Skip to content
Snippets Groups Projects
  1. Jun 01, 2017
  2. May 09, 2017
    • Akihiko Odaki's avatar
      Use CLD3 (#2949) · d5cabfe5
      Akihiko Odaki authored
      Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
      used in the previous implementation. CLD3 includes improvements since CLD2,
      and supports newer compilers. On the other hand, it has additional
      requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
      be still inmature.
      
      Though CLD3 is named after CLD2, it is implemented with a neural network
      model, different from the old implementation, which is based on a Naïve
      Bayesian classifier.
      
      CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
      GCC 6 because it assigns negative values to varibales typed unsigned.
      (see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
      newer compilers are essential today, when some server operating system
      such as Ubuntu Server 16.10 has GCC 6 by default.
      
      On the one hand, CLD3 requires C++11 support. Environments with old
      compilers such as Ubuntu Server 14.04 needs to update the system or install
      a newer compiler.
      
      CLD3 needs protocol buffers as a new dependency. However,it is not
      considered problematic because major server operating systems, CentOS and
      Ubuntu Server provide them.
      
      The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
      It is still new and may be inmature, but confirmed to pass existing tests.
      d5cabfe5
  3. May 08, 2017
  4. May 03, 2017
  5. Apr 22, 2017
  6. Apr 18, 2017
    • Matt Jankowski's avatar
      Language detection refactor (#2099) · 297c11db
      Matt Jankowski authored
      * Extract detect_language to separate class
      
      * Use default locale, not just en
      
      * Add spec to confirm that whatlanguage cant identify empty string
      
      * Allow account locale to override default in language detector
      
      * PostStatusService supplies an account to detect language
      297c11db
Loading