tokenizer 0.2.0

A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.

Gemfile:
=

install:
=

Versions:

0.3.0 January 20, 2016 (10 KB)
0.2.0 January 11, 2016 (10 KB)
0.1.2 September 03, 2015 (9.5 KB)
0.1.1 August 25, 2011 (8 KB)
0.1.0 May 19, 2011 (5.5 KB)

Show all versions (6 total)

Owners:

Authors:

Andrei Beliankou

SHA 256 checksum:

=

Total downloads 236,117

For this version 73,195

Version Released:
January 11, 2016 5:09pm

License:
MIT

Required Ruby Version: >= 1.9.3

Links:

Homepage Documentation Bug Tracker Download Review changes Badge RSS Report abuse Reverse dependencies