标签：his break lib and any code rdp var rip
A part of Natural Language Processing (NLP) is processing text by “tokenizing” language strings. This means we can break up a string of text into parts by word, sentence, etc. In this lesson, we will use the natural library to tokenize a string. First, we will break the string into words using WordTokenizer, WordPunctTokenizer, and TreebankWordTokenizer. Then we will break the string into sentences using RegexpTokenizer.
var natural = require(‘natural‘), tokenizer = new natural.WordTokenizer(); console.log(tokenizer.tokenize("your dog has fleas.")); // [ ‘your‘, ‘dog‘, ‘has‘, ‘fleas‘ ]
tokenizer = new natural.TreebankWordTokenizer(); console.log(tokenizer.tokenize("my dog hasn‘t any fleas.")); // [ ‘my‘, ‘dog‘, ‘has‘, ‘n\‘t‘, ‘any‘, ‘fleas‘, ‘.‘ ] tokenizer = new natural.RegexpTokenizer({pattern: /\-/}); console.log(tokenizer.tokenize("flea-dog")); // [ ‘flea‘, ‘dog‘ ] tokenizer = new natural.WordPunctTokenizer(); console.log(tokenizer.tokenize("my dog hasn‘t any fleas.")); // [ ‘my‘, ‘dog‘, ‘hasn‘, ‘\‘‘, ‘t‘, ‘any‘, ‘fleas‘, ‘.‘ ]
[Javascript Natural] Break up language strings into parts using Natural
标签：his break lib and any code rdp var rip
原文地址：http://www.cnblogs.com/Answer1215/p/6041263.html
