TokenStream::NEW (open)

Create a new TokenStream

Syntax

LOADLIB "wh::util/langspecific.whlib";

MACRO NEW(STRING text, STRING langcode)

Parameters

STRING text

The text to parse

STRING langcode

The two-letter code of the language to use for normalizing and stemming