PhoenixmlDb.XQuery

WordTokenizer

Finds the words of a text by the word boundary rules of Unicode Standard Annex #29 (Unicode 6.3 data), and returns those that hold a letter, a digit or an ideograph. Three additions, which the annex leaves to the implementation, keep the tokens of the earlier analysis: a run of Line_Break=Complex_Context characters (Thai, Lao, Khmer, Myanmar) is one token; a Han or Hiragana character is a token by itself; and digits in the Halfwidth and Fullwidth Forms block are numeric.

#WordTokenizer

Namespace: PhoenixmlDb.XQuery.FullText

Finds the words of a text by the word boundary rules of Unicode Standard Annex #29 (Unicode 6.3 data), and returns those that hold a letter, a digit or an ideograph. Three additions, which the annex leaves to the implementation, keep the tokens of the earlier analysis: a run of Line_Break=Complex_Context characters (Thai, Lao, Khmer, Myanmar) is one token; a Han or Hiragana character is a token by itself; and digits in the Halfwidth and Fullwidth Forms block are numeric.

#Methods

#AttachExtends(String,Int32)

The index after any Extend and Format characters from

(WB4).

#MatchComplexContext(String,Int32)

A run of Complex_Context characters; their own combining marks are Complex_Context too, but no other Extend or Format character joins the run.

#MatchWord(String,Int32)

The end of the longest word token from

, or -1.

#SkipRegionalIndicators(String,Int32)

The end of two or more regional indicators with their attached marks, or -1.

#Tokenize(String,Collections.Generic.List<ValueTuple<Int32,Int32>>)

Adds the [start, end) UTF-16 span of each token of

.

#Unit(String,Int32,Int32@)

A character with its attached Extend and Format characters.