PhoenixmlDb.XQuery
WordTokenizer
Finds the words of a text by the word boundary rules of Unicode Standard Annex #29 (Unicode 6.3 data), and returns those that hold a letter, a digit or an ideograph. Three additions, which the annex leaves to the implementation, keep the tokens of the earlier analysis: a run of Line_Break=Complex_Context characters (Thai, Lao, Khmer, Myanmar) is one token; a Han or Hiragana character is a token by itself; and digits in the Halfwidth and Fullwidth Forms block are numeric.
#WordTokenizer
Namespace: PhoenixmlDb.XQuery.FullText
Finds the words of a text by the word boundary rules of Unicode Standard Annex #29 (Unicode 6.3 data), and returns those that hold a letter, a digit or an ideograph. Three additions, which the annex leaves to the implementation, keep the tokens of the earlier analysis: a run of Line_Break=Complex_Context characters (Thai, Lao, Khmer, Myanmar) is one token; a Han or Hiragana character is a token by itself; and digits in the Halfwidth and Fullwidth Forms block are numeric.
#Methods
#AttachExtends(String,Int32)
The index after any Extend and Format characters from
(WB4).
#MatchComplexContext(String,Int32)
A run of Complex_Context characters; their own combining marks are Complex_Context too, but no other Extend or Format character joins the run.
#MatchWord(String,Int32)
The end of the longest word token from
, or -1.
#SkipRegionalIndicators(String,Int32)
The end of two or more regional indicators with their attached marks, or -1.
#Tokenize(String,Collections.Generic.List<ValueTuple<Int32,Int32>>)
Adds the [start, end) UTF-16 span of each token of
.
#Unit(String,Int32,Int32@)
A character with its attached Extend and Format characters.