Class ICUTransformFilter
- All Implemented Interfaces:
Closeable
,AutoCloseable
,Unwrappable<TokenStream>
TokenFilter
that transforms text with ICU.
ICU provides text-transformation functionality via its Transliteration API. Although script conversion is its most common use, a Transliterator can actually perform a more general class of tasks. In fact, Transliterator defines a very general API which specifies only that a segment of the input text is replaced by new text. The particulars of this conversion are determined entirely by subclasses of Transliterator.
Some useful transformations for search are built-in:
- Conversion from Traditional to Simplified Chinese characters
- Conversion from Hiragana to Katakana
- Conversion from Fullwidth to Halfwidth forms.
- Script conversions, for example Serbian Cyrillic to Latin
Example usage:
stream = new ICUTransformFilter(stream, Transliterator.getInstance("Traditional-Simplified"));
For more details, see the ICU User Guide.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescription(package private) static final class
Wrap aCharTermAttribute
with the Replaceable API.Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final com.ibm.icu.text.Transliterator.Position
private final ICUTransformFilter.ReplaceableTermAttribute
private final CharTermAttribute
private final com.ibm.icu.text.Transliterator
Fields inherited from class org.apache.lucene.analysis.TokenFilter
input
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
Constructor Summary
ConstructorsConstructorDescriptionICUTransformFilter
(TokenStream input, com.ibm.icu.text.Transliterator transform) Create a new ICUTransformFilter that transforms text on the given stream. -
Method Summary
Modifier and TypeMethodDescriptionboolean
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token.Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end, reset, unwrap
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
Field Details
-
transform
private final com.ibm.icu.text.Transliterator transform -
position
private final com.ibm.icu.text.Transliterator.Position position -
termAtt
-
replaceableAttribute
-
-
Constructor Details
-
ICUTransformFilter
Create a new ICUTransformFilter that transforms text on the given stream.- Parameters:
input
-TokenStream
to filter.transform
- Transliterator to transform the text.
-
-
Method Details
-
incrementToken
Description copied from class:TokenStream
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriateAttributeImpl
s with the attributes of the next token.The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
andAttributeSource.getAttribute(Class)
, references to allAttributeImpl
s that this stream uses should be retrieved during instantiation.To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in
TokenStream.incrementToken()
.- Specified by:
incrementToken
in classTokenStream
- Returns:
- false for end of stream; true otherwise
- Throws:
IOException
-