org.opensolaris.opengrok.analysis
Class JFlexTokenizer
java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.opensolaris.opengrok.analysis.JFlexTokenizer
- All Implemented Interfaces:
- java.io.Closeable
- Direct Known Subclasses:
- CSymbolTokenizer, CxxSymbolTokenizer, FortranSymbolTokenizer, JavaSymbolTokenizer, LispSymbolTokenizer, PlainFullTokenizer, PlainSymbolTokenizer, ShSymbolTokenizer, TclSymbolTokenizer, TroffFullTokenizer
public abstract class JFlexTokenizer
- extends org.apache.lucene.analysis.Tokenizer
this class was created because of lucene 2.4.1 update which introduced char[] in Tokens instead of String
lucene 3.0.0 uses AttributeSource instead of Tokens to make things even easier :-D
Generally this is a "template" for all new Tokenizers, so be carefull when changing it,
it will impact almost ALL symbol tokenizers in OpenGrok ...
Created on August 24, 2009
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource |
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State |
Field Summary |
protected org.apache.lucene.analysis.tokenattributes.OffsetAttribute |
offsetAtt
|
protected org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute |
posIncrAtt
|
protected org.apache.lucene.analysis.tokenattributes.TermAttribute |
termAtt
|
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
input |
Method Summary |
boolean |
incrementToken()
This will reinitalize internal AttributeImpls, or it returns false if end of input Reader ... |
protected void |
setAttribs(java.lang.String str,
int start,
int end)
|
abstract boolean |
yylex()
|
Methods inherited from class org.apache.lucene.analysis.Tokenizer |
close, correctOffset, reset |
Methods inherited from class org.apache.lucene.analysis.TokenStream |
end, reset |
Methods inherited from class org.apache.lucene.util.AttributeSource |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString |
Methods inherited from class java.lang.Object |
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
termAtt
protected org.apache.lucene.analysis.tokenattributes.TermAttribute termAtt
offsetAtt
protected org.apache.lucene.analysis.tokenattributes.OffsetAttribute offsetAtt
posIncrAtt
protected org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute posIncrAtt
JFlexTokenizer
public JFlexTokenizer()
yylex
public abstract boolean yylex()
throws java.io.IOException
- Throws:
java.io.IOException
incrementToken
public boolean incrementToken()
throws java.io.IOException
- This will reinitalize internal AttributeImpls, or it returns false if end of input Reader ...
- Specified by:
incrementToken
in class org.apache.lucene.analysis.TokenStream
- Returns:
- false if no more tokens, otherwise true
- Throws:
java.io.IOException
setAttribs
protected void setAttribs(java.lang.String str,
int start,
int end)