Public Member Functions
zorba::TokenizerProvider Class Reference

A TokenizerProvider provides a Tokenizer for a given language. More...

#include <zorba/tokenizer.h>

List of all members.

Public Member Functions

virtual bool getTokenizer (locale::iso639_1::type lang, Tokenizer::State *state=0, Tokenizer::ptr *t=0) const =0
 Creates a new Tokenizer.
virtual ~TokenizerProvider ()

Detailed Description

A TokenizerProvider provides a Tokenizer for a given language.

Definition at line 284 of file tokenizer.h.


Constructor & Destructor Documentation

virtual zorba::TokenizerProvider::~TokenizerProvider ( )
virtual

Member Function Documentation

virtual bool zorba::TokenizerProvider::getTokenizer ( locale::iso639_1::type  lang,
Tokenizer::State state = 0,
Tokenizer::ptr t = 0 
) const
pure virtual

Creates a new Tokenizer.

Parameters:
langThe language of the text that the tokenizer will tokenize.
stateThe State to use. If null, t is not set.
tIf not null, set to point to a Tokenizer for lang.
Returns:
Returns true only if this provider can provide a tokenizer for lang.

The documentation for this class was generated from the following file:
blog comments powered by Disqus