Public Types | Public Member Functions
zorba::Tokenizer::Callback Class Reference

A Callback is called once per token. More...

#include <zorba/tokenizer.h>

List of all members.

Public Types

typedef Tokenizer::size_type size_type

Public Member Functions

virtual void item (Item const &item, bool entering)
 This member-function is called whenever an item that is being tokenized is entered or exited.
virtual void token (char const *utf8_s, size_type utf8_len, locale::iso639_1::type lang, size_type token_no, size_type sent_no, size_type para_no, Item const *item=0)=0
 This member-function is called once per token.
virtual ~Callback ()

Detailed Description

A Callback is called once per token.

This is only internally by Zorba. You do not need to derive from this class.

Definition at line 74 of file tokenizer.h.


Member Typedef Documentation

Definition at line 76 of file tokenizer.h.


Constructor & Destructor Documentation

virtual zorba::Tokenizer::Callback::~Callback ( )
virtual

Member Function Documentation

virtual void zorba::Tokenizer::Callback::item ( Item const &  item,
bool  entering 
)
virtual

This member-function is called whenever an item that is being tokenized is entered or exited.

The default implementation does nothing.

 @param item The item being entered or exited.
 @param entering If \c true, the item is being entered; if \c false, the
 item is being exited.
virtual void zorba::Tokenizer::Callback::token ( char const *  utf8_s,
size_type  utf8_len,
locale::iso639_1::type  lang,
size_type  token_no,
size_type  sent_no,
size_type  para_no,
Item const *  item = 0 
)
pure virtual

This member-function is called once per token.

Parameters:
utf8_sThe UTF-8 token string. It is not null-terminated.
utf8_lenThe number of bytes in the token string.
langThe language of the token.
token_noThe token number. Token numbers start at 0.
sent_noThe sentence number. Sentence numbers start at 1.
para_noThe paragraph number. Paragraph numbers start at 1.
itemThe Item this token is from, if any.

The documentation for this class was generated from the following file:
blog comments powered by Disqus