Various properties of this Tokenizer. More...
#include <zorba/tokenizer.h>
Public Types | |
typedef std::vector < locale::iso639_1::type > | languages_type |
Public Attributes | |
bool | comments_separate_tokens |
If true , XML comments separate tokens. | |
bool | elements_separate_tokens |
If true , XML elements separate tokens. | |
languages_type | languages |
The set of languages supported. | |
bool | processing_instructions_separate_tokens |
If true , XML processing instructions separate tokens. | |
char const * | uri |
The URI that uniquely identifies this Tokenizer. |
Various properties of this Tokenizer.
Definition at line 112 of file tokenizer.h.
typedef std::vector<locale::iso639_1::type> zorba::Tokenizer::Properties::languages_type |
Definition at line 113 of file tokenizer.h.
bool zorba::Tokenizer::Properties::comments_separate_tokens |
If true
, XML comments separate tokens.
For example, net<!—->work
would be 2 tokens instead of 1.
Definition at line 119 of file tokenizer.h.
bool zorba::Tokenizer::Properties::elements_separate_tokens |
If true
, XML elements separate tokens.
For example, <b>B</b>old
would be 2 tokens instead of 1.
Definition at line 125 of file tokenizer.h.
languages_type zorba::Tokenizer::Properties::languages |
The set of languages supported.
Definition at line 136 of file tokenizer.h.
bool zorba::Tokenizer::Properties::processing_instructions_separate_tokens |
If true
, XML processing instructions separate tokens.
For example, net<?PI pi?>work
would be 2 tokens instead of 1.
Definition at line 131 of file tokenizer.h.
char const* zorba::Tokenizer::Properties::uri |
The URI that uniquely identifies this Tokenizer.
Definition at line 141 of file tokenizer.h.