Public Types | Public Attributes | List of all members
zorba::Tokenizer::Properties Struct Reference

Various properties of this Tokenizer. More...

#include <zorba/tokenizer.h>

Public Types

typedef std::vector
< locale::iso639_1::type
languages_type
 

Public Attributes

bool comments_separate_tokens
 If true, XML comments separate tokens. More...
 
bool elements_separate_tokens
 If true, XML elements separate tokens. More...
 
languages_type languages
 The set of languages supported. More...
 
bool processing_instructions_separate_tokens
 If true, XML processing instructions separate tokens. More...
 
char const * uri
 The URI that uniquely identifies this Tokenizer. More...
 

Detailed Description

Various properties of this Tokenizer.

Definition at line 112 of file tokenizer.h.

Member Typedef Documentation

Definition at line 113 of file tokenizer.h.

Member Data Documentation

bool zorba::Tokenizer::Properties::comments_separate_tokens

If true, XML comments separate tokens.

For example, net<!-—>work would be 2 tokens instead of 1.

Definition at line 119 of file tokenizer.h.

bool zorba::Tokenizer::Properties::elements_separate_tokens

If true, XML elements separate tokens.

For example, <b>B</b>old would be 2 tokens instead of 1.

Definition at line 125 of file tokenizer.h.

languages_type zorba::Tokenizer::Properties::languages

The set of languages supported.

Definition at line 136 of file tokenizer.h.

bool zorba::Tokenizer::Properties::processing_instructions_separate_tokens

If true, XML processing instructions separate tokens.

For example, net<?PI pi?>work would be 2 tokens instead of 1.

Definition at line 131 of file tokenizer.h.

char const* zorba::Tokenizer::Properties::uri

The URI that uniquely identifies this Tokenizer.

Definition at line 141 of file tokenizer.h.


The documentation for this struct was generated from the following file:
blog comments powered by Disqus