The Zorba XQuery engine implements the XQuery and XPath Full Text 1.0 specification that, among other things, adds the ability to use a thesaurus for text-matching via the thesaurus option. For example, the query:
returns true
because $x
contains "wealthy" that the thesaurus identified as a synonym of "affluent".
The initial implementation of the thesaurus option uses the WordNet lexical database, version 3.0.
The stock WordNet database files are plain ASCII text files. In many ways this is very convenient for portability, grep-ability, vi-ability, etc. However, the sum total of the files is approximately 27MB (which is quite large) and accessing the database would be inefficient since the files would have to be parsed for every access.
Instead, the database files are compiled into a single binary file that is 6MB and can be efficiently accessed from Zorba using mmap(2) with no parsing of the data. The only caveat of the binary format is that it is endian-dependent, i.e., a binary file created on a computer having a little-endian CPU won't work on a computer having a big-endian CPU.
To download and install the WordNet database on a Unix-like system, follow these steps:
WNdb-3.0.tar.gz
)./usr/local/wordnet-3.0/dict
.dict
directory into a Zorba-compatible binary thesaurus as described below.To compile the WordNet database files, use the zt-wn-compile
script found in the scripts
subdirectory of the Zorba distribution. (Note: this script is written in perl.) The usage message is:
-v
option specifies verbose output.dict
directory.wordnet-en.zth
("en" for English and "zth" for "Zorba Thesaurus file").For example:
To install the wordnet-en.zth
file, move it onto Zorba's library path:
Alternatively, you can download a precompiled, little-endian (Intel) CPU WordNet database from here.
In order to use thesauri, you need to specify what symbolic URI(s) map to what thesauri. A mapping is of the form:
from_uri:=
implementation-scheme:
to_uri
For example:
says that the symbolic URI http://wordnet.princeton.edu
maps to the WordNet implementation having a database file at the given sub-path edu/princeton/wordnet
on Zorba's library path. Once a mapping is established for a symbolic URI, it can be used in a query:
As a special-case, the from_uri can be default
or ##default
to allow for specifying the default thesaurus as was done for the first example on this page.
To specify the location of the thesaurus to Zorba from the command-line, use one or more –thesaurus options:
Using the WordNet database, Zorba supports all of the thesaurus relationships specified by [ISO 2788] and [ANSI/NISO Z39.19-2005] with the exceptions of "HN" (history note) and "X SN" (see scope note for).
These relationships are:
Rel. | Meaning | WordNet Rel. |
---|---|---|
BT | broader term | hypernym |
BTG | broader term generic | hypernym |
BTI | broader term instance | instance hypernym |
BTP | broader term partitive | part meronym |
NT | narrower term | hyponym |
NTG | narrower term generic | hyponym |
NTI | narrower term instance | instance hyponym |
NTP | narrower term partitive | part holonym |
RT | related term | also see |
SN | scope note | n/a |
TT | top term | hypernym |
UF | non-preferred term | n/a |
USE | preferred term | n/a |
and can be used in a query like:
that returns true
because $x
contains "breakfast" that the thesaurus identified as a "narrower term" (NT) of "meal."
Note that you can specify relationships either by their abbreviation or their meaning. Relationships are case-insensitive. The above query is equivalent to:
Since Zorba's thesaurus is implemented using WordNet, the [ISO 2788] relationships map to WordNet relationships that are shown in the "WordNet Rel." column. WordNet relationships are explained in the next section.
In addition to the [ISO 2788] and [ANSI/NISO Z39.19-2005] relationships, Zorba also supports all of the relationships offered by WordNet. These relationships are:
Relationship | Meaning |
---|---|
also see | A word that is related to another, e.g., for "varnished" (furniture) one should also see "finished." |
antonym | A word opposite in meaning to another, e.g., "light" is an antonym for "heavy." |
attribute | A noun for which adjectives express values, e.g., "weight" is an attribute for which the adjectives "light" and "heavy" express values. |
cause | A verb that causes another, e.g., "show" is a cause of "see." |
derivationally related form | A word that is derived from a root word, e.g., "metric" is a derivationally related form of "meter." |
derived from adjective | An adverb that is derived from an adjective, e.g., "correctly" is derived from the adjective "correct." |
entailment | A verb that presupposes another, e.g., "snoring" entails "sleeping." |
hypernym | A word with a broad meaning that more specific words fall under, e.g., "meal" is a hypernym of "breakfast." |
hyponym | A word of more specific meaning than a general term applicable to it, e.g., "breakfast" is a hyponym of "meal." |
instance hypernym | A word that denotes a category of some specific instance, e.g., "author" is an instance hypernym of "Asimov." |
instance hyponym | A term that donotes a specific instance of some general category, e.g., "Asimov" is an instance hyponym of "author." |
member holonym | A word that denotes a collection of individuals, e.g., "faculty" is a member holonym of "professor." |
member meronym | A word that denotes a member of a larger group, e.g., a "person" is a member meronym of a "crowd." |
part holonym | A word that denotes a larger whole comprised of some part, e.g., "car" is a part holonym of "engine." |
part meronym | A word that denotes a part of a larger whole, e.g., an "engine" is part meronym of a "car." |
participle of verb | An adjective that is the participle of some verb, e.g., "breaking" is the participle of the verb "break." |
pertainym | An adjective that classifies its noun, e.g., "musical" is a pertainym in "musical instrument." |
similar to | Similar, though not necessarily interchangeable, adjectives. For example, "shiny" is similar to "bright", but they have subtle differences. |
substance holonym | A word that denotes a larger whole containing some constituent substance, e.g., "bread" is a substance holonym of "flour." |
substance meronym | A word that denotes a constituant substance of some larger whole, e.g., "flour" is a substance meronym of "bread." |
verb group | A verb that is a member of a group of similar verbs, e.g., "live" is in the verb group of "dwell", "live", "inhabit", etc. |
If no levels are specified in a query, Zorba defaults the WordNet implementation to be 2 levels. (The rationale can be found here.)
Using the Zorba C++ API, you can provide your own thesaurus by deriving from four classes: Thesaurus
, Thesaurus::iterator
, ThesaurusProvider
, and URLResolver
.
The Thesaurus
class is:
For details about the ptr
types, the destroy()
functions, and why the destructors are protected
, see the Memory Management document.
To implement the Thesaurus
you need to implement the lookup()
function where:
phrase | The phrase to be looked-up. |
relationship | The relationship the results are to have to the phrase, if any. |
at_least | The minimum number of levels within the thesaurus to be traversed. |
at_most | The maximum number of levels within the thesaurus to be traversed. |
The lookup()
function returns a pointer to an iterator
that is used to iterate over the phrase's synonyms. You also need to implement an iterator
. A very simple Thesaurus
and its iterator
can be implemented as:
A real thesaurus would load a large number of synonyms, of course.
The ThesaurusProvider
class is:
To implement a ThesaurusProvider
, you need to implement the getThesaurus()
function where:
lang | The desired language of the thesaurus. |
thesaurus | If not null , set to point to a thesaurus for lang . |
The getThesaurus()
function returns true
only if it can provide a thesaurus for the given language. Continuing with the example, a very simple ThesaurusProvider
can be implemented as:
In addition to a Thesaurus
and ThesaurusProvider
, you must also implement a "thesaurus resolver" class that, given a URI, provides a ThesaurusProvider
for that URI. A simple ThesaurusURLResolver
for our simple thesaurus can be implemented as:
For more on URLResolver
, see URI Resolvers.
To enable your thesaurus to be used, you need to add it to a static context:
You can then perform a query using your thesaurus: