Introduction to textwolf

How to Use textwolf

To use textwolf you have to include include/textwolf.hpp

#include "textwolf.hpp"

The namespace textwolf is defined there for all templates introduced here.

W3C Standards and textwolf

textwolf follows the standard of W3C XML 1.0 but does not implement all of it. The following list shows the exceptions:

Documents are not validated against any schema definition and they are assumed to be well formed.
Only entity mappings to single characters are supported.
ENTITY definitions in the XML are ignored. Entity definitions have to be passed as read only table.
Declarative "<! ... >" definitions like DOCTYPE, ELEMENT and ATTLIST are ignored (skipped over). Changing of tokenization (ATTLIST) has to be done by calling the "doTokenize(bool)" method of the scanner or XML path parser. You can do this at any point when you are iterating through the document.
Commands of the form "<? ..." like PI (process instructions) are ignored (skipped over).
Context information given by namespaces is not treated specially. A namespace prefix with colon is treated as part of the tag name and not as context information. For example an attribute "xmlns:xsi" will be treated as an attribute with name "xmlns:xsi".

Wide Character Set Encodings

textwolf is a template library that is instantiated with and encoding type as template parameter. If you want to parse the character set encoding according to what is in the XML header, you have to parse the XML header first to determine the character set encoding. Then you can use the right instance of textwolf.

Definitions

textwolf defines the following templates and types

TextScanner

The TextScanner class template defines an iterator on the characters of the input as

Unicode characters
ASCII characters
XML control characters

Template parameters

InputIterator (explained in the section iterator)
Charset = character set encoding of the input (explained in the section character set encodings)

Example

char* input = ...;
textwolf::TextScanner<char*,charset::UCS2<LE> > itr( input);
while (itr->chr()) itr++;

XMLScanner

The XMLScanner class template defines the state of a parser scanning the XML elements like tags,atrributes,values,content,etc.. on a source defined with an input iterator.

Template parameters

InputIterator = input iterator type (explained in the section iterator)
InputCharSet = character set encoding of the input (explained in the section character set encodings)
OutputCharSet = character set encoding of the output (explained in the section character set encodings)
OutputBuffer = buffer type to use for the tokens parsed (STL back insertion sequence interface)

Constructor arguments

InputIterator& src = an input iterator reference
OutputBuffer outbuf = the buffer to use for output
EntityMap* emap = pointer to a read only map of entities (optional)

Example

char* input = ...;
std::string outputbuf;
typedef textwolf::XMLScanner
	<
		char*,
		charset::IsoLatin,
		charset::IsoLatin,
		std::string
	> Scan;
Scan xs( input, outputbuf);
for (Scan::iterator itr = xs.begin(); itr != xs.end(); itr++)
{
	switch (itr->type())
	{
		case MyXMLScanner::ErrorOccurred: throw(...);
		case MyXMLScanner::OpenTag: ... break;
		case MyXMLScanner::Content: ... break;
		...
	}
}

End of tag events come without tag name. So it is not possible to validate an XML with 'textwolf'.

XMLPathSelectAutomaton

This class template defines an automaton for selecting XML path expression and to assign a type as integer to the filtered tokens. It has one template argument:

OutputCharSet = the character set in which the tokens are stored and therefore in which format they are processed.

The automaton construction is described in the section "how to define an XML path expression automaton".

XMLPathSelect

The XMLPathSelect class template defines the state of a set of XML path selections to filter the output of an XMLScanner iterator. It is constructed by passing

textwolf::Automaton* atm = the pointer to an XML path selection automaton

The class template definition has the following parameters

CharSet = character set encoding used for processing

The XMLPathSelect class is fed with the output of an XMLScanner. The idea is to iterate on an input with XMLScanner and to push every element fetched to the XMLPathSelect. After every push you can iterate on the new results you got with this push.

Character Set Encodings

Predefined

The following character set encodings are defined in the textwolf::charset namespace as examples:

UTF8 = utf-8
UTF16LE = utf-16 Litle Endian
UTF16BE = utf-16 (Big Endian = default for utf-16)
IsoLatin = Iso-Latin-1 .. Iso-Latin-9 (depending on codepage)
UCS2LE = UCS2 Little Endian
UCS2BE = UCS2 Big Endian
UCS4LE = UCS4 Little Endian
UCS4BE = UCS4 Big Endian

How to Define Your Own

You can define your own character set encodings. A structure with the following interface is passed to a textwolf iterator as a character set encoding definition. Textwolf assumes some form of a single Unicode character (UChar) to be able to map them to each other.

/// \class Interface
/// \brief Interface that has to be implemented for a character set encoding
struct Interface
{
	/// \brief Defines the highest unicode character representation 
	///	of a character in this encoding.
	enum {MaxChar=0xFF};

	/// \brief Skip to start of the next character
	/// \param [in] buf buffer for the character data
	/// \param [in,out] bufpos position in 'buf'
	/// \param [in,out] itr iterator to skip
	template <class Iterator>
	static void skip( char* buf, unsigned int& bufpos, Iterator& itr);

	/// \brief Fetches the ascii char representation of the current character
	/// \param [in] buf buffer for the parses character data
	/// \param [in,out] bufpos position in 'buf'
	/// \param [in,out] itr iterator on the source
	/// \return the value of the ascii character or -1
	template <class Iterator>
	static char asciichar( char* buf, unsigned int& bufpos, Iterator& itr);

	/// \brief Fetches the bytes of the current character into a buffer
	/// \param [in] buf buffer for the parses character data
	/// \param [in,out] bufpos position in 'buf'
	/// \param [in,out] itr iterator on the source
	template <class Iterator>
	static void fetchbytes( char* buf, unsigned int& bufpos, Iterator& itr);

	/// \brief Fetches the unicode character representation of the current character
	/// \param [in] buf buffer for the parses character data
	/// \param [in,out] bufpos position in 'buf'
	/// \param [in,out] itr iterator on the source
	/// \return the value of the unicode character
	template <class Iterator>
	static UChar value( char* buf, unsigned int& bufpos, Iterator& itr);

	/// \brief Prints a unicode character to a buffer
	/// \tparam Buffer STL back insertion sequence
	/// \param [in] chr character to print
	/// \param [out] buf buffer to print to
	template <class Buffer>_
	static void print( UChar chr, Buffer& buf);
};

Iterators

Input

The TextScanner and XMLScanner template classes expect an input iterator on a sequence of bytes with the following properties:

It implements pre increment:
```
iterator& operator++()
```
It implements one byte access, that returns a sequence of 0 (zeros) after reaching end of data:
```
char operator*() const
```

The following iterators for various purposes are already defined:

CStringIterator (cstringiterator.hpp): Defines an iterator on a complete C string
IStreamIterator (istreamiterator.hpp): Defines an iterator on a std::istream
SrcIterator (sourceiterator.hpp): Defines an iterator on C buffers for chunk-wise processing

The following example shows how you can define your own input iterator. It shows a simple textwolf iterator based on an STL iterator. You have to define a structure like this:

template <typename iterator>
struct twiterator
{
	iterator m_itr;
	iterator m_end;

	twiterator( const iterator& b, const iterator& e)
			:m_itr(b),m_end(e) {}

	char operator*() const
	{
		if (m_itr >= m_end)
		{
			return 0;
		}
		else
		{
			return *m_itr;
		}
	}
	twiterator& operator++()
	{
		++m_itr; return *this;
	}
};

Chunkwise Processing of Input

For chunk by chunk feeding of input a longjmp structure can be defined and passed to the source iterator. The iterator can issue a longjump if it reaches the end of the current chunk processed. The longjmp structure has to be initialized by the caller of textwolf. textwolf just ensures to save its state and that it can be called again, if it has data again and can continue. Because textwolf is completely table driven it has no problem to save its state in a stable way. The caller can continue iteration when the next chunk is prepared and has been passed to the source iterator. If no such longjmp structure is defined then the end of chunk is also seen as the end of the data. The following example shows the iterator from the example above with a longjmp structure declared for handling end of data in chunkwise processing of input:

template <typename iterator>
struct twiterator
{
	iterator m_itr;
	iterator m_end;
	jmp_buf* m_eom;

	twiterator( const iterator& b, const iterator& e, jmp_buf* eom=0)
			:m_itr(b),m_end(e),m_eom(eom) {}

	char operator*() const
	{
		if (m_itr >= m_end)
		{
			if (m_eom) longjmp(*m_eom,1);
			return 0;
		}
		else
		{
			return *m_itr;
		}
	}
	twiterator& operator++()
	{
		++m_itr; return *this;
	}
};

Using the iterator example for chunk by chunk data processing:

jmp_buf eom;
typedef twiterator ItrType;
std::string outputbuf;
typedef textwolf::XMLScanner
	<
		ItrType,
		charset::IsoLatin,
		charset::IsoLatin,
		std::string
	> Scan;
std::string chunk = <first chunk>;
ItrType itr;
Scan xs( ItrType( chunk.begin(), chunk.end(), &eom), outputbuf);
Scan::iterator itr = xs.begin();

if (setjmp(eom) != 0)
{
	//... the iterator processed the whole chunk and needs a new one
	chunk = <call for getting the next chunk>;
	xs.setSource( ItrType( chunk.begin(), chunk.end(), &eom));
}
for (; itr != xs.end(); itr++)
{
	... < processing the found elements > ...
}

Output

The following iterators refer to InputIterator as the required input iterator type (see input):

Iterator on the characters of the input

InputIterator in = input.begin();
textwolf::TextScanner
	<
		InputIterator,
		textwolf::charset::UTF8
	> itr( in);
for (; itr.control() != textwolf::EndOfText; itr++)
{
	UChar chr = *itr; // parse the unicode character
}

Iterator on the XML element (tags,attributes,values,etc.) of the input

typedef XMLScanner
	<
		InputIterator,
		charset::IsoLatin,
		charset::IsoLatin,
		std::string
	> MyXMLScanner;
std::string outputbuf;
InputIterator in = input.begin();
MyXMLScanner xs( in, outputbuf);

MyXMLScanner::iterator itr;
for (itr=xs.begin(); itr != xs.end(); itr++)
{
	const char* typestr = 0;
	switch (itr->type())
	{
		case MyXMLScanner::ErrorOccurred:
			throw std::runtime_error( itr->content());
		case MyXMLScanner::HeaderEnd:
			typestr = "end of header"; break;
		case MyXMLScanner::DocAttribValue:
			typestr = "entity token"; break;
		case MyXMLScanner::DocAttribEnd:
			typestr = "end of entity definition"; break;
		case MyXMLScanner::TagAttribName:
		case MyXMLScanner::HeaderAttribName:
			typestr = "attribute name"; break;
		case MyXMLScanner::TagAttribValue:
		case MyXMLScanner::HeaderAttribValue:
			typestr = "attribute value"; break;
		case MyXMLScanner::OpenTag:
			typestr = "open tag"; break;
		case MyXMLScanner::CloseTag:
		case MyXMLScanner::CloseTagIm:
			typestr = "close tag"; break;
		case MyXMLScanner::Content:
			typestr = "content"; break;
		case MyXMLScanner::Exit:
			typestr = "end of document"; break;
	}
	std::cout
		<< "Element "
		<< typestr
		<< " ("
		<< itr->name()
		<< "): "
		<< itr->content()
		<< std::endl;
	if (itr->type() == MyXMLScanner::ErrorOccurred) break;
}

Iterator on the XML path expressions filtered in the input

//define the automaton
typedef XMLPathSelectAutomaton<charset::UTF8> Automaton;
Automaton atm;
//return '1' for all "//element/@id" attribute values:
(*atm)--["element"]("id") = 1;
//return '2' for all text content elements of "/doc":
(*atm)["doc"]() = 2;

// define the selection on the source by an automaton:
typedef XMLPathSelect<charset::UTF8> PathSelect;
typedef XMLScanner<char*,charset::UTF8,charset::UTF8,std::string> Scanner;

Scanner xc( src);
PathSelect xs( &atm);

// iterating through the source tokens
Scanner::iterator ci,ce;
for (ci=xc.begin(),ce=xc.end(); ci!=ce; ci++)
{
	// feed the automaton with each token encountered
	MyXMLPathSelect::iterator
		itr = xs.push( ci->type(), ci->content(), ci->size()),
		end=xs.end();

	// iterate through the result elements of the last token fed
	for (; itr!=end; itr++)
	{
		...
	}
}

How to define an XML path expression automaton

An XML Path expression automaton in defined as tree. Every expression defines a set containing the root node as start of the selection. Subsequent node selections in the expression refer to the previously selected nodes and create a new set of selected nodes. The set of nodes you get at the end of the selection defines the set of nodes selected by the expression. The following operators are defined for building node selection expressions.
For a node 'A'

select the tag with the name "doc" following node A
```
A["doc"]
```
or
```
A.selectTag("doc")
```
select the value of the attribute "id"
```
A("id")
```
or
```
A.selectAttribute("id")
```
seek the attribute "id" (also defined as function ifAttribute)
```
A("id",0)
```
or
```
A.ifAttribute("id",0)
```
seek the attribute "id" with value "188" (same ifAttribute)
```
A("id","188")
```
or
```
A.ifAttribute("id","188")
```
select all content values of the node selected by A (also as function selectContent)
```
A()
```
or
```
A.selectContent()
```
stop selecting elements with index beyond 24 counted from 0
```
A.TO(25)
```
ignore elements with an index smaller than 2 (counted from 0)
```
A.FROM(2)
```
assign the value 3 as type to all values selected by the expression A (also defined as function assignType)
```
A = 3
```
or
```
A.assignType(3)
```
Select all nodes below the current node, so that all conditions expressed in this context apply transitively to all successors of A (also as function doFollow)
```
A--
```
or
```
A.doFollow()
```

A special role has the operator '--'. it corresponds to the operator '//' in abbreviated syntax of XPath expressions. It says that the following selection applies also for a successors of the current node.

The root node is selector with the operator '*' on the automaton. The following code selects all 'alt/prd' tag content elements in the document and assigns them the type 1:
```
typedef XMLPathSelectAutomaton Automaton;
Automaton atm;
(*atm)--["alt"]["prd"]() = 1;
```
If no content or attribute value is selected then textwolf just triggers an event when the tag or attribute appears:
Get an event '111' for every new document:
```
(*atm) = 111;
```
Get an event '234' for every 'txt' tag:
```
(*atm)["txt"] = 234;
```
Get an event '761' for every attribute 'id' of a 'pers' tag:
```
(*atm)["pers"]("id") = 761;
```

textwolf and XPath

textwolf has not the power of XPath and it does not aim to. It does not buffer than the currently processed token so it can't detect patterns that require buffering. It can't even cope with the fact that tag attributes in XML have no order. For expressions that are not expressible in this model, you have to build the logic around textwolf. textwolf is not XPath, but with some addional effort you get an engine that is able to process at least 'abbreviated syntax of XPath' without parent references and content conditions.

For example

A//ter[@id='5' and @name='kaspar']

has to be translated to

A--["ter"]("id","5")("name","kaspar")
A--["ter"]("name","kaspar")("id","5")

and

A//ter[@id='5' or @name='kaspar']

A--["ter"]("id","5")
A--["ter"]("name","kaspar")

Some cases are even worse. If you select attribute values where you have attribute conditions, then you can solve it only in the filter functions on the iterator after calling textwolf. Selections have to be at the end, because they are not buffered. Therefore something like

A//person[@id='se1']@name

cannot be expressed in textwolf for the case where 'name' appears before 'id' in the XML. The expression

A--["ter"]("id","se1")("name",0)

works only for the case where 'id' appears before 'name'. A possible solution is to define

A--["ter"]("id","se1") = 201;
A--["ter"]("name",0) = 202;

and to set the value with 202 on the element and a flag with 201 that together with 202 enables the element created.