Contents User Forum Source Index APIs by Task APIs by Level Data

G3D::TextInput Class Reference

A simple style tokenizer for reading text files that also supports C++-like syntaxes. More...

#include <TextInput.h>

List of all members.

Public Types

enum FS { FROM_STRING }

Public Member Functions

const std::string & filename () const

bool hasMore ()

Token peek ()

int peekCharacterNumber ()

int peekLineNumber ()

void push (const Token &t)

Token read ()

bool readBoolean ()

double readNumber ()

void readString (const std::string &s)

std::string readString ()

Token readStringToken ()

void readSymbol (const std::string &symbol)

std::string readSymbol ()

void readSymbols (const std::string &s1, const std::string &s2, const std::string &s3, const std::string &s4)

void readSymbols (const std::string &s1, const std::string &s2, const std::string &s3)

void readSymbols (const std::string &s1, const std::string &s2)

Token readSymbolToken ()

TextInput (FS fs, const std::string &str, const Settings &settings=Settings())

TextInput (const std::string &filename, const Settings &settings=Settings())

Classes

class BadMSVCSpecial

While parsing a number of the form 1. More...

class Settings

Tokenizer configuration options. More...

class TokenException

class WrongString

String read from input did not match expected string. More...

class WrongSymbol

class WrongTokenType

Thrown by the read methods. More...

Detailed Description

A simple style tokenizer for reading text files that also supports C++-like syntaxes.

TextInput handles a superset of C++/Java-like text including single line comments, block comments, quoted strings with escape sequences, and operators. TextInput recognizes four categories of tokens, which are separated by white space, quotation marks, or the end of a recognized operator:

Token::SINGLE_QUOTED_TYPE string of characters surrounded by single quotes, e.g., 'x', '', 'foo'.
Token::DOUBLE_QUOTED_TYPE string of characters surrounded by double quotes, e.g., "x", "abc\txyz", "b o b".
Token::SYMBOL_TYPE legal C++ operators, keywords, and identifiers. e.g., >=, Foo, _X, class, {
Token::INTEGER_TYPE numbers without decimal places or exponential notation. e.g., 10, 0x17F, 32, 0, -155
Token::FLOATING_POINT_TYPE numbers with decimal places or exponential notation. e.g., 1e3, -1.2, .4, 0.5
Token::BOOLEAN_TYPE special symbols like "true" and "false"; the exact details can be configured in TextInput::Settings

The special ".." and "..." tokens are recognized in addition to normal C++ operators.

Negative numbers are handled specially because of the ambiguity between unary minus and negative numbers-- see the note on TextInput::read.

TextInput does not have helper functions for types with non-obvious formatting, or helpers that would be redundant. Use the serialize methods instead for parsing specific types like int, Vector3, and Color3.

Inside quoted strings escape sequences are converted. Thus the string token for ["a\nb"] is 'a', followed by a newline, followed by 'b'. Outside of quoted strings, escape sequences are not converted, so the token sequence for [a] is symbol 'a', symbol '\', symbol 'nb' (this matches what a C++ parser would do). The exception is that a specified TextInput::Settings::otherCommentCharacter preceeded by a backslash is assumed to be an escaped comment character and is returned as a symbol token instead of being parsed as a comment (this is what a LaTex or VRML parser would do).

Examples

  TextInput ti(TextInput::FROM_STRING, "name = \"Max\", height = 6");

  Token t;

  t = ti.read(); 
  debugAssert(t.type == Token::SYMBOL);
  debugAssert(t.sval == "name");

  ti.read();
  debugAssert(t.type == Token::SYMBOL);
  debugAssert(t.sval == "=");

  std::string name = ti.read().sval;
  ti.read();

  TextInput ti(TextInput::FROM_STRING, "name = \"Max\", height = 6");
  ti.readSymbols("name", "=");
  std::string name = ti.readString();
  ti.readSymbols(",", "height", "=");
  double height = ti. readNumber();

Member Enumeration Documentation

enum G3D::TextInput::FS

Enumerator:

FROM_STRING

Constructor & Destructor Documentation

G3D::TextInput::TextInput	(	const std::string &	filename,
		const Settings &	settings = `Settings()`
	)

G3D::TextInput::TextInput	(	FS	fs,
		const std::string &	str,
		const Settings &	settings = `Settings()`
	)

Creates input directly from a string.

The first argument must be TextInput::FROM_STRING.

Member Function Documentation

const std::string& G3D::TextInput::filename ( ) const

Returns the filename from which this input is drawn, or the first few characters of the string if created from a string.

If options::filename is non-empty that will replace the true filename.

bool G3D::TextInput::hasMore ( )

Returns true while there are tokens remaining.

Token G3D::TextInput::peek ( )

Return a copy of the next token in the input stream, but don't remove it from the input stream.

int G3D::TextInput::peekCharacterNumber ( )

Returns the character number (relative to the line) for the next token in the input stream.

See also peek.

void G3D::TextInput::push ( const Token & t )

Take a previously read token and push it back at the front of the input stream.

Can be used in the case where more than one token of read-ahead is needed (i.e., when peek doesn't suffice).

Token G3D::TextInput::read ( )

Read the next token (which will be the END token if ! hasMore()).

Signed numbers can be handled in one of two modes. If the option TextInput::Settings::signedNumbers is true, A '+' or '-' immediately before a number is prepended onto that number and if there is intervening whitespace, it is read as a separate symbol.

If TextInput::Settings::signedNumbers is false, read() does not distinguish between a plus or minus symbol next to a number and a positive/negative number itself. For example, "x - 1" and "x -1" will be parsed the same way by read().

In both cases, readNumber() will contract a leading "-" or "+" onto a number.

bool G3D::TextInput::readBoolean ( )

double G3D::TextInput::readNumber ( )

Read one token (or possibly two) as a number or throws WrongTokenType, and returns the number.

If the first token in the input is a number, it is returned directly.

If TextInput::Settings::signedNumbers is false and the input stream contains a '+' or '-' symbol token immediately followed by a number token, both tokens will be consumed and a single token will be returned by this method.

WrongTokenType will be thrown if one of the input conditions described above is not satisfied. When an exception is thrown, no tokens are consumed.

void G3D::TextInput::readString ( const std::string & s )

Reads a specific string token or throws either WrongTokenType or WrongString.

If the next token in the input is a string matching s, it will be consumed.

Use this method if you want to match a specific string from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a string. WrongString will be thrown if the next token in the input stream is a string but does not match the s parameter. When an exception is thrown, no tokens are consumed.

std::string G3D::TextInput::readString ( )

Like readStringToken, but returns the token's string.

Use this method (rather than readStringToken) if you want the token's value but don't really care about its location in the input. Use of readStringToken is encouraged for better error reporting.

Token G3D::TextInput::readStringToken ( )

Reads a string token or throws WrongTokenType, and returns the token.

Use this method (rather than readString) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a string. When an exception is thrown, no tokens are consumed.

void G3D::TextInput::readSymbol ( const std::string & symbol )

Reads a specific symbol token or throws either WrongTokenType or WrongSymbol.

If the next token in the input is a symbol matching symbol, it will be consumed.

Use this method if you want to match a specific symbol from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a symbol. WrongSymbol will be thrown if the next token in the input stream is a symbol but does not match the symbol parameter. When an exception is thrown, no tokens are consumed.

std::string G3D::TextInput::readSymbol ( )

Like readSymbolToken, but returns the token's string.

Use this method (rather than readSymbolToken) if you want the token's value but don't really care about its location in the input. Use of readSymbolToken is encouraged for better error reporting.

void G3D::TextInput::readSymbols	(	const std::string &	s1,
		const std::string &	s2,
		const std::string &	s3,
		const std::string &	s4
	)			`[inline]`

Read a series of four specific symbols.

See readSymbol.

void G3D::TextInput::readSymbols	(	const std::string &	s1,
		const std::string &	s2,
		const std::string &	s3
	)			`[inline]`

Read a series of three specific symbols.

See readSymbol.

void G3D::TextInput::readSymbols	(	const std::string &	s1,
		const std::string &	s2
	)			`[inline]`

Read a series of two specific symbols.

See readSymbol.

Token G3D::TextInput::readSymbolToken ( )

Reads a symbol token or throws WrongTokenType, and returns the token.

Use this method (rather than readSymbol) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a symbol. When an exception is thrown, no tokens are consumed.

The documentation for this class was generated from the following file:

TextInput.h

Generated on Thu Aug 2 11:40:48 2007 for G3D by

1.5.2

Hosted by