presage 0.9.1
ForwardTokenizer Class Reference

#include <forwardTokenizer.h>

Inheritance diagram for ForwardTokenizer:
Inheritance graph
Collaboration diagram for ForwardTokenizer:
Collaboration graph

Public Member Functions

 ForwardTokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual ~ForwardTokenizer ()
virtual int countTokens ()
virtual bool hasMoreTokens () const
virtual std::string nextToken ()
virtual double progress () const
Public Member Functions inherited from Tokenizer
 Tokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual ~Tokenizer ()
void blankspaceChars (const std::string)
std::string blankspaceChars () const
void separatorChars (const std::string)
std::string separatorChars () const
void lowercaseMode (const bool)
bool lowercaseMode () const
std::string streamToString () const

Additional Inherited Members

Protected Member Functions inherited from Tokenizer
bool isBlankspace (const int character) const
bool isSeparator (const int character) const
Protected Attributes inherited from Tokenizer
std::istream & stream
std::ios::iostate sstate
std::streamoff offbeg
std::streamoff offend
std::streamoff offset

Detailed Description

Tokenizes a stream from the beginning to the end.


|_|_|_|_|_|_|_| | | | | | offend | offset offbeg

Definition at line 38 of file forwardTokenizer.h.

Constructor & Destructor Documentation

◆ ForwardTokenizer()

ForwardTokenizer::ForwardTokenizer ( std::istream & stream,
const std::string blankspaces,
const std::string separators )

Definition at line 27 of file forwardTokenizer.cpp.

References Tokenizer::blankspaces, Tokenizer::offbeg, Tokenizer::offset, Tokenizer::separators, Tokenizer::stream, and Tokenizer::Tokenizer().

Referenced by StringForwardTokenizer::StringForwardTokenizer().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ ~ForwardTokenizer()

ForwardTokenizer::~ForwardTokenizer ( )
virtual

Definition at line 36 of file forwardTokenizer.cpp.

Member Function Documentation

◆ countTokens()

int ForwardTokenizer::countTokens ( )
virtual

Returns the number of tokens left.

Implements Tokenizer.

Definition at line 39 of file forwardTokenizer.cpp.

References hasMoreTokens(), nextToken(), Tokenizer::offbeg, Tokenizer::offset, and Tokenizer::stream.

Here is the call graph for this function:

◆ hasMoreTokens()

bool ForwardTokenizer::hasMoreTokens ( ) const
virtual

Tests if there are more tokens.

Implements Tokenizer.

Definition at line 61 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by countTokens(), ContextTracker::learn(), and main().

Here is the caller graph for this function:

◆ nextToken()

std::string ForwardTokenizer::nextToken ( )
virtual

Returns the next token.

Implements Tokenizer.

Definition at line 72 of file forwardTokenizer.cpp.

References Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Referenced by countTokens(), ContextTracker::learn(), and main().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ progress()

double ForwardTokenizer::progress ( ) const
virtual

Returns progress percentage.

Implements Tokenizer.

Definition at line 151 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by main().

Here is the caller graph for this function:

The documentation for this class was generated from the following files: