presage 0.9.1
ReverseTokenizer Class Reference

#include <reverseTokenizer.h>

Inheritance diagram for ReverseTokenizer:
Inheritance graph
Collaboration diagram for ReverseTokenizer:
Collaboration graph

Public Member Functions

 ReverseTokenizer (std::istream &stream, const std::string blanks, const std::string separs)
virtual ~ReverseTokenizer ()
virtual int countTokens ()
virtual bool hasMoreTokens () const
virtual std::string nextToken ()
virtual double progress () const
Public Member Functions inherited from Tokenizer
 Tokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual ~Tokenizer ()
void blankspaceChars (const std::string)
std::string blankspaceChars () const
void separatorChars (const std::string)
std::string separatorChars () const
void lowercaseMode (const bool)
bool lowercaseMode () const
std::string streamToString () const

Additional Inherited Members

Protected Member Functions inherited from Tokenizer
bool isBlankspace (const int character) const
bool isSeparator (const int character) const
Protected Attributes inherited from Tokenizer
std::istream & stream
std::ios::iostate sstate
std::streamoff offbeg
std::streamoff offend
std::streamoff offset

Detailed Description

Tokenizes a stream from the end to the beginning.


|_|_|_|_|_|_|_| | | | | | offend | offset offbeg

Definition at line 38 of file reverseTokenizer.h.

Constructor & Destructor Documentation

◆ ReverseTokenizer()

ReverseTokenizer::ReverseTokenizer ( std::istream & stream,
const std::string blanks,
const std::string separs )

Definition at line 27 of file reverseTokenizer.cpp.

References Tokenizer::offend, Tokenizer::offset, Tokenizer::stream, and Tokenizer::Tokenizer().

Here is the call graph for this function:

◆ ~ReverseTokenizer()

ReverseTokenizer::~ReverseTokenizer ( )
virtual

Definition at line 44 of file reverseTokenizer.cpp.

Member Function Documentation

◆ countTokens()

int ReverseTokenizer::countTokens ( )
virtual

Returns the number of tokens left.

Implements Tokenizer.

Definition at line 47 of file reverseTokenizer.cpp.

References hasMoreTokens(), nextToken(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Here is the call graph for this function:

◆ hasMoreTokens()

bool ReverseTokenizer::hasMoreTokens ( ) const
virtual

Tests if there are more tokens.

Implements Tokenizer.

Definition at line 69 of file reverseTokenizer.cpp.

References Tokenizer::offbeg, and Tokenizer::offset.

Referenced by countTokens(), and ContextTracker::getToken().

Here is the caller graph for this function:

◆ nextToken()

std::string ReverseTokenizer::nextToken ( )
virtual

Returns the next token.

Implements Tokenizer.

Definition at line 80 of file reverseTokenizer.cpp.

References Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offbeg, Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Referenced by ContextChangeDetector::change(), countTokens(), and ContextTracker::getToken().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ progress()

double ReverseTokenizer::progress ( ) const
virtual

Returns progress percentage.

Implements Tokenizer.

Definition at line 174 of file reverseTokenizer.cpp.

References Tokenizer::offbeg, Tokenizer::offend, and Tokenizer::offset.


The documentation for this class was generated from the following files: