Class: Ariel::Token

Tokens populate a TokenStream. They know their position in the original document, can list the wildcards that match them and determine whether a given string or wildcard is a valid match. During the process of parsing a labeled document, some tokens may be marked as being a label_tag. These are filtered from the TokenStream before the rule learning phase.

Methods

<=> == is_label_tag? matches? matching_wildcards new

Attributes

 end_loc
 start_loc
 text

Public Class methods

new(text, start_loc, end_loc, label_tag=false)

Each new Token must have a string representing its content, its start position in the original document (start_loc) and the point at which it ends (end_loc). For instance, in str="This is an example", if "is" were to be made a Token it would be given a start_loc of 5 and and end_loc of 7, which is str

Public Instance methods

<=>(t)

Tokens are sorted based on their start_loc

==(t)

Tokens are only equal if they have an equal start_loc, end_loc and text.

is_label_tag?()

Returns true or false depending on whether the token was marked as a label tag when it was initialized.

matches?(landmark)

Accepts either a string a symbol representing a wildcard in Wildcards#list or an an arbitrary regex. Returns true if the whole Token is consumed by the wildcard or the string is equal to Token#text, and false if the match fails. Raises an error if the passed symbol is not a member of Wildcards#list.

matching_wildcards()

Returns an array of symbols corresponding to the Wildcards that match the Token.