byte markTokensImpl(final byte token, @NotNull final Segment line)
An abstract method that splits a line up into tokens.
This package contains the other part of the script editor.
void addToken(final int length, final byte id)
Adds a token to the token list.
A TokenMarker for the message field of Crossfire objects allowing.
static final byte OPERATOR
Operator token id.
static final byte INVALID
Invalid token id.
static final byte KEYWORD1
Keyword 1 token id.
static final byte LITERAL1
Literal 1 token id.
A token marker that splits lines of text into tokens.
int length
The number of lines in the model being tokenized.