2.2 Other tokens
Besides NEWLINE, INDENT and DEDENT, the following categories of tokens exist: identifiers,
keywords, literals, operators, and delimiters. Whitespace
characters (other than line terminators, discussed earlier) are not tokens, but serve to
delimit tokens. Where ambiguity exists, a token comprises the longest possible string that
forms a legal token, when read from left to right.
|