Grammar (ANTLR4)

Grammar (ANTLR4)

Concrete syntax for Pine Script in PYNE is defined by a split lexer/parser ANTLR4 grammar plus hand-written base classes that inject Python-like indentation tokens.

Abstract

Two grammars live under src/pynescript/ast/grammar/antlr4/resource/:

• PinescriptLexer.g4 — keywords, operators, literals (including v6 triple-quoted strings), comments, hidden whitespace.
• PinescriptParser.g4 — statements, expressions, type/enum/function/method declarations, control structures.

Python runtime classes under generated/ are outputs of antlr4 -Dlanguage=Python3. They are committed so end users never need a JDK. Never edit files under generated/ — changes evaporate on the next regeneration and fight the build pipeline.

Hand-written bases (copied into generated/ by the generate tool):

• resource/PinescriptLexerBase.py — INDENT/DEDENT, newline elision, multiline string handling.
• resource/PinescriptParserBase.py — parser superClass hooks.

Conceptual model

Rendering…

Interface surface

Application code should not import generated symbols directly when a wrapper exists:

from pynescript.ast.grammar.antlr4.lexer import PinescriptLexer
from pynescript.ast.grammar.antlr4.parser import PinescriptParser
from pynescript.ast.grammar.antlr4.visitor import PinescriptParserVisitor
from pynescript.ast.grammar.antlr4.error_listener import PinescriptErrorListener

The public parse path is higher level — Helper API — which constructs lexer, token stream, parser, and strips default console error listeners in favor of PinescriptErrorListener.

Entry rules (PinescriptParser.g4)

Lexer structural facts

Keywords include var, varip, method, export, enum, type, qualifiers const/input/simple/series, and control forms if/for/while/switch/once (August 2026). Many of those words are soft in identifier position (name alternatives) so as = input(...) and once = 3 parse.

Parser structural facts

• Compound vs simple statements. Functions, methods, types, enums, and structure-valued assignments are compound; imports, break/continue, expression statements, and simple assignments terminate with NEWLINE.
• Trailing structures. a = 0, for i = 0 to n is trailing_structure_statements (comma-separated simples ending in a structure).
• Structures are dual-use. if/for/while/switch appear both as statements and as expressions (structure_expression) — the classic Pine “if returns a value” form. once is statement-only (once_structure is not in value_structure); x = once … is a parse error.
• Local blocks. Either indented (NEWLINE INDENT statements DEDENT) or inline single statement.
• Library export. EXPORT? prefixes function/method/type/enum and June-2025 export const T name = … compound initializers.
• UDF return types. function_declaration / method_declaration accept an optional leading type_specification (int ilog2(int n) => …). Builder stores it on FunctionDef.returns.
• Bitwise layer. Between and and comparisons: | ^ & then <</>>, then additive. Unary ~ is Invert.
• Assign vs reassignment (0.3.9). = on a bare name is initialization (Assign via variable_declaration). Reassignment of a name is := (ReAssign). = is reassignment only for attribute / subscript targets (obj.f =, a[i] =).
• Left-factored typed names. variable_declaration is declaration_mode? type_specification? name_store with the typed alternative first so int x = 1 does not consume int as the identifier. Same left-factor on for_iterator and parameter lists.

Internals

Paths

LexerBase responsibilities

PinescriptLexerBase is not optional sugar; it is part of the language definition:

• Ignore leading / collapse consecutive newlines; ensure trailing newline.
• Suppress newlines inside open () / [], after operators, and for soft wrap lines whose indent width is not a multiple of four.
• Push INDENT/DEDENT with tab length and indent length = 4.
• Special-case multiline string tokens so wrap indentation inside strings is not mis-tokenized as structure.

Indent mistakes surface as pynescript.ast.error.IndentationError (subclass of the project SyntaxError).

Regeneration

# Project-aware entry (sets -o generated, -lib resource, copies *Base.py)
python -m pynescript.ast.grammar.antlr4.tool.generate

# hatch lint:gen-parser is only the antlr4 CLI (`antlr4 {args}`) — pass flags yourself.
# hatch run lint:gen-parser -- -o …/generated -lib …/resource -listener -visitor -Dlanguage=Python3 …/resource/*.g4

tool/generate.py invokes $(sys.executable)/../antlr4 with -o generated -lib resource -listener -visitor -Dlanguage=Python3 on resource/*.g4, then copies resource/*.py into generated/. Requires a working antlr4 CLI (hatch lint env pulls antlr4-cli). Generated modules are excluded from mypy/ruff strictness in pyproject.toml — do not “lint-fix” them by hand.

Downstream after grammar change

1. Regenerate (or selectively refresh lexer — see case study).
2. Add/adjust visit_* methods in src/pynescript/ast/builder.py for new or renamed rules.
3. If the shape of the language changed (new statement/expression kind), update Pinescript.asdl and regenerate ASDL nodes — ASDL schema.
4. Smoke-test with parse / unparse on a minimal first-party snippet before broader regression.

Invariants

1. Never hand-edit generated/. Patches belong in resource/*.g4 or resource/*Base.py.
2. Visitor contract. Generated PinescriptParserVisitor method names follow ANTLR rule names; the builder subclass must track renames.
3. Comments are off the default channel. Annotation logic in helper._collect_comment_nodes iterates token_stream.tokens for COMMENT type after fill(), and keeps only texts containing @. HASH_COMMENT / BACKTICKS are remapped to that same token type.
4. Indent unit is four spaces. Soft-wrap lines that are not multiples of four are newline-elided, not nested blocks.

Worked examples

Minimal grammar-level mental model

//@version=6
indicator("x")
if close > open
    plot(1)
else
    plot(0)

Lexer emits NEWLINE, INDENT, DEDENT around the if bodies. Parser builds if_structure → builder yields ast.If with body / orelse statement lists.

v6 triple-quoted strings

Resource lexer rules (illustrative — see live PinescriptLexer.g4):

TRIPLE_DQ_STRING : '"""' ( ~'"' | '"' ~'"' | '""' ~'"' )* '"""' -> type(STRING) ;
TRIPLE_SQ_STRING : '\'\'\'' ( ~'\'' | '\'' ~'\'' | '\'\'' ~'\'' )* '\'\'\'' -> type(STRING) ;

ANTLR quoting pitfall: putting "'''" or '"""' as fragment literals can produce tool errors (quote came as a complete surprise). Factor starters when needed:

fragment TRIPLE_SQ_START: '\'' '\'' '\'';

Content and source indentation inside the triple quotes are preserved literally (no automatic dedent).

Typed names and = vs := (0.3.9)

Pine lets type names occupy identifier positions (int is both a type and a legal NAME alternative via soft keywords). Alternatives must try the typed form first:

variable_declaration
    : declaration_mode? type_specification name_store
    | declaration_mode? name_store
    ;

simple_name_initialization: EXPORT? variable_declaration EQUAL expression;

simple_reassignment
    : assignment_target_attribute EQUAL expression
    | assignment_target_subscript EQUAL expression
    | primary_expression COLONEQUAL expression
    ;

Putting name_store before type_specification name_store made int x = 1 parse int as the variable and then fail on x. Do not invert those alternatives.

Case study: targeted lexer refresh (2026-07)

Full regeneration once produced a PinescriptParser.py whose context accessors diverged from what the hand-written builder expected (e.g. missing template_spec_suffix()), breaking even trivial parses. The practical recovery:

1. Edit only resource/ grammar.
2. Generate the lexer in a clean temp directory (avoids nested src/ mirror paths).
3. Copy only PinescriptLexer.py (+ refreshed LexerBase.py) into generated/.
4. Leave committed PinescriptParser.py and visitors untouched unless prepared to patch the builder in the same change.
5. Verify immediately:

from pynescript.ast.helper import parse, unparse
ast = parse('indicator("x")\ns = """\nfoo\n  bar\n"""\n')
assert "foo" in unparse(ast)

Failure modes

See also

• ASDL schema
• Builder
• Error model
• Helper API
• Missing features