Semantic Tokens

textDocument/semanticTokens/full — AST visitor, custom legend, and LSP delta encoding.

This page

Semantic Tokens

Abstract

Semantic tokens let the server paint identifiers with roles richer than TextMate scopes (library namespaces vs user variables, function defs vs properties). PYNE advertises a full semantic-tokens provider with a short custom legend (not the full LSP standard set) and implements textDocument/semanticTokens/full by walking the workspace AST.

Conceptual model

Diagram

Rendering…

Interface surface

Capability (config.py → semantic_tokens_provider):

legend.token_types (index order is contract):
  namespace, type, class, function, method, variable,
  parameter, property, keyword, string, number, operator, comment

legend.token_modifiers (bit 0 = declaration):
  declaration, definition, readonly, defaultLibrary

range: false
full: true

Handler: features/semantic_tokens.py → handle_semantic_tokens(params, source, tree=…).

Response type: lsp.SemanticTokens(data=[...]) — LSP-encoded five-tuples as a flat int array (relative line/character deltas).

What is emitted

ASTToken typeModifiers
FunctionDef.namefunctiondefinition | declaration
TypeDef.nameclassdefinition | declaration
EnumDef.nametypedefinition | declaration
Assign to Namevariabledeclaration
Attribute whose value is a builtin namespace (ta, math, strategy, …)namespace + methoddefaultLibrary (+ readonly on ns)
Other Attribute.attrproperty—

Builtin namespace set (_BUILTIN_NS): ta, math, str, array, matrix, map, strategy, request, input, color, line, label, box, table, polyline, log, ticker, timeframe, chart, runtime, syminfo, barstate, session, time.

Internals

PathRole
features/semantic_tokens.pyCollector + _encode
config.pyLegend + capability — indices must match _TT
server.pyTEXT_DOCUMENT_SEMANTIC_TOKENS_FULL; passes cached doc.ast

Attribute columns prefer end_col_offset - len(attr) when present; otherwise parent col_offset (approximate).

TextMate from vscode-extension/syntaxes/pinescript.tmLanguage.json still colors keywords, strings, and comments — the visitor does not emit those yet even though the legend reserves the slots.

Invariants and edge cases

  1. Never throws — missing source, tree=None, or parse exception yields empty data.
  2. No range provider — clients must not request semanticTokens/range.
  3. Legend is fixed at initialize; changing indices later would break cached clients.
  4. Empty data is valid protocol (empty file or parse failure), not an error.
  5. Tokens are sorted by (line, col) before delta encoding.

Worked example

//@version=6
indicator("tok")
length = 14
plot(ta.sma(close, length))

Expect tokens for length (variable/declaration), ta (namespace/defaultLibrary), sma (method/defaultLibrary). Keywords such as indicator remain TextMate-only.

Failure modes

SymptomCause
Semantic highlighting off despite tokensClient editor.semanticHighlighting.enabled false
Colors look “shifted” after a legend editClient cached old type indices — restart the language client
Client errors on range requestrange=False in capabilities; client should not call it
Attribute token sits on the wrong columnMissing end_col_offset on the AST node — approximate path

See also