Numba path

Numba path

Abstract

Numeric compile mode emits a single @numba.njit function whose body is the script’s bar loop. Pine builtins that cannot cross the JIT boundary are replaced by hand-written kernels in numba_builtins.py that take full arrays plus the current index (or a small float state vector for *_inc forms). The result is large speedups on multi-thousand-bar series after a one-time JIT warm-up—at the cost of a restricted language subset and np.nan as the missing-value sentinel.

Kernel contracts below match interpret ta.* / calendar helpers where parity was fixed; intentional deviations should be documented on the kernel docstring and here.

Conceptual model

Rendering…

Interface surface

Kernels are defined with @numba.njit(cache=True) in numba_builtins.py. Generated entry points use @numba.njit(cache=False) in the in-memory IR string; when the engine writes a disk IR module it rewrites that decorator to cache=True so Numba can store machine code next to a real file path (see Cache).

Generated code imports from pynescript.compiler.numba_builtins import * so call sites are bare names. Prefer *_inc when the visitor allocates fixed state (__ema0_st, …).

Kernel table (interpret-parity highlights)

Full inventory lives in src/pynescript/compiler/numba_builtins.py (njit batch + *_inc + pure-Python object-mode helpers).

Calendar / time_arr

Bar open time is the host time_arr series (synthetic when the runner omits timestamps).

History access

close[n] lowers to array indexing at __bar_idx - n with bounds → nan (parallel to interpreter None, different sentinel).

Control flow

if / for / while / once emit Python control flow inside the njit function—supported Numba subset only (no heap objects, no Python lists of mixed types). once lowers to a persistent __once_fired_N bool set when the body runs (compile is historical: barstate.isconfirmed is always true). A once inside a UDF forces object mode so the flag is module-level.

Cache

Three layers interact:

Numba function cache (.nbi / .nbc):

• Hand-written kernels: @njit(cache=True) → artifacts under …/pynescript/compiler/__pycache__/.
• Generated entry: IR is emitted with cache=False (safe for exec from <string>). Disk write rewrites @numba.njit(cache=False) → cache=True so Numba’s file locator can cache machine code under the disk IR __pycache__/.
• Truncated/corrupt pickle loads raise EOFError / UnpicklingError. The engine purges known .nbi/.nbc via clear_numba_function_caches() and recompiles once instead of failing the script.

from pynescript.compiler import clear_numba_function_caches, clear_disk_compile_cache
clear_numba_function_caches()
clear_disk_compile_cache()

Internals

Expanding coverage is primarily new kernels + visitor call mapping, not changes to the bar-loop skeleton.

Invariants & edge cases

1. No Python objects in numeric mode. Strings, UDTs, maps, drawings force object mode before njit is applied.
2. Warm-up cost. First compile_script invokes a dummy run; production servers should cache CompiledScript and prewarm (PYNE_COMPILE_PREWARM default on, POST /compile/prewarm).
3. EMA/RMA seed uses sliding all-finite SMA windows—not “seed once at bar period-1 only if that fixed slice is clean.” Leading NaN on tr / nested EMA sources must eventually produce values.
4. RSI is Wilder, first valid at i == period. Do not compare against a simple N-bar average-gain RSI.
5. highestbars/lowestbars return negative offsets and -1 on incomplete windows; ties keep the oldest bar.
6. Disk entry cache requires the rewrite to a real file path; pure exec of cache=False IR does not populate Numba’s on-disk function cache for the entry function.
7. Anaconda static libpython. Packaging notes in AGENTS/build docs apply when shipping JIT-heavy binaries.

Worked examples

Benchmark mental model

interpret:  O(bars × ast_nodes × python_dispatch)
numeric:    O(bars × lowered_ops) in machine code after JIT

For simple SMA scripts, internal notes cite order-of-magnitude speedups versus list-based interpretation on long series (see repo docs/COMPILER_PLAN.md qualitative claims).

Using from Pro API

Runtime(...).run(source, ohlcv, mode="compile")

Converts bar dicts → arrays (including time= bar-open ms), compiles, reshapes plots into the standard envelope, flags object_mode when applicable. generated_code is omitted unless PYNESCRIPT_RETURN_GENERATED_CODE=1.

Failure modes

Performance notes

• Prefer one compile, many runs with different OHLCV.
• Keep scripts in the numeric subset for realtime ticks.
• Host prewarm of numba_builtins + common scripts shifts cold JIT off the first user request.
• Object mode still wins over AST walking for UDT-heavy scripts but will not match peak njit throughput.

See also

• Compiler overview
• Technical builtins (interpret)
• Series & history