Skip to content

Commit 3df5b24

Browse files
committed
feat(zynml): add stdlib traits, extern calls, and type system foundations
Grammar additions: - Add trait definitions with generics and associated types - Add impl blocks with trait arguments and associated type impls - Add type definitions (@opaque for ZRTL-backed types, type aliases) - Add generic type parameters with bounds (T: Display + Clone) - Add extern call syntax for invoking ZRTL builtins Standard library foundation: - Create stdlib/prelude.zynml with core traits: - Display, Debug (formatting) - Add, Sub, Mul, Div, Mod, MatMul, Neg (arithmetic operators) - Eq, Ord (comparison operators) - BitAnd, BitOr, BitXor, Not (bitwise operators) - Index, IndexMut (indexing) - Clone, Drop (lifecycle) - Iterator, IntoIterator (iteration) - Create stdlib/tensor.zynml with Tensor module: - @opaque("$Tensor") type backed by zrtl_tensor plugin - Display impl using extern tensor_to_string - Operator trait impls (Add, Sub, Mul, Div, MatMul, Neg) - Clone and Drop trait impls - Tensor creation functions (from_array, zeros, ones, etc.) - @method functions for tensor operations ZRTL plugin updates: - Add tensor_to_string, tensor_clone, tensor_free builtins - Add operator builtins (tensor_add, tensor_sub, tensor_mul, etc.) - Enhance IO plugin with type-aware printing infrastructure Compiler updates: - Fix function definition parsing with optional parameters - Split fn_def into explicit variants to avoid index shifting - Add extern call resolution through @Builtin mappings
1 parent a50c90e commit 3df5b24

22 files changed

Lines changed: 2862 additions & 101 deletions

File tree

book/05-semantic-actions.md

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -443,6 +443,231 @@ typed_var_decl = { "const" ~ identifier ~ ":" ~ type_expr ~ "=" ~ expr ~ ";" }
443443
}
444444
```
445445

446+
## The `fold_left_ops` Command
447+
448+
This command handles left-associative binary expressions where operators are interleaved with operands in the child list.
449+
450+
### Difference from `fold_binary`
451+
452+
While `fold_binary` expects named rules for operands and operators, `fold_left_ops` works with children in a flat pattern: `[operand, operator, operand, operator, operand, ...]`
453+
454+
### Usage
455+
456+
```zyn
457+
// Multiplication with operators in child list
458+
multiplicative_expr = { unary_expr ~ (multiplicative_op ~ unary_expr)* }
459+
-> TypedExpression {
460+
"get_all_children": true,
461+
"fold_left_ops": true
462+
}
463+
464+
multiplicative_op = { "*" | "/" | "%" }
465+
-> String {
466+
"get_text": true
467+
}
468+
```
469+
470+
### How It Works
471+
472+
For input `2 * 3 / 4`:
473+
474+
1. Parse produces: `[unary(2), "*", unary(3), "/", unary(4)]`
475+
2. Start with first operand: `result = 2`
476+
3. Process pairs: `result = binary(*, result, 3)``(2 * 3)`
477+
4. Continue: `result = binary(/, result, 4)``((2 * 3) / 4)`
478+
479+
### Important Notes
480+
481+
- Requires `get_all_children: true` before `fold_left_ops`
482+
- Operators should be extracted as strings (use `get_text: true`)
483+
- Automatically unwraps nested single-element lists
484+
485+
## The `apply_unary` Command
486+
487+
This command handles optional unary prefix operators.
488+
489+
### Problem It Solves
490+
491+
When parsing unary expressions like `-x` or `!flag`, the unary operator is optional:
492+
- `-42` has a unary prefix
493+
- `42` has no unary prefix
494+
495+
Without `apply_unary`, you'd need to split into separate rules.
496+
497+
### Usage
498+
499+
```zyn
500+
// Unary operators: -, !
501+
unary_expr = { unary_op? ~ postfix_expr }
502+
-> TypedExpression {
503+
"get_all_children": true,
504+
"apply_unary": true
505+
}
506+
507+
unary_op = { "-" | "!" }
508+
-> String {
509+
"get_text": true
510+
}
511+
```
512+
513+
### How It Works
514+
515+
1. Collects all children: `[operator?, operand]`
516+
2. If operator present: creates `unary(op, operand)`
517+
3. If no operator: passes through the operand unchanged
518+
4. Unwraps nested single-element lists from expression cascading
519+
520+
## The `fold_left` Command
521+
522+
This command provides custom left-folding behavior for special operators like the pipe operator.
523+
524+
### Usage
525+
526+
```zyn
527+
// Pipe operator: x |> f(args) |> g()
528+
// Transforms: a |> f(b) into f(a, b)
529+
pipe_expr = { or_expr ~ ("|>" ~ pipe_call)* }
530+
-> TypedExpression {
531+
"get_all_children": true,
532+
"fold_left": {
533+
"op": "pipe",
534+
"transform": "prepend_arg"
535+
}
536+
}
537+
538+
pipe_call = { identifier ~ "(" ~ call_args? ~ ")" }
539+
-> TypedExpression {
540+
"commands": [
541+
{ "define": "pipe_target", "args": {
542+
"callee": { "define": "variable", "args": { "name": "$1" } },
543+
"args": "$2"
544+
}}
545+
]
546+
}
547+
```
548+
549+
### Parameters
550+
551+
- `op`: The operator name ("pipe", "||", "&&", etc.)
552+
- `transform` (optional): Special transformation to apply
553+
- `"prepend_arg"`: For pipe operator, prepends left-hand side to function args
554+
555+
### How It Works for Pipe Operator
556+
557+
For input `data |> filter(pred) |> map(fn)`:
558+
559+
1. Parse produces: `[data, pipe_call(filter, [pred]), pipe_call(map, [fn])]`
560+
2. Start with `result = data`
561+
3. Transform: `filter(result, pred)` (prepends result to args)
562+
4. Transform: `map(result2, fn)` (prepends to args)
563+
564+
Result is equivalent to: `map(filter(data, pred), fn)`
565+
566+
### Logical Operators
567+
568+
For simple left-folding (like `||` and `&&`):
569+
570+
```zyn
571+
or_expr = { and_expr ~ ("||" ~ and_expr)* }
572+
-> TypedExpression {
573+
"get_all_children": true,
574+
"fold_left": { "op": "||" }
575+
}
576+
```
577+
578+
## The `fold_postfix` Command
579+
580+
This command handles postfix operations like function calls, array indexing, and member access.
581+
582+
### Problem It Solves
583+
584+
Postfix expressions chain operations: `obj.field[0](arg)` needs to fold left-to-right into nested AST nodes.
585+
586+
### Usage
587+
588+
```zyn
589+
// Postfix: function calls, indexing, member access
590+
postfix_expr = { primary_expr ~ postfix_op* }
591+
-> TypedExpression {
592+
"get_all_children": true,
593+
"fold_postfix": true
594+
}
595+
596+
postfix_op = { call_op | index_op | member_op }
597+
-> TypedExpression {
598+
"get_child": { "index": 0 }
599+
}
600+
601+
// Function call: f(args)
602+
call_op = { "(" ~ call_args? ~ ")" }
603+
-> TypedExpression {
604+
"get_child": { "index": 0 },
605+
"define": "call_args",
606+
"args": { "args": "$result" }
607+
}
608+
609+
// Indexing: x[i]
610+
index_op = { "[" ~ expr ~ "]" }
611+
-> TypedExpression {
612+
"define": "index",
613+
"args": { "index": "$1" }
614+
}
615+
616+
// Member access: x.field
617+
member_op = { "." ~ identifier }
618+
-> TypedExpression {
619+
"define": "member",
620+
"args": { "field": "$1" }
621+
}
622+
```
623+
624+
### How It Works
625+
626+
For input `arr[0].length`:
627+
628+
1. Parse produces: `[primary(arr), index_op(0), member_op(length)]`
629+
2. Start with `result = arr`
630+
3. Apply index: `result = index(result, 0)`
631+
4. Apply member: `result = field_access(result, "length")`
632+
633+
### Postfix Operation Types
634+
635+
Each postfix operation is wrapped with a marker so `fold_postfix` knows how to apply it:
636+
637+
| Marker | Creates |
638+
|--------|---------|
639+
| `call_args` | Function call with args |
640+
| `index` | Array/map index access |
641+
| `member` | Field/property access |
642+
643+
## Type Inference Markers
644+
645+
When defining variables in dynamically-typed or type-inferred languages, use these special type markers:
646+
647+
### The `infer_type` Define
648+
649+
```zyn
650+
let_stmt = { "let" ~ identifier ~ "=" ~ expr }
651+
-> TypedStatement {
652+
"commands": [
653+
{ "define": "let_stmt", "args": {
654+
"name": "$1",
655+
"type": { "define": "infer_type" },
656+
"init": "$2",
657+
"is_const": false
658+
}}
659+
]
660+
}
661+
```
662+
663+
The `infer_type` define returns a special null marker indicating the type should be inferred from the initializer expression by the type checker.
664+
665+
### Aliases
666+
667+
These aliases also work for type inference:
668+
- `"define": "auto"` - C++ style auto type
669+
- `"define": "var"` - TypeScript/C# style var
670+
446671
## Handling Optional Children
447672

448673
When a child might be absent, the builder handles null/missing gracefully:

crates/compiler/src/llvm_backend.rs

Lines changed: 80 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,9 @@ pub struct LLVMBackend<'ctx> {
7474

7575
/// Maps HIR global IDs to compiled LLVM global values (persists across functions)
7676
globals_map: IndexMap<HirId, BasicValueEnum<'ctx>>,
77+
78+
/// Symbol signatures for auto-boxing (symbol name → signature)
79+
symbol_signatures: std::collections::HashMap<String, crate::zrtl::ZrtlSymbolSig>,
7780
}
7881

7982
impl<'ctx> LLVMBackend<'ctx> {
@@ -97,9 +100,27 @@ impl<'ctx> LLVMBackend<'ctx> {
97100
phi_map: IndexMap::new(),
98101
current_function: None,
99102
globals_map: IndexMap::new(),
103+
symbol_signatures: std::collections::HashMap::new(),
104+
}
105+
}
106+
107+
/// Register symbol signatures for auto-boxing support
108+
pub fn register_symbol_signatures(&mut self, symbols: &[crate::zrtl::RuntimeSymbolInfo]) {
109+
for sym in symbols {
110+
if let Some(sig) = &sym.sig {
111+
self.symbol_signatures.insert(sym.name.to_string(), sig.clone());
112+
}
100113
}
101114
}
102115

116+
/// Check if a symbol parameter expects DynamicBox
117+
fn param_needs_boxing(&self, symbol_name: &str, param_index: usize) -> bool {
118+
self.symbol_signatures
119+
.get(symbol_name)
120+
.map(|sig| sig.param_is_dynamic(param_index))
121+
.unwrap_or(false)
122+
}
123+
103124
/// Compile an entire HIR module to LLVM IR
104125
///
105126
/// This is the main entry point for compilation. It:
@@ -2230,17 +2251,68 @@ impl<'ctx> LLVMBackend<'ctx> {
22302251
}
22312252
HirCallable::Symbol(symbol_name) => {
22322253
// Call external runtime symbol by name (e.g., "$haxe$trace$int")
2254+
// Check if any parameters need auto-boxing based on symbol signature
2255+
let sig_info = self.symbol_signatures.get(symbol_name).cloned();
2256+
22332257
// Compile arguments first to infer their types
2234-
let arg_values: Vec<BasicMetadataValueEnum> = args
2258+
let raw_arg_values: Vec<BasicValueEnum> = args
22352259
.iter()
2236-
.map(|arg_id| {
2237-
self.get_value(*arg_id)
2238-
.map(|v| v.into())
2239-
})
2260+
.map(|arg_id| self.get_value(*arg_id))
22402261
.collect::<CompilerResult<Vec<_>>>()?;
22412262

2242-
// Infer parameter types from argument values
2243-
let param_types: Vec<BasicMetadataTypeEnum> = arg_values
2263+
// Process arguments - box if needed
2264+
let final_arg_values: Vec<BasicMetadataValueEnum> = if let Some(ref sig) = sig_info {
2265+
raw_arg_values.iter().enumerate().map(|(i, &arg_val)| {
2266+
if sig.param_is_dynamic(i) {
2267+
// This argument needs to be boxed as DynamicBox
2268+
// Determine which boxing function to call based on type
2269+
let func_name = if arg_val.is_int_value() {
2270+
let int_ty = arg_val.into_int_value().get_type();
2271+
if int_ty == self.context.i32_type() {
2272+
"zyntax_box_i32"
2273+
} else if int_ty == self.context.i64_type() {
2274+
"zyntax_box_i64"
2275+
} else if int_ty == self.context.i8_type() {
2276+
"zyntax_box_bool"
2277+
} else {
2278+
"zyntax_box_i64"
2279+
}
2280+
} else if arg_val.is_float_value() {
2281+
let float_ty = arg_val.into_float_value().get_type();
2282+
if float_ty == self.context.f32_type() {
2283+
"zyntax_box_f32"
2284+
} else {
2285+
"zyntax_box_f64"
2286+
}
2287+
} else {
2288+
// Pointers and other types
2289+
"zyntax_box_ptr"
2290+
};
2291+
2292+
// Declare and call boxing function
2293+
let box_fn_type = self.context.i64_type().fn_type(
2294+
&[arg_val.get_type().into()],
2295+
false
2296+
);
2297+
let box_fn = self.module.get_function(func_name).unwrap_or_else(|| {
2298+
self.module.add_function(func_name, box_fn_type, None)
2299+
});
2300+
2301+
if let Ok(call_site) = self.builder.build_call(box_fn, &[arg_val.into()], "box") {
2302+
call_site.try_as_basic_value().left().unwrap_or(arg_val).into()
2303+
} else {
2304+
arg_val.into()
2305+
}
2306+
} else {
2307+
arg_val.into()
2308+
}
2309+
}).collect()
2310+
} else {
2311+
raw_arg_values.iter().map(|&v| v.into()).collect()
2312+
};
2313+
2314+
// Infer parameter types from (potentially boxed) argument values
2315+
let param_types: Vec<BasicMetadataTypeEnum> = final_arg_values
22442316
.iter()
22452317
.map(|v| match v {
22462318
BasicMetadataValueEnum::IntValue(i) => i.get_type().into(),
@@ -2260,7 +2332,7 @@ impl<'ctx> LLVMBackend<'ctx> {
22602332
});
22612333

22622334
// Build call
2263-
self.builder.build_call(func, &arg_values, symbol_name)?;
2335+
self.builder.build_call(func, &final_arg_values, symbol_name)?;
22642336

22652337
// Return a dummy value (void functions don't return anything meaningful)
22662338
Ok(self.context.i32_type().const_zero().into())

crates/compiler/src/llvm_jit_backend.rs

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,9 @@ pub struct LLVMJitBackend<'ctx> {
5050
/// Runtime symbols to register with the execution engine
5151
/// Maps function name to function pointer address
5252
runtime_symbols: IndexMap<String, usize>,
53+
54+
/// Symbol signatures for auto-boxing (symbol name → signature)
55+
symbol_signatures: Vec<crate::zrtl::RuntimeSymbolInfo>,
5356
}
5457

5558
impl<'ctx> LLVMJitBackend<'ctx> {
@@ -73,9 +76,15 @@ impl<'ctx> LLVMJitBackend<'ctx> {
7376
function_pointers: IndexMap::new(),
7477
opt_level,
7578
runtime_symbols: IndexMap::new(),
79+
symbol_signatures: Vec::new(),
7680
})
7781
}
7882

83+
/// Register symbol signatures for auto-boxing support
84+
pub fn register_symbol_signatures(&mut self, symbols: &[crate::zrtl::RuntimeSymbolInfo]) {
85+
self.symbol_signatures.extend(symbols.iter().cloned());
86+
}
87+
7988
/// Register a runtime symbol that will be available to JIT-compiled code
8089
///
8190
/// Call this before `compile_module` to make external functions available.
@@ -100,6 +109,10 @@ impl<'ctx> LLVMJitBackend<'ctx> {
100109
pub fn compile_module(&mut self, hir_module: &HirModule) -> CompilerResult<()> {
101110
// Step 1: Create backend and compile HIR → LLVM IR
102111
let mut backend = LLVMBackend::new(self.context, "zyntax_jit");
112+
113+
// Register symbol signatures for auto-boxing
114+
backend.register_symbol_signatures(&self.symbol_signatures);
115+
103116
let _llvm_ir = backend.compile_module(hir_module)?;
104117

105118
// Step 2: Collect external function declarations from the module BEFORE consuming it

0 commit comments

Comments
 (0)