Yash Nagda

Blog post

Week 6: GSoC Project Progress

03 Jul 2026

Hours this week: 18

Total hours: 104

At the end of the fifth week, I had the following tasks to work upon:

  • Start working on the benchmarking speed of execution of the mlir backend, by comparing the performance of tinyc compiler with the new backend with Clang.
  • Implement the optimizations like elimination of string comparisions, malloc etc for improving the speed of execution of the mlir backend.
  • Once these steps are completed, I shall be working on the integration of the new mlir backend with the lfortran compiler.

Continuing from here, this week, I focused on making the Linux x86_64 ELF path usable for tinyC self-hosting from webassembly modules (wasm). After getting that path working, I also tried simplifying the wasm pipeline, since the current path can be optimized for better performance.

In this blog, I would like to discuss the work done during this week in two parts:

  • First, the backend work to make the --from-wasm --emit=elf flow work, which has the same design as the existing aarch64 flow.
  • Second, the simplifications I tried in the pipeline, and the issues faced while trying them.

Task

The main task for this week was to support the following self-host path in the certik/mlir repository:

flowchart TB
  A["linked wasm"] --> B["wasmstack"]
  B --> C["wasmssa"]
  C --> D["LLVM dialect"]
  D --> E["x86_64"]
  E --> F["ELF"]

The end goal was to make the ELF backend work for the self-host flow of tinyC as well. Along with that, I also added a benchmarking setup so that the tinyC self-host flow on Linux can be compared against clang on the same workload.

PR Made

This PR is created here: PR: 218

The high level idea here is similar to the aarch64 flow. In the aarch64 path, we already have a staged flow where we lower the input through MLIR, add in the required helper functions codes and then run the target-specific backend. I followed the same overall structure for x86_64 ELF, but adapted it for Linux runtime support:

  • compile helper modules such as platform_linux.c, buddy.c, and the WASI adapter as separate MLIR modules
  • add the needed globals and functions into the lifted LLVM module
  • run the normal LLVM-side cleanup and lowering passes
  • lower the final LLVM dialect module through the x86_64 backend
  • synthesize the required startup and data-section setup for ELF
  • also, lower the scf, arith and other dialects that were still bypassing the normal LLVM-dialect route

Why this PR was needed

Even after the earlier ELF fixes, the self-host path still had a few missing pieces:

  • the driver did not have a full --from-wasm --emit=elf path
  • helper runtime files had to be compiled and spliced into the lifted module in a controlled way
  • the x86_64 backend needed better startup handling for lifted wasm modules
  • global data setup had to work for both normal globals and wasm-lifted globals
  • benchmarking support for the Linux ELF path was missing

Without these changes, the backend could pass many direct tests, but the self-hosting tinyC compiler flow was still incomplete.

In the aarch64 flow, the backend already works as a staged lowering pipeline: first produce LLVM-side MLIR, then run the target-specific lowering, then emit the final object format. The helper code is also brought into the module in a controlled way.

For x86_64 ELF, I followed the same broad approach:

  • keep the driver responsible for preparing the full module
  • splice host/platform helpers into the LLVM module instead of depending on ad-hoc runtime assumptions
  • keep target-specific startup and relocation logic inside the x86_64 backend
  • make the final path work as one staged lowering pipeline rather than scattered target-specific special cases

The difference is that the x86_64 ELF path additionally has to set up wasm linear memory, pass argc/argv for lifted wasm programs, resolve the correct entry symbol, and emit ELF.

Latest benchmark

After the PR-sized backend work, I also recorded the latest timing from benchmarks.md. The latest best-of-3 Linux/x86_64 numbers are:

BENCHMARK 1 — compiler speed
tinyc  (clang -O3 -flto) compiling tinyc : 23.593s
clang  -O0               compiling tinyc : 8.363s
ratio tinyc / clang-O0                   : 2.82x

BENCHMARK 2 — generated-code quality
tinyc built by clang -O0         self-compile : 63.884s
tinyc built by tinyc ELF backend self-compile : 198.722s
ratio tinyc-codegen / clang-O0-codegen       : 3.11x

These numbers are useful because they show two things clearly:

  • the ELF self-host flow is now complete enough to benchmark end-to-end
  • even though the path is working, the generated-code quality and the full self-host flow still have enough overhead to perform more performance optimizations

Commit 1: Handle scf and arith dialects in LLVM lowering

Commit: fix: handle scf and arith dialects in llvm lowering

Problem: Some helper/platform files still had a few scf and arith operations that were being handled directly in the Mach-O backend. The x86_64 ELF backend does not implement that structured lowering, so when those ops reached codegen the self-host path failed.

Fix: This commit moves scf (while, index_switch, if), arith.index_cast, and related cases into mlir_lower_to_llvm.c, so platform backends only ever see LLVM dialect IR. This is in line with the design, where the platform binaries only see LLVM Dialect, for easier handling, and not support multiple dialects at that level.

Commit 2: Add ELF benchmarking module to measure tinyC vs Clang

Commit: fix: add ELF benchmarking module to measure tinyC vs Clang

Problem: The Linux x86_64 ELF self-host flow had no scripted way to compare tinyC against clang on the same workload.

Fix: This commit adds selfhost_tinyc_elf.sh and bench_tinyc_vs_clang_linux.sh for the Linux x86_64 ELF self-host and benchmark flow. It adds the route to add path for self-hosting lowering, and benchmark the time using various configurations, namely:

  • unoptimized clang
  • self hosting built using unoptimized clang
  • self hosting built using optimized clang
  • optimized clang

Commit 3: Emit arith extf/truncf and fnptr field support

Commit: fix: emit arith extf/truncf and fnptr field support

Problem: While testing the larger self-host path, compilation could fail in the emitter even when parsing succeeded. Float widening and narrowing were emitted as unregistered arith.extf / arith.truncf ops that never reached the LLVM lowering path. Struct copy and field-load paths also did not handle function-pointer fields or pointer-to-pointer fields, which show up in vtable-style layouts.

Fix: This commit fixes those gaps in examples/tinyc/emit.c: conversions now use typed OP_TYPE_ARITH_EXTF / OP_TYPE_ARITH_TRUNCF, struct assignment copies TY_FNPTR and TY_PTR_PTR fields like other pointer-like members, and field loads recover the correct fnptr or pointer-to-pointer type for nested callback fields. This adds support for these additional syntaxes in lowering.

Commit 4: Add support for few parsing operations

Commit: fix: add support for few parsing operations

Problem: The self-host script compiles mlir runtime sources with tinyc itself, which exposed parser gaps that build_tinyc_native.sh never hits (that script compiles mlir .c files with clang). One hard stop was in mlir_llvm_to_x64.c:

static const uint8_t ARG_REGS[6] = { RDI, RSI, RDX, RCX, R8, R9 };

Each element is an enumerator name, not an integer literal. Scalar globals already accepted enum constants, but global array initializers only allowed integer literals, NULL, or (void*)0, producing six copies of:

global array initializer element must be an integer literal, NULL, or (cast)0

Fix: This commit links enum_lookup into the global-array initializer path. It also fixes inline function-pointer parameters and struct fields (e.g. void (*progress)(const char *)): the parser reuses the captured fnptr field name instead of expecting a separate identifier token afterward in this new setup.

Commit 5: Add support in driver.c for elf backend selfhost

Commit: fix: add support in driver.c for elf backend selfhost

Problem: The driver lacked the full --from-wasm --emit=elf path needed to splice helper runtime modules into the lifted LLVM module and drive the self-host flow end-to-end.

Fix: This is one of the central commits of the week. It adds the full --from-wasm --emit=elf path in driver.c, compiles helper runtime files into standalone MLIR modules, and splices the needed globals and functions into the lifted LLVM module. This is the main driver-side part that makes the whole self-host path possible. The ELF backend is now compiled in whenever TINYC_ENABLE_ELF is set, not only on native Linux/x86_64 hosts, so a wasm-built tinyc can still emit native ELF after lifting. The old fixed-size pick-buffer splice is replaced by module walks via driver_splice_helper_mod, which moves matching llvm.func and llvm.mlir.global ops with no symbol cap; TinycHelperKind selects platform, buddy, or WASI symbols. Mach-O and aarch64 debug paths were updated to use the same splice infrastructure and also pull in buddy.c for heap globals.

Commit 6: Support wasm-lift ELF linking and unify .data section setup

Commit: feat(x64): support wasm-lift ELF linking and unify .data section setup

Problem: The x86_64 ELF backend treated lifted wasm modules like ordinary native programs, with incomplete startup, global layout, and entry resolution for the self-host path.

Fix: This is the backend-side counterpart of the previous commit. At the x86_64 ELF layer, the backend now treats lifted wasm modules differently from ordinary native programs instead of assuming a single startup model. Global and data-section setup was unified into one layout pass, which reduced duplicate scanning and made it easier to handle globals that optimization passes had already removed. Startup was split into two paths: native binaries keep the usual host init-then-entry flow, while wasm-lifted binaries first set up linear memory of 4GB, link host command-line arguments into the wasm runtime, run host-side platform initialization, and then jump to the correct user entry. The backend also resolves the entry point dynamically, emits only reachable code from that root, and creates small adapter functions when lifted code still references runtime helpers that now live in the spliced host modules. Symbol-name copies in the backend use a local heapdup_cstr helper rather than POSIX strdup, keeping the tinyc bootstrap surface smaller. Together, these changes let the self-host flow produce a standalone Linux ELF binary without relying on libc or a dynamic linker.

These changes were made in response to the following issues faced while compiling the tinyC code. Since these are platform segfault fixes, I took alot of time to debug and fix them:

ELF runtime bugs on the self-host path

Once tinyc could compile and lift the linked wasm module, the resulting tinyc_stage1_elf binary still did not behave like tinyc_native. These were runtime failures on the native-host ELF path.

Segfault at 0xffffffff80000000 on first compile

bench_tinyc_vs_clang_linux and selfhost_tinyc_elf.sh would segfault as soon as stage-1 tinyc tried to compile a real source file (e.g. corec/base/io.c), even though ./tinyc_stage1_elf with no arguments sometimes printed usage and exited cleanly.

On running strace, it showed a successful 4 GiB mmap for wasm linear memory, then:

SIGSEGV {si_code=SEGV_MAPERR, si_addr=0xffffffff80000000}

That address is sign-extended 0x80000000 . It is a valid wasm i32 linmem offset with the high bit set, turned into an invalid host pointer. In the ELF-native WASI adapter, native_out_ptr treated negative i32 offsets as host stack addresses instead of always mapping through wasm linear memory. A related bug wrote 64-bit usize values into 4-byte wasm slots in args_sizes_get, corrupting adjacent stack data. In parallel, some builds picked the wrong crt0 target (e.g. the largest llvm.func in the module instead of the real program entry). Entry resolution of finding main file was updated to prefer app_main, then main, then wasi_start, then _start.

Floating-point exception when argc >= 2

After fixing the pointer bug, compiling with real arguments could still segfault with FPE_INTDIV (div %ecx with rcx = 0) inside app_main before any input file was opened. gdb showed the fault in buddy-heap setup: the wasm startup path normally runs platform init (which calls buddy_init) before user code, but the ELF crt0 was jumping straight to app_main. With the heap uninitialized, arena_create() corrupted buddy metadata.

The fix was to run host-side platform initialization before the resolved user entry on the wasm-lifted path, and to keep argc/argv bridging wasm-sized.

Apart from these fixes, few other smaller fixes were made in fucntion arguments handling, file linking etc, which made the elf backend ultimately self-compile correctly.

Wasm lifting optimizations

After getting the self-host ELF path into a working shape, I also tried simplifying the wasm lifting pipeline.

The stable pipeline today is still:

flowchart TB
  A["linked wasm"] --> B["wasmstack"]
  B --> C["wasmssa"]
  C --> D["LLVM"]
  D --> E["x64"]
  E --> F["ELF"]

For a large linked wasm module, this is expensive because the full wasmstack IR is built and then immediately re-lifted to wasmssa, call sites still resolve signatures by name, and type lists are hex-encoded between stages. I tried several changes to reduce that cost; all were reverted after hitting the issues described below. Currently, they are work in progress, and not pushed to restore stability. This is because since they give issues in assmebly, debugging them take a lot time.

1. Module-level signature tables (ModTables)

Why this was tried: On the self-host path, selfhost_tinyc_elf.sh eventually produces one large linked wasm module with thousands of functions. In the existing wasmstack -> wasmssa lifter, every call and call_indirect had to recover param/result valtypes — often by scanning sibling ops, decoding hex type strings, or walking the module again. That repeated work scales with both function count and call-site count. The idea was to pay the signature cost once up front: build type_sigs[] and func_sigs[] when the wasmstack module is first seen, then have call lowering read a dense table instead of re-deriving types on every site.

What was tried: Added a ModTables structure in mlir_wasmstack_to_wasmssa.c and populated it from a first pass over the wasmstack module before per-function lifting began. Introduced module-level wasmstack.type_sig ops (OP_TYPE_WASMSTACK_TYPE_SIG) carrying type_idx plus dense valtype arrays instead of repeating full signatures on every function. Wired call and call_indirect lowering to read from the table rather than re-scanning by name. In mlir_wasmstack_to_bin.c, added type_to_sig[] so wasmstack-to-binary emission could map type_idx back to the interned signature bytes when writing CALL / CALL_INDIRECT back to wasm.

Error observed: On the full self-host run, the pipeline could segfault after compiling and linking many per-file wasm objects, while tinyc --from-wasm "$LINKED_WASM" --emit=elf was lifting the linked module or writing the ELF output — with no log line pointing at particular signature-table construction. This made it hard to debug the changes.

2. Function-index call resolution (func_idx)

Why this was tried: Wasm links functions by index, not by source name. After tinyc --link merges dozens of per-file .wasm.o objects, the linked module can contain multiple functions that share a linker-visible name from different translation units. The old lifter papered over that by renaming duplicates to foo.<index> and then doing a linear name scan at each call. That is both slow on large modules and easy to get wrong when names collide. Carrying func_idx on wasmstack.func, wasmstack.import_func, and wasmstack.call matches the wasm object model and should make callee resolution O(1) instead of a module walk.

What was tried: Added func_idx and type_idx attributes on wasmstack.func and wasmstack.import_func when wasmstack IR is built from raw wasm. Extended wasmstack.call with a func_idx field while keeping the existing target string for wasmssa emission. Replaced lookup_callee_sig() (name-based scan over module ops) with lookup_func_sig(mod, func_idx) indexed into ModTables. Updated mlir_wasmstack_to_bin.c so CALL encoding prefers func_idx and only falls back to name lookup for older IR. Kept lookup_callee_sig_legacy() as a fallback path. Removed the duplicate-name suffix hack in mlir_wasm_to_wasmstack.c once index-based resolution was in place.

Error observed: No func_idx-specific lifter message (stack underflow / call target not found). The self-host script still hit segfaults on the same stages: during --from-wasm --emit=elf while lowering that linked wasm toward the final ELF binary.

3. Remove hex-encoded type strings between stages

Why this was tried: Wasm valtypes (i32, i64, f32, f64, etc.) were being serialized as hex strings on MLIR attributes — for example param_types = "7f7e" for two i32 parameters. The wasmstack builder encoded bytes to hex, the wasmstack-to-wasmssa lifter parsed hex back to bytes, and wasmstack-to-bin could encode again. On a module with thousands of functions and many calls, that allocate/parse loop shows up in both memory traffic and CPU time. Storing types as type_idx plus a dense array<i32: …> attribute should let each stage pass handles instead of re-parsing strings.

What was tried: On wasmstack function declarations: replaced hex param_types, result_types, and local_types with type_idx and a local_vts dense i32 array attribute. On call_indirect: dropped inline sig_params / sig_results hex blobs and kept only type_idx, relying on the module type table. Added dense_i32_decode() and parse_dense_i32_vts() in mlir_wasmstack_to_bin.c and mlir_wasmstack_to_wasmssa.c to read the new attribute form. Left wasmssa output still hex-encoded where wasmssa_to_llvm still expected hex on wasmssa.func, so the change was staged rather than end-to-end. Added sig_bytes_from_op() with a legacy hex fallback in wasmstack_to_bin.c so mixed old/new IR could coexist during transition.

Error observed:

Compile-time (first):

tinyc emit error at line 177: undefined function: strtol

Location: mlir_wasmstack_to_bin.c, inside dense_i32_decode while parsing array<i32: …> attributes.

After fixing strtol, the self-host run could get past compiling individual mlir sources but still segfault while linking the wasm objects.

4. Direct wasm -> wasmssa fast path

Why this was tried: The linked tinyc wasm blob produced after compiling every mlir runtime source to .wasm.o and running tinyc --link is a high latency process. Today the driver always materializes a full wasmstack MLIR module from those bytes, then immediately runs a second pass to lift wasmstack to wasmssa. ELF emission only needs wasmssa (and then LLVM). Skipping the retained wasmstack module, ie parsing wasm once and emitting wasmssa directly — was meant to cut roughly half the IR construction work and a large slice of peak memory on the lift stage.

What was tried: Added mlir_wasm_module.h with a parsed WasmModule view, mlir_wasm_parse() over linked wasm bytes, and mlir_wasm_decode_func_body() to decode individual function bodies without building a whole wasmstack module first. Implemented mlir_wasm_to_wasmssa() to walk the parsed module, build temporary per-function wasmstack stubs only as lift scaffolding, and emit the final wasmssa module in one pass. Changed driver.c so that for --from-wasm targets other than --emit=wasmstack, the driver calls mlir_wasm_to_wasmssa() instead of mlir_wasm_to_wasmstack() followed by mlir_wasmstack_to_wasmssa(). The fast path reused the same ModTables / dense-type helpers added for items 1 and 3 inside the per-function stub lifter.

Errors observed:

Compile-time:

tinyc emit error at line 198: undefined function: alloca

Location: mlir_wasmstack_to_wasmssa.c, in attr_vts_i32 while building dense i32 attributes.

After replacing alloca with arena allocation, runtime segfault during --from-wasm --emit=elf (linked wasm already built from many per-file .wasm.o objects):

[elf] lifting wasm -> wasmssa...
examples/tinyc/selfhost_tinyc_elf.sh: line 156: ... Segmentation fault (core dumped)

What was tried to fix: alloca → MLIR_GetArenaAllocator(ctx) + arena_alloc in attr_vts_i32(). The segfault was not debugged further before revert.

5. Reuse one unknown location per function lift

Why this was tried: During wasmstack -> wasmssa lifting, almost every emitted MLIR op and result was given a fresh unknown location via MLIR_CreateLocationUnknown. On large functions with tens of thousands of wasm instructions, that means tens of thousands of small location allocations and intern-table traffic that do not affect semantics. The hypothesis was that a single unknown location per lift_func — good enough for debug printing — would reduce allocator pressure without changing IR shape.

What was tried: Introduced a file-scope g_lift_loc in the wasmssa lifter. At the start of each lift_func, create one unknown location and store it in g_lift_loc. Updated build_op_* helpers to pass g_lift_loc instead of calling MLIR_CreateLocationUnknown per op. Cleared g_lift_loc when the function lift finished so the next function got its own location.

Error observed: Self-host runs on the linked tinyc wasm module still segfaulted during --from-wasm --emit=elf (wasm lift or x86_64 / ELF write), due to incorrect resolutions of function name. This was after the per-file wasm compile and link steps had already completed.

6. Pre-size operand and frame stacks

Why this was tried: The wasmstack-to-wasmssa lifter maintains an operand stack and a small control-flow frame stack while walking each function body. In the original code, both stacks started small and grew through realloc as the lifter encountered more wasm instructions or nested control flow. On large compiler functions (many basic blocks, deep nesting), repeated reallocation can dominate micro-ops even when total memory stays modest. Pre-sizing from a cheap estimate of body size should remove most of those growth steps.

What was tried: In lift_func, before walking the wasm body, set F.stack.cap from the wasm instruction count (or wasmstack body op count) so the operand stack vector is allocated once at roughly the right size. Pre-allocated the control-flow frame stack with an initial capacity of 32 entries instead of growing from zero on the first if / loop / block. Left the same lift_body walk logic; only the container sizing changed.

Error observed: On large linked wasm inputs, segfaults still appeared during lift of the linked module or while the driver was emitting ELF — the same crash class as the other optimization attempts, without a separate stack-realloc error string.

Next Steps:

  • Implement the optimizations like elimination of string comparisions, malloc etc for improving the speed of execution of the mlir backend.

Acknowledgments

A huge thanks to my mentors, Ondřej Čertík, Pranav Goswami, Harshita Kalani & other contributors for helping me to work on this project.