8 October 2025

Benben v0.7.0 Postmortem (or "why I moved it to Lisp")

This post is just in response to a question I’ve gotten a small number of times: why did you rewrite Benben in Common Lisp? Rather than linking them to two posts that mentioned it in passing (and, in hindsight, probably didn’t explicitly state why clearly enough), I’ll just do a short final post on the subject. I don’t want to keep dragging out “why”, either.

But first: I like Crystal, I really do. It’s a really nice language that’s productive to work in, and it’s fun to boot. There are a few additions/changes that I’m not 100% sold on yet in recent and upcoming versions, but as these mainly don’t affect the type of code I work on, they aren’t a big deal. I’ll still use Crystal in some of my other projects.

But, it wasn’t the right fit for Benben. It almost was… but not quite.

Threads and Parallel Execution

First, remember parallel is not the same as concurrency. To use an analogy, with concurrency, you cut your meat, then put it in the pan to cook. While it’s cooking, you turn and set up your side dish. Then you turn back to your pan, stir it a bit, then turn away and set the table. Parallelism is where one hand is cutting the meat, the other is preparing the side dish, one foot is turning on the stove, the other is taking out the pan, and you’re also having a conversation with your cat about the new anime episode you saw last night, all at the same time.

Benben was designed around parallel multi-threading because, from the onset, I figured it was going to mainly run on everyday laptop/desktop hardware, and these tend to have multiple cores. Plus, I generally like the concept of splitting tasks up across threads in parallel. It’s a fun way to solve problems, and it can be really efficient. Plus, rendering files to disk is an embarrassingly parallel problem well-suited to parallelism.

Anyway, Crystal does not give you true access to OS threads. You can kinda do it using undocumented APIs, but those can break between versions (see more about this in the “Smaller Things” section). Instead it favors “Fibers”, a type of green thread that exists in the runtime and is then scheduled on one or more real OS threads (which are created and owned by the runtime). This is also called “N:M” scheduling because N fibers are scheduled on M threads. More recent versions are slowly introducing what the Crystal folks call “Execution Contexts”, where (roughly) rather than having a global CRYSTAL_WORKERS environment variable that dictates how many OS threads the runtime uses for its fibers, you instead get to say “I want X number of threads”, it gives you a context with that many, and then you can use that to run your Fibers. In truth there’s more to it than “I want a context for X threads for my Fibers”, but it’s what Benben would have ended up using.

The older method was causing severe hiccups/stuttering when CRYSTAL_WORKERS was too small. Benben actually tried to override this environment variable at startup, but starting with Crystal 1.15.0, that override method stopped working. This was explored in my previous blog post that I’ve already linked above.

Unfortunately it was a difficult problem to solve, so I was stuck. I needed to support versions of Crystal back to 1.5.x due to that being the version in some BSD repos (and I fully believe the average user should not be arsed to install versions of compilers outside of what comes with their system packages if they don’t want). Execution Contexts weren’t ready when I started 0.7.0, so I couldn’t really do conditional code internally, either. I could just sit and wait, but… yeah, no. That wasn’t going to happen. Every solution I thought of was a bandage, while what I actually needed was simple: access to OS-level threads (or at least a better runtime scheduler).

Of the languages I’m willing to use, it boiled down to Object Pascal or Common Lisp (specifically SBCL). With how much I love Lisp, I chose the latter.

Write-Compile-Test Cycle

Moving to Common Lisp meant that I was also gaining a better write-compile-test cycle. SLIME mode for Emacs is just plain wonderful to use. After the initial startup, I can write code and just C-c C-c to compile/hot-load the function (or C-c C-l to load the whole file) into the running Lisp implementation, which compiles it to native machine code near instantly. I can do this even as the program is running and see the effects immediately. The debugger is built-in, and with a few workarounds, I can also start up SBCL in a terminal window, load Benben’s code there, and then connect to it “remotely” over a socket so that I can also work on the TUI (which doesn’t work in a Slime buffer in Emacs). Thus development has, for the most part, instantaneous feedback.

Compare this to writing code, tabbing out, waiting for Crystal to compile a semi-release version (because debug versions were sometimes too slow, depending on the code I was writing), launching the program cold, then repeating the cycle. Crystal’s compile times tend to be on the long side, not because of the compiler itself (Crystal itself is damn fast), but because of the LLVM backend (nearly all the time each compile was spent in the LLVM compilation stage), and it was getting to be frustrating.

Moving to Lisp made development less stressful overall.

Smaller Things

There are just a few more minor things to mention, all of which contributed to my decision to port Benben once they were taken into account collectively with the other aforementioned reasons:

• Maintaining compatibility to older Crystal versions was getting out of hand. I would have had to vomited conditional code all over the codebase if I moved to Execution Contexts, possibly to the point of it being near unmaintainable code. Common Lisp hasn’t changed since 1994, and SBCL hasn’t had any backwards compatibility issues that would cause this sort of conditional-compilation-spewage. Crystal really just isn’t a good choice for parallel code unless you always run newer version… 😅
• SBCL would give me really nice SIMD support out-of-the-box. No assembly needed (though I can still do assembly with it if I need).
• No LLVM. My system-wide LLVM is 13.0 on my Slackware 15 boxes, and 21.1.2 on my Slackware-current box. Crystal sometimes doesn’t support what I have, and that’s annoying. Plus I just don’t like LLVM for various other reasons. SBCL, however, has its own compiler and linker and all that good stuff.

The Downsides

There were always going to be a few downsides, however, but I knew of them from the very start. I have, after all, written Common Lisp code a lot longer than I’ve written Crystal code. In the end, I determined that none of these were serious enough to move me away from Common Lisp.

• Larger binaries. Common Lisp uses image-based development similar to Smalltalk, so the entire runtime gets included in every binary (which is just a small kernel + the image core), and you can’t call strip on them. SBCL (and the build system for Benben) does offer optional ZStandard compression of the resulting binary, though (and the AppImages are always compressed as well). Uncompressed and outside of an AppImage, a Benben binary now stands at 103mb.
• Higher memory usage. Common Lisp simply uses more RAM, though technically not too much more. If you look at it in htop, it’ll seem a bit larger than it actually is. This is because the image core is mmap()‘ed into its memory space, and this sorta muddies what’s reported in tools like htop. See footnote number 1 in this post for a better explanation.
• I had to rewrite a lot code. Benben isn’t just its own repository. It’s every support library I’ve written for it, from the VGM emulators (which are not C-bindings, but native Lisp code), to all the DSP/effect/format/resampler/FLAC codec/most other codecs (mostly not C bindings), to the MIDI synth (also not C-bindings)… it’s a lot. If I use gocloc to add everything up, it comes out to roughly 115k LOC. Some of that already existed, some had to be reworked, and most was new code ported from my equivalent Crystal libraries.
• No shards program. It’s now harder for non-Lispers to compile Benben from source, partially because of the number of dependencies it pulls in. The vast majority of these aren’t used by any of my code directly, but are dependencies of dependencies. The shards program made it awfully easy for end users to compile the program, even though they didn’t call it directly. I do plan to implement my own solution for Benben v1.0.0 that’ll be similar to shards, however, as well as do some NIH to reduce external dependencies. Plus I offer AppImages, mostly for this very reason.

One notable omission above is “speed”. As it turns out, SBCL has given me code that is mostly close to the performance of the older Crystal version. Some bits are faster, some the same, some slower. But overall I’d still put it in the same overall category of “quite fast”, and that makes me especially happy.

Wrap Up

So there you have it. I don’t want to revisit this topic again. I’ll be using Crystal for other stuff going forward, just not for anything that requires a lot of parallel code. Benben v0.7.0 is exactly where I want it.