22 January 2025

Plans for Benben v0.7.0, and porting v0.8.0 to Common Lisp

If you remember, I had a plan back in November of 2023 to port Benben to Common Lisp. That ended up not happening at the time for various reasons (specifically a bug that I had no idea how to work around until much more recently), but the desire to port it remained in the back of my mind. Well, I’ve since worked around that showstopper of a bug, and after weighing the pros and cons of porting it, I think it’s finally time to actually port it.

Benben has changed a LOT since November of 2023, though. It’s now a general purpose player, supporting a lot more formats, and it has added quite a few new features. This means that there’s a lot more stuff to port this time around, not just the VGM code. But I also have a better game plan this time, where I basically do the port in three general steps: library consolidation, updates to my Common Lisp libraries, and porting Benben while I also update the Crystal version. The first two parts of the plan kind of fit together, so I’ll explain that first.

Having library code spread out over a bunch of individual libraries is kind of a dual-edged sword. Sometimes it’s good: do one thing, and do it well. But it’s also difficult with something like Benben, where these Crystal libraries (all but one of which are maintained by me) are conceptually separate, yet still closely tied with Benben’s overall development. This leads to a situation where I’m updating multiple things in parallel, while also maintaining things like a shards.yml, gets to be pretty dang hectic. It’s even worse as a release comes up and I’m having to release not just a new Benben version, but potentially up to seven other libraries.

Now this is the Crystal version we’re talking about. The Common Lisp version will similar, but it will also use more 3rd party libraries simply because the standard library for Common Lisp is smaller than the standard lib in Crystal.

This “stuff spread out over multiple libraries” situation becomes worse when you have packages that live outside of any sort of package manager, which is the case for basically all of my Common Lisp libraries (I don’t think any of them are in Quicklisp yet). Having a user who wishes to compile from source download and install 30 packages is a lot more frustrating than downloading seven, for example.

So, to help me better maintain my various libraries, and make it easier on end users, I’ll be consolidating my libraries further. If you’ve been paying attention, I already did some consolidation with Benben v0.6.0, where the various codecs got moved into a single RACodecs library, and a few other things (e.g. the config file handling) got moved into libremiliacr. Well, I’m going to take this a bit further going forward. Crystal’s compiler will handle the removal of dead code anyway, and I can separate modules into individual ASDF systems that all live within one logical library for Common Lisp.

As for updating things, I’ve maintained Lisp equivalents for most of the Crystal libraries I maintain… well, forever. They’re usually a bit behind, though, so the porting effort will also include various cleanups and updates to my Common Lisp libraries to prepare them for Benben. This is the part that will probably take the most amount of time.

Once the updates are done, I’ll start work on Benben v0.7.0, which will remain in Crystal, and will be a much more incremental update than the previous major releases. Parallel to this, I’ll begin doing the actual port of it to Common Lisp, which will eventually be released as v0.8.0. Doing things in parallel like this will, in my experience, help catch bugs and make v0.8.0 rock solid.

So, the goal is to have v0.7.0 in Crystal, and v0.8.0 in Common Lisp. The overall plan to achieve this is:

1. Consolidate libraries, both on the Crystal and Common Lisp side, while updating my Common Lisp libraries to be feature-equivalent with my Crystal versions.
2. Start work on v0.7.0 proper. As I do, I port Benben to Common Lisp.
3. Release v0.7.0, which is more of an incremental update.
4. Work on new stuff for v0.8.0 begins.

I’ve been working on the first step since the beginning of the year and have been making good progress. Performance is almost identical and memory usage is looking good. So things are underway. I’ll be updating the milestones in the repository soon.