Overview

At the beginning of March 2020 I started working on FirnÂș. At the time, I wanted to create an MVP of a static site generator for org-mode files. According to my task list, this involved the following:

  • Setup Parser [100%]

    • [X] Find a good org-parser.

    • [X] Slurp single file

    • [X] Parse file

    • [X] Display one basic element (properties?) with hiccup.

  • Render a single org file to html [100%]

    • [X] Slurp File

    • [X] render it to file as is.

Things have come a long way since then. I found an excellent parser called Orgize written in Rust. The parser is fast and seems well written. As someone who knows very little about Rust I can't really comment on code quality, but I was blown away by the scope of org that is covered. In my research I had also found org-rs which seemed much more popular based on it's GitHub star count. While the owner has clearly put in a lot of work, they had made note in an issue that they no longer had the bandwidth to keep developing the project. And as usual, GitHub stars can't really be counted on as a measure of quality or future-proof-ness. Org-rs had been posted to Hackernews at one point, where I imagine a lot of interest came from.

I also investigated a few Clojure org-parsers, but each that I found appeared fairly abandoned / untouched. I only cloned and messed around with one seemingly actively developed project; 200ok-ch/org-parser, but by that point I was already invested in Orgize.

I achieve my MVP pretty quickly, using Clojure/Java's standard library to shell out to a parser binary I compiled with Orgize. To my surprise, it worked very nicely. Soon I dove deeper and deeper into what might be possible to create a static site generator for org-mode. Meanwhile, at the back of my mind remained the interest and desire to bundle my code base as a native-image with the GraalVM. The farther along I got, the more I grew doubtful that it would be possible.

How I got things compiling

My previous attempts with GraalVM on my project Pod-DodgerÂș had left me skeptical when it came to getting Clojure code compiled without reflection errors. So, I continued puttering on the project, developing more features and cobbling together a working prototype. Then, I stumbled on some of Bork Dude's (Michiel Borken) work. He seems to be quite invested in the success of GraalVM and has used it very successfully with his projects (Babashka and Clj-Kondo, both use the GraalVM to build native images). He's quite present on the Clojurians Slack channel, and had helped me out several times when I got stuck. He contributes to a useful repository called graalvm-clojure which provides lots of useful information.

But beyond these resources, I also found an example he built of compiling a native-image that can interface with other languages via Graal. By this time I was starting to grow quite hopeful that this would be something possible with Firn. I knew from some helpful articles, however, that use of eval would prevent compilation. I was using eval and read-string in Firn to enable users to create their own layout functions that could be read and applied to the parsed org tree as edn.

This was the only remaining blocker, and Michiel's work saved the day again. He built sci - a Small Clojure Interpreter for ... well stuff like this, I suppose. I was thrilled to find that I was able to drop the library directly into Firn and I was eval'ing functions outside of the binary with no problem at all. I definitely don't quite understand what is happening, but this talk goes into some details that covers SCI. I will probably watch it again soon.

Anyway, with that accomplished, I had a native-binary on my hands before I knew it. I was super pleased. I didn't have to compromise the flexibility of the REPL driven work-flow that had me quickly developing the prototype for Firn - I simply replaced the call to the parser with the original shelling out code so that I could do that from the REPL when in Dev-mode, and then have the proper JNI FFI behaviour in the native binary.

The only thing that is a downside, and it really isn't a big deal, is that the.dylib or .so artifacts of the Rust library have to exist on disk somewhere, and can't be called from the resources folder of the Java class path. So, as I borrowed from Michiel's work, he just copies the lib into a unique folder at the root directory at runtime.

What's next?

I've since moved on to re-writing the test suite so that I can catch my breaking changes more easily. It's been a while since I've written Clojure tests, so they are probably a bit more brittle than they could be, and I'm a little out of touch with how to test a CLI tool, but it's coming along nonetheless.

Next up I have one of two features that I would like to do, once I merge my test-suite re-write: either a dev-server, or a file-watcher. Eventually, both should work together, but I haven't decided which will be more useful. Currently, whenever I change org files, or styling for the generated site, I have to re-run the binary, or my compile commands in the REPL and then refresh in the browser where I'm running an http-server of the file output.

I know less about file-watching then setting up a server, but I think both should be sufficiently feasible to do, at least in the REPL. Whether or not they will be able to compile to a native image is another issue. The GraalVM is a bit of a rabbit hole; it has tons of flags, and configuration that I know very little about. It has been giving me interesting insight into the Clojure<>Java ecosystem (class paths, reflection, etc) which is great, but there is lots of fiddling->compiling->repeat.

Further along down the line, if the project maintains enough velocity past it's first release, I could see a plugin system similar to Jekyll's being developed (and it probably wouldn't be too hard thanks to the function pipelines ).

What's the point of failure?

I don't want to think about this stuff too much, because I'm fairly invested in this pretty fragile, glued together solution. The most likely failure point I can think of currently would be the abandonment of the orgize library. This wouldn't be a huge problem, as I've forked and saved the code for reference and would be willing to hack on the library at the added benefit of learning some Rust along the way. Beside that, I'm not sure how maintained the project is right now; it has covered pretty much every use case of org-mode that I personally have, and since this is a personal project, that's totally fine to me (I'm sure that if I post the project online lots of folks would have a different experience). I suppose the org mode "spec" could change a bit, but that seems pretty unlikely, at least for the basic functions I am making use of right now (clocking, logging, properties, tables, src blocks quote blocks).

Wrapping up

It's been a super fun project for the past little while, and I'm already iteratively using my compiled native-image to build the wiki for this site.