Subject matter overview: A long post digging into details on a very small, inconsequential (I do not say this in a self-deprecating way) command line tool, written first in Clojure, and then in Crystal. I wrote half this post while very tired, late at night, and then the rest in the morning when I was refreshed.

Topics: Clojureº | GraalVM | Crystalº | Native Binaries | CLI Tools

Overview

This weekend I dove into Crystal to re-write a half-finished program called Pod-Dodgerº. Pod-Dodger is a tool for bulk downloading episodes of Podcasts. You feed it a yummy RSS string, and then it starts downloading episodes for you. Perfect for long road-trips and those moments of stepping into ghost circles (à la Bone) where you find yourself with intermittent-to-no internet access. It seems to happen to me quite a bit these days.

From "Ghost Circles (BONE #7)"

Pod-Dodger is a command line tool. I originally wrote it (a half-working thing) in Clojure, to test out GraalVM native image compilation. In this log, I'll mostly talk about how weird it has been to wrap my head around object oriented programming in Crystal and differences in work-flow when working with Clojure. I've jumped the fence, and I'm in a new pasture.

Original lispy intentions

Let's go over the original program a bit. When I'm using Clojure, I try and think in capital-D-Data. You'll hear that in the lisp world, or at least, the Clojure world. It's one of those phrases. It's all about the data. This is a simple project and so it is built around filling up and operating on a small map of data that looks like this:

(def config {:out-dir "resources/tmp/"
             :feed         ""
             :episodes     []
             :num-episodes nil
             :curr-file    {:name  nil
                            :url   nil}})

Just some data that will get passed around.

What's next? Well, we have a -main function that parses a few command line arguments, and then sets up a config as per above, passes it through some functions, and then the program ends. Is that really so different from a world of objects, stemming from other objects? Do any differences really appear or even matter in such a small project? Who knows. It's past my bed time and I'm still writing.

(defn -main
  ""
  [& args]
  (let [c (assoc config :feed (first args))] ; attach the url of a feed to the config map.
    (setup (config :out-dir))                ; run a setup function
    (get-episodes c)))                       ; run get-episodes, passing in our config.

For the original Pod-Dodger in Clojure, I pulled in a library for parsing rss feeds. This was my first mistake. Another thing you'll hear about in the Clojure ecosystem is that it's all about pulling together disparate small libraries that do a few good jobs. Well, this project was small enough that I should have looked into rolling my own solution. Here's why:

  • All I needed was to find the mp3 URL links in the RSS feeds (ie, XML) were URLs to mp3 files. I could have probably just found a really smart regex that could have done it.

  • You are rolling the dice that a library will compile with GraalVM without some kind of modification.

Let's expand on that last point.

In which I talk much on Native Binaries

I'm all about native binaries. I just think they're really cool. Sometimes I talk about them when I have lunch and the people/inanimate objects who have to suffer my enthusiasm must begin to plumb the depths of their mind for internal entertainment while I rattle on. Is this a symptom of slogging through strange front-end build processes? Maybe.

I was excited when I found out that I could make native-binaries with Clojure, one of my favourite languages, allowing me to make quick-starting, multi-platform binary applications. My poor plants and the few remaining friends that still had the glimmer of interest in their eyes! Soon they too would lower themselves into a vacuous emptiness, a shell of their former selves while I talked; GraalVM was going to make it possible to use Clojure in ALL THE SITUATIONS, and wouldn't you like to hear me talk about it?

But it's not that simple. For several reasons, programs do not "just compile" on GraalVM; with Clojure, you mostly need to figure out how to avoid reflection. This usually means telling your programs to warn you if any interop with Java is going to cause reflection. You can get Clojure to warn you about reflection by putting this line at the top of your file:

(set! *warn-on-reflection* true)

; I still don't know what those stars mean on either side of the variable name. I'll look it up someday.

Ok, not a big deal. But when you're using someone else's libraries, there's a good chance some reflection is going on there (we all need to reflect a little more, don't we?) and that it's going to break your native image. For me, this meant I had to go in and vendor the library and make some changes to the source code. This was a good exercise for me, because I often forget how to do type hinting in Clojure and because it makes you dig into other people's code. But it's also a bit of a wild goose chase.

Finally, it takes a while to compile your programs in Graal. It takes a while, and a lot of ram. I think this is improving with new releases of GraalVM, but 5 minutes and laptop fans-a-blasting is a rough feedback cycle for a toy project. I think I finally gave up in favor of starting a new project. I left Pod-Dodger in the dust for a few months.

An original naive attempt

I started thinking about Crystal a week or so ago, after having written some ruby for workº. I don't have a ton of experience with Ruby. I worked at an old-growth-startup (you get to make up what that means in your own head!) and did a bit of Rails, but that's about it. At the time I only had experience with Javascript, and was pretty new to... a lot of things. I had built a few toy applications in Rails, but mostly felt like it was a weird magical world where the route to the other end of the maze was pretty well-trodden, and you had to work fairly hard to get lost. With that said, you never really had to spend too much time gazing around at the structure of the maze in the first place (are you seeing what I'm doing here? Metaphor.)

So, I had some fun with Ruby at work recently and started thinking about how it was actually pretty nice to write. Then I remembered Crystal and decided it would be a good candidate to re-write the ol' PoddyDodger in. My first attempt, which worked, was very NOT classy. I could not reformat my brain to think in classes. I just wanted to use functions. The original attempt looked something like this:

def get_xml_from_url(url)
  # ...
end

def save_file(mp3)
  # ...
end

def get_mp3s_from_xml(xml) fc = xml.first_element_child out = [] of Hash(String,
  String)

  # this is messy, could be recursive + use .next_sibling?
  if fc
    fc.children.each do |child|
      if child.name == "channel"
        items = child.children.select { |c| c.name == "item" }
        items.each do |item|
          enclosure = item.children.find { |f| f.name == "enclosure" }
          title = item.children.find { |f| f.name == "title" }.try &.content
          url = enclosure["url"] unless enclosure.nil?
          out << {"title" => title, "url" => url} unless title.nil? || url.nil?
        end
      end
    end
  end
  out ## << Trying to be functional.
end

def do_all(url) ## < Prototype names are best names.
  xml = get_xml_from_url(url)
  mp3s = get_mp3s_from_xml(XML.parse(xml))
  mp3s.each do |mp3|
    save_file(mp3) unless mp3s.nil?
  end
  mp3s.each { |mp3| save_file(mp3) } unless mp3s.nil?
end

module Poddodger VERSION = "0.1.0"
  # My "state"
  feed = ""
  xml = ""

  p = OptionParser.parse() do |parser| parser.on("-f FEED", "--feed=FEED", "URL
    of podcast rss feed") { |url| do_all(url) } # where things happen. end

  if ARGV.empty? puts p; exit 0 end end

Look at me! Wanting to use recursion! No classes in sight, no instance variables, a bunch of functions that took input and returned an output.

Trying hard to be classy

The whole time I wrote the above code I knew I was going against conventions. I eventually made some changes, as you will see below. I re-wrote my program with a class.

But I have to tell you something.

It felt so weird to modify variables.

(Pause to re-read. Is this all I'm saying here?)

class Pododger
  property :feed_url, :out_dir

  @feed_url = String
  @out_dir = String # should this be  or 
  @mp3s = [] of Hash(String, String)

  def initialize
    @out_dir = nil
    @feed_url = "" # should be nilable?
  end

  def download
    get_mp3s_from_xml
    @mp3s.each { |mp3| save_file(mp3) } unless @mp3s.nil?
  end

  def out_dir=(f : String)
    @out_dir = f
  end

  def save_file(mp3)
    spinner = Term::Spinner.new("[:spinner] Fetching: #{mp3["title"][(0..64)]} ...", format: :dots)
    spinner.auto_spin
    cleaned_title = mp3["title"].gsub("/", "-") # replace problem chars.
    resp = Halite.follow.get("#{mp3["url"]}")

    if @out_dir
      file_path = "#{@out_dir}/#{cleaned_title}.mp3"
    else
      file_path = "#{cleaned_title}.mp3"
    end

    #....

Look at all those instance variables, happily grazing and ready to mutate on command! I, the functional no-body, has arrived to the classy party. I do not think anyone has discovered my unchanging, immutable, true identity.

Can you believe it? Has functional programming warped my mind (for better or worse? I'm biased: better) so much that I'm becoming a programmer that tends to only think in one sort of process? Oh no! Am I invariably becoming the one-trick-pony who is inflexible, and has to go to bed at 9:30? I thought I was a juggler, juggling at least three one-trick pony's at a time.

Oh well.

An unnecessary back-story.

Listen, listen, listen. I am not going to give you the whole time-line of programming for me. But basically, when I entered this world Angular 1.x was the in-thing. I missed the whole world of Java, the Ruby train, and the whole carnival of classes, and constructors. It was JavaScript from the beginning! I was jumping stairs for JS! From JS, I strayed into functional programming with Clojure. Aside from playing with Processing, oop was pretty much goop.

I felt like I was doing everything backward. The narrative I had seen quite often was that of the scarred Object-Oriented-ist who discovers functional programming, has an a-ha! moment and never wants to go back to anything else.

But I didn't really have anything to compare and contrast. So what's the point of this log? Maybe one point is that starting your programming journey in the functional world is very possible, and just as some may find going to functional programming difficult and strange, the reverse can be similar (so... don't fret?)

Little projects are little joys

I really enjoy working on little projects like this. At this point in my puttering-ways, I know when I'm starting to get into dangerous-over-ambition territory with projects. Little projects like this definitely scratch an itch to try new languages, new ways of thinking with low risk, get done in a reasonable time, and provide something a little useful, at least for me. I saw someone who wrote an OS or something in Crystal... I'm pretty far from that, but that's okay too!