Clojure

2023-01-24, 2023-01-27, 2023-02-12, 2023-03-30, 2023-07-29

I’m attempting to learn some Clojure. I have very little Lisp experience aside from starting to read “The Little Typer”. The FP style appeals though and Clojure continues to appear as an Interesting and Important Thing. I’m also interested in playing with Clerk after watching the Stop Writing Dead Programs talk.

I now have a small pet project I want to try which seems like a good excuse to pick it up (or a way to make a simple pet project 5x more difficult).

To keep things simple, I’ll not use details of my real project below. Instead, I’ll just substitute it with toy examples.

Plans

Goals

I have a (fairly) simple project in mind. It requires making a query to an API, storing the results somewhere (just a file will do), and then displaying those results on a web page.

Non-goals

In the early stages of this project I’m very happy for pages to be static. Using a static site generator to convert some markdown to HTML seems like an easy first step, so I can move that part out of my project for now.

So - read an API, write some results, create some markdown is the goal.

Learning resources

Books

I started to go through the about guide on the Clojure site but it seemed to require a bit more of a Lisp background (I now realise that I was looking in the wrong place) so I looked for some books to follow along with. Programming Clojure looked good, but for some reason the O’Reilly app crapped out on me (I’m in Bali while writing this, so I’m blaming the network) so I’ve gone with Clojure for the Brave and True. I’m very much a “show me the codes” type of learner, so the informal style and liberal code blocks help me skim around happily.

Setting up

Bootstrapping a Clojure environment with Nix and Leiningen

I’m using nix to manage my dev environments. In my first attempt at creating a project I first initialised my environment with Leiningen as a dependency to create the project in place.

$ lein new app <projectname> --to-dir . --force  # Force because I've got my flake in the directory already

However, because of i) this, or ii) because I ran lein new without the app template and then ran it again with it, or iii) something else wrong I did that I haven’t spotted yet, I ran into some trouble getting test-refresh running.

Instead, I recommend this flow. First create the project

$ nix-shell -p leiningen
$ cd ~/code # or wherever you keep your projects
$ lein new app <projectname>
$ exit

Then create your Nix enviornment with [jre clojure leiningen] as its dependencies. I’ve included jre so that I can run things like java -jar target/uberjar/project-0.1.0 after building, otherwise I get the following error

The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.

Picking an IDE (it’s Vim for me, friends)

I’m going to use Neovim. I’ve so far resisted the urge to pick up Spacemacs so I can limit the number of moving parts. The editor guide on the Clojure site gives vim-iced and Conjure as options for Neovim plugins. Conjure + vim-jack-in installed easily via Nix’s vimPlugins directive, so that’s what I’ll stick with for now. The :ConjureSchool interactive tutorial was pretty helpful too.

I’ve done minimal customisation so far. Only using splitright so REPL results appear on the right, and setting my local leader (run the tutorial to see more) to _.

let maplocalleader = "_"
set splitright

To connect Clojure to Vim I first start the REPL…

$ lein repl  # The first run takes ages, just wait it out

nREPL server started on port 53585 on host 127.0.0.1 - nrepl://127.0.0.1:53585
REPL-y 0.5.1, nREPL 1.0.0
Clojure 1.11.1
OpenJDK 64-Bit Server VM 19.0.1+10
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

project.core=>

Then inside Vim run :Lein (if it doesn’t connect automatically). That’s it. Now I can execute the various Conjure commands, eg _ee, to evaluate Clojure forms.

Getting going

OK, enough with the setup. I want to start playing around! I’ve read a chapter or two of …Brave and True. I barely know anything! That should be enough to get going, right?

Making HTTP requests

I want to make HTTP requests. One tiny Google search gives me clj-http and the ability to add it to my dependencies.

  ;; project.clj

  :dependencies [[org.clojure/clojure "1.11.1"]
                 [clj-http "3.12.3"]]

Restarting the REPL allows me to make an HTTP request!

nREPL server started on port 60059 on host 127.0.0.1 - nrepl://127.0.0.1:60059
REPL-y 0.5.1, nREPL 1.0.0
Clojure 1.11.1
OpenJDK 64-Bit Server VM 19.0.1+10
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

project.core=> (require [clj-http.client :as client])
nil
project.core=> (client/get "https://mastodon.return12.net/@jammus.json")
{:cached nil, :request-time 890, :repeatable? false, :protocol-version...}

Let’s try the same inside Vim by editing the default core.clj.

;; src/project/core.clj

(ns project.core
  (:require [clj-http.client :as client])
  (:gen-class))

(client/get "https://mastodon.return12.net/@jammus.json")

Executing the whole file - _ef - gives similar, but better formatted, output.

I can then name the response and inspect its data by either typing commands straight into the REPL or by executing the forms in Vim. I can get the values of keys on the response object by using the name as a function.

(def response (client/get "https://mastodon.return12.net/@jammus.json"))

(:request-time response) ; => 741
(:body response) ; => "{\"@context\":[\"https://www.w3.org/ns/activitystr...

This feels quite neat.

Parsing JSON

OK, this turns out to be pretty easy. I can add :as :json to the get call.

(client/get "https://mastodon.return12.net/@jammus.json" {:as :json})

This initially errors.

;; (err) Execution error (AssertionError) at clj-http.client/coerce-json-body (client.clj:461).
;; (err) Assert failed: json-enabled?

This is because I’m missing cheshire as a dependency. Adding it to project.clj and restarting the REPL (is there a way to do this automatically?) fixes the issue.

  :dependencies [[org.clojure/clojure "1.11.1"]
                 [clj-http "3.12.3"]
                 [cheshire "5.11.0"]]

Now I can query the json body.

(:inbox (:body response)) ; => "https://mastodon.return12.net/users/jammus/inbox"

Iterating over things

I want to extract data from a list of things. The map function looks to be what I want.

(def response (client/get "https://mastodon.return12.net/api/v1/timelines/public" {:as :json}))
(map (fn [toot] (:id toot)) (:body response)) ; =>
   ; ("109747906354007378"
   ; "109747767282221519"
   ; "109747601974560741"
   ; ...
   ; "109746421529975578")

Looking up a value from a map using the short form anonymous function was a bit tricky to figure out, but it looks like this:

(map #(% :id) (:body response)) ; =>
   ; ("109747906354007378"
   ; "109747767282221519"
   ; "109747601974560741"
   ; ...
   ; "109746421529975578")

Functions / Naming things

I’m feeling a bit weird having these bits of code floating around so I’m going to give them some names.

(defn fetch-timeline 
  [] 
  (:body (client/get "https://mastodon.return12.net/api/v1/timelines/public" {:as :json})))

(defn extract-details
  [toots]
  (map #(% :id) toots))

I can then wire them together like this:

(extract-details ; => ("109747906354007378" ...)
  (fetch-timeline))

Extracting multiple values from the json object

I’ll want multiple properties from each item, so I’ll create a new map instead of just extracting the ID. Again I found this hard to reason about in the short form. Very verbose it looks like this:

(defn extract-details
  [toots]
  (map (fn [toot] {:id (:id toot) :url (:url toot)}) toots))

I don’t like that at all. Far too noisy. Ah! I can use select-keys!

  (map (fn [toot] (select-keys toot [:id :url])) toots))

This is much easier to put back into the short form, and helps me understand the % placement better.

(defn extract-details
  [toots]
  (map #(select-keys % [:id :url]) toots))

OK, cool. I guess. I feel a little uneasy about the structure here - I might just be creating work for myself - but we’ll see how we get on as I learn more. I’m happy to be ugly/non-idiomatic for now (but maybe not for very long).

Writing to a file

I want to store this extracted details into a file so I can reuse it later as an input to another part of the project. This prevents me from having to do API calls when I just want to regenerate the final output.

There are a pair of functions called slurp and spit which deal with reading and writing files. I’ll also need to convert the map back to json with cheshire’s generate-string.

(ns project.core
  (:require [clj-http.client :as client])
  (:require [cheshire.core :refer [generate-string]])
  (:gen-class))

(spit "details.json"
  (generate-string
    (extract-details
      (fetch-timeline))))

A little bit of refactoring

I don’t really like the nesting of the main function and I alwats prefer the visualisation of a pipeline to run left-to-right. I can use Clojure’s thread-last macro (-->) to apply the result of a form as the last argument to the next.

(->> (fetch-timeline)
     (extract-details)
     (generate-string)
     (spit "details.json"))

Now I don’t like the lack of symmetry or the configuration of the file name being prominent. Instead, I’ll turn the use of spit in to a unary function and move the config out. I’ll do this by creating a little helper function to write json and then create a partial of it with the filename defined.

(defn write-json
  [filename data]
  (spit filename (generate-string data)))

(def write-details-json
  (partial write-json "details.json"))

(->> (fetch-timeline)
     (extract-details)
     (write-details-json))

Writing tests

So far we don’t have much in the way of logic, but that will soon change. I want to get into the rhythm of testing now so that it doesn’t hurt me more later.

When creating a new project with lein new a core_test.clj is created with a default failing test inside.

(ns project.core-test
  (:require [clojure.test :refer :all]
            [project.core :refer :all]))

(deftest a-test
  (testing "FIXME, I fail."
    (is (= 0 1))))

I can run this from the command line with lein test but it’s quite slow to run. Well, it’s four seconds, but that’s almost enough time for me to get completely distracted.

I should be able to use run-tests in the REPL but it doesn’t find the tests.

project.core=> (use 'clojure.test)
nil
project.core=> (run-tests)

Testing project.core

Ran 0 tests containing 0 assertions.
0 failures, 0 errors.
{:test 0, :pass 0, :fail 0, :error 0, :type :summary}
project.core=>

Practicalli gives a hint that it might be classpath related.

Ensure test directory is on the class path when evaluating tests in the REPL,
otherwise the (deftest) test definitions may not be found.

However, when I print the classpath I can see the test directory is included.

I’m able to add a (run-tests) to the bottom of core_test.clj and evaluate the whole file but I don’t like scattering artifacts like that around the place cos I know I’ll forget to remove them. Also, having to constantly evaluate tests and the function(s) under test is going to require a lot of keystrokes.

test-refresh seems to do what I want. Adding it to my profiles.clj as a plugin allows me to run lein test-refresh and, on any change, tests are re-ran.

  ;; ~/.lein/profiles.clj
  {:user {:plugins [[com.jakemccrary/lein-test-refresh "0.25.0"]]}}

However, due to some error in my setup, tests are not being ran.

*********************************************
*************** Running tests ***************
:reloading (project.core-test .)
:error-while-loading .

Error refreshing environment: java.io.FileNotFoundException: Could not locate /__init.class, /.clj or /.cljc on classpath. java.io.FileNotFoundException: Could not locate /__init.class, /.clj or /.cljc on classpath.
Finished at 12:15:41.289 (run time: 0.281s)

This seems to be a problem with how I first created my project as it works fine on a completely fresh one. I couldn’t see any obvious differences in configuration or the code files, so I just created a new project and copied the source files back across.

Hmm, now that I’ve got it working I’m not so sure it’s what I want. Maybe executing forms inside Vim is better… we’ll see.

Another option is to bind :ConjureEval (use 'clojure.test)(run-tests) to something and run this after evaluating test/functions.

I used a combination of both to put together the following simple test.

(ns project.core-test
  (:require [clojure.test :refer :all]
            [project.core :refer :all]))

(def single-toot (
 list {:id 4 :url "https://google.com" :other "property" }
))

(def multiple-toots (
 list
   {:id 4 :url "https://google.com" :other "property" }
   {:id 6 :url "https://example" :ignore "this" }
   {:id 3 :url "https://last.fm" :no "good" }
))

(deftest extract-details-test
  (testing "empty list when empty"
    (is (empty? (extract-details []))))
  (testing "strips unrequired properties"
    (is (=
         (first (extract-details single-toot))
         {:id 4 :url "https://google.com"})))
  (testing "strips from multiple toots"
    (is (=
         (extract-details multiple-toots)
         (list
           {:id 4 :url "https://google.com"}
           {:id 6 :url "https://example"}
           {:id 3 :url "https://last.fm"}))))))

Accepting command line arguments

I’m imagining this project as a number of small, somewhat independent command line utilities likely triggered by cron jobs. I don’t want any individual piece to get too complicated while learning Clojure. Also, I don’t want to do too much infrastructure type work before it’s necessary. So no long-running tasks or processes. Each job might need a little bit of configuration so I need a way to pass that in at run time. Command line arguments or environment variables are good candidates.

Accepting command line args is as simple as binding them to a variable in the main function.

(defn -main
  [single-arg]
  (println (str "Called with " single-arg)))

This will fail with a Execution error (ArityException) error if the argument is omitted. Which might be what you want but not necessarily user-friendly. Instead, it’s better to use an ampersand to bind all received arguments to a sequence.

(defn -main
  [& args]
  (println (str "Called with" args)

It’s also possible to bind individual arguments using destructuring and then check for their existance.

(defn -main [& [domain]]
  (if (nil? domain)
    (println "Missing domain argument")
    (println (str "Domain set: " url))))

This is fine for positional arguments but for something even more user-friendly there’s tools.cli.

(ns project.core
  (:require [clojure.tools.cli :refer [parse-opts]])
  (:gen-class))

(def cli-options
  [["-d" "--domain DOMAIN" "Mastodon domain to query"]
   ["-h" "--help"]])

(defn -main
  [& args]
  (let [opts (parse-opts args cli-options)]
  (println (str "Called with " opts)))

parse-opts will construct a map based on the options definition and the arguments passed. Passing in --help, eg lein run -- --help, will give

{:options {:help true},
 :arguments [],
 :summary "  -d, --domain DOMAIN  Mastodon domain to query\n  -h, --help",
 :errors nil}

Which you can use to print some help text with (println summary).

Setting --domain mastodon.return12.net will mean options becomes:

{:domain "mastodon.return12.net"}

You can then respond to each case quite simply (see the tools.cli example for more options).

(defn -main
  [& args]
  (let [{:keys [options summary]} (parse-opts args cli-options)]
    (cond
      (:help options)
      (println (str "Query the supplied Mastodon instance\n" summary)) 
      (:domain options)
      (println (str "Called with " (:domain options)))
      :else
      (println "Missing domain argument"))))

One mild gotcha I ran into was that my value for :domain kept coming through as true rather than the domain I’d passed in. This was because I had accidentally missed off the DOMAIN from the argument definition.

Using this map of options, I can then change the fetch-timeline function to accept a domain.

(defn fetch-timeline 
  [] 
  (:body (client/get (str "https://" domain "/api/v1/timelines/public") {:as :json})))

(defn -main
  [& args]
  (let [{:keys [options summary]} (parse-opts args cli-options)]
    (cond
      (:help options)
      (println (str "Query the supplied Mastodon instance\n" summary)) 
      (nil? (:domain options))
      (println "Missing domain argument")
      :else
      (->> (fetch-timeline (:domain options))
           (extract-details)
           (write-details-json)))))

If I want to accept the same parameter multiple time, eg multiple domain, I can use :multi and :update-fn.

(def cli-options
  [["-d" "--domain DOMAIN" "Mastodon domain to query"
    :multi true
    :update-fn conj]
   ["-h" "--help"]])

...
(->> (:domain options)
     (map fetch-timeline)
     (flatten)
     (extract-details)
     (write-details-json))

or replace (flatten (map)) with mapcat.

(->> (:domain options)
     (mapcat fetch-timeline)
     (extract-details)
     (write-details-json))