The week of the misplaced parenthesis
After leaving MetaMed, Alyssa spent 2015–2016 building a knowledge-graph-backed CRM (customer relationship management) tool of her own, written in Clojure. Each week she sent a short technical progress note to a small group of friends and former colleagues — fellow programmers and AI researchers — describing what she had built or fought with. This installment is one of those notes: a debugging detective story about a single misplaced parenthesis that swallowed three hours, plus brief explanations of two Clojure features she was newly using, Selenium for browser automation and destructuring for cleaner function arguments.
I did lots of smallish things this week, of which the hardest was probably finding a misplaced parenthesis. That sounds like a joke, but it was really quite tricky and took about three hours. It started off with a function that looked like this: (defn fix-errors [stuff] (map #(if (is-fixable-error? %) (fix %) %) stuff)) Normally, this looks at each item in the collection “stuff”, checks to see if there’s an error, and fixes it if there is one. Simple. But there was a wrong parenthesis, like this: (defn fix-errors [stuff] (map #(if (is-fixable-error? %) (fix %) %)) stuff) In most languages, this would be a syntax error, which would get flagged during compilation — the function “map” isn’t being applied to anything, since “stuff” is outside the parentheses for “map”. But in Clojure, if you call “map” without “mapping” anything, you get an object called a transducer, which is a special kind of higher-order function (functions that map functions to other functions). I love Clojure, but transducers still mystify me; I’ve yet to see an example where they make code shorter rather than longer.
Anyway, this caused errors to propagate through, until they triggered an exception about twenty functions later. Normally, I’d just pull out the Emacs debugger and trace the error back… except that I didn’t see any error message. To make development easier, Clojure provides a way of running the entire HTTP server and everything attached to it from inside Emacs. And since there wasn’t a special case to catch that exception, Emacs apparently decided to route the exception message and all the stack traces to /dev/null. So I wound up looking at code like this: (defn do-stuff [foo] (...do some stuff...)
(print "do-stuff finished") bar)
(defn do-more-stuff [bar] (do-stuff bar) (print "do-more-stuff finished")) and “do-stuff finished” would print, but “do-more-stuff finished” wouldn’t — it was like the computer just spontaneously fell asleep in the middle of running the program. Lesson learned: always have default exception handlers, where you can log the error message into a nice file you control from a nice logging library that’s designed for this stuff, and it won’t disappear into the aether. :)
Fortunately, I got to spend most of my time on more interesting things, two of which were Selenium and destructuring. Selenium is like an API for the web browser — it’s a library controlled by your program which can instruct the browser to do various things. For example, here’s the Selenium code I wrote to simulate a user signing up for my CRM: (def signup-button "input.pure-button.pure-button-primary") (def gmail-button "a.pure-button.pure-button-primary > h3") (def permission-button "submit_approve_access")
(defn create-account [] (let [driver (firefox-driver)] (.get driver (env :app-domain)) (-> "signupUsername" by-id find (send-keys (env :test-acct-email))) (-> "signupPassword" by-id find (send-keys (env :test-acct-pwd))) (-> "signupConfirm" by-id find (send-keys (env :test-acct-pwd))) (-> signup-button by-css find .click) (-> gmail-button by-css find .click) (-> permission-button by-id find .click))) This piece of code will fire up a copy of Firefox, load the application, find where all the buttons are, type in the username and password, and click through the signup dialog boxes, all in about half a second (except for the time it takes GMail’s turtle servers to approve email access :p). Selenium can also check to see if, after you’ve clicked through some stuff, the webpage you get at the end looks like it ought to look, which is great for integration testing.
Destructuring is yet another interesting feature of Clojure, which I’ve started using a lot only recently. For example, suppose that you’re calculating the score for a sports game. The total number of points is (making something up) three times the number of passes, plus eleven times the number of goals, minus the number of players multiplied by the number of minutes it takes to finish. You could write a score function like this: (defn point-total [game] (+ (* 3 (:passes game)) (* 11 (:goals game)) (* -1 (:players game) (:length game)))) But that’s redundant. With destructuring, you can define all the variables you need implicitly, like so: (defn point-total [{:keys [passes goals players length]}] (+ (* 3 passes) (* 11 goals) (* -1 players length))) I saved like 150 lines of code this way, it’s great.
Now that there are good filters for junk data at every level, I’ve also been working on taking a bunch of low-reliability data from various NLP algorithms, and incorporating it into the graph. Coming along nicely, IMO, but not yet finished. Once that’s done, though, it should be able to just scale and scale and scale — instead of just things like “where does Bob live?” and “what is Bob’s job?”, it’ll pick up data like (for the real estate agent application) what the square footage of Bob’s house is, when the house was built, what color paint the house has in the kitchen, what style of roof Bob installed, and all this rich domain-specific stuff that you’d never get anywhere else.