Go-like concurrency (and profiling!) in Hy and PyPy

Waking up at 6:30 AM has bestowed upon me a blissful, irritant-free period that is perfect for contemplation, so of course I used it for a few experiments instead.

With all the hoopla around Go concurrency, a surprisingly large number of folk have been pursuing similar avenues for writing nice, clean concurrent code, and analogues have sprung up all over the place – like Clojure‘s core.async, for instance, which is around a year old and which I’m quite fond of.

In Python land, one of the available options is Rob Galanakis’ goless, which I’ve been tracking fairly closely. It provides Go-like primitives and works wonderfully out-of-the-box with PyPy, providing in-process coroutines with pretty much zero hassle.

As it happens, PyPy is now my default Python interpreter1, so goless is a very nice addition to my toolkit, all the more so now that I’m starting to benchmark and profile most of my code as a matter of course – partly to figure out where the bottlenecks are, and partly to figure out ways to make it go faster2.

And it’s been quite entertaining, really – I’ve found a few interesting things here and there, and profile results from even trivial bits of code is quite informative.

For instance, here’s one of the goless examples (the very first one I picked up, really), rewritten in Hy (and with inline profiling until #618 is addressed):

; A straight port of the goless library example (runs fine with the PyPy backend)
(import [goless [go chan]]
        [cProfile [Profile]]
        [pstats [Stats]]
        [functools [partial]])

(defn produce [msgs done count]
    (for [i (xrange count)]
        (.send msgs i))
    (.send done))

(defn consume [msgs out name]
    (for [msg msgs]
        (.send out (% "%s:%s" (, name msg)))))

(defn logger [out]
    (for [msg out]
        (print msg)))

(let [[p (Profile)]
      [done (chan)]
      [msgs (chan)]
      [out  (chan)]]
      ; enable profiler
      (.enable p)
      ; start a producer, three consumers and a logger
      (go produce msgs done 10000)
      (map (partial go consume msgs out) ["one" "two" "three"])
      (go logger out)
      ; wait for completion, stop profiler and dump stats
      (.recv done)
      ; stop profiler
      (.disable p)
      (.dump_stats (Stats p) "out.pstats"))

…and here’s the profile result (in SVG format), courtesy of gprof2dot:

Next up, applying this to a number of things I’m building around an MQTT broker…

  1. Since 2.2.x, and for most things that don’t require numpy, that is. Ironically, I’m falling back to the system Python via pyenv to do most of my number-crunching for the moment. ↩︎

  2. Which brings to mind that I need to get to grips with profiling in Go as well. It’s not magical, you know. ↩︎