Now you’re talking!

If we make the transducer run on 6 item lag, we can have exactly 6 futures in flight at any time.

All we have to do is to write a bit of code to create a lag. We can recruit clojure.lang.PersistentQueue to do this for us:

(defn build-lagging-transducer "creates a transducer that will always run n items behind. this is convenient if the pipeline contains futures, which you want to start deref-ing only when a certain number are in flight" [n] (fn [rf] (let [qv (volatile! PersistentQueue/EMPTY)] (fn ([] (rf)) ([acc] (reduce rf acc @qv)) ([acc v] (vswap! qv conj v) (if (< (count @qv) n) acc (let [h (peek @qv)] (vswap! qv pop) (rf acc h))))))))

And we can use this to write the transducer generating function: