Ensure more concurrency

236 views
Skip to first unread message

bertschi

unread,
Mar 6, 2017, 6:06:46 AM3/6/17
to Clojure
For a lecture I have implemented the write-skew example from Mark Volkmann's article:

(defn write-skew []
  (let [cats (ref 1)
        dogs (ref 1)

        john (future
               (dosync
                 (when (< (+ @cats @dogs) 3)
                   (alter cats inc))))
        mary (future
               (dosync
                 (when (< (+ @cats @dogs) 3)
                   (alter dogs inc))))]
    (do @john @mary)
    (dosync
     (> (+ @cats @dogs) 3))))

(defn write-skew-demo []
  (let [count-write-skews (atom 0)]
    (doseq [_ (range 25)]
      (when (write-skew)
        (swap! count-write-skews inc)))
    @count-write-skews))

(write-skew-demo)

When I try to fix this program using ensure, i.e. using (ensure dogs) in john and (ensure cats) in mary, I find that it sometimes runs very slow.

From the docs it says "Allows for more concurrency than (ref-set ref @ref)". I would read that as "runs at least as fast as (ref-set ref @ref)", but using (ref-set dogs @dogs) instead of (ensure dogs) and the same with cats, does not exhibit these occasional runtime spikes.

Am I misunderstanding something or is it a bug in ensure?


Nils

lawrence...@gmail.com

unread,
Mar 7, 2017, 5:35:09 PM3/7/17
to Clojure



Must be called in a transaction. Protects the ref from modification
by other transactions.  Returns the in-transaction-value of
ref. Allows for more concurrency than (ref-set ref @ref)


This can be read in two contradictory ways. Protecting a ref during a transaction sounds like it increases contention and slows the app down. "Allows for more concurrency" can be read as "Allows you to use a huge number of refs while still ensuring consistency." In other words, it sounds like it offers safety at the cost of speed. 

Herwig Hochleitner

unread,
Mar 8, 2017, 8:40:01 AM3/8/17
to clo...@googlegroups.com
2017-03-06 12:06 GMT+01:00 'bertschi' via Clojure <clo...@googlegroups.com>:

> From the docs it says "Allows for more concurrency than (ref-set ref @ref)".
> I would read that as "runs at least as fast as (ref-set ref @ref)", but
> using (ref-set dogs @dogs) instead of (ensure dogs) and the same with cats,
> does not exhibit these occasional runtime spikes.
>
> Am I misunderstanding something or is it a bug in ensure?

If ensure does indeed show worse (or not even better) performance than
(ref-set ref @ref) in benchmarks, it certainly would go against my
understanding of clojure's STM and I'd consider this a bug (since
ensure could just be implemented by ref-set for a free performance
boost, in this case).

To show this behavior and move this along into a ticket, it would be
helpful to run those two examples in a microbenchmark runner, like
criterium and attach a profiler to uncover locking problems and
excessive retries, that would likely be the root cause.

bertschi

unread,
Mar 9, 2017, 6:34:21 AM3/9/17
to Clojure
Thanks for your comments. As suggested I ran a small benchmark of both versions. Turns out that the difference between (ref-set ref @ref) and ensure is huge ...

(defn write-skew [de-ref]

  (let [cats (ref 1)
        dogs (ref 1)

        john (future
               (dosync
                 (when (< (+ @cats (de-ref dogs)) 3)

                   (alter cats inc))))
        mary (future
               (dosync
                 (when (< (+ (de-ref cats) @dogs) 3)

                   (alter dogs inc))))]
    (do @john @mary)
    (dosync
     (> (+ @cats @dogs) 3))))

user> (bench/with-progress-reporting
        (bench/bench (write-skew (fn [x] (ref-set x @x))) :verbose))
Warming up for JIT optimisations 10000000000 ...
  compilation occurred before 18580 iterations
Estimating execution count ...
Sampling ...
Final GC...
Checking GC...
Finding outliers ...
Bootstrapping ...
Checking outlier significance
amd64 Linux 3.16.0-4-amd64 4 cpu(s)
OpenJDK 64-Bit Server VM 24.121-b00
Runtime arguments: -Dfile.encoding=UTF-8 -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -XX:-OmitStackTraceInFastThrow -Dclojure.compile.path=/home/bertschi/GitRepos/Vorlesungen/PS2/target/classes -Dps2.version=0.0.0 -Dclojure.debug=false
Evaluation count : 1675560 in 60 samples of 27926 calls.
      Execution time sample mean : 37,327714 µs
             Execution time mean : 37,325021 µs
Execution time sample std-deviation : 1,137506 µs
    Execution time std-deviation : 1,147482 µs
   Execution time lower quantile : 34,971387 µs ( 2,5%)
   Execution time upper quantile : 39,511899 µs (97,5%)
                   Overhead used : 12,902114 ns
nil

user> (bench/with-progress-reporting
        (bench/bench (write-skew ensure)))
Warming up for JIT optimisations 10000000000 ...
  compilation occurred before 40 iterations
  compilation occurred before 61 iterations
Estimating execution count ...
Sampling ...
Final GC...
Checking GC...
Finding outliers ...
Bootstrapping ...
Checking outlier significance
Evaluation count : 60 in 60 samples of 1 calls.
             Execution time mean : 1,662454 sec
    Execution time std-deviation : 1,906919 sec
   Execution time lower quantile : 186,394000 µs ( 2,5%)
   Execution time upper quantile : 7,630998 sec (97,5%)
                   Overhead used : 12,902114 ns

Found 4 outliers in 60 samples (6,6667 %)
    low-severe     2 (3,3333 %)
    low-mild     2 (3,3333 %)
 Variance from outliers : 98,3142 % Variance is severely inflated by outliers
nil

I'm running clojure 1.8.0 by the way. According to VisualVM the (ref-set ref @ref) version spends most of its time in clojure.lang.LockingTransaction$RetryEx.<init> () while ensure gets basically stuck (> 99%) at clojure.lang.LockingTransaction.tryWriteLock ().
Reply all
Reply to author
Forward
0 new messages