[ANN] Avout: Distributed State in Clojure

[liebke]

Today we are releasing Avout, which brings Clojure's in-memory model
of state to distributed application development by providing a
distributed implementation of Clojure's Multiversion Concurrency
Control (MVCC) STM along with distributable, durable, and extendable
versions of Clojure's Atom and Ref concurrency primitives.

Here's the post announcing the project: http://clojure.com/blog/2011/11/29/avout.html

And here's the project's website: http://avout.io

David

[Chris Perkins]

Wow.  It will take a while to digest this before I can even dream of what possibilities this opens up.

In the meantime, a couple of simple questions:

1) On the avout.io site, is the diagram of conflicting transactions correct?  It looks to me like the red arrow is in the wrong place (and it doesn't match the description below it, points 5 and 6).

2) Contributing:  pull requests or patches?  (not that I have one yet :)  If patches, where do they go?

thanks,

- Chris

[Gary Trakhman]

The issue with transactions not overlapping with in-memory ones implies some separation to deal with distributed coordination, I think.  Are there any guidelines or interesting papers on how to create an effective distributed architecture with these semantics?

[liebke]

1) On the avout.io site, is the diagram of conflicting transactions correct?  It looks to me like the red arrow is in the wrong place (and it doesn't match the description below it, points 5 and 6).

Great catch, I had intended to fix the figure before release but forgot. It's fixed now.

 

2) Contributing:  pull requests or patches?  (not that I have one yet :)  If patches, where do they go?

Great question, the goal is to leave the door open for including Avout in Clojure-Contrib, which means following as closely as necessary to contrib.'s patch procedure, including requiring signed Clojure CAs, but I haven't pursued the details of this yet.

 

David

[kovas boguta]

Congrats on the release! Looks like the world just got a bit more civilized :)

Particularly excited to see how far this concept of distributed refs
can go while remaining simple:

- Using S3 as the backing store

- Massively distributed STM. For example, every user of clojure
sharing datastructures through a single, massively distributed
datastructure (possibly backed by s3, or user-created stores)

- Integration with clojurescript/browsers (for example via jsonp),
both in the distributed STM and the hypothetical massively distributed
STM

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

[Tim Robinson]

Fantastic !!! .... I'm looking forward to giving this a try. It has
the potential to solve some problems I am currently working on.

so thanks for your efforts.
Tim

[Harrison Maseko]

Could anyone please recommend a good introductory book about
distributed application development? The release of Avout has gotten
me interested in the subject.
Thanks,
Harrison.

[AndyK]

If you're willing to dig into another language, 'Erlang and OTP in
Action' gives a great overview of Erlang and the distributed
principles underlying that language. Though different from the
approach of distributed STM, the concepts of distributed applications
are baked into the core of Erlang.

[Linus Ericsson]

In Clojure in Action (still in MEAP i think) there's a chapter about
using (Erlang/OTP-based) RabbitMQ message queue server for making
Clojure scalable in a Hadoopish map-reduce-way.

Avout seems to solve many of the problems that easily could occur in
such an approach by not needing a centralized "reduce server" for every
calculation.

Another similar way of solving problems of concurrency seems to be
Operational Transformation, which is used in many Google products.

http://en.wikipedia.org/wiki/Operational_transformation

Otherwise +1 for Erlang.

/Linus

[Sean Corfield]

On Wed, Nov 30, 2011 at 2:39 AM, Linus Ericsson
<oscarlinu...@gmail.com> wrote:
> In Clojure in Action (still in MEAP i think) there's a chapter about

CiA was officially released at the Conj. Everyone who bought the MEAP
should have had a notification by now about downloading the final
version (my final CiA eBook is dated November 15th).
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Sam Aaron]

Hi David,

I'm super excited by Avout. It seems *better* than magic in that it not only appears to make complicated things possible, but also in a conceptually transparent way. Crazy cool.

I'm about to look into this in detail, but I thought I'd just post an issue I'm having with the basic example just on the off-chance that I'm doing something obviously wrong.

I have zookeeper 3.3.3 installed running on port 2181 with a valid dataDir path with the appropriate permissions. I'm got avout 0.5.0 and in a REPL session I do the following:

; SLIME 2009-10-15
user> (use 'avout.core)
nil
user> (def client (connect "127.0.0.1"))
#'user/client
user> (def r0 (zk-ref client "/r0" 0))
run-in-transaction exception: #<IllegalArgumentException java.lang.IllegalArgumentException: Path must not end with / character> nil
#'user/r0
user> r0
#<DistributedReference@761b2f32: nil>
user> @r0
nil

Sam

---
http://sam.aaron.name

[David Edgar Liebke]

Hi Sam,

> run-in-transaction exception: #<IllegalArgumentException java.lang.IllegalArgumentException: Path must not end with / character> nil

Very interesting, I wouldn't expect that particular exception unless you named the zk-ref "/r0/" instead of "/r0", which you apparently didn't.

And even when I do call it "/r0/" I get the "Path must not end with / character" exception at a different point in the code, which is probably the result of a patch I added to zookeeper-clj yesterday.

Can you try deleting the zookeeper-clj jar file from your dependencies, blowing about your ~/.m2/repository/zookeeper-clj directory, reloading your deps, and trying again?

David

[Edmund]

Hey David,

I get an identical exception here here w/ zookeeper version
3.2.2. Because I'm crazy that way I also tried to call the ref "/r0/"
(just to see) and the exception came up. However it was different in
that with "/r0/" I got the usual exception handling in emacs, whereas
with "/r0" it just reported the exception in the repl, and then returned
nil.

Edmund

[Edmund]

Hey Sam & Dave,

I'm following along at home. I followed David's advice and there's
no change. FWIW here's the client datastructure and the exception
stacktrace:

#<ZooKeeper State:CONNECTED Timeout:5000 sessionid:0x133fa42621c0004
local:/10.50.0.195:58347 remoteserver:tyrol/10.50.0.184:2181 lastZxid:44
xid:12 sent:56 recv:56 queuedpkts:0 pendingresp:0 queuedevents:0>

java.lang.IllegalArgumentException: Path must not end with / character

at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:58)
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1025)
at zookeeper$set_data.doInvoke(zookeeper.clj:394)
at clojure.lang.RestFn.invoke(RestFn.java:470)
at avout.transaction$update_txn_state.invoke(transaction.clj:82)
at avout.transaction$stop$fn__2604.invoke(transaction.clj:239)
at avout.transaction$stop.invoke(transaction.clj:238)
at
avout.transaction.LockingTransaction$fn__2617.invoke(transaction.clj:321)
at
avout.transaction.LockingTransaction.runInTransaction(transaction.clj:301)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:92)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:30)
at avout.transaction$run_in_transaction.invoke(transaction.clj:354)
at avout.core$zk_ref.doInvoke(core.clj:51)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at clojure.lang.AFn.applyToHelper(AFn.java:167)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3337)
at clojure.lang.Compiler$DefExpr.eval(Compiler.java:375)
at clojure.lang.Compiler.eval(Compiler.java:6470)
at clojure.lang.Compiler.eval(Compiler.java:6431)
at clojure.core$eval.invoke(core.clj:2795)
at swank.commands.basic$eval_region.invoke(basic.clj:47)
at swank.commands.basic$eval_region.invoke(basic.clj:37)
at
swank.commands.basic$eval760$interactive_eval__761.invoke(basic.clj:66)
at clojure.lang.Var.invoke(Var.java:401)
at tavout.core$eval2926.invoke(NO_SOURCE_FILE)
at clojure.lang.Compiler.eval(Compiler.java:6465)
at clojure.lang.Compiler.eval(Compiler.java:6431)
at clojure.core$eval.invoke(core.clj:2795)
at swank.core$eval_in_emacs_package.invoke(core.clj:92)
at swank.core$eval_for_emacs.invoke(core.clj:239)
at clojure.lang.Var.invoke(Var.java:409)
at clojure.lang.AFn.applyToHelper(AFn.java:167)
at clojure.lang.Var.applyTo(Var.java:518)
at clojure.core$apply.invoke(core.clj:600)
at swank.core$eval_from_control.invoke(core.clj:99)
at swank.core$spawn_worker_thread$fn__493$fn__494.invoke(core.clj:298)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:600)
at swank.core$spawn_worker_thread$fn__493.doInvoke(core.clj:294)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:680)

Edmund

[Sam Aaron]

Hi David,

I nuked all my zookeeper deps in my lib and ~/.m2 dirs, but similar to Edmund experience it doesn't fix anything. My stacktrace is also identical:

∴ /Users/sam/tmp/avv
λ lein deps
Downloading: zookeeper-clj/zookeeper-clj/0.9.0/zookeeper-clj-0.9.0.pom from repository clojars at http://clojars.org/repo/
Transferring 2K from clojars
Downloading: zookeeper-clj/zookeeper-clj/0.9.0/zookeeper-clj-0.9.0.jar from repository clojars at http://clojars.org/repo/
Transferring 9K from clojars
Copying 8 files to /Users/sam/tmp/avv/lib
Copying 5 files to /Users/sam/tmp/avv/lib/dev

∴ /Users/sam/tmp/avv
λ lein repl
Listening for transport dt_socket at address: 52602
REPL started; server listening on localhost port 33063
user=> (use 'avout.core)
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
nil
user=> (def client (connect "127.0.0.1"))
#'user/client
user=> (def r0 (zk-ref client "/r0" 0))

java.lang.IllegalArgumentException: Path must not end with / character
run-in-transaction exception: #<IllegalArgumentException java.lang.IllegalArgumentException: Path must not end with / character> nil

at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:58)
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1025)
at zookeeper$set_data.doInvoke(zookeeper.clj:394)
at clojure.lang.RestFn.invoke(RestFn.java:470)
at avout.transaction$update_txn_state.invoke(transaction.clj:82)

at avout.transaction$stop$fn__1104.invoke(transaction.clj:239)
at avout.transaction$stop.invoke(transaction.clj:238)
at avout.transaction.LockingTransaction$fn__1117.invoke(transaction.clj:321)

at avout.transaction.LockingTransaction.runInTransaction(transaction.clj:301)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:92)

at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:30#'user/r0
user=> )

at avout.transaction$run_in_transaction.invoke(transaction.clj:354)
at avout.core$zk_ref.doInvoke(core.clj:51)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at clojure.lang.AFn.applyToHelper(AFn.java:167)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3337)
at clojure.lang.Compiler$DefExpr.eval(Compiler.java:375)
at clojure.lang.Compiler.eval(Compiler.java:6470)
at clojure.lang.Compiler.eval(Compiler.java:6431)
at clojure.core$eval.invoke(core.clj:2795)

at clojure.main$repl$read_eval_print__5967.invoke(main.clj:244)
at clojure.main$repl$fn__5972.invoke(main.clj:265)
at clojure.main$repl.doInvoke(main.clj:265)
at clojure.lang.RestFn.invoke(RestFn.java:512)
at user$eval27$acc__3869__auto____30$fn__32.invoke(NO_SOURCE_FILE:1)

at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:680)

Sam

---
http://sam.aaron.name

[David Edgar Liebke]

Thanks Sam and Edmund,

The stack traces were helpful, I think I understand what the immediate problem is. It appears that the transaction ID in these cases is not getting set, and then Avout is trying to write data to the ZooKeeper node /stm/history/ instead of /stm/history/txid.

Since I can't replicate the problem with my setup, I can't be certain of the cause, but I've added a patch to make the STM more robust in the face of a nil transaction ID, and have updated Avout to version 0.5.1.

Can you guys give the version 0.5.1 a try and see if makes a difference?

If you still see the problem, another diagnostic would be to create a ref without an initial value, and see if that succeeds, then try setting the value.

(def r1 (zk-ref client "/r1"))
(dosync!! client (ref-set!! r1 0))

Thanks again,
David

[Sam Aaron]

Hi David,

thanks for looking into this so promptly. Sadly 0.5.1 just throws a different exception:

user=> (def client (connect "127.0.0.1"))
#'user/client
user=> (def r0 (zk-ref client "/r0" 0))

java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /stm/history/t-
run-in-transaction exception: #<RuntimeException java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /stm/history/t-> nil
at clojure.lang.Util.runtimeException(Util.java:165)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:35)
at avout.transaction$run_in_transaction.invoke(transaction.clj:356)

at avout.core$zk_ref.doInvoke(core.clj:51)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at clojure.lang.AFn.applyToHelper(AFn.java:167)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3337)
at clojure.lang.Compiler$DefExpr.eval(Compiler.java:375)
at clojure.lang.Compiler.eval(Compiler.java:6470)
at clojure.lang.Compiler.eval(Compiler.java:6431)
at clojure.core$eval.invoke(core.clj:2795)
at clojure.main$repl$read_eval_print__5967.invoke(main.clj:244)
at clojure.main$repl$fn__5972.invoke(main.clj:265)
at clojure.main$repl.doInvoke(main.clj:265)
at clojure.lang.RestFn.invoke(RestFn.java:512)
at user$eval27$acc__3869__auto____30$fn__32.invoke(NO_SOURCE_FILE:1)
at clojure.lang.AFn.run(AFn.java:24)

at java.lang.Thread.run(Thread.java#'user/r0
user=> :680)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /stm/history/t-
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:92)

at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:30)
at zookeeper$create.doInvoke(zookeeper.clj:158)
at clojure.lang.RestFn.invoke(RestFn.java:521)
at avout.transaction$next_point.invoke(transaction.clj:62)
at avout.transaction$reincarnate_txn.invoke(transaction.clj:233)
at avout.transaction$stop$fn__1104.invoke(transaction.clj:243)
at avout.transaction$stop.invoke(transaction.clj:238)
at avout.transaction.LockingTransaction$fn__1117.invoke(transaction.clj:323)
at avout.transaction.LockingTransaction.runInTransaction(transaction.clj:303)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:92)

at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:30)
... 17 more

also, trying to avoid setting the value on creation of the ref doesn't work either:

user=> (def r1 (zk-ref client "/r1"))
#'user/r1
user=> (dosync!! client (ref-set!! r1 0))
java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /stm/history/t-
run-in-transaction exception: #<RuntimeException java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /stm/history/t-> nil
nil
at clojure.lang.Util.runtimeException(Util.java:165)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:35)
at avout.transaction$run_in_transaction.invoke(transaction.clj:356)
at user$eval1426.invoke(NO_SOURCE_FILE:49)
at clojure.lang.Compiler.eval(Compiler.java:6465)

at clojure.lang.Compiler.eval(Compiler.java:6431)
at clojure.core$eval.invoke(core.clj:2795)
at clojure.main$repl$read_eval_print__5967.invoke(main.clj:244)
at clojure.main$repl$fn__5972.invoke(main.clj:265)
at clojure.main$repl.doInvoke(main.clj:265)
at clojure.lang.RestFn.invoke(RestFn.java:512)
at user$eval27$acc__3869__auto____30$fn__32.invoke(NO_SOURCE_FILE:1)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:680)

Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /stm/history/t-
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
atuser=> org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:92)

at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:30)
at zookeeper$create.doInvoke(zookeeper.clj:158)
at clojure.lang.RestFn.invoke(RestFn.java:521)
at avout.transaction$next_point.invoke(transaction.clj:62)
at avout.transaction$reincarnate_txn.invoke(transaction.clj:233)
at avout.transaction$stop$fn__1104.invoke(transaction.clj:243)
at avout.transaction$stop.invoke(transaction.clj:238)
at avout.transaction.LockingTransaction$fn__1117.invoke(transaction.clj:323)
at avout.transaction.LockingTransaction.runInTransaction(transaction.clj:303)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:92)

at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:30)
... 12 more

Sam

---
http://sam.aaron.name

[David Edgar Liebke]

Did you initialize the STM?

(init-stm client)

You only need to do it the first time, to set up the necessary zookeeper nodes, it's described in the main tutorial but not the snippet on the top of the avout site.

David

[Edmund]

That's the ticket! Thanks David, its working for me now.

On 01/12/2011 17:21, David Edgar Liebke wrote:
> (init-stm client)

[Sam Aaron]

On 1 Dec 2011, at 17:21, David Edgar Liebke wrote:

> Did you initialize the STM?
>
> (init-stm client)

Works perfectly for me too. Perhaps it might help to add that to the example snippet to stop idiots like myself falling into that trap :-)

Sam

---
http://sam.aaron.name

[David Edgar Liebke]

Fantastic, I've added the init-stm step to the code snippet at the top of the Avout site, it was an accident to leave it out.

It's been great "pairing" with you two :)

David

> Works perfectly for me too. Perhaps it might help to add that to the example snippet to stop idiots like myself falling into that trap :-)
>
> Sam

[Sam Aaron]

Out of interest, why is #'init-stm a separate step to #'connect

I tried looking at the docstrings for each fn but they were both nil :-(

Sam

---
http://sam.aaron.name

[David Edgar Liebke]

init-stm (and reset-stm) are only used for creating an STM that will be used by every client. You only want to call it once, not every time a client connects.

I could be more clever about calling it when it's clear that it hasn't been called before though. I'll dedicate that patch to you :)

> I tried looking at the docstrings for each fn but they were both nil :-(

Oops, I'll fix that.

David

[liebke]

Just released Avout 0.5.2, which now includes automatic STM
initialization (no more pesky init-stm step).

David

> >>>      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:25)

> >>>      at java.lang.reflect.Method.invoke(Method.java:597)
> >>>      at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:92)
> >>>      at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:30)
> >>>      at zookeeper$create.doInvoke(zookeeper.clj:158)
> >>>      at clojure.lang.RestFn.invoke(RestFn.java:521)
> >>>      at avout.transaction$next_point.invoke(transaction.clj:62)
> >>>      at avout.transaction$reincarnate_txn.invoke(transaction.clj:233)
> >>>      at avout.transaction$stop$fn__1104.invoke(transaction.clj:243)
> >>>      at avout.transaction$stop.invoke(transaction.clj:238)
> >>>      at avout.transaction.LockingTransaction$fn__1117.invoke(transaction.clj:323)
> >>>      at avout.transaction.LockingTransaction.runInTransaction(transaction.clj:303)
> >>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3 9)
> >>>      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp l.java:25)
> >>>      at java.lang.reflect.Method.invoke(Method.java:597)
>

> ...
>
> read more »

[Sam Aaron]

On 1 Dec 2011, at 18:26, liebke wrote:

> Just released Avout 0.5.2, which now includes automatic STM
> initialization (no more pesky init-stm step).

Ha, throw down the gauntlet, then beat me to it ;-)

Great work,

Sam

---
http://sam.aaron.name

[Glen Stampoultzis]

On 2 December 2011 05:26, liebke <lie...@gmail.com> wrote:

Just released Avout 0.5.2, which now includes automatic STM
initialization (no more pesky init-stm step).

The init-stm step is still referenced in the documentation as being required BTW.

I had a couple of questions.

I noticed that when I create a reference (zk-ref) I need to provide an initial value.  For each VM I do this for - it ends up clobbering the previous value.  Is there anyway to create a reference without necessarily clobbering existing data?

My second question is to do with derefs. The documentation says that Avout caches multiple derefs and that it will invalidate the cache when the ref is locally or remotely updated.  My own testing seems to indicate that the deref still see the old values after a remote change is made.  If I wrap the deref in a dosync!! however that seems to trigger the invalidation and I see the correct value. Am I missing something?

Great work on this library.  Distributed STM has a lot of potential.

-- Glen

[Chris Shea]

On Fri, Dec 2, 2011 at 12:33 AM, Glen Stampoultzis <gst...@gmail.com> wrote:

I noticed that when I create a reference (zk-ref) I need to provide an initial value.  For each VM I do this for - it ends up clobbering the previous value.  Is there anyway to create a reference without necessarily clobbering existing data?

You don't need to supply an initial value. I had to look at the source to figure that out, though. Leaving off the value will just create a reference to an already existing ref. See the second set of arguments here: https://github.com/liebke/avout/blob/master/src/avout/core.clj#L59

Chris

[David Edgar Liebke]

Hi Glen,

>
> The init-stm step is still referenced in the documentation as being required BTW.
>

Thanks, I'll remove the reference.

> I had a couple of questions.
>
> I noticed that when I create a reference (zk-ref) I need to provide an initial value. For each VM I do this for - it ends up clobbering the previous value. Is there anyway to create a reference without necessarily clobbering existing data?

Yep, like Chris said, you can just leave off the initial value when creating a Ref or Atom, just like with the in-memory versions.

>
> My second question is to do with derefs. The documentation says that Avout caches multiple derefs and that it will invalidate the cache when the ref is locally or remotely updated. My own testing seems to indicate that the deref still see the old values after a remote change is made. If I wrap the deref in a dosync!! however that seems to trigger the invalidation and I see the correct value. Am I missing something?
>

I can't reproduce this behavior, can you provide a code snippet where the cache isn't getting invalidated and maybe some additional details on your setup?

David

[Glen Stampoultzis]

On 3 December 2011 01:44, David Edgar Liebke <lie...@gmail.com> wrote:

Hi Glen,

>
> The init-stm step is still referenced in the documentation as being required BTW.
>

Thanks, I'll remove the reference.

> I had a couple of questions.
>
> I noticed that when I create a reference (zk-ref) I need to provide an initial value.  For each VM I do this for - it ends up clobbering the previous value.  Is there anyway to create a reference without necessarily clobbering existing data?

Yep, like Chris said, you can just leave off the initial value when creating a Ref or Atom, just like with the in-memory versions.

Okay, thanks. Not sure how I missed the second arg list when I looked up the docs.  

A related question.  If I wanted to do a once-off initialization of the value (ie, the first VM to create the distributed ref will set the value) how would I go about it?

 

>
> My second question is to do with derefs. The documentation says that Avout caches multiple derefs and that it will invalidate the cache when the ref is locally or remotely updated.  My own testing seems to indicate that the deref still see the old values after a remote change is made.  If I wrap the deref in a dosync!! however that seems to trigger the invalidation and I see the correct value. Am I missing something?
>

I can't reproduce this behavior, can you provide a code snippet where the cache isn't getting invalidated and maybe some additional details on your setup?

I think I've narrowed down what was triggering the problem for me.  In my code I had something like this:

    (dosync!! client

              (alter!! r0 inc)

              (alter!! r1 conj @r0)

              (Thread/sleep 20000))

If I deref r1 in the second VM before the thread has finished sleeping I get the expected (unmodified) value. Once the thread finishes sleeping if I deref it again it doesn't update.  If instead I wait until after the thread finishes before doing the deref then it shows the correct value.

Let me know if you still can't reproduce it and I'll try to provide more detail.

Regards,

Glen

[David Edgar Liebke]

A related question.  If I wanted to do a once-off initialization of the value (ie, the first VM to create the distributed ref will set the value) how would I go about it?

I would just create the Ref without an initial value, and if the current value is non-nil, you know that it has already been initialized.

I think I've narrowed down what was triggering the problem for me.  In my code I had something like this:

    (dosync!! client

              (alter!! r0 inc)

              (alter!! r1 conj @r0)

              (Thread/sleep 20000))

If I deref r1 in the second VM before the thread has finished sleeping I get the expected (unmodified) value. Once the thread finishes sleeping if I deref it again it doesn't update.  If instead I wait until after the thread finishes before doing the deref then it shows the correct value.

Let me know if you still can't reproduce it and I'll try to provide more detail.

That was perfect, the problem occurs because I was invalidating the cache as soon as the alter!! function was called, you then deref-ed it on the VM, which asked for the current value (because it's cache had just been invalidated) where it received the unmodified version (and cached it), then the commit occurs on the first VM without invalidating the cache again. 

So, I've changed the strategy, I invalidate the cache at commit instead of at alter!! (actually doSet), which causes your example to work correctly.

The new version is:

[Avout 0.5.3]

Let me know if you run into any other issues.

Thanks for the feedback,

David