Description of memory sharing between master and slaves in pipelines?

[Doug DesCombaz]

I'm trying to find some documentation of how Jenkins shares information between master and slaves. I'm not necessarily interested in the channels, or mechanisms, but the guarantees (wishes) on how the memory would be distributed (I am also interested in the channels, and mechanisms, but that's another question).

So, I know (or rather I think I know) there are two connections, a bi-directional tcp, and a bi-directional of standard io passed through the ssh connection. This is what I keep finding when asking the question, but it's not quite the answer that I am interested in. So, I will ask w/ an example:

In master I have a map of data ([ somekey: "some value"]). I make this data available to a closure that I am passing to slave executors slave-a, and slave-b. If slave-b modifies this map (or it's local copy of it), 1) is the change reflected in the master?, 2) is the change reflected on the other slave-a?, 3) If yes, what are the guarantees (or (un)documented behaviors) here?

I don't know, but I suspect that my self-defined objects are not synchronized throughout the "cluster" (especially since Jenkins distances itself from being considered a cluster). Is that true? I wonder however if properties defined in "env" are (simply because it's a data structure under their control)?

Anyway, if you are able to answer any of those, maybe you can describe how it's being done too just to satisfy my curiosity.

[Ted Xiao]

just serialize and deserialize, it is one-way only, check out https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/UserRequest.java

If you want data shared among slaves, check out http://jgroups.org/javadoc4/org/jgroups/blocks/PartitionedHashMap.html

[Ted Xiao]

More info https://wiki.jenkins-ci.org/display/JENKINS/Making+your+plugin+behave+in+distributed+Jenkins

[Stephen Connolly]

It depends.

You are going to have to just learn about Channels

Ignore Stdio and friends, they are distracting you.

What you have is a Channel... how you got that channel is irrelevant... I have an implementation of Channel (in closed source product) that doesn't even have a persistent connection between the two endpoints... though most "normal" channels are established using either the "JNLP" protocol family or the SSH mechanism.

At its fundamental, Channel sends Callable instances from one side to the other where they get invoked.

The sending involves serialisation, so it is all pass-by-value.

Because that would be somewhat limiting, you can mark some objects as "exported" (providing that they implement an interface to proxy) when you do that, remoting replaces the object with a proxy, where each method call will result in a request being sent over and back... so exported objects are pass-by-reference 

There are a few other special case objects: I/O streams mostly, and also classloaders

For the I/O streams, a more intelligent "proxy" is used (as you cannot proxy classes, only interfaces) which does things like chunking of data to improve performance

For the classloaders, there is some deep voodoo in order to allow reconciling which classloaders to resolve classes from in order to make everything seem seamless.

HTH 

--


You received this message because you are subscribed to the Google Groups "Jenkins Users" group.


To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.


To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2f600c3a-fd9f-4b50-a026-53a3faacc288%40googlegroups.com.


For more options, visit https://groups.google.com/d/optout.

--

Sent from my phone