Cantata's Scheduler; Data Objects
Arnold Rots
Mark Young responded on January 16, Jeremy Worley on January 17.
There is not a whole lot to be added to that discussion:
-The idea of tabs in procedures sounds like a great idea.
-The notion that glyphs only be executed once in a do-loop in
order to control implied loops seems quite reasonable, although
I assume that explicit nested loops are exempted from this.
-Funny to see that there is apparently no agreement on how the
trigger glyph works.
However, in another project, in the meantime, an old issue returned:
how to properly handle objects in a dataflow-driven system?
Consider an object that consists of a collection of data points,
model parameters, and instrumental parameters. The idea is to
iterate on the model parameters (say, in a count-loop) in order
to fit the model to the data. I dislike having to pass that object
from glyph to glyph since that generates several copies (of several
MBytes each), while in reality there is only one object around
at any point in time - the "current" one.
So I thought that I could solve this be letting the object be
generated by the input section of the workspace and then providing
it as input to all glyphs in the iterative fitting loop. The iterative
process would be controlled by the individual glyphs' passing
fairly small things, such as chisq and derivatives, and they would
all have access, through an input, to the one-and-only object that
contains the data and model information; the key is that they would
also be allowed to modify this input. Although the model would keep
its identity and stay in the same place, it would gradually change
as the iterations went on, until the final fit is reached.
Everything looked even better when I realized that this big object
could be accessed by every glyph through shared memory and be
memory-mapped. There were recently some discussions on how
to conserve disk and memory space while processing large objects
or images, and this seemed like an excellent way to go.
But, in the exchange between Matthew Yeates and Mark Young on
February 6 on "strange use of Sun virtual memory" it was made
clear that such practice may not only be frowned upon, but even
be impossible under Khoros 2.0.
Mark writes:
>Currently the transport mechanisms can attach to a sun virtual memory map or
>a shared memory segment, but it shouldn't allow a routine to re-use the input
>as the output. What would happen is that it would directly change the image in
>shared memory or in the case of sun virtual memory on disk. I think most users
>would be frustrated to have their input image corrupted. Or in the case of
>Sun Virtual Memory when a file is opened with kopen() for reading if you try
>to write to the data a bus error is generated. For this reason we have decided
>to wait until Khoros 2.0, when we rewrite these routines, before adding this
>capibility.
>
>If there is enough demand we maybe able to add this alternate readimage() or
>make it available as an optional set of utilities via a toolbox!?!? This
>way if you agreed to abide by the rules that you would not corrupt the input
>image you would have access to this capibility.
I would submit that not all users may be frustrated when their input
data are changed; that it may even be their intention; and that there
may be good reasons for wanting to do this.
So, I would suggest that you do nothing rash, but think carefully
before excluding any options.
I have a suspicion that the issue of object integrity and object
identity may become more and more important, and I think it should
be kept carefully in mind in all design considerations.
Have a nice day,
- Arnold Rots