event spawning subevents...how to assure their priority

[Steve Cumming]

Hi, gang.

A module "yieldTables" is designed to load and preprocess an arbitrary number of datasets that are to be converted into stand Volume Age Tables. It's convenient that this be done by having the module's init event spawn specialized events within the same module, one for each unique dataset that turns up. The problem is to ensure that these spawn-of-init are schedule between now and any subsequent event that depends on the module. 

Can you advise how this may be arranged? 

I also suggest a new tag: scheduling, details of.

Steve

[Steve Cumming]

Also, too, setting eventPriority=1 does not do the trick.

sc

[Eliot McIntire]

You can put a scheduleEvent(   ) anywhere. So, you can put those inside the functions that must spawn different events.  So, judicious and precise use of scheduling event calls, outside of the doEvent(  ) function could be an option. 

eventPriority=1 will is only a secondary sorting key. The primary sort key has to be exactly the same in two events in order for the eventPriority to be relevant.

If this doesn't answer your question, you can try clarifying more.

[Steve Cumming]

The scheduleEvent calls I am occur within the init event of moduleA. These events need to occur before the init event of ModuleB. The sequence of the init events is more or less as expected given the dependencies, but it seems there may be no way to insert events into the midst of a sequence of module init events. Is this correct? 

sc

[Eliot McIntire]

Your description needs to be clearer. We have not yet implemented conditional scheduling, though we have planned it out. Issues 143,https://github.com/PredictiveEcology/SpaDES/issues/143
Is that what you are talking about?

[Steve Cumming]

No, Eliot, conditional scheduling is not what I was trying to do.

Let's see if I can be clear.

modules<-list("foo", "bar")
mySim<-simInit(...,...modules,...)
spades(mySim)

The resultant schedule is
eventTime  moduleName eventType eventPriority
1:         0   foo      init             5
2:         0   bar      init             5

properly, because the bar has dependencies on foo, as specified in the
modules' inputObjects and outputObjects

within foo.init these is code like this:

  ytFs <- file.path(inputPath(sim),ytFs)
  ytFs <- dir(ytFs)  #this will be a vector of names of visible files
  for(i in ytFs)     #and yes, various sanity chex would be appropriate
    sim <- scheduleEvent(sim, time(sim), "foo", i, .highest())

The resultant schedule is

eventTime  moduleName eventType eventPriority
1:         0   foo      init             5
2:         0   bar      init             5
3:         0   foo      AB             1

but what I want is

The resultant schedule is
eventTime  moduleName eventType eventPriority
1:         0   foo      init             5
2:         0   foo      AB             1
3:         0   bar      init             5

Is there a way to bring this about? It's not crucial, there are other ways
I can the job done other than through the scheduler, but it just seemed
more elegant to try.


Steve

[Eliot McIntire]

Ah. I see. So, by definition, inits come first by all. If you want to have more control over these things, then make a post init event section for each module. Then you can make whatever you want. i.e., in both module A and B, you schedule the postInit events at time zero, with explicity eventPriorities.

in A

scheduleEvent(postInit, A, 0, eventPriority=1),  

in B

scheduleEvent(postInit, B, 0, eventPriority=5)

[Steve Cumming]

That solution, Eliot, would break the pieces up more than I think is desirable or safe....I want all this to happen in the init. 

The use case is having multiple sources of volume-age data for different regions, each requiring specialized handling. The cases reveal themselves in the source directory for G&Y data, and the cases (subdirectories) ones that have an appropriate method are processed. 

I'll just have to replicate that despatch mechanism within the YieldTable modules init event. 

sc

[Eliot McIntire]

Thinking about the priority mechanism, it should work as you proposed above. We can look into this next week: why doesn't the eventPriority=1 preempt the already scheduled events with weaker priority? @achubaty may already know.

[Alex Chubaty]

so what you want is a post-init event that happens immediately after the init event for that module, but before the init events of other modules?

why not put this second event into the init, especially if you say that creating a a separate post-init event for the module could be too dangerous? if this post-init event is so closely coupled with the init, then why is it a separate event?

that being said, it looks like there might be a bug with how we are keeping the event list sorted -- the secondary key for event priority isn't working the way we would expect it to:

# current implementation in `scheduleEvent` follows this:
set.seed(10)
n <- 20
dt <- data.table(a = sample(1:100, n, replace = TRUE), b = sample(letters, n, replace = TRUE))
dt
setkey(dt, 'a') # pay attention to rows 11, 12, 13 -- note the order of the second column ('b')
dt
set2key(dt, 'b') # note that setting the secondary key isn't sorting on column 'b'
dt


# proposed fix:
set.seed(10)
n <- 20
dt <- data.table(a = sample(1:100, n, replace = TRUE), b = sample(letters, n, replace = TRUE))
dt
setkey(dt, 'a', 'b') # now rows 11, 12, 13 are sorted correctly first by 'a' then by 'b'
dt

However, this fix is not so simple to implement! Basically, an event that schedules another event for the same time but with higher priority will always place that other event at the top of the event list, so when the current event completes, the first item in the list is removed but this no longer corresponds to the current event! The result is an endless loop where event A schedules event B, but event A never gets removed from the list so is always run, scheduling more event B's.

@eliotmcintire and I need to think about how to properly fix this

[Steve Cumming]

Thanks for this, Alex.

I will explain the use case I have in mind at later time. I can work around it.

I don't know enough about the details of the scheduler, but your explanation is as if the even queue was a stack. Would it not be possible to implement anà

"insert after" version of scedule_event?

Steve

[Eliot McIntire]

@achubaty and I are discussing offline various solutions. It is not clear that an insert after would work. What happens if you have one event that has 2 insertAfter calls? Which one is really "next" and which is "after next". 

We have found the bug in the data.table sorting mechanism and will fix it, so you can use eventPriority as you initially suggested. 

However, it seems that the only way you can truly guarantee that something happens immediately after something else is if you put them in the same event. Any other mechanism, once it gets put into the queue, can be overridden by another module (placing a higher priority) or your own module (e.g., 2 insertAfter calls in the same event will cause it to break). 

One could assume that insertAfter is First In First Out, but that would undermine the entire scheduling structure of the current philosophy, which is three ways:

1. time based first, 

2. then FIFO for identically timed events, 

3. then event Priority for overriding FIFO for identically timed events

[Steve Cumming]

Your fix will work for me.

You are right about an insert after not doing the job. What I want is that the scheduled events from the init even of module A happen before in init event of module B, which depends on A.

This is just a way of exploiting the dispatcher mechanism in the module itself to gracefully fail in processing certain kinds of input that a handler has not been written for.

Steve

[Alex Chubaty]

fixed in development branch

https://github.com/PredictiveEcology/SpaDES/commit/edd0916fa3e4d964d3df19f2217630a1b8f4f640

[Eliot McIntire]

We did not implement a "insertNext" or any equivalent. We just fixed the bug that was preventing the original mechanism from working. In so doing, we added a new function, current(sim) which gives the current event being run, instead of using the "top" event in the event queue as the one that is being run.