parallel processing

[Jasmine Wu]

Is there a way to run parallel modeling in eventnet?

My application involves a large-scale healthcare communication dataset with clinician teams coordinating patient care in emergency departments. For each of the 3,000+ patients in the dataset, I would construct a separate clinician collaboration network and estimate REMs independently. From what I understand, Butt’s R package for REMs does not support interaction terms, so I have been exploring eventnet as an alternative. I was wondering whether eventnet supports efficient large-scale or parallelized model estimation for this type of workflow.

Response: 

This is not implemented explicitly in the eventnet software (although the java virtual machine will often use more than one processor for memory management, which is a time-consuming part - but this happens automatically). In general, eventnet is rather efficient. So depending on how large the collaboration network is for each of the 3000+ patients, I would say it should still be possible (we have already processed data with several hundreds of millions of events). Some further comments:

• You don't have to specify a new configuration for each of the 3000 patients. Eventnet allows you to specify directories (instead of files) in the first tab ("files") and then it processes all, say, .csv files in these directories, one after the other and independent of each other (a new file starts a new empty "network of past events"). (This assumes that each CSV file has the same columns, uses the same event types, etc.) Each of the input files is turned into a separate output file (perhaps best in a new empty output directory that you specify).
• Of course, if your computer has K processors, you may distribute the 3000 input files over K separate directories and start a new eventnet process for each of them. Then you achieve parallelization trivially.

I think some combination of this should work.