About the "Calibrated emulators"

[Jing Liu]

Hi,

I am curious about the part five calibrated emulators in the Pantheon research paper, would you please give some information about the implementation? Could I reproduce it?

The reason I post this is I encountered problems when I optimize BBR's Startup. 

I separately collected some RTT and bandwidth traces from production environment before( I changed the original BBR algorithm a bit for this collection) ,

and randomly reproduced them via Pantheon locally for testing my work. I found the performance of BBR or other CC is very strange under some traces cases! 

For instance, under the highly varing bandwidth trace, the 95th perct one way delay is large than 3500, and I think it is impossible in practice.

So I am reflecting if my approach is wrong and thinking of Pantheon's approach, and thinking about could I get the "right" traces or trace combinations from the performance perspective motivated by Pantheon.

Looking forward to response. Any insight would be greatly appraciated!

Jing Liu

[Francis Y. Yan]

Hey Jing,

Reproducing calibrated emulators is definitely possible, but I'm not sure if you want to go through the lengthy process :( Our code is based on the well-known implementation of Bayesian optimization called Spearmint, and you could find it at https://github.com/StanfordSNR/Spearmint. Note that we adapt the optimization to be distributed across multiple AWS/GCP instances in order to speed it up. To be honest, we haven't run the code for over two years and we are not actively maintaining it anymore. Our attentions these days is mostly on Puffer, and you could find our paper and slides here.

If your question is more about the weird behavior of BBR in some cases, you might want to test other combinations of Mahimahi parameters and try to understand its behaviors. Calibrated emulators, on the other hand, are designed to bridge the simulation-to-reality gap in machine learning. For instance, if you would like to train a learning-based algorithm in simulation and test it on real network paths, the calibrated emulators are closer to the real Internet and may help generalize the training performance to the real life.

Best,

Francis 

--
You received this message because you are subscribed to the Google Groups "Pantheon" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pantheon-stanf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pantheon-stanford/e1c58687-6853-437b-aec9-4ffd02eaf4bb%40googlegroups.com.

[Keith Winstein]

It also depends what you mean by "reproduce." Do you want to repeat the calibration of the emulators from the raw data (which is what Francis is talking about), or just actually run the emulators and compare them with data from the corresponding links in the Pantheon dataset?

If the latter, everything is already on the website and linked from the paper. Each performance measurement (e.g. in Figure 4) and graph has a "P" number that links to that dataset on the live website. The exact emulation commands for mahimahi are listed on the website along with the full results in the "Emulation" tab under "Find results" (https://pantheon.stanford.edu/measurements/emu/). So you can just copy-and-paste that command (sometimes you need a trace that's also available on the website) and get the same emulation environment locally.

Best regards,

Keith

To view this discussion on the web visit https://groups.google.com/d/msgid/pantheon-stanford/CAK50sgvDNHSXY1YfrP4xJ1FhEWuN2FxoMHzp4%3DkL%3D_a7s24N%2BA%40mail.gmail.com.

[Jing Liu]

Thanks, Francis and Keith!

I think I need some time to read or run the Spearmint implementation to understand how the calibrated emulator works and assess whether I should use it to solve my problem.

 

My problem is about using the “Data-Driven Network” approach, I got the real-world datasets from our production environment with many users and countries, but replaying the dataset can’t perfectly imitate real-world network environment because under some cases the performance of CC algorithms looks like impossible in practice. So, I think I need to calibrate the dataset and add more other suitable network parameters to get the final replay settings, not just for the gap of simulation-to-reality in machine learning, but also for the same gap in traditional CC’s optimization. Further, my question is about how could I calibrate the dataset and add other parameters rationally to close to CC algorithms’ performances in the wild. “Reproduce” may have two steps, the first is to repeat it as Keith said, the second is porting it to my situation for calibrating our dataset and obtaining suitable Mahimahi emulation settings. I do feel this process might be laborious, and I’m not sure if I would be allowed to work on it even though I think this would be beneficial for CC optimization.  

 

Also, I think this is a little different from purely constructing network parameter combinations to imitate real-world network environments. Containing some real-world datasets is more complex and challenging, and easily makes CC algorithms expose their problems even though they were well-designed, and helps find CC optimization directions.

 

Last but not the least, congratulations to you! I saw some posts from twitter about Puffer got the NSDI Community Award! Well deserved!

 

Thanks again for your reply!

 

Jing

Keith Winstein <kei...@cs.stanford.edu> 于2020年3月25日周三 上午6:30写道：

[Francis Y. Yan]

Thank you, Jing! Exactly -- I always believe better network emulators that can replicate the real world should deserve more attention. Please let us know how it goes and if you'd like to reproduce and maybe extend the work of calibrate emulators.That sounds exciting!

Best,
Francis

To view this discussion on the web visit https://groups.google.com/d/msgid/pantheon-stanford/CA%2BPdy1iFvEu8Wiw25Ju3eBKAhjRzgC7SLgvSbf2Bbf8Viyb7TA%40mail.gmail.com.