Accuracy of CloudSim Plus

[S M]

Hello together, 

after reading through the publications, docs and architecture I still can't get my head around how CloudSimPlus achieves accurracy. It is not mentioned in the original cloudsim paper from 2010, but in the cloudsimplus publication from 2017 it states that the implementation of the linux completely fair scheduler and the use of integrationtests are the main reasons for higher accuracy nowadays, at least the conclusion is drawn.

Maybe I'm missing something, but I'd like to know: Have there been studies or tests comparing cloudsimplus simulations scenarios to realworld cloudinfrastructures?

Has someone proven that the simulation metrics are in a certain tolerance to the realworld infrastructure? Or would I need to tweak them as a researcher till I'm there?

Hope my question is not too inaccurate ;)

Thanks you very much in advance for your time, I appreciate it 

Greetings from Germany

Silas

[Prof. Manoel Campos]

Hello Silas,

Sorry for the really late reply. Some messages may be going straight to my spam folder. 

after reading through the publications, docs and architecture I still can't get my head around how CloudSimPlus achieves accurracy.

That was a short paper with a very limited number of pages. So, we couldn't elaborate on that.

 

It is not mentioned in the original cloudsim paper from 2010, but in the cloudsimplus publication from 2017 it states that the implementation of the linux completely fair scheduler and the use of integrationtests are the main reasons for higher accuracy nowadays, at least the conclusion is drawn.

 

The Linux CFS is just one example of higher accuracy in CloudSim Plus. The CloudletSchedulerTimeShared is not realistic at all. It's an oversimplified implementation of a time-shared scheduler. With such an implementation, Cloudlets are never preempted. This way, there is no wait time at all. That implementation considers that all Cloudlets are always executed in parallel. If there aren't enough CPUs for all of them, it just splits the CPU MIPS capacity between all existing Cloudlets. In practice, if we have 2 CPUs with hyper-treading (HT) technology and 4 applications, we can still execute them in parallel since HT gives us 2 additional virtual CPUs. But if we have one more app, that will have to wait for the CPU.

If you want a more realistic scheduler, the CSF in CloudSim Plus provides that. More details here.

The use of integration tests is an additional way to ensure CloudSim Plus is much more accurate than CloudSim. But the main reason is the large number of bugs that were fixed in CloudSim Plus. Just check the GitHub repo to confirm that.

The code duplication in CloudSim is the source of many issues and makes it really difficult to extend without needing to change the framework code. The amount of duplicated code in CloudSim is just scary. If you have a bug, that same bug probably exists elsewhere, mainly after the container support was implemented by duplicating entire packages of classes.

I've been improving the framework accuracy by including additional features and changing some simulation behaviors (such as CSF, destruction of Vms after they finish, accurate power consumption computation, etc). Check the list of the main features at https://cloudsimplus.org and the issue at GitHub. The number of fixed issues is huge, which provides increased accuracy.

 

Maybe I'm missing something, but I'd like to know: Have there been studies or tests comparing cloudsimplus simulations scenarios to realworld cloudinfrastructures?

No. And there is no plan for doing that. We are talking about better accuracy compared to CloudSim and how given features should work in the real world (such as presented in the links provided). For instance, if all hosts inside a given datacenter are overloaded and you have available hosts in a different datacenter, you should be able to migrate VMs across datacenters. However, that is not possible in CloudSim and was implemented in CloudSim Plus (such a feature is one of many implemented after the paper was published). 

 

Has someone proven that the simulation metrics are in a certain tolerance to the realworld infrastructure? Or would I need to tweak them as a researcher till I'm there?

If you are talking about simulation results, I haven't seen any paper that compared simulation results with a testbed running in a real cloud infrastructure. I remember there is a PhD thesis from a CloudSim's former member that executed some real testbed, but I don't know if the results between such experiments were compared. Anyway, CloudSim itself has a lot of bugs that may put in check the scientific validity of some experiment results presented in many papers, mainly those ones performed without using multiple simulation runs and computing confidence intervals for presented metrics.

 

If you have any more questions, feel free to keep in touch.

[S M]

Thanks very much Manoel, 

that provides a great feedback, I'll use the information to formulate a working hypothesis and maybe reach out again. For now this is a good starting point, as I plan to provide a concept for migration recommendations using CloudSim as a core part. There the use of multiple simulation runs (As I saw around 30 at least in some papers) and metric tolerances/confidence intervalls seems to be the right choice.

Maybe I'll be able to compare in a small testbed as well. I'll try reaching out to Anton Beloglazov for some information regarding his setup and experiences.

Greetings

Silas