Any experience how vagrant + virtual box + jenkins perform in the cloud?

[Thomas Baumann]

Recently, I tried virtualising a jenkins instance in my company, meaning jenkins should go into some KVM domain. Well, jenkins itself ran just fine under these conditions, but virtual box did not perform very well. We made sure all nested virtualisation options were set correctly. The jenkins guest is able to use processor-flags vmx, ept and vpid. /sys/module/kvm_intel/parameters/nested states nested paging is enabled with an 'Y'.

I asked at the virtual box forums for experiences running virtual box inside a KVM domain, but they could not really give me a hint.

Anyway, I meanwhile accept answers like "nested virtualisation does not perform very well". We just gave it a try. Unless anybody here tells me he has managed to run VB inside KVM with good performance results, that is fine, we accept that.

The only thing I am still wondering about is, how do people run a setup of vagrant + virtual box + jenkins in the cloud? Cloud servers are always virtualised, so we have nested virtualisation there, too.

Can anybody confirm vagrant + virtual box + jenkins is running fine in the cloud? Let's say with an Amazon EC2 instance? The guys from the virtual box forum sent me here, as they actually do not use virtual box in CI environments. So they cannot say.

Any experiences?

Thanks!

[Alvaro Miranda Aguilera]

hello

how do people run a setup of vagrant + virtual box + jenkins in the cloud? 

they don't

what do you want to achieve that vagrant + virtual box + jenkins in the cloud is the answer ?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/vagrant/issues
IRC: #vagrant on Freenode
---
You received this message because you are subscribed to the Google Groups "Vagrant" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vagrant-up+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vagrant-up/cd0e6653-6adf-4ba1-ad75-3588c121951e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alvaro

[Antony Stone]

On Thursday 08 February 2018 at 18:21:43, Alvaro Miranda Aguilera wrote:

> hello
>
> how do people run a setup of vagrant + virtual box + jenkins in the cloud?
>
> they don't

Is that because it doesn't work, or because it's just not a useful idea?

> what do you want to achieve that vagrant + virtual box + jenkins in the
> cloud is the answer ?

I got the impression that the original question was simply "how to run Jenkins
in Virtualbox on a KVM guest machine". The "cloud" aspect was simply
introduced as an example where people would have to be running Virtualbox
inside an already-virtualised server. I don't think Thomas intends to
actually run things that way, he was just giving context to his question.


Antony.

> On Thu, Feb 8, 2018 at 12:15 PM, Thomas Baumann wrote:
> > Recently, I tried virtualising a jenkins instance in my company, meaning
> > jenkins should go into some KVM domain. Well, jenkins itself ran just
> > fine under these conditions, but virtual box did not perform very well.
> > We made sure all nested virtualisation options were set correctly. The
> > jenkins guest is able to use processor-flags vmx, ept and
> > vpid. /sys/module/kvm_intel/parameters/nested states nested paging is
> > enabled with an 'Y'.
> >
> > I asked at the virtual box forums for experiences running virtual box
> > inside a KVM domain, but they could not really give me a hint.
> >
> > Anyway, I meanwhile accept answers like "nested virtualisation does not
> > perform very well". We just gave it a try. Unless anybody here tells me
> > he has managed to run VB inside KVM with good performance results, that
> > is fine, we accept that.
> >
> > The only thing I am still wondering about is, how do people run a setup
> > of vagrant + virtual box + jenkins in the cloud? Cloud servers are
> > always virtualised, so we have nested virtualisation there, too.
> >
> > Can anybody confirm vagrant + virtual box + jenkins is running fine in
> > the cloud? Let's say with an Amazon EC2 instance? The guys from the
> > virtual box forum sent me here, as they actually do not use virtual box
> > in CI environments. So they cannot say.
> >
> > Any experiences?
> >
> > Thanks!

--

I conclude that there are two ways of constructing a software design: One way
is to make it so simple that there are _obviously_ no deficiencies, and the
other way is to make it so complicated that there are no _obvious_
deficiencies.

- C A R Hoare

Please reply to the list;
please *don't* CC me.

[Thomas Baumann]

No no it is different. Jenkins runs inside a KVM guest. Jenkins uses vagrant + virtual box to build the testing environment in which the actual application tests run. The latter is common practice.

In deed this setup does not perform very well. Virtual box instances can be set up, they do run with multiple cores, but performance is weak.

Now let's assume virtual box does really not like to be run inside a guest, and given that using vagrant + virtual box to build the testing environments is common practice with Jenkins, I wonder how people run such a setup in cloud environments, then. They will not have the KVM part, but as soon as they use EC2 instances, they are actually running Jenkins inside a virtualized environment. Don't they? AFAIK amazon uses Xen, which is a type 1 hypervisor. KVM on the other hand is considered to be a type 2 hypervisor,

So why is it my setup does not play along nicely? Is it the specific KVM and virtual box configurations, that might be wrong or does virtual box just not like to be virtualized? Does it just not like type 2 hypervisors or no hypervisors in general? If people really do not do it the vagrant+virtual box way in the cloud, what other strategy do they follow to build their virtualized testing environments?

I think the configurations are correct. An answer to the other questions would help me to get a bigger picture on the issue and narrow down the problem.

[Alvaro Miranda Aguilera]

hello

there are same technical aspects chosen before the scope of the requirement, like you want to use virtualbox nested.

If that is the case, then you should look for:

- bare metal

- ideally per minute or per hour

check terraform + packet.net if you want to build and use later

now, if you explain more, i can help you more, but the technical requirements (how) should came after the business requirements (what/why)

if you will be using this as part of your pipeline, say ci/cd then i will say looks more prod than dev, and i would ask " is vagrant the best box for this ? "

then we can move into there are tools that build around this already and its easy (vs simple) to implement, and you may have this in house and runs fine, so you want to make it scale and move it to the cloud

valid points but guessing whats the backstory is not good, better have the whole story first, so ppl can actually share how/what they do.

vagrant defaults to virtualbox, but with plugins you can use aws or digital ocean or others, not sure will be easy to move from what you got to that

happy to comment and help, but need more context please.

Thanks

Alvaro

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/vagrant/issues
IRC: #vagrant on Freenode
---
You received this message because you are subscribed to the Google Groups "Vagrant" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vagrant-up+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vagrant-up/4f8083bd-ca46-46eb-afdb-693285222c33%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Alvaro

[Thomas Baumann]

The bare metal scenario:

----------------------------------

now, if you explain more, i can help you more, but the technical requirements (how) should came after the business requirements (what/why)

Forget about the outer virtualisation for a moment. In a bare metal world, vagrant+vb are used to build testing environments. It is used to run unit and functional tests.

Let's take a simple web project as an example:

- the project has ansible playbooks tied to it

- the playbooks define a web and a database server

- to be able to run the tests inside the specified environment, jenkins needs to build the servers according to current specifications

- all this is done to ensure the code runs in the currently defined systems (e.g. there may be php upgrades in the feature branch or any other nasty things)

As far as I know, it is common practice to use vagrant for that scenario. Vagrant is also used for development. So the pretty much same environment is used for development, for tests and later also for deployment. Production and staging servers are not deployed using vagrant and they do not run inside virtual box, but the ansible-playbooks provision them the same way, as the virtual boxes in development or during tests in Jenkins are.

Does that make sense so far?

 

if you will be using this as part of your pipeline, say ci/cd then i will say looks more prod than dev, and i would ask " is vagrant the best box for this ? "

I think vagrant is the best solution for that. This is why there are vagrant plugins for jenkins, I guess.

Now we come to nested virtualisation:

---------------------------------------------------

Our operations department would like us to run Jenkins inside a KVM domain, because they want to avoid to maintain a bare metal box. This is the business requirement and this is also why I tried to set up things inside KVM. I found everything is working inside the KVM domain, but virtual box performs weak. Although the performance of the web project would be good enough for development, it is too weak for unit tests. The tests should run as fast as possible.

After some investigation, people tend to say virtual box does not like to be nested. Well, according to my findings, this may be true.

This is what brings us to the cloud:

----------------------------------------------

If the bare metal scenario with vagrant+vb+jenkins is common practice, I wonder how people do this in a cloud environment. I do not want to move Jenkins to the cloud, nor do I care why other people do it. But I am sure there are people out there who run jenkins in the cloud. This is the only thing, that counts at the moment.

And if they do it in the cloud, they would run VB inside a VM, as any cloud server is virtualised, isn't it? Is anybody able to follow my arguments?

So one of the following is true:

1) People do not use VB to build their testing environments, neither bare metal nor in the cloud (what else do they do to make sure the test system is well defined?)

2) People use VB to build the testing environments and so they use it in the cloud.

If 1) applies, I am eagerly interested in what they use instead of VB? When doing it in the cloud, do they set up new EC2 instances for any test run?!?!

If 2) applies, it is an indicator that our local setup with KVM is wrong. This does not yet prove it is possible with KVM, but at least we know that virtual box is able to be run as a nested hypervisor somehow.

I need some people to tell me about their experience running Jenkins+vagrant+vb in the cloud. People having this setup run in KVM are also welcome, but it is probably easier to find someone doing this in the cloud. Experiences of somebody having done (or having tried) this in in the cloud, would give me valuable hints.

Where is the failure in the line?

PS: Docker is not an option, as we are not able to deploy containers to our production servers.

 

Thanks

Alvaro

Thank you, too!

 

To unsubscribe from this group and stop receiving emails from it, send an email to vagrant-up+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vagrant-up/4f8083bd-ca46-46eb-afdb-693285222c33%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alvaro