Re: [babushka] vagrant

[Torsten Curdt]

Thanks for sharing, Ben!

I've run into a couple of issues though. When Vagrant is running Babushka the output is a bit broken for me. (The "spinner" creates many newlines) that's just cosmetics though.

When Vagrant is running my dep I don't come past the setting of the locale:

    tcurdt:vserver {

      tcurdt:hostname {

        meet {

        }

      } ✓ tcurdt:hostname

      tcurdt:base {

        tcurdt:set locale {

          tcurdt:exists locale {

          } ✓ tcurdt:exists locale

          meet {

            Setting the locale doesn't take effect until you log out and back in.

          }

        } ✗ tcurdt:set locale

      } ✗ tcurdt:base

    } ✗ tcurdt:vserver

    You can view a more detailed log at '/root/.babushka/logs/tcurdt:vserver'.

    The following SSH command responded with a non-zero exit status.

    Vagrant assumes that this means the command failed!

    babushka --update --defaults --colour  'tcurdt:vserver'

Looking at the locale dep

    met? {

      shell('locale').val_for('LANG')[locale_regex(locale_name)]

    }

    on :apt do

      meet {

        sudo("echo 'LANG=#{local_locale(locale_name)}' > /etc/default/locale")

      }

      after {

        log "Setting the locale doesn't take effect until you log out and back in."

      }

it sound like changing to login_shell could fix this.

When I "sudo -i" to root it runs just fine.

Another issue I've run into is that the ssh agent does not forward the authentication on the first run. Not sure why that is the case yet. Only after a second "sudo -i" I can clone private repos.

So much for feedback and progress :)

cheers,

Torsten

[Ben Hoskings]

On 11/12/2012, at 5:28 AM, Torsten Curdt <tcu...@vafer.org> wrote:

> When Vagrant is running my dep I don't come past the setting of the locale:

Yep, this is expected.

> it sound like changing to login_shell could fix this.
> When I "sudo -i" to root it runs just fine.

As far as I know there's no way other than to refresh the session. You can certainly run the check by sudoing, which will confirm the locale is set properly, but the setting won't take effect for the current session.

That means that if the locale dep runs as a dependency of (e.g.) postgres, then postgres will still install on the old locale -- so it will create its template databases in the old locale, and so on.

Since the actual session babushka is running in needs to have the correct locale, I set it in a separate session. A babushka-only example:

https://github.com/conversation/babushka-deps/blob/master/provision.rb#L123-L125

- Ben

[Torsten Curdt]

> When Vagrant is running my dep I don't come past the setting of the locale:

Yep, this is expected.

Meh. While this makes sense - it also sucks.

How would solve this to setup a system? Especially through Vagrant.

 

> it sound like changing to login_shell could fix this.
> When I "sudo -i" to root it runs just fine.

As far as I know there's no way other than to refresh the session. You can certainly run the check by sudoing, which will confirm the locale is set properly, but the setting won't take effect for the current session.

A bit of a surprise but it seems like not even a sudo gets the new locale :-/ Seems you have to really exit the session.

 

Since the actual session babushka is running in needs to have the correct locale, I set it in a separate session. A babushka-only example:

https://github.com/conversation/babushka-deps/blob/master/provision.rb#L123-L125

Maybe there is a need to solve problem in a general manner?

cheers,

Torsten 

 

[Ben Hoskings]

On 11/12/2012, at 10:41 PM, Torsten Curdt <tcu...@vafer.org> wrote:

> > When Vagrant is running my dep I don't come past the setting of the locale:
>
> Yep, this is expected.
>
> Meh. While this makes sense - it also sucks.
> How would solve this to setup a system? Especially through Vagrant.

It's inherent to the way locales are set. Since they're not set on the shell, but on the surrounding session, this is just a reality -- it's how they work, and so if you want to be real-world about it, it's also how you have to manually check if the locale is correct (and hence how met?{} should do it).

The vagrant plugin could easily be extended with a #failable_dep call. I'm not sure if that's the best place for it, though.

- Ben

[Torsten Curdt]

> Meh. While this makes sense - it also sucks.
> How would solve this to setup a system? Especially through Vagrant.

It's inherent to the way locales are set. Since they're not set on the shell, but on the surrounding session, this is just a reality -- it's how they work, and so if you want to be real-world about it, it's also how you have to manually check if the locale is correct (and hence how met?{} should do it).

Indeed - that's what I have done to work around this.

 

The vagrant plugin could easily be extended with a #failable_dep call. I'm not sure if that's the best place for it, though.

I am wondering if we should expose information on whether a dep needs a new session and then have babushka handle the creation of a new session if needed.

Does that make sense?

Another thing I came across. Through Vagrant stdin is not connected and so not interactive reads from the CLI are not possible.

Would be really cool if there was an easy way to fix this. But I fear that means digging through Vagrant. Question is whether they would like the changes. If that's not in the cards at least we should make it easy to pass variables to deps from the Vagrant file. (Even though I am not a fan of the idea)

cheers,

Torsten

[Ben Hoskings]

On 11/12/2012, at 11:41 PM, Torsten Curdt <tcu...@vafer.org> wrote:

> The vagrant plugin could easily be extended with a #failable_dep call. I'm not sure if that's the best place for it, though.
>
> I am wondering if we should expose information on whether a dep needs a new session and then have babushka handle the creation of a new session if needed.

The issue with this is that then you have deps that only run properly in a specific state, that isn't the default. I'd rather avoid dependencies of state.

My principle is to do things how I'd do them manually, which is just to log out of the box and back in. That's how you check the locale is _really_ set properly, and so that's what I want babushka to do. I think that's how you get real reliability.

> Another thing I came across. Through Vagrant stdin is not connected and so not interactive reads from the CLI are not possible.
> Would be really cool if there was an easy way to fix this. But I fear that means digging through Vagrant. Question is whether they would like the changes. If that's not in the cards at least we should make it easy to pass variables to deps from the Vagrant file. (Even though I am not a fan of the idea)

Interactive reads probably won't be easily passable, but you can already pass parameters from the Vagrantfile. Is that what you mean?

(Old-style vars are deprecated and about to be removed.)

- Ben

[Torsten Curdt]

> I am wondering if we should expose information on whether a dep needs a new session and then have babushka handle the creation of a new session if needed.

The issue with this is that then you have deps that only run properly in a specific state, that isn't the default. I'd rather avoid dependencies of state.

Not sure we are on the same page here. Could you elaborate?

Right now I would have to split provisioning into sessions manually. Think of it as provisioning phases:

  babuska tcurdt:vserver-phase1

  babuska tcurdt:vserver-phase2

Also keeping track of the interdependencies manually - which sucks balls. Now if a dependency was marked as "needs new session" babushka could analyse the tree and create new sessions whenever necessary. Does that make sense?

It's essentially what I am doing manually for a Vagrant deployment.

How does that introduce state?

 

My principle is to do things how I'd do them manually, which is just to log out of the box and back in.

Yuck - that's not really an option for me. Especially not with Vagrant. "Just keep running until there is no error anymore". Meh.

 

That's how you check the locale is _really_ set properly, and so that's what I want babushka to do. I think that's how you get real reliability.

Well, you would still have the same thing. You would be re-running

  babuska tcurdt:vserver-phase1

  babuska tcurdt:vserver-phase2

all the time. It's just that babushka would be a bit smarter on when to create new sessions and so an error re-run comes distinguishable from a new run caused by the need for a new session.

 

> Another thing I came across. Through Vagrant stdin is not connected and so not interactive reads from the CLI are not possible.
> Would be really cool if there was an easy way to fix this. But I fear that means digging through Vagrant. Question is whether they would like the changes. If that's not in the cards at least we should make it easy to pass variables to deps from the Vagrant file. (Even though I am not a fan of the idea)

Interactive reads probably won't be easily passable,

Will, dig a bit into Vagrant - would be really nice to have.

 

but you can already pass parameters from the Vagrantfile. Is that what you mean?

Yeah, I saw it in the code but how do pass the variables to specific dependency? Can you give an example?

 

(Old-style vars are deprecated and about to be removed.)

What's the old style?

 

cheers,

Torsten

[Ben Hoskings]

On 12/12/2012, at 9:09 PM, Torsten Curdt <tcu...@vafer.org> wrote:

>>> I am wondering if we should expose information on whether a dep needs a new session and then have babushka handle the creation of a new session if needed.
>>
>> The issue with this is that then you have deps that only run properly in a specific state, that isn't the default. I'd rather avoid dependencies of state.
>
> Not sure we are on the same page here. Could you elaborate?
>
> Right now I would have to split provisioning into sessions manually. Think of it as provisioning phases:
>
> babuska tcurdt:vserver-phase1
> babuska tcurdt:vserver-phase2

That's exactly what I do. For example, when provisioning our CI agents (which are vagrant VMs), I have two deps:

'ci prepared' # Sets the locale and installs the correct ruby
'ci provisioned' # Does everything else

The locale issue is only one of a more general case: sometimes the running babushka would run differently because of a change that a dep made. Whether that's a new locale, or a new ruby that babushka will run against next time, etc.

> Also keeping track of the interdependencies manually

This isn't an issue, because 'ci provisioned' requires 'ci prepared'. By the time you get to running 'provisioned', 'prepared' should already be met -- but there's no need to manually track dependencies. The requirements are still just the actual requirements of each dep; there's just an initial subset that run separately.

> Now if a dependency was marked as "needs new session" babushka could analyse the tree and create new sessions whenever necessary. Does that make sense?
>
> It's essentially what I am doing manually for a Vagrant deployment.
>
> How does that introduce state?

The state is that session that babushka would be creating. It's not the real session, only a simulation.

For example, say you run "postgres.bin", which calls "set.locale"; the locale is set, and postgres is installed within the simulated environment, so the locale is correct. Babushka says "all done" and drops you back to a shell -- but that shell still has the old locale, and so creating a database using it wouldn't do what you expect.

The state involved in constructing that temporary session is the problem. This stuff can only be reliable when it's running against an actual, current session, otherwise it's hypothetical and will cause surprises.

>> My principle is to do things how I'd do them manually, which is just to log out of the box and back in.
>
> Yuck - that's not really an option for me. Especially not with Vagrant. "Just keep running until there is no error anymore". Meh.

As I explained above, I don't think that's what this is :) It's not about retrying after a failure at all.

It's simply being honest about the fact that after you change locale, rubies, or whatever, the current session is no longer valid: it no longer represents the system, and so should be replaced with a new one before continuing.

> Yeah, I saw it in the code but how do pass the variables to specific dependency? Can you give an example?

You can only pass an argument to the top-level dep -- that's by design, because I intended dep parameters to feel as much like method parameters as possible.

If you have to pass a value to a specific dep, one option is to run that dep separately in the Vagrantfile first. Then when the dep that requires it is run, it will already be met.

>> (Old-style vars are deprecated and about to be removed.)
>
> What's the old style?

Dep parameters are a replacement for vars -- whose removal date has now passed, woohoo I can delete so much code now! :)

They were inline-defined, e.g.

met? { shell('whoami') == var(:user) }

The main problem with vars was that they were global across all deps, and hence were a whole bunch of set state. It was really hard to tell why a certain var had the value that it did, i.e. which dep set it, and when. It also led to a lot of cases where changing a dep would break something in another dep.

Dep parameters have totally fixed that; I'm really pleased with how they've turned out.


Cheers
Ben

[Torsten Curdt]

 

> Also keeping track of the interdependencies manually

This isn't an issue, because 'ci provisioned' requires 'ci prepared'. By the time you get to running 'provisioned', 'prepared' should already be met -- but there's no need to manually track dependencies. The requirements are still just the actual requirements of each dep; there's just an initial subset that run separately.

IMO it's a bit more messy than it sounds when you describe it. Transitive dependencies could pull things into the first phase - unintentionally. So you do need to manually make sure you keep the sessions separate.

As an extreme you could create a new session for each dependency to solve this. Of course that's not what we want.

 

> Now if a dependency was marked as "needs new session" babushka could analyse the tree and create new sessions whenever necessary. Does that make sense?
>
> It's essentially what I am doing manually for a Vagrant deployment.
>
> How does that introduce state?

The state is that session that babushka would be creating. It's not the real session, only a simulation.

For example, say you run "postgres.bin", which calls "set.locale"; the locale is set, and postgres is installed within the simulated environment, so the locale is correct. Babushka says "all done" and drops you back to a shell -- but that shell still has the old locale, and so creating a database using it wouldn't do what you expect.

The state involved in constructing that temporary session is the problem. This stuff can only be reliable when it's running against an actual, current session, otherwise it's hypothetical and will cause surprises.

I think we are not on the same page yet.

At the moment Babushka only has a single session - so of course updating the locale is a problem in your postgres example. For Vagrant we solve this by creating two separate sessions - do separate runs if you like. They are completely independent and so there is really is no state other than the modifications on the actual system. I think that's the beauty of Babushka :)

Now - what if Babushka knew how to create those different sessions?

By marking nodes in the dependency tree we then could analyse the tree and actually could come up with these separate sessions automatically.

Does that make sense now?

cheers,

Torsten

[Ben Hoskings]

(Sorry for the delay; I've been on holidays and moving house.)

On 27 December 2012 10:54, Torsten Curdt <tcu...@vafer.org> wrote:

I think we are not on the same page yet.

At the moment Babushka only has a single session - so of course updating the locale is a problem in your postgres example. For Vagrant we solve this by creating two separate sessions - do separate runs if you like. They are completely independent and so there is really is no state other than the modifications on the actual system. I think that's the beauty of Babushka :)

We're in agreement on this part :)

Now - what if Babushka knew how to create those different sessions?

Then it could do this test more easily, yes. But then when babushka exits, saying "✔ job done", the job actually isn't done for the current session. There are many cases where this will result in surprising behaviour and would ultimately mean that I didn't trust babushka's "✔" anymore.

I want to avoid situations where using babushka introduces unknowns over just doing the work manually. If you have to sidestep the tool to work out why it's not doing what you thought, then the tool has a design flaw.

--

If the deps are being run on a remote box, this whole thing is a non-issue, because the process of logging in a second time can be trivially automated. This is how I provision servers, for example: https://github.com/conversation/babushka-deps/blob/master/provision.rb#L123-L131

I agree that it can be problematic for local runs, but I don't think that this is a problem that babushka should attempt to solve. It's just a reality of how sessions work.

- Ben

[Torsten Curdt]

 

Now - what if Babushka knew how to create those different sessions?

Then it could do this test more easily, yes. But then when babushka exits, saying "✔ job done", the job actually isn't done for the current session. There are many cases where this will result in surprising behaviour and would ultimately mean that I didn't trust babushka's "✔" anymore.

Not sure I can fully follow your arguments here...

"✔ job done" can (for the cases we are having in mind here) never be reached in the current session. I think that we have established as a fact. The question is now how to deal with this better.

Manually establishing new sessions and running babushka until it is passes with "✔ job done" sounds neither efficient nor elegant or well designed to me. I think we can do better :)

I would not want to change the behaviour of babushka as it is today ..but I would like to aid something that sits a level above and can take care of efficient session creations. But this tool/layer needs access to the dependency graph - and the graph nodes would need an attribute that defines the session requirements.

A compromise could be to to print out which dependencies need to be run in separate sessions.

This information could be used by a "babushka driver" or (whatever we want to call it). 

When this thing is done with creating sessions and running babushka the "✔ job done" is well deserved. Just not for the current session - so afterwards you need to close the session. But that's and easy to understand contract that could maybe even be enforced.

I want to avoid situations where using babushka introduces unknowns over just doing the work manually. If you have to sidestep the tool to work out why it's not doing what you thought, then the tool has a design flaw.

While it's not babushka's fault - I would argue that's how it is today ;)

It does not deal well with a common problem.

cheers,

Torsten

PS: how is the new house? :)

[Ben Hoskings]

On 28 January 2013 01:03, Torsten Curdt <tcu...@vafer.org> wrote:

"✔ job done" can (for the cases we are having in mind here) never be reached in the current session.

Right. The new state can't be applied, and babushka is reflecting that.

 

Manually establishing new sessions and running babushka until it is passes with "✔ job done" sounds neither efficient nor elegant or well designed to me. I think we can do better :)

It's not a matter of design; it's a matter of reflecting reality. (And there's no reason it has to be a manual process, as the deps I linked above show.)

There are two cases:

(a) Running locally, e.g. to configure something on your laptop

(b) Running remotely, e.g. to provision a server

The (b) case is already solved with something like the deps I linked above, and as far as I can see, the (a) case can't be solved by any tool.

Say we build test-in-a-new-session support into babushka, what do you do in the (a) case when it says "✔ job done"? That is, when babushka says "locale set", or "$PATH updated", or "default shell changed", what do you do at your terminal to actually have that take effect, short of manually restarting the terminal?

- Ben

P.S. The house is great, thanks. :)

[Torsten Curdt]

 

"✔ job done" can (for the cases we are having in mind here) never be reached in the current session.

Right. The new state can't be applied, and babushka is reflecting that.

yupp

 

 

Manually establishing new sessions and running babushka until it is passes with "✔ job done" sounds neither efficient nor elegant or well designed to me. I think we can do better :)

It's not a matter of design; it's a matter of reflecting reality. (And there's no reason it has to be a manual process, as the deps I linked above show.)

you mean https://github.com/conversation/babushka-deps/blob/master/provision.rb#L123-L131 ?

so "remote_babushka" runs another session?

 

There are two cases:

(a) Running locally, e.g. to configure something on your laptop

you only have the one session

 

(b) Running remotely, e.g. to provision a server

The (b) case is already solved with something like the deps I linked above, and as far as I can see,

this creates multiple session via "remote_babushka" IIUC

 

the (a) case can't be solved by any tool.

I envision to align these two cases. Whether remote or local does not really matter then.

Let's say you let babushka run on your laptop machine. The dependency graph shows that setting the locale is part of the babushka run. Which means the run has to be split into more than one session. Babushka would then run the locale session and open a new one to run the rest of the dependencies. The new session will of course have the locale be set correctly. The initial session will of course not reflect that. But that's fine IMO. You would only have close the initial session. Babushka could report this and all is good.

The only session that can be a problem is the initial session. And that's just a matter of communication and contracts.

cheers,

Torsten

[Ben Hoskings]

Right, I see where you're coming from.

My instinct says the design will be cleaner and more composable if babushka doesn't know about sessions, because that keeps the ceiling much clearer. (For example, I'm starting to think that perhaps babushka shouldn't know how to sudo either.)

But maybe there's a way to achieve this cleanly. Does anyone else have any thoughts?

Cheers

Ben

--
--
To post, email babush...@googlegroups.com
To unsubscribe, email babushka_app...@googlegroups.com
~
http://babushka.me
http://github.com/benhoskings/babushka
http://groups.google.com/group/babushka_app
 
---
You received this message because you are subscribed to the Google Groups "babushka_app" group.
To unsubscribe from this group and stop receiving emails from it, send an email to babushka_app...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Torsten Curdt]

> Right, I see where you're coming from.

Yay! :)

> My instinct says the design will be cleaner and more composable if babushka
> doesn't know about sessions, because that keeps the ceiling much clearer.

That could in fact well be true. On the other hand then there probably
shouldn't be remote_sessions either, right?

> (For example, I'm starting to think that perhaps babushka shouldn't know how
> to sudo either.)

Indeed - the sudo thing is a little awkward, too.
Right now I write my dependencies with the assumption that they have
enough permissions.

> But maybe there's a way to achieve this cleanly. Does anyone else have any
> thoughts?

Maybe we just need to expose the dependency tree and then we can have
something else that sits on top.

Would also love to hear more input on this.

cheers,
Torsten

[Ben Hoskings]

I think a good solution would involve nested babushka runs, with a clean separation: a running babushka is either processing deps, or orchestrating other babushkas, but not both.

Whether it's babushka or something else handling this process, I think the important thing is that clear separation.

- Ben

cheers,
Torsten

--
--
To post, email babush...@googlegroups.com
To unsubscribe, email babushka_app...@googlegroups.com
~
http://babushka.me
http://github.com/benhoskings/babushka
http://groups.google.com/group/babushka_app

---
You received this message because you are subscribed to the Google Groups "babushka_app" group.
To unsubscribe from this group and stop receiving emails from it, send an email to babushka_app...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Cheers

Ben

[Torsten Curdt]

> I think a good solution would involve nested babushka runs, with a clean
> separation: a running babushka is either processing deps, or orchestrating
> other babushkas, but not both.
>
> Whether it's babushka or something else handling this process, I think the
> important thing is that clear separation.

agreed