Discussion about private_network issue (continued from gh-2881)

[Alan Pinstein]

Alvaro asked me to move this discussion to the ML: https://github.com/mitchellh/vagrant/issues/2881#issuecomment-34147376

In summary I see a few issues:

• As a user I don't expect placing network config to be handled differently when placed in a global config block vs a provider config block, esp since vagrant seems to be heading in a direction where one would use it to configure the same box across different providers (dev on VBox, prod on AWS for instance), being able to have per-provider network config work gracefully would seem to be the goal.
• The way vagrant handles "re-configuring" the network is fragile; it stops the network, re-generates some temp files, ssh's them over, then re-starts the network. You can easily see how this would break if ssh only listens on the port that is being taken down, as the subsequent config/ifup commands will not succeed. It would seem that the config files should be pre-generated, copied over in a single ssh request, and ifdown/ifup in a single request to eliminate the network suicide problem currently exhibited.

I think in the end those are the major issues that I feel exist after working through this particular issue.

Thoughts?

Alan

[Teemu Matilainen]

Hi Alan,

On Thu, Feb 6, 2014 at 1:19 AM, Alan Pinstein <apin...@mac.com> wrote:

• 
  As a user I don't expect placing network config to be handled differently when placed in a global config block vs a provider config block, esp since vagrant seems to be heading in a direction where one would use it to configure the same box across different providers (dev on VBox, prod on AWS for instance), being able to have per-provider network config work gracefully would seem to be the goal.

This is totally supported. But you have to use the `override` variable in the `provider` block. See "Overriding Configuration" in the docs: http://docs.vagrantup.com/v2/providers/configuration.html

-- 

Cheers,

  - Teemu

[Alvaro Miranda Aguilera]

Ok, so the issue of 3 networks can be resolved by:

  config.vm.provider :virtualbox do |vb, override|                                                                                                                                            
    override.vm.box     = 'Tourbuzz_Basebox_centos_65'                                                                                                                                        
    override.vm.box_url = 'https://s3.amazonaws.com/util.cloud.tourbuzz.net/base_boxes/opscode_centos-6.5_chef-11.8.2.box'                                                                    
                                                                                                                                                                                              
    # private network gives a static IP; ideal for juggling multiple machines                                                                                                                 
    override.vm.network :private_network, ip: $vagrant[:network]                                                                                                                              
  end                                                                                                                                                                                         

  config.vm.provider :virtualbox do |vb|                                                                                                                                                      
                                                                                                                                                                                              
    vb.gui  = $vagrant[:gui]                                                                                                                                                                  
    vb.name = $vagrant[:vm_name] if $vagrant[:vm_name]                                                                                                                                        
                                                                                                                                                                                              
    vb.customize ["modifyvm", :id, "--memory",              $vagrant[:memory_size]  ]                                                                                                         
    # enable multi-cpu shit                                                                                                                                                                   
    vb.customize ['modifyvm', :id, "--cpus",                $vagrant[:cpus]         ]                                                                                                         
    vb.customize ["modifyvm", :id, "--ioapic",              $vagrant[:ioapic]       ]  # this is buggy, crashes on some machines. if you have issues turn it off in Vagrantfile.local.rb      
                                                                                                                                                                                              
    # makes DNS work correctly (fast)                                                                                                                                                         
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"                    ]                                                                                                         
  end                                                                                                                                                                                         



Tested, and works cool.

Thanks to Teemu. 

--
You received this message because you are subscribed to the Google Groups "Vagrant" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vagrant-up+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Alvaro Miranda Aguilera]

For the ssh port issue, I have to change your Vagrantfile a bit.

As you are concerned of the security reasons, I did add a rule in iptables that will allow only connections from interface eth0, that is nat, so nothing bad will happen

I add port 22 and 1855 to sshd, but 22 is protected by the iptables firewall.

A. On the first run, the normal portforwarded for ssh is [default] -- 22 => 2222 (adapter 1), so the normal behaviour will fail on the 2nd run, as ssh won't be on port 22 anymore

So, we need to fix that for the 2nd and next runs

*AND*

on each run, on top, remember this happen:

[default] Clearing any previously set forwarded ports...                                                                                                                                      
[default] Clearing any previously set network interfaces...                                                                                                                                   

B. so, you won't be able to connect over the network that is not yet created, that's why the ssh on the 2nd run fail.

Please try this on your Vagrantfile

  if !::File.exists? ".vagrant/machines/default/virtualbox/id" # first-time bootstrapping -- this is the best approximation of once-per-setup that can be done, even mitchellh says so.
    puts "NOTICE: Detected first-run of instance. Will connect localhost/2222 and run first-time-only script."
    # security configuration
    config.vm.provision :shell, :inline => 'grep "root.*NOPASSWD" /etc/sudoers || echo "root ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers'
    config.vm.provision :shell, :inline => 'grep "SSH_AUTH_SOCK" /etc/sudoers || echo "Defaults env_keep+=SSH_AUTH_SOCK" >> /etc/sudoers'
                                                                                                                                                                                              
    # configure iptables for our custom port                                                                                                                                                  
    # install basic ssh port 22 rule if there is no existing iptables config                                                                                                                  
    config.vm.provision :shell, :inline => "grep 'dport 22 ' /etc/sysconfig/iptables 2>/dev/null || ( iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT && service iptables save)"         
                                                                                                                                                                                              
    # configure ssh for our custom port                                                                                                                                                       
    config.vm.provision :shell, :inline => "grep '^Port 1855$' || echo 'Port 1855' >> /etc/ssh/sshd_config && service sshd restart"                                                           
    config.vm.provision :shell, :inline => "grep '^Port 22$' || echo 'Port 22' >> /etc/ssh/sshd_config && service sshd restart"                                                               
    config.vm.provision :shell, :inline => "sed -i 's/--dport 22 /--dport 1855 /gi' /etc/sysconfig/iptables && service iptables restart"                                                      
    config.vm.provision :shell, :inline => "grep 'dport 22 ' /etc/sysconfig/iptables 2>/dev/null || ( iptables -A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT && service iptables save)" 
  else                                                                                                                                                                                        
    # once we're on our custom port, disable the built-in host -> guest:22 forwarding so that we can spin up multiple vagrants without incident.                                              
    # https://github.com/mitchellh/vagrant/issues/1922                                                                                                                                        
    # https://github.com/mitchellh/vagrant/search?q=collision&ref=cmdform&type=Issues                                                                                                         
    #config.vm.network :forwarded_port,     guest: 1855,     host: 2222,     id: "ssh",     auto_correct: true, disabled: true                                                                
                                                                                                                                                                                              
                                                                                                                                                 
                                                                                                                                                                                              
  end                                                                                                                                                                                         

--

[Alvaro Miranda Aguilera]

have noticied a little error:

should be

    config.vm.provision :shell, :inline => "grep '^Port 1855$' /etc/ssh/sshd_config  || echo 'Port 1855' >> /etc/ssh/sshd_config && service sshd restart"                                                           

    config.vm.provision :shell, :inline => "grep '^Port 22$' /etc/ssh/sshd_config  || echo 'Port 22' >> /etc/ssh/sshd_config && service sshd restart"

[Alan Pinstein]

Thanks for the recommendations on the workaround, I will try it. Any reason you split these out over 2 different blocks?

I definitely see now how what I was doing was wrong with config vs override; this is one of those goofy errors due to closure. It's very subtle. Many rubyists have seen this code and no one even noticed the error. Thinking about it more I am not sure that you could really produce warnings for this case unless you decide to just mask "config" since people use it by convention; but AFAIK it's just a plain-old closure bug.

[Alan Pinstein]

I understand why the network breaks. However I made a point on the issue (https://github.com/mitchellh/vagrant/issues/2881#issuecomment-34139326) that in this case it'd seem  you can actually fix the bug. I don't see any reason to shut down the network, edit configs, then start the network. Why not edit the config, then restart the network in a single command, thus preventing any issue with my approach. I am sure I'm not the only one to run into this type of thing.

The other thing is that your fix is to add a rule for eth0, but there's no explicit config that makes the NAT on eth0. That might change in a future vagrant, and that'd be a subtle & dangerous regression. Unless/until you can map specific network configs to specific adapters, that seems like a bad idea, no?

[Alan Pinstein]

I should add.. I'm not really concerned about security with VirtualBox provider, but we wanted to start trying out Vagrant to bootstrap our cloud instances, and thus I wanted to automate moving of the ssh port.

That said, is there a place/way to put a block code that only runs on certain providers? So that way I could locate the code that kills port 22 ssh in certain providers. My concern here is that even if I could do this, wouldn't I still have the same problem but just on that provider?

I really think the fix here is to make vagrant's network munging much more robust. Change config then restart, and optionally maybe skip all of that if the config doesn't change...

Thoughts?

On Thursday, February 6, 2014 7:12:02 PM UTC-5, Alvaro Miranda Aguilera wrote:

[Alan Pinstein]

Actually, now that I try it with a real setup, I don't see how using:

  config.ssh.host = [private network ip] 

ever works... because in this situation, vagrant will always kill the private network on all runs 2..n trying to re-configure the network... see log below.

 INFO ssh: Attempting SSH connnection...

 INFO ssh: Attempting to connect to SSH...

 INFO ssh:   - Host: 33.33.33.50

 INFO ssh:   - Port: 1855

 INFO ssh:   - Username: vagrant

 INFO ssh:   - Key Path: ["/Users/apinstein/.vagrant.d/insecure_private_key"]

 INFO subprocess: Starting process: ["/usr/bin/VBoxManage", "showvminfo", "9e49fd36-ef99-49df-9219-b1b867e2ee64", "--machinereadable"]

 INFO ssh: SSH is ready!

 INFO interface: info: Machine booted and ready!

[default] Machine booted and ready!

 INFO warden: Calling IN action: #<VagrantPlugins::ProviderVirtualBox::Action::CheckGuestAdditions:0x00000100f8a0d8>

 INFO subprocess: Starting process: ["/usr/bin/VBoxManage", "guestproperty", "get", "9e49fd36-ef99-49df-9219-b1b867e2ee64", "/VirtualBox/GuestAdd/Version"]

 INFO warden: Calling OUT action: #<VagrantPlugins::ProviderVirtualBox::Action::CheckGuestAdditions:0x00000100f8a0d8>

 INFO warden: Calling OUT action: #<Vagrant::Action::Builtin::WaitForCommunicator:0x00000100f880a8>

 INFO warden: Calling OUT action: #<VagrantPlugins::ProviderVirtualBox::Action::Customize:0x00000100f80060>

 INFO warden: Calling OUT action: #<VagrantPlugins::ProviderVirtualBox::Action::Boot:0x00000100f800b0>

 INFO warden: Calling OUT action: #<VagrantPlugins::ProviderVirtualBox::Action::Customize:0x00000100f80178>

 INFO warden: Calling OUT action: #<VagrantPlugins::ProviderVirtualBox::Action::SaneDefaults:0x00000100ed1100>

 INFO warden: Calling OUT action: #<Vagrant::Action::Builtin::SetHostname:0x00000100ed1150>

 INFO warden: Calling OUT action: #<VagrantPlugins::ProviderVirtualBox::Action::ForwardPorts:0x00000100ed12e0>

 INFO subprocess: Starting process: ["/usr/bin/VBoxManage", "showvminfo", "9e49fd36-ef99-49df-9219-b1b867e2ee64", "--machinereadable"]

 INFO interface: info: Configuring and enabling network interfaces...

[default] Configuring and enabling network interfaces...

 INFO ssh: SSH is ready!

 INFO guest: Execute capability: configure_networks (redhat)

 INFO ssh: SSH is ready!

 INFO guest: Execute capability: network_scripts_dir (redhat)

 INFO ssh: Execute: /sbin/ifdown eth1 2> /dev/null (sudo=true)

 INFO subprocess: Starting process: ["/usr/bin/VBoxManage", "showvminfo", "9e49fd36-ef99-49df-9219-b1b867e2ee64", "--machinereadable"]

 INFO subprocess: Starting process: ["/usr/bin/VBoxManage", "showvminfo", "9e49fd36-ef99-49df-9219-b1b867e2ee64", "--machinereadable"]

 INFO ssh: Execute: echo; printf $SSH_AUTH_SOCK (sudo=false)

 INFO subprocess: Starting process: ["/usr/bin/VBoxManage", "showvminfo", "9e49fd36-ef99-49df-9219-b1b867e2ee64", "--machinereadable"]

 INFO subprocess: Starting process: ["/usr/bin/VBoxManage", "showvminfo", "9e49fd36-ef99-49df-9219-b1b867e2ee64", "--machinereadable"]

 INFO ssh: Setting SSH_AUTH_SOCK remotely: /tmp/ssh-UqBfSa1371/agent.1371

[hangs.... ssh vag...@33.33.33.50 will never work b/c vagrant just ran `/sbin/ifdown eth1` ]

Right?

[Alan Pinstein]

Actually you have to disable the custom ssh host & port, as with the private networking and custom ssh, the default ssh port-forwading doesn't run, and the box is inaccessible...

[Alvaro Miranda Aguilera]

I think the way you are approaching is causing you so much troubles.

1. At this moment, and until that change, the virtualbox provider will use eth0 as nat.

2. at the boot time, the port forwarders and the nics are cleared out and reconfigured

3. all the reconfiguration is done by the forwarded port done in 1.

so you have a chicken egg you are trying to break, but the way it is, it works, just you want to make it work diffeerently.

I think at the moment, you should have 2 configuration blocks, one for the destop/virtualbox and other for your cloud provider.

I did test those config.ssh.port/host and wasn't able to make them work due the point 1 and 2. 

After chekcing past issues, I can see you have been months trying to get those to work, however.

I thing the best way to go is the 2 block for virtualbox and cloud.

Alvaro.

[Alan Pinstein]

I think the way you are approaching is causing you so much troubles.

I agree; though I am talking about it not for my benefit, but for vagrant’s. Vagrant has features, which when used in expected and trivial ways, completely break vagrant. I am suggesting architectural approaches that will make it more robust. Vagrant is a really cool idea and useful project, but I ran into these issues very quickly, and judging by the issues/stackexchange q’s I’ve seen trying to solve my problem, I am not alone. So I am just trying to contribute to the project to make it more robust and accessible for vagrant’s user base.

1. At this moment, and until that change, the virtualbox provider will use eth0 as nat.

2. at the boot time, the port forwarders and the nics are cleared out and reconfigured

3. all the reconfiguration is done by the forwarded port done in 1.

so you have a chicken egg you are trying to break, but the way it is, it works, just you want to make it work diffeerently.

I think at the moment, you should have 2 configuration blocks, one for the destop/virtualbox and other for your cloud provider.

I did test those config.ssh.port/host and wasn't able to make them work due the point 1 and 2. 

After chekcing past issues, I can see you have been months trying to get those to work, however.

I thing the best way to go is the 2 block for virtualbox and cloud.

Right, and that’s what I have done for now, and commented it as a “lucky hack” that might break in the future. I do appreciate your efforts in helping me elucidate the problem. I also do hope, however, that vagrant can improve in its robustness so that it works cleanly with non-standard ssh ports on private networks. But for now I totally agree that leaving the vagrant NAT up for the “control channel” is a reasonable workaround. I don’t think that continues to be reasonable as people use vagrant to manage production infrastructure. I am probably outpacing most people’s use of vagrant at this time, and I understand that.

Anyway, thanks again for all the help and best of luck with the project.

Alan

You received this message because you are subscribed to a topic in the Google Groups "Vagrant" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vagrant-up/mkHSTiKnBpM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vagrant-up+...@googlegroups.com.

[Alvaro Miranda Aguilera]

I am an end user like you.

Just happen that I have worked years in support for unix/linux and other stuff, so I enjoy doing troubleshooting.

As time permits, I check over the mailing list and open bugs, and check what I believe are OS issues, and try to help people to get those request closed, so Vagrant dev people can focus on the new cool features that around the corner :D

Alvaro.