PFE VM never boots past "Volume not properly unmounted" error message

[ro...@apstra.com]

I know this is not the right place for a question like that, but I'm hoping someone faced similar issue and may provide some guidance.

The problem appears when I'm trying to start vagrant environment using full-2qfx-4srv-evpnvxlan, see description below.

Overview

Symptom 1

Both `vqfx1-pfe` and `vqfx2-pfe` VMs in `full-2qfx-4srv-evpnvxlan` topology get stuck with following error message:

FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

VM boot process doesn't go any further.

Symptom 2

`vqfx1` and `vqfx2` VMs do not have a single `xe-a/b/c` interface:

vagrant@vqfx1> show interfaces terse
Interface               Admin Link Proto    Local                 Remote
gr-0/0/0                up    up
bme0                    up    up
bme0.0                  up    up   inet     128.0.0.1/2
                                            128.0.0.4/2
                                            128.0.0.16/2
                                            128.0.0.63/2
cbp0                    up    up
dsc                     up    up
em0                     up    up
em0.0                   up    up   inet     10.0.2.15/24
em1                     up    up
em1.0                   up    up   inet     169.254.0.2/24
em2                     up    up
em2.32768               up    up   inet     192.168.1.2/24
em3                     up    up
em4                     up    up
em4.32768               up    up   inet     192.0.2.2/24
em5                     up    up
em6                     up    up
em7                     up    up
esi                     up    up
gre                     up    up
ipip                    up    up
irb                     up    up
irb.10000               up    down inet     10.10.1.251/24
                                            10.10.1.254/24
irb.20000               up    down inet     10.10.2.251/24
                                            10.10.2.254/24
jsrv                    up    up
jsrv.1                  up    up   inet     128.0.0.127/2
lo0                     up    up
lo0.0                   up    up   inet     9.9.9.1             --> 0/0
                                   inet6    fe80::205:860f:fc71:ea00
lo0.16385               up    up   inet
lsi                     up    up
mtun                    up    up
pimd                    up    up
pime                    up    up
pip0                    up    up
tap                     up    up
vme                     up    down
vtep                    up    up
vtep.32768              up    up

Steps to reproduce

1. Bring up `full-2qfx-4srv-evpnvxlan` topology:

$ cd full-2qfx-4srv-evpnvxlan
$ vagrant up

Expected result

- All VMs in topology are running

Actual result

- VMs are running, but VM provisioning for PFE instances fails with timeout:

$ vagrant up vqfx1-pfe
Bringing machine 'vqfx1-pfe' up with 'virtualbox' provider...
==> vqfx1-pfe: Importing base box 'juniper/vqfx10k-pfe'...
==> vqfx1-pfe: Matching MAC address for NAT networking...
==> vqfx1-pfe: Checking if box 'juniper/vqfx10k-pfe' is up to date...
==> vqfx1-pfe: Setting the name of the VM: full-2qfx-4srv-evpnvxlan_vqfx1-pfe_1536100593522_60660
==> vqfx1-pfe: Fixed port collision for 22 => 2222. Now on port 2201.
==> vqfx1-pfe: Clearing any previously set network interfaces...
==> vqfx1-pfe: Preparing network interfaces based on configuration...
    vqfx1-pfe: Adapter 1: nat
    vqfx1-pfe: Adapter 2: intnet
==> vqfx1-pfe: Forwarding ports...
    vqfx1-pfe: 22 (guest) => 2201 (host) (adapter 1)
==> vqfx1-pfe: Running 'pre-boot' VM customizations...
==> vqfx1-pfe: Booting VM...
==> vqfx1-pfe: Waiting for machine to boot. This may take a few minutes...
    vqfx1-pfe: SSH address: 127.0.0.1:2201
    vqfx1-pfe: SSH username: vagrant
    vqfx1-pfe: SSH auth method: private key
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.


If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.


If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.


If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.

Environment

 - Vagrant

$ vagrant --version
Vagrant 2.0.1

 - Virtualbox

$ VBoxManage --version
5.2.16r123759