Not enough space in root partition

[Giorgio Gambino]

Hi,

I'm trying to install Eucalyptus on a bare metal machine with the 

Faststart script bash <(curl -Ls hphelion.com/eucalyptus-install). 

When I run it I get this warning message: 

"WARNING: we recommend at least 100G of disk space available

in /var for a Eucalyptus Faststart installation.  Running with

less disk space may result in issues with image and volume

management, and may dramatically reduce the number of instances

your cloud can run simultaneously." and then the installation fails.

Is there a way to resize the root partition in order to have enough space

or to create from scratch a custom disk image with the requirements 

that i need?

Thank you in advance,

Giorgio Gambino

[Brian Kroth]

The Utah folks might have some ideas that might fit better with the disk
imaging infrastructure that CloudLab uses, but here's some general Linux
disk management thoughts to maybe get you started.

Does it have to be all of /var that needs 100G or is some subdir
alright?

If it's the latter, you could make use of the extra space in the last
partition, maybe via LVM to give you added flexibility. Looks like you
might need to tweak that install script check in that case though.

If it's the former, well, you could do the same thing, only mount it
over /var instead. You'd want to (r)sync things from the existing /var
(dir) first though and is probably best done from a rescue console,
though should also be possible to do while booted too (though you might
lose a little bit of runtime state or log info that way). For instance:
# pvcreate /dev/sda4
# vgcreate vgname /dev/sda4
# lvcreate -L 100G -n var vgname
# mkfs.ext4 -L var /dev/vgname/var

(tweak this step's target mount point for the first case)
# echo "/dev/vgname/var /var ext4 defaults 0 0" >> /etc/fstab

(skip these steps in the first case)
# mount /dev/vgname/var /mnt
# rsync -aviPHAXS /var/ /mnt/
# umount /mnt

# mount /dev/vgname/var
# reboot

Cheers,
Brian

Giorgio Gambino <gambino...@gmail.com> 2015-10-16 06:32:

>--
>You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
>To post to this group, send email to cloudla...@googlegroups.com.
>To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/55c6c4b3-3e74-461d-b00d-d9c993f14b11%40googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

[Leigh Stoller]

> If it's the latter, you could make use of the extra space in the last partition, maybe via LVM to give you added flexibility. Looks like you might need to tweak that install script check in that case though.

Just a heads up, that if you use an LVM you cannot use Cloudlab’s Clone
or Snapshot functions, which means every time you have to terminate
(because the clusters are busy and we deny an extension request) you will
lose everything.

I suggest holding off until we figure out what to tell you.

Leigh

[Mike Hibler]

An easier variant of the "mount extra space on /var" approach might be
to see if it uses a specific subdir of /var, like /var/eucalyptus, and
just mount the extra FS there. Then you avoid all the issues of copying
existing /var directories.

> https://groups.google.com/d/msgid/cloudlab-users/20151016154807.GC14122%40cs.wisc.edu.

[Mike Hibler]

Whether there is a "better" solution depends on how much data there is and
what the workflow is.

If this is a one-shot experiment, then it makes sense to create a local FS
on the node and just copy the data in.

If you want to instantiate the experiment again and again over time with
the same dataset, then the best solution depends on the size of the dataset
and whether you want changes to the data to be persistent across
instantiations.

If the actual data it is less than around 10GB and you don't need persistence,
then loading the data every time is still reasonable using a CloudLab
"image-backed" dataset. Actually, you can use image-backed datasets if
you want to persist changes, it just takes extra time and effort to copy
all the data back out again (i.e., you need to remember to "snapshot"
the dataset before you destroy your experiment).

If you have a huge amount of data, then it would be better to have a
solution where you don't have to populate a node each time. For this you
can use the SAN-based ("short term" and "long term") datasets.

The big caveats here are:

- SAN-based datasets can only be used on the cluster where they are created
- you can currently only create them on Clemson and Apt clusters
- only one node per experiment can attach to the dataset

But with that in mind, you can create a persistent dataset via the portal:

https://www.cloudlab.us/create-dataset.php

and an example profile that uses a persistent dataset:

https://www.cloudlab.us/show-profile.php?uuid=387de1aa-6e13-11e5-96c6-38eaa71273fa

In fact, you can just instantiate the profile, as it will prompt you for
the dataset to use and where to mount it.

On Fri, Oct 16, 2015 at 10:48:07AM -0500, Brian Kroth wrote:

> https://groups.google.com/d/msgid/cloudlab-users/20151016154807.GC14122%40cs.wisc.edu.

[Giorgio Gambino]

>If you want to instantiate the experiment again and again over time with 
>the same dataset, then the best solution depends on the size of the dataset 
>and whether you want changes to the data to be persistent across 
>instantiations. 

Hi Mike and thanks for your suggestions. 

This looks like a good way to obtain an extra space and mount it wherever i need .

The method indicated from Brian also works but i need to take snapshots.

>and an example profile that uses a persistent dataset: 

  >  https://www.cloudlab.us/show-profile.php?uuid=387de1aa-6e13-11e5-96c6-38eaa71273fa 

>In fact, you can just instantiate the profile, as it will prompt you for 
>the dataset to use and where to mount it. 

I wrote this script using geni-lib:

#BEGIN

import geni.portal as portal

import geni.rspec.pg as rspec

import geni.rspec.igext as igext

_pcon = portal.Context()

_pcon.defineParameter("DATASET","URN of the dataset", portal.ParameterType.STRING,"urn:publicid:IDN+clemson.cloudlab.us:fuzzysdn-pg0+dataset+fuzzysdn-PG0:data_set")

_pcon.defineParameter("M_POINT","Dataset mountpoint", portal.ParameterType.STRING,"/var")

_params = _pcon.bindParameters()

_rspec = rspec.Request()

#euca-server

euca_server = rspec.RawPC("euca-server")

euca_inf = euca_server.addInterface("if0")

euca_server.disk_image = "urn:publicid:IDN+emulab.net+image+emulab-ops:CENTOS66-64-STD"

#blockstore

blockstore = igext.RemoteBlockstore("blockstore",_params.M_POINT)

block_inf = blockstore.interface

blockstore.dataset = _params.DATASET

#linking euca_server and blockstore

link = rspec.Link("link")

link.addInterface(euca_inf)

link.addInterface(block_inf)

#special parameters

link.best_effort = True

link.vlan_tagging = True

_doc = igext.Tour()

_doc.Description(_doc.TEXT,"Desc")

_doc.Instructions(_doc.MARKDOWN,"Istr")

_rspec.addTour(_doc)

_rspec.addResource(euca_server)

_rspec.addResource(blockstore)

_rspec.addResource(link)

_pcon.printRequestRSpec(_rspec)

#END

it's based on the example profile contained in the link that

you provided to me. But once started an experiment with this profile I can't ssh 

into the machines.

Should i add something in the script to enable ssh or something like that?

Thanks, Giorgio

[Leigh Stoller]

> But once started an experiment with this profile I can't ssh
> into the machines.
> Should i add something in the script to enable ssh or something like that?

Hi there, I see your experiment as started okay. When you click on the
euca-server node, you should get a context menu, and the first option is
Shell. What happens when you click on that?

Leigh

[Giorgio Gambino]

>Hi there, I see your experiment as started okay. When you click on the 
>euca-server node, you should get a context menu, and the first option is 
>Shell. What happens when you click on that? 

>Leigh 

Hi Leigh,

If I try to open a shell on euca-server I get this error:

  ssh_exchange_identification: Connection closed by remote host

If I try to open a shell on blockstore I get this error:

 ssh: connect to host clstore03.clemson.cloudlab.us port 30010: Connection refused           

[Leigh Stoller]

> Hi Leigh,
>
> If I try to open a shell on euca-server I get this error:
> ssh_exchange_identification: Connection closed by remote host

Okay, that is odd. Leave that experiment active so I can poke around.
Stay tuned …

Leigh

[Leigh Stoller]

Oh, I see. You mounted your blockstore on /var. That will not work,
you have buried all the stuff that the OS needs to boot properly.
You need to mount it someplace else or on a subdir of /var (I think
that subdir will need to exist already, but not sure).

Leigh

[Giorgio Gambino]

Ok,

this is a problem because I'm doing all this stuff because I need more than 100 GB 

under /var directory to run the script properly.

What can I do? 

Giorgio

[Leigh Stoller]

I think Mike mentioned this in his reply; where in /var does it need this
space? Typically, it is just one directory. If you can find that out, it
should be possible to mount it there instead of /var

Leigh

[Mike Hibler]

There is no problem mounting it on a subdirectory of /var, you just cannot
mount it on /var proper. A quick google turns up:

https://www.eucalyptus.com/docs/eucalyptus/4.0/install-guide/space_verifying_ha.html

which shows that mostly Eucalyptus uses space in /var/lib/eucalyptus. Try
mounting it there.

> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To post to this group, send email to cloudla...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/431978ef-f6d7-4f97-94d1-68ac823fbe30%40googlegroups.com.

[Brian Kroth]

Also, like I mentioned earlier, once you've done as Mike and Leigh
suggested, you might just need to tweak that particular sanity check in
the install script you were using and have it check for ~100G in, say
/var/lib/eucalytpus, instead of /var. It (the install script) is
probably just being lazy in only checking that /var has 100GB available
rather than some subdir.

Good luck.

Cheers,
Brian

Mike Hibler <mi...@flux.utah.edu> 2015-10-19 11:58:

>To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/20151019175806.GM78947%40flux.utah.edu.