systemsize calculation on lvm-based image

[Pavol Domin]

Hi kiwis,

I have an impression, that kiwi oem image is larger than necessary on lvm-based definition:

We specify systemdisk as

<systemdisk name="vg00">
  <volume name="usr" size="5G"></volume>
  <volume name="home" size="1G"></volume>
  ...

</systemdisk>

 i.e. most of installed data is on dedicated lvol usr. However, the calculation of systemsize do not take this into account; as far as I see, it initiates image size with

$sizeBytes = qxx ("du -s --block-size=1 $system | cut -f1")

and does not seem to subtract the utilization on dedicated lvols, like /usr in this case.

Later on we update this initial size again by counted size of each volume.

So we end with the image with the grand total size, plus the sizes per each specified lvol.. 

Is this the indeed the case, or I miss something?

Kind regards

Pavol

[Marcus Schäfer]

Hi,

yes this is the expected behavior. for the vmx type (which I guess this
is the one you are using) kiwi calculates how much space plus some free
space is needed to store all data that's the min start size of the disk.
After that it looks at your systemdisk requests and adds the requested
sizes to the disk size. You requested a fixed size of 5G and 1G volumes
so kiwi adds this values to the overall disk size

After that it creates the volumes which are now of the requested
size. It then mounts everything together and syncs all data which
causes the data to be shared over the volumes and the rest.

If you intend to specify a relative size free in a volume you can
use freespace="..." instead of a fixed size="..." value

makes sense ?

Regards,
Marcus
--
Public Key available
gpg --keyserver pgp.mit.edu --recv-keys 0xCCE3C6A2
-------------------------------------------------------
Marcus Schäfer (Res. & Dev.) SUSE LINUX Products GmbH
Tel: 0911-740 53 0 Maxfeldstrasse 5
FAX: 0911-740 53 479 D-90409 Nürnberg
GF: Jeff Hawn,Jennifer Guild, Felix Imendörffer
HRB: 21284 (AG Nürnberg) Germany
http://www.suse.de
-------------------------------------------------------

[Pavol Domin]

Hi

On 16 May 2013 15:29, Marcus Schäfer <m...@suse.de> wrote:

yes this is the expected behavior. for the vmx type (which I guess this
is the one you are using) kiwi calculates how much space plus some free
space is needed to store all data that's the min start size of the disk.

This is my point. I think the calculation is not good: it calculates sum of all used
space in unpacked tree, but it does not take into account, that some of data
(like the /usr) will be copied "outside" the root lvol: to mounted volume,
i.e. we do not need that space in systemsize.

for example assume, in '--prepare' we copy all data into '/usr' and we define

<volume name="usr" size="5G"></volume>

The '--create' than counts 'du -s' on whole unpacked tree + it adds /usr size to it again, as we define it as volume; we end

with the image twice as big as we need (if i don't consider compression).

Pavol

[Marcus Schäfer]

Hi,

sorry I don't get it, in the setup you requested a size of 5G for the
/usr volume, what other than creating a 5G volume should kiwi do now ?

when you create an 'oem' image instead of a 'vmx' image kiwi just stores
the requested sizes and increases to the requested sizes on first boot
thus the image itself is just as big as it needs to be to store the
data in that case

I'm sorry but I still don't understand your use case

[Pavol Domin]

Hi

On 16 May 2013 16:15, Marcus Schäfer <m...@suse.de> wrote:

sorry I don't get it, in the setup you requested a size of 5G for the
/usr volume, what other than creating a 5G volume should kiwi do now ?

when you create an 'oem' image instead of a 'vmx' image kiwi just stores
the requested sizes and increases to the requested sizes on first boot
thus the image itself is just as big as it needs to be to store the
data in that case

Sorry for being confusing. I am creating OEM image indeed not the vmx one.

The extensions to the requested sizes at firstboot time is as expected.

My concern is, that the OEM ISO image itself is bigger than it could be and I think it's because occupation in volumes is counted twice - first time in global "du -s" on whole unpacked tree, and second time in loop, adding "du -s" from the volumes.

Or - I terribly overlook something. Still the dumped raw image shows big, but almost empty LVRoot and I think this is not wanted in OEM image.

Kind regards

Pavol

 

I'm sorry but I still don't understand your use case

Regards,
Marcus
--
 Public Key available
 gpg --keyserver pgp.mit.edu --recv-keys 0xCCE3C6A2
 -------------------------------------------------------
 Marcus Schäfer (Res. & Dev.)   SUSE LINUX Products GmbH
 Tel: 0911-740 53 0             Maxfeldstrasse 5
 FAX: 0911-740 53 479           D-90409 Nürnberg
 GF:  Jeff Hawn,Jennifer Guild, Felix Imendörffer
 HRB: 21284 (AG Nürnberg)       Germany
 http://www.suse.de
 -------------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "kiwi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kiwi-images...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Marcus Schäfer]

Hi,

> Sorry for being confusing. I am creating OEM image indeed not the vmx
> one.
> The extensions to the requested sizes at firstboot time is as expected.
> My concern is, that the OEM ISO image itself is bigger than it could be
> and I think it's because occupation in volumes is counted twice - first
> time in global "du -s" on whole unpacked tree, and second time in loop,
> adding "du -s" from the volumes.
> Or - I terribly overlook something. Still the dumped raw image shows
> big, but almost empty LVRoot and I think this is not wanted in OEM
> image.

Oh you are saying the install iso is bigger than you would expect it ?
I'll give it a test and come back with some numbers. Stay tuned ;)

[Marcus Schäfer]

Hi,

> > one.
> > The extensions to the requested sizes at firstboot time is as expected.
> > My concern is, that the OEM ISO image itself is bigger than it could be
> > and I think it's because occupation in volumes is counted twice - first
> > time in global "du -s" on whole unpacked tree, and second time in loop,
> > adding "du -s" from the volumes.
> > Or - I terribly overlook something. Still the dumped raw image shows
> > big, but almost empty LVRoot and I think this is not wanted in OEM
> > image.
>
> Oh you are saying the install iso is bigger than you would expect it ?
> I'll give it a test and come back with some numbers. Stay tuned ;)

ok I build a JeOS image with the following type and systemdisk

<type image="oem"
filesystem="ext3"
boot="oemboot/suse-12.3"
installiso="true"
bootloader="grub2">
<systemdisk>

<volume name="usr" size="5G"/>

</systemdisk>
<oemconfig>
<oem-swap>true</oem-swap>
</oemconfig>
</type>

the unpacked tree need ~900M according to a "du -ks" inside the tree
after prepare. Here is the result:

385M May 17 11:57 LimeJeOS-openSUSE-12.3.x86_64-1.12.3.install.iso
1.3G May 17 11:55 LimeJeOS-openSUSE-12.3.x86_64-1.12.3.raw

So the 1.3G for the plain raw disk is ok, because custom partitions
are based on LVM and kiwi adds a boot partition of at least 200M plus
some free space which is required to make the image work even without
a repartition step. So we have 900M + 200M + some-free... ok for me

The result install iso image is 385M which is ok and the result of
the compression kiwi did with the .raw image file. So on the iso there
is the kernel, initrd and the image as compressed version.

All in all that are good numbers for me.
How are yours different from mine

Thanks

[Pavol Domin]

Hi,

OK, i reproduced our scenario. We have following config:

<type image="oem"
      filesystem="ext3" fsmountoptions="errors=remount-ro,data_err=abort"
      boot="oemboot/suse-SLES11"
      installiso="true"
      bootpartsize="500"
      bootfilesystem="ext3"
      kernelcmdline="elevator=noop console=tty0 crashkernel=0-12G:256M,12G-48G:512M,48G-128G:1G,128G-:2G">

        <systemdisk name="vg00">
                <volume name="usr" size="5G"></volume>
                <volume name="home" size="1G"></volume>

                <volume name="tmp" size="2G"></volume>
                <volume name="opt" size="2G"></volume>
                <volume name="var" size="1G"></volume>
                <volume name="var/opt" size="1G"></volume>
                <volume name="var/log" size="1G"></volume>
                <volume name="var/crash" size="all"></volume>
                <volume name="usr/sap" size="500M"></volume>

        </systemdisk>
        <oemconfig>
                <oem-align-partition>true</oem-align-partition>
                <oem-swap>false</oem-swap>
                <oem-reboot>false</oem-reboot>
        </oemconfig>
</type>

Created kiwi image and loop mounted lvols inside:

img # losetup /dev/loop0 SLESS11.2.x86_64-0.0.15.raw -o525340672
img # lvm pvscan /dev/loop0
  PV /dev/loop0   VG vg00   lvm2 [4.95 GiB / 0    free]
  Total: 1 [4.95 GiB] / in use: 1 [4.95 GiB] / in no VG: 0 [0   ]
img # lvs
  LV          VG   Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  LVRoot      vg00 -wi---  1.55g
  LVhome      vg00 -wi--- 32.00m
  LVopt       vg00 -wi--- 56.00m
  LVtmp       vg00 -wi--- 32.00m
  LVusr       vg00 -wi---  3.06g
  LVusr_sap   vg00 -wi--- 32.00m
  LVvar       vg00 -wi--- 96.00m
  LVvar_crash vg00 -wi--- 32.00m
  LVvar_log   vg00 -wi--- 36.00m
  LVvar_opt   vg00 -wi--- 32.00m
img # mkdir /tmp/kiwiroot
img # lvchange -a y /dev/vg00/LVRoot
img # lvchange -a y /dev/vg00/LVusr
img # mount /dev/vg00/LVRoot /tmp/kiwiroot
img # mount /dev/vg00/LVusr /tmp/kiwiroot/usr
img # df /tmp/kiwiroot /tmp/kiwiroot/usr
Filesystem              1K-blocks    Used Available Use% Mounted on
/dev/mapper/vg00-LVRoot   1552860  259952   1211808  18% /tmp/kiwiroot
/dev/mapper/vg00-LVusr    3110616 2772892    177164  94% /tmp/kiwiroot/usr

The question is: why LVRoot in the raw image is that big, when only 18% fraction is utilized?

Pavol

[Marcus Schäfer]

Thanks for the details, yes this sounds weird. I will reproduce this
with your setup and see what happens

Thanks

Regards,
Marcus
--
Public Key available

gpg --keyserver gpg-keyserver.de --recv-keys 0xCCE3C6A2

[Marcus Schäfer]

Hi,

> > The question is: why LVRoot in the raw image is that big, when only 18%
> > fraction is utilized?
>
> Thanks for the details, yes this sounds weird. I will reproduce this
> with your setup and see what happens

This problem should be fixed in the next version of kiwi.
Here the commit:

commit 3560ead05525f851fd107ff1f1d797223a828cc7
Author: Marcus Schäfer <m...@suse.de>
Date: Mon May 20 19:16:45 2013 +0200

- fixed and optimized disk size calculation
* I had by default 40% spare space on the root path which seems to
be too much, I have 20% spare on each volume so I adapted that
to the root path as well.
* for the size estimation I now use the --apparent-size option in
du, which works better on files with holes and other or other
attributes which makes a difference when copying those data.
* Use a fixed value for the filesystem overhead each volume adds
to the disk. It is wrong to calculate and add a 20% spare/volume
according to the size of the data in the volume to the disk,
because that value has already been taken into account with
the data spare calculation of the entire tree

by intention I want each volume to have ~20% of free space.
This is an empiric number which worked for me. Unfortunately
it is really hard to exactly precalculate the needed filesystem
size and even harder to precalculate the needed disk size

I will submit a new kiwi version by the end of this week latest
Would be nice if you can give it a test too

I got these values with a similar setup as yours:

lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
LVRoot vg00 -wi-a---- 356.00m
LVhome vg00 -wi-a---- 32.00m
LVopt vg00 -wi-a---- 32.00m
LVtmp vg00 -wi-a---- 32.00m
LVusr vg00 -wi-a---- 4.00g
LVusr_sap vg00 -wi-a---- 32.00m
LVvar vg00 -wi-a---- 52.00m
LVvar_crash vg00 -wi-a---- 32.00m
LVvar_log vg00 -wi-a---- 36.00m
LVvar_opt vg00 -wi-a---- 32.00m

mount /dev/mapper/vg00-LVRoot /mnt
/dev/mapper/vg00-LVRoot 321M 227M 77M 75% /mnt

mount /dev/mapper/vg00-LVusr /mnt
/dev/mapper/vg00-LVusr 3.9G 3.5G 180M 96% /mnt

...

[Pavol Domin]

super thanks! I'll check once the next version appears on obs.

[Pavol Domin]

Hi,

checked with v5.05.16; --create failed, rsync exited with error:

May-21 20:15:25 <1> : EXEC [rsync -aHXA --one-file-system /kiwi/build/kiwi/ /tmp/kiwiloop.EG2dFe 2>&1]
May-21 20:20:36 <3> : Can't copy image tree to disk: rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (3101935 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

It filled LVusr filesystem, not clue why; these are actual sizes un unpacked tree

# du -s --apparent-size --block-size=1 kiwi/usr/
2429780340      kiwi/usr/

# du -s --block-size=1 kiwi/usr/
2740830208      kiwi/usr/

# du -s --apparent-size --block-size=1 kiwi/usr/*
16399   kiwi/usr/X11R6
187217320       kiwi/usr/bin
4096    kiwi/usr/games
13169221        kiwi/usr/include
219283728       kiwi/usr/lib
815886840       kiwi/usr/lib64
81957   kiwi/usr/local
4096    kiwi/usr/sap
38265607        kiwi/usr/sbin
704526944       kiwi/usr/share
451303542       kiwi/usr/src
10      kiwi/usr/tmp
16484   kiwi/usr/x86_64-suse-linux

and mounted raw volume:
# df --block-size=1 /tmp/kiwiloop.EG2dFe/usr
Filesystem              1B-blocks       Used Available Use% Mounted on
/dev/mapper/vg00-LVusr 2827902976 2827890688         0 100% /tmp/kiwiloop.EG2dFe/usr

# du -s --apparent-size --block-size=1 /tmp/kiwiloop.EG2dFe/usr/*
16399   /tmp/kiwiloop.EG2dFe/usr/X11R6
187217320       /tmp/kiwiloop.EG2dFe/usr/bin
4096    /tmp/kiwiloop.EG2dFe/usr/games
13169221        /tmp/kiwiloop.EG2dFe/usr/include
219283728       /tmp/kiwiloop.EG2dFe/usr/lib
815878648       /tmp/kiwiloop.EG2dFe/usr/lib64
81957   /tmp/kiwiloop.EG2dFe/usr/local
16384   /tmp/kiwiloop.EG2dFe/usr/lost+found
13312   /tmp/kiwiloop.EG2dFe/usr/sap
38265607        /tmp/kiwiloop.EG2dFe/usr/sbin
704526944       /tmp/kiwiloop.EG2dFe/usr/share
442471329       /tmp/kiwiloop.EG2dFe/usr/src
10      /tmp/kiwiloop.EG2dFe/usr/tmp
4096    /tmp/kiwiloop.EG2dFe/usr/x86_64-suse-linux

Can it be because we mount (empty) /usr/sap in kiwi below /usr
<volume name="usr/sap" size="500M"></volume> ?

Kind regards

Pavol

[Pavol Domin]

...just noting this message during prepare (was there ale before this patch i think):

Argument "all" isn't numeric in addition (+) at /usr/share/kiwi/modules/KIWIRuntimeChecker.pm line 312.

Kind regards

Pavol

[Marcus Schäfer]

Hi,

> ...just noting this message during prepare (was there ale before this
> patch i think):
> Argument "all" isn't numeric in addition (+) at
> /usr/share/kiwi/modules/KIWIRuntimeChecker.pm line 312.

I already fixed this here:

commit dcba40c06d415a626495aec8c0a41e2ea2a7a28a
Author: Marcus Schäfer <m...@suse.de>
Date: Mon May 20 17:37:46 2013 +0200

- fixed __checkSystemDiskData runtime check to work correctly
with the 'all' size specification

[Marcus Schäfer]

Hi,

> checked with v5.05.16; --create failed, rsync exited with error:
> May-21 20:15:25 <1> : EXEC [rsync -aHXA --one-file-system
> /kiwi/build/kiwi/ /tmp/kiwiloop.EG2dFe 2>&1]
> May-21 20:20:36 <3> : Can't copy image tree to disk: rsync:
> writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken
> pipe (32)
> rsync: connection unexpectedly closed (3101935 bytes received so far)
> [sender]
> rsync error: error in rsync protocol data stream (code 12) at io.c(605)
> [sender=3.0.9]
> It filled LVusr filesystem, not clue why; these are actual sizes un
> unpacked tree

Hmm, yes it runs out of space on the usr volume now.
I think it's because of the --apparent-size option I use

> # du -s --apparent-size --block-size=1 kiwi/usr/
> 2429780340 kiwi/usr/
> # du -s --block-size=1 kiwi/usr/
> 2740830208 kiwi/usr/

so without --apparent-size we got a bigger value. But it would
create wrong numbers for sparse files.

--apparent-size
print apparent sizes, rather than disk usage; although the
apparent size is usually smaller, it may be larger due to holes
in ('sparse') files, internal fragmentation, indirect blocks,
and the like

I'll tend to revert that change but then we will get the errors
back if spares files are used in the tree which is often the
case if people preserve database files or similar

anybody with a good solution on that ?

[Roman Neuhauser]

# m...@suse.de / 2013-05-22 11:43:47 +0200:

> Hmm, yes it runs out of space on the usr volume now.
> I think it's because of the --apparent-size option I use
>
> > # du -s --apparent-size --block-size=1 kiwi/usr/
> > 2429780340 kiwi/usr/
> > # du -s --block-size=1 kiwi/usr/
> > 2740830208 kiwi/usr/
>
> so without --apparent-size we got a bigger value. But it would
> create wrong numbers for sparse files.
>
> --apparent-size
> print apparent sizes, rather than disk usage; although the
> apparent size is usually smaller, it may be larger due to holes
> in ('sparse') files, internal fragmentation, indirect blocks,
> and the like
>
> I'll tend to revert that change but then we will get the errors
> back if spares files are used in the tree which is often the
> case if people preserve database files or similar
>
> anybody with a good solution on that ?

retrieve both values, use the larger one of them?

--
roman

[Marcus Schäfer]

Hi,

> retrieve both values, use the larger one of them?

It will increase the build time especially for large root trees
but yes a possible solution. There is no other utility except 'du'
to sum up file sizes ?

[Pavol Domin]

Hi,

I'll tend to revert that change but then we will get the errors
back if spares files are used in the tree which is often the
case if people preserve database files or similar

anybody with a good solution on that ?

What about reverting --apparent-size, and calling rsync with the "--sparse"
-S, --sparse
              Try to handle sparse files efficiently so they take up less space on the destination.  Conflicts with --inplace because it's not possible to overwrite data in a sparse fashion.

Kind regards

Pavol

[Marcus Schäfer]

Hi,

> I'll tend to revert that change but then we will get the errors
> back if spares files are used in the tree which is often the
> case if people preserve database files or similar
> anybody with a good solution on that ?
>
> What about reverting --apparent-size, and calling rsync with the
> "--sparse"
> -S, --sparse

had that some time before and it causes conflicts for different
filesystems especially if block allocations are in use with
fallocate.

until we find a better solution here I picked up the approach
and check the size twice. Would you mind to give it a try with
the commit in:

commit 3476479b97c4d1c36ab6f2579b569d5dfbac1797
Author: Marcus Schäfer <m...@suse.de>
Date: Wed May 22 12:32:49 2013 +0200

- create global function dsize() which implements a size method
to calculate the size of all entities in a directory using the
du utility. du is called in two passes first in standard mode
counting disk usage and secondly in apparent mode also checking
for files with holes, fragmentation, indirect blocks and the
like. The biggest value is returned

I'm not really happy with it but if it fixes your build it is at
least a better solution than none

[Pavol Domin]

Hi

I already fixed this here:

commit dcba40c06d415a626495aec8c0a41e2ea2a7a28a
Author: Marcus Schäfer <m...@suse.de>
Date:   Mon May 20 17:37:46 2013 +0200

I have this patch already (included in my Tuesday's v5.05.16 from the OBS)

This is another bug, line 312 in modules/KIWIRuntimeChecker.pm:
 $volSizes += $size;

Kind regards

Pavol

[Marcus Schäfer]

Hi,

> I have this patch already (included in my Tuesday's v5.05.16 from the
> OBS)
> This is another bug, line 312 in modules/KIWIRuntimeChecker.pm:
> $volSizes += $size;

you are righ, i fixed it