new beaglebone image-builder script fails when downloading Cloud9

[Dennis Cote]

I tried to use the image builder linked at the end of http://www.beagleboard.org/blog/2014-01-04-happy-new-year/ to build a new Debian based image for a BBB. I used the following commands in my home directory to download and run the image builder script.

mkdir BBB

cd BBB

git clone https://github.com/beagleboard/image-builder

cd image-builder

./beagleboard.org_image.sh

The script worked until it tried to clone the Cloud9 git repository. There were many subsequent errorsafter that as copied below.

Cloning into '/opt/cloud9'...

error: Problem with the SSL CA cert (path? access rights?) while accessing https://github.com/ajaxorg/cloud9.git/info/refs

fatal: HTTP request failed

/bin/chown: invalid user: `debian:debian'

Cloning into '/var/www'...

error: Problem with the SSL CA cert (path? access rights?) while accessing https://github.com/beagleboard/bone101/info/refs

fatal: HTTP request failed

Cloning into '/var/lib/cloud9'...

error: Problem with the SSL CA cert (path? access rights?) while accessing https://github.com/beagleboard/bonescript/info/refs

fatal: HTTP request failed

/bin/chown: invalid user: `debian:debian'

Cloning into '/opt/source/libsoc'...

error: Problem with the SSL CA cert (path? access rights?) while accessing https://github.com/jackmitch/libsoc/info/refs

fatal: HTTP request failed

final.sh: 160: cd: can't cd to /opt/source/libsoc/

final.sh: 161: final.sh: ./autogen.sh: not found

final.sh: 162: final.sh: ./configure: not found

final.sh: 163: final.sh: make: not found

final.sh: 164: final.sh: make: not found

final.sh: 165: final.sh: make: not found

Cloning into '/opt/source/Userspace-Arduino'...

error: Problem with the SSL CA cert (path? access rights?) while accessing https://github.com/prpplague/Userspace-Arduino/info/refs

fatal: HTTP request failed

/bin/sed: can't read /etc/ssh/sshd_config: No such file or directory

/bin/sed: can't read /etc/ssh/sshd_config: No such file or directory

Does anyone know what went wrong, or more importantly, what I need to do to fix it?

TIA

Dennis Cote

[Robert Nelson]

On Tue, Jan 14, 2014 at 1:57 PM, Dennis Cote <den...@harding.ca> wrote:
> I tried to use the image builder linked at the end of
> http://www.beagleboard.org/blog/2014-01-04-happy-new-year/ to build a new
> Debian based image for a BBB. I used the following commands in my home
> directory to download and run the image builder script.
>
> mkdir BBB
> cd BBB
> git clone https://github.com/beagleboard/image-builder
> cd image-builder
> ./beagleboard.org_image.sh
>
> The script worked until it tried to clone the Cloud9 git repository. There
> were many subsequent errorsafter that as copied below.

I'm guessing this was on x86?

So what OS and what version and what version of qemu-arm-static

qemu-arm-static -version

> Cloning into '/opt/cloud9'...
> error: Problem with the SSL CA cert (path? access rights?) while accessing
> https://github.com/ajaxorg/cloud9.git/info/refs
> fatal: HTTP request failed
> /bin/chown: invalid user: `debian:debian'

I'm not 100% sure of this is qemu to blame or the network went down..

BTW: due to issues with qemu across the board, I only run this script
on native arm hardware (Debian Jessie)..

Regards,

--
Robert Nelson
http://www.rcn-ee.com/

[Dennis Cote]

On Tuesday, January 14, 2014 1:27:44 PM UTC-7, RobertCNelson wrote:

I'm guessing this was on x86?

So what OS and what version and

Yes, you are correct. A Ubuntu 13.10 VM running on virtualbox. 

Linux dennis-VirtualBox 3.11.0-15-generic #23-Ubuntu SMP Mon Dec 9 18:17:04 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

 

what version of qemu-arm-static

qemu-arm-static -version

qemu-arm version 1.5.0 (Debian 1.5.0+dfsg-3ubuntu5.2), Copyright (c) 2003-2008 Fabrice Bellard

 

I'm not 100% sure of this is qemu to blame or the network went down..

BTW: due to issues with qemu across the board, I only run this script
on native arm hardware (Debian Jessie)..

 OK I can try to build with a Ubuntu 13.10 install I have running on an SD card on the BBB. 

Linux ubuntu-armhf 3.8.13-bone30 #1 SMP Thu Nov 14 06:23:24 UTC 2013 armv7l armv7l armv7l GNU/Linux

How do you get the image off the SD card used to build it and onto another SD card to test booting? I guess I'll copy it up to another computer (like my Ubuntu PC) and write the image to an SD card there.

Thanks.

Dennis Cote

[Robert Nelson]

> Yes, you are correct. A Ubuntu 13.10 VM running on virtualbox.
>
> Linux dennis-VirtualBox 3.11.0-15-generic #23-Ubuntu SMP Mon Dec 9 18:17:04
> UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>
>> what version of qemu-arm-static
>>
>> qemu-arm-static -version
>
> qemu-arm version 1.5.0 (Debian 1.5.0+dfsg-3ubuntu5.2), Copyright (c)
> 2003-2008 Fabrice Bellard

For me.. Sometimes this version works..

voodoo@work-e6400:~$ qemu-arm-static -version
qemu-arm version 1.7.0 (Debian 1.7.0+dfsg-2), Copyright (c) 2003-2008
Fabrice Bellard

depending on mode the git calls just fail..

>> I'm not 100% sure of this is qemu to blame or the network went down..
>>
>> BTW: due to issues with qemu across the board, I only run this script
>> on native arm hardware (Debian Jessie)..

btw, this should help the script atleast complete on totaly failure
qemu/git failure..

https://github.com/beagleboard/image-builder/commit/7e308e293e6d524e00406590a8cefc57d2173115

> OK I can try to build with a Ubuntu 13.10 install I have running on an SD
> card on the BBB.
>
> Linux ubuntu-armhf 3.8.13-bone30 #1 SMP Thu Nov 14 06:23:24 UTC 2013 armv7l
> armv7l armv7l GNU/Linux
>
> How do you get the image off the SD card used to build it and onto another
> SD card to test booting? I guess I'll copy it up to another computer (like
> my Ubuntu PC) and write the image to an SD card there.

Just copy it via any means, rsync/apache/etc.

When you run ./beagleboard.org_image.sh, your final image "base" image
will be under the deploy directory.

You'll also see a "ship.sh" script, this created the final 3 images i
uploaded to here:

http://rcn-ee.net/deb/testing/2014-01-10/

Normally i copy these 2 files (ship.sh/debian*-.tar) to an x86 and
then run ./ship.sh as the xz compression takes a lot of resources
before uploading..

[Charles Steinkuehler]

On 1/14/2014 2:27 PM, Robert Nelson wrote:
>
> BTW: due to issues with qemu across the board, I only run this script
> on native arm hardware (Debian Jessie)..

As a point of reference, I have run all the MachineKit builds on a
fairly potent amd64 Debian Wheezy box (16-core Opteron 6320) with the
stock Debian qemu version (1.1.2+dfsg-6a).

I haven't had any issues completing the Debian install inside the
chroot, and I even build the user-mode Xenomai tools and LinuxCNC from
source (pulled via git running in the ARM chroot environment).

So...qemu works fine here, but then I haven't tried playing with Cloud9.
I'm still running the rcn-ee_image.sh script, and building just the
MachineKit release.

Maybe I'm just lucky?!? If so, that would be a first... :)

--
Charles Steinkuehler
cha...@steinkuehler.net

[Dennis Cote]

My build on the BBB itself just failed in *exactly* the same fashion. I got the same error after starting to clone Cloud9. Also, the same follow on errors since I hadn't fixed the script as you did in your patch.

So this does not look like it is caused by qemu.

Interestingly, I think this build completed slightly faster than the x86 build. I think it was mostly because the network speed for the downloads was better. I suspect the network translation slows down the virtual machine build enough to make a difference. On the other hand it could have been the time of day or something else. Ultimately the downloads were about twice the speed on the BBB as my VM, and both eventually lead to the same ISP. 

In any case, the native build is definitely not radically slower than the PC build (as I had expected).

On Tuesday, January 14, 2014 2:09:03 PM UTC-7, RobertCNelson wrote:

For me.. Sometimes this version works..

voodoo@work-e6400:~$ qemu-arm-static -version
qemu-arm version 1.7.0 (Debian 1.7.0+dfsg-2), Copyright (c) 2003-2008
Fabrice Bellard

depending on mode the git calls just fail..


>> I'm not 100% sure of this is qemu to blame or the network went down..
>>
>> BTW: due to issues with qemu across the board, I only run this script
>> on native arm hardware (Debian Jessie)..

btw, this should help the script atleast complete on totaly failure
qemu/git failure..

https://github.com/beagleboard/image-builder/commit/7e308e293e6d524e00406590a8cefc57d2173115

How do I use this link to update the script? I'm quite new to git.

 

Just copy it via any means, rsync/apache/etc.

When you run ./beagleboard.org_image.sh, your final image "base" image
will be under the deploy directory.

You'll also see a "ship.sh" script, this created the final 3 images i
uploaded to here:

http://rcn-ee.net/deb/testing/2014-01-10/

Normally i copy these 2 files (ship.sh/debian*-.tar) to an x86 and
then run ./ship.sh as the xz compression takes a lot of resources
before uploading..

Thanks for the tips. I'll do that when I can get the build to complete. 

Dennis Cote

[Robert Nelson]

On Tue, Jan 14, 2014 at 4:06 PM, Dennis Cote <den...@harding.ca> wrote:
> My build on the BBB itself just failed in *exactly* the same fashion. I got
> the same error after starting to clone Cloud9. Also, the same follow on
> errors since I hadn't fixed the script as you did in your patch.

I think this was the real issue..

https://github.com/beagleboard/image-builder/commit/95f61a4c3c050492c252152f7fff1379c4aa50b4

I had re-ordered the pkg list to make things easier for another user,
but missed a space, so the default package set never got installed.
Running a full rerun right now and so far it looks good..

>
> So this does not look like it is caused by qemu.
>
> Interestingly, I think this build completed slightly faster than the x86
> build. I think it was mostly because the network speed for the downloads was
> better. I suspect the network translation slows down the virtual machine
> build enough to make a difference. On the other hand it could have been the
> time of day or something else. Ultimately the downloads were about twice the
> speed on the BBB as my VM, and both eventually lead to the same ISP.
>
> In any case, the native build is definitely not radically slower than the PC
> build (as I had expected).

[Dennis Cote]

On Tuesday, January 14, 2014 3:06:39 PM UTC-7, Dennis Cote wrote:

How do I use this link to update the script? I'm quite new to git.

 

Google answered that one I think. I did 

git pull 

in the directory I had cloned from github. It updated the correct file as expected.

Dennis Cote

[Robert Nelson]

OH sorry missed that.. Yeah, just "git pull" we don't normally bother
to create any branches/etc this repo is pretty linear..

Just a follow on my previous message, it just finished creating the
image fine on my native quad/a9.. So you shouldn't have any more
issues with the base script..

[John Syn]

Hi Robert,

What is the quad/a9?

BTW, have you done any work on OMAP5?

Regards,
John

>
>Regards,
>
>--
>Robert Nelson
>http://www.rcn-ee.com/
>

>--
>For more options, visit http://beagleboard.org/discuss
>---
>You received this message because you are subscribed to the Google Groups
>"BeagleBoard" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to beagleboard...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.

[Robert Nelson]

> Hi Robert,
>
> What is the quad/a9?

It's just the imx6 1Ghz/quad/2GB/sata system from boundary devices..
It's nice and fast when doing native builds. (such as chromium)

> BTW, have you done any work on OMAP5?

I have one.. It doesn't really like to boot at the moment. There's a
few nice git repo's queued for v3.14-rc0 so i'll be revisiting it a
week or two.

[John Syn]

On 1/14/14, 7:32 PM, "Robert Nelson" <robert...@gmail.com> wrote:

>> Hi Robert,
>>
>> What is the quad/a9?
>
>It's just the imx6 1Ghz/quad/2GB/sata system from boundary devices..
>It's nice and fast when doing native builds. (such as chromium)

Nice.

>
>> BTW, have you done any work on OMAP5?
>
>I have one.. It doesn't really like to boot at the moment. There's a
>few nice git repo's queued for v3.14-rc0 so i'll be revisiting it a
>week or two.

Hi Robert,

I have one also. I just need the time work on it. I¹ve managed run code on
the dual CortexM4¹s using GEL scripts and next I want to get RPMSG
working. My goal is to do all real-time I/O, comms and time sync via the
CortexM4s and leave the CortexA15s to run apps.

Regards,
John

>
>Regards,
>
>--
>Robert Nelson
>http://www.rcn-ee.com/
>

[Dennis Cote]

On Tuesday, January 14, 2014 3:14:29 PM UTC-7, RobertCNelson wrote:

I think this was the real issue..

https://github.com/beagleboard/image-builder/commit/95f61a4c3c050492c252152f7fff1379c4aa50b4

Running a full rerun right now and so far it looks good..

Hi Robert,

I haven't had any luck yet. The native build on the BBB had a panic while building node.js. The cross build on my PC is still running (it was waiting for a password when I returned this morning).

Putty wouldn't let me copy the text from the BBB console, so had to do a screen capture. See file attached.

It looks like something went wrong with the SD card driver. I thought the root partition may have run out of space, but it had 1.9GB free after the reset. 

I am using an 8GB SD card with ubuntu 13.10. Do you know how much space is required to run the native image builder script?

Also, is there a way to restart the script without redoing all the downloads etc again?

Thanks again.

Dennis Cote

 

[Robert Nelson]

> Hi Robert,
>
> I haven't had any luck yet. The native build on the BBB had a panic while
> building node.js. The cross build on my PC is still running (it was waiting
> for a password when I returned this morning).
>
> Putty wouldn't let me copy the text from the BBB console, so had to do a
> screen capture. See file attached.

Humm, i thought we fixed that.. what kernel version on your BBB?

> It looks like something went wrong with the SD card driver. I thought the
> root partition may have run out of space, but it had 1.9GB free after the
> reset.
>
> I am using an 8GB SD card with ubuntu 13.10. Do you know how much space is
> required to run the native image builder script?

Humm, honestly I'm not sure on the "minimum" spec's. I really don't
mess around with the script, using a quad A9 with 2GB of ram and a
fast sata drive..

I know python needs 256Mb of dedicated memory when building nodejs..

The source *.tar is around 1.2G

The final image (in uncompressed form) is around 1.2G

Just started a fresh run right now, so i'll get those numbers..

> Also, is there a way to restart the script without redoing all the downloads
> etc again?

No on restarting..

But if you build alot, look at setting up "apt-cacher-ng" on a local server..

then just set the apt-proxy variable

like: https://github.com/beagleboard/image-builder/blob/master/host/rcn-ee-host.sh#L12

Then on the 2nd run every is gotten from the local cache server...

[Dennis Cote]

On Wednesday, January 15, 2014 9:27:05 AM UTC-7, RobertCNelson wrote:

Humm, i thought we fixed that.. what kernel version on your BBB?

ubuntu@ubuntu-armhf:~$ uname -a

Linux ubuntu-armhf 3.8.13-bone30 #1 SMP Thu Nov 14 06:23:24 UTC 2013 armv7l armv7l armv7l GNU/Linux

ubuntu@ubuntu-armhf:~$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 13.10

Release: 13.10

Codename: saucy

 

Humm, honestly I'm not sure on the "minimum" spec's.  I really don't
mess around with the script, using a quad A9 with 2GB of ram and a
fast sata drive..

Just started a fresh run right now, so i'll get those numbers..

OK. I'll wait until then to try the native build again.

The good news is the cross build on my PC completed successfully. So I can try using your Wheezy image next time.

 

No on restarting..

But if you build alot, look at setting up "apt-cacher-ng" on a local server..

then just set the apt-proxy variable

like: https://github.com/beagleboard/image-builder/blob/master/host/rcn-ee-host.sh#L12

Then on the 2nd run every is gotten from the local cache server...

Again, thanks for the tips. 

Dennis Cote 

[Robert Nelson]

> ubuntu@ubuntu-armhf:~$ uname -a
> Linux ubuntu-armhf 3.8.13-bone30 #1 SMP Thu Nov 14 06:23:24 UTC 2013 armv7l
> armv7l armv7l GNU/Linux
> ubuntu@ubuntu-armhf:~$ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 13.10
> Release: 13.10
> Codename: saucy

Hum, that should have been fine.. dpkg really stress out mmc cards...

>> Humm, honestly I'm not sure on the "minimum" spec's. I really don't
>> mess around with the script, using a quad A9 with 2GB of ram and a
>> fast sata drive..
>>
>> Just started a fresh run right now, so i'll get those numbers..
>>
>
> OK. I'll wait until then to try the native build again.
>
> The good news is the cross build on my PC completed successfully. So I can
> try using your Wheezy image next time.

2.8GB of space needed right as my sata drive took a dive in the lab.
(this drive has been abused specially with sata bring up on the
imx53/6 so it's not exactly 100% anymore..)

btw, you can also nuke the ./ignore directory as not everything gets
cleaned up before runs...

[Dennis Cote]

On Tuesday, January 14, 2014 2:09:03 PM UTC-7, RobertCNelson wrote:

When you run ./beagleboard.org_image.sh, your final image "base" image
will be under the deploy directory.

You'll also see a "ship.sh" script, this created the final 3 images i
uploaded to here:

http://rcn-ee.net/deb/testing/2014-01-10/

Normally i copy these 2 files (ship.sh/debian*-.tar) to an x86 and
then run ./ship.sh as the xz compression takes a lot of resources
before uploading..

Robert,

When my PC was done I ran the ship.sh script in the deploy directory. When I look at the files I see the following:

dennis@dennis-VirtualBox:~/BBB/image-builder/deploy$ ls -hl

total 3.1G

-rw-r--r-- 1 root   root   1.2G Jan 15 09:02 armhf-rootfs-debian-wheezy-2014-01-14-src.tar

-rw-r--r-- 1 root   root   1.7G Jan 15 09:46 BBB-eMMC-flasher-debian-7.3-2014-01-14-2gb.img

-rw-r--r-- 1 dennis dennis 254K Jan 15 09:38 BBB-eMMC-flasher-debian-7.3-2014-01-14-2gb.img.xz

-rw-r--r-- 1 dennis dennis 362M Jan 15 09:48 bone-debian-7.3-2014-01-14-4gb.img.xz

-rw-r--r-- 1 dennis dennis 300M Jan 15 09:05 debian-7.3-lxde-armhf-2014-01-14.tar.xz

-rwxr-xr-x 1 dennis dennis  689 Jan 15 09:05 ship.sh

  

I notice that the file sizes are different than your website which has:

      BBB-eMMC-flasher-deb..> 10-Jan-2014 10:52  358M  
      bone-debian-7.3-2014..> 10-Jan-2014 10:54  358M  
      debian-7.3-console-a..> 10-Jan-2014 10:36  302M  

My flasher image seems to be substantially smaller than yours. Too small to be correct.

My debian... file has "lxde" in the name rather than "console", and is about the same size even though lxde would imply an X11 based system as opposed to a console only system.  

My 4G bone... image file is about the same size as yours.

Where does the armhf... file come from? I tried tracing through the build scripts, but I couldn't see where it was created. 

Do you know why any of these differences exist?

BTW I noticed that the tar command in ship.sh has no "-" before the xf options. The tar help says it should be there, but perhaps it is not required. 

Thanks again.

Dennis Cote

[Robert Nelson]

On Wed, Jan 15, 2014 at 12:27 PM, Dennis Cote <den...@harding.ca> wrote:
>
>
> On Tuesday, January 14, 2014 2:09:03 PM UTC-7, RobertCNelson wrote:
>>
>> When you run ./beagleboard.org_image.sh, your final image "base" image
>> will be under the deploy directory.
>>
>> You'll also see a "ship.sh" script, this created the final 3 images i
>> uploaded to here:
>>
>> http://rcn-ee.net/deb/testing/2014-01-10/
>>
>> Normally i copy these 2 files (ship.sh/debian*-.tar) to an x86 and
>> then run ./ship.sh as the xz compression takes a lot of resources
>> before uploading..
>>
>
> Robert,
>
> When my PC was done I ran the ship.sh script in the deploy directory. When I
> look at the files I see the following:
>
> dennis@dennis-VirtualBox:~/BBB/image-builder/deploy$ ls -hl
> total 3.1G
> -rw-r--r-- 1 root root 1.2G Jan 15 09:02
> armhf-rootfs-debian-wheezy-2014-01-14-src.tar
> -rw-r--r-- 1 root root 1.7G Jan 15 09:46

> BBB-eMMC-flasher-debian-7.3-2014-01-14-2gb.img
> -rw-r--r-- 1 dennis dennis 254K Jan 15 09:38
> BBB-eMMC-flasher-debian-7.3-2014-01-14-2gb.img.xz
> -rw-r--r-- 1 dennis dennis 362M Jan 15 09:48

It looks like xz just died, it should remove
"BBB-eMMC-flasher-debian-7.3-2014-01-14-2gb.img" when done..

> bone-debian-7.3-2014-01-14-4gb.img.xz
> -rw-r--r-- 1 dennis dennis 300M Jan 15 09:05
> debian-7.3-lxde-armhf-2014-01-14.tar.xz
> -rwxr-xr-x 1 dennis dennis 689 Jan 15 09:05 ship.sh
>
> I notice that the file sizes are different than your website which has:
>
> BBB-eMMC-flasher-deb..> 10-Jan-2014 10:52 358M
> bone-debian-7.3-2014..> 10-Jan-2014 10:54 358M
> debian-7.3-console-a..> 10-Jan-2014 10:36 302M
>
>
> My flasher image seems to be substantially smaller than yours. Too small to
> be correct.
>
> My debian... file has "lxde" in the name rather than "console", and is about
> the same size even though lxde would imply an X11 based system as opposed to
> a console only system.

I changed that this week, seemed kinda silly calling it "console" when
obviously an 'lxde' image..

https://github.com/beagleboard/image-builder/commit/07af6e7249f224cbf96145f686c40efab516ece3

>
> My 4G bone... image file is about the same size as yours.
>
> Where does the armhf... file come from? I tried tracing through the build
> scripts, but I couldn't see where it was created.

The "armhf" is the name of the debian port we are using..

https://wiki.debian.org/ArmHardFloatPort

Default gcc compile settings: --with-arch=armv7-a --with-fpu=vfpv3-d16
--with-float=hard --with-mode=thumb

This script also supports the older "armel" and plans for the "arm64"
archs.. You could do x86, but that would be silly..

> Do you know why any of these differences exist?
>
> BTW I noticed that the tar command in ship.sh has no "-" before the xf
> options. The tar help says it should be there, but perhaps it is not
> required.

The ship.sh file came out of me being lazy about a week ago, it's not
really tested beyond debian jessie..

[Dennis Cote]

On Wednesday, January 15, 2014 11:27:10 AM UTC-7, Dennis Cote wrote:

When my PC was done I ran the ship.sh script in the deploy directory. When I look at the files I see the following:

dennis@dennis-VirtualBox:~/BBB/image-builder/deploy$ ls -hl

total 3.1G

-rw-r--r-- 1 root   root   1.2G Jan 15 09:02 armhf-rootfs-debian-wheezy-2014-01-14-src.tar

-rw-r--r-- 1 root   root   1.7G Jan 15 09:46 BBB-eMMC-flasher-debian-7.3-2014-01-14-2gb.img

-rw-r--r-- 1 dennis dennis 254K Jan 15 09:38 BBB-eMMC-flasher-debian-7.3-2014-01-14-2gb.img.xz

-rw-r--r-- 1 dennis dennis 362M Jan 15 09:48 bone-debian-7.3-2014-01-14-4gb.img.xz

-rw-r--r-- 1 dennis dennis 300M Jan 15 09:05 debian-7.3-lxde-armhf-2014-01-14.tar.xz

-rwxr-xr-x 1 dennis dennis  689 Jan 15 09:05 ship.sh

  

I noticed that the timestamp of the compressed flasher image was before the timestamp of the uncompressed image. This file was probably created the first time I ran the ship.sh script. It did some operations and then told be I was missing some dependencies. I installed the missing tools and then reran the script. which produced they files above.

It seems like I should be able to delete the output files and rerun the script, but it is not clear to me which files are the outputs at this point. Should I delete the two BBB... flasher image files and the *-4gb.img.xz file, and uncompress the debian...tar.xz file to get back to square one before running the ship.sh script again?

Thanks 

Dennis Cote  

[Robert Nelson]

This will work around that.. ;)

https://github.com/beagleboard/image-builder/commit/d856d07c209f6af4022c95fd8fecf9d1f86514c6

essentially leaving "debian-7.3-lxde-armhf-2014-01-14.tar.xz" as is.
As it is now already compressed after the first run and building all
the final images off it..

[Dennis Cote]

On Wednesday, January 15, 2014 11:38:20 AM UTC-7, RobertCNelson wrote:

It looks like xz just died, it should remove
"BBB-eMMC-flasher-debian-7.3-2014-01-14-2gb.img" when done..

That explains what happened to the debian-7.3-lxde-armhf-2014-01-14.tar file that ship.sh is looking for to start with.

 

I changed that this week, seemed kinda silly calling it "console" when
obviously an 'lxde' image..

Good to know. That difference is explained away. 

 

> Where does the armhf... file come from? I tried tracing through the build
> scripts, but I couldn't see where it was created.

The "armhf" is the name of the debian port we are using..

https://wiki.debian.org/ArmHardFloatPort

Yes, I understand what "armhf" means (after I looked it up a few days ago), but I was asking where does the file armhf-rootfs-debian-wheezy-2014-01-14-src.tar come from? I couldn't see where it was created in the scripts.

Thanks again.

Dennis Cote

[Robert Nelson]

> Yes, I understand what "armhf" means (after I looked it up a few days ago),
> but I was asking where does the file
> armhf-rootfs-debian-wheezy-2014-01-14-src.tar come from? I couldn't see
> where it was created in the scripts.

Oh that, it gets dumped out here:
https://github.com/beagleboard/image-builder/blob/master/scripts/chroot.sh#L719

just un-comment the "chroot_ENABLE_DEB_SRC=enable" line..

[Dennis Cote]

On Wednesday, January 15, 2014 11:54:17 AM UTC-7, RobertCNelson wrote:

This will work around that.. ;)

https://github.com/beagleboard/image-builder/commit/d856d07c209f6af4022c95fd8fecf9d1f86514c6

Thanks, that did it (after a little copying and editing to update my existing ship.sh script without rebuilding everything).

I now have what appear to be a complete set of image files.

Dennis Cote 

[Dennis Cote]

On Wednesday, January 15, 2014 12:15:51 PM UTC-7, RobertCNelson wrote:

Oh that, it gets dumped out here:
https://github.com/beagleboard/image-builder/blob/master/scripts/chroot.sh#L719

just un-comment the "chroot_ENABLE_DEB_SRC=enable" line..

Is it used for anything in this build, or is it simply an extraneous file that doesn't need to be produced?

Dennis Cote 

[Robert Nelson]

It serves only one purpose. If someone was to ask you.

Do you have the source to package xyz which was installed in release_xyz?

You'd be able to give it to them..

It's just a tar file with the source package for everything that was installed.

[Dennis Cote]

On Wednesday, January 15, 2014 1:06:28 PM UTC-7, RobertCNelson wrote:

It serves only one purpose. If someone was to ask you.

Do you have the source to package xyz which was installed in release_xyz?

You'd be able to give it to them..

It's just a tar file with the source package for everything that was installed.

I see, *-src.tar now its clear. :-)

Thanks again for all your help.

Dennis Cote

[Dennis Cote]

Still no joy. :-(

I am getting an error when writing the 4GB image to a 4 GB SD card.

root@dennis-VirtualBox:/home/dennis/BBB/image-builder/deploy# xz -cd bone-debian-7.3-2014-01-14-4gb.img.xz > /dev/sdd

xz: (stdout): Write error: No space left on device

I expanded the image to check its size and it comes back as 3.7GB which should fit. 

So I then unmounted the two partitions and copied the image file to the SD card (/dev/sdd) again. This also failed

root@dennis-VirtualBox:/home/dennis/BBB/image-builder/deploy# cp bone-debian-7.3-2014-01-14-4gb.img /dev/sdd

cp: writing ‘/dev/sdd’: No space left on device

cp: failed to extend ‘/dev/sdd’: No space left on device

I then removed and reinserted the SD card. It mounted the BOOT partition on /dev/sdd1, but failed to mount the rootfs partition from /dev/sdd2 with the following error:

Error mounting /dev/sdd2 at /media/dennis/rootfs: Command-line `mount -t "ext4" -o "uhelper=udisks2,nodev,nosuid" "/dev/sdd2" "/media/dennis/rootfs"' exited with non-zero exit status 32: mount: wrong fs type, bad option, bad superblock on /dev/sdd2

Next I checked the size of the SD card. 

dennis@dennis-VirtualBox:~$ cat /sys/block/sdd/size

7626752

I believe this is units of 512B blocks. This corresponds to 3.904 GB or 3.637 GiB.

The image file size is:

root@dennis-VirtualBox:/home/dennis/BBB/image-builder/deploy# ls -l bone-debian-7.3-2014-01-14-4gb.img

-rw-r--r-- 1 dennis dennis 3932160000 Jan 15 12:26 bone-debian-7.3-2014-01-14-4gb.img

This size is 3.932 GB or 3.662 GiB (i.e. the 3.7GB reported earlier).

Therefore this image is slightly too large for my nominal 4GB SD card.

Can this image be resized so that it is small enough to fit? 

I know that the armhf.com images are 2 GB. These can be copied to a larger SD card and then fdisk is used to resize the second partition to fill the SD card, and finally the filesystem is resized to match the new partition size. This seems like a better approach than arbitrarily reducing the image size to fit on undersized SD cards, though it is more complicated.

Dennis Cote

[Robert Nelson]

On Wed, Jan 15, 2014 at 5:45 PM, Dennis Cote <den...@harding.ca> wrote:
>
>
> Still no joy. :-(
>
> I am getting an error when writing the 4GB image to a 4 GB SD card.
>
> root@dennis-VirtualBox:/home/dennis/BBB/image-builder/deploy# xz -cd
> bone-debian-7.3-2014-01-14-4gb.img.xz > /dev/sdd
> xz: (stdout): Write error: No space left on device
>
> I expanded the image to check its size and it comes back as 3.7GB which
> should fit.
>
> So I then unmounted the two partitions and copied the image file to the SD
> card (/dev/sdd) again. This also failed
>
> root@dennis-VirtualBox:/home/dennis/BBB/image-builder/deploy# cp
> bone-debian-7.3-2014-01-14-4gb.img /dev/sdd
> cp: writing ‘/dev/sdd’: No space left on device
> cp: failed to extend ‘/dev/sdd’: No space left on device
>
> I then removed and reinserted the SD card. It mounted the BOOT partition on
> /dev/sdd1, but failed to mount the rootfs partition from /dev/sdd2 with the
> following error:
>
> Error mounting /dev/sdd2 at /media/dennis/rootfs: Command-line `mount -t
> "ext4" -o "uhelper=udisks2,nodev,nosuid" "/dev/sdd2" "/media/dennis/rootfs"'
> exited with non-zero exit status 32: mount: wrong fs type, bad option, bad
> superblock on /dev/sdd2
>
> Next I checked the size of the SD card.

how about just plain old:

sudo dd if=./bone-debian-7.3-2014-01-14-4gb.img of=/dev/sdd

> dennis@dennis-VirtualBox:~$ cat /sys/block/sdd/size
> 7626752
>
> I believe this is units of 512B blocks. This corresponds to 3.904 GB or
> 3.637 GiB.
>
> The image file size is:
>
> root@dennis-VirtualBox:/home/dennis/BBB/image-builder/deploy# ls -l
> bone-debian-7.3-2014-01-14-4gb.img
> -rw-r--r-- 1 dennis dennis 3932160000 Jan 15 12:26
> bone-debian-7.3-2014-01-14-4gb.img
>
> This size is 3.932 GB or 3.662 GiB (i.e. the 3.7GB reported earlier).

Other than the fact i HATE debugging VMware/VirtualBox with a
passion.. What brand are these 4GB microSD cards?

The magic number I've been using is 3750, which is working on
SanDisk/Kingston/Samsung 4GB
https://github.com/RobertCNelson/omap-image-builder/blob/master/tools/setup_sdcard.sh#L1540

>
> Therefore this image is slightly too large for my nominal 4GB SD card.
>
> Can this image be resized so that it is small enough to fit?
>
> I know that the armhf.com images are 2 GB. These can be copied to a larger
> SD card and then fdisk is used to resize the second partition to fill the SD
> card, and finally the filesystem is resized to match the new partition size.
> This seems like a better approach than arbitrarily reducing the image size
> to fit on undersized SD cards, though it is more complicated.

If you want a 2Gb image, pass the "--img" option vs the "img-4gb"
option to setup_sdcard.sh..

[William Hermans]

Whats all this talk of "debugging vmware/virtualbox" all about ? 

[Dennis Cote]

On Wednesday, January 15, 2014 5:10:59 PM UTC-7, RobertCNelson wrote:

how about just plain old:

sudo dd if=./bone-debian-7.3-2014-01-14-4gb.img of=/dev/sdd

I can try but I doubt if it will be any different than cp or xz -cd >. 

 

> dennis@dennis-VirtualBox:~$ cat /sys/block/sdd/size
> 7626752
>
> I believe this is units of 512B blocks. This corresponds to 3.904 GB or
> 3.637 GiB.
>
> The image file size is:
>
> root@dennis-VirtualBox:/home/dennis/BBB/image-builder/deploy# ls -l
> bone-debian-7.3-2014-01-14-4gb.img
> -rw-r--r-- 1 dennis dennis 3932160000 Jan 15 12:26
> bone-debian-7.3-2014-01-14-4gb.img
>
> This size is 3.932 GB or 3.662 GiB (i.e. the 3.7GB reported earlier).

Other than the fact i HATE debugging VMware/VirtualBox with a
passion.. What brand are these 4GB microSD cards?

Kingston Technology.

To rule out virtualbox's involvment I inserted the same SD card into my BBB and checked its size as reported by Angstrom linux.

 root@beaglebone:~# cat /sys/block/mmcblk1/size

7626752

As you can see it reports exactly the same size.

 

The magic number I've been using is 3750, which is working on
SanDisk/Kingston/Samsung 4GB
https://github.com/RobertCNelson/omap-image-builder/blob/master/tools/setup_sdcard.sh#L1540

Yes, 3750*1024*1024 is 3.932 GB, the size of the image file.

Reducing this to 3750 * 3.904/3.932 or 3723 would make it fit this SD card. I would suggest reducing it slightly more, to 3700 to allow for other smaller cards.

> I know that the armhf.com images are 2 GB. These can be copied to a larger
> SD card and then fdisk is used to resize the second partition to fill the SD
> card, and finally the filesystem is resized to match the new partition size.
> This seems like a better approach than arbitrarily reducing the image size
> to fit on undersized SD cards, though it is more complicated.

If you want a 2Gb image, pass the "--img" option vs the "img-4gb"
option to setup_sdcard.sh..

I think I'll do that. I will also look at creating a script to automate the expansion process so that the result automatically fills the SD card to its full capacity, regardless of its exact size.

Thanks again.

Dennis Cote 

[Robert Nelson]

>> The magic number I've been using is 3750, which is working on
>> SanDisk/Kingston/Samsung 4GB
>>
>> https://github.com/RobertCNelson/omap-image-builder/blob/master/tools/setup_sdcard.sh#L1540
>>
>
> Yes, 3750*1024*1024 is 3.932 GB, the size of the image file.
>
> Reducing this to 3750 * 3.904/3.932 or 3723 would make it fit this SD card.
> I would suggest reducing it slightly more, to 3700 to allow for other
> smaller cards.

Okay, just dropped it to 3700..

https://github.com/beagleboard/image-builder/commit/b6288e9ebbe3a688b4c9623ce3c94bfdb9e5e305

>
>> > I know that the armhf.com images are 2 GB. These can be copied to a
>> > larger
>> > SD card and then fdisk is used to resize the second partition to fill
>> > the SD
>> > card, and finally the filesystem is resized to match the new partition
>> > size.
>> > This seems like a better approach than arbitrarily reducing the image
>> > size
>> > to fit on undersized SD cards, though it is more complicated.
>>
>> If you want a 2Gb image, pass the "--img" option vs the "img-4gb"
>> option to setup_sdcard.sh..
>>
>
> I think I'll do that. I will also look at creating a script to automate the
> expansion process so that the result automatically fills the SD card to its
> full capacity, regardless of its exact size.

What i'd really like to see is a script that can be run on target to
automatically re-size it, even on bootup...

Because, if your already running linux, instead of resizing the *.img
for your microSD, you can just already call "--mmc /dev/sdX" with
setup_sdcard.sh and it'll auto partition your microSD using all the
available space.

Of course the "--img" option allows you to easily share a completed
image that doesn't need network access to pull in the extra bits

[Dennis Cote]

On Thursday, January 16, 2014 9:22:12 AM UTC-7, RobertCNelson wrote:

Okay, just dropped it to 3700..

https://github.com/beagleboard/image-builder/commit/b6288e9ebbe3a688b4c9623ce3c94bfdb9e5e305

Success... finally. :-)

It took a while to figure out ship.sh was using the setup_sdcard.sh inside the image file. I had pulled the new version into the image_builder dir, but I didn't rerun the entire build. I was just rerunning the ship.sh script. After editing the setup_sdcard.sh in the image it worked.

I have booted my BBB using the new 4GB wheezy SD card. Everything seems to be working as expected.

 

What i'd really like to see is a script that can be run on target to
automatically re-size it, even on bootup...

Because, if your already running linux, instead of resizing the *.img
for your microSD, you can just already call "--mmc /dev/sdX" with
setup_sdcard.sh and it'll auto partition your microSD using all the
available space.

Of course the "--img" option allows you to easily share a completed
image that doesn't need network access to pull in the extra bits
setup_sdcard.sh..

I was thinking more along the lines of always building a standard image the exact size needed for the BBB eMMC (i.e. 2 GB) since they are all the same size. Then using a minimal system and a compressed copy of this standard image to build the eMMC_flasher image which I think would also fit into 2 GB. The same compressed standard image could be distributed for use with a small script that would expand the image onto an SD card of any larger size (i.e. 4 GB, 8 GB, 16 GB, etc) and then expand the partition and filesystem to match the size of the SD card. 

This would work well for linux host systems, but perhaps not so well for Windows or OS X hosts. So perhaps the larger images will still be needed for these systems.

I noticed that the ship.sh script seems to build each image from scratch rather than basing one image on the other (i.e. it calls setup_sdcard.sh once for each image). The only thing I saw that is done for the eMMC_flasher image is the creation of a file called flash-eMMC.txt. It wasn't obvious to me how that file is used. I assume the startup code looks for that file and runs the flashing commands if it's found as the system boots the flasher image. 

Are there other differences between a normal image (say the 4GB image) and a eMMC_flasher image as they are currently being built?

Dennis Cote

[Robert Nelson]

> Success... finally. :-)

That's good..

> It took a while to figure out ship.sh was using the setup_sdcard.sh inside
> the image file. I had pulled the new version into the image_builder dir, but
> I didn't rerun the entire build. I was just rerunning the ship.sh script.
> After editing the setup_sdcard.sh in the image it worked.

Yeah, it's just years of bolt-on's.. I originally wrote
setup_sdcard.sh to automate the task of installing the output of
"project-rootstock". Then i needed to manage what options we called
"project-rootstock" with. Eventually re-writing 'project-rootstock'..
Then finally adding the ship.sh file as i couldn't remember what i
shipped the month before.

It was much easier when i just pushed out:
"debian-7.3-lxde-armhf-2014-01-16.tar.xz" and made everyone run
"setup_sdcard.sh"...

>> What i'd really like to see is a script that can be run on target to
>> automatically re-size it, even on bootup...
>>
>> Because, if your already running linux, instead of resizing the *.img
>> for your microSD, you can just already call "--mmc /dev/sdX" with
>> setup_sdcard.sh and it'll auto partition your microSD using all the
>> available space.
>>
>> Of course the "--img" option allows you to easily share a completed
>> image that doesn't need network access to pull in the extra bits
>> setup_sdcard.sh..
>
>
> I was thinking more along the lines of always building a standard image the
> exact size needed for the BBB eMMC (i.e. 2 GB) since they are all the same
> size. Then using a minimal system and a compressed copy of this standard
> image to build the eMMC_flasher image which I think would also fit into 2
> GB. The same compressed standard image could be distributed for use with a
> small script that would expand the image onto an SD card of any larger size
> (i.e. 4 GB, 8 GB, 16 GB, etc) and then expand the partition and filesystem
> to match the size of the SD card.

If you consider "armhf-rootfs-debian-wheezy.tar" to be the "minimal
system" you've just described what "setup_sdcard.sh --mmc /dev/sdX"
does already.. But i'm not going to stop you from writing a script to
dump the *.img and then expanding it..

>
> I noticed that the ship.sh script seems to build each image from scratch
> rather than basing one image on the other (i.e. it calls setup_sdcard.sh
> once for each image). The only thing I saw that is done for the eMMC_flasher
> image is the creation of a file called flash-eMMC.txt. It wasn't obvious to
> me how that file is used. I assume the startup code looks for that file and
> runs the flashing commands if it's found as the system boots the flasher
> image.
>
> Are there other differences between a normal image (say the 4GB image) and a
> eMMC_flasher image as they are currently being built?

I've limited the flasher to 2GB in size.. No sense in flashing more then 2GB's..

The extra flash file triggers this call in the board init script:

https://github.com/RobertCNelson/boot-scripts/blob/master/boot/am335x_evm.sh#L76