New firmware based on Linux-4.4, with bugs fixed

[Neil Brown]

As you might know, I've been working toward full mainline support for thje GnuBee in Linux.  We aren't quite there yet and it might take a while.  So I thought a diversion was in order.

The Linux-4.4 kernel that is available is quite good but has a few bugs.  Fixing them and creating new firmware images using LEDE or openwrt or libreCMC is more complex than seems necessary.  For those of us using a Debian root filesystem, we don't need most of the things that those packages provide.

So I've taken a simpler approach, building on things I put together for mainline kernel testing.  I have a simple initramfs which is created by copying files out of a Debian install (rather than building everything from scratch).  I can build this together with a new kernel in about 5 minutes.  I'll be using this until I'm happy with the status of mainline support.

The kernel can be found at https://github.com/neilbrown/linux/tree/gnubee/v4.4
It contains v4.4.131 plus all the relevant patches from LEDE plus fixes for

- occasional network lockups (ping works, tcp doesn't)
- /sbin/halt - it now halts instead of rebooting
- sd card works
- device tree support for both 880MHz and 900MHz versions.

The initramfs tools can be found at https://github.com/neilbrown/gnubee-tools
The README.md contains an introduction and build instructions. You can build entirely on the GnuBee in a little over 100 minutes, once you get the git trees in place.

I plan to add a number of other features to what it there, such as making a way to SSH in to the gnubee while in the initramfs state so if there is something wrong with your root filesystem you can still get in and look around - or if it is encrypted, you can get in and enter the password.

You can get pre-built images at http://neil.brown.name/gnubee/ one for 880MHz and one for 900MHz.  I don't have anything specificall taylored for the GnuBee-pc2 - the pc1 images should work except for the extra network port.

I'll probably try to get prebuilt images into gnubee-git/GnuBee_Docs at some stage, maybe when its a bit more complete and tested.

Please test it out if you feel inclined - let me know how it goes.
It is only expected to work if you have an ext4 Debian root filesystem called GNUBEE-ROOT.

[Brett Neumeier]

On Sat, May 12, 2018 at 3:32 AM, Neil Brown <ne...@brown.name> wrote:

The Linux-4.4 kernel that is available is quite good but has a few bugs.  Fixing them and creating new firmware images using LEDE or openwrt or libreCMC is more complex than seems necessary.  For those of us using a Debian root filesystem, we don't need most of the things that those packages provide.

​ [...]​

Please test it out if you feel inclined - let me know how it goes.
It is only expected to work if you have an ext4 Debian root filesystem called GNUBEE-ROOT.

​Hi Neil,

Thank you for all of your efforts! This is great.

I built​ a new firmware using your 4.4.131 kernel and gnubee-tools repositories, as in the README. I had issues with the mkinitramfs script -- it refused to find the "findfs" program, because that's in /sbin and I didn't have /sbin on my PATH -- so I tweaked the mkinitramfs to add /sbin and /usr/sbin to PATH before trying to find the required/optional programs. I also noticed that after reboot, my mdraid arrays didn't have the correct names, so I also tweaked mkinitramfs to copy the mdadm.conf file, if there is one, into the initramfs. The result works fine on both GnuBee PC1 and PC2; the only infelicity on PC2 is that the third ethernet port isn't recognized. The massive clock drift I've seen on other kernels has been corrected, which is a relief!

I also tweaked the README in a couple of places, basically just proofreading.

The linux kernel build had a bunch of irritating compilation warning messages, so I cleaned those up as well. There is still one warning message from a Kconfig file; I don't understand how the kernel configuration stuff works well enough to silence that one.

Patches are attached, for your consideration.

​

​Cheers,

Brett​

--

Brett Neumeier (bneu...@gmail.com)

[NeilBrown]

Thanks so much.

I've applied all of your changes except the mdadm.conf addition.
I'm not sure that is the right (or best) solution for the problem.

I think there are two possible name problems. Which one are you
hitting.
Are you getting /dev/md127 where you wanted /dev/md0 (or similar)
or are you getting /dev/md/hostname:arrayname when you wanted
/dev/md/arrayname ??

Also, is your root filesystem on the md array?

Thanks,
NeilBrown

[Brett Neumeier]

On Tue, May 15, 2018 at 4:37 PM, NeilBrown <ne...@brown.name> wrote:

>> Patches are attached, for your consideration.

Thanks so much.

I've applied all of your changes except the mdadm.conf addition.
I'm not sure that is the right (or best) solution for the problem.

I think there are two possible name problems.  Which one are you
hitting.
Are you getting /dev/md127 where you wanted /dev/md0 (or similar)

​Yes, ​the thing where at the time the array is assembled the system doesn't know that they are local arrays and gives them names like /dev/md127 and /dev/md126. There are two ways to fix that, that I know of: either set the hostname of the system to the one baked into the md array before assembling it, or provide an mdadm.conf that specifies the arrays. I have no real preference; I copied mdadm.conf into the initramfs because it was easy and I knew it would work. (It's also what the "dracut" program in redhat does, if I recall correctly.)

Also, is your root filesystem on the md array?

​No, but I was thinking of changing that.​

(​My real goal is to have the firmware's job be to locate the real kernel on a /boot partition and then kexec into it. But I haven't even gotten to the point of putting kexec tools into the initramfs, or compiling the kernel with kexec support.)​

--

Brett Neumeier (bneu...@gmail.com)

[NeilBrown]

On Tue, May 15 2018, Brett Neumeier wrote:

>> I think there are two possible name problems. Which one are you
>> hitting.
>> Are you getting /dev/md127 where you wanted /dev/md0 (or similar)
>>
>
> ​Yes, ​the thing where at the time the array is assembled the system
> doesn't know that they are local arrays and gives them names like
> /dev/md127 and /dev/md126. There are two ways to fix that, that I know of:
> either set the hostname of the system to the one baked into the md array
> before assembling it, or provide an mdadm.conf that specifies the arrays. I

There is another way - --homehost=any will assume that all arrays found
are local. I don't much like any of them.
I'm probably just paranoid, but I imagine a situation where you have two
similar machines and one dies, so you pull the drives out of the dead one
and plug them into the working machine, and boot it.
I want to be sure that the "foreign" arrays don't get assembled as
though they are the local arrays.
Part of ensuring this works right is to have a strict ordering of
devices - so the *first* GNUBEE-ROOT is used as the root filesysytem.
That doesn't work so well for md arrays - hence the reliance on host
names.
Burning the hostname into the initramfs on flash is certainly an
option, but it is currently a little to hard to do that. People
shouldn't *have* to compile their own kernel.
One possibility is to add a host name to the u-boot environment.
Then the initramfs could run
fw_printenv hostname
to get a hostname to pass to mdadm.
We would need a way to ensure
fw_setenv host `hostname`
were run at some suitable time. These tools are in u-boot-tools.
The kernel needs a small tweak for this to work as the u-boot-env
partitions is currently read-only.

> have no real preference; I copied mdadm.conf into the initramfs because it
> was easy and I knew it would work. (It's also what the "dracut" program in
> redhat does, if I recall correctly.)
>
> Also, is your root filesystem on the md array?
>>
>
> ​No, but I was thinking of changing that.​

For that case I can easily fix it.
Before
umount /proc /sys /dev
in the init script run
mdadm -Ss
rm -dr /dev/md/

that leaves it completely up to the booting system to start the arrays.

Thanks,
NeilBrown

[Damon Lane]

Thanks Neil! I am using the 900 MHz image and everything seemed as it was before except shutdown/halt work now. And it may be that bootup is faster and more RAM is free, but didn't pay enough attention before to know for sure. I was previously using the "2017-11-28 Debian firmware," librecmc-ramips-mt7621-gb-pc1-squashfs-sysupgrade_2017-11-28.bin

[Tim DuBois]

Hi Niel,

Is it at all possible to set up NIC bonding with this kernel? `modprobe bonding` tells me the module is missing. I've only started to configure my GnuBee now, so I'm not even sure if this is something that anyone does generally -- but it seems like a useful configuration.

[Neil Brown]

Is it at all possible to set up NIC bonding with this kernel?

Sure, you just need to add CONFIG_BONDING=m to the kernel .config file, and rebuild.
I've pushed out an update to gnubee-tool which adds the config option.
I'll upload a new firmware to  http://neil.brown.name/gnubee/

[Damon Lane]

Thanks Neil, it seems like you are the only one still working on this project. I appreciate the updates and the instructions and tools so we can add different options or features. Anyway, I had successfully been using your earlier firmware, but wanted to add the usb sound driver and access to /proc/config. I had tried to rebuild the firmware following your instructions under gnubee-tools. In between the two lines from your instructions below, I did "make menuconfig" to add the additional kernel options I wanted, and then I did gbmake because I assumed that takes the place of make, make modules_install and the other commands for a typical kernel rebuild on a pc.

cd linux
../gnubee-tools/scripts/gbmake firmware gbpc1-900-4.4

However, when I reboot to the firmware I built, or actually now even to the firmware you provided, I don't get the features I added and ssh access is gone. Looking at services status, ssh is up, but iptables complains about a module. Looking at dmesg, I see that the /lib/modules you mentioned for first boot didn't mount:

[ 16.774088] Partition GNUBEE-ROOT found. Starting...
mount: mounting tmpfs on /mnt/root/lib/modules failed: No such file or directory
cp: can't create directory '/mnt/root/lib/modules/': Read-only file system
cp: can't create '/mnt/root/lib/modules': Read-only file system
cp: can't create '/mnt/root/lib/modules/': Is a directory

and some, but not all modules are available to the system. I created the folders mentioned in dmesg and rebooted, but the same messages appeared. I read that this type of thing could be a filesystem problem, but I don't see any issue when I boot with the 11-28-2017 firmware from the project.

Incidentally, I think you added usb sound modules in between these versions, but I think I need one that is one more step down in the menus.

Am I doing something wrong building the firmware? Is there another step to prepare for /lib/modules?

As a side note, earlier I had mentioned more RAM being available, but through some checks with "free -m" during a reinstallation process, it appears that this firmware does help the free RAM a bit, but the big difference is how much less RAM Debian Stretch uses than Jessie. With Stretch and Neil's firmware, about 20 MB is used after a fresh boot. Other combinations were 60-90.

[Zenaan Harkness]

On Sat, Jul 28, 2018 at 07:06:57AM -0700, Damon Lane wrote:
> Thanks Neil, it seems like you are the only one still working on this
> project. I appreciate the updates and the instructions and tools so we can
> add different options or features. Anyway, I had successfully been using

Ack. Thanks Neil - truly awesome to steadily get the GnuBee support
upstreamed.

I am looking forward to getting involved in some way - waiting on
a few GnuBees to arrive (could be awaiting pickup). I used to
configure my own Linux kernels regularly, but that was a
few years ago now.

They should be perfect to make use of a bunch of older drives lying
around.

No doubt I shall be in contact again soon :)

[Neil Brown]

Hi Damon,
 If you want to change the config options, you need to start with an appropriate default config.  You also need to specify the correct directory with "O=" which gbmake will do for you.
e.g.
> cd linux
> ../gnubee-tools/scripts/gbmake defconfig-900-4.4
> ../gnubee-tools/scripts/gbmake menuconfig
> ../gnubee-tools/scripts/gbmake firmware

There error message
> mount: mounting tmpfs on /mnt/root/lib/modules failed: No such file or directory

suggests that your GNUBEE-ROOT filesystem doesn't have a /lib/modules directory.  Can you check if /lib/modules exists on the GNUBEE-ROOT?


[Damon Lane]

Thanks again Neil!

I had based my changes on your default config by copying it from gnubee-tools/kern_config, so I think it was that middle line that I did wrong. Now that I used "gbmake menuconfig" instead of "make menuconfig", things work as expected.

My default Debian installation and upgrade to Stretch did not include /lib/modules, but once I added it, things worked as in your instructions and I was able to run keep to copy the modules.

Ok, now then, back to setting up the programs I want to use!

[Andy Zwirko]

Hi Neil,

  I was wondering if I could apply this firmware image to my existing system.  I'm running an original PC1 with Kernel 4.4.87 LEDE Reboot 17.01-SNAPSHOT r3533-e84c2c26ec / LuCI Master (git-17.267.26012-a2ea9fd).  I've got a multi disk RAID currently running with it with 4 TB of data I wish to preserve in place.  I've stumble across the network lock up issue when transferring lots of files and was hoping I could try the new firmware without too much effort / hassle.  

  So, can I apply this through the web UI firmware updater or through the boot from USB thumb drive (gnubee.bin of VFAT) and if so will I be able to restore a LEDE backup tgz file to restore my RAID, or is this not doable?  And therefore do I have any other options?

Thanks for your efforts!

andyz

[Neil Brown]

If you have a root filesystem called GNUBEE-ROOT, then this firmware should work for you.  The RAID array shouldn't be affected.
If you don't, then you cannot use this firmware without getting hold of one.
Hope that helps.

[Dan Nowlin]

Hey Neil.  I am running a PC1 with librecmc-ramips-mt7621-gb-pc1-

squashfs-sysupgrade_2017-11-28 firmware running jessie and omv.  I would like to run your version with the updates but can not find where to download a bin file.  Can you point me in the right direction?  Thank YOU for all of your work.

[Neil Brown]

Sorry, I've been out of the loop for a while.

You should be able to access

http://neil.brown.name/gnubee/

again now. It has some firmware for you to try.

[sec.rese...@gmail.com]

Thanks for this Neil. I have a comment and follow up question.

There seems to be something wrong in the ordering of the targets with running gbmake firmware. I had to run gbmake initramfs firmware in order for the initramfs to be generated otherwise the linux/usr/Makefile was failing on rnning scripts/gen_initramfs_list.sh saying initramfs not found. I didn't try and debug the issue too much but just thought you should know.

Secondly my question is, is there anything in the kernel specifically for debian/systemd. I was thinking of build a void linux image which doesn't use systemd and using your kernel. Any pointers you can give are appreciated.

Thanks!

[Neil Brown]

I thought I noticed something funny with the ordering too - I'll look into it.

The kernel just has stuff in it that was needed for different reasons.

I don't think there is anything specific to Debian.

I think the only reason that cgroups was included was because systemd needs it.  So if you won't be using systemd, then you might not need cgroups.

[sec.rese...@gmail.com]

The ordering could be my fault. I had an incorrectly setup config file which resulted in an error. After fixing the file and cleaning up I reran the script without any problems.

Perhaps for added safety add a 'clean' target which get rids of built image + generated files, so it's easier to clean up if desired.

Secondly is it still true that the GNUBEE-ROOT needs to be on a hard disk and can't be on a sdcard ? The wiki mentions this but I was not sure if this is still the case.

Thanks

[Todd Adam]

I have found network is slow. Rsyncing from another machine I get around 10 MB/s with the original firmware I get average of 140MB/s. Any idea why.

[Neil Brown]

I have GNUBEE-ROOT on an Sd card - it works fine.  There was a bug in some kernels that caused problems, but that is fixed in my 4.4 kernel.

[Neil Brown]

Sorry, no idea. I can confirm that I get around 10MB/s too.  I vaguely remember looking into speed issues some months ago, I don't remember where I got to.  I might have another look, but cannot say when.

Thanks for highlighting this.

[Neil Brown]

I've been looking at this a bit more.

With iperf3, I measure about 930Mbit/sec with my 4.4 kernel (quite a bit less with the mainline kernel).

With scp, I get 10MB/sec, which is about 80Mbit/set.

With wget, I get 99.8MB/sec, which is at least 800Mbit/sec.

So it looks like ssh is causing the slowdown - maybe the encryption is too slow.

I assume your rsync is running over ssh?

When you say "original firmware" do you mean the 3.0 kernel?  Do you have an exact version number?

On Tuesday, December 4, 2018 at 7:44:17 AM UTC+11, Todd Adam wrote:

[Todd Adam]

No it was with the librecmc firmware.  I get the same results as you. Rsync with rsh instead of ssh gets me to the low 20's so it seems it is the encryption that is slowing the transfer down.  To bad they removed the lower ciphers from ssh.  But since it is an internal network rsh should work for me.

[Neil Brown]

I've pushed out a small update to my 4.4 firmware which contains three changes worth mentioning.

1. Update to 4.4.169
2. Kernel now detects CPU speed, so there is just one firmware image for both 880MHz and 900MHz versions
3. Firmware includes debootstrap so it can be used to create a Debian root filesystem - use the "config" script.

A pre-built image is in http://neil.brown.name/gnubee, and the kernel and gnubee-tools code is in my (neilbrown) github area.

(I used this firmware to build a new root filesystem, then followed the README to build a new firmware in that filesystem - It took about 110 minutes to build.  This is the image that I uploaded).

[Anton Feldmann]

Hello Neil,

Thanks for that awesome firmware. I had to change my network scripts to get the network run. All files are in /etc/network/interfaces.d/{02-eth0,02-eth0-1,02-eth0-2}

02-eth0:
   allow-hotplug eth0
   iface eth0 inet dhcp

02-eth0-1:
   allow-hotplug eth0.1
   iface eth0. inet dhcp

02-eth0-2:
   allow-hotplug eth0.2
   iface eth0.2 inet dhcp

I could not make the eth0.1 static. I am working on that. 

[Neil Brown]

Just a heads-up.  I've uploaded a new firmware to http://neil.brown.name/gnubee - see the README in that directory.

Headline improvement is that you can use it to install Debian on a fresh gnubee, even if you don't have a serial cable.  You can ssh into the initramfs shell and go from there.

[Luke Picciau]

This is super cool, I'll try this out today and let you know how it goes.

On 5/2/19 11:00 am, Neil Brown wrote:

Just a heads-up.  I've uploaded a new firmware to http://neil.brown.name/gnubee - see the README in that directory.

Headline improvement is that you can use it to install Debian on a fresh gnubee, even if you don't have a serial cable.  You can ssh into the initramfs shell and go from there.

--
You received this message because you are subscribed to the Google Groups "GnuBee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gnubee+un...@googlegroups.com.
To post to this group, send email to gnu...@googlegroups.com.
Visit this group at https://groups.google.com/group/gnubee.
To view this discussion on the web visit https://groups.google.com/d/msgid/gnubee/f2845f08-3cd2-42db-95da-5ad720a96d40%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luke Picciau]

I followed the readme with a gnubee pc2 and when I booted it up with the button pressed I see a message pop up on my desktop saying "Activation of network failed". This is on Fedora with gnome. Tested both the black and blue ports connected to my desktop and got the same error. Attempting to SSH in fails.

On 5/2/19 11:00 am, Neil Brown wrote:

Just a heads-up.  I've uploaded a new firmware to http://neil.brown.name/gnubee - see the README in that directory.

Headline improvement is that you can use it to install Debian on a fresh gnubee, even if you don't have a serial cable.  You can ssh into the initramfs shell and go from there.

[Neil Brown]

Thanks for testing.

If you plug into the black port, you need to manually configure the network on your computer to use a 192.168.10.XX address, then connect to 192.168.10.1.

If you plug into the blue port, you need to have a dhcp server on the network, and need to find out what IP address was assigned.

I'll arrange for the blue port to run a dhcp server, which should make it possible to connect with no manual configuration.