Kernel-hacking workflow with RobertCNelson's linux-dev

[ChrisSchuku]

Hi all,

I used to tweak and compile the kernel for my beaglebone in a very old fashioned way I guess. I downloaded the source, tweaked it, compiled them (make .... uImage) and copied the generated uImage over to the boot partition of the SD card and was good to go.

Yesterday I tried to do it the "GIT way" and got stuck pretty bad. Using Robert C Nelson's linux-dev repo I can compile and deploy his kernel, but I have no clue how I make my changes to kernel sources permanent.

Here's what I did:

1) cloned RobertCNelson's linux-dev repo 

2) checked out the am33x-v3.2 branch

3) cloned the linux-stable.git

4) edited the system.sh file in linux-dev/ defining the cross compiler, LINUX_GIT, ZRELADDR and MMC

5) executed build_kernel.sh -> KERNEL directory is built

6) edited the kernel sources in the generated KERNEL directory in linux-dev

7) added and committed my changes inside the KERNEL directory

8) executed the build_kernel.sh script

Obviously this script applies all patches from the linux-dev repo to the stable kernel and thus generates new kernel sources - overwriting all changes a just made.

Can someone outline the intended workflow please? I figure that this is a rooky question, but I really couldn't figure it out.

Thanks,

Chris

[Robert Nelson]

On Thu, Oct 18, 2012 at 10:43 AM, ChrisSchuku <abus...@gmail.com> wrote:
> Hi all,
>
> I used to tweak and compile the kernel for my beaglebone in a very old
> fashioned way I guess. I downloaded the source, tweaked it, compiled them
> (make .... uImage) and copied the generated uImage over to the boot
> partition of the SD card and was good to go.
>
> Yesterday I tried to do it the "GIT way" and got stuck pretty bad. Using
> Robert C Nelson's linux-dev repo I can compile and deploy his kernel, but I
> have no clue how I make my changes to kernel sources permanent.
>
> Here's what I did:
>
> 1) cloned RobertCNelson's linux-dev repo
> 2) checked out the am33x-v3.2 branch
> 3) cloned the linux-stable.git
> 4) edited the system.sh file in linux-dev/ defining the cross compiler,
> LINUX_GIT, ZRELADDR and MMC
> 5) executed build_kernel.sh -> KERNEL directory is built
> 6) edited the kernel sources in the generated KERNEL directory in linux-dev
> 7) added and committed my changes inside the KERNEL directory

> 8) executed the build_kernel.sh script

Step 8 was your mistake.. Run "./tools/rebuild.sh" instead....
"./build_kernel.sh" will do everything possible to nuke the KERNEL
directory and setup the tree just as specified in patch.sh..
"./tools/rebuild.sh" will only 'rebuild' what is in the KERNEL
directory, thus allowing you to work on your new patch.. (btw, before
running ./tools/rebuild.sh any changes to 'patches/defconfig' will
also be ignored and reset, if your looking to also change the config
options edit "KERNEL/.config" first or via menuconfig when
./tools/rebuild.sh is ran..)

I updated/cleaned-up the readme late last week with more details on
this so it should help users going forward..

Regards,

--
Robert Nelson
http://www.rcn-ee.com/

[meino....@gmx.de]

Robert Nelson <robert...@gmail.com> [12-10-18 18:04]:

> --
>
>

Hi,

...that means: everytime I update my copy with git pull I need to
run build_kernel.sh instead of tools/rebuild.sh to insert the updated
files (and all other) in KERNEL?

Is this correct?

Best regards,
mcc

[Robert Nelson]

> Hi,
>
> ...that means: everytime I update my copy with git pull I need to
> run build_kernel.sh instead of tools/rebuild.sh to insert the updated
> files (and all other) in KERNEL?
>
> Is this correct?

Correct... ./tools/rebuild.sh only operates inside KERNEL ignoring
everything outside..

[George Lu]

Hi Chris,

This is my workflow for making customizations:

1) cloned RobertCNelson's linux-dev repo 

2) checked out the am33x-v3.2 branch

3) cloned the linux-stable.git

4) edited the system.sh file in linux-dev/ defining the cross compiler, LINUX_GIT, ZRELADDR and MMC

5) executed build_kernel.sh -> KERNEL directory is built

6) create a new local branch that contains your customizations
cd KERNEL
git branch #tell you what branch you are in, let's say it is v3.2-psp24
git branch chris-v3.2-psp24 #create new branch with the name chris-v3.2-psp24
git checkout chris-v3.2-psp24
7) edit the kernel source files, for example:
vi arch/arm/mach-omap2/board-am335xevm.c
8) rebuild kernel
in linux-dev, run $ tools/rebuild_deb.sh

9) if the customization is good, commit the changes
(in KERNEL directory)
git status #tell you which files you have changed
git commit -a #commit those changes with a 1-liner to say what you have done
git log #verify that your latest commit is there

At this point I would also create a patch (in the dir above linux-dev) of my customizations. This is useful for sharing with collaborators or replaying the changes on a new git repo.

10) create a patch for future kernels
git format-patch v3.2-psp24 --stdout > ../../mycustom.patch24

For example, when Robert releases psp25:
11) in linux-dev, $ git pull
12) build_kernel.sh to get the "off-the-shelf" kernel image built
13) create a local branch for psp25 in KERNEL
cd KERNEL
git branch chris-v3.2-psp25
git checkout chris-v3.2-psp25
14) apply your customization patch
patch -p1 < ../../mycustom.patch24
15) rebuild kernel
in linux-dev, run $ tools/rebuild_deb.sh

Also want to say a big "THANK YOU" to Robert for his contribution that makes Ubuntu easy to work with on beaglebone and beagleboard.

regards,
George

--

[ChrisSchuku]

Thanks a lot! That really helps me out!

However, I now have another problem that's probably to do with the .config:

When I run rebuild.sh on my modified kernel I get the error:

arch/arm/mach-versatile/core.c:42:22: fatal error: asm/leds.h: No such file or directory

When I looked up this error I found a post of yours in this group saying: 

" It looks like you forgot to copy the defconfig 

from patches/beagleboard/defconfig and save it as .config in kernel 
directory right.. "

As I'm compiling not for beagleboard but for beaglebone I looked in patches/beaglebone, but no defconfig there. So I tried all config files I found in the patches directory (current_defconfig, defconfig, ref_omap2plus_defconfig) - but not luck. 

I still get either the mach-versatile/core.c error or errors in mach-omap2/board-omap3evm.c.

So I figured that I messed up the kernel sources somehow and ran rebuild_kernel.sh again. But oddly this also terminates with the mach-versatile/core.c error!?

What is it that I'm missing?

Thanks,

Chris

[Robert Nelson]

On Fri, Oct 19, 2012 at 8:10 AM, ChrisSchuku <abus...@gmail.com> wrote:
> Thanks a lot! That really helps me out!
>
> However, I now have another problem that's probably to do with the .config:
>
> When I run rebuild.sh on my modified kernel I get the error:
> arch/arm/mach-versatile/core.c:42:22: fatal error: asm/leds.h: No such file
> or directory

"mach-versatile" ? you really messed up the .config...

> When I looked up this error I found a post of yours in this group saying:
> " It looks like you forgot to copy the defconfig
> from patches/beagleboard/defconfig and save it as .config in kernel
> directory right.. "

That's a completely different git repo... The config being used in
"linux-dev" is patches/defconfig

> As I'm compiling not for beagleboard but for beaglebone I looked in
> patches/beaglebone, but no defconfig there. So I tried all config files I
> found in the patches directory (current_defconfig, defconfig,
> ref_omap2plus_defconfig) - but not luck.

Well, "origin/master" doesn't support the Bone, as it's more of a
staging ground for my other repo "stable-kernel"..

For the Bone use the: "am33x-v3.2" or "am33x-v3.6" branches..

> I still get either the mach-versatile/core.c error or errors in
> mach-omap2/board-omap3evm.c.
>
> So I figured that I messed up the kernel sources somehow and ran
> rebuild_kernel.sh again. But oddly this also terminates with the
> mach-versatile/core.c error!?
>
> What is it that I'm missing?

Yeah, it sounds like you did..

git checkout master -f
git reset HEAD --hard
git pull

Then for the Bone:

git checkout origin/am33x-v3.2 -b tmp-am33x-v3.2
or
git checkout origin/am33x-v3.6 -b tmp-am33x-v3.6

and run "./build_kernel.sh" to populate KERNEL..

[ChrisSchuku]

I was on am33x-v3.2 but something strange must have happened to the config.

Anyway, I restored the kernel the way you suggested and applied the same modifications to the kernel as before.

Now it compiles like a charm...

Thanks very much!

[mickeyf]

Pardon my newbie-ness. I cannot find

arch/arm/mach-omap2/board-am335xevm.c

anywhere. The most closely named file I see is "arch/arm/mach-omap2/board-am3517evm.c", in either the source for stable 3.6.3 from kernel.org, or 3.2.17 source. I have done a 'git clone' of RCNelson's patches and scripts and have that available.

This relates to this post of mine from a short while back:
https://groups.google.com/forum/#!topic/beagleboard/Iyy4MF55zj4

My goal is to configure the kernel to enable the serial port using UART 4, not the default of UART 2, which will be needed for something else.

What have I misunderstood?

Thanks

[Robert Nelson]

You'll never find "arch/arm/mach-omap2/board-am335xevm.c" in the pure
mainline kernel source..

You'll find it in the v3.2.x community kernel patchset..

[mickeyf]

You'll never find "arch/arm/mach-omap2/board-am335xevm.c" in the pure
mainline kernel source..

You'll find it in the v3.2.x community kernel patchset..

Again, pardon my ignorance, but I have no idea where this is.  Looking in the obvious (to me) places, and Googling has not helped so far.

Thanks,

Mickey

[Robert Nelson]

Since you've cloned my linux-dev tree, it's under:

git checkout origin/am33x-v3.2 -b am33x-v3.2-tmp
./build_kernel.sh

(then under KERNEL directory)

or, from:

https://github.com/beagleboard/kernel

git checkout origin/beaglebone-3.2 -b beaglebone-3.2
./patch.sh

(then under kernel directory)

[mickeyf]

Since you've cloned my linux-dev tree, it's under:

git checkout origin/am33x-v3.2 -b am33x-v3.2-tmp
./build_kernel.sh

(then under KERNEL directory)

or, from:

https://github.com/beagleboard/kernel

git checkout origin/beaglebone-3.2 -b beaglebone-3.2
./patch.sh

(then under kernel directory)

Still struggling here I'm afraid.

Made sure cross compiler was Installed by running :
sudo apt-get install gcc-arm-linux-gnueabihf

Edited system.sh, pointed to my local copy of linux-src, and set CC=gcc-arm-linux-gnueabihf, as I understood from the README. If all I want to do at the moment is create a set of kernel source files to study and understand how to tweak the UARTS, I should be able to ignore ZRELADDR and MMC for now? When I am ready to actually create an image those will matter - correct?

When I try to run build_kernel.sh I get a complaint about the CC variable, and "gcc-arm-linux-gnueabihfgcc: command not found" Not sure where that gcc on the end is from.

I'm new to git and don't know it's subtleties yet. Trying to "git checkout origin/beaglebone-3.2 -b beaglebone-3.2" gets me an error message about "updating paths is incompatible with switching branches"

I'm thinking that I should start from scratch with a clean system, and that knowing the exact commands used for steps 1 through 3 (in particular) of George Lu's process above would help.

Thanks in advance to all.


 

[Robert Nelson]

On Tue, Oct 23, 2012 at 10:01 AM, mickeyf <mic...@thesweetoasis.com> wrote:
> Since you've cloned my linux-dev tree, it's under:
>>
>>
>> git checkout origin/am33x-v3.2 -b am33x-v3.2-tmp
>> ./build_kernel.sh
>>
>> (then under KERNEL directory)
>>
>> or, from:
>>
>> https://github.com/beagleboard/kernel
>>
>> git checkout origin/beaglebone-3.2 -b beaglebone-3.2
>> ./patch.sh
>>
>> (then under kernel directory)
>>
>>
>
> Still struggling here I'm afraid.
>
> Made sure cross compiler was Installed by running :
> sudo apt-get install gcc-arm-linux-gnueabihf
>
> Edited system.sh, pointed to my local copy of linux-src, and set
> CC=gcc-arm-linux-gnueabihf, as I understood from the README. If all I want

In that case you would use:
CC=arm-linux-gnueabihf-

> to do at the moment is create a set of kernel source files to study and
> understand how to tweak the UARTS, I should be able to ignore ZRELADDR and
> MMC for now? When I am ready to actually create an image those will matter -
> correct?

Correct, hence they are listed under "OPTIONAL:"

> When I try to run build_kernel.sh I get a complaint about the CC variable,
> and "gcc-arm-linux-gnueabihfgcc: command not found" Not sure where that gcc
> on the end is from.

It's just detected your CC was inproper..

> I'm new to git and don't know it's subtleties yet. Trying to "git checkout
> origin/beaglebone-3.2 -b beaglebone-3.2" gets me an error message about
> "updating paths is incompatible with switching branches"

Your not going to find that in the linux-dev repo, it's in:
https://github.com/beagleboard/kernel

[Robert Nelson]

On Tue, Oct 23, 2012 at 10:08 AM, Robert Nelson <robert...@gmail.com> wrote:
> On Tue, Oct 23, 2012 at 10:01 AM, mickeyf <mic...@thesweetoasis.com> wrote:
>> Since you've cloned my linux-dev tree, it's under:
>>>
>>>
>>> git checkout origin/am33x-v3.2 -b am33x-v3.2-tmp
>>> ./build_kernel.sh
>>>
>>> (then under KERNEL directory)
>>>
>>> or, from:
>>>
>>> https://github.com/beagleboard/kernel
>>>
>>> git checkout origin/beaglebone-3.2 -b beaglebone-3.2
>>> ./patch.sh
>>>
>>> (then under kernel directory)
>>>
>>>
>>
>> Still struggling here I'm afraid.
>>
>> Made sure cross compiler was Installed by running :
>> sudo apt-get install gcc-arm-linux-gnueabihf
>>
>> Edited system.sh, pointed to my local copy of linux-src, and set
>> CC=gcc-arm-linux-gnueabihf, as I understood from the README. If all I want
>
> In that case you would use:
> CC=arm-linux-gnueabihf-

btw: This "CC" hint is now in the README as of sometime last week.. ;)
It's a common-ish problem for new users..

https://github.com/RobertCNelson/linux-dev/blob/master/README#L33

[mickeyf]

> In that case you would use:

> CC=arm-linux-gnueabihf-

Aha. apt-get wants "gcc-arm-linux-gnueabihf" but the compiler as actually installed is arm-linux-gnueabihf-gcc, and the trailing hyphen is needed in system.sh.

Couple of speed bumps still:

I ran build_kernel.sh, and it started patch.sh which got pretty far, but then :

"fatal: Unable to create '/home/mfeldman/stable-kernel/KERNEL/.git/index.lock': File exists.
If no other git process is currently running, this probably means a
git process crashed in this repository earlier. Make sure no other git
process is running and remove the file manually to continue."

I can well believe I crashed git earlier, but there was no index.lock to be found.
Re-ran it, since there was no lock. Got as far as "Your display is too small to run Menuconfig!"
Enlarged terminal window & re-ran.
Finally ran to completion, creating ../deploy and apparently updating files in KERNEL/arch/arm/mach-omap2, but still no sign of "board-am335xevm.c".

Guessing that I still don't have the required source files, I try

git clone https://github.com/beagleboard/kernel
cd kernel
./patch.sh

Got as far as "cannot access blah..blah../kernel/export/dirs/*.patch: No such file or directory

I feel like I'm missing some basic concept here. I'm understanding (rightly or wrongly) that the main branch source files get patched to make a Beagle-specific set, but it's not 100% clear to me which files I need to start with (or if there are alternatives) or where they need to come from. I now have a couple of different sets of kernel files, none of which has the particular one I think I care about.

I'm ready to blow them away and start from square one, but I'm not sure where square one is.



 

[Robert Nelson]

On Tue, Oct 23, 2012 at 12:57 PM, mickeyf <mic...@thesweetoasis.com> wrote:
>> In that case you would use:
>>
>> > CC=arm-linux-gnueabihf-
>>
>>
> Aha. apt-get wants "gcc-arm-linux-gnueabihf" but the compiler as actually
> installed is arm-linux-gnueabihf-gcc, and the trailing hyphen is needed in
> system.sh.
>
> Couple of speed bumps still:
>
> I ran build_kernel.sh, and it started patch.sh which got pretty far, but
> then :
>
> "fatal: Unable to create
> '/home/mfeldman/stable-kernel/KERNEL/.git/index.lock': File exists.
> If no other git process is currently running, this probably means a
> git process crashed in this repository earlier. Make sure no other git
> process is running and remove the file manually to continue."
>
> I can well believe I crashed git earlier, but there was no index.lock to be
> found.
> Re-ran it, since there was no lock. Got as far as "Your display is too small
> to run Menuconfig!"
> Enlarged terminal window & re-ran.
> Finally ran to completion, creating ../deploy and apparently updating files
> in KERNEL/arch/arm/mach-omap2, but still no sign of "board-am335xevm.c".

Then your still on "master" switch to the am33x-v3.2 branch as
mentioned earlier:

git checkout origin/am33x-v3.2 -b am33x-v3.2-tmp

[mickeyf]

Then your still on "master" switch to the am33x-v3.2 branch as

mentioned earlier:

git checkout origin/am33x-v3.2 -b am33x-v3.2-tmp

This generates the same error I had earlier with this. I don't understand git enough to get around it.

"fatal: git checkout: updating paths is incompatible with switching branches.
Did you intend to checkout 'origin/am33x-v3.2' which can not be resolved as commit?"
 

[Robert Nelson]

First are you on master?
voodoo@a53t:/opt/github/linux-dev$ git branch
* master

if not:
git checkout master -f

2nd, do the branches show up? (hit tab after "origin/" to show options)
voodoo@a53t:/opt/github/linux-dev$ git checkout origin/
origin/am33x-v3.1 origin/am33x-v3.6 origin/master
origin/v2.6.39.x origin/v3.1.x
origin/am33x-v3.2 origin/HEAD origin/v2.6.38.x
origin/v3.0.x origin/v3.3.x

if not:
git pull --all
git fetch --all

then:
voodoo@a53t:/opt/github/linux-dev$ git checkout origin/am33x-v3.2 -b am33x-v3.2
Branch am33x-v3.2 set up to track remote branch am33x-v3.2 from origin.
Switched to a new branch 'am33x-v3.2'

Beyond that, i have no idea.. either you system is just old, or some
other random thing..

[mickeyf]

First are you on master?
voodoo@a53t:/opt/github/linux-dev$ git branch
* master

if not:
git checkout master -f

2nd, do the branches show up? (hit tab after "origin/" to show options)
voodoo@a53t:/opt/github/linux-dev$ git checkout origin/
origin/am33x-v3.1   origin/am33x-v3.6   origin/master
origin/v2.6.39.x    origin/v3.1.x
origin/am33x-v3.2   origin/HEAD         origin/v2.6.38.x
origin/v3.0.x       origin/v3.3.x

if not:
git pull --all
git fetch --all

then:
voodoo@a53t:/opt/github/linux-dev$ git checkout origin/am33x-v3.2 -b am33x-v3.2
Branch am33x-v3.2 set up to track remote branch am33x-v3.2 from origin.
Switched to a new branch 'am33x-v3.2'

Beyond that, i have no idea.. either you system is just old, or some
other random thing..

git branch did show that I was on master. I did a pull --all, and a fetch --all.

git checkout origin/ showed a number of branches, but am33x-v3.2 was not among them. What I did see was:

origin/3.0.x-rt2870sta   origin/v2.6.37.x         origin/v3.2-camera       origin/v3.4.x
origin/HEAD              origin/v2.6.38.x         origin/v3.2-merge        origin/v3.5.x
origin/master            origin/v2.6.39.x         origin/v3.2-spitest      origin/v3.6.x
origin/sgx_4.05.00.03    origin/v3.0.x            origin/v3.3-merge       
origin/v2.6.35.x         origin/v3.1.x            origin/v3.3.x

...just in case this offers some clues to what I'm beginning to suspect I may have initially checked out incorrectly...

 

[Robert Nelson]

Wrong git repo:

git clone git://github.com/RobertCNelson/linux-dev.git

[mickeyf]

Thank you, that did it. I think 95% of my problems were due to unfamiliarity with git. George Lu's recipe was essentially it, except for me not understanding the details.

Here's what worked, a bit more spelled out, for beginners like me:


1) Get rcn's patch scripts:

git clone git://github.com/RobertCNelson/linux-dev.git

2) Switch to newly created directory

cd linux-dev

3) Edit system.sh.sample to indicate cross compiler. We are using Ubuntu 12.04, so we want (note trailing hyphen):

CC=arm-linux-gnueabihf-

3a) Ensure compiler is available:

sudo apt-get install gcc-arm-linux-gnueabihf

3b) Possibly also to indicate local copy of linux source if it was git cloned previously:

LINUX_GIT=/home/user/linux-src/

3c) Save edited system.sh.sample as "system.sh"

4) Make sure the terminal window is at least 19 lines high and 80 chars wide. kernel configuration menu requires this, and the script will halt if this condition is not met.

5) Switch to am33x-v3.2 branch:

git checkout origin/am33x-v3.2 -b am33x-v3.2-tmp

6) Run build script. It will call the patch script. This will all take a while:

./build_kernel.sh

7) Optionally set kernel configuration when the kernel configuration menu comes up. For a production build we will likely want to eliminate many unnecessary drivers - we have no expectation of supporting audio, video, bluetooth, keyboards, cameras, ham radio...

8) When build_kernel.sh has completed the kernel source will be in KERNEL, and board-am335xevm.c (or whatever) can be edited.


Thanks again Robert for your patient help.

[Andreas]

Hi,

I changed the board-omap3beagle.c (changing i2c-bus freq) and after that I used tools/rebuild.sh but it does not rebuild the file.

Do I need to clean something first?

Best,

Andreas

Den söndagen den 25:e november 2012 kl. 18:04:11 UTC+1 skrev AlbertoLopSie:

Hi Robert,

What repository/branch should I clone if I want to build a kernel supporting 1GHz on BB-xM ?  I need to load a Debian Wheezy armhf distro over it.

Is there any pre-built image you are aware ?

Thanks !!!

Alberto