GSoC 2017 progress report

[Nikita Korobov]

Hi MINIX3! As you remember, I'm GSoC 2017 participant. Jean-Baptiste Boric and Lionel Sambuc are my great mentors. I'm trying to make the MINIX more portable and port it to the Raspberry Pi. I had finished my first month of work and would like to tell you about my results and ideas. I'm ready to answer any questions. :)

 

Firstly, I'm trying to improve existed simple port by Jean-Baptiste Boric. It includes a lot of dirty hacks and problems such as versions of kernel for the each platform. I think, that I don’t have to explain you why it isn't good idea. What I did: We decided that, platform should resolve at run-time. All platform-dependent code will be chosen dynamically by global table. For the each platform we must make the function-table and initialize global variable depending on platform where we are. All others calls of these functions we must to do by this variable.

You'll ask: How do we know where we are? It's easy on the embedded platforms like ARM. It has a device tree, which contains information about the peripheral and platform. It is chosen and passed to the kernel by the hardware, we only parse it and know all what we need. We've decided that libfdt under the BSD license is the best for these targets. The device tree information need in the early boot stage, thus I've added it in the pre_init.c.

The booting process is also changed. We had a great idea(as for our modest opinion): enable paging in the early bootstrap (in the head.S). Thus we have only the one file with unpaged code. It makes the kernel linker script, makefile and the memory structure easier. It isn't the last idea about the bootloader.
https://github.com/nekorobov/minix/commits/rpi_clean_dev

We have great plans, how to make the MINIX3 boot process general and convenient for understanding and development in the future. We thought that it would be good: to make a general bootloader with a general structure for the all architectures and the all platforms. Multiboot2.h format was very helpful for these aims. The new kernel-loader can enable paging, it allows us don't worry about paging in the kernel, because the kernel load is in the virtual memory now. The kernel-loader parse the device tree (if it exist) or the command line parameters and packs the multiboot structure by the information above. It doesn't affect on, where are we are, because the multiboot has a universal header for the each architecture. It passed through the kernel argv, where the kernel sets up all the system parameters. Furthermore, the second parameter may have the special architecture structure (with the device tree information e.g.). I have done the great part of this, but stopped because of a couple of problems, which I can't solve now. We decided to put it aside until the free time, but the start was given. :)

 

At the end of the month, I've started development the SDCard driver. The RPI has differ register set compared with the Omap.Therefore there are two mmc and emmc drivers (for each platform), but I'm trying to find ways how to make it general. Determination of the driver constants will be from the device tree.
https://github.com/nekorobov/minix/commit/03eaf9dac896a403d855892e016dc6b4dc137bc5

[Denis Obrezkov]

Hi MINIX3! As you remember, I'm GSoC 2017 participant. Jean-Baptiste Boric and Lionel Sambuc are my great mentors. I'm trying to make the MINIX more portable and port it to the Raspberry Pi. I had finished my first month of work and would like to tell you about my results and ideas. I'm ready to answer any questions. :)

 

Firstly, I'm trying to improve existed simple port by Jean-Baptiste Boric. It includes a lot of dirty hacks and problems such as versions of kernel for the each platform. I think, that I don’t have to explain you why it isn't good idea. What I did: We decided that, platform should resolve at run-time. All platform-dependent code will be chosen dynamically by global table. For the each platform we must make the function-table and initialize global variable depending on platform where we are. All others calls of these functions we must to do by this variable.

You'll ask: How do we know where we are? It's easy on the embedded platforms like ARM. It has a device tree, which contains information about the peripheral and platform. It is chosen and passed to the kernel by the hardware, we only parse it and know all what we need. We've decided that libfdt under the BSD license is the best for these targets. The device tree information need in the early boot stage, thus I've added it in the pre_init.c.

 

Can you describe the process of hardware passing device tree information to a kernel?

The booting process is also changed. We had a great idea(as for our modest opinion): enable paging in the early bootstrap (in the head.S). Thus we have only the one file with unpaged code. It makes the kernel linker script, makefile and the memory structure easier. It isn't the last idea about the bootloader.
https://github.com/nekorobov/minix/commits/rpi_clean_dev

Hello, I am glad to see that you have decided so much to do.

So, my questions.

You have provided the link:

https://github.com/nekorobov/minix/commits/rpi_clean_dev

why can I see that the last commit is at the 1st of June?

So, let's go through your proposal.

You proposed:

"1st half of June - improve tty, clean up booting process. 
End of my exams.

2nd half of June - I2C, mailbox. "

So, what about tty issues?

Could you describe more thoroughly what was improved?

And what about I2C and mailbox drivers?  

We have great plans, how to make the MINIX3 boot process general and convenient for understanding and development in the future. We thought that it would be good: to make a general bootloader with a general structure for the all architectures and the all platforms. Multiboot2.h format was very helpful for these aims. The new kernel-loader can enable paging, it allows us don't worry about paging in the kernel, because the kernel load is in the virtual memory now. The kernel-loader parse the device tree (if it exist) or the command line parameters and packs the multiboot structure by the information above. It doesn't affect on, where are we are, because the multiboot has a universal header for the each architecture. It passed through the kernel argv, where the kernel sets up all the system parameters. Furthermore, the second parameter may have the special architecture structure (with the device tree information e.g.). I have done the great part of this, but stopped because of a couple of problems, which I can't solve now. We decided to put it aside until the free time, but the start was given. :)

Why not to use some already existing bootloader?

 

 

At the end of the month, I've started development the SDCard driver. The RPI has differ register set compared with the Omap.Therefore there are two mmc and emmc drivers (for each platform), but I'm trying to find ways how to make it general. Determination of the driver constants will be from the device tree.
https://github.com/nekorobov/minix/commit/03eaf9dac896a403d855892e016dc6b4dc137bc5

Does this mean  that there are no plans on I2C and mailboxes? Could you provide your renewed schedule? 

[Jean-Baptiste Boric]

Hi, I'm one of the co-mentors.

Hi MINIX3! As you remember, I'm GSoC 2017 participant. Jean-Baptiste Boric and Lionel Sambuc are my great mentors. I'm trying to make the MINIX more portable and port it to the Raspberry Pi. I had finished my first month of work and would like to tell you about my results and ideas. I'm ready to answer any questions. :)

 

Firstly, I'm trying to improve existed simple port by Jean-Baptiste Boric. It includes a lot of dirty hacks and problems such as versions of kernel for the each platform. I think, that I don’t have to explain you why it isn't good idea. What I did: We decided that, platform should resolve at run-time. All platform-dependent code will be chosen dynamically by global table. For the each platform we must make the function-table and initialize global variable depending on platform where we are. All others calls of these functions we must to do by this variable.

You'll ask: How do we know where we are? It's easy on the embedded platforms like ARM. It has a device tree, which contains information about the peripheral and platform. It is chosen and passed to the kernel by the hardware, we only parse it and know all what we need. We've decided that libfdt under the BSD license is the best for these targets. The device tree information need in the early boot stage, thus I've added it in the pre_init.c.

 

Can you describe the process of hardware passing device tree information to a kernel?

The Raspberry Pi firmware gets the device tree from the SD card and pass its address in memory to the kernel in a register.

For now, the kernel gets the hardware platform type it is running on from the device tree. Other properties can be extracted (libfdt has been imported into the kernel), but it'll be done at a later point.

 

The booting process is also changed. We had a great idea(as for our modest opinion): enable paging in the early bootstrap (in the head.S). Thus we have only the one file with unpaged code. It makes the kernel linker script, makefile and the memory structure easier. It isn't the last idea about the bootloader.
https://github.com/nekorobov/minix/commits/rpi_clean_dev

Hello, I am glad to see that you have decided so much to do.

So, my questions.

You have provided the link:

https://github.com/nekorobov/minix/commits/rpi_clean_dev

why can I see that the last commit is at the 1st of June?

So, let's go through your proposal.

You proposed:

"1st half of June - improve tty, clean up booting process. 
End of my exams.

2nd half of June - I2C, mailbox. "

So, what about tty issues?

Could you describe more thoroughly what was improved?

And what about I2C and mailbox drivers?  

The latest commit is at https://github.com/nekorobov/minix/commits/rpi_clean_driver, dated June 30th. Exams were in the way during mid-June and we realized somewhere during the second half of June that cleaning up the existing ARM port (and especially early bootstrap/kernel) for proper multi-vendor board support would take a LOT of time and be very difficult.

We decided to postpone this for later and focus instead on writing drivers, the SD card one first. TTY issues are not blocking the port for the moment, so this has been set aside too for the time being. 

 

We have great plans, how to make the MINIX3 boot process general and convenient for understanding and development in the future. We thought that it would be good: to make a general bootloader with a general structure for the all architectures and the all platforms. Multiboot2.h format was very helpful for these aims. The new kernel-loader can enable paging, it allows us don't worry about paging in the kernel, because the kernel load is in the virtual memory now. The kernel-loader parse the device tree (if it exist) or the command line parameters and packs the multiboot structure by the information above. It doesn't affect on, where are we are, because the multiboot has a universal header for the each architecture. It passed through the kernel argv, where the kernel sets up all the system parameters. Furthermore, the second parameter may have the special architecture structure (with the device tree information e.g.). I have done the great part of this, but stopped because of a couple of problems, which I can't solve now. We decided to put it aside until the free time, but the start was given. :)

Why not to use some already existing bootloader?

We want to make a separate pre-boot execution environment that sets up paging, extract the most interesting properties from the device tree to pass to the kernel, unpacks the boot modules and pass execution to the kernel. Among other things, this would get rid of unpaged code in the kernel binary, remove device tree parsing from the kernel (unlike usermode drivers, the kernel itself needs little information from the device tree) and remove most board-specific bootstrap code from the kernel.

So our "bootloader" is not a replacement for a real bootloader like U-Boot, it's just a way to simplify the early kernel initialization (which is rather complicated and crufty right now). As noted previously, due to sheer complexity and time requirements this has been postponed in favor of drivers.

 

At the end of the month, I've started development the SDCard driver. The RPI has differ register set compared with the Omap.Therefore there are two mmc and emmc drivers (for each platform), but I'm trying to find ways how to make it general. Determination of the driver constants will be from the device tree.
https://github.com/nekorobov/minix/commit/03eaf9dac896a403d855892e016dc6b4dc137bc5

Does this mean  that there are no plans on I2C and mailboxes? Could you provide your renewed schedule? 

We haven't decided on a new schedule yet. Porting operating systems is a hard task with many pitfalls (especially when multi-vendor board support is nonexistent in the original ARM port), so as long as reasonable progress is made I'm happy about it.

[Nikita Korobov]

Can you describe the process of hardware passing device tree information to a kernel?

Yeah, the RPI's boot process starts with the GPU. It loads some firmware images like a start.elf, bootcode.bin, etc, CPU is disabled at this moment. GPU reads config.txt and cmdline.txt, it generates device tree depending on it and copy the device tree at the 0x100 by default (but we can tune it) and copy address of the device tree into r2.

why can I see that the last commit is at the 1st of June?

I started work in May, because I'm sophomore of MIPT and I have exams in June, I wanted to do a great part of my job in May. I've started the new bootloader in June, but it isn't success, thus all commits only in my local repo, I'll push it later, as soon as I make it works.
 

So, what about tty issues?

Could you describe more thoroughly what was improved?

And what about I2C and mailbox drivers?  

We decided, that we have a lack of memory and services for stable debug system, the SDCard driver will solve this problem. I2C driver will be realized, but later. Mailbox driver is very easy, I've done it, when I did a trial task in March, but it must be formated as a driver.
 

While I have written this post, Jean-Baptiste Boric are making it faster than me :)

[Denis Obrezkov]

воскресенье, 2 июля 2017 г., 20:19:36 UTC+2 пользователь Nikita Korobov написал:

Can you describe the process of hardware passing device tree information to a kernel?

Yeah, the RPI's boot process starts with the GPU. It loads some firmware images like a start.elf, bootcode.bin, etc, CPU is disabled at this moment. GPU reads config.txt and cmdline.txt, it generates device tree depending on it and copy the device tree at the 0x100 by default (but we can tune it) and copy address of the device tree into r2.

why can I see that the last commit is at the 1st of June?

I started work in May, because I'm sophomore of MIPT and I have exams in June, I wanted to do a great part of my job in May. I've started the new bootloader in June, but it isn't success, thus all commits only in my local repo, I'll push it later, as soon as I make it works.

I want to note that all students from exUSSR have exams in June, me too, so what? 

 

So, what about tty issues?

Could you describe more thoroughly what was improved?

And what about I2C and mailbox drivers?  

We decided, that we have a lack of memory and services for stable debug system, the SDCard driver will solve this problem. I2C driver will be realized, but later. Mailbox driver is very easy, I've done it, when I did a trial task in March, but it must be formated as a driver.
 

Lack of memory on RPi? Really? 1 GiB of RAM! You can place everything there.

My board has 16 KiB of RAM and I am porting RTEMS on it and debugging works pretty well.   

[David van Moolenbroek]

Hello Denis,

We understand that you are not happy with the fact that we did not pick you as the GSoC student for MINIX this year. However, you have no right to take that out on the GSoC student that we did select. As such, this is your first and last warning. If you continue to post these kinds of unconstructive, negative, and even aggressive posts, your access to this group will be revoked. I would strongly suggest that you move on instead.

Regards,
David

[Denis Obrezkov]

воскресенье, 2 июля 2017 г., 22:03:16 UTC+2 пользователь David van Moolenbroek написал:

Ha, guys, really, I don't care about myself, I am working on a MUCH better project, I care about another girl's proposal. It was really really good, creative and mature. And she proposed to work on the existing port that currently doesn't work. The Nikolay's one was unrealistic and you can see that I am right. But, anyway, since you aren't able to even define your goal or schedule your students' work, I will leave this group by myself.  Tschüss.