Pine Linux distro

[Jeremy Kahn]

Pine, among other things, is going to be a custom Linux distro.  I have never made a Linux distro before, so this will be interesting to say the least.  I'll ping this thread with my progress as I go.

I am doing all distro development in an Ubuntu VM.  So far, I have compiled the Linux kernel with these instructions.  I don't actually have anything working yet, but it's a start.  I am going to look at relinux for packaging the distro.

[Jeremy Kahn]

I managed to compile a custom Linux kernel (at least according to tutorials), but I can't boot it in QEMU with the rootfs's I've downloaded.  It looks like I need to compile an RPi kernel with QEMU specific hacks, which seems kinda wrong, but I will give it a shot.  I'm also learning all about Linux Rootfs, which is fun.  My goal is to build a Linux environment from scratch that runs in QEMU and eventually on the RPi hardware.  Even if we don't use it, it's important to have a general understanding of distro development fundamentals like this.

[Jeremy Kahn]

I'm still doing research (there's a lot to learn).  At this point, it seems like a good way forward is to make a custom version of Raspbian.  It is an RPi-optimized Debian distro.  I believe that this distro is where all of Hexxeh's Chromium optimization work is happening, and it's also pretty minimalist.  It seems like a good fit for us.  Importantly, there is a bunch of documentation to help me out.

--
You received this message because you are subscribed to the Google Groups "Pine" group.
To post to this group, send email to pine-d...@googlegroups.com.
To unsubscribe from this group, send email to pine-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/pine-discuss/-/zkbCoFeHELQJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Jeremy Kahn

jerem...@gmail.com

https://github.com/jeremyckahn

@jeremyckahn

[Jeremy Kahn]

At this point, I'm wondering if all we have to do is take a Raspbian image and just write some shell scripts to customize it post-install.  Then we would just instruct users to download the image and run the scripts.  It's kinda lazy, but it seems like that might be the fastest way to get a new Linux distro up and running.

[Alex Wilson]

I say we go with whatever's fastest now and later we can polish things up. Once we have a working platform we can benchmark and optimize.

[Jeremy Kahn]

Good news everyone!

After lots of tinkering, I now have Chromium running in fullscreen and booting with a request a local Node server.  Chromium is running in kiosk mode, which means that there is no window chrome or desktop bogging the system down.  This means that all components of the Pine stack work!  This isn't really groundbreaking, but I at least know how it all fits together so I can build on top of it.

I ended up just taking a stock Raspbian image and modifying it.  Kinda dumb, but this is probably easier than trying to fork Raspbian and keeping it all on Github.  The idea is that people can just get Raspbian however they please and then run some shell scripts to Pine-ify it.  We'll also have pre-built images, but we don't need to worry about that now.  I have some scripts Pine-ify Raspbian here, but they really need to be cleaned up, tested and documented.  It's not worth trying to optimize for performance until I have real hardware, but I do want to make Raspbian boot into a basic Pine environment.  By "Pine environment," I am referring to Chromium kiosk instance loading a local Node app.

Woohoo!

On Sat, Jul 28, 2012 at 7:09 PM, Alex Wilson <arex...@gmail.com> wrote:

I say we go with whatever's fastest now and later we can polish things up. Once we have a working platform we can benchmark and optimize.

--

You received this message because you are subscribed to the Google Groups "Pine" group.
To post to this group, send email to pine-d...@googlegroups.com.
To unsubscribe from this group, send email to pine-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/pine-discuss/-/C6t4mzeRB2IJ.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jeremy Kahn]

More good news.  I have figured out how to log in as pine-user without prompting for a password when the system boots.  Additionally, I figured out how to start a background Node process and launch Chrome when pine-user logs in.  This means that when you boot up, you are dropped into Chrome requesting a local Node app.  I still want to clean up and test the script.  

There's little value in developing the Linux environment further until I have hardware to test on.  I'm going to start working on the Pine app (the UI that users use to launch games and interact with the system).  I'll start another thread for that.

[Luke Stebner]

Good work! Glad to hear the progress. I kinda like the idea of being able to "pineify" any existing Pi with some scripts because that lets people who already own them capable of turning theirs into a Pine. Again, well done.

[Jeremy Kahn]

Thanks!  I suppose I should clarify, I don't want Pine to serve as an augmentation to an OS already running on an RPi. It should be its own standalone OS. By isolating the OS, we can minimize weird conflicts with other software. Also, users probably won't want their general use computer booting into a Pine environment. 

Fortunately, swapping OSes on an RPi is as simple changing out the SD card. SD cards are cheap enough if people want to have separate environments. 

--

Jeremy Kahn

jerem...@gmail.com

Tel: 847.347.8086

On Jul 31, 2012, at 11:45 PM, Luke Stebner <luke.s...@gmail.com> wrote:

Good work! Glad to hear the progress. I kinda like the idea of being able to "pineify" any existing Pi with some scripts because that lets people who already own them capable of turning theirs into a Pine. Again, well done.

--

You received this message because you are subscribed to the Google Groups "Pine" group.
To post to this group, send email to pine-d...@googlegroups.com.
To unsubscribe from this group, send email to pine-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/pine-discuss/-/cmJQUkznLY0J.

[Scott Elcomb]

On Wed, Aug 1, 2012 at 11:14 AM, Jeremy Kahn <jerem...@gmail.com> wrote:
> Thanks! I suppose I should clarify, I don't want Pine to serve as an
> augmentation to an OS already running on an RPi. It should be its own
> standalone OS. By isolating the OS, we can minimize weird conflicts with
> other software. Also, users probably won't want their general use computer
> booting into a Pine environment.
>
> Fortunately, swapping OSes on an RPi is as simple changing out the SD card.
> SD cards are cheap enough if people want to have separate environments.

This weekend I'll try getting an NFS image[1] to work. If it does,
nightly builds and automated testing on real hardware should be
doable. Another avenue I'm pursuing is to find something like a
Flu-card[2] to trick the Pi into booting from a remote SD card image
(hosted on a dev box).

I'll also start running through the Raspbian distro sources - it'll
almost certainly be easiest to start with modifying Raspbian as the
initial OS. A finely tuned LFS would be a good light-weight &
long-term solution, though I've not yet tried building one for an ARM
platform. (An excellent excuse to try it!)

[1] <http://raspberrypi.stackexchange.com/questions/628/how-do-i-configure-the-raspberry-pi-to-boot-with-an-nfs-root>
[2] <www.flu-card.com/>

--
Scott Elcomb
@psema4 on Twitter / Identi.ca / Github & more

Atomic OS: Self Contained Microsystems
http://code.google.com/p/atomos/

Member of the Pirate Party of Canada
http://www.pirateparty.ca/

[Jeremy Kahn]

Very ambitious.  It sounds like your goal is to be able to boot from a filesystem via a network?  That will certainly make developing and iterating the OS much easier.  Is this something that other Pine OS developers will have to set up in order to collaborate?

Don't be surprised if I have a lot rookie Linux questions.  I'm... well, a Linux rookie.  However, I definitely want to understand everything that you're doing so that I can learn and contribute.

--
You received this message because you are subscribed to the Google Groups "Pine" group.
To post to this group, send email to pine-d...@googlegroups.com.
To unsubscribe from this group, send email to pine-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Scott Elcomb]

On Wed, Aug 1, 2012 at 10:17 PM, Jeremy Kahn <jerem...@gmail.com> wrote:
> Very ambitious. It sounds like your goal is to be able to boot from a
> filesystem via a network? That will certainly make developing and iterating
> the OS much easier. Is this something that other Pine OS developers will
> have to set up in order to collaborate?

That's part of the goal, the other part is running automated tests.
I'm thinking to use a simple CI process using something like Concrete
<http://ryankee.github.com/concrete/>.

Unless devs want to dedicate a box towards a distributed test farm,
they shouldn't need to do anything different:

Once a day the CI server will pull the project from it's repository,
build it then test and post the results.

Ideally there would be multiple sites running & sharing these tests,
but even one (which I'd be happy to contribute) would be beneficial.

> Don't be surprised if I have a lot rookie Linux questions. I'm... well, a
> Linux rookie. However, I definitely want to understand everything that
> you're doing so that I can learn and contribute.

No worries - I moved to Linux years ago and am still learning. Will
add documentation on the wiki as things progress as well.

[Jeremy Kahn]

Continuous Integration would be awesome.  I've never set up or managed a CI system before, but that's another thing I'd love to learn.  I can provision my desktop to run the tests, since it's always on.  

Minor concern:  Is this something we should be implementing this early on?  I'm all for planning for the long term, but I want to make sure we focus on the right things at the appropriate stages of the project.  What do you think is the most important first step to take for developing the Pine OS?  It might be worth putting together a high-level ordered list of tasks so we can calibrate our efforts.

--
You received this message because you are subscribed to the Google Groups "Pine" group.
To post to this group, send email to pine-d...@googlegroups.com.
To unsubscribe from this group, send email to pine-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Scott Elcomb]

On Thu, Aug 2, 2012 at 1:05 AM, Jeremy Kahn <jerem...@gmail.com> wrote:
> Continuous Integration would be awesome. I've never set up or managed a CI
> system before, but that's another thing I'd love to learn. I can provision
> my desktop to run the tests, since it's always on.
>
> Minor concern: Is this something we should be implementing this early on?
> I'm all for planning for the long term, but I want to make sure we focus on
> the right things at the appropriate stages of the project. What do you
> think is the most important first step to take for developing the Pine OS?
> It might be worth putting together a high-level ordered list of tasks so we
> can calibrate our efforts.

While I've made use of CI systems, I've never set one up either so
this will be a bit of a learning experience for me as well. My
primary motivation for setting up CI is that it provides a kind of
guarantee to users (including developers) that the latest nightly will
"just work" on stock hardware. Another motivation is to get notified
in a timely fashion when there is a show-stopping issue.

For complex projects (like systems) it's incredibly important to plan
first. Placing early priority on automation & testing is a worthwhile
step that should save time and effort in the long run. I can't count
the number of projects I've worked on that didn't consider testing
and/or documentation requirements before beginning... It's always a
struggle to fix after things are rolling. As a contractor, I've even
stopped taking contracts from clients who refuse to take documentation
seriously - it's just too much pain to figure everything out, all the
time.

Anyway, what I'm planning to do this weekend is to lay a foundation
for CI; implementation will take a while - it should grow with the
project (by adding new tests).

To start with I'd just like it to create a fresh Raspbian image, boot
the hardware and apply the Pine scripts. In the future, we can add
tests to check specific functionality (say USB drivers in a custom
distro or test a UI interface with Selenium), but the foundation work
- like how to setup NFS and write simple tests - will be out of the
way and documented.

As for the task list, do we have any Google Docs, etherpads/piratepads
or similar to work with? (Or just write into a page on the wiki and
update via pull requests?)

I'd also like to recommend a (semi?) regular IRC meeting once or twice
a month to discuss major issues in real-time.

[Jeremy Kahn]

> For complex projects (like systems) it's incredibly important to plan
> first. Placing early priority on automation & testing is a worthwhile
> step that should save time and effort in the long run. I can't count
> the number of projects I've worked on that didn't consider testing
> and/or documentation requirements before beginning... It's always a
> struggle to fix after things are rolling.

I completely agree that planning is important. I'm unfamiliar with OS development, and your reasoning makes a lot of sense. If you think it's important to get these systems in place at the outset, let's do it.

If you look at my Rekapi project, you'll see that I take documentation very seriously. :)

> To start with I'd just like it to create a fresh Raspbian image, boot
> the hardware and apply the Pine scripts. In the future, we can add
> tests to check specific functionality (say USB drivers in a custom
> distro or test a UI interface with Selenium), but the foundation work
> - like how to setup NFS and write simple tests - will be out of the
> way and documented.

Sounds like a good plan. Be aware that the scripts may be a little dicey, I haven't run them as is. I've been adding commands to them as I figure out what works. Please document your steps for all of this in the wiki (or just add Markdown files to the repo itself).

> As for the task list, do we have any Google Docs, etherpads/piratepads
> or similar to work with? (Or just write into a page on the wiki and
> update via pull requests?)

I've found that the Github wiki is a pretty effective tool for this sort of thing. Let's stick with that until we need something better. You can edit it straight from the web interface, no need to make a Pull Request.

> I'd also like to recommend a (semi?) regular IRC meeting once or twice
> a month to discuss major issues in real-time.

Great idea. I don't think we need to do this just yet, but we can definitely plan something when we get deeper into development. So far it's been more planning than anything, and the mailing list has been pretty effective for that.

As a minor aside, I don't intend to question your practices for developing Pine. I just want to make sure I understand everything you're doing any why. I want to be able to defend your decisions. :)

[Jeremy Kahn]

Scott, I just merged your awesome Pull Request on Github.  Great work!  I have some catching up to do, my UI branch isn't nearly as impressive as what you put together. :)

Sorry I can't give you better technical feedback, but I trust that you know what you are doing.

My understanding is that your Pull Request creates Pine distribution that can be loaded both in QEMU and on a Raspberry Pi.  Is that correct?  What are your next steps?

[Jeremy Kahn]

Scott, FYI:  I just gave you commit privileges on my fork.  You've got a handle on the Linux stuff, so I don't want to hold you up with waiting on me to merge your changes.

[Scott Elcomb]

On Tue, Aug 7, 2012 at 1:38 AM, Jeremy Kahn <jerem...@gmail.com> wrote:
> Scott, I just merged your awesome Pull Request on Github. Great work! I
> have some catching up to do, my UI branch isn't nearly as impressive as what
> you put together. :)
>
> Sorry I can't give you better technical feedback, but I trust that you know
> what you are doing.

No worries. :)

I'm a bit concerned about whether these scripts will run on OS X as
well as they do on Ubuntu. Any feedback from devs that use it would
be most helpful; I know virtually nothing about OS X.

> My understanding is that your Pull Request creates Pine distribution that
> can be loaded both in QEMU and on a Raspberry Pi. Is that correct? What
> are your next steps?

That's correct. On my dev laptop, the entire process (downloading,
unpacking, modifying, running first boot, modifying again) is fully
automated and takes about 10 minutes.

Next steps:
* Cleanup scripts (remove redundant blocks)
* Document each stage of the process (started at
<https://github.com/psema4/pine/wiki/Distro>)
* Run the auto-pine scripts nightly through concrete (I expect
sometime this week)
* Identify superfluous packages in the raspbian image that should be
removed (during pine_setup.sh)

Cheers,
- Scott.

PS - Thanks for the commit bit and shoutout on Twitter Jeremy! :-)

[Jeremy Kahn]

I don't think we need to worry about OS X environments in respect to developing the Pine OS. Most people who want to develop a Linux will be using Linux. Alternatively, they can use a Linux virtual machine, which is what I've been doing. Ubuntu-only is fine, and it will speed us up in the short term.

Your next steps sound good to me. Please update this thread as you progress.

> On my dev laptop, the entire process (downloading,
> unpacking, modifying, running first boot, modifying again) is fully
> automated and takes about 10 minutes.

Wow, impressive! That will really help make developing and testing easier. It's worth noting that time figure in the documentation somewhere, if it isn't already.

Have you gotten a chance to test Chromium on actual hardware yet? Since I'm waiting on my RPi, I'd love to know how it actually performs.

For my own part, I am still working on the UI menus. There's one more small feature I want to add, and then I will merge back to master. I also need to think of a practical way to benchmark browser performance in a Pine environment.

> PS - Thanks for the commit bit and shoutout on Twitter Jeremy! :-)

Always happy to celebrate progress!

[Scott Elcomb]

On Tue, Aug 7, 2012 at 11:21 AM, Jeremy Kahn <jerem...@gmail.com> wrote:
>> On my dev laptop, the entire process (downloading,
>> unpacking, modifying, running first boot, modifying again) is fully
>> automated and takes about 10 minutes.
>
> Wow, impressive! That will really help make developing and testing easier. It's worth noting that time figure in the documentation somewhere, if it isn't already.

Thought about that and initially decided against it. I'll note it
along with a caveat: w/o broadband, the process will take a good bit
longer. Also the number of seeders for the raspbian torrent will
likely fluctuate over time.

> Have you gotten a chance to test Chromium on actual hardware yet? Since I'm waiting on my RPi, I'd love to know how it actually performs.

It runs slow but rather well otherwise. Snake at least (from
html5games.com) runs just fine. =D

IIRC, hexxeh's working on getting more of the code to run in the GPU.
When that happens, we should be good as far as the browser is
concerned.

One concern however: the chromium launcher (/home/pine-user/.bashrc)
sets the window size to 640x480 which is great when using NTSC/PAL
out. Unfortunately, it does not work so well with HDMI as it leaves
about 2/3 of my test display black.

Going into monitor properties I get an error for both NTSC/PAL and
HDMI. We'll need to find a way to detect the output resolution before
pine-user's bashrc runs.

> For my own part, I am still working on the UI menus. There's one more small feature I want to add, and then I will merge back to master. I also need to think of a practical way to benchmark browser performance in a Pine environment.

I haven't tried this but perhaps Selenium could be used for
benchmarking performance?

Testing tools & frameworks like Selenium can be installed using
auto-pine's first_boot.sh and pine_setup.sh - for distribution
images, mksd & mksd-nfs could be modified to remove testing tools/data
if a flag is present (say --dist).

Eventually, I'd like to have a dev images running live tests and
reporting back to the CI server during nightly builds.

[Jeremy Kahn]

It runs slow but rather well otherwise.  Snake at least (from
html5games.com) runs just fine. =D

Do you mean that it starts slowly?  I've heard that can be an issue, and it's probably due to the fact that SD cards are generally pretty slow.  We'll have to do some research so we can make SD card recommendations to users.  Aside from that, it's a really good sign that it can handle Snake!  I'm curious how well it can handle my Rekapi demos.  They run at about 60 fps on my Macs, and I wonder how smoothly they run on Pine.  Unfortunately I don't have framerate recording functionality for Rekapi, but I can build that in as a way to benchmark performance.  I expect that the bubbles.html demo will run slowly.

One concern however:  the chromium launcher (/home/pine-user/.bashrc)
sets the window size to 640x480 which is great when using NTSC/PAL
out.  Unfortunately, it does not work so well with HDMI as it leaves
about 2/3 of my test display black.

Going into monitor properties I get an error for both NTSC/PAL and
HDMI.  We'll need to find a way to detect the output resolution before
pine-user's bashrc runs. 

Yeah, that's definitely something we need to fix.  This seems like a problem that has been solved before, so it's likely just a matter of Googling around and figuring it out.  I'm happy to help with this once I have hardware and some free time.

I haven't tried this but perhaps Selenium could be used for
benchmarking performance?

 

Testing tools & frameworks like Selenium can be installed using
auto-pine's first_boot.sh and pine_setup.sh -  for distribution
images, mksd & mksd-nfs could be modified to remove testing tools/data
if a flag is present (say --dist).

I haven't used Selenium before, so I can't argue for or against it.  However, I wonder if this approach might be a bit heavy-handed?  I was envisioning a set of JavaScript/DOM/Canvas tests that we can run and record.  We could just send an AJAX request to a remote server to log the results.  I'm open to other approaches, we just need to collect metrics that would be helpful to game developers.  For instance, a game developer would want to know at what point the framerate starts to drop off when animating a growing number of divs.

Eventually, I'd like to have a dev images running live tests and
reporting back to the CI server during nightly builds.

That sounds great.  My next task (before wrapping up my UI stuff) is to run your Asphalt instructions.  Hopefully I can finish that tonight. 

--
You received this message because you are subscribed to the Google Groups "Pine" group.
To post to this group, send email to pine-d...@googlegroups.com.
To unsubscribe from this group, send email to pine-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Scott Elcomb]

On Tue, Aug 7, 2012 at 10:53 PM, Jeremy Kahn <jerem...@gmail.com> wrote:
>> It runs slow but rather well otherwise. Snake at least (from
>> html5games.com) runs just fine. =D
>
> Do you mean that it starts slowly? I've heard that can be an issue, and
> it's probably due to the fact that SD cards are generally pretty slow.
> We'll have to do some research so we can make SD card recommendations to
> users. Aside from that, it's a really good sign that it can handle Snake!
> I'm curious how well it can handle my Rekapi demos. They run at about 60
> fps on my Macs, and I wonder how smoothly they run on Pine. Unfortunately I
> don't have framerate recording functionality for Rekapi, but I can build
> that in as a way to benchmark performance. I expect that the bubbles.html
> demo will run slowly.

Haven't tried your Rekapi demos yet, but generally speaking Chrome
runs slow because the CPU's doing everything. When the GPU gets
involved there should be a noticeable performance boost.

>> One concern however: the chromium launcher (/home/pine-user/.bashrc)
>> sets the window size to 640x480 which is great when using NTSC/PAL
>> out. Unfortunately, it does not work so well with HDMI as it leaves
>> about 2/3 of my test display black.

I added some alternate resolutions (commented out) to pine-user's
.bashrc file. I've tested 720p entry and it works rather well. Lots
more real estate to work with. Will try to automate.

...

Being buried in work all last week, I wanted to try something fun over
the weekend... namely testing the Gamepad API. I'll post about it on
the associated thread.

Basically it did not go nearly as smoothly as I'd hoped - I started
with a fresh copy of the Pine repo and ran auto-pine from scratch,
quickly discovering that last weekend I must've got confused somewhere
between the different images. I spent most of yesterday fixing things
up (with the critical bit of removing raspi-config from
/etc/profile.d), now auto-pine works as advertised with one exception:

Currently when writing to an SD card, you'll get the NFS-boot version;
to fix it I'll move the meat of boot_parts.sh into mksd-nfs -
shouldn't take long at all.

[Jeremy Kahn]

Sounds good, thanks for making the fixes. To be honest, I haven't had
a chance to actually test out your auto-pine work, I should have made
that a bit clearer. I'm busy for the next few days but I will try out
auto-pine as soon as I can side aside enough time.

Sorry about not helping you to catch that. If you'd like me to test
something specifically in the future, please just let me know. I'm
happy to do it. :)

[Scott Elcomb]

Thanks and no worries - wasn't specifically looking for anyone to
test. I like to hit the reset button rather often (starting a fresh
repo/build from scratch), especially when writing code to catch
situations like this. =D

Cheers,