Minimal, self-contained Crossbar deployment for embedded systems

[Emile Cormier]

First off, please forgive my ignorance about Python.

How would one go about creating a compact, self-contained Crossbar deployment for an embedded Linux-ARM system? Ideally, I'd like to be able to dump a "compiled" Crossbar to the target platform without having to install a bunch of Python libraries. Afterwards, if I want to remotely update to a newer version of Crossbar, I just extract a compressed archive to temporary folder and then swap it out with the old one.

I'd like to somehow bake the Python runtime into the Crossbar deployment. That way, if Crossbar depends on changes to Python itself, then I simply replace the runtime along with Crossbar.

Our WAMP-based product will be a headless network appliance, and we don't expect our users to update library dependencies on their device. That's why I want to have a compact Crossbar deployment that can be easily swapped out with a newer one when needed.

Any pointers on how one would go about doing this would be greatly appreciated. If you know of an unrelated Python app that can be deployed in the manner I'm referring to, please let me know.

Cheers,

Emile Cormier

[Emile Cormier]

I was thinking about the instructions for the RaspberryPi when I originally posted. The problem I had with those instructions was that all the dependencies were installed "auto-magically" in an apparently non-self-contained manner.

After reading the instructions for installing from source on Ubuntu/Debian, I see that it's possible to deploy everything in a "virtual environment". I think this is the self-contained deployment package I'm looking. Whether or not it's compact remains to be seen.

But a question still remains. For my intended usage, am I better off with CPython or PyPy? Apparently, PyPy has better execution performance, but is there a difference in the total size of the virtual environment? I guess I might have to try both ways and see for myself.

[Gareth Bult]

Hi Emile,

If you use CPython, then you could try something like "bbfreeze" to produce a binary package .. not tried it myself .. but in terms of a comparison, I've recently converted my Crossbar application to "pypy" and it uses a lot more memory.  Previously I was running comfortably on a 512M Digital Ocean instance with no swap .. for production I'm now considering a 1G instance "with" swap. So in my case, slightly over 2x usage would be a reasonable estimate. Can't really say how well it performs, "pypy" feels slower, although I know it must be quicker, I suspect the first-time compilation is slower than CPython and I suspect that is what is making is "seem" a little slower .. but once I stop reloading every 30 seconds while testing, I expect this issue to evaporate ...

But if you're working on RaspberryPi devices, I would think you're pretty much going to be stuck on CPython .. if you expose yourself to pypy memory usage with your swap on an SD card, speed aside, you're going to stuff your SD card in fairly short order ...

[Emile Cormier]

Thanks, Gareth, that information is quite useful! My device is a BeagleBone Black with 512MB RAM, so I guess I'll stick with CPython. I'll give bbfreeze a try and will let you know how it works out. I'm not going to tackle this anytime soon; I was just planning ahead.

[Gareth Bult]

Sure, I know things like bbfreeze "sound" good (and may well be if you're on Windows) but if you're using Linux, you may find it just as easy (or easier) to build your own package (.rpm/.deb etc) containing the required files and deploy that as opposed to a binary .. you won't be compiling as you develop, so if you deploy post-compiled code, you're relying completely in your test suite, rather than ongoing developer testing you would normally have with Python apps.

Alternatively (!) I deploy to a raw device using Ansible, taking source files directly from a "git" tag .. (I hear purists screaming in the background) but this works well for me and I don't have to worry about keeping up different package builds for lots of different distributions .. just a few extra conditionals in my Ansible scripts.

[Emile Cormier]

There already are precompiled deb packages for desktop. Do you know if the build scripts that generate those are available? If so, then perhaps I could tweak them to generate deb packages for my ARM platform.

[Gareth Bult]

Mmm, not sure which bits you mean .. my previous project involved Pi's and Crossbar (door entry system) and I was able to install the whole thing using standard .deb packages and "pip" .. (again, all via ansible). So if you're using Pi's, then everything you need should be available ... minus your actual project code - which you would need to either deploy directly or via your own .deb package ... (?)

[Tobias Oberstein]

Hi Emile,

we already provide completely self-contained binary packages of
Crossbar.io for Debian/Ubuntu x86-64. These bundle everything but the C
run-time.

We might expand that to provide the same for Linux/ARM. Eg deb packages.

In the future, we likely will also provide snaps for Ubuntu Core. You
might want to look into Ubuntu Core - the concepts and design makes a
lot of sense.

In fact, I think what Ubuntu Core is doing is really the way forward for
the "how to update embedded Linux devices and apps" problem.

Cheers,
/Tobias

[Emile Cormier]

Thanks, Gareth and Tobias. For some reason, I thought deb packages were tied to a specific architecture. But if the Crossbar deb package only contains Python script code, then it makes that you would be able to install it on the Raspberry Pi.

--
You received this message because you are subscribed to a topic in the Google Groups "Crossbar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/FWZWJFrmFYU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/57506BF7.7050802%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Gareth Bult]

Hi, they are, for obvious reasons. Python however (and other scripting languages) are generally architecture agnostic and for such occurrences we have an architecture labelled as "noarch". (this is also used for documentation, "man" pages and the like) If you have a Python application that will run on any system, typically when you build the package, choose "noarch" as the architecture and it will install on any platform.

If you use the stock Ubuntu for Raspberry Pi, it should come with access to "apt" and packages built for ARM architecture, including things like "pip".

I must admit I didn't actually run Crossbar "on" a Pi, we used Pi/Autobahn clients with a cloud based Crossbar .. but I was sort of expecting "pip" to be able to pull crossbar down and compile it up .. (??)

[Tobias Oberstein]

The deb package is arch dependent .. as it bundles PyPy and also binary extensions. So what we need is to extend our build infra to produce ARM based debs as well ..

Sent from Mobile

> You received this message because you are subscribed to the Google Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to crossbario+...@googlegroups.com.

> To post to this group, send email to cross...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/CAM70yxDCE8jpCYHOie6VcWGEgRGpKOCG5s8hryBpO8AGRrjnig%40mail.gmail.com.

[Emile Cormier]

On Friday, June 3, 2016 at 12:07:39 PM UTC-3, Tobias Oberstein wrote:

The deb package is arch dependent .. as it bundles PyPy and also binary extensions. So what we need is to extend our build infra to produce ARM based debs as well ..

If you do, please provide support for the gcc-arm-linux-gnueabihf architecture.

[Tobias Oberstein]

Hi Emile,

> If you do, please provide support for the gcc-arm-linux-gnueabi*hf*
> architecture.
>

sure. as far as I know the following holds true anyways:

- neither CPy nor PyPy runs on ARM softfp (ARM11 lacking HW FPU)
- the Pi 3 has a 64 bit ARM core, but a 32 bit userland

since our build system creates a completely self-contained binary
packages (it even bundles its own OpenSSL), we only depend on the C
run-time.

we can't ship that / paper over diffs in glibc between different Linuxes
with incompat. C run-times, and (at least short term) will only be able
to provide builds for glibc, not other C run-times ..

cheers,
/Tobias

PS: have you looked at Ubuntu Snappy Core? Opinions?

[Emile Cormier]

Tobias,

I've only looked at Ubuntu Core very briefly. It seems promising; I'll have to study it further when I have the chance.

For the BeagleBone, the primary distro being supported is Debian. I don't know how well that platform will run Ubuntu Core. In theory, I should be able to use the same kernel provided with the BeagleBone Debian image, and install the Ubuntu Core distro on top of it.

We still have to decide whether we want to use apt as the mechanism to deploy updates, or roll our own. Our system has to very robust against failed or interrupted updates.

Cheers,

Emile

[firefighterblu3]

FYI, I use Arch Linux on multiple single board computers, 32bit and 64. I've had no problems installing or using crossbar and autobahn python

Sent from my T-Mobile 4G LTE Device

--

You received this message because you are subscribed to the Google Groups "Crossbar" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/d1ce0816-2bab-49c9-9c75-2e91b62cd95a%40googlegroups.com.

[Greg Keys]

you could use docker.