Creating packages that reverse-proxy "standalone" applications

[Dan Krol]

Hello,

I was thinking of packaging a few services for UBOS that run their own web server in their own process, rather than being a web app built for Apache/Nginx or even Tomcat. For instance, an IRC web client such as KiwiIRC. I would want to reverse proxy it so that it would still run through Apache and fit the UBOS paradigm. I've played around with this a bit, and have the very basics of UBOS packaging figured out. However this seems like a slightly unusual scenario for UBOS, so I'm wondering if I could get some help here.

The first hurdle is that a few of these have no Arch packages, or exist only in the AUR. How do we go about setting such a dependency? (For instance https://aur.archlinux.org/packages/kiwiirc/).

The second hurdle is that, taking Tomcat as an example, it looks like each application that I wrap in this manner needs to be created as a role. And for that matter, I think that in the case of IRC, both the client and the server should be configured as roles. (Since UBOS is about independence, I imagine people wanting to run their own IRC servers.) Is it feasible for a newcomer like me to set up an IRC client and server as roles?

One interesting thing I noticed, btw, is that Tomcat is listening on a hardcoded port. What if you install two Tomcat applications on one UBOS system? I'd think you'd want to be able to generate a unique port number per app. Or is there only ever one Tomcat instance?

I'll likely have some smaller questions, but these are the big ones that can get me started on understanding.

Thanks a lot!

Dan

[Johannes Ernst]

Hi Dan, welcome!

These are all excellent questions:

I was thinking of packaging a few services for UBOS that run their own web server in their own process, rather than being a web app built for Apache/Nginx or even Tomcat. For instance, an IRC web client such as KiwiIRC. I would want to reverse proxy it so that it would still run through Apache and fit the UBOS paradigm.

That’s the way to do it. For an example in case you hadn’t found it yet, look at the Java version of gladiwashere: https://github.com/uboslinux/ubos-toyapps/tree/master/gladiwashere-java which runs the app in Tomcat behind Apache as a reverse proxy.

I've played around with this a bit, and have the very basics of UBOS packaging figured out. However this seems like a slightly unusual scenario for UBOS, so I'm wondering if I could get some help here.

The first hurdle is that a few of these have no Arch packages, or exist only in the AUR. How do we go about setting such a dependency? (For instance https://aur.archlinux.org/packages/kiwiirc/).

That’s okay. There are all sorts of packages in UBOS that do not exist in Arch — ubos-admin for example.

Let’s take kiwiirc. First, you would create/modify/… a PKGBUILD (and ubos-manifest.json) so that you can build the package on your development machine, and so you can install it on a UBOS box with ubos-admin. (usually ftp or such from dev machine to UBOS box, pacman -U <package>, ubos-admin createsite or something like it.)

If you’d like to make it available to other people running UBOS, currently you have two choices.

1. You convince me that it is a good idea to include this as an app in mainline UBOS, and point me to the PKGBUILD that you created. I include this into the UBOS build, by adding it here: https://github.com/uboslinux/ubos-buildconfig/tree/master/config/hl/us

2. You put your PKGBUILD somewhere, and invite people to download from there, package themselves, and manually install themselves. I do that myself with rasptimer, which some people do run on UBOS, but which IMHO isn’t quite good enough to include in UBOS. (I wrote it, so I know what I am talking about!) See https://github.com/jernst/rasptimer

Soon, there will be another option (and that’s the reason that this isn’t quite documented well; no point if it is about to change), which is this:

3. You can setup your own package repository, i.e. basically the same as what we run at depot.ubos.net, and publish packages there. The only thing users will have to do is to add a repository path to their /etc/pacman.conf.d/ . This avoids the conflict best known as “Apple didn’t approve our app”.

In this case, I’d love to get the app into UBOS mainline.

The second hurdle is that, taking Tomcat as an example, it looks like each application that I wrap in this manner needs to be created as a role. And for that matter, I think that in the case of IRC, both the client and the server should be configured as roles. (Since UBOS is about independence, I imagine people wanting to run their own IRC servers.) Is it feasible for a newcomer like me to set up an IRC client and server as roles?

No. Roles are for things that run apps, like Apache and Tomcat today, could also be JBoss etc and databases, like mysql and postgres today and perhaps mongo etc.

One interesting thing I noticed, btw, is that Tomcat is listening on a hardcoded port. What if you install two Tomcat applications on one UBOS system? I'd think you'd want to be able to generate a unique port number per app. Or is there only ever one Tomcat instance?

There should only be one Tomcat instance. But you are correct about port numbers. The idea (not implemented yet … volunteers welcome) is to augment the ResourceManager.pm in ubos-admin to only allocate databases for apps that ask, but also allocate port numbers. The bitcoin daemon I’ve been working on could also use this.

I'll likely have some smaller questions, but these are the big ones that can get me started on understanding.

Keep on asking!

Cheers,

Johannes.

Thanks a lot!

Dan

--
You received this message because you are subscribed to the Google Groups "UBOS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ubos-dev+u...@googlegroups.com.
To post to this group, send email to ubos...@googlegroups.com.
Visit this group at http://groups.google.com/group/ubos-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/ubos-dev/c6768401-7f1e-4df8-b03a-44a4b40118ab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Krol]

Thanks Johannes.

That’s the way to do it. For an example in case you hadn’t found it yet, look at the Java version of gladiwashere: https://github.com/uboslinux/ubos-toyapps/tree/master/gladiwashere-java which runs the app in Tomcat behind Apache as a reverse proxy.

Yep, I figured out how to get it to do an http reverse proxy as well (Tomcat uses some special Java protocol). I got it to work with one of the programs I was trying to package for. However I had to manually start said application, so obviously my job isn't done here. BTW for the sake of your examples, I'm using Virtualbox to develop.

As far as packaging the application, I think there may be a misunderstanding possibly based on a bad assumption I'm making, which I should probably lay out. My understanding is that the application package, and the "UBOS package", would be two different different things. For instance, the UBOS package for Wordpress does not contain the entire source code for Wordpress, but merely a reference to it. Though, interestingly enough, what it references doesn't seem to be a separate Arch or UBOS package either; rather it's a URL to download straight from the Wordpress project's website, plus an md5sum. What if the referenced package had binaries, would you do the same? Wouldn't really work for Raspberry Pi, unless the project also supplied ARM binaries. So, assuming it required a build, don't most OS repositories host the built binaries? It would seem you'd want something it to be in the Arch or UBOS package repo. (I've never packaged for an OS before so please forgive any ignorance here).

So following this pattern, if I were to make a UBOS package for the IRC client/server ("ubos-irc"), I wouldn't want to include the binaries or the source as part of ubos-irc, right? Only a reference to it. But then, if the server is a binary that requires building, I would want to make a package for it. Would that package go in the UBOS repo, or maybe the Arch AUR?

And as far as why I thought it should be a role: Most of the packages on UBOS say, "Here are some files. Put them over there. Configure Apache and maybe Tomcat thusly." and it works. However in my case, I have a whole new executable (or two) to run. How do I specify that the system should run the program? How do I tell it to monitor that program (supervisord, etc)? One interesting thing is that in some cases, I may want one instance of the application running for the whole system, such as the IRC server. On the other hand I may want one program per "UBOS app", such as the IRC client. (maybe a user wants multiple IRC clients pointing at the same server, who knows.)

BTW any reason you use md5 and not sha256?

Thanks again,

- Dan

[Johannes Ernst]

On Aug 1, 2015, at 18:50, Dan Krol <orbl...@gmail.com> wrote:

As far as packaging the application, I think there may be a misunderstanding possibly based on a bad assumption I'm making, which I should probably lay out. My understanding is that the application package, and the "UBOS package", would be two different different things. For instance, the UBOS package for Wordpress does not contain the entire source code for Wordpress, but merely a reference to it.

It looks like it but no. If you go to https://github.com/uboslinux/ubos-wordpress/tree/master/wordpress, you indeed find a very small number of files, and none of the Wordpress code itself. But once you have run

> makepkg

the resulting package (wordpress-…pkg…xz) contains all of Wordpress.

Though, interestingly enough, what it references doesn't seem to be a separate Arch or UBOS package either; rather it's a URL to download straight from the Wordpress project's website, plus an md5sum.

This is straight Arch and has nothing UBOS-specific to it: https://wiki.archlinux.org/index.php/Makepkg

What if the referenced package had binaries, would you do the same?

Yes, the makepkg script (which reads PKGBUILD) needs to compile whatever needs compiling. The PKGBUILD specifies which architectures are allowed.

So if you want your package to run on ARM7 and x86_64, you need to makepkg twice on suitable hardware (or set up cross-compile, although I don’t know how to do that — I build on x86_64, Raspberry Pi 1 and Raspberry Pi 2.)

Wouldn't really work for Raspberry Pi, unless the project also supplied ARM binaries. So, assuming it required a build, don't most OS repositories host the built binaries? It would seem you'd want something it to be in the Arch or UBOS package repo. (I've never packaged for an OS before so please forgive any ignorance here).

UBOS has things like http://depot.ubos.net/yellow/x86_64 and http://depot.ubos.net/yellow/armv6l etc.

So following this pattern, if I were to make a UBOS package for the IRC client/server ("ubos-irc"), I wouldn't want to include the binaries or the source as part of ubos-irc, right? Only a reference to it. But then, if the server is a binary that requires building, I would want to make a package for it. Would that package go in the UBOS repo, or maybe the Arch AUR?

UBOS is unrelated to AUR, although one could conceivably pull from there. 

The primary application code should remain wherever the developers of the app like to have it. E.g. Ben W has Known’s releases at assets.withknown.com, which is why UBOS’s PKGBUILD for Known/idno points to there: https://github.com/uboslinux/ubos-idno/blob/master/idno/PKGBUILD

The packager in this case is different from the developer; the packager (here, me) chooses were to put it, and I put it on github. When I (or anybody else) runs the build of the package, they checkout the packager’s PKGBUILD and aux files, run makepkg, which will automatically (if needed), download the code from the developer. Based on what PKGBUILD says, it might make patches first before creating the package (we do that for owncloud, in case you want an example)

And as far as why I thought it should be a role: Most of the packages on UBOS say, "Here are some files. Put them over there. Configure Apache and maybe Tomcat thusly." and it works. However in my case, I have a whole new executable (or two) to run. How do I specify that the system should run the program?

What exactly do you need to run and how? Is it a daemon? If so, you should run it as a systemd .service.

How do I tell it to monitor that program (supervisord, etc)? One interesting thing is that in some cases, I may want one instance of the application running for the whole system, such as the IRC server. On the other hand I may want one program per "UBOS app", such as the IRC client. (maybe a user wants multiple IRC clients pointing at the same server, who knows.)

systemd is pretty flexible in that regard: foo.service vs foo@.service, and it’s strongly recommended, from a UBOS perspective, to use systemd facilities.

BTW any reason you use md5 and not sha256?

You choice, either works.

Thanks again,

- Dan

--
You received this message because you are subscribed to the Google Groups "UBOS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ubos-dev+u...@googlegroups.com.
To post to this group, send email to ubos...@googlegroups.com.
Visit this group at http://groups.google.com/group/ubos-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/ubos-dev/CAAWRcS9SAjrE7arwaE2PWNjseAbqomSKn810EY%3DW7QR3Akc9rg%40mail.gmail.com.

[Dan Krol]

Hello,

> Yes, the makepkg script (which reads PKGBUILD) needs to compile whatever needs compiling. The PKGBUILD specifies which architectures are allowed.

Oh, okay, I think it just clicked for me. What goes into the various UBOS package git repos are not packages as such, but rather code that is used to build packages (same as would be for any Arch packages). It is either up to the (savvy) UBOS user to git clone, build, and pacman-install the packages before installing them with ubos-admin, or up to the UBOS repository maintainer to git clone, build, and commit the packages for the (not necessarily as savvy) UBOS user to pacman-install them before using ubos-admin. So, that's how binaries would be made available from a repo. Not sure why this didn't occur to me before. I think this was part of the source of my confusion. So, it sounds like if UBOS were to accept such a package as I've described, your build system would have to be running makepkg twice, and sticking the result into your repo, if x86 and Raspberry Pi would be supported. Do I get the right idea now?

What exactly do you need to run and how? Is it a daemon? If so, you should run it as a systemd .service.

Yes, it is a daemon. Cool, I will look into that. However, I don't see an example in the UBOS documentation. Is there one you could point me to?

Thanks,

Dan

[Dan Krol]

Ah, found one in mediagoblin. Interesting that it's under apache in the manifest.

[Dan Krol]

Cool, got the systemd part to work!

[Johannes Ernst]

On Aug 22, 2015, at 18:50, Dan Krol <orbl...@gmail.com> wrote:

Hello,

> Yes, the makepkg script (which reads PKGBUILD) needs to compile whatever needs compiling. The PKGBUILD specifies which architectures are allowed.

Oh, okay, I think it just clicked for me. What goes into the various UBOS package git repos are not packages as such, but rather code that is used to build packages (same as would be for any Arch packages). It is either up to the (savvy) UBOS user to git clone, build, and pacman-install the packages before installing them with ubos-admin, or up to the UBOS repository maintainer to git clone, build, and commit the packages for the (not necessarily as savvy) UBOS user to pacman-install them before using ubos-admin. So, that's how binaries would be made available from a repo. Not sure why this didn't occur to me before. I think this was part of the source of my confusion. So, it sounds like if UBOS were to accept such a package as I've described, your build system would have to be running makepkg twice, and sticking the result into your repo, if x86 and Raspberry Pi would be supported. Do I get the right idea now?

Yes. I’m currently running builds for x86_64 (on a PC), arm v6 (on a Raspberry Pi) and arm v7 (on a Raspberry Pi 2, for itself and other systems like the BeagleBone Black).

What exactly do you need to run and how? Is it a daemon? If so, you should run it as a systemd .service.

Yes, it is a daemon. Cool, I will look into that. However, I don't see an example in the UBOS documentation. Is there one you could point me to?

It depends on whether you need to run a single daemon, or one daemon per installation of your app. If the latter, you want to have an AppConfigItem in your manifest of type systemd-service, see http://ubos.net/docs/developers/manifest/roles.html

Thanks,

Dan

--
You received this message because you are subscribed to the Google Groups "UBOS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ubos-dev+u...@googlegroups.com.
To post to this group, send email to ubos...@googlegroups.com.
Visit this group at http://groups.google.com/group/ubos-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/ubos-dev/CAAWRcS_o3j%2Bo9dJsBUBnu5z01svVrQsmVSQFVBiNajVdgBv06w%40mail.gmail.com.

[Johannes Ernst]

On Aug 22, 2015, at 19:00, Dan Krol <orbl...@gmail.com> wrote:

Ah, found one in mediagoblin. Interesting that it's under apache in the manifest.

Yes, because it needs to run wherever Apache runs. (The roles, while currently not supported, are also intended to be deployable on different computers, so you can get, say, a three-tier architecture.)

-- 
You received this message because you are subscribed to the Google Groups "UBOS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ubos-dev+u...@googlegroups.com.
To post to this group, send email to ubos...@googlegroups.com.
Visit this group at http://groups.google.com/group/ubos-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/ubos-dev/CAAWRcS_T8XjDohe8qekWgV4r2k7%2BeAL777cDXFujH5p_iYXK4w%40mail.gmail.com.

[Johannes Ernst]

On Aug 22, 2015, at 20:21, Dan Krol <orbl...@gmail.com> wrote:

Cool, got the systemd part to work!

Apparently I’m too late, congratulations!

-- 
You received this message because you are subscribed to the Google Groups "UBOS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ubos-dev+u...@googlegroups.com.
To post to this group, send email to ubos...@googlegroups.com.
Visit this group at http://groups.google.com/group/ubos-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/ubos-dev/CAAWRcS-VyPiSvkDH%3DEJxqLc%2BvHLzvFNw0HiCZ%3DzHBtXojwN0wQ%40mail.gmail.com.