kraileth initial request

[John Marino]

On Sat, Sep 23, 2017 at 4:29 AM, 'kraileth' via Ravenports <raven...@googlegroups.com> wrote:

Hi John,

thanks for your efforts with Ravenports! I tried it out on FreeBSD and (Debian) Linux. I'd like to do a little project with it next for which I had planned to use Pkgsrc but I'm very much in favor of Ravenport's more modern approach. (Eventually I hope to introduce it at work, too, but I need to get familiar with it before I can make a decision if I should attempt to convince management)

Here's a quick list of things that I'm currently missing for my test project:

sysutils/py-salt
devel/py-pygit2
devel/gitolite
sysutils/iocell
security/sudo
www/uwsgi
www/py-flask
dns/bind911

This should suffice to put up a jailhost system and create as well as configure both a web jail and a DNS jail automatically using SaltStack.

I've read the Github wiki (a lot of the rules for the specfiles are really pedantic - love it!) and tried to create a port myself. I managed to compile a buildsheet just to get an idea of how everything is working. It doesn't compile, yet, and I run out of time yesterday. But the whole process looks nice. And while you asked which ports are needed by people, in the long run I'll most likely dig deeper into creating some more ports myself.

Hi kraileth,

This is just confirming that all of these ports have since been added to Ravenports.

[kraileth]

On Saturday, September 30, 2017 at 7:36:33 PM UTC+2, John Marino wrote:

On Saturday, September 30, 2017 at 6:22:42 AM UTC-5, kraileth wrote:

Another question: As long as zstd isn't used - do you see a problem with building a package repository with ravenadm and then pointing pkg on a fresh FreeBSD installation to it? If I disable the FreeBSD repo and put /raven/bin and so on into the PATH, this should work, right?

It's unclear to me which case you mean.  A) use an existing pkg(8) and add a ravenports repository conf file or B) use the ravenports-built pkg(8) and add a conf file for an official freebsd pkg(8).

Regardless of which case you mean, I think you'll have problems doing it. In my experience, if you try to update one repository, pkg(8) may try to delete or update packages from the other repository.  It gets confused.   So my recommendation is to keep two pkg(8)'s on the system: One for freebsd repository and one for ravenports repository.  Refer to the pkg executable by the full path.  Then for sure you won't have any cross-repository confusion.

I'm copying this over from the other thread. No, I didn't mean to mix packages from FreeBSD and from Ravenports in either variant. I was just thinking to use FreeBSD's Pkg (i.e. doing a "pkg bootstrap" on a fresh installation) and then immediately disable the FreeBSD repo + enable a Ravenports one. I've done this in the meantime (before zstd was implemented) and it works fine.

I did this for a simple reason: I'm trying to eventually introduce Ravenports in the company that I'm working for. Most of the other admins do not share my passion for *BSD. They know how to use Pkg, though. Different package names should not be too much of a problem but in general I'd like to keep things as they are as much as possible. Now with zstd implemented, I don't have a choice in that regard, anyway (execpt FreeBSD is going to adopt the changes, that is. Allan Jude mentioned that he'd be interested in getting zstd compression an option for ZFS - probably there's also interest in pulling it in for mainline Pkg?).

Somebody else already asked about changing localbase to something else outside of /raven. I read your answer that this requires rebuilding the whole Ravenports infrastructure and is not an easy thing to do. It's not very high priority, but in the long run I'd like to take a look at setting it to /usr/local so that Ravenports is getting much closer to a drop-in replacement for FreeBSD packages. For the time being I'm going to "cheat" by symlinking /usr/local/bin -> /raven/bin, /usr/local/sbin -> /raven/sbin and /usr/local/etc -> /raven/etc and so on.

I'll be starting over with a clean system and rebuilding my repo. Looking forward to working with the new zstd archives!

[John Marino]

On Sunday, October 8, 2017 at 6:23:33 AM UTC-5, kraileth wrote:

I did this for a simple reason: I'm trying to eventually introduce Ravenports in the company that I'm working for. Most of the other admins do not share my passion for *BSD. They know how to use Pkg, though. Different package names should not be too much of a problem but in general I'd like to keep things as they are as much as possible. Now with zstd implemented, I don't have a choice in that regard, anyway (execpt FreeBSD is going to adopt the changes, that is. Allan Jude mentioned that he'd be interested in getting zstd compression an option for ZFS - probably there's also interest in pulling it in for mainline Pkg?).

I would guess FreeBSD is not going to take my modifications to package for 2 reasons:

1) They have a long-term plan to remove libarchive and create a new package format, and then use zstd directly to compress packages.  That is probably a year away, but it is a superior approach.  When that modification is released, Ravenports will likely adopt it (which is not only better performance, it's less linkage and I can drop the custom patches).

2) the "repo" command is hard coded to .txz format.  You can't modify it.  Since they aren't going to drop txz before (1) happens, they can't adopt my patches unless they also fix the "repo" command to allow the package format to be something other than "txz".  (which means all the other support formats are useless, meaning libarchive is mostly useless)

 

Somebody else already asked about changing localbase to something else outside of /raven. I read your answer that this requires rebuilding the whole Ravenports infrastructure and is not an easy thing to do. It's not very high priority, but in the long run I'd like to take a look at setting it to /usr/local so that Ravenports is getting much closer to a drop-in replacement for FreeBSD packages. For the time being I'm going to "cheat" by symlinking /usr/local/bin -> /raven/bin, /usr/local/sbin -> /raven/sbin and /usr/local/etc -> /raven/etc and so on.

I wouldn't advise this.  Even if you take care of the linking issue by using symlinks instead of /bin/mv, the config files are still looked for at /raven/etc.

I wouldn't say relocating the localbase is a particularly hard thing to do either.  I haven't done it myself (so there may be a gotcha) but basically it's just starting with an empty package set, then rebuilding the ravensys-root:* package, the gcc7 package, then the ravensys-toolchain package with the new localbase).  Then you install them, use another empty package directory and rebuild all the needed packages.  

But you are doomed to building your own packages always at that point.

And you lose the option to use freebsd packages simultaneously with ravenports.  Currently they can coexist.

Don't forget third-party vendors also like to use /usr/local,

 

I'll be starting over with a clean system and rebuilding my repo. Looking forward to working with the new zstd archives!

They make a huge difference for a package builder.

llvm40 build time dropped about 50 minutes.  Instead of taking 52 minutes or so to compress packages, zstd did it in about 90 seconds.  The packages are bigger (maybe 25%) but that just means compression drops from 7.5% to 10%.  (using general numbers, every package is different of course).  The bigger packages are well worth the speed increases for the builder and even the user (they unpack a bit faster than xz)

 

[John Marino]

Also, don't forget your company can actually contract me to do some of this (including building packages for you at any localbase for regularly scheduled deliveries).  The possibilities are on the website.

(btw, I just mirrored ravenports.ironwolf.systems to http://www.ravenports.com which I haven't announced yet)

[kraileth]

On Sunday, October 8, 2017 at 2:39:23 PM UTC+2, John Marino wrote:

Also, don't forget your company can actually contract me to do some of this (including building packages for you at any localbase for regularly scheduled deliveries).  The possibilities are on the website.

Hi John,

kind of forgot to reply to this. I'm aware that you are offering professional services - the problem here is that I'm only an employed operator. For that reason I cannot make any decisions on my own. And while I think that Ravenports would fit very well with the company I work for, I have to convince others first. Fortunately my boss is fairly open to improvements and not generally afraid of trying new things. Still I would rather take my time, learning how to use Ravenports properly and do a little project in my free time so that I actually have something to show. That's always better than just talking about a possible concept.

Also I'm having another small issue: I use a formerly spare machine now dedicated to tinker with Ravenports. It's not very fancy hardware, but it's all I have available right now. In general things are working great, but building llvm fails. From my understanding it finishes to compile but the linking phase takes too much time and the process eventually gets killed by the watchdog. Well, llvm4.0 is a beast of a package and I've read that there were issues even with the much smaller llvm37 or so with Synth. It looks like you increased the timeout quite a bit, but in that special case it still doesn't suffice. With Synth there's the compile-time option to deactivate the watchdog monitor. Is it possible to deactivate it somehow, too, for Ravenadm or do I have to use a prebuilt package from either the official repo or from another system?

[John Marino]

The base time is 40 minutes, which means in practical terms you're probably getting 60 to 120 minutes of time to complete a single link.  Are you sure it's not really locked up?  even a weak machine should be able to link one library in less than an hour.

I've hit llvm problems like this before.  It's the reason llvm is limited to 2 jobs.

First question though, are you sure you have enough swap space?  You need something ridiculous like 20G.  If you run out of swap, it may lock up rather than exit.

On FreeBSD for me where there is no SSD-based swap, I had a create a large swap file and use it in addition to the existing 5G swap partition.  Compiled with a 2-job limit, it finally started compiling.

I would look at your swap situation on that computer before trying to disable a very (too?) generous watchdog timeout.