offline sepkg

[kevin kuh]

 I am interested in installing studioware on a bunch of offline computers. Instinctively I'd like to put an rsync mirror of your package repository in a directory on a usb pen and install straight from there. This does not seem to be an option.

Should I presume that - hacking the sepkg script to screw with MAINURL and PKGSERVER, and serve the files to myself at a localhost address - is the recommended way to do an offline install?

Kev

[David Woodfall]

On Wednesday 13 June 2018 06:07,
kevin kuh <kuh...@gmail.com> put forth the proposition:

Hmm I've not tried doing any offline installs.

I guess hacking the script URLs would be the way to do it at the
moment. I'll have to look into a proper way of doing this. Like
supporting a local folder/mount like slackpkg does.

Let me know how you get on.

-Dave

--

Linux poses a real challenge for those with a taste for late-night
hacking (and/or conversations with God).
-- Matt Welsh

.--. oo
(____)//
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~'

[David Woodfall]

On Wednesday 13 June 2018 14:18,
David Woodfall <da...@studioware.org> put forth the proposition:

> On Wednesday 13 June 2018 06:07,
> kevin kuh <kuh...@gmail.com> put forth the proposition:
> > I am interested in installing studioware on a bunch of offline computers.
> > Instinctively I'd like to put an rsync mirror of your package repository in
> > a directory on a usb pen and install straight from there. This does not
> > seem to be an option.
> >
> > Should I presume that - hacking the sepkg script to screw with MAINURL and
> > PKGSERVER, and serve the files to myself at a localhost address - is the
> > recommended way to do an offline install?
> >
> > Kev

You realise that sepkg will still want to download the source code
tarballs going by the URL in the info?

I think that you'll need to do a mass download of sources first.

> Hmm I've not tried doing any offline installs.
>
> I guess hacking the script URLs would be the way to do it at the
> moment. I'll have to look into a proper way of doing this. Like
> supporting a local folder/mount like slackpkg does.
>
> Let me know how you get on.
>
> -Dave
>
> --
>
> Linux poses a real challenge for those with a taste for late-night
> hacking (and/or conversations with God).
> -- Matt Welsh
>
> .--. oo
> (____)//
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~'
>

--

.. I used to get in more fights with SCO than I did my girlfriend, but
now, thanks to Linux, she has more than happily accepted her place back at
number one antagonist in my life..
-- Jason Stiefel, kry...@s30.nmex.com

.--. oo
(____)//
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~'

[kevin kuh]

Thank you. It has worked out fine in the end.

I have writen up a rough outlined of what I did (for other third parties who want to travel this path in the future), and added a couple of trivial suggestions below:

After your replies I thought I would be clever and save myself some effort. I thought to boot up the AlienBob live studioware ISO image then install that with his setup2hd script.

Using the live image was a big mistake and wasted a lot of time. The download was clean but the actual install image seemed badly setup. Every problem I solved brought up new problems. The C compiler was linked to a missing version of a shared library. The X11 server would not even run on the on-board intel graphics of the first system I tried. While the X11 in the stock slackware 14.2 disc was fine on the same system. You are right not to attempt to support it.


In the end I went back to Plan A.
I grabbed your SLACKBUILDS.TXT and VERSIONINFO.TXT files then used rsync to grab your sqf files. Hacked the sepkg script to point to my localhost Apache server.

> You realise that sepkg will still want to
> download the source code tarballs
> going by the URL in the info?

Yes. It wasn't too bad: thanks to you.
It was easy because your scripts already cache downloaded files. I pre-populated '/var/cache/sepkg/14.2/' with the files I would need.

The main sequence of events went like this:

awk '//{print "wget -x http://www.studioware.org/files/slackbuilds/14.2/" $2}' SLACKBUILDS.TXT > getme1.bat

wget is available for linux and windows. So redirect the output of the above command to a file and thus generate a bash script or a Windows-cmd '.bat' script that will get the required SlackBuild files with an online machine.

Once obtained; put them where you tricked sepkg to look for them.

And then to get the actual downloads:

find -iname *.tar.gz |xargs -n 1 tar xzvf

..and then to generate a download script..

#!/bin/bash
for F in `find -iname *.info`
do
(                                                    
source $F                                            
for D in $DOWNLOAD                                    
do                                                    
echo "wget --no-check-certificate $D"                
done                                                  
)                                                    
                                                     
done

                                                 

Again, redirect the output of the above script to a file and thus generate a script compatible with running from bash or being a Windows-cmd '.bat' script. Either way use it to get the files with an online machine.
Then put the obtained files in '/var/cache/sepkg/14.2/'.


Curiously, After the first time I ran

sepkg -u

..offline, I was suprised to find that

sepkg -q audio

..Did nothing. But

sepkg -i < audio.sqf

..worked fine, and seemed to kick the dust out whatever was clogging up the system.
I must have done something wrong since -q worked normally after that.
I figured I'd mention it here if anyone else on the same course has trouble there.


I had a few other hickups, so here are some Suggestions:

1,  The package protobuf-3.1.0 has hidden dependencies on Google Mock (gmock-1.7.0) and Google Test (1.7.0).
When protobuf's autogen.sh discovers they are not installed, it quietly tries to download and install them itself. It is an ugly oddity. There should probably be explicit gmock packages in studioware to install before protobuf. Alternatively, maybe include the googlemock and googletest downloads in the DOWNLOAD="" section of 'protobuf.info' and extract them where protobuf expects them?

Having one slackware package download and install other packages on it's own - is not cool.


2, It would be nice if you rechecked the cache before redownloading missing files.

When a download fails during a build: you helpfully offer an option to redownload it. Unfortunately when this option is chosen you do not recheck if the file has mysteriously appeared in the cache first. You literally try to redownload it. Your script is sensible enough, but in my case I found this quite inconvienient. It often took well over half an hour into a queued build to realise a file was missing. This then ment aborting the whole install.. getting the file into the cache.. restarting the build from scratch.. and then waiting for the next file that github or sourceforce had moved deleted or otherwise screwed up. GraphicsMagick-1.3.24.tar.xz for example seems to have been one of the files dropped from sourceforce which needed hunting down.

If you made this change and were to add an option to sepkg to specify what path to use as the cache directory (call it 'downloads-path' or something): it would get you 90% of the way to having an offline version.

In the end I got fed up and wrote a simple script to produce a script that would list all the missing files from the cache up front:

#!/bin/bash
echo "echo List of missing files:"
echo "cd /var/cache/sepkg/14.2/"
for F in `find -iname *.info`                                                      
do                                                                                
(                                                                                  
#echo "M $F E"                                                                    
source $F                                                                          
#echo "M $F $DOWNLOAD E"                                                          
for D in $DOWNLOAD                                                                
do                                                                                
#echo "A $F $D B"                                                                  
echo "$D" | sed 's%.*/%%' | xargs -n 1 -I Y echo "test -s Y || echo Y"            
done                                                                              
)
echo "echo --End of list"

3, Irregularly capitalised sqf filepaths.
I downloaded some files via a new windows 10 machine. Somehow, it still managed to screw up filepath captialisations in the year 2018.

There are literally only three files and three directory names I had trouble with for this reason. It is not a big deal, and best solved by nuking Microsoft from orbit. But when you make a new major version you could maybe consider dropping capital letters from your own directory tree. People normally write 'OpenGL' as 'opengl' in filenames, so why should OpenAV be 'openAV' twice in your sqf filepath? Besides these mixed capitals are so infrequently used in your tree that they look kinda ugly; and they don't really fit the standard linux style. But mostly - I request the change because it took me ages to track-down and work out why my offline build wasn't finding the three files that I thought were obviously present! Doh!

system/system/DisplayCAL/                                ->    system/system/displaycal/
system/system/DisplayCAL/DisplayCAL.sqf          ->    system/system/displaycal/displaycal.sqf

audio/sequencers/terminatorX/                              ->    audio/sequencers/terminatorx/
audio/sequencers/terminatorX/terminatorX.sqf        ->    audio/sequencers/terminatorx/terminatorx.sqf

audio/plugins/openAV-ArtyFX/                               ->    audio/plugins/openav-artyfx/
audio/plugins/openAV-ArtyFX/openAV-ArtyFX.sqf    ->    audio/plugins/openav-artyfx/openav-artyfx.sqf

But I am happy now anyway.

Thanks. Studioware is Great Stuff. I much appreciate the work you have put in.