Building Qubes 4rc4 iso from source
Tim W
I have tried with a standard builder.conf as well as using the ./setup script to create the builder.conf.
Doing the builder.conf from example when I uncomment whonix I am asked for git username and password during #make get-sources for that. Even with it removed during #make qubes I get failures.
Using ./setup I have tried numerous combos of the various templates offered removing ones that create errors. Still I can not get a clean build. It maybe because I am picking templates that while listed do not actually work with 4.0. If this is the case which templates should work with building 4.0?
"Example last time I had it fail during the initial build steps of FC26+fully loaded with error:
Building template fc26 (logfile: build-logs/template-fc26.log)...
Makefile:297: recipe for target 'template-local-fc26+fullyloaded' failed
make: *** [template-local-fc26+fullyloaded] Error 1"
I did have some null failures but I think it had to do with misdirection of log.txt as it did not stop the build example below:
" Installing core RPM packages...
install-info: No such file or directory for /dev/null
install-info: No such file or directory for /dev/null
install-info: No such file or directory for /dev/null
install-info: No such file or directory for /dev/null
Failed to connect to bus: No such file or directory
Failed to set locale, defaulting to C
Running in chroot, ignoring request.
-> Installing package groups...
Failed to set locale, defaulting to C"
All I really care about in the end is a way to get a clean build of the latest release being 4.0rc4.
I did not have these issues before building 3.1 and 3.2.
Ideally I would like to build an iso with FC25, Fc26, Debian Stretch, Whonix, Centos7 would like the standard and minimal where they are available. Not sure exactly what is difference between the standard Template and fully-loaded one. I get minimal but did not realize there were fully loaded and standard. I tried picking all of them but got failures. After removing a number of them and still getting failures I decided it was time to ask some questions as my search-fu turned up nothing helpful.
I am working in a 3.1 fc23 standalone VM in it phython2.sh is not found in repo so I loaded it as a rpm. If you load only python.sh you get error looking for python2.sh Other than that I have not seen anything error specific to building in FC23
Tim
Cheers,
awokd
> Has anyone accomplished a clean build?
sudo dnf install git createrepo rpm-build make wget rpmdevtools dialog
rpm-sign gnupg dpkg-dev debootstrap python2-sh
I used the Fedora 26-standard, Debian 9-standard, and whonix-gw & ws
templates. They should be the safest bets because they ship with Qubes.
I'll kick off another build in case there's an rc4 regression; haven't
tried recently.
Tim W
Thanks will try that as the most basic.
Did you use ./setup or just run off the master build.conf in examples directory?
awokd
>
> Thanks will try that as the most basic.
>
>
> Did you use ./setup or just run off the master build.conf in examples
> directory?
https://github.com/QubesOS/qubes-issues/issues/3426, just skip the text
file edits and choose 4.0 instead of 3.2.
Tim W
in https://www.qubes-os.org/doc/qubes-builder/ Building Qubes From Scratch
Then you have the build directions that there is notes to follow the Archlinux directions for using the builder tools and they do not have them but then it includes python.sh rather than python2.sh . The latter is definitely needed.
Then finally in the README.md file in the qubes-builder which you are directed to read it has you install perl-open but not dpkg-dev debootstrap and a few others. So we are basically given directions to go to at least 3 different places that all have from slightly different to completely different instructions for building qubes.
IMHO for the advanced user who wants to build qubes from source the ./setup script is by far the easiest to follow as it being a all inclusive script handles many steps for you in a nice basic gui. Issue is we need to have some very basic steps of what will work with each OS version. It would only take 10-15 min to update a markup page for each os. Everything is there except what works with what OS and frankly that's all in terms of instructions that should have to change ever in qubes current xen based format. If a dev could even just post a list of what templates work with the version in the group I or others can easily update the doc page. I have helped in that area before as its one area IMO any and all qubes users can help out with and give back to the project.
Tim W
Thanks am starting the process now. I always prefer to build my own OS and prog from source whenever possible. Makes updating a bit more tedious but it keeps me more uptodate and with less possible points for code to be altered + if I can't build something it concerns me.
Tim W
It failed with below:
Downloading Packages:
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=fedora-25&arch=x86_64
/home/user/qubes-builder/qubes-src/builder-fedora/Makefile.fedora:111: recipe for target 'dist-build-dep' failed
make[2]: *** [dist-build-dep] Error 1
make[2]: Leaving directory '/home/user/qubes-builder'
Makefile.generic:156: recipe for target 'packages' failed
make[1]: *** [packages] Error 1
make[1]: Leaving directory '/home/user/qubes-builder'
Makefile:224: recipe for target 'core-libvirt-dom0' failed
make: *** [core-libvirt-dom0] Error 1
An issue with the link in the makefile?
awokd
> packages'. Error: Error downloading packages:
> Status code: 503 for
> https://mirrors.fedoraproject.org/metalink?repo=fedora-25&arch=x86_64
>
a bunch of those. All you can do is run "make qubes" again and cross your
fingers.
Tim W
Well that royally sucks. Its not as if the build process is only 5-10 mins. That was 30-45 mins to hit that error.
looked at the makefile.fedora and it looks like this section. Does it look correct I was wondering if something might change with depeciation of f25
ifdef FEDORA_MIRROR
YUM_OPTS += --setopt=fedora.baseurl=$(patsubst %/,%,$(FEDORA_MIRROR))/releases/$(subst fc,,$(DIST))/Everything/x86_64/os/
YUM_OPTS += --setopt=updates.baseurl=$(patsubst %/,%,$(FEDORA_MIRROR))/updates/$(subst fc,,$(DIST))/x86_64/
endif
Ill try running it yet again.
awokd
> On Saturday, February 3, 2018 at 11:39:09 PM UTC-5, awokd wrote:
>
>> On Sun, February 4, 2018 4:21 am, Tim W wrote:
>>
>>
>>> packages'. Error: Error downloading packages: Status code: 503 for
>>> https://mirrors.fedoraproject.org/metalink?repo=fedora-25&arch=x86_64
>>>
>>
>>>
>>> An issue with the link in the makefile?
>>>
>>
>> Probably a Fedora repo issue, maybe it's being updated. You might run
>> into a bunch of those. All you can do is run "make qubes" again and
>> cross your fingers.
>
> Well that royally sucks. Its not as if the build process is only 5-10
> mins. That was 30-45 mins to hit that error.
>
> looked at the makefile.fedora and it looks like this section. Does it
> look correct I was wondering if something might change with depeciation
> of f25
Another thing you can try if you're frequently getting download issues
(status code 503 is server issue, not missing file) is to build one group
of components at a time instead of the full list (which "make qubes"
does).
If you want to try that instead, do "make help" and look for the list of
components towards the end. It's like 10 lines long. Then start copying
and pasting them into make, one batch at a time, like "make vmm-xen
core-libvirt core-vchan-xen" for the first batch. That way if it fails you
don't have to go so far back. Make sure to run them in order.
Tim W
great idea. I had done that on templates when working out issues back before we all got archlinux building well. Did not think of if for this. So far this time its still running on the build so hopefully in a few hours I will have an iso. We will see.
After that I think I will start adding in other templates to make a list of what works. As I can leave it to do its thing will not be a big waste of time. I am thinking minimal templates should also work. Then its onto Centos. I prefer it leaps and bounds over Fedora. IMO its the ideal dom0 if we need to stick to rpm based with its long LTS. Now with the longer linux kernel support it will go great together for a long term stable system without having to upgrade.
Tim W
OK I was able to get a standard build done to iso. That is standard FC26 + Strech-Debain + Whonix.
I also successful built standard + minimal of those to iso as well.
Next I will try adding in certain other templates 1 by 1.
A correction on the instructions to build these 4.0 ISOs:
You need significantly more private store drive space in the VM. I am using a stand-alone VM FC23 so I have more initial space taken up. Still with an app vm its 2x what the old docs recommend as a minimum.
For just the standard build I needed 60gigbyes.
For the standard + minimal 65 gig just barely made it. Best to have 70 gigs.
My guess if you wanted to build all the non EOL templates for 4.0rc4 if they all can be built would be 100 gigs.
I will continue to see what will build and the space needed. My guess is some laptops especially that are running ssd will not have the drive space to build depending on what's used up already for your system data. Looks like I finally have to add the 1tb Samsung Pro SED SSD in the optical bay for vm user data.
Still not sure what program differences between the full-loaded-template vs Standard-templates. For me I would rather manually add the programs I wanted in each area. I may try the builds for those so we know if the build or not.
I may also try the last template version to be EOL'd such as fc25 for fedora etc but that will be last. Too bad I do not have quad xeons 1000 +gigs of ram workstations vs the T440 with I7 + 16gig. I could get these built in 15-30 min rather than 3 hrs.
In the attachment is the QM showing Build-Qubes VM Size used for at the end of #make iso with FC26-Standard+FC26-Minimal & Stretch-Standard + Stretch-Minimal + Whonix 64309MB
Cheers,
Tim
Unman
the currently supported version, and should be complete.
The archlinux instructions are quite out of date and the README definitely
needs to be updated.
I'm not sure what you mean by OS version, or most of your last
paragraph. Can you clarify?
Unman
You should be able to build all the templates referenced in the setup
program. If you can't, then (barring network problems) it's a bug.
Its a moot point whether you should be building some of the older,
eol templates, and whether they should be available to build.
Tim W
Correct on the older templates. The oldest I would consider is fc25 and wheezy etc.. for it to be working correctly without bug I would expect to be able to choose all none EOL templates together and have it build. Would need 100+ gigs but it should work. That is what I am working towards.
My original issue with setup script might have been part network and part gpg sig issue. Just would like to know and then update the offical Doc etc pages with correct info where needed.
The issue with archlinux is it is referenced at least twice in other docs as to being the most uptodate instructions for building qubes and templates. Basically much of the docs are a mess as they have conflicting missing dif info.
Tim W
I just added Centos 7 Standard template with the ones that had already built correct and as before it errored out with the following:
-----------------------------------
Building core-admin-client (rpm_spec/qubes-core-admin-client.spec) for centos7 vm (logfile: build-logs/core-admin-client-vm-centos7.log)
--> build failed!
reading sources... [ 92%] manpages/qvm-tags
reading sources... [ 96%] manpages/qvm-unpause
reading sources... [100%] manpages/qvm-volume
/home/user/qubes-src/core-admin-client/doc/manpages/qvm-features.rst:91: ERROR: Unknown interpreted text role "py:pbj".
/home/user/qubes-src/core-admin-client/doc/manpages/qvm-firewall.rst:73: WARNING: Bullet list ends without a blank line; unexpected unindent.
looking for now-outdated files... none found
pickling environment... done
checking consistency... /home/user/qubes-src/core-admin-client/doc/manpages/index.rst:: WARNING: document isn't included in any toctree
done
writing... qvm-backup-restore.1 { } Checking arguments for 'qvm-backup-restore'
Exception occurred:
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/user/qubes-src/core-admin-client/qubesadmin/tools/qvm_backup_restore.py", line 4
SyntaxError: Non-ASCII character '\xc3' in file /home/user/qubes-src/core-admin-client/qubesadmin/tools/qvm_backup_restore.py on line 4, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
The full traceback has been saved in /tmp/sphinx-err-JOw4sG.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
Either send bugs to the mailing list at <http://groups.google.com/group/sphinx-dev/>,
or report them in the tracker at <http://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks!
make: *** [man] Error 1
make: Leaving directory `/home/user/qubes-src/core-admin-client/doc'
error: Bad exit status from /var/tmp/rpm-tmp.Hd675A (%build)
-----------------------------------
Any ideas?
Tim W
Next I guess I will try the fully loaded versions of FC26 and Stretch as they have the most likelihood of working without a bug. Then move onto archlinux as that has seen a long line of attention over the years.
Cheers,
Tim