qubes-tor needs to be updated(?)

[Axon]

Attempting to install qubes-tor tries to download from
http://deb.torproject.org/torproject.org/rpm/fc/18/[...], which no
longer exists. It looks like there now exist directories only for 19 and
20. See here: http://deb.torproject.org/torproject.org/rpm/fc

(Is it even possible to fix this without upgrading the template to
Fedora 19 or 20?)

[Marek Marczykowski-Górecki]

I'm afraid the only option is to build tor yourself, or switch to older
version available in Fedora repository.

We are planning to release Fedora 20 template soon, but this doesn't solve
fc18 problem. Perhaps there is some mirror for fc18 packages?

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[clew...@gmail.com]

Hi Marek,

Would it be possible to update the qubes-tor package to use this location instead?

https://deb.torproject.org/torproject.org/rpm/tor-testing/fc/18/

I see this is a newer version of tor, so I don't know if it would break anything.

Would it be possible to update the qubes-tor package to use this location instead?

[Marek Marczykowski-Górecki]

Worth a try - just update an URL in /etc/yum.repos.d/torproject.repo should do
the work.

[Christopher Lewis]

I was afraid you would say that.  I did try that, but yum says it cannot connect when I issue the command "sudo yum install qubes-tor".  I believe it is a 500 error.  I will check again and give more specifics.
 

[Marek Marczykowski-Górecki]

On 24.12.2013 23:29, Christopher Lewis wrote:
>
>
> On Tuesday, December 24, 2013 4:41:50 PM UTC-5, Marek Marczykowski-Górecki
> wrote:
>>

> I was afraid you would say that. I did try that, but yum says it cannot
> connect when I issue the command "sudo yum install qubes-tor". I believe
> it is a 500 error. I will check again and give more specifics.

Have you pointed x86_64 directory?

[sam]

Not to pry but how would users be able to upgrade to Fedora 20 ?
I am asking because I want to use the more updated version of Fedora and Tor to solve a lot of security issues.

What is usually needed to under go for the Qubes fedora template to be ready for the public as a posed to just downloading the latest version and installing it as a template ?

[sam]

After editing the repo file to use the tor-testing URL

"https://deb.torproject.org/torproject.org/rpm/tor-testing/fc/18/x86_64/"

It now works and will download albeit an alpha version of the latest Tor release.

I have found a problem tho after installing and finding that it did not respond to the check.torproject.org status page ( server not found )
In the log messages it displays an error: " Couldn't open /rw/userlocal/lib/qubes-tor/state.tmp " (/rw/userlocal/lib/qubes-tor/state) for writing: No such file or directory.
The directory is there and I can see it placing files in there so it must be working on the retry it states in the log.

On the TorVM docs it says that in R2B3 the user does not need to create the folder "/rw/usrlocal/etc/qubes-tor"
But from checking it isn't automatically created either and I can not locate the torrc file and think that this might be why I am unable to use Tor to connect to anything.

[Christopher Lewis]

Don't forget to set the netvm for your torvm, otherwise there's no network connection.

Well, I got to fool around with this some more, and finally have a working torvm.  Unfortunately, I never was able to get yum to connect to the torproject.org repo.  Instead, I just downloaded the tor RPM from the tor-testing location and installed it manually.  Then, I could install qubes-tor and it just used the existing tor installation.  Otherwise, I did everything just as the torvm guide said.  I did not create the /rw/usrlocal/etc/qubes-tor files, and they are still not there now.

[Marek Marczykowski-Górecki]

The torrc file simply isn't required to start tor - it was in R2B2. If you
want some custom settings, you can create the file.

[patrick...@gmail.com]

Any update on this? Even the workaround doesn't seem to work any more (updated repo doesn't have the file any more, either). I tried going through the original tutorial [1] but it is also no longer accurate. On a side note, for people (like myself) that are really interested in how it all fits together, an updated guide like Joanna's (without qubes-specific packages) would be really nice.

Thanks,
Patrick

[1] http://theinvisiblethings.blogspot.com/2011/09/playing-with-qubes-networking-for-fun.html

[Joanna Rutkowska]

On 03/23/14 01:23, patrick...@gmail.com wrote:
> Any update on this? Even the workaround doesn't seem to work any more
> (updated repo doesn't have the file any more, either). I tried going
> through the original tutorial [1] but it is also no longer accurate. On a
> side note, for people (like myself) that are really interested in how it
> all fits together, an updated guide like Joanna's (without qubes-specific
> packages) would be really nice.
>
> Thanks,
> Patrick
>
> [1]
> http://theinvisiblethings.blogspot.com/2011/09/playing-with-qubes-networking-for-fun.html
>

What kind of updates would you expect? All the principles are still the
same...

joanna.

[Manuel Amador (Rudd-O)]

Joanna,

On 03/23/14 03:01, Joanna Rutkowska wrote:
> What kind of updates would you expect? All the principles are still
> the same... joanna.

If the documentation is out of date, you can safely assume your
interlocutor expects a documentation update.

If the software can't be made to work, you can safely assume your
interlocutor expects a software update, workaround, or some other method
to get the software to work. And then a documentation update.

[Patrick Schless]

> What kind of updates would you expect? All the principles are still
> the same... joanna.

If the documentation is out of date, you can safely assume your
interlocutor expects a documentation update.

If the software can't be made to work, you can safely assume your
interlocutor expects a software update, workaround, or some other method
to get the software to work.  And then a documentation update.

I should have been more clear, as there are a few different things going on here.

- This thread seems to say that the fedora 20 update is the real solution to the qubes-tor package problem. So I'm wondering if that's imminent (in which case I shouldn't bother trying to get a work-around figured out).

- Joanna's blog post about setting up proxy vms for various purposes (TOR, work VPN, etc) was well-written and interesting. The steps listed, though, are no longer valid. I would be happy create new documentation on this, if I can figure out how to get it to work. The work VPN proxyvm is my ultimate goal, which is why, while I'm curious about the qubes-tor package, ultimately I'm more interested in updated docs. Since this is a separate question from the qubes-tor package, maybe it makes more sense to break it out into it's own thread. I'll list the steps I went through and be more specific about what I'm seeing in a post to qubes-users later today.

Thanks,
Patrick

 

[Patrick Schless]

I figured out the problem I was having with following the blog post:

- The start_tor_proxy.sh script reads qubes_ip from xenstore. That should be qubes-ip

- The blog post doesn't say this (or at least I didn't see it), but you have to set a netvm for the torvm (qvm-prefs -s torvm netvm firewallvm) otherwise qubes-ip is not present (because there's no network interface)

After those two changes I restarted the torvm and it came up correctly. Now the "check.torproject.org -> anon-web -> torvm -> firewallvm -> netvm" chain works like a charm (very exciting!)

Thanks,

Patrick

[Abel Luck]

On Monday 24 March 2014 00:02:58 Patrick Schless wrote:
> I figured out the problem I was having with following the blog post:
>
> - The start_tor_proxy.sh script reads qubes_ip from xenstore. That should
> be qubes-ip
> - The blog post doesn't say this (or at least I didn't see it), but you
> have to set a netvm for the torvm (qvm-prefs -s torvm netvm firewallvm)
> otherwise qubes-ip is not present (because there's no network interface)
>
> After those two changes I restarted the torvm and it came up correctly. Now
> the "check.torproject.org -> anon-web -> torvm -> firewallvm -> netvm"
> chain works like a charm (very exciting!)
>

This is documented in the TorVM documentation:

http://qubes-os.org/trac/wiki/UserDoc/TorVM

As for the qubes_ip/qubes-ip issue, that is a bug, but I've no desire to
update it because I personally consider "Update to Fedora 20 AppVMs" the
correct solution.

For the record, qubes-tor works great in Fedora 20 without any hacks, assuming
you follow the documentation.

~abel

[Bahtiar Gadimov]

Hi,

2014-03-25 10:25 GMT+01:00 Abel Luck <ab...@guardianproject.info>:
> On Monday 24 March 2014 00:02:58 Patrick Schless wrote:
>> I figured out the problem I was having with following the blog post:
>>
>> - The start_tor_proxy.sh script reads qubes_ip from xenstore. That should
>> be qubes-ip
>> - The blog post doesn't say this (or at least I didn't see it), but you
>> have to set a netvm for the torvm (qvm-prefs -s torvm netvm firewallvm)
>> otherwise qubes-ip is not present (because there's no network interface)
>>
>> After those two changes I restarted the torvm and it came up correctly. Now
>> the "check.torproject.org -> anon-web -> torvm -> firewallvm -> netvm"
>> chain works like a charm (very exciting!)
>>
>
> This is documented in the TorVM documentation:
>
> http://qubes-os.org/trac/wiki/UserDoc/TorVM
>
> As for the qubes_ip/qubes-ip issue, that is a bug, but I've no desire to
> update it because I personally consider "Update to Fedora 20 AppVMs" the
> correct solution.
>
> For the record, qubes-tor works great in Fedora 20 without any hacks, assuming
> you follow the documentation.

Nop it does not. sudo yum install qubes-tor-repo in fedora 20 and 18
does not work. This was noticed on the qubes-users around a month ago.
No one cared.

> ~abel


The only documentation available is the outdated wiki page, joannas
blog which just describes some networking/separation basics and some
discussions on the mailing list. From the discussions i see that
people solved the problem, but no where was a solution posted or it
seemed like a crude hack.

So after wasting a whole day trying to do it the "qubes way" I gave
up and just setup tor and routing by installing default tor from
fedora and configuring the fedora-20-tor-vm properly as i would with a
normal server. Works like a charm.

This makes me sad. It seems like qubes is starting to adopt a crude
works-for-me-do-not-care-about-the-rest-culture, this is extremely
worsened by lack of documentation. Most of the time i end up reading
the source code. Most of the time it kind of works I.e to figure out
what --default-script really means in qvm-create-default-dvm. But i'm
still not able to find out why window resizing in qubes only works
with xfce and kde, but any other window manger will not work.

Sorry if i disappoint some one, but this just my own personal
observation (which my be biased/wrong).

kalkin-

[Abel Luck]

On Wednesday 26 March 2014 00:56:08 Bahtiar Gadimov wrote:
> Hi,
>
> 2014-03-25 10:25 GMT+01:00 Abel Luck <ab...@guardianproject.info>:
> > On Monday 24 March 2014 00:02:58 Patrick Schless wrote:
> >> I figured out the problem I was having with following the blog post:
> >>
> >> - The start_tor_proxy.sh script reads qubes_ip from xenstore. That should
> >> be qubes-ip
> >> - The blog post doesn't say this (or at least I didn't see it), but you
> >> have to set a netvm for the torvm (qvm-prefs -s torvm netvm firewallvm)
> >> otherwise qubes-ip is not present (because there's no network interface)
> >>
> >> After those two changes I restarted the torvm and it came up correctly.
> >> Now
> >> the "check.torproject.org -> anon-web -> torvm -> firewallvm -> netvm"
> >> chain works like a charm (very exciting!)
> >
> > This is documented in the TorVM documentation:
> >
> > http://qubes-os.org/trac/wiki/UserDoc/TorVM
> >
> > As for the qubes_ip/qubes-ip issue, that is a bug, but I've no desire to
> > update it because I personally consider "Update to Fedora 20 AppVMs" the
> > correct solution.
> >
> > For the record, qubes-tor works great in Fedora 20 without any hacks,
> > assuming you follow the documentation.
>
> Nop it does not. sudo yum install qubes-tor-repo in fedora 20 and 18
> does not work. This was noticed on the qubes-users around a month ago.
> No one cared.
>

It is definitely broken in fedora 18, but I just installed qubes-tor on a
freshly upgraded Fedora 20 TemplateVM without any hitches. Which isn't too say
it is without bugs, I just didn't experience any.

> The only documentation available is the outdated wiki page, joannas
> blog which just describes some networking/separation basics and some
> discussions on the mailing list. From the discussions i see that
> people solved the problem, but no where was a solution posted or it
> seemed like a crude hack.
>

The only outdated part of the documentation I'm aware of is the name of the
TemplateVM which still references fedora 18. Is there anything else out of
date?

> So after wasting a whole day trying to do it the "qubes way" I gave
> up and just setup tor and routing by installing default tor from
> fedora and configuring the fedora-20-tor-vm properly as i would with a
> normal server. Works like a charm.
>
> This makes me sad. It seems like qubes is starting to adopt a crude
> works-for-me-do-not-care-about-the-rest-culture, this is extremely
> worsened by lack of documentation. Most of the time i end up reading
> the source code. Most of the time it kind of works I.e to figure out
> what --default-script really means in qvm-create-default-dvm. But i'm
> still not able to find out why window resizing in qubes only works
> with xfce and kde, but any other window manger will not work.
>
> Sorry if i disappoint some one, but this just my own personal
> observation (which my be biased/wrong).
>

When it comes to qubes-tor, I've been slacking in my maintainer duties, but
Marek and Vincent stepped and fixed all the outstanding bugs I know of.

That said the installation process isn't as simple as I would like. Installing
a TorVM in dom0 from an RPM would be swell, as it it would get rid of all the
manual configuration. I should look into how to do this.

Sorry you came away with this impression, but I understand. Going "off the
rails" so to speak, doing anything even a bit out of the ordinary, with Qubes
can be difficult as it is still a wild west of sorts.

Still a fantastic project though, one I'm glad to use every day.

~abel

[Rob Townley]

Haven't used TOR. How does a user of TOR test that the outgoing
"gateways" are randomized? Does a traceroute "8.8.8.8" give a
different path each time?

[Vincent Penquerc'h]

On 26/03/14 21:55, Rob Townley wrote:
> Haven't used TOR. How does a user of TOR test that the outgoing
> "gateways" are randomized? Does a traceroute "8.8.8.8" give a
> different path each time?

No, Tor will route TCP streams, but traceroute will use UDP. Tor will
*not* mean that all of your traffic will go through Tor. See the Tor FAQ
at torproject.org. check.torproject.org will let you know if your IP is
a Tor exit node. Additionally, Tor selects a new circuit every 10
minutes (IIRC), so if you load several web pages in a row, they'll go
through the same circuit.

[Joanna Rutkowska]

On 03/26/14 23:13, Vincent Penquerc'h wrote:
> On 26/03/14 21:55, Rob Townley wrote:
>> Haven't used TOR. How does a user of TOR test that the outgoing
>> "gateways" are randomized? Does a traceroute "8.8.8.8" give a
>> different path each time?
>
> No, Tor will route TCP streams, but traceroute will use UDP.

... unless you pass '-T' to traceroute...

> Tor will
> *not* mean that all of your traffic will go through Tor.

... but Qubes Tor VM will ensure that (more specifically, that all the
traffic generated from an AppVM connected to torvm will got thorugh tor,
or will get discarded).

> See the Tor FAQ
> at torproject.org. check.torproject.org will let you know if your IP is
> a Tor exit node. Additionally, Tor selects a new circuit every 10
> minutes (IIRC), so if you load several web pages in a row, they'll go
> through the same circuit.
>

joanna.

[Rob Townley]

Thanks for reminding me about -T.

[Axon]

Abel Luck:

I think I know what Bahtiar is referring to. For some reason, if you had
an existing F18 template with qubes-tor installed, then upgraded that
template to F20, it now works (i.e., you can update the template in the
normal way and end up with the most recent version of Tor). But for some
reason, if you just create a new F20 template and attempt to install
qubes-tor in it (by following the wiki doc), it fails. At least, that's
what some users are/were experiencing. I don't know why.

[Marek Marczykowski-Górecki]

This is because qubes-tor packages are missing in current fc20 repo (only in
testing repo). Joanna, can you upload the tor packages to current/vm/fc20 repo?

[Patrick Schless]

for anyone else trying to make this work, I've written up the specifics steps I took (without using any qubes-specific packages). I prefer this method to the qubes-tor package because I'm new to Qubes and having to type out each step helps me learn what's going on. As I said above, I basically used Joanna's blog post, with a couple changes (qubes-ip, torvm's netvm, and the start_tor bin dir).

http://www.plainlystated.com/2014/03/torvm-for-qubesos/

Hope this helps,
Patrick

--
You received this message because you are subscribed to a topic in the Google Groups "qubes-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/qubes-devel/IVPlA0_vb4I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to qubes-devel...@googlegroups.com.
To post to this group, send email to qubes...@googlegroups.com.
Visit this group at http://groups.google.com/group/qubes-devel.
For more options, visit https://groups.google.com/d/optout.

[Joanna Rutkowska]

Done.
j.

[Outback Dingo]

On Sat, Mar 29, 2014 at 1:31 AM, Patrick Schless <patrick...@gmail.com> wrote:

for anyone else trying to make this work, I've written up the specifics steps I took (without using any qubes-specific packages). I prefer this method to the qubes-tor package because I'm new to Qubes and having to type out each step helps me learn what's going on. As I said above, I basically used Joanna's blog post, with a couple changes (qubes-ip, torvm's netvm, and the start_tor bin dir).

http://www.plainlystated.com/2014/03/torvm-for-qubesos/

Hope this helps,
Patrick

This was a big help, though now I can confirm torvm based on fedora-20-x64 templates do now work, since they updated the packages

--
You received this message because you are subscribed to the Google Groups "qubes-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.

[Axon]

Outback Dingo:

> On Sat, Mar 29, 2014 at 1:31 AM, Patrick Schless
> <patrick...@gmail.com>wrote:
>
>> for anyone else trying to make this work, I've written up the specifics
>> steps I took (without using any qubes-specific packages). I prefer this
>> method to the qubes-tor package because I'm new to Qubes and having to type
>> out each step helps me learn what's going on. As I said above, I basically
>> used Joanna's blog post, with a couple changes (qubes-ip, torvm's netvm,
>> and the start_tor bin dir).
>>

>> *http://www.plainlystated.com/2014/03/torvm-for-qubesos/
>> <http://www.plainlystated.com/2014/03/torvm-for-qubesos/>*

qubes-tor updates now appear to be failing. Not sure if this is already
known:

> Downloading packages:
> No Presto metadata available for qubes-vm-r2b3-current
> qubes-tor-repo-0.1.4-1.fc20.x8 FAILED
> http://yum.qubes-os.org/r2-beta3/current/vm/fc20/qubes-tor-repo-0.1.4-1.fc20.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found
> Trying other mirror.
> qubes-tor-0.1.4-1.fc20.x86_64. FAILED
> http://yum.qubes-os.org/r2-beta3/current/vm/fc20/qubes-tor-0.1.4-1.fc20.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found ETA
> Trying other mirror.
>
>
> Error downloading packages:
> qubes-tor-0.1.4-1.fc20.x86_64: [Errno 256] No more mirrors to try.
> qubes-tor-repo-0.1.4-1.fc20.x86_64: [Errno 256] No more mirrors to try.

[Joanna Rutkowska]

On 04/09/14 14:06, Axon wrote:

> qubes-tor updates now appear to be failing. Not sure if this is already
> known:
>
>> Downloading packages:
>> No Presto metadata available for qubes-vm-r2b3-current
>> qubes-tor-repo-0.1.4-1.fc20.x8 FAILED
>> http://yum.qubes-os.org/r2-beta3/current/vm/fc20/qubes-tor-repo-0.1.4-1.fc20.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found
>> Trying other mirror.
>> qubes-tor-0.1.4-1.fc20.x86_64. FAILED
>> http://yum.qubes-os.org/r2-beta3/current/vm/fc20/qubes-tor-0.1.4-1.fc20.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found ETA
>> Trying other mirror.
>>
>>
>> Error downloading packages:
>> qubes-tor-0.1.4-1.fc20.x86_64: [Errno 256] No more mirrors to try.
>> qubes-tor-repo-0.1.4-1.fc20.x86_64: [Errno 256] No more mirrors to try.
>
>
>
>

That's probably because I'm *just* uploading a bunch of new pkgs and
some of the repodata still didn't get updated (or got updated by pkgs
didnt).

We will need to change the upload procedure to avoid such "race
conditions" soon. I think we could have a spool directory on our yum
server where we could do upload of rpms, and then fire a script on the
server to quickly copy the rpms to the appropriate repo (current,
current-testing, unstable) and generate repodata.

Better yet: have each of the "real" repo directory have its shadow
directory, copy pkgs there, generate repodata, and then only switch
simlinks?

Or perhaps somebody has a better solution?

joanna.

[Marek Marczykowski-Górecki]

On 09.04.2014 14:33, Joanna Rutkowska wrote:
> On 04/09/14 14:06, Axon wrote:
>
>> qubes-tor updates now appear to be failing. Not sure if this is already
>> known:
>>
>>> Downloading packages:
>>> No Presto metadata available for qubes-vm-r2b3-current
>>> qubes-tor-repo-0.1.4-1.fc20.x8 FAILED
>>> http://yum.qubes-os.org/r2-beta3/current/vm/fc20/qubes-tor-repo-0.1.4-1.fc20.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found
>>> Trying other mirror.
>>> qubes-tor-0.1.4-1.fc20.x86_64. FAILED
>>> http://yum.qubes-os.org/r2-beta3/current/vm/fc20/qubes-tor-0.1.4-1.fc20.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found ETA
>>> Trying other mirror.
>>>
>>>
>>> Error downloading packages:
>>> qubes-tor-0.1.4-1.fc20.x86_64: [Errno 256] No more mirrors to try.
>>> qubes-tor-repo-0.1.4-1.fc20.x86_64: [Errno 256] No more mirrors to try.
>>
>>
>>
>>
> That's probably because I'm *just* uploading a bunch of new pkgs and
> some of the repodata still didn't get updated (or got updated by pkgs
> didnt).

This isn't the case, the metadata is broken (see missing "rpm/" in package
url). I'm working on fixing that script...

> We will need to change the upload procedure to avoid such "race
> conditions" soon. I think we could have a spool directory on our yum
> server where we could do upload of rpms, and then fire a script on the
> server to quickly copy the rpms to the appropriate repo (current,
> current-testing, unstable) and generate repodata.

Simpler solution (used on fedora mirrors) is to transfer rpms and only then
metadata. Or even generate metadata on server after uploading the packages.

[Joanna Rutkowska]

On 04/09/14 14:40, Marek Marczykowski-Górecki wrote:
> On 09.04.2014 14:33, Joanna Rutkowska wrote:
>> On 04/09/14 14:06, Axon wrote:
>>
>>> qubes-tor updates now appear to be failing. Not sure if this is already
>>> known:
>>>
>>>> Downloading packages:
>>>> No Presto metadata available for qubes-vm-r2b3-current
>>>> qubes-tor-repo-0.1.4-1.fc20.x8 FAILED
>>>> http://yum.qubes-os.org/r2-beta3/current/vm/fc20/qubes-tor-repo-0.1.4-1.fc20.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found
>>>> Trying other mirror.
>>>> qubes-tor-0.1.4-1.fc20.x86_64. FAILED
>>>> http://yum.qubes-os.org/r2-beta3/current/vm/fc20/qubes-tor-0.1.4-1.fc20.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found ETA
>>>> Trying other mirror.
>>>>
>>>>
>>>> Error downloading packages:
>>>> qubes-tor-0.1.4-1.fc20.x86_64: [Errno 256] No more mirrors to try.
>>>> qubes-tor-repo-0.1.4-1.fc20.x86_64: [Errno 256] No more mirrors to try.
>>>
>>>
>>>
>>>
>> That's probably because I'm *just* uploading a bunch of new pkgs and
>> some of the repodata still didn't get updated (or got updated by pkgs
>> didnt).
>
> This isn't the case, the metadata is broken (see missing "rpm/" in package
> url). I'm working on fixing that script...
>

Ah, indeed.

>> We will need to change the upload procedure to avoid such "race
>> conditions" soon. I think we could have a spool directory on our yum
>> server where we could do upload of rpms, and then fire a script on the
>> server to quickly copy the rpms to the appropriate repo (current,
>> current-testing, unstable) and generate repodata.
>
> Simpler solution (used on fedora mirrors) is to transfer rpms and only then
> metadata. Or even generate metadata on server after uploading the packages.
>

Still, running createrepo takes some time (15 sec or so)... But perhaps
createrepo somehow "locks" the metadata (e.g. by creating some file in
the repodata) that yum clients can detect when connecting and just wait?

j.

[Marek Marczykowski-Górecki]

No lock, but it creates new metadata in new files (see long hashes in
filenames), and at the end updates repomd.xml with path to the new files (and
remove old files).

[Joanna Rutkowska]

Aha, so if we're not overwriting old rpms, which we don't do, the
sensitive period should only be the repomd.xml overwrite, which should
be negligible, correct? Actually, that could even be done without any
window of inconsistencies thanks to file locking.

So we should change our upload scripts to do:
1) upload rpms only
2) fire createrepo on the server

At least this should be for current-testing/unstable. For current, I
think the only way to get packages there should be by doing local cp
(ln) on the server from current-testing and running createrepo again (or
can we just copy the repodata too to another directory?)

j.

[Marek Marczykowski-Górecki]

I'm not sure, but I expect createrepo writes new repomd.xml in some temporary
file and then renames it over original repomd.xml.

> So we should change our upload scripts to do:
> 1) upload rpms only
> 2) fire createrepo on the server
>
> At least this should be for current-testing/unstable. For current, I
> think the only way to get packages there should be by doing local cp
> (ln) on the server from current-testing and running createrepo again (or
> can we just copy the repodata too to another directory?)

No, metadata need to be regenerated. Package set in those repositories are
most likely different (not all packages from current-testing ends up in current).

[cprise]

I think the symlink idea works best of all. I have noticed that Fedora
repo source data gets temporarily borked from time to time when trying
to an update, making it necessary to abort and re-try later. I have even
seen one template successfully update, then the next template fail to
update because the latter was accessing a repo while it was updating. I
never experienced this with distros that used apt, and never experienced
it with CentOS either.

The point is that a symlink gets read at one point in time, and then
whatever versioned dir-name that points to gets used for the entire
update process.