Bug#874397: debian mirror ftp.arnes.si issues

[Peter Palfrader]

Package: mirrors
User: mir...@packages.debian.org
Usertags: mirror-problems may-auto-close

[Sent to the mirror maintainer]

Hi!

It seems ftp.arnes.si is not updating regularly.

cf.
https://mirror-master.debian.org/status/mirror-info/ftp.arnes.si.html

We recommend four mirror runs a day for publicly accessible and listed
mirrors. Please change your config to that interval.

Cheers,
--
| .''`. ** Debian **
Peter Palfrader | : :' : The universal
https://www.palfrader.org/ | `. `' Operating System
| `- https://www.debian.org/

[Peter Palfrader]

On Tue, 05 Sep 2017, Peter Palfrader wrote:

> Hi!
>
> It seems ftp.arnes.si is not updating regularly.
>
> cf.
> https://mirror-master.debian.org/status/mirror-info/ftp.arnes.si.html
>
> We recommend four mirror runs a day for publicly accessible and listed
> mirrors. Please change your config to that interval.

ping.

[Mitja Mihelič]

Hi!

Thank you for the reminder. I have now set the mirror sync rate to
4x/day. I have also started a manual sync to get up to date.
Until now the mirror was syncing regularly once per week.

Regards,
Mitja

[Peter Palfrader]

Hi Mitja!

On Mon, 11 Sep 2017, Mitja Mihelič wrote:

> Thank you for the reminder. I have now set the mirror sync rate to 4x/day. I
> have also started a manual sync to get up to date.
> Until now the mirror was syncing regularly once per week.

Thanks.

o The trace file at
http://ftp.arnes.si/pub/packages/debian/project/trace/ftp.arnes.si
does not contain much information.

Please use our ftpsync script to mirror Debian.

http://ftp.debian.org/debian/project/ftpsync/ftpsync-current.tar.gz

Using a modern ftpsync ensures updates are done in the correct order
so apt clients don't get confused. In particular, it processes
translations, contents, and more files that have been added to the
archive in recent years in the correct stage. It also should produce
trace files that contain more information that is useful for us.

o Do you think you could make an alias for
http://ftp.arnes.si/pub/packages/debian/
at
http://ftp.arnes.si/debian/
to have the archive available at a more standard location?

o Also, is there a reason you have rsync, but don't offer Debian via it?

[Mitja Mihelič]

The answer to most of you question about our configuration is "It is so
due to historical reasons." :)
I will address your suggestions in-line with updates of our configuration.

I have noticed, that ftp.si.debian.org is an alias for ftp.debian.at.
Can we get that to point at our mirror?

On 18/09/2017 10:53, Peter Palfrader wrote:
> Hi Mitja!
>
> On Mon, 11 Sep 2017, Mitja Mihelič wrote:
>
>> Thank you for the reminder. I have now set the mirror sync rate to 4x/day. I
>> have also started a manual sync to get up to date.
>> Until now the mirror was syncing regularly once per week.
> Thanks.
>
> o The trace file at
> http://ftp.arnes.si/pub/packages/debian/project/trace/ftp.arnes.si
> does not contain much information.
>
> Please use our ftpsync script to mirror Debian.
>
> http://ftp.debian.org/debian/project/ftpsync/ftpsync-current.tar.gz
>
> Using a modern ftpsync ensures updates are done in the correct order
> so apt clients don't get confused. In particular, it processes
> translations, contents, and more files that have been added to the
> archive in recent years in the correct stage. It also should produce
> trace files that contain more information that is useful for us.

I will work on this and report back.

>
> o Do you think you could make an alias for
> http://ftp.arnes.si/pub/packages/debian/
> at
> http://ftp.arnes.si/debian/
> to have the archive available at a more standard location?

The mirror is now available at a new location which is consistent with
our mirror policy.
http://ftp.arnes.si/mirrors/debian/
https://ftp.arnes.si/mirrors/debian/

>
> o Also, is there a reason you have rsync, but don't offer Debian via it?

Debian is now available via rsync:
- rsync://ftp.arnes.si:/mirrors/debian
- ftp.arnes.si::debian
>
> Cheers,
Kind regards,
Mitja

[Bastian Blank]

Hi

On Mon, Sep 25, 2017 at 02:53:09PM +0200, Mitja Mihelič wrote:
> The answer to most of you question about our configuration is "It is so due
> to historical reasons." :)
> I will address your suggestions in-line with updates of our configuration.

Thank you for coming back to us.

> I have noticed, that ftp.si.debian.org is an alias for ftp.debian.at. Can we
> get that to point at our mirror?

Given that ftp.*.debian.org _must_ offer Debian at /debian, I don't see
how this would work.

> On 18/09/2017 10:53, Peter Palfrader wrote:
> > o The trace file at
> > http://ftp.arnes.si/pub/packages/debian/project/trace/ftp.arnes.si
> > does not contain much information.

> I will work on this and report back.

Okay. Our monitoring will tell us about it.

> > o Do you think you could make an alias for
> > http://ftp.arnes.si/pub/packages/debian/
> > at
> > http://ftp.arnes.si/debian/
> > to have the archive available at a more standard location?
> The mirror is now available at a new location which is consistent with our
> mirror policy.
> http://ftp.arnes.si/mirrors/debian/
> https://ftp.arnes.si/mirrors/debian/

Okay, I'll change that in our list, but as said before, this does not
help us.

Regards,
Bastian

--
It is a human characteristic to love little animals, especially if
they're attractive in some way.
-- McCoy, "The Trouble with Tribbles", stardate 4525.6

[Peter Palfrader]

On Mon, 25 Sep 2017, Bastian Blank wrote:

> Hi

>
> Thank you for coming back to us.
>
> > I have noticed, that ftp.si.debian.org is an alias for ftp.debian.at. Can we
> > get that to point at our mirror?

It used to be an alias for debmirror.amis.net, but that hasn't updated
in weeks so we had to remove it from our list of mirrors.

> Given that ftp.*.debian.org _must_ offer Debian at /debian, I don't see
> how this would work.

Can you make /debian/ be a (hidden) alias?

[Mitja Mihelič]

On 25/09/2017 17:49, Peter Palfrader wrote:
> On Mon, 25 Sep 2017, Bastian Blank wrote:
>
>> Hi
>>
>> Thank you for coming back to us.
>>
>>> I have noticed, that ftp.si.debian.org is an alias for ftp.debian.at. Can we
>>> get that to point at our mirror?
> It used to be an alias for debmirror.amis.net, but that hasn't updated
> in weeks so we had to remove it from our list of mirrors.

I see. I'll get our mirror up to specs, and then we can see about
redirecting the alias.

>
>> Given that ftp.*.debian.org _must_ offer Debian at /debian, I don't see
>> how this would work.
> Can you make /debian/ be a (hidden) alias?

The Debian mirror is now also available as an alias.
http://ftp.arnes.si/debian/
https://ftp.arnes.si/debian/

I am having trouble with ftpsync. I am using the ftpsync-cron script. I
ran it manually from the commandline and got the folowing error:
Sep 26 13:29:37 poprovec ftpsync-cron[12403]: Failed to parse archive
serial from http://debian.ethz.ch/debian/project/trace/debian.ethz.ch

I checked the upstream trace file manually and the serial is there. Can
you help me with setting up the ftpsync cronjob?
Until I have ftpsync up and running I will be using the old rsync.

Regards,
Mitja

[Bastian Blank]

Hi Mitja

On Tue, Sep 26, 2017 at 01:40:08PM +0200, Mitja Mihelič wrote:
> I am having trouble with ftpsync. I am using the ftpsync-cron script. I ran
> it manually from the commandline and got the folowing error:
> Sep 26 13:29:37 poprovec ftpsync-cron[12403]: Failed to parse archive serial
> from http://debian.ethz.ch/debian/project/trace/debian.ethz.ch

Could you please show me the output of "bash -x ftpsync-cron"? For me
this works fine.

But I'll change the script to output the complete content of the trace
file in this condition.

Regards,
Bastian

--
There are always alternatives.
-- Spock, "The Galileo Seven", stardate 2822.3

[Peter Palfrader]

On Tue, 26 Sep 2017, Mitja Mihelič wrote:

> The Debian mirror is now also available as an alias.
> http://ftp.arnes.si/debian/
> https://ftp.arnes.si/debian/

Thanks.

Regarding rsync, I note ftp.arnes.si::mirrors/debian/ works but
also ftp.arnes.si::debian/ is advertised but doesn't:

| weasel@orinoco:~$ rsync ftp.arnes.si::debian/
| @ERROR: invalid uid debian
| rsync error: error starting client-server protocol (code 5) at main.c(1666) [Receiver=3.1.2]

Should this work?

[Mitja Mihelič]

On 26/09/2017 15:18, Peter Palfrader wrote:
> On Tue, 26 Sep 2017, Mitja Mihelič wrote:
>
>> The Debian mirror is now also available as an alias.
>> http://ftp.arnes.si/debian/
>> https://ftp.arnes.si/debian/
> Thanks.

You're welcome.

>
> Regarding rsync, I note ftp.arnes.si::mirrors/debian/ works but
> also ftp.arnes.si::debian/ is advertised but doesn't:
>
> | weasel@orinoco:~$ rsync ftp.arnes.si::debian/
> | @ERROR: invalid uid debian
> | rsync error: error starting client-server protocol (code 5) at main.c(1666) [Receiver=3.1.2]
>
> Should this work?

The configuration is fixed now. It should work.

Regards,
Mitja Mihelič

[Mitja Mihelič]

On 26/09/2017 15:13, Bastian Blank wrote:
> Hi Mitja
>
> On Tue, Sep 26, 2017 at 01:40:08PM +0200, Mitja Mihelič wrote:
>> I am having trouble with ftpsync. I am using the ftpsync-cron script. I ran
>> it manually from the commandline and got the folowing error:
>> Sep 26 13:29:37 poprovec ftpsync-cron[12403]: Failed to parse archive serial
>> from http://debian.ethz.ch/debian/project/trace/debian.ethz.ch
> Could you please show me the output of "bash -x ftpsync-cron"? For me
> this works fine.
>
> But I'll change the script to output the complete content of the trace
> file in this condition.
>
> Regards,
> Bastian
>

I took a look at the "bash -x" output. I pasted the relevant lines into
a separate script.
When the double quotes were omitted the extract_trace_serial simply got
an empty set of arguments.
It all led me to make the following change to the ftpsync-cron script:
[ftpsync@poprovec bin]$ diff ftpsync-cron ftpsync-cron_original
423c423
<     if ! serial_remote=$(extract_trace_serial <<< "$trace"); then
---
>     if ! serial_remote=$(extract_trace_serial <<< $trace); then
426c426
<     elif ! serial_local=$(extract_trace_serial "$trace_local"); then
---
>     elif ! serial_local=$(extract_trace_serial $trace_local); then

As requested, here is the trace from "bash -x ftpsync-cron". I don't
know if attachments are okay, so here is a direct paste.
I see that the LOGDIR variable at the top of the output is set to
"/home/ftpsync/log". Which is not what I set it to in the config file.
Further down it is seen as LOGDIR=/var/log/mirror/debian. This is what I
set in the config file. Is difference OK?

[ftpsync@poprovec bin]$ bash -x ftpsync-cron
+ set -euE
+ VERSION=20170920
+++ readlink -f ftpsync-cron
++ dirname /home/ftpsync/bin/ftpsync-cron
+ BINDIR=/home/ftpsync/bin
+++ dirname ftpsync-cron
++ readlink -f ./..
+ BASEDIR=/home/ftpsync
+ CONFDIR=/home/ftpsync/etc
+ LOCKDIR=/home/ftpsync/locks
+ LOGDIR=/home/ftpsync/log
+ export BASEDIR
+ ARCHIVE=
+ CONFIG=ftpsync.conf
+ NAME=ftpsync-cron
+ PROGRAM=ftpsync-cron
+ read_config ftpsync.conf
++ echo ftpsync.conf
++ sed -e 's/[^A-Za-z0-9._-]/_/g'
+ local name=ftpsync.conf
++ search_config ftpsync.conf
++ local file
++ '[' /home/ftpsync/etc ']'
++ file=/home/ftpsync/etc/ftpsync.conf
++ '[' -f /home/ftpsync/etc/ftpsync.conf ']'
++ echo /home/ftpsync/etc/ftpsync.conf
++ return
+ local config=/home/ftpsync/etc/ftpsync.conf
+ '[' /home/ftpsync/etc/ftpsync.conf ']'
+ . /home/ftpsync/etc/ftpsync.conf
++ MIRRORNAME=ftp.arnes.si
++ TO=/big/ftp-data/mirrors/debian
++ RSYNC_PATH=debian
++ RSYNC_HOST=debian.ethz.ch
++ INFO_MAINTAINER=ftp-...@arnes.si
++ INFO_SPONSOR='ARNES <https://www.arnes.si>'
++ INFO_COUNTRY=SI
++ INFO_THROUGHPUT=1Gb
++ LOGDIR=/var/log/mirror/debian
++ LOG=/var/log/mirror/debian/ftpsync-cron.log
++ MAILTO=root
++ ERRORSONLY=true
++ FULLLOGS=false
++ ARCH_EXCLUDE='alpha amd64 arm arm64 armel armhf hppa ia64 m68k mipsel
mips powerpc ppc64el s390 s390x sparc'
++ HUB=false
++ LOGROTATE=14
++ LOCK=/big/ftp-data/mirrors/debian/Archive-Update-in-Progress-ftp.arnes.si
++
UPDATEREQUIRED=/big/ftp-data/mirrors/debian/Archive-Update-Required-ftp.arnes.si
++ UIPSLEEP=1200
++ UIPRETRIES=3
++ TRACE=project/trace/ftp.arnes.si
+++ hostname -f
++ TRACEHOST=poprovec.arnes.si
++ RSYNC=rsync
++ RSYNC_EXTRA=-a
++ RSYNC_OPTIONS='-prltvHSB8192 --chmod=D755,F644 --timeout 3600 --stats'
++ RSYNC_OPTIONS1='--exclude=Packages* --exclude=Sources*
--exclude=Release* --exclude=InRelease --include=i18n/by-hash/**
--exclude=i18n/* --exclude=ls-lR*'
++ RSYNC_OPTIONS2='--max-delete=40000 --delay-updates --delete
--delete-excluded'
+ CURRENT_CONFIG=/home/ftpsync/etc/ftpsync.conf
+ return 0
+ MAILTO=root
+ MIRRORNAME=ftp.arnes.si
+ LOCK=/big/ftp-data/mirrors/debian/Archive-Update-in-Progress-ftp.arnes.si
+ RSYNC_PATH=debian
+ TRACE_UPSTREAM=project/trace/debian.ethz.ch
+ cmp_trace
+ local trace_uri=http://debian.ethz.ch/debian/project/trace/debian.ethz.ch
+ local
trace_local=/big/ftp-data/mirrors/debian/project/trace/debian.ethz.ch
+ local trace ret
+ [[ -e
/big/ftp-data/mirrors/debian/Archive-Update-in-Progress-ftp.arnes.si ]]
++ curl --silent --fail --show-error --location
http://debian.ethz.ch/debian/project/trace/debian.ethz.ch
+ trace='Thu Sep 28 03:13:07 UTC 2017
Date: Thu, 28 Sep 2017 03:13:07 +0000
Date-Started: Thu, 28 Sep 2017 03:07:43 +0000
Archive serial: 2017092801
Used ftpsync version: 20170204
Running on host: plattenberg.ethz.ch
Architectures: all amd64 arm64 armel armhf hurd-i386 i386 ia64
kfreebsd-amd64 kfreebsd-i386 mips mips64el mipsel powerpc ppc64el s390
s390x source sparc
Architectures-Configuration: ALL
Upstream-mirror: syncproxy2.eu.debian.org
SSL: false
Total bytes received in rsync: 2483023385
Total time spent in stage1 rsync: 250
Total time spent in stage2 rsync: 74
Total time spent in rsync: 324
Average rate: 7663652 B/s'
+ ret=0
+ [[ 0 != 0 ]]
++ extract_trace_serial
+++ awk '-F: ' ' $1=="Archive serial" {print $2}'
++ local serial=
++ [[ -n '' ]]
++ return 1
+ serial_remote=
+ error 'Failed to parse archive serial from
http://debian.ethz.ch/debian/project/trace/debian.ethz.ch'
+ log 'Failed to parse archive serial from
http://debian.ethz.ch/debian/project/trace/debian.ethz.ch'
+ [[ -z ftpsync-cron ]]
++ date '+%b %d %H:%M:%S'
++ hostname -s
+ echo 'Sep 28 09:32:21 poprovec ftpsync-cron[29806]: Failed to parse
archive serial from
http://debian.ethz.ch/debian/project/trace/debian.ethz.ch'
Sep 28 09:32:21 poprovec ftpsync-cron[29806]: Failed to parse archive
serial from http://debian.ethz.ch/debian/project/trace/debian.ethz.ch
+ [[ -n root ]]
+ echo 'Failed to parse archive serial from
http://debian.ethz.ch/debian/project/trace/debian.ethz.ch'
++ hostname -s
+ mailx -e -s '[ftpsync-cron@poprovec] ERROR [29806]' root
+ exit 1

Regards,
Mitja

[Mitja Mihelič]

Hi!

Did you get a chance to go over my last reply and questions about
ftpsync-cron?

Regards,
Mitja Mihelič