Bug#874574: Debian mirror mirrors.tuna.tsinghua.edu.cn/nanomirrors.tuna.tsinghua.edu.cn/neomirrors.tuna.tsinghua.edu.cn: out of date, tracefile-name, ftpsync-version

[Peter Palfrader]

Package: mirrors
User: mir...@packages.debian.org
Usertags: mirror-problem

Hi!

One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
out of date.

| weasel@melartin:~$ curl -s http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn | grep serial
| Archive serial: 2017090602
| weasel@melartin:~$ curl -s http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn | grep serial
| Archive serial: 2017090702

o Please investigate why nanomirrors does not update

o Please set MIRRORNAME on both hosts to their respective hostname, such that
the tracefiles are
http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/nanomirrors.tuna.tsinghua.edu.cn
and
http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/neomirrors.tuna.tsinghua.edu.cn
respectively.

o The tracefiles suggest that the ftpsync version you are using is not the
current one. Please update

Cheers,
--
| .''`. ** Debian **
Peter Palfrader | : :' : The universal
https://www.palfrader.org/ | `. `' Operating System
| `- https://www.debian.org/

[Shanker Wang]

Thanks for notification!

> 在 2017年09月07日，20:29，Peter Palfrader <wea...@debian.org> 写道：
>
> Package: mirrors
> User: mir...@packages.debian.org
> Usertags: mirror-problem
>
> Hi!
>
> One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
> out of date.
>
> | weasel@melartin:~$ curl -s http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn | grep serial
> | Archive serial: 2017090602
> | weasel@melartin:~$ curl -s http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn | grep serial
> | Archive serial: 2017090702
>
> o Please investigate why nanomirrors does not update

Because of an unexpected termination of the sync process, the lock file was not removed and blocked further retries. The lock file has been removed and the sync is now in progress.

>
> o Please set MIRRORNAME on both hosts to their respective hostname, such that
> the tracefiles are
> http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/nanomirrors.tuna.tsinghua.edu.cn
> and
> http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/neomirrors.tuna.tsinghua.edu.cn
> respectively.

We have changed this setting. Thanks for your advice.

>
> o The tracefiles suggest that the ftpsync version you are using is not the
> current one. Please update

We will soon perform that upgrade.

>
> Cheers,
> --
> | .''`. ** Debian **
> Peter Palfrader | : :' : The universal
> https://www.palfrader.org/ | `. `' Operating System
> | `- https://www.debian.org/
>

> --
> 您收到此邮件是因为您订阅了 Google 网上论坛的“THU Opensource Mirror Admin”群组。
> 要退订此群组并停止接收此群组的电子邮件，请发送电子邮件到thu-opensource-mirr...@googlegroups.com。
> 要查看更多选项，请访问 https://groups.google.com/d/optout。

[Shanker Wang]

Thanks for notification! 

在 2017年09月07日，20:29，Peter Palfrader <wea...@debian.org> 写道：

Package: mirrors
User: mir...@packages.debian.org
Usertags: mirror-problem

Hi!

One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
out of date.

| weasel@melartin:~$ curl -s http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn | grep serial 
| Archive serial: 2017090602
| weasel@melartin:~$ curl -s http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn | grep serial 
| Archive serial: 2017090702

o Please investigate why nanomirrors does not update

Because of an unexpected termination of the sync process, the lock file was not removed and blocked further retries. The lock file has been removed and the sync is now in progress. 

o Please set MIRRORNAME on both hosts to their respective hostname, such that
the tracefiles are
  http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/nanomirrors.tuna.tsinghua.edu.cn
and
  http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/neomirrors.tuna.tsinghua.edu.cn
respectively.

We have changed this setting. Thanks for your advice.

o The tracefiles suggest that the ftpsync version you are using is not the
current one.  Please update

We will soon perform that upgrade.

Cheers,
-- 
                          |  .''`.       ** Debian **
    Peter Palfrader       | : :' :      The  universal
https://www.palfrader.org/ | `. `'      Operating System
                          |   `-    https://www.debian.org/

[Shanker Wang]

> 在 2017年09月15日，18:05，Bastian Blank <wa...@debian.org> 写道：
>
> Hi

>
> On Thu, Sep 07, 2017 at 11:02:40PM +0800, Shanker Wang wrote:
>>> One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
>>> out of date.
>

> Currently both mirrors are updated almost three hours apart, even if
> triggered at the same time. Please investigate why this happens.

That's because the trigger cannot assure that sync is started immediately.
When the limit of concurrent syncing jobs is reached, other jobs will be
pending in queue.

>
>>> | weasel@melartin:~$ curl -s http://nanomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn | grep serial
>>> | Archive serial: 2017090602
>>> | weasel@melartin:~$ curl -s http://neomirrors.tuna.tsinghua.edu.cn/debian/project/trace/mirrors.tuna.tsinghua.edu.cn | grep serial
>>> | Archive serial: 2017090702
>

> Regards,
> Bastian
>
> --
> Extreme feminine beauty is always disturbing.
> -- Spock, "The Cloud Minders", stardate 5818.4

[Shanker Wang]

> 在 2017年09月16日，16:47，Peter Palfrader <wea...@debian.org> 写道：

>
> On Fri, 15 Sep 2017, Shanker Wang wrote:
>
>>> On Thu, Sep 07, 2017 at 11:02:40PM +0800, Shanker Wang wrote:
>>>>> One of the two mirrors behind mirrors.tuna.tsinghua.edu.cn is currently
>>>>> out of date.
>>>
>>> Currently both mirrors are updated almost three hours apart, even if
>>> triggered at the same time. Please investigate why this happens.
>>
>> That's because the trigger cannot assure that sync is started immediately.
>> When the limit of concurrent syncing jobs is reached, other jobs will be
>> pending in queue.
>

> Then these services probably should not be available behind a common
> name. Apt gets very confused if the data on "one server" changes
> between requests, therefore the window where mirrors have a different
> state should be kept as small as possible.
>

You are right. The main reason why there are two servers balancing the
traffic is that the throughput of the storage is becoming the bottleneck.
The most ideal solution should be utilizing a load balancer to "pin" a
user onto a specific server. However, we do not have an additional server
as a load balancer currently.

I admit that the inconsistence between the two servers should be a problem.
To partly work around this, the following measures were taken:

* Load balancing is achieved by randomly return one DNS record of two on
the name server, whose ttl is about 24 hours, so that for a period of
time, the name resolution result of our common name will be stable.

* For requests for a file which does not exist on one server, the server
will internally forward the request to the other and will response with
the file on the other server if exists.


Those are all we have done to work around this problem. Though we have not
received issues from users about the inconsistence, we will keep a close
eye on that. Any suggestions are welcomed.

> To which of the two mirrors should ftp2.cn.debian.org point, and should
> we replace mirrors.tuna.tsinghua.edu.cn in our list with nano and neo?

[Bastian Blank]

Hi

It happened again, this time on neomirror, and for 12(!) days.

https://mirror-master.debian.org/status/mirror-info/neomirrors.tuna.tsinghua.edu.cn.html

Please fix your setup. In the meantime I will remove it as candicate
for any *.d.o.

Bastian

--
Men will always be men -- no matter where they are.
-- Harry Mudd, "Mudd's Women", stardate 1329.8

[Shanker Wang]

Hi, the sync is still in progress, which started on Nov 11 09:39:26

Nov 11 09:39:26 4b36bab6e1bc ftpsync-debian[7]: Mirrorsync start
Nov 11 09:39:26 4b36bab6e1bc ftpsync-debian[7]: Running mirrorsync, update is required, /data/mirrors/debian//Archive-Update-Required-neomirrors.tuna.tsinghua.edu.cn exists
Nov 11 09:39:26 4b36bab6e1bc ftpsync-debian[7]: Running stage1: rsync -e /ftpsync/bin/rsync-ssl-tunnel -6 --safe-links --bwlimit=0 -prltvHSB8192 --timeout 3600 --stats --filter=protect_Archive-Update-in-Progress-neomirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/neomirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/neomirrors.tuna.tsinghua.edu.cn-stage1 --exclude=project/trace/_hierarchy --filter=protect_project/trace/_hierarchy --exclude=project/trace/_traces --filter=protect_project/trace/_traces --filter=protect_Archive-Update-Required-neomirrors.tuna.tsinghua.edu.cn --exclude=Packages* --exclude=Sources* --exclude=Release* --exclude=InRelease --include=i18n/by-hash/** --exclude=i18n/* --exclude=ls-lR* --exclude=.~tmp~/ mirrors.tuna.t...@syncproxy.cna.debian.org::debian /data/mirrors/debian/

I'm trying to figure out why the sync is taking so much time.

Cheers,

Miao Wang

[Wang Shanker]

Finally we managed to get the server in pace.

BTW, due to the slow connection speed to syncpxy.cna.d.o, could you please also open an account on syncproxy2.wna.d.o for us? It seems that the connection to that server is better.

[Bastian Blank]

On Fri, Nov 17, 2017 at 08:24:37AM -0700, Wang Shanker wrote:
> Finally we managed to get the server in pace.
>
> BTW, due to the slow connection speed to syncpxy.cna.d.o, could you please also open an account on syncproxy2.wna.d.o for us? It seems that the connection to that server is better.

How does it come that both systems are behind the same router, but only
one got connection problems?

Sorry, but this have been a problem for too long. So please make sure
that:
- all sync pushs start a ftpsync without lag, aka drop your queueing
between the request and the sync, and
- if you really have connection problems, start with rewireing your
setup to sync to one system from the outside and update the second
system locally.

Regards,
Bastian

--
Totally illogical, there was no chance.
-- Spock, "The Galileo Seven", stardate 2822.3

[Shanker Wang]

> 在 2017年11月26日，19:58，Bastian Blank <wa...@debian.org> 写道：
>
> On Fri, Nov 17, 2017 at 08:24:37AM -0700, Wang Shanker wrote:
>> Finally we managed to get the server in pace.
>>
>> BTW, due to the slow connection speed to syncpxy.cna.d.o, could you please also open an account on syncproxy2.wna.d.o for us? It seems that the connection to that server is better.
>
> How does it come that both systems are behind the same router, but only
> one got connection problems?

Not only one but both. The connection is slow but normally both servers
can get in pace with the upstream. However, in late October, we has
performed an storage upgrade on the server in question, which made the
sever missed several updates during that time in a few days. After it
brought back online, it was not put into service immediately but a manual
sync from the other server was performed. After the successful sync, the
server in question was put back online. Because the re-sync itself took a
long time, a few updates for debian was missed, which made a large amount
of data to transfer in the later sync from syncpxy.cna.d.o. With as slow
connection speed as about 1 Mbps, such transfer could not be completed
in an appropriate time. I first didn't realize it and tried to restart
the ftpsync for many times but such measure made no luck. Finally, I
manually started rsync from its peer and then started the normal ftpsync.
Everything worked fine after that.

What I want to emphasis is that such slow connection to syncpxy.cna.d.o
won't lead to problems if the servers are kept in pace with the upstream.
But it can be a problem if a server is out of sync for one or two days.
I think maybe another account on syncproxy2.wna.d.o can be opened and
we can get a faster connection.