Broken ssh trigger of mirrors.tuna.tsinghua.edu.cn from syncproxy.cna.debian.org

20 views
Skip to first unread message

Bastian Blank

unread,
Jan 26, 2017, 9:27:36 AM1/26/17
to TUNA, mir...@debian.org
Hi

I'm contacting you because you are listed as contact for
mirrors.tuna.tsinghua.edu.cn.

The ssh trigger from syncproxy.cna.debian.org does not work with the
following error message:

| ssh: connect to host mirrors.tuna.tsinghua.edu.cn port 23222: Connection refused

Please provide us with up-to-date information how we can do ssh triggers
to this mirror.

I attached the information how to setup ssh triggers for your convinience.

Please add the following SSH keys to the authorized_keys file of the
user:

command="~/bin/ftpsync",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNZoV9j9KFMFssqLLbKF0b35npFAE3USHgtyN3pUsFm8Wsh1i4ZVly+8Nks7LVJrAI8uCrfvZKyQrqdcjAyy7UAGXfz5YrMxj0PZbvz9rHWa7Hxl+wO5mYwtklN8ek7uFgqDCmMOkE+4uIxOlPzDJ95BUBx5SVm8jAaMmaBoX20rxO5BGvguVU4RJ1V9EZKB4B9pGbdScRJOWAGUrg8HSKJGtlNQTF0xIU6P6quF0O+qpjF9CiB2GVzRJZXNpAhlekpVx5/Xg6N+sIN7QxxHViO7xwTQgteUehbrpToNc1N4jf4kciHmP/jdgEus/W61U410QfXVUYKOTDuGzAotl/

Please do not restrict access to the system or the keys to specific IP.
If you must, please whitelist the following IP for access:

- 128.101.240.212
- 2607:ea00:101:3c0b::1deb:212

Regards,
Bastian, for Debian mirror team

--
The sight of death frightens them [Earthers].
-- Kras the Klingon, "Friday's Child", stardate 3497.2

Miao Wang

unread,
Jan 26, 2017, 11:37:17 AM1/26/17
to mir...@debian.org, thu-opensourc...@googlegroups.com, TUNA Mirror Team
Hi

I feel sorry about not informing you after last hardware update.
The ssh key has been set up. Here's the updated information about
ssh triggers.

There are now two servers serving the mirror files, providing
redundant and load balance. The domain `mirrors.tuna.tsinghua.edu.cn`
will randomly point to one of them. Therefore, please do not
connect directly to the domain. Instead, connect and trigger
both of them.

* Server1
Domain: neomirrors.tuna.tsinghua.edu.cn
Username: tunasync
Port: 22222
Host public key fingerprint:
256 5e:2c:35:24:1a:66:02:b0:2e:9a:6b:40:09:7e:ee:ac (ECDSA)
256 77:a8:f8:05:81:e2:e3:f5:ac:24:dd:f5:ea:64:1f:37 (ED25519)
2048 cf:90:ad:5b:01:e7:5e:a9:7d:dc:49:b9:39:b5:e0:55 (RSA)

* Server2
Domain nanomirrors.tuna.tsinghua.edu.cn
Username: tunasync
Port: 22222
Host public key fingerprint:
256 c6:2d:4d:0f:16:0a:fe:6f:fb:b3:f6:b8:c5:00:34:3a (ECDSA)
256 3a:7b:dd:54:26:62:50:53:f6:25:88:ec:0c:3a:22:47 (ED25519)
2048 bf:78:29:c1:44:bb:ef:e4:26:e3:78:d7:bb:22:41:03 (RSA)

If there should any further questions, please do not hesitate to
contact us.

Cheers,


**Please verify the digital signature attached with the e-mail.**
Miao Wang
Department of Computer Science and Technology, Tsinghua University
Add.: Zijing Apartment, Tsinghua University, Peking. P.R.C. 100084
> --
> 您收到此邮件是因为您订阅了 Google 网上论坛的“THU Opensource Mirror Admin”群组。
> 要退订此群组并停止接收此群组的电子邮件,请发送电子邮件到thu-opensource-mirr...@googlegroups.com
> 要查看更多选项,请访问 https://groups.google.com/d/optout



Justin Wong

unread,
Jan 28, 2017, 12:39:34 PM1/28/17
to Bastian Blank, TUNA Mirror Team, mir...@debian.org, thu-opensourc...@googlegroups.com
Hi there,

tunasync [1] is our mirror job scheduler. We do run our mirror job using
ftpsync, which is scheduled by tunasync.
Besides scheduling, tunasync also collects logs and job status.

As we have to manage a bunch of mirror jobs for different types of
mirror protocols, this scheduler, or middleware is
essential to us.

The output info you got is the returned by the `tunasynctl start debian`
command, which signals the backend to start syncing debian with ftpsync.
As the tick unit of the scheduler is 5 seconds, the job does not start
intermediately. There are also some other conditions that may delay the
syncing, such as concurrent job limitation.

So don't worry, we do sync debian with ftpsync, and every push is queued
and ensured to trigger the syncing.

Cheers,
Justin

[1] https://github.com/tuna/tunasync/

--
Justin Wong

On Sun, Jan 29, 2017, at 01:28, Bastian Blank wrote:
> On Fri, Jan 27, 2017 at 12:37:03AM +0800, Miao Wang wrote:
> > If there should any further questions, please do not hesitate to
> > contact us.
>
> I can connect to both of them. However I neither see the usual ftpsync
> reaction on invalid archive names, nor an rsync connection back to the
> syncproxy. Instead I get on the ssh connection:
>
> | /home/tunasync/.config/tunasync/ctl.conf
> | [17-01-29 01:25:35][INFO] Use manager address: http://127.0.0.1:14242
> | [17-01-29 01:25:35][INFO] Succesfully send command
>
> Can you please that ftpsync is actually called?
>
> Regards,
> Bastian
>
> --
> Every living thing wants to survive.
> -- Spock, "The Ultimate Computer", stardate 4731.3

Shanker Wang

unread,
Jan 29, 2017, 10:16:31 AM1/29/17
to Bastian Blank, mir...@debian.org, TUNA Mirror Team, thu-opensourc...@googlegroups.com
Hi there,

As log from our server, the trigger functions well. Here are the detailed
logs and the time zone is UTC+8 (except log from ftpsync).

## log on neomirrors.tuna.tsinghua.edu.cn

login log:

```
Jan 29 01:19:30 localhost sshd[44950]: Accepted publickey for tunasync from 2607:ea00:101:3c0b::1deb:216 port 36693 ssh2: RSA dd:0a:0d:0e:fd:bf:74:48:0d:3b:be:25:6c:e9:c0:6a
Jan 29 01:19:30 localhost sshd[44950]: pam_unix(sshd:session): session opened for user tunasync by (uid=0)
Jan 29 01:19:32 localhost sshd[44995]: Received disconnect from 2607:ea00:101:3c0b::1deb:216: 11: disconnected by user
Jan 29 01:19:32 localhost sshd[44950]: pam_unix(sshd:session): session closed for user tunasync
```

scheduler log:

```
1月 29 01:19:32 neomirrors tunasync[35589]: [NOTICE] Received command: start (debian)
1月 29 01:19:32 neomirrors tunasync[35589]: [NOTICE] start syncing: debian
1月 29 02:56:19 neomirrors tunasync[35589]: [NOTICE] succeeded syncing debian
1月 29 02:56:19 neomirrors tunasync[35589]: [NOTICE] Next scheduled time for debian: 2017-01-29 14:56:19
```

ftpsync-debian.log
```
Jan 28 17:19:35 14ff00e2939d ftpsync-debian[7]: Mirrorsync start
Jan 28 17:19:35 14ff00e2939d ftpsync-debian[7]: Running mirrorsync, update is required, /data/mirrors/debian//Archive-Update-Required-mirrors.tuna.tsinghua.edu.cn exists
Jan 28 17:19:35 14ff00e2939d ftpsync-debian[7]: Running stage1: rsync -e /ftpsync/bin/rsync-ssl-tunnel -6 --safe-links --bwlimit=0 -prltvHSB8192 --timeout 3600 --stats --filter=protect_Archive-Update-in-Progress-mirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/mirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/mirrors.tuna.tsinghua.edu.cn-stage1 --exclude=project/trace/_hierarchy --filter=protect_project/trace/_hierarchy --exclude=project/trace/_traces --filter=protect_project/trace/_traces --filter=protect_Archive-Update-Required-mirrors.tuna.tsinghua.edu.cn --exclude=Packages* --exclude=Sources* --exclude=Release* --exclude=InRelease --include=i18n/by-hash/** --exclude=i18n/* --exclude=ls-lR* --exclude=.~tmp~/ mirrors.tuna.t...@syncproxy.cna.debian.org::debian /data/mirrors/debian/
Jan 28 18:09:35 14ff00e2939d ftpsync-debian[7]: Back from rsync with returncode 0
Jan 28 18:09:36 14ff00e2939d ftpsync-debian[7]: Latest recorded rsync transfer speed: 792 KB/s
Jan 28 18:09:36 14ff00e2939d ftpsync-debian[7]: Running stage2: rsync -e /ftpsync/bin/rsync-ssl-tunnel -6 --safe-links --bwlimit=0 -prltvHSB8192 --timeout 3600 --stats --filter=protect_Archive-Update-in-Progress-mirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/mirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/mirrors.tuna.tsinghua.edu.cn-stage1 --exclude=project/trace/_hierarchy --filter=protect_project/trace/_hierarchy --exclude=project/trace/_traces --filter=protect_project/trace/_traces --filter=protect_Archive-Update-Required-mirrors.tuna.tsinghua.edu.cn --max-delete=40000 --delay-updates --delete --delete-excluded --delete-delay --exclude=.~tmp~/ mirrors.tuna.t...@syncproxy.cna.debian.org::debian /data/mirrors/debian/
Jan 28 18:53:51 14ff00e2939d ftpsync-debian[7]: Back from rsync with returncode 0
Jan 28 18:53:51 14ff00e2939d ftpsync-debian[7]: Latest recorded rsync transfer speed: 104 KB/s
Jan 28 18:53:51 14ff00e2939d ftpsync-debian[7]: Creating a full trace file
Jan 28 18:56:19 14ff00e2939d ftpsync-debian[7]: Mirrorsync done
```

In summary, the sync on server neomirrors.tuna.tsinghua.edu.cn started
immediately when triggered, and finished successfully in one hour and
40 minute.

## log on nano mirrors.tuna.tsinghua.edu.cn

login log:

```
Jan 29 01:20:16 localhost sshd[3807]: Accepted publickey for tunasync from 2607:ea00:101:3c0b::1deb:216 port 50774 ssh2: RSA dd:0a:0d:0e:fd:bf:74:48:0d:3b:be:25:6c:e9:c0:6a
Jan 29 01:20:16 localhost sshd[3807]: pam_unix(sshd:session): session opened for user tunasync by (uid=0)
Jan 29 01:20:17 localhost sshd[3812]: Received disconnect from 2607:ea00:101:3c0b::1deb:216: 11: disconnected by user
Jan 29 01:20:17 localhost sshd[3807]: pam_unix(sshd:session): session closed for user tunasync
Jan 29 01:20:54 localhost sshd[4171]: Accepted publickey for tunasync from 2607:ea00:101:3c0b::1deb:216 port 39195 ssh2: RSA dd:0a:0d:0e:fd:bf:74:48:0d:3b:be:25:6c:e9:c0:6a
Jan 29 01:20:54 localhost sshd[4171]: pam_unix(sshd:session): session opened for user tunasync by (uid=0)
Jan 29 01:20:55 localhost sshd[4174]: Received disconnect from 2607:ea00:101:3c0b::1deb:216: 11: disconnected by user
Jan 29 01:20:55 localhost sshd[4171]: pam_unix(sshd:session): session closed for user tunasync
Jan 29 01:24:11 localhost sshd[10126]: Accepted publickey for tunasync from 2607:ea00:101:3c0b::1deb:216 port 50120 ssh2: RSA dd:0a:0d:0e:fd:bf:74:48:0d:3b:be:25:6c:e9:c0:6a
Jan 29 01:24:12 localhost sshd[10126]: pam_unix(sshd:session): session opened for user tunasync by (uid=0)
Jan 29 01:24:12 localhost sshd[10196]: Received disconnect from 2607:ea00:101:3c0b::1deb:216: 11: disconnected by user
Jan 29 01:24:12 localhost sshd[10126]: pam_unix(sshd:session): session closed for user tunasync
Jan 29 01:25:34 localhost sshd[14268]: Accepted publickey for tunasync from 2607:ea00:101:3c0b::1deb:216 port 36572 ssh2: RSA dd:0a:0d:0e:fd:bf:74:48:0d:3b:be:25:6c:e9:c0:6a
Jan 29 01:25:34 localhost sshd[14268]: pam_unix(sshd:session): session opened for user tunasync by (uid=0)
Jan 29 01:25:35 localhost sshd[14369]: Received disconnect from 2607:ea00:101:3c0b::1deb:216: 11: disconnected by user
Jan 29 01:25:35 localhost sshd[14268]: pam_unix(sshd:session): session closed for user tunasync
```

scheduler log:

```
1月 28 18:05:33 nanomirrors tunasync[13005]: [NOTICE] start syncing: debian
1月 29 01:20:17 nanomirrors tunasync[13005]: [NOTICE] Received command: start (debian)
1月 29 01:20:54 nanomirrors tunasync[13005]: [NOTICE] Received command: start (debian)
1月 29 01:25:35 nanomirrors tunasync[13005]: [NOTICE] Received command: start (debian)
1月 29 06:55:39 nanomirrors tunasync[13005]: [NOTICE] succeeded syncing debian
1月 29 06:55:39 nanomirrors tunasync[13005]: [NOTICE] Next scheduled time for debian: 2017-01-29 18:55:39
```

ftpsync-debian.log:

```
Jan 28 10:05:33 8b6c293292ae ftpsync-debian[7]: Mirrorsync start
Jan 28 10:05:33 8b6c293292ae ftpsync-debian[7]: Running mirrorsync, update is required, /data/mirrors/debian//Archive-Update-Required-mirrors.tuna.tsinghua.edu.cn exists
Jan 28 10:05:33 8b6c293292ae ftpsync-debian[7]: Running stage1: rsync -e /ftpsync/bin/rsync-ssl-tunnel -6 --safe-links --bwlimit=0 -prltvHSB8192 --timeout 3600 --stats --filter=protect_Archive-Update-in-Progress-mirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/mirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/mirrors.tuna.tsinghua.edu.cn-stage1 --exclude=project/trace/_hierarchy --filter=protect_project/trace/_hierarchy --exclude=project/trace/_traces --filter=protect_project/trace/_traces --filter=protect_Archive-Update-Required-mirrors.tuna.tsinghua.edu.cn --exclude=Packages* --exclude=Sources* --exclude=Release* --exclude=InRelease --include=i18n/by-hash/** --exclude=i18n/* --exclude=ls-lR* --exclude=.~tmp~/ mirrors.tuna.t...@syncproxy.cna.debian.org::debian /data/mirrors/debian/
Jan 28 21:35:27 8b6c293292ae ftpsync-debian[7]: Back from rsync with returncode 0
Jan 28 21:35:27 8b6c293292ae ftpsync-debian[7]: Latest recorded rsync transfer speed: 983 KB/s
Jan 28 21:35:27 8b6c293292ae ftpsync-debian[7]: Running stage2: rsync -e /ftpsync/bin/rsync-ssl-tunnel -6 --safe-links --bwlimit=0 -prltvHSB8192 --timeout 3600 --stats --filter=protect_Archive-Update-in-Progress-mirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/mirrors.tuna.tsinghua.edu.cn --filter=protect_project/trace/mirrors.tuna.tsinghua.edu.cn-stage1 --exclude=project/trace/_hierarchy --filter=protect_project/trace/_hierarchy --exclude=project/trace/_traces --filter=protect_project/trace/_traces --filter=protect_Archive-Update-Required-mirrors.tuna.tsinghua.edu.cn --max-delete=40000 --delay-updates --delete --delete-excluded --delete-delay --exclude=.~tmp~/ mirrors.tuna.t...@syncproxy.cna.debian.org::debian /data/mirrors/debian/
Jan 28 22:55:21 8b6c293292ae ftpsync-debian[7]: Back from rsync with returncode 0
Jan 28 22:55:21 8b6c293292ae ftpsync-debian[7]: Latest recorded rsync transfer speed: 459 KB/s
Jan 28 22:55:21 8b6c293292ae ftpsync-debian[7]: Creating a full trace file
Jan 28 22:55:39 8b6c293292ae ftpsync-debian[7]: Mirrorsync done
```
In summary, the sync on neomirrors.tuna.tsinghua.edu.cn was in
progress and the trigger was ignored as a result.


I think by adding the scheduler, the default behavior of `ftpsync`
trigger is not changed. If there is further problem, please do not
hesitate to contact us.

Cheers,

**Please verify the digital signature attached with the e-mail.**
Miao Wang
Department of Computer Science and Technology, Tsinghua University
Add.: Zijing Apartment, Tsinghua University, Peking. P.R.C. 100084

Reply all
Reply to author
Forward
0 new messages