ISD connectivity problem

[Dmitry Dugaev]

Dear SCION community,

I have the following SCION configuration:

<local AS 19-ffaa:1:232> --- <19-ffaa:0:1301 (OVGU)> --- <WAN> --- <17-ffaa:0:1107 (ETHZ)> --- <local AS 17-ffaa:1:233>

   (VM with public IP)                                                                                                                               (VM with public IP)

This configuration was working well, I was able to "pinpong" my 2 VMs with scion-daemon between each other.

Last week I tried to run the same configuration, however, the connectivity with the adjacent ISDs (19 and 17) seems to be broken now.

The beacon server keeps issuing "revocations" in the log-files:

2019-02-14 21:50:42.491280+0000 [INFO] (BS._handle_if_timeouts) Issuing revocation: RevInfo: 17-ffaa:1:233 IF: 1 Link type: PARENT Timestamp: 2019-02-14 21:50:42+00:00 TTL: 20s

2019-02-14 21:50:42.494335+0000 [DEBUG] (BS._handle_if_timeouts) IFState update to 127.0.0.1:30042

2019-02-14 21:50:42.495009+0000 [DEBUG] (BS._handle_if_timeouts) IFState update to 127.0.0.1:31044

2019-02-14 21:50:44.988965+0000 [DEBUG] (BS.worker) Expired 1 old entries from /17-ffaa:1:233/bs/rev_cache

Is this issue somehow connected to the SCIONLab update, which was announced on 23rd of January?

How can I update the SCION daemon and/or topology in order to get this configuration working again?

Thanks!

[Juan A. Garcia Pardo]

Hi Dmitry,

It could be that one or both of your ASes are not up to date with the SCIONLab code. 

If your ASes are running inside a VM obtained via the Coordinator, they should update automatically. If you are running the ASes services in a dedicated machine, meaning that you installed SCION yourself in the machines, then you need to trigger the update manually. In this case please read https://netsec-ethz.github.io/scion-tutorials/scion_tricks/updating_scion/#running-scion-in-a-dedicated-machine

Please tell us if you still experience problems.

Best regards,

Juan A.

[Dmitry Dugaev]

Hello Juan,

I tried to follow the steps you suggested: https://www.google.com/url?q=https%3A%2F%2Fnetsec-ethz.github.io%2Fscion-tutorials%2Fscion_tricks%2Fupdating_scion%2F%23running-scion-in-a-dedicated-machine&sa=D&sntz=1&usg=AFQjCNGriNZ3_zzElEL8FxOLjcBmaCXHvw

however, the update script was failing for some reason, so I came up with more radical solution.

Instead, I just re-installed the system on my machines, and deployed a fresh SCION installation there again.

It went okay, but then I tried to pingpong the remote AS (ISD18), and got the following issue I haven't observed before:

./bin/pingpong -local 19-ffaa:1:232,[127.0.0.1]:0 -remote 18-ffaa:0:1202,[128.105.21.208]:40002

2019-02-21 20:44:28.476227+0000 [CRIT] Unable to initialize QUIC/SCION err=

>  squic: Unable to load TLS cert/key

>      open gen-certs/tls.pem: no such file or directory

Could you please give me a hint where the problem is?

The sciond seems to be working, I can see some successful verifications in the BS logs.

Thanks!

--

Dmitrii

[Juan A. Garcia Pardo]

Hi Dmitry,

It would be great if you could send us the failure for us to reproduce.

Aside from that, the error you are getting comes from `pingpong` not finding a certificate in `gen-certs/tls.pem`. This is not directly related to the SCION architecture, but to QUIC. You can run the following to get a new certificate:

---------------8<----------

old=$(umask)

mkdir -p "gen-certs"

umask 0177

openssl genrsa -out "gen-certs/tls.key" 2048

umask "$old"

openssl req -new -x509 -key "gen-certs/tls.key" -out "gen-certs/tls.pem" -days 3650 -subj /CN=scion_def_srv

---------------8<----------

This is what we do in `./scion.sh` when testing a new topology. 

Please check that `scmp echo` works (even without this certificate), and that `pingpong` works again after running the steps above.

Best regards,

Juan A.

[Dmitry Dugaev]

Hello Juan,

I've executed those commands, and the certificate error was resolved. Thanks.

However, there was another problem after that. Eventually, I managed to solve it and get my SCION installation work, but I just want to describe the problem to you, so you could pay attention to it later.

So, after solving the problem with the certificates, I tried to "pingpong" the ISD19 with the following command, but got this error:

./bin/pingpong -local 19-ffaa:1:232,[127.0.0.1]:0 -remote 18-ffaa:0:1202,[128.105.21.208]:40002

2019-02-22 23:23:18.644478+0000 [CRIT] No paths available to remote destination

I've observed that issue before, and in my case, the problem was that I use AWS t2.micro instances, which are behind "amazon NAT". I.e., my AWS instance has an internal IP which is different from the public IP of the same instance, if one tries to reach it from the Internet and vice-versa.

So, I tried to execute the command, mentioned here: https://netsec-ethz.github.io/scion-tutorials/general_scion_configuration/public_ip_nat/

BIND_IP=<my_internal_ip>; sed -e '/"Bind"/{:x n;s/\("Addr": \)".*"/\1"'${BIND_IP}'"/;Tx}' -i ./gen/*/*/*/topology.json

This command worked for me before your major SCION updates.

But now, the command does not actually work since nothing is changed in the topology.json files after its execution.

Maybe the format of the "topology.json" file has also been changed during the update? I'm not very good at "sed", so I'm just guessing.

Anyway, then I just manually replaced <instance_public_ip> with <instance_local_ip> in the "topology.json" files, and everything started to work just fine.

I was able to successfully run "pingpong" and "scmp echo" commands:

./bin/pingpong -local 19-ffaa:1:232,[120.0.1]:0 -remote 18-ffaa:0:1202,[128.105.21.208]:40002

Using path:

  Hops: [19-ffaa:1:232 1>69 19-ffaa:0:1303 1>5 19-ffaa:0:1301 3>5 18-ffaa:0:1201 6>1 18-ffaa:0:1202] Mtu: 1472

Received 13 bytes from 18-ffaa:0:1202,[128.105.21.208]:40002 (UDP): seq=0 RTT=275.897ms

Received 13 bytes from 18-ffaa:0:1202,[128.105.21.208]:40002 (UDP): seq=1 RTT=274.911ms

Received 13 bytes from 18-ffaa:0:1202,[128.105.21.208]:40002 (UDP): seq=2 RTT=274.974ms

Received 13 bytes from 18-ffaa:0:1202,[128.105.21.208]:40002 (UDP): seq=3 RTT=274.965ms

./bin/scmp echo -local 19-ffaa:1:232,[127.0.0.1] -remote 18-ffaa:0:1202,[128.105.21.208]

Using path:

  Hops: [19-ffaa:1:232 1>69 19-ffaa:0:1303 1>5 19-ffaa:0:1301 3>5 18-ffaa:0:1201 6>1 18-ffaa:0:1202] Mtu: 1472

152 bytes from 18-ffaa:0:1202,[128.105.21.208] scmp_seq=0 time=275.008ms

152 bytes from 18-ffaa:0:1202,[128.105.21.208] scmp_seq=1 time=274.139ms

152 bytes from 18-ffaa:0:1202,[128.105.21.208] scmp_seq=2 time=275.186ms

^C

--- 18-ffaa:0:1202,[[128.105.21.208]] statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2.591083s

Hope this information was useful.

Thanks a lot for the help!

p.s.

regarding the update script, which was failing before all of this. Unfortunately, I've erased my previous installations, so I can't reproduce the problem. Sorry about that.

BR,

Dmitrii

[FR4NK]

Hello Dmitry

Thank you for bringing this to our attention.

We will update the sed command in the tutorial you linked.

Cheers