scp over ssh -R is slow

[Charles William Dick]

I'd like some help diagnosing a performance issue with ssh tunnels in the Secure Shell App.

Background

I'm working from home on ChromeOS.

My build server is remote, and my test device is connected to my local network. I flash my test device by using an `ssh -R` tunnel to get from my local (corp joined) ChromeOS laptop to my remote build server, so that the build server can then use localhost to connect to my test device.

This all works very well, except that flashing the test device is very slow. I can only get max ~1MiB/s through the tunnel.

What I've Done

I also noticed that CPU was quite high on my corp ChromeBook:

Between the Native Client module and the App, it used ~110-120% of one core, which made me suspect it was CPU bound on a lock or something.

Both ChromeOS tracing and Perfetto traces didn't really tell me much about what was going on.

A coworker with a MacBook is able to push ~13MiB/s through a -R tunnel, and I was able to push ~10MiB/s through a -L tunnel on a FreeBSD box on my local network.

Call To Action

Does anyone know how to get scheduling/CPU profiles of the Secure Shell App and native client module?

Does anyone know what the issue could be? It might just be that I need to specify a larger window somewhere...

Thanks for any help,

-Charles

[Mike Frysinger]

1MB/s sounds a bit low, but at this point, that's somewhat in ballpark of expectations.  the killer part is really going through the relay server.  b/155522198 has some stats, but basically atm, i think 3MB/s might be the ceiling to expect.

-mike

--
You received this message because you are subscribed to the Google Groups "chromium-hterm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-hter...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-hterm/f32b280a-b771-4eb1-b9b5-bde6a7fc1d24n%40chromium.org.

[Charles William Dick]

My coworkers are able to get 12MiB/s when using MacOS ssh for the `-R`.

I am able to get ~10MiB/s when changing the direction of the tunnel and doing an `ssh -L` from my build server to a FreeBSD box I have on my network.

I would like to verify by profiling ChromeOS sshd. Is there any tool I can use for that? I have not had luck with Perfetto or chrome://tracing/, but I might not have the correct tracing configuration. I am concerned about the high CPU reported by task manager.

This (and lack of mosh support/how easily ssh disconnects) is my biggest pain point with ChromeOS.

[Mike Frysinger]

i assume the macOS testing is using the native ssh which isn't really comparable

i don't know what you mean by "ChromeOS sshd".  there is no sshd in CrOS by design.  you using ssh+port forwarding on CrOS to connect a corp desktop with a DUT on your LAN is only running the client.  there are no tools i'm aware of to profile the NaCl stack regardless of the OS you're using.

mosh support exists now, but it doesn't work through corp relays.  there's not much we can do on the client side when the relay only supports TCP.

-mike

[Charles William Dick]

Sorry, I meant ssh, not sshd.

Why is the mac ssh client not comparable? Do you think the bottleneck is the relay or the ssh client?

--

-Charles

[Mike Frysinger]

the macOS client is single process native code hitting the kernel directly through the network stack.  Chrome Secure Shell is running multiprocess/multithreaded in the browser with data traversing the NaCl/browser/JS layers before it even hits the kernel.  i think it's unrealistic to expect these to be comparable.

that said, we don't know how close is theoretically possible.  we're not investigating it because the NaCl stack is being killed entirely.  once the WASM port is more active, we can spend some time on profiling the new stack as the expectation is we'll be on a platform that is (mostly) going to be around for the next 10+ years.

i think there were some threads on g/chromeos-discuss for WFH.  i never use the flow you describe (as i do the vast majority of my work in VMs), so i don't have immediate tips for you.  sorry.

-mike

[Charles William Dick]

Is it possible to try out the new version?

--

-Charles

[Mike Frysinger]

unfortunately, not yet. the CLs are still in progress. tbh, I'm expecting there to be regressions in perf at first. so don't bank on it ;).

if you really want to dive into the current code to see if there are improvements to be found, it's possible from CrOS. imo you'd want to use crostini to checkout the code and host your tools as it'll be a lot easier. it's how I do 100% of my development. probably want to focus on the JS side since that's what the relay forces -- pipes over a websocket.

I don't want to discourage helping hands, just set expectations that the current stack is messy. that's what you get from a 10year code base -- the state of the world has changed around us.

-mike