Packet Timing and Data Leaks

[Chris Rapier]

Howdy all,

So, one night over beers I was telling a friend how you could use the
timing between key presses on a type writer to extract information.
Basically, you make some assumptions about the person typing (touch
typing at so many words per second and then fuzzing the parameters until
words come out).

The I found a paper written back in 2001 talked about using the
interpacket timing in interactive sessions to leak information.
https://people.eecs.berkeley.edu/~dawnsong/papers/ssh-timing.pdf

I'm sure this has been addressed (or dismissed) but I'm looking for the
specific section of code that might deal with this. Any pointers?

Thanks,

Chris
_______________________________________________
openssh-unix-dev mailing list
openssh-...@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

[Nico Kadel-Garcia]

On Thu, Aug 3, 2023 at 2:35 PM Chris Rapier <rap...@psc.edu> wrote:
>
> Howdy all,
>
> So, one night over beers I was telling a friend how you could use the
> timing between key presses on a type writer to extract information.
> Basically, you make some assumptions about the person typing (touch
> typing at so many words per second and then fuzzing the parameters until
> words come out).
>
> The I found a paper written back in 2001 talked about using the
> interpacket timing in interactive sessions to leak information.
> https://people.eecs.berkeley.edu/~dawnsong/papers/ssh-timing.pdf
>
> I'm sure this has been addressed (or dismissed) but I'm looking for the
> specific section of code that might deal with this. Any pointers?
>
> Thanks,
>
> Chris

Why would any code "deal with this"? What you're describing is a form
of steganography, embedding another subtler, more dangerous message
inside the larger, safer, apparent message. Because there are so many
ways to hide such data, it's not even theoretically possible to avoid
altogether. OpenSSH would be the wrong layer of the stack to
manipulate this, especially given the attempts to improve interactive
behavior by improving responsiveness and reducing latency. If you're
making a system as swift and responsive as possible, it becomes very
difficult to regulate the timing of what may be human driven
interactions. And if you're going to manipulate packet delays.....
that's at the network layer or data layer, layer 2 or layer 3 of the
OSI stack. OpenSSH is more at the transport layer.

I don't see how OpenSSH could be written to avoid this.without
sacrificing responsive of interactive sessions.

[Damien Miller]

On Thu, 3 Aug 2023, Chris Rapier wrote:

> Howdy all,
>
> So, one night over beers I was telling a friend how you could use the timing
> between key presses on a type writer to extract information. Basically, you
> make some assumptions about the person typing (touch typing at so many words
> per second and then fuzzing the parameters until words come out).
>
> The I found a paper written back in 2001 talked about using the interpacket
> timing in interactive sessions to leak information.
> https://people.eecs.berkeley.edu/~dawnsong/papers/ssh-timing.pdf
>
> I'm sure this has been addressed (or dismissed) but I'm looking for the
> specific section of code that might deal with this. Any pointers?

The main issue raised in that paper was that it was trivially detectable
when terminal echo was switched off and so an attacker could specifically
observe the moments when users were typing their passwords into (say)
sudo. This got fixed around the time the paper was released IIRC,
search for "Simulate echo" in channels.c:channel_handle_wfd().

The broader issue of hiding all potential keystroke timing is not yet fixed.

-d

[Howard Chu]

Damien Miller wrote:
> On Thu, 3 Aug 2023, Chris Rapier wrote:
>
>> Howdy all,
>>
>> So, one night over beers I was telling a friend how you could use the timing
>> between key presses on a type writer to extract information. Basically, you
>> make some assumptions about the person typing (touch typing at so many words
>> per second and then fuzzing the parameters until words come out).
>>
>> The I found a paper written back in 2001 talked about using the interpacket
>> timing in interactive sessions to leak information.
>> https://people.eecs.berkeley.edu/~dawnsong/papers/ssh-timing.pdf
>>
>> I'm sure this has been addressed (or dismissed) but I'm looking for the
>> specific section of code that might deal with this. Any pointers?
>
> The main issue raised in that paper was that it was trivially detectable
> when terminal echo was switched off and so an attacker could specifically
> observe the moments when users were typing their passwords into (say)
> sudo. This got fixed around the time the paper was released IIRC,
> search for "Simulate echo" in channels.c:channel_handle_wfd().
>
> The broader issue of hiding all potential keystroke timing is not yet fixed.

The keystroke timing issue would be solved by adding LINEMODE support as I did back in 2010.
https://lists.mindrot.org/pipermail/openssh-unix-dev/2010-June/028732.html

The code is still available here https://github.com/hyc/OpenSSH-LINEMODE/
If there's sufficient interest this time, I can probably bring it all up to date with
a current OpenSSH version. I won't bother if it meets the same apathy as last time.

--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/

[Thorsten Glaser]

On Sun, 6 Aug 2023, Howard Chu wrote:

>The keystroke timing issue would be solved by adding LINEMODE support as I did back in 2010.
>https://lists.mindrot.org/pipermail/openssh-unix-dev/2010-June/028732.html

Local line editing by using GNU libreadline? *shudder* No, thanks.

bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also, https://www.tarent.de/newsletter
╱ ╲ header encryption!
****************************************************

[Chris Rapier]

Nico,

This isn't steganography but a timing attack. However, the timing attack
isn't focused on the cipher but the user. Basically, let's assume that
you are an attacker. All you have are the timing sequences of packets
between a client and a host. The question is if you can take the timing
between each packet to tell you anything about the data being sent.

You can make some assumptions -
1) It's an interactive session.
2) The person behind the interactive session is trained in touch typing
(this won't work as well with self taught/hunk and peck/whatever typists
as well because the "fist" (if we want to go back the old Morse code
terminology is going be much more idiosyncratic and harder to model).

What you know is that it takes a certain amount of time for a finger to
travel from the home row to a specific key and from that key to another
key in a well known language. You also know that SSH doesn't delay
sending packets and that each individual character is represented by a
new packet. As such the pause between each individual packet roughly
corresponds to the time between each key press.

Since you can model the time between key presses (given that person is
trained in touch typing) for pairs of letters you may be able to use
that to gain some entry into what the person is actually typing during
the session.

If you can figure out even a relatively small percentage of letters you
can possibly recreate large portions of the text and, from that, maybe
even figure out the encryption key being used.

The Soviets did something similar to this when they bugged the American
embassy in Moscow. However, in this case they did timing attack on the
movement of the type ball in IBM Selectrics which might have made it
easier.
https://arstechnica.com/information-technology/2015/10/how-soviets-used-ibm-selectric-keyloggers-to-spy-on-us-diplomats/

Chris

[Chris Rapier]

On 8/6/23 3:46 AM, Damien Miller wrote:
> On Thu, 3 Aug 2023, Chris Rapier wrote:
>
>> Howdy all,
>>
>> So, one night over beers I was telling a friend how you could use the timing
>> between key presses on a type writer to extract information. Basically, you
>> make some assumptions about the person typing (touch typing at so many words
>> per second and then fuzzing the parameters until words come out).
>>
>> The I found a paper written back in 2001 talked about using the interpacket
>> timing in interactive sessions to leak information.
>> https://people.eecs.berkeley.edu/~dawnsong/papers/ssh-timing.pdf
>>
>> I'm sure this has been addressed (or dismissed) but I'm looking for the
>> specific section of code that might deal with this. Any pointers?
>
> The main issue raised in that paper was that it was trivially detectable
> when terminal echo was switched off and so an attacker could specifically
> observe the moments when users were typing their passwords into (say)
> sudo. This got fixed around the time the paper was released IIRC,
> search for "Simulate echo" in channels.c:channel_handle_wfd().

Excellent! Thank you for the pointer.

>
> The broader issue of hiding all potential keystroke timing is not yet fixed.

Could some level of obfuscation come from enabling Nagle for interactive
sessions that has an associated TTY? Though that would be of limited
usefulness in low RTT environments. I don't like the idea of having a
steady drip of packets as that seems problematic both in terms of code
complexity and network usage. I also don't like the idea of imposing
random jitter though that might be easier to implement. However, without
actual modeling I have no idea if that would actually improve things.

Anyway, I was simply thinking about this because the whole bugging of
typewriters thing crossed my mind recently.

Chris

[Howard Chu]

Thorsten Glaser wrote:
> On Sun, 6 Aug 2023, Howard Chu wrote:
>
>> The keystroke timing issue would be solved by adding LINEMODE support as I did back in 2010.
>> https://lists.mindrot.org/pipermail/openssh-unix-dev/2010-June/028732.html
>
> Local line editing by using GNU libreadline? *shudder* No, thanks.

I also ported it to use libedit instead, but readline is more widely used.

--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/

[Thorsten Glaser]

On Mon, 7 Aug 2023, Howard Chu wrote:

>>> The keystroke timing issue would be solved by adding LINEMODE support as I did back in 2010.
>>> https://lists.mindrot.org/pipermail/openssh-unix-dev/2010-June/028732.html
>>
>> Local line editing by using GNU libreadline? *shudder* No, thanks.
>
>I also ported it to use libedit instead, but readline is more widely used.

Yeah, same point though. I actually did work with such a system once,
namely Android adb before they removed the local line editing part
once they had imported mksh, and it was awful. You lose any sort of
connection to the command line input mode of the remote shell (not
everyone uses a shell backed by libreadline/libedit), and even
passwords would show up in the scrollback, etc. but the worst is the
missing tab completion.

I also doubt it will catch many relevant use cases, e.g. editors.

bye,
//mirabilos
--
(gnutls can also be used, but if you are compiling lynx for your own use,
there is no reason to consider using that package)
-- Thomas E. Dickey on the Lynx mailing list, about OpenSSL

[Chris Rapier]

On 8/7/23 1:06 PM, Thorsten Glaser wrote:
> On Mon, 7 Aug 2023, Howard Chu wrote:
>
>>>> The keystroke timing issue would be solved by adding LINEMODE support as I did back in 2010.
>>>> https://lists.mindrot.org/pipermail/openssh-unix-dev/2010-June/028732.html
>>>
>>> Local line editing by using GNU libreadline? *shudder* No, thanks.
>>
>> I also ported it to use libedit instead, but readline is more widely used.
>
> Yeah, same point though. I actually did work with such a system once,
> namely Android adb before they removed the local line editing part
> once they had imported mksh, and it was awful. You lose any sort of
> connection to the command line input mode of the remote shell (not
> everyone uses a shell backed by libreadline/libedit), and even
> passwords would show up in the scrollback, etc. but the worst is the
> missing tab completion.
>
> I also doubt it will catch many relevant use cases, e.g. editors.

I think these are valid critiques and using something like this against
a "maybe it's an issue" thing is a bit heavy at this point. However, as
an intellectual exercise, could interpacket timing actually be a
potential information leak in an interactive ssh session? If so, then
how much a threat is it really? So assuming that it could be done and
that it's a reasonable threat how would we go about mitigating it?

Honestly, just curious about what people think. I don't know if this
could ever be a real issue or if I'm just being overly imaginative.

Chris

[Damien Miller]

On Mon, 7 Aug 2023, Chris Rapier wrote:

> > The broader issue of hiding all potential keystroke timing is not yet fixed.
>
> Could some level of obfuscation come from enabling Nagle for interactive
> sessions that has an associated TTY? Though that would be of limited
> usefulness in low RTT environments. I don't like the idea of having a steady
> drip of packets as that seems problematic both in terms of code complexity and
> network usage. I also don't like the idea of imposing random jitter though
> that might be easier to implement. However, without actual modeling I have no
> idea if that would actually improve things.

Nagle is usually turned off because it causes annoying perceptions of
non-deterministic latency.

For ssh, IMO sending interactive traffic on a fixed clock (e.g. every
2-4ms) instead of as soon as possible, and adding fake keystroke packets
for some interval after the user stops generating traffic is the way to
fix it. I've slowly been preparing for this by reworking the mainloop and
associated timers.

[Chris Rapier]

Sorry for top posting but I'm on my phone. My colleague just sent me this.
https://www.theregister.com/2023/08/07/audio_keystroke_security/

I don't know how relevant this is as of yet but based on a quick reading it
seems related.

So I fully support this work. If there is anything I can do please let me
know.

[Damien Miller]

On Mon, 7 Aug 2023, Chris Rapier wrote:

> Sorry for top posting but I'm on my phone. My colleague just sent me
> this. https://www.theregister.com/2023/08/07/audio_keystroke_security/
> I don't know how relevant this is as of yet but based on a quick
> reading it seems related. 

AFAIK using audio is a much easier target, as each key probably has
some acoustically-unique signal to define it in addition to the
inter-keystroke timing signal.

[Philipp Marek]

The linked paper says 5.7 bits of password entropy can be recovered by timing data; while the brute-force time will have changed in 22 years, this number should still be valid.

An easy workaround is to use a password manager (a plain file as a minimum) and to copy/paste passwords in - though that might violate other security preferences.

[Jochen Bern]

On 08.08.23 00:30, Damien Miller wrote:
> For ssh, IMO sending interactive traffic on a fixed clock (e.g. every
> 2-4ms) instead of as soon as possible, and adding fake keystroke packets
> for some interval after the user stops generating traffic is the way to
> fix it.

Top touch typing speeds get to about 100 wpm; with an average of 4.7
letters per word in English, that's a bit short of 10 keystrokes per
second, or 100 ms between. So, I'd guess that you could make that
quantization a *lot* coarser (*if* you can keep it reliably applying
only to manual input, and, e.g., side-step the algo when the user is
copy-pasting into the terminal window instead).

Of course, this will telegraph to any snoop that this mechanism *is* in
effect - and that minute changes in inter-packet delay might instead be
leaking information from the cryptalgorithm. If we want to keep them
guessing (and desparately doing more and more complex statistics),
sending characters in groups of two or three (as soon as either that
many have been entered, or we reach a dynamic-and-randomized timeout
waiting for that) might be a better approach.

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

[Chris Rapier]

On 8/7/23 10:14 PM, Damien Miller wrote:
> On Mon, 7 Aug 2023, Chris Rapier wrote:
>
>> Sorry for top posting but I'm on my phone. My colleague just sent me
>> this. https://www.theregister.com/2023/08/07/audio_keystroke_security/
>> I don't know how relevant this is as of yet but based on a quick
>> reading it seems related.
>
> AFAIK using audio is a much easier target, as each key probably has
> some acoustically-unique signal to define it in addition to the
> inter-keystroke timing signal.

I still have to read through the paper in more detail (reading a PDF on
my phone was less than optimal) to get more detail on how they did it.
It could be acoustic signals based on minute variations in each key. On
my cursory read it seemed like a lot of Deep Learning special sauce
thrown on top without enough details to really know what they were
really doing.

I might just write the authors and see what they have to say.

Chris

[Howard Chu]

Thorsten Glaser wrote:
> On Mon, 7 Aug 2023, Howard Chu wrote:
>
>>>> The keystroke timing issue would be solved by adding LINEMODE support as I did back in 2010.
>>>> https://lists.mindrot.org/pipermail/openssh-unix-dev/2010-June/028732.html
>>>
>>> Local line editing by using GNU libreadline? *shudder* No, thanks.
>>
>> I also ported it to use libedit instead, but readline is more widely used.
>
> Yeah, same point though. I actually did work with such a system once,
> namely Android adb before they removed the local line editing part
> once they had imported mksh, and it was awful. You lose any sort of
> connection to the command line input mode of the remote shell (not
> everyone uses a shell backed by libreadline/libedit), and even
> passwords would show up in the scrollback,

There's no reason for passwords to be mishandled. All ioctl's sent to the pty are
still forwarded to the other side, so the ssh client will know when the server
wants to turn echo off. It just sounds like the adb implementation was incompetent.

> etc. but the worst is the
> missing tab completion.

Tab completiom still works. The local client forwards TABs immediately to the server.

>
> I also doubt it will catch many relevant use cases, e.g. editors.

It works fine with ed and ex. A full screen editor like emacs doesn't use input lines
anyway, so that's not going to change.

>
> bye,
> //mirabilos

>


--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/

[Peter Stuge]

Philipp Marek wrote:
> An easy workaround is to use a password manager (a plain file as a minimum)

If you can/want to use a file then consider using a key instead.

publickey authentication is non-interactive on the wire and the key is
already unlocked so packet timing leaks nothing about your passphrase.


//Peter

[Philipp Marek]

>If you can/want to use a file then consider using a key instead.
>
>publickey authentication is non-interactive on the wire and the key is
>already unlocked so packet timing leaks nothing about your passphrase.

Well, I interpreted the original request as "needing a passphrase for non-ssh access" (like su, or local ipmi, etc).

If you can use a public/private key pair instead, of course that should be preferred.

[Darren Tucker]

On Thu, 10 Aug 2023 at 20:44, Peter Stuge <pe...@stuge.se> wrote:
> Philipp Marek wrote:
> > An easy workaround is to use a password manager (a plain file as a minimum)
>
> If you can/want to use a file then consider using a key instead.
>
> publickey authentication is non-interactive on the wire and the key is
> already unlocked so packet timing leaks nothing about your passphrase.

Just to be clear: SSH's password authentication is not subject to this
attack. The client sends the whole password in a single
SSH_MSG_USERAUTH_REQUEST packet (see RFC4252 section 8) so there's no
inter-keystroke timing. Same goes for keyboard-interactive (RFC4256).

The issue is that if during an interactive session you invoke
something that wants a password which it reads from the tty (eg su,
sudo, ssh-add or a second ssh invocation) that the potential exists to
leak keystroke timing.

If you needed to do a two-hop ssh then using either of pubkey via
agent forwarding or jumphost would avoid the timing leaving the client
machine.

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

[Damien Miller]

On Tue, 8 Aug 2023, Damien Miller wrote:

> On Mon, 7 Aug 2023, Chris Rapier wrote:
>
> > > The broader issue of hiding all potential keystroke timing is not yet fixed.
> >
> > Could some level of obfuscation come from enabling Nagle for interactive
> > sessions that has an associated TTY? Though that would be of limited
> > usefulness in low RTT environments. I don't like the idea of having a steady
> > drip of packets as that seems problematic both in terms of code complexity and
> > network usage. I also don't like the idea of imposing random jitter though
> > that might be easier to implement. However, without actual modeling I have no
> > idea if that would actually improve things.
>
> Nagle is usually turned off because it causes annoying perceptions of
> non-deterministic latency.
>
> For ssh, IMO sending interactive traffic on a fixed clock (e.g. every
> 2-4ms) instead of as soon as possible, and adding fake keystroke packets
> for some interval after the user stops generating traffic is the way to
> fix it. I've slowly been preparing for this by reworking the mainloop and
> associated timers.

Here's the beginning of an implementation of this. Far from complete,
but the timing quantisation part seems to work at least.

https://github.com/djmdjm/openssh-wip/pull/24

-d

[Chris Rapier]

As an aside, I talked to the lead author on that acoustic side channel
attack. He says "This is a very interesting parallel to our paper and I
am happy to tell you that the timing between keystrokes was not used as
part of our model. The only features extracted for our classifier were
the acoustic emanations of the individual keystrokes, so no TDoA or
inter-key timing was used."

Hopefully, I haven't give him and his team ideas for a new attack :)

Mitch and I will take a look at your pull if you like. It may take a bit
as we are finishing up some work on ChaCha20.

Chris