AKICIF: Undeciphered Enigma messages?

[Keith F. Lynch]

I've been reading up on the wartime German Enigma machines, and on how
they were deciphered. I think I understand the process well enough to
write a program to do it.

Even after deciphering was semi-automated with electro-mechanical
"bombes," there was a lot of manual tedium before and after, and many
messages weren't deciphered in time to be useful, in which case they
were usually permanently skipped over.

I'm curious whether there's an archive of intercepted but never
deciphered Enigma messages. If there is, I assume they're no longer
classified. I think it would be fun to decipher them all. Or has
someone else with the same idea beat me to it? Thanks.
--
Keith F. Lynch - http://keithlynch.net/
Please see http://keithlynch.net/email.html before emailing me.

[Martha Adams]

On 08/14/2018 10:18 PM, Keith F. Lynch wrote:
> I've been reading up on the wartime German Enigma machines, and on how
> they were deciphered. I think I understand the process well enough to
> write a program to do it.
>
> Even after deciphering was semi-automated with electro-mechanical
> "bombes," there was a lot of manual tedium before and after, and many
> messages weren't deciphered in time to be useful, in which case they
> were usually permanently skipped over.
>
> I'm curious whether there's an archive of intercepted but never
> deciphered Enigma messages. If there is, I assume they're no longer
> classified. I think it would be fun to decipher them all. Or has
> someone else with the same idea beat me to it? Thanks.

This is a really interesting idea, not least because it illustrates
technological / knowhow progress over time. Back in the late 1940's,
people imagined if you just destroyed the files and machines, the
special crypto knowledge that had developed thru WW2 was suppressed
for all time. After all, those fantastic Bombe machines, who could
imagine building one of those if they didn't know what their builders
knew? Using *advanced vacuum tube technology* no less?

Now here we are in today. I don't think you could excite much
interest in producing Bombes and their manuals except as a giveaway
to crypto freaks, who might see them as toys not for any serious
work. Likewise the special knowledge, now to be found nearly
anywhere among makers of crypto systems and resources. Yes, what
"man can do," man can do again, and pretty soon, as tech develops.
And whoever thinks that's an exaggeration wants to look at how the
tech for nuclear bombs and for chemical warfare, has grown and got
around today's world.

I think the central point in this that got missed, was that such
tech resources and warfare, are seriously dangerous to the existence
of humans and their cultures, if those exist in only one single
place in our solar system and universe. Our racial propensity to be
human, includes a propensity to be dangerous and destructive cranks
and fascists. See today's news out of Washington and other capitol
cities. So let's get our human culture *out there now*.

Go with it, Keith! Maybe you could work up and publish a *book* of
challenging Enigma files! Interesting to coming generations of young
people who might prefer cryptography to puzzles and games.

Titeotwawki -- Martha Adams [Tues 2018 Aug 21]

[Scott Dorsey]

Martha Adams <m...@mhada.info> wrote:
>This is a really interesting idea, not least because it illustrates
>technological / knowhow progress over time. Back in the late 1940's,
>people imagined if you just destroyed the files and machines, the
>special crypto knowledge that had developed thru WW2 was suppressed
>for all time. After all, those fantastic Bombe machines, who could
>imagine building one of those if they didn't know what their builders
>knew? Using *advanced vacuum tube technology* no less?

Actually... they were gas tubes. All thyratron logic. Ingenious as hell
although they were still just looking for certain distributions of letters
in the final output and spitting out a result for a human to check whenever
the test key seemed to produce a decrypted message that looked like German.

>Now here we are in today. I don't think you could excite much
>interest in producing Bombes and their manuals except as a giveaway
>to crypto freaks, who might see them as toys not for any serious
>work. Likewise the special knowledge, now to be found nearly
>anywhere among makers of crypto systems and resources. Yes, what
>"man can do," man can do again, and pretty soon, as tech develops.
>And whoever thinks that's an exaggeration wants to look at how the
>tech for nuclear bombs and for chemical warfare, has grown and got
>around today's world.

For many years the standard Unix encryption using crypt(1) used an
Enigma-like cypher algorithm. It was pretty hard to break in the seventies
with brute force methods but by the late eighties most folks had abandoned
it for more secure options.

>I think the central point in this that got missed, was that such
>tech resources and warfare, are seriously dangerous to the existence
>of humans and their cultures, if those exist in only one single
>place in our solar system and universe. Our racial propensity to be
>human, includes a propensity to be dangerous and destructive cranks
>and fascists. See today's news out of Washington and other capitol
>cities. So let's get our human culture *out there now*.

It's not the tech resources, it's the warfare that is the problem. Technology
keeps getting better and better, you cannot stop it. Everything is a dual-use
technology in the end. The day after someone discovered fire, someone else
burned down his neighbor's hut with it. Suppressing technology never works,
you have to suppress the warfare.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."

[pete...@gmail.com]

Its been done:

http://www.bytereef.org/m4_project.html

https://www.computerworld.com/article/2518647/computer-hardware/archive-project-will-digitize-wwii-enigma-messages.html

https://bletchleypark.org.uk/our-story/collections-and-research/archives-and-library

Enigma with known wiring of the rotors has about 76 bits of security against
pure brute force attack. The largest key I personally know to have been brute
forced is 64 bits.

Brute force is a 'dumb' attack - Turing etal did much smarter things.

Distributed.net has been trying to brute force a 72 bit key since 2002. They've
checked 5.3% of the keyspace to date: 250,093,833,063,336,247,296 keys, or about
half a trillion a second.


(of course, Keith won't see this)

pt

[Keith F. Lynch]

Scott Dorsey <klu...@panix.com> wrote:
> Actually... they were gas tubes. All thyratron logic.

Not from what I've read. They were electro-mechanical, consisting
mostly of rotors and plug boards, much like the Enigma machines
themselves, only much faster and with enormously more rotors and plug
boards. The movie The Imitation Game shows one in action, but in real
life the fastest rotors spun much faster, about as fast as they could
get something to spin without flying apart.

> Ingenious as hell although they were still just looking for certain
> distributions of letters in the final output and spitting out a
> result for a human to check whenever the test key seemed to produce
> a decrypted message that looked like German.

No. They were mostly looking for contradictions, to narrow the search
space. Secondly, they were looking for uneven distributions of
letters, as appear in plaintext or in a simple substitution cipher,
rather than the even distribution of letters usually seen in
ciphertext. Since the Enigmas had plugboards in addition to rotors,
the last step was usually to manually break the simple substitution
cipher that the plugboard alone implemented.

There was no automated German detector. Anyhow, the "plaintext"
didn't look much like standard German. It was mostly abbreviations,
there was no punctuation, no mixed case, and no accented letters,
and the letter X substituted for spaces. The messages were sent
in Morse code.

To this day, if you get an automated Morse code decoding app and tune
your shortwave to the ham bands, what you see won't look much like
English or any other natural language. Indeed, two hams successfully
communicating in Morse may never realize that they don't speak the
same language.

>> Now here we are in today. I don't think you could excite much
>> interest in producing Bombes and their manuals except as a giveaway
>> to crypto freaks, who might see them as toys not for any serious
>> work.

True. For one thing, computer simulations of mechanical bombes would
be enormously faster. For another, computer algorithms completely
unlike mechanical bombes would be faster yet. For a third, nobody
uses ciphers anything like Enigma anymore anyway.

>> Likewise the special knowledge, now to be found nearly anywhere
>> among makers of crypto systems and resources. Yes, what "man can
>> do," man can do again, and pretty soon, as tech develops. And
>> whoever thinks that's an exaggeration wants to look at how the
>> tech for nuclear bombs and for chemical warfare, has grown and
>> got around today's world.

Certainly. One of the most unrealistic things in Stapledon's _Last
and First Men_ was that after the inventors of nuclear power decide
to destroy their work and commit suicide, nobody reinvents it for
hundreds of thousands of years, despite lots of very smart people
trying very hard.

> For many years the standard Unix encryption using crypt(1) used an
> Enigma-like cypher algorithm.

I think it always used DES, which is nothing like Enigma. Enigma is
all nested rotating single-character substitution ciphers, very much
electromechanical flavored even though it could be implemented on
digital computers. DES is all bit swapping and flipping, very much
binary digital computer flavored, though it could be implemented
electro-mechanically.

> It's not the tech resources, it's the warfare that is the problem.
> Technology keeps getting better and better, you cannot stop it.
> Everything is a dual-use technology in the end. The day after
> someone discovered fire, someone else burned down his neighbor's hut
> with it. Suppressing technology never works, you have to suppress
> the warfare.

Agreed.

[Keith F. Lynch]

Martha Adams <m...@mhada.info> wrote:
> This is a really interesting idea, not least because it illustrates
> technological / knowhow progress over time. Back in the late
> 1940's, people imagined if you just destroyed the files and
> machines, the special crypto knowledge that had developed thru
> WW2 was suppressed for all time.

But they didn't destroy any of that stuff. To this day, there are
plenty of Enigma machines, and plenty of information about how they
worked and how they were cracked.

> Go with it, Keith! Maybe you could work up and publish a *book*
> of challenging Enigma files! Interesting to coming generations of
> young people who might prefer cryptography to puzzles and games.

I've since read that the last undecrypted Enigma messages were cracked
about a decade ago. I read that even today it's unfeasible to use
brute force. They used a "hill-climbing algorithm." I'm not sure
that's a distinction I'd make. Apparently by brute force they mean
to try every possible choice of rotors, initial rotor positions,
plugboard setup, etc., independent of all the others, which would be
a silly thing to do. I'd count "brute force" as any algorithm that
doesn't rely on any "kisses," "cribs," or other outside knowledge.

A "crib" is when they correctly guess that a particular long word or
short phrase is in a certain position. A "kiss" is when they already
have a copy of a plaintext and its corresponding ciphertext.

The allies cheated a lot -- after all, WWII was not a sporting event
-- by doing things such as planting mines in specific positions just
to learn what the codes for those positions were. Or by having a
German spy whom they caught and turned send a specific message,
knowing that it would be relayed verbatim after being encrypted.
Also, on several occasions they captured a U-boat complete with key
schedules and rotors.

It also helped that the encryption keys were in several parts, each
separate part of which was changed on a regular schedule. It would
have been much harder had they changed the whole key at once, and if
they had done so on an irregular schedule.

Of course nothing keeps people from creating new Enigma ciphertexts,
with an actual Enigma machine, a re-created one, or a computer
simulation of one, and challenging people to break it. But it's
a solved problem.

[Scott Dorsey]

Keith F. Lynch <k...@KeithLynch.net> wrote:
>Scott Dorsey <klu...@panix.com> wrote:
>> Actually... they were gas tubes. All thyratron logic.
>
>Not from what I've read. They were electro-mechanical, consisting
>mostly of rotors and plug boards, much like the Enigma machines
>themselves, only much faster and with enormously more rotors and plug
>boards. The movie The Imitation Game shows one in action, but in real
>life the fastest rotors spun much faster, about as fast as they could
>get something to spin without flying apart.

Go see them. You can see one, though not in operation, at the NSA museum
at Ft. Meade. It is well worth the trip, which is not far for you.
There are rotor matrices and gas tube logic and printing mechanisms.
It is all very ingenious.

>> For many years the standard Unix encryption using crypt(1) used an
>> Enigma-like cypher algorithm.
>
>I think it always used DES, which is nothing like Enigma. Enigma is
>all nested rotating single-character substitution ciphers, very much
>electromechanical flavored even though it could be implemented on
>digital computers. DES is all bit swapping and flipping, very much
>binary digital computer flavored, though it could be implemented
>electro-mechanically.

You're thinking of crypt(3) which was used for the password database and
which was a lot harder to break.

[David Goldfarb]

In article <plibk5$7jk$1...@reader1.panix.com>,

Keith F. Lynch <k...@KeithLynch.net> wrote:

>It also helped that the encryption keys were in several parts, each
>separate part of which was changed on a regular schedule. It would
>have been much harder had they changed the whole key at once, and if
>they had done so on an irregular schedule.

Also, the Germans could have done a double encryption -- run the
text through Engima, and then take the ciphertext and encrypt *that*.
It would have made sending and receiving messages a great deal more
laborious, but it would have protected all messages from known-plaintext
attacks; and from what I've read (and Katie and I just visited
Bletchley Park a few weeks ago) such attacks were hugely important.

--
David Goldfarb |"I want instant gratification -- and I'm
goldf...@gmail.com | willing to wait for it."
gold...@ocf.berkeley.edu | -- Patrick Nielsen Hayden

[Keith F. Lynch]

David Goldfarb <goldf...@gmail.com> wrote:
> Also, the Germans could have done a double encryption -- run the

> text through Enigma, and then take the ciphertext and encrypt *that*.

They often did. But the other encryptions were always weaker, and had
usually long since been broken by the Allies.

Not just encryptions, but codes. Common terms and phrases useful
for their line of business had single code words, mostly just for
succinctness. And for those words, they substituted other words,
in a classic code.

For instance instead of using latitudes and longitudes they divided
the Atlantic into two-letter grid squares, and subdivided each of
them into 9-by-9 numeric grids, each of which were then themselves
subdivided into 9-by-9 numeric grids. That was precise enough to
get two vessels within sight of each other. Later, they encoded the
two-letter grid squares with different pairs of two letters, and added
numbers to the subgrid numbers to disguise those too. But the British
were almost always a step ahead of them.

> It would have made sending and receiving messages a great deal
> more laborious, but it would have protected all messages from
> known-plaintext attacks; and from what I've read (and Katie and
> I just visited Bletchley Park a few weeks ago) such attacks were
> hugely important.

There were often "kisses," in which the exact same message was sent
in a known cipher or code and in Enigma. It didn't make much difference
to the difficulty of breaking the message whether what was encrypted in
Enigma was the plaintext of that message or the known cipher.

Also, one change they made intending to make the system both easier to
use and more secure, reflecting the encrypted message back through all
the rotors to double encrypt it, actually made it much less secure.
Mostly because it ensured that no letter could ever represent itself.
Sometimes a particularly lazy operator would send a test message
consisting of a single letter repeated hundreds of times. When that
happened it was obvious because that letter didn't appear at all in
the ciphertext, and that made key recovery almost trivial.