[erlang-questions] public_key & OpenSSL format weirdness
zxq9
regarding RSA key formats generated by OpenSSL.
If I generate a private key in PEM format, public_key:pem_entry_decode/1 will
not return an #'RSAPrivateKey'{} record, but if I generate the private key in
DER format public_key:der_decode('RSAPrivateKey', KeyFile) will return the
correct #'RSAPrivateKey'{}.
Conversely, if I generate the public key in DER format
public_key:der_decode('RSAPublicKey', PubFile) will give me an asn1 error, but
if I generate the public key in PEM format public_key:pem_entry_decode/1
returns the correct #'RSAPublicKey'{} record.
If I go one step further and decode the public key from a PEM file, then use
public_key:der_encode/2 and write the resulting binary to a file as DER, then
I can use the Erlang-generated DER of the public key with no problem.
The I'm using "OpenSSL 1.0.1f 6 Jan 2014" on Linux.
The following is a screen dump illustrating the problem, then another dump of
some code illustrating the workaround:
%%%%%%%%%%%%%%%%
ceverett@changa:~/Code/erlang/keytest$ cat keygen-der
#! /bin/bash
prefix=${1:?"Cannot proceed without a file prefix."}
keyfile="$prefix"".key.der"
pubfile="$prefix"".pub.der"
openssl genpkey \
-algorithm rsa \
-out $keyfile \
-outform DER \
-pkeyopt rsa_keygen_bits:8192
openssl rsa \
-inform DER \
-in $keyfile \
-outform DER \
-pubout \
-out $pubfile
ceverett@changa:~/Code/erlang/keytest$ cat keygen-pem
#! /bin/bash
prefix=${1:?"Cannot proceed without a file prefix."}
keyfile="$prefix"".key.pem"
pubfile="$prefix"".pub.pem"
openssl genpkey \
-algorithm rsa \
-out $keyfile \
-outform PEM \
-pkeyopt rsa_keygen_bits:8192
openssl rsa \
-inform PEM \
-in $keyfile \
-outform PEM \
-pubout \
-out $pubfile
ceverett@changa:~/Code/erlang/keytest$ ./keygen-der rsa1
.....................................................................................................................................................................................................................................................................................................................
++
.............................................................................................................................................................................................................................................................................
++
writing RSA key
ceverett@changa:~/Code/erlang/keytest$ ./keygen-pem rsa2
..........................................................................................................................................................................................................................................................................................................................................................................................................................
++
........................................................++
writing RSA key
ceverett@changa:~/Code/erlang/keytest$ erl
Erlang/OTP 17 [erts-6.3] [source] [64-bit] [smp:2:2] [async-threads:10]
[kernel-poll:false]
Eshell V6.3 (abort with ^G)
1> rr("/usr/lib/erlang/lib/public_key-0.22.1/include/public_key.hrl").
['AAControls','ACClearAttrs','AccessDescription',
'Algorithm','AlgorithmIdentifier',
'AlgorithmIdentifierPKCS-10','AlgorithmIdentifierPKCS-8',
'AlgorithmIdentifierPKCS5v2-0','AlgorithmIdentifierPKSC-7',
'AlgorithmNull','AnotherName','AttCertValidityPeriod',
'Attribute','AttributeCertificate',
'AttributeCertificateInfo','AttributePKCS-10',
'AttributePKCS-7','AttributeTypeAndValue',
'Attributes_SETOF',
'Attributes_SETOF_valuesWithContext_SETOF',
'AuthorityKeyIdentifier','BasicConstraints',
'BuiltInDomainDefinedAttribute','BuiltInStandardAttributes',
'Certificate','CertificateList','CertificationRequest',
'CertificationRequestInfo',
'CertificationRequestInfo_attributes_SETOF'|...]
2> {ok, DerKeyBin} = file:read_file("rsa1.key.der").
{ok,<<48,130,18,41,2,1,0,2,130,4,1,0,189,228,172,204,135,
175,97,84,205,254,58,197,47,41,196,...>>}
3> DerKey = public_key:der_decode('RSAPrivateKey', DerKeyBin).
#'RSAPrivateKey'{version = 'two-prime',
modulus = 8090848...,
publicExponent = 65537,
privateExponent = 7670114...,
prime1 = 9729823...,
prime2 = 8315513...,
exponent1 = 4754084...,
exponent2 = 6031370...,
coefficient = 4497486...,
otherPrimeInfos = asn1_NOVALUE}
4> {ok, DerPubBin} = file:read_file("rsa1.pub.der").
{ok,<<48,130,4,34,48,13,6,9,42,134,72,134,247,13,1,1,1,5,
0,3,130,4,15,0,48,130,4,...>>}
5> DerPub = public_key:der_decode('RSAPublicKey', DerPubBin).
** exception error: no match of right hand side value {error,{asn1,{wrong_tag,
{{expected,2},
{got,16,
{16,
[{6,<<42,134,72,134,247,13,1,1,1>>},{5,<<>>}]}}}}}}
in function public_key:der_decode/2 (public_key.erl, line 219)
6> {ok, PemKeyBin} = file:read_file("rsa2.key.pem").
{ok,<<"-----BEGIN PRIVATE
KEY-----\nMIISQwIBADANBgkqhkiG9w0BAQEFAASCEi0wghIpAgEAAoIEAQCs758ufZpw6gVn\nlcb3GDri0nWovuJ"...>>}
7> [PemKeyData] = public_key:pem_decode(PemKeyBin).
[{'PrivateKeyInfo',<<48,130,18,67,2,1,0,48,13,6,9,42,134,
72,134,247,13,1,1,1,5,0,4,130,18,45,
...>>,
not_encrypted}]
8> PemKey = public_key:pem_entry_decode(PemKeyData).
#'PrivateKeyInfo'{version = v1,
privateKeyAlgorithm = #'PrivateKeyInfo_privateKeyAlgorithm'{
algorithm = {1,2,840,113549,1,1,1},
parameters = <<5,0>>},
privateKey = [48,130,18,41,2,1,0,2,130,4,1,0,172,239,159,46,
125,154,112,234,5,103,149,198,247|...],
attributes = asn1_NOVALUE}
9> {ok, PemPubBin} = file:read_file("rsa2.pub.pem").
{ok,<<"-----BEGIN PUBLIC
KEY-----\nMIIEIjANBgkqhkiG9w0BAQEFAAOCBA8AMIIECgKCBAEArO+fLn2acOoFZ5XG9xg6\n4tJ1qL7iUiVxLzWM"...>>}
10> [PemPubData] = public_key:pem_decode(PemPubBin).
[#'SubjectPublicKeyInfo'{algorithm = <<48,130,4,34,48,13,
6,9,42,134,72,134,
247,13,1,1,1,5,0,3,
130,4,15,0,48,130,
...>>,
subjectPublicKey = not_encrypted}]
11> PemPub = public_key:pem_entry_decode(PemPubData).
#'RSAPublicKey'{modulus = 7368345...,
publicExponent = 65537}
%%%%%%%%%%%%%%%%
The PEM -> DER workaround for the public key:
%%%%%%%%%%%%%%%%
ceverett@changa:~/Code/erlang/keytest$ cat keygen-asym
#! /bin/bash
prefix=${1:?"Cannot proceed without a file prefix."}
keyfile="$prefix"".key.der"
pubfile="$prefix"".pub.pem"
openssl genpkey \
-algorithm rsa \
-out $keyfile \
-outform DER \
-pkeyopt rsa_keygen_bits:8192
openssl rsa \
-inform DER \
-in $keyfile \
-outform PEM \
-pubout \
-out $pubfile
ceverett@changa:~/Code/erlang/keytest$ ./keygen-asym rsa3
............++
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
................................................................................
++
writing RSA key
ceverett@changa:~/Code/erlang/keytest$ erl
Erlang/OTP 17 [erts-6.3] [source] [64-bit] [smp:2:2] [async-threads:10]
[kernel-poll:false]
Eshell V6.3 (abort with ^G)
1> {ok, PubPemBin} = file:read_file("rsa3.pub.pem").
{ok,<<"-----BEGIN PUBLIC
KEY-----\nMIIEIjANBgkqhkiG9w0BAQEFAAOCBA8AMIIECgKCBAEAyqeCmfJNxPynjp8RDUWU\nKaEyLIrwv1WLfkmy"...>>}
2> [PubPemData] = public_key:pem_decode(PubPemBin).
[{'SubjectPublicKeyInfo',<<48,130,4,34,48,13,6,9,42,134,
72,134,247,13,1,1,1,5,0,3,130,
4,15,0,48,130,...>>,
not_encrypted}]
3> Pub1 = public_key:pem_entry_decode(PubPemData).
{'RSAPublicKey',8634564...,65537}
4> PubDer = public_key:der_encode('RSAPublicKey', Pub1).
<<48,130,4,10,2,130,4,1,0,202,167,130,153,242,77,196,252,
167,142,159,17,13,69,148,41,161,50,44,138,...>>
5> file:write_file("rsa3.pub.der", PubDer).
ok
6> {ok, PubDerBin} = file:read_file("rsa3.pub.der").
{ok,<<48,130,4,10,2,130,4,1,0,202,167,130,153,242,77,196,
252,167,142,159,17,13,69,148,41,161,50,...>>}
7> Pub2 = public_key:der_decode('RSAPublicKey', PubDerBin).
{'RSAPublicKey',8634564...,65537}
8> Pub1 = Pub2.
{'RSAPublicKey',8634564...,65537}
9> {ok, KeyDerBin} = file:read_file("rsa3.key.der").
{ok,<<48,130,18,40,2,1,0,2,130,4,1,0,202,167,130,153,242,
77,196,252,167,142,159,17,13,69,148,...>>}
10> Key = public_key:der_decode('RSAPrivateKey', KeyDerBin).
{'RSAPrivateKey','two-prime',
8634564...,
65537,
5179789...,
9406113...,
9179736...,
8502631...,
7664604...,
4453412...,
asn1_NOVALUE}
11> TestMessage = <<"A test message.">>.
<<"A test message.">>
12> Sig = public_key:sign(TestMessage, sha512, Key).
<<129,61,138,16,14,168,75,66,38,125,157,33,233,114,150,
135,45,97,237,181,2,255,162,180,242,124,36,223,208,...>>
13> public_key:verify(TestMessage, sha512, Sig, Pub2).
true
14> public_key:verify(TestMessage, sha512, Sig, Pub1).
true
%%%%%%%%%%%%%%%%
What is going on here?
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
zxq9
> I have encountered a strange issue with the public_key and crypto modules
> regarding RSA key formats generated by OpenSSL.
>
> If I generate a private key in PEM format, public_key:pem_entry_decode/1
> will not return an #'RSAPrivateKey'{} record, but if I generate the private
> key in DER format public_key:der_decode('RSAPrivateKey', KeyFile) will
> return the correct #'RSAPrivateKey'{}.
>
> Conversely, if I generate the public key in DER format
> public_key:der_decode('RSAPublicKey', PubFile) will give me an asn1 error,
> but if I generate the public key in PEM format
> public_key:pem_entry_decode/1 returns the correct #'RSAPublicKey'{} record.
>
> If I go one step further and decode the public key from a PEM file, then use
> public_key:der_encode/2 and write the resulting binary to a file as DER,
> then I can use the Erlang-generated DER of the public key with no problem.
>
> The I'm using "OpenSSL 1.0.1f 6 Jan 2014" on Linux.
Comparing the OpenSSL generated DER and the Erlang generated DER, I found
this:
%%%%%%%%%%%%%%
ceverett@changa:~/Code/erlang/keytest$ cp rsa3.pub.der rsa3.pub.der.erlang
ceverett@changa:~/Code/erlang/keytest$ openssl rsa -inform DER -in
rsa3.key.der -outform DER -pubout -out rsa3.pub.der.openssl
writing RSA key
ceverett@changa:~/Code/erlang/keytest$ ls -l rsa3.pub.der.{erlang,openssl}
-rw-rw-r-- 1 ceverett ceverett 1038 2月 26 10:58 rsa3.pub.der.erlang
-rw-rw-r-- 1 ceverett ceverett 1062 2月 26 10:58 rsa3.pub.der.openssl
ceverett@changa:~/Code/erlang/keytest$ cmp -b rsa3.pub.der.erlang
rsa3.pub.der.openssl
rsa3.pub.der.erlang rsa3.pub.der.openssl 異なります: バイト 4、行 1 12 ^J 42 "
ceverett@changa:~/Code/erlang/keytest$ cmp -b --ignore-initial=0:24
rsa3.pub.der.erlang rsa3.pub.der.openssl
ceverett@changa:~/Code/erlang/keytest$ erl
Erlang/OTP 17 [erts-6.3] [source] [64-bit] [smp:2:2] [async-threads:10]
[kernel-poll:false]
Running $HOME/.erlang.
Eshell V6.3 (abort with ^G)
1> {ok, OpenSSLBin} = file:read_file("rsa3.pub.der.openssl").
{ok,<<48,130,4,34,48,13,6,9,42,134,72,134,247,13,1,1,1,5,
0,3,130,4,15,0,48,130,4,...>>}
2> {ok, ErlangBin} = file:read_file("rsa3.pub.der.erlang").
{ok,<<48,130,4,10,2,130,4,1,0,202,167,130,153,242,77,196,
252,167,142,159,17,13,69,148,41,161,50,...>>}
3> <<_:24/binary, ChoppedBin/binary>> = OpenSSLBin.
<<48,130,4,34,48,13,6,9,42,134,72,134,247,13,1,1,1,5,0,3,
130,4,15,0,48,130,4,10,2,...>>
4> ChoppedBin = ErlangBin.
<<48,130,4,10,2,130,4,1,0,202,167,130,153,242,77,196,252,
167,142,159,17,13,69,148,41,161,50,44,138,...>>
5>
%%%%%%%%%%%%%%
So it appears there are 24 extra bytes appended to the OpenSSL generated DER
that are confusing the asn1 import function, but only in the case of public
RSA keys, as private OpenSSL generated DER keys match what is expected.
zxq9
> So it appears there are 24 extra bytes appended to the OpenSSL generated DER
> that are confusing the asn1 import function, but only in the case of public
> RSA keys, as private OpenSSL generated DER keys match what is expected.
Apparently OpenSSL adds an asn1 header, either on its own or because PKCS#1
demands it (or PKCS#8? I haven't found a clear reference, but it should be one
of those), and Erlang, iOS and a few other environments that expect a public
RSA key in DER format do not expect this header.
The header is always
<<48,130,4,34,48,13,6,9,42,134,72,134,247,13,1,1,1,5,0,3,130,4,15,0>>
So I wonder if it wouldn't be better to either add a clause to
public_key:der_decode/2 to catch this situation:
der_decode(Asn1Type,
<<48,130,4,34,48,13,6,9,42,134,72,
134,247,13,1,1,1,5,0,3,130,4,15,0,
Der>>) -> der_decode(Asn1Type, Der);
or (more properly) change the ASN.1 definition that generates OTP-PUB-KEY.erl
in a way that makes 'dec_RSAPublicKey' derive from a CHOICE instead of a
SEQUENCE type (or something along those lines -- I'm not sure where the asn1
module definition is... ?).
In any case, this issue appears to be a persistent annoyance for folks using
various RSA utilities:
(http://blog.flirble.org/2011/01/05/rsa-public-key-openssl-ios/
http://blog.wingsofhermes.org/?p=42
http://blog.wingsofhermes.org/?p=75)
-Craig
zxq9
> On 2015年2月26日 木曜日 11:01:22 zxq9 wrote:
> > So it appears there are 24 extra bytes appended to the OpenSSL generated
> > DER that are confusing the asn1 import function, but only in the case of
> > public RSA keys, as private OpenSSL generated DER keys match what is
> > expected.
> Apparently OpenSSL adds an asn1 header, either on its own or because PKCS#1
> demands it (or PKCS#8? I haven't found a clear reference, but it should be
> one of those), and Erlang, iOS and a few other environments that expect a
> public RSA key in DER format do not expect this header.
Ah ha! So... perhaps the big problem is really a lack of documentation. It
also seems that the canonical PKCS#1 asn1 definition is a bit ambiguous. In
any case, there is an openssl tool that has the same effect as chopping off
the first 24 bytes if invoked that way:
$ openssl asn1parse -strparse 24 -inform DER -in rsa3.pub.der.openssl -out
rsa3.pub.der.asn1parse
0:d=0 hl=4 l=1034 cons: SEQUENCE
4:d=1 hl=4 l=1025 prim: INTEGER :CAA7829...
1033:d=1 hl=2 l= 3 prim: INTEGER :010001
$ cmp -b rsa3.pub.der.asn1parse rsa3.pub.der.erlang
$
# No differences now!
This stupid little issue was really bothersome to get to the bottom of (and
whatever the problem with using PEM formatted private keys is, it is probably
related, since PEM is a base64 encoded DER binary inside). If our tools are
going to be built on OpenSSL I think they should at least detect and comply
with whatever (silly, maybe even wrong) output it produces instead of failing
on the most likely form it is likely to encounter.
The asn1 defintion in lib/public_key-0.22.1/asn1/PKCS-1.asn1 should be changed
to allow a CHOICE type among the current canonical RSAPublicKey and an
'OpenSSL-RSAPublicKey' option, OpenSSL upstream should be "fixed" (if its even
wrong, which technically it probably isn't) and document this to make it less
of a pointless speedbump, or we might want to add an optional clause to
public_key.erl to make sure we can accept the actual output of OpenSSL (since
adding an OID header to RSA public keys appears to be what its always done).
Robert Wilkinson
> On 2015年2月26日 木曜日 12:18:48 zxq9 wrote:
> > On 2015年2月26日 木曜日 11:01:22 zxq9 wrote:
> > > So it appears there are 24 extra bytes appended to the OpenSSL generated
> the first 24 bytes if invoked that way:
Hello Craig
Please can you clarify? I think that the 24 bytes are added at the
beginning, yet your use of the word "appended" implies to me that
the bytes are added at the end?
If the bytes are at the beginning, then "prepended" would be more
properly used than "appended".
Thank you for taking the time to go through the subtleties and
nuances of OpenSSL, and PEM, BER, DER, ASN.1 et al. I know it is
tricky!
Bob
zxq9
> On Thu, Feb 26, 2015 at 12:46:38PM +0900, zxq9 wrote:
> > On 2015年2月26日 木曜日 12:18:48 zxq9 wrote:
> > > On 2015年2月26日 木曜日 11:01:22 zxq9 wrote:
> > > > So it appears there are 24 extra bytes appended to the OpenSSL
> > > > generated
> >
> > the first 24 bytes if invoked that way:
> Hello Craig
>
> Please can you clarify? I think that the 24 bytes are added at the
> beginning, yet your use of the word "appended" implies to me that
> the bytes are added at the end?
>
> If the bytes are at the beginning, then "prepended" would be more
> properly used than "appended".
>
> Thank you for taking the time to go through the subtleties and
> nuances of OpenSSL, and PEM, BER, DER, ASN.1 et al. I know it is
> tricky!
>
> Bob
Indeed, I mis-typed that. They are prepended, so skipping the first 24 bytes
of the file yields a remainder of the form most RSA public key asn1 DER
decoders expect.
This first 24 bytes of the OpenSSL DER output for RSA public keys appears to
be an extra OSI header -- which most non-OpenSSL decode functions (Erlang's,
Java, iOS keyring, etc.) do not expect and trips them up.
This was a pain to track down, not because it is actually that mysterious, but
because its not documented anywhere I could easily find -- if I didn't happen
to be familiar with how asn1 works I would never have figured this out. The
canonical asn1 definitions for PKCS#1 (or at least seem canonical, as every
project but OpenSSL seems to use the same ones) don't include this particular
OSI header. It might be optional per spec (haven't located that part yet --
its probably spelled out in the actual PKCS#1 spec) but in any case, it would
be nice if asn1 decoding tools took this into account. OpenSSL is a very
common, or perhaps the most common, source of RSA key pairs in open source
projects; having its output clash with what most DER key decoders expect is
pretty silly.
-Craig