Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Fatal Error: Bad Record MAC

2,529 views
Skip to first unread message

Jmail Clist

unread,
Jun 8, 2011, 2:33:34 PM6/8/11
to
Hello,
 
I am new to the list and definitely lack knowledge regarding the inner workings of the openssl stack. I will attempt to post all relevant information in hopes of getting feedback on this issue.
 
Basically, I have an IBM Datatpower appliance that cannot complete a successful handshake with a F5 LTM (load balancer).  After the client and server hellos, i get a "Fatal Alert" Bad Record Mac". Can someone explain this error more clearly and what are the possible causes along with some tips on how to debug/troubleshoot this issue? I have also traces available if anyone wants them. Please refer to frame 7 below for the error.
 
Frame 5 (192 bytes on wire, 192 bytes captured)
Ethernet II, Src: Cisco_08:34:00 (00:1b:2b:08:34:00), Dst: Ibm_f1:c2:24 (00:14:5e:f1:c2:24)
Internet Protocol, Src: 10.97.127.7 (10.97.127.7), Dst: 10.97.85.73 (10.97.85.73)
Transmission Control Protocol, Src Port: https (443), Dst Port: 27608 (27608), Seq: 1, Ack: 106, Len: 126
Secure Socket Layer
    TLSv1 Record Layer: Handshake Protocol: Server Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 74
        Handshake Protocol: Server Hello
    TLSv1 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
        Content Type: Change Cipher Spec (20)
        Version: TLS 1.0 (0x0301)
        Length: 1
        Change Cipher Spec Message
    TLSv1 Record Layer: Handshake Protocol: Encrypted Handshake Message
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 36
        Handshake Protocol: Encrypted Handshake Message
No.     Time        Delta       Source                tcp win size Len   Total Bytes Destination           Protocol Info
      6 0.000699    0.000008    10.97.85.73           5888         66    647         10.97.127.7           TCP      27608 > https [ACK] Seq=106 Ack=127 Win=5888 Len=0 TSV=154345430 TSER=1789553040
Frame 6 (66 bytes on wire, 66 bytes captured)
Ethernet II, Src: Ibm_f1:c2:24 (00:14:5e:f1:c2:24), Dst: All-HSRP-routers_0a (00:00:0c:07:ac:0a)
Internet Protocol, Src: 10.97.85.73 (10.97.85.73), Dst: 10.97.127.7 (10.97.127.7)
Transmission Control Protocol, Src Port: 27608 (27608), Dst Port: https (443), Seq: 106, Ack: 127, Len: 0
No.     Time        Delta       Source                tcp win size Len   Total Bytes Destination           Protocol Info
      7 0.000771    0.000072    10.97.85.73           5888         73    720         10.97.127.7           TLSv1    Alert (Level: Fatal, Description: Bad Record MAC)
Frame 7 (73 bytes on wire, 73 bytes captured)
Ethernet II, Src: Ibm_f1:c2:24 (00:14:5e:f1:c2:24), Dst: All-HSRP-routers_0a (00:00:0c:07:ac:0a)
Internet Protocol, Src: 10.97.85.73 (10.97.85.73), Dst: 10.97.127.7 (10.97.127.7)
Transmission Control Protocol, Src Port: 27608 (27608), Dst Port: https (443), Seq: 106, Ack: 127, Len: 7
Secure Socket Layer
    TLSv1 Record Layer: Alert (Level: Fatal, Description: Bad Record MAC)
        Content Type: Alert (21)
        Version: TLS 1.0 (0x0301)
        Length: 2
        Alert Message
            Level: Fatal (2)
            Description: Bad Record MAC (20)
After this the communication closes cleanly - Fin-Ack-Fin-Ack.
 
Thank you,
 

Eduardo Navarro

unread,
Jun 8, 2011, 3:30:02 PM6/8/11
to
Well, textbook explanation of SSL is not short, but once the connection is established, each party will have a set keys composed of a MAC key (message authentication code) and an encryption key. Within the SSL record, the payload is encrypted, and the MAC is basically a hash of the MAC Key + data + sequence + nonce + etc (I don’t remember the exact list of parameters that are authenticated by the MAC off the top of my head).
Also, at the end of the handshake, there is a final exchange of the MAC of all of the Records sent before the connection was “settled”.
If any of the items of the SSL Record change the client will be able to detect that because the MAC will not match. First place I would look is at the firewall logs, or maybe any app (such as HIDS/NIDS) that might be doing something to the packet.
 
Hope this helps.
-Eduardo

Dave Thompson

unread,
Jun 9, 2011, 7:58:56 PM6/9/11
to
> From: owner-ope...@openssl.org On Behalf Of Jmail Clist
> Sent: Wednesday, 08 June, 2011 14:34

> Basically, I have an IBM Datatpower appliance that cannot complete
> a successful handshake with a F5 LTM (load balancer). After the client
> and server hellos, i get a "Fatal Alert" Bad Record Mac". Can someone
> explain this error more clearly and what are the possible causes along
> with some tips on how to debug/troubleshoot this issue? I have also traces

> available if anyone wants them. Please refer to frame 7 below for the
error.

(trimmed)
> Frame 5 (192 bytes on wire, 192 bytes captured) from server contains

> TLSv1 Record Layer: Handshake Protocol: Server Hello len 74

> TLSv1 Record Layer: Change Cipher Spec Protocol: Change Cipher
Spec

> TLSv1 Record Layer: Handshake Protocol: Encrypted msg len 36
> Frame 6 (66 bytes on wire, 66 bytes captured) TCP-level ACK
> Frame 7 (73 bytes on wire, 73 bytes captured) from client contains

> TLSv1 Record Layer: Alert (Level: Fatal, Description: Bad Record
MAC)

The only case server should send ServerHello and immediately
ChangeCipher like that is "resuming" a session, i.e., using the
same handshake result (masterkey) for this connection as a previous
one between same client and server. If this is wireshark (as it looks)
click the box-plus to left of "Handshake Protocol: Server Hello"
and see if you have SessionIDLength=32 and SessiondID=bunchofhex.
(I *think* the only ciphersuite that makes encrypted Finish 36 is
RSA_WITH_RC4_128_SHA, abbreviated just RC4-SHA to openssl; you might
confirm that's what the trace shows selected.) Resume should be
accepted by the server only if there was a previous connection
and the client (correctly) remembers at least the (a?) sessionID.

I think if there was a previous successful handshake, it is
less likely (though not impossible) that something is damaging
the data between the endpoints, as described by Eduardo Navarro.
Maybe either the client or server is misremembering or misapplying
the session so that encryption/MAC on the new connection fails.

Note that openssl uses the "bad MAC" alert code for most(!)
decryption failures as well, because an attacker could benefit
from knowing which error was detected. But neither of these should
occur if both ends are correct and the data isn't damaged between.

Was there in fact a previous full-handshake connection from same
client to same server? Can you configure or direct client to do
only initial connection(s) not resume, or similarly on server
(perhaps only for this client)? Do other clients work okay
to same server, and can you verify if they are using resume?
Do you know this client works to other server(s)? With resume?

Can you try openssl commandline s_client to that server, with
appropriate certs etc and -reconnect, or with -sess_out and then
-sess_in using a scratch file? (But not sending any data over
the connection if you get one, just Q+return or EOF.)

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org

0 new messages