NetMQ.Security vs. Router/Dealer
Christoph Neumann
Hi netmq-dev,
Doron Somech did not post about NetMQ.Security for some time, hopefully this group might provide some helpful advices.
(I also created an issue in NetMQ.Security github [1], but Doron might not monitor these issues any more.)
I neet to apply Router/Dealer ("Harmony Pattern") [2] in my peer-to-peer messaging project.
For NetMQ.Security, I am just doing the first steps, all experiments are still decoupled from my actual code.
Dorons post from 2013 [3] was helpful with first steps about NetMQ.Security.
I got the Dealer/Dealer example up and running, in a NUnit class.
I am using BouncyCastle to successfully generate certificates at test run-time and could share the exact code with you, if necessary.
Then, I simply tried to change the server-side socket in his example to type RouterSocket; his comment suggested, that this should be possbible.
However, I get an NetMQ.Security.NetMQSecurityException ("Wrong length for protocol version frame").
Details:
The client initiates the connection and is waiting in line:
NetMQMessage incomingMessage= socket.ReceiveMultipartMessage();
The server is calling secureChanel.ProcessMessage the first time via line:
while (!secureChannel.ProcessMessage(incomingMessage, outgoingMessages))
This line throws an exception:
NetMQ.Security.NetMQSecurityException
HResult=0x80131500
Message=Wrong length for protocol version frame
Source=NetMQ.Security
StackTrace:
at NetMQ.Security.V0_1.SecureChannel.ProcessMessage(NetMQMessage incomingMessage, IList`1 outgoingMesssages) in C:\...\NetMQ.Security\V0_1\SecureChannel.cs:line 103
I suspect, that the additional identity as part of the extended structure of NetMQMessage in case of Router/Dealer is the reason.
Kind regards
Christoph
[1] https://github.com/NetMQ/NetMQ.Security/issues/3
[2] https://netmq.readthedocs.io/en/latest/router-dealer/
[3] https://somdoron.com/2013/05/securing-netmq/
PS: I am using NetMQ 4.0.x in a hybrid .NET Framework and .NET Core environment.
Thus, I am using NetMQ.Security not from nutget, but from github, which has been migrated to .NET Standard 1.6.
Christoph Neumann
Christoph Neumann
Solved this. As I have guessed, on the Server-side with the RouterSocket, upon receive, another first frame is added, with the identity (of the dealer). The NetMQ.Security does not handle this on ist own, thus, one needs to pop the frame, i.e. ignore it from SecureChannel perspective.
I will provide my solution / NUnit example to the community; its .NET Standard 2.0 compatible, thus, it can be used in hybrid .NET Framework / .NET Core environments:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Diagnostics.CodeAnalysis;
using System.IO;
using System.Net;
using System.Net.NetworkInformation;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using NetMQ;
using NetMQ.Security.V0_1;
using NetMQ.Sockets;
using NUnit.Framework;
using Org.BouncyCastle.Asn1.X509;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Crypto.Generators;
using Org.BouncyCastle.Crypto.Prng;
using Org.BouncyCastle.Math;
using Org.BouncyCastle.Pkcs;
using Org.BouncyCastle.Security;
using Org.BouncyCastle.X509;
namespace TwoPeersNetMQ
{
[TestFixture]
[Category("CI")]
[ExcludeFromCodeCoverage]
[Timeout(120000)]
[Parallelizable(ParallelScope.Fixtures)]
public class NetMQSecurityRouterDealerTests
{
internal readonly ISerializerWrapper serializerWrapper = new SlimSerializerWrapper();
internal string routerHost = "tcp://127.0.0.1";
internal int routerPort = -1;
internal ManualResetEvent mre = new ManualResetEvent(false);
[Test]
public void NetMQSecurity_RouterDealer_SecureChannel()
{
Action actionServer = this.Server;
Action actionClient = this.Client;
var taskServer = Task.Run(() => actionServer.Invoke());
var taskClient = Task.Run(() => actionClient.Invoke());
taskServer.Wait();
taskClient.Wait();
}
protected internal void Client()
{
mre.WaitOne();
using (var dealerSocket = new DealerSocket())
{
dealerSocket.Connect($"{routerHost}:{routerPort}");
dealerSocket.Options.Identity = Encoding.ASCII.GetBytes("IDENTITY: Client in Router/Dealer");
// ===== Init SecureChannel =====
SecureChannel secureChannel
= new SecureChannel(ConnectionEnd.Client);
secureChannel.SetVerifyCertificate(c => true);
List<NetMQMessage> outgoingMessages = new List<NetMQMessage>();
// call the process message with null as the incoming message
// because the client is initiating the connection
secureChannel.ProcessMessage(null, outgoingMessages);
foreach (NetMQMessage message in outgoingMessages)
{
dealerSocket.SendMultipartMessage(message);
}
outgoingMessages.Clear();
// waiting for a message from the server
NetMQMessage incomingMessage = dealerSocket.ReceiveMultipartMessage();
// calling ProcessMessage until ProcessMessage return true and
// the SecureChannel is ready to encrypt and decrypt messages
while (!secureChannel.ProcessMessage(incomingMessage, outgoingMessages))
{
foreach (NetMQMessage message in outgoingMessages)
{
dealerSocket.SendMultipartMessage(message);
}
outgoingMessages.Clear();
incomingMessage = dealerSocket.ReceiveMultipartMessage();
}
foreach (NetMQMessage message in outgoingMessages)
{
dealerSocket.SendMultipartMessage(message);
}
outgoingMessages.Clear();
// ===== Send Message =====
// you can now use the secure channel to encrypt messages
NetMQMessage sendPlainMessage = new NetMQMessage();
byte[] messageBytesOutgoing;
try
{
messageBytesOutgoing = serializerWrapper.Serialize("Hello");
}
catch (MessagingSerializationException)
{
throw;
}
sendPlainMessage.Append(messageBytesOutgoing);
// encrypting the message and sending it over the socket
dealerSocket.SendMultipartMessage(secureChannel.EncryptApplicationMessage(sendPlainMessage));
// ===== Receive Message =====
NetMQMessage receivedCipherMessage = dealerSocket.ReceiveMultipartMessage();
// decrypting the message
NetMQMessage receivedPlainMessage = secureChannel.DecryptApplicationMessage(receivedCipherMessage);
string receivedPayload;
try
{
receivedPayload = serializerWrapper.Deserialize<string>(receivedPlainMessage.First.ToByteArray());
}
catch (Exception)
{
throw;
}
TestContext.Out.WriteLine($"Client has received: {receivedPayload}");
}
}
protected internal void Server()
{
// we are using dealer here, but we can use router as well, we just
// have to manager SecureChannel for each identity
using (var routerSocket = new RouterSocket())
{
routerSocket.Options.RouterMandatory = true;
routerSocket.Options.Linger = TimeSpan.Zero;
routerPort = routerSocket.BindRandomPort(routerHost);
mre.Set();
// waiting for message from client
NetMQMessage incomingMessage = routerSocket.ReceiveMultipartMessage();
NetMQFrame identityFrame = null;
if (routerSocket is RouterSocket)
{
// ignore identity in netMQMessage[0] ...
identityFrame = incomingMessage.Pop();
var identityString = Encoding.ASCII.GetString(identityFrame.ToByteArray());
TestContext.Out.WriteLine( $"Received message from {identityString}");
}
// ===== Init SecureChannel =====
using (SecureChannel secureChannel = new SecureChannel(ConnectionEnd.Server))
{
// For the server, we need to provide a X509Certificate2 with a private key
var cert = GenerateSecureChannelCert(
$"NetMQ-Peer" +
$"-{Process.GetCurrentProcess().Id}" +
$"-{DateTime.Now.Ticks}" +
$"-{routerSocket.GetType().Name}"
);
secureChannel.Certificate = cert;
List<NetMQMessage> outgoingMessages = new List<NetMQMessage>();
// calling ProcessMessage until ProcessMessage return true
// and the SecureChannel is ready to encrypt and decrypt messages
while (!secureChannel.ProcessMessage(incomingMessage, outgoingMessages))
{
foreach (NetMQMessage message in outgoingMessages)
{
if (identityFrame == null)
routerSocket.SendMultipartMessage(message);
else
{
// add identity as frist frame:
NetMQMessage outgoingMessage = new NetMQMessage(1 + message.FrameCount);
outgoingMessage.Append(identityFrame);
foreach (NetMQFrame frame in message)
outgoingMessage.Append(frame);
// send
routerSocket.SendMultipartMessage(outgoingMessage);
}
}
outgoingMessages.Clear();
incomingMessage = routerSocket.ReceiveMultipartMessage();
if (routerSocket is RouterSocket)
// ignore identity in netMQMessage[0] ...
incomingMessage.Pop();
}
foreach (NetMQMessage message in outgoingMessages)
{
if (identityFrame == null)
routerSocket.SendMultipartMessage(message);
else
{
// add identity as first frame:
NetMQMessage outgoingMessage = new NetMQMessage(1 + message.FrameCount);
outgoingMessage.Append(identityFrame);
foreach (NetMQFrame frame in message)
outgoingMessage.Append(frame);
// send
routerSocket.SendMultipartMessage(outgoingMessage);
}
}
outgoingMessages.Clear();
// ===== Receive Message =====
// this message is now encrypted
NetMQMessage receivedCipherMessage = routerSocket.ReceiveMultipartMessage();
if (routerSocket is RouterSocket)
// ignore identity in netMQMessage[0] ...
receivedCipherMessage.Pop();
// decrypting the message
NetMQMessage receivedPlainMessage = secureChannel.DecryptApplicationMessage(receivedCipherMessage);
string receivedPayload;
try
{
receivedPayload = serializerWrapper.Deserialize<string>(
receivedPlainMessage.First.ToByteArray()
);
}
catch (Exception)
{
throw;
}
TestContext.Out.WriteLine($"Server has received: {receivedPayload}");
// ===== Send Message =====
NetMQMessage sendPlainMessage = new NetMQMessage();
byte[] messageBytesOutgoing;
try
{
messageBytesOutgoing = serializerWrapper.Serialize("World");
}
catch (MessagingSerializationException)
{
throw;
}
sendPlainMessage.Append(messageBytesOutgoing);
// encrypting the message and sending it over the socket
var sendCipherMessage = secureChannel.EncryptApplicationMessage(sendPlainMessage);
if (identityFrame == null)
routerSocket.SendMultipartMessage(sendCipherMessage);
else
{
// add identity as frist frame:
NetMQMessage outgoingMessage = new NetMQMessage(1 + sendCipherMessage.FrameCount);
outgoingMessage.Append(identityFrame);
foreach (NetMQFrame frame in sendCipherMessage)
outgoingMessage.Append(frame);
// send
routerSocket.SendMultipartMessage(outgoingMessage);
}
}
}
}
private static X509Certificate2 GenerateCert(
string certCN,
string signerCN = null,
AsymmetricKeyParameter signerKey = null,
int bitStrength = 2048)
{
// Phase 1: Use Bouncy Castle to generate a Org.BouncyCastle.X509.X509Certificate
GenerateBouncyCert(certCN, signerCN, signerKey, bitStrength, out var newBouncyCert, out var privateKey);
// Phase 2: Transform it into X509Certificate2 with private key.
// Use a relatively absurd approach with Pkcs12Store; however, the Pkcs12Store approach
// is the ONLY approach that is known to work in .NET Standard 2.0 (due to several
// issues by Microsoft concerning X509Certificate2.PrivateKey) and, thus, the only known
// approach to work in hybrid .NET Framework / .NET Core environments.
FromBouncyCertToDotNetCert2WithPrivateKey(newBouncyCert, privateKey, out var newDotNetCert2);
return newDotNetCert2;
}
internal static void GenerateBouncyCert(string certCN, string signerCN, AsymmetricKeyParameter signerKey, int bitStrength, out Org.BouncyCastle.X509.X509Certificate newBouncyCert, out AsymmetricKeyParameter privateKey)
{
// Keypair Generator
var kpGenerator = new RsaKeyPairGenerator();
kpGenerator.Init(new KeyGenerationParameters(new SecureRandom(new CryptoApiRandomGenerator()), bitStrength));
var kp = kpGenerator.GenerateKeyPair();
// Certificate Generator
var cGenerator = new X509V3CertificateGenerator();
cGenerator.SetSerialNumber(BigInteger.ProbablePrime(120, new Random()));
cGenerator.SetSubjectDN(new X509Name("CN=" + certCN));
cGenerator.SetIssuerDN(new X509Name("CN=" + (signerCN ?? certCN)));
cGenerator.SetNotBefore(DateTime.UtcNow);
cGenerator.SetNotAfter(DateTime.MaxValue);
cGenerator.SetSignatureAlgorithm("SHA256withRSA");
cGenerator.SetPublicKey(kp.Public);
// Add DNS name as Subject Alternative Name:
var hostFQDN = Dns.GetHostName();
var domainName = IPGlobalProperties.GetIPGlobalProperties().DomainName;
if (!String.IsNullOrEmpty(domainName)
&& !hostFQDN.EndsWith(domainName, StringComparison.OrdinalIgnoreCase))
hostFQDN = hostFQDN + "." + domainName;
GeneralName dnsName = new GeneralName(GeneralName.DnsName, $"{hostFQDN}");
GeneralNames subjectAltName = new GeneralNames(dnsName);
cGenerator.AddExtension(X509Extensions.SubjectAlternativeName, false, subjectAltName);
// Generate:
newBouncyCert = cGenerator.Generate(signerKey ?? kp.Private);
privateKey = kp.Private;
}
internal static void FromBouncyCertToDotNetCert2WithPrivateKey(Org.BouncyCastle.X509.X509Certificate bouncyCastleCert, AsymmetricKeyParameter privateKey, out X509Certificate2 newDotNetCert2)
{
string alias = bouncyCastleCert.SubjectDN.ToString();
Pkcs12Store store = new Pkcs12StoreBuilder().Build();
X509CertificateEntry certEntry = new X509CertificateEntry(bouncyCastleCert);
store.SetCertificateEntry(alias, certEntry);
AsymmetricKeyEntry keyEntry = new AsymmetricKeyEntry(privateKey);
store.SetKeyEntry(alias, keyEntry, new X509CertificateEntry[] { certEntry });
byte[] certificateData;
string password = Path.GetRandomFileName().Replace(".", ""); // 12 chars long
using (MemoryStream memoryStream = new MemoryStream())
{
store.Save(memoryStream, password.ToCharArray(), new SecureRandom());
memoryStream.Flush();
certificateData = memoryStream.ToArray();
}
newDotNetCert2 = new X509Certificate2(certificateData, password, X509KeyStorageFlags.Exportable);
}
protected internal static X509Certificate2 GenerateSecureChannelCert(string certCN)
{
// Generate self-signed certificate with bit strength 2048:
var cert = GenerateCert(certCN, null, null, 2048);
// NetMQ.Security, in HandshakeLayer.cs, will access the private key
// via X509Certificate2.GetRSAPrivateKey(), thus, assert:
if (cert.GetRSAPrivateKey() == null)
throw new ApplicationException("Missing RSA private key in generated X509Certificate2");
return cert;
}
}
public interface ISerializerWrapper
{
T Deserialize<T>(byte[] messageBytes);
byte[] Serialize<T>(T messageObject);
}
// SlimSerializer => https://github.com/azist/azos => MIT license
//
// For hybrid .NET Framework / .NET Core (.NET Standard 2.0) environments
// you have to patch Azos and in general any known binary serializer:
//
// 1) detect the run-time type:
// and 2) translate the assembly name from "mscorlib" to "System.Private.CoreLib" and vice versa:
// https://programmingflow.com/2020/02/18/could-not-load-system-private-corelib.html
//
// For Azos this can be done in Azos/Serialization/Slim/TypeRegistry.cs and RefPool.cs.
//
public sealed class SlimSerializerWrapper : ISerializerWrapper
{
public T Deserialize<T>(byte[] messageBytes)
{
try
{
T messageObject;
using (MemoryStream stream = new MemoryStream(messageBytes))
{
messageObject = (T)new Azos.Serialization.Slim.SlimSerializer().Deserialize(stream);
}
return messageObject;
}
catch (Exception)
{
throw;
}
}
public byte[] Serialize<T>(T messageObject)
{
try
{
byte[] messageBytes;
using (MemoryStream stream = new MemoryStream())
{
new Azos.Serialization.Slim.SlimSerializer().Serialize(stream, messageObject);
messageBytes = stream.ToArray();
}
return messageBytes;
}
catch (Exception)
{
throw;
}
}
}
}