Gremlin .NET Driver & Websockets with Signed Headers

[Taylor Riggan]

All,

I'm working on developing an example for connecting to Amazon Neptune using the Gremlin .NET driver and connecting using the Websocket endpoint. I'm also attempting to use SigV4 Signing [1] for authentication.  This last part is key (everything else works but this).

I'm running into some issues with the Authorization header that I'm attaching to the WebsocketConfiguration that I'm using.  I'm essentially getting an error back that the header is malformed.  SigV4 headers are not compliant with IETF standards, so there's a need to disable any sort of validation.  It appears that in .NET, you can disable header validation for an HTTP Request [2].  I can't seem to find the same for the WebsocketConfiguration headers.

Has anyone else come across this from .NET perspective?  Any ideas?

Here's a short code snippet showing effectively what I'm trying to do:

            var webSocketConfiguration = new Action<ClientWebSocketOptions>(options => {

                    options.SetRequestHeader("x-amz-date", "20190301T194322Z");

                    options.SetRequestHeader("Authorization","AWS4-HMAC-SHA256 Credential= AKIAXXXXXXXXXXXYSZA /20190301/us-east-2/neptune-db/aws4_request, SignedHeaders=host;x-amz-date,

                               Signature=b8155de54d9faab00000000000000000000000000a07e0d7dda49902e4d9202");

                    });

            

            var gremlinServer = new GremlinServer("neptune-abcdefgh12345678.elb.us-east-2.amazonaws.com", 8182);

            var gremlinClient = new GremlinClient(gremlinServer, webSocketConfiguration: webSocketConfiguration);

var remoteConnection = new DriverRemoteConnection(gremlinClient);

var g = Traversal().WithRemote(remoteConnection);

Here's the error message I'm receiving:

Unhandled Exception: System.Net.WebSockets.WebSocketException: Unable to connect to the remote server ---> System.FormatException: The format of value 'AWS4-HMAC-SHA256%20Credential=AKIAXXXXXXXXXXXYSZA/20190301/us-east-2/neptune-db/aws4_request,%20SignedHeaders=host;x-amz-date,%20Signature=b8155de54d9faab00000000000000000000000000a07e0d7dda49902e4d9202' is invalid.

   at System.Net.Http.Headers.HttpHeaderParser.ParseValue(String value, Object storeValue, Int32& index)

   at System.Net.Http.Headers.HttpHeaders.ParseAndAddValue(HeaderDescriptor descriptor, HeaderStoreItemInfo info, String value)

   at System.Net.Http.Headers.HttpHeaders.Add(HeaderDescriptor descriptor, String value)

   at System.Net.Http.Headers.HttpHeaders.Add(String name, String value)

   at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)

   --- End of inner exception stack trace ---

   at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)

   at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)

   at Gremlin.Net.Driver.WebSocketConnection.ConnectAsync(Uri uri)

   at Gremlin.Net.Driver.Connection.ConnectAsync()

   at Gremlin.Net.Driver.ConnectionPool.CreateNewConnectionAsync()

   at Gremlin.Net.Driver.ConnectionPool.PopulatePoolAsync()

   at Gremlin.Net.Process.Utils.WaitUnwrap(Task task)

   at Gremlin.Net.Driver.GremlinClient..ctor(GremlinServer gremlinServer, GraphSONReader graphSONReader, GraphSONWriter graphSONWriter, String mimeType, ConnectionPoolSettings connectionPoolSettings, Action`1 webSocketConfiguration)

   at Aws4RequestSigner.Program.Main(String[] args) in /Users/triggan/Documents/code/Aws4RequestSigner/Program.cs:line 180

 

[1] https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html

[2] https://docs.microsoft.com/en-us/dotnet/api/system.net.http.headers.httpheaders.tryaddwithoutvalidation?view=netframework-4.7.2#System_Net_Http_Headers_HttpHeaders_TryAddWithoutValidation_System_String_System_Collections_Generic_IEnumerable_System_String__

Thanks in advance!

Taylor

[Florian Hockmann]

Hi Taylor,

I didn't know about this signing mechanism AWS is using before, but it sounds to me more like something that needs to be supported in System.Net.WebSockets. If they support this somehow with a configuration option or something similar, then we can provide access to that from Gremlin.Net.

So, I would take this to the .NET team, but maybe someone else here has more knowledge in this area than me and can give you a better answer.

Does that mean that Neptune users cannot use Gremlin.Net at all right now? Or is this signature optional right now?

[Kelvin Lawrence]

Hi Florian, the signature is needed once the Neptune database has had IAM Authentication turned on. While this is technically optional, it is likely that in production deployments it will be common for this to be turned on. Today it is possible to perform the signing using the Java, Python and Javascript Gremlin clients as well (I believe) as one of the Go open source Gremlin clients. It would be nice to have the .Net client likewise enabled. There is more background available at [1].

[1] https://docs.aws.amazon.com/neptune/latest/userguide/iam-auth.html

Cheers

Kelvin

[Taylor Riggan]

Just wanted to update this and let you know what I've found.

Starting in .NET Core 2.1, the WebSocketHandler.Managed [1] class significantly changed how it was constructing the HTTP request for instantiating and upgrading a connection to a Websocket.  Prior to 2.1, this class contained a StringBuilder that constructed the headers and added them to the HTTP request.  At 2.1 and afterwards, this class now uses the methods within the HttpHeaders class to create a collection of headers to add to the HTTP request.  The HttpHeaders class [2] has a default add() method that requires all headers to be properly formatted - known headers have associated parsers for which their values must pass in order to be valid.  Authorization is a known header value and it expects a single Bearer value in return.  Ideally, we should have the option to allow certain headers outside of this.  the HttpHeaders class also includes a TryAddWithoutValidation() that could be used to pass these types of headers.

I'm looking for thoughts on how best to address this for the long term.

[1] https://github.com/dotnet/corefx/blob/master/src/System.Net.WebSockets.Client/src/System/Net/WebSockets/WebSocketHandle.Managed.cs

[2] https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/Headers/HttpHeaders.cs 

[Austin Malpede]

Hi Florian,

I've created an issue on the dotnet corefx github page here: https://github.com/dotnet/corefx/issues/35862

I request that everyone here responds to it for visibility.

Thanks,

Austin

[Austin Malpede]

Microsoft has heard our pleas! It looks like this will be fixed in .Net Core 3.0 https://github.com/dotnet/corefx/issues/35862

-Austin

[Taylor Riggan]

Just confirmed.  This is fixed starting in 3.0.100-preview4-010715.

[Austin Malpede]

Just got confirmation that the port has been approved to .Net Core 2.1.11 and 2.2.5. These should be released sometime in May.

Cheers!