Spanish special characters making NewtonSoft bomb

[Joshua Gough]

Hello,

We just noticed something odd happening when one of our team members makes GitHub commits that contain:

ñ

We take the JSON that GitHub sends us, like this:

"commits": [

    {

      "id": "something",

      "distinct": true,

      "message": "Added status configuration to app.config",

      "timestamp": "2015-10-07T12:49:15-03:00",

      "url": "https://hidden",

      "author": {

        "name": "Santiago Aceñolaza",

        "username": "acenolaza"

      },

      "committer": {

        "name": "Santiago Aceñolaza",

        "username": "acenolaza"

      },

      "added": [

      ],

      "removed": [

      ],

      "modified": [

        "VersionOne.TeamSync.JiraConnector/Config/JiraSettings.cs",

        "VersionOne.TeamSync.Service/App.config"

      ]

    }

  ],

And then we attempt to persist into EventStore. It actually appears, on the surface to be working because we aren't getting errors on the initial write, but I have verified that we do not see the events inside the actual target stream, and then we do get this in the log:

[PID:04092:033 2015.10.07 16:13:08.743 ERROR JsonCodec           ] '[{"eventId":"25122d37-c84d-4707-ad53-19eed1f15f46","eventType":"GitHubCommitReceived","data":{"sha":"something","commit":{"author":{"name":"Santiago Aceñolaza","email":"em...@gmail.com","username":"acenolaza"},"committer":{"name":"Santiago Aceñolaza","email":"em...@gmail.com","date":"2015-10-07T12:49:15-03:00"},"message":"Added status configuration to app.config"},"html_url":"https://github.com/versionone/VersionOne.TeamSync/commit/42bc908f88cfbab3075b700aa4b0806db723104d","repository":{"id":34900311,"name":"VersionOne.TeamSync"},"branch":"S-53254_TeamSyncSyncsStatuses","originalMessage":{"id":"42bc908f88cfbab3075b700aa4b0806db723104d","distinct":true,"message":"Added status configuration to app.config","timestamp":"2015-10-07T12:49:15-03:00","url":"https://github.com/versionone/VersionOne.TeamSync/commit/something","author":{"name":"Santiago Aceñolaza","email":"em...@gmail.com","username":"acenolaza"},"committer":{"name":"Santiago Aceñolaza","email":"em...@gmail.com","username":"acenolaza"},"added":[],"removed":[],"modified":["VersionOne.TeamSync.JiraConnector/Config/JiraSettings.cs","VersionOne.TeamSync.Service/App.config"]}},"metadata":{"instanceId":"something","digestId":"something","inboxId":"something' is not a valid serialized EventStore.Core.Messages.HttpClientMessageDto+ClientEventDynamic[]

Newtonsoft.Json.JsonReaderException: Unterminated string. Expected delimiter: ". Path '[0].metadata.inboxId', line 1, position 1401.

   at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 554

   at Newtonsoft.Json.JsonTextReader.ParseString(Char quote) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 118

   at Newtonsoft.Json.JsonTextReader.ParseValue() in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 925

   at Newtonsoft.Json.JsonTextReader.ReadInternal() in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 382

   at Newtonsoft.Json.JsonTextReader.Read() in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonTextReader.cs:line 303

   at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Int32 initialDepth, Boolean writeChildren, Boolean writeDateConstructorAsDate) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonWriter.cs:line 557

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJObject(JsonReader reader) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 236

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 370

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 258

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 767

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 1794

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 388

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 258

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 1280

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 648

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 260

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\JsonSerializerInternalReader.cs:line 184

   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonSerializer.cs:line 701

   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonConvert.cs:line 833

   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) in c:\Temp\Release\Working\Newtonsoft.Json\Src\Newtonsoft.Json\JsonConvert.cs:line 792

   at EventStore.Transport.Http.Codecs.JsonCodec.From[T](String text) in c:\work\EventStore\src\EventStore.Transport.Http\Codecs\JsonCodec.cs:line 64

I don't see anything here that indicates any special exclusions: https://github.com/EventStore/EventStore/blob/release-v3.4.0/src/EventStore.Transport.Http/Codecs/JsonCodec.cs

Nor, anything in the NewtonSoft code itself, nor in their issue list at https://github.com/JamesNK/Newtonsoft.Json

Is this a bug?

Thanks,

Josh

[João Bragança]

Can you post what metadata you are sending?

--
You received this message because you are subscribed to the Google Groups "Event Store" group.
To unsubscribe from this group and stop receiving emails from it, send an email to event-store...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my regular computer

http://twitter.com/thefringeninja

http://www.thefringeninja.com/

[Greg Young]

I'm just guessing:

"inboxId":"something'

Single quote terminator? Can you send your actual post that you are
doing? We don't do anything on top of newtonsoft.

> --
> You received this message because you are subscribed to the Google Groups
> "Event Store" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to event-store...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Studying for the Turing test

[João Bragança]

ñ is in the ascii character set, weird that it would cause a problem

[Joshua Gough]

I obfuscated the meta data, but it's not actually that sensitive, it's just some guids:

"metadata":{"instanceId":"d80e4bf6-e7fe-420b-8abc-9cfad80f0428","digestId":"4fb047a9-e7fe-420b-8abc-fb7517a3cd0e","inboxId":"d80e4bf6-6f3f-44eb-9e51-2e16381fb06e' is not a valid serialized EventStore.Core.Messages.HttpClientMessageDto+ClientEventDynamic[]

It's almost like the number of characters is off because it is not recognizing the trailing } on the metadata. It's fine on all other messages, except when we toss the ñ in there. We tried it where we put the character inside the "message" property of the GitHub commit, as opposed to the author name, and got the same resulting error in the ES logs.

[Greg Young]

Any chance we can try to get a reproduction of this with curl?

[Joshua Gough]

Sorry about that, lost track of this. I'll dig it up and post here.

You received this message because you are subscribed to a topic in the Google Groups "Event Store" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/event-store/WKlP29SkzM0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to event-store...@googlegroups.com.