MongoDB issue with multi-dimensional arrays

[Steve. Warren]

HI,

Having an issue with the MongoDB adapter and multi-dimensional arrays. Seems that "Null" fields are being inserted with the data. 

Example Data posted to end point VIA Chrome Postman plugin:

{
"app":"test",
"config":
  {
      "levels" :[
      [5   ,5  ,10 ,10     ,2      ,3   ,1     ,3      ,1      ,5          ,10     ,1],
      [5   ,10 ,8  ,3      ,1      ,5   ,1     ,5      ,2      ,5          ,5      ,1],
      [10  ,5  ,7  ,6      ,2      ,2   ,1     ,5      ,2      ,5          ,8      ,1],
      [15  ,5  ,5  ,10     ,1      ,5   ,1     ,5      ,2      ,5          ,7      ,1],
      [5   ,15 ,10 ,6      ,1      ,3   ,1     ,5      ,2      ,5          ,10     ,1]
      ],
      "zones": {
          "one": "XXXXXXX",
          "two": "XXXXXX",
          "three": "XXXXXX"
      },
      "frequency":120
  }
}

The result in Ghengis:

{ _id: ObjectId("5624396d4837a389fbc4e503"), app: "test", objectId: "be15218617", config: { zones: { three: "XXXXXX", one: "XXXXXXX", two: "XXXXXX" }, levels: [ null, [ null, 5, 5, 10, 10, 2, 3, 1, 3, 1, 5, 10, 1 ], [ null, 5, 10, 8, 3, 1, 5, 1, 5, 2, 5, 5, 1 ], [ null, 10, 5, 7, 6, 2, 2, 1, 5, 2, 5, 8, 1 ], [ null, 15, 5, 5, 10, 1, 5, 1, 5, 2, 5, 7, 1 ], [ null, 5, 15, 10, 6, 1, 3, 1, 5, 2, 5, 10, 1 ] ], frequency: 120 } }

And the server-side Lua code to replicate the problem:

local in_data = coronium.input()
local obj = coronium.asJson(in_data)
local answer = coronium.mongo:createObject( "dsc", in_data)
if not answer.error then
  coronium.error(answer.error)
end
coronium.output(coronium.answer(answer))

As you can see in MongoDB the levels field's value has null added to each of the arrays. I am feel this may be a problem with the serializer so I added the "obj = coronium.asJson(in_data)" line and logged it. the result was the same as the body posted using Postman so the problem is definitely occurring in the mongo module. 

Any help would be appreciated.

[develephant]

Hi,

I'll see if I can get it to work. Most often, because of the Lua to JSON juggling, this has to do with how the data is set up for the call. Have you built the data up as a Lua table? I am not familiar with Postman, but you can't post JSON directly, you need to use the Coronium client.

Also, which version of Coronium are you using, and where are you hosting?

Cheers.

[Steve. Warren]

That is interesting to now about the client having to be a Lua client. I've been using Postman so far to test the endpoints before setting up the client in Lua and haven't had an issue, but I also have only been using simple objects up until now. I will setup a test app real quick in Corona and see if the same issue exists. 

[Steve. Warren]

Just tested in Corona and same issue happens. So it doesn't appear related to the client. Sample test code:

local coronium = require( "mod_coronium" )

local globals = require( "globals" )

local obj = {

app="test",

config=

  {

      levels = {

      {5   ,5  ,10 ,10     ,2      ,3   ,1     ,3      ,1      ,5          ,10     ,1},

      {5   ,10 ,8  ,3      ,1      ,5   ,1     ,5      ,2      ,5          ,5      ,1},

      {10  ,5  ,7  ,6      ,2      ,2   ,1     ,5      ,2      ,5          ,8      ,1},

      {15  ,5  ,5  ,10     ,1      ,5   ,1     ,5      ,2      ,5          ,7      ,1},

      {5   ,15 ,10 ,6      ,1      ,3   ,1     ,5      ,2      ,5          ,10     ,1}

      },

      zones= {

          one= "XXXXXXX",

          two= "XXXXXX",

          three= "XXXXXX"

      },

      frequency=120

  }

}

local function _callback(e)

    print("callback")

    --dump(e)

end

coronium:init( { appId = globals.appid, apiKey = globals.apikey } )

coronium:run( "dsc_create", obj, _callback )

Also realized I hadn't answered your other questions "Also, which version of Coronium are you using, and where are you hosting?". Hosting version 1.9.2 on a Digital Ocean droplet. 

[Steve. Warren]

OK, more research and discovered that this is a known issue in resty-mongol as addressed here (https://github.com/bigplum/lua-resty-mongol/issues/29). It seems it has to do with BSON's array index starting at 0 while Lua starts at 1. This causes the 0 indexed element to be null. Going to code and test a quick fix for mod_mongo that works around this issue. Will post back once I figure it out.

[Jonathan Sharp]

Hi Steve, I'm having the same issue.  Were you able to figure it out?

Thanks,

Jonathan

[Steve. Warren]

Unfortunately, I have not been able to spend much time on this outside of what I already documented here. As a work around, I ended up changing my data structure such that I had an index on all embedded arrays.  

[develephant]

You could also try discarding the first value of the array:

if my_arr[1] == coronium.json.null then

  table.remove( my_arr, 1 )

end

Not tested, but should work.

Cheers.