json.HasOwnProperty for anyone coming from node.js

[maxpow...@gmail.com]

Note to the group:  This message is intended for future people who like me are coming to golang after spending a good deal of time out in nodejs land.

I've been a programmer for a lot of years, C/C++, Java, Node.js you name it I've probably put it to good use at some point.

Lately I've found that I like Go alot and I really want to keep liking Go, but having used node.js as my go to webdev language for the last couple of years has spoiled me in the way I interact with loosely structured data.

This is most obvious when I start dealing with JSON being passed back and forth across the wire.
I really do hate Javascript & callback hell and all the other reasons I decided this current project would not be built with node, but I love dealing with JSON in node because it's native and feels natural.

I saw that Go advertises strong support for JSON, and I'll admit that the language features a very good parser.
However if you need data from an arbitrary field, it's just not able to do it and messing with the idiomatic methods of doing it just looks ugly in my opinion.
 
I spent a few days struggling with this, because I guess most folks who use Go are used to dealing with predictable structured data so most of the JSON examples don't really go into any great depth of how to deal with data that doesn't fit into structures that can be built at compile time.

In my case actual data interaction is minimal, but when I need a field it may be buried 10 layers deep in a JSON object, and I don't even get to control whether or not that field is present. 
I just need the language to get out of my way and let me have the data at field x.y.z.a.n.w.t.f  or tell me something along the lines of "Sorry that field couldn't be found."

This is the one place where Javascript really shines, I can simply say something like

if(someObject.hasOwnProperty('some.field')){
    doSomethingWith(someObject)
}

And it just works!

As far as I can tell, go has no equivalent.

So I was faced with a few choices.
#1 Ditch Go and run back node.js (not going to happen even if I have to hard code every possible data structure, lives will literally depend on this code)
#2 Hard code structs for the expected types merely for the purpose of checking for field values, (that's a LOT of code and I am a very lazy person)
#3 Use one of the Javascript interpreters for Go (too much overhead just for this, and leaning on JS could lead to very bad things)
#4 Take a close look at my previous node projects and see what features were actually being used the most and try to replicate them in Go.

I opted for number 4. 

The most common things I do with server side javascript
#1 Marshall Strings into JSON
#2 Check JSON for presence of fields
#3 Take logical actions based on data contained in said fields
#4 Ship JSON out somewhere else

Actual data manipulation is very, very rare in my past projects.  Usually once data has been JSONized somewhere, the primary purpose of said data is to tell some controller somewhere to take some action.
Thus I did not include any way to actually manipulate JSON Objects.

The result is here, https://play.golang.org/p/cRgh7j2TW2

I've given you two functions. 
The first one is json.Parse which takes a string and gives you a map[string]interface{} which is the closest thing to an actual JSON object that you are going to find in Go.
The second function is json.HasOwnProperty which takes a field label and the json object created by Parse and will return to you either the value at said label or nil

It does differ from Javascript here in that it will return the actual value of a label rather than a boolean indicating whether it was present or not.
I did that because asking if a field is present and then asking it to retrieve that field are two steps that really should be just one, imho.

You can use the code like this...

First get a JSON string and parse it into a JSON object

const B = `{
    "allowedtypes" :{
        "medical": {
            "sources":{
                "myself": ["*"],
                "mylocation":["r"],
                "other": ["a"]
            }
        }
    }
}`

var objB := json.Parse(B)

Next query and assign the field to a variable so you can take action on it...

permissions := json.HasOwnProperty("allowedtypes.medical.sources.myself", objB)
    fmt.Printf("Permissions for allowedtypes.medical.sources.myself %+v\n", permissions)

That's it!
Thanks for taking the time to read this and I hope someone who stumbles on this later can put it to good use.

[Egon]

For arbitrary data there is also: https://godoc.org/github.com/bitly/go-simplejson

 + Egon

[Klaus Post]

Hi!

Good call by Egon - you should go for that.

Also it is trivial to implement what you need, see:

http://play.golang.org/p/5oeZxliMyn

Of course you would need to type assert the results you get, but that is the deal when dealing with loosely structured data.

/Klaus

[Egon]

On Thursday, 22 January 2015 13:31:16 UTC+2, Klaus Post wrote:

Hi!

Good call by Egon - you should go for that.

Actually, my recommended approach for that problem was http://play.golang.org/p/4SqbLxr_lg. But for dealing random arbitrary data is simplejson is better. For something inbetween, there are multiple approaches, i.e. use map[string]interface{} directly, or properly marshal into a nice type. Using a proper type simplifies code else-where while keeping the problematic part in a single place.

+ Egon

[maxpow...@gmail.com]

Yeah Egon has some great ideas and reading his code replies to my previous posts is where I noticed this particular opportunity.

I agree that there are a lot of approaches, to dealing with unstructured data.

My intent was more narrow and was intended to show how easy it is to reimplement something akin to hasOwnProperty from the JS world in Go.

This is not to say that either approach mentioned by Egon is in anyway deficient, just the particular use case of needing an object or value at some arbitrary depth in unstructured data does not appear to be addressed any where that I have been able to find.

I showed an example using permissions since that was the thing that was buggered up when I started, but consider a different use case for a moment.

When you store a document in Mongo you tend to store the whole document, these things do not lend themselves to relational structuring very easily and can get heavy if you're not careful.

Again I'm not addressing the stupidity of the idea of trying to store *.world+dog , but merely, what do you do when you are presented with a need to extract something particular from an unstructured document containing dog+world and at completely arbitrary depth?  You can say things like "just make an struct that addresses it" or "you shouldn't do it that way in the first place!".  But that's not helpful when what you have has been handed to you and your job is just to make it work.

Use case...
There are custom forms created using a custom tool that runs clientside, built on top of  angular.
Any form, can contain any field those fields can and do contain subforms to any arbitrary depth.

This allows for the creation of business forms that meet regulatory compliance by stringing them together from other forms that also meet compliance.  In otherwords the person creating the forms does not want to have to re-invent the wheel.  To their mind they are just adding new pages to existing forms.
  
These were persisted wholesale in node and now the "legacy" node app is being replaced, we need to ensure that anything that node may have returned previously is also returned by go.
 
In this particular case, the User has a custom reporting system that needs to extract data from field z of subform y of form x, which itself is embedded in form n (and n could be embedded in l etc). 

What you have as far as information about the thing you need to return is a document id generated by mongo during the last upsert, and a command coming from the browser that looks like
[query: {collection: 'forms', _id: 100, path:'n.x.y.z' },...]

The server neither knows, nor cares about the type of data stored there, it's job is to isolate requested field(s), extract the data from those fields and pass it back to the caller assuming they had permission to read it :).

This was handled in node with nothing more than(pseudocode follows)

document = db.fetchOne(query._id)
if(document.hasOwnProperty(query.path)){
   res.write(JSON.stringify(document[query.path]))
}else{
  res.send(404)
}

I can't fathom what that would need to look like in Go, using the recommended or idiomatic approach, but replacing that lovely little tidbit was a task, way up high on my todo list.
The replacement code now looks about the same
(again, psuedo code here)

collection.Find(bson.M{'_id': query.ID}).One(&document)
section := json.HasOwnProperty(query.path,&document)
if section !=nil {
    fmt.Fprintf(w, "%s",json.Marshall(&document))
}else{
   http.Error(404)
}

Obviously the "correct" answer here is to dedup and normalize the data , however the goal of the project is to unify disparate information systems and to do so while having the smallest impact possible to existing systems all the while maintaining current levels of data integrity.  Easiest path to maintaining data integrity is to leave legacy data laying where it sits while gradually phasing in replacment front ends and ensuring new data ends up stored in a more coherent fashion.  Eventually the old data ends up in long term storage anyways, but the retention period on some of these records is 10 years before they go to archives.
I'm in a position where I didn't make the data that I'm expected to deal with, but I have to ensure that every user of that data has at least the same quality of experience they do now.
On the other hand I have no desire to take 1,000 lines of node and convert it into 100k lines of Go just to cover every edge case.

My instincts which are informed by over a decade of coming in and cleaning up after 1st year college freshman, H1Bs who couldn't spell SQL without "My" in front of it and *shudder* PHP devs, tell me that things like I've described tend to be the norm after any "super pumped, agile, web 2.0, html11, kitchen sink" project reaches more than about 6 months of development.
 
I have a feeling that as node and other "web server" tech becomes supplanted by Go, things like this are going to come to light pretty frequently, and my goal with this posting was to leave a trace for anyone in the future bumping up against similar things.  Hope it's helpful to someone.

[Egon]

Also a possibility: https://play.golang.org/p/eSw9PMgQrF

collection.Find(bson.M{'_id': query.ID}).One(&document)

if section, ok := document.Get(query.path); ok {
    fmt.Fprintf(w, "%s",json.Marshal(&document))
}else{
   http.Error(404)
}

Obviously the "correct" answer here is to dedup and normalize the data , however the goal of the project is to unify disparate information systems and to do so while having the smallest impact possible to existing systems all the while maintaining current levels of data integrity.  Easiest path to maintaining data integrity is to leave legacy data laying where it sits while gradually phasing in replacment front ends and ensuring new data ends up stored in a more coherent fashion.  Eventually the old data ends up in long term storage anyways, but the retention period on some of these records is 10 years before they go to archives.
I'm in a position where I didn't make the data that I'm expected to deal with, but I have to ensure that every user of that data has at least the same quality of experience they do now.
On the other hand I have no desire to take 1,000 lines of node and convert it into 100k lines of Go just to cover every edge case.

In that situation your approach makes perfectly sense.

[maxpow...@gmail.com]

Ooh I really like that one Egon.  You never cease to amaze me.
One thing though, I would always treat words like class & object as though they were reserved words (for later maintenance, it can get confusing especially if like me you need to language hop a lot).
Instead of

type Object map[string]interface{}

It might be clearer in the future if it were

type JSONObject map[string]interface{}

I say this after having worked on a project where somebody decided to redefine boolean into 3 states, true, false & null/never set
Not sure if they were trying to create Job security by being the only one "in the know" or if they had some legit reason, but eventually they left and my team was tasked with fixing some oddities that had crept up.

Just imagine how much fun that caused for us at debug time :)

[Egon]

On Friday, 23 January 2015 02:36:56 UTC+2, maxpow...@gmail.com wrote:

Ooh I really like that one Egon.  You never cease to amaze me.
One thing though, I would always treat words like class & object as though they were reserved words (for later maintenance, it can get confusing especially if like me you need to language hop a lot).
Instead of

type Object map[string]interface{}

It might be clearer in the future if it were

type JSONObject map[string]interface{}

Name the package "js" then the fully qualified name will be "js.Object"

+ Egon

[C Banning]

http://godoc.org/github.com/clbanning/mxj#Map.NewMapJson and  http://godoc.org/github.com/clbanning/mxj#Map.ValuesForKey would have worked too - but if that's all you need the mxi package is probably overkill.