Loading JSON with missing properties in mapping plugin

[alex.d...@gmail.com]

First some backstory. The question is at the end, if you want to jump there.

I'm loading a JSON array that represents records in a database from a REST endpoint. When a record's field has no value, it's not included as a property of that object in the JSON array.

Now, when I load that JSON string into the mapping plugin to create my view model, the mapping plugin crashes, citing "Uncaught Error: Unable to parse bindings." for the attribute that wasn't included.

Here's the table cell to which this property is bound.

<td class="cityField" data-bind="text: City"></td>

Here's the mock data I'm using:

var testData = [{Id:"a", FirstName:"Me", LastName:"LastName", City:"Chicago", State:"IL", Company:"MyCo"}, {Id:"b", FirstName:"You", LastName:"LastName"}];

self.leadsFromServer = ko.mapping.fromJS(testData, self.mapping, self.personCache);

The first record seems to load fine, but it chokes on the second one. I'm guessing that as it loads this data, it is trying to update the UI. In the table loop, it loops over each object in the array and grabs the property (City, in this case). Because the 'City' property isn't on that object, an exception is thrown. Does this sound right? It seems like Knockout should just fill it with an 'error' value and continue, instead of crashing everything. Am I mistaken?

My question: What's the normal way of dealing with this non-normalized data? Can Knockout handle incomplete JSON like this with a plugin? Or is this something the dev must handle before passing it to the mapping plugin?

[alex.d...@gmail.com]

I've found these two topics that are similar to my problem:
   https://groups.google.com/d/topic/knockoutjs/ZwXCIkgce3U/discussion
   http://stackoverflow.com/questions/5086003/knockoutjs-using-updatefromjs-replacing-values-when-it-should-be-adding

It seems that there should be a plugin to handle this problem. In fact, there seems to be one attempt at one:
   http://janhartigan.com/articles/an-extension-to-the-knockout-js-mapping-plugin

However, this plugin requires the data model to be hand-written, pre-defined, in the Javascript. This isn't the best option, as the page could be using any number of fields that are returned from the database. I'd like Knockout to be invisible in this aspect - if the REST resource returns fields a, b, c, I should be able to plainly reference them in my data table... I wonder if I can modify the above-linked extension to not require a data model as input, but rather create one that sums all seen model properties as we receive records from the REST request...

[jga...@gmail.com]

I'm no expert by any stretch, but it seems the problem is the state of the data going in; generally with any mapping function, you have no 'memory' of previous or future objects in the array; each object is treated 'as is' and processed by the mapping function in isolation. In effect, you need the knockout mapping function to pre-evaluate the array, and look for the high-water mark of available object properties, and then assign those are missing a null value or whatever, which doesn't normally lie in the purview of a mapping function.

If you adjust your data model on the fly that you pass to the mapping function, it's likely still going to choke when it next updates the view model and finds objects missing properties that are referenced in the view yet not present in older objects.

The brute force approach would be generate a data model with all needed properties, based upon a high water mark after looking at each object in the array; then process your raw data array through the new data model via a jquery .map so they all have the same properties, and finally run your normalised data array through a 3rd transformation with ko.mapping. Brutal, but it would work.

A more elegant approach would be to skip the 2nd step from the brute force method; you'll still need to examine your raw array to make a list of all required properties - i.e. all properties that are present in at least one object, and then feed it to ko.mapping as an include, i.e.

resultOfPreProcessFunction = ["Id", "FirstName", "LastName", "City", "State", "MyCo"]

var mapping = {

'include' : resultOfPreProcessFunction;

}

var viewModel = ko.mapping.fromJS(rawData, mapping);

I'm afraid I don't see a way of skipping the first step if what properties are in your AJAX response objects are unknown, and you don't want to hard-code a list of 'acceptable' properties that must be present (and ignore any others) in a data model - given any subsequent objects may or may not have the known properties, and may have new ones to boot means you have to normalise what gets passed to ko.mapping somehow.

[alex.d...@gmail.com]

Woah, you're telling me that I can use the options.include for that? The docs say it is used only for the toJS function. I checked the source for the mapping, and I don't see any references to include being used this way.

I manually add items to my array before passing it to fromJS, like this:

$.each(data, function(i, mockItem){

    ko.mapping.addMissingProps(mockItem, self.modelSchema);

});

Where this addMissingProps mapping extension is defined like this:

// Extension function to the mapping plugin.

// This function adds missing fields to a JS object, based on the specified dataModel as a minimal required set of properties.

(function () {

    ko.mapping.addMissingProps = function(jsObject, dataModel) {

        if (arguments.length < 2) throw new Error("When calling ko.mapping.addMissingProps, pass: the updated data and the data model.");

        if (!jsObject) throw new Error("The jsObject is undefined.");

        if (!dataModel) throw new Error("The dataModel is undefined.");

        for (var stdProp in dataModel) {

            if (!(stdProp in jsObject)) {

                jsObject[stdProp] = "";

            }

        }

        return jsObject;

    }

    

    ko.exportSymbol('ko.mapping.addMissingProps', ko.mapping.addMissingProps);

})();

My initial tests show that options.include does not have this functionality when using the fromJS function. What do you think?

[jga...@gmail.com]

No, you're right, it's only on toJS. Sorry for leading you up the garden path! In my defence, the example at http://knockoutjs.com/documentation/plugins-mapping.html uses fromJS, even though the text says otherwise. Thought it was available for both.

Including certain properties using “include”

var mapping = {

    'include': ["propertyToInclude", "alsoIncludeThis"]

}

var viewModel = ko.mapping.fromJS(data, mapping);

Will have to flag that!

Still, it did lead me to an alternative approach; generating the columns from the raw data model for the viewmodel, then you don't need to process the data you pass to ko.mapping at all.

http://jsfiddle.net/bcBUu/2/

The <thead> is just for looks, the key part is the nested foreach loop. I've used underscore for the map/reduce/unique list of properties available in your data model, but obviously you can do that however you like. It might also be worth looking at the 'repeat' binding instead of the nested foreach if you've a large dataset, as it's supposedly much quicker.

https://github.com/mbest/knockout-repeat

[jga...@gmail.com]

Updated the jsfiddle slightly so the ko.computed updates from the processed data - so that data added with additional properties also adds new columns to the view.

http://jsfiddle.net/bcBUu/3/

[alex.d...@gmail.com]

Ah, brilliant! This looks like exactly my goal! Some new concepts for me here, so I'll have to study this in a bit before integrating it. I'll let you know how it goes. Thanks for the tip!