Request for comments: ko.fromJS, ko.fromJSON, ko.toJS, ko.toJSON

1,389 views
Skip to first unread message

fla...@gmail.com

unread,
Oct 25, 2010, 3:33:01 PM10/25/10
to KnockoutJS
Hi folks

Since releasing v1.1.0, the #1 most requested feature has been an
easier way of getting data in and out of Knockout.

=== How it works in v1.1.0 ===
The approach I've used is receiving data in JSON form from the server
and manually looping over it, converting it into client-side models
with selected properties represented as observables. There are
benefits to my approach - for example you can add extra model
functions and dependentObservables as part of the mapping - but it
seems that most newcomers expect a more automated solution.

For example, given some incoming data,

function personModel(data) {
this.firstName = ko.observable(data.firstName);
this.job = {
title : ko.observable(data.job.title),
isVoluntary : ko.observable(data.job.isVoluntary)
};
this.pets = ko.observableArray(data.pets);

// Can add other dependentObservable properties or model functions
here
}

// Now map the actual data
var viewModel = {
people : ko.utils.arrayMay(incomingPeople, function(personData) {
return new personModel(personData);
})
}

=== Proposed new helpers ===
To automate some of this, I'm considering adding 4 new helper
methods:
* ko.fromJS - receives a plain JavaScript object graph (including
arrays, nested properties, etc) and makes certain parts of it
observable:
- all arrays are mapped to observableArrays
- leaf properties (that is, properties of type string, number, or
boolean) are mapped to observables
- all other structure in the object graph is mapped unchanged (not
made observable)
* ko.fromJSON - receives a JSON string, parses it, and then calls
ko.fromJS on the result
* ko.toJS - takes an arbitrary object graph (e.g., your view model)
and copies it to a new object graph in which all observables (and
observable arrays) are replaced by their current values. In other
words, strips out everything to do with KO and makes plain old
JavaScript objects/arrays. This might be more convenient for sending
to the server.
* ko.toJSON - takes an arbitrary object graph, copies it to a new
object graph stripping out the observables (by calling ko.toJS) and
then stringifies the result into JSON format. This will be easy to
send to the server.

The reason why ko.fromJS only makes *leaf* properties observable (and
not every point in the object graph) is that it's much less confusing
for the developer, and I think is more likely to be what you want. For
example, the following data structure:

{
firstName : "Steve",
job : {
title : "Spaceman",
isVoluntary : true
},
pets : ["Rod", "Jane", "Freddy"]
}

... gets mapped to become equivalent to:

{
firstName : ko.observable("Steve"),
job : {
title : ko.observable("Spaceman"),
isVoluntary : ko.observable(true)
},
pets : ko.observableArray(["Rod", "Jane", "Freddy"])
}

Notice that "job" itself isn't observable (it's unlikely that you'd
want it to be), but the leaf properties on "job" are. Similarly,
"Rod", "Jane" and "Freddy" aren't put inside observable wrappers (most
developers would find that unexpected) - they are just entries on an
observable array.

Now you can bind this to some DOM elements and templates and will be
able to update "firstName", "job.title", "job.isVoluntary", and the
entries in "pets", and the UI gets updated.

Then, when you want to send the data back to the server, you call
ko.toJS(myModel) or ko.toJSON(myModel) and you get back a purely non-
observable data structure suitable for transmission to the server,
without having to manually map it.

=== Possible drawbacks ===
The reason I have been a bit reluctant to implement helpers like this
is that the mapping rules are pretty subjective. In most cases you'll
want leaf properties to be observable, but in some case you might not,
or you might also want intermediate levels in the graph to be
observable. Also, sometimes you'll want to add dependentObservables or
other functions at certain points in the object graph.

If you have any custom requirements at all, then ko.fromJS/ko.fromJSON
isn't going to work for you - you'll have to write custom mapping
logic like before. That's fine, but I'm just slightly concerned that
some developers will think that ko.fromJS/ko.fromJSON is the only
possible way to handle data and will forego the ability to make nice
rich client-side view models with appropriate functions on them.

=== Please try it out and give me your feedback ===
I've implemented the helpers described above in a prototype release of
Knockout, version 1.1.1pre, which you can get from
http://github.com/SteveSanderson/knockout/tree/master/build/output/

I'd appreciate it if you would practically try using these helpers
with your current code to see if it actually meets your needs. That's
the only way to really know if this is the right way forwards or
whether it's actually a dead end. Do they reduce the amount of code
you have to write, or in the end do you always need to write custom
mapping logic anyway?

Final note: Don't start relying on these new helpers for your
production code just yet as they may be removed if they turn out to
confuse us more than they help us :)

Thanks!

fla...@gmail.com

unread,
Oct 25, 2010, 3:34:25 PM10/25/10
to KnockoutJS
Typo: when I wrote "ko.utils.arrayMay", I meant "ko.utils.arrayMap".
> Knockout, version 1.1.1pre, which you can get fromhttp://github.com/SteveSanderson/knockout/tree/master/build/output/

Ω Alisson

unread,
Oct 27, 2010, 2:11:45 PM10/27/10
to knock...@googlegroups.com
I like these new features!

MrM

unread,
Oct 31, 2010, 3:55:13 PM10/31/10
to KnockoutJS
The toJSON worked great for my purposes of persisting state. fromJSON
looks good but I have some custom code that might prevent me from
using it. Using the example on http://knockoutjs.com/ , how would you
hydrate the viewmodel using fromJSON supposing tickets wasn't hard
coded. Would you do something like

var viewModel = new viewModel();
var data = ko.fromJSON(jsonData);
viewModel.tickets = data;


In my case my model has nested models which fromJson appears to handle
well but I have observables similiar to chosenTicket (on http://knockoutjs.com/)
that i dont know how to handle.

I like it though, how soon will you be releasing this to the trunk?

Thanks
> Knockout, version 1.1.1pre, which you can get fromhttp://github.com/SteveSanderson/knockout/tree/master/build/output/

Steven Sanderson

unread,
Nov 1, 2010, 2:08:58 PM11/1/10
to knock...@googlegroups.com
Hi

I've been discussing this a lot with Roy Jacobs, and we're going to move ko.fromJS/ko.fromJSON into a separate plugin that can become more sophisticated to handle these sorts of cases. Roy's already implemented a callback mechanism at http://github.com/SteveSanderson/knockout.mapping that can deal with more complex mapping requirements, though there aren't any docs for this yet.

The plan now is:
 * toJS/toJSON will stay in the Knockout core, because they're pretty simple and unambiguous
 * if you have fairly predictable, programmatic mapping requirements, you'll be able to use fromJS/fromJSON in the knockout.mapping plugin to map things across, supplying your own mapping callbacks where required
 * for richer, more complex view models, the more direct way to handle incoming data will be just to directly construct your view model using plain JavaScript code (without the plugin) so you can set up whatever object graph, observables, dependentObservables, etc meet your needs (e.g., using ko.utils.arrayMap as in the original post in this thread)

Cheers
Steve

MrM

unread,
Nov 4, 2010, 10:03:42 AM11/4/10
to KnockoutJS
Thanks Steve, I look forward to the changes.

On Nov 1, 2:08 pm, Steven Sanderson <ste...@stevensanderson.com>
wrote:
> Hi
>
> I've been discussing this a lot with Roy Jacobs, and we're going to move
> ko.fromJS/ko.fromJSON into a separate plugin that can become more
> sophisticated to handle these sorts of cases. Roy's already implemented a
> callback mechanism athttp://github.com/SteveSanderson/knockout.mappingthat
> can deal with more complex mapping requirements, though there aren't any
> docs for this yet.
>
> The plan now is:
>  * toJS/toJSON will stay in the Knockout core, because they're pretty simple
> and unambiguous
>  * if you have fairly predictable, programmatic mapping requirements, you'll
> be able to use fromJS/fromJSON in the knockout.mapping plugin to map things
> across, supplying your own mapping callbacks where required
>  * for richer, more complex view models, the more direct way to handle
> incoming data will be just to directly construct your view model using plain
> JavaScript code (without the plugin) so you can set up whatever object
> graph, observables, dependentObservables, etc meet your needs (e.g., using
> ko.utils.arrayMap as in the original post in this thread)
>
> Cheers
> Steve
>
> On 31 October 2010 19:55, MrM <mr.matthew.sm...@gmail.com> wrote:
>
>
>
> > The toJSON worked great for my purposes of persisting state. fromJSON
> > looks good but I have some custom code that might prevent me from
> > using it. Using the example onhttp://knockoutjs.com/, how would you
> > > Thanks!- Hide quoted text -
>
> - Show quoted text -

Roy Jacobs

unread,
Nov 4, 2010, 11:32:07 AM11/4/10
to KnockoutJS
Hi all,

As Steve mentioned I'm currently on writing a mapping plugin for
Knockout that will allow you to take arbitrarily complex JS objects
and convert them into a viewmodel that can be updated as well. We're
currently using a version in-house and I'm currently tweaking the API
and fixing bugs as we encounter them. Also, documentation and examples
need to be written.

The main features are:
* Convert from arbitrary JS
* Update from arbitrary JS
* Automatically creates observables and observable arrays
* Allows you to specify a 'key' callback to indicate to the mapper
how to compare different items in an observable array (to
automatically add/remove complex types from your arrays)
* Allows you to specify a 'create' callback if you want to
construct specific parts of the viewmodel yourself
* Allows you to specify an 'arrayChanged' callback that will tell
you when something was added/removed from an array as a result of an
update

I expect to get a beta of the plugin out in a week or two!

With regards,
Roy

Victor S

unread,
Nov 14, 2010, 11:59:51 PM11/14/10
to KnockoutJS
Something I'd like to watch out for, especially when it comes to
helping newcomers like to grok KO, is API documentation. The website
doesn't mention (in any easily searchable/findable way)
ko.utils.arrayMap... At least, not in a way that new here, like me has
yet to come across it (maybe I forgot if it was mentioned in some
obscure example, once or twice).

Is there a document with all functions listed and explained? A very
good example of thorough API documentation is Node.js and
Backbone.js...

On topic: The examples you state above a great help, and are clearing
much confusion on my part.

Something that would interest me, and I think many newcomers, is some
thorough examples of how getting some JSON through jQuery AJAX and
then mapping it to a model and or View Model. I know most of the
examples on the website are aimed at highlighting the main features
and purpose of KO, and what I'm asking for may be perhaps a more
general issue of taking in JSON value through AJAX, but I think it
makes more sense to give such examples, rather than using neat hand
crafted JSON models, which are un-realistic of the usual problems that
a developer would deal with on a real world app... In other words, if
I may make a request for more pragmatic examples, as opposed to the
"academic" looking ones on the site... at least in some specially
designated area that aims to clarify the general workflow and how KO
fits into a real world app. dev. environment.

Sorry if this is a bit off topic generally speaking...

Also, on the theme of adding functionality: how do you feel about
Sammy.js? I was considering that in a complete JS dev stack, having
some way to route uri's or #uri's is very useful if not necessary.
Backbone.JS just introduced that exact type of functionality on their
latest 0.3.0 release, and I was wondering if this kind of thing would
be useful to KO, and what people's opinion on this matter would be.

Cheers,
- V
> Knockout, version 1.1.1pre, which you can get fromhttp://github.com/SteveSanderson/knockout/tree/master/build/output/

Steven Sanderson

unread,
Nov 15, 2010, 3:22:09 AM11/15/10
to knockoutjs
Hi Victor

Thanks very much for your feedback. You make a lot of good points, and I think it will be possible to enhance KO in some of the ways you've suggested.

[1] API documentation for utility methods
Yes, the documentation doesn't cover these in detail right now, and the reason is that I had thought of the utility methods as being just internal implementation helpers that aren't really central to the way you'd use Knockout. For example, instead of using ko.utils.arrayMap, you might use jQuery's $.map or underscore.js's array mapping function. However, lots of people have raised questions about this (like, are these functions actually part of the public API?) so I think you're right - we should add documentation to cover each of the ones that are exposed in the public API so there's no doubt about what they do and whether it's preferable to use KO's utilities or equivalent ones in jQuery or underscore.js or prototype.js or whatever. I'll work on writing this.

[2] End-to-end examples integrated into a full technology stack
This is another good point that has also been raised before. So far I've resisted tying KO to any particular technology stack (because if the examples used MySQL or SQL Server or Mongo, people may assume that's the one and only database you're supposed to use with KO, which wouldn't be true), but I appreciate how useful it would be to see some specific cases of using KO with certain popular technology stacks. I blogged one example of using KO with ASP.NET MVC at http://blog.stevensanderson.com/2010/07/12/editing-a-variable-length-list-knockout-style/, and have already prepared (but not yet published) a "complex dynamic forms" example that uses KO with Ruby on Rails. I'll try to publish further examples, but may need help from you or others in the community to show using KO with popular technology stacks that I'm not very knowledgeable about (node.js being a key example).

[3] Using the url #hash for back button support and routing to places in a dynamic UI
This is absolutely possible already, and only takes a few lines of code. There's some discussion and an example at http://groups.google.com/group/knockoutjs/browse_thread/thread/c262315002cec3b1/54d3e409ae1dfecd?lnk=gst&q=jquery+address#54d3e409ae1dfecd - is this what you're talking about? If this turns out to be popular functionality we could consider merging it into the core (possibly reproducing some of the functionality of the jQuery Address plugin to eliminate the external dependency), and adding an example of it to the main knockoutjs.com website.

Regards
Steve

Victor S

unread,
Nov 15, 2010, 9:21:31 AM11/15/10
to KnockoutJS
Thanks, Steve for considering the suggestions,

For No. 2: while those cases are useful, I do understand why you would
not want to get all the way to the server-side. What I was trying to
suggest, is having examples that go all the way to requesting json
data or other data and then munging that into something KO can use
seamlessly. So I think this will go a bit further than just the
examples with plain handwritten objects, but not as far as suggesting
some tie-in to a particular server site tech. Otherwise, once I have
something interesting enough with KO, I wouldn't mind writing up a
blog post or short tutorial on it as well...


On Nov 15, 3:22 am, Steven Sanderson <ste...@stevensanderson.com>
wrote:
> Hi Victor
>
> Thanks very much for your feedback. You make a lot of good points, and I
> think it will be possible to enhance KO in some of the ways you've
> suggested.
>
> [1] API documentation for utility methods
> Yes, the documentation doesn't cover these in detail right now, and the
> reason is that I had thought of the utility methods as being just internal
> implementation helpers that aren't really central to the way you'd use
> Knockout. For example, instead of using ko.utils.arrayMap, you might use
> jQuery's $.map or underscore.js's array mapping function. However, lots of
> people have raised questions about this (like, are these functions actually
> part of the public API?) so I think you're right - we should add
> documentation to cover each of the ones that are exposed in the public API
> so there's no doubt about what they do and whether it's preferable to use
> KO's utilities or equivalent ones in jQuery or underscore.js or prototype.js
> or whatever. I'll work on writing this.
>
> [2] End-to-end examples integrated into a full technology stack
> This is another good point that has also been raised before. So far I've
> resisted tying KO to any particular technology stack (because if the
> examples used MySQL or SQL Server or Mongo, people may assume that's the one
> and only database you're supposed to use with KO, which wouldn't be true),
> but I appreciate how useful it would be to see some specific cases of using
> KO with certain popular technology stacks. I blogged one example of using KO
> with ASP.NET MVC athttp://blog.stevensanderson.com/2010/07/12/editing-a-variable-length-...,
> and have already prepared (but not yet published) a "complex dynamic forms"
> example that uses KO with Ruby on Rails. I'll try to publish further
> examples, but may need help from you or others in the community to show
> using KO with popular technology stacks that I'm not very knowledgeable
> about (node.js being a key example).
>
> [3] Using the url #hash for back button support and routing to places in a
> dynamic UI
> This is absolutely possible already, and only takes a few lines of code.
> There's some discussion and an example athttp://groups.google.com/group/knockoutjs/browse_thread/thread/c26231...
> -
> is this what you're talking about? If this turns out to be popular
> functionality we could consider merging it into the core (possibly
> reproducing some of the functionality of the jQuery Address plugin to
> eliminate the external dependency), and adding an example of it to the main
> knockoutjs.com website.
>
> Regards
> Steve
>
> On 15 November 2010 04:59, Victor S <victor.s...@gmail.com> wrote:> Something I'd like to watch out for, especially when it comes to

XASD

unread,
Nov 15, 2010, 12:22:57 PM11/15/10
to KnockoutJS
Seems must have feature to me.
If I pass something to ko.applyBindings(some_obj),I want to make all
these thing observable-no exceptions(it work by default onway in any
case and even twoway for binded inputs,why not for arrays?).I don't
want to add ko.observableArray() ko.observable() for every single
property I have in HUGE hierarcy.
All I care is data binding instructions and my view model-that's all.I
can prepare VM and merge(IF I need it) subsequently,without any
problems.
By the way,how I can understand all this mess cause by legacy browsers
not supporting getters/setters from js. But how about browsers which
support such feature? I need to pay tax anyway?
I always dream about libraries which begin to see such a useful
feature(yes,exactly,databinding related). AFAIK jquery use native
JSON.stringify/decode,why knockoutjs does not give any kudos to modern
js implementations?

Best wishes.

funkatron

unread,
Nov 15, 2010, 12:59:01 PM11/15/10
to KnockoutJS
I'd be interested in providing an example of PHP <-> Knockout.js.
Using it currently in this way at my day job.

--
Ed Finkler
http://funkatron.com
@funkatron
AIM: funka7ron / ICQ: 3922133 / XMPP:funk...@gmail.com


On Nov 15, 3:22 am, Steven Sanderson <ste...@stevensanderson.com>
wrote:
> Hi Victor
>
> Thanks very much for your feedback. You make a lot of good points, and I
> think it will be possible to enhance KO in some of the ways you've
> suggested.
>
> [1] API documentation for utility methods
> Yes, the documentation doesn't cover these in detail right now, and the
> reason is that I had thought of the utility methods as being just internal
> implementation helpers that aren't really central to the way you'd use
> Knockout. For example, instead of using ko.utils.arrayMap, you might use
> jQuery's $.map or underscore.js's array mapping function. However, lots of
> people have raised questions about this (like, are these functions actually
> part of the public API?) so I think you're right - we should add
> documentation to cover each of the ones that are exposed in the public API
> so there's no doubt about what they do and whether it's preferable to use
> KO's utilities or equivalent ones in jQuery or underscore.js or prototype.js
> or whatever. I'll work on writing this.
>
> [2] End-to-end examples integrated into a full technology stack
> This is another good point that has also been raised before. So far I've
> resisted tying KO to any particular technology stack (because if the
> examples used MySQL or SQL Server or Mongo, people may assume that's the one
> and only database you're supposed to use with KO, which wouldn't be true),
> but I appreciate how useful it would be to see some specific cases of using
> KO with certain popular technology stacks. I blogged one example of using KO
> with ASP.NET MVC athttp://blog.stevensanderson.com/2010/07/12/editing-a-variable-length-...,
> and have already prepared (but not yet published) a "complex dynamic forms"
> example that uses KO with Ruby on Rails. I'll try to publish further
> examples, but may need help from you or others in the community to show
> using KO with popular technology stacks that I'm not very knowledgeable
> about (node.js being a key example).
>
> [3] Using the url #hash for back button support and routing to places in a
> dynamic UI
> This is absolutely possible already, and only takes a few lines of code.
> There's some discussion and an example athttp://groups.google.com/group/knockoutjs/browse_thread/thread/c26231...
> -
> is this what you're talking about? If this turns out to be popular
> functionality we could consider merging it into the core (possibly
> reproducing some of the functionality of the jQuery Address plugin to
> eliminate the external dependency), and adding an example of it to the main
> knockoutjs.com website.
>
> Regards
> Steve
>
> ...
>
> read more »

Steven Sanderson

unread,
Nov 16, 2010, 3:58:52 AM11/16/10
to knockoutjs
XASD, 

Roy Jacobs is working on a KO plugin that will do what what you want - map an object graph to another object graph in which all the leaf properties are observable. You can get the current version from https://github.com/SteveSanderson/knockout.mapping.

> All I care is data binding instructions and my view model-that's all

I'm not sure if you're saying this based on experience of just on an idea of how you'd like to work, but what I find in practice is that it's very useful to control which specific parts of an object graph are observable (or, the "observability boundaries" as I sometimes find it useful to call them) so that the right parts of the UI respond to specific changes. If you haven't yet been working with KO you might need to give it a go to fully appreciate this.

> By the way,how I can understand all this mess cause by legacy browsers not supporting getters/setters from js.
> But how about browsers which support such feature? I need to pay tax anyway

If you want your code to run on all mainstream browsers (and for KO, that's one of the design goals), then yes, you have to be restricted to functionality supported by those browsers. For KO to work correctly on IE, for instance, its API can't depend on JavaScript getters and setters.

AFAIK jquery use native JSON.stringify/decode,why knockoutjs does not give any kudos to modern js implementations

KO does use the native JSON methods where appropriate. However, some of the elegance of the declarative binding API comes from being able to reference templating variables that are in scope wherever your element is in the template, and to be able to reference variables in scope, it's necessary to use "eval" in certain places rather than JSON.decode. If we abandoned this, then we'd also have to abandon some of the simplicity and elegance of the templating syntax. 

Hope this helps clarify the design a little! Thanks,
Steve

Josh Beall

unread,
Nov 6, 2012, 1:43:05 PM11/6/12
to knock...@googlegroups.com, ste...@stevensanderson.com
On Monday, November 15, 2010 3:22:09 AM UTC-5, Steven Sanderson wrote:
Hi Victor

Thanks very much for your feedback. You make a lot of good points, and I think it will be possible to enhance KO in some of the ways you've suggested.

[1] API documentation for utility methods
Yes, the documentation doesn't cover these in detail right now, and the reason is that I had thought of the utility methods as being just internal implementation helpers that aren't really central to the way you'd use Knockout. For example, instead of using ko.utils.arrayMap, you might use jQuery's $.map or underscore.js's array mapping function. However, lots of people have raised questions about this (like, are these functions actually part of the public API?) so I think you're right - we should add documentation to cover each of the ones that are exposed in the public API so there's no doubt about what they do and whether it's preferable to use KO's utilities or equivalent ones in jQuery or underscore.js or prototype.js or whatever. I'll work on writing this.

 It's been two years since this post... how's this coming?  I'm eager to see a full set of documentation so I can know when I'm using and when I'm abusing the system. :-)  And I see I'm not alone--if you do a search for "API documentation" you'll find numerous requests for this.

Unfortunately, it's not something that requires familiarity with the codebase and the intended usage of each function.  The documentation author needs to know, "Was this supposed to be a part of the public API?  Is it safe to use in this case?"  Etc.

  -Josh
Reply all
Reply to author
Forward
0 new messages