Browser persistence via IndexDB? LocalStorage?

[wuzz...@gmail.com]

I’d like to make an Eve app that persists data in the most basic way, say, TodoMVC that survives refresh. What would my path be to flushing all records to IndexDB (or LocalStorage if necessary)?

Thank you!

[harmat...@gmail.com]

This feature request is tracked in #792.

[co...@kodowa.com]

There is a path toward this using the DSL in the pre-release v0.3 branch (refactor-runtime). This branch has a JavaScript/TypeScript DSL for Eve, so you can intersperse Eve and Typescript. Then you can combine the facilities to import and export EAVs to get data into and load data from localStorage.

I'm currently writing a guide to make this more clear, but the summary of the DSL is that you can write blocks with very similar rules to the Eve syntax. In addition to this, you can write "watchers" which as the name suggests, watches for changes and then allows you to act on them. So you can watch for changes to the database or a particular record, and then save them using localStorage.setItem or however you would like to persist the data.

The second part is inputEavs(), which is called on a program instance. Again, as the name suggests, this allows you to input raw Entity Attribute Value triples into your program. You would read these in from localStorage.getItem(), for example.

Here is a small program that displays the current second, and saves it to local storage. It also reads in all saved times, and displays them on the screen. So when you refresh the page or restart the Eve server, all the times previously seen by the program will be printed on the screen.

let persistedEavs = [];

for (var key in localStorage){
   let storedEav : any = localStorage.getItem(key);
   let parsedEav : any = JSON.parse(storedEav);
   persistedEavs.push(parsedEav);
}

function RunProgram(input: any) {
  let prog = new Program("test persistence");
  prog.attach("html");
  prog.attach("system");
  prog.block("Display saved timestamps", ({find, record}) => {
    let timestamp = find("timestamp");
    return [
      record("html/element", {tagname: "div", sort: "0", text: `Saved Time: ${timestamp.time}!`}),
    ];
  });
  prog.block("Save the current time", ({find, record}) => {
    let time = find("timer");
    return [
      record("html/element", "time-message", {tagname: "div", sort: "0", text: `The time is: ${time.seconds}`}),
      record("timestamp", {time: `${time.seconds}`}),
    ];
  });
  prog.watch("Export timestamp", ({find, lookup, record}) => {
    let timestamp = find("timestamp");
    let {attribute, value} = lookup(timestamp);
    return [
      timestamp.add(attribute, value)
    ];
  })
  .asDiffs((diff) => {
    for(let [e, a, v] of diff.adds) {
      saveEav(e,a,v);
    }
  });
  prog.test(0, [
    [0, "tag", "timer"],
    [0, "tag", "system/timer"],
    [0, "resolution", 2000],
  ]);
  if (input.length > 0) {
    prog.inputEavs(input);
  }
}

function saveEav(e: any, a: any, v: any) {
  localStorage.setItem(`${e}|${a}`, JSON.stringify([e,a,v]));
}

RunProgram(persistedEavs);

Again, I'm going to be posting a guide soon to make this clearer, but it's a good place to start at least.

Corey

[wuzz...@gmail.com]

Thank you, this is super-helpful! This should be sufficient to adapt it to IndexDB.

Uhm, this is super-off-topic but maybe I’ll be forgiven since it’s my thread—Eve is built with TypeScript (after Lua, Rust, &c.), do y’all love TypeScript? Is it better, “better” than PureScript?

[co...@kodowa.com]

I'm glad I could be helpful.

We keep coming back to TypeScript from other languages because it allows us to leverage the browser as a platform, which Rust, Lua, C, etc. make more difficult. We use TypeScript in particular because we like the type system, we have a guy (Josh) who's good with the particulars, and it's well supported. I don't think there's any specific reason we're not using e.g. Purescript, and I'm not too sure how it compares to TS, but TS has been convenient and good to work with, so we haven't had any reason to look beyond it.

Corey

[William Taysom]

As one who futzed with the Rust branch when you were trying it out, TypeScript strikes me as a much, much better choice.

[Josh Cole]

I suspect the answer to whether TypeScript is better than PureScript depends heavily on your relationship with Haskell. :) As to whether 1) TypeScript is a good language on it's own, or 2) better than the others we've tried for certain tasks, I can provide some opinions.

1) Is TypeScript a good language?

Short answer: Yeah, it's pretty good. Longer answer:

Microsoft knows their way around a compiler. The tooling is rock solid and well designed, so third party editors have no trouble providing a proper editing experience. Compile times are fairly impressive for the amount of verification TSC provides, and errors have always been fairly intelligible (if sometimes verbose). In all the time I used it I managed to trick the compiler into doing the wrong thing only once, and that was when it was still pretty new. The type system is intuitive, expressive, and pay as you go; which works well in a prototype-heavy environment like this. Finally, (and importantly for an open source project) anyone who can work with a JS stack can sight-read typescript pretty easily. Since switching from other more esoteric languages, we've seen a noticeable uptick of contribution from the community.

Of course, it's not perfect. You're still playing in JS land, with all the problems that entails. ES6 (which TS will happily transpile) lessens that blow, but you'll feel it nonetheless. Performance is still just JS performance, you're limited to JS data structures, equality semantics will still make you sad in situations that you can't statically guarantee, package management for the web is junk, etc. I also have some nitpicks about the generics system, but that's been improving of late and comes up so infrequently that it's hardly worth mentioning.

All in all, it depends on your use case, which brings us to:

2) Is it better than other languages we've tried for this?

More or less. The qualifier there is that different languages suit different tasks.

Eve has a pretty deep stack, from parser and compiler up to web components and other UX considerations. No one conventional language is really going to be best across that gamut. However, for initial prototyping the most important thing is reaching a proof of concept as quickly and obviously as possible. As mentioned above, TS excels at this. Once the new runtime has stabilized, we'll look into rewriting the executor (the most performance intensive portion of our codebase) in Rust and compiling to wASM. Rust is definitely a language worth your time if you have strict performance requirements, but it's a bit of a bear to get into, and it's compile times are not great (even with optimizing disabled). It's a much smoother process if you go into it with a clear picture of exactly what you're trying to build first. We also plan to pull the last mile UX into Eve itself along the way, but I expect we'll be keeping a middle layer of TypeScript for everything that doesn't quite fit on either extreme.

Lua's also a fine language, but without LuaJIT it's pretty slow, and with LuaJIT it's a bit unpredictable. I also stumbled into a couple LuaJIT bugs without trying very hard. That said, props to the LuaJIT team, as soon as I provided sufficient diagnostic info they got things patched up very quickly [1]. Just be prepared to get your hands dirty if you want to play in that sandbox.

We've talked a little about why we're not using Clojure/ClojureScript right now, but the gist of it is that while they're both great languages, most of the features that make them nice are too slow for us to use even outside our hot paths.

[1] https://github.com/LuaJIT/LuaJIT/issues/180

On Wednesday, March 22, 2017 at 9:07:32 PM UTC-7, wuzz...@gmail.com wrote:

[Brian Theado]

Corey,

On Wed, Mar 22, 2017 at 11:06 PM, <co...@kodowa.com> wrote:
[...]

> The second part is inputEavs(), which is called on a program instance.
> Again, as the name suggests, this allows you to input raw Entity Attribute
> Value triples into your program. You would read these in from
> localStorage.getItem(), for example.

I noticed in the tag-browser watcher that [e, a, v, 1] is being passed
to inputEAVs for adding and [e, a, v, -1] for removing. Is that an
official part of the API and can it be added to the documentation at
https://github.com/witheve/docs/blob/master/guides/dsl.md?

>
> Here is a small program that displays the current second, and saves it to
> local storage. It also reads in all saved times, and displays them on the
> screen. So when you refresh the page or restart the Eve server, all the
> times previously seen by the program will be printed on the screen.

[...]

> localStorage.setItem(`${e}|${a}`, JSON.stringify([e,a,v]));

In the case of your simple program, [e,a] will be unique. I was
thinking that probably won't always be true. Like when you call add
multiple times for the same entity/attribute? In that case will
calling setItem in that way cause data to be overwritten?

It would help me to have such EAV cardinality details documented on
that DSL page.

[co...@kodowa.com]

Brian,

You're right, that usage should be documented.

> I was thinking that probably won't always be true.

I think the example I've shown is probably not the best way to go about this, and a more robust solution should be used in its place. If you look at diff.adds, you could potentially see multiple attributes with the same name on an entity. A good example of this is the "tag" attribute, so my toy program will fail for any records with more than one tag.

Corey

[Brian Theado]

Corey,

On Mon, May 8, 2017 at 1:34 PM, <co...@kodowa.com> wrote:
>> I was thinking that probably won't always be true.
>
> I think the example I've shown is probably not the best way to go about
> this, and a more robust solution should be used in its place. If you look at
> diff.adds, you could potentially see multiple attributes with the same name
> on an entity. A good example of this is the "tag" attribute, so my toy
> program will fail for any records with more than one tag.

Thanks. One way I thought of to make it more general is to use setItem
to store an array of eav instead of a single eav.

With this approach follow these steps for diffs.add

1. If the e|a key doesn't exist, then just call setItem on the
stringified [[e,a,v]]
2. If the e|a key already exists
2a. Use getItem to read the string representation of the json array
2b. Parse the string into a json array
2c. Append the new [e,a,v] to the end of the array
2d. Stringify the new array and pass it to setItem

For diffs.remove

1. Use getItem to read the string representation of the json array
2. Parse the string into a json array
3. Find and remove the [e,a,v] from the array
4. Stringify the resulting array and pass it to setItem

Have you guys given thought to this and have better ideas on how you
would do it?

[co...@kodowa.com]

Brian,

I think your plan sounds good, at least for a JS client. As we get Eve networked, the database is going to become distributed, so the persistence story becomes a little more complicated. I'm not quite sure what that looks like, long term yet.

Corey

[Felix George]

It sounds like in a distributed setup, the pieces of the database are scattered all over and there will be far more complicated ways for how we reliably watch for records in the networked version of Eve.  That is the sound of my heart beating.  I hope I'm wrong on that.

[Josh Cole]

Yeah the work required gets a lot harder in the distributed context. That's something we should be able to insulate you from as a user though, since it's all at the implementation level. You write the blocks, we'll take care of the rest. Luckily, it's not an immediate concern. Performance is looking good enough in 0.4 that the scale of most projects shouldn't really require distribution in the first place.