Livedb 0.4 with drivers and projections

[Joseph Gentle]

Hi all

I've been rewriting the guts of livedb over the last couple weeks, and
I've published livedb 0.4.

IMPORTANT TLDR: If you want to keep using redis, you need to change
how you're initializing livedb with 0.4



There's two important new features you should be aware of:

- Livedb no longer requires redis. Redis is still used as a pub-sub
message bus & single source of truth when you have multiple servers.
But a lot of the time you only have one server. I've factored out &
isolated the redis code behind a driver interface, and added a simple
in-process version of the same API.

If you aren't scaling across multiple servers, you can get rid of your
redis server and remove references to it and everything will keep
working.

If you are using redis, you need to instantiate livedb a little differently:
livedb.createClient({db:snapshotDb, driver:livedb.redisDriver(oplogDb,
redisClient, redisObserver), ...})
If your snapshots and oplog are stored in the same place, pass your
database into both redisDriver and livedb.createClient.

Sorry about the API churn - in another version or two I'll pull the
redis driver entirely out of livedb, and you'll have to change how
you're instantiating everything again.

- Projections! - Here at lever we're sending way too much data to our
clients. I've added a new (experimental) feature to livedb whereby you
can expose a fake collection which maps a few fields from a real
collection. Any operations (gets, queries, sets, etc) on the fake
collection work, but you only see a small portion of the data. You can
use this to drop server & db load dramatically and speed up page
times. I think this is the equivalent to SQL VIEWs. For now, this only
works on JSON documents. (I don't know what it would look like for
text documents).

Its worth noting that you still need to restrict users from making
queries. You can make clever use of queries to find out any data in
hidden fields. If you want to stop people from accessing some of your
data, use sharejs middleware just like before.

Use it by calling:
livedbclient.addProjection("users_small", "users", "json0",
{name:true, age:true});

Limitations:
- You can only whitelist fields (not blacklist them).
- The third parameter must be 'json0'.
- Projections can only limit / allow fields at the top level of the document
- If you're using racer/derby, you should enable {id:true} in the projection.


I've also dramatically increased the test coverage of livedb (I think
there's about 2x as many tests as there were before).

There's still more feature work to do in livedb before I move on to
other areas of my stack. If nothing major comes up, 0.5 should come
out soon.

As always, let me know if you run into any issues.
-Joseph

[Artur Zayats]

It's awesome Joseph. Thanks!!!

[Pavel Zhukov]

This is so great! Projections was the thing that we really lack. At last there is no need to manually create collection subsets to divide public/private data and/or to keep the amount of data transferred small enough for a client. And making redis an optional dependency is great for the development environment/stage of a project. Thank you!

[Artur Zayats]

Joseph, could you please show us a little derby-code projections example?

On Saturday, May 3, 2014 4:38:06 AM UTC+4, Joseph Gentle wrote:

[Artur Zayats]

I think I found the way to use the two feature with derby: 

// Create store without redis

var store = derby.createStore({db: liveDbMongo(mongoUrl + '?auto_reconnect', {safe: true})});

// Add projection

store.shareClient.backend.addProjection("users_small", "users", "json0", {id: true, a:true, b:true});

 

[Alexander Wenzowski]

added these usage instructions to derby-starter

https://github.com/wenzowski/derby-starter/blob/9c91fba86949d3fcd34e47d62a9e7dd598781305/lib/store.js#L9

[Сергей Дроганов]

Hello,

Why does addProjection hide id from output?

Here is the code:

# projections

store

.shareClient

.backend

.addProjection "lenta", "posts", "json0",

id : true

type : true

# query

q = model.query "lenta",

visible : true

type : "post"

q.fetch (err, data)->

return next err if err

model.set "_page.lenta", q.get()

page.render()

# view

{{ each #root._page.lenta }}

<p>

{{ Object.keys(this) }}

</p>

{{ / }}

# output

<p>type</p>

Where is the id? How to add id to an output? Why it's present when I query original collection and missing when I query projection?

суббота, 3 мая 2014 г., 4:38:06 UTC+4 пользователь Joseph Gentle написал:

[Сергей Дроганов]

That's must be some rendering issue, If I trace the query result in controller I can see is has id

суббота, 31 мая 2014 г., 23:17:40 UTC+4 пользователь Сергей Дроганов написал:

[Dennis Torres]

Are there any plans to allow nesting fields in projections?

On Friday, May 2, 2014 8:38:06 PM UTC-4, Joseph Gentle wrote:

[Nate Smith]

Would be great, but no one is working on it right now.

--
You received this message because you are subscribed to the Google Groups "Derby" group.
To unsubscribe from this group and stop receiving emails from it, send an email to derbyjs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Browne]

Hi,

I'm reviving this thread because I'm wondering about the status of this feature and whether there are any plans to add support for blacklisting fields.

My site is a database with information on musical pieces that requires purchasing a subscription to access the full database. For security reasons, it would be great if the only data that could be retrieved in bulk by the client was a few basic fields about each song - the same fields that appear in the table of search results (title, composer, etc.) - but not the full details for each song. It would also be great if it were still possible for the client to retrieve all the fields for a particular song, but only one song at a time, with some time limit if attempting to retrieve the data for many songs consecutively.

I realize that especially the latter part would probably require custom code in a server-only route and manually retrieving the data via AJAX, but please let me know if livedb currently has any current or planned features that might help with this.

This sort of use case isn't limited to my app so I think it would be great if Derby has more features to support it. Let me know if I can be of assistance in that effort.

Thanks,

Matt