Long running operations...

[shansen]

Hi

I think Ayende announced that queues was being removed a while back,
making http://www.ravendb.net/documentation/triggers/long-running
obsolete. Since we are using this in a couple of plugins, I would like
to know how we can implement long running operations moving forward.
Would that just be using regular triggers or?

Thanks,
Steffen

[Ayende Rahien]

What long running operations did you have in mind?

[shansen]

Hi

The two plugins I have in mind at the moment is for notifications of
end users. One of them contacts a SMTP server and sends an email. The
other contacts a SMS gateway and sends a text to mobile phones. This
is triggered by the end user creating a mail or SMS document.

In a few weeks I will be doing another one for onetime replication
from an external datasource (Could be any type of datasource, but
would normally be a HTTP endpoint of some kind, or SQL). The workflow
for this is that the end user creates a document, which then gets one
or more fields replicated automatically based on a reference provided
by the user.

Thanks,
Steffen

On Jul 28, 2:43 pm, Ayende Rahien <aye...@ayende.com> wrote:
> What long running operations did you have in mind?
>
> On Thu, Jul 28, 2011 at 4:11 PM, shansen <steffen%grundsoe.dk@
>
>
>
>
>
>
>
> gtempaccount.com> wrote:
> > Hi
>
> > I think Ayende announced that queues was being removed a while back,

> > makinghttp://www.ravendb.net/documentation/triggers/long-running

[Ayende Rahien]

You can do that using the documents themselves.

Use the metadata to indicate the send state, then query them periodically to get the items that you want to work on

[Matt Warren]

I've been using it as well to enable Faceted search, see
https://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven.Database/Indexing/FacetedTrigger.cs.

Basically I detect when an index has been updated (using a trigger)
and then post a msg to the background queue where the actual
(expensive) work of update the facet values takes place. How would I
get round this without using a background worker, or am I going about
it in the wrong way? I'm doing is this was so that the facet creation
doesn't force the index update to take longer that it would normally.

Also (while I'm on the subject) would you consider adding a
IndexUpdateComplete trigger that fired when a whole batch of docs have
been added to the index, not just a single doc?

On Jul 28, 2:43 pm, Ayende Rahien <aye...@ayende.com> wrote:

> What long running operations did you have in mind?
>
> On Thu, Jul 28, 2011 at 4:11 PM, shansen <steffen%grundsoe.dk@
>
> gtempaccount.com> wrote:
> > Hi
>
> > I think Ayende announced that queues was being removed a while back,

> > makinghttp://www.ravendb.net/documentation/triggers/long-running

[Ayende Rahien]

Hi,

I am going to take a deeper look at this, then I'll get back to you with better answers.

[Matt Warren]

Thanks,

Just a couple of things, the higher-level outline of what I'm doing is
here http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/readme.txt.

The codes not finished, I have to write some more unit tests and cover
a hole load of edge cases before I submit a pull-request, but the
basic idea is there.

Most of the work takes place in these files:

http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven.Database/Indexing/FacetedTrigger.cs
http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven.Database/Indexing/FacetedQueryHelper.cs
and the CalculateFacets() function here
http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven.Database/Indexing/Index.cs

On Jul 28, 6:13 pm, Ayende Rahien <aye...@ayende.com> wrote:
> Hi,
> I am going to take a deeper look at this, then I'll get back to you with
> better answers.
>

> On Thu, Jul 28, 2011 at 8:05 PM, Matt Warren <mattd...@gmail.com> wrote:
> > I've been using it as well to enable Faceted search, see
>

> >https://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven...
> > .

[Ayende Rahien]

Some notes on that.

* It is probably better to have separate docs for each index, rather than a single doc for all indexes.

* UpdateIndexFacets seems to be doing something fairly strange, you seems to be saving the default term count there, but I don't really understand why.

You seems to be doing a lot of work that isn't really required. .

At any rate, I did some work to get a new version going, you can see it here:

https://github.com/ayende/ravendb/tree/facets

Here are the documents data:

[

{

"Name": "Android",

"Price": "069.9",

"Manufacterer": "Goggle",

"@metadata": {

"Raven-Entity-Name": "Products",

"@id": "products/3",

"Last-Modified": "2011-07-31T12:51:47.4740000+03:00",

"@etag": "00000000-0000-0900-0000-000000000003",

"Non-Authoritive-Information": false

}

},

{

"Name": "iPad",

"Price": "299.9",

"Manufacterer": "Apple",

"@metadata": {

"Raven-Entity-Name": "Products",

"@id": "products/2",

"Last-Modified": "2011-07-31T12:51:27.9390000+03:00",

"@etag": "00000000-0000-0900-0000-000000000002",

"Non-Authoritive-Information": false

}

},

{

"Name": "iPod",

"Price": "099.9",

"Manufacterer": "Apple",

"@metadata": {

"Raven-Entity-Name": "Products",

"@id": "products/1",

"Last-Modified": "2011-07-31T12:51:15.5390000+03:00",

"@etag": "00000000-0000-0900-0000-000000000001",

"Non-Authoritive-Information": false

}

}

]

Here is the index definition:

{

"Index": {

"Name": "Products",

"Map": "from p in docs.Products select new { p.Manufacterer, p.Price }",

"Reduce": null,

"TransformResults": null,

"IsMapReduce": false,

"IsCompiled": false,

"IsTemp": false,

"Stores": {

"Manufacterer": "Yes",

"Price": "Yes"

},

"Indexes": {

"Manufacterer": "Default",

"Price": "Default"

},

"Fields": [

"__document_id",

"Manufacterer",

"Price"

],

"Type": "Map"

}

}

I issued the following query:

http://localhost:8080/databases/Tiger/facets/Products?Facet.Manufacterer.Mode=Default&Facet.Price.Mode=Ranges&Facet.Price.Range=[NULL TO 100]&Facet.Price.Range=[100 TO 200]&Facet.Price.Range=[200 TO NULL]

Here are the actual parameters in a way that we can read it easily:

Facet.Manufacterer.Mode=Default

Facet.Price.Mode=Ranges

Facet.Price.Range=[NULL TO 100]

Facet.Price.Range=[100 TO 200]

Facet.Price.Range=[200 TO NULL]

The result of this query si:

{

"Manufacterer": [

{

"Range": "apple",

"Count": 2

},

{

"Range": "goggle",

"Count": 1

}

],

"Price": [

{

"Range": "[NULL TO 100]",

"Count": 2

},

{

"Range": "[200 TO NULL]",

"Count": 1

}

]

}

We can also add facets to a query, like this:

http://localhost:8080/databases/Tiger/facets/Products?Facet.Manufacterer.Mode=Default&Facet.Price.Mode=Ranges&Facet.Price.Range=[NULL TO 100]&Facet.Price.Range=[100 TO 200]&Facet.Price.Range=[200 TO NULL]&query=Manufacterer:apple

Which then gives us:

{

"Manufacterer": [

{

"Range": "apple",

"Count": 2

}

],

"Price": [

{

"Range": "[NULL TO 100]",

"Count": 1

},

{

"Range": "[200 TO NULL]",

"Count": 1

}

]

}

The code isn't done, and I would love for you to take a look at that and see how we can build client API for this, but it seems like it would be much easier approach.

[Ryan Heath]

Ayende,

This is really nice! Perhaps you may want to store the facet
definitions (or group of facet defs) on the server
and return them when asked for?

// Ryan

[Ayende Rahien]

Ryan,

Why would I want to do that? Using my approach, Facets are totally dynamic, so there is no need to store them on the server.

[Ryan Heath]

I can imagine that one would want a lot of facets all having their own
ranges and definitions.
It could be a service to store those definitions on the server opposed
to send it with each and every request.
Also by storing it on the server, we can skip the parsing and probably
win some ms there.

If we agree on this, we could group facets and invoke them by a
certain command. something like this:

define the group:
http://localhost:8080/databases/Tiger/facets-define/Products?FacetGroup=MyFacetName&Facet.Manufacterer.Mode=Default&Facet.Price.Mode=Ranges&Facet.Price.Range=[NULL

TO 100]&Facet.Price.Range=[100 TO 200]&Facet.Price.Range=[200 TO NULL]

invoke:
http://localhost:8080/databases/Tiger/facets/Products?FacetGroup=MyFacetName

which would return your results. And, of course, you should also
support the adhoc approach.

How about that?

// Ryan

[Ayende Rahien]

I don't think that this is valuable, but we can probably just save this to a document, and then read from the document directly 

[Matt Warren]

Thanks for taking time to look at my code, you're right I probably
over-engineered it, but at least I learnt a bit more about the Raven
internals :-) I started off using the BitSet method and it kinda grew
from there, but your method of adding each facet to the query as an
"AND" clause is much more elegant and probably more efficient.

> http://localhost:8080/databases/Tiger/facets/Products?Facet.Manufacte...[NULL

> TO 100]&Facet.Price.Range=[100 TO 200]&Facet.Price.Range=[200 TO NULL]
>
> Here are the actual parameters in a way that we can read it easily:
>
> Facet.Manufacterer.Mode=Default
> Facet.Price.Mode=Ranges
> Facet.Price.Range=[NULL TO 100]
> Facet.Price.Range=[100 TO 200]
> Facet.Price.Range=[200 TO NULL]
>
> The result of this query si:
>
> {
> "Manufacterer": [
>  {
> "Range": "apple",
> "Count": 2
>  },
> {
> "Range": "goggle",
>  "Count": 1}
>
> ],
>  "Price": [
> {
> "Range": "[NULL TO 100]",
>  "Count": 2},
>
> {
>  "Range": "[200 TO NULL]",
> "Count": 1
>
> }
>  ]
> }
>
> We can also add facets to a query, like this:
>

> http://localhost:8080/databases/Tiger/facets/Products?Facet.Manufacte...[NULLTO

> 100]&Facet.Price.Range=[100 TO 200]&Facet.Price.Range=[200 TO
> NULL]&query=Manufacterer:apple
>
> Which then gives us:
>
> {
> "Manufacterer": [
> {
>  "Range": "apple",
> "Count": 2}
>
>  ],
> "Price": [
> {
>  "Range": "[NULL TO 100]",
> "Count": 1},
>
>  {
> "Range": "[200 TO NULL]",
> "Count": 1
>  }
> ]
>
> }
>
> The code isn't done, and I would love for you to take a look at that and see
> how we can build client API for this, but it seems like it would be much
> easier approach.
>
>
>
>
>
>
>
> On Thu, Jul 28, 2011 at 8:28 PM, Matt Warren <mattd...@gmail.com> wrote:
> > Thanks,
>
> > Just a couple of things, the higher-level outline of what I'm doing is
> > here

> >http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/readme....

>
> > The codes not finished, I have to write some more unit tests and cover
> > a hole load of edge cases before I submit a pull-request, but the
> > basic idea is there.
>
> > Most of the work takes place in these files:
>

> >http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven....
>
> >http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven....

> > and the CalculateFacets() function here
>

> >http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven....

[Ayende Rahien]

My only concern is that I am not really sure how to expose this to the client API.

Ideas?

[Matt Warren]

> The code isn't done, and I would love for you to take a look at that and see
> how we can build client API for this, but it seems like it would be much
> easier approach.

The best I could come up with was this:

facetQuery.("cameraInfo", new List<FacetMode>
{
new FacetMode {Name = "Manufacturer"}, //default is term query
//In Lucene [ is inclusive, { is exclusive
new FacetMode {Name = "Cost", DataType = TypeCode.Decimal,
Ranges = {
"[NULL TO 200.0}",
"[200.0 TO 400.0}",
"[400.0 TO 600.0}",
"[600.0 TO 800.0}",
"[800.0 TO NULL]",
}},
new FacetMode {Name = "Megapixels", DataType = TypeCode.Decimal,
Ranges = {
"[NULL TO 3.0}",
"[3.0 TO 7.0}",
"[7.0 TO 10.0}",
"[10.0 TO NULL]",
}}
});

But this wouldn't fit well with LINQ and it's really just a strongly
typed wrapper over the query string you manually write. Plus it would
get even more verbose if the ranges weren't string, but some so or
class with upper/lower properties.

> http://localhost:8080/databases/Tiger/facets/Products?Facet.Manufacte...[NULL

> TO 100]&Facet.Price.Range=[100 TO 200]&Facet.Price.Range=[200 TO NULL]
>
> Here are the actual parameters in a way that we can read it easily:
>
> Facet.Manufacterer.Mode=Default
> Facet.Price.Mode=Ranges
> Facet.Price.Range=[NULL TO 100]
> Facet.Price.Range=[100 TO 200]
> Facet.Price.Range=[200 TO NULL]
>
> The result of this query si:
>
> {
> "Manufacterer": [
>  {
> "Range": "apple",
> "Count": 2
>  },
> {
> "Range": "goggle",
>  "Count": 1}
>
> ],
>  "Price": [
> {
> "Range": "[NULL TO 100]",
>  "Count": 2},
>
> {
>  "Range": "[200 TO NULL]",
> "Count": 1
>
> }
>  ]
> }
>
> We can also add facets to a query, like this:
>

> http://localhost:8080/databases/Tiger/facets/Products?Facet.Manufacte...[NULLTO

> 100]&Facet.Price.Range=[100 TO 200]&Facet.Price.Range=[200 TO
> NULL]&query=Manufacterer:apple
>
> Which then gives us:
>
> {
> "Manufacterer": [
> {
>  "Range": "apple",
> "Count": 2}
>
>  ],
> "Price": [
> {
>  "Range": "[NULL TO 100]",
> "Count": 1},
>
>  {
> "Range": "[200 TO NULL]",
> "Count": 1
>  }
> ]
>
> }
>
> The code isn't done, and I would love for you to take a look at that and see
> how we can build client API for this, but it seems like it would be much
> easier approach.
>

> On Thu, Jul 28, 2011 at 8:28 PM, Matt Warren <mattd...@gmail.com> wrote:
> > Thanks,
>
> > Just a couple of things, the higher-level outline of what I'm doing is
> > here

> >http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/readme....

>
> > The codes not finished, I have to write some more unit tests and cover
> > a hole load of edge cases before I submit a pull-request, but the
> > basic idea is there.
>
> > Most of the work takes place in these files:
>

> >http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven....
>
> >http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven....

> > and the CalculateFacets() function here
>

> >http://github.com/mattwarren/ravendb-FacetedSearch/blob/master/Raven....

[Ayende Rahien]

What if we would split things?

We store the facet definition in a document, then do something like:

GET /facets/CameraFacets

Where "CameraFacets" gives us the doc name.

We can then do something like:

session.Query<Camera>().Where(x=>x.Name.Contains("Z10")).ToFacets("CameraFacets");

???

[Matt Warren]

I like that approach and that was similar to what my code did (but for
a different reason).

It also fits in with how other libraries handle facets, i.e. you
define the facet fields/ranges up-front and then queries include
those. But having the option of doing it dynamically is also really
nice!

On Aug 2, 1:58 pm, Ayende Rahien <aye...@ayende.com> wrote:
> What if we would split things?
> We store the facet definition in a document, then do something like:
>
> GET /facets/CameraFacets
>
> Where "CameraFacets" gives us the doc name.
>
> We can then do something like:
>

> session.Query<Camera>().Where(x=>x.Name.Contains("Z10")).ToFacets("CameraFa cets");
>
> ???

[Ayende Rahien]

Okay, would you be able to complete this and create a pull request?

[Matt Warren]

Yeah no problem, I should get some time over the next few days

On Aug 2, 2:21 pm, Ayende Rahien <aye...@ayende.com> wrote:
> Okay, would you be able to complete this and create a pull request?
>

[Ayende Rahien]

Matt, did you get the chance to look at this?

[Matt Warren]

Sorry just got back from 2 weeks holiday, if you can give me another
week I can finish it off for you.

On Aug 24, 12:28 pm, Ayende Rahien <aye...@ayende.com> wrote:
> Matt, did you get the chance to look at this?
>

> ...
>
> read more »

[Ayende Rahien]

Hi, any news on this?

[Matt Warren]

Yeah, got quite a bit working, see the commits here
https://github.com/mattwarren/ravendb/tree/facets

Currently you can issue URL's like this:

http://localhost:8080/facets/CameraCost?facetDoc=facets%2fCameraFacets&query=Cost_Range[Dx100
TO NULL] AND Cost_Range[Dx200 TO Dx300]

and it will load the doc "facets/CameraFacet" and use that.

The Client API looks like this
var facetResults = s.Query<Camera>("CameraCost")
.Where(x => x.Cost >= 100 && x.Cost <=
300)
.ToFacets("facets/CameraFacets");

See here for the full test https://github.com/mattwarren/ravendb/blob/facets/Raven.Tests/Faceted/FacetedIndex.cs

Just having some wierdness between "[Dx100 TO Dx200]" and "[100 TO
200]", when using Client/Server compared to Embedded mode, 1 works and
1 doesn't!! I think it's to do with where the "Dx" does or doesn't get
added by the Client API. But I'm too tired to do much more now!

> ...
>
> read more »

[Ayende Rahien]

Hm...

Matt, I looked on your stuff, and it:

a) looks good

b) I can't get it to fail using embedded testing

c) I push it to the unstable branch

If you can write some docs for it, that would be wonderful

[Matt Warren]

> Matt, I looked on your stuff, and it:
> a) looks good

Cheers, I tried to do it in a sensible way

> b) I can't get it to fail using embedded testing

Okay, I try and write a unit test that breaks it and send an update if
necessary. However it was late when I wrote that code so I might've
been imaging it!!

> c) I push it to the unstable branch
>
> If you can write some docs for it, that would be wonderful

Sure, I'd be happy to.

Just a couple of other things

- Would it be beneficial for the facet results to be returend with the
query results, in 1 call? I.e. extend QueryResult to have a "Facet"
property, like if has one for "Includes"
- There'a a bug I need to fix with the Range strings. At the moment
you can't mix and match "{" and "[", inclusive and exclusive, because
the Lucene Query Parser fails to parse it. So you can't write "{100 TO
200]", "{200 TO 300]", which might be useful in certain scenarios.
I'll send a pull-request for this.

> On Sat, Sep 3, 2011 at 3:01 AM, Matt Warren <mattd...@gmail.com> wrote:
> > Yeah, got quite a bit working, see the commits here
> >https://github.com/mattwarren/ravendb/tree/facets
>
> > Currently you can issue URL's like this:
>

> >http://localhost:8080/facets/CameraCost?facetDoc=facets%2fCameraFacet...[Dx100

> > TO NULL] AND Cost_Range[Dx200 TO Dx300]
>
> > and it will load the doc "facets/CameraFacet" and use that.
>
> > The Client API looks like this
> >    var facetResults = s.Query<Camera>("CameraCost")
> >                            .Where(x => x.Cost >= 100 && x.Cost <=
> > 300)
> >                            .ToFacets("facets/CameraFacets");
>
> > See here for the full test

> >https://github.com/mattwarren/ravendb/blob/facets/Raven.Tests/Faceted...

> ...
>
> read more »

[Ayende Rahien]

inline

On Sun, Sep 4, 2011 at 11:34 PM, Matt Warren <matt...@gmail.com> wrote:

> c) I push it to the unstable branch

>
> If you can write some docs for it, that would be wonderful

Sure, I'd be happy to.

Great

 

Just a couple of other things

- Would it be beneficial for the facet results to be returend with the
query results, in 1 call? I.e. extend QueryResult to have a "Facet"
property, like if has one for "Includes"

Hm,

I would rather implement support for calling this lazily and async, that should handle this better.

 

- There'a a bug I need to fix with the Range strings. At the moment
you can't mix and match "{" and "[", inclusive and exclusive, because
the Lucene Query Parser fails to parse it. So you can't write "{100 TO
200]", "{200 TO 300]", which might be useful in certain scenarios.
I'll send a pull-request for this.

It is not a bug, it is logically consistent, and I am fine with {100 to 201}

[Matt Warren]

See http://ravendb.net/documentation/faceted-search.

It's not linked to from anywhere else on the site at the moment, I
thought you'd want to take a look first.

BTW are you happy for me to post a thread on this board about it, so
that it gets some more visibility?

On Sep 5, 4:39 am, Ayende Rahien <aye...@ayende.com> wrote:
> inline
>

[Ayende Rahien]

inline

On Mon, Sep 5, 2011 at 7:13 PM, Matt Warren <matt...@gmail.com> wrote:

See http://ravendb.net/documentation/faceted-search.

It's not linked to from anywhere else on the site at the moment, I
thought you'd want to take a look first.

Notes:

a) the size of the image make it harder to see the facets, can you provide a linked larger version?

b) the picture and the domain in the sample aren't the same, which may be confusing, could you use a shopping site picture instead?

c) the sample facet doc and the sample data do not agree (not Megapixels facet).

Can you talk a bit about how it works and in what situations it would be useful?

 

BTW are you happy for me to post a thread on this board about it, so
that it gets some more visibility?

Absolutely!

[Matt Warren]

I'll make those changes, no probs.

I've also found some weirdness around the Client/Server mode compared
to Embedded. I'll need to change the unit test because it currently
doesn't catch the problem.

I'll send a pull-request in the next couple of days to fix it as well.

On Sep 5, 6:38 pm, Ayende Rahien <aye...@ayende.com> wrote:
> inline
>

> On Mon, Sep 5, 2011 at 7:13 PM, Matt Warren <mattd...@gmail.com> wrote:
> > Seehttp://ravendb.net/documentation/faceted-search.

> > > > > > > > > > > > > > > "Non-Authoritive-Information": false...
>
> read more »

[Ayende Rahien]

thanks

[Matt Warren]

It turned out that the unit tests were broken and were not testing
everything, so I've sent a pull request that fixes this.

Also embedded and client/server work fine as you said, the new test
confirms this. In the next few days I'll send another pull request
that re-enables the "dynamic facets" as per your original code (where
you pass all the facet definitions in the query string) and some tests
for this.

I'll also fix the doc as per your mail further up the thread.

On Sep 5, 6:44 pm, Ayende Rahien <aye...@ayende.com> wrote:
> thanks
>

> ...
>
> read more »

[Ayende Rahien]

Thanks, very cool.

Started to look at your advanced patching, looks _very_ nice so far.

[Matt Warren]

Thanks, it's based around the suggestion here, I've just put it all
together and added a "Map" function to make iterating over arrays
easier.

I'm thinking of pulling in LINQ-to-Javascript (like this
http://jslinq.codeplex.com/), so that you can write LINQ code inside
the patch script, what do you think?

On Sep 11, 6:49 am, Ayende Rahien <aye...@ayende.com> wrote:
> Thanks, very cool.
> Started to look at your advanced patching, looks _very_ nice so far.
>

> ...
>
> read more »

[Matt Warren]

PArt of the reason for this is that it would make a C# client API
easier as we could just convert a LINQ query in "LINQ javascript"

On Sep 11, 8:26 am, Matt Warren <mattd...@gmail.com> wrote:
> Thanks, it's based around the suggestion here, I've just put it all
> together and added a "Map" function to make iterating over arrays
> easier.
>

> I'm thinking of pulling in LINQ-to-Javascript (like thishttp://jslinq.codeplex.com/), so that you can write LINQ code inside

> ...
>
> read more »

[Ayende Rahien]

I think that this is nicer, yes.

[Ayende Rahien]

And that would be the holy grail, yes.

[Matt Warren]

I just wanted to check are there any licensing issues using the
following libraries/tools:

- IronJs from https://github.com/fholm/IronJS/
- json2.js from https://github.com/douglascrockford/JSON-js
- "Map" function, a modified version of https://gist.github.com/1195667

On Sep 11, 8:38 am, Ayende Rahien <aye...@ayende.com> wrote:
> And that would be the holy grail, yes.
>

> ...
>
> read more »

[Ayende Rahien]

Shouldn't be an issue, no.