Re: [chicagoboss] [boss_db] Issues using Riak and Associations

[Evan Miller]

Hi Daniel,

Searching within Riak (and hence associations) has always been wacky.
The riakc library API changes with annoying frequency.

Anyway try this patch and let me know if it works better:

https://github.com/evanmiller/boss_db/commit/5fd838542868de28ffb6f2782480dfff8ecb3cbd

On Sun, Mar 31, 2013 at 4:06 PM, <daniel....@gmail.com> wrote:
> Hi -,
>
> I'm toying around with CB 0.8.5 and Riak 1.3, and I think I have encountered
> a bug (but it seems so fundamental that I'm mostly confused about what's
> happening).
>
> I currently have three models:
>
> account
> comment
> post
>
> With the following associations set up between the three:
>
> account
> -has({posts, many}).
> comment
> -belongs_to(account).
> -belongs_to(post).
> post
> -belongs_to(account).
> -has({comments, many}).
>
> Everything is working fine when using the mock adapter, but once I switch to
> riak, I'm seeing errors like this one:
>
> {{function_clause,
> [{lists,map,
> [#Fun,{search_results,[],0.0,0}],
> [{file,"lists.erl"},{line,1172}]},
> {boss_db_adapter_riak,find,7,
> [{file,"src/db_adapters/boss_db_adapter_riak.erl"},{line,63}]},
> {boss_db_controller,handle_call,3,
> [{file,"src/boss_db_controller.erl"},{line,114}]},
> {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,588}]},
> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},
> {gen_server,call,
> [<0.95.0>,
> {find,post,
> [{account_id,equals,"account-VypnEwwcghmSLe9mQczhezcTZoN"}],
> all,0,id,ascending,[]},
> 30000]}}
>
>
> This is happening when fetching an account (and trying to access the posts
> association in a template), so it looks like BossDB is trying to fetch all
> posts related to an account (using the account_id field on posts) and then
> fails. At first I thought that it's because the user simply has no posts,
> but it also happens after I added one (and made sure the account_id field
> points to the right user).
>
>
> Here is what I did to "set up" Riak (only what the README said):
>
> Enabled riak_search via app.config
> "Installed" the following search commands:
> - account
> - comments
> - posts
>
> When I check the buckets they all say search: true and have the following
> precommit hook:
>
> {
>
> "mod": "riak_search_kv_hook",
>
> "fun": "precommit"
>
> }
>
>
> So I'm pretty sure I've set up Riak correctly.
>
> After searching around for a bit, I found this:
> https://groups.google.com/d/msg/chicagoboss/9XBTg2HHmrI/TJGtVFiYZIIJ It
> sounds like this could be related, but Im really not sure.
>
> Btw. Running boss_db:find("account-VypnEwwcghmSLe9mQczhezcTZoN") (in the
> REPL) works just fine, because it looks like it's not fetching the
> association. What doesn't work however, is something like this:
> boss_db:find(post, [{account_id, "account-VypnEwwcghmSLe9mQczhezcTZoN"}])
> (getting the same error as above).
>
> I do realize that the Riak adapter is marked as experimental, but
> associations should work, right?
>
> Best regards.
>
> --
> You received this message because you are subscribed to the Google Groups
> "ChicagoBoss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to chicagoboss...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Evan Miller
http://www.evanmiller.org/

[Daniel Kempkens]

Hi Evan,

first: Such a quick response, awesome :)

I did a quick test and the error is gone now. However, there seems to be another problem: Having only one user in the DB (username=daniel), running boss_db:find_first(account, []) returns the correct record. Running the following, sadly, returns undefined :(

boss_db:find_first(account, [{username, "daniel"}])

Best regards.

[Daniel Kempkens]

I looked into this a bit more, but I know way too much about Erlang (or Riak/riak_search) in order to figure out what's going on …

What I did notice, however, is that riak_search doesn't seem to index the proplist. I'm unable to run any query on it (doesn't matter what client library I use). SO I started digging around a bit and found this:

Riak Search is able to handle several standard data encodings with zero configuration. Simply set the Content-Type metadata on your objects to the appropriate mime-type. Out of the box, XML, JSON, and plain-text encodings are supported.

No mention of Erlang proplists … Later they mention that:

If your object contains Erlang terms, you can set your Content-Type to “application/x-erlang”. This expects either an Erlang term that is a proplist or a nested proplist. In the case of a proplist, the key is used as the field name, and the value as the field value. When the object's value is a nested proplist, field names are constructed by concatenating the nested keys together with underscores in between.

Looking through the source, I found that the adapter sets the Content-type to "application/x-erlang-term", so I changed it to what the documentation says. This didn't help either, so I kept and search and found this:

If the data was originally stored using the distributed erlang client (riak_client), the server will automatically term_to_binary/1 the value before sending it, with the content type set to application/x-erlang-binary (replacing any user-set value). The application is responsible for calling binary_to_term to access the content and calling term_to_binary when modifying it.

So I checked the bucket entry using Rested (HTTP Client) and it returns the following content type (for the already mentioned account object):

Content-Type: application/x-erlang-binary

Which leads me to believe that this is why riak_search seems to be unable to index any entries.

Am Montag, 1. April 2013 01:18:56 UTC+2 schrieb Evan Miller:

[Daniel Kempkens]

Me again. Figured out a bit more. It looks like it's not related to the Content-Type, but to how the proplist is "generated."

Manually inserting the following works as expected:

[{<<"username">>, <<"daniel">>}]

But boss_db inserts:

[{username, "daniel"}]

Which doesn't work.

Any help would be much appreciated :)

Am Montag, 1. April 2013 01:18:56 UTC+2 schrieb Evan Miller:

[Evan Miller]

Great, thanks for investigating this. I've long been waiting for a
knight in shining armor to figure out what the heck is going on with
Riak search.

I think the line you'll want to change is this:

https://github.com/evanmiller/boss_db/blob/master/src/db_adapters/boss_db_adapter_riak.erl#L107

Probably just need to check the types of K and V to binaries and
convert if needed.

Evan

[Daniel Kempkens]

Hi Evan,

https://github.com/nifoc/boss_db/tree/fixes/riak_search

Changing the way a record is saved also required changing searching a bit, but I did all that :) I'm not submitting a pull request (yet) for two reasons:

1. I only did some rudimentary testing
2. The way this currently works is rather stupid. We already have the entire record after the :search call, no need for find_acc to fetch again based on the ID (see commit)

Btw: I only started learning Erlang last week … if my code seems stupid, than that's why :D 

Best regards,

Daniel

[Evan Miller]

Great, thanks. find_acc is an artifact of an older API that did not
return full records, so feel free to trash it.

A couple code comments:

1. binary_to_atom is relatively new so I would avoid it in favor of
list_to_atom(binary_to_list()) for backward compatibility

2. try/catch should be used sparingly... stick to case statements and
guards if you can. Also it would be nice if the decoder were aware of
the model's type-hints, see e.g.

https://github.com/evanmiller/boss_db/blob/master/src/db_adapters/boss_db_adapter_mysql.erl#L241

Happy to answer any questions about this.

Evan

On Mon, Apr 1, 2013 at 2:09 PM, Daniel Kempkens

[Miguel Benitez]

Hi community, I have the same problem, any update on this? I have ChicagoBoss 0.8.12 and riak 2.0 with solr support enabled. I would greatly appreciate any help in this regard.