Reference API change proposal
Ben Mabey
First of all, I'd like to thank Wilkes and co for creating karras. It
has some great features and is what we (my work) had hoped to create
ourselves once we got the time. Thanks for releasing it!
Our biggest pain point with karras right now deals with
associations/references. We are using defaggregate where appropriate
but for certain collections we use the :reference(s) type. Ideally we
would like to interact with entities with references as if they are like
aggregates (embedded documents) but have karras handle the serialization
of them... For example, here is some pseudo-repl-like code (assume
with-mongo isn't needed):
(defentity Person
[:first-name
:last-name])
(defentity Company
[:name
:employees {:type :references :of Person}
:boss {:type :reference :of Person}])
(def michael (make Person {:first-name "Michael" :last-name "Scott"}))
; notice how this is unsaved
(def pam (create Person {:first-name "Pam" :last-name "Beasly"}) ;
this one is saved
(def dm (create Company {:name "Dunder Mifflin" :boss michael
:employees [pam]))
; this should auto-save the unsaved references, store the ids, but
preserve the real entity records in the :boss and :employees locations
after the save.. So...
(= [pam] (:employees dm)) ; true
(= (fetch-one Person {:last-name "Scott"}) (:boss dm)) ; true
When you fetch entities that have references you generally don't want to
fetch the references. However, a pipeline-able function that loads and
assocs the references (that maybe uses the get-reference fn internally)
would be useful for the times when you do want to references loaded onto
the parent entity:
(def dm-with-boss-loaded (-> (fetch-by-id Company {:_id dm)
(load-references :boss)))
(get-in dm-with-boss-loaded [:boss :last-name]) ; Scott
(def dm-with-all-refs-loaded (-> (fetch-by-id Company {:_id dm)
load-references)) ; fetches and assocs both :boss and :employees
We also played around with the idea of having get-reference memoize the
fetched reference with an atom in the metadata of the parent to prevent
repeat calls to the DB. However, that felt a little dirty....
Many times when we fetch an entity with references we generally don't
need to load them and we just need the id. To keep our functions
consistent though it might be nice to have the references always map to
the appropriate entity type. So...
(def dm-no-refs (fetch-by-id Company {:_id dm))
(:boss Company) ; user.Person{:_id #<ObjectId
4c3ce194fa40c2d7246d7efe>} ; so.. a shell of the Person would be
returned with just the _id
The advantage of this is that you can treat/query entities with
un-loaded references the same as if you would if they were loaded
(assuming all you need is the id). Meta-data could perhaps be added to
the entities indicating if they have been loaded or not.
This is what our ideal API for dealing with references, more or less,
would be. We have somewhat approximated this with some callbacks I
created. The callbacks are just a temporary solution until (if) karras
adds such functionality. (The callback solution falls apart when
aggregates have references though, so this should really be something
more fundamental to karras IMO.) Implementation details aside, how does
such an API sound? Do you see any problems with the proposed API changes?
Sorry for the long-winded message and thanks again,
Ben
Seth Buntin
yesterday with DBRefs. I'll take a look at this more and see where it
goes. I have only glanced at DBRefs and haven't even looked at how
the Java library implements it but I'll put it on the list of things
to do. If I hammer out something I'll push it to a separate branch
and have you test it.
Ben Mabey
understand about them it seemed like a good fit.
Somewhat tangential, but another use case we have WRT references is
being able to store the id in a different name than the reference.
Using the previous example we would store the reference ids as :boss_id
and :employee_ids (but still want to refer to the reference as :boss,
etc). This is because we are using Rails as the front-end for this app
which traditionally stores FKs with the _id suffix. (Yes, the underscore
annoys us to no end on the clojure side of things but since Ruby doesn't
like hyphens it is the only option.) With my current implementation I
take that into account and pivot the reference based on that
convention. I'm not asking, nor suggesting, that karras adopts such a
convention but at some point I'd like to see karras be flexible enough
to allow this. I'm more than willing to provide any needed patches, but
I thought I'd let everyone know so it can be kept in mind as different
implementations are tried out.
-Ben
Wilkes Joiner
> Awesome! Tim had suggested the use of DBRefs as well. From what I
> understand about them it seemed like a good fit.
the Javascript's API for storage for future use. However, the Java
driver prevented us from storing exactly their way. It prevents
storage of keys that start with $. So the new structure for references
is {:_db "db-name" :_id ObjectId :_ref "collection-name")}. We'll
make the karras.collection api aware of this structure once the
reference api pans out.
> Somewhat tangential, but another use case we have WRT references is
> being able to store the id in a different name than the reference.
> Using the previous example we would store the reference ids as :boss_id
> and :employee_ids (but still want to refer to the reference as :boss,
> etc). This is because we are using Rails as the front-end for this app
> which traditionally stores FKs with the _id suffix. (Yes, the underscore
> annoys us to no end on the clojure side of things but since Ruby doesn't
> like hyphens it is the only option.) With my current implementation I
> take that into account and pivot the reference based on that
> convention. I'm not asking, nor suggesting, that karras adopts such a
> convention but at some point I'd like to see karras be flexible enough
> to allow this. I'm more than willing to provide any needed patches, but
> I thought I'd let everyone know so it can be kept in mind as different
> implementations are tried out.
need to give this more thought.
To accomodate your first use we add 3 functions relate, grab, and grab-
in. It's not transparent but it does remove a lot heavy lifting.
I've pushed them to the references branch:
http://github.com/wilkes/karras/tree/references
relate associates a reference and saves the reference if it has not
been saved already.
grab and grab-in are analogous to get and get-in but will fetch any
references as need. The references are cached, but grab and grab-in
can take an optional refresh flag to force a fetch.
Here's your original example with the new api:
(defentity Person
[:first-name
:last-name])
(defentity Company
[:name
:employees {:type :references :of Person}
:boss {:type :reference :of Person}])
(def michael (make Person {:first-name "Michael" :last-name "Scott"}))
this one is saved
(def dm (-> (make Company {:name "Dunder Mifflin"))
(relate :boss michael) ; michael will be saved
(relate :employees pam)
save))
preserve the real entity records in the :boss and :employees locations
after the save.. So...
(= (fetch-one Person {:last-name "Scott"}) (grab dm :boss)) ; true
(def dm (-> (fetch-by-id Company {:_id dm))))
(grab-in dm [:boss :last-name]) ; "Scott" boss is now cached
(-> (grab dm :employees) first :last-name) ; "Beasly" employees are
now cached
(def dm (fetch-by-id Company {:_id dm)))
(:boss dm) ; {:_db "corporate" :_id #<ObjectId
4c3ce194fa40c2d7246d7efe> :_ref "people"}
Let me know how this works out for you guys. I think with these
functions in place we could tweak create to save after the references
have been associated. This would make the Company creation above a
little less awkward.
- Wilkes