Questions about writing a storage backend for Solr for nodes/edges in addition to indexes

20 views
Skip to first unread message

Russell Jurney

unread,
Jan 23, 2020, 2:54:50 PM1/23/20
to JanusGraph developers
Hello fellow JanusGraphers, 

I'm interested in writing a storage backend for Solr to store nodes/edges in addition to indexes and would like some pointers on doing so. We would open source the implementation as a part of JanusGraph.

I work at an analytics startup and we're using JanusGraph to analyze the open source ecosystem. We have a Solr cluster to search over the same data that will appear in our JanusGraph database. It occurred to me that it would be nice to have one less system and so rather than store our graph data in duplicate some place like Cassandra or BigTable, why not access it on a Solr cluster? Solr is a capable database. It would be cool to use the same index but even if it were a different index served from different machines in the cluster, we would have significantly less operational overhead than if we ran something else and had to develop expertise in both systems.

So, my questions are:

* Is a Solr storage backend for nodes/edges a good idea? Terrible idea? Smart or stupid? Thoughts in general? Would you use this?

* What is the best resource on creating a storage backend? Google results are not promising.

* How much work would be involved to extend the janusgraph-solr module to also store nodes/edges?

* Is there a particular existing storage backend that I could refer to that would be similar to a Solr backend for nodes/edges? Is the in-memory the most agnostic one, or is another one more specific to something like Solr?

* How much work would be involved? The existing storage backends vary a lot in how much code they are made up of, so I have no idea of an estimate.
 
* Would anyone be interested in doing this work as a consultant? Advising or building? If you built it, how long would it take you?
 
Thanks in advance!

Russell Jurney, Founding Engineer @ Archipelo.co (we do foss graphs)

Dmitry Kovalev

unread,
Jan 24, 2020, 8:09:07 AM1/24/20
to JanusGraph developers
Hi Russell,

I wonder if your Google results included the following fairly recent thread in this same maillist:
Anyone with experience of adding new Storage backend for JanusGraph ? [Help needed w.r.t SnowFlake]  24/10/2019 

I believe it covers most of the questions above, except for Solr specifics, and should give you the basic understanding of what you need to do in order to implement a new storage backend. As to how much effort it is to implement it using Solr specifically - having never seen Solr myself, I cannot help you with that. You might have observed already that, depending on how well the target database data model is aligned with the Janusgraph expectations, the backend implementation effort can vary a lot.

Hope this helps,
Dmitry


--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/4627b368-a1a5-4066-a6de-53e3e8dce500%40googlegroups.com.

Russell Jurney

unread,
Jan 24, 2020, 1:25:50 PM1/24/20
to janusgr...@googlegroups.com
Thank you! No, I missed that. I appreciate your help!

Russ

--

Debasish Kanhar

unread,
Feb 3, 2020, 3:45:44 PM2/3/20
to JanusGraph developers
Hi Russel,

Just posted an update on the other Thread, and well as you might have guessed already, its finally all about extending the right set of interfaces for enabling JanusGraph with your target backend. While those are easy, the prime issue you face is how do you model your data in target backend when there is too much variability? Do you model as Key Value store or as Document store? or as something else? We faced that issue for modelling in SnowFlake. And once done, you will need to write set of custom transformers to transform JanusGraph's Interfaces operations into corresponding backend queries.

Debasish
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@googlegroups.com.
--
Reply all
Reply to author
Forward
0 new messages