To Contribute Our Code Enhancement Back to JanusGraph-FoundationDB Storage Plugin

[Jun Li]

Hello,

In eBay, we now have JanusGraph deployed in production, with FoundationDB being the backend store. The details on the design, deployment and the eBay use cases can be found in last year’s FoundationDB Summit (https://www.youtube.com/watch?v=EtB1BPG00PE). We took the earlier version of JanusGraph-FoundationDB plugin developed by Ted Wilmes in February 2019, and have made bug fixing and improvement over the last two years.  Now we would like to contribute our code back to the open source community, with the following improved features:

(1)Some bug fixing for the current JanusGraph FoundationDB plugin, and support of Async. Iterator. With Asynchronous Iterator, we can greatly leverage FoundationDB’s Asynchronous Iterator in terms of on-demand data stream pulling, better memory efficiency, better parallelism for MultiQuery support. We introduce “iterator” mode, along with the current mode that we call “list” mode.

(2)Enable Prefetching on Global Read Version of FoundationDB. This allows read-only query to use a slightly delayed GRV for snap-shot read (configurable, say 100 ms delayed), but save the GRV fetching time from the Primary DC. In the cross-datacenter production environment, the saving can be as large as > 50 ms due to this GRV pre-fetching.

(3) Support Read-Only GraphDB from the specified environment variables at GraphDB engine start-up.

(4)Context propagation that allows the Client Application to express “read-only” optimization support from Feature 2 mentioned above, to the FoundationDB storage plugin. So that write-involved queries will not be optimized. Furthermore, we can allow transaction reset with “read-commit-write” mode to be per-transaction, and only for the “read-only” queries, further optimization for “read-only” queries.The write-involved queries will still follow "serializable" mode for strong consistency.

(5)Enable TLS that is supported by FoundationDB. Security is important in the production environment.

(6)Enable Prometheus Metrics Monitoring. So that we can have good visibility across multiple datacenter deployment in the production environment. From the metrics collected, dashboards and alerts can be further developed easily. We can share our Grafana dashboards JSON configuration files and Prometheus alert rules used in our production environment  to the open-source community also. 

The first PR on Feature 1 above can be found at: https://github.com/JanusGraph/janusgraph-foundationdb/pull/50

Regards,

Jun Li and Hieu Nguyen

[Jason Plurad]

+1 thank you both for the enhancements. Looking forward to getting these contributions integrated into the project!

[jackson.chr...@gmail.com]

These all are welcome contributions, thanks so much for taking the time to share these with the community it's greatly appreciated! 

On Sunday, October 4, 2020 at 10:11:39 AM UTC-4 Jun Li wrote:

[Florian Grieskamp]

+1 Thank you for sharing your improvements with the JanusGraph community. From what you describe, the storage adapter will substantially profit of your modifications. I'm looking forward to have a closer look at it and rewiew the PR in detail. I'm very excited about the iterator mode, which sounds like a great idea!

[Henry Saputra]

Just noticed this one.

Congrats, Jun Li and Hieu Nguyen

It is great to see the hard work done inside eBay can be shared with the community!

- Henry

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/43fd48ac-9211-48ee-a1ea-240354010dc2n%40googlegroups.com.