Schema management tools

[Robert Dale]

With so many high-profile companies and others out there with Titan/Janus in production, I can only imagine that most, if not all, have written some higher-level tools for managing the database and, in particular, the schema.

This is a call to publish your tools.  I'll go first.  I whipped up a little 'describe' command to help make it easier to see the schema and statuses.

https://gist.github.com/robertdale/ad4c63910009dd1118abe67b33ce41e1

This shows the output after loading GraphOfTheGods and running through some of the examples in the docs.

./bin/gremlin.sh -i describe.groovy

gremlin> graph = JanusGraphFactory.open('conf/janusgraph-cassandra-es.properties')
==>standardjanusgraph[cassandrathrift:[127.0.0.1]]
gremlin> GraphOfTheGodsFactory.load(graph)
gremlin> :schema
Usage: must provide command: describe [ vertex | edge | propertykey | index ]
Unknown command: null
gremlin>
gremlin> :schema describe
Vertex Label                   | Partitioned | Static
------------                   | ----------- | ------
titan                          |       false |  false
location                       |       false |  false
god                            |       false |  false
demigod                        |       false |  false
human                          |       false |  false
monster                        |       false |  false
person                         |       false |  false
------------                   | ----------- | ------


Edge Name                      |            Type |        Directed |     Unidirected |    Multiplicity
---------                      |            ---- |        -------- |     ----------- |    ------------
brother                        |            Edge |            true |           false |           MULTI
follow                         |            Edge |            true |           false |           MULTI
father                         |            Edge |            true |           false |        MANY2ONE
mother                         |            Edge |            true |           false |        MANY2ONE
battled                        |            Edge |            true |           false |           MULTI
lives                          |            Edge |            true |           false |           MULTI
pet                            |            Edge |            true |           false |           MULTI
---------                      |            ---- |        -------- |     ----------- |    ------------


PropertyKey Name               |            Type |     Cardinality |            Data Type
----------------               |            ---- |     ----------- |            ---------
birthDate                      |     PropertyKey |          SINGLE | class java.lang.Long
rating                         |     PropertyKey |          SINGLE | class java.lang.Double
name                           |     PropertyKey |          SINGLE | class java.lang.String
age                            |     PropertyKey |          SINGLE | class java.lang.Integer
time                           |     PropertyKey |          SINGLE | class java.lang.Integer
reason                         |     PropertyKey |          SINGLE | class java.lang.String
location                       |     PropertyKey |          SINGLE | class org.janusgraph.core.attribute.Geoshape
----------------               |            ---- |     ----------- |            ---------


Graph Index                    |      Type |          Element | Unique |       Backing |     PropertyKey |               Status
-----------                    |      ---- |          ------- | ------ |       ------- |     ----------- |               ------
name                           | Composite | JanusGraphVertex |   true | internalindex |                 |                    
                               |           |                  |        |               |            name |              ENABLED
vertices                       |     Mixed | JanusGraphVertex |  false |        search |                 |                    
                               |           |                  |        |               |             age |              ENABLED
byNameComposite                | Composite | JanusGraphVertex |  false | internalindex |                 |                    
                               |           |                  |        |               |            name |              ENABLED
byNameAndAgeComposite          | Composite | JanusGraphVertex |  false | internalindex |                 |                    
                               |           |                  |        |               |            name |              ENABLED
                               |           |                  |        |               |             age |              ENABLED
byNameUnique                   | Composite | JanusGraphVertex |   true | internalindex |                 |                    
                               |           |                  |        |               |            name |              ENABLED
nameAndAge                     |     Mixed | JanusGraphVertex |  false |        search |                 |                    
                               |           |                  |        |               |            name |              ENABLED
                               |           |                  |        |               |             age |              ENABLED
byNameAndLabel                 | Composite | JanusGraphVertex |  false | internalindex |                 |                    
                               |           |                  |        |               |            name |            INSTALLED
edges                          |     Mixed |   JanusGraphEdge |  false |        search |                 |                    
                               |           |                  |        |               |          reason |              ENABLED
                               |           |                  |        |               |        location |              ENABLED
----------                     |      ---- |          ------- | ------ |       ------- |     ----------- |               ------


Relation Index                 |                 Type |  Direction |   Sort Key | Sort Order |               Status
--------------                 |                 ---- |  --------- | ---------- |   -------- |               ------
battlesByTime                  |              battled |       BOTH |       time |       decr |              ENABLED
battlesByRatingAndTime         |              battled |        OUT |     rating |       decr |            INSTALLED
                               |                      |            |       time |            |                    
--------------                 |                 ---- |  --------- | ---------- |   -------- |               ------


gremlin>

[HadoopMarc]

Hi Robert,

Cool!  Actually, I use a fixed schema myself which I maintain in a json file and look at if needed. Your tool is nice though for other users of my graphs.

If your tool "wins", it would also be nice to document how to use it for remote execution on gremlin server.

Cheers,     Marc


Op woensdag 5 juli 2017 19:40:43 UTC+2 schreef Robert Dale:

[Florian Hockmann]

Thanks for publishing your tool! It looks really helpful to get an overview of the schema.

At my company we developed a tool that generates the schema creation groovy script together with C# and TypeScript entity classes (support for other languages would probably be easy to add) from a schema defined in a custom XML format. This has the advantage for us that we can define restrictions for the schema to make it usable for our use case. We for example require each vertex label to have a property with a composite index that is expected to be unique. Another restriction for us is that each edge label can only connect vertices with certain vertex labels. This would for example specify that the pet edge label can only connect human vertices with monster vertices.

Such a "strict" schema allows to generate helpful entity classes, but it may not be appropriate for everyone.

I can check if we can open source this tool when there is some interest from others.

[Nigel Brown]

This is a good tool, thanks.

 

[Jason Plurad]

Thanks for stirring up the conversation, Robert.

Viewing the schema definitions in tabular form is nice and a similar experience to what you'd find in other database REPLs. I really like the simple access with :schema describe without having to fuss around with the mgmt object directly.

I know some folks in my company have a JSON-based schema tool, which sounds similar to what Marc has. It allows for a relatively painless creation (and exporting) a static schema rather than going directly through the ManagementSystem API. The benefit is that you can manipulate the JSON programmatically, plus it works through the Gremlin Server.

I'd think that Robert's Gremlin Console integration and JSON-based schema definition aren't necessarily only one tool can win. Both could co-exist since they have different objectives. For example, if the ManagementSystem returns a JSON representation, the Gremlin Console plugin does a transform from JSON to text tables.

-- Jason

[Robert Dale]

Jason, this looks like the tool you mentioned.   https://github.com/yhwang/janusgraph-utils  and slide share:  https://www.slideshare.net/ChinHuang4/onboarding-with-janusgraph-performance

[ngrig...@gmail.com]

https://github.com/graph-lab/janusgraph-schema-manager

Work in progress but we successfully use it here at Newforma. The main purpose was to have something to define the schema and, most importantly, to maintain the schema documentation for the developers.