Nested properties

1,347 views
Skip to first unread message

Ashish Dalela

unread,
May 9, 2012, 1:59:24 AM5/9/12
to Neo4j
Folks,

I'm new to neo4j and need help. I read the documentation and found the
following which is a problem for me - "Nesting properties is not
supported". My question is - do we have a plan to reverse this
limitation?

My data objects are composite. E.g. A device comprises of cards, and
cards comprise of ports which contain virtual ports, each of which can
have multiple tiers of configurations. To represent this in neo4j with
the above limitation, I have to break this down into individual nodes
and connect them through "contains" relationships. Then, my XPath
expressions such as /device/card/port/virtualport/attrlevel1/
attrlevel2 will have to span across multiple nodes.

XML databases allow us to represent such complex objects easily, but
they have no relations. Graph databases have relations but the objects
are forced to be primitive. Can this either / or situation be
corrected?

Thanks much for your time.

-Ashish

Peter Neubauer

unread,
May 9, 2012, 2:32:38 AM5/9/12
to ne...@googlegroups.com

Ashish,
This has been a very early decision to force explicit data structures with the least amount of surprises when dealing with it. That said, there are plans for the introduction of more property types, among them date and nested maps (documents). We are not sure yet if it makes sense to treat them as a property type or internally actually decompose them into subgraphs and only expose unified api to that.

Wdyt?

Send from mobile.

Ashish Dalela

unread,
May 9, 2012, 2:47:18 AM5/9/12
to ne...@googlegroups.com
Peter,

The ability to collapse a graph into a node or expand a node into a graph would make it really powerful, and also a logical extension of the present schemes. Whether we model that "graph inside a node" as a new property of the node or a recursive graph would depend on how hard it is to do either one of them, and I don't know the implementation so it's hard to say.

The basic capability needed is that of being able to build abstractions on graphs so you can zoom in or out of a graph. 

Ashish

Axel Morgner

unread,
May 9, 2012, 3:22:40 AM5/9/12
to ne...@googlegroups.com
Maybe you might have a look at structr [1]. It is completely Open Source and has a Java layer which allows for building domain-specific JSON REST APIs on top of (embedded) Neo4j.

The JSON properties can be either stored as node or relationship properties, or even remotely (e.g. "user_id" of a given JSON object points to the id of a User, connected by a relationship to the node represented by the JSON object). structr sports search (fulltext, keyword), has neo4i-spatial integrated for spatial operations (only distance search for now), allows background operations, software agents with cron-like time control, and even more complex stuff like traversers, filters etc. if needed.

It supports exposing nodes or relationships as REST resources including nested properties, property groups, property converter, paging, sorting, default values, value constraints (not null-, range-, composed-, regex-), cascading delete, etc..

One application of structr is a CMS (but not limited to it). As an example, you can have a look at [2] which is the representation of a DOM element as a structr entity.

While structr is still in a beta state and not yet well documented, it is actively developed and f.e. powers https://splink.de, a sports-oriented social and knowledge network in Germany. 

Does that sound interesting?

Axel

Peter Neubauer

unread,
May 9, 2012, 3:34:38 AM5/9/12
to ne...@googlegroups.com
Also,
if you looks at Spring Data Neo4j
(http://www.springsource.org/spring-data/neo4j#documentation), objects
can be transparently serialized into properties via annotations. If
you zoom out into the object layer you will be able to see even
complex properties, it is just that we don't want to do overly complex
properties at the DB layer because of the reasons I mentioned above :)



Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

If you can write, you can code - @coderdojomalmo
If you can sketch, you can use a graph database - @neo4j

Ashish Dalela

unread,
May 9, 2012, 3:57:23 AM5/9/12
to ne...@googlegroups.com

Axel,

Thanks for the pointers. I will investigate and get back.

Ashish

Ashish Dalela

unread,
May 9, 2012, 4:00:20 AM5/9/12
to ne...@googlegroups.com

Peter,

In my limited understanding, I thought that the schemas were fixed with Spring (i.e. you have to define the mappings in advance). If that's true, then I would like to avoid that since my goal is to not use any schema (the basic idea with no-sql). Of course, I might be wrong, and if so, please correct me.

Ashish

Michael Hunger

unread,
May 9, 2012, 4:04:26 AM5/9/12
to ne...@googlegroups.com
I agree with Axel,

if you don't want to evolve your complex properties into subgraphs, storing the data as json structures is the best you can do for now.

Cheers

Michael

Axel Morgner

unread,
May 9, 2012, 12:11:04 PM5/9/12
to ne...@googlegroups.com
Hm, we're not storing JSON docs as a whole in structr. We're using GSON for parsing a JSON request and dynamically translate it into graph operations. On the other side, we're composing complex JSON objects from graph objects.
--
Axel Morgner
Morgner UG (haftungsbeschränkt)
Hanauer Landstr. 291a
60314 Frankfurt Germany


Matthew Young

unread,
May 10, 2012, 9:12:40 AM5/10/12
to ne...@googlegroups.com
@Axel, been thinking in the same lines as Ashish in CRUDing on complex typed properties (bean containing beans) rather than flat lining a node into simple typed properties.  Problem is uniqueness across multiple nodes.  And uniqueness in Neo4j relies on locking the index (as far as I know) in the putIfAbscent/getOrCreate methods.  Looked at the Structr code (nicely written by the way) and it looks like the indexes are done on individual nodes not across nodes:

but I haven't hunted down where the additions to the NodeService indexes are added so correct if I am wrong (that indexes are constrained like normal Neo4j to a node rather than a conglomeration of nodes)?

I wonder if it is possible to use establish an index on more than Nodes/Relationship like a tree of these entities?  Haven't looked at the internals of the Index.  Just assuming it is radical break from the underlying design in Neo4j.  Which puts me back at solving it in the service layer living with the potential for overwrites (even if I implement an optimistic lock, like versioning in Hibernate) or creating a custom Index implementation that gets injected via the UniqueFactory if this is even possible?

A great example I want to solve is the idea of representing Software in Neo4j.  Software like modules in Ivy generally is made up of an Organization and Name and Version.  At least the Organization property I want to not be a simple string rather a relational to a Organization node.  Since organizations are reusable metadata and a viable node.  Both Software and Organization have different uniqueness.  Software is unique when the composition of org, name, and version is unique.  And Organization is unique in it's name property is unique.

Matthew Young

unread,
May 11, 2012, 4:08:53 AM5/11/12
to ne...@googlegroups.com
Dug into the Neo4j Lucene to look at how a Lucene document is built up against the Value sent in.  A light bulb switched on for me.  Basically the key for the index (based of the entity id, ex. the node id or relationship id) can be anything and the Value an array of "fields".  So in my example about with Software against Organization if the Software is what I am looking at then I can add an key with a human readable name like "Software.name, Software.Organization.name, Software.version" and hash that into a numeric or whatever then the Value will be the values of "Software.name", "Software.Organization.name" and "Software.version" as an array.  This index will never be used for searching rather only when using the putIfAbsent method.  But I will raise this as a separate issue (question).

Peter Neubauer

unread,
May 11, 2012, 8:21:19 AM5/11/12
to ne...@googlegroups.com
Interesting approach!

Would love to see an example and maybe a blog post on this when you
get it working?

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

If you can write, you can code - @coderdojomalmo
If you can sketch, you can use a graph database - @neo4j


Axel Morgner

unread,
May 12, 2012, 5:20:53 AM5/12/12
to ne...@googlegroups.com
Matthew,

it's right, we do not have composite indexing across nodes or rels. We
do complex uniquess (or other) checks in the onCreation method of a
node's entity class, preventing the transaction from being committed if
one fails. You could f.e. combine uniquess checks with other validators,
e.g. RegEx, minimum length or the like. In some Neo4j-related projects,
we found that in real-world use-cases, you need a lot of flexibilty when
doing the validation in your REST API.

For your use case

Software.name
Software.version
Software.Organization.name

we'd define two node entity classes Software and Organization like this
(simplified, untested code):

public class Software extends AbstractNode {

public enum Key implements PropertyKey{ name, version, organization }

static {

// Make "name", "version" and "organization" valid properties
EntityContext.registerPropertySet(Software.class,
PropertyView.All, Key.values());

// Define "organization" as being the "name" property of a
related Organization node
// The second argument of the PropertyNotion constructor means
'createIfNotExists'
EntityContext.registerEntityRelation(Software.class,
Organization.class, RelType.HAS, Direction.OUTGOING,
Cardinality.OneToMany, new PropertyNotion(Organization.Key.name, true));

// Uniquess for Software.name
EntityContext.registerPropertyValidator(Software.class,
Key.name, new TypeUniquenessValidator("Software"));

}
[...]

@Override
public boolean onCreation(SecurityContext securityContext,
ErrorBuffer errorBuffer) {

List<SearchAttribute> searchAttributes = new
LinkedList<SearchAttribute>();
Map<String, String> fields = new
LinkedHashMap<String, String>();
String name =
getStringProperty(Key.name);
String version =
getStringProperty(Key.city);
String org =
getStringProperty(Key.organization);

if (name == null) {
name = "";
}

if (version== null) {
version= "";
}

if (org== null) {
org= "";
}

// add type to search attributes

searchAttributes.add(Search.andExactType(Software.class.getSimpleName()));

// add new fields to search attributes
searchAttributes.add(Search.andExactProperty(Key.name, name));
searchAttributes.add(Search.andExactProperty(Key.version,
version));
searchAttributes.add(Search.andExactProperty(Key.org, org));
fields.put(Key.name.name(), name);
fields.put(Key.version.name(), version);
fields.put(Key.org.name(), org);

// do search
try {

List<GraphObject> results = (List<GraphObject>)
Services.command(securityContext, SearchNodeCommand.class).execute(null,
false, false, searchAttributes);

if (!results.isEmpty()) {

for (Entry<String, String> entry : fields.entrySet()) {

if (StringUtils.isNotBlank(entry.getValue())) {

String id = ((AbstractNode)
results.get(0)).getUuid();

errorBuffer.add(Software.class.getSimpleName(),
new UniqueToken(id, entry.getKey(), entry.getValue()));

}

}

return true;

}

} catch (FrameworkException fex) {
logger.log(Level.WARNING, "Unable to validate uniqueness of
Software");
}

// everything's ok
return false;
}
}


public class Organization extends AbstractNode {

public enum Key implements PropertyKey{ name, software }

static {

EntityContext.registerPropertySet(Organization.class,
PropertyView.All, Key.values());

EntityContext.registerEntityRelation(Organization.class,Software.class,
RelType.HAS, Direction.INCOMING, Cardinality.ManyToOne);
EntityContext.registerPropertyValidator(Organization.class,
Key.name, new TypeUniquenessValidator("Organization"));
}
[...]
}


The structr REST servlet would understand GET, POST, PUT and DELETE
requests to those two new resources.

F.e. a POST to /software '{"name":"Software
X","version":"1.0","organization":"Organization Y"}' would internally
create two nodes:

A node of type "Software" with the properties "name", "version" and
"organization"
A node of type "Organization" with the properties "name" and "software"

GET /software would return a JSON object like this:

{
"query_time": "0.002025186",
"result_count": 1,
"result": [
{
"id": "c6c098b9517f4b61b9e490aa93029192",
"type": "Software",
"name": "Software X",
"version": "1.0",
"organization": "Organization Y"
}
]
}

And a GET /organizations would return:

{
"query_time": "0.002047697",
"result_count": 1,
"result": [
{
"id": "a53d59240ba64f2584b7aeda348eb0d3",
"type": "Organization",
"name": "Organization Y",
"softwares" : [
{
"id": "c6c098b9517f4b61b9e490aa93029192",
"type": "Software",
"name": "Software X",
"version": "1.0",
"organization": "Organization Y"
}
]
}
]
}

The uniqueness of Software objects would be ensured in the onCreation
method above, and it's basically a single index query using the
integrated lucene, searching for a node of type "Software" with the
given properties. If the uniqueness test fails, the transaction will not
be comitted, so no object created (no Software, no Organization).

The trick is, that the getter (and setter) methods of our entity classes
respect the property type defintions, e.g. local or related,
cardinality, uniqueness etc..

Best regards
Axel

Matthew Young

unread,
May 12, 2012, 9:07:06 AM5/12/12
to ne...@googlegroups.com
@axel, thanks for taking time to show how it is done in your project. Cool stuff. Going to adjust Spring Neo4j and play w an UniqueFactory. Always fun to how other solve simular problems.

Peter Neubauer

unread,
May 12, 2012, 9:26:07 AM5/12/12
to ne...@googlegroups.com

Yeah contributions there are very welcome. Thanks guys!

Send from mobile.

Ashish Dalela

unread,
May 13, 2012, 12:49:16 AM5/13/12
to ne...@googlegroups.com

I'm taking a slightly different approach and need help.

I created a ServerPlugin using instructions here -  http://docs.neo4j.org/chunked/stable/server-plugins.html 

This plugin exposes two methods putXMLProp and getXMLProp. The PUT method basically decomposes a XML doc into sub-nodes and relationships (SUB_NODE). Using these methods, I can do things like:


which will return {"prop1" : "val1", "prop2", "val2"}

That is a rough XML equivalent of GET/PUT on "node/100/properties/abc".

My problem is that I don't see this extension in the GET to "db/data/node/{nodeId}". What am I not doing right?

1. I have a .jar with the class in the plugins directory
2. The .jar file has META-INF/services/org.neo4j.server.plugins.ServerPlugin file with my package name
I looked at the logs and I don't see anything wrong, and I don't see anything about the plugin being loaded.

Thanks, Ashish

Michael Hunger

unread,
May 13, 2012, 5:14:22 AM5/13/12
to ne...@googlegroups.com
2. The .jar file has META-INF/services/org.neo4j.server.plugins.ServerPlugin file with my package name

it must contain the fully qualified class name of your plugin, not the package name.

Michael

wang guan

unread,
Jan 14, 2014, 2:00:22 AM1/14/14
to ne...@googlegroups.com
I think the composite properties like hashmap is very important.
Json format data is very important and general data format, for example, now i have a json data or an php array data,they are both nested,but i want to import them into neo4j,but I found it's difficult,even important,the essential reason is neo4j can't support comosite properties,so i have to give up neo4j.

在 2012年5月9日星期三UTC+8下午2时32分38秒,Peter Neubauer写道:
Reply all
Reply to author
Forward
0 new messages