Re: [Neo4j] Neo4j creating a file system. with paths / journal / travel / alaskin-natives-of-the-north

[Michael Hunger]

Joshua,

you can simply create structures like that as trees in the graph.

For accessing all the nodes from a page-root, you could use a query that follows a certain CHILD relationship to the end

start root=node:pages(page="my-page") match path = root-[:CHILD*]->leaf return path

or return NODES(path)

If you start at the inner node (e.g. travel you can also create a path to the root (journal))

start travel=node:travels(name="my-name") match path = travel<-[:CHILD*]->journal return path, journal

HTH

Michael

See the cypher cheat sheet for some hints: http://neo4j.org/resources/cypher and feel free to play around with http://console.neo4j.org

There are even CMS like structr which mimic pages structures in Neo4j.

Am 26.08.2012 um 19:13 schrieb Joshua Anderson:

I am considering Neo4j;  I have watched a video entitled, "Getting started with Neo4j." I have a basic (low level) understanding of relationships and nodes.  I want to have nodes that represent pages on a normal web server. I want to be able to store lets say a journal page and then create a relationship called "inside" to other nodes (its children).  

I must be able to query down the nodes to a particular item.  For example we have a journal and a travel node.  Travel is inside journal.  Routing to travel/journal  must obtain the journal node and all of its data.  I understand I will probably have to split at the slashes, but what would a syntax look like for this.

I am using nodejs, there are some restful plugins for Neo4j; how will this affect my ability to use all of Neo4j's?

I am hoping that Neo4j is right for this.  I hate creating a parent/sibbling relationship in a relational database.  Querying is terrible and reliability of queries is bad.

Thanks.

Joshua.

--
 
 

[Niels Hoogeveen]

Hi Joshua,

 

I would advice against nodes representing pages.

 

In my experience, building multispective.com, it's nicer to model the URL path as nodes, and have separate nodes representing the content related to the path. That way you separate the concerns of URL-resolution and presentation.

 

Having a separate node for the URL, and a separate node for the page-content, make things like redirects and changing the content of related to a URL, more manageable.

 

Niels

[Joshua Anderson]

So separate the data of a page its [date, author, views, tags, data] from the paths represented; so then, create a node with path="journal" and then associate a page to it like [travel].  But how does this provide any benefits over the other method.  Because path="journal" may be a page its self that must contain data. It needs to be dynamic, because travel may be a page or it may not be.

@Michael Hunger, Michael your example syntax helps me understand what is going on here; I cannot thank you enough for taking the time out of your day to assit me in this.

[Michael Hunger]

You're welcome.

Perhaps Axel can chime in too

Michael

Sent from mobile device

--
 
 

[Joshua Anderson]

Mich, Installed Neo4j on a ubuntu system in my home, so far so good, haven't tested it with nodejs yet; at first I was confused because functions I was using in the Neo4j console on neo4j.org were not working on my server.  I then realized it was because I wasn't using 1.8 build.  lol my head was turning left and right.

Joshua.

[Axel Morgner]

Hi Joshua,

IMHO, as a first step, it's completely ok to have a page node and to connect your content nodes as children. You might give the page a unique ID and access it at http://yourhost/<page_id>. To be more flexible, as Niels noted, you can add some path-to-page mapping afterwards.

But from our experience with structr, the most challenging part is not path mapping but the question how to integrate your business data into the rendering graph while avoiding serious complexity. We ended up with a very "graphy" model where markup, data and content are in the same graph, and the page content is determined by just a few control structures.

Some of the control elements "listen" to request parameters (or path elements) and change their behaviour in a pre-defined matter, creating a subgraph constisting of both, markup and business data, to be rendered in their place. The control elements "know" how a business object should look like (e.g. in which markup elements they should be embedded).

So the overall workflow is like that:

URL ---> find entry point (page) ---> render page elements (children). When hitting a control element, let the control element decide what to render out (e.g. render a list of data elements of a certain kind, or render the details of a data object with a certain ID, determined by analysis of request parameters).

One thing we learned while working with graphs, maybe can help you too: Try to draw your use case on a flipchart and then model it exactly like that. Start as simple as possible and avoid over-engineering.

HTH

Cheers
Axel

--
 
 

--

Axel Morgner · ax...@morgner.de · @amorgner

c/o Morgner UG · Hanauer Landstr. 291a · 60314 Frankfurt · Germany
phone: +49 151 40522060 · skype: axel.morgner

http://structr.org
http://www.meetup.com/graphdb-frankfurt
https://splink.de

[Niels Hoogeveen]

I agree with Axel, that the most challenging part IS the integration of the data into the pages.

 

There are different ways to go about this. From what I understand, structr doesn't use templates, and instead maps HTML structures onto the data store.

 

In multispective, we have chosen to embrace templates and store those in the data store as zipped String properties.

 

Templates help guide the traversal through the data store. Thanks to the tree-like nature of HTML, it can serve really well as a traversal description.

 

<example>

 

Suppose you want to show a blog entry, associated to a certain URL. The blog entry has a title, a text, an author and some tags.

 

In multispective we have a template that looks like this:

<div class="no-tabs content">
 <h2 id="common.title" class="append no-label"></h2>
 <div class="author-date">
  <span id="common.author:out"> <a id="base.user_name"
   href="/user/"
   class="attr(href,base.user_name,append) replace-inner no-label"></a>
  </span> <span id="root.created" class="append no-label"> / </span>
 </div>
 <div id="common.text" class="no-label replace-inner"
  style="margin-top: 1em;"></div>
 <div class="topic-list">
  <ul id="base.has_topic:out">
   <li><a id="base.topic_term" href="/topics/"
    class="attr(href,base.topic_term,append) replace-inner no-label"></a>
   </li>
  </ul>
 </div>
 <div id="comments" class="embed"></div>
</div>

 

The code above is valid HTML and can be created with any HTML editor, yet it contains enough information to dynamically create a page based on content in the attributes of the template.

The template transformation is started by putting the node associated with the blog entry into scope.

The h2 element contains an id telling the template engine to lookup a property named "title" within the namespace "common" and put that property in scope. The class attribute "append no-label" tells the engine to add the value of the property to the children of the h2 tag.

The span tag starting in the div "author-date", has an id "common.author:out", which instructs the template engine to look up a relationhip named "author" in the namespace "common" and put the iteration of the nodes found in the outgoing direction into scope,

The following a-tag, fetches the user_name of the author and puts that into scope. to place that as the text of the tag.

</example>

In short, we use the tree structure of HTML to give our templating engine traversal descriptions, so the engine can maintain positions in the graph.

Of course this is only one solution to the challenge to render data from the graph and our solution may not suit your specific need, but I wanted to share it with you anyway, so you get an idea of what can be done.

Niels

 

[Joshua Anderson]

How do we properly follow a complex url like journal/travel/alaskin-natives/story-1

We cannot just say start at journal and tell it to end at story-1 we are then missing all of the middle; Also, what if there is an immediate relationship of another node with that name story-1 that is attached directly to journal(example: journal/story-1) or story-1 could be in the next folder journal/travel.  At any given point we have to follow a direct url provided to ensure we have retrieved the exact item requested.  As far as following back up the tree I guess this would work, but what if for some odd reason (which I'd never do, but could mistakenly happen) the travel folders contains an item called journal.  How do you avoid conflicts and ensure the graph database is actually following a requested url, through each node.

I hope this makes sense.  If not, I may post a flow chart to demonstrate conflicts.

Joshua

[Christian Morgner]

Joshua,

to address your issue from a more technical standpoint, let me give an
example of how you could match the URL to the path you provided:

Take the URL "/journal/travel/alaskin-natives/story-1" and split it on
the "/" character. Now you have a hierarchically (descending) ordered
list of path parts you must match with your graph.

So first you need to find the node with the name "journal" and start a
traversal from there, following any outgoing (e.g. CHILD, ...)
relationship until you find a node with the name "travel". Now repeat
this process with all elements of the URL part list, and you have
successfully matched the URL path with your graph.

You can of course stop the traversal at any point and return a 404,
for example if there is no node with the desired name, or if there are
security restrictions that prevent the traverser from reaching a
specific node.

HTH

-Christian

On 27.08.2012 19:50, Joshua Anderson wrote:
> How do we properly follow a complex url like
> journal/travel/alaskin-natives/story-1
>

> We cannot just say *start* at /journal/ and tell it to *end* at
> /story-1 /we are then missing all of the middle; Also, what if
> there is an *immediate* relationship of another node with that name
> /story-1/ that is attached directly to /journal/(example:
> /journal/story-1/) or /story-1/ could be in the next folder
> /journal/travel./ At any given point we have to follow a direct

> url provided to ensure we have retrieved the exact item requested.
> As far as following back up the tree I guess this would work, but

> *what if* for some *odd* reason (which I'd never do, but could

> mistakenly happen) the travel folders contains an item called
> journal. How do you avoid conflicts and ensure the graph database
> is actually following a requested url, through each node.
>

> I hope this /makes sense/. If not, I may post a flow chart to
> /demonstrate/ /conflicts/.
>
> /Joshua/

>> path="*journal*" and then associate a page to it like
>> [*travel*]. But how does this provide any benefits over the other

>> method. Because path="journal" may be a page its self that must
>> contain data. It needs to be dynamic, because travel may be a
>> page or it may not be.
>>
>> @Michael Hunger, Michael your example syntax helps me understand
>> what is going on here; I cannot thank you enough for taking the
>> time out of your day to assit me in this.
>>
>> On Sunday, August 26, 2012 7:39:20 PM UTC-4, Niels Hoogeveen
>> wrote:
>>
>> Hi Joshua,
>>
>> I would advice against nodes representing pages.
>>
>> In my experience, building multispective.com

>> <http://multispective.com>, it's nicer to model the URL path as

>> nodes, and have separate nodes representing the content related
>> to the path. That way you separate the concerns of URL-resolution
>> and presentation.
>>
>> Having a separate node for the URL, and a separate node for the
>> page-content, make things like redirects and changing the content
>> of related to a URL, more manageable.
>>
>> Niels
>>
>>
>> On Sunday, August 26, 2012 7:13:06 PM UTC+2, Joshua Anderson
>> wrote:
>>
>> I am considering Neo4j; I have watched a video entitled,
>> "Getting started with Neo4j." I have a basic (low level)
>> understanding of relationships and nodes. I want to have nodes
>> that represent pages on a normal web server. I want to be able to
>> store lets say a journal page and then create a relationship
>> called "inside" to other nodes (its children).
>>
>> I must be able to query down the nodes to a particular item. For

>> example we have a journal and a travel node. Travel is /inside/

>> journal. Routing to travel/journal must obtain the journal node
>> and all of its data. I understand I will probably have to split
>> at the slashes, but what would a syntax look like for this.
>>
>> I am using nodejs, there are some restful plugins for Neo4j; how
>> will this affect my ability to use all of Neo4j's?
>>
>> I am hoping that Neo4j is right for this. I hate creating a

>> parent/sibbling relationship in a *relational database. Querying
>> is terrible and reliability of queries is bad.* * * *Thanks.*
>> *Joshua.*
>>
>> --
>>
>>
>
>
> --
>
> Axel Morgner � ax...@morgner.de <javascript:> � @amorgner

>
> c/o Morgner UG � Hanauer Landstr. 291a � 60314 Frankfurt � Germany
> phone: +49 151 40522060 � skype: axel.morgner
>
> http://structr.org http://www.meetup.com/graphdb-frankfurt

> <http://www.meetup.com/graphdb-frankfurt> https://splink.de
>
> --
>
>

[Niels Hoogeveen]

Here's how we handle this in multispective:

 

Each URL is stored in the Lucene index, which points to a node representing that URL. This node is a path entry and has a relationship with a node higher up the hierarchy.

 

So as a result of serving a URL /journal/travel/alaskin-natives/story-1, there will be the following path entry nodes:

 

node5: /journal/travel/alaskin-natives/story-1

node4: /journal/travel/alaskin-natives

node3: /journal/travel

node2: /journal

node1:/

 

Each of node1 to node5, has a URL propery stored in the index and may have a relationship to content. If no content relationship exists, a 404 page is presented, unless there is a redirect relationship to another path entry, in which case a redirect respone to the related URL is given.

 

Now if you add a page with the URL /journal/story-1, a new path entry node is created for that URL, with a NEXT_IN_PATH relationship coming from the node representing the URL /journal.

 

By following the NEXT_IN_PATH relationships coming into the path entry node related to your request, you can even create a breadcrum, with links to intermediate path entries if they have an associated content item.

 

Niels

[Joshua Anderson]

@Niels Hoogeveen ==> Nice approach; When you are moving thousands of files around to different parents reiterating over those objects to update there absolute url has  large effect on resources.  I'm going to try a dynamic method where moving a node to a different node doesn't require much maintenance (pertaining to the fact we don't have to update a path_url attribute) on the node we are moving.  For example, moving a node is simply a matter of changing it's relationship and then done.

@Christian Morgner ==> iterating over these nodes could be done with query syntax right (I want to try and keep this all in the database) correct?  After splicing up the slashes and having a hierarchy of relationships we need to create a query.  Could you provide an example of a query where this is performed.  Remember it must be dynamic and route through all instances of the hierarchy.

[Christian Morgner]

If by query syntax you mean the Cypher query language, then yes, I
think it can be done by query.

Your example from before ("/journal/travel/alaskin-natives/story-1")
would translate to the following Cypher query. Note that you may need
an index (or use the auto index) to find the start node. In the
example I will assume that there is a keyword index named "nodes".

The list of URL parts is as follows:
{ "journal", "travel", "alaskin-natives", "story-1" }

So the query would be:
START a = node:nodes(name = "journal")
MATCH (a)-[:CHILD]->(b)-[:CHILD]->(c)-[:CHILD]->(d)
WHERE b.name = "travel"
AND c.name = "alaskin-natives"
AND d.name = "story1"
RETURN d

Please note that I did not actually test the query against a Neo4j
database, so there might be missing something or it might as well be
plain wrong. :)

There might even be a more elegant, recursive way to query such a
structure. Maybe you can use a single query for various path lengths
and inputs..

You can find the Cypher Query Language reference here[1].


-Christian

[1] http://docs.neo4j.org/chunked/stable/cypher-query-lang.html

> --
>
>

[Niels Hoogeveen]

The reason to separate path from content is exactly for the reason to make moving of content around paths easier.

 

Of course there is a trade off here between quick lookups and updateability of the path structure.

 

Using an index, as we do at multispective, has slightly more overhead creating/deleting paths, but it allows for fast lookups.

 

Travering over path relationships makes the overhead of creating/deleting paths very low, but can have a steep cost when doing lookups.

 

Suppose you want to look up the path:  /journal/travel/alaskin-natives/story-1

 

If we use the traversal approach, you will have to have to start with some root node, follow all outgoing NEXT_IN_PATH relationships and filter out the one named "journal", from there you follow all outgoing NEXT_IN_PATH relationships and filter out those named "travel"... etc.

 

This approach only works as long as you have few entries within a sub-path.

 

Lets's assume there are 10,000 stories within the path  /journal/travel/alaskin-natives. With the traversal approach, you will need to read all 10,000 story nodes, to determine if it's name matches the one you are looking.

 

As I said earlier, every solution to this problem is a trade off. If updating paths is the prime use-case of your application, then the traversal method will work best, at the cost of potentially slow URL look-ups. If look-up speed of a page is most important, then the indexing method works better, at the cost of a slightly more complicated updating mechanism.

 

Niels

[Joshua Anderson]

Niels, Am I understanding this correctly? Via the image. 

[Niels Hoogeveen]

That is almost what I meant, though to be complete, there would also be a node with path /journal/travel/alaskin-natives/story1, so you have an immediate lookup of the node, otherwise you will have to loop through all content nodes attached to the "alaskin-natives" node, which can potentially be a large number, if you add many stories.

 

So I would put one more node under the "alaskin-natives" node with name "story1" and path "/journal/travel/alaskin-natives/story1", then attach you story node to that added node.

[Joshua Anderson]

Honestly, I still have no clue how I am going to solve this issue.  I don't want to iterate over absolute paths to update all nodes in folder.  It just isn't right for what I'm trying to achieve.  I only want to remove a relationship and add it to another node to modify a path [each node restricted to one relationship];  Then all the paths connected to the moved one will follow.I think putting an index on the path name will speed up the query tremendously. 

I need to learn more and practice before I get into something this complex.  At the moment.  I guess I'll go over the docs because I'm obviously still confused.

Joshua

[Axel Morgner]

Hi Joshua,

I wouldn't think too much about (additional) indexing or performance.

Here are some links that might be worth reading:

http://lists.neo4j.org/pipermail/user/2010-April/003309.html
http://blog.neo4j.org/2010/02/access-control-lists-graph-database-way.html
And there's even example code: https://github.com/neo4j-examples/java-tree-traverse/blob/master/src/main/java/org/neo4j/examples/treetraverse/TreeTraverse.java

Axel

--
 
 

[Niels Hoogeveen]

Since you said, you are going to use Neo4j for a website, you will likely have external links to you pages (Google will index your site and other websites may refer to some of your entries).

 

Suppose you have a working setup, and added 10 stories, so you have the paths /journal/travel/alaskin-natives/story1 up to  /journal/travel/alaskin-natives/story10

 

Now you realize you made a typo and it should actually read "alaskan-natives" instead of "alaskin-natives".

 

You can's simply change the name of the "alaskin-natives" path-entry into "alaskan-natives", because some of the external links will refer to URLs based on the naming "alaskin-natives"

 

To service both old URLs and new URLS, you will have to maintain both your old URL paths and your new URL paths in the database anyway. The old URL paths will need their CONTENT relationship removed and instead have a REDIRECT_TO relationship to the new URL path.  

 

Unlike a filesystem on a computer, URLs should be treated as immutable, because of external indexing services, you have no control over, in which case it is perfectly reasonable to maintain an index to it yourself. Why refuse to maintain an index yourself, when the outside world does.

 

Niels

[Craig Taverner]

Travering over path relationships makes the overhead of creating/deleting paths very low, but can have a steep cost when doing lookups.

Well, that depends. The traversal of the tree is a kind of index lookup too (both are tree's). The performance of this depends on the tree depth, and numbers of child nodes that need to be checked. The lucene index on the other hand is also a tree, but one that depends on the total number of URL's indexed (generic indexes slow down with total index size). It is always possible to make a domain-specific data structure that out-performs a generic index, just as it is possible to make a bad structure. Getting the structure right depends on the queries you are likely to perform. In the 'directory structure' case a very, very common use case is to expand a single directory node (ie. list the children of a single node). This is certainly faster in the graph than in lucene, since it touches only the local graph, not the entire index. However a query from root down a long path is likely faster in lucene, since the lucene tree is optimized for that.

My vote would be to build the graph version first, allowing easy maintenance, as Joshua said, and only add the lucene index in if and when you find it become necessary (as a performance optimization).

[Niels Hoogeveen]

You are certainly right that boh a lucene index and a graph based path layout are trees, but they are also different from one another. An index tree in Lucene is balanced, while a path tree in the graph is not.  Furthermore a Lucene index is more fine-grained, since its tree is based on the lexical structure of the URL, not on the path entries in it.

 

A URL path can have an almost unlimited number of children (only limited by the maximum number of characters a URL can take and the maximum number of relationships a node can have), while an index node in the Lucene index has a fixed maximum number of children.

 

As soon as there is one path entry in a URL with many children, a Lucene index will easily outperform a graph based lookup.

 

In short, a Lucene lookup has predictable performance in the order of milliseconds, while a graph based lookup has varying response times, some slightly faster than a Lucene lookup, others much slower than a Lucene lookup.

 

As to the maintenance issue, I believe I have addressed that in a previous message in this thread, showing that there is no easy solution, since URLs have a fixed existence in the outside world.

 

Niels

[Joshua Anderson]

Interesting points; I still have some reading to carry out. Hopefully, all of you will check back later in the event you find useful information to assist in making this process dry [not repeating our-selves].  I hope all of you will share new information and innovations as it pertains to this particular topic.

Thanks

Joshua.

=]

[Joshua Anderson]

Everyone involved in this topic, I have amazing logical approach I believe will work.  We find all traversals from the root node to the end result in the current case (journal/travel/alaskan-natives/story1); we start at journal and find all traversals that result in story1, yet limit it to a certain depth.  Since this is simply for my website I won't have thousands of pages (not really worried about the resources on this query).  I don't know how to write this in cypher/gremline but I'm assuming we start at journal find relationships of HAS_CHILD_NODE that end in story1 at x - a given number - depth  who's parent is alaskan-natives. so the search only uses the first, last, and parent node of last.  I think this would work any person who can show me a query you are awesome.

I know someone will bring up the fact that this will use a lot of resources; to address this, would storing the names of pages inside a relationship (HAS_CHILD_NODE) property help solve issues with heavy queries?

Joshua

[Niels Hoogeveen]

Your solution may be faster, but it may also be slower and is certainly not lineair.

 

Take eg. /journal/travel/alaskan-natives/story1.

 

A reverse search action would mean, asking some root node to list all nodes connected to it with a name "story1", then we need to follow all "has_parent" relationships and filter out those nodes that have the name "alaskan-natives", again follow the "has_parent" relationships and filter out the ones with name "travel", finally follow "has_parent" and filter out the one named "journal" and check that it has no further outgoing "has_parent" relationships.

 

Having one root node with relationships to all pages, is likely to be slow, so you may need an index on that, though it will remain slow if you have many pages with the same name.

 

As you see, search is not linear. The only thing that is linear is establishing the URL of a given node, then you can simply follow the relations from right to left.

 

Niels

On Thursday, September 6, 2012 10:26:00 PM UTC+2, Peter Schrammel wrote:

Stupid question:

Why don't you put the relation the other way round (has_parent) and start from the right side of the url?

Traversal time would not depend on number of child nodes but be linear withe the parts of the Url.

[Mattias Persson]

Perhaps traversing both from the left side and the right side would yield fewer relationships visited. Something like (if going for embedded traversal):

    Index<Node> index = db.index().forNodes( "index" );
    String[] parts = fullPath.split( "/" );
   
    TraversalDescription sideBase = traversal( NODE_PATH ).relationships( Types.PARENT );
    for ( Path path : bidirectionalTraversal()
        .startSide( sideBase.evaluator( new PartEvaluator( parts ) ) )
        .endSide( sideBase.evaluator( new PartEvaluator( reversed( parts ) ) ) )
        .traverse( index.get( "name", parts[0] ), index.get( "name", parts[parts.length-1] ) ) )
    {
        // Here's your match
    }

with the PartEvaluator#evaluate method simply like:

    return path.length() < parts.length && path.endNode().getProperty( "name" ).equals( parts[path.length()] ) ?
        INCLUDE_AND_CONTINUE : EXCLUDE_AND_PRUNE;

2012/9/7 Niels Hoogeveen <nielsh...@gmail.com>

--
 
 

--
Mattias Persson, [mat...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com