TinkerPop 3.0 work cycle begins

[Michael Klishin]

Now that we have new Ogre, Archimedes and Titanium releases on Clojars [1][2][3],
it's time to start the TP 3.0 cycle.

I've bumped Archimedes and Ogre to 3.0.0-SNAPSHOT. I think the next steps
should be to fold the two into Ogre. Any takers?

Another task we can't forego is putting together a list of new and changed
things in 3.0 that we want to support/support better/make more prominent/etc.

I feel that Stephen is the most knowledgeable person in that area ;)

Then we can begin looking into what improvements Titanium needs before
we can put the 1.0 tag on it. 

1. https://clojars.org/clojurewerkz/archimedes/versions/2.5.0.0
2. https://clojars.org/clojurewerkz/ogre/versions/2.5.0.0
3. https://clojars.org/clojurewerkz/titanium/versions/1.0.0-beta2
--
@michaelklishin, github.com/michaelklishin

[Stephen Mallette]

I imagine it will take a bit to stabilize Ogre/Archimedes over the new interfaces in TP3.  I created a "tp3" branch in Ogre as a place to start this effort.  

https://github.com/clojurewerkz/ogre/tree/tp3

I think that getting Ogre running the various TP3 test suites, the way gremlin-groovy, gremlin-scala, etc. do, is a good way to get rolling.  I assume that same pattern should apply here in clojure and helps Ogre in two ways:

1. It ensures that Ogre is compatible as a wrapper to TP3 across a wide variety of scenarios

2. Ogre gets an entire battery of tests for free which prevents the need for coming up with tons of different test cases.  That's not to say that Ogre might not need its own tests, but the TP3 test-suite can go a long way to ensuring high confidence in the library and that all TP3 features are covered.

I'll work with Victor to see if we can get the basics of that working.  If we can do that much, it should be pretty easy for us to collaborate with Ray and others who might want to contribute.

> I feel that Stephen is the most knowledgeable person in that area ;)

Michael, I wonder what makes you say that. :)  There are so many new pieces, I don't quite know where to start, however, I think that if we get the TP3 test suite running we will surface the biggest of those changes pretty easily.  From there I can point out the other areas where we will want better clojure interop with the TP3 API.

As for the latest news from TinkerPop Land , we intend to release TinkerPop 3.0.0 M3 sometime this week.  We're getting very good feeback from implementers (graph database vendors, gremlin flavor implementers, and driver/plugin developers) and are making adjustments based on their feedback on these milestone releases.

Best regards,

Stephen

[Michael Klishin]

On 29 September 2014 at 01:48:14, Stephen Mallette (spmal...@gmail.com) wrote:
> I think that getting Ogre running the various TP3 test suites,
> the way gremlin-groovy, gremlin-scala, etc. do, is a good way
> to get rolling. I assume that same pattern should apply here in
> clojure and helps Ogre in two ways:
>
> 1. It ensures that Ogre is compatible as a wrapper to TP3 across
> a wide variety of scenarios
> 2. Ogre gets an entire battery of tests for free which prevents
> the need for coming up with tons of different test cases. That's
> not to say that Ogre might not need its own tests, but the TP3 test-suite
> can go a long way to ensuring high confidence in the library and
> that all TP3 features are covered.
>
> I'll work with Victor to see if we can get the basics of that working.
> If we can do that much, it should be pretty easy for us to collaborate
> with Ray and others who might want to contribute.

Sounds good. I can assist you with folding Archimedes into Ogre in the 2nd
half of the week.

I'll also publish a post announcing the releases and the move to TP3,
Archimedes merging into Ogre, etc. 
--
@michaelklishin, github.com/michaelklishin

[Andrew Fitzgerald]

I'd like to help out if possible, but I'm not really sure what the best approach is. Just put in a (naive?) PR which just bulk copies the archimedes code into ogre, and modifies the project to get the configs to play nice with each other.

Would it make more sense to integrate the two more cleanly first, then switch to gremlin3, or just start immediately working on integrating the new gremlin structure?

[Andrew Fitzgerald]

PR build fails on Java 6 due to archimedes test dependency on Titan.

It probably makes sense for Titan to not be a dependency (neo4j is the tinkerpop reference implementation.

Also, I believe that tinkerpop3 requires java 8.

[Michael Klishin]

 On 30 September 2014 at 07:10:32, Andrew Fitzgerald (andrewcf...@gmail.com) wrote:
> I'd like to help out if possible, but I'm not really sure what
> the best approach is. Just put in a (naive?) PR which just bulk
> copies the archimedes code into ogre, and modifies the project
> to get the configs to play nice with each other.

The configs? We need to rename namespaces, I assume.

> Would it make more sense to integrate the two more cleanly first,
> then switch to gremlin3, or just start immediately working on
> integrating the new gremlin structure?

Integrate Archimedes first. We'll also need to update Archimedes README to say
it has been merged into Ogre.
--
@michaelklishin, github.com/michaelklishin

[Michael Klishin]

 On 30 September 2014 at 07:15:37, Andrew Fitzgerald (andrewcf...@gmail.com) wrote:
> PR build fails on Java 6 due to archimedes test dependency on
> Titan.
>
> It probably makes sense for Titan to not be a dependency (neo4j
> is the tinkerpop reference implementation.
>
> Also, I believe that tinkerpop3 requires java 8.

Only test on Java 7+ or even 8 then. I'm fine with testing Ogre against Neo4J if
Stephen confirms it is the reference implementation.
--
@michaelklishin, github.com/michaelklishin

[Ray Miller]

Most of the tests can run against Tinkergraph-Gremlin. It's only the integration tests and transactional tests that need Titan (or another engine that supports transactions).

Ray.

[Ray Miller]

On 27 September 2014 16:34, Michael Klishin <michael....@gmail.com> wrote:

Now that we have new Ogre, Archimedes and Titanium releases on Clojars [1][2][3],
it's time to start the TP 3.0 cycle.

I've bumped Archimedes and Ogre to 3.0.0-SNAPSHOT. I think the next steps
should be to fold the two into Ogre. Any takers?

We have a pull request from Andrew, but I've asked for the namespaces to be renamed before considering a merge.

 

Then we can begin looking into what improvements Titanium needs before
we can put the 1.0 tag on it. 

I'm keen to see a 1.0 Titanium release with Titan 0.5.0 and Blueprints 2.5.0, as that would already be really useful; it's likely we'll need to maintain 2.5.0.x Archimedes and Ogre branches alongside this before making a big switch-over when Titan itself targets TinkePop3.

If you agree this is a good idea, I'll focus on updating the Titanium documentation and dealing with any bugfix / feature requests that need to be done before a 'final' release against Titan 0.5.0.  What do you think?

Ray.

[Michael Klishin]

 On 30 September 2014 at 11:22:46, Ray Miller (r...@1729.org.uk) wrote:
> I'm keen to see a 1.0 Titanium release with Titan 0.5.0 and Blueprints
> 2.5.0, as that would already be really useful; it's likely we'll
> need to maintain 2.5.0.x Archimedes and Ogre branches alongside
> this before making a big switch-over when Titan itself targets
> TinkePop3.
>
> If you agree this is a good idea, I'll focus on updating the Titanium
> documentation and dealing with any bugfix / feature requests
> that need to be done before a 'final' release against Titan 0.5.0.
> What do you think?

OK, go ahead.
--
@michaelklishin, github.com/michaelklishin

[Andrew Fitzgerald]

I think that getting Ogre running the various TP3 test suites, the way gremlin-groovy, gremlin-scala, etc. do, is a good way to get rolling.  I assume that same pattern should apply here in clojure and helps Ogre in two ways:

1. It ensures that Ogre is compatible as a wrapper to TP3 across a wide variety of scenarios

2. Ogre gets an entire battery of tests for free which prevents the need for coming up with tons of different test cases.  That's not to say that Ogre might not need its own tests, but the TP3 test-suite can go a long way to ensuring high confidence in the library and that all TP3 features are covered.

I'll work with Victor to see if we can get the basics of that working.  If we can do that much, it should be pretty easy for us to collaborate with Ray and others who might want to contribute.

Have you been able to get anywhere with this?

I did some work on merging Ogre and Archimedes, and think it's about ready to start working on TP3 integration.

[Michael Klishin]

 On 5 October 2014 at 00:53:42, Andrew Fitzgerald (andrewcf...@gmail.com) wrote:
> I did some work on merging Ogre and Archimedes, and think it's
> about ready to start working on TP3 integration.

+1
--
@michaelklishin, github.com/michaelklishin

[Stephen Mallette]

Yeah - those pull requests were good stuff.  Thanks Andrew.  

Victor and I pulled back from the repo while you were actively making those big changes.  As we saw things quiet down the last few days with respect to pull requests, we started talking yesterday and today on an approach to getting to Ogre to TP3.  Victor started experimenting in his fork on it a few hours ago.  Unfortunately, i think the code base is going to look kind of ugly for a while as we make the transition.  TP3 came with some breaking changes to accomplish the things that it does.  Anyway, we're trying to come up with the fastest way to get Ogre building against TP3 so that we can get past the worst of the breaking change which should then hopefully allow more collaboration and pull requests.

We'll post our progress later in the week.

Stephen

--
@michaelklishin, github.com/michaelklishin

--
You received this message because you are subscribed to the Google Groups "Clojure Titanium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-titani...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Victor Su]

As a follow up to Stephen's message I wanted to provide a progress update.  We've been able

to convert some of the classes to work with TP3, but are currently working on figuring out how

to convert the pipeline to use traversals.  This is the next big hurdle and until it is completed,

the repo will still be in a broken state.

-Victor

[Michael Klishin]

 On 9 October 2014 at 04:15:12, Victor Su (vsu...@gmail.com) wrote:
> As a follow up to Stephen's message I wanted to provide a progress
> update. We've been able
> to convert some of the classes to work with TP3, but are currently
> working on figuring out how
> to convert the pipeline to use traversals. This is the next big
> hurdle and until it is completed,
> the repo will still be in a broken state.

Thank you for the update, Victor!
--
@michaelklishin, github.com/michaelklishin

[Stephen Mallette]

Hi all,

Just wanted to provide another quick update on Ogre, since Victor's post last week.  We've been collaborating in Victor's fork of Ogre and have made what we think is some good progress.  We have just one more test case to sort out before Ogre is on his feet.  In getting to this point, we've dropped a few bits that were no longer relevant or had drastically changed given TP3 APIs.  Despite all the change we've had to implement, I don't believe we've lost the spirit of what Ogre was doing or the way in which it was doing it.  I think that once we sort out this final test issue, we will look to tidy up a bit (we still have a bit of a mess organizationally with respect to the archimedes/ogre merge and the semantic differences between TP2 and TP3) and offer a pull request back to the main repository.  I think that should make it possible for more collaboration to happen, as major renaming and shifting should hopefully be settled and tests cases should be operational to help protect us a bit during the continued development against TP3.  

As far as next steps after that, I was thinking that when the code is merged back to the Clojurewerkz master, I could try to outline things that need to be implemented based on what I've learned from tinkering with Ogre these past few weeks and from what I know about TP3 from developing it for the past year.  Does that sounds like a good approach?  If so, would it also be a good idea to do that outline by creating issues in the Ogre issue tracker?  In that way, if people want to jump in and help with Ogre they could do so without us all stepping on each other. Does that sound like the right way to organize things going forward?  Other suggestions?

Best regards,

Stephen

--
@michaelklishin, github.com/michaelklishin

--
You received this message because you are subscribed to the Google Groups "Clojure Titanium" group.

[Michael Klishin]

 On 17 October 2014 at 02:28:13, Stephen Mallette (spmal...@gmail.com) wrote:
> As far as next steps after that, I was thinking that when the code
> is merged back to the Clojurewerkz master, I could try to outline
> things that need to be implemented based on what I've learned
> from tinkering with Ogre these past few weeks and from what I know
> about TP3 from developing it for the past year. Does that sounds
> like a good approach? If so, would it also be a good idea to do that
> outline by creating issues in the Ogre issue tracker? In that
> way, if people want to jump in and help with Ogre they could do so
> without us all stepping on each other. Does that sound like the
> right way to organize things going forward?

All of this makes sense. After TP3 (preview?) is merged, we'll add you and Victor to the repo, too,
to make sure you can close issues, merge pull requests, and generally
feel in control :)
--
@michaelklishin, github.com/michaelklishin

[Andrew Fitzgerald]

Sounds good to me as well.

[Ray Miller]

Hi Stephen,

On 16 October 2014 23:28, Stephen Mallette <spmal...@gmail.com> wrote:

As far as next steps after that, I was thinking that when the code is merged back to the Clojurewerkz master, I could try to outline things that need to be implemented based on what I've learned from tinkering with Ogre these past few weeks and from what I know about TP3 from developing it for the past year.  Does that sounds like a good approach?  If so, would it also be a good idea to do that outline by creating issues in the Ogre issue tracker?  In that way, if people want to jump in and help with Ogre they could do so without us all stepping on each other. Does that sound like the right way to organize things going forward?  Other suggestions?

Sounds like you're making good progress. Yes, it would be great to see outstanding tasks as Github issues, especially as more people start collaborating on the project.

Ray. 

[Stephen Mallette]

Hi all,

Victor and I think we are at a point where we could get development happening again in the Clojurewerkz master branch.  We are not complete with respect to the TinkerPop3 migration and there is much still left to do, however, we do believe that we are past the point of significant code upheaval which would have made it hard for others to collaborate who wanted to.  Some of the major remaining items include:

+ improving transaction support - the TP3 model is more robust than TP2

+ complete wrapping of all the traversal steps - we still have a good number of outstanding steps including some "harder" ones like match() step which might require some deeper thought on how to make it nice with clojure

+ figure out how to leverage the gremlin test suite to test Ogre - my clojure skills seem to fall short there - if it's not possible, then we'll likely want to manually migrate the tests to ogre to ensure good coverage (or just write up our own tests).

i think completing those major items (and a few other odds and ends) would make Ogre suitable for a release candidate to stay in line with the TinkerPop3 GA.  As an update there, we just released M4 and are continuing to get feedback from many vendors on the API.  Despite all the feedback we are feeling like we are seeing things solidifying and most of the core structures/abstractions remain unchanged in all this.  

So, all that said, as we were in general agreement with respect to how to proceed (given the previous emails in this thread), the next steps would be for:

1. Victor completes any final odds and ends he's currently working on

2. He issues his fatty pull request back to Clojurewerkz and it gets merged to master

3. Development for TP3 support will continue in the Clojurewerkz repository

4. I will create Ogre GitHub issues for remaining work.  I'll try to do them in small workable chunks that are easy for folks to dive into

Sound good?

[Michael Klishin]

 On 25 October 2014 at 00:53:07, Stephen Mallette (spmal...@gmail.com) wrote:
> 1. Victor completes any final odds and ends he's currently working
> on
> 2. He issues his fatty pull request back to Clojurewerkz and it
> gets merged to master
> 3. Development for TP3 support will continue in the Clojurewerkz
> repository
> 4. I will create Ogre GitHub issues for remaining work. I'll try
> to do them in small workable chunks that are easy for folks to dive
> into
>
> Sound good?

Yup. Please tell me your and Victor's github usernames, I'll add you to the
repo after the merge.

Thank you very much for your work!
--
@michaelklishin, github.com/michaelklishin

[Stephen Mallette]

cool....mine is 

https://github.com/spmallette

but i think you've already added me. thanks.

[Victor Su]

Hi all,

I've completed my changes and have submitted a pull request back to master.

Michael, my username is vsu:  http://github.com/vsu

It's been great working on ogre.  Looking forward to moving it along further!

-Victor

[Michael Klishin]

 On 25 October 2014 at 17:00:13, Victor Su (vsu...@gmail.com) wrote:
> I've completed my changes and have submitted a pull request
> back to master.
>
> Michael, my username is vsu: http://github.com/vsu
>
> It's been great working on ogre. Looking forward to moving it
> along further!

Thank you, Victor!

Merged. You should be able to push to clojurewerkz/ogre now. Feel free to create
a branch (e.g. `tp3`) there and continue there.

Also, feel free to file and comment on issues, etc. It's very exciting to see Ogre
moving to TP3 in such a small amount of time.
--
@michaelklishin, github.com/michaelklishin

[Michael Klishin]

 On 25 October 2014 at 17:00:13, Victor Su (vsu...@gmail.com) wrote:
> Merged. You should be able to push to clojurewerkz/ogre now.
> Feel free to create
> a branch (e.g. `tp3`) there and continue there.

…or simply work on master, since it already has been switched to TP 3.0 :)
--
@michaelklishin, github.com/michaelklishin

[Stephen Mallette]

I've added what I think are the major remaining issues for Ogre/TP3 to the GitHub issue tracker.  I tried to break them down into small manageable bits to work on.  Please let me know if any of the issues aren't clear or otherwise need more description.

Thanks,

Stephen

--
@michaelklishin, github.com/michaelklishin

[Andrew Fitzgerald]

I'm going to bump this and hold of on making the inevitable "Future of Titanium" post.

I'm not the most experienced Clojure programmer, but I've been following Ogre development for a while and have made a couple contributions.

Not entirely sure of the best way forward, but this is my feedback so far.

As an intermediate Clojure programmer, the number of macros, and the nested usage of them makes it pretty difficult to figure out the execution flow, especially given the complexity/uniqueness of the TP3 api.

The definition of inV/outV and the aliases in ogre.core is very clever, but probably much more complicated than necessary.

There also appear to be some fairly large issues with the current api. I don't believe there is currently an equivalent of g.V(),

with the closest being get-all-vertices, which eagerly fetches all vertices and puts them into a vec.

I'm not attempting to trash the contributers or the project, I mostly wanted to bring attention to the current barriers to entry.

A wish list for me at the moment would be a more simplified code base that tracked closer to the TP3 api, and as much wholesale copying from the TP3 test suite as possible.

Any thoughts or comments?

Cheers,

Andrew Fitzgerald 

[Andrew Fitzgerald]

Nevermind about the g.V() part!

[Stephen Mallette]

Hi Andrew, I generally agree with your sentiments with respect to the Ogre.  I would say that Ogre is going through a slow evolution toward your wish list as you called it:

1. A more simplified code base closer to TP3.  

2. Better use of the TP3 test suite - that one is close to my heart because I spent a ton of time on the TP3 test suite for vendors and after success with groovy and scala, I thought it would cleanly apply to any JVM language and when I got to clojure interop, i was like uhhhhhhhhh............perhaps there are smarter clojure folks than me who can figure out how to make it work.

So, how does this slow evolution get us there:

1. It started when Archimedes got combined into Ogre - over time we've slowly cleaned out duplicate functions and reorganized to make it all just Ogre.  I'm still not so sure this is completely done.

2. Get Ogre running against the latest TP3 milestone M7 without too much change to the original spirit of Ogre (e.g. keep the clever Ogrish deviations from TP3 and other helper functions without too much change).  

3. Start releasing Ogre milestones in step with TP3 milestones as gremlin-scala has been doing successfully since M1.  My guess is that the last milestone will be M8 as TinkerPop will then be under Apache and we will likely release GA.

4. Then, with Ogre community involvement (perhaps for Ogre's GA against TP3 GA), start to look at the Ogre API itself and:

  a. remove some indirection in the API - maybe inV is just inV 

  b. remove functions that might not be so useful - make Ogre a bit more lean

  c. add new functions that showcase clojure strengths over the TP3 API - note the "sugar plugin" for TP3 and how groovy get's leveraged in this fashion: http://www.tinkerpop.com/docs/3.0.0-SNAPSHOT/#sugar-plugin

5. Beyond that - the TP3 API is expansive.  There are many areas where Ogre might be extended into, like Strategies, other methods of IO, a driver for Gremlin Server, etc.

So - that's has been the thinking Victor and I have had on where Ogre should be heading since we started tinkering on it to get it running for TP3.  It would be great to hear what others think on this approach to turning Ogre into a beautiful clojure representation of the TP3 API.

Thanks,

Stephen

[Andrew Fitzgerald]

Just wanted to poke this thread and see if there's anyone else out there who is interested on getting this moving again.

[Stephen Mallette]

Hi Andrew, I think you've been following the TP3 code base.  What do you think about Ogre just being a wrapper over the Traversal API (with enough support over the Core Graph API to instantiate it)?  The reason I say this is because every time I think about Ogre I struggle with the depth it goes to wrap the various interfaces TinkerPop interfaces when TinkerPop 3 mostly espouses use of just the Traversal API for most all operations.  A solid implementation of Ogre over just that aspect of the code base with nice idiomatic clojure seems like it would vastly simplify the code on the Ogre side.  Anyway, do you have any thoughts on that?

[Andrew Fitzgerald]

Hey Stephen, Yeah, I'm fairly up to date on TP3 stuff. I definitely agree with you that a more minimal wrapper around the Traversal API would probably be the most useful/maintainable version of Ogre.  I had hoped to be able to nudge ogre more closely in that direction as it migrated to TP3, but the 2-3 migration has been a huge wall every time I've forked/branched and taken a go at it.  I feel like adding more features/abstractions on top of tp3 might make sense somewhere down the road, but keeping all the currently existing ogre features is a non-starter if we want a working TP3 version to happen any time in the near future.

I'm not sure what the best approach to take at the moment is, but my gut feeling is that some sort of rewrite is in order, possibly driven by the traversal portion of gremlin-test.  The primary caveat with that approach is making sure that the creator/maintainers don't feel like they're having their toes stepped on, and making sure that everything is done in a respectful manner.

I'd love to hear thoughts from anyone else who's been lurking on the list.

-Fitz

[Stephen Mallette]

>  possibly driven by the traversal portion of gremlin-test

You'd made some headway on that some time back - I don't quite recall the blocker though. do you think that you had enough of a model to figure it out?  

fwiw, gremlin-scala doesn't bother with the test suite.  michael pollmeier listed out a bunch of reasons why when I'd asked, but I suppose that what stood out for me was that it sounded like he didn't feel it necessary to re-test all of gremlin for his "wrapper" - he felt he could be more strategic in developing his own tests.  I was kinda bummed as we'd try to gear it up to be easy for "language providers" to hook into it, but it must have been a burden on gremlin-scala somehow anyway. perhaps it wouldn't hurt ogre to go the same way.  I don't know scala so well but if you do, you might take a look at what he's done over there to see if it's a pattern worth copying.

[Andrew Fitzgerald]

Not really blocked by anything, there's just a whole lot of wiring required to get it up and running.

Just put in a PR (https://github.com/clojurewerkz/ogre/pull/65) which uses gremlin-test for the has step (as that seemed like the most straightforward place to start).

There's probably a better pattern or some macro magic that could make it a little cleaner, but it's a starting point.