Titan's Future

[Effy]

Hi,

I'm becoming worried regarding the future of Titan, and would appreciate feedback from either Titan's devs (DataStax nowadays, I guess), or the community.

For the last 6-7 months, there haven't been any significant development (or even version minor version updates, in order to support the latest TinkerPop releases).

This forces me to re-think whether I (or anyone else for that matter) should continue relying on Titan for my applications.

Is it time to abandon the ship and leave Titan for another Graph Database solution (TP based, or alternatives)?

Does anyone know what are DataStax plans? Are they planning on maintaining Titan? Are they planning to release some commercial product based on Titan? If so, did anyone have any clue regarding their schedule?

Thanks,

Effy

[HadoopMarc]

I share your worries. Some information is provided here:

http://www.datastax.com/products/datastax-enterprise-graph

Cheers,   Marc

[Adam Phelps]

On 4/13/16 12:32 AM, HadoopMarc wrote:
> I share your worries. Some information is provided here:
>
> http://www.datastax.com/products/datastax-enterprise-graph

Yeah, I have to admit I'm having similar concerns over here, especially
since we solely use HBase (not just for Titan, but for massive amounts
of other data) and would prefer not to move to yet another clustered
product we need to figure out how to maintain.

I'm also a little put off by this products top level marketing blurb:

'DataStax Enterprise Graph is the first graph database fast enough to
power customer facing applications, capable of scaling to massive
datasets and advanced integrated tools that power deep analytical queries.'

As we've been directly powering customer facing applications from Titan
on a pretty massive dataset for a while now.

If Titan isn't going to be maintained going forward by the original dev
team (and while I'd like to have the time to actively develop on it, I
know my team over hear doesn't have the capacity to do so), it would be
good to hear that explicitly stated for our planning purposes.

- Adam

[Dylan Bethune-Waddell]

Hi all,

Beyond the plans of the original dev team with respect to maintaining Titan, it would be nice to hear from the Titan/TinkerPop devs about what Titan might need going forward so that the community could pick some of the work up in a targeted way. Stephen/Marko/David have sprinkled breadcrumbs out there that the community could follow up on in terms of issues/bugs both for what Titan needs for a 1.1 release and how to upgrade to TinkerPop 3.2.x for providers. David mentioned at one point that Titan would need an update to its storage model for performance, and I'm sure everyone from DataStax have probably picked up some high level concepts from working on DSEG that would hopefully be okay to divulge. 

I think a discussion focused around distributing the workload enough so that Titan can stay up to date with TinkerPop and Marko/Matthias/Dan/David/Stephen etc. can chime in as the voice of experience instead of having to do all the work themselves - I'm working on upgrading Fulgora and some other parts of the Titan code so that it will build against 3.2.x, and also pulling in David's changes which bump a lot of dependency versions of particular note thrift, Cassandra, and HBase:

https://github.com/graben1437/titan1withtp3.1

And the PR from James Cahill which updates the ES version to the 2.x line and the Lucene version to 5.x

https://github.com/thinkaurelius/titan/pull/1153

I think all the pieces are there to just kind of knock Titan up to parity with TinkerPop development which might make it more appealing for the DataStax/TinkerPop guys to drop into the Titan codebase when they get a spare minute. The latest and greatest versions for HBase/Cassandra/ES/Lucene are also there for storage backends, with Jan Kotek of MapDB having said a more license friendly and better performing replacement for BerkeleyDB in Titan-MapDB will be coming for Titan 1.x soon, and for OLAP it might be worth it to get a CassandraInputRDD and CassandraOutputFormat going as well as write a TitanBulkLoader with some configurations to throttle BulkLoaderVertexProgram when the ES backend is getting a lot of MergesInFlight which seems to cause Spark executors to fail for me when loading dense.

Also might be worth pinging the Amazon guys and asking if they have any updates or thoughts they would be willing to share with the community based on their experience with Titan-DynamoDB - I've seen IBM and some others do this with Jupyter and other open source projects, so it might not be a totally crazy thing to propose to them...

Just some thoughts,

Dylan

[David]

Hi Everyone,

I have a whole list of "improvements" that we could discuss and prioritize with regard to moving Titan into 2016 and beyond.

I will be pushing a few more things to my Titan repo in the next few weeks which I hope are helpful including a
TinkerPop 3.1.2 upgrade, in case that matters to anyone.

But the "project" conversation happening here is the most important. Kudos for bringing it up.

What do we want as a "Titan" ...er...better to say "graph" community ?
May I suggest the following list:

a) we want a technically relevant and vibrant open source graph database solution that works well with
    our big data environments - hbase, hdfs, spark graph frames, spark olap, kafka, cassandra, maybe
    even map/reduce etc.  That is why we started here in the first place.

b) we want some sort of continuity, or as much as possible, with our Titan investment.

c) we want open governance going forward, so we don't get stuck or controlled like we are right now
     as a community.  Titan was never open governance.  Don't hold your breath (and be careful
     about chasing shiny objects in this space, they are just fishing lures with sharp hooks)

d) we want great partners and community members ...like we have here

Assuming the above is close to accurate for our community...how do we get there ?
There are options !

1) private "forks" of Titan.  This isn't really community.  It is what we have now.
     It is tough to even provide support for users with this sort of thing.

2) "public" or community forks of Titan.  This is a way to re-boot and get going again. 
     This provides the most continuity.  If you take a hard look at Titan and the list of things
    needed to bring it to 2016 (reference to my list above) do we really want to invest here ?
    It is a lot of work, and even disruption.

3) We agree as a community to collectively move efforts to S2 Graph or Gaffer (not open governance...we
    don't want that again) and try to make them become our new Titan.

4) Titan-Nextgen.  Not easy.  Big up side.  A project that achieves a-d (and more)
    with a look and feel of Titan to ease transition, adds new things, and includes modern internals.
    Wouldn't it be a surprise if something like this, with running code, turned up at the Apachecon
    Big Data conference in a few weeks ?

5) Combine thoughts from 3 and 4.

[Stephen Mallette]

I can't really answer these questions very well, so I've pointed Matthias at this thread.

David,  regarding:

> I will be pushing a few more things to my Titan repo in the next few weeks which I hope are helpful including a TinkerPop 3.1.2 upgrade, in case that matters to anyone.

What kind of testing are you doing on your fork of the repo?

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/d79518c3-19d1-47b2-841b-741b023f4528%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dennis]

Hi all,

Politics aside... I think the product vision for Titan is quite murky right now. Originally, it's about providing a distributed graph database. Now, what's next?

Under TP3, it's up to some database vendor to provide gremlin-X as a Graph implementation. If we are trying to implement these "connectors" against many backends, we won't have the best quality across the board. For example, how do you ensure highly-performant ACID transactions? This is a big challenge for all graph databases. If we are trying to do that, we will need to be more "opinionated" about our stack.

Sure, everybody wants flexibility. But we get flexibility with no governance at all. But if we want better quality, like higher performance and stronger guarantees, we will need to re-architect Titan. And people are already doing that privately. 

If we can get some co-design happening and we find some worthy goals, then we will have the basis to work out a governance structure.

Dennis

在 2016年4月14日星期四 UTC+8下午11:55:00，David写道：

[Venkata Phani Kumar Mangipudi]

Hi All,

In between there were even talks about folding titan into apache? is this even possible looking at the current state of affairs? 

How difficult is this activity going to be, can we request for a fit gap analysis done by the Tinkerpop team to help us understand our situation better (because they are currently well versed with the apache process) Stephen is actively involved in answering the questions on Titan. 

Stephen, can we request you to please help here? 

Can Titan be trimmed down to support few (say 2) most used back ends to let it run fast and get it into apache? and later add the support for other back ends? Strategy employed in movie 300 at the end to hit the target!

Hypothetically if we assume the 2 back ends are going to be Cassandra, HBase (as they were mentioned in above discussions). How complex / easy it is going to be to fold this into Apache? What are the activities involved and how the community should go about doing this?

There could be another strategy to "Complement Tinkerpop - Example Implementation of Tinkerpop"! 

1. Trimming any overlapping functionality / support from Titan which already exists in Tinkerpop or is in near future roadmap of Tinkerpop 

2. Bring only those modules / backends which have a commitment from the current users to support the apache inclusion!

Regards,

Phni

[Dylan Bethune-Waddell]

I like what David had to say about why Titan was more appealing than alternatives in the first place. A technically relevant piece of FOSS with a strong community that brought a graph solution into our existing or desired big data environments. Titan was for many a big investment to learn and integrate into those environments, though friction seems to have varied a lot here. To my mind, keeping the community we already have around it from fragmenting and moving on seems like the worthiest goal while the how/who/where/what of revitalizing the project are sorted. A collaborative design document as Dennis suggested, and trying on the Apache model with some guidance from the TinkerPop as Phni suggested, both appeal to me as ways to structure the process of sorting it out. I remember seeing something from Matthias or Stephen a while back about how Titan would need more committers to incubate at Apache, so this might be good timing.

The fact that Titan's technical relevance is decreasing (with respect to "technical debt" at least, TP3/Cassandra/HBase/ES/Hadoop and other dependency versions etc.) while "murkiness" is increasing - whether those changes are truly appreciable or simply perceived because DSEG and TinkerPop are higher priority right now for the DataStax/TinkerPop guys - is to my mind the biggest risk to the investment we have made in Titan and what should be averted immediately by the community. Pulling David's build into the main Titan repository, TinkerPop 3.2.x support, updating and merging James Cahill's commit for ES 2.x and Lucene 5.x support, and solving the github issues Stephen has listed in this forum as blocking the Titan 1.1 release are the most impactful things I can come up with that can be done to this end. From working on each of these things myself at some point, with the exception of the "blocking" issues, I don't think any of them are actually that hard - to me they have been, because I'm not a very good developer yet, but at Marko's pace and level of understanding for example I would be surprised.

Conjecture here, and not an attempt to pander to anyone - letting Titan and its community fragment or dissolve would be bad for DSEG. Doesn't a fading and broken up Titan community reduce the pool of ready-and-willing DSEG customers, and kind of go against the Cassandra / DataStaxEnterprise model that has already proven successful at DataStax? What do you gain from saying that DSEG is based on Titan, if people are talking about jumping the Titan ship? And this works in reverse in my eyes - what does the Titan community gain if the rockstar team at DataStax no longer has a strong stake in the community codebase? 

I think continuity of the initial investment everyone made in Titan, including Aurelius, DataStax, and the TinkerPop, is best served by an outcome where Titan is re-FOSS'd through some kind of collaborative move towards a more active and possibly Apache project, and simultaneously revitalized in the repo it lives in right now. As everyone has been pointing out, we must de-murk this thing and see if and how Titan can renew the technical relevance and role we all bought into it for in the first place - but publicly, as David/Dennis point out.

Kind Regards,

Dylan

[Ranger Tsao]

+1

在 2016年4月17日星期日 UTC+8上午8:19:24，Dylan Bethune-Waddell写道：

[Bezalel Lim]

++1!

[Sinus Rhythm]

titandb++

[Dylan Bethune-Waddell]

Hi guys,

I just submitted a PR based off the titan11 branch that at the very least gets Titan to build with TinkerPop 3.2.0-incubating and for most modules gets past mvn clean install

https://github.com/thinkaurelius/titan/pull/1312

The HBase test suite had a Zookeeper issue, the Hadoop 1 module failed all tests, and there was one failed Solr module test which looks like its an artifact of the test environment not being what it should be (like the HBase one) as it just complains about there not being enough Solr nodes.

This PR is mainly intended to be a work in progress that highlights the points in the codebase that are relevant to an effort to properly update Titan to the newest version of TP3 and thereby (hopefully) engage some people who will have a better inkling than I as to how to do it right in steering the PR as needed. Best case scenario is that if this gets fixed up and merged, the outstanding blockers for a Titan 1.1 release will become more appealing to solve for any of us that have time and we will see that release sooner rather than later.

TLDR; Please take it for a test drive, let me know what's broken so I can try and fix it up.

Cheers,

Dylan

[Gavin Baumanis]

This thread was started in April - and it is now nearly at the end of July.

And there is still no word of the roadmap for TitamDB - or if there even is one.

When taken over by DAtaStax, I assumed that TitanDB would be the community-based and open-sourced factory for technology that gets used in DSE.

Where DSE would include things, like OpsCenter and "paid" support for matters that you can't work out for yourself with the help of the community - or where the timeframe of a community-based support system was too slow for the level of risk you were willing to take on-board.

I "thought" it would pretty much be - like, well, pretty much every other commercially-backed / successful open-source projects.

it is somewhat mitigated by the "start-up" program offered by DataStax for DSE... but the silence is quite worrying none the less.

How can you possibly choose TitanDB for your enterprise applications in its current environment of seemingly - no (new code) activity / no statement of the project's future by it's founders?

And this isn't the only thread on this mailing list - where clarification of the project's future has been asked and seemingly ignored.

Perhaps it is simply answered, by all the silence?

[Robert Dale]

"90% of DSE Graph is completely new code and 10% is Titan-inspired" - http://www.datastax.com/2016/04/introducing-datastax-enterprise-graph

I interpret that to mean that DSE Graph has 0% in common with Titan's codebase. I wouldn't expect any support from them.

[Grant Haywood]

If I where going to build a new product, and wanted to use the Tinkerpop ecosystem, I would consider contributing to Unipop at this point (or using one of the other TP enabled graph vendors)

https://github.com/rmagen/unipop

a younger project with less baggage (both technological and political)

either way, at the end of the day, its now up to the community (volunteers) to contribute energy/work to make progress

maybe this has not happened for titan because there is alot of baggage to ramp up on?

--

You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/6ed61e46-fb12-4180-8548-821bc166bb41%40googlegroups.com.

[Jason Plurad]

Perhaps folks have been waiting to hear back from Titan's developers, namely Matthias Broecheler and Dan LaRocque. I've put them on cc. The last mail here from Matthias: September 2015, Dan: March 2016. It is clear, and it has been said before, they are working on DataStax Enterprise Graph now (which is awesome, btw, so be sure to consider it if you're looking for an Apache TinkerPop-compliant backend).

Here's how I see it. Titan 1.0 is a gift that Matthias and Dan left for the open source community. THANK YOU, Matthias and Dan, for getting it released.

If you are interested in helping to evolve it going forward, we need to hear from you so we can get organized. Keep in mind that contributing to an open source project is not limited to just code contributions. There's code reviews, testing, mailing list, documentation, coding examples, tutorials, blogs, meetups, conferences, etc. Reply on this thread, email me directly, tweet at me directly, and let us know how you want to contribute. +1s are nice, but contributions are what make open source work.

One open issue the Titan community has is that its GitHub repository is owned by the former-Aurelius team. Stephen has been a saint (borrowing Marko's term) in terms of getting documentation and small bug fixes integrated into the `titan11` branch, but some of the more complex fixes have stalled in the queue without reviews from Matthias and Dan. I don't know what possibility there is for opening up the repo's committer access list or what kind of process there would be. In the meantime, what you can do is test out the code (definitely check out TinkerPop 3.2 support from @dylanht and Elasticsearch 2.2.3 support from @sjudeng). Try it in your environments and report back. We also need people to start learning the code and testcases and documenting the internals -- one page of doc is not enough to enable a future wave of Titan development.

Titan has racked up 3855 stars on GitHub: 1300+ stars on GitHub since the DataStax acquisition in February 2015, including 800+ stars since the Titan 1.0 release in September 2015. It seems like the interest in Titan is still there. Let's hear from all of you and start getting more contributions rolling in.


-- Jason

[Michael Woytowitz]

I did discover that IBM is using Titan as the basis for its IBM Graph offering. 
Maybe this industry giant will start to contribute and take over raising Titan now that his birth parents have abandoned him?

https://ibm-graph-docs.ng.bluemix.net/

I am still deciding if our company will continue with Titan 1.x (on Cassandra), migrate to DSE Graph, migrate to IBM Graph or look else where for our Graph database needs.

Cheers,

Michael

[Sandeep Srinivasa]

Hi guys
Just curious - if you all had the choice to start from scratch today...would you still choose Titan ?

Or would you go neo4j/orientdb/DSE/etc

[dgr...@cgnal.com]

I would stick with Titan, besides the new DSE Graph it's only scalable graph db integrated with the hadoop world, no other options available yet.

[Gavin Baumanis]

Hi there;

Are you saying that;

1. DSE is only the OTHER option that does Hadoop? or;

2. That DSE ONLY does Hadoop?

Either way, let me offer you this...

1.

Just about everyone I know is choosing Spark or some other "stream" processing technology.

The requirements for programming multiple times....

Once for the "application side" in whatever is the chosen technology stack is...

And then additionally, for Map / Reduce tasks (Hadoop) - is just too much work,

Too much hardware / too many different moving pieces.

Too many different skillsets required.

Too many staff resources (cost) required.

Unless you already have a significant Hadoop investment : Hadoop is "practically" a dead technology.

I know a couple of companies that are actually transitioning off genuinely impressive Hadoop infrastructure to "stream processing" to lessen the complexity of their technology stack.

2. 

DSE comes with Spark integration / SOLR integration, baked right in.

So it certainly isn't Hadoop centric.

(You don't have to use Spark or SOLR - they are config on/off items)

Lastly,

I am really new to Graph DB's - and was genuinely considering the use of TitanDB.

Not simply because it is free either...

I genuinely thought that Titan would end up being the community driver "to" DSE.

And so thought that all the "excitement" would be happening here.

As a result of the extremely generous "start-up" programme, offered by DataStax - I am using DSE - to future proof my technology stack.

(and let's be honest - if you have over 5 million in annual turnover - you can afford to pay for a commercial license!) 

And with the "future-proofing" hat on - really wanted to ensure that the entire technology stack lives on peer-to-peer distributed network.

(again why I was initially thinking of using Titan - because it can use Cassandra.)

If the application was simple enough that we genuinley "never" have a requirement for distributed storage / processing - I would have used Neo4j - it is stable / actively developed / has awesome amounts of documentation and examples.

As always thanks!

Gavin Baumanis

E: gavinb...@gmail.com

I, for one, like Roman numerals.

--
You received this message because you are subscribed to a topic in the Google Groups "Aurelius" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/aureliusgraphs/R0RJnvVbgCs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/a7ad1f22-46a1-476b-94e8-aea68e01d2b1%40googlegroups.com.