The future of OpenTSDB

[Frederik Kraus]

Hi everyone!

as long time (heavy) users, contributors (sort of - yes, we also have our own fork ;)) and great fans of OpenTSDB I'd like to start a discussion on the current status of opentsdb development but even more about how development can look like in the future.

Talking to people and also reading on the list I have the impression that everyone is currently working on his own fork of OpenTSDB. Most are creating their own UI, some are adding/extending functionality, others are working on different persistence options. At the same time (most) of the great work doesn't find its way back into the main git repo from stumbleupon leading to forks drifting further and further apart.

We could obviously let this process continue and each have our truly "own" OpenTSDB as integration costs between the (in the future) very different forks become too high. At the same time investments from others become less likely.

… or we could try to find a way of better coordination and at the same time acceleration of the development process (and allow for changing priorities of individuals)

Any thoughts?

Fred.

[Kevin Ortman]

I completely agree.  I would love to see a community-supported repository that maintains a stable branch, takes a thoughtful approach to feature add, and incorporates build/test processes to ensure quality of new features.

I would also be willing to assist with that in any way I can.  

Are there others interested?

---

Date: Mon, 7 Jan 2013 22:38:20 +0100
From: frederi...@gmail.com
To: open...@googlegroups.com
Subject: The future of OpenTSDB

[Krzysztof Kaczmarski]

Yes, indeed. Perhaps not with the repository but I can help in testing.

On 01/07/2013 11:58 PM, Kevin Ortman wrote:

I completely agree. �I would love to see a community-supported repository that maintains a stable branch, takes a thoughtful approach to feature add, and incorporates build/test processes to ensure quality of new features.

I would also be willing to assist with that in any way I can. �

Are there others interested?

---

Date: Mon, 7 Jan 2013 22:38:20 +0100
From: frederi...@gmail.com
To: open...@googlegroups.com
Subject: The future of OpenTSDB

Hi everyone!

as long time (heavy) users, contributors (sort of - yes, we also have our own fork ;)) and great fans of OpenTSDB I'd like to start a discussion on the current status of opentsdb development but even more about how development can look like in the future.

Talking to people and also reading on the list I have the impression that everyone is currently working on his own fork of OpenTSDB. Most are creating their own UI, some are adding/extending functionality, others are working on different persistence options. At the same time (most) of the great work doesn't find its way back into the main git repo from stumbleupon leading to forks drifting further and further apart.

We could obviously let this process continue and each have our truly "own" OpenTSDB as integration costs between the (in the future) very different forks become too high. At the same time investments from others become less likely.

� or we could try to find a way of better coordination and at the same time acceleration of the development process (and allow for changing priorities of individuals)

Any thoughts?

Fred.

[m...@one.com]

We've recently begun looking seriously into OpenTSDB, so I'm quite curious about what's going to happen here. We are investing in Hadoop AND out-growing our monitoring-systems, so OpenTSDB fits quite well into our current plans.

As you stated, the "official" OpenTSDB master looks somewhere between dead and on extended life-support, so any need to get patches in, would effectively force us to do a fork of our own, which we have very little desire to do.

On the upside, I've found references to work on OpenTSDB 2.0 (http://www.euphoriaaudio.com/opentsdb/new20.html), which seem to happen on https://github.com/manolama/opentsdb/tree/Scratch. The commits mention, among other, Cassandra, RabbitMQ a few times (neither of which is mentioned in the 'new20'-document), which makes me wonder where it's actually going. (And if so, is OpenTSDB 2.0 switching to Cassandra, or "just" support it as an alternate back-end?).

Then there's the main OpenTSDB-repos, which sees very little activity overall, most pull-requests seem to be closed without explanation (though some of the commits are cherry-picked, from what I could find). And only tsuna has commit-access (from what is public on GitHub), leaves an effective Bus Factor of one...

As to going forwards, I see three main options; (please correct me if I'm missing anything!)

1) Tsuna getting some serious work put in.

2) A group does a fork (possibly an existing one), effectively forking the community as well as the code.

3) Letting the community (which I *guess* is becoming large enough) take over, either completely, by taking the BDFL-approach with some semi-vague plan and have lieutenants carry that out, or some similar construct.

Of these, option 1 historically tend to burn people out (and still leave the bus factor), 2 is a variation of 1 or 3, depending on circumstances. Given this and that there is a obvious interest to put in work from multiple parties on this list, three is the very obvious choice IMHO.

As tsuna is the de-facto BDFL and it is his "baby", I guess it is ultimately up to him to make a decision.

[ManOLamancha]

On Thursday, January 10, 2013 5:18:31 AM UTC-5, m...@one.com wrote:

On the upside, I've found references to work on OpenTSDB 2.0 (http://www.euphoriaaudio.com/opentsdb/new20.html), which seem to happen on https://github.com/manolama/opentsdb/tree/Scratch. The commits mention, among other, Cassandra, RabbitMQ a few times (neither of which is mentioned in the 'new20'-document), which makes me wonder where it's actually going. (And if so, is OpenTSDB 2.0 switching to Cassandra, or "just" support it as an alternate back-end?).

For the 2.0 version, I abstracted the backend so folks could use Cassandra or other storage systems as options since we don't know what will be around in a few years. I'd hate to see OpenTSDB die out if HBase goes under. Since I haven't worked out the bugs in the Cass code, or finished the MQ stuff, I haven't doced it yet, but when I get to that part I will.

My goal my 2.0 work is to pull in all of the tweaks folks have been making in their own forks, along with stuff we need at our company, and get them ready for integration into the main branch by Tsuna or Dave Barr. I'm hoping to earn committer status if my code is good enough. So far all of my work is really rough, hence the "Scratch" branch, and within the next month I'm going to be cleaning and moving it into the root of my fork for review.

I would also like to see more committers but also more talk in the forums here about the features we're all developing. I've been trying to post additions I've been making but haven't received as much feedback as I'd like since we're all duplicating efforts. So maybe we need to figure out what everyone wants in 2.0, how the features should work, then try to figure out how we want to develop and integrate all of the changes.

[Simon Matic Langford]

I think one of the reasons for not having provided feedback personally is that since the branch is large and also commits are currently slow to make it back to OpenTSDB main, it feels like there's a high chance of it never making it back in.

It would be nice to see improvements split out rather than having a big bang approach, but I appreciate this makes life hard when adding features you need soon.

At Betfair, we've generally worked around the slowness of commits back (there are a number of improvements people are working on that we'd like to see) by running a proxy in front of OpenTSDB and implementing new features in there temporarily until those features make it into the core to save us pain if those implementations look different before they make it in.

[Andrew Purtell]

</lurk>

On Thu, Jan 10, 2013 at 2:18 AM, <m...@one.com> wrote:

As to going forwards, I see three main options; (please correct me if I'm missing anything!)

As a community you (but really tsuna) also may have the option of relicensing the code as ASL 2 and submitting this project for incubation at the ASF. I think what the ASF buys an OSS community is a neutral third party host that won't "go away" and free hosted dev infrastructure. Whether or not this is a desirable option, OpenTSDB as a project is already in a better state than others that have been in incubation and graduated: > 1 developer with >1 affiliation, an active user base, integration with other ASF projects (HBase, Cassandra).

--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)

[Elliott Clark]

I know there would be people willing to help with moving it to ASF (myself for one). So that's an option if it's wanted. 

[Andrew Purtell]

> I know there would be people willing to help with moving it to ASF (myself for one).

I should have explicitly included myself on that list, apologies. 

[Frederik Kraus]

Thanks everyone for your feedback! 

ASF sounds like a great idea to me. IMHO anything that reduces the bus factor and opens up coordinated development is a must. The great advantage of the ASF would really be the neutrality + the infrastructure to run such a project independent of individuals and organizations.

@Tsuna: What do you think? 

@ManOLamancha: We had started investing heavily into OpenTSDB (Peter was doing most of the work ;)) about a year ago, but once one gets the feeling, that patches don't find their way back into the main repo, efforts shift from "community work" to "internal development". This is exactly the point why I first started this discussion. If we all want OpenTSDB to succeed (and survive), we'll have to do it together.

[ManOLamancha]

On Thursday, January 10, 2013 5:42:51 PM UTC-5, Fred wrote:

@ManOLamancha: We had started investing heavily into OpenTSDB (Peter was doing most of the work ;)) about a year ago, but once one gets the feeling, that patches don't find their way back into the main repo, efforts shift from "community work" to "internal development". This is exactly the point why I first started this discussion. If we all want OpenTSDB to succeed (and survive), we'll have to do it together.

Absolutely, and I do like the ASF idea.

[Jim Kleckner]

Note that tsuna says that having moved to Arista, he may not reply to email more than once per month:

  https://groups.google.com/d/msg/opentsdb/RI2qpE7cUmY/Y8v36J-DiKsJ

 

[tsuna]

On Thu, Jan 10, 2013 at 5:12 PM, Jim Kleckner <j...@cloudphysics.com> wrote:
> Note that tsuna says that having moved to Arista, he may not reply to email
> more than once per month:

It's true that since I changed jobs I had to focus primarily on
getting up to speed at Arista.
It's also true that traditionally I haven't been a good maintainer.
I consistently took a lot of time to answer questions on the mailing
list, follow up to issues / pull requests, and integrate changes
upstream (as a matter of fact there are 10 commits in tsuna/master not
yet in OpenTSDB/master, and for no good reason other than just "I
haven't pushed yet").

I would certainly appreciate extra help to integrate changes upstream
faster. There isn't really a notion of "commit access" since anyone
can commit anything on a GitHub fork, but there is indeed a bottleneck
to review and integrate changes in the "official" repo.

As far as moving to the ASF, I personally would like to stay
independent, because I'm happier with GitHub than with SVN/JIRA, and
I'm not a big fan of the overhead and politics that usually comes with
having a PMC, having an explicit list of committers, and other things
typically mandated by the ASF for the projects it hosts.


How about we come up with a plan to:
- Integrate latest changes and any must-fix bugs and release that as
OpenTSDB 1.1.0
- Come up with a plan for OpenTSDB 2.0

I know I personally want a number of things in 2.0:
- New read path (asynchronous, non-blocking, that doesn't require
O(N) RAM with N = # of data points).
- Proper JSON API (with at least JSONP in a first phase, CORS maybe later).
- Fix the stupid little bugs / annoyances (autocompletion
misbehaving, the &exact business which really just requires removing
unnecessary data points from the output, etc).
- Abstracting out HBase interactions behind an interface so that
people can (try to) use storage backends other than HBase.
- Store extra meta data in the tsdb-uid table (other than UIDs, e.g.
metric descriptions, whether a metric is known to be a rate or not,
unit of the metric, etc).
- Add the ability to add annotations on a graph.

And the list goes on. For me the big ticket item is the first on the
list. Amongst the rest, several of these things have already been
tackled, at least to a certain extent, in other forks. ManOLamancha's
"Scratch" fork is by far the farthest ahead.

What do you guys think? Without a plan it's harder to know where we're going.

--
Benoit "tsuna" Sigoure

[Kevin Ortman]

Hi Tsuna,

I think many here would be grateful to assist with change reviews and to help you plan/implement the 2.0 features.

How can the community help you here?  

Do you intend to bring others on board?

Thanks,

Kevin

[Rubberman]

We have started using OpenTSDB in a serious way in my organization. At the moment I am gathering something like 350M metrics per day from just one of our data centers (we have 4 or 5 of similar size) that is getting pushed into our hadoop cluster in the AWS EC2 cloud, and will be adding additional metrics over time that will reach about 1 billion data points per day per data center. Naturally, this encourages us to become contributors to this community. Right now, this is working really well at least from the data collection perspective, but we want to improve read performance, and to add additional programming language support (API's) to better extract data for analytic purposes.

-Rubberman

[tsuna]

On Thu, Jan 10, 2013 at 6:31 PM, Kevin Ortman <kevin....@gmail.com> wrote:
> How can the community help you here?

First we need to better define what we want, and how people want to
prioritize things. Of course different people have different needs
and priorities, but we need to get a sense of who's gonna do what.
For instance, I personally very much want to work on rewriting the
read path.

I just pushed my last round of changes upstream. Do we cut v1.1.0
from there, or are there any other must-fix bugs that you guys think
should absolutely go into the release?

Amongst the various pull requests and forks that already exist out
there, which changes should we start looking at first that we'll want
to integrate in 2.0? Which changes do we feel like are already "good
to go" that we could start cherry-picking upstream immediately?

> Do you intend to bring others on board?

Yes, I don't want to be the only maintainer merging things upstream.

--
Benoit "tsuna" Sigoure

[Peter Speybrouck]

With my background of OSIsoft PI (process historian), I am too interested in storing extra metadata for metrics like unit of measure (°C, m/s, psi, MB/s, ...)
This metadata could also contain other fields like:
* groups to make it easier to shield certain metrics from specific users. This does not mean that we need to implement a full security layer, but make it easier for custom frontends to limit users to a group of metrics
* Zero and span to indicate a range where you expect the datapoints to be. This can help for displaying or alerting (when it goes out of the range)
* Configuration to reduce resolution. I know that tcollector does deduplication, but with a fixed timerange (10 minutes if I remember correctly) for all metrics. In some cases it might be interesting to configure such a period per metric or even have a configuration for reducing noise. Of course, it would then be up to tcollector or another collector interface to actually use that configuration, but I think it is good to store such information together with metric info instead of relying on local storage that might get lost or changed without knowing it.
* Descriptor, if you search for metrics, this can be very useful to pick the right one.
* some general fields that can be interpreted by collectors to do certain actions (like scaling or go from a range [-100,100] to [0,200])

One thing that I am missing a little bit is a sort of SDK or RPC interface that allows to read/write data, search metrics/tags/tagValues, read/change tsd configuration.
I know there is the HTTP api (and with the JSON support it is useful) but some things need to be done locally, commandline. It would make it easier if you could do all that remotely through a uniform interface (for instance to manage different instances of tsdb).
This is probably not a high priority and might have a big impact on the code.


My java skills are very rusty, so having project files for Eclipse or Netbeans would make it easier for me to start understanding all the parts of the code and hopefully contribute something useful. I have seen some instructions to get everything in Eclipse, I will play with that a little.


These are just a few ideas, but I agree, making a plan of what is wanted and with some priorities sounds like a good way to make progress.

Peter

On Friday, January 11, 2013 2:59:26 AM UTC+1, tsuna wrote:

  - Store extra meta data in the tsdb-uid table (other than UIDs, e.g.
metric descriptions, whether a metric is known to be a rate or not,
unit of the metric, etc).
  - Add the ability to add annotations on a graph.

[Simon Matic Langford]

I think ManOLamancha was working on a lot of this (metadata/annotations), which we're also very interested in.

[m...@one.com]

On Friday, 11 January 2013 10:06:54 UTC+1, tsuna wrote:

On Thu, Jan 10, 2013 at 6:31 PM, Kevin Ortman <kevin....@gmail.com> wrote:
> How can the community help you here?

First we need to better define what we want, and how people want to
prioritize things.  Of course different people have different needs
and priorities, but we need to get a sense of who's gonna do what.
For instance, I personally very much want to work on rewriting the
read path.

There's this list or GitHub issues (tagged "ideas", perhaps?) for discussions. Whatever fits people the best, I guess.

 

I just pushed my last round of changes upstream.  Do we cut v1.1.0
from there, or are there any other must-fix bugs that you guys think
should absolutely go into the release?

Definitely a way of getting data out as JSON. Possibly getting full lists of metrics, tagk and tagv's out too. It is *the* major road-block for getting data out (be it other API's, UI's &c), IMHO.

 

Amongst the various pull requests and forks that already exist out
there, which changes should we start looking at first that we'll want
to integrate in 2.0?  Which changes do we feel like are already "good
to go" that we could start cherry-picking upstream immediately?

Can't we take this in a separate thread(s)/GH issues?

I'd be happy to start a buch of threads/issues for this. Just say where.

 

> Do you intend to bring others on board?

Yes, I don't want to be the only maintainer merging things upstream.

Great! 

[Frederik Kraus]

I'm not sure we should dive in to a roadmap / version / feature discussion at this point before working out how we want to proceed with the actual project setup. 

Can't we go one step back and focus on how we want to setup everything and maybe continue the feature / roadmap discussion in a different thread?

[Morten Siebuhr]

On 01/10/2013 04:34 PM, ManOLamancha wrote:
> On Thursday, January 10, 2013 5:18:31 AM UTC-5, m...@one.com wrote:
>
> On the upside, I've found references to work on OpenTSDB
> 2.0 (http://www.euphoriaaudio.com/opentsdb/new20.html

> <http://www.euphoriaaudio.com/opentsdb/new20.html>), which seem to
> happen on https://github.com/manolama/opentsdb/tree/Scratch
> <https://github.com/manolama/opentsdb/tree/Scratch>. The

> commits mention, among other, Cassandra, RabbitMQ a few times
> (neither of which is mentioned in the 'new20'-document), which makes
> me wonder where it's actually going. (And if so, is OpenTSDB 2.0
> switching to Cassandra, or "just" support it as an alternate back-end?).
>
> For the 2.0 version, I abstracted the backend so folks could use
> Cassandra or other storage systems as options since we don't know what
> will be around in a few years. I'd hate to see OpenTSDB die out if HBase
> goes under. Since I haven't worked out the bugs in the Cass code, or
> finished the MQ stuff, I haven't doced it yet, but when I get to that
> part I will.

Nice. I've also noticed an Accumulo-backend
(http://github.com/ericnewton/opentsdb), that might benefit from this as
well.

> My goal my 2.0 work is to pull in all of the tweaks folks have been
> making in their own forks, along with stuff we need at our company, and
> get them ready for integration into the main branch by Tsuna or Dave
> Barr. I'm hoping to earn committer status if my code is good enough. So
> far all of my work is really rough, hence the "Scratch" branch, and
> within the next month I'm going to be cleaning and moving it into the
> root of my fork for review.

From personal experience working with large projects, I tend to find it
difficult to work with multiple branches - except when it comes to
merging. Say tsuna doesn't like some of your work, the whole merge is in
risk of failing.

That's why I'm keen on having more active maintenance on master - it
will make smaller (and thus less involved) changes possible. Currently I
see a number of active forks, most of which are probably mutually
exclusive, should they be merged back into master.

> I would also like to see more committers but also more talk in the
> forums here about the features we're all developing. I've been trying to
> post additions I've been making but haven't received as much feedback as
> I'd like since we're all duplicating efforts. So maybe we need to figure
> out what everyone wants in 2.0, how the features should work, then try
> to figure out how we want to develop and integrate all of the changes.

Yeah. I've just found your post detailing this
(https://groups.google.com/forum/#!msg/opentsdb/AeRnSbW3dMo/Io4RMTDkaSUJ).
I'll comment on that once I've ingested it properly.

--
Siebuhr

[ma...@markfarnan.com]

Personally, 

I support a move to ASF, and APL-2.    For all sorts of reasons.   The licence is one, but the community and acceptance factors are actually more important. 

For a number of reasons, good or bad, the LGPL is just a  problem in some situations, where APL 2 is just fine.  

I also think the comiter model to main branch would take significant load off tsuana / any other single individual.     Lets face it, it works very very well as shown by many successfull projects. 

I'd hate to see OpenTSDB die due to the 'many fork / internals'  problem that has killed off other projects in the past. 

I do understand Tsuna's reluctance,  and there is some politics.  But OpenTSDB is already far ahead of many ASF projects that have graduated. 

Personally I think it would be a strong sign of acceptance, and encourage more take up, and therefore more contribution. 

Regards

Mark

[Jonathan Creasy]

I really support the idea of moving to an ASF type license and also the committer model.

[tsuna]

On Thu, Jun 13, 2013 at 11:32 PM, Jonathan Creasy <j...@box.com> wrote:
> I really support the idea of moving to an ASF type license and also the
> committer model.

As I mentioned previously, I'm not a big fan of that model. I much
prefer the model that is used by the Linux & Git development communities.

--
Benoit "tsuna" Sigoure

[Jonathan Creasy]

How does that work?

[Jonathan Creasy]

Is the license something that could be independently discussed or are they somewhat coupled to the development models?