[DISCUSS] Moving toward an initial release

[P. Taylor Goetz]

I’d like to start a discussion to flesh out a list of tasks necessary to make an initial release. Having a downloadable binary release and associated maven artifacts, IMO would help in terms of expanding the community by allowing potential users “kick the tires” of JanusGraph and get and idea of what’s required to migrate off of TitanDB.

From a process perspective, we should figure out how releases are planned and executed. In Apache projects, that typically falls to the Release Manager. That role typically rotates among commiters, and any committer is free to propose a release at any time. The process typically starts with a DISCUSS thread where the community discusses what features, bug fixes, etc. should to into the release, and deciding who will act as release manager for that release. Once the release is ready to be cut, the release manager builds the source release, binary release, and stages the Maven artifacts in Nexus. The next step is to start a VOTE thread to approve the release (the VOTE message contains links to all release artifacts: source/binary archives, associated checksums and signatures, and links to the Nexus staging repository. For the vote to pass, it requires a minimum of 3 positive votes, and more positive than negative votes. If the vote passes, the release artifacts are made available for download, and the Maven artifacts released from Nexus.

The Apache release policy can be found here: http://www.apache.org/legal/release-policy.html

I that a process we would like to adopt?

From an operational perspective, I can think of a few things:

Any potential release managers would need a Nexus account. I already have one, so I can help facilitate the creation of the JanusGraph organization in Nexus (presumably using the “org.janusgraph” maven group ID). I would just need JIRA usernames for everyone. (You can create a JIRA account here: https://issues.sonatype.org/secure/Dashboard.jspa) I would want to do this in one fell swoop because the process requires human review and can take up to 48 hours.

Where would we host the downloads? The janusgraph.org website is in git, but I’m not sure we would want to store potentially big binaries there.

In terms of technical issues that should be resolved for the first release, do we want to list them out in this thread, use a GitHub issue label, or something else?

-Taylor

[Misha Brukman]

Hi Taylor,

On Fri, Jan 27, 2017 at 4:38 PM, P. Taylor Goetz <ptg...@gmail.com> wrote:

I’d like to start a discussion to flesh out a list of tasks necessary to make an initial release. Having a downloadable binary release and associated maven artifacts, IMO would help in terms of expanding the community by allowing potential users “kick the tires” of JanusGraph and get and idea of what’s required to migrate off of TitanDB.

Definitely agree on this! And we've had a number of users ask for this explicitly as well.

 

From a process perspective, we should figure out how releases are planned and executed. In Apache projects, that typically falls to the Release Manager. That role typically rotates among commiters, and any committer is free to propose a release at any time. The process typically starts with a DISCUSS thread where the community discusses what features, bug fixes, etc. should to into the release, and deciding who will act as release manager for that release. Once the release is ready to be cut, the release manager builds the source release, binary release, and stages the Maven artifacts in Nexus. The next step is to start a VOTE thread to approve the release (the VOTE message contains links to all release artifacts: source/binary archives, associated checksums and signatures, and links to the Nexus staging repository. For the vote to pass, it requires a minimum of 3 positive votes, and more positive than negative votes. If the vote passes, the release artifacts are made available for download, and the Maven artifacts released from Nexus.

The Apache release policy can be found here: http://www.apache.org/legal/release-policy.html

I that a process we would like to adopt?

No strong feelings on this matter, the Apache release process sounds good to me.

 

From an operational perspective, I can think of a few things:

Any potential release managers would need a Nexus account. I already have one, so I can help facilitate the creation of the JanusGraph organization in Nexus (presumably using the “org.janusgraph” maven group ID). I would just need JIRA usernames for everyone. (You can create a JIRA account here: https://issues.sonatype.org/secure/Dashboard.jspa) I would want to do this in one fell swoop because the process requires human review and can take up to 48 hours.

We can start a GitHub issue and request that everyone posts their desired usernames there, collect them all after some period of time, and submit a single request. Do you want to own this?

Where would we host the downloads? The janusgraph.org website is in git, but I’m not sure we would want to store potentially big binaries there.

We don't need to host binaries in version control, GitHub provides binary hosting outside of version control, so it won't affect the source repo: https://github.com/blog/1547-release-your-software

 

In terms of technical issues that should be resolved for the first release, do we want to list them out in this thread, use a GitHub issue label, or something else?

I'm fine with either, but for discussion purposes, email might be easier (?).

Misha

[Ted Wilmes]

Apache release process sounds good to me.  Perhaps we can discuss what to include on this list and then mark the relevant issues in github with the 0.1 milestone.

As far as what to include, I think it would be good to at a minimum get the build & tests in a repeatedly passing state and then possibly update things to TinkerPop 3.2.4 which will be released shortly.

I just entered this other issue which is serious, but I'm torn if it's serious enough require its inclusion in the first release.  Essentially if batch queries to the backends are flipped on, certain traversals will

fail.  At this point, getting a release that is at least at parity with the Titan 1.1 branch + a TinkerPop bump is probably a good first step.  What are you all's thoughts? 

--Ted

[Jerry He]

+1

We are on the same page.

The first release will be an introduction release. New face, updated dependencies as appropriate, on par with Titan 1.0/1.1 in term of testing and quality, with consideration/ documentation for migration from Titan.

Jerry  

[Misha Brukman]

FYI, I created a milestone for the 0.1.0 release (since we're on 0.1.0-SNAPSHOT now): https://github.com/JanusGraph/janusgraph/milestone/1

Let's start adding all blockers to that milestone so that it's easy to track where we stand. For example, I just added the following issue:

• https://github.com/JanusGraph/janusgraph/issues/73 (compatibility with other versions of JanusGraph as well as Titan)
  

I think some of these should also be blockers:

• https://github.com/JanusGraph/janusgraph/issues/5 (categorize tests by runtime)
  
• https://github.com/JanusGraph/janusgraph/issues/18 (add timeouts for test methods)
  

though what it's really trying to get to is to get a stable set of tests that we can rely on, that Travis also runs for every PR. Right now, we're just verifying builds, but since the full test suite times out, we are not running any tests continuously at all, which is dangerous.

--
You received this message because you are subscribed to the Google Groups "JanusGraph developer list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[P. Taylor Goetz]

I created the following issue to get Sonatype Nexus setup so we can publish artifacts to maven central.

https://github.com/JanusGraph/janusgraph/issues/97

Any committers interested in having the ability to perform a release should follow those instructions. Once we get a list of usernames, I can go ahead and setup the Nexus account for the “org.janusgraph” group ID.

Thanks,

-Taylor*

[sjudeng]

Let me know if this should be moved to another thread but is there any interest in scoping down the initial JanusGraph release to just be a "compatibility" release? There has been concern raised in discussions (for example here) that some users on production Titan systems would benefit from an initial release that has no breaking changes, especially to core storage, indexing and compute system components. There are already potentially breaking changes in master such as the TinkerPop update and others are in work.

Beyond mitigating concerns regarding having an initially easy migration step from Titan to JanusGraph, pursing a release branch without the additional features and dependency updates currently in work would allow for a relatively more rapid (and hopefully more stable) initial JanusGraph release. Any feedback on inevitable compatibility issues could then be addressed to the benefit of the subsequent JanusGraph release.

[P. Taylor Goetz]

I think that’s a sound approach. Having an easy migration path from Titan to JanusGraph will encourage adoption and help build community.

-Taylor

On Mar 9, 2017, at 12:00 PM, sjudeng <sju...@gmail.com> wrote:

Let me know if this should be moved to another thread but is there any interest in scoping down the initial JanusGraph release to just be a "compatibility" release? There has been concern raised in discussions (for example here) that some users on production Titan systems would benefit from an initial release that has no breaking changes, especially to core storage, indexing and compute system components. There are already potentially breaking changes in master such as the TinkerPop update and others are in work.

Beyond mitigating concerns regarding having an initially easy migration step from Titan to JanusGraph, pursing a release branch without the additional features and dependency updates currently in work would allow for a relatively more rapid (and hopefully more stable) initial JanusGraph release. Any feedback on inevitable compatibility issues could then be addressed to the benefit of the subsequent JanusGraph release.

--
You received this message because you are subscribed to the Google Groups "JanusGraph developer list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-de...@googlegroups.com.

[Adam Phelps]

Yeah, I agree completely here.

The TinkerPop update to me is a very different breaking change, since it
just requires some minor code changes vs needing to upgrade clusters.

- Adam Phelps

On 3/9/17 10:35 AM, P. Taylor Goetz wrote:
> I think that’s a sound approach. Having an easy migration path from
> Titan to JanusGraph will encourage adoption and help build community.
>
> -Taylor
>
>> On Mar 9, 2017, at 12:00 PM, sjudeng <sju...@gmail.com

>> <mailto:sju...@gmail.com>> wrote:
>>
>> Let me know if this should be moved to another thread but is there any
>> interest in scoping down the initial JanusGraph release to just be a
>> "compatibility" release? There has been concern raised in discussions
>> (for example here

>> <https://groups.google.com/forum/#!msg/janusgraph-dev/F-zfIsuiIoQ/TFsMMa59BAAJ>)

[sjudeng]

The TinkerPop update is relevant because it would require production users of OLAP capabilities to upgrade their compute clusters (e.g. from Spark 1.2 to Spark 1.6).

[P. Taylor Goetz]

My guess would be that end users might want as close to drop in replacement for Titan as possible, at least initially. That would at least give them the opportunity to move from an essentially EOL'd solution (Titan) to one with a future (JanusGraph). In other words, it would send the message that JanusGraph is a serious and viable continuation of Titan.

In terms of community development, IMHO it would be best to get users moved over to JanusGraph as quickly as possible. The sooner that happens, the sooner we get actual user feedback that can help better inform the project's technical direction.

-Taylor

On Mar 9, 2017, at 7:06 PM, sjudeng <sju...@gmail.com> wrote:

The TinkerPop update is relevant because it would require production users of OLAP capabilities to upgrade their compute clusters (e.g. from Spark 1.2 to Spark 1.6).

--

[Ted Wilmes]

In that case, since the disk storage format is unchanged, could we simply suggest that for the time being they could continue running OLAP jobs using their Titan 1.0 install?  There isn't a lot of interaction between OLAP and OLTP at this point aside from bulk loading/exporting.

I agree we should minimize leaving folks behind with version jumps that will require infrastructure upgrades (search & storage), but the titan11 branch that Janus forked off of, though it wasn't at TinkerPop 3.2.3, had progressed beyond the 3.0.2 release and I feel that there was a general consensus that it was a reasonable starting point.  I suppose we could cut a release from pre-upgrade to 3.2.3, but at that point, we'd still be ahead of folks running Titan 1.0 with the TP version 3.1.1.  I thinking getting Janus up to TinkerPop 3.2.3 or possibly even 3.2.4 depending on the timing is a big value add for people interested in kicking the tires and production users.  Having said that, perhaps we should start a thread that includes concrete examples of what a TinkerPop upgrade would mean to a Janus user coming from Titan and get feedback from the community on the level of pain this would cause them.

--Ted

On Thu, Mar 9, 2017 at 6:06 PM, sjudeng <sju...@gmail.com> wrote:

The TinkerPop update is relevant because it would require production users of OLAP capabilities to upgrade their compute clusters (e.g. from Spark 1.2 to Spark 1.6).

--
You received this message because you are subscribed to a topic in the Google Groups "JanusGraph developer list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/janusgraph-dev/8jkMnkKzmC0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to janusgraph-dev+unsubscribe@googlegroups.com.

[sjudeng]

That's a good point that titan11 was already at Spark 1.5 via TinkerPop 3.1.0. At that point including TinkerPop 3.2.3 probably isn't going to make much difference in terms of migration. If that's fine then a release branch (0.1-SNAPSHOT) could be created right now off the tip of master. If going with TinkerPop 3.2.3 it might be worth considering releasing JanusGraph as 0.2.0 if there was intention to tie versioning to TinkerPop.

If the goal is to try and move out relatively quickly on an initial release then I wouldn't wait for the update to TinkerPop 3.2.4 or really any other new feature/dependency update unless it's a true blocker. Looking over existing PRs and 0.1.0 release milestone issues it appears to me that resolving the below items would be sufficient for the initial "compatibility" release. The other issues are either done or in my opinion don't need to block a release if they can't be immediately addressed.

• #80 - Backward compatibility with Titan. Action: Needs to be merged.
  
• #132 - Non-permissive license issue. Action: Need confirmation that dependency licenses are okay as-is at least for the initial release
• (Optionally) #98 - GitHub releases deployment via Travis. Necessary only if using GitHub releases for deploying zipfile artifacts for tagged releases. Action: Needs to be merged.
  

[Ted Wilmes]

I think that's a solid list. If #159 gets through review, I'd be interested in having it included. WIthout it, bulk loads via the BulkLoaderVertexProgram become untenable due to unnecessary full vertex scans. If we can come to consensus on this list, shall we set a freeze date at some point in the not too distant future and then start working through the release steps? Maybe step 1 at that point should be publishing a snapshot followed by going through the full process.

--Ted

--

[sjudeng]

I'm not familiar with the proposed released process. The JanusGraph release docs appear to start with a candidate release already ready to be deployed but not how it's prepared/vetted. Personally I'd like to see ongoing development and release preparation work be broken out into different branches. On the release side it puts good visibility on what exactly will be the basis for the release and then also allows non-release development to continue without getting held up.

It sounds like #159 should definitely be included. I'd hope after this long that any other critical features/fixes for an initial release have already been identified on GitHub issues/PRs and there should be some resistance to delaying the release at all at this point except for really critical blockers. Hopefully keeping it scoped down to just a couple PRs to review/merge will help make this more manageable for committers (which I'm not, so I can't help on this part).

[Jason Plurad]

Agreed, creating a separate branch for the initial release and ongoing development makes sense. I'll be reviewing #159 this afternoon. Others are encouraged to review it also.

> help make this more manageable for committers (which I'm not, so I can't help on this part).

Disagree on the part in parens... this is an open community project and anybody who wants to contribute to JanusGraph can help move the project forward can do so by communicating on the mailing lists, on issues, and on pull requests. Your participation in all venues has done a lot for JanusGraph so far and is greatly appreciated!

[sjudeng]

Now that the initial snapshot deployment is behind us what are the next steps towards an initial release? PR#80 and #159 have been merged so it seems to me that the tip of master right now might be a good starting point for a release. We could work on release steps directly on master or create a release branch. Below is one possible approach. Or is there another process already defined or other alternatives?

* Create a release branch, 0.1-SNAPSHOT based on current tip of master (8914cae)
* On master branch: Bump versions from 0.1.0-SNAPSHOT to 0.2.0-SNAPSHOT
* On 0.1-SNAPSHOT branch
  - Run full test suite including TP tests
  - Update documentation as described in RELEASING.md
  - Validate release (what are the steps here? running janusgraph.sh?)
  - Tag version 0.1.0 as 0.1.0-RC1
  - Call vote on 0.1.0 release
  - (Optional) Repeat previous steps (0.1.0-RC2, etc.) as necessary until vote passes
  - Push artifacts to Nexus
  - Upload release distribution zipfile (where?)
* Merge 0.1-SNAPSHOT into master

One question is where release distribution zipfile will be hosted, assuming this should be provided somewhere aside from (or in addition to) Nexus. Titan used S3 I think and janusgraph-dist/src/release/release.sh looks like it might still be assuming this? Another option would be GitHub releases, which would provide them on https://github.com/JanusGraph/janusgraph/releases (see this page for an example).

[P. Taylor Goetz]

I would say, like Apache projects, the release should consist of the following:

* Source tar.gz/zip, with associated GPG signature of the release manager.

* Convenience binary tar.gz/zip + GPG sig.

* Maven artifacts published to mirrors.

The typical process works like this:

1. DISCUSS thread to form consensus on what to include.

2. Create a branch to release from, merge patches for the release.

3. Create build the release with `mvn release:prepare && mvn release:perform`. This will generate the tar.gz/zip files and stage the maven artifacts in Nexus.

4. Release Manager closes the Nexus repository, making the jar files available for review.

5. RM uploads tar.gz/zip files to staging area for review

5. RM calls a VOTE on janusgraph-dev@ for the release with links to: The tag/SHA for the release, CHANGELOG, tar.gz/zip files, Nexus staging repository.

If the VOTE is successful, release the Maven artifacts from Nexus, and move the staged tar.gz/zip files to the public download location. Then announce the release on user@ and dev@ mailing lists.

The Apache parent pom [1] can help automating this, though we may need to change it for whatever infrastructure we use. I don’t have an answer on where to various download sites should be, aside from Nexus. Misha?

When I looked at the TitanDB/JanusGraph build system, it seemed pretty convoluted and IMHO is ripe for refactoring and simplification. I’d be happy to help with this, but my availability is going to limited for the next two weeks.

-Taylor

--

You received this message because you are subscribed to the Google Groups "JanusGraph developer list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-de...@googlegroups.com.

[sjudeng]

The process you describe sounds good to me. I would call out tests specifically in there somewhere. For a JanusGraph release I think this should include running the full TinkerPop test suite before the candidate release is generated. Also some of the configuration/process could be manual for the initial release and then simplified/automated for the second release.

I'm really interested in getting master freed up for post-initial release development. But this could happen in parallel to release preparation once we create the release branch. Given there has been some discussion here on features to include (#80, #159, #132), can we start with master as it is now and then start the discuss thread in parallel? In the discussion I hope we can keep focus that this is an initial compatibility release only. Then hopefully we can agree that any new features, even those with complete PRs, can wait for the second release. There's nothing saying that the second release can't happen a month after the first.

If this is acceptable I'd like to get started at least with testing (which takes 2-3 days) as soon as possible. Once the tests are confirmed good hopefully others can help out with the documentation (changelog, etc.) updates and definitely to serve as the RM to actually tag and push out the release artifacts. Again in my opinion we could do some of the configuration/process steps manually the first time and then simplify/automate them later maybe as part of the second release. Is this approach too aggressive?

[Samik Raychaudhuri]

On hosting the zip file, I think we should just use the github hosting. It's easy and comes bundled with github, so why not use it?
Thanks.
-Samik

[Ted Wilmes]

That's a helpful summation of the Apache release process Taylor, thanks for that.  I say yes, we should get the process started.  I'll kick things off with a DISCUSS thread and create a 0.1.0 release branch that we can work out of.

Thanks,

Ted

To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-dev+unsubscribe@googlegroups.com.

[sjudeng]

Thanks for creating the branch. I'm starting working through TP tests now. You might consider changing the branch name to 0.1-SNAPSHOT or jg01 or something since presumably it would service all potential 0.1.x releases and also to avoid confusion with the eventual 0.1.0 tag.