Release of v4.0.0

[Daniel Lamblin]

Could I find out where to get more information so I can answer these questions I have to help me decide if I can work to adopt the current release or should instead work on adopting a v4 rc release with the expectation that v4 is production ready soon?:

1) a roadmap or a high level summary of v3 vs v4 (and beyond? last thing I could find was talking about looking towards v2 from v1)

1a) EG what's not great in v3 that's handled in v4

1b) Is there anything known to be "basically broken/not-as-intended" in v3 that would ONLY be addressed in v4

2) a summary of how v4 would get released; EG when, or what criteria should be met with estimates.

3) a discussion of what the relationship is between bug fixes going into milestone v4 and back-porting to v3.3 point releases (does releasing 4 cut off backports to v3?)

4) guides for how much is involved in upgrading from production v3.3.9 to v4.0.0 when ready or testing v.4 rc X

5) known issues between rc X and target v4.0.0

6) how the is the user testing of RCs suggested to be performed and what channels are available to note issues or vote for release

7) with pygenie split into its own project, how ready is it to work with v4.0.0 when released? or will it stay on v3 api, if so how long?

Thanks,

-Daniel

[Tom Gianos]

Hi Daniel,

That's quite a list of questions and I'll try to answer as best I can at this time.

Let me preface by stating one of the reasons there isn't an official V4 release yet is that we haven't had time to generate much of the types of documentation you're asking about in here.

1. We hope to document this kind of thing before release but at a high level the main goal of V4 is to allow deployments of Genie where jobs aren't restricted to running on the same hardware as the Genie web server. The agent was developed to allow an agent process to run 1-1 with a given job and communicate back to the Genie servers with status information rather than the Genie servers monitoring jobs locally directly. Additionally the agent can back jobs launched directly via the agent CLI so it is hoped it will unify an organizations interactive and batch use cases with Genie. We hope to eventually provide pluggable ways to let people decide how to launch the Agent and one of the targeted plugins we'd provide would likely be for Titus. There was nothing inherently wrong with Genie 3 and it has served us well for many years. We're just evolving it to meet a growing number of use cases here at Netflix.

2. The Genie 4.0.0 release candidates were started being used internally on our main production clusters last week. This is why there has been an uptick in release candidate's as we fix bugs and issues we find. As for when an official 4.0.0 will be out in OSS we probably are looking sometime in Q3 before we have the bandwidth to handle the documentation though help is always welcome! We also have some known hardening we'd like to do with the agent process.

3. As of right now unless there is a compelling reason for us to backport further fixes into the 3.3.x line (of which 3.3.21 is currently the latest) we probably won't prioritize it. Since we're now running the 4.x line in production internally pulling onto 3.3.x probably isn't the best use of our time vs. moving forward on 4.x features/fixes as they continue to diverge. The initial implementation of V4 server side maintains the v3 job execution logic mostly as is and we only currently have a V3 API so it is intended to be fully backwards compatibly at the REST API level. No internal clients were changed as part of our V4 server upgrade for instance. So anyone who has V3 working should be able to upgrade to V4.

4. We're hoping to produce this as part of the documentation and each environment may change but there are some property changes but migration shouldn't be too difficult. Genie ships with Flyway to handle the database migrations but you may need a downtime to allow the migration to complete if your database is large.

5. All known issues with the 4.0.0 relative to 3.3.21 backwards interoperability have already been patched. Anything new we're finding we're actively working on as they come up. We're hoping this stabilizes in the next week or so and then we'll switch to hardening the agent for production. Since there was no Agent in < 4.x this would have no impact on people migrating and the agent is currently optional with 4.0.

6. Any testing users can provide in their own environment would be greatly appreciated as we can only test our own use cases. Any feedback is best provided via github issues most likely as they're most visible to the development team and allow us to make commits/messages directly against any known bugs.

7. pygenie is completely compatible with V4 currently as the existing REST API has stayed the same. It's still being used in the internal library which wraps it without modification. We'll see what happens when we get around to a V4 API but for now the main "client" we're working on is the agent.

Thanks for the questions. It gives us a good template for what to produce when it comes to documentation for V4. Feel free to let me know if this doesn't help/make sense or if you have further questions.

Tom

[Daniel Lamblin]

Thanks Tom,

Apologies if the questions were a lot to cover. I thought they were all related and aspects of the core question of whether adopting v3 would be a mistake as v4 getting active testing.

I appreciate your explanations, so it seems that it would be perfectly safe to make clients with the v3 api, start using v3.3.21, and move to a release of v4 if we can accept a bit of Genie downtime during a db upgrade that would be needed. Alternatively using a v4-rc might mean frequent updates to genie but could also contribute back issue reports (if any) and probably would have less downtime when updating to a release in the v4 line. (Until db changes come in, possibly in point releases?).

Thanks again,

-Daniel

[Tom Gianos]

No problem at all on the questions they were all reasonable.

If I were you starting fresh I'd probably just use the 4.x stuff to save an unnecessary upgrade but that's entirely up to you.

I would hope that we only made DB changes in major or minor releases. I don't think we have any desire to make patch releases with DB changes unless we find some massive bug but that likely wouldn't be related to a DB schema as it would be to code sitting on top of the DB.

Tom