GOCQL and Cassandra 5 plans

[Jackson Fleming]

Hi,

Just wondering it there are plans for support and the upcoming Cassandra 5 release.

I can see that there is an issue around vector support on github https://github.com/gocql/gocql/issues/1734 but doesn't look like anyones had a chance to respond to it yet

Just wondering if the GOCQL team have any plans in this area yet?

Thanks,

Jackson

[Martin Sucha]

Hi Jackson,

There is a great deal of technical debt accumulated over the years in the gocql repository and v2 is long overdue.

Support for protocol version 5 is not a trivial change either.

Personally, I don't have motivation to add new features or to maintain gocql anymore.

It would be great if somebody else is willing to step up as a maintainer. Ideally multiple people.

I don't have permission to administer the gocql/gocql repository though, I only have commit access there.

Chris is the person who could help you with gaining commit access if you are willing to become a maintainer.

Alternatively, feel free to fork gocql or try asking maintainers of the Scylla fork (https://github.com/scylladb/gocql),

which still seems to be active, if they are willing to accept contributions for the V5 protocol support.

As far as I know the Scylla database server does not support the protocol V5 currently though (it has been a while since I last checked).

Also check out https://github.com/scylladb/scylla-go-driver.

There does not seem to be any activity either, but may be a better starting point for someone looking to build a new CQL driver in Go.

This one also only supports protocol V4 as far as I know.

If the only thing you need is support for the new vector type using gocql, you might be able to support it

with a custom Go type with gocql.Marshaler and gocql.Unmarshaler implementation, even in a third-party package.

There is a native vector type in CQL protocol v5, but according to https://github.com/datastax/gocql-astra/issues/17,

it seems that in protocol V4, Cassandra returns the vector as a custom type, which a custom unmarshaller could handle.

You can find the description of the wire format at https://github.com/apache/cassandra/blob/997302d8a9a9b89f8e7fbc1821d693e04d314a66/doc/native_protocol_v5.spec#L1210

A custom gocql.Unmarshaler will not map the vector type to any plain slice though, if you need to support that,

you need to update the Marshal and Unmarshal functions directly in gocql accordingly.

Feel free to send a pull request, but I am not promising when and if I merge it.

I hope the information here will help you in some way.

Best regards,

Martin

---

This email, including attached files, may contain confidential information and is intended only for the use of the individual and/or entity to which it is addressed. If you are not the intended recipient, disclosure, copying, use, or distribution of the information included in this email and/or in its attachments is prohibited.
If you have received it by mistake, please do not read, copy or use it, or disclose its contents to others. Please notify the sender that you have received this email by mistake by replying to the email, and then delete the email and any copies and attachments of it. Thank you.

[Fleming, Jackson]

Hi Martin,

 

Apologies for the very delayed response, we’ve got a few people who are keen to contribute to this project.

 

We’re starting looking at the project and how we can best contribute to it. Are there any burning or higher priority issues in the project that would be good for some new contributors to get involved with? If you don’t have any ideas that’s ok, we’ve got some issues identified already that we’ll start looking into.

 

I’ve cc’d Carlos in as he’s managing this effort currently.

 

Looking forward to hearing from you.

 

Regards,

 

Jackson Fleming

 

 

From: Martin Sucha <martin...@kiwi.com>
Date: Monday, 5 February 2024 at 7:55 PM
To: Fleming, Jackson <Jackson...@netapp.com>, Chris Bannister <c.ban...@gmail.com>
Cc: gocql <go...@googlegroups.com>
Subject: Re: GOCQL and Cassandra 5 plans

You don't often get email from martin...@kiwi.com. Learn why this is important

 

EXTERNAL EMAIL - USE CAUTION when clicking links or attachments

[Mick Semb Wever]

  

 

Apologies for the very delayed response, we’ve got a few people who are keen to contribute to this project.

We’re starting looking at the project and how we can best contribute to it. Are there any burning or higher priority issues in the project that would be good for some new contributors to get involved with? If you don’t have any ideas that’s ok, we’ve got some issues identified already that we’ll start looking into.

Hi Jackson ! 

The Cassandra project is discussing  whether to adopt the gocql driver.  We are looking for a show of support here:  https://lists.apache.org/thread/h8q3lgqgsb87vptxr1j6x322yqz9fk7l 

regards,

Mick

[Martin Sucha]

Hi Jackson and Carlos, 

apologies for the delayed response.

On Thu, May 2, 2024 at 9:43 AM Fleming, Jackson <Jackson...@netapp.com> wrote: 

we’ve got a few people who are keen to contribute to this project.

That's great to hear!

The Cassandra project is discussing adoption of the gocql driver, which I support, as it could attract more contributors from the Cassandra community.

If you support this move and are willing to help maintain the project, it would be great if you could reply to the thread in the Cassandra dev mailing list as well: https://lists.apache.org/thread/h8q3lgqgsb87vptxr1j6x322yqz9fk7l 

 

We’re starting looking at the project and how we can best contribute to it. Are there any burning or higher priority issues in the project that would be good for some new contributors to get involved with? If you don’t have any ideas that’s ok, we’ve got some issues identified already that we’ll start looking into.

There are many issues and pull requests open, it would help to triage them and close obsolete ones.

Some of the issues require breaking changes to the API, which is needed to clean up the deprecated stuff as well,

so it would be great to have an overview of how the API will evolve.

Some of the things that need a breaking change are already marked with the semver-major label,

examples include removing global logger or global functions like NewBatch.

The major thing that is missing in the driver currently is support for the final version of CQL protocol v5

and the data types from the latest Cassandra versions.

The current implementation of the v5 protocol support was written before the framing changes were introduced to the protocol,

so it is incompatible with the protocol versions released in stable Cassandra versions.

One of the other requested features that multiple people asked for is exposing the topology of the cluster to the user code.

That is needed mainly for two reasons:

- When you are doing scans using token ranges.

- When you need to implement a custom host selection policy.

There is a pull request implementing this change, which needs to be reviewed: https://github.com/gocql/gocql/pull/1664

The API there could probably be simplified a bit.

Speaking of host selection policies, it is not currently possible to implement a host selection policy outside of the package.

Possibly other exported interfaces too.

The above change is one of the steps needed to allow that.

Another would be to make sure that you can mock the data such as HostInfo to allow testing your host selection policy.

The CI pipeline needs some work to make it more robust. Currently it uses ccm for setting up the cluster and the whole setup is fragile

and it is not as straightforward for contributors to run the integration tests locally.

I think the best would be to have a harness that spawns the Cassandra containers written in Go, for example using testcontainers-go or another similar approach.

Ideally these integration tests should be usable with other implementations of the CQL protocol as well, like AWS keyspaces, Scylla or Yugabyte,

so the vendors could run pipelines against their server implementations using the same test suite.

This would also enable more types of integration tests, as the test coverage needs to be increased as well to prevent regressions.

Another important part of testing that is not covered currently is fuzzing. Go 1.18 has native fuzzing support, but gocql does not take advantage of it yet.

There were also suggestions to continuously fuzz the code https://github.com/gocql/gocql/issues/1471, which I think may help as well.

The global Marshal and Unmarshal functions and Marshaler/Unmarshaler interfaces have limitations on how the serialization can be customized,

so a new version of the API should not use global Marshal/Unmarshal functions and instead pass the implementation for marshaling/unmarshaling

as a parameter in the Marshaler/Unmarshaler interface.

Changing the error reporting to clearly distinguish client errors from server errors should also improve the user experience.

That's all I can think of right now, I'm sure I forgot something, but I guess it's a good start.

I hope the gocql project will see more activity again.

Best regards,

Martin

[Martin Sucha]

Hi!

On Fri, May 17, 2024 at 4:54 PM Martin Sucha <martin...@kiwi.com> wrote:

On Thu, May 2, 2024 at 9:43 AM Fleming, Jackson <Jackson...@netapp.com> wrote: 

we’ve got a few people who are keen to contribute to this project.

That's great to hear!

The Cassandra project is discussing adoption of the gocql driver, which I support, as it could attract more contributors from the Cassandra community.

If you support this move and are willing to help maintain the project, it would be great if you could reply to the thread in the Cassandra dev mailing list as well: https://lists.apache.org/thread/h8q3lgqgsb87vptxr1j6x322yqz9fk7l 

Oh, I just noticed that Carlos already posted to the Cassandra mailing list on May 3rd: https://lists.apache.org/thread/wxspsypk8821htnn5g7w3bsklpj565jf 

Martin