Connect classpath and provided dependencies

241 views
Skip to first unread message

Vasco Figueira

unread,
Oct 17, 2016, 1:22:28 PM10/17/16
to Confluent Platform
Hi,

I'm running into classpath problems and would like to clarify my understanding of how kafka-connect deals with it.

A quick look at bin/connect-distributed and the command that it actually executes, tells me that "kafka", "confluent-common", "kafka-serde-tools", "monitoring-interceptors" and any other subdir of share/java that starts with "connect-" are added to the classpath of the process.

1. Am I right in assuming that all the jars in the above directories end up in the same single undivided classpath without any isolation from each other?

2. If so, that would mean that in order to support running any connector of ours, all of them would have to be added to said directories.

3. If said dependencies have version conflicts we either a) shade the jar renaming their usage, b) align versions across all the connectors deployed to the same cluster or c) rely on the ordering of resources in the classpath definition.

If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Thanks very much for your time.

Regards,

Vasco

Shikhar Bhushan

unread,
Oct 17, 2016, 7:13:52 PM10/17/16
to Confluent Platform
1. That is correct. Currently classpath isolation is not implemented by Connect, it's certainly a gap we'd like to address.
2. Yes, your connector's JAR (and any dependencies) would need to be on the CLASSPATH.
3. Shading is advisable if you are worried about conflicts, but you should exclude `org.apache.kafka:connect-api` by marking as provided.

> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/2d250747-a8a1-4261-ad47-99e6d5f93c9f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Lucy Zhang

unread,
Oct 27, 2016, 5:10:31 PM10/27/16
to Confluent Platform
> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?
Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

I can give an example of an issue. We have a connector that uses Apache HttpClient, and we also use the HDFS connector. The scripts for running Kafka Connect add httpclient-4.2.5.jar and httpcore-4.2.4.jar to the classpath, since they're needed by the HDFS connector. We'd prefer to use HttpClient 4.5 for our own connector, in part because HttpClient 4.2 has a SSL-related vulnerability (CVE-2014-3577).

Based on your suggestion I'm going to try to do jar shading for our own connector, but other suggestions would be welcome if there's a better solution to this problem.

 On Monday, October 17, 2016 at 7:13:52 PM UTC-4, Shikhar Bhushan wrote:
1. That is correct. Currently classpath isolation is not implemented by Connect, it's certainly a gap we'd like to address.
2. Yes, your connector's JAR (and any dependencies) would need to be on the CLASSPATH.
3. Shading is advisable if you are worried about conflicts, but you should exclude `org.apache.kafka:connect-api` by marking as provided.

> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

On Mon, Oct 17, 2016 at 10:22 AM Vasco Figueira <vasco.f...@gmail.com> wrote:
Hi,

I'm running into classpath problems and would like to clarify my understanding of how kafka-connect deals with it.

A quick look at bin/connect-distributed and the command that it actually executes, tells me that "kafka", "confluent-common", "kafka-serde-tools", "monitoring-interceptors" and any other subdir of share/java that starts with "connect-" are added to the classpath of the process.

1. Am I right in assuming that all the jars in the above directories end up in the same single undivided classpath without any isolation from each other?

2. If so, that would mean that in order to support running any connector of ours, all of them would have to be added to said directories.

3. If said dependencies have version conflicts we either a) shade the jar renaming their usage, b) align versions across all the connectors deployed to the same cluster or c) rely on the ordering of resources in the classpath definition.

If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Thanks very much for your time.

Regards,

Vasco

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

On Monday, October 17, 2016 at 7:13:52 PM UTC-4, Shikhar Bhushan wrote:
1. That is correct. Currently classpath isolation is not implemented by Connect, it's certainly a gap we'd like to address.
2. Yes, your connector's JAR (and any dependencies) would need to be on the CLASSPATH.
3. Shading is advisable if you are worried about conflicts, but you should exclude `org.apache.kafka:connect-api` by marking as provided.

> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

On Mon, Oct 17, 2016 at 10:22 AM Vasco Figueira <vasco.f...@gmail.com> wrote:
Hi,

I'm running into classpath problems and would like to clarify my understanding of how kafka-connect deals with it.

A quick look at bin/connect-distributed and the command that it actually executes, tells me that "kafka", "confluent-common", "kafka-serde-tools", "monitoring-interceptors" and any other subdir of share/java that starts with "connect-" are added to the classpath of the process.

1. Am I right in assuming that all the jars in the above directories end up in the same single undivided classpath without any isolation from each other?

2. If so, that would mean that in order to support running any connector of ours, all of them would have to be added to said directories.

3. If said dependencies have version conflicts we either a) shade the jar renaming their usage, b) align versions across all the connectors deployed to the same cluster or c) rely on the ordering of resources in the classpath definition.

If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Thanks very much for your time.

Regards,

Vasco

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

On Monday, October 17, 2016 at 7:13:52 PM UTC-4, Shikhar Bhushan wrote:
1. That is correct. Currently classpath isolation is not implemented by Connect, it's certainly a gap we'd like to address.
2. Yes, your connector's JAR (and any dependencies) would need to be on the CLASSPATH.
3. Shading is advisable if you are worried about conflicts, but you should exclude `org.apache.kafka:connect-api` by marking as provided.

> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

On Mon, Oct 17, 2016 at 10:22 AM Vasco Figueira <vasco.f...@gmail.com> wrote:
Hi,

I'm running into classpath problems and would like to clarify my understanding of how kafka-connect deals with it.

A quick look at bin/connect-distributed and the command that it actually executes, tells me that "kafka", "confluent-common", "kafka-serde-tools", "monitoring-interceptors" and any other subdir of share/java that starts with "connect-" are added to the classpath of the process.

1. Am I right in assuming that all the jars in the above directories end up in the same single undivided classpath without any isolation from each other?

2. If so, that would mean that in order to support running any connector of ours, all of them would have to be added to said directories.

3. If said dependencies have version conflicts we either a) shade the jar renaming their usage, b) align versions across all the connectors deployed to the same cluster or c) rely on the ordering of resources in the classpath definition.

If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Thanks very much for your time.

Regards,

Vasco

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

On Monday, October 17, 2016 at 7:13:52 PM UTC-4, Shikhar Bhushan wrote:
1. That is correct. Currently classpath isolation is not implemented by Connect, it's certainly a gap we'd like to address.
2. Yes, your connector's JAR (and any dependencies) would need to be on the CLASSPATH.
3. Shading is advisable if you are worried about conflicts, but you should exclude `org.apache.kafka:connect-api` by marking as provided.

> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

On Mon, Oct 17, 2016 at 10:22 AM Vasco Figueira <vasco.f...@gmail.com> wrote:
Hi,

I'm running into classpath problems and would like to clarify my understanding of how kafka-connect deals with it.

A quick look at bin/connect-distributed and the command that it actually executes, tells me that "kafka", "confluent-common", "kafka-serde-tools", "monitoring-interceptors" and any other subdir of share/java that starts with "connect-" are added to the classpath of the process.

1. Am I right in assuming that all the jars in the above directories end up in the same single undivided classpath without any isolation from each other?

2. If so, that would mean that in order to support running any connector of ours, all of them would have to be added to said directories.

3. If said dependencies have version conflicts we either a) shade the jar renaming their usage, b) align versions across all the connectors deployed to the same cluster or c) rely on the ordering of resources in the classpath definition.

If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Thanks very much for your time.

Regards,

Vasco

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Ewen Cheslack-Postava

unread,
Nov 1, 2016, 10:25:53 PM11/1/16
to Confluent Platform
Right, this is the type of conflict we might expect. It can also often be resolved (and we agree this is not ideal) by careful ordering of the CLASSPATH. I'm not sure about HttpClient, though ideally minor versions wouldn't be backwards incompatible so 4.2.x and 4.5.x would be compatible such that putting the former earlier in the classpath wouldn't be a problem. (We also know this won't be the case for all libraries.)

We know this is increasingly becoming a problem for folks using multiple connectors. We're actively looking into fixing this.

-Ewen

On Thu, Oct 27, 2016 at 2:10 PM, Lucy Zhang <lucy....@appboy.com> wrote:
> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?
Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

I can give an example of an issue. We have a connector that uses Apache HttpClient, and we also use the HDFS connector. The scripts for running Kafka Connect add httpclient-4.2.5.jar and httpcore-4.2.4.jar to the classpath, since they're needed by the HDFS connector. We'd prefer to use HttpClient 4.5 for our own connector, in part because HttpClient 4.2 has a SSL-related vulnerability (CVE-2014-3577).

Based on your suggestion I'm going to try to do jar shading for our own connector, but other suggestions would be welcome if there's a better solution to this problem.

 On Monday, October 17, 2016 at 7:13:52 PM UTC-4, Shikhar Bhushan wrote:
1. That is correct. Currently classpath isolation is not implemented by Connect, it's certainly a gap we'd like to address.
2. Yes, your connector's JAR (and any dependencies) would need to be on the CLASSPATH.
3. Shading is advisable if you are worried about conflicts, but you should exclude `org.apache.kafka:connect-api` by marking as provided.

> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

On Mon, Oct 17, 2016 at 10:22 AM Vasco Figueira <vasco.f...@gmail.com> wrote:
Hi,

I'm running into classpath problems and would like to clarify my understanding of how kafka-connect deals with it.

A quick look at bin/connect-distributed and the command that it actually executes, tells me that "kafka", "confluent-common", "kafka-serde-tools", "monitoring-interceptors" and any other subdir of share/java that starts with "connect-" are added to the classpath of the process.

1. Am I right in assuming that all the jars in the above directories end up in the same single undivided classpath without any isolation from each other?

2. If so, that would mean that in order to support running any connector of ours, all of them would have to be added to said directories.

3. If said dependencies have version conflicts we either a) shade the jar renaming their usage, b) align versions across all the connectors deployed to the same cluster or c) rely on the ordering of resources in the classpath definition.

If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Thanks very much for your time.

Regards,

Vasco

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

On Monday, October 17, 2016 at 7:13:52 PM UTC-4, Shikhar Bhushan wrote:
1. That is correct. Currently classpath isolation is not implemented by Connect, it's certainly a gap we'd like to address.
2. Yes, your connector's JAR (and any dependencies) would need to be on the CLASSPATH.
3. Shading is advisable if you are worried about conflicts, but you should exclude `org.apache.kafka:connect-api` by marking as provided.

> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

On Mon, Oct 17, 2016 at 10:22 AM Vasco Figueira <vasco.f...@gmail.com> wrote:
Hi,

I'm running into classpath problems and would like to clarify my understanding of how kafka-connect deals with it.

A quick look at bin/connect-distributed and the command that it actually executes, tells me that "kafka", "confluent-common", "kafka-serde-tools", "monitoring-interceptors" and any other subdir of share/java that starts with "connect-" are added to the classpath of the process.

1. Am I right in assuming that all the jars in the above directories end up in the same single undivided classpath without any isolation from each other?

2. If so, that would mean that in order to support running any connector of ours, all of them would have to be added to said directories.

3. If said dependencies have version conflicts we either a) shade the jar renaming their usage, b) align versions across all the connectors deployed to the same cluster or c) rely on the ordering of resources in the classpath definition.

If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Thanks very much for your time.

Regards,

Vasco

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

On Monday, October 17, 2016 at 7:13:52 PM UTC-4, Shikhar Bhushan wrote:
1. That is correct. Currently classpath isolation is not implemented by Connect, it's certainly a gap we'd like to address.
2. Yes, your connector's JAR (and any dependencies) would need to be on the CLASSPATH.
3. Shading is advisable if you are worried about conflicts, but you should exclude `org.apache.kafka:connect-api` by marking as provided.

> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

On Mon, Oct 17, 2016 at 10:22 AM Vasco Figueira <vasco.f...@gmail.com> wrote:
Hi,

I'm running into classpath problems and would like to clarify my understanding of how kafka-connect deals with it.

A quick look at bin/connect-distributed and the command that it actually executes, tells me that "kafka", "confluent-common", "kafka-serde-tools", "monitoring-interceptors" and any other subdir of share/java that starts with "connect-" are added to the classpath of the process.

1. Am I right in assuming that all the jars in the above directories end up in the same single undivided classpath without any isolation from each other?

2. If so, that would mean that in order to support running any connector of ours, all of them would have to be added to said directories.

3. If said dependencies have version conflicts we either a) shade the jar renaming their usage, b) align versions across all the connectors deployed to the same cluster or c) rely on the ordering of resources in the classpath definition.

If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Thanks very much for your time.

Regards,

Vasco

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

On Monday, October 17, 2016 at 7:13:52 PM UTC-4, Shikhar Bhushan wrote:
1. That is correct. Currently classpath isolation is not implemented by Connect, it's certainly a gap we'd like to address.
2. Yes, your connector's JAR (and any dependencies) would need to be on the CLASSPATH.
3. Shading is advisable if you are worried about conflicts, but you should exclude `org.apache.kafka:connect-api` by marking as provided.

> If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Which dependencies do you need for your connector? In general you should be good with 2 version constants, `confluent.version` (e.g. 3.0.1) for `io.confluent.*` deps, and `kafka.version` (e.g. `0.10.1`) for `org.apache.kafka` deps.

On Mon, Oct 17, 2016 at 10:22 AM Vasco Figueira <vasco.f...@gmail.com> wrote:
Hi,

I'm running into classpath problems and would like to clarify my understanding of how kafka-connect deals with it.

A quick look at bin/connect-distributed and the command that it actually executes, tells me that "kafka", "confluent-common", "kafka-serde-tools", "monitoring-interceptors" and any other subdir of share/java that starts with "connect-" are added to the classpath of the process.

1. Am I right in assuming that all the jars in the above directories end up in the same single undivided classpath without any isolation from each other?

2. If so, that would mean that in order to support running any connector of ours, all of them would have to be added to said directories.

3. If said dependencies have version conflicts we either a) shade the jar renaming their usage, b) align versions across all the connectors deployed to the same cluster or c) rely on the ordering of resources in the classpath definition.

If we were to go for b), is there an easy way to get the definition of what those dependencies and their versions are, so that we can have our projects build against them as dependencies provided in the platform?

Thanks very much for your time.

Regards,

Vasco

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

Appboy named Forbes Cloud 100: Rising Star, and Digiday Signal Award Finalist for Best Marketing Automation Platform.


--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/ba766b6f-4cc5-4bda-8e64-bc3c2069f5db%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks,
Ewen
Reply all
Reply to author
Forward
0 new messages