Writing custom connectors

[Anand Kane]

Hello Folks

For the last few days, I am trying to write a sample custom connector (for the existing camel component), but have not succeeded to date. I followed the documentation at the below links as references;

https://access.redhat.com/documentation/en-us/red_hat_fuse/7.5/html/integrating_applications_with_fuse_online/customizing_ug#developing-extensions_custom

https://syndesis.io/manual/developing_extensions/

https://syndesis.io/docs/extensions/create/#overview-of-how-to-develop-an-extension

https://syndesis.io/docs/connectors/create/

The first two links have the same content; one is for Fuse and the other for Syndesis.

I have also observed that the documentations at different links differ from each other significantly.

Could someone kindly point me to the documentation that I can follow to write custom connectors? Alternatively, the source code of connectors like PostgresDB will also be a great learning resource. I tried to build and deploy the connector examples present at;

https://github.com/syndesisio/syndesis (app/connector folder)

https://github.com/syndesisio/syndesis-extensions

But none could be deployed on Fuse Online environment (version: 1.9.0.fuse-760020-redhat-00001).

Thanks in advance

Anand

[Zoran Regvart]

Hi Anand,
the definite documentation on creating connectors is this (one you linked):

https://syndesis.io/docs/connectors/create/

That process involves building and then deploying a version of
Syndesis that includes the new connector.

On an existing Syndesis installation connectors can be added via
extensions, and the definite documentation on that (you also linked
is):

https://syndesis.io/manual/developing_extensions/

Red Hat publishes documentation sourced from these upstream documents.

Bear in mind, for future versions of Syndesis (2.0), the notion of
extensions and how we bundle connectors within a distribution of
Syndesis will certainly differ, this is the work that's ahead of us
basing Syndesis on top of Camel-K and Kamelets. We're a bit ways off
from that future, and we still do releases from the 1.x version.

I would also encourage you, if you have a support contract with Red
Hat, to contact Red Hat support with any specific questions about the
version you get from Red Hat. This helps your issues be prioritized,
and you're getting your money's worth :)

zoran

> --
> You received this message because you are subscribed to the Google Groups "Syndesis" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syndesis+u...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/syndesis/b0ff498f-c645-4b43-bb98-cdb5818667ccn%40googlegroups.com.



--
Zoran Regvart

[Anand Kane]

Thanks, Zoran.

I have been attempting to get a custom connector to work using both methods; by developing a connector extension and by building Syndesis locally with a new connection. Unfortunately, I succeeded in neither. After having searched for the solution and trying out many combinations, I am once again reaching out to you for help. I hope that not much of a trouble.

A quick note on  Red Hat; I personally do not have a Red Hat subscription and I am told that there are some problems in getting support from Red Hat. I have fallen back to Syndesis on Minishift.

Problems with extension

• I could create and import connector extension following the documentation https://syndesis.io/manual/developing_extensions/.
• I could also create a connection from the connector to be used in integrations.
• However, I get exceptions listed in the attached file, extension-exceptions.txt, when I,
•         Validate the custom connector during its import
•         Use the connection in an integration

To fix the exceptions, I followed the example of SQL connector in app/connector/sql to write my implementations of ComponentMetadataRetrieval and ComponentVerifier (both imported via maven dependency io.syndesis.connector:connector-support-verifier:1.10.0 in my extension project). But I think, they are not being picked up. I have these files in place; META-INF/syndesis/connector/meta/sql and META-INF/syndesis/connector/verifier/ext-sql.

Problems with building a new connection

• I copied the 'sql' connector project into 'sql2' and changed the package from io.syndesis.connector.sql to io.syndesis.connector.sql2.
• Built and installed Syndesis by following instructions at https://syndesis.io/docs/connectors/create/.

Syndesis build and installation is successful. However, the installation does not contain 'sql2' connector. Instead, it contains 'sql' connector that I omitted from app/connector/pom.xml. I think this is happening because the new image is not pushed to Minishift. I confirmed this by logging into syndesis-server pod and checking the contents of 'sql.json' in the 'runtime.jar' file. The contents belong to the original 'sql' connector and not the new one. 

Running 'syndesis build -f -i -m s2i' has not solved the problem.

Could you please point out what am I missing? I am really stuck at this and your help will be much appreciated.

Thanks

Anand

[Anand Kane]

Hi Zoran

There is an update; I finally got the custom connector 'sql2' to appear in the connectors list. But now it is in the same state as that of the connector extension I mentioned in the previous mail. I can create a connection from this connector but cannot use it in integrations. Clicking 'Next' on the actions page of the connection results in the same exceptions that I mentioned in the attachment of the previous mail. I am attaching the same file here again for quick access. The extension connector and this new connection are in exactly the same state now.

Thanks

Anand

[Zoran Regvart]

Hi Anand,
I'll reply to both of your emails here.

On Tue, Dec 29, 2020 at 2:34 PM 'Anand Kane' via Syndesis
<synd...@googlegroups.com> wrote:
> A quick note on Red Hat; I personally do not have a Red Hat subscription and I am told that there are some problems in getting support from Red Hat. I have fallen back to Syndesis on Minishift.

Sad to hear that, the support is the only thing Red Hat sells, we
should be doing a good job at it.

> Problems with extension
>
> I could create and import connector extension following the documentation https://syndesis.io/manual/developing_extensions/.
> I could also create a connection from the connector to be used in integrations.
> However, I get exceptions listed in the attached file, extension-exceptions.txt, when I,
> Validate the custom connector during its import
> Use the connection in an integration

I'm not sure if we support metadata operations (such as validation)
with extensions, though I never fully understood what we do or don't
support with extensions since it's such an open-ended technology. Are
you using the version from master (2.0)? That could also be unstable.
Without seeing code it's a bit difficult to find out what's happening.
The exception is about missing file
`META-INF/syndesis/connector/meta/ext-sql`, and I don't really
understand why we need to load that, as far as I understood all the
dynamic bits, like loading classes from extensions should not happen
on the server JVM but on the metadata JVM, it appears we broke that
(could be a while ago).

> To fix the exceptions, I followed the example of SQL connector in app/connector/sql to write my implementations of ComponentMetadataRetrieval and ComponentVerifier (both imported via maven dependency io.syndesis.connector:connector-support-verifier:1.10.0 in my extension project). But I think, they are not being picked up. I have these files in place; META-INF/syndesis/connector/meta/sql and META-INF/syndesis/connector/verifier/ext-sql.

Yeah, there seems to be a need, for whatever reason, for those to
appear on the server classpath, and we don't have a way to do that, we
can change the classpath of the meta (by restarting it at runtime),
but not of the server. This is a strange requirement we added, not
sure why we did that.

> Problems with building a new connection
>
> I copied the 'sql' connector project into 'sql2' and changed the package from io.syndesis.connector.sql to io.syndesis.connector.sql2.
> Built and installed Syndesis by following instructions at https://syndesis.io/docs/connectors/create/.
>
> Syndesis build and installation is successful. However, the installation does not contain 'sql2' connector. Instead, it contains 'sql' connector that I omitted from app/connector/pom.xml. I think this is happening because the new image is not pushed to Minishift. I confirmed this by logging into syndesis-server pod and checking the contents of 'sql.json' in the 'runtime.jar' file. The contents belong to the original 'sql' connector and not the new one.
>
> Running 'syndesis build -f -i -m s2i' has not solved the problem.
>
> Could you please point out what am I missing? I am really stuck at this and your help will be much appreciated.

When you installed Syndesis on minishift did you use the `--dev`
option, again I'm not sure if this was broken with the operator, it
used to create an image stream that you can push your changes to. I
also think you probably need to build the server and the metadata
projects with `-i` to push the images to that image stream. Also make
sure you've added the dependency on the new connector, it needs to be
added in several places, see[1].

Quick way to check is to do a `oc get is` and see where the image
streams are pointing to, if they're pointing to docker.io then you
don't have the installation configured for development.

I usually don't bother with minishift, it's too slow and flaky for me,
I usually just run Syndesis locally[2], or straight from the IDE.

zoran

[1] https://syndesis.io/docs/connectors/create/#integrating-a-new-connector-with-existing-connectors
[2] https://syndesis.io/docs/development_quickstart/#running-without-kubernetes-openshift
--
Zoran Regvart

[Anand Kane]

Hi Zoran

Thanks for taking out time for the detailed response. I will try out the suggestions.

Regards

Anand