How To Continuously Feed Off Of Multiple Topics

[Brett Renshaw]

Hi all,

I have a few questions about multiple Kafka loaders running in TigerGraph.

Is it that currently only one Kafka loader can run per node?

I've tried starting multiple Kafka load jobs on a single node, but it seems that the second can only start after the first is complete and so on.

Basically my use case is that I gave several different vertex/edge types that each have their own topics,

and I would always like to have a Kafka loader running for each.

Is that a common thing or is it not a good practice regarding TigerGraph?

If it is not advisable to have multiple Kafka loaders always running continuously, then is a more orchestrated

approach the usual solution? One where load jobs are periodically kicked off by a CI/CD server or something?

In that case, how can I make sure that when a specific load job is done, that when is starts back up later, that it starts at the offset it left off at?

[Xinyu Chang]

Hi Brett,

There is a Kafka config for concurrent Kafka loader numbers. 

1. gadmin --config kafka

2. change KAFKA-LOADER.count to a larger value

3. save

4. gadmin config-apply

5. gadmin restart restpp -fy

6. restart the loading jobs.

Thanks.

Xinyu Chang

Director of Customer Solutions

TigerGraph, Inc.

xi...@tigergraph.com

--
Welcome to GSQL-user group.
- our mission is bringing the power of graph databases to everyone www.opengsql.org
- technical resource can be found here https://docs.tigergraph.com/
---
You received this message because you are subscribed to the Google Groups "gsql-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gsql-users+...@opengsql.org.
To view this discussion on the web visit https://groups.google.com/a/opengsql.org/d/msgid/gsql-users/67c22663-3534-495e-9ab8-94d0fa3f3e50%40opengsql.org.