job lock issue in stress tests.

[Vicky Kak]

I have started doing some stress tests for the platform that we have been building from past some time.

>>gobblin.runtime.JobException: Previous instance of job ingestor_flickr_92 is still running, skipping this scheduled run

Actually the rate of triggering ingestion is too high and hence the subsequent ingestion in being called before the earlier one is completed, the work around seems to disable the flag job.lock.enabled.

We have never got this issue so often, now for every run of the stress tests we see this error

I am curious to know how other gobblin users are handling this scenario.

I am going to disable the job lock and see how stress tests goes, will like to hear more from others.

Thanks,

Vicky

[Vicky Kak]

I did made the changes in the gobblin-standalone.properties, disabled the job lock and it worked as expected.

[Issac Buenrostro]

Hi Vicky,

The lock is in place because it is sometimes unsafe to run jobs using the same state-store in parallel. Furthermore, many constructs use scratch space in FileSystem that is just identified by the job name. If two jobs with the same name are running at the same time, they may corrupt each other's scratch space.

In a stress test, if you are repeteadly running the same job, whenever a job starts and it sees the old job still running, it will throw the above exception and abort. This is normal and expected. If you are sure that your jobs will not interfere with each other, you can disable the lock as you did (but again, it may corrupt state or produce unexpected output).

Best,

Issac

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-users+unsubscribe@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/277b423d-eea1-49a8-9f38-be225db24c1a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.