Error while deploying secor on Kubernetes

[Sanath Krishna]

Hello,

I am trying to deploy secor to fetch messages from kafka producer and push it to an S3 bucket on AWS cloud. But I am running into a few errors.

1. The docker image "us.gcr.io/secor-demo/" isn't available or might have been taken  down. 

2. To go around this error and have a working docker image, I tried to build my own docker image on the "Dockerfile" present in the secor repository. But while building the docker file, I ran into an error at the ADD command mentioned in the docker file.

3. When I tried debugging the error mentioned in step 2, I understood that I need to install secor on my local machine, by running the mvn package command. When I tried running that, I came across another error concerning the pom.xml. It mentions that the dependencies could not be resolved. 

It'd be really helpful and I'd appreciate it if someone can help me with these blockages as soon as possible.

Regards,

Sanath.

[hc...@pinterest.com]

I just tried 'mvn clean package' build, looks like fine.

[Sanath Krishna]

There was an issue with my organization's network and hence I was getting that error.

Now it says the maven build is successful.

But I am trying to run secor by creating a new folder for binaries and all the mentioned steps, but I am getting an error which says the topic does not exist.

Can someone help me out with this.

I am posting the image of the error for your reference.

 
P.S :- I have created the topic and there are messages in that topic. Also the zookeeper and kafka broker service are running in another terminal.

[Henry Cai]

The is just a warning since secor hasn't been run before.  Do you see data file generated by secor while you adding new records into kafka?

--
You received this message because you are subscribed to a topic in the Google Groups "secor-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/secor-users/YtlEi7M1zFI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to secor-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/secor-users/37c269ef-8183-4b4a-ae4d-52ce2bf40c19n%40googlegroups.com.

[Sanath Krishna]

You mean the log file?

[Sanath Krishna]

I also don't see any data being pushed into the s3 bucket that I have created.

If it is just a warning, it must at least create a folder or something in the s3 bucket.

[Henry Cai]

If you generate some new data into kafka, do you see any data generated by secor?  If not, you can change the logging category in log4j to INFO or DEBUG to see what was happening.

To view this discussion on the web visit https://groups.google.com/d/msgid/secor-users/fdac1332-f3ed-4a8e-9cdd-75ee4ce49been%40googlegroups.com.

[Sanath Krishna]

I tried changing the logging category to go through the logs. This is what I came across:

I went through the logs all over and I could see it was connecting to the topic. But wasn't reading the messages from the topic and sending it to the S3. 

Also I couldn't find a new folder being created when I pushed data into kafka. 

I am expecting a file to be created in the following path: /tmp/secor_dev/message_logs/partition

PS :- I am seeing the message INFO: Starting LatchedStatsListener as soon as I run the secor command.