How to configure Log severity in Dataproc Batches

[Prabha Arivalagan]

I am submitting a job to Dataproc batches (Serverless) , We are charged thousands of dollars for clod logging. I tries setting the properties but no luck. Any help or pointers would be greatly appreciated.

[Pushkar Sharan]

Hello Prabha,

You can use Cloud Logs Exclusion to remove unwanted logs and save cost on cloud logging.

https://cloud.google.com/logging/docs/routing/overview#exclusions

Operations --> Logging --> Log Router -->  Cloud Logging Bucket (_Default)  --> Edit Sink  --> Choose logs to filter out of Sink --> Add filters to exclude type of logs not needed

this will not generate unwanted logs and you can control the type of logs you want to see and pay for them only.

Thanks,
Pushkar Sharan

[Prabha Arivalagan]

Awesome, Thanks, Pushkar. That is good to know.

I am wondering if that can be configured in the Dataproc job?

[Pushkar Sharan]

Hi Prabha,

Yes you can try doing below to set logging level at a more granular level for each job

https://cloud.google.com/dataproc/docs/guides/logging#configuring_logging