Issue with Video Recording and Upload to S3 on Selenium Grid in Kubernetes

33 views

Skip to first unread message

Jasim Abdul Jamal

unread,

Oct 11, 2024, 1:20:49 AM10/11/24

to Selenium Users

I'm facing an issue with setting up video recording and upload to S3 for browser nodes in my Selenium Grid running on Kubernetes. I’ve followed the instructions in the Selenium documentation here: Selenium Grid Video Recording and Upload Configuration.

The video recording seems to start and stop correctly, but the upload to S3 fails with a 403 Forbidden error. Below are the relevant logs:

2024-10-07T18:31:15UTC [video.recorder] - Starting to record video
2024-10-07T18:31:15UTC [video.recorder] - Video recording started
[x11grab @ 0x5636e152b800] Stream #0: not enough frames to estimate rate; consider increasing probesize
2024-10-07T18:31:16UTC [video.recorder] - Video recording in progress
2024-10-07T18:31:18UTC [video.recorder] - Video recording in progress
2024-10-07T18:31:19UTC [video.recorder] - Video recording in progress
2024-10-07T18:31:20UTC [video.recorder] - Video recording stopped
2024-10-07T18:31:20UTC [video.recorder] - Add to pipe a signal Uploading video to s3://*****/5c3113be5449a1ef29db7767956dab81.mp4
2024-10-07T18:31:20UTC [video.recorder] - Node will be drained since max sessions reached count number (1)
2024-10-07T18:31:20UTC [video.uploader] - Uploading /videos/5c3113be5449a1ef29db7767956dab81.mp4 to s3://*****
2024-10-07T18:31:20UTC [video.recorder] - Trapped SIGTERM/SIGINT/x so shutting down recorder
2024-10-07T18:31:20UTC [video.recorder] - Sending a signal to force exit the uploader
2024-10-07T18:31:20UTC [video.uploader] - Received exit signal. Aborting upload process
2024-10-07T18:31:20UTC [video.recorder] - Recorder is waiting for RCLONE to finish

2024/10/07 18:31:20 ERROR : Attempt 1/3 failed with 1 errors and: operation error S3: HeadObject, https response error StatusCode: 403, RequestID: 9XSQ7D5A6J210CBF, HostID: fSFfRkNF/NsizItSiScSQr6ZYwT/5ocKGgfrWuTfaQIaWcxuHeUp+dogBHPs/Lejhwa5SxKa3bYcJbgMo9b8+A==, api error Forbidden: Forbidden
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.1s
2024/10/07 18:31:20 ERROR : Attempt 2/3 failed with 1 errors and: operation error S3: HeadObject, https response error StatusCode: 403, RequestID: 9XSQR0CHEZ01CTH0, HostID: jJfMOUdaCFAWZyBLMkQ6MewJu4w3DPL7HCcdiSYH9R4vyuyg0AkzNaosYY17oyAtv9xxYM76rpNBE6DSEgmKAg==, api error Forbidden: Forbidden
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.1s
2024/10/07 18:31:20 ERROR : Attempt 3/3 failed with 1 errors and: operation error S3: HeadObject, https response error StatusCode: 403, RequestID: 9XSQN213GH561CGG, HostID: hUSTAODtddJpNnpAyhe1DCdXAUYZ2glIuiCmYl5HKLPgF2VPktVbGCIBgKY8zWf15XNfXencpcvMFPzK8g0sZg==, api error Forbidden: Forbidden
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.1sTransferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.1s
2024/10/07 18:31:20 NOTICE: Failed to move: operation error S3: HeadObject, https response error StatusCode: 403, RequestID: 9XSQN213GH561CGG, HostID: hUSTAODtddJpNnpAyhe1DCdXAUYZ2glIuiCmYl5HKLPgF2VPktVbGCIBgKY8zWf15XNfXencpcvMFPzK8g0sZg==, api error Forbidden: Forbidden
2024-10-07T18:31:21UTC [video.uploader] - Waiting for /tmp/uploadpipe to be present
2024-10-07T18:31:21UTC [video.uploader] - Named pipe /tmp/uploadpipe exists
2024-10-07T18:31:21UTC [video.uploader] - Start consuming pipe file to upload
2024-10-07T18:31:21UTC [video.uploader] - Received exit signal. Aborting upload process
2024-10-07T18:31:21UTC [video.recorder] - Ready to shutdown the recorder
2024-10-07 18:31:21,690 INFO exited: video-recording (exit status 0; expected)
2024-10-07 18:31:21,691 WARN received SIGTERM indicating exit request
2024-10-07 18:31:21,691 INFO waiting for video-ready, video-upload to die
2024-10-07T18:31:22UTC [video.uploader] - Trapped SIGTERM/SIGINT/x so shutting down uploader
2024-10-07T18:31:22UTC [video.uploader] - Start consuming pipe file to upload
2024-10-07T18:31:22UTC [video.uploader] - Received exit signal. Aborting upload process
2024-10-07T18:31:22UTC [video.uploader] - Uploader consumed all files in the pipe
2024-10-07T18:31:22UTC [video.uploader] - Uploader is ready to shutdown
2024-10-07T18:31:22UTC [video.uploader] - Trapped SIGTERM/SIGINT/x so shutting down uploader
2024-10-07T18:31:22UTC [video.uploader] - Start consuming pipe file to upload
2024-10-07T18:31:22UTC [video.uploader] - Received exit signal. Aborting upload process
2024-10-07T18:31:22UTC [video.uploader] - Uploader consumed all files in the pipe
2024-10-07T18:31:22UTC [video.uploader] - Uploader is ready to shutdown
2024-10-07 18:31:22,438 INFO stopped: video-upload (exit status 0)
2024-10-07 18:31:22,440 WARN stopped: video-ready (terminated by SIGKILL)

I have double-checked my AWS S3 credentials and permissions, and everything seems to be in order. However, the upload process still fails with the same 403 error after multiple retries.

It appears the video uploader process is receiving an exit signal and aborts the upload:

Has anyone encountered a similar issue with video recording/upload to S3 on Selenium Grid in Kubernetes? Any help with troubleshooting the 403 error or configuring the video upload process would be greatly appreciated.