Bug#1035902: obs-studio: NVENC codec fails unless I use ffmpeg to encode a video first

[Rishi Cutchin]

Package: obs-studio
Version: 29.0.2+dfsg-1+b1
Severity: normal
X-Debbugs-Cc: rishin...@gmail.com

Dear Maintainer,


Run obs-studio, and attempted to record a video with NVENC selected as
the encoder.

Fails with this error:
rishi@tripodhost:~$ [h264_nvenc @ 0x56294719dfc0] dl_fn->cuda_dl->cuInit(0) failed -> CUDA_ERROR_U_SNKNOWN: unknown error
warning: [NVENC encoder] nvenc_create_internal failed, trying again without Psycho Visual Tuning
info: ---------------------------------
info: [FFmpeg NVENC encoder: 'advanced_video_recording'] settings:
encoder: NVIDIA NVENC H.264 (FFmpeg)
rate_control: CBR
bitrate: 2500
cqp: 0
keyint: 250
preset: p5
tuning: hq
multipass: qres
profile: high
width: 1920
height: 1080
b-frames: 2
psycho-aq: 0
GPU: 0

[h264_nvenc @ 0x5629471a2c80] dl_fn->cuda_dl->cuInit(0) failed -> CUDA_ERROR_UNKNOWN: unknown error

But after encoding a video with ffmpeg on the command line with nvenc:
$ffmpeg -i output.mp4 -c:v h264_nvenc -b:v 1M outputtwo.mp4
Subsequent attempts to use NVENC on obs succeed.

-- System Information:
Debian Release: 12.0
APT prefers testing
APT policy: (990, 'testing'), (500, 'testing-security'), (50, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.1.0-8-amd64 (SMP w/4 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages obs-studio depends on:
ii libavcodec59 7:5.1.3-1
ii libavdevice59 7:5.1.3-1
ii libavformat59 7:5.1.3-1
ii libavutil57 7:5.1.3-1
ii libc6 2.36-9
ii libcurl3-gnutls 7.88.1-9
ii libfontconfig1 2.14.1-4
ii libfreetype6 2.12.1+dfsg-5
ii libgcc-s1 12.2.0-14
ii libjansson4 2.14-2
ii libluajit-5.1-2 2.1.0~beta3+git20220320+dfsg-4.1
ii libmbedcrypto7 2.28.3-1
ii libmbedtls14 2.28.3-1
ii libmbedx509-1 2.28.3-1
ii libobs0 29.0.2+dfsg-1+b1
ii libpci3 1:3.9.0-4
ii libpulse0 16.1+dfsg1-2+b1
ii libpython3.11 3.11.2-6
ii libqt5core5a 5.15.8+dfsg-7
ii libqt5gui5 5.15.8+dfsg-7
ii libqt5network5 5.15.8+dfsg-7
ii libqt5svg5 5.15.8-2
ii libqt5widgets5 5.15.8+dfsg-7
ii libqt5xml5 5.15.8+dfsg-7
ii librist4 0.2.7+dfsg-1
ii libspeexdsp1 1.2.1-1
ii libsrt1.5-openssl 1.5.1-1
ii libstdc++6 12.2.0-14
ii libswscale6 7:5.1.3-1
ii libudev1 252.6-1
ii libv4l-0 1.22.1-5+b2
ii libva-drm2 2.17.0-1
ii libva2 2.17.0-1
ii libx11-6 2:1.8.4-2
ii libx264-164 2:0.164.3095+gitbaee400-3
ii libxcb-composite0 1.15-1
ii libxcb-randr0 1.15-1
ii libxcb-shm0 1.15-1
ii libxcb-xfixes0 1.15-1
ii libxcb-xinerama0 1.15-1
ii libxcb1 1.15-1
ii python3 3.11.2-1+b1
ii python3.11 3.11.2-6

Versions of packages obs-studio recommends:
ii obs-plugins 29.0.2+dfsg-1+b1

Versions of packages obs-studio suggests:
ii pkexec 122-3
ii policykit-1 122-3
pn v4l2loopback-dkms <none>

-- no debconf information

[IOhannes m zmölnig]

Package: obs-studio
Followup-For: Bug #1035902

Hi,

i cannot reproduce this problem.

i'm using ffmpeg_7:6.1.1-1 and nvidia-driver_535.43.02-1 (from
experimental, as the version in stable currently cannot be used by
ffmpeg to run nvenc).

Does the problem still exist on your side?

-- System Information:
Debian Release: trixie/sid
APT prefers unstable-debug
APT policy: (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'stable-updates'), (500, 'stable-security'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')

Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.6.9-amd64 (SMP w/24 CPU threads; PREEMPT)

Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE

Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set

Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages obs-studio depends on:

ii libavcodec60 7:6.1.1-1
ii libavdevice60 7:6.1.1-1
ii libavformat60 7:6.1.1-1
ii libavutil58 7:6.1.1-1
ii libc6 2.37-13
ii libcurl3-gnutls 8.5.0-2
ii libfontconfig1 2.14.2-6+b1
ii libfreetype6 2.13.2+dfsg-1+b1
ii libgcc-s1 13.2.0-9
ii libglx0 1.7.0-1
ii libjansson4 2.14-2+b2
ii libluajit-5.1-2 2.1.0~beta3+git20220320+dfsg-4.1
ii libmbedcrypto7 2.28.6-1
ii libmbedtls14 2.28.6-1
ii libmbedx509-1 2.28.6-1
ii libobs0 30.0.2+dfsg-2
ii libopengl0 1.7.0-1
ii libpci3 1:3.10.0-2
ii libpulse0 16.1+dfsg1-3
ii libpython3.11 3.11.7-2
ii libqrcodegencpp1 1.8.0-1.2
ii libqt6core6 6.4.2+dfsg-20
ii libqt6gui6 6.4.2+dfsg-20
ii libqt6network6 6.4.2+dfsg-20
ii libqt6svg6 6.4.2-4
ii libqt6widgets6 6.4.2+dfsg-20
ii libqt6xml6 6.4.2+dfsg-20
ii librist4 0.2.10+dfsg-1
ii libspeexdsp1 1.2.1-1
ii libsrt1.5-openssl 1.5.3-1
ii libstdc++6 13.2.0-9
ii libswscale7 7:6.1.1-1
ii libudev1 255.2-4
ii libv4l-0 1.26.1-2+b1
ii libva-drm2 2.20.0-2
ii libva2 2.20.0-2
ii libx11-6 2:1.8.7-1
ii libx264-164 2:0.164.3095+gitbaee400-3+b2

ii libxcb-composite0 1.15-1
ii libxcb-randr0 1.15-1
ii libxcb-shm0 1.15-1
ii libxcb-xfixes0 1.15-1
ii libxcb-xinerama0 1.15-1
ii libxcb1 1.15-1

ii libxkbcommon0 1.6.0-1
ii python3 3.11.6-1
ii python3.11 3.11.7-2
ii qt6-image-formats-plugins 6.4.2-5
ii qt6-wayland 6.4.2-5

Versions of packages obs-studio recommends:

ii obs-plugins 30.0.2+dfsg-2

Versions of packages obs-studio suggests:

ii pkexec 123-3
ii policykit-1 123-3
ii v4l2loopback-dkms 0.12.7-2

-- no debconf information

[Sebastian Ramacher]

On 2024-01-11 10:13:41 +0100, IOhannes m zmölnig wrote:
> Package: obs-studio
> Followup-For: Bug #1035902
>
> Hi,
>
> i cannot reproduce this problem.
>
> i'm using ffmpeg_7:6.1.1-1 and nvidia-driver_535.43.02-1 (from
> experimental, as the version in stable currently cannot be used by
> ffmpeg to run nvenc).

Current ffmpeg requires nvidia-driver from experimental for nvenc. This
issue will be fixed once nvidia-driver from experimental is uploaded to
unstable.

Cheers
--
Sebastian Ramacher