b2ff380d5b (master): [ruby/openssl] Fix memory leak if ossl_bn_new() fails

[ndossche]

ndossche 2026-02-24 22:12:19 +0000 (Tue, 24 Feb 2026)

New Revision: b2ff380d5b

https://github.com/ruby/ruby/commit/b2ff380d5b

Log:
[ruby/openssl] Fix memory leak if ossl_bn_new() fails

When that call fails, the `bn` BIGNUM is never freed in
asn1integer_to_num(). To solve this, use rb_protect().

Example Valgrind report:
```
32 (24 direct, 8 indirect) bytes in 1 blocks are definitely lost in loss record 11,113 of 25,910
malloc (at /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
CRYPTO_zalloc (at /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
BN_new (at /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
BN_bin2bn (at /usr/lib/x86_64-linux-gnu/libcrypto.so.3)
<unknown stack frame>
*asn1integer_to_num (ossl_asn1.c:136)
*asn1integer_to_num_i (ossl_asn1.c:165)
rb_protect (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
*decode_int (ossl_asn1.c:356)
*int_ossl_asn1_decode0_prim (ossl_asn1.c:777)
*ossl_asn1_decode0 (ossl_asn1.c:936)
*ossl_asn1_decode_all (ossl_asn1.c:1058)
<unknown stack frame>
<unknown stack frame>
<unknown stack frame>
rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
<unknown stack frame>
<unknown stack frame>
rb_catch_obj (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
<unknown stack frame>
<unknown stack frame>
<unknown stack frame>
rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
rb_yield (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
rb_ary_each (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
<unknown stack frame>
<unknown stack frame>
<unknown stack frame>
rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
rb_yield (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
rb_ary_each (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
<unknown stack frame>
<unknown stack frame>
<unknown stack frame>
rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
rb_yield (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
rb_ary_each (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
<unknown stack frame>
<unknown stack frame>
<unknown stack frame>
rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
<unknown stack frame>
<unknown stack frame>
rb_catch_obj (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
<unknown stack frame>
<unknown stack frame>
<unknown stack frame>
rb_vm_exec (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
rb_vm_invoke_proc (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
rb_proc_call_kw (at /usr/lib/x86_64-linux-gnu/libruby-3.2.so.3.2.3)
```

https://github.com/ruby/openssl/commit/309c55d755

Modified files:
ext/openssl/ossl_asn1.c