Strange JSON error with the 5.0.2 nix package

[Bruno Bzeznik]

Hello,

I recently upgraded my irods servers from 4.3.4 to the latest 5.0.2 version using Debian packages. I don't have any problem with the Debian packages, servers and clients are ok once the environment.json file hase been updated.

We are using the pam_password auth scheme.

I used to provide the icommands on every nodes of my clusters (which are heterogeneous in term of OS version) through the Nix packages. But the latest version, while compiling correctly, leads to this error when using iinit (or any icommand)

$ iinit
Connecting as xxxxx#zzzzz to zzzzz.u-ga.fr:1247 ...                                            
Level 0: JSON error occurred while authenticating user [xxxxx] [[json.exception.type_error.305] cannot use operator[] with a string argument with array]

What could cause that? How can I debug this? I tried export IRODS_LOG_LEVEL=9 and above, but I get nothing very useful:

[...]
DEBUG: load_plugin - calling plugin_factory() in [/nix/store/q9g68vm8if3jy2ckgncwvdk3a1xinmpz-irods-5.0.2/lib/irods/plugins/api/libirods_touch_client.so]
DEBUG: load_plugin - loaded [irods_touch_client]
DEBUG: captured env [irods_log_level]-[9]
DEBUG: load_plugin - calling plugin_factory() in [/nix/store/q9g68vm8if3jy2ckgncwvdk3a1xinmpz-irods-5.0.2/lib/irods/plugins/authentication/libirods_auth_plugin-pam_password_client.so]
DEBUG: load_plugin - loaded [irods_auth_plugin-pam_password_client]
Level 0: JSON error occurred while authenticating user [xxxxx] [[json.exception.type_error.305] cannot use operator[] with a string argument with array]
DEBUG: captured env [irods_log_level]-[9]
[...]

Here's my irods_environment.json file (working fine with the debian packages clients):

{
    "irods_authentication_scheme": "pam_password",
    "irods_client_server_negotiation": "request_server_negotiation",
    "irods_client_server_policy": "CS_NEG_REQUIRE",
    "irods_connection_pool_refresh_time_in_seconds": 300,
    "irods_cwd": "/zzzzz/home/xxxxx",
    "irods_default_hash_scheme": "SHA256",
    "irods_default_number_of_transfer_threads": 4,
    "irods_default_resource": "xxxx",
    "irods_encryption_algorithm": "AES-256-CBC",
    "irods_encryption_key_size": 32,
    "irods_encryption_num_hash_rounds": 16,
    "irods_encryption_salt_size": 8,
    "irods_home": "/zzzzz/home/xxxxx",
    "irods_host": "xxxxx.u-ga.fr",
    "irods_port": 1247,
    "irods_ssl_ca_certificate_file": "/etc/irods/chain.pem",
    "irods_ssl_verify_server": "cert",
    "irods_user_name": "xxxxx",
    "irods_zone_name": "zzzzz"
}

[Bruno Bzeznik]

EDIT: The Nix package source code:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/by-name/ir/irods/package.nix
https://github.com/NixOS/nixpkgs/blob/master/pkgs/by-name/ir/irods-icommands/package.nix

I also tried to recompile with '"IRODS_BUILD_WITH_CLANG" true' as suggested by this link: https://github.com/ll4strw/irods_easybuild/tree/main
but I keep getting the same JSON error.

[Terrell Russell]

Hi Bruno,

Did you see this error with the 5.0.1 recipe before updating to 5.0.2?

The Consortium doesn't test building with GCC quite yet, so that 'false' might be worth investigating more.

https://github.com/NixOS/nixpkgs/blob/7b781d7c0829b36dd4fff8f873fd75f56c5bcfa5/pkgs/by-name/ir/irods/package.nix#L81

I'm not sure what the comment just above it means...

"# We already use Clang in the `stdenv`."

Terrell

 

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/irod-chat/e22fafbd-4939-4f25-b161-f1eb1509ba00n%40googlegroups.com.

[Bruno Bzeznik]

I upgraded directly from 4.3.4 to 5.0.2 and the problem does not come from the servers by the way as the Debian packages work ok, both server and client 5.0.2. The problem only occurs when I try to use the clients (icommands) from the Nix 5.0.2 package.

This comment "# We already use Clang in the `stdenv`"  means that as we use llvmPackages.stdenv.mkDerivation to build, we already use Clang as the compiler.

But I tested to set "IRODS_BUILD_WITH_CLANG to True" and the problem is the same.

Do you think that this Json error might come from a dependency built with GCC? Which one? How could we add more debug to see exactly what happens to the Json data?

[Markus Kitsinger]

We build our dependencies with GCC. We're on the path to switching to distro-provided compilers, but for now, we still build irods, irods clients, and irods plugins with clang 16. So I doubt the problem is from a GCC-build dependency.

While I'd like to add stack traces to our exception handling, std::stacktrace is a C++23 feature that is still experimental in libstdc++ 14, and boost.stacktrace didn't add from_current_exception until 1.85 (and it requires building against libbacktrace, which can be challenging).

My initial suspicion was that this problem was caused by an incompatibility with nlohmann-json 3.12, but I can't reproduce the issue even when building irods and icommands against this version. So I would suggest patching to add some diagnostic output.

The exception handler that's being triggered is here: https://github.com/irods/irods/blob/5.0.2/lib/core/include/irods/authentication_plugin_framework.hpp#L231-L240

I would suggest adding a few lines in the try block above to print out the state of the json objects and watch how it changes.

[Bruno Bzeznik]

Thanks for your replies! 
I managed to add some debug (thanks to gpt-oss!) and I could determine that the error occurs in this block: https://github.com/irods/irods/blob/e16f20424192f24db2dcc3dbaaa95cee0334a2b6/lib/core/include/irods/authentication_plugin_framework.hpp#L189-L192

Here are the json objects dumps before each line, when doing a simple `ils`:

/// [JSON DEBUG] _ctx = {
///     "scheme": "pam_password"
/// }

        json req{_ctx}; 

/// [JSON DEBUG] req = [
///     {
///         "scheme": "pam_password"
///     }
/// ]

        json resp; 

        req["scheme"] = scheme;   // <- Crash here with "cannot use operator[] with a string argument with array"

So the problem is that `req` is instanciated as an array with our Nix package compilation chain. 

I'm going try to compile with an older version of nlohmann-json...

[Bruno Bzeznik]

Same thing with nlohmann-json v3.11.13

Shouldn't we have `json req( _ctx );` in place of `json req{ _ctx };` ? in this code? https://github.com/irods/irods/blob/e16f20424192f24db2dcc3dbaaa95cee0334a2b6/lib/core/include/irods/authentication_plugin_framework.hpp#L189

[Bruno Bzeznik]

https://github.com/nlohmann/json/issues/2583

https://json.nlohmann.me/home/faq/#brace-initialization-yields-arrays

[Alan King]

Hi Bruno,

What you found seems correct. Please try out that change and let us know if things improve. Feel free to open a pull request with the fix and we will include it in the next release: https://github.com/irods/irods/pulls

Alan

To view this discussion visit https://groups.google.com/d/msgid/irod-chat/3003d40d-7985-494e-b641-f2e0a0d84902n%40googlegroups.com.

--

Alan King

Senior Software Developer | iRODS Consortium

[Bruno Bzeznik]

Yes, `json req( _ctx );` fixes the bug at line 189, and the code now go further, but still fails with the same error (cannot use operator[] with a string argument with array]), at this line:

https://github.com/irods/irods/blob/e16f20424192f24db2dcc3dbaaa95cee0334a2b6/lib/core/include/irods/authentication_plugin_framework.hpp#L196

resp = auth->call(_comm, *next_operation, req)

But I can't figure out why... probably the same issue somewhere else in the code

[Markus Kitsinger]

I've opened https://github.com/irods/irods/issues/8839 to address this. There's some info in there I found from the chunk of time I spent experimenting today.