setting dlopen flags while loading addons

[Wayne]

While writing a node addon I ran into what I believe is the same
problem (with a different underlying library) that was reported by
@springmeyer in issue 436,
https://github.com/joyent/node/issues#issue/436

My problem is a result of the dlopen call in node.cc being called only
with the flag RTLD_LAZY. By default, if no other flag is specified,
dlopen assumes RTLD_LOCAL which means any symbols in the loaded file
are local to that library and not available when resolving references
from subsequently loaded libraries. The alternative, RTLD_GLOBAL,
makes loaded symbols globally available.

That seems like sensible default behavior (avoid polluting the global
namespace). However, if the file being loaded in turn uses dlopen to
load another file, and that second file has a symbol referencing the
first, an attempt to load the second file will fail.

That's exactly the problem in my case. I linked my node addon to a
third party library I am wrapping, and that library attempts to load
plugins on the fly using dlopen. There is a plugin registry in the
library and since node loads my addon with RTLD_LOCAL the registry is
not visible to the plugins loaded in the library. Failure to resolve
the plugin registry symbol properly causes a crash. If node's dlOpen
uses RTLD_GLOBAL, the addon works fine.

So in some cases the sensible default behavior can be problematic. I
liked the idea suggested by Dane in issue 436. His suggestion was to
emulate python and add a function in sys that lets you set the flags
for dlopen before loading the module. I implemented his suggestion,
putting the variable in process rather than sys, so that you can do
the following,

var c = require('constants');
var oldFlags = process.dlOpenFlags;
process.dlOpenFlags = c.RTLD_LAZY | c.RTLD_GLOBAL;
var x = require('some_dynamically_loaded_node_extension');
process.dlOpenFlags = oldFlags;

and submitted a pull request, #855:
https://github.com/joyent/node/pull/855

I think this change or one like it is necessary. Some alternatives
with the reason for rejecting:
- Take care of it in the addon itself: You're still going to have to
do use RTLD_GLOBAL which means you're potentially polluting the global
namespace, but now it's buried in the addon which makes it more
difficult for the end node user to track down problems if they occur.
- "Fix" the underlying library being wrapped: Lots of problems with
that--might make the lib incompatible with other applications, forces
an upgrade of the lib, you'd probably be modifying stable code that
changes/should be changed rarely with the risk of introducing new
bugs.
- Figure out some way to flag an addon as requiring RTLD_GLOBAL and do
that switch within the node dlOpen function (not sure of a smart way
to do this but for example you could have a ".node-global" suffix or
something instead of just '.node'): same problem as first approach,
you pollute the namespace and it's still kind of buried behind the
scenes.
- Load all addons with RTLD_GLOBAL instead of RTLD_LOCAL: that doesn't
seem like a very bright approach since most things will be fine with
the more conservative RTLD_LOCAL.

The apparent disadvantage with Dane's approach, of course, is that
every time you want to use such an addon you are required to set the
dlOpenFlags parameter before the require. However you thereby force
the issue out into the open at a level of the code making a plain node
coder aware that he or she is doing something unusual. If two such
modules are being loaded and they have a conflicting symbol I would
think it'll be much easier to track things down.

By the way if I understand Dane correctly he fixed his particular
problem by switching RTLD_LAZY to RTLD_NOW. That doesn't change the
above, it seems to me to just further support the more flexible
approach of being able to set the dlopen flags directly at the
javascript level versus handling it in some other way.

Thoughts?

[Marco Rogers]

Wow, thanks for the thorough write up Wayne. I'm researching how node handles addons right now and this was very informative.

I don't yet know enough to comment on your proposal. But I will comment on the API :)  I don't like the idea of hanging random properties off of process that have such a critical effect on behavior. Especially since it's directly writable.

Having this be a function of require() seems to make more sense. There is already require.paths that affects what happens when you require modules. Maybe we add require.dlOpenFlags. The other thing that bothers me is that you have to make sure to preserve oldFlags and reset it. This feels too PHP-ish to me and I hope we can do better. I would like an API that provided a temporary context for require that uses my configuration options.

    require.withConfig('mylib', {dlOpenFlags: c.RTLD_LAZY | c.RTLD_GLOBAL});


Note I intentionally made this different from adding optional parameters to require itself.

    require('mylib', {dlOpenFlags: c.RTLD_LAZY | c.RTLD_GLOBAL});

I'm sure there will be some type of objection to that.

:Marco

[Wayne]

If you happen to get bit by this and no mechanism for setting dlopen
flags makes it into node, just try the first alternative, i.e. take
care of it in your addon by calling dlopen with appropriate flags
set. That should work.

[Ilya Usvyatsky]

Unfortunately, the real issue is that if your addon uses dlopen to load a module, said module won't see symbols from your addon because the addon is loaded with RTLD_LOCAL scope.
I like the approach that Wayne suggested, but since the thread has been dormant for almost four years, and the proposal is not implemented, I assume there is another way of helping the situation.
I just wonder, what is it.