[std-proposals] Attributes useful for shared objects (import, export, symbol locations)

[Antony Polukhin]

Hello,

There's a bunch of problems with dynamic loading entities defined in C++ shared libraries:

1) No portable way to define symbol as being imported/exported. All the C++ libraries are forced to reinvent BOOST_SYMBOL_EXPORT and BOOST_SYMBOL_IMPORT like macro [1].

2) All modern binary formats store symbols in sections that are not currently controlled from C++, but provide a rich set of useful functionality. For example Linux kernel developers use sections to store pointers to specific entry points (macro __init http://lxr.free-electrons.com/source/include/linux/init.h), store data that must be changed at load (https://lwn.net/Articles/164121/) and so on.

I'm proposing to add following attributes to C++ Standard:

* [[import]] - for importing symbol (same as BOOST_SYMBOL_IMPORT)

* [[export]] - for exporting symbol in default section with default C++ mangling (same as BOOST_SYMBOL_EXPORT)

* [[section "section name"]] - places the symbol into section with specified name, constructs section if it does not exist(close to BOOST_DLL_SECTION [2])

Here are some usage examples:

namespace boost {

    // exporting function with C++ mangled name.

    [[export]] void export_me(std::sting, variant<int, short>);

    // importing function with C++ mangled name

    [[import]] void import_me(variant<int, char>, int);

    extern "C" {

        [[export]] void export_me_c(std::sting v1, variant<int, short> v2) {

            export_me(v1, v2);

        }

        // Now we can load function from shared library using "export_me_c" name:

        //   dlsym(dlopen(library), "export_me_c");

        // importing function with C mangled name

        [[import]] void import_me_c(variant<int, char>, int);

    }

    // Creates section named to_strip and places variable `data` into that section

    [[section "to_strip"]] char data[] = "Some minor data that will be stripped later";

}

////////////

// Now we can make nicer macro for header files that declare API, for example:

#if EXPORTING

#   define API [[export]]

#else

#   define API [[import]]

#endif

#define PLUGIN extern "C" API [[section "plugin"]]

namespace my_namespace {

    // Shared object that contains "foo2" could be dynamically loaded or dynamically linked.

    PLUGIN void foo2(std::sting);

    // Class that will be dynamically linked

    class API foo_class {

        // ...

    };

}

All the proposed attributes are either simple to implement or are already supported by modern compilers (MSVC has dllimport, dllexport, __pragma(section); GCC has dllimport, dllexport, section, visibility).

Any comments are wellcomed!

[1] Boost C++ libraries. Macros controlling shared library symbol visibility

http://www.boost.org/doc/libs/1_60_0/libs/config/doc/html/boost_config/boost_macro_reference.html#boost_config.boost_macro_reference.macros_for_libraries_with_separate_source_code.macros_controlling_shared_library_symbol_visibility

[2] Boost C++ libraries. Macro BOOST_DLL_SECTION

http://apolukhin.github.io/Boost.DLL/BOOST_DLL_SECTION.html

--

Best regards,
Antony Polukhin

[Magnus Fromreide]

On Mon, Jan 18, 2016 at 10:37:11PM +0300, Antony Polukhin wrote:
> Hello,
>
> There's a bunch of problems with dynamic loading entities defined in C++
> shared libraries:
> 1) No portable way to define symbol as being imported/exported. All the C++
> libraries are forced to reinvent BOOST_SYMBOL_EXPORT and
> BOOST_SYMBOL_IMPORT like macro [1].
> 2) All modern binary formats store symbols in sections that are not
> currently controlled from C++, but provide a rich set of useful
> functionality. For example Linux kernel developers use sections to store
> pointers to specific entry points (macro __init
> http://lxr.free-electrons.com/source/include/linux/init.h), store data that
> must be changed at load (https://lwn.net/Articles/164121/) and so on.
>
>
> I'm proposing to add following attributes to C++ Standard:
> * [[import]] - for importing symbol (same as BOOST_SYMBOL_IMPORT)
> * [[export]] - for exporting symbol in default section with default C++
> mangling (same as BOOST_SYMBOL_EXPORT)
> * [[section "section name"]] - places the symbol into section with
> specified name, constructs section if it does not exist(close to
> BOOST_DLL_SECTION [2])

* How does this interact with modules?
* Do you intend to bring dlopen into the standard as well?
* What about binary formats that don't support some of these features,
are all of them supposed to be optionally supported?
* What does importing actually mean? When using ELF there are four different
modes a symbol could be imported in with different features and effects.

/MF

[Matthew Woehlke]

On 2016-01-18 14:37, Antony Polukhin wrote:
> There's a bunch of problems with dynamic loading entities defined in C++
> shared libraries:
> 1) No portable way to define symbol as being imported/exported. All the C++
> libraries are forced to reinvent BOOST_SYMBOL_EXPORT and
> BOOST_SYMBOL_IMPORT like macro [1].

CMake¹ might beg to differ ;-).

> I'm proposing to add following attributes to C++ Standard:
> * [[import]] - for importing symbol (same as BOOST_SYMBOL_IMPORT)
> * [[export]] - for exporting symbol in default section with default C++
> mangling (same as BOOST_SYMBOL_EXPORT)

While I don't object to this as such, I think it's worth noting that it
doesn't solve the problem of needing to use macros to decorate symbols.
As long as platforms (Windows/PExx) define these differently, library
authors must use a macro so that it is defined differently when building
the library vs. someone else using the library.

This borders on being a build system issue, with which the C++ standard
usually does not get involved, and we already have many good solutions
for that problem (e.g. CMake).

(¹ https://cmake.org/cmake/help/v3.4/module/GenerateExportHeader.html)

--
Matthew

[Sean Middleditch]

On Monday, January 18, 2016 at 12:03:01 PM UTC-8, Matthew Woehlke wrote:

On 2016-01-18 14:37, Antony Polukhin wrote:
> I'm proposing to add following attributes to C++ Standard:
> * [[import]] - for importing symbol (same as BOOST_SYMBOL_IMPORT)
> * [[export]] - for exporting symbol in default section with default C++
> mangling (same as BOOST_SYMBOL_EXPORT)

While I don't object to this as such, I think it's worth noting that it
doesn't solve the problem of needing to use macros to decorate symbols.
As long as platforms (Windows/PExx) define these differently, library
authors must use a macro so that it is defined differently when building
the library vs. someone else using the library.

It has been publicly mentioned that with the modules feature, this is no longer the case. Symbols need to merely be marked as being available for DLL linkage (using __declspec(dllexport) only) and then the linker automatically exports symbols in modules linked into the DLL and imports symbols otherwise.

See Andrew Pardoe's response to Matthias in the comments on http://blogs.msdn.com/b/vcblog/archive/2015/12/03/c-modules-in-vs-2015-update-1.aspx

I think that should be taken into account for this whole feature idea. :)

--
Matthew

[Thiago Macieira]

On Monday 18 January 2016 20:50:10 Magnus Fromreide wrote:
> * What does importing actually mean? When using ELF there are four different
> modes a symbol could be imported in with different features and effects.

Off-topic, but this is a missed optimisation in ELF. I've been trying to get
the ABI folks to come up with a proper way to import. GCC 6's -fno-plt is
half-way there.

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center
PGP/GPG: 0x6EF45358; fingerprint:
E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358

[Antony Polukhin]

2016-01-18 22:50 GMT+03:00 Magnus Fromreide <ma...@lysator.liu.se>:

* How does this interact with modules?

Both proposals use `import` as a keyword, however use cases are different and must not bother each other.

 

* Do you intend to bring dlopen into the standard as well?

Yes. But later, only after the Boost.DLL library will be tested by Boost users. Probably in half a year.

 

* What about binary formats that don't support some of these features,
  are all of them supposed to be optionally supported?

Probably will just follow the existing GCC practice for unsupported attributes: warn about them 

 

* What does importing actually mean? When using ELF there are four different
  modes a symbol could be imported in with different features and effects.

In GCC case it would be visibility=default, nonweak symbol. Or did I misunderstand the question/miss something?

[Antony Polukhin]

2016-01-19 0:23 GMT+03:00 Sean Middleditch <sean.mid...@gmail.com>:

It has been publicly mentioned that with the modules feature, this is no longer the case. Symbols need to merely be marked as being available for DLL linkage (using __declspec(dllexport) only) and then the linker automatically exports symbols in modules linked into the DLL and imports symbols otherwise.

See Andrew Pardoe's response to Matthias in the comments on http://blogs.msdn.com/b/vcblog/archive/2015/12/03/c-modules-in-vs-2015-update-1.aspx

I think that should be taken into account for this whole feature idea. :)

Yes. 

__declspec(dllexport) is a MSVC extension, that must be made part of the Standard and must follow the existing [[attributes syntax]]. And that's what the proposal is trying to achieve - make attributes for shared objects parts of the Standard. "dll" prefix was dropped from [[import]] and [[export]] to make proposal less platform specific, because on different platforms shared objects have different names (dll/so/dynlib).

[Sean Middleditch]

I'm quite aware.

What I was getting at is that with C++ modules we might not need
separate import and export attributes, at least on one major platform
that had forced us to explicitly import symbols and deal with separate
import/export macros. Your proposal should take that into account,
since modules will almost certainly be a part of C++ before your new
attributes ever would be.

>
> --
> Best regards,
> Antony Polukhin
>

> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/a/isocpp.org/d/topic/std-proposals/jzc7d_MQUZg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> std-proposal...@isocpp.org.
> To post to this group, send email to std-pr...@isocpp.org.
> Visit this group at
> https://groups.google.com/a/isocpp.org/group/std-proposals/.



--
Sean Middleditch
http://seanmiddleditch.com

[Antony Polukhin]

Oh, I've finally get that one. You're 100% right, [[import]] and [[export]] may become deprecated before even getting into the Standard.

How about following the GCC approach and using [[visible]] attribute? With [[visible]] all the export/import macro will go away. Just use [[visible]] if in binary that symbol must be available for dynamic load.

What do you and others think of it?

[Matthew Woehlke]

On 2016-01-20 13:09, Antony Polukhin wrote:
> Oh, I've finally get that one. You're 100% right, [[import]] and [[export]]
> may become deprecated before even getting into the Standard.
>
> How about following the GCC approach and using [[visible]] attribute? With
> [[visible]] all the export/import macro will go away. Just use [[visible]]
> if in binary that symbol must be available for dynamic load.
>
> What do you and others think of it?

...or [[exported]]? :-)

--
Matthew

[Sean Middleditch]

Honestly, I'd be a little hesitant to use anything remotely like
"import" or "export" just to avoid user confusion with modules.

>
> --
> Matthew

[Klemens Morgenstern]

If you build a binary, why wouldn't export be possible instead of [[visible]] ? 

If it's exported in the module, it must be visible in the binary. And if it's visible in the binary, why wouldn't it be exported in the module?

[David Krauss]

On 2016–02–13, at 7:25 PM, Klemens Morgenstern <klemens.m...@gmx.net> wrote:

If you build a binary, why wouldn't export be possible instead of [[visible]] ? 

If it's exported in the module, it must be visible in the binary. And if it's visible in the binary, why wouldn't it be exported in the module?

Module export is merely equivalent to presence in a header file. Under modules, headers are replaced by .cpp files wherein everything is exported. (Microsoft may use module interface membership as a linker hint, but that’s tangential. Other platforms have long switched automatically between importing and exporting.)

According to N4466, “Export declaration” ¶2, export implies extern. I wonder when you would need extern without export under modules. Maybe the keyword can actually be reclaimed, and then it would be free for specifying dynamic library interfaces.

Attributes are not supposed to be semantic, and dynamic library linkage may not be subtle enough to fit them. A static local declaration in a non-visible inline function may generate several variables, which is not conforming. The implementation of a dynamic library may define an extern function with the same signature as another library’s internal-but-extern function. Surely Microsoft’s latest compiler allows a DLL to use a module without exposing it to the world. This is because the ODR is at odds with the dynamic library distribution model.

The standard needs to admit that another linkage level exists inside the current shared object file — extern but multiply defined. Or equivalently, with modules, it will need to admit that a program can multiply contain one module. Considering the push for modules as a simplification that will make linkage obsolete, neither seems likely to add complexity anytime soon. (My 2¢.)

[Klemens Morgenstern]

Module export is merely equivalent to presence in a header file. Under modules, headers are replaced by .cpp files wherein everything is exported. (Microsoft may use module interface membership as a linker hint, but that’s tangential. Other platforms have long switched automatically between importing and exporting.)

True, and I think the bigger issue is the visibility attribute, which you do have on other platforms. And here's were I think, if it's declared export, it's visible. So, wouldn't it be smarter so reuse export when building a runtime-loaded shared libary?

 

According to N4466, “Export declaration” ¶2, export implies extern. I wonder when you would need extern without export under modules. Maybe the keyword can actually be reclaimed, and then it would be free for specifying dynamic library interfaces.

That depends on the actual implementation. That is, if module A needs a global variable from module B, but module B depends on A. Sloppy design for sure, but we don't know yet if we can compile this then. I don't think so.

It could even happen, that extern thereby becomes a bit more used for exactly those cases. 

 

The standard needs to admit that another linkage level exists inside the current shared object file — extern but multiply defined. Or equivalently, with modules, it will need to admit that a program can multiply contain one module. Considering the push for modules as a simplification that will make linkage obsolete, neither seems likely to add complexity anytime soon. (My 2¢.)

 Well afaik it doesn't consider loading libraries at runtime. However: once we have a binary format for modules which contains a lot of information, If there is then a need for dlls it maybe could be build atop modules, which would allow an amazing amount of information about the data. But that is mere speculation.

Currently, you don't even have the size of types in dlls.

[Antony Polukhin]

I've made a draft proposal for [[visible]] attribute http://apolukhin.github.io/papers/D0276R0:%20Attribute%20[[visible]].html

I'd appreciate any comments and suggestions.

[Thiago Macieira]

On sábado, 13 de fevereiro de 2016 21:54:47 PST David Krauss wrote:
> Other platforms have long switched automatically between importing and
> exporting

I wish they wouldn't. The MS DLL model may be cumbersome, but it generates the
best code.

[Antony Polukhin]

Big update of the proposal draft (2016-03-16). https://apolukhin.github.io/papers/D0276R0:%20Attribute%20[[visible]].html

Changes:

* big rewrite of the wording

* more experiments to determinate existing practices

* clarify behavior of attribute on nested classes and namespaces

* clarify behavior on platforms that do not support shared libraries

* better code highlighting and formatting

* multiple minor fixes 

[Antony Polukhin]

2016-02-14 6:30 GMT+03:00 Thiago Macieira <thi...@macieira.org>:

On sábado, 13 de fevereiro de 2016 21:54:47 PST David Krauss wrote:
> Other platforms have long switched automatically between importing and
> exporting

I wish they wouldn't. The MS DLL model may be cumbersome, but it generates the
best code.

Could you explain in more details, how the MS DLL model allows to generate a better code (comparing to a shared library compiled with -fvisibility=hidden on GCC or Clang)?