Linking data files

[Kenneth Porter]

I saw mention of linking to the Windows manifest on the devel list (part of
a pull request) and wondered if there's some machinery useful for pulling
generated data files into my executable.

I've got some XML files in one DLL and a JSON file in another application
that are parsed at startup into C++ structures. That's partly done as a
convenience to the non-coders on the team to tweak values used in our
robotics work. (We try to make this as data-driven as possible so that a
single binary works with many products.) It would be nice if I could pull
these text files into the shipping binaries, possibly with some way to
override them with an external file for patching in the field.

[Steve Barnes]

Hi Kenneth,

This is not really a wx mechanism as much as a question of how you are preparing your binaries for distribution and then distributing them.

- If you are preparing as a zip file (or other archive), then simply add the configuration files with their default values into the archive.
- If you are creating an installer such as Windows MSI, etc., then just about every installer creation tool that I have come across has mechanisms for bundling and placing data & config files.
- If you are allowing the end users to download the binaries from somewhere like GitHub releases then you can simply add them to the releases (again a feature of every hosting service that I have used).
- If you have written your own installer then it should be relatively simple to add these files and the location(s) that you want to store & load them from.

In any of the above it should be simple to add the default files to your build & release mechanism.

I would encourage you to look at the mechanisms in wxStandardPaths and wxConfig and either have your installer place the configuration in the platform specific configuration directory and change your load mechanism to load them from there - this will allow for cases where your executables are placed in directories that the users don't have write access to but need to change configuration values.

Hope that is some help.

Steve Barnes

--
Please read https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.wxwidgets.org%2Fsupport%2Fmlhowto.htm&data=05%7C01%7C%7C74c4d699da3f412c615508da2a379921%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637868715518101445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R3YJhg1QTrLVbSdSzZLiIx1D%2B%2Ftx7kTt24uZAzR6OQU%3D&reserved=0 before posting.
---
You received this message because you are subscribed to the Google Groups "wx-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wx-users+u...@googlegroups.com.
To view this discussion on the web visit https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fwx-users%2F1E205564420CBDEA8DE58C38%2540%255B192.168.69.69%255D&data=05%7C01%7C%7C74c4d699da3f412c615508da2a379921%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637868715518101445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xBxfGV%2FRQdMU9S9fw03Vd0n3LZUf03ym3jyqRrKd7cw%3D&reserved=0.

[Kenneth Porter]

On 4/29/2022 9:58 PM, Steve Barnes wrote:
> In any of the above it should be simple to add the default files to your build & release mechanism.

I don't want the files to be separate deliverables in a package. I want
them to be bound inside the executable or DLL, similar to the way icons
and manifests are delivered as an integral part of the final binary. I'd
still consult an external file as a "back door" for a savvy user. While
I'm currently delivering on Windows, I expect ELF has a similar
mechanism for other platforms.

[PB]

Hi,

On Saturday, April 30, 2022 at 8:58:08 AM UTC+2 SpareSimian wrote:

I don't want the files to be separate deliverables in a package. I want
them to be bound inside the executable or DLL, similar to the way icons
and manifests are delivered as an integral part of the final binary.

Why not use a resource then? In case you did not know, you can embed virtually anything there.

For example, in the .rc file, add

APP_XML_DATA RCDATA "app.xml"

where APP_XML_DATA is the identifier you will use to load this resource,  RCDATA indicates this is a user resource type, and "app.xml" is the path to your file.

And now you can load the resource easily in wxWidgets with wxLoadUserResource(), for example load it into wxCharBuffer

wxCharBuffer buffer = wxCharBuffer::CreateNonOwned(wxLoadUserResource("APP_XML_DATA", RT_RCDATA));
// use the buffer as read-only

or into wxMemoryInputStream that can be used as any input stream

const void* xmlData = nullptr;
size_t      xmlDataSize = 0;

if ( wxLoadUserResource(&xmlData, &xmlDataSize, "APP_XML_DATA", RT_RCDATA) )
{
    wxMemoryInputStream(xmlData, xmlDataSize);
    // read from the stream
}

Obviously, any resource can still be viewed and modified, using a tool such as Resource Hacker. 

I am Windows only, so I have no idea how this would be done on other platforms; however, I do not think MacOS or Linux have a similar mechanism for embedding data into the executable or library.

Regards,

PB

[PB]

On Saturday, April 30, 2022 at 9:54:40 AM UTC+2 PB wrote:

wxCharBuffer buffer = wxCharBuffer::CreateNonOwned(wxLoadUserResource("APP_XML_DATA", RT_RCDATA));

Oops, I made a mistake, wxCharBuffer must be created as owned, since that overload of wxLoadUserResource() returns a copy of the resource data, so the code must be

     wxCharBuffer buffer = wxCharBuffer::CreateOwned(wxLoadUserResource("APP_XML_DATA", RT_RCDATA));

I also forgot to actually declare a variable when creating wxMemoryInputStream, the code should be

     wxMemoryInputStream xmlStream(xmlData, xmlDataSize);

Sorry,
PB

[Gunter Königsmann]

If you don't have excessive amounts of data xxd or similar tools can make .h files out of them that can be compiled into the executable. I did so for wxMaxima, since every single packager choose a different location for the files (there is a flavour of Linux with a windows-like directory structure. For the Mac there is fink, ports, homebrew and the app bundle, debian, archlinux and red hat differ slightly,... ...and all of them expected my program to find the files in their new location; wxWidgets can load the .h files from xxd into a wxMemoryBuffer. There are other programs like xxd. For wxMaxima we switched to a cmake-only mechanism, instead that does the same.

On the Mac any GUI application is required to be an app bundle (which is more or less a .zip file containing the executable and all of it's resources) and on linux there would have been ways to append zip files to binaries (and for the binary to find them afterwards). Additionally there are flatpacks, which is a zip file appended to a binary that unpacks your binary and all of the resources and for your application makes them look like they were in the system's file system. But none of these methods is really multiplatform so I didn't use them.

Kind regards,

Gunter

[Lauri Nurmi]

Gunter Königsmann kirjoitti 30.4.2022 klo 12:23:
> On the Mac any GUI application is required to be an app bundle (which
> is more or less a .zip file containing the executable and all of it's
> resources)

An app bundle is a folder structure with subfolders named after a
specific convention, and files in specific places. App bundles are
sometimes distributed as .zip files, but an app bundle in itself is not
a .zip file. And an app bundle inside a .zip cannot be executed without
first extracting it from the .zip.


L

[Kenneth Porter]

On 4/30/2022 12:54 AM, PB wrote:
> I am Windows only, so I have no idea how this would be done on other
> platforms; however, I do not think MacOS or Linux have a similar
> mechanism for embedding data into the executable or library.

Thanks. It looks like only Windows has a formal mechanism for bundling
runtime data in the binary (the resource).

The ELF format used by Linux has "sections" for code, data, and other
stuff, and includes a section type of "note" that could be used. But I
haven't found any effort to use it for the kinds of things that appear
in Windows resources. It does get used for the Gnu ABI version and build ID.

https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-18048.html

https://stackoverflow.com/questions/9401217/is-the-elf-notes-section-really-needed

I'd guess the Gnu linker has some code buried within that builds the
notes section and perhaps it could be leveraged to add arbitrary data
like resources, but I haven't found anyone proposing that.

[Igor Korot]

Hi,

On Sat, Apr 30, 2022 at 4:24 PM Kenneth Porter <sh...@sewingwitch.com> wrote:
>
> On 4/30/2022 12:54 AM, PB wrote:
> > I am Windows only, so I have no idea how this would be done on other
> > platforms; however, I do not think MacOS or Linux have a similar
> > mechanism for embedding data into the executable or library.
>
> Thanks. It looks like only Windows has a formal mechanism for bundling
> runtime data in the binary (the resource).

You can put the data files in the Application Bundle Resources folder.
Or maybe use the Data folder of the Bundle.

Check the official Apple documentation for that.

Thank you.

>
> The ELF format used by Linux has "sections" for code, data, and other
> stuff, and includes a section type of "note" that could be used. But I
> haven't found any effort to use it for the kinds of things that appear
> in Windows resources. It does get used for the Gnu ABI version and build ID.
>
> https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-18048.html
>
> https://stackoverflow.com/questions/9401217/is-the-elf-notes-section-really-needed
>
> I'd guess the Gnu linker has some code buried within that builds the
> notes section and perhaps it could be leveraged to add arbitrary data
> like resources, but I haven't found anyone proposing that.
>
>

> --
> Please read https://www.wxwidgets.org/support/mlhowto.htm before posting.

> ---
> You received this message because you are subscribed to the Google Groups "wx-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to wx-users+u...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/wx-users/48928e08-0fa4-a509-4c8f-38dbcf373168%40sewingwitch.com.