App crashes on startup with LIBDECOR_FORCE_CSD

[Christopher Reitz]

Hello everyone,

in my current quest to find an enjoyable GUI library for Linux I'm playing around with FLTK. I'm using a Wayland compositor (Hyprland) which doesn't provide server side decorations. As I understand the docs, FLTK should find that out and draw client side decorations via libdecor. But on my system this is not the case, even though the app is linked against libdecor. Furthermore the documentation mentions the environment variable LIBDECOR_FORCE_CSD. But when I set it to the value 1, the app crashes immediately on start up:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
[1]    13054 IOT instruction (core dumped)  ./main

This happens already with the code of the hello world example from the docs:

=================================================

#include <FL/Fl.H>
#include <FL/Fl_Window.H>
#include <FL/Fl_Box.H>

int main(int argc, char **argv) {
    Fl_Window *window = new Fl_Window(340, 180);
    Fl_Box *box = new Fl_Box(20, 40, 300, 100, "Hello, World!");
    box->box(FL_UP_BOX);
    box->labelfont(FL_BOLD + FL_ITALIC);
    box->labelsize(36);
    box->labeltype(FL_SHADOW_LABEL);
    window->end();
    window->show(argc, argv);
    return Fl::run();
}
=================================================

I tried different ways to compile the code, which doesn't make any difference. I only put the simplest variant here:

=================================================

fltk-config --compile main.cpp    
/usr/bin/c++ -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/fltk/src=/usr/src/debug/fltk -flto=auto -ffat-lto-objects -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D_REENTRANT -o main main.cpp /usr/lib/libfltk.a -lm -lpthread -lXinerama -lXfixes -lXcursor -L/usr/lib -lpangoxft-1.0 -lpangoft2-1.0 -lpango-1.0 -lXft -lfontconfig -lharfbuzz -lfreetype -lgobject-2.0 -lglib-2.0 -lpangocairo-1.0 -lcairo -lX11 -lXrender -lwayland-cursor -lwayland-client -lxkbcommon -ldbus-1 -ldecor-0 -ldl

=================================================

Without LIBDECOR_FORCE_CSD (or set to 0) the sample code and the little more sophisticated app I already build works just fine, but of course without having a title bar.

Before I file a bug report I wanted to ask here if I'm missing something and/or the problem is maybe with my setup:

OS: Arch Linux x86_64 

WM: Hyprland 0.54.3

fltk 1.4.4-4

libdecor 0.2.5-1

fltk and libdecor are the official packages from the Arch pacman repository. 

This installs both the libdecore cairo and the gtk plugin. Setting the other env var mentioned in the docs, LIBDECOR_PLUGIN_DIR to /usr/lib/libdecor/plugins-1/ where libdecor-cairo.so and libdecor-gtk.so are located on my machine doesn't seem to make any difference.

The full core dump and the output of ldd is attached.

Thanks in advance for any help.

Greetings

Chris

[imm]

Hmm, don't know what that would be.

I have very limited experience using Arch - I have a dev SBC board I was running Arch on for a few weeks, but there were issues with some of the tools so that boards now running Armbian instead.

FWIW, in both cases it was running Wayland rather than X11, and FLTK was (is) running fine. TBH I don't know what compositor the Armbian is using, off hand.

However, I'm not using the distro FLTK, I always build my own so that might make a difference?

Note that FLTK bundles in some libdecor support, so I've never had to "consciously select" an external libdecor lib at compile time.

The FLTK build I have on that SBC is based on the 1.4 git, maybe a month old, so probably not all that different from the 1.4.4 your Arch distro provides.

Um... So, nothing useful to suggest....

--
Ian
From my Fairphone FP6

[Jackk Sonn]

Ok

[Christopher Reitz]

Hi Ian,

so I followed your hint and built fltk (v1.4.4) by myself. First I used basically the default build config and ended up with the same result. No window decorations by default and when I set LIBDECOR_FORCE_CSD=1 it also crashes with the bad_alloc exception. 

Then I built the example apps that are in the repo and made an interesting discovery. All of them also crashed in the same way except glpuzzle and sudoku, they run and display window decorations! What makes these two different I've yet to figure out. 

Anyways, what I've tried next was setting the following build options:

FLTK_USE_LIBDECOR_GTK=OFF 
FLTK_USE_SYSTEM_LIBDECOR=OFF 

With this build everything is working now, the hello world code and my real app I'm currently working on as well. The window decorations look a little bit different than with my systems libdecor, but I can live with that.

Thank you and have a nice weekend!

Chris

[imm]

On Sat, 25 Apr 2026, 09:35 Christopher Reitz wrote:

Then I built the example apps that are in the repo and made an interesting discovery. All of them also crashed in the same way except glpuzzle and sudoku, they run and display window decorations! What makes these two different I've yet to figure out. 

Anyways, what I've tried next was setting the following build options:

FLTK_USE_LIBDECOR_GTK=OFF 
FLTK_USE_SYSTEM_LIBDECOR=OFF 

With this build everything is working now, the hello world code and my real app I'm currently working on as well. The window decorations look a little bit different than with my systems libdecor, but I can live with that.

That's interesting.

So, in essence, by forcing FLTK to use its built-in libdecor it works (hence looks a bit different to system.)

Which I guess implies that whatever lib combination cmake is detecting at build time is not working at run time...

Unfortunate and confusing.

What state did cmake report?

Does your system have Xwayland and does setting the FLTK environment variable allow your build to run in X11 mode, or is your build Wayland only?

(FLTK_BACKEND=x11 ./my_app)

[Manolo]

> Then I built the example apps that are in the repo and made an interesting discovery. All of them also crashed in the same way except glpuzzle and sudoku, they run and display window decorations! What makes these two different I've yet to figure out. 

glpuzzle and sudoku use libdecor and interact with the Wayland compositor exactly as other FLTK apps as far as titlebar is concerned. The difference suggests there's a mixture of FLTK header files for different versions and therefore different internal layouts when building the FLTK library.

Could you remove all FLTK versions and all libdecor versions from Linux packages and then rebuild FLTK from a local source tree copied from github or obtained by uncompressing an FLTK source release tarball. I suggest you set FLTK_USE_SYSTEM_LIBDECOR=OFF  when running CMake but don't change FLTK_USE_LIBDECOR_GTK.

[Manolo]

I'm trying to install a Arch Linux system in a VM over my macOS machine here. I started with a basic Arch linux VM and ran successfully

  pacman -Syu hyprland kitty

Then I try to run a Hyprland session with

  start-hyprland

This fails with these messages:

Any clue to help me progress?

[Manolo]

Ok. I can now run a Hyprland session in my Ubuntu VM. I do see that many FLTK and non-FLTK apps run without titlebars. I will try to look into this.

[Manolo]

Here are my findings running FLTK apps under the Hyprland Wayland compositor on Ubuntu 26.04:

- I get the same results running Ubuntu's packaged FLTK v 1.4.4 or building FLTK locally with FLTK_USE_SYSTEM_LIBDECOR=0

- FLTK apps generally run correctly, except those having a non-resizable window (more below).

- Hyprland declares it supports the "XDG decoration" Wayland protocol whose purpose is :

      "This interface allows a compositor to announce support for server-side decorations."

- Unexpectedly to me, Hyprland claims to support SSD but draws no window decoration. I tried a couple of non-FLTK apps

(cmake-gui, gnome-terminal)  that, under KWin (Plasma's Wayland compositor) get a titlebar drawn by the compositor 

and observed they behave as FLTK apps: no titlebar is visible.

- When an FLTK app is run with LIBDECOR_FORCE_CSD=1 a titlebar does appear but it's often mostly (but not totally)

hidden behind other Hyprland stuff: a few pixels of the bottom part of the titlebar are visible above the window.

- This affects FLTK menu windows that are not correctly positioned vertically.

- if several FLTK apps are run with LIBDECOR_FORCE_CSD=1 so that an FLTK window appears below another window 

and the FLTK window is made active, that window's titlebar becomes visible, but returns mostly invisible if another window is made active.

- FLTK apps such as test/hello.cxx having a non-resizable window crash at launch when run with LIBDECOR_FORCE_CSD=1:

 they enter a loop that reduces the vertical height of the window and finally crash.

When run without, they produce a tall, centered window, which is not correct in my view in terms of window size.

I'm tempted to conclude that either Hyprland doesn't seem to be totally mature yet , or libdecor doesn't support Hyprland correctly:

- claims to support SSD but doesn't draw any titlebar;

- partially hides the subwindow that libdecor creates around the main window. When in CSD mode, libdecor works creating a subwindow

where the titlebar gets drawn and positioning it above the main window. It seems that either Hyprland doesn't obey libdecor

instructions that a surface and a subsurface above it are to be mapped, or libdecor doesn't talk correctly to Hyprland.

This analysis indicates, in my view, that the sources of the issues are either in Hyprland or in libdecor, but not in FLTK.

[imm]

On Sun, 26 Apr 2026, 14:32 Manolo wrote:

I'm tempted to conclude that either Hyprland doesn't seem to be totally mature yet , or libdecor doesn't support Hyprland correctly:

- claims to support SSD but doesn't draw any titlebar;

- partially hides the subwindow that libdecor creates around the main window. When in CSD mode, libdecor works creating a subwindow

where the titlebar gets drawn and positioning it above the main window. It seems that either Hyprland doesn't obey libdecor

instructions that a surface and a subsurface above it are to be mapped, or libdecor doesn't talk correctly to Hyprland.

This analysis indicates, in my view, that the sources of the issues are either in Hyprland or in libdecor, but not in FLTK.

Sounds about right: I've not tried Hyprland as a compositor, but I've tried a few different Wayland systems (both Intel and Arm) and they all seem fine.

There's things I don't much like about Wayland (currently annoyed by it not reliably pushing my active window topmost, for example) but it does seem to mostly work!

Still wonder what our cmake configure is detecting - it sounds like Hyprland is asserting it supports SSD but then doesn't?

[Manolo]

Le lundi 27 avril 2026 à 14:39:22 UTC+2, Ian MacArthur a écrit :

Still wonder what our cmake configure is detecting - it sounds like Hyprland is asserting it supports SSD but then doesn't?

Ian

 At build time, no detection of what will be the Wayland compositor is done. Everything is done at run time.

A Wayland client app starts, connects to the Wayland compositor it has chosen, often the default one at 

WAYLAND_DISPLAY = wayland-0, and asks it for the list of Wayland protocols it supports. That's the only 

information the client can use to distinguish between this or that compositor.

Hyprland declares to support the "xdg decoration" protocol which, as I understand it, means it offers to

draw window titlebars itself. But, to my surprise, Hyprland doesn't draw any titlebar. May be that's because

it considers a titlebar isn't useful to a tiling compositor.