In addition to the way fluid approaches it, you can also design your program to have separation more deeply.
In my program, we have a batch version that can be built with none of the GUI and OpenGL stuff at all. That way, the batch executable has no dependencies on any graphics libraries from the operating system at all -- no OpenGL, no X, no Wayland, etc.
This is important because my program is sometimes run in HPC (cluster) environments where the headless nodes don't have those libraries installed -- and getting the sys admins to install things like that on their machine is not practical.
My main() is the same for both the headless and the GUI application. It gets compiled twice - once with an #define for NO_GRAPHICS. There are some #ifdef's that basically turn some things like gui_start() into a no-op.
Then, all the GUI code is written pretty normal -- it has full access into the core code, it can do whatever it wants.
The core code however must be able to be compiled without any knowledge of the GUI / OpenGL code at all. This means it can't even #include any headers from the GUI side of things (fltk, or the application's own code).
So, put your core code in one directory and your GUI code in a different directory. Then, set up your include paths such that the GUI sees the core, but not the other way around.
For the situation where the core code needs to communicate with the GUI, we devised a message passing system. Listeners subscribe to the messages, but senders never know who is subscribed. So, the GUI subscribes and the core sends messages that will be received if the listener is there -- but won't if it isn't.
As it turns out, there is basically only one message that needs to be sent from the core to the GUI -- "Needs_Update". I.e. something in the core state has changed and you need to trigger a GUI update to reflect that state.
The GUI side of things can #include the core code and has full access, so when it goes through and does Update(), it has full access to all the classes, data, and state of the core.
This approach seems like it would be very limiting, but it really isn't that bad.
On my Mac desktop, the GUI executable (with everything statically linked) is 25MB, the headless executable is 20MB.
The GUI application requires these libraries (otool -L ./app):
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 24.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.120.2)
/System/Library/Frameworks/UniformTypeIdentifiers.framework/Versions/A/UniformTypeIdentifiers (compatibility version 1.0.0, current version 709.0.0, weak)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1700.255.5)
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2487.60.105)
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 64.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 2503.1.0)
/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1774.5.3)
/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 2503.1.0)
/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
While the headless app requires:
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1700.255.5)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.120.2)
The strict headless environments are all Linux, but you get the idea.
Rob