I'm really interested to hear your ideas about how to improve the GUI -- if you have some mockups or some examples you can point at, that would be great.
OpenVSP uses FLTK as a GUI Widget library -- FLTK handles all the windows, buttons, and sliders. FLTK is kinda old and clunky, but it is quite flexible and can be very powerful.
If you're mostly interested in changing some of the purely aesthetic things (colors, borders, typeface, spacing, etc (dark mode anyone?)), then you can probably accomplish this relatively easily by adjusting some FLTK settings and/or applying 'themes'. FLTK's support for things like themes is not awesome, but the FLTK team recently started working on v1.5 and improving things like themes is one of the themes of the next version.
If you're interested in changing how the UI works more fundamentally, that may also be possible. For example, I have long considered adding a docking capability (where you can move windows around inside the main application, tabbing them, rearranging them, minimizing them, etc). FLTK has never had a first-class docking framework. There was an old partial attempt, but it wasn't worth getting deep into. I am optimistic that the FLTK 1.5 dev cycle might add a docking capability that we could adopt in OpenVSP.
FLTK 1.5 will be getting improved multitouch gesture support. So things like pinch to zoom and two-finger rotate (on the screen or a touchpad) will be a reality. That alone probably isn't enough to make OpenVSP tablet friendly, but I think the combination of a docking framework and multitouch gestures will go a long way towards supporting things like tablets.
If you're interested in something fundamentally more expansive, then it is going to be very difficult. I won't say it is impossible, but it would not be a small endeavour.
Most FLTK GUI's are developed in a GUI drawing program called Fluid. You drag-n-drop, resize, and interact with a visual representation of the GUI. This information is stored in a *.fl file and at compile time, Fluid converts that into C++ source to compile into your program.
OpenVSP doesn't actually do that. Instead, we generate the entire GUI programmatically (i.e. with code). We have some custom helper tools that wrap around FLTK Widgets -- they're in GuiDevice.cpp. We handle the layout of GuiDevices with GroupLayout.cpp. The great thing about this approach is that it improves consistency across our GUI. You change things in one place and every place is updated. Adding a new slider or button to a GUI is relatively straightforward (not like doing it in Fluid). It also means that if you wanted to replace FLTK with something else, this is approximately where you would want to start.
However, I strongly recommend that you don't go down that path -- certainly not at the start. I believe that if you can articulate what you want to improve in the GUI, then you can probably accomplish that using FLTK and the current frameworks.
You aren't the first person to observe that OpenVSP's GUI is not 'modern'. So far, nobody who has levied that criticism has been able to articulate what they would do to improve the GUI -- other than "make it modern".
If you go to AI and ask it to "Make the OpenVSP GUI modern", I do not think it will give a helpful reply. Instead of giving you advice of how to adjust the existing GUI to improve various aesthetic choices, it will instead recommend a bunch of fundamental software engineering technology stack choices. It will say that OpenVSP needs to be a browser based application with remote rendering and XYZ API and all the latest 'modern' stuff. I.e. its definition of 'Modern' is not likely to align with your definition. Put another way, you can make an ugly GUI in a 'modern' framework just as easily as you can make an ugly GUI in FLTK. I do not believe that FLTK is the part of the problem that needs to be solved.
I hope this helps start the conversation. What do you have in mind?
Rob