Windows 10: Weird behavior of fltk /SUBSYSTEM:CONSOLE apps

[Greg Ercolano]

Curious if anyone else encounters this.

Normally, if you build an FLTK app with /SUBSYSTEM:CONSOLE:

• You can run it from a DOS window and stdout/err show up in that DOS window, and the window is blocked while the GUI is open
• If you click on it from a desktop or browser, it opens a DOS window and the GUI window opens over that

..which is all fine in the Windows universe as I've seen it for the last 20 years.

However, with windows 10, there's some new weird behavior I haven't seen before,
wondering if anyone else encounters this:

    1) If I invoke it from a DOS window, an /extra/ dos window opens when it runs (WTF?)
    2) If I invoke it from a DOS window /running as administrator/, no extra DOS window appears (OK)
    3) If I invoke it from a PowerShell, no extra DOS window appears (OK)

#1 seems really really odd to me. Never seen that before.

The customer sees this too, and I don't know what to make of it, and wasn't able
to find info on this, as I imagine GUI apps aren't usually also console apps.

Trying to figure out what's wrong here; I assume Windows is seeing this console
app as something different from normal, probably because FLTK is creating a WinMain()
internally, and that's somehow triggering Windows with this new behavior.

Might be a bug, as it's weird that the old normal behavior happens if it's an administrator owned
DOS window.

[Greg Ercolano]

On 4/19/21 7:29 PM, Greg Ercolano wrote:

[..] However, with windows 10 [..]

    1) If I invoke it from a DOS window, an /extra/ dos window opens when it runs (WTF?)

[..]

Might be a bug, as it's weird that the old normal behavior happens if it's an administrator owned DOS window.

    Oh, and by "bug" I mean a bug in Windows 10.
    I was on "Home Edition" when I tried this, but I think the customer is on Win 10 Pro.

[Daniel G.]

Hey, slightly offtopic but just FYI:

I always program my stuff as subsystem console (-mconsole on mingw), so I can pass/parse commandline parameters like I do on Unix based systems. You can get rid of the console after parsing the parameters using the FreeConsole() call. (See here: https://github.com/nwrkbiz/simple-webcam/blob/11a0503d7e8a5e5ba252104203e559564ff4030e/main.cpp#L157)

BR,
Giri

[Daniel G.]

> 1) If I invoke it from a DOS window, an /extra/ dos window opens when it runs (WTF?)

Oh and I cannot confirm this observation. The executable is launched within the calling DOS window (and also prints cout / cerr there).

I am using FLTK 1.3.5

BR,
Giri

[Greg Ercolano]

On 4/20/21 1:51 AM, 'Daniel G.' via fltk.general wrote:

1) If I invoke it from a DOS window, an /extra/ dos window opens when it runs (WTF?)

Oh and I cannot confirm this observation. The executable is launched within the calling DOS window (and also prints cout / cerr there).
I am using FLTK 1.3.5

    Ya, it's the weirdest thing I've seen Windows do in a while.

    I should mention I'm using fltk-1.3.6 + Visual Studio 2017 building on a Windows8 machine,
    then running the static executable on a separate Windows 10 machine.

    My compile/link flags look like this, FWIW:

        cl /nologo /TP /MT /EHsc /DVERSION=\"0.96\" /DWIN32 /Ifltk-1.3.6 /I./hidapi /I. /c myapp.cpp

        link /nologo /SUBSYSTEM:CONSOLE myapp.obj fltk-1.3.6\lib\fltk.lib ./hidapi/hidapi.lib \
            Setupapi.lib comctl32.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib \
            advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib  odbccp32.lib shlwapi.lib \
            /OUT:myapp.exe

    The spec is for a single application that can do both a GUI and command line,
    but I'm beginning to think I better make separate executables..

[Daniel G.]

Hey, I just did a quick test:

does your program have the option "Right Click -> Compatibility -> Run this program as an administrator" checked? If I check this, I will also get a second DOS window when the first one was not started with

admin privileges. (Which somehow also makes sense to me)

BR,
Giri

[Albrecht Schlosser]

On 4/20/21 11:20 AM Greg Ercolano wrote:
> On 4/20/21 1:51 AM, 'Daniel G.' via fltk.general wrote:
>>> 1) If I invoke it from a DOS window, an /extra/ dos window opens when
>>> it runs (WTF?)
>>

>> Oh and I *cannot *confirm this observation. The executable is launched

>> within the calling DOS window (and also prints cout / cerr there).
>> I am using FLTK 1.3.5
>
>     Ya, it's the weirdest thing I've seen Windows do in a while.
>

>     I should mention I'm using *fltk-1.3.6* + *Visual Studio 2017*

> building on a Windows8 machine,
>     then running the static executable on a separate Windows 10 machine.

Greg, I *believe* this is caused by our (FLTK) WinMain() function and
the build flags. I did a test on my Win10 machine and built the stock
FLTK 1.3.6 test programs with some modifications [1] with VS 2019.

Since you are using FLTK, how did you build FLTK? Debug or Release mode?
This can make a difference (see below).

>     My compile/link flags look like this, FWIW:

I don't understand the MS commandline, but what I can say is: If I build
the FLTK lib in debug mode and set the WIN32 option on the CMake the
resultant program is a GUI program, otherwise it's a console application.

[1] For my modifications see attached file debug_console.diff

If I remove the WIN32 flag from fl_create_example there's no console
window and both debug and release versions write to the DOS window w/o
an extra console window.

If I set the WIN32 flag (the default) I can see that the FLTK WinMain()
is called and opens the (extra) console window even if called from the
DOS window.

My guess is that older Windows versions suppressed the extra console
window created by AllocConsole whereas Win10 always creates an extra window.

I suggest that you patch your FLTK version with my diff file for
debugging to see if the FLTK WinMain() function is called and if that
creates the extra console window.

Daniel G. (Giri) wrote that he could not confirm what you see. I assume
that he built a non-FLTK program (or he built FLTK in Release mode).

Note: in my tests I used test/glpuzzle.cxx, built in Debug and Release
mode with and w/o the WIN32 flag in fl_create_example (see patch). HTH.

[Albrecht Schlosser]

On 4/20/21 1:17 PM Albrecht Schlosser wrote:

> I don't understand the MS commandline, but what I can say is: If I build
> the FLTK lib in debug mode and set the WIN32 option on the CMake the
> resultant program is a GUI program, otherwise it's a console application.

Should read: ... set the WIN32 option on the CMake add_executable() call
the resultant program ...

[Daniel G.]

> Daniel G. (Giri) wrote that he could not confirm what you see. I assume
> that he built a non-FLTK program (or he built FLTK in Release mode).

I did build a FLTK program, but FLTK was indeed built using the Release configuration.
Did any of you check if the  "Right Click -> Compatibility -> Run this program as an administrator" is enabled on your executables?

[Daniel G.]

@All,

When building glpuzzle using VS19 on Windows 10 (debug & release) with "Linker -> System -> SubSystem" set to /SUBSYSTEM:CONSOLE and (FLTK 1.3.5) and none of the executables did open a new DOS window. (Also added std::cout to verify if I can print stuff to the calling console).

... It really seems to me, like one of these compatibility flags are set on  Greg Ercolano's executables. (Maybe because the executable was built on Win8?)

Albrecht  Schlosser,
I think when building /SUSBSYTEM:WINDOWS executables with the Debug configuration the DOS window is a "feature", as you will be able to see your iostream/debugging traces, even when working with the wrong subsystem.

BR,
Giri

[Greg Ercolano]

On 4/20/21 4:17 AM, Albrecht Schlosser wrote:

Greg, I *believe* this is caused by our (FLTK) WinMain() function and the build flags. I did a test on my Win10 machine and built the stock FLTK 1.3.6 test programs with some modifications [1] with VS 2019.

Since you are using FLTK, how did you build FLTK? Debug or Release mode? This can make a difference (see below).

    Built in release mode, and switched the libs to use the /MT flag for static linking.

     My compile/link flags look like this, FWIW:

I don't understand the MS commandline, but what I can say is: If I build the FLTK lib in debug mode and set the WIN32 option on the CMake the resultant program is a GUI program, otherwise it's a console application.

    My goal is (or well, was) to compile the FLTK app to interact with stdout/err,
    so that command line flags could control the app and have output be redirectable.

    I've since had to retool the app's output to an Fl_Simple_Terminal, which I simply copied
    from 1.4.x into my 1.3.6 project, and direct stdout/err to that using messages.

    I'm on a tight deadline, so I can't dive into figuring this out right now, I realize
    the misbehavior seems to be something new and unique to Win10.

If I remove the WIN32 flag from fl_create_example there's no console window and both debug and release versions write to the DOS window w/o an extra console window.

    Oh, interesting -- by WIN32 flag, I assume you mean the macro setting, e.g. /DWIN32 (or on
    mingw, I guess it's -DWIN32)

    I thought I absolutely needed the WIN32 flag or my app wouldn't build; non windows
    stuff was being triggered in the FLTK .H files.

If I set the WIN32 flag (the default) I can see that the FLTK WinMain() is called and opens the (extra) console window even if called from the DOS window.

    Good to know, I'll try some more experiments when I'm done w/this project.
    Ran into all kinds of weird things on this quick project; another one is problems
    with DLLs not loading when you're chdir'd to a network drive, and get "Runtime Error/
    process terminated unexpectedly" dialogs (on both Win8 and Win7) that can be solved
    by invoking the app with absolute paths.. *shakes head* Microsoft Windows makes me nuts.

My guess is that older Windows versions suppressed the extra console window created by
AllocConsole whereas Win10 always creates an extra window.

I suggest that you patch your FLTK version with my diff file for debugging to see if the FLTK WinMain() function is called and if that creates the extra console window.

    Hmm, I didn't know we had code in FLTK that triggered AllocConsole.
    As I said I'll look into it after..

Daniel G. (Giri) wrote that he could not confirm what you see. I assume that he built a non-FLTK program (or he built FLTK in Release mode).

    I'm built in release mode for sure.

Note: in my tests I used test/glpuzzle.cxx, built in Debug and Release mode with and w/o the WIN32 flag in fl_create_example (see patch). HTH.

    Thanks -- I'll check when I have some time.
    I already retooled my app into separate programs to work around. It's probably
    better this way, because the customer didn't like the DOS window opening when
    the app was 'clicked on' from the desktop/file explorer, so I had to do the conversion
    anyway..

[Greg Ercolano]

On 4/20/21 2:24 AM, 'Daniel G.' via fltk.general wrote:

Hey, I just did a quick test:

does your program have the option "Right Click -> Compatibility -> Run this program as an administrator" checked?

    Hmm, my copy of Win10 Home Edition doesn't have a "Compatibility" submenu when I right click on it,
    I only have a "Troubleshoot Compatibility" which eventually opens a window with two options; 'Try recommended settings'
    and 'Troubleshoot program'. If I choose the latter, it shows me a dialog with 4 radio button options:

        ( ) Program worked in earlier versions of Windows but won't install or run now
        ( ) Program opens but doesn't display correctly
        ( ) Program requires additional permissions
        ( ) I don't see my problem listed

    So I'm not sure I can follow your test, at least not from the "file explorer window" I have,
    but I see what you're saying in how if it was set to automatically run as admin, it would
    affect the behavior between a regular DOS window and a DOS window running as admin.

    The only thing that might complicate this is:

        (a) I'm compiling on Win8 with VS 2017 "CL" command line + nmake,
        (b) The build directory is a network drive served by a mac over Samba, which /might/ affect execution flags somehow

    I did try copying the binary to c:\temp and right click on it, same results with the right click
    menu (no "Compatibility" submenu), and same results with DOS window / powershell behavior,
    where it works /normally/ when a powershell is used to open it in both DOS windows (normal and admin).
    The misbehavior is only from the regular DOS window when there's no powershell.

    I did get a window about "This program didn't seem to be installed correctly" but it ran it
    anyway, and subsequent executions don't throw that dialog. I don't know what that is either.

    But if I run the C:\temp version of the executable, I get the same results I do on the network drive;
    extra window shows up only in the 'regular DOS window', not the 'admin' DOS window, and if
    I run 'powershell' in both, the app can be run without the extra DOS window in both. Friggin' weird!

    Between the UAC and this batty 'compatibility mode' stuff with the damn 'program might not be
    installed correctly' crap, it's confusing as hell to be a developer.. what is all this crap, lol

[Daniel G.]

> Hmm, my copy of Win10 Home Edition doesn't have a "Compatibility" submenu when I right click on it

Oh I am sorry, I missed one step in my description.
You should find it via:

"Right Click -> Properties -> (Tab) Compatibility -> (Checkbox) Run this program as an administrator"

Also I cannot confirm your behaviour with the latest changes on the 1.3 branch.. (Neither Debug build nor Release build with /SUBSYSTEM:CONSOLE, built on Win10 with VS19)
I only can reproduce your behaviour when enabling the checkbox described above.

[Greg Ercolano]

On 4/20/21 9:03 AM, 'Daniel G.' via fltk.general wrote:

> Hmm, my copy of Win10 Home Edition doesn't have a "Compatibility" submenu when I right click on it

Oh I am sorry, I missed one step in my description.
You should find it via:

"Right Click -> Properties -> (Tab) Compatibility -> (Checkbox) Run this program as an administrator"

    Ah, OK.

    All the radio buttons are "unclicked", so that's not it for me.
    (And if I set the flag, the behavior is unchanged)

Also I cannot confirm your behaviour with the latest changes on the 1.3 branch.. (Neither Debug build nor Release build with /SUBSYSTEM:CONSOLE, built on Win10 with VS19)

    Hmm, when I get a chance, I'll try using VS2019.
    Might be a week till I can get to that, tho.

[Albrecht Schlosser]

On 4/20/21 5:13 PM Greg Ercolano wrote:
>
> On 4/20/21 4:17 AM, Albrecht Schlosser wrote:
>> Greg, I *believe* this is caused by our (FLTK) WinMain() function and
>> the build flags. I did a test on my Win10 machine and built the stock
>> FLTK 1.3.6 test programs with some modifications [1] with VS 2019.
>>
>> Since you are using FLTK, how did you build FLTK? Debug or Release
>> mode? This can make a difference (see below).
>
>     Built in release mode, and switched the libs to use the /MT flag
> for static linking.

OK, and I read that you're on a tight deadline, but I recommend to patch
FLTK with the tiny patch I posted to see *if* it's FLTK's
AttachConsole() call that creates the extra console. If that's the case
you could also disable it in src/fl_call_main.c (see more on that below).

>     I'm on a tight deadline, so I can't dive into figuring this out
> right now, I realize
>     the misbehavior seems to be something new and unique to Win10.

As I wrote before: my assumption is that older Windows versions actively
*suppress* the extra console window if it is created by AllocConsole in
the FLTK WinMain() function.

>> If I remove the WIN32 flag from fl_create_example there's no console
>> window and both debug and release versions write to the DOS window w/o
>> an extra console window.
>
>     Oh, interesting -- by WIN32 flag, I assume you mean the macro
> setting, e.g. /DWIN32 (or on mingw, I guess it's -DWIN32)

No, I mean the WIN32 in the CMake code building the FLTK test programs.
Take a tiny little time to look at my patch and look at the changes
introduced in CMake/fl_create_example.cmake. I *know* that this option
sets either /subsystem:console or /subsystem:windows or whatever it's
called.

>     I thought I absolutely needed the WIN32 flag or my app wouldn't
> build; non windows stuff was being triggered
>     in the FLTK .H files.

Yep, that's the preprocessor macro WIN32, a totally different thing. In
FLTK 1.3.x you need to define it yourself whereas in FLTK 1.4 we're
using _WIN32 (leading underscore) which should be defined by the compiler.

>> If I set the WIN32 flag (the default) I can see that the FLTK
>> WinMain() is called and opens the (extra) console window even if
>> called from the DOS window.

Again, here I mean the CMake WIN32 flag used in add_executable().

>> My guess is that older Windows versions suppressed the extra console
>> window created by
>> AllocConsole whereas Win10 always creates an extra window.
>>
>> I suggest that you patch your FLTK version with my diff file for
>> debugging to see if the FLTK WinMain() function is called and if that
>> creates the extra console window.
>
>     Hmm, I didn't know we had code in FLTK that triggered AllocConsole.
>     As I said I'll look into it after..

Just for completeness: it's in src/fl_call_main.c which defines the
WinMain() function that FLTK supplies if your program doesn't have one
and is a GUI application (if I understand that correctly).

I think Windows needs "WinMain()" only if you build a GUI app
(/subsystem:windows) but that may be wrong.

Anyway, this is important:

If your program needs a WinMain() function and you don't provide one
(you use main() only) *and* you link your program /statically/ with
fltk.lib, then (and only then) the linker will link in the WinMain()
function from the FLTK library which is defined in 'src/fl_call_main.c'
and may under certain circumstances (see #if ... in source file) use
AllocConsole() to create a console for your program.

My patch adds only a few printf() statements so you can see if this code
is used.

[Daniel G.]

Hmm still cannot reproduce (now tried with VS2015).

Here is what I did:

1.) Downloaded current state of branch-1.3
2.) Generated FLTK solution to a "build" directory within the source tree for VS2015 
3.) Built fltk.lib
4.) Created following file (main.cpp) within the "build" directory:

#include <FL/Fl.H>

#include <FL/fl_message.H>

#include <iostream>

int main(int, char**) {

  std::cout << "Schinkenstangerl" << std::endl;

  fl_alert("Kipferl");

  return 0;

}

5.) Opened VS2015 Developer command prompt and navigated to the build directory
6.) cl /nologo /TP /MT /EHsc /DVERSION=\"0.96\" /DWIN32 /I. /I.. /c main.cpp 

7.) link /nologo /SUBSYSTEM:CONSOLE main.obj lib\Release\fltk.lib Setupapi.lib comctl32.lib kernel32.lib user32.lib gdi32.lib winspool.lib msvcrt.lib msvcmrt.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib  odbccp32.lib shlwapi.lib /OUT:myapp.exe
8.) link /nologo /SUBSYSTEM:CONSOLE main.obj lib\Debug\fltkd.lib Setupapi.lib comctl32.lib kernel32.lib user32.lib gdi32.lib winspool.lib msvcrt.lib msvcmrt.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib  odbccp32.lib shlwapi.lib /OUT:myapp_debug.exe

[Albrecht Schlosser]

On 4/20/21 3:35 PM 'Daniel G.' via fltk.general wrote:
>
> Albrecht Schlosser,
> I think when building /SUSBSYTEM:WINDOWS executables with the Debug
> configuration the DOS window is a "feature", as you will be able to see
> your iostream/debugging traces, even when working with the wrong subsystem.

Yes, it's a feature, that's clear so far. The only question is why it is
apparantly sometimes "suppressed" by the Windows OS (Win7) or not
(Win10) as Greg reported.


FYI: This feature is as old as FLTK 1.0 and mostly in its current shape
since FLTK 1.0.6. It got some compiler fixes (#ifdef...) and some
modifications for UTF-8 handling of commandline arguments since FLTK 1.3.0.

There's an old comment that certainly predates WinXP, Win7, etc.:

/*
* Currently the debug version of this library will create a
* console window for your application so you can put printf()
* statements for debugging or informational purposes. Ultimately
* we want to update this to always use the parent's console,
* but at present we have not identified a function or API in
* Microsoft(r) Windows(r) that allows for it.
*/


As I read this (i.e. the part "Ultimately..."), the extra console window
was already an issue in the pre-XP aera which IMHO might assert my
assumption that *some* windows versions have actively suppressed the
extra console whereas Win10 does not (in Greg's case) suppress it.

Maybe it's a /bug fix/ rather than a /bug/ in Windows 10? ;-)


That said, the code in fl_call_main.c is essentially the same in 1.3 and
1.4 with the difference that 1.3 uses the "WIN32" and 1.4 uses the
"_WIN32" preprocessor definition. The current code in src/fl_call_main.c
(stripped to the essential parts) with my comments below is:


#if defined(WIN32) && !defined(FL_DLL) && !defined (__GNUC__)


Means: the following applies only

- to Windows (WIN32)
- if statically linked (!defined(FL_DLL))
- and to non-GNU compilers (MSVC included)

The latter should IMHO better read "&& defined _MSC_VER" so it only
applies to MS Visual Studio compilers, but that's a different story.


int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR
lpCmdLine, int nCmdShow) {


This defines the WinMain() entry function.


# ifdef _DEBUG
AllocConsole();
freopen("conin$", "r", stdin);
freopen("conout$", "w", stdout);
freopen("conout$", "w", stderr);
# endif /* _DEBUG */


This opens the extra console window in debug mode, i.e. if "_DEBUG" is
defined.

The main question is if this is (was) defined when the FLTK library is
(was) built.


/* Run the standard main entry point function... */
rc = main(__argc, ar);


Obviously: this calls the standard main() function.

The rest of the code closes the console and returns.


Interesting link regarding main() vs. WinMain() and more:
https://stackoverflow.com/questions/13871617/winmain-and-main-in-c-extended

[Greg Ercolano]

On 4/20/21 11:18 AM, Albrecht Schlosser wrote:

This defines the WinMain() entry function.


#  ifdef _DEBUG
  AllocConsole();
  freopen("conin$", "r", stdin);
  freopen("conout$", "w", stdout);
  freopen("conout$", "w", stderr);
#  endif /* _DEBUG */


This opens the extra console window in debug mode, i.e. if "_DEBUG" is defined.
The main question is if this is (was) defined when the FLTK library is (was) built.

 
    Definitely not compiled debug mode in my case.
    And if it were, the AllocConsole() would always appear, but it doesn't..

    I'll try your patch test later today -- for reasons, I can't apply changes to FLTK right now..
    need to keep stability until I get through a delivery cycle.

[lifeatt...@gmail.com]

However, with windows 10, there's some new weird behavior I haven't seen before, wondering if anyone else encounters this: 

    1) If I invoke it from a DOS window, an /extra/ dos window opens when it runs (WTF?)

Sorry for being late to the party.

Reading the docs re: the /SUBSYSTEM linker command, there's one subtlety which might be relevant:

WINDOWS
Application does not require a console, probably because it creates its own windows for interaction with the user. If WinMain or wWinMain is defined for native code [...] WINDOWS is the default.

 

I interpret this to mean that if WinMain() is defined, then the linker will use /SUBSYSTEM:WINDOWS even if you specify /SUBSYSTEM:CONSOLE !

I.e. I think this means if a program uses main() won't see the problem, as /SUBSYSTEM:CONSOLE is the default. 

I've not tested/verified this yet myself, thought I'd throw it out there ASAP.

Kevin

[lifeatt...@gmail.com]

>I interpret this to mean that if WinMain() is defined, then the linker will use /SUBSYSTEM:WINDOWS even if you specify /SUBSYSTEM:CONSOLE !

>I.e. I think this means if a program uses main() won't see the problem, as /SUBSYSTEM:CONSOLE is the default. 

Whelp, never mind, I got it wrong.

 If I build using /SUBSYSTEM:CONSOLE, the program won't link unless main() exists.

[Daniel G.]

Greg Ercolano,

does your powershell report 'True' when executing these two commands:

$currentPrincipal = New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentity]::GetCurrent())
$currentPrincipal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)

I still think the DOS window pops up because of privilege escalation..

[Ian MacArthur]

On 20 Apr 2021, at 19:18, Albrecht Schlosser wrote:
>
> Yes, it's a feature, that's clear so far. The only question is why it is apparantly sometimes "suppressed" by the Windows OS (Win7) or not (Win10) as Greg reported.

It is entirely possible that the different compiler versions matter - MS have a long standing habit of adding metadata flags and attributes to their binaries, that are then interpreted by later OS variants and ignored (or simply not understood) by older OS versions...

I’ve tripped over this a few times, since (using mingw a lot) my binaries tend not to have these bits set at all, and I have seen code do “unexpected” things when later compiled with some or other VS version. And also different “unexpected” things when compiled with different VS versions (2010 vs. 2017 usually in my case...)