CFileDialog::DoModal() never returns
jbreher
Something deep within the call tree of CFileDialog::DoModal() is (apparently)
failing, causing the dialog to never be displayed, and the f() never to
return. Here's the failing code:
void CPvtDlg::OnBnClickedBnFileBd()
{
// specify a file
CFileDialog fileOpenBox ( TRUE,
// BOOL bOpenFileDialog
NULL,
// LPCTSTR lpszDefExt
NULL,
// LPCTSTR lpszFileName
(OFN_HIDEREADONLY | OFN_OVERWRITEPROMPT),
// DWORD dwFlags
NULL, // LPCTSTR lpszFilter
NULL );
// CWnd* pParentWnd
// extract file name
if( IDOK == fileOpenBox.DoModal() )
{
m_strFileNameBd = fileOpenBox.GetFileName();
m_strPathNameBd = fileOpenBox.GetPathName();
}
...
Single stepping, I am unable to step over the DoModal() call. Stepping into
the call to DoModal(), I find myself inside dlgfile.cpp:
INT_PTR CFileDialog::DoModal()
{
ASSERT_VALID(this);
ASSERT(m_ofn.Flags & OFN_ENABLEHOOK);
ASSERT(m_ofn.lpfnHook != NULL); // can still be a user hook
// zero out the file buffer for consistent parsing later
ASSERT(AfxIsValidAddress(m_ofn.lpstrFile, m_ofn.nMaxFile));
DWORD nOffset = lstrlen(m_ofn.lpstrFile)+1;
ASSERT(nOffset <= m_ofn.nMaxFile);
memset(m_ofn.lpstrFile+nOffset, 0, (m_ofn.nMaxFile-nOffset)*sizeof(TCHAR));
// This is a special case for the file open/save dialog,
// which sometimes pumps while it is coming up but before it has
// disabled the main window.
HWND hWndFocus = ::GetFocus();
BOOL bEnableParent = FALSE;
m_ofn.hwndOwner = PreModal();
AfxUnhookWindowCreate();
if (m_ofn.hwndOwner != NULL && ::IsWindowEnabled(m_ofn.hwndOwner))
{
bEnableParent = TRUE;
::EnableWindow(m_ofn.hwndOwner, FALSE);
}
_AFX_THREAD_STATE* pThreadState = AfxGetThreadState();
ASSERT(pThreadState->m_pAlternateWndInit == NULL);
if (m_ofn.Flags & OFN_EXPLORER)
pThreadState->m_pAlternateWndInit = this;
else
AfxHookWindowCreate(this);
INT_PTR nResult;
if (m_bOpenFileDialog)
nResult = ::AfxCtxGetOpenFileName(&m_ofn);
else
nResult = ::AfxCtxGetSaveFileName(&m_ofn);
...
I am unable to step over the call to ::AfxCtxGetOpenFileName(&m_ofn); in the
if branch above. Stepping into it allows me to step through
OPENFILENAME& CFileDialog::GetOFN()
{
return *m_pOFN;
}
and
COMMDLG_AFXCTXFUNC(BOOL,GetOpenFileNameW,(LPOPENFILENAMEW
unnamed1),(unnamed1))
to
AFX_MODULE_STATE* AFXAPI AfxGetModuleState()
{
_AFX_THREAD_STATE* pState = _afxThreadState;
ENSURE(pState);
AFX_MODULE_STATE* pResult;
if (pState->m_pModuleState != NULL)
{
// thread state's module state serves as override
pResult = pState->m_pModuleState;
}
else
{
// otherwise, use global app state
pResult = _afxBaseModuleState.GetData();
}
ENSURE(pResult != NULL);
return pResult;
}
back through
COMMDLG_AFXCTXFUNC(BOOL,GetOpenFileNameW,(LPOPENFILENAMEW
unnamed1),(unnamed1))
I could probably continue, but it seems to me that I am inside the innards
of an area of the libs that I have no business in. What is it up in the API
level that I am screwing up?
Mikel
On 21 feb, 09:27, jbreher <jbre...@discussions.microsoft.com> wrote:
> I am encountering a wierd error, and I can't figure out which way to turn.
> Something deep within the call tree of CFileDialog::DoModal() is (apparently)
> failing, causing the dialog to never be displayed, and the f() never to
> return. Here's the failing code:
*************
Well, DoModal returns when you select a file o close the dialog, so if
the dialog is never displayed, it's logical that it doesn't return.
*************
>
> void CPvtDlg::OnBnClickedBnFileBd()
> {
>
> // specify a file
> CFileDialog fileOpenBox ( TRUE,
> // BOOL bOpenFileDialog
> NULL,
> // LPCTSTR lpszDefExt
> NULL,
> // LPCTSTR lpszFileName
> (OFN_HIDEREADONLY | OFN_OVERWRITEPROMPT),
> // DWORD dwFlags
> NULL, // LPCTSTR lpszFilter
> NULL );
> // CWnd* pParentWnd
************
You are using the defaults here, so why not remove them and use just
CFileDialog fileOpenBox(TRUE); ?
************
*******
I have tried stepping into all of the functions until it finally
displays the dialog, so I really don't know what is going wrong. Those
are MFC/WIN32 internals and I am not an expert on that...
Have you checked wether the dialog is under other windows, or outside
the screen? It shouldn't, but anyway.
Another thing that comes to mind, try setting breakpoints in all the
functions in CFileDialog and see where it stops...
Joseph M. Newcomer
you should see a dialog box.
Presumably you are going to fill in all those parameters at some point, because you are
using the defaults, so there is no reason to provide default parameters to the
constructor.
I see no function f() here.
What version of VS are you using, what platform SDK, what OS? Not enough information here
to even make a good guess. There are some known issues with using older Platform SDKs on
modern operating systems because the size of the m_ofn member changed, but they have
typically led to unpleasant crashes, usually due to access faults or heap corruption.
joe
Joseph M. Newcomer [MVP]
email: newc...@flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
jbreher
"Mikel" wrote:
> See below:
>
> *************
> Well, DoModal returns when you select a file o close the dialog, so if
> the dialog is never displayed, it's logical that it doesn't return.
> *************
Understood. Hence, my confusion.
> >
> > void CPvtDlg::OnBnClickedBnFileBd()
> > {
> >
> > // specify a file
> > CFileDialog fileOpenBox ( TRUE, // BOOL bOpenFileDialog
> > NULL, // LPCTSTR lpszDefExt
> > NULL, // LPCTSTR lpszFileName
> > (OFN_HIDEREADONLY | OFN_OVERWRITEPROMPT),
> > // DWORD dwFlags
> > NULL, // LPCTSTR lpszFilter
> > NULL ); // CWnd* pParentWnd
>
> ************
> You are using the defaults here, so why not remove them and use just
> CFileDialog fileOpenBox(TRUE); ?
> ************
OK, I tried that. No apparent change. Still unable to step over the call
nResult = ::AfxCtxGetOpenFileName(&m_ofn);
witihin the implementation of INT_PTR CFileDialog::DoModal().
> > I could probably continue, but it seems to me that I am inside the innards
> > of an area of the libs that I have no business in. What is it up in the API
> > level that I am screwing up?
>
> *******
> I have tried stepping into all of the functions until it finally
> displays the dialog, so I really don't know what is going wrong. Those
> are MFC/WIN32 internals and I am not an expert on that...
Likewise.
> Have you checked wether the dialog is under other windows, or outside
> the screen? It shouldn't, but anyway.
I can't seem to find it. The parent dialog *seems* to retain focus. However,
it refuses to repaint itself. Perhaps that is to be expected, anyway? Also,
all my past experience wit hthe CFileDialog is that it is displayed above all
other windows when DoModal() is invoked. At least that is my recollection.
Should it not have top-most position, and focus, in all cases?
How do I check for it outside the screen?
> Another thing that comes to mind, try setting breakpoints in all the
> functions in CFileDialog and see where it stops...
Well, as reported, I can step into the MFC functions (at least to a point).
However, that should be irrelevant, right? I shouldn't have to reverse
engineer MFC in order to use MFC, should I? I never had to in the past -
although I must admit to noobishness.
If you think it would help, I'm certainly willing to dive into the guts of
MFC, to get more specific about where things go astray. It is rather obtuse
code, however.
jbreher
"Joseph M. Newcomer" wrote:
> This is not a good description. If DoModal "fails", it will return 0. If it succeeds,
> you should see a dialog box.
Sorry for my imprecise language. What shall we call it when an API
invocation does not do what the documentation purports it to do? You are
right, it did not 'fail' in the sense that it did not return a value
indicating failure. It never returned, period. However, I apparently have no
File Open dialog. At least none that I can find.
> Presumably you are going to fill in all those parameters at some point, because you are
> using the defaults, so there is no reason to provide default parameters to the
> constructor.
As suggested by another poster, I have also tried invoking it as CFileDialog
fileOpenBox(TRUE);, with no change in behavior.
> I see no function f() here.
True. Sorry again for my imprecision. f() here is meant to be shorthand for
'the function we are discussing'. More specifically, f() was meant to denote
CFileDialog::DoModal().
Accordingly, the symptons could be described as follows:
Something deep within the call tree of CFileDialog::DoModal() is (apparently)
failing, causing the dialog to never be displayed, and the
CFileDialog::DoModal() never to return.
> What version of VS are you using,
Help>About has the following to say:
Visual Studio 2005 Professional Edition
Microsoft Visual Studio 2005
Version 8.0.50727.762 (SP.050727-7600)
> what platform SDK,
Quite frankly, I don't know how to check this. Is the SDK version not
inferred from the VS version?
> what OS?
System Information says:
Microsoft Wondows XP Professional
5.1.2600 Service Pack 2 Build 2600
> Not enough information here
> to even make a good guess. There are some known issues with using older > Platform SDKs on
> modern operating systems because the size of the m_ofn member changed, but they have
> typically led to unpleasant crashes, usually due to access faults or heap corruption.
How old is old? I guess I just assumed that VS installed whatever it needed
to build working code. I have certainly used this particular MFC API call in
other projects, built on this machine, using this exact installation of VS.
Unless Windows Update changed something germane out from underneath me.
Is there any more info that may be of use in diagnosing the problem? I was
kind of hoping someone could point to some glaring error in the parameter
list of the DoModal call, but I guess nothing there looks obviously amiss?
Joseph M. Newcomer
dialog.
Note that phenomena such as having too many windows, running out of GDI space, etc. can
result in some of the problems you are seeing. So key here is to replicate the
environment. I (and others) can do
CFileDialog dlg(TRUE);
dlg.DoModal();
without problem, so if it is failing for you, there is something else wrong. But since it
is impossible to intuit what your entire environment must be like, it is hard to do more
than guess.
I'd look at resource exhaustion as a likely candidate. Can you write a trivial example
that just does the CFileDialog dlg(TRUE) as above, in a little test app, and see it work?
If so, then the problem is the environment your app is presenting to it. Programs like
the process viewer (from www.sysinternals.com) could give some figures about things like
GDI handle usage; if the numbers are far out of line, this could be the problem.
joe
Joseph M. Newcomer
joe
jbreher
"Joseph M. Newcomer" wrote:
> So that means you haven't stepped deeply enough; you should have tried to Step Into.
As reported in the OP, I did step into the DoModal() call. Indeed, I stepped
deep enough that I was soon lost in a forest of dense inscrutable macros.
This may speak more to my limid mental capacity more than to anything else.
However, it would seem to me that, if MFC works as advertised, I should not
need to dig that deep.
All ranting aside, I abandoned the hunt when I seemed to enter into a
(seemingly) recursive loop, bringing me repeatedly through this
indecipherable line in afxcomctl32.h:
AFX_ISOLATIONAWARE_FUNC(BOOL,GetOpenFileNameW,(LPOPENFILENAMEW
unnamed1),(unnamed1),FALSE)
Why the code would tread through that invocation over a handful of times is
beyond me. Any guesses?
jbreher
"Joseph M. Newcomer" wrote:
> There was a disaster in the VS6-XP versions because of a change in the size of the file
> dialog.
So am I to assume that I should not have any such issues, as I am using a
subsequent release of VS?
> Note that phenomena such as having too many windows, running out of GDI space, etc. can
> result in some of the problems you are seeing. So key here is to replicate the
> environment. I (and others) can do
> CFileDialog dlg(TRUE);
> dlg.DoModal();
>
> without problem, so if it is failing for you, there is something else wrong. But since it
> is impossible to intuit what your entire environment must be like, it is hard to do more
> than guess.
This gives me my first insight. I had been thinking in terms of something
wrong with my application directly, rather than an environmental issue. I
took a detour to run another app that I built last week, which uses the same
code. While it used to operate just fine, it now exhibits the same inability
to DoModal().
> I'd look at resource exhaustion as a likely candidate. Can you write a trivial example
> that just does the CFileDialog dlg(TRUE) as above, in a little test app, and see it work?
> If so, then the problem is the environment your app is presenting to it. Programs like
> the process viewer (from www.sysinternals.com) could give some figures about things like
> GDI handle usage; if the numbers are far out of line, this could be the problem.
I'll see if I can find said process viewer. I have used other sysinternals
tools before (I recall something that monitored accesses to COM and LPT
ports). I'll report back with any findings. In the meantime, if you have any
links to a 'process viewer 101 training doc', it would be most appreciated.
Norbert Unterberg
>
> "Joseph M. Newcomer" wrote:
>
>> This is not a good description. If DoModal "fails", it will return 0. If it succeeds,
>> you should see a dialog box.
>
> Sorry for my imprecise language. What shall we call it when an API
> invocation does not do what the documentation purports it to do? You are
> right, it did not 'fail' in the sense that it did not return a value
> indicating failure. It never returned, period. However, I apparently have no
> File Open dialog. At least none that I can find.
Opening the file open dialog causes possibly all explorer shell
extensions and plug-ins to be loaded into your application's context.
The file open dialog contains a full featured explorer window.
So it can actually happen that some shell extension does not work
correctly within your application.
You should try to remove some of the shell extension. Does anybody know
a tool to display (and disable) all extensions that are registered and
hooked into explorer?
Norbert
Alexander Grigoriev
When you get the call to hang, break inside the debugger and check the call
stack.
"jbreher" <jbr...@discussions.microsoft.com> wrote in message
news:4B01B52F-ED20-4E7D...@microsoft.com...
jbreher
"Norbert Unterberg" wrote:
>
> Opening the file open dialog causes possibly all explorer shell
> extensions and plug-ins to be loaded into your application's context.
> The file open dialog contains a full featured explorer window.
>
> So it can actually happen that some shell extension does not work
> correctly within your application.
>
> You should try to remove some of the shell extension. Does anybody know
> a tool to display (and disable) all extensions that are registered and
> hooked into explorer?
>
> Norbert
>
I am afraid that I do not have the necessary background to understand how to
make use of this information. By 'shell extension', are you speaking of
third-party things that integrate themselves into the Explorer interface?
(e.g. WinZip, Tortoise-SVN, etc.)
I am certainly more than happy to disable such extensions - as a debugging
mechanism if nothing else. However, I do not know how to determine what
extensions I may have running, nor how to actually disable them once
identified.
jbreher
"Alexander Grigoriev" wrote:
> Make sure you don't call CoInitializeEx(COINIT_MULTITHREADED) before that.
I certainly do not call it explicitly. However, if operations upon some
other MFC object may have invoked this 'under the hood', I wouldn't know.
> When you get the call to hang, break inside the debugger and check the call
> stack.
I tried that before my orignal post (guess I should have indicated such -
sorry). Upon Hitting the Break All button, I get the following dialog:
Microsoft Visual Studio [?][X]
<!> No symbols can be loaded for any call stack frame. The source code
cannot be displayed.
[_] Show disassembly automatically
[ OK ] [Show Disassembly]
I have tried to decipher what the assembly code is trying to do. However,
the lack of symbols makes this rather inscrutable as well.
Drew
If you think you may have (or some other program may have) exhausted your
resources just reboot and try your app again.
Drew
Joseph M. Newcomer
But I wouldn't know how to disable them, either. Note that if it is only one app that has
this problem, the problem may be the app environment as I indicated earlier.
Shell extensions are somewhat of a nightmare in this regard.
joe
Joseph M. Newcomer
have some serious stack damage. An example would be stack damage that causes a return to
a place where there is a set of instructions that go into an infinite loop.
This is just a guess. Confronted with this problem, I'd do a LOT of single step
debugging.
Also, if you have access to a machine that is fairly simply configured (no shell
extensions) try running it on that machine. If it comes up successfully, it *might* be
the shell extensions problem suggested earlier.
joe
Joseph M. Newcomer
On Thu, 21 Feb 2008 22:22:01 -0800, jbreher <jbr...@discussions.microsoft.com> wrote:
>
>
>"Joseph M. Newcomer" wrote:
>
>> There was a disaster in the VS6-XP versions because of a change in the size of the file
>> dialog.
>
>So am I to assume that I should not have any such issues, as I am using a
>subsequent release of VS?
****
Yes. Newer versions of VS would not show this problem
****
>
>> Note that phenomena such as having too many windows, running out of GDI space, etc. can
>> result in some of the problems you are seeing. So key here is to replicate the
>> environment. I (and others) can do
>> CFileDialog dlg(TRUE);
>> dlg.DoModal();
>>
>> without problem, so if it is failing for you, there is something else wrong. But since it
>> is impossible to intuit what your entire environment must be like, it is hard to do more
>> than guess.
>
>This gives me my first insight. I had been thinking in terms of something
>wrong with my application directly, rather than an environmental issue. I
>took a detour to run another app that I built last week, which uses the same
>code. While it used to operate just fine, it now exhibits the same inability
>to DoModal().
****
OK, going back to some of the other suggestions that have been made:
if it is a problem with shell extensions, you should see it in all apps
if it is a problem with resource exhaustion, you should see it if you run the failing
program, and while the failing program is running, the other program fails. The suggested
reboot-and-retry would indicate if it is the result of some progressive debilitation of
the system (perhaps some other random program is consuming all the resources, for example,
and your program, and now the other program, are victims of a drive-by resource hogging)
A weird idea, but just for jollies, try to launch a color dialog, which uses a lot of
resources. If it is ONLY CFileDialog, then it could be the explorer, or some of the other
ideas I suggested. If CColorDialog doesn't come up, it is likely to be a resource
problem. Whether it is your app consuming the resources, or some other app (watch out for
background programs that come up when you log in!), you would see the same kind of
failure. I'd give it a shot just to see if the problem appears when any complex dialog
comes up, or if it is restricted to CFileDialog.
****
>
>> I'd look at resource exhaustion as a likely candidate. Can you write a trivial example
>> that just does the CFileDialog dlg(TRUE) as above, in a little test app, and see it work?
>> If so, then the problem is the environment your app is presenting to it. Programs like
>> the process viewer (from www.sysinternals.com) could give some figures about things like
>> GDI handle usage; if the numbers are far out of line, this could be the problem.
>
>I'll see if I can find said process viewer. I have used other sysinternals
>tools before (I recall something that monitored accesses to COM and LPT
>ports). I'll report back with any findings. In the meantime, if you have any
>links to a 'process viewer 101 training doc', it would be most appreciated.
****
It comes with documentation. What you're looking for is handle consumption and GDI
resource consumption.
joe
****
Joseph M. Newcomer
Getting lost is not the problem. Seeing where it finally hangs can be interesting.
Your problem is probably completely unrelated to MFC; it is probably much deeper. For
example, stack damage is not MFC-related.
I looked in my VS2005 source listings, and there is no such line in my copy of
afxcomctl32.h. The fact that it goes through it several times, though, is definitely
suspect.
joe
Joseph M. Newcomer
joe
On Thu, 21 Feb 2008 22:27:01 -0800, jbreher <jbr...@discussions.microsoft.com> wrote:
>
>
>"Joseph M. Newcomer" wrote:
>
>
>> If so, then the problem is the environment your app is presenting to it. Programs like
>> the process viewer (from www.sysinternals.com) could give some figures about
>
>Quick follow-up question - I don't seem to be able to locate any "process
>viewer" from sysinternals. Might you be referring to their "Process Explorer"?
Alexander Grigoriev
server.
See KB319037 for VS2002 instructions. I think for 2003+ you don't have to
get symsrv.dll.
"jbreher" <jbr...@discussions.microsoft.com> wrote in message
news:6CACF5B7-3B47-4BBB...@microsoft.com...