Read blocking nsInputStream without blocking
PM
I need to read a blocking nsInputStream (that doesn’t implement
nsIAsyncInputStream interface) in main thread without blocking and save
it into output stream. Does it exist any helper class doing so(in C++)?
I need to use it with nsIMsgMessageService::StreamMessage.
Any help is welcome.
Thanks
Boris Zbarsky
> I need to read a blocking nsInputStream (that doesn’t implement
> nsIAsyncInputStream interface) in main thread without blocking and save
> it into output stream. Does it exist any helper class doing so(in C++)?
NS_AsyncCopy?
-Boris
PM
nsInputStreamTee doesn't implement nsIAsyncInputStream.
It cause deadlock of main thread.
nsCOMPtr<nsIEventQueueService> eqs =
do_GetService(kEventQueueServiceCID, &rv);
if ( NS_FAILED(rv) || NULL == eqs.get() )
{
return SetStatusCode(rv);
}
nsCOMPtr<nsIEventQueue> targetEventQueue;
rv = eqs->ResolveEventQueue(NS_CURRENT_EVENTQ,
getter_AddRefs(targetEventQueue));
if ( NS_FAILED(rv) || NULL == targetEventQueue.get() )
{
return SetStatusCode(rv);
}
rv = NS_AsyncCopy(aInputStream, m_outputStream,
/*targetEventQueue*/NULL, NS_ASYNCCOPY_VIA_READSEGMENTS, aCount,
AsyncCopyCallbackFun, this);
Have you any idea?
Boris Zbarsky
> It doesn't help because inputstream is nsInputStreamTee.
> nsInputStreamTee doesn't implement nsIAsyncInputStream.
Have you considered wrapping it in a stream that does?
> It cause deadlock of main thread.
It should just cause the copy to terminate early.
> rv = NS_AsyncCopy(aInputStream, m_outputStream,
> /*targetEventQueue*/NULL, NS_ASYNCCOPY_VIA_READSEGMENTS, aCount,
> AsyncCopyCallbackFun, this);
Doesn't the NS_AsyncCopy clearly say that a null target is not permitted?
-Boris
PM
> PM wrote:
>> It doesn't help because inputstream is nsInputStreamTee.
>> nsInputStreamTee doesn't implement nsIAsyncInputStream.
>
> Have you considered wrapping it in a stream that does?
nsCOMPtr<nsIStreamTransportService> sts =
do_GetService(NS_STREAMTRANSPORTSERVICE_CONTRACTID, &rv);
nsCOMPtr<nsITransport> srcTransport;
rv = sts->CreateInputTransport(aInputStream, nsInt64(-1), nsInt64(-1),
PR_TRUE, getter_AddRefs(srcTransport));
nsCOMPtr<nsIInputStream> asyncInputStream;
rv = srcTransport->OpenInputStream(nsITransport::OPEN_UNBUFFERED, 0, 0,
getter_AddRefs(asyncInputStream));
if (NS_FAILED(rv)) return rv;
rv = NS_AsyncCopy(asyncInputStream, m_outputStream, targetEventQueue,
NS_ASYNCCOPY_VIA_READSEGMENTS, aCount, AsyncCopyCallbackFun, this);
It seems that this work but I get demaged data in output stream.
I published this on:
mozilla.dev.apps.thunderbird: StreamMessage and GMail
Have you any idea?
>> It cause deadlock of main thread.
>
> It should just cause the copy to terminate early.
>
>> rv = NS_AsyncCopy(aInputStream, m_outputStream,
>> /*targetEventQueue*/NULL, NS_ASYNCCOPY_VIA_READSEGMENTS, aCount,
>> AsyncCopyCallbackFun, this);
>
> Doesn't the NS_AsyncCopy clearly say that a null target is not permitted?
Sorry my mistake.
I get NS_OK as result of NS_AsyncCopy call in both cases:
- NS_AsyncCopy(aInputStream, m_outputStream, NULL, ...
- NS_AsyncCopy(aInputStream, m_outputStream, targetEventQueue
I got NS_OK even aInputStream doesn't implement nsIAsyncInputStream.
PM-
Boris Zbarsky
>>> It doesn't help because inputstream is nsInputStreamTee.
>>> nsInputStreamTee doesn't implement nsIAsyncInputStream.
>>
>> Have you considered wrapping it in a stream that does?
> You mean this?
No, I meant something like a buffered stream, but those don't seem to
implement nsIAsyncInputStream either.
That said, if the stream really is blocking, you shouldn't need
nsIAsyncInputStream, no?
>>> It cause deadlock of main thread.
>>
>> It should just cause the copy to terminate early.
This only if the stream is non-blocking.
> I get NS_OK as result of NS_AsyncCopy call in both cases:
> - NS_AsyncCopy(aInputStream, m_outputStream, NULL, ...
> - NS_AsyncCopy(aInputStream, m_outputStream, targetEventQueue
Yes, true. Arguably a bug.
> I got NS_OK even aInputStream doesn't implement nsIAsyncInputStream.
That's because it doesn't have to for the copy to work. It needs to
either be blocking or implement nsIAsyncInputStream.
-Boris
Boris Zbarsky
> That said, if the stream really is blocking, you shouldn't need
> nsIAsyncInputStream, no?
Key here being that if it's blocking, the copy will just block and not
need the nsIAsync*Stream machinert for knowing when to resume.
>> I get NS_OK as result of NS_AsyncCopy call in both cases:
>> - NS_AsyncCopy(aInputStream, m_outputStream, NULL, ...
>> - NS_AsyncCopy(aInputStream, m_outputStream, targetEventQueue
>
> Yes, true. Arguably a bug.
>
>> I got NS_OK even aInputStream doesn't implement nsIAsyncInputStream.
>
> That's because it doesn't have to for the copy to work. It needs to
> either be blocking or implement nsIAsyncInputStream.
One thing that does confuse me here... you pass null for the target
parameter to NS_AsyncCopy. NS_AsyncCopy creates an nsStreamCopierIB (in
your case, since you passed NS_ASYNCCOPY_VIA_READSEGMENTS) and calls
Start() on it with that target. Start() assigns the target to mTarget
and calls PostContinuationEvent(). PostContinuationEvent() calls a
method on mTarget.
So your call should be crashing.
-Boris
Honza Bambas
http://209.85.129.132/search?q=cache:CUw7ViW_vcoJ:www.xulplanet.com/references/xpcomref/ifaces/nsIAsyncStreamCopier.html+site:www.xulplanet.com+nsIAsyncStreamCopier&cd=1&hl=cs&ct=clnk&gl=cz&client=firefox-a
This makes an async stream from a "normal" stream. Sorry,
www.xulplanet.com is dead, this is google cache.
Other useful reference here:
http://209.85.129.132/search?q=cache:tT5gmBV9atIJ:www.xulplanet.com/references/xpcomref/group_Streams.html+site:www.xulplanet.com+streams&cd=1&hl=cs&ct=clnk&gl=cz&client=firefox-a
-hb-
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
>
Pavol Misik
> Boris Zbarsky wrote:
>> That said, if the stream really is blocking, you shouldn't need
>> nsIAsyncInputStream, no?
>
> Key here being that if it's blocking, the copy will just block and not
> need the nsIAsync*Stream machinert for knowing when to resume.
It seems that it was wasting of time.
Here is a reason (whole story).
I created my class MyStreamListener which implements nsIStreamListener
interface.
Instance of my class I pass as parameter aConsumer into call
nsIMsgMessageService::StreamMessage(const char *aMessageURI, nsISupports
*aConsumer, nsIMsgWindow *aMsgWindow, nsIUrlListener *aUrlListener,
PRBool aConvertData, const char *aAdditionalHeader, nsIURI **_retval);
When MyStreamListener::OnDataAvailable(nsIRequest *aRequest, nsISupports
*aContext, nsIInputStream *aInputStream, PRUint32 aOffset, PRUint32 aCount)
was called I tried to read aCount from aInputStream.
I can read it without blocking by nsIInputStream::Read, but I always got
damaged data (email in rfc822 format).
Then I tried to use nsIInputStream::ReadSegments. I find
nsPipeInputStream::Read implementation in nsPipe3.cpp.
I copy pasted it into my code.
static NS_METHOD
nsWriteToRawBuffer(nsIInputStream* inStr,
void *closure,
const char *fromRawSegment,
PRUint32 offset,
PRUint32 count,
PRUint32 *writeCount)
{
char *toBuf = (char*)closure;
memcpy(&toBuf[offset], fromRawSegment, count);
*writeCount = count;
return NS_OK;
}
NS_IMETHODIMP
MyRead(char* toBuf, PRUint32 bufLen, PRUint32 *readCount)
{
return ReadSegments(nsWriteToRawBuffer, toBuf, bufLen, readCount);
}
but I dot the same result damaged data.
Then I took I look into nsIInputStream.idl where I found *The writer
function may be called multiple times for segmented buffers.*
I realized that it can overwrite beginning of the buffer if it is called
by nsIInputstream::ReadSegments with wrong offset or cause buffer
overrun if is called with wrong count (greater then buffer toBuf really
is). This can happened if nsIInputstream::ReadSegments is written badly.
And I think that it is potential weakness or security bug of Mozilla
code. Just trust that caller call callback with correct parameters
without checking parameters is way to hell.
I compared gecko 1.8 and 1.9.1 and there is the same just
nsWriteToRawBuffer has new name NS_CopySegmentToBuffer
I wrote my own "safe" version - which appended data with checking range
and detecting potential data overwrite.
I always got damaged data.
Again I took a look into nsIInputStream.idl and I find IsNonBlocking. I
started to thing that I have to wait until stream is nonblocking until I
can read data.
I used NS_AsyncCopy this way:
nsCOMPtr<nsIStreamTransportService> sts =
do_GetService(NS_STREAMTRANSPORTSERVICE_CONTRACTID, &rv);
nsCOMPtr<nsITransport> srcTransport;
rv = sts->CreateInputTransport(aInputStream, nsInt64(-1), nsInt64(-1),
PR_TRUE, getter_AddRefs(srcTransport));
if (NS_FAILED(rv)) return rv;
nsCOMPtr<nsIInputStream> asyncInputStream;
rv = srcTransport->OpenInputStream(nsITransport::OPEN_UNBUFFERED, 0, 0,
getter_AddRefs(asyncInputStream));
if (NS_FAILED(rv)) return rv;
AddRef(); // because closure is this
rv = NS_AsyncCopy(asyncInputStream, m_outputStream,
targetEventQueue, NS_ASYNCCOPY_VIA_READSEGMENTS, aCount,
AsyncCopyCallbackFun, this);
if (NS_FAILED(rv)) Release(); // because closure is this
Which finished without crash but I again got the same result as at the
beginning.
Now I took a look into nsIStreamListener.idl. There is written this:
*Called when the next chunk of data (corresponding to the request) may
* be read without blocking the calling thread.*
Therefore it seems my approach is bad and problem is somewhere else.
Now I am hopeless. I have no idea what I'm doing wrong.
> One thing that does confuse me here... you pass null for the target
> parameter to NS_AsyncCopy. NS_AsyncCopy creates an nsStreamCopierIB (in
> your case, since you passed NS_ASYNCCOPY_VIA_READSEGMENTS) and calls
> Start() on it with that target. Start() assigns the target to mTarget
> and calls PostContinuationEvent(). PostContinuationEvent() calls a
> method on mTarget.
>
> So your call should be crashing.
Yes, it crashed when i pass null for the target parameter.
First of all I didn't noticed that NS_ASSERTION(target, "non-null target
required");
Now I generated preprocessed file and that macro was preprocessed into
do { } while (0); Therefore it continues without returning error and
later crashed. I copy-pasted wrong version. As you can see in my second
message of this thread there was commented out target parameter.
rv = NS_AsyncCopy(aInputStream, m_outputStream,
/*targetEventQueue*/NULL, NS_ASYNCCOPY_VIA_READSEGMENTS, aCount,
AsyncCopyCallbackFun, this);
Sorry for wrong information.
PM-
Boris Zbarsky
> I tried to copy it in main thread of Thunderbird without blocking.
Um. How do you expect that to work? The stream is blocking. Whatever
thread the Read() call happens on will block. So your options are to
either do the copy on a worker thread (and it'll block, but that's ok)
or ... what, exactly?
> When MyStreamListener::OnDataAvailable(nsIRequest *aRequest, nsISupports
> *aContext, nsIInputStream *aInputStream, PRUint32 aOffset, PRUint32 aCount)
>
> was called I tried to read aCount from aInputStream.
> I can read it without blocking by nsIInputStream::Read, but I always got
> damaged data (email in rfc822 format).
Sounds like whoever is calling you is buggy, then. Did you try
examining the given stream in a debugger to see what's up?
> Then I took I look into nsIInputStream.idl where I found *The writer
> function may be called multiple times for segmented buffers.*
>
> I realized that it can overwrite beginning of the buffer if it is called
> by nsIInputstream::ReadSegments with wrong offset
If it is, the input stream implementation is buggy. If you implement
ReadSegments, you promise to pass correct offsets to the writer.
> This can happened if nsIInputstream::ReadSegments is written badly.
Yes. Of course if it's written badly it can also just read from memory
it shouldn't be reading from and crash. Or it can just write to
arbitrary memory itself and crash.
> And I think that it is potential weakness or security bug of Mozilla
> code. Just trust that caller call callback with correct parameters
> without checking parameters is way to hell.
Well... You have the option of not using ReadSegments if you don't
trust your stream implementation. But then you probably shouldn't trust
the data you get from it either.
> I always got damaged data.
Right; you're still doing things that are equivalent to that Read() call
above.
> Again I took a look into nsIInputStream.idl and I find IsNonBlocking. I
> started to thing that I have to wait until stream is nonblocking until I
> can read data.
A stream is either blocking or not. This state doesn't change during
the lifetime of the stream. Blocking just means that if you try to read
more data than is present in the stream then the Read() call will block
waiting for the extra data. Non-blocking means that instead of blocking
it will return the NS_ERROR_BASE_STREAM_WOULD_BLOCK error code.
> Which finished without crash but I again got the same result as at the
> beginning.
Yes. Because the stream is just handing you bogus data, no matter how
you read it.
> *Called when the next chunk of data (corresponding to the request) may
> * be read without blocking the calling thread.*
>
> Therefore it seems my approach is bad and problem is somewhere else.
Right. The problem is in whatever code is creating the stream, sounds like.
> Now I am hopeless. I have no idea what I'm doing wrong.
Nothing, unless you're also implementing the StreamMessage code.
> Now I generated preprocessed file and that macro was preprocessed into
> do { } while (0);
In a non-debug build, yes.
-Boris
Pavol Misik
> Sounds like whoever is calling you is buggy, then. Did you try
> examining the given stream in a debugger to see what's up?
This is callstack:
eplgTb.dll!eAsyncStreamListener::OnDataAvailable is my implementation.
>kb
Index Function
--------------------------------------------------------------------------------
*1 eplgTb.dll!eAsyncStreamListener::OnDataAvailable(nsIRequest *
aRequest=0x0557ec98, nsISupports * aContext=0x0537cbec, nsIInputStream *
aInputStream=0x051bb590, unsigned int aOffset=0x00000000, unsigned int
aCount=0x000007b2)
2 mail.dll!nsImapCacheStreamListener::OnDataAvailable(nsIRequest
* request=0x053a12e0, nsISupports * aCtxt=0x0537cbec, nsIInputStream *
aInStream=0x051bb590, unsigned int aSourceOffset=0x00000000, unsigned
int aCount=0x000007b2)
3 necko.dll!nsInputStreamPump::OnStateTransfer()
4
necko.dll!nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream *
stream=0x051bb590)
5 xpcom_core.dll!nsInputStreamReadyEvent::EventHandler(PLEvent *
plevent=0x053b7de4)
6 xpcom_core.dll!PL_HandleEvent(PLEvent * self=0x053b7de4)
7 xpcom_core.dll!PL_ProcessPendingEvents(PLEventQueue *
self=0x00e13a68)
8 xpcom_core.dll!_md_TimerProc(HWND__ * hwnd=0x001f0a5a, unsigned
int uMsg=0x00000113, unsigned int idEvent=0x00000000, unsigned long
dwTime=0x071c3ac2)
9 user32.dll!_InternalCallWinProc@20()
10 user32.dll!_UserCallWinProc@24()
11 user32.dll!_DispatchMessageWorker@8()
12 user32.dll!_DispatchMessageW@4()
13 gkwidget.dll!nsAppShell::Run()
14 tkitcmps.dll!nsAppStartup::Run()
15 thunderbird.exe!XRE_main(int argc=0x00000002, char * *
argv=0x003d8290, const nsXREAppData * aAppData=0x004268bc)
16 thunderbird.exe!main(int argc=0x00000002, char * *
argv=0x003d8290)
17 thunderbird.exe!__tmainCRTStartup()
18 thunderbird.exe!mainCRTStartup()
19 kernel32.dll!_BaseProcessStart@4()
> A stream is either blocking or not. This state doesn't change during
> the lifetime of the stream. Blocking just means that if you try to read
> more data than is present in the stream then the Read() call will block
> waiting for the extra data. Non-blocking means that instead of blocking
> it will return the NS_ERROR_BASE_STREAM_WOULD_BLOCK error code.
Thanks for explanation.
>> Which finished without crash but I again got the same result as at the
>> beginning.
>
> Yes. Because the stream is just handing you bogus data, no matter how
> you read it.
>
>> *Called when the next chunk of data (corresponding to the request) may
>> * be read without blocking the calling thread.*
>>
>> Therefore it seems my approach is bad and problem is somewhere else.
>
> Right. The problem is in whatever code is creating the stream, sounds
> like.
Therefore I thing that problem is in mail.dll!nsImapCacheStreamListener.
>> Now I am hopeless. I have no idea what I'm doing wrong.
>
> Nothing, unless you're also implementing the StreamMessage code.
No, I don't implement it. It is implemented by Mozilla. It calls
nsImapService::StreamMessage.
here are my call with parameters
rv = msgService->StreamMessage(aMessageURI,
(nsISupports*)((nsIStreamListener*)listener), msgWnd, nsnull, PR_FALSE,
nsnull, nsnull);
on POP3 it call nsMailboxService::StreamMessage.
I didn't reproduced this bug on POP3 account, unfortunately we had a
feedback - that our extension damaged on POP3 accounts too.
PM-
Boris Zbarsky
> Boris Zbarsky wrote:
>> Sounds like whoever is calling you is buggy, then. Did you try
>> examining the given stream in a debugger to see what's up?
>
> This is callstack:
I really did mean examining the stream object, not just the stack
> Therefore I thing that problem is in mail.dll!nsImapCacheStreamListener.
That's just something that takes the data and passes it on, no? If it
does this correctly (and I suspect it does), then the real issue is in
whatever is setting up that stream pump, if the data is really wrong.
Or possibly the issue is in the assumption about what the data should
look like.
-Boris
Pavol Misik
> I really did mean examining the stream object, not just the stack
I noticed that it exists at least two sources which calls my
eAsyncStreamListener::OnDataAvailable.
One is necko.dll!nsStreamListenerTee::OnDataAvailable
other is mail.dll!nsImapCacheStreamListener::OnDataAvailable
>kb
Index Function
--------------------------------------------------------------------------------
*1 eplgTb.dll!eAsyncStreamListener::OnDataAvailable(nsIRequest *
aRequest=0x0535ff18, nsISupports * aContext=0x053890ac, nsIInputStream *
aInputStream=0x050c61d8, unsigned int aOffset=0x00000000, unsigned int
aCount=0x00000b10)
2 necko.dll!nsStreamListenerTee::OnDataAvailable(nsIRequest *
request=0x0535ff18, nsISupports * context=0x053890ac, nsIInputStream *
input=0x0516e4d8, unsigned int offset=0x00000000, unsigned int
count=0x00000b10)
3 necko.dll!nsOnDataAvailableEvent0::HandleEvent()
4 necko.dll!nsStreamListenerEvent0::HandlePLEvent(PLEvent *
aEvent=0x054e7e68)
5 xpcom_core.dll!PL_HandleEvent(PLEvent * self=0x054e7e68)
6 xpcom_core.dll!PL_ProcessPendingEvents(PLEventQueue *
self=0x00e13a68)
7 xpcom_core.dll!_md_TimerProc(HWND__ * hwnd=0x003f0bda, unsigned
int uMsg=0x00000113, unsigned int idEvent=0x00000000, unsigned long
dwTime=0x076f749b)
8 user32.dll!_InternalCallWinProc@20()
9 user32.dll!_UserCallWinProc@24()
10 user32.dll!_DispatchMessageWorker@8()
11 user32.dll!_DispatchMessageW@4()
12 gkwidget.dll!nsAppShell::Run()
13 tkitcmps.dll!nsAppStartup::Run()
14 thunderbird.exe!XRE_main(int argc=0x00000002, char * *
argv=0x003d8290, const nsXREAppData * aAppData=0x004268bc)
15 thunderbird.exe!main(int argc=0x00000002, char * *
argv=0x003d8290)
16 thunderbird.exe!__tmainCRTStartup()
17 thunderbird.exe!mainCRTStartup()
18 kernel32.dll!_BaseProcessStart@4()
>
>> Therefore I thing that problem is in mail.dll!nsImapCacheStreamListener.
>
> That's just something that takes the data and passes it on, no? If it
> does this correctly (and I suspect it does), then the real issue is in
> whatever is setting up that stream pump, if the data is really wrong.
I have to investigate yet.
> Or possibly the issue is in the assumption about what the data should
> look like.
I don't think. I agree that it is not documented.
I investigated way how Thunderbird saves messages to file. It uses
nsIMsgMessageService::StreamMessage. Even I tried to use that function
to save email to file from my event handler executed from main thread
and it saved badly too. If I do it from Thunderbird (Save message) I get
result I expected and correct.
It happened just in case message is downloaded from server I try to read
it with nsIMsgMessageService::StreamMessage. It looks like thread
concurrence problem or cache and offline storage inconsistence.
Unfortunately when source is
mail.dll!nsImapCacheStreamListener::OnDataAvailable I don't receive any
error code nsIStreamListener::OnStopRequest or nsIUrlListener.
PM-