Brix Resources Streamer (images, css etc.) - partly content/ problematic flushing
27 views
Skip to first unread message
Korbinian
Jun 19, 2012, 5:15:23 AM6/19/12
to brix-cms-discuss
Hi,
our brix app is an e-commerce application. While its production we
regularly trace it and try to find weak spots in our code. During this
we found 2 problematic issues:
1st.: DiskPageStore can sometimes increase page delivery time by about
200 - 1000ms as it sometimes needs to wait for writes; We don't have
yet much traffic, so this is something we didnt expect but now try to
solve it by changing its file handles from 50 to 100; (done last
night) - I'll keep an eye onto this (possible other solution would be
some kind of Ram-Disk for it - HttpSessionStore can't be used as it
breaks ajax and kills multitab)
2nd.: Especially resource files like JPG, PNG, CSS gave us a headache.
Those sometimes where reported to take up to 120 sek's running time.
Investigation took a while till we found where time is lost, its
basically the current implementation of
org.brixcms.plugin.site.resource.Streamer.stream();
Problematic is the fact that the content is tried to get by parsing of
an header ("Range") that at least on Glassfish and JBoss wont work
(returns null/null initaliezed Range object).
So, the status will be given an 200 in that cases, whats ok. However,
the write to response.getOutputStream().write(....)
is followed by
response.flushBuffer();
As current implementation sets the buffer to int bufferSize =1024*10
this means that every 10KB of data a response.flush is thrown;
Beside the fact that its questionable that brix can handle this better
than the container (at least it would be his job to multipart and
deliver data), it now has a impact on th handling of bigger images/
files.
If the file is a 150KB file we issue at least 15s flush(), forcing 15
different writes to the client and during that time, we also have the
InputStream open, wich means we run into concurrency trouble (e.g:
imagein this 150KB image is a main-page image on an webpage) as we
depend (!) on the time of the client to receive the data (flush writes
to client and needs that time - if client is slow, we're bugged, as 15
x slow = very very slow in the end);
Of course it might make sense not to directly put in a 5 MB file, and
so I'm not sure how to fix this in Brix.
My current idea is to introduce a boolean switch that at least
suppresses all flush in case we set an 200' header (SC_OK) and else we
atleast wait for 250KB (25's) till we issue a flush in other case -
however, I'm still not sure the current implementation is a good idea
as this is IMHO not Brix's job to do that after all.
Any Idea on that would really be welcome
our brix app is an e-commerce application. While its production we
regularly trace it and try to find weak spots in our code. During this
we found 2 problematic issues:
1st.: DiskPageStore can sometimes increase page delivery time by about
200 - 1000ms as it sometimes needs to wait for writes; We don't have
yet much traffic, so this is something we didnt expect but now try to
solve it by changing its file handles from 50 to 100; (done last
night) - I'll keep an eye onto this (possible other solution would be
some kind of Ram-Disk for it - HttpSessionStore can't be used as it
breaks ajax and kills multitab)
2nd.: Especially resource files like JPG, PNG, CSS gave us a headache.
Those sometimes where reported to take up to 120 sek's running time.
Investigation took a while till we found where time is lost, its
basically the current implementation of
org.brixcms.plugin.site.resource.Streamer.stream();
Problematic is the fact that the content is tried to get by parsing of
an header ("Range") that at least on Glassfish and JBoss wont work
(returns null/null initaliezed Range object).
So, the status will be given an 200 in that cases, whats ok. However,
the write to response.getOutputStream().write(....)
is followed by
response.flushBuffer();
As current implementation sets the buffer to int bufferSize =1024*10
this means that every 10KB of data a response.flush is thrown;
Beside the fact that its questionable that brix can handle this better
than the container (at least it would be his job to multipart and
deliver data), it now has a impact on th handling of bigger images/
files.
If the file is a 150KB file we issue at least 15s flush(), forcing 15
different writes to the client and during that time, we also have the
InputStream open, wich means we run into concurrency trouble (e.g:
imagein this 150KB image is a main-page image on an webpage) as we
depend (!) on the time of the client to receive the data (flush writes
to client and needs that time - if client is slow, we're bugged, as 15
x slow = very very slow in the end);
Of course it might make sense not to directly put in a 5 MB file, and
so I'm not sure how to fix this in Brix.
My current idea is to introduce a boolean switch that at least
suppresses all flush in case we set an 200' header (SC_OK) and else we
atleast wait for 250KB (25's) till we issue a flush in other case -
however, I'm still not sure the current implementation is a good idea
as this is IMHO not Brix's job to do that after all.
Any Idea on that would really be welcome
Martin Grigorov
Jun 19, 2012, 5:34:03 AM6/19/12
to brix-cms...@googlegroups.com
Hi,
Issue 1) I think you still use Wicket 1.4 so I cannot tell you how
exactly to configure this but I'm sure there is a support for
asynchronous write to the disk.
In 1.5 you can use
org.apache.wicket.settings.def.StoreSettings#setAsynchronous(true) to
make the write asynchronous. In this case the http thread just adds a
job that should write the page bytes but doesn't wait for that task to
be executed.
Issue 2)
Again in 1.5 there is
org.apache.wicket.request.resource.AbstractResource#flushResponseAfterHeaders().
A comment from Matej says that the flush is needed because otherwise
Firefox somehow breaks the image rendering.
We also hit this problem and I made this flushing configurable by
overriding the method above.
I guess something similar can be done for Brix's Streamer.
> --
> You received this message because you are subscribed to the Google Groups "brix-cms-discuss" group.
> To post to this group, send email to brix-cms...@googlegroups.com.
> To unsubscribe from this group, send email to brix-cms-discu...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/brix-cms-discuss?hl=en.
>
Issue 1) I think you still use Wicket 1.4 so I cannot tell you how
exactly to configure this but I'm sure there is a support for
asynchronous write to the disk.
In 1.5 you can use
org.apache.wicket.settings.def.StoreSettings#setAsynchronous(true) to
make the write asynchronous. In this case the http thread just adds a
job that should write the page bytes but doesn't wait for that task to
be executed.
Issue 2)
Again in 1.5 there is
org.apache.wicket.request.resource.AbstractResource#flushResponseAfterHeaders().
A comment from Matej says that the flush is needed because otherwise
Firefox somehow breaks the image rendering.
We also hit this problem and I made this flushing configurable by
overriding the method above.
I guess something similar can be done for Brix's Streamer.
> You received this message because you are subscribed to the Google Groups "brix-cms-discuss" group.
> To post to this group, send email to brix-cms...@googlegroups.com.
> To unsubscribe from this group, send email to brix-cms-discu...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/brix-cms-discuss?hl=en.
>
Korbinian
Jun 19, 2012, 3:30:52 PM6/19/12
to brix-cms-discuss
Hi Martin,
yes, its on wicket 1.4 - I'm working on making brix happy with wicket
6.0 and I hope that by the weekend I can send the first updates to
github.
Reason why I'm advancing from our current 1.4 wicket production is
that we switch to brix 6 / wicket 6 quite fast and I need to trace
down all possible problems we might face.
I saw some async in wicket 1.4's DiskPageStore as well, will have a
look at it - thx for this hint!
Regarding the flushing:
I didnt see any problems with firefox (its our own default browser),
but of course there could be some problems. Can you point me to any
trace Matej left about this?
I personally would understand that a flush is needed (even I then
don't understand whats so special here, as every container should
handle this?), but I think it should be enough to issue 1 flush at the
end and not every 10KB of data?
yes, its on wicket 1.4 - I'm working on making brix happy with wicket
6.0 and I hope that by the weekend I can send the first updates to
github.
Reason why I'm advancing from our current 1.4 wicket production is
that we switch to brix 6 / wicket 6 quite fast and I need to trace
down all possible problems we might face.
I saw some async in wicket 1.4's DiskPageStore as well, will have a
look at it - thx for this hint!
Regarding the flushing:
I didnt see any problems with firefox (its our own default browser),
but of course there could be some problems. Can you point me to any
trace Matej left about this?
I personally would understand that a flush is needed (even I then
don't understand whats so special here, as every container should
handle this?), but I think it should be enough to issue 1 flush at the
end and not every 10KB of data?
Korbinian
Jun 19, 2012, 3:36:37 PM6/19/12
to brix-cms-discuss
PS: according to Streamer.stream() there is no explicit flush of the
headers, only of headers + content (every 10KB)
headers, only of headers + content (every 10KB)
Martin Grigorov
Jun 19, 2012, 3:41:20 PM6/19/12
to brix-cms...@googlegroups.com
On Tue, Jun 19, 2012 at 10:30 PM, Korbinian
<korbini...@googlemail.com> wrote:
> Hi Martin,
>
> yes, its on wicket 1.4 - I'm working on making brix happy with wicket
> 6.0 and I hope that by the weekend I can send the first updates to
> github.
>
> Reason why I'm advancing from our current 1.4 wicket production is
> that we switch to brix 6 / wicket 6 quite fast and I need to trace
> down all possible problems we might face.
>
> I saw some async in wicket 1.4's DiskPageStore as well, will have a
> look at it - thx for this hint!
>
> Regarding the flushing:
>
> I didnt see any problems with firefox (its our own default browser),
> but of course there could be some problems. Can you point me to any
> trace Matej left about this?
https://github.com/apache/wicket/blob/master/wicket-core/src/main/java/org/apache/wicket/request/resource/AbstractResource.java#L618
<korbini...@googlemail.com> wrote:
> Hi Martin,
>
> yes, its on wicket 1.4 - I'm working on making brix happy with wicket
> 6.0 and I hope that by the weekend I can send the first updates to
> github.
>
> Reason why I'm advancing from our current 1.4 wicket production is
> that we switch to brix 6 / wicket 6 quite fast and I need to trace
> down all possible problems we might face.
>
> I saw some async in wicket 1.4's DiskPageStore as well, will have a
> look at it - thx for this hint!
>
> Regarding the flushing:
>
> I didnt see any problems with firefox (its our own default browser),
> but of course there could be some problems. Can you point me to any
> trace Matej left about this?
It could be that it was some older version of Firefox and the bug is
already fixed ...
Korbinian
Jun 19, 2012, 3:54:48 PM6/19/12
to brix-cms-discuss
I introduced a flushBuffer right after the headers (those are quite
few data) and tested it against FireFox 2.20 - beside the fact that
this old 2006 thing didnt even understand all of our css at least all
images worked as expected;
I think with that flush introduced after headers the current flushing
in the while can be removed - at least the transfer times for bigger
images also shrink and performance goes up....
On 19 Jun., 21:41, Martin Grigorov <martin.grigo...@gmail.com> wrote:
> On Tue, Jun 19, 2012 at 10:30 PM, Korbinian
>
>
>
>
>
>
>
>
>
> <korbinian.ba...@googlemail.com> wrote:
> > Hi Martin,
>
> > yes, its on wicket 1.4 - I'm working on making brix happy with wicket
> > 6.0 and I hope that by the weekend I can send the first updates to
> > github.
>
> > Reason why I'm advancing from our current 1.4 wicket production is
> > that we switch to brix 6 / wicket 6 quite fast and I need to trace
> > down all possible problems we might face.
>
> > I saw some async in wicket 1.4's DiskPageStore as well, will have a
> > look at it - thx for this hint!
>
> > Regarding the flushing:
>
> > I didnt see any problems with firefox (its our own default browser),
> > but of course there could be some problems. Can you point me to any
> > trace Matej left about this?
>
> https://github.com/apache/wicket/blob/master/wicket-core/src/main/jav...
few data) and tested it against FireFox 2.20 - beside the fact that
this old 2006 thing didnt even understand all of our css at least all
images worked as expected;
I think with that flush introduced after headers the current flushing
in the while can be removed - at least the transfer times for bigger
images also shrink and performance goes up....
On 19 Jun., 21:41, Martin Grigorov <martin.grigo...@gmail.com> wrote:
> On Tue, Jun 19, 2012 at 10:30 PM, Korbinian
>
>
>
>
>
>
>
>
>
> <korbinian.ba...@googlemail.com> wrote:
> > Hi Martin,
>
> > yes, its on wicket 1.4 - I'm working on making brix happy with wicket
> > 6.0 and I hope that by the weekend I can send the first updates to
> > github.
>
> > Reason why I'm advancing from our current 1.4 wicket production is
> > that we switch to brix 6 / wicket 6 quite fast and I need to trace
> > down all possible problems we might face.
>
> > I saw some async in wicket 1.4's DiskPageStore as well, will have a
> > look at it - thx for this hint!
>
> > Regarding the flushing:
>
> > I didnt see any problems with firefox (its our own default browser),
> > but of course there could be some problems. Can you point me to any
> > trace Matej left about this?
>
Korbinian
Jun 19, 2012, 4:03:22 PM6/19/12
to brix-cms-discuss
PS: the fix in wicket was introduced on Jan 11th, 2010 :
https://github.com/apache/wicket/tree/da1077e12e690fda9a7f47a745308a156df738bd/wicket/src/main/java/org/apache/wicket/ng/resource/AbstractResource.java
So it may be a temp bug - or simply not affecting wicket but I expect
this to be save to put into brix then as a flush of just the headers
is not a big problem;
https://github.com/apache/wicket/tree/da1077e12e690fda9a7f47a745308a156df738bd/wicket/src/main/java/org/apache/wicket/ng/resource/AbstractResource.java
So it may be a temp bug - or simply not affecting wicket but I expect
this to be save to put into brix then as a flush of just the headers
is not a big problem;
Korbinian
Jun 23, 2012, 8:51:08 AM6/23/12
to brix-cms-discuss
Follow up: keep-alive is also not currently enabled, will have to
overwork streamer for Brix 6
On 19 Jun., 22:03, Korbinian <korbinian.ba...@googlemail.com> wrote:
> PS: the fix in wicket was introduced on Jan 11th, 2010 :https://github.com/apache/wicket/tree/da1077e12e690fda9a7f47a745308a1...
overwork streamer for Brix 6
On 19 Jun., 22:03, Korbinian <korbinian.ba...@googlemail.com> wrote:
> PS: the fix in wicket was introduced on Jan 11th, 2010 :https://github.com/apache/wicket/tree/da1077e12e690fda9a7f47a745308a1...
Ingo
Jun 24, 2012, 10:29:21 AM6/24/12
to brix-cms...@googlegroups.com
Hi,
is there a public repository where I can get the current state of Brix 6?
Thanks,
Ingo
is there a public repository where I can get the current state of Brix 6?
Thanks,
Ingo
> > > >> > To post to this group, send email to brix-cms-discuss@googlegroups.com.
> > > >> > To unsubscribe from this group, send email to brix-cms-discuss+unsubscribe@googlegroups.com.
> > > >> > For more options, visit this group athttp://groups.google.com/group/brix-cms-discuss?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google Groups "brix-cms-discuss" group.
> > > > To post to this group, send email to brix-cms-discuss@googlegroups.com.
> > > > To unsubscribe from this group, send email to brix-cms-discuss+unsubscribe@googlegroups.com.
Martin Grigorov
Jun 25, 2012, 3:11:44 AM6/25/12
to brix-cms...@googlegroups.com
Hi Ingo,
There is a branch at the GitHub repo:
https://github.com/brix-cms/brix-cms/tree/wicket6
P.S. Next time don't steal threads ;-)
>> > > > >> > brix-cms...@googlegroups.com.
> https://groups.google.com/d/msg/brix-cms-discuss/-/9RY55pt_tOUJ.
>
> To post to this group, send email to brix-cms...@googlegroups.com.
There is a branch at the GitHub repo:
https://github.com/brix-cms/brix-cms/tree/wicket6
P.S. Next time don't steal threads ;-)
>> > > > >> > To unsubscribe from this group, send email to
>> > > > >> > brix-cms-discu...@googlegroups.com.
>> > > > >> > For more options, visit this group
>> > > > >> > athttp://groups.google.com/group/brix-cms-discuss?hl=en.
>> >
>> > > > > --
>> > > > > You received this message because you are subscribed to the Google
>> > > > > Groups "brix-cms-discuss" group.
>> > > > > To post to this group, send email to
>> > > > > brix-cms...@googlegroups.com.
>> > > > >> > athttp://groups.google.com/group/brix-cms-discuss?hl=en.
>> >
>> > > > > --
>> > > > > You received this message because you are subscribed to the Google
>> > > > > Groups "brix-cms-discuss" group.
>> > > > > To post to this group, send email to
>> > > > > To unsubscribe from this group, send email to
>> > > > > brix-cms-discu...@googlegroups.com.
>> > > > > For more options, visit this group
>> > > > > athttp://groups.google.com/group/brix-cms-discuss?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "brix-cms-discuss" group.
> To view this discussion on the web visit
>> > > > > athttp://groups.google.com/group/brix-cms-discuss?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "brix-cms-discuss" group.
> https://groups.google.com/d/msg/brix-cms-discuss/-/9RY55pt_tOUJ.
>
> To post to this group, send email to brix-cms...@googlegroups.com.
> To unsubscribe from this group, send email to
> brix-cms-discu...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages