Weird problem with fileServe

[tphyahoo]

So, I am developing this site for managing darcs repos in a web 2.0-
ish way, and I recently gave it a bit of a graphic design "facelift",
which increased the number of graphic elements I was serving from the
"static" directory using the fileServe method from HAppS.Server.

And I am having this really weird bug. Basically, the page looks
different every time I reload it, because a significant portions of
the graphic elements (.jpg/.gif files) don't load, in a non-
deterministic way.

At first I thought this was a problem with fileServe, but I now
suspect this isn't a case, because when I put a trace on the top lovel
of fileServe, it appears that fileServe itself is being called in a
non-deterministic way.

I.e., I load the page once, and fileServe has been called for

img1.gif, img2.gif, img4.git (img3.gif skipped)

I reload the page, and fileServe was now called for

img1.gif, img3.gif, img4.git (img2.gif skipped)

I have also noticed that in happstutorial.com, sometimes (maybe one in
every 200 loads) the logo doesn't display, but since this happened
rarely I just ignored it.

But now it's happening all the time with my new site.

Here's another weird detail, when I have the same static content
served via the fileServe handler from within the happstutorial app,
the page actually loads correctly (10 times in a row anyway); it's
only when I have the handler in my other app that it behaves the
nondeterministic bug in a way that is problematic on every load.

So then I thought that maybe it was a memory issue, that my other app
was somehow taxing my system resources in a way that happstutorial
wasn't. I still think this may be the case, but a problem with this
theory is that in htop, happtutorial actually appears to be consuming
*more* cpu and ram than the other app.

I am wondering if anyone else has observed this problem with fileServe
on a large quantity of static content, and has a workaround?

thomas.

[Eelco Lempsink]

On 12 dec 2008, at 15:09, tphyahoo wrote:
> At first I thought this was a problem with fileServe, but I now
> suspect this isn't a case, because when I put a trace on the top lovel
> of fileServe, it appears that fileServe itself is being called in a
> non-deterministic way.
>
> I.e., I load the page once, and fileServe has been called for
>
> img1.gif, img2.gif, img4.git (img3.gif skipped)
>
> I reload the page, and fileServe was now called for
>
> img1.gif, img3.gif, img4.git (img2.gif skipped)

A thread is spawned for each request, so that should explain the non-
deterministic behaviour. I'm curious about what happens with the
requests that fail. From your debugging I'd conclude it fails
somewhere before fileServe is reached. Do you have any error catching/
logging mechanisms in place that could provide a hint?

--
Regards,

Eelco Lempsink

[tphyahoo]

Bah, this was a horrible bug, and it's still open although I've made
significant progress in isolating it.

Indeed, the problem appears to be not with fileServe, but with a
HStringTemplate machinery handler
(which I wrote) packaged on hackage in HStringTemplateHandlers.

I have managed to create a test case for it, and integrated it into
the happstutorial code in darcs at
http://code.haskell.org/happs-tutorial. This online url (keep
refreshing) shows the buggy behavior:

http://happstutorial.com/static/Html2/index.html

There are 16 gifs total, usually at least one or two goes missing. (On
my development laptop
typically 12-14 gifs go missing, so at least partly a hardware/
environment issue?)

The bug comes down to 2 lines of code in Controller.hs. If these two
lines are commented out
and the server is restarted, Html2/index.html shows all 16 gifs as it
should.

The test case is in Controller.tutorial:

http://happstutorial.com/src/Controller.hs

" -- If the following two lines are commented out, (?) --
http:localhost:5001/static/Html2/index.html
-- shows all images correctly
tDirGroups <- liftIO $ directoryGroupsSafer $ "templates"
let ts = getTemplateGroup "." tDirGroups
"

To make debugging easier, I moved the directoryGroupSafer out of
hackaged library code and included it in the darcs distribution in
StringTemplateHelpers.hs:

http://happstutorial.com/src/StringTemplateHelpers.hs

I suspect a strict/lazy IO bug (mainly because lazy IO seems to be the
source of so many "weird" haskell bugs rather than any well-founded
reason) in the following functions, pasted below.

If anybody out there sees anything that seems that it could be a
culprit, please please share!

There is a workaround -- a couple actually.
1) I can get the template handlers in the main function and pass them
in as a pure value. That works, but it means I have to stop and
restart the server every time I want to make a change, which can take
up to a minute on my rinky-dink laptop unfortunately. (Perhaps time to
upgrade?)
2) I can simply use directoryGroupSafer (note no "s") rather than
directoryGroupsSafer.
This is ok since all the templates in happstutorial are grouped
into one top-level directory anyway.
However, in my other app (and eventually in happstutorial when I
get around to it) I found it a lot
easier to work with template groups when they were broken into
subdirectories, eg for header,
menu, various subcompartments with common functionality. So, I
really like directoryGroupsSafer
(with the s) and hope to keep using it. Unfortunately, at the
moment it appears that it is borked.

Well, that's about all I can think of to say on the subject. The lines
I suspect / hope have the bug are pasted below, but of course it might
be somewhere else. From StringTemplateHelpers, in darcs and linked
above:





directoryGroupsSafer d = bracketCD d $ do
subDirs <- findDirectories $ "."
-- attempt to make strict, but doesn't make
any difference
--putStrLn . show . last $ show
subDirs
return . M.fromList =<< mapM f
subDirs
where
f d = do g <- directoryGroupSafer d
return (d,g)
-- this seems suspect. Perhaps try using the find module on
hackage?
-- | wrapper over find \/path\/to\/top\/dir -type d
findDirectories :: FilePath -> IO [FilePath]
findDirectories d = runStrings $ render1 [("d",d)] "find $d$ -type
d"
where
runStrings :: String -> IO [String]
runStrings = ( return . lines =<< ) . run


-- | calculate the STGroup for a given directory, filtering out files
that are probably errors (eg emacs backups)
--directoryGroupSafer :: (Stringable a) => FilePath -> IO (STGroup a)
directoryGroupSafer :: FilePath -> IO (String -> StFirst
(StringTemplate String) )
directoryGroupSafer path = do
files <- mapM checkTmplName =<< return . filter isTemplateFile =<<
getDirectoryContents path
contents <- mapM readFile $ map (path </>) files
-- does this make it strict? what's the right way to do it?
-- mm, the buggy behavior persists, with or without putStrLn .
show . last
--putStrLn . show . last . show $ contents
let sts = map newSTMP contents
tnames = map dropExtension files
return $ groupStringTemplates $ zip tnames sts
where
checkTmplName t = if ( badTmplVarName . takeBaseName ) t
then fail $ "directoryGroupSafer, bad
template name: " ++ t
else return t
isTemplateFile f = ( (".st" ==) . takeExtension $ f )
&& (not . or . map (=='#') $ f ) {-filename doesn't
contain naughty emacs backup character-}

>  PGP.sig
> < 1KViewDownload

[Sterling Clover]

There are a number of issues with this code, but I don't know if it
is to blame for the problem you're seeing. First, if you want to
specify an inheritance hierarchy between these directories, that
needs to be done explicitly and programmatically. I realize this is
my fault for underdocumenting HStringTemplate, but it seems like the
feature you want (which maybe should actually rolled into the main
package if we can specify cleanly how exactly it should work) can be
accomplished through use of addSubGroup, and addSuperGroup, rather
than a new type. Your use of the find command can be replaced with
calls to System.Directory (see doesDirectoryExist, used in
combination with getDirectoryContents)

Next, your comments are right about potential issues -- you're not
forcing readFile to be strict enough, for one. As it is, it seems
like each file will be read once, but lazily, which I'm not sure is
what you want.

Finally, I don't know how any of these problems would relate to
missing gifs however, and I suspect the problem must be elsewhere.
Since I assume that your images are not being served through
HStringTemplate, there's no direct reason that it should affect your
fileServe stuff unless there is some sort of (ick) rts bug or bug in
a library that associates with the rts.

Removing the dependency on HSH may help things (or it may not).

Good luck,
Sterl.

[tphyahoo]

Indeed using a strict readFile fixes an even more serious problem,
though I still can't have multiple gifs loading if I leave the
template IO in the controller loop:

http://groups.google.com/group/HAppS/browse_thread/thread/cff2b9098c2b7a14/2b64b94525cd28a4#2b64b94525cd28a4

t.

[tphyahoo]

I am looking into another StringTemplate solution I should have
thought of sooner: just use unsafeVolatileDirectory group during
development, and switch to something else for real deployment --
something that won't do IO in the controller loop every time.

thomas.

On Dec 15, 4:14 pm, tphyahoo <thomashartm...@googlemail.com> wrote:
> Indeed using a strict readFile fixes an even more serious problem,
> though I still can't have multiple gifs loading if I leave the
> template IO in the controller loop:
>

> http://groups.google.com/group/HAppS/browse_thread/thread/cff2b9098c2...

[tphyahoo]

Hi, I actually don't think I need an inheritance relation in this
case.

For now, I just want the convenience of being able to specify
additional ST groupings by creating a directory and filling it with
templates, without having to bother to stop/start the server, which on
my laptop is a time suck.

I use find because it will recursively search for directories, which
getDirectoryContents won't (I think).

Might be better off using native haskell find module on hackage
though.

Perhaps there could be two versions of something like what I did in
the StringTemplate distro: one that recursively adds subgroups based
on directory hierarchy, and one that just creates a bunch of
disconnected subgroups like I have, and lets the program logic handle
how they are to be put together.

thomas.

On Dec 13, 6:10 pm, Sterling Clover <s.clo...@gmail.com> wrote:

[tphyahoo]

So, I finally figured out what the problem was, and did a version bump
of happs-tutorial and HStringTemplateHelpers to propagate the fix.

It's two problems actually.

First of all, the readFile for HStringTemplate shoule definitely be
strict (System.IO.Strict.readFile).

With lazy readFile, I can crash my happs instance via apache bench:

ab -c10 -n1000 http://localhost:5001/

With strict readFile, no crash.

The gifs still load non-deterministically though.

Part two of the fix is to reorder main handlers so that the static
content handler (images, css, etc) comes before all others. See snip
below.

I'm not sure if this qualifies as a HAppS bug, but my gut feeling is
yes. At least, this behavior concerning static content serving when
there's IO in the controller loop should be documented somewhere
prominent.
Maybe in the haddock for fileServe?

Thomas.

staticfiles = [ staticserve "static"
, staticserve "userdata"
, browsedir "projectroot" "."
, browsedirHS "templates" "templates"
, browsedirHS "src" "src"
]
where staticserve d = dir d [ fileServe [] d ]

-- main controller
controller :: STDirGroups String -> Bool -> Bool -> [ServerPartT IO
Response]
controller tDirGroups dynamicTemplateReload allowStressTests = map
cookieFixer $
-- staticfiles handler *has* to go first, or some content (eg
images) will fail to load nondeterministically,
-- eg http://localhost:5001/static/Html2/index.html (this loads ok
when staticfiles handler goes first,
-- but has the problem when staticfiles handler goes after
tutorial handler)
-- Also interesting: the order doesn't matter when
dynamicTemplateReload is false
-- This still feels to me like a bug: it was quite a headache to
diagnose, and why should
-- the order of the static content handler matter anyway?
-- At the very least, fileServer should have a highly visible
comment warning about this problem.
staticfiles
++ ( tutorial tDirGroups dynamicTemplateReload allowStressTests )
++ simpleHandlers
++ [ myFavoriteAnimal ]
++ [ msgToSp "Quoth this server... 404." ]

[Lemmih]

On Thu, Dec 18, 2008 at 3:08 PM, tphyahoo <thomash...@googlemail.com> wrote:
>
> So, I finally figured out what the problem was, and did a version bump
> of happs-tutorial and HStringTemplateHelpers to propagate the fix.
>
> It's two problems actually.
>
> First of all, the readFile for HStringTemplate shoule definitely be
> strict (System.IO.Strict.readFile).
>
> With lazy readFile, I can crash my happs instance via apache bench:
>
> ab -c10 -n1000 http://localhost:5001/
>
> With strict readFile, no crash.
>
> The gifs still load non-deterministically though.
>
> Part two of the fix is to reorder main handlers so that the static
> content handler (images, css, etc) comes before all others. See snip
> below.
>
> I'm not sure if this qualifies as a HAppS bug, but my gut feeling is
> yes. At least, this behavior concerning static content serving when
> there's IO in the controller loop should be documented somewhere
> prominent.
> Maybe in the haddock for fileServe?

I still don't understand what the problem is. Why do you have to put
the static content handlers before the others?

--
Cheers,
Lemmih

[tphyahoo]

I don't understand it either.

But if I don't, the static content doesn't load right.

This behavior can be replicated by using the head version of happs-
tutorial and moving the static content handler out of first position,
as stated in the comment.

On Dec 18, 4:45 pm, Lemmih <lem...@gmail.com> wrote:

> On Thu, Dec 18, 2008 at 3:08 PM, tphyahoo <thomashartm...@googlemail.com> wrote:
>
> > So, I finally figured out what the problem was, and did a version bump
> > of happs-tutorial and HStringTemplateHelpers to propagate the fix.
>
> > It's two problems actually.
>
> > First of all, the readFile for HStringTemplate shoule definitely be
> > strict (System.IO.Strict.readFile).
>
> > With lazy readFile, I can crash my happs instance via apache bench:
>

> > ab -c10 -n1000http://localhost:5001/

[tphyahoo]

In order to replicate the bug you have to run happs-tutorial in
dynamic template reload mode, eg

$ happs-tutorial 5001 True False