Blocking requests

[AnanthRK]

We are writing our first application using Play and have been enjoying
the experience so far.

The application is a REST API used to query & update several entities
in our Database (Vertica). In addition to this, we also have (long-
running) jobs that keep the tables up-to-date by updating their status
by querying a different database. When we deployed this app in
production, most requests come back within milliseconds, but there are
cases when the same requests take minutes (yes, minutes!!) to return a
response. Since I am new to Play, I would like to understand the
following:

(1) How do I track the lifecycle of a request so that the root-cause
can be identified?

(2) Will the background jobs be a reason for this behavior? If yes,
what is the right way to implement them. For now, they are scheduled
to run 3 times in an hour and directly update the models. Thus, they
are not part of the request/response flow.

(3) Since the jobs are updating the models, would a request that
involves querying the same model be blocked, thus causing this
latency?

Any pointers in tracking down this problem is much appreciated.

Thanks,
Ananth

[Morten Kjetland]

Can it be the case that your update-jobs locks the database while updating it?

-Morten

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To post to this group, send email to play-fr...@googlegroups.com.
To unsubscribe from this group, send email to play-framewor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/play-framework?hl=en.

[AnanthRK]

Yes. It is quite possible, though am not sure of it now. Does this
mean I should run my jobs differently in order to avoid this behavior?

- Ananth

[grandfatha]

Jobs are run in the background by default, so I dont think it is the
way you are running the job, but merely the job having a side-effect
that causes the request processing to halt (like waiting to resume for
a released database lock).

One option is to add logging to your application so you can "tail -f"
the logfile and see whats going on or another alternative is to create
a thread-dump in the situation of those minute-long request and see
what each thread of the JVM is doing at that point. Depending on your
production environment this could be done using command-line tools or
GUI tools such as VisualVM.

[mtsr]

Assuming you're using something like MySQL you could use a Process
Monitor. These are usually build into MySQL tools, but processes can
also be viewed using the query "SHOW PROCESSLIST" which shows you a
list of processes running on the databases, including locks and such.

[AnanthRK]

Thanks. So, if I understand right, the jobs running in the background
per-se does not affect my request/response lifecycle, but the job
could be (b)locking objects, such as the models, that are processed in
the request/response cycle. I will look into monitoring my server &
database state.

Thanks once again.

- Ananth

[Alison Winters]

You might be hitting the same problem as this guy:
http://groups.google.com/group/play-framework/browse_thread/thread/70eece92d9432fcf
I.e. you have a background job that opens a transaction and leaves it
open without ever closing it. If you have long-running jobs that only
need to do periodic database updates, those jobs should be run without
a transaction and should spawn off transactional jobs to do their
database work.

Alison