Django: creating formset is very slow

[Kottiyath Nair]

Hi all,
   My web application sends a medium size data grid (20 elements). I was using formsets for the same.
   The issue I am facing is that the formset instantiation is very very slow. I timed it and it is taking ~4-7 seconds for it to instantiate.
   Is there someway the speed can be increased?

There are no files sent. I am planning to, later.
The code:
            logging.info('Start - %s' %time.clock())
            DataFormSet = formset_factory(DataForm, extra=25)
            logging.info('Formset Class created- %s' %time.clock())
            formset = DataFormSet(request.POST, request.FILES)
            logging.info('Created new formset- %s'%time.clock())

From my logs:
2009-01-06 22:53:30,671 INFO Start - 0
2009-01-06 22:53:30,671 INFO Formset Class created- 0.000403403225829
2009-01-06 22:53:34,296 INFO Created new formset- 3.62182316468
  or later
2009-01-06 22:56:37,500 INFO Start - 186.836136716
2009-01-06 22:56:37,500 INFO Formset Class created- 186.836445135
2009-01-06 22:56:43,108 INFO Created new formset- 192.440754621


   Please note that I am running the whole thing under the django development server in my laptop itself and not a server.

[Kottiyath Nair]

I tried with 500 forms instead of 25. Now the time is becoming appaling.--- 117 seconds. Second time it hung.

2009-01-07 01:42:13,812 INFO Start - 4.46984183744e-006
2009-01-07 01:42:13,812 INFO Formset Class created- 0.000422958783868
2009-01-07 01:44:11,703 INFO Created new formset- 117.90750668
2009-01-07 01:44:17,203 INFO All forms done - 123.39991647
2009-01-07 01:44:17,217 INFO Start - 123.416734808
2009-01-07 01:44:17,217 INFO Formset Class created- 123.41704658

Regards
K

[Jeff FW]

Post the code for DataForm--I'll bet it's hitting the database a
number of times. That would be the only reason I can think of that it
would take that long. I just created a formset containing simple
forms, and it instantiated almost instantly--even with 2000 forms.

-Jeff

On Jan 6, 3:21 pm, "Kottiyath Nair" <n.kottiy...@gmail.com> wrote:
> I tried with 500 forms instead of 25. Now the time is becoming appaling.---
> 117 seconds. Second time it hung.
>
> 2009-01-07 01:42:13,812 INFO Start - 4.46984183744e-006
> 2009-01-07 01:42:13,812 INFO Formset Class created- 0.000422958783868
> 2009-01-07 01:44:11,703 INFO Created new formset- 117.90750668
> 2009-01-07 01:44:17,203 INFO All forms done - 123.39991647
> 2009-01-07 01:44:17,217 INFO Start - 123.416734808
> 2009-01-07 01:44:17,217 INFO Formset Class created- 123.41704658
>
> Regards
> K
>

[Kottiyath]

Hi,
I was able to solve it.
The issue was not with formsets, nor database.
It was slow because request.POST was getting delayed - ranging from
3.5 seconds to 3+ minutes.
I even tried having an assignment as follows:
x = request.POST
and even that was taking 3seconds to 3 minutes.
On going through the depths of wsgi code (yay, opensource), I found
the parsing is much slower in case on multipart messages - and I had
my http content-type and encoding as multipart.
Since having files was a rare thing in my code, I modified the code
to encode as basic encoding unless files was present.
Now, the same code - with 500 elements - take 0.7 seconds instead of
the earlier 117 seconds.

Open source is amazing.

Regards
K