good pratices to write many lines on database

[Bruno Barbosa]

Hello everybody!

I have an application that receives about 1000 POST requests per second. On each request I writte a line on my database table. On this scenario my database can shutdown during execution.

Is it a good pratice to store these data on array in cache (redis) and on each minute through celery get then and use a bulk_create to insert all data on database?

Bruno

--

[Vinicius Assef]

Or you could return a http status 400 if the transaction couldn't be
committed. So, the client should re-run the request later.

If this is not an option, you should lay upon a queue as you suggested.

Bad things always can happen. So, try your best to avoid i/o problems
if your clients cannot re-make requests in case of problems in your
side.

--
Vinicius Assef

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAHxcCH5t7FWRoXbCFvUez-hAzLk0b08KideZ_D_A6oj6a_8L1A%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Avraham Serour]

can you elaborate on this scenario? how and why can your database shutdown?

I second the suggestion about returning error and telling the client to try later, but it should be >=500 not 400. 4xx Are for client errors, 5xx are for server errors. see http://www.restapitutorial.com/httpstatuscodes.html

On Wed, Jan 4, 2017 at 10:47 PM, Vinicius Assef <vinic...@gmail.com> wrote:

Or you could return a http status 400 if the transaction couldn't be
committed. So, the client should re-run the request later.

If this is not an option, you should lay upon a queue as you suggested.

Bad things always can happen. So, try your best to avoid i/o problems
if your clients cannot re-make requests in case of problems in your
side.

--
Vinicius Assef

On 4 January 2017 at 14:39, Bruno Barbosa <bsbr...@gmail.com> wrote:
> Hello everybody!
>
> I have an application that receives about 1000 POST requests per second. On
> each request I writte a line on my database table. On this scenario my
> database can shutdown during execution.
>
> Is it a good pratice to store these data on array in cache (redis) and on
> each minute through celery get then and use a bulk_create to insert all data
> on database?
>
> Bruno
> --
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to django-users+unsubscribe@googlegroups.com.

> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAHxcCH5t7FWRoXbCFvUez-hAzLk0b08KideZ_D_A6oj6a_8L1A%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFmXjSDsYp44V8mVwkdwX-FVww%3D-NhYm4w-KnDZ4rw5SQo5owg%40mail.gmail.com.

[Vijay Khemlani]

You could queue the tasks in a broker (rabbitmq for example) and use a
fixed number of celery workers to process them and save them to the
database, assuming:

- you don't have 1000 requests per second all the time
- you don't need the data to be stored immediately

>> > email to django-users...@googlegroups.com.

>> > To post to this group, send email to django...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/django-users.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msgid/django-users/CAHxcCH5t7FWRoXbCFvUez-
>> hAzLk0b08KideZ_D_A6oj6a_8L1A%40mail.gmail.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to django-users...@googlegroups.com.

>> To post to this group, send email to django...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit https://groups.google.com/d/

>> msgid/django-users/CAFmXjSDsYp44V8mVwkdwX-FVww%3D-NhYm4w-KnDZ4rw5SQo5owg%
>> 40mail.gmail.com.

>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to django-users...@googlegroups.com.

> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/CAFWa6tJQCkmJ%2BnM4nbw2snR0TguG7DPYvKLqN2__OtdFK314Dg%40mail.gmail.com.