Loading data from Django into RedShift - ORM vs SQL?

[Victor Hooi]

Hi,

We have a Django eCommerce site, from where we want to load transactional data into Amazon RedShift (i.e. basically ETL).

This would probably be a batch load run either once a day, or at intervals throughout the day.

(I'm also curious about whether it's possible to stream the data in, but I don't think that will work well with RedShift).

My question is regarding how we should be extracting the data - we can either hook directly into the Django ORM, or just use SQL by hand.

What are people's experiences with either approach? Any thoughts on pros/cons of either?

Cheers,

Victor

[Avraham Serour]

The whole idea of having an ORM is not having ot deal with SQL directly unless necessary. I would try to do it using the ORM first but there's not general rule, each case should be analised individually

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/c020f34a-37c0-4259-9251-dd5ccad00968%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Arnold Krille]

Am Thu, 17 Oct 2013 13:32:44 +0300
schrieb Avraham Serour <tov...@gmail.com>:

> The whole idea of having an ORM is not having ot deal with SQL
> directly unless necessary. I would try to do it using the ORM first
> but there's not general rule, each case should be analised
> individually

Actually you shouldn't decide each case individually but use the ORM by
default! The ORM is tested for a lot of cases whereas your SQL is only
working in your own case. And there are a lot of problems you might
think are better to solve in raw SQL, but actually after thinking
about it using the ORM will give you simpler code, simpler algorithms,
more optimization and less runtime... At least its our experience that
on 2/3 of the places where we 'needed' to do raw SQL replacing that by
using the ORM actually improved everything.

Have fun,

Arnold

[damond...@gmail.com]

Hi,

Does anyone here have experience in using the django-redshift ORM available in: https://github.com/binarydud/django-redshift

in a django web app? If so could you share it with us?

Is it robust? Scalably efficient?

Many thanks.