Loading timezone naive data into your test database with USE_TZ = True

[Robert Rollins]

I have a legacy database from which my Django application must migrate data into a Django database. The relevant dates fields are actually TIMESTAMP columns in the database, but something (perhaps Django, or python's MySQL driver?) loads these columns as timezone naive datetime objects, rather than integers. So I wrote my migration code under the assumption that the dates coming out of the legacy database are timezone naive.

Unfortunately, now that I'm trying to write tests for this migrator, I can't find any way to load timezone naive datetimes into my test legacy database. I can't use integer timestamps, because the DateTimeField doesn't accept that kind of input (I get a JSON serialization error when I try), so I'm using datetime strings like this: "2014-08-01T00:00:00" in my fixture. But regardless of whether or not I include a UTC offset in the string, the datetime objects that come out of the database during my tests are somehow timezone aware. This causes my code to crash because it calls make_aware(), which throws ValueError('Not naive datetime (tzinfo is already set)'). 

It seems like having USE_TZ = True is forcibly making my fixture dates timezone aware, which I don't want. But USE_TZ will be True during the actual migration, so I can't just turn it off during the tests. So how can I load timezone naive dates into my test database?

[Wim Feijen]

Hi Robert, 

Timezones confuse me, maybe Aymeric can answer this one if he has time?

Off topic, your question might be a better fit for the django-users mailing list, but perhaps you posted to django-developers intentionally, because you are thinking of a bug report?

Wim

[Carl Meyer]

Hi Robert,

I think your analysis is correct; USE_TZ = True does enforce that only aware datetimes are saved to the database, by design.

I'm not immediately thinking of a better approach for your situation than to load your test data into the source database via raw SQL inserts rather than via Django's fixture mechanism.

[Aymeric Augustin]

2014-09-18 23:29 GMT+02:00 Wim Feijen <w...@go2people.nl>:

Timezones confuse me, maybe Aymeric can answer this one if he has time?

I've bookmarked this thread to answer at some point but I have some work-related matters to deal with first.

--
Aymeric.

[Aymeric Augustin]

Hi Robert,

On 17 sept. 2014, at 01:54, Robert Rollins <coredu...@gmail.com> wrote:

> I have a legacy database from which my Django application must migrate data into a Django database. The relevant dates fields are actually TIMESTAMP columns in the database, but something (perhaps Django, or python's MySQL driver?) loads these columns as timezone naive datetime objects, rather than integers. So I wrote my migration code under the assumption that the dates coming out of the legacy database are timezone naive.

Unfortunately, you’re running into a known and documented limitation of time zone support in Django:
https://docs.djangoproject.com/en/dev/ref/databases/#timestamp-columns

> It seems like having USE_TZ = True is forcibly making my fixture dates timezone aware, which I don't want. But USE_TZ will be True during the actual migration, so I can't just turn it off during the tests. So how can I load timezone naive dates into my test database?

Would it be possible to:
- use USE_TZ = False during the migration process,
- convert datetimes to the proper timezone while migrating,
- then switch back to USE_TZ = True?

Based on my understanding of your situation, I believe this is the easiest path if you want to use the ORM.

Alternatives include:
- pull data with raw SQL queries — cursor.execute(…) — and insert it into the main database through Django,
- export data to a CSV file and re-import that.

I hope this helps,

--
Aymeric.