Playing with fixtures and django serializers -> Errors...
Jens Diemer
I'm playing with django.core.serializers... so far with few success :(
With JSON i get the Error: "ContentType matching query does not exist."
If i use the "python" format, i get from serializers.serialize() a list.
How should I to store these list in a file? With pickle, repr() or
unicode()???
With pickle, i got the Error: "can't pickle array objects"
if i use "python" and store the result with repr() or unicode() in a
file... I got an error, if i load the file and deserialize it:
------------------------------------------------------------------------
File ".\django\core\serializers\python.py", line 59, in Deserializer
Model = _get_model(d["model"])
TypeError: string indices must be integers
------------------------------------------------------------------------
If i use the "xml" format, i get an Error if i deserialize the
serialized string:
------------------------------------------------------------------------
File "serializers_test.py", line 77, in test_xml
for object in objects2:
File ".\django\core\serializers\xml_serializer.py", line 115, in next
return self._handle_object(node)
File ".\django\core\serializers\xml_serializer.py", line 155, in
_handle_object
value =
field.to_python(getInnerText(field_node).strip().encode(self.encoding))
File ".\django\db\models\fields\__init__.py", line 525, in to_python
raise validators.ValidationError, gettext('Enter a valid date/time
in YYYY-MM-DD HH:MM format.')
django.core.validators.ValidationError: ['Enter a valid date/time in
YYYY-MM-DD HH:MM format.']
------------------------------------------------------------------------
I examined this. The XML file is correct. In the DB (the source) i have
datetime fields with a NULL value.
XML file cutout:
------------------------------------------------------------------------
<object pk="1" model="PyLucid.group">
<field type="IntegerField" name="pluginID">0</field>
<field type="CharField" name="name">managePages</field>
<field type="CharField" name="section">core</field>
<field type="CharField" name="description">blabla</field>
<field type="DateTimeField" name="lastupdatetime"></field>
<field type="IntegerField" name="lastupdateby">None</field>
<field type="DateTimeField" name="createtime"></field>
</object>
------------------------------------------------------------------------
The Traceback is interesting (cutout):
.\django\core\serializers\xml_serializer.py in _handle_object line 155:
------------------------------------------------------------------------
Model: <class 'PyLucid.models.Group'>
data: {'description': 'This group is able to add/edit/delete pages.',
'id': u'1', 'name': 'managePages', 'pluginID': '0', 'section': 'core'}
field: <django.db.models.fields.DateTimeField object at 0x01477570>
field_name: u'lastupdatetime'
field_node: <DOM Element: field at 0x16568a0>
m2m_data: {}
node: <DOM Element: object at 0x1659990>
pk: u'1'
self:<django.core.serializers.xml_serializer.Deserializer object at
0x016A7190>
value: 'blabla'
------------------------------------------------------------------------
In 'data' i missing the keys 'lastupdatetime' and 'createtime'.
I find field_name == 'lastupdatetime', but value == 'blabla' also
amusing. Because you can see from above: 'blabla' is the value for the
key 'description' and not from 'lastupdatetime'
--
Mfg.
Jens Diemer
----
CMS in pure Python CGI: http://www.pylucid.org
Russell Keith-Magee
>
>
> I'm playing with django.core.serializers... so far with few success :(
Hi Jens,
I've just finished writing a new set of tests for the serializers,
which included fixing a number of bugs. These fixes have been checked
in as of [4719]. Can you retry your tests and see if you still have
problems?
Many Thanks,
Russ Magee %-)
Jens Diemer
> I've just finished writing a new set of tests for the serializers,
> which included fixing a number of bugs. These fixes have been checked
> in as of [4719]. Can you retry your tests and see if you still have
> problems?
I can't trying this. Because now i have a error before the serializers
starts working: """ContentType matching query does not exist."""
------------------------------------------------------------------------
110. objects = []
111. for app in app_list:
112. for model in get_models(app):
113. model_objects = model.objects.all()
114. response.write(repr(model_objects))
115. objects.extend(model_objects)
------------------------------------------------------------------------
The Exception Location is in line 114.
I used plain "django-admin dumpdata" with the same error.
I think i have a error in my DB data. But i can't isolate it with there
error messages :(
Jens Diemer
> I can't trying this. Because now i have a error before the serializers
> starts working: """ContentType matching query does not exist."""
On a other platform it works!
I don't known why...
But if i import the dump i got some errors:
Error: columns app_label, model are not unique
and some Errors like: 'ascii' codec can't decode byte 0xc3 in position
4: ordinal not in range(128)
Russell Keith-Magee
> > I can't trying this. Because now i have a error before the serializers
> > starts working: """ContentType matching query does not exist."""
>
> On a other platform it works!
> I don't known why...
>
> But if i import the dump i got some errors:
> Error: columns app_label, model are not unique
>
> and some Errors like: 'ascii' codec can't decode byte 0xc3 in position
> 4: ordinal not in range(128)
You are obviously having problems, and you want help, but the only
assistance you are giving me is parts of some of the error messages
you are seeing, evidently across two different platforms.
Help me help you. In order to solve your problem, I need to know
everything you are doing - not just bits of the problem. This means
you need to provide your model, and the smallest set of sample data
that you can produce that demonstrates the problem. Tell me which SVN
checkout you are using, which database backend, what operating system
- these are the details that will help me solve your problem.
Yours,
Russ Magee %-)
Jens Diemer
> Help me help you. In order to solve your problem, I need to know
> everything you are doing - not just bits of the problem.
Yes you are right, sorry ;)
Here some information: I'm rewrite PyLucid CMS: http://www.pylucid.org
The emphasis is thereby: Using PyLucid in a shared Webhosting
environment. So I implement a complete Web-Based installation.
I write a "init DB data" routine:
http://pylucid.net/trac/browser/branches/0.8%28django%29/PyLucid/install/install.py?rev=925#L82
It used a fixture file, witch i have made with this:
http://pylucid.net/trac/browser/branches/0.8%28django%29/PyLucid/install/low_level_admin.py?rev=925#L48
The source fixture file is a dump from a MySQL DB.
The current fixture file, can you see here:
http://pylucid.net/trac/browser/branches/0.8%28django%29/PyLucid/fixtures/initial_data.xml?rev=925&format=raw
I have some open questions: How am I to store the data?
I tried:
- file(fixture_filename, "w")
- file(fixture_filename, "wb")
- codecs.open(fixture_filename, "w", "utf-8")
(reading accordingly)
All the same which I used, I get unicode errors.
Russell Keith-Magee
>
> Here some information: I'm rewrite PyLucid CMS: http://www.pylucid.org
> The emphasis is thereby: Using PyLucid in a shared Webhosting
> environment. So I implement a complete Web-Based installation.
>
> I write a "init DB data" routine:
First off, for clarification - you're not using the Django native
fixtures. The Cookbook code your code is based off is not the same as
the Django fixtures that are documented as part of the test framework.
> The source fixture file is a dump from a MySQL DB.
As a warning - using MySQL will cause you some problems with the
serializers. MySQL doesn't support forward references if you use the
InnoDB backend, and it doesn't support row-referential integrity or
transactions if you use the MyISAM backend.
> The current fixture file, can you see here:
> http://pylucid.net/trac/browser/branches/0.8%28django%29/PyLucid/fixtures/initial_data.xml?rev=925&format=raw
Ok; the error you reported previously (app_label, model not unique) is
being caused when deserializing the contenttype model in the
contenttypes application. This model requires that the fields
app_label and model are a unique pair. The error will be generated
either because:
1) your data file contains a duplicate entry, or
2) you aren't flushing the existing data before inserting the new
data, so when you deserialize, you are duplicating data in the
database.
> I have some open questions: How am I to store the data?
> I tried:
> - file(fixture_filename, "w")
> - file(fixture_filename, "wb")
> - codecs.open(fixture_filename, "w", "utf-8")
> (reading accordingly)
You will need to read the file in the same mode as which it was
written, but other than that, the answer depends entirely on what you
are saving. In your case, you have unicode characters in your data, so
using unicode files would be appropriate.
Yours,
Russ Magee %-)
Jens Diemer
Generally: I have possibly wrong data in the data base. But i think
django does not help, to find these wrong data.
I made "django-admin dumpdata" and i got this error:
"""
Unable to serialize database: ContentType matching query does not exist.
"""
I can set "--verbosity=2", but i get no more information :(
I examined and make a small test app... conclusion: It does not have to
do anything with the serialize process!
The error comes if I access the model object:
-------------------------------------------------------------------
from django.db.models import get_apps, get_models
for app in get_apps():
for model in get_models(app):
model_objects = model.objects.all()
print "test:", model_objects
-------------------------------------------------------------------
tree print lines appears. The the error:
-------------------------------------------------------------------
Traceback (most recent call last):
File "W:\SVN\SVN PyLucid\branches\0.8(django)\test2.py", line 14, in ?
print "test:", model_objects
File "./django/db/models/query.py", line 102, in __repr__
File "./django/db/models/base.py", line 81, in __repr__
File "./django/contrib/auth/models.py", line 48, in __str__
File "./django/db/models/fields/related.py", line 171, in __get__
File "./django/db/models/manager.py", line 73, in get
File "./django/db/models/query.py", line 250, in get
django.db.models.base.DoesNotExist: ContentType matching query does not
exist.
-------------------------------------------------------------------
What am I to do, now?
Russell Keith-Magee schrieb:
> Ok; the error you reported previously (app_label, model not unique) is
> being caused when deserializing the contenttype model in the
> contenttypes application. This model requires that the fields
> app_label and model are a unique pair. The error will be generated
> either because:
>
> 1) your data file contains a duplicate entry, or
IMHO, that cannot happen actually. I create the fixture file with django
and not write it by hand ;)
> 2) you aren't flushing the existing data before inserting the new
> data, so when you deserialize, you are duplicating data in the
> database.
I make only this:
-------------------------------------------------------------------
objects = serializers.deserialize(format, fixture)
for object in objects:
object.save()
-------------------------------------------------------------------
Do I have to delete first the tables? But that cannot be also. I deleted
the SQLite file. Thus no data are present.
the complete routines are here:
http://pylucid.net/trac/browser/branches/0.8%28django%29/PyLucid/install/install.py
Russell Keith-Magee
>
> Generally: I have possibly wrong data in the data base. But i think
> django does not help, to find these wrong data.
> I made "django-admin dumpdata" and i got this error:
> """
> Unable to serialize database: ContentType matching query does not exist.
> """
> I can set "--verbosity=2", but i get no more information :(
Well... I don't think you need much more information. I'm going to go
out on a limb here, and say that a Content Type definintion is
missing.
This suggests that somewhere along the line, your database got out of sync.
Start from the beginning with a completely clean database, run syncdb,
then serialize - does the problem go away? Then, reintroduce data
manually and serialize. Repeat until the problem re-emerges. I suspect
that if you start from a clean setup, the problem will go away.
Yours,
Russ Magee %-)
jj
not unique").
I've found a work-around:
- delete the database
- recreate the db from your model (using syncdb)
- edit the XML (from dumpdata), removing all but your data
(i.e. keep instances only, since your model is already in the db -
recreated by syncdb. Also, removing permissions, content-type
definitions, log.entries, etc. I.e. really just your own data)
- if you've edited with, for example, XML Notepad, change the empty
date fields from:
>
<None>
</None>
</field>
back to:
><None></None></field> (i.e. single line no spaces)
since django's current XML parsing seems rather untolerant (spaces and
newlines should be allowed)
- reload your data (using loaddata)
- it should be back now
- if not, try splitting your XML into chunks and load each chunk
individually
Couple of suggestions:
- django's XML parser should really allow spaces and newlines
- it should be possible dump only your own data
- verbosity does not help for loaddata/dumpdata (no extra debug
information)
- dumpdata should definitely insert newlines and tabs for human
readibility
- the current XML format should probably be made more generic to allow
for greater interop.
JJ.
On Mar 17, 12:57 am, "Russell Keith-Magee" <freakboy3...@gmail.com>
wrote:
Jens Diemer
> I've run into a similar problem (error: "columns app_label, model are
> not unique").
>
> I've found a work-around:
A easier way is to use db_dump.py ;)
Look at:
http://groups.google.com/group/django-users/browse_thread/thread/342efea6e3fc693d/a6807b7770b6b165
--
Mfg.
Jens Diemer
----
A django powered CMS: http://www.pylucid.org
Russell Keith-Magee
> Couple of suggestions:
> - django's XML parser should really allow spaces and newlines
It does accept whitespace, in certain places. You should also remember
that depending on the schema, whitespace can be significant in XML
documents.
> - it should be possible dump only your own data
This is already possible.
By default, dumpdata will dump _every_ table from _every_ application
- but if you specifically name applications, the dump will be
restricted to just those applications.
`manage.py dumpdata myapp` will only dump data from the tables from myapp.
The reason that the other tables are included is because contenttypes,
authentication, sessions, sites and admin are all individual
applications that must be explicitly loaded into your project.
> - verbosity does not help for loaddata/dumpdata (no extra debug
> information)
What extra debug information are you looking for? IMHO, loaddata
--verbosity=3 is quite verbose.
> - dumpdata should definitely insert newlines and tabs for human
> readibility
It does - see ./manage.py --format=xml --indent=4 dumpdata
> - the current XML format should probably be made more generic to allow
> for greater interop.
Sure. Do you have a specific suggestion?
Yours,
Russ Magee %-)
akaihola
written a tool (dbpickle.py) for dumping and loading Django databases.
I've used it both for moving data from one database engine to another
and for migrating schema changes to production databases.
I wonder if it would be possible to use dumpdata and loaddata for
migration instead.
The migration approach I took was to allow for plugins to hook on
dbpickle.py's object retrieval and saving. This way I can add/remove/
rename fields of objects on the fly when loading a dumped database.
It's also possible to populate new fields with default values or even
values computed based on the object's other properties.
I include a database migration plugin with my projects for each schema
change and use them with dbpickle.py.
For the source code, see:
http://trac.ambitone.com/ambidjangolib/browser/trunk/dbpickle/dbpickle.py
Any thoughts about the feasibility of this with dumpdata/loaddata? It
would be great to have this kind of functionality in Django and have
it done The Right Way.
Russell Keith-Magee
>
> I wonder if it would be possible to use dumpdata and loaddata for
> migration instead.
Broadly speaking, it sounds viable. My only reservation would be that
for large databases, dumping the entire contents to a text file isn't
really a good migration solution (think Amazon.com dumping their
database to a text file for migration purposes)
However, for small databases, or during initial project development,
this could be a good approach.
> Any thoughts about the feasibility of this with dumpdata/loaddata? It
> would be great to have this kind of functionality in Django and have
> it done The Right Way.
At present, the deserializer doesn't have any hooks like the ones you
describe. Rather than add hooks to the deserializer, I would be
inclined to write the conversion tool as an external tool that
operates on serialized data files. The evolution process would look
something like this:
1- Dump database to data.old
2- Write a conversion script
3- Run the conversion script over data.old to produce data.new
4- Check that the converted data file is OK
5- Drop, recreate and sync the database
6- Install data.new
Long term, we (i.e., the Django project as a whole) needs to address
Schema Evolution properly (i.e., at a database ALTER TABLE level).
However, an 'evolution lite' solution like this one would be a
valuable resource in the interim.
Yours,
Russ Magee %-)