Call for testing: streaming uploads (#2070)
Jacob Kaplan-Moss
As far as I'm concerned, #2070, adding large streaming uploads, is
done. I'd like to get some public kicking-of-the-tires before I push
the change to trunk (which won't happen before Tuesday: I'm taking a
long weekend off).
You can get the code either from my git repository
(git://djangoproject.com/django; branch "2070-streaming-uploads") or
as a patch against [7728] from
http://code.djangoproject.com/attachment/ticket/2070/2070-r7728.patch.
There's also a gitweb:
http://code.djangoproject.com/git/?p=django;a=shortlog;h=refs/heads/2070-streaming-uploads.
Documentation of the new features is can be found in
``docs/upload_handling.txt`` or
http://code.djangoproject.com/git/?p=django;a=blob_plain;f=docs/upload_handling.txt;hb=refs/heads/2070-streaming-uploads.
I need a hand on the following things:
* Running the test suite -- coverage is quite good now, but I've only
been able to test on a limited set of OSes/DBs. Looks elsewhere just
in case, very good.
* Real-world tests -- try uploading huge files. I can't really
simulate enormous files in the test suite, so someone willing to
upload a bunch of porn^H^H^H^H video would help.
* Custom upload handlers: write some, and let me know how it goes.
Also, anyone with better git-fu than me might want to try merging this
into the newforms-admin branch to see if anything crazy blows up; I
don't have any idea how this'll affect that (though I hope it won't).
Please report any problems back here; the ticket is nearly impossible
to follow now.
Thanks!
Jacob
Marty Alchin
<jacob.ka...@gmail.com> wrote:
>
> Hi folks --
>
> As far as I'm concerned, #2070, adding large streaming uploads, is
> done. I'd like to get some public kicking-of-the-tires before I push
> the change to trunk (which won't happen before Tuesday: I'm taking a
> long weekend off).
Yay!
I can't help too much with most of your needs, except possibly coming
up with a custom upload filter just to figure out how it works.
Instead, I'll work on getting the file storage patch to play nicely
with it, so that once it's merged, we're already a step ahead on
another item in the list.
I don't expect it to take much work, but I've been holding off until
2070 either merged or hit a stage like this, so I wasn't working with
too much of a moving target. I'll stay in the loop on any changes that
are made as a result of this testing, and get us in better shape once
it lands.
-Gul
Mike Malone
FYI: Our environment isn't that bizarre (Apache/Debian), but we've been running patch #2070 in production on Pownce for a couple weeks now (we actually ran a back-ported version of #2070 on 0.96 before we moved to trunk). It's been working beautifully, and has really improved performance for large file uploads. Our current cap is 250MB, but I've tested uploads that are closer to a gig in our staging environment with no problems.
Mike
Simon Willison
wrote:
Just a small thing:
"""
Like any data supplied by the user, you shouldn't
trust that the uploaded file is actually this type. You'll still need
to
validate that the file contains the content that the content-type
header
claims -- "trust but verify."
"""
That's not "trust but verify" - that's "distrust and verify". You're
not trusting the content-type header at all!
The next line mentions verifying the charset of the uploaded file as
well. It would be useful to provide an example of code that confirms
that something is valid utf-8 here:
try:
chunk.decode('utf8')
except UnicodeDecodeError:
# It's not valid utf8
A link to http://chardet.feedparser.org/ would be useful here as well.
Finally, a question: from reading the documentation it isn't clear how
you would go about implementing an Ajax file progress bar - to build
one of those, you need to know how large the browser claims the file
will be right at the start of the stream (so you can calculate the
percentage done). Does a FileUploadHandler object have access to the
content length sent by the browser?
Cheers,
Simon
Simon Willison
> Does a FileUploadHandler object have access to the
> content length sent by the browser?
field_name, file_name, content_type, content_length, charset)
Never mind then,
Simon
David Reynolds
Hopefully this will be of some use..
On 26 Jun 2008, at 8:14 pm, Jacob Kaplan-Moss wrote:
> * Running the test suite -- coverage is quite good now, but I've only
> been able to test on a limited set of OSes/DBs. Looks elsewhere just
> in case, very good.
I've just checked the branch and ran the test suite on the following
setup:
- Debian Lenny
- Python 2.5.2
- MySQLdb 1.2.2 final
- python-sqlite 2.3.2
- sqlite 3.5.9
- MySQL 5.0.51a
Running with sqlite backend all tests went through fine.
Running with mysql backend created a few errors (I'm not sure if
they're related so I've attached them in full, hope that's ok):
/var/www/django/git/django/tests/regressiontests/templates/loaders.py:
14: UserWarning: Module _mysql was already imported from /var/lib/
python-support/python2.5/_mysql.so, but /var/lib/python-support/
python2.5 is being added to sys.path
import pkg_resources
/var/www/django/git/django/django/db/models/sql/query.py:1502:
Warning: Incorrect string value: '\xE6\x9E\x97\xE5\x8E\x9F...' for
column 'headline' at row 1
cursor.execute(sql, params)
======================================================================
FAIL: Doctest: modeltests.basic.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py", line
2180, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
modeltests.basic.models.__test__.API_TESTS
File "/var/www/django/git/django/tests/modeltests/basic/
models.py", line unknown line number, in API_TESTS
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/modeltests/basic/models.py",
line ?, in modeltests.basic.models.__test__.API_TESTS
Failed example:
Article.objects.get(pk=a.id).headline
Expected:
u'\u6797\u539f \u3081\u3050\u307f'
Got:
u'?? ???'
======================================================================
FAIL: Doctest: modeltests.custom_pk.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py", line
2180, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
modeltests.custom_pk.models.__test__.API_TESTS
File "/var/www/django/git/django/tests/modeltests/custom_pk/
models.py", line unknown line number, in API_TESTS
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/modeltests/custom_pk/
models.py", line ?, in modeltests.custom_pk.models.__test__.API_TESTS
Failed example:
emp.save()
Exception raised:
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py",
line 1267, in __run
compileflags, 1) in test.globs
File "<doctest modeltests.custom_pk.models.__test__.API_TESTS
[34]>", line 1, in <module>
emp.save()
File "/var/www/django/git/django/django/db/models/base.py",
line 274, in save
self.save_base()
File "/var/www/django/git/django/django/db/models/base.py",
line 315, in save_base
if manager.filter(pk=pk_val).extra(select={'a': 1}).values
('a').order_by():
File "/var/www/django/git/django/django/db/models/query.py",
line 170, in __nonzero__
iter(self).next()
File "/var/www/django/git/django/django/db/models/query.py",
line 164, in _result_iter
self._fill_cache()
File "/var/www/django/git/django/django/db/models/query.py",
line 586, in _fill_cache
self._result_cache.append(self._iter.next())
File "/var/www/django/git/django/django/db/models/query.py",
line 612, in iterator
for row in self.query.results_iter():
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 200, in results_iter
for rows in self.execute_sql(MULTI):
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 1502, in execute_sql
cursor.execute(sql, params)
File "/var/lib/python-support/python2.5/MySQLdb/cursors.py",
line 166, in execute
self.errorhandler(self, exc, value)
File "/var/lib/python-support/python2.5/MySQLdb/
connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1267, "Illegal mix of collations
(latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for
operation '='")
======================================================================
FAIL: Doctest: modeltests.serializers.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py", line
2180, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
modeltests.serializers.models.__test__.API_TESTS
File "/var/www/django/git/django/tests/modeltests/serializers/
models.py", line unknown line number, in API_TESTS
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/modeltests/serializers/
models.py", line ?, in modeltests.serializers.models.__test__.API_TESTS
Failed example:
ac.save(); mv.save()
Exception raised:
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py",
line 1267, in __run
compileflags, 1) in test.globs
File "<doctest modeltests.serializers.models.__test__.API_TESTS
[52]>", line 1, in <module>
ac.save(); mv.save()
File "/var/www/django/git/django/django/db/models/base.py",
line 274, in save
self.save_base()
File "/var/www/django/git/django/django/db/models/base.py",
line 315, in save_base
if manager.filter(pk=pk_val).extra(select={'a': 1}).values
('a').order_by():
File "/var/www/django/git/django/django/db/models/query.py",
line 170, in __nonzero__
iter(self).next()
File "/var/www/django/git/django/django/db/models/query.py",
line 164, in _result_iter
self._fill_cache()
File "/var/www/django/git/django/django/db/models/query.py",
line 586, in _fill_cache
self._result_cache.append(self._iter.next())
File "/var/www/django/git/django/django/db/models/query.py",
line 612, in iterator
for row in self.query.results_iter():
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 200, in results_iter
for rows in self.execute_sql(MULTI):
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 1502, in execute_sql
cursor.execute(sql, params)
File "/var/lib/python-support/python2.5/MySQLdb/cursors.py",
line 166, in execute
self.errorhandler(self, exc, value)
File "/var/lib/python-support/python2.5/MySQLdb/
connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1267, "Illegal mix of collations
(latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for
operation '='")
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/modeltests/serializers/
models.py", line ?, in modeltests.serializers.models.__test__.API_TESTS
Failed example:
print serializers.serialize("json", [mv])
Exception raised:
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py",
line 1267, in __run
compileflags, 1) in test.globs
File "<doctest modeltests.serializers.models.__test__.API_TESTS
[53]>", line 1, in <module>
print serializers.serialize("json", [mv])
File "/var/www/django/git/django/django/core/serializers/
__init__.py", line 72, in serialize
s.serialize(queryset, **options)
File "/var/www/django/git/django/django/core/serializers/
base.py", line 48, in serialize
self.handle_fk_field(obj, field)
File "/var/www/django/git/django/django/core/serializers/
python.py", line 41, in handle_fk_field
related = getattr(obj, field.name)
File "/var/www/django/git/django/django/db/models/fields/
related.py", line 239, in __get__
rel_obj = self.field.rel.to._default_manager.get(**params)
File "/var/www/django/git/django/django/db/models/manager.py",
line 82, in get
return self.get_query_set().get(*args, **kwargs)
File "/var/www/django/git/django/django/db/models/query.py",
line 282, in get
num = len(clone)
File "/var/www/django/git/django/django/db/models/query.py",
line 139, in __len__
self._result_cache = list(self.iterator())
File "/var/www/django/git/django/django/db/models/query.py",
line 252, in iterator
for row in self.query.results_iter():
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 200, in results_iter
for rows in self.execute_sql(MULTI):
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 1502, in execute_sql
cursor.execute(sql, params)
File "/var/lib/python-support/python2.5/MySQLdb/cursors.py",
line 166, in execute
self.errorhandler(self, exc, value)
File "/var/lib/python-support/python2.5/MySQLdb/
connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1267, "Illegal mix of collations
(latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for
operation '='")
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/modeltests/serializers/
models.py", line ?, in modeltests.serializers.models.__test__.API_TESTS
Failed example:
list(serializers.deserialize('json', serializers.serialize
('json', [mv])))[0].object.title
Exception raised:
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py",
line 1267, in __run
compileflags, 1) in test.globs
File "<doctest modeltests.serializers.models.__test__.API_TESTS
[54]>", line 1, in <module>
list(serializers.deserialize('json', serializers.serialize
('json', [mv])))[0].object.title
File "/var/www/django/git/django/django/core/serializers/
__init__.py", line 72, in serialize
s.serialize(queryset, **options)
File "/var/www/django/git/django/django/core/serializers/
base.py", line 48, in serialize
self.handle_fk_field(obj, field)
File "/var/www/django/git/django/django/core/serializers/
python.py", line 41, in handle_fk_field
related = getattr(obj, field.name)
File "/var/www/django/git/django/django/db/models/fields/
related.py", line 239, in __get__
rel_obj = self.field.rel.to._default_manager.get(**params)
File "/var/www/django/git/django/django/db/models/manager.py",
line 82, in get
return self.get_query_set().get(*args, **kwargs)
File "/var/www/django/git/django/django/db/models/query.py",
line 282, in get
num = len(clone)
File "/var/www/django/git/django/django/db/models/query.py",
line 139, in __len__
self._result_cache = list(self.iterator())
File "/var/www/django/git/django/django/db/models/query.py",
line 252, in iterator
for row in self.query.results_iter():
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 200, in results_iter
for rows in self.execute_sql(MULTI):
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 1502, in execute_sql
cursor.execute(sql, params)
File "/var/lib/python-support/python2.5/MySQLdb/cursors.py",
line 166, in execute
self.errorhandler(self, exc, value)
File "/var/lib/python-support/python2.5/MySQLdb/
connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1267, "Illegal mix of collations
(latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for
operation '='")
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/modeltests/serializers/
models.py", line ?, in modeltests.serializers.models.__test__.API_TESTS
Failed example:
print serializers.serialize("json", [mv2])
Exception raised:
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py",
line 1267, in __run
compileflags, 1) in test.globs
File "<doctest modeltests.serializers.models.__test__.API_TESTS
[56]>", line 1, in <module>
print serializers.serialize("json", [mv2])
File "/var/www/django/git/django/django/core/serializers/
__init__.py", line 72, in serialize
s.serialize(queryset, **options)
File "/var/www/django/git/django/django/core/serializers/
base.py", line 48, in serialize
self.handle_fk_field(obj, field)
File "/var/www/django/git/django/django/core/serializers/
python.py", line 41, in handle_fk_field
related = getattr(obj, field.name)
File "/var/www/django/git/django/django/db/models/fields/
related.py", line 239, in __get__
rel_obj = self.field.rel.to._default_manager.get(**params)
File "/var/www/django/git/django/django/db/models/manager.py",
line 82, in get
return self.get_query_set().get(*args, **kwargs)
File "/var/www/django/git/django/django/db/models/query.py",
line 282, in get
num = len(clone)
File "/var/www/django/git/django/django/db/models/query.py",
line 139, in __len__
self._result_cache = list(self.iterator())
File "/var/www/django/git/django/django/db/models/query.py",
line 252, in iterator
for row in self.query.results_iter():
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 200, in results_iter
for rows in self.execute_sql(MULTI):
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 1502, in execute_sql
cursor.execute(sql, params)
File "/var/lib/python-support/python2.5/MySQLdb/cursors.py",
line 166, in execute
self.errorhandler(self, exc, value)
File "/var/lib/python-support/python2.5/MySQLdb/
connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1267, "Illegal mix of collations
(latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for
operation '='")
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/modeltests/serializers/
models.py", line ?, in modeltests.serializers.models.__test__.API_TESTS
Failed example:
print list(serializers.deserialize('json', serializers.serialize
('json', [mv2])))[0].object.id
Exception raised:
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py",
line 1267, in __run
compileflags, 1) in test.globs
File "<doctest modeltests.serializers.models.__test__.API_TESTS
[57]>", line 1, in <module>
print list(serializers.deserialize('json',
serializers.serialize('json', [mv2])))[0].object.id
File "/var/www/django/git/django/django/core/serializers/
__init__.py", line 72, in serialize
s.serialize(queryset, **options)
File "/var/www/django/git/django/django/core/serializers/
base.py", line 48, in serialize
self.handle_fk_field(obj, field)
File "/var/www/django/git/django/django/core/serializers/
python.py", line 41, in handle_fk_field
related = getattr(obj, field.name)
File "/var/www/django/git/django/django/db/models/fields/
related.py", line 239, in __get__
rel_obj = self.field.rel.to._default_manager.get(**params)
File "/var/www/django/git/django/django/db/models/manager.py",
line 82, in get
return self.get_query_set().get(*args, **kwargs)
File "/var/www/django/git/django/django/db/models/query.py",
line 282, in get
num = len(clone)
File "/var/www/django/git/django/django/db/models/query.py",
line 139, in __len__
self._result_cache = list(self.iterator())
File "/var/www/django/git/django/django/db/models/query.py",
line 252, in iterator
for row in self.query.results_iter():
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 200, in results_iter
for rows in self.execute_sql(MULTI):
File "/var/www/django/git/django/django/db/models/sql/
query.py", line 1502, in execute_sql
cursor.execute(sql, params)
File "/var/lib/python-support/python2.5/MySQLdb/cursors.py",
line 166, in execute
self.errorhandler(self, exc, value)
File "/var/lib/python-support/python2.5/MySQLdb/
connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1267, "Illegal mix of collations
(latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for
operation '='")
======================================================================
FAIL: Doctest:
regressiontests.model_inheritance_regress.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py", line
2180, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
regressiontests.model_inheritance_regress.models.__test__.API_TESTS
File "/var/www/django/git/django/tests/regressiontests/
model_inheritance_regress/models.py", line unknown line number, in
API_TESTS
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/regressiontests/
model_inheritance_regress/models.py", line ?, in
regressiontests.model_inheritance_regress.models.__test__.API_TESTS
Failed example:
[sorted(d.items()) for d in dicts]
Expected:
[[('name', u"Guido's House of Pasta"), ('serves_hot_dogs', True)]]
Got:
[[('name', u"Guido's House of Pasta"), ('serves_hot_dogs', 1)]]
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/regressiontests/
model_inheritance_regress/models.py", line ?, in
regressiontests.model_inheritance_regress.models.__test__.API_TESTS
Failed example:
[sorted(d.items()) for d in dicts]
Expected:
[[('name', u"Guido's House of Pasta"), ('serves_gnocchi', True),
('serves_hot_dogs', True)]]
Got:
[[('name', u"Guido's House of Pasta"), ('serves_gnocchi', 1),
('serves_hot_dogs', 1)]]
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/regressiontests/
model_inheritance_regress/models.py", line ?, in
regressiontests.model_inheritance_regress.models.__test__.API_TESTS
Failed example:
[sorted(d.items()) for d in dicts]
Expected:
[[('name', u"Guido's All New House of Pasta"),
('serves_hot_dogs', False)]]
Got:
[[('name', u"Guido's All New House of Pasta"),
('serves_hot_dogs', 0)]]
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/regressiontests/
model_inheritance_regress/models.py", line ?, in
regressiontests.model_inheritance_regress.models.__test__.API_TESTS
Failed example:
[sorted(d.items()) for d in dicts]
Expected:
[[('name', u"Guido's All New House of Pasta"),
('serves_gnocchi', False), ('serves_hot_dogs', False)]]
Got:
[[('name', u"Guido's All New House of Pasta"),
('serves_gnocchi', 0), ('serves_hot_dogs', 0)]]
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/regressiontests/
model_inheritance_regress/models.py", line ?, in
regressiontests.model_inheritance_regress.models.__test__.API_TESTS
Failed example:
[sorted(d.items()) for d in dicts]
Expected:
[[('name', u"Guido's All New House of Pasta"),
('serves_gnocchi', False), ('serves_hot_dogs', False)]]
Got:
[[('name', u"Guido's All New House of Pasta"),
('serves_gnocchi', 0), ('serves_hot_dogs', 0)]]
======================================================================
FAIL: Doctest: regressiontests.string_lookup.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py", line
2180, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for
regressiontests.string_lookup.models.__test__.API_TESTS
File "/var/www/django/git/django/tests/regressiontests/
string_lookup/models.py", line unknown line number, in API_TESTS
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/regressiontests/string_lookup/
models.py", line ?, in
regressiontests.string_lookup.models.__test__.API_TESTS
Failed example:
Foo.objects.get(friend__contains=u'\xe7')
Exception raised:
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py",
line 1267, in __run
compileflags, 1) in test.globs
File "<doctest
regressiontests.string_lookup.models.__test__.API_TESTS[18]>", line
1, in <module>
Foo.objects.get(friend__contains=u'\xe7')
File "/var/www/django/git/django/django/db/models/manager.py",
line 82, in get
return self.get_query_set().get(*args, **kwargs)
File "/var/www/django/git/django/django/db/models/query.py",
line 287, in get
% self.model._meta.object_name)
DoesNotExist: Foo matching query does not exist.
----------------------------------------------------------------------
File "/var/www/django/git/django/tests/regressiontests/string_lookup/
models.py", line ?, in
regressiontests.string_lookup.models.__test__.API_TESTS
Failed example:
Foo.objects.get(friend__contains='\xc3\xa7')
Exception raised:
Traceback (most recent call last):
File "/var/www/django/git/django/django/test/_doctest.py",
line 1267, in __run
compileflags, 1) in test.globs
File "<doctest
regressiontests.string_lookup.models.__test__.API_TESTS[19]>", line
1, in <module>
Foo.objects.get(friend__contains='\xc3\xa7')
File "/var/www/django/git/django/django/db/models/manager.py",
line 82, in get
return self.get_query_set().get(*args, **kwargs)
File "/var/www/django/git/django/django/db/models/query.py",
line 287, in get
% self.model._meta.object_name)
DoesNotExist: Foo matching query does not exist.
----------------------------------------------------------------------
Ran 264 tests in 852.132s
FAILED (failures=5)
--
David Reynolds
da...@reynoldsfamily.org.uk
alex....@gmail.com
also did some general testing of uploading files, which were
succesful, and I was able to observe large uploads going to my /tmp
dir.
On Jun 27, 4:41 am, David Reynolds <da...@reynoldsfamily.org.uk>
wrote:
>
> read more »
Russell Keith-Magee
<da...@reynoldsfamily.org.uk> wrote:
> Running with mysql backend created a few errors (I'm not sure if
> they're related so I've attached them in full, hope that's ok):
Hi David,
Thanks for doing this testing, but from the look of it, these errors
are all known, and unrelated to #2070 - they are due to two MySQL
specific issues:
1) The confusion between 0/1 and False/True on BooleanField
This is an issue with MySQL support that has existed for some time.
Strictly, it's only an issue with the test suite because 0 is not
literally the same as False, but it would be good to be able to ensure
that BooleanFields always return bool. I'm pretty sure there is a
ticket for it, but I'm not sure what progress has been made on a fix.
2) Errors caused when your default collation for text fields isn't UTF-8.
This is a configuration problem on your setup. There is a ticket open
to document this precondition for running the Django test suite.
Yours,
Russ Magee %-)
Jonathan Hseu
he'd get to them but that I should post them on the 2070 ticket. I
forgot to post them, so he might've missed it before it was committed
to SVN. I made a new ticket for them:
http://code.djangoproject.com/ticket/7593
A quick overview of the issues:
1. The temporary file is not deleted in many cases. This is a problem
when handling large uploads. The fix is to use tempfile.TemporaryFile
instead of uploadhandler.py's TemporaryFile (why does it even
exist?). tempfile.TemporaryFile unlinks the file immediately after
creation. uploadhandler.py's TemporaryFile also does not act
completely like a file object, which would be fixed by moving to
tempfile.TemporaryFile.
2. An extremely common use-case requires direct access to the file
object. Both upload handlers (memory & temporary file) have file
objects that UploadedFile can return, so why not put it in the API?
Thanks,
Jonathan Hseu
On Jun 26, 2:14 pm, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:
>
> As far as I'm concerned, #2070, adding large streaming uploads, is
> done. I'd like to get some public kicking-of-the-tires before I push
> the change to trunk (which won't happen before Tuesday: I'm taking a
> long weekend off).
>
> You can get the code either from my git repository
> (git://djangoproject.com/django; branch "2070-streaming-uploads") or
> There's also a gitweb:http://code.djangoproject.com/git/?p=django;a=shortlog;h=refs/heads/2....
> Documentation of the new features is can be found in
David Cramer
the 10k trac emails about it :P) but a few concerns:
- Is there a setting (e.g. settings.py) to restrict the maximum file
size in an upload?
- For test suites couldn't you while (xrange(n)) pass /dev/random or
whatever to make a larger file upload simulation?
Jeremy Dunck
> - Is there a setting (e.g. settings.py) to restrict the maximum file
> size in an upload?
FILE_UPLOAD_MAX_MEMORY_SIZE
Jacob Kaplan-Moss
> I haven't read this over in too much detail yet (I just auto ignore
> the 10k trac emails about it :P) but a few concerns:
I haven't read the rest of your email yet, but I think the answers are
"77" and "see a doctor if the rash doesn't go away."
Seriously, David, please read the at *least* the docs before asking
questions. Your first question is pretty obviously answered by a
glance at settings.txt.
> - For test suites couldn't you while (xrange(n)) pass /dev/random or
> whatever to make a larger file upload simulation?
That's basically what the test does (well, "a" * 10MB, not
/dev/random). Unfortunately the way the test suite works the file gets
all read into memory anyway (by the test suite, that is) so we can't
make it too big without making the test suite more expensive than it
already is. 10MB is a fine test that stream-to-disk works, and for
tests of large files I'm OK relying on anecdotal evidence that it
works.
Jacob
Julien Phalip
I've spotted what I think is a bug. See ticket #7651.
Basically, there is a duplicate entry in the file list (i.e.
request.FILES.getlist(file_input_name)), when using the jQuery
MultiFile plugin. It used to work just fine before #2070 was merged.
Cheers,
Julien
[1] http://www.fyneworks.com/jquery/multiple-file-upload/
On Jul 2, 9:53 am, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote: