Encoding error

cvk

unread,

Feb 8, 2010, 5:07:57 AM2/8/10

to Panda

I'm constantly getting this weird error message when i try to
transcode large files (> 1 GB) on a S3 bucket. The bucket location is
in the US.
Any ideas?

Thanks in advance
Chris

~ Fetch from S3
~ Fetch from S3
~ Unable to transcode file 2af39020-f52c-012c-46b8-1231390a9532:
REXML::ParseException - #<REXML::ParseException: malformed XML:
missing tag
start
Line:
Position:
Last 80 unconsumed characters:
<Net::ReadAdapter:0xb74b32fc>>
/usr/local/lib/ruby/1.8/rexml/parsers/baseparser.rb:346:in `pull'
/usr/local/lib/ruby/1.8/rexml/parsers/treeparser.rb:21:in `parse'
/usr/local/lib/ruby/1.8/rexml/document.rb:204:in `build'
/usr/local/lib/ruby/1.8/rexml/document.rb:42:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:
952:in `new'
/usr/local/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:
952:in `parse'
/usr/local/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:
164:in `xml_in'
/usr/local/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:
203:in `xml_in'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/parsing.rb:
64:in `parse'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/parsing.rb:
55:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/response.rb:
55:in `new'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/response.rb:
55:in `unmemoized_parsed_1265054339'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/
extensions.rb:144:in `send'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/
extensions.rb:144:in `parsed'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/
extensions.rb:116:in `memoize'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/
extensions.rb:143:in `parsed'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/response.rb:
45:in `error?'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/base.rb:
72:in `request'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/base.rb:
83:in `get'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/object.rb:
134:in `value'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/object.rb:
138:in `stream'
/var/local/www/panda/lib/s3_store.rb:44:in `get'
/var/local/www/panda/lib/s3_store.rb:42:in `open'
/var/local/www/panda/lib/s3_store.rb:42:in `get'
/var/local/www/panda/lib/core_extensions/kernel.rb:26:in `retryable'
/var/local/www/panda/lib/s3_store.rb:41:in `get'
/var/local/www/panda/app/models/video.rb:184:in `fetch_from_store'
/var/local/www/panda/app/models/video.rb:621:in `encode'
/usr/local/lib/ruby/gems/1.8/gems/merb-core-0.9.7/lib/merb-core/rack/
adapter/runner.rb:30
/usr/local/lib/ruby/gems/1.8/gems/merb-core-0.9.7/lib/merb-core/rack/
adapter/runner.rb:23:in `loop'
/usr/local/lib/ruby/gems/1.8/gems/merb-core-0.9.7/lib/merb-core/rack/
adapter/runner.rb:23
...
malformed XML: missing tag start
Line:
Position:
Last 80 unconsumed characters:
<Net::ReadAdapter:0xb74b32fc>
Line:
Position:
Last 80 unconsumed characters:
<Net::ReadAdapter:0xb74b32fc>
~ Error email sent to …@webrunners.de
~ Error encoding 2af39020-f52c-012c-46b8-1231390a9532

#<REXML::ParseException: malformed XML: missing tag start
Line:
Position:
Last 80 unconsumed characters:
<Net::ReadAdapter:0xb74b32fc>>
/usr/local/lib/ruby/1.8/rexml/parsers/baseparser.rb:346:in `pull'
/usr/local/lib/ruby/1.8/rexml/parsers/treeparser.rb:21:in `parse'
/usr/local/lib/ruby/1.8/rexml/document.rb:204:in `build'
/usr/local/lib/ruby/1.8/rexml/document.rb:42:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:
952:in `new'
/usr/local/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:
952:in `parse'
/usr/local/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:
164:in `xml_in'
/usr/local/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:
203:in `xml_in'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/parsing.rb:
64:in `parse'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/parsing.rb:
55:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/response.rb:
55:in `new'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/response.rb:
55:in `unmemoized_parsed_1265054339'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/
extensions.rb:144:in `send'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/
extensions.rb:144:in `parsed'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/
extensions.rb:116:in `memoize'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/
extensions.rb:143:in `parsed'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/response.rb:
45:in `error?'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/base.rb:
72:in `request'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/base.rb:
83:in `get'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/object.rb:
134:in `value'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/lib/aws/s3/object.rb:
138:in `stream'
/var/local/www/panda/lib/s3_store.rb:44:in `get'
/var/local/www/panda/lib/s3_store.rb:42:in `open'
/var/local/www/panda/lib/s3_store.rb:42:in `get'
/var/local/www/panda/lib/core_extensions/kernel.rb:26:in `retryable'
/var/local/www/panda/lib/s3_store.rb:41:in `get'
/var/local/www/panda/app/models/video.rb:184:in `fetch_from_store'
/var/local/www/panda/app/models/video.rb:621:in `encode'
/usr/local/lib/ruby/gems/1.8/gems/merb-core-0.9.7/lib/merb-core/rack/
adapter/runner.rb:30
/usr/local/lib/ruby/gems/1.8/gems/merb-core-0.9.7/lib/merb-core/rack/
adapter/runner.rb:23:in `loop'
/usr/local/lib/ruby/gems/1.8/gems/merb-core-0.9.7/lib/merb-core/rack/
adapter/runner.rb:23
...
malformed XML: missing tag start
Line:
Position:
Last 80 unconsumed characters:
<Net::ReadAdapter:0xb74b32fc>
Line:
Position:
Last 80 unconsumed characters:
<Net::ReadAdapter:0xb74b32fc>

PARENT ATTRS

============================================================
---
status: original
video_codec: mpeg4
updated_at: "2010-02-06T08:18:15+00:00"
container: avi
video_bitrate: "00000000000000000000000000001540"
audio_sample_rate: 00000000000000000000000000048000
audio_codec: mp3
height: "00000000000000000000000000000352"
filename: 88ed6880-f51a-012c-46b7-1231390a9532
fps: "25.00"
duration: 00000000000000000000000005930800
width: "00000000000000000000000000000640"
original_filename: 88ed6880-f51a-012c-46b7-1231390a9532
created_at: "2010-02-06T06:55:29+00:00"

============================================================

ENCODING ATTRS

============================================================
---
status: error
video_codec: ""
notification: 0
player: flash
updated_at: 2010-02-06 09:01:54.882885 +00:00
container: mp4
video_bitrate: "00000000000000000000000000000400"
profile: 9626ef20-aab8-012c-467f-1231390a9532
audio_bitrate: 00000000000000000000000000000048
audio_sample_rate: ""
audio_codec: aac
profile_title: Flash h264 HI
height: "00000000000000000000000000000360"
filename: 2af39020-f52c-012c-46b8-1231390a9532.mp4
fps: "00000000000000000000000000000024"
duration: 00000000000000000000000005930800
width: 00000000000000000000000000000480
parent: 88ed6880-f51a-012c-46b7-1231390a9532
original_filename: 88ed6880-f51a-012c-46b7-1231390a9532
created_at: "2010-02-06T09:01:42+00:00"

============================================================

Damien Tanner

unread,

Feb 9, 2010, 10:20:00 AM2/9/10

to panda...@googlegroups.com

Hi Chris,

I'm not sure about what is exactly causing the error. But I do know that S3 library isn't the best suited for large files. We are going to be migrating to the RightAWS lib in future.

Damien Tanner - Co-Founder and Director, Panda
+44 (0)20 7099 7486 - pandastream.com - @pandastream

--
You received this message because you are subscribed to the Google Groups "Panda" group.
To post to this group, send email to panda...@googlegroups.com.
To unsubscribe from this group, send email to pandastream...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pandastream?hl=en.

schneck

unread,

Feb 10, 2010, 9:50:25 AM2/10/10

to Panda

Hi Damien,

I'm working with Chris on the project, and we have investigated a bit
further. It seems that merb is causing the problem: We changed all
proxy timeout settings in nginx to very high values, but merb returns
after about 30 seconds if the request has not been processed (what can
be often the case, especially when dealing with very large video
files). In the nginx-logfile, we find things entries like this:

*75 upstream prematurely closed connection while reading response
header from upstream,

Is there any way to tell merb not to timeout?

Thanks,
Stefan

Damien Tanner

unread,

Feb 10, 2010, 11:59:50 AM2/10/10

to panda...@googlegroups.com

Hi Stefan,

This is likely to be because it's taking more than 30s to transfer the uploaded data from nginx to mongrel - not unlikely considering all the data is going through Ruby.

In the new managed service we have solved this using the nginx upload module which passes a reference to the uploaded file on the filesystem, instead of the actual raw file data.

See: http://www.grid.net.ru/nginx/upload.en.html

You will also need to make some changes to the upload action as well. For example:

if (params["file.name"] && params["file.path"])

params[:file] = {:filename => params["file.name"], :tempfile => OpenStruct.new({ :path => params["file.path"] }) }

end

Regards,

Damien Tanner - Co-Founder and Director, Panda
+44 (0)20 7099 7486 - pandastream.com - @pandastream

--

Reply all

Reply to author

Forward