Fundamental problem with Paperclip and unusual image formats

[Rodeoclash]

I've been trying to solve a fundamental problem with Paperclip and
unusual image formats, such as targa.

When files are uploaded with Paperclip they are stored as temporary
files, with semi random file names. With JPG and GIF files it does not
matter what the filename is, it's possible to determine the file type
with the identify command. With targa (tga) files, the file must have
an explicit extension .tga for the identify command to work correctly,
this extension is lost when the temporary file is created.

When the thumbnail generation is performed, it attempts to find the
width/height of the original image, now stored in the Tempfile. This
is where Paperclip fails with a “not recognized by the 'identify'
command.

I've tested running the identify command on a tga file copied from my /
tmp directory and renamed with a .tga extension, this works fine. I
can also run the identify command from the root of my rails site.

Going forwards, I'm not sure how to correct this? I either have to
patch the way Tempfile stores its filenames to make sure they include
the .tga extension. I've had limited success with this, or find a way
of patching ImageMagick to identify tga files without an explicit
extension on them.

Finally, I could alter the way Paperclip generates temporary files and
generate permanent files that are garbage collected somehow.

I'd like to get the thoughts on the list about this.

Sam

[Rodeoclash]

I've also posted this question on Stack Overflow with some additional
details.

http://stackoverflow.com/questions/1658581/using-paperclip-to-process-tga-targa-files

Sam

[Jonathan Yurek]

I'm not sure why you're seeing this problem. Paperclip has a custom
Tempfile subclass specifically to handle this case, which makes sure
the Tempfile ends with the correct extension. What version of
Paperclip are you using?

irb(main):002:0> Paperclip::VERSION
=> "2.3.1"
irb(main):003:0> Paperclip::Tempfile.new("image.tga")
=> #<File:/var/folders/MC/MCMYsNIwEeSNb5+PlFcmU++++TI/-Tmp-/image,
23562,0.tga>
irb(main):004:0>

This is what should be happening. Are you sure your installation of
ImageMagick can handle Targas?

For verification, please set Paperclip.options[:log_command] = true in
your paperclip.rb initializer and try to process a Targa. You will see
the exact command in the log. Please paste it here.

--
Jonathan Yurek, Founder and CTO
thoughtbot, inc.
organic brains. digital solutions.

617.482.1300 x114
http://www.thoughtbot.com/

[Jeff Pritchard]

Hi Jonathan,

Here's the console output of the two things you asked for:
JP-iMac:jp jeffpritchard$ script/console development
Loading development environment (Rails 2.1.0)
>> Paperclip::VERSION
=> "2.3.1.1"
>> Paperclip::Tempfile.new("image.tga")
=> #<File:/var/folders/SD/SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/image,
7289,0.tga>
>>

The log output from create when trying to attach a jpeg file is like
this:
Processing ImagesController#create (for 127.0.0.1 at 2009-11-01
22:28:24) [POST]
Session ID:
BAh7BzoMY3NyZl9pZCIlM2FkNmRkNzA1ZGEyNzBkOTBhY2Y2MzEyOWU2Y2Zj
NDEiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhh
c2h7AAY6CkB1c2VkewA=--9eeae5e415deba3d3a493d3445580ddd1a1553b7
Parameters: {"commit"=>"Create",
"authenticity_token"=>"99bb89e5b9f6409d35dbd7f878eb8b11b4daa20c",
"action"=>"create", "controller"=>"images", "image"=>{"name"=>"test1",
"photo"=>#<File:/var/folders/SD/SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/CGI.
7305.2>}}
[4;36;1mSQL (0.000166) [0m [0;1mSET NAMES 'utf8' [0m
[4;35;1mSQL (0.000076) [0m [0mSET SQL_AUTO_IS_NULL=0 [0m
[4;36;1mImage Columns (0.001497) [0m [0;1mSHOW FIELDS FROM
`images` [0m
[paperclip] /usr/local/bin/identify -format "%wx%h" "/var/folders/SD/
SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/stream.7305.0"[0] 2>/dev/null
[paperclip] An error was received while processing:
#<Paperclip::NotIdentifiedByImageMagickError: /var/folders/SD/
SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/stream.7305.0 is not recognized by
the 'identify' command.>
[4;35;1mSQL (0.000134) [0m [0mBEGIN [0m
[4;36;1mSQL (0.000058) [0m [0;1mCOMMIT [0m
Rendering images/new
Completed in 0.09581 (10 reqs/sec) | Rendering: 0.00527 (5%) | DB:
0.00193 (2%) | 200 OK [http://localhost/images]


I don't know how to "process a targa". If you give me some more info
on that I'll send that output along as well.

thanks much for the help!

jp

[Jonathan Yurek]

By process, I meant take a .tga file and try to identify it with
ImageMagick. But I can see by your log output that the command isn't
being executed properly for identify. I'll have to look into that. I
know it does for convert, because the format conversion depends on it.
So I assume it's just identify that's the issue. Thanks for your help.

[Rodeoclash]

Hi Jonathan,

It's just the identify command. I tested this by taking the generated
temp file and appending .tga on the end of it, this worked fine but
running identify without appending .tga caused an error.

Thanks,

Sam

[Anton Mironov]

I have the same problem with .ico files.
ImageMagick is not able to identify these files without extension:
~ > identify favicon.ico
favicon.ico[0] ICO 32x32 32x32+0+0 8-bit DirectClass 5.3kb
favicon.ico[1] ICO 16x16 16x16+0+0 8-bit DirectClass 5.3kb
~ > mv favicon.ico favicon
~ > identify favicon
identify: no decode delegate for this image format `favicon' @
constitute.c/ReadImage/503.


The rails log when trying to attach an .ico file is like this:
Processing RoomsController#update (for 127.0.0.1 at 2009-11-03
19:01:17) [PUT]
Parameters: {"action"=>"update", "_method"=>"put",
"controller"=>"rooms", "room"=>{"icon"=>#<File:/var/folders/K-/K-Y-
f0DXEVKNmKQxVh1YKk+++TI/-Tmp-/RackMultipart.29300.0>}}
...
[paperclip] identify -format "%wx%h" "/var/folders/K-/K-Y-
f0DXEVKNmKQxVh1YKk+++TI/-Tmp-/stream.29300.0"[0] 2>/dev/null

[paperclip] An error was received while processing:

#<Paperclip::NotIdentifiedByImageMagickError: /var/folders/K-/K-Y-
f0DXEVKNmKQxVh1YKk+++TI/-Tmp-/stream.29300.0 is not recognized by the
'identify' command.>


Mac OS 10.5.8
ImageMagick 6.5.3-7
Rails 2.3.4
Paperclip 2.3.1.1

Hope it will help.

[Jeff Pritchard]

Thanks Sam.

Sam and Jonathan, is this a bug that snuck in to a recent new version
of paperclip? Or do you think it is something caused by my (2)
environments or how I'm using paperclip? Paperclip has been around
for awhile and adored by many. It seems unlikely that nobody has
tried it with non-gif image files before now. :)

thanks for any help in getting around this,
cheers,
jp

[Rodeoclash]

I'm not sure, I've never used Paperclip before about two days ago :)

However, when I had a look at the way it was generating the first
tempfile, if it's been like that in the past then I can't see how it
could have supported anything else other than png, gif, jpg (and
whatever else the identify command can recognise without needing an
explicit file extension)

Nice code by the way Jonathan, I've only been using Ruby for about six
months and looking through your classes has taught me a thing or
two :)

[Jonathan Yurek]

Ok I've pushed a fix to this to github. Can anyone who had a problem
with ImageMagick not identifying things because of the filename please
pull this and try again?

[Jeff Pritchard]

Hi Jonathan,
I noticed the "blame" link on git hub. I verified that the iostream
file in my plugins dir has:
def to_tempfile
name = respond_to?(:original_filename) ? original_filename :
(respond_to?(:path) ? path : "stream")
tempfile = Paperclip::Tempfile.new(File.basename(name))
tempfile.binmode
self.stream_to(tempfile)
end

So I think I have your fix.

I also uninstalled the gem to make sure that wasn't hosing things up.
No joy. Same error.

thanks,
jp

[Jonathan Yurek]

That's strange, because I've verified that it was working with .tga
files in a sample app I created just for that. Are you sure you've got
that version installed as the plugin in the app? Can you try
processing one and pasting what the log spits out? And, as a sanity
check, after you downloaded the new code and copied it into your
vendor/plugins, you did restart the app, right?

[Jonathan Yurek]

Ok, from the looks of this, the Tempfile is doing what it should be,
but you actually have a problem with ImageMagick. Can you try a
similar command from the command line manually? Perhaps using the same
file you're trying to upload so we're using the same reference.

(also, this email appears to have been sent only to me, so I'm
replying to the whole list. I also didn't see it before I saw your
other email to the whole list, so sorry I asked for information you
already supplied)

On Nov 4, 2009, at 9:50 PM, Jeff Pritchard wrote:

> Hi Jonathan,

> I tried it with a jpeg file and it failed (looks like the same thing
> again).
>
> I got this in the log:
> Processing ImagesController#create (for 127.0.0.1 at 2009-11-04
> 18:23:28) [POST]

> Session ID:
> BAh7BzoMY3NyZl9pZCIlM2FkNmRkNzA1ZGEyNzBkOTBhY2Y2MzEyOWU2Y2Zj
> NDEiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhh
> c2h7AAY6CkB1c2VkewA=--9eeae5e415deba3d3a493d3445580ddd1a1553b7
> Parameters: {"commit"=>"Create",
> "authenticity_token"=>"99bb89e5b9f6409d35dbd7f878eb8b11b4daa20c",
> "action"=>"create", "controller"=>"images", "image"=>{"name"=>"test

> it", "photo"=>#<File:/var/folders/SD/SDx8I2aFFImM18OrHU-e8U+++TI/-
> Tmp-/CGI.9016.2>}}
> Image Columns (0.001576) SHOW FIELDS FROM `images`

> [paperclip] /usr/local/bin/identify -format "%wx%h" "/var/folders/SD/

> SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo,9016,0.jpeg"[0] 2>/dev/null

> [paperclip] An error was received while processing:
> #<Paperclip::NotIdentifiedByImageMagickError: /var/folders/SD/

> SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo,9016,0.jpeg is not recognized
> by the 'identify' command.>

> [paperclip] /usr/local/bin/identify -format "%wx%h" "/var/folders/SD/

> SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo,9016,0.jpeg"[0] 2>/dev/null

> [paperclip] An error was received while processing:
> #<Paperclip::NotIdentifiedByImageMagickError: /var/folders/SD/

> SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo,9016,0.jpeg is not recognized
> by the 'identify' command.>

> [paperclip] /usr/local/bin/identify -format "%wx%h" "/var/folders/SD/

> SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo,9016,0.jpeg"[0] 2>/dev/null

> [paperclip] An error was received while processing:
> #<Paperclip::NotIdentifiedByImageMagickError: /var/folders/SD/

> SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo,9016,0.jpeg is not recognized
> by the 'identify' command.>
> SQL (0.024559) BEGIN
> SQL (0.000070) COMMIT
> Rendering images/new
> Completed in 0.52305 (1 reqs/sec) | Rendering: 0.00306 (0%) | DB:
> 0.02620 (5%) | 200 OK [http://localhost/images]
>
>
>
> I did this to get your new version:
> JP-iMac:jp jeffpritchard$ script/plugin install --force git://
> github.com/thoughtbot/paperclip.git
> removing: /Users/jeffpritchard/Desktop/jp/vendor/plugins/
> paperclip/.git
> Initialized empty Git repository in /Users/jeffpritchard/Desktop/jp/
> vendor/plugins/paperclip/.git/
> remote: Counting objects: 181, done.
> remote: Compressing objects: 100% (140/140), done.
> remote: Total 181 (delta 60), reused 79 (delta 20)
> Receiving objects: 100% (181/181), 113.58 KiB | 169 KiB/s, done.
> Resolving deltas: 100% (60/60), done.
>
>
> What can I look at to verify that I got the new stuff? Or what is
> the proper way to get it?

>
> thanks,
> jp
>
> On Nov 4, 2009, at 4:00 PM, Jonathan Yurek wrote:
>
>>

[Jeff Pritchard]

I appreciate all the help Jonathan,

I tried this:
JP-iMac:jp jeffpritchard$ identify /Users/jeffpritchard/Desktop/test.jpg
identify: no decode delegate for this image format `/Users/
jeffpritchard/Desktop/test.jpg' @ constitute.c/ReadImage/503.

I absolutely hate trying to install magick (and its pre-reqs). Worst
time I've ever had installing anything (on both my mac and my server
as I recall).

Any idea how to get the "delegates" I need for other image formats?

thanks,
jp

[Jonathan Yurek]

Whenever I've installed it, it's been with MacPorts or some other
package management system. I don't recall the last time I've done it
from source, so I'm afraid I can't help much there.

[Rodeoclash]

Jonathan,

Can confirm it's working perfectly with .tga files here!

Thanks for the fix!

Sam

> ...
>
> read more »

[Rodeoclash]

Jeff,

I'd install MacPorts and use that to install ImageMagick. I had it up
and running and working with Paperclip in about 5 minutes.

Thanks,

Sam

> ...
>
> read more »

[Anton Mironov]

Can confirm that newest paperclip working perfectly with .ico files
too!

Thank you, Jonathan!

[Jeff Pritchard]

Hi everyone,
Sorry to create so much traffic on this.

I used MacPorts to reinstall imagemagick on my mac. I had tried that
before but it was failing then . Now it installed successfully. I
rebooted.

From the command line I now get:
JP-iMac:~ jeffpritchard$ identify /Users/jeffpritchard/Desktop/foo2.jpg
/Users/jeffpritchard/Desktop/foo2.jpg JPEG 418x640 418x640+0+0 8-bit
DirectClass 107KiB 0.000u 0:00.009

which looks real good.

For some reason though, it seems to have put it in:
JP-iMac:~ jeffpritchard$ which identify
/opt/local/bin/identify


Sadly though, my use of paperclip still fails and I get this log which
looks the same to me:
Processing ImagesController#create (for 127.0.0.1 at 2009-11-05
08:02:36) [POST]
Session ID:
BAh7BzoMY3NyZl9pZCIlNTFjNTliMGY1NjdjYTc3YjE4ZDY3OTAzZWQ5Y2Fj
ZDEiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhh
c2h7AAY6CkB1c2VkewA=--48bfc74c0d046b256f06a9bba55e8eaf17b7e587

Parameters: {"commit"=>"Create",

"authenticity_token"=>"a64f421816a9a808c7742726091ee4924419339a",

"action"=>"create", "controller"=>"images",

"image"=>{"name"=>"finally", "photo"=>#<File:/var/folders/SD/
SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/CGI.220.2>}}
Image Columns (0.001402) SHOW FIELDS FROM `images`

[paperclip] /usr/local/bin/identify -format "%wx%h" "/var/folders/SD/

SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo2,220,0.jpg"[0] 2>/dev/null

[paperclip] An error was received while processing:
#<Paperclip::NotIdentifiedByImageMagickError: /var/folders/SD/

SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo2,220,0.jpg is not recognized by

the 'identify' command.>
[paperclip] /usr/local/bin/identify -format "%wx%h" "/var/folders/SD/

SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo2,220,0.jpg"[0] 2>/dev/null

[paperclip] An error was received while processing:
#<Paperclip::NotIdentifiedByImageMagickError: /var/folders/SD/

SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo2,220,0.jpg is not recognized by

the 'identify' command.>
[paperclip] /usr/local/bin/identify -format "%wx%h" "/var/folders/SD/

SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo2,220,0.jpg"[0] 2>/dev/null

[paperclip] An error was received while processing:
#<Paperclip::NotIdentifiedByImageMagickError: /var/folders/SD/

SDx8I2aFFImM18OrHU-e8U+++TI/-Tmp-/foo2,220,0.jpg is not recognized by
the 'identify' command.>
SQL (0.000146) BEGIN
SQL (0.000055) COMMIT
Rendering images/new
Completed in 1.19324 (0 reqs/sec) | Rendering: 0.00237 (0%) | DB:
0.00160 (0%) | 200 OK [http://localhost/images]


Is paperclip somehow finding some other/old magick installation and
using that even though the command line version works?
From the log it looks like paperclip is using the "magick" in /usr/
local/bin

The MacPorts installer only lets you choose a volume to install to,
not an actual location. ;(

thanks in advance for any words of wisdom,
jp

>>>>>>>>>> [paperclip] An error was received while processing:
>>>>>>>>>> #<Paperclip::NotIdentifiedByImageMagickError: /var/folders/
>>>>>>>>>> SD/

[Jonathan Yurek]

You seem to have Paperclip.options[:command_path] set to "/usr/local/
bin" according to what's getting run in the log. Set that to "/opt/
local/bin" where your new ImageMagick is installed and it should make
a difference.

[Jeff Pritchard]

Ok,
Finally got this working. Found a place with better instructions on
installing the jpeg library and recompiled that and magick.

Thanks much for all the help! (and the great attachment plugin)

cheers,
jp