Re: [CarrierWave] Getting the real file from S3

[Jacob Tjørnholm]

Hi, 

You should be able to do this: 

   open(@document.file.url)

This gives you a File object you can read from. (so append ".read" if you want the contents of the file instead).

/Jacob

On Mon, Aug 13, 2012 at 7:10 PM, Luiz Eduardo Kowalski <luizeduar...@gmail.com> wrote:

Hi all
I have an app where users uploads documents (pdf files, office files, texts, spreadsheets, etc) and my app indexes it so users can search for it after...
I was uploading everything to a /upload folder, but now I'm using Amazon S3...
Before this, I could get the real file via `@document.file` where file is my uploader mounted, and I passed this @documet to a background worker and he gets the file and indexes it...
Now using S3 I can't get the real file...If I try like before, the app fails, saying that there's no file like '/bucket/document/id/name'
Is there a way that I can download it, save it on a tmp folder and index, or something?
By now, I'm using the tmp file that is passed to the form, but if the worker fails at index, I can't retry the worker, because the file doesn't exists anymore...
Any ideas?
If you guys need something to help clarify this question, let me know

--
You received this message because you are subscribed to the Google Groups "carrierwave" group.
To view this discussion on the web visit https://groups.google.com/d/msg/carrierwave/-/FFLcweHJff8J.
To post to this group, send email to carri...@googlegroups.com.
To unsubscribe from this group, send email to carrierwave...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/carrierwave?hl=en.

[Luiz E.]

where did this 'open' came from?
from File?

[Jacob Tjørnholm]

No, it's part of the Standard Library: 

http://www.ruby-doc.org/stdlib-1.9.3/libdoc/open-uri/rdoc/Kernel.html#method-c-open

Not sure where it gets required, but I thinks it's part of every Rails projects. If not, require 'open-uri'. 

/Jacob

On Mon, Aug 13, 2012 at 9:53 PM, Luiz E. <luizeduar...@gmail.com> wrote:

where did this 'open' came from?
from File?

--
You received this message because you are subscribed to the Google Groups "carrierwave" group.

To post to this group, send email to carri...@googlegroups.com.

To unsubscribe from this group, send email to carrierwave+unsubscribe@googlegroups.com.

[Luiz E.]

I'll try it and post a feedback later
thank you

[Luiz E.]

I've tried on console


doc = Document.first
file = open(doc.file.file.authenticated_url)

and I got this
OpenURI::HTTPError: 404 Not Found


I tried with doc.file.file.url too, but Carrierwave gives me this
OpenURI::HTTPError: 403 Forbidden

What am I missing? should I set fog config  fog_public to true?

[Jacob Tjørnholm]

I think so, yes. Take a look at the url and make sure you can download from it in a web browser of using curl.

The 403 you see is Amazon denying to serve the file. IIRC  files are not publicly downloadable by default when stored through fog.

/Jacob

Sent from my phone

--

You received this message because you are subscribed to the Google Groups "carrierwave" group.

[Luiz E.]

Worked, but I'm seeing a strage behavior
I've uploaded two files: a .txt and .pdf
When I read the .txt, I got a StringIO class, and when I read a .pdf, I
got a tmp file...
is this expected? how can I ensure that I always have a File instead a
StringIO?
btw, thank you very much :D

[Jacob Tjørnholm]

This is an implementation detail somewhere. The point is that you shouldn't rely on the class of the object being returned. 

See the accepted answer here: 

http://stackoverflow.com/questions/694115/why-does-ruby-open-uris-open-return-a-stringio-in-my-unit-test-but-a-fileio-in

If you're sure that you need a file object, you can create a tempfile: 

http://ruby-doc.org/stdlib-1.9.3/libdoc/tempfile/rdoc/Tempfile.html

/Jacob

--
You received this message because you are subscribed to the Google Groups "carrierwave" group.
To post to this group, send email to carri...@googlegroups.com.

To unsubscribe from this group, send email to carrierwave+unsubscribe@googlegroups.com.

[Luiz E.]

As I can see, ruby will return a StringIO if the file size itself is
less than 10k...
I think I will go with the Tempfile...
Thank you, Jacob

[Thomas]

> --
> You received this message because you are subscribed to the Google Groups
> "carrierwave" group.
> To post to this group, send email to carri...@googlegroups.com.
> To unsubscribe from this group, send email to

> carrierwave...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/carrierwave?hl=en.
>

hi,

the open will work, be careful about the rights thing, you might need
to generated signed urls to make things work if you have restricted
access to your bucket.

I've been playing quite a lot with this kind of direct access in the
last 18 months and I ended up writing a module to handle it using the
aws gem (https://github.com/appoxy/aws/) to connect to the account,
bucket and get the object using its path. basicaly calls look like
this :

MyStore.get(an_object.image.path)
MyStore.put("some/wanted/path", an_image.to_s)
etc...

why use that ? the product handle a lot of files and need to store
some of them directly (tmp use etc ...)

to avoid any trouble I added a limited number of trials in case of
networks errors etc ... and a caching part using redis

I recently found out Ruby has a built-in timeout class/module that
could also make things nicer but did not take time to add it yet

iirc Fog might be able to do similar stuff, but I like the idea to
have a lib that just take care of files handling

--
Thomas Riboulet
+33 (0) 698 926 057