Multiple FileStorage.save() operations - pocoo-libs

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Multiple FileStorage.save() operations
Received-SPF: pass (google.com: domain of mr.me...@gmail.com designates 209.85.212.46 as permitted sender) client-ip=209.85.212.46;
MIME-Version: 1.0
In-Reply-To: <e321db95-279e-40fb-9aa2-fbf05bb93036@googlegroups.com>
References: <e321db95-279e-40fb-9aa2-fbf05bb93036@googlegroups.com>
Date: Tue, 31 Jul 2012 16:31:38 -0700
Message-ID: <CA+5DKYwCBC4Pck2qBm3K+sOVjogp9qDTDgJ6+ZRxM07yJQx...@mail.gmail.com>
Subject: Re: Multiple FileStorage.save() operations
From: Gordon Haag <mr.me...@gmail.com>
To: pocoo-libs@googlegroups.com
Content-Type: multipart/alternative; boundary=bcaec501604b7e8c1104c62894f1

--bcaec501604b7e8c1104c62894f1
Content-Type: text/plain; charset=ISO-8859-1

You shouldn't need to save a file twice. You are doing extra disk IO when
you should keep it in RAM until it needs to hit the disk. This is what you
should use instead of writing the file to /tmp.
http://docs.python.org/library/stringio.html

On Tue, Jul 31, 2012 at 3:27 PM, wgoulet <wgou...@gmail.com> wrote:

> Hi,
>
> I'm a relatively new Python and brand new Flask user, so please bear with
> me.
>
> I'm developing a webapp that requires that I permit users to upload files
> which are validated before I store them in their final location on the web
> server's local filesystem. To satisfy this requirement, I have defined
> helper methods that I use to create a temporary copy of a FileStorage
> object that is processed to determine if it is valid. Once this check
> passes, I then want to save the uploaded file in a permanent location.
>
> Here's an example subset of my code:
>
> def confirm():
>     amfile = request.files['zipfile']
>     if validate_file(amfile):
>
>  amfile.save(os.path.join("/www/docs",secure_filename(amfile.filename))
>
> def validate_file(infile):
>     infile.save(os.path.join("/tmp",secure_filename(infile.filename))
>     # My validation code goes in here; I read the file in /tmp and return
> true or false depending on the results
>
>
> The problem I'm running into is that when I call the save() function on a
> FileStorage object twice in a row, the second save() function creates an
> empty copy of the file. In the code above, I have a valid copy of my file
> stored in /tmp, but the file in /www/docs is zero file size. I don't think
> that copying the file from /tmp to /www/docs is the right solution, because
> the validation code could potentially destroy or overwrite the temp copy
> (say if the file is a zip file as it is in my case)
>
> Looking at the source of FileStorage.save(), it looks like the reason for
> this behavior is that the shutil.copyfileobj function advances the
> filepointer when it copies from the source, but it doesn't move it back to
> the beginning of the file stream when it's finished (the copyfileobj docs
> state as much).
>
> As a simple test, I modified the FileStorage.save() function in my local
> werkzeug install to add a file seek call before the copyfileobj call as
> follows:
>
>  from shutil import copyfileobj
>         close_dst = False
>         if isinstance(dst, basestring):
>             dst = file(dst, 'wb')
>             close_dst = True
>         try:
>             # Reset file pointer before copying from object
>             self.stream.seek(0)
>             copyfileobj(self.stream, dst, buffer_size)
>         finally:
>             if close_dst:
>                 dst.close()
>
> With this change, I can use the FileStorage.save() multiple times to save
> multiple copies of the FileStorage file.
>
> Would it make sense to modify FileStorage.save() as I've done here, or is
> there another, better way to achieve my goal?
>
> Thanks,
> Walter
>
> --
> You received this message because you are subscribed to the Google Groups
> "pocoo-libs" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/pocoo-libs/-/kH6rN1tQemoJ.
> To post to this group, send email to pocoo-libs@googlegroups.com.
> To unsubscribe from this group, send email to
> pocoo-libs+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/pocoo-libs?hl=en.
>

--bcaec501604b7e8c1104c62894f1
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You shouldn&#39;t need to save a file twice. You are doing extra disk IO wh=
en you should keep it in RAM until it needs to hit the disk. This is what y=
ou should use instead of writing the file to /tmp.=A0<a href=3D"http://docs=
.python.org/library/stringio.html">http://docs.python.org/library/stringio.=
html</a><br>
<br><div class=3D"gmail_quote">On Tue, Jul 31, 2012 at 3:27 PM, wgoulet <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:wgou...@gmail.com" target=3D"_blank">w=
gou...@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote"=
 style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<div><br></div><div>I&#39;m a relatively new Python and brand new Flask =
user, so please bear with me.</div><div><br></div><div>I&#39;m developing a=
 webapp that requires that I permit users to upload files which are validat=
ed before I store them in their final location on the web server&#39;s loca=
l filesystem. To satisfy this requirement, I have defined helper methods th=
at I use to create a temporary copy of a FileStorage object that is process=
ed to determine if it is valid. Once this check passes, I then want to save=
 the uploaded file in a permanent location.</div>
<div><br></div><div>Here&#39;s an example subset of my code:</div><div><br>=
</div><div>def confirm():</div><div>=A0 =A0 amfile =3D request.files[&#39;z=
ipfile&#39;]</div><div>=A0 =A0 if validate_file(amfile):</div><div>=A0 =A0 =
=A0 =A0 =A0 =A0 =A0amfile.save(os.path.join(&quot;/www/docs&quot;,secure_fi=
lename(amfile.filename))=A0</div>
<div><br></div><div>def validate_file(infile):</div><div>=A0 =A0 infile.sav=
e(os.path.join(&quot;/tmp&quot;,secure_filename(infile.filename))</div><div=
>=A0 =A0 # My validation code goes in here; I read the file in /tmp and ret=
urn true or false depending on the results</div>
<div>=A0 =A0=A0</div><div><br></div><div>The problem I&#39;m running into i=
s that when I call the save() function on a FileStorage object twice in a r=
ow, the second save() function creates an empty copy of the file. In the co=
de above, I have a valid copy of my file stored in /tmp, but the file in /w=
ww/docs is zero file size. I don&#39;t think that copying the file from /tm=
p to /www/docs is the right solution, because the validation code could pot=
entially destroy or overwrite the temp copy (say if the file is a zip file =
as it is in my case)</div>
<div><br></div><div>Looking at the source of FileStorage.save(), it looks l=
ike the reason for this behavior is that the shutil.copyfileobj function ad=
vances the filepointer when it copies from the source, but it doesn&#39;t m=
ove it back to the beginning of the file stream when it&#39;s finished (the=
 copyfileobj docs state as much).</div>
<div><br></div><div>As a simple test, I modified the FileStorage.save() fun=
ction in my local werkzeug install to add a file seek call before the copyf=
ileobj call as follows:</div><div><br></div><div><div>=A0from shutil import=
 copyfileobj</div>
<div>=A0 =A0 =A0 =A0 close_dst =3D False</div><div>=A0 =A0 =A0 =A0 if isins=
tance(dst, basestring):</div><div>=A0 =A0 =A0 =A0 =A0 =A0 dst =3D file(dst,=
 &#39;wb&#39;)</div><div>=A0 =A0 =A0 =A0 =A0 =A0 close_dst =3D True</div><d=
iv>=A0 =A0 =A0 =A0 try:</div><div>=A0 =A0 =A0 =A0 =A0 =A0 # Reset file poin=
ter before copying from object</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 self.stream.seek(0)</div><div>=A0 =A0 =A0 =A0 =
=A0 =A0 copyfileobj(self.stream, dst, buffer_size)</div><div>=A0 =A0 =A0 =
=A0 finally:</div><div>=A0 =A0 =A0 =A0 =A0 =A0 if close_dst:</div><div>=A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 dst.close()</div></div><div><br>
</div><div>With this change, I can use the FileStorage.save() multiple time=
s to save multiple copies of the FileStorage file.</div><div><br></div><div=
>Would it make sense to modify FileStorage.save() as I&#39;ve done here, or=
 is there another, better way to achieve my goal?</div>
<div><br></div><div>Thanks,</div><div>Walter</div><span class=3D"HOEnZb"><f=
ont color=3D"#888888">

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;pocoo-libs&quot; group.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/d/msg/pocoo-libs/-/kH6rN1tQemoJ" target=3D"_blank">https://groups.google=
.com/d/msg/pocoo-libs/-/kH6rN1tQemoJ</a>.<br>=20
To post to this group, send email to <a href=3D"mailto:pocoo-libs@googlegro=
ups.com" target=3D"_blank">pocoo-libs@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:pocoo-libs%=
2Bunsubscribe@googlegroups.com" target=3D"_blank">pocoo-libs+unsubscribe@go=
oglegroups.com</a>.<br>

For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/pocoo-libs?hl=3Den" target=3D"_blank">http://groups.google.com/group/p=
ocoo-libs?hl=3Den</a>.<br>


</font></span></blockquote></div><br>

--bcaec501604b7e8c1104c62894f1--
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
