Received: by 10.204.5.194 with SMTP id 2mr1375432bkw.7.1350910351675;
Mon, 22 Oct 2012 05:52:31 -0700 (PDT)
X-BeenThere: silverstripe-dev@googlegroups.com
Received: by 10.205.124.16 with SMTP id gm16ls3667522bkc.7.gmail; Mon, 22 Oct
2012 05:52:30 -0700 (PDT)
Received: by 10.204.127.19 with SMTP id e19mr1372083bks.4.1350910350065;
Mon, 22 Oct 2012 05:52:30 -0700 (PDT)
Received: by 10.204.127.19 with SMTP id e19mr1372082bks.4.1350910350035;
Mon, 22 Oct 2012 05:52:30 -0700 (PDT)
Return-Path: <bummz...@gmail.com>
Received: from mail-bk0-f44.google.com (mail-bk0-f44.google.com [209.85.214.44])
by gmr-mx.google.com with ESMTPS id o9si1019742bko.2.2012.10.22.05.52.29
(version=TLSv1/SSLv3 cipher=OTHER);
Mon, 22 Oct 2012 05:52:30 -0700 (PDT)
Received-SPF: pass (google.com: domain of bummz...@gmail.com designates 209.85.214.44 as permitted sender) client-ip=209.85.214.44;
Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of bummz...@gmail.com designates 209.85.214.44 as permitted sender) smtp.mail=bummz...@gmail.com; dkim=pass header...@gmail.com
Received: by mail-bk0-f44.google.com with SMTP id jc3so856749bkc.3
for <silverstripe-dev@googlegroups.com>; Mon, 22 Oct 2012 05:52:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=message-id:date:from:user-agent:mime-version:to:subject:references
:in-reply-to:content-type;
bh=EwvDBy/zZlQvC9UnBtypfad/TvlF6ovQVGFjukHWRMM=;
b=trhMK9AdhGUdaNcG4JK9RH+aqXFhoKboAoT1G5ODizqYGYCmCf03IvUo/kYZXf31s5
p6zTE1ZHtjJpOEWoivnQOfkG3XQG/ctsjUeP9yV/FSpZX7vo3vK0sFX0Bvaz4KcHPrBo
W55/HO+kDoBecIK8Ma5q5Bz1x7vvR0s9RpUGwO8aBchNPgWCmnudH+ejkMUTcKycfQBF
bF2vDTystLOr9/6rRMS38Vbi2Q8KeH1CXCHPqhmW+ACTLxqYg/EchI64gdEOrckqWKAT
7AMZkh2z77mrHKwLeWtCLXyqlSS8xaWfe1CVxTAKXbrNZhODAYdYkWWxdH9JRkNk7SdU
VtyQ==
Received: by 10.205.126.131 with SMTP id gw3mr2661839bkc.137.1350910349781;
Mon, 22 Oct 2012 05:52:29 -0700 (PDT)
Return-Path: <bummz...@gmail.com>
Received: from bummzack.fritz.box (zux221-120-045.adsl.green.ch. [81.221.120.45])
by mx.google.com with ESMTPS id fm5sm3512997bkc.5.2012.10.22.05.52.28
(version=SSLv3 cipher=OTHER);
Mon, 22 Oct 2012 05:52:29 -0700 (PDT)
Message-ID: <5085418B.6010...@gmail.com>
Date: Mon, 22 Oct 2012 14:52:27 +0200
From: Roman <bummz...@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:16.0) Gecko/20121010 Thunderbird/16.0.1
MIME-Version: 1.0
To: silverstripe-dev@googlegroups.com
Subject: Re: [silverstripe-dev] Block high compression/large dimension images,
that crash the system on resize
References: <29f387c2-01b4-4416-b72e-b360ce2094f7@googlegroups.com>
In-Reply-To: <29f387c2-01b4-4416-b72e-b360ce2094f7@googlegroups.com>
Content-Type: multipart/alternative;
boundary="------------010108060204000507060000"
This is a multi-part message in MIME format.
--------------010108060204000507060000
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Hi Martine
Thanks a lot for your work on this.
I'm pretty sure GD has two different modes when loading images into memory:
Palette images will have 8 bits per pixels and true-color images 32 bits
per pixel. Not sure how monochrome images are handled and I think it
highly depends on the parser at hand.
To be entirely sure how memory will be allocated by the GD library, it
would probably be best to look at its source code directly:
https://bitbucket.org/pierrejoye/gd-libgd/src
But I think it's safe to assume, images will load as 32bit in most of
the cases, which means the memory consumption will be /at least/ *width
* height * 4 bytes
*That's for every image buffer that will be created, and since you
usually have at least a source and a target buffer, these have to be
added up.
In the comments of the PHP documentation, there's a code example of
somebody using a dynamic approach, where he checks the available memory
before processing the image:
http://www.php.net/manual/en/function.imagecreatetruecolor.php#99623
This approach seems flawed though, as it doesn't factor in the size of
the target buffer. If the target buffer is large (eg. resizing a
4000x3000 pixels source to a 3000x2000 target), the multiplier constant
of 1.7 won't be enough.
Personally I think it would be best to allow configuration of a max.
pixel sum for uploaded images. The default setting should be something
sensible that works well with the SS3 system requirements (48mb memory).
Also a customizable handler-function for such images would be nice... so
that it would be possible to write your own routine that deals with
images that are too large (using "ImageMagick" as fallback comes to mind).
The most important use-case to cover is when a user uploads an image
that will consume too much memory. Ideally, the user would get an
error-message, prompting him to upload a low-res version of the image.
This won't cover the asset-sync cases or the unlikely situation where a
huge image will be generated by the system itself though. Since this
check is really simple and computationally inexpensive compared to the
image-manipulation itself, I suggest that a memory-check should be done
whenever a GD object is about to be created. Not sure how error-handling
should work in this case though. Throwing an exception won't do any
good, as it could stall the system as well. Maybe dynamically creating a
"Not enough memory" image and using this instead would be a possible
solution...
If there is a customizable handler function for these cases, the default
implementation could be a dynamic "Not enough memory" image, but
developers could implement their own handler and use ImageMagick instead
or come up with other forms of dealing with these images.
Just my $0.02
Best - Roman
On 10/22/12 1:42 PM, Martimiz wrote:
> Hi all,
>
> I took the liberty to start a new thread for this topic, because it
> was very mixed up in this thread (my fault initially). I hope that's
> allright with you.
>
> Original thread: UploadField documentation
> <%20https://groups.google.com/forum/?fromgroups=#%21topic/silverstripe-dev/RwWB3hoholI>
>
> To summarize:
> Files with very large dimensions but very high compression (like CAD
> files) will often slip the max file size settings, and then crash PHP
> once GD loads them in memory for resizing.
>
> 1. possible solution: use getImageSize() to check the file dimensions.
> GetImageSize() seems to extract width and hight from a file without
> having to create a memory map.
>
> Simon:
> > <?php
> > var_dump($start = memory_get_usage());
> > $size = getimagesize('/Users/simon/
> > Downloads/20121016_190253.jpg');
> > var_dump($size, $end = memory_get_usage());
> > var_dump($end - $start);
>
> > This showed a difference of about 2 KB between $end and $start. The
> image is around 3.2 MB.
>
>
> Question: does anyone know if 24 bit images create larger maps then 8
> bit images? Or are memorymaps all the same anyway?)
>
> Issues:
>
> 1. Decide on a max value:
> a. do we set an (adjustable) limit on the number of pixels (h*w)?
> b. do we try to use existing memory settings, as Jacob is proposing?
> (see:
> c. both?
>
> My first proposal:
> https://github.com/Martimiz/sapphire/commit/b6c29366f00489ff349df32bb72358a4e467b605
>
>
> 2. Decide on where/what to check 'at the gate':
> a. On upload - before ever creating the file
> b. On File Synchronization - ignore large files, rmove large files
> from DB when a user has overwritten the file from FTP?
> c. In GD, before any resizing is done, just to catch any file that has
> slipped through?
>
> As Sam suggested:
> > If that's the case, I would suggest adding the ability to set
> maxWidth, maxHeight
> > or maxPixels (just W x H) for either site-wide, or for specific
> upload fields.
>
> > As a default, perhaps we set max pixels to 10,000,000?
>
>
> 3. Extra checks for images slipping through.
> As jacob suggests:
> a. check the dimensions before resizing, just in case an image slipped
> through
> b. create a lock-file for any image before resizing, and then remove
> it again on completion, to prevent the system from ever crashing on
> the same image more then once.
>
> See Jacobs proposal here: https://github.com/jakr/sapphire/tree/trac5284
>
> I hope I've done right by everone and this sort of sums it up. Now I'm
> wondering how far should we go with this. And should protecting users
> against 'illegal' FTP uploads at all cost be part of it?
>
> Thanks, Martine
>
> --
> You received this message because you are subscribed to the Google
> Groups "SilverStripe Core Development" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/silverstripe-dev/-/fUjsQQnfEPAJ.
> To post to this group, send email to silverstripe-dev@googlegroups.com.
> To unsubscribe from this group, send email to
> silverstripe-dev+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/silverstripe-dev?hl=en.
--------------010108060204000507060000
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Martine<br>
<br>
Thanks a lot for your work on this.<br>
<br>
I'm pretty sure GD has two different modes when loading images
into memory: <br>
Palette images will have 8 bits per pixels and true-color images
32 bits per pixel. Not sure how monochrome images are handled and
I think it highly depends on the parser at hand.<br>
<br>
To be entirely sure how memory will be allocated by the GD
library, it would probably be best to look at its source code
directly: <a class="moz-txt-link-freetext" href="https://bitbucket.org/pierrejoye/gd-libgd/src">https://bitbucket.org/pierrejoye/gd-libgd/src</a><br>
But I think it's safe to assume, images will load as 32bit in most
of the cases, which means the memory consumption will be <i>at
least</i> <b>width * height * 4 bytes<br>
</b>That's for every image buffer that will be created, and since
you usually have at least a source and a target buffer, these have
to be added up.<br>
<br>
In the comments of the PHP documentation, there's a code example
of somebody using a dynamic approach, where he checks the
available memory before processing the image:
<a class="moz-txt-link-freetext" href="http://www.php.net/manual/en/function.imagecreatetruecolor.php#99623">http://www.php.net/manual/en/function.imagecreatetruecolor.php#99623</a><br>
This approach seems flawed though, as it doesn't factor in the
size of the target buffer. If the target buffer is large (eg.
resizing a 4000x3000 pixels source to a 3000x2000 target), the
multiplier constant of 1.7 won't be enough.<br>
<br>
Personally I think it would be best to allow configuration of a
max. pixel sum for uploaded images. The default setting should be
something sensible that works well with the SS3 system
requirements (48mb memory). Also a customizable handler-function
for such images would be nice... so that it would be possible to
write your own routine that deals with images that are too large
(using "ImageMagick" as fallback comes to mind).<br>
<br>
The most important use-case to cover is when a user uploads an
image that will consume too much memory. Ideally, the user would
get an error-message, prompting him to upload a low-res version of
the image. This won't cover the asset-sync cases or the unlikely
situation where a huge image will be generated by the system
itself though. Since this check is really simple and
computationally inexpensive compared to the image-manipulation
itself, I suggest that a memory-check should be done whenever a GD
object is about to be created. Not sure how error-handling should
work in this case though. Throwing an exception won't do any good,
as it could stall the system as well. Maybe dynamically creating a
"Not enough memory" image and using this instead would be a
possible solution...<br>
<br>
If there is a customizable handler function for these cases, the
default implementation could be a dynamic "Not enough memory"
image, but developers could implement their own handler and use
ImageMagick instead or come up with other forms of dealing with
these images.<br>
<br>
Just my $0.02<br>
<br>
Best - Roman<br>
<br>
On 10/22/12 1:42 PM, Martimiz wrote:<br>
</div>
<blockquote
cite="mid:29f387c2-01b4-4416-b72e-b360ce2094f7@googlegroups.com"
type="cite">Hi all,<br>
<br>
I took the liberty to start a new thread for this topic, because
it was very mixed up in this thread (my fault initially). I hope
that's allright with you. <br>
<br>
Original thread: <a moz-do-not-send="true"
href="%20https://groups.google.com/forum/?fromgroups=#%21topic/silverstripe-dev/RwWB3hoholI">UploadField
documentation</a><br>
<br>
To summarize:<br>
Files with very large dimensions but very high compression (like
CAD files) will often slip the max file size settings, and then
crash PHP once GD loads them in memory for resizing.<br>
<br>
1. possible solution: use getImageSize() to check the file
dimensions.<br>
GetImageSize() seems to extract width and hight from a file
without having to create a memory map. <br>
<br>
Simon:<br>
> <?php
<br>
> var_dump($start = memory_get_usage());
<br>
> $size = getimagesize('/Users/simon/
<div><wbr>> Downloads/20121016_190253.jpg'<wbr>);
<br>
> var_dump($size, $end = memory_get_usage());
<br>
> var_dump($end - $start);
<br>
<br>
> This showed a difference of about 2 KB between $end and
$start. The image is around 3.2 MB.<br>
<br>
<br>
Question: does anyone know if 24 bit images create larger maps
then 8 bit images? Or are memorymaps all the same anyway?)<br>
<br>
Issues:<br>
<br>
1. Decide on a max value:<br>
a. do we set an (adjustable) limit on the number of pixels
(h*w)?<br>
b. do we try to use existing memory settings, as Jacob is
proposing?<br>
(see: <br>
c. both?<br>
<br>
My first proposal:
<a class="moz-txt-link-freetext" href="https://github.com/Martimiz/sapphire/commit/b6c29366f00489ff349df32bb72358a4e467b605">https://github.com/Martimiz/sapphire/commit/b6c29366f00489ff349df32bb72358a4e467b605</a><br>
<br>
<br>
2. Decide on where/what to check 'at the gate':<br>
a. On upload - before ever creating the file<br>
b. On File Synchronization - ignore large files, rmove large
files from DB when a user has overwritten the file from FTP?<br>
c. In GD, before any resizing is done, just to catch any file
that has slipped through?<br>
<br>
As Sam suggested:<br>
<div>> If that's the case, I would suggest adding the ability
to set maxWidth, maxHeight <br>
> or maxPixels (just W x H) for either site-wide, or for
specific upload fields.</div>
<div><br>
</div>
<div>> As a default, perhaps we set max pixels to 10,000,000?</div>
<br>
<br>
3. Extra checks for images slipping through. <br>
As jacob suggests: <br>
a. check the dimensions before resizing, just in case an image
slipped through<br>
b. create a lock-file for any image before resizing, and then
remove it again on completion, to prevent the system from ever
crashing on the same image more then once.<br>
<br>
See Jacobs proposal here:
<a class="moz-txt-link-freetext" href="https://github.com/jakr/sapphire/tree/trac5284">https://github.com/jakr/sapphire/tree/trac5284</a><br>
<br>
I hope I've done right by everone and this sort of sums it up.
Now I'm wondering how far should we go with this. And should
protecting users against 'illegal' FTP uploads at all cost be
part of it?<br>
<br>
Thanks, Martine<br>
</div>
<br>
-- <br>
You received this message because you are subscribed to the Google
Groups "SilverStripe Core Development" group.<br>
To view this discussion on the web visit <a
moz-do-not-send="true"
href="https://groups.google.com/d/msg/silverstripe-dev/-/fUjsQQnfEPAJ">https://groups.google.com/d/msg/silverstripe-dev/-/fUjsQQnfEPAJ</a>.<br>
To post to this group, send email to
<a class="moz-txt-link-abbreviated" href="mailto:silverstripe-dev@googlegroups.com">silverstripe-dev@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to
<a class="moz-txt-link-abbreviated" href="mailto:silverstripe-dev+unsubscribe@googlegroups.com">silverstripe-dev+unsubscribe@googlegroups.com</a>.<br>
For more options, visit this group at
<a class="moz-txt-link-freetext" href="http://groups.google.com/group/silverstripe-dev?hl=en">http://groups.google.com/group/silverstripe-dev?hl=en</a>.<br>
</blockquote>
<br>
</body>
</html>
--------------010108060204000507060000--
Message from discussion Block high compression/large dimension images, that crash the system on resize
| Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy |
| ©2013 Google |