Vips::Error:VipsForeignLoad During heavy image processing

[da...@harris.org.nz]

Hi,

I have a image processing application which has quite spiky load - a lot of photographs are edited and then uploaded one after another to my application.

Images are uploaded directly to S3 using the standard pre-sign, and then the 'promotion' is performed in a background job:

Shrine::Attacher.promote { |data| Shrine::PromoteWorker.perform_in(30.seconds, data) }

Most of the time, this all works perfectly, however on really heavy days, I get a lot of Vips::Error:VipsForeignLoad errors. Weirdly, even once the servers have calmed down, it takes a while for the jobs to complete successfully. I've confirmed multiple times that source files are perfectly fine. I can process them locally, and eventually (after exponential backoff) the job succeeds.

The Exception message is:

Vips::Error: VipsForeignLoad: "/tmp/shrine-s320190512-12411-1jfqxfo.jpg" is not a known file format

I've had some issues with hard drive space in the past, but that is well and truly solved and confirmed via free space monitoring.

My uploader is relatively complex (it resizes to 3 sizes, one of which is watermarked), but the exception always occurs at the first resize, so I think it's unrelated to any of that.

process(:store) do |io, context|
  versions = { printable: io } # retain original


  io.download do |original|
    pipeline = ImageProcessing::Vips.source(original)


    versions[:mobile_index] = pipeline.resize_to_limit!(768, 768, sharpen: false) # Full-width. # <---- EXCEPTION ALWAYS OCCURS HERE
    versions[:index] = pipeline.resize_to_limit!(360, 360, sharpen: false) # 2 up on tablet, 3 up on desktop


    # Watermark example
    # https://groups.google.com/d/msg/ruby-shrine/6v6dMVG-ygs/LxEbXw5YCAAJ
    #
    watermark = host_logo_file(context)


    versions[:shareable] = pipeline
      .resize_to_limit(1200, 1200, sharpen: false)
      .composite(watermark,    # Set the watermark as the overlay image
        mode: "over",          # Apply the watermark 'over' the source (default)
        gravity: "south-east", # Apply the watermark in the bottom right corner
        offset: [55, 55],      # Apply the watermark 55 pixels up and left from ':gravity' setting
      )
      .call                    # Apply operation


    watermark.close!
  end


  versions # return the hash of processed files
end

Could it be the case that ImageProcessing::Vips.source returns before the image is full downloaded from S3? Therefore that when I try and call .resize_to_limit! it has a file which is corrupted. That doesn't quite explain why 10 mins later the job fails, yet after an hour it does.

I haven't been able get my hands on the /tmp/shrine-s320190512-12411-1jfqxfo.jpg file, it gets cleaned by before I can get there. I assume Shrine creates a TempFile object which is why the file no longer exists. Weirdly though, there are quite a few files left in /tmp/, such as:

$ ls -1 /tmp/
image_processing20190505-29621-109gfjd.jpg
image_processing20190505-29621-15q8swz.jpg
image_processing20190505-29621-1c7psbu.jpg

However, these files are all the successful output of the process step, not the input.

Sorry for the long and detailed question. Does anyone have any idea how Vips::Error:VipsForeignLoad can be raised?

Any help greatly appreciated, it's becoming a thorn in my side.

Thanks,

Dave

[jesse....@gmail.com]

Hi Janko,

I want to report that I've begun seeing this on my production-like system as well - although not under heavy load as Dave has described and using simple file upload to S3 for both cache and store I get the same issue as described with:

```

Vips::Error (VipsForeignLoad: "/tmp/shrine20190528-1049-1fw9lic.jpg" is not a known file format

):

app/uploaders/image_uploader.rb:77:in `block in <class:ImageUploader>'

```

I do see the same jpg file referenced above in the /tmp folder.

and seems to be happening around the same point as Dave as well during the versions step:

```

medium = pipeline.resize_to_limit!(700, 700)

```

ImageUploader class

```

# frozen_string_literal: true

require 'image_processing/vips'

class ImageUploader < Shrine

  VERSION_TYPES = {

    ORIGINAL: :original,

    MEDIUM: :medium,

    SMALL: :small,

    THUMBNAIL: :thumb,

    SQUARE: :square

  }.freeze

  VERSION_LOCATION_TYPES = {

    ORIGINALS: 'originals',

    MEDIUMS: 'mediums',

    SMALLS: 'smalls',

    THUMBNAILS: 'thumbs',

    SQUARES: 'squares'

  }.freeze

  plugin :determine_mime_type

  plugin :delete_promoted

  plugin :infer_extension, inferrer: :mini_mime

  plugin :processing

  plugin :store_dimensions, analyzer: :ruby_vips

  plugin :upload_options, store: (lambda do |_io, _context|

    { acl: 'public-read' }

  end)

  plugin :validation_helpers

  plugin :versions

  def generate_location(io, context)

    version =

      if context[:version] == ImageUploader::VERSION_TYPES[:MEDIUM]

        ImageUploader::VERSION_LOCATION_TYPES[:MEDIUMS]

      elsif context[:version] == ImageUploader::VERSION_TYPES[:SMALL]

        ImageUploader::VERSION_LOCATION_TYPES[:SMALLS]

      elsif context[:version] == ImageUploader::VERSION_TYPES[:THUMBNAIL]

        ImageUploader::VERSION_LOCATION_TYPES[:THUMBNAILS]

      elsif context[:version] == ImageUploader::VERSION_TYPES[:SQUARE]

        ImageUploader::VERSION_LOCATION_TYPES[:SQUARES]

      else

        ImageUploader::VERSION_LOCATION_TYPES[:ORIGINALS]

      end

    ['images', version, super].compact.join('/')

  end

  process(:store) do |io, _|

    original = io.download

    pipeline = ImageProcessing::Vips

      .source(original)

    medium = pipeline

      .resize_to_limit!(700, 700)

    small = pipeline

      .resize_to_limit!(400, 400)

    thumb = pipeline

      .resize_to_limit!(200, 200)

    square = pipeline

      .resize_to_fill!(100, 100)

    original.close!

    { original: io, medium: medium, small: small, thumb: thumb, square: square }

  end

end

```

shrine.rb config

```

# frozen_string_literal: true

require 'shrine'

require 'shrine/storage/s3'

s3_options = {

    region: Rails.configuration.s3_region,

    bucket: Rails.configuration.s3_bucket_name,

    access_key_id: Rails.configuration.s3_access_key_id,

    secret_access_key: Rails.configuration.s3_secret_access_key

}

Shrine.storages = {

    cache: Shrine::Storage::S3.new(prefix: 'tmp/uploads', upload_options: { acl: 'public-read' }, **s3_options),

    store: Shrine::Storage::S3.new(upload_options: { acl: 'public-read' }, **s3_options)

}

Shrine.plugin :logging, logger: Rails.logger

Shrine.plugin :cached_attachment_data

Shrine.plugin :upload_options

Shrine.plugin :activerecord

```

Using:

gem 'shrine', '~> 2.17.1'

libvips 8.6.4 (and 8.8.0 as most recent update, same behavior)

Any help or ideas here would be awesome!

Cheers and thanks

Jesse

[da...@harris.org.nz]

Hi,

Ok well at least it's not just me. Hopefully we can all worth together to get to the bottom of it. We should post this to Github issues if we don't get anywhere now it's affecting 2 people.

Jesse, you said that you were able to find the file in /tmp, can you inspect it with file or anything. If it's not sensitive, can you upload it here so we can try and see what's 'wrong' with it. Is it the same file size as the version on S3? What about its sha1? My current theory is that ImageProcessing::Vips.source(original) is returning before the file is fully downloaded, but without access to the file, I can't see what is going on.

Just to compare setups:

• libvips v8.6.5
• shrine v2.11.0
• image_processing v1.6.0
  
• ruby-vips v2.0.12
  
• Shrine plugins (* denotes in common with Jesse)
• add_metadata
• validation_helpers*
• processing*
• versions*
• activerecord*
• cached_attachment_data*
• logging*
• determine_mime_type
• backgrounding
• default_url_options
• Not a plugin but 
• We are both over-riding #generate_location
  
• We are both storing and the original version
• I am using backgrounding via Sidekiq but Jesse is not, so that's not part of problem
• Deployed on Amazon EC2 (so you would've thought that download from S3 would be nearly instant and reliable given there is no 'internet' involved)

My #generate_location method:

def generate_location(io, context)
  default_unique_identifier = [context[:version], super].compact.join('-')
  client_id = context[:record].client_id
  appointment_id = context[:record].appointment_id


  # Keep in sync with Shrine.storages[:cache].presign (UploadableAppointment#upload_details)
  ['clients', client_id, 'appointments', appointment_id, 'images', default_unique_identifier].join('/')
end

Jesse, it would be great if you could take a look at the file in /tmp/ so we can work out what wrong with a test-case.

If there is any more information I can provide let me know, I'm eager to help out.

Dave

[jesse....@gmail.com]

Hi Dave,

I was playing around with this about more yesterday evening.

I have some new findings but not promising with respect to libvips.

By switching out the pipeline processor from Vips to MiniMagick I was able to get things working.

I used the code example that Janko had provided for our ImageUploader class from https://github.com/shrinerb/shrine/blob/master/doc/plugins/versions.md):

process(:store) do |io, _|

    versions = { original: io }

    io.download do |original|
      pipeline = ImageProcessing::MiniMagick.source(original)

      versions[:medium] = pipeline.resize_to_limit!(700, 700)
      versions[:small] = pipeline.resize_to_limit!(400, 400)
      versions[:thumb] = pipeline.resize_to_limit!(200, 200)
      versions[:square] = pipeline.resize_to_fill!(100, 100)
    end

    versions
  end

Code that doesn't work (used elsewhere, original examples):

process(:store) do |io, _|

    versions = { original: io }

    io.download do |original|
      pipeline = ImageProcessing::MiniMagick.source(original)

      versions[:medium] = pipeline.resize_to_limit!(700, 700)
      versions[:small] = pipeline.resize_to_limit!(400, 400)
      versions[:thumb] = pipeline.resize_to_limit!(200, 200)
      versions[:square] = pipeline.resize_to_fill!(100, 100)
    end

    versions
  end

  process(:store) do |io, _|
    original = io.download
    pipeline = ImageProcessing::Vips.source(original)

    medium = pipeline.resize_to_limit!(700, 700)
    small = pipeline.resize_to_limit!(400, 400)
    thumb = pipeline.resize_to_limit!(200, 200)

    square = pipeline.resize_to_fill!(100, 100)

    original.close!

    { original: io, medium: medium, small: small, thumb: thumb, square: square }
  end

I've got some more additional tests to run, and for some reason width and height are not being extracted, so that's next (not related to this specific issue I think).

Modules and plugins used:

libvips 8.8.0 (although was at 8.6.4 at started having the same issue, bumped it up to newest for the fun of it).

Shrine.rb

Shrine.plugin :logging, logger: Rails.logger
Shrine.plugin :cached_attachment_data
Shrine.plugin :upload_options
Shrine.plugin :activerecord

ImageUploader.rb

 

plugin :delete_promoted
plugin :determine_mime_type
plugin :infer_extension, inferrer: :mini_mime
plugin :processing
plugin :store_dimensions, analyzer: :ruby_vips
plugin :validation_helpers
plugin :versions

I do have another uploader for background processing (accepting images from mobile submission via API as base64 encoded strings), using the same module attribute in fact:

MobileImageUploader.rb

plugin :backgrounding
plugin :data_uri
plugin :determine_mime_type
plugin :delete_promoted
plugin :infer_extension, inferrer: :mini_mime
plugin :processing
plugin :store_dimensions, analyzer: :ruby_vips

plugin :upload_options, store: (lambda do |_io, _context|

  { acl: 'public-read' }
end)

plugin :validation_helpers
plugin :versions

Interesting gems:

gem 'aws-sdk-s3', '~> 1.40.0'
gem 'image_processing', '~> 1.9.0'
gem 'ruby-vips', '~> 2.0.13'

gem 'shrine', '~> 2.17.1'

On the Shrine .download method...

I've seen some interesting platform dependent behavior where I've actually had timing issues locally on MacOS where it was storing to my Rails tmp folder (instead of a preferred system /tmp folder to better resemble production-like platforms).  It wasn't in a processing step per se but when I was using the attachers/uploaders directly in service object calls doing more interesting stuff pulling a PDF template from S3, attempting to temp store it locally for an operation (form fill) which would then be sent back up to S3.  .download caused problems on the original pull down where the file didn't exist or was not ready for the next step, however, using proper a temporary file with Ruby operations (not TempFile as it's the same implementation that Shrine uses) in place of .download worked as expected (and placed my temp file in the preferred MacOS system /tmp folder).  Possible that the file is not getting closed with .download?

The new code block that I'm using above is rescuing and deleting the temp file at the end of the process step as it includes cleanup, whereas the code I had before would leave it in place.  

Using the original code the file is left behind (shrine20190529-28338-10kxpc2.jpg) is the correct size and I'm able to scp download it locally and view it as the original, so looks ok there.

Hopefully all my thoughts and writings here are coherent, did lots of jumping around with different tests the last hour!

Not sure what to do at this point - I might just switch back to MiniMagick temporarily in order to get this feature pushed through and continue to work through this with you and Janko.

Cheers,

Jess

[jesse....@gmail.com]

One other note to add as an observation, probably doesn't help much, but image dimensions using the store_dimensions plugin with ruby_vips isn't doing its job (nulls in the db), however using mini_magick works.  Locally on my dev box I wasn't having any issues with libvips.  I'll try to check some other environments today other than staging to see what happens (sudden bad environment config that used to work fine?).

I'll be using mini_magick in the meantime to move this work I'm doing through but will keep in touch here as I'd like to move back to vips for handling everything.

Jess

[da...@harris.org.nz]

Hi Jesse,

Yes I suspect you are right about the TempFile (or not a real TempFile) being a potential source of the problem. The thing that doesn't sit right with me though is how when this happens, I can re-try the job and it fails again and again, until it eventually works. That really implies it's some kind of network issue or something.

I switched the libvips when doing a huge migration of one system to another with something like 200,000 images. Using mini_magick was about 3 times slower in thumbnailing so it would've otherwise taken months! I guess I could theoretically switch back now that migration is over. Having a dependency on libvips is actually quite a large pain in the neck as I need to build it from source in production and Semaphore test environments... Anyway, that's beside the point, it should work.

It's interesting you talk about platform specific behaviour, I've never seen it in development on MacOS, only in production on Amazon Linux. But also, it seems that it only does it on "high" (not sure how high) load so that's probably why.

Did you say that you were able to get this error reliably in your development environment? Can you attach byebug or something to halt execution and take a good look around with the file etc. I may not help if the file is being downloaded from S3, it hits the exception, and trigger you breakpoint, by which time the file is fully downloaded... If you can reliably get this error are you able to create a dummy project so a) I can try it and b) people like Janko and other maintainers can see it also. I know this takes time but I don't think we are going to get far without it.

Dave

[da...@harris.org.nz]

Hi,

Just another data point I had over the weekend. Even though my servers have about 8GB free space, during processing I ran out of hard drive space:

Errno::ENOSPC: No space left on device

Vips::Error: vips2png: unable to write "/tmp/image_processing20190610-12659-1rb4dde.png"

Looking at it now, it's all cleaned itself up again ...

# df -h

Filesystem      Size  Used Avail Use% Mounted on

devtmpfs        969M   56K  969M   1% /dev

tmpfs           979M     0  979M   0% /dev/shm

/dev/nvme0n1p1   15G  6.4G  8.2G  44% /

But that implies that the TempFile isn't being cleanup properly. The weird thing is that I only run 2 jobs at once (in an attempt to reduce the original Vips::Error:VipsForeignLoad issue) so at maximum there should only be 2 8MB images being loaded through the pipeline at anytime.

It's a shame that it happened overnight so I can't see what the state was at the time, but feels like it might be related. I really don't want to have to switch back to mini_magick...

Jessie, did you get anywhere with the dummy project to replicate the Vips::Error:VipsForeignLoad issue?

Thanks,

Dave

[Jess Latham]

Hi Dave and Janko,

I had a chance yesterday evening to try both approaches listed but sadly no luck.

1) Replace io.download with manual solution 'original = Tempfile.new(["shrine", ".#{io.extension}"], binmode: true)`

2) Downgrade to Down 4.7.0

Interesting to note for our discussion here, I was able to reproduce this issue using local file system storage (so no S3 in play) where I again received a "Vips::Error (VipsForeignLoad "/tmp/shrine20190718-17554-1hw1uk6.png" is not a known file format)"

Does this help at all?

And apologies as at some point we ended up disconnecting the conversation here from the google group.

Thanks,

Jess

On Mon, Jul 15, 2019 at 2:42 PM Dave Harris <da...@harris.org.nz> wrote:

Hi Jess,

No sorry I haven't had a chance, I've been kinda swamped in other areas. I'm starting to raise my head above the parapet this week though so will make time this week and give it a go.

Dave

On Sun, 14 Jul 2019 at 08:08, Jess Latham <jesse....@gmail.com> wrote:

Hi Dave,

Just checking in to see if you've made any headway here with Janko's suggestion of replacing io.download?

Cheers,

Jess

On Wed, Jun 19, 2019 at 4:13 PM Dave Harris <da...@harris.org.nz> wrote:

Hi Jess,

Yeah replacing io.download with the code above is something that I can do relatively easily. I still don't know how it's triggered so I will just have to wait for a busy time and see if it happens again.

I should probably add the delete_raw plugin too. That might explain the reason why it runs out of hard drive space.

I'll let you know how I get on.

Dave

On Thu, 20 Jun 2019 at 05:36, Jess Latham <jesse....@gmail.com> wrote:

Thanks for the insight @Janko Marohnić!  It's something approachable that I could try here as well.

Dave - what do you think?  Anything ringing a bell here?

Cheers guys,

Jess

On Sat, Jun 15, 2019 at 2:53 PM Janko Marohnić <janko.m...@gmail.com> wrote:

Thanks for all the details, sorry for the late response.

First of all, the VipsForeignLoad error does sound like the file could be corrupted. When Shrine downloads the file to disk, it closes the file descriptor to force a flush and then opens it again. So, from the perspective of IO system buffers we should be good.

However, S3#open (which Shrine::UploadedFile#download eventually calls) internally uses Down::ChunkedIO to wrap S3's streaming, which could theoretically have a bug, or maybe just in conjunction with aws-sdk-s3. I would recommend trying downloading only with aws-sdk-s3 (avoiding Down::ChunkedIO) and see whether that resolves the issue. So, you could replace `io.download` with:

  process(:store) do |io, _|

    original = Tempfile.new(["shrine", ".#{io.extension}"], binmode: true)

    object = io.storage.object(io.id)

    object.get(response_target: original)

    original.open # refresh file descriptor, which forces flush

    # ...

  end

Alternatively, if you're locking to Down 4.8.0, you could try downgrading to 4.7.0. 4.8.0 introduced a security measure which unlinks Down::ChunkedIO's internal Tempfile. But it should work fine, because I stole that trick from Rack and Unicorn, which are pretty battle tested.

Some other notes from the discussion:

• `io.download` with a block ensures the Tempfile is deleted even in case of exceptions
• without `delete_raw` plugin the processed files won't get deleted after uploading (Ruby's GC should theoretically delete Tempfiles which aren't used anymore, but it's known to take its sweet time)
• `store_dimensions` plugin swallows any Vips::Error exceptions, so it could be that dimensions are blank because a Vips::Error has happened (unfortunately Shrine doesn't log these exceptions yet, but it's in my TODO list)

Kind regards,

Janko

On 11 Jun 2019, 00:50 +0200, Jess Latham <jesse....@gmail.com>, wrote:

Hey Dave,

Sorry man, apologies for not getting back to you sooner.  Thanks for bubbling back up and the extra details you've just provided.

I had not yet had a chance to put together a dummy project to try and repro the issue here.  Unfortunately got a small team and I'm trying to help move some big pieces through at the moment and so don't have spare cycles available to properly experiment and put together a production-level test case. :(

For my own experience it's really interesting that on my side that I have two separate products/environments and they're behaving differently due to some possibly low level configuration (one successful and one unsuccessful).

I wonder if @Janko Marohnić  has any insights here to what we're experiencing? Maybe some interesting bits had come up during implementation on his side?

Maybe we'll see what he says and regroup afterwards?

Cheers,

Jess

--
You received this message because you are subscribed to a topic in the Google Groups "Shrine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ruby-shrine/KmlmMVO6f-Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ruby-shrine...@googlegroups.com.
To post to this group, send email to ruby-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-shrine/0ccaebb5-1d21-4b76-9950-2b597a83c507%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Janko Marohnić]

Hi Jess,

Thanks a lot for the update. I'm actually kind of relieved that the bug wasn't in the Down gem, or even in aws-sdk-s3. Even though your problem still exists, at least we were able to discard some possibilities.

If I remember correctly, you said that when the job is retried, it eventually succeeds? That means that the original file from the storage is not corrupted, but I guess it gets corrupted after it's downloaded to disk? In that case I'm guessing the content somehow isn't properly flushed to disk.

In that case I still have one more idea you could try. Maybe Shrine calling `Tempfile#open` at the end in `UploadedFile#download` method to flush the buffer to disk doesn't properly flush it. So I would try the following (assuming you're still using FileSystem storage and not S3):

  process(:store) do |io, _|
    original = Tempfile.new(["shrine", ".#{io.extension}"], binmode: true)

    io.open do

      IO.copy_stream(io, original)

      original.fsync # this should flush for sure

      original.rewind

    end

    # ...
  end

While searching more for the behaviour of `IO#close`, `IO#flush` and `IO#fsync`, I came across this StackOverflow answer. It suggests that `IO#close` doesn't in fact imply that `fsync()` will be called (although it appears to imply IO#flush, which would make sense, as if it didn't there would be errors much sooner).

If you try the above and it works, I will add it to Shrine. Just to make sure, you were calling regular `UploadedFile#download` before?

Kind regards,
Janko

You received this message because you are subscribed to the Google Groups "Shrine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-shrine...@googlegroups.com.

To post to this group, send email to ruby-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-shrine/CAPRTHQgOaU6vLXszkdjZPnM%2B7cxnufyJ%3DMX_OE0wioBks6pLbw%40mail.gmail.com.

[Jess Latham]

Hi Janko,

So, bad news/good news/bad news.

Bad news. 

The suggestion you provided didn't help - I'm still receiving an issue from libvips relating to foreignload.

Good news. 

Digging in further on this today, using the original code below

-------------

io.download do |original|
      pipeline = ImageProcessing::Vips.source(original)

      versions[:medium] = pipeline.resize_to_limit!(700, 700)
      versions[:small] = pipeline.resize_to_limit!(400, 400)
      versions[:thumb] = pipeline.resize_to_limit!(200, 200)
      versions[:square] = pipeline.resize_to_fill!(100, 100)

end

-------------

and trying to focus more on libvips (8.8.0) on the box.  

I still receive the error "Vips::Error (VipsForeignLoad: "/tmp/shrine20190720-6104-v4mgzk.jpg" is not a known file format)" when uploading an image.

Made the app fail so that I have a `shrine20190720-6104-zbi7co.jpg` image in my tmp folder.  I've downloaded this off the server and confirmed that it's valid (I can open it locally).  I also tried a PNG for good measure - same results.  Image was successfully downloaded.

The next step here was to try and perform a simple vips operation on it with vips as `vips rot shrine20190720-6104-zbi7co.jpg shrine20190720-6104-zbi7co-new.jpg d90`.  Well, guess what.  I receive the same stupid error of Vips::Error (VipsForeignLoad: "/tmp/shrine20190720-6104-v4mgzk.jpg" is not a known file format).  I did the same for an uploaded PNG file.  So now I'm investigating what has happened since my update of libvips 8.6.4 to 8.8.0.  

To summarize the good news here - Shrine appears to be working correctly.

Bad news:

From the above, this looks to be an issue with my configuration for some reason.

Not to be too noisy here, and I understand that this is looking more and more like a libvips/my configuration issue, here is the output of ./configure when attempting to start a libvips build process from start/source (which is how I've been installing).  Interesting to note that PNG and JPEG support both seem to be found and included.  Note that ImageMagick or MagickCore couldn't be found so there's no fallback there to those libraries providing support for PNG/JPEG/others.

That said, maybe either of you have an idea of why libvips can't process PNGs or JPEGs currently from the output of configure (remember vips 8.8.0 on Ubuntu 16.04).

Thanks guys.  I'll keep you updated on what I find/am able to work through myself here.

Jess

Output of ./configure

------------------

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for gobject-introspection... no
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for needs -lstdc++... yes
checking for native Win32... no
checking for binary open needed... no
checking for Mac OS X... no
checking whether make supports the include directive... yes (GNU style)
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for gawk... (cached) gawk
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) gcc3
checking for gcc option to accept ISO C99... none needed
checking for gcc option to accept ISO Standard C... (cached) none needed
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking for ranlib... ranlib
checking whether ln -s works... yes
checking if malloc debugging is wanted... no
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking for LC_MESSAGES... yes
checking for CFPreferencesCopyAppValue... no
checking for CFLocaleCopyCurrent... no
checking libintl.h usability... yes
checking libintl.h presence... yes
checking for libintl.h... yes
checking for ngettext in libc... yes
checking for dgettext in libc... yes
checking for bind_textdomain_codeset... yes
checking for msgfmt... no
checking for gcc version... 5.4.0
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking for ANSI C header files... (cached) yes
checking errno.h usability... yes
checking errno.h presence... yes
checking for errno.h... yes
checking math.h usability... yes
checking math.h presence... yes
checking for math.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking sys/file.h usability... yes
checking sys/file.h presence... yes
checking for sys/file.h... yes
checking sys/ioctl.h usability... yes
checking sys/ioctl.h presence... yes
checking for sys/ioctl.h... yes
checking sys/param.h usability... yes
checking sys/param.h presence... yes
checking for sys/param.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking sys/mman.h usability... yes
checking sys/mman.h presence... yes
checking for sys/mman.h... yes
checking for sys/types.h... (cached) yes
checking for sys/stat.h... (cached) yes
checking for unistd.h... (cached) yes
checking io.h usability... no
checking io.h presence... no
checking for io.h... no
checking direct.h usability... no
checking direct.h presence... no
checking for direct.h... no
checking windows.h usability... no
checking windows.h presence... no
checking for windows.h... no
checking for dllwrap... no
checking for dlltool... dlltool
checking for objdump... objdump
checking for ranlib... (cached) ranlib
checking for strip... strip
checking for ar... ar
checking for as... as
checking for ld... ld
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for fgrep... /bin/grep -F
checking for ld used by gcc... ld
checking if the linker (ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for ld option to reload object files... -r
checking for objdump... (cached) objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... (cached) dlltool
checking how to associate runtime and link libraries... printf %s\n
checking for archiver @FILE support... @
checking for strip... (cached) strip
checking for ranlib... (cached) ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... mt
checking if mt is a manifest tool... no
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... ld -m elf_x86_64
checking if the linker (ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking for C/C++ restrict keyword... __restrict
checking for __attribute__((vector_size))... yes
checking for an ANSI C-conforming const... yes
checking for mode_t... yes
checking for off_t... yes
checking for size_t... yes
checking for gcc with working vector support... no
checking for working memcmp... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... (cached) yes
checking for getpagesize... yes
checking for working mmap... yes
checking for vprintf... yes
checking for _doprnt... no
checking for getcwd... yes
checking for gettimeofday... yes
checking for getwd... yes
checking for memset... yes
checking for munmap... yes
checking for putenv... yes
checking for realpath... yes
checking for strcasecmp... yes
checking for strchr... yes
checking for strcspn... yes
checking for strdup... yes
checking for strerror... yes
checking for strrchr... yes
checking for strspn... yes
checking for vsnprintf... yes
checking for realpath... (cached) yes
checking for mkstemp... yes
checking for mktemp... yes
checking for random... yes
checking for rand... yes
checking for sysconf... yes
checking for atexit... yes
checking for cbrt in -lm... yes
checking for hypot in -lm... yes
checking for atan2 in -lm... yes
checking for pthread_setattr_default_np in -lpthread... yes
checking for REQUIRED... yes
checking for BASE64_ENCODE... yes
checking for CONTEXT_GET_HELP... yes
checking for MONOTONIC... yes
checking for THREADS... yes
checking for TYPE_INIT... no
checking for WIN32_GET_COMMAND_LINE... yes
checking for STR_TO_ASCII... yes
checking for HAVE_CHECKED_MUL... yes
checking for gtk-doc... no
configure: WARNING:
  You will not be able to create source packages with 'make dist'
  because gtk-doc >= 1.14 is not found.
checking for gtkdoc-check... no
checking for gtkdoc-check... no
checking for gtkdoc-rebase... no
checking for gtkdoc-mkpdf... no
checking whether to build gtk-doc documentation... no
checking for GTKDOC_DEPS... yes
checking for XML_ParserCreate in -lexpat... yes
checking expat.h usability... yes
checking expat.h presence... yes
checking for expat.h... yes
checking for GSF... yes
checking for GSF_ZIP64... yes
checking for FFTW... no
configure: WARNING: fftw not found; disabling fftw support
checking for MAGICK_WAND... no
checking for IMAGE_MAGICK... no
configure: WARNING: neither MagickCore nor ImageMagick found; disabling Magick support
checking for ImportImagePixels... no
checking for ImagesToBlob... no
checking for ORC... no
configure: WARNING: orc-0.4.11 or later not found; disabling orc support
checking for LCMS... no
configure: WARNING: lcms2 not found; disabling ICC profile support
checking for OPENEXR... no
configure: WARNING: OpenEXR not found; disabling OpenEXR support
checking for X... no
checking for NIFTI... libraries (none), headers (none)
checking for HEIF... no
configure: WARNING: libheif not found; disabling HEIF support
checking for PDFIUM... libraries (none), headers (none)
checking for POPPLER... no
configure: WARNING: poppler-glib >= 0.16.0 or cairo >= 1.2 not found; disabling PDF load via poppler
checking for RSVG... no
configure: WARNING: librsvg-2.0 >= 2.34.0 or cairo >= 1.2 not found; disabling SVG load via rsvg
checking for ZLIB... yes
checking for OPENSLIDE... no
configure: OpenSlide >= 3.4.0 not found; checking for >= 3.3.0
checking for OPENSLIDE... no
configure: WARNING: OpenSlide >= 3.3.0 not found; disabling virtual slide support
checking for MATIO... no
configure: WARNING: matio not found; disabling matio support
checking for CFITSIO... no
configure: WARNING: cfitsio not found; disabling cfitsio support
checking for LIBWEBP... no
configure: WARNING: libwebp, mux, demux not found; disabling WEBP support
checking for PANGOFT2... no
configure: WARNING: pangoft2 not found; disabling pangoft2 support
checking for TIFF... yes
checking for giflib... libraries (none), headers (none)
configure: WARNING: giflib not found; disabling direct GIF support
checking for PNG... yes
checking for png_set_chunk_malloc_max... no
checking for IMAGEQUANT... no
configure: WARNING: libimagequant not found; disabling 8bpp PNG support
checking for JPEG... yes
checking for jpeg_c_bool_param_supported... no
checking for EXIF... yes
checking exif-data.h usability... yes
checking exif-data.h presence... yes
checking for exif-data.h... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating vips.pc
config.status: creating vips-cpp.pc
config.status: creating Makefile
config.status: creating libvips/include/vips/version.h
config.status: creating libvips/include/Makefile
config.status: creating libvips/include/vips/Makefile
config.status: creating libvips/Makefile
config.status: creating libvips/arithmetic/Makefile
config.status: creating libvips/colour/Makefile
config.status: creating libvips/conversion/Makefile
config.status: creating libvips/convolution/Makefile
config.status: creating libvips/deprecated/Makefile
config.status: creating libvips/foreign/Makefile
config.status: creating libvips/freqfilt/Makefile
config.status: creating libvips/histogram/Makefile
config.status: creating libvips/draw/Makefile
config.status: creating libvips/iofuncs/Makefile
config.status: creating libvips/morphology/Makefile
config.status: creating libvips/mosaicing/Makefile
config.status: creating libvips/create/Makefile
config.status: creating libvips/resample/Makefile
config.status: creating cplusplus/include/Makefile
config.status: creating cplusplus/include/vips/Makefile
config.status: creating cplusplus/Makefile
config.status: creating tools/Makefile
config.status: creating tools/batch_crop
config.status: creating tools/batch_image_convert
config.status: creating tools/batch_rubber_sheet
config.status: creating tools/light_correct
config.status: creating tools/shrink_width
config.status: creating test/Makefile
config.status: creating test/variables.sh
config.status: creating test/test-suite/Makefile
config.status: creating test/test-suite/helpers/Makefile
config.status: creating man/Makefile
config.status: creating doc/Makefile
config.status: creating doc/libvips-docs.xml
config.status: creating po/Makefile.in
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing default-1 commands
config.status: executing libtool commands
* build options
native win32: no
native OS X: no
open files in binary mode: no
enable debug: no
enable deprecated library components: yes
enable docs with gtkdoc: no
gobject introspection: no
enable radiance support: yes
enable analyze support: yes
enable PPM support: yes

* optional dependencies
use fftw3 for FFT: no
Magick package: none
Magick API version: none
load with libMagick: no
save with libMagick: no
accelerate loops with orc: no
  (requires orc-0.4.11 or later)
ICC profile support with lcms: no
file import with niftiio: no
file import with libheif: no
file import with OpenEXR: no
file import with OpenSlide: no
  (requires openslide-3.3.0 or later)
file import with matio: no
PDF import with PDFium no
PDF import with poppler-glib: no
  (requires poppler-glib 0.16.0 or later)
SVG import with librsvg-2.0: no
  (requires librsvg-2.0 2.34.0 or later)
zlib: yes
file import with cfitsio: no
file import/export with libwebp: no
  (requires libwebp, libwebpmux, libwebpdemux 0.5.0 or later)
text rendering with pangoft2: no
file import/export with libpng: yes (pkg-config libpng >= 1.2.9)
  (requires libpng-1.2.9 or later)
support 8bpp PNG quantisation: no
  (requires libimagequant)
file import/export with libtiff: yes (pkg-config libtiff-4)
file import/export with giflib: no
file import/export with libjpeg: yes (pkg-config)
image pyramid export: yes
  (requires libgsf-1 1.14.26 or later)
use libexif to load/save JPEG metadata: yes

------------------

[Janko Marohnić]

Hi Jess,

Thanks a lot for the details. From what you've said it is starting to look like a libvips issue. I'm not unfortunately familiar with how a correct ./configure output should look like, but now that you have the failing image you can report it on the libvips repository. John Cupitt, the lead maintainer of libvips, is very responsive and super helpful, so I don't doubt you will get a useful answer from him.

I'm curious whether the `vipsheader` command is able to recognize the image:

  $ vipsheader path/to/image.jpg

For a JPEG image I get an output like this:

  path/to/image.jpg: 600x800 uchar, 3 bands, srgb, jpegload

And also whether ImageMagick's `magick identify` is able to recognize the format:

  $ magick identify path/to/image.jpg

Kind regards,
Janko

To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-shrine/CAPRTHQgZf7Bmf5c-LfHpZ6vtnqd8hqxg7WPuS96qkjvy9RH1Bw%40mail.gmail.com.