Why does TIFF/PDF document text detection take so long?

[parth....@corecompete.com]

I have a multi-frame TIFF image that I sent to the vision API and I was surprised that the request takes ~25 seconds to complete. The image I used had only 2 frames/pages. 

I tested converting the pages of the TIFF to individual PNG and running normal Vision API requests (in serial because, for some reason, "batch" annotate doesn't support batch requests??)  and recombining the output. I was able to do that in ~15-20 seconds. I didn't check the quality of the output response but I'm curious why there is such a huge difference. 

Am I doing something wrong or is this just expected of the Vision API? What are some ways to get this speedy enough to the point where I can safely call it in a Google Cloud Function?

[Ji Zhu]

Parth:

The offline batch API is not designed to take short running time as the first priority. What is your expected running time that fits your use case?

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-dis...@googlegroups.com.
To post to this group, send email to cloud-visi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/840099f5-1ff8-4bfb-88a1-c7d904635c83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[parth....@corecompete.com]

That makes sense, and I guess the real pain point here is that if I have a large number of mutli-page TIFF/PDFs, there is no way to batch those requests. I'm trying to keep this all >60 seconds in a Cloud function which is why I'm concerned about runtime.

On Wednesday, October 3, 2018 at 1:28:18 AM UTC-4, Ji Zhu wrote:

Parth:

The offline batch API is not designed to take short running time as the first priority. What is your expected running time that fits your use case?

On Tue, Oct 2, 2018 at 7:03 PM <parth....@corecompete.com> wrote:

I have a multi-frame TIFF image that I sent to the vision API and I was surprised that the request takes ~25 seconds to complete. The image I used had only 2 frames/pages. 

I tested converting the pages of the TIFF to individual PNG and running normal Vision API requests (in serial because, for some reason, "batch" annotate doesn't support batch requests??)  and recombining the output. I was able to do that in ~15-20 seconds. I didn't check the quality of the output response but I'm curious why there is such a huge difference. 

Am I doing something wrong or is this just expected of the Vision API? What are some ways to get this speedy enough to the point where I can safely call it in a Google Cloud Function?

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-discuss+unsub...@googlegroups.com.

[Ji Zhu]

Parth:

Instead of reducing the latency of processing each individual PDF/TIFF file, the design of offline batch PDF aims to provide scheduling for a large number of multi-page PDF/TIFF files according to user quota.

So instead of sending us PDF/TIFF files one by one and wait for each one to succeed, the typical way to use it is to send us as many PDF/TIFF files as possible at one time or continuously, track each operation id to 

get the final result of each PDF/TIFF processing.

We will schedule your works according to your batch quota (pages per second).

On Wed, Oct 3, 2018 at 8:31 AM <parth....@corecompete.com> wrote:

That makes sense, and I guess the real pain point here is that if I have a large number of mutli-page TIFF/PDFs, there is no way to batch those requests. I'm trying to keep this all >60 seconds in a Cloud function which is why I'm concerned about runtime.

On Wednesday, October 3, 2018 at 1:28:18 AM UTC-4, Ji Zhu wrote:

Parth:

The offline batch API is not designed to take short running time as the first priority. What is your expected running time that fits your use case?

On Tue, Oct 2, 2018 at 7:03 PM <parth....@corecompete.com> wrote:

I have a multi-frame TIFF image that I sent to the vision API and I was surprised that the request takes ~25 seconds to complete. The image I used had only 2 frames/pages. 

I tested converting the pages of the TIFF to individual PNG and running normal Vision API requests (in serial because, for some reason, "batch" annotate doesn't support batch requests??)  and recombining the output. I was able to do that in ~15-20 seconds. I didn't check the quality of the output response but I'm curious why there is such a huge difference. 

Am I doing something wrong or is this just expected of the Vision API? What are some ways to get this speedy enough to the point where I can safely call it in a Google Cloud Function?

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-dis...@googlegroups.com.

To post to this group, send email to cloud-visi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/840099f5-1ff8-4bfb-88a1-c7d904635c83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-dis...@googlegroups.com.

To post to this group, send email to cloud-visi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/d9f6e8cf-6d85-4f20-a89f-6f0cbef070a9%40googlegroups.com.

[parth....@corecompete.com]

Ok that makes sense, I wasn't clear on that from the documentation. Also, could you elaborate on the "continuous" sending of PDF/TIFF files? Is it possible to dynamically add new requests to an async batch operation?

On Wednesday, October 3, 2018 at 4:53:35 PM UTC-4, Ji Zhu wrote:

Parth:

Instead of reducing the latency of processing each individual PDF/TIFF file, the design of offline batch PDF aims to provide scheduling for a large number of multi-page PDF/TIFF files according to user quota.

So instead of sending us PDF/TIFF files one by one and wait for each one to succeed, the typical way to use it is to send us as many PDF/TIFF files as possible at one time or continuously, track each operation id to 

get the final result of each PDF/TIFF processing.

We will schedule your works according to your batch quota (pages per second).

On Wed, Oct 3, 2018 at 8:31 AM <parth....@corecompete.com> wrote:

That makes sense, and I guess the real pain point here is that if I have a large number of mutli-page TIFF/PDFs, there is no way to batch those requests. I'm trying to keep this all >60 seconds in a Cloud function which is why I'm concerned about runtime.

On Wednesday, October 3, 2018 at 1:28:18 AM UTC-4, Ji Zhu wrote:

Parth:

The offline batch API is not designed to take short running time as the first priority. What is your expected running time that fits your use case?

On Tue, Oct 2, 2018 at 7:03 PM <parth....@corecompete.com> wrote:

I have a multi-frame TIFF image that I sent to the vision API and I was surprised that the request takes ~25 seconds to complete. The image I used had only 2 frames/pages. 

I tested converting the pages of the TIFF to individual PNG and running normal Vision API requests (in serial because, for some reason, "batch" annotate doesn't support batch requests??)  and recombining the output. I was able to do that in ~15-20 seconds. I didn't check the quality of the output response but I'm curious why there is such a huge difference. 

Am I doing something wrong or is this just expected of the Vision API? What are some ways to get this speedy enough to the point where I can safely call it in a Google Cloud Function?

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-visi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/840099f5-1ff8-4bfb-88a1-c7d904635c83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-discuss+unsub...@googlegroups.com.

[Ji Zhu]

Parth:

You can not add new requests to an existing batch operation but you can create as many new requests as you can, before the previous requests finishes. 

The workflow looks like this:

SEND async request 1 -> store operation id

SEND async request 2 -> store operation id

SEND async request 3 -> store operation id

GET result of request 1

GET result of request 2

GET result of request 3

[Jeroen Bobbeldijk]

I understand this is how it's meant to be used. But we don't have the possibility to batch requests. We provide a real-time data extraction service API.

When someone calls the API with a document, they get the response back in the same request. This works great for images, it takes a couple of seconds at most.

We used to convert PDF's to image, and send that to Vision, it worked fine, but the quality wasn't that good, especially for large documents. So we were happy when the PDF/TIFF option was added, but it's really much slower than image extraction. The amount of time it takes can really differ per scan too.

 

I compared it (these times include converting the image and parsing the Vision result): 

PDF converted to PNG using ImageMagick and using Vision AnnotateImage: ~5s (consistently)

PDF using the AsyncBatchAnnotateFiles: between 20s and 50s

It would be great if something like AnnotateImage could be created for PDF's. It doesn't really matter to me whether it accepts raw bytes or a gs:// URL, both are fine to me. I just want it to be faster and have a more consistent speed.

On Wednesday, October 3, 2018 at 10:53:35 PM UTC+2, Ji Zhu wrote:

Parth:

Instead of reducing the latency of processing each individual PDF/TIFF file, the design of offline batch PDF aims to provide scheduling for a large number of multi-page PDF/TIFF files according to user quota.

So instead of sending us PDF/TIFF files one by one and wait for each one to succeed, the typical way to use it is to send us as many PDF/TIFF files as possible at one time or continuously, track each operation id to 

get the final result of each PDF/TIFF processing.

We will schedule your works according to your batch quota (pages per second).

On Wed, Oct 3, 2018 at 8:31 AM <parth....@corecompete.com> wrote:

That makes sense, and I guess the real pain point here is that if I have a large number of mutli-page TIFF/PDFs, there is no way to batch those requests. I'm trying to keep this all >60 seconds in a Cloud function which is why I'm concerned about runtime.

On Wednesday, October 3, 2018 at 1:28:18 AM UTC-4, Ji Zhu wrote:

Parth:

The offline batch API is not designed to take short running time as the first priority. What is your expected running time that fits your use case?

On Tue, Oct 2, 2018 at 7:03 PM <parth....@corecompete.com> wrote:

I have a multi-frame TIFF image that I sent to the vision API and I was surprised that the request takes ~25 seconds to complete. The image I used had only 2 frames/pages. 

I tested converting the pages of the TIFF to individual PNG and running normal Vision API requests (in serial because, for some reason, "batch" annotate doesn't support batch requests??)  and recombining the output. I was able to do that in ~15-20 seconds. I didn't check the quality of the output response but I'm curious why there is such a huge difference. 

Am I doing something wrong or is this just expected of the Vision API? What are some ways to get this speedy enough to the point where I can safely call it in a Google Cloud Function?

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-visi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/840099f5-1ff8-4bfb-88a1-c7d904635c83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-discuss+unsub...@googlegroups.com.

[George Du]

Jeroen, we hear your feedback. We're working on your use case so stay tuned.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-dis...@googlegroups.com.

To post to this group, send email to cloud-visi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/840099f5-1ff8-4bfb-88a1-c7d904635c83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-dis...@googlegroups.com.

To post to this group, send email to cloud-visi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/d9f6e8cf-6d85-4f20-a89f-6f0cbef070a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Cloud Vision Discussion Google Group (cloud-visi...@googlegroups.com) to participate in discussions with other members of the Google Cloud Vision community and the Google Cloud Vision Team.
---
You received this message because you are subscribed to the Google Groups "cloud-vision-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-vision-dis...@googlegroups.com.

To post to this group, send email to cloud-visi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-vision-discuss/4c9a0a07-add7-4c1f-809e-fe1ea09560eb%40googlegroups.com.