Empty Search result from previously Indexed Product set

[ifedapo olarewaju]

Hi,

When using the Product Search API, I have noticed that a previously indexed product-set will sometimes return an empty result with the following error:

error {
  code: 5
  message: "No matching products found. Please verify that the ProductSet exists, has images and has been indexed."
}

The pattern I'm noticing is that, if I haven't searched with that product-set for some days, the next time I search with it, it will return the empty result above. If I fetch the product-set to get the index info, the indexTime returns 0.

However, after about an hour, it will start to work again, and the indexTime would have been updated to a more recent time.

Is this a known issue? Do indexes become stale and deleted after remaining unused for a period? This behaviour makes the Product Search API erratic as I can't reliably search for products.

Could anyone share some insights or some workaround regarding this?

[Olusayo Akinlaja]

Hello,

As you seemed to have guessed, the error[0] often indicates that it is either the product set has not be done properly, hence it is not searchable OR you are searching for the product without waiting for the indexing to happen.

As highlighted in this doc[1], If the product set has NEVER been indexed, then the output timestamp is the default value "1970-01-01T00:00:00Z". The indextime should regularly be updated to reflect a recent time that the product set was last indexed[2].

In your case, you explained that your product sets were once previously indexed but because you hadn't searched for them in a while, the timestamp is now set to "0".  When the timestamp is set to "0", it also suggests that the product set hasn't been indexed yet[3]. It is possible that there was an error with indexing the product set at some point in time. Did you check the indexError[4] Status to find out if there was any error reported while trying to index the product set?

The Product Search index of products is updated approximately every 30 minutes[5].

[0]No matching products found. Please verify that the ProductSet exists, has images and has been indexed."
[1]https://cloud.google.com/vision/product-search/docs/reference/rest/v1/projects.locations.productSets#ProductSet.FIELDS.index_time
[2]https://stackoverflow.com/questions/59713331/vision-product-search-indexing-time#answer-59725104
[3]https://stackoverflow.com/questions/52257269/product-search-api-no-matching-products-caused-by-index-issue#answer-55770947
[4]https://cloud.google.com/vision/product-search/docs/reference/rest/v1/projects.locations.productSets#ProductSet.FIELDS.index_error
[5]https://cloud.google.com/vision/product-search/docs/tutorial

[ifedapo olarewaju]

Thank you for your response Sayo.

So from what I've noticed, the Product-Set HAS been indexed before without errors. In fact, it has been used for product searches already. It only seems like after a search hasn't been executed for some days, its index is no longer available. When I check for the product-set info, I get this:

>>> pset.index_time
DatetimeWithNanoseconds(1970, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
>>> pset.index_error

>>> 

Hence, there is no indexError, and the index time has fallen back to 1970. But I must re-iterate that this Product-set has previously been indexed and used for searches. Also, after one failed search attempt, the product-set index then becomes available within an hour. So this tells me that running a search after some days triggers it to be indexed in the next indexing cycle. This behaviour however is not reliable and makes the service seem somewhat erratic.

Is this a known, documented behaviour? Is there a way to avoid it? Or could there be an isolated Google vision operation that is triggering this deletion of indexes?

[Carlos Clement Bellido]

Hello

A possible cause for this error is that you could be exceeding the ProductSet maximum reference images [1-2]; this limits to a million reference images per ProductSet and, if this happens, the periodic indexing could fail. Additionally, the GitHub repository [3-4] and the Java documentation [5] also mentions that “[...] the total number of distinct product_labels over all products in one ProductSet cannot exceed 1M, otherwise the product search pipeline will refuse to work for that ProductSet”.

[1]: https://github.com/googleapis/googleapis/blob/73da6697f598f1ba30618924936a59f8e457ec89/google/cloud/vision/v1/product_search_service.proto#L420

[2]: https://cloud.google.com/vision/docs/reference/rest/v1/projects.locations.productSets#resource:-productset 

[3]: https://github.com/googleapis/googleapis/blob/73da6697f598f1ba30618924936a59f8e457ec89/google/cloud/vision/v1/product_search_service.proto#L414

[4] ProductLabels: https://developers.google.com/resources/api-libraries/documentation/vision/v1/csharp/latest/classGoogle_1_1Apis_1_1Vision_1_1v1_1_1Data_1_1Product.html#a627ad27124e2dbf06c961ae2441fcedb

[5]: https://cloud.google.com/java/docs/reference/google-cloud-vision/latest/com.google.cloud.vision.v1p4beta1.Product.Builder

[ifedapo olarewaju]

Thank you for your response.

In this case, the reference images within the product-set are about 2000 (which is well under 1M).

Also keep in mind that after I try to do a search once, it then gets indexed in the next cycle.

Meaning that it CAN get indexed, It just doesn't get indexed seemingly because the product set hadn't been used for a while.

[Carlos Clement Bellido]

It’s possible that you’re hitting some type of limitation [1] that affects the indexation. Do you know the time a product gets un-indexed because of not searching for it? Is it periodical? E.g.: “every 7 days that a product hasn’t been searched it gets un-indexed” or the time before getting un-indexed varies?

Regarding the limitations, I don’t have the tools to look into your project. Perhaps, if this issue is affecting you drastically, you could open a support case and a colleague will look deeply into your project to see if there is something wrong.

[1]: https://cloud.google.com/vision/product-search/quotas 

[2]: https://support.google.com/cloud/answer/6282346?hl=en

[Côme Schaeffer]

Indeed, I have the same issue, product sets used to work and get "unindexed" after some time.

I confirm that I there is no index error and I am well below current quotas limits...

[Kun-Da Li]

I have the same issue.
These are same product sets get indexed without error, but after some time if not queried, product sets get "unindexed".

Is it a normal feature of the API? if so where can I read more about it?