cant import catalog

[muhammad ahmad]

i am trying to import my catalog ( which i uploaded to bucket as json format) but it gives error
{"code":3,"message":"Parsing terminated before end of input.

i copied the data from sample json as given here
but the error is same.
here is my json which i copied from reference website
{ "id": "1234", "category_hierarchies": [ { "categories": [ "athletic wear", "shoes" ] } ], "title": "ABC sneakers" }}
{ "id": "5839", "category_hierarchies": [ { "categories": [ "casual attire", "t-shirts" ] } ], "title": "Crew t-shirt" }}
and here is the error

{"code":3,"message":"Parsing terminated before end of input.\n: \"ABC sneakers\" }}\n ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":1}}]} {"code":3,"message":"Parsing terminated before end of input.\n: \"Crew t-shirt\" }}\n ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":2}}]}

i will really appreciate any help

[Peng Ren]

Hi Muhammad,

Could I ask which (version of) import API you were using? The referenced sample json is for our v1beta1 API. For V2 API, the schema is a bit different so that if you mistakenly used v1 schema for V2 import, the error is expected. BTW, you can find V2 sample json here. Thanks.

Best,

Peng

--
You received this message because you are subscribed to the Google Groups "cloud-recommendations-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-recommendatio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-recommendations-users/392f1bdc-c193-4c3d-aca0-d3e2c5dff915n%40googlegroups.com.

[Peng Ren]

Hi Muhammad,

It turns out there's some issue on the json sample in our documentation. We are already fixing it now.

Could you try this sample for now?

    {
      "id": "1234",
      "categories": "Apparel & Accessories > Shoes",
      "title": "ABC sneakers",
      "description": "Sneakers for the rest of us",
      "attributes": { "vendor": {"text": ["vendor123", "vendor456"]} },
      "language_code": "en",
      "tags": [ "black-friday" ],
      "priceInfo": {"currencyCode": "USD", "price":100, "originalPrice":200, "cost": 50},
      "availableTime": "2020-01-01T03:33:33.000001Z",
      "availableQuantity": "1",
      "uri":"http://foobar",
      "images": [{"uri": "http://foobar/img1", "height": 320, "width": 320 }]
    }

Sorry for the inconvenience.

Thanks,

Peng

On Thu, Feb 11, 2021 at 10:46 AM muhammad ahmad <ahmadman...@gmail.com> wrote:

Thank you so much Peng for your quick response. I am using online portal for data import here I do not know the version. As i am using the GUI version and I have also tried the JSON schema which you mentioned here. But still it is giving error. Below is the JSon i am using now.
{
  "id": "WAG141579D",
  "categories": "Apparel & Accessories > Shoes",
  "category_hierarchies": [ { "categories": ["Subjects", "People"] },{ "categories": ["Subjects", "People", "Famous People"] },{ "categories": ["Styles", "Cultural"] },{ "categories": ["Styles", "Cultural", "Hispanic", "Mexican"] },{ "categories": ["Styles", "Modern"] },{ "categories": ["Styles", "Modern", "Pop"] },{ "categories": ["Subjects", "People", "Women"] },{ "categories": ["Styles", "Cultural", "Hispanic"] },{ "categories": ["Styles", "Modern", "Pop", "Celebrity Pop"] }],
  "title": "Frida Floral I",
  "description": "Sneakers for the rest of us",
  "attributes": { "artist": {"text": ["Regina Moore", "Regina Moore"]} },
  "language_code": "en",
  "tags": [ "black-friday" ],
  "priceInfo": {"currencyCode": "USD", "price":100, "originalPrice":200, "cost": 50},
  "availableTime": "2020-01-01T03:33:33.000001Z",
  "availableQuantity": "1"
}

And again i am getting this error

{"code":3,"message":"Unexpected end of string. Expected an object key or }.\n\n^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":1}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"id\": \"WAG141579D\",\n      ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":2}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"categories\": \"Apparel & Accesso\n              ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":3}}]}
{"code":3,"message":"Parsing terminated before end of input.\nategory_hierarchies\": [ { \"categories\": \n                    ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":4}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"title\": \"Frida Floral I\",\n         ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":5}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"description\": \"Sneakers for the \n               ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":6}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"attributes\": { \"artist\": {\"text\n              ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":7}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"language_code\": \"en\",\n                 ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":8}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"tags\": [ \"black-friday\" ]\n        ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":9}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"priceInfo\": {\"currencyCode\": \"\n             ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":10}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"availableTime\": \"2020-01-01T03:33:\n                 ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":11}}]}
{"code":3,"message":"Parsing terminated before end of input.\n \"availableQuantity\": \"1\"\n                    ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":12}}]}
{"code":3,"message":"Unexpected token.\n}\n^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":13}}]}

I even validated the josn and it is valid but i am still getting issue. Please help me find out the problem.
Best Regards
Ahmad

[Peng Ren]

Hi Muhammad,

We use line-delimited json format so that you need to make sure to have one json object per line. Please try this format:

{"id": "1234","categories": "Apparel & Accessories > Shoes","title": "ABC sneakers","description": "Sneakers for the rest of us","attributes": { "vendor": {"text": ["vendor123", "vendor456"]} },"language_code": "en","tags": [ "black-friday" ],"priceInfo": {"currencyCode": "USD", "price":100, "originalPrice":200, "cost": 50},"availableTime": "2020-01-01T03:33:33.000001Z","availableQuantity": "1","uri":"http://foobar","images": [{"uri": "http://foobar/img1", "height": 320, "width": 320 }]}

Also,please make sure to select "Retail Product Schema" for the schema of data. Please let me know if you still encounter problems with this. Thanks!

Best,

Peng

On Fri, Feb 12, 2021 at 11:02 AM muhammad ahmad <ahmadman...@gmail.com> wrote:

Thank you peng for your help. But problem still persists. i tried this now but still problem, seems like it is not working at all like error is on even 1st line of json which seems illogical.
here is the error again for your reference

{"code":3,"message":"Unexpected end of string. Expected an object key or }.\n\n^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":1}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"id\": \"WAG141579D\",\n      ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":2}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"categories\": \"Apparel & Accesso\n              ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":3}}]}

{"code":3,"message":"Parsing terminated before end of input.\n  \"title\": \"Frida Floral I\",\n         ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":4}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"attributes\": { \"artist\": {\"text\n              ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":5}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"language_code\": \"en\",\n                 ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":6}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"tags\": [ \"black-friday\" ]\n        ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":7}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"priceInfo\": {\"currencyCode\": \"\n             ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":8}}]}
{"code":3,"message":"Parsing terminated before end of input.\n  \"availableTime\": \"2020-01-01T03:33:\n                 ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":9}}]}
{"code":3,"message":"Parsing terminated before end of input.\n \"availableQuantity\": \"1\"\n                    ^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":10}}]}
{"code":3,"message":"Unexpected token.\n}\n^","details":[{"@type":"type.googleapis.com/google.protobuf.Struct","value":{"line_number":11}}]}

[muhammad ahmad]

Thank you so much Peng it worked, i have another query about images. I was wondering how images are handled in AI recommendations. i.e are they treated as these are treated in google vision AI like its features and whats inside picture is actually used in model learnng or training? or is it just for reference?
Best Regards 
Ahmad

[Kuan Liu]

Hi Ahmad, we are not using images to train our recommendation models. In our existing models, product ids, and text information (title, category, categorical features) are found to be more helpful.

We do use image urls (if uploaded by customers) to help evaluate models internally (whether our models recommend the most relevant products for customers). +Yuening Hu +Guangsha Shi 

Hope that helps.

Kuan

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-recommendations-users/CALYnwyeH2mZYMyB5g-bSp_osdgapQB8w8b-W%3DrzLFb-2j8e%3DJA%40mail.gmail.com.

[Peng Ren]

Yes Muhammad, that's correct.

Sorry for the late reply. Just in the future, please ensure to reply all so that other team members could receive your questions as well(in case I miss it sometime :D). Thanks!

On Mon, Feb 22, 2021 at 11:56 AM muhammad ahmad <ahmadman...@gmail.com> wrote:

Thank you for this information. I misunderstood the format for categories. But its not a problem as I can quickly reformat accordingly. But before I do so, I want to be clear about the correct format so perhaps you can help me out with that. So let's assume a product belongs to the following three categories: Subjects > Nature Subjects > Nature > Geode Styles > Watercolor ...the the correct format in the Product Data sheet categoryHierarchies column should be: “Subjects > Nature”, “Subjects > Nature > Geode”, “Styles > Watercolor” Please confirm if I have this right.

[muhammad ahmad]

Sure Peng, Thank you for your response :) just a confusion in your past comment you gave me a sample json to test, it did not have categoryHierarchies so will it come in categoryHierarchies or in "category" field?  and in this format?  { "id": "WAG141579", "categories": "[“Subjects > Nature”],[“Subjects > Nature > Geode”],[“Styles > Watercolor”]", "title": "Frida Floral I"}  2nd question i was uploading some data to test the format how can i delete that. Best Ahmad 

[Peng Ren]

No problem Muhammad.

'categoryHierarchies' field is in our V1 API format and in V2 API we rename it to 'categories' and streamlined the structure. We recommend you to use V2 API.

You can use ProductService.DeleteProduct API call to delete those items. 

Thanks,

Peng

[muhammad ahmad]

Hi i was adding data and i noted it accepts categories in these three format
{ "id": "WAG141582","title":"Florida II", "categories": "Styles, Modern, Abstract , Styles, Modern , Subjects Decorating Ideas , Subjects, Decorating Ideas , Fresh Neutrals"}
{"id": "WAG141580","title":"Florida III", "categories":  [["Styles", "Modern", "Abstract"],["Styles", "Modern"],["Subjects", "Decorating Ideas"],["Subjects", "Decorating Ideas", "Fresh Neutrals"]]}
{"id": "WAG141581","title":"Florida IV", "categories": "[“Subjects > Nature”],[“Subjects > Nature > Geode”],[“Styles > Watercolor”]", "title": "Frida Floral I"}

I was wondering in which format categories will be more useful as i think currently it is taking category just as string so i was wondering how to add multiple categories or hierarchy of categories. I am aslo attaching the screenshot of how data looks like when we upload it in different ways of categories.

2nd question why it is showing that there is only 1 product 
Best regards
Ahmad

[muhammad ahmad]

I am waiting for your reply, any comment will be really appreciated

[muhammad ahmad]

Hi all. My project is stuck due to this information. Please clear this query

[Patrick Hess]

Muhammad,

Either format should be fine as long as you are consistent - hierarchy of categories is probably best though. +Yuening Hu to confirm.

As to the product count in UI, is that updated now?

Cheers,

Patrick

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-recommendations-users/CALYnwyeUm9rMoHxbZrwUQkT0Ddz3xrZbfoL9rOzWqP11SoTtjg%40mail.gmail.com.

[Yuening Hu]

Hi, Muhammad,

For category ingestion, we support multiple category hierarchies for each item, and each level of the category hierarchy may include multiple words.

For the three formats you listed, the second way as below is more aligned with the proto. However, there are some overlapping, ["Styles", "Modern", "Abstract"] and ["Styles", "Modern"],  ["Subjects", "Decorating Ideas"] and ["Subjects", "Decorating Ideas", "Fresh Neutrals"] 

 

{"id": "WAG141580","title":"Florida III", "categories":  [["Styles", "Modern", "Abstract"],["Styles", "Modern"],["Subjects", "Decorating Ideas"],["Subjects", "Decorating Ideas", "Fresh Neutrals"]]}

Does this item have two categories or four categories? If it originally has four categories, what you did sounds good. If it only has two categories, you don't need to expand, just need to do,

{"id": "WAG141580","title":"Florida III", "categories":  [["Styles", "Modern", "Abstract"], ["Subjects", "Decorating Ideas", "Fresh Neutrals"]]}

Hope it helps.

Thanks,

Yuening

[muhammad ahmad]

Thank you for your response. Category Hierarchy is clear to me now, But I cant really delete my old products.This is very complex i cant understand this Product.Service.delete.product . hence i tried this curl But it is giving 403 error of permission. 
Here is my Curl
curl "https://retail.googleapis.com/v2/projects/303001/locations/global/catalogs/default_catalog/branches/0/products/WAG141579"
-H "Accept: application/json"
    -H "Authorization: Bearer ya29.c.Kp8B9gfWZIAiE60KnwL5nQnCNCkj1vg_Wm079Y1dguJuJf1lziGDXafwXEo5acd5mhinlb7BMpJ0GasiueRg2FCausyBEld2IX-pM9XWsTrLTGiF8KZ1ZHyv0Aln3AE5CJXZAuxlkRC9gRK_HDtxC-9v0IwQcwk7GT-Axqvz24lZb8eAX6Shgf-_0jgLlSI9kAWfBJuZLJtYWtP-rDzzuLv2"
    
here is error in response
{
    "error": {
        "code": 403,
        "message": "Permission 'retail.products.get' denied on resource '//retail.googleapis.com/projects/303001/locations/global/catalogs/default_catalog/branches/0/products/WAG141579' (or it may not exist).",
        "status": "PERMISSION_DENIED",
        "details": [{
            "@type": "type.googleapis.com/google.rpc.ErrorInfo",
            "reason": "IAM_PERMISSION_DENIED",
            "domain": "retail.googleapis.com",
            "metadata": {
                "resource": "projects/303001/locations/global/catalogs/default_catalog/branches/0/products/WAG141579",
                "permission": "retail.products.get"
            }
        }]
    }
NOTE. i have created json with this role 

Service Accounts > Service Account Token Creator and there is no option of adding this role. Recommendations AI > Recommendations AI Editor Isnt there any way of deleting products manually just like we import it manually ? :( Waiting for your response Ahmad

[muhammad ahmad]

waiting patiently for your reply

Best Regards

Ahmad

[Patrick Hess]

Can you assign the Retail AI Editor role :) ?

[muhammad ahmad]

Sure I am adding patri...@google.com this as editor. Thank you Patrick

[muhammad ahmad]

@Patrick Hess i have added you now. Please have a look.

[Patrick Hess]

Hi Muhammad,

Actually, I meant can you assign "Retail AI Editor" to the service account you are using, in project 303001? Is 303001 a valid project?

Cheers,
Patrick

[muhammad ahmad]

Hi i have added Retail AI Editor . I have also added you to the project . 
Project Id is nodal-unity-303001  i have also tried with this id and tried with number only of project ID but still getting the same error of  

'{\n  "error": {\n    "code": 403,\n    "message": "Permission \'retail.products.delete\' denied on resource \'//retail.googleapis.com/projects/nodal-unity-303001/locations/global/catalogs/default_catalog/branches/0/products/WAG141579D\' (or it may not exist).",\n    "status": "PERMISSION_DENIED",\n    "details": [\n      {\n        "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n        "reason": "IAM_PERMISSION_DENIED",\n        "domain": "retail.googleapis.com",\n        "metadata": {\n          "permission": "retail.products.delete",\n          "resource": "projects/nodal-unity-303001/locations/global/catalogs/default_catalog/branches/0/products/WAG141579D"\n        }\n      }\n    ]\n  }\n}\n'

Please advise.
Best Regards
Ahmad

[Patrick Hess]

Hi Muhammad,

Do you mind also giving me the "Viewer" role? Thanks.

Can you provide a screenshot of the service account  you are using with the appropriate role assigned? 

Cheers,
Patrick

[Yuening Hu]

Hi, Ahmad,

Would you mind sharing your project_number? I can check from the backend. But  I am not exactly sure how it is displayed on the UI. +Patrick Hess who may know more about this part.

Thanks,

Yuening

On Thu, Mar 11, 2021 at 11:38 AM muhammad ahmad <ahmadman...@gmail.com> wrote:

Hi Yuening,
I have used your suggested format for category hierarchies

{"id": "WAG141580","title":"Florida III", "categories":  [["Styles", "Modern", "Abstract"],["Styles", "Modern"],["Subjects", "Decorating Ideas"],["Subjects", "Decorating Ideas", "Fresh Neutrals"]]}

attached is Screenshot of data

But when i uploaded/imported the data  to AI recommendation, it is getting only Subjects i.e first word from category hierarchy instead of taking all the hierarchies

Moreover it is also showing that there is only 1 product. Why so?
Waiting for your reply

Best Regards
Ahmad

On Thu, Mar 4, 2021 at 5:35 AM Yuening Hu <yn...@google.com> wrote:

[Patrick Hess]

Hi Yuening,

Muhammad ended up resolving the issue in a separate email thread with me. 

Cheers,

Patrick

[Yuening Hu]

Great to know. Thanks, Patrick!

[muhammad ahmad]

Hi Everyone I have only product data and i do not have any user events. As stated i9n FAQs of Recommendations AI

"How does Recommendations AI handle cold starts for new catalog items? For items that do not have any purchase history, we make recommendations based on similar items. In these cases, it is especially important to have good product titles, categories, and descriptions defined in the catalog. For cold-start users (visitors with no history) the models start with the most popular general items, and become more personalized in real time as more user events are received." 

But Now i cant train my model because Create button is disabled and seems like i need user events data 
Please guide me what should i do? @nd question. After training the model Which API i should use to get recommendations All i can find on documentations is manual placement method but no API to call to integrate to website Best Regards Ahmad 

[Patrick Hess]

Hi Muhammad,

Yes you will need user events to train a model - here are docs on this topic: https://cloud.google.com/retail/recommendations-ai/docs/record-events

I would suggest following this tutorial if you find yourself stuck: https://cloud.google.com/retail/recommendations-ai/docs/movie-rec-tutorial. It goes step by step all the way to model training + calling the api.

Let me know if that makes sense.

Cheers,
Patrick

[Muhammad Ahmad]

Thank you so much for your reply. I want to record events of user on my site so i can us ethos data later to train AI recommendations model. I have website developed in django and bootstrap. floatpic.com Can you please recommend me the easiest way to record events? i read there are multiple ways of recording those. I would be grateful if you could recommend me the easiest way of doing this which can be easily used later for training model
Waiting anxiously for your response.
Thanks
Ahmad

[Muhammad Ahmad]

Sir Please guide me 
Thanks

[Patrick Hess]

Hi Muhammad,

Please refer to https://cloud.google.com/retail/recommendations-ai/docs/record-events for advice on ingesting user events.

Thanks,
Patrick