Firestore adding data from Python. Arrays being added as blobs. Bug? Or I'm I just doing something wrong?

[Cory Wolff]

Hey all,

I seem to be running into trouble when adding data to a collection in Firestore with Python.

This is part of my transfer from RTDB to Firestore. I exported data from RTDB in JSON and I'm looping through the JSON file with Python to add to Firestore.

Now everything else gets added just fine, except when I try to add a list(array) to my collection. They're getting added as binary blobs instead of strings in an array.

It's for a list of tags. My current json field is quite dirty and needs to be cleaned up before being added to Firestore. So for example I have something like this:

"tags": "Group, Inexpensive, Outdoors,"

I need to break up that field by ","

tags.split(",")

Then looping through those results I strip the whitespace and add to a new clean list:

cleaned_tags = []

for t in tags:
   t.strip()
   cleaned_tags.append(t)

Then I try and add it with the rest of my data:

data = {
   "name":name,
   "description":description,
   "tags":cleaned_tags
}

db.collection("ideas").add(data)

This adds the new document with name and description being correct. My tags field will be an array but the items in the array are binary blobs instead of strings.

I've also tried explicitly converting to strings just to see what would happen:

    cleaned_tags.append(str(t))

and

data = {
   "name":name,
   "description":description,
   "tags": str(cleaned_tags)
}

and converting to list:

data = {
   "name":name,
   "description":description,
   "tags": list(cleaned_tags)
}

I've also tried creating a new list and adding the cleaned_tags by index as a test (there could be more than 2 items, just wanted to see what would happen):

new_new_list = [cleaned_tags[0],cleaned_tags[1]]
data = {
   "name":name,
   "description":description,
   "tags": new_new_list
}

and also making sure that they're unicode:

cleaned_tags.append(t.encode("utf-8"))

Every time they're added as binary blobs.

Am I missing something? Or is this some kind of bug? I've been working off of this guide regarding data types and adding data.

[Hiranya Jayathilaka]

Are you on Python 2.7? If so can you try something like the following?

for t in tags:

  cleaned_tags.append(t.strip().decode())

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/eac7840e-306a-4cc3-8205-110f745429e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hiranya Jayathilaka | Software Engineer | h...@google.com | 650-203-0128

[Cory Wolff]

Yes on Python 2.7 and it worked!

Any idea why it's like that? Shouldn't it already be a unicode string?

Thank you!

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/eac7840e-306a-4cc3-8205-110f745429e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hiranya Jayathilaka]

Python 2.7 considers raw byte sequences as strings. So following 2 statements, both return True:

isinstance(b'hello', str)

isinstance('hello', str)

But the Firestore DB engine knows the difference, and would save b'hello' as a binary blob. So you will have to decode()  such byte sequences in your code.

Note that Python 3 has addressed this issue. There only the 2nd statement will return True.

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/7100bb93-de9b-4f89-b9ab-bf1994019a00%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[romenyrr]

Hi Corey, just curious if you have your script available on GitHub? 

[Cory Wolff]

Hey romenyrr,

No I did not add this to Github :/ It was just a one-off script I wrote to move data to Firestore. It's pretty easy though. Just download your RTDB as json and loop through all of the entries and add them to Firestore.

[Sai Kiran]

strip().decode() does not work in Python 3, so we can use decode('utf-8') which works on both 3 and 2.7

for t in tags:

  cleaned_tags.append(t.decode('utf-8'))