Unexpected UTF-8 BOM in JSON response

[Albert Filice]

I wanted to use the API with Python, so I did the following:

>>> import requests

>>> agencies = requests.get('http://api.511.org/transit/operators?api_key=TOKENTOKENTOKENTOKENTOKEN')
>>> agencies.json()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/albertfilice/Desktop/511python3venv/lib/python3.6/site-packages/requests/models.py", line 896, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 344, in loads
    s, 0)
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

When I look at just the text from the response, I can see it at the very start...

>>> agencies.text
'\ufeff[{"Id":"5E","Name":"511 Emergency","ShortName":"511 Emergency","SiriOperatorRef":null,"TimeZone":"America/Vancouver","DefaultLanguage":"en","ContactTelephoneNumber":null,"WebSite":null,"PrimaryMode":"other","PrivateCode":"5E","Montiored":false,"OtherModes":""},{"Id":"5F","Name":"511 Flap Sign","ShortName":"511 Flap Sign","SiriOperatorRef":null,"TimeZone":"America/Vancouver","DefaultLanguage":"en","ContactTelephoneNumber":null,"WebSite":null,"PrimaryMode":"other","PrivateCode":"5F","Montiored":false,"OtherModes":""},{"Id":"5O","Name":"511 Operations","ShortName":"511 Emergency","SiriOperatorRef":null,"TimeZone":"America/Vancouver","DefaultLanguage":"en","ContactTelephoneNumber":null,"WebSite":null,"PrimaryMode":"other","PrivateCode":"5O","Montiored":false,"OtherModes":""} ... ... ...

Is that supposed to be there? Why doesn't it just return a clean JSON response?

[Nisar Ahmed]

To get JSON response, you will need to add the format filter in your request, like

 

http://api.511.org/transit/operators?api_key=[your_token]&format=json

 

The default is XML. Hope this helps.

 

--Nisar

--
You received this message because you are subscribed to the Google Groups "511SFBayDeveloperResources" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperre...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Albert Filice]

Thanks for the response Nisar.

I was following the api posted here: https://511.org/developers/list/apis/

It says that JSON is the default format. 

I did still try both, and here were my results:

When I specify JSON as the format I get the same result. 

When I specify XML as the format I get XML. 

Looking at the agencies.text that I posted above should also show that it is JSON but it is prepended with \ufeff which causes problems when trying to parse. 

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperresources+unsub...@googlegroups.com.

[Nisar Ahmed]

Please try explicitly specifying encoding to UTF-8 when saving the file. We think Python may be defaulting to a different encoding.

 

--Nisar

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperre...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "511SFBayDeveloperResources" group.

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperre...@googlegroups.com.

[Albert Filice]

I get the list of agencies like so:

>>> import requests

>>> agencies = requests.get('http://api.511.org/transit/operators?api_key=TOKENTOKENTOKENTOKEN&format=JSON')

>>> agencies.encoding

'utf-8'

It seems to show that the encoding is 'utf-8' already. 

Are you referring to something different? Can you show me an example? 

The code I posted is extremely simple, and I'm just trying to get the JSON response with the list of agencies, but I can't seem to parse the JSON. 

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperresources+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "511SFBayDeveloperResources" group.

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperresources+unsub...@googlegroups.com.

[Nisar Ahmed]

Here is what my technical team is telling me:

 

He needs to specify the encoding when creating the file where the JSON response is being saved. I think he has the encoding on the HTTP request.

 

Please ask him to provide the code and I'll give it a shot. (in case his above advice didn’t work)

 

--Nisar

 

From: 511sfbaydevel...@googlegroups.com [mailto:511sfbaydevel...@googlegroups.com] On Behalf Of Albert Filice
Sent: Wednesday, October 10, 2018 10:52 AM
To: 511SFBayDeveloperResources <511sfbaydevel...@googlegroups.com>
Subject: Re: [511SFBayDeveloperResources] Unexpected UTF-8 BOM in JSON response

 

I get the list of agencies like so:

 

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperre...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "511SFBayDeveloperResources" group.

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperre...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "511SFBayDeveloperResources" group.

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperre...@googlegroups.com.

[Tony Laidig]

I had this same problem and fixed it here:

https://github.com/laidig/511-gtfsrt-list/blob/master/main.py#L18

agencyList = json.loads(codecs.decode(agencyList.content, encoding='utf-8-sig'))

Of course, it would be nice if this could be fixed on 511's end-- it's the first time I've encountered it.

[Nisar Ahmed]

Thanks, Tony.

 

We will look into this issue and try to find a fix at our end.

 

--Nisar, 511 SF Bay

 

From: 511sfbaydevel...@googlegroups.com [mailto:511sfbaydevel...@googlegroups.com] On Behalf Of Tony Laidig

Sent: Wednesday, October 10, 2018 9:52 PM
To: 511SFBayDeveloperResources <511sfbaydevel...@googlegroups.com>

--

[Albert Filice]

I am not saving anything to a file. I'm trying to print out the json, and in my script I will be parsing the json for information. Still, not saving anything to any file. 

To replicate my results...

You need the python requests package.

Then open Terminal or Command Prompt and run the following commands:

$ python3

>>> import requests

>>> agencies = requests.get('http://api.511.org/transit/operators?api_key=TOKENTOKENTOKENTOKEN&format=JSON')

>>> agencies.json()

Same code I posted in the first message. If you still don't understand, let me know and I'll try to explain. Maybe nobody on your team has ever used Python before. If that is the case I might need to provide a more detailed explanation. 

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperresources+unsub...@googlegroups.com.

[Albert Filice]

Any update?

On Thursday, October 11, 2018 at 9:18:11 AM UTC-7, nahmed wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to 511sfbaydeveloperresources+unsub...@googlegroups.com.

[Chris Thacker]

I just spent quite a bit of time troubleshooting my usage of the 511.org API due to this Byte Order Mark (BOM) issue. I didn't see it documented anywhere and went down a fairly deep rabbit hole. Also, all the documentation says the default format is JSON but that doesn't appear to be the case.

[Elisabeth Henkens]

I just ran into the same issue - the BOM at the start of the JSON response breaks all built-in decoding of JSON text responses. To get around it I dropped the first character and can parse JSON successfully.

I would recommend the team revisit this and drop the BOM from your end.

On Sunday, October 28, 2018 at 2:27:32 PM UTC-7 Albert Filice wrote: