API feed is not returning results

[Elan Trybuch]

Hi, I am the developer of a magic mirror module: MMM-nyc-transit, I have been using a module to help transform the api response into json.

I am no longer getting results in the data, and am wondering if its my API key or something else...

this is the URL the module I use hits:

datamine.mta.info/mta_esi.php?key=${API_KEY}&feed_id=318

and this is the module I use that uses the above API url: https://github.com/ericandrewlewis/mta-realtime-subway-departures/

[Underway]

Hello Elan,

That endpoint has been deprecated, I believe. Read about the new endpoint at api.mta.info

Daniel

Underway

https://underway.nyc/download 

--
You received this message because you are subscribed to the Google Groups "mtadeveloperresources" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtadeveloperreso...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtadeveloperresources/fc758e2d-95ab-4099-8de9-16edf33a9df4o%40googlegroups.com.

--

Daniel

Underway

https://itunes.apple.com/app/id1118548566

https://instagram.com/underwayNYC

https://twitter.com/underwayNYC

[Musanov Darkhan]

Thank you, Daniel.

I've been looking on the website for a new endpoint URL and couldn't find it for the life of me. If you have a minute, do you mind sharing a link that has the endpoint?

Have a good day,
~Darkhan

University of Illinois

On Thursday, July 9, 2020 at 11:28:36 AM UTC-4, Underway wrote:

Hello Elan,

That endpoint has been deprecated, I believe. Read about the new endpoint at api.mta.info

Daniel

Underway

https://underway.nyc/download 

On Thu, Jul 9, 2020 at 11:25 AM Elan Trybuch <eitr...@gmail.com> wrote:

Hi, I am the developer of a magic mirror module: MMM-nyc-transit, I have been using a module to help transform the api response into json.

I am no longer getting results in the data, and am wondering if its my API key or something else...

this is the URL the module I use hits:

datamine.mta.info/mta_esi.php?key=${API_KEY}&feed_id=318

and this is the module I use that uses the above API url: https://github.com/ericandrewlewis/mta-realtime-subway-departures/

--
You received this message because you are subscribed to the Google Groups "mtadeveloperresources" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mtadeveloperresources+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mtadeveloperresources/fc758e2d-95ab-4099-8de9-16edf33a9df4o%40googlegroups.com.

[Membership Pertaining To Mark]

Hello Musanov,

To see the listing of new endpoints:

1 -  Go here: https://api.mta.info/#/landing

2 - Sign in

3 - click on 'Subway Feeds'

Afterwards you will be rewarded with all the new URLs as such:

[Evan Wood]

Hello,

I am having the same issue -- I used to call response = requests.get('http://datamine.mta.info/mta_esi.php?key=[API key]&feed_id=1') but this no longer works for me. 

I updated my account and registered for a new key, but when I click on a Subway Realtime Feed (https://api.mta.info/#/subwayRealTimeFeeds), I get the error: {"message":"Forbidden"}. 

How can I download GTFS data now that the old endpoint is deprecated? 

Best,

Evan 

[1.mille....@gmail.com]

Hello Evan,

As referenced here, the header must include the formatted to include a new key-value pairing (being x-api-key: '<API_KEY>')

With regards to the requests library, when inserting this code you see below, simply

calling the get() function and passing a url does NOT modify the header dictionary.

=================================ISSUE REPLICATION=================================

post = requests.get(url='https://api-endpoint.mta.info/Dataservice/mtagtfsfeeds/nyct%2Fgtfs')

print(post.request.headers)
print("Status Code: " + str(post.status_code))
print(post.content)

REFER TO:  PROBLEM.png Attachment

=================================SOLN=================================

When you pass in an authentication parameter, underneath the hood the requests library ingests this object and

uses whatever authentication data contained within the object's fields to build the properly formatted header.

For example, some APIs require generic username and password fields which can be passed to the header using HTTPBasicAuth() class.

The extra hurdle here is that this MTA API requires a custom authentication header entry (being 'x-api-key': 'API_KEY') which can be created

using TokenAuth() wrapper class as such:

post = requests.get(url='https://api-endpoint.mta.info/Dataservice/mtagtfsfeeds/nyct%2Fgtfs', auth=TokenAuth(<API_KEY>))

print(post.request.headers)
print("Status Code: " + str(post.status_code))
print(post.content)

REFER TO:  SOLN.png Attachment

As you can see, the status is a 200 == success.

Refer to this link: https://realpython.com/python-requests/#authentication

for more details on how to setup the custom TokenAuth() class (or whatever name you want to call it).

It's pretty boilerplate so you should get it to work exactly without issue. Just alter 'X-TokenAuth' in the example to 'x-api-key' ... let me know if this helps.

[Evan Wood]

Hello,

Your response was super helpful, I was able to get to your solution! One additional question I had was regarding "post.content". I see that when you enter print(post.content), it gives you a long string ('\nn\n\x031.0....'). Is this the same output as the deprecated version? 

My next step of the code calls:

feed = gtfs_realtime_pb2.FeedMessage()

feed.ParseFromString(post.content)

but I am getting a "DecodeError: Error parsing message". I wonder if something has changed or if I am doing something wrong. Any ideas would be much appreciated! 

Best,

Evan 

[1.mille....@gmail.com]

No problem! 

Yes the output is the same as the deprecated version. 

Prior to the url change,  we had to insert the api key through a query parameter of the old url and now it's passed through the header of the new url but the data is identical.

I have the same 2 lines of code and I've only received the DecodeError when 'post.status_code' != 200.

But if you're saying that you see the binary data through 'post.content' then that should imply you're hitting the API correctly and your status is indeed a 200 == OK.

It's hard to assist without viewing your source code but I suggest you slap on a breakpoint and run in debug mode to find that aha moment. Make sure you inspect

all of your variable values each and every time you step through. Let me know if you're still facing this issue after trying to debug.

Best,

Kes

[Robert Boscacci]

I am also having decode errors after pointing my Arduino code at the new API endpoint (and switching to an HTTP lib that can handle SSL) and getting a 200 response. I have been mucking around trying to figure out if there's something different now about the separation between the response headers and body...no dice so far

[Bill Bernsen]

Echoing Rob above, are you sure nothing's changed about the response body? The data from the deprecated feed used chunked transfer encoding. I don't see that indicated in the headers on the new feed or the CRLFs that would indicate that in the body. I had to write my own HTTP parser for a microcontroller for *reasons* (it's the one that Rob's using) and can confirm there's definitely a difference.

On Tuesday, October 6, 2020 at 10:19:27 AM UTC-4 cinem...@gmail.com wrote:

For a little more context,

My code assumes a "chunked" transfer-encoding. In the response I'm now getting over 443/SSL since the API move, there isn't a transfer-encoding specified and the content-type is "text/plain". I think I'm looking for CRLF characters to separate chunks but not seeing them anymore. 

[Robert Boscacci]

For a little more context, (sorry to repost, just condensing)

Bill's arduino code assumes a "chunked" transfer-encoding. In the response we're now getting over 443/SSL since the API move, there isn't a transfer-encoding specified and the content-type is "text/plain". I think we're looking for CRLF symbols to separate chunks but not seeing them anymore. Instead, all we're seeing are 0x0A hex codes, where there should really be 0x0D 0x0:

I have tried requesting with "Accept-Encoding" set to gzip or br just to see if that would yield something different but non.

On Thursday, October 1, 2020 at 10:44:41 AM UTC-4 Robert Boscacci wrote: