GTFS-realtime archiver tools?

[Sean Barbeau]

Hi all,
Is anyone aware of any existing GTFS-realtime open-source archiving tools?

For example, I'd like to point a client app at a GTFS-realtime source to log data from that feed into a long-term database/file.

Thanks,
Sean

[@makar]

Hi Sean,

We have a cloud service that allows GTFS realtime integration and data logging with retrieval API.

http://cloud.doroga.tv

Regards

Evgeni

[Matt Conway]

Hi Sean,

I wrote gtfsrdb (https://github.com/mattwigway/gtfsrdb) a while back; it supports logging GTFS-realtime to various types of databases (Postgres, MySQL, SQLite, &c.). It might be worth a look.

Matt

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To post to this group, send email to gtfs-r...@googlegroups.com.
Visit this group at http://groups.google.com/group/gtfs-realtime?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Barbeau, Sean]

Thanks Matt!  I’ll check it out.

 

Sean

[Sean Barbeau]

Matt,
I got gtfsrdb setup and am logging BART's GTFS-realtime to a sqlite database, as shown in your sample:
---------------
D:\Git Projects\gtfsrdb>gtfsrdb.py -t http://www.bart.gov/dev/gtrtfs/tripupdate.
aspx -d sqlite:///test.db -c
Warning: no alert URL specified, proceeding without alerts
Creating table trip_updates
Creating table alerts
Creating table stop_time_updates
Creating table entity_selectors
Adding 80 trip updates
Adding 80 trip updates
Adding 80 trip updates
Adding 82 trip updates
Adding 82 trip updates
Adding 82 trip updates
Adding 82 trip updates
Adding 82 trip updates
---------------

However, when I connect to and query the database via sqlite command line tool, its not returning any records:
---------------
D:\Git Projects\gtfsrdb>sqlite3 test.db
SQLite version 3.7.11 2012-03-20 11:35:50
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> SELECT * from trip_updates;
sqlite> .tables
alerts             entity_selectors   stop_time_updates  trip_updates
sqlite> SELECT * FROM stop_time_updates;
sqlite> SELECT * FROM stop_time_updates;
sqlite>
-------------

Any ideas?  Am I missing something simple here with sqlite?

Thanks,
Sean

On Tuesday, February 19, 2013 2:01:40 PM UTC-5, Sean Barbeau wrote:

Thanks Matt!  I’ll check it out.

 

Sean

 

 

From: gtfs-r...@googlegroups.com [mailto:gtfs-realtime@googlegroups.com] On Behalf Of Matt Conway
Sent: Tuesday, February 19, 2013 1:37 PM
To: gtfs-r...@googlegroups.com
Subject: Re: [GTFS-realtime] GTFS-realtime archiver tools?

 

Hi Sean,

I wrote gtfsrdb (https://github.com/mattwigway/gtfsrdb) a while back; it supports logging GTFS-realtime to various types of databases (Postgres, MySQL, SQLite, &c.). It might be worth a look.

 

Matt

 

On Feb 18, 2013 2:40 PM, "Sean Barbeau" <bar...@cutr.usf.edu> wrote:

Hi all,
Is anyone aware of any existing GTFS-realtime open-source archiving tools?

For example, I'd like to point a client app at a GTFS-realtime source to log data from that feed into a long-term database/file.

Thanks,
Sean

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtime+unsubscribe@googlegroups.com.

To post to this group, send email to gtfs-r...@googlegroups.com.
Visit this group at http://groups.google.com/group/gtfs-realtime?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtime+unsubscribe@googlegroups.com.

[Matt Conway]

Yep, looks like I accidentally had the commit statement inside the if statement that does alerts, so if you aren't using alerts none of the data was getting committed. I've pushed a fix to this and it also closes the database cleanly on exit.

Sorry about that!

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.

[Barbeau, Sean]

Thanks Matt, looks like that did it!  I’m getting results from the database query now.

 

Sean

 

From: gtfs-r...@googlegroups.com [mailto:gtfs-r...@googlegroups.com] On Behalf Of Matt Conway
Sent: Saturday, March 02, 2013 2:47 AM
To: gtfs-r...@googlegroups.com
Subject: Re: [GTFS-realtime] GTFS-realtime archiver tools?

 

Yep, looks like I accidentally had the commit statement inside the if statement that does alerts, so if you aren't using alerts none of the data was getting committed. I've pushed a fix to this and it also closes the database cleanly on exit.

Sorry about that!

 

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To post to this group, send email to gtfs-r...@googlegroups.com.
Visit this group at http://groups.google.com/group/gtfs-realtime?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To post to this group, send email to gtfs-r...@googlegroups.com.
Visit this group at http://groups.google.com/group/gtfs-realtime?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To post to this group, send email to gtfs-r...@googlegroups.com.
Visit this group at http://groups.google.com/group/gtfs-realtime?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

 

--
You received this message because you are subscribed to a topic in the Google Groups "GTFS-realtime" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gtfs-realtime/RziZFgykRkA/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to gtfs-realtim...@googlegroups.com.

[rakesh.ra...@gmail.com]

Matt -

         I am unable to run the python script and getting the following error. Please let me know how to proceed. Also, let me know what are the pre-requisite to run the program.

D:\Downloads\gtfsrdb-master>python gtfsrdb.py install
Traceback (most recent call last):
  File "gtfsrdb.py", line 24, in <module>
    import gtfs_realtime_pb2
  File "D:\Downloads\gtfsrdb-master\gtfs_realtime_pb2.py", line 3, in <module>
    from google.protobuf import descriptor
ImportError: No module named google.protobuf

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To post to this group, send email to gtfs-r...@googlegroups.com.
Visit this group at http://groups.google.com/group/gtfs-realtime?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.
To post to this group, send email to gtfs-r...@googlegroups.com.
Visit this group at http://groups.google.com/group/gtfs-realtime?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Barbeau, Sean]

Rakesh,

I believe you need to install the protobuf packages.  It’s been a while since I set this up, but here are my notes on this, for the Windows environment:

 

Install Python 2.7 (32bit) and add to Path (also add \Scripts directory to path to get easy_install)

 

Download protobuf win32 binary AND source package:

https://code.google.com/p/protobuf/downloads/detail?name=protoc-2.5.0-win32.zip&can=2&q=

https://code.google.com/p/protobuf/downloads/detail?name=protobuf-2.5.0.zip&can=2&q=

 

run “easy_install protobuf”

 

run “protobuf-2.5.0\python\ez_setup.py -U setuptools“

run “protobuf-2.5.0\python\setup.py build“  ←(Note: you MUST run this and next command from the \python directory)

run “protobuf-2.5.0\python\setup.py install”

then test with “setup.py test”

(see - http://stackoverflow.com/questions/10969149/compilation-needed-for-installing-google-protobuf)

 

And win32 binary to Windows PATH

 

Sean

--
You received this message because you are subscribed to a topic in the Google Groups "GTFS-realtime" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gtfs-realtime/RziZFgykRkA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gtfs-realtim...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/5154bb4a-4338-4aca-8a78-502852bf8e2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pankaj Mankotia]

Hi Matt,

I have done setup for gtfsrdb.I have few questions:

1. At step :-

 test with “setup.py test”

(see - http://stackoverflow.com/questions/10969149/compilation-needed-for-installing-google-protobuf)

I am facing same error again (PFA-Capture1.png)

2. After execution of gtfsrdb.py -a http://realtime.commuterpage.com/rtt/public/utility/gtfsrealtime.aspx/alert -t http://realtime.commuterpage.com/rtt/public/utility/gtfsrealtime.aspx/tripupdate -d mssql+pyodbc://gtfsrdb:cutr12!@@db1.cutr.usf.edu/gtfsrdb_Arlington -c

I am getting another error(PFA Capture.png). Actually I am trying to store Data locally on my machine.

[Pankaj Mankotia]

Hi,

After running below  command 

                                        gtfsrdb.py -t http://www.bart.gov/dev/gtrtfs/tripupdate.aspx -d sqlite:///test.db -c

 I am getting some Exception(PFA). A database is created in sqlite but it is containing only tables.There is no data in Tables. 

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.

To post to this group, send email to gtfs-r...@googlegroups.com.
Visit this group at http://groups.google.com/group/gtfs-realtime?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "GTFS-realtime" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gtfs-realtim...@googlegroups.com.

[Barbeau, Sean]

Looks like:

http://www.bart.gov/dev/gtrtfs/tripupdate.aspx

 

…is broken and is no longer BART’s GTFS-rt endpoint. 

 

As mentioned here:

http://www.bart.gov/schedules/developers/gtfs-realtime

 

…try this URL instead:

http://api.bart.gov/gtfsrt/tripupdate.aspx

 

Sean

--

You received this message because you are subscribed to a topic in the Google Groups "GTFS-realtime" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gtfs-realtime/RziZFgykRkA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gtfs-realtim...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gtfs-realtime/6dc22d9d-726d-470a-99e4-d6b35b97d2ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.