Reg: FROM and TO field under C-class messages

[sabeesh senthil]

Hi All,

Myself Sabeesh doing my MSC in Data Science . I am new to this data . I am doing my research with the open rail data railway network feed.co.uk.

Issue:
Under TD Feeds we have TD Feed 

• C-Class Messages • S-Class Messages • Train Describers • TD Berths 

When I try to download a feed I am able to see all the fields in a data frame format  except from , to  and reporting_time Field. 

Can someone please help me to get this issue get resolved for my studies. It will be highly appreciable and it will be very helpful, Thanks in Advance.

Best regards,

Sabeesh. S.

[Peter Hicks]

Hi Sabeesh

That doesn't sound right - do you see the 'description' field in the frame?

Can you post some example code here which we can use to reproduce the problem?

Peter Hicks

Director

OpenTrainTimes Ltd.

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/b8d2fcfd-990a-44b4-b2b8-d4b29a0e9136n%40googlegroups.com.

OpenTrainTimes Ltd. registered in England and Wales, company no. 09504022.

Registered office: Suite 1-3, Hop Exchange, 24 Southwark Street, London SE1 1TY

[sabeesh senthil]

Dear Peter Hicks,

First of all thanks much for your reply. I am using the code which is located in the following GitHub repository.

Python TD network rail feeder project:

https://github.com/KaitaiD/py-network-rail-feeder.

Please find the following sample image of excel csv file which I downloaded using the Python code from above github URL link for TD Feeds. Here we can see the All fields except from , to and reporting message fields which are under C- Class Messages- Link attached Below, Thanks in advance.

https://wiki.openraildata.com/index.php?title=C_Class_Messages 

Best Regards,

Sabeesh .S

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/CAKGkhzb7iAP6vmUrZ99%3DsmZcxKLJPAKBSROg5_yHes2wW%3D9oLg%40mail.gmail.com.

[Peter Hicks]

Hi Sabeesh

The SF_MSG, SG_MSG and SH_MSG messages have an address and data field, and the CA_MSG, CB_MSG and CC_MSG fields have a ‘from’ and ’to’ field.  In your Python code, you have:

td_schema = {
	"MSG": "TEXT",
	"time": "TEXT",
	"area_id": "TEXT",
	"msg_type": "TEXT",
	"address": "INTEGER",
	"data": "INTEGER",
	"descr": "TEXT"
	}

I think you need to put in ‘from’ and ’to’ keys, or perhaps have a different schema for S-Class (SF, SG, SH) and C-Class (CA, CB, CC, CT) messages.

Peter

On 22 May 2021, at 16:29, sabeesh senthil <sabeesh...@gmail.com> wrote:

Dear Peter Hicks,

First of all thanks much for your reply. I am using the code which is located in the following GitHub repository.

Python TD network rail feeder project:

https://github.com/KaitaiD/py-network-rail-feeder.

Please find the following sample image of excel csv file which I downloaded using the Python code from above github URL link for TD Feeds. Here we can see the All fields except from , to and reporting message fields which are under C- Class Messages- Link attached Below, Thanks in advance.

https://wiki.openraildata.com/index.php?title=C_Class_Messages 

<image.png>

Best Regards,

Sabeesh .S

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/CAJ7pmWgsmTmhncw3TqmfetLREoRa%3DBPkwEc7oraA4XkkSCYotg%40mail.gmail.com.

[sabeesh senthil]

Dear Peter hicks,

I have tried the same before  but nothing is populated.Eventhough if i added a from and to field keys to the resepcetive td schema the from and to fields are not populating.I am not sure how to resolve this,Thanks in advance.

Best regards,

Sabeesh ,S

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/B9786EAA-5C4E-4B0B-8075-DF087316C009%40opentraintimes.com.

[sabeesh senthil]

Hi all,

When I try to download the TD FEED ,I am able to see all the messages which is uploaded except the from and to berth fileds. After analysing the data feed I could see that from and to are the reserved key words for sqlite db similar to some other db(database).

so we can resolve this by changing  the schema.replace( "from " '"from"' ,"to" '"to"'). I am sharing here may be its useful for someone.

Also I would like to know whether we have from and to berth location name instead of from and to berth id number .If, yes can someone please tell how and from where they are referencing the data and replacing the from and to berth location to from and to berth id,Thanks in advance.

Example ( from id( 1234)<< -->>from ( from berth location name).

Best regards,

Sabeesh .S

[Peter Hicks]

Hi Sabeesh

The closest you will get to berth locations is by looking at the SMART 'All Berth Steps' file - details at https://wiki.openraildata.com/index.php?title=Reference_Data#SMART:_Berth_Stepping_Data.

This will let you match up a berth to a STANOX - if you use the 'from' berth for departures and 'to' berth from arrivals, you can create a map of which berths are at which locations.  Bear in mind that this won't include every possible berth - the vast majority are between locations used for train movement reporting.

Peter Hicks

Director

OpenTrainTimes Ltd.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/CAJ7pmWhWGugGGeFgrdjVU0i%3DijBxk0tkBMMqfTohHqFtjq68TA%40mail.gmail.com.

[Thomas Wood]

You need to quote your column names if using keywords in them.

sqlite> create table example("to" text, "from" text, "keywords should be quoted" text);
sqlite> insert into example ("to", "from", "keywords should be quoted") values ("0123", "0456", "9M00");
sqlite> select * from example;
0123|0456|9M00

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/CAJ7pmWhWGugGGeFgrdjVU0i%3DijBxk0tkBMMqfTohHqFtjq68TA%40mail.gmail.com.

[sabeesh senthil]

Dear Peter Hicks,

Thank you for the information provided, It really helps, Thanks much.

Best,

Sabeesh S

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/CAKGkhzZb-e%2Bao523bvP0QY%3DGvsw9Jtpea_8H5z0GGE4%2BOXsjqw%40mail.gmail.com.

[sabeesh senthil]

Dear Thomas Wood,

Yes ,I did the same thing which you mentioned in the mail. Anyways thanks for the information which might be useful for the near future, Thanks.

This community is really rocking and the people who are here in this community are really helpful and kind for the freshers like me, Thanks a lot and thanks in advance for my future queries..

Best Regards,

Sabeesh S

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/CAG4WJGrJKDq73s%3DaBr_eCgwuxJPfF2os--m5gMu9O0MM03La_Q%40mail.gmail.com.

[1978mo...@gmail.com]

Hi all,

I am very new to all of this but have come up against this same problem. I am working with the same code schema mentioned higher up in this thread but cannot add "from" or "to"  column names to the TD_schema. I don't fully understand why this is but from reading above it seems that this is an SQLite3 issue? Has anyone used the schema.replace as described above and could you tell me a bit more about how to use this please?

Thanks,

Martin 

[1978mo...@gmail.com]

Hi again,

Further to my previous message - I have been able to add the 'to' and 'from' fields to the table by using the inverted commas around the but do not receive any data from the feed in these fields. I am sure that I am doing something silly! 

Thanks in advance for any help you can offer.

Martin 

[Peter Hicks]

Hi Martin

Have you thought about naming the columns something else, e.g. "to_berth" and "from_berth"?  That is probably the cleanest way to fix this.

Peter Hicks

Director

OpenTrainTimes Ltd.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/397c4175-3093-4a2f-8ddc-d0b51eaff4cdn%40googlegroups.com.

[1978mo...@gmail.com]

Hi Peter,

Thanks for your response. 

I have tried this and cannot pull the data through. When I view the stream data I can see it is all there (as 'from ' and 'to') but it does not get pulled through into my SQLite database.  

This is code for the schema:

td_schema = {
    "MSG": "TEXT",

    "'to'": "TEXT",

    "time": "TEXT",
    "area_id": "TEXT",
    "msg_type": "TEXT",
    "address": "INTEGER",
    "data": "INTEGER",

    "'from'": "TEXT",
    "descr": "TEXT"

I have tried changing the to and from labels as suggested. On each occasion, the columns are added to my database but to not get populated with the data. 

Best Wishes,

Martin

[Peter Hicks]

Hi Martin

> On 6 Jan 2022, at 20:06, 1978mo...@gmail.com <1978mo...@gmail.com> wrote:
>
> I have tried changing the to and from labels as suggested. On each occasion, the columns are added to my database but to not get populated with the data.

You will of course need to change the code that inserts data in to the database. Simply changing the column will create somewhere to put the data, but unless your code knows where to put the data..!

Do you have some sample code you can share here?


Peter



--

[1978mo...@gmail.com]

Hi Peter,

Sorry for the delay in responding (busy with work!!)

I am working with GitHub - KaitaiD/py-network-rail-feeder:

I have had to fix a few minor issues but have no problem running the example without issue. The data is being collected in an SQLite database. I am using SQLitestudio to view the data it but have written some additional code to use the data  - I am hoping to make a mimic panel the Merseyrail network using a Raspberry pi to drive addressable LEDs. 

I currently have the topic set to TD. It is probably my lack of understanding of how the code works! 

Thanks again for any help you can offer.

Martin 

[1978mo...@gmail.com]

Hi all,

I have worked out where I was going wrong! Thanks for your time and help.

Cheers,

Martin 

[Peter Hicks]

Hi Martin

So that the next person can fix this too - can you say here what changes you made or, ideally, submit a patch and PR (pull request) for the original GitHub project?

Don't worry if you're not confident submitting a PR - if you posted the changes you made here, somebody else can submit it on your behalf!

Cheers,

Peter Hicks

Director

OpenTrainTimes Ltd.

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/3a5f6abf-8c1c-4e27-88a3-c7fe03ceffa8n%40googlegroups.com.

[1978mo...@gmail.com]

Hi Peter,

It was based upon this project  GitHub - KaitaiD/py-network-rail-feeder:

but I modified topicmapping.py and listener.py to make it possible amend the incoming 'to' and 'from' headers so that they can be stored in an SQLite database using the column headings  'fromberth' and 'toberth'. (attacjed)

I am sure that this could have been done more efficiently that I have done it but its great to have it working! I can now move on to the next step of my project. 

It is worth noting that you must use a circa 2019 version of stomp.py (as per the original scripts) but I will probably modify the listener to correct this next.

It would be great if someone could help me share this so that others can benefit and improve it.

Regards,

Martin