CIF Schedule transaction_type = delete

[Andrew Powell]

Hi

I have been running the CIF importer for a good few months now. I have a large amount of schedule entries that I believe are no longer required within the database, this also means I have the same for schedule_segment and schedule_location data. In the locations I have 23 millions rows.

I wanted to ask when a consist is pulled in and marked as deleted, Would it be ok to remove this schedule and associated schedule_location's?

For example from 1.3 millions schedule entries I have 300,000 of them that have been marked as delete.

Many thanks

Andrew

[Phil Wieland]

The answer to this depends on what you want to do with the data. 

For trains running today and in the future, the schedules marked as deleted can be removed from your database, and you should also note that you don't get a delete transaction for schedules which have expired (i.e. schedule_end_date is in the past), so you can delete those as well.

However, if you want to do any historical analysis of train running, then you need to keep deleted schedules, recording when they were deleted, so that if you are looking at data from before the deletion, the schedule is still valid.

I hope that make some kind of sense.

My database retains records for the last 400 days, and currently has about 45 million schedule location records.

Phil