RDM - Train Allocation and Consists

[Marsh Lane]

With regards to the RDM, has anyone taken the Allocations/consist data
(or indeed any other subscriber data set) and got it to automatically
put the files to AWS S3?  We've had a bucket set up on the archive since
two days after the archive product was added, and nothing has yet gone
over to it.  We've tried removing the bucket and adding it back, each
time RDM says it can contact the bucket ok, and we can write files
locally to it, so the permissions must be right.

I've logged as a fault with RDM over a week ago, but had nothing back. 
Just wondered if it was just us, or whether anyone else had been successful?

Cheers

Richie

[Peter Hicks]

Hi Richie

I had a whole heap of trouble with this some months ago. If I gave an IAM user full permissions over S3 and used that on RDM, it would work but would be helluva insecure as the credentials could escape from RDM.

It wasn't clear what the minimal set of permissions was, and there was no 'Test' button to allow checking whether file(s) could be pushed with a new set of permissions assigned to an IAM user. Through trial and error (and some calls with TCS who operate RDM), I found that the minimal permissions are:

s3:ListBucket on the target bucket (e.g. arn:aws:s3:::bucketName)
s3:ListAllMyBuckets on arn:aws:s3:::*
s3:ListObjectsV2 on arn:aws:s3:::bucketName/*
s3:*Object on arn:aws:s3:::bucketName/*

Note that you have to both add a new destination *and* under the 'Data Files' tab, select that destination.



Peter

[Christoper Stafford]

is ListAllMyBuckets really part of the minimal set? I think I'd regard sharing that with a 3rd party as a security incident.

[Peter Hicks]

On Monday, 20 October 2025 at 16:19, 'Christoper Stafford' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

is ListAllMyBuckets really part of the minimal set? I think I'd regard sharing that with a 3rd party as a security incident.

It's not ideal, is it?  The S3 push functionality doesn't seem to work without it though - I built up the policy bit by bit through looking at detailed IAM logs and allowing calls that were denied, as I couldn't get a list of required permissions from TCS.

Peter

[Marsh Lane]

For reference, I must apologise to the RDM team - I said in a previous email that I'd had no response after a week to a support ticket.  It turns out that they did respond, although for some odd reason, it became categorised as junk, so I didnt see it until after they had closed the ticket!  I have responded back again, so will report whatever comes out.  For reference, Peter (I cannot remember if I emailed you off list or not) having set up the policy as you suggested, it is still not delivering to the S3 bucket.

Richie

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/gWj0wRW2L6btmHd77uN3SkjNxeFrlFLYSeGyz1o_nGF41jWUK-MyLP2hAK3qXF-3nEqZ3mUhP73lBgzYV_r0WaaUD-gzD2mp61ZdjnuM0ec%3D%40poggs.co.uk.

[Marsh Lane]

Morning all,

I thought it was worth a quick update, following the previous messages.

I had a call with RDMs technology supplier, TCS, this morning to try and
resolve this problem, where the TAAC archive files that Peter uploads
each day, were not going to S3, despite a successful validation.  It has
been confirmed the issue appears to lie within the RDM side, as they
have been able to manually upload using the access details, but the
system is not automatically dealing with the upload.  That is now being
investigated internally, having checked that the IAM/S3 Bucket Policies
are correct.

I would offer thanks to Myles at RDM for her help/support and keenness
in resolving the problem, and that of her colleague, Vaibhav, on the
technical side.

Richie


On 18/10/2025 16:31, 'Peter Hicks' via A gathering place for the Open

[Peter Hicks]

On Wednesday, 29 October 2025 at 12:30, Marsh Lane <marsh...@rail.photos> wrote:

> I had a call with RDMs technology supplier, TCS, this morning to try and
> resolve this problem, where the TAAC archive files that Peter uploads
> each day, were not going to S3, despite a successful validation. It has
> been confirmed the issue appears to lie within the RDM side, as they
> have been able to manually upload using the access details, but the
> system is not automatically dealing with the upload. That is now being
> investigated internally, having checked that the IAM/S3 Bucket Policies
> are correct.

Something appears to have changed recently, because I noticed yesterday that I'm no longer receiving any data products via S3 as of some time in mid-September.

I'll raise my own support ticket...!

Best wishes,


Peter