How to write into S3 using fixed filename

jayesh wasnik

unread,

Oct 12, 2021, 2:58:47 AM10/12/21

to sdc-user

Hi,

I am using " Amazon S3" Destination to write some file into S3 everyday.
I am putting the same JSON file into dated folders,so everyday this file would be placed in the respective date folder.
But the name of file is different each day,as in the time basis field I have added ${time:now()} so this gets appended to the name of the file.

Can you please help me in removing this timestamp value,or to find a way to write the file with same name everyday into separate folder.

I have attached a screenshot of my configurations too.

jayesh wasnik

unread,

Oct 12, 2021, 3:01:13 AM10/12/21

to sdc-user, jayesh wasnik

Sujata hs

unread,

Feb 16, 2022, 9:58:41 AM2/16/22

to sdc-user, jayeshw...@gmail.com

Hey Jayesh,

You can use 2 pipeline to achieve this.

1. first write original pipeline data to LocalFS

2. second pipeline - Read from local FS & write to your actual destination as "wholeFileTransfer"

You Can use the required filename in "FileNameExpression" configuration [attaching the screenshot]

localFS_toS3_pipeline.PNG

Reply all

Reply to author

Forward