How to write into S3 using fixed filename

25 views
Skip to first unread message

jayesh wasnik

unread,
Oct 12, 2021, 2:58:47 AM10/12/21
to sdc-user
Hi,

I am using " Amazon S3" Destination to write some file into S3 everyday.
I am putting the same JSON file into dated folders,so everyday this file would be placed in the respective date folder.
But the name of file is different each day,as in the time basis field I have added ${time:now()} so this gets appended to the name of the file.

Can you please help me in removing this timestamp value,or to find a way to write the file with same name everyday into separate folder.

I have attached a screenshot of my configurations too.


jayesh wasnik

unread,
Oct 12, 2021, 3:01:13 AM10/12/21
to sdc-user, jayesh wasnik

Capture231.PNG

Sujata hs

unread,
Feb 16, 2022, 9:58:41 AMFeb 16
to sdc-user, jayeshw...@gmail.com
Hey Jayesh,

You can use 2 pipeline to achieve this.

1. first write original pipeline data to LocalFS
2. second pipeline - Read from local FS & write to your actual destination as "wholeFileTransfer"
    You Can use the  required filename in "FileNameExpression" configuration [attaching the screenshot]

localFS_toS3_pipeline.PNG
Reply all
Reply to author
Forward
0 new messages