Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
File transfer for fastness
26 views
Skip to first unread message
t.sathya...@gmail.com
Jun 14, 2020, 9:43:33 AM6/14/20
to
Hi,
Im trying to find best solution for parallel file transfer, which is event driven
after the file written to a directory with extension like .done, means the file is ready for file transfer.
1. Monitoring the directory for .done is cpu cycle time waster
2. running a cron on particular interval to transfer files will not help fasteness to the data.
what are other options existing for file transfer.
Thanks,
Sathya
Im trying to find best solution for parallel file transfer, which is event driven
after the file written to a directory with extension like .done, means the file is ready for file transfer.
1. Monitoring the directory for .done is cpu cycle time waster
2. running a cron on particular interval to transfer files will not help fasteness to the data.
what are other options existing for file transfer.
Thanks,
Sathya
Jorgen Grahn
Jun 14, 2020, 10:39:24 AM6/14/20
to
On Sun, 2020-06-14, t.sathya...@gmail.com wrote:
> Hi,
>
> Im trying to find best solution for parallel file transfer, which is
> event driven after the file written to a directory with extension
> like .done, means the file is ready for file transfer.
>
> 1. Monitoring the directory for .done is cpu cycle time waster
> 2. running a cron on particular interval to transfer files will
> not help fasteness to the data.
I cannot understand what you want to do, or how you want to do it;
> Hi,
>
> Im trying to find best solution for parallel file transfer, which is
> event driven after the file written to a directory with extension
> like .done, means the file is ready for file transfer.
>
> 1. Monitoring the directory for .done is cpu cycle time waster
> 2. running a cron on particular interval to transfer files will
> not help fasteness to the data.
you would have to explain better.
> what are other options existing for file transfer.
http, then I use firefox or curl. Bittorrent is another option.
/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
Grant Edwards
Jun 14, 2020, 9:31:14 PM6/14/20
to
On 2020-06-14, t.sathya...@gmail.com <t.sathya...@gmail.com> wrote:
> Im trying to find best solution for parallel file transfer, which is
> event driven after the file written to a directory with extension
> like .done, means the file is ready for file transfer.
>
> 1. Monitoring the directory for .done is cpu cycle time waster
>
> 2. running a cron on particular interval to transfer files will not
> help fasteness to the data.
https://man7.org/linux/man-pages/man7/inotify.7.html
> Im trying to find best solution for parallel file transfer, which is
> event driven after the file written to a directory with extension
> like .done, means the file is ready for file transfer.
>
> 1. Monitoring the directory for .done is cpu cycle time waster
>
> 2. running a cron on particular interval to transfer files will not
> help fasteness to the data.
https://en.wikipedia.org/wiki/Inotify
Jorgen Grahn
Jun 15, 2020, 4:42:01 AM6/15/20
to
the file system. I'd still like to understand what the OP is trying
to do ...
Sathya Narayanan Thangavelu
Jun 15, 2020, 5:14:17 AM6/15/20
to
we have a distrubuted systems which collects peta bytes of data and deliver the data to netstorage, the problem we have is transmission of files at faster rate, the one problem we sees as soon as the file comes in, we are not delivering the data, we have to wait for cron to run and pick those files and to deliver netstorage, the problem in cron is we schedule a time to run but practically this will lead to big data backlog, my question here is there any event based for monitoring file system to initiate the job as soons as files that comes into the directory, like (epoll mechanism)
Jorgen Grahn
Jun 15, 2020, 7:42:33 AM6/15/20
to
On Mon, 2020-06-15, Sathya Narayanan Thangavelu wrote:
> On Sunday, 14 June 2020 20:09:24 UTC+5:30, Jorgen Grahn wrote:
>> On Sun, 2020-06-14, t.sathya...@gmail.com wrote:
>> > Hi,
>> >
>> > Im trying to find best solution for parallel file transfer, which is
>> > event driven after the file written to a directory with extension
>> > like .done, means the file is ready for file transfer.
>> >
>> > 1. Monitoring the directory for .done is cpu cycle time waster
>> > 2. running a cron on particular interval to transfer files will
>> > not help fasteness to the data.
>>
>> I cannot understand what you want to do, or how you want to do it;
>> you would have to explain better.
>>
>> > what are other options existing for file transfer.
>>
>> Personally I always use rsync over ssh. Unless I download files over
>> http, then I use firefox or curl. Bittorrent is another option.
...
> On Sunday, 14 June 2020 20:09:24 UTC+5:30, Jorgen Grahn wrote:
>> On Sun, 2020-06-14, t.sathya...@gmail.com wrote:
>> > Hi,
>> >
>> > Im trying to find best solution for parallel file transfer, which is
>> > event driven after the file written to a directory with extension
>> > like .done, means the file is ready for file transfer.
>> >
>> > 1. Monitoring the directory for .done is cpu cycle time waster
>> > 2. running a cron on particular interval to transfer files will
>> > not help fasteness to the data.
>>
>> I cannot understand what you want to do, or how you want to do it;
>> you would have to explain better.
>>
>> > what are other options existing for file transfer.
>>
>> Personally I always use rsync over ssh. Unless I download files over
>> http, then I use firefox or curl. Bittorrent is another option.
>
> we have a distrubuted systems which collects peta bytes of data and
> deliver the data to netstorage, the problem we have is transmission
> of files at faster rate, the one problem we sees as soon as the file
> comes in, we are not delivering the data, we have to wait for cron
> to run and pick those files and to deliver netstorage, the problem
> in cron is we schedule a time to run but practically this will lead
> to big data backlog, my question here is there any event based for
> monitoring file system to initiate the job as soons as files that
> comes into the directory, like (epoll mechanism)
I think I see. Yes, that's the Inotify mechanism someone else wrote
> we have a distrubuted systems which collects peta bytes of data and
> deliver the data to netstorage, the problem we have is transmission
> of files at faster rate, the one problem we sees as soon as the file
> comes in, we are not delivering the data, we have to wait for cron
> to run and pick those files and to deliver netstorage, the problem
> in cron is we schedule a time to run but practically this will lead
> to big data backlog, my question here is there any event based for
> monitoring file system to initiate the job as soons as files that
> comes into the directory, like (epoll mechanism)
about.
I think it would be better to avoid that (files appearing in a
directory as an IPC mechanism) but I can see how design decisions can
make it unavoidable.
0 new messages