Uniq muliple duplicate messages from Watch node

[Aaron Estrada]

Does anyone have a good recipe for doing a unique on multiple duplicate messages from the Watch node?

I'm on Linux.

What I'm trying to do is watch a dir for new files to show up and then process the files. Unfortunately, the Watch node is sending multiple messages when new files are copied into the dir (with cp or the file browser).  Using an mv to rename the files into the dir seems atomic and the Watch node only sends one message when I use mv. However,  only being able to use mv puts too many restrictions on how I want to use the flow. Hence my question... Is there a good recipe to accumulate all the messaged and then filter them to become a single unique message?

I'm basically looking for the counterpart to  node-red-contrib-splitter  (message combiner or uniquer)

Thanks!


[Dave C-J]

Hi,

not tried it... but you should be able to use a trigger node - triggered by "all" the messages - but with a long enough timeout to not send it's second output until the copies have finished... - meanwhile also feed "all" the messages into a function node - and also the output from the trigger node. - in the function use the first value of the trigger node message to "enable"  adding all the other messages to an array - as they arrive add them - and then on the second message output from the trigger - send the accumulated array.

[Aaron Estrada]

Thanks Dave. I will give that a try. But wouldn't I then just have an array of all the messages, repeating? Are you suggesting I then run a unique on the array captured by the function node?  I'm not sure I can guess how long it will take for all the messages to arrive. The issue appears to be that Watch is detecting the changes in file size as a new file is copied into the dir. I may be copying very large files into the dir and they may take some time to arrive. What if another file starts copying in just before my time out dumps the last accumulation of messages? Wouldn't I have a sort of race condition where I still end up with two copies of the message naming the file? (part of in in the first array, and part in the second) They would still unique to what look like duplicate messages. This is why I'm so confused how to solve this. It seems like there is no elegant solution short of a smarter Watch node.

The Watch node doesn't seem to have enough options. It notices every single file change. It would be nice if it exposed the option to only send a signal only in the case of specific changes or combination of changes. For my particular use, detecting a file close or MV would be the most useful.

[Jim Williams]

I solved this a while ago by changing the script that was doing the copying. After it had finished copying it created a symlink to the file in another directory and the watch node was set to watch that directory. It then only got one message for the creation of the symlink. In my case the script then waited 10 seconds for Node-red to process the file and then deleted the file and the symlink so it cleaned up after itself.

(See http://jekw.co.uk/2014/01/12/node-red-flow-for-the-robot-arm/ for details)

Jim....

[Aaron Estrada]

Thanks Jim. This is a variation of what I was doing as my work-around externally (using a mv to get the file to the watch dir rather than a cp) I like this solution since it's so simple.