Naming Csv Files with FileName Externalized Property

[Nicholas Guilbeault]

Hi,

I am wondering if I have encountered a bug or whether I have encountered a problem with the structure of my workflow. What happens is I create a string to be used as a base file name that I then feed into multiple CsvWriter and VideoWriter nodes using the FileName externalized property for each node. The first time I run the workflow, the names of the csv and video files are correct. However, when I run the workflow a second time, after I have changed the string to produce a new base file name, for some reason the names of the csv files do not update (i.e. the file name is the same as the previous file name). Interestingly, the names of the videos do update, so there is no issue there. The issue is not that the string being passed to the CsvWriter's FileName property is not updated, because I check what the string being passed to the filename is and it is correct. I've also tried implementing a TakeUntil node such that the data being passed to the CsvWriter is only sent when the new FileName has been created. This also does not solve the problem. I am wondering, what is happening here and how can I fix this? Thanks for your help!

Cheers,

Nick

[Gonçalo Lopes]

Hey Nicholas,

Would it be possible to attach the workflow you are currently using? The issue here is that the VideoWriter and CsvWriter nodes use the FileName property at the beginning of the workflow execution. What can happen is that the FileName is updated, but too late, after the nodes have already started writing, so that you see the properties being updated but still the old filenames were used.

The reason why this is happening will depend on the specific workflow structure you are using so it is hard to advise further without seeing the particular construction.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/bonsai-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/d1834731-8928-4c89-a8a0-238a526a3464%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicholas Guilbeault]

Hi Goncalo,

Here is the workflow. I am running bonsai with the --noboot flag because I am currently using the VimbaCapture node.

Cheers,

Nick

On Wednesday, August 2, 2017 at 3:22:03 PM UTC-4, goncaloclopes wrote:

Hey Nicholas,

Would it be possible to attach the workflow you are currently using? The issue here is that the VideoWriter and CsvWriter nodes use the FileName property at the beginning of the workflow execution. What can happen is that the FileName is updated, but too late, after the nodes have already started writing, so that you see the properties being updated but still the old filenames were used.

The reason why this is happening will depend on the specific workflow structure you are using so it is hard to advise further without seeing the particular construction.

On 2 August 2017 at 19:52, Nicholas Guilbeault <ncguil...@gmail.com> wrote:

Hi,

I am wondering if I have encountered a bug or whether I have encountered a problem with the structure of my workflow. What happens is I create a string to be used as a base file name that I then feed into multiple CsvWriter and VideoWriter nodes using the FileName externalized property for each node. The first time I run the workflow, the names of the csv and video files are correct. However, when I run the workflow a second time, after I have changed the string to produce a new base file name, for some reason the names of the csv files do not update (i.e. the file name is the same as the previous file name). Interestingly, the names of the videos do update, so there is no issue there. The issue is not that the string being passed to the CsvWriter's FileName property is not updated, because I check what the string being passed to the filename is and it is correct. I've also tried implementing a TakeUntil node such that the data being passed to the CsvWriter is only sent when the new FileName has been created. This also does not solve the problem. I am wondering, what is happening here and how can I fix this? Thanks for your help!

Cheers,

Nick

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.

[Gonçalo Lopes]

Hey Nicholas,

As I suspected, the issue was branching. In Bonsai, branching has a very specific initialization behavior.

Basically, in order to guarantee that all the branches see the exact same messages no matter what, Bonsai only subscribes to the source after all branches have been initialized. Usually, this leads to the exact same behavior for data processing pipelines, since stuff only happens when inputs start arriving anyway.

However, for filename properties specifically this is not the case, since they need to be available immediately on initialization. If you are feeding the names from independent streams, this is not a problem, since the CsvWriter/VideoWriter node will try to subscribe to the FileName property first. However, if you are branching, it may not get the name immediately on subscription, because the branch is still waiting for everyone to be ready.

I am attaching a modified version of the workflow where I use a ReplaySubject instead of a branch in order to share the prefix. This should ensure that the filename prefix is propagated immediately to each property. The replay feature ensures that all "branches" still see the same message (it gets replayed to all of them), but this way initialization can happen immediately.

I'm sorry if this is confusing, this is one area of the Bonsai language that I am considering heavily for future improvement, but for now hopefully this workaround will work.

P.S.: I have also removed some of the Zip nodes from the workflow. You can add constant suffixes to strings directly in the Add node, without zipping.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/3f57af2f-59c0-4e07-88d8-3e2461813e68%40googlegroups.com.

[Nicholas Guilbeault]

Hey Gancalo,

Thanks so much for your help! This part works great now. On a related note, I have another question. I would like to have a workflow where each time I press a key, a new 30 second trial starts (call this one Trial_1). After the 30 second trial is complete, all of the data from that trial (video and csv files) are saved. When I press the key again, a new trial starts (call this one Trial_2) and after 30 seconds, save the data from the new trial into new files.  Is it possible to stop and save files with CsvWriter and VideoWriter, without having to stop the workflow entirely? Here's what I have so far. Thanks so much for your help!

Cheers,

Nick

[Gonçalo Lopes]

Hey Nicholas,

That sounds very similar to Kensaku's last question, but where you would replace the time of day timers by a 30-second fixed Timer and a KeyDown source.

If you run into the difficulties, would you mind starting a new thread? It helps to organize references for future users. 

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/43c910a9-7f73-45a3-b5f1-b206fe857aa2%40googlegroups.com.

[Nicholas Guilbeault]

Awesome! Got it working now. Thanks again! 

[LJ]

Hello, 

Sorry to ressurect an old post, but is there a simpler way to do this now? Having to write the path and filename as a string everytime is a pain. Is there a way to do the same thing but using the file explorer (where I can just browse and choose the video file I want)? For one file only, not for batch processing.

Thank you,

Luis

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai...@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

[Gonçalo Lopes]

Hi LJ,

Do you mean for selecting the FileName of the video writer? You can just select the VideoWriter node and choose the filename with an explorer dialog by clicking the button next to the FileName property.

Or are you referring to formatting the string from a text node? You can use the new Format operator for much simpler and more flexible formatting operations.

Hope one of these helps.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/864751c6-9435-447f-b59f-7b921c0d783f%40googlegroups.com.

[LJ]

Olá Gonçalo,

Actually, I was asking about something else. As discussed in this thread, if I want to use the loaded file's name (in FileCapture) as the name of a bonsai output file (in my case a .csv) I need to initialize the file name as string so it's immediately available to both File Capture and CSVWriter when bonsai is initialized. See images below:

What I was asking was, if there was a way of actually opening the file browser to select the file name that goes into the string that actually initializes the FileName for both input and output. The reason why I am asking is because it's a pain having to write the full path and file name each time on the String Node, instead of just using the file browser to select a file which carries the desired name (in this case, it's the input file, anyway).

Hope my question makes more sense now.

Thank you,

Luis

sexta-feira, 1 de Novembro de 2019 às 00:30:30 UTC, goncaloclopes escreveu:

Hi LJ,

Do you mean for selecting the FileName of the video writer? You can just select the VideoWriter node and choose the filename with an explorer dialog by clicking the button next to the FileName property.

Or are you referring to formatting the string from a text node? You can use the new Format operator for much simpler and more flexible formatting operations.

Hope one of these helps.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai...@googlegroups.com.

[Gonçalo Lopes]

Hi Luis,

This is exactly the case where you would use a PropertySource :)

Basically, instead of using a String source to define your file name, you want to create a PropertySource from the FileName property of the node. Property sources behave exactly like the built-in type sources (e.g. String, Float, etc), but the editor for the value uses the editor of the original property (e.g. the FileName popup dialog).

You create property sources by right-clicking on the original node (e.g. CsvWriter) and then Create Property Source > FileName.

Hope this helps!

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/b882b02f-295b-45ae-8624-748e07a14c67%40googlegroups.com.