naming output csv files based on incoming video files

[Danbee Kim]

Hi folks, 

I'm trying to batch process some videos, where I'm generating (among other things) a csv file for each video. I want to automatically name each csv file to correspond with the video file that generated it, but I'm having trouble writing the python sink that would do this. I need to get both the FileName and the actual values getting written to the csv into the python sink node, and I can't figure out how to get the FileName in there properly. 

Here's a screenshot of my workflow:

Any advice very much appreciated!

Cheers,

Danbee

[Danbee Kim]

Tried adding a zip and delay, now the error message I get is "The device is not ready". advice for parsing this error?

[Danbee Kim]

lol what does "root element is missing" mean?

[Danbee Kim]

apologies, to properly document, bonsai broke after I added "PythonSelectMany" before my python sink: 

[goncaloclopes]

Hi Danbee,

This is a nice complexity test case!

I'm curious to know why the workflow broke exactly, so if you can attach the workflow file directly to the forum thread, that would be very useful.

Anyway, regarding the goal of passing an external "property" onto a Python script node, one of the current "limitations" of these nodes is that you can't really have proper properties. You can only define the functions that will handle your input types, so any external data that you need to pass to the Python script needs to come through there.

I see you were trying to use the load() and unload() functions to open and close a file handle and you would need a dynamically generated filename at that time. I'm afraid this strategy is currently not possible with the existing implementation of the Python script nodes.

One workaround though, would be to declare the global file handle variable, but initialize it only when the first input to the node arrives. You can then still use unload() to close the handle in case it has been initialized (i.e. is not None). You could then use CombineLatest to pair your input with the desired filename, and you can then use that data inside process() to open the file handle if needed. Probably it would end up being something like this:

import os
import csv
from System.IO import Path

AvgLumFreqs = None

def process(value):
  global AvgLumFreqs
  if AvgLumFreqs is None:
    path = Path.GetFileNameWithoutExtension(value.Item2) + "_AvgLumFreqs.csv"
    AvgLumFreqs = csv.writer(open(path, 'wb'), delimiter = ',')
  AvgLumFreqs.writerow(value)

def unload():
  global AvgLumFreqs
  if AvgLumFreqs is not None:
    AvgLumFreqs.close()
    AvgLumFreqs = None

Hope this helps.

[Danbee Kim]

Thanks for the help! and attached is the workflow file, though I still can't get it to even open...

[goncaloclopes]

Hmm, there is an easy explanation for why the file does not open: it's empty.

I'm curious as to how exactly this happened as Bonsai should not write to the file in case there are XML serialization errors. If you can reproduce the steps that lead to this problem I would be very interested to understand better what happened.

[Danbee Kim]

Hmm ok now the csv file generates and is named correctly, but the values don't get written in, and then bonsai crashes. I changed the python sink to look like this: 

import os

import clr

clr.AddReference("Bonsai.System")

from System.IO import Path

from Bonsai.IO import PathHelper, PathSuffix

import csv

AvgLumFreqs = None

def process(value):

  global AvgLumFreqs

  if AvgLumFreqs is None:

    path = Path.GetFileNameWithoutExtension(value.Item2) + "_AvgLumFreqs.csv"

    AvgLumFreqs = csv.writer(open(path, 'wb'), delimiter = ',')

  AvgLumFreqs.writerow(str(value))

def unload():

  global AvgLumFreqs

  if AvgLumFreqs is not None:

    AvgLumFreqs.close()

    AvgLumFreqs = None

 I tried with both stringifying the value getting written into the csv file and not, and they both make bonsai crash, though making value a str did write to the csv, albeit with each character in its own cell. I played around with the delays in front of the video writer and the csv writer as well, on the guess that having both running synchronously might be crashing everything - no luck unfortunately. here's a screenshot of how it broke:

and attached is the bonsai file. 

[Danbee Kim]

A different strategy, made with the help of George, found in the attached workflow. The only issue with this is that the first csv file it tries to generate needs to be given the filepath manually. could this be tweaked?

[goncaloclopes]

Ok, thanks for test running this, there were a number of issues but I think everything is sorted now.

First, PythonSink's error reporting seems to be broken. I was surprised that you were seeing so many Bonsai crashes; these days it has been relatively robust to such things except on memory failures and other system critical errors. However, it looks like PythonSink has somehow managed to find some loophole to throw an unhandled exception. I've logged an issue on bitbucket to take care of it.

Ok, so what errors were it throwing? If you were to actually scroll up the black console window to the top of the exception stack, it was actually telling you that the csv.writer object has no close() function. This is a perk of python's csv module. It seems that you need to actually keep a reference to the original file handle and close that if you want to shutdown writing cleanly.

There was another problem with the python code, when you called the writerow function. It seems that this python function actually expects you to pass in a sequence of values, and it will not work if you pass in a single value. So you need to modify the function call to actually create a list contaning a single element (your value) that you then pass in to the writer. I guess in this case if you only wanted to write a single value to a file you wouldn't need the csv module anyway and could just use the file handle directly, but anyway, I've left as an example in case more complicated things come up.

I've attached the final workflow. Hope this helps.

[Danbee Kim]

sweet! so does the node SubscribeSubject basically make a property "global"? and it needs to be used in conjunction with an externalize property node?

thank you!!! 

[goncaloclopes]

Actually it's the other way around: SubscribeSubject reads from a "global" variable. There are two nodes that create these variables now, specifically PublishSubject and ReplaySubject. In this case I used ReplaySubject right after the Source1 node in order to store the value of the variable that is then replayed throughout the workflow wherever you need it.

Not only do these variables sometimes make the workflow cleaner in these situations, but they actually also give you a number of guarantees and nice automatic validations that help keep everything tidy.

Glad this worked for you :-)

[Vincent Prevosto]

Hi all,
I was trying to adapt the above workflow to export ROIs booleans from .avi files. The workflow runs, but no file is created, although the correct inputs seem to go to the python node.
This is an example, each line correspond to one frame, and each videoclip has a few thousands of frames:

(C:\Data\Video\tests\PrV77_108__HSCamClips202.avi, ((True, False, False, False), (True, True, False), (False, False, False), (False, False, False), (False, False, False), (False, False, False)))
(C:\Data\Video\tests\PrV77_108__HSCamClips202.avi, ((False, False, False, True), (False, False, False), (False, False, False), (False, False, False), (True, True, False), (False, False, False)))
(C:\Data\Video\tests\PrV77_108__HSCamClips202.avi, ((False, False, False, False), (False, False, False), (False, False, False), (False, False, False), (False, False, False), (False, False, False)))

Here's the python code:

import os
import clr
clr.AddReference("Bonsai.System")
from System.IO import Path
from Bonsai.IO import PathHelper, PathSuffix
import

csv

ROIS = None
writer = None

def process(value):
  global ROIS, writer
  if ROIS is None:
    path = Path.GetFileNameWithoutExtension(value.Item1) + "_ROIs.csv"
    ROIS = open(path, 'wb')
    writer = csv.writer(ROIS)
  writer.writerow([value.Item2])

def unload():
  global ROIS, writer
  if ROIS is not None:
    ROIS.close()
    ROIS = None
    writer = None

The workflow also works if using the CSVwriter Node, but I don't know how to automatically name output files according to the source file.
What am I doing wrong?
Thanks!

[Vincent Prevosto]

Just solved this one - admittedly a simpler case than the one above.
All I needed was to write this Python transform:

@returns(str)
def process(value):
  return value.replace('.avi', '_ROIs.csv')

and use it as follow:

[Edgar Bermudez]

Hi all,

I am also trying to adapt this workflow (many thanks for sharing btw) to quantify some whisker motion in a bunch stimulation trials. I have a single video file for each trial. I have a workflow that analyzes each trial (attached). However, I would like to analyze all in batch, in a similar way you guys are going here. The only difference, is that for each video file that I load, I want to generate a csv file (each corresponding to the video file) with the positions of the whisker.

I have a couple of questions and wonder if you could help me to try to understand how to do it:

1) How can I tell the python sink that I want to write a csv with the filename (or part of it) of the video file that was loaded?

I tried following your examples, but I get it wrong when trying to combine the extracted whisker positions and the filename (nothing is written).

2) How could I combine all the videos analyzed into a single csv file with the whisker positions (num video files (trials) x num of frames x positions (x,y))?

I know is a bit of a stretch after more than two years from this thread but many thanks for your help.

Cheers,
Edgar

[Edgar Bermudez]

Sorry I forgot to attach my failed attempt...

[Gonçalo Lopes]

Hi Edgar,

Sorry for late reply. Running a bit busy but here's a quick fix, I will try to explain later, let me know if you can get it to work.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/968c9377-590a-48c6-b35b-ca61e481e09c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Edgar Bermudez]

Hi Gonzalo,

Thank you for taking the time to look into this. I tried your workflow but I found a problem. The CsvWriter at the end causes an error because the filename is not specified. So I am not sure to make the csvWriter take the filename from the ExternalizeProperty node?

Thanks again,

Edgar

[Gonçalo Lopes]

Hi Edgar,

If you remove the CsvWriter, do the videos actually play through? I am trying to understand if this is a problem with just the CsvWriter, or more generally with the propagation of the filename.

[Mikail Weston]

Hi Goncalo,

I think I've figured out where the bug is:

The file capture and CSVwriter nodes both need to be initialized before first running. Even if I start out with some garbage filename for the File capture node, it automatically changes to the first file in the folder on starting. 

However the CSVwriter is one step behind in the file list, if it starts with the first file, bonsai crashes after the first file finishes as it attempts to make a new CSV file with the same name it has just created.

I hope that makes sense?

Regards,

Mikail

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

[Gonçalo Lopes]

Hi Mikail,

Nice investigative work :)

Yes, both FileCapture and CsvWriter filenames need to be fully initialized as soon as each processing workflow starts.

The problem is in the branching of the initial filename input, as that holds the propagation of values until all branches have initialized (hence defeating the constraint).

The way to work around this limitation of branching is to use a subject such as AsyncSubject to ensure the filename value is stored.

You can then use SubscribeSubject to connect it to both the FileCapture filename and CsvWriter without branching, which should solve the issue.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/28bc4861-679d-415b-8887-6b94a199d4e6%40googlegroups.com.