setting kernel in filter2D

[Francisco Romero]

Hi Bonsai users!

I am trying to filter a video using Bonsai with the Filter2D node but I don't get to set the kernel. I have tried to externalize the property and load a .png image with the proper pixel size to perform the convolution but it does not work. When I try to "add" a member to the Kernel I do not know what I have to input in the value property. 

Has anybody done it before and can help me?

Thanks, 

[goncaloclopes]

Welcome to the forums!

Filter2D expects a 2D rectangular array of floating point numbers. You can write it directly in the property grid using something like "0,1,0;1,0,1;0,1,0". This would be a 3x3 kernel. Basically commas (,) separate columns and semi-colons (;) separate the different rows.

If you want to use an image as kernel, you would need to convert it into a rectangular array first. Ideally you could use ConvertToArray for this, but unfortunately it doesn't support creating rectangular arrays yet. I've put up an issue to solve this for the next version of Bonsai.

In the meantime, if you really need your kernels to come from images, you can use the following PythonTransform:

import clr
clr.AddReference("OpenCV.Net")
from OpenCV.Net import CV, Mat
from System import Array, Single, Type

def getOutputType():
  return Type.GetType("System.Single[,]")

def process(value):
  output = Array.CreateInstance(Single, value.Height, value.Width)
  header = Mat.CreateMatHeader(output)
  CV.Convert(value, header)
  header.Dispose()
  return output

Sorry that it's a bit awkward, but apparently it's non-trivial to specify the type of rectangular 2D arrays in IronPython (will look into it better). Also maybe the conversion of OpenCV matrices and images into rectangular arrays in code could be easier... well, it's good feedback for next versions!

Also, don't forget to convert and normalize your kernel images into floating point, or else the kernel values may not be appropriate for what you want.

Hope this helps,

G

Edit: Fixed order of dimensions in CreateInstance.

[Francisco Romero]

Hi again, I try the Python Transform alone with the Load Image (the image I am loading is a 100x100 pixels .png image I created in Illustrator with a particular pixel size), a Timer and a Combine Latest and it seems to work (the output is a window with "System.Single[,]" inside. However, when I try to send the output of the Python Transform to the externalized Kernel of the Filter2D I cannot save the workflow and it gives the following error. Do you know what am I doing wrong? It seems to me that I cannot send the output of the Python Transform to the Kernel as easily as I would expect. Do I need to define something in the Kernel? Thank you again!

[goncaloclopes]

Hi,

This is actually an issue with the serialization of rectangular arrays. The .NET XML serialization engine doesn't support them, so Filter2D uses a specific workaround to be able to save its configuration successfully. The problem is that this workaround doesn't propagate to the externalized property. In fact, there's another related issue which is that the editor probably shouldn't be saving the value of the externalized property in the first place if you have inputs connected to them. Even more specifically, Filter2D should probably just have an overload to allow you to specify a kernel dynamically.

I've added all these issues to the bug tracker (#385, #384, #383), thanks for pushing the system! Solving any one of them will get this working and I will surely have a solution by the time the next version is released.

In the meantime, though, there's an "easy" workaround. Filter2D is really just a wrapper for the OpenCV function with the same name, so you can just call it from inside a PythonTransform, like so:

import clr
clr.AddReference("OpenCV.Net")

from OpenCV.Net import *

@returns(IplImage)
def process(value):
  image = value.Item1
  kernel = value.Item2
  output = IplImage(image.Size, image.Depth, image.Channels)
  CV.Filter2D(image, output, kernel, Point(-1, -1))
  return output

This transform takes a tuple of two images (the image and the kernel) and applies the convolution. In fact, this will even avoid the need to have the previous transform to convert a matrix into a rectangular array, so in that sense it's even easier. You can use it with something like the following workflow:

Hope this helps and sorry for the trouble.

Cheers,

G

[Francisco Romero]

Thanks again! I will try it asap. In the mean time I have been learning OpenCV with python and I have managed to go a bit further. But I still think that it is worth it to make it works in Bonsai. I let you know my progresses. 

Cheers, 

[goncaloclopes]

Cool. Yeah, let me know if there's additional functionality you need. The next version of Bonsai will be a minor version update (2.2), not just a patch, so it's an opportunity to add in any new features that are currently not available.

Thanks,

G