Signals

[Matt Bauman]

Hello all,

I've been mulling over a new package for some time now.  I'd really like to simplify the handling of time with respect to DSP and a few other packages.  A lot of what I do involves futzing with time: interpolating, filtering, partitioning (breaking things into arrays of parts), and windowing (getting small subsets about some event).  I've always felt that many of these steps could be greatly simplified with a basic type that handles the time nicely for you.

The stats folk have DataFrames, but I think there's something to be gained from an explicit time vector:

- By using a float-range, we have a very definitive source for the sampling rate. No more diff(time[1:2]) or mean(diff(time))

- We can ensure that the time vector is always monotonically increasing in the constructor

- Often my channels are just numbered; it'd be nice to simply refer to them as numbers (and not, e.g., `c1`, `c2`, etc).

- But it is a great package with good support.  Are folks here using DataFrames for this kind of analysis?  I'd be interested in hearing your workflows and experiences.

The Quant guys have TimeSeries and TimeData, where their time is in calendar days.  We're looking at things several orders of magnitude faster.

I've not found any existing packages that do this, so I've started sketching out Signals.jl: https://github.com/mbauman/Signals.jl

A Signal is vector (<: AbstractVector) of channels.  Iteration is done across channels (not along data within the channels), and skipping the time vector.  All channels must have the same length as the time vector.  It's still very basic, but I have the core vector together, with interpolation (using Grid.jl) and some time windowing functionality.

My signals are all biosignals, so that's where my example on the readme comes from, but I think this is something that other fields could use, too.  Are folks interested in this functionality?  Swing by and check out the open issues on things I'm currently pondering.  It'd be great to have some other use cases to help flush out the functionality and API.

I just wanted to alert anyone who may be interested in getting involved to its existence.  If there's demand, I'd very happily transfer it over to the JuliaDSP org.

Best regards,

Matt

[Spencer Russell]

Hi Matt,

Thanks for the work on this, it looks pretty awesome.

I've thought a little bit about this in the context of AudioIO, but don't have anything nailed down yet.

It looks like you have the # of channels and the sample type as type parameters of Signal. This is great for multichannel audio streams, because then i can dispatch on it. For instance, if I'm playing out of a 2-channel audio interface, I can define a playback method for single, double, and multichannel audio to handle the mixing behavior.

Would it make sense also to have the sampling rate also as a type parameter so that we can define methods that auto-resample?

I like the time-base stuff in there. I'm not sure if I'll take a performance hit for it, but I'll try to get around to doing some testing soon.

peace,

s

--
You received this message because you are subscribed to the Google Groups "julia-dsp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to julia-dsp+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Bauman]

Thanks for your thoughts.  Right now the Signal type is extremely flexible via its parameters… maybe too flexible.  I've been thinking about splitting it into a hierarchy of AbstractSignals.  Then we could possibly have a type parameter for the sampling frequency for RegularSignals… but only if its an integer number.   I deal with some systems with crazy frequencies like 24414.0625.  I'll keep it in mind as I keep flushing out the API.

Right now, I want to focus on adding functionality and let that drive the design of the Signal type/types.  It might be helpful to sketch out what an auto-resampling function would look like.

Best,

Matt

[Simon Kornblith]

Hi Matt,

I'd be interested in using this to organize channels in PLX.jl. It also seems like a great starting point for a common API for reading electrophysiological data.

Would you be interested in supporting some kind of piecewise linear time vector in this package? This is necessary for recordings that are sampled at a constant rate but have gaps. I have seen far too many people struggle with windowing such signals in MATLAB, and I suspect we could do much better. I already have an implementation in PLX.jl, but that code predates immutables and FloatRanges and should probably be completely rewritten. I'm happy to contribute this if you think it belongs.

Simon

[Matt Bauman]

On Jul 18, 2014, at 1:34 PM, Simon Kornblith <si...@simonster.com> wrote:

> I'd be interested in using this to organize channels in PLX.jl. It also seems like a great starting point for a common API for reading electrophysiological data.

That is precisely my field of work and my motivation. Just yesterday a post-doc asked me for some help in his windowing code in MATLAB: depending upon the event time (if it landed exactly on a sample point), he would sometimes get both fenceposts of the signal, giving him major troubles with matrix sizes. This stuff is surprisingly difficult to do right.

> Would you be interested in supporting some kind of piecewise linear time vector in this package? This is necessary for recordings that are sampled at a constant rate but have gaps.

We also have a Plexon system. The pause button is completely verboten because of the difficulties it introduces in our analysis. I’ve re-run trials where I accidentally touched the pause button at the end. We also have some very boring, very large files where nothing happens for some time in the middle because we didn’t want to hit pause. So, yes, I think it would be wonderful if we could support that feature natively. I would be very happy to work with you on this.

As an aside: I’ve been meaning to take a look at PLX.jl at some point, too, but for now I’ve been using our custom-built mex’ed Matlab importer and then grabbing the data from there. We have a C++ guru in our lab who, in the words of Plexon support staff, “basically implemented PL2 on his own” with a companion file format that provides a lookup table for all the seek locations. It ends up being about 5% of the size of the plx file itself, and it takes a pass through the file at first to construct it, but then we can pull out just the data we need with random access.

Best,
Matt

[Simon Kornblith]

I made a PiecewiseIncreasingRanges package that should be suitable for efficiently storing timestamps with gaps.

[Matt Bauman]

Very cool.  I've not had much chance to work on Signals (my limited free Julia time has been diverted to projects in Base), but I've been experimenting with a more general format: https://github.com/mbauman/AxisArrays.jl

It's just still pretty sparse in functionality, but I'm hoping to get back to working on it soon. A PiecewiseIncreasingRange should work as an axis without any trouble.

I'm curious - do you have a use case for different step sizes within the same PiecewiseIncreasingRange type?  For my (imagined) uses, I think I'd favor something that enforces a constant step (and maybe even makes sure that each subsequent range falls exactly on a "tick" from the first range — is that how Plexon's pausing works?).

Thanks for sharing it with us!

Matt

[Simon Kornblith]

I don't actually have a use for different step sizes. I'm not entirely positive whether pausing recording would result in the first point of a subsequent range falling on a subsequent tick from the first range. I'd imagine this is the case when the data are digitized at the same rate as they're saved, but in cases where the data are subsampled before saving I'm not sure. In any case, I think it should be easy for users of PiecewiseIncreasingRanges to verify that the appropriate constraints for their application hold, and we could have utility functions for doing so. I made PiecewiseIncreasingRanges more general because it didn't make it any harder to write, and I don't think that imposing additional restrictions would confer much of a performance benefit.

Simon

[Simon Kornblith]

After playing around a little, I'm reconsidering this. Math with rationals seems slow enough to be a possible bottleneck, so I think I may introduce a divisor into PiecewiseLinearRange.

Simon