Re: [bedtools-discuss] groupby and missing values

[Aaron Quinlan]

Hi Daryl,

I think it would be doable to specify a single missing value (e.g., -missing ".") that would be ignored for _all_ columns and _all_ operations.  It would be more complex to specify individual missing values for the various columns upon which one wishes to operate. 

Would others find utility in specifying a common "missing" value?

On Thu, Nov 15, 2012 at 3:11 PM, Daryl Waggott <dwag...@gmail.com> wrote:

Is it possible to use missing values in groupby queries?  For example, any value indicated by a missing value character, say a dot, would be ignored in the calculation of an aggregate function i.e. mode and sum of the following are 1 not dot (. . 1) 

Daryl

[Aaron Quinlan]

Hi Daryl,

I have been giving this some thought and wonder if perhaps it would be better to simply filter out lines (or lines with specific columns) having the "missing" value with things like "grep -vw "\."" or awk '$3 != "."', where the former excludes lines where any column on the line has solely the missing character, and the latter only allows lines where, for example, the 3rd column is not the missing character.

So, you'd do something like:

> cat foo.txt | grep -vw "." | bedtools groupby -g 1,2,3 -c 4 -o sum

Would that satisfy your needs?  If so, I'd be inclined to use this instead so as to not overly complicate groupby.

Thanks,

Aaron

[Daryl Waggott]

Hi Aaron,
That would work for some of the simpler cases but generally I do groupby queries on matrices of data values.  Each row is a mix of missing and values and I'd like to aggregate.  Removing the whole row would be throwing out information.
Daryl

[Aaron Quinlan]

Very good point, thanks Daryl.  It is back on the "to do" list.

- Aaron