Connecting continuous and sporadic variables
Xt
point. She notes the time and the number of birds in each flock she
sees. Her data looks like this 8:00 start, 8:17 2 birds, 8:47 12
birds, ..., time number ... etc... 14:45 1 bird, 15:00 end. Over the
same time perid, she has an almost continuous record of (for example)
% cloud cover (every 10 minutes, say). Data 8:00 20%, 8:10,
22%, ...time cover ... etc ... 14:50 85%, 15:00 87%. There is data
for several days.
The problem is to test whether or not there is an association between
the number of birds seen per hour and the cloud cover. You can
connect the cloud cover when the birds are seen, but what about the
cover when the birds are not seen?
Any ideas? Derek
Ray Koopman
First, why not interpolate the cloud cover? If the record is truly
almost continuous then the method of interpolation shouldn't matter
much, and linear should suffice.
Second, it's not clear what the question is. If it's the relation
between cloud cover and the number of birds seen then I don't
understand why you think there's a problem. Or do you want to relate
cloud cover to the total number of birds passing overhead, seen plus
unseen?
Xt
Thanks for your comments, Ray. Perhaps I didn't explain where I saw
the problem.
Your first comment - yes, the interpolation is fine.
Your second comment - a simplified version of the problem is deciding
if more birds fly past when it is cloudy than when it is clear.
Instead of continuous cloud cover, assume that you classify the
weather as clear or cloudy at any particular time. Say you record
that 180 birds pass when it is cloudy and only 20 when it is clear.
Then you may be tempted to say that birds favour cloudy conditions.
But what if the day itself was 90% cloudy and only 10% clear, then the
birds are could well be at random and there no connection. On the
other hand, if the day was 50% cloudy and 50% clear then there
probably is a connection. In this simplified case, if they were all
independent sightings of single birds, you could use chi square and
compare what you see with what you would have expected from the
proportion of cloud conditions. Things are more difficult if you are
counting birds in flocks because birds within flocks are not
independent. Things are even more difficult if the cover is
continouus but i would be happy to put the data into say 20% bins.
Derek
Ray Koopman
As a first step, why not plot the number of birds in each flock as
a function of the (interpolated) cloud cover, and get the ordinary
Pearson correlation? Then proceed to fancier models as necessary.
Xt
>
> - Show quoted text -
Thanks again, Ray. I don’t have the data yet but I’ll try it when it
comes. But, as I see it, the correlation between flock size and cover
is problematic. We need to consider both flock size and flock
frequency. If flock size is a random and independent of cover, then I
don’t think flock size vs cover will show anything. Flocks will just
be closer together. On the other hand, different cloud covers may
cause flocks to spit up into smaller ones so that there will be a
correlation between cover and flock size but the actual numbers in
each level of cover are much the same. So there could be no
correlation and a difference in numbers, or a correlation and no
difference in numbers.
Can I please have your thoughts on this little randomization test I
have just devised. I plan to put the cloud data into say 5 bins and
work out the proportion of the whole survey time that cover is in each
bin. From the total number of ducks seen I can then work out the
expected number of birds in each bin assuming cloud cover has no
effect. Then I will use the chi square formula sum((o - e)^2/e) to
work out a measure of how far the observed is from the expected. So
far so good. The normal chi square table is not applicable of course,
so the trick is to effectively make a new table from the data. Keep
the original set of bird observations but allocate them a cloud cover
at random according to the proportion of the whole survey time in each
bin (or perhaps just shuffle the bins to different birds) and
recalculate sum((o - e)^2/e). Repeat 100 times and rank them (all
automated, of course). See where the chi square number from real data
fits into this list. If it is near the middle, then there is no
significant connection. If it is in the top few, then there is a
significant connection.
I don’t know if sum((o - e)^2/e) is the best measure. We only use it
because we have the theory to calculate the percentiles. Perhaps a
proportional thing like sum(abs(o - e)/e) may be more powerful?
Derek
RichUlrich
I think that it is nice to ask the question before you collect the
data, so that you can collect the data that you *need*.
But I have to say, it is still not entirely obvious what you are
trying to determine. Geese do fly on partly-cloudy days -- don't
they fly all day when they are flying? Do they "sit out" bad storms?
In this case, if the question is whether "cloud cover" hides some
geese, some times, it seems like it would be better to record how
many geese spend how much time hidden by *partial* cloud cover.
Are you, maybe, trying to estimate a complete total? If so, it might
be useful to model the size-of-flock... If half the total pass by as
one group, which I (in total ignorance) consider conceivable, then
(a) you have a pretty good start if you count the big group, or
(b) you are really messed up if you miss it.
If the geese are all flying the same path, you might need to make
comments on the shape of the cloud cover, as well as the amount.
If you see part of a migration, partly hidden, can you guess how
much of it could be hidden?
It could be useful to actually do some observations, and then
write out a criticism of what could be wrong with it.
And then, can't you *hear* migrating geese, at times, even if they
are hidden?
--
Rich Ulrich
Ray Koopman
It's not clear how you would do the randomization. My earlier
suggestion took the flock as the unit of analysis, but that was
wrong. From your comments, it seems that time period should be the
unit. You have an n x 2 table, where each of the n rows contains
data for one time period; one column contains the cloud cover (x),
the other column contains the number of birds that passed over (y;
yes, there may be many zeros). To randomize, shuffle the values in
one column.
The question is what descriptive statistic to compute from each such
n x 2 table. You want a statistic that is sensitive to differences
between the average number of birds per period for different degrees
of cloud cover; that is, to differences between the conditional mean
of y|x for different values of x. That's exactly what correlation
looks at, so just correlate x with y and see where the correlation
for the actual data comes in the distribution of correlations for
the randomized data. There is no need to quantize cloud cover.
Xt
>
> - Show quoted text -
That sounds good, Ray. It takes care of the problem of accounting for
when we don't see birds. The only problem I can see is that it
assumes an increasing or decreasing function of numbers with cover.
If midrange cover encourages more birds to fly then the correlation
idea won't work.
My idea of shuffling the covers with the sightings won't work, but the
other idea - allocating random cover for each flock should. If you
know the proportions overall, then you can allocate each flock a cover
at random according to the proportions over the whole survey. If 15%
of the period is 0-20% then make the chance of allocating the 0-20 bin
to any particular flock 0.15 and so on.
Derek
Ray Koopman
> when we don't see birds. The only problem I can see is that it
> assumes an increasing or decreasing function of numbers with cover.
> If midrange cover encourages more birds to fly then the correlation
> idea won't work.
Plot the data. As I suggested earliier, start with ordinary
Pearson correlation, then proceed to fancier models as necessary.
Xt
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -
Thanks Ray. I'll sign off now and try your suggestions over the next
few weeks. Derek