size of the CMB fluctuations
Ted Sung
Hi,
I was wondering if there is a way to understand why the temperature
fluctuations of the cosmic microwave background radiation is about 1
in 100,000.
For example, if the photons were a a perfect gas, what would the size
of
the fluctuations be just from random fluctuations?
Thanks,
Ted
Greg Kuperberg
Ted Sung <te...@intex.com> wrote:
A crucial point is that the fluctuations do not depend on time. The CMB
is a static radiation map at the age that the universe became transparent
to photons. IIRC, this occurred about 100,000 years after the Big Bang.
There isn't any elementary explanation for the size of the fluctuations.
For one thing, the fluctuations do not have a single magnitude. The map
is a sum of fluctuations on different angular scales; the magnitude of a
fluctuation does depend on its angular wavelength. This is discussed in a
great Physics Today article, http://www.aip.org/pt/vol-53/iss-7/p17.html .
One explanation for the fluctuations is entirely theoretical: They
are predicted by inflation models. The other is entirely based on
observation: They are necessary for galaxy formation. In other words,
the non-uniformity of the universe grows over time, but only so quickly.
If it is now lumpy enough to consist of galaxies, it would have been
lumpy enough at 100,000 years to be observable.
--
/\ Greg Kuperberg (UC Davis)
/ \
\ / Visit the Math ArXiv Front at http://front.math.ucdavis.edu/
\/ * All the math that's fit to e-print *
greywolf42
"Ted Sung" <te...@intex.com> wrote in message
news:86dea8f5.02092...@posting.google.com...
One possible reason is that the physical sensitivity of the COBE detector is
1 part in 10,000. The CMBR fluctuations are "computer enhancements" of the
actual data. There is always the possibility that these "enhancements"
don't really exist.
greywolf42
ubi dubium ibi libertas
eb...@lfa221051.richmond.edu
[Crossposted to sci.astro.research.]
In article <upef7eh...@corp.supernews.com>,
greywolf42 <min...@sim-ss.com> wrote:
>One possible reason is that the physical sensitivity of the COBE detector is
>1 part in 10,000. The CMBR fluctuations are "computer enhancements" of the
>actual data. There is always the possibility that these "enhancements"
>don't really exist.
I can't tell if you're talking about fraud or error when you use the
term "computer enhancements." I'll assume the latter. (In case it's
the former, I'll simply point out that it's customery to have at least
a modicum of evidence before accusing people of gross ethical and
probably legal misconduct.)
People naturally worried quite a lot about the possibility of
systematic errors or artifacts in the data analysis when the COBE DMR
data first came out in the early 1990s. The tests that were done, by
both the original experimenters and scientists who weren't on the
team, to check for such errors were extremely extensive. You can read
about them in the voluminous peer-reviewed literature on COBE if
you're interested.
Of course, the best way to check for the possibility of systematic or
analysis error is to have a completely separate experiment replicate
the results. Fortunately, in the ten years since COBE, this has been
done repeatedly: microwave background anisotropy has now been detected
by something like 20 different experiments. The various experiments
are so different in detector technology, observing location,
frequency, angular scale, and analysis methods that it's extremely
hard to imagine any explanation for how they could all have come out
with consistent results, other than the obvious one: that what they're
seeing is there.
By the way, an up-to-date listing of CMB experiments, including
the recent detection of CMB polarization by DASI, can be found
at
http://background.uchicago.edu/~whu/cmbex.html
-Ted
--
[My posts come from a machine that doesn't accept incoming mail. To
e-mail me, use an address of the form user...@domain.edu, as opposed
to user...@machinename.domain.edu.]
Jonathan Silverlight
eb...@lfa221051.richmond.edu writes
>People naturally worried quite a lot about the possibility of
>systematic errors or artifacts in the data analysis when the COBE DMR
>data first came out in the early 1990s. The tests that were done, by
>both the original experimenters and scientists who weren't on the
>team, to check for such errors were extremely extensive. You can read
>about them in the voluminous peer-reviewed literature on COBE if
>you're interested.
[...]
There's no doubt that the COBE data was subjected to a great deal of
computer processing. There seems no doubt that the fluctuations actually
exist - as you say, they've been confirmed by other experiments and at
other scales. _However_, soon after the original reports ISTR reading
that the original results aren't "real", in the sense that if you
collected new data you would get a different picture. I can't find
anything in "Wrinkles in Time", but does this make any sense?
--
mail to jsilverlight AT merseia.fsnet.co.uk is welcome
[s.a.r. mod. note: quoted text trimmed -- mjh.]
eb...@lfa221051.richmond.edu
sci.astro.research.]
In article <86dea8f5.02092...@posting.google.com>,
In a gas in thermal equilibrium, the fluctuations would be
much, much smaller than a part in 10^5. I guess that the
fluctuation in, say, the total energy in a box of a certain
size should be about 1/sqrt(N), where N is a typical number
of particles in that box. The fluctuations that matter
for the microwave background are on scales of hundreds of
megaparsecs (comoving), and there are something like 10^75 photons
in a box of that size. (I may have that number wrong, but
anyway it's truly enormous.) So the thermal fluctuations
should be a part in 10^37 or so.
That just means that the Universe wasn't in thermal equilibrium
on those scales. That's OK: there's no reason it should have been!
In the early Universe, scales like that were larger than the horizon
length -- that is, light didn't have time to travel across such
a length scale. Since no influences travel faster than light, there's
no mechanism to reach thermal equilibrium.
So there's no problem in explaining why the fluctuations are
as large as 10^(-5). The problem is explaining why they're as small
as 10^(-5). If the Universe had no mechanism to reach thermal
equilibrium, why is it even close to thermal equilibrium? Why doesn't
the temperature vary wildly from place to place?
The leading theory to explain this is inflation. In inflationary models,
the Universe expanded extremely rapidly at extremely early times.
Before inflation, everything was so close together that thermal
equilibrium should have been reached. This was one of the leading
arguments in favor of inflation when it was first proposed around 1980:
it explains why the Universe is so uniform.
Shortly after that (meaning shortly after the 1980s, not shortly after
inflation -- although there was a lot of inflation in the 1970s), people
realized that inflation could also explain the 1 part in 10^5 fluctuations.
During inflation, quantum fluctuations in the vacuum get "stretched out"
to large (classical) scales and become honest-to-goodness density
variations.
So the most popular answer to your question (why 10^(-5)?) is
inflation. It's not a complete answer: inflation doesn't firmly
predict 10^(-5) as opposed to, say, 10^(-4), but it's not bad. The
amplitude of the fluctuations produced in this mechanism is a free
parameter, but it's got to be considerably less than one, and it turns
out that 10^(-5) is a "reasonable" level in a lot of inflationary
models.
Models based on inflation do an excellent job of explaining lots
of things about the Universe, so most cosmologists think inflation
stands a good chance of being correct. There's not the sort
of stunningly direct evidence for it that could convince a die-hard
skeptic, though.
If inflation didn't happen, then I think it's safe to say
that nobody knows why 10^(-5).
-Ted
John Devers
> One explanation for the fluctuations is entirely theoretical: They
> are predicted by inflation models. The other is entirely based on
> observation: They are necessary for galaxy formation. In other words,
> the non-uniformity of the universe grows over time, but only so quickly.
> If it is now lumpy enough to consist of galaxies, it would have been
> lumpy enough at 100,000 years to be observable.
This quote from an astronomer I know and picture from COBE may help.
"Gravitational clumping was also going on right from the start - cut
on huge scales -
Here is an observation of the clumping of the universe at the time of
recombination - all gravitationally induced"
http://aether.lbl.gov/www/projects/cobe/COBE_Home/physics_today.gif
(bottom piccy is the important one - red is overdensities of matter,
blue is underdensities)
eb...@lfa221051.richmond.edu
Jonathan Silverlight <jsi...@merseia.fsnet.co.uk> wrote:
>There's no doubt that the COBE data was subjected to a great deal of
>computer processing. There seems no doubt that the fluctuations actually
>exist - as you say, they've been confirmed by other experiments and at
>other scales. _However_, soon after the original reports ISTR reading
>that the original results aren't "real", in the sense that if you
>collected new data you would get a different picture.
There are actually two completely different senses in which this is
true:
1. The signal-to-noise ratio per pixel in the COBE maps was only about
one. That is, each pixel had roughly equal contributions from
actual CMB fluctuations and from instrument noise. If you rebuilt
COBE and flew it again, you'd get another map with the same signal
but different noise. At the level of individual pixels, it would
be quite different.
Note, though, that most COBE maps you're likely to see in
presentations have been smoothed. Smoothing reduces the noise by
averaging together many pixels (at the cost, of course, of reducing
the number of independent data points). The signal-to-noise in such
smoothed maps is considerably higher than one, so the hot and cold
spots you see in smoothed maps are real. If you flew COBE again,
you'd see the same spots in the smoothed map.
Smoothing throws away information, so for actual quantitative analysis
of the data, people use unsmoothed maps whenever possible. Smoothed
maps are typically used only for "pretty pictures" (or for answering
questions that only depend on the largest-scale properties of the
map).
2. The particular pattern of hot and cold spots that we see in the
COBE map are thought to be fundamentally random. In
inflation-based models, for instance, they're produced ultimately
by quantum fluctuations. So if you could somehow rerun the entire
Universe from scratch, you wouldn't see the same spots. Or, if you
want a less drastic scenario, if you could redo the COBE experiment
from a different spot in the Universe (many billions of light-years
away), you'd see different hot and cold spots.
greywolf42
news:mt2.0-14162...@star.bris.ac.uk...
> In article <mt2.0-10264...@star.bris.ac.uk>,
> Jonathan Silverlight <jsi...@merseia.fsnet.co.uk> wrote:
>
> >There's no doubt that the COBE data was subjected to a great deal of
> >computer processing. There seems no doubt that the fluctuations actually
> >exist - as you say, they've been confirmed by other experiments and at
> >other scales. _However_, soon after the original reports ISTR reading
> >that the original results aren't "real", in the sense that if you
> >collected new data you would get a different picture.
>
> There are actually two completely different senses in which this is
> true:
>
> 1. The signal-to-noise ratio per pixel in the COBE maps was only about
> one. That is, each pixel had roughly equal contributions from
> actual CMB fluctuations
Good clarification of the theory.
> and from instrument noise. If you rebuilt
> COBE and flew it again, you'd get another map with the same signal
> but different noise. At the level of individual pixels, it would
> be quite different.
Yes. You'd get a different map. Showing different "signals."
> Note, though, that most COBE maps you're likely to see in
> presentations have been smoothed.
I believe you have this backwards. The actual COBE data was "too smooth."
There was no signal discernable at the 1 to 10,000 level (the level where
your theoretical variation in the "signal" was equal to the noise level).
> Smoothing reduces the noise by
> averaging together many pixels (at the cost, of course, of reducing
> the number of independent data points).
Smoothing does not reduce the number of independent data points. Are you
referring to data selection when you use the term "smoothing?"
> The signal-to-noise in such
> smoothed maps is considerably higher than one,
Smoothing has no effect whatsoever on the signal to noise ratio.
> so the hot and cold
> spots you see in smoothed maps are real. If you flew COBE again,
> you'd see the same spots in the smoothed map.
Not if your description of the process is correct. Unless you arranged a
way to "select" data with the same values.
> Smoothing throws away information, so for actual quantitative analysis
> of the data, people use unsmoothed maps whenever possible. Smoothed
> maps are typically used only for "pretty pictures" (or for answering
> questions that only depend on the largest-scale properties of the
> map).
Smoothing (the averaging of many data points) does not throw away ANY
information. You seem to be talking about data selection. Is data
selection the process used in "data enhancement?"
{snip}
eb...@lfa221051.richmond.edu
greywolf42 <min...@sim-ss.com> wrote:
><eb...@lfa221051.richmond.edu> wrote in message
>> and from instrument noise. If you rebuilt
>> COBE and flew it again, you'd get another map with the same signal
>> but different noise. At the level of individual pixels, it would
>> be quite different.
>
>Yes. You'd get a different map. Showing different "signals."
I'd prefer to say that it would have the same signal but different
noise. Maybe that's what you mean here; it's not clear to me. (If
it's not what you mean, then I have no idea what you do mean!)
Just to be 100% explicit here, each pixel of the map, T_i, is the
sum of a "signal part" and a "noise part,"
T_i = s_i + n_i.
The noise would be different each time you ran the experiment; the
signal part would be the same. The signal part is what you'd
really like to know, but of course all the experiment gives you
is the combination s_i+n_i.
(In case anyone thinks I'm actually trying to say something
interesting here, let me assure you otherwise. All of the above is
incredibly boring, standard, and obvious stuff that applies to pretty
much any measurement.)
>> Note, though, that most COBE maps you're likely to see in
>> presentations have been smoothed.
>
>I believe you have this backwards. The actual COBE data was "too smooth."
>There was no signal discernable at the 1 to 10,000 level (the level where
>your theoretical variation in the "signal" was equal to the noise level).
I think we've just misunderstood each other. Here's what I meant by the
above sentence:
The COBE maps that you're likely to see in presentations are not maps
of the raw data. They're maps of the raw data after it has been
convolved with some sort of smoothing kernel (usually a Gaussian with
a 7 degree FWHM beam, if you want the details).
For instance, there's this image
http://space.gsfc.nasa.gov/astro/cobe/phys_today_cover_big.gif
which was on the cover of Physics Today way back when the COBE data first
came out. Or there's this one
http://space.gsfc.nasa.gov/astro/cobe/dmr_4yr_cmb_stereo.gif
which is a different map projection and is based on the full four
years of data (unlike the first one, which was generated before
all four years of data had been gathered).
These are both smoothed maps, meaning simply that the temperature
plotted at each point is a weighted average (convolution) of the
measured temperatures in a neighborhood of that point.
To see the difference, you should compare this to an unsmoothed map.
Oddly, there doesn't seem to be one anywhere at the COBE web site.
(You can get the raw data, but not a picture.) Fortunately, I've made
some myself. You can see one, for instance, at
http://www.richmond.edu/~ebunn/rawmap.ps
http://www.richmond.edu/~ebunn/rawmap.pdf
(same map, different file formats). In this map, I just cut out the
roughly 1/3 of the sky with the most Galactic contamination, rather
than doing anything fancy as the COBE folks did in some of their maps.
It's the full four years of COBE data (average of the two 53 GHz and
the two 90 GHz channels, if you must know). The point here is that
there's considerable pixel-to-pixel variation in this map, most of
which is due to noise. It's hard to pick out by eye the "real"
hot and cold spots.
If you smooth (convolve) the map, you get something that looks
prettier. Since the noise is (roughly) uncorrelated from pixel to
pixel, if you average together a bunch of nearby pixels (which is all
convolving is), you beat down the noise. The signal is almost the
same in neighboring pixels, though, so smoothing doesn't reduce the
signal too much. (It does reduce it some, which is why smoothed maps
are good for "pretty pictures" but not so good for serious data
analysis.)
>> Smoothing reduces the noise by
>> averaging together many pixels (at the cost, of course, of reducing
>> the number of independent data points).
>
>Smoothing does not reduce the number of independent data points. Are you
>referring to data selection when you use the term "smoothing?"
I'm referring to convolution with a smoothing kernel. That throws
away small-scale power in the data set, or in other words it makes
the signal in neighboring pixels more strongly correlated than
it was before. Either way, it means you can measure fewer independent
numbers in the smoothed data set than you could in the original.
>> The signal-to-noise in such
>> smoothed maps is considerably higher than one,
>
>Smoothing has no effect whatsoever on the signal to noise ratio.
I promise it does, for the reason I tried to explain above. Let's
take a one-dimensional example. Suppose you have a data set that
consists of 1000 points arranged along a straight line. Each point
has noise in it. The noise is uncorrelated from point to point, but
the signal isn't. Say you smooth this map by replacing the value at
pixel i with the average of the 101 pixels around i:
T'_i = (1/101) * sum from j = -50 to 50 of T_{i+j}
Your noise in each pixel will be about ten times less than it was,
because you averaged together 101 independent random numbers. But if
the signal is coherent on that scale (i.e., if the signal part of T
doesn't change too rapidly as a function of i), then the signal will
not have been reduced by a factor of 10. Presto! Higher
signal-to-noise ratio.
At the risk of being repetitive, the price you pay for this is
a smaller number of independent data points. In this case, T'_{500}
and T'_{501} are "less independent" than they were. They now
contain almost exactly the same information.
>> so the hot and cold
>> spots you see in smoothed maps are real. If you flew COBE again,
>> you'd see the same spots in the smoothed map.
>
>Not if your description of the process is correct. Unless you arranged a
>way to "select" data with the same values.
I can't begin to guess what you mean here. Sorry!
>> Smoothing throws away information, so for actual quantitative analysis
>> of the data, people use unsmoothed maps whenever possible. Smoothed
>> maps are typically used only for "pretty pictures" (or for answering
>> questions that only depend on the largest-scale properties of the
>> map).
>
>Smoothing (the averaging of many data points) does not throw away ANY
>information. You seem to be talking about data selection. Is data
>selection the process used in "data enhancement?"
Of course it does! Suppose I give you the following information:
3, 7, 14.
Haven't I given you more information than if I just tell you
8
which is the average of those numbers?
I confess I'm indulging in a tiny bit of sophistry here. Convolution
is in principle an invertible operation, whereas averaging three
numbers to get one isn't. But in practice deconvolution is horribly
unstable. If I gave you a smoothed COBE map, in practice there's no
way you could get the unsmoothed map from it. In that sense,
smoothing throws away information.
If you don't believe me, let's try it! I'll give you a smoothed
simulated COBE map, and I'll leave the original, unsmoothed map in the
custody of an independent trusted third party. You try deconvolving
the smoothed map to get the original.
Greg Kuperberg
<eb...@lfa221051.richmond.edu> wrote:
>In article <mt2.0-29784...@star.bris.ac.uk>,
>I confess I'm indulging in a tiny bit of sophistry here. Convolution
>is in principle an invertible operation, whereas averaging three
>numbers to get one isn't. But in practice deconvolution is horribly
>unstable.
It anything but a pretense to say that convolution loses information.
Relative to reasonable noise models on function space, it does. What is
"sophistry", or more politely a bad information model, is to use a real
number as a unit of information. Obvously a real number contains an
infinite amount of information: you can records two real numbers into
one by interleaving digits. To sensibly interpret the information in a
real number, you need both a macroscopic cutoff, typically called a power
spectrum, and a microscopic cutoff, typically called a noise spectrum.
Mathematical deconvolution on function space is discontinuous, which
means that it need not obey rules of information theory. Computational
deconvolution on discretized functions, on the other hand, is technically
okay as a linear map on a finite-dimensional vector space. However, since
it magnifies noise along with signal, there is no gain in information.
I have the impression that information content is increasingly important
in CMB studies. It might therefore make sense to move to a more
rigorous point of view than merely noting that deconvolution doesn't
work in practice.
jacob navia
> sum of a "signal part" and a "noise part,"
>
> T_i = s_i + n_i.
>
> The noise would be different each time you ran the experiment; the
> signal part would be the same. The signal part is what you'd
> really like to know, but of course all the experiment gives you
> is the combination s_i+n_i.
Wait a minute.
"Noise" in this context could be any source, even an astronomical source.
Actually
T_i = signal_i + noise1_i + noise2_i + noise3_i... etc
where we could have several kinds of superposed "noise" signals represented
above by several noise parameters. Hot pixels would be only *one* kind of
noise. Other kinds are surely possible!
Imagine an astronomical object that emits a signal between the observer and
the background radiation (such an object is not difficult to assume since by
definition the background radiation is the most distant "object" we are
trying to measure). Its signal could be misunderstood as a fluctuation in
the background radiation isn't it?
Just a question. Thanks for your attention.
eb...@lfa221051.richmond.edu
jacob navia <ja...@jacob.remcomp.fr> wrote:
>> Just to be 100% explicit here, each pixel of the map, T_i, is the
>> sum of a "signal part" and a "noise part,"
>>
>> T_i = s_i + n_i.
>>
>> The noise would be different each time you ran the experiment; the
>> signal part would be the same. The signal part is what you'd
>> really like to know, but of course all the experiment gives you
>> is the combination s_i+n_i.
>
>Wait a minute.
>"Noise" in this context could be any source, even an astronomical source.
>Actually
>
>T_i = signal_i + noise1_i + noise2_i + noise3_i... etc
>
>where we could have several kinds of superposed "noise" signals represented
>above by several noise parameters. Hot pixels would be only *one* kind of
>noise. Other kinds are surely possible!
Absolutely right. Of course, whether you call those various
contributions "signal" or "noise" is a matter of taste. Microwave
background data sets generally contain contributions from Galactic
dust, radio point sources, etc. To a cosmologist like me, those
things are noise; to someone who studies Galactic dust, radio point
sources, etc., they're signal!
>Imagine an astronomical object that emits a signal between the observer and
>the background radiation (such an object is not difficult to assume since by
>definition the background radiation is the most distant "object" we are
>trying to measure). Its signal could be misunderstood as a fluctuation in
>the background radiation isn't it?
Yes, indeed. In fact, separating actual CMB fluctuations from
foreground sources is one of the more difficult aspects of the whole
CMB data analysis game. The simplest tricks are cross-correlation
with templates of known sources and use of multifrequency data to
separate the component with the CMB spectrum from components with
other spectra. Most of the techniques that are used are variants of
these two ideas, ranging from incredibly crude to moderately
sophisticated.
Of course these techniques aren't perfect, but they may be good enough
for some purposes. Telling whether they're good enough is not
trivial, though: we don't really have reliable models for the
foreground contaminants, so it's hard to quantify the errors
associated with component separation. This is the biggest problem
that I personally am worried about in the next generation of microwave
background experiments (MAP and Planck).
It may get even worse as attention shifts from temperature
measurements to polarization measurements, by the way, as we know even
less about the polarization properties of foregrounds. Maybe the
foregrounds are weakly polarized and the data will be largely free of
foreground contamination, but maybe they won't. And it's not really
clear how we'll know.
eb...@lfa221051.richmond.edu
Greg Kuperberg <gr...@conifold.math.ucdavis.edu> wrote:
>In article <mt2.0-7101...@star.bris.ac.uk>,
> <eb...@lfa221051.richmond.edu> wrote:
>>In article <mt2.0-29784...@star.bris.ac.uk>,
>>I confess I'm indulging in a tiny bit of sophistry here. Convolution
>>is in principle an invertible operation, whereas averaging three
>>numbers to get one isn't. But in practice deconvolution is horribly
>>unstable.
>
>It anything but a pretense to say that convolution loses information.
You're right, of course. I put that comment in just to forestall the
complaint that I was being unfair in comparing convolution (which
turns a set of N numbers into another set of N numbers) to averaging a
set of N numbers together to get 1 number.
>Relative to reasonable noise models on function space, it does. What is
>"sophistry", or more politely a bad information model, is to use a real
>number as a unit of information.
I wasn't trying to do that! I was just trying to illustrate my point
in a very crude way. I can see how that may have been misleading,
though. Thanks for pointing this out.
>I have the impression that information content is increasingly important
>in CMB studies. It might therefore make sense to move to a more
>rigorous point of view than merely noting that deconvolution doesn't
>work in practice.
In fact, CMB analysis has grown enormously in sophistication in these
matters in the past decade or so. When I first got started in the
business, it was incredibly crude; it's really not anymore. For instance,
you can't swing a dead cat at a CMB conference without hitting
a Fisher information matrix these days.
That's not to say that we don't need to move further in the direction
of statistical and information-theoretic rigor; maybe we do.
In any case, please don't take the statements I've made in this thread
as indications of the state of the art in CMB analysis. I've been
speaking quite vaguely and nontechnically in this thread, which is
what I thought (perhaps incorrectly) the situation warranted.
Steve Willner
In article <mt2.0-25776...@star.bris.ac.uk>,
jacob navia <ja...@jacob.remcomp.fr> writes:
> Imagine an astronomical object that emits a signal between the observer and
> fluctuation in the background radiation isn't it?
Ted (who is a pro in this business if he is the person I think he is)
has given a good answer, but let me add the simple-minded one.
In the jargon of the field, such a source would be called a
"foreground," and it represents a systematic error, not a noise
source. The difference is that a foreground would be seen by _all_
similar observations, whereas a noise source would be different for
each independent observation.
In this and many other experimental contexts, it is quite important
to distinguish between "noise," which you can decrease simply by
observing longer, and "systematic errors," which remain the same no
matter how long you observe. The only remedy for systematic errors
is to do a different type of experiment or observation. COBE, for
example, observed at many different wavelengths because the
foregrounds are different at each one.
Of course both noise and systematic errors can lead to an error in
the interpretation of an experiment's results, but observers try very
hard to quantify how big the errors are likely to be.
--
Steve Willner Phone 617-495-7123 swil...@cfa.harvard.edu
Cambridge, MA 02138 USA
(Please email your reply if you want to be sure I see it; include a
valid Reply-To address to receive an acknowledgement. Commercial
email may be sent to your ISP.)
greywolf42
news:mt2.0-7101...@star.bris.ac.uk...
> In article <mt2.0-29784...@star.bris.ac.uk>,
> greywolf42 <min...@sim-ss.com> wrote:
> ><eb...@lfa221051.richmond.edu> wrote in message
>
> >> and from instrument noise. If you rebuilt
> >> COBE and flew it again, you'd get another map with the same signal
> >> but different noise. At the level of individual pixels, it would
> >> be quite different.
> >
> >Yes. You'd get a different map. Showing different "signals."
>
> I'd prefer to say that it would have the same signal but different
> noise. Maybe that's what you mean here; it's not clear to me. (If
> it's not what you mean, then I have no idea what you do mean!)
>
> Just to be 100% explicit here, each pixel of the map, T_i, is the
> sum of a "signal part" and a "noise part,"
>
> T_i = s_i + n_i.
>
> The noise would be different each time you ran the experiment; the
> signal part would be the same. The signal part is what you'd
> really like to know, but of course all the experiment gives you
> is the combination s_i+n_i.
>
> (In case anyone thinks I'm actually trying to say something
> interesting here, let me assure you otherwise. All of the above is
> incredibly boring, standard, and obvious stuff that applies to pretty
> much any measurement.)
True. But in this case, you're merely assuming that s_i <> 0. Since
signal-to-noise ratio is 1 or less. To be a valid scientific experiment,
you'll need signal greater than 1 (i.e. no data enhancement).
That's not odd at all. A map that looks like "nothing" won't encourage
interest or money.
> Fortunately, I've made
> some myself. You can see one, for instance, at
>
> http://www.richmond.edu/~ebunn/rawmap.ps
> http://www.richmond.edu/~ebunn/rawmap.pdf
>
> (same map, different file formats). In this map, I just cut out the
> roughly 1/3 of the sky with the most Galactic contamination, rather
> than doing anything fancy as the COBE folks did in some of their maps.
> It's the full four years of COBE data (average of the two 53 GHz and
> the two 90 GHz channels, if you must know). The point here is that
> there's considerable pixel-to-pixel variation in this map, most of
> which is due to noise. It's hard to pick out by eye the "real"
> hot and cold spots.
Which would -- of course -- be the result from a "zero" signal.
> If you smooth (convolve) the map, you get something that looks
> prettier. Since the noise is (roughly) uncorrelated from pixel to
> pixel, if you average together a bunch of nearby pixels (which is all
> convolving is), you beat down the noise.
But this is a Bayesian assumption. If you were to generate numbers at
random over a large number of pixels, and then "smooth" the results, you'd
get the same effect. Such data enhancements are a standard "method" in psi
experiments. There's no end to the "correlations" you can get if you just
massage the data enough.
> The signal is almost the
> same in neighboring pixels, though, so smoothing doesn't reduce the
> signal too much. (It does reduce it some, which is why smoothed maps
> are good for "pretty pictures" but not so good for serious data
> analysis.)
>
> >> Smoothing reduces the noise by
> >> averaging together many pixels (at the cost, of course, of reducing
> >> the number of independent data points).
> >
> >Smoothing does not reduce the number of independent data points. Are you
> >referring to data selection when you use the term "smoothing?"
>
> I'm referring to convolution with a smoothing kernel. That throws
> away small-scale power in the data set, or in other words it makes
> the signal in neighboring pixels more strongly correlated than
> it was before. Either way, it means you can measure fewer independent
> numbers in the smoothed data set than you could in the original.
This comes under data selection. And data selection is antiscience.
The rest of the post is repeats of the basic theme. So I'll snip here..
eb...@lfa221051.richmond.edu
>> Just to be 100% explicit here, each pixel of the map, T_i, is the
>> sum of a "signal part" and a "noise part,"
>>
>> T_i = s_i + n_i.
>>
>> The noise would be different each time you ran the experiment; the
>> signal part would be the same. The signal part is what you'd
>> really like to know, but of course all the experiment gives you
>> is the combination s_i+n_i.
>>
>> (In case anyone thinks I'm actually trying to say something
>> interesting here, let me assure you otherwise. All of the above is
>> incredibly boring, standard, and obvious stuff that applies to pretty
>> much any measurement.)
>
>True. But in this case, you're merely assuming that s_i <> 0.
Well, in what I said above, I didn't assume anything. The above
statements, banal and boring as they are, are valid whether or not
s_i = 0.
Nonetheless, if it'll make you happy, I'll gladly make the statement
you attribute to me:
The signal s_i in the COBE experiment was nonzero.
I'm not "assuming" that, though. I checked it. I made my living
(such as it was) for a number of years analyzing the COBE data.
>Since
>signal-to-noise ratio is 1 or less.
The signal-to-noise ratio *per pixel* in the COBE experiment was less
than (but not much less than) one.
Don't leave off the "per pixel" part -- it's important! Since there
are many pixels, the overall statistical weight of the experiment is
very high: the no-signal hypothesis is ruled out at very high
confidence.
There's nothing specific to COBE about this sort of thing. It's
extremely common in experiments for each individual sampling of the
data to have low signal-to-noise. If you have many measurements, you
beat down the noise and get a high-confidence measurement anyway.
>To be a valid scientific experiment,
>you'll need signal greater than 1 (i.e. no data enhancement).
You've referred repeatedly to "data enhancement." I worked on the
COBE data for years, and I honestly have no idea what you mean by
it. Anyone in the world can go to the COBE web site, grab the raw
data, and do statistical tests on it to see if there's a signal in it.
There is.
Can you explain exactly what you mean by "enhancement"? Try being
about a billion times more specific than you have been so far, or
better yet provide a reference.
>> The point here is that
>> there's considerable pixel-to-pixel variation in this map, most of
>> which is due to noise. It's hard to pick out by eye the "real"
>> hot and cold spots.
>
>Which would -- of course -- be the result from a "zero" signal.
Right. If you look at the *raw data* (not the smoothed data) *by eye*
(without doing any sort of statistical analysis), you couldn't tell
whether there was signal or not. (Or at least it'd be hard. If
you're good at doing convolutions in your head, you could do it.)
Fortunately, we're not required to do our data analysis by looking at
the raw maps by eye. We're allowed to, well, analyze them! (We also
don't have to do it with one hand tied behind our backs.) If you do,
you find that the no-signal hypothesis is ruled out at high
confidence. In other words, there is signal there.
>> If you smooth (convolve) the map, you get something that looks
>> prettier. Since the noise is (roughly) uncorrelated from pixel to
>> pixel, if you average together a bunch of nearby pixels (which is all
>> convolving is), you beat down the noise.
>
>But this is a Bayesian assumption.
No. This is true regardless of your statistical approach.
>If you were to generate numbers at
>random over a large number of pixels, and then "smooth" the results, you'd
>get the same effect.
This is false. If you generate a white-noise map and then smooth it,
you would see a smoothed noise map. It would look dramatically
different from the actual data.
To be specific, the power spectrum of the smoothed noise map would be
a constant, multiplied by the Fourier transform squared of the
smoothing kernel. The power spectrum of a smoothed map with signal
in it would look quite different, as indeed the power spectrum
of the real COBE data does. To put this more intuitively,
if you take a white-noise map and smooth it, there will be no correlations
in the map on scales larger than the smoothing scale. In the
COBE map, there are coherent structures on scales much larger
than the smoothing scale.
>Such data enhancements are a standard "method" in psi
>experiments. There's no end to the "correlations" you can get if you just
>massage the data enough.
Everything that was done to the COBE data is exhaustively documented
in the published literature. If you think that any of the documented
steps in the data reduction led to the introduction of spurious
signals, explain why. Be sure to be specific. Using ill-defined yet
pejorative terms like "massaging" won't cut it.
If you think that the data were "massaged" in ways other than those
documented in the published literature, then you're accusing the
researchers of fraud. In that case, you might want to consider
supplying some evidence.
>> I'm referring to convolution with a smoothing kernel. That throws
>> away small-scale power in the data set, or in other words it makes
>> the signal in neighboring pixels more strongly correlated than
>> it was before. Either way, it means you can measure fewer independent
>> numbers in the smoothed data set than you could in the original.
>
>This comes under data selection. And data selection is antiscience.
As I said repeatedly, actual analysis of the COBE data was always
done on the unsmoothed maps. The smoothed maps are used only
for illustration. So even if smoothing is "antiscience" by your
own idiosyncratic definition, it doesn't matter.
-Ted
--
[E-mail me at na...@domain.edu, as opposed to na...@machine.domain.edu.]
eb...@lfa221051.richmond.edu
> Right. If you look at the *raw data* (not the smoothed data) *by eye*
> (without doing any sort of statistical analysis), you couldn't tell
> whether there was signal or not. (Or at least it'd be hard. If
> you're good at doing convolutions in your head, you could do it.)
But in fact, upon reconsideration, I want to take that back. I
conceded way too much with that statement. Even looking at the raw
COBE map by eye, without doing any sort of data analysis at all, it's
pretty easy to tell that you're not looking at a pure noise map.
For an illustration of this, you can look at
http://www.richmond.edu/~ebunn/maps/maps.html
Three sky maps are shown there. One of them is the raw COBE data (no
smoothing). The other two are pure noise maps, made with noise that
has the same statistical properties as the noise in the real data.
Take a look and see if you can tell by eye which one is different. I
think it's easy, but it's hard for me to tell, since I know the right
answer.
[Fine print: In all three cases, the gray scale is temperature,
measured in units of the standard deviation of the map. If I'd just
used raw temperature, without scaling by the standard deviation, it
would have been much too easy to tell the difference, because the
r.m.s. pixel value is about 50% higher in the real data than in a
pure noise map. (That's just because the signal-to-noise pixel is
not much less than one.) In each map, I removed the same set of
low-Galactic-latitude pixels as were removed in most actual
analyses.]
The statement I made above was true for the first COBE maps released.
Those were based on one year of data, not four, and the
signal-to-noise per pixel was correspondingly worse by about a factor
of 2. In those maps, there was statistically significant signal, but
you couldn't pick it out by eye in the raw maps.
I suspect that this is where the notion that the COBE signal is only
there when you "enhance" or "massage" the data came from. Even then,
this conclusion wasn't true -- at least, not in any malign sense of
"enhance" or "massage." Some analysis beyond just eyeballing the data
was necessary to extract the signal, but who cares? No one expected
particle physicists to detect the top quark by eyeballing the data
stream pouring out of the accelerator; no one has yet outlawed
analyzing data.
Urs Schreiber
> For an illustration of this, you can look at
>
> http://www.richmond.edu/~ebunn/maps/maps.html
>
> Three sky maps are shown there. One of them is the raw COBE data (no
> smoothing). The other two are pure noise maps, made with noise that
> has the same statistical properties as the noise in the real data.
> Take a look and see if you can tell by eye which one is different. I
> think it's easy, but it's hard for me to tell, since I know the right
> answer.
[Moderator's note: Just in case anyone doesn't want to see the answer,
I'll put some blank space before Urs Schreiber's post, in which he
gives the correct answer. I won't do this after today. -TB]
I'd say the third, the one at the bottom of the page, is the
raw data.
Kevin A. Scaldeferri
<eb...@lfa221051.richmond.edu> wrote:
>
>http://www.richmond.edu/~ebunn/maps/maps.html
>
>Three sky maps are shown there. One of them is the raw COBE data (no
>smoothing). The other two are pure noise maps, made with noise that
>has the same statistical properties as the noise in the real data.
>Take a look and see if you can tell by eye which one is different. I
>think it's easy, but it's hard for me to tell, since I know the right
>answer.
my vote, and explanation below...
[s.a.r. mod. note: some newsreaders don't deal properly with
control-L, so I've added some spoiler space by hand -- mjh.]
I vote for C
I may have cheated, though, by applying a "smoothing filter" provided
me by nature. I.e., I took off my glasses.
--
======================================================================
Kevin Scaldeferri Calif. Institute of Technology
The INTJ's Prayer:
Lord keep me open to others' ideas, WRONG though they may be.
Ned Wright
>
> For an illustration of this, you can look at
>
> http://www.richmond.edu/~ebunn/maps/maps.html
>
> Three sky maps are shown there. One of them is the raw COBE data (no
> smoothing). The other two are pure noise maps, made with noise that
> has the same statistical properties as the noise in the real data.
> Take a look and see if you can tell by eye which one is different. I
> think it's easy, but it's hard for me to tell, since I know the right
> answer.
This test is a bit too easy, since it looks like the mean of the real
map was zeroed before the galaxy cut was applied instead of removing
the monopole+dipole after cutting the galaxy. So the real map has
a negative background which makes it lighter.
Now if you fixed this and redid the COBE maps in res6 HEALpix with
49152 pixels, then the SNR per pixel would be very low and it would
be hard to tell. For the 1 year maps the signal was obvious in 500
"pixels" for the 10^o smoothed resolution, and the above maps show
that the 4 year COBE has enough SNR so the signal is obvious in
6144 pixels. This is a very resolution dependent test, but easier
to understand than some advanced statistical method.
--Edward L. (Ned) Wright, UCLA Professor of Physics and Astronomy
See http:www.astro.ucla.edu/~wright/cosmolog.htm
eb...@lfa221051.richmond.edu
Ned Wright <mapp...@yahoo.com> wrote:
>eb...@lfa221051.richmond.edu wrote in message news:<mt2.0-17162...@star.bris.ac.uk>...
>>
>> For an illustration of this, you can look at
>>
>> http://www.richmond.edu/~ebunn/maps/maps.html
>>
>> Three sky maps are shown there. One of them is the raw COBE data (no
>> smoothing). The other two are pure noise maps, made with noise that
>> has the same statistical properties as the noise in the real data.
>> Take a look and see if you can tell by eye which one is different. I
>> think it's easy, but it's hard for me to tell, since I know the right
>> answer.
>
>This test is a bit too easy, since it looks like the mean of the real
>map was zeroed before the galaxy cut was applied instead of removing
>the monopole+dipole after cutting the galaxy. So the real map has
>a negative background which makes it lighter.
You're right. Sorry about that! I should have been more careful.
I replaced it with a correct version, and I also reshuffled the maps.
The test is definitely harder now, but I think it's still quite doable.
Kevin Scaldeferri's suggestion of the "poor man's convolution" -- taking
off your eyeglasses -- does seem to help, actually (although it won't
help those of you who were cursed with perfect vision).
>Now if you fixed this and redid the COBE maps in res6 HEALpix with
>49152 pixels, then the SNR per pixel would be very low and it would
>be hard to tell. For the 1 year maps the signal was obvious in 500
>"pixels" for the 10^o smoothed resolution, and the above maps show
>that the 4 year COBE has enough SNR so the signal is obvious in
>6144 pixels. This is a very resolution dependent test, but easier
>to understand than some advanced statistical method.
Absolutely. This is an important point. greywolf's original
objection (as near as I could tell) is that smoothing the data to
increase the signal-to-noise per pixel is some sort of unfair
"enhancement." But in fact, the choice of pixel size, and hence the
signal-to-noise per pixel, was somewhat arbitrary to begin with: if
the COBE team had chosen to make maps with large numbers of very tiny
pixels, the signal-to-noise per pixel would have been low; if they'd
chosen to make maps with very large pixels, it would have been high.
So it's a bit silly to assign much significance to the signal-to-noise
per pixel in the released maps.