Supernova data
Oh No
near SN (which should much improve the dataset) , while I wasn't looking
http://arxiv.org/abs/0804.4142
They have a website where they intend to keep regular updates, and from
which the data can be downloaded.
http://supernova.lbl.gov/Union/
Regards
--
Charles Francis
moderator sci.physics.foundations.
charles (dot) e (dot) h (dot) francis (at) googlemail.com (remove spaces and
braces)
Chalky
> There has been a new, and substantially larger compilation including new
> near SN (which should much improve the dataset) , while I wasn't looking
>
> http://arxiv.org/abs/0804.4142
>
> They have a website where they intend to keep regular updates, and from
> which the data can be downloaded.
>
> http://supernova.lbl.gov/Union/
>
> Regards
>
> --
> Charles Francis
> moderator sci.physics.foundations.
Excellent!
Lucky I was visiting here when you noticed!
I look forward to finishing off my graphical analysis program as the
nights start drawing in, with the added incentive of knowing that the
data set is now larger, and growing.
Meanwhile, if you come up with more detailed chi squared results, I
would be delighted to read about them.
As you know, I was struggling a bit with the statistical error margins
earlier, but one interesting thing I did notice was that I got less
visible scatter (irrespective of theoretical reference model) when the
program computed unweighted mean values per z range than when it
weighted them according to declared error margins. This does seem to
confirm that, graphically, the advantages of an effectively larger set
outweigh the advantages of giving greater importance to results
presented with greater (accuracy) confidence, at least, so far.
Oh No
>On Sep 1, 11:45 am, Oh No <N...@charlesfrancis.wanadoo.co.uk> wrote:
>> There has been a new, and substantially larger compilation including new
>> near SN (which should much improve the dataset) , while I wasn't looking
>>
>> http://arxiv.org/abs/0804.4142
>>
>> They have a website where they intend to keep regular updates, and from
>> which the data can be downloaded.
>>
>> http://supernova.lbl.gov/Union/
>>
>> Regards
>>
>> --
>> Charles Francis
>> moderator sci.physics.foundations.
>
>Excellent!
>
>Lucky I was visiting here when you noticed!
>
>I look forward to finishing off my graphical analysis program as the
>nights start drawing in, with the added incentive of knowing that the
>data set is now larger, and growing.
>
>Meanwhile, if you come up with more detailed chi squared results, I
>would be delighted to read about them.
>
Initial result (I am not sure how much I will do, but I want to get a
new graph with the more up to date data, and I feel I may as well used
their cuts, which seem quite robust.
Standard flat space model, Omega=0.29, chi^2 = 311.0
Teleconnection no-Lambda model, Omega = 1.92, chi^2=309.4
So very similar to the 225 points I had from the 04 gold plus Astier
dataset.
I believe you may have a new form of Chalky's law, but for the one which
remains programmed into my system, it gave chi^2=312.4. All very
inconclusive.
Regards
--
Charles Francis
moderator sci.physics.foundations.
Chalky
> near SN (which should much improve the dataset) , while I wasn't looking
>
> http://arxiv.org/abs/0804.4142
A quick perusal indicates that what is really exciting about this is
that not only have they added new data, they have also reinterpreted
all pre-existing data in a consistent format, and applied an identical
quality filter to everything.
> They have a website where they intend to keep regular updates, and from
> which the data can be downloaded.
>
> http://supernova.lbl.gov/Union/
I find the nitty gritty on the available data is at:
http://supernova.lbl.gov/Union/figures/SCPUnion_mu_vs_z.txt
listed as an ASCII table with tab-separated columns: Supernova Name,
Redshift, Distance Modulus, and Distance Modulus Error.
That is certainly all I need for my number cruncher. This makes it
almost too easy. Last time I had to program to translate mag to DM,
skip superfluous data columns, compensate for differences between
different sources, and allow user flexibility to select which sources
could be trusted.
At a rough count this new list also covers ~ 610 Sn1a, which is a huge
improvement on what you could obtain hitherto, even when using relaxed
quality constraints.
Chalky
> Thus spake Chalky <chalkys...@bleachboys.co.uk>
>
>
>
> >On Sep 1, 11:45 am, Oh No <N...@charlesfrancis.wanadoo.co.uk> wrote:
> >> There has been a new, and substantially larger compilation including new
> >> near SN (which should much improve the dataset) , while I wasn't looking
>
> >>http://arxiv.org/abs/0804.4142
>
> >> They have a website where they intend to keep regular updates, and from
> >> which the data can be downloaded.
>
> >>http://supernova.lbl.gov/Union/
>
> >Excellent!
>
> >Lucky I was visiting here when you noticed!
>
> >I look forward to finishing off my graphical analysis program as the
> >nights start drawing in, with the added incentive of knowing that the
> >data set is now larger, and growing.
>
> >Meanwhile, if you come up with more detailed chi squared results, I
> >would be delighted to read about them.
>
> Initial result (I am not sure how much I will do, but I want to get a
> new graph with the more up to date data, and I feel I may as well used
> their cuts, which seem quite robust.
>
> Standard flat space model, Omega=0.29, chi^2 = 311.0
> Teleconnection no-Lambda model, Omega = 1.92, chi^2=309.4
>
> So very similar to the 225 points I had from the 04 gold plus Astier
> dataset.
>
> I believe you may have a new form of Chalky's law, but for the one which
> remains programmed into my system, it gave chi^2=312.4. All very
> inconclusive.
Yes, that is certainly too close to call. What figures did you get for
the best fit offsets in each case? Still similar too I imagine. Of
course, with much more near field data, we should be starting to get
closer to a point where plausible/implausible offsets could be hinted
at by the near field data.
I note the authors have used the method of robust statistics in
interpreting results (as did Wright). I don't recall precisely what
that was, offhand, but it involved de-emphasis of statistical
magnitude outliers.
Oh No
They all got shunted a bit
standard teleconnection chalky
0.172 0.174 0.225
compared to
-0.072 -0.055 0.034
But I took it that this was just due to a change in the value of
Hubble's constant assumed by the data. I think I had Riess's value
programmed in (64.5) while the new data probably assumes something
around 70. I didn't bother to check as this isn't important for my
fits..
>Of
>course, with much more near field data, we should be starting to get
>closer to a point where plausible/implausible offsets could be hinted
>at by the near field data.
It isn't possible to extract Hubble's constant from this data. Iirc we
need additional info about the distance of the galaxies in which the SN
appear.
>I note the authors have used the method of robust statistics in
>interpreting results (as did Wright). I don't recall precisely what
>that was, offhand, but it involved de-emphasis of statistical
>magnitude outliers.
>
Don't know, but they do appear to have gone to some effort to reduce
problems due to the heterogeneous nature of the data, and certainly the
most obvious statistical anomalies I was finding in the previous data
have been removed.
Chalky
> Thus spake Chalky <chalkys...@bleachboys.co.uk>
> >On Sep 1, 4:15 pm, Oh No <N...@charlesfrancis.wanadoo.co.uk> wrote:
> >> Thus spake Chalky <chalkys...@bleachboys.co.uk>
> >> >Meanwhile, if you come up with more detailed chi squared results, I
> >> >would be delighted to read about them.
>
> >> Initial result (I am not sure how much I will do, but I want to get a
> >> new graph with the more up to date data, and I feel I may as well used
> >> their cuts, which seem quite robust.
>
> >> Standard flat space model, Omega=0.29, chi^2 = 311.0
> >> Teleconnection no-Lambda model, Omega = 1.92, chi^2=309.4
>
> >> So very similar to the 225 points I had from the 04 gold plus Astier
> >> dataset.
I should have asked this earlier. What no. of points did you get for
the new set?
> >> I believe you may have a new form of Chalky's law, but for the one which
> >> remains programmed into my system, it gave chi^2=312.4. All very
> >> inconclusive.
>
> >Yes, that is certainly too close to call. What figures did you get for
> >the best fit offsets in each case? Still similar too I imagine.
>
> They all got shunted a bit
>
> standard teleconnection chalky
> 0.172 0.174 0.225
>
> compared to
>
> -0.072 -0.055 0.034
New data seems to have reduced required model offset differences from
standard model for both of us.
difference old to new
standard teleconnection chalky
0.244 0.229 0.259
> But I took it that this was just due to a change in the value of
> Hubble's constant assumed by the data. I think I had Riess's value
> programmed in (64.5) while the new data probably assumes something
> around 70.
Yes. Pretty sure I read 72 (or perhaps 73) yesterday.
72 would give 0.239 difference, and 73 would give 0.269
> >Of
> >course, with much more near field data, we should be starting to get
> >closer to a point where plausible/implausible offsets could be hinted
> >at by the near field data.
>
> It isn't possible to extract Hubble's constant from this data. Iirc we
> need additional info about the distance of the galaxies in which the SN
> appear.
You missed my point here. This offset is merely a nuisance factor
which covers both error in assumed Ho and error in assumed absolute
mag. In model independent format, the most plausible data offset would
be the one which causes the data curve to hit the origin.
> >I note the authors have used the method of robust statistics in
> >interpreting results (as did Wright). I don't recall precisely what
> >that was, offhand, but it involved de-emphasis of statistical
> >magnitude outliers.
>
> Don't know,
The central principle is that, for a graph, you take the median value
of a z range bin, not the mean value, which makes sense.
http://en.wikipedia.org/wiki/Robust_statistics and
http://en.wikipedia.org/wiki/Median (popular explanation)
> but they do appear to have gone to some effort to reduce
> problems due to the heterogeneous nature of the data, and certainly the
> most obvious statistical anomalies I was finding in the previous data
> have been removed.
Absolutely, as indicated by your chi^2 figures. Of course, you really
do have to divide these all by n, to get a realistic indication of how
well the error margins have been tightened up.
Oh No
>On Sep 2, 8:26 am, Oh No <N...@charlesfrancis.wanadoo.co.uk> wrote:
>> Thus spake Chalky <chalkys...@bleachboys.co.uk>
>
>> >On Sep 1, 4:15 pm, Oh No <N...@charlesfrancis.wanadoo.co.uk> wrote:
>> >> Thus spake Chalky <chalkys...@bleachboys.co.uk>
>
>> >> >Meanwhile, if you come up with more detailed chi squared results, I
>> >> >would be delighted to read about them.
>>
>> >> Initial result (I am not sure how much I will do, but I want to get a
>> >> new graph with the more up to date data, and I feel I may as well used
>> >> their cuts, which seem quite robust.
>>
>> >> Standard flat space model, Omega=0.29, chi^2 = 311.0
>> >> Teleconnection no-Lambda model, Omega = 1.92, chi^2=309.4
>>
>> >> So very similar to the 225 points I had from the 04 gold plus Astier
>> >> dataset.
>
>I should have asked this earlier. What no. of points did you get for
>the new set?
307, as they say.
>
>> It isn't possible to extract Hubble's constant from this data. Iirc we
>> need additional info about the distance of the galaxies in which the SN
>> appear.
>
>You missed my point here. This offset is merely a nuisance factor
>which covers both error in assumed Ho and error in assumed absolute
>mag. In model independent format, the most plausible data offset would
>be the one which causes the data curve to hit the origin.
It can't. Magnitude is based on a log law, and goes to -infinity at zero
distance.
> Of course, you really
>do have to divide these all by n, to get a realistic indication of how
>well the error margins have been tightened up.
I suspect they have normalised the error figures at some point. I didn't
actually get a chi^2 of 307, but wasn't far off.
Chalky
> Thus spake Chalky <chalkys...@bleachboys.co.uk>
>
>
>
> >On Sep 2, 8:26 am, Oh No <N...@charlesfrancis.wanadoo.co.uk> wrote:
> >> Thus spake Chalky <chalkys...@bleachboys.co.uk>
>
> >> >On Sep 1, 4:15 pm, Oh No <N...@charlesfrancis.wanadoo.co.uk> wrote:
> >> >> Thus spake Chalky <chalkys...@bleachboys.co.uk>
>
> >> >> >Meanwhile, if you come up with more detailed chi squared results, I
> >> >> >would be delighted to read about them.
>
> >> >> Initial result (I am not sure how much I will do, but I want to get a
> >> >> new graph with the more up to date data, and I feel I may as well used
> >> >> their cuts, which seem quite robust.
>
> >> >> Standard flat space model, Omega=0.29, chi^2 = 311.0
> >> >> Teleconnection no-Lambda model, Omega = 1.92, chi^2=309.4
>
> >> >> So very similar to the 225 points I had from the 04 gold plus Astier
> >> >> dataset.
>
> >I should have asked this earlier. What no. of points did you get for
> >the new set?
>
> 307, as they say.
You are right. I obviously miscounted them (by mental arithmetic),
first time around.
> >> It isn't possible to extract Hubble's constant from this data. Iirc we
> >> need additional info about the distance of the galaxies in which the SN
> >> appear.
>
> >You missed my point here. This offset is merely a nuisance factor
> >which covers both error in assumed Ho and error in assumed absolute
> >mag. In model independent format, the most plausible data offset would
> >be the one which causes the data curve to hit the origin.
>
> It can't. Magnitude is based on a log law, and goes to -infinity at zero
> distance.
Sorry, I should have clarified. The graph of del DM against distance
as used by Wright, where del DM is the difference between observe and
theoretical DM using any arbitrary theoretical reference model. This
allows you to crank up the visual resolution of any resultant graph to
far beyond that employed by the newest authors.
> > Of course, you really
> >do have to divide these all by n, to get a realistic indication of how
> >well the error margins have been tightened up.
>
> I suspect they have normalised the error figures at some point. I didn't
> actually get a chi^2 of 307, but wasn't far off.
I presume from this that they indicated a chi^2/n figure of 1
somewhere. I only skimmed through the paper because all I really
wanted was the resultant figures for each supernova.
Chalky
FRW cosmology, using the retrofit parameters Omega_Lambda and Omega_M,
but this paper has me foxed, since they don't specify those values,
just giving a range of options for Omega_M , Omega_k, and w (the
equation of state parameter).
Can anyone explain how to get from that information to the effective
best fit values of Omega_M and Omega_Lambda that I know how to use?
Chalky
Since my last posting, I have managed to make a little more sense of
this paper, and have worked out that Omega_k must be either
Omega_lambda + Omega_m -1 or 1 - Omega_lambda-Omega_m, and would
appreciate clarification of which of these conclusions is actually
true.
As for the equation of state parameter w, I think it looks like, not
content with the two additional free retrofit parametes Omega_lambda
and Omega_m in GR, some theorists and cosmologists have concluded that
it might be helpful to introduce a still further 'retrofit fudge
factor' which allows Omega_lambda to vary with time.
However, since Kowalski et al have concluded that such additional
retrofit flexibility is not particularly helpful, and is not well
constrained, I don't think it is worth me complicating my own
comparative analysis to accommodate this latest resurrection of the
Ptolemaic trick of arbitrary introduction of epicycles on epicycles,
to achieve compatibility with observation.
As for the underlying reason why I was completely foxed earlier, this
turns out to be because the best fit cosmology Kowalski et al employed
for outlier cuts is completely unrelated to any of the options they
give for a potentially viable cosmology. Instead, it is essentially
the highly implausible (on other grounds) closed dark energy model of
GR that was discussed briefly (and discounted) last year, at
sci.astro.research.
Oh No
>
>
> Since my last posting, I have managed to make a little more sense of
> this paper, and have worked out that Omega_k must be either
> Omega_lambda + Omega_m -1 or 1 - Omega_lambda-Omega_m, and would
> appreciate clarification of which of these conclusions is actually
> true.
Omega_k + Omega_lambda + Omega_m = 1
is an identity for Friedmann cosmologies.
>
> As for the equation of state parameter w, I think it looks like, not
> content with the two additional free retrofit parametes Omega_lambda
> and Omega_m in GR, some theorists and cosmologists have concluded that
> it might be helpful to introduce a still further 'retrofit fudge
> factor' which allows Omega_lambda to vary with time.
>
> However, since Kowalski et al have concluded that such additional
> retrofit flexibility is not particularly helpful, and is not well
> constrained, I don't think it is worth me complicating my own
> comparative analysis to accommodate this latest resurrection of the
> Ptolemaic trick of arbitrary introduction of epicycles on epicycles,
> to achieve compatibility with observation.
>
I agree. w is an additional fudge factor. I find the exercise becomes
meaningless when it is included, especially as there is no solid
theoretical reason for its inclusion. Imv, scientific theory does not
allow the inclusion of arbitrary parameters simply because we do not
have a reason to exclude them. Instead it should be based on what we
know of fundamental structures of matter.
As Von neumann said "with four free parameters, I can model an
elephant. With five, I can make it wiggle its trunk".
Chalky
> On 19 Oct, 18:35, Chalky <chalkys...@bleachboys.co.uk> wrote:
>
>
>
> > Since my last posting, I have managed to make a little more sense of
> > this paper, and have worked out that Omega_k must be either
> > Omega_lambda + Omega_m -1 or 1 - Omega_lambda-Omega_m, and would
> > appreciate clarification of which of these conclusions is actually
> > true.
>
> Omega_k + Omega_lambda + Omega_m = 1
>
> is an identity for Friedmann cosmologies.
Thanks
> > As for the equation of state parameter w, I think it looks like, not
> > content with the two additional free retrofit parametes Omega_lambda
> > and Omega_m in GR, some theorists and cosmologists have concluded that
> > it might be helpful to introduce a still further 'retrofit fudge
> > factor' which allows Omega_lambda to vary with time.
>
> > However, since Kowalski et al have concluded that such additional
> > retrofit flexibility is not particularly helpful, and is not well
> > constrained, I don't think it is worth me complicating my own
> > comparative analysis to accommodate this latest resurrection of the
> > Ptolemaic trick of arbitrary introduction of epicycles on epicycles,
> > to achieve compatibility with observation.
>
> I agree. w is an additional fudge factor. I find the exercise becomes
> meaningless when it is included, especially as there is no solid
> theoretical reason for its inclusion. Imv, scientific theory does not
> allow the inclusion of arbitrary parameters simply because we do not
> have a reason to exclude them. Instead it should be based on what we
> know of fundamental structures of matter.
Yes, and, ideally, understand why, in addition to know.
> As Von neumann said "with four free parameters, I can model an
> elephant. With five, I can make it wiggle its trunk".
Nice one
: )