Proposing hypotheses and designing experiments

[Ariel Haim Hecht]

Hi all,

In response to John’s email from earlier in the week, we should articulate the clear hypotheses that we can evaluate within this group.

The overarching hypothesis could be something like: “Calibrating flow cytometry data with a stable artifact leads to better understanding of the quality of results, and of sources of bias and uncertainty in the measurements.”

There are a number of sources of uncertainty and bias in measurements of fluorescent protein expression, including:

1)    variability within one population of cells,
2)    nominally identical and different culturing process,
3)    expression of different fluorescent proteins,
4)    measurements from nominally identical, nominally similar, and different cytometers

These are the types of issues that we are equipped to explore within this group. We should strive to propose hypotheses and design experiments along these lines.

I think it would also be useful to clarify the limits of the beads that we’re using, and clearly identify what we cannot do with them. Some have already been discussed in this group, but I think it would be useful to reiterate them and make sure that everyone is on the same page:

1)    Beads that are not spectrally matched to a fluorophore cannot be used to establish a generalizable calibration method for comparability of measurements of that fluorophore across different labs or different instruments.

To state this more explicitly, calibration of fluorescent protein expression to the Spherotech beads cannot be used to compare measurements from different instruments or in different labs.

As has been mentioned before, we could explore using the Clontech EGFP and mCherry beads for calibrating measurements of those two fluorescent proteins, with the understanding that the method would be applicable to just those two fluorophores.

2)    We should be careful with units. Since the Spherotech beads do not contain fluorescein, I don’t think we can use MEFL units on their measurements. I think we could use an “MEF” unit, where “MEF” stands for molecules of equivalent fluorophore, and the fluorophore is unknown.

Thanks,

Ariel

[Ross, David J. (Fed)]

I agree that we should have a clear “hypothesis” that we should attempt to test. But I would put the emphasis on ‘clear,’ and to me that means being very specific in identifying: 1) a measureand (mean, median, geometric mean, variance, absolute, ratio, etc.), 2) what exactly we mean by comparable (same instrument over time, different instruments, different labs…), and 3) a quantitative measure of utility/success.

 

So, for example: “Our stakeholders need to be able to use flow cytometry to measure the geometric mean number of GFP molecules (absolute) per cell, with comparability across different labs and instruments with 1.2-fold or better geometric standard deviation. Can we develop a method, using fluorescent beads as a calibrant, that would meet this need?”

 

Now, in that example, I intentionally picked a target that I think many of you would be very skeptical of – so the answer to that one is probably, “No.”

So, a more realistic example, might be: “Our stakeholders need to be able to use flow cytometry to measure the ratio of two bacterial cell populations’ geometric mean GFP copy number, with comparability across different labs and instruments with 2-fold or better geometric standard deviation. Can we develop a method, using fluorescent beads as a calibrant, that would meet this need?”

 

I think if we can make a list of specific, needs-driven hypothesis statements along the lines of these examples, it will help us communicate with each other more clearly, and provide specific targets that we can think about addressing.

 

 

But before going this way, I would like to get consensus from the group. This is, after all, my first week of active participation, so it may be that the group already has an alternate (but equally clear) way to state a set of potential objectives/hypotheses.

 

So, I ask all of you to please respond with a Yes/No/other on the question of whether you think a list of needs-driven hypothesis statements, similar to my examples, is the right way to go.

 

Thanks-

--
You received this message because you are subscribed to the Google Groups "SBSC Flow Cytometry Working Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sbsc-flow-cytom...@googlegroups.com.
To post to this group, send email to sbsc-flow...@googlegroups.com.
Visit this group at https://groups.google.com/group/sbsc-flow-cytometry.
To view this discussion on the web visit https://groups.google.com/d/msgid/sbsc-flow-cytometry/7257B42E-54C2-41A0-8B6A-FA29CFA5A130%40stanford.edu.
For more options, visit https://groups.google.com/d/optout.

[Ross, David J. (Fed)]

Some additional comments on Ariel’s comments:

 

1)      “…calibration of fluorescent protein expression to the Spherotech beads cannot be used to compare measurements from different instruments or in different labs.” Taking this as a general statement, I would disagree with it. The key question to me is not whether or not some method/calibrant can be used to compare different measurements, but how well it allows you to compare. So, for this case, I think Spherotech beads CAN be used to compare measurements from different instruments/labs, but it may be that the level of comparability it would give you is 1000-fold geometric standard deviation (or pick your favorite quantitative measure of comparability), which would really not be that useful.

With regard to fluorescent protein beads: The fluorescent properties of a molecule (protein, fluorescein, etc.) are not just determined by the molecule, but also by the medium it is in. So, GFP in a bead is not going to have the same excitation and emission spectra as GFP in an E. coli. So, qualitatively, there is no difference between GFP beads and Rainbow beads. Quantitatively, there probably is a difference: the GFP-beads spectra is likely to be closer than the Rainbow-beads spectra to the GFP-coli spectra. But whether or not that makes the difference between ‘good enough’ and ‘not good enough’ depends on the details – starting with what level of comparability do our stakeholders need?

 

 

There is a story about German and American nuclear scientists comparing notes after WWII. The German said, “The German program was very much superior to the American program. We were able to enrich uranium so that it was nearly pure U-235.” The American responded, “We figured out what level of enrichment we would need for a reliable bomb, and we stopped there.”

 

From: sbsc-flow...@googlegroups.com [mailto:sbsc-flow...@googlegroups.com] On Behalf Of Ariel Haim Hecht
Sent: Thursday, December 08, 2016 6:54 PM
To: sbsc-flow...@googlegroups.com
Subject: [sbsc-fc] Proposing hypotheses and designing experiments

 

Hi all,

--

[Ariel Haim Hecht]

Hey Dave,

All fair points. 

I think your key question is correct, the question is how well a method works. The problem with using Spherotech beads is that I don’t know how we’d ever be able to know how well, as a generalizable method, it will work. There are unknowable sources of uncertainty, there’s no traceability to a standard. It’s one thing to see if it works within our small group, but it’s an entirely different proposition to encourage the community at large to do it. The method would not be robust enough to be generalizable. As NIST, we have a responsibility not to lead people astray. If we encourage people to rely on a measurement, that we don’t know how reliable it is, we can be doing more harm than good. We have to be very careful. We will not be able to make a general statement like “this method will give you x-fold geometric standard deviations of comparability" because it is not traceable to a standard, we don’t know the values of the uncertainties, and we won’t be able to validate it across all of the potential use cases. 

With regards to the fluorescent protein beads, you’re absolutely right that GFP on/in a bead is different than GFP in a cell. Fluorescence is very sensitive to environmental parameters. GFP-beads would be better than Rainbow beads, but you’re right that they may not be good enough. Fluorescence spectroscopy is hard, calibrating fluorescence spectroscopy is harder. It also highlights why people tend to report fluorescence as ratios between cell populations, because it controls for a lot of the condition-dependent spectroscopic issues. 

The question of stakeholder needs is also very important. We do need to figure out what our stakeholders need. There seem to be two somewhat orthogonal questions here: What do our stakeholders need? and What can we possibly deliver to our stakeholders? The work that we should do probably lies within the intersection of those two answer spaces.

-Ariel

To view this discussion on the web visit https://groups.google.com/d/msgid/sbsc-flow-cytometry/BY1PR09MB04533935C36A44E0B73D5295E7870%40BY1PR09MB0453.namprd09.prod.outlook.com.

[Sebastián Castillo Hair]

Ariel,

What you mention about Spherotech beads not being traceable to a NIST standard is interesting to me, and something I haven't considered before. Are there any other commercial beads that are traceable that we can use? We don't have to use Spherotech beads.

I don't know how many resources NIST is willing to put into developing new calibration artifacts, but an alternative would be to develop a standard strain that expresses a fluorescent protein, and quantify the number of proteins per cell with a method similar to this: https://www.ncbi.nlm.nih.gov/pubmed/21907700 (I'm not an expert in microscopy or single molecule measurements, so we could probably find a better method). Then, the strain could be frozen and shipped ready to be measured in a flow cytometer. Of course, this would require characterizing how well cells survive storage and shipping conditions, as we discussed before.

Regarding the stakeholder's needs, I would argue that synthetic biologists need a flow cytometry measurement technique that, when used in different labs with the same fluorescent strain and the same culturing conditions, result in the same number (and we can qualify this last part with "within 2x-fold of each other" or some other number. 1.5x-2x seems to me like a good compromise between what I think is realistic and useful). Even better would be to have a measurement technique that gives us the exact number of molecules of reporter per cell, but that seems beyond the scope of this group's work for the moment (although shipping frozen cells as I described above could potentially achieve this).

To view this discussion on the web visit https://groups.google.com/d/msgid/sbsc-flow-cytometry/A57B5022-04FB-4D3B-A94D-063250127157%40stanford.edu.

[Ariel Haim Hecht]

Hi Sebastian,

Thanks. Yes, the issue of traceability is a bit of a subtle one, but very important. To the best of my knowledge, there are not any beads that are traceable to a standard. 

The question you ask about a biological reference material is also a good question. The biggest challenge with a biological reference material is stability over time. Practically, there are a ton of issues, as you mention and as we have discussed, because cells are living things, they don’t stay static. NIST has never developed a biological reference material, because there are so many practical issues that we haven’t figured out yet how to solve. 

Regarding stakeholder needs, that sounds like a lovely goal, but one that is probably unachievable given any sort of realistic, generalizable constraints. The “Effects of Sample Dilution” SBC Technical Report that Nicholas sent out in the other thread provides an excellent example. Look at Figure 1A. The measured fluorescence of a population changes by more than 3x based on the dilution factor alone, and that’s with everything else held constant. 

-Ariel

To view this discussion on the web visit https://groups.google.com/d/msgid/sbsc-flow-cytometry/CAAXZg%3DYYNNG7TJay-y4%3Dfsnny64cYBupBTEh9POV%2Bfw0jprCpA%40mail.gmail.com.

[Ross, David J. (Fed)]

From Paul DeRose (Fluorescence Guy at NIST) in response to my question on whether there are fluorescent beads that are NIST traceable:

 

“Yes, we now offer ERF assignments to members of the Quantitative Flow Cytometry Consortium.

This is being done as individual CRADAs with each company interested, all calibration bead manufacturers so far.

We assign the mean ERF units/bead. The ERF units are the number of molecules of reference fluorophore

corresponding to the fluorescence intensity of the bead. The SI unit is moles of fluorophore. During this process

we also measure the bead concentration and fluorophore concentration of reference solutions,

both traceable to moles/dm^3.

 

Note that we do not have a calibration bead SRM.

We are instead certifying the ERF values for the customer’s beads.”

To view this discussion on the web visit https://groups.google.com/d/msgid/sbsc-flow-cytometry/07F2BADD-6A12-4BA9-B9C5-FA28AB41CF26%40stanford.edu.