Re: CytoSeg help

[Rick Giuly]

OK, a couple questions:

In this folder:
output/voxelOutput/mitochondria/composite
Do you have some images with some green highlights? How many images?

Is there anything in this folder:
output/voxelOutput/blobs/composite
If so, how many images?

-Rick

kurt weiss wrote:
>
> You cleared up a few things, but I still have a misunderstanding. I did
> not alter the _train or _seg folder contents. They both contain the
> exact same matching set of images that you provided. All I changed was
> the input folder (which I now realized I did not actually need to
> resize, but I did anyway). So the _train and _seg folders have your
> 16-image training sets (that match perfectly), and the input folder has
> 100 of my SBF-SEM images 700x700 with 5nm/pixel xy resolution (z=20nm).
> But I'm getting the
> IOError: [Errno 2] No such file or directory: 'output/Volumes,default_
> testInputProbabilityMapAllFastMarchBlobs.pickle'
> error.
>
> Unless of course, the first 15 images of the input folder must also
> match the train and seg images? But I was under the impression that we
> could train with one image set (yours or mine or say DataSetA) then use
> that same training set to segment/quantify DataSetB and DataSetC, so
> they were both tested under the same unbiased conditions - is this the
> right idea?
>
> Please help me get that pickle!
>
> Thank you,
> Kurt
>
>
> On Wed, Oct 31, 2012 at 12:15 PM, Rick Giuly <rgi...@gmail.com
> <mailto:rgi...@gmail.com>> wrote:
>
>
> Hi Kurt, these are good questions that others will probably have. Is
> it OK with you if I cc this thread to the forum:
> http://groups.google.com/__group/cytoseg
> <http://groups.google.com/group/cytoseg> ?
>
> The error you got occurs when no valid contours are detected.
>
> Here's the reason that it's happening:
> Since you are replacing the training image volume, it's very
> important that you also replace the training segmentation with a
> segmentation that matches your training data.
>
> For a quick test you can train with only about 6 slices. However,
> for better accuracy, you will need about 30 slices of training data.
>
> Note 1: The data (training and input) should be resized (if needed)
> so that the voxel size in about 5nm in XY. This will make detection
> more accurate.
>
> Note 2: The training segmentation should have the same dimension as
> the training image volume. It's important that the two align exactly
> for the learning to work properly.
>
> However, the training segmentation doesn't have to have the same
> dimension as the input image volume. The input is typically much larger.
>
> Usually a bit of tweaking is needed to get good results, so if you
> let me know how things go, I can give suggestions.
>
> Hope that helps!
>
> -Rick
>
>
> kurt weiss wrote:
>
> Hello again Rick,
>
> I have cytoseg setup and working (Ubuntu 12.04) for the example
> images
> you have provided. It looks good, but I need some help
> implementing with
> my own data: If I am correct, I need to use Imod to generate my own
> training sets - correct? (I'm working on learning this step now).
> However, I though I could eliminate one source of error, and
> save some
> time by simply using your training data, since I also have SBF-SEM
> images of similar resolution. I found the note that training
> sets must
> be the same dimensions as the input data set, so I cropped my
> input data
> to 700 x 700 pixels for a 100 image stack (same as your training
> set). I
> then placed this stack in the folder
> cytoseg-read-only/cytoseg/__testing/data/example/input (and
> removed your
> example stack). I then ran it with this input:
>
> krw@ubuntu:~/cytoseg-read-__only/cytoseg/testing$ python
> run_pipeline_test.py data/example/input output
> --trainingImage=data/example/__train_images
> --trainingSeg=data/example/__train_seg
> --voxelTrainingLowerBound=*,*,__*
> --voxelTrainingUpperBound=*,*,__*
> --voxelProcessingLowerBound=*,__*,*
> --voxelProcessingUpperBound=*,__*,*
> --contourTrainingLowerBound=*,__*,*
> --contourTrainingUpperBound=*,__*,*
> --contourProcessingLowerBound=__*,*,*
> --contourProcessingUpperBound=__*,*,* --accuracyCalcLowerBound=*,*,*
> --accuracyCalcUpperBound=*,*,* --labelConfigFile=settings3.py
> --voxelWeights=0.0130,0.00064 --contourListWeights=7,1
> --contourListThreshold=0.8 --step1 --step2
>
> and eventually get the ERROR:
>
> Traceback (most recent call last):
> File "run_pipeline_test.py", line 348, in <module>
> contourStepSet()
> File "run_pipeline_test.py", line 128, in contourStepSet
> contourChunkSize)
> File "../batch_process.py", line 103, in batchProcessContours
> runSteps(**params)
> File "../run_steps.py", line 129, in runSteps
> detector.run(steps)
> File "../segmentation_manager.py", line 211, in run
> self.componentDetector.__runWrite3DBlobsVolume()
> File "../component_detector.py", line 2300, in
> runWrite3DBlobsVolume
> 'AllFastMarchBlobs')
> File "../data_viewer.py", line 443, in getPersistentVolume_old
> return self.getPersistentObject(('__Volumes', name))
> File "../data_viewer.py", line 432, in getPersistentObject
> node = self.mainDoc.dataTree.__getSubtree(pathToNode)
> File "../tree.py", line 240, in getSubtree
> self.readSubtree(__pathToSubtree)
> File "../tree.py", line 227, in readSubtree
> f = open(fullFilename)
> IOError: [Errno 2] No such file or directory:
> 'output/Volumes,default___testInputProbabilityMapAllFast__MarchBlobs.pickle'
>
> I have attached all of the the terminal text with my input
> re-called at
> the bottom since I could not see the entire terminal output. Let
> me know
> if there is something horribly wrong with my approach, and
> where you
> might have other documentation explaining where/how to change
> necessary
> parameters to run cytoseg with my own data.
>
> Thank you very much,
> Kurt
>
> PS If you would prefer that I move my questions to a public forum to
> reduce your time answering questions, I can do that as well, but I
> assume I'm still at a point most people don't need help with.
>
> On Wed, Oct 10, 2012 at 2:29 PM, Rick Giuly <rgi...@gmail.com
> <mailto:rgi...@gmail.com>
> <mailto:rgi...@gmail.com <mailto:rgi...@gmail.com>>> wrote:
>
>
> Hi Kurt,
>
> Interesting project. I gave you a quick call but didn't get
> through.
> My number is 858-531-0069 <tel:858-531-0069>
> <tel:858-531-0069 <tel:858-531-0069>> if you'll like to chat
>
> sometime.
>
> To answer your questions:
>
> Here's the difference: Ilastik uses only pixel
> classification (Step
> 1). In my experience, this leads to noisy results that
> require a
> large amount of manual correction (and often are not smooth
> in 3D).
> The fundamental problem is that local texture does not always
> determine if pixel is really a part of a particular object
> or not.
>
> Cytoseg uses 3 steps of automatic processing. The first is
> pixel
> classification, much like Ilastik. The second step detects
> contours
> at multiple thresholds and uses another classification step to
> decide which contours most likely belong to the object of
> interest.
> By analysing contour geometry, Cytoseg can make decisions
> based on
> more that just local texture. Effectively, the reduces the
> about of
> incorrect objects detected after threshold of the pixel
> classification. Results after step 2 still typically lack
> some 3D
> smoothness. The 3rd step helps to increase the smoothness and
> typically increases accuracy also.
>
> Cytoseg is still a command line based tool. To install it,
> you'll
> need to follow the installation instructions. It's best to
> first try
> the example that's on the Cytoseg site, and then modify it
> slightly
> for your task.
>
> Also a warning: it typically takes significant time for
> Cytoseg to
> process a dataset. I usually start the process and leave it
> for a
> while. We're working on a faster parallel version, but it's not
> quite available yet.
>
> Let me know if you have any more questions
>
> If you are interested in using Cytoseg, I can help you
> directly with
> processing your data
>
> Good luck,
> -Rick
>
> Rick Giuly
> National Center for Microscopy and Imaging Research
>
>
> On 10/10/2012 10:57 AM, kurt weiss wrote:
>
> Dear Richard,
>
> I am a postdoc in the lab of Chiara Cirelli and Giulio
> Tononi
> working on reconstruction and segmentation of SBF-SEM
> data to
> study the ultrastructural effects of sleep loss. We
> are using the
> Gatan 3-view system and have been following the work of
> your lab
> closely. I would greatly appreciate some help from you
> in my
> research as both myself and the lab are new to these
> techniques,
> and we are interested in using your software to quantify
> mitochondria in rat and fly tissues. I have a few
> questions so
> I'll just get to it, but if you would prefer to answer
> via voice
> (my phone: 262 617 4973 <tel:262%20617%204973>
> <tel:262%20617%204973>) or Gchat (i'm
>
> usually invisible, but online), just let me know.
>
> 1. How is the Cytoseg process fundamentally different
> from that
> used in the Ilastik software (http://www.ilastik.org/)?
> (both seem to use 2-D random forest classification,
> followed by
> seeding contours to render in 3D).
>
> 2. Where exactly do I find the cytoseg software/source
> code? I
> have been to this site:
> http://code.google.com/p/__cytoseg/source/checkout
> <http://code.google.com/p/cytoseg/source/checkout>
> and tried running
> */http/*://cytoseg.googlecode.__com/svn/trunk/
> <http://cytoseg.googlecode.com/svn/trunk/>
>
> <http://cytoseg.googlecode.__com/svn/trunk/
> <http://cytoseg.googlecode.com/svn/trunk/>> cytoseg-read-only
>
>
> from the command line, but I get an error and I have
> not been able
> to use the software or find the source code.
>
> 3. Will you be attending SFN this weekend?
>
> Thank You for you help.
> Regards,
> Kurt
>
>
>
> ___
> Kurt Weiss, PhD
> Postdoctoral Fellow, Cirelli/Tononi Lab
> University of Wisconsin Madison - Psychiatry Dept.
> 6001 Research Blvd
> Madison, WI 53719
>
> Ph: (262) 617 4973 <tel:%28262%29%20617%204973>
> <tel:%28262%29%20617%204973>
> Email: kurtrich...@gmail.com
> <mailto:kurtrich...@gmail.com>
> <mailto:kurtrichardweiss@__gmail.com
> <mailto:kurtrich...@gmail.com>>
>
>
> .
>
>
>
>
>
> --
> .
>
>
>
>
>
> --
> .

[Rick Giuly]

Hi Kurt,

Got it, this means that the process is running OK. The issue is just
accuracy. What you'll need to do next is create a training image stack
and a training segmentation stack directly from your data. The learning
is very sensitive to the properties of the data. You can start with
about 6 slices for testing and then move to about 30 later for better
accuracy. You'll need to make sure that there are good representative
mitochondria within the training volume that you choose from your volume.

You won't need to create the blobs folder. It should show up
automatically. The process just has to be a little more accurate for all
of the stages to complete.

When it's working more accurately, those green highlights should show up
more on the mitochondria and less on outer cell membranes.

Also, with our data, the intensity varied through the stack, probably
from the physics of imaging. So, I ran autocontrast on each slice to
normalize them. This might be useful, but it's not required.


-Rick


kurt weiss wrote:
> output/voxelOutput/mitochondria/composite : contains 100 images (my
> full input) with the green highlights around membranes.
>
> outtput/voxelOutput : contains folders called "mitochondria" "other"
> and "training" but there is no blobs folder. The contents of the 'other'
> folder is the negative of the mito folder. So I guess I'll start by
> creating a blobs folder? by the way, are you calling all membrane
> mitochondria (as in mito plus all possible false positives)? and blobs
> are actual bona-fide mitochondria?
>
> -Kurt

>
> On Wed, Oct 31, 2012 at 1:24 PM, Rick Giuly <rgi...@gmail.com
> <mailto:rgi...@gmail.com>> wrote:
>
>
> OK, a couple questions:
>
> In this folder:

> output/voxelOutput/__mitochondria/composite

> Do you have some images with some green highlights? How many images?
>
> Is there anything in this folder:

> output/voxelOutput/blobs/__composite

> If so, how many images?
>
> -Rick
>
> kurt weiss wrote:
>
>
> You cleared up a few things, but I still have a
> misunderstanding. I did
> not alter the _train or _seg folder contents. They both contain the
> exact same matching set of images that you provided. All I
> changed was
> the input folder (which I now realized I did not actually need to
> resize, but I did anyway). So the _train and _seg folders have your
> 16-image training sets (that match perfectly), and the input
> folder has
> 100 of my SBF-SEM images 700x700 with 5nm/pixel xy resolution
> (z=20nm).
> But I'm getting the
> IOError: [Errno 2] No such file or directory:
> 'output/Volumes,default_

> testInputProbabilityMapAllFast__MarchBlobs.pickle'

> error.
>
> Unless of course, the first 15 images of the input folder must also
> match the train and seg images? But I was under the impression
> that we
> could train with one image set (yours or mine or say DataSetA)
> then use
> that same training set to segment/quantify DataSetB and DataSetC, so
> they were both tested under the same unbiased conditions - is
> this the
> right idea?
>
> Please help me get that pickle!
>
> Thank you,
> Kurt
>
>
> On Wed, Oct 31, 2012 at 12:15 PM, Rick Giuly <rgi...@gmail.com
> <mailto:rgi...@gmail.com>

> <mailto:rgi...@gmail.com <mailto:rgi...@gmail.com>>> wrote:
>
>
> Hi Kurt, these are good questions that others will probably
> have. Is
> it OK with you if I cc this thread to the forum:

> http://groups.google.com/____group/cytoseg
> <http://groups.google.com/__group/cytoseg>

> cytoseg-read-only/cytoseg/____testing/data/example/input (and

>
> removed your
> example stack). I then ran it with this input:
>

> krw@ubuntu:~/cytoseg-read-____only/cytoseg/testing$ python
> run_pipeline_test.py data/example/input output
> --trainingImage=data/example/____train_images
> --trainingSeg=data/example/____train_seg
> --voxelTrainingLowerBound=*,*,____*
> --voxelTrainingUpperBound=*,*,____*
> --voxelProcessingLowerBound=*,____*,*
> --voxelProcessingUpperBound=*,____*,*
> --contourTrainingLowerBound=*,____*,*
> --contourTrainingUpperBound=*,____*,*
> --contourProcessingLowerBound=____*,*,*
> --contourProcessingUpperBound=____*,*,*

> --accuracyCalcLowerBound=*,*,*
>
> --accuracyCalcUpperBound=*,*,*
> --labelConfigFile=settings3.py
> --voxelWeights=0.0130,0.00064 --contourListWeights=7,1
> --contourListThreshold=0.8 --step1 --step2
>
> and eventually get the ERROR:
>
> Traceback (most recent call last):
> File "run_pipeline_test.py", line 348, in <module>
> contourStepSet()
> File "run_pipeline_test.py", line 128, in
> contourStepSet
> contourChunkSize)
> File "../batch_process.py", line 103, in
> batchProcessContours
> runSteps(**params)
> File "../run_steps.py", line 129, in runSteps
> detector.run(steps)
> File "../segmentation_manager.py", line 211, in run

> self.componentDetector.____runWrite3DBlobsVolume()

>
> File "../component_detector.py", line 2300, in
> runWrite3DBlobsVolume
> 'AllFastMarchBlobs')
> File "../data_viewer.py", line 443, in
> getPersistentVolume_old

> return self.getPersistentObject(('____Volumes',

> name))
>
> File "../data_viewer.py", line 432, in
> getPersistentObject
> node =

> self.mainDoc.dataTree.____getSubtree(pathToNode)

>
> File "../tree.py", line 240, in getSubtree

> self.readSubtree(____pathToSubtree)

>
> File "../tree.py", line 227, in readSubtree
> f = open(fullFilename)
> IOError: [Errno 2] No such file or directory:

> 'output/Volumes,default_____testInputProbabilityMapAllFast____MarchBlobs.pickle'

> http://code.google.com/p/____cytoseg/source/checkout
> <http://code.google.com/p/__cytoseg/source/checkout>

>
> <http://code.google.com/p/__cytoseg/source/checkout
> <http://code.google.com/p/cytoseg/source/checkout>>
> and tried running

> */http/*://cytoseg.googlecode.____com/svn/trunk/
> <http://cytoseg.googlecode.__com/svn/trunk/
> <http://cytoseg.googlecode.com/svn/trunk/>>
>
> <http://cytoseg.googlecode.____com/svn/trunk/

>
> <http://cytoseg.googlecode.__com/svn/trunk/
> <http://cytoseg.googlecode.com/svn/trunk/>>> cytoseg-read-only
>
>
> from the command line, but I get an error and
> I have
> not been able
> to use the software or find the source code.
>
> 3. Will you be attending SFN this weekend?
>
> Thank You for you help.
> Regards,
> Kurt
>
>
>
> ___
> Kurt Weiss, PhD
> Postdoctoral Fellow, Cirelli/Tononi Lab
> University of Wisconsin Madison - Psychiatry Dept.
> 6001 Research Blvd
> Madison, WI 53719
>
> Ph: (262) 617 4973
> <tel:%28262%29%20617%204973> <tel:%28262%29%20617%204973>
> <tel:%28262%29%20617%204973>
> Email: kurtrich...@gmail.com
> <mailto:kurtrich...@gmail.com>
> <mailto:kurtrichardweiss@__gmail.com
> <mailto:kurtrich...@gmail.com>>

> <mailto:kurtrichardweiss@
> <mailto:kurtrichardweiss@>__gma__il.com <http://gmail.com>

>
> <mailto:kurtrichardweiss@__gmail.com
> <mailto:kurtrich...@gmail.com>>>
>
>
> .
>
>
>
>
>
> --
> .
>
>
>
>
>
> --
> .
>
>
>
>
>

> --
> .