Re: Adding new traced neurons to myneurons

[Gregory Jefferis]

Hi Chris,

In brief, start R,

# load the nat package
library(nat)

# load MyNeurons
load(url("http://flybrain.mrc-lmb.cam.ac.uk/si/grosjean11/MyNeuronsFCIR.rda"))

# read in your amira neurons
new_neurons=read.neurons("/dir/containing/my/neurons")

MyNeurons=c(MyNeurons,new_neurons)

The only tricky bit is adding a data.frame containing useful metadata

You could do something like this:

newdf=read.table(text="
Glomerulus Sex PNType source
DA1 M iPN Potter2015
DL1 F iPN Potter2015", head=T)

or make a spreadsheet e.g. in Excel, save as .csv and use read.csv

newdf=read.csv("sample_metadata.csv", row.names="File")

There must be one entry per tracing file - see attached for an example.

Then you need to do:

new_neurons=read.neurons("/dir/containing/my/neurons", df=newdf)
MyNeurons=c(MyNeurons,new_neurons)

Best,

Greg.

PS we've just started a google group to help with this kind of thing and
I've taken the liberty of forwarding this response since it may be
useful to others.

On 6 Apr 2015, at 22:51, Christopher Potter wrote:

> We have a few new traced and warped PNs that we’d like to directly
> compare to the other PNs. We’ve followed the detailed instructions
> on your website related to the Grosjean, Benton et al publication, but
> are a bit stymied as to how to incorporate our traced PN files into
> the master dataset for comparison.
>
> Specifically, 1) do we need to convert our Amira traced PN files into
> another format? If so, how do you do the conversions?
> 2) How do we then take this traced file, and add it into the PN
> database for the cluster analysis?
>
> This is likely a standard protocol in your lab, so please feel free to
> pass me onto a lab member!
>
> Thanks!
>
> Cheers,
> Chris


--
Gregory Jefferis, PhD
Division of Neurobiology
MRC Laboratory of Molecular Biology
Francis Crick Avenue
Cambridge Biomedical Campus
Cambridge, CB2 OQH, UK

http://www2.mrc-lmb.cam.ac.uk/group-leaders/h-to-m/g-jefferis
http://jefferislab.org
http://flybrain.stanford.edu

[Sonia Chin]

Hi Greg,

Sonia here from Chris's lab.  I was able to add some new traced neurons to the existing dataset with the aforementioned code, but I had to remove the initial 

MyNeurons=c(MyNeurons,new_neurons) 

command because it looks like the program couldn't add the subsequent iterations of new_neurons when those file names were already added. When I look up tail(MyNeurons), it looks like R properly added the files and dataframe.   

I'm not sure if losing the initial command somehow messed things up because now when I try to run the new file, with new neuron traces included, through some other code, it gets stuck on a portion that (I think) starts to compare neurons to other neurons (nblast_allbyall function for the selected lateral horn region - line 41 of the code we've been previously discussing).  R is saying "Error: Cloud has no points" when it comes to this line.

Is there something else I need to do when adding new traces?  Did eliminating that first MyNeurons command somehow cause the dimensions of the data to get wonky?

Thanks for your help!

Sonia 

[Gregory Jefferis]

Dear Sonia,

On 16 Oct 2015, at 21:08, Sonia Chin <aitc...@gmail.com> wrote:

Sonia here from Chris's lab.  I was able to add some new traced neurons to the existing dataset with the aforementioned code, but I had to remove the initial 

MyNeurons=c(MyNeurons,new_neurons) 

command because it looks like the program couldn't add the subsequent iterations of new_neurons when those file names were already added. When I look up tail(MyNeurons), it looks like R properly added the files and data frame.   

you’re right should only run this command once. You are not allowed to put two or more neurons with the same name into a neuronlist. In the code discussion that you quoted, my intention was that the code blocks 

# read in your amira neurons 
new_neurons=read.neurons("/dir/containing/my/neurons") 

MyNeurons=c(MyNeurons,new_neurons) 

and then

newdf=read.table(text=" 
   Glomerulus Sex PNType source 
   DA1 M iPN Potter2015 
   DL1 F iPN Potter2015", head=T) 

or make a spreadsheet e.g. in Excel, save as .csv and use read.csv 

newdf=read.csv("sample_metadata.csv", row.names="File") 

There must be one entry per tracing file - see attached for an example. 

Then you need to do: 

new_neurons=read.neurons("/dir/containing/my/neurons", df=newdf) 
MyNeurons=c(MyNeurons,new_neurons) 

were an either/or alternative.

I'm not sure if losing the initial command somehow messed things up because now when I try to run the new file, with new neuron traces included, through some other code, it gets stuck on a portion that (I think) starts to compare neurons to other neurons (nblast_allbyall function for the selected lateral horn region - line 41 of the code we've been previously discussing).  R is saying "Error: Cloud has no points" when it comes to this line.

It sounds like you may have some neurons without points. You can try something like:

points_per_neuron=sapply(MyNeurons, function(x) nrow(x$d))

points_per_neuron

sum(any(points_per_neuron==0))

to see how many points are in each neuron and check if any neurons have a small number of points (fewer than 10 or so will cause trouble for nblast).

Best wishes,

Greg.

[Sonia Chin]

Just discussed with Chris.  Is it possible that since I registered according to a different template brain - a cropped version of the JFRC template suggested for nc82 staining - that this could be a bridging problem?  The LH/MBC mask included in Analysis Suite no longer fits.

[Dr Gregory Jefferis]

Dear Sonia,

On 16 Oct 2015, at 21:31, Sonia Chin wrote:

> Just discussed with Chris.  Is it possible that since I registered
> according to a different template brain - a cropped version of the
> JFRC template suggested for nc82 staining - that this could be a
> bridging problem?  The LH/MBC mask included in Analysis Suite no
> longer fits.

Thanks for sharing your data off list. The problem is simply that you as
you say you have used a cropped template brain, rather than the original
JFRC2 template brain. Therefore you need to do a little extra work to
combine your neurons with the PN dataset in FCWB template space. In
outline

# read in your amira neurons
new_neurons=read.neurons("/dir/containing/my/neurons")

# fix X offset introduced by cropping the JFRC2 template brain
new_neurons=new_neurons+c(240.126,0,0)

library(nat.flybrains)
new_neurons.FCWB=xform_brain(new_neurons, sample=JFRC2, ref=FCWB)

Now you they are in the right spatial frame of reference to be combined
with these neurons

load(url("http://flybrain.mrc-lmb.cam.ac.uk/si/grosjean11/MyNeuronsFCIR.rda"))

Hope this works! Best,

Greg.

--
Gregory Jefferis, PhD Tel: +44 1223 267048

[Sonia Chin]

Hi Greg,

I think something in the code just mentioned above needs an updated version of CMTK.  I updated using the instructions on your website using FIJI (Install CMTK in Plugins), but I am getting the following:

Error in cmtk.bindir(check = TRUE) : 
  Cannot find CMTK. Please install from http://www.nitrc.org/projects/cmtk and make sure that it is your path!

Pardon my ignorance, but I'm not sure where the download must go for R to be able to access it.

Thanks for your help and patience,

Sonia 

On Wednesday, April 15, 2015 at 2:35:26 PM UTC-4, Gregory Jefferis wrote:

[Greg Jefferis]

Dear Sonia,

On Tuesday, October 20, 2015 at 10:54:56 PM UTC+1, Sonia Chin wrote:

 

I think something in the code just mentioned above needs an updated version of CMTK.

transforming points using CMTK registrations depends on nat package finding CMTK. Not specifically an updated version of CMTK, just any version of CMTK that it can find.

 

 I updated using the instructions on your website using FIJI (Install CMTK in Plugins), but I am getting the following:

Error in cmtk.bindir(check = TRUE) : 
  Cannot find CMTK. Please install from http://www.nitrc.org/projects/cmtk and make sure that it is your path!

Pardon my ignorance, but I'm not sure where the download must go for R to be able to access it.

The Fiji CMTK GUI plugin installs CMTK inside Fiji essentially solely for its own use. 

In order to use CMTK with nat to transform neurons, you must either

1.  (Recommended) install it by going to the web page mentioned in that error message. I am not sure which platform you are using, but on mac for example, this file is the version I would currently recommend:

https://www.nitrc.org/frs/download.php/6806/CMTK-3.2.2-MacOSX-10.6-x86_64.dmg

which is the last version to have an automated installer. You could have found the file yourself by clicking on the "See All Files" button.

Once you have run the installer, you can check that it is seen by nat's cmtk.bindir() function.

2. Alternatively, you use the cmtk.bindir() funciton to tell nat where to find CMTK, something like this for my machine:

cmtk.bindir("/Applications/Fiji.app/bin/cmtk/", set = T, check = T)

It will remember this for the rest of the session.

Best,

Greg.

[Sonia Chin]

Hi Greg,

Thanks for all your help with this.  

The bridging code you provided puts everything into the same space in new_neurons.FCWB.  The next thing to do is to add newdf (dataframe with all the neuron information) to new_neurons.FCWB.  Previously, you had used read.neurons, and I think that put everything in the right data structure to add newdf through df=newdf tagged into the read.neurons arguments.  However, I'm having trouble adding a dataframe to new_neurons.FCWB.

Since read.neurons generates a neuronlist with an optional dataframe, I thought turning new_neurons.FCWB into a neuronlist might do the trick so I ran this code:

new_neurons=neuronlist(new_neurons.FCWB, df = newdf)

However, it doesn't seem to add the newdf information properly.  Instead, I get a data frame added with the appropriate number of rows but 0 columns.  How do I properly add newdf to new_neurons.FCWB?

Again, thanks for all your help and quick responses,

Sonia

[Greg Jefferis]

Dear Sonia,

On Friday, October 23, 2015 at 8:04:40 PM UTC+1, Sonia Chin wrote:

The bridging code you provided puts everything into the same space in new_neurons.FCWB.

Great.

 

 The next thing to do is to add newdf (dataframe with all the neuron information) to new_neurons.FCWB.  Previously, you had used read.neurons, and I think that put everything in the right data structure to add newdf through df=newdf tagged into the read.neurons arguments.  

and you can still do exactly the same at the start of the script i.e.

new_neurons=read.neurons("/dir/containing/my/neurons", df = newdf) 

 

However, I'm having trouble adding a dataframe to new_neurons.FCWB.

Since read.neurons generates a neuronlist with an optional dataframe, I thought turning new_neurons.FCWB into a neuronlist might do the trick so I ran this code:

new_neurons=neuronlist(new_neurons.FCWB, df = newdf)

If you already have a neuronlist, the preferred way to set its data.frame is actually like this

data.frame(new_neurons)=newdf

However there are a variety of ways of achieving the same goal and 

new_neurons=as.neuronlist(new_neurons.FCWB, df = newdf)

would also do the trick. It so happens that you cannot do this with the neuronlist function because that is designed to construct a neuronlist from a number of individually specified neurons i.e.

myneuronlist=neuronlist(n1, n2, n3)

Finally if already have a neuronlist it is sometimes helpful to set individual columns rather than the whole data.frame. You can do that like this:

new_neurons[,'Glomerulus']='DL1' 

to set all neurons to have the same value, or this:

new_neurons[,'Glomerulus']=c('DL1', 'DL1', 'DL1' , 'DA1', 'DA1')

if there are 5 neurons and you want to give them those values for the Glomerulus column. Note that this is designed to work just as if you were with the neuronlists's data.frame object directly, so that you can use the notation e.g. to add arbitrary columns or set/fetch only part of the data.frame, e.g. .

new_neurons[1:3,'Glomerulus']='DL1'

new_neurons[4:5,'Glomerulus']='DA1'

Best wishes,

Greg.