neuPrint Janelia hemibrain skeleton -> NBLAST

[Abby]

Hi,

I would like to use NBLAST to search for GAL4 lines using a skeleton of a neuron that I downloaded from neuprint.janelia.org but I'm not sure how to prepare my input for NBLAST. I thought the hemibrain EM data had been registered to the JRC2018 template brain and that I might be able to use the Saalfeld Lab registrations to do something like what I pasted below, but when I plot my untransformed neuron and the JRC2018F template brain they are not aligned so I must be misunderstanding the necessary steps! I would appreciate any advice you might have on what additional bridging registrations or formatting is required before running NBLAST on hemibrain skeletons. Thank you so much!!

n = read.neuron("file.swc")

n_trans = xform_brain(my_neuron, sample=JRC2018F, reference=FCWB)

n.dps = dotprops(my_neuron_transf) 

## then run nblast on n.dps

[John Bogovic]

Hi,

I'm not a core nat contributor but I made the JRC2018 template, and the Saalfeld Lab registrations you referred to.

Just one misunderstanding - the "raw" hemibrain dataset is not aligned to any standard template.  It's in its own "space."

I'm about to release a transformation that will take neurons from the hemibrain space to the JRC2018F,

and once that's ready you'll be able to do what you'd like.  It will be soon.

John

P.S.  It's been slow to release this transform, because some biologists here at Janelia were unsatisfied with a

preliminary registration we had, and we've been spending the last week tuning and evaluating it. We don't want

to release a sub-standard "product".  Thanks for your patience.

[Abby]

Hi John,

Thank you very much for your fast response and helpful explanation! I look forward to the new release whenever it is ready :) 

-Abby

[Greg Jefferis]

Dear John, Thanks for responding to this!

Dear Abby, As John explains you need a transformation to take you from the raw hemibrain dataset to JRC2018F and he is the one who generated that. We have been using a pre-release version of the transform for some time now for the purposes that you describe. The natverse has infrastructure in place to make use of the final transform when he is able to make publicly available.

The likelihood is that you will use the package

https://github.com/natverse/nat.jrcbrains

Once you have this installed and the relevant registrations installed 

library(nat.jrcbrains)
download_saalfeldlab_registrations()

This already lets you use John's registrations for light level data. You will soon be able to do something like:

n_trans = xform_brain(my_neuron, sample=JRC2018FIBF, reference=FCWB)

n.dps = dotprops(my_neuron_transf) 

But once again John is the one providing the key missing ingredient here!

All the best,

Greg.

--
You received this message because you are subscribed to the Google Groups "nat-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/c20c8be4-3b1a-4b16-a100-96df80f0f5d6%40googlegroups.com.

[James Jeanne]

Hi all,

I am also interested in performing NBLAST comparisons between hemibrain neurons and neurons registered to other brain templates. I see that John Bogovic and the Saalfeld Lab have posted a JRC2018F to JRCFIB2018F registration to their website. Is this registration currently accessible and/or appropriate to use within the natverse environment? In principle, I don't think I necessarily need the JRCFIB2018F template as long as the registration exists, but I don't see how to apply the registration in NAT without specifying  the template (i.e. the call to "xform_brain(my_neuron, sample= JRCFIB2018F, reference=JRC2018F)" fails). Am I missing something about how to use the existing JRC2018F to JRCFIB2018F registration? Or are we still waiting on missing ingredients?

Thanks in advance for the help, and thanks for making such useful tools.

Jamie

Greg.

To unsubscribe from this group and stop receiving emails from it, send an email to nat-...@googlegroups.com.

[Greg Jefferis]

Dear Jamie,

As you've noticed John has now posted the registration, which is great. In theory you should be able to do something like:

remotes::install_github("natverse/nat.jrcbrains")

# once only
nat.jrcbrains::download_saalfeldlab_registrations()

library(nat.jrcbrains)

# check natverse knows about the registrations

plot(bridging_graph(), vertex.size=15, edge.arrow.size=.5)

# notice use of quotes for the sample brain
# however it seems that JRCFIB2018Fraw is not yet available in the natverse ...
my_neuront=xform_brain(my_neuron, reference=JRC2018F, sample="JRCFIB2018Fraw")
# so manually deal with the raw to µm mapping
# *8 to go from raw pixels to nm
# /1000 to go from nm to µm 
my_neuront=xform_brain(my_neuron*8/1000, reference=JRC2018F, sample="JRCFIB2018F")

However all is not quite right yet even after dealing with the incomplete natverse integration  – I am seeing a scaling issue of 2x for the new registration and I will need to look into this with John.

All the best,

Greg.

To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/5527d88c-78c9-480b-afa5-e2b4fa131bd6%40googlegroups.com.

[James Jeanne]

Thanks, Greg!

When I run

xform_brain(my_neuron*8/1000, reference=JRC2018F, sample="JRCFIB2018F")

I get a popup message telling me "To use 'java' command-line tool you need to install a JDK"

In addition the error message in the RStudio console is:

Error in saalfeld_xform(points, reg, inverse = !swapped, ...) : 

  Error running saalfeld xform!

Do I need a specific java development kit installed? Can you point me to an appropriate option?

Thanks,

Jamie

To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/5527d88c-78c9-480b-afa5-e2b4fa131bd6%40googlegroups.com.

[Gregory Jefferis]

You should be able to use any JDK. I have the oracle one installed. Best, Greg.

Sent from my iPhone

To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/7c87da46-20c8-491d-9bb9-7367860e7a08%40googlegroups.com.

[James Jeanne]

Terrific. Thanks. Can you explain further "what is not quite right"? With the JDK installed and adding in the factor of 8/1000, the registration transformations look fine (at least judging by eye). Are there remaining inaccuracies that should be fixed before I can rely on this registration?

Jamie

To unsubscribe from this group and stop receiving emails from it, send an email to nat-...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/7c87da46-20c8-491d-9bb9-7367860e7a08%40googlegroups.com.

[Greg Jefferis]

Dear Jamie,

Actually I've gone through everything again and you're right, all looks fine. I think I had some left over older transforms interfering. Full worked example below.

All the best,

Greg.

library(nat.jrcbrains)

dl1.hemibrain=neuprint_read_neurons('DL1 adPN')

dl1.hemibrain.fafb=xform_brain(dl1.hemibrain/125, reference= "FAFB14", sample="JRCFIB2018F")

dl1.hemibrain.jrc2018f=xform_brain(dl1.hemibrain*8/1000, reference= "JRC2018F", sample="JRCFIB2018F")

fafbconn=catmaid::catmaid_login(server='https://fafb.catmaid.virtualflybrain.org/')

dl1.fafb=catmaid::read.neurons.catmaid('name:DL1', conn=fafbconn)

dl1.fafb.jrc2018f=xform_brain(dl1.fafb, reference = "JRC2018F", sample="FAFB14")

plot3d(dl1.fafb.jrc2018f, col='red')

plot3d(dl1.hemibrain.jrc2018f, col='cyan')

plot3d(JRC2018F)

To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/84d38ef2-61b6-42dd-890d-24972a5762b4%40googlegroups.com.

[Avirut Mehta]

Hi,

Extremely related question so I'll continue in this thread since I'm trying to do pretty much exactly the same thing! I'm able to get the neuron I want from neuprint following the steps above and transform it to FCWB for NBLAST, but can't seem to get nat.nblast to work. Here's the code I'm running:

conn = neuprint_login(server=Sys.getenv('npserver'), token=Sys.getenv('nptoken'))

retneurons = neuprint_search('MBON01', fixed=TRUE)

ofinterest = retneurons[1,]

bodyid = ofinterest[1, 'bodyid']

neuron.hemibrain.raw = neuprint_read_neurons(bodyid)[[1]]

neuron.hemibrain.fcwb=xform_brain(neuron.hemibrain.raw/125, reference=FCWB, sample="JRCFIB2018F")

neuron.hemibrain.dps = dotprops(neuron.hemibrain.fcwb)

nblast(neuron.hemibrain.dps, options("nat.default.neuronlist"))

And here's the output I get:

Error in query[, c("X", "Y", "Z")] : incorrect number of dimensions

I'm not sure if it's possible to get a stack trace from RStudio, but from what I can tell this is probably coming from line 426 in neuriteblast.r:

b=query[,c("X","Y","Z")]

I've also tried with:

fafbconn=catmaid::catmaid_login(server='https://fafb.catmaid.virtualflybrain.org/')

dl1.fafb=catmaid::read.neurons.catmaid('name:DL1', conn=fafbconn)

dl1.fcwb=xform_brain(dl1.fafb, reference = FCWB, sample="FAFB14")

nblast(dl1.fafb, options("nat.default.neuronlist"))

...and get the same error. Please let me know if you have any ideas why this might be happening. Thanks!

- Avirut

To unsubscribe from this group and stop receiving emails from it, send an email to nat-...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/84d38ef2-61b6-42dd-890d-24972a5762b4%40googlegroups.com.

[Greg Jefferis]

On 7 Mar 2020, at 19:54, James Jeanne <jje...@gmail.com> wrote:

Terrific. Thanks. Can you explain further "what is not quite right"? With the JDK installed and adding in the factor of 8/1000, the registration transformations look fine (at least judging by eye). Are there remaining inaccuracies that should be fixed before I can rely on this registration?

Hi Jamie, Sorry, I missed replying to this, but although the process may become slightly more streamlined, yes this is now the best registration available. All the best, Greg.

[Greg Jefferis]

Dear Avirut,

Thanks for your interest in the natverse tools. Although I have made a few suggestions to improve your code below, AFAICS the only problem is with the last line in which you are trying to nblast your transformed neuron

 nblast(dl1.fafb, options("nat.default.neuronlist"))

The statement 

options("nat.default.neuronlist")

is supposed to be run once per R session, to set the default neuronlist for searching or plotting. See e.g. http://natverse.org/nat/reference/nat-package.html#package-options

In this case it needs to be set to the collection of 16,000 flycircuit neurons. You can find out more about NBLASTing them here http://flybrain.mrc-lmb.cam.ac.uk/si/nblast/www/nblast_desktop/

but I have put the code that you need to run in the full script below.

All the best,

Greg.

library(neuprintr)

# server=Sys.getenv('npserver'), token=Sys.getenv('nptoken')

# As per the neuprint readme, https://github.com/natverse/neuprintr

# we recommend setting the following two environment variables

# This is only necessary if they should be something other than these defaults

# neuprint_server = "https://neuprint.janelia.org"

# neuprint_dataset = "hemibrain:v1.0.1"

conn = neuprint_login()

retneurons = neuprint_search('MBON01', fixed=TRUE)

ofinterest = retneurons[1,]

bodyid = ofinterest[1, 'bodyid']

# bodyid = retneurons[1, 'bodyid'] also works

# you don't necessarily need to subset out the first neuron with [[1]]

# if you do, then there are various places in the plot3d statements

# below where I have set WithNodes = F

neuron.hemibrain.raw = neuprint_read_neurons(bodyid)[[1]]

library(nat.jrcbrains)

neuron.hemibrain.fcwb=xform_brain(neuron.hemibrain.raw/125, reference=FCWB, sample="JRCFIB2018F")

# check that the neuron looks ok

plot3d(neuron.hemibrain.fcwb, WithNodes=F, col='green')

plot3d(neuron.hemibrain.dps, col='blue')

plot3d(FCWB)

nview3d()

neuron.hemibrain.fcwb.pruned <- prune_twigs(neuron.hemibrain.fcwb, twig_length = 5)

neuron.hemibrain.dps = dotprops(neuron.hemibrain.fcwb.pruned, k=5, resample=1)

# load FlyCircuit neurons

library(flycircuit)

dps<-read.neuronlistfh("http://flybrain.mrc-lmb.cam.ac.uk/si/nblast/flycircuit/dpscanon.rds",

                       localdir=getOption('flycircuit.datadir'))

options('nat.default.neuronlist'='dps')

# As a one time step, you need to download all the flycircuit neurons.

# you can do this like this

remotesync(dps)

library(nat.nblast)

# this implicitly will use the 'dps' object as the query neurons

sc=nblast(neuron.hemibrain.dps, .progress='text')

hist(sc)

top10=sort(sc, decreasing = T)[1:10]

# look at the results

nclear3d()

# this implicitly will use the 'dps' object as the source of neurons to plot

plot3d(names(top10)[1:3], soma=T)

plot3d(neuron.hemibrain.fcwb.pruned, col='black', WithNodes = F, lwd=2, soma=T)

# Look at neurons that actually have matches

mbony5s=neuprint_read_neurons(neuprint_search('MBON.*y5.*', field = 'name'))

mbony5s.fcwb=xform_brain(mbony5s/125, reference=FCWB, sample="JRCFIB2018F")

nclear3d()

plot3d(names(top10)[1:3], soma=T)

nlscan(mbony5s.fcwb, col='black', soma=T)

# You might find it useful to plot the hemibrain surface to see 

# how neurons are cut off

# Needs another package, hemibrainr

# see https://github.com/flyconnectome/hemibrainr

hemibrain.surf.fcwb <- xform_brain(hemibrainr::hemibrain.surf/125, reference=FCWB, sample="JRCFIB2018F")

plot3d(hemibrain.surf.fcwb, alpha=.2)

[Avirut Mehta]

Thank you, this works perfectly and things in general about how to make neuprintr/nblastr make a lot more sense to me now!

[Isabel D'Alessandro]

Somewhat related question- is it possible to perform an NBLAST search against the entire collection of hemibrain neurons (i.e. use the hemibrain dataset as the target in an NBLAST search)? Is there a way to download the entire collection of hemibrain neurons using neuprintr for this purpose? 

Thank you! 

performing NBLAST comparisons between hemibrain neurons and neurons registered to other brain templates.

On Wednesday, February 12, 2020 at 7:41:23 PM UTC-5, Abby wrote:

[Greg Jefferis]

Dear Isabel,

There is of course a way to do this – and we have been doing so for internal use for a while now. In theory, you can just download the SWC files en masse by following the link from the hemibrain website to the DVID blog, convert them to the right template space (as described earlier in this thread) and then to dotprops format.

There are some wrinkles though as you need to decide which template space to use for your calculations (it needs to be calibrated in microns). We have also found that some preprocessing is necessary to remove numerous small twigs, especially when comparing light level neurons with EM neurons. Otherwise the twigs dominate the score over the neuronal backbone. We normally use twig thresholds of 2-5 µm.

http://natverse.org/nat/reference/prune_twigs.html

We have not distributed NBLAST ready versions of all the skeletons because FlyEM should soon be releasing their own. However if you want to know more in the interim, feel free to contact me off list.

All the best,

Greg.

[Isabel D'Alessandro]

Hi Greg, 

Thanks for your help! I've been able to complete almost all of the steps you mentioned, except removing the small twigs. When I try to run the prune_twigs function on a neuron object, I get the following error message: 'I can't prune neurons with more than 1 root!'. Here's an example of the type of code I'm trying to run (reading in a single hemibrain .swc file) 

test=read.neuron('D:/hemibrain_skeletons/skeletons/200326126.swc',class='neuron')
test.fcwb=xform_brain(test*8/1000, reference="FCWB", sample="JRCFIB2018F")
test.fcwb.pruned=prune_twigs(test.fcwb, twig_length = 5)

Do you know what might be the problem in this case? 

Thanks again! 

Isabel 

[Gregory Jefferis]

Dear Isabel,

You need to “heal” the skeletons.  You can use 

http://natverse.org/nat/reference/stitch_neurons_mst.html

To do this. 

neuprint_read_skeletons/neuprint_read_neurons will do this for you based on the heal argument

http://natverse.org/neuprintr/reference/neuprint_read_neurons.html

I have to say that several people have been asking me highly related questions recently, so we should think about a public release of a good quality NBLAST compatible dataset.

All the best,

Greg.

Sent from my iPhone

On 30 Jun 2020, at 21:08, Isabel D'Alessandro <idisa...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "nat-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/bfa8f737-5371-4ed4-9068-af73b7eade83o%40googlegroups.com.

[Altyn Rymbek]

Dear Dr. Jefferis,

I have a related question. I am also trying to NBLAST the neuron against all neuPrint skeletons.

Is there any way to download all neuPrint swc files or as neuronlist in order to NBLAST my skeletonized neuron against them? May be there is a way to first get all of the bodyids and then input them into the read.neuron(), similar to the following two functions from the flycircuit package?

fc.ids = fc_get_ids()

fcns <- fc_read_neurons(fc.ids)

So far, I have downloaded hemibrain_v1.0.1_neo4j_inputs.zip   file that contains only csv files and no swc skeletons.

Any help would be very much appreciated!

Thank you very much.

Kind regards,

Altyn

вторник, 30 июня 2020 г., 15:37:40 UTC-7 пользователь Gregory Jefferis написал:

To unsubscribe from this group and stop receiving emails from it, send an email to nat-...@googlegroups.com.

[Greg Jefferis]

Dear Altyn,

You have to 

1. Choose what neurons to include in your search

2. Use http://natverse.org/neuprintr/reference/neuprint_read_neurons.html to read them in.

For example

# find all neurons with at least one character in their name
named.neurons=neuprint_ids("/name:.+")
# ensure that every single piece is stitched, no need to fetch synapses
alln=neuprint_read_neurons(named.neurons, heal.threshold=1e9, connectors=FALSE)

Then you need to proceed with making your dotprops etc. If you wish you can directly download the 1.0 version of the skeletons (from Jan) here:

https://dvid.io/blog/

https://storage.cloud.google.com/hemibrain-release/skeletons.tar.gz

Best,

Greg.

To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/d6885b7e-bb2f-4d05-bbc5-0e69170a4740o%40googlegroups.com.