Translating Tracing Data in CATMAID

[Marcel Sayre]

Hello everyone,

I have a stack of image data on CATMAID which has been incorrectly positioned in 3D coordinate space. Unfortunately, I realized this only after we began tracing and we now have ~70 neurons + synapses and annotations which all need to be translated to the correct coordinate values. In short, I need to do the following things:

1.) Export all tracing data from the current CATMAID instance

2.) Translate all tracing data (i.e. Skeletons/Connectors/Annotations/Tags) in the following way:

x (new) = x (old) + 103935.96

y (new) = y (old) + 77665.5567

z (new) = z (old) + 166425.408

3.) Import adjusted tracing data back into CATMAID instance

As of now, I am able to read neurons one at a time from our instance using read.neuron.catmaid (I either get a curl related error or "Error in catmaid_fetch...Method Not Allowed (HTTP 405)" if I try loading more than one) and translate these neurons to their new values using the following (where "x", "y", and "z" are the above functions):

```{r}

#for a single neuron n407

shift.407 = n407

shift.407$d$X = x(shift.407$d$X)

shift.407$d$Y = y(shift.407$d$Y)

shift.407$d$Z = z(shift.407$d$Z)

```

I can then write these as new .swc files and import them back into our instance using the Import/Export widget. However, after doing this, the connector data seems to be lost and no connectors can be found when searching in the Connector Table. I'm sure there is a more efficient way to do this (maybe using nlapply?) and greatly appreciate any advice! I'm happy to include more info about the error I mentioned above or anything else which might help. 

Thanks so much,

Marcel

[Gregory Jefferis]

You can do the nat stuff quite efficiently 

nl=read.neurons.catmaid(skids)

nl.shift=nl+c(dx,dy,dz)

# write a zip file containing all SWCs

write.neurons(nl.shift, “nl.shift.zip”)

Note that you can replace (skids) with (“myannotation”) if you have annotated all your neurons. 

Now the thing is that we do not currently have an export/import pathway for synapses as you note. There isn’t a default file format that works for synapses. 

It may be possible to use the catnat package to do an upload straight from R

http://jefferislab.github.io/catnat/reference/catmaid_upload_neurons.html

but this is a sort of unofficial package and I’m not sure exactly how replacing the preexisting skeletons is handled. 

I think that it is probably worth writing to the catmaid google group to see if they/Tom Kazimiers have a suggestion for how to do an in database update.

All the best,

Greg.

Sent from my iPhone

On 20 Apr 2020, at 13:48, Marcel Sayre <marcel...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "nat-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/c8eb2f66-510a-4eb6-a43a-d8c59dff249e%40googlegroups.com.

[Marcel Sayre]

Hello Greg,

Thanks so much for your quick response! We are now able to correctly shift the neurons and I will post to the CATMAID group re connectors. I'm wondering if you could help us troubleshoot the error message I mentioned above. We are able to connect to our instance just fine using token based auth and we are able to read single neurons when using read.neuron.catmaid, however we are unable to read multiple neurons at once. Here's an example of the error we receive:

```{r}

n.val = read.neurons.catmaid("OLD", pid = 2, conn=conn)

```

Error in catmaid_fetch(path = path, body = body, ..., include_headers = F) : Method Not Allowed (HTTP 405).

7.	stop(http_condition(x, "error", task = task, call = call))

6.	stop_for_status(req)

5.	catmaid_fetch(path = path, body = body, ..., include_headers = F)

4.	query_by_neuron_or_annotation(paste0(pid, "/annotations/query-targets"), body = list(annotated_with = query), maxresults = maxresults, type = return_type, raw = raw, conn = conn, ... = ...)

3.	catmaid_query_by_annotation(substr(x, 12, nchar(x)), type = "neuron", conn = conn, ...)

2.	catmaid_skids(skids, conn = conn, pid = pid)

1.	read.neurons.catmaid("OLD", pid = 2, conn = conn)

Our conn contains http auth username, auth pass and api token using the following:

conn = catmaid_login(server="servername",

  authname="authname",authpassword="authpass",

  token="token")

Do we need to include additional auth verification somewhere? 

Thanks!
Cheers,
Marcel

On Monday, April 20, 2020 at 3:56:02 PM UTC+2, Gregory Jefferis wrote:

You can do the nat stuff quite efficiently 

nl=read.neurons.catmaid(skids)

nl.shift=nl+c(dx,dy,dz)

# write a zip file containing all SWCs

write.neurons(nl.shift, “nl.shift.zip”)

Note that you can replace (skids) with (“myannotation”) if you have annotated all your neurons. 

Now the thing is that we do not currently have an export/import pathway for synapses as you note. There isn’t a default file format that works for synapses. 

It may be possible to use the catnat package to do an upload straight from R

http://jefferislab.github.io/catnat/reference/catmaid_upload_neurons.html

but this is a sort of unofficial package and I’m not sure exactly how replacing the preexisting skeletons is handled. 

I think that it is probably worth writing to the catmaid google group to see if they/Tom Kazimiers have a suggestion for how to do an in database update.

All the best,

Greg.

Sent from my iPhone

On 20 Apr 2020, at 13:48, Marcel Sayre <marce...@gmail.com> wrote:



Hello everyone,

I have a stack of image data on CATMAID which has been incorrectly positioned in 3D coordinate space. Unfortunately, I realized this only after we began tracing and we now have ~70 neurons + synapses and annotations which all need to be translated to the correct coordinate values. In short, I need to do the following things:

1.) Export all tracing data from the current CATMAID instance

2.) Translate all tracing data (i.e. Skeletons/Connectors/Annotations/Tags) in the following way:

x (new) = x (old) + 103935.96

y (new) = y (old) + 77665.5567

z (new) = z (old) + 166425.408

3.) Import adjusted tracing data back into CATMAID instance

As of now, I am able to read neurons one at a time from our instance using read.neuron.catmaid (I either get a curl related error or "Error in catmaid_fetch...Method Not Allowed (HTTP 405)" if I try loading more than one) and translate these neurons to their new values using the following (where "x", "y", and "z" are the above functions):

```{r}

#for a single neuron n407

shift.407 = n407

shift.407$d$X = x(shift.407$d$X)

shift.407$d$Y = y(shift.407$d$Y)

shift.407$d$Z = z(shift.407$d$Z)

```

I can then write these as new .swc files and import them back into our instance using the Import/Export widget. However, after doing this, the connector data seems to be lost and no connectors can be found when searching in the Connector Table. I'm sure there is a more efficient way to do this (maybe using nlapply?) and greatly appreciate any advice! I'm happy to include more info about the error I mentioned above or anything else which might help. 

Thanks so much,

Marcel

--
You received this message because you are subscribed to the Google Groups "nat-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nat-...@googlegroups.com.

[Gregory Jefferis]

It looks a bit like a permissions error but there is a possibility that there’s a bug with passing on the non default project id (pid=2). 

Trying breaking down that query by doing 

catmaid_skids(“OLD”, conn=conn, pid=2)

If you still get the same error then try 

debug(catmaid_fetch)

And tell me exactly what the `path` variable that defines the query is as you step through the catmsid_fetch function. 

Finally can you check your version of the catmaid package is up to date with the version on github. You need to check the sha1 — you can use the natverse package update functions for this. 

http://natverse.org/install/#update

All the best,

Greg. 

Sent from my iPhone

On 21 Apr 2020, at 09:26, Marcel Sayre <marcel...@gmail.com> wrote:



To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/d9bf224a-f5c4-4adb-bf03-be47027d7272%40googlegroups.com.

[Marcel Sayre]

Hey Greg,

I posted the original question on the CATMAID Google Group and Albert and Tom provided a solution for translating all tracing data by updating the psql tables directly in case anyone is interested: https://groups.google.com/forum/#!topic/catmaid/fJ2zkN1yiOU

I apologize for the belated response on this- I just got around to trying your advice and, after breaking down the query, I'm now receiving a new (but maybe related?) error:

catmaid_skids("OLD", pid = 2, conn = server_auth)

Error in curl::curl_fetch_memory(url, handle = handle) : Could not resolve host: catmaid

12. curl::curl_fetch_memory(url, handle = handle)

11. request_fetch.write_memory(req$output, req$url, handle)

10. request_fetch(req$output, req$url, handle)

9. request_perform(req, hu$handle$handle)

8. GET(url = file.path(conn$server, path, fsep = "/"), ...)

7. force(expr)

6. with_config(conn$config, { if (is.null(body)) { GET(url = file.path(conn$server, path, fsep = "/"), ...) } ...

5. catmaid_fetch(paste0("/", pid, "/annotations/"), conn = conn, parse.json = TRUE, ...)

4. catmaid_get_annotationlist(conn = conn, pid = pid)

3. catmaid_aids(query, conn = conn, pid = pid, rval = "data.frame")

2. catmaid_query_by_annotation(substr(x, 12, nchar(x)), type = "neuron", conn = conn, ...)

1. catmaid_skids("annotation:OLD", pid = 2, conn = server_auth)

If I run debug(catmaid_fetch) on this, I'm returned with the screen shot I've attached here. It doesn't look to me like path is defined anywhere in this function- is this what you're looking for? If not - I apologize, I'm still relatively new to R so let me know if I misunderstood!  

Thanks!

Cheers,
Marcel

On Tuesday, April 21, 2020 at 8:46:47 PM UTC+10, Gregory Jefferis wrote:

It looks a bit like a permissions error but there is a possibility that there’s a bug with passing on the non default project id (pid=2). 

Trying breaking down that query by doing 

catmaid_skids(“OLD”, conn=conn, pid=2)

If you still get the same error then try 

debug(catmaid_fetch)

And tell me exactly what the `path` variable that defines the query is as you step through the catmsid_fetch function. 

Finally can you check your version of the catmaid package is up to date with the version on github. You need to check the sha1 — you can use the natverse package update functions for this. 

http://natverse.org/install/#update

All the best,

Greg. 

Sent from my iPhone

To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/d9bf224a-f5c4-4adb-bf03-be47027d7272%40googlegroups.com.

[Marcel Sayre]

I forgot to mention- I updated natverse to the most recent catmaid packages so all should be good there! 

[Greg Jefferis]

Dear Marcel,

As noted in separate correspondence, the problem was that you had provided an http URL for your server but it should have been an https URL. The http URL was being redirected by the server but some POST queries failed after redirection. I am not sure why that is the case, but I will try to give some kind of warning on server login for this situation in the future.

All the best,

Greg.

To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/d9bf224a-f5c4-4adb-bf03-be47027d7272%40googlegroups.com.

[Greg Jefferis]

Dear Marcel,

I eventually got to the bottom of this. It seems that there is a bug in how libcurl (the most common way to interact programmatically with web resources) handles certain passwords. You can see all the details here if you wish:

https://github.com/jeroen/curl/issues/224

Bottom line, although I could try to work around this bug until there is a fix in libcurl (which will take a while to propagate), I would suggest that you avoid the use of # and @ characters in the basic authorisation passwords protecting your catmaid website. If you can change your password to remove the #, then everything will work fine.

You definitely win a prize for the most convoluted problem ever!

All the best,

Greg.

To unsubscribe from this group and stop receiving emails from it, send an email to nat-user+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/27d25b3d-51da-4147-bc34-fa736094d633%40googlegroups.com.
<debug-catmaid-fetch.png>

[Marcel Sayre]

Wo! I'll gladly accept my most convoluted problem ever prize. We changed the password as you suggested and that solved the issue! Thanks so much for digging into this, I really appreciate it!! 

Cheers,
Marcel

 

On Thursday, May 14, 2020 at 11:08:55 PM UTC+10, Greg Jefferis wrote:

Dear Marcel,

I eventually got to the bottom of this. It seems that there is a bug in how libcurl (the most common way to interact programmatically with web resources) handles certain passwords. You can see all the details here if you wish:

https://github.com/jeroen/curl/issues/224

Bottom line, although I could try to work around this bug until there is a fix in libcurl (which will take a while to propagate), I would suggest that you avoid the use of # and @ characters in the basic authorisation passwords protecting your catmaid website. If you can change your password to remove the #, then everything will work fine.

You definitely win a prize for the most convoluted problem ever!

All the best,

Greg.

To view this discussion on the web, visit https://groups.google.com/d/msgid/nat-user/27d25b3d-51da-4147-bc34-fa736094d633%40googlegroups.com.
<debug-catmaid-fetch.png>