New CATMAID release: 2019.06.20
15 views
Skip to first unread message
Tom Kazimiers
Jun 21, 2019, 9:39:18 AM6/21/19
to CATMAID
Hi all,
We just released a new CATMAID version, 2019.06.20:
https://github.com/catmaid/CATMAID/releases/tag/2019.06.20
You will find a list of changes as we well as some update notes at the
end of this email and on the website linked above. As always, please let
us know if you run into any problems, both as administrator and user.
If you update a CATMAID instance, please use the most recent commit of
the master branch in our GitHub repository, which includes all future
bug fixes for this release:
https://github.com/catmaid/CATMAID
If you are administering a CATMAID server, the "Notes" section provides
information on how to migrate to the new version. The file UPDATE.md
contains the migration steps for all versions. Please also note that
this is the last version of CATMAID that supports Python 3.5 and
Postgres 10. With the next release, Python 3.6 or 3.7, Postgres 11 and
PostGIS 2.5 will be required.
Cheers,
Tom
2019.06.20
==========
Contributors: Chris Barnes, Albert Cardona, Andrew Champion,
Stephan Gerhard, Pat Gunn, Tom Kazimiers, William Patton
Notes
-----
- A virtualenv update is required.
- All \*.pyc files in the folders `django/applications/catmaid/migrations/`,
`django/applications/performancetests/migrations/` and
`django/applications/pgcompat/migrations/` need to be deleted.
- Python 3.7 is now supported.
- Heads-up: The next CATMAID version will require Postgres 11, PostGIS 2.5 and
Python 3.6 or 3.7.
- Should migration 0057 fail due a permission error, the Postgres extension
"pg_trgm" has to be installed manually into the CATMAID database using a
Postgres superuser:
`sudo -u postgres psql -d <catmaid-db> -c 'CREATE EXTENSION pg_trgm;'`
- CATMAID's version information changes back to a plain `git describe` format.
This results overall in a simpler setup and makes live easier for some
third-party front-ends, because the commit counter is included again. The
version display is now the same `git describe` format for both regular setups
and Docker containers.
- Tile loading is clamped to (0,0) again, i.e. there are no negative tile
coordinates anymore by default. If you need them, set the respective stack's
`metadata` field to `{"clamp": false}`.
- To write to CATMAID through its API using an API token, users need to have
now dedicated "API write" permission, called "Can annotate project using
API token" in the admin UI. To restore the previous behavior (regular annotate
permission allows API write access) the settings.py variable
`REQUIRE_EXTRA_TOKEN_PERMISSIONS` can be set to `False`. This is done as a
safety measure to prevent accidental changes through automation.
- If R based NBLAST is used, make sure to execute to update all dependencies:
`manage.py catmaid_setup_nblast_environment`.
- The main documentation on catmaid.org has now a place for widget specific
documentation as well. Only a few widgets have been updated yet, but more will
follow.
Features and enhancements
-------------------------
Tracing tool:
- Add a new Tracing Tool icon button to compute the distance between two nodes
on a skeleton. Respects virtual nodes.
- The globally nearest node can now be selected and brought into view using
Alt + G, as opposed to selecting the nearest node in the current section
using the G key without modifier.
- Using the H shortcut now without an active skeleton, the most recently edited
node of the current user (Alt: anyone) in the last active skeleton will be
selected. Using Shift will look in all skeletons.
- Using Shift + Alt + Click with a connector selected, will will now create a
presynaptic node. This is consistent with the already existing Shift + Alt +
Click behavior with a treenode selected, which creates a presynaptic
connector.
Graph widget:
- GraphML files can now be imported, positions and colors are respected. This is
useful if layouting is done in e.g. Gephi and coloring should be done in
CATMAID. The help page explains a possible workflow for this.
- The button "Group equally colored" in the "Main" tab will group all skeletons
with the same color into a single group. The user is asked for group names for
each color.
3D viewer:
- A new synapse coloring mode has been added: polyadicity. The number of partner
nodes for each connector is color coded (for synaptic connectors, this is the
number of postsynapses). The colors and ranges can be configured through the
"Polyadicity colors" button in the "Shading parameters" tab. This is basically
a configurable version of an absolute "N with partner" coloring.
- Branches with leaf nodes tagged "not a branch" can now be collapsed using the
'Collapse "not a branch"' checkbox in the Skeleton Filters tab.
- The visibility of radius information can now be controlled using the "Show
radius" checkbox in the "Views settings" tab.
- History animations can now be exported in full length without requiring to
guess the number of frames for the export. The animation export dialog will
show an additional checkbox ("Complete history") if a history animation should
be exported. If complete history is enabled, CATMAID will export the complete
history of the exported skeletons.
- Animations can now be exported as stream directly to a file, which allows for
much larger exports (32GB maximum at the moment).
- Fractional rotations are now allowed in the animation export.
- The default time per rotation in the animation export is set to 15 seconds now,
slowing down the default by a factor of 3, which makes it easier to look at.
- Stack Z slices can now be animated. Configurable are the change frequency and
the change step in terms of sections. This is available for animation exports
as well.
- Stack Z slices can now be thresholded to replace a background color with
another color. If enabled and the sum of all channels is in a configurable
range [a,b] it will be replaced with another color.
- The rotation time for animations can now specified in seconds rather than
angular distance.
- The background color is now fully adjustable through the "backgorund" button
in the "View settings" tab.
- Basic support for Virtual Reality is now available on Windows platforms using
a Mixed Reality / SteamVR.3D setup and Firefox >= v65. To enable, check the
VR checkbox in the "View" tab and click the "Enter" button right next to it.
Skeleton history widget:
- A basic view of the change of a set of skeleton IDs over time based on all
nodes that are part of a given skeleton ID or that have been in the past.
- Skeleton history can also be used with past skeleton IDs to see into what
skeleton they changed (if any).
- All past and present treenodes with a passed in skeleton ID are tracked
through the complete history and their path of skeleton ID changes is
recorded along with the number of treenodes following a given skeleton path.
- The widget shows a graph from origin skeletons to the final skeleton IDs in
every available path, summing the treenode counts for each contributing path.
- Existing skeletons are colored in yellow, past skeletons are colored in cyan.
Selected skeletons are colored green.
- Ctrl+Click on skeleton will select it and go to the closest location in it.
Shift+Click allows selecting multiple skeletons. All selected skeletons are
available through the Skeleton Source interface.
Node and skeleton filters:
- Filter rules support now an "invert" option during creation, which allows to
create filters that include everything but whatever is matched by a particular
filter strategy. This can be useful e.g. during neuron review to only look at
segments that have been created by people other than oneself or connectivity
everywhere excluding a particular compartment.
Measurement table:
- Node filters are now supported. Like in other widgets, the respective panel
can be opened through the funnel icon in the widget title bar. Measurements
are displayed for all independent fragments of a skeleton after a set of node
filter rules has been applied.
- With the help of the "Sum fragments" toggle all fragments that result from a
node filter application can be aggregated into one row by summing individual
values.
Remote CATMAID instances
- Some tools in CATMAID gained support to communicate with other CATMAID
instances, e.g. the Landmark Widget (see below). To enable this functionality,
remote CATMAID instances can now be added to the user settings.
- The Settings Widget contains now a section labeled "Other CATMAID instances".
It allows to manage a list of other CATMAID servers based on their URL, an API
key and optional HTTP authentication. This information is stored under an
alias.
Landmarks:
- Clicking on the name of a landmark group in the Landmarks tab will now open
the group member edit dialog that was previously accessible by clicking any
landmark name. Clicking a landmark name will now show a new dialog, which
allows to specify of which landmark groups the clicked landmark is a member
of.
- The color of each active display transformation can be established separately
in the respective table row. Color changes are now also visible in the 3D
Viewer.
- It is now possible to load skeletons and landmarks from other CATMAID
instances. Point matches are done on the basis of matching landmark names. To
enable the UI for this, check the "Source other projects" checkbox in the
Display tab.
- With the "other projects" UI enabled and a remote CATMAID instance configured
in the Setting Widget (see above), it is now possible to select a remote
CATMAID instance from the "Source remote" dropdown menu. Alternatively, the
local instance can be selected if another project from the same instance
should be used.
- Next the source project in the selected instance needs to be specified. This
list is updated when a different remote instance is selected.
- Skeletons from a remote instance are collected through annotations. The
respective annotation has to be entered in the "Source skeleton annotation".
With the help of the "Preview" button it is possible to load the matching
skeletons from their remote CATMAID to inspect if the correct ones are
selected.
- As a last step for the remote data configuration, the source landmark group
has to be defined. This list is updated if the source project changes.
Landmarks from this group are mapped to the selected target group. The
matching is done by name, i.e. no landmarks can have the same names in a
group.
- Adding such a transformation adds it to the list at the bottom of the widget,
just like with regular transformations and they can be used in the same way.
The can be shown in 3D Viewers, superimposed on the Tracing Layer and used in
NBLAST queries from the Neuron Similarity Widget (see below).
- The new checkbox 'Multiple mappings' displays additional user interface
elements to add multiple source and target landmark group mappings for a
single display transformation. Point matches are created independently for
each pair and only merged for finding the final transformation.
- Using a transformation is now optional (but enabled by default). To disable
landmark transformations, uncheck the "Apply transformation" checkbox. This
allows e.g. to display skeletons from remote CATMAID instances without
modification.
Neuron Similarity:
- The computation of the default "mean" normalized similarity scores is now
considerably faster.
- The new "Top N" option allows to store only the Top N best matches in the result
set. For "mean" normalization, it will also only compute the mean score for
the top N forward hits (mean is the average between forward and backward
score). A value of zero disables this cutoff (default). This cutoff can be
used speed up computation for very large sets of neurons or large point
clouds.
- The new "Reverse" option allows to rank similar objects by there reverse
score. That means that the stored score for a particular object pair is target
versus query rather than query versus target. NBLAST uses the query neuron as
reference, and thereforoe the reverse option can make a difference as long as
no "mean" scoring is used.
- It is now possible to use display transformations defined in the Landmark
Widget that reference another CATMAID server as their data source. They can be
used both as source and target for NBLAST comparisons. This makes it
essentially possible to compare skeletons from another CATMAID project,
possibly on another CATMAID server, to local skeletons using NBLAST.
- Similarity matrices can now include transformed skeletons in their set of
similar objects. To do so, create a Display Transformation in the Landmark
Widget that represents the wanted transformation and the select it from the
"Sim. transformed skeletons" drop down in the Configurations tab of the Neuron
Similarity Widget when creating a similarity matrix.
- Similarity matrices can now be created with more control over which skeletons
are looked at as similar. The drop down menu "Similarity mode" allows to
select different groupings of the input skeletons (and transformed skeletons).
It is for instance possible to group a neuron and its contralateral partner
that has been transformed to the side of the first neuron as similar. A
confirmation dialog presents the computed grouping before actually creating a
similarity matrix.
- Similarity matrices can now be imported from CSV files, optionally including
the distance binning and dot binning as header row and column. As part of the
import, the distance binning can be scaled to adjust for differences in
datasets. This is available through the "Create from CSV file" button in the
Configurations tab.
- Similarity query results can now be used as skeleton source in other widgets,
if the target type of the query are skeletons.
System check widget:
- A subset of important database statistics are now displayed if a user has the
"can_administer" permission in a project. These statistics include e.g. cache
hit ratio, temporary files, requested checkpoints and replication lag. For
most of them a rule of thumb suggestion on how a value should behave is
provided as well.
Extension: CATMAID-autoproofreader:
- This extension is written by Will Patton and has to be installed separately,
because it requires additional setup steps for segmentation data handling.
Details can be found here: https://github.com/pattonw/CATMAID-autoproofreader.
- This tool suggests locations in a skeleton reconstructions where branches
might be missing or wrongly connected. Users can step through these
suggestions to fix potential problems. After reviewing a node it can be marked
as reviewed for future reference.
- In the CATMAID front-end the autoproofreader provides a widget that relies on
a compute server to handle the computations involved with automatically
proofreading a neuron reconstruction. Proofreading jobs are performed
asyncronously and might take a few minutes to complete.
Administration:
- A grid based node query cache can now be used to speed up tracing data
retrieval by precomputing intersection results. This can be useful for larger
field of views in big data set. It supports multiple levels of detail per grid
cell and can therefore provide a somewhat uniform sampling when limiting the
number of displayed nodes. The sorting of this LOD configuration can be
controlled so that e.g. only the top N largest skeletons will be shown per
cell with growing LOD. The documentation has more details. This cache can be
kept up updated using two extra worker processes that listen to database
changes, implemented as management commands `catmaid_spatial_update_worker`
and `catmaid_cache_update_worker`.
- The database can now emit notifications on tracing data changes using the
"catmaid.spatial-update" channel and when grid cache cells get marked dirty
on the "catmaid.dirty-cacche" channel. These can be subscribed to using
Postgres' LISTEN command. To enable emitting these events and let cache update
workers work, set `SPATIAL_UPDATE_NOTIFICATIONS = True` in `settings.py`. This
is disabled by default, because sending events for spatial cache updates can
add slightly more time to spatial queries, which will mainly be relevant for
importing and merging large skeleotons.
- Projects can now be deleted along with the data that reference them (e.g.
treenodes, ontologies, volumes). To do so select the projects to delete in the
admin project list and select "Delete selected" from action drop down.
- Users can now be set to an inactive state after a specified amount of time.
This time is configured for a user group and it applies to users if they are
members of such a group. A message can optionally be displayed as well as
users to contact that could potentially help. This is configurable as "Group
inactivity period" in the admin view. Users to contact for support, can be
either configured there or from individual user views.
CLI Exporter:
- If both required annotations and exclusion annotations are provided, the
importer will now only exclude a skeleton if it is not also annotated with a
sub-annotation of the required annotations set. This behavior can be disabled
and exclusion can be enforced when a skeleton is annotated with the exclusion
annotation or one of its sub-annotations. To do so, use the new
"--exclusion-is-final" switch.
- Volumes can now be exported by using the `--volumes` switch. By default all
volumes of the exported project will be included. This can be further
constrained by using `--volume-annotation <annotation-name>` arguments.
- The way she exported objects are specified through the command line interface
changed. Instead of writing e.g. `--notreenodes` to not import treenodes from
a source, `--treenodes false` has to be used now. This is the case for
treenodes, connectors, tags and annotations. The defaults (when the argument
is not provided) stay the same. To explicitly disable the export of a type
`false`, `no`, `f`, `n` and `0` can be provided as argument, e.g. `-treenodes
false` or `--users n`. For a positive parameter use `true`, `yes`, `t`, and
`1`.
CLI Importer:
- Precompute materializations (edges, connectors) explicitly only for imported
data, which improves performance in typical scenarios (import is small
compared to existing data). If the old behavior of recomputing everything in
the target project should be used in cases where a full data base is imported,
the --update-project-materializations switch can be used.
- The way she imported objects are specified through the command line interface
changed. Instead of writing e.g. `--notreenodes` to not import treenodes from
a source, `--treenodes false` has to be used now. This is the case for
treenodes, connectors, tags and annotations. The defaults (when the argument
is not provided) stay the same. To explicitly disable the import of a type
`false`, `no`, `f`, `n` and `0` can be provided as argument, e.g. `-treenodes
false` or `--users n`. For a positive parameter use `true`, `yes`, `t`, and
`1`.
- Imported usernames can now be mapped to existing usernames and mapped
selectively by using one or more parameters of the form
--username-mapping="import-username=target-username". The example would map
all references to the user import-username in the imported data to the
existing user target-username. The mapping is performed even with --map-users
off.
- It is now possible to import volumes from CATMAID export files and other
projects..
Miscellaneous:
- Tracing layer: a minimum skeleton length can now be specified in the layer
options, causing the Tracing Layer to show only nodes of skeletons of at least
this cable length (nm).
- Tracing layer: a minimum number of nodes can now be configured to only show
skeletons of at certain minimum size (just like the existing minimum cable
length constraint). This is configurable through the layer configuration,
accessible by clicking the blue/white box in the lower left corner of stack
viewers.
- Settings widget: visibility group conditions can now be inverted.
- Confirming a radius selection (Shift + Y) can now also be done using the Enter
key.
- Neuron name searches and annotations should be much faster on larger
instances.
- Basic support for touch screens (e.g. phones and tablets) is now implemented.
- CATMAID can now export files (e.g. CSVs, videos, etc.) of much larger size
than before, if "Save exported files in streaming mode" is enabled in the
Settings Widget. To make this work, the browser settings (chrome://settings)
"Ask where to save each file before downloading" has to be enabled.
- Context aware help: by clicking the new question mark icon in the upper right
corner, a context aware help dialog can be displayed on top of all other
CATMAID tools. It provides some general guidance for the intended workflow
with individual tools. This can be enable to be displayed by default as part
of the Settings Widget. The display of this can also be enabled in a link to a
view by appending `&help=true`. This will be added automatically if a help
dialog is open during URL creation.
- Connectivity widget: the new "List links' button will open a Connector List
with all links of the selected neurons.
- Selection table: the new checkbox "Link visibility" allows to change the
behavior of the visibility checkboxes. So far the pre/post/text/meta
select-all controls were only affecting the respective visibility options of
visible/selected skeletons (default). With unlinked visibility
pre/post/text/meta can also be controlled for all neurons, regardless of their
visibility. This allows e.g. to only show synapses, but no skeletons.
- General search widget: a more flexible table is now used for display, which
allows sorting, filtering and pagination.
- Neuron dendrogram: select currently active node by default when loading the
dendrogram for the active skeleton.
- The right end of the status bar contains now a button to toggle the visibility
of the top toolbars.
- Updating the copy of client side annotations will only request new data if
there are actually new annotations. This makes loading of e.g. the Neuron
Search widget faster.
- Boss databases can now be used as tile source type so that image data is
loaded from them. More details: https://docs.theboss.io/docs/image.
- Docker: HTTP basic authentication can be configured by using the environment
variables HTTP_AUTH_ENABLED, HTTP_AUTH_USER and HTTP_AUTH_PASS in the web
container of the `docker-compose.yml` file (an example is given).
- Performance: selecting the closest node in a skeleton should now be faster.
Bug fixes
---------
- A race condition has been fixed that could in rare cases lead to inconsistent
skeleton IDs in a skeleton that is merge into different skeletons by different
requests.
- Landmark transformations: fix Moving Least Square transformations for skeleton
fragments outside of source group bounding box.
- The Connectivity Graph Plot draws now individual bars in each sub-plot
side-by-side and uses the same neuron names as other widgets (from CATMAID's
neuron name service).
- Information based on historic information like the history animation in the 3D
Viewer and the Neuron History widget includes merge information now correctly.
Before, merge nodes were represented twice in a time slice that included the
merge.
- The default connector type is now properly persisted as a setting.
- Neuron search: no duplicate entries are shown anymore, which could result e.g.
when sub-annotations where allowed.
- Neuron similarity: when showing result skeletons in 3D, skeletons are now
appended to the Selection Table of a 3D Viewer, rather than the 3D Viewer
directly.
- Connectivity widget: the auto-connectivity CSV will now use formatted neuron
names as column and row headers.
- Connectivity widget: the 'Export CSV' button will now respect name and
annotation filters.
- Graph widget: ungrouping of one-element groups is now allowed.
- Graph widget: merging of grouped skeletons is now handled properly.
- 3D viewer: the depth test for connector partner spheres is now performed
correctly and spheres should be rendered in the correct Z order.
- 3D viewer: mouse controls now work correctly in fullscreen mode.
- 3D viewer: the suggested width and height of the animation export are now by
default a factor of two. This is required by the WebM encoder. Off values are
reduce by one.
- 3D viewer: spatial select works now also without the active node highlighted.
- 3D viewer: no error is shown any more when attempting to move while in radius
edit mode.
- 3D viewer: landmark transformed skeletons show now also radius spheres.
- Connector table: the section and tag columns are now part of the CSV export.
- Tracing tool: unneeded node updates are removed from initial tracing layer
loading.
- Selection table: don't try to load missing skeletons from JSON file.
- Settings widget: default values for tracing layer skeleton limits can now be
configured under Tracing Overlay > Tracing layer skeleton filters.
- The numpad delete key is now recognized as regular delete (if numlock is off).
- SWC neuron import: providing a neuron ID for an import works again and will
now also set an optionally provided neuron name. Additionally, it is now
required that a user importing skeleton data for an existing neuron ID has
the rights to edit the skeleton instance as well as all its treenodes.
- Volume widget: volumes/meshes with annotation can now be removed properly.
- Volume widget: editing a volume by double clicking its table entry works
again.
- Neuron dendrogram: correctly reload skeleton if it changes as a result of a
split or merge. If a dendrogram node is selected, the respective skeleton will
be reloaded after a split, even if its ID changed.
- The Notification Table can be opened again without errors.
- Exporter: connector links are now exported properly if the parameter
--original-placeholder-context is used.
- Docker: the docker-compose setup now uses Postgis 2.5 internally and therefore
allows upgrades from Postgres versions < 10 with Postgis 2.4.
- Docker: stale Postgres PID files will now be removed during a database upgrade
in a docker-compose setup. PID files without actually running database
processes prevented some updates before.
- Initial setup: create_configuration.py now also prints the media files
directory as part of the webserver example configuration. This is the folder
where e.g. some exported files are made available.
API changes
-----------
Additions
---------
- `GET /{project_id}/nodes/nearest`:
Replaces `POST /{project_id}/node/nearest`. The parameters are the same, but
the API allows now to look globally for the nearest node in the project, if no
skeleton ID or neuron ID is provided.
- `GET /{project_id}/nodes/most-recent`:
Replaces `POST /{project_id}/node/nearest`. A skeleton_id parameter can still
be provided, but now also a user_id parameter is available to further
constrain.
Modifications
-------------
- `POST|GET /{project_id}/node/list`:
Offers a new optional parameter "min_skeleton_length", which can be used to
constrain the returned neurons to only those of at least this cable length.
- `POST|GET /{project_id}/pointclouds/`:
Offers a new optional parameter "order_by", which accepts the strings 'id' and
'name' to define in what order the list of pointclouds should be retuned
(default: id).
- `POST /{project_id}/landmarks/{landmark_id}/`:
Offers a new optional parameter "group_ids", an array of integers, which
allows to set the landmark group memberships of a specific landmark. The new
boolean parameter "append_memberships" allows to only append new group IDs as
memberships, without removing any. Otherwise the whole set of memberships is
replaced.
- `POST /{project_id}/skeletons/import`:
The new parameter `skeleton_id` makes it possible to request a particular
skeleton ID during import, just like it is done for neurons using `neuron_id`.
If a skeleton or neuron with this ID exists already, a new object is created
and the existing one is not touched. If an error should be raised instead, set
the `auto_id` parameter to `false`. If instead the passed in IDs should
replace existing data, the `force` parameter can be set to `true`. Both
options apply to both neurons and skeletons.
- `POST /{project_id}/skeletons/connectivity/csv`:
The new optional parameter `names` makes it possible to pass in a mapping of
skeleton IDs to names used in the CSV export as column and row headers. If
this parameter is not provided, the plain skeleton IDs will be used as it was
done before. If it is provided, it has to be a list of two-element lists, each
of the form [<skeleton-id>, <name>], which provides the mapping.
- `POST /{project_id}/skeletons/connectivity`:
The `source_skeleton_ids` parameter can now also be specified in regular form
format (multiple arguments with the exact same name), rather than only the
square braces style.
Deprecations
------------
None.
Removals
--------
- `POST /{project_id}/node/nearest`:
Replaced with `GET /{project_id}/nodes/nearest` (note the plural of nodes).
- `POST /{project_id}/node/most-recent`:
Replaced with `GET /{project_id}/nodes/most-recent`.
We just released a new CATMAID version, 2019.06.20:
https://github.com/catmaid/CATMAID/releases/tag/2019.06.20
You will find a list of changes as we well as some update notes at the
end of this email and on the website linked above. As always, please let
us know if you run into any problems, both as administrator and user.
If you update a CATMAID instance, please use the most recent commit of
the master branch in our GitHub repository, which includes all future
bug fixes for this release:
https://github.com/catmaid/CATMAID
If you are administering a CATMAID server, the "Notes" section provides
information on how to migrate to the new version. The file UPDATE.md
contains the migration steps for all versions. Please also note that
this is the last version of CATMAID that supports Python 3.5 and
Postgres 10. With the next release, Python 3.6 or 3.7, Postgres 11 and
PostGIS 2.5 will be required.
Cheers,
Tom
2019.06.20
==========
Contributors: Chris Barnes, Albert Cardona, Andrew Champion,
Stephan Gerhard, Pat Gunn, Tom Kazimiers, William Patton
Notes
-----
- A virtualenv update is required.
- All \*.pyc files in the folders `django/applications/catmaid/migrations/`,
`django/applications/performancetests/migrations/` and
`django/applications/pgcompat/migrations/` need to be deleted.
- Python 3.7 is now supported.
- Heads-up: The next CATMAID version will require Postgres 11, PostGIS 2.5 and
Python 3.6 or 3.7.
- Should migration 0057 fail due a permission error, the Postgres extension
"pg_trgm" has to be installed manually into the CATMAID database using a
Postgres superuser:
`sudo -u postgres psql -d <catmaid-db> -c 'CREATE EXTENSION pg_trgm;'`
- CATMAID's version information changes back to a plain `git describe` format.
This results overall in a simpler setup and makes live easier for some
third-party front-ends, because the commit counter is included again. The
version display is now the same `git describe` format for both regular setups
and Docker containers.
- Tile loading is clamped to (0,0) again, i.e. there are no negative tile
coordinates anymore by default. If you need them, set the respective stack's
`metadata` field to `{"clamp": false}`.
- To write to CATMAID through its API using an API token, users need to have
now dedicated "API write" permission, called "Can annotate project using
API token" in the admin UI. To restore the previous behavior (regular annotate
permission allows API write access) the settings.py variable
`REQUIRE_EXTRA_TOKEN_PERMISSIONS` can be set to `False`. This is done as a
safety measure to prevent accidental changes through automation.
- If R based NBLAST is used, make sure to execute to update all dependencies:
`manage.py catmaid_setup_nblast_environment`.
- The main documentation on catmaid.org has now a place for widget specific
documentation as well. Only a few widgets have been updated yet, but more will
follow.
Features and enhancements
-------------------------
Tracing tool:
- Add a new Tracing Tool icon button to compute the distance between two nodes
on a skeleton. Respects virtual nodes.
- The globally nearest node can now be selected and brought into view using
Alt + G, as opposed to selecting the nearest node in the current section
using the G key without modifier.
- Using the H shortcut now without an active skeleton, the most recently edited
node of the current user (Alt: anyone) in the last active skeleton will be
selected. Using Shift will look in all skeletons.
- Using Shift + Alt + Click with a connector selected, will will now create a
presynaptic node. This is consistent with the already existing Shift + Alt +
Click behavior with a treenode selected, which creates a presynaptic
connector.
Graph widget:
- GraphML files can now be imported, positions and colors are respected. This is
useful if layouting is done in e.g. Gephi and coloring should be done in
CATMAID. The help page explains a possible workflow for this.
- The button "Group equally colored" in the "Main" tab will group all skeletons
with the same color into a single group. The user is asked for group names for
each color.
3D viewer:
- A new synapse coloring mode has been added: polyadicity. The number of partner
nodes for each connector is color coded (for synaptic connectors, this is the
number of postsynapses). The colors and ranges can be configured through the
"Polyadicity colors" button in the "Shading parameters" tab. This is basically
a configurable version of an absolute "N with partner" coloring.
- Branches with leaf nodes tagged "not a branch" can now be collapsed using the
'Collapse "not a branch"' checkbox in the Skeleton Filters tab.
- The visibility of radius information can now be controlled using the "Show
radius" checkbox in the "Views settings" tab.
- History animations can now be exported in full length without requiring to
guess the number of frames for the export. The animation export dialog will
show an additional checkbox ("Complete history") if a history animation should
be exported. If complete history is enabled, CATMAID will export the complete
history of the exported skeletons.
- Animations can now be exported as stream directly to a file, which allows for
much larger exports (32GB maximum at the moment).
- Fractional rotations are now allowed in the animation export.
- The default time per rotation in the animation export is set to 15 seconds now,
slowing down the default by a factor of 3, which makes it easier to look at.
- Stack Z slices can now be animated. Configurable are the change frequency and
the change step in terms of sections. This is available for animation exports
as well.
- Stack Z slices can now be thresholded to replace a background color with
another color. If enabled and the sum of all channels is in a configurable
range [a,b] it will be replaced with another color.
- The rotation time for animations can now specified in seconds rather than
angular distance.
- The background color is now fully adjustable through the "backgorund" button
in the "View settings" tab.
- Basic support for Virtual Reality is now available on Windows platforms using
a Mixed Reality / SteamVR.3D setup and Firefox >= v65. To enable, check the
VR checkbox in the "View" tab and click the "Enter" button right next to it.
Skeleton history widget:
- A basic view of the change of a set of skeleton IDs over time based on all
nodes that are part of a given skeleton ID or that have been in the past.
- Skeleton history can also be used with past skeleton IDs to see into what
skeleton they changed (if any).
- All past and present treenodes with a passed in skeleton ID are tracked
through the complete history and their path of skeleton ID changes is
recorded along with the number of treenodes following a given skeleton path.
- The widget shows a graph from origin skeletons to the final skeleton IDs in
every available path, summing the treenode counts for each contributing path.
- Existing skeletons are colored in yellow, past skeletons are colored in cyan.
Selected skeletons are colored green.
- Ctrl+Click on skeleton will select it and go to the closest location in it.
Shift+Click allows selecting multiple skeletons. All selected skeletons are
available through the Skeleton Source interface.
Node and skeleton filters:
- Filter rules support now an "invert" option during creation, which allows to
create filters that include everything but whatever is matched by a particular
filter strategy. This can be useful e.g. during neuron review to only look at
segments that have been created by people other than oneself or connectivity
everywhere excluding a particular compartment.
Measurement table:
- Node filters are now supported. Like in other widgets, the respective panel
can be opened through the funnel icon in the widget title bar. Measurements
are displayed for all independent fragments of a skeleton after a set of node
filter rules has been applied.
- With the help of the "Sum fragments" toggle all fragments that result from a
node filter application can be aggregated into one row by summing individual
values.
Remote CATMAID instances
- Some tools in CATMAID gained support to communicate with other CATMAID
instances, e.g. the Landmark Widget (see below). To enable this functionality,
remote CATMAID instances can now be added to the user settings.
- The Settings Widget contains now a section labeled "Other CATMAID instances".
It allows to manage a list of other CATMAID servers based on their URL, an API
key and optional HTTP authentication. This information is stored under an
alias.
Landmarks:
- Clicking on the name of a landmark group in the Landmarks tab will now open
the group member edit dialog that was previously accessible by clicking any
landmark name. Clicking a landmark name will now show a new dialog, which
allows to specify of which landmark groups the clicked landmark is a member
of.
- The color of each active display transformation can be established separately
in the respective table row. Color changes are now also visible in the 3D
Viewer.
- It is now possible to load skeletons and landmarks from other CATMAID
instances. Point matches are done on the basis of matching landmark names. To
enable the UI for this, check the "Source other projects" checkbox in the
Display tab.
- With the "other projects" UI enabled and a remote CATMAID instance configured
in the Setting Widget (see above), it is now possible to select a remote
CATMAID instance from the "Source remote" dropdown menu. Alternatively, the
local instance can be selected if another project from the same instance
should be used.
- Next the source project in the selected instance needs to be specified. This
list is updated when a different remote instance is selected.
- Skeletons from a remote instance are collected through annotations. The
respective annotation has to be entered in the "Source skeleton annotation".
With the help of the "Preview" button it is possible to load the matching
skeletons from their remote CATMAID to inspect if the correct ones are
selected.
- As a last step for the remote data configuration, the source landmark group
has to be defined. This list is updated if the source project changes.
Landmarks from this group are mapped to the selected target group. The
matching is done by name, i.e. no landmarks can have the same names in a
group.
- Adding such a transformation adds it to the list at the bottom of the widget,
just like with regular transformations and they can be used in the same way.
The can be shown in 3D Viewers, superimposed on the Tracing Layer and used in
NBLAST queries from the Neuron Similarity Widget (see below).
- The new checkbox 'Multiple mappings' displays additional user interface
elements to add multiple source and target landmark group mappings for a
single display transformation. Point matches are created independently for
each pair and only merged for finding the final transformation.
- Using a transformation is now optional (but enabled by default). To disable
landmark transformations, uncheck the "Apply transformation" checkbox. This
allows e.g. to display skeletons from remote CATMAID instances without
modification.
Neuron Similarity:
- The computation of the default "mean" normalized similarity scores is now
considerably faster.
- The new "Top N" option allows to store only the Top N best matches in the result
set. For "mean" normalization, it will also only compute the mean score for
the top N forward hits (mean is the average between forward and backward
score). A value of zero disables this cutoff (default). This cutoff can be
used speed up computation for very large sets of neurons or large point
clouds.
- The new "Reverse" option allows to rank similar objects by there reverse
score. That means that the stored score for a particular object pair is target
versus query rather than query versus target. NBLAST uses the query neuron as
reference, and thereforoe the reverse option can make a difference as long as
no "mean" scoring is used.
- It is now possible to use display transformations defined in the Landmark
Widget that reference another CATMAID server as their data source. They can be
used both as source and target for NBLAST comparisons. This makes it
essentially possible to compare skeletons from another CATMAID project,
possibly on another CATMAID server, to local skeletons using NBLAST.
- Similarity matrices can now include transformed skeletons in their set of
similar objects. To do so, create a Display Transformation in the Landmark
Widget that represents the wanted transformation and the select it from the
"Sim. transformed skeletons" drop down in the Configurations tab of the Neuron
Similarity Widget when creating a similarity matrix.
- Similarity matrices can now be created with more control over which skeletons
are looked at as similar. The drop down menu "Similarity mode" allows to
select different groupings of the input skeletons (and transformed skeletons).
It is for instance possible to group a neuron and its contralateral partner
that has been transformed to the side of the first neuron as similar. A
confirmation dialog presents the computed grouping before actually creating a
similarity matrix.
- Similarity matrices can now be imported from CSV files, optionally including
the distance binning and dot binning as header row and column. As part of the
import, the distance binning can be scaled to adjust for differences in
datasets. This is available through the "Create from CSV file" button in the
Configurations tab.
- Similarity query results can now be used as skeleton source in other widgets,
if the target type of the query are skeletons.
System check widget:
- A subset of important database statistics are now displayed if a user has the
"can_administer" permission in a project. These statistics include e.g. cache
hit ratio, temporary files, requested checkpoints and replication lag. For
most of them a rule of thumb suggestion on how a value should behave is
provided as well.
Extension: CATMAID-autoproofreader:
- This extension is written by Will Patton and has to be installed separately,
because it requires additional setup steps for segmentation data handling.
Details can be found here: https://github.com/pattonw/CATMAID-autoproofreader.
- This tool suggests locations in a skeleton reconstructions where branches
might be missing or wrongly connected. Users can step through these
suggestions to fix potential problems. After reviewing a node it can be marked
as reviewed for future reference.
- In the CATMAID front-end the autoproofreader provides a widget that relies on
a compute server to handle the computations involved with automatically
proofreading a neuron reconstruction. Proofreading jobs are performed
asyncronously and might take a few minutes to complete.
Administration:
- A grid based node query cache can now be used to speed up tracing data
retrieval by precomputing intersection results. This can be useful for larger
field of views in big data set. It supports multiple levels of detail per grid
cell and can therefore provide a somewhat uniform sampling when limiting the
number of displayed nodes. The sorting of this LOD configuration can be
controlled so that e.g. only the top N largest skeletons will be shown per
cell with growing LOD. The documentation has more details. This cache can be
kept up updated using two extra worker processes that listen to database
changes, implemented as management commands `catmaid_spatial_update_worker`
and `catmaid_cache_update_worker`.
- The database can now emit notifications on tracing data changes using the
"catmaid.spatial-update" channel and when grid cache cells get marked dirty
on the "catmaid.dirty-cacche" channel. These can be subscribed to using
Postgres' LISTEN command. To enable emitting these events and let cache update
workers work, set `SPATIAL_UPDATE_NOTIFICATIONS = True` in `settings.py`. This
is disabled by default, because sending events for spatial cache updates can
add slightly more time to spatial queries, which will mainly be relevant for
importing and merging large skeleotons.
- Projects can now be deleted along with the data that reference them (e.g.
treenodes, ontologies, volumes). To do so select the projects to delete in the
admin project list and select "Delete selected" from action drop down.
- Users can now be set to an inactive state after a specified amount of time.
This time is configured for a user group and it applies to users if they are
members of such a group. A message can optionally be displayed as well as
users to contact that could potentially help. This is configurable as "Group
inactivity period" in the admin view. Users to contact for support, can be
either configured there or from individual user views.
CLI Exporter:
- If both required annotations and exclusion annotations are provided, the
importer will now only exclude a skeleton if it is not also annotated with a
sub-annotation of the required annotations set. This behavior can be disabled
and exclusion can be enforced when a skeleton is annotated with the exclusion
annotation or one of its sub-annotations. To do so, use the new
"--exclusion-is-final" switch.
- Volumes can now be exported by using the `--volumes` switch. By default all
volumes of the exported project will be included. This can be further
constrained by using `--volume-annotation <annotation-name>` arguments.
- The way she exported objects are specified through the command line interface
changed. Instead of writing e.g. `--notreenodes` to not import treenodes from
a source, `--treenodes false` has to be used now. This is the case for
treenodes, connectors, tags and annotations. The defaults (when the argument
is not provided) stay the same. To explicitly disable the export of a type
`false`, `no`, `f`, `n` and `0` can be provided as argument, e.g. `-treenodes
false` or `--users n`. For a positive parameter use `true`, `yes`, `t`, and
`1`.
CLI Importer:
- Precompute materializations (edges, connectors) explicitly only for imported
data, which improves performance in typical scenarios (import is small
compared to existing data). If the old behavior of recomputing everything in
the target project should be used in cases where a full data base is imported,
the --update-project-materializations switch can be used.
- The way she imported objects are specified through the command line interface
changed. Instead of writing e.g. `--notreenodes` to not import treenodes from
a source, `--treenodes false` has to be used now. This is the case for
treenodes, connectors, tags and annotations. The defaults (when the argument
is not provided) stay the same. To explicitly disable the import of a type
`false`, `no`, `f`, `n` and `0` can be provided as argument, e.g. `-treenodes
false` or `--users n`. For a positive parameter use `true`, `yes`, `t`, and
`1`.
- Imported usernames can now be mapped to existing usernames and mapped
selectively by using one or more parameters of the form
--username-mapping="import-username=target-username". The example would map
all references to the user import-username in the imported data to the
existing user target-username. The mapping is performed even with --map-users
off.
- It is now possible to import volumes from CATMAID export files and other
projects..
Miscellaneous:
- Tracing layer: a minimum skeleton length can now be specified in the layer
options, causing the Tracing Layer to show only nodes of skeletons of at least
this cable length (nm).
- Tracing layer: a minimum number of nodes can now be configured to only show
skeletons of at certain minimum size (just like the existing minimum cable
length constraint). This is configurable through the layer configuration,
accessible by clicking the blue/white box in the lower left corner of stack
viewers.
- Settings widget: visibility group conditions can now be inverted.
- Confirming a radius selection (Shift + Y) can now also be done using the Enter
key.
- Neuron name searches and annotations should be much faster on larger
instances.
- Basic support for touch screens (e.g. phones and tablets) is now implemented.
- CATMAID can now export files (e.g. CSVs, videos, etc.) of much larger size
than before, if "Save exported files in streaming mode" is enabled in the
Settings Widget. To make this work, the browser settings (chrome://settings)
"Ask where to save each file before downloading" has to be enabled.
- Context aware help: by clicking the new question mark icon in the upper right
corner, a context aware help dialog can be displayed on top of all other
CATMAID tools. It provides some general guidance for the intended workflow
with individual tools. This can be enable to be displayed by default as part
of the Settings Widget. The display of this can also be enabled in a link to a
view by appending `&help=true`. This will be added automatically if a help
dialog is open during URL creation.
- Connectivity widget: the new "List links' button will open a Connector List
with all links of the selected neurons.
- Selection table: the new checkbox "Link visibility" allows to change the
behavior of the visibility checkboxes. So far the pre/post/text/meta
select-all controls were only affecting the respective visibility options of
visible/selected skeletons (default). With unlinked visibility
pre/post/text/meta can also be controlled for all neurons, regardless of their
visibility. This allows e.g. to only show synapses, but no skeletons.
- General search widget: a more flexible table is now used for display, which
allows sorting, filtering and pagination.
- Neuron dendrogram: select currently active node by default when loading the
dendrogram for the active skeleton.
- The right end of the status bar contains now a button to toggle the visibility
of the top toolbars.
- Updating the copy of client side annotations will only request new data if
there are actually new annotations. This makes loading of e.g. the Neuron
Search widget faster.
- Boss databases can now be used as tile source type so that image data is
loaded from them. More details: https://docs.theboss.io/docs/image.
- Docker: HTTP basic authentication can be configured by using the environment
variables HTTP_AUTH_ENABLED, HTTP_AUTH_USER and HTTP_AUTH_PASS in the web
container of the `docker-compose.yml` file (an example is given).
- Performance: selecting the closest node in a skeleton should now be faster.
Bug fixes
---------
- A race condition has been fixed that could in rare cases lead to inconsistent
skeleton IDs in a skeleton that is merge into different skeletons by different
requests.
- Landmark transformations: fix Moving Least Square transformations for skeleton
fragments outside of source group bounding box.
- The Connectivity Graph Plot draws now individual bars in each sub-plot
side-by-side and uses the same neuron names as other widgets (from CATMAID's
neuron name service).
- Information based on historic information like the history animation in the 3D
Viewer and the Neuron History widget includes merge information now correctly.
Before, merge nodes were represented twice in a time slice that included the
merge.
- The default connector type is now properly persisted as a setting.
- Neuron search: no duplicate entries are shown anymore, which could result e.g.
when sub-annotations where allowed.
- Neuron similarity: when showing result skeletons in 3D, skeletons are now
appended to the Selection Table of a 3D Viewer, rather than the 3D Viewer
directly.
- Connectivity widget: the auto-connectivity CSV will now use formatted neuron
names as column and row headers.
- Connectivity widget: the 'Export CSV' button will now respect name and
annotation filters.
- Graph widget: ungrouping of one-element groups is now allowed.
- Graph widget: merging of grouped skeletons is now handled properly.
- 3D viewer: the depth test for connector partner spheres is now performed
correctly and spheres should be rendered in the correct Z order.
- 3D viewer: mouse controls now work correctly in fullscreen mode.
- 3D viewer: the suggested width and height of the animation export are now by
default a factor of two. This is required by the WebM encoder. Off values are
reduce by one.
- 3D viewer: spatial select works now also without the active node highlighted.
- 3D viewer: no error is shown any more when attempting to move while in radius
edit mode.
- 3D viewer: landmark transformed skeletons show now also radius spheres.
- Connector table: the section and tag columns are now part of the CSV export.
- Tracing tool: unneeded node updates are removed from initial tracing layer
loading.
- Selection table: don't try to load missing skeletons from JSON file.
- Settings widget: default values for tracing layer skeleton limits can now be
configured under Tracing Overlay > Tracing layer skeleton filters.
- The numpad delete key is now recognized as regular delete (if numlock is off).
- SWC neuron import: providing a neuron ID for an import works again and will
now also set an optionally provided neuron name. Additionally, it is now
required that a user importing skeleton data for an existing neuron ID has
the rights to edit the skeleton instance as well as all its treenodes.
- Volume widget: volumes/meshes with annotation can now be removed properly.
- Volume widget: editing a volume by double clicking its table entry works
again.
- Neuron dendrogram: correctly reload skeleton if it changes as a result of a
split or merge. If a dendrogram node is selected, the respective skeleton will
be reloaded after a split, even if its ID changed.
- The Notification Table can be opened again without errors.
- Exporter: connector links are now exported properly if the parameter
--original-placeholder-context is used.
- Docker: the docker-compose setup now uses Postgis 2.5 internally and therefore
allows upgrades from Postgres versions < 10 with Postgis 2.4.
- Docker: stale Postgres PID files will now be removed during a database upgrade
in a docker-compose setup. PID files without actually running database
processes prevented some updates before.
- Initial setup: create_configuration.py now also prints the media files
directory as part of the webserver example configuration. This is the folder
where e.g. some exported files are made available.
API changes
-----------
Additions
---------
- `GET /{project_id}/nodes/nearest`:
Replaces `POST /{project_id}/node/nearest`. The parameters are the same, but
the API allows now to look globally for the nearest node in the project, if no
skeleton ID or neuron ID is provided.
- `GET /{project_id}/nodes/most-recent`:
Replaces `POST /{project_id}/node/nearest`. A skeleton_id parameter can still
be provided, but now also a user_id parameter is available to further
constrain.
Modifications
-------------
- `POST|GET /{project_id}/node/list`:
Offers a new optional parameter "min_skeleton_length", which can be used to
constrain the returned neurons to only those of at least this cable length.
- `POST|GET /{project_id}/pointclouds/`:
Offers a new optional parameter "order_by", which accepts the strings 'id' and
'name' to define in what order the list of pointclouds should be retuned
(default: id).
- `POST /{project_id}/landmarks/{landmark_id}/`:
Offers a new optional parameter "group_ids", an array of integers, which
allows to set the landmark group memberships of a specific landmark. The new
boolean parameter "append_memberships" allows to only append new group IDs as
memberships, without removing any. Otherwise the whole set of memberships is
replaced.
- `POST /{project_id}/skeletons/import`:
The new parameter `skeleton_id` makes it possible to request a particular
skeleton ID during import, just like it is done for neurons using `neuron_id`.
If a skeleton or neuron with this ID exists already, a new object is created
and the existing one is not touched. If an error should be raised instead, set
the `auto_id` parameter to `false`. If instead the passed in IDs should
replace existing data, the `force` parameter can be set to `true`. Both
options apply to both neurons and skeletons.
- `POST /{project_id}/skeletons/connectivity/csv`:
The new optional parameter `names` makes it possible to pass in a mapping of
skeleton IDs to names used in the CSV export as column and row headers. If
this parameter is not provided, the plain skeleton IDs will be used as it was
done before. If it is provided, it has to be a list of two-element lists, each
of the form [<skeleton-id>, <name>], which provides the mapping.
- `POST /{project_id}/skeletons/connectivity`:
The `source_skeleton_ids` parameter can now also be specified in regular form
format (multiple arguments with the exact same name), rather than only the
square braces style.
Deprecations
------------
None.
Removals
--------
- `POST /{project_id}/node/nearest`:
Replaced with `GET /{project_id}/nodes/nearest` (note the plural of nodes).
- `POST /{project_id}/node/most-recent`:
Replaced with `GET /{project_id}/nodes/most-recent`.
Reply all
Reply to author
Forward
0 new messages