New CATMAID release: 2018.11.09

[Tom Kazimiers]

Hi all,

We just released a new CATMAID version, 2018.11.09:

https://github.com/catmaid/CATMAID/releases/tag/2018.11.09

You will find a list of changes as we well as some update notes at the
end of this email and on the website linked above. As always, please let
us know if you run into any problems, both as administrator and user.

If you update a CATMAID instance, please use the most recent commit of
the master branch in our GitHub repository, which includes all future
bug fixes for this release:

https://github.com/catmaid/CATMAID

If you are administering a CATMAID server,, please note that this
version of CATMAID drops support for Python 2. You need Python 3.5 or
Python 3.6 to run CATMAID, we recommend Python 3.6.

Cheers,
Tom


2018.11.09
==========

Contributors: Andrew Champion, Chris Barnes, Tom Kazimiers, William
Patton, Eric Trautman

Notes
-----

- Python 3 is now required for the back-end. We recommend the use of
Python 3.6.

- CATMAID's version information is now presented in a different form. It
follows the pattern <base-version>[-dev]-<commit>. The <base-version>
is baked into the source code on a release. The -dev part will only be
present if CATMAID's dev branch is used for deployment. It won't be
present for master branch based setups. The <commit> part is the 10
digit version of the Git commit ID. This version representation is now
also consistent with what is display in Docker images. In the rare
event that no commit information can be found, <commit> will fallback
to "unknown". This version will now also logged during start-up of the
back-end.

Features and enhancements
-------------------------

Connectivity matrix:

- The new checkbox labeled "Fractions" in the "Main" tab makes it now possible
to display connector fraction instead of an absolute link number in each cell.
The number of connections from one source row to a target column is divided by
the number of total posynaptic connections that are made to the target
skeleton (column). This makes columns better comparable to each other.

- The auto-connectivity matrix of a large set of skeletons can now be exported
as CSV without displaying it using the "Auto-connectivity CSV" button. This
makes it possible to export larger connectivity matrices in a usable format.

- The aggregation method for the connectivity count in groups can now be
selected. Available are: sum (default), min, max and average.

3D viewer:

- Treenodes that are linked to connector nodes can now be scaled independently
from other node handles using the "Link node scaling" option.

- Volume picking is now optional and disabled by default, i.e Shift + Click will
go through volumes. To enable volume surface location selection, the
"Pickable" checkbox needs to be checked.

- If Reconstruction Sampler domain shading or interval shading is used, a list
of valid domains and intervals can now be specified in the "Shadings
parameter" tab.

- The X/Y/Z rainbow coloring modes are now also available normalized to each
individual skeleton.

- Add custom connector coloring. The pre and post colors can be adjusted in the
"Shading parameters" tab.

- The X/Y/Z/ axes can now be displayed in the lower right corner using the
"Axes" checkbox in the "View settings" tab.

Reconstruction sampler:

- Improve performance of interval length computation

- Sampled skeletons can now be split after the users confirm this is their
intention. The split-off part of the skeleton will not contain any sampler
information anymore. Intervals on the split-off part are removed, split-point
crossing intervals are shortened and domain end points will be removed and
recreated as needed.

- Sampled skeletons can now be merged into. All samplers that reference the
merged-in fragment are deleted. If the merged fragment is merged outside
of a domain, nothing special is happening---it is a regular merge. If the
merge treenode is in a sampler domain, there are currently three options,
"Branch", "Domain end" and "New domain": 1. Branch: add the new fragment to the
skeleton without changing domain end nodes or intervals. This is only allowed
if the merge target is not the start or end of an interval. 2. Domain end: add
a new domain end node right where the merged in fragment starts. This keeps
the new fragment isolated from the sampled domain. 3. New domain: a new
domain is created for the merged in fragment. This also adds the domain end
node from (2).

- Domain completion is now shown in percent along with interval coverage of the
domain in "Interval" step.

- Merge decisions can now be limited when the 'merge' or 'merge-or-create' leaf
handling mode is selected. This means a percentage can be specified which
defined below which ratio of extra cable versus interval length the extra
cable should be merged into the previous interval (if possible) rather than a
new interval is created.

- Ignored lead segments can now be inspected in more detail using the
"Uncovered domain parts" button the "Interval" step. This will open a dialog
window with a histogram on all ignored leaf segments in the current domain.
Clicking on individual bins will open a treenode table containing the
respective start nodes of the ignored leaf segments. From this dialog, it is
also possible to list all downstream/upstream partners linked to nodes in
ignored segments of a domain.

- For samplers using the 'ignore' leaf handing mode, it is now possible to
update this to 'short-interval' mode including the generation of missing
intervals for existing domains. A visual confirmation dialog is shown. To use
this, press the "Set short-interval leaf mode" button in the Domain tab.

- The Synapse tab now also shows all leaf nodes of an interval. This makes it
easier to find the places where an interval needs to be continued.

Tracing layer:

- Tracign layer: cycle open end in reverse using Shift + Alt + R.

- Alt + Click now opens consistently the link type context menu, regardless of
whether a treenode or connector node is currently selected.

- The layer options now allow to select a user who's tracing data won't be
fetched from the server. The main motivation is to hide data imported by a
dedicated import user by default and not even fetch it from the server.

- Similar to image data mirrors, it is now possible to configure read-only
tracing data mirror servers from which the tracing data will load all data
except for the active node, which will be read from the main server. This is
particularly useful if connecting to the main server from a remote location.
To make this work reliably, it is expected that physical replication is setup
on the database level that mirrors the main server constantly. A separate
CATMAID instance needs to be setup on the mirror server as well. To configure
this, the "Read-only CATMAID mirrors" option in the settings widget can be
used together with the "Read-only mirror index".

Neuron similarity:

- The new Neuron Similarity Widget makes it possible to compare neurons to each
other, to neurons transformed based on landmarks as well as to arbitrary point
clouds. Point clouds can for instance be created from light microscopy data.
It creates a similarity ranking based on NBLAST. To open the widget, use Ctrl
+ Space or the Open Widget button and then search for "Neuron similarity".

- To compare two different objects, NBLAST will compare a query object pairwise
with potential target objects. It iterates over each point of the query
object, find the closest point in a target object and computes a score based
on the distance of these points and their orientation to each other.

- This scoring is done based on a scoring matrix, which needs to be created
before any comparisons can be made. Scoring matrices are typically reused and
don't need to be recomputed every time. The "Configurations" tab allows to
create new similarity matrices and lists existing ones. For a new scoring
matrix, probabilities for distance and orientation are computed for both a set
of of similar neurons and a representative sample of random neurons. Both are
combined into a single matrix in which a value of zero makes a particular a
pair of points equally likely to be random or to be a match. Values above zero
make a match more likely. Computed similarity matrices can be visualized by
clicking the "View" link in the Scoring column of the respective similarity
configuration.

- With a similarity matrix computed, similarity queries ca be performed from the
"Neuron similarity" tab. In its most basic form, this compares neurons to
other neurons. It is also possible to select transformed neurons and point
clouds as query type or target type in a search. This however requires
additional setup (see below). Query and target skeletons can be selected by
selecting a skeleton source for each. A similarity matrix has to be selected
as well, but all other options have reasonable defaults. A click on "Compute
similarity" queues a new similarity request, which is computed asynchronously.
Once the task is complete its table entry will switch its status to
"complete".

- Once completed, the similarity query results can be viewed by clicking on
"View" in the "Scoring" column. This will open a new result window (or
dialog, if selected in the "View" option), which shows the similarity ranking.

- To query with or against transformed skeletons, a landamark based "display
transformation" has to be created. To do so, open the Landmark Widget, and
create a transformation in its "Display" tab. Transformations created this way
are selectable from the Similarity Widget, if "transformed skeletons" is
selected for either query or target. Depending on available landmark groups,
this could be for instance a skeleton transformation to its contralateral
location.

- The Point cloud tab allows to import individual point clouds, along with an
optional transformation and representative images. It also provides a list of
all point clouds that are visible to the current user. A group visibility
option during import allows to restrict visibility of imported point clouds to
selected groups (which need to be added from the admin interface).

- The "Point cloud import" tab allows to import many point clouds at the same
time, optionally transformed and with linked representative images.

Volumes:

- The general widget controls are now distributed across tabs.

- Annotations can now be used on volumes too. The "Annotate" button in the
Volume Manager can be used to annotate all selected volumes.

- The "Add from file" button in the Volume Manager can now be used to import
volumes from STL files.

- The "Skeleton innervations" tab allows to search for volumes that intersect
with a set of query skeletons. A volume annotation can be specified to look
only at volumes having this annotation. Optionally, exact result computation
can be disabled to only compute skeleton/bounding-box intersections. This is
slightly faster, but leads to false positives.

Docker:

- More CATMAID configuration options are now accessible through Docker
environment cariables: CM_DEBUG, CM_FORCE_CONFIG_UPDATE, CM_WRITABLE_PATH,
CM_NODE_LIMIT, CM_NODE_PROVIDERS, CM_SUBDIRECTORY, CM_CSRF_TRUSTED_ORIGINS.

- The Git commit from which a Docker image was built is now preserved and
included in the CATMAID's version information.

CLI importer:

- If no user information except for IDs is present in the imported data, the
importer will by default ask for a username and create new inactive users for
those IDs (and update the referenced IDs). Alternatively, the --map-user-ids
parameter can be specified, which will make the importer map referenced IDs to
existing users. If an existing user with the respective ID is not available,
the user is asked for a username and a new user will be created.

- With the help of a few additional progress bars, import progress can be better
monitored.

- Database statistics are now automatically recomputed after an import, i.e.
ANALYZE is run.

Miscellaneous:

- Each Tracing Layer is now listed as a skeleton source. All skeletons visible
in their fields of view are made available to other widgets that way.

- Tracing tool: add icon button to toggle a node coloring mode in which each
node is colored according to the length of their respective skeleton. The
colors and cable length values when to use it can be set for three colors in
the Settings Widget in the "Skeleton length coloring" section.

- The SWC exporter can now optionally mark a node as soma if it either is tagged
as "soma", if it has a radius larger than a defined value or if it is the root
node. All three conditions can be selected, and they will be applied in the
order they are listed above. If selected, a soma tagged node will always take
precedence.

- There is small copy-to-clipboard button left to the "URL to the view" link in
the upper right corner of the user interface.

- The Ctrl modifier can now be used with - and + to animate zooming.

- New widget: stack info, which displays properties for stacks related to the
active project.

- The node cache update management command `catmaid_update_cache_tables` can now
update all caches configured in the NODE_PROVIDERS settings variable
automatically when the `--from-config` option is provided.

- Admin: a projects/export JSON export of the visible project/stack structure
can now be used directly in the Project/stack Importer by selection "JSON
representation" as source and pasting the data into the text field.

- Neuron search: regular expressions are now optional for the neuron name.
Unless the search string starts with '/', no regular expressions are used,
but a regular case insensitive text search.

- Two new connector types are available: "tight junction" and "desmosome". Both
can be created through the Alt + Click menu, are reciprocal and two links at
one connector are allowed for each type.

- Connectivity widget: all available link types can no be selected to be
displayed. Both the gapjunction and attachment checkboxes have been removed in
favor of a more generic list select element.

- Tracing layer: the displayed tracing data can now be constrained to show only
the N most recently edited skeletons. This settings adds to the skeletons
selected by the N largest skeletons filter.

- WebGL layers are now preferred by default.

- Project statistics: a top ten of the largest neurons is now displayed.

- Admin: a user import view is now available to import users from other CATMAID
instances. It requires superuser permissions on the remote instance.

- DB integrity check management command: volumes are now checked to make sure
all faces are triangles. The --tracing [true|false] and --volumes [true|false]
command line parameters now allow to explicitly test only some parts of the
database. By default all is tested.


Bug fixes
---------

- 3D viewer: loading a single node skeleton with smoothing enabled no longer
causes an error.

- 3D viewer: nodes taged with 'uncertain' can be loaded again.

- 3D viewer: various rendering bugs for Reconstruction Sampler domains and
intervals have been fixed.

- 3D viewer: stored node scaling settings are now properly restored.

- 3D viewer: TODO tag coloring doesn't override custom label colors anymore.

- 3D viewer: all synaptic site spheres are created and colored again.

- 3D viewer: camera won't flip anymore if it is upside down during animations.

- Tracing overlay: the border of the tracing window is now properly rendered.

- Tracing overlay: child node edition times are now correctly updated if the
parent is deleted, which fixes occasional state matching errors.

- Tracing overlay: Shift + Click now works also with attachment connectors,
should they be selected as default connector type in the settings.

- Reconstruction sampler: deleting samplers while other samplers on the same
neuron refer to the same created boundary nodes no longer causes an error.

- Reconstruction sampler: cable length columns are now sorted numercially.

- Reconstruction sampler: all settings are now correctly reset when a sampler is
selected or "New session" is pressed.

- Reconstruction sampler: a few corner cases for binary interval coloring have
been fixed. Colors should now alternate in most cases.

- Connectivity matrix: synapse count based ordering works again.

- Connectivity widget: annotations on neurons (used for filtering) are now
properly updated when they are changed in another part of CATMAID.

- Graph widget: edge color updates now trigger a redraw operation again.

- CLI importer: the ID sequence for the auth_user table is now properly reset
after an import.

- CLI importer: missing treenode-connector links are now imported between
connectors and placeholer nodes.

- CLI impoerter: the ID of reused objects is now proplery updated in imported
data when --preserve-ids is used.

- CLI importer: skeleton summaries and edge tables aew correctly created again.

- CLI importer: unmapped imported users are now correctly saved.

- The error dialog prints now linen breaks and spaces correctly again, which
improves its formatting.


API changes
-----------

Additions:

- `POST /{project_id}/skeletons/cable-length`:
The POST version of the already existing GET endpoint. It allows passing in
more skeleton IDs for which to get the cable length.

- `POST /{project_id}/skeletons/connectivity-counts`:
Allows to get the number connector links per relation type for each passed in
skeleton. Also accepts GET parameters.

- `POST /{project_id}/skeletons/connectivity_matrix/csv`:
Return a CSV file containing the connectivity matrix for the passed in `rows`
and `columns`.

- `POST /{project_id}/skeletons/sampler-count`:
Return the number of samplers linked to each skeleton in the passed in
`skeleton_ids` list.

- `GET /{project_id}/skeletons/{skeleton_id}/sampler-count`:
Return the number of samplers linked to this skeleton.

- `GET /{project_id}/labels/detail`:
Returns a list of of label objects, each with a name field and an ID field.

- `POST /{project_id}/volumes/import`:
Import volumes as STL files.

- `GET /{project_id}/volumes/{volume_id}/export.{extension}`:
Export a particular volume. Currentl only exports AS STL are supported.

- `GET /{projec_id}/stats/cable-length`:
Get the top N largest skeletons. N can be specified using the 'n_skeletons'
parameter.

- `POST /{project_id}/volumes/skeleton-innervations`:
Get a list of all volume/skeleton combinations that intersect. Skeletons are
provided with a 'skeleton_ids' parameter. This can be constrained by volume
annotation, cable length and node count.

- `POST /{project_id}/volumes/`:
Previously only the GET method would be supported. With the introduction of an
optional `volume_ids` parameter, POST can be used as well. This parameter can
be used to constrain the returned set of volumes.

Modifications:

- `POST /{project_id}/skeletons/node-label`:
The new `label_names` parameter accepts a list of strings that an be used
instead of or together with label IDs to get skeletons with nodes that have
particular labels.

- `GET /{project_id}/stacks/{stack_id}/info`:
Now includes the `comment` field for each stack.

- `GET /{project_id}/connectors/`:
The partner data returned when the `with_partners` is set, includes now
additionally the creator of the link and the creation and edition time (in
epoch format).

- `GET /user-list`:
Accepts a new boolean parameter `with_passwords` to export encrypted
passwords, which in turn can be imported into other CATMAID instances. This
requires the requesting user to be a superuser.

- `GET /{project_id}/skeleton/{skeleton_id}/swc`:
A new optional parameter `soma_markers` can be supplied. It is a list of
'tag:soma', 'radius:<n>' and 'root'. These are conditions for when a node is
marked as soma in the SWC export. The first matching condition in this order
wins.

- `GET /{project_id}/volumes/`:
The return format changed. Instead of a list of volume objects an object with
a 'columns' field and a 'data' field are returned. The data fields contains a
list of lists, with each inner list being a volume. The entries are described
by the 'columns' field. Along with the already returned fields, annotations
are now retuned as well.

- `GET /{project_id}/volumes/`:
An optional `volume_ids` parameter can now be used and the POST method is
supported as well (to transfer larger volume ID lists). This parameter can be
used to constrain the returned set of volumes.

- `POST /{project_id}/skeletons/connectivity`:
A new optional option 'link_types' allows to specify which connectivity types
should be returned. By default 'incoming' and 'outgoing' is returned.
Additionally, the following options are allowed: gapjunction, abutting,
attachment, close_object. The return field names have also been adapted to
these parameters and match them. Instead of 'gapjunctions' and 'attachments',
the fields are now named 'gapjunction' and 'attachment'.

- `GET /{project_id}/annotations/`:
Supports now an optional boolean parameter named `simple`, which returns only
names and IDs and is therefore much faster.

Deprecations
------------

None.

Removals
--------

None.