New CATMAID release: 2021.12.21
38 views
Skip to first unread message
Tom Kazimiers
Dec 21, 2021, 8:50:01 AM12/21/21
to CATMAID
Hi all,
We just released a new CATMAID version, 2021.12.21
https://github.com/catmaid/CATMAID/releases/tag/2021.12.21
You will find a list of changes as we well as some update notes at the
end of this email and on the website linked above. As always, please let
us know if you run into any problems, both as an admin or a user. The
docker images tagged stable and 2021.12.21 have been (re)built.
If you update a CATMAID instance, please use the most recent commit of
the master branch in our GitHub repository, which includes all future
bug fixes for this release:
https://github.com/catmaid/CATMAID
If you are administering a CATMAID server, the "Notes" section provides
information on how to migrate to the new version. The file UPDATE.md
contains a copy of the migration steps as well. Please note that this
CATMAID version is the last with Python 3.6 support. This release
requires also Postgres 12+ and PostGIS 2.5+. The CATMAID version after
this one will require Postgres 13+ and PostGIS 3.2+, both of which are
supported already.
If you have to update your database, please make sure to update to the
target version of PostGIS in all databases *before* you update the
database cluster, it will make the update go more smoothly. This means
installing the new PostGIS version (e.g. v3), connecting to each
database and running `ALTER EXTENSION postgis UPDATE` (check version
using \dx). The same database version needs to be available for the
target version of Postgres. If you need to update your database, go to
Postgres 13 directly, because the next version of CATMAID will require
Postgres 13 and PostGIS 3.
Cheers,
Tom
2020.12.21
==========
Contributors: Chris Barnes, Albert Cardona, Andrew Champion, Stephan
Gerhard, Sanja Jasek, Tom Kazimiers
Notes
-----
- The version requires PostgreSQL 12. If you also want to upgrade PostGIS,
update PostGIS first and run ``ALTER EXTENSION postgis UPDATE;`` in every
existing database in the cluster that should be upgraded. For docker-compose
setups this database update is performed automatically if `DB_UPDATE=true` is
set for the `db` container (watch the Docker output). CATMAID's documentation
Docker has more information. If a replication setup is in use, the database
configuration changes for Postgres 12. CATMAID's replication documentation
explains what needs to be done.
In Postgres 12, JIT compilation is available to the planner by default. We
found that being more conservative with the use of it helped a few common
queries (like the tracing data field of view). We set the required minimum
cost of JIT use to 1,000,000 in ``postgresql.conf``::
jit_above_cost = 1000000
- If R extensions are used, make sure to use R 3.6. On Ubuntu this can be made
available by first installing the official R PPA repository:
sudo gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-key E084DAB9
sudo gpg -a --export E084DAB9 | sudo apt-key add -
echo "deb https://cloud.r-project.org/bin/linux/ubuntu xenial-cran35/" | sudo tee -a /etc/apt/sources.list
And second update the package index and update the local R setup:
sudo apt-get update
sudo apt-get install r-base r-base-dev mesa-common-dev libglu1-mesa-dev \
libssl-dev libssh2-1-dev libcurl4-openssl-dev cmtk
This also requires updating all installed R packages. In all likelihood this
requires executing "manage.py catmaid_setup_nblast_environment".
- The default downsample factors for a stack without explicit factors defined,
changed slightly: if you relied on the automatic creation of downsample
factors for image stacks, make sure the changed behavior still works for you
or enforce the previous configuration with an explicit specification of
downsample factors. Previously as many zoom levels as needed were created to
map the image size in a quadratic fashion per level onto a 1024x1024px tile.
Instead of hard-coding the 1024px, the minimum tile size defined in the mirror
set is used. Without any mirrors, no extra zoom levels will be assumed.
- The recommended python version is now 3.8, although support for 3.6 is maintained.
- A virtualenv update is required.
Features and enhancements
-------------------------
Notes widget:
- This new and simple widget can be used to take notes either only for the
current project, shared across all projects or globally visible for all users.
The last option is only available to admin users for writing. Use the "Open
Widget" dialog to open it using the "Notes" keyword.
Publication widget:
- This new widget makes management of publication related annotations easier. It
lists all annotations that annotated with one of the publication annotations
defined in the Settings Widget in the Publication section. Each of them
represents a publication for the widget. The export options for each one of
them can now be configured through the table.
Project management widget:
- This new widget allows superusers and users with can_administer role in a
project to edit user and group permissions for this project. This interface is
easier to find and manage then the admin view and allows for more granular
admin permissions. It is also useful to share own spaces (own copies of
projects) with other users.
- The widget allows to define for both user and groups the following
permissions: can browse (read), can annotate (write), can administer, can
import, can queue, compute taks, can write through API, can fork
- Clicking any checkbox will store the value right away.
- If a user has "can_administer" permissions in a project, they can edit the
project title and project description (comment) in the second tab.
- If a user has "can_administer" and "delete_project" permissions in a project,
the project management widget allows to delete the current project. This is
mainly useful for personal spaces, in which users typically have
delete_project permissions.
- The "User data permissions" tab allows to configure user level edit
permissions (who is allowed to edit who's data).
- The "Project token" tab allows project admins to view existing project tokens
("invitation codes") available for a project. For each generated token, the
table allows to copy both the token itself as well as an invitation link using
this token to the clipboard. If a user opens an invitation link, they will
automatically apply the token. Anonymous users are asked to log in first.
- As admin user in a project, it is also possible from to manually change
skeleton IDs from the "ID updates" tab. This is only allowed if the new ID
isn't used already. Also, if the new ID is a new maximum ID number, all new
IDs will be automatically be larger than this ID. This functionality is also
available as the back-end managment command catmaid_change_skeleton_id.
Skeleton ID change management command:
- Just like the Project Management functionality explained in the last
paragraph, the management command catmaid_change_skeleton_id can be used to
change skeleton IDs (and optionally neuron IDs) from the command line.
Link widget and deep links:
- This new widget allows users to create persistent deep links into the
dataset using an alias, allowing for friendlier URLs of the form
`<catmaid-url>/<project-id>/links/<alias>`. The stored parameters are the same
as for regular deep links, with most of them being optional.
- Like with fully parameterized URLs, features like layout, widget settings and
loaded skeletons can be used.
- Links can be marked private, making them only accessible (and listable) by the
creator of the link.
- An optional message can be stored with the link and is shown to the user when
the links is opened.
- The context menu for creating different types of links moved from entries
in the Layouts menu to its own menu that is shown when hovering the mouse
cursor over the "URL to this view" link. The first menu entry is new and opens
a link creation dialog, similar to the Link Widget's 'Add link' dialog.
Neuron renaming:
- There is now a new neuron renaming dialog, which allows a search/replace
renaming workflow for many neurons at the same time. This can be launched
through the "Rename" button in both the Neuron Search widget and the neuron
lists of the Neuron Navigator (e.g. neurons with a particular annotation).
- The dialog supports simple find/replace, but also regular expressions
including capture groups and presents live previews.
Neuron search:
- Neuron search: add option to show meta data columns (cable length, number of
nodes, creation time and last edition time) for neurons.
- Removal tools for annotations are now opt-in, i.e. not shown by default. To
enable them during annotation display, click the "Show removal tools"
checkbox.
- The optional annotation column has now the option to only show annotations
with a particular meta-annotation. This meta-annotation can be configured in
the input field in the table column header.
- The CSV export comes now with a header row to make clear the first column is a
neuron ID.
- The CSV export includes now the skeleton ID as well and will export metadata
like cable length, if metadata display is enabled in the UI.
- Expanded elements (e.g. annotations) are now sorted correctly within their
expansion group. This also fixes the out-of-order expansion display.
- Escape underscore and percent sign properly. They were treated as wildcards
before.
Connectivity Matrix:
- The new buttons "Save JSON" and "Open JSON" can be used to save the row and
column skeletons, including groups and widget settings.
- The loading speed has been improved significantly.
- With the new "Link type" selector in the Main tab it is now possible to show
the connectivity matrix for other link types (like desmosomes or gap
junctions). The selected link type is also respected by the CSV export.
Layouts:
- New layout menu item: "Copy URL to view with layout". This will create a URL
to the current view like the link button on the right hand side of the second
toolbar, but with a layout spec included. It will open all widgets in the new
CATMAID like in the current one. This includes skeletons, their colors and
widget configurations by default. A second menu entry with the "(no
skeletons)" suffix allows to create link with only the layout and no
skeletons. And a third menu entry with the suffix ("no widget settings)"
allows to copy only the layout plus skeletons, without any widget
configuration.
- New layout menu item: "Copy current layout spec". This will copy the layout
specification of the current view to the clipboard. This can be useful when
constructing URLs to specific views.
- Widgets in layouts can now optionally specify a list of skeletons to load.
This is done by adding the `skeletons` parameter:
"skeletons": [16035815, {"id": 16035701, "color": "rgb(0.5,1,0.2)"}]
The list of skeletons can consist of skeleton IDs and/or optionally objects
that can also specify a color in terms of common CSS definitions.
- The Selection Table can now receive an options parameter from the layout
initialization. If it contains a list of skeleton IDs like this, then
selection table will load this automatically:
options: {"skeleton-ids": [16035815, 16035701]}
This has to be added to the URL manually at the moment, by adding it to the
selection table config part in the layout spec.
Volume manager:
- The table of filters applied to input neurons for volume creation has now a
new "Actions" column. At the moment this includes a "Remove" link for each
filter. This makes it easier to try different filters without removing all
other filters
- Annotations can now also be removed from volumes if "Show removal tools" is
enabled.
- For new node filters during volume creation, the merge mode can now be
selected, i.e. whether additional filters will be combined using OR or AND.
- The new "Import from CATMAID" tab allows users with import permission to
search and import volumes/meshes from other CATMAID projects, both local and
remote. Volumes can be searched by name and annotation.
- Don't show removal options by default. It happens generally rarely that one
wants to remove volumes, especially in the skeleton innervation tab. To reduce
the risk of accidental removals (even though a confirmation dialog is shown),
removal buttons are now only shown if the "Show removal options" checkbox is
enabled in the Main tab.
- The connector listing for a volume will now include unlinked nodes.
- The checkbox to show the removal tools is now colored red, to make it easier
to see how to remove volumes.
Graph widget:
- Add options to configure the edge text outline size and opacity. The options
area available from the properties dialog.
- Grouping with undirected edges like desmosomes won't lead to multiple edges
between the same groups anymore.
- Ungrouping groups with members having non-synaptic links (e.g. desmosomes)
doesn't lead to an error anymore.
- Remember selected link types in local widget state storage.
- Applying node filters works now correctly and only removes an edge if both
source and target node are not allowed according to the filter configuration.
It is now also possible to use additional subgraphs (e.g. axon/dendrite
split) with filters applied.
Connectivity widget:
- Larger input and partner sets can now be displayed.
- Make node filter test consistent with filter application in the Graph Widget.
A connection is now valid according to the selected filters if it uses a
connector node that is linked to a node allowed by the node filters. The node
filters are still only applied to the query skeletons. This makes the
filtering numbers consistent with the Graph Widget.
3D viewer:
- The new tab "Volumes & Geometry" contains now all settings related to volumes,
landmarks and point clouds. This makes the "Volume settings" tab a little
less overloaded.
- The View tab has two new controls to control the sensitivity of both camera
zoom and position change when using the scroll wheel without an with the Alt
key, respectively.
- Connector restrictions (View tab) now offer a new option: show only connectors
with a specific tag. If selected, a dialog will ask for the desired tag and
will hide all connectors that aren't tagged with it.
- For both in-widget and exported animations a duration time in seconds can now
be specified. When enabled and an animation is played in the 3D Viewer, the
duration will just stop rotation and any other ongoing animation after the
specified time in seconds. If enabled for an animation export, it determines
the total length of the video, regardless of history or rotation settings.
- The history export in the animation export doesn't use a frame number input
field anymore and the "Complete history" checkbox has been removed, too.
Instead, the duration input can be used. If the duration input isn't enabled,
the complete history is exported.
- Both in-widget and exported animations don't need to have rotation anymore. A
new checkbox can be used to disabled rotation. Without rotation the camera
will just keep its position at the moment.
- Zoom behavior of the camera can be animated both for in-widget and export
animations. When enabled, a speed of nm/sec can be defined for the camera with
which it will move towards its target. This setting is both available in the
animation tab and the export dialog.
- A scaled version of the Strahler analysis shading mode has been added. The
scaling factor can be adjusted in the Shading parameters tab. This is mainly
useful for visualization.
- The PNG export dialog allows now to select whether the background should be
transparent.
- The "Focus skeleton(s)" button in the Main tab will now focus the center of
mass of all skeletons loaded in 3D, if no skeleton is active or the active
skeleton is not loaded in the 3D Viewer.
- The current view is now part of the saved state/settings. This is used as
initial view when used in a URL or when saved through the 'Save settings'
button in the panel opened by the window icon in the widget title bar.
- The "Shading" tab features now an input text field that allows to change the
location of the hemispheric light source. The checkbox "Lock light to camera"
next the location input allows to automatically set the light location to the
camera location, causing the light to come always from the viewing direction.
- The export "Connectors as CSV" 3D Viewer button now also exports the spatial
coordinates of each connector. The coordinates are represented with three
additional columns.
Skeleton history widget:
- Skeletons to query can now also be appened through the common skeleton source
controls.
- Now past skeletons with different skeleton IDs than the query IDs are
displayed as well as part of the graph.
- Edges show now more information in their labels: the number of nodes added as
well as the number of nodes deleted from the transition of nodes from skeleton
A to skeleton B.
- By default skeleton refreshs keep all nodes in-place. To try a different
version of the current layout, use the "Relayout" button.
Neuron navigator:
- Neuron widget nodes (like the active neuron display) show now also the initial
creator of the neuron and the initial creation time.
- The textual representation of neurons now follows the global settings.
- The neuron list filter input now remains focused when typing. The neuron
table should also update quicker initially.
- The pagination in neuron lists is fixed and does work on the
first attempt now. Previously it took a second click to actually jump to a
page.
- Sorting annotation lists works again for all columns.
- The annotation filter/search term survives now refreshes of the currently
displayed data (e.g. by clicking on the name in the navigator path).
- The select-all checkboxes in neuron lists now actually select all neurons
across all pages. This allows for bulk annotation, deannotation and deletion.
Morphology plot:
- Radial density can now be shown also for desmosomes and gap junctions.
Treenode table:
- The new node type "Adjacent to gap" will show all nodes that are in a Z slice
marked as broken or adjacent to it. Such nodes are marked with an additional
"G" in the node type flags.
Docker:
- The new environment variable CM_SERVER_SETTINGS can be used to define any
settings.py based setting. A valid Python code sting is expected, \n can be
used for newlines.
- Asynchronous tasks can now also be run inside the Docker container. Celery and
RabbitMQ are run by default, but can also be run in separate containers (see
CM_CELERY_BROKER_URL, CM_CELERY_WORKER_CONCURRENCY and CM_RUN_CELERY
Docker environment variables).
- Maintenance tasks are now automatically executed inside the Docker container
if Celery is enabled. This happens around midnight UTC by default, but the
time zone can be adjusted using the CM_CELERY_TIMEZONE environment variable.
For instance, use "America/New_York' for US east coast time. This way,
maintenance tasks can be moved to whenever local midnight is.
- CORS header are now enabled by default and allow also authentication. This is
useful to allow other web services to access a Docker back-end directly. This
can be disabled by setting "-e CORS_OPEN=false" with e.g. "docker run".
Vagrant:
- We now support a Vagrant configuration for setting up a full replicable development environment with minimal effort.
- This is distinct from the Docker image, which is for creating an ephemeral production environment.
- See the relevant developer documentation page for more details.
Data sources:
- A new tile source has been added: Neuroglancer Precomputed data, with ID 14.
This image block source works very similar to the N5 tile source. At the
moment only non-compressed and non-sharded image volumes are supported. In
order for the voxel space coordinates to match between CATMAID and
Neuroglancer, if the Neuroglancer dataset defines a voxel offset, the
respective CATMAID stack needs to have its zoom-level zero voxel offset
defined in the stack meta data in the admin view, e.g. `{"voxelOffset":
[-3072, -3072, 0]}`.
- A second new tile source is available: CloudVolume based tiles, generated by
the back-end. While slow for many users, it is a convenient way of make
neuroglancer volumes available to CATMAID if they aren't supported by the
Neuroglancer Precomputed source. It works better for a small set of users and
smaller block sizes. The new tile source has ID 13.
Administration:
- User profiles can now include a "home view", which is a reference to the
preferred data view for a particular user. When a user signs-in, this data
view is displayed. If none is set, the default data view is used.
- External accounts can now be used to sign-in to a CATMAID instance (if
enabled) using OAuth2. The documentation has details on how to configure this.
This can allow users with e.g. an existing Orcid ID or GitHub account to
sign-in to a CATMAID instance. If enabled, there is now a menu displayed when
hovering the mouse over the "Login" button in the upper right corner.
- Like every other new user, users created this way can be in default user
groups (`NEW_USER_DEFAULT_GROUPS` setting) and the default tool visibility
settings apply as well.
- If new users are created, there is now also a user group created with the same
name as the new user's username. This group can be used to manage edit permissions
the user's data. If groups shouldn't be created automatically, set the
`NEW_USER_CREATE_USER_GROUP` to `False` in the settings.
- Project importer: YAML data can now also be pasted into a text box for
project import, just like JSON data.
- The basic scripts/tiles/tile Bash 2D tiling script now by default creates as
many zoom levels until the whole image fits on one tile (rather than two like
before). It also accepts now an additional optional argument with which the
number of zoom levels to generate can be specified. Moreover it also provides
an option to specify an output directory and knows the common default value
for the overview size (192px).
- System check widget: more information regarding a potential transaction ID
wraparound has been added to the database section.
- User account creation now supports confirmation emails and welcome emails. The
User Accounts section of the manual has more details on this.
- Superusers will now see an extra menu item in their user menu (upper right
corner user name): Instance configuration. It opens a dialog which currently
presents two options: whether to show local and external login controls.
Tracing data export:
- The management command catmaid_export_data supports now the specification of
export options for tags, annotations and connectors per publication. To tell
the exporter which annotations/publications are allowed to carry these
settings, the "--settings-meta-annotation <annotation>" can be provided to the
exporter. For instance, assuming the provided <annotation> is "Published",
then all annotations that are annotated with "Published" are checked for
additional annotations that specify explicitly whether to export tags,
annotations and connectivity. The following annotations are respected:
"export: no-connectors", "export: no-tags", "export: annotations",
"export: no-annotations", "export: no-connectors",
"export: intra-connectors-only", "export: intra-connectors-and-placeholders",
and "export: intra-connectors-and-original-placeholders".
- The --conector-placeholders and the --original-placeholder-context options
have been removed. Instead the --connector option can now have four different
options: false, intra_connectors_only, intra_connectors_and_placeholders,
intra_connectors_and_original_placeholders. This provides the default
configuration if no publication specific options are provided.
- With the help of the new --allowed-tags-only parameter it is possible to limit
the kinds of tags that are exported. Without this parameter, all tags are
exported. If --allowed-tags-only is provided without arguments, a standard set
of known tags is included (uncertain end, posterior end, ends, anterior end,
not a branch, really ends, uncertain continuation, soma, microtubules end, out
to nerve). Alternatively, such a list can be provided as argument to this
parameter.
- In case all (minus exclusions) skeletons should be exported, now also all
annotations are exported if requested by the user. This wasn't the case so far
and in this mode no annotations were exported before.
Tracing data import:
- The catmaid_import_data management command will now truly update only project
wide data if --update-project-materializations is provided (as opposed to
selective updates for the imported data). Before, the summary information was
still computed instance wide.
- The new option --update-instance-materializations can now be used to update
the summary table information instance wide.
Import/export widget:
- A new 'Import log' tab will show users with import permission a list of their
imports. Double clicking entries moves the few to a representative location.
- So far, if no name was provided in the UI, the imported neuron would just have
no name. Now the user is asked whether the import should be canceled or if the
file name should be used as default name, if no name was provided by the user.
- If multiple SWC files are imported and a name is provided, an incrementing
numeric suffic will be added to each name.
- After one or more SWC files have been imported, there is now a list of links
displayed at the bottom of the widget. Each link is a successful import and,
when clicked, selects and moves to the respective skeleton.
- The NetowrkX JSON graph export is now in networkx 2.x format, rather than in
v1.x format like before. The main difference is that start and target nodes of
an edge are now defined explicitly by ID rather than an index.
Project statistics widget:
- Subsections will now show more clearly when they are still fetching data.
- A new data overview section has been added to the bottom of the content. It
shows currently the total cable length of skeletons in the project along with
the total number of treenodes and connectors.
Selection table:
- A minim review percentage can now be used as a filter using the new numeric
input in the name column header table cell.
- Duplicate skeleton IDs are now ignored in a CSV file import.
TrakEM2 import managmenet command:
- The new catmaid_import_from_trakem2 management command can be used to import
data from a TrakEM2 XML file in a similar fashion as the
scripts/export/export_from_trakem2.py script. It can however import in
addition TrakEM2 AreaList objects and connectivity.
- It generates meshes and skeletons by first generating a marching cubes based
mesh from the AreaList (using TrakEM2 functionality), then using Blender to
remesh that mesh to get a watertight manifold. This mesh is then first
imported as a triangle volume into CATMAID and then it is skeletonized. The
skeletonization can be done in different ways, by using skeletor or stl2swc.
The latter seems to generate better skeletons at the moment, but might require
a bit more setup time.
TrakEM2 based node transformation management command:
- The new catmaid_update_tracing_data_using_trakem2_xml management command can
be used to transform all spatial data in a project based on the
transformations defined in a TrakEM2 XML file. This includes
- This command uses Java and Fiji directly using pyjnius, which needs to be set
up separately. The management command will expect the PYJNIUS_JAR environment
variable to point to it (or provided using the --pyjnius parameter).
Data views:
- Add the options sample_image, sample_mirror and sanmple_slice to the front-end
based "Simple project list" data view. These options behave like in the other
data views.
- If more than one data view is available, users can now select a home view by
clicking the home icon in the 'Home' menu in the upper left corner. Upon next
reload of the page, this makes the respective data view the default one the
users sees.
- The icon to create a link to a data view changed to a more common chain icon.
- A click on the Home button in the upper left corner will now bring the user to
their home view (if defined), otherwise the global default data view.
- The data view menu (Home button) will now highlight the currently active data
view.
- A new front-end based data view type is available: resources and spaces. It
organizes projects in resources (read-only) and spaces (write) and indicates
this with different colors.
- The front-end data views now support the option "with_controls" (false by
default). If enabled favorite projects are indicated with a little star, which
also allows to toggle the favorite state by clicking on it. Only for admins
visible, right next to it, there is a pencil button to open a dialog to edit
the project title and description. If a project token was used to make a
project visible, the controls will also show an option to revoke a previously
used token. This will undo all permissions added by the token.
Cropping tool:
- A new radio button in front of both the top and the bottom Z index sliders
allows users now to choose whether the link the displayed Z slice to either
the top or the bottom of the copping box. This makes it easier to visually
confirm top and bottom of the bounding box.
- The ARTIST TIFF tag is now used to store meta information about the cropping
task: the min and max coordinates of the bounding box as well as the
resolution. ImageJ/Fiji read this particular tag.
Landmark widget:
- Landmarks can now be exported from the widget as CSVs
- Skeletons in the list of displayed transformations are now shown using their
Neuron Name Service based name, rather than only a skeleton ID. This works for
local and remote skeletons.
Tracing layer:
- Shift + Delete will now attempt to delete a node even if it is not in view.
This is mainly useful for deleting nodes in previously visible broken
sections.
- It's now possible to add the personal tag set to newly created nodes. Both the
personal tag set and the new option can be configured in the Tracing section
of the Settings Widget.
- Fallback to WebGL 1, if WebGL 2 is not available or can't be initialized
properly. In these cases image block rendering (e.g. N5) will be unavailable.
(Applies to image layer as well)
Tracing tool:
- Node distance measurements (icon in tracing tool bar) are now allowed also
between nodes of different skeletons. In this case only the straight line
distance will be displayed. Both the straight line distance and the path
distance will be displayed for nodes of the same skeleton.
- Centering on the active node reads now the active node only once all present
tasks (like node creation) are done, which makes centering on newly created
nodes using 'A' center reliably on the new node rather than the parent.
- The position and ID input box on the right side of the second tool bar accepts
now also location triplets that are delimited by space or tabs.
Miscellaneous:
- The dialog to add new annotations (F3 key) accepts now multiple annotations as
input, separated by commas. To include a literal coma, use baslash to escape
it: \,
- Regular view links (URL to this view): virtual nodes can now be referenced.
Only the parent and child ID are looked at when selecting a virtual node,
which means URLs keep working if parent or child location of a virtual node
change.
- The initial loading of client settings is now faster, because it performs
fewer requests.
- Each stack viewer will now show the project name by default along with the
stack and mirror name in its title bar. This can be adjusted in the "Stack
view" settings of the Settings Manager.
- Layer settings like opacity, blending more or filters can now be stored as
part of the regular settings and can be defined for the regular scopes like in
the Settings Widget (e.g. session or project). These settings are stored per
layer type and source reference (e.g. stack ID). In order to save, restore and
clear those settings, there are now three new buttons at the bottom of each
layer that supports settings storage.
- The minimum zoom level can now be configured in the Settings Widget using the
"Min zoom level" control of the "Stack view" section.
- Stack viewers can now have an optional layer offset configured. Accessible
through the Layer Controls, it allows users to configure one or more Z steps
from a stack space Z coordinate z1 to z2, that will cause an offset of the
current location in XY if the move from z1 to z2 is performed by the user
(e.g. by pressing comma or period). This can optionally be constrained by a
location and a radius. Generally, this helps working with registration
problems e.g. at TEM gaps.
- Client-side widget settings aren't stored by default anymore, if the widget is
closed. The former default caused a lot of confusion, especially with the 3D
Viewer. If widget settings should be stored, they have to be stored explicitly
using the "Store settings" button in the widget settings panel, available by
clicking the window icon right next to the widget title.
- Client-side widget state can now optionally store UI interaction information
like the camera location / details in the 3D Viewer. In the widget settings
panel (window icon in widget title bar), this can be done through the "Save
settings & UI state" if a widget supports this. Deep links will include the UI
state by default.
- Auto-completion should now be faster by displaying only 15 filtering results
by default. Clicking on the "..." entry will expand the auto-completion list
to its full length.
- Volume widget: box volumes can now be created also based on the active
skeleton's bounding box.
- Connector table: a search filter is now available for the table that allows
filtering across all columns.
- Global Search (looking glass icon): Neuron names are now properly displayed
using the general naming service. This makes the naming respect global naming
patterns.
- Connector selection: a search filter is now available for the table that allows
filtering across all columns.
- Settings widget: adding a remote CATMAID instance without supplying an API key
will now make CATMAID try and get the API key of the remote anonymous user.
This makes it easy to add publicly accessible CATMAID instances, because it is
now enough to provide the URL of the remote CATMAID instance. Additionally,
hash characters (#) at the end of passed in URLs are now removed
automatically.
- The user registration form fits now the overall layout better and include first
and last name as well as the email address. Also, a link to the user
registration form is now displayed right next to the login link in the top
right corner, if user registration is enabled. It also allows to optionally
require users to accept a certain set of terms and conditions, which can be
configured through the settings variables USER_REGISTRATION_CONFIRM_TERMS and
USER_REGISTRATION_CONFIRM_TERMS_TEXT (disabled by default).
- The overview image in the lower right corner of the stack viewer is now
displayed by default. This behavior can be adjusted for users, projects and
the whole instance in the Settings Widget.
- Open window: a new button "Open with active skeleton" makes it quicker to open
a new widget with the active skeleton loaded.
- Log/history tables: the relevant data is now fetched in parallel and doesn't
block the rest of CATMAID anymore.
- Layer settings: the color transform filter matrix input elements use now
background colors to better indicate the meaning of rows and columns.
- Key shortcut widget: the content is now refreshed when the selected tool
changes.
- If Caps-Lock is enabled, users now see a warning by default if they press down
any key. This can be disabled in the Settings Widget.
- Client settings that are stored in the browser's local storage are now
properly namespaced so that they work with multiple instances on the same
server.
- The new settings.py setting `PROJECT_TOKEN_USER_VISIBILITY` allows to
constrain the users that are visible to non-superuser users. If enabled, only
users that share knowledge about the same project tokens can be seen.
- Export management command: the new option --public-deep-links allows to export
public deep links of the selected projects.
Bug fixes
---------
- Cropping tool: if a single pixel past the start of a new image tile
represented part of the outer boundary of a cropping area, this part was
rendered only as black pixels. This is fixed now.
- Cropping tool: fix error on tool destruction when clicking on Home link
- Graph widget: fraction edge labels work again. They produced an error before.
- Graph widget: edge color updates of non-synaptic links works now properly.
- Graph widget: fix node width/height setting.
- 3D viewer: PNG exports can now be transparent again.
- 3D viewer: the default settings for orthographic mode and control lock are no
correct (i.e. in accordance with what has been saved as default).
- 3D viewer: the scale bar is now properly initialized if orthographic mode is
stored as default view.
- 3D viewer: fix camera movement using the mouse wheel.
- 3D viewer: the volume list is now also updated if a volume is deleted.
- 3D viewer: the landmark group list is now updated when landmark groups are
added or deleted.
- 3D viewer: the volume visibility controls are now correctly enabled if a
volume is made visible from another widget.
- 3D viewer: prevent accidental double loading of volumes, if the first attempt
didn't finish before the second.
- 3D viewer: fix saving a loaded volume twice in a saved state or URL.
- 3D viewer: mesh transparency is now handled in a more robust fashion.
- 3D viewer: landmark group event handlers are now correctly unregistered,
fixing an error in the Landmark Widget showing if a 3D Viewer was closed.
- 3D viewer: history animations can be created again.
- Volume lists (drop down menus) are now automatically refreshed if a volume is
added, updated or removed.
- Volume manager: the "Invert" checkbox for new filter rules for input skeletons
during volume creation is now position properly.
- Volume manager: the colors used for showing result volumes from an innervation
query in a new 3D Viewer are now computed correctly so that they match the
listing colors.
- Node filters: default values for extra options like merge operation,
skeleton/name filters or invert, are now correct on repeated filter creation
if these options were changed for the first filter.
- CSVs imported into the skeleton selection widget may contain nonexistent
skeleton IDs, as intended.
- Skeleton bulk updates like splits or joins should now be faster on setups with
spatial change events disabled (default).
- Connectivity widget: the rendering of large sets of input skeletons is now
much faster.
- Connectivity widget: showing a list of displayed links ("List links" button)
is working again.
- Connector list: shows now neuron names rather than skeleton IDs and can
optionally include them in exported CSVs as well.
- Neuron search: row highlighting for the active skeleton works now again for
the whole row and registers deselection properly.
- Neuron search: the active skeleton is now highlighted as soon as it part of
the result shown in the table. Before, the row of the active skeleton wouldn't
be highlighted after a search and only upon new selection of a skeleton.
- Tracing data importer: The catmaid_import_data management command was not
update edge data for project-wide materialization updates if there were no
connectors imported. This if fixed now.
- Tracing tool: a tracing data mirror server can now also be used with
authentication.
- Tracing tool: the "More tools" menu is now properly closed when a remote data
source is selected.
- Tracing tool: the mouse cursor icon is now initialized correctly.
- Remote tracing layers: mouse bindings are now correctly updated when a remote
layer is added. Before this could sometimes lead to mouse clicks not been
caught in the remote tracing layer.
- Skeleton import API: new annotations are now added to existing annotations by
default, rather than overriding them. This can be changed by setting the
replace_annotations parameter to true.
- Stack viewer: fix inconsistent state when removing copies of stack layers.
- Docker: when the container is stopped, all processes are now gracefully
stopped as well. I.e. the database will be notified about an imminent
shutdown, etc.
- Layouts: subscriptions, skeletons and options are now now properly loaded from
tab nodes in layout specs.
- Tracing data export: management command: connector tags are now exported
properly. Before in some cases inconsistent data was exported where either
connectors or tags where wrongly referenced even though they were not included
in the export.
- Split/merge dialog: both annotation lists (one for each partner) maintain now
a height of 50% in the dialog and don't cause enlarging the dialog with long
lists.
- Neuron navigator: require less neuron name updates when updating the active
skeleton display.
- Node filters: the "Pruned arbor" filter works again.
- Neuron search: row highlighting for the active skeleton works now again for
the whole row and registers deselection properly.
- Connector list: can now display connectors without links.
- Measurements table: "N presynaptic sites" column includes now by default also
presynaptic connectors that don't have any postsynaptic link. To restore the
previous behavior, the "Include half links/synapses" checkbox can be
unchecked.
- Navigator tool and sub-tools like Tracing tool: the Z slider is now properly
hidden if the image data set has only one section.
API Changes
-----------
Additions:
- POST `/{project_id}/neurons/rename`:
Rename multiple neurons at the same time.
- GET|POST `/{project_id}/skeletons/summary`:
Retrieve meta information like cable length or number of nodes for skeletons.
- GET `/accounts/anonymous-api-token`:
Get the API token for the anonymous user.
- POST `/{project_id}/neurons/all-skeletons`:
Map a list of neuron IDs to the skeleton IDs that model them.
- GET `/{project_id}/`:
Get project details like the title and comment.
- POST `/{project_id}/`:
Update the project title if the user has admin permissions in the project.
Modifications:
- `GET /{project_id}/transactions/`:
Now accepts the optional parameters user_id and type to further constrain the
result. Type is a transaction label string, e.g. 'skeletons.import'.
- `POST /{project_id}/skeletons/import`:
Now accepts the optional boolean parameter replace_annotations, which can be
used to remove all existing annotations if an existing neuron/skeleton is
replaced with force=true.
- `POST /{project_id}/graphexport/json`:
The returned graph export is now in networkx 2.x format, rather than in v1.x
format like before. The main difference is that start and target nodes of an
edge are now defined explicitly by ID rather than an index.
- `POST /{project_id}/skeleton/connectivity_matrix`:
The new parameters row_relations and col_relations allow callers now to
specify the relations used to connect two skeletons for each side. They
default to "presynaptic_to" and "postsynaptic_to".
- `POST /{project_ids}/skeletons/in-bounding-box`:
Returns now also unlinked connectors by default. To only get linked connectors
like before, pass in `only_linked = true`.
Deprecations:
None.
Removals:
None.
We just released a new CATMAID version, 2021.12.21
https://github.com/catmaid/CATMAID/releases/tag/2021.12.21
You will find a list of changes as we well as some update notes at the
end of this email and on the website linked above. As always, please let
us know if you run into any problems, both as an admin or a user. The
docker images tagged stable and 2021.12.21 have been (re)built.
If you update a CATMAID instance, please use the most recent commit of
the master branch in our GitHub repository, which includes all future
bug fixes for this release:
https://github.com/catmaid/CATMAID
If you are administering a CATMAID server, the "Notes" section provides
information on how to migrate to the new version. The file UPDATE.md
contains a copy of the migration steps as well. Please note that this
CATMAID version is the last with Python 3.6 support. This release
requires also Postgres 12+ and PostGIS 2.5+. The CATMAID version after
this one will require Postgres 13+ and PostGIS 3.2+, both of which are
supported already.
If you have to update your database, please make sure to update to the
target version of PostGIS in all databases *before* you update the
database cluster, it will make the update go more smoothly. This means
installing the new PostGIS version (e.g. v3), connecting to each
database and running `ALTER EXTENSION postgis UPDATE` (check version
using \dx). The same database version needs to be available for the
target version of Postgres. If you need to update your database, go to
Postgres 13 directly, because the next version of CATMAID will require
Postgres 13 and PostGIS 3.
Cheers,
Tom
2020.12.21
==========
Contributors: Chris Barnes, Albert Cardona, Andrew Champion, Stephan
Gerhard, Sanja Jasek, Tom Kazimiers
Notes
-----
- The version requires PostgreSQL 12. If you also want to upgrade PostGIS,
update PostGIS first and run ``ALTER EXTENSION postgis UPDATE;`` in every
existing database in the cluster that should be upgraded. For docker-compose
setups this database update is performed automatically if `DB_UPDATE=true` is
set for the `db` container (watch the Docker output). CATMAID's documentation
Docker has more information. If a replication setup is in use, the database
configuration changes for Postgres 12. CATMAID's replication documentation
explains what needs to be done.
In Postgres 12, JIT compilation is available to the planner by default. We
found that being more conservative with the use of it helped a few common
queries (like the tracing data field of view). We set the required minimum
cost of JIT use to 1,000,000 in ``postgresql.conf``::
jit_above_cost = 1000000
- If R extensions are used, make sure to use R 3.6. On Ubuntu this can be made
available by first installing the official R PPA repository:
sudo gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-key E084DAB9
sudo gpg -a --export E084DAB9 | sudo apt-key add -
echo "deb https://cloud.r-project.org/bin/linux/ubuntu xenial-cran35/" | sudo tee -a /etc/apt/sources.list
And second update the package index and update the local R setup:
sudo apt-get update
sudo apt-get install r-base r-base-dev mesa-common-dev libglu1-mesa-dev \
libssl-dev libssh2-1-dev libcurl4-openssl-dev cmtk
This also requires updating all installed R packages. In all likelihood this
requires executing "manage.py catmaid_setup_nblast_environment".
- The default downsample factors for a stack without explicit factors defined,
changed slightly: if you relied on the automatic creation of downsample
factors for image stacks, make sure the changed behavior still works for you
or enforce the previous configuration with an explicit specification of
downsample factors. Previously as many zoom levels as needed were created to
map the image size in a quadratic fashion per level onto a 1024x1024px tile.
Instead of hard-coding the 1024px, the minimum tile size defined in the mirror
set is used. Without any mirrors, no extra zoom levels will be assumed.
- The recommended python version is now 3.8, although support for 3.6 is maintained.
- A virtualenv update is required.
Features and enhancements
-------------------------
Notes widget:
- This new and simple widget can be used to take notes either only for the
current project, shared across all projects or globally visible for all users.
The last option is only available to admin users for writing. Use the "Open
Widget" dialog to open it using the "Notes" keyword.
Publication widget:
- This new widget makes management of publication related annotations easier. It
lists all annotations that annotated with one of the publication annotations
defined in the Settings Widget in the Publication section. Each of them
represents a publication for the widget. The export options for each one of
them can now be configured through the table.
Project management widget:
- This new widget allows superusers and users with can_administer role in a
project to edit user and group permissions for this project. This interface is
easier to find and manage then the admin view and allows for more granular
admin permissions. It is also useful to share own spaces (own copies of
projects) with other users.
- The widget allows to define for both user and groups the following
permissions: can browse (read), can annotate (write), can administer, can
import, can queue, compute taks, can write through API, can fork
- Clicking any checkbox will store the value right away.
- If a user has "can_administer" permissions in a project, they can edit the
project title and project description (comment) in the second tab.
- If a user has "can_administer" and "delete_project" permissions in a project,
the project management widget allows to delete the current project. This is
mainly useful for personal spaces, in which users typically have
delete_project permissions.
- The "User data permissions" tab allows to configure user level edit
permissions (who is allowed to edit who's data).
- The "Project token" tab allows project admins to view existing project tokens
("invitation codes") available for a project. For each generated token, the
table allows to copy both the token itself as well as an invitation link using
this token to the clipboard. If a user opens an invitation link, they will
automatically apply the token. Anonymous users are asked to log in first.
- As admin user in a project, it is also possible from to manually change
skeleton IDs from the "ID updates" tab. This is only allowed if the new ID
isn't used already. Also, if the new ID is a new maximum ID number, all new
IDs will be automatically be larger than this ID. This functionality is also
available as the back-end managment command catmaid_change_skeleton_id.
Skeleton ID change management command:
- Just like the Project Management functionality explained in the last
paragraph, the management command catmaid_change_skeleton_id can be used to
change skeleton IDs (and optionally neuron IDs) from the command line.
Link widget and deep links:
- This new widget allows users to create persistent deep links into the
dataset using an alias, allowing for friendlier URLs of the form
`<catmaid-url>/<project-id>/links/<alias>`. The stored parameters are the same
as for regular deep links, with most of them being optional.
- Like with fully parameterized URLs, features like layout, widget settings and
loaded skeletons can be used.
- Links can be marked private, making them only accessible (and listable) by the
creator of the link.
- An optional message can be stored with the link and is shown to the user when
the links is opened.
- The context menu for creating different types of links moved from entries
in the Layouts menu to its own menu that is shown when hovering the mouse
cursor over the "URL to this view" link. The first menu entry is new and opens
a link creation dialog, similar to the Link Widget's 'Add link' dialog.
Neuron renaming:
- There is now a new neuron renaming dialog, which allows a search/replace
renaming workflow for many neurons at the same time. This can be launched
through the "Rename" button in both the Neuron Search widget and the neuron
lists of the Neuron Navigator (e.g. neurons with a particular annotation).
- The dialog supports simple find/replace, but also regular expressions
including capture groups and presents live previews.
Neuron search:
- Neuron search: add option to show meta data columns (cable length, number of
nodes, creation time and last edition time) for neurons.
- Removal tools for annotations are now opt-in, i.e. not shown by default. To
enable them during annotation display, click the "Show removal tools"
checkbox.
- The optional annotation column has now the option to only show annotations
with a particular meta-annotation. This meta-annotation can be configured in
the input field in the table column header.
- The CSV export comes now with a header row to make clear the first column is a
neuron ID.
- The CSV export includes now the skeleton ID as well and will export metadata
like cable length, if metadata display is enabled in the UI.
- Expanded elements (e.g. annotations) are now sorted correctly within their
expansion group. This also fixes the out-of-order expansion display.
- Escape underscore and percent sign properly. They were treated as wildcards
before.
Connectivity Matrix:
- The new buttons "Save JSON" and "Open JSON" can be used to save the row and
column skeletons, including groups and widget settings.
- The loading speed has been improved significantly.
- With the new "Link type" selector in the Main tab it is now possible to show
the connectivity matrix for other link types (like desmosomes or gap
junctions). The selected link type is also respected by the CSV export.
Layouts:
- New layout menu item: "Copy URL to view with layout". This will create a URL
to the current view like the link button on the right hand side of the second
toolbar, but with a layout spec included. It will open all widgets in the new
CATMAID like in the current one. This includes skeletons, their colors and
widget configurations by default. A second menu entry with the "(no
skeletons)" suffix allows to create link with only the layout and no
skeletons. And a third menu entry with the suffix ("no widget settings)"
allows to copy only the layout plus skeletons, without any widget
configuration.
- New layout menu item: "Copy current layout spec". This will copy the layout
specification of the current view to the clipboard. This can be useful when
constructing URLs to specific views.
- Widgets in layouts can now optionally specify a list of skeletons to load.
This is done by adding the `skeletons` parameter:
"skeletons": [16035815, {"id": 16035701, "color": "rgb(0.5,1,0.2)"}]
The list of skeletons can consist of skeleton IDs and/or optionally objects
that can also specify a color in terms of common CSS definitions.
- The Selection Table can now receive an options parameter from the layout
initialization. If it contains a list of skeleton IDs like this, then
selection table will load this automatically:
options: {"skeleton-ids": [16035815, 16035701]}
This has to be added to the URL manually at the moment, by adding it to the
selection table config part in the layout spec.
Volume manager:
- The table of filters applied to input neurons for volume creation has now a
new "Actions" column. At the moment this includes a "Remove" link for each
filter. This makes it easier to try different filters without removing all
other filters
- Annotations can now also be removed from volumes if "Show removal tools" is
enabled.
- For new node filters during volume creation, the merge mode can now be
selected, i.e. whether additional filters will be combined using OR or AND.
- The new "Import from CATMAID" tab allows users with import permission to
search and import volumes/meshes from other CATMAID projects, both local and
remote. Volumes can be searched by name and annotation.
- Don't show removal options by default. It happens generally rarely that one
wants to remove volumes, especially in the skeleton innervation tab. To reduce
the risk of accidental removals (even though a confirmation dialog is shown),
removal buttons are now only shown if the "Show removal options" checkbox is
enabled in the Main tab.
- The connector listing for a volume will now include unlinked nodes.
- The checkbox to show the removal tools is now colored red, to make it easier
to see how to remove volumes.
Graph widget:
- Add options to configure the edge text outline size and opacity. The options
area available from the properties dialog.
- Grouping with undirected edges like desmosomes won't lead to multiple edges
between the same groups anymore.
- Ungrouping groups with members having non-synaptic links (e.g. desmosomes)
doesn't lead to an error anymore.
- Remember selected link types in local widget state storage.
- Applying node filters works now correctly and only removes an edge if both
source and target node are not allowed according to the filter configuration.
It is now also possible to use additional subgraphs (e.g. axon/dendrite
split) with filters applied.
Connectivity widget:
- Larger input and partner sets can now be displayed.
- Make node filter test consistent with filter application in the Graph Widget.
A connection is now valid according to the selected filters if it uses a
connector node that is linked to a node allowed by the node filters. The node
filters are still only applied to the query skeletons. This makes the
filtering numbers consistent with the Graph Widget.
3D viewer:
- The new tab "Volumes & Geometry" contains now all settings related to volumes,
landmarks and point clouds. This makes the "Volume settings" tab a little
less overloaded.
- The View tab has two new controls to control the sensitivity of both camera
zoom and position change when using the scroll wheel without an with the Alt
key, respectively.
- Connector restrictions (View tab) now offer a new option: show only connectors
with a specific tag. If selected, a dialog will ask for the desired tag and
will hide all connectors that aren't tagged with it.
- For both in-widget and exported animations a duration time in seconds can now
be specified. When enabled and an animation is played in the 3D Viewer, the
duration will just stop rotation and any other ongoing animation after the
specified time in seconds. If enabled for an animation export, it determines
the total length of the video, regardless of history or rotation settings.
- The history export in the animation export doesn't use a frame number input
field anymore and the "Complete history" checkbox has been removed, too.
Instead, the duration input can be used. If the duration input isn't enabled,
the complete history is exported.
- Both in-widget and exported animations don't need to have rotation anymore. A
new checkbox can be used to disabled rotation. Without rotation the camera
will just keep its position at the moment.
- Zoom behavior of the camera can be animated both for in-widget and export
animations. When enabled, a speed of nm/sec can be defined for the camera with
which it will move towards its target. This setting is both available in the
animation tab and the export dialog.
- A scaled version of the Strahler analysis shading mode has been added. The
scaling factor can be adjusted in the Shading parameters tab. This is mainly
useful for visualization.
- The PNG export dialog allows now to select whether the background should be
transparent.
- The "Focus skeleton(s)" button in the Main tab will now focus the center of
mass of all skeletons loaded in 3D, if no skeleton is active or the active
skeleton is not loaded in the 3D Viewer.
- The current view is now part of the saved state/settings. This is used as
initial view when used in a URL or when saved through the 'Save settings'
button in the panel opened by the window icon in the widget title bar.
- The "Shading" tab features now an input text field that allows to change the
location of the hemispheric light source. The checkbox "Lock light to camera"
next the location input allows to automatically set the light location to the
camera location, causing the light to come always from the viewing direction.
- The export "Connectors as CSV" 3D Viewer button now also exports the spatial
coordinates of each connector. The coordinates are represented with three
additional columns.
Skeleton history widget:
- Skeletons to query can now also be appened through the common skeleton source
controls.
- Now past skeletons with different skeleton IDs than the query IDs are
displayed as well as part of the graph.
- Edges show now more information in their labels: the number of nodes added as
well as the number of nodes deleted from the transition of nodes from skeleton
A to skeleton B.
- By default skeleton refreshs keep all nodes in-place. To try a different
version of the current layout, use the "Relayout" button.
Neuron navigator:
- Neuron widget nodes (like the active neuron display) show now also the initial
creator of the neuron and the initial creation time.
- The textual representation of neurons now follows the global settings.
- The neuron list filter input now remains focused when typing. The neuron
table should also update quicker initially.
- The pagination in neuron lists is fixed and does work on the
first attempt now. Previously it took a second click to actually jump to a
page.
- Sorting annotation lists works again for all columns.
- The annotation filter/search term survives now refreshes of the currently
displayed data (e.g. by clicking on the name in the navigator path).
- The select-all checkboxes in neuron lists now actually select all neurons
across all pages. This allows for bulk annotation, deannotation and deletion.
Morphology plot:
- Radial density can now be shown also for desmosomes and gap junctions.
Treenode table:
- The new node type "Adjacent to gap" will show all nodes that are in a Z slice
marked as broken or adjacent to it. Such nodes are marked with an additional
"G" in the node type flags.
Docker:
- The new environment variable CM_SERVER_SETTINGS can be used to define any
settings.py based setting. A valid Python code sting is expected, \n can be
used for newlines.
- Asynchronous tasks can now also be run inside the Docker container. Celery and
RabbitMQ are run by default, but can also be run in separate containers (see
CM_CELERY_BROKER_URL, CM_CELERY_WORKER_CONCURRENCY and CM_RUN_CELERY
Docker environment variables).
- Maintenance tasks are now automatically executed inside the Docker container
if Celery is enabled. This happens around midnight UTC by default, but the
time zone can be adjusted using the CM_CELERY_TIMEZONE environment variable.
For instance, use "America/New_York' for US east coast time. This way,
maintenance tasks can be moved to whenever local midnight is.
- CORS header are now enabled by default and allow also authentication. This is
useful to allow other web services to access a Docker back-end directly. This
can be disabled by setting "-e CORS_OPEN=false" with e.g. "docker run".
Vagrant:
- We now support a Vagrant configuration for setting up a full replicable development environment with minimal effort.
- This is distinct from the Docker image, which is for creating an ephemeral production environment.
- See the relevant developer documentation page for more details.
Data sources:
- A new tile source has been added: Neuroglancer Precomputed data, with ID 14.
This image block source works very similar to the N5 tile source. At the
moment only non-compressed and non-sharded image volumes are supported. In
order for the voxel space coordinates to match between CATMAID and
Neuroglancer, if the Neuroglancer dataset defines a voxel offset, the
respective CATMAID stack needs to have its zoom-level zero voxel offset
defined in the stack meta data in the admin view, e.g. `{"voxelOffset":
[-3072, -3072, 0]}`.
- A second new tile source is available: CloudVolume based tiles, generated by
the back-end. While slow for many users, it is a convenient way of make
neuroglancer volumes available to CATMAID if they aren't supported by the
Neuroglancer Precomputed source. It works better for a small set of users and
smaller block sizes. The new tile source has ID 13.
Administration:
- User profiles can now include a "home view", which is a reference to the
preferred data view for a particular user. When a user signs-in, this data
view is displayed. If none is set, the default data view is used.
- External accounts can now be used to sign-in to a CATMAID instance (if
enabled) using OAuth2. The documentation has details on how to configure this.
This can allow users with e.g. an existing Orcid ID or GitHub account to
sign-in to a CATMAID instance. If enabled, there is now a menu displayed when
hovering the mouse over the "Login" button in the upper right corner.
- Like every other new user, users created this way can be in default user
groups (`NEW_USER_DEFAULT_GROUPS` setting) and the default tool visibility
settings apply as well.
- If new users are created, there is now also a user group created with the same
name as the new user's username. This group can be used to manage edit permissions
the user's data. If groups shouldn't be created automatically, set the
`NEW_USER_CREATE_USER_GROUP` to `False` in the settings.
- Project importer: YAML data can now also be pasted into a text box for
project import, just like JSON data.
- The basic scripts/tiles/tile Bash 2D tiling script now by default creates as
many zoom levels until the whole image fits on one tile (rather than two like
before). It also accepts now an additional optional argument with which the
number of zoom levels to generate can be specified. Moreover it also provides
an option to specify an output directory and knows the common default value
for the overview size (192px).
- System check widget: more information regarding a potential transaction ID
wraparound has been added to the database section.
- User account creation now supports confirmation emails and welcome emails. The
User Accounts section of the manual has more details on this.
- Superusers will now see an extra menu item in their user menu (upper right
corner user name): Instance configuration. It opens a dialog which currently
presents two options: whether to show local and external login controls.
Tracing data export:
- The management command catmaid_export_data supports now the specification of
export options for tags, annotations and connectors per publication. To tell
the exporter which annotations/publications are allowed to carry these
settings, the "--settings-meta-annotation <annotation>" can be provided to the
exporter. For instance, assuming the provided <annotation> is "Published",
then all annotations that are annotated with "Published" are checked for
additional annotations that specify explicitly whether to export tags,
annotations and connectivity. The following annotations are respected:
"export: no-connectors", "export: no-tags", "export: annotations",
"export: no-annotations", "export: no-connectors",
"export: intra-connectors-only", "export: intra-connectors-and-placeholders",
and "export: intra-connectors-and-original-placeholders".
- The --conector-placeholders and the --original-placeholder-context options
have been removed. Instead the --connector option can now have four different
options: false, intra_connectors_only, intra_connectors_and_placeholders,
intra_connectors_and_original_placeholders. This provides the default
configuration if no publication specific options are provided.
- With the help of the new --allowed-tags-only parameter it is possible to limit
the kinds of tags that are exported. Without this parameter, all tags are
exported. If --allowed-tags-only is provided without arguments, a standard set
of known tags is included (uncertain end, posterior end, ends, anterior end,
not a branch, really ends, uncertain continuation, soma, microtubules end, out
to nerve). Alternatively, such a list can be provided as argument to this
parameter.
- In case all (minus exclusions) skeletons should be exported, now also all
annotations are exported if requested by the user. This wasn't the case so far
and in this mode no annotations were exported before.
Tracing data import:
- The catmaid_import_data management command will now truly update only project
wide data if --update-project-materializations is provided (as opposed to
selective updates for the imported data). Before, the summary information was
still computed instance wide.
- The new option --update-instance-materializations can now be used to update
the summary table information instance wide.
Import/export widget:
- A new 'Import log' tab will show users with import permission a list of their
imports. Double clicking entries moves the few to a representative location.
- So far, if no name was provided in the UI, the imported neuron would just have
no name. Now the user is asked whether the import should be canceled or if the
file name should be used as default name, if no name was provided by the user.
- If multiple SWC files are imported and a name is provided, an incrementing
numeric suffic will be added to each name.
- After one or more SWC files have been imported, there is now a list of links
displayed at the bottom of the widget. Each link is a successful import and,
when clicked, selects and moves to the respective skeleton.
- The NetowrkX JSON graph export is now in networkx 2.x format, rather than in
v1.x format like before. The main difference is that start and target nodes of
an edge are now defined explicitly by ID rather than an index.
Project statistics widget:
- Subsections will now show more clearly when they are still fetching data.
- A new data overview section has been added to the bottom of the content. It
shows currently the total cable length of skeletons in the project along with
the total number of treenodes and connectors.
Selection table:
- A minim review percentage can now be used as a filter using the new numeric
input in the name column header table cell.
- Duplicate skeleton IDs are now ignored in a CSV file import.
TrakEM2 import managmenet command:
- The new catmaid_import_from_trakem2 management command can be used to import
data from a TrakEM2 XML file in a similar fashion as the
scripts/export/export_from_trakem2.py script. It can however import in
addition TrakEM2 AreaList objects and connectivity.
- It generates meshes and skeletons by first generating a marching cubes based
mesh from the AreaList (using TrakEM2 functionality), then using Blender to
remesh that mesh to get a watertight manifold. This mesh is then first
imported as a triangle volume into CATMAID and then it is skeletonized. The
skeletonization can be done in different ways, by using skeletor or stl2swc.
The latter seems to generate better skeletons at the moment, but might require
a bit more setup time.
TrakEM2 based node transformation management command:
- The new catmaid_update_tracing_data_using_trakem2_xml management command can
be used to transform all spatial data in a project based on the
transformations defined in a TrakEM2 XML file. This includes
- This command uses Java and Fiji directly using pyjnius, which needs to be set
up separately. The management command will expect the PYJNIUS_JAR environment
variable to point to it (or provided using the --pyjnius parameter).
Data views:
- Add the options sample_image, sample_mirror and sanmple_slice to the front-end
based "Simple project list" data view. These options behave like in the other
data views.
- If more than one data view is available, users can now select a home view by
clicking the home icon in the 'Home' menu in the upper left corner. Upon next
reload of the page, this makes the respective data view the default one the
users sees.
- The icon to create a link to a data view changed to a more common chain icon.
- A click on the Home button in the upper left corner will now bring the user to
their home view (if defined), otherwise the global default data view.
- The data view menu (Home button) will now highlight the currently active data
view.
- A new front-end based data view type is available: resources and spaces. It
organizes projects in resources (read-only) and spaces (write) and indicates
this with different colors.
- The front-end data views now support the option "with_controls" (false by
default). If enabled favorite projects are indicated with a little star, which
also allows to toggle the favorite state by clicking on it. Only for admins
visible, right next to it, there is a pencil button to open a dialog to edit
the project title and description. If a project token was used to make a
project visible, the controls will also show an option to revoke a previously
used token. This will undo all permissions added by the token.
Cropping tool:
- A new radio button in front of both the top and the bottom Z index sliders
allows users now to choose whether the link the displayed Z slice to either
the top or the bottom of the copping box. This makes it easier to visually
confirm top and bottom of the bounding box.
- The ARTIST TIFF tag is now used to store meta information about the cropping
task: the min and max coordinates of the bounding box as well as the
resolution. ImageJ/Fiji read this particular tag.
Landmark widget:
- Landmarks can now be exported from the widget as CSVs
- Skeletons in the list of displayed transformations are now shown using their
Neuron Name Service based name, rather than only a skeleton ID. This works for
local and remote skeletons.
Tracing layer:
- Shift + Delete will now attempt to delete a node even if it is not in view.
This is mainly useful for deleting nodes in previously visible broken
sections.
- It's now possible to add the personal tag set to newly created nodes. Both the
personal tag set and the new option can be configured in the Tracing section
of the Settings Widget.
- Fallback to WebGL 1, if WebGL 2 is not available or can't be initialized
properly. In these cases image block rendering (e.g. N5) will be unavailable.
(Applies to image layer as well)
Tracing tool:
- Node distance measurements (icon in tracing tool bar) are now allowed also
between nodes of different skeletons. In this case only the straight line
distance will be displayed. Both the straight line distance and the path
distance will be displayed for nodes of the same skeleton.
- Centering on the active node reads now the active node only once all present
tasks (like node creation) are done, which makes centering on newly created
nodes using 'A' center reliably on the new node rather than the parent.
- The position and ID input box on the right side of the second tool bar accepts
now also location triplets that are delimited by space or tabs.
Miscellaneous:
- The dialog to add new annotations (F3 key) accepts now multiple annotations as
input, separated by commas. To include a literal coma, use baslash to escape
it: \,
- Regular view links (URL to this view): virtual nodes can now be referenced.
Only the parent and child ID are looked at when selecting a virtual node,
which means URLs keep working if parent or child location of a virtual node
change.
- The initial loading of client settings is now faster, because it performs
fewer requests.
- Each stack viewer will now show the project name by default along with the
stack and mirror name in its title bar. This can be adjusted in the "Stack
view" settings of the Settings Manager.
- Layer settings like opacity, blending more or filters can now be stored as
part of the regular settings and can be defined for the regular scopes like in
the Settings Widget (e.g. session or project). These settings are stored per
layer type and source reference (e.g. stack ID). In order to save, restore and
clear those settings, there are now three new buttons at the bottom of each
layer that supports settings storage.
- The minimum zoom level can now be configured in the Settings Widget using the
"Min zoom level" control of the "Stack view" section.
- Stack viewers can now have an optional layer offset configured. Accessible
through the Layer Controls, it allows users to configure one or more Z steps
from a stack space Z coordinate z1 to z2, that will cause an offset of the
current location in XY if the move from z1 to z2 is performed by the user
(e.g. by pressing comma or period). This can optionally be constrained by a
location and a radius. Generally, this helps working with registration
problems e.g. at TEM gaps.
- Client-side widget settings aren't stored by default anymore, if the widget is
closed. The former default caused a lot of confusion, especially with the 3D
Viewer. If widget settings should be stored, they have to be stored explicitly
using the "Store settings" button in the widget settings panel, available by
clicking the window icon right next to the widget title.
- Client-side widget state can now optionally store UI interaction information
like the camera location / details in the 3D Viewer. In the widget settings
panel (window icon in widget title bar), this can be done through the "Save
settings & UI state" if a widget supports this. Deep links will include the UI
state by default.
- Auto-completion should now be faster by displaying only 15 filtering results
by default. Clicking on the "..." entry will expand the auto-completion list
to its full length.
- Volume widget: box volumes can now be created also based on the active
skeleton's bounding box.
- Connector table: a search filter is now available for the table that allows
filtering across all columns.
- Global Search (looking glass icon): Neuron names are now properly displayed
using the general naming service. This makes the naming respect global naming
patterns.
- Connector selection: a search filter is now available for the table that allows
filtering across all columns.
- Settings widget: adding a remote CATMAID instance without supplying an API key
will now make CATMAID try and get the API key of the remote anonymous user.
This makes it easy to add publicly accessible CATMAID instances, because it is
now enough to provide the URL of the remote CATMAID instance. Additionally,
hash characters (#) at the end of passed in URLs are now removed
automatically.
- The user registration form fits now the overall layout better and include first
and last name as well as the email address. Also, a link to the user
registration form is now displayed right next to the login link in the top
right corner, if user registration is enabled. It also allows to optionally
require users to accept a certain set of terms and conditions, which can be
configured through the settings variables USER_REGISTRATION_CONFIRM_TERMS and
USER_REGISTRATION_CONFIRM_TERMS_TEXT (disabled by default).
- The overview image in the lower right corner of the stack viewer is now
displayed by default. This behavior can be adjusted for users, projects and
the whole instance in the Settings Widget.
- Open window: a new button "Open with active skeleton" makes it quicker to open
a new widget with the active skeleton loaded.
- Log/history tables: the relevant data is now fetched in parallel and doesn't
block the rest of CATMAID anymore.
- Layer settings: the color transform filter matrix input elements use now
background colors to better indicate the meaning of rows and columns.
- Key shortcut widget: the content is now refreshed when the selected tool
changes.
- If Caps-Lock is enabled, users now see a warning by default if they press down
any key. This can be disabled in the Settings Widget.
- Client settings that are stored in the browser's local storage are now
properly namespaced so that they work with multiple instances on the same
server.
- The new settings.py setting `PROJECT_TOKEN_USER_VISIBILITY` allows to
constrain the users that are visible to non-superuser users. If enabled, only
users that share knowledge about the same project tokens can be seen.
- Export management command: the new option --public-deep-links allows to export
public deep links of the selected projects.
Bug fixes
---------
- Cropping tool: if a single pixel past the start of a new image tile
represented part of the outer boundary of a cropping area, this part was
rendered only as black pixels. This is fixed now.
- Cropping tool: fix error on tool destruction when clicking on Home link
- Graph widget: fraction edge labels work again. They produced an error before.
- Graph widget: edge color updates of non-synaptic links works now properly.
- Graph widget: fix node width/height setting.
- 3D viewer: PNG exports can now be transparent again.
- 3D viewer: the default settings for orthographic mode and control lock are no
correct (i.e. in accordance with what has been saved as default).
- 3D viewer: the scale bar is now properly initialized if orthographic mode is
stored as default view.
- 3D viewer: fix camera movement using the mouse wheel.
- 3D viewer: the volume list is now also updated if a volume is deleted.
- 3D viewer: the landmark group list is now updated when landmark groups are
added or deleted.
- 3D viewer: the volume visibility controls are now correctly enabled if a
volume is made visible from another widget.
- 3D viewer: prevent accidental double loading of volumes, if the first attempt
didn't finish before the second.
- 3D viewer: fix saving a loaded volume twice in a saved state or URL.
- 3D viewer: mesh transparency is now handled in a more robust fashion.
- 3D viewer: landmark group event handlers are now correctly unregistered,
fixing an error in the Landmark Widget showing if a 3D Viewer was closed.
- 3D viewer: history animations can be created again.
- Volume lists (drop down menus) are now automatically refreshed if a volume is
added, updated or removed.
- Volume manager: the "Invert" checkbox for new filter rules for input skeletons
during volume creation is now position properly.
- Volume manager: the colors used for showing result volumes from an innervation
query in a new 3D Viewer are now computed correctly so that they match the
listing colors.
- Node filters: default values for extra options like merge operation,
skeleton/name filters or invert, are now correct on repeated filter creation
if these options were changed for the first filter.
- CSVs imported into the skeleton selection widget may contain nonexistent
skeleton IDs, as intended.
- Skeleton bulk updates like splits or joins should now be faster on setups with
spatial change events disabled (default).
- Connectivity widget: the rendering of large sets of input skeletons is now
much faster.
- Connectivity widget: showing a list of displayed links ("List links" button)
is working again.
- Connector list: shows now neuron names rather than skeleton IDs and can
optionally include them in exported CSVs as well.
- Neuron search: row highlighting for the active skeleton works now again for
the whole row and registers deselection properly.
- Neuron search: the active skeleton is now highlighted as soon as it part of
the result shown in the table. Before, the row of the active skeleton wouldn't
be highlighted after a search and only upon new selection of a skeleton.
- Tracing data importer: The catmaid_import_data management command was not
update edge data for project-wide materialization updates if there were no
connectors imported. This if fixed now.
- Tracing tool: a tracing data mirror server can now also be used with
authentication.
- Tracing tool: the "More tools" menu is now properly closed when a remote data
source is selected.
- Tracing tool: the mouse cursor icon is now initialized correctly.
- Remote tracing layers: mouse bindings are now correctly updated when a remote
layer is added. Before this could sometimes lead to mouse clicks not been
caught in the remote tracing layer.
- Skeleton import API: new annotations are now added to existing annotations by
default, rather than overriding them. This can be changed by setting the
replace_annotations parameter to true.
- Stack viewer: fix inconsistent state when removing copies of stack layers.
- Docker: when the container is stopped, all processes are now gracefully
stopped as well. I.e. the database will be notified about an imminent
shutdown, etc.
- Layouts: subscriptions, skeletons and options are now now properly loaded from
tab nodes in layout specs.
- Tracing data export: management command: connector tags are now exported
properly. Before in some cases inconsistent data was exported where either
connectors or tags where wrongly referenced even though they were not included
in the export.
- Split/merge dialog: both annotation lists (one for each partner) maintain now
a height of 50% in the dialog and don't cause enlarging the dialog with long
lists.
- Neuron navigator: require less neuron name updates when updating the active
skeleton display.
- Node filters: the "Pruned arbor" filter works again.
- Neuron search: row highlighting for the active skeleton works now again for
the whole row and registers deselection properly.
- Connector list: can now display connectors without links.
- Measurements table: "N presynaptic sites" column includes now by default also
presynaptic connectors that don't have any postsynaptic link. To restore the
previous behavior, the "Include half links/synapses" checkbox can be
unchecked.
- Navigator tool and sub-tools like Tracing tool: the Z slider is now properly
hidden if the image data set has only one section.
API Changes
-----------
Additions:
- POST `/{project_id}/neurons/rename`:
Rename multiple neurons at the same time.
- GET|POST `/{project_id}/skeletons/summary`:
Retrieve meta information like cable length or number of nodes for skeletons.
- GET `/accounts/anonymous-api-token`:
Get the API token for the anonymous user.
- POST `/{project_id}/neurons/all-skeletons`:
Map a list of neuron IDs to the skeleton IDs that model them.
- GET `/{project_id}/`:
Get project details like the title and comment.
- POST `/{project_id}/`:
Update the project title if the user has admin permissions in the project.
Modifications:
- `GET /{project_id}/transactions/`:
Now accepts the optional parameters user_id and type to further constrain the
result. Type is a transaction label string, e.g. 'skeletons.import'.
- `POST /{project_id}/skeletons/import`:
Now accepts the optional boolean parameter replace_annotations, which can be
used to remove all existing annotations if an existing neuron/skeleton is
replaced with force=true.
- `POST /{project_id}/graphexport/json`:
The returned graph export is now in networkx 2.x format, rather than in v1.x
format like before. The main difference is that start and target nodes of an
edge are now defined explicitly by ID rather than an index.
- `POST /{project_id}/skeleton/connectivity_matrix`:
The new parameters row_relations and col_relations allow callers now to
specify the relations used to connect two skeletons for each side. They
default to "presynaptic_to" and "postsynaptic_to".
- `POST /{project_ids}/skeletons/in-bounding-box`:
Returns now also unlinked connectors by default. To only get linked connectors
like before, pass in `only_linked = true`.
Deprecations:
None.
Removals:
None.
Reply all
Reply to author
Forward
0 new messages