cryo-ET: understanding how to combine filament curve-picking with neural net picking

[Esther Bullitt]

Hi, 

 I would like to use the Segmentation/Manual Segmentation/Draw curves to pick some of my filaments, and then get data from many more tomograms. 

If I manually box training references, segment them, train and apply the neural network, it finds individual particles along the filament, but very few, as the boxes can only be close enough to touch.  I would like to end up with overlapping boxes (as is possible by extracting from manually picked curves data).  Is auto-picking using the neural net possible?

e2version.py
EMAN 2.99.47 ( GITHUB: 2023-03-04 13:31 - commit: 3f313008c3185410fe859663e763dffb9c0b6fcc )
Your EMAN2 is running on: Linux-4.18.0-553.89.1.el8_10.x86_64-x86_64-with-glibc2.28 4.18.0-553.89.1.el8_10.x86_64
Your Python version is: 3.9.16

One other question - if I start a command and decide to change parameters, I know how to kill the job, but I would like some instruction on how to determine what can and cannot be deleted without effecting further processing.  - for example, I extract particles I no longer want; I can delete them in the particles directory, but what about the info/*.json file; do I need to edit that?

thanks very much.

Esther

-- 

Esther Bullitt, PhD

Professor of Pharmacology, Physiology & Biophysics

Director, Cryogenic Electron Microscopy Core

Boston University Chobanian & Avedisian School of Medicine

700 Albany St, W-302

Boston, MA. 02118

[Steve Ludtke]

Hi Esther,

cc'ing Muyuan here to draw attention to the discussion, since he has done filament work himself in the not too distant past

On Apr 1, 2026, at 2:08 PM, 'Esther Bullitt' via EMAN2 <em...@googlegroups.com> wrote:

Hi, 

 I would like to use the Segmentation/Manual Segmentation/Draw curves to pick some of my filaments, and then get data from many more tomograms. 

If I manually box training references, segment them, train and apply the neural network, it finds individual particles along the filament, but very few, as the boxes can only be close enough to touch.  I would like to end up with overlapping boxes (as is possible by extracting from manually picked curves data).  Is auto-picking using the neural net possible?

So, I haven't dealt with filaments personally in a long time, but as far as I recall we don't have a tool specifically designed for deep learning based filament tracing. There are 2 approaches we have/had. 

1) use the manual filament tracer on each tomogram. Yes, this is labor intensive, but

- allows you to control the direction of each filament, for example, if you were tracing axonal microtubules, you might want to get their polarity self-consistent

- allows you to generate boxes with specific overlap and known location with respect to each filament

2) you can use the deep learning annotation tool (not the particle picker) and have it annotate all of the filament density, then you can ask it to generate random boxes along filament-annotated density

The particle based deep learning picker is not optimized for filaments. Indeed, it is conceptually based on the idea of monodisperse particles, which filament chunks obviously aren't

e2version.py
EMAN 2.99.47 ( GITHUB: 2023-03-04 13:31 - commit: 3f313008c3185410fe859663e763dffb9c0b6fcc )
Your EMAN2 is running on: Linux-4.18.0-553.89.1.el8_10.x86_64-x86_64-with-glibc2.28 4.18.0-553.89.1.el8_10.x86_64
Your Python version is: 3.9.16

I thought you had updated your machine(s) to a fairly recent release?  This is about 3 years old?

One other question - if I start a command and decide to change parameters, I know how to kill the job, but I would like some instruction on how to determine what can and cannot be deleted without effecting further processing.  - for example, I extract particles I no longer want; I can delete them in the particles directory, but what about the info/*.json file; do I need to edit that?

It's pretty hard to offer sweeping advice here... it depends on what you're doing specifically. 

Generally speaking you can delete anything in particles or particles3d as long as you make sure to also delete any .lst files you may have created which reference them, and as long as you don't have existing refinement/initial model folders

Generally speaking you can completely delete any _XX numbered folder. ie - if you run e2spt_refine_new.py, you can completely delete any spt_XX folders without harm to the project. If you have created spt_01, spt_02, spt_03, spt_04, then delete spt_02 and spt_04, the next run you do will make a new spt_04. ie - when new numbered folders are created it is the highest current numbered folder of that type +1.

You can, of course, delete things in tomograms/ as well, but if you do you'll have to recreate them again.

For the most part you should avoid deleting anything in info/. Each of those files contains the aggregated metadata for each tilt series throughout the processing you've done. Yes, boxed out particles are part of that information, and if you decide you want to, for example, eliminate all of the previous particle picks you've made, you should use e2procjson.py rather than manually editing the files (since you may unintentionally produce syntax errors):

To completely erase all of the picked particles in all tomograms, for example:

e2procjson.py --removekey boxes_3d  info/*json

that will only remove box locations, not other related metadata.  If you are trying to figure out what metadata keys that are available to delete:

e2procjscon.py --listkeys info/<pick one json file>

Hope that helps...

thanks very much.

Esther

-- 

Esther Bullitt, PhD

Professor of Pharmacology, Physiology & Biophysics

Director, Cryogenic Electron Microscopy Core

Boston University Chobanian & Avedisian School of Medicine

700 Albany St, W-302

Boston, MA. 02118

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/eman2/32aa8838-78c9-4458-84e3-56a3902fcbfen%40googlegroups.com.

[Bullitt, Esther]

Hi Steve,

Thank you.  I think for now I will go ahead and do my own filament tracing, as I will end up with many more particles this way (unless Muyuan has a suggestion).

The e2procjson.py command will be very helpful — not sure why I ended up with 50K particles from one tomogram by mistake, but this command will be better than my editing the json with a text editor!

Yes, my computer has the latest eman2, but this is a newer/more powerful computer that I thought would help with this project.  I’ll investigate why it is running such an old version.

Thanks again,

Esther

From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Steve Ludtke <slud...@gmail.com>
Date: Thursday, April 2, 2026 at 8:15 AM
To: em...@googlegroups.com <em...@googlegroups.com>
Cc: Muyuan Chen <g5v...@gmail.com>
Subject: Re: [EMAN2] cryo-ET: understanding how to combine filament curve-picking with neural net picking

You don't often get email from slud...@gmail.com. Learn why this is important

You received this message because you are subscribed to a topic in the Google Groups "EMAN2" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/eman2/MPpA-a1D8oM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to eman2+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/eman2/FC15BC8E-C84C-451F-ABBB-9CB8F3E06C67%40gmail.com.

[Steve Ludtke]

Just make sure you only run that json command on the specific files you want to clear out, or you'll lose all of your particles on all of your tomograms!   In case you're paranoid, it isn't a terrible idea to 

tar cvzf info.tgz info

before you do any major changes on the info files. They are all text files and are tiny so the backup takes no time, and can potentially save you a LOT of work  :^)

To view this discussion visit https://groups.google.com/d/msgid/eman2/SJ0PR03MB6253A6D9AC8AD684EC4B5080C951A%40SJ0PR03MB6253.namprd03.prod.outlook.com.