SpamAssassin (Spam/Ham) support for Sup

[Robert Winkler]

Hi, am looking for a long time for a high-performance email solution (see http://www.meteomex.com/serendipity/index.php?/archives/10-Rule-your-Inboxes-with-RAcS-ReadAction-Spam.html).

Sup has most of the features I need; especially fully keyboard-driven/ vim-like (tried mu4e, but emacs is just not for me..).

However: How could I mark mails as Spam/ Ham and move them into a separate IMAP Folder?

Ideally, I would just tag them with ",s" for Spam and ",h" as Ham, and when synching, the mails are moved (or better copied?) for training.

Best regards, 

Robert 

[Iain Parris]

Hi Robert,

Excerpts from Robert Winkler's message of 2020-07-06 11:36:45 -0700:

> However: How could I mark mails as Spam/ Ham and move them into a separate
> IMAP Folder?
>
> Ideally, I would just tag them with ",s" for Spam and ",h" as Ham, and when
> synching, the mails are moved (or better copied?) for training.

I understand what you are aiming for. This is an interesting question.
:-)

Sup has hooks, and I believe these could be used to extend Sup with this
functionality. For a full list of hooks, launch sup with "--list-hooks".
(See <https://github.com/sup-heliotrope/sup/wiki/Hooks>.)

Are you looking to operate on threads (groups of messages) or individual
emails? For example, existing hook "mark-as-spam" operates on an entire
thread, not an individual message - which I suspect is probably not what
you're looking for

Instead, to operate on individual messages (which I think is what
SpamAssassin would prefer), I think creating a custom variant inspired
by the existing "pipe_message" method may succeed. This allows you to
pipe an entire email message to any arbitrary *nix command (e.g., could
be used to write the message contents to a new file in either a special
"ham" or "spam" directory, which in turn could be fed into sa-learn).

(For a demonstration of pipe_message, open a message in Sup, then type
"|" and enter, then: "wc -l" and enter. This calls "wc -l" with the
message contents, and returns a line count.)

So I think it would be possible to accomplish what you are looking for
with two hooks:

- "startup" hook: Extend Redwood::ThreadViewMode with new methods, e.g.,
send_to_sa_ham and send_to_sa_spam. For inspiration, see
Redwood::ThreadViewMode#pipe_message (in file
lib/sup/modes/thread_view_mode.rb).

- "keybindings" hook: To bind keys to the new functions.

For inspiration from another user, see:
<https://christop.club/2014/01/19/sup/#hooks> - this shows an example of
extending Sup (Redwood) classes with new methods, and binding keys to
these new methods.

I know this is only a very rough outline of a possible solution - but
does this make sense, and sound like a path you may be interested in?

Kind regards,
Iain

[Robert Winkler]

Thanks, Good hint, Iain!

The '|' is an interesting option, but I did not get it working for direct file operations. Instead, I am working on a different approach:

Usually, the real Spam and real Ham emails can be identified already in the Inbox view and labeled:

startup.rb:

class Redwood::InboxMode
  def label_spamassassin_spam
   t = cursor_thread or return
   t.apply_label :spam
   multi_toggle_new [t]
   regen_text
   Index.save_thread t
   flush_index
  end
end


class Redwood::InboxMode
  def label_spamassassin_ham
   t = cursor_thread or return
   t.apply_label :ham
   regen_text
   Index.save_thread t
   flush_index
  end
end


class Redwood::SearchResultsMode
  def label_spamassassin_spam
   t = cursor_thread or return
   t.apply_label :spam
   multi_toggle_new [t]
   regen_text
   Index.save_thread t
   flush_index
  end
end


class Redwood::SearchResultsMode
  def label_spamassassin_ham
   t = cursor_thread or return
   t.apply_label :ham
   regen_text
   Index.save_thread t
   flush_index
  end
end

keybindings.rb:

modes["inbox-mode"].keymap.add :label_spamassassin_spam, "Label as SpamAssassin Spam", 's'
modes["inbox-mode"].keymap.add :label_spamassassin_ham, "Label as SpamAssassin Ham", 'h'
modes["search-results-mode"].keymap.add :label_spamassassin_spam, "Label as SpamAssassin Spam", 's'
modes["search-results-mode"].keymap.add :label_spamassassin_ham, "Label as SpamAssassin Ham", 'h'

I want to move the mails in a second step (yet to be resolved).

Any hints on possible external programs/ scripts for moving mails with a particular label to a special /Maildir folder (e.g. ~/Maildir/SpamAssassin/Spam)?

Best regards, 

Robert 

[Iain Parris]

Hi Robert,

Excerpts from Robert Winkler's message of 2020-07-08 10:06:55 -0700:
> Thanks, Good hint, Iain!

Thank you!

> I want to move the mails in a second step (yet to be resolved).
>
> Any hints on possible external programs/ scripts for moving mails with a
> particular label to a special /Maildir folder (e.g.
> ~/Maildir/SpamAssassin/Spam)?

Question: do the mails need to be moved, or can they be copied?

(Side note: Sup should de-duplicate copies of mails that appear in more
than one source, i.e., should recognise that it is a duplicate of the
same message, and not show it twice. Or for simplicity, don't tell Sup
to index the SpamAssassin Maildirs!)

If making a copy is sufficient, then you can write the email to a new
file in the appropriate Maildir (since a Maildir is just a collection of
files, with each file being one email). All you need is a random
filename. For example, you could pipe to cat to create with a random
filename (a UUID), using something like:

cat >~/Maildir/SpamAssassin/Spam/$(cat /proc/sys/kernel/random/uuid)

Kind regards,
Iain

[Robert Winkler]

Hi Iain,

Yes, this is a good option. I modified my workflow a bit (Spam/Ham are not watched by Sup anymore. The mails should be copied to the Spam/Ham IMAP server for learning and are automatically deleted after 5 days). 

Moving the mail from the | pipe in the thread view with

  cat > ~/Maildir/SpamAssassin/Spam/cur/$(cat /proc/sys/kernel/random/uuid)

works; The mail is synced to the IMAP server with OfflineIMAP. However, how can I pass the mail below the cursor directly to a pipe command?

The following code causes an error:

class Redwood::InboxMode
  def label_spamassassin_spam
   t = cursor_thread or return
   t.apply_label :

spam
   t.remove_label :inbox
   multi_toggle_new [t]
   hide_thread t
   pipe_to_process('cat > ~/Maildir/SpamAssassin/Spam/cur/$(cat /proc/sys/kernel/random/uuid)')

   regen_text
   Index.save_thread t
   flush_index
  end
end

--- LocalJumpError from thread: main
no block given (yield)
/usr/local/sup/lib/sup/mode.rb:125:in `block in pipe_to_process'
/usr/lib/ruby/2.5.0/open3.rb:205:in `popen_run'
/usr/lib/ruby/2.5.0/open3.rb:95:in `popen3'
/usr/local/sup/lib/sup/mode.rb:107:in `pipe_to_process'
/home/rob/.sup/hooks/startup.rb:8:in `label_spamassassin_spam'
/usr/local/sup/lib/sup/mode.rb:59:in `handle_input'
/usr/local/sup/lib/sup/buffer.rb:222:in `handle_input'
/usr/local/sup/bin/sup:257:in `<module:Redwood>'
/usr/local/sup/bin/sup:76:in `<main>'

Best regards, 

Robert

[Iain Parris]

Hi Robert,

Excerpts from Robert Winkler's message of 2020-07-09 15:20:12 -0700:

> Yes, this is a good option. I modified my workflow a bit (Spam/Ham are not
> watched by Sup anymore. The mails should be copied to the Spam/Ham IMAP
> server for learning and are automatically deleted after 5 days).

OK, sounds good.

> Moving the mail from the | pipe in the thread view with
> cat > ~/Maildir/SpamAssassin/Spam/cur/$(cat /proc/sys/kernel/random/uuid)
> works; The mail is synced to the IMAP server with OfflineIMAP. However, how
> can I pass the mail below the cursor directly to a pipe command?

If we're acting on an entire thread, then we can't use pipe_to_process
directly. That would only work for an individual message.

However, I had a look, and I think I've found a better way to do this.

Assuming your source is also another Maildir, then we can find the full
path for the original message file. Then we can do a direct file copy of
this file into your SpamAssassin Spam Maildir.

Here's a demonstration of the concept:


=== ~/.sup/hooks/keybindings.rb ===

modes["inbox-mode"].keymap.add :label_spamassassin_spam, "Label as SpamAssassin Spam", 's'

=== ~/.sup/hooks/startup.rb ===

class Redwood::InboxMode
def label_spamassassin_spam

thread = cursor_thread or return
spamassassin_maildir_path = "/home/youruser/Maildir/SpamAssassin/Spam/cur/"
message_file = File.join(
thread.latest_message.source.file_path,
thread.latest_message.source_info
)
toggle_spam
raise "Label 'spam' not set" unless thread.latest_message.labels.include? :spam
FileUtils.cp(message_file, spamassassin_maildir_path)
BufferManager.flash "Copied to SpamAssassin Spam file: #{File.basename message_file}"
end
end


Kind regards,
Iain

[Iain Parris]

Excerpts from Iain Parris's message of 2020-07-10 21:26:12 +0100:

> Assuming your source is also another Maildir, then we can find the full
> path for the original message file. Then we can do a direct file copy of
> this file into your SpamAssassin Spam Maildir.

I should add: in this demo, we copy only the *latest* message in the
thread.

Kind regards,
Iain

[Robert Winkler]

Thanks for all the useful code, Iain.

Here the final solution. Explication: Spam (= 'undesired') messages and Ham (= 'desired') messages are copied into separate Maildir folders. 

Those are synchronized with an IMAP server (using OfflineImap) for training a SpamAssassin filter (running daily on a server, using CRON):

The last message of a thread is chosen by 's' (Spam) or 'h' (Ham) in the Inbox or Search Results Mode (e.g. in the display of unread messages).

Spam messages are removed from the Inbox listing. Ham messages get a label 'ham' which indicates that the message was already sent to training.

The SpamAssassin folders are not observed by sup. Mails older than 5 days are automatically deleted by (email and password are different, of course):

archive-mails.sh

echo 'cleaning Spam/Ham folders'
archivemail -d 5 --delete imaps://"spa...@bioprocess.org":"email-terminator2020!"@imap.ionos.de/Spam
archivemail -d 5 --delete imaps://"spa...@bioprocess.org":"email-terminator2020!"@imap.ionos.de/Ham

keybindings.rb

modes["inbox-mode"].keymap.add :label_spamassassin_spam, "Label as SpamAssassin Spam", 's'

modes["inbox-mode"].keymap.add :label_spamassassin_ham, "Label as SpamAssassin Ham", 'h'
modes["search-results-mode"].keymap.add :label_spamassassin_spam, "Label as SpamAssassin Spam", 's'
modes["search-results-mode"].keymap.add :label_spamassassin_ham, "Label as SpamAssassin Ham", 'h'

startup.rb

class Redwood::InboxMode
  def label_spamassassin_spam
    thread = cursor_thread or return

    spamassassin_maildir_path = "/home/rob/Maildir/SpamAssassin/Spam/cur/"
    message_file = File.join(
      thread.latest_message.source.file_path,
      thread.latest_message.source_info.to_s)
    toggle_spam
    FileUtils.cp(message_file, spamassassin_maildir_path)

    BufferManager.flash "Copied to SpamAssassin Spam file: #{File.basename message_file}"
  end
end

class Redwood::InboxMode
  def label_spamassassin_ham
 thread = cursor_thread or return
 spamassassin_maildir_path = "/home/rob/Maildir/SpamAssassin/Ham/cur/"
 message_file = File.join(
 thread.latest_message.source.file_path,
 thread.latest_message.source_info.to_s)
 thread.apply_label :ham
 FileUtils.cp(message_file, spamassassin_maildir_path)
    BufferManager.flash "Copied to SpamAssassin Ham file: #{File.basename message_file}"
  end
end

class Redwood::SearchResultsMode

  def label_spamassassin_spam
    thread = cursor_thread or return

    spamassassin_maildir_path = "/home/rob/Maildir/SpamAssassin/Spam/cur/"
    message_file = File.join(
      thread.latest_message.source.file_path,
      thread.latest_message.source_info.to_s)
    toggle_spam
    FileUtils.cp(message_file, spamassassin_maildir_path)

    BufferManager.flash "Copied to SpamAssassin Spam file: #{File.basename message_file}"
  end
end

class Redwood::SearchResultsMode
  def label_spamassassin_ham
 thread = cursor_thread or return
 spamassassin_maildir_path = "/home/rob/Maildir/SpamAssassin/Ham/cur/"
 message_file = File.join(
 thread.latest_message.source.file_path,
 thread.latest_message.source_info.to_s)
 thread.apply_label :ham
 FileUtils.cp(message_file, spamassassin_maildir_path)
    BufferManager.flash "Copied to SpamAssassin Ham file: #{File.basename message_file}"
  end
end

note: The source_info needed a conversion to string: .to_s

Thanks a lot again; sup is really mighty!

Best regards, 

Robert 

[Iain Parris]

Excerpts from Robert Winkler's message of 2020-07-10 14:58:16 -0700:

> Thanks for all the useful code, Iain.
>
> Here the final solution.

Thank you, Robert - I'm happy that I could help.

I intend to update the Sup wiki in the near future with a new section
for user-submitted hooks, and would be happy to include yours there as
an example to inspire future users.

> Thanks a lot again; sup is really mighty!

Absolutely - long live Sup! :-)

Kind regards,
Iain

[Robert Winkler]

Correction: The :unread label of spam mails has to be removed after copyingl. Why? If you remove :unread, the mail file is moved from the /new directory and not found any more. My working function looks like this:

class Redwood::InboxMode
  def label_spamassassin_spam
    thread = cursor_thread or return
    spamassassin_maildir_path = "/home/rob/Maildir/SpamAssassin/Spam/cur/"
    message_file = File.join(
      thread.latest_message.source.file_path,
      thread.latest_message.source_info.to_s)

    FileUtils.cp(message_file, spamassassin_maildir_path)
    BufferManager.flash "Copied to SpamAssassin Spam file: #{File.basename message_file}"

    toggle_spam
    thread.remove_label :unread
    flush_index
  end
end

Best, Robert

On Friday, 10 July 2020 16:58:16 UTC-5, Robert Winkler wrote:

Thanks for all the useful code, Iain.

Here the final solution. Explication: Spam (= 'undesired') messages and Ham (= 'desired') messages are copied into separate Maildir folders. 

Those are synchronized with an IMAP server (using OfflineImap) for training a SpamAssassin filter (running daily on a server, using CRON):

The last message of a thread is chosen by 's' (Spam) or 'h' (Ham) in the Inbox or Search Results Mode (e.g. in the display of unread messages).

Spam messages are removed from the Inbox listing. Ham messages get a label 'ham' which indicates that the message was already sent to training.

The SpamAssassin folders are not observed by sup. Mails older than 5 days are automatically deleted by (email and password are different, of course):

archive-mails.sh

echo 'cleaning Spam/Ham folders'
archivemail -d 5 --delete imaps://"spamass@bioprocess.org":"email-terminator2020!"@imap.ionos.de/Spam
archivemail -d 5 --delete imaps://"spamass@bioprocess.org":"email-terminator2020!"@imap.ionos.de/Ham