patch - Use newer datastore for mod_muc_log

[JC Brand]

Hi folks

I need something like mod_muc_log but I need to log to Postgres, so I wrote a
patch for mod_muc_log that does this by using the newer storage API.

In order to do this, I copied liberally from mod_muc_mam.

Some changes made:

- Use newer module storage API so that we can also store in SQL
- Adhere to config option `muc_log_all_rooms` (also used by mod_muc_mam)
- Add affiliation information in the logged stanza
- Remove code that set (and then removed) an "alreadyJoined" dummy element

I'm not really sure whether I should be using the `archive` store (like
mod_muc_mam does). Is this OK, or is it better to use a different store, or to
create a new one?

I removed the "alreadyJoined" element code since it doesn't look to me like
it's actually doing anything useful, so I assumed it's leftovers from previous
refactoring. Is that so, or why was that code there?

I don't think my changes are entirely backwards compatible because the stanza
being logged is no longer wrapped with `<stanza time=...>`. I'm not sure how to
make it backwards compatible aside from having an `if` statement and then
logging data the old way if internal storage is used.

Alternatively, I could publish this as a new module instead of patching
mod_muc_log.

I look forward to your feedback.

Thanks
JC

[JC Brand]

Hi guys,

friendly reminder that I sent this patch in May and would appreciate some feedback.

Thanks
JC

> --- src/prosody-modules/mod_muc_log/mod_muc_log.lua 2019-05-07 14:03:06.836030801 +0200
> +++ modules/mod_muc_archive/mod_muc_archive.lua 2019-05-07 16:28:37.276266654 +0200
> @@ -3,18 +3,40 @@
> -- Copyright (C) 2009-2013 Matthew Wild
> -- Copyright (C) 2013 Kim Alvefur
> -- Copyright (C) 2013 Marco Cirillo
> +-- Copyright (C) 2019 JC Brand
>
> local hosts = prosody.hosts;
> local tostring = tostring;
> +local st = require "util.stanza";
> local split_jid = require "util.jid".split;
> -local datamanager = require"core.storagemanager".olddm;
> -local data_load, data_store = datamanager.load, datamanager.store;
> +local jid_bare = require "util.jid".bare;
> +local time_now = os.time;
> +
> local datastore = "muc_log";
> local muc_form_config_option = "muc#roomconfig_enablelogging"
>
> +local log_all_rooms = module:get_option_boolean("muc_log_all_rooms", false);
> local log_by_default = module:get_option_boolean("muc_log_by_default", false);
> local log_presences = module:get_option_boolean("muc_log_presences", true);
>
> +local archive_store = "muc_logging_archive";
> +local archive = module:open_store(archive_store, "archive");
> +
> +local xmlns_muc_user = "http://jabber.org/protocol/muc#user";
> +
> +if archive.name == "null" or not archive.find then
> + if not archive.find then
> + module:log("error", "Attempt to open archive storage returned a driver without archive API support");
> + module:log("error", "mod_%s does not support archiving",
> + archive._provided_by or archive.name and "storage_"..archive.name.."(?)" or "<unknown>");
> + else
> + module:log("error", "Attempt to open archive storage returned null driver");
> + end
> + module:log("info", "See https://prosody.im/doc/storage and https://prosody.im/doc/archiving for more information");
> + return false;
> +end
> +
> +
> -- Module Definitions
>
> local function get_room_from_jid(jid)
> @@ -35,6 +57,12 @@
> end
>
> local function logging_enabled(room)
> + if log_all_rooms then
> + return true;
> + end
> + if room._data.hidden then -- do not log data of private rooms
> + return false;
> + end
> local enabled = room._data.logging;
> if enabled == nil then
> return log_by_default;
> @@ -42,6 +70,7 @@
> return enabled;
> end
>
> +
> function log_if_needed(event)
> local stanza = event.stanza;
>
> @@ -51,112 +80,90 @@
> then
> local node, host = split_jid(stanza.attr.to);
> if node and host then
> - local bare = node .. "@" .. host;
> - if get_room_from_jid(bare) then
> - local room = get_room_from_jid(bare)
> -
> - local today = os.date("!%y%m%d");
> - local now = os.date("!%H:%M:%S");
> -
> - local muc_to = nil
> - local muc_from = nil;
> - local already_joined = false;
> -
> - if room._data.hidden then -- do not log any data of private rooms
> - return;
> - end
> - if not logging_enabled(room) then -- do not log where logging is not enabled
> - return;
> - end
> -
> - if stanza.name == "presence" and stanza.attr.type == nil then
> - muc_from = stanza.attr.to;
> - if room._occupants and room._occupants[stanza.attr.to] then
> - already_joined = true;
> - stanza:tag("alreadyJoined"):text("true");
> - end
> - elseif stanza.name == "iq" and stanza.attr.type == "set" then -- kick, to is the room, from is the admin, nick who is kicked is attr of iq->query->item
> - if stanza.tags[1] and stanza.tags[1].name == "query" then
> - local tmp = stanza.tags[1];
> - if tmp.tags[1] ~= nil and tmp.tags[1].name == "item" and tmp.tags[1].attr.nick then
> - tmp = tmp.tags[1];
> - for jid, nick in pairs(room._jid_nick) do
> - if nick == stanza.attr.to .. "/" .. tmp.attr.nick then
> - muc_to = nick;
> - break;
> - end
> + local room = get_room_from_jid(node.."@"..host)
> + if not room then return end
> + -- Policy check
> + if not logging_enabled(room) then return end
> +
> + local muc_to = nil
> + local muc_from = nil;
> +
> + if stanza.name == "presence" and stanza.attr.type == nil then
> + muc_from = stanza.attr.to;
> + elseif stanza.name == "iq" and stanza.attr.type == "set" then -- kick, to is the room, from is the admin, nick who is kicked is attr of iq->query->item
> + if stanza.tags[1] and stanza.tags[1].name == "query" then
> + local tmp = stanza.tags[1];
> + if tmp.tags[1] ~= nil and tmp.tags[1].name == "item" and tmp.tags[1].attr.nick then
> + tmp = tmp.tags[1];
> + for jid, nick in pairs(room._jid_nick) do
> + if nick == stanza.attr.to .. "/" .. tmp.attr.nick then
> + muc_to = nick;
> + break;
> end
> end
> end
> - else
> - for jid, nick in pairs(room._jid_nick) do
> - if jid == stanza.attr.from then
> - muc_from = nick;
> - break;
> - end
> + end
> + else
> + for jid, nick in pairs(room._jid_nick) do
> + if jid == stanza.attr.from then
> + muc_from = nick;
> + break;
> end
> end
> + end
>
> - if (muc_from or muc_to) then
> - local data = data_load(node, host, datastore .. "/" .. today);
> - local realFrom = stanza.attr.from;
> - local realTo = stanza.attr.to;
> -
> - if data == nil then
> - data = {};
> - end
> + if (muc_from or muc_to) then
> + local stored_stanza = st.clone(stanza);
> + stored_stanza.attr.from = muc_from;
> + stored_stanza.attr.to = muc_to;
> +
> + if stanza.name == "message" then
> + local actor = jid_bare(room._occupants[muc_from].jid);
> + local affiliation = room:get_affiliation(actor) or "none";
> + local role = room:get_role(actor) or room:get_default_role(affiliation);
> + stored_stanza:add_direct_child(st.stanza("x", { xmlns = xmlns_muc_user })
> + :tag("item", { affiliation = affiliation; role = role; jid = actor }));
> + end
>
> - stanza.attr.from = muc_from;
> - stanza.attr.to = muc_to;
> - data[#data + 1] = "<stanza time=\"".. now .. "\">" .. tostring(stanza) .. "</stanza>\n";
> - stanza.attr.from = realFrom;
> - stanza.attr.to = realTo;
> - if already_joined == true then
> - if stanza[#stanza].name == "alreadyJoined" then -- normaly the faked element should be the last, remove it when it is the last
> - stanza[#stanza] = nil;
> - else
> - for i = 1, #stanza, 1 do
> - if stanza[i].name == "alreadyJoined" then -- remove the faked element
> - stanza[i] = nil;
> - break;
> - end
> - end
> - end
> - end
> - datamanager.getpath(node, host, datastore, nil, true); -- create the datastore dir
> - data_store(node, host, datastore .. "/" .. today, data);
> + local with = stanza.name
> + if stanza.attr.type then
> + with = with .. "<" .. stanza.attr.type
> end
> + archive:append(node, nil, stored_stanza, time_now(), with);
> end
> end
> end
> end
>
> -module:hook("muc-config-form", function(event)
> - local room, form = event.room, event.form;
> - table.insert(form,
> - {
> - name = muc_form_config_option,
> - type = "boolean",
> - label = "Enable Logging?",
> - value = logging_enabled(room),
> - }
> - );
> -end);
> -
> -module:hook("muc-config-submitted", function(event)
> - local room, fields, changed = event.room, event.fields, event.changed;
> - local new = fields[muc_form_config_option];
> - if new ~= room._data.logging then
> - room._data.logging = new;
> - if type(changed) == "table" then
> - changed[muc_form_config_option] = true;
> - else
> - event.changed = true;
> +if not log_all_rooms then
> + module:hook("muc-config-form", function(event)
> + local room, form = event.room, event.form;
> + table.insert(form,
> + {
> + name = muc_form_config_option,
> + type = "boolean",
> + label = "Enable Logging?",
> + value = logging_enabled(room),
> + }
> + );
> + end);
> +
> + module:hook("muc-config-submitted", function(event)
> + local room, fields, changed = event.room, event.fields, event.changed;
> + local new = fields[muc_form_config_option];
> + if new ~= room._data.logging then
> + room._data.logging = new;
> + if type(changed) == "table" then
> + changed[muc_form_config_option] = true;
> + else
> + event.changed = true;
> + end
> end
> - end
> -end);
> + end);
> +end
>
> module:hook("message/bare", log_if_needed, 1);
> +
> if log_presences then
> module:hook("iq/bare", log_if_needed, 1);
> module:hook("presence/full", log_if_needed, 1);

[Matthew Wild]

Hey,

Sorry, thanks for the reminder - I somehow missed the original and our
patch tracker is down at the moment.

On Wed, 2 Oct 2019 at 10:22, JC Brand <li...@opkode.com> wrote:
>
> Hi guys,
>
> friendly reminder that I sent this patch in May and would appreciate some feedback.
>
> Thanks
> JC
>
> On Tue, May 07, 2019 at 04:45:12PM +0200, JC Brand wrote:
> > Hi folks
> >
> > I need something like mod_muc_log but I need to log to Postgres, so I wrote a
> > patch for mod_muc_log that does this by using the newer storage API.
> >
> > In order to do this, I copied liberally from mod_muc_mam.

Is the high level goal that you just want MAM but without the client's
ability to query it? Does it matter to you if the client /can/ query
it? In other words, why not just use mod_muc_mam?

Regards,
Matthew

[li...@opkode.com]

We use mod_muc_mam as well, with an expiration of a few days.

The goal with this module is to have a permanent SQL-based archive for
compliance/reporting requirements.