Trace Plugin Design

[Artyom Abakumov]

Hello. I have a custom trace plugin to calculate some statistics. It works well until the client wants to use both my plugin and internal 'fbtrace' plugins. The same config file from AuditTraceConfigFile comes to the both plugins so config of my plugin should be compatible with the fbtrace (and no exstantable). This can be avoided by explicitly loading different configurations, but without a AuditTraceConfigFile value, there will be no trace session, and no plugin will be run. So the outcomes:

1. When using only my plugin: set my configuration in AuditTraceConfigFile.
2. When using both fbtrace and my plugin: set AuditTraceConfigFile = fbtrace.conf and load my configuration from a different place

Another issue is version control. It is necessary to wait for a DBMS restart since it is impossible to update the plugin on the fly. Additionally, the interface does not provide any getVersion functions.

Maybe it is possible to implement some improvements to the design:

1) Introduce a new interface function getVersion and a corresponding service action print_version (this seems relatively straightforward to implement).

2) Add the ability to reload plugins on the fly. (From what I can see, this could be implemented in TraceManager::update_sessions(), before realoding the sessions and then update plugin for each session)

3) Move TracePlugin and AuditTraceConfigFile to plugins.conf (in FB6 or FB7). This would allow specifying configuration file for each plugin (or no configuration if a plugin does not require it).

4) Allow to sepcify trace plugin in tracemgr

What do you think of these suggestions?

[Dimitry Sibiryakov]

Artyom Abakumov wrote 18.08.2025 10:31:
> What do you think of these suggestions?

Half of them is wrong:
1. Trace session doesn't need AuditTraceConfigFile, it receives configuration
from client side.
2. Moving configuration to plugins.conf won't help because there can be only one
trace plugin name set in firebird.conf.

All of them can be replaced with one (which may be already in the tracker
IIRC): "allow to set trace plugin name for each trace session separately".

--
WBR, SD.

[Artyom Abakumov]

Artyom Abakumov wrote 18.08.2025 10:31:
> What do you think of these suggestions?

Half of them is wrong:
1. Trace session doesn't need AuditTraceConfigFile, it receives configuration
from client side.

I am speaking of the system session. Of course, you can run a user session, but it is not so convenient to run a separate application 24/7 just to start, keep, check and restart a trace session.

 

2. Moving configuration to plugins.conf won't help because there can be only one
trace plugin name set in firebird.conf.

I think I didn’t clearly express my ideas. I propose to remove AuditTraceConfigFile and TracePlugin from firebird.conf and add something like this to the plugins.conf:

Plugin = FBTRACE {

Type = TracePlugin

Module = $(dir_plugins)/fbtrace

Config = fbtrace_config

}

Config = fbtrace_config {
path = $(root)/fbtrace.conf
}

 Plugin = STAT_TRACE {

Type = TracePlugin

Module = $(dir_plugins)/stattrace

Config = stats_config

}

Config = stats_config {
path = $(root)/my_config.conf
}

The TraceManager will read the plugins.conf and accumulate all the Trace plugins

[Dimitry Sibiryakov]

Artyom Abakumov wrote 18.08.2025 11:39:
> I propose to remove AuditTraceConfigFile and TracePlugin from firebird.conf and
> add something like this to the plugins.conf

1. plugins.conf isn't for this.
2. There can be only one audit session and obviously it can use only one plugin.

There is nothing bad with user session using fbtracemgr or an application
being run 24/7.

--
WBR, SD.

[Artyom Abakumov]

1. plugins.conf isn't for this.

 Maybe. But there must be a place for custom plugin settings

2. There can be only one audit session and obviously it can use only one plugin.

However, Firebird allows setting two or more values in TracePlugin 

...

for (GetPlugins<ITraceFactory> traceItr(IPluginManager::TYPE_TRACE); traceItr.hasData(); traceItr.next())

...

There is nothing bad with user session using fbtracemgr or an application
being run 24/7.

 don't say it's bad; it's just inconvenient

[Dimitry Sibiryakov]

Artyom Abakumov wrote 18.08.2025 11:58:
> 1. plugins.conf isn't for this.
>
>  Maybe. But there must be a place for custom plugin settings

Yes, and the place for "custom plugin settings" IS plugins.conf. These
settings are delivered to plugin via IPluginFactory::createPlugin() parameter.
Your custom trace plugin is free to use these parameters instead of those,
delivered via ItraceFactory::trace_create() parameter.


--
WBR, SD.

[Artyom Abakumov]

And still, if I want to use my plugin as a system trace session, I have to set something in the AuditTraceConfigFile option

понедельник, 18 августа 2025 г. в 13:26:06 UTC+3, Dimitry Sibiryakov:

[Artyom Abakumov]

*As a single plugin

понедельник, 18 августа 2025 г. в 14:17:58 UTC+3, Artyom Abakumov:

[Dimitry Sibiryakov]

Artyom Abakumov wrote 18.08.2025 13:17:
> And still, if I want to use my plugin as a system trace session, I have to set
> something in the AuditTraceConfigFile option

Yes, any file. You'll get its content "as is" in trace_create(). You are free
to ignore it.

--
WBR, SD.

[Artyom Abakumov]

I understand your vision, but for me, you're suggesting a workaround rather than a solution. For me, this clearly indicates that the plugin design has some flaws. I think, at the very least, the AuditTracePlugin could be removed from firebird.conf and moved to plugins.conf instead.

 

[Dimitry Sibiryakov]

Artyom Abakumov wrote 18.08.2025 13:46:
> I understand your vision, but for me, you're suggesting a workaround rather than
> a solution.

I'm suggesting nothing, I describe things as they works now.

--
WBR, SD.

[Dimitry Sibiryakov]

Artyom Abakumov wrote 18.08.2025 13:46:
> I understand your vision,

And no, it is not a vision, this is working code from ibptrace plugin.

--
WBR, SD.

[Vlad Khorsun]

18.08.2025 11:31, Artyom Abakumov:

> Hello. I have a custom trace plugin to calculate some statistics. It works well until the client wants to use both my plugin and
> internal 'fbtrace' plugins. The same config file from AuditTraceConfigFile comes to the both plugins so config of my plugin should
> be compatible with the fbtrace (and no exstantable). This can be avoided by explicitly loading different configurations, but without
> a AuditTraceConfigFile value, there will be no trace session, and no plugin will be run. So the outcomes:
>
> 1. When using only my plugin: set my configuration in AuditTraceConfigFile.
> 2. When using both fbtrace and my plugin: set AuditTraceConfigFile = fbtrace.conf and load my configuration from a different place
>
> Another issue is version control. It is necessary to wait for a DBMS restart since it is impossible to update the plugin on the fly.
> Additionally, the interface does not provide any getVersion functions.
>
> Maybe it is possible to implement some improvements to the design:
>
> 1) Introduce a new interface function getVersion and a corresponding service action print_version (this seems relatively
> straightforward to implement).

Version of what ? For what ?

> 2) Add the ability to reload plugins on the fly. (From what I can see, this could be implemented in TraceManager::update_sessions(),
> before realoding the sessions and then update plugin for each session)

Trace Manager **is not** a Plugin Manager, don't ask it to do others job.
Also, PluginManager already unloads plugins after 60 sec of last use.

> 3) Move TracePlugin and AuditTraceConfigFile to plugins.conf (in FB6 or FB7). This would allow specifying configuration file for
> each plugin (or no configuration if a plugin does not require it).

Both settings belongs to the engine, so - no, this is wrong.
BTW, did you try to specify list of plugins for TracePlugin value ?

> 4) Allow to sepcify trace plugin in tracemgr

fbtracemgr is command-line interface for the Trace API. You can add any kind of command
in it, but it makes no sense without support at engine side. I.e. we need implement a way
of how to specify desired trace plugin(s) for the trace session. It also should address a
task of specifying custom configuration for custom plugin.
> What do you think of these suggestions? You misunderstand concept of server audit: it must be non-interruptible and controlled by
privileged person only (DBA in our case). Therefore it is intentional not possible to replace
audit configuration without a full stop of the server. You may agree or not, but this was
original idea of audit session.

What you want is non-interactive trace session like audit but without such strong requirements.
Initially, it was intended to run separate instance of fbtracemgr for it, btw, therefore there is
no support in the engine for such kind of sessions. Of course it can be implemented. With no need
to screw up audit session.


Regards,
Vlad

[Artyom Abakumov]

18.08.2025 11:31, Artyom Abakumov:
> Hello. I have a custom trace plugin to calculate some statistics. It works well until the client wants to use both my plugin and
> internal 'fbtrace' plugins. The same config file from AuditTraceConfigFile comes to the both plugins so config of my plugin should
> be compatible with the fbtrace (and no exstantable). This can be avoided by explicitly loading different configurations, but without
> a AuditTraceConfigFile value, there will be no trace session, and no plugin will be run. So the outcomes:
>
> 1. When using only my plugin: set my configuration in AuditTraceConfigFile.
> 2. When using both fbtrace and my plugin: set AuditTraceConfigFile = fbtrace.conf and load my configuration from a different place
>
> Another issue is version control. It is necessary to wait for a DBMS restart since it is impossible to update the plugin on the fly.
> Additionally, the interface does not provide any getVersion functions.
>
> Maybe it is possible to implement some improvements to the design:
>
> 1) Introduce a new interface function getVersion and a corresponding service action print_version (this seems relatively
> straightforward to implement).

Version of what ? For what ?

Just an interface to validate the installed plugin version. This is helpful for administrative purposes (to check that the correct version of the plugin is being used).

 

> 2) Add the ability to reload plugins on the fly. (From what I can see, this could be implemented in TraceManager::update_sessions(),
> before realoding the sessions and then update plugin for each session)

Trace Manager **is not** a Plugin Manager, don't ask it to do others job.
Also, PluginManager already unloads plugins after 60 sec of last use.

So, if I replace a plugin with a new one, even old connections can use the new plugin?

 

> 3) Move TracePlugin and AuditTraceConfigFile to plugins.conf (in FB6 or FB7). This would allow specifying configuration file for
> each plugin (or no configuration if a plugin does not require it).

Both settings belongs to the engine, so - no, this is wrong.
BTW, did you try to specify list of plugins for TracePlugin value ?

I tried two plugins, and it works well.

 

> What do you think of these suggestions? You misunderstand concept of server audit: it must be non-interruptible and controlled by
privileged person only (DBA in our case). Therefore it is intentional not possible to replace
audit configuration without a full stop of the server. You may agree or not, but this was
original idea of audit session. 

Ok. it's just a pain that in production, I have to wait several days before the server gets restarted.

 

What you want is non-interactive trace session like audit but without such strong requirements.
Initially, it was intended to run separate instance of fbtracemgr for it, btw, therefore there is
no support in the engine for such kind of sessions. Of course it can be implemented. With no need
to screw up audit session.

Okay, I got it. By design, there must be a single trace session with the most important logging. All additional needs should be requested via user sessions.

Adding plugin as a user session option should solve all the issues. Sounds not too hard to implement. I can try to implement this and create a pull request

[Vlad Khorsun]

18.08.2025 17:51, Artyom Abakumov:

>
> 18.08.2025 11:31, Artyom Abakumov:
> > Hello. I have a custom trace plugin to calculate some statistics. It works well until the client wants to use both my plugin and
> > internal 'fbtrace' plugins. The same config file from AuditTraceConfigFile comes to the both plugins so config of my plugin
> should
> > be compatible with the fbtrace (and no exstantable). This can be avoided by explicitly loading different configurations, but
> without
> > a AuditTraceConfigFile value, there will be no trace session, and no plugin will be run. So the outcomes:
> >
> > 1. When using only my plugin: set my configuration in AuditTraceConfigFile.
> > 2. When using both fbtrace and my plugin: set AuditTraceConfigFile = fbtrace.conf and load my configuration from a different
> place
> >
> > Another issue is version control. It is necessary to wait for a DBMS restart since it is impossible to update the plugin on
> the fly.
> > Additionally, the interface does not provide any getVersion functions.
> >
> > Maybe it is possible to implement some improvements to the design:
> >
> > 1) Introduce a new interface function getVersion and a corresponding service action print_version (this seems relatively
> > straightforward to implement).
>
> Version of what ? For what ?
>
> Just an interface to validate the installed plugin version. This is helpful for administrative purposes (to check that the correct
> version of the plugin is being used).

Again, not a job of the TraceManager.

> > 2) Add the ability to reload plugins on the fly. (From what I can see, this could be implemented in
> TraceManager::update_sessions(),
> > before realoding the sessions and then update plugin for each session)
>
> Trace Manager **is not** a Plugin Manager, don't ask it to do others job.
> Also, PluginManager already unloads plugins after 60 sec of last use.
>
> So, if I replace a plugin with a new one, even old connections can use the new plugin?

Not in the current code, unfortunately. TraceManager was implemented long before plugins-based
architecture, thus it is still do some job that should be done by PluginManager now
(see TraceManager::factories). But it can be changed, of course.

> > 3) Move TracePlugin and AuditTraceConfigFile to plugins.conf (in FB6 or FB7). This would allow specifying configuration file for
> > each plugin (or no configuration if a plugin does not require it).
>
> Both settings belongs to the engine, so - no, this is wrong.
> BTW, did you try to specify list of plugins for TracePlugin value ?
>
> I tried two plugins, and it works well.

Good, one problem less

> > What do you think of these suggestions? You misunderstand concept of server audit: it must be non-interruptible and
> controlled by
> privileged person only (DBA in our case). Therefore it is intentional not possible to replace
> audit configuration without a full stop of the server. You may agree or not, but this was
> original idea of audit session.
>
> Ok. it's just a pain that in production, I have to wait several days before the server gets restarted.

Of course it is much better than just run separate instance of fbtracemgr and redirect its
output to the file. Or create own simple application...

> What you want is non-interactive trace session like audit but without such strong requirements.
> Initially, it was intended to run separate instance of fbtracemgr for it, btw, therefore there is
> no support in the engine for such kind of sessions. Of course it can be implemented. With no need
> to screw up audit session.
>
> Okay, I got it. By design, there must be a single trace session with the most important logging. All additional needs should be
> requested via user sessions.

Seems so.

> Adding plugin as a user session option should solve all the issues. Sounds not too hard to implement. I can try to implement this
> and create a pull request

I suggest to discuss it here before start implementation.

Regards,
Vlad

[Artyom Abakumov]

Seems so.
> Adding plugin as a user session option should solve all the issues. Sounds not too hard to implement. I can try to implement this
> and create a pull request
I suggest to discuss it here before start implementation.

 

1) Add a string field `ses_plugin` to TraceSession and the corresponding isc_ tag and option to the fbtracemgr.
2) Add a new static field `Factories* userPlugins` to the TraceManager.

3) In `TraceManager::update_session`, add a check for ses_plugin content. If a plugin name is present, check the userPlugins for the plugin. If it is present, use the existing factory. Otherwise, create and add the factory to the userPlugins via GetPlugins.

ses_plugin must contain only one plugin name.

 

[Dimitry Sibiryakov]

Artyom Abakumov wrote 18.08.2025 16:51:
> BTW, did you try to specify list of plugins for TracePlugin value ?
>
> I tried two plugins, and it works well.

How output did look like? Did strings from different plugins interleave?

--
WBR, SD.

[Vlad Khorsun]

18.08.2025 18:57, Artyom Abakumov:

> Seems so.
> > Adding plugin as a user session option should solve all the issues. Sounds not too hard to implement. I can try to implement
> this
> > and create a pull request
> I suggest to discuss it here before start implementation.
>
> 1) Add a string field `ses_plugin` to TraceSession and the corresponding isc_ tag and option to the fbtracemgr.

Ok

> 2) Add a new static field `Factories* userPlugins` to the TraceManager.
> 3) In `TraceManager::update_session`, add a check for ses_plugin content. If a plugin name is present, check the userPlugins for the
> plugin. If it is present, use the existing factory. Otherwise, create and add the factory to the userPlugins via GetPlugins.
> ses_plugin must contain only one plugin name.

Here I would remove TraceManager::factories (and load_plugins() method) and use
GetPlugins instead (in update_session()) - let PluginManager load and unload plugins.
GetPlugins::name() should be used to filter required plugins. Or desired plugins list
could be passed into GetPlugins() ctor. I see no need to limit session by single plugin.
If plugins list was not set by the user, use NULL value and GetPlugins() will use the
value of TracePlugin setting.

Next, merge FactoryInfo into SessionInfo and keep reference on plugin (ITraceFactory
actually) here. Release it togheter with reference on ITracePlugin.

I could miss some details, of course.

Regards,
Vlad

[Artyom Abakumov]

My plugin writes to a separate predefined file, so the output isn't an issue for me. A also tried setting two fbtrace plugins and each event was doubled

[Artyom Abakumov]

Here I would remove TraceManager::factories (and load_plugins() method) and use
GetPlugins instead (in update_session()) - let PluginManager load and unload plugins.

 

I thought factories were intended to avoid plugin manager routines and library loading. Wouldn't this add to the overhead?

[Vlad Khorsun]

18.08.2025 19:30, Artyom Abakumov:

> Here I would remove TraceManager::factories (and load_plugins() method) and use
> GetPlugins instead (in update_session()) - let PluginManager load and unload plugins.
>
> I thought factories were intended to avoid plugin manager routines

No. It was implemented when there was no plugins, no Plugin Manager.

> and library loading. Wouldn't this add to the overhead?

No. Plugin Manager knows what modules it already loaded and when its should be
unloaded. This is a job of Plugin Manager, no need to duplicate it.


Regards,
Vlad

[Artyom Abakumov]

Got it. Then I will start implementing the improvements 

[Artyom Abakumov]

The PR is ready 

https://github.com/FirebirdSQL/firebird/pull/8707

понедельник, 18 августа 2025 г. в 19:42:56 UTC+3, Artyom Abakumov: