Unstable API for op_info_database statistics?

[Mark Rotteveel]

I'm looking at implementing an op_info_database request using the
isc_info_read_seq_count, isc_info_read_idx_count (etc) request items,
but looking at the implementation, it seems that the item length is
unstable.

Based on the InterBase 6.0 API guide (page 55), I expect:

"""
When any of these information items is requested, InterBase returns to
the result buffer:
* 1 byte specifying the item type (for example, isc_info_insert_count).
* 2 bytes telling how many bytes compose the subsequent value pairs.
* A pair of values for each table in the database on which the requested
type of operation has occurred since the database was last attached.
Each pair consists of:
* 2 bytes specifying the table ID.
* 4 bytes listing the number of operations (for example, inserts) done
on that table.
"""

However, looking at the implementation on master, the statistics are
held in a SINT64, and the values are encoded with INF_convert, which
will either use 4 bytes or 8 bytes depending on the value.

This means that the length of the statistics are unknowable if it is
possible for them to exceed 2^31 - 1, because the length itself is not
encoded in the info response.

Is this assessment correct? Or did I miss something?

Mark
--
Mark Rotteveel

[Dmitry Yemanov]

Looks correct. The issue is that global counters (fetches/reads/etc) are
returned as 2-byte length followed by the 4-8 byte value, but
per-relation counters are missing the value length. While this was
always documented, IMHO this looks like violation of the API rules. And
yes, it became broken with 64-bit counters.


Dmitry

[Mark Rotteveel]

Do you want me to create an issue in the tracker?

--
Mark Rotteveel

[Dmitry Yemanov]

07.12.2022 17:33, Mark Rotteveel wrote:

>> Looks correct. The issue is that global counters (fetches/reads/etc)
>> are returned as 2-byte length followed by the 4-8 byte value, but
>> per-relation counters are missing the value length. While this was
>> always documented, IMHO this looks like violation of the API rules.
>> And yes, it became broken with 64-bit counters.
>
> Do you want me to create an issue in the tracker?

Can you think of any other solution than switching to a whole new set of
opcodes?


Dmitry

[Mark Rotteveel]

The current API will need to be fixed to always return 4 byte values.

If we want to expose the full 64-bit range then we indeed need to add
new info items. If we do that, we also need to consider extensibility,
because I think the current implementation has problems if you have a
lot of tables with statistics, that is, at 6 bytes per table, it can
currently can only report statistics on 10922 tables. If we go to 10
bytes per table (2 byte table, 8 byte value), this goes down to 6553
tables, or if we go to 12 bytes (including 2 byte table, 2 byte length,
4 or 8 byte value), it goes down to 8191 - 5461. In practice, a
connection is probably unlikely to have so many active statistics, but
maybe this is something we should take into account.

Mark
--
Mark Rotteveel

[Dimitry Sibiryakov]

Mark Rotteveel wrote 07.12.2022 16:20:
> If we want to expose the full 64-bit range then we indeed need to add new info
> items. If we do that, we also need to consider extensibility, because I think
> the current implementation has problems if you have a lot of tables with
> statistics, that is, at 6 bytes per table, it can currently can only report
> statistics on 10922 tables. If we go to 10 bytes per table (2 byte table, 8 byte
> value), this goes down to 6553 tables, or if we go to 12 bytes (including 2 byte
> table, 2 byte length, 4 or 8 byte value), it goes down to 8191 - 5461. In
> practice, a connection is probably unlikely to have so many active statistics,
> but maybe this is something we should take into account.

I would like to point up to two things about new API:

1. We can have separate block with the same tag for each single table instead of
single solid buffer for them all (perhaps this trick could work for current API
as well).
2. Integer value may have length 1-8 bytes, not just 4 or 8.
isc_portable_integer() is prepared for this.

--
WBR, SD.

[Mark Rotteveel]

On 07-12-2022 16:26, 'Dimitry Sibiryakov' via firebird-devel wrote:
> Mark Rotteveel wrote 07.12.2022 16:20:
>> If we want to expose the full 64-bit range then we indeed need to add
>> new info items. If we do that, we also need to consider extensibility,
>> because I think the current implementation has problems if you have a
>> lot of tables with statistics, that is, at 6 bytes per table, it can
>> currently can only report statistics on 10922 tables. If we go to 10
>> bytes per table (2 byte table, 8 byte value), this goes down to 6553
>> tables, or if we go to 12 bytes (including 2 byte table, 2 byte
>> length, 4 or 8 byte value), it goes down to 8191 - 5461. In practice,
>> a connection is probably unlikely to have so many active statistics,
>> but maybe this is something we should take into account.
>
>   I would like to point up to two things about new API:
>
> 1. We can have separate block with the same tag for each single table
> instead of single solid buffer for them all (perhaps this trick could
> work for current API as well).

It will probably work, but it would be a bit of a gamble if existing
implementations are actually able to handle it.

> 2. Integer value may have length 1-8 bytes, not just 4 or 8.
> isc_portable_integer() is prepared for this.

The specific usage within Firebird only *encodes* the values as either 4
or 8 bytes.

Mark
--
Mark Rotteveel

[Mark Rotteveel]

On 07-12-2022 15:27, Dmitry Yemanov wrote:

> Looks correct. The issue is that global counters (fetches/reads/etc) are
> returned as 2-byte length followed by the 4-8 byte value, but
> per-relation counters are missing the value length. While this was
> always documented, IMHO this looks like violation of the API rules. And
> yes, it became broken with 64-bit counters.

I created https://github.com/FirebirdSQL/firebird/issues/7414

Mark
--
Mark Rotteveel