An idea to replace GDML and GPRE in Firebird code

[Adriano dos Santos Fernandes]

Hi!

GPRE usage in Firebird is very problematic:
- GPRE itself needs a fake engine, so we have gpre_boot.
- The written by user code uses GDML, that is not maintained.
- GDML is not powerful as SQL.
- GPRE code is totally crap.
- We write huge epp files with many C++ code not related directly with
the queries. IDE experience is crap with these epp files.
- The generated code is crap to debug.

But usage of GPRE has a strenght:
- Static types leads to less errors.
- Queries do not need to run to be detected with syntax errors.

So I was thinking on a way to improve situation and I think this idea works.

First, engine code is already prepared to run system queries coded in
SQL (text, not necessarily pre-compiled). As soon database is
initialized we fill metadata cache from structures defined in code.

Second, different than maintain GPRE, it's not difficult to write a
preprocessor tool from scratch.

So the new tool will catch SQL code inside CPP files - syntax to be
defined and will generate corresponding header file.

The user written CPP file will include this generated header.

But to understand SQL and generate types, this tool needs an engine. An
engine that may not be already be built.

Here comes the trick. User code will read fields using methods like
asInt(), asShort(), asString().

The preprocessor will work in two modes: disconnected and connected.

In disconnected mode, it will generate field types in a way that all
field read methods will be valid. In runtime this may fail, convert, or
convert with a warning. To be decided.

In connected mode, the preprocessor will use the engine to prepare
queries and discover real field types and generate them.

And here another trick. When preprocessor is run in connected mode and
emit types for a given query, if this query is changed later in the cpp
code, the cpp code will automatically switch types to the disconnected
mode way.

That will happen because generated types will be based on constexpr
function that hashes query text.

When it's run again in connected mode, new types will be generated and
user code using wrong field read methods will fail to compile and may be
fixed.

This is all to handle development flux, where a query is changed and we
do not want to put and maintain types manually, nor have failing queries
discovered on in runtime when it's run.

It's very raw idea from today, but I would say this may be easy to do
and will make our code much better.

It also allow incremental change from GDML + epp to pure CPP + SQL +
generated header with the same strengths as the current solution and
much less drawbacks.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 11.10.2023 13:00:
> So I was thinking on a way to improve situation and I think this idea works.
>
> First, engine code is already prepared to run system queries coded in
> SQL (text, not necessarily pre-compiled). As soon database is
> initialized we fill metadata cache from structures defined in code.
>
> Second, different than maintain GPRE, it's not difficult to write a
> preprocessor tool from scratch.

Too complicated IMHO.
Most GDML queries in the engine is access to single record of single table or
simple join or small set of records.
It can be easily handled by simple functions:

Relation Database::getTable(tableId);
Record Relation::getRecord(fieldId, value);
std::iterator<Record> Table::getRecords(fieldId, value);

and so on.

Table and field ids are from relations.h and fields.h so any mistype is
caught by compiler.

--
WBR, SD.

[Adriano dos Santos Fernandes]

Em qua., 11 de out. de 2023 10:26, 'Dimitry Sibiryakov' via firebird-devel <firebir...@googlegroups.com> escreveu:

   Too complicated IMHO.
   Most GDML queries in the engine is access to single record of single table or
simple join or small set of records.
   It can be easily handled by simple functions:

   Relation Database::getTable(tableId);
   Record Relation::getRecord(fieldId, value);
   std::iterator<Record> Table::getRecords(fieldId, value);

   and so on.

   Table and field ids are from relations.h and fields.h so any mistype is
caught by compiler.

A ugly problem awaiting 20+ years to be fixed is not simple as you mention.

There are where conditions, there are joins, there are operations being done in c++ code because gdml do not support that should better be pushed to the queries...

Time ago i tried path to transform c++ expressions to BLR at compile time. It would be complex and another point of maintainability.

What I describe here is not complex and is small piece of software, without points of constant maintainability, and that solve a superset (not subset as you described) of the current problems.

Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 11.10.2023 17:14:
> What I describe here is not complex and is small piece of software, without
> points of constant maintainability, and that solve a superset (not subset as you
> described) of the current problems.

But it is not as effective. One of main advantages of GDML is that it bypass
parser and go directly to BLR. A set of low-level access functions can bypass
BLR as well and use VIO directly that increase speed even more.
Joins are just nested loops, conditions are either handled inside of loop on
iterator or (if index available) use indexed access inside of the access methods.
Such kind of table access level is necessary in any case if Firebird ever
plans to have cross-database queries so why to use provisory solutions if
ultimate one can be done?..

--
WBR, SD.

[Alex Peshkoff]

On 10/11/23 14:00, Adriano dos Santos Fernandes wrote:
> Hi!
>
> GPRE usage in Firebird is very problematic:
> - GPRE itself needs a fake engine, so we have gpre_boot.
> - The written by user code uses GDML, that is not maintained.
> - GDML is not powerful as SQL.
> - GPRE code is totally crap.
> - We write huge epp files with many C++ code not related directly with
> the queries. IDE experience is crap with these epp files.
> - The generated code is crap to debug.
>
> But usage of GPRE has a strenght:
> - Static types leads to less errors.
> - Queries do not need to run to be detected with syntax errors.
>
> So I was thinking on a way to improve situation and I think this idea works.
>
> First, engine code is already prepared to run system queries coded in
> SQL (text, not necessarily pre-compiled). As soon database is
> initialized we fill metadata cache from structures defined in code.
>
> Second, different than maintain GPRE, it's not difficult to write a
> preprocessor tool from scratch.
>
> So the new tool will catch SQL code inside CPP files - syntax to be
> defined and will generate corresponding header file.
>
> The user written CPP file will include this generated header.
>
> But to understand SQL and generate types, this tool needs an engine. An
> engine that may not be already be built.

Current solution with gpre_boot appears to be not as bad as it may
appear. When running in connected state such preprocessor will anyway
have to load metadata from database. What's wrong if in disconnected
state it loads that metadata from that same structures used when
database is initialized like gpre_boot does? Provided this is special
compile-time tool we do not need to build it twice - when engine
unavailable it simply loads metadata in another way. This avoids need to
preprocess same files twice - with or without engine.

Very interesting to see an ideas about builtin SQL syntax. I do not ask
for complete syntax description, just main ideas pls.

[Jim Starkey]

I'm sorry you don't like the GPRE code.  It was my first C program.  It
was written on a DEC Pro/350 (PDP-11) running the ZENIX operating
system.  Perhaps you might give some latitude if you understood more
about writing code for an unsegmented 16 bit address space.  It is,
however, 39 years old and while, I'm sure, getting long in tooth, it
still seems to be giving some useful service.  Perhaps you have other 39
year old programs that you like better.

Of course GPRE needed a "boot" mode.  It had to work before I could
start on the engine (the engine started on the Pro/350 but moved to an 2
MB Apollo DN300 with virtual memory!).

And I will quibble about GDML vs SQL.  GMDL was a bona fide programming
language, which was critical for reducing network overhead on 10 MB
Ethernet and primitive network adapters.  As an aside, you might be
interested to know that Interbase was the first database to support
heterogeneous networks.  Around the time that Interbase (then Groton
Database Systems) first shipped, workstations were typically 2 MB almost
never over 8 MB.

BLR existed because it wasn't the least bit obvious which would prevail:
DEC's data access language (another name for GDML), Oracle's SQL, or
Ingres's Quel, so a syntax neutral representation made sense.  And so
did a really cheap BLR based engine when alternative was a precedence
driven compile bogging down a database server.

But time have changed, we're learned a lot about OO systems, pure
virtual interfaces, and, in generally we have cycles to waste and memory
to burn.  So pox on SQL: May it live forever in software archaeology.  
Dogmatic, dumb language, that violates it "mathematical foundation" on
every "select" statement.

If it were me, I'd design a whole new API from scratch (oh, wait, I just
did!), an API that was only plumbing with no semantics. Then you could
run may data access languages over a simple API and plumbing and let
creativity bloom.  But I digress.

--
Jim Starkey, AmorphousDB, LLC

[Jim Starkey]

Exposing internal interfaces like VIO to application code is a crime
against software architecture.  Give it up.

Jim Starkey, AmorphousDB, LLC

[Adriano dos Santos Fernandes]

On 11/10/2023 12:28, 'Dimitry Sibiryakov' via firebird-devel wrote:
> Adriano dos Santos Fernandes wrote 11.10.2023 17:14:
>> What I describe here is not complex and is small piece of software,
>> without points of constant maintainability, and that solve a superset
>> (not subset as you described) of the current problems.
>
>   But it is not as effective. One of main advantages of GDML is that it
> bypass parser and go directly to BLR.

I didn't said because I think this is not important...

But the "connected" mode may generate BLR (preparing the SQL and asking
for BLR, like SET EXEC_PATH_DISPLAY BLR does), which may work in the
same way (based on constexpr hash of the SQL text, a changed query
(waiting to be re-generated through the connected mode), could use SQL,
while an non-modified generated query may use the precompiled BLR directly.

The idea is that the "connected" mode should be in use in 99% cases,
except when query is modified. Or, when tree has not the generated files.

This process works without versioning the generated files.

But as the generated files is not going to be ugly, things will be clear
and less error-prone also versioning these generated files.

Note that these files are going to have pieces only related to the
queries, not a mix of user C++ code as was present in the same cpp file.



Adriano

[Adriano dos Santos Fernandes]

On 11/10/2023 12:55, Alex Peshkoff wrote:
> On 10/11/23 14:00, Adriano dos Santos Fernandes wrote:
>> Hi!
>>
>> GPRE usage in Firebird is very problematic:
>> - GPRE itself needs a fake engine, so we have gpre_boot.
>> - The written by user code uses GDML, that is not maintained.
>> - GDML is not powerful as SQL.
>> - GPRE code is totally crap.
>> - We write huge epp files with many C++ code not related directly with
>> the queries. IDE experience is crap with these epp files.
>> - The generated code is crap to debug.
>>
>

> Current solution with gpre_boot appears to be not as bad as it may
> appear.

For me, points above make things very unproductive.

I tried alternative approaches before, like make thin epp file with
queries and std::function callback, but the way gpre generates codes
make this also not possible.

> When running in connected state such preprocessor will anyway
> have to load metadata from database. What's wrong if in disconnected
> state it loads that metadata from that same structures used when
> database is initialized like gpre_boot does? Provided this is special
> compile-time tool we do not need to build it twice - when engine
> unavailable it simply loads metadata in another way. This avoids need to
> preprocess same files twice - with or without engine.
>

The tool will preprocess real SQL code.

There is alternative, that is, separate almost dsql code related to
statement compile from the engine, making the compiler use an abstract
metadata loader, that may be switched to a static one.

But of course, this is *much* more difficult.

> Very interesting to see an ideas about builtin SQL syntax. I do not ask
> for complete syntax description, just main ideas pls.
>

I may prepare some hand-made source plus "generated" code that works in
both connected and disconnected mode using fbclient for discussing this
area.


Adriano

[Artyom Abakumov]

One more problem with the current GPRE code is the large amount of duplicate code. `backup.epp`, `show.epp`, and `extract.epp` basically use the same selects from all system tables. When a new field is added, it needs to be changed manually in all of these files. So my proposal for the new tool is to use iterators instead of loops. We can generate a select iterator in a header file and use it in all the CPP files.

среда, 11 октября 2023 г. в 14:00:16 UTC+3, Adriano dos Santos Fernandes:

[Alex Peshkoff]

On 10/12/23 03:40, Adriano dos Santos Fernandes wrote:

>> When running in connected state such preprocessor will anyway
>> have to load metadata from database. What's wrong if in disconnected
>> state it loads that metadata from that same structures used when
>> database is initialized like gpre_boot does? Provided this is special
>> compile-time tool we do not need to build it twice - when engine
>> unavailable it simply loads metadata in another way. This avoids need to
>> preprocess same files twice - with or without engine.
>>
> The tool will preprocess real SQL code.
>
> There is alternative, that is, separate almost dsql code related to
> statement compile from the engine, making the compiler use an abstract
> metadata loader, that may be switched to a static one.

I have not understood that your plan is to prepare statements in engine
and get MessageMetadata from prepared statement. I've thought only about
getting database metadata from engine. Well, now I understand better why
you say that preprocessor is going to be very simple.

> But of course, this is *much* more difficult.

Yes, certainly it's too difficult.

[Dimitry Sibiryakov]

Jim Starkey wrote 12.10.2023 0:29:
> Exposing internal interfaces like VIO to application code is a crime against
> software architecture.  Give it up.

I'm not talking about application code. I'm talking about Firebird Engine
code. It IS internal.

--
WBR, SD.

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 12.10.2023 2:28:
> But the "connected" mode may generate BLR (preparing the SQL and asking
> for BLR, like SET EXEC_PATH_DISPLAY BLR does), which may work in the
> same way (based on constexpr hash of the SQL text, a changed query
> (waiting to be re-generated through the connected mode), could use SQL,
> while an non-modified generated query may use the precompiled BLR directly.
>
> The idea is that the "connected" mode should be in use in 99% cases,
> except when query is modified. Or, when tree has not the generated files.

I don't understand what "connected mode" you are talking about. This topic is
about using of GDML in Firebird code, right?
Firebird code can query only system tables which structure is already known
and defined in named headers. Everything can and must be done at compile time.
Queries are never modified at runtime.

--
WBR, SD.

[Alex Peshkoff]

One more question - when we build release on GH, in what mode will it
run? I suppose in disconnected.  Is it OK that release will contain
fields that have not passed compile-time type control?

[Adriano dos Santos Fernandes]

Maybe we should replace terms first:
- connected -> strict
- disconnected -> relaxed

Releases will use strict mode, user's c++ code needs to use correct
.asType() method for each field.

Relaxed mode exists for a single purpose: build a (modified) engine when
there is no engine available for the preprocessor to work preparing
queries and discovering types. For example, when new ODS is being
developed, or when user checks out tree and starts working on queries
before first build.

Since we're going to commit generated files that matches what the
developer just did, releases will use the strict mode.

I think this workflow may be clear when there is some thing to demonstrate.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 12.10.2023 14:14:
> Since we're going to commit generated files that matches what the
> developer just did, releases will use the strict mode.

relations.h and fields.h are not generated. No generated CPP files are kept
in repository. We are not going to commit them.

Preprocessing of queries that involve only system tables don't need engine or
database at all.
The only purpose on "connected" mode is to process user application code
working with user defined tables (or hosted on server of different version).
None of these circumstances apply to Firebird code itself.

--
WBR, SD.

[Alex Peshkoff]

On 10/12/23 15:14, Adriano dos Santos Fernandes wrote:
> Since we're going to commit generated files that matches what the
> developer just did, releases will use the strict mode.

That was a missing point - commit generated files.
Now looks like this should work.

[Adriano dos Santos Fernandes]

Hi!

I put an example of the concept and flow mechanics here:

https://github.com/asfernandes/temp-fb-preprocessor-concept-demo/tree/pass1

https://github.com/asfernandes/temp-fb-preprocessor-concept-demo/tree/pass2

https://github.com/asfernandes/temp-fb-preprocessor-concept-demo/tree/pass3

It's 3 commits and 3 tags.

Please do not attach to the access interfaces here, it's not the focus
of the demo.

Nor to the runtime and preprocessor tool - they do not exist there.

In pass1 we start with the framework and an already made query
(user-query.cpp) and its generated file (generated-query.h).

In Firebird, user-query.cpp is likely each individual epp file we have
currently.

In pass2 we wanted to add another field to the query, but we had not the
engine built, so the preprocessor cannot prepare queries. So we added
the field to the query text (in user-query.cpp), but to access this
field we had to manually add the field to the generated file. In this
case, I purposely added the field using a wrong type, because it does
not matter. It only need to be listed there in the correct order.

Note that in the context of Firebird development, in most cases when the
developer edits a query in the cpp file, it already has a working engine
built that can be used to preprocess and regenerate the file correctly,
not needing to manually edit the generated file.

But in the example I wanted to go this path, considering the currently
engine is not built or is broken, and the edit is very simple.

In pass3, after pass2 was done and an engine was built and is working,
we used the preprocessor that can use the engine to prepare queries,
discover its types and regenerate the file.

So you see, in pass2 (when query in cpp file is different than query as
present in the generated file), field types are weak. pass2 is
intermediary step in developer's machine and is never expected to be
committed.

In pass1 and pass3, user and generated queries matches, so field types
are strong based in how Firebird engine described the query.

In CI build, we can use special define that makes the build fail instead
of use weak types if one commits user and generated files that do not match.


Adriano

[Alex Peshkoff]

On 10/12/23 20:26, Adriano dos Santos Fernandes wrote:

> Nor to the runtime and preprocessor tool - they do not exist there.
>
> In pass1 we start with the framework and an already made query
> (user-query.cpp) and its generated file (generated-query.h).
>
> In Firebird, user-query.cpp is likely each individual epp file we have
> currently.
>
> In pass2 we wanted to add another field to the query, but we had not the
> engine built, so the preprocessor cannot prepare queries. So we added
> the field to the query text (in user-query.cpp), but to access this
> field we had to manually add the field to the generated file. In this
> case, I purposely added the field using a wrong type, because it does
> not matter. It only need to be listed there in the correct order.
>
> Note that in the context of Firebird development, in most cases when the
> developer edits a query in the cpp file, it already has a working engine
> built that can be used to preprocess and regenerate the file correctly,
> not needing to manually edit the generated file.
>
> But in the example I wanted to go this path, considering the currently
> engine is not built or is broken, and the edit is very simple.

Well understanding that runtime is missing now want to ask - how do you
plan to handle call like fields.RDB$RELATION_ID.asString() ? Raise error
I think?

And one small suggestion - taking into an account that this is very
specific tool, may be use field names in lower case? Not to pollute src
code.

> In pass3, after pass2 was done and an engine was built and is working,
> we used the preprocessor that can use the engine to prepare queries,
> discover its types and regenerate the file.
>
> So you see, in pass2 (when query in cpp file is different than query as
> present in the generated file), field types are weak. pass2 is
> intermediary step in developer's machine and is never expected to be
> committed.
>
> In pass1 and pass3, user and generated queries matches, so field types
> are strong based in how Firebird engine described the query.
>
> In CI build, we can use special define that makes the build fail instead
> of use weak types if one commits user and generated files that do not match.

Enable weak types only in DEV_BUILD?

[Adriano dos Santos Fernandes]

On 13/10/2023 12:07, Alex Peshkoff wrote:
>
> Well understanding that runtime is missing now want to ask - how do you
> plan to handle call like fields.RDB$RELATION_ID.asString() ? Raise error
> I think?
>

I think an assert is ok, if it's dev build only feature.

> And one small suggestion - taking into an account that this is very
> specific tool, may be use field names in lower case? Not to pollute src
> code.
>

Ok for me.

>
> Enable weak types only in DEV_BUILD?
>

Seems good.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 14.10.2023 17:52:
> On 13/10/2023 12:07, Alex Peshkoff wrote:
>> Well understanding that runtime is missing now want to ask - how do you
>> plan to handle call like fields.RDB$RELATION_ID.asString() ? Raise error
>> I think?
>>
> I think an assert is ok, if it's dev build only feature.

IMHO, it either should follow common type coercion rules (i.e. allow implicit
conversion) or produce compilation error.

--
WBR, SD.

[Adriano dos Santos Fernandes]

Another idea is to not have a c++ custom pre-processor, but define
queries in a different file, say, SQL files.

So the tool will act on SQL files and generate header files.

Here is an example, for MET.

----------
-- Query: GetDomain

select fld.rdb$field_type,
fld.rdb$field_scale,
fld.rdb$field_length,
fld.rdb$field_sub_type,
fld.rdb$character_set_id,
fld.rdb$collation_id
from rdb$fields
where rdb$field_name = :fieldName;


-- Query: GetRelationField

select fld.rdb$field_type,
fld.rdb$field_scale,
fld.rdb$field_length,
fld.rdb$field_sub_type,
fld.rdb$character_set_id,
fld.rdb$collation_id,
rfl.rdb$collation_id
from rdb$relation_fields rfl
join rdb$fields fld
on fld.rdb$field_name = rfl.rdb$field_source
where rfl.rdb$relation_name = :relationNamed and
rfl.rdb$field_name = :fieldName;
----------

In c++ file (and in the generated header), the queries will be
references by their names (-- Query syntax, or something else).

As we're going to have the generated queries in git, it's not a problem
even for the first build.

This would make the creation of the tool even easier, and would avoid
SQL mixed with C++ (IMO it's very problematic to have SQL queries
formatted with tabs).

The parse of the SQL file would only need to separate queries and do
some processing needing to have named input parameters (until it's
engine feature).


Adriano

[Adriano dos Santos Fernandes]

I would like to describe path chosen when I started to implement this,
and a problem I missed.

The queries are described in yaml files, and a tool (sqlgen) read these
files and are capable to do these things:
- Generate a C++ header file, to be included and used in the cpp files
where previously GDML was used
- Fix a yaml file, that is, regenerate parameters and field types
- Validate (not yet implemented) that a currently yaml file matches what
the "fix" command would generate

So user write this code:

met.sql.yml:

-----
options:
internal: true
namespace: Jrd

queries:
MetGetDomain:
sql: |

select fld.rdb$field_type,
fld.rdb$field_scale,
fld.rdb$field_length,
fld.rdb$field_sub_type,
fld.rdb$character_set_id,
fld.rdb$collation_id,

fld.rdb$null_flag,
fld.rdb$default_value,
fld.rdb$validation_blr
from rdb$fields fld
where fld.rdb$field_name = :field_name
parameters:
- name: field_name
type: char
size: 63
charSet: utf8
fields:
- name: rdb$field_type
type: smallint
- name: rdb$field_scale
type: smallint
- name: rdb$field_length
type: smallint
- name: rdb$field_sub_type
type: smallint
- name: rdb$character_set_id
type: smallint
- name: rdb$collation_id
type: smallint
- name: rdb$null_flag
type: smallint
- name: rdb$default_value
type: blob
- name: rdb$validation_blr
type: blob
-----

Without sections "parameters" and "fields", sqlgen is able to generate
these sections with the "fix" command. Of course, to do it, an already
built engine must exist.

User can manually write these sections and types need not be the exact
ones. If one do that, then after engine is built, "fix" may be used to
have completely correct types written.

This file is going to be stored in git.

From this file, met.sql.h is generated. Fields and parameters are
FB_MESSAGE-based, so implementation is easy. It just needs small library
code in common and jrd (for system queries).

Here is the generated header:

-----
#ifndef JRD_MET_SQL_YML_H
#define JRD_MET_SQL_YML_H

#include "firebird.h"
#include "../jrd/StaticInternalSql.h"

namespace Jrd
{


class MetGetDomain
{
private:
static constexpr auto SQL =
R"""(select fld.rdb$field_type,

fld.rdb$field_scale,
fld.rdb$field_length,
fld.rdb$field_sub_type,
fld.rdb$character_set_id,
fld.rdb$collation_id,

fld.rdb$null_flag,
fld.rdb$default_value,
fld.rdb$validation_blr
from rdb$fields fld
where fld.rdb$field_name = ?
)""";

public:
FB_MESSAGE(ParametersMessage, Firebird::ThrowWrapper,
(FB_INTL_CHAR(63, CS_UTF8), field_name)
);

FB_MESSAGE(FieldsMessage, Firebird::ThrowWrapper,
(FB_SMALLINT, rdb$field_type)
(FB_SMALLINT, rdb$field_scale)
(FB_SMALLINT, rdb$field_length)
(FB_SMALLINT, rdb$field_sub_type)
(FB_SMALLINT, rdb$character_set_id)
(FB_SMALLINT, rdb$collation_id)
(FB_SMALLINT, rdb$null_flag)
(FB_BLOB, rdb$default_value)
(FB_BLOB, rdb$validation_blr)
);

public:
class Query : public Jrd::StaticInternalSqlQuery<ParametersMessage,
FieldsMessage>
{
public:
Query(Jrd::Attachment* attachment, Jrd::jrd_tra* transaction)
: StaticInternalSqlQuery(attachment, transaction, SQL)
{}
};
};


} // Jrd

#endif // JRD_MET_SQL_YML_H
-----

This file is going to be generated and not stored in git. Much like in
places gpre is currently used, but sqlgen does not need an already built
engine to do it.

And here is its usage:

-----
void MET_get_domain(thread_db* tdbb, MemoryPool& csbPool, const
MetaName& name, dsc* desc,
FieldInfo* fieldInfo)
{
/**************************************
*
* M E T _ g e t _ d o m a i n
*
**************************************
*
* Functional description
* Get domain descriptor and informations.
*
**************************************/
SET_TDBB(tdbb);
const auto attachment = tdbb->getAttachment();
const auto transaction = attachment->getSysTransaction();
bool found = false;

// FIXME: AutoCacheRequest handle(tdbb, irq_l_domain, IRQ_REQUESTS);

MetGetDomain::Query query(attachment, transaction);

query.forEach(
[&](auto&& p) {
p->field_name.set(name.c_str());
},

[&](auto&& f) {
if (!DSC_make_descriptor(desc,
f->rdb$field_type,
f->rdb$field_scale,
f->rdb$field_length,
f->rdb$field_sub_type,
f->rdb$character_set_id,
f->rdb$collation_id))
{
return;
}

found = true;

if (!fieldInfo)
return;

fieldInfo->nullable = f->rdb$null_flagNull || f->rdb$null_flag == 0;

Jrd::ContextPoolHolder context(tdbb, &csbPool);

if (f->rdb$default_valueNull)
fieldInfo->defaultValue = nullptr;
else
fieldInfo->defaultValue = parse_field_default_blr(tdbb, (bid*)
&f->rdb$default_value);

if (f->rdb$validation_blrNull)
fieldInfo->validationExpr = nullptr;
else
{
fieldInfo->validationExpr = parse_field_validation_blr(tdbb,
(bid*) &f->rdb$validation_blr, name);
}
}
);

if (!found)
ERR_post(Arg::Gds(isc_domnotdef) << Arg::Str(name));
}
-----

I think that path works better than what I initially described and it
covers the same workflows: build engine code just checked out, change
system queries during development, validate system queries in
development or CI/production build.

Now the problem: I think that since v2.5, our GDML queries can access
nonexistent system fields in runtime, treating them as NULL in read and
nop in write. That simplified a lot handling of multiple ODS and in
ISQL, make it faster as there is no need to run multiple queries to get
additional fields.

With SQL that is going to be difficult. I'd hate to return to that
multiple queries approach.

A path I didn't explore yet but I believe may work is to let sqlgen
"fix" store the BLR of just prepared queries in the yaml file. With some
hashes, in runtime it will be possible to chose SQL or BLR-based execution.

SQL runtime, to be used when yml is outdated (i.e., when one is changing
system queries), will work only with current ODS.

BLR runtime, to be used when yaml files are not modified, hence just
checked out tree build, andproduction builds, will work like GDML now.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 21.10.2023 22:32:
> The queries are described in yaml files, and a tool (sqlgen) read these files
> and are capable to do these things: - Generate a C++ header file, to be included
> and used in the cpp files where previously GDML was used - Fix a yaml file, that
> is, regenerate parameters and field types - Validate (not yet implemented) that
> a currently yaml file matches what the "fix" command would generate

I'm sorry, but for me it looks even worse that current GDML.

--
WBR, SD.

[Adriano dos Santos Fernandes]

I've started this topic showing the problems we should solve IMO:

- GPRE itself needs a fake engine, so we have gpre_boot.
- The written by user code uses GDML, that is not maintained.
- GDML is not powerful as SQL.
- GPRE code is totally crap.
- We write huge epp files with many C++ code not related directly with
the queries. IDE experience is crap with these epp files.
- The generated code is crap to debug.

The proposal fix all of this, and also:
- Simplify build, removing the boot phase
- Allow write and test of system queries in tools (isql, other) and
easily dump them in a yaml file to be used in Firebird code
- Allow us to deprecate GPRE right now, and remove it in v6
- It start fixing our very problematic, slow and outdated dependency
management: yaml-cpp was included using vcpkg (already used in the MacOS
build)

So the least expected thing is that you give reasons to hate it,
otherwise it looks personal or emotional.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 22.10.2023 14:21:
> I've started this topic showing the problems we should solve IMO:

>- GPRE itself needs a fake engine, so we have gpre_boot.

That's wrong. To compile queries to system tables (Firebird sources has no
others) any engine is not needed at all because system tables are defined by
headers jrd/relations.h, fields.h, etc.

> - The written by user code uses GDML, that is not maintained.

That's not a problem, existing functionality is enough.

> - GDML is not powerful as SQL.

Existing functionality is enough. If you miss something - GPRE supports SQL
as well. It is called "ESQL".

> - GPRE code is totally crap.

So it is time for someone to clean it up which is faster that reinventing a
wheel from scratch.

> - We write huge epp files with many C++ code not related directly with the
> queries. IDE experience is crap with these epp files.

Cannot say about MSVC, Code::Block is good enough with them for me.

> - The generated code is crap to debug.

That's the reason to change the code generator, that's all. Much simpler task
that creating a new one from scratch.

> The proposal fix all of this, and also:
> - Simplify build, removing the boot phase

I can be done with GPRE as well as I wrote above.

> - Allow write and test of system queries in tools (isql, other) and
> easily dump them in a yaml file to be used in Firebird code

The last thing any project needed is an another programming language to
study. As I already said GPRE supports SQL via "EXEC SQL".

> - Allow us to deprecate GPRE right now, and remove it in v6

If you are talking about customer supports and including GPRE into
distribution packages - it is not related to internal build process at all and
can be done any time. But it hasn't been done yet for a reason: some valuable
customers still use ESQL.

> - It start fixing our very problematic, slow and outdated dependency
> management: yaml-cpp was included using vcpkg (already used in the MacOS
> build)

That's an another reason NOT to use yaml at all.

--
WBR, SD.

[Adriano dos Santos Fernandes]

On 22/10/2023 09:34, 'Dimitry Sibiryakov' via firebird-devel wrote:
> Adriano dos Santos Fernandes wrote 22.10.2023 14:21:
>> I've started this topic showing the problems we should solve IMO:
>
>> - GPRE itself needs a fake engine, so we have gpre_boot.
>
>   That's wrong. To compile queries to system tables (Firebird sources
> has no others) any engine is not needed at all because system tables are
> defined by headers jrd/relations.h, fields.h, etc.
>

That is gpre_boot, a special version.

>> - The written by user code uses GDML, that is not maintained.
>
>   That's not a problem, existing functionality is enough.
>

It's not. DFW needs recursive queries already, that is done using
multiple queries.

ISQL is a pig shit of code and slow due to bad query language.

It's horrible code to put even basic features, so only the minimal
things is being done there.

>> - GDML is not powerful as SQL.
>
>   Existing functionality is enough.

It's not.

> If you miss something - GPRE
> supports SQL as well. It is called "ESQL".
>

It does not work for JRD's system queries and is far from currently SQL
dialect. I'd say it's v1.5 or v1.0 SQL dialect almost.

>> - GPRE code is totally crap.
>
>   So it is time for someone to clean it up which is faster that
> reinventing a wheel from scratch.
>

Someone arguing against others proposals may do it, but did not.

>> - We write huge epp files with many C++ code not related directly with
>> the queries. IDE experience is crap with these epp files.
>
>   Cannot say about MSVC, Code::Block is good enough with them for me.
>

Didn't know this tools has users today.

>> - The generated code is crap to debug.
>
>   That's the reason to change the code generator, that's all. Much
> simpler task that creating a new one from scratch.
>

Feel free to start a counter proposal and implement it.

>> The proposal fix all of this, and also:
>> - Simplify build, removing the boot phase
>
>   I can be done with GPRE as well as I wrote above.
>

Feel free.

>> - Allow write and test of system queries in tools (isql, other) and
>> easily dump them in a yaml file to be used in Firebird code
>
>   The last thing any project needed is an another programming language
> to study. As I already said GPRE supports SQL via "EXEC SQL".
>

Let's remove automated builds if yaml is a programming language.

>> - Allow us to deprecate GPRE right now, and remove it in v6
>
>   If you are talking about customer supports and including GPRE into
> distribution packages - it is not related to internal build process at
> all and can be done any time. But it hasn't been done yet for a reason:
> some valuable customers still use ESQL.
>

I would think very fondly if they are in this list:
https://firebirdsql.org/en/sponsors/

Anyway, this is unmaintained client tools and Firebird server/engine
always works with old clients.

But deprecation should be discussed apart. Importantly here is that it
enables the deprecation.

>> - It start fixing our very problematic, slow and outdated dependency
>> management: yaml-cpp was included using vcpkg (already used in the MacOS
>> build)
>
>   That's an another reason NOT to use yaml at all.
>

Libraries are bad! Let's then remove unicode, time zones, big decimals,
cryptography, regexp, compression. Let's re-release v2 as v6 :)

I know many of your opinions: no template, no C++, no libraries, no
deprecate of bad and almost unused tools, etc.

With resistance, we deprecated GPRE DDL, GDEF, QLI, and *nobody* complained.

My main worry would be that some day Firebird server/engine is
"deprecated" and nobody worries.

With totally unpleasant and unproductive tools, bad development
experience, and consequentially bad tools for users, that day gets
closer and closer.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 22.10.2023 15:44:
>>   That's wrong. To compile queries to system tables (Firebird sources
>> has no others) any engine is not needed at all because system tables are
>> defined by headers jrd/relations.h, fields.h, etc.
>>
> That is gpre_boot, a special version.

Is there any reason why it is a special version and not a feature of standard
GPRE?

> Didn't know this tools has users today.

Now you know.

>> That's the reason to change the code generator, that's all. Much
>> simpler task that creating a new one from scratch.
> Feel free to start a counter proposal and implement it.

Proposal for what? Something that nobody needs because is satisfied with
current status quo?

>> The last thing any project needed is an another programming language
>> to study. As I already said GPRE supports SQL via "EXEC SQL".
>
> Let's remove automated builds if yaml is a programming language.

Good idea. At least you wouldn't wonder about your previous discussion with
Vlad who said that nobody but you understand anything in this crap.

--
WBR, SD.

[Adriano dos Santos Fernandes]

On 22/10/2023 10:51, 'Dimitry Sibiryakov' via firebird-devel wrote:
>> Let's remove automated builds if yaml is a programming language.
>
>   Good idea. At least you wouldn't wonder about your previous discussion
> with Vlad who said that nobody but you understand anything in this crap.

Let's see it from another perspective: it's GitHub Actions, an
established, free service, widely used and documented product. And
completely defined in the Firebird source code and that anyone can learn
and change for good. It also started gradually, with builds only for
PRs, in different environments, replacing outdated services (swallowed
by GH Actions superiority), then for snapshots and now releases. A
process that took years.

Versus a system that previously depended on two people, their machines,
their knowledge and that nobody else may do.

But anyway I understand it can upset who previously do "certified
builds" if "certified builds" is only building with information that was
not publicly or easily available, as now any user can do a "certified"
build.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 22.10.2023 16:12:
> Let's see it from another perspective: it's GitHub Actions, an established, free
> service, widely used and documented product.

Right, it is MS choice and they force everybody else to use it for Github's
CI. They have rights to do so as owners of the Github.
But Firebird is a separate project and while it has to use YAML for CI there
is no reason to draw it into anything else.

> anyone can learn and change for good.

Yes, that's my point. One have to learn YAML to change CI settings. That's
fine because we have no choice in this regard and these settings are rarely need
changes so everyone can push this task to someone else.
But learning it to work on Firebird code itself is different. It is an
unnecessary complication of everybody's work.


--
WBR, SD.

[Adriano dos Santos Fernandes]

Em dom., 22 de out. de 2023 12:16, 'Dimitry Sibiryakov' via firebird-devel <firebir...@googlegroups.com> escreveu:

Adriano dos Santos Fernandes wrote 22.10.2023 16:12:
> Let's see it from another perspective: it's GitHub Actions, an established, free
> service, widely used and documented product.

   Right, it is MS choice and they force everybody else to use it for Github's
CI. They have rights to do so as owners of the Github.

They force who and how?

   But Firebird is a separate project and while it has to use YAML for CI there
is no reason to draw it into anything else.

?

Almost all ci system uses yaml as nobody need to "learn" yaml really.

> anyone can learn and change for good.

   Yes, that's my point. One have to learn YAML to change CI settings. 

Learn yaml?

Usage of yaml is straightforward and nobody talks about advanced feature or master its specification.

Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 22.10.2023 17:48:
>>    Right, it is MS choice and they force everybody else to use it for Github's
>> CI. They have rights to do so as owners of the Github.
>
> They force who and how?

Isn't answer "everybody who would like to use CI they offer by not offering
any other way to use it" obvious...?

> Almost all ci system uses yaml as nobody need to "learn" yaml really.

You missed "who already studied this area" after "nobody".

--
WBR, SD.

[Jim Starkey]

Adriano, I don't particularly care about how you feel about my code or
architectures, but it is a shame that your mother didn't teach you more
about civility in your public comments.

Guys, there were perfectly good reasons that internal engine modules
depended on pre-processed modules for system table access, but they all
have to do with a) machines that had about .1% processing power as
today's machines, b) machines that had about .001% the memory as modern
machines, and c) an architecture where a client typically ran with the
engine linked in and had to pay the overhead of engine startup every
time it ran.

I submit, gentlemen, that you spend less time pissing on on 39 year old
code and spend a little more time thinking about what makes sense
architecturally in 2023 and going forward.  Hint: Adriano's proposed
architecture is just as bad as his manners, possibly even worse.

What you need is a squeaky clean OO API can be used internally or, with
necessary and appropriate extensions to the protocol(s), externally as well.

I'd be perfectly happy to give you a copy of the Amorphous API which, in
fact, is a generic database interface and not tied into AmorphousDB, per
se.  In Amorphous, the API has an internal implementation, a remote
client implementation with the server layered on the internal, a native
Java remote client, and a JNI implementation for running Java code in
server context.  I'm sure it wouldn't meet what Adriano's perception of
Adriano's high standards, but it might be something the more enlightened
members of the project might find interesting, even if only in an
academic sense.

Of course, if Adriano spent 5% of the time that he spends pissing and
moaning about my code thinking about an appropriate architecture for
internal requests, the problem would have been solved 10 or 20 years ago.

--
Jim Starkey, AmorphousDB, LLC

[Adriano dos Santos Fernandes]

On 22/10/2023 13:26, Jim Starkey wrote:
> Adriano, I don't particularly care about how you feel about my code or
> architectures, but it is a shame that your mother didn't teach you more
> about civility in your public comments.
>
> Guys, there were perfectly good reasons that internal engine modules
> depended on pre-processed modules for system table access, but they all
> have to do with a) machines that had about .1% processing power as
> today's machines, b) machines that had about .001% the memory as modern
> machines, and c) an architecture where a client typically ran with the
> engine linked in and had to pay the overhead of engine startup every
> time it ran.
>

I have no idea what you are saying about me telling about your things.

Since I started working with Firebird code, more than 20 years ago,
probably there isn't any commit of you. I suppose any code of you
entered in Firebird code was some Vulcan code that list files from a
directory and that I get from it and added to Firebird.

If I criticize Firebird code or architecture, it's about *some* of 30 or
more years decisions and implementations are still there. Note that some
of these things are very good, some good, some bad, and some very bad
for today.

So be it less hysteric. It's not about you, it's about Firebird code,
that derived from some code and decisions you made a lot of time ago,
sold it, has passed by many and many peoples even before started as
Firebird. It's about any codebase that need to change for better over
days, months, years.


Adriano

[Pavel Cisar]

Dne 22. 10. 23 v 19:07 Adriano dos Santos Fernandes napsal(a):

>
> I have no idea what you are saying about me telling about your things.
>
> Since I started working with Firebird code, more than 20 years ago,
> probably there isn't any commit of you. I suppose any code of you
> entered in Firebird code was some Vulcan code that list files from a
> directory and that I get from it and added to Firebird.
>
> If I criticize Firebird code or architecture, it's about *some* of 30 or
> more years decisions and implementations are still there. Note that some
> of these things are very good, some good, some bad, and some very bad
> for today.

Adriano, you obviously forgot that Jim designed and implemented the
InterBase, which eventually became Firebird. So you actually DO bitch
about Jim's code and design.

> So be it less hysteric. It's not about you, it's about Firebird code,
> that derived from some code and decisions you made a lot of time ago,
> sold it, has passed by many and many peoples even before started as
> Firebird. It's about any codebase that need to change for better over
> days, months, years.

No offense, but you certainly have issues to understand the meaning of
written text sometimes. Or are you just quick and trigger-happy?

P.S. to Jim. Nice to see you're still around, and that you care about
what's going on.

best regards
Pavel

[Dmitry Yemanov]

22.10.2023 15:21, Adriano dos Santos Fernandes wrote:

> I've started this topic showing the problems we should solve IMO:
> - GPRE itself needs a fake engine, so we have gpre_boot.

This may be weird for newcomers but is this *really* a problem?

> - The written by user code uses GDML, that is not maintained.
> - GDML is not powerful as SQL.

True. But that's problems of GDML which surely shows its age. I agree we
should be using SQL instead, but there may be different ways to achieve
that goal.

> - GPRE code is totally crap.

Maybe, I only once added something new there. Doable, but hard to
understand at the first glance.

> - We write huge epp files with many C++ code not related directly with
> the queries. IDE experience is crap with these epp files.

Having queries together with code is exactly the purpose of Embedded SQL
;-) Personally, I find it easier to understand things when I see the
queries. And changing things in one file is easier than modifying
queries and logic in different files (think e.g. about the DSQL
refactoring you did that had the same issue before and now became handier).

> - The generated code is crap to debug.

IMHO more inconvenient rather than unusable.

> - Allow us to deprecate GPRE right now, and remove it in v6

Please no.


Dmitry

[Mark Rotteveel]

On 22-10-2023 18:26, Jim Starkey wrote:
> Adriano, I don't particularly care about how you feel about my code or
> architectures, but it is a shame that your mother didn't teach you more
> about civility in your public comments.

[..]

> Of course, if Adriano spent 5% of the time that he spends pissing and
> moaning about my code thinking about an appropriate architecture for
> internal requests, the problem would have been solved 10 or 20 years ago.

First off, I don't see where or how Adriano is being uncivil, but even
so, the comment about his mother is uncalled for.

However, more importantly, as a moderator of this list I want to remind
everyone here to stop taking things personal and making things personal.
Lets keep in mind here that we're all working together to make Firebird
better, and we all have our own temperaments and discussion styles.

Mark
--
Mark Rotteveel

[Dmitry Yemanov]

22.10.2023 16:44, Adriano dos Santos Fernandes wrote:
>
>> If you miss something - GPRE
>> supports SQL as well. It is called "ESQL".
>
> It does not work for JRD's system queries and is far from currently SQL
> dialect. I'd say it's v1.5 or v1.0 SQL dialect almost.

I'd consider improving GPRE in this regard. Maybe even rewriting it, if
you wish. That would be useful for both the project and our users.

>>   That's the reason to change the code generator, that's all. Much
>> simpler task that creating a new one from scratch.
>
> Feel free to start a counter proposal and implement it.

Do you have any arguments against it or just don't want to do that yourself?

>>   If you are talking about customer supports and including GPRE into
>> distribution packages - it is not related to internal build process at
>> all and can be done any time. But it hasn't been done yet for a reason:
>> some valuable customers still use ESQL.
>
> I would think very fondly if they are in this list:
> https://firebirdsql.org/en/sponsors/

I can see at least one of them there.

> But deprecation should be discussed apart. Importantly here is that it
> enables the deprecation.

That's surely more reasonable approach than just deprecating it
immediately ;-)

> With totally unpleasant and unproductive tools, bad development
> experience, and consequentially bad tools for users, that day gets
> closer and closer.

This is too much exaggerated, IMHO.


Dmitry

[Dmitry Yemanov]

21.10.2023 23:32, Adriano dos Santos Fernandes wrote:
>
> Now the problem: I think that since v2.5, our GDML queries can access
> nonexistent system fields in runtime, treating them as NULL in read and
> nop in write. That simplified a lot handling of multiple ODS and in
> ISQL, make it faster as there is no need to run multiple queries to get
> additional fields.
>
> With SQL that is going to be difficult. I'd hate to return to that
> multiple queries approach.

Me too, but could you please explain why it's a problem with SQL. GDML
vs SQL are just different grammar frontends, but the code generator
should be mostly the same. I'm supposedly missing something.


Dmitry

[Vlad Khorsun]

23.10.2023 11:29, 'Mark Rotteveel' via firebird-devel:

> On 22-10-2023 18:26, Jim Starkey wrote:
>> Adriano, I don't particularly care about how you feel about my code or architectures, but it is a shame that your mother didn't
>> teach you more about civility in your public comments.
> [..]
>> Of course, if Adriano spent 5% of the time that he spends pissing and moaning about my code thinking about an appropriate
>> architecture for internal requests, the problem would have been solved 10 or 20 years ago.
>
> First off, I don't see where or how Adriano is being uncivil, but even so, the comment about his mother is uncalled for.

Just count number of 'crap' and consider Jim answer as very-very soft and polite.

Regards,
Vlad

[Vlad Khorsun]

23.10.2023 10:57, Pavel Cisar:

+1

Regards,
Vlad

[Mark Rotteveel]

On 23-10-2023 11:00, Vlad Khorsun wrote:
> 23.10.2023 11:29, 'Mark Rotteveel' via firebird-devel:

>> First off, I don't see where or how Adriano is being uncivil, but even
>> so, the comment about his mother is uncalled for.
>
>   Just count number of 'crap' and consider Jim answer as very-very soft
> and polite.

I don't think using "crap" as an expression of opinion on code quality
or usability of code is uncivil, but I guess everyone has their own
level of tolerance. Personally, I find ad hominems (or should I say, ad
matriams) worse.

To be honest, I share the opinion that the experience with *.epp files
and their processed *.cpp counterparts is sub-optimal, and I don't even
write code in the engine, but every time I open such a file in Visual
Studio and see the lack of syntax highlighting in *.epp, or the amount
of hard to read generated code in their *.cpp counterparts, I recoil a bit.

It might have been a great solution back when it was created, but it
feels outdated now and I think there are probably better solutions out
there or possible.

Mark
--
Mark Rotteveel

[Vlad Khorsun]

23.10.2023 12:17, 'Mark Rotteveel' via firebird-devel:
...

> To be honest, I share the opinion that the experience with *.epp files and their processed *.cpp counterparts is sub-optimal, and I
> don't even write code in the engine, but every time I open such a file in Visual Studio and see the lack of syntax highlighting in
> *.epp, or the amount of hard to read generated code in their *.cpp counterparts, I recoil a bit.

Add epp as "C++" at text editor settings (options\text editor\file extensions)

> It might have been a great solution back when it was created, but it feels outdated now and I think there are probably better
> solutions out there or possible.

There are so many things that could be improved and "epp problem" is somewhere at
the very end of list, IMHO.

Regards,
Vlad

PS "sub-optimal" is much more suitable than "crap", isn't is ?

[Mark Rotteveel]

On 23-10-2023 11:32, Vlad Khorsun wrote:
> 23.10.2023 12:17, 'Mark Rotteveel' via firebird-devel:

[..]

>   Add epp as "C++" at text editor settings (options\text editor\file
> extensions)

Thanks, I knew there was something for it, but never bothered to look it
up (as I don't generally look too much at *.epp anyway).

>> It might have been a great solution back when it was created, but it
>> feels outdated now and I think there are probably better solutions out
>> there or possible.
>
>   There are so many things that could be improved and "epp problem" is
> somewhere at
> the very end of list, IMHO.

Sure, but this is an open-source project, and every contributor has
their own priorities, pet-peeves or things they'd like to change.

> PS "sub-optimal" is much more suitable than "crap", isn't is ?

Maybe, but as I said, everyone is different, and in some cultures, using
swearwords or coarse language is pretty normal. And some people can be
totally OK with use of coarse language, while get in a fit of rage when
words like "sub-optimal" are used in a critique.

Mark
--
Mark Rotteveel

[Artyom Abakumov]

The discussion is getting pretty heated, isn't it? You all have a lot of experience with code, over 20 years, and it's understandable why you all have a different view. But let me describe the interaction with GPRE from the perspective of a newbie trying to counterpunch into code for the first time.

So, you find the code to change, explore the file. And then you see a huge clutter of structs and a weird array of constants with a blr prefix. You find where it is used and see a comment. At the time, I thought this was a SQL query. And the first thing I thought was to change this blr in the cpp file itself. Then you realize that there is an epp file, there is GPRE, which you have heard about for the first time in your life, and for which there is hardly any documentation.

And my God, I can't even count how many times I changed the code, only to realize later that I had screwed up because I changed a cpp file instead of the epp one and all my changes were erased. And editing an epp file is a pain. The compiler points to the cpp file, and you have to find the same line in the epp one. At the same time, all the breakpoints are screwed up. All this makes it 100 times more difficult for beginners like me.

понедельник, 23 октября 2023 г. в 12:41:35 UTC+3, Mark Rotteveel:

[Dimitry Sibiryakov]

'Mark Rotteveel' via firebird-devel wrote 23.10.2023 11:41:
> Sure, but this is an open-source project, and every contributor has their own
> priorities, pet-peeves or things they'd like to change.

But also it is not a personal project so changing thing in a way that others
consider worsening is not a good idea.

Sometimes I do it myself and that's why it is good that my changes are
pre-moderated.
But Adriano has rights for direct commits so when he decided that instead of
rethinking OO API because of a bug in FPC/Delphi whole CLOOP things must be
implemented - there was nobody to stop him so now we have another code generator
that gets some variation of IDL and produce strange code which is hard to debug.
I personally don't want to get third one - this time from YAML.

--
WBR, SD.

[Adriano dos Santos Fernandes]

On 23/10/2023 06:32, Vlad Khorsun wrote:
>
> PS "sub-optimal" is much more suitable than "crap", isn't is ?
>

Should we call the internet police to look at Vlad's and Jim's words
around the time?


Adriano

PS: Jim's was not answered as I liked, you know. I did that in respect
to this list. Maybe sometime if he come back to Brazil, I can answer it
in person.

[Mark Rotteveel]

On 23-10-2023 12:25, 'Dimitry Sibiryakov' via firebird-devel wrote:
> 'Mark Rotteveel' via firebird-devel wrote 23.10.2023 11:41:
>> Sure, but this is an open-source project, and every contributor has
>> their own priorities, pet-peeves or things they'd like to change.
>
>   But also it is not a personal project so changing thing in a way that
> others consider worsening is not a good idea.

I think that is what this discussion is for.

>   Sometimes I do it myself and that's why it is good that my changes
> are pre-moderated.
>   But Adriano has rights for direct commits so when he decided that
> instead of rethinking OO API because of a bug in FPC/Delphi whole CLOOP
> things must be implemented - there was nobody to stop him so now we have
> another code generator that gets some variation of IDL and produce
> strange code which is hard to debug.

To be honest, I don't recall the exact details from back then, and I
don't want to dig into my archive to confirm, but I believe that change
was discussed before it was integrated.

Mark
--
Mark Rotteveel

[Adriano dos Santos Fernandes]

On 23/10/2023 07:06, Artyom Abakumov wrote:
> And my God, I can't even count how many times I changed the code, only
> to realize later that I had screwed up because I changed a cpp file
> instead of the epp one and all my changes were erased. And editing an
> epp file is a pain. The compiler points to the cpp file, and you have to
> find the same line in the epp one. At the same time, all the breakpoints
> are screwed up. All this makes it 100 times more difficult for beginners
> like me.

This happened to me as well many times.

And it's not the first time I try to improve it.

Other time, Firebird developers was against IIRC, even to issue "#line"
directives in the generated file so compile error line numbers and debug
matches the original epp code.

While I have implemented it, I gave up. It would make this situation of
edit the generated cpp file much more common, so it's better to be
difficult here.


Adriano

PS: Would like to know why people think work with Firebird code is so
different than modern development in general and don't want to change.

Embedded SQL is ditched by so many time in the world, but not in
Firebird even with a not ideal (polite mode on) tool.

Software in general is in many times 100% covered by unit tests, but in
Firebird there is people against unit tests being added to the code.


Adriano

[Adriano dos Santos Fernandes]

On 23/10/2023 07:25, 'Dimitry Sibiryakov' via firebird-devel wrote:
>   But Adriano has rights for direct commits so when he decided that
> instead of rethinking OO API because of a bug in FPC/Delphi whole CLOOP
> things must be implemented - there was nobody to stop him so now we have
> another code generator that gets some variation of IDL and produce
> strange code which is hard to debug.

And I moderate myself opening pull requests when I think is appropriate.

I do the same here describing the idea for this topic as it was changing.

cloop (as well any change) is agreed by the team. (If you were against
it and it's there, the team didn't considered your opinion and
considered others).

I remember pretty well that Alex was an enthusiast of it and we both
changed the old code together. It may not be ideal, but it still serving
well for the purposes it was written.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 23.10.2023 12:38:
> This happened to me as well many times. And it's not the first time I try to
> improve it.

Just remove generated .cpp files from the VS projects so there will be no way
to open them in one click. That's all.

--
WBR, SD.

[Vlad Khorsun]

23.10.2023 13:06, Artyom Abakumov:

While this was a reply to the Mark message, let me answer on it.

> The discussion is getting pretty heated, isn't it? You all have a lot of experience with code, over 20 years, and it's
> understandable why you all have a different view. But let me describe the interaction with GPRE from the perspective of a newbie
> trying to counterpunch into code for the first time.

I also was a newby some time ago. And in that time there was much less
info in the much smaller internet to search for answers.

> So, you find the code to change, explore the file. And then you see a huge clutter of structs and a weird array of constants with a
> blr prefix. You find where it is used and see a comment. At the time, I thought this was a SQL query. And the first thing I thought
> was to change this blr in the cpp file itself. Then you realize that there is an epp file, there is GPRE, which you have heard about
> for the first time in your life, and for which there is hardly any documentation.

This is not true. There is IB6 docs that explains gpre. Also, did you
asked for help in support list, or sql.ru, or here ?

> And my God, I can't even count how many times I changed the code, only to realize later that I had screwed up because I changed a
> cpp file instead of the epp one and all my changes were erased.

Again, why you not ask someone more experienced ?

> And editing an epp file is a pain. The compiler points to the cpp
> file, and you have to find the same line in the epp one.

VS IDE works with epp files as with cpp ones, it allows to see and jump to the
any functions\methods, for example.

> At the same time, all the breakpoints are screwed up. All this makes it 100
> times more difficult for beginners like me.

Do you consider the offered replacement as more clear way for beginners ?
If you had known what is epp and how it is related with cpp at time you begin,
would it have been same pain as you wrote above ?

The point is that tool (gpre) is not as bad as lack of knowledge of how to use it.
And this applicable to any tool, not only to the gpre.

Regards,
Vlad

[Vlad Khorsun]

23.10.2023 13:29, Adriano dos Santos Fernandes:

> On 23/10/2023 06:32, Vlad Khorsun wrote:
>>
>> PS "sub-optimal" is much more suitable than "crap", isn't is ?
>>
>
> Should we call the internet police to look at Vlad's and Jim's words
> around the time?

Feel free (c)

Vlad

[Ann Harrison]

On Sun, Oct 22, 2023 at 1:07 PM Adriano dos Santos Fernandes <adri...@gmail.com> wrote:

Since I started working with Firebird code, more than 20 years ago,

probably there isn't any commit of you [Jim Starkey]. I suppose any code of you

entered in Firebird code was some Vulcan code that list files from a
directory and that I get from it and added to Firebird.

Just in case there are people on the project who aren't familiar with the history  of Firebird,

"jrd" in the code base is short for Jim's Relational Database, and the Jim in question is my 

husband, Jim Starkey.  His name was taken off the code when Borland bought InterBase

and changed the copyright statements, but there are still lots of his commits in the code base.

Even 35 years later, much of the basic architecture and ODS is his design.

Best regards,

Ann

[Jim Starkey]

Actually, for the record, my name isn't on any commit, as commit was a git concept that didn't appear until decades later.

The originally, I didn't use any code management system.  Builds were performed either by Unix "make" on most platforms or my own implementation of "make" for VMS (DEC had a make clone called MMS for which they wanted $2,500 per copy).

As we added engineers this became untenable, so I implemented "Marion" (the librarian) layered on, of course, JRD.  Marion required module checkout to avoid collisions but was otherwise a reasonable code management system (for its day).  I clearly remember the moment that I loaded the Interbase code base into Interbase, wondering whether this was a very good idea.  We also had a test control system, obscurely called TCS, that ran on every development system every night and Test Analyzer (TAN) that collected and displayed all test failures.

We also had a distributed email program written by the support group, again layered on the database.  The Internet was known but still closed to non-government sponsored organization, though eventually I think a connection to UseNet (may it burn in hell) was added.

I could go on about the evolution of lock managers, etc.  But the low point came when I got a call from Apollo's senior architect that we had detected an architecturally unfixable conflict between their brand new threading system and their signal handling system.  This required a major architectural change to product implementation to shift from signal handling to threads which, in turn, required that I write the first VMS threading package.

--
You received this message because you are subscribed to the Google Groups "firebird-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebird-deve...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebird-devel/CALAsn_LDuodBySY6_yQD2uax7oXLptTvHBpA0DB66SW7AyiURA%40mail.gmail.com.

[Mark Rotteveel]

On 23-10-2023 16:51, Jim Starkey wrote:
> Actually, for the record, my name isn't on any commit, as commit was a
> git concept that didn't appear until decades later.

If we're going to pedantic, commit is not a git concept, it predates git
by a few decades in versioning systems like RCS, CVS, Subversion and others.

Mark
--
Mark Rotteveel

[Adriano dos Santos Fernandes]

Don't you see a problem if users writes:

select non_existent_field from rdb$database

And Firebird return NULL?

It may be a flag, but do you want a flag that allow this?

It's the same with GDML, but it need a initial database with the field
so the source can be compiled and BLR be generated.


Adriano

[Adriano dos Santos Fernandes]

On 23/10/2023 05:23, Dmitry Yemanov wrote:
>> - The written by user code uses GDML, that is not maintained.
>> - GDML is not powerful as SQL.
>
> True. But that's problems of GDML which surely shows its age. I agree we
> should be using SQL instead, but there may be different ways to achieve
> that goal.
>

Nobody proposed different ways in the last 20 years.

>> - GPRE code is totally crap.
>
> Maybe, I only once added something new there. Doable, but hard to
> understand at the first glance.
>

If you tell about GDML, it's recreation of Firebird DSQL in different
way in worse codebase.

If you tell about SQL, SQL approach as present in GPRE is incompatible
with usage of SQL in engine without additional pre-built engine. We
don't want to go back to this time when to build Firebird, a previous
Firebird should be running.

So it's doable to add a system function to it. More advanced and/or
complete things, it's not.

>> - We write huge epp files with many C++ code not related directly with
>> the queries. IDE experience is crap with these epp files.
>
> Having queries together with code is exactly the purpose of Embedded SQL

Of course.

But it was never stated that Firebird needs embedded SQL. For me it was
just an implementation detail.

If you change perspective and think that the queries are a service
(sometimes even reusable) and the other code is someone needing that
service, it totally makes sense to have them in separate places.

And this is very important factor to write internally testable software.
But not an important argument when developers do not bother with this
and thinks only integration and slow tests matters.

So summarizing:
- We can't expect people should understand a new "programming language"
(yaml)
- We want embedded SQL in C++
- We want users to still have GPRE, even it being an unmaintained tool

We want more things?

We want GPRE boot? Everybody is happy with what we have today?


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 24.10.2023 13:10:
> So it's doable to add a system function to it. More advanced and/or complete
> things, it's not.

IMHO, it is doable to teach DSQL use static definitions for system tables
instead of actual metadata.
It is doable to make DSQL nodes to generate BLR (they do it anyway).
So it is doable for GPRE to call the parser and write into output file
whatever BLR has been generated by DSQL.

> We want GPRE boot? Everybody is happy with what we have today?

As I already said: using static definitions of system tables in detached mode
must be a feature of standard GPRE, separate boot build is not needed.

--
WBR, SD.

[Adriano dos Santos Fernandes]

On 24/10/2023 08:18, 'Dimitry Sibiryakov' via firebird-devel wrote:
> Adriano dos Santos Fernandes wrote 24.10.2023 13:10:
>> So it's doable to add a system function to it. More advanced and/or
>> complete things, it's not.
>
>   IMHO, it is doable to teach DSQL use static definitions for system
> tables instead of actual metadata.
>   It is doable to make DSQL nodes to generate BLR (they do it anyway).
>   So it is doable for GPRE to call the parser and write into output file
> whatever BLR has been generated by DSQL.
>

It may be possible (not easy) to build DSQL, accepting only system
metadata, inside a client tool that do not interact with the engine.
Correct and it may be a piece to compile SQL to BLR and be faster and
avoid problems with non-existent system fields.

But it make no sense to integrate it with GPRE as it is. It will not
solve problem of how bad are the generated files to debug and to
understand compile errors.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 24.10.2023 13:35:
> But it make no sense to integrate it with GPRE as it is. It will not solve
> problem of how bad are the generated files to debug and to understand compile
> errors.

I see no difference in generated files regarding debugging between GPRE and
CLOOP so both of them has a huge room for improvements. Still doable.

--
WBR, SD.

[Alex Peshkoff]

On 10/24/23 14:10, Adriano dos Santos Fernandes wrote:
> On 23/10/2023 05:23, Dmitry Yemanov wrote:
>>> - The written by user code uses GDML, that is not maintained.
>>> - GDML is not powerful as SQL.
>> True. But that's problems of GDML which surely shows its age. I agree we
>> should be using SQL instead, but there may be different ways to achieve
>> that goal.
>>
> Nobody proposed different ways in the last 20 years.

May be because current state satisfies (more or less) most of us?
As for me I do not see GDML as bad as you, moreover - it's enough for
99% of what we need to access system tables in engine and utilities.
It's syntax certainly differs from SQL - but in some aspects it's even
better (compare "tableA cross tableB over CommonField" with ugly join
syntax). When SQL power is really needed we (thanks to Adriano!) have a
way to do it.
Need of gpre_boot is not a problem at all - it's small piece of code
(single file as far as I remember), getting DB metadata from tables
anyway needed in ini.epp.
Code, written in almost K&R dialect of C, certainly looks a bit strange
today. But taking into an account that it remains useful for so many
years - can it be called ugly? IMO not.

>
>>> - GPRE code is totally crap.
>> Maybe, I only once added something new there. Doable, but hard to
>> understand at the first glance.
>>
> If you tell about GDML, it's recreation of Firebird DSQL in different
> way in worse codebase.

Historically DSQL is recreation of GDML :D

> If you tell about SQL, SQL approach as present in GPRE is incompatible
> with usage of SQL in engine without additional pre-built engine. We
> don't want to go back to this time when to build Firebird, a previous
> Firebird should be running.
>
> So it's doable to add a system function to it. More advanced and/or
> complete things, it's not.
>
>
>>> - We write huge epp files with many C++ code not related directly with
>>> the queries. IDE experience is crap with these epp files.
>> Having queries together with code is exactly the purpose of Embedded SQL
> Of course.
>
> But it was never stated that Firebird needs embedded SQL. For me it was
> just an implementation detail.
>
> If you change perspective and think that the queries are a service
> (sometimes even reusable) and the other code is someone needing that
> service, it totally makes sense to have them in separate places.

Too many beautiful words. As for me having GDML/SQL in same file is very
convenient.

> And this is very important factor to write internally testable software.

Ability to test software depends upon presence of embedded database
access statements in C++ code? On my mind that means problems with
approach to tests.

> But not an important argument when developers do not bother with this
> and thinks only integration and slow tests matters.
>
> So summarizing:
> - We can't expect people should understand a new "programming language"
> (yaml)

not a problem for me but if others object...

> - We want embedded SQL in C++

Yes.

> - We want users to still have GPRE, even it being an unmaintained tool

COBOL is also still supported language, and yes - as long as people keep
using it they will definitely prefer ummaitained, not missing.

> We want more things?
>
> We want GPRE boot?

Still no idea why you so much dislike it.

> Everybody is happy with what we have today?

Certainly not, but if first suggestion with preprocessor putting it's
result into .h file was something to talk about current suggestion
appears to me very wet. Sorry.

[Jim Starkey]

It revolves around the semantics of the word "commit" which, as a
database guy, you should be acutely sensitive to.   Some code management
systems like Marion and CVS supported a check out operation to prevent
collisions.  All code management systems support a check in operation. 
Both CVS and git use the term "commit" to check in modules, but the
semantics are radically different.

CVS manages modules but not relationships between modules.  CVS
"commits" and "updates" are distinctly non-atomic.  It was always
possible to update a directory with some but not all modules for a
single feature change.  Not good.

Git manages atomic transactions of related module changes.  In a real
sense, git is an MVCC module database.  Not only are commits atomic, but
so are pushes to a repository.

I believe I uses CVS for Vulcan.  So, yeah, there were hundreds of CVS
"commits", but no commit is the git -- or database -- sense.

I used the term "commit" in the sense that I presumed Adriano did.  And
since Vulcan had a separate CVS repository, none of my CVS "commit"
comments would have made it into Firebird even though much -- or most --
of the Vulcan code eventually found its way into the Firebird mainstream
(which was fine with me for a number of reasons).

If you're lucky, I won't explain my views on open source development and
why I find watching Firebird instructive.

Jim Starkey, AmorphousDB, LLC

[James Starkey]

Of course it was K&R.  That was all that existed in 1984.  Even after C89/C90 was official, Sun supported function prototypes, etc., only in their expensive, unbundled compiler; the freebie compiler was basically K&R.  Like almost everyone else, we developed a set of macros that could expand with or without functional prototypes but only at the cost of even more ugliness.

--
You received this message because you are subscribed to the Google Groups "firebird-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebird-deve...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebird-devel/c4a99441-f644-4cbf-b3ff-36dca20711db%40gmail.com.

--

Jim Starkey

[Adriano dos Santos Fernandes]

On 24/10/2023 09:38, Alex Peshkoff wrote:
> On 10/24/23 14:10, Adriano dos Santos Fernandes wrote:
>> On 23/10/2023 05:23, Dmitry Yemanov wrote:
>>>> - The written by user code uses GDML, that is not maintained.
>>>> - GDML is not powerful as SQL.
>>> True. But that's problems of GDML which surely shows its age. I agree we
>>> should be using SQL instead, but there may be different ways to achieve
>>> that goal.
>>>
>> Nobody proposed different ways in the last 20 years.
>
> May be because current state satisfies (more or less) most of us?
> As for me I do not see GDML as bad as you, moreover - it's enough for
> 99% of what we need to access system tables in engine and utilities.

It's does not cover well 99% of cases for sure.

There is workaround and workarounds.

In the client, that make very slow tools over the internet.

And we don't want different tools and tools with pure dynamic SQL and
untyped code for the other cases, AFAIR.

>> If you change perspective and think that the queries are a service
>> (sometimes even reusable) and the other code is someone needing that
>> service, it totally makes sense to have them in separate places.
>
> Too many beautiful words. As for me having GDML/SQL in same file is very
> convenient.
>

For sure Mark and Jiri knows what I'm talking.

Testable software, where components can be replaced in unit tests.

>> And this is very important factor to write internally testable software.
>
> Ability to test software depends upon presence of embedded database
> access statements in C++ code? On my mind that means problems with
> approach to tests.
>

Another funny case. In the past I said we need to remove the viral
"tdbb" from every code in Firebird, passing proper arguments like
attachment, request, transaction and so on. People here complained.

Now Dimitry says that it's easy to create standalone tool with parser
not touching the engine.

And also there is talk that we want to maintain self-contained nodes, in
another topic, for example, not replacing parts with visitors (say,
separate execution from compile).

Then tell me how not adaptable code as we have, due to people's not
wanting then to be, can be reused like this.

>> But not an important argument when developers do not bother with this
>> and thinks only integration and slow tests matters.
>>
>> So summarizing:
>> - We can't expect people should understand a new "programming language"
>> (yaml)
>
> not a problem for me but if others object...
>
>> - We want embedded SQL in C++
>
> Yes.
>

I asked because while I don't think it's the better approach, it would
be possible to go back to the idea of the preprocessor code, generating
inline messages, i.e., mixing parts of first idea with the latest one.

But I don't think this is going anywhere when people is attached to old
ideas and are rigid against changes.

PS: when building the code today in Windows:

..\..\..\gen\burp\restore.cpp(15032,9): warning C4101: 'len':
unreferenced local variable [C:\...\builds\win32\msvc15\burp.vcxproj]
..\..\..\gen\burp\restore.cpp(15031,7): warning C4101: 'temp':
unreferenced local variable [C:\...\builds\win32\msvc15\burp.vcxproj]
..\..\..\gen\burp\restore.cpp(15165,9): warning C4101: 'len':
unreferenced local variable [C:\...\builds\win32\msvc15\burp.vcxproj]
..\..\..\gen\burp\restore.cpp(15164,7): warning C4101: 'temp':
unreferenced local variable [C:\...\builds\win32\msvc15\burp.vcxproj]

I hope nobody double clicks in the errors to fix them.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 26.10.2023 2:23:
> In the client, that make very slow tools over the internet.

No, it is not the reason. DSQL won't be faster if it mindlessly take the
place of GDML, only slower.
In many cases ISQL sends much more queries than necessary. Before my cleanup
SHOW command requested character set data for the same field three times in row.
Now it is requested only once but still for each field individually so there is
still a room for improvement.

>> Too many beautiful words. As for me having GDML/SQL in same file is very
>> convenient.
>
> For sure Mark and Jiri knows what I'm talking.

Haven't seen any commit from them to Firebird for ages. Do they still write
any code?

> But I don't think this is going anywhere when people is attached to old
> ideas and are rigid against changes.

Yes if new ideas are worse than these old ones. This topic is an excellent
example of bargaining one disaster for another.

--
WBR, SD.

[Mark Rotteveel]

On 26-10-2023 11:30, 'Dimitry Sibiryakov' via firebird-devel wrote:
> Adriano dos Santos Fernandes wrote 26.10.2023 2:23:
>>> Too many beautiful words. As for me having GDML/SQL in same file is very
>>> convenient.
>>
>> For sure Mark and Jiri knows what I'm talking.
>
>   Haven't seen any commit from them to Firebird for ages. Do they still
> write any code?

I think Adriano was referring to _"Testable software, where components
can be replaced in unit tests."_, as that is the normal and accepted way
of working in the C# and Java world. Though to be honest, in Jaybird, a
lot of tests are not actually unit tests, but integration tests against
the database, because there is a trade-off in complexity and
testability, so some things in Jaybird are unit tests, and others are
integration tests.

That said, the lack of (unit) tests in the main Firebird codebase, and
the reliance on only a separate test project, with only tests written by
someone else never ceases to amaze me.

Mark
--
Mark Rotteveel

[Dimitry Sibiryakov]

'Mark Rotteveel' via firebird-devel wrote 26.10.2023 13:24:
> That said, the lack of (unit) tests in the main Firebird codebase, and the
> reliance on only a separate test project, with only tests written by someone
> else never ceases to amaze me.

As I already said: unit-tests (actually any tests) has to be written by
someone else because they must find situations which code writer didn't think
about. And if they didn't think about such things they cannot write tests to
check for situations they cannot imagine. Because of that TheDailyWTF is full of
examples of bad unit tests.
AFAICS even Adriano himself only created test projects but didn't write any
actual test. Even in his own CLOOP (which is an obvious candidate for test
covering) from time to time demonstrate some bugs that has to be fixed by
someone else.

And getting back to the topic: I still don't see how creation of another
preprocessor and extraction of every query from code to separate files can
improve testing, stability or whatever.

--
WBR, SD.

[Mark Rotteveel]

On 26-10-2023 13:49, 'Dimitry Sibiryakov' via firebird-devel wrote:
> 'Mark Rotteveel' via firebird-devel wrote 26.10.2023 13:24:
>> That said, the lack of (unit) tests in the main Firebird codebase, and
>> the reliance on only a separate test project, with only tests written
>> by someone else never ceases to amaze me.
>
>   As I already said: unit-tests (actually any tests) has to be written
> by someone else because they must find situations which code writer
> didn't think about. And if they didn't think about such things they
> cannot write tests to check for situations they cannot imagine. Because
> of that TheDailyWTF is full of examples of bad unit tests.

No, unit tests are used to check, verify and inform your own work, and
prevent regressions, and if you're using TDD (Test-Driven Development),
you'll even write tests before you write the code that is verified by a
test (that is done in an iterative manner, switching between writing
tests and writing main code).

That doesn't mean that someone else can't write (unit/integration) tests
*as well*, but it is pretty alien to me to not be writing any tests
yourself while developing something.

Mark
--
Mark Rotteveel

[Omacht András]

+1000

[Dimitry Sibiryakov]

'Mark Rotteveel' via firebird-devel wrote 26.10.2023 13:57:
> That doesn't mean that someone else can't write (unit/integration) tests *as
> well*, but it is pretty alien to me to not be writing any tests yourself while
> developing something.

I personally have no idea how to write tests for anything that is not a
classic function, returning single deterministic result from given parameters
without any side effect and previous set up.
Unfortunately such functions even if exist in code are too much trivial and
never touched after been written once so regression tests has no change to find
any regression.
And even for functions with clear contract (i.e. part of API) tests may be
very complex and expensive. As an example, my test for just two parameters of
WinAPI function CreateFile() must check several hundreds of thousand
combinations of input values. That's too much for functions that are made as we go.

--
WBR, SD.

[Mark Rotteveel]

On 26-10-2023 14:07, 'Dimitry Sibiryakov' via firebird-devel wrote:
> 'Mark Rotteveel' via firebird-devel wrote 26.10.2023 13:57:
>> That doesn't mean that someone else can't write (unit/integration)
>> tests *as well*, but it is pretty alien to me to not be writing any
>> tests yourself while developing something.
>
>   I personally have no idea how to write tests for anything that is not
> a classic function, returning single deterministic result from given
> parameters without any side effect and previous set up.

That is where mocks, stubs and drivers come in, and obviously writing
code that is testable.

>   Unfortunately such functions even if exist in code are too much
> trivial and never touched after been written once so regression tests
> has no change to find any regression.

The point is that unit tests are cheap to run, so if the code never
changes, then fine, the test will happily run green every time, but if
the code is touched (or something else the code depends on changes, like
a dependency/library, or other code in the project) you will catch it
sooner rather than later.

>   And even for functions with clear contract (i.e. part of API) tests
> may be very complex and expensive. As an example, my test for just two
> parameters of WinAPI function CreateFile() must check several hundreds
> of thousand combinations of input values. That's too much for functions
> that are made as we go.

First of all, I think you're exaggerating that you need to generate
hundreds of thousands of combination to test something with only two
input variables.

But even if that were true, having 3 tests of such a function (e.g
(<valid>, <valid>), (<valid>, <invalid>), (<invalid>, <valid)) is better
than having *no* test (hell, even testing only the happy path of
(<valid>, <valid>) would be better than nothing). Will it catch
everything, probably not (unless there is only one equivalence class for
<valid> and one equivalence class for <invalid>).

There is a lot of literature on how you can reduce testcases and still
provide adequate coverage, and tools that allow you to check for
coverage to find missing testcase. Good test frameworks also provide you
ways for generating testcases dynamically, without having to enumerate
them all yourself (though you will need to specify what/how it needs to
generate), though personally I prefer to write or generate them once and
use that as stable input.

In any case, your hyperbolic argument is equivalent to arguing against
seat belts and airbags because they don't prevent all injuries or deaths
in a car-crash, ignoring the fact that they do prevent a lot of injury
and death.

Similarly for tests, just because it is hard to achieve total coverage
(which BTW is an anti-pattern in itself), doesn't mean you shouldn't try
and achieve some coverage and improve it incrementally.

Mark
--
Mark Rotteveel

[Dimitry Sibiryakov]

'Mark Rotteveel' via firebird-devel wrote 26.10.2023 14:31:
> In any case, your hyperbolic argument is equivalent to arguing against seat
> belts and airbags because they don't prevent all injuries or deaths in a
> car-crash, ignoring the fact that they do prevent a lot of injury and death.
>
> Similarly for tests, just because it is hard to achieve total coverage (which
> BTW is an anti-pattern in itself), doesn't mean you shouldn't try and achieve
> some coverage and improve it incrementally.

And this is exactly what our QA does: it covers something now and expanded
with every ticket in tracker.
I would be glad if it become a part of our CI process and alarm about
breaking commits. Unfortunately full run is taking several hours.

--
WBR, SD.

[Mark Rotteveel]

It is a good *second* layer, but what is absent right now is a *first*
layer. Developer-written unit tests that are part of the project code
help with verifying your own assumptions when writing code, help you
inform the design and catch things early, and obviously they prevent
(some forms of) regression from passing the build itself (if test are
run as part of the build).

Mark
--
Mark Rotteveel

[Adriano dos Santos Fernandes]

Em qui., 26 de out. de 2023 09:07, 'Dimitry Sibiryakov' via firebird-devel <firebir...@googlegroups.com> escreveu:

'Mark Rotteveel' via firebird-devel wrote 26.10.2023 13:57:
> That doesn't mean that someone else can't write (unit/integration) tests *as
> well*, but it is pretty alien to me to not be writing any tests yourself while
> developing something.

   I personally have no idea how to write tests for anything that is not a
classic function, returning single deterministic result from given parameters
without any side effect and previous set up

Good point.

It's easy to be complaining about everything in the internet than study.

Adriano

[Adriano dos Santos Fernandes]

Another good point.

That's how you break the code for days recently and fixed with subsequently confused changes.

Adriano

[Dimitry Sibiryakov]

'Mark Rotteveel' via firebird-devel wrote 26.10.2023 14:41:
> It is a good *second* layer, but what is absent right now is a *first* layer.
> Developer-written unit tests that are part of the project code help with
> verifying your own assumptions when writing code, help you inform the design and
> catch things early, and obviously they prevent (some forms of) regression from
> passing the build itself (if test are run as part of the build).

That's why I suggest Adriano to spend some time to writing tests for already
existing functions instead of developing new ones.
If such things were done before I would probably don't need to finish
incomplete DEFAULT COLLATION feature.

--
WBR, SD.

[Dimitry Sibiryakov]

'Mark Rotteveel' via firebird-devel wrote 26.10.2023 13:24:
> Though to be honest, in Jaybird, a lot of tests are not actually unit tests, but
> integration tests against the database, because there is a trade-off in
> complexity and testability, so some things in Jaybird are unit tests, and others
> are integration tests.

BTW, do you ever seen an automated test suite that can check any JDBC driver
for conformance to JDBC specs?
I tried to find one for ODBC but it looks like nobody managed to create one
so far.

--
WBR, SD.

[Mark Rotteveel]

On 26-10-2023 15:24, 'Dimitry Sibiryakov' via firebird-devel wrote:
> 'Mark Rotteveel' via firebird-devel wrote 26.10.2023 13:24:
>> Though to be honest, in Jaybird, a lot of tests are not actually unit
>> tests, but integration tests against the database, because there is a
>> trade-off in complexity and testability, so some things in Jaybird are
>> unit tests, and others are integration tests.
>
>   BTW, do you ever seen an automated test suite that can check any JDBC
> driver for conformance to JDBC specs?

Well, there is one, but you either need to pay Oracle to get access, or
apply for a "scholarship" with a lot of hoops to jump through to get
access for free. I considered doing that a few years back, but I gave up
when trying to wade through Oracle's legalese for applying (not to
mention that I couldn't get a straight answer exactly which TCK I needed
to apply for...).

>   I tried to find one for ODBC but it looks like nobody managed to
> create one so far.

If one exist, it is probably managed by Microsoft.

Mark
--
Mark Rotteveel

[Adrian Marius Popa]

Prophetic words : " Moving away from GDML and gpre is hard, but necessary. "

https://firebirdsql.org/rlsnotesh/vulcan-sql.html

Also dudley was removed long time ago 

https://sourceforge.net/p/firebird/mailman/firebird-devel/thread/B86BC856102C4699854EE83B8C1513FD%40atenea/#msg24083676