Testing DbLinq
Jonathan Pryor
I'm gratified that DbLinq contains unit tests for all the providers.
However, as a developer I dislike any tests that require an actually
running database, particularly in the case of DbLinq which is trying to
write a LINQ provider for 7 different databases. I do not look forward
to installing and configuring 7 different database systems, much less
running tests for all of them...
For a project predicated upon databases, this does seem to be an odd
complaint. ;-)
There is a method to my madness, though: "removing" the database
requirement would make it easier to run unit tests on systems that don't
have any databases (most useful from my perspective, as I want to get
some form of unit tests within Mono for System.Data.Linq). This should
also make life easier for new developers -- less "grunt" work to do (in
the form of setting up a DB server) before working on more interesting
development, etc.
Note: I am NOT advocating that we drop the existing unit tests.
What I am advocating is a layered testing strategy; for DbLinq, there
are logically 3 layers (at least) that can be tested (more or less)
separately:
1. IQueryable -> SQL command (to send to backend provider)
2. IDataReader -> result sets
3. Database -> IDataReader
Strictly speaking, (3) is under the responsibility of the database
provider, not the LINQ provider, so can be ignored (e.g. there's no
reason for DbLinq to test SqlDataReader...).
The existing unit tests would remain as "end-to-end" integration tests,
ensuring that things actually work when the more targeted tests
suggested above work.
How would this work?
I haven't looked at layer (2) yet, but layer (1) can be solved through
string comparisons on two "well known" types: DataContext.Log and
DbCommand.CommandText (as returned from DataContext.GetCommand). For
example, this works quite nicely on current DbLinq w/o requiring any
database support:
DataContext context = new DataContext(new NullConnection());
var foos =
from p in context.GetTable<Person>()
where p.FirstName == "foo"
select p;
var cmd = context.GetCommand(foos);
Assert.AreEqual(ExpectedText, cmd.CommandText);
(NullConnection is a DbConnection subclass that does nothing on required
void-returning methods and throws an exception on all other methods.
It's enough to create a DataContext and elicit "interesting" behavior,
as seen above, without requiring a full-blown database.)
DataContext.Log can be used for checking the SQL generated for e.g.
DataContext.ExecuteCommand() and DataContext.ExecuteQuery() (along with
LINQ expressions that are immediately evaluated, such as .Count()).
A prototype of such an implementation is attached.
Is this testing strategy something the DbLinq team would be interested
in implementing?
Thanks,
- Jon
Pascal Craponne
Jonathan Pryor
> the idea is interesting, because I also have problems maintaining a VM
> with all the databases.
> Anyway I see several limitations:
> - This will require to rewrite all current tests
Hence my suggestion that this would be /in addition to/, not a
replacement for, the current tests. We want to keep the current tests
as integration tests, to ensure that "end-to-end" functionality works.
> - Making any small change (like the internal temporary objects naming)
> would require to change all tests
This is to be expected, somewhat. I'd consider it to be a feature,
myself, so that the "externally visible" effects of an internal change
are more noticeable.
> - This makes any further optimization a pain.
> As an example, I rewrote the SQL generation engine last spring/summer,
> and even if the generated SQL is different is all ways from previous
> engines, the existing tests helped me validating the engine. Also
> (still my own story), some tests were complex, and I had no idea of
> what it would look like in SQL (I'm not an SQL expert at all, and yes,
> I rewrote an SQL generation engine :)), so until the engine worked I
> had no idea of what tests should look like.
It's not necessary to have the tests written and fully complete before
the functionality is complete. In this case, I'd suggest doing the
refactoring, then writing the tests to ensure that the functionality
doesn't change in the future. (This also removes the "I don't know what
the SQL should look like" issue, as you can just use the actual
resulting SQL in the test, after you've written the functionality.)
> - SQL is very different from one vendor to another. A good example is
> how Skip() and Take() are handled in Oracle, SQL Server, and others.
You'll notice that my approach solved this, by having a DataContractTest
abstract base class (which has all the [Test] methods), and a
MsSqlDataContextTest subclass which overrides DataContextTest methods.
This allows subclasses to provide the actual SQL that's specific to that
provider, without requiring that all databases accept the same SQL.
> Regarding the tests, we could instead of a hard SQL string comparison
> (which will totally break all tests after first change), use regular
> expressions, so ignore internal names could be safely ignored, just
> like maybe tables or columns names (without ignoring their
> occurrence), whitespaces, comments, and so on?
I'd be leery about using regular expressions, and would think that the
current subclass approach would be sufficient.
However, I wonder what sort of "internal names" would be present in the
SQL that is being sent to the database....
> I also suggest to keep the current tests, but restructure them in real
> projects with real directories (currently they are all in the same
> folder, we could probably create new projets in new directories with
> links to shared files, instead of having all test projects in one
> folder).
Please! :-)
- Jon
Pascal Craponne
Jonathan Pryor
> contributors (and I would be very pleased to)
My Google Code identifier is: jonmpryor
- Jon
Pascal Craponne
Justin Collum
Pascal Craponne
Justin Collum
Pascal Craponne
Justin Collum
Pascal Craponne
Justin Collum
Pascal Craponne
Pascal Craponne
3. Criteria (the "where" you were writing about)
Justin Collum
I think this didn't come up yet because there's no boolean fields in northwind that I can see. Probably because of the fact that many databases don't support them. Still, dblinq ought to have some way to translate from the boolean object type in .net to a something in the target db.
Might be related, might not, but dbmetal did interpret the 'Boolean' type in the database as a boolean in .net. The create script from SQLite admin looks like this:
CREATE TABLE [Users] (
[Id] INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
[UserName] varchar(20) UNIQUE NOT NULL,
[Password] varchAR(20) NULL,
[FirstName] vaRCHAR(20) NULL,
[LastName] varchar(20) NULL,
[Active] BOOLEAN DEFAULT 'Y' NOT NULL
)
It seems likely that SQLite Administrator is doing something weird with booleans. It's making some assumptions, probably generating a varchar(1) instead. I'm not sure what to do with it.
Avery Pennarun
> Might be related, might not, but dbmetal did interpret the 'Boolean' type in
> the database as a boolean in .net. The create script from SQLite admin looks
> like this:
>
> CREATE TABLE [Users] (
> [Id] INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
> [UserName] varchar(20) UNIQUE NOT NULL,
> [Password] varchAR(20) NULL,
> [FirstName] vaRCHAR(20) NULL,
> [LastName] varchar(20) NULL,
> [Active] BOOLEAN DEFAULT 'Y' NOT NULL
> )
>
> It seems likely that SQLite Administrator is doing something weird with
> booleans. It's making some assumptions, probably generating a varchar(1)
> instead. I'm not sure what to do with it.
The important thing about sqlite is it doesn't really have "data
types" exactly. You can name your data types anything you want. You
could call it a "bigfoolean" if you wanted, and it would still work.
Because of that, there's no completely standard convention on what
goes into a boolean type. With that in mind, I'm not really sure how
you'd do it comfortably in Dblinq. Maybe an attribute on the
table/column object that declares what the true/false values of the
field are.
Avery
Justin Collum
Avery Pennarun
> That makes sense to me. I solved my problem by switching those columns over
> to ints. Just have to modify my linq and dbml and i'm done. I think you
> could argue that the issue I'm having is a configuration issue. I'm not sure
> why SQLite Admin lets you define a boolean type if there is no such thing in
> the list of SQLite types I referenced.
Well, people do like their data types :)
Up until sqlite3, there apparently weren't *any* types. Even ints
were just stored as varchar. That doesn't mean you don't want to
think of it as an integer, though.
Have fun,
Avery
Justin Collum
Justin Collum
Jonathan Pryor
On Mon, 2009-02-23 at 10:01 +0100, Pascal Craponne wrote:
I also suggest to keep the current tests, but restructure them in real projects with real directories (currently they are all in the same folder, we could probably create new projets in new directories with links to shared files, instead of having all test projects in one folder).
I don't know if there is a common convention, but I do know Mono's convention is to place unit tests for an assembly into a Test directory within the assembly's directory. Mapping this to DbLinq, we would thus have the directories:
| src/DbLinq | [ existing directory ] |
| src/DbLinq/Test | [ from tests/Test_NUnit/Internals, tests/DbLinqTest ] |
| src/DbLinq.Firebird | [ existing directory ] |
| src/DbLinq.Firebird/Test | [ from tests/Test_NUnit/Test_NUnit_Firebird.csproj ] |
| src/DbLinq.Ingres | [ existing directory ] |
| src/DbLinq.Ingres/Test | [ from tests/Test_NUnit/Test_NUnit_Ingres.csproj ] |
This would be a "radical" departure from the current setup of keeping the unit tests in a directory structure parallel to that of src, so I was wondering if this would be acceptable, or if you'd instead prefer a "parallel" structure within tests, e.g. have tests/DbLinq.Firebird, tests/DbLinq.Ingres, etc. directories.
- Jon
Pascal Craponne
Jonathan Pryor
no, there is no IRC channel.
If there is interest, I would suggest starting a ##dblinq channel on freenode. This would require registering the channel; directions can be found at: http://freenode.net/group_registration.shtml
Regarding the tests, I have no specific idea about how the directories should be organized (and since I'm the only one to answer to you, I must be right :)).
The remaining problem is: where should the common tests (95% of them) be placed?
If you choose a structure under a common directory, as you suggested as second option, the problem is probably easier to solve.
To prove so, I've created a new branch:
https://dblinq2007.googlecode.com/svn/branches/tests-without-db/
In this branch, I've committed the previously suggested test split, so that tests are located within a src/Assembly/Test directory.
Shared files are located within src/DbLinq/Test/Providers.
The .csproj, .vbproj, and .sln files have been updated to cope with this change.
That's the good news.
The bad news is that I only have Visual Studio Express to play with, and loading the DbLinq.sln file throws up a ton of error about the project files using features that aren't supported within Express, so I would appreciate it if someone else could take a peek and ensure things are still sane. (A subset of DbLinq.sln is supported in Express, so I was able to build most of the projects, but I don't know that all of them will build.)
This brings me to a related issue: file duplication in the build process. At present, all the project files are setup so that they copy dependent assemblies into their output directories (which makes sense, as this allows the assemblies to actually work properly ;-).
Unfortunately, for my DbLinq.sln-under-Express build, this results in 18 copies of DbLinq.dll running around, which seems...excessive. Similarly, there are 3 copies of DbLinq.SqlServer.dll, 4 of DbLinq.Oracle.dll, 5 of DbLinq.Sqlite.dll....
We could greatly reduce all this duplication by configuring the projects to place their output files into a common directory, e.g. into the top-level lib directory (which also contains Npgsql.dll, etc.), or perhaps into a src/build directory. (Personally, I'd prefer throwing them into the toplevel `lib' directory, but that's just me...).
Would this be a reasonable change to make? Or is the current setup preferred?
Thanks,
- Jon
Jonathan Pryor
On Wed, 2009-02-25 at 22:40 +0100, Pascal Craponne wrote:
no, there is no IRC channel.
Should we start one? Or is there no interest among the current maintainers?
If there is interest, I would suggest starting a ##dblinq channel on freenode. This would require registering the channel; directions can be found at: http://freenode.net/group_registration.shtml
My identifier on IRC is `jonp'.
- Jon
Pascal Craponne
Justin Collum
Adam Tauno Williams
> I agree. My exposure to IRC was a mildly frustrating experience. I
> think people check their email way more often than they log on to irc.
> And it's easier to formulate a thought in email. Where you have
> paragraphs and formatting.
Ditto; and putting code snippet's etc... in IRC is pretty wretched.
IMO, nothing beats maillist (hands down). You can sort, archive, post
(with attachments, quotations, etc..) and then the wonderful listservs
archive for everyone else. I've never understood some people aversion's
to lists.
I usually have an IRC client open to three channels (#GRLUG, #gtk, and
#ogo). Other than chatting with some fellow OGo hackers on #ogo the
amount of useful traffic is very close to zero.
--
OpenGroupware developer: awil...@whitemice.org
<http://whitemiceconsulting.blogspot.com/>
Justin Collum
Contrast that with: send an email to a group of people, wait for a response.
Option 2 is a much better use of my time.