Tasks

[Pascal Craponne]

Hi everyone,

here's a list of tasks to be done soon:

1. Subqueries. A thread is still in progress on it. I may work on it one of these days, even if the thread is still in progress

2. Automatic vendor identification, using connection string tricks. I think we all agreed, so we are looking for volunteers.

3. Insert fixes (so it can work with most of vendors without extra round). Looking for volunteers.

4. Fixes on string manipulation. Looking for volunteers.

5. Fixes in DbMetal (I made some breaking changes... too breaking, apparently). --> Me.

Did I forget something?

Pascal.

jabber/gtalk: pas...@jabber.fr
msn: pas...@craponne.org

[Pascal Craponne]

I forgot: 6.Registering DbMetal as a Visual Studio custom tool, to build .cs files from .dbml in a suitable way for DbLinq.

--

[Jiri Moudry]

Here's Jiri. I will take (2).
Can you post a link to more information on 4 and 5?

Cheers,
Jiri

[Pascal Craponne]

Holy crap, the boss's back! :)

4. Most (all?) SQL engines parse strings with index starting at 1, and .NET CLR starts at 0. At the moment, we have a hack directly at SQL generation level, and it can't be optimized there. The idea was to perform the transformation where the SpecialExpression is generated, according to a flag given in IVendor.

(the thread is http://groups.google.com/group/dblinq/browse_frm/thread/b9d1cc3d4a278470?tvc=1&q=string)

5. I made a cleanup for namespaces in generated .cs file, and some "using"s are missing when we generate entities implementing INotifyPropertyChanging or INotifyPropertyChanged. Instead of adding those usings directly in .cs, it's better to add a requirement in IImplementation interface. It would list required namespaces and be used only if matching interface is requested (hope I'm clear, sure I'm not).

I'm going to work on point 5 tonight.

[Pascal Craponne]

Point 5 is done. Good night.

[Pablo Iñigo Blasco]

I'll work in the 4th task.

Regards.

--
Saludos. Pablo Iñigo Blasco .

[Jiri Moudry]

I have checked in first half of part (2).
We now have a DataContext constructor which takes a connection string
similar to:
"DbLinqProvider=DbLinq.Sqlite.dll; Server=xx; etc.."
The connection string gets searched for DbLinqProvider=xyz substring.
If it's there, we load a DLL of that name and load up an IVendor
class.
If DbLinqProvider substring is not there, we load up an MS
SqlServerVendor.

What I am going to do next is modify the codegen - anyone who's
generating Northwind should now get an additional constructor.

Zdravi Jiri

[Pascal Craponne]

How does it work with Mono, where all vendors are (will be) embedded in the System.Data.Linq assembly?

[Jiri Moudry]

So in the Mono scheme, we will have the System.Data.Linq contain
multiple IVendor implementations? Then I don't think a single config
variable "DbLinqProvider" is enough.

Maybe I should change it to two config variables, similar to:
connectionString =
"DbLinqProviderAssembly=System.Data.Linq.dll,DbLinqProviderClass=PgsqlVendor;server=x;UID=y;..."

[Atsushi Eno]

This could just be simply short names, instead of fully qualified
type names, at least for some known providers. It is like we can
specify a CodeDomProvider in xsd command (/l option).

Atsushi Eno

[Atsushi Eno]

Hi Jiri,

Actually r927 broke MONO_STRICT build, where no concrete DB providers
cannot be referenced (there is no reference to FirebirdSql, Ingres,
Pgsql etc. etc.).

It should be done by some reflection based factories that provides
those connections, without using strong (straightforward) references
to those connection types.

Atsushi Eno

[Pascal Craponne]

Ahem, I tend to support Atsushi:

- as discussed, it is better NOT to reference drivers in the vendors, since we don't want (and Mono people just probably can't) stick to any driver or version.

- regarding the driver option in the connection string, maybe we should have discussed this a bit further before implementation.

The concerns for vendor in connection string are:

- shall the connection string use an assembly identification or a pseudo-type that would be declared by each vendor (using a class attribute on IVendor implementation for example)?

- if using assembly identification, how to distinguish several vendors in the same assembly? no idea for me here

- if using a "vendor id", how to load the assembly if not loaded? can we consider that it is application's responsibility to load its vendor, so DbLinq would just need to scan current appdomain for corresponding IVendor implementation?

[Jiri Moudry]

Apologies guys - I am fixing this right now.
Nobody answered my question - how do you typically compile
MONO_STRICT?
Is there a script checked in, or do you use MonoDevelop/SharpDevelop?

[Jiri Moudry]

Second iteration is checked in:
References removed, and I implemented Atsushi's suggestion of using
short names,
eg. "DbLinqProvider=MySql; Server=localhost; ..."

Atsushi - is MONO_STRICT now building correctly?

[Atsushi Eno]

Oops, yes, I use only Makefile and relevant mono scripts that
mono build system generates. I've added vendor files (things
like SqlServerProvider.cs but for every other vendor) only in
mono build script. I've just added files to System.Data.Linq.csproj
as well.

Jiri, thanks for taking care of it. Now it should build (it builds
at least with System.Data.Linq.csproj). It's already 6:30 AM, so
I'll try it on mono tomorrow.

Atsushi Eno

For now I'm going to sleep but if

Atsushi Eno

[Atsushi Eno]

So, I tried it under Mono. It now builds, but does not run.
There seem a couple of reasons why:

- Assembly.Load("System.Data.Linq.dll" won't work, since it is
not in current current directory but in the GAC (you can try
Assembly.Load("System.Data.dll") on .NET - it won't work).

For now I just skipped assembly load phase by just supplying
typeof(DataContext).Assembly (= S.D.linq.dll itself).

I'm not sure if it doesn't really matter in DBLinq land. It will
fail when DbLinq.dll and DbLinq.VendorFoo.dll reside differently.

- DbLinq.Vendor.Implementation.Vendor.CreateDbConnection() tries
to create IDbConnection and then it simply fails, saying:

> DataContext ctor: Assembly load failed for System.Data.SQLite.DLL:
> System.IO.FileNotFoundException: Could not load file or assembly
> 'System.Data.SQLite.DLL' or one of its dependencies. The system
> cannot find the file specified.
> File name: 'System.Data.SQLite.DLL'

I guess it may happen to non-strict version of DBLinq when referenced
-by-name assemblies do not exist at run-time (only in-use DB provider
assembly has to exist, right?).

I also noticed that .ctor(string, MappingSource) could be implemented
easily. I'm attaching a patch. (Note: it contains mono workaround
for the first issue. Second issue is not fixed.)

Atsushi Eno

[Pascal Craponne]

Hi guys,

I took a quick look at the code, and I definitely think that it is not DbLinq's responsibility to directly load the required assembly.

The application know what vendors it will use, is then its own responsibility.

To me, identifiers given in the connection string MUST just allow to find an IVendor implementation that is already loaded.

Maybe, for DbLinq in extended mode, we may consider later to use an extension to dynamically load the assembly, but this is not the first priority.

The code currently in SVN show that dynamically loading assemblies is a complex task that is far beyond the DataContext's scope.

Pascal.

Index: src/DbLinq/Data/Linq/DataContext.cs
===================================================================
--- src/DbLinq/Data/Linq/DataContext.cs (revision 116918)
+++ src/DbLinq/Data/Linq/DataContext.cs (working copy)
@@ -106,12 +106,6 @@
            Init(new DatabaseContext(connection), null, null);
        }

-        [DbLinqToDo]
-        public DataContext(string fileOrServerOrConnection, MappingSource mapping)
-        {
-            throw new NotImplementedException();
-        }
-
        /// <summary>
        /// Construct DataContext, given a connectionString.
        /// To determine which DB type to go against, we look for 'DbLinqProvider=xxx' substring.
@@ -121,14 +115,22 @@
        /// DbLinqProvider=Mysql
        /// DbLinqProvider=Oracle etc.
        /// </summary>
-        /// <param name="connectionString">specifies file or server connection</param>
+        /// <param name="fileOrServerOrConnection">specifies file or server connection</param>
        [DbLinqToDo]
-        public DataContext(string connectionString)
+        public DataContext(string fileOrServerOrConnection)
+               : this (fileOrServerOrConnection, null)
        {
+        }
+
+        [DbLinqToDo]
+        public DataContext(string fileOrServerOrConnection, MappingSource mapping)
+        {
            #region DataContext connectionString ctor
-            if (connectionString == null)
-                throw new ArgumentNullException("connectionString");
+            if (fileOrServerOrConnection == null)
+                throw new ArgumentNullException("fileOrServerOrConnection");

+            string connectionString = fileOrServerOrConnection;
+
            System.Text.RegularExpressions.Regex reProvider
                = new System.Text.RegularExpressions.Regex(@"DbLinqProvider=([\w\.]+)");

@@ -146,7 +148,7 @@
 #if MONO_STRICT
                //Pascal says on the forum:
                //[in MONO] "all vendors are (will be) embedded in the System.Data.Linq assembly"
-                assemblyToLoad = "System.Data.Linq.dll";
+               assemblyToLoad = null;
                vendorClassToLoad = match.Groups[1].Value; //eg. "MySql"
 #else
                //plain DbLinq - non MONO:
@@ -172,8 +174,12 @@
            Assembly assy;
            try
            {
+#if MONO_STRICT
+               assy = typeof (DataContext).Assembly;
+#else
                //TODO: check if DLL is already loaded?
                assy = Assembly.LoadFrom(assemblyToLoad);
+#endif
            }
            catch (Exception ex)
            {

[Atsushi Eno]

Hello,

Regardless of whether current implementation strategy is to change
or not, Dynamic type loading will also happen for loading DB
provider assemblies (e.g. System.Data.SQLite.dll), like currently
DbLinq.Vendor.Implementation.Vendor.CreateDbConnection() does.

To not load assembly dynamically, you will have to add dependencies
onto all DB providers, which you don't want (unless I miss some
other possibilities).

Atsushi Eno

[Pascal Craponne]

Argh, you're right. This prooves that this complicated problem should have been discussed first :)

Anyway the vendor still must not provide, in any way (by reference or type name), a method to indicate a required driver, since we can not stick to a driver implementation or version.

Ideally, all drivers should be in the AppDomain, so we just would need to identify them and use them.

The problem here, is that even if an assembly is referenced in a project, it is not loaded until the application uses some of its classes. The missing link (ah ah ah) is just here.

So if we find a way to ensure that a referenced assembly is loaded even if not used by code, then we probably win the game.

[Andrus]

Other possibility may be to use app.config /web.config / maschine.config file entries like ADO .NET standard Dataprovider factory or

separate config file like NHibernate

 

Andrus.

[Atsushi Eno]

Does this simplification match what you wanted i.e. provider
assembly should be loaded in prior?

Atsushi Eno

[Pascal Craponne]

I took a look at how .NET works when loading assemblies (and couldn't find any precise documentation on this. If someone gets it, please give the link):

- assemblies are loaded only when types are used

- assemblies do not contain any trace of referenced assemblies

- when another assembly is used, it is by the referenced type.

- assemblies are searched in GAC and currently directory for type resolution (and some other directories can be specified too).

So I think we can not load extra assemblies to find what's in, because we would need to search the whole application directory, and much worse, the GAC.

This leaves to options, and we can use both:

- scan the current AppDomain (this is enough for Mono where all vendors are embedded into System.Data.Linq)

- for DbLinq in extended mode, as Andrus suggested, we can also use an extra configuration section (or better, a vendor provider that can be changed).

This was for vendors.

For database drivers, the problem is probably worse. I'd like to find a solution similar to what I wrote in paragraphs above.

Anyway, I suggest that we stop changes on this part until we all agree.

[Jiri Moudry]

Pascal, Astushi - Ohayo gozaimasu & Bon soir,
seems that both of you like scanning the AppDomain for the IVendor
class and the IDbConnection implementation.
I like it also, because it results in fewer assumptions and shifts
responsibility to the user. Should we go ahread with that approach?

[Pascal Craponne]

What options do we have for the database driver (the ADO one)?

How to make the guess? Shall be make a choice given another parameter in the connection string? Shall DbLinq also scan the AppDomain, and ask vendor if any found IDbConnection is suitable (this condition is not sufficient, we also need to dig more).

[Atsushi Eno]

Actually, personally I'm positive to include version dependency
on those driver assemblies. The reasons are:

- It is the database driver vendor who claims that a future version
of the assembly is compatible with older versions of the driver or
not (by "publisher policy file" i.e. policy.*.*.dll in general).
- Usually a user (dblinq here) should depend on specific version of
the dependency assemblies in each version of his or her application
- A user (we) can finally determine whether a future version of the
assembly is compatible with older versions in respect to his or her
application. It can be specified by our assembly configuration
(/configuration/runtime/assemblyBinding/dependentAssembly).

While we would likely want to avoid loading "duplicate" assemblies
(not precisely though; they have different versions), we can first
iterate Assemblies in CurrentDomain, and then try Assembly.Load() with
fully qualified name. Then users don't have to add extra code to load
the driver assembly in prior to use dblinq.

Adding custom configuration section may also look like a solution, if
we want to let users to make first decision on which Type should be
regarded as the target driver (it is typical scenario). But the problem
here is that it works only when the dependent assemblies MUST exist
*while proving the target driver*. Otherwise it will raise an error
saying that the configuration system cannot find the target Type.
(Am I understanding the situation correctly? I'm unsure.)

Atsushi Eno

> jabber/gtalk: pas...@jabber.fr <mailto:pas...@jabber.fr>
> msn: pas...@craponne.org <mailto:pas...@craponne.org>
>
>
> >

[Pascal Craponne]

I may die alone (at least), but I still don't agree:

- Sticking to a driver will cause problems where there are two driver implementations, such as Oracle drivers, where there are two different implementations, MS and ODP.

- Sticking to a version will require DbLinq to be built and integrate each new version. Today DbLinq supports 7 databases. How many releases will it imply each year? How many releases when DbLinq will support 10 databases? Too many.

- Sticking to a version will also leave behind the applications who can't upgrade to the version linked by DbLinq.

Each actor has its responsibility and it's better if the application has the responsibility to assemble all parts and make a consistent result.

Atsushi, are you talking in your message about loading vendor or database driver?

[Atsushi Eno]

Hello,

Pascal Craponne wrote:
> I may die alone (at least), but I still don't agree:

Well, it is all about "when we couldn't prove the target
vendor/provider after iterating assemblies in the current domain.

It is not to stick to specific driver (at least what I intended).
Discussed details below.

But before digging in depth, I'd like to confirm whether we take the
approach to check assemblies in the current domain. Right now
MONO_STRICT doesn't work at all for connectionString-based use due to
the current assembly load strategy.

> - Sticking to a driver will cause problems where there are two driver
> implementations, such as Oracle drivers, where there are two different
> implementations, MS and ODP.

I'm curious - can/should they be the same implementation while the
"vendors" are different? Is it still valid when we start to consider
creating IDbConnection instance from connection string i.e. how can
we distinguish a connection string for OracleClient and for ODP.NET
within one vendor API? Add another connection string item such as
"OracleDriver=ODP" ?

> - Sticking to a version will require DbLinq to be built and integrate
> each new version. Today DbLinq supports 7 databases. How many releases
> will it imply each year? How many releases when DbLinq will support 10
> databases? Too many.

Note that I didn't suggest to "actually" stick to specific versions
of database drivers. As I wrote in the previous message, they should
be redirected to newer versions as long as the driver vendors ship
policies. Or you could even put assembly configuration for dblinq.
That is, for each <bindingRedirect> element you just give the oldest
supported version of the driver like:

<dependentAssembly>
<assemblyIdentity name="MySql.Data" publicKeyToken="(snip)"
culture=""/>
<bindingRedirect oldVersion="0.0.0.0-65535.65535.65535.65535"
newVersion="(the oldest supported version" />
</dependentAssembly>

seealso http://msdn.microsoft.com/en-us/library/9w519wzk(VS.71).aspx

It'd happen just once. I don't think there really is such a task to
manage every driver every year, unless they became very different
(probably like Npgsql -> Npgsql2, where we should really care,
at least the assembly name has changed).

> - Sticking to a version will also leave behind the applications who
> can't upgrade to the version linked by DbLinq.

So it won't happen, unless the driver assembly name changes (and
even it might not happen as you seem to have some magic to have
OracleClient and ODP.NET coexist within a vendor implementation).

> Each actor has its responsibility and it's better if the application has
> the responsibility to assemble all parts and make a consistent result.
>
> Atsushi, are you talking in your message about loading vendor or
> database driver?

Both. But note that under MONO_STRICT I don't need any trick for
loading vendors ;-)

Atsushi Eno

> > <mailto:gmo...@gmail.com <mailto:gmo...@gmail.com>>> wrote:
> >
> >
> > Pascal, Astushi - Ohayo gozaimasu & Bon soir,
> > seems that both of you like scanning the AppDomain for the
> IVendor
> > class and the IDbConnection implementation.
> > I like it also, because it results in fewer assumptions and
> shifts
> > responsibility to the user. Should we go ahread with that
> approach?
> >
> >
> >
> >
> >
> > --
> > Pascal.
> >
> > jabber/gtalk: pas...@jabber.fr <mailto:pas...@jabber.fr>

> <mailto:pas...@jabber.fr <mailto:pas...@jabber.fr>>
> > msn: pas...@craponne.org <mailto:pas...@craponne.org>
> <mailto:pas...@craponne.org <mailto:pas...@craponne.org>>

[Atsushi Eno]

Mmm, never mind :( I was under the impression that the assembly of
the redirected version does not have to exist, but maybe it does.

Atsushi Eno

[Andrus]

> (probably like Npgsql -> Npgsql2, where we should really care,
> at least the assembly name has changed).

Assembly name is still npgsql.dll

http://pgfoundry.org/forum/message.php?msg_id=1003866

So I don't understand why you wrote that assembly name has changed.

Andrus.

[Atsushi Eno]

Oops. Then it was my another misunderstanding (regarding your "why").

Atsushi Eno

[Pascal Craponne]

This thread is getting longer each day (and we're now probably out of its original scope).

Here's my concrete suggestion for implementation:

Needs:
- we need to optionally specify a IVendor and a IDbConnection
- since we stick to MS.NET API, the only workaround we have is to use the connection string

Specifications:
- for both (IVendor and IDbConnection), use more or less qualified type:
  * more qualified type when the assembly isn't loaded, so we need to use a type name+assembly to find and load it
  * average qualified type when the assembly is already loaded, so we need here only the qualified type
  * less, like a vendor ID, if the vendor is a well-known vendor, and is already loaded in memory (optional, let me know if you like it)
- Today the IVendor is identified by "DbLinqProvider", I suggest to rename it "DbProvider" (because of mono) and use "DbConnection" to identify the IDbConnection

Examples:
- fully qualified vendor and connection "DbProvider=DbLinq.PostgreSql.PgsqlVendor, DbLinq.PostgreSql; DbConnection=Npgsql.NpgsqlConnection, Npgsql; ..."
- partially qualified: "DbProvider=DbLinq.PostgreSql.PgsqlVendor; DbConnection=Npgsql.NpgsqlConnection; ..."
- ID qualitifed: "DbProvider=PostgreSQL; DbConnection=PostgreSQL; ..."

Implementation:
- Create or extend the Util.TypeLoader utility class to find a type or load it (today the class doesn't load assemblies)
- Use it for both IVendor and IDbConnection
- Regarding the ID qualification, this requires:
  * adding some attribute to vendor implementations, so we could mark vendors with an additional attribute such as "[VendorID("PostgreSQL")]"
  * AppDomain scan
I'm not the sure the ID thing is a good one, at least in the current solution.

So. Who agrees, who disagrees? Jiri, how long to you think it will take to change?

Thanks,
Pascal.

[Atsushi Eno]

Hi, I was somehow alive.

I agree to most of the points. I only have a few concern:

> Examples:
> - fully qualified vendor and connection
> "DbProvider=DbLinq.PostgreSql.PgsqlVendor, DbLinq.PostgreSql;
> DbConnection=Npgsql.NpgsqlConnection, Npgsql; ..."
> - partially qualified: "DbProvider=DbLinq.PostgreSql.PgsqlVendor;
> DbConnection=Npgsql.NpgsqlConnection; ..."
> - ID qualitifed: "DbProvider=PostgreSQL; DbConnection=PostgreSQL; ..."

The first vendor name looks rather like a partial name, and I'm not
sure if it works fine with Assembly.Load*() methods. (There is
Assembly.LoadWithPartialName() but it is obsoleted.) A fully
qualified name involves version numbers, which does not seem good.

And I don't think the second "partially qualified" case is useful.

I have no preference on which/how dependencies will be loaded, as
long as it does not force users to add extra code (such as loading
dependencies in prior to the actual use). DBLinq is designed to be
dependency version agnostic and I show doubts on that principle
only when it rather looks like a blocker to more features.

Atsushi Eno