Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
[Samba] Increasing database size limit
54 views
Skip to first unread message
mathias dufresne
Jun 11, 2015, 5:10:03 AM6/11/15
to
Hi all,
Would it be really complex to add some code to have more database files?
The database I have to fill is too large to be hosted into
$samba_db/private/DC=DOMAIN,DC=EXAMPLE,DC=COM which is limited to 4GB.
The idea would be to create files to host OUs created a domain's root level.
For example an OU to host groups: OU=Groups,DC=DOMAIN,DC=EXAMPLE,DC=COM
would be hosted in file named:
$samba_db/private/OU=Groups,DC=DOMAIN,DC=EXAMPLE,DC=COM
As there is already something to have separated DB files (CN=CONFIGURATION,
DC=DOMAINDNSZONES...) perhaps this change would not be to complex...
Having that we would be able to create files to host users in their own
file, computers in some other (if displaced from builtin "computers" OU)
and so on.
Best regards,
mathias
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba
Would it be really complex to add some code to have more database files?
The database I have to fill is too large to be hosted into
$samba_db/private/DC=DOMAIN,DC=EXAMPLE,DC=COM which is limited to 4GB.
The idea would be to create files to host OUs created a domain's root level.
For example an OU to host groups: OU=Groups,DC=DOMAIN,DC=EXAMPLE,DC=COM
would be hosted in file named:
$samba_db/private/OU=Groups,DC=DOMAIN,DC=EXAMPLE,DC=COM
As there is already something to have separated DB files (CN=CONFIGURATION,
DC=DOMAINDNSZONES...) perhaps this change would not be to complex...
Having that we would be able to create files to host users in their own
file, computers in some other (if displaced from builtin "computers" OU)
and so on.
Best regards,
mathias
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba
mathias dufresne
Jun 11, 2015, 6:30:03 AM6/11/15
to
Here is the answer from Andrew Bartlett sent to samba-technical (which I
don't read enough, so I missed the reply)
Sent by Andrew Bartlett June the 10th at 00:49
Title was "Samba and LDB at 150k+ (was: Re: [Samba] 32 bits limit?)"
....
No, it isn't possible to split it up that way, because then we would
need complex logic to allow move operations between LDB files, as well
as logic to allow a subtree search correctly. The split of one ldb file
per partition works very well for a number of operations, because it
matches the LDAP semantics we need.
On the other hand, changing the key-value store under the database, but
keeping the layer above the key-value store largely the same seems like
a massive change, but both layers are themselves quite mature, they just
have not been used together before.
What I'm getting at is that with the right testing, I think this (or
perhaps an alternative using ntdb) is the less risky approach, and the
one the follows the direction the Samba Team would like to go in.
The point of database It would also be the appropriate way to introduce
other efficiencies in the database format (such as a dual
normalised/original data store, improved indexes or SD caching).
....
don't read enough, so I missed the reply)
Sent by Andrew Bartlett June the 10th at 00:49
Title was "Samba and LDB at 150k+ (was: Re: [Samba] 32 bits limit?)"
....
No, it isn't possible to split it up that way, because then we would
need complex logic to allow move operations between LDB files, as well
as logic to allow a subtree search correctly. The split of one ldb file
per partition works very well for a number of operations, because it
matches the LDAP semantics we need.
On the other hand, changing the key-value store under the database, but
keeping the layer above the key-value store largely the same seems like
a massive change, but both layers are themselves quite mature, they just
have not been used together before.
What I'm getting at is that with the right testing, I think this (or
perhaps an alternative using ntdb) is the less risky approach, and the
one the follows the direction the Samba Team would like to go in.
The point of database It would also be the appropriate way to introduce
other efficiencies in the database format (such as a dual
normalised/original data store, improved indexes or SD caching).
....
Volker Lendecke
Jun 11, 2015, 7:30:03 AM6/11/15
to
On Thu, Jun 11, 2015 at 11:03:08AM +0200, mathias dufresne wrote:
> Hi all,
>
> Would it be really complex to add some code to have more database files?
Would trusts help you? Individual domains could become
> Hi all,
>
> Would it be really complex to add some code to have more database files?
smaller.
Volker
--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kon...@sernet.de
mathias dufresne
Nov 19, 2015, 4:30:04 AM11/19/15
to
Hi Volker,
First I apologize for very late reply. I did tests in the meantime...
Trust relationships are the option we keep in mind. As trusts are for now
bidirectional-full-trust they give us possibility to increase domain size.
If we have to split our domain we will split in one domain for users and
another for computers (meaning users' MS Windows computers as they need to
be included into AD, which is not the case for UNIX/Linux boxes, and
because MS Windows clients are almost as numerous as users).
The others choices are to have users + computers in same domain, with
several domains. This implies we reproduce management processes on each
domain.
Having one domain for users and another for computers we won't have to
reproduce processes. Of course processes to manage computers and users
should be quiet similar so somewhere we will have to reproduce these
processes, but it seems to me (I can be wrong ;) that would generate less
work during run...
Another good point is once we don't need any more trust relationships we
could move users from their domain to computers' domain almost
automagically.
Having computers spread on several domain means to remove these computers
from their domain then integrate them in the one chosen as main domain when
we'll aggregate them to remove trust relationship(s).
In parallel Oliver Liebel tells around 2 months ago he'll be able to find
funds to develop LMDB backend (in some mail with title "s4 ldb tdb limits")
which could solve the DB size limitation. And the company I'm working for
is still discussing internally (without me :p) about investing for same
development (and others).
That's where are my thoughts regarding scaling this domain... Critics are
welcome :)
Cheers,
mathias
First I apologize for very late reply. I did tests in the meantime...
Trust relationships are the option we keep in mind. As trusts are for now
bidirectional-full-trust they give us possibility to increase domain size.
If we have to split our domain we will split in one domain for users and
another for computers (meaning users' MS Windows computers as they need to
be included into AD, which is not the case for UNIX/Linux boxes, and
because MS Windows clients are almost as numerous as users).
The others choices are to have users + computers in same domain, with
several domains. This implies we reproduce management processes on each
domain.
Having one domain for users and another for computers we won't have to
reproduce processes. Of course processes to manage computers and users
should be quiet similar so somewhere we will have to reproduce these
processes, but it seems to me (I can be wrong ;) that would generate less
work during run...
Another good point is once we don't need any more trust relationships we
could move users from their domain to computers' domain almost
automagically.
Having computers spread on several domain means to remove these computers
from their domain then integrate them in the one chosen as main domain when
we'll aggregate them to remove trust relationship(s).
In parallel Oliver Liebel tells around 2 months ago he'll be able to find
funds to develop LMDB backend (in some mail with title "s4 ldb tdb limits")
which could solve the DB size limitation. And the company I'm working for
is still discussing internally (without me :p) about investing for same
development (and others).
That's where are my thoughts regarding scaling this domain... Critics are
welcome :)
Cheers,
mathias
Andreas Schneider
Nov 19, 2015, 11:30:04 AM11/19/15
to
On Thursday 19 November 2015 10:17:24 mathias dufresne wrote:
> Hi Volker,
Hi Mathias,
> In parallel Oliver Liebel tells around 2 months ago he'll be able to find
> funds to develop LMDB backend (in some mail with title "s4 ldb tdb limits")
> which could solve the DB size limitation. And the company I'm working for
> is still discussing internally (without me :p) about investing for same
> development (and others).
you should take a look at the talk Nadya gave at SambaXP 2014:
http://www.sambaxp.org/archive_data/SambaXP2014-DATA/thu/track1/Nadezhd_Ivanova-Samba4withOpenLDAPBackend-It_sAlive.pdf
http://www.sambaxp.org/archive_data/SambaXP2014-AUDIO/thu/track1/sambaxp2014-thu-track1-Nadezhda_Ivanova-Samba4withOpenLDAPBackend_-_It_s_Alive.mp3
Cheers,
-- andreas
--
Andreas Schneider GPG-ID: CC014E3D
Samba Team a...@samba.org
www.samba.org
> Hi Volker,
Hi Mathias,
> In parallel Oliver Liebel tells around 2 months ago he'll be able to find
> funds to develop LMDB backend (in some mail with title "s4 ldb tdb limits")
> which could solve the DB size limitation. And the company I'm working for
> is still discussing internally (without me :p) about investing for same
> development (and others).
http://www.sambaxp.org/archive_data/SambaXP2014-DATA/thu/track1/Nadezhd_Ivanova-Samba4withOpenLDAPBackend-It_sAlive.pdf
http://www.sambaxp.org/archive_data/SambaXP2014-AUDIO/thu/track1/sambaxp2014-thu-track1-Nadezhda_Ivanova-Samba4withOpenLDAPBackend_-_It_s_Alive.mp3
Cheers,
-- andreas
--
Andreas Schneider GPG-ID: CC014E3D
Samba Team a...@samba.org
www.samba.org
Nadezhda Ivanova
Nov 20, 2015, 6:00:04 AM11/20/15
to
Jakub's approach is different that ours, we are rewriting the better
part of the ldb stack into OpenLDAP, so he should be done faster. Even
if it does not solve all the performance issues, an ldb_mdb backend will
still be very useful - kudos Jakub for undertaking this.
As for mdb indexing - I haven't "been" in that part of the code yet and
unfortunately cannot give a meaningful advice. If something is
happening, it is somewhere in the servers/slapd/back_mdb :).
On 11/19/2015 06:20 PM, Andreas Schneider wrote:
> On Thursday 19 November 2015 10:17:24 mathias dufresne wrote:
>> Hi Volker,
> Hi Mathias,
>
>> In parallel Oliver Liebel tells around 2 months ago he'll be able to find
>> funds to develop LMDB backend (in some mail with title "s4 ldb tdb limits")
>> which could solve the DB size limitation. And the company I'm working for
>> is still discussing internally (without me :p) about investing for same
>> development (and others).
> you should take a look at the talk Nadya gave at SambaXP 2014:
>
> http://www.sambaxp.org/archive_data/SambaXP2014-DATA/thu/track1/Nadezhd_Ivanova-Samba4withOpenLDAPBackend-It_sAlive.pdf
> http://www.sambaxp.org/archive_data/SambaXP2014-AUDIO/thu/track1/sambaxp2014-thu-track1-Nadezhda_Ivanova-Samba4withOpenLDAPBackend_-_It_s_Alive.mp3
>
> Cheers,
>
>
> -- andreas
>
--
part of the ldb stack into OpenLDAP, so he should be done faster. Even
if it does not solve all the performance issues, an ldb_mdb backend will
still be very useful - kudos Jakub for undertaking this.
As for mdb indexing - I haven't "been" in that part of the code yet and
unfortunately cannot give a meaningful advice. If something is
happening, it is somewhere in the servers/slapd/back_mdb :).
On 11/19/2015 06:20 PM, Andreas Schneider wrote:
> On Thursday 19 November 2015 10:17:24 mathias dufresne wrote:
>> Hi Volker,
> Hi Mathias,
>
>> In parallel Oliver Liebel tells around 2 months ago he'll be able to find
>> funds to develop LMDB backend (in some mail with title "s4 ldb tdb limits")
>> which could solve the DB size limitation. And the company I'm working for
>> is still discussing internally (without me :p) about investing for same
>> development (and others).
> you should take a look at the talk Nadya gave at SambaXP 2014:
>
> http://www.sambaxp.org/archive_data/SambaXP2014-DATA/thu/track1/Nadezhd_Ivanova-Samba4withOpenLDAPBackend-It_sAlive.pdf
> http://www.sambaxp.org/archive_data/SambaXP2014-AUDIO/thu/track1/sambaxp2014-thu-track1-Nadezhda_Ivanova-Samba4withOpenLDAPBackend_-_It_s_Alive.mp3
>
> Cheers,
>
>
> -- andreas
>
--
0 new messages