[Samba] CTDB and locking issues in 4.4.6 (Classic domain)
Alex Crow via samba
We recently upgraded our fileservers from Centos supplied 4.2.10 to
Sernet 4.4.6, and then our DCs from 3.6.x to 4.4.6.
It seems that since then we've had problems with locks not being obeyed
on all nodes - they only seem to work when a second client opens a file
on the same node as the first client.
For example, when a user opens an Excel file I will see something like
this with smbstatus -L:
1:30578 1608 DENY_WRITE 0x2019f RDWR NONE
/sharename path/foo.xls
This output is identical on all nodes, so smbstatus at least can see the
locks.
However if another client opens the same XLS file, they only get the
"This file is in use by <username>" if they happen to hit the same
server as the first user. If they happen to land on another server they
don't get this prompt, and if the original user modifies the file, when
they try to save their changes they get prompted whether they want to
overwrite. I'm sure we didn't have this issue before the upgrade.
CTDB status reports as OK on all nodes.
Our filesystem passes ping_pong in both read and read/write, data
increment behaving as expected.
We have a lot of users trying to edit Excel and Word docs (Office 2010)
so this is happening quite often.
Best regards,
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba
Alex Crow via samba
>
> It appears that samba is still using local databases and not clustered
> databases.
>
> What does "ctdb getdbmap" list? Is locking.tdb a clustered database?
> Do you have "clustering = yes" in smb.conf on all the nodes?
>
> Amitay.
FYI as stated smbstatus -L shows locks from all hosts, prefixed by the PNN.
Alex Crow via samba
On 20/10/16 14:10, Amitay Isaacs wrote:
> It appears that samba is still using local databases and not clustered
> databases.
>
> What does "ctdb getdbmap" list? Is locking.tdb a clustered database?
> Do you have "clustering = yes" in smb.conf on all the nodes?
>
> Amitay.
Hi Amitay,
Here you go:
Server 1:
# ctdb getdbmap
Number of databases:20
dbid:0x4d2a432b name:g_lock.tdb path:/var/lib/ctdb/g_lock.tdb.1
dbid:0x2d608c16 name:netlogon_creds_cli.tdb
path:/var/lib/ctdb/netlogon_creds_cli.tdb.1 READONLY
dbid:0x9ec2a880 name:serverid.tdb path:/var/lib/ctdb/serverid.tdb.1
dbid:0x6afb8c09 name:dbwrap_watchers.tdb
path:/var/lib/ctdb/dbwrap_watchers.tdb.1
dbid:0x521b7544 name:smbXsrv_version_global.tdb
path:/var/lib/ctdb/smbXsrv_version_global.tdb.1
dbid:0x6b06a26d name:smbXsrv_session_global.tdb
path:/var/lib/ctdb/smbXsrv_session_global.tdb.1
dbid:0x68c12c2c name:smbXsrv_tcon_global.tdb
path:/var/lib/ctdb/smbXsrv_tcon_global.tdb.1
dbid:0x4e66c2b2 name:brlock.tdb path:/var/lib/ctdb/brlock.tdb.1
dbid:0x7a19d84d name:locking.tdb path:/var/lib/ctdb/locking.tdb.1
dbid:0x06916e77 name:leases.tdb path:/var/lib/ctdb/leases.tdb.1
dbid:0x66f71b8c name:smbXsrv_open_global.tdb
path:/var/lib/ctdb/smbXsrv_open_global.tdb.1
dbid:0x5bcfcbd7 name:printer_list.tdb path:/var/lib/ctdb/printer_list.tdb.1
dbid:0x477d2e20 name:smbXsrv_client_global.tdb
path:/var/lib/ctdb/smbXsrv_client_global.tdb.1
dbid:0x2ca251cf name:account_policy.tdb
path:/var/lib/ctdb/persistent/account_policy.tdb.1 PERSISTENT
dbid:0x3ef19640 name:passdb.tdb
path:/var/lib/ctdb/persistent/passdb.tdb.1 PERSISTENT
dbid:0xc3078fba name:share_info.tdb
path:/var/lib/ctdb/persistent/share_info.tdb.1 PERSISTENT
dbid:0x6cf2837d name:registry.tdb
path:/var/lib/ctdb/persistent/registry.tdb.1 PERSISTENT
dbid:0xa1413774 name:group_mapping.tdb
path:/var/lib/ctdb/persistent/group_mapping.tdb.1 PERSISTENT
dbid:0x7132c184 name:secrets.tdb
path:/var/lib/ctdb/persistent/secrets.tdb.1 PERSISTENT
dbid:0x6645c6c4 name:ctdb.tdb path:/var/lib/ctdb/persistent/ctdb.tdb.1
PERSISTENT
Server 2:
Number of databases:20
dbid:0x4d2a432b name:g_lock.tdb path:/var/lib/ctdb/g_lock.tdb.0
dbid:0x2d608c16 name:netlogon_creds_cli.tdb
path:/var/lib/ctdb/netlogon_creds_cli.tdb.0 READONLY
dbid:0x9ec2a880 name:serverid.tdb path:/var/lib/ctdb/serverid.tdb.0
dbid:0x6afb8c09 name:dbwrap_watchers.tdb
path:/var/lib/ctdb/dbwrap_watchers.tdb.0
dbid:0x521b7544 name:smbXsrv_version_global.tdb
path:/var/lib/ctdb/smbXsrv_version_global.tdb.0
dbid:0x6b06a26d name:smbXsrv_session_global.tdb
path:/var/lib/ctdb/smbXsrv_session_global.tdb.0
dbid:0x68c12c2c name:smbXsrv_tcon_global.tdb
path:/var/lib/ctdb/smbXsrv_tcon_global.tdb.0
dbid:0x4e66c2b2 name:brlock.tdb path:/var/lib/ctdb/brlock.tdb.0
dbid:0x7a19d84d name:locking.tdb path:/var/lib/ctdb/locking.tdb.0
dbid:0x06916e77 name:leases.tdb path:/var/lib/ctdb/leases.tdb.0
dbid:0x66f71b8c name:smbXsrv_open_global.tdb
path:/var/lib/ctdb/smbXsrv_open_global.tdb.0
dbid:0x5bcfcbd7 name:printer_list.tdb path:/var/lib/ctdb/printer_list.tdb.0
dbid:0x477d2e20 name:smbXsrv_client_global.tdb
path:/var/lib/ctdb/smbXsrv_client_global.tdb.0
dbid:0x2ca251cf name:account_policy.tdb
path:/var/lib/ctdb/persistent/account_policy.tdb.0 PERSISTENT
dbid:0x3ef19640 name:passdb.tdb
path:/var/lib/ctdb/persistent/passdb.tdb.0 PERSISTENT
dbid:0xc3078fba name:share_info.tdb
path:/var/lib/ctdb/persistent/share_info.tdb.0 PERSISTENT
dbid:0x6cf2837d name:registry.tdb
path:/var/lib/ctdb/persistent/registry.tdb.0 PERSISTENT
dbid:0xa1413774 name:group_mapping.tdb
path:/var/lib/ctdb/persistent/group_mapping.tdb.0 PERSISTENT
dbid:0x7132c184 name:secrets.tdb
path:/var/lib/ctdb/persistent/secrets.tdb.0 PERSISTENT
dbid:0x6645c6c4 name:ctdb.tdb path:/var/lib/ctdb/persistent/ctdb.tdb.0
PERSISTENT
Server 3:
Number of databases:20
dbid:0x4d2a432b name:g_lock.tdb path:/var/lib/ctdb/g_lock.tdb.2
dbid:0x2d608c16 name:netlogon_creds_cli.tdb
path:/var/lib/ctdb/netlogon_creds_cli.tdb.2 READONLY
dbid:0x9ec2a880 name:serverid.tdb path:/var/lib/ctdb/serverid.tdb.2
dbid:0x6afb8c09 name:dbwrap_watchers.tdb
path:/var/lib/ctdb/dbwrap_watchers.tdb.2
dbid:0x521b7544 name:smbXsrv_version_global.tdb
path:/var/lib/ctdb/smbXsrv_version_global.tdb.2
dbid:0x6b06a26d name:smbXsrv_session_global.tdb
path:/var/lib/ctdb/smbXsrv_session_global.tdb.2
dbid:0x68c12c2c name:smbXsrv_tcon_global.tdb
path:/var/lib/ctdb/smbXsrv_tcon_global.tdb.2
dbid:0x4e66c2b2 name:brlock.tdb path:/var/lib/ctdb/brlock.tdb.2
dbid:0x7a19d84d name:locking.tdb path:/var/lib/ctdb/locking.tdb.2
dbid:0x06916e77 name:leases.tdb path:/var/lib/ctdb/leases.tdb.2
dbid:0x66f71b8c name:smbXsrv_open_global.tdb
path:/var/lib/ctdb/smbXsrv_open_global.tdb.2
dbid:0x5bcfcbd7 name:printer_list.tdb path:/var/lib/ctdb/printer_list.tdb.2
dbid:0x477d2e20 name:smbXsrv_client_global.tdb
path:/var/lib/ctdb/smbXsrv_client_global.tdb.2
dbid:0x2ca251cf name:account_policy.tdb
path:/var/lib/ctdb/persistent/account_policy.tdb.2 PERSISTENT
dbid:0x3ef19640 name:passdb.tdb
path:/var/lib/ctdb/persistent/passdb.tdb.2 PERSISTENT
dbid:0xc3078fba name:share_info.tdb
path:/var/lib/ctdb/persistent/share_info.tdb.2 PERSISTENT
dbid:0x6cf2837d name:registry.tdb
path:/var/lib/ctdb/persistent/registry.tdb.2 PERSISTENT
dbid:0xa1413774 name:group_mapping.tdb
path:/var/lib/ctdb/persistent/group_mapping.tdb.2 PERSISTENT
dbid:0x7132c184 name:secrets.tdb
path:/var/lib/ctdb/persistent/secrets.tdb.2 PERSISTENT
dbid:0x6645c6c4 name:ctdb.tdb path:/var/lib/ctdb/persistent/ctdb.tdb.2
PERSISTENT
All the servers load an smb.conf from the cluster FS that includes
clustering=yes.
As I said we did not seem to have this problem when we were running
CentOS 4.2.10 (but it had a crash bug so we had to move).
Regards,
Amitay Isaacs via samba
It appears that samba is still using local databases and not clustered
databases.
What does "ctdb getdbmap" list? Is locking.tdb a clustered database? Do
you have "clustering = yes" in smb.conf on all the nodes?
Amitay.
Martin Schwenke via samba
On Thu, 20 Oct 2016 14:28:38 +0100, Alex Crow via samba
<sa...@lists.samba.org> wrote:
> >
> > It appears that samba is still using local databases and not clustered
> > databases.
> >
> > What does "ctdb getdbmap" list? Is locking.tdb a clustered database?
> > Do you have "clustering = yes" in smb.conf on all the nodes?
> >
> > Amitay.
>
> FYI as stated smbstatus -L shows locks from all hosts, prefixed by the PNN.
That tells you that the locks are correctly stored in CTDB and
smbstatus can find them. This means that smbd should be able to find
them too.
I can't help but think that smbd is somehow using different
configuration than smbstatus.
You don't have smbd/smbstatus installed from a package in /usr/ and the
other left over in /usr/local/, or similar? In that case, they could
be getting their configurations from different places...
peace & happiness,
martin
Alex Crow via samba
>> FYI as stated smbstatus -L shows locks from all hosts, prefixed by the PNN.
> That tells you that the locks are correctly stored in CTDB and
> smbstatus can find them. This means that smbd should be able to find
> them too.
>
> I can't help but think that smbd is somehow using different
> configuration than smbstatus.
>
> You don't have smbd/smbstatus installed from a package in /usr/ and the
> other left over in /usr/local/, or similar? In that case, they could
> be getting their configurations from different places...
>
> peace & happiness,
> martin
Hi Martin,
No, nothing like that. Installing the Sernet packages obsoletes the
distro ones.
FYI:
# smbstatus -V
Version 4.4.6-SerNet-RedHat-31.el7
# smbd -V
Version 4.4.6-SerNet-RedHat-31.el7
# find /usr -name smbd
/usr/sbin/smbd
# find /usr -name smbstatus
/usr/bin/smbstatus
Thanks very much,
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Alex Crow via samba
> Interestingly I have a setup here for testing for rollout of S4 AD.
> The file servers are in an AD domain and are running Sernet 4.4.5, so
> one version older than production. I have tested the locking on these
> and it works. The only difference I can find that in prod,
> /etc/default/sernet-samba-ctdb contains these lines:
>
> CTDB_MANAGES_SAMBA=yes
> CTDB_MANAGES_WINBIND=yes
>
> Whereas in the test environment they are:
> CTDB_SERVICE_WINBIND="sernet-samba-winbindd"
> CTDB_SERVICE_SMB="sernet-samba-smbd"
>
> I'm wondering if this is what's making the difference?
>
> Cheers,
>
> Alex
>
>
The above is not what I meant, apologies. In the prod cluster, these lines:
CTDB_SERVICE_WINBIND="sernet-samba-winbindd"
CTDB_SERVICE_SMB="sernet-samba-smbd"
are actually absent. The CTDB_MANAGES_... ones are the same on both setups.
Alex Crow via samba
>> You don't have smbd/smbstatus installed from a package in /usr/ and the
>> other left over in /usr/local/, or similar? In that case, they could
>> be getting their configurations from different places...
>>
>> peace & happiness,
>> martin
> Hi Martin,
>
> No, nothing like that. Installing the Sernet packages obsoletes the
> distro ones.
>
> FYI:
>
> # smbstatus -V
> Version 4.4.6-SerNet-RedHat-31.el7
> # smbd -V
> Version 4.4.6-SerNet-RedHat-31.el7
>
> # find /usr -name smbd
> /usr/sbin/smbd
> # find /usr -name smbstatus
> /usr/bin/smbstatus
>
> Thanks very much,
>
> Alex
> --
Interestingly I have a setup here for testing for rollout of S4 AD. The
file servers are in an AD domain and are running Sernet 4.4.5, so one
version older than production. I have tested the locking on these and it
works. The only difference I can find that in prod,
/etc/default/sernet-samba-ctdb contains these lines:
CTDB_MANAGES_SAMBA=yes
CTDB_MANAGES_WINBIND=yes
Whereas in the test environment they are:
CTDB_SERVICE_WINBIND="sernet-samba-winbindd"
CTDB_SERVICE_SMB="sernet-samba-smbd"
I'm wondering if this is what's making the difference?
Cheers,
Alex
Martin Schwenke via samba
<sa...@lists.samba.org> wrote:
> > Interestingly I have a setup here for testing for rollout of S4 AD.
> > The file servers are in an AD domain and are running Sernet 4.4.5, so
> > one version older than production. I have tested the locking on these
> > and it works. The only difference I can find that in prod,
> > /etc/default/sernet-samba-ctdb contains these lines:
> >
> > CTDB_MANAGES_SAMBA=yes
> > CTDB_MANAGES_WINBIND=yes
> >
> > Whereas in the test environment they are:
> > CTDB_SERVICE_WINBIND="sernet-samba-winbindd"
> > CTDB_SERVICE_SMB="sernet-samba-smbd"
> >
> > I'm wondering if this is what's making the difference?
> The above is not what I meant, apologies. In the prod cluster, these lines:
>
> CTDB_SERVICE_WINBIND="sernet-samba-winbindd"
> CTDB_SERVICE_SMB="sernet-samba-smbd"
>
> are actually absent. The CTDB_MANAGES_... ones are the same on both setups.
I'm guessing that's it. Instead of running the Sernet initscripts (or
systemd thingies), some other initscript is being run and it is
probably not making smbd use the correct configuration, so smbd is
probably running without clustering configured...
... but that's obviously just speculation! I don't know anything about
the Sernet packages...
Possibly time to talk to someone at Sernet?
peace & happiness,
martin
Ralph Böhme via samba
> Interestingly I have a setup here for testing for rollout of S4 AD. The file
> servers are in an AD domain and are running Sernet 4.4.5, so one version
> older than production.
with attached patch that reverts the change?
Cheerio!
-slow
L.P.H. van Belle via samba
I suggest have a look in :
https://bugzilla.samba.org/show_bug.cgi?id=12371
i think that wil help you.
Greetz,
Louis
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-...@lists.samba.org] Namens Martin Schwenke
> via samba
> Verzonden: vrijdag 21 oktober 2016 11:46
> Aan: sa...@lists.samba.org
> Onderwerp: Re: [Samba] CTDB and locking issues in 4.4.6 (Classic domain)
Alex Crow via samba
> I suggest have a look in :
> https://bugzilla.samba.org/show_bug.cgi?id=12371
>
> i think that wil help you.
>
> Greetz,
>
> Louis
>
>
>
The thing is, that CTDB /is/ successfully starting the services but just
the locking isn't working correctly.
Cheers
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Alex Crow via samba
I could test that but in production running from direct source builds is
not permitted here. There must be a repo available or at least the
ability to build RPMs.
I've tried before rebuilding upstream Samba using RH spec files but I've
never succeeded in getting all the packages to build correctly :-(
That's why I use the Sernet repo.
Another interesting point is that I've just upgraded the test cluster to
4.4.6 and it appears to be working, although that is only two hosts. The
prod setup is 6 hosts, with two sub-clusters, so there could be a
difference there.
Cheers
Alex Crow via samba
Hi Ralph,
Some more tests I've done indicate that locking is only working between
2 of my 6 VIPs (one per host). If I open a file on either of these two
"working" hosts, it's locked on the other working one, but not on any
others. Opening a file on any of the "non-working" hosts means the file
is not locked anywhere else.
On my test setup with only two nodes it seems to work normally with both
4.4.5 and 4.4.6.
All hosts have the same smb.conf.
Cheers
Alex Crow via samba
On 21/10/16 12:07, Alex Crow via samba wrote:
> On 21/10/16 10:52, Ralph Böhme wrote:
>> On Fri, Oct 21, 2016 at 10:30:46AM +0100, Alex Crow via samba wrote:
>>> Interestingly I have a setup here for testing for rollout of S4 AD.
>>> The file
>>> servers are in an AD domain and are running Sernet 4.4.5, so one
>>> version
>>> older than production.
>> maybe it's related to bug 12005 that was fixed in 4.4.6? Can you test
>> with attached patch that reverts the change?
>>
>> Cheerio!
>> -slow
>
> Hi Ralph,
>
> Some more tests I've done indicate that locking is only working
> between 2 of my 6 VIPs (one per host). If I open a file on either of
> these two "working" hosts, it's locked on the other working one, but
> not on any others. Opening a file on any of the "non-working" hosts
> means the file is not locked anywhere else.
>
> On my test setup with only two nodes it seems to work normally with
> both 4.4.5 and 4.4.6.
>
> All hosts have the same smb.conf.
>
> Cheers
>
> Alex
I've now managed to replicate the error on my test setup. I added a
third host and all hell broke loose - no locking at all when opening
Excel files apart from same-host. It seemed fine with two.
I then downgraded to 4.4.5 and the problem persisted. So it cannot be a
regression from that patch.
Regards,
Alex Crow via samba
> I've now managed to replicate the error on my test setup. I added a
> third host and all hell broke loose - no locking at all when opening
> Excel files apart from same-host. It seemed fine with two.
>
> I then downgraded to 4.4.5 and the problem persisted. So it cannot be
> a regression from that patch.
>
> Regards,
>
> Alex
>
Locked files:
Pid Uid DenyMode Access R/W Oplock
SharePath Name Time
--------------------------------------------------------------------------------------------------
1:11489 1046 DENY_WRITE 0x12019f RDWR NONE
/mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd
Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls
Fri Oct 21 17:27:22 2016
0:9375 1046 DENY_WRITE 0x12019f RDWR NONE
/mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd
Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls
Fri Oct 21 17:26:38 2016
This is what I'm seeing. I could open the same file r/w off two
different hosts. Both seem to have a DENY_WRITE lock - which is surely
supposed to be impossible?
This is on 4.4.5 with only two hosts operational.
Ralph Böhme via samba
Can you try to reproduce the issue with a specially patched smbclient?
Attached patch modifies the smbclient open command to request deny
read and write sharemodes, so any two opens should conflict:
smb: \> open test
open file \test: for read/write fnum 1
smb: \> open test
Failed to open file \test. NT_STATUS_SHARING_VIOLATION
Without the patch the opens would succeed:
smb: \> open test
open file \test: for read/write fnum 1
smb: \> open test
open file \test: for read/write fnum 2
If you can reproduce the issue with smbclient, please set loglevel to
10 on both nodes and reproduce the issue with smbclient. Reproducing
the issue with smbclient instead of Excel reduces the noise and should
make it easier to analyse.
Please setup logging in a way so that we get per client logfiles, eg
"log file = /var/log/samba/%m.log".
Cheerio!
-slow
Alex Crow via samba
> *phew*! At least it's not a regression by the patches for bug 12005.
Hi Ralph,
Looking back through the changelogs suggests to me that it's been around
a long time....
I did however find it odd that even after disabling the 3rd host,
stopping ctdb everywhere and removing *all* the db files (including
persistent) I still could reproduce on 4.4.5.
>
> Can you try to reproduce the issue with a specially patched smbclient?
> Attached patch modifies the smbclient open command to request deny
> read and write sharemodes, so any two opens should conflict:
Luckily I've realised that sernet has source packages so I'll be able to
rebuild the rpms and install individually. Can you tell me if the client
patch should be applied to 4.4.5 or 4.4.6 (or is valid for both)?
>
> smb: \> open test
> open file \test: for read/write fnum 1
>
> smb: \> open test
> Failed to open file \test. NT_STATUS_SHARING_VIOLATION
>
> Without the patch the opens would succeed:
>
> smb: \> open test
> open file \test: for read/write fnum 1
> smb: \> open test
> open file \test: for read/write fnum 2
>
> If you can reproduce the issue with smbclient, please set loglevel to
> 10 on both nodes and reproduce the issue with smbclient. Reproducing
> the issue with smbclient instead of Excel reduces the noise and should
> make it easier to analyse.
>
> Please setup logging in a way so that we get per client logfiles, eg
> "log file = /var/log/samba/%m.log".
That is our existing logging method. We prefer to know which client is
misbehaving and it makes it easy to diagnose. I think it should be a
default!
> Cheerio!
> -slow
Many thanks for your help so far. I'd guess this might also affect
people like Rozo who market appliances with a similar focus to our
internal solution's aims (although we're using another distributed FS -
MooseFS). I don't think we're doing anything quirky - I fact I've
followed the wiki docs very closely.
I've been worrying about our FS but I even tested with "posix locking =
no" and had the same result, so given we pass ping-pong that would rule
that out, yes?
Cheerio to you too. I may be able to get this test done over the weekend
but I have a couple of kids to keep happy ;-)
I have pasted our test smb.conf (anonymised) after this in case I've
done something idiotic.
Cheers
Alex
[global]
workgroup = MY_NET
realm = samba.my.net
netbios name = S4FILES
security = ADS
#bind interfaces only = yes
#interfaces = eth0, lo
#dedicated keytab file = /etc/krb5.keytab
#kerberos method = secrets and keytab
idmap_ldb:use rfc2307 = yes
clustering = yes
#private dir = /mfs/ctdb/private
idmap config *:backend = tdb
idmap config *:range = 200000-299999
idmap config MY_NET:backend = ad
idmap config MY_NET:default = yes
idmap config MY_NET:schema_mode = rfc2307
idmap config MY_NET:range = 500-199999
idmap config ALEX:backend = rid
idmap config ALEX:range = 300000-399999
winbind nss info = rfc2307
winbind trusted domains only = no
winbind use default domain = yes
winbind enum users = yes
winbind enum groups = yes
winbind refresh tickets = Yes
[homes]
comment = Home Directories
valid users = %S
read only = no
browseable = no
path = /mfs/samba/home/%S
[test]
comment = test
path = /mfs/samba/test
read only = no
browseable = yes
oplocks = yes
.. other shares.. (most have oplocks = yes set, but other shares that
don't also suffer the issue
Alex Crow via samba
I applied the patch to a new build of Sernet 4.4.6 SRPM. I could still
open the file from both hosts, and in fact this time I could actually
open it on the same host without any problems:
host 1:
smbclient -U ajc //172.31.0.120/ifa_v
...
smb: \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\> open "201609
213_ ISL Balance sheet variance analysis.xls"
open file \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\201609
213_ ISL Balance sheet variance analysis.xls: for read/write fnum 56287
host 2:
smbclient -U ajc //172.31.0.120/ifa_v
...
smb: \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\> open "201609
213_ ISL Balance sheet variance analysis.xls"
open file \jmi\Corporate Accounting\ISL Ltd Accounts\2016\1609\201609
213_ ISL Balance sheet variance analysis.xls: for read/write fnum 17669
But this:
# smbstatus -L
Locked files:
Pid Uid DenyMode Access R/W
Oplock SharePath Name Time
--------------------------------------------------------------------------------------------------
1:19650 1046 DENY_NONE 0x83 RDWR
NONE /mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd
Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls
Sat Oct 22 19:26:06 2016
0:13503 1046 DENY_NONE 0x83 RDWR
NONE /mfs/samba/drive_v jmi/Corporate Accounting/ISL Ltd
Accounts/2016/1609/201609 213_ ISL Balance sheet variance analysis.xls
Sat Oct 22 18:58:06 2016
Is odd as the patch was applied fine during the RPM build process.
Is there something I need to do when building from sRPMs to make this stick?
Cheers
Alex
On 21/10/16 18:06, Ralph Böhme wrote:
> diff --git a/source3/client/client.c b/source3/client/client.c
> index 831b9bc..28e98af 100644
> --- a/source3/client/client.c
> +++ b/source3/client/client.c
> @@ -2498,12 +2498,12 @@ static int cmd_open(void)
>
> status = cli_ntcreate(targetcli, targetname, 0,
> FILE_READ_DATA|FILE_WRITE_DATA, 0,
> - FILE_SHARE_READ|FILE_SHARE_WRITE, FILE_OPEN,
> + 0, FILE_OPEN,
> 0x0, 0x0, &fnum, NULL);
> if (!NT_STATUS_IS_OK(status)) {
> status = cli_ntcreate(targetcli, targetname, 0,
> FILE_READ_DATA, 0,
> - FILE_SHARE_READ|FILE_SHARE_WRITE, FILE_OPEN,
> + 0, FILE_OPEN,
> 0x0, 0x0, &fnum, NULL);
> if (NT_STATUS_IS_OK(status)) {
> d_printf("open file %s: for read/write fnum %d\n", targetname, fnum);
> -- 2.7.4
Ralph Böhme via samba
On Sat, Oct 22, 2016 at 07:30:01PM +0100, Alex Crow wrote:
> I applied the patch to a new build of Sernet 4.4.6 SRPM. I could still
> open the file from both hosts, and in fact this time I could actually
> open it on the same host without any problems:
I don't see where you're opening the file twice from one smbclient
below. Am I missing something?
And then there's no need to use an xls file, simply create an empty
file in the root of the share with `touch file` and use that.
Is this from an open with the patched smbclient or from Excel?
> Is odd as the patch was applied fine during the RPM build process.
>
> Is there something I need to do when building from sRPMs to make
> this stick?
You could double check in ~/rpmbuild/BUILD/ if the patch was really
applied. If it was, an smbclient open should result in DENY_ALL
denymode in smbstatus -L, eg:
$ ./bin/smbclient -Uslow%x -msmb3 //10.10.11.100/test
Domain=[SLOWSERVER] OS=[] Server=[]
smb: \> open test
open file \test: for read/write fnum 1
$ sudo ./bin/smbstatus -L
Locked files:
Pid Uid DenyMode Access R/W Oplock SharePath Name Time
--------------------------------------------------------------------------------------------------
12757 1000 DENY_ALL 0x3 RDWR NONE /shares/test test Sun Oct 23 10:39:25 2016
Cheerio!
-slow
Alex Crow via samba
On 23/10/16 10:00, Ralph Böhme wrote:
> Alex,
>
> On Sat, Oct 22, 2016 at 07:30:01PM +0100, Alex Crow wrote:
>> I applied the patch to a new build of Sernet 4.4.6 SRPM. I could still
>> open the file from both hosts, and in fact this time I could actually
>> open it on the same host without any problems:
> I don't see where you're opening the file twice from one smbclient
> below. Am I missing something?
Not the same client session, but two sessions connecting to the same host.
>
> And then there's no need to use an xls file, simply create an empty
> file in the root of the share with `touch file` and use that.
Yes, it's just I had been testing with that file the whole time so I had
it in my clipboard...
smbclient.
>
>> Is odd as the patch was applied fine during the RPM build process.
>>
>> Is there something I need to do when building from sRPMs to make
>> this stick?
> You could double check in ~/rpmbuild/BUILD/ if the patch was really
> applied. If it was, an smbclient open should result in DENY_ALL
> denymode in smbstatus -L, eg:
>
> $ ./bin/smbclient -Uslow%x -msmb3 //10.10.11.100/test
> Domain=[SLOWSERVER] OS=[] Server=[]
> smb: \> open test
> open file \test: for read/write fnum 1
>
> $ sudo ./bin/smbstatus -L
> Locked files:
> Pid Uid DenyMode Access R/W Oplock SharePath Name Time
> --------------------------------------------------------------------------------------------------
> 12757 1000 DENY_ALL 0x3 RDWR NONE /shares/test test Sun Oct 23 10:39:25 2016
>
> Cheerio!
> -slow
Does look like it didn't get applied. Odd, as it was in the specfile and
the SOURCES dir and the build finished. I'll check,
Cheers
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Alex Crow via samba
> Does look like it didn't get applied. Odd, as it was in the specfile and
> the SOURCES dir and the build finished. I'll check,
>
> Cheers
>
> Alex
>
>
Hi Ralph,
FYI I fluffed the SPEC file. I added the filename of the patch but not
the line to apply it!
Going out for my b'day lunch now, so I might be a bit too tipsy to try
it today.
For now in prod I've stopped CTDB on all but one non-DR servers so users
should have the correct experience on Monday morning.
This will give us more time to get to the root of the issue.
Cheers for you help so far, it is very much appreciated.
Alex Crow via samba
break (as by definition it accesses the same host).
[root@s4files1 ~]# smbclient -U"MY_NET\ajc" //172.31.0.120/ifa_x
Enter MY_NET\user's password:
Domain=[MY_NET] OS=[Windows 6.1] Server=[Samba 4.4.6-SerNet-RedHat-33.el7]
smb: \> ls
. D 0 Sun Oct 23 19:18:58 2016
.. D 0 Thu Oct 20 19:19:21 2016
test.txt A 0 Sun Oct 23 19:18:58 2016
SW_DVD5_Office_2010_W32_English_MLF_X16-51904 D 0 Wed Mar
31 03:02:55 2010
171358464 blocks of size 1024. 167913856 blocks available
smb: \> open test.txt
open file \test.txt: for read/write fnum 50796
smb: \> open test.txt
Failed to open file \test.txt. NT_STATUS_SHARING_VIOLATION
smb: \>
OK, so the locking works fine from smbclient just pointing to one host
in the CTDB cluster.
So now I'll try accessing the same file again, but using the second host
(leaving the first client open):
[root@s4files2 ~]# smbclient -U"MY_NET\ajc" //172.31.0.121/ifa_x
Enter MY_NET\ajc's password:
Domain=[MY_NET] OS=[Windows 6.1] Server=[Samba 4.4.6-SerNet-RedHat-33.el7]
smb: \> ls
. D 0 Sun Oct 23 19:18:58 2016
.. D 0 Thu Oct 20 19:19:21 2016
test.txt A 0 Sun Oct 23 19:18:58 2016
SW_DVD5_Office_2010_W32_English_MLF_X16-51904 D 0 Wed Mar
31 03:02:55 2010
171358464 blocks of size 1024. 167913856 blocks available
smb: \> open test.txt
open file \test.txt: for read/write fnum 26479
smb: \>
Bingo - seems smbd does not care about the CTDB locks reported by:
[root@s4files1 ~]# smbstatus -L
Locked files:
Pid Uid DenyMode Access R/W
Oplock SharePath Name Time
--------------------------------------------------------------------------------------------------
0:12886 1046 DENY_ALL 0x83 RDWR
NONE /mfs/samba/drive_x test.txt Sun Oct 23 19:27:17 2016
1:1593 1046 DENY_ALL 0x83 RDWR
NONE /mfs/samba/drive_x test.txt Sun Oct 23 19:19:13 2016
When I get the chance I will up the server logs to level 10 and get back
to you.
Cheers
Alex
Alex Crow via samba
> On Tue, Nov 01, 2016 at 09:56:06PM +0000, Alex Crow wrote:
>>> well, yes and no. I did find some time to look into those logs and
>>> those just tell use, that the second smbd on the second node that
>>> fetches the lock record from the (clustered) locking db doesn't see
>>> the record from the first node. ctdb debug logs might be telling, so
>>> posting a full set of smbd and ctdb logs to the mailing list would be
>>> the next step.
>>>
>>> Currently I have no idea how this can happen, getting to the bottom of
>>> this is certainly fun and entertaining, but also very time
>>> consuming. At the moment I don't have the time to do this in my free
>>> time, maybe someone else from the mailing list has.
>>>
>>> -slow
>> Hi Ralph,
>>
>> Thanks for that. Can I post the above back to the ML with links to
>> appropriate logs?
> sure.
>
> Btw, I forgot to mention that I did test this on my own devel cluster,
> both development version from git master as well as production 4.4,
> and could not reproduce the issue.
>
> -slow
Is anyone able to assist with this issue? I have produced level 10 logs
but not sure how to debug CTDB.
As a quick reminder the problem is that locks on one server are not
being honoured when a client connects to another server in the cluster.
Cheers
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Alex Crow via samba
FYI When I reset my cluster for two nodes and upgraded from 4,4,6 to
4.4.7, restarted all services the locking worked between two clients
accessing different servers.
So my next step was to add back the 3rd server. At this point, clients
would see locking working OK on the 1st and 2nd servers but opening the
same file from the 3rd server from the 2nd client allowed all access.
It's almost as if if you establish a cluster with 2 CTDB servers
everything works OK but it all goes to hell when you add a 3rd member.,
In all cases the file was initially opened R/W from the 1st client to
the 1st server.
To clarify: by "Nth server" I mean the virtual IPs set up by CTDB in
numerical order.
I would hope this helps anyone that's looking at this.
Best regards
Martin Schwenke via samba
On Wed, 2 Nov 2016 11:29:13 +0000, Alex Crow via samba
<sa...@lists.samba.org> wrote:
> On 02/11/16 07:29, Ralph Böhme wrote:
>
> > On Tue, Nov 01, 2016 at 09:56:06PM +0000, Alex Crow wrote:
> [...]
> >> Hi Ralph,
> >>
> >> Thanks for that. Can I post the above back to the ML with links to
> >> appropriate logs?
> > sure.
> >
> > Btw, I forgot to mention that I did test this on my own devel cluster,
> > both development version from git master as well as production 4.4,
> > and could not reproduce the issue.
> >
> > -slow
>
> Is anyone able to assist with this issue? I have produced level 10 logs
> but not sure how to debug CTDB.
If you lock the file from a client attached to 1 node and
"smbstatus -L" shows the lock from all the nodes then CTDB is doing
what it is meant to. That is, it makes sure that Samba processes see
the same TDB records on all nodes.
Not sure what's going on with the rest of it. Doing a bit of testing...
peace & happiness,
martin
Ralph Böhme via samba
On Thu, Nov 03, 2016 at 05:34:21PM +1100, Martin Schwenke via samba wrote:
> On Wed, 2 Nov 2016 11:29:13 +0000, Alex Crow via samba
> <sa...@lists.samba.org> wrote:
>
> > On 02/11/16 07:29, Ralph Böhme wrote:
> >
> > > On Tue, Nov 01, 2016 at 09:56:06PM +0000, Alex Crow wrote:
> > [...]
> > >> Hi Ralph,
> > >>
> > >> Thanks for that. Can I post the above back to the ML with links to
> > >> appropriate logs?
> > > sure.
> > >
> > > Btw, I forgot to mention that I did test this on my own devel cluster,
> > > both development version from git master as well as production 4.4,
> > > and could not reproduce the issue.
> > >
> > > -slow
> >
> > Is anyone able to assist with this issue? I have produced level 10 logs
> > but not sure how to debug CTDB.
>
> If you lock the file from a client attached to 1 node and
> "smbstatus -L" shows the lock from all the nodes then CTDB is doing
> what it is meant to. That is, it makes sure that Samba processes see
> the same TDB records on all nodes.
hm, smbstatus does a db traverse while smbclient will trigger a
migrate record, so both use different ctdb protcol ops, don't they?
> Not sure what's going on with the rest of it. Doing a bit of testing...
I tested on a 3 node test cluster, with 4.4.x and master, works as
expected.
-slow
Martin Schwenke via samba
<sa...@lists.samba.org> wrote:
> On Thu, Nov 03, 2016 at 05:34:21PM +1100, Martin Schwenke via samba wrote:
> > If you lock the file from a client attached to 1 node and
> > "smbstatus -L" shows the lock from all the nodes then CTDB is doing
> > what it is meant to. That is, it makes sure that Samba processes see
> > the same TDB records on all nodes.
>
> hm, smbstatus does a db traverse while smbclient will trigger a
> migrate record, so both use different ctdb protcol ops, don't they?
Sure. I was glossing over a lot of details... but I hope we would
notice major breakage like that... :-)
> > Not sure what's going on with the rest of it. Doing a bit of testing...
>
> I tested on a 3 node test cluster, with 4.4.x and master, works as
> expected.
Yay!
peace & happiness,
martin
Alex Crow via samba
On 03/11/16 08:51, Martin Schwenke via samba wrote:
> On Thu, 3 Nov 2016 08:13:53 +0100, Ralph Böhme via samba
> <sa...@lists.samba.org> wrote:
>
>> On Thu, Nov 03, 2016 at 05:34:21PM +1100, Martin Schwenke via samba wrote:
>>> If you lock the file from a client attached to 1 node and
>>> "smbstatus -L" shows the lock from all the nodes then CTDB is doing
>>> what it is meant to. That is, it makes sure that Samba processes see
>>> the same TDB records on all nodes.
>> hm, smbstatus does a db traverse while smbclient will trigger a
>> migrate record, so both use different ctdb protcol ops, don't they?
> Sure. I was glossing over a lot of details... but I hope we would
> notice major breakage like that... :-)
>
>>> Not sure what's going on with the rest of it. Doing a bit of testing...
>> I tested on a 3 node test cluster, with 4.4.x and master, works as
>> expected.
> Yay!
>
> peace & happiness,
> martin
>
This is very strange. I've tested with 3 separate clusters and can
replicate the problem every time if a 3rd node is in the picture.
The filesystem could not be an issue could it? It passes ping_pong just
fine.
Cheers
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Alex Crow via samba
>> Not sure what's going on with the rest of it. Doing a bit of testing...
> I tested on a 3 node test cluster, with 4.4.x and master, works as
> expected.
>
> -slow
>
Hi there,
Can you send your smb.conf so I can compare it to mine?
Cheers,
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Ralph Böhme via samba
>
> > > Not sure what's going on with the rest of it. Doing a bit of testing...
> > I tested on a 3 node test cluster, with 4.4.x and master, works as
> > expected.
> >
> > -slow
> >
>
> Hi there,
>
> Can you send your smb.conf so I can compare it to mine?
sure. It's a virtual cluster in LXC containers with 3 nodes,
filesystem is a bind mount with ext4.
# cat /var/lib/lxc/ctdb-node0/rootfs/opt/samba/etc/smb.conf
[global]
clustering = yes
ctdbd socket = /opt/samba/var/run/ctdb/ctdbd.socket
log file = /var/log/samba/log.%m
debug pid = yes
log level = 2
netbios name = CTDB-CLUSTER
workgroup = CLUSTER
security = user
idmap config *:backend = tdb
idmap config *:range = 10000-99999
store dos attributes = yes
read only = no
ea support = yes
include = registry
disable spoolss = yes
[test]
path = /cluster/data/share
-slow
Alex Crow via samba
Ah, my smb.conf is missing the "ctdb socket" parameter. According to
"man smb.conf" this should be set, but I don't remember seeing it on the
wiki pages.
I'll have a go with that and report back.
Cheers
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Ralph Böhme via samba
if it's not set, the default is used which works in many cases, but
not in mine. If the default wasn't working in your setup, you'd
notice, as clients wouldn't be able to talk to ctdb at all.
-slow
Alex Crow via samba
On 03/11/16 14:28, Ralph Böhme wrote:
>
>> Ah, my smb.conf is missing the "ctdb socket" parameter. According to "man
>> smb.conf" this should be set, but I don't remember seeing it on the wiki
>> pages.
> if it's not set, the default is used which works in many cases, but
> not in mine. If the default wasn't working in your setup, you'd
> notice, as clients wouldn't be able to talk to ctdb at all.
>
> -slow
I can confirm this made no difference. I see the locking working from
windows between the first two servers but *not* the third. This seems to
match what I had with smbclient. Again, could it be that an FS that
passes ping_pong not be able to work in a CTDB cluster?
The developers of my fs (MooseFS) have said that while file flock() is
working perfectly there is still no directory flock. Could that be a
pointer?
Cheers,
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Volker Lendecke via samba
>
>
> On 03/11/16 14:28, Ralph Böhme wrote:
> >
> >> Ah, my smb.conf is missing the "ctdb socket" parameter. According to "man
> >> smb.conf" this should be set, but I don't remember seeing it on the wiki
> >> pages.
> > if it's not set, the default is used which works in many cases, but
> > not in mine. If the default wasn't working in your setup, you'd
> > notice, as clients wouldn't be able to talk to ctdb at all.
> >
> > -slow
>
> I can confirm this made no difference. I see the locking working from
> windows between the first two servers but *not* the third. This seems to
> match what I had with smbclient. Again, could it be that an FS that
> passes ping_pong not be able to work in a CTDB cluster?
>
> The developers of my fs (MooseFS) have said that while file flock() is
> working perfectly there is still no directory flock. Could that be a
> pointer?
>
Does your system have consistent inodes and device ids? Does
stat /some/shared/file
give exactly the same information on all nodes, in particular the
Device/Inode fields?
Volker
Alex Crow via samba
On 03/11/16 19:37, Volker Lendecke wrote:
> On Thu, Nov 03, 2016 at 07:11:12PM +0000, Alex Crow via samba wrote:
>>
>> On 03/11/16 14:28, Ralph Böhme wrote:
>>>> Ah, my smb.conf is missing the "ctdb socket" parameter. According to "man
>>>> smb.conf" this should be set, but I don't remember seeing it on the wiki
>>>> pages.
>>> if it's not set, the default is used which works in many cases, but
>>> not in mine. If the default wasn't working in your setup, you'd
>>> notice, as clients wouldn't be able to talk to ctdb at all.
>>>
>>> -slow
>> I can confirm this made no difference. I see the locking working from
>> windows between the first two servers but *not* the third. This seems to
>> match what I had with smbclient. Again, could it be that an FS that
>> passes ping_pong not be able to work in a CTDB cluster?
>>
>> The developers of my fs (MooseFS) have said that while file flock() is
>> working perfectly there is still no directory flock. Could that be a
>> pointer?
>>
> Does your system have consistent inodes and device ids? Does
>
> stat /some/shared/file
>
> give exactly the same information on all nodes, in particular the
> Device/Inode fields?
>
> Volker
Thanks Volker,
No, it seems it's OK on two servers but not on the third:
[root@zalma ~]# stat testfile
...
Size: 1286700 Blocks: 2514 IO Block: 65536 regular file
Device: 29h/41d Inode: 35820037 Links: 1
Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users)
Access: 2016-11-03 19:51:46.000000000 +0000
Modify: 2016-11-01 13:06:04.000000000 +0000
Change: 2016-11-01 13:06:04.000000000 +0000
Birth: -
[root@zalma ~]# ssh zearing "stat testfile"
root@zearing's password:
...
Size: 1286700 Blocks: 2514 IO Block: 65536 regular file
Device: 29h/41d Inode: 35820037 Links: 1
Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users)
Access: 2016-11-03 19:51:46.000000000 +0000
Modify: 2016-11-01 13:06:04.000000000 +0000
Change: 2016-11-01 13:06:04.000000000 +0000
Birth: -
[root@zalma ~]# ssh metropolis "stat testfile"
root@metropolis's password:
...
Size: 1286700 Blocks: 2514 IO Block: 65536 regular file
Device: 26h/38d Inode: 35820037 Links: 1
Access: (0774/-rwxrwxr--) Uid: ( 3535/ jh3) Gid: ( 513/Domain Users)
Access: 2016-11-03 19:51:46.000000000 +0000
Modify: 2016-11-01 13:06:04.000000000 +0000
Change: 2016-11-01 13:06:04.000000000 +0000
Birth: -
NB this is a FUSE mounted FS.
What does the Device: XXh/YYd mean?
Cheers
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Jeremy Allison via samba
Ah, well that explains a lot :-).
Samba uses the device/ino pair to determine if two files
are the same. If they're different - to smbd then they're
different files.
That's why the locking doesn't work.
Alex Crow via samba
Your and Volker's insight much appreciated. Thanks to Ralph and everyone else who's piled into this!
Goodnight,
Alex
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Volker Lendecke via samba
That's the so-called device number. Unix concept from the early days.
You can use
vfs objects = fileid
and
fileid:mapping = fsname
to fix this issue.
Volker
Alex Crow via samba
Can that go in the global section so it applies to all shares?
And this should fix the CTDB locking problem? If so, brilliant!
Cheers
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Volker Lendecke via samba
> Can that go in the global section so it applies to all shares?
Yep.
> And this should fix the CTDB locking problem? If so, brilliant!
Just try :-)
Volker
Alex Crow via samba
> On Fri, Nov 04, 2016 at 07:20:22AM +0000, Alex Crow wrote:
>> Can that go in the global section so it applies to all shares?
> Yep.
>
>> And this should fix the CTDB locking problem? If so, brilliant!
> Just try :-)
>
> Volker
It wooooorks!!! Fantastic. I've been banging my head against the wall
for the best part of a month on that, and the fix is so simple.
However I will also get in touch with the FS devs and tell them about
the difference in the Device ID.
Thanks very much to all who have helped get to the bottom of this, much
appreciated.
Cheers,
Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
--
Jeremy Allison via samba
Well that's an interesting little vfs module I don't
remember existing ! You learn something new every day :-).
I still think your filesystem should try and get dev/ino
pairs the same on different nodes though in the long run.
Martin Schwenke via samba
I'm going to document this in the wiki (at current non-existent
https://wiki.samba.org/index.php?title=Setting_up_a_cluster_filesystem&action=edit&redlink=1).
It is already mentioned under OCFS2 section
(https://wiki.samba.org/index.php/CTDB_Setup#OCFS2) without explanation.
According to
https://www.samba.org/samba/docs/man/manpages/vfs_fileid.8.html
which I will point to from the wiki, the modern way seems to be to use
fileid:algorithm = fsname
instead of
fileid:mapping = fsname
Just a few quick questions:
* This is only done at connect time so there's no performance penalty?
That's my reading of the code but I didn't want to turn this into
a "let's learn all about the VFS" exercise. :-)
* If the device name is different across nodes then this doesn't help.
I was wondering why there isn't an choice to hash m->mnt_dir since
that will always be the same on each node (if you want a consistent
share definition across the cluster). Perhaps there's no cluster
filesystem so far that has needed this but has had inconsistent
device names?
* Is there anything similar for inconsistent/unstable inode numbers?
That would clearly come with a performance penalty... unless
caching, which is always an easy problem to solve... :-)
Thanks...
peace & happiness,
martin