[Samba] Samba AD/DC crashed again, third time in as many months

[IT Admin]

Hello to the samba users list again. I've got an emergency here, a Samba
AD DC I've got deployed has crashed again, this is the third time since
12/17/2015 that this domain has failed completely.

No power outages or unexpected shutdowns, samba simply fails to start, it
appears my database is yet again corrupt:

[2016/03/02 11:27:17.025235, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.025346, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.025377, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.025404, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.025431, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.025460, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.032681, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.033626, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
[2016/03/02 11:27:17.033776, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [nbtd failed to open samdb]
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.034676, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.035224, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.035838, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.036392, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.038675, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
[2016/03/02 11:27:17.040353, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
[2016/03/02 11:27:17.042764, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.042857, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.042887, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.042917, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.042942, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.042971, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.045981, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.046598, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.047302, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
[2016/03/02 11:27:17.047523, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [dreplsrv: Failed to connect to local samdb:
WERR_DS_UNAVAILABLE
]
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.048503, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.048540, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.050402, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [kdc: krb5_init_context samdb connect failed]
task_server_terminate: [cldapd failed to open samdb]
[2016/03/02 11:27:17.072670, 0]
../source4/smbd/server.c:211(samba_terminate)
samba_terminate: dreplsrv: Failed to connect to local samdb:
WERR_DS_UNAVAILABLE

[2016/03/02 11:27:17.073896, 0]
../source4/smbd/server.c:211(samba_terminate)
samba_terminate: nbtd failed to open samdb
[2016/03/02 11:27:17.075031, 0]
../source4/smbd/server.c:211(samba_terminate)
samba_terminate: cldapd failed to open samdb
[2016/03/02 11:27:17.075614, 0]
../source4/smbd/server.c:211(samba_terminate)
samba_terminate: kdc: krb5_init_context samdb connect failed
[2016/03/02 11:27:17.086051, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.091786, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.091860, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.091890, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.091919, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.091946, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.091974, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.093919, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [dns: samdb_connect failed]
[2016/03/02 11:27:17.095175, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.095232, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.095261, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.095288, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.095317, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.095940, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.097255, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
[2016/03/02 11:27:17.098404, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
task_server_terminate: [kccsrv: Failed to connect to local samdb:
WERR_DS_UNAVAILABLE
]
[2016/03/02 11:27:17.102052, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.102608, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.103164, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.103210, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.103241, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.103939, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.103974, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.104003, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.107759, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [dnsupdate: Failed to connect to local samdb
]
[2016/03/02 11:27:17.109612, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.109672, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.114754, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [ntp_signd failed to open samdb]
[2016/03/02 11:27:17.731614, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.731721, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.731750, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.731779, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.731807, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.731835, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.762162, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: dsdb_get_schema: refresh_fn() failed
[2016/03/02 11:27:17.762247, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.762276, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module schema_load initialization failed : Operations error
[2016/03/02 11:27:17.762305, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module rootdse initialization failed : Operations error
[2016/03/02 11:27:17.762332, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: module samba_dsdb initialization failed : Operations error
[2016/03/02 11:27:17.762361, 0]
../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
ldb: Unable to load modules for /usr/local/samba/private/sam.ldb:
schema_load_init: dsdb_get_schema failed
[2016/03/02 11:27:17.763923, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [Failed to startup ldap server task]
[2016/03/02 11:27:18.247740, 0]
../file_server/file_server.c:46(file_server_smbd_done)
file_server smbd daemon died with exit status 1
[2016/03/02 11:27:18.247875, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [smbd child process exited]
[2016/03/02 11:27:23.987725, 0]
../source4/winbind/winbindd.c:49(winbindd_done)
winbindd daemon exited normally
[2016/03/02 11:27:23.989217, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [winbindd child process exited]

Relevant info:

itwerks@filer:~$ samba -V
Version 4.3.3
itwerks@filer:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.10
Release: 15.10
Codename: wily
itwerks@filer:~$ uname -a
Linux filer 4.2.0-30-generic #36-Ubuntu SMP Fri Feb 26 00:58:07 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux
itwerks@filer:~$

Please advise. It is end of year for this company, I simply cannot afford
failures like this every three to four weeks, and I cannot blow the entire
domain out again and rebuild from scratch.

Frustrated.

JS
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[IT Admin]

I should mention that I have a backup generated using samba_backup script
from 2/27/2016. I'm guessing I need to restore from this backup to
recover.

Please advise, these failures are killing me.

JS

[Marc Muehlfeld]

Hello,

Am 02.03.2016 um 17:39 schrieb IT Admin:
> ../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
> ldb: dsdb_get_schema: refresh_fn() failed
> [2016/03/02 11:27:17.042857, 0]
> ../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
> ldb: schema_load_init: dsdb_get_schema failed
> [2016/03/02 11:27:17.042887, 0]
> ../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
> ldb: module schema_load initialization failed : Operations error

* Did you made any schema modifications or joined a 2012 or later DC?

* Is this the only DC in your forest?


Regards,
Marc

[IT Admin]

Hi Marc,

No, no schema adjustments. This is the only domain controller, there have
been no changes to this domain since I rebuilt it about a month ago. Samba
keeps failing like this almost every month.

JS

[Rowland penny]

On 02/03/16 17:07, IT Admin wrote:
> Hi Marc,
>
> No, no schema adjustments. This is the only domain controller, there have
> been no changes to this domain since I rebuilt it about a month ago. Samba
> keeps failing like this almost every month.
>
> JS
> On Mar 2, 2016 11:59 AM, "Marc Muehlfeld" <mmueh...@samba.org> wrote:
>
>> Hello,
>>
>> Am 02.03.2016 um 17:39 schrieb IT Admin:
>>> ../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
>>> ldb: dsdb_get_schema: refresh_fn() failed
>>> [2016/03/02 11:27:17.042857, 0]
>>> ../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
>>> ldb: schema_load_init: dsdb_get_schema failed
>>> [2016/03/02 11:27:17.042887, 0]
>>> ../lib/ldb-samba/ldb_wrap.c:72(ldb_wrap_debug)
>>> ldb: module schema_load initialization failed : Operations error
>>
>> * Did you made any schema modifications or joined a 2012 or later DC?
>>
>> * Is this the only DC in your forest?
>>
>>
>> Regards,
>> Marc
>>

If I remember correctly, aren't you the guy that had the problem with
provision not finding the schema files ?

Can you run this in a terminal:

env | grep PATH

and post the result ?

Rowland

[Marc Muehlfeld]

Am 02.03.2016 um 18:07 schrieb IT Admin:
> No, no schema adjustments. This is the only domain controller, there have
> been no changes to this domain since I rebuilt it about a month ago. Samba
> keeps failing like this almost every month.

Can you tell us some more information about your environment
* Samba DBs are stored on a local HDD, NFS, Gluster, iSCSI,...
* Is the AD only used for authentication? Or do other applications are
connected, too? Especially if there are any that are trying to write to AD
* How do you do changes in AD? ADUC? samba-tool, other tools?
* etc.

[IT Admin]

Samba is compiled from source.
Samba DB is stored on local RAID array.
Changes to AD are done using ADUC from a Windows 7 box.
AD is used for authentication, user shares (folder redirection), and shared
folders.

JS

[IT Admin]

Hi Rowland,

I'm the guy who had this exact issue in January, you worried extensively
with me to resolve the issue. We ended up building samba from source,
provisioning again (which led to rooting out remnants of the older samba
release), and then you worked with me to help me get samba_backup
implemented.

Here's the output of the command you requested I run:

PATH=/usr/local/samba/bin:/usr/local/samba/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/samba/bin:/usr/local/samba/sbin

JS

[Rowland penny]

On 02/03/16 17:49, IT Admin wrote:
>
> Hi Rowland,
>
> I'm the guy who had this exact issue in January, you worried
> extensively with me to resolve the issue. We ended up building samba
> from source, provisioning again (which led to rooting out remnants of
> the older samba release), and then you worked with me to help me get
> samba_backup implemented.
>
> Here's the output of the command you requested I run:
>
> PATH=/usr/local/samba/bin:/usr/local/samba/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/samba/bin:/usr/local/samba/sbin
>
> JS
>
>

Hmm, that shows you should be using the correct Samba files.

Can you answer Marc's questions plus tell us if Apparmor is running.

[IT Admin]

Apparmor status:

sudo apparmor_status
[sudo] password for itwerks:
apparmor module is loaded.
7 profiles are loaded.
7 profiles are in enforce mode.
/sbin/dhclient
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/NetworkManager/nm-dhcp-helper
/usr/lib/connman/scripts/dhclient-script
/usr/sbin/mysqld
/usr/sbin/ntpd
/usr/sbin/tcpdump
0 profiles are in complain mode.
1 processes have profiles defined.
1 processes are in enforce mode.
/usr/sbin/mysqld (1443)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

JS

[IT Admin]

Any further information required? Any troubleshooting steps? I need to
get this domain functional ASAP.

Please advise.

JS

On Mar 2, 2016 11:39 AM, "IT Admin" <i...@cliffbells.com> wrote:

[Rowland penny]

Have you considered that it may be a hardware problem ? or is your raid
setup ok . You could also try turning Apparmor off as a test (Is there
anything in syslog about Samba and Apparmor).

Your problem is not a normal one, or one that is common, it may help if
we could see your smb.conf (you may have posted it before, but if so,
please post it again)

[mj]

> On 02/03/16 20:46, IT Admin wrote:
>> Any further information required? Any troubleshooting steps? I need to
>> get this domain functional ASAP.
>>
>> Please advise.
>>
>> JS

I would also install a second DC. It would give you redundancy, and if
the second one does NOT fail when the first one does...then at least you
know that the problem is specific to that one DC.

(i realise that this does not help you to get the domain back up, now)

[IT Admin]

I built this machine, and while it isn't the most robust box in the world
it has been stable otherwise. The RAID array is configured RAID1, I can't
see how that could cause corruption issues and I haven't experienced any
other data corruption issues apart from SAMBA collapsing. I did start
seeing samba failures about two weeks ago, restarting the machine brought
the domain up in those cases, associated errors revolved around samba dns
functions iirc. I have intended to upgrade to the latest release (which
went public a week after the last rebuild), I had hoped that update would
eradicate the errors I was experiencing.

My smb.conf:

cat /etc/samba/smb.conf
# Global parameters
[global]
workgroup = CB
realm = CB.CLIFFBELLS.COM
netbios name = FILER
server role = active directory domain controller
dns forwarder = 192.168.37.254
allow dns updates = nonsecure and secure
idmap_ldb:use rfc2307 = yes
printing = CUPS
printcap name = /dev/null
# below added from dragon.org guide
idmap config *:backend = tdb
# This local range must not overlap the BLACK:range above
idmap config *:range = 5000-9999
idmap config CB:backend = ad
idmap config CB:schema_mode = rfc2307
idmap config CB:range = 10000-29999
# Use home directory and shell information from AD
winbind nss info = rfc2307

[netlogon]
path = /usr/local/samba/var/locks/sysvol/cb.cliffbells.com/scripts
read only = No

[sysvol]
path = /usr/local/samba/var/locks/sysvol
read only = No
[Users]
directory_mode: parameter = 0700
read only = no
path = /storage/Users
csc policy = documents

The above smb.conf has been in use since late January, and the following I
cannot explain... There should be declarations for "data" and "backups" in
that file but they have magically vanished. I worked at this location last
weekend and know for a fact that those shares were live and accessible.

It appears an update mangled the smb.conf file on the 17th of february:

ls -la /etc/samba/
total 68
drwxr-xr-x 3 root root 4096 Mar 2 16:34 .
drwxr-xr-x 135 root root 12288 Mar 2 06:51 ..
-rw-r--r-- 1 root root 8 Nov 12 13:07 gdbcommands
-rw-r--r-- 1 root root 972 Feb 17 06:34 smb.conf
-rw-r--r-- 1 root root 998 Dec 13 19:44 smb.conf.BAD
-rw-r--r-- 1 root root 345 Jan 10 04:36 smb.conf.BADDY
-rw-r--r-- 1 root root 1529 Jan 6 06:54 smb.conf.ONE.CLIFFBELLS.COM
-rw-r--r-- 1 root root 9542 Dec 7 18:43 smb.conf.ORIG
-rw-r--r-- 1 root root 345 Jan 8 21:24 smb.conf.SRC
-rw-r--r-- 1 root root 9535 Feb 17 06:34 smb.conf.ucf-dist
drwxr-xr-x 2 root root 4096 Nov 12 13:06 tls

I'm going to restore it to an earlier version and reboot.

JS

[Rowland penny]

On 02/03/16 21:42, IT Admin wrote:
>
> I built this machine, and while it isn't the most robust box in the
> world it has been stable otherwise. The RAID array is configured
> RAID1, I can't see how that could cause corruption issues and I
> haven't experienced any other data corruption issues apart from SAMBA
> collapsing. I did start seeing samba failures about two weeks ago,
> restarting the machine brought the domain up in those cases,
> associated errors revolved around samba dns functions iirc. I have
> intended to upgrade to the latest release (which went public a week
> after the last rebuild), I had hoped that update would eradicate the
> errors I was experiencing.
>

I have never trusted raid since the server at the place I was working
went down, this was traced to one of the HDs dying. the raid was
composed of 4 HDs, two in stripped mode, backed up by the other two, if
one set was faulty the other set was supposed to take over, only they
didn't, took three days to get the server back up.

> My smb.conf:
>
> cat /etc/samba/smb.conf
> # Global parameters
> [global]
> workgroup = CB

> realm = CB.CLIFFBELLS.COM <http://CB.CLIFFBELLS.COM>

> netbios name = FILER
> server role = active directory domain controller
> dns forwarder = 192.168.37.254
> allow dns updates = nonsecure and secure
> idmap_ldb:use rfc2307 = yes
> printing = CUPS
> printcap name = /dev/null
>

> # below added from dragon.org <http://dragon.org> guide

> idmap config *:backend = tdb
> # This local range must not overlap the BLACK:range above
> idmap config *:range = 5000-9999
> idmap config CB:backend = ad
> idmap config CB:schema_mode = rfc2307
> idmap config CB:range = 10000-29999
> # Use home directory and shell information from AD
> winbind nss info = rfc2307
>

You might as well remove the above 9 lines from smb.conf, they do
*nothing* on a DC.

> [netlogon]
> path =
> /usr/local/samba/var/locks/sysvol/cb.cliffbells.com/scripts
> <http://cb.cliffbells.com/scripts>

> read only = No
>
> [sysvol]
> path = /usr/local/samba/var/locks/sysvol
> read only = No
> [Users]
> directory_mode: parameter = 0700
> read only = no
> path = /storage/Users
> csc policy = documents
>
> The above smb.conf has been in use since late January, and the
> following I cannot explain... There should be declarations for "data"
> and "backups" in that file but they have magically vanished. I worked
> at this location last weekend and know for a fact that those shares
> were live and accessible.
>
> It appears an update mangled the smb.conf file on the 17th of february:
>

AHA, I take you are talking about an Ubuntu update, as you are using a
self compiled version of Samba, no Ubuntu update should touch smb.conf,
or are you talking about some other update ?


> ls -la /etc/samba/
>

Why is smb.conf in /etc/samba/ ? It should be in /usr/local/samba/etc/

Rowland

> total 68
> drwxr-xr-x 3 root root 4096 Mar 2 16:34 .
> drwxr-xr-x 135 root root 12288 Mar 2 06:51 ..
> -rw-r--r-- 1 root root 8 Nov 12 13:07 gdbcommands
> -rw-r--r-- 1 root root 972 Feb 17 06:34 smb.conf
> -rw-r--r-- 1 root root 998 Dec 13 19:44 smb.conf.BAD
> -rw-r--r-- 1 root root 345 Jan 10 04:36 smb.conf.BADDY
> -rw-r--r-- 1 root root 1529 Jan 6 06:54

> smb.conf.ONE.CLIFFBELLS.COM <http://smb.conf.ONE.CLIFFBELLS.COM>

> -rw-r--r-- 1 root root 9542 Dec 7 18:43 smb.conf.ORIG
> -rw-r--r-- 1 root root 345 Jan 8 21:24 smb.conf.SRC
> -rw-r--r-- 1 root root 9535 Feb 17 06:34 smb.conf.ucf-dist
> drwxr-xr-x 2 root root 4096 Nov 12 13:06 tls
>
> I'm going to restore it to an earlier version and reboot.
>
> JS
>
>

[IT Admin]

I forgot that when compiled from source samba looks at /usr/local/samba/etc
for smb.conf, not /etc/samba, ignore my confusion re: mangled conf file,
the actual smb.conf is correct and follows:

cat /usr/local/samba/etc/smb.conf

# Global parameters
[global]
workgroup = CB
realm = CB.CLIFFBELLS.COM
netbios name = FILER
server role = active directory domain controller
dns forwarder = 192.168.37.254

idmap_ldb:use rfc2307 = yes
printing = CUPS
printcap name = /dev/null
# below added from dragon.org guide
idmap config *:backend = tdb
# This local range must not overlap the BLACK:range above
idmap config *:range = 5000-9999
idmap config CB:backend = ad
idmap config CB:schema_mode = rfc2307
idmap config CB:range = 10000-29999
# Use home directory and shell information from AD
winbind nss info = rfc2307

[netlogon]
path = /usr/local/samba/var/locks/sysvol/cb.cliffbells.com/scripts
read only = No

[sysvol]
path = /usr/local/samba/var/locks/sysvol
read only = No

[Users]
directory_mode: parameter = 0700
read only = no
path = /storage/Users
csc policy = documents

[data]
writeable = yes
directory mode = 777
delete readonly = yes
comment = Cliff Bell's Shared Data Directory
path = /storage/data
create mode = 777

[backups]
comment = Cliff Bell's Backup Directory
path = /storage/backups
directory mode = 777
delete readonly = yes
writeable = yes
create mode = 777

JS

[Marc Muehlfeld]

Am 02.03.2016 um 18:52 schrieb IT Admin:
> Samba is compiled from source.
> Samba DB is stored on local RAID array.
> Changes to AD are done using ADUC from a Windows 7 box.
> AD is used for authentication, user shares (folder redirection), and shared
> folders.

Can you add
log level = 10
to your smb.conf, empty your log directory and start Samba. It should
generate a new log, that captures all output. Then put it to cpaste.org
or some other paste service and share the link with us. Maybe we see
something interesting.


Does something changed when the problem occured the first time? Some
package updates, crashes, etc.?


Can you make sure that no kind of Samba package (daemon, libs, etc.) is
installed on the system? Maybe your selfcompiled version overwrites some
stuff and your OS installed an update, that mixes now with the self
compiled version. Just a guess.

[IT Admin]

Hi Marc,

I added "log level = 10" to my smb.conf, the new log is here:
http://www.anonpaste.net/?p=f1ec8

The initial failure of this domain occurred in mid-december, I associated
it with a failure of the raid array after an ubuntu upgrade, however, I
suspect that wasn't really the case. I rebuilt the domain, it failed again
about a week later with roughly the same behaviour as I'm seeing now. I
worked with the samba-users group over a period of about three weeks to
troubleshoot the issue, we settled on my database being corrupted beyond
repair, at the time I believed CrashPlan backup was to blame, but again I
suspect that may not have been the case, or at least it wasn't the only
issue in play. I removed all Canonical samba packages, downloaded the
latest samba source from samba.org, compiled and installed samba 4.3.3. I
then provisioned the current domain and rebuilt their infrastructure after
which I worked with Rowland a bit to implement the samba_backup script.
The domain seemed fine up until about two weeks ago at which time I started
seeing failures in log.samba, typically referencing dns update failures and
NT_STATUS_UNAVAILABLE etc. A reboot of the DC always brought samba back, I
had hoped upgrading to the latest release would alleviate those issues.
Unfortunately the entire thing fell apart this morning, samba failing to
start, their entire domain is noon-functional.

I'm not sure how to verify all other unwanted samba packages are gone, what
would you suggest?

I look forward to any guidance you may be able to offer.

[IT Admin]

I poked around the system using locate, I don't think there are any
packages installed on this system other than those associated with the
4.3.3 build I compiled.

smbd

/usr/lib/python2.7/dist-packages/samba/samba3/smbd.so
/usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0
/usr/lib/x86_64-linux-gnu/samba/libsmbd_conn.so.0
/usr/lib/x86_64-linux-gnu/samba/libsmbd_shim.so.0
/usr/local/samba/lib/private/libsmbd-base-samba4.so
/usr/local/samba/lib/private/libsmbd-conn-samba4.so
/usr/local/samba/lib/private/libsmbd-shim-samba4.so
/usr/local/samba/lib/python2.7/site-packages/samba/samba3/smbd.so
/usr/local/samba/private/smbd.tmp
/usr/local/samba/sbin/smbd
/usr/local/samba/share/man/man8/smbd.8
/usr/local/samba/var/log.smbd
/usr/local/samba/var/log.smbd.old
/usr/local/samba/var/run/smbd.pid
/var/log/upstart/smbd.log.1.gz

nmbd

/usr/local/samba/sbin/nmbd
/usr/local/samba/share/man/man8/nmbd.8
/var/log/upstart/nmbd.log.1.gz

samba

/usr/local/samba/sbin/nmbd
/usr/local/samba/sbin/samba
/usr/local/samba/sbin/samba_dnsupdate
/usr/local/samba/sbin/samba_kcc
/usr/local/samba/sbin/samba_spnupdate
/usr/local/samba/sbin/samba_upgradedns
/usr/local/samba/sbin/smbd
/usr/local/samba/sbin/winbind


If I'm overlooking something obvious please let me know.

JS

[Sketch]

Also check for pytalloc|libtalloc and libtdb.

[IT Admin]

pytalloc:

/usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2
/usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2.1.2
/usr/local/samba/include/pytalloc.h
/usr/local/samba/lib/private/libpytalloc-util.so.2
/usr/local/samba/lib/private/libpytalloc-util.so.2.1.3

libtalloc:

/usr/lib/x86_64-linux-gnu/libtalloc.so.2
/usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.2
/usr/local/samba/lib/private/libtalloc-report-samba4.so
/usr/local/samba/lib/private/libtalloc.so.2
/usr/local/samba/lib/private/libtalloc.so.2.1.3


libtdb:

/usr/lib/x86_64-linux-gnu/libtdb.so.1
/usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5
/usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0
/usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0
/usr/local/samba/lib/private/libtdb-wrap-samba4.so
/usr/local/samba/lib/private/libtdb.so.1
/usr/local/samba/lib/private/libtdb.so.1.3.7


I also noticed that I've got a log.smbd in /usr/local/samba/var:
http://www.anonpaste.net/?p=bdfa3

JS

[Sketch]

I'd remove the distro packages providing those libs in /usr/lib, as they
could possibly cause problems. One more I forgot, which might possibly be
responsible for corrupting your ldb database if the wrong one is loaded
by samba at runtime, is libldb.

From the log, I'm guessing your database is corrupt if it can't read the
schema, but someone else might have more insight.

[IT Admin]

Apt and to think those packages aren't installed:

sudo apt-get remove libtdb-dev libtalloc-dev python-talloc-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'libtalloc-dev' is not installed, so not removed
Package 'libtdb-dev' is not installed, so not removed
Package 'python-talloc-dev' is not installed, so not removed
The following packages were automatically installed and are no longer
required:
linux-image-4.2.0-27-generic linux-image-extra-4.2.0-27-generic
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Should I just rename the files?

If the database is corrupt my only recourse is to follow the restore
guidelines in the Samba wiki and roll back to a previous version of the db,
correct?

JS

[L.P.H. van Belle]

Best thing you can do if you need 4.3.3 ( or higher ) packages.

Add the xenial deb-src to apt.
Run : apt-get update
apt-get source samba ( gets latest xenial sources )
apt-get build-dep samba ( get the build depends for samba )

!! skip this line for now, read on first.
apt-get source samba -b (builds packages)


if you missing one or more, repeat above again for these packages.

Use in local repo in higly recommends, this helps dependicy problems.
Marking these packages as your own to prevent mixing with original ubuntu packages.

Edit the files in .. source-package-nr/debian/changelog
For example : 2:4.3.4+custom1-lvb1
All i did here was edit, and add in the top new part.
Like :

samba (2:4.3.4+custom1-lvb1) unstable; urgency=medium

* Rebuild from Debian Sid to Debian Jessie

-- Louis van Belle <lo...@van-belle.nl> Fri, 12 Feb 2016 13:09:01 +0200

IF you mix packges from different distros version, you can get problems like memory corruption.. etc..

So this is why you need to recompile all the packages from a higher version.

And if people are thinking there is no debian 4.3.4 correct.
Get the 4.3.3 source and apply the diff 4.3.3-4.3.4 from samba.org,
and rebuild, apply the samba patch BEFORE the debian patches.

I havent succeeded to build 4.3.5 on Jessie jet, problems with socket-wrapper, and not time to look into that yet..


Greetz,

Louis


> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-...@lists.samba.org] Namens IT Admin
> Verzonden: donderdag 3 maart 2016 16:32
> Aan: Sketch
> CC: sa...@lists.samba.org
> Onderwerp: Re: [Samba] Samba AD/DC crashed again, third time in as many
> months

[mathias dufresne]

2016-03-03 16:32 GMT+01:00 IT Admin <i...@cliffbells.com>:

> Apt and to think those packages aren't installed:
>
> sudo apt-get remove libtdb-dev libtalloc-dev python-talloc-dev
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Package 'libtalloc-dev' is not installed, so not removed
> Package 'libtdb-dev' is not installed, so not removed
> Package 'python-talloc-dev' is not installed, so not removed
> The following packages were automatically installed and are no longer
> required:
> linux-image-4.2.0-27-generic linux-image-extra-4.2.0-27-generic
> Use 'apt-get autoremove' to remove them.
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>
> Should I just rename the files?
>
> If the database is corrupt my only recourse is to follow the restore
> guidelines in the Samba wiki and roll back to a previous version of the db,
> correct?
>
> JS
>

I'd say that depend on how complex is that corruption and how much you have
around you to understand that corruption.

If it is too complex (main reason to be too complex is a boss in a hurry)
restore the whole thing.

During restoration what you need is the same kind of system, you don't
really need to restore on the very same system. What I mean is you can take
advantage of that issue to deploy a new system (using same version of that
system, ie if it was centos6, use centos6).
As you seem to have a big system doing lot of things, part of these things
is running a Samba and even that software is configured to do several
things (AD + file server) which is not advised, I would use that down time
to switch Samba from that big system to some virtual machine with
minimalistic system doing one and only one thing: Samba as AD. Then in a
second VM I would install file server. In fact before installing file
server I would create another VM to host a second DC.

AD with one DC is not advised, anywhere. If you were having several DC
perhaps you would not have the whole AD broken but only one DC broken and
the other(s) one(s) working well. This was already explained to you today,
I was just insisting a little bit : )

[IT Admin]

Well, this puts me in a catch-22 situation... I can see the benefit of
spinning up VMs as ADCs, unfortunately this machine is already leveraged to
the limit and there aren't any resources available to support a single
additional VM, let alone two of them. And I'm scratching my head a bit
here as I have another Samba ADC deployed on another network, similar host
OS, similar RAID setup, and it has been rock solid...

I think the best course of action for this specific situation short-term is
to restore one of the samba backups to get their domain up and running
ASAP. I'll then have to figure out how to shuffle resources to get a
second ADC running in a VM and with any luck redundancy will mitigate
future corruption issues. That being said I'm very keen to understand what
the source of this corruption is, as I mentioned earlier I haven't had any
other issues with data corruption on this host and suspect that mismatched
libraries are a big part of the problem.

So, I need to verify, what is the proper way to remove the unwanted
packages in /usr/lib? Am I trying to remove the correct packages with
apt? Can I simply rename the offending files and reboot?

Once I've gotten rid of those files I'll follow the restore procedure and
attempt to get samba running again with last week's backup, that will allow
the client to move forward with end of year accounting work and give me a
chance to figure out how to shuffle resources so I can implement the dual
VM architecture.

Please advise, I really need to get this domain functional again by end of
day, I've got about 5 hours to do so.

Thanks again everyone for all of your help.

JS

[mj]

On 03/03/2016 06:34 PM, IT Admin wrote:
> So, I need to verify, what is the proper way to remove the unwanted
> packages in /usr/lib? Am I trying to remove the correct packages with
> apt? Can I simply rename the offending files and reboot?

To be absolutely sure to have about a clean and non-corrupt system, I'd
reccommend to reinstall on a CLEAN os, without manually
renaming/removing files.

Just install whatever OS you use, like and know on a empty fresh machine
(I DO like mdadm raid1, and have only good experiences with it, contrary
to Rowland) One tip: don't use btrfs for your AD server, use ext4, or xfs.

I really advise to NOT start manually deleting stuff etc on your current
install. You want to be as safe as possible after your misery. Even a
unused desktop machine with raid1 will be better than patching your
current misbehaving machine, is my advise.

That way you'd also be able to seperate AD DC functionality from your
fileserver. If you want to be REALLY cheap, you could even start like this:

desktop machine, raid one, install kvm, and run TWO dc's on that
machine. It's very easy to move around those kvm machines to different
hosts, if you get some budget, or another spare machine.

(but: running two DCs on the same host of course does NOT give you
redundancy if that host goes down, it could just help against the kind
of corruption you experienced)

MJ

[Marc Muehlfeld]

Hello,

Am 03.03.2016 um 01:08 schrieb IT Admin:
> I added "log level = 10" to my smb.conf, the new log is here:
> http://www.anonpaste.net/?p=f1ec8

Do you still have the compiled sources or can re-compile the 4.3.3
sources with the same ./configure parameters? If yes, do an "make
install" over your existing installation and see what happens.

[IT Admin]

I do have the 4.3.3 sources. The only configure flag i set was
--enable-debug. I can execute that install, however the issue of unwanted
libs still lingers. I'm confused, is it or is it not an issue that i have
libtld, libtalloc and pytalloc libs in /usr/lib? And if it is an issue
whats the best path forward to resolve it?

JS

[IT Admin]

Thanks for your input, I could spin up a VM on a workstation but a) it
would be 32bit, and b) current samba ADC is on a 64 bit host... Not sure if
I could restore my backup there it not.

I think I can swap the roles of the current file server and another machine
to get greater resources to play with, so I likely will be implementing the
dual VM approach you outlined in the future but my primary concern at this
moment is to get the domain up again, they need access no later than 8am
tomorrow.

If I can get answers re removing the offending libs from /usr/lib I'll
hopefully me moving forward here in the next couple of hours.

JS

[Rowland penny]

On 03/03/16 20:33, IT Admin wrote:
> I do have the 4.3.3 sources. The only configure flag i set was
> --enable-debug. I can execute that install, however the issue of unwanted
> libs still lingers. I'm confused, is it or is it not an issue that i have
> libtld, libtalloc and pytalloc libs in /usr/lib? And if it is an issue
> whats the best path forward to resolve it?
>
> JS
>

No, I don't think it is a problem, except for libtld and that is only
because I have never heard of it :-) Do you mean libldb ?

My debian server running 4.3.1:

locate libldb
/usr/lib/x86_64-linux-gnu/libldb.so.1
/usr/lib/x86_64-linux-gnu/libldb.so.1.1.17

/usr/local/samba/lib/private/libldb.so.1
/usr/local/samba/lib/private/libldb.so.1.1.21

locate libtalloc
/usr/lib/x86_64-linux-gnu/libtalloc.so.2
/usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.1

/usr/local/samba/lib/private/libtalloc.so.2
/usr/local/samba/lib/private/libtalloc.so.2.1.3

locate pytalloc
/usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2
/usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2.1.1

/usr/local/samba/lib/private/libpytalloc-util.so.2
/usr/local/samba/lib/private/libpytalloc-util.so.2.1.3

Rowland

[IT Admin]

Ha! That should be libtdb:

/usr/lib/x86_64-linux-gnu/libtdb.so.1
/usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5
/usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0
/usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0
/usr/local/samba/lib/private/libtdb-wrap-samba4.so
/usr/local/samba/lib/private/libtdb.so.1
/usr/local/samba/lib/private/libtdb.so.1.3.7

Sorry for the typo, I compose most of my replies on my phone in the car as
this case pertains to work I do outside the scope of my primary source of
employment.

Ok, if Rowland says they aren't an issue then I'm moving forward. I'll
recompile 4.3.3 and overwrite my existing install, then fire up samba and
see what happens. My guess is it fails all over itself at which point I'll
make sure samba process is stopped and follow the restore guidelines in the
wiki to restore an older copy of the database. Fingers crossed I can raise
this thing from the dead.

Thanks again everyone, really appreciate the community stepping up to help
out.

JS

[IT Admin]

Thanks Rowland,

They've been there all along and the thing ran for a month so my gut said
they weren't problematic.

JS
On Mar 3, 2016 4:30 PM, "Rowland penny" <rpe...@samba.org> wrote:

> On 03/03/16 21:24, IT Admin wrote:
>
>>
>> Ha! That should be libtdb:
>>
>> /usr/lib/x86_64-linux-gnu/libtdb.so.1
>> /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5
>> /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0
>> /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0
>> /usr/local/samba/lib/private/libtdb-wrap-samba4.so
>> /usr/local/samba/lib/private/libtdb.so.1
>> /usr/local/samba/lib/private/libtdb.so.1.3.7
>>
>> Sorry for the typo, I compose most of my replies on my phone in the car
>> as this case pertains to work I do outside the scope of my primary source
>> of employment.
>>
>> Ok, if Rowland says they aren't an issue then I'm moving forward. I'll
>> recompile 4.3.3 and overwrite my existing install, then fire up samba and
>> see what happens. My guess is it fails all over itself at which point I'll
>> make sure samba process is stopped and follow the restore guidelines in the
>> wiki to restore an older copy of the database. Fingers crossed I can raise
>> this thing from the dead.
>>
>> Thanks again everyone, really appreciate the community stepping up to
>> help out.
>>
>> JS
>>

>> On Mar 3, 2016 4:02 PM, "Rowland penny" <rpe...@samba.org <mailto:

> OK, I have these in two places as well:
>
> locate libtdb
> /usr/lib/x86_64-linux-gnu/libtdb.so.1
> /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.1

> /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0
> /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0
>
> /usr/local/samba/lib/private/libtdb-wrap-samba4.so
> /usr/local/samba/lib/private/libtdb.so.1
> /usr/local/samba/lib/private/libtdb.so.1.3.7
>

> All I can say is that I haven't had a problem with both of them installed.

[Rowland penny]

On 03/03/16 21:24, IT Admin wrote:
>

> Ha! That should be libtdb:
>
> /usr/lib/x86_64-linux-gnu/libtdb.so.1
> /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5
> /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0
> /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0
> /usr/local/samba/lib/private/libtdb-wrap-samba4.so
> /usr/local/samba/lib/private/libtdb.so.1
> /usr/local/samba/lib/private/libtdb.so.1.3.7
>
> Sorry for the typo, I compose most of my replies on my phone in the
> car as this case pertains to work I do outside the scope of my primary
> source of employment.
>
> Ok, if Rowland says they aren't an issue then I'm moving forward.
> I'll recompile 4.3.3 and overwrite my existing install, then fire up
> samba and see what happens. My guess is it fails all over itself at
> which point I'll make sure samba process is stopped and follow the
> restore guidelines in the wiki to restore an older copy of the
> database. Fingers crossed I can raise this thing from the dead.
>
> Thanks again everyone, really appreciate the community stepping up to
> help out.
>
> JS
>
> On Mar 3, 2016 4:02 PM, "Rowland penny" <rpe...@samba.org

OK, I have these in two places as well:

locate libtdb
/usr/lib/x86_64-linux-gnu/libtdb.so.1
/usr/lib/x86_64-linux-gnu/libtdb.so.1.3.1

/usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0
/usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0

/usr/local/samba/lib/private/libtdb-wrap-samba4.so
/usr/local/samba/lib/private/libtdb.so.1
/usr/local/samba/lib/private/libtdb.so.1.3.7

All I can say is that I haven't had a problem with both of them installed.

[Bob of Donelson Trophy]

I have been following this and I am going to add my two cents . . . had
a computer awhile back that did strange things after a few weeks.
Re-install the OS and try again. Did this twice too many times.

Some how, quite by accident I discovered that there was a bad sata
cable. Replaced the cable and the computer is still running today, about
two years later.

We forget that we are dealing with mechanical as well as electronic
issues, not just software. A bad electrical connection, no matter how
minor, can be confusing and difficult to sort out. What if your problem
_is_ a bad sata I (one) cable and should be a SATAII cable? Have you
tested your memory (RAM)? Bad power supply . . . and the list goes on
and on.

Well, I think you get the idea.

Just my two cents . . .
---

_______________________________

Bob Wooden of Donelson Trophy

615.885.2846
www.donelsontrophy.com [2]

"Everyone deserves an award!!"

On 2016-03-03 15:37, IT Admin wrote:

> Thanks Rowland,
>
> They've been there all along and the thing ran for a month so my gut said
> they weren't problematic.
>
> JS
> On Mar 3, 2016 4:30 PM, "Rowland penny" <rpe...@samba.org> wrote:
> On 03/03/16 21:24, IT Admin wrote: Ha! That should be libtdb: /usr/lib/x86_64-linux-gnu/libtdb.so.1 /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5 /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0 /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0 /usr/local/samba/lib/private/libtdb-wrap-samba4.so /usr/local/samba/lib/private/libtdb.so.1 /usr/local/samba/lib/private/libtdb.so.1.3.7 Sorry for the typo, I compose most of my replies on my phone in the car as this case pertains to work I do outside the scope of my primary source of employment. Ok, if Rowland says they aren't an issue then I'm moving forward. I'll recompile 4.3.3 and overwrite my existing install, then fire up samba and see what happens. My guess is it fails all over itself at which point I'll make sure samba process is stopped and follow the restore guidelines in the wiki to restore an older copy of the database. Fingers crossed I can raise this thing from the dead. Thanks again everyone, really appreciate the community
stepping up to help out. JS On Mar 3, 2016 4:02 PM, "Rowland penny" <rpe...@samba.org <mailto: rpe...@samba.org>> wrote: On 03/03/16 20:33, IT Admin wrote: I do have the 4.3.3 sources. The only configure flag i set was --enable-debug. I can execute that install, however the issue of unwanted libs still lingers. I'm confused, is it or is it not an issue that i have libtld, libtalloc and pytalloc libs in /usr/lib? And if it is an issue whats the best path forward to resolve it? JS No, I don't think it is a problem, except for libtld and that is only because I have never heard of it :-) Do you mean libldb ? My debian server running 4.3.1: locate libldb /usr/lib/x86_64-linux-gnu/libldb.so.1 /usr/lib/x86_64-linux-gnu/libldb.so.1.1.17 /usr/local/samba/lib/private/libldb.so.1 /usr/local/samba/lib/private/libldb.so.1.1.21 locate libtalloc /usr/lib/x86_64-linux-gnu/libtalloc.so.2 /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.1 /usr/local/samba/lib/private/libtalloc.so.2

/usr/local/samba/lib/private/libtalloc.so.2.1.3 locate pytalloc /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2 /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2.1.1 /usr/local/samba/lib/private/libpytalloc-util.so.2 /usr/local/samba/lib/private/libpytalloc-util.so.2.1.3 Rowland -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba [1] OK, I have these in two places as well: locate libtdb /usr/lib/x86_64-linux-gnu/libtdb.so.1 /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.1 /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0 /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0 /usr/local/samba/lib/private/libtdb-wrap-samba4.so /usr/local/samba/lib/private/libtdb.so.1 /usr/local/samba/lib/private/libtdb.so.1.3.7 All I can say is that I haven't had a problem with both of them installed. Rowland -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
[1]


Links:
------
[1] https://lists.samba.org/mailman/options/samba
[2] http://www.donelsontrophy.com

[IT Admin]

Appreciate the input, I had the build failures during my recompile, am
going to test ram the next chance I get, will swap out those cables for
giggles too.

Thanks again,

JS
On Mar 3, 2016 5:35 PM, "Bob of Donelson Trophy" <b...@donelsontrophy.net>
wrote:

[IT Admin]

I rebuilt samba 4.3.3 and overwrote my existing install, samba still
collapses unceremoniously. I've now stopped samba, moved necessary dirs to
a safe backup location, and executed the required steps to restore backup
files created with samba_backup per the wiki. I've hit a roadblock
though... I chose to backup ACLs when I ran the backup script but the
section on restoration in the wiki makes no mention of what to do to
restore the archives that contains them... Please advise, this is the last
step prior to cranking her over to determine if I'm staying up all night
rebuilding this domain from scratch.

JS

[IT Admin]

Could I get some input regarding restoring ACLs? There's nothing in the
Samba wiki about what to do with the ACL archive. I've finished all of the
other steps already, what do I need to do to restore this archive?

JS

[IT Admin]

I had to move forward with my restore process, couldn't wsit any longer for
confirmation on extended ACLs. As I understand it the version of
samba_backup that Rowland supplied me in January backs up these ACLs as
long as my version of gnutar supports it, and afaik the version of gnutar I
have installed does, so I took a leap of faith and just restarted samba
using the init script I have in /etc/init.d.

After executing *sudo /etc/init.d/samba4 start *I then issued the
command *sudo
/etc/init.d/samba4 status *and received the following output:

sudo /etc/init.d/samba4 start
[ ok ] Starting samba4 (via systemctl): samba4.service.
itwerks@filer:~$ sudo /etc/init.d/samba4 status
● samba4.service - LSB: start Samba daemons
Loaded: loaded (/etc/init.d/samba4)
Active: active (running) since Thu 2016-03-03 20:47:23 EST; 7s ago
Docs: man:systemd-sysv-generator(8)
Process: 25404 ExecStop=/etc/init.d/samba4 stop (code=exited,
status=0/SUCCESS)
Process: 4798 ExecStart=/etc/init.d/samba4 start (code=exited,
status=0/SUCCESS)
CGroup: /system.slice/samba4.service
├─4802 /usr/local/samba/sbin/samba -D
├─4824 /usr/local/samba/sbin/samba -D
├─4825 /usr/local/samba/sbin/samba -D
├─4826 /usr/local/samba/sbin/samba -D
├─4828 /usr/local/samba/sbin/smbd -D --option=server role
check:inhibit=yes --foreground
├─4829 /usr/local/samba/sbin/samba -D
├─4830 /usr/local/samba/sbin/samba -D
├─4831 /usr/local/samba/sbin/samba -D
├─4832 /usr/local/samba/sbin/samba -D
├─4833 /usr/local/samba/sbin/samba -D
├─4834 /usr/local/samba/sbin/samba -D
├─4835 /usr/local/samba/sbin/samba -D
├─4836 /usr/local/samba/sbin/winbindd -D --option=server role
check:inhibit=yes --foreground
├─4837 /usr/local/samba/sbin/samba -D
├─4838 /usr/local/samba/sbin/samba -D
├─4839 /usr/local/samba/sbin/samba -D
├─4840 python /usr/local/samba/sbin/samba_dnsupdate
├─4841 python /usr/local/samba/sbin/samba_spnupdate
├─4844 /usr/local/samba/sbin/winbindd -D --option=server role
check:inhibit=yes --foreground
├─4845 /usr/local/samba/sbin/smbd -D --option=server role
check:inhibit=yes --foreground
├─4848 /usr/local/samba/sbin/winbindd -D --option=server role
check:inhibit=yes --foreground
├─4850 /usr/local/samba/sbin/winbindd -D --option=server role
check:inhibit=yes --foreground
└─4855 /usr/local/samba/sbin/smbd -D --option=server role
check:inhibit=yes --foreground

Mar 03 20:47:25 filer samba[4802]: [2016/03/03 20:47:25.386200, 0]
../source4/smbd/server.c:490(binary_smbd_main)
Mar 03 20:47:25 filer samba[4802]: samba: using 'standard' process model
Mar 03 20:47:25 filer samba[4802]: [2016/03/03 20:47:25.503751, 0]
../lib/util/become_daemon.c:124(daemon_ready)
Mar 03 20:47:25 filer samba[4802]: STATUS=daemon 'samba' finished
starting up and ready to serve connections
Mar 03 20:47:27 filer winbindd[4836]: [2016/03/03 20:47:27.287385, 0]
../source3/winbindd/winbindd_cache.c:3245(initialize_winbindd_cache)
Mar 03 20:47:27 filer winbindd[4836]: initialize_winbindd_cache: clearing
cache and re-creating with version number 2
Mar 03 20:47:28 filer winbindd[4836]: [2016/03/03 20:47:28.686333, 0]
../lib/util/become_daemon.c:124(daemon_ready)
Mar 03 20:47:28 filer winbindd[4836]: STATUS=daemon 'winbindd' finished
starting up and ready to serve connections
Mar 03 20:47:30 filer smbd[4828]: [2016/03/03 20:47:30.057642, 0]
../lib/util/become_daemon.c:124(daemon_ready)
Mar 03 20:47:30 filer smbd[4828]: STATUS=daemon 'smbd' finished starting
up and ready to serve connections

/usr/local/samba/var/log.samba:

[2016/03/03 20:47:23.075237, 0]
../source4/smbd/server.c:372(binary_smbd_main)
samba version 4.3.3 started.
Copyright Andrew Tridgell and the Samba Team 1992-2015
[2016/03/03 20:47:25.386200, 0]
../source4/smbd/server.c:490(binary_smbd_main)
samba: using 'standard' process model
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
[2016/03/03 20:47:25.503751, 0]
../lib/util/become_daemon.c:124(daemon_ready)
STATUS=daemon 'samba' finished starting up and ready to serve connections
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.


/usr/local/samba/var/log.smbd:

[2016/03/03 20:47:30.057642, 0]
../lib/util/become_daemon.c:124(daemon_ready)
STATUS=daemon 'smbd' finished starting up and ready to serve connections


The domain is up again, ACLs appear to have been retained, user accounts
and groups are intact, shared folders are accessible.

I'll be doing an immediate samba_backup and will be taking a hard look at
the hardware behind this ADC. I'll also probably be spinning up a second
ADC in a VM hosted on another machine in an attempt to make this deployment
more robust, and I'll be moving this ADC implementation to a VM as well.

Thanks again to everyone here who offered advice and support, I appreciate
your time and expertise.

Kind Regards,

JS

PS - if you're ever in Detroit, MI and like Jazz, ping me and I'll take
care of you.

[Andrew Bartlett]

On Wed, 2016-03-02 at 16:42 -0500, IT Admin wrote:
> I built this machine, and while it isn't the most robust box in the
> world
> it has been stable otherwise.  The RAID array is configured RAID1, I
> can't
> see how that could cause corruption issues and I haven't experienced
> any
> other data corruption issues apart from SAMBA collapsing

I know it is hard to swallow, but I really think this is hardware, or
the OS configuration under it, combined with unexpected shutdown or
some other corruption vector.

We have at this stage 10,000 or more domains running Samba4, and this
is only the second I've heard of with this kind of symptom.  The first
I blamed on a use of DRDB that I postulated was not preserving 'write
barriers' (that is, the thing that makes fsync() work) and a poweroff,
but I didn't really have any proof.

You do need to run a second DC, as well as run tools like memcheck on
this DC.  Make sure you regularly run the backup script, so you can
work out when the corruption happens, and verify your DB with dbcheck.

The second DC has the advantage that this kind of low-level corruption
doesn't easily spread across DRS replication (it would instead fail
replication). 

The error shown indicates that for some reason or other, it can't read
the schema.  This is very odd, as the schema doesn't change!  

We would love to get to the bottom of this. 

Unlike others, I don't think this has anything to do with packaging
(that would just make us not start at all), but a clean install on a
clean machine is my best advise, keeping the rest aside (and off) for
forensics if you have the patience. 

Finally, always keep the steps simple - otherwise we might start
confusing admin errors for hardware errors or vice verca.  The things
we all do in the panic are always the hardest to de-construct in the
cold light of day.

Thanks,

Andrew Bartlett

--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team https://samba.org
Samba Development and Support, Catalyst IT
https://catalyst.net.nz/services/samba

[IT Admin]

Hi Andrew,

Thanks for your input. I intend to take a look at the memory on this
machine asap to see if that is the cause of my issues, and I figure I may
as well swap out the data cables while I'm there for good measure.

I didn't think package conflicts could be an issue as I spent a fair amount
of time double checking for conflicts when I initially moved from
canonical's distribution of samba to a compiled build as the domain had
been functional for awhile.

My only frustration at this point is that at no point does the wiki state
that in order to stably deploy active directory using samba one MUST deploy
at least to ADCs. If that is true it would be helpful to state the
requirement in the docs. I've got a few ADs deployed with single ADCs, now
I feel compelled to make those environments more robust and in many cases
lack the resources to do so without added cost for the client and/or a fair
amount of labor on my part.

I'll update this thread once I've had an opportunity to inspect the
hardware.

Kind regards,

JS

[IT Admin]

As advised I have begun the process of adding ADCs to this domain and
currently have a second samba ADC joined to the domain. I would like to
demote the initial ADC and make this secondary ADC the primary as the
problematic machine is still crashing and when samba has failed DNS fails
across the domain. I'd also like to know if it is possible to create
redundancy when using SAMBA_INTERNAL as a DNS backend.

Please advise.

JS
On Mar 6, 2016 6:56 PM, "Andrew Bartlett" <abar...@samba.org> wrote:

[mathias dufresne]

Answering to previous mail:
AD is hearth of infrastructure. That's where all accounts are stored. That
last affirmation implies few times after you start deploying AD most of
your IT infrastructure depends on AD (all applications need accounts, they
are in AD, no AD, no accounts, nothing work) and that you take security in
consideration and that you do that seriously: an attacker with
administrator account can do almost everything everywhere on machines
joined to AD.

So redundancy, every times.

You could also think about your own issue: is it the whole DB which is
broken or is it the DB on the broken DC? With one DC, the whole DB is the
one DC, so you always break the whole DB.
With several you get a chance to break only one DC and to have others with
a coherent DB. That do not means you will never break the whole DB (backup
and a working process to restore is still needed).

Second mail:
You want to remove your FSMO owner. The FSMO owner is SOA.
These two are really important notion in AD:
- FSMO is kind of PDC in NT4 domain, these roels must belong to one DC.
Seize role before demoting the old one.
- SOA is about DNS, it refers the one server where some client can push DNS
modification. Change SOA before you try to add a replacement server to the
one you demoted. If you don't the DC you would join to replace demoted DC
won't be able to send DNS update!

And yes it possible to get redundancy with dns-backend=SAMBA_INTERNAL.

How to test your DNS servers are well configured: samba_dnsupdate gives no
error on all DC (this a test related to DNS service only).

[Rowland penny]

On 07/03/16 15:07, mathias dufresne wrote:
> Answering to previous mail:
> AD is hearth of infrastructure. That's where all accounts are stored. That
> last affirmation implies few times after you start deploying AD most of
> your IT infrastructure depends on AD (all applications need accounts, they
> are in AD, no AD, no accounts, nothing work) and that you take security in
> consideration and that you do that seriously: an attacker with
> administrator account can do almost everything everywhere on machines
> joined to AD.
>
> So redundancy, every times.

Totally agree

>
> You could also think about your own issue: is it the whole DB which is
> broken or is it the DB on the broken DC? With one DC, the whole DB is the
> one DC, so you always break the whole DB.
> With several you get a chance to break only one DC and to have others with
> a coherent DB. That do not means you will never break the whole DB (backup
> and a working process to restore is still needed).

Again agree

>
> Second mail:
> You want to remove your FSMO owner. The FSMO owner is SOA.

Not necessarily, there are no FSMO roles on my second DC, but it has a SOA.

> These two are really important notion in AD:
> - FSMO is kind of PDC in NT4 domain,

Well, to a certain extent and only when you are describing the PDC
emulator FSMO role

> these roels must belong to one DC.

Totally wrong, you can, and probably should, share these about if you
have more than one DC.

> Seize role before demoting the old one.

Again wrong, you should try to transfer the role first, only seize it if
you have to i.e. the FSMO role owner DC is dead.

> - SOA is about DNS, it refers the one server where some client can push DNS
> modification. Change SOA before you try to add a replacement server to the
> one you demoted. If you don't the DC you would join to replace demoted DC
> won't be able to send DNS update!

It would seem that something has changed and I need to some more
testing, must add it to my todo list.

Rowland

>
> And yes it possible to get redundancy with dns-backend=SAMBA_INTERNAL.
>
> How to test your DNS servers are well configured: samba_dnsupdate gives no
> error on all DC (this a test related to DNS service only).
>
>
>
>