[Samba] Samba 4 AD - Samba Fails to Start, hdb_samba4_create_kdc (setup KDC database) failed

[IT Admin]

Hello to the Samba Mailing List,

This is my first post, so please, should I commit any faux pas, nudge me in
the right direction and I will adjust accordingly.

I'm experiencing a complete failure of the PDC in a Samba 4 AD Domain I've
deployed for a client. Samba failed a few days ago and I've been unable to
resolve the issue on my own. Google searches are leading me in circles,
I'm hoping the list can help me get this deployment back in working order.

Some details on the failing machine:

Release: 15.10
Linux 4.2.0-22-generic #27-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux
Samba Version: 4.1.17-Ubuntu

I provisioned this domain a few months ago, everything was going smoothly
until a hardware failure forced me to reprovision a couple of weeks ago.
Having just got their network stable again I was rather disheartened to
discover Samba had taken a nosedive for Christmas.

Relevant info from Samba's logs (debug level 4):

/var/log/samba/log/samba:

samba version 4.1.17-Ubuntu started.
Copyright Andrew Tridgell and the Samba Team 1992-2013
[2015/12/28 21:12:05.907126, 3]
../source4/smbd/server.c:381(binary_smbd_main)
Becoming a daemon.
[2015/12/28 21:12:05.919238, 3]
../auth/gensec/gensec_start.c:870(gensec_register)
GENSEC backend 'gssapi_spnego' registered
[2015/12/28 21:12:05.919327, 3]
../auth/gensec/gensec_start.c:870(gensec_register)
GENSEC backend 'gssapi_krb5' registered
[2015/12/28 21:12:05.919360, 3]
../auth/gensec/gensec_start.c:870(gensec_register)
GENSEC backend 'gssapi_krb5_sasl' registered
[2015/12/28 21:12:05.919437, 3]
../auth/gensec/gensec_start.c:870(gensec_register)
GENSEC backend 'schannel' registered
[2015/12/28 21:12:05.919472, 3]
../auth/gensec/gensec_start.c:870(gensec_register)
GENSEC backend 'spnego' registered
[2015/12/28 21:12:05.919503, 3]
../auth/gensec/gensec_start.c:870(gensec_register)
GENSEC backend 'ntlmssp' registered
[2015/12/28 21:12:05.919537, 3]
../auth/gensec/gensec_start.c:870(gensec_register)
GENSEC backend 'krb5' registered
[2015/12/28 21:12:05.919567, 3]
../auth/gensec/gensec_start.c:870(gensec_register)
GENSEC backend 'fake_gssapi_krb5' registered
[2015/12/28 21:12:05.919643, 3]
../source4/ntptr/ntptr_base.c:67(ntptr_register)
NTPTR backend 'simple_ldb'
[2015/12/28 21:12:05.919714, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'default' for type 1 registered
[2015/12/28 21:12:05.919753, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'posix' for type 1 registered
[2015/12/28 21:12:05.919791, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'unixuid' for type 1 registered
[2015/12/28 21:12:05.919821, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'unixuid' for type 3 registered
[2015/12/28 21:12:05.919852, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'unixuid' for type 2 registered
[2015/12/28 21:12:05.919884, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'cifs' for type 1 registered
[2015/12/28 21:12:05.919915, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'smb2' for type 1 registered
[2015/12/28 21:12:05.919946, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'simple' for type 1 registered
[2015/12/28 21:12:05.919977, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'cifsposix' for type 1 registered
[2015/12/28 21:12:05.920010, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'default' for type 3 registered
[2015/12/28 21:12:05.920041, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'default' for type 2 registered
[2015/12/28 21:12:05.920078, 3]
../source4/ntvfs/ntvfs_base.c:79(ntvfs_register)
NTVFS backend 'nbench' for type 1 registered
[2015/12/28 21:12:05.921420, 3]
../source4/smbd/process_model.c:97(register_process_model)
PROCESS_MODEL 'single' registered
[2015/12/28 21:12:05.921479, 3]
../source4/smbd/process_model.c:97(register_process_model)
PROCESS_MODEL 'standard' registered
[2015/12/28 21:12:05.921510, 3]
../source4/smbd/process_model.c:97(register_process_model)
PROCESS_MODEL 'onefork' registered
[2015/12/28 21:12:05.921540, 3]
../source4/smbd/process_model.c:97(register_process_model)
PROCESS_MODEL 'prefork' registered
[2015/12/28 21:12:06.064097, 3]
../source4/auth/ntlm/auth.c:673(auth_register)
AUTH backend 'sam' registered
[2015/12/28 21:12:06.064187, 3]
../source4/auth/ntlm/auth.c:673(auth_register)
AUTH backend 'sam_ignoredomain' registered
[2015/12/28 21:12:06.064220, 3]
../source4/auth/ntlm/auth.c:673(auth_register)
AUTH backend 'anonymous' registered
[2015/12/28 21:12:06.064251, 3]
../source4/auth/ntlm/auth.c:673(auth_register)
AUTH backend 'winbind' registered
[2015/12/28 21:12:06.064284, 3]
../source4/auth/ntlm/auth.c:673(auth_register)
AUTH backend 'winbind_wbclient' registered
[2015/12/28 21:12:06.064316, 3]
../source4/auth/ntlm/auth.c:673(auth_register)
AUTH backend 'name_to_ntstatus' registered
[2015/12/28 21:12:06.064347, 3]
../source4/auth/ntlm/auth.c:673(auth_register)
AUTH backend 'unix' registered
[2015/12/28 21:12:06.064401, 3]
../source4/param/share.c:124(share_register)
SHARE backend [classic] registered.
[2015/12/28 21:12:06.697309, 3]
../lib/ldb-samba/ldb_wrap.c:320(ldb_wrap_connect)
ldb_wrap open of privilege.ldb
[2015/12/28 21:12:06.748805, 0]
../source4/smbd/server.c:488(binary_smbd_main)
samba: using 'standard' process model
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
[2015/12/28 21:12:06.779495, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'rpcecho' registered
[2015/12/28 21:12:06.764776, 0]
../source4/dsdb/common/util.c:1693(samdb_reference_dn_is_our_ntdsa)
Failed to find object DC=one,DC=cliffbells,DC=com for attribute
fsmoRoleOwner - Cannot find DN DC=one,DC=cliffbells,DC=com to get attribute
fsmoRoleOwner for reference dn: (null)
[2015/12/28 21:12:06.780250, 1]
../source4/dsdb/common/util.c:1877(samdb_is_pdc)
Failed to find if we are the PDC for this ldb: Searching for
fSMORoleOwner in DC=one,DC=cliffbells,DC=com failed: Cannot find DN
DC=one,DC=cliffbells,DC=com to get attribute fsmoRoleOwner for reference
dn: (null)
[2015/12/28 21:12:06.788717, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'epmapper' registered
[2015/12/28 21:12:06.789079, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'remote' registered
[2015/12/28 21:12:06.789535, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'srvsvc' registered
[2015/12/28 21:12:06.789597, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'wkssvc' registered
[2015/12/28 21:12:06.789634, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'unixinfo' registered
[2015/12/28 21:12:06.790292, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'samr' registered
[2015/12/28 21:12:06.790372, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'winreg' registered
[2015/12/28 21:12:06.790410, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'netlogon' registered
[2015/12/28 21:12:06.790654, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'dssetup' registered
[2015/12/28 21:12:06.790702, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'lsarpc' registered
[2015/12/28 21:12:06.790739, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'backupkey' registered
[2015/12/28 21:12:06.790783, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'spoolss' registered
[2015/12/28 21:12:06.790818, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'drsuapi' registered
[2015/12/28 21:12:06.790864, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'browser' registered
[2015/12/28 21:12:06.790897, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'eventlog6' registered
[2015/12/28 21:12:06.790941, 3]
../source4/rpc_server/dcerpc_server.c:1208(dcerpc_register_ep_server)
DCERPC endpoint server 'dnsserver' registered
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
[2015/12/28 21:12:06.842176, 3]
../lib/ldb-samba/ldb_wrap.c:320(ldb_wrap_connect)
ldb_wrap open of secrets.ldb
[2015/12/28 21:12:06.843155, 3]
../lib/ldb-samba/ldb_wrap.c:320(ldb_wrap_connect)
ldb_wrap open of idmap.ldb
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
[2015/12/28 21:12:06.865340, 1]
../source4/kdc/db-glue.c:1956(samba_kdc_setup_db_ctx)
samba_kdc_fetch: could not find own KRBTGT in DB: (null)
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
[2015/12/28 21:12:06.869471, 2]
../source4/dsdb/repl/drepl_partitions.c:116(dreplsrv_load_partitions)
dreplsrv_partition[CN=Configuration,DC=one,DC=cliffbells,DC=com] loaded
[2015/12/28 21:12:06.869600, 2]
../source4/dsdb/repl/drepl_partitions.c:116(dreplsrv_load_partitions)

dreplsrv_partition[CN=Schema,CN=Configuration,DC=one,DC=cliffbells,DC=com]
loaded
[2015/12/28 21:12:06.869648, 2]
../source4/dsdb/repl/drepl_partitions.c:116(dreplsrv_load_partitions)
dreplsrv_partition[DC=one,DC=cliffbells,DC=com] loaded
[2015/12/28 21:12:06.869742, 2]
../source4/dsdb/repl/drepl_partitions.c:116(dreplsrv_load_partitions)
dreplsrv_partition[DC=DomainDnsZones,DC=one,DC=cliffbells,DC=com] loaded
[2015/12/28 21:12:06.869789, 2]
../source4/dsdb/repl/drepl_partitions.c:116(dreplsrv_load_partitions)
dreplsrv_partition[DC=ForestDnsZones,DC=one,DC=cliffbells,DC=com] loaded
[2015/12/28 21:12:06.865437, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [kdc: hdb_samba4_create_kdc (setup KDC database)
failed]
[2015/12/28 21:12:06.878911, 3]
../source4/dsdb/dns/dns_update.c:340(dnsupdate_check_names)
Calling DNS name update script
[2015/12/28 21:12:06.888121, 3]
../source4/dsdb/dns/dns_update.c:355(dnsupdate_check_names)
Calling SPN name update script
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.
[2015/12/28 21:12:06.902840, 2]
../source4/dsdb/kcc/kcc_service.c:127(kccsrv_load_partitions)
kccsrv_partition[DC=ONE,DC=CLIFFBELLS,DC=COM] loaded
[2015/12/28 21:12:06.902998, 2]
../source4/dsdb/kcc/kcc_service.c:127(kccsrv_load_partitions)
kccsrv_partition[CN=Configuration,DC=one,DC=cliffbells,DC=com] loaded
[2015/12/28 21:12:06.903036, 2]
../source4/dsdb/kcc/kcc_service.c:127(kccsrv_load_partitions)
kccsrv_partition[CN=Schema,CN=Configuration,DC=one,DC=cliffbells,DC=com]
loaded
[2015/12/28 21:12:06.903072, 2]
../source4/dsdb/kcc/kcc_service.c:127(kccsrv_load_partitions)
kccsrv_partition[DC=DomainDnsZones,DC=one,DC=cliffbells,DC=com] loaded
[2015/12/28 21:12:06.903107, 2]
../source4/dsdb/kcc/kcc_service.c:127(kccsrv_load_partitions)
kccsrv_partition[DC=ForestDnsZones,DC=one,DC=cliffbells,DC=com] loaded
[2015/12/28 21:12:06.884922, 0]
../lib/util/become_daemon.c:136(daemon_ready)
STATUS=daemon 'samba' finished starting up and ready to serve
connectionssamba_terminate: kdc: hdb_samba4_create_kdc (setup KDC database)
failed
[2015/12/28 21:12:06.930079, 3]
../lib/ldb-samba/ldb_wrap.c:320(ldb_wrap_connect)
ldb_wrap open of secrets.ldb
[2015/12/28 21:12:07.752016, 0]
../file_server/file_server.c:48(file_server_smbd_done)
file_server smbd daemon exited normally
[2015/12/28 21:12:07.752994, 0]
../source4/smbd/service_task.c:35(task_server_terminate)
task_server_terminate: [smbd child process exited]


/var/log/samba/log.smbd:

smbd version 4.1.17-Ubuntu started.
Copyright Andrew Tridgell and the Samba Team 1992-2013
[2015/12/28 21:12:06, 2]
../source3/lib/tallocmsg.c:124(register_msg_pool_usage)
Registered MSG_REQ_POOL_USAGE
[2015/12/28 21:12:06, 2]
../source3/lib/dmallocmsg.c:78(register_dmalloc_msgs)
Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED
[2015/12/28 21:12:06.975569, 3]
../source3/param/loadparm.c:4839(lp_load_ex)
lp_load_ex: refreshing parameters
[2015/12/28 21:12:06.975630, 3]
../source3/param/loadparm.c:750(init_globals)
Initialising global parameters
[2015/12/28 21:12:06.975672, 2]
../source3/param/loadparm.c:543(max_open_files)
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
[2015/12/28 21:12:06.975752, 3] ../lib/util/params.c:550(pm_process)
params.c:pm_process() - Processing configuration file
"/etc/samba/smb.conf"
[2015/12/28 21:12:06.975787, 3]
../source3/param/loadparm.c:3565(do_section)
Processing section "[global]"
[2015/12/28 21:12:06.976003, 2]
../source3/param/loadparm.c:3582(do_section)
Processing section "[netlogon]"
[2015/12/28 21:12:06.976125, 2]
../source3/param/loadparm.c:3582(do_section)
Processing section "[sysvol]"
[2015/12/28 21:12:06.976193, 2]
../source3/param/loadparm.c:3582(do_section)
Processing section "[accounting]"
[2015/12/28 21:12:06.976277, 2]
../source3/param/loadparm.c:3582(do_section)
Processing section "[data]"
[2015/12/28 21:12:06.976359, 2]
../source3/param/loadparm.c:3582(do_section)
Processing section "[backups]"
[2015/12/28 21:12:06.976472, 3]
../source3/param/loadparm.c:1774(lp_add_ipc)
adding IPC service
[2015/12/28 21:12:06.976790, 2]
../source3/lib/interface.c:341(add_interface)
added interface eth0 ip=192.168.37.2 bcast=192.168.37.255
netmask=255.255.255.0
[2015/12/28 21:12:06.976876, 3] ../source3/smbd/server.c:1248(main)
loaded services
[2015/12/28 21:12:06.977004, 3] ../source3/smbd/server.c:1280(main)
Becoming a daemon.
[2015/12/28 21:12:07.738688, 3]
../lib/ldb-samba/ldb_wrap.c:320(ldb_wrap_connect)
ldb_wrap open of idmap.ldb
[2015/12/28 21:12:07.740665, 0]
../source3/passdb/pdb_interface.c:178(make_pdb_method_name)
pdb backend samba_dsdb did not correctly init (error was
NT_STATUS_UNSUCCESSFUL)


I am at a loss, Samba simply does not start. Any help/guidance the list
could provide to assist me in restoring Samba to a working state would be
greatly appreciated.

Regards,

JS
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[L.P.H. van Belle]

Hai,

Can be incorrect rights, of corrupted db.

Can you give the output of

ls -al /var/lib/samba/
ls -al /var/lib/samba/private
ls -al /var/lib/samba/private/dns



Greetz,

Louis


> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-...@lists.samba.org] Namens IT Admin
> Verzonden: woensdag 30 december 2015 4:37
> Aan: sa...@lists.samba.org
> Onderwerp: [Samba] Samba 4 AD - Samba Fails to Start,

[Rowland penny]

Why are you using ntvfs ?
Also how are you starting the Samba deamons ?

Rowland

[JS]

<=?windows-1252?Q?L.P.H._van_Belle?=> writes:

>
> Hai,
>
> Can be incorrect rights, of corrupted db.
>
> Can you give the output of
>
> ls -al /var/lib/samba/
> ls -al /var/lib/samba/private
> ls -al /var/lib/samba/private/dns
>
> Greetz,
>
> Louis
>

Hi Louis, thanks for your reply, here is the info you requested:

ls -al /var/lib/samba/
total 1376
drwxr-xr-x 8 root root 4096 Dec 13 21:07 .
drwxr-xr-x 59 root root 4096 Dec 13 20:16 ..
-rw------- 1 root root 421888 Dec 13 21:07 account_policy.tdb
drwxr-x--- 2 root root 4096 Dec 28 21:12 ntp_signd
drwxr-xr-x 10 root root 4096 Dec 13 20:51 printers
drwxr-xr-x 6 root root 4096 Dec 28 21:12 private
-rw------- 1 root root 528384 Dec 13 21:07 registry.tdb
-rw------- 1 root root 421888 Dec 13 21:07 share_info.tdb
drwxrwx---+ 6 root 3000000 4096 Dec 13 21:59 sysvol
drwxrwx--T 2 root sambashare 4096 Dec 13 20:36 usershares
drwxr-x--- 2 root root 4096 Dec 28 21:12 winbindd_privileged


ls -al /var/lib/samba/private/
total 11220
drwxr-xr-x 6 root root 4096 Dec 28 21:12 .
drwxr-xr-x 8 root root 4096 Dec 13 21:07 ..
-rw------- 1 root root 2085 Dec 13 21:07 dns_update_cache
-rw-r--r-- 1 root root 3183 Dec 13 21:03 dns_update_list
-rw------- 1 root root 1286144 Dec 13 21:02 hklm.ldb
-rw------- 1 root root 1609728 Dec 23 20:15 idmap.ldb
-rw-r--r-- 1 root root 99 Dec 13 21:03 krb5.conf
srwxrwxrwx 1 root root 0 Dec 28 21:12 ldapi
drwxr-x--- 2 root root 4096 Dec 28 21:12 ldap_priv
-r--r--r-- 1 root root 242 Dec 13 21:07 named.conf.update
-rw------- 1 root root 1286144 Dec 13 21:41 privilege.ldb
-rw------- 1 root root 696 Dec 13 21:07 randseed.tdb
-rw------- 1 root root 4247552 Dec 28 07:22 sam.ldb
drwx------ 2 root root 4096 Dec 13 21:02 sam.ldb.d
-rw------- 1 root root 696 Dec 28 21:12 schannel_store.tdb
-rw------- 1 root root 1212 Dec 13 21:03 secrets.keytab
-rw------- 1 root root 1286144 Dec 13 21:03 secrets.ldb
-rw------- 1 root root 430080 Dec 13 21:03 secrets.tdb
-rw------- 1 root root 1286144 Dec 13 21:02 share.ldb
drwxr-xr-x 3 root root 4096 Dec 13 21:07 smbd.tmp
-rw-r--r-- 1 root root 955 Dec 13 21:03 spn_update_list
drwx------ 2 root root 4096 Dec 13 21:07 tls


I have no /var/lib/samba/private/dns directory. Note that I am using
Samba's internal DNS server as opposed to Bind9 or anything else.

[JS]

Rowland penny <rpenny <at> samba.org> writes:

> Why are you using ntvfs ?
> Also how are you starting the Samba deamons ?
>
> Rowland
>

Hi Rowland,

Your mention of ntvfs is the first I've heard of it. A cursory search
reveals it was implemented in alpha versions of Samba4, did I provision this
domain incorrectly? Below are the commands I used when provisioning this
domain:

sudo samba-tool domain provision --use-rfc2307 --interactive
sudo samba-tool domain level raise --domain-level 2008_R2 --forest-level 2008_R2

Thanks for your reply.

[L.P.H. van Belle]

Ok,

First things is see.

NTP

drwxr-x--- 2 root root 4096 Dec 28 21:12 ntp_signd

should be root:ntp

SYVOL
drwxrwx---+ 3 root BUILTIN\administrators 4096 Apr 28 2015 sysvol
your shows 300000 while mine gives : BUILTIN\administrators
but i have winbind/nsswitch etc configured on my DC, dont ask why, but i need it, and it works good for me.

so besides your ntp folder this looks all ok.

Can you tell more about the hardware failure?
Disk problems, power outage etc what exact happend?
Did you see an filesystem check the first time starting up after the failuere?

I asume its the only server, do no other DC's.
Stop all samba processes and backup at least these folders.
/etc/samba
/var/lib/samba
/var/cache/samba

When you run : samba-tool fsmo show
You probely get an error, so try the following.
samba-tool fsmo sieze

( i dont think i will work, but give it a try, any outputs is most welkom )

These do worry me.

Failed to find object DC=one,DC=cliffbells,DC=com for attribute fsmoRoleOwner - Cannot find DN DC=one,DC=cliffbells,DC=com to get attribute fsmoRoleOwner for reference dn: (null)

./source4/dsdb/common/util.c:1877(samdb_is_pdc)
Failed to find if we are the PDC for this ldb: Searching for fSMORoleOwner in DC=one,DC=cliffbells,DC=com failed: Cannot find DN DC=one,DC=cliffbells,DC=com to get attribute fsmoRoleOwner for reference
dn: (null)

which looks like you samba DB is corrected, probely due to the hardware failure.

Do you have a backup, made with samba_backup ?
( shown here : https://wiki.samba.org/index.php/Backup_and_restore_an_Samba_AD_DC )

Because i think you db is corrected and beyond recovery.

If you have backupped :
/etc/samba
/var/lib/samba
/var/cache/samba

You can remove the content of
/var/lib/samba
/var/cache/samba

And reprovision, bases on the posts here and the things i see.
If you have a backup "any" which have also the samba databases, thats the first you can try.


Greetz,

Louis



> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-...@lists.samba.org] Namens JS
> Verzonden: woensdag 30 december 2015 23:42
> Aan: sa...@lists.samba.org
> Onderwerp: Re: [Samba] Samba 4 AD - Samba Fails to Start,

> hdb_samba4_create_kdc (setup KDC database) failed
>

[L.P.H. van Belle]

In addition.
You can try :
samba-tool dbcheck --cross-ncs --fix

but again, i think quicker with a backup restore or new provisioning.

Greetz,

Louis

> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-...@lists.samba.org] Namens L.P.H. van Belle
> Verzonden: donderdag 31 december 2015 9:46

[Rowland penny]

On 30/12/15 22:55, JS wrote:
> Rowland penny <rpenny <at> samba.org> writes:
>
>> Why are you using ntvfs ?
>> Also how are you starting the Samba deamons ?
>>
>> Rowland
>>
> Hi Rowland,
>
> Your mention of ntvfs is the first I've heard of it. A cursory search
> reveals it was implemented in alpha versions of Samba4, did I provision this
> domain incorrectly? Below are the commands I used when provisioning this
> domain:
>
> sudo samba-tool domain provision --use-rfc2307 --interactive

Yes, but what were your answers to the questions you were asked ?

Having read your reply to Louis and his answers, I am with him, you will
probably be better of provisioning again, but this time don't bother
with the interactive provision, set the required settings in the
provision command, run 'samba-tool domain provision --help' for what you
can set.

Rowland

[JS]

<=?windows-1252?Q?L.P.H._van_Belle?=> writes:

>
> Ok,
>
>

Hi Louis,

Thank you again for taking the time to help me out, I do appreciate it, and
I hope you had a safe and Happy New Year's eve. I'm going to work my way
through the questions/comments in your response from top to bottom:

> First things is see.
>
> NTP
> drwxr-x--- 2 root root 4096 Dec 28 21:12 ntp_signd
> should be root:ntp

No idea why the ownership is incorrect for that directory but I have
executed the following to fix it:

sudo chown -R root:ntp /var/lib/samba/ntp_signd

and now the security settings on that dir look like:

sudo ls -la /var/lib/samba/ntp_signd/
total 8
drwxr-x--- 2 root ntp 4096 Dec 28 21:12 .

drwxr-xr-x 8 root root 4096 Dec 13 21:07 ..

srwxrwxrwx 1 root ntp 0 Dec 28 21:12 socket

> SYVOL
> drwxrwx---+ 3 root BUILTIN\administrators 4096 Apr 28 2015 sysvol
> your shows 300000 while mine gives : BUILTIN\administrators
> but i have winbind/nsswitch etc configured on my DC, dont ask why, but i
need it, and it works good for me.

Regarding the SYSVOL permissions, I checked the permissions of
/var/lib/samba/ on another PDC I have deployed on a different network and
ntp_signd is owned by root:3000000 as well.

> Can you tell more about the hardware failure?
> Disk problems, power outage etc what exact happend?
> Did you see an filesystem check the first time starting up after the failuere?

The initial hardware failure was a RAID array failure, I replaced the failed
devices and rebuilt the array and then rebuilt their domain from scratch
provisioning under a new domain.

> I asume its the only server, do no other DC's.

Yes, that is correct, this machine is the only domain controller on this
network.

> Stop all samba processes and backup at least these folders.
> /etc/samba
> /var/lib/samba
> /var/cache/samba

Samba fails at boot, I've already made a couple of safety backups but for
good measure I stopped smbd, nmbd, and samba services and backed up the
directories you listed.

> When you run : samba-tool fsmo show

> You probely get an error...

I do receive an error, note I did not start any of the aforementioned
services prior to executing the samba-tool command below:

sudo samba-tool fsmo show
ldb_wrap open of secrets.ldb
ERROR(assert): uncaught exception
File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py", line
175, in _run
return self.run(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/samba/netcmd/fsmo.py", line 196, in run
assert len(res) == 1

> , so try the following.
> samba-tool fsmo sieze

I receive a second error when executing the seize command:

sudo samba-tool fsmo seize
ldb_wrap open of secrets.ldb
ERROR: Invalid FSMO role.

> ( i dont think i will work, but give it a try, any outputs is most welkom )
>
> These do worry me.
> Failed to find object DC=one,DC=cliffbells,DC=com for attribute
fsmoRoleOwner - Cannot find DN
> DC=one,DC=cliffbells,DC=com to get attribute fsmoRoleOwner for reference
dn: (null)
>
> ./source4/dsdb/common/util.c:1877(samdb_is_pdc)
> Failed to find if we are the PDC for this ldb: Searching for
fSMORoleOwner in DC=one,DC=cliffbells,DC=com
> failed: Cannot find DN DC=one,DC=cliffbells,DC=com to get attribute
fsmoRoleOwner for reference
> dn: (null)
>
> which looks like you samba DB is corrected, probely due to the hardware
failure.

If your hunch that the database is corrupt holds true it couldn't be from
hardware failure as this domain was provisioned after that incident. I do
believe I may have traced where any possible corruption might have
originated though... I (apparently foolishly) started backing up
/var/lib/samba with CrashPlan after the hardware failure incident... I'm
guessing that was a bad idea.

> Do you have a backup, made with samba_backup ?
> ( shown here :
https://wiki.samba.org/index.php/Backup_and_restore_an_Samba_AD_DC )
>
> Because i think you db is corrected and beyond recovery.

No, I do not have that backup mechanism implemented, and from reading that
wiki page's notes about backing up live databases I have come to the
conclusion that CrashPlan backed up /var/lib/samba/ while the databases were
live and irreparably damaged them. I don't know what the relationship
between /var/lib/samba/ and /var/cache/samba/ is exactly, but I assume that
any backup I had created via CrashPlan (if it had worked instead of wreaking
havoc) probably wouldn't have been valid lacking the /var/cache/samba/
directory contents... I will be implementing the Samba backup script from
your wiki link immediately on the other Samba ADCs I have deployed and will
utilize it here when I've rebuilt the domain, using CrashPlan for offsite
storage of archives it creates.

Which leads us your closing statement:

> If you have backupped :
> /etc/samba
> /var/lib/samba
> /var/cache/samba
>
> You can remove the content of
> /var/lib/samba
> /var/cache/samba
>
> And reprovision, bases on the posts here and the things i see.
> If you have a backup "any" which have also the samba databases, thats the
first you can try.
>
> Greetz,
>
> Louis

Other than the python error I received after running samba-tool fsmo show, I
believe I've built a pretty solid case for poor backup strategy being the
cause of this failure, and that reprovisioning the domain is my only course
of action at this time. If you believe I'm getting ahead of myself, or if
you think that Python error could lead to another failure after I've
reprovisioned, please let me know. I intend to execute the new domain
provisioning tomorrow (Sunday Jan 03 2016) in the late afternoon/early
evening (EST), and would hate to go through the process of rebuilding their
infrastructure only to have a Python issue trash the domain again.


Thanks again Louis et al for helping me troubleshoot this issue, I'm still
green when it comes to Samba.

Kind Regards,

[Rowland penny]

Just how are you starting Samba ? If you are running Samba as an AD DC,
you should only start the samba deamon, yet you say that you 'stopped
smbd, nmbd, and samba services', 'nmbd' should not be running on an AD
DC, it interferes with 'nbt' built into the samba deamon.

>> When you run : samba-tool fsmo show
>> You probely get an error...
> I do receive an error, note I did not start any of the aforementioned
> services prior to executing the samba-tool command below:
>
> sudo samba-tool fsmo show
> ldb_wrap open of secrets.ldb
> ERROR(assert): uncaught exception
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py", line
> 175, in _run
> return self.run(*args, **kwargs)
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/fsmo.py", line 196, in run
> assert len(res) == 1

Known problem that I have fixed in master, mind you, your version of
fsmo.py will only show 5 of the seven roles. Your problem seems to be
that at least one of your FSMO roles doesn't have a roleowner, hence
when the python code says it has (assert len(res) == 1), it throws an error.

As far as I am aware, you cannot backup a running Samba AD DC with
anything that doesn't use tdbbackup, unless you stop samba.

One of your problems is that you are using the stock Ubuntu samba, this
is getting a bit long in the tooth now, can I suggest you use either the
latest freely available samba from Sernet or better still, compile it
yourself and use the latest version 4.3.3. This will get you a much
improved fsmo.py and will also cover you for several CVEs.

Rowland

[Andrew Bartlett]

On Sun, 2016-01-03 at 08:37 +0000, Rowland penny wrote:
>
> As far as I am aware, you cannot backup a running Samba AD DC with
> anything that doesn't use tdbbackup, unless you stop samba.

To be clear, as long as the backup is only making reads, the impact
should only be on the backed-up DB, not on Samba.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

[Andrew Bartlett]

On Sun, 2016-01-03 at 06:00 +0000, JS wrote:
>
>
> Other than the python error I received after running samba-tool fsmo
> show, I
> believe I've built a pretty solid case for poor backup strategy being
> the
> cause of this failure, and that reprovisioning the domain is my only
> course
> of action at this time. If you believe I'm getting ahead of myself,
> or if
> you think that Python error could lead to another failure after I've
> reprovisioned, please let me know. I intend to execute the new
> domain
> provisioning tomorrow (Sunday Jan 03 2016) in the late
> afternoon/early
> evening (EST), and would hate to go through the process of rebuilding
> their
> infrastructure only to have a Python issue trash the domain again.

I've not seen an error like yours before. It suggests one of the key
objects that the KDC needs to start is not present in the DB.

This particular error is pretty damming:

> Failed to find object DC=one,DC=cliffbells,DC=com for attribute
> fsmoRoleOwner - Cannot find DN DC=one,DC=cliffbells,DC=com to get
> attribute
> fsmoRoleOwner for reference dn: (null)

That is, it can't find the base object for the whole domain.

What does 'samba-tool dbcheck' say? After a backup, does running it
with --fix resolve the issue or at least run clear?

If that is fixed (somehow), then what does 'samba-tool domain
exportkeytab' or 'pdbedit -L -v' say? Try turning up the debug level
to get a failure message if it fails.

But all said and done, it seems unlikely that that domain is in a
'good' enough state to continue.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

[JS]

Andrew Bartlett <abartlet <at> samba.org> writes:


>
> What does 'samba-tool dbcheck' say?

Running "sudo samba-tool dbfix" produces the following Python error:

sudo samba-tool dbcheck
ERROR(<type 'exceptions.IndexError'>): uncaught exception - list index out
of range

File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py", line
175, in _run
return self.run(*args, **kwargs)

File "/usr/lib/python2.7/dist-packages/samba/netcmd/dbcheck.py", line 120,
in run
reset_well_known_acls=reset_well_known_acls)
File "/usr/lib/python2.7/dist-packages/samba/dbchecker.py", line 87, in
__init__
dnsadmins_sid = ndr_unpack(security.dom_sid, res[0]["objectSid"][0])

Appreciate you joining the conversation Andrew, do you think CrashPlan
corrupted this database? I can't think of anything else I could have done
that would've caused such a drastic failure and would like to know so I
don't repeat the blunder in the future, this has been a royal PITA.

JS

[JS]

Rowland penny <rpenny <at> samba.org> writes:

>

A couple questions...

1) I've downloaded the latest samba source files and obtained the backup
script. Ubuntu's distribution of samba installs to different locations than
compiling from source from what I can tell yet I haven't been able to find
any references online that dictate exactly what directories need to be
backed up to create valid archives on Ubuntu, could you provide some insight
in this regard?

2) How would I go about 'upgrading' a deployed Samba4 PDC from the version
provided by Canonical to one compiled from source? Is it possible to do an
'in-place' upgrade or will I need to uninstall, compile the new version and
then redeploy?

Thanks again for all the help.

[Rowland penny]

On 03/01/16 10:38, JS wrote:
> Rowland penny <rpenny <at> samba.org> writes:
>
>> On 03/01/16 06:00, JS wrote:
>>> <=?windows-1252?Q?L.P.H._van_Belle?=> writes:
>>>
>> One of your problems is that you are using the stock Ubuntu samba, this
>> is getting a bit long in the tooth now, can I suggest you use either the
>> latest freely available samba from Sernet or better still, compile it
>> yourself and use the latest version 4.3.3. This will get you a much
>> improved fsmo.py and will also cover you for several CVEs.
>>
>> Rowland
>>> Kind Regards,
>>>
>>> JS
>>>
>>>
> A couple questions...
>
> 1) I've downloaded the latest samba source files and obtained the backup
> script. Ubuntu's distribution of samba installs to different locations than
> compiling from source from what I can tell yet I haven't been able to find
> any references online that dictate exactly what directories need to be
> backed up to create valid archives on Ubuntu, could you provide some insight
> in this regard?

This has, I believe, been done, but not by me, so I am unsure just what
you need to backup and more importantly, where you need to the various
backup files. What to backup is fairly easy, Ubuntu uses two locations,
/var/lib/samba and /var/cache/samba, just backup everything in these two
locations that end in .ldb & .tdb using tdbbackup. If you self compile
samba, everything goes into /usr/local/samba and you would need to put
the files you backed up into the required places, if it would help, I
can provide a directory listing for /usr/loca/samba.

>
> 2) How would I go about 'upgrading' a deployed Samba4 PDC from the version
> provided by Canonical to one compiled from source? Is it possible to do an
> 'in-place' upgrade or will I need to uninstall, compile the new version and
> then redeploy?

Got to be honest here, from what you have posted so far, I get the
feeling your Samba database is badly corrupted and you will have to
start again. It should be possible to upgrade if your database is ok,
but I would do it first on a test machine.

Rowland

[JS]

Rowland penny <rpenny <at> samba.org> writes:

>
>
> This has, I believe, been done, but not by me, so I am unsure just what
> you need to backup and more importantly, where you need to the various
> backup files. What to backup is fairly easy, Ubuntu uses two locations,
> /var/lib/samba and /var/cache/samba, just backup everything in these two
> locations that end in .ldb & .tdb using tdbbackup. If you self compile
> samba, everything goes into /usr/local/samba and you would need to put
> the files you backed up into the required places, if it would help, I
> can provide a directory listing for /usr/loca/samba.
>

So, samba_backup contains the following variables:

FROMWHERE=/usr/local/samba
WHERE=/usr/local/backups
DAYS=90

which are condigured to work with Samba compiled from source.

Ubuntu utilizes /var/lib/samba and /var/cache/samba, does samba_backup's
FROMWHERE variable support multiple directories, or would I have to modify
the script to handle two source targets? The latter option here is a bit
out of my scope...

> Got to be honest here, from what you have posted so far, I get the
> feeling your Samba database is badly corrupted and you will have to
> start again. It should be possible to upgrade if your database is ok,
> but I would do it first on a test machine.
>

I understand and agree completely that this domain in particular is likely
beyond repair and will need to be reprovisioned, and that uninstalling the
current version of Samba and compiling from source prior to doing so is my
best course of action. This question was in relation to other PDCs I have
deployed elsewhere, after chatting with you here on the list it seems
modifying those clients would be in my best interest, I'm just not sure how
to go about executing the upgrade.

Regards,

[Andrew Bartlett]

Is there really a Samba database in the location shown by:

bin/testparm --parameter-name=privatedir --suppress-prompt

That is, a sam.ldb, a secrets.ldb and (importantly) sam.ldb.d/ with the
usual files in that (metadata.tdb, other files ending in .ldb named
after your domain).

It looks to me like this has been removed (or we have been pointed at
the wrong location), and Samba has re-created an empty DB for sam.ldb,
with nothing in it. I mention this because the alternative is that it
is damaged beyond (costly/tedious/manual) repair involving a rebuild
and putting back some of the old values.

The last time I came across a DB failure like this, I blamed a DRDB
setup that didn't honour 'barriers' and an unexpected power-off. The
DB was only able to be partially rescued with the new 'ldbdump' tool we
wrote. In that case the domain was able to hobble on for a few weeks,
but was rebuilt.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

[JS]

Andrew Bartlett <abartlet <at> samba.org> writes:

> Is there really a Samba database in the location shown by:
>
> bin/testparm --parameter-name=privatedir --suppress-prompt
>
> That is, a sam.ldb, a secrets.ldb and (importantly) sam.ldb.d/ with the
> usual files in that (metadata.tdb, other files ending in .ldb named
> after your domain).

Hi Andrew,

Here is the result of your testparm command:

sudo testparm --parameter-name=privatedir --suppress-prompt

Load smb config files from /etc/samba/smb.conf

rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)

Processing section "[netlogon]"
Processing section "[sysvol]"
Processing section "[accounting]"
Processing section "[data]"
Processing section "[backups]"
Loaded services file OK.
/var/lib/samba/private



ls -la /var/lib/samba/private/
total 11220
drwxr-xr-x 6 root root 4096 Dec 28 21:12 .

drwxr-xr-x 8 root root 4096 Dec 13 21:07 ..

-rw------- 1 root root 2085 Dec 13 21:07 dns_update_cache
-rw-r--r-- 1 root root 3183 Dec 13 21:03 dns_update_list
-rw------- 1 root root 1286144 Dec 13 21:02 hklm.ldb
-rw------- 1 root root 1609728 Dec 23 20:15 idmap.ldb
-rw-r--r-- 1 root root 99 Dec 13 21:03 krb5.conf
srwxrwxrwx 1 root root 0 Dec 28 21:12 ldapi
drwxr-x--- 2 root root 4096 Dec 28 21:12 ldap_priv
-r--r--r-- 1 root root 242 Dec 13 21:07 named.conf.update
-rw------- 1 root root 1286144 Dec 13 21:41 privilege.ldb
-rw------- 1 root root 696 Dec 13 21:07 randseed.tdb
-rw------- 1 root root 4247552 Dec 28 07:22 sam.ldb
drwx------ 2 root root 4096 Dec 13 21:02 sam.ldb.d
-rw------- 1 root root 696 Dec 28 21:12 schannel_store.tdb
-rw------- 1 root root 1212 Dec 13 21:03 secrets.keytab
-rw------- 1 root root 1286144 Dec 13 21:03 secrets.ldb
-rw------- 1 root root 430080 Dec 13 21:03 secrets.tdb
-rw------- 1 root root 1286144 Dec 13 21:02 share.ldb
drwxr-xr-x 3 root root 4096 Dec 13 21:07 smbd.tmp
-rw-r--r-- 1 root root 955 Dec 13 21:03 spn_update_list

drwx------ 2 root root 4096 Dec 13 21:07 tls



sudo ls -la /var/lib/samba/private/sam.ldb.d/
total 39000
drwx------ 2 root root 4096 Dec 13 21:02 .
drwxr-xr-x 6 root root 4096 Dec 28 21:12 ..
-rw------- 1 root root 16384000 Dec 28 07:22
CN=CONFIGURATION,DC=ONE,DC=CLIFFBELLS,DC=COM.ldb
-rw------- 1 root root 10383360 Dec 28 07:22
CN=SCHEMA,CN=CONFIGURATION,DC=ONE,DC=CLIFFBELLS,DC=COM.ldb

-rw------- 1 root root 4247552 Dec 28 07:22

DC=DOMAINDNSZONES,DC=ONE,DC=CLIFFBELLS,DC=COM.ldb

-rw------- 1 root root 4247552 Dec 28 07:22

DC=FORESTDNSZONES,DC=ONE,DC=CLIFFBELLS,DC=COM.ldb
-rw------- 1 root root 4243456 Dec 28 07:22 DC=ONE,DC=CLIFFBELLS,DC=COM.ldb
-rw-r----- 1 root root 421888 Dec 27 21:44 metadata.tdb

>
> It looks to me like this has been removed (or we have been pointed at
> the wrong location), and Samba has re-created an empty DB for sam.ldb,
> with nothing in it. I mention this because the alternative is that it
> is damaged beyond (costly/tedious/manual) repair involving a rebuild
> and putting back some of the old values.

It looks to me like everything is correct there...

>
> The last time I came across a DB failure like this, I blamed a DRDB
> setup that didn't honour 'barriers' and an unexpected power-off. The
> DB was only able to be partially rescued with the new 'ldbdump' tool we
> wrote. In that case the domain was able to hobble on for a few weeks,
> but was rebuilt.
>
> Andrew Bartlett
>

Not sre if the ldbdump tool you mention could help in this scenario or not.
This machine is on an APC UPS so sudden shutdown shouldn't have been an issue.

JS