Windows server 2008 crashes at startup – c00002e2

/in , /by

STOP: c00002e2 Directory Services could not start because of the following error: a device attached to the system is not functioning.

Applies to: Windows Server 2008, Windows Server 2008 R2 with Exchange 2007 installed on single node.

You might face this issue while you are running your daily work, first you will panic and think that there is a huge problem and your domain controller is failing, not to mention the heat you will face if that domain controller machine is hosting your exchange 2007 too, which is typically the scenario I have seen in small business environment.

Cause: 

This problem occurs because one or more of the following conditions are true:

– The NTFS file system permissions on the root of the drive are too restrictive.

– The NTFS file system permissions on the NTDS folder are too restrictive.

– The drive letter of the volume that contains the Active Directory database has changed.

– The Active Directory database (Ntds.dit) is corrupted.

– The NTDS folder is compressed.

Troubleshooting steps:

  1. Boot into Directory Services Restore Mode (F8). You might need to do it twice if you have an error on your HDD.
  2. Run CHKDSK in read only mode to verify that there are no errors on your hard drive. If the system prompts you that there was an error on the hard disk, then feel safe to run /R command CHKDSK /R will not ruin your hard disk data, so don’t believe all the myths said.
    1. Boot again into Directory Services Restore Mode (F8).
    2. Go to command prompt (administrative privileges) and run the following command:
    3. NTDSUTIL
    4. activate instance NTDS
    5. files info

You will see an error similar to the following:

“Error: Could not initialize the Jet engine: Jet Error -501. Failed to open DIT for AD DS/LDS instance NTDS. Error -2147418113”

Event log: 

  • Error 1003: Active Directory Domain Services could not be initialized. The directory service cannot recover from this error. Restore the local directory service from backup media. Error value: -501 JET_errLogFileCorrupt, Log file is corrupt
  • Error 465: NTDS (2156) Corruption was detected during soft recovery in logfile C:\Windows\NTDS\edb.log. The failing checksum record is located at position END. Data not matching the log-file fill pattern first appeared in sector 6697 (0x00001A29). This logfile has been damaged and is unusable.
  • Error 454: NTDS (2256) Database recovery/restore failed with unexpected error -501.

Solution: 

The solution of this error is very simple and will be done in one step:

Go to C:\Windows\NTDS folder and rename all *.log files to .old and restart system.

Example, edb.log should be renamed to edb.log.old

This will solve the issue and your active directory domain services will be started again.

It might be wise in this phase to consider replacing your HDD and checking your GPO policies, if you see any more file corruption.

If you find this article helpful, please send us a note to WDallal@bostonIT.com so we can keep on adding quality hands-on articles.

Anyconnect Memory Locks up and Cert8

/in , /by

AnyConnect Locks up on Linux before it finally connects because of high memory usage that could go up to 100%. Machine would need to be rebooted to recover. Kubuntu, Ubuntu and Linux.

Scenario:

When using the Linux Cisco AnyConnect client x64 (like Kubuntu), memory usage gradually starts going up until it’s all used up. Anyconnect won’t connect.

I ran the VPN client within gdb to try and get a sense of what it was doing when trying to allocate so much memory.

Thread 3 of the process below is the only active thread and you can see that it is performing certificate related activities. Below are more related logs that I have collected

 

(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff28b3700 (LWP 3656))]
#0 0x00007ffff49f8180 in PR_Free () from /usr/lib/x86_64-linux-gnu/libnspr4.so
(gdb) bt
#0 0x00007ffff49f8180 in PR_Free () from /usr/lib/x86_64-linux-gnu/libnspr4.so
#1 0x00007ffff35bab1a in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#2 0x00007ffff35bcb15 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#3 0x00007ffff35be6eb in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#4 0x00007ffff35c3880 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#5 0x00007ffff35c413b in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#6 0x00007ffff35c41e7 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#7 0x00007ffff35bf262 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#8 0x00007ffff35bf751 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#9 0x00007ffff3a9c8e2 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so
#10 0x00007ffff3a87e75 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so
#11 0x00007ffff3a8c78c in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so
#12 0x00007ffff52b0d60 in ?? () from /usr/lib/x86_64-linux-gnu/libnss3.so
#13 0x00007ffff52aa4d2 in ?? () from /usr/lib/x86_64-linux-gnu/libnss3.so
#14 0x00007ffff5272091 in CERT_GetCertNicknames () from /usr/lib/x86_64-linux-gnu/libnss3.so
#15 0x00007ffff5272149 in CERT_FindUserCertsByUsage () from /usr/lib/x86_64-linux-gnu/libnss3.so
#16 0x00007ffff7844cc5 in CNSSCertStore::Enumerate(eCertType,
std::list<CCertificate*, std::allocator<CCertificate*> >&) () from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#17 0x00007ffff7818474 in CCollectiveCertStore::Enumerate(eCertType,
std::list<CCertificate*, std::allocator<CCertificate*> >&) () from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#18 0x00007ffff781363c in CCertStore::GetCertificates(CERT_ENTRY*,
CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*>
>&) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#19 0x00007ffff7818575 in
CCollectiveCertStore::GetCertificates(CERT_ENTRY*, CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*> >&) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#20 0x00007ffff780fb1c in
CCertHelper::GetClientCertificates(CERT_ENTRY*, CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*> >&, unsigned int) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#21 0x00007ffff7abb5d9 in ApiCert::getCertList(CERT_ENTRY*, CCertNameList*, std::string const&, ConnectProtocolType) () from /opt/cisco/anyconnect/lib/libvpnapi.so
#22 0x00007ffff7abbcb5 in ApiCert::getCertList(CERT_ENTRY*, std::string const&, ConnectProtocolType) () from /opt/cisco/anyconnect/lib/libvpnapi.so
#23 0x00007ffff7ace406 in ConnectMgr::resetCertRegistration(std::string) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#24 0x00007ffff7ae565b in ConnectMgr::setConnectionData(std::string
const&) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#25 0x00007ffff7aed002 in ConnectMgr::initiateConnect(std::string
const&, bool) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#26 0x00007ffff7af3c1a in ConnectMgr::run() () from /opt/cisco/anyconnect/lib/libvpnapi.so
#27 0x00007ffff7ac608a in ApiThread::threadProcedure(void*) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#28 0x00007ffff6753e9a in start_thread (arg=0x7ffff28b3700) at
pthread_create.c:308
#29 0x00007ffff5c6eccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112

SYSLOG logs:

std::list<CCertificate*, std::allocator<CCertificate*> >&) () from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#18 0x00007ffff781363c in CCertStore::GetCertificates(CERT_ENTRY*,
CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*>
>&) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#19 0x00007ffff7818575 in
CCollectiveCertStore::GetCertificates(CERT_ENTRY*, CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*> >&) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#20 0x00007ffff780fb1c in
CCertHelper::GetClientCertificates(CERT_ENTRY*, CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*> >&, unsigned int) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#21 0x00007ffff7abb5d9 in ApiCert::getCertList(CERT_ENTRY*, CCertNameList*, std::string const&, ConnectProtocolType) () from /opt/cisco/anyconnect/lib/libvpnapi.so
#22 0x00007ffff7abbcb5 in ApiCert::getCertList(CERT_ENTRY*, std::string const&, ConnectProtocolType) () from /opt/cisco/anyconnect/lib/libvpnapi.so
#23 0x00007ffff7ace406 in ConnectMgr::resetCertRegistration(std::string) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#24 0x00007ffff7ae565b in ConnectMgr::setConnectionData(std::string
const&) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#25 0x00007ffff7aed002 in ConnectMgr::initiateConnect(std::string
const&, bool) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#26 0x00007ffff7af3c1a in ConnectMgr::run() () from /opt/cisco/anyconnect/lib/libvpnapi.so
#27 0x00007ffff7ac608a in ApiThread::threadProcedure(void*) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#28 0x00007ffff6753e9a in start_thread (arg=0x7ffff28b3700) at
pthread_create.c:308
#29 0x00007ffff5c6eccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112

The relevant portion of the syslog follows. Note that the kernel experiences a page allocation failure shortly after the VPN connection is initiated.

Oct 31 15:08:01 Kubuntu acvpncli[1509]: Initializing vpnapi version
3.1.04072 ().
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: loadProfiles File:
../../vpn/Api/ProfileMgr.cpp Line: 100 No profile is available.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: getCurrentState
File: ../../vpn/Api/ClientIfcBase.cpp Line: 2058 API service not ready Oct 31 15:08:01 Kubuntu acvpncli[1509]: Current Preference Settings:
ServiceDisable: false CertificateStoreOverride: false CertificateStore:
All ShowPreConnectMessage: false AutoConnectOnStart: false
MinimizeOnConnect: true LocalLanAccess: true AutoReconnect: true
AutoUpdate: true ProxySettings: Native AllowLocalProxyConnections: true
PPPExclusion: Disable PPPExclusionServerIP: EnableScripting: false
TerminateScriptOnNextEvent: false AuthenticationTimeout: 12
IPProtocolSupport: IPv4,IPv6 AllowManualHostInput: true
BlockUntrustedServers: false PublicProxyServerAddress:
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
OnNegotiateMessageTypesComplete File: ../../vpn/Api/ApiIpc.cpp Line: 726 Master Agent Connection started.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: VPN state: Disconnected Network state: Network Accessible Network control state: Network Access:
Available Network type: Undefined
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
setConnectRequestComplete File: ../../vpn/Api/ConnectMgr.cpp Line: 9133 Connect request complete. Proceeding to cleanup.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
activateConnectEvent File: ../../vpn/Api/ConnectMgr.cpp Line: 1352 NULL object. Cannot establish a connection at this time.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Message type information sent to the user: Ready to connect.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: attach File:
../../vpn/Api/ClientIfcBase.cpp Line: 629 Client successfully attached.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: WMHintCB File:
../../vpn/Api/ClientIfc.cpp Line: 146 User did not implement WMHintCB.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: WMHintCB File:
../../vpn/Api/ClientIfc.cpp Line: 146 User did not implement WMHintCB.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: An SSL VPN connection to MyVPN.Server.com has been requested by the user.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
getProfileNameFromHost File: ../../vpn/Api/ProfileMgr.cpp Line: 793 No profile available for host MyVPN.Server.com.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: getHostInitSettings
File: ../../vpn/Api/ProfileMgr.cpp Line: 873 Profile () not found. Using default settings.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
deliverWebLaunchHostCB File: ../../vpn/Api/ClientIfc.cpp Line: 152 User did not implement deliverWebLaunchHostCB.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: loadProfiles File:
../../vpn/Api/ProfileMgr.cpp Line: 100 No profile is available.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
getProfileNameFromHost File: ../../vpn/Api/ProfileMgr.cpp Line: 793 No profile available for host MyVPN.Server.com.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Using default preferences.
Some settings (e.g. certificate matching) may not function as expected if a local profile is expected to be used. Verify that the selected host is in the server list section of the profile and that the profile is configured on the secure gateway.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
getProfileNameFromHost File: ../../vpn/Api/ProfileMgr.cpp Line: 793 No profile available for host MyVPN.Server.com.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: getHostInitSettings
File: ../../vpn/Api/ProfileMgr.cpp Line: 873 Profile () not found. Using default settings.
Oct 31 15:08:03 Kubuntu kernel: [395314.511185] kworker/u:2: page allocation failure: order:1, mode:0x4020

 

Resolution:

After troubleshooting, it turned out that Firefox cert8.db might have been corrupted (Anyconnect relies on some components of Firefox especially certs), In the case above, anyconnect is just not liking something about that file on this particular machine, or the issue might be an undocumented bug in Anyconnect on Kubuntu or Linux in general

Renaming the following two files allowed me to correct the issue (make sure you close Firefox first)

/home/Kubuntu/.mozilla/firefox/8n6w3k0u.default/cert8.db

/home/Kubuntu/.mozilla/firefox/8n6w3k0u.default/key3.db

After you rename the two files, start Firefox once and it will auto-recreate them.

Note: those files are profile specific, i.e. if you login to the machine with a different user and start Anyconnect, you might not face the same issue.

If you find this article helpful, please send us a note to Mike@bostonIT.com so I can keep on adding quality hands-on articles.