Windows Updates Fail – Error 80070216

/in /by

Windows 7 – Windows Update keeps failing with Error 80070216

Scenario:

When trying to install Windows updates on a Windows 7, it keeps failing with the error above.

Resolutions:

Machine is infected with a rootkit or a virus. Download and run Kaspersky TDSSKiller, then download and run Combofix (you can google them). These two tools will clean-up your machine. Windows Update should work after that.

If you find this article helpful, please send me a note to Mike@bostonIT.com and so I can keep on adding more hands-on knowledgebase articles.

Anyconnect Memory Locks up and Cert8

/in , /by

AnyConnect Locks up on Linux before it finally connects because of high memory usage that could go up to 100%. Machine would need to be rebooted to recover. Kubuntu, Ubuntu and Linux.

Scenario:

When using the Linux Cisco AnyConnect client x64 (like Kubuntu), memory usage gradually starts going up until it’s all used up. Anyconnect won’t connect.

I ran the VPN client within gdb to try and get a sense of what it was doing when trying to allocate so much memory.

Thread 3 of the process below is the only active thread and you can see that it is performing certificate related activities. Below are more related logs that I have collected

 

(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff28b3700 (LWP 3656))]
#0 0x00007ffff49f8180 in PR_Free () from /usr/lib/x86_64-linux-gnu/libnspr4.so
(gdb) bt
#0 0x00007ffff49f8180 in PR_Free () from /usr/lib/x86_64-linux-gnu/libnspr4.so
#1 0x00007ffff35bab1a in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#2 0x00007ffff35bcb15 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#3 0x00007ffff35be6eb in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#4 0x00007ffff35c3880 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#5 0x00007ffff35c413b in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#6 0x00007ffff35c41e7 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#7 0x00007ffff35bf262 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#8 0x00007ffff35bf751 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libnssdbm3.so
#9 0x00007ffff3a9c8e2 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so
#10 0x00007ffff3a87e75 in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so
#11 0x00007ffff3a8c78c in ?? () from
/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so
#12 0x00007ffff52b0d60 in ?? () from /usr/lib/x86_64-linux-gnu/libnss3.so
#13 0x00007ffff52aa4d2 in ?? () from /usr/lib/x86_64-linux-gnu/libnss3.so
#14 0x00007ffff5272091 in CERT_GetCertNicknames () from /usr/lib/x86_64-linux-gnu/libnss3.so
#15 0x00007ffff5272149 in CERT_FindUserCertsByUsage () from /usr/lib/x86_64-linux-gnu/libnss3.so
#16 0x00007ffff7844cc5 in CNSSCertStore::Enumerate(eCertType,
std::list<CCertificate*, std::allocator<CCertificate*> >&) () from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#17 0x00007ffff7818474 in CCollectiveCertStore::Enumerate(eCertType,
std::list<CCertificate*, std::allocator<CCertificate*> >&) () from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#18 0x00007ffff781363c in CCertStore::GetCertificates(CERT_ENTRY*,
CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*>
>&) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#19 0x00007ffff7818575 in
CCollectiveCertStore::GetCertificates(CERT_ENTRY*, CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*> >&) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#20 0x00007ffff780fb1c in
CCertHelper::GetClientCertificates(CERT_ENTRY*, CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*> >&, unsigned int) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#21 0x00007ffff7abb5d9 in ApiCert::getCertList(CERT_ENTRY*, CCertNameList*, std::string const&, ConnectProtocolType) () from /opt/cisco/anyconnect/lib/libvpnapi.so
#22 0x00007ffff7abbcb5 in ApiCert::getCertList(CERT_ENTRY*, std::string const&, ConnectProtocolType) () from /opt/cisco/anyconnect/lib/libvpnapi.so
#23 0x00007ffff7ace406 in ConnectMgr::resetCertRegistration(std::string) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#24 0x00007ffff7ae565b in ConnectMgr::setConnectionData(std::string
const&) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#25 0x00007ffff7aed002 in ConnectMgr::initiateConnect(std::string
const&, bool) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#26 0x00007ffff7af3c1a in ConnectMgr::run() () from /opt/cisco/anyconnect/lib/libvpnapi.so
#27 0x00007ffff7ac608a in ApiThread::threadProcedure(void*) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#28 0x00007ffff6753e9a in start_thread (arg=0x7ffff28b3700) at
pthread_create.c:308
#29 0x00007ffff5c6eccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112

SYSLOG logs:

std::list<CCertificate*, std::allocator<CCertificate*> >&) () from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#18 0x00007ffff781363c in CCertStore::GetCertificates(CERT_ENTRY*,
CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*>
>&) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#19 0x00007ffff7818575 in
CCollectiveCertStore::GetCertificates(CERT_ENTRY*, CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*> >&) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#20 0x00007ffff780fb1c in
CCertHelper::GetClientCertificates(CERT_ENTRY*, CCertNameList*, std::list<CCertificate*, std::allocator<CCertificate*> >&, unsigned int) ()
from /opt/cisco/anyconnect/lib/libvpncommoncrypt.so
#21 0x00007ffff7abb5d9 in ApiCert::getCertList(CERT_ENTRY*, CCertNameList*, std::string const&, ConnectProtocolType) () from /opt/cisco/anyconnect/lib/libvpnapi.so
#22 0x00007ffff7abbcb5 in ApiCert::getCertList(CERT_ENTRY*, std::string const&, ConnectProtocolType) () from /opt/cisco/anyconnect/lib/libvpnapi.so
#23 0x00007ffff7ace406 in ConnectMgr::resetCertRegistration(std::string) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#24 0x00007ffff7ae565b in ConnectMgr::setConnectionData(std::string
const&) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#25 0x00007ffff7aed002 in ConnectMgr::initiateConnect(std::string
const&, bool) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#26 0x00007ffff7af3c1a in ConnectMgr::run() () from /opt/cisco/anyconnect/lib/libvpnapi.so
#27 0x00007ffff7ac608a in ApiThread::threadProcedure(void*) ()
from /opt/cisco/anyconnect/lib/libvpnapi.so
#28 0x00007ffff6753e9a in start_thread (arg=0x7ffff28b3700) at
pthread_create.c:308
#29 0x00007ffff5c6eccd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112

The relevant portion of the syslog follows. Note that the kernel experiences a page allocation failure shortly after the VPN connection is initiated.

Oct 31 15:08:01 Kubuntu acvpncli[1509]: Initializing vpnapi version
3.1.04072 ().
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: loadProfiles File:
../../vpn/Api/ProfileMgr.cpp Line: 100 No profile is available.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: getCurrentState
File: ../../vpn/Api/ClientIfcBase.cpp Line: 2058 API service not ready Oct 31 15:08:01 Kubuntu acvpncli[1509]: Current Preference Settings:
ServiceDisable: false CertificateStoreOverride: false CertificateStore:
All ShowPreConnectMessage: false AutoConnectOnStart: false
MinimizeOnConnect: true LocalLanAccess: true AutoReconnect: true
AutoUpdate: true ProxySettings: Native AllowLocalProxyConnections: true
PPPExclusion: Disable PPPExclusionServerIP: EnableScripting: false
TerminateScriptOnNextEvent: false AuthenticationTimeout: 12
IPProtocolSupport: IPv4,IPv6 AllowManualHostInput: true
BlockUntrustedServers: false PublicProxyServerAddress:
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
OnNegotiateMessageTypesComplete File: ../../vpn/Api/ApiIpc.cpp Line: 726 Master Agent Connection started.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: VPN state: Disconnected Network state: Network Accessible Network control state: Network Access:
Available Network type: Undefined
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
setConnectRequestComplete File: ../../vpn/Api/ConnectMgr.cpp Line: 9133 Connect request complete. Proceeding to cleanup.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
activateConnectEvent File: ../../vpn/Api/ConnectMgr.cpp Line: 1352 NULL object. Cannot establish a connection at this time.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Message type information sent to the user: Ready to connect.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: attach File:
../../vpn/Api/ClientIfcBase.cpp Line: 629 Client successfully attached.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: WMHintCB File:
../../vpn/Api/ClientIfc.cpp Line: 146 User did not implement WMHintCB.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: WMHintCB File:
../../vpn/Api/ClientIfc.cpp Line: 146 User did not implement WMHintCB.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: An SSL VPN connection to MyVPN.Server.com has been requested by the user.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
getProfileNameFromHost File: ../../vpn/Api/ProfileMgr.cpp Line: 793 No profile available for host MyVPN.Server.com.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: getHostInitSettings
File: ../../vpn/Api/ProfileMgr.cpp Line: 873 Profile () not found. Using default settings.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
deliverWebLaunchHostCB File: ../../vpn/Api/ClientIfc.cpp Line: 152 User did not implement deliverWebLaunchHostCB.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: loadProfiles File:
../../vpn/Api/ProfileMgr.cpp Line: 100 No profile is available.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
getProfileNameFromHost File: ../../vpn/Api/ProfileMgr.cpp Line: 793 No profile available for host MyVPN.Server.com.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Using default preferences.
Some settings (e.g. certificate matching) may not function as expected if a local profile is expected to be used. Verify that the selected host is in the server list section of the profile and that the profile is configured on the secure gateway.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function:
getProfileNameFromHost File: ../../vpn/Api/ProfileMgr.cpp Line: 793 No profile available for host MyVPN.Server.com.
Oct 31 15:08:01 Kubuntu acvpncli[1509]: Function: getHostInitSettings
File: ../../vpn/Api/ProfileMgr.cpp Line: 873 Profile () not found. Using default settings.
Oct 31 15:08:03 Kubuntu kernel: [395314.511185] kworker/u:2: page allocation failure: order:1, mode:0x4020

 

Resolution:

After troubleshooting, it turned out that Firefox cert8.db might have been corrupted (Anyconnect relies on some components of Firefox especially certs), In the case above, anyconnect is just not liking something about that file on this particular machine, or the issue might be an undocumented bug in Anyconnect on Kubuntu or Linux in general

Renaming the following two files allowed me to correct the issue (make sure you close Firefox first)

/home/Kubuntu/.mozilla/firefox/8n6w3k0u.default/cert8.db

/home/Kubuntu/.mozilla/firefox/8n6w3k0u.default/key3.db

After you rename the two files, start Firefox once and it will auto-recreate them.

Note: those files are profile specific, i.e. if you login to the machine with a different user and start Anyconnect, you might not face the same issue.

If you find this article helpful, please send us a note to Mike@bostonIT.com so I can keep on adding quality hands-on articles.

Cisco RV082 NAT Example

/in /by

Cisco RV082 One To One NAT – Access Rules Example

Example:
Public IP Address: 75.75.75.75
Private IP Address: 192.168.1.10

In this example we will NAT 192.168.1.10 to 75.75.75.75 and open up TCP Port 23 (Telnet) to the inside host from outside.

– Login to the Cisco RV 082 through the browser
– On the left click on Setup then One-To-One NAT
– Click Enable One-To-One NAT. In the Private Range Begin: fill in192.168.1.10. In the Public Range Begin type 75.75.75.75. In theRange Length type 1. Click Add to List. Click Save.

Click on Firewall on the left.. Click Access Rules. Click Add.
For Action click Allow
For Service choose TELNET TCP 23-23
For Log: choose the option you want
For Source: choose the outside interface of the Router. In my case it isWAN 1
For Source IP: Choose ANY (if you would like it to be available to anyone on the Internet
For Destination: Choose the Inside IP address of the NAT 192.168.1.10 (NOT The Public IP Address)
Click Save

Now create a new rule to deny any other access to the NAT’d host:
Click on Firewall on the left.. Click Access Rules. Click Add.
For Action click Deny
For Service choose Any
For Log: choose the option you want
For Source: choose the outside interface of the Router. In my case it isWAN 1
For Source IP: Choose ANY (if you would like it to be available to anyone on the Internet
For Destination: Choose the Inside IP address of the NAT 192.168.1.10 (NOT The Public IP Address)
Click Save

If you find this article helpful, please send me a note to Mike@bostonIT.com so I can keep on adding more hands-on knowledgebase articles.

Backup Exec fails to backup Exchange server with VSS error

/in /by

Windows Server 2008 R2 – Backup Exec 2010 / 12.5 – AOFO: Initialization failure on: “\\SERVER01\Microsoft Information Store\Information Store”. Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS). V-79-10000-11226 – VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: “Unexpected provider error”. Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.

Scenario:

Windows Server 2008 R2 – Symantec Backup Exec fails backing up Exchange Server. The following error is recorded in Windows System Log:

Log Name: Application
Source: VSS
Date: 8/15/2013 6:34:55 PM
Event ID: 8193
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: Server01.Domain.local
Description:
Volume Shadow Copy Service error: Unexpected error calling routine Cannot find
anymore diff area candidates for volume \\?\Volume{f6cd5a9b-04cf-11e1-b482-5cf3fc2b627f}\ [0].
hr = 0x8000ffff, Catastrophic failure.

Operation:
Automatically choosing a diff-area volume
Processing EndPrepareSnapshots

Context:

Volume Name: \\?\Volume{f6cd5a9b-04cf-11e1-b482-5cf3fc2b627f}\
Execution Context: System Provider
Event Xml:
Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”
System
Provider Name=”VSS”
EventID Qualifiers=”0″8193
2
0
Keywords0x80000000000000 Keywords
TimeCreated SystemTime=”2013-08-15T22:34:55.000000000Z”
EventRecordID316735 /EventRecordID
Channel Application Channel
Computer Server01.Domain.local/Computer
EventData
Cannot find anymore diff area candidates for
0x8000ffff, Catastrophic failure

Operation:
Automatically choosing a diff-area volume
Processing EndPrepareSnapshots

Context:
Volume Name: \\?\Volume{f6cd5a9b-04cf-11e1-b482-5cf3fc2b627f}\
Execution Context: System Provider
2D20436F64653A20535052414C
4C4F4330303030313137342D2043616C6C3A20535052414C4C4F4330
303030303739302D205049443A202030303031343838342D205449443A20203
0303030393234302D20434D443A2020433A5C57696E646F77735C5379737
4656D33325C737663686F73742E657865202D6B2073777072762D20557365723A204E616D653
A204E5420415554484F524954595C53595354454D2C205349443A532D312D352D313820 Binary
EventData
Event

Resolution:

Took me a while to figure it out. The error was happening to me because the Shadow Copy setting for C:\ was set low for VSS to operate. Increasing that disk space fixed it! Right click on C: drive, Properties, Shadow Copies, Highlight C:\ (Even though shadow copy might be disabled),click Settings –> Under Maximum size, change it to a bigger size., I set it to 300G to test it and that worked for me.

If Exchange Stores are located on a different drive, adjust that drive shadow copy setting.

If you find this article helpful, please send us a note to Mike@bostonIT.com so I can keep on adding quality hands-on articles.

Cisco AnyConnect VPN Client, Debian 5.0.3 and Firefox

/in /by

After you connect to VPN using Cisco Anyconnect on Debian 5.0.3, you are able to ping, traceroute and browse the Internet using Google Chrome but Mozilla Firefox, FTP, Dig, Telnet and Thunderbird won’t work

Resolution:

I had to troubleshoot this issue for a customer. During that I ran Wireshark, tcpdump and Microsoft Network Monitor traces on Debian, Ubuntu and Microsoft Windows hosts and came up with the conclusion and proposed solutions. At the end it was shown that IPv6 didn’t seem to be compatible with Cisco Anyconnect on Debian 5.0.3. The solution was to make the host machine totally rely on IPv4 for DNS resolution – in another word disable IPv6.

Before you disable IPv6 in Debian and to confirm the above finding, try to disable IPv6 in Firefox only and test. Here how you can do it:

Open Firefox and in the address bar type about:config, then confirm warning message, go to the line network.dns.disableIPv6 and change to true.

If that makes Firefox work, then go ahead and disable it in the operating system so other programs such as Thunderbird, telnet and FTP work. You can Google how to disable IPv6 in Debian and that should fix it for you. I didn’t want to write about that because you couldfind 100s of links about how to disable IPv6 in Linux/Debian.

One other method that worked for me while testing (but might not work for everybody) is to change MTU packet size of Ethernet 0 to 1200 from 1499. MTU 1200 is recommended by Cisco. Also IPv6 minimum MTU requirement is 1280 so by setting it to 1200, that might stop the use of IPv6

sudo /sbin/ifconfig eth0 mtu 1200

If you find this article helpful, please send us a note to Mike@bostonIT.com so I can keep on adding quality hands-on articles.