Tag Archives: troubleshooting

Microsoft Security Update for Windows 7 for x64-based Systems (KB2984976), titled RDP 8.0 update for restricted administration on Windows 7 or Windows Server 2008 R2 and released on October 14, 2014, as one of that month’s Patch Tuesday updates, appears to cause multiple restarts when applied during the Install Software Updates step within a System Center Configuration Manager Task Sequence. The second restart is not controlled by the Task Sequence engine and causes the engine to be unable to resume the Task Sequence when the computer comes back up after the second restart. The Task Sequence therefore fails to complete.

This behavior is a known issue with software updates that require multiple restarts, as documented in KB2894518, titled Task sequence fails in Configuration Manager if software updates require multiple restarts. Microsoft’s recommendation in KB2894518 is to deploy those updates that require multiple restarts outside of a Task Sequence.

If your patching procedure is to deploy Software Updates and other application updates during a maintenance Task Sequence and a hotfix is applied during the Install Software Updates step that causes multiple restarts, then the Task Sequence fails and potentially causes computers to go unpatched until they next run the Task Sequence. Therefore, it is important to quickly identify problematic hotfixes and deploy them outside of the Install Software Updates step in the Task Sequence.

When the Task Sequence fails due to a hotfix forcing multiple restarts, the TSManager component writes the following entries to smsts.log:

Failed to restore logs from cache. Execution history may be lost.	TSManager	10/17/2014 2:21:52 AM	2644 (0x0A54)
Task Sequence Manager executing as service main thread	TSManager	10/17/2014 2:21:52 AM	2644 (0x0A54)
Environment scope successfully created: Global\{51A016B6-F0DE-4752-B97C-54E6F386A912}	TSManager	10/17/2014 2:21:52 AM	2644 (0x0A54)
Environment scope successfully created: Global\{BA3A3900-CA6D-4ac1-8C28-5073AFC22B03}	TSManager	10/17/2014 2:21:52 AM	2644 (0x0A54)
Failed to locate the local data path. The files needed to resume the task sequence are missing.  This could be because the task sequence finished while in Windows PE.  Please check the largest available partition for SMSTSLog\smsts.log file for more information.
The system cannot find the file specified. (Error: 80070002; Source: Windows)	TSManager	10/17/2014 2:21:52 AM	2644 (0x0A54)
Task Sequence Manager could not initialize Task Sequence Environment. code 80070002	TSManager	10/17/2014 2:21:52 AM	2644 (0x0A54)
Task sequence execution failed with error code 80070002	TSManager	10/17/2014 2:21:52 AM	2644 (0x0A54)
Cleaning Up.	TSManager	10/17/2014 2:21:52 AM	2644 (0x0A54)

The exact language of the entries will vary based on the version of SCCM, with the above coming from SCCM 2012 R2.

A method of determining the culprit

Until I can figure out how to tell whether an update is going to schedule its own restart by observing some property of the hotfix itself, I’m left using a time-consuming process of elimination to determine which hotfix is responsible for the second restart during a Task Sequence.

In our environment, we run updates through SCCM on both our 64-bit Windows 7 workstations and our Windows Server 2008 R2 servers, and sometimes these two platforms receive different updates. When the Task Sequence failure occurs on only one platform, the first thing I do is to look at the hotfixes that are not applied to both platforms.

This month, fourteen hotfixes that require a reboot are installed on our workstations during the Task Sequence. Below are the entries written to the Setup log in Event Viewer on a Windows 7 workstation, where a second reboot occurred. Items followed by an asterisk “*” also appear in the Setup log in Event Viewer on a Windows Server 2008 R2 server, where a second reboot did not occur.

A reboot is necessary before package KB2984972 can be changed to the Installed state. *
A reboot is necessary before package KB2972100 can be changed to the Installed state. *
A reboot is necessary before package KB3001554 can be changed to the Installed state.
A reboot is necessary before package KB2977292 can be changed to the Installed state. *
A reboot is necessary before package KB3000061 can be changed to the Installed state. *
A reboot is necessary before package KB3000988 can be changed to the Installed state. *
A reboot is necessary before package KB3000869 can be changed to the Installed state. *
A reboot is necessary before package KB2949927 can be changed to the Installed state. *
A reboot is necessary before package KB2968294 can be changed to the Installed state. *
A reboot is necessary before package KB2952664 can be changed to the Installed state.
A reboot is necessary before package KB2984976 can be changed to the Installed state.
A reboot is necessary before package KB2987107 can be changed to the Installed state. *
A reboot is necessary before package KB2979570 can be changed to the Installed state. *
A reboot is necessary before package KB2994023 can be changed to the Installed state. *

The process of elimination begins, then, with those hotfixes that were installed on the workstations but not on the servers. Thankfully, in our environment this is just three hotfixes:

To work through the suspect hotfixes, I set up a VMware virtual machine that was patched up to September 9, 2014. I created a snapshot of this test computer and then ran the maintenance Task Sequence on it to confirm that the Task Sequence abruptly ended during the Install Software Updates step.

Next, I selected one of the hotfixes to eliminate. My first choice was KB2952664 because it has a long and troubled history and seems to not be necessary in our enterprise environment. Even this month, it seems to be problematic, as it was released on October 14, then re-released on October 16. See the InfoWorld article Windows 7 patch KB 2952664 fails with error 80242016 for more on the struggles Microsoft has had with this hotfix. I removed KB2952664 from the Software Update Group deployed to the test computer, reverted to my snapshot and ran the maintenance Task Sequence again. It still ended abruptly, so KB2952664 was not the culprit.

Without re-adding KB2952664 to the Software Update Group, I removed KB2984976 from the Software Update Group. The KB2984976 hotfix deals with RDP and shares files in common with the last hotfix to require multiple restarts, KB2965788 (which was the subject of a similar blog post that I wrote about this problem back in June). Microsoft has now included KB2965788 among those hotfixes listed in KB2894518.

I reverted to my snapshot again and with KB2984976 unavailable to my test computer, the Task Sequence was able to proceed through the Install Software Updates step, indicating that KB2984976 is responsible for the multiple restarts among the October Patch Tuesday updates.

If anyone has a better method of identifying whether a hotfix is going to schedule a restart that is not controlled by SCCM, please post a comment. I would love to find a faster way, possibly by looking at how these hotfixes schedule the restart differently than most updates.

The October 2014 Patch Tuesday updates did not go smoothly for Microsoft:
http://www.infoworld.com/article/2834535/security/four-more-botched-black-tuesday-patches-kb-3000061-kb-2984972-kb-2949927-and-kb-2995388.html
http://www.infoworld.com/article/2834930/security/microsoft-yanks-botched-patch-kb-2949927-re-issues-kb-2952664.html

Microsoft Security Update for Windows 7 for x64-based Systems (KB2965788), which was released on June 10, 2014, as one of that month’s Patch Tuesday updates and titled MS14-030: Description of the security update for Remote Desktop Security Release for Windows: June 10, 2014, appears to cause multiple restarts when applied during the Install Software Updates step within a System Center Configuration Manager Task Sequence. The second restart is not controlled by the Task Sequence engine and causes the engine to be unable to resume the Task Sequence when the computer comes back up after the second restart. The Task Sequence therefore fails to complete.

This behavior is a known issue with software updates that require multiple restarts, as documented in KB2894518, titled Task sequence fails in Configuration Manager if software updates require multiple restarts.

At my firm, we deploy Software Updates and other application updates during a maintenance Task Sequence. When the Task Sequence fails to complete after the Install Software Updates step, the TSManager component writes the following entries to smsts.log:

Failed to restore logs from cache. Execution history may be lost.
...
Failed to locate the local data path. The files needed to resume the task sequence are missing.  This could be because the task sequence finished while in Windows PE.  Please check the largest available partition for SMSTSLog\smsts.log file for more information.
The system cannot find the file specified. (Error: 80070002; Source: Windows)
Task Sequence Manager could not initialize Task Sequence Environment. code 80070002
Task sequence execution failed with error code 80070002
...
Error executing Task Sequence Manager service. Code 0x80070002
MP name must be set in an environment variable
Non fatal error 0x80004005 in sending task sequence execution status message to MP
Successfully finalized logs to SMS client log directory from C:\Windows\CCM\Logs

Microsoft’s recommendation in KB2894518 is to deploy updates that require multiple restarts outside of a Task Sequence. If you choose to deploy hotfix KB2965788 as a traditional package or an application using a required deployment, it can be downloaded from Security Update for Windows 7 for x64-based Systems (KB2965788).

I was running into a problem with the installation of Microsoft .NET Framework 4.5.2 during an SCCM 2012 SP1 build and capture Task Sequence, both in Windows 7 and Windows 8.1, wherein the installer was running but the log files were not being created.

I was using the Application model in SCCM and executing the offline installer executable with the command line “NDP452-KB2901907-x86-x64-AllOS-ENU.exe /q /norestart” through a VBScript wrapper script.

When the application failed during the task sequence, the error in smsts.log read, in part:

Execution status received: 4 (Application failed to install )
Installation failed.
...
Install application action failed: 'Microsoft .NET Framework 4.5.2'. Error Code 0x80004005
...
Install Static Applications failed, hr=0x80004005
Process completed with exit code 2147500037
!--------------------------------------------------------------------------------------------!
Failed to run the action: Microsoft .NET Framework 4.5.2. 
Unspecified error (Error: 80004005; Source: Windows)

If I ran my installation wrapper script from the ccmcache subdirectory while logged on as the local administrator account, the .NET Framework 4.5.2 installation ran successfully.

According to the MSDN page .NET Framework Deployment Guide for Administrators, log files are written to “%temp%\Microsoft .NET Framework 4.5*.txt” and “%temp%\Microsoft .NET Framework 4.5*.html”, but neither of these logs existed on my Windows 7 and Windows 8.1 systems after the installation failed.

However, a log file at C:\Windows\Temp\dd_NDP452-KB2901907-x86-x64-AllOS-ENU_decompression_log.txt caught my eye. The contents of this decompression log read:

[5/22/2014, 16:42:35] === Logging started: 2014/05/22 16:42:35 ===
[5/22/2014, 16:42:35] Executable: C:\Windows\ccmcache\1\NDP452-KB2901907-x86-x64-AllOS-ENU.exe v4.5.51209.34209
[5/22/2014, 16:42:35] --- logging level: standard ---
[5/22/2014, 16:42:35] Successfully bound to the ClusApi.dll
[5/22/2014, 16:42:35] Error 0x80070424: Failed to open the current cluster
[5/22/2014, 16:42:35] Cluster drive map: ''
[5/22/2014, 16:42:35] Considering drive: 'A:\'...
[5/22/2014, 16:42:36] Drive 'A:\' is rejected because of the unknown or unsuitable drive type
[5/22/2014, 16:42:36] Considering drive: 'C:\'...
[5/22/2014, 16:42:36] Considering drive: 'D:\'...
[5/22/2014, 16:42:36] Drive 'D:\' is rejected because of the unknown or unsuitable drive type
[5/22/2014, 16:42:36] Drive 'C:\' has been selected as the largest fixed drive
[5/22/2014, 16:42:36] Directory 'C:\26169df5670339bdf66775485ff857\' has been selected for file extraction
[5/22/2014, 16:42:36] Extracting files to: C:\26169df5670339bdf66775485ff857\
[5/22/2014, 16:42:36] Error 0x80004005: Failed to extract all files out of box container #0.
[5/22/2014, 16:42:36] Error 0x80004005: Failed to extract
[5/22/2014, 16:42:36] Exiting with result code: 0x80004005
[5/22/2014, 16:42:36] === Logging stopped: 2014/05/22 16:42:36 ===

I Googled the rather generic error messages from the decompression log a little bit and didn’t find anything helpful, although I did find some recommendations to 1) extract the file using 7-Zip and then run the setup.exe, which I did not want to do, or 2) use the traditional Package/Program method, which I considered. On the day that I was going to give up on the Application model and just create a Package, I did a little more Googling on installing .NET Framework 4.5 through an SCCM task sequence and found a thread on the Technet forums that describes the problem exactly and gives a working solution:

.NET 4.5.1 Install only works when running on a UI session
http://social.technet.microsoft.com/Forums/en-US/4808233e-1410-4305-a8d1-0e88f3a6fdc8/net-451-install-only-works-when-running-on-a-ui-session?forum=configmanagerapps

The resolution described in the thread is to edit the properties of the Deployment Type to enabled “Run installation and uninstall program as 32-bit process on 64-bit clients.” (This setting is found on the Programs tab.) I modified the Deployment Type properties to run the installer as a 32-bit process and indeed it did install successfully during the task sequence.

For the Detection Rule, I chose to use a registry setting must exist:

HKLM\SOFTWARE\Classes\Installer\Products\6414876250E69FF3395387C6C7F05BEB\ProductName = Microsoft .NET Framework 4.5.2

Using the registry to determine the installed version of .NET Framework is rather tricky, but for versions of .NET Framework 4.5 and later, see How to: Determine Which .NET Framework Versions Are Installed.

Best of luck.

P.S. I expect, but will have to validate, that this method of running the installer as a 32-bit process might help with other applications that fail to install during a task sequence but run successfully via Software Center or Application Catalog when a user is logged on.

This post is a collection of some of the more commonly used command line utilities when doing basic troubleshooting in a Windows domain environment.

To open a command window within a directory from Windows Explorer, hold the Shift key and right-click on the directory, then choose “Open command window here”.

cd

Displays the name of the current directory or changes the current folder.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ntcmds.mspx?mfr=true

Used within a command window to change the current active directory, allowing navigation through the computer’s mapped drives and their directory structures.

Usage:

cd
Displays the current directory path.

cd
Moves to the root of the current drive.

cd /d e:
Moves to the E: drive from another drive. It’s also possible to move to a different drive by typing only the drive letter followed by a colon, ex: D:

cd..
Moves to the parent directory of the current directory (move up one directory toward the root).

cd “People to sue next”
Moves from the current directory into the subdirectory named “People to sue next”. A handy trick is to just type the first few characters of the directory name, and then hit the tab key to auto-complete the rest of the directory name from the first alphabetical match found, and even wrap it in double quotes if it contains spaces. For example, the same command as above can by typed: cd peop <tab>

If the current directory contains multiple matches for the characters typed, hitting tab again will cycle to the next match.

The tab method can be used more than once, to chain together a series of directories. For example, to move to the C:\Users\Public\Documents directory from a command prompt at the root of C:, one can type: cd u <tab> p <tab> d <tab> <tab> <enter>

dir

Displays a list of a directory’s files and subdirectories.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/dir.mspx?mfr=true

Usage:

dir
Displays the directories and files in the current directory.

dir /s
Displays the directories and files in the current directory and all sub directories.

Dir can also be used to search for a file, and in many cases it works better than the Windows Explorer search.

dir c:\findme.txt /s
Displays a list of all instances of a file named “findme.txt” on the C: drive. It’s also possible to navigate to a location, such as the root of C:, and type: dir /s findme.txt to search that location and all subdirectories for a file named “findme.txt”.

Wildcards are allowed in the form of an asterisk. For example, type: dir c:\*.doc /s to search the C: drive for all files with a .doc or .docx extension (I’m not sure why it also locates .docx files, when there is no wildcard specified at the end of the extension, but it does).

Another command line utility for searching for files is where, but the syntax is slightly more complicated.

gpupdate

Refreshes local and Active Directory-based Group Policy settings, including security settings.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/refrgp.mspx?mfr=true

If you absolutely must reapply all settings, you can use the /force switch. After reading about the difference between gupdate and gpupdate /force, I now feel that gupdate is sufficient to reapply group policy nearly all of the time, and the /force switch shouldn’t automatically be used.

Usage:

gpupdate
Reapplies group policy.

gpresult

Displays Group Policy settings and Resultant Set of Policy (RSOP) for a user or a computer.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ntcmds.mspx?mfr=true

Usage:

gpresult /r
Displays RSoP summary data, which includes the last time group policy was applied, from which server group policy was applied, and the groups for which the current user is a member.

gpresult /h gpreport.html
Generates a report of the applied group policy settings and saves it in HTML format as a file named gpreport.html. When generating a report as a user that is not a local administrator, either supply a full path to a valid location for gpreport.html, or navigate to a location (like the Public Documents directory) before running the command, or else the utility may be unable to create the report due to insufficient rights to the current directory.

ipconfig

Displays all current TCP/IP network configuration values and refreshes Dynamic Host Configuration Protocol (DHCP) and Domain Name System (DNS) settings. Used without parameters, ipconfig displays the IP address, subnet mask, and default gateway for all adapters.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ipconfig.mspx?mfr=true

Usage:

ipconfig
Display the computer’s IP address and default gateway, for each network adapter.

ipconfig /all
Displays full TCP/IP information, including the MAC address, DHCP server, and DNS servers, for each network adapter.

net use

Connects a computer to or disconnects a computer from a shared resource, or displays information about computer connections. The command also controls persistent net connections. Used without parameters, net use retrieves a list of network connections.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/net_use.mspx?mfr=true

Usage:

net use
Lists all of the computer’s connections (mapped network drives).

net use e: \\ComputerName\ShareName
Maps the E: drive to the ShareName shared resource on the ComputerName computer. To map the local E: drive to the C: drive (which is a hidden share) of a remote machine named Loomer, type: net use e: \\loomer\c$

net use e: /delete
Removes the connection currently mapped to the local E: drive.

If you are connecting to a network share that your regular account does not have rights to access, you will be prompted for a username. You will need to also supply the domain, ex: domainusername

nslookup

Displays information that you can use to diagnose Domain Name System (DNS) infrastructure.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/nslookup.mspx?mfr=true

Usage:

nslookup <ipaddress or computername>
Queries the local computer’s default DNS name server for information on the specified IP address or computer name. Supply either piece of information and nslookup will return both pieces. It’s also possible to specify a particular DNS name server to be queried, which is useful when troubleshooting whether DNS is propagating/replicating correctly.

ping

Verifies IP-level connectivity to another TCP/IP computer by sending Internet Control Message Protocol (ICMP) Echo Request messages. The receipt of corresponding Echo Reply messages are displayed, along with round-trip times. Ping is the primary TCP/IP command used to troubleshoot connectivity, reachability, and name resolution.

You can use ping to test both the computer name and the IP address of the computer. If pinging the IP address is successful, but pinging the computer name is not, you might have a name resolution problem.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ping.mspx?mfr=true

Usage:

ping <ipaddress or computername>
Makes four attempts to contact the computer at the specified IP Address or with the specified computer name, and reports back whether the machine could be contacted and the time taken for the request to travel to the remote computer, be acknowledged, and the acknowledgement received by the local computer.

ping <ipaddress or computername> -t
Repeatedly attempts to contact the remote computer until interrupted by pressing Ctrl+Break or Ctrl+C. This is sometimes called a persistent ping.

systeminfo

Displays detailed configuration information about a computer and its operating system, including operating system configuration, security information, product ID, and hardware properties, such as RAM, disk space, and network cards.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/systeminfo.mspx?mfr=true

The systeminfo command also reveals installed hotfixes and some information about the computer that isn’t readily available in Device Manager or other MMC Snap-ins, such as the BIOS version.

Usage:

systeminfo
Displays information about the local computer.

systeminfo /s computername /u domainuser
Displays information about a remote computer named computername.

systeminfo /s computername | find “System Model:”
Retrieves information about a remote computer named computername, but pipes the output of systeminfo to the find command, which returns only the line containing the string “System Model:”. This output in the command window shows only “System Model:” followed by the model of the remote computer.

The systeminfo report can be sent to a text file, ex: systeminfo > systeminforeport.txt

Bonus commands

getmac

Returns the media access control (MAC) address and list of network protocols associated with each address for all network cards in each computer, either locally or across a network.
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/getmac.mspx?mfr=true

Usage:

getmac /v
Shows MAC addresses for the local computer.

getmac /s computername /u domainusername /v
Shows MAC addresses for a remote computer named computername while authenticating as a different user.

(Need to test this.)

msg

Sends a message to a user (this may be turned off in many environments). Run msg /? for usage information.

During the development of our Windows 7 image with Office 2010, we began seeing a problem around our users’ Outlook 2010 profiles on the pre-production builds. On occasion, after logging into a machine for the first time, our users would be prompted to choose an Outlook profile upon the first launch of Outlook. Every time the Choose Profile dialog box was presented, it had only a single option in the profile name menu, and that option was always “BACKUP OF Outlook”, where Outlook was our customized profile as configured in a .PRF and applied via the Office OCT.

Background

We were not using .PST files and we were not using Windows Roaming Profiles, but we were using Group Policy logon and logoff scripts to roam certain portions of our user profiles, including the entire registry key at [HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles]. With 20/20 hindsight, it’s clear that this unwanted behavior was not happening when we logged into a machine for the first time as a brand-new user (ie, while also preventing the logon script from merging the Profiles key from another machine into HKCU before Outlook was opened), but before a pattern had emerged, we considered the problem to be intermittent.

We were building machines using System Center 2012 Configuration Manager and using the Microsoft Office Customization Tool for configuring our .MSP and .PRF files. We were familiar with Active Setup and recognized that Outlook was doing a similar first-run process to set up a profile for what it thought was a new user. When it discovered that an Outlook profile already existed, it created a new profile named “BACKUP OF Outlook” and offered the user a Choose Profile dialog box with this profile as the only choice, presumably because “BACKUP OF Outlook” was not yet set to be the default profile.

Observed symptoms

When the logon script had roamed a user profile from another machine by importing the Profiles key, and before Outlook was launched for the first time, our Profiles key looked similar to this (the snippet has had the juicy bits removed):

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles]
"DefaultProfile"="Outlook"

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Outlook]

After Outlook had been launched and the Choose Profile dialog box had been presented, the registry looked similar to this:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\DeletedProfiles]

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\DeletedProfiles\Outlook]

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles]

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\Outlook]

[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\BACKUP OF Outlook]

The changes we noted were that a new profile named BACKUP OF Outlook had been created, a new DeletedProfiles key had been created, our desired profile had been flagged for deletion via a subkey under DeletedProfiles, and the DefaultProfile string value under the Profiles key that had been pointing to our profile had been deleted.

If we launched Outlook but cancelled out of the Choose Profile dialog box, closed Outlook, and put the Profiles key back to the state before Outlook was launched, we could then relaunch Outlook without issue. It had no objection to using the roamed profile on the second launch or any subsequent launches. The problem might arise if this user roamed the Profile key to yet another machine, but we had not yet identified a pattern or means of reproducing the problem.

Complicating factors

This was happening during a time of rapid development, when the Office OCT was being changed frequently, and machines were hitting the floor with different Office builds. The users logging into these machines were not always aware of all of the changes between builds, and in many cases we were not roaming the user’s profile in an attempt to get a ‘clean’ test of the new build. These factors contributed to the difficulty in establishing a pattern or recognizing commonalities.

A decent amount of time was spent researching problems with Outlook profiles in general, and this research turned up a few forum threads (http://social.technet.microsoft.com/Forums/en-US/outlook/thread/f05c057b-d226-4b7f-bf0d-0406db5acdb1/) that indicated the problem stemmed from the .PRF file. (We also found some mentions of an “undocumented” property named BackupProfile in the .PRF: http://www.slipstick.com/outlook/tips-for-using-outlook-prf-files-to-configure-profiles/ and http://www.slipstick.com/outlook/config/understanding-microsoft-outlook-profile-file-prf/.) We experimented briefly with making changes to the .PRF as hinted at in these threads, but felt that such trial-and-error experimentation was not the best use of our time.

The epiphany

A tip passed along by one of our Kraft Kennedy consultants lead to the break-through.

When 32-bit Office is installed with an .MSP generated by the OCT, a GUID-named key is created under [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Office\14.0\User Settings] that contains a value named Count with a data of 1. For example:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Office\14.0\User Settings\{75BB133B-F5DD-423C-8321-3BD0B50322A5}]
"Count"="1"

Much like Active Setup, when Outlook launches, it looks for a corresponding key in [HKEY_CURRENT_USER\Software\Microsoft\Office\14.0\User Settings], also with a
value named Count with a data of 1. If the matching key is not found, Outlook does its first-run process, applies the .PRF, and then writes the GUID-named key to HKCU so that the first-run behavior doesn’t happen on that machine again.

In our case, each time we made a change to the OCT and cooked up a new OSD image in SCCM, the GUID-named key under HKLM changed. When a user from an old machine roamed to one of these new machines (or when a user initially on a new machine roamed to an older one), the GUID-named keys did not match and the first-run behavior fired off.

The resolution

Now that we better understood what was happening, we could evaluate a few ways to handle the situation. One way was to detect whether the user already had a default profile, and then add the current machine’s HKLM GUID key to her HKCU. Another way was to try to get a better handle on the .PRF and configure it to not create the backup, even when the first-run behavior was triggered. The latter seemed preferable, because we weren’t sure that avoiding the first-run behavior entirely was desirable. We suspected that there might be an advantage to allowing Office’s first-run process to play out, for example, if future changes to the OCT were made that needed to be added to the user’s environment.

After some communication with Microsoft, we made two changes to our .PRF that suppressed the creation of the BACKUP OF profile. The first change was to add BackupProfile=False to the Section 1, General area. The second change was to use UniqueService=Yes in the Section 4, Service1 area.

The corrected .PRF, in part, looks like this:

;Automatically generated PRF file from the Microsoft Office Customization and Installation Wizard

; **************************************************************
; Section 1 - Profile Defaults
; **************************************************************

[General]
Custom=1
ProfileName=Outlook
DefaultProfile=Yes
OverwriteProfile=Yes
ModifyDefaultProfileIfPresent=False
BackupProfile=False

...

;***************************************************************
; Section 4 - Default values for each service.
;***************************************************************

[Service1]
OverwriteExistingService=No
UniqueService=Yes
MailboxName=%UserName%

Some final thoughts

I would note that we are using BackupProfile=False, while many of the forum threads on the subject (incorrectly?) reference the property value as No, as in BackupProfile=No.

This “undocumented” BackupProfile property is actually quite well-documented, and even highlighted as important, in the TechNet article at http://technet.microsoft.com/en-us/library/cc179062.aspx.

A TechNet blog post from August, 2010, at http://blogs.technet.com/b/odsupport/archive/2010/08/06/multiple-exchange-accounts-created-in-outlook-2010-with-existing-outlook-profiles-after-upgrading-from-an-earlier-office-version-using-a-custom-msp.aspx is dedicated to a problem with multiple Exchange accounts that is resolved by making the same two changes to the .PRF. The blog post helpfully points out that the manually edited .PRF file must exist in the same location as the .PRF originally used with the OCT:

NOTE: If Outlook/Exchange settings in the MSP file need to be edited in the future, the custom PRF file created to work around this issue must be copied to the same location as it was when originally imported into the OCT (i.e., C:\Custom14.PRF) on the machine that you’re running the Office Customization Tool on when modifying the MSP file.
http://blogs.technet.com/b/odsupport/archive/2010/08/06/multiple-exchange-accounts-created-in-outlook-2010-with-existing-outlook-profiles-after-upgrading-from-an-earlier-office-version-using-a-custom-msp.aspx

The work yet to be done

Without further testing, it remains unclear whether the BackupProfile=False instruction, possibly in combination with other options in the .PRF, causes settings in the .PRF to be merged into the existing Outlook profile, or whether the presence of an existing profile means Outlook just doesn’t do anything with the .PRF.

Customize Outlook profiles by using an Outlook Profile (PRF) file
An existing profile can be either overwritten or updated when a new .prf file is executed. Several settings control how the new settings are applied:

The OverwriteProfile setting can be set to Yes, Append, or No. To update existing profiles, set the value to Append. This preserves the existing profile and updates the sections that have been changed. To overwrite existing profiles with a new profile, set the value to Yes. To prevent overwriting an existing profile, set the value to No.

The ModifyDefaultProfileIfPresent setting can be set to True or False. When set to True, Outlook will modify the default profile even if the new and existing profile names are different.
http://technet.microsoft.com/en-us/library/cc179062.aspx

I would also like to better understand how the GUID-named key gets its name. The TechNet article on the Office Customization Tool in Office 2010 seems to possibly allude to the GUID being a timestamp.

Every time that you save a customization file in the OCT, the tool updates the customization file’s sequencing number with the current computer date and time stamp and generates a new update globally unique identifier (GUID).
http://technet.microsoft.com/en-us/library/cc179097.aspx

As it seems to be the subject of some debate, I want to point out that Microsoft supports applying Setup customization .msp files to existing installations of Office 2010. I suspect, but have not attempted to confirm, that this would generate additional GUID-named keys.

Sure, Wbemtest.exe is pretty neat, and it gets points for being built-in. http://blogs.technet.com/b/chad/archive/2012/03/08/tip-45-wbemtest-the-underappreciated-tool.aspx

Microsoft's Wbemtest.exe displaying a WMI query

Microsoft’s Wbemtest.exe displaying a WMI query

But when it comes to building WMI queries for use in scripting languages, Microsoft’s WMI Code Creator is even slicker. https://technet.microsoft.com/en-us/magazine/2006.01.utilityspotlight.aspx

The WMI Code Creator tool allows you to generate VBScript, C#, and VB .NET code that uses WMI to complete a management task such as querying for management data, executing a method from a WMI class, or receiving event notifications using WMI.
https://technet.microsoft.com/en-us/scriptcenter/dd823314.aspx

Microsoft's WMI Code Creator displaying a WMI query and VBScript

Microsoft’s WMI Code Creator displaying a WMI query and VBScript (click for full-size)

The tool also allows you to browse through the available WMI namespaces and classes on the local computer to find their descriptions, properties, methods, and qualifiers.
https://www.microsoft.com/en-us/download/details.aspx?id=8572

The WMI Code Creator utility can be downloaded from microsoft.com at WMI Code Creator v1.0.

I want the SparkFun VC830L multimeter, to replace a $10, no-name multimeter that I’ve had for a few years (and it still works, but the probes are falling apart).

SparkFun VC830L multimeter

SparkFun VC830L multimeter

This is for all those starving students that need to buy their first good, low-cost multimeter. This meter has really impressive overall feel for being so low-cost. We’ve played with a lot of cheap-o DMMs and found that this unit’s function selector has a great, solid, clicking feel to it. This unit has good continuity (rare for low-cost units), and decent probes.
http://www.sparkfun.com/products/9141

And while I’m at it, I’d also like a GE 50957 GFCI Tester. I’m weird about plugging a multimeter into an outlet in order to test for incorrect wiring.

GE 50957 GFCI Tester

GE 50957 GFCI Tester

I’ve been a fan of the simple and effective Windows 7 USB/DVD Download Tool for quite awhile, and have often used it to create a bootable USB flash drive for installing Windows 7.

But I recently ran into a problem with a flash drive after connecting it to my Xbox 360 and using it to move my profile. The Xbox 360 must have made some change to the MBR on the flash drive that the WUDT didn’t like, because it was unable to format the drive.

The WUDT would begin to format the drive, then report:

We were unable to copy your files. Please check your USB device and the selected ISO file and try again.

Windows 7 had no problems formatting the drive, but something was obviously missing from the process.

A quick search in Google turned up the solution. The formatting done by Windows 7 or the WUDT wasn’t cleaning the MBR and partition table.

To thoroughly format the drive so that it can be used by the WUDT, open an elevated command prompt and enter the following commands, using the drive number of the USB drive reported in list disk for the value of select disk #.

diskpart
list disk
select disk #
clean
create partition primary
select partition 1
active
format quick fs=fat32
assign
exit

For the curious, here’s a more detailed explanation of the clean command:

Removes any and all partition or volume formatting from the disk with focus. On master boot record (MBR) disks, only the MBR partitioning information and hidden sector information are overwritten. On GUID partition table (GPT) disks, the GPT partitioning information, including the Protective MBR, is overwritten; there is no hidden sector information.
http://technet.microsoft.com/en-us/library/cc766465(v=ws.10).aspx

I would expect that the MBR and partition table would need to be cleaned after formatting a drive for booting Mac OS X, too.

Credit: http://4sysops.com/archives/windows-7-usbdvd-download-tool-wudt-is-unable-to-copy-files/

I recently reinstalled Windows 7 and then VM Workstation 7 on a machine that I’d been using as a VMware Workstation host running an XP virtual machine. The VM is named “XP-Office2007” and it resides on a separate physical hard drive in the machine. Since the reinstall, upon clicking the “Power on this virtual machine” link, VMware returns a cryptic “Internal error.” and fails to start the VM.

VMware Workstation - Internal error.

VMware Workstation - Internal error.

The vmware log in %temp% had the following entries:

Mar 01 09:28:46.750: vmui| CVMUIStatusVM::OnPowerVM clicked
Mar 01 09:28:46.920: vmui-1176| /vm/#6f7088f493f32788/: VMHSVMCbPower: Setting state of VM to powerOn with option hard
Mar 01 09:28:46.921: vmui-1176| VMHSGetVMX failed: Empty vmxFilePath
Mar 01 09:28:46.921: vmui-1176| VMHSLaunchVM failed: VMDB failure
Mar 01 09:28:46.957: vmui| Internal VMDB error: VMDB failure
Mar 01 09:28:46.957: vmui| Internal error.
Mar 01 09:28:46.957: vmui| VMDlg::ShowDialog: Internal error.

Of course, I immediately started Googling these errors and failure messages, but found very little. At least one page indicated that a similar sounding problem wasn’t resolved by re-installing VMware. I like to think I’m pretty decent with VMware, so I decided I’d roll up my sleeves, dig into the VM, and try to figure it out.

Trying to get lightning to strike twice, I tried my old standby fix of editing the name of the hard drive in the *.vmx file. Instead of pointing the IDE drive to the “XP-Office2007-000001.vmdk” file, I moved that file out of my VM directory and then changed the ide0:0.fileName value to point to the older *.vmdk file, named “XP-Office2007.vmdk”. The IDE lines now looked like this:

ide0:0.present = "TRUE"
ide0:0.fileName = "XP-Office2007.vmdk"
ide1:0.present = "TRUE"
ide1:0.fileName = "E:"

When the machine was powered on, it still failed, but with a different and more descriptive error:

VMware Workstation - Unable to open file

VMware Workstation - Unable to open file

VMware Workstation
Unable to open file “D:\VMware\Virtual Machines\XP-Office2007\XP-Office2007.vmdk”: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot.

The more verbose message from the vmware log in %temp% read:

Mar 01 10:31:22.814: vmui| CVMUIStatusVM::OnPowerVM clicked
Mar 01 10:31:23.016: vmui-3992| SNAPSHOT: SnapshotDiskTreeAddFromSnapshot: Trying to add snapshot XP-Office2007-Snapshot2.vmsn to disk D:\VMware\Virtual Machines\XP-Office2007\XP-Office2007.vmdk which already has snapshot currentState.
Mar 01 10:31:23.016: vmui-3992| Cannot open D:\VMware\Virtual Machines\XP-Office2007\XP-Office2007.vmdk of type disk: One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot.
Mar 01 10:31:23.016: vmui-3992| Cmd /vm/#6f7088f493f32788/cmd/##94/op/checkMissingFiles/ failed: Unable to open file "D:\VMware\Virtual Machines\XP-Office2007\XP-Office2007.vmdk": One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot.
Mar 01 10:31:23.016: vmui| Unable to open file "D:\VMware\Virtual Machines\XP-Office2007\XP-Office2007.vmdk": One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot.
Mar 01 10:31:23.016: vmui| VMDlg::ShowDialog: Unable to open file "D:\VMware\Virtual Machines\XP-Office2007\XP-Office2007.vmdk": One of the disks in this virtual machine is already in use by a virtual machine or by a snapshot.

I copied only the *.vmx and *.vmdk files into a new folder and tried to launch the *vmx as a new VM, but got the Internal error message again. This was something of a relief, as it suggested that the problem wasn’t with the VM itself.

I tried to create a new, empty virtual machine using the New Virtual Machine Wizard, but the process didn’t complete. After setting up the basic machine, I clicked the Finish button in the wizard, but the window did not close. The *.vmx and *.vmdk files were created, however. When I closed the window with the red X and tried to power on the VM, I got the same Internal error message.

I found a VMware community article suggesting that I needed to run Windows Updates on the host. This seemed logical, as perhaps the old host had more recent updates, but I found that the current host was up to date.

As I was running out of ideas, I decided that maybe re-installing Workstation was worth a shot, so I first checked in SCCM Software Center and found it had a Failed status. I then looked in Windows Programs and Features and found it wasn’t listed at all.

I reinstalled Workstation from Software Center and lo and behold, the VMs powered up normally. If I’d just kept my self-confidence in check, I would probably have gotten around to verifying the install much sooner.

7:21 PM 2/26/2012

I recently ran the spider at www.xml-sitemaps.com against www.ardamis.com and it returned a list of URLs that included a few pages with some suspicious-looking parameters. This is the second time I’ve come across these URLs, so I decided to document what was going on. The first time, I just cleared the cache, spidered the site to preload the cache, and confirmed that the spider didn’t encounter the pages. And then I forgot all about it. But now I’m mad.

Normally, a URL list for a WordPress site includes the various pages of the site, like so:

//ardamis.com/
//ardamis.com/page/2/
//ardamis.com/page/3/

But in the suspicious URL list, there are additional URLs for the pages directly off of the site’s root.

//ardamis.com/
//ardamis.com/?option=com_google&controller=..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F%2Fproc%2Fself%2Fenviron%0000
//ardamis.com/page/2/
//ardamis.com/page/2/?option=com_google&controller=..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F%2Fproc%2Fself%2Fenviron%0000
//ardamis.com/page/3/
//ardamis.com/page/3/?option=com_google&controller=..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F..%2F%2F%2Fproc%2Fself%2Fenviron%0000

This occurs only for the pagination of the main site’s pages. I did not find URLs containing the parameter ?option=com_google&controller= for any pages that exist under a category or tag, but that also use the /page/2/ convention.

The parameter is the urlencoded version of the text:

?option=com_google&controller=..//..//..//..//..//..//..//..///proc/self/environ00

Exploration

I compared the source code of the pages at the clean URLs vs that of the pages at the bad URLs and found that there was a difference in the pagination code generated by the WP-Paginate plugin.

The good pages had normal-looking pagination links.

<div class="navigation">
<ol class="wp-paginate">
<li><span class="title">Navigation:</span></li>
<li><a href="//ardamis.com/page/2/" class="prev">&laquo;</a></li>
<li><a href='//ardamis.com/' title='1' class='page'>1</a></li>
<li><a href='//ardamis.com/page/2/' title='2' class='page'>2</a></li>
<li><span class='page current'>3</span></li>
<li><a href='//ardamis.com/page/4/' title='4' class='page'>4</a></li>
<li><a href='//ardamis.com/page/5/' title='5' class='page'>5</a></li>
<li><a href='//ardamis.com/page/6/' title='6' class='page'>6</a></li>
<li><a href='//ardamis.com/page/7/' title='7' class='page'>7</a></li>
<li><span class='gap'>...</span></li>
<li><a href='//ardamis.com/page/17/' title='17' class='page'>17</a></li>
<li><a href="//ardamis.com/page/4/" class="next">&raquo;</a></li>
</ol>
</div>    

The bad pages had the suspicious URLs, but were otherwise identical. Other than the URLs in the navigation, there was nothing alarming about the HTML on the bad pages.

I downloaded the entire site and ran a malware scan against the files, which turned up nothing. I also did some full-text searching of the files for the usual base64 decode eval type stuff, but nothing was found. I searched through the tables in my database, but didn’t see any instances of com_google or proc or environ that I could connect to the suspicious URLs.

Google it

Google has turned up a few good links about this problem, including:

  1. http://www.exploitsdownload.com/search/com_/36 – AntiSecurity/Joomla Component Contact Us Google Map com_google Local File Inclusion Vulnerability
  2. http://forums.oscommerce.com/topic/369813-silly-hacker/ – “On a poorly-secured LAMP stack, that would read out your server’s environment variables. That is one step in a process that would grant the hacker root access to your box. Be thankful it’s not working. Hacker is a bad term for this. This is more on the Script Kiddie level.”

    The poster also provided a few lines of code for blocking these URLs in an .htaccess file.

    # Block another hacker
    RewriteCond %{QUERY_STRING} ^(.*)/self/(.*)$ [NC]
    RewriteRule ^.* - [F]
    
  3. http://forums.oscommerce.com/topic/369813-silly-hacker/ – “This was trying for Local File Inclusion vulnerabilities via the Joomla/Mambo script.”
  4. http://core.trac.wordpress.org/ticket/14556 – a bug ticket submitted to WordPress over a year earlier identifying a security hole if the function that generates the pagination isn’t wrapped in a url_esc function that sanitizes the URL. WP-Paginate’s author submits a comment to the thread, and the plugin does use url_esc.

So, what would evidence of an old Joomla exploit be doing on my WordPress site? And what is happening within the WP-Paginate plugin to cause these parameters to appear?

Plugins

It seemed prudent to take a closer look at two of the plugins used on the site.

Ardamis uses the WP-Paginate plugin. The business of generating the /page/2/, /page/3/ URLs is a native WordPress function, so it’s strange to see how those URLs become subject to some sort of injection by way of the WP-Paginate plugin. I tried passing a nonsense parameter in a URL (//ardamis.com/page/3/?foobar) and confirmed that the navigation links created by WP-Paginate contained that ?foobar parameter within each link. This happens on category pages, too. This behavior of adding any parameters passed in the URL to the links it is writing into the page, even if they are urlencoded, is certainly unsettling.

The site also uses the WP Super Cache plugin. While this plugin seems to have been acting up lately, in that it’s not reliably preloading the cache, I can’t make a connection between it and the problem. I also downloaded the cache folder and didn’t see cached copies of these URLs. I turned off caching in WP Super Cache but left the plugin activated, cleared the cache, and then sent the spider against the site again. This time, the URL list didn’t contain any of the bad URLs. Otherwise, the lists were identical. I re-enabled the plugin, attempted to preload the cache (it got through about 70 pages and then stopped), and then ran a few spiders against the site to finish up the preloading. I generated another URL list and the bad URLs didn’t appear in it, either.

A simple fix for the WP-Paginate behavior

The unwanted behavior of the WP-Paginate plugin can be corrected by changing a few lines of code to strip off the GET parameters from the URL. The lines to be changed all reference the function get_pagenum_link. I’m wrapping that function in the string tokenizing function strtok to strip the question mark and everything that follows.

The relevant snippets of the plugin are below.

			
$prevlink = ($this->type === 'posts')
? esc_url(strtok(get_pagenum_link($page - 1), '?'))
: get_comments_pagenum_link($page - 1);
$nextlink = ($this->type === 'posts')
? esc_url(strtok(get_pagenum_link($page + 1), '?'))
: get_comments_pagenum_link($page + 1);
			
function paginate_loop($start, $max, $page = 0) {
    $output = "";
    for ($i = $start; $i <= $max; $i++) {
        $p = ($this->type === 'posts') ? esc_url(strtok(get_pagenum_link($i), '?')) : get_comments_pagenum_link($i);
        $output .= ($page == intval($i))
        ? "<li><span class='page current'>$i</span></li>"
        : "<li><a href='$p' title='$i' class='page'>$i</a></li>";
    }
    return $output;
}

Once these changes are made, WP-Paginate will no longer insert any passed GET parameters into the links it’s writing into that page.

Bandaid

The change to the WP-Paginate plugin is what we tend to call a bandaid – it doesn’t fix the problem, it just suppresses the symptom.

I’ve found that once the site picks up the bad URLs, they can be temporarily cleaned by clearing the cache and then using a spider to recreate it. The only thing left to do is determine where they are coming from in the first place.

The facts

Let’s pause to review the facts.

  1. The http://www.xml-sitemaps.com spider sent against //ardamis.com discovers pages with odd parameters that shouldn’t be naturally occurring on the pages
  2. The behavior of the WP-Paginate plugin is to accept any parameters passed and tack them onto the URLs it is generating
  3. Deleting the cached pages created by WP Super Cache and respidering produces a clean list – the bad URLs are absent

So how is the spider finding pages with these bad URLs? How are they first getting added to a page on the site? It would seem likely that they are originating only on the home page, and the absence of the parameters on other pages that use pagination seems to support that theory.

An unsatisfying ending

Well, the day is over. I’ve added my updated WP-Paginate plugin to the site, so hopefully Ardamis has seen the last of the problem, but I’m deeply unsatisfied that I haven’t been able to get to the root cause. I’ve scoured the site and the database, and I can’t find any evidence of the URLs anywhere. If the bad URLs come back again, I’ll not be so quick to clean up the damage, and will instead try to preserve it long enough to make a determination as to their origin.

Update 07 April 2012: It’s happened again. When I spider the site, two pages have the com_google URL. These page have the code appended to the end of the URL created by the WordPress function cancel_comment_reply_link(). This function generates the anchor link in the comments area with an ID of cancel-comment-reply-link. This time, though, I see the hijacked URL used in the link even when I visit the clean URL of the page.

This code is somehow getting onto the site in such a way that it only shows up in the WP Super Cache’d pages. Clearing the cache and revisiting the page returns a clean page. My suspicion is that someone is visiting my pages with the com_google code as part of the URL. WordPress puts the code into a self-referencing link in the comment area. WP Super Cache then updates the cache with this page. I don’t think WordPress can help but work this way with nested comments, but WP Super Cache should know better than to create a cached page from anything but the content from the server.

In the end, because I wasn’t using nested comments to begin with, I chose to remove the block of code that was inserting the link from my theme’s comments.php file.

    <div class="cancel_comment_reply">
        <small><?php cancel_comment_reply_link(); ?></small>
    </div>

I expect that this will be the last time I find this type of exploit on ardamis.com, as I don’t think there is any other mechanism that will echo out on the page the contents of a parameter passed in the URL.