Hack 80 Beginners Guide to Enterprise Patch Management


managing the patch-management cycle effectively in a large enterprise
environment, written by an expert on the subject.One of the most heated and wildly debated subjects in many
organizations today is the subject of desktop security. When large IT
organizations have an enterprise product like Systems Management
Server (SMS) deployed, security teams usually push to the desktop
teams the task of ensuring that the latest security updates released
by Microsoft are installed. This means that the responsibility of
ensuring that a high percentage of clients in the environment are
patched rests on the shoulders of desktop support personnel members
or SMS team members. In order to better distribute this
responsibility, it's best to understand the
functionality that can be extended in the following steps:Identify vulnerable systemsAssess the business impact of patchingPackage patches for distributionTest patchesEvaluate successes and failuresFinish up
Before we discuss each of these steps in detail, if you are currently
looking to evaluate which tools would be best for your organizations
patch-management strategies, take a look at these helpful links from
Microsoft's two heaviest hitters:Patch Management Using Microsoft Software Update Services
http://www.microsoft.com/technet/treeview/default.asp?url=/technet/itsolutions/msm/swdist/pmsusog.asp
Patch Management Using Microsoft Systems Management Server
http://www.microsoft.com/technet/treeview/default.asp?url=/technet/itsolutions/msm/swdist/pmsmsog.asp
Identifying Vulnerable Systems
SMS stands out from SUS the most in its ability to report on the current
client state. In the past, SMS_DEF.MOF updates
accomplished this by pulling data from the Registry in
HKLM\SOFTWARE\Microsoft\Windows
NT\CurrentVersion\HotFix and basing distributions
on that data. However, with the advent of the SUS Feature Pack for
SMS (SUSFP), this is no longer necessary. A more robust security
hotfix and reporting mechanism can now be added to your
infrastructure. Installed and missing updates are reported through
normal SMS inventory, so collections and queries can be based on of
this information. Microsoft also supplies a web-based reporting
package with built-in reports to give high-level overviews of an
organization's patch state. The job of an IT
administrator to manage the environment will be significantly easier
if she is able to view which updates are already installed and which
are a priority to be deployed.
Assessing the Business Impact of Patching
The business impact of patching
is mostly comprised of developing patch
distribution schedules and policies. Each company's
needs are different and patch distribution should conform to those
specific needs. Many desktop IT divisions let mandates from their
security teams determine when and how patches should be deployed;
generally, this causes more harm than good. Do your customers have
schedules to meet as well? Shipping dates for when the product must
be out the door? What if you send a package with 10 critical system
updates to 100 workstations? Problems might occur through no fault of
your own, no matter how much testing is done prior to deployment.
Optimal success is gained by talking to your customer base to
determine the best dates for distribution. If a major product order
is being shipped on the 15th, you should probably wait until the 20th
to send out updates.What, then, guarantees the least amount of interruption to users,
while still ensuring success? Most of the time, this is encapsulated
in your company's workstation reboot policy. If most
IT administrators had their druthers, every workstation would get
restarted on a daily basis or the administrators would be allowed to
force system restarts whenever they want to during distributions. But
in the real world, especially for engineering or R&D-driven
companies, this is not a reality. Even if you are using the SUS
Feature Pack (SUSFP) to give your user base the ability to install
their updates early, it is still difficult to manage workstations
that might not get logged onto for weeks at a time or workstations
that run test simulations and cannot get restarted even while a
distribution is happening. There is no easy answer to this, and many
different internal solutions have been developed by various
organizations. If you are faced with this dilemma and are not able to
use the built-in tools (such as the SUSFP), the best option is to
speak with other members of the IT community in forums, newsgroups,
or mailing lists and learn which methods they use to overcome this
problem. However, it is crucial to keep two ideas in mind: the impact
to your user base and whether the solution still ensures successful
installation.
Packaging Patches for Distribution
If your organization is
already
using the SMS SUS Feature Pack's patch-installation
agent (i.e., the Distribute Software Updates Wizard functionality)
for distribution, you can pretty much ignore this section because
everything you need is already done for you. Microsoft has developed
a common template in the SUSFP that consistently applies patches in a
safe fashion, so the SMS administrator no longer needs to perform
constant scripting. Most companies using SMS that have not made the
transition to SUSFP to distribute patches haven't
done so either because their organizations have either specific
user-base needs (such as workstations in labs that might go long
periods without being looked at) or a scattered hierarchy of sites
and multiple administrators that would need to replicate the same
settings repeatedly (this scenario introduces an increased
possibility for human error, along with other issues).If you decide to write your own package to suit your
company's needs, make sure you perform the following
major steps in your code repeatedly for each patch:Determine if the patch is already present by checking for a Registry
key's existence, file version, and so on.
You'll spend a lot of time reading through Microsoft
security bulletins to figure out which data to look for.Detect which operating system your package is running on and install
the correct version of that patch for each bulletin.Verify successful installation by again checking for a Registry
key's existence, file version, and so on. But this
time you will need to code in failure messages (MIFs for SMS) and
logging so that you can troubleshoot issues that might arise later.QChain the patches. Many people forget to do this step.
This process is a simplified flow of what needs to be done for a
solid patch-distribution package. It might look easy at first glance,
but when you are dealing with over 15 hotfixes, the package size can
grow quite considerably, and the time the administrator spends coding
and working out bugs greatly increases as well. In the past, packages
that do just these simple tasks have been known to grow to a few
thousand lines of code!It's easy to see why using the SUSFP Patch
Installation agent would make an SMS administrators job easier, since
these steps are already done. Proper hotfix command-line switches are
really all you need to research.
Testing Patches
Proper testing procedures should
be
at the foundation of every software distribution, regardless of
weather they are hotfixes. Your company's client
base might have any number of different configurations that could
effect distribution. I won't discuss package testing
in detail here, because entire books can be written on the subject,
but I will add a couple of notes.First, test the patch in its raw installer form from Microsoft to
eliminate any chance of causing problems on the system. Next, when
testing configurations on workstations that are designated for
testing, do the majority of your tests via SMS once you are confident
that the package functions correctly. Many problems can be uncovered
when distributing via SMS. Because the majority of your enterprise
will execute the installation in this method, the majority of your
testing should mirror that use.Also, note that there is no substitute for beta
testers among your customer base. Not only will you get more feedback
if a problem arises, but they are also using the tools that could be
affected. Finally, base distributions on testing;
don't base testing on distributions
Evaluating Successes and Failures
Verifying that the client base is up-to-date after distribution might
be the most important phase in this entire process. Managing the
patch state in your enterprise will never be finished as long as
there is person out there who is smarter than the Microsoft
developers, which is why companies need to patch in the first place.
Unfortunately, many organizations distribute an update and never look
at reports to make sure that a high-enough percentage of machines are
updated. If your infrastructure is functioning properly, the
reporting methods you used earlier to identify vulnerable systems
(such as the SUSFP) should show that the patch-installation state is
higher after distribution. If this is not the case (if installations
are failing but notifications of the failure are not sent back to the
IT administrator), you will need to look into either your reporting
mechanism or your packaging technique.
Finishing Up
Thoroughness and attention to detail are the most important aspects
to managing a company's patch state. Although there
are many delivery machines (such as SMS) that can be used to apply
patches to your workstations, the same basic premises should be
followed throughout the entire process. Even in this heightened time
of hacking, viruses, and worms, a solid patch-management process and
attentive IT administration can avoid almost any vulnerability by
actively keeping workstations' patch state updated.Do some research on Microsoft patch release dates and virus/worm
release dates. You'll find that in almost every
instance a patch has been released far ahead of time. When
administrators fail to apply these updates, frantic patching is often
required when an attack happens. Do you and your customers a favor:
avoid these emergencies by having a stable enterprise
patch-management solution in place.
See Also
Here is a brief list of useful links on different aspects of
enterprise patch management:Security bulletin email notifications (http://www.microsoft.com/security/security_bulletins/decision.asp)White paper on improving patch management (http://www.microsoft.com/security/whitepapers/patch_management.asp)Security policy, assessment, and vulnerability analysis (http://www.microsoft.com/technet/treeview/?url=/technet/security/topics/assess/)Choosing a security update management solution (http://www.microsoft.com/windows2000/windowsupdate/sus/suschoosing.asp)How the SMS Software Update Services Feature Pack works (http://www.microsoft.com/smserver/techinfo/administration/20/using/suspackhowto.asp)Windows patch-management tools (http://www.nwfusion.com/reviews/2003/0303patchrevl)BigFix (http://www.bigfix.com/website/indexl)Shavlik (http://www.shavlik.com)
Richard Threlkeld