Working with ISA Server 2004 Logs and Reports - Dr. Tom Shinderamp;#039;s Configuring ISA Server 1002004 [Electronic resources] نسخه متنی

Working with ISA Server 2004 Logs and Reports

ISA Server 2004's logging and reporting features take monitoring a step further and provide you with permanent documentation of the activities related to your ISA server. In the following sections, we take a look at how ISA Server logs data, how to configure the logs, and how to generate reports based on the logged information.

Understanding ISA Server 2004 Logs

ISA Server 2004 logs all components by default. These logs include the following:

Web Proxy

Firewall Service

SMTP Message Screener

Log Types

The default log type is a Microsoft Data Engine (MSDE) database. The MSDE service is installed along with ISA Server 2004. If you have a SQL server on the network, you can configure the logs to be saved to the SQL database, or you can save the information to a file (World Wide Web Consortium or W3C format, or ISA Server format). There are advantages and disadvantages to each.

Logging to an MSDE Database

You can use ISA Server 2004's log viewer to display information saved in an MSDE database. You can query the database to find specific information. This is one of the reasons we like the MSDE format. The logs themselves are limited to 2GB each, but the log viewer will display all the information in separate log files as if it came from the same file. If a log reaches the 2GB limit, ISA Server 2004 will automatically start a new one.

You can also export the information from the log viewer to a text file, which is handy if you use analysis tools that require text files.

By default, MSDE log information is saved to the ISALogs folder within the ISA Server installation folder.

Logging to a SQL Server

Logging to a SQL server allows you to use standard SQL tools to query the database. There is also some fault tolerance in having the logs located on a remote SQL server. However, if connectivity with the SQL server is lost, the Firewall service shuts down.

There are also a number of security issues involved in logging to a remote SQL server. If you choose to do so, Microsoft recommends using Windows authentication rather than SQL authentication, and you should also consider encrypting the log information and implementing IPSec for the data transmitted from the ISA Server to the SQL server. In order to log to a SQL database, you will need to ensure that the system policy rule to allow remote logging using NetBios transport to trusted servers is enabled on the ISA server.

Logging to a File

If you select to use the W3C file format, the data is stored along with information about the version, log date, and logged fields. The W3C format creates a tab-delimited file.

If you select to use the ISA Server file format, only the data itself is saved, and all fields are logged, whether selected or not, but unselected fields are shown as empty (marked by a dash). This format creates a comma-delimited file.

Another difference between the two formats is that W3C files denote the date and time in Coordinated Universal Time (UTC), whereas the ISA Server format uses local time as configured on the computer.

The files are stored by default in the ISALogs folder. You can change this location if you want. If the partition on which the logs are stored is formatted in NTFS (which we recommend), you can compress the log files to save space, although this may cause a reduction in performance (access time).

W3C and ISA Server log files, like MSDE files, are limited to 2GB, but a new file is started automatically when the limit is reached. ISA Server monitors log file size at ten minute intervals.

How to Configure Logging

You can configure logging separately for each of the three services (Firewall, Web Proxy, and SMTP Message Screener). Click the Logging tab in the Monitoring node, and select Configure Firewall Logging, Configure Web Proxy Logging, or Configure SMTP Message Screener Logging in the right task pane, as shown in Figure 12.36.

Figure 12.36: Configuring Logging Separately

Configuration is basically the same for each service, with a few differences. In our example, we will configure logging for the Firewall service. The first step is to ensure that the Enable logging for this service box at the bottom of the Log tab is checked (it is enabled by default). Next, you need to configure the log storage format, as shown in Figure 12.37.

Figure 12.37: Configuring Log Storage Format

Configuring MSDE Database Logging

By default, the MSDE database format is selected. To configure it, click Options, which will display the Options dialog box shown in Figure 12.38.

Figure 12.38: Configuring MSDE Database Logging

Here you can select whether to store the MSDE database files in the default location in the ISALogs folder, or in a different folder. To do the latter, type in the folder path or click the Browse button to browse to the folder where you want to store the logs.

Next, you can set a limit on the total size of all log files, in gigabytes. The default is 8 GB. You can also set an amount of free disk space that is to be maintained, in megabytes. The default is 2048MB (2 GB).

You can select also determine what ISA Server will do when the log limits are reached: either delete the oldest files to make room for new ones, or discard the new log entries. Finally, you can select to automatically delete files that are older than a specified number of days (by default, this option is selected and the default time period is 7 days).

On the Fields tab, you can check the fields that you want logged or uncheck those you don't want logged. If you want to log all fields, you can click the Select All button or you can clear all fields with the Clear All button. By default, all fields are logged except the following:

Bidirectional

Source proxy

Destination proxy

Client host name

Destination host name

Network Interface

Raw IP Header

Raw Payload

You can log only the default fields by clicking the Restore Defaults button.

Configuring Logging to a File

If you choose to log to a file, you will need to select the file format from the drop-down box: either ISA Server file format or W3C extended log file format. When you click Options, you will see the same options you were given for MSDE logging (location to store the log file, storage limits, actions for maintaining storage limits), but you will also see that the Compress log files checkbox is now available.

Configuring Logging to a SQL Database

If you choose to log to a SQL database, you will first need to set up a SQL server for ISA Server logging. This involves configuring the SQL server to accept the Open Database Connectivity (ODBC) connection from the ISA Server. You'll need to create a SQL server account if the SQL Server and the ISA Server aren't in the same Windows domain. If the two are in the same domain, you can use Windows authentication; if they are in different domains that do not have an appropriate trust relationship, you have to use SQL authentication.

To set up logging to a SQL server, you will first have to prepare the SQL database and tables. Fortunately, the ISA Server installation CD contains two SQL scripts, fwsrv.sql and w3proxy.sql, that will create the tables used to record the Web Proxy and Firewall service data. You will have to modify these scripts by adding SQL statements to use an existing database or create a new one to store the tables. You can use the SQL Query Analyzer tool that comes with SQL Server to run these scripts to create the tables. After setting up the SQL tables, you will have to configure appropriate permission for the Windows or SQL account used by ISA Server to be able to query and insert data.

Once you have the SQL server set up, on the Log tab of the Firewall Logging Properties dialog box, you'll need to enter the name of the ODBC data source and a table name. Then you may need to set a user account. To do so, click Set Account, and enter the user name and password (twice) in the Set Account dialog box. You can browse for a user by clicking the Browse button.

You'll need to enable the necessary Remote Logging configuration groups in the System Policy Editor.

To configure Web Proxy or SMTP Message Screener logging, the procedure is the same as for Firewall logging. The primary difference is in the available fields to be logged. In addition, you'll find that all log storage formats except File are grayed out in the SMTP Message Screener Logging properties dialog box.

How to Use the Log Viewer

The log viewer will show you entries being logged in real time as they happen. Each event is displayed in the log viewer as soon as it is logged. Click the Logging tab to use the log viewer. The default filter displays all log records for the Firewall or Web Proxy logs. To display these records, click Start Query in the task pane. Entries will continue to be added to the display in real time until you click Stop Query.

Because the log viewer contains many columns, you might want to close the console tree pane and/or the task pane to provide more room. Even if you do, you will probably still have to scroll to see all the default columns. The log viewer is shown in Figure 12.39.

Figure 12.39: The Log Viewer with Default Filter

By default, the following columns are shown:

Log time

Destination IP

Destination port

Protocol

Action

Rule

Client IP

Client user name

Source network

Destination network

HTTP method

URL

You can add additional columns, such as MIME type, source or destination proxy, referring server, and many others. To do so, or just to view a list of available column headers, right-click any column header and select Add/Remove Columns.

How to Filter the Log Information

You can filter the information in log viewer similarly to the way you filtered the sessions information. As with the sessions filters, only those entries that meet all of your specified criteria will be displayed.

If you have logged to an MSDE database, you can also filter by log time. This allows you to display log data entered during a specific time period (rather than live data). You can only set the log time to something other than live for MSDE databases. This is referred to as offline viewing.

To configure a filter, click Edit Filter in the task pane. The Edit Filter dialog box is shown in Figure 12.40.

Figure 12.40: Editing a Log Filter

In the Filter by field, select the desired criteria.

Note that you cannot remove the default entries. When you click either the Log Record Type or Log Time entry, the Remove button is grayed out. If you try to create a new entry for Log Record Type or Log time, when you click Add to List, you will get a message that only one Log Record Type (or Log Time) expression is allowed in a query.

So how do you change these parameters? Here's the secret: Click the one you want to change to highlight it, make the change in the Value field, and then click Update.

You can choose from the following criteria by which to filter:

Action

Authenticated client

Bidirectional

Bytes received

Bytes sent

Cache information

Client agent

Client host name

Client IP

Client user name

Destination host name

Destination IP

Destination network

Destination port

Destination proxy

Error information

Filter information

HTTP method

HTTP status code

Log Record type

Log time

MIME type

Network interface

Object source

Original client IP

Processing time

Protocol

Raw IP header

Raw payload

Referring server

Result code

Rule

Server name

Service

Source network

Source port

Source proxy

Transport

URL

Some of these criteria apply only to one or the other log type (Firewall or Web Proxy).

When you configure the log record type, you can select to display entries from the Firewall or Web Proxy filter, from the Firewall filter only, or from the Web Proxy filter only. Note that you cannot display entries from the SMTP Message Screener logs.

When you configure the log time, in the Condition field the default is Live (and that is the only option if you are not logging to an MSDE database). If you're logging to MSDE, you can select any of the following:

Last 24 hours

Last 30 days

Last 7 days

Last hour

Live

On or after

On or before

If you choose one of the last two, you'll need to select a date and time in the Value field.

After you have specified all the desired criteria for filtering, click Start Query to display the entries filtered by your criteria.

Saving Log Viewer Data to a File

You can save the data displayed in the log viewer to a file by copying all results, or only selected results, to the Windows clipboard. To copy selected results, highlight the entries you want to copy (you can select multiple entries by holding down the CTRL or SHIFT keys). Click Copy Selected Results to the Clipboard. To copy all results, click Copy All Results to the Clipboard.

Then you can paste the copied results into a text editor, such as Notepad, as shown in Figure 12.41.

Figure 12.41: Saving Log Viewer Data by Copying to the Clipboard

Once you have the data in a text editor, you can save it as a text file. However, note that you can only display up to ten thousand results in the log viewer, so even if you copy all results, you may not get all entries in the log.

Exporting and Importing Filter Definitions

You can save your filter definitions in the same way you did with sessions filters, by selecting Export Filter Definitions in the task pane and selecting a location and file name. The filters are saved as .xml files. You can then load them by selecting the Import Filter Definitions in the task pane.

Because there are so many different filtering criteria available for filtering log information, it is handy to be able to save a number of different filters and import them when they are needed.

Generating, Viewing, and Publishing Reports with ISA Server 2004

The reporting function is where it all comes together; this is where you create reports that summarize or detail the information in the log files in such a way that allows you to easily analyze the data and spot patterns, trends, and anomalies.

You can track usage for bandwidth allocation purposes, or you can track access for security purposes. With the reporting feature, you can generate reports manually or schedule report jobs to be run on a regular basis. The reporting component creates a database in the ISASummaries folder (by default) on the ISA Server computer. Reports are based on summaries of the Firewall and Web Proxy logs.

How to Generate a One-time Report

To create a report, click the Reports tab in the Monitoring node. This will show you a listing of all reports that have been generated, or are in the process of generating, as shown in Figure 12.42.

Figure 12.42: The Reports Display

To create a new report, click Generate a New Report in the task pane. This will invoke the New Report Wizard. This wizard manually creates a single, one-time report as soon as you finish configuring the wizard. On the first page, you'll be asked to give your report a name.

On the next page of the wizard, you can select the type of content to include in this report. You can choose any of all of the following:

Summary

Web Usage

Application Usage

Traffic and Utilization

Security

In our example, shown in Figure 12.43, we've selected to include only Web usage data.

Figure 12.43: Configuring Report Content

Click Next, and you'll be asked to specify a reporting period (start date and end date). Because the reports are based on daily log summaries, you cannot include the present date as the end date.

On the next page, you will have the option to publish the report to a directory. You can type in a path or browse to the folder where you want to save the report. If you click Browse, you can use the Make New Folder button to create a new folder in which to save the reports. It will be named New Folder by default, but you can right-click it and rename it from within the Browse for Folder dialog box.

You may need to enter an account name and password of an account that has permission to write to the specified directory. If so, check the Publish using this account checkbox, as shown in Figure 12.44, and click Set Account to enter the account name and credentials.

Figure 12.44: Configuring Report Publishing

The report is automatically saved in HTML format.

On the next page, you can choose to have an e-mail notification sent when the report is completed. You'll need to enter the following information:

SMTP server name or IP address

Address from which the notification is to be sent

Address to which the notification is to be sent

CC: addresses of additional recipients, if any

Message for the body of the e-mail.

You can also check a checkbox to include a link to the completed published report within the e-mail message.

The last page of the wizard summarizes your choices. Use the Back button to make any changes, then click Finish to begin generating the report. The report will immediately appear in the Reports list, with the status shown as 'Generating,' as shown in Figure 12.45.

Figure 12.45: Generating the Report Upon Completion of the Wizard

Because it is a one-time report, the Period column will indicate 'Custom.'

How to Configure an Automated Report Job

You can configure a report job to generate reports on a daily, weekly, monthly, or yearly basis. This is handy for comparative purposes. For example, you might want to create a daily summary report, or a weekly Web usage report.

To create a report job, click Create and Configure Report Jobs in the right task pane. This will bring up the Report Jobs Properties dialog box, shown in Figure 12.46.

Figure 12.46: Creating Report Jobs

Here you will see a list of all scheduled report jobs. To add a new report job, click the Add button. This invokes the New Report Job Wizard. On the first page of the wizard, you'll be asked to give your report job a name (for example, Weekly Web Usage Report).

On the next page, you can configure the report content in the same way you did for a one-time report.

On the third page, as shown in Figure 12.47, you can select the time interval to run the report job: daily, weekly, or monthly.

Figure 12.47: Scheduling the Report Job

If you choose to run the job weekly, you can select on which day of the week to run the job. If you choose to run the job monthly, you will be asked to specify a day of the month on which to run the job. You should not use days that some months don't have (29, 30 or 31) if you want the job to run every month. If you want a report that covers the entire preceding month, you should set this value to 1.

On the next page, you can configure the job to publish the reports to a directory in the same way you did with the one-time report. The following page allows you to configure an e-mail message to be sent when the report completes, also in the same way as was done for the one-time report.

Finally, the last page of the wizard summarizes your choices. When you click Finish, the job will be scheduled to run on the day(s) you specified. By default, the report will start generating at 1:00

a.m. on the specified day. You can change this by selecting the report job in the Report Job Properties dialog box and clicking Edit. Click the Schedule tab, and you can change the generation hour, as shown in Figure 12.48.

Figure 12.48: Editing the Report Job Properties

Other Report Tasks

There are a number of other report-related tasks you can perform from the task pane. You can configure the log summary by clicking Configure Log Summary. This brings up the Log Summary Properties dialog box, from which you can enable or disable daily and monthly summaries by checking a checkbox, as shown in Figure 12.49.

Figure 12.49: Configuring the Log Summary

You can also change the default report generation time here, and specify where the summaries are to be saved (by default, they are saved in the ISASummaries folder). You can also configure the number of daily and monthly summaries to save (from a minimum of 35 to a maximum of 999 for daily summaries, and from a minimum of 13 to a maximum of 999 for monthly summaries).

You can also customize each of the report content types, using the following task pane selections:

Customize Summary Content: You can specify the number of protocols to include, specify the number of top users to report on, specify the sort order for determining top usage, specify the number of top Web sites and the sort order for determining top sites, and specify the sort order for the cache hit ratio, either by requests or by bytes.

Customize Web Usage Content: You can specify the number of top protocols to include and specify the sort order for determining top protocols (requests, users, bytes in, bytes out, or total bytes), specify the number of top Web sites and the sort order, specify the number of top users and the sort order, specify the number of object types and the sort order, specify the number of Web browsers and the sort order, and specify the number of operating systems and the sort order.

Customize Application Usage Content: You can specify the number of top protocols and the sort order, number of top users and sort order, number of client applications and sort order, number of destinations and sort order, and number of operating systems and sort order.

Customize Traffic and Utilization Content: You can specify the number of top protocols and the sort order for cache hit ratio.

Customize Security Content: You can specify the number of clients who generate the most dropped packets and the number of users who cause the most authorization failures.

How to View Reports

Once a report has been generated, you can view it from the Reports tab in the Monitoring node of the ISA Server Management console. Double-click the report name, and it will open in your Web browser, as shown in Figure 12.50.

Figure 12.50: Viewing Reports

As you can see, the reports use graphs and tables to make the information easy to access and analyze. You can quickly move to different sections of the report by clicking the hyperlinks on the left side of the page.

Publishing Reports

If you didn't select to automatically publish the report to a directory when you configured the report job, you can publish it after it has been generated. Just highlight the report you want to publish and click Publish Selected Report in the task pane.

You will be asked to select a destination location for the report files folder. Click OK, and the report will be published to the folder. A new subfolder will be created within the selected folder (the folder name will be the report name plus the date). All the HTML and graphics files for the report will be stored there. To open the report itself from the folder, double-click the file named Report.

Reports need to be published if you want to view them on computers other than the ISA Server computer.