Linux Security Cookbook [Electronic resources] نسخه متنی

اینجــــا یک کتابخانه دیجیتالی است

با بیش از 100000 منبع الکترونیکی رایگان به زبان فارسی ، عربی و انگلیسی

Linux Security Cookbook [Electronic resources] - نسخه متنی

Daniel J. Barrett, Robert G. Byrnes, Richard Silverman

| نمايش فراداده ، افزودن یک نقد و بررسی
افزودن به کتابخانه شخصی
ارسال به دوستان
جستجو در متن کتاب
بیشتر
تنظیمات قلم

فونت

اندازه قلم

+ - پیش فرض

حالت نمایش

روز نیمروز شب
جستجو در لغت نامه
بیشتر
لیست موضوعات
توضیحات
افزودن یادداشت جدید










Recipe 9.35 Combining Log Files



9.35.1 Problem



You want to merge a collection of log files
into a single, chronological log file.


9.35.2 Solution


#!/bin/sh
perl -ne 'print $last, /last message repeated \d+ times$/ ? "\0" : "\n" if $last;
chomp($last = $_);
if (eof) {
print;
undef $last;
}' "$@" | sort -s -k 1,1M -k 2,2n -k 3,3 | tr '\0' '\n'


9.35.3 Discussion


The system logger automatically prepends a timestamp to each message,
like this:

Feb 21 12:34:56 buster kernel: device eth0 entered promiscuous mode

To merge log files, sort each one by its
timestamp entries, using the first three
fields (month, date, and time) as keys.

A complication arises because the system logger inserts
"repetition messages" to conserve
log file space:

Feb 21 12:48:16 buster last message repeated 7923 times

The timestamp for the repetition message is often later than the last
message. It would be terribly misleading if possibly unrelated
messages from other log files were merged between the last message
and its associated repetition message.

To avoid this, our Perl script glues together the last message with a
subsequent repetition message (if present), inserting a null
character between them: this is reliable because the system logger
never writes null characters to log files. The script writes out the
final line before the end of each file and then forgets the last
line, to avoid any possibility of confusion if the next file happens
to start with an unrelated repetition message.

The sort command sees these null-glued
combinations as single lines, and keeps them together as the files
are merged. The null characters are translated back to newlines after
the files are sorted, to split the combinations back into separate
lines.

We use sort -s to avoid sorting entire lines if
all of the keys are equal: this preserves the original order of
messages with the same timestamp, at least within each original log
file.

If you have configured the system logger to write messages to
multiple log files, then you may wish to remove duplicates as you
merge. This can be done by using sort -u instead
of -s, and adding an extra sort key -k
4
to compare the message contents. There is a drawback,
however: messages could be rearranged if they have the same
timestamp. All of the issues related to sort -s
and -u are consequences of the one-second
resolution of the timestamps used by the system logger.

We'll note a few other pitfalls related to
timestamps. The system logger does not record the year, so if your
log files cross a year boundary, then you will need to merge the log
files for each year separately, and concatenate the results.
Similarly, the system logger writes timestamps using the local time
zone, so you should avoid merging log files that cross a daylight
saving time boundary, when the timestamps can go backward. Again,
split the log files on either side of the discontinuity, merge
separately, and then concatenate.

If your system logger is configured to receive messages from other
machines, note that the timestamps are generated on the machine where
the log files are stored. This allows consistent sorting of messages
even from machines in different time zones.


9.35.4 See Also


sort(1).

/ 247