20.10. Mirroring Web Pages
20.10.1. Problem
You want a local copy of a web page kept
up-to-date.
20.10.2. Solution
Use LWP::Simple''s mirror
function:
use LWP::Simple;
mirror($URL, $local_filename);
20.10.3. Discussion
Although closely related to the get function
discussed in Recipe 20.1, the
mirror function doesn''t download the file
unconditionally. It adds the If-Modified-Since
header to the GET request it creates, so the server does not transfer
the file unless the file has been
updated. The mirror function mirrors only a single page,
not a full tree. To mirror a set of pages, use this recipe in
conjunction with Recipe 20.3. A good solution
to mirroring an entire directory hierarchy can be found in the
w3mir program, also found on CPAN, and the
wget program from
ftp.gnu.org.Be careful! It''s possible (and easy) to write programs that run amok
and begin downloading all web pages on the net. This is not only poor
etiquette, it''s also an infinite task, since some pages are
dynamically generated. It could also get you into trouble with
someone who doesn''t want their pages downloaded en
masse.
20.10.4. See Also
The documentation for the CPAN module LWP::Simple; the HTTP
specification at http://www.w3.org/pub/WWW/Protocols/HTTP/