Perl Cd Bookshelf [Electronic resources] نسخه متنی

20.19. Extracting Table Data

20.19.1. Problem

You have data
in an HTML table, and you would like to turn that into a Perl data
structure. For example, you want to monitor changes to an author's
CPAN module list.

20.19.2. Solution

Use the HTML::TableContentParser module from
CPAN:

use HTML::TableContentParser;
$tcp = HTML::TableContentParser->new;
$tables = $tcp->parse($HTML);
foreach $table (@$tables) {
@headers = map { $_->{data} } @{ $table->{headers} };
# attributes of table tag available as keys in hash
$table_width = $table->{width};
foreach $row (@{ $tables->{rows} }) {
# attributes of tr tag available as keys in hash
foreach $col (@{ $row->{cols} }) {
# attributes of td tag available as keys in hash
$data = $col->{data};
}
}
}

20.19.3. Discussion

The HTML::TableContentParser module converts all tables in the HTML
document into a Perl data structure. As with HTML tables, there are
three layers of nesting in the data structure: the table, the row,
and the data in that row.

Each table, row, and data tag is represented as a hash reference. The
hash keys correspond to attributes of the tag that defined that
table, row, or cell. In addition, the value for a special key gives
the contents of the table, row, or cell. In a table, the value for
the rows key is a reference to an array of rows.
In a row, the cols key points to an array of
cells. In a cell, the data key holds the HTML
contents of the data tag.

For example, take the following table:

<table width="100%" bgcolor="#ffffff">
<tr>
<td>Larry &amp; Gloria</td>
<td>Mountain View</td>
<td>California</td>
</tr>
<tr>
<td><b>Tom</b></td>
<td>Boulder</td>
<td>Colorado</td>
</tr>
<tr>
<td>Nathan &amp; Jenine</td>
<td>Fort Collins</td>
<td>Colorado</td>
</tr>
</table>

The parse method returns this data
structure:

[
{
'width' => '100%',
'bgcolor' => '#ffffff',
'rows' => [
{
'cells' => [
{ 'data' => 'Larry & Gloria' },
{ 'data' => 'Mountain View' },
{ 'data' => 'California' },
],
'data' => "\n      "
},
{
'cells' => [
{ 'data' => '<b>Tom</b>' },
{ 'data' => 'Boulder' },
{ 'data' => 'Colorado' },
],
'data' => "\n      "
},
{
'cells' => [
{ 'data' => 'Nathan &amp; Jenine' },
{ 'data' => 'Fort Collins' },
{ 'data' => 'Colorado' },
],
'data' => "\n      "
}
]
}
]

The data tags still contain tags and entities. If you don't want the
tags and entities, remove them by hand using techniques from Recipe 20.6.

Example 20-11 fetches a particular CPAN author's page
and displays in plain text the modules they own. You could use this
as part of a system that notifies you when your favorite CPAN authors
do something new.

Example 20-11. Dump modules for a particular CPAN author

  #!/usr/bin/perl -w
# dump-cpan-modules-for-author - display modules a CPAN author owns
use LWP::Simple;
use URI;
use HTML::TableContentParser;
use HTML::Entities;
use strict;
our $URL = shift || 'http://search.cpan.org/author/TOMC/';
my $tables = get_tables($URL);
my $modules = $tables->[4];    # 5th table holds module data
foreach my $r (@{ $modules->{rows} }) {
my ($module_name, $module_link, $status, $description) = 
parse_module_row($r, $URL);
print "$module_name <$module_link>\n\t$status\n\t$description\n\n";
} 
sub get_tables {
my $URL = shift;
my $page = get($URL);
my $tcp = new HTML::TableContentParser;
return $tcp->parse($page);
}
sub parse_module_row {
my ($row, $URL) = @_;
my ($module_html, $module_link, $module_name, $status, $description);
# extract cells
$module_html = $row->{cells}[0]{data};  # link and name in HTML
$status      = $row->{cells}[1]{data};  # status string and link
$description = $row->{cells}[2]{data};  # description only
$status =~ s{<.*?>}{  }g; # naive link removal, works on this simple HTML
# separate module link and name from html
($module_link, $module_name) = $module_html =~ m{href=".*?>(.*)<}i;
$module_link = URI->new_abs($module_link, $URL); # resolve relative links
# clean up entities and tags
decode_entities($module_name);
decode_entities($description);
return ($module_name, $module_link, $status, $description);
}

20.19.4. See Also

The documentation for the CPAN module HTML::TableContentParser;
http://search.cpan.org