Google Hacks 2Nd Edition [Electronic resources] نسخه متنی

Hack 57. Scrape Google Groups

Pull results from Google Groups searches a
comma-delimited file .

It's

easy to
look at the Internet and say that it's a group of
web pages or computers or networks. But look a little deeper and
you'll see that the core of the Internet is
discussions: mailing lists, online forums, and even web sites, where
people hold forth in glorious HTML, waiting for people to drop by,
consider their philosophies, make contact, or buy their products and
services.

Nowhere is the Internet-as-conversation idea more prevalent than in
Usenet newsgroups. Google Groups has an archive of over 800 million
messages from years of Usenet traffic. If you're
researching a particular time, searching and saving Google Groups
message pointers comes in really handy.

Because Google Groups is not searchable by the current version of the
Google API, you can't build an automated Google
Groups query tool without violating Google's Terms
of Service. However, you can scrape the HTML of a page you visit
personally and save to your hard drive.

The first thing that you need to do is run a Google Groups Search.
See the "Google Groups" section
earlier in this chapter for some hints on the best practices for
searching this massive message archive.

This hack works with Google Groups, not Google Groups 2. While any sort of scraping is brittle, we expect Version 2 to change form many times in the very near future and wanted to be sure you had the best chance of success with this hack.

It's best to put pages that you're
going to scrape in order of date; that way if you're
going to scrape more pages later, it's easy to look
at them and check the last date that the search results changed.
Let's say that you're trying to
keep up with uses of Perl in programming the Google API; your query
might look like this:

perl group:google.public.web-apis On the right side of the results page is an option to sort either by
relevance or date; click the "Sort by
date" link. Your results page should look something
like Figure 4-11.

Save this page to your hard drive, naming it something memorable,
like groupsl.

Scraping is brittle at best. A single change in the HTML code underlying Google Groups pages and the script won't get very far.

At the time of this writing, a typical Google Groups Search result
looks like this:

<a href= or results ?</a><font size=-1><br> <b>...</b>
Yet when making a query, via the <b>perl</b> Net::Google module, setting
max_results to 50 works fine and returns 50 results, which was not what I had expected.
<b>...</b> <br><font color=green><a href= class=a>google.public.
web-apis</a> - Aug 31, 2004 by sean - <a href= class=a>View Thread (1 article)</a>

As with the HTML example given for Google News in [Hack #54] , this might be
utter gobbledygook for some of you. Those of you with an
understanding of the code below should see why the regular expression
matching was written in the way it was.

4.16.1. The Code

Save the following code as groups2csv.pl:

#!/usr/bin/perl
# groups2csv.pl
# Google Groups results exported to CSV suitable for import into Excel.
# Usage: perl groups2csv.pl < groupsl > groups.csv
# The CSV Header.
print qq{"title","url","group","date","author","number of articles"\n};
# The base URL for Google Groups.
my $url = "http://groups.google.com";
# Rake in those results.
my($results) = (join '', <>);
# Perform a regular expression match to glean individual results.
while ( $results =~ m!<a href= href=" - (.+?) by
(.+?)\s+.*?\(([0-9]+) article!mgis ) {
my($path, $title, $snippet, $group, $date, $author, $articles) =
($1||'',$2||'',$3||'',$4||'',$5||'',$6||'',$7||'');
$title =~ s!"!"!g; # double escape " marks
$title =~ s!<.+?>!!g; # drop all HTML tags
print qq{"$title","$url$path","$group","$date","$author","$articles"\n\n};
}

4.16.2. Running the Hack

Run the script from the command line ["How to Run
the Hacks" in the Preface], specifying the Google
Groups results filename that you saved earlier and the name of the
CSV file that you wish to create or to which you wish to append
additional results. For example, use groupsl
as your input and groups.csv as your output:

$ perl groups2csv.pl < groupsl
> groups.csv Leaving off the > and CSV filename sends the
results to the screen for your perusal.

Using >> before the CSV filename appends the
current set of results to the CSV file, creating it if it
doesn't already exist. This is useful for combining
more than one set of results, represented by more than one saved
results page:

$ perl groups2csv.pl
< results_1l
> results.csv
$ perl groups2csv.pl
< results_2l
>> results.csv

4.16.3. The Results

Scraping the results of a search for perl
group:google.public.web-apis for anything mentioning the
Perl programming language on the Google API's
discussion forum looks like this:

$ perl groups2csv.pl < groupsl
"title","url","group","date","author","number of articles"
"queries or results ?","http://groups.google.com/groups?q=perl+group:google.public.
web-apis&hl=en&lr=&c2coff=1&safe=off&scoring=d&selm=bfd91813.
0408311406.21d2bb89%40posting.google.com&rnum=1","google.public.web-apis","Aug 31,
2004","sean",
"1"
...
"Re: Whats the Difference between using the API and ordinary ... ","http://groups.google.
com/groups?q=perl+group:google.public.web-apis&hl=en&lr=&c2coff=1&safe=
off&scoring=d&selm=882fdb00.0405052309.44fe831b%40posting.google.com&rnum=7",
"google.public.web-apis","May 6, 2004","tonio","4"
...