Better Faster Lighter Java [Electronic resources] نسخه متنی

اینجــــا یک کتابخانه دیجیتالی است

با بیش از 100000 منبع الکترونیکی رایگان به زبان فارسی ، عربی و انگلیسی

Better Faster Lighter Java [Electronic resources] - نسخه متنی

Justin Gehtland; Bruce A. Tate

نمايش فراداده ، افزودن یک نقد و بررسی
افزودن به کتابخانه شخصی
ارسال به دوستان
جستجو در متن کتاب
تنظیمات قلم


اندازه قلم

+ - پیش فرض

حالت نمایش

روز نیمروز شب
جستجو در لغت نامه
لیست موضوعات
افزودن یادداشت
افزودن یادداشت جدید

9.1 What Is the Spider?

One of the most valuable features of any web site is the ability to search for what you need. Companies with web sites are constantly looking for the right tool to provide those features; they can write their own or purchase something from one of the big vendors. The problem with writing your own is mastering the tools. The problem with purchasing is usually vast expense. Google, the world's leading search provider, sells a boxed solution at $18,000 per unitnot including the yearly license.

Customized search engines are often built around the act of querying the database that sits behind a web site. Programmers immediately jump to this solution becausetools and libraries make querying a database simple. However, these customized search solutions often miss entire sections of a web site; no matter how stringently a company tries to build an all-dynamic, data-driven web site, they almost always end up with a few static HTML files mixed in. A data-driven query won't discover those pages.

Crawling a web site is usually the answer, but don't attack it naively. Let's look at what crawling means. When you crawl a web site, you start at some initial page. After cataloging the text of the page, you parse it, looking for and following any hyperlinks to other endpoints, where you repeat the process. If you aren't careful, crawling a web site invites the most ancient of programming errors: the infinite loop.

Take a look at Figure 9-1. The web site is only four pages, but no simple crawler will survive it. Given Page1 as a starting point, the crawler finds a link to Page2. After indexing Page1, the crawler moves on to Page2. There, it finds links to Page3 and Page4. Page4 is a nice little cul-de-sac on the site, and closes down one avenue of exploration. Page3 is the killer. Not only does it have a reference back to Page1, starting the whole cycle again, but it also has an off-site link (to Anyone who wants a crawler to navigate this beast has more processor cycles than brain cells.

Figure 9-1. A simple, four-page web site that breaks any naïve crawler

I had a client who couldn't afford the $18,000 expense to buy search capabilities and didn't want to sit down and write something custom that might cost them the same amount in development dollars. They came to me and provided a set of straightforward requirements for an application that would enable them to search on their web site. Here's what they asked me to do:

Provide a service for crawling a web site, following all links from a provided starting point.

The crawling service must ignore links to image files.

The crawler must be configurable to only follow a maximum number of links.

Provide a service for indexing the resulting set of web pages. The indexing service should be schedulable; initially, it should run every night at midnight.

Each result of a search of the index should return a filename and a rank indicating the relative merit of each result.

Create two interfaces for accessing the spider:

A console interface for local searches and testing.

A web service that returns an XML document representing the results of all the searches.

My solution was to write an open source web site indexing and search engine. The goal was to have an application that could be pointed at any arbitrary web site, crawl it to create the domain of searchable pages, and allow a simple search language for querying the index. The crawler would be configurable to either allow or deny specific kinds of links, based on the link prefix (for example, ONLY follow links starting with or NEVER follow links starting with The indexer would operate on the results of the crawler and the search engine would query the index. Here are the advantages this engine would provide:

No $18,000 to Google.

No $18,000 to the IT department.

General enough to work with any web site.

A layered architecture that would allow it to easily be used in a variety of UI environments.

/ 111