Crawling The Deep Web
Researchers have been exploring how the deep Web can be crawled in an automatic fashion. In 2001, Sriram Raghavan and Hector Garcia-Molina presented an architectural model for a hidden-Web crawler that used key terms provided by users or collected from the query interfaces to query a Web form and crawl the deep Web resources. Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho of UCLA created a hidden-Web crawler that automatically generated meaningful queries to issue against search forms. Several form query languages (e.g., DEQUEL) have been proposed that, besides issuing a query, also allow to extract structured data from result pages. Another effort is DeepPeep, a project of the University of Utah sponsored by the National Science Foundation, which gathered hidden-Web sources (Web forms) in different domains based on novel focused crawler techniques.
Commercial search engines have begun exploring alternative methods to crawl the deep Web. The Sitemap Protocol (first developed by Google) and mod oai are mechanisms that allow search engines and other interested parties to discover deep Web resources on particular Web servers. Both mechanisms allow Web servers to advertise the URLs that are accessible on them, thereby allowing automatic discovery of resources that are not directly linked to the surface Web. Google's deep Web surfacing system pre-computes submissions for each HTML form and adds the resulting HTML pages into the Google search engine index. The surfaced results account for a thousand queries per second to deep Web content. In this system, the pre-computation of submissions is done using three algorithms: (1) selecting input values for text search inputs that accept keywords, (2) identifying inputs which accept only values of a specific type (e.g., date), and (3) selecting a small number of input combinations that generate URLs suitable for inclusion into the Web search index.
Read more about this topic: Deep Web
Famous quotes containing the words crawling, deep and/or web:
“You know, Frank, I’m beginning to get a new perspective on this crawling little animal known as man. Why a dog or a cat or a bird is cleverer than any human. They sense me immediately. But these shrewd detectives of yours—. Take away one of man’s senses and you render him helpless.”
—Lester Cole (1904–1985)
“There is a pleasure in the pathless woods,
There is a rapture on the lonely shore,
There is society where none intrudes
By the deep sea, and music in its roar:
I love not man the less, but nature more,”
—George Gordon Noel Byron (1788–1824)
“Any newspaper, from the first line to the last, is nothing but a web of horrors.... I cannot understand how an innocent hand can touch a newspaper without convulsing in disgust.”
—Charles Baudelaire (1821–1867)