U.C. Berkeley



Library Web
Finding Information on the Internet: A Tutorial
http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html
Invisible or Deep Web:
What it is, How to find it, and Its inherent ambiguity
UC Berkeley - Teaching Library Internet Workshops
About This Tutorial | Table of Contents | Contact us

What is the "Invisible Web", a.k.a. the "Deep Web"?

The "visible web" is what you can find using general web search engines. It's also what you see in almost all subject directories. The "invisible web" is what you cannot find using these types of tools.

The first version of this web page was written in 2000, when this topic was new and baffling to many web searchers. Since then, search engines' crawlers and indexing programs have overcome many of the technical barriers that made it impossible for them to find "invisible" web pages.

These types of pages used to be invisible but can now be found in most search engine results:

Why isn't everything visible?

There are still some hurdles search engine crawlers cannot leap. Here are some examples of material that remains hidden from general search engines:


How to Find the Invisible Web

Simply think "databases" and keep your eyes open. You can find searchable databases containing invisible web pages in the course of routine searching in most general web directories. Of particular value in academic research are:

Use Google and other search engines to locate searchable databases by searching a subject term and the word "database". If the database uses the word database in its own pages, you are likely to find it in Google. The word "database" is also useful in searching a topic in the Yahoo! directory, because they sometimes use the term to describe searchable databases in their listings.

Examples:
plane crash database
languages database
toxic chemicals database

Remember that the Invisible Web exists. In addition to what you find in search engine results (including Google Scholar) and most web directories, there are other gold mines you have to search directly. This includes all of the licensed article, magazine, reference, news archives, and other research resources that libraries and some industries buy for those authorized to use them.

As part of your web search strategy, spend a little time looking for databases in your field or topic of study or research. The contents of these may not be freely available: libraries and corporations buy the rights for their authorized users to view the contents. If they appear free, it's because you are somehow authorized to search and read the contents (library card holder, company employee, etc.).


The Ambiguity Inherent in the Invisible Web:

It is very difficult to predict what sites or kinds of sites or portions of sites will or won't be part of the Invisible Web. There are several factors involved:


Want to learn more about the Invisible Web?


Quick Links
Search Engines |Subject Directories | Meta-Search Engines | Invisible Web

Creative Commons License
Invisible Web: What it is, Why it exists, How to find it, and Its inherent ambiguity
Copyright © 2012 The Regents of the University of California is licensed
under a Creative Commons Attribution-NonCommercial 3.0 Unported License.
Permissions beyond the scope of this license may be available at
http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/contact.html.


[ HELP/SEARCH ] [ CATALOGS ] [ COMMENTS ] [ HOME ]
Last update 05/08/12. Server manager: Contact