How Do Search Engines Work - internet Crawlers

 Search Engines use spiders to index websites. 

after you submit your website pages to a pursuit engine by finishing their needed submission page, the computer program spider can index your entire website.

How Do Search Engines work


A ‘spider’ is an automatic program that goes past the computer program system.
 Spider visits an internet website, scans the content on the particular website, {the website|the location|the positioning}'s Meta tags, and additionally follows the links that the site connects.
 The spider then returns all that data back to a central repository, wherever the info is indexed. it'll visit every link you have got on your website and index those sites further. 

Some spiders can solely index a definite variety of pages on your website, therefore don’t produce a website with five hundred pages!

The spider can sporadically come back to the sites to envision any data that has been modified. The frequency with that this happens is decided by the moderators of the computer program. A spider is sort of a book wherever it contains the table of contents, the particular content, and also the links and references for all the websites it finds throughout its search, and it's going to index up to 1,000,000 pages every day.


  1.  Excite
  2.  Lycos
  3. AltaVista
  4.  Google 

When you raise a pursuit engine to find data, it's truly rummaging through the index that it's created and not truly looking out the net. totally {different|completely different} computer programs turn out different rankings as a result of not each search engine using a similar algorithmic rule to look through the indices.

One of the items that a pursuit engine algorithmic rule

  • One of the items that a pursuit engine algorithmic rule scans for is the frequency and site of keywords on an internet page, however, it also can sight artificial keyword stuffing or spamdexing.
  •  Then the algorithms analyze the method by which pages link to different pages on the internet.
  •  By checking which pages link to every different, the associate degree engine will each confirm what a page is regarding if the keywords of the joined pages are the same as the keywords on the initial page.