| Topic S |
| Crawler |
| The program designed to browse World Wide Web automatically is well known as , or robots. There is content created by and images which is not visible to . But which is readable to the spiders because they are designed to read page. |
| How do spiders help search engines ? |
| Crawling over the pages to update information |
As mentioned in the introduction spiders are the robots intended to read Meta tag, & of page. They read the information given in these tags. After a specific time interval they crawl over the web page & as well as that the site connects to perform indexing. These supervise the changes of the web page and all information is finally returned to the central depository.
|
| Indexing in other terms updating of the pages is the major task of . Because when we search in the we do not hit the web but the database where the updated information is stored. |
| Dead links and spiders |
| Some times it also happens that the which we have removed from our records (i.e. dead link) is also displayed by search engines because that data is stored in the database of search engines. This is all due to the spiders that while crawling on the particular page store information in the database of the search engines. |
| Spider as an expert technician |
| Web site owners often try to deceive search with their tricks like hidden text, , pages etc. to get better ranking in the search engine. But as an expert technician these find out the technique of and thus they penalize orthe site. |
| Ranking & indexing |
| Search engine spiders have thousands of pages to crawl &INDEX. That is why they drop the newly submitted site from indexing. You have to wait for six month to one year to get your site indexed by search engines. In this essence also you have to your page again & again because of their tendency of dropping your site again & again. Once your site is indexed by search engine you have to take care about the interest of your visitors that what they actually want. |
| Spiders are inflexible machine. They are software program. They can not read the graphics, objects or images. So the care should be taken that to give proper coding in the page to make it visible and readable to the spiders. |
| Observation of spiders during ranking |
1* Many search engine spiders stress more on the . Make use of targeted & meaningful keywords in “title”& “description” Meta tags.
2* Search engine result also depends on the KEYWORED in which the result are more close or it matches with the quarry.
3* is any word on the web page which is used to catch attention. |
| After the keen inspection of all these elements search engine compare the web site and finally rank the. |
| Robot.txt |
| I would like to mention here about the well known program of search engine that is . This file disallow search engine spiders to crawl over some area of the web site. |
| |
| |
