Pronunciation

A type of bot that peruses the internet to log indices of any information it can retrieve.

Spiders are often employed by search engines to effectively track and record information across the Internet within an index maintained by the search engine itself. With up-to-date information housed in its index, search engines can provide better, more accurate information for user queries. Though the amount of web pages that spiders monitor is vast, not all information available online can be indexed due to designers disallowing bots on the website or if a website lives on a private server.

Technology