A method of collecting, arranging and categorizing the information that crawlers, bots, or spiders find on websites.