Google significantly changed the crawler documentation, resulting in a higher level of information density and tighter topical coverage Google has launched a major revam p of its Crawler documentation, shrinking the main overview page and splitting content into three new, more focused pages. Although the changelog downplays the changes there is an entirely new section and basically a rewrite of the entire crawler overview page. The additional pages allows Google to increase the information density of all the crawler pages and improves topical coverage. What Changed? Google’s documentation changelog notes two changes but there is actually a lot more. Here are some of the changes: Added an updated user agent string for the GoogleProducer crawler Added content encoding information Added a new section about technical properties The technical properties section contains entirely new information that didn’t previously exist. There are no changes to the crawler behavior, but by creating thr