Skip to main content

Google Revamps Entire Crawler Documentation

Google significantly changed the crawler documentation, resulting in a higher level of information density and tighter topical coverage



Google has launched a major revamp of its Crawler documentation, shrinking the main overview page and splitting content into three new, more focused pages.  Although the changelog downplays the changes there is an entirely new section and basically a rewrite of the entire crawler overview page. The additional pages allows Google to increase the information density of all the crawler pages and improves topical coverage.

What Changed?

Google’s documentation changelog notes two changes but there is actually a lot more.

Here are some of the changes:

Added an updated user agent string for the GoogleProducer crawler

Added content encoding information

Added a new section about technical properties

The technical properties section contains entirely new information that didn’t previously exist. There are no changes to the crawler behavior, but by creating three topically specific pages Google is able to add more information to the crawler overview page while simultaneously making it smaller.

There is additional information about crawling over HTTP/1.1 and HTTP/2, plus a statement about their goal being to crawl as many pages as possible without impacting the website server.

What Is The Goal Of The Revamp?

The change to the documentation was due to the fact that the overview page had become large. Additional crawler information would make the overview page even larger. A decision was made to break the page into three subtopics so that the specific crawler content could continue to grow and making room for more general information on the overviews page. Spinning off subtopics into their own pages is a brilliant solution to the problem of how best to serve users.



Labels:

Comments

Post a Comment

Popular posts from this blog

What is SEO & How Does SEO Work?

 What is SEO & How Does SEO Work? SEO stands for Search Engine Optimization, a digital marketing strategy that focuses on improving your website position in search results on search engine like google. Since search engine used hundreds of factors to generate search results Google uses more than 200 SEO works by optimising  your website for these factors, along with getting your sites crawled and indexed. When you come to know how SEO works you can use the other tactics, like keyword research, content creation and page speed optimisation, to increase your visibility in search result. How does SEO works for search engines? Search engine optimization is the product of search engines- search engine companies develop ranking factors and use those factors to determine the most relevant content for the search. However, before search engine can determine the most relevant content, it must crawl and index it. Crawling: Search Engine use crawling to discover and re-discover new and ...
Post a Comment
Read more

Google: Avoid Duplicate Content In Business Profile Posts

Duplicate content in post published by Google Business Profiles is now considered as spam, according to the content policy updates. Google added a line to its Business Profile post content policy under the section cautioning users to avoid spam. Among the list of content users should avoid posting on their Business Profile. It is important to be aware of what this change means for our content strategy so you don't find yourself in violation of the updated policy.  NO DUPLICATE CONTENT IN GOOGLE POSTS.  With this update content policy, Google 8s sending a clear message that it wants businesses to post unique content. Posting the same photo, video, or text block isn't allowed. If you aren't careful, breaking google new rule is possible, even if you are not trying to spam. In addition, it sounds like Google wants businesses to limit their use of logos putting a logo on every photo, for example, may get you in trouble. The best way to stay within the guidelines is to ensure eve...
Post a Comment
Read more