Webmaster level: all

It's been a bit more than five years now that our Webmaster Help Forum has been up and running, helping webmasters around the world. Over the years, over tens of thousands of users have discussed various topics in well over 100,000 threads, helping each other to improve their websites and to solve a variety of issues that web publishers are confronted with. Among those users is a group that we call the "Bionic Posters," users who have proven to be consistently helpful and knowledgeable, selflessly helping others to tackle seemingly insurmountable problems.

It's great to have such an awesome community -- but thanks is best said by those who are helped. Here is some of the feedback that we collected this year:

• "Thank you for the time you have spent helping me. It is genuinely appreciated."
• "Thank you to everyone who helped with my problem! My creaky old 1996 era website is all cleaned up and doing just fine now! Good Guys Rule!"
• "WOW!!!!!!!! Thank you so much for your help! I was reluctant to post because I thought you guys might think my site is too small, too insignificant, etc. Thanks so much! To me, it's a BIG deal!"
• "My traffic has doubled and I am now either top or close to the top in search terms"
• "Thanks. Hopefully my late night paid off then, all the help and information has been great!!!!!!!"
• "Wow Webado Thank You Again.. You Really Know Your stuff! (…) You Are a True professional and Seriously The Only Person That Could Even Figure This out. I even Spoke to Other Top Specialists and YOU were the Only one who told me what to do and what was wrong."
• "You are AMAZING! Thank you so much!"
• "Finally, thanks so much for your concern and prompt reply, Squibble. Can't imagine a person would deligated his efforts to someones you don't even know and met. (…) Thanks for your help!! It means everything to me!"
• "Wow thanks so much! I have just been to the hot springs in Banos Ecuador, the volcano is rumbling and the town has evacuated but I am still here talking Apache, now that is dedication, I will test your helpfulness out 2moro! Thanks so much!!!"
• "THANKYOU THANKYOU THANKYOU! YOU ARE SOOOO AMAZING! MY BLOG IS GONE FROM GOOGLE AND ALL THE OLD POSTS ASSOCIATED WITH THAT TOO! THANK YOU! I LOVE YOU!"
• "It worked! Thank you very much for your help Cristina :)"
• "I could NEVER have seen what's possible without this forum. I am so grateful."
• "Thank you so much for your detailed response to my questions. In 10 years of me having a website, no one has explained these concepts better to me than you did."
• "IT WORKED!!! Thank you so much for saving me the grief and embarrassment of this problem. I truly appreciate both your knowledge and guidance."
• "Thank you so much for such a detailed answer and putting into terms I can easily understand. (…) Where shall I send the batch of brownies?"
• "Thank you so much Squibble you are a hero. I have done what it says and i will check to see if i appear in google! Thanks again!"
• "you guys are amazing! thank you so much redleg! (…) and if you happen to ever be in san carlos give me a shout - you deserve at least a beer and a lunch! "
• "Thank you Squibble, Vanessa, Cristina, and Ishigaki for weighing in on this and helping me. I hope I can pay it forward one day."
• "Great info Guys. I really appreciate it. It was awesome of y'all to help me out. I really appreciate it. Thank you."
• "Your amazing :-) i love you lol xx im sorted now thanks and never in a million years would i have found that out!"
• "THANK YOU! I love Google and appreciate that they have these safety precautions in place for those nasty hackets - especially when we can fix the problem! Thanks so much for your help. Whew."
• "Thank you very much, Robbo! With a little tweaking it worked perfectly!"
• "yay!!! I think it finally went through - THANK YOU SO MUCH!!!"

If you have a question that you would like to ask, a problem that you need help with, we'd love to see you in the forums! We just ask that you please take the time to read through our frequently asked questions, and search the forum before posting. Chances are high that a question like yours has already been answered. Tell us a little bit about yourself and then join us to learn more and help others!

Posted by John Mueller, Webmaster Trends Analyst

Webmaster Level: Intermediate to Advanced

To the fabulous, savvy audience that attended our Video Sitemap webinar several months ago, please accept our re-gift: a summary of your questions from the Video Sitemaps Q&A!

To those who were unable to attend the webinar, please enjoy our gift of the summarized Q&A -- it’s like new!

Either way, happy holidays from all of us on the Webmaster Central Team. :)

1. Can the source/content of the video (perhaps a third-party vendor) be hosted on another site? For example, can I host my videos on YouTube and still be eligible for Video Search traffic?
   
   Yes, you can use a third party to host videos. Only the play page--the URL within the <loc> tag--needs to be on your site. <video:content_loc> and <video:player_loc> can list URLs on a different site or subdomain.
   
   For example, here’s a snippet from a valid Video Sitemap that shows content hosted on a different subdomain from the play page:
   
   

   <url>
     <loc>http://www.example.com/videos/some_video_landing_page.html</loc>
       <video:video>
         <video:thumbnail_loc>http://www.example.com/thumbs/123.jpg</video:thumbnail_loc>
         <video:title>Grilling steaks for summer</video:title>
         <video:description>Alkis shows you how to get perfectly done steaks every time</video:description>
         <video:content_loc>http://video-hoster.example.com/video123.flv</video:content_loc>
         <video:player_loc allow_embed="yes" autoplay="ap=1">http://www.example.com/videoplayer.swf?video=123</video:player_loc>
       </video:video>
   </url>



2. If I’m using YouTube to host my videos, can Google verify that I’m the legitimate owner?
   
   Currently, there doesn’t exist functionality that allows you, as the uploader, to verify that you’re the owner of a video. The issue of authorship is a hard problem on the web, not just for videos, but nearly all types of content.


3. Because Google owns YouTube, should users who embed YouTube videos still submit Video Sitemaps or is it unnecessary?
   
   Google treats YouTube as just another source for video content -- though you don’t need to submit a Video Sitemap if you only want your YouTube-hosted videos indexed. If, however, you’re using YouTube as a online video platform (i.e., with play pages on your own site), then we do recommend Sitemap submission.


4. How long does it take for Google to accept and verify a Video Sitemap?
   
   Video Sitemap submission is a two-part process:
   
   
   
   1. We fetch the Sitemap and parse it for syntax errors. This happens within minutes.
   
   
   2. We fetch the assets referenced in the Sitemap, perform checks, validate metadata, do more cool stuff, and last, index the video. This step can require varied amounts of time depending on your site and our system load.
   
   
5. What tags and categories are most important in Video Sitemaps or mRSS? Should I create my own categories or is there a list that I should conform to?
   
   Currently, the most important metadata to include is title and description -- both are required. The category tag is optional, and there isn’t a list from which to select.


6. Do I have to use HTML5 to use Video Sitemaps?
   Does HTML5 help with discovery?
   Or, if my site is HTML5 compliant, do I still need to submit a Video Sitemap?
   
   None of the Video Search principles change with HTML5. We still recommend using a Video Sitemap regardless of the markup on your site. HTML5 can be helpful, though, because tags like <video> make it easier for our systems to verify that video exists on the page.


7. If I use an iframe rather than embedding my videos, can Google still find it?
   
   We do not recommend using iframes to embed video content on your pages.


8. Can I have multiple videos on one URL?
   
   You can. We’ve found, however, that users may not consider it the best experience. When users click on a video search result, they most often don’t like being forced to locate the correct video among multiple videos on the resulting page.


9. Do I need to specifically create a robots.txt file that allows Googlebot, or do I just need to make sure Googlebot isn’t blocked?
   
   Just make sure that Googlebot isn’t blocked.


10. I provided a thumbnail, but it’s not being used. Does Google create their own thumbnails from my videos?
    
    We try to use the thumbnail you provide if it’s valid. If not, we’ll try to generate a thumbnail ourselves. We recommend that you provide thumbnails that are at least 120x90 pixels. We also accept many thumbnail formats, such as PNG and JPEG.


11. Any video filesize limitations?
    
    At this time, there aren’t video filesize limitations on content submitted through VIdeo Sitemaps.


12. Is there any way to indicate a transcript or closed captioning for a video?
    
    Currently there isn’t, but perhaps down the road.


13. What if I’m using Lightbox or a popup to display a video; can it still be indexed?
    
    Depends on the use case and how it’s rendered, but if indexing by search engines is important to you, it’s not the safest method. In the Webmaster Help Center, we explain that “When designing your site, it's important to configure your video pages without any overly complex JavaScript or Flash setup.” Most often, for bots, simpler is safer.

Have a safe and happy holiday!

Written by Maile Ohye, Developer Programs Tech Lead

Webmaster Level: Intermediate to Advanced

What are the benefits of submitting feeds like Video Sitemaps and mRSS vs. the benefits of Facebook Share and RDFa? Is one better than the other? Let’s start the discussion.

Functionality of feeds vs. on-page markup

Google accepts information from both video feeds, such as Video Sitemaps and mRSS, as well as on-page markup, such as Facebook Share and RDFa. We recommend that you use both!

If you have limited resources, however, here’s a chart explaining the pros and cons of each method. The key differentiators include:

• While both feeds and on-page markup give search engines metadata, Video Sitemaps/mRSS also help with crawl discovery. We may find a new URL through your feed that we wouldn’t have easily discovered otherwise.


• Using Video Sitemaps/mRSS requires that the search engine support these formats and not all engines do. Because on-page markup is just that -- on the page -- crawlers can gather the metadata through organic means as they index the URL. No feed support is required.

If you’re further wondering about the benefits of specific feeds (Video Sitemaps vs. mRSS), we can help with clarification there, too. First of all, you can use either. We’re agnostic. :) One benefit of Video Sitemaps is that, because it’s a format we’re actively enhancing, we can quickly extend it to allow for more specifications.

All this said, if you’re going to start from scratch, Video Sitemaps is our recommended start.



“Starving” to start conversation about feeds or on-page markup? Join us in the Sitemaps section of the Webmaster discussion forum.

Written by Maile Ohye, Developer Programs Tech Lead

Webmaster level: Beginner

Webmaster Level: All

Just in time for the holidays, the Webmaster Tools team has updated the "Search queries" and "Links to your site" features.

Search queries with top pages:
Throughout the past year we’ve made some significant changes to the search queries feature in Webmaster Tools. We've received lots of feedback about this tremendously popular feature. One frequent request we heard was that people wanted to be able to see search queries data for their site’s individual pages. Well, we totally agreed that this would be useful and promptly set out to add this functionality to search queries. The fruits of our effort have finally ripened enough on the vine and are ready for you to enjoy. Now when you visit the search queries feature in Webmaster Tools you'll see a new tab titled "Top Pages." The "Top Pages" tab lists impression, click, and position data for the top pages on your site based on their performance in Google's search results.


If you click on one of the individual pages listed you’ll see a list of the queries driving traffic to that page, along with impressions and number of clicks for each query.


Just like in the "Top queries" view, you can click on a specific query to see more detailed data and evaluate how the query is performing across the whole site.


To make filtering in search queries even easier, we've added pie charts to show visually the proportions of search type, location and traffic. Also in the "Top queries" view, you can now specify “containing” or “not containing” when filtering queries.


Links to your site with intermediate links:
Links to your site now shows when a particular URL redirects. If there's a link to your site that links to URL1 which then redirects to URL2, we are now showing the link from URL1 to URL2 as an intermediate link. We also added a "Download all links" option to all the tables in the Links to your site feature.


Now that you know about these updates, please take a few minutes to check them out for your site. We hope it will provide you with a little bit of extra joy this holiday season. Let us know what you think by submitting a comment here or posting in our Webmaster Help Forum. Happy Holidays from the Webmaster Tools team!

Posted by Jonathan Simon, Webmaster Trends Analyst

Webmaster level: All

Do you know how Google's crawler, Googlebot, handles conflicting directives in your robots.txt file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot’s favorite song? The answers to these questions (except for the last one :)), along with lots of other information about controlling the crawling and indexing of your site, are now available on code.google.com:

Controlling crawling and indexing



Now site owners have a comprehensive resource where they can learn about robots.txt files, robots meta tags, and X-Robots-Tag HTTP header directives. Please share your comments, and if you have questions you can post them in our Webmaster Help Forum.

Posted by Jonathan Simon, Webmaster Trends Analyst

Webmaster Level: All

Recently we made a change to show more results from a domain for certain types of queries -- this helped searchers get to their desired result even faster. Today we’re expanding the feature so that, when appropriate, more queries show additional results from a domain. As a webmaster, you’ll appreciate the fact that these results may bring targeted visitors directly to the pages they’re interested in.

Here’s an example: in the past, the query [moma] (the Museum of Modern Art), might have triggered two results from the official site:


With this iteration, our search results may show:

• Up to four web results from each domain (i.e., several domains may have multiple results)
• Single-line snippets for the additional results, to keep them compact

As before, we still provide links to results from a variety of domains to ensure people find a diverse set of sources relevant to their searches. However, when our algorithms predict pages from a particular site are likely to be most relevant, it makes sense to provide additional direct links in our search results.


Like all the hundreds of changes we make a year, we’re trying to help users quickly reach their desired result. Even though we’re constantly improving our algorithms, our general advice still holds true: create compelling, search-engine friendly sites in order to attract users, buzz, and often targeted traffic!

Written by Harvey Jones, Software Engineer

Webmaster Level: All

We often get questions from webmasters about how we index content designed for Flash Player, so we wanted to take a moment to update you on some of our latest progress.

About two years ago we announced that through a collaboration with Adobe we had significantly improved Google’s capability to index Flash technology based content. Last year we followed up with an announcement that we had added external resource loading to our SWF indexing capabilities. This work has allowed us to index all kinds of textual content in SWF files, from Flash buttons and menus to self-contained Flash technology based websites. Currently almost any text a user can see as they interact with a SWF file on your site can be indexed by Googlebot and used to generate a snippet or match query terms in Google searches. Additionally, Googlebot can also discover URLs in SWF files and follow those links, so if your SWF content contains links to pages inside your website, Google may be able to crawl and index those pages as well.

Last month we expanded our SWF indexing capabilities thanks to our continued collaboration with Adobe and a new library that is more robust and compatible with features supported by Flash Player 10.1. Additionally, thanks to improvements in the way we handle JavaScript, we are also now significantly better at recognizing and indexing sites that use JavaScript to embed SWF content. Finally, we have made improvements in our video indexing technology, resulting in better detection of when a page has a video and better extraction of metadata such as alternate thumbnails from Flash technology based videos. All in all, our SWF indexing technology now allows us to see content from SWF files on hundreds of millions of pages across the web.

While we’ve made great progress indexing SWF content over the past few years, we’re not done yet. We are continuing to work on our ability to index deep linking (content within a Flash technology based application that is linked to from the same application) as well as further improving indexing of SWF files executed through JavaScript. You can help us improve these capabilities by creating unique links for each page that is linked from within a single Flash object and by submitting a Sitemap through Google Webmaster Tools.

We’re excited about the progress we’ve made so far and we look forward to keeping you updated about further progress.

Written by Jifeng Situ and Sverre Sundsdal, Software Engineers

Webmaster Level: Intermediate to Advanced

Today Google introduced Instant Previews, a new search feature that helps people find information faster by showing a visual preview of each result. Traditionally, elements of the search results like the title, URL, and snippet—the text description in each result—help people determine which results are best for them. Instant Previews achieves the same goal with a visual representation of each page and where the relevant content is, instead of a text description. For our webmaster community, this presents an opportunity to reveal the design of your site and why your page is relevant for a particular query. We'd like to offer some thoughts on how to take advantage of the feature.

First of all, it's important to understand what the new feature does. When someone clicks on the magnifying glass on any result, a zoomed-out snapshot of the underlying page appears to the right of the results. Orange highlights indicate where highly relevant content on the page is, and text call outs show search terms in context.

Here’s the Instant Preview for the Google Webmaster Forum.

These elements let people know what to expect if they click on that result, and why it's relevant for their query. Our testing shows that the feature really does help with picking the right result—using Instant Previews makes searchers 5% more likely to be satisfied with the results they click.

Many of you have put a lot of thought and effort into the structure of your sites, the layout of your pages, and the information you provide to visitors. Instant Previews gives people a glimpse into that design and indicates why your pages are relevant to their query. Here are some details about how to make good use of the feature.

• Keep your pages clearly laid out and structured, with a minimum of distractions or extraneous content. This is always good advice, since it improves the experience for visitors, and the simplicity and clarity of your site will be apparent via Instant Previews.
• Try to avoid interstitial pages, ad pop-ups, or other elements that interfere with your content. In some cases, these distracting elements may be picked up in the preview of your page, making the screenshots less attractive.
• Many pages have their previews generated as part of our regular crawl process. Occasionally, we will generate screenshots on the fly when a user needs it, and in these situations we will retrieve information from web pages using a new "Google Web Preview" user-agent.
• Instant Previews does not change our search algorithm or ranking in any way. It's the same results, in the same order. There is also no change to how clicks are tracked. If a user clicks on the title of a result and visits your site, it will count as a normal click, regardless of whether the result was previewed. Previewing a result, however, doesn't count as a click by itself.
• Currently, adding the nosnippet meta tag to your pages will cause them to not show a text snippet in our results. Since Instant Previews serves a similar purpose to snippets, pages with the nosnippet tag will also not show previews. However, we encourage you to think carefully about opting out of Instant Previews. Just like regular snippets, previews tend to be helpful to users—in our studies, results which were previewed were more than four times as likely to be clicked on. URLs that have been disallowed in the robots.txt file will also not show Instant Previews.
• Currently, some videos or Flash content in previews appear as a "puzzle piece" icon or a black square. We're working on rendering these rich content types accurately.

We hope you're as excited about this next step in the search results as we are. We're looking forward to many more improvements to Instant Previews in the future.

Posted by Jeremy Silber, Software Engineer

Note: The information in this post may be outdated. See our latest post about reporting spam.

Webmaster Level: All

At Google, we continually strive to improve our algorithms to keep search results relevant and clean. You have been supporting us on this mission by sending spam reports for websites that violate our Webmaster Guidelines, using the spam report form in Google Webmaster Tools. While you might not see changes right away, we take your reports seriously and use them to fine-tune our algorithms -- the feedback is much appreciated and helps us to protect the integrity of our search results. We also take manual action on many of these spam reports. A recent blog post covers more information on how to identify webspam.

For those of you who regularly report spam, or would like to do so, we’ve now published a Chrome extension for reporting spam that makes the process more convenient and simple. The extension adds “Report spam” links to search results and your Web History, taking you directly to the spam report form and autocompleting some form fields for you. With this extension, Google’s spam report form is always just one click away.

The Google Webspam Report Chrome extension provides further tools to help you quickly fill out a spam report:

• a browser button to report the currently viewed page
• an option to retrieve recent Google searches from your Chrome history
• an option to retrieve recently visited URLs from your Chrome history

As before, you need to be logged into your Google Account to report spam. You can find a more detailed walkthrough of the use cases and features in this presentation and on the Chrome Extensions Gallery page, where you can also provide feedback and suggestions. We hope that you find this extension useful and that you continue to help us fight spam.

The extension is available in 16 languages. If your Chrome browser is set to a language supported by the extension, it will automatically use the localized version, otherwise defaulting to English.

Note: We care about your privacy. The Google Webspam Report Chrome extension allows you to access your personal Chrome history for the purpose of reporting spam, but does not send data retrieved from it to our servers. The source code of the extension has been published under an open source license.

Posted by Manuel Holtz, Support Engineer, Search Quality team

Note: The information in this post may be outdated. See our latest post about reporting spam.

Webmaster level: All

Everyone who uses the web knows how frustrating it is to land on a page that sounds promising in the search results but ends up being useless when you visit it. We work hard to make sure Google’s algorithms catch as much as possible, but sometimes spammy sites still make it into search results. We appreciate the numerous spam reports sent in by users like you who find these issues; the reports help us improve our search results and make sure that great content is treated accordingly. Good spam reports are important to us. Here’s how to maximize the impact of any spam reports you submit:

Why report spam to Google?

Google’s search quality team uses spam reports as a basis for further improving the quality of the results that we show you, to provide a level playing field for webmasters, and to help with our scalable spam fighting efforts. With the release of new tools like our Chrome extension to report spam, we’ve seen people filing more spam reports and we have to allocate appropriate resources to the spam reports that are mostly likely to be useful.

Spam reports are prioritized by looking at how much visibility a potentially spammy site has in our search results, in order to help us focus on high-impact sites in a timely manner. For instance, we’re likely to prioritize the investigation of a site that regularly ranks on the first or second page over that of a site that only gets a few search impressions per month. A spam report for a page that is almost never seen by users is less likely to be reviewed compared to higher-impact pages or sites. We generally use spam reports to help improve our algorithms so that we can not only recognize and handle this particular site, but also cover any similar sites. In a few cases, we may additionally choose to immediately remove or otherwise take action on a site.

Which sites should I report?

We love seeing reports about spammy sites that our algorithms have missed. That said, it’s a poor use of your time to report sites that are not spammy. Sites submitted through the spam report form are reviewed for spam content only. Sites that you think should be tackled for other reasons should be submitted to us through the appropriate channels: for example, for those that contain content which you have removed, use our URL removal tools; for sites with malware, use the malware report form; for paid links that you find on sites, use the paid links reporting form. If you want to report spammy links for a page, make sure that you read how to report linkspam. If you have a complaint because someone is copying your content, we have a different copyright process--see our official documentation pages for more info. There’s generally no need to report sites with technical problems or parked domains because these are typically handled automatically.

The same applies to redirecting legitimate sites from one top level domain to another, e.g. example.de redirecting to example.com/de. As long as the content presented is not spammy, the technique of redirecting one domain to another does not automatically violate the Google Webmaster Guidelines.


The best way to submit a compelling spam report is to take a good look at the website in question and compare it against the Google Webmaster Guidelines. For instance, these would be good reasons to report a site through the spam report form:

• the cached version contains significantly different (often keyword-rich) content from the live version
• you’re redirected to a completely different domain with off-topic, commercial content
• the site is filled with auto-generated or keyword-stuffed content that seems to make no sense

These are just a few examples of techniques that might be potentially spammy, and which we would appreciate seeing in the form of a spam report. When in doubt, please feel free to discuss your concerns on the Help Forum with other users and Google guides.

What should I include in a spam report?

Some spam reports are easier to understand than others; having a clear and easy-to-understand report makes it much easier for us to analyze the issue and take appropriate actions. Here are some things to keep in mind when submitting the spam report:

• Submit the URLs of the pages where you see spam (not just the domain name). This makes it easy for us to verify the problem on those specific pages.
• Try to specify the issue as clearly as possible using the checkboxes. Don’t just check every single box--such reports are less likely to be reviewed.
• If only a part of the page uses spammy techniques, for example if it uses cloaking or has hidden text on an otherwise good page, provide a short explanation on how to look for the spam you’re seeing. If you’re reporting a site for spammy backlinks rather than on-page content, mention that.

By following these guidelines, your spam reports will be reproducible and clear, making them easier to analyze on our side.

What happens next?

After reviewing the feedback from these reports (we want to confirm that the reported sites are actually spammy, not just sites that someone didn’t like), it may take a bit of time before we update our algorithms and a change is visible in the search results. Keep in mind that sometimes our algorithms may already be treating those techniques appropriately; for instance, perhaps we’re already ignoring all the hidden text or the exchanged links that you have reported. Submitting the same spam report multiple times is not necessary. Rest assured that we actively review spam reports and take appropriate actions, even if the changes are not immediately visible to you.

With your help, we hope that we can improve the quality of and fairness in our search results for everyone! Thank you for continuing to submit spam reports and feel free to post here or in our Help Forum should you have any questions.

Written by Kaspar Szymanski, Search Quality Strategist & John Mueller, Webmaster Trends Analyst