XML Sitemaps are an easy way for webmasters to tell search engines about pages on their site that may be crawled by their robots.
A typical XML Sitemap file lists each URL, together with information about when it was last updated, how often it typically changes, and how important it is, relative to other pages in the site). This helps search engines to more intelligently crawl your site.
In November 2006, Google, Yahoo! and MSN joined forces to sustain a new industry standard for sitemaps: Sitemaps 0.90. As long as webmasters follow the protocol, they can ensure their sites are fully and consistently indexed across all the major search engines (a real step forward). This article is important for all those with missing or poorly ranked pages.
The official site for the joint venture is at sitemaps.org and contains a lot of info about the new standard and it’s syntax. What the site particularly fails to do is explain correctly how to submit your sitemap to the big three! The format suggested on the site of:
does not currently work at any of the three sites! Until it does, this short article provides instructions for how to (a) create your sitemap and (b) how to submit to each of the three main search engines…
Creating your Sitemap
Some hosting providers (for example 1and1) provide utilities via their web control panel, to create your sitemap, so you should always check with your provider first. If this service is not obtainable, then make a visit to xml-sitemaps.com and go into your site URL into the generator box. Copy-and-paste the resulting sitemap into notepad, then save-and-upload to your site with the file name: sitemap.xml
If you want to validate the XML prior to uploading to the search engines (useful if you have made any manual amendments), look at the XML validator (at the same site) where you can put in the URL of your sitemap and check it against the standard.
Submit sitemap to MSN
MSN have in addition to implement a formal interface for Sitemap submission (as at July 2007). To monitor the situation, please visit (now and then) the MSN Official Livesearch Blog (where where future announcements are likely to be found).
Whilst MSN have in addition to implement a front door, there is a recognised back door for submitting your sitemap to the MSN Search index; namely additionally.com! You should use the following syntax directly in your browser URL box:
Since February 2005, additionally.com have been the official provider of RSS feeds to the myMSN portal (see press release) and reliable evidence indicates that submission to additionally will consequence in MSN spidering your pages within 2-3 weeks.
observe that, whilst MSN nevertheless do not sustain direct submission, they do suggest on their blog that you add a reference to your Sitemap into your robots.txt file (something now supported by sitemaps.org). For example:
This would tell MSN (and all other engines) to crawl your sitemap file but not to crawl your cgi-bin directory. For more info on how to implement a robots.txt file (in the root of your site webserver) please visit: http://www.robotstxt.org
Submit sitemap to Google
Google originally developed the XML schema for sitemaps and have developed a dedicated portal for webmasters, from where you can submit your sitemap:
First, you need to tell Google all the sites you own, then verfiy that you indeed own them. The verifiaction is achieved by adding a metatag between the head tags on your site homepage. The syntax for the tag is as follows:
There are complete instructions on how to do this on the Google site.
Submit sitemap to Yahoo
Yahoo follows a similar approach to Google. Again, there is a dedicated service for webmasters (Yahoo! Site Explorer) and a procedure for verifying your ownsership of the site. First go to:
Add a site, then click on the verification button. You can then download a verification meaningful html file – which you will need to upload to the root directory of your webserver. Then you can return to Site Explorer and tell Yahoo to start authentication. This will take up to 24 hours. At the same time you can also add your sitemap by clicking on the manage button and then adding the sitemap as a satisfy.
Submit sitemap to Ask
Ask follows a simpler approach to the other three. To submit you sitemap, you simply go into a ping URL, followed by the complete URL of where your sitemap is located:
After clicking return, you will get a reassuring message from Ask that they have received your submission. Very neat!