Sitemap and Sitemap Generator
The sitemap in XML helps crawlers of search engines and and the HTML version helps users, for guidance as needed on the site.
Now, sitemaps are extended with image and video tags and even a set a tags that turn them into a RSS feed.
You can generate a site map with just a command and edit the generated document
from within the built-in viewer (or any text or XML editor), and then upload
it directly at the root of your site.
Finally you must register the map if the format is XML or text. The XML format is the standard from Google adopted by Yahoo and Live Search (Microsoft).
- Sitemap concepts.
- How to create the map of a website?
- Why to make a sitemap?
- XML, text or HTML, which format to choose?
- Sitemap formats.
- Sitemap index.
- Multiple contents in one sitemap.
- Tips, important advices about XML, text or HTML site maps.
- Validating the sitemap.xml file.
- Submitting the sitemap.
- The tool.
With the graphical interface, you have just to type the name of the home page and click on the "Generate" button!
A sitemap, either in the XML or HTML format, helps search engines to index
all pages of a website.
Additionally, Google, when an XML sitemap is registered, produces an analysis of problems encountered and a report of errors, and statistics.
Google returns which requests to search engines will lead to your pages and the pages that have not been indexed.
Simple Map, the screen
The XML format is now recognized by main search engines. It is intended to
give various information to Googlebot and other crawlers. This XML document
is generated by Simple Map according to the standard specification.
- The priority tag indicates which pages are the most important ones.
- The lastmod tag, gives the date of the last modification. Used along with the frequency attribute.
- The changefreq tag, is the frequency of scanning by the robot, from always, for a very big website with pages changing continuously, to yearly or never, for static pages. As for W3C specifications of formats with a version number.
The text format gives only the list of URL of pages to be indexed. It is accepted by Google.
The HTML format is for visitors of your website. It may display links, titles, descriptions or other infos. It is scanned by search engines and allows to give URL of pages that are not indexed, specially in the case of multi-level sub-directories, since deeper levels are not always scanned
The plain text or HTML files are simple list of URLs, but the XML format is made of tags corresponding to a standard format.
The container is urlset and it holds a set of url tags matching the pages of the site.
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://www.scriptol.com/</loc> <lastmod>2005-01-01</lastmod> <changefreq>monthly</changefreq> <priority>0.8</priority> </url> </urlset>
Images in sitemap
To have an image indexed by search engines, the format is as following:
More on the Center for Webmasters from Google.
Videos in sitemap
For your articles to be published on Google News, it must, in addition to the URL containing a unique ID, a specific sitemap. It is the standard XML sitemap with tags added.
In fact these tags are transforming the sitemap in a RSS file:
- <publication> is equivalent to channel. It includes the <name> and <language> tags.
- <access> with value "publication" for free access or "registration" for a limited access.
- <genre>, optional, is used to describe the type of article.
- <publication_date>, date and time of publication.
- <title>, title of the article.
- <keywords> optional.
- plus legacy sitemap tags for URL, weight ...
A News Sitemap should only contain articles published in the last two days.
View the News Sitemap format.
A sitemap index is a file holding a list of sitemaps. It allows if you have several maps or if the file is splitted into several file, to give the URLs.
You do not need to create an index if you have only one sitemap file and even if you have multiple sitemaps for different contents, they may be joined in a single file, as explained further.
The index file has a standard XML format too.
The container is sitemapindex and it holds a set of inner sitemap tags.
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap> <loc>http://www.scriptol.com/sitemap1.xml</loc> <lastmod>2004-10-01T18:23:17+00:00</lastmod> </sitemap> </sitemapindex>
Multiple contents in one sitemap
To cope with the proliferation of types of sitemap files, Google has decided to integrate all types of content in a single file.
The multiple file contents looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<video:title>Look growing the youngest.</video:title>
Thus three types of tags in the URL tag: loc for a page, image plus image:loc to an image file, and video with video:content_loc.
- The XML format is now supported at least by Google, Yahoo, Live Search, Ask...
- Don't use an XML sitemap if all pages of your site are already indexed at Google. (You can verify by typing in the search box site:www.sitexyz.com.)
- If some pages in your website are not indexed yet, give these pages a higher priority in the XML sitemap (the "priority" tag).
- To let a page ignored by crawlers of search engines you should use a Robots.txt file or the "ROBOTS" meta tag with value noindex.
- The sitemap is for the whole site. Don't create a map with only links of files that are not indexed by Google.
- You can omit optional tags (priority, lastmod, changefreq) if you are not sure you need for them.
- The time option (in lastmod tag) is for very big websites! Only date is useful in most cases.
- A sitemap with all pages same higher priority and faster frequency is of null interest for Google. Give to pages the lower priority and the slower frequency if they are already indexed and not frequently changed.
- For videos, a tag has been added to the sitemap protocol. See the tutorial for videos in sitemap from Google.
- You may create a HTML map for visitors and an XML map for robots of search engines.
- Put the link to the HTML sitemap on the home page.
- When a page is added to your site, it is not indexed immediately. You have to wait for days or for weeks before it becomes visible. Meanwhile search engine's robots scan your site daily, the database is updated from time to time (weeks or months).
- A RSS file is a valid sitemap for Google and other search engines, but
for recent pages only.
- An index may contain the URLs of 50 000 sitemaps each containing up to 50 000 Web page URLs.
To validate your XML site map, you need for a validator and two files:
- sitemap.xsd, the schema of the format, included in the archive
- sitemap.xml, the list of web pages, on your website or locally on your computer.
See resources for a validator.
The XML map file must stays at root of your site, belong the home page, index.html or index.php or other name.
You can submit it in three ways:
- Register at the site of the search engine.
- By a ping request.
- With the robots.txt file.
Create an account if you don't have one.
Google will provide you a test file to upload into your website, and once this is done, you have to return and click on the verify button... and forget them for a day before to return again for the results.
You can also submit the sitemap with a ping, see at "What do I do after
I create my Sitemap? " in the FAQ in resources below.
Once the sitemap is registered, when it is updated you don't have to register it again, you can inform the search engine that the file is modified by a ping:
Replace scriptol.com by the URL of your website, and google.com by the URL provided by the search engine: yahoo, ask, etc.
According to the blog of Google, you can now add a sitemap entry to the robots.txt file, and the sitemap is parsed when the robot of Google and other search engines encounters this file. the syntax is:
User-Agent: * Disallow: Sitemap: http://www.scriptol.com/sitemap.xml
The robots.txt file is stored at root of the website along with the sitemap file and the home page, index.html or other.
It is possible for the owner of several websites, to define in the robots.txt file of a site, the URL of sitemaps for each website, one per line. Référence.
User-Agent:* Disallow: Sitemap: http://www.scriptol.com/sitemap.xml Sitemap: http://www.scriptol.fr/sitemap.xml
The program parses recursively the content of a website, from the main page
to each page that is linked, and builds a list of all pages to be indexed
by search engines. A list of extensions in the source allow to select the
type of files to index.
The program works locally offline on the image of the site for now. There is a lot of websites that offer you to build the sitemap of your Internet site, directly online.
The source code is included in the archive. This is a Scriptol
program, and it is very clear and short, thanks to the text processing functions
of the Scriptol programming language.
You can compile the source to PHP or C++, or binary.
The source of the graphical user interface is furnished free of charge also.
- Full specification of the XML standard.
- Validator. Check your sitemap for validity.
- Sitemaps.org - Official website of the standard, common to Google, Yahoo, Live Search.
- Robotstxt.org. More infos about robots.txt and indexing of your website.
- Video Sitemap. Videos can be indexed too.