How to Redesign Your Site Free of SEO Risk

by Anvil on November 17, 2014Search Engine Marketing

Congratulations on your impending new site! Given the furious pace at which design and usability best practices evolve these days, there’s hardly any doubt that a site redesign is a good move, no matter what business you’re in.

Here’s the thing, though: if your bottom line depends on the organic search traffic that your current site pulls down (and chances are overwhelmingly high that it does), you can’t go moving things around on your site without potentially putting that traffic at risk.

For instance: I’d bet you’re planning to change some of your site’s URLs, right? What do you think would happen if you took a bunch of key pages on your site — pages that have earned high-quality backlinks and thus boosted your whole domain’s search visibility — and changed the URLs one day? I’ll tell you: Google wouldn’t know that the URLs had been changed; it would see only that those old URLs were no longer on the site and that some new URLs had been added, and would make no connection on its own between the old ones and the new ones. Consequently, all the referral traffic and all the SEO authority that the old pages’ links had conferred to your site would vanish as soon as the index turned over.

Or: what if, when you made the leap from wireframes to a staging site and started getting your new site’s HTML in order, Google came around and crawled and indexed the staging site while it was still under construction? If that were to happen — and it would, if you didn’t take the proper precautions — you’d have a second site (and one decidedly not ready for primetime) sitting next to your real, not-yet-replaced site in Google’s index, and since both sites presumably make the same prolific mention of your brand name, Google would put your staging site into competition with your real site for the same pool of traffic. Also not good.

What you need is an overview of best practices for site redesign SEO, and this post is just that. It will teach you the SEO measures that are critical to each designated stage of development, and empower you to make site changes as dramatic as you like without any probability of losing your precious organic traffic.

During Development

While your new site is being developed, it must be hidden from search engines so that it doesn’t start showing up in search results while still under construction, thereby polluting your brand’s presence in search. Full “hiding” requires two steps: one to prevent crawling, and another to prevent indexation. These are carried out as follows:

To prevent crawling, create a robots.txt file for the site and compose it of the following two lines:
- Make sure that you name this file robots.txt and host it in your dev site’s root directory, so that it will be found at the following (example) URL: https://dev.mysite.com/robots.txt
To prevent indexation, add the following tag to the <head> section of every page on the dev site:

Additionally, there are certain metrics of SEO success on which your old site’s scores should be formally noted somewhere as benchmarks against which to test the SEO health of the new site. These are as follows:

Domain Authority. Visit https://moz.com/researchtools/ose and enter your main domain’s URL. Record the site’s Domain Authority score. This crawler’s index turns over every four to five weeks, so check back four to five weeks after your new site’s launch to see whether the Domain Authority score has moved.
Top Pages. This same Open Site Explorer dashboard contains a tab called “Top Pages” that allows you to see which pages on your current site have the highest Page Authority scores. These are, objectively speaking, the pages most important to your site in search. You’ll want to make a record of them now so that you can visit them after your new site’s launch and verify the proper functioning of your 301 redirects (more on those to follow in the next section). This is most easily done by requesting a .csv export and filing it away somewhere safe.
Rankings. It is advised that you record your old site’s organic search rankings on the keywords most important to your brand so that you can easily determine later how your site redesign has affected them.

Just Before Launch

Just before you push your new site live, you’ll need to take several measures to ensure that it will be fully search-ready upon launch, and that all of the SEO authority accrued by your old site will successfully transfer to the new one. These measures are as follows:

Keep your robots.txt file in place, but remove the “Disallow: /” instruction.
This instruction had previously been used to keep your entire dev site from being crawled, but naturally, we’ll want the site to be crawled after launch. If there are sections of the site that genuinely do not serve your search audience and would be a waste of crawlers’ resources, however, then this Disallow instruction can be used to indicate these sections. See https://www.robotstxt.org/orig.html#format for full instructions on how to format a robots.txt file.
Add a sitemap autodiscovery line to your robots.txt file. There is one crucial addition that should be made to a robots.txt file immediately before the new site launches: a line at the bottom informing crawlers of the location of your XML sitemap. This line should read as follows:

Note that you must use an absolute, and not a relative, URL here; this can be confusing because it inverts the formatting standard for Disallow instructions, but it is the case.

Remove the meta robots noindex, nofollow tag from all pages as well. As with the Disallow instruction on the robots.txt file, this tag can still be used on any pages that you feel do not serve your search audience and wish to keep out of search results. However, when using this tag on the live site, it is recommended that you amend the “nofollow” instruction to read “follow”, so that search engines know to look at the pages linked to from the page in question, even if they are not meant to index the page itself. Thus, the version of this tag that should be used on the live site, if at all, reads:
Set up 301 redirects from every URL on your old site to whichever URLs on the new site correspond most closely. In the case of URLs that did not change at all between the old site and the new, this will not be necessary; however, for URLs that did change and/or URLs of pages from the old site that have been slated for deletion, a 301 redirect must be set up from the old URL to whichever URL on the new site represents the best content match. This will not only help returning visitors find their way back to the most current versions of any pages that they have bookmarked, but will help search engines connect your old site to your new site on a page-by-page basis. That latter benefit also carries an explicit SEO perk: when search engines see a 301 redirect, they interpret it as an instruction to transfer all link equity and attendant authority from the old URL to the new.

Just After Launch

Immediately after launch, once the new site’s basic functionality has been verified, take the following measures to tip off the search engines and keep your own house in order:

Verify that the site can be found by Google. This is as simple as visiting Google Webmaster Tools, selecting “Crawl” from the sidebar menu, selecting “Fetch as Google”, and clicking “Fetch” (which, by default, will retrieve the homepage).
Get the site indexed. The quickest means to this end is to write an announcement of your new site launch and post it on your company’s Google+ page, with a link to the new homepage. Google+ posts are indexed almost immediately, and pages linked to from Google+ posts are typically indexed within 10 hours. Send the search engine spiders to your homepage and they’ll quickly find your XML sitemap; soon enough, the whole site will be in the index.
Check your analytics tracking and add an annotation. Visit your Google Analytics dashboard and check the real-time report to see whether you currently have visitors on your new site. This is the surest way to confirm that tracking is working. Once tracking is confirmed, add an annotation indicating the date of the new site launch. This will be a useful reference for the next year or more of reporting.

What to Expect

Even with the benefit of all the above precautions, it is all but certain that a redesign will temporarily disrupt your site’s rankings, and most likely your organic search traffic as well. This is because your presence in the search engine indices will have to be refreshed wholesale; even if you’ve done everything in your power to help search engines connect the authority of your old site to the content of your new one, the process by which they make this connection can take several weeks, especially if the transition entails the addition and/or subtraction of a large number of pages. The essential message is: once you’ve verified that your robots.txt and meta robots tags are in order, your 301 redirects are working properly, and your metadata is up-to-date, your redesign work is done, but this does not mean that your SEO work on the whole is done. The tasks of creating and nurturing online communities, writing content of substance and value, optimizing your public relations messages, and amplifying earned media all fall under the SEO umbrella today, and these tasks are never complete so long as your business still exists. In view of this larger context, a business redesigning its website is akin to an airline renovating the concourse and lounge at its principal airport: a significant gesture for improving customer experience, but not the game itself. The game never ends!