Digital Marketing Agency | Portland PPC SEO Services | Anvil Media -
  • Services
    • Paid Media
    • SEO
    • Social Media
    • Email Marketing
    • Website Development
  • Outcomes
    • Clients
    • Portfolio
    • Case Studies
    • Testimonials
    • Awards
    • Industry Expertise
  • Insights
    • Blog
    • Articles
    • Cheat Sheets
    • White Papers
    • eBooks
    • Podcasts
    • Webinars
  • About
    • Our Philosophy
    • Our Culture
    • Executive Team
    • News & Events
    • Careers
  • Contact
    • Contact Us
    • Anvil Newsletter
  • Twitter
  • Facebook
  • Linked in

Controlling how search engines access and index your website

by Kent Lewis on February 1, 2007Uncategorized

Controlling how search engines access and index your website

by Kent Lewis on February 1, 2007Uncategorized

I’m often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and non-searchable? The good news is that those who publish on the web have a lot of control over which pages should appear in search results.

The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels — the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.

Read Full Article >>

  • Contact Us
    Contact Us
  • Search
  • Recent Posts
    • 5 Reasons Why Your Website Needs a BlogMarch 4, 2021
    • Advertising with Apple iOS 14 – What You Need to KnowFebruary 24, 2021
    • A Guide to Google Ads Bid Strategies: Best Practices to Use with Your CampaignsFebruary 19, 2021
    • How COVID-19 Changed our Online BehaviorFebruary 11, 2021
  • Categories

Related Posts

  • A Guide to Google Ads Bid Strategies: Best Practices to Use with Your Campaigns
  • Prepare for Google’s Newest Ranking Signals – Core Web Vitals & Page Experience
  • 11 Tips to Improve Your Presentation Skills
Ready to get started?

Take the next step toward growing your business. Tell us a little more about your goals and schedule an appointment.

Talk to Us

Blog

  • 5 Reasons Why Your Website Needs a Blog by Mike Stormberg on March 4, 2021
  • Advertising with Apple iOS 14 – What You Need to Know by Jared Carpenter on February 24, 2021

Twitter

Anvil Media, Inc. @anvilmedia

RT @kentjlewis: Why incumbent brands prefer @Instagram #Reels over @TikTokhttps://t.co/B5YzgeOK4M

ReplyRetweetFavorite

Testimonials

Anvil media has been instrumental in helping Icon Time execute on our eCommerce GTM strategy. From responsive account management to ...Icon Time

©Anvil Media, Inc. 2021
310 NE Failing St. Portland, OR 97212

  • Services
  • Outcomes
  • Insights
  • About
  • Contact
  • Legal
  • Facebook
  • Twitter
  • Linked in
  • Instagram
  • Youtube
  • Pinterest