Pages Not Indexed By Google Because Of Autofresh

About a year ago, I decided to add an autorefresh to pages that contain feeds so that content would always stay fresh. I set it to autofresh every 3 minutes. I could have been done this via ajax, but my ajax kung fu wasn’t good enough then (or now). I went with the easiest method, which is a simple meta tag addition to those pages. I got the idea from Drudge Report.

It was fine for about a year until the week of the Panda Update 2.2, when all pages with the autorefresh would give the Googlebot errors and not get indexed.

I only discovered it by accident while Googling info about Panda. It’s not related, but because I was losing traffic, I was wondering if pages were being indexed. You can find out by searching Google for “,” click on “show search tools” at the bottom of the left sidebar, and then “Past hour.” That will show the number pages from your site that’s indexed in the past hour. For my site, it was just my blog and the main channels. None of the featured headlines and teasers from the main sections were indexed.

That affected traffic, of course, because less pages on Google means less search referrals.

I went into Webmaster Tools and started looking for crawl errors. There were lots of “not followed” errors. I learned that one of the causes for those errors are autorefresh meta tags.

As soon as I commented it out, new pages started appearing on Google’s index within minutes.

I got a slight traffic boost from this, but is still down overall because of Panda.


URLs not followed errors > Redirect error

Sections:  note from the editor   
Topics:  autorefresh   coding   google   google index   html   panda update 2.2   search engine   seo   site updates   wopular   
  • Moved SimplePie File Caching to DatabaseMoved SimplePie File Caching to Database
    A while back, My ISP told me my backups took half a day to complete, which is too long. If anything goes wrong during that period, the process would have to start all over again, which happened quite often. The customer rep told me that my server had too many files, and even though they didn’t take up much disk space when added up, the process of copying each one of them takes a lot of time. More
  • Pixlr’s Online Image Editor, When All You Want To Do Is Resize An Image
    I was looking for a Chrome app that would allow me to resize images and came upon Pixlr’s online image editor. Open up their website on any browser (I’ve only used it on Firefox and Chrome), and you get a fast simpler version of Photoshop online. More
  • Bet On HTML 5 Kept Facebook from Entering Mobile Market Early
    I was reading an article on BBC about Facebook’s future and came upon the reason why the social network got into the mobile market so late. BBC has the answer from Facebook’s COO Sheryl Sandberg: More
  • Looks Like Facebook Home Is Not For Me
    There’s been lots of news lately regarding Facebook Home. I use Facebook on a daily basis, but I’m not sure if I want them to take over my phone. I don’t even want Google to take over my phone; the first thing I did with my new phone was get rid of their search bar. More
  • Got the Galaxy Note 2Got the Galaxy Note 2
    Ever since Samsung came out with the Galaxy Note, I was already interested. It has a 5.3” screen and a pressure-sensitive stylus. Because I use my phone mostly for web browsing, the larger screen allows me to see more of a website without zooming in and out. More
  • Google's Panda Update Took a Big Bite Out of My Traffic
    Google have been tweating their search algorithms this year to battle content farms and scrapers. They named this tweak Panda Update. In February of this year, they released the first version, which was aimed at lowering the rankings of content farms - i.e. More
  • Don't Put Google Analytics at the Top
    A new tracking code from Google Analytics was released recently. GA is a free web traffic reporting tool that let's you track how many users are viewing your website. The new code snippet uses an asynchronous process, meaning your site would load up without having to wait for the tracking code to finish executing. More
  • Google's New Search Algorithm Change Took Out 40% of My Google Traffic
    Last Thursday, Google launched a new tweak to their search algorithm. It was targeted at content farms, which many consider to be spammy sites whose content are created solely to attract search engine traffic for specific keywords. I do notice more and more of these sites on Google’s search results. More
  • Want Your Articles on Google Instantaneously? Digg it.
    I'm just starting to submit stories from my blog to Digg, hoping against hope that they'll get some Digg love and drive some traffic back here. So far, no luck. The only Digg love my blog entries get is from me - and ONLY me. More
  • Sorry, I Don't Have Time to Work on Your SEO: Backlink RemovalSorry, I Don't Have Time to Work on Your SEO: Backlink Removal
    Ever since Google released its Penguin update last year, which penalizes sites for being linked to by low quality sites, I’ve been getting a regular stream of link removal requests. (Yeah, I know this implies that my site is low quality, but I’ll let that go.) More


Comment On This Story