Pages Not Indexed By Google Because Of Autofresh

Pages Not Indexed by Google Because of Autofresh

About a year ago, I decided to add an autorefresh to pages that contain feeds so that content would always stay fresh. I set it to autofresh every 3 minutes. I could have been done this via ajax, but my ajax kung fu wasn’t good enough then (or now). I went with the easiest method, which is a simple meta tag addition to those pages. I got the idea from Drudge Report.

It was fine for about a year until the week of the Panda Update 2.2, when all pages with the autorefresh would give the Googlebot errors and not get indexed.

I only discovered it by accident while Googling info about Panda. It’s not related, but because I was losing traffic, I was wondering if pages were being indexed. You can find out by searching Google for “,” click on “show search tools” at the bottom of the left sidebar, and then “Past hour.” That will show the number pages from your site that’s indexed in the past hour. For my site, it was just my blog and the main channels. None of the featured headlines and teasers from the main sections were indexed.

That affected traffic, of course, because less pages on Google means less search referrals.

I went into Webmaster Tools and started looking for crawl errors. There were lots of “not followed” errors. I learned that one of the causes for those errors are autorefresh meta tags.

As soon as I commented it out, new pages started appearing on Google’s index within minutes.

I got a slight traffic boost from this, but is still down overall because of Panda.


URLs not followed errors > Redirect error

Sections:  note from the editor   
Topics:  autorefresh   coding   google   google index   html   panda update 2.2   search engine   seo   site updates   wopular   
  • Wait, Business Insider Does Sports Now?Wait, Business Insider Does Sports Now?
    I didn’t realize Business Insider does sports, until I saw a link to an article on Yahoo’s homepage titled “LeBron James referenced a 4-month-old Kevin Love quote in a tweet, and the fallout is getting absurd.” More
  • Broken Feeds: Is RSS Going the Way of the Dinosaurs?Broken Feeds: Is RSS Going the Way of the Dinosaurs?
    I’m been fixing a bunch of broken feeds since the new year. It started with CNN, which launched a redesign at the beginning of the year. At the time, all of their RSS feeds were broken as a result of that (more about this later). More
  • Finally Upgraded Sites to Use New Hardware from Linode Cloud: SSDs, Double RAM, Faster CPUsFinally Upgraded Sites to Use New Hardware from Linode Cloud: SSDs, Double RAM, Faster CPUs
    Around April, my VPS provider Linode announced that they’ve just finished a major $45M upgrade to their hardware and network, which means free upgrades for their existing users. The new hardware uses SSD RAID drives and faster CPUs. I hadn’t bothered because Wopular and Movies With Butter have been relatively stable. More
  • The Identical Look of Firefox and Chrome Caused Site ShutdownThe Identical Look of Firefox and Chrome Caused Site Shutdown
    In my previous article, I mention how the latest version of Firefox has a nearly identical look to Google’s Chrome web browser. When I’m working on new features for my sites, I use Firefox for testing and Chrome to see what the results look like to the users. I knew one of these days, because both browsers look the same now, I would confuse one for the other and put up untested programming code that would cause havoc on my sites. More
  • Firefox 29: Looks Just Like Chrome NowFirefox 29: Looks Just Like Chrome Now
    I just upgraded to Firefox 29, the latest version. Well, it’s more like it upgraded itself. Although I like the newer cleaner look, it’s harder and harder to tell the difference between it and Chrome, except the latter is still seemingly snappier. Firefox 29 has buttons at the top left for both bookmarking and access to the bookmark list. More
  • Google's Panda Update Took a Big Bite Out of My Traffic
    Google have been tweating their search algorithms this year to battle content farms and scrapers. They named this tweak Panda Update. In February of this year, they released the first version, which was aimed at lowering the rankings of content farms - i.e. More
  • Don't Put Google Analytics at the Top
    A new tracking code from Google Analytics was released recently. GA is a free web traffic reporting tool that let's you track how many users are viewing your website. The new code snippet uses an asynchronous process, meaning your site would load up without having to wait for the tracking code to finish executing. More
  • Link Removal Part II: I'm Still Not Removing Your Link, But ...Link Removal Part II: I'm Still Not Removing Your Link, But ...
    About a year ago, I wrote about getting daily requests/demands/threats to remove links to random sites on Wopular. I didn’t add those external links myself. They’re all done automatically through RSS feeds. If another site’s content is related to a headline on Wopular, then a link to it is generated. More
  • Google's New Search Algorithm Change Took Out 40% of My Google Traffic
    Last Thursday, Google launched a new tweak to their search algorithm. It was targeted at content farms, which many consider to be spammy sites whose content are created solely to attract search engine traffic for specific keywords. I do notice more and more of these sites on Google’s search results. More
  • Want Your Articles on Google Instantaneously? Digg it.
    I'm just starting to submit stories from my blog to Digg, hoping against hope that they'll get some Digg love and drive some traffic back here. So far, no luck. The only Digg love my blog entries get is from me - and ONLY me. More


Comment On This Story