Don’t Let Google Crawl Your Secure Pages – FIFA Case Study
Over the course of the day I may use one of 3 browsers fairly often to have a look at any manner of website. Sometimes I’m simply frustrated at the memory sucking of Firefox or confused as to why there are 12 Google Chrome sessions running with one browser window open with 3 tabs. Whatever the reason I seem to always navigate back to Chrome. However when FireBug is calling, it’s Firefox that wins over hands down. Firebug for Chrome is virtually useless. Perhaps I’ll get more into that sometime. If you use it you know what I mean.
Anyhow, I was checking out the “Home Grown Quota” rule for the English football league (yes, the football which is actually played with the FEET) and while scanning Google’s search results I noticed a link to FIFA.com. I felt this was a very high authority site and clicked the link. With great surprise I was confronted with the following Google Chrome nag screen.
I was naturally surprised at first given that the site in question is probably the most authoritative Football related site on the planet. Only when I looked a little closer did I clue into why this error popped for me.
At first glance it might seem as though Google Chrome is picking on the Akamai Network (my initial reaction). The Akamai Network is probably the largest global content distribution network around and allows for global proxying of very large websites. Wanna learn more, check out Akamai.com. Well this isn’t the reason why this link from the search engine will probably result in a bounced visitor. The reason is that FIFA is duplicating their website by allowing Google to crawl their HTTPS pages. This is a bad thing.
Secure Pages in Google’s Index
If you notice the Address Bar, this was a link to the secure (HTTPS) fifa site. Someone in the Web department screwed up pretty big here as there should be no HTTPS files listed in Google’s search index. This link was simply to a news item that I wanted to read up on, not some super secure sign up form or some other type of form where a secure connection would be appreciated. Nope, this was a cock up by the webmaster or masters.
Why is this Wrong?
Well, there are actually quite a few topics here that I could go on in detail about but I will stick to the core issue, which is the access to the secure pages. This is a common mistake by website administrators who are not receiving the correct feedback from their internal SEO or Internet Marketing department. You’re website should not be accessible via HTTPS except for the pages which require a secure connection, I.E. Forms. If you can type in the URL, rest assured Google will most likely find the pages. Secure pages also carry a specific Secure Certificate which is installed at the server using a specific Top Level Domain (https://www.fifa.com). The browser must find that domain during the HEAD request or it will return a security error. In this case, with FIFA using the Akamai network, the first hop was an Akamai proxy server. Booya! So that’s why Chrome appeared to crumble at the sight of the Akamai URL. Because the Search Engine Result was an HTTPS FIFA page, it invoked the Security Certificate and ceased once it did not immediately find the Associated URL (https://www.fifa.com), and instead found an Akamia jump off point.
Yikes, I hope that made sense.
Okay Man, What do I do now that You’ve Taken the Piss out of my Website Set up?
So in a nutshell, don’t allow your secure pages to serve unless absolutely necessary.
Return users to the ‘non-secure’ website once their transaction is complete (form filled out, product purchases).
Apply ROBOTS NOFOLLOW tags to the entire HTTPS branch of the website.
Set up a separate ‘Web Instance’ for use with the Secure side of the site which allows you to apply unique settings against the files (restrictions to folders, files, etc.).
Add a 301 moved permanently redirect on all HTTPS pages which should not be secured.
Fifa will lose a large amount of this visiting traffic because somewhere down the road this news items was allowed to propagate across the net from the Secure portion of the website and was subsequently linked to from all over the net.
Latest posts by Jade (see all)
- Caorda Time Lord – A Forgotten Relic of the Time War - February 9, 2017
- Sage Stomp 6 Hour Mini Rogaine – Gardens Creek Ranch - March 17, 2016
- Ankles and Ultras - February 3, 2015