What did that Google Webmaster message mean?

Channel Digital

Written by Channel on .

User Rating:  / 0

For those website owners using Google Webmaster tools – you really should do – being both free, and a source of important information about how to improve your website.  

Here it is: https://www.google.com/webmasters/tools

Website owners may have noticed recently notifications coming out from Google relating to make the JavaScript and CSS files indexable.  The actual message is:

“Googlebot cannot access CSS and JS files on http://www.yourwebsitename.com

To: Webmaster of http://www.yourwebsitename.com/

Google systems have recently detected an issue with your homepage that affects how well our algorithms render and index your content. Specifically, Googlebot cannot access your JavaScript and/or CSS files because of restrictions in your robots.txt file. These files help Google understand that your website works properly, so blocking access to these assets can result in sub-optimal rankings.”

So although this looks technical it is really quite straightforward.

A file in your website called “Robots.txt” tells the search engines which parts of the site to index, and which to ignore. 

Here is an example of the file, from  this website, so you can see what I mean: http://www.channeldigital.co.uk/robots.txt

In the case of a modern CMS website like Joomla, WordPress of Drupal, all of its workings are in subfolders in the web space, folders like: “administrator”,  “bin”, “cache”, “components”, “libraries” and so on.  Now none of the actual content of your website is in these folders so you don’t need Google to index them.  In fact its best they don’t, so in the robots.txt file some lines say simply “don’t go there”, like this:

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/

However SOME of the files are helpful for Google to read. Particularly JavaScript and CSS relate to how the page is displayed, and with the ever growing importance of mobile and responsive websites, it is these files which work out what type of device the visitor is using, and adapts the display to suit.  For example a responsive template will resize itself according to the viewer’s device. 

Given that Google have announced they are building the “Mobile friendliness” of  a website into their search algorithm, it follows that they need see the JavaScript and CSS files, in order to understand how mobile-friendly the website is.

The change required is a fairly simple one for a web designer, clients will either need to have FTP or C-panel access to the web site files, or if they are on our servers just let us know and we’ll be able to make the changes.

To summarise however, the lines to add to the robots.txt file, rather than “Disallow” are simply “Allow” lines instructing the Googlebot that it CAN access the relevant folders.  But which folders should you give it access to?

Well here Google Webmaster Tools comes to you rescue again, because there is a feature in there called “Fetch as Google”.  This grabs and renders the web page, and tells you if there are any resources it needs to render the page, which are blocked to it.  By resources we mean these very CSS, Javascript, and image files that we would like Google to index.

So by reading the list that Google gives us of “Blocked Resource s”, we are able to compile a comprehensive list of the folders that we must Allow access to.

Here it is for one particular website that we manage.

Allow: /templates/mytemplate/
Allow: /templates/mytemplate/css/
Allow: /templates/mytemplate/images/
Allow: /images/
Allow: /media/com_acymailing/css/
Allow: /media/system/js/
Allow: /plugins/content/avreloaded/
Allow: /plugins/content/avreloaded/
Allow: /plugins/system/modalizer/modals/

So these lines can be added to the end of the Robots.txt file, test again with the “Fetch as Google” feature,  and before we know it the waring message from Google has gone away, and Google is developing the understanding that it needs of the mobile-friendliness of the website.

Now apologies to all you non-tecchie website owners out there, I appreciate that this can be terrifying.  Do refer this to your web designer though, they will know what to do, and if Channel are involved in your website please give us a call, we will be very happy to make the required changes.

We use cookies to improve our website and your experience when using it. To find out more about the cookies we use and how to delete them, see our Privacy Policy.

I accept cookies from this site
EU Cookie Directive plugin by www.channeldigital.co.uk