Bandwidth Stealing
June 22nd, 2004
The Problem
Recently, a fellow computer technology website warned a number of us that a website in China was stealing content from a number of other review sites. In fact, not only were they stealing the textual content of our reviews, this web site was also stealing the valuable bandwidth that we use to deliver information to the viewing audience. How you might ask? The web pages that this foreign site served up had HTML tags referring back to our website and hence we were still delivering the image content. Suffice it to say that we were a bit peeved!
The Solution
After some digging around, a number of users recommended using an Apache mod to fix the problem. The solution involves creating or modifying the .htaccess file that can or is stored in a specific directory on your server. The mod in question is the mod_rewrite module and is usually distributed with all Apache distributables.
The mod_rewrite module is able to intercept incoming URLs and modify them according to a set of rules that you specify. What we are interested in doing is to change an incoming request for an image to instead either:
- deliver no image or
- deliver a predetermined image condemning the act of theft
To make this happen, you can use the mod_rewrite module to inspect the incoming HTTP header. The field we're looking for is the referer field - or basically the URL that the current request originated from. Now this isn't always populated but in the case of images - usually a web page is where you will find a reference to an image.
As an example: if http://www.domain.com/index.html has a <img src="http://www.domain.com/blah.jpg"> tag within the HTML source code and a web browser reads the index.html file - the browser will also make a request to the domain.com webserver for the blah.jpg file. The HTTP request will have a referer field populated with http://www.domain.com
What all of this means is ... since you can usually figure out who is making the http requests for an image, you can selectively pick who can see the images and who can not. There are two ways usual ways of approaching this.
The first way is to just show a broken image if the site that is making the request is not the site you intended.
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?domain.com(/)?.*$ [NC]
RewriteRule .*\.(gif|GIF|jpg|JPG)$ - [F]
The secone way is to show a different image than the image requested:
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?domain.com(/)?.*$ [NC]
RewriteRule .*\.(gif|jpg|jpeg|bmp)$ http://domain.com/alternate.jpg [R,NC]
So pick either option and store the text into an .htaccess file. If you've never created an .htaccess file, just create a blank text file and then copy the source text above (picking only one of the options) and then paste it into the text file. Save the file as .htaccess and then upload it to the webserver. Wherever you place the .htaccess file, all resulting subdirectories will be affected. If you want this to work on a global scale, place the .htaccess file in the root folder of your website.
Keep in mind that this only works for Apache web servers with the mod_rewrite module. We applied the fix to our webservers here at WhiningDog and we're happy to say that no more bandwidth is being stolen by this foreign website. However, this doesn't mean that our text can be protected unfortunately. Oh well..
Any questions? Post them here in our forums.
Here are other useful links around the web on this topic:
|
