Filters

(Based on Fred Cohen's guide. Updated by Jargoon, edited by Leto.)

Introduction

Filters gives you control over what files HTTrack will or will not download. You can both expand and restrict the access to websites you are trying to mirror.

Whenever you make a mirror of a website HTTrack tries to download everything inside the starting directory and in any sub-directory associated. It ignores anything outside that domain by default. If you need any different behaviour, you may wish to have a try with filters: they will let you add some parts of other websites, or deny certain sub-directories of current website, and offer an opportunity to get only certain kind of files.

Important: It's important to remember that filters help control HTTrack and only apply to pages and files that it discovers when crawling through websites defined in your Start URL settings.

Syntax

To include certain kind of things use a plus sign (+). To exclude anything you don't need, use a minus sign (-). Asterisks (*) work like wildcards to match any number of characters.

Example:

+www.all.net/i_want_this/*
-www.all.net/i_dont_want_that/*
+*.edu.au/*.jpg

Priorities

A list of filters works from least important to most important (latter filters take precedence over previous filters).

In the following example, even though a restriction has been added with the minus filter, all GIF files found will be downloaded because the second filter is overriding the first filter (more specifically, the first filter is not even applied due to the second filter):

-*/images/*.gif
+*.gif

However in this example, the two filters work together nicely. The first filter is initially allowing GIF files from any domain/server, but the second filter restricts that to deny any GIF files that are inside an "images" directory. Another way to think of what these filters are doing is: if a GIF file (on any domain) is not in an "images" directory then permit it to be downloaded

+*.gif
-*/images/*.gif

Examples of filter types

File extension filters

These filters will control certain files depending on their extension (e.g. zip, gif, tgz, pdf, mpeg).

File size filters

After a file has been added to the download queue, you still have an opportunity to abort the actual download. You can control the size range of your downloaded files. But beware: this will be only helpful if server sends correct image size before downloading the file. If that's not the case, the file will be downloaded and deleted afterwards.

MIME filters

These are important for dynamically generated websites (pages created with PHP, ASP, CGI, etc): the file extension filters rely on the assumption that HTTrack knows the type of file it is downloading. That's not the case with dynamic pages as the file may be downloaded before its file type is "applied". The MIME type of the file hopefully sent by the server will do the trick.

Important: MIME type checking is always done over links currently added to download queue (the files that have passed other filters), so rules such as -mime:*/* +mime:text/html +mime:image/gif will only polish other filters' results up.

Resources

• Fred Cohen's HTTrack User Guide
• Spanish User Guide (Fred Cohen's guide updated and translated into Spanish)