JoBo is a simple program to download complete websites to your local computer. Internally it is basically a web spider. The main advantage to other download tools is that it can automatically fill out forms (e.g. for automated login) and also use cookies for session handling. Compared to other products the GUI seems to be very simple, but the internal features matters ! Do you know any download tool that allows it to login to a web server and download content if that server uses a web forms for login and cookies for session handling? It also features very flexible rules to limit downloads by URL, size and/or MIME type. For programmers it features a very flexible object model and is easily expandable - expect new modules in the future ! It is implemented in Java and the source code is available. If you want to implement your own web spider, the WebRobot class will be a good starting point. Even if you don't want to use it as a download tool but for indexing, link checking or whatever you want, JoBo is the right tool. Retrieving documents and handling these documents are completely seperated - therefore you can plug in your own module easily. Features * command line and graphical version (but command line version needs a major update, currently the GUI version has much more features) * recursive search of all documents starting from a given start document * support of tags (with fault tolerance) * support of the robot exclusion protocol * user controlled maximal search depth * user agent name can be defined * support of referrer headers * support of automated form handling (JoBo can fill fields with predefined values) * cookie support * XML configuration * used bandwidth can be limited * allow/deny downloads by mime type and document size (e.g. ignore all image/* files) * allow/deny downloads by regular expressions (e.g. don't download /cgi-bin) * can convert absolute links to relative * download only files newer then a given age * resume job JoBo Crawler Home Page http://www.matuschek.net/jobo/ JoBo Crawler Download http://www.matuschek.net/jobo-download/

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt. Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity. System Runtime Requirements Java Runtime Environment The Heritrix crawler is implemented purely in java. This means that the only true requirement for running it is that you have a JRE installed. The Heritrix crawler makes use of Java 5.0 features so your JRE must be at least of a 5.0 (1.5.0+) pedigree. We currently include all of the free/open source third-party libraries necessary to run Heritrix in the distribution package. See dependencies for the complete list (Licenses for all of the listed libraries are listed in the dependencies section of the raw project.xml at the root of the src download or here on sourceforge). Hardware Default heap size is 256MB RAM. This should be suitable for crawls that range over hundreds of hosts. Linux The Heritrix crawler has been built and tested primarily on Linux. It has seen some informal use on Macintosh, Windows 2000 and Windows XP, but is not tested, packaged, nor supported on platforms other than Linux at this time. Heritrix Home Page http://crawler.archive.org/ Heritrix Documentation http://crawler.archive.org/articles/user_manual/index.html Download Heritrix http://crawler.archive.org/downloads.html More Java Open Source Applications