| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@252 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@250 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@242 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
output files are not covered by our copyright
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@186 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
redirect loop detection code
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@141 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
links, etc) and link problems (errors retreiving the document)
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@138 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@135 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@134 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
a host open to do multiple requests (this greatly speeds up crawling of ftp sites)
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@133 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
handling errors more gracefully and also crawl normal ftp directories
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@132 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
mimetypes module
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@129 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
content if we can parse the content type
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@128 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
useful, based on a partial patch by Evelyn Mitchell <efm@tummy.com>
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@99 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@72 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
making children and parents link objects instead of URLs and giving link member variables better names, change plugins accordingly, make scheme handling more pluggable and only use one function call and have a better pluggable structure for content parsing (currently only html)
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@66 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
upper-case URL
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@65 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
more clearly mark internal functions and do some major clean-up of the scheme modules code
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@61 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@59 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
change) and update notices
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@51 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
debug command line option
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@33 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@17 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|