Arthur de Jong

Open Source / Free Software developer

summaryrefslogtreecommitdiffstats
path: root/schemes/ftp.py
Commit message (Collapse)AuthorAgeFilesLines
* some more small code improvements thanks to pycheckerArthur de Jong2006-05-071-1/+4
| | | | git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@252 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* bump copyright noticesArthur de Jong2006-05-061-1/+1
| | | | git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@250 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* code improvements thanks to pylintArthur de Jong2006-04-231-11/+10
| | | | git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@242 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* add copyright clarification to specify that generated ↵Arthur de Jong2005-12-171-0/+3
| | | | | | output files are not covered by our copyright git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@186 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* move redirect handling code to crawler module, including ↵Arthur de Jong2005-08-191-2/+1
| | | | | | redirect loop detection code git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@141 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* split problems into page problems (parsing errors, wrong ↵Arthur de Jong2005-08-191-1/+2
| | | | | | links, etc) and link problems (errors retreiving the document) git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@138 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* pick up configured filenames if present in directoriesArthur de Jong2005-08-161-33/+51
| | | | git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@135 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* add extra debugging infoArthur de Jong2005-08-161-8/+15
| | | | git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@134 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* use a pool of ftp connections to keep ftp connection to ↵Arthur de Jong2005-08-131-18/+25
| | | | | | a host open to do multiple requests (this greatly speeds up crawling of ftp sites) git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@133 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* almost complete reimplementation of the ftp scheme, ↵Arthur de Jong2005-08-131-62/+64
| | | | | | handling errors more gracefully and also crawl normal ftp directories git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@132 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* rename parameter to acceptedtypes to not conflict with ↵Arthur de Jong2005-08-121-1/+1
| | | | | | mimetypes module git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@129 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* also pass mimetypes to scheme modules to only fetch ↵Arthur de Jong2005-08-121-1/+1
| | | | | | content if we can parse the content type git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@128 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* remove references to email addresses where they are not ↵Arthur de Jong2005-07-291-3/+3
| | | | | | useful, based on a partial patch by Evelyn Mitchell <efm@tummy.com> git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@99 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* Mike Meyer -> Mike W. MeyerArthur de Jong2005-07-231-1/+1
| | | | git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@72 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* almost complete rewrite of crawling and site state code ↵Arthur de Jong2005-07-221-4/+5
| | | | | | making children and parents link objects instead of URLs and giving link member variables better names, change plugins accordingly, make scheme handling more pluggable and only use one function call and have a better pluggable structure for content parsing (currently only html) git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@66 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* use lower-case URL attribute in Link instead of ↵Arthur de Jong2005-07-171-4/+4
| | | | | | upper-case URL git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@65 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* rework scheme code to use more logical function names, ↵Arthur de Jong2005-07-101-73/+33
| | | | | | more clearly mark internal functions and do some major clean-up of the scheme modules code git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@61 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* remove unneeded import and printArthur de Jong2005-07-101-1/+0
| | | | git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@59 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* change to most recent version of the GPL (FSF address ↵Arthur de Jong2005-06-221-1/+1
| | | | | | change) and update notices git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@51 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* redo output writing using a cleaner debugio and change ↵Arthur de Jong2005-06-061-4/+4
| | | | | | debug command line option git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@33 86f53f14-5ff3-0310-afe5-9b438ce3f40c
* remove link part from scheme modulesArthur de Jong2005-04-081-0/+148
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@17 86f53f14-5ff3-0310-afe5-9b438ce3f40c