path: root/ChangeLog

2005-07-31 20:44  arthur

	* [r119] parsers/html.py: also catch AttributeError for problem in
	  HTMLParser not fully supporting continuing after errors

2005-07-31 10:50  arthur

	* [r118] README: add note about supported versions of python

2005-07-31 09:45  arthur

	* [r117] parsers/html.py: replace numeric entity refs with their
	  proper values based on patch by UNKNOWN

2005-07-31 09:21  arthur

	* [r116] parsers/html.py: put new html parser in place

2005-07-31 09:14  arthur

	* [r115] schemes/https.py: add https module as a wrapper to the
	  http module

2005-07-31 09:02  arthur

	* [r114] crawler.py: while cleaning urls also make host part
	  lowercase and also clean added internal urls

2005-07-30 15:34  arthur

	* [r113] crawler.py: fix a thinko

2005-07-30 15:32  arthur

	* [r112] crawler.py: fix typo

2005-07-30 15:20  arthur

	* [r111] crawler.py: follow_link() now returns None when trying to
	  follow a redirect who's target is not crawled, also don't add
	  children and embeds when we are an external link

2005-07-30 14:05  arthur

	* [r110] plugins/__init__.py: remove version and author from
	  module as no other module has one (except the plugins themselves)

2005-07-30 14:04  arthur

	* [r109] config.py: remove support for extra configurable headers
	* [r108] schemes/http.py: reimplement http module to be a little
	  more generic and clean and handle errors cleaner and more
	  consistently

2005-07-30 14:00  arthur

	* [r107] crawler.py: give second search through website a slightly
	  different debug message

2005-07-30 13:59  arthur

	* [r106] crawler.py: also ignore io errors when retrieving
	  robots.txt files
	* [r105] crawler.py: make a _urlclean() function to always store a
	  proper url without a fragment and with at least a slash for urls
	  with path elements

2005-07-30 13:55  arthur

	* [r104] README: some minor tweaks in the documentation

2005-07-29 14:36  arthur

	* [r103] crawler.py: import time as we need it for sleep

2005-07-29 14:32  arthur

	* [r102] crawler.py, plugins/sitemap.py: do an extra breadth first
	  traversal of the site to combine links into pages, combining
	  page children and determining depth of every page and using all
	  this in the sitemap

2005-07-29 10:20  arthur

	* [r101] AUTHORS, README, config.py, webcheck.1: change email
	  address from arthur@tiefighter.et.tudelft.nl to
	  arthur@ch.tudelft.nl (including urls etc)

2005-07-29 10:18  arthur

	* [r100] webcheck.css: remove another reference of an email address

2005-07-29 10:11  arthur

	* [r99] NEWS, README, config.py, crawler.py, debugio.py,
	  parsers/__init__.py, parsers/css.py, parsers/html.py,
	  plugins/__init__.py, plugins/about.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/new.py,
	  plugins/notchkd.py, plugins/notitles.py, plugins/old.py,
	  plugins/problems.py, plugins/sitemap.py, plugins/slow.py,
	  plugins/urllist.py, schemes/__init__.py, schemes/file.py,
	  schemes/ftp.py, schemes/http.py, webcheck.py: remove references
	  to email addresses where they are not useful, based on a partial
	  patch by Evelyn Mitchell <efm@tummy.com>

2005-07-27 20:38  arthur

	* [r98] plugins/__init__.py, plugins/badlinks.py,
	  plugins/problems.py, plugins/sitemap.py: fix a couple of typos,
	  also thanks to Scott Kirkwood <scottakirkwood@gmail.com> for
	  spotting another one

2005-07-27 20:32  arthur

	* [r97] crawler.py: turn tocheck list into fifo queue

2005-07-26 20:40  arthur

	* [r96] plugins/new.py, plugins/old.py: fix typo spotted by Scott
	  Kirkwood <scottakirkwood@gmail.com>

2005-07-25 17:29  arthur

	* [r94] ChangeLog, NEWS, config.py: get files ready for 1.9.1
	  release

2005-07-25 17:17  arthur

	* [r93] webcheck.1: fix typo, thanks to Stefan Schröder
	  <stefan@tokonoma.de>

2005-07-25 17:16  arthur

	* [r92] plugins/slow.py: only report on internal links

2005-07-25 17:13  arthur

	* [r91] parsers/css.py: empty module as placeholder to parse css
	  (referenced from __init__.py already)

2005-07-25 17:11  arthur

	* [r90] parsers/html.py: don't replace an allready set title

2005-07-24 09:32  arthur

	* [r88] ChangeLog: add ChangeLog for release

2005-07-24 09:30  arthur

	* [r87] NEWS, TODO: get files ready for release

2005-07-24 08:56  arthur

	* [r86] README: clean up README removing sections that should be
	  in the manual page

2005-07-24 08:55  arthur

	* [r85] config.py, plugins/new.py, plugins/old.py,
	  plugins/whatsnew.py, plugins/whatsold.py: rename whatsold and
	  whatsnew plugins to old and new

2005-07-24 08:52  arthur

	* [r84] schemes/http.py: handle socket errors properly
	* [r83] schemes/http.py: fix for incomplete change in r76, now
	  version should not be referenced any more

2005-07-24 08:49  arthur

	* [r82] plugins/__init__.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/notchkd.py,
	  plugins/notitles.py, plugins/problems.py, plugins/sitemap.py,
	  plugins/slow.py, plugins/urllist.py, plugins/whatsnew.py,
	  plugins/whatsold.py: call make_link() with a link object instead
	  of a url, removing the need for a mySite in plugins

2005-07-24 08:47  arthur

	* [r81] plugins/badlinks.py: remove HTTP status code handling from
	  here as this should be done by the HTTP module
	* [r80] plugins/whatsnew.py, plugins/whatsold.py: only report on
	  internal links

2005-07-24 08:46  arthur

	* [r79] crawler.py: only add links to crawl list if they are not
	  in there allready

2005-07-24 08:45  arthur

	* [r78] debugio.py: flush stdout after each message so that
	  redirecting stdout and stderr together to a file works reliably

2005-07-23 14:02  arthur

	* [r77] crawler.py: fix regular expression matching

2005-07-23 12:55  arthur

	* [r76] config.py, plugins/__init__.py, schemes/http.py,
	  version.py, webcheck.1, webcheck.py: integrate versio.py into
	  config.py, clean up config.py removing unused settings and clean
	  up boolean types

2005-07-23 11:00  arthur

	* [r75] config.py, webcheck.1, webcheck.py: remove logo option
	  since the current output does not use one

2005-07-23 10:53  arthur

	* [r74] schemes/file.py: most systems already know about .shtml
	  files

2005-07-23 08:34  arthur

	* [r73] BUGS, INSTALL, README, webcheck.1: first step in cleaning
	  up documentation, integrating INSTALL in README and BUGS in
	  manual page and adding section on robots handling in manual

2005-07-23 08:28  arthur

	* [r72] AUTHORS, crawler.py, debugio.py, parsers/html.py,
	  plugins/__init__.py, plugins/about.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/notchkd.py,
	  plugins/notitles.py, plugins/problems.py, plugins/sitemap.py,
	  plugins/slow.py, plugins/whatsnew.py, plugins/whatsold.py,
	  schemes/file.py, schemes/ftp.py, schemes/http.py, version.py,
	  webcheck.1, webcheck.py: Mike Meyer -> Mike W. Meyer

2005-07-22 21:21  arthur

	* [r71] crawler.py: add support for sleep between requests

2005-07-22 21:11  arthur

	* [r70] webcheck.py: don't add . to python path as it's not needed
	  and put command line handling in same order as options

2005-07-22 21:05  arthur

	* [r69] plugins/__init__.py, webcheck.css: change layout to have a
	  simpler layout that also should work in MSIE

2005-07-22 21:04  arthur

	* [r68] debugio.py: fix docstrings

2005-07-22 21:01  arthur

	* [r67] plugins/__init__.py, webcheck.py: do not use start_time
	  from webcheck saving an import

2005-07-22 19:17  arthur

	* [r66] crawler.py, myUrlLib.py, parsers/__init__.py,
	  parsers/html.py, plugins/__init__.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/notchkd.py,
	  plugins/notitles.py, plugins/sitemap.py, plugins/slow.py,
	  plugins/urllist.py, plugins/whatsnew.py, plugins/whatsold.py,
	  schemes/__init__.py, schemes/file.py, schemes/ftp.py,
	  schemes/http.py, webcheck.py: almost complete rewrite of
	  crawling and site state code making children and parents link
	  objects instead of urls and giving link member variables better
	  names, change plugins accordingly, make scheme handling more
	  pluggable and only use one function call and have a better
	  pluggable structure for content parsing (currently only html)

2005-07-17 08:46  arthur

	* [r65] myUrlLib.py, plugins/__init__.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/notitles.py,
	  plugins/problems.py, plugins/sitemap.py, plugins/slow.py,
	  plugins/urllist.py, plugins/whatsnew.py, plugins/whatsold.py,
	  schemes/file.py, schemes/ftp.py, schemes/http.py, webcheck.py:
	  use lowercase url attribute in Link instead of uppercase URL

2005-07-16 15:35  arthur

	* [r64] plugins/__init__.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/notchkd.py,
	  plugins/notitles.py, plugins/problems.py, plugins/rptlib.py,
	  plugins/sitemap.py, plugins/slow.py, plugins/urllist.py,
	  plugins/whatsnew.py, plugins/whatsold.py, webcheck.py: move
	  functionality of rptlib.py to __init__.py so that we can just
	  use the plugins package

2005-07-16 15:33  arthur

	* [r63] plugins/__init__.py: remove __init__.py to be replaced by
	  contents of rptlib.py

2005-07-16 10:24  arthur

	* [r62] webcheck.1: add note about pattern matching

2005-07-10 14:08  arthur

	* [r61] myUrlLib.py, schemes/__init__.py, schemes/file.py,
	  schemes/ftp.py, schemes/http.py: rework scheme code to use more
	  logical function names, more clearly mark internal functions and
	  do some major cleanup of the scheme modules code

2005-07-10 12:26  arthur

	* [r60] myUrlLib.py, plugins/whatsnew.py, plugins/whatsold.py,
	  schemes/file.py, schemes/http.py: store mtime in link object
	  instead of age in days

2005-07-10 12:00  arthur

	* [r59] schemes/ftp.py, webcheck.py: remove unneeded import and
	  print

2005-07-09 20:22  arthur

	* [r58] htmlparse.py, myUrlLib.py, parsers, parsers/__init__.py,
	  parsers/html.py: move htmlparse to a more generic parsers
	  package, cleaning up the code and simplefying dependencies

2005-07-09 13:54  arthur

	* [r57] plugins/about.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/notchkd.py,
	  plugins/notitles.py, plugins/problems.py, plugins/rptlib.py,
	  plugins/sitemap.py, plugins/slow.py, plugins/urllist.py,
	  plugins/whatsnew.py, plugins/whatsold.py, webcheck.css,
	  webcheck.py: clean up html output generating xhtml 1.1 without
	  frames and using css for styling also getting rid of the images

2005-07-04 21:25  arthur

	* [r56] config.py: put plugins in a more logical order

2005-07-04 20:39  arthur

	* [r55] plugins/badlinks.py, plugins/external.py,
	  plugins/images.py, plugins/notchkd.py, plugins/notitles.py,
	  plugins/rptlib.py, plugins/sitemap.py, plugins/slow.py,
	  plugins/urllist.py, plugins/whatsnew.py, plugins/whatsold.py:
	  implement consistent sorting of all lists removing sort
	  functions from rptlib and using lambda functions where needed

2005-07-03 07:04  arthur

	* [r54] config.py, plugins/rptlib.py, schemes/http.py, webcheck.1:
	  handle and document proxy settings with environment variables

2005-07-03 06:36  arthur

	* [r53] INSTALL, README, config.py, myUrlLib.py,
	  plugins/rptlib.py, schemes/http.py, webcheck.1, webcheck.py:
	  name webcheck with lower case

2005-06-28 20:32  arthur

	* [r52] schemes/http.py: clean up get_reply() function to uses
	  proper recursion and don't use self where it doesn't make sense

2005-06-22 19:24  arthur

	* [r51] COPYING, debugio.py, htmlparse.py, myUrlLib.py,
	  plugins/about.py, plugins/badlinks.py, plugins/external.py,
	  plugins/images.py, plugins/notchkd.py, plugins/notitles.py,
	  plugins/problems.py, plugins/rptlib.py, plugins/sitemap.py,
	  plugins/slow.py, plugins/urllist.py, plugins/whatsnew.py,
	  plugins/whatsold.py, schemes/file.py, schemes/ftp.py,
	  schemes/http.py, version.py, webcheck.1, webcheck.py: change to
	  most recent version of the GPL (FSF address change) and update
	  notices

2005-06-18 19:59  arthur

	* [r50] plugins/external.py: sort external links by url

2005-06-18 13:48  arthur

	* [r49] webcheck.py: split main() part into it's own function

2005-06-18 13:32  arthur

	* [r48] plugins/rptlib.py, webcheck.py: restructure a couple of
	  things to reduce the number of mutual imports and reduce the
	  number of sutff gathered in webcheck.py

2005-06-18 13:31  arthur

	* [r47] config.py, plugins/urllist.py: add simple urllist plugin
	  to list all visited urls

2005-06-18 13:20  arthur

	* [r46] plugins/sitemap.py: only include internal links in sitemap

2005-06-18 12:49  arthur

	* [r45] config.py, webcheck.py: add problems plugin to config
	  instead of hard-coding

2005-06-18 10:25  arthur

	* [r44] plugins/rptlib.py: remove ugly redirection for overwrite
	  file question since we now write all html through a file
	  descriptor

2005-06-15 21:01  arthur

	* [r43] TODO, myUrlLib.py, plugins/about.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/notchkd.py,
	  plugins/notitles.py, plugins/problems.py, plugins/rptlib.py,
	  plugins/sitemap.py, plugins/slow.py, plugins/whatsnew.py,
	  plugins/whatsold.py, schemes/http.py, webcheck.py: pass
	  reference to Link class to plugins with parameter and make
	  import config where it is used instead of accessing it through
	  another module

2005-06-15 20:55  arthur

	* [r42] myUrlLib.py, plugins/rptlib.py, plugins/sitemap.py,
	  webcheck.py: make use of base consistent, do not modify it to
	  make a nicer url (at least not now) and do not overwrite it with
	  something silly from webcheck.py

2005-06-14 19:17  arthur

	* [r41] myUrlLib.py: also set URL attribute on yaked links

2005-06-12 06:21  arthur

	* [r40] plugins/badlinks.py, plugins/images.py,
	  plugins/notchkd.py, plugins/notitles.py: again use the url as
	  link title for some links

2005-06-11 21:52  arthur

	* [r39] httpcodes.py, plugins/about.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/notchkd.py,
	  plugins/notitles.py, plugins/problems.py, plugins/rptlib.py,
	  plugins/sitemap.py, plugins/slow.py, plugins/whatsnew.py,
	  plugins/whatsold.py: general cleanup of plugins structure and
	  code, moving httpcodes to the only place they were used,
	  cleaning up plugin titles, version numbers and descriptios,
	  adding docstrings and using slightly more logical and consistent
	  names (plus some other cleanups)

2005-06-11 21:39  arthur

	* [r38] plugins/rptlib.py: make_link(): if no title is specified,
	  try to look up the title of the page and fallback to the url as
	  title

2005-06-11 21:24  arthur

	* [r37] plugins/about.py: adapt plugin to using file descriptor etc

2005-06-11 18:52  arthur

	* [r36] contrib, plugins/about.py: move about plugin to plugins
	  directory

2005-06-08 19:29  arthur

	* [r35] plugins/badlinks.py, plugins/external.py,
	  plugins/images.py, plugins/notchkd.py, plugins/notitles.py,
	  plugins/problems.py, plugins/rptlib.py, plugins/sitemap.py,
	  plugins/slow.py, plugins/whatsnew.py, plugins/whatsold.py,
	  webcheck.py: write html files using file descriptors instead of
	  through redirection using stdout, split writing of navigation
	  frame and plugin pages plus some minor cleanups to calling
	  plugins

2005-06-08 19:10  arthur

	* [r34] plugins/__init__.py, schemes/__init__.py: claiming
	  copyright on empty files is silly

2005-06-06 21:22  arthur

	* [r33] debugio.py, htmlparse.py, myUrlLib.py, plugins/rptlib.py,
	  schemes/ftp.py, schemes/http.py, webcheck.1, webcheck.py: redo
	  output writing using a cleaner debugio and change debug command
	  line option

2005-06-06 20:11  arthur

	* [r32] plugins/badlinks.py, plugins/notchkd.py: replace a couple
	  more tabs

2005-06-06 20:05  arthur

	* [r31] webcheck.1: initial version of manual page loosely based
	  on documentation

2005-06-06 19:22  arthur

	* [r30] AUTHORS: added myself as copyright holder and added
	  Bastian Kleineidam (previous debian package maintainer) as
	  contributor

2005-06-06 19:20  arthur

	* [r29] webcheck.py: small text improvement

2005-05-27 20:39  arthur

	* [r28] webcheck.sh: remove unneeded shell script

2005-05-27 20:28  arthur

	* [r27] webcheck.py: also support --force

2005-05-27 20:18  arthur

	* [r26] webcheck.py: redo command-line checking

2005-04-13 19:41  arthur

	* [r25] contrib/plugins/about.py: general cleanup
	* [r24] plugins/sitemap.py: rework recursion to make it simpler
	  plus some general cleanups

2005-04-13 19:20  arthur

	* [r23] contrib/plugins/about.py, myUrlLib.py,
	  plugins/badlinks.py, plugins/external.py, plugins/images.py,
	  plugins/notchkd.py, plugins/notitles.py, plugins/problems.py,
	  plugins/rptlib.py, plugins/sitemap.py, plugins/slow.py,
	  plugins/whatsnew.py, plugins/whatsold.py, schemes/http.py,
	  webcheck.py: rename linkList to linkMap

2005-04-13 19:18  arthur

	* [r22] myUrlLib.py, robotparser.py: remove local copy of
	  robotparser, just use python\'s

2005-04-09 20:03  arthur

	* [r21] myUrlLib.py: qualify references to types functions

2005-04-09 13:48  arthur

	* [r20] htmlparse.py, myUrlLib.py, plugins/badlinks.py,
	  plugins/external.py, plugins/images.py, plugins/notchkd.py,
	  plugins/notitles.py, plugins/rptlib.py, plugins/slow.py,
	  plugins/whatsnew.py, plugins/whatsold.py, schemes/http.py:
	  indent with spaces instead of tabs (tabs are evil)

2005-04-08 21:31  arthur

	* [r19] myUrlLib.py: move finding of scheme module to separate
	  function

2005-04-08 21:25  arthur

	* [r18] schemes/http.py: rebump loglevel to debug

2005-04-08 16:24  arthur

	* [r17] myUrlLib.py, schemes/file.py, schemes/filelink.py,
	  schemes/ftp.py, schemes/ftplink.py, schemes/http.py,
	  schemes/httplink.py: remove link part from scheme modules

2005-04-07 22:37  arthur

	* [r16] schemes/httplink.py: clean up http request code a little
	  and do not set host header (it is sent by HTTPConnection already

2005-04-07 20:29  arthur

	* [r15] contrib/plugins/about.py, debugio.py, htmlparse.py,
	  httpcodes.py, myUrlLib.py, plugins/__init__.py,
	  plugins/badlinks.py, plugins/external.py, plugins/images.py,
	  plugins/notchkd.py, plugins/notitles.py, plugins/problems.py,
	  plugins/rptlib.py, plugins/sitemap.py, plugins/slow.py,
	  plugins/whatsnew.py, plugins/whatsold.py, schemes/__init__.py,
	  schemes/filelink.py, schemes/ftplink.py, version.py,
	  webcheck.py: make nicer file (copyrights) headers

2005-04-07 20:23  arthur

	* [r14] schemes/httplink.py: fix problem with incorrect indent

2005-04-07 20:06  arthur

	* [r13] config.py, httpcodes.py, plugins/notitles.py: tabs to
	  spaces (tabs are evil)

2005-04-07 20:05  arthur

	* [r12] config.py, contrib/plugins/about.py, httpcodes.py,
	  plugins/badlinks.py, plugins/external.py, plugins/notchkd.py,
	  plugins/notitles.py, plugins/problems.py, plugins/rptlib.py,
	  plugins/sitemap.py, plugins/slow.py, plugins/whatsnew.py,
	  plugins/whatsold.py, schemes/filelink.py, schemes/ftplink.py,
	  schemes/httplink.py: tabs to spaces (tabs are evil)

2005-04-07 20:04  arthur

	* [r11] AUTHORS, schemes/httplink.py: include patch from Sebastien
	  Delafond <sdelafond@gmx.net> (from
	  http://bugs.debian.org/286017) to fix problems with recent
	  versions of python

2005-04-06 19:38  arthur

	* [r10] INSTALL, config.py, htmlparse.py, plugins/images.py,
	  plugins/rptlib.py, schemes/ftplink.py, schemes/httplink.py,
	  webcheck.css, webcheck.py: import Debian package patches

2005-03-31 12:47  arthur

	* [r9] COPYING: install updated file without millenium bug

2005-03-31 12:45  arthur

	* [r8] AUTHORS: reformat file to better match suggested layout

2005-03-31 12:44  arthur

	* [r7] NEWS: put news items in a little more standard format

2005-03-31 12:42  arthur

	* [r6] AUTHORS, CHANGES, CREDITS, ChangeLog-1999, ChangeLog-2002,
	  HISTORY, HISTORY.linbot, NEWS: rename files to more standard
	  names

2005-03-31 12:32  arthur

	* [r5] config.py, plugins/rptlib.py, version.py: remove checks for
	  updates (registry)

2005-03-31 12:28  arthur

	* [r4] ., contrib, contrib/plugins, plugins, schemes: ignore
	  compiled python objects

2005-03-29 12:08  arthur

	* [r2] BUGS, CHANGES, COPYING, CREDITS, HISTORY, HISTORY.linbot,
	  INSTALL, README, TODO, config.py, contrib, contrib/plugins,
	  contrib/plugins/about.py, debugio.py, htmlparse.py,
	  httpcodes.py, myUrlLib.py, plugins, plugins/__init__.py,
	  plugins/badlinks.py, plugins/external.py, plugins/images.py,
	  plugins/notchkd.py, plugins/notitles.py, plugins/problems.py,
	  plugins/rptlib.py, plugins/sitemap.py, plugins/slow.py,
	  plugins/whatsnew.py, plugins/whatsold.py, robotparser.py,
	  schemes, schemes/__init__.py, schemes/filelink.py,
	  schemes/ftplink.py, schemes/httplink.py, version.py,
	  webcheck.css, webcheck.py, webcheck.sh: import of release 1.0

2005-03-28 12:57  arthur

	* [r1] .: create webcheck directory