| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
| |
This splits some common functionality from Link._get_child() and
Crawler.get_link() to the new Link.get_or_create() function.
|
|
|
|
|
| |
This should make some functions clearer and marks internal functions
with a leading underscore.
|
| |
|
| |
|
|
|
|
|
| |
This moves all static files to be installed into the webcheck Python
path and uses pkg_resources to load the files.
|
| |
|
|
|
|
|
|
|
|
| |
Exposing crawler.bases leaks the sqlalchemy session to the plugins which
seems to cause problems in some cases.
As a consequence of this change, the sitemap plugin now uses its own
session.
|
| |
|
| |
|
|
|
|
|
|
| |
This uses the Jinja template engine to produce the report HTML files.
This also renames the util module to output to better describe its
purpose.
|
|
|
|
|
| |
This tries to close the session when the function is done with it to
avoid using too much memory.
|
|
|
|
|
|
| |
This changes the constructor to accept a dict configuration of the
crawler. This is currently combined with the configuration in the config
module but the goal is to replace it completely.
|
|
|
|
| |
This avoids having module loading code in different places.
|
| |
|
|
|
|
|
|
| |
with unicode
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@471 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@464 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
crawling based on a patch by Devin Bayer
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@459 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@457 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@456 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@454 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
reading the response times out)
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@453 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
the code to webcheck.db
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@452 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
has issues with some dates (http://bugs.python.org/issue5537)
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@451 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
automatically initialise database connection when needed
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@450 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@448 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
make imports of config and debugio consistent
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@447 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@446 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
|
|
| |
webcheck package
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@441 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@438 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
|
|
| |
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@436 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|
|
the webcheck package and reorganise imports accordingly
git-svn-id: http://arthurdejong.org/svn/webcheck/webcheck@435 86f53f14-5ff3-0310-afe5-9b438ce3f40c
|