Arthur de Jong

Open Source / Free Software developer

summaryrefslogtreecommitdiffstats
path: root/backup.py
Commit message (Collapse)AuthorAgeFilesLines
* Fix logic error in handling --no-extractlistsArthur de Jong2015-07-031-1/+1
|
* Do not cache full backup contentsArthur de Jong2015-06-301-73/+33
| | | | | | Storing this in SQLite is slow and grows the cache to a huge size. The approach of reading these files lists may be a bit slower but saves a lot of space and overhead and removes quite some complexity.
* Explicitly specify filename encodingArthur de Jong2015-06-251-1/+2
|
* Change metadata informationArthur de Jong2015-06-251-11/+14
| | | | | | | | | | This changes the information in the metadata dict to include the file type in a separate field and limit the mode information to standard permissions only. Upon reading files lists from the repository the old format is automatically converted. This changes local cache file to ensure all information is re-read (the previous commit also already required this).
* Ensure file lists are written in archive orderArthur de Jong2015-06-211-1/+2
| | | | | This is apparently needed because at least GNU tar expects the extractlists to be in the same order as files in the archive itself.
* Split out backend functions from repoArthur de Jong2015-06-201-20/+6
| | | | | | This creates a FileBackend class (from parts of the FileRepository class that is now renamed to Repository) that implements a simple API that can also be provided by other backends.
* Implement --no-extractlistsArthur de Jong2015-06-011-6/+7
| | | | | | This option ensures that a full restore does not require extractlists. Note that this is less efficient than with extractlists because existing archives need to be discarded earlier.
* Add an fsck commandArthur de Jong2015-05-171-0/+3
| | | | | | | | This command checks the repository to see if any files are missing or corrupt and whether backups can still be restored. This currently only reads meta-data files and only checks archive files presence.
* Use build_extractlist() in select_archives()Arthur de Jong2015-05-151-26/+9
|
* Implement a restore commandArthur de Jong2015-05-151-0/+144
| | | | This implements a command to restore files from a specific backup.
* Make filters context managersArthur de Jong2015-05-151-41/+36
| | | | | This ensures that open files and streams are properly closed when an exception occurs.
* Catch certain problems reading JSON filesArthur de Jong2015-05-071-2/+2
| | | | | This also replaces catches IOError by catching EnvironmentError which covers a wider range of errors.
* Implement a --keep option for rmArthur de Jong2015-05-071-2/+68
| | | | | | | | This allows for specifying a policy for backups to keep. All other backups will be removed. This also implements a --no-act option to only print which backups would be removed.
* Add extractlists to infofileArthur de Jong2015-05-071-1/+2
| | | | | This adds the list of archives that have an extractlist to the infofile so the backup can be checked to see if all needed files are present.
* Change minimum archive usageArthur de Jong2015-04-101-8/+10
| | | | | | | | This ignores archives that will only be for 40% or less by default and make the percentage count towards effective use and use relative to block size. Also renames --min-files to --archive-min-files.
* Refactor out repository and cacheArthur de Jong2015-04-101-467/+14
| | | | | This moves functionality related to the metadata cache and repository files to separate modules.
* Ignore exceptions removing filesArthur de Jong2015-04-081-2/+10
| | | | | This ignores any exceptions when trying to remove files from the repository.
* Improve command-line handlingArthur de Jong2015-04-021-3/+0
| | | | | This moves a validation to cmdline, supports natural values for --block-size and does simple validation of the BACKUP argument.
* Add option for minimum number of filesMichel Wilson2015-03-271-1/+10
| | | | | | When backing up directories containing large files, archives with only one or two files are created. The --min-files option forces the archives to always contain at least a set amount of files.
* Do not store contents after backupArthur de Jong2015-03-261-6/+0
| | | | | This tries to avoid filling the SQLite cache and only fills the backup contents cache when it is needed.
* Fix key encryption and uuid propertyArthur de Jong2015-03-261-3/+4
|
* Save a UUID in the repositoryArthur de Jong2015-03-161-8/+32
| | | | | | | | This UUID is used to distinguish repository cache files from each other which allows running multiple backups without manually specifying different cache directories. This also changes the default cache directory to ~/.cache/sloth.
* Auto-detect key encryptionArthur de Jong2015-03-161-22/+12
|
* Implement a command to remove backupsArthur de Jong2015-03-151-3/+92
| | | | | This allows removing backups from the repository along with archives that have become redundant.
* Implement find commandArthur de Jong2015-03-081-0/+41
| | | | WIP: Implement a find command that finds files in archives
* Store backup contents in the cacheArthur de Jong2015-03-081-18/+97
| | | | | This stores the file list for the backup in the cache and modifies the ls command to use the cache.
* Store backup metadata in the cacheArthur de Jong2015-03-081-37/+111
| | | | | This adds the information from the backup (stored in info.json) in the cache so it can be queried easier.
* Use fetchall() where neededArthur de Jong2015-03-081-90/+95
| | | | | | | | | | | | | | Use fetchall() on a cursor because SQLite cannot handle partial reads from a cursor if the database is being modified in another cursor. This clearly uses more memory during the backup. Set the synchronous and journal_mode SQLite pragmas to have better performance but be less safe. Since it is a cache, the data can be reconstructed from the repository if needed. This also uses the connection as a context manager instead of manually calling commit, changes some of the transactions around to have better performance and includes a few consistency improvements.
* Use better names for tablesArthur de Jong2015-03-021-37/+37
| | | | The names now better reflect the purpose and contents of the table.
* Rename snapshot to backupArthur de Jong2015-03-011-36/+35
| | | | Don't use the term snapshot any more and use backup instead.
* Refactor out path-handling functionsArthur de Jong2015-03-011-13/+1
|
* Refactor out ls moduleArthur de Jong2015-03-011-44/+2
|
* Implement --exclude-from optionArthur de Jong2015-03-011-1/+11
| | | | This reads exclude patterns from a file.
* Implement --exclude optionArthur de Jong2015-03-011-1/+1
| | | | | | | This instructs the crawler to skip certain patterns from the backup. It supports * for matching any part of a file name, ** to also match /, ending the pattern with / to only match directories and starting the pattern with / to match the full path.
* Support both python 2 and 3Arthur de Jong2015-03-011-36/+36
|
* Improve database performanceArthur de Jong2015-03-011-63/+111
| | | | | | | | | This avoids problems with long extractlists and long lists of archives where more than SQLite's maximum number of SQL variables were passed. This also uses database transactions in order to improve performance and moves some database actions around in the code to make for example temporary table creation closer to where it is actually used.
* Implement a command to list backup contentsArthur de Jong2015-03-011-0/+77
| | | | | This reads snapshot file list and filters and formats the output to be like ls.
* Implement a command to list backupsArthur de Jong2015-02-211-0/+83
| | | | | This writes an info.json file that contains information on each backup that was made and shows that information with a backups command.
* Improve database performanceArthur de Jong2015-02-211-29/+40
| | | | | | | Create indexes (some after crawling which is a minor improvement) in the tables to improve queries, and use explicit transactions to improve performance (small improvement). Also, move temporary table creation to the functions where they are used (instead of global).
* Handle non-existing directories in repositoryArthur de Jong2015-02-211-0/+8
|
* Handle failing to add a file more gracefullyArthur de Jong2015-02-211-1/+5
|
* Use XDG cache directory by defaultArthur de Jong2015-02-211-2/+10
|
* Fix writing of passphrase filesArthur de Jong2015-02-121-7/+16
| | | | | | This ensures that the correct old files are removed and the current file is correctly renamed to avoid the situation when the encryption type is changed.
* Simplify read_file() and write_file()Arthur de Jong2015-02-121-44/+40
| | | | | | This integrates compression and encryption detection into the read_file() function and filename expansion into the write_file() function (adding arguments for compression and encryption).
* Refactor writing restore scriptArthur de Jong2015-02-121-85/+104
| | | | | Move writing to separate function and use file name in the repository to select the correct extract and decrypt commands for use in restore.sh.
* Implement xz compressionArthur de Jong2015-02-121-1/+9
|
* Update configuration from command-lineArthur de Jong2015-02-121-25/+23
| | | | | This updates properties of the Config class with arguments on the command line. It also includes a few configuration changes.
* Implement a set-keys commandArthur de Jong2015-02-121-35/+103
| | | | | | This configures which PGP keys should be able to decrypt the backup. At least for one of the keys specified the private key should available during backup runs to be able to verify repository meta-data.
* Implement bzip2 compressionArthur de Jong2015-02-121-1/+4
|
* Initial version of command-line handlingArthur de Jong2015-02-121-14/+32
|