Joeri Verdeyen bio photo

Joeri Verdeyen

Web-engineer, cyclist, Nespresso lover, Strava pusher.

Twitter LinkedIn Instagram Github Stackoverflow Strava

Duplicity out of memory

For years I’am using duplicity to create encrypted backups and store them on amazon s3. I’m using a bash wrapper script for automated backups with duplicity supporting Amazon’s S3 online storage as well as other storage destinations (ftp, rsync, sftp, local storage…).

Backup error

But I received a message on my backup notification slack channel that one backup failed. The message buffer dmesg on an ubuntu 16.04 machine showed an out of memory message.

[29152765.948135] Out of memory: Kill process 2519 (python2) score 606 or sacrifice child
[29152765.949376] Killed process 2519 (python2) total-vm:1331852kB, anon-rss:845472kB, file-rss:488kB

Duplicity version upgrade needed

This was due to a bug which is resolved in version 0.7.15. (bug report)

* Fixed bug #1720159 - Cannot allocate memory with large manifest file since 0.7.03
  - filelist is not read if --file-changed option in collection-status not present
  - This will keep memory usage lower in non collection-status operations

The server was using a version from the default ubuntu repo, which is version 0.7.06.

$ apt show duplicity
Package: duplicity
Version: 0.7.06-2ubuntu2
Priority: optional
Section: utils
Origin: Ubuntu

Install duplicity package from duplicity-team repo

Install the duplicity package from ppa:duplicity-team/ppa. I also suggest to add this to your ansible roles or configuration management tool you are using.

sudo add-apt-repository ppa:duplicity-team/ppa
sudo apt-get update
sudo apt-get purge duplicity
sudo apt-get install duplicity

Check the package version is at least 0.7.15.

$ duplicity --version

Thanks for reading

Feel free to leave a comment if you have remarks or like this post