Blob store
Most of the bytes a mail server stores belong to a small number of large objects: messages with their attachments, and the files users upload. The blob store holds these large binary objects, or blobs, apart from the structured records in the data store. The separation keeps the data store small and fast, and it allows the bulk of the data to be placed on inexpensive object storage.
Each blob is identified by its content rather than by the item that uses it. Identical contents are therefore stored only once, which matters for a message delivered to several accounts on the same server: every account refers to the same blob.
Contents
Section titled “Contents”The blob store holds the following data:
- Raw email messages, including their attachments.
- Files stored by users through WebDAV and JMAP file storage.
- Sieve scripts.
- Spam classifier training samples and the trained model.
- Blobs uploaded by JMAP clients that have not yet been attached to a message. Unused uploads expire and are removed by the blob purge.
- Snapshots of account caches, when cache swapping uses the
BlobStorebackend with its default store.
Account records, mailboxes, flags, calendar events and contacts are structured data, so they are kept in the data store rather than here.
Backends
Section titled “Backends”The backend is selected by the @type of the BlobStore object (found in the WebUI under Settings › Storage › Blob Store):
@type | Where blobs are kept | Shared by cluster nodes |
|---|---|---|
Default | In the data store (default) | When the data store is shared |
S3 | An S3-compatible bucket | Yes |
Azure | An Azure Blob Storage container | Yes |
FileSystem | A directory on the local file system | Only on a shared file system |
PostgreSql, MySql, FoundationDb | A database separate from the data store | Yes |
Sharded | Several of the above, with blobs distributed among them (composite stores, Enterprise) | Yes |
The Default variant is the simplest choice. Blobs then share the data store’s backups, replication and scaling, and no other service is needed. A separate blob store becomes worthwhile in two situations. The first is when the volume of message contents makes the data store too large to back up or replicate within the available time. The second is when object storage costs less than database storage. The fields of each variant are described on its backend page and in the BlobStore reference.
One combination needs care. The FileSystem backend cannot hold cache swap snapshots, so a server whose blob store is FileSystem uses the LocalFile or InMemory swap backend instead.
Changing the blob store
Section titled “Changing the blob store”A change to the BlobStore object selects where new blobs are written and where existing ones are looked up. It does not copy the blobs that are already stored. As a result, after a change the messages, files and scripts written before it can no longer be read. To change the blob store of a server that already holds data, migrate the data as described in Database migration. Like any store change, the new setting takes effect after a restart.
Compression
Section titled “Compression”Blobs are compressed before they are written. Email messages, Sieve scripts, queued messages and JMAP uploads follow the compressionAlgorithm field of the Email object ( Settings › Email › Storage), and files follow the field of the same name in the FileStorage object ( Settings › Files & Sharing › File Storage). Out of the box both are lz4. Three values are available:
lz4applies LZ4 compression, which favours speed over compression ratio. It finds little to compress in base64-encoded attachments.zstdapplies Zstandard compression. It compresses text more tightly than LZ4 and, unlike LZ4, also shrinks base64-encoded attachments, at the cost of slower compression and decompression.nonestores blobs uncompressed.
A blob that compression would not shrink, such as an already compressed image, is stored uncompressed whatever the setting. A change of algorithm applies only to the blobs written afterwards. Existing blobs are read in the form in which they were written, so no conversion is needed; the BlobRecode task described in Blob encryption converts them when that is wanted.
Encryption
Section titled “Encryption”The blob store can also encrypt blobs with keys held by the server, so that the storage backend only holds ciphertext. Blob encryption describes what it protects and how to set it up.
Maintenance
Section titled “Maintenance”A blob is deleted once nothing references it any more. This happens, for example, after the last message that uses it is expunged, or when an upload expires without being used. The deletion itself is performed by the blob purge, which runs daily at 04:00 unless its schedule is changed. Data retention and reclaiming disk space describes the schedule, and when each backend returns the freed space.