HISSQLITE(5)

NAME
hissqlite - SQLite-based history storage method for INN

DESCRIPTION
This method uses SQLite to store the history database, as an alternative to the default "hisv6" method. It requires version 3.24.0 or later of the SQLite library.

SQLite source, documentation, etc. are available at <https://www.sqlite.org/>. One of the stated goals of the SQLite file format is long-term stability and compatibility, which makes it a good choice for a history database.

Unlike "hisv6", hissqlite keeps everything in a single SQLite database file: there is no separate history text file and no dbz index files. Consequently the makedbz and prunehistory text-file tools do not apply, and the history path simply names the database file (see "CONFIGURATION"). Each entry is keyed on the 16-byte MD5 hash of its Message-ID and is in one of three states: a real article (with a storage API token), a remembered Message-ID (no token, recorded so a cancelled or expired article is not re-offered), or absent.

Only one schema version of the hissqlite method currently exists, implemented since INN 2.8.0.

This method comes with a migration tool (hissqlite-convert) and a support utility (hissqlite-util). See their manual pages for details.

INSTALLATION
The configure script will automatically enable SQLite support if it finds the SQLite library. If the library isn't installed in a standard location, you may have to specify the --with-sqlite3 option to help configure find it. For complicated cases, you can use separate --with-sqlite3-include and --with-sqlite3-lib options. Finally, if you don't want SQLite support even when your system has the SQLite library, you can use the --without-sqlite3 option.

CONFIGURATION
To select hissqlite as your history method, set the hismethod parameter in inn.conf to "hissqlite". The database is stored as history.sqlite in the directory named by the pathhistory parameter (by default the value of pathdb). After changing hismethod, migrate or rebuild your history (see "MIGRATION") and restart INN.

To start with an empty history database on a fresh server, run makehistory once: on an empty spool, it simply creates an empty history.sqlite ("make install" does this when hismethod is already set to "hissqlite"). Do not create the file by hand; innd requires an initialized database and will refuse to start on an empty file.

The following inn.conf parameters tune the database; their defaults suit most sites.

hissqlitecachesize
    The page cache size in kilobytes for the innd writer connection. The default is 65536 (64 MB).

hissqlitemmapsize
    The maximum number of bytes of the database that SQLite memory-maps, for both the writer and the direct readers. When enabled, memory-mapped pages are served from the shared operating system page cache, which can keep the B-tree hot and lets reader processes share physical memory. This is of little benefit on ZFS and other systems without a unified buffer cache, where the pages would simply be cached twice, and memory-mapped I/O turns a read error into a fatal signal ("SIGBUS") rather than a recoverable error. The default value of 0 therefore disables it, matching ovsqlite(5). A good starting value, if you enable it after benchmarking, is the size of the database file or the amount of available memory, whichever is smaller.

hissqlitepagesize
    The SQLite page size in bytes, applied when the database is created (it cannot be changed afterwards without rebuilding). Must be a power of two between 512 and 65536. The default is 4096, matching the common filesystem block size so that a random single-row insert or point lookup on the random-MD5 keys touches exactly one block (no read-modify-write). For best results, the page size should match the underlying storage block size: on ZFS, create a dedicated history dataset with "recordsize=4k" and point the pathhistory parameter at it. A small page on a coarse-recordsize ZFS dataset (the 128k default) forces read-modify-write on every random write and will perform worse, not better.

hissqlitereadercachesize
    The page cache size in kilobytes for each nnrpd reader process, which opens the database directly for read-only access. Each reader has its own cache, so this is kept small; history lookups are not nnrpd's hot path (overview is), and readers share the operating system page cache. The default is 2000 (about 2 MB).

The database is opened in WAL journal mode with "synchronous" set to "NORMAL" (see "DURABILITY").

DURABILITY
hissqlite uses WAL (Write-Ahead Logging) journal mode with "synchronous" set to "NORMAL". A committed write is a buffered append to the WAL: it is durable across an application crash (for instance, if innd is killed), but a sudden power loss or kernel panic can lose the transactions written since the last checkpoint. The database itself is never corrupted. This matches the durability contract of the "hisv6"/dbz backend, where history updates are likewise periodically flushed rather than synchronously committed; in both cases the affected articles are simply re-offered by peers. The WAL is checkpointed in the background and truncated on a clean shutdown.

Note: WAL mode requires shared-memory support and does not work on network filesystems such as NFS if duties are split across multiple machines. Keep pathhistory on local storage or only use a single instance to run innd and other programs like nnrpd.

MIGRATION
To convert an existing "hisv6" history into a hissqlite database without losing remembered entries or exact timestamps, use hissqlite-convert; see hissqlite-convert(8). Alternatively, you can rebuild the history from the article spool with makehistory, though a from-spool rebuild cannot reconstruct remembered Message-IDs. Both bulk-load in large batched transactions, substantially faster than innd's steady-state one-commit-per-article write path.

RUNNING
hissqlite has no separate server daemon. innd writes the database directly and is the steady-state writer; the only other writer is expire, which runs in place (see "EXPIRATION") rather than rebuilding and swapping files, so innd is not paused for it. SQLite's WAL locking and busy timeout serialize the two writers, and because writes autocommit there is no broker process to funnel them through (unlike ovsqlite). Reader processes and the offline tools open the same WAL database read-only.

While innd is running, hissqlite-util can safely read the database: innd holds the write lock for only a single statement at a time, so it waits at most briefly. (hissqlite-convert is a one-time migration tool that reads a separate source history, which must not be written while it runs; see hissqlite-convert(8).)

"hissqlite-util -s" reports database storage and entry statistics. "hissqlite-util -v" vacuums the database when freelist space should be reclaimed; see "VACUUMING".

VACUUMING
hissqlite does not vacuum the database automatically, and that is intentional.

When rows are deleted (for example when expire drops remembered Message-IDs past /remember/), SQLite does not usually shrink the file. It places free pages on a freelist and reuses them for later inserts. For a news spool that receives and expires at a steady rate, the database therefore grows to a high-water mark and then stays there: freelist pages absorb the churn of expire and intake without constantly extending or truncating the file. That behaviour is efficient. Keeping the high-water mark avoids repeated filesystem allocations, keeps more of the B-tree's working set in a stable file layout, and means expire does not pay the cost of a full rewrite.

A full "VACUUM" rewrites the entire database into a compact copy and replaces the original. It reclaims freelist space (and can apply a new page size if "PRAGMA page_size" was set beforehand), but it needs free disk space on the order of the current file size, takes time proportional to the database size, and needs exclusive write access while it runs. Automatic vacuuming after every expire would therefore be expensive and usually pointless: the next day of traffic would grow the file back toward the same high-water mark.

Vacuum manually with "hissqlite-util -v" only when the smaller size is actually wanted for the long term, for example:

•   After a large, permanent reduction in retention (so the spool will not grow back to the old size).

•   After a bulk delete or migration that left a large freelist (check with "hissqlite-util -s"; freelist share is reported there).

•   When changing hissqlitepagesize on an existing database. The page size is applied only when the database is created; changing it later requires setting "PRAGMA page_size" and vacuuming (or rebuilding with makehistory / hissqlite-convert). Prefer matching the page size to the filesystem block or ZFS recordsize from the start; see hissqlitepagesize above.

In normal operation, leave freelist pages alone and treat the high-water mark as the steady-state size of an expiring spool.

EXPIRATION
expire runs against the live database in place: real articles past their retention are transitioned to remembered entries (their token is cleared), and remembered entries older than the /remember/ threshold in expire.ctl are deleted. Because hissqlite expires in place, expire cannot write a rebuilt history to a side file, so its -d and -f options are rejected. Use -t for a dry run: it lists the tokens that would be removed and changes neither the history nor the spool (no history.n file is produced, unlike with "hisv6"). -x on its own is also rejected, since suppressing the history rewrite while still removing articles would leave the history pointing at deleted articles. See expire(8).

HISTORY
Written by Kevin Bowling for InterNetNews.

SEE ALSO
expire(8), expire.ctl(5), hissqlite-convert(8), hissqlite-util(8), history(5), inn.conf(5), makehistory(8), ovsqlite(5).