=head1 NAME

dbz - Database routines for InterNetNews

=head1 SYNOPSIS

    #include <inn/dbz.h>

    #define DBZMAXKEY              ...
    #define DBZ_INTERNAL_HASH_SIZE ...

    typedef enum
    {
        DBZSTORE_OK,
        DBZSTORE_EXISTS,
        DBZSTORE_ERROR
    } DBZSTORE_RESULT;

    typedef enum
    {
        INCORE_NO,
        INCORE_MEM,
        INCORE_MMAP
    } dbz_incore_val;

    typedef struct {
        bool writethrough;
        dbz_incore_val pag_incore;
        dbz_incore_val exists_incore;
        bool nonblock;
    } dbzoptions;

    typedef struct {
        char hash[DBZ_INTERNAL_HASH_SIZE];
    } __attribute__((__packed__)) erec;

    extern bool dbzinit(const char *name);
    extern bool dbzclose(void);

    extern bool dbzfresh(const char *name, off_t size);
    extern bool dbzagain(const char *name, const char *oldname);
    extern bool dbzexists(const HASH key);
    extern bool dbzfetch(const HASH key, off_t *value);
    extern DBZSTORE_RESULT dbzstore(const HASH key, off_t data);
    extern bool dbzsync(void);
    extern long dbzsize(off_t contents);
    extern void dbzsetoptions(const dbzoptions options);
    extern void dbzgetoptions(dbzoptions *options);

=head1 DESCRIPTION

These functions provide an indexing system for rapid random access to a text
file, hereafter named the I<base> file, notably used for the C<hisv6> history
storage method.

I<dbz> stores offsets into the base file for rapid retrieval.  All retrievals
are keyed on a hash value that is generated by the B<HashMessageID> function
in libinn(3).

B<dbzinit> opens a database, an index into the base file I<name>, consisting
of files I<name>F<.dir>, I<name>F<.index>, and I<name>F<.hash> which must
already exist.  (If the database is new, they should be zero-length files.)
Subsequent accesses go to that database until B<dbzclose> is called to close
the database.  When tagged hash format is used (if B<--enable-tagged-hash> was
given at configure time), a I<name>F<.pag> file is used instead of F<.index>
and F<.hash>.

B<dbzfetch> searches the database for the specified I<key>, assigning the
offset of the base file for the corresponding I<key> to I<value>, if any.

B<dbzstore> stores the I<key>-I<data> pair in the database.  It will
return C<DBZSTORE_EXISTS> for duplicates (already existing entries), and
C<DBZSTORE_OK> for success.  It will fail with C<DBZSTORE_ERROR> if the
database files are not writable or not opened, or if any other error occurs.

B<dbzexists> will verify whether or not the given hash exists or not.  I<dbz>
is optimized for this operation and it may be significantly faster than
B<dbzfetch>.

B<dbzfresh> is a variant of B<dbzinit> for creating a new database with
more control over details.  The I<size> parameter specifies the size of
the first hash table within the database, in number of key-value pairs.
Performance will be best if the number of key-value pairs stored in the
database does not exceed about 2/3 of I<size>, or 1/2 of I<size> when
the tagged hash format is used.  (The B<dbzsize> function,
given the expected number of key-value pairs, will suggest a database size
that meets these criteria.)  Assuming that an I<fseek> offset is 4 bytes,
the F<.index> file will be 4 * I<size> bytes.  The F<.hash> file will be
C<DBZ_INTERNAL_HASH_SIZE> * I<size> bytes (the F<.dir> file is tiny and
roughly constant in size) until the number of key-value pairs exceeds about
80% of I<size>.  (Nothing awful will happen if the database grows beyond 100%
of I<size>, but accesses will slow down quite a bit and the F<.index> and
F<.hash> files will grow somewhat.)

I<dbz> stores up to C<DBZ_INTERNAL_HASH_SIZE> bytes (by default, 4 bytes
if tagged hash format is used, 6 otherwise) of the Message-ID's hash in the
F<.hash> file to confirm a hit.  This eliminates the need to read the base
file to handle collisions.

A I<size> of C<0> given to B<dbzfresh> is synonymous with the local default;
the normal default is suitable for tables of about 6,000,000 key-value
pairs (or 500,000 key-value pairs when the tagged hash format is used).
That default value is used by B<dbzinit>.

When databases are regenerated periodically, as it is the case for the
F<history> file, it is simplest to pick the parameters for a new database
based on the old one.  This also permits some memory of past sizes of the
old database, so that a new database size can be chosen to cover expected
fluctuations.  B<dbzagain> is a variant of B<dbzinit> for creating a new
database as a new generation of an old database.  The database files for
I<oldname> must exist.  B<dbzagain> is equivalent to calling B<dbzfresh> with
a I<size> equal to the result of applying B<dbzsize> to the largest number of
entries in the I<oldname> database and its previous 10 generations.

When many accesses are being done by the same program, I<dbz> is massively
faster if its first hash table is in memory.  If the I<pag_incore> flag
is set to C<INCORE_MEM>, an attempt is made to read the table in when the
database is opened, and B<dbzclose> writes it out to disk again (if it was
read successfully and has been modified).  B<dbzsetoptions> can be used to
set the I<pag_incore> and I<exists_incore> flags to different values which
should be C<INCORE_NO> (read from disk), C<INCORE_MEM> (read from memory)
or C<INCORE_MMAP> (read from a mmap'ed file) for the F<.hash> and F<.index>
files separately; this does not affect the status of a database that has
already been opened.  The default is C<INCORE_NO> for the F<.index> file and
C<INCORE_MMAP> for the F<.hash> file.  The attempt to read the table in may
fail due to memory shortage; in this case I<dbz> fails with an error.  Stores
to an in-memory database are not (in general) written out to the file until
B<dbzclose> or B<dbzsync>, so if robustness in the presence of crashes or
concurrent accesses is crucial, in-memory databases should probably be avoided
or the I<writethrough> option should be set to true (telling to systematically
write to the filesystem in addition to updating the in-memory database).

If the I<nonblock> option is true, then writes to the F<.hash> and F<.index>
files will be done using non-blocking I/O.  This can be significantly faster
if your platform supports non-blocking I/O with files.  It is only applicable
if you're not mmap'ing the database.

B<dbzsync> causes all buffers etc. to be flushed out to the files.  It is
typically used as a precaution against crashes or concurrent accesses when
a I<dbz>-using process will be running for a long time.  It is a somewhat
expensive operation, especially for an in-memory database.

Concurrent reading of databases is fairly safe, but there is no
(inter)locking, so concurrent updating is not.

An open database occupies three I<stdio> streams and two file descriptors;
Memory consumption is negligible except for in-memory databases (and I<stdio>
buffers).

=head1 DIAGNOSTICS

Functions returning I<bool> values return true for success, false for failure.

B<dbzinit> attempts to have I<errno> set plausibly on return, but otherwise
this is not guaranteed.  An I<errno> of C<EDOM> from B<dbzinit> indicates that
the database did not appear to be in I<dbz> format.

If C<DBZTEST> is defined at compile-time, then a B<main()> function will be
included.  This will do performance tests and integrity test.

=head1 BUGS

Unlike I<dbm>, I<dbz> will refuse to B<dbzstore> with a key already in the
database.  The user is responsible for avoiding this.

The RFC5322 case mapper implements only a first approximation to the
hideously-complex RFC5322 case rules.

I<dbz> no longer tries to be call-compatible with I<dbm> in any way.

=head1 HISTORY

The original I<dbz> was written by Jon Zeeff <zeeff@b-tech.ann-arbor.mi.us>.
Later contributions by David Butler and Mark Moraes.  Extensive reworking,
including this documentation, by Henry Spencer <henry@zoo.toronto.edu> as part
of the C News project.  MD5 code borrowed from RSA.  Extensive reworking to
remove backward compatibility and to add hashes into I<dbz> files by Clayton
O'Neill <coneill@oneill.net>.  Rewritten into POD by Julien Elie.

=head1 SEE ALSO

dbm(3), history(5), libinn(3).

=cut
