journal

A simple log-structured library for event-driven applications.

Journal is essentially an append-only data file (*.dat) with an index file (*.idx) used to speed up lookups. No complex data structures, no sophisticated algorithms, only basic file access. We rely on the filesystem cache (managed by the operating system) to ensure read performance.

Main features:

• Variable length record type
• Records uniquely identified by a sequential number (seqnum)
• There are no other indexes other than seqnum
• Records can be appended and read by seqnum
• Records cannot be updated or deleted
• Allows reverting the last entries (rollback)
• Supports read-write concurrency (multi-thread)
• Automatic data recovery in case of catastrophic events
• Minimal memory footprint
• No dependencies

Operation modes:

• Online: 1-thread writes (append, check, rollback), N-threads reads (read, range).
• Offline: maintenance operations (split, join).

File format

Journal file formats (.dat, .idx) use native host byte order for integer fields (endianness). Files copied to a machine with different endianness may not be readable.

dat file format

    header        record1          data1          record2       data2
┌──────┴──────┐┌─────┴─────┐┌────────┴────────┐┌─────┴─────┐┌─────┴─────┐...
  magic number    seqnum1       raw bytes 1       seqnum2    raw bytes 2
     format       length1                         length2
    metadata      chksum1                         chksum2

idx file format

     header      record1       record2
┌──────┴──────┐┌─────┴─────┐┌─────┴─────┐...
  magic number    offset1      offset2
     format
     seqnum1

Usage

Drop journal.h and journal.c into your project and start using it.

#include "journal.h"

#define MAX_ENTRIES 100

ldb_journal_t *journal = NULL;
ldb_entry_t entries[MAX_ENTRIES] = {{0}};
ldb_range_t range = {0};
char buf[10 * 1024];
size_t num = 0;
int rc = 0;

journal = ldb_alloc();
ldb_open(journal, "/my/directory", "example", LDB_OPEN_CREATE);

...

rc = ldb_append(journal, entries, MAX_ENTRIES, NULL);

if (rc != LDB_OK)
    printf("Error: %s\n", ldb_strerror(rc));

...

range = ldb_get_range(journal);

print("Min seqnum = %zu\n", range.min_seqnum);
print("Max seqnum = %zu\n", range.max_seqnum);

rc = ldb_read(journal, range.min_seqnum, entries, MAX_ENTRIES, buf, sizeof(buf), &num);

if (rc == LDB_OK)
{
    if (num == MAX_ENTRIES)
        printf("Read all requested data\n");
    else if (entries[num].seqnum == 0)
        printf("Last record reached\n");
    else {
        printf("Not enough memory in buffer, only read %zu entries\n", num);
        printf("Next entry requires a minimum of %zu bytes\n", entries[num].data_len + 24);
    }

    for (int i = 0; i < num; i++) {
        printf("entry[%zu].data = %s\n", entry[i].seqnum, entry[i].data);
    }
}

ldb_close(journal);
ldb_free(journal);

Read the function documentation in journal.h.
See tests.c for unit tests.
See example.c for basic function usage.
See performance.c for concurrent usage.
See jtools.c for a maintenance tool.

Contributors

Name	Contribution
Gerard Torrent	Initial work Code maintainer
J_H	Code review
Harith	Code review

License

This project is licensed under the MIT License - see the LICENSE file for details.