Description

Archive Manager and Archive objects for managing CDCv2 data container files [v26.06.1].

Overview

The Archive module provides management for CDC (CRM64Pro Data Container) files through the Archive Manager and Archive objects. CDCv2 is the only archive container implemented by CoreArchive. Legacy CDC implementations are not loaded by this module. CDCv2 is a block-based container designed specifically for CRM64Pro GDK as the successor to the old DPF format developed for CRM32Pro SDK.

Key features

Block-based storage: Flexible data organization using typed and named blocks
Fast indexing: Variable size index for rapid search, load, save and delete operations
Data integrity: Fast non-cryptographic xxHash64 checks for accidental IO corruption and malformed writes
Streaming: Raw block reads and writes for large resources without full memory buffering
Compression: Optional zlib compression for data blocks
Lightweight confidentiality: Non-authenticated AES-256-CTR encrypted storage for selected blocks and for the archive index
Soft deletion: Recoverable block deletion with defragmentation support

CDC format

CDCv2 uses a fixed binary header, a variable-size block index and a payload area.

Magic header	8-byte C64CDCv2 signature and versioned header
Block identification	Blocks identified by fixed-size type and name fields
Maximum file size	64-bit offsets and sizes

CDCv2 implementation

The CDCv2 implementation provides the following features:

Header	512 bytes for storing copyright, creation and modification dates, flags and index metadata
Index	Reserved variable-size index, default 512 entries, with automatic growth when needed
Integrity	Uses xxHash64 to detect accidental corruption, truncated writes, wrong-key index decrypts and malformed stored payloads
Storage modes	Raw, compressed (zlib) or non-authenticated encrypted storage (AES-256-CTR). Raw blocks can be streamed. Compressed and encrypted blocks currently use in-memory load/save paths and are limited to 1GB decoded size
Archive confidentiality	Non-authenticated AES-256-CTR index confidentiality with a salted key derived from the caller-provided 1-32 byte key

File operations

load()	Load CDC file with IO data corruption check
save()	Save CDC header and index automatically after successful metadata changes
defrag()	Physically delete soft-deleted blocks, remove fragmentation and resize maximum blocks
getFragmentation()	Get fragmentation level caused by soft deletions and overwrites

Block operations

blockAdd()	Add or replace a block: skips identical data and otherwise appends new payload before updating the index
blockAddFile()	Add file as block with type "C64_FILE" and name as [directory]+filename+[extension]. Source file is loaded in memory and limited to 1GB
blockAddStream()	Add or replace a raw block from a callback source in 1 MiB chunks without buffering the full block. Designed for future streamed encryption support
blockLoad()	Load block data into memory. Decoded block size is limited to 1GB
blockVerify()	Verify stored and decoded block hashes. Raw streamable blocks are verified without loading the full block into memory
blockSearch()	Search for block (extremely fast due to fixed index)
blockAt()	Access block metadata by index (extremely fast due to fixed index)
blockDelete()	Soft-delete block (remains physically present but not loadable)
blockUndelete()	Recover a soft-deleted block
getBlockCount()	Get number of active data blocks
getIndexCount()	Get number of used index entries, including active and deleted blocks

Security operations

encrypt()	Apply non-authenticated AES-256-CTR confidentiality to the CDC index. setKey() must be called first
decrypt()	Remove non-authenticated AES-256-CTR confidentiality from the CDC index. The archive must have been loaded with the correct key

Security model

CDCv2 provides lightweight confidentiality / non-authenticated encrypted storage. AES-256-CTR encrypts selected block payloads and, when requested, the archive index. xxHash64 is used for fast non-cryptographic checks of the used index, stored payload bytes and decoded raw payload bytes. This is useful for detecting accidental corruption and most implementation or IO errors, but it is not cryptographic authentication. CDCv2 does not provide authenticated encryption, a keyed MAC or tamper-proof storage; applications that need protection against intentional modification should add an external signature/MAC or use a future authenticated container mode.

Large blocks

blockLoad() and deep validation allocate decoded block data in memory, so CDCv2 limits decoded in-memory loads to 1GB. Blocks larger than 1GB must currently use ABS_RAW, be written with Archive::blockAddStream() and be read with Archive::blockOpenStream(). Archive::blockLoad(), Archive::blockAdd() and Archive::blockAddFile() remain in-memory paths for their supported modes. Read, seek and close operations are performed on the resulting Archive::BlockStream cursor.

Metadata writes

CDCv2 writes and flushes the index before writing and flushing the header, so the header is the final metadata commit record. This reduces the header-new/index-old failure window, but metadata updates are still not fully crash-atomic because the index is overwritten in place. A future journaled or double-index layout is required for full crash-safe commits.

Manager type

This is a sharing manager: it shares a CDC ID among multiple owners when they request to open an already opened file based on the file path [directory]+file+[extension] as the key. The CDC is effectively closed only when the last owner closes it. Shared opens must use a compatible key. Encrypted CDC files require the same key for every shared open. Unencrypted CDC files can be shared without a key, but a different key is rejected when the already opened CDC has a key attached.

Best practices

Use compression (zlib) for medium in-memory data blocks to reduce file size
Use raw streaming for very large media/resources that should not be fully buffered
Use CDCv2 encrypted storage only for lightweight confidentiality of sensitive data blocks or of the archive index
Run defrag() periodically to reclaim space from soft-deleted blocks and reduce fragmentation
Store archive keys securely, they cannot be recovered if lost
Use blockAddFile() for adding external files with automatic type and naming
Access the manager exclusively through Main::archiveMgr()

Note: The Archive Manager is a singleton, automatically created once Main is instantiated. You can get a reference to this manager using Main::archiveMgr() method.
The Archive Manager is automatically released when Main::terminate() is called. At this time, any resource still loaded will be released, avoiding resource leaks.

Classes
class	CRM64Pro::Archive
	Archive Object class. More...
class	CRM64Pro::ArchiveMgr
	Archive Manager class. More...

Macros
#define	CRM64PRO_CDC2_BLOCK_NAME_SIZE 64
#define	CRM64PRO_CDC2_BLOCK_TYPE_SIZE 16

Enumerations
enum	CRM64Pro::eArchiveBlockStore : Sint32 { CRM64Pro::ABS_RAW = 1 , CRM64Pro::ABS_COMPRESSED = 2 , CRM64Pro::ABS_ENCRYPTED = 3 }
	Archive block storing method. More...

Macro Definition Documentation

◆ CRM64PRO_CDC2_BLOCK_NAME_SIZE

#define CRM64PRO_CDC2_BLOCK_NAME_SIZE 64

CDCv2: size of block name char array.

◆ CRM64PRO_CDC2_BLOCK_TYPE_SIZE

#define CRM64PRO_CDC2_BLOCK_TYPE_SIZE 16

CDCv2: size of block type char array.

Enumeration Type Documentation

◆ eArchiveBlockStore

enum CRM64Pro::eArchiveBlockStore : Sint32

Archive block storing method.

Enumerator
ABS_RAW	Store block in raw form.
ABS_COMPRESSED	Store block compressed.
ABS_ENCRYPTED	Store block using non-authenticated AES-256-CTR confidentiality.