Description

Archive Manager and Archive objects for managing CDC data container files [v26.01.0].

Overview

The Archive module provides management for CDC (CRM64Pro Data Container) files through the Archive Manager and Archive objects. CDC is an extensible and open data container format based on data blocks, designed specifically for CRM64Pro GDK as the successor to the old DPF format developed for CRM32Pro SDK.

Key features

Block-based storage: Flexible data organization using typed and named blocks
Fast indexing: Variable size index for rapid search, load, save and delete operations
Data integrity: IO corruption and file tampering detection
Compression: Optional zlib compression for data blocks
Encryption: AES-256 encryption for individual blocks or entire archive
Soft deletion: Recoverable block deletion with defragmentation support

CDC format

The CDC format defines a minimal 8-byte magic header containing the implementation version and a set of interface methods. The format is designed to be extensible, allowing different implementations.

Magic header	8 bytes containing CDC implementation version
Block identification	Blocks identified by type (cannot start with "?SD_") and name (cannot start with "#")
Maximum file size	4GB

CDCv1.1 implementation

Currently, the only available implementation is CDCv1.1 with the following features:

Header	256 bytes for storing copyright, creation and modification dates, etc.
Index	Variable size index (minimum 128 blocks) for fast operations
Integrity	Detects IO data corruption and file tampering
Storage modes	Raw, compressed (zlib) or encrypted (AES-256)
Archive security	AES-256 index encryption with 4-32 byte key

File operations

load()	Load CDC file with IO data corruption check
save()	Save CDC file (automatically called after successful block addition)
defrag()	Physically delete soft-deleted blocks, remove fragmentation and resize maximum blocks
getFragmentation()	Get fragmentation level caused by soft deletions and overwrites

Block operations

blockAdd()	Add new block: skips if identical exists, overwrites payload if fits, or adds new
blockAddFile()	Add file as block with type "C64_FILE" and name as [directory]+filename+[extension]
blockLoad()	Load block data (soft-deleted blocks are not loaded)
blockSearch()	Search for block (extremely fast due to fixed index)
blockAt()	Access block by index (extremely fast due to fixed index)
blockDelete()	Soft-delete block (remains physically present but not loadable)
blockUndelete()	Recover a soft-deleted block
getNumberOfBlocks()	Get total number of data blocks

Security operations

secure()	Encrypt index with AES-256 to prevent block access without password
unSecure()	Decrypt index with valid password (password is unrecoverable if lost)

Manager type

This is a sharing manager: it shares a CDC ID among multiple owners when they request to open an already opened file based on the file path [directory]+file+[extension] as the key. The CDC is effectively closed only when the last owner closes it.

Best practices

Use compression (zlib) for large data blocks to reduce file size
Use AES-256 encryption for sensitive data blocks
Run defrag() periodically to reclaim space from soft-deleted blocks and reduce fragmentation
Store the archive password securely, it cannot be recovered if lost
Avoid block type names starting with "?SD_" and block names starting with "#" (reserved)
Use blockAddFile() for adding external files with automatic type and naming
Access the manager exclusively through Main::IArchiveMgr()

Note: The Archive Manager is a singleton, automatically created once Main is instantiated. You can get a reference to this manager using Main::IArchiveMgr() method.
The Archive Manager is automatically released when Main::Terminate() is called. At this time, any resource still loaded will be released, avoiding resource leaks.

Classes
class	CRM64Pro::Archive
	Archive Object class. More...
class	CRM64Pro::ArchiveMgr
	Archive Manager class. More...

Macros
#define	CDCv1_sPARAM1 64
#define	CDCv1_sPARAM2 16

Enumerations
enum	CRM64Pro::eArchiveBlockStore { CRM64Pro::ABS_RAW = 1 , CRM64Pro::ABS_COMPRESSED = 2 , CRM64Pro::ABS_CRYPTED = 3 }
	Archive block storing method. More...

Functions
bool	CRM64Pro::Archive::info (Sint32 iMode=0) override
	Request CDC object information.
const string &	CRM64Pro::Archive::getName () const override
	Get the name.
Uint32	CRM64Pro::Archive::getID () const override
	Get the ID.
Sint32	CRM64Pro::Archive::blockAdd (Block &myBlock)
	Add a data block.
Sint32	CRM64Pro::Archive::blockAddFile (const string &sFile, eArchiveBlockStore eABS=ABS_COMPRESSED)
	Add a file.
Sint32	CRM64Pro::Archive::blockDelete (Block &myBlock)
	Delete a data block.
Sint32	CRM64Pro::Archive::blockUndelete (Uint32 iBlock, Block &myBlock)
	Recover a deleted data block.
Sint32	CRM64Pro::Archive::blockSearch (Block &myBlock)
	Search for a given data block.
Sint32	CRM64Pro::Archive::blockAt (Uint32 iBlock, Block &myBlock)
	Access to the given data block position.
Sint32	CRM64Pro::Archive::blockLoad (Block &myBlock)
	Load a data block.
Sint32	CRM64Pro::Archive::blockLoadReleaseData (Block &myBlock)
	Free a loaded data block.
Uint32	CRM64Pro::Archive::getBlockCount () const
	Get the number of used data blocks.
Uint32	CRM64Pro::Archive::getBlockLimit () const
	Get the number of maximum data blocks.
float	CRM64Pro::Archive::getFragmentation () const
	Get the fragmentation.
Sint32	CRM64Pro::Archive::secure (const unsigned char *myKey, Uint32 iKeySize)
	Secure the CDC.
Sint32	CRM64Pro::Archive::unSecure ()
	Unsecure a CDC file.
bool	CRM64Pro::ArchiveMgr::info (Sint32 iMode=0) override
	Request Archive Manager information.
Sint32	CRM64Pro::ArchiveMgr::create (const string &sFile, Uint32 iNumBlocks=0) override
	Create a new CDC file.
bool	CRM64Pro::ArchiveMgr::close (Sint32 idCDC) override
	Close and destroy a CDC object.
Sint32	CRM64Pro::ArchiveMgr::getCount () const override
	Get number of loaded objects.
Sint32	CRM64Pro::ArchiveMgr::setName (Sint32, const string &) override
	This method is not implemented.
Archive *	CRM64Pro::ArchiveMgr::get (Sint32 idCDC=0)
	Get a pointer to the CDC using its handler.
Sint32	CRM64Pro::ArchiveMgr::defrag (const string &sFile, Uint32 iNumBlocks=0)
	Defragment a CDC file.
Sint32	CRM64Pro::ArchiveMgr::load (const string &sFile, const unsigned char *myKey=nullptr, Uint32 iKeySize=0)
	Load a CDC file.

Macro Definition Documentation

◆ CDCv1_sPARAM1

#define CDCv1_sPARAM1 64

CDC v1.x: size of custom char array attribute 1

◆ CDCv1_sPARAM2

#define CDCv1_sPARAM2 16

CDC v1.x: size of custom char array attribute 2

Enumeration Type Documentation

◆ eArchiveBlockStore

enum CRM64Pro::eArchiveBlockStore

Archive block storing method.

Enumerator
ABS_RAW	Store block in raw form.
ABS_COMPRESSED	Store block compressed.
ABS_CRYPTED	Store block encrypted.

Function Documentation

◆ info() [1/2]

bool CRM64Pro::Archive::info ( Sint32 iMode = 0 )

override

Request CDC object information.

Writes information to the default log.

Parameters

iMode unused for the time being.

Returns: true on success or false on failure.

◆ getName()

const string & CRM64Pro::Archive::getName ( ) const

override

Get the name.

Returns: The object name.

◆ getID()

Uint32 CRM64Pro::Archive::getID ( ) const

override

Get the ID.

Returns: Object ID.

◆ blockAdd()

Sint32 CRM64Pro::Archive::blockAdd ( Block & myBlock )

Add a data block.

Parameters

myBlock filled Block struct, at least it needs pData and iSize. Custom attributes depend on CDC version.

Returns: 0 on success, or a negative error code on failure.

Note: Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ blockAddFile()

Sint32 CRM64Pro::Archive::blockAddFile	(	const string &	sFile,
		eArchiveBlockStore	eABS = ABS_COMPRESSED )

Add a file.

Parameters

sFile	string containing the [directory]+filename+[extension] to be added. Directory separators '\' and '/' are supported.
eABS	Check eArchiveBlockStore enum for details. Default is ABS_COMPRESSED.

Returns: 0 on success, or a negative error code on failure.

Note: Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ blockDelete()

Sint32 CRM64Pro::Archive::blockDelete ( Block & myBlock )

Delete a data block.

Parameters

myBlock filled Block struct with custom attributes depending on CDC version.

Returns: 0 on success, or a negative error code on failure.

Note: Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ blockUndelete()

Sint32 CRM64Pro::Archive::blockUndelete	(	Uint32	iBlock,
		Block &	myBlock )

Recover a deleted data block.

Parameters

iBlock	block index position from [0 to used_blocks - 1] which contains a deleted data block. Use getBlockCount() for getting used_blocks.
myBlock	filled Block struct with custom attributes depending on CDC version. pData must be nullptr. If recovered, myBlock will contain all attributes except pData.

Returns: 0 on success, or a negative error code on failure.

Note: Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ blockSearch()

Sint32 CRM64Pro::Archive::blockSearch ( Block & myBlock )

Search for a given data block.

Parameters

myBlock filled Block struct with custom attributes depending on CDC version. If found, myBlock will contain all attributes except pData.

Returns: [0 to used_blocks - 1] on success indicating the block index position, or a negative error code on failure. Use getBlockCount() for the used_blocks value.

Note: Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ blockAt()

Sint32 CRM64Pro::Archive::blockAt	(	Uint32	iBlock,
		Block &	myBlock )

Access to the given data block position.

Parameters

iBlock	block index position from [0 to used_blocks - 1]. Use getBlockCount() for the used_blocks value.
myBlock	if the block index position is found, myBlock will contain all attributes except pData.

Returns: 0 on success, or a negative error code on failure.

Note: Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ blockLoad()

Sint32 CRM64Pro::Archive::blockLoad ( Block & myBlock )

Load a data block.

Parameters

myBlock filled Block struct with custom attributes depending on CDC version. pData must be nullptr. If found, myBlock will contain all attributes plus the pData loaded.

Returns: 0 on success, or a negative error code on failure.

Note: Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ blockLoadReleaseData()

Sint32 CRM64Pro::Archive::blockLoadReleaseData ( Block & myBlock )

Free a loaded data block.

Parameters

myBlock Block struct with pData loaded by blockLoad() method.

Returns: Always returns 0.

◆ getBlockCount()

Uint32 CRM64Pro::Archive::getBlockCount ( ) const

Get the number of used data blocks.

Returns: [0 to block_limit] the number of used blocks (used_blocks).

◆ getBlockLimit()

Uint32 CRM64Pro::Archive::getBlockLimit ( ) const

Get the number of maximum data blocks.

This limit can be increased using defrag() method.

Returns: Positive maximum number (block_limit) of blocks.

◆ getFragmentation()

float CRM64Pro::Archive::getFragmentation ( ) const

Get the fragmentation.

Returns: Float value in the [0-100] range expressed as a percentage (lower is better, meaning less fragmentation), or a negative error code on failure.

Note: Depending on CDC version, deleting blocks can fragment the store.; Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ secure()

Sint32 CRM64Pro::Archive::secure	(	const unsigned char *	myKey,
		Uint32	iKeySize )

Secure the CDC.

Parameters

myKey	unsigned char array of iKeySize containing the key.
iKeySize	the key size in bytes. For CDC v1.1, the key size must be from 4 to 32 bytes.

Returns: 0 on success, or a negative error code on failure.

Warning: Securing a CDC file and forgetting the key will make its content unreadable.

Note: Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ unSecure()

Sint32 CRM64Pro::Archive::unSecure ( )

Unsecure a CDC file.

Returns: 0 on success, or a negative error code on failure.

Note: The exact behaviour of this method depends on the CDC version. Please check the CDC implementation description on ArchiveMgr Detailed Description.

◆ info() [2/2]

bool CRM64Pro::ArchiveMgr::info ( Sint32 iMode = 0 )

override

Request Archive Manager information.

Writes information to the default log.

Parameters

iMode -1 to display only manager information. 0 (default) to display manager and all objects. Specific object ID to display manager and only that object.

Returns: true on success or false on failure.

◆ create()

Sint32 CRM64Pro::ArchiveMgr::create	(	const string &	sFile,
		Uint32	iNumBlocks = 0 )

override

Create a new CDC file.

Parameters

sFile	string containing [directory]+filename+[extension]. The .cdc extension is added automatically if missing. Directory separators '\' and '/' are supported.
iNumBlocks	Maximum number of data blocks. The minimum is 64.

Returns: greater than 0 on success (the CDC id) or a negative error code on failure.

Note: If the file already exists, returns an error. New CDCs use the latest version available, though older versions can still be loaded.

◆ close()

bool CRM64Pro::ArchiveMgr::close ( Sint32 idCDC )

override

Close and destroy a CDC object.

Parameters

idCDC 0 for closing all CDCs or the CDC id.

Returns: true on success or false on failure.

Note: If you forget to close a CDC, it will be automatically closed once the GDK is terminated.

◆ getCount()

Sint32 CRM64Pro::ArchiveMgr::getCount ( ) const

override

Get number of loaded objects.

Returns: the number of CDC files.

◆ setName()

Sint32 CRM64Pro::ArchiveMgr::setName	(	Sint32	idCDC,
		const string &	sName )

override

This method is not implemented.

Parameters

idCDC	Unused
sName	Unused

Returns: Does not return any value.

◆ get()

Archive * CRM64Pro::ArchiveMgr::get ( Sint32 idCDC = 0 )

Get a pointer to the CDC using its handler.

By default it returns the first CDC loaded.

Parameters

idCDC CDC id.

Returns: nullptr if the CDC was not found, otherwise a pointer to the CDC object.

◆ defrag()

Sint32 CRM64Pro::ArchiveMgr::defrag	(	const string &	sFile,
		Uint32	iNumBlocks = 0 )

Defragment a CDC file.

Parameters

sFile	string containing [directory]+filename+[extension]. Directory separators '\' and '/' are supported.
iNumBlocks	Maximum number of data blocks (minimum 64). Default 0 keeps current max block number. Can be used to resize the maximum number of data blocks.

Returns: 0 on success, or a negative error code on failure.

Note: Depending on CDC version, the storing method can produce fragmentation when blocks are deleted. The CDC file must be closed and unsecured to defragment.; Exact behavior depends on CDC version. See ArchiveMgr Detailed Description for CDC implementation details.

◆ load()

Sint32 CRM64Pro::ArchiveMgr::load	(	const string &	sFile,
		const unsigned char *	myKey = nullptr,
		Uint32	iKeySize = 0 )

Load a CDC file.

The Archive Manager shares ids based on file path as the key, so it checks if a CDC is already loaded before creating a new one.

Parameters

sFile	string containing [directory]+filename+[extension]. Directory separators '\' and '/' are supported.
myKey	unsigned char array containing the key. Only used for secured CDC files. Default nullptr.
iKeySize	key size from 4 to 32 bytes. 0 for unsecured CDC (default).

Returns: greater than 0 on success (the CDC id) or a negative error code on failure.

Description

Overview

Key features

CDC format

CDCv1.1 implementation

Manager type

Best practices

Classes

Macros

Enumerations

Functions

Macro Definition Documentation

◆ CDCv1_sPARAM1

◆ CDCv1_sPARAM2

Enumeration Type Documentation

◆ eArchiveBlockStore

Function Documentation

◆ info() [1/2]

◆ getName()

◆ getID()

◆ blockAdd()

◆ blockAddFile()

◆ blockDelete()

◆ blockUndelete()

◆ blockSearch()

◆ blockAt()

◆ blockLoad()

◆ blockLoadReleaseData()

◆ getBlockCount()

◆ getBlockLimit()

◆ getFragmentation()

◆ secure()

◆ unSecure()

◆ info() [2/2]

◆ create()

◆ close()

◆ getCount()

◆ setName()

◆ get()

◆ defrag()

◆ load()