Guides:MAME - chdman
Compressed Hunks of Data Manager
Introduction
A (C)ompressed (H)unks of (D)ata file or CHD for short is a standard format developed by the MAME project for storing hunks of data.
The format was originally developed to image later 200x arcade machines that had begun shipping with internal hard drives. Beyond that some arcade hardware systems also began shipping with interchangeable discs (e.g. NAOMI) and a format was also needed to capture these.
If you have been around from the late 90s or have grabbed dumps from around that time you will be aware of how many different disc image formats there are. Different formats took hold for short periods depending n their features set. Some example historical formats are mdf, nrg, img, iso and bin. Emulator authors then had the decision to add support for all these formats or provide a limited number of formats and force users to convert from one format to another during use. In some cases not all the data can be converted between formats.
CHD is expected to address this by generalizing the data storage format providing a solution for both of the above scenarios and more. The CHD format itself has undergone multiple iterations to add support for various media features since its release. The current chd hashes registered in MAME 0.217 are format version 5.
Machine Support
We can take a look at the supported image formats for a couple of systems by querying MAME with -listmedia. Let's look at a couple of consoles to start with that have cdrom drives
Sony PlayStation (USA)
$ ./mame64 psu -listmedia
SYSTEM MEDIA NAME (brief) IMAGE FILE EXTENSIONS SUPPORTED
---------------- --------------------------- -------------------------------
psu memcard1 (memc1) .mc
memcard2 (memc2) .mc
quickload (quik) .cpe .exe .psf .psx
cdrom (cdrm) .chd .cue .toc .nrg .gdi .iso .cdr
Sega Saturn (USA)
$ ./mame64 saturn -listmedia
SYSTEM MEDIA NAME (brief) IMAGE FILE EXTENSIONS SUPPORTED
---------------- --------------------------- -------------------------------
saturn cdrom (cdrm) .chd .cue .toc .nrg .gdi .iso .cdr
cartridge (cart) .bin
You'll notice neither of these systems support mdf/mds format. The mdf/mds format was developed for use with a very popular tool during the early 2000s called Alcohol 120%.
Another popular tool of the early 2000's CloneCD was the only tool for a very long time to properly support sub channel data dumping and became popular with groups dumping pc titles. The deafult format for CloneCD was ccd/img or ccd/img/sub when sub channel data was dumped. Again looking at our MAME output these images are not supported in either system above although dumps exist.
You will notice the nrg format is in the list of compatible file extensions, this was a proprietary format from the tool Nero Burning ROM, it is interesting that it is supported and we will need to find out why and update information here.
We can see that chd is supported on both systems, so if we can convert our discs to chd we will remain compatible with MAME and both these systems therein.
We mentioned previously chdman was created to be used as a tool to image hard disks originally. chdman can also create blank chd images for use with machines where a hdd may be managed by an Operating System. Lets take a look at the ct486, a 486SX/25 Mhz Personal Computer emulated in MAME.
Start by having a look at the output of -listmedia for this machine.
PC/AT 486 with CS4031 chipset
$ ./mame64 ct496 -listmedia
SYSTEM MEDIA NAME (brief) IMAGE FILE EXTENSIONS SUPPORTED
---------------- --------------------------- -------------------------------
ct486 floppydisk1 (flop1) .dsk .ima .img .ufi .360 .d77 .d88 .1dd .dfi .hfe .imd .ipf .mfi .mfm .td0 .cqm .cqi
floppydisk2 (flop2) .dsk .ima .img .ufi .360 .d77 .d88 .1dd .dfi .hfe .imd .ipf .mfi .mfm .td0 .cqm .cqi
harddisk (hard) .chd .hd .hdv .2mg .hdi
printout (prin) .prn
The harddisk option does have the ability to attach a chd file to it along with a number of other formats but we will focus on chd in this document.
Once the machine is started with the disk attached using -hard <chdfile.chd> the chd will be attached on the internal machine on ide0.
Usage
The tool itself is called chdman depending on how you installed it, it may be accessible by typing chdman into your terminal. I compile MAME to custom location so I will need to execute chdman from the current directory by including a ./ before the command. Substitute as necessary for your own system.
Firstly lets take a look at the usage summary for chdman
$ ./chdman -help chdman - MAME Compressed Hunks of Data (CHD) manager 0.217 (mame0217-611-ga7fec751ed) Usage: ./chdman info: displays information about a CHD ./chdman verify: verifies a CHD's integrity ./chdman createraw: create a raw CHD from the input file ./chdman createhd: create a hard disk CHD from the input file ./chdman createcd: create a CD CHD from the input file ./chdman createld: create a laserdisc CHD from the input file ./chdman extractraw: extract raw file from a CHD input file ./chdman extracthd: extract raw hard disk file from a CHD input file ./chdman extractcd: extract CD file from a CHD input file ./chdman extractld: extract laserdisc AVI from a CHD input file ./chdman copy: copy data from one CHD to another of the same type ./chdman addmeta: add metadata to the CHD ./chdman delmeta: remove metadata from the CHD ./chdman dumpmeta: dump metadata from the CHD to stdout or to a file ./chdman listtemplates: list hard disk templates For help with any command, run: ./chdman help <command>
The first thing you may note is the version number, this build of MAME was compiled around the time of the MAME 0.218 code freeze. Your version will differ.
chdman - MAME Compressed Hunks of Data (CHD) manager 0.217 (mame0217-611-ga7fec751ed)
After reviewing the version, you can take a look at the rest of the available options. At the bottom of all the options chdman tells us we can get further help on each of these commands.
For help with any command, run: ./chdman help <command>
Lets now get help on the verify command to see how we would use it to confirm the chd image we may want to use is valid for this version of MAME.
Note: CHDs may change between versions of MAME, they may be redumped or recreated or copied to bring the original data up to a new version of the format.
$ ./chdman help verify
chdman - MAME Compressed Hunks of Data (CHD) manager 0.217 (mame0217-611-ga7fec751ed)
Usage:
./chdman verify [options], where valid options are:
--input, -i <filename>: input file name (required)
--inputparent, -ip <filename>: parent file name for input CHD
We wont cover parent CHDs (--inputparent) yet, but we will verify a standard CHD in the next test. You will need to obtain a CHD (upload a test chd here slacker) to use next
$ ./chdman verify --input <chdfile.chd> chdman - MAME Compressed Hunks of Data (CHD) manager 0.217 (mame0217-611-ga7fec751ed) Raw SHA1 verification successful! Overall SHA1 verification successful!
It is worth mentioning here that CHDs must be told to be --compressed when they are created. The default create action for chdman (ignoring the name) is to build an uncompressed chd file. These cannot be verified because the verification is of the compressed vs raw data. Uncompressed CHDs are only raw data. There are other property differences between compressed and uncompressed chds we will cover later.
If you attempt to run a verify on an uncompressed image, MAME will let you know there was nothing to do.
$ ./chdman verify --input <uncompressed.chd> chdman - MAME Compressed Hunks of Data (CHD) manager 0.217 (mame0217-611-ga7fec751ed) No verification to be done; CHD is uncompressed Fatal error occurred: 0
We can also query information about the CHD as it was stored as metadata during creation.
$ ./chdman info --input chdfile.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.217 (mame0217-611-ga7fec751ed)
Input file: chdfile.chd
File Version: 5
Logical size: 104,857,600 bytes
Hunk Size: 4,096 bytes
Total Hunks: 25,600
Unit Size: 512 bytes
Total Units: 204,800
Compression: none
CHD size: 102,574 bytes
Metadata: Tag='GDDD' Index=0 Length=34 bytes
CYLS:256,HEADS:16,SECS:50,BPS:512.
Here we can see the metadata stored with the CHD data, it lists information we can use for setting up a machine and info the state of the CHD.
For example let's say this chd was being attached to the ct486 machine in MAME. The line CYLS:256,HEADS:16,SECS:50,BPS:512 can be used to configure the HDD in the bios. There is no HDD autodetect mechanism in the bios on the ct486 machine, it must be manually configured.
Converting between versions is completed with the copy functionality. You do not extract to raw then createhd/cd etc generally, it should be fine but if you forget to pass the --compress parameter the resulting CHD will be invalid. The copy function maintains the original condition of the chd and copies it to a new file. You can understand how to use this function but requesting help on it.
$ ./chdman help copy
chdman - MAME Compressed Hunks of Data (CHD) manager 0.217 (mame0217-611-ga7fec751ed)
Usage:
./chdman copy [options], where valid options are:
--output, -o <filename>: output file name (required)
--outputparent, -op <filename>: parent file name for output CHD
--force, -f: force overwriting an existing file
--input, -i <filename>: input file name (required)
--inputparent, -ip <filename>: parent file name for input CHD
--inputstartbyte, -isb <offset>: starting byte offset within the input
--inputstarthunk, -ish <offset>: starting hunk offset within the input
--inputbytes, -ib <length>: effective length of input in bytes
--inputhunks, -ih <length>: effective length of input in hunks
--hunksize, -hs <bytes>: size of each hunk, in bytes
--compression, -c <none|type1[,type2[,...]]>: which compression codecs to use (up to 4)
--numprocessors, -np <processors>: limit the number of processors to use during compression
Compressed vs Uncompressed
The main key difference (apart from size) in compressed vs uncompressed chds is (as of 252)
- compressed chds will generate a diff file on run to store the writable data instead of writing back to the original chd
- uncompressed chds are writable therefore all runtime write operations modify the original image.
CHD creation workflows
These are unofficial proposals only, to serve as a reminder for myself
Dump
- dump hdd to raw
- convert raw to chd compressed
- release compressed chd
Software packaging
- create raw hdd
- convert raw to chd uncompressed
- attach to machine, install software
- copy uncompressed chd to compressed chd
- release compressed chd
Within the software packaging scope you may want to consider parent chds where the OS installation is used as a parent and a copy is made for the remaining software installed (e.g. DOS in parent, DOOM in child) but I need to test this particular approach.
Documentation
For Linux systems chdman ships with a man page, this includes a lot more information on the tool and should be reviewed as a starting point for any operation. You will have this available with the packaged version fropm your distro. If you compile it yourself you will need to generate and install the man pages which is outside the scope of this document.
$ man chdman
CHDMAN(1) MAME Compressed Hunks of Data (CHD) manager CHDMAN(1)
NAME
chdman - MAME Compressed Hunks of Data (CHD) manager
SYNOPSIS
chdman option
DESCRIPTION
chdman is the (C)ompressed (H)unks of (D)ata (CHD) manager for MAME. It is a tool to manage CHD images.
You can also find the man page on self compiled version of MAME in the docs/man/ folder and read it with the man command
$ man docs/man/chdman.1
Examples
Create a 100MB hdd chd from a disk image on linux (currently testing)
$ IMAGE=disk.raw
$ dd if=/dev/zero of=$IMAGE bs=1024 count=0 seek=$((1024*100))
$ parted -s -a optimal -- $IMAGE unit B print free
Error: disk.raw: unrecognised disk label
Model: (file)
Disk disk.raw: 104857600B
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
$ ./chdman createhd --input $IMAGE --output disk.chd --chs 256,16,50 --sectorsize 512 --hunksize 4096 --size 102574 --compression none
$ ./chdman info -i disk.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.229 (mame0229-dirty)
Input file: disk.hdd.chd
File Version: 5
Logical size: 104,857,600 bytes
Hunk Size: 4,096 bytes
Total Hunks: 25,600
Unit Size: 512 bytes
Total Units: 204,800
Compression: none
CHD size: 6,139,904 bytes
Metadata: Tag='GDDD' Index=0 Length=34 bytes
CYLS:256,HEADS:16,SECS:50,BPS:512.