Troubleshooting MAME

MAME is a beast these days and when something doesn't work it can be difficult to know where to start, I am going to document some methods here, trying to help out on github issues. This is a teach a man to fish scenario.

Searching for Information

This will cover the simplest forms of finding information

Audit your roms

If you are receiving messages at launch about missing roms, this is the time to understand that roms evolve with MAME, i.e. dumps may be renamed or redumped over time. If you have downloaded a rom you found linked off a site it may be for an older version of MAME than what you have downloaded. Unless the site lists the compatible MAME version there is no way to tell _which_ version of MAME the set may be compatible with.

MAME however includes are database of compatible titles with each version either in the form of the main listxml output or a software list xml located in the hash directory. These databases contain the correct naming, set collections and file hashes to allow us and MAME to audit the rom you are trying to use.

These databases can be combined with tools like CLRMAME Pro (recommended) or | Romcenter to audit your romset and confirm its compatibility against each version.

Check your installation

Is your installation complete and up to date? Missing files or folders may be the cause of random issues

e.g. this cryptic message was the result of missing files from the bgfx folder which were discovered to during a git bisect by the reporter

MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0

X Error of failed request:  BadDrawable (invalid Pixmap or Window parameter)
  Major opcode of failed request:  149 ()
  Minor opcode of failed request:  4
  Resource id in failed request:  0x700000e
  Serial number of failed request:  285
  Current serial number in output stream:  290

Check the emulation status

MAME tells you the emulation status of each title in the UI (at the bottom of the screen and info tab) and at launch when using a frontend, if you have turned on skip_gameinfo in the ini file turn that off while troubleshooting

Confirm the emulation status if you are troubleshooting a machine or title to ensure it is not still considered work in progress

Read the documentation

MAME provides documentation in the forms listed below (nb: see the Documentation menu on mamedev.org)

Check social media

Check your general social media platforms for solutions to issues, you may find both community knowledge and devs floating around these platforms (nb: see the Community menu on mamedev.org)

Official

Unofficial

MAMETesters

MAMETesters is used as a community bug reporting tool more-so aligned to testing and reporting on specific title performance. You may search freely but will need an account to submit

github issues

Start your search with the github issues list, it may have been solved in a later code change or simply be documented there

Basic Troubleshooting

Simple troubleshooting with any type of build, some of these may be obvious but we will document them anyway

One step at a time

Make small change(s) to help identify the change that influenced the fix, you can report on this and help MAME progress. If you jump forward and change a lot document what was changed then backtrack until you break it again to try and isolate the problem.

Run in verbose mode

Run mame in -verbose mode and review the output

-verbose

Test an older version

Download (or build) and older version of MAME and test your configuration, find a build that works and report that to the MAME team, this can help identify a subset of changes that may be to blame.

Clean installation

Run with a clean installation, move your $HOME/.mame folder and run mame without the influences of any pre-existing configuration to see if the problem persists

BGFX

If troubleshooting BGFX issues, try a different chain and backend, up to version 0.236 the vulkan backend has been particularly troublesome whereas the more mature opengl backend is OK

Project

BGFX is a standalone project which is used by MAME to do the fancy display simulation but you already knew this.

Take a look for known project issues
Read the official documentation

Available Chains

You can see the available chains under ./bgfx/chains/ they are json files and will be called on the cli without the .json extension

UI Controls

See the sliders menu on the ingame menu (default key: tab) to change between chains

Consider your graphics stack

Look at your graphics stack,

what does it comprise?
Take into account any known issues with it

Include information in your graphics stack as part of you ticket.

Command Line Options

The cli options you will want are

-video bgfx -bgfx_backend (vulkan|opengl) -bgfx_chain <chain name>

Identify the running driver

Understand the impacted driver if you are looking at sf2ce that runs on the cps1 driver, you can identify this in your report

mame -listxml | awk '/name="sf2ce"/{print $3}'
sourcefile="cps1.cpp">

If you can launch the game refer to the Driver: entry on the game info screen

Further Troubleshooting

Compiling

Compile your own build, this will give you the control you need to start narrowing down when the issue first presented.

This is a large topic please see the official documentation

I currently compile with

-j12 STRIP_SYMBOLS=1 NOWERROR=1 CFLAGS='-U_FORTIFY_SOURCE' TOOLS=1 REGENIE=1

Compile options
Option	Usage	Reference
j12	Threaded compile, can use up to (n)+1 cores	-
STRIP_SYMBOLS=1	Do not include debugging symbols	-
NOWERROR=1	Disable Warnings as Errors	bannister.org
CFLAGS='-U_FORTIFY_SOURCE'	This is a compiler flag that is required to work around particulars with gcc	bannister.org Jira
TOOLS=1	Build external tools (i.e. chdmam, floptool)	-
REGENIE=1	Regenerate settings for build	-

Compile a single driver

I can be quicker while troubleshooting to compile a single driver instead of everything

Shaders

Compile the shaders, not all shaders are compiled at any time so if you are testing CRT simulation issues compile the latest shaders

make shaders

Debug Symbols

Symbols help to give legible/useful output when debugging mame with an external debugger, retaining symbols will result in a larger executable, to retain symbols set STRIP_SYMBOLS to 0 when compiling (the default)

STRIP_SYMBOLS=0

Debug Builds

Once you have learnt to compile your own builds, you can now have the opportunity of building a debug version, to build a debug version pass DEBUG=1 to your build flags

DEBUG=1

This will build mamed (note the d)

When run you can enabled the debugger with -debug on the command line

Check your buildenv versiom(s)

I currently use gcc, other tools are available (see MAME Documentation Compiling and Github)

$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Timelines

Look at the release date of the version you have identified as problematic and/or the settings which cause the issue, take that release date and have a look at the changes to the source code around at this time, was there updates to the feature you are having issues with?

Review the source code

Pull down a copy of the source code and start to take a look over it, perhaps by now you have identified a particular file that was changed for a feature where you have identified a crash, perhaps you just have an error message.

grep the source code for the error, what does the code seem to do? Trace this back through files until you are either completely lost or find what you are looking for.

Use the cppreference to try and understand what is going.

External Tools

git

git is the revision control tool used by the project

bisect

TBA

gdb

Debugging with gdb can be useful to get information (at a minimum a backtrace) to include in you issue report.

Missing Symbols

If you have not compiled mame with symbols do so now, you will be able to tell if you do not have symbols enabled in your build if you get a message when started

gdb ./mame
--- snip ---
Reading symbols from ./mame...
(No debugging symbols found in ./mame)

or your backtrace looks like this note the "in ??"

Thread 1 "mame" received signal SIGSEGV, Segmentation fault.
0x00007ffff2444cb0 in ?? () from /lib/x86_64-linux-gnu/libGLX_mesa.so.0
(gdb) bt
#0  0x00007ffff2444cb0 in ?? () from /lib/x86_64-linux-gnu/libGLX_mesa.so.0
#1  0x00007ffff24331d9 in ?? () from /lib/x86_64-linux-gnu/libGLX_mesa.so.0
#2  0x00007ffff7f25331 in ?? () from /lib/x86_64-linux-gnu/libSDL2-2.0.so.0
#3  0x00007ffff7e9d9cf in ?? () from /lib/x86_64-linux-gnu/libSDL2-2.0.so.0
#4  0x000055555ecaf0bf in ?? ()
#5  0x000055555ecaf1ad in ?? ()
#6  0x000055555ec79ad7 in ?? ()
#7  0x000055555ec82be1 in ?? ()
#8  0x000055555ec82eee in ?? ()
#9  0x000055555ec825f8 in ?? ()
#10 0x000055555ec78c99 in ?? ()
#11 0x0000555562d1a9be in ?? ()
#12 0x000055555ed6d573 in ?? ()
#13 0x0000555562c57ead in ?? ()
#14 0x0000555562c5944e in ?? ()
#15 0x000055555ed70729 in ?? ()
#16 0x000055555ee1c09e in ?? ()
#17 0x000055555ee1c332 in ?? ()
#18 0x000055555ed6e16a in ?? ()
#19 0x000055555a08212f in main ()
(gdb)

Start Using gdb

Start gdb with mame, loading mamed directly will auto load the symbols

gdb ./mamed
--- snip ---
Reading symbols from ./mame...

Confirm mame is loaded with list

(gdb) list
1	// license:BSD-3-Clause
2	// copyright-holders:Aaron Giles
3	/***************************************************************************
4	
5	    mame.cpp
6	
7	    Specific (per target) constants
8	
9	****************************************************************************/
10	
(gdb)

Run mame in windowed mode from within gdb (any mame params will work here so use as you like)

run -window

Perform the actions you need to make MAME fail for debugging and the gdb will halt execution

--Type <RET> for more, q to quit, c to continue without paging--

Press the Return to drop to a prompt

Thread 1 "mamed" received signal SIGBUS, Bus error.
0x00007ffff242f776 in ?? () from /lib/x86_64-linux-gnu/libGLX_mesa.so.0
(gdb)

View a backtrace of the failure with the bt command

Thread 1 "mamed" received signal SIGBUS, Bus error.
0x00007ffff242f776 in ?? () from /lib/x86_64-linux-gnu/libGLX_mesa.so.0
(gdb) bt
#0  0x00007ffff242f776 in ?? () from /lib/x86_64-linux-gnu/libGLX_mesa.so.0
#1  0x00007ffff2444c96 in ?? () from /lib/x86_64-linux-gnu/libGLX_mesa.so.0
#2  0x00007ffff24331d9 in ?? () from /lib/x86_64-linux-gnu/libGLX_mesa.so.0
#3  0x00007ffff7f25331 in ?? () from /lib/x86_64-linux-gnu/libSDL2-2.0.so.0
#4  0x00007ffff7e9d9cf in ?? () from /lib/x86_64-linux-gnu/libSDL2-2.0.so.0
#5  0x000055555f6a732f in renderer_sdl1::~renderer_sdl1() ()
#6  0x000055555f6a741d in renderer_sdl1::~renderer_sdl1() ()
#7  0x000055555f670ea7 in sdl_window_info::~sdl_window_info() ()
#8  0x000055555f67a4a9 in sdl_event_manager::process_window_event(running_machine&, SDL_Event&) ()
#9  0x000055555f67a7ee in sdl_event_manager::process_events(running_machine&) ()
#10 0x000055555f679e88 in sdl_osd_interface::poll_inputs(running_machine&) ()
#11 0x000055555f66f91e in sdl_osd_interface::input_update() ()
#12 0x0000555563a0f507 in video_manager::frame_update(bool) ()
#13 0x000055555f769253 in mame_machine_manager::create_ui(running_machine&) ()
#14 0x000055556394374d in running_machine::start() ()
#15 0x0000555563944d00 in running_machine::run(bool) ()
#16 0x000055555f76c549 in mame_machine_manager::execute() ()
#17 0x000055555f81c2ff in cli_frontend::start_execution(mame_machine_manager*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) ()
#18 0x000055555f81c592 in cli_frontend::execute(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) ()
#19 0x000055555f769f2a in emulator_info::start_frontend(emu_options&, osd_interface&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) ()
#20 0x000055555a08697f in main () at ../../../../../src/mame/mame.cpp:23
(gdb)

You probably noted in the above output we are missing symbols for some systems libraries (sdl and mesa) to debug further we will need to enable the appropriate symbols (see distro doco) for these libraries.

Include the backtrace output in your report to assist with further troubleshooting

valgrind

WIP:: Valgrind "is an instrumentation framework for building dynamic analysis tools." ... yep sure thing, if you are asked to run a Memory Analyser by one of the devs to help out this may be your tool. I am currently testing this after being asked the same | here

Currently testing with

$ valgrind --track-origins=yes --tool=memcheck --leak-check=yes --show-reachable=yes --track-fds=yes --log-file=$HOME/logs/valgrind-$(date +'%s').log ./mamed -numscreens 2 -video opengl -window

mtrace

WIP:: requires modifying MAME to include mtrace support(?)

Final thoughts

Don't stress

The MAMEdev team know this source, you may not and it is a huge project to jump into. Raise an issue with your findings, a dev may make suggestions on where to look next or you may have managed to do 70% of the investigation for them, they do appreciate it.

Own your ignorance and ask for help, it is a learning opportunity that may have long term impact.

Guides:MAME - Troubleshooting