Poor man's parallel MPI debugger with tmux
Have you ever had a parallel MPI program, say a complicated FORTRAN code, and desired to debug it in parallel? Up to some days ago I used commands such mpirun -np 2 xterm -e "gdb my_mpi_program" and manually type gdb commands in every gdb window. This immediately becomes very tedious, and you cannot do it for more than 2 processes at the same time. There is a better solution: tmux. With this terminal multiplexer it is possible to broadcast all the character that you type on the keyboard on multiple terminals, so you can use it to begin a parallel debug session and step into the code of many mpi instances of your parallel program without having to type the commands in all the windows. How do I use it to debug an MPI program?
There is a script that you can find on github (or you can download it here) that is called tmpi that does all the hard work for you. Simply install tmux (something like sudo apt-get install tmux on Ubuntu-like linux distributions) and add this little configuration file, to enable/disable key broadcasting in an easy way. Put the following text inside the tmux configuration file ~/.tmux.conf: bind -n C-x setw synchronize-panes In this way enabling/disabling broadcasting is as simple as pressing Ctrl + B Ctrl + X. At this point the poor man's MPI debugger is simply the command: ./tmpi 2 "gdb my_mpi_program" where 2 is the number of MPI processes. That's it!
Tested with Ubuntu 18.04.3 LTS, gcc 7.4, Open MPI 2.1.1, gdb 8.1.0.20180409-git2019-09-09
Interesting software
- terminator very interesting terminal. It can split itself into the same window and broadcasts keystrokes to multiple terminals. Very useful!
- screen: mantains a terminal opened in beetween ssh connections/disconnections (like tmux)
- to be completed...