This directory contains the components necessary to perform remote
debugging of the Linux kernel using two machines and gdb.

The setup goes as follows:

Designate one machine as the "development" machine.  This is the
machine on which you run your compiles and which has your source
code for the kernel.  Designate a second machine as the "target"
machine.  This is the machine that will run your experimental
kernel.

The two machines will be connected together via a serial line out
one or the other of the COM ports of the PC.  You will need a modem
eliminator and the appropriate cables.

On the development machine you need to apply the patch for the gdb
hooks.  The patches are contained in the file "patches.gdbstub".
You can do that as follows.  For purposes of example, I will assume
that you have un-tarred the gdbstub stuff into the directory
/usr/src/gdbstub.

	cd /usr/src/linux
	patch -p1 </usr/src/gdbstub/patches.gdbstub 2>patches.errs

This should produce an error file that is substantially similar to
/usr/src/gdbstub/patches.errs from the tar archive.

If the patch looks successful, then you are ready for a kernel
rebuild.  Do a "make clean" before you rebuild your kernel.  The patch
inserts a "-g" into the master Makefile so as to compile the kernel
with debugging info set.  You need to rebuild the whole thing so that
all the pieces have debugging info.

Copy this kernel to your target machine using your usual procedures.
I usually arrange to copy development:/usr/src/linux/arch/i386/boot/zImage
to /vmlinuz on the TARGET machine via a LAN based NFS access.  That is,
I run the cp command on the target and copy from the development machine
via the LAN.  Run Lilo on the new kernel on the target machine so that it
will boot.  Then boot the kernel on the target machine.

Back on the development machine, cd /usr/src/gdbstub and enter "make".
This will build a small utility program that you should copy over
to your target machine, probably into /usr/local/bin.  This utility
program, gdbstub, is run on the target machine to activate the kernel
hooks for the debugger.  It is invoked as follows:


    gdbstub [-s speed] [-t tty-dev]
    defaults:  /dev/ttyS0 with speed unmodified by gdbstub

Don't run the program just yet.  We'll get to that in a bit.

Decide on which tty port you want the machines to communicate, then
cable them up back-to-back using the null modem.  COM1 is /dev/ttys0
and COM2 is /dev/ttyS1.

On the development machine, create a file called .gdbinit in the
directory /usr/src/linux.  You will probably find it handy to add
the stuff from the file /usr/src/gdbstub/gdbinit to your .gdbinit
file.  You can include all your favorite gdb macros in this file.

Assuming that you added my gdbinit stuff to your .gdbinit, edit .gdbinit
and find the section that looks like this:


	define rmt
	target remote /dev/ttyS0
	end

Change the "target" definition so that it specifies the tty port that
you intend to use.

On your development machine, edit the file /etc/rc.d/rc.local and
add the following line to it (assuming that you settled on /dev/ttyS0):

	stty 38400 </dev/ttyS0

Or, whatever data rate you decide to run the line.  Also, type in this
command so as to make the data rate on your tty port what you want it
to be.

On the target machine I find it helpful to create shell script file
named "debug" in the root home directory with the following contents:

	gdbstub -s 38400 -t /dev/ttyS0 <<EOF
	<blank line>
	EOF

This runs the gdbstub proram and gives it the carriage return that
it prompts for.  This sets the data rate from the target machine's side.

You are now ready to try it out.

On your target machine, freshly rebooted with your gdbstub-equipped
kernel, type "debug" in the root directory.  The system will appear
to hang with some messages on the screen from the debug stub.  What
it is doing is waiting for contact from the development machine.

On your development machine, cd /usr/src/linux and enter "gdb vmlinux".
When gdb gets the symbols loaded and prompts you, enter "rmt" (that's
the macro from the .gdbinit file that you just edited).  If everything
is working correctly you should see gdb print out a few lines indicating
that a breakpoint has been taken.  It will actually show a line of
code in the target kernel inside the gdbstub activation code.

The gdb interaction should look something like this:

	linux-dev:/usr/src/linux# gdb vmlinux
	GDB is free software and you are welcome to distribute copies of it
	 under certain conditions; type "show copying" to see the conditions.
	There is absolutely no warranty for GDB; type "show warranty" for details.
	GDB 4.15.1 (i486-slackware-linux), 
	Copyright 1995 Free Software Foundation, Inc...
	(gdb) rmt
	breakpoint () at i386-stub.c:750
	750     }
	(gdb) 


You can now use whatever gdb commands you like to set breakpoints.
Enter "continue" to start your target machine executing again.  At this
point the target system will run at full speed until it encounters
your breakpoint or gets a segment violation in the kernel, or whatever.

Debugging hints:

There is unfortunately no way of breaking into the kernel if it is
in a loop, so if this happens to you then you need to place exploratory
breakpoints into the kernel to find out where it is looping.

If you can possibly do so, when you are "done" with a debugging session,
use the "continue" command as the last thing so that you can reboot your
target kernel gracefully (saves fsck time on reboot).  When gdb is watching
the target machine, rather than your keyboard, you can type two control-Cs
and it will come back to you.  You can then quit out of gdb.

If it doesn't work, you will have to troubleshoot it.  Do the easy things
first like double checking your cabling and data rates.  You might
try some non-kernel based programs to see if the back-to-back connection
works properly.  Just something simple like cat /etc/hosts >/dev/ttyS0
on one machine and cat /dev/ttyS0 on the other will tell you if you
can send data from one machine to the other.  There is no point in tearing
out you hair in the kernel if the line doesn't work.

All of the real action takes place in the file
/usr/src/linux/arch/i386/kernel/i386-stub.c.  That is the code on the target
machine that interacts with gdb on the development machine.  In gdb you can
turn on a debug switch with the following command:

	set remotedebug

This will print out the protocol messages that gdb is exchanging with
the target machine.

Another place to look is /usr/src/linux/drivers/char/serialstub.c.
That is the code that talks to the serial port on the target side.
There might be a problem there.

If you are really desperate you can use printf debugging in the
gdbstub code in the target kernel until you get it working.  In particular,
there is a global variable in /usr/src/linux/arch/i386/kernel/i386-stub.c
named "remote_debug".  Compile your kernel with this set to 1, rather
than 0 and the debug stub will print out lots of stuff as it does
what it does.

I picked up this code from a gdb source code release and modified it
to make it work.  Most of the hard work was done by the original
authors.

If you make some really cool modification to this stuff, or if you 
fix a bug, please let me know.

David Grothe
<dave@gcom.com>
