0. Overview

    There are three different variables used to coordinate the
  activities of the kernel, signal and user thread:
  under_kernel_control, emu_local_lock and sigs_handled. The first one
  is a member of the tss structure and is therefore accessible only
  within the kernel. The other ones are member of the exclusive page
  of a process and are there accessible in user and kernel space. They
  are used to coordinate activities between the user/signal thread and
  the kernel thread.

1.1 under_kernel_control

    under_kernel_control is a member of the tss structure and is used
  to coordinate the activity within the kernel. If its value is not
  zero its meaning is essentially: "I'm in the kernel and I'm
  executing some system code". under_kernel_control is set to one
  shortly after the kernel received a msg (either a system call, an
  exception or a page fault) and is reset just before sending the reply
  to the user.

    There is one case where the user is within the kernel but isn't
  under kernel control anymore. That happens after an execve system
  call. Execve deletes the old task and creates a new one that isn't
  under kernel control and will start to send new requests to the
  Linux kernel. This is signaled through setting under_kernel_control
  to zero within execve.

    Since under_kernel_control tells us whether a process is within
  the kernel or not it is used by schedule to select the next process
  to activate.

1.2 emu_local_lock and sigs_handled 

    In our redesigned version emu_local_lock and sigs_handled have new
   semantics. To be able to talk about the changes and discuss the new
   design I will first list the old semantics and the discuss the new
   one in terms of what has changed.

1.2.1 emu_local_lock and sigs_handled in our first version

    emu_local_lock and sigs_handled are used to coordinate the
  activities within the emulib and to make certain states within
  user space visible for the kernel. The usage of both variables is
  rather complex therefore I can't discuss one after the other and
  will rather explain their usage in certain situations.

    emu_local_lock is used to synchronize the activities regarding to
  the exclusive page and signal delivery. A thread making a system
  call writes its own state into the exclusive page before calling the
  server. Therefore it has to acquire the emu_local_lock before writing
  to the exclusive page.

    A signal thread propagating a signal to the user thread will force
  the user thread into a special system call which changes the
  exclusive page. Because the signal thread changes the exclusive page
  indirectly it has to acquire the emu_local_lock before starting any
  activity. If it can acquire the emu_local_lock it is save to
  manipulate the state of the user thread because the user will try to
  acquire the same lock before actually starting to make a system call.

    If it can't acquire the lock the user thread is just making a
  system call and the linux server will normally arrange for the
  thread to handle the signal upon return from system call. Therefore
  the signal thread could forget about the incoming signal and do
  nothing. But it may happen, that the linux server has finished
  signal delivery and the return to the user is delayed for some
  reasons. Then it may happen that another process is sending a new
  signal which is lost because the signal thread does nothing and the
  server has finished signal delivery. Therefore the server sets
  sigs_handled to indicate that the signal thread should not forget
  the signal. The signal thread waits until the emu_local_lock is free
  again and propagates the signal.

    Another problem is the relation between page fault handling and
  signal delivery. It may happen, that the linux server tries to
  handle a page fault but hasn't set under_kernel_control yet. Then
  another may send a message to the signal thread to propagate a
  signal which leads to a second message to the linux server which
  already tries to handle the page fault. Therefore the server locks
  the emu_local_lock to prevent signal delivery during page fault
  handling. 

    And last but not least there are some problems with page faults
  raised from emulation code. The emu_local_lock is usually locked if
  the user executes emulation code. Because of the server expects an
  upcoming signal if the emulib is locked he does nothing. Therefore
  we introduce a second meaning for sigs_handled. If the emulib is
  locked and sigs_handled is set we have a page fault within the
  emulation library that needs to be handled. Therefore the signal
  thread sets sigs_handled before manipulating the user stack and the
  server sets sigs_handled before returning to the user to allow the
  user thread to raise page faults while accessing its own stack.
 
1.2.1 emu_local_lock and sigs_handled in our new version

    In our new version we only have one server for all linux
  processes therefore some of our original problems have vanished.
    We don't need to cope with:

       - signal delivery during a page fault, because there is only
         one server which sets under_kernel_control just after
         receiving the page fault message. There are no other servers
         which can use the time between receiving a message and
         setting under_kernel_control to propagate a signal. That was
         possible due to our kernel lock:

		  server a			server b

					receive message
	    receive message		acquire kernel_lock
	    acquire kernel_lock      <-- propagate signal to user
	    set under_kernel_control    release kernel_lock
	 
	   Therefore we don't need to lock the emu_local_lock within our
	 page fault handling anymore.

       - lost signals on the way back to the user; again there is no
         parallel execution leading to lost signals. In our first
         version it was possible to have the following sequence:

 		  server a			server b

					receive message
	    receive message		acquire kernel_lock
	    acquire kernel_lock      
	    set under_kernel_control    
	    do something
	    reset under_kernel_control
	    release kernel_lock
					propagate signal to user
					release kernel_lock
	    send reply			send reply 

	 In this situation the signal thread forgets the signal
         because the emulib is locked. Therefore we needed
         sigs_handled. But with one server we don't need it anymore.

    The only semantics we need is emu_local_lock preventing user and
  signal thread from interfering with each other.

2. Where do we use these variables
2.1 under_kernel_control

  There are several places where under_kernel_control is set, reset
  and tested to achieve the described semantics:

  - dispatch message start (linux_ipc.c)

    if (tss->under_kernel_control)
    {
      /* woken  up while still under kernel control, check for
        SIG_KILL */
    }

  - page fault path (linux_ipc.c):

      tss->under_kernel_control = 1;
      handle_page_fault(...);
      tss->under_kernel_control = 0;	

  - system call entry and exit (linux_ipc.c):

      tss->under_kernel_control = 1;
      if (handle_message(...))
      {
        tss->under_kernel_control = 0;		
        return -1
      }
      if (! tss->under_kernel_control)
      {
  	  /* aha, someone has reset the client.  we must not
	     answer it, but instead wait for new instructions. */
	  return -1;
      }
      tss->under_kernel_control = 0;	

   
  - sys_idle/dsched: check whether we are already within the kernel or
                     not (process.c)

      if (active->tss.under_kernel_control)
	{
	  /* oops, we received a message but are already within the
	     kernel */	  
	  enter_kdebug("unexpected syscall");
	  continue;
	}

  - task_pair_create_user (part of fork/copy_thread and execve)

      /* ensure, the new process is marked as not under kernel control */
      t->under_kernel_control = 0;

  - task_pair_create (part of fork/copy_thread)

      /* oha, we are creating a kernel thread, mark it as under kernel
         control to make it selectable by schedule */
      t->under_kernel_control = 1;

