There are various kinds of security related works. Our goal is to create a novel IDS which uses the state of the process to distinguish intrusions or malicious actions. Though there are many security related softwares such as anti-viruses, firewalls, IDS and so on, most of them have predefined signatures, created by the vendor, which are compared with the suspicious program or data (e.g. network packets, files). The vendors of these softwares work their socks off to catch up with the endless "tsunami" of new viruses and attacks.
Our work is to create a system which will not use simple byte-to-byte matching of predefined signatures. Instead it uses the traces of the process for detection. To be more precise, our system will use the NT System Service traces to distinguish between normal and abnormal behaviours. An NT System Service corresponds to a system call on UNIX and UNIX compatible operating systems (e.g. Linux, BSD).
The NT System Service is explained in more detail here.
The basic idea of using system calls for intrusion detections is not new and has been around for quite long in the UNIX world. Operating systems such as Linux or BSD have an open-source kernel and thus the internals are well-known. This lead to many works on system call based IDS. On the other hand Microsoft does not want programmers to be able to look inside their softwares (which is nothing special for commercial software vendors) and thus it is difficult for the same kind of work to be done.
Our work is still under development and currently, the software is capable of capturing the NT System Service traces.
There are several ways to implement this feature. One is to intercept the switch from user mode to kernel mode (by modifying the Interruption Descriptor Table (IDT) or the SYSENTER_EIP_MSR for SYSENTER). The second is rewriting the table which keeps the address of the NT System Services (which is called the System Service Descriptor Table (SSDT)). The last is to modify each NT System Services by rewriting the binary code itself.
We use the first approach which is to overwrite the SYSENTER_MIP_MSR. This register indicates the address to jump to when executing SYSENTER. The address will be changed to our code which will keep track of the NT System Service called by the target process.
For more detail, please read my graduation thesis(English).
The below are slides which I've created for my talk (in Japanese, sorry).