Guest Virtual Machine (VM) operation is one of main threat vectors in the cloud system. Therefore, a good VM monitoring system is a necessary. We argue that a good guest VM monitoring system should be performed outside the monitored VM and the monitoring process should not rely on the monitored VM to provide the observation data. Moreover, to ensure the monitoring quality, the observation data should not lose the guest VM operation context and therefore it should minimise the semantic gap issue. Finally, the monitoring cost, in term of computing resource usage, for either the host and the guest VM should be at minimum.
In this thesis, we propose a guest VM monitoring method that can work from the host independently, while preserving a high semantic level without high computation cost for either the host and the guest VM. We propose the use of static instrumentation technique on the Virtual Machine Monitor (VMM) and using the collected tracepoints to dynamically collect the guest VM operation data. We use the tracepoints pattern of each guest VM to decide the status of guest VM operation.
We conducted experiment on userspace layer and kernel layer of KVM-Qemu hypervisor system. We use multiple real life scenarios, both for normal scenarios and attack scenarios (either host-based attack, network-based attack and virtualization specific attack). One of our study is the first to tackle the detection problem of Flush+Flush Cache-based Side Channel attack. We compare the use of several machine learning method to analyse the data. We gave insight on each given results. Our conclusion gives the advantage and disadvantage of our proposal.