To detect deception in user's statements, I chose multi-modal approach, which was shown to be very effective to detect human's lies. Previous works have demonstrated that acoustic features and facial features are useful cues to decide whether someone is lying or not. Thus, in this research, I decide to combine acoustic and facial features to classify the user's statement into lie or truth. To combine acoustic and facial features, I chose hierachical combination method, which is proposed in a previous work for emotion detection; this research is among the first to apply this combination method for the task of deception detection. Evaluation results showed that hierachical combination outperforms baseline methods of combining acoustic and facial features. However, the highest accuracy of detection is about 70%, which means we cannot reach a concrete conclusion whether user is lying or not.
For dialog management, I have designed a dialog behavior for the system on how should system react when user is lying. Using this dialog behavior, the system reacts to user based on both user's action and user's deception. However, as mentioned above, we cannot have a exact conclusion about user's deception. Therefore, the dialog decision process of the system is modeled using partially observable Markov decision process (POMDP.) To train the system to acts according to the designed dialog behavior, reinforcement learning was chosen. The reward that system receives when interacting with user is defined based on the design dialog behavior and system also receives additional reward when it successfully persuade user to agree with the system.
In the evaluation of the system, first I show the learning curve created from reward that system received in the traning phase. Results proved that POMDP and reinforcement learning were successfully used to train the dialog manager of the system. Using evaluation with simulated user, I showed that the proposed system outperforms baseline in term of negotiation performance. Using evaluation with real human-to-human dialogs, I showed that the system also achieved better accuracy than baselin in term of dialog acts selection.
In conclusion, this research proposed a negotiaion dialog system that detect user's lies and user this deception information for dialog management. Evaluation results proved that the proposed system outperforms baseline system that react to user without using deception information.
In the past few years, there have been an increasing number of works in negotiation dialog topic. A typical example of negotiation is the conversation between doctor and patient where doctor negotiates with the patient to find a treatment plan that satisfy both of them. In this negotiation, patient can tell lies so that doctor changes the treatment plan according to the patient's preferences. According to my knowledge, almost all of existing negotiation systems assume that its interlocutor (the user) is telling the truth. As a result, we cannot use existing systems to act as a doctor and handle the doctor-patient conversation. In this research, I proposed a negotiation dialog system that detects user's lies and designed a dialog behavior on how should system react when user is lying. I showed that we can use partially observable Markov decision process to model this conversation and use reinforcement learning to train the system to act according to the designed behavior. Experimental evaluations proved that the proposed system outperforms baseline negotiation system that does not use user's deception information for dialog management.