This book is a small endeavor towards creating the possibility for the machines to understand human behavior and emotions. While significant work is being done in order to develop empathetic agents which can identify emotions of the user through eye gaze and facial expressions, a neglected area, especially in the pedagogical context, is the use of voice for detection of alertness, fatigue and emotions. However, such a system has advantages - the ability to work where visual monitoring is expensive, in darkness or where mobile devices cannot provide adequate visual feedback. This book proposes a model for a system capable of identifying emotions as well as alertness and fatigue based entirely on voice interaction. Furthermore, it proposes the development of an engine which can intelligently improve its prediction of emotions and cognitive states based on earlier interaction, and suggest appropriate measures to improve emotions, reduce distractions and mitigate fatigue.