A Multiple Instance Learning and Relevance Feedback Framework for Retrieving Abnormal Incidents in Surveillance Videos

Abstract

This paper incorporates coupled hidden Markov models (CHMM) with relevance feedback (RF) and multiple-instance learning (MIL) for retrieving various abnormal events in surveillance videos. CHMM is suitable for modeling not only the object’s behavior itself but also the interactions between objects. In addition, to address the challenges posed by the “semantic gap” between high level human concepts and the machine-readable low level visual features, we introduce relevance feedback (RF) to bridge the semantic gap by progressively collecting feedback from the user, which allows the machine to discover the semantic meanings of an event by exploring the patterns behind lowlevel features. The adopted multiple-instance learning algorithm enables the proposed framework to provide a user-friendly video retrieval platform with the use of queryby- example (QBE) interface. The experimental results show the effectiveness of the proposed framework in detecting “chasing”, “fighting”, and “robbery” events by demonstrating the increase of retrieval accuracy through iterations and comparing with other methods. By tightly integrating these key components in a learning system, we ease the surveillance video retrieval problem.