PhD defense : Vision-based approaches for surgical activity recognition using laparoscopic and RGBD videos

PhD defense : Andru Putra TWINANDA

Team : AVR

Title : Vision-based approaches for surgical activity recognition using laparoscopic and RGBD videos

Abstract : The main objective of this thesis is to address the problem of activity recognition in the operating room (OR). Activity recognition is an essential component in the development of context-aware and data management systems, which will allow various applications, such as automated assistance during difficult procedures and automatic report generation. Here, we focus on vision-based approaches since cameras are a common source of information to observe the OR without disrupting the surgical workflow. Specifically, we propose to use two complementary types of videos: laparoscopic and OR-scene RGBD videos. Laparoscopic videos record in detail the tool-tissue interactions inside the patients during minimally invasive surgeries, while OR-scene RGBD videos are recordings from a multi-view ceiling-mounted camera system which captures the activities occurring in the whole room. Despite the vast literature on activity recognition in computer vision, activity recognition in surgical setups is still far from being solved. The OR is a very particular and challenging environment where the objects are of similar colors and the scene contains a lot of clutter and occlusions. Furthermore, laparoscopic videos capture a scene completely different from the conventional videos used in the computer vision community, which typically contain humans in the scene. The laparoscopic videos also contain inherent visual challenges, such as rapid camera motion and specular reflection.

In this thesis, we investigate how state-of-the-art computer vision approaches perform on these videos and propose novel approaches to overcome  some of the aforementioned challenges. First, we establish recognition pipelines to address activity recognition problems on both laparoscopic and OR-scene RGBD videos using the bag-of-word (BOW) approach, Support Vector Machines (SVM), and hidden Markov models (HMM). Second, we propose an extension to the BOW approach used on multi-view RGBD data to retain more spatial and temporal information during the encoding process. Ultimately, to alleviate the difficulties in manually designing the visual features to represent the data, we design deep learning architectures for surgical activity recognition. We also propose an end-to-end approach with an LSTM network to eliminate the need for SVM and HMM. To evaluate our proposed approaches, we generate large datasets of real surgical videos, including either laparoscopic videos or multi-view RGBD videos. The results demonstrate that the proposed approaches outperform the state-of-the-art methods in performing surgical activity recognition on these new datasets.

This thesis was directed by Michel de Mathelin.

The presentation in english will take place at IRCAD auditorium Hirsch on Friday, January 27th 2016 at 1.30 pm.