Real-time selective markerless tracking of forepaws of head fixed mice using deep neural networks

Real-time selective markerless tracking of forepaws of head fixed mice using deep neural networks, Here, we describe a system capable of tracking specific mouse paw movements at high frame rates (70.17 Hz) with a high level of accuracy (M = 0.95, SD = 0.01). Short-latency markerless tracking of specific body parts opens up the possibility of manipulating motor feedback.,

Here, we describe a system capable of tracking specific mouse paw movements at high frame rates (70.17 Hz) with a high level of accuracy (M = 0.95, SD = 0.01). Short-latency markerless tracking of specific body parts opens up the possibility of manipulating motor feedback. We present a software and hardware scheme built on DeepLabCut – a robust movement-tracking deep neural network framework – that enables real-time estimation of paw and digit movements of mice. Using this approach, we demonstrate movement-generated feedback by triggering a USB-CGPIO controlled LED when the movement of one paw, but not the other, selectively exceeds a pre-set threshold. The time delay between paw movement initiation and LED flash was M = 44.41 ms, SD = 36.39 ms, a latency sufficient for applying behaviorally-triggered feedback. We adapt DeepLabCut for real-time tracking as an open-source package we term DeepCut2RealTime. The package’s ability to rapidly assess animal behavior was demonstrated by reinforcing specific movements within water-restricted, head-fixed mice. This system could inform future work on a behaviorally triggered ‘closed loop’ brain-machine interface that could reinforce behaviors or deliver feedback to brain regions based on pre-specified body movements.

Significance statement We present a software and hardware scheme modified from DeepLabCut – a robust movement-tracking deep neural network framework – that enables real-time estimation of paw and digit movements of mice. Coupled to the body part tracking is the ability to rapidly trigger external events such as rewards upon the detection of specific behaviors. This system lays the groundwork for a behaviorally triggered ‘closed loop’ brain-machine interface that could reinforce behaviors and deliver feedback to brain regions based on pre-specified body movements.

, admin,