A multi-behavior recognition method for macaques based on improved SlowFast network_Journal of Biomedical Engineering

Authors：

ZHONG Weifeng ¹ , XU Zhe ^1,2 , ZHU Xiangyu ² ,  MA Xibo ²

1. School of Automation, Harbin University of Science and Technology School, Harbin 150000, P. R. China;
2. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing100000, P. R. China;

Corresponding author：

MA Xibo, Email: xibo.ma@ia.ac.cn

Keywords：

Drug safety assessment; Behavior recognition; Macaque behavior dataset; Convolutional neural network

DOI：

10.7507/1001-5515.202112014

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Macaque is a common animal model in drug safety assessment. Its behavior reflects its health condition before and after drug administration, which can effectively reveal the side effects of drugs. At present, researchers usually rely on artificial methods to observe the behavior of macaque, which cannot achieve uninterrupted 24-hour monitoring. Therefore, it is urgent to develop a system to realize 24-hour observation and recognition of macaque behavior. In order to solve this problem, this paper constructs a video dataset containing nine kinds of macaque behaviors (MBVD-9), and proposes a network called Transformer-augmented SlowFast for macaque behavior recognition (TAS-MBR) based on this dataset. Specifically, the TAS-MBR network converts the red, green and blue (RGB) color mode frame input by its fast branches into residual frames on the basis of SlowFast network and introduces the Transformer module after the convolution operation to obtain sports information more effectively. The results show that the average classification accuracy of TAS-MBR network for macaque behavior is 94.53%, which is significantly improved compared with the original SlowFast network, proving the effectiveness and superiority of the proposed method in macaque behavior recognition. This work provides a new idea for the continuous observation and recognition of the behavior of macaque, and lays the technical foundation for the calculation of monkey behaviors before and after medication in drug safety evaluation.

Citation： ZHONG Weifeng, XU Zhe, ZHU Xiangyu, MA Xibo. A multi-behavior recognition method for macaques based on improved SlowFast network. Journal of Biomedical Engineering, 2023, 40(2): 257-264. doi: 10.7507/1001-5515.202112014 Copy

1.	Harrer S, Shah P, Antony B, et al. Artificial intelligence for clinical trial design. Trends Pharmacol Sci, 2019, 40(8): 577-591.
2.	Wang H, Brown P C, Chow E C Y, et al. 3D cell culture models: drug pharmacokinetics, safety assessment, and regulatory consideration. Clin Transl Sci, 2021, 14(5): 1659-1680.
3.	Plagenhoef M R, Callahan P M, Beck W D, et al. Aged rhesus monkeys: cognitive performance categorizations and preclinical drug testing. Neuropharmacology, 2021, 187: 108489.
4.	Wu L, Wu D, Chen J, et al. Intranasal salvinorin a improves neurological outcome in rhesus monkey ischemic stroke model using autologous blood clot. J Cereb Blood Flow Metab, 2021, 41(4): 723-730.
5.	童安炀, 唐超, 王文剑. 基于双流网络与支持向量机融合的人体行为识别. 模式识别与人工智能, 2021, 34(9): 863-870.
6.	Klaser A, Marszałek M, Schmid C. A spatio-temporal descriptor based on 3D-gradients//19th British Machine Vision Conference (BMVC), Leeds: British Machine Vision Association, 2008, 275: 1-10.
7.	Laptev I, Marszalek M, Schmid C, et al. Learning realistic human actions from movies//IEEE Conference on Computer Vision and Pattern Recognition, Alaska: IEEE, 2008. DOI: 10.1109/CVPR.2008.4587756.
8.	Dalal N, Triggs B, Schmid C. Human detection using oriented histograms of flow and appearance//European Conference on Computer Vision, Graz: IEEE, 2006: 428-441.
9.	Messing R, Pal C, Kautz H. Activity recognition using the velocity histories of tracked keypoints//2009 IEEE 12th International Conference on Computer Vision, Kyoto: IEEE, 2009: 104-111.
10.	Wang H, Klser A, Schmid C, et al. Dense trajectories and motion boundary descriptors for action recognition. International Journal of Computer Vision, 2013, 103(1): 60-79.
11.	周波, 李俊峰. 结合目标检测的人体行为识别. 自动化学报, 2020, 46(9): 1961-1970.
12.	Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos//Advances in Neural Information Processing Systems, Montreal: IEEE, 2014: 568-576.
13.	Liu P, Lyu M, King I, et al. Selflow: self-supervised learning of optical flow//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, California: IEEE, 2019: 4571-4580.
14.	Tran D, Bourdev L, Fergus R. Learning spatiotemporal features with 3D convolutional networks//Proceedings of the IEEE International Conference on Computer Vision, Santiago: IEEE, 2015: 4489-4497.
15.	Christoph R, Pinz F A. Spatiotemporal residual networks for video action recognition//Advances in Neural Information Processing Systems, Barcelona: IEEE, 2016: 3468-3476.
16.	He K, Zhang X, Ren S. Deep residual learning for image recognition//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas: IEEE, 2016: 770-778.
17.	Donahue J, Anne Hendricks L, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas: IEEE, 2015: 2625-2634.
18.	Feichtenhofer C, Fan H, Malik J. SlowFast networks for video recognition//Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul: IEEE, 2019: 6202-6211.
19.	Li D, Zhang K, Li Z, et al. A spatiotemporal convolutional network for multi-behavior recognition of pigs. Sensors (Basel), 2020, 20(8): 2381-2399.
20.	Li C, Zhong Q, Xie D, et al. Skeleton-based action recognition with convolutional neural networks//2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Hong Kong: IEEE, 2017: 597-600.
21.	Girdhar R, Carreira J, Doersch C, et al. Video action transformer network//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, California: IEEE, 2019: 244-253.
22.	Vaswani A, Shazeer N, Parmar N. Attention is all you need//Advances in Neural Information Processing Systems, California: IEEE, 2017: 5998-6008.
23.	Tao L, Wang X, Yamasaki T. Motion representation using residual frames with 3D CNN//IEEE International Conference on Image Processing, Abu Dhabi: IEEE, 2020: 1786-1790.
24.	Bala P C, Eisenreich B R, Yoo S B M, et al. Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Nat Commun, 2020, 11(1): 4560.
25.	Tang D H, Wang C Y, Huang X, et al. Inosine induces acute hyperuricaemia in rhesus monkey (Macaca mulatta) as a potential disease animal model. Pharmaceutical Biology, 2021, 59(1): 175-182.
26.	Carreira J, Zisserman A. Quo vadis, action recognition? a new model and the kinetics dataset//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Venice: IEEE, 2017: 6299-6308.
27.	Tran D, Wang H, Torresani L, et al. A closer look at spatiotemporal convolutions for action recognition//Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Salt Lake City and UT: IEEE, 2018: 6450-6459.
28.	Wang L, Xiong Y, Wang Z, et al. Temporal segment networks: towards good practices for deep action recognition//European Conference on Computer Vision, Amsterdam: IEEE, 2016: 20-36.
29.	Bertasius G, Wang H, Torresani L. Is space-time attention all you need for video understanding//International Conference on Machine Learning (ICMC), 2021. arXiv: 2102.05095.

1. Harrer S, Shah P, Antony B, et al. Artificial intelligence for clinical trial design. Trends Pharmacol Sci, 2019, 40(8): 577-591.
2. Wang H, Brown P C, Chow E C Y, et al. 3D cell culture models: drug pharmacokinetics, safety assessment, and regulatory consideration. Clin Transl Sci, 2021, 14(5): 1659-1680.
3. Plagenhoef M R, Callahan P M, Beck W D, et al. Aged rhesus monkeys: cognitive performance categorizations and preclinical drug testing. Neuropharmacology, 2021, 187: 108489.
4. Wu L, Wu D, Chen J, et al. Intranasal salvinorin a improves neurological outcome in rhesus monkey ischemic stroke model using autologous blood clot. J Cereb Blood Flow Metab, 2021, 41(4): 723-730.
5. 童安炀, 唐超, 王文剑. 基于双流网络与支持向量机融合的人体行为识别. 模式识别与人工智能, 2021, 34(9): 863-870.
6. Klaser A, Marszałek M, Schmid C. A spatio-temporal descriptor based on 3D-gradients//19th British Machine Vision Conference (BMVC), Leeds: British Machine Vision Association, 2008, 275: 1-10.
7. Laptev I, Marszalek M, Schmid C, et al. Learning realistic human actions from movies//IEEE Conference on Computer Vision and Pattern Recognition, Alaska: IEEE, 2008. DOI: 10.1109/CVPR.2008.4587756.
8. Dalal N, Triggs B, Schmid C. Human detection using oriented histograms of flow and appearance//European Conference on Computer Vision, Graz: IEEE, 2006: 428-441.
9. Messing R, Pal C, Kautz H. Activity recognition using the velocity histories of tracked keypoints//2009 IEEE 12th International Conference on Computer Vision, Kyoto: IEEE, 2009: 104-111.
10. Wang H, Klser A, Schmid C, et al. Dense trajectories and motion boundary descriptors for action recognition. International Journal of Computer Vision, 2013, 103(1): 60-79.
11. 周波, 李俊峰. 结合目标检测的人体行为识别. 自动化学报, 2020, 46(9): 1961-1970.
12. Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos//Advances in Neural Information Processing Systems, Montreal: IEEE, 2014: 568-576.
13. Liu P, Lyu M, King I, et al. Selflow: self-supervised learning of optical flow//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, California: IEEE, 2019: 4571-4580.
14. Tran D, Bourdev L, Fergus R. Learning spatiotemporal features with 3D convolutional networks//Proceedings of the IEEE International Conference on Computer Vision, Santiago: IEEE, 2015: 4489-4497.
15. Christoph R, Pinz F A. Spatiotemporal residual networks for video action recognition//Advances in Neural Information Processing Systems, Barcelona: IEEE, 2016: 3468-3476.
16. He K, Zhang X, Ren S. Deep residual learning for image recognition//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas: IEEE, 2016: 770-778.
17. Donahue J, Anne Hendricks L, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas: IEEE, 2015: 2625-2634.
18. Feichtenhofer C, Fan H, Malik J. SlowFast networks for video recognition//Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul: IEEE, 2019: 6202-6211.
19. Li D, Zhang K, Li Z, et al. A spatiotemporal convolutional network for multi-behavior recognition of pigs. Sensors (Basel), 2020, 20(8): 2381-2399.
20. Li C, Zhong Q, Xie D, et al. Skeleton-based action recognition with convolutional neural networks//2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Hong Kong: IEEE, 2017: 597-600.
21. Girdhar R, Carreira J, Doersch C, et al. Video action transformer network//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, California: IEEE, 2019: 244-253.
22. Vaswani A, Shazeer N, Parmar N. Attention is all you need//Advances in Neural Information Processing Systems, California: IEEE, 2017: 5998-6008.
23. Tao L, Wang X, Yamasaki T. Motion representation using residual frames with 3D CNN//IEEE International Conference on Image Processing, Abu Dhabi: IEEE, 2020: 1786-1790.
24. Bala P C, Eisenreich B R, Yoo S B M, et al. Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Nat Commun, 2020, 11(1): 4560.
25. Tang D H, Wang C Y, Huang X, et al. Inosine induces acute hyperuricaemia in rhesus monkey (Macaca mulatta) as a potential disease animal model. Pharmaceutical Biology, 2021, 59(1): 175-182.
26. Carreira J, Zisserman A. Quo vadis, action recognition? a new model and the kinetics dataset//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Venice: IEEE, 2017: 6299-6308.
27. Tran D, Wang H, Torresani L, et al. A closer look at spatiotemporal convolutions for action recognition//Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Salt Lake City and UT: IEEE, 2018: 6450-6459.
28. Wang L, Xiong Y, Wang Z, et al. Temporal segment networks: towards good practices for deep action recognition//European Conference on Computer Vision, Amsterdam: IEEE, 2016: 20-36.
29. Bertasius G, Wang H, Torresani L. Is space-time attention all you need for video understanding//International Conference on Machine Learning (ICMC), 2021. arXiv: 2102.05095.

Journal of Biomedical Engineering

A multi-behavior recognition method for macaques based on improved SlowFast network

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content