Multimodal Human Action Recognition Base on Sparse Joints Graph Convolutional Network and Dynamic Image ROI Sampling
Human action recognition is an important field that aims to train computers to understand human actions in videos. Cameras are increasingly used to monitor human activities across various settings, including hospitals, elderly care facilities, smart homes, smart environments, factories, surveillance systems, robotics, and human-computer interaction. Accurate action recognition enables intelligent systems to respond appropriately, detect emergencies, and support more natural interactions between humans and machines. However, many everyday activities, such as drinking water, making a phone call, and brushing teeth, often involve very similar body movements. While humans can distinguish these actions effortlessly, computer vision systems often struggle with subtle visual differences.
Skeleton-based action recognition is a common method that analyzes only the positions and movements of human body joints. Traditional approaches typically focus on neighboring joints. This approach emphasizes body motion rather than appearance, making it less susceptible to changes in lighting or background. However, it cannot identify the objects a person is interacting with. As a result, actions such as reading, writing, and using a tablet often appear almost identical as they involve similar body postures.
On the other hand, another common approach, image-based action recognition, identifies the objects in video images. While these images contain the necessary visual information, they often also include substantial amounts of irrelevant content, such as walls, floors, furniture, and other objects. This approach identifies both the person and the surrounding objects using color images. Although this method provides richer visual information, its performance can be negatively impacted by cluttered backgrounds and unrelated objects in the scene.
To address the limitations of existing methods, the researchers developed a multimodal action recognition framework that combines the strengths of skeleton-based and image-based approaches. The limitations of skeleton-based action recognition were addressed using a Sparse Joints Graph Convolutional Network, which enables the recognition system to capture relationships between distant body parts, providing a more comprehensive understanding of full-body movements. It enables the recognition system to effectively capture relationships between various and non-adjacent body joints, facilitating the flow of information more directly across the body. Additionally, the system automatically focuses on the body regions that exhibit the most movement, enabling it to identify the objects most relevant to the action being performed.
The challenges of image-based action recognition were addressed by introducing Dynamic Image ROI Sampling, where ROI stands for “region of interest.” Instead of scanning and analyzing the entire image, this system dynamically selects key areas around body joints that exhibit the most movement, thereby focusing attention on the parts of the image crucial for identification.
By combining global body movements with key visual information, this method achieves a more comprehensive understanding of human activities. This approach consistently outperformed previous methods that are skeleton-based, image-based, or multimodal. It is particularly effective at distinguishing between actions that could be easily confused, such as drinking water versus making a phone call, or writing versus typing. These advancements could benefit various applications, including smart healthcare, elderly monitoring, smart manufacturing, and human-robot collaboration.