Researchers at the University of Virginia’s School of Engineering and Applied Science have engineered an AI-driven intelligent video analyser capable of performing action detection technology on humans.
The action detection system, called the Semantic and Motion-Aware Spatiotemporal Transformer Network (SMAST), promises a wide range of societal benefits.
These include enhancing surveillance systems and improving public safety, advanced motion tracking in healthcare, and refining how autonomous vehicles navigate through complex environments.
“This AI technology opens doors for real-time action detection in some of the most demanding environments,” said professor and chair of the Department of Electrical and Computer Engineering, Scott T Acton, who led the project.
AI technology for complex video analysis
The system relies on two key components to detect and understand complex human behaviours.
The first is a multi-feature action detection model, which helps the AI focus on the most important parts of a scene — like a person or object — while ignoring unnecessary details. This makes the system more accurate at identifying what’s happening, such as recognising someone throwing a ball instead of just moving their arm.
The second key feature is a motion-aware 2D positional encoding algorithm, which helps the AI track how things move over time.
By integrating these features, SMAST can accurately recognise complex actions in real-time, making it more effective in high-stakes scenarios like surveillance, healthcare diagnostics, or autonomous driving.
SMAST redefines how machines detect and interpret human actions. Current systems struggle with chaotic, unedited, contiguous video footage, often missing the context of events.
SMAST’s innovative design allows it to capture the dynamic relationships between people and objects with remarkable accuracy, powered by AI components that allow it to learn and adapt from data.
New standards in action detection technology
This huge leap in action detection technology means the AI system can identify actions like a runner crossing a street, a doctor performing a precise procedure or even a security threat in a crowded space.
SMAST has already outperformed top-tier solutions across key academic benchmarks, including AVA, UCF101-24 and EPIC-Kitchens, setting new standards for accuracy and efficiency.
“The societal impact could be huge,” said Matthew Korban, a postdoctoral research associate in Acton’s lab working on the project.
“We’re excited to see how this AI technology might transform industries, making video-based systems more intelligent and capable of real-time understanding.”
