Michael Black3 ,
1 University of Colorado Boulder, USA,
2 University of Grenoble, France,
3 Max Planck Institute for Intelligent Systems, Germany,
4 University of Colorado School of Medicine, USA. .
To enhance the accuracy of surface-guided RT (SGRT) for abdominal SBRT by designing an artificial intelligence (AI) enhanced computer-vision (CV) patient setup technique that predicts skeletal anatomy from surface imaging.
We have designed a modified SGRT technique, 'avatar guided-RT' (AgRT), that employs patient-specific "avatars” based on the recently published Sparse Trained Articulated Human Body Regressor (STAR) model. STAR is a realistic 3D model of human surface anatomy learned from >10,000 3D body scans that considers gender and BMI for pose-dependent surface variation and can be fitted to CT-based surface contours or surface meshes acquired from 2D video/depth images. We utilize a pre-existing neural network trained on 2,400 soft-tissue/skeleton training pairs obtained from dual-energy X-ray absorptiometry (DXA) scans to predict the skeletal anatomy from the body surface of a patient in treatment position.
AgRT was tested using a calibrated multiple camera system. Real-time 3D pose extraction from multiple 2D images was tested in a virtual treatment room to optimize camera numbers and positions. Testing was then conducted on a healthy volunteer to track various treatment poses. The patient's 3D pose was mapped to an avatar with matching gender and BMI. The skeletal alignment technique was assessed on XCAT phantom data and retrospective patient CTs. Skeletal anatomy was predicted from surface imaging with sub-cm accuracy.
Figure 1 shows an example of skeletal misalignment between days of treatment for a pelvic patient based on a typical SGRT matching algorithm, due to the limited field of view available to the SGRT algorithm. Realistic full- body modelling has the potential to overcome current issues due to lack of surface anatomic variation which can lead to poor correlation and large random errors.
This project consists of two major innovations. Our first innovation is the use of a novel patient-specific skin mesh model to enable marker-less 3D human anatomy tracking. Our patient-specific skin mesh model is based on a recently published Sparse Trained Articulated Human Body Regressor (STAR), which is a realistic 3D model of the human body learned from >10,000 3D body scans. It allows estimation, synthesis, and analysis of 3D human pose and shape with sub-centimeter accuracy (Fig. 2). Our second innovation is an improved internal to external anatomical model that fuses patient specific body models to their internal skeletal anatomy, improving the correspondence between surface imaging and internal anatomy. We have adapted the technique outlined in Keller et al, 2022, using a neural network trained on 1000s of dual-energy X-ray absorptiometry (DXA) scans to predict the skeleton from the body surface when patients are in the treatment position (Fig. 3b-c). Accounting for patient pose and body type will be particularly valuable for overcoming two health-equity issues of current SGRT techniques,