Research Article| Volume 48, ISSUE 1, P55-60, March 2023

A clinical and time savings evaluation of a deep learning automatic contouring algorithm

Published:December 20, 2022DOI:


      Automatic contouring algorithms may streamline clinical workflows by reducing normal organ-at-risk (OAR) contouring time. Here we report the first comprehensive quantitative and qualitative evaluation, along with time savings assessment for a prototype deep learning segmentation algorithm from Siemens Healthineers. The accuracy of contours generated by the prototype were evaluated quantitatively using the Sorensen-Dice coefficient (Dice), Jaccard index (JC), and Hausdorff distance (Haus). Normal pelvic and head and neck OAR contours were evaluated retrospectively comparing the automatic and manual clinical contours in 100 patient cases. Contouring performance outliers were investigated. To quantify the time savings, a certified medical dosimetrist manually contoured de novo and, separately, edited the generated OARs for 10 head and neck and 10 pelvic patients. The automatic, edited, and manually generated contours were visually evaluated and scored by a practicing radiation oncologist on a scale of 1-4, where a higher score indicated better performance. The quantitative comparison revealed high (> 0.8) Dice and JC performance for relatively large organs such as the lungs, brain, femurs, and kidneys. Smaller elongated structures that had relatively low Dice and JC values tended to have low Hausdorff distances. Poor performing outlier cases revealed common anatomical inconsistencies including overestimation of the bladder and incorrect superior-inferior truncation of the spinal cord and femur contours. In all cases, editing contours was faster than manual contouring with an average time saving of 43.4% or 11.8 minutes per patient. The physician scored 240 structures with > 95% of structures receiving a score of 3 or 4. Of the structures reviewed, only 11 structures needed major revision or to be redone entirely. Our results indicate the evaluated auto-contouring solution has the potential to reduce clinical contouring time. The algorithm's performance is promising, but human review and some editing is required prior to clinical use.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Medical Dosimetry
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Deasy JO
        • Moiseenko V
        • Marks L
        • Chao KC
        • Nam J
        • Eisbruch A.
        Radiotherapy dose–volume effects on salivary gland function.
        International Journal of Radiation Oncology* Biology* Physics. 2010.; 76: S58-S63
        • Zabel WJ
        • Conway JL
        • Gladwish A
        • et al.
        Clinical evaluation of deep learning and atlas-based auto-contouring of bladder and rectum for prostate radiation therapy.
        Practical Radiation Oncology. 2021; 11: e80-e89
        • Lustberg T
        • van Soest J
        • Gooding M
        • et al.
        Clinical evaluation of atlas and deep learning based automatic contouring for lung cancer.
        Radiotherapy and Oncology. 2018; 126: 312-317
        • Vaassen F
        • Hazelaar C
        • Vaniqui A
        • et al.
        Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy.
        Physics and Imaging in Radiation Oncology. 2020; 13: 1-6
        • Cha E
        • Elguindi S
        • Onochie I
        • et al.
        Clinical implementation of deep learning contour autosegmentation for prostate radiotherapy.
        Radiotherapy and Oncology. 2021; 159: 1-7
        • Van der Veen J
        • Willems S
        • Deschuymer S
        • et al.
        Benefits of deep learning for delineation of organs at risk in head and neck cancer.
        Radiotherapy and Oncology. 2019; 138: 68-74
        • Young AV
        • Wortham A
        • Wernick I
        • Evans A
        • Ennis RD.
        Atlas-based segmentation improves consistency and decreases time required for contouring postoperative endometrial cancer nodal volumes.
        International Journal of Radiation Oncology* Biology* Physics. 2011; 79: 943-947
        • Ghesu F-C
        • Georgescu B
        • Zheng Y
        • et al.
        Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans.
        IEEE transactions on pattern analysis and machine intelligence. 2017; 41: 176-189
        • Goodfellow I
        • Pouget-Abadie J
        • Mirza M
        • et al.
        Generative adversarial nets.
        Advances in neural information processing systems. 2014: 27
        • Yi X
        • Walia E
        • Babyn P.
        Generative adversarial network in medical imaging: A review.
        Medical image analysis. 2019; 58101552
        • Sherer MV
        • Lin D
        • Elguindi S
        • et al.
        Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: A critical review.
        Radiotherapy and Oncology. 2021; 160
        • Meyer P
        • Noblet V
        • Mazzara C
        • Lallement A.
        Survey on deep learning for radiotherapy.
        Computers in biology and medicine. 2018; 98: 126-146
        • Kiljunen T
        • Akram S
        • Niemelä J
        • et al.
        A deep learning-based automated CT segmentation of prostate cancer anatomy for radiation therapy planning-A retrospective multicenter study.
        Diagnostics. 2020; 10: 959
        • Deeley M
        • Chen A
        • Datteri R
        • et al.
        Comparison of manual and automatic segmentation methods for brain structures in the presence of space-occupying lesions: a multi-expert study.
        Physics in Medicine & Biology. 2011; 56: 4557
        • Qazi AA
        • Pekar V
        • Kim J
        • Xie J
        • Breen SL
        • Jaffray DA.
        Auto-segmentation of normal and target structures in head and neck CT images: A feature-driven model-based approach.
        Medical physics. 2011; 38: 6160-6170
        • Savenije MH
        • Maspero M
        • Sikkes GG
        • et al.
        Clinical implementation of MRI-based organs-at-risk auto-segmentation with convolutional networks for prostate radiotherapy.
        Radiation Oncology. 2020; 15: 1-12
        • Zhang T
        • Chi Y
        • Meldolesi E
        • Yan D.
        Automatic delineation of on-line head-and-neck computed tomography images: toward on-line adaptive radiotherapy.
        International Journal of Radiation Oncology* Biology* Physics. 2007; 68: 522-530