Ultrasonography > Volume 42(1); 2023 > Article
Xholli, Londero, Jakimovska, Zermano, Puma, Pasqual, Kobal, and Cagnacci: An implementation of the learning curve cumulative summation test to evaluate a practicum for endometrial cancer ultrasound staging



The learning curve cumulative summation (LC-CUSUM) test is commonly used as a quantitative assessment of the individual learning process. This study aimed to evaluate the skill acquisition process for performing ultrasound staging of endometrial cancer using the LC-CUSUM test.


Sixty-seven ultrasound examinations performed by two operators were evaluated using the LC-CUSUM test according to their rate of success or failure to correctly stage myometrial invasion, serosa involvement, and pelvic and para-aortic lymph node involvement. The reference standard was the pathologic result. After the learning phase, the cumulative summation (CUSUM) test was applied to assess performance maintenance.


The processes achieved satisfactory performance in the majority of the cases according to the established definitions. Operator 1 reached adequate performance within the 30th procedure for all the locations considered, while operator 2 performed fewer than 30 ultrasound examinations by the audit time. The CUSUM test confirmed that the target quality was preserved after the learning phase. Moreover, the ultrasound staging for endometrial cancer was highly accurate.


Using the LC-CUSUM test made it possible to monitor the achievement and maintenance of a satisfactory performance quantitatively. The LC-CUSUM test could be a valuable instrument to survey diagnostic pathways in gynecological ultrasonography quantitatively.


In high-income countries, endometrial cancer is the most common gynecological malignancy, including 4% of all female cancers worldwide [1,2]. Moreover, the prevalence of endometrial cancer is expected to increase in the near future [3]. Women from low-income and middle-income countries are significantly more likely to die from endometrial cancer than those from high-income countries, reflecting poor access to adequate resources [1,2]. Surgery is the primary staging and treatment approach for early endometrial cancer [3]. Lymphadenectomy is required for surgical staging in endometrial cancer; however, it does not offer a survival advantage and is associated with an increased surgical burden [4,5]. An accurate and cost-effective presurgical workup can improve the treatment quality, reducing the frequency of overtreatment or undertreatment and enabling broader access to appropriate staging in resource-limited settings.
For several years, staging ultrasonography in endometrial cancer has proven to be a valid and reproducible method [5–7]. Expert ultrasonography and magnetic resonance imaging (MRI) are both critical in the diagnostic workup of endometrial cancer for local tumor extension [5,7]. Models based on ultrasound staging have also been proposed to predict lymph node involvement [4]. Local tumor extension and the prediction of lymph node involvement influence the patient’s management [4,5]. These results direct the surgical management in terms of whether to perform systematic lymph node dissection or lymph node biopsy assessment [4,5]. Moreover, imaging can serve as a decision instrument for further management in women with failed sentinel lymph node biopsy [4].
Evaluating the learning curve (LC) in a structured program is useful for standardizing medical procedures [8–10]. Similar studies have been proposed in gynecological ultrasound examinations for other conditions, such as ovarian cancer, endometriosis, or congenital uterine anomalies [8–10].
Completing an advised number of ultrasound examinations is the typical approach for professional training in ultrasound imaging. However, this method lacks objective evidence regarding individual competency. Statistical methods have been applied to analyze interventional and diagnostic LCs to overcome this issue. The cumulative summation (CUSUM) test and the LC-CUSUM test have been used multiple times for this purpose [9,11–13]. However, to the best of the authors’ knowledge, the LC-CUSUM test has not yet been well characterized in the ultrasound staging of endometrial cancer.
Therefore, this study aimed to use the LC-CUSUM test to assess the skill acquisition process for performing ultrasound staging of endometrial cancer.

Materials and Methods

Compliance with Ethical Standards

In this retrospectively designed study, data were collected after reviewing medical records during the 2017-2019 period at Santa Maria della Misericordia University Hospital (a tertiary referral center). This investigation was carried out with internal review board approval, following the Helsinki Declaration, and according to local data protection authority dictates (number 049/2021). The need for informed consent, according to national legislation, was waived by the IRB listed above because this was a retrospective study.

Setting and Sample

The study was conducted at a tertiary academic hospital with a reference population of more than 500,000 inhabitants and an annual average of 35,000 hospitalizations. Endometrial cancer patients were routinely managed during the study period according to the local guidelines [14]. During the same period, beyond MRI and the computed tomography imaging, an advanced ultrasonography examination was also introduced to map the disease extension preoperatively. This ultrasound examination was recorded in the hospitalization files and used by the surgeons as additional information during the surgical procedure. From the inception of the preoperative ultrasound examinations to April 2019, data on the ultrasound staging, intraoperative assessment, and final pathology report were collected. All consecutive eligible patients were included. The exclusion criteria included the absence of ultrasound data and patients not undergoing surgery after the preoperative workup.

Data Collection and Measurement

Clinical data were recorded on a standard spreadsheet. The data contained patient demographics, ultrasound examinations, surgical reports, and definitive pathology data. The following outcomes were considered: myometrial invasion, uterus serosa invasion, pelvic lymph nodes positivity, and para-aortic lymph nodes positivity. The success or failure of the ultrasound examination assessment was established according to the intraoperative exploration and the definitive pathology report.
The extent of myometrial tumor invasion was evaluated using the Karlsson ratio [15]. The degree of infiltration into the myometrium was determined by measuring the anteroposterior diameter of the tumor (d1) and relating it to the anteroposterior diameter of the uterus (d2) in the sagittal plane [15]. In detail, if the ratio of d1 to d2 is <50%, myometrial invasion was estimated as less than half of the myometrium (stage 1a); in contrast, if the ratio is >50%, it was classified as more than half of the myometrium (stage 1b) [15]. Interruption of the hyperechoic line surrounding the uterus serosa by the tumor infiltrating the myometrium wall was considered as serosal involvement. Lymph nodes were examined principally by transabdominal sonography using a convex-array probe [16,17]. The standard approach started from the inguinal to iliac and para-aortic nodes. The inguinal lymph nodes are situated caudally to the level of the inguinal ligament. Once the inguinal lymph nodes were assessed, the probe was placed in an oblique section to evaluate the external vessels located on the psoas major muscle. The obturator fossa is under the external iliac vessels, and at the pelvic margin it follows the bifurcation of the internal and external iliac vessels. The examination continued with an evaluation of the common iliac vessels and periaortic and peri-caval vessels. The sagittal plane was obtained to check these nodes and move from the aortic/caval bifurcation to the diaphragm. The probe was placed from left to right and back by small movements during the entire course of the aorta and cava [16,17].
In all cases, the examiners used both approaches, with transvaginal and transabdominal probes, as part of the standard methodology for staging endometrial cancer. Transvaginal ultrasonography was used to assess the pelvic organs and myometrial invasion, while transabdominal ultrasonography was used to evaluate the inguinal, iliac, and para-aortic lymph nodes [16,17].
Both operators were experienced ultrasonographers (more than 5 years of experience in ultrasonography) blinded to the other imaging results, and the pathologist was blinded to the ultrasound examination results. All the ultrasound examinations were performed with a Voluson E10 General Electric (General Electric Medical Systems, Zipf, Austria) or a Voluson E6 (General Electric Medical Systems) instrument. All investigations were performed with a high-resolution ultrasound transabdominal probe (2-8 MHz convex transducer) or a high-resolution ultrasound transvaginal probe (5–9 MHz transducer) (General Electric Medical Systems). All the ultrasound probes had three-dimensional capabilities.

Data Analysis

The LC-CUSUM test was designed to inform when the operator performance is sufficiently far away from an unsatisfactory performance level to evaluate the performance as acceptable [11,13]. It supposes that a newcomer operator is not performing adequately at the start of monitoring. Afterward, it reports when the operator achieves a satisfactory level of performance [13,18]. Unlike LC-CUSUM, the CUSUM test is designed for monitoring the transition from an adequate to an inadequate performance level, and it can be used after LC-CUSUM to monitor whether the performance is maintained within an acceptable range [13]. Acceptable and unacceptable rates were designated according to the previously published literature [9]. For the LC-CUSUM test, the null hypothesis was set at a failure rate of 25% (suboptimal implementation), while the alternative hypothesis was set at a failure rate of 10% (optimal implementation) [9]. For the CUSUM test, the null hypothesis was set at a failure rate of 10% (optimal implementation), while the alternative hypothesis was set at a failure rate of 25% (suboptimal implementation) [9]. The accepted type I and II errors of 0.1 and 0.1, respectively, entailed a limit of h=2.0 for both curves [9].
All statistical analyses were executed with R software (version 3.6.3, R Core Team 2020, R: A language and environment for statistical computing; R Foundation for Statistical Computing, Vienna, Austria; https://www.R-project.org/). Continuous variables are shown as the median and interquartile range (IQR) or mean and standard deviation. Dichotomous variables are shown as frequencies (%) and absolute values, and missing values were excluded from the analysis. Furthermore, sensitivity, specificity, and accuracy were analyzed and presented with their 95% confidence intervals.


The two operators performed 67 procedures during the study period. Table 1 shows the detailed numbers of ultrasound examinations eligible for inclusion per site. The number of procedures required to achieve competence is reported in Table 1. Operator 1 performed 42 procedures for the myometrial invasion diagnosis, and the number required to achieve competence (NRTAC) according to LC-CUSUM was 29 (Table 1, Fig. 1A). Operator 2 had only a limited number of procedures, and it was not possible to establish the NRTAC (Fig. 1B). For serosa invasion, operator 1 had an NRTAC of 39 and operator 2 of 13, with a mean value of 26 procedures (Table 1, Fig. 1C). For pelvic lymph nodes assessment, the NRTAC was only calculated for operator 1, resulting in 13 procedures (Table 1, Fig. 2A). Operator 2 did not reach the limit to calculate the NRTAC (Fig. 2B). The NRTAC was 13 procedures in both operators to diagnose positive paraaortic lymph nodes (Table 1, Fig. 2C, D).
Figs. 1 and 2 show the LC-CUSUM. In all the curves where the learning target was achieved, the performance was maintained within the established definitions of adequacy.
Table 2 shows the diagnostic performance of ultrasound staging in patients affected by endometrial cancer. The accuracy was above 50% in all the assessed locations, with higher specificity than sensitivity (Table 2).


The present study tested the implementation of a new diagnostic pathway in the presurgical workup of endometrial cancer using LC-CUSUM, which represents a tailored approach to individual learning. The processes reached satisfactory performance in the majority of the cases according to the established definitions. Moreover, the ultrasound staging for endometrial cancer was highly accurate.
This study demonstrated the feasibility of using the LC-CUSUM test to assess the performance for auditing the introduction of a presurgical staging pathway to the traditional management. Satisfactory performance was achieved by operator 1 within the first 67 subjects examined, and moreover, the performance was maintained according to the established definitions. The second operator performed fewer procedures and did not achieve adequate performance in all the predefined outcomes at the audit time. Previous studies on gynecological ultrasonography adopted the LC-CUSUM method to assess the efficacy of a predetermined training program [8–10]. Those previous studies all highlighted the satisfactory use of the LC-CUSUM test to assess progress and monitor the skills achieved by the trainees [8–10].
The ultrasound staging of endometrial cancer is a hot topic. Ultrasound imaging is a simple and affordable methodology that has several advantages [17]. This technique is increasingly supported for the staging of endometrial cancer [5,19]. In the literature, the sensitivity of ultrasound imaging for assessing myometrial invasion ranges between 70% and 100% and the specificity between 70% and 90% [19–21]. In addition, the sensitivity and specificity of the subjective assessment of cervical stromal invasion range from 25% to 93% and from 85% to 99%, respectively [22,23]. Different studies and a recent meta-analysis comparing MRI and ultrasonography did not highlight any significant differences between the accuracy of the two methods [21,24–26]. The only thing that stands out is the more substantial variability in the accuracy of the ultrasound scans, probably because ultrasonography is a more subjective technique or simply because of improvements in the equipment available in recent years [21]. Moreover, recent studies have proposed less subjective and more standardized methods to assess myometrial invasion in endometrial cancer, showing an improvement in diagnostic accuracy [5,19]. The data of the present study showed an overall accuracy overlapping with previously published literature [5,20].
Some possible limitations of this study should be noted. First, the LC-CUSUM test was employed by a small number of operators for a relatively limited number of women affected by endometrial cancer. Thus datasets with other numbers and different working conditions might show dissimilar results. However, even with this limitation, the LC-CUSUM test showed its potential value for monitoring the introduction and the continuation of a new diagnostic pathway, ensuring adequate quality achievement and quality maintenance. Second, this is a retrospective study with all the limitations of a retrospective design. Future studies should consider a prospective design with a predefined protocol to avoid any possible bias innate in the retrospective design.
This study investigated the usefulness of applying the LC-CUSUM test to endometrial cancer ultrasound staging. The data showed the utility of the LC-CUSUM method for monitoring the LC and preserving adequate standards.


Author Contributions

Conceptualization: Xholli A, Londero AP, Jakimovska M, Puma L, Pasqual EM, Cagnacci A. Data acquisition: Xholli A, Jakimovska M, Zermano S. Data analysis or interpretation: Xholli A, Londero AP, Kobal B, Cagnacci A. Drafting of the manuscript: Xholli A, Londero AP, Jakimovska M, Zermano S, Puma L, Pasqual EM, Kobal B, Cagnacci A. Critical revision of the manuscript: Xholli A, Londero AP, Jakimovska M, Zermano S, Puma L, Pasqual EM, Kobal B, Cagnacci A. Approval of the final version of the manuscript: all authors.

Conflict of Interest

No potential conflict of interest relevant to this article was reported.


The authors would like to thank the entire staff for their collaboration in article collection, selection, and reading, and in paper writing and reviewing.


1. Crosbie EJ, Kitson SJ, McAlpine JN, Mukhopadhyay A, Powell ME, Singh N. Endometrial cancer. Lancet 2022;399:1412–1428.
crossref pmid
2. Viola L, Londero AP, Bertozzi S, Orsaria M, Marzinotto S, Antoniazzi F, et al. Prognostic role of Kruppel-like factors 5, 9, and 11 in endometrial endometrioid cancer. Pathol Oncol Res 2020;26:2265–2272.
crossref pmid pdf
3. Londero AP, Parisi N, Tassi A, Bertozzi S, Cagnacci A. Hormone replacement therapy in endometrial cancer survivors: a metaanalysis. J Clin Med 2021;10:3165.
crossref pmid pmc
4. Eriksson LSE, Epstein E, Testa AC, Fischerova D, Valentin L, Sladkevicius P, et al. Ultrasound-based risk model for preoperative prediction of lymph-node metastases in women with endometrial cancer: model-development study. Ultrasound Obstet Gynecol 2020;56:443–452.
crossref pmid pdf
5. Green RW, Epstein E. Dynamic contrast-enhanced ultrasound improves diagnostic performance in endometrial cancer staging. Ultrasound Obstet Gynecol 2020;56:96–105.
crossref pmid pdf
6. Fischerova D, Cibula D. Ultrasound in gynecological cancer: is it time for re-evaluation of its uses? Curr Oncol Rep 2015;17:28.
crossref pmid pdf
7. Colombo N, Creutzberg C, Amant F, Bosse T, Gonzalez-Martin A, Ledermann J, et al. ESMO-ESGO-ESTRO Consensus Conference on Endometrial Cancer: diagnosis, treatment and follow-up. Ann Oncol 2016;27:16–41.
crossref pmid
8. Alcazar JL, Diaz L, Florez P, Guerriero S, Jurado M. Intensive training program for ultrasound diagnosis of adnexal masses: protocol and preliminary results. Ultrasound Obstet Gynecol 2013;42:218–223.
crossref pmid
9. Guerriero S, Pascual MA, Ajossa S, Rodriguez I, Zajicek M, Rolla M, et al. Learning curve for ultrasonographic diagnosis of deep infiltrating endometriosis using structured offline training program. Ultrasound Obstet Gynecol 2019;54:262–269.
crossref pmid pdf
10. Pascual MA, Alcazar JL, Graupera B, Rodriguez I, Guerriero S, Mallafre A, et al. Training performance in diagnosis of congenital uterine anomalies with 3-dimensional sonography: a learning curve cumulative summation analysis. J Ultrasound Med 2016;35:2589–2594.
11. Biau DJ, Williams SM, Schlup MM, Nizard RS, Porcher R. Quantitative and individualized assessment of the learning curve using LC-CUSUM. Br J Surg 2008;95:925–929.
crossref pmid pdf
12. Bolsin S, Colson M. The use of the Cusum technique in the assessment of trainee competence in new procedures. Int J Qual Health Care 2000;12:433–438.
crossref pmid
13. Je S, Cho Y, Choi HJ, Kang B, Lim T, Kang H. An application of the learning curve-cumulative summation test to evaluate training for endotracheal intubation in emergency medicine. Emerg Med J 2015;32:291–294.
crossref pmid
14. Associazione Italiana di Oncologia Medica. Neoplasie dell’utero: endometrio e cervice 2017. Milan: Associazione Italiana di Oncologia Medica, 2017.

15. Karlsson B, Norstrom A, Granberg S, Wikland M. The use of endovaginal ultrasound to diagnose invasion of endometrial carcinoma. Ultrasound Obstet Gynecol 1992;2:35–39.
crossref pmid pdf
16. Fischerova D. Ultrasound scanning of the pelvis and abdomen for staging of gynecological tumors: a review. Ultrasound Obstet Gynecol 2011;38:246–266.
crossref pmid
17. Xholli A, Cecchini F, Londero A, Cagnacci A. Ultrasound nodal staging in cervical and endometrial cancer. Londero AP, Bertozzi S, Cedolini C. The sentinel lymph node: detection, procedures and clinical implications. New York: Nova Medicine & Health, 2019:201–217.

18. Biau DJ, Porcher R. A method for monitoring a process from an out of control to an in control state: application to the learning curve. Stat Med 2010;29:1900–1909.
crossref pmid pdf
19. Verbakel JY, Mascilini F, Wynants L, Fischerova D, Testa AC, Franchi D, et al. Validation of ultrasound strategies to assess tumor extension and to predict high-risk endometrial cancer in women from the prospective IETA (International Endometrial Tumor Analysis)-4 cohort. Ultrasound Obstet Gynecol 2020;55:115–124.
crossref pmid pdf
20. Epstein E, Blomqvist L. Imaging in endometrial cancer. Best Pract Res Clin Obstet Gynaecol 2014;28:721–739.
crossref pmid
21. Alcazar JL, Gaston B, Navarro B, Salas R, Aranda J, Guerriero S. Transvaginal ultrasound versus magnetic resonance imaging for preoperative assessment of myometrial infiltration in patients with endometrial cancer: a systematic review and meta-analysis. J Gynecol Oncol 2017;28:e86.
pmid pmc
22. Eriksson LS, Lindqvist PG, Floter Radestad A, Dueholm M, Fischerova D, Franchi D, et al. Transvaginal ultrasound assessment of myometrial and cervical stromal invasion in women with endometrial cancer: interobserver reproducibility among ultrasound experts and gynecologists. Ultrasound Obstet Gynecol 2015;45:476–482.
crossref pmid
23. Mascilini F, Testa AC, Van Holsbeke C, Ameye L, Timmerman D, Epstein E. Evaluating myometrial and cervical invasion in women with endometrial cancer: comparing subjective assessment with objective measurement techniques. Ultrasound Obstet Gynecol 2013;42:353–358.
crossref pmid
24. Antonsen SL, Jensen LN, Loft A, Berthelsen AK, Costa J, Tabor A, et al. MRI, PET/CT and ultrasound in the preoperative staging of endometrial cancer: a multicenter prospective comparative study. Gynecol Oncol 2013;128:300–308.
crossref pmid
25. Celik C, Ozdemir S, Kiresi D, Emlik D, Tazegul A, Esen H. Evaluation of cervical involvement in endometrial cancer by transvaginal sonography, magnetic resonance imaging and frozen section. J Obstet Gynaecol 2010;30:302–307.
crossref pmid
26. Savelli L, Ceccarini M, Ludovisi M, Fruscella E, De Iaco PA, Salizzoni E, et al. Preoperative local staging of endometrial cancer: transvaginal sonography vs. magnetic resonance imaging magnetic resonance imaging;magnetic resonance imaging:magnetic resonance imaging–magnetic resonance imaging.

Learning curve cumulative summation (LC-CUSUM) and cumulative summation (CUSUM) curves with the upper decision limit (UDL) and the lower decision limit (LDL) (limits set h=2) for the diagnosis of myometrial invasion >50% and serosa invasion.

Diagnosis of myometrial invasion >50% was performed by operator 1 (A) and operator 2 (B). Diagnosis of serosa invasion was performed by operator 1 (C) and operator 2 (D).
Fig. 1.

Learning curve cumulative summation (LC-CUSUM) and cumulative summation (CUSUM) curves with the upper decision limit (UDL) and the lower decision limit (LDL) (limits set h=2) for the diagnosis of positive pelvic lymph nodes and positive para-aortic lymph nodes.

Diagnosis of positive pelvic lymph nodes was performed by operator 1 (A) and operator 2 (B). Diagnosis of positive para-aortic lymph nodes was performed by operator 1 (C) and operator 2 (D).
Fig. 2.
Table 1.
Number of procedures per operator and number of procedures necessary to achieve competence
Myometrial invasion >50% Serosa invasion Positive pelvic lymph nodes Positive para-aortic lymph nodes
Operator 1
Procedures performed 42 41 27 26
NRTAC 29 39 13 13
Operator 2
Procedures performed 25 25 23 22
NRTAC - 13 - 13
Procedures performed 67 66 50 48
Mean NRTAC 29 26 13 13

NRTAC, number required to achieve competence.

Table 2.
Diagnostic performance stratified by operator and disease location
Sensitivity (95% CI) Specificity (95% CI) Accuracy (95% CI)
Myometrial invasion >50%
 Operator 1 82.61 (62.86–93.02) 78.95 (56.67–91.49) 80.95 (66.70–90.02)
 Operator 2 92.31 (66.69–98.63) 50.00 (25.38–74.62) 72.00 (52.42–85.72)
 All cases 86.11 (71.34–93.92) 67.74 (50.14–81.43) 77.61 (66.29–85.94)
Serosa invasion
 Operator 1 42.86 (15.82–74.95) 94.12 (80.91–98.37) 85.37 (71.56–93.12)
 Operator 2 100.00 (43.85–100.00) 100.00 (85.13–100.00) 100.00 (86.68–100.00)
 All cases 60.00 (31.27–83.18) 96.43 (87.88–99.02) 90.91 (81.55–95.77)
Positive pelvic lymph nodes
 Operator 1 50.00 (9.45–90.55) 96.00 (80.46–99.29) 92.59 (76.63–97.94)
 Operator 2 50.00 (15.00–85.00) 84.21 (62.43–94.48) 78.26 (58.10–90.34)
 All cases 50.00 (18.76–81.24) 90.91 (78.84–96.41) 86.00 (73.81–93.05)
Positive para-aortic lymph nodes
 Operator 1 100.00 (86.68–100.00) 96.15 (81.11–99.32)
 Operator 2 50.00 (9.45–90.55) 100.00 (83.89–100.00) 95.45 (78.20–99.19)
 All cases 33.33 (6.15–79.23) 100.00 (92.13–100.00) 95.83 (86.02–98.85)

CI, confidence interval.

Web of Science
We recommend
Editorial Office
A-304 Mapo Trapalace, 53 Mapo-daero, Mapo-gu, Seoul 04158, Korea
TEL : +82-2-763-5627   FAX : +82-2-763-6909   E-mail : office@ultrasound.or.kr
About |  Browse Articles |  Current Issue |  For Authors and Reviewers
Copyright © Korean Society of Ultrasound in Medicine.                 Developed in M2PI
Zoom in Close layer