Artificial intelligence in breast ultrasound: application in clinical practice

Breast Lesion Detection, Characterization, and Classification

Utilizing deep learning (DL) neural network techniques and instance segmentation, the application of AI research to clinical practice can potentially improve identification, characterization, and classification of US imaged breast lesions. When a breast lesion is detected, the suspicion for malignancy is primarily based on the radiologist's qualitative visual assessment guided by a standardized lexicon, part of the Breast Imaging Reporting and Data System (BI-RADS) Atlas [5], which contains a specifically developed section for breast US. Despite this, breast US has low specificity and low positive predictive values (PPVs). Large interobserver variability for lesion management recommendations has also been reported in clinical practice [6-11].

Intensively investigated in the field of breast imaging in recent years, several computer-aided detection/diagnosis (CAD) systems have emerged to enhance detection and diagnostic accuracy while reducing interpretation variability. Using machine learning and AI, these systems generate a probability of malignancy for a finding included in a user-selected region of interest (ROI), assisting the radiologist in the decision-making process when assigning a BI-RADS category and making management recommendations. As of today, there are local AI governance committees, with country-specific approved applications for clinical use. The United States Food and Drug Administration (FDA) approved a few AI-powered decision support applications for breast US clinical use (Table 1), which are commercially available on various vendor platforms [12].

Table 1.

Overview of commercially available artificial intelligence applications in breast handheld US

BU-CAD (TaiHao Medical Inc.)

BU-CAD is an FDA-approved, commercially available AI software for computer assisted detection (CADe) and diagnosis (CADx DS) for breast US. To assist lesion detection, the CADe function identifies regions of interest (automated ROIs) of a suspicious lesion in up to two orthogonal US images. An adjunctive smart system named breast free-hand US (BR-FHUS) which is comprised of two subsystems (BR-FHUS Navigation and Viewer) leverages the BU-CAD, allowing both real-time and batched recorded CADe assistance in identifying suspicious lesions for whole breast HHUS (Fig. 1A, B) [12]. Once a breast lesion is identified by either the human operator and/or the software and a ROI identified, the (CADx DS) component generates a numerical assessment of a lesion being malignant or benign based on its’ characteristics, termed by the manufacturer as "score of lesion characteristics (SLC)." The software also provides a corresponding BI-RADS category and descriptors including shape, orientation, margin, echo pattern, and posterior features (Fig. 2). The software was evaluated by Lai et al. [13] in a multi-reader study comparing the diagnostic performance and the interpretation time of breast US examinations between reading without and with the AI system as a concurrent reading aid. Statistically significant improvement of reader’s diagnostic performance was achieved with the addition of BU-CAD CADx DS, increasing area under the receiver operating characteristic curve (AUC) from 0.758 to 0.829 (P<0.001). Additionally, the readers’ mean reading time has decreased from 30.15 seconds without the software to 18.11 seconds with the aid of the AI system, reflecting a significant decrease in interpretation time of nearly 40% (P<0.001) [13]. The potential for AI to enable faster and more accurate interpretation by the reader could have tremendous clinical implications if these results are supported in future studies in a prospective clinical environment.

Fig. 1.

BR-FHUS Navigation - an adjunct artificial intelligence tool to assist breast ulrasoundscreening.

A. The software provides a route map panel to display scanning information including probe location and scanning coverage (scanned areas of the breast indicated by gray bars). In addition, the system generates a "lesion detection module" to assist with real-time detection. Example of a suspicious lesion (red square) detected by the software with display of clock axis and distance to the nipple (purple circle) is shown. The images and cine loops are stored with their spatial position coordinates in DICOM format (figure provided by TaiHao Medical Inc. and used with permission). B. BR-FHUS Viewer assists physicians in reviewing a series of 2-D ultrasound images recorded by BR-FHUS Navigation. It supports the computer-aided detection method to detect breast lesions in the recorded images. An example of a snapshot taken by Navigation which is automatically showed in the Viewer so physicians can choose which lesion would appear in the report. By clicking a CADe button (not displayed on this image) the software automatically detects the suspicious areas which are displayed in the suspicious area list (purple rectangle), indicating breast laterality (green rectangle) and location within the breast (yellow rectangle) for each lesion. The user can capture the image which suits for reporting (in this case marked by blue background from the list area on the right side). Subsequently, the diagnostics results including DICOM images and reports are uploaded to PACS or stored in a local storage (figure provided by TaiHao Medical Inc. and used with permission). BR-FHUS, breast free-hand ultrasound; CADe, computer assisted detection; PACs, Picture Archiving and Communication System.

Fig. 2.

Example of BU-CAD showing orthogonal B-mode ultrasound images of a breast mass (indicated by the red box), where the artificial intelligence decision support output (DS) displayed a high score of lesion characteristics (SLC=61), corresponding to a Breast Imaging Reporting and Data System (BI-RADS) assessment category 4B (figure provided by TaiHao Medical Inc. and used with permission).

Koios Decision Support (DS) (Koios Medical Inc.)

The potential clinical impact of another breast US AI decision support system, Koios DS for breast, was recently evaluated by Mango et al. [14]. Koios DS is a software application designed to assist physicians in analyzing breast US images by generating a likelihood of malignancy for a user-selected ROI that contains a breast lesion (Fig. 3). In this multicenter retrospective study, 900 breast lesions seen on US images (470 benign and 430 malignant), were evaluated by 15 physicians with and without the software. The mean AUC with DS system alone (0.88; 95% confidence interval [CI], 0.86 to 0.91) and mean reader AUC with US plus DS (0.87; 95% CI, 0.84 to 0.90) were significantly higher than mean reader AUC with US only (0.83; 95% CI, 0.78 to 0.89; P<0.001), demonstrating improved accuracy of sonographic breast lesion assessment using AI-based DS. Interobserver agreement, quantified by the Kendall τ -b correlation coefficient, was higher for US plus DS (0.68; 95% CI, 0.67 to 0.69) vs. US only (0.54; 95% CI, 0.53 to 0.55). Additionally, the integration of DS reduced the intra-observer variability, as evidenced by the decrease in the rates of cases with different BI-RADS assessments between the two reading sessions (10.8% vs. 13.6%, P=0.04). There are inherent limitations of retrospective breast US AI studies, as they do not replicate a true clinical environment in which lesions are often scanned in real time by the radiologist and evaluated in the context of patient symptoms, risk factors, and correlation with mammography, prior imaging, or both. Although AI-based DS reduced inter and intra-observer variability and improved correct assessment of sonographic breast lesions by most physicians, when the impact of DS on each reader's sensitivity and specificity was analyzed, improvements appeared to depend on the reader's subspecialty. This finding was in line with results from a prior study by Chabi et al. [15], who evaluated the accuracy of CAD in breast US according to the radiologist’s experience. The authors found improved sensitivity for junior radiologists but decreased specificity for experienced radiologists. This implies that although AI can enhance the diagnosis of malignant lesions when employed by junior radiologists, it might not surpass or enhance the precision of experienced, specialized breast radiologists [15]. This trend must be explored further and carefully considered when implementing breast US AI in clinical practice as the experience of the radiologists in a given practice may affect the impact of the AI system.

Fig. 3.

Example of Koios decision support (DS) for Breast showing the B-mode ultrasound images of invasive carcinoma with micropapillary features in a 68-year-old patient.

The artificial intelligence DS output is displayed in a graphical form on the right panel, with the DS-generated output (in this case correctly classified as "suspicious") and the confidence of assessment within that category as marked by the triangular marker.

Additionally, Berg et al. [16] conducted a Koios reader study to assess the impact of original and artificially enhanced AI-based CADx on breast US interpretation. The study included 319 lesions identified on screening, with 88 (27.6%) being cancers. Nine breast imaging radiologists reviewed orthogonal-paired US images, evaluating the findings both without and with CADx assistance in its original mode (AUC=0.77) and modified to high sensitivity (AUC=0.89) or specificity (AUC=0.86).

Results showed no overall improvement in accuracy when using the original CADx. However, when the DS outputs were adjusted to provide a binary categorization (benign or malignant) in high sensitivity or high specificity mode, all readers significantly enhanced their accuracies (average AUC increase of 0.045 for high sensitivity mode and 0.071 for high specificity mode, P<0.001). The authors concluded that radiologists acted more appropriately on CADx output when it presented fewer false-positive cues, perhaps indicating readers tend to trust a more specific tool. These results highlight that issues related to user trust of AI need to be considered in AI development, implementation, and radiologist training, as ultimately the human-AI interaction will affect the impact AI has on patient care.

The Koios software impact on the PPV of US-guided breast biopsies has also been investigated. Browne et al. [17] retrospectively compared pathology results of 403 biopsied breast lesions from a single institution against the original radiologist’s BI-RADS classification based on breast US diagnostic images and the result of processing the same images through the AI algorithm. At the authors’ institution, cytology is performed at the radiologist’s discretion for some palpable breast lesions that are assessed as BI-RADS 3. While the BI-RADS Atlas indicates if biopsy is recommended a BI-RADS 4 or 5 assessment should be given, this study highlights the challenging clinical scenario of managing US findings that appear very likely to be benign but are palpable. The authors report that in their clinical practice if cytology renders indeterminate or suspicious results a core needle biopsy is recommended. According to study results, the software was more successful at determining which BI-RADS 3 lesions selected for biopsy by the radiologist actually required biopsy, potentially avoiding 44.1% (19/43) of biopsies yielding a benign result, without missing any cancers in this category. The use of AI DS may assist radiologists in this practice setting to more confidently recommend a 6-month follow-up US for a true BI-RADS 3 finding or upgrading it appropriately to a BI-RADS 4 with clear recommendation for biopsy. In this study, using Koios did not significantly increase the PPV for lesions categorized BI-RADS 4/5/6 by the reading radiologists and potentially would have missed 10 cancers, indicating that suspicious lesions categorized as BI-RADS 4b and higher by reading radiologists, still warrants a biopsy if recommended by the human reader [17]. Research studies that focus only on lesions recommended for biopsy by the radiologist fail to capture cancers missed by the radiologist that may have been accurately categorized by the AI system. Such limitations should be considered when applying research results to real-world clinical practice.

S-Detect (Samsung Medison, Co., Ltd., Seoul, South Korea)

"S-Detect," is a commercially available CADx DS tool for breast US which employs a DL algorithm to offer a valuable second opinion, aiding operators in the interpretation and diagnosis of breast lesions. Once a clinician selects a breast lesion, S-Detect promptly generates an ROI encompassing the lesion, which the operator can manually adjust as needed. The software analyzes the morphological characteristics of the lesion according to the BI-RADS lexicon, provides a detailed report of each US descriptor and combines the information with manual input from the operator regarding specific features (such as associated calcifications) and produces a dichotomized evaluation result of "possibly benign" or "possibly malignant" (Fig. 4) [18]. The impact of S-Detect on diagnostic performance has been investigated by multiple studies, reinforcing that CADx is a useful additional diagnostic tool in breast US for radiologists [19], with benefits varying depending on the radiologist’s level of experience, primarily benefiting less experienced radiologists [20-25].

Fig. 4.

Example of S-Detect showing the B-Mode ultrasound image of breast mass.

The artificial intelligence decision support output outlined the region of interest around the mass margins (yellow line); classified descriptors as irregular shape, non-parallel orientation, spiculated margin, and heterogenous echo pattern. S-Detect produced a final assessment of "possibly malignant” supporting a recommendation to biopsy this finding (figure provided by Samsung Electronics Co., Ltd.).

The S-Detect system was evaluated via a multicenter prospective study in China by Zhao et al. [26], investigating the feasibility of S-Detect in enhancing the diagnostic performance of breast US for screen-detected lesions. Analyzing 757 breast masses (460 benign, 297 malignant), S-Detect exhibited significantly higher AUC (0.83 [0.80-0.85]) and specificity (74.35% [70.10%-78.28%]) than radiologists reading conventional US (0.74 [0.70-0.77] and 54.13% [51.42%-60.29%] respectively; P<0.001), with no decrease in sensitivity [26].

Improved specificity has important implications for patient care by reducing biopsies of benign lesions, which would spare the patient the associated discomfort, cost, anxiety, and time involved with undergoing a biopsy. Evaluating the potential of AI to reduce benign biopsies by downgrading a BI-RADS 4A findings to BI-RADS 3, Wang et al. [27] described two separate downgrading stratifications based on S-Detect AI assessments, according to whether the "possibly benign" AI-output was for both orthogonal US images vs. just one of the US images. While assessing 43 BI-RADS 4A lesions using the first strategy, the biopsy rate decreased from 100% to 67.4% (P<0.001) with no missed cancers when both US images were assessed as "possibly benign"; however, when the assessment was "possibly benign" on just one of the images, the biopsy rate decrease was greater (from 100% to 37.2% [P<0.001]) but two cancers were missed [27]. Comparable to the previously mentioned Koios readers study findings, intra-reader [28] and inter-reader agreement among radiologists regarding final BI-RADS assessment [20,29], and specific BI-RADS morphological descriptors, was improved by using S-Detect, with greater improvement noted among less experienced readers [29].

The overall diagnostic accuracy of S-Detect is best summarized by a meta-analysis by Wang and Meng [30], which included 11 studies in which 951 malignant and 1,866 breast masses were assessed. The results of the meta-analysis indicate that the system has high diagnostic accuracy in differentiating benign and malignant masses, with pooled sensitivity of 0.82 (95% CI, 0.74 to 0.88); pooled specificity 0.83 (95% CI, 0.78 to 0.88) and AUC of 0.90 (standard error=0.0166) [30].

Prediction of Tumor Biology and Molecular Subtypes of Breast Cancer

Gene expression profiling has significantly influenced the comprehension of breast cancer biology and biomarkers. These encompass histologic grade, estrogen receptor, and progesterone receptor expressions, human epidermal growth factor receptor 2 (HER2) expression, and oncogene activity. These factors have been integrated into breast cancer staging by the American Joint Committee for Cancer (AJCC) [31]. Several studies have examined the prediction of breast cancer molecular subtypes based on AI systems for breast US.

The role of breast US AI in triple-negative cancer was investigated by Ma et al. [32] in a retrospective review of 600 patients with breast cancer, randomly divided into training (n=450) and testing (n=150) sets. Five AI models were trained based on clinical characteristics and imaging features of both mammography and US. One model excelled in distinguishing between triple-negative breast cancer and the other subtypes (AUC, 0.971; 95% CI, 0.947 to 0.995), significantly improving the accuracy, sensitivity, and specificity of four radiologists with the help of the model.

In a search of potential non-invasive preoperative methods to predict tumor molecular subtypes, a US-based assembled convolutional neural network multi-model was developed by Zhou et al. [33] trained on combined grayscale US images together with color Doppler and shear wave elastography features. The performance of the multi-model (macro-average AUC, 0.89-0.96) was superior to that of other models which were based on greyscale only (macro-average AUC, 0.73-0.75) or greyscale with color Doppler images (macro-average AUC, 0.81-0.84) in predicting four-classification breast cancer molecular subtypes. Surprisingly the authors reported that the model was also better than preoperative core needle biopsy (AUC, 0.89-0.99 vs. 0.67-0.82; P<0.05), possibly due to partial sampling obtained by CNB which might not represent the entire lesion owing to the heterogeneity of breast cancer. The ability of AI to evaluate the entire lesion via the US images as opposed to partial sampling by core biopsy may prove to be a benefit and is an area of future research.

Prediction of Response to Neoadjuvant Chemotherapy in Breast Cancer

Neoadjuvant chemotherapy (NAC) is used to downstage locally advanced breast cancer and additionally serves as in vivo drug sensitivity testing. Evaluating the pattern of response to NAC can be used to further tailor systemic treatment [34]. Pathologic complete response (pCR) is associated with favorable disease-free and overall survival and is the most commonly used endpoint in neoadjuvant trials [35]. While surgery is currently required to confirm a pCR post-NAC, with further study the omission of surgery could potentially be explored if pCR could be identified non-invasively. Although breast magnetic resonance imaging is currently the most accurate imaging modality for assessing pCR [36,37], the ability to predict pCR preoperatively using a less costly, less complex, and more accessible modality such as US has enormous clinical potential and economic benefits. This has encouraged the investigation of the added value of AI and deep learning radiomics (DLR) to US images in patients undergoing NAC. Jiang et al. [38] developed a novel preoperative pCR prediction model which is based on pre- and post-NAC US images, showing an AUC of 0.94 (95% CI, 0.91 to 0.77), outperforming two experienced radiologists.

In a prospective study by Gu et al. [39] two DLR models (DLR-2 and DLR-4) corresponding to two post-treatment time points (after second and forth courses of chemotherapy) were used to assess response to NAC based on US images of 168 patients. DLR-2 achieved an AUC of 0.812 (95% CI, 0.77 to 0.85) and DLR-4 achieved an AUC of 0.937 (95% CI, 0.913 to 0.955). Furthermore, the deep learning radiomics pipeline model which is a stepwise prediction model that combines DLR-2 and DLR-4, successfully identified 90% (19/21 patients) of non-responders, triaging them to a different treatment strategy from which they could potentially benefit more.

Prediction of Axillary Nodal Metastases

The axillary lymph node status plays a crucial role in staging, prognosis, and treatment decisions for breast cancer patients [40]. Over time, axillary surgical management has evolved, moving away from routine axillary lymph node dissection (ALND) for all breast cancer patients. Instead, it focuses on avoiding ALND in cases of negative sentinel lymph node biopsy (SLNB) with similarly low axillary failure rates and significantly reduced lymphedema rates [41]. More recently the use of SLNB has expanded to women with low-volume nodal disease based on results of randomized controlled trials demonstrating equivalent locoregional control and survival [42,43]. Conducted by the American College of Surgeons Oncology Group (ACOSOG), the Z0011 trial showed that in women with early-stage, clinically node-negative (cN0) patients with less than three positive sentinel lymph nodes undergoing breast conserving surgery and whole breast radiation therapy, ALND can be omitted [44,45]. Large meta-analysis reported that preoperative axillary US combined with LN biopsy in the diagnostic workup of breast cancer patients will identify 50% of cases with axillary metastatic disease whereas in 25% of the patients with negative US-guided biopsy, a positive sentinel node may still be retrieved at SLNB [46]. The current use of axillary imaging and image-guided biopsy in patients with breast cancer who meet Z0011 criteria or are undergoing NAC is institution- and surgeon-dependent, but also has the potential to overtreat the axilla and increase surgical morbidity [47]. Given the evolving surgical approach to axillary management, the role of axillary imaging must evolve in tandem, with the aim of improving diagnostic performance and supporting best patients’ outcomes. Potentially, AI can assist in predicting axillary lymph nodes status based on sonographic images, enhancing the utility and accuracy of preoperative US.

A predictive model of lymph node metastasis in patients with breast cancer by using DL neural networks was developed and investigated by Zhou et al. [48] Three representative deep convolutional neural networks (CNNs) models were evaluated to predict lymph node metastasis based on US images of primary breast cancer and of axillary lymph nodes. The performance of the models was compared with that of five radiologists while surgical pathologic results were used as the reference standard. Performance was analyzed in terms of accuracy, sensitivity, specificity, receiver operating characteristic curves, AUC, and heat maps. The best-performing CNN model was superior to a consensus of five radiologists, achieving higher sensitivity of 85% (95% CI, 70% to 94%) vs. 73% (95% CI, 57% to 85%; P=0.17) respectively and higher specificity of 73% (95% CI, 56% to 85%) compared to 63% (95% CI, 46% to 77%; P=0.34) achieved by the radiologists. Relatively high AUC of 0.89 (95% CI, 0.83 to 0.95) in the prediction of axillary lymph node metastasis was achieved by the model, demonstrating the feasibility of using CNNs to predict whether early primary breast cancer has metastasized to axillary lymph nodes.

Zheng et al. [49] investigated the incorporation of DLR to conventional US and shear wave elastography of breast cancer for predicting axillary lymph node status preoperatively in patients with early-stage breast cancer, reporting high diagnostic performance of combined clinical information and DLR output, with AUC of 0.90 (95% CI, 0.84 to 0.96) in comparison to lower AUC of 0.74 (95% CI, 0.69 to 0.78) achieved by routine axillary US done by radiologist. Nevertheless, there was no significant difference between DLR and radiologists' evaluation in terms of sensitivity, specificity, PPV or negative predictive value [49].