The clinical value of artificial intelligence in assisting junior radiologists

Patients

This prospective study was approved by the local ethical committee of each medical center. Written informed consent was obtained from each patient prior to undergoing US examinations. Patients were continuously admitted to three medical centers, i.e., The Cancer Hospital of the University of Chinese Academy of Sciences (Medical Center 1), The Second Affiliated Hospital of Shantou University (Medical Center 2), and The Second Affiliated Hospital of Nanchang University (Medical Center 3) for thyroid nodule examinations. The US examinations were carried out following a previous guideline [11] using central frequency in the range of 5–10 MHz with Colour Doppler US machines. Details of US machines are supplemented in Additional file 1: Table S1.

The needed sample size for this study was estimated using the equation [15] for a one-sided test:

in which P_T represents the expected sensitivity or specificity, P₀ represents a clinically acceptable lower bound for sensitivity or specificity, Z_1-α is the normal deviate at 1-α confidence level, and Z_{1– β} is the normal deviate at 1-β power, while α and β represent the probability of type I and type II errors respectively. The expected sensitivity and specificity for the AI system were 90% and 85% while the targeted sensitivity and specificity were 85% and 80%. For a confidence level of 95% and power of 80%, assuming a loss of 20% during data collection, 363 positive and 471 negative cases were needed.

The data collection started from November 2, 2021, and ended on February 21, 2022, to fulfill the need for sample size.

Inclusion and exclusion criteria

From November 2, 2021, to February 21, 2022, 1040 consecutive patients with 2309 thyroid nodules who underwent thyroid US examination at three medical centers were initially enrolled. Only patients with nodules who met all of the following criteria (Fig. 1) were included in this clinical study:

1. 1)

   Age ≥ 18 years, no gender restrictions;

2. 2)

   Thyroid nodules detected during US examination;

3. 3)

   Patients voluntarily participated in this study and signed informed consent forms; and

4. 4)

   Patients not recruited for any other clinical trials or studies within the past 30 days.

However, patients were excluded from this study if they met the following criteria:

1. 1)

   History of thyroidectomy, thyroid ablation, chemotherapy, or radiation therapy ( n = 1);

2. 2)

   Upon the patient’s or the family members’ request to withdraw from the study (n = 5); and

3. 3)

   Poor image quality (e.g., swallowing, breathing, coughing, speaking, neck movement) or improper technique (e.g., intermittent scanning, improper probe pressure), incomplete image data, and related examination reports (n = 7).

A total of 2296 thyroid nodules from 1036 patients were finally included for analysis.

Acquisition and quality control of US images

In this study, a total of 5 radiologists (2 radiologists from Medical Center 1, 1 radiologist from Medical Center 2, and 2 radiologists from Medical Center 3) acquired US images following their respective hospital’s US examination protocol. After US image acquisitions, these radiologists selected the most representative transverse and longitudinal planes of each nodule, de-identified and serialized the images, manually segmented regions of interests (ROIs) around the target nodules and subsequently transmitted the original US images and ROIs to the AI system for display and analysis. This is done not because the AI system is not capable of segmenting the target nodules, but to eliminate the compounding effect of mixing nodule segmentation with diagnostic performance, as we focused primarily on evaluating its diagnosis capability.

Based on the original images and supplied ROIs, the AI system provided its US…