Carotid Plaque Measurement by Ultrasound Is Highly Reliable, Large Review Finds

Table of Contents

• Key Points
• Background: Why This Research Matters
• How the Research Was Conducted
• Key Findings: Reliability of Plaque Measurements
• Reliability of Total Plaque Area (TPA)
• Reliability of Total Plaque Volume (TPV)
• Supporting Evidence from Major Studies
• Plaque Thickness and Area: Both Reliable and Predictive
• What This Means for Patients
• Limitations of the Study
• Recommendations
• Frequently Asked Questions
• Source Information

Key Points

• A review of 7 studies found almost perfect reliability for ultrasound measurement of carotid plaque burden, both between different observers and within the same observer.
• For total plaque area, inter-observer reliability was 0.94 and intra-observer reliability was 0.96; for total plaque volume, both were 0.91.
• These findings counter criticism that plaque measurement is unreliable and support its use in vascular prevention clinics.
• Reliable plaque measurements let doctors track whether treatments are reducing plaque and intensify therapy if needed.
• Plaque measurements provided better risk prediction than intima-media thickness (IMT) in the studies reviewed.

Background: Why This Research Matters

Carotid plaque is a buildup of fatty deposits in the arteries of the neck that supply blood to the brain. This plaque buildup is a major risk factor for stroke and other cardiovascular events. Doctors can measure the amount of plaque using ultrasound imaging, a safe and non-invasive technique.

Two key measurements are used to quantify plaque burden. Total plaque area (TPA) measures the surface area of plaque seen on a two-dimensional (2D) ultrasound image. Total plaque volume (TPV) measures the three-dimensional (3D) volume of plaque. Both measurements help doctors assess a patient's risk of future cardiovascular problems and guide treatment decisions.

However, some critics have questioned whether these measurements are reliable enough to be used in routine clinical practice. They have raised concerns about whether different ultrasound technicians would get the same results when measuring the same patient's plaque. This systematic review and meta-analysis was designed to answer that question definitively by pooling data from all available studies on the topic.

The importance of this question is significant. If plaque measurements are unreliable, doctors cannot confidently use them to track whether a patient's treatment is working or to decide when to intensify therapy. If the measurements are reliable, as this study found, then plaque burden becomes a valuable tool for personalized cardiovascular prevention.

How the Research Was Conducted

The researchers followed two internationally recognized sets of guidelines for their review. They used the PRISMA guidelines (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and the GRRAS guidelines (Guidelines for Reporting Reliability and Agreement Studies). The study was registered under the number CRD42020210453.

The research team performed a comprehensive search of the medical literature as of May 19, 2021. They searched four major databases: MEDLINE/PubMed, Scopus, Embase, and Web of Science. They used combinations of keywords and medical subject headings to find studies that investigated the accuracy and reproducibility of carotid plaque measurement using 2D and 3D ultrasound.

The search terms included "Carotid Stenosis," "Carotid plaque," "Plaque Burden," "Total Plaque Area," "Total Plaque Volume," "TPV," and "TPA." These were combined with terms related to reliability. The reliability terms included "Inter-rater reliability," "Inter-observer reliability," "Intra-rater reliability," "Kappa value," "Intraclass correlation coefficient," "Correlation coefficient," and "Cohen's kappa."

The researchers used EndNote X9 software to manage their findings and remove duplicate studies. They also checked the reference lists of relevant studies and reviews to identify any additional studies that might have been missed by the database search.

Inclusion and exclusion criteria: The team included all relevant papers that reported the accuracy and reproducibility of carotid plaque measurement using 2D and 3D ultrasound. Reproducibility was expressed as either inter-rater reliability (agreement between different observers) or intra-rater reliability (agreement when the same observer repeats the measurement).

According to the GRRAS guidelines, studies needed to report the following information to be included: sample size, number of raters and subjects, sampling method, rating process, statistical analysis, and reliability coefficients. The researchers gathered all available statistical information, including Kappa coefficients (weighted and unweighted), intraclass correlation coefficients, Pearson correlation coefficients, and Spearman rank correlation coefficients.

When a single study reported both a correlation coefficient and a kappa value, the researchers used only the correlation coefficient for the final analysis. They excluded studies that did not confirm ethics approval, animal studies, short reports or letters without adequate information, case studies, retracted or withdrawn articles, previous reviews, and studies with insufficient information.

Two authors independently assessed the search findings based on the inclusion criteria. Any disagreements were resolved through discussion and consensus.

Data extraction: The researchers extracted specific information from each included study using a predefined spreadsheet. This included sample size, number of raters and subjects, sampling method, rating process, statistical analysis, and reliability coefficients. Each sample was considered a unit of analysis. If the same sample was reported in more than two studies, it was included only once. If several samples were reported in one study, each was included separately.

Quality assessment: Each selected study was scored on eight quality criteria. Studies with a score greater than 6 out of 8 were considered qualified. The quality criteria included whether the study explained how the sample size was chosen, described the sampling method, described the measurement process, and stated whether measurements were conducted independently. The quality criteria also included whether the study described the statistical analysis, stated the actual number of raters and subjects, and described the characteristics of raters and subjects. The quality criteria also included whether the study reported estimates of reliability with measures of statistical uncertainty.

Statistical analysis: The researchers pooled data from all studies for different types of reliability measurements. When studies reported reliability as a kappa statistic, the researchers converted these to an r-type coefficient ranging from −1.00 to +1.00. They categorized agreement as follows:

• Poor: kappa = 0.00–0.20
• Fair: kappa = 0.21–0.40
• Moderate: kappa = 0.41–0.60
• Substantial: kappa = 0.61–0.80
• Almost perfect: kappa = 0.81–1.00

In meta-analyses, the kappa statistic can be treated as a correlation coefficient. Therefore, the researchers transformed all reliability coefficients into Fisher's Z values. This transformation stabilizes variance and normalizes data distributions when pooling results. They then performed a meta-analysis using a random-effects model for each type of reliability coefficient. Finally, they transformed the pooled results back from Fisher's Z to reliability coefficients to make them easier to interpret. All statistical analyses were performed using the 'meta' package in R software.

Key Findings: Reliability of Plaque Measurements

The initial search retrieved a total of 1,421 papers. After removing duplicates and reviewing abstracts, titles, and full texts, the researchers selected 7 articles for the final review. The two independent reviewers had near-perfect agreement on which papers to include, with a kappa value of 0.89.

Of the 7 selected articles, 6 were included in the meta-analysis for inter-rater reliability and 5 articles (reporting 6 studies) were included for intra-rater reliability. All selected studies reported correlation coefficients, so no kappa values were entered into the final analysis.

The quality assessment showed that all included studies scored 6 or 7 out of 8 possible points, indicating good methodological quality. The studies came from several countries, including Canada, the United States, Switzerland, Germany, Spain, and Denmark.

Reliability of Total Plaque Area (TPA)

The reliability of total plaque area measurement was excellent. The inter-observer reliability was 0.94 (95% CI: 0.83–0.99). This means that when two different ultrasound technicians measured the same patient's plaque area, their results agreed 94% of the time. The intra-observer reliability was 0.96 (95% CI: 0.94–0.97). This means that when the same technician measured the same patient's plaque area on two different occasions, the results agreed 96% of the time.

These values fall into the "almost perfect" category of agreement. In practical terms, this means that TPA measurement is highly consistent regardless of who performs the measurement or when it is performed.

The researchers noted that TPA is reliable and easy to teach and learn. Most experienced and skillful vascular ultrasound technologists can learn to measure TPA reliably in a single day. This is an important practical advantage because it means clinics can quickly train staff to perform this measurement.

Unlike intima-media thickness (IMT), another ultrasound measurement sometimes used to assess cardiovascular risk, the quantities measured with TPA are much larger than the spatial resolution of ultrasound. This means that measurement errors represent a very small fraction of the total quantity. The interobserver coefficient of variation for TPA (standard deviation divided by the mean) is approximately 1.5%, which is very low.

Reliability of Total Plaque Volume (TPV)

Total plaque volume is measured using three-dimensional (3D) ultrasound. The inter-observer reliability of TPV was 0.91 (95% CI: 0.62–0.98). The intra-observer reliability was also 0.91 (95% CI: 0.83–0.95).

These values also fall into the "almost perfect" category. The confidence interval for inter-observer reliability was wider (0.62–0.98) than for intra-observer reliability (0.83–0.95), which reflects the smaller number of studies available for TPV analysis. However, both measures indicate strong reliability.

The 3D ultrasound technique captures the full volume of plaque rather than just a cross-sectional area. This provides additional information about plaque burden that may be useful in certain clinical and research settings.

Supporting Evidence from Major Studies

Several individual studies included in the review provided detailed reliability data that support the overall findings.

The Northern Manhattan Study (NOMAS): This National Institutes of Health-funded study examined stroke risk and cognitive decline. Researchers conducted a reproducibility analysis for carotid plaque burden measured by two sonologists trained in performing and reading research ultrasound studies. Among 105 randomly selected subjects, both sonologists identified 101 plaques and were in complete agreement on the number of plaques and plaque locations within each individual (kappa = 1). The intraclass correlation coefficient was 0.94 for measurements of total carotid plaque area between the two readers. The NOMAS team also reported excellent reproducibility of maximum carotid plaque thickness, with an interrater intraclass correlation coefficient of 0.87 and an intra-rater intraclass correlation coefficient of 0.94.

The Tromsø Study: This large population-based study from Norway reported detailed reproducibility data for TPA measurements. The inter-observer mean arithmetic difference was −1.0 mm² with a standard deviation of 4.3, and the limits of agreement were ± 8.4 mm². For the first sonographer, the intra-observer mean arithmetic difference was 0.2 mm² with a standard deviation of 3.0, and the limits of agreement were ± 6.0 mm². For the second sonographer, the corresponding values were 0.01 mm² with a standard deviation of 3.6, and limits of agreement of ± 7.1 mm². These differences are very small compared to the average plaque burden of vascular patients.

Stroke Prevention & Atherosclerosis Research Centre (SPARC) Database: Among 9,864 patients referred for vascular prevention in London, Canada, the mean age was 62.4 years with a standard deviation of 14 years. The range of TPA was from 0 to 1,289 mm², with a mean of 119 mm² and a standard deviation of 135 mm². Among patients with TPA greater than 0 mm², the mean age was 64.56 years with a standard deviation of 13.07 years, and the mean TPA was 134.46 mm² with a standard deviation of 137.42 mm².

Multicenter Reliability Study: To demonstrate that the results apply to other ultrasound laboratories and clinics, researchers conducted a study of interobserver reliability in which plaque area measurements in 25 patients were repeated a week apart by 2 technicians using 2 different machines. The senior technologist, who had been performing these measurements for 8 years, used a new high-resolution ATL HDI 5000 scanner. The junior technologist, who had been performing such measurements for 1 year, used an ATL Mark 9 duplex scanner. The reliability (intraclass correlation) was 0.85, with the senior technician using the higher-resolution machine systematically measuring more plaque.

Swiss and German Cohort Study: Romanens and colleagues reported intra-observer reproducibility of TPA in a study of 2 middle-aged cohorts from Switzerland and Germany. For the cut-offs of TPA (0–9, 10–49, 50–99, and ≥100 mm²), the linear weighted kappa value was excellent at 0.97 (95% CI 0.92–1.00).

Plaque Thickness and Area: Both Reliable and Predictive

Plaque thickness (also called plaque height) and plaque area are both highly reproducible measurements. In the Northern Manhattan Study, the intraclass correlation coefficient for inter-observer reliability of plaque thickness was 0.77, and for intra-observer reliability it was 0.94. The intraclass correlation coefficients for plaque area were 0.78 and 0.94, respectively.

Both plaque thickness and plaque area showed similar predictive value for future cardiovascular events. In models fully adjusted for risk factors, the hazard ratio for carotid plaque thickness was 3.13 (95% CI 1.80–5.51, P < 0.001) for all major adverse cardiovascular events (MACE). This was similar to the hazard ratio for carotid plaque burden (plaque area).

Importantly, intima-media thickness (IMT) did not improve risk prediction significantly. The non-categorical net reclassification index for carotid plaque thickness was 0.173 (95% CI 0.109–0.243, P < 0.001) for all MACE, similar to carotid plaque burden. IMT assessment did not result in a significant net reclassification index.

This finding suggests that plaque measurements provide better risk prediction than IMT, which has been more commonly used in some clinical settings.

What This Means for Patients

This study provides strong evidence that ultrasound measurement of carotid plaque burden is a reliable and reproducible method. Patients can have confidence that their plaque measurements will be consistent, regardless of which technician performs the ultrasound or when the measurement is taken.

The findings directly address a common criticism that plaque measurement is unreliable. The critics were wrong. The data show almost perfect inter-observer and intra-observer reliability for both 2D and 3D ultrasound measurements.

For patients, this means that:

• Plaque burden measurements can be trusted to guide treatment decisions
• Changes in plaque measurements over time likely reflect real changes in disease, not measurement error
• Different clinics and technicians should produce comparable results
• Plaque measurement is a valuable tool for monitoring the effectiveness of cardiovascular prevention therapies

The authors conclude that plaque measurement with ultrasound is an accurate and reproducible method that should be used widely in vascular prevention clinics for targeted management of vascular diseases. This represents a shift from traditional risk factor-based approaches to a more personalized, imaging-based approach to cardiovascular prevention.

The ability to reliably measure plaque burden means that doctors can track whether treatments such as cholesterol-lowering medications, blood pressure control, and lifestyle changes are actually reducing plaque. If plaque is not shrinking or is still growing despite treatment, doctors can intensify therapy. This personalized approach may prevent more heart attacks and strokes than a one-size-fits-all strategy.

Limitations of the Study

The review included only 7 studies, which is a relatively small number. For total plaque volume, only a few studies were available, which resulted in wider confidence intervals for the reliability estimates.

The studies included in the review came from specialized research centers with experienced sonographers. The reliability may be lower in community settings with less experienced technicians. However, the multicenter reliability study described above suggests that good reliability can be achieved even with less experienced staff.

The review focused on reliability (consistency of measurements) rather than accuracy (whether the measurements correctly reflect the true amount of plaque). However, the authors note that accuracy of plaque measurement assessed by phantoms and by histology has been reported by Lopez-Melgar and colleagues, supporting the validity of the technique.

Most studies measured plaque in the carotid arteries. The reliability of plaque measurement in other arteries may differ, although the principles are likely similar.

Recommendations

Based on this research, the following recommendations can be made for patients and healthcare providers:

1. Ask about plaque measurement: If you have risk factors for cardiovascular disease, ask your doctor whether carotid plaque measurement might be appropriate for assessing your risk.
2. Trust the measurements: If your doctor orders a carotid plaque measurement, you can have confidence that the results are reliable and reproducible.
3. Use plaque measurements to track progress: If you are receiving treatment to reduce cardiovascular risk, plaque measurements can help determine whether the treatment is working.
4. Consider plaque measurement over IMT: Plaque measurements appear to provide better risk prediction than intima-media thickness (IMT) measurements.
5. Discuss results with your doctor: Plaque burden measurements should be interpreted in the context of your overall cardiovascular risk profile.

The authors strongly recommend that plaque measurement with ultrasound should be used widely in vascular prevention clinics. This technique offers a reliable, non-invasive way to assess cardiovascular risk and to monitor the effectiveness of prevention strategies.

Frequently Asked Questions

What is carotid plaque and why does it matter?

Carotid plaque is a buildup of fatty deposits in the arteries of the neck that supply blood to the brain. This plaque buildup is a major risk factor for stroke and other cardiovascular events. Doctors can measure the amount of plaque using ultrasound imaging, a safe and non-invasive technique. Two key measurements are used to quantify plaque burden: total plaque area (TPA) and total plaque volume (TPV).

How reliable are ultrasound measurements of carotid plaque?

A systematic review and meta-analysis of 7 studies found that ultrasound measurement of carotid plaque burden has almost perfect reliability between different observers and within the same observer. For total plaque area, inter-observer reliability was 0.94 and intra-observer reliability was 0.96. For total plaque volume, both inter-observer and intra-observer reliability were 0.91. These findings directly counter the criticism that plaque measurement is unreliable.

What do the reliability numbers mean for my plaque measurements?

Reliability values like 0.94 and 0.96 fall into the 'almost perfect' category of agreement. This means that when two different ultrasound technicians measured the same patient's plaque area, their results agreed 94% of the time. When the same technician measured the same patient's plaque area on two different occasions, the results agreed 96% of the time. So you can have confidence that your plaque measurements will be consistent.

Can I trust that changes in my plaque measurements over time are real?

Yes. Because the measurements are highly reliable, changes in plaque measurements over time likely reflect real changes in disease, not measurement error. This means doctors can track whether treatments such as cholesterol-lowering medications, blood pressure control, and lifestyle changes are actually reducing plaque. If plaque is not shrinking or is still growing despite treatment, doctors can intensify therapy.

How is carotid plaque measured and is it safe?

Carotid plaque is measured using ultrasound imaging, a safe and non-invasive technique. Two key measurements are used: total plaque area (TPA), which measures the surface area of plaque seen on a two-dimensional ultrasound image, and total plaque volume (TPV), which measures the three-dimensional volume of plaque using 3D ultrasound. Both measurements help doctors assess a patient's risk of future cardiovascular problems and guide treatment decisions.

How quickly can a technician learn to measure carotid plaque reliably?

Most experienced and skillful vascular ultrasound technologists can learn to measure total plaque area reliably in a single day. This is an important practical advantage because it means clinics can quickly train staff to perform this measurement. The researchers noted that total plaque area is reliable and easy to teach and learn, which supports its use in routine clinical practice.

Is carotid plaque measurement better than intima-media thickness (IMT) for predicting risk?

In models fully adjusted for risk factors, the hazard ratio for carotid plaque thickness was 3.13 for all major adverse cardiovascular events, similar to carotid plaque burden. Importantly, intima-media thickness (IMT) did not improve risk prediction significantly. The non-categorical net reclassification index for carotid plaque thickness was 0.173 for all major adverse cardiovascular events, similar to carotid plaque burden. IMT assessment did not result in a significant net reclassification index.

If my carotid ultrasound shows plaque, when should I get a second opinion on the measurement?

A second opinion is reasonable when a plaque measurement is being used to guide treatment decisions, since total plaque area and total plaque volume show almost perfect reliability between different technicians and within the same technician. If your results seem inconsistent with your risk profile, or if a clinic proposes changing therapy based on a single reading, an independent review of the images can confirm whether the measurement is trustworthy. Plaque measurements should be interpreted alongside your overall cardiovascular risk profile. Diagnostic Detectives Network provides independent expert second opinions.

Source Information

Original article title: Spence Supplemental 2020 Reliability and reproducibility of measurement of carotid plaque burden

Authors: Spence JD et al.

Publication year: 2020

Study registration: CRD42020210453

Note: This patient-friendly article is based on peer-reviewed research. The original article was a systematic review and meta-analysis of 7 studies examining the reliability and reproducibility of carotid plaque measurement using 2D and 3D ultrasound.