QNT Method Comparison

QNT Method Comparison

Quantitative Method Comparison (MC)

MC Overview

Other Names - Comparison of Methods, Comparative Method Analysis, Intermethod Comparison, Cross-Method Validation, Method Equivalence Study, Analytical Method Comparison, Benchmark Method Comparison, Performance Comparison Study, Parallel Method Testing, Method Correlation Study, Alternative Method Evaluation

Method Comparison is one of the primary MV experiments. It involves comparing your instrument’s actual results with those of a true reference instrument. This verifies that the new method produces results that are in agreement

Definitions

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
Actual

Actual represents the real-world value measured by the instrument being evaluated.

Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100

Agreements

The total number of measurements between actual and reference value in agreement.

Bias

The difference in Results when comparing laboratory Quantitative methods with established methods.

Clinical Reportable Range

Range of analyte values that a method can measure, allowing for specimen dilution, concentration, or other pretreatment used to extend the direct analytical measurement range. The distinction between CRR and RR is that CRR may be up to the medical director's discretion of what range of results to report.

Control

A Quality Control material, usually developed by a manufacturer, with a known Expected Result used for monitoring Test performance.

Eligible (Sample and Results)

Samples that provide the necessary information for statistical calculations to be performed. Usually this is possessing both an actual and expected result.

Error Allowance Type

The type of error (TEa, SEa or REa) that will be used for calculating Error Index agreement between two results.

Experiment

A sampling of results that are statistically analyzed and interpreted in the evaluation of testing performance.

Label

The general identification used to identify a specific Sample.

Laboratory Developed Test (LDT)

A method developed by the laboratory or a regulator approved/validated method that has been modified.

Method Comparison (MC) Experiment

An MV experiment used to estimate the systematic difference on the basis of the differences observed between the methods. See: QL Method ComparisonQL Method Comparison

Method Validation

A systematic process to evaluate whether the performance of a medical Test meets quality goals to be used for medical testing.

Method Verification

A systematic process to evaluate whether the performance of a medical Test meets quality goals set by a validation. Performance evaluations may usually be found in a manufacturer insert.

Min. Samples

An Acceptance Criteria of the minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

MV

A broad term that can be used synonymously with Performance Evaluation, Method Validation and/or Method Verification.

Reference Method

Reference methods are well-established, highly accurate, and precise analytical procedures used as standards to evaluate the performance of other methods.

Reference Value

The value/result that represents the true value from a trusted reference source such as a verified instrument, EQA, or a commercial product.

Result(s)

A value or determination collected by measurement or calculation.

Sample Source

The original source of the Sample such as Sample, EQA, or the name of the Manufacturer.

Sample(s)

Individual specimens collected for testing representing the source. In MV experiments with Runs, this can refer to to the number of concentration levels used.

Spiked

A prepared sample, often using a Blank that has been "spiked" with a measurable analyte.

What is MC?

Method Comparison in medical laboratory verification is a process where a new method of testing is compared with a standard or established method. This is done to ensure that the new method is reliable, accurate, and consistent with the existing method. The new method should yield results that are clinically equivalent to the established method.

During this process, a variety of samples are tested using both methods and the results are compared. If the new method's results are within acceptable limits of the established method, the new method can be considered for implementation.

Method Comparison can be performed on Qualitative and Quantitative tests.

Experiment Settings and Acceptance Criterias

Quantitative Method Comparison allows for 2 different modes of evaluation:

image

Mode 1: TEa Mode

Use an error allowance to determine the agreement between measured and reference values. This is useful for evaluating an instrument with the same expected measurements as the reference. You may set the following minimum values for passing.

Min. Samples: The minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

Cualia Recommendations: Verifications: Quantitative 20-30 samples. Qualitative 10-30 samples. Validations/LDTs: 80-100 samples

Min. Agreement (%): The minimum percent of N results where the actual measurements are in agreement with their expected values.

Generally it is recommended to set Min. Agreement to 95% because it strikes a balance between clinical significance and statistical reliability in many statistical tests such as the Bland-Altman.

Error Allowance Type - The type of error (TEa, SEa or REa) that will be used for calculating Error Index agreement between two results. The TEa error values may be set in the test’s settings.

⚠️

If TEa values have not been set in the test, only an exact match between two results will yield agreement.

Mode 2: Regression Mode

image

In regression mode, measured and reference results are compared to assess the strength of the relationship. This approach is ideal for different methodologies or reagent changes that may have different reporting ranges.

Min. Samples: The minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

Cualia Recommendations: Verifications: Quantitative 20-30 samples. Qualitative 10-30 samples. Validations/LDTs: 80-100 samples

Regression Type - Calculates a correlation value to assess the relationship between two data sets. This value can be set to be greater than or less than a cutoff value in order to determine experiment passing.

Options: Deming Variance Ratio (vr), Passing Bablok Correlation Coefficient (r), Spearman Rank Correlation Coefficient (p), Sample , Correlation Coefficient (R), Pearson Correlation Coefficient (r) or R-squared (R²) Coefficient

Sample Selection

Choose a representative set samples that reflect the diversity and variability expected in routine testing. Ensure samples cover the full range of analyte concentrations and conditions relevant to the test.

It is recommended that most of the samples come from the true population. Some manufacturer products such as commercial kits and controls may be used. These may often be necessary for particularly difficult to obtain concentrations and conditions.

Testing Samples

Measure each sample on a reference method (expected) and on your new instrumentation (actual). Record the results, label and source either in the Cualia app or on your own platform. Include the label and the source.

Use the Cualia MV App

Make sure the data is entered into the experiment with the right acceptance criterias.

Preparation Checklist

Analyzer is set up in Cualia → AnalyzersAnalyzers
Tests are set up in the analyzer → Quantitative TestsQuantitative Tests and Qualitative TestsQualitative Tests
The initial MV details are prepared → MV Overview and DetailsMV Overview and Details

General Experiment Recommendations: Dos and Don’ts

❌
Don’t: Enter private patient information, identifiers or data into Cualia.
❌
Don’t: Rush Through an MV: Sometimes an MV can take months waiting for the right samples to come through. Based on our experience, the most time consuming part of an MV is finding out after days after measurements and painstakingly entering in the data to find that something was done incorrectly and the cycle must be repeated.
❌
Don’t: Rely too much on controls, calibrators and spiked samples: The goal of an MV is not to pass inspection but to truly evaluate that your instrumentation’s performance is up to clinical standards. Using samples that reflect the lab’s testing population will offer the best insights into the evaluation
✅
Do: Prepare your Cualia MV before taking measurements: Having a blueprint of work provides a smooth experience.
✅
Do: Enter Results Directly into Cualia: Taking down results to enter them into a spreadsheet just to copy them into Cualia will increase sources of error. When a result is returned, enter it directly into Cualia. You will be able to immediately have feedback into the success state of the experiment, identifying any missing variables that will hinder your MV.
✅
Do: Ask for clarification. Talk to regulators, auditor, consultants and don’t hesitate to reach out to support@cualia.io with questions.

Data Table

https://prod-files-secure.s3.us-west-2.amazonaws.com/e13ce2c5-b3e6-4355-a0db-3229e8cb15be/ac54c7ab-a23d-4597-ac5f-41e770aa64b5/2024-10-22_125735_Microsoft_Edge.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIAT73L2G45HZZMZUHI%2F20241022%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20241022T045951Z&X-Amz-Expires=86400&X-Amz-Signature=974ba8183303cb666e787cda4d1a894630e002ad2bcc7f0f5be2acc911e5db39&X-Amz-SignedHeaders=host&x-id=GetObject

Data Table Columns

Label | Free text - The general identification used to identify a specific Sample.

Source | Free text - The original source of the Sample such as Patient, EQA, or the name of the Manufacturer

Actual | Number only - Represents the real-world value measured by the instrument being evaluated.

Reference | Number only - The true value as measured from the reference method.

Result | Read Only - Indicates whether the measurements are within agreeable range.

Calculations

Bias

Bias - The difference between the actual value and the reference value for calculating agreement.

Bias=Actual−Reference\text{Bias} = {\text{Actual} - \text{Reference}}

Bias %

Bias % - The % difference between the actual value and the reference value for calculating agreement.

%Bias=Actual−ReferenceReference×100\text{\%Bias} = \frac{\text{Actual} - \text{Reference}}{\text{Reference}} \times 100

Error Index (Ei)

Error Index (Ei) - Proportion of the Bias to the Error Allowance. An Error Index must be between -1 and 1 to be considered in agreement. The Ei column will determine if agreement exists and is color coded to indicate agreement.

⚠️

A TEa / Allowable Error value must be set for an Error Index to be calculated

ErrorIndex(Ei)=∣Actual−Reference∣TEaError Index (Ei) = \frac{|\text{Actual} - \text{Reference}|}{\text{TEa}}

R-squared Coefficient of Determination (R²)

R-squared Coefficient of Determination (R²): R-squared is a statistical measure that represents the proportion of the variance in the dependent variable that is explained by the independent variables in a regression model. It ranges from 0 to 1, with higher values indicating a better fit. Usually >0.95 is considered statistical correlation.

R2=1−∑i=1n(yi−y^i)2∑i=1n(yi−yˉ)2R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}i)^2}{\sum{i=1}^{n} (y_i - \bar{y})^2}

R2R^2 = Coefficient of determination yiy_i = Observed value y^i\hat{y}_i= Predicted value yˉ\bar{y} = Mean of observed values nn = Sample size

Spearman Rank Correlation Coefficient (p)

Spearman Rank Correlation Coefficient (p): Measures the strength and direction of a monotonic relationship between two variables. A minimum absolute value of 0.95 is standard for strong correlation.

p=1−6∑di2n(n2−1)p = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)}

did_i= Difference between ranks of X and Y nn = Number of pairs

Passing Bablok Correlation Coefficient (r)

Passing Bablok Correlation Coefficient (r): Measures the strength and direction of the linear relationship between two measurement methods using Passing Bablok regression. A minimum absolute value of 0.95 is standard for strong correlation.

r=Cov(X,Y)Var(X)+Var(Y)r = \frac{Cov(X, Y)}{Var(X) + Var(Y)}

Cov(X,Y)Cov(X, Y) = Covariance of X and Y

Var(X)Var(X) = Variance of X

Var(Y)Var(Y) = Variance of Y

Spearman Rank Correlation Coefficient (p)

Spearman Rank Correlation Coefficient (p): Measures the strength and direction of a monotonic relationship between two variables. A minimum absolute value of 0.95 is standard for strong correlation.

p=1−6∑di2n(n2−1)p = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)}

did_i= Difference between ranks of X and Y nn = Number of pairs

Deming Variance Ratio (vr)

Deming Variance Ratio (vr): Quantifies the agreement between two measurement methods, considering measurement errors. A maximum allowable difference from 1 is standard.

VR=Var(X)+Var(Y)Var(X−Y)VR = \frac{Var(X) + Var(Y)}{Var(X - Y)}

Var(X)Var(X) = Variance of X Var(Y)Var(Y) = Variance of Y

Sample Correlation Coefficient (R)

Sample Correlation Coefficient (R): The sample correlation coefficient (R) quantifies the strength and direction of the linear relationship between two variables based on sample data. A min value of 0.95 is standard.

R=r2R = \sqrt{r^2}

r2r^2 = Coefficient of Determination

Pearson Correlation Coefficient (r)

Pearson Correlation Coefficient (r): Measures the strength and direction of a linear relationship between two variables. A minimum absolute value of 0.95 is standard for strong correlation.

r=Cov(X,Y)σXσYr = \frac{Cov(X, Y)}{\sigma_X \sigma_Y}

Cov(X,Y)Cov(X, Y) = Covariance of X and Y σXσ_X = Standard Deviation of X σYσ_Y = Standard Deviation of Y

Cov(X,Y)=1n−1∑i=1n(xi−xˉ)(yi−yˉ)Cov(X, Y) = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})

nn = Number of data points xix_i = Individual data point of variable XyiXy_i = Individual data point of variable Y xˉ\bar{x} = Mean of variable X yˉ\bar{y} = Mean of variable Y

Results

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
Min. Samples

An Acceptance Criteria of the minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

Min. Agreement

The minimum percentage of agreement required between the test results and the reference results from the acceptance criteria for the experiment to pass.

Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

Agreements

The total number of measurements between actual and reference value in agreement.

Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100

Mean Error Index (MEi)

Mean of all the Error Indexes calculated from a group of samples. A large distance from 0 can indicate a shift or Systematic error between the Actual and Reference result values.

Range Verified

Range of values from lowest to highest that was measured within a group of samples.

R-squared (R²) Coefficient

R-squared is a statistical measure that represents the proportion of the variance in the dependent variable that is explained by the independent variables in a regression model. It ranges from 0 to 1, with higher values indicating a better fit. Usually >0.95 is considered statistical correlation. R2=1−∑i=1n(yi−y^i)2∑i=1n(yi−yˉ)2R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}i)^2}{\sum{i=1}^{n} (y_i - \bar{y})^2} R2R^2 = Coefficient of determination yiy_i = Observed value y^i\hat{y}_i= Predicted value yˉ\bar{y} = Mean of observed values nn = Sample size

Mean (x̅)

Average value amongst a sample group

Random Allowable Error (REa)

The maximum amount of Random Error allowed for two results to be in agreement

Standard Deviation (σ or StdDev)

Statistical measure of the amount of variation from the mean. σ=1n−1∑i=1n(xi−xˉ)2\sigma = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2} σ = Standard Deviation n = Sample Size xix_i = Individual data point xˉ\bar{x} = Mean

Systematic Error Allowance (SEa)

Acceptable shift across the detectable range of a test in a single direction.

Pearson Correlation Coefficient (r)

Measures the strength and direction of a linear relationship between two variables. A minimum absolute value of 0.95 is standard for strong correlation. r=Cov(X,Y)σXσYr = \frac{Cov(X, Y)}{\sigma_X \sigma_Y} Cov(X,Y)Cov(X, Y) = Covariance of X and Y σXσ_X = Standard Deviation of X σYσ_Y = Standard Deviation of Y Cov(X,Y)=1n−1∑i=1n(xi−xˉ)(yi−yˉ)Cov(X, Y) = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) nn = Number of data points xix_i = Individual data point of variable XyiXy_i = Individual data point of variable Y xˉ\bar{x} = Mean of variable X yˉ\bar{y} = Mean of variable Y

Sample Correlation Coefficient (R)

The sample correlation coefficient (R) quantifies the strength and direction of the linear relationship between two variables based on sample data. A min value of 0.95 is standard. R=r2R = \sqrt{r^2} r2r^2 = Coefficient of Determination

Deming Variance Ratio (vr)

Quantifies the agreement between two measurement methods, considering measurement errors. A maximum allowable difference from 1 is standard. VR=Var(X)+Var(Y)Var(X−Y)VR = \frac{Var(X) + Var(Y)}{Var(X - Y)} Var(X)Var(X) = Variance of X Var(Y)Var(Y) = Variance of Y

Passing Bablok Correlation Coefficient (r)

Measures the strength and direction of the linear relationship between two measurement methods using Passing Bablok regression. A minimum absolute value of 0.95 is standard for strong correlation. r=Cov(X,Y)Var(X)+Var(Y)r = \frac{Cov(X, Y)}{Var(X) + Var(Y)} Cov(X,Y)Cov(X, Y) = Covariance of X and Y Var(X)Var(X) = Variance of X Var(Y)Var(Y) = Variance of Y

Spearman Rank Correlation Coefficient (p)

Measures the strength and direction of a monotonic relationship between two variables. A minimum absolute value of 0.95 is standard for strong correlation. p=1−6∑di2n(n2−1)p = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)} did_i= Difference between ranks of X and Y nn = Number of pairs

Line Chart

The line chart plots the results in a line chart. The min and max ranges on the X and Y axis are from the Range Low and Range High (AMR) set within the Test Settings.

X-axis = Reference results

Y-axis = Actual results

Solid Black Line = Line of best fit

Dotted Red Line = TEa min and max

Dotted Blue Line = If using SEa or REa, the min and max acceptable for that Error Allowance

image

Error Index Chart

Only on MC experiments set to Error Allowance mode.

Error Index Chart is used to visualize the differences between actual and the Error Allowance from the reference value, helping to assess the accuracy of a model.

image

Y-axis - Error Index: Error Allowance from the reference value.

X-axis -Actual Result.

0 Line - Ideal prediction, no error.

Red Lines - Error thresholds at +1 and -1 – acceptable error range.

Data Points - Error for each observation – closer to 0 means better prediction.

Bias Chart

The bias chart visually represents the difference between measured and reference values, highlighting systematic and random error.

Y-Axis - Bias between measured and reference results.

X-Axis - Measured result value

Dotted Line - Mean bias. A large mean bias can indicate a shift or systemic error.

image

Percentage Bias Chart

The percentage bias chart visually represents the difference between measured and reference values.

Y-Axis - Percent bias between measured and reference results.

X-Axis - Measured result value

image

Experiment Pass / Fail

Error Allowance Mode

image

When both samples and agreement satisfy the acceptance criterias, the experiment will be in a Passed state indicated with a green check.

Regression Mode

image

Samples

Samples - The number of eligible samples tested. In the results bar, the color will turn green when the number of samples has satisfied the Min. Samples acceptance criteria to indicate it has passed.

Agreement

Agreement - The proportion (%) of test results that match the reference results. In the results bar, the color will be green and be considered passed when it satisfies the Min. Agreement or Min. Between-Day Agreement acceptance criteria.

Sample Coefficient

Sample Coefficient - This value show the regression type set in the experiment settings. The value will be red or green depending on if the results satisfy the requirements.