QL Method Comparison

QL Method Comparison

Qualitative Method Comparison (MC)

Overview

Other Names - Comparison of Methods, Comparative Method Analysis, Intermethod Comparison, Cross-Method Validation, Method Equivalence Study, Analytical Method Comparison, Benchmark Method Comparison, Performance Comparison Study, Parallel Method Testing, Method Correlation Study, Alternative Method Evaluation

Method Comparison is one of the primary MV experiments. It involves comparing your instrument’s actual results with those of a true reference instrument. This verifies that the new method produces results that are in agreement

Lab Juice

Definitions

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
Result(s)

A value or determination collected by measurement or calculation.

Agreements

The total number of measurements between actual and reference value in agreement.

Specificity (Qualitative)

The ability of the test to correctly identify true negatives. Specificity=TNTN+FP×100%\text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}} \times 100\%

Sensitivity (Qualitative)

The ability of the test to correctly identify true positives. Sensitivity=TPTP+FN×100%\text{Sensitivity} = \frac{\text{TP}}{\text{TP} + \text{FN}} \times 100\%

Negative Predictive Value (NPV)

The proportion of negative test results that are true negatives. NPV=TNTN+FN×100%\text{NPV} = \frac{\text{TN}}{\text{TN} + \text{FN}} \times 100\%

False Positive (FP)

The number of false positive results where the test incorrectly detected the presence of the substrate when compared to the reference method.

False Negative (FN)

The number of false negative results where the test incorrectly missed the presence of the substrate when compared to the reference method.

True Negative (TN)

The number of true negative results where the test correctly identified the absence of the substrate when compared to the reference method.

True Positive (TP)

The number of true positive results where the test correctly detected the presence of the substrate when compared to the reference method.

Reference Method

Reference methods are well-established, highly accurate, and precise analytical procedures used as standards to evaluate the performance of other methods.

Negative

Value from Qualitative or Semi-Quantitative set in Tests that expresses non-detection of the measured substrate.

Min. Samples

An Acceptance Criteria of the minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

Eligible (Sample and Results)

Samples that provide the necessary information for statistical calculations to be performed. Usually this is possessing both an actual and expected result.

Method Comparison (MC) Experiment

An MV experiment used to estimate the systematic difference on the basis of the differences observed between the methods. See: QL Method ComparisonQL Method Comparison

Pos Label

A label used in a Qualitative test to express a that a sample has detectable levels of substrate

Neg Label

A label used in a Qualitative test to express a that a sample does not have detectable levels of substrate

Method Verification

A systematic process to evaluate whether the performance of a medical Test meets quality goals set by a validation. Performance evaluations may usually be found in a manufacturer insert.

Experiment

A sampling of results that are statistically analyzed and interpreted in the evaluation of testing performance.

Reference Value

The value/result that represents the true value from a trusted reference source such as a verified instrument, EQA, or a commercial product.

Spiked

A prepared sample, often using a Blank that has been "spiked" with a measurable analyte.

MV

A broad term that can be used synonymously with Performance Evaluation, Method Validation and/or Method Verification.

Sample Source

The original source of the Sample such as Sample, EQA, or the name of the Manufacturer.

Qualitative

A Result Type for a Test that can only have two possible options

Qualitative Agreement

In Qualitative testing a matching pair of Actual and Reference Results are considered in Agreement

Label

The general identification used to identify a specific Sample.

Method Validation

A systematic process to evaluate whether the performance of a medical Test meets quality goals to be used for medical testing.

Actual

Actual represents the real-world value measured by the instrument being evaluated.

Sample(s)

Individual specimens collected for testing representing the source. In MV experiments with Runs, this can refer to to the number of concentration levels used.

Control

A Quality Control material, usually developed by a manufacturer, with a known Expected Result used for monitoring Test performance.

Laboratory Developed Test (LDT)

A method developed by the laboratory or a regulator approved/validated method that has been modified.

Positive

Values Qualitative or Semi-Quantitative set in Tests that can express a Result that indicates a Positive Result.

Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100 See: Agreements

Positive Predictive Value (PPV)

The probability that a Positive Result indicates the Expected analyte Result is a positive. PPV=TPTP+FPPPV = \frac{TP}{TP + FP}

What is Method Comparison?

Method Comparison in medical laboratory verification is a process where a new method of testing is compared with a standard or established method. This is done to ensure that the new method is reliable, accurate, and consistent with the existing method. The new method should yield results that are clinically equivalent to the established method.

During this process, a variety of samples are tested using both methods and the results are compared. If the new method's results are within acceptable limits of the established method, the new method can be considered for implementation.

Method Comparison can be performed on Qualitative and Quantitative tests.

How to Perform Qualitative Method Comparison

1. Define Acceptance Criterias

In Cualia’s Method Comparison Experiment there are two Acceptance Criterias:

Min. Samples: The minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

Cualia Recommendations: Verifications: Quantitative 20-30 samples. Qualitative 10-30 samples. Validations/LDTs: 80-100 samples

Min. Agreement (%): The minimum percent of N results where the actual measurements are in agreement with their expected values.

Generally it is recommended to set Min. Agreement to 95% because it strikes a balance between clinical significance and statistical reliability in many statistical tests such as the Bland-Altman.

Sample Selection

Choose a representative set samples that reflect the diversity and variability expected in routine testing. Ensure samples cover the full range of analyte concentrations and conditions relevant to the test.

It is recommended that most of the samples come from the true population. Some manufacturer products such as commercial kits and controls may be used. These may often be necessary for particularly difficult to obtain concentrations and conditions.

Testing Samples

Measure each sample on a reference method (expected) and on your new instrumentation (actual). Record the results, label and source either in the Cualia app or on your own platform. Include the label and the source.

Use the Cualia MV App

Make sure the data is entered into the experiment with the right acceptance criterias.

Reference Methods

Reference Methods are well-established, highly accurate, and precise analytical procedures used as standards to evaluate the performance of other methods.

For testing you will need to select a Reference Method to act as your reference result to match with the actual results of the new test.

It is recommended to use a validated and verified source of the same model/make as the analyzer being tested to ensure compatibility of the measurements.

Preparation Checklist

Analyzer is set up in Cualia → AnalyzersAnalyzers
Tests are set up in the analyzer → Quantitative TestsQuantitative Tests and Qualitative TestsQualitative Tests
The initial MV details are prepared → MV Overview and DetailsMV Overview and Details

General Experiment Recommendations: Dos and Don’ts

Don’t: Enter private patient information, identifiers or data into Cualia.
Don’t: Rush Through an MV: Sometimes an MV can take months waiting for the right samples to come through. Based on our experience, the most time consuming part of an MV is finding out after days after measurements and painstakingly entering in the data to find that something was done incorrectly and the cycle must be repeated.
Don’t: Rely too much on controls, calibrators and spiked samples: The goal of an MV is not to pass inspection but to truly evaluate that your instrumentation’s performance is up to clinical standards. Using samples that reflect the lab’s testing population will offer the best insights into the evaluation
Do: Prepare your Cualia MV before taking measurements: Having a blueprint of work provides a smooth experience.
Do: Enter Results Directly into Cualia: Taking down results to enter them into a spreadsheet just to copy them into Cualia will increase sources of error. When a result is returned, enter it directly into Cualia. You will be able to immediately have feedback into the success state of the experiment, identifying any missing variables that will hinder your MV.
Do: Ask for clarification. Talk to regulators, auditor, consultants and don’t hesitate to reach out to support@cualia.io with questions.

Data Table

image

There are 5 relevant columns in the Qualitative Method Comparison Data Table:

Label | Free text - The general identification used to identify a specific Sample.

Source | Free text - The original source of the Sample such as Patient, EQA, or the name of the Manufacturer

Actual | Pos or Neg Label only - Represents the real-world value measured by the instrument being evaluated.

Reference | Pos or Neg Label only - The true value as measured from the reference method.

Result | Read Only - Indicates whether the measurements are within agreeable range.

Qualitative Method Comparison Results

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
Min. Samples

An Acceptance Criteria of the minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

Min. Agreement

The minimum percentage of agreement required between the test results and the reference results from the acceptance criteria for the experiment to pass.

Sample(s)

Individual specimens collected for testing representing the source. In MV experiments with Runs, this can refer to to the number of concentration levels used.

Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

Agreements

The total number of measurements between actual and reference value in agreement.

Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100 See: Agreements

True Positive (TP)

The number of true positive results where the test correctly detected the presence of the substrate when compared to the reference method.

True Negative (TN)

The number of true negative results where the test correctly identified the absence of the substrate when compared to the reference method.

False Positive (FP)

The number of false positive results where the test incorrectly detected the presence of the substrate when compared to the reference method.

False Negative (FN)

The number of false negative results where the test incorrectly missed the presence of the substrate when compared to the reference method.

Sensitivity (Qualitative)

The ability of the test to correctly identify true positives. Sensitivity=TPTP+FN×100%\text{Sensitivity} = \frac{\text{TP}}{\text{TP} + \text{FN}} \times 100\%

Specificity (Qualitative)

The ability of the test to correctly identify true negatives. Specificity=TNTN+FP×100%\text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}} \times 100\%

Negative Predictive Value (NPV)

The proportion of negative test results that are true negatives. NPV=TNTN+FN×100%\text{NPV} = \frac{\text{TN}}{\text{TN} + \text{FN}} \times 100\%

Positive Predictive Value (PPV)

The probability that a Positive Result indicates the Expected analyte Result is a positive. PPV=TPTP+FPPPV = \frac{TP}{TP + FP}

Qualitative Method Comparison Acceptance

image

When both samples and agreement satisfy the acceptance criterias, the experiment will be in a Passed state indicated with a green check.

Samples

Samples - The number of eligible samples tested. In the results bar, the color will turn green when the number of samples has satisfied the Min. Samples acceptance criteria to indicate it has passed.

Agreement

Agreement - The proportion (%) of test results that match the reference results. In the results bar, the color will be green and be considered passed when it satisfies the Min. Agreement or Min. Between-Day Agreement acceptance criteria.