QL Trueness

QL Trueness

🛠️ QL Trueness

Overview

Other Names - Accuracy, Bias Test, Correctness Verification, Accuracy Assessment, Method Accuracy Evaluation, Bias Measurement, Correctness Validation, Accuracy Verification, Bias Analysis, Method Accuracy Check.

Trueness is one of the primary MV experiments. It involves comparing your instrument’s actual results with those an EQA qualified sample. The mechanism is similar to Method Comparison, but compares actual results with highly validated results for comparison. Few samples may be used for comparison due to stringent sample requirements.

Lab Juice

Definitions

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
Actual

Actual represents the real-world value measured by the instrument being evaluated.

Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100

Agreements

The total number of measurements between actual and reference value in agreement.

Eligible (Sample and Results)

Samples that provide the necessary information for statistical calculations to be performed. Usually this is possessing both an actual and expected result.

EQA Sample

A specimen provided by an external quality assessment organization to participating laboratories for testing. The results from these samples are used to evaluate the laboratory's performance and accuracy by comparing it against a benchmark or other participating laboratories.

Experiment

A sampling of results that are statistically analyzed and interpreted in the evaluation of testing performance.

External Quality Assessment (EQA)

External Quality Assessment (EQA) evaluates a laboratory's performance by comparing its results with other labs through external sample testing. This ensures accuracy and consistency in diagnostic testing.

False Negative (FN)

The number of false negative results where the test incorrectly missed the presence of the substrate when compared to the reference method.

False Positive (FP)

The number of false positive results where the test incorrectly detected the presence of the substrate when compared to the reference method.

Label

The general identification used to identify a specific Sample.

Method Comparison (MC) Experiment

An MV experiment used to estimate the systematic difference on the basis of the differences observed between the methods. See: QL Method ComparisonQL Method Comparison

Method Verification

A systematic process to evaluate whether the performance of a medical Test meets quality goals set by a validation. Performance evaluations may usually be found in a manufacturer insert.

Min. Agreement

The minimum percentage of agreement required between the test results and the reference results from the acceptance criteria for the experiment to pass.

Min. Samples

An Acceptance Criteria of the minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

Neg Label

A label used in a Qualitative test to express a that a sample does not have detectable levels of substrate

Negative

Value from Qualitative or Semi-Quantitative set in Tests that expresses non-detection of the measured substrate.

Negative Predictive Value (NPV)

The proportion of negative test results that are true negatives. NPV=TNTN+FN×100%\text{NPV} = \frac{\text{TN}}{\text{TN} + \text{FN}} \times 100\%

Pos Label

A label used in a Qualitative test to express a that a sample has detectable levels of substrate

Positive

Values Qualitative or Semi-Quantitative set in Tests that can express a Result that indicates a Positive Result.

Positive Predictive Value (PPV)

The probability that a Positive Result indicates the Expected analyte Result is a positive. PPV=TPTP+FPPPV = \frac{TP}{TP + FP}

Qualitative

A Result Type for a Test that can only have two possible options

Qualitative Agreement

In Qualitative testing a matching pair of Actual and Reference Results are considered in Agreement

Reference Method

Reference methods are well-established, highly accurate, and precise analytical procedures used as standards to evaluate the performance of other methods.

Reference Value

The value/result that represents the true value from a trusted reference source such as a verified instrument, EQA, or a commercial product.

Result(s)

A value or determination collected by measurement or calculation.

Sample(s)

Individual specimens collected for testing representing the source. In MV experiments with Runs, this can refer to to the number of concentration levels used.

Sensitivity (Qualitative)

The ability of the test to correctly identify true positives. Sensitivity=TPTP+FN×100%\text{Sensitivity} = \frac{\text{TP}}{\text{TP} + \text{FN}} \times 100\%

Specificity (Qualitative)

The ability of the test to correctly identify true negatives. Specificity=TNTN+FP×100%\text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}} \times 100\%

True Negative (TN)

The number of true negative results where the test correctly identified the absence of the substrate when compared to the reference method.

True Positive (TP)

The number of true positive results where the test correctly detected the presence of the substrate when compared to the reference method.

What is the Trueness Experiment?

The Trueness Experiment assesses the accuracy of a measurement method, indicating how close the average of a series of test results is to the true value of the analyte. High trueness means that the method produces results that are consistently accurate and free from systematic errors or bias. This is crucial for ensuring the reliability and validity of diagnostic tests.

The Trueness Experiment involves comparing the test results from the method being verified against known reference values or a gold standard.

External Quality Assessments (EQAs) and Interlaboratory Comparisons (Interlab) are integral to evaluating trueness. In EQAs, samples are distributed to multiple laboratories to assess their performance and ensure accuracy. Interlab comparisons involve multiple laboratories testing the same samples and comparing their results to ensure consistency and accuracy across different settings.

How to Perform Qualitative Trueness

1. Define Acceptance Criterias

In Cualia’s Trueness Experiment there are two Acceptance Criterias:

Min. Samples: The minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

💡

Cualia Trueness has a limit of max 10 samples. They should be of high EQA validity.

Cualia Recommendations: Verifications: 5-10 samples Validations/LDTs: 20-30 samples

Min. Agreement (%): The minimum percent of N results where the actual measurements are in agreement with their expected values.

Generally it is recommended to set Min. Agreement to 95% because it strikes a balance between clinical significance and statistical reliability in many statistical tests such as the Bland-Altman.

2. Sample Selection

For Trueness in Method Verification (MV), sample selection is crucial and should involve using gold standard samples that have been validated and agreed upon by multiple sources, such as External Quality Assessments (EQA) and Interlaboratory Comparisons (Interlab). These samples ensure that the method’s accuracy is assessed against the highest standards, providing a reliable benchmark for trueness evaluation.

3. Testing Samples

Measure each sample on a new instrumentation (actual). Record the results in comparison with the gold standard reference values, label and source either in the Cualia app or on your own platform. Include the label and the source.

4. Use the Cualia MV App

Make sure the data is entered into the experiment with the right acceptance criterias.

Preparation Checklist

Analyzer is set up in Cualia → AnalyzersAnalyzers
Tests are set up in the analyzer → Quantitative TestsQuantitative Tests and Qualitative TestsQualitative Tests
The initial MV details are prepared → MV Overview and DetailsMV Overview and Details

General Experiment Recommendations: Dos and Don’ts

Don’t: Enter private patient information, identifiers or data into Cualia.
Don’t: Rush Through an MV: Sometimes an MV can take months waiting for the right samples to come through. Based on our experience, the most time consuming part of an MV is finding out after days after measurements and painstakingly entering in the data to find that something was done incorrectly and the cycle must be repeated.
Don’t: Rely too much on controls, calibrators and spiked samples: The goal of an MV is not to pass inspection but to truly evaluate that your instrumentation’s performance is up to clinical standards. Using samples that reflect the lab’s testing population will offer the best insights into the evaluation
Do: Prepare your Cualia MV before taking measurements: Having a blueprint of work provides a smooth experience.
Do: Enter Results Directly into Cualia: Taking down results to enter them into a spreadsheet just to copy them into Cualia will increase sources of error. When a result is returned, enter it directly into Cualia. You will be able to immediately have feedback into the success state of the experiment, identifying any missing variables that will hinder your MV.
Do: Ask for clarification. Talk to regulators, auditor, consultants and don’t hesitate to reach out to support@cualia.io with questions.

Data Table

There are 5 relevant columns in the Qualitative Trueness Data Table:

Label | Free text - The general identification used to identify a specific Sample.

Source | Free text - The original source of the Sample such as Patient, EQA, or the name of the Manufacturer

Actual | Pos or Neg Label only - Represents the real-world value measured by the instrument being evaluated.

Reference | Pos or Neg Label only - The true value as measured from the reference method.

Result | Read Only - Indicates whether the measurements are within agreeable range.

image

Qualitative Trueness Results

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
Min. Samples

An Acceptance Criteria of the minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

Min. Agreement

The minimum percentage of agreement required between the test results and the reference results from the acceptance criteria for the experiment to pass.

Sample(s)

Individual specimens collected for testing representing the source. In MV experiments with Runs, this can refer to to the number of concentration levels used.

Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

Agreements

The total number of measurements between actual and reference value in agreement.

Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100

True Positive (TP)

The number of true positive results where the test correctly detected the presence of the substrate when compared to the reference method.

True Negative (TN)

The number of true negative results where the test correctly identified the absence of the substrate when compared to the reference method.

False Positive (FP)

The number of false positive results where the test incorrectly detected the presence of the substrate when compared to the reference method.

False Negative (FN)

The number of false negative results where the test incorrectly missed the presence of the substrate when compared to the reference method.

Sensitivity (Qualitative)

The ability of the test to correctly identify true positives. Sensitivity=TPTP+FN×100%\text{Sensitivity} = \frac{\text{TP}}{\text{TP} + \text{FN}} \times 100\%

Specificity (Qualitative)

The ability of the test to correctly identify true negatives. Specificity=TNTN+FP×100%\text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}} \times 100\%

Negative Predictive Value (NPV)

The proportion of negative test results that are true negatives. NPV=TNTN+FN×100%\text{NPV} = \frac{\text{TN}}{\text{TN} + \text{FN}} \times 100\%

Positive Predictive Value (PPV)

The probability that a Positive Result indicates the Expected analyte Result is a positive. PPV=TPTP+FPPPV = \frac{TP}{TP + FP}

Qualitative Trueness Acceptance

image

When both samples and agreement values satisfy the acceptance criterias, the experiment will be in a Passed state indicated with a green check.

Samples

Samples - The number of eligible samples tested. In the results bar, the color will turn green when the number of samples has satisfied the Min. Samples acceptance criteria to indicate it has passed.

Agreement

Agreement - The proportion (%) of test results that match the reference results. In the results bar, the color will be green and be considered passed when it satisfies the Min. Agreement or Min. Between-Day Agreement acceptance criteria.