QNT Trueness

QNT Trueness

Quantitative Trueness (TRN)

Overview

Other Names - Accuracy, Bias Test, Correctness Verification, Accuracy Assessment, Method Accuracy Evaluation, Bias Measurement, Correctness Validation, Accuracy Verification, Bias Analysis, Method Accuracy Check.

Trueness is one of the primary MV experiments. It involves comparing your instrument’s actual results with those an EQA qualified sample. The mechanism is similar to Method Comparison, but compares actual results with highly validated results for comparison. Few samples may be used for comparison due to stringent sample requirements.

Definitions

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
Actual

Actual represents the real-world value measured by the instrument being evaluated.

Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100 See: Agreements

Agreements

The total number of measurements between actual and reference value in agreement.

Bias

The difference in Results when comparing laboratory Quantitative methods with established methods.

Clinical Reportable Range

Range of analyte values that a method can measure, allowing for specimen dilution, concentration, or other pretreatment used to extend the direct analytical measurement range. The distinction between CRR and RR is that CRR may be up to the medical director's discretion of what range of results to report.

Eligible (Sample and Results)

Samples that provide the necessary information for statistical calculations to be performed. Usually this is possessing both an actual and expected result.

Error Allowance Type

The type of error (TEa, SEa or REa) that will be used for calculating Error Index agreement between two results.

Experiment

A sampling of results that are statistically analyzed and interpreted in the evaluation of testing performance.

Label

The general identification used to identify a specific Sample.

Laboratory Developed Test (LDT)

A method developed by the laboratory or a regulator approved/validated method that has been modified.

Method Validation

A systematic process to evaluate whether the performance of a medical Test meets quality goals to be used for medical testing.

Method Verification

A systematic process to evaluate whether the performance of a medical Test meets quality goals set by a validation. Performance evaluations may usually be found in a manufacturer insert.

Min. Samples

An Acceptance Criteria of the minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

MV

A broad term that can be used synonymously with Performance Evaluation, Method Validation and/or Method Verification.

Reference Method

Reference methods are well-established, highly accurate, and precise analytical procedures used as standards to evaluate the performance of other methods.

Reference Value

The value/result that represents the true value from a trusted reference source such as a verified instrument, EQA, or a commercial product.

Result(s)

A value or determination collected by measurement or calculation.

Sample Source

The original source of the Sample such as Sample, EQA, or the name of the Manufacturer.

Sample(s)

Individual specimens collected for testing representing the source. In MV experiments with Runs, this can refer to to the number of concentration levels used.

Trueness (TN) Experiment

Trueness is an MV Experiment that evaluates the performance of a test by comparing sample test results with true values. Usually an EQA.

What is the Trueness Experiment?

The Trueness Experiment assesses the accuracy of a measurement method, indicating how close the average of a series of test results is to the true value of the analyte. High trueness means that the method produces results that are consistently accurate and free from systematic errors or bias. This is crucial for ensuring the reliability and validity of diagnostic tests.

The Trueness Experiment involves comparing the test results from the method being verified against known reference values or a gold standard.

External Quality Assessments (EQAs) and Interlaboratory Comparisons (Interlab) are integral to evaluating trueness. In EQAs, samples are distributed to multiple laboratories to assess their performance and ensure accuracy. Interlab comparisons involve multiple laboratories testing the same samples and comparing their results to ensure consistency and accuracy across different settings.

Experiment Settings and Acceptance Criterias

Cualia Recommendations: Verifications: Quantitative 1-5 samples. Qualitative 1-5 samples. Validations/LDTs: 3-10 samples

Quantitative Trueness allows for 2 different modes of evaluation:

Error Allowance Mode

Use an error allowance to determine the agreement between measured and reference values. This is useful for evaluating an instrument with the same expected measurements as the reference. You may set the following minimum values for passing.

Min. Samples: The minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

💡

Cualia Trueness has a limit of max 10 samples. They should be of high EQA validity.

image

Min. Agreement (%): The minimum percent of N results where the actual measurements are in agreement with their expected values.

Generally it is recommended to set Min. Agreement to 95% because it strikes a balance between clinical significance and statistical reliability in many statistical tests such as the Bland-Altman.

Error Allowance Type - The type of error (TEa, SEa or REa) that will be used for calculating Error Index agreement between two results. The TEa error values may be set in the test’s settings.

⚠️

If TEa values have not been set in the test, only an exact match between two results will yield agreement.

Range Mode

Set a Min and Max acceptable range for each EQA sample. These will usually be the 2SD upper and lower thresholds on EQA reports. If the result is within this range the sample is in agreement.

Min. Samples: The minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

💡

Cualia Trueness has a limit of max 10 samples. They should be of high EQA validity.

Min. Agreement (%): The minimum percent of N results where the actual measurements are in agreement with their expected values.

Generally it is recommended to set Min. Agreement to 95% because it strikes a balance between clinical significance and statistical reliability in many statistical tests such as the Bland-Altman.

image

Sample Selection

Choose a representative set samples that reflect the diversity and variability expected in routine testing. Ensure samples cover the full range of analyte concentrations and conditions relevant to the test.

It is recommended that most of the samples come from the true population. Some manufacturer products such as commercial kits and controls may be used. These may often be necessary for particularly difficult to obtain concentrations and conditions.

Testing Samples

Measure each sample on a reference method (expected) and on your new instrumentation (actual). Record the results, label and source either in the Cualia app or on your own platform. Include the label and the source.

Use the Cualia MV App

Make sure the data is entered into the experiment with the right acceptance criterias.

Preparation Checklist

Analyzer is set up in Cualia → AnalyzersAnalyzers
Tests are set up in the analyzer → Quantitative TestsQuantitative Tests and Qualitative TestsQualitative Tests
The initial MV details are prepared → MV Overview and DetailsMV Overview and Details

General Experiment Recommendations: Dos and Don’ts

❌
Don’t: Enter private patient information, identifiers or data into Cualia.
❌
Don’t: Rush Through an MV: Sometimes an MV can take months waiting for the right samples to come through. Based on our experience, the most time consuming part of an MV is finding out after days after measurements and painstakingly entering in the data to find that something was done incorrectly and the cycle must be repeated.
❌
Don’t: Rely too much on controls, calibrators and spiked samples: The goal of an MV is not to pass inspection but to truly evaluate that your instrumentation’s performance is up to clinical standards. Using samples that reflect the lab’s testing population will offer the best insights into the evaluation
✅
Do: Prepare your Cualia MV before taking measurements: Having a blueprint of work provides a smooth experience.
✅
Do: Enter Results Directly into Cualia: Taking down results to enter them into a spreadsheet just to copy them into Cualia will increase sources of error. When a result is returned, enter it directly into Cualia. You will be able to immediately have feedback into the success state of the experiment, identifying any missing variables that will hinder your MV.
✅
Do: Ask for clarification. Talk to regulators, auditor, consultants and don’t hesitate to reach out to support@cualia.io with questions.

Data Table

image

Data Table Columns

Label | Free text - The general identification used to identify a specific Sample.

Source | Free text - The original source of the Sample such as Patient, EQA, or the name of the Manufacturer

Actual | Pos or Neg Label only - Represents the real-world value measured by the instrument being evaluated.

CV Mode Only

Reference | Pos or Neg Label only - The true value as measured from the reference method.

Result | Read Only - Indicates whether the measurements are within agreeable range.

Bias | Read Only - The bias column shows the Bias and the %Bias. These represent the difference between the actual value and the reference value for calculating agreement.

Error Index | Read Only - The Error Index (Ei) proportion of the Bias to the Error Allowance. An Error Index must be between -1 and 1 to be considered in agreement. The Ei column will determine if agreement exists and is color coded to indicate agreement.

Range Mode Only

Min - The cutoff value for the minimum value the measured result can be for it to be in agreement.

Max - The cutoff value for the maximum value the measured result can be for it to be in agreement.

Mean - The average between the min and max range.

Bias | Read Only - The bias column shows the Bias and the %Bias. These represent the difference between the actual value and the reference value for calculating agreement.

Calculations

Bias

Bias - The difference between the actual value and the reference value for calculating agreement.

Bias=Actual−Reference\text{Bias} = {\text{Actual} - \text{Reference}}

Bias %

Bias % - The % difference between the actual value and the reference value for calculating agreement.

%Bias=Actual−ReferenceReference×100\text{\%Bias} = \frac{\text{Actual} - \text{Reference}}{\text{Reference}} \times 100

Error Index (Ei)

Error Index (Ei) - Proportion of the Bias to the Error Allowance. An Error Index must be between -1 and 1 to be considered in agreement. The Ei column will determine if agreement exists and is color coded to indicate agreement.

⚠️

A TEa / Allowable Error value must be set for an Error Index to be calculated

ErrorIndex(Ei)=∣Actual−Reference∣TEaError Index (Ei) = \frac{|\text{Actual} - \text{Reference}|}{\text{TEa}}

Error Index shows on Error Allowance mode only.

Quantitative Trueness Results

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
Pages
Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

QNT TestsQL TestsMV OverviewQL MCQL TNQL TN ResultsQL MC ResultsQL PrecQL Prec ResultsQL LoDQL LoD ResultsQNT MCQNT MC ResultsQNT PrecQNT Prec ResultsQNT TNQNT TN Results
Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100 See: Agreements

QL MCQL TNQL TN ResultsQL MC ResultsQL PrecQL Prec ResultsQL LoDQL LoD ResultsQNT MCQNT MC ResultsQNT PrecQNT Prec ResultsQNT TNQNT TN Results
Agreements

The total number of measurements between actual and reference value in agreement.

QL TNQNT MCQL MCQL TestsQNT TestsQL TN ResultsQL MC ResultsQL PrecQL Prec ResultsQL LoDQL LoD ResultsQNT MC ResultsQNT PrecQNT Prec ResultsWorking with ExperimentsQNT TNQNT TN Results
Mean Error Index (MEi)

Mean of all the Error Indexes calculated from a group of samples. A large distance from 0 can indicate a shift or Systematic error between the Actual and Reference result values.

QNT MC ResultsQNT TN Results
Min. Agreement

The minimum percentage of agreement required between the test results and the reference results from the acceptance criteria for the experiment to pass.

QL TNQL TN ResultsQL MC ResultsQNT MC ResultsQNT TN Results
Min. Samples

An Acceptance Criteria of the minimum number of eligible samples required in order for the evaluation’s requirements to be fulfilled.

QL MCQL TNQL TN ResultsQL MC ResultsQNT MCQNT MC ResultsQNT TNQNT TN Results
Random Allowable Error (REa)

The maximum amount of Random Error allowed for two results to be in agreement

QNT TestsQNT MC ResultsQNT PrecQNT Prec ResultsQNT TN Results
Range Verified

Range of values from lowest to highest that was measured within a group of samples.

QNT MC ResultsQNT TN ResultsQNT LINQNT LIN Results
Sample(s)

Individual specimens collected for testing representing the source. In MV experiments with Runs, this can refer to to the number of concentration levels used.

QNT TestsQL TestsWorking with ExperimentsQL TNQNT MCQL TN ResultsQL MCQL MC ResultsQL PrecQNT PrecQNT TNQNT TN ResultsQNT LoBQNT LoQQNT RIQNT LINQL LoDQNT Carryover
Standard Deviation (σ or StdDev)

Statistical measure of the amount of variation from the mean. σ=1n−1∑i=1n(xi−xˉ)2\sigma = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2} σ = Standard Deviation n = Sample Size xix_i = Individual data point xˉ\bar{x} = Mean

QNT MC ResultsQNT Prec ResultsQNT TN ResultsQNT LoQQNT LoQ ResultsQNT CarryoverQNT Carry Calc
Systematic Error Allowance (SEa)

Acceptable shift across the detectable range of a test in a single direction.

QNT TestsQNT MC ResultsQNT Prec ResultsQNT TN Results

Quantitative Trueness Acceptance

image

When both samples and agreement satisfy the acceptance criterias, the experiment will be in a Passed state indicated with a green check.

Samples

Samples - The number of eligible samples tested. In the results bar, the color will turn green when the number of samples has satisfied the Min. Samples acceptance criteria to indicate it has passed.

Agreement

Agreement - The proportion (%) of test results that match the reference results. In the results bar, the color will be green and be considered passed when it satisfies the Min. Agreement or Min. Between-Day Agreement acceptance criteria.