QL Precision

QL Precision

Qualitative Precision

Overview

Other Names - Repeatability Test, Reproducibility Test, Consistency Test, Variability Test, Precision Study, Repeatability Study, Reproducibility Study, Intra-assay Precision, Inter-assay Precision, Consistency Measurement.

Precision refers to the consistency of test results when the same test is repeated under identical conditions. It measures the reproducibility of categorical outcomes, ensuring the method produces the same results consistently across different runs and operators. Verified precision is essential for the reliability of diagnostic tests.

Lab Juice

Definitions

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
5 Day Test

A 5 Day Test is a standardized procedure used for method validation/verification and experiments in clinical and medical laboratories. It involves performing 5 replicates/results in each of the 5 runs conducted over a course of 5 days to ensure precision and accuracy of the method.

Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100 See: Agreements

Agreements

The total number of measurements between actual and reference value in agreement.

Between Day/Run

Comparison of Run replicate measurement(s) that occur on different runs/days. Other Names → Inter-Assay

Days (MV Experiments)

Days refers to the number of distinct days on which a test is repeated to assess the consistency of results over time.

Eligible (Sample and Results)

Samples that provide the necessary information for statistical calculations to be performed. Usually this is possessing both an actual and expected result.

Experiment

A sampling of results that are statistically analyzed and interpreted in the evaluation of testing performance.

Level(s)

A reference Level within a range to determine performance around that value.

Method Verification

A systematic process to evaluate whether the performance of a medical Test meets quality goals set by a validation. Performance evaluations may usually be found in a manufacturer insert.

Qualitative

A Result Type for a Test that can only have two possible options

Qualitative Agreement

In Qualitative testing a matching pair of Actual and Reference Results are considered in Agreement

Reference Value

The value/result that represents the true value from a trusted reference source such as a verified instrument, EQA, or a commercial product.

Replicate

Multiple tests on the same sample to assess precision and repeatability, conducted within a run or across multiple runs.

Result(s)

A value or determination collected by measurement or calculation.

Results per Day (MV Experiments)

In MV experiments, the Results Per Day is the number of replicate measurements that will be taken per day.

Run

A sequence of measurements on a set of samples under the same conditions, performed within a defined time period.

Sample(s)

Individual specimens collected for testing representing the source. In MV experiments with Runs, this can refer to to the number of concentration levels used.

Within Day/Run

Combined Results across Runs of the same Level on the same run, day or time period. Oftentimes a run load volume can be instrument specific and “Within Day” is commonly used. Other Names → Intra-Assay

What is the Precision Experiment?

The precision experiment in Method Verification (MV) assesses the consistency and repeatability of a test method. It involves repeatedly testing the same sample under identical conditions to determine the variability in the results. Precision experiments can be conducted within a single run (Within Day Precision) or across multiple runs (Between Day Precision), providing insights into the method's reliability over time and different conditions. The goal is to ensure that the test produces stable and consistent results, minimizing random errors.

High precision is crucial for the reliability of any diagnostic test, as it ensures that repeated measurements yield similar results, thereby enhancing confidence in the test method. This experiment typically involves calculating the standard deviation or coefficient of variation (CV) of the repeated measurements. By demonstrating low variability, the precision experiment confirms that the test method is robust and dependable for clinical or diagnostic use.

The experimental design for Precision involves repeatedly testing multiple samples under consistent conditions to assess the consistency of categorical outcomes, such as positive or negative results, over time and across different operations.

How to Perform Qualitative Precision Experiments

1. Define Acceptance Criterias

Samples

In Cualia’s Qualitative Precision Experiment the experimental design should be considered first with the number of Samples. Each sample should be of a different concentration to assess the precision at different levels.

It is recommended to use at least 3 samples: a blank, a low concentration and a high concentration. 2 samples may be used if the assessor confirms acceptability.

Days and Results per Day

The number of days and measurements per day that will be performed for each sample/level. It is best to use a number of days and results per day that will effectively evaluate the precision of your instrumentation.

image

Min. Between Day Agreement (%)

The Min. Between Day Agreement % is the Agreement % threshold for each sample across all of its days and replicates for the sample to be considered passed.

The 95% agreement threshold is widely recommended in MV experiments to ensure that the new method reliably matches the reference method. This level of agreement is chosen because it strikes a balance between clinical significance and statistical reliability.

Specifically, the Bland-Altman method, which is a commonly used approach for assessing agreement, calculates the 95% limits of agreement as the mean difference plus and minus 1.96 times the standard deviation of the differences. This approach ensures that 95% of the differences between the new method and the reference method fall within these limits, providing a high degree of confidence that the two methods produce comparable results in most cases.

The Standard 5x5 Precision Experiment

Oftentimes, the Standard 5x5 approach is the way to go. This follows a 3x5x5 format of 75 replicates. It is a standard design used in method validation to assess the precision of a measurement method. This design involves three distinct samples at different concentrations, each performed on five different days, with five replicates tested in each run.

The Standard 5x5 is the default setting in Cualia MVs.

2. Sample Selection

In general practice is it often acceptable to use controls as samples. However this should be discussed with the assessor to confirm.

💡
You can save on resources by using the same sample/results from the LoD Experiment as your low control.

3. Testing Samples

Measure each sample on a reference method (reference) and on your new instrumentation (actual) the appropriate number of times.

For example if running a 3x4x5 experiment 3 samples will be run on the MV test 5 times on 4 separate days. It is recommended to make variations in time of day and operator to ensure robustness and rule out operator biases.

4. Use the Cualia MV App

Make sure the data is entered into the experiment with the right acceptance criterias.

Reference Methods

Reference Methods are well-established, highly accurate, and precise analytical procedures used as standards to evaluate the performance of other methods.

For testing you will need to select a Reference Method to act as your reference result to match with the actual results of the new test.

It is recommended to use a validated and verified source of the same model/make as the analyzer being tested to ensure compatibility of the measurements.

Qualitative Precision Structure

  1. Samples Tabs - Navigate between samples/levels from the Samples Tabs. These will show different tables for each of the samples.
  1. Sample Label - The Sample Label field allows for given the sample/level an identifier.
  1. Reference Value - Expected value from the reference. In qualitative tests this value should be the same as the actual measurements.
  1. Data Table - Area where measurement values may be entered. See the Data Table section below for more information.
  1. Sample Results - Shows the label, reference value, the number of results and the agreement % for the sample/level. The numbers will turn green when the acceptance criterias are satisfied.
  1. Experiment Results - Overall results of the experiment including the number of samples with measurements, results and the agreement % across all results.
image

Data Table

There are 4 relevant columns types in the Qualitative Precision Data Table:

Run | Free Text - This column identifies the run that the replicate group was measured. It represents Within-Day/Run combined results. The default value is the Day number.

Replicate 1 to X | Pos or Neg Label only - The measured value of the replicate.

Result | Read Only - Indicates whether the Within-Day results are within agreeable range.

(blank) - No result. The sample is likely to be ineligible for calculation because it is missing an actual and/or reference value.

(✓) → The individual result is out of agreement range and/or the run is not within agreeable parameters

(✓) → The individual result is out of agreement range however the run/day is still within agreeable parameters

(✓) → The individual result is within agreeable parameters (pass)

Qualitative Precision Results

🗣️
**Cualia Support Docs Definitions (Public)**

Name
Definition
Result(s)

A value or determination collected by measurement or calculation.

Agreement

Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.

Agreements

The total number of measurements between actual and reference value in agreement.

Agreement Percent

The percentage (%) of paired Results within Agreement amongst the number of Samples measured. Agreement %=(Results in AgreementTotal Number of Comparisons)×100\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100 See: Agreements

Qualitative Precision Acceptance

Sample Results

image

Results

Results - The total number of measurements collected. This value will be yellow if it doesn’t satisfy the requirements of the acceptance criteria and green when it passes.

Agreement

Agreement - The proportion (%) of test results that match the reference results. In the results bar, the color will be green and be considered passed when it satisfies the Min. Agreement or Min. Between-Day Agreement acceptance criteria.

Experiment Results

image

When all samples and their agreement values satisfy the acceptance criterias, the experiment will be in a Passed state indicated with a green check.

Samples

Samples - The number of eligible samples tested. In the results bar, the color will turn green when the number of samples has satisfied the Min. Samples acceptance criteria to indicate it has passed.

Results

Results - The total number of measurements collected. This value will be yellow if it doesn’t satisfy the requirements of the acceptance criteria and green when it passes.

Agreement

Agreement - The proportion (%) of test results that match the reference results. In the results bar, the color will be green and be considered passed when it satisfies the Min. Agreement or Min. Between-Day Agreement acceptance criteria.

Sources

Cualia Support Docs Citations

Source
Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 327(8476), 307-310.
Clinical and Laboratory Standards Institute (CLSI). (2016). EP05-A3: Evaluation of Precision of Quantitative Measurement Procedures; Approved Guideline—Third Edition. Clinical and Laboratory Standards Institute.
International Organization for Standardization (ISO). (2015). ISO 5725-2:1994 Accuracy (trueness and precision) of measurement methods and results — Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method. International Organization for Standardization. Retrieved from https://www.iso.org/standard/11833.html
International Organization for Standardization (ISO). (2015). ISO 15189:2012 Medical laboratories — Requirements for quality and competence. International Organization for Standardization.
Sisu.ut.ee. (n.d.). Validation of liquid chromatography mass spectrometry (LC-MS) methods. Retrieved from https://sisu.ut.ee/lcms_method_validation/calculation-precision