Sections

The Organization

MVs

MV Overview and Details

Experiment Design

Working with Experiments

Qualitative Experiments

QL Precision

QL Limit of Detection

QL Method Comparison

QL Trueness

Quantitative Experiments

QNT Precision

QNT Limit of Blank

QNT Limit of Quantitation

QNT Linearity

QNT Method Comparison

QNT Reference Interval

QNT Carryover

QNT Trueness

Settings

General Settings

Quantitative Precision

Overview

Other Names - Repeatability Test, Reproducibility Test, Consistency Test, Variability Test, Precision Study, Repeatability Study, Reproducibility Study, Intra-assay Precision, Inter-assay Precision, Consistency Measurement.

Precision refers to the consistency of test results when the same test is repeated under identical conditions. It measures the reproducibility of categorical outcomes, ensuring the method produces the same results consistently across different runs and operators. Verified precision is essential for the reliability of diagnostic tests.

Definitions

Name	Definition
5 Day Test	A 5 Day Test is a standardized procedure used for method validation/verification and experiments in clinical and medical laboratories. It involves performing 5 replicates/results in each of the 5 runs conducted over a course of 5 days to ensure precision and accuracy of the method.
Agreement	Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.
Agreement Percent	The percentage (%) of paired Results within Agreement amongst the number of Samples measured. $\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100$ See: Agreements
Agreements	The total number of measurements between actual and reference value in agreement.
Average/Mean Error Index	The Mean of the Error Index's within a collection.
Between Day/Run	Comparison of Run replicate measurement(s) that occur on different runs/days. Other Names → Inter-Assay
Blank	A sample of similar matrix to method's expected sample with none of the analyte present. Oftentimes this can be a zero standard from a calibrator.
Coefficient of Variation (CV)	Ratio of the Standard Deviation of the Mean, often expressed as a percentage. See: CV%. It provides a relative comparison of the Standard Deviation independent of the size of the value. $\text{CV} = \frac{\sigma}{\mu}$ $\sigma$ = Standard Seviation ${\mu}$ = Mean
Coefficient of Variation % (CV%)	Coefficient of variation expressed as a percent
Days (MV Experiments)	Days refers to the number of distinct days on which a test is repeated to assess the consistency of results over time.
Eligible (Sample and Results)	Samples that provide the necessary information for statistical calculations to be performed. Usually this is possessing both an actual and expected result.
Error Index (Ei)	The proportion of the Bias to the Error Allowance. An Error Index must be between -1 and 1 to be considered in agreement.
Experiment	A sampling of results that are statistically analyzed and interpreted in the evaluation of testing performance.
Label	The general identification used to identify a specific Sample.
Level(s)	A reference Level within a range to determine performance around that value.
Method Validation	A systematic process to evaluate whether the performance of a medical Test meets quality goals to be used for medical testing.
Method Verification	A systematic process to evaluate whether the performance of a medical Test meets quality goals set by a validation. Performance evaluations may usually be found in a manufacturer insert.
Random Allowable Error (REa)	The maximum amount of Random Error allowed for two results to be in agreement
Random Error	An error in measurement caused by factors which vary from one measurement to another.
Reference Value	The value/result that represents the true value from a trusted reference source such as a verified instrument, EQA, or a commercial product.
Replicate	Multiple tests on the same sample to assess precision and repeatability, conducted within a run or across multiple runs.
Result(s)	A value or determination collected by measurement or calculation.
Results per Day (MV Experiments)	In MV experiments, the Results Per Day is the number of replicate measurements that will be taken per day.
Run	A sequence of measurements on a set of samples under the same conditions, performed within a defined time period.
Sample(s)	Individual specimens collected for testing representing the source. In MV experiments with Runs, this can refer to to the number of concentration levels used.
Systematic Error	A consistent, predictable error that skews results in a specific direction. Systematic errors affect the accuracy of the test results, leading to a consistent deviation from the true value.
Total Allowable Error (TEa)	The set limit of combined REa and SEa tolerable for a single result. It represents the error range around an Expected result that can be considered in Agreement to an Actual result. TEa = REa + SEa
Within Day/Run	Combined Results across Runs of the same Level on the same run, day or time period. Oftentimes a run load volume can be instrument specific and “Within Day” is commonly used. Other Names → Intra-Assay

What is the Precision Experiment?

The precision experiment in Method Verification (MV) assesses the consistency and repeatability of a test method. It involves repeatedly testing the same sample under identical conditions to determine the variability in the results. Precision experiments can be conducted within a single run (Within Day Precision) or across multiple runs (Between Day Precision), providing insights into the method's reliability over time and different conditions. The goal is to ensure that the test produces stable and consistent results, minimizing random errors.

High precision is crucial for the reliability of any diagnostic test, as it ensures that repeated measurements yield similar results, thereby enhancing confidence in the test method. This experiment typically involves calculating the standard deviation or coefficient of variation (CV) of the repeated measurements. By demonstrating low variability, the precision experiment confirms that the test method is robust and dependable for clinical or diagnostic use.

The experimental design for Precision involves repeatedly testing multiple samples under consistent conditions to assess the consistency of categorical outcomes, such as positive or negative results, over time and across different operations.

Experiment Settings and Acceptance Criterias

1. Define Acceptance Criterias

Setting TEa or CV Mode

Quantitative Precision testing may be set to use Error Allowance settings from TEa or by calculating a CV.

In TEa mode, the TEa’s from the test settings are used for calculating agreement between Actual and Reference values. Under this setting the SEa or REa may be selected.

In CV mode, the Coefficient of Variation is calculated from Between-Day results and compared with the maximum CV% allowable.

https://images.spr.so/cdn-cgi/imagedelivery/j42No7y-dcokJuNgXeA0ig/9ccc3279-e2d6-4f08-89af-39d88664efa9/2024-10-23_163154_Microsoft_Edge/w=1080,quality=90,fit=scale-down

Samples

In Cualia’s Precision Experiment the experimental design should be considered first with the number of Samples. Each sample should be of a different concentration to assess the precision at different levels.

It is recommended to use at least 3 samples: a blank, a low concentration and a high concentration. 2 samples may be used if the assessor confirms acceptability.

Days and Results/Replicates per Day

The number of days and measurements per day that will be performed for each sample/level. It is best to use a number of days and results per day that will effectively evaluate the precision of your instrumentation.

https://images.spr.so/cdn-cgi/imagedelivery/j42No7y-dcokJuNgXeA0ig/6d314b12-61a6-4d9d-8c5e-c25ba3c3af4b/2024-10-23_162741_Microsoft_Edge/w=1080,quality=90,fit=scale-down

Min. Between Day Agreement (%)

The Min. Between Day Agreement % is the Agreement % threshold for each sample across all of its days and replicates for the sample to be considered passed.

The 95% agreement threshold is widely recommended in MV experiments to ensure that the new method reliably matches the reference method. This level of agreement is chosen because it strikes a balance between clinical significance and statistical reliability.

Specifically, the Bland-Altman method, which is a commonly used approach for assessing agreement, calculates the 95% limits of agreement as the mean difference plus and minus 1.96 times the standard deviation of the differences. This approach ensures that 95% of the differences between the new method and the reference method fall within these limits, providing a high degree of confidence that the two methods produce comparable results in most cases.

The Standard 5x5 Precision Experiment

Oftentimes, the Standard 5x5 approach is the way to go. This follows a 3x5x5 format of 75 replicates. It is a standard design used in method validation to assess the precision of a measurement method. This design involves three distinct samples at different concentrations, each performed on five different days, with five replicates tested in each run.

The Standard 5x5 is the default setting in Cualia MVs.

2. Sample Selection

In general practice is it often acceptable to use controls as samples. However this should be discussed with the assessor to confirm.

💡

MV Tip: You can save on resources by using the same sample/results from the LoD Experiment as your low control.

3. Testing Samples

Measure each sample on a reference method (reference) and on your new instrumentation (actual) the appropriate number of times.

For example if running a 3x4x5 experiment 3 samples will be run on the MV test 5 times on 4 separate days. It is recommended to make variations in time of day and operator to ensure robustness and rule out operator biases.

4. Use the Cualia MV App

Make sure the data is entered into the experiment with the right acceptance criterias.

Reference Methods and Value

Reference Methods are well-established, highly accurate, and precise analytical procedures used as standards to evaluate the performance of other methods.

For testing you will need to select a Reference Method to act as your reference result to match with the actual results of the new test.

It is recommended to use a validated and verified source of the same model/make as the analyzer being tested to ensure compatibility of the measurements.

Quantitative Precision Structure

Samples Tabs - Navigate between samples/levels from the Samples Tabs. These will show different tables for each of the samples.

Sample Label - The Sample Label field allows for given the sample/level an identifier.

Reference Value - Expected value from the reference. In qualitative tests this value should be the same as the actual measurements.

Data Table - Area where measurement values may be entered. See the Data Table section below for more information.

Sample Results - Shows the label, reference value, the number of results and the agreement % for the sample/level. The numbers will turn green when the acceptance criterias are satisfied.

Experiment Results - Overall results of the experiment including the number of samples with measurements, results and the agreement % across all results.

https://images.spr.so/cdn-cgi/imagedelivery/j42No7y-dcokJuNgXeA0ig/296cd8d8-d2cb-4131-9591-2cf788770f27/2024-10-24_093620_Microsoft_Edge/w=1080,quality=90,fit=scale-down

Data Table

Run | Free Text - This column identifies the run that the replicate group was measured. It represents Within-Day/Run combined results. The default value is the Day number.

Replicate(R) 1 to X | Number only - The measured value of the replicate. A color may indicate whether the result is within agreeable parameters.

(blank) - No result. The sample is likely to be ineligible for calculation because it is missing an actual and/or reference value.

(✓) → The individual result is out of agreement range and/or the run is not within agreeable parameters

(✓) → The individual result is out of agreement range however the run/day is still within agreeable parameters

(✓) → The individual result is within agreeable parameters (pass)

Within Run | Read only - Shows the number of results within agreeable parameters and the total number of samples present within the run. A color will indicate whether the result is within agreeable parameters.

(blank) - No result. The sample is likely to be ineligible for calculation because it is missing an actual and/or reference value.

(✓) → The individual result is out of agreement range and/or the run is not within agreeable parameters

(✓) → The individual result is out of agreement range however the run/day is still within agreeable parameters

(✓) → The individual result is within agreeable parameters (pass)

Calculations

TEa mode

In TEa mode, agreement is determined through calculation between the Actual and Reference result based on the experiment setting (TEa, SEa or REa).

The bias and %bias are calculated and used with the TEa to determine the Error Index value. If the Error Index value between -1 and 1, the Actual and Reference result are considered in agreement.

There are two TEa values, one in units and one in a percent. Both TEas will have the Error Index calculation performed and the lowest one will be used for determining experiment agreement.

Bias

Bias - The difference between the actual value and the reference value for calculating agreement.

\text{Bias} = {\text{Actual} - \text{Reference}}

Bias %

Bias % - The % difference between the actual value and the reference value for calculating agreement.

\text{\%Bias} = \frac{\text{Actual} - \text{Reference}}{\text{Reference}} \times 100

Error Index (Ei)

Error Index (Ei) - Proportion of the Bias to the Error Allowance. An Error Index must be between -1 and 1 to be considered in agreement. The Ei column will determine if agreement exists and is color coded to indicate agreement.

⚠️

A TEa / Allowable Error value must be set for an Error Index to be calculated

Error Index (Ei) = \frac{|\text{Actual} - \text{Reference}|}{\text{TEa}}

CV Mode

In CV mode, the CV% for within-run and between-run must be less than the CV% set for the sample level. The passing of the experiment is based on the CV% of the between-run results.

⚠️

Remember to set the expected CV % max for each sample level or else only an exact match will be in agreement. The default value is 0.

Coefficient of Variation (CV)

Coefficient of Variation (CV) - The ratio of the Standard Deviation of the Mean, often expressed as a percentage. It provides a relative comparison of the Standard Deviation independent of the size of the value.

\text{CV} = \frac{\sigma}{\mu}

${\sigma}$ = standard deviation

${\mu}$ = mean

\text{CV\%} = \text{CV} \times 100

Standard Deviation (σ or StdDev)

Standard Deviation (σ or StdDev) - The statistical measure of the amount of variation from the mean.

\sigma = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2}

σ = Standard Deviation

n = Sample Size

$x_i$ = Individual data point

$\bar{x}$ = Mean

Results

Name	Definition
Result(s)	A value or determination collected by measurement or calculation.
Agreement	Whether two values, usually the actual and reference values acceptably agreeable, based on testing requirements and/or Error Allowance.
Agreements	The total number of measurements between actual and reference value in agreement.
Agreement Percent	The percentage (%) of paired Results within Agreement amongst the number of Samples measured. $\text{Agreement \%} = \left( \frac{\text{Results in Agreement}}{\text{Total Number of Comparisons}} \right) \times 100$ See: Agreements
Error Index (Ei)	The proportion of the Bias to the Error Allowance. An Error Index must be between -1 and 1 to be considered in agreement.
Random Allowable Error (REa)	The maximum amount of Random Error allowed for two results to be in agreement
Standard Deviation (σ or StdDev)	Statistical measure of the amount of variation from the mean. $\sigma = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2}$ σ = Standard Deviation n = Sample Size $x_i$ = Individual data point $\bar{x}$ = Mean
Coefficient of Variation (CV)	Ratio of the Standard Deviation of the Mean, often expressed as a percentage. See: CV%. It provides a relative comparison of the Standard Deviation independent of the size of the value. $\text{CV} = \frac{\sigma}{\mu}$ $\sigma$ = Standard Seviation ${\mu}$ = Mean
Total Allowable Error (TEa)	The set limit of combined REa and SEa tolerable for a single result. It represents the error range around an Expected result that can be considered in Agreement to an Actual result. TEa = REa + SEa
Coefficient of Variation % (CV%)	Coefficient of variation expressed as a percent
Systematic Error Allowance (SEa)	Acceptable shift across the detectable range of a test in a single direction.

Experiment Pass / Fail

Sample Results

Error Index (Ei)

Error Index (Ei) - Proportion of the Bias to the Error Allowance. An Error Index must be between -1 and 1 to be considered in agreement. The Ei column will determine if agreement exists and is color coded to indicate agreement.

⚠️

A TEa / Allowable Error value must be set for an Error Index to be calculated

Error Index is only shown Error Allowance Mode.

Mean

Mean - The average, calculated by summing all measured values and dividing by the total number of values.

Coefficient of Variation (CV)

Coefficient of Variation (CV) - The ratio of the Standard Deviation of the Mean, often expressed as a percentage. It provides a relative comparison of the Standard Deviation independent of the size of the value.

Results

Results - The total number of measurements collected. This value will be yellow if it doesn’t satisfy the requirements of the acceptance criteria and green when it passes.

Agreement

Agreement - The proportion (%) of test results that match the reference results. In the results bar, the color will be green and be considered passed when it satisfies the Min. Agreement or Min. Between-Day Agreement acceptance criteria.

Experiment Results

When all samples and their agreement values satisfy the acceptance criterias, the experiment will be in a Passed state indicated with a green check.