Quality Assurance Tools
Mathmet has developed a draft version of Quality Assurance Tools to ensure that research outputs in the forms of data, software and guidelines are fit-for-purpose, achieve a sufficient level of quality, and are consistent with the aims of National Measurement Institutes & Designated Institutes to provide quality-assured and trusted outputs.
The Quality Assurance Tools for software, data and guidelines are available for download below.
For research outputs in the form of data and software, the tools are structured according to iterative life cycles for developing software and builds upon existing good practices for data management and software development, following the process-based approach of ISO 9001 and ISO 8000.
For research outputs in the form of guidelines, a distinction is made between existing guidelines and future guidelines. For existing guidelines, the Quality Assurance Tools are applied to provide a statement of relevance rather than a recommendation about their fitness-for-purpose. A focus is given to their provenance bearing in mind that ISO Standards and BIPM Guides, for example, can be expected to be thoroughly reviewed.
Collaboration between Mathmet partners on applying the Quality Assurance Tools to various cases gave useful insight into assuring the quality of data, software, and guidelines. A pragmatic approach has been taken to the design of the Quality Assurance Tools. Where practicable, the tools provide a high-level framework into which the processes in place and adopted by individual partners can comfortably fit.
Components of the Quality Assurance Tools
The Quality Assurance Tools consist of separate components for data, software, and guidelines. For data and software an online interactive risk assessment tool guides the user in developing a quality management plan. For guidelines (future and existing), the Quality Assurance Tools involve completing an interactive checklist comprising a set of questions, and making a recommendation based on the answers to those questions.
The components are summarised below:
- A quality management plan is key to the Quality Assurance Tools' components for data and software. This plan lists the quality management activities needed for a particular dataset or piece of software. These activities follow a typical life cycle from requirements capture to design and development, verification, and validation through to release and maintenance. Review is an important activity that is carried out throughout the life cycle.
- Quality management plans are generated using an online, interactive risk assessment tool. Risk is quantified using a value called an integrity level. The integrity level is a number between 1 and 4, where 1 indicates the lowest level of risk (e.g., prototypes of software for internal use within an organisation) and 4 indicates the highest level (e.g., software that is safety critical). The integrity level is used to decide quality management activities, i.e., the activities listed on the plan, to be undertaken. Mathmet provides an online risk assessment tool to guide the user through the process of calculating an integrity level and generating a quality management plan.
- The concept is analogous to, but should not be confused with, the 'safety integrity level' of IEC 61508 'Functional safety of electrical/electronic/programmable electronic safety related systems'. This value is also a number between 1 and 4, where 1 specifies the lowest level of functional safety and 4 the highest.
- For software, the Quality Assurance Tools can include established quality procedures and templates from the Mathmet members. However, the activities listed in the quality management plan must be carried out.
- Quality management of data is less well established than quality management of software. Accordingly, developing this component of the Quality Assurance Tools was arguably a research activity to some extent. A key concept is that integrity levels are adapted to apply to data as well as software.
As with software, an online risk assessment tool will assist with assigning integrity levels and generating quality management plans. - The aim of the guidelines component of the Quality Assurance Tools is to ensure a sufficient level of quality in the development, assessment, and recommendation of existing and future guidelines for mathematics and statistics in metrology.
A process has been developed to provide advice and feedback that will enable third parties to adapt and improve mathematical, and statistical, guidelines to the needs and requirements of the European metrology community and its stakeholders.
Integration of the Quality Assurance Tools into research data management
Data quality is an overarching theme, which aims to be harmonised within broader recommendations for the management of data from research and metrological services. For this purpose Mathmet is collaborating with the EURAMET project TC-IM 1449 "Research data management in European metrology" to incorporate some aspects of the Mathmet QATs within their guidelines. In particular:
- Aspects of data quality will be included within 1449's Guide on Research data management in Metrology,
- As well as the PDF versions, downloadable from this website, the QAT data quality assurance plan and software quality assurance plan generators will also be implemented in the metrology research data management organiser (MetRDMO) tool
- The RDMO version of the data quality assurance plan generator will provide structured, re-usable, machine-actionable data quality evaluation. It will also strengthen the integration of quality aspects in data documentation procedures.
- The RDMO version of the software quality assurance plan generator will provide structured, re-usable, machine-actionable software quality evaluation.
Training
The online training course for Mathmet's Quality Assurance Tools is now available! It is accessible in EURAMET's section on the BIPM e-learning platform:
Quality Assurance Tools for Software, Data and Guidelines EMN Mathmet >>
The Quality Assurance Tools for data, software and guidelines has been provided by the Members and Partners of the European Metrology Network for Mathematics and Statistics (Mathmet). EURAMET has no influence on its correctness and completeness and does not assume any liability for it.
The European Metrology Network for Mathematics and Statistics is supported by the Joint Network Project ‘Support for a European Metrology Network for mathematics and statistics’ (18NET05).