Ensuring the Success of AI: The Importance of Maintaining Quality and the Key Attributes of AI Engineering
“I’m pretty sure AI stands for ‘Almost Inevitable’ because whether we like it or not, it’s coming for us. So, let’s ensure it’s high quality, or at least has a good sense of humor.”
As the Director of Technology at Speire, I have had the opportunity to witness firsthand the incredible transformation that AI-enabled systems have brought to businesses. One such example that stands out in my mind is a healthcare company that was struggling with the manual analysis of patient data. This led to missed diagnoses and delayed treatments, resulting in negative patient outcomes and increased healthcare costs.
By introducing AI into its data analysis process, the company was able to streamline its workflow and optimize its data analysis procedures. The AI system was able to quickly analyze large amounts of patient data and identify patterns that human analysts may have overlooked. This resulted in improved patient outcomes, as doctors could make more informed decisions based on the insights provided by the AI system. Additionally, the company reduced healthcare costs, as the AI system allowed them to optimize their operations and allocate their resources more efficiently.
However, the success of AI-enabled systems depends heavily on the quality of the AI models used. In fact, I vividly recall an incident from four years ago, where we were assisting a financial startup in implementing an AI-powered vulnerability scoring system. Unfortunately, due to differences in the training data and the distribution of data post-deployment in production, the AI system produced numerous false positives. This resulted in significant financial losses for the startup and customer dissatisfaction, ultimately leading to the AI model being taken down.
As such, it is crucial for businesses to prioritize the quality of their AI models and the processes used to create them. Subpar AI models can have devastating consequences, leading to inaccurate predictions, unreliable recommendations, and a decrease in overall business value. That’s why I always emphasize the importance of meeting strict quality standards at every stage of the AI product lifecycle, from data collection and processing to model training and deployment. By doing so, businesses can ensure that their AI-enabled systems operate at peak performance, resulting in numerous benefits, such as increased efficiency, cost savings, and, ultimately, business success.
Importance of maintaining AI Quality
As someone who has designed, developed, and worked with AI-enabled systems, I understand the significance of maintaining high AI quality in such systems. Failure to adhere to quality boundaries and requirements in developing AI-enabled systems can lead to inaccurate predictions, and unreliable recommendations, finally leading to declining business value. One true-life example is Microsoft’s AI-powered chatbot, Tay, launched on March 23, 2016, which was designed to learn and respond to Twitter users’ conversations. Unfortunately, within 24 hours of launch, Tay began spewing racist and sexist messages, causing a public relations crisis for Microsoft. This incident underlines the importance of maintaining quality in AI-enabled systems and the potential repercussions of neglecting it.
To prevent such catastrophic outcomes, it is crucial to ensure that AI-enabled systems satisfy specific quality attributes throughout their lifecycle. These quality attributes include security, privacy, data centricity, sustainability, and explainability, which are essential in developing reliable and robust software across the four dimensions of AI engineering — robust systems, data, human-machine interaction, and models. As per the study by the Software Engineering Institute at Carnegie Mellon University, these quality attributes are critical in creating trustworthy software [1].
Quality Attributes of AI Engineering
AI Quality attributes such as security, privacy, data centricity, sustainability, and explainability are crucial for the development and operation of AI systems. Ensuring sustainability involves balancing the need for improvement with maintaining safety and reliability over time, even as technology or data changes. Security is a significant challenge for AI-enabled systems, with the learning element of some AI algorithms providing a vector for attackers to change the output of a system. Privacy involves giving individuals control over the collection, use, and disclosure of their information, and AI systems must be designed with privacy boundaries in mind. Explainability is fundamental to the trustworthiness of an AI system, allowing users to understand how an algorithm produces a result. Data centricity is critical for producing accurate and reliable results and promoting ethical and responsible data use. Most AI systems rely on machine learning models trained on data, which can degrade system performance if changed, making sustainability testing necessary. Overall, incorporating these quality attributes can help mitigate risks and promote the long-term success of AI systems. The remainder of the article provides a concise yet thorough explanation of the quality attributes of AI engineering.
1.Sustainability requires considering the rate of change, data uncertainty, and scalability in the design of AI systems. In the case of AI-enabled Software as a Medical Device (SaMD), sustainability involves balancing the need to improve the system’s quality with maintaining its safety for use in a clinical setting. Retraining models in AI/ML systems can cause disruption, and determining the pace of updates is a challenge. While continuous learning is powerful, regulatory barriers exist due to the unpredictable risks of continuously evolving and deploying. The FDA has approved AI-enabled systems that preclude changing a deployed model but recognize the need for a recognizable sustainability model. Ensuring sustainability requires testing the system under a range of conditions and data inputs to ensure that it remains reliable over time, even as the technology or data changes. Sustainability is critical to ensuring the long-term success and usefulness of AI systems.
1.1 Accuracy: Accuracy is critical for determining the reliability of an AI system, as it measures how well the system can perform its intended function. To assess accuracy, it is necessary to compare the results produced by the AI system against a benchmark or ground truth. This comparison can help identify errors and inaccuracies in the system, allowing developers to refine the algorithms and data inputs to improve accuracy.
1.2 Robustness: Robustness is also essential for determining the sustainability of an AI system, as it measures the system’s ability to perform well under unexpected or adverse conditions. Robustness is evaluated by subjecting the system to various inputs or data variations to determine whether it can produce reliable results. For example, a robust image recognition system should be able to identify objects in an image regardless of variations in lighting, color, or background.
1.3 Scalability: Scalability is another critical attribute for evaluating the sustainability of an AI system. Scalability measures the system’s ability to handle increasing amounts of data or users without a decrease in performance. A scalable system should be able to operate effectively under heavy loads, without sacrificing accuracy or robustness. Scalability is particularly important for AI systems that need to process large volumes of data, such as those used for natural language processing or image recognition.
2. Security of AI-enabled systems presents a greater challenge than securing traditional software systems. AI systems have new attack surfaces, with the learning element of some AI algorithms providing a powerful vector for attackers to change the output of a system. When considering a system designed to handle billions of features, the test space for an AI system can explode. AI-enabled systems frequently abstract input data into a non-human interpretable representation, making verification and validation difficult or impossible for humans. Consequently, this can create barriers to deployment for public sector software systems with substantial legal or financial ramifications. To secure AI systems, new threat modeling is necessary, which can impact how the systems are designed. Adversarial machine learning attacks rely on taking advantage of the openness of the input space for an AI system. As a result, traditional testing approaches like fuzzing cannot ensure the security of these systems.
3. Privacy in software and AI involves individuals having control and freedom of choice over the collection, use, and disclosure of their information. Guidelines and principles are available to help engineers understand privacy by design principles. Central AI concerns related to privacy include providing data owners with control over what is shared, ensuring that derivative inferences are explained, and designing AI systems with privacy boundaries in mind. Adversarial machine learning attacks, such as model stealing attacks, can be implicated in privacy breaches that expose individuals’ information. Unicity and re-identifiability in large data collections are frequently possible, hindering research in healthcare applications.
3.1 Fairness is a critical concern in the development and deployment of AI algorithms, despite their ability to enhance competitive advantage. A systematic review of 47 articles reveals a lack of formalized AI terminology and definitions, resulting in contrasting views of AI algorithmic fairness [2]. Most research has focused on the technical aspects of narrow AI, particularly in criminal justice and immigration and the health sector. Currently, AI algorithmic fairness is more focused on the technical and social/human aspects than the economic aspects. Evaluating the fairness of an AI system involves assessing its ability to produce unbiased results by evaluating the data inputs, algorithms, and decision-making processes to ensure they are free from bias. This is particularly important in areas such as hiring, lending, or insurance, where bias can have a significant impact on outcomes. A fair AI system treats all users equally, without bias or discrimination, and ensures unbiased results regardless of the user’s characteristics.
4. Explainability is an essential attribute of AI systems. It refers to the system’s ability to explain its decisions or actions in a way humans can understand, by providing transparency and interpretability. The trustworthiness of an AI system depends on a user’s ability to understand how an algorithm produces a result. Developing explainable systems is a fundamental research question of AI. A tension in engineering AI systems is that some methods do not permit the analyst to trace what feature(s) resulted in a prediction. Explainability is a fundamental concern of deploying AI systems in the clinic. Techniques for explaining how outputs arise from AI systems will enable fielding these systems and reduce risk. Storing and presenting intermediate output may aid in the explainability problem. Deep learning algorithms remain an evolving research area for explainability. This requires developers to design algorithms that are explainable and to provide clear explanations of the system’s decision-making processes.
3.1 Transparency: Transparency refers to the ability of an AI system to provide clear and understandable information about how it works, including the data it uses and the algorithms it employs. A transparent system can help users understand how the system works, the inputs it uses to make decisions, and the outcomes it produces. This requires developers to provide clear documentation, user manuals, and data visualizations that help users understand the system’s operation.
3.2 Interpretability: Interpretability refers to the ability of an AI system to provide explanations of its decisions or actions in a way that is understandable to humans. An interpretable system can provide insight into how it reached a particular conclusion, allowing users to understand the reasoning behind the system’s decisions. This requires developers to design algorithms that are interpretable and to provide clear explanations of the system’s decision-making processes.
5. Data centricity is a critical attribute of AI systems that emphasizes the importance of data quality, governance, and ethics in the development and operation of the system. By prioritizing these factors, AI systems can produce more accurate and reliable results, while also promoting ethical and responsible data use. Software elements need to be explicitly architected with data uncertainty, availability, and scalability in mind, as data is the key aspect that influences every aspect of AI system design. Most AI systems rely on a machine learning model trained on data, which is the weakest link and can degrade system performance if changed. Continuously learning models are fragile to adversarial attacks and may produce patterns that are incomprehensible to human experts. The scale of data poses engineering challenges and requires new storage and database architectures. However, ensuring data quality is difficult, which adds to the attack surface of AI systems.
5.1 Data quality: The first sub-attribute of data centricity is data quality, which refers to the accuracy, completeness, and consistency of the data used to train and operate an AI system. Data quality is essential for ensuring the AI system produces reliable and accurate results. To evaluate data quality, developers need to assess the data sources, the completeness and accuracy of the data, and the data cleaning and preprocessing techniques used.
5.2 Data governance: The second sub-attribute of data centricity is data governance, which refers to the management of data throughout its lifecycle, including the acquisition, storage, and use of data. Data governance is critical for ensuring that data is used ethically and responsibly and that it complies with legal and regulatory requirements. To evaluate data governance, developers need to assess the policies and procedures for data acquisition, storage, and use, and the measures in place to protect data privacy and security.
5.3 Data ethics: The third sub-attribute of data centricity is data ethics, which refers to the moral principles and values that guide the use of data in an AI system. Ethical considerations may include issues related to privacy, bias, and fairness. To evaluate data ethics, developers need to assess the ethical implications of the AI system, such as the potential for bias or discrimination, and the measures in place to address these issues.
Conclusion
In conclusion, maintaining quality is crucial for the success of AI systems. Quality attributes such as security, privacy, data centricity, sustainability, and explainability are essential to ensure the trustworthiness and reliability of AI systems. While AI systems may not require upfront requirement specifications, the overall quality concerns and expectations should still follow principles that can be known a priori and evolve with the systems. As an open research topic, AI engineering needs to progress on building robust systems, data, human-machine interaction, and models. By prioritizing these dimensions, we can develop trustworthy and robust AI software that meets our business and mission goals.
References:
[1] https://arxiv.org/pdf/1911.02912.pdf
[2] https://link.springer.com/chapter/10.1007/978-3-030-85447-8_24
