Cover page:
Responsible Innovation: Ethics and risks of new technologies
Responsible Innovation: Ethics and risks of new technologies
Editors
Introduction
0.1 When is innovation good for society?
0.3 Acknowledgements
Part I: General Introduction to RI
1. Introduction to responsible innovation
1.1 The real-world context of responsible innovation: dilemmas
Dilemma 4: Feedback and democratic influence
Should ordinary citizens have some level of influence on the design and availability of new technologies; or not? To what extent can societal actors, NGOs, citizens and other public groups influence technological development? Should they have the powe...
1.2 Why discuss responsible innovation?
1.3 Defining RI
1.4 Substantive and process aspect of RI
1.5 EU-definition of RI
Part II: Applied Ethics for Responsible Innovation
2. Applied ethics for responsible innovation
2.1 Applied ethic: thought experiments
The “Fat Man” case
2.2 How engineers answer the Trolley Problem
2.3 Individual moral responsibility
What does individual moral responsibility entail?
2. 4 Collective moral responsibility
The problem of freeriding But you might be wondering why the fishermen would stick to this scheme. Think back to individual rational self-interest and consider only fisherman A. If all the other boats comply with the quota scheme, then it is in fisher...
The limits of enforcement
2.5 Responsibility in complex systems
Introduction
The conditions for moral responsibility
The problem of many hands
Building responsibility into technology
2.6 Emotions and values
Introduction
The difference between risk and risk perception
Emotions as a guide to acceptable risk
2.7 Moral dilemmas and moral overload
Part III: Institutions and Values
Case study #1: Smart meters and conflicting values as an opportunity to innovate
Case study #2: Medical ethics in the age of AI and big data
IntroductionAs discussed in this chapter, the core idea of responsible innovation is that we try to accommodate as many of our moral values as we can by design, by tweaking the world, by innovation, by creativity. This also applies to medical ethics in a world of big data and AI. We should to accommodate our moral concerns AND make use of big data and AI in health care. There is no guarantee that this will always be possible. But because the stakes are high, we have the obligation to explore whether there are suitable solutions.
Digital technology and its impact on health care.
Digital technology affects health care in all its dimensions: research & development, clinical practice, policy, innovation, entrepreneurship, insurance and financing. It changes everything. It not only enables new practice, but is a constitutive technology. It is obvious that data and AI can reduce costs in health care, improve patient safety, empower patients, improve the quality of diagnosis, therapy and patient journeys, and create more efficient billing and logistics. Smart phones and watches with health apps, wearables and so-called digi-ceuticals are part of an Internet of Things revolution that is well underway in the health sector. Wearable devices can be used to detect arrhythmia, predict Parkinson’s disease (via the accelerometer in the phone), and measure a range of biomarkers such as blood sugar, blood pressure, fat percentage, oxygen and stress levels. They can diagnose skin cancer and retina damage and assist in the management of eating disorders, phobias, depression, chronic pain and PTSD. They can even gauge the risk of suicide on the basis of social media posts – looking at the time of day of the post, the number of human faces it contains and the colours.
Serious concerns
However, there are serious limitations to a purely data-driven approach and the glorification of statistical correlation. In medicine – and in other fields of great social importance – the data-driven approach, while it can be clinically useful and morally responsible, needs to be complemented by theory-driven approaches which aim at uncovering causal mechanisms.
Attitude problemsAnother problem is caused by the tech companies’ attitude to health care. The digital industry and Silicon Valley approach to health care is a solutionist approach, which focuses exclusively on problems for which we have nice and clean technological solutions at our disposal. David Lazer has called their approach ‘Big Data Hubris’: the idea that there are simple digital solutions to complex problems in the very complex world of health care, with its very complex institutional settings, multiple stakeholders, plurality of moral values and great cultural diversity. This is culpably naïve.
Moral problemsThere are not only epistemic failings in the digital usurpation of the health domain; there are also moral concerns. We know that there are race and gender - and many other - biases in health care data and they may become entrenched in algorithms, or even be built into medical systems with the conscious aim to deceive, in order to save money or make profits. US-based company Aspire Health, for example, tries to save money in palliative care by estimating which patients will die soon. Moreover, algorithms in decision support systems affect the fiduciary relationships between doctors and patients. Furthermore, there have been massive breaches of security and privacy in health care in the last decade. Deep Mind Health has been reprimanded by the Information Commission of the UK for its processing of NHS patient data. Deep Mind replied that it had “underestimated the complexity of the NHS and of the rules around patient data, as well as the potential fears about a well-known tech company working in health.”
It is all about trust. But the key question of course is: why would we trust Facebook, Uber, Google, Amazon and Microsoft with all of our sensitive medical data? They can’t even fix basic problems regarding fake news, data security, filter bubbles and bias, nor can they prevent the data of 50 million users being abused to run political campaigns. Big tech is essentially about quarterly revenues. These companies come to health care with a Silicon Valley approach to innovation: innovate in the grey zone, move fast, break things first and apologize later. This is not a very helpful approach in health care.
Formal and informal rules
3. Institutional context of innovations
3.1 Introduction
Substantive and procedural values
Institutions and their values
Accounting and designing for public values
Accounting for institutional values in innovation
Level 4: Resource Allocation & Employment
Applying the Four-Layer model of Institutions
These are the four different layers or categories for institutions that we can identify. A very interesting aspect of these different layers is indicated in the scheme shown above by the top-down and bottom-up arrows. The yellow arrow in figure 3.3 me...
Part IV: Management and innovation
Case study # 4: Self-Driving Vehicles
The problem of “many hands”The first area of concern is a problem we have already encountered a few times: the problem of “many hands”, resulting from several different actors playing parts of varying influence in the design and deployment of these vehicles. This raises concerns regarding the accountability of any one actor. Imagine an AV causes an accident due to a failed sensor, partly due to bad weather. Who is responsible? The driver? The car manufacturer? The company providing sensors? The company providing the key software? Perhaps the dealer doing vehicle maintenance? Or even the road authority which allowed these vehicles in the road despite the bad weather? Even if one actor is identified to be legally accountable, that does not mean the other actors are not involved, and their culpability is far from settled. Let us assume for the sake of argument that the human driver is held responsible. We can expect that a debate will inevitably arise about his/her responsibility, because the driver feels there was no wrongdoing on their part.
The “trolley problem”We can also ask how even the most complex algorithmic intelligence might deal with the “trolley problem”. What should the AV do if there is an oncoming vehicle on an impact trajectory, and the only options are to a) crash against that vehicle, endangering both drivers or b) make a sudden turn that will inevitably endanger a pedestrian nearby? AVs might very well face situations like these where a choice between two alternative accident scenarios needs to be made. Whatever the choice, that outcome would be based on the instructions it has been programmed with.From a consequentialist’s perspective, hitting, or even killing, the pedestrian would be the preferred option, because only one person will be at risk, rather than two if the two cars were to crash into one another. But from a Kantian perspective, this may be not the preferred option. Of course, this is not to mention that pedestrians might change their behaviors in the (ubiquitous) presence of AVs.
Distribution of utilityA third issue is the potential trade-off between travel times, safety and sustainability. Any optimization of the system from one of these three perspectives may not result in equal outcomes. Taking a safety perspective for instance, it is preferable to maintain longer distances between vehicles, but this arrangement would induce higher fuel consumption due to higher air resistance. Also, the utilized capacity of the road would not be optimal, possibly resulting in more congestion and longer travel times.Or let us assume AVs can drive short distances at 160 km/h without any risks. This may result in shorter travel times, but at the same time increased CO2 emissions. Or consider that AVs may in time become so convenient that they become preferable to public transportation even over long distances, potentially increasing emissions, but also indirectly inducing urban sprawl and increased land demand. Of course, we should not forget that trade-offs of this variety exist even now, in the current status quo.
Economic disparityFourth, there is the question of fair distribution when it comes to financial or economic considerations. At least initially, AVs will be more expensive than regular cars. Experts project that the cost of an AVs may be at least € 10,000 higher than that of comparable normal cars. We could argue that this is not a problem considering the benefits.However, if the road authorities were to allocate dedicated road space - say one lane of the highway - specifically for AVs, this would effectively reduce the available road capacity for normal vehicles, possibly leading to more congestion. Moreover, we could argue that since only affluent consumers would purchase expensive AVs, such a scheme would indirectly benefit only people in higher socioeconomic brackets, at the cost of those in lower brackets.
Decrease in demand for public transportationAVs could make individual car ownership more attractive. The logic is as follows. One of the competitive advantages of public transport for individuals is that they can work, read or sleep while commuting, for example by train. On the other hand, personal cars provide the option of an on-demand mode of transportation. However, AVs can essentially provide the best of both worlds, serving both as personalized on-demand transportation and freeing the driver from actively having to drive.In the long term, there could be a large-scale shift from public transportation to AVs, in turn exacerbating issues such as congestion on highways, pollution, emissions etc. Moreover, the decline of demand for public transportation could hurt the population from lower socio-economic brackets disproportionately, since this is the group that depends the most on public transportation and moreover, cannot afford AVs in the first place.
Increased safetyIn the long run, AVs could significantly improve safety not only for AV users, but also for non-users, such as pedestrians, (motor)cyclists and other drivers. Another overlooked area where we would see immediate and significant safety benefits would be the decrease in incidents due to driving under the influence of alcohol or other drugs.
Lower pollution and emissionsOverall energy use and emissions may also be reduced, perhaps directly, due to better design and more precise handling of AVs, and indirectly, as people shift to AVs that are known for their fuel-efficiency. After all, what sense is there in having a powerful but inefficient car?
Transition issues!It is also important to pay attention to the transition period between traditional cars and AVs. Experts think that, immediately after the initial market introduction of AVs, the capacity of roads might decrease, as well as road safety. This could be expected due to ‘growing pains’, failures in technology, or a period of real-world learning. Eventually though, AVs would be more efficient, solve highway capacity concerns and improve safety. In other words, the initial years of AVs could decrease the performance of the transportation system, but due to learning effects and increasing market penetration of AVs, the system could improve over time, eventually exceeding the performance of the status quo. In the meantime, of course, there will be an inter-temporal ethical issue.Moreover, we could argue that the relative safety of AVs as compared to human drivers would introduce an interesting trade-off when it comes to car insurance. Over time, insurance premiums for AVs - if they consistently have fewer and less severe incidents than human drivers - could become lower. This would effectively make AVs a preferred investment in the long term, pricing out traditional cars on purely economic grounds.
Embracing cautious optimism
4. Innovation and business
4.1 Incremental and radical innovation
A taxonomy of innovation
The link between radical an responsible innovation
Ethical considerations of radical innovations
4.2 Determinants of innovation
Innovative actors and their motivations
Economic determinants of innovation
4.3 Management of innovation
Management of innovation in companies
Innovation as a simple project
Innovation as a complex process
The modern innovation process
Case study #5: The development and diffusion of television
Case Study #6: Coolants
5 Frugal innovation
5.1 What is frugal innovation
5.2 The case for frugal innovations
5.3 The link between frugal innovation and responsible innovation
5.4 Innovation and social standards
How social standards impact frugal innovation
Caveats for frugal innovation
5.4 Innovation and inclusive development
The need for inclusive development
Achieving inclusive development with frugal innovation
Serving poor consumers
Serving poor producers We may also ask if frugal innovations are more inclusive towards poor producers than other types of innovation. The majority of poor producers can be found in the so-called informal sector. Poor producers generally have micro, ...
5.5 Conclusion
Case Study #7: TAHMO weather stations
6. Implementation of RI by companies: new standard
6.1 Introduction
6.2 Roadmap
Step 1: Top Management commitment and leadership A pre-requisite for RRI implementation is top management commitment. This commitment is necessary but not sufficient to achieve RRI intended outcomes, as the top-down approach should be integrated wi...
Step 2: Context analysis RRI is connected to a broad spectrum of factors related to the type and management policies of a company, the technology and products it works on, the sectors and markets, the pertinent regulatory frameworks and stakeholder...
Step 3: Materiality analysis
Step 4: Experiment and engage
Step 5: Validation The success of RRI up-take is strongly context-dependent and is affected by several factors, as underlined in the context analysis clause (e.g. company size, complexity of the organization, features of the technology, the level o...
Final step: Roadmap design Based on the outcomes of the above-mentioned steps, a RRI roadmap is designed to guide an organization to put RRI in practice RRI implementation. So, as we have seen, this means:
5.3 Template for RRI-Roadmap
6.4 SWOT analysis for RRI implementation
Part V: Risk assessment and safety
7. Understanding risk
7.1. Risk, Uncertainty and Ignorance
The difference between risk and uncertainty
The difference between uncertainty and ignorance
Dealing with risk, uncertainty and ignorance
7.2 Extreme uncertainty of unknown unknowns
Drawbacks of the Precautionary Principle
7.3 Technology assessment
Forerunners of responsible innovation
Types of technology assessment
Ad 1. Constructive Technology Assessment (CTA)
Ad. 3 Network Approach for Moral Evaluation (NAME)
Finally
Case Study 8 #: The debate on nuclear energy
Case study #9: When Big data meets Big brother
8. Risk management and safety engineering
8.1 Introduction
8.2 Definitions
8.3 Cost-benefit analysis
Anticipating various types of incidents and events
Net Present Value
Costs and benefits of safety measures
Disproportion factor
8.4 Quantifying and comparing risks
Performing risk analysis We now turn to the topic of risk analysis. Of course, the main questions for risk analysis are: what can go wrong, and how? (Each answer, and there could be more than one, is a risk scenario.) We can also ask: what is the lik...
Risk contours
Defining the system and its boundaries is of practical importance. In a way, we can understand events that occur within the system boundaries as outcomes we can prevent or control, whereas events that occur outside system boundaries should be seen as ...
Hazard analysis When we have defined the context and the system, we can think about the hazards. We have three questions to guide us: what can go wrong; how might this it happen; and what measures or controls do we have to contain the hazard? We thus...
Fault Tree Analysis A Fault Tree Analysis is a logical structuring of events leading to the top event, the outcome that is to be avoided as much as possible. Because of its logical structure, we can use fault trees to quantify risks.
Although it has one particular event as its top event, we can also use the Fault Tree for events that have not happened yet - that is to say, in a prospective way.
Consequence analysis In the next step of our risk analysis, we look at the consequences of a scenario. We often define consequences in terms of fatalities, injuries or money.
Anticipating risk scenarios
Risk assessment
Safety measures Finally, we come to the treatment of the identified risks. Here, safety comes into full view. Basically, we have four possibilities: risk avoidance, risk reduction, risk transfer and risk acceptance. These measures are illustrated in ...
Risk analysis in practice In the previous sections, we have seen the main steps of a risk analysis so far. Note, however, that this is not a linear process with a fixed endpoint. Risk analysis ideally never ends; it is a continuous process of anticip...
Part VI Value Sensitive Design
9. Value Sensitive Design
9.1 Introduction to Value Sensitive Design
Converging lines of thought
‘Do artifacts have politics?’
Science and Technology Studies
Concerns by engineers
9.2 Defining the method of Value Sensitive Design
The most clear and precise formulation of the VSD concept originated in a movement at Stanford in the 1970s-80s in the field of Computer Science, advocated strongly by Terry Winograd. It has now been adopted by many research groups and is often referr...
9.3 Applying VSD in practice
Does technology embody values?
Instrumentalism
What values should be included in technology design?
9.4 How can we translate moral values into design specifications?
9.5 Complicated process
Case study #10: Autonomous weapons
Case study #11: Care robots
Summary
Appendices
Appendix 1: Overview of EU funded Projects in the field of RI
Appendix 2: Questions for consideration
Questions for par. 2.2
Questions for par. 2.3
Questions for par. 2.5
Questions for par. 2.6
Questions for par. 4.3
Questions for par. 5.4
Questions for par. 6.4
Questions for par. 7.3
Questions for par. 8.1
Questions for par. 8.3
Questions for par. 9.3
Appendix 3: Teachers and link to weblectures
Appendix 4: Credit figures