How do Algorithms decide? Peering into the Black Box

As an employer, Amazon is much in demand and the company receives a flood of applications. Little wonder, therefore that they are seeking ways to automate the pre-selection process, which is why the company developed an algorithm to filter out the most promising applications. This AI algorithm was trained using employee data sets to enable it to learn who would be a good fit for the company. However, the algorithm systematically disadvantaged women. Because more men had been recruited in the past, far more of the training data sets related to men than women, as a result of which the algorithm identified gender as a knockout criterion. Amazon finally abandoned the system when it was found that this bias could not be reliably ruled out despite adjustments to the algorithm.

This example shows how quickly someone could be placed at a disadvantage in a world of algorithms, without ever knowing why, and often without even knowing it. “Should this happen with automated music recommendations or machine translation, it may not be critical,” says Marco Huber, “yet it is a completely different matter when it comes to legally and medically relevant issues or in safety-critical industrial applications.”

One approach involves decision tree algorithms, which present a series of structured yesno (binary) questions. These are even familiar from school: whoever has been asked to graph all possible combinations of heads and tails when flipping a coin multiple times will have drawn a decision tree. Of course, the decision trees Huber's team uses are more complex.

“Neural networks need to be trained with data before they can even come up with reasonable solutions,” he explains, whereby “solution” means that the network makes meaningful predictions. The training represents an optimization problem to different solutions are possible, which in addition to the input data, also depend on boundary conditions, which is where decision trees come in. “We apply a mathematical constraint to the training to ensure that the smallest possible decision tree can be extracted from the neural network,” Huber explains. And because the decision tree renders the forecasts comprehensible, the network (black box) is rendered “white”. “We nudge it to adopt a specific solution from among the many potential solutions,” says the computer scientist: “probably not the optimal solution, but one that we can retrace and understand.”

Many youngsters in Britain might have wished for a counterfactual explanation of that kind this year. Final exams were cancelled due to the Covid-19 pandemic, after which the Ministry of Education then decided to use an algorithm to generate final grades. The result was that some students were given grades that were well below what they expected to receive, which resulted in an outcry throughout the country. The algorithm took account of two main aspects: an assessment of individual’s general performance and exam results at the respective school from previous years. As such, the algorithm reinforced existing inequalities: a gifted student automatically fared worse in an at-risk school than in a prestigious school.

In Sarah Oppold’s opinion, this is an example of an algorithm implemented in an inadequate manner. “The input data was unsuitable and the problem to be solved was poorly formulated,” says the computer scientist, who is currently completing her doctoral studies at the University of Stuttgart’s Institute of Parallel and Distributed Systems (IPVS), where she is researching how best to design AI algorithms in a transparent manner. “Whilst many research groups are primarily focusing on the model underlying the algorithm,” Oppold explains, “we are attempting to cover the entire chain, from the collection and pre-processing of the data through the development and parameterization of the AI method to the visualization of the results.” Thus, the objective in this case is not to produce a white box for individual AI applications, but rather to represent the entire life cycle of the algorithm in a transparent and traceable manner.

The result is a kind of regulatory framework. In the same way that a digital image contains metadata, such as exposure time, camera type and location, the framework would insert explanatory notes to an algorithm – for example, that the training data refers to Germany and that the results, therefore, are not transferable to other countries. “You could think of it like a drug,” says Oppold: “It has a specific medical application and a specific dosage, but there are also associated risks and side effects. Based on that information, the health care provider will decide which patients the drug is appropriate for.”

The framework has not yet been developed to the point where it can perform comparable tasks for an algorithm. “It currently only takes tabular data into account,” Oppold explains: “We now want to expand it to take in imaging and streaming data.” A practical framework would also need to incorporate interdisciplinary expertise, for example from AI developers, the social sciences and lawyers. “As soon as the framework reaches a certain level of maturity,” the computer scientist explains, “it would make sense to collaborate with the industrial sector to develop it further and make the algorithms used in industry more transparent .”

Text: Michael Vogel