“It’s not about shaming. It’s just about being honest.”
Artificial Intelligence is having what some call a “reproducibility crisis”, as its been reported that only 6% of researchers share their code, and only third share their data (1). It has been argued that the issue of reproducibility goes beyond just reluctancy to report failed replications, but that factor such as the random numbers generated to kick-off training and the high sensitivity to the exact code should also be considered (1). Furthermore, it has been said that there’s too little emphasis on what is termed “hyperparameters” when discussing reproducibility. The latter can be described as settings which are not core to the algorithm but affect how quickly it can learn (1).
Figure 1: ‘The same algorithm can learn to walk in wildly different ways’ (1).
Whilst closely linked to the “black box” problem in which the mechanisms behind the functioning of the algorithm are not clear, the reproducibility problem has been categorised as its own challenge (2). The problem of reproducibility has mostly been deduced to both poor and inconsistent experimental and publication practices (2).
Leave a comment and find out more below!
Written by Nefti-Eboni Bempong