AI’s reproducibility problem

“It’s not about shaming. It’s just about being honest.”

Artificial Intelligence is having what some call a “reproducibility crisis”, as its been reported that only 6% of researchers share their code, and only third share their data (1).  It has been argued that the issue of reproducibility goes beyond just reluctancy to report failed replications, but that factor such as the random numbers generated to kick-off training and the high sensitivity to the exact code should also be considered  (1).  Furthermore, it has been said that there’s too little emphasis on what is termed “hyperparameters” when discussing reproducibility.  The latter can be described as settings which are not core to the algorithm but affect how quickly it can learn (1).

Bildergebnis für AI reproducibility

Figure 1: ‘The same algorithm can learn to walk in wildly different ways’ (1).

Whilst closely linked to the “black box” problem in which the mechanisms behind the functioning of the algorithm are not clear, the reproducibility problem has been categorised as its own challenge (2). The problem of reproducibility has mostly been deduced to both poor and inconsistent experimental and publication practices (2).

Leave a comment and find out more below! 

 

References 

[1] http://science.sciencemag.org/content/sci/359/6377/725.full.pdf

[2] http://www.sciencemag.org/news/2018/05/ai-researchers-allege-machine-learning-alchemy

 

Written by Nefti-Eboni Bempong

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: