As research and technological development in artificial intelligence progress, questions around AI evaluation become more pressing. How can we evaluate and compare a wide range of competing or complementary AI techniques? How can we ensure that experiments can be replicated – ensuring the long-term success of AI research? And how can we make sure that what we measure in some way reflects “intelligence”? This talk presents Project Malmö, which enables AI experimentation on top of the computer game Minecraft. After the talk, there will be an open discussion with students and academics about possible projects using Malmö.