Toward a common language to facilitate reproducible research and technology transfer. Grigori Fursin
Toward a common language to facilitate reproducible research and technology transfer: challenges and solutions
Grigori Fursin, Co-chair of the MLCommons task force on automation and reproducibility, President of the cTuning foundation, and Founder of cKnowledge.org
Abstract: During the past 10 years, we have considerably improved the reproducibility of experimental results from published papers by introducing the artifact evaluation process with a unified artifact appendix and reproducibility checklists, Jupyter notebooks, containers, and Git repositories. On the other hand, our experience reproducing more than 200 papers shows that it can take weeks and months of painful and repetitive interactions between teams to reproduce artifacts. This effort includes decrypting numerous README files, examining ad-hoc artifacts and containers, and figuring out how to reproduce computational results. Furthermore, snapshot containers pose a challenge to optimize algorithms’ performance, accuracy, power consumption and operational costs across diverse and rapidly evolving software, hardware, and data used in the real world.
In this talk, I will explain how our practical artifact evaluation experience and the feedback from researchers and evaluators motivated us to develop a simple, intuitive, technology agnostic, and English-like scripting language called Collective Mind (CM). It helps to automatically adapt any given experiment to any software, hardware, and data while automatically generating unified README files and synthesizing modular containers with a unified API. It is being developed by MLCommons to facilitate reproducible AI/ML Systems research and minimizing manual and repetitive benchmarking and optimization efforts, reduce time and costs for reproducible research, and simplify technology transfer to production. I will also present several recent use cases of how CM helps MLCommons, the Student Cluster Competition, and artifact evaluation at ACM/IEEE conferences. I will conclude with our development plans, new challenges, possible solutions, and upcoming reproducibility and optimization challenges powered by the MLCommons Collective Knowledge platform and CM: access.cKnowledge.org.
Смотрите видео Toward a common language to facilitate reproducible research and technology transfer. Grigori Fursin онлайн без регистрации, длительностью часов минут секунд в хорошем качестве. Это видео добавил пользователь Association for Computing Machinery (ACM) 31 Октябрь 2023, не забудьте поделиться им ссылкой с друзьями и знакомыми, на нашем сайте его посмотрели 26 раз и оно понравилось людям.