A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems

Neural Networks, 2023
PDF  

Abstract: Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to “real world” events, where the input distributions and tasks encountered by the deployed systems will not be limited to the original training context, and systems will instead need to adapt to novel distributions and tasks while deployed. This critical gap may be addressed through the development of “Lifelong Learning” systems that are capable of 1) Continuous Learning, 2) Transfer and Adaptation, and 3) Scalability. Unfortunately, efforts to improve these capabilities are typically treated as distinct areas of research that are assessed independently, without regard to the impact of each separate capability on other aspects of the system. We instead propose a holistic approach, using a suite of metrics and an evaluation framework to assess Lifelong Learning in a principled way that is agnostic to specific domains or system techniques. Through five case studies, we show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems. We highlight how the proposed suite of metrics quantifies performance trade-offs present during Lifelong Learning system development - both the widely discussed Stability-Plasticity dilemma and the newly proposed relationship between Sample Efficient and Robust Learning. Further, we make recommendations for the formulation and use of metrics to guide the continuing development of Lifelong Learning systems and assess their progress in the future.

Recommended citation:

@article{baker2023domain, 
 author = {Baker, Megan M. and New, Alexander and Aguilar-Simon, Mario and Al-Halah, Ziad and Arnold, Sébastien M. R. and Ben-Iwhiwhu, Ese and Brna, Andrew P. and Brooks, Ethan and Brown, Ryan C. and Daniels, Zachary and Daram, Anurag and Delattre, Fabien and Dellana, Ryan and Eaton, Eric and Fu, Haotian and Grauman, Kristen and Hostetler, Jesse and Iqbal, Shariq and Kent, Cassandra and Ketz, Nicholas and Kolouri, Soheil and Konidaris, George and Kudithipudi, Dhireesha and Learned-Miller, Erik and Lee, Seungwon and Littman, Michael L. and Madireddy, Sandeep and Mendez-Mendez, Jorge and Nguyen, Eric Q. and Piatko, Christine D. and Pilly, Praveen K. and Raghavan, Aswin and Rahman, Abrar and Ramakrishnan, Santhosh Kumar and Ratzlaff, Neale and Soltoggio, Andrea and Stone, Peter and Sur, Indranil and Tang, Zhipeng and Tiwari, Saket and Vedder, Kyle and Wang, Felix and Xu, Zifan and Yanguas-Gil, Angel and Yedidsion, Harel and Yu, Shangqun and Vallabha, Gautam K.}, 
 title = {A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems}, 
 journal = {Neural Networks}, 
 year = {2023}, 
 volume = {160}, 
 pages = {274--296}, 
 url = {https://arxiv.org/abs/2301.07799}}