Federated Self-Supervised Learning and Deep Clustering

Abstract: Self-supervised learning (SSL) is a key technique in the pretraining stage of large language models (LLMs), used across various tasks including natural language processing, computer vision, and speech processing. In SSL, models are trained on unlabeled data to learn meaningful representations that can be transferred to other downstream tasks. However, concerns around data privacy and limited computing resources for users and customers have led to the development of federated learning (FL) to address these issues. In our work, we focus on training models with self-supervised learning and extending it in a federated setting. We aim to train a global model that produces robust data representations and performs data clustering by tackling challenges from both self-supervised learning and federated learning. We propose a federated momentum contrastive clustering framework that simultaneously generates data representations and performs clustering through a novel two-stage federated learning setup. Additionally, we introduce a clustering-guided federated learning framework, designed to improve the quality of data representations. This is achieved by incorporating a novel clustering loss with a data selection strategy and a dynamic controller to update local networks during communication round, effectively mitigating client drift in federated learning. Moreover, we develop a resource-efficient federated clustering scheme that introduces a past negatives pool to combat client drift in FL and relief class collision issue in SSL. Finally, we apply federated learning to address a tomographic reconstruction problem with quantized inter-client communication, successfully demonstrating image recovery through compression schemes.