Abstract: Non-independent and identically distributed (Non-IID) data across clients remains a fundamental challenge in federated learning. A critical gap exists: the lack of a universal metric capable of both quantifying distributional differences across diverse Non-IID scenarios and fairly evaluating algorithmic robustness. This work addresses this gap by developing a generic metric for the degree of Non-IIDness among distributed nodes. We propose a Domain-shift-based Metric (DSM). The core idea is to circumvent the intractability of measuring divergence in high-dimensional raw data by leveraging a pre-trained feature extractor. DSM transforms the problem into estimating domain shift within a learned feature space. Specifically, it quantifies the Non-IID degree as theweighted sum of Wasserstein distances between client feature distributions and the global feature distribution. We validated DSM on MNIST and Fashion-MNIST datasets, simulating five canonical Non-IID scenarios. Our experiments demonstrate that: DSM effectively differentiates bias levels both within and across these five scenarios, showing superior sensitivity compared to four conventional metrics (e.g., Euclidean, Cosine distances); After calibrating different scenarios to an equivalent Non-IID degree using DSM, we conducted a fair comparison of four federated classification algorithms and four federated generative algorithms. The evaluation consistently identified Scaffold as the top performer for classification and BGAN for generation, validating DSM's utility for fair cross-scenario algorithm assessment. The proposed DSM metric establishes the first unified, quantitative benchmark for Non-IIDness across scenarios in federated learning. As a model-agnostic tool, it provides a principled basis for algorithm evaluation and selection. Furthermore, it offers a crucial quantitative foundation for designing collaborative learning algorithms in distributed systems with inherent Non-IID data, such as satellite constellations in aerospace applications.

Key words: federated learning, Non-IID, distribution discrepancy measurement, domain adaptation, spaceborne distributed learning