To counter the vulnerabilities of model parameters to inference and Byzantine attacks during federated learning, we propose a robust and privacy-preserving federated learning scheme based on user selection, enhancing the security and reliability of model training. We first design a user selection algorithm based on the concept of groups constructed on fog servers. The purpose of this algorithm is to select users with high credit scores to contribute to the training of the global mode. Next, we construct a method for filtering local model parameters and updating user scores using the test set from the cloud server, effectively mitigating the interference from malicious users in the model training process and progressively excluding them from training, thereby enhancing the robustness of the global model. Finally, we design a lightweight encryption algorithm based on cloud-fog collaboration, which not only effectively protects the privacy of user local model parameters but also ensures their security during the aggregation process, all while maintaining high computational and communication efficiency. Building upon the computational challenge of the Diffie-Hellman (CDH) problem, we have demonstrated the security of our scheme, which resists various attacks, ensuring the global model's robustness while safeguarding user data privacy. By comparing with existing schemes and through performance analysis and experimental results, our proposal excels in efficiency. When facing malicious attackers, the accuracy rate of directly aggregated global models drops to about 65%, whereas our scheme maintains an accuracy rate close to that of a scenario without attacks, effectively mitigating the impact of attacks. Thus, our solution offers a practical and effective strategy for federated learning systems to deal with inference and Byzantine attacks.