How CERN analyzed 1 PetaByte per second using K8s with Ricardo Rocha
One PetaByte is the equivalent of 11000 4k movies. And CERN's Large Hadron Collider (LHC) generates this every single second. Only a fraction of this data (~1 GB/s) is stored and analyzed using a multicluster batch job dispatcher with Kueue running on Kubernetes.
In this episode we have Ricardo Rocha, Platform Engineering Lead at CERN and CNCF Advocate, explaining why after 20 years at CERN he is still excited about the work he and his colleagues at CERN are doing. To kick things off we learn about the impact that the CNCF has on the scientific community, how to best balance an implementation of that scale between "easy of use" vs "optimized for throughput". Tune in and learn about custom hardware being built 20 years ago and how the advent of the latest chip generation has impacted the evolution of data scientists around the globe
Links we discussed
Ricardo's LinkedIn: https://www.linkedin.com/in/ricardo-rocha-739aa718/
KubeCon SLC Keynote: https://www.youtube.com/watch?v=xMmskWIlktA&list=PLj6h78yzYM2Pw4mRw4S-1p_xLARMqPkA7&index=5
Kueue CNCF Project: https://kubernetes.io/blog/2022/10/04/introducing-kueue/