#90 Sharing Data Reliably in Hyperscale Mode - Interview w/ Björn Smedman
Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.Björn's LinkedIn: https://www.linkedin.com/in/bjornsmedman/In this episode, Scott interviewed Björn Smedman, Engineering Manager at Communication Platform-as-a-Service (CPaaS) company Sinch.Some interesting thoughts or takeaways:A good indicator for when decentralizing your data team might make sense is the cognitive load of a centralized data team. How many systems - including a measure of how complex - are they managing? How much of their time is spent in meetings, especially trying to understand context/requests? Is there starting to be combative prioritization from multiple domains? It can be very beneficial and scalable to apply data mesh principles to non analytical use cases, especially sharing data for application purposes. It is still often difficult to prioritize creating a data product for machine learning without knowing the business value of the ML model. But the ML team needs the data first before they can figure out the business value of the ML model. You have to make speculative bets.If you see the data platform team start to dig into the semantics of a use case, that's a red flag that people are trying to leverage them as a data team. And while you want a centralized data platform team, you probably don't want them to become a centralized data team.Since December 2020, Sinch raised nearly $2 billion USD. With this funding, they have made a number of sizeable acquisitions, with the company growing from 500 employees to over 3,000 in about a year. This has led to some interesting challenges in sharing data in a hyper-scaling environment. Per Björn, data is a very key part of Sinch's plans for growth. Sinch's operational systems are often very transactional, as some product lines can process tens of thousands of monetary transactions a second, so data that might be typically shared on the operational plane in other companies is shared on the data plane lest the operational data stores deal with billions of events, making the data challenges even more complex than for most organizations. Then add in the regulatory requirements of telecom.Björn helped lead the move to decentralizing the data...