Arroyo with Micah Wylde

In this episode, we have Micah Wylde from Arroyo as our guest. Micah introduces us to Arroyo, a real-time data processing engine that simplifies stream processing for data engineers using Rust. They explain how Arroyo enables users to write SQL queries with Rust user-defined functions on top of streaming data, highlighting the advantages of real-time data processing and discussing the challenges posed by competitors like Apache Flink. Moving on, we dive into the use of Rust in Arroyo and its benefits in terms of performance and memory safety. We explore the complementarity of workflow engines and stream processors and examine Arroyo's approach to real-time SQL and its compatibility with Postgres. Micah delves into memory and lifetime concerns and elaborates on how Arroyo manages them in its storage layer. Shifting gears, we explore the use of the Tokyo framework in the Arroyo system and how it has enhanced speed and efficiency. Micah shares insights into the challenges and advantages of utilizing Rust, drawing from their experiences with Arroyo projects. Looking ahead, we discuss the future of the Rust ecosystem, addressing the current state of the Rust core and standard library, as well as the challenges of interacting with other languages using FFI or dynamically loading code. We touch upon Rust's limitations regarding a stable ABI and explore potential solutions like WebAssembly. We also touch upon industry perceptions of Rust, investor perspectives, and the hiring process for Rust engineers. The conversation takes us through the crates used in the Arroyo system, our wishlist for Rust ecosystem improvements, and the cost-conscious nature of companies that make Rust an attractive choice in the current macroeconomic environment. As we wrap up, we discuss the challenges Rust faces in competing with slower Java systems and ponder the potential for new languages to disrupt the trend in the future. We touch upon efficiency challenges in application software and the potential for a new language to emerge in this space. We delve into the increasing interest in using Rust in data science and the promising prospects of combining Rust with higher-level languages. Finally, we discuss the importance of fostering a welcoming and drama-free Rust community. I would like to thank Micah for joining us today and sharing their insights. To find more resources related to today's discussion, please refer to the show notes. Stay tuned for our next episode, and thank you for listening!

Om Podcasten

This is "Rust in Production", a podcast about companies who use Rust to shape the future of infrastructure. We follow their journey in pursuit of more reliable and efficient software as they solve some of the most challenging technical problems in the world. Each episode dives deep into real-world applications of Rust, showcasing how this powerful systems programming language is revolutionizing the way we build and maintain critical infrastructure. From startups to tech giants, we explore the diverse landscape of organizations leveraging Rust's unique features to create safer, faster, and more scalable systems. Our guests share their experiences, challenges, and triumphs in adopting Rust for production environments. Listen in as we discuss topics such as concurrent programming, memory safety, performance optimization, and how Rust's ownership model contributes to building robust software systems. Whether you're a seasoned Rust developer, an infrastructure engineer, or a tech leader considering Rust for your next project, "Rust in Production" offers valuable insights and practical knowledge. Release Schedule "Rust in Production" releases new episodes every other Thursday at 4 PM UTC. Our podcast is structured into seasons, each featuring a diverse range of companies and experts in the Rust ecosystem. Recent episodes have included: - Season 2: Interviews with representatives from System76, Fusion Engineering, OxidOS, Matic, Thunderbird, AMP, and curl. - Season 1: Conversations with leaders from Sentry, Tweede Golf, Arroyo, Apollo, PubNub, and InfluxData. What You'll Learn - Real-world case studies of Rust implementation in production environments - Insights into how companies overcome technical challenges using Rust - Best practices for adopting Rust in various infrastructure contexts - The impact of Rust on software reliability, efficiency, and scalability - Future trends in systems programming and infrastructure development Join us as we uncover the latest trends in Rust development, explore best practices for using Rust in production, and examine how this language is addressing some of the most pressing issues in modern software engineering. From web services and databases to embedded systems and cloud infrastructure, we cover the full spectrum of Rust's impact on the tech industry.