Datto Engineering Blog

Tearing apart IPMI to learn how it's put together

Have you ever looked at the backplane of a server? You know, the kind that’s meant to go into a rack in a datacenter? They’ll usually have more than one ethernet port, but oddly, a lot of the time one of those ethernet ports is gonna be off to the side, over with the USB ports. That’s your server’s IPMI interface, and it can do a lot. We go over how to inspect and tear apart an IPMI image to learn how they're put together.

Causing ZFS corruption for fun and profit (and quality assurance purposes)

Datto backs up data, a lot of it. At the time of writing Datto has over 500 PB of data stored on ZFS. This count includes both backup appliances that are sent to customer sites, as well as cloud storage servers that are used for secondary and tertiary backup of those appliances. At this scale drive swaps are a daily occurrence, and data corruption is inevitable. How we handle this corruption when it happens determines whether we truly lose data, or successfully restore from secondary backup. In this post we'll be showing you how at Datto we intentionally cause corruption in our testing environments, to ensure we're building software that can properly handle these scenarios.

Flexible and fast software delivery with the Open Build Service

Delivering and deployment of software is hard. Continuously doing so while dealing with ever-changing requirements and scenarios in a secure and reproducible way? Even harder! We show how we use the Open Build Service at Datto to build and deliver software for dozens of Linux distributions every day.

How we upgrade the software and operating system of thousands of appliances every two weeks

In this post, we describe how we moved from Debian-based deployments in our fleet of >80,000 devices to image based upgrades. We show the nitty gritty details of how we use Grub and loop devices to boot from image to image seamlessly, every two weeks.