Datto Engineering Blog

Providing remote access to Datto devices via SSH tunnels

Our backup devices are typically physically located inside the LAN of our end users. Under normal circumstances that means that they are behind a NAT and are not reachable from the public Internet without a VPN or other tunneling mechanisms. For our customers, the Managed Service Provider (MSP), only being able to access their Datto devices with direct physical access would be a major inconvenience. In this post, we talk about how we implemented "Remote Web", a feature that lets customers remotely access the device, even when it is behind a NAT.

Cracking passwords to prevent credential stuffing

People tend to reuse the same password for all of their online accounts. With an attack technique called "credential stuffing", a single data breach can result in millions of compromised accounts across the web. Hackers use this technique for evil; here's how we used it to protect our customers.

Tearing apart IPMI to learn how it's put together

Have you ever looked at the backplane of a server? You know, the kind that’s meant to go into a rack in a datacenter? They’ll usually have more than one ethernet port, but oddly, a lot of the time one of those ethernet ports is gonna be off to the side, over with the USB ports. That’s your server’s IPMI interface, and it can do a lot. We go over how to inspect and tear apart an IPMI image to learn how they're put together.

Causing ZFS corruption for fun and profit (and quality assurance purposes)

Datto backs up data, a lot of it. At the time of writing Datto has over 500 PB of data stored on ZFS. This count includes both backup appliances that are sent to customer sites, as well as cloud storage servers that are used for secondary and tertiary backup of those appliances. At this scale drive swaps are a daily occurrence, and data corruption is inevitable. How we handle this corruption when it happens determines whether we truly lose data, or successfully restore from secondary backup. In this post we'll be showing you how at Datto we intentionally cause corruption in our testing environments, to ensure we're building software that can properly handle these scenarios.

Flexible and fast software delivery with the Open Build Service

Delivering and deployment of software is hard. Continuously doing so while dealing with ever-changing requirements and scenarios in a secure and reproducible way? Even harder! We show how we use the Open Build Service at Datto to build and deliver software for dozens of Linux distributions every day.

How we upgrade the software and operating system of thousands of appliances every two weeks

In this post, we describe how we moved from Debian-based deployments in our fleet of >80,000 devices to image based upgrades. We show the nitty gritty details of how we use Grub and loop devices to boot from image to image seamlessly, every two weeks.