Datto Engineering Blog

A few police-action stories from working on the Datto Linux backup agent

A few years ago, Datto released its very own Linux backup agent. In this post, we reminisce about lessons learned from developing and supporting the Datto Linux Agent (DLA) shortly after its production release.

Wi-Fi Roaming: how it works, how to successfully deploy it and how to troubleshoot it

Understanding how Wi-Fi roaming works is key to a successful Wi-Fi deployment. This post focuses on the most difficult Wi-Fi environment, the enterprise business, and explains how to fix problems and why things sometimes don't work.

Automating Vault and Consul Template Management

Configuring and managing Vault isn't too difficult, but integrating it with our existing configuration management tools provided a unique challenge. We wanted to ensure that we could continue using Puppet, our config management tool of choice, to automatically handle the day-to-day operations of our new Vault deployment.

Understanding chkdsk for NTFS

chkdsk is a utility from Microsoft that tries to fix a volume’s file system without losing data. It checks for both logical and physical errors. chkdsk usually checks the file system metadata information first and applies file system repair functions if necessary. At Datto, we mostly deal with NTFS, so in this post, I'll only discuss the NTFS file system.

Keeping it simple: Using Server-Sent Events (SSE) to power a live UI

Ever wonder how your favorite web-based app gets real-time updates without a page refresh? We wanted to do this for one of our networking products, but there are many ways to go about it. Here's how we ended up at one less well-known solution for building a live-updating web UI.

Putting the devices you know and love into our cloud

One of Datto’s ongoing missions is continually improving and advancing our cloud offerings. The SIRIS has proven itself over and over again to perform well with local restores, so why haven’t we put them in Datto’s cloud? By putting SIRIS in Datto's cloud we can improve two products offerings simultaneously without needing to adjust our test suites or even change our hardware.

Providing remote access to Datto devices via SSH tunnels

Our backup devices are typically physically located inside the LAN of our end users. Under normal circumstances that means that they are behind a NAT and are not reachable from the public Internet without a VPN or other tunneling mechanisms. For our customers, the Managed Service Provider (MSP), only being able to access their Datto devices with direct physical access would be a major inconvenience. In this post, we talk about how we implemented "Remote Web", a feature that lets customers remotely access the device, even when it is behind a NAT.

Cracking passwords to prevent credential stuffing

People tend to reuse the same password for all of their online accounts. With an attack technique called "credential stuffing", a single data breach can result in millions of compromised accounts across the web. Hackers use this technique for evil; here's how we used it to protect our customers.

Tearing apart IPMI to learn how it's put together

Have you ever looked at the backplane of a server? You know, the kind that’s meant to go into a rack in a datacenter? They’ll usually have more than one ethernet port, but oddly, a lot of the time one of those ethernet ports is gonna be off to the side, over with the USB ports. That’s your server’s IPMI interface, and it can do a lot. We go over how to inspect and tear apart an IPMI image to learn how they're put together.

Causing ZFS corruption for fun and profit (and quality assurance purposes)

Datto backs up data, a lot of it. At the time of writing Datto has over 500 PB of data stored on ZFS. This count includes both backup appliances that are sent to customer sites, as well as cloud storage servers that are used for secondary and tertiary backup of those appliances. At this scale drive swaps are a daily occurrence, and data corruption is inevitable. How we handle this corruption when it happens determines whether we truly lose data, or successfully restore from secondary backup. In this post we'll be showing you how at Datto we intentionally cause corruption in our testing environments, to ensure we're building software that can properly handle these scenarios.