Configuring and managing Vault isn't too difficult, but integrating it with our existing configuration management tools provided a unique challenge. We wanted to ensure that we could continue using Puppet, our config management tool of choice, to automatically handle the day-to-day operations of our new Vault deployment.
chkdsk is a utility from Microsoft that tries to fix a volume’s file system without losing data. It checks for both logical and physical errors. chkdsk usually checks the file system metadata information first and applies file system repair functions if necessary. At Datto, we mostly deal with NTFS, so in this post, I'll only discuss the NTFS file system.
Ever wonder how your favorite web-based app gets real-time updates without a page refresh? We wanted to do this for one of our networking products, but there are many ways to go about it. Here's how we ended up at one less well-known solution for building a live-updating web UI.
One of Datto’s ongoing missions is continually improving and advancing our cloud offerings. The SIRIS has proven itself over and over again to perform well with local restores, so why haven’t we put them in Datto’s cloud? By putting SIRIS in Datto's cloud we can improve two products offerings simultaneously without needing to adjust our test suites or even change our hardware.
Our backup devices are typically physically located inside the LAN of our end users. Under normal circumstances that means that they are behind a NAT and are not reachable from the public Internet without a VPN or other tunneling mechanisms. For our customers, the Managed Service Provider (MSP), only being able to access their Datto devices with direct physical access would be a major inconvenience. In this post, we talk about how we implemented "Remote Web", a feature that lets customers remotely access the device, even when it is behind a NAT.
People tend to reuse the same password for all of their online accounts. With an attack technique called "credential stuffing", a single data breach can result in millions of compromised accounts across the web. Hackers use this technique for evil; here's how we used it to protect our customers.
Have you ever looked at the backplane of a server? You know, the kind that’s meant to go into a rack in a datacenter? They’ll usually have more than one ethernet port, but oddly, a lot of the time one of those ethernet ports is gonna be off to the side, over with the USB ports. That’s your server’s IPMI interface, and it can do a lot. We go over how to inspect and tear apart an IPMI image to learn how they're put together.
Datto backs up data, a lot of it. At the time of writing Datto has over 500 PB of data stored on ZFS. This count includes both backup appliances that are sent to customer sites, as well as cloud storage servers that are used for secondary and tertiary backup of those appliances. At this scale drive swaps are a daily occurrence, and data corruption is inevitable. How we handle this corruption when it happens determines whether we truly lose data, or successfully restore from secondary backup. In this post we'll be showing you how at Datto we intentionally cause corruption in our testing environments, to ensure we're building software that can properly handle these scenarios.
Delivering and deployment of software is hard. Continuously doing so while dealing with ever-changing requirements and scenarios in a secure and reproducible way? Even harder! We show how we use the Open Build Service at Datto to build and deliver software for dozens of Linux distributions every day.
In this post, we describe how we moved from Debian-based deployments in our fleet of >80,000 devices to image based upgrades. We show the nitty gritty details of how we use Grub and loop devices to boot from image to image seamlessly, every two weeks.