Inside Sketch: What happens when you save a document to your Cloud Workspace
Find out how our render farm takes the strain off your Mac and makes Cloud Workspace uploads faster and more reliable
You probably work on Sketch documents stored in your Cloud Workspace multiple times a day. But what happens when you actually hit Save? In this article, we’ll dig a little deeper into why we developed our render farm, and how it manages the heavy lifting when it comes to uploading your document to the cloud.
Last year we set out to improve the way in which we uploaded documents to your Cloud Workspace and make the process faster and more efficient for everyone. We also thought about how we could offer new features — like developer handoff and real-time collaboration. Our solution was to build our own render farm.
Wait, did you say “render farm”?
Yes, you read that right. For the last twelve months, we’ve been using a render farm to process your documents so that they’re ready for viewing, inspecting, and asset export in the web app.
Prior to that, whenever you uploaded a document to a Workspace, the Mac app would first need to render every page, Artboard, and Symbol Source (at 1x and 2x) on your computer. The Mac app would then hash all these images and start uploading them to the web app, so you could view your document in the browser. As you can imagine, uploading your document and a bunch of images every time you hit Save was not a fast process.
For this reason (and many others, which we’ll explore more below), we wanted to take the responsibility of rendering previews of your Sketch document away from the Mac app. The answer was a render farm — a collection of servers that receive your document, process it, and generate the previews you see in the web app, as quickly as possible.
But building a render farm for Sketch wasn’t straightforward. Our rendering engine is built on top of Apple’s technologies, which means it needs to run on macOS servers. Plus, in order to perform effectively, we needed to build an orchestrated system that could render hundreds of documents in parallel.
Why build a render farm?
The very first iteration of Sketch for the web was built to simply help you share your documents with others via a link. However, this was just a starting point. Our real goal was to improve collaboration across Sketch. And to do that, we needed Sketch documents to truly live on the server. This wouldn’t just take the rendering workload off your Mac — it would help the browser understand Sketch documents on a deep level.
This was an important goal for us, and it has allowed us to develop some exciting features. The first was developer handoff in the browser, which gives developers ability to inspect any part of the design, as well as the option to download precisely the assets they need — including flattened Symbols. Now, there’s no need for them to open a Sketch file to figure out how it was built up.
None of this would’ve been possible without the render farm. But these were just the first steps. Ultimately, we wanted to do truly innovative things with Sketch documents on the server — things like real-time collaboration in our native Mac app. And to do it, our server-side systems needed a ‘Sketch brain’.
Building a render farm with Sketch smarts
We wanted our render farm to behave like a ‘black box’. An API takes the input — a Sketch document — and pushes it to the render farm, which turns it into an output — the images and other useful metadata that appear in your web app. Meanwhile, the rest of our services don’t need to know what happened between input and output.
We also wanted our render farm to be a distributed system. The render farm API pushes every document onto the rendering queue, and one of the many Mac servers picks up the task (while running other rendering tasks at the same time). The system monitors the duration of render tasks and load-balances to minimize rendering delays. And as we provision new servers, they fit into the overall system seamlessly.
We wrote the render farm in Go, then deployed the API to AWS and the agent software to a small army of securely hosted Mac minis. Because we’re using Mac hardware, we can run the sketchtool command-line interface — the same one that ships inside every Sketch build. This means our render farm and Mac app use the same rendering engine, so when our engineers add new features and improve performance for the Mac app, the render farm benefits, too.
One of the main reasons we chose Go was because it lets us cross-compile our code. This means we can share functionality between our Linux-hosted render farm API and our macOS-hosted agent processes. Go has also proven to be very performant, and it was easy to take advantage of its clean and simple approach to concurrency. We’re also hoping that Go 1.16 will let us compile our software for macOS on ARM — which would allow us to take advantage of the M1 chip’s major power boost!
How we upload and render documents today
Sketch documents themselves are typically pretty small in terms of file size (unless they include large embedded bitmap images) — even those with a lot of Artboards and Symbols. The document upload to the server is typically near-instant, and thanks to the render farm your computer no longer needs to do any intensive rendering before uploading a Sketch document. Instead, Sketch simply uploads the document itself, and the render farm takes care of the rest.
This high-level process chart explains what happens between the moment you save a document in the Mac app and it being ready to view in the web app.
As we expected, this new process has cut upload times dramatically, and it‘s far more reliable for people with unstable internet connections. The render farm also uses the latest compatible version of our rendering engine for any given document — even if you’re uploading from an older version of Sketch. So you can still take advantage of the web app’s latest features, such as developer handoff and asset exports in their browser.
Over the last year, the render farm has proved itself a solid and reliable service, and continues to help us blur the lines between the Mac app and the web app. This was an important goal for us, as a fast, scaleable render farm is a vital building block of our new real-time collaboration in the Mac app. Our implementation of real-time collaboration will stream your updates to the web, in real-time, giving our server-side systems more control over how they process this granular information. For example, we’ll be able to utilize partial rendering to cut processing times even further, with the aim of updating your document in the web app the moment you hit Save in the Mac app.
This was an important goal for us, as we knew a fast and scalable render farm would be an essential building block of our new big cloud initiative; real-time collaboration in the Mac app for everybody in your team.
Our render farm in numbers
One way of illustrating the success of our render farm is to share some numbers with you!
All figures accurate at the time of writing.
We’re really excited about what the render farm means for the future of Sketch — especially how it will help us improve the Sketch platform. We now know that if we can do it on the Mac, then we can do it on the web too.
This is the latest in our series diving a little deeper into the technical side of Sketch. Is there a particular aspect of the platform that you want to hear more about? Let us know on Twitter.
