By: Xin Liu, Giles Lavelle, Lucas Boyer, Tom Genoni
Email is a crucial channel for us to connect customers and service professionals. When customers submit project requests with requirements such as the type of the job, location, schedule, etc. on our platform, we send emails to matched service professionals with those request details. When service professionals reply with job cost estimates, we send the messages to customers through emails. Above are only two examples out of more than 100 types of emails we have in the product. Most of them were coded in PHP Twig templates within our main website application. The deep coupling with both the language and the data generation logic in website make it very hard for us to bring our modern design styles to emails, send emails from other languages and systems, and preview and test emails.
In order to unleash the power of email, we leverage Foundation for Emails to write responsive templates, Handlebars to populate dynamic data without complex logic, and decouple frontend templates with backend data generation. This has enabled designers and frontend engineers to preview emails and iterate on designs quickly. At the same time, we can send emails from any languages and services by providing only template name and required data in JSON format. Almost all our emails have been migrated to this new system at the time of writing. Based on our A/B testing results, many of them have higher click through rates due to better design and faster iterations enabled by the new system.
Architecture
The following graph shows a high level view of the new email templates system. The email templates live in their own Git repository. They pull in the styles from another shared CSS styles repository, allowing them to use styles shared with the rest of the Thumbtack product. The templates repository has its own deployment process, which takes advantage of Jenkins to deploy builds to Amazon S3. Given the relatively small number of templates we have and potentially large number of emails we need to send, we retrieve all templates from Amazon S3 and cache them in our backend email delivery service. Finally, with the templates in the cache and the JSON data provided by clients, we render HTML emails accordingly and send them to SendGrid, the email service provider we use.
Frontend: Email Templates Repository
All our email templates exist in their own repo. We keep this repo very minimal—it contains the email templates themselves, some dummy test data for each, and a few tools to aid development and deployment. We have avoided implementing any complex functionality here (for instance, custom Handlebars helpers) in order to ensure that our templates remain simple and easy to reason about.
Tooling
It is always a good idea to know what an email will look like before sending it, so we keep a set of tools in our frontend repo to help us visualize our templates. The primary tool for this is a local preview server powered by Browsersync which populates every email with dummy data. This lets us quickly see how a new email will look and lets us browse through our existing emails for design inspiration. Unfortunately, simply previewing an email in your web browser isn’t good enough. Anyone developing emails will quickly realize that not all email clients are created equally. Some clients use modern renderers like WebKit or Gecko, which have first-rate support for modern web features. Others just use Microsoft Word. Thus, it is crucial to visualize your email in a wide range of clients. We have two tools for this. One lets you send a local email template to any email address, allowing you to view it inside a real email client. The second uploads an email to Litmus, a powerful tool for previewing an email in many different clients. Once we’re happy with how the email looks, the final tool will let us upload a local email file to our shared development environment, so that engineers can test triggering and rendering the email using actual data.
Versioning
We take advantage of versioning to help with testing and to enable quick deploys and rollbacks. Every new build of our repo constitutes a new version, which is made available on Amazon S3 as soon as it is built. The currently active version and most recent versions are specified in a special version file. Clients of this system regularly poll the version file to determine if they should switch to a newer version. Thus, rolling back to a previous version is as simple as updating this version file. Furthermore, our dev, staging, and prod environments can be set to different versions, allowing us to test new emails in one environment before deploying to another.
Email CSS Kit
Due to the large number of email clients and their notoriously poor support for modern web standards, building HTML emails requires coding with nested tables and inline styles, error-prone techniques most developers haven’t used in a decade. Fortunately a number of frameworks have emerged that abstract away the ugly complexities, among them Zurb’s Foundation for Emails. By restricting our email designs to the limited number of styling and layout options it provides, along with minimal Thumbtack-specific styling, we ensure our emails have the highest chance of being delivered and rendering correctly on all our customers’ email clients.
Backend: Email Templates Management
We have built an email delivery service on top of SendGrid. Previously, all clients (in different languages) needed to have their own email rendering engine and every email had to be transferred over the wire. In our new design, the email Delivery service API only requires two fields: the template name and the corresponding data. Based on the template name, the delivery service retrieves the corresponding template from the cached template store and populates dynamic fields in the template using the provided data in JSON format. After template partials assembling, CSS inlining, and final rendering, emails are ready to be sent to SendGrid. Note that in this process, we have minimized the data transfer because only dynamic data (instead of full HTMLs) need to be transferred and templates are cached.
Caching
As we have hundreds of email templates, we cache all of them in memory for fast retrieval. Given a new email request, we compare the current build version in memory with the one in remote build storage (Amazon S3) and only load templates when the versions mismatch. If the templates loading is successful, we update the current version and templates in memory accordingly. Otherwise, the next request will trigger the cache update again.
This design of email templates backend is agnostic to any specific template. It has been serving most of our email templates flawlessly without much updating since launch.
Future Work
The new email templates system described in this document has been widely adopted by our engineering teams. The original design goal of easy designing, writing, and previewing emails have been achieved. However, due to limited engineering resources back then, our design did leave certain limitations we would like to address in the future:
- Frontend and backend used different languages (Node.js and Golang, respectively) and libraries to render emails. This led to potential style inconsistencies in previewed emails from frontend and delivered emails from backend. Moving forward, we plan to use one language to consolidate the email rending into one component.
- The capability of previewing emails has been well received. However, previews are limited to email bodies. We plan to use an iframe, which enables us to show subject line and preheader text around it. We also plan to enhance the preview functionalities to include searching/filtering, sending test emails directly from preview UI, collecting test results from Litmus and SendGrid automatically, etc. Stay tuned!
If you are excited about solving real-world problems in the local services domain, join us and help millions of customers get things done!