Late last month, Verizon Digital Media Services (VDMS) introduced waflz, its new global CDN Web Application Firewall (WAF). A WAF acts as the first line of defense in network security by monitoring all incoming and outgoing network traffic, and deciding which traffic should be blocked and which let through typically based on a set of predefined security rules. With the right tuning, a WAF can more quickly and robustly protect websites and applications against various attacks than patching is able to.
There is often a trade-off between security and performance when it comes to WAFs. In the case of a global CDN at the scale of VDMS, which has 3,000+ global interconnections that guarantee “100% uptime SLA”, it was critical that WAF functionality did not impede performance. With the introduction of a WAF engine, VDMS wasn’t prepared to let latency be compromised.
ModSecurity and its Challenges at CDN Scale
On first introducing a WAF engine to the VDMS stack three years ago, the VDMS engineers initially worked with ModSecurity Rules Engine, the popular open source WAF. Verizon says it found ModSecurity to be “first-rate for individual WAF use cases” and enjoyed its support of the OWASP Core Rule Set, along with its powerful rule language and API access to the HTTP traffic stream in real time.
As the number of its customers using the WAF went up, however, VDMS found that latency issues were indeed beginning to occur. When the dense ruleset of ModSecurity was multiplied across every single customer, CPU and memory utilization went up across its network, which increased operational costs. VDMS also found the testing and developing of new rules was often challenging, with the writing and parsing of rulesets being clunky and leading to a slow response to new security threats.
VDMS initiated a series of modifications and workarounds, but ultimately decided to build its own WAF from the ground up in order to protect the global scale of its CDN without compromising performance. Its new WAF is called waflz, which VDMS says “rewrites the standards for WAF performance and scale” and “processes requests 3x faster than the previous version”. It also “improves target accuracy to more effectively detect and mitigate the latest threats”. For VDMS, this is a major priority. Since 2016, cyberthreats have become more pervasive, more varied and more costly, and using its WAF to protect its customers against the latest threats is critical.
VDMS’ new WAF: waflz
waflz is the new WAF that VDMS developed internally (it has been running on the VDMS network for over a year) and has now been released as an open source WAF engine, published under the Apache 2.0 license. waflz is a high performance WAF engine that is native to the VDMS technology stack and can operate at massive scale while also enabling highly efficient granular testing and customization of rules. It still offers a subset of ModSecurity capabilities, the OWASP Core Rule Sets 2.x and 3.x in addition to many third party rule sets. This allows waflz to support thousands of different client configurations.
The new WAF is a sizeable improvement than the previous solution for several reasons:
- It uses less memory (the global ruleset map is shared across all VDMS customer configs and doesn’t have to be spun up separately each time);
- It can operate at the global scale of VDMS;
- It offers significantly better performance;
- It is API-driven;
- It offers efficient testing of ModSecurity rule sets and allows for easy customization of rules.
How waflz was designed
Optimizing Performance
In the post announcing the new WAF, Reed Morrison, Software Developer at VDMS, writes, “waflz is designed from the ground up to deliver high performance and multitenancy”. As the CDN has chosen to prioritize performance, there are times in which latency has taken precedence over flexibility. Therefore, waflz supports a restricted amount of ModSecurity functionality, as some of its directives, such as SecRemoteRules and inspectFile, were considered not suitable for running on the edge network.
At the same time, there are some aspects of waflz that ModSecurity is not capable of, such as dual WAF mode, which supports the testing and tuning of new WAF profiles for emerging and current threats without any gaps in service.
waflz runs in C/C+ and was designed to be API-first. The new WAF is also configurable with JSON rules as both inputs and outputs (one of the VDMS team’s goals was to improve the representation of rule language in code).
One of the fundamental components to the waflz architecture is its use of Google Protocol buffers internally to represent configuration (including rules) and alert formats. Protocol buffers are Google’s mechanism for serializing and deserializing structural data, built to create a smaller and faster version of XML. Protocol buffers allow waflz to be interoperable with JSON and ModSecurity, in addition to adding strictly typed schemas for inputs and outputs in both. waflz offers a specific conversion tool between JSON, Protocol buffers and ModSecurity.
Enabling Multitenancy
In addition to performance, the other main engineering goal behind waflz as Morrison writes, was multitenancy, and “dealing with the high concurrency and multitenancy that comes from serving thousands of customers”. Each of VDMS’ global edge servers needs to be capable of processing a request for any customer as quickly as possible. In addition, edge server applications need to offer real-time patching and processing for any type of customer configuration.
With ModSecurity, when a request reaches a server, it has to serve that request based on a customer’s configuration. The principle resource issue with this method when you are running at scale is that too many customer configurations loading the same three or four WAF rule set definitions into a server’s process memory can overload it.
VDMS set up waflz so that rule sets only need to be loaded once in memory and the waflz global rule set map is shared internally on a read-only basis across all its customer configs, meaning no copying is required. This lets VDMS continue to handle the customization of each customer’s rules modifications, but the method to do so in waflz is more straightforward and less resource intensive. waflz can also do this for more complicated rule modifications such as SecRuleUpdateTargetById by deduping rulesets loaded previously, saving process memory in production.
Other performance optimizations were also identified and tweaked, including:
- Aho-Corasick: For operators such as PM/PMFROMFILE (multiple substring matching like “grep -F/fgrep”), an Aho-Corasick data structure is designed for quicker parallel searching of substrings. Our construction is similar to the acmp object in the standard implementation but “more space efficient, as it prunes node meta information”. Search performance is similar as the tree is traversed in a similar way in both implementations.
- IP Tree: VDMS has had an internal IP Tree in its internal repos, that is faster than the msc_tree in the standard implementation and reusable outside of its library.
- XPath: For “XML:<path>” targets in the rules to mitigate the performance overhead of recalculating the same expression in the rules during request processing, VDMS built in XPath cache-ing.
Rules Testing and Customization
An additional benefit to waflz is the fact it easily enables the testing of one-off ModSecurity rules, which removes the need for a more sophisticated test harness. It also has built-in features to enable the validation of engine behavior.
When writing a new feature for waflz, there is no need to bring up an entire web server to test; you just mark the connectors and parse in all the HTTP request components needed to inspect it. waflz has a lightweight standalone server to test rules. You can rapidly prototype a WAF rule (built-in or custom-made) on the standalone waflz engine without having to run it in a production environment or set up a CDN & WAF configuration first. waflz can also be run in proxy mode, which allows you to run any website through the waflz server on your local machine. This will let you do rapid prototyping of rules/rulesets on any production website.
As waflz was built to be interoperable, customers can create their own custom rules for their unique application requirements. They can do so from the OWASP rule set, other similar directives, JSON rules and VDMS’ own offerings. This allows VDMS to provide a comprehensive and unique application security ruleset that adapts quickly to emerging threats in cyberspace.
Overall Improvements
The overall result of the VDMS WAF is higher performance and fewer server overloads because HTTP requests are run through just one configuration. This means that waflz can run at CDN scale as part of VDMS’ overall technology stack. It offers the side benefit of highly efficient granular rule testing and customization. VDMS highlights the fact it has added to the ModSecurity community and contributed back to its development as an open source WAF rules engine.
waflz has now been in operation for over a year at VDMS and supported and protected thousands of different client configurations “without any loss in performance”. It is exciting to see what happens next with its development, both internally at VDMS and beyond.