NSS Labs, founded in 1991 is one of the most respected information security research companies in the world. Besides publishing research reports, they offer a first class testing environment where products from various security vendors are tested. They currently have 19 senior level security analysts on staff with the majority of them having 10+ years of experience working in the security industry. Their analysts come from all walks of life including Gartner, Fortinet, 451 Research, Trustwave and so on.
The intellectual brain power at NSS Labs is mind blogging. The last thing you ever want to do at a cocktail party is ask them a question about malware, because their answer will probably make your head spin. Yesterday, analysts Thomas Skybakmoen and Jason Pappalexis published a comparative analysis called “Security Value Map for Breach Detection” comparing security platforms from six well-known companies: AhnLabs, Fedelis, FireEye, Fortinet, Sourcefire and Trend Micro.
NSS Labs Background
- Started in 1999
- Raised: Undisclosed from LiveOak Venture Partners
- Service Offering: Leading Information Security Research and Testing Lab of security products
- LinkedIn Employee Count: 60
- Senior Level Security Analyst on Staff: 19
- Executives: Vikram Phatak (CEO) and Bob Walder (COO)
- HQ: Austin Texas
FireEye vs NSS Labs
NSS Labs evaluated the products on a wide ranges of metrics, including security effectiveness, performance, the ability to detect malware, TCO, and so on. NSS Labs claims to be the first company to conduct a test of this kind. After the report was published, FireEye’s stock drop from $65/share to $55/share that wiped out $800M+ in market valuation. The initial thought that comes to mind, “are you serious”. First of all, the reports that NSS Labs publishes are informative, educational, interesting, well thought out, and offer a unique perspective.
However, NSS Lab reports are not the Final word, but only a solid data point that is to be used with other data points when making a decision. Secondly, can you really test an enterprise class, comprehensive security platform in a lab? Testing the performance and TCO of a Cisco router vs a Juniper router is doable. Testing and comparing Oracle RDMS to IBM DB2 is more difficult. Testing and comparing Akamai against EdgeCast is really difficult. Testing and comparing comprehensive security platforms from different vendors is just not possible in a lab, no matter how many PHD’s you have on staff.
How can NSS Labs create a test environment that mimics the real world? The real world has thousands of computers, spread across dozens of locations around the world, dozens of business applications on the backend, more applications on the front end, hundreds or thousands of smart phones and tablets from different manufacturers, and on and on.
There are literally thousands of moving parts internally and externally to every large enterprise. How can you mimic that in a lab? And then you have the hackers, from the rookies to the mathematicians, who are creating thousands of malware variants and zero-day attacks. It’s kind of a shame that NSS Labs created something out of nothing, and in the process detonated $800M in market valuation in FireEyes stock. FireEye is fuming. $800M drop in valuation is no laughing matter.
FireEye, welcome to Wall Street, the land of the cowboys where misinformation can wipe out months of hard work in hours. Lesson learned for FireEye – next time before NSS Labs publishes the edition of its report that you decline to participate in, make sure you alert the entire press before hand, stating your objections loudly to the test, pointing out the flaws of their testing methodology, and neutralizing their arguments. Wall Street hates bad surprises.