Some Light on False Positives and Static Analysis
I’ve noticed my blog has been getting a lot of traffic lately regarding false positives and custom rules for Fortify. Full disclosure: I am a Fortify consultant. The number one complaint I hear about Fortify’s static analysis product (Fortify SCA) is that it produces too many false positives. To understand why this happens, some context is needed on Fortify’s design and purpose.
When the cavemen wrote code, they would use manual code review to look for bugs and security holes. There is a place for manual code review, but its costly and the effectiveness depends on the security education level of the reviewers. Manual code review usually leads to a high false negative rate, especially in complex applications. To make sure everyone is on the same page here, a false positive is where an issue is found that is really not an issue. A false negative is where an issue is missed, leaving a vulnerability in the code.
Fortify SCA was created to help automate the manual code review process. SCA looks through your source code and finds possible vulnerabilities…emphasis on possible. SCA errs on the side of reducing false negatives. The corollary to that is we produce more false positives. We do that because we would rather produce some false positives to avoid missing real issues. We do our best to understand the application, but let’s face it: static analysis is just one algorithm scanning another. There’s no way for ANY static analysis software to not produce false positives.
I also encounter many developers calling real vulnerabilities false positives because they don’t fully understand the finding. I recently received an email from a developer saying SCA found hundreds of persistent cross site scripting issues and that they were all false positives. He was convinced that since they do input validation, trusting the database was not a problem. We talked some about why the database is not a trusted source and he got the big picture.
For the most part, SCA does a decent job of reducing false positives. Many times the reviewers don’t understand the vulnerability or SCA’s criticality is too high for the given scenario. When it comes down to it, SCA is just a way of identifying potential issues. Its up to the reviewers to decide how they want to handle the output.
If you’re having problems with Fortify, send an email to email@example.com. Our support group is fantastic, so give that a go first. If you need help integrating Fortify into your SDLC, contact your Fortify rep to get a consultant on site.