Crawl coverage and accurate vulnerability detection are the two most important characteristics of a scanner. A combination of good web application resource coverage with good vulnerability detection (without false positives) makes for the perfect scanner, and this is what we strive for.
A common way to test and compare the crawl and vulnerability detection capabilities of today’s scanners is via the WAVSEP benchmark. This page provides up-to-date scores for the latest Arachni version so that you can make an informed decision when it comes to selecting the right scanner.
Crawl coverage scores represent the efficacy of a scanner when it comes to interacting with web applications and finding as many resources to audit as possible. Simply put, a high crawl coverage means that a scanner can perform a thorough audit and not miss possibly vulnerable resources.
The coverage benchmark uses WIVETv3 and the scores can be found at: http://sectoolmarket.com/wivet-score-unified-list.html
Arachni has a score of 96%, which places it at the top of the scoreboard (the missed cases are due to lack of support for SWF).
Security is an important matter and we would encourage you not to take our obviously biased word for it, hence, you can use the following instructions to reproduce our results.
You need to:
- Visit WIVET with your browser and pass its “PHPSESSD” cookie to Arachni, to force it to maintain a single session. That will also allow you to easily inspect its progress via the “Current run” page.
- Load a check that submits forms (in this case it’s the “trainer”), so that Arachni can train from the web app’s behaviour and pass the wizard test.
- Exclude the logout link, so that Arachni won’t nullify its session.
Run a scan against the WIVETv3 application with:
./bin/arachni http://192.168.1.7/wivet/ --checks trainer \
--audit-links --audit-forms --scope-exclude-pattern=logout \
The other options are there to force Arachni to only submit links and forms and skip cookies, which would have been enabled by default, but that doesn’t affect the results; it just makes the process faster and makes the progress output less hectic.
Vulnerability detection scores represent the ability of a scanner to detect different types and permutations of vulnerabilities, as well as the accuracy of those results when dealing with pitfalls that commonly cause false positives.
Vulnerability detection and accuracy scores can be found at: http://sectoolmarket.com/price-and-feature-comparison-of-web-application-scanners-unified-list.html
- SQL injection: 100% (0% false positives)
- Reflected XSS: 90.91% (0% false positives) — Misses cases which require support for the now obsolete VBScript language.
- Local file inclusion: 100% (0% false positives)
- Remote file inclusion: 100% (0% false positives)
- Unvalidated redirect: 100% (0% false positives)
- Backup files: 100% (0% false positives)
These results stem from the fact that Arachni learns from the web application’s behavior and intelligently adapts to each web application resource, leading to great vulnerability coverage and accuracy — amongst other things.
Again, don’t take our word for it, here is how the benchmark was performed:
- No “special” optimizations were enabled for these tests, they were performed using the default settings.
- Only applicable checks were loaded for each test category.
- The tests were not run per case, but rather per category.
- Official WAVSEP benchmarks are performed per-case, to help scanners return consistent results. The WAVSEP benchmark is rather limited in its design to handle concurrent DB connections and when stressed can make scanners miss cases or return false-positives.
- Arachni doesn’t need that sort of pampering, it is designed to handle bad network conditions and broken web applications (up to a point), since that’s what you’ll face in real-life.
As you can see, in an effort to be fair, we have actually put Arachni at a disadvantage during these tests; although, even Arachni has its limits, so if you try to do the same your mileage may vary.
If you get different scores you may have found a bug, please let us know and we will fix it ASAP.