It’s that time again, although only the Framework has been updated, so let’s go over the changes.
Overall, this release should significantly reduce the amount of RAM that is consumed and the percentage of CPU that is being utilized.
Most of the changes fall under this category as the focus of this release has been on CPU and RAM usage optimizations.
In addition to what will be mentioned in following sections, the system has been profiled up the wazoo and caches have been put in place for operations which introduce latencies.
- The default HTTP queue size has been lowered from 500 to 100 requests.
- This will reduce RAM consumption at any given time.
- Automated platform fingerprinting now only takes place for key requests.
- This will significantly reduce CPU utilization.
- A garbage collection is forced before the HTTP requests are performed and after the HTTP responses have been processed.
- This will reduce RAM spikes by ensuring that old objects are collected, before new ones are allocated, during these resource-heavy operations.
- Bodies are now streamed to enforce the maximum response size limit when no Content-Length is available.
- The proxy server’s SSL interception has been updated to dynamically generate a certificate for the requested domain, based on a common Arachni CA.
- Pause operations are now near instant.
- Instead of waiting for the current page to finish auditing, it’ll take place as soon as the active HTTP requests have completed.
- Latency between page audits has been removed. Application of DOM metadata to non-browser-generated pages now happens asynchronously.
- Except for authenticated scans, where a login check and possible re-login needs to occur.
- Updated to extract CDNs from response bodies and whitelist them.
- Limits the amount of logged sinks during data and execution flow tracing.
- Added support for tracking jQuery delegated events.
- #job_timeout — Increased from 15 to 25 seconds.
- Waits for element matching a CSS selector to appear, when visiting a page whose URL matches the given pattern.
- Prior to parsing HTML in search of auditable elements, performs a preliminary text-based check to avoid a full parse if not necessary.
- Enforces a maximum size on the acceptable input values.
- Prevents ever-growing element values from causing excess RAM consumption.
Updated to add proofs to as many issues as possible.
- When the case involves payloads landing in textareas, break out of them to prevent possible FPs.
- Added double-encoded payloads.
- Don’t perform redundant audits.
- Don’t process custom events.
- Updated to handle cases where a button needs to be clicked after filling in the inputs.
- Added progress messages.
- Escalated severity to ‘High’.
- Only perform straight payload injections.
- Escalated severity to ‘High’.
- Updated /etc/passwd content matching pattern.
Newcommon_admin_intefaces — By Brendan Coles.
Updatedbackdoors, backup_directories, backup_files, common_files, directory_listing
- Added MVC frameworks as exempt platforms since they do their own routing.
- restrict_to_dom_state — Restricts the audit to a single page’s DOM state, based on a URL fragment.
- metrics — Captures metrics about multiple aspects of the scan and the web application.
- autologin — Updated to fail gracefully in cases of an invisible form DOM elements.
- Updated to show JSON and XML inputs in the inspection page.
- Added output message with instructions for servers that use SSL.
- vector_feed — Updated to support XML and JSON elements.
Optimized across the board to prefer less resource intensive checks.
- ASP.NET MVC
- Rack — Expanded signatures.
- JSP renamed to Java and expanded signatures.
- PHP — Expanded signatures.
- Python — Expanded signatures.
- Tomcat — Expanded signatures.
Moving away from OSS, introducing the Arachni Public Source License
The title sounds scary, but don’t worry, you’ll still be able to use Arachni as part of your pen-testing toolkit for free.
I jokingly tweeted recently about the amount of time I’ve spent on Arachni since its inception, modifying around 3 million LOC over the course of 8k commits and 5 years (for the Framework alone); however, that makes up pretty much 100% of all code.
The point of OSS is having a community work on a piece of software and make that software available back to the community, with few to no strings attached. Unfortunately, in the case of Arachni that didn’t happen.
Contributions have been virtually non-existent, and when they do rarely occur are usually typo fixes in comments or updates to metadata, like issue descriptions, references and the like, appreciated of course, but a drop in the proverbial ocean.
That’s somewhat understandable when talking about the Framework, it’s a complex system with little room for error. Changes need to happen at the right place, the right way, otherwise they’ll drag the entire system down. It makes sense that people generally trust me to fix bugs and implement new features.
However, I don’t see why the WebUI got the same treatment. It’s basically a bog standard Ruby-on-Rails web application, prime candidate for contributions.
So, it’s time to adapt the license to the reality of the situation — with said reality being that even though the Framework’s code has been (and still is, and will be) open and has been (and still is, and will be) free (for 99.9% of use cases at least), it has never in fact been a community F/OSS project, at least not in any sustainable way.
Arachni is a highly specialized system and I’ve had to re-invent a lot of wheels for it, alas it makes sense that I’d also have to write a custom license, although I was hoping I wouldn’t.
The new license can be found at the License page.
If you’ve got some familiarity with F/OSS licenses, I think I know what you’re thinking right about now:
That was unnecessary, you could have just used the AGPL. People who want to incorporate Arachni into proprietary products would need to acquire a non-free license, people who want to just use it for pen-tests get to use it for free and the code and knowledge would be available to anyone. Plus, it’d still be Open Source Software.
Yes, that’s true, I’d still get the same outcomes as with the new license, however, the problem with the AGPL is the GPL part.
Arachni originally started as GPLv2 and I used to get e-mails from people asking whether I would consider changing to something like BSD or MIT, because the company they work for does not allow the use of GPL software. I even got a book author once asking me the same thing, because he’d rather not include GPL software in the book since companies avoid it.
And that was way back when, very few people had even heard of Arachni at that time.
Many more optimizations
There are many more optimizations planned which will have a massive impact on the overall performance of the system — from HTTP requests to parsing pages to the analysis techniques.
However, those won’t just be for system resources, but network ones as well. You can expect future versions to use massively less bandwidth, which will hopefully result in much quicker scans.
Still, the entire list is quite long, with some optimizations being more radical and game changing than others; so, it may take up to v2.0 to get through all of them. I can tell you this though, at that point Arachni will run like nothing you’ve ever seen before.
As some of you will have probably noticed, the current WebUI has been put out out to pasture for a long time now. It hasn’t received many updates, except for the occasional bugfix.
I’ve got some good news on that front though: a replacement is in the works, and is pretty far along; trust me when I say, the new interface will amaze you by its ease of use, intuitive design, scan management, scan/issue filtering, issue reviewing capabilities and many more.
You will quite literally fall in love with it, I know I have. I’ve been playing with it for no reason, just navigating and filtering and reviewing issues because it is actually an enjoyable experience.
Things are pretty fluid right now, once it gets more mature I’ll post demos and screenshots.
So, that’s about it for now; as usual, you can grab the latest version from the download page.