Go to Top

New engine sneak peek

Hello everybody,

I’d like to share something really special with you today.

I found myself idling a few months ago and wanted to tackle an issue that had been bugging me for a long time in Arachni, performance and resource utilization.

You see, Arachni has vulnerability identification, accuracy and coverage well in hand, but it has to do a lot of work to get those results and even though there are plenty of ways to optimize the system via configuration, I wasn’t completely satisfied with that approach. Any sort of substantial improvement had to come from the insides of the system, the way it approaches each web application, its very essence.

With the above in mind, I started redesigning and rewriting a few things in Arachni, but while I was doing that I ended up changing so much that the result was a new engine.

It looks basically the same from the outside (the REST and RPC APIs, interface options and reports are the same because there was no need to change them) which is good news because when the time comes for the new engine to be released, you won’t have to refamiliarise yourself with it or change any integration code you may have written.

Do not be fooled though from its appearance, because inside lies a different kind of beast. I won’t go into the technical details because they’re boring and I want to keep them as a surprise, but the gist is the following:

  • Every type of resource usage has been massively reduced — CPU, RAM, bandwidth.
  • CPU intensive code has been rewritten and key parts of the system are now 2 to 55 times faster, depending on where you look.
  • The scheduling of all scan operations has been completely redesigned.
  • DOM operations have been massively optimized and require much less time and overall resources.
  • Suspension to disk is now near instant.
    • Previously browser jobs could not be dumped to disk and had to be completed, which could cause large delays depending on the amount of queued jobs.
  • Default configuration is much less aggressive, further reducing the amount of resource usage and web application stress.

Talk is cheap though, so let’s look as some numbers:

http://testhtml5.vulnweb.com
Duration RAM HTTP requests HTTP requests/second Browser jobs Seconds per browser job
New engine 00:02:14 150MB 14,504 113.756 211 1.784
Arachni 00:06:33 210MB 34,109 101.851 524 3.88
Larger real production site (cannot disclose)
Duration RAM HTTP requests HTTP requests/second Browser jobs Seconds per browser job
New engine 00:45:31 617MB 60,024 47.415 9404 2.354
Arachni 12:27:12 1,621MB 123,399 59.516 9180 48.337

As you can see, the impact of the improvements becomes more substantial as the target’s complexity and size increases, especially when it comes to scan duration and RAM usage — and for the production site the new engine consistently yielded better coverage, which is why it performed more browser jobs.

End result:

  • Runs fast on underpowered machines.
  • You can run many more scans at the same time.
  • You can complete scans many times faster than before.
  • If you’re running scans in the “cloud”, it means that it’ll cost you many, many times less than before.

That’s all for now, but I’ll be keeping you updated on my progress.

Cheers,

Tasos L.

, ,

About Tasos Laskos

CEO of Sarosys LLC, founder and lead developer of Arachni.

6 Responses to "New engine sneak peek"

  • Chrisg
    March 10, 2017 - 6:19 am Reply

    Wow, hats off and great job! Really looking testing out the new engine for myself.

  • Vishal Patel
    March 11, 2017 - 11:50 pm Reply

    Great work. Is it out yet?

  • Vladimir Garbuz
    March 12, 2017 - 12:04 am Reply

    Great progress, congrats! Is the new engine already available in the nightlies?

    • Tasos Laskos
      March 12, 2017 - 12:06 am Reply

      No, it’s going to be a while until it’s available.

  • erv
    March 24, 2017 - 4:48 am Reply

    Great Job Mate!! Arachni is a Beast!

Leave a Reply