Go to Top

Help us test Windows support and XML, JSON auditing

Hello folks,

There have been exciting developments for v1.1 and I’d like to ask for your help in testing the new goodies. We’ve got support for a new Ruby interpreter (JRuby), support for a new OS (MS Windows) support for new elements (XML and JSON) as well as some serious performance improvements.

MS Windows support via JRuby

It’s been a long time coming, but native Windows support seems to finally be here — barring any unforeseen show-stoppers, of course.

The Framework, WebUI and their dependencies have been updated to work on Windows, via the JRuby interpreter utilizing the Java Virtual Machine, with very encouraging results.

This feature has been requested for a long time so don’t be shy, help us test it, in hopes of providing an official Arachni Windows package for either the v1.1 release or a follow-up version.

Another cool thing about this is that Java application can now directly integrate with Arachni and Arachni can now directly use Java libraries, both due to JRuby. Don’t know how this may help, but it’s a nice capability. :)

Truthfully, Arachni does indeed run on Windows via the official Ruby interpreter, however there have been serious stability issues, which involve the process dying without any indication of what went wrong. Windows ErrorLevel codes point towards a memory access violation error, but that’s as far as I’ve gotten with that. If you’ve got experience with Windows debugging and the Ruby interpreter please do get in touch.

XML and JSON element support

A natural follow-up to adding real browser analysis is extracting XML and JSON inputs from requests and auditing them like any other element. The Framework can now do this for you.

There’s not much to say here really, auditing these elements is enabled by default and everything is automated.

If you’ve got sites that use XML or JSON in your pentest TODO list please give Arachni a spin and let us know how it does on that front.

Performance improvements

The focus during the development of Framework v1.0 was, of course, the architecture surrounding the browser analysis. It was a conscious decision to provide a clean, scalable design but avoid clever optimizations, and instead introduce them carefully and in small batches.

The sheer complexity of that system mandated that stability and feedback from real-world use-cases be the top priority — I’d rather optimize a working system rather than fix a fast one.

Here’s the first batch of optimizations:

  • An inter-browser, inter-scan, shared, HTTP resource disk cache has been enabled. The first time a browser hits a cacheable resource it will save it to disk and subsequent browser calls (from any browser worker) and subsequent scans won’t have to request it again.
  • The browsers are now smarter and can determine when forms and cookies are actually involved in DOM processing, and thus avoid unnecessary (and costly) DOM audits.
    • This wasn’t that hard for forms, just paying closer attention to their associated DOM events.
    • This was a bit tricky for cookies, as they’re not DOM elements, so the browsers are using their JS data-flow tracing capabilities to determine whether a page is using cookies at each given DOM state snapshot.

All in all, the above will save thousands of HTTP requests (which would be very slow, coming from the browsers) and thousands of browser DOM audit operations (and thus more thousands of HTTP requests).

The cache business was quite frankly an oversight, it should have been used from the start, my bad.

Take it for a spin

Now, down to business, as there are no packages yet, setting up Arachni can be a bit of a hustle, but clear instructions are available for both the Framework and the WebUI — the WebUI will automatically pull in the Framework, so you don’t need to setup both.

So, take it for a spin and let me know how it works. :)

Cheers

, , , , , , , , , , , , ,

About Tasos Laskos

CEO of Sarosys LLC, founder and lead developer of Arachni.

11 Responses to "Help us test Windows support and XML, JSON auditing"

  • Dhaya KS
    June 3, 2015 - 11:10 am Reply

    Hi I dont see a windows installation instructions can you pls help

    • Tasos Laskos
      June 3, 2015 - 1:44 pm Reply

      Hello,

      I’ve had to pull the Windows instructions as there was an issues when running on the platform.
      I’ll be coming back to that in the future though.

      Cheers

      • Dan Farrell
        June 15, 2015 - 7:53 pm Reply

        We would definitely be interested in that when you’re ready to test again – if you need a windows hosting provider to test with, we’re happy to be it. :)

        • Tasos Laskos
          June 17, 2015 - 8:48 am Reply

          Thanks man, waiting on a dependency to fix a Ruby 2.2.2 Windows issue and then I’ll start testing again.
          Ideally I’d like to do this without using JRuby, so that may give me the chance.

          • Dan Farrell
            June 17, 2015 - 9:49 pm Reply

            Excellent! When you’re ready let us know… I’ve set this thread to inform me of a new post so I’ll check back here for that :)

  • name
    October 31, 2015 - 8:18 pm Reply

    Any news on windows support? any news on instructions? It’s been a while.. :)

    • Tasos Laskos
      November 1, 2015 - 12:10 pm Reply

      No good news I’m afraid, newer versions of JRuby introduced incompatibilities which I don’t have time to hunt down at the moment.
      And, again. due to time constraints, I’m not sure how the current version of the official Ruby interpreter works on Windows, older ones crashed without providing any info.

    • Tasos Laskos
      November 1, 2015 - 4:04 pm Reply

      As it happened, I did have some time today and since you mentioned it I took another look at the situation.

      The thing that caused the biggest issue on JRuby turned out to be the same thing that caused the official interpreter to crash on Windows, and the fix was quite simple.

      Initial tests show the Framework working reliably on MS Windows using the official Ruby interpreter.

      I haven’t tested the WebUI yet but thus far results look very promising, if something real comes out of this I’ll write a new blog post with the details.

    • name
      November 4, 2015 - 2:46 am Reply

      Thanks! Way to go! Really appreciate you decided to dedicate some time to it! I will finally have the opportunity to test Arachni and possibly spread out the word about it !!! I love when something as little as a comment can trigger big advances :) What was the error you were experiencing with Ruby / JRuby on Windows? How did you fix it? Now that you fixed it, are you going to swich back to JRuby (for faster performace)? Again, thanks!

      • Tasos Laskos
        November 4, 2015 - 3:25 am Reply

        Thank you for reminding me. :)

        On JRuby its effect was quite benign, causing mixed up HTTP requests/responses, so I figured the JRuby bug would be easier to fix.
        That was the only reason I was considering JRuby as Arachni can’t benefit from its support for real threads, because it takes an alternative approach — it generally prefers non-blocking I/O instead of threads.

        On MRI on Windows it was an outright crash of the interpreter, with the process exiting with an %errorlevel% pointing to a memory access violation error with no other info. So I was completely stumped.

        But, while I was revisiting the JRuby issue after your comment, I made a change that fixed it.
        Then I figured I’d see if that had any effect for MRI on Windows, and the crash went away.

        Must have been C pointers getting mixed-up/shared between threads.

        MRI has a GVL and on Linux it must be harsher than on Windows, so the pointer mix-up thing didn’t happen.
        JRuby has no lock and real threads so the pointer resulted in objects getting mixed up.
        On MRI on Windows though the GVL must have not been enforced in time, pointer got shared or something, Windows killed the process.

        Mind you, I’m just guessing about the technical stuff based on the behaviours of different interpreters, I didn’t do any low level debugging.

        See: https://github.com/typhoeus/typhoeus/issues/449

        Typhoeus is the global Arachni HTTP client and it’s awesome, so I figured I’d use it for the Selenium->PhantomJS communications as well, which by default used Net::HTTP.

        However, Selenium uses threads to make those requests in parallel, and that caused this whole mess.
        Turns out all I had to do was just use the default HTTP client for Selenium and the issue went away.

Leave a Reply