Can scrapy be used to extract elements generated by javascript code?

[Hugh Jass]

This is the site I am trying to extract from:

https://www.oanda.com/forex-trading/analysis/margin-calculator

The goal is to scrape the 'Margin Used' value for a given set of inputs (Account Currency, Currency Pair, Margin Ratio, Number of Units).

I thought that the value is computed via a XHR request but when I examined the XHR tab in Chrome Developer Tools, I do not see any requests being generated. After examining the JS tab, I get a feeling that the computations are done using javascript on the client side. I'm not a web developer so I may be wrong.

So, how is this site performing the computation and can scrapy be used to scrape it or should I consider using another tool?

[David Fishburn]

Yes, it is using JavaScript.

Here is the function is calls:

var marginAvailableCalculator = Class.create(FXMathBase, {
    initialize: function($super) {
        $super($A(["accountCurrencies", "currencyPairs", "customRateBase", "marginRatios", "unitAmount"]), $A(["marginUsed"]))
    },
    marginUsed: function() {
        return this.createInput("Margin Used", "marginUsed", {
            listenTo: "accountCurrencies"
        })
    },
    calculate: function() {
        if (this.startCalc() && this.checkFieldErrors()) {
            var a = "(customRateBase*unitAmount)/marginRatios";
            this.results.get("marginUsed").node.value = this.round(this.evaluateFormula(a), 2)
        }
    }
});

I found that in the debugger, under www.oanda.com\wandacache\fxmatch-*

If you wanted to capture that you would have to use something that supports JavaScript.

I use selenium / PhantomJS and Scrapy together in order to do this.

David

[Hugh Jass]

Hi David,

Thank you for your answer.

You mentioned that you used selenium and Scrapy together to capture the information.

I know how to accomplish this using selenium but I don't see how Scrapy can be of any help.

Could you please elaborate more on the roles played by selenium and Scrapy for this case?