Login to Office365 OWA

[Duncan Hepple]

Hi All,

I have been tasked with monitoring our Office 365 OWA application by logging in with a test user and clicking on some links after login.

I originally set out to use Webinject (http://www.webinject.org/) to do this but soon discovered that I needed to POST a value that was created by Javascript - which is something that Webinject cannot process.  Looking for alternatives, I soon discovered phantomjs and casperjs, and started writing a script for logging into the application and clicking on a few links.  This all started coming together very well and for the first couple of attempts this worked well.  And then suddenly, I am getting "PhantomJS has crashed" prompts.   

Nothing in my environment has changed - it's almost as if Microsoft have detected the use of a headless browser from their end and stopped my script from working.  I then took the same script file and used on a different machine with the same version of phantomjs and casperjs - I saw the same pattern... it worked perfectly well for first couple of log ins and then "Phantomjs has crashed"..  Has anyone come across anything like this?  I am not trying to create a DDOS against Microsoft's Online Services, we simply want to create a monitoring check for Outlook Web App logins so we can see if our O365 tenancy is affected by a service outage (as these are always very vague and don't always apply to us).  Are there any tips to avoid being detected as a headless browser? 

Any help would be greatly appreciated.

Cheers

Duncan

[Eloi Tay]

Are you doing this in a infinite loop? If it is in a loop, if you restart your script does it work again? To my knowledge phantomjs seems to leak a little, I used to have a script that scrapes website by following links but it keep growing and growing in memory usage before it blows up. So maybe you can check if that is what is happening first. If you are only calling it once from a scheduler and this is happening, use something like charles to look at the http packet being sent back and fore, look at the latest return that cause it to crash maybe something messed up on OWA.

My experience in using OWA is it works pretty terrible in chrome which phantomjs is sitting on. In that case maybe you can change to slimerjs which is based off firefox's engine.

[Richard Wilde]

You can also try messing about with the user agent string to see if problem persists.  Casper allows you to set this as an option

Sent from my Windows Phone

---

From: Duncan Hepple
Sent: ‎19/‎03/‎2015 22:52
To: casp...@googlegroups.com
Subject: [casperjs] Login to Office365 OWA

--
CasperJS homepage & documentation: http://casperjs.org/
CasperJS @github: https://github.com/n1k0/casperjs
 
You received this message because you are subscribed to the Google Groups "casperjs" group.
Visit this group at http://groups.google.com/group/casperjs?hl=en.
---
You received this message because you are subscribed to the Google Groups "CasperJS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to casperjs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Duncan Hepple]

Thanks Eloi - I'm not running it in a loop.  But because we are using a federated login, it includes a lot of redirects so perhaps they are not liking the quick succession of requests using a headless browser.  I never really considered using SlimerJS before - so I'll give this a go

Cheers

[Duncan Hepple]

Hi Richard,

I have tried using a more up-to-date useragent, but unfortunately get the same problem.  This is the verbose output I see when it crashes:-

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "MSISAuthenticated=MjAvMDMvMjAxNSAxdfdfMDoxMzowMg==; secure; HttpOnly; domain=

sts.consoso.com; path=/adfs/ls"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "MSISLoopDetectionCookie=MjAxNS0wMy0yMDdffdoxMDoxMzowMlpcMQ==; secure; HttpOnl

y; domain=sts.contoso.com; path=/adfs/ls"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "MSPRequ=lt=1426846382&co=1&id=N; domain=login.microsoftonline.com; path=/

"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "PPAuth=AXVKIt*XsBKRVMqiGx*eu7CQOH731czGciITNjV7vVKtelIGR9WcsWkNgSmx2LCIGc

4np5LPk9N0jpYDpOgpbwGtWBl5tts78vVWICQF31sl25j6mCfWWOzYGQ!Fe0zNhsfsdfsfsdDTNrGZ3j*I61foSVCXGSTimDnOr!2J78SAVnr8C6He6VtQGE76mPjSgS6

XhttAnlxtUv4JtPiGYMOsetvLhgIPawZror*aB3ienTa3wUmNo3diZI7WiElOnUP7Fz2R9*YcS8PC5fjN0uOSwI8i2NuhnHUNX; secure; HttpOnly; do

main=.login.microsoftonline.com; path=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "PPLState=1; domain=.microsoftonline.com; path=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "clntrealm=contoso.com; HttpOnly; domain=.login.microsoftonline.com; path

=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "MSPShared=1; HttpOnly; expires=Wed, 30-Dec-2037 16:00:00 GMT; domain=.log

in.microsoftonline.com; path=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "MSPVis=$260563; domain=.login.microsoftonline.com; path=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "LOpt=0; domain=.login.microsoftonline.com; path=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "MSPSoftVis=@72127960634622307@:@; domain=.login.microsoftonline.com; path

=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "MSPBack=0; domain=.login.microsoftonline.com; path=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "x-ms-gateway-slice=orgidprod; secure; HttpOnly; domain=login.microsoftonl

ine.com; path=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "stsservicecookie=orgidprod; secure; HttpOnly; domain=login.microsoftonlin

e.com; path=/"

2015-03-20T10:13:02 [DEBUG] CookieJar - Saved "ClientId=FJSW1XWKOREPT2UKBLVG; secure; HttpOnly; expires=Sat, 19-Mar-2016

 10:13:02 GMT; domain=outlook.office365.com; path=/"

Resource received:https://outlook.office365.com/owa/?exsvurl=1&wa=wsignin1.0

Count = 1

2015-03-20T10:13:02 [DEBUG] Network - Resource request error: 299 ( "Error downloading https://outlook.office365.com/owa

/?exsvurl=1&wa=wsignin1.0 - server replied: Login Timeout" ) URL: "https://outlook.office365.com/owa/?exsvurl=1&wa=wsign

in1.0"

Resource received:https://outlook.office365.com/owa/?exsvurl=1&wa=wsignin1.0

Count = 1

2015-03-20T10:13:02 [DEBUG] WebPage - updateLoadingProgress: 50

2015-03-20T10:13:02 [DEBUG] WebPage - setupFrame ""

2015-03-20T10:13:02 [DEBUG] WebPage - updateLoadingProgress: 100

2015-03-20T10:13:02 [DEBUG] WebPage - updateLoadingProgress: 10

PhantomJS has crashed. Please read the crash reporting guide at

<http://phantomjs.org/crash-reporting.html> and file a bug report at

<https://github.com/ariya/phantomjs/issues/new>.

Please attach the crash dump file:

  C:\Users\bob\AppData\Local\Temp\7e27592f-132a-4533-8c7b-385cd7097cb6.dmp

I'm almost certain that I'm being blocked by Microsoft now - I think they are keeping a log of who uses a headless browser.  Bizarrely, I changed my IP address and still had the same issue, so not quite sure what the identifier is.  Dammit

Cheers

Duncan

[Eloi Tay]

Hi 

If you can perhaps upload the script that you use, I could run it on my end to see if it is just affecting your company's office 365. I seriously doubt that they can detect a headless browser that support javascript like phantomjs. Since in theory it works exactly like the browser except it does not render on screen but instead render it offscreen. I believe it is some other issue.

On Friday, March 20, 2015 at 6:52:14 AM UTC+8, Duncan Hepple wrote:

[Patrick Gormley]

Duncan,

I have been tasked with the same. Could you share your script?

[Ken]

On the question on avoiding detection as automated browser, check this detail guide - https://franciskim.co/dont-need-no-stinking-api-web-scraping-2016-beyond/

Although I'll respect the website owner intention and don't automate on a website if it does not want to serve automated / headless browsers.