IsMobileDevice -- Mobile Devices and Search Engine Spiders

[Brad Griffey]

We are using the IsMobileDevice property to automatically redirect mobile
devices to our customized mobile site, while keeping standard web browser
traffic flowing into our conventional website. The problem we have is
search engine spiders. The code inside the Machine.Config that is added via
the MMIT is setting the IsMobileDevice property to TRUE for all devices
except Mozilla. Spiders such as Google are not Mozilla, so when they
attempt to index our site, they get redirected to the mobile site instead.

Has anyone encountered a similar problem, or have ideas on how best to
resolve this?

Thanks.

[Mike Moore [MS]]

Hi Brad,

Though I don't have a ready-made solution for you, I can point you in the
right direction. Use the information below to compile a list of "user
agent" strings that you wish to identify. Then you change your
machine.config file so that it uses the list you have compiled.

BACKGROUND INFO
When a web device (such as a browser, or a automated paged indexer)
requests a page from a server, it includes a "user agent" string in the
request. This string, called HTTP_USER_AGENT, identifies that the request
is coming from Internet Explorer 5 or Netscape Communicator 6 or .... there
are a huge number of possibilities.

The machine.config file contains a section named "browserCaps". This
section contains code which attempts to parse or match the text in the user
agent string to identify the browser. This section is designed to identify
browsers used by end users (such as IE or Netscape). With MMIT installed,
it also performs some minimal testing to identify mobile devices. To a
large degree, the logic simply marks the user agent as being from a mobile
device if it was not able to otherwise identify the user agent.

SOLUTION
1) Compile a list of user agent strings
The latest browscap.ini file is available at
http://www.cyscape.com/browscap/.
http://www.jafsoft.com/searchengines/webbots.html
http://www.robotstxt.org/wc/active/html/index.html
http://www.siteware.ch/webresources/useragents/
I found the above by searching the web. Some good search strings to do
further searches are:
"user agent" "browscap.ini" "http_user_agent"

2) Update the browserCaps section of machine.config
Look at the syntax of what is already in this section, then expand on it
with your new list.

---
Do you have any more questions on this issue?

Thank you, Mike Moore
Microsoft, ASP.NET

This posting is provided "AS IS", with no warranties, and confers no rights.

--------------------
>From: "Brad Griffey" <bgri...@dollar.com>
>Subject: IsMobileDevice -- Mobile Devices and Search Engine Spiders
>Date: Thu, 21 Nov 2002 17:40:57 -0600
>Lines: 14
>X-Priority: 3
>X-MSMail-Priority: Normal
>X-Newsreader: Microsoft Outlook Express 6.00.2800.1106
>X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
>Message-ID: <u7w3vdbkCHA.2472@tkmsftngp08>
>Newsgroups: microsoft.public.dotnet.framework.aspnet.mobile
>NNTP-Posting-Host: 208.2.126.4
>Path: cpmsftngxa08!tkmsftngp01!tkmsftngp08
>Xref: cpmsftngxa08 microsoft.public.dotnet.framework.aspnet.mobile:4985
>X-Tomcat-NG: microsoft.public.dotnet.framework.aspnet.mobile

[Edward J. Stembler]

I have a similar need in my application to detect if it is
running on a PDA or via the Web. Here's what I use:

Private Function isPDA() As Boolean

Dim mc As MobileCapabilities = Request.Browser
Dim op As String = ""

Dim isMyPalm As Boolean = mc.HasCapability
("isMyPalm", op)
Dim isPocketIE As Boolean = mc.HasCapability
("isPocketIE", op)
Dim isIE As Boolean = mc.HasCapability("isIE", op)
Dim isBlazer As Boolean = mc.HasCapability
("isBlazer", op)
Dim isOpera As Boolean = mc.HasCapability
("isOpera", op)

Return (isMyPalm Or isPocketIE Or isIE Or isBlazer
Or isOpera)

End Function

Corresponding device filters needed in your Web.config:

<filter name="isMyPalm" compare="Browser"
argument="MyPalm" />
<filter name="isPocketIE" compare="Browser"
argument="Pocket IE" />
<filter name="isIE" compare="Browser" argument="IE" />
<filter name="isOpera" compare="Browser"
argument="Opera" />
<filter name="isBlazer" compare="Browser"
argument="Handspring Blazer" />

>.
>

[Edward J. Stembler]

What I failed to mention in my previous post is that I
used the isPDA function mainly to differenciate between
PDA and cell phone users. The particular feature which
needed this allowed it to be used on the web & pda, but
not cell phones. So, the function probably should've been
name isPDAOrWeb. In any event, you get the idea. You
could just remove the IE and Opera lines. Incidentially,
I added Opera support because I heard that Opera 7
supported Smartphones and PDAs with it's small-screen
rendering. Later, I found out that by PDA they really
meant devices which run on Symbian OS, which is mainly a
cell phone OS. MyPalm and Blazer are WAP browsers which
run on Palm Pilots. PocketIE runs on Pocket PCs.

>.
>

[Bill Ehrreich]

I'm having the same problem. Spiders are getting redirected to the
mobile site. Did Mike Moore's suggested solution work for you?

Bill Ehrreich

*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

[Brad Griffey]

I guess the concern I have with this fix is I'm tracking spiders based on
their existing user-agent strings (or some other means of identifying them).
I know that new spiders are created all the time, and existing search
engines upgrade their existing spiders. I'm just worried that my user-agent
list within my machine.config will need to be routinely maintained.

One of the methods I used prior to our site going .NET was to check the
pixel height and width of the calling device (if it was less than 640x480 we
assumed it was a mobile device). I'm assuming spiders would return 0x0 or
null for this, which would still be a good way to detect them and flag them
as non-mobile. Would this be a viable solution as well? At least it would
prevent me from having to maintain a user-agent list in my machine.config.

Thanks for your feedback on this.

"Mike Moore [MS]" <mic...@online.microsoft.com> wrote in message
news:iUeBu3nkCHA.2528@cpmsftngxa08...

[Mike Moore [MS]]

Hi Brad,

I'm looking into this. I hope to have something for you tomorrow or
Thursday.

Thank you, Mike Moore
Microsoft, ASP.NET

This posting is provided "AS IS", with no warranties, and confers no rights.

--------------------
>From: "Brad Griffey" <bgri...@dollar.com>

>References: <u7w3vdbkCHA.2472@tkmsftngp08> <iUeBu3nkCHA.2528@cpmsftngxa08>
>Subject: Re: IsMobileDevice -- Mobile Devices and Search Engine Spiders
>Date: Mon, 9 Dec 2002 08:12:15 -0600
>Lines: 101

>X-Priority: 3
>X-MSMail-Priority: Normal
>X-Newsreader: Microsoft Outlook Express 6.00.2800.1106
>X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106

>Message-ID: <ObH82z4nCHA.1448@TK2MSFTNGP11>
>Newsgroups: microsoft.public.dotnet.framework.aspnet.mobile
>NNTP-Posting-Host: mail.dollar.com 208.2.126.4
>Path: cpmsftngxa06!cpmsftngxa10!tkmsftngp01!TK2MSFTNGP11
>Xref: cpmsftngxa06 microsoft.public.dotnet.framework.aspnet.mobile:4993
>X-Tomcat-NG: microsoft.public.dotnet.framework.aspnet.mobile

[Mike Moore [MS]]

Hi Brad,

QUESTION
IsMobileDevice is not accurate. How can I get a better determination of
which type of device sent a given request?

The current method for testing if an incoming request is from a mobile
device is IsMobileDevice. This tests for specifically known devices and
then classifies all requests as mobile if "mozilla" is not included in the
user agent string. However, there are many other user agent strings in use
today which are not mobile devices. This includes "spiders" which search
engines use to index web sites. If spiders get classified as mobile
devices, then they get directed to the sites mobile section. The result is
that the regular section of the site does not get indexed by the search
engine.

ANSWER
I agree with your idea of using the pixel height/width. I will research
this further and let you know what I find.

Regarding the height/width. This is dependant upon the device sending its
screen height/width as part of the request. Here are the related headers
for a request from a Pocket PC (it also had a couple other headers which
were not of interest for this topic):
UA-OS: Windows CE (POCKET PC) - Version 3 0
UA-color: color16
UA-pixels: 240x320
UA-CPU: ARM SA1110
UA-Voice: FALSE
UA-Language: JavaScript
User-Agent: Mozilla/2.0 (compatible; MSIE 3.02; Windows CE; PPC; 240x320)

All of these are different from the headers sent by IE. Three are of
particular interest: UA-OS, UA-pixels, and User-Agent. Parsing these
header strings can reveal that this is a Pocket PC and that it's
height/width is 240x320.

Since IE does not send height/width information, I expect that spiders will
not send height/width information either.

I previously wrote about compiling a list of user agent strings and
updating your machine.config file. Then the IsMobileDevice would use the
new information and it would be more accurate. For those interested in
seeing this reply, go to:
http://www.google.com/groups?safe=images&ie=
UTF-8&oe=UTF-8&as_umsgid=iUeBu3nkCHA.2528
@cpmsftngxa08&lr=&hl=en

Again, I will look into this further and post my findings. This will
probably take several days to track down.

[Mike Moore [MS]]

Hi Brad,

I'm still looking for more information on this. So far, your own idea of
using window height/width, along with IsMobileDevice, is looking like it
may be the best solution we can offer at this time. I am still looking into
this and will keep you posted.

Thank you, Mike Moore
Microsoft, ASP.NET

This posting is provided "AS IS", with no warranties, and confers no rights.

--------------------
>From: "Brad Griffey" <bgri...@dollar.com>

>References: <u7w3vdbkCHA.2472@tkmsftngp08> <iUeBu3nkCHA.2528@cpmsftngxa08>
>Subject: Re: IsMobileDevice -- Mobile Devices and Search Engine Spiders
>Date: Mon, 9 Dec 2002 08:12:15 -0600
>Lines: 101

>X-Priority: 3
>X-MSMail-Priority: Normal
>X-Newsreader: Microsoft Outlook Express 6.00.2800.1106
>X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106

>Message-ID: <ObH82z4nCHA.1448@TK2MSFTNGP11>
>Newsgroups: microsoft.public.dotnet.framework.aspnet.mobile
>NNTP-Posting-Host: mail.dollar.com 208.2.126.4
>Path: cpmsftngxa06!cpmsftngxa10!tkmsftngp01!TK2MSFTNGP11
>Xref: cpmsftngxa06 microsoft.public.dotnet.framework.aspnet.mobile:4993
>X-Tomcat-NG: microsoft.public.dotnet.framework.aspnet.mobile
>

[Brad Griffey]

Mike,

Thanks for your help with this. We had to implement a fix last week, so we
went with the list of user-agent strings in the machine.config file.
Although I would like to return to the pixel height-width solution at some
point (which would require less maintenance), I wanted to let you know the
user-agent solution is working for us, and spiders are now being properly
identified as non-mobile devices.

Thanks.

""Mike Moore [MS]"" <mic...@online.microsoft.com> wrote in message

news:acAtQqvoCHA.3224@cpmsftngxa06...

[Mike Moore [MS]]

Hi Brad,

QUESTION
The IsMobileDevice property is sometimes inadequate. It identifies some
incoming requests as mobile which are more accurately described as
"unknown". In particular, some search engines of interest to us are using
User-Agent-Strings which are not currently recognized by the
System.Web.Mobile.MobileCapabilities object. Currently, the
MobileCapabilities object sets IsMobileDevice to True for unrecognized
requests. The result is that requests from unrecognized search engines are
directed to the mobile section of our web site. Then the main-stream
section of our site does not get indexed by the search engine.

The requests sent by search engines and other automated queries are often
called "spiders" or "crawlers". Many of the larger search engines use a
User-Agent-String containing the word "mozilla" which identifies them as a
desktop type browser. However, some search engines use other strings and
even change their strings occasionally.

ANSWER
I am glad to hear that you have been successful in setting up your own list
of user-agent-strings.

I'm still looking into this. Third party user-agent-strings are not
specifically supported. This means that the .NET Framework does not promise
to recognize them. However, I am hoping I can find something for you that
is easier to work with over time than for you to maintain your own list of
user-agent-strings.

So far, I've found that the height/width is only useful if your list of
user-agent-strings is fairly up to date with regard to mobile devices. I
found that the framework does a good job of identifying known mobile
devices and setting height/width information for them. Therefore, a device
with IsMobileDevice set to True and no height/width is probably an
UN-recognized request. So, if the browsercaps section of your
machine.config file contains an up-to-date list of known mobile devices,
then most requests that truly are mobile requests should have both
IsMobileDevice set to True and have some height/width other than zero or
null.

Again, I'm still looking for a better solution. I don't know if I will
succeed. And, this may take a few more days.

[Mike Moore [MS]]

Hi Brad,

QUESTION
The IsMobileDevice property is sometimes inadequate. It identifies some
incoming requests as mobile which are more accurately described as
"unknown". In particular, some search engines of interest to us are using
User-Agent-Strings which are not currently recognized by the
System.Web.Mobile.MobileCapabilities object. Currently, the
MobileCapabilities object sets IsMobileDevice to True for unrecognized
requests. The result is that requests from unrecognized search engines are
directed to the mobile section of our web site. Then the main-stream
section of our site does not get indexed by the search engine.

The requests sent by search engines and other automated queries are often
called "spiders" or "crawlers". Many of the larger search engines use a
User-Agent-String containing the word "mozilla" which identifies them as a
desktop type browser. However, some search engines use other strings and
even change their strings occasionally.

ANSWER
I am glad to hear that you have been successful in setting up your own list
of user-agent-strings.

I looked into this and I had some co-workers help me. Our conclusion is
that we don't have an ideal solution for you. Using height/width is not
perfect. A superior option to height/width is to change the default value
in machine.config for the IsMobileDevice property from True to False. With
the value False, all devices not specifically recognized will be marked as
NOT mobile.

Using height/width actually turns out to be very similar to doing just
this. Except, that using height/width could mark devices as being NOT
mobile when they did get recognized as a known mobile type, but one which
does not have height/width specified.

That's the "easy" option. As you can guess, this leaves an opening for
genuine mobile devices that are not recognized being directed to your HTML
site rather than your mobile site. So, the easy solution forces you to
choose whether unrecognized devices are always HTML or always mobile.

The more accurate solution is the same as given previously. Do some
research to find the user-agent-strings that are important to you and add
them to your machine.config file. Below is a repeat of that answer.

----
Here is a repeat of the previous post.

[Mike Moore [MS]]

Hi Brad,

I was just looking at my last response. I tried to make it legible to new
comers to the thread. Now, I think I buried my answer to you.

The answer is that we don't have a lot more to offer. Earlier, you
suggested using height/width in addition to IsMobileDevice. I found
something that is superior to that, but it's still just an incremental
improvement.

The height/width only gets set if the machine.config browsercaps section
recognizes the device within the mobile section. So, height/width is a
subset of recognized mobile devices. It you change the default value of
IsMobileDevice from true to false, then you will have a better solution
compared to using height/width with IsMobileDevice. Unfortunately, that's
all I have to offer at this time (aside from the advice about finding and
adding your own list of user-agent-strings).

What that leaves, is to either leave the default for IsMobileDevice set to
true and risk redirecting non-mobile devices to your mobile site. Or,
setting it to false and risk sending mobile devices to your HTTP site.

Clearly, it's better if you can add your own user-agent-strings.

Thank you, Mike Moore
Microsoft, ASP.NET

This posting is provided "AS IS", with no warranties, and confers no rights.

--------------------
>From: "Brad Griffey" <bgri...@dollar.com>
>References: <u7w3vdbkCHA.2472@tkmsftngp08> <iUeBu3nkCHA.2528@cpmsftngxa08>

<ObH82z4nCHA.1448@TK2MSFTNGP11> <acAtQqvoCHA.3224@cpmsftngxa06>

>Subject: Re: IsMobileDevice -- Mobile Devices and Search Engine Spiders

>Date: Mon, 16 Dec 2002 08:06:03 -0600
>Lines: 168

>X-Priority: 3
>X-MSMail-Priority: Normal
>X-Newsreader: Microsoft Outlook Express 6.00.2800.1106
>X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106

>Message-ID: <eva4AxQpCHA.2432@TK2MSFTNGP12>

>Newsgroups: microsoft.public.dotnet.framework.aspnet.mobile
>NNTP-Posting-Host: mail.dollar.com 208.2.126.4

>Path: cpmsftngxa06!tkmsftngp01!TK2MSFTNGP08!TK2MSFTNGP12
>Xref: cpmsftngxa06 microsoft.public.dotnet.framework.aspnet.mobile:5028
>X-Tomcat-NG: microsoft.public.dotnet.framework.aspnet.mobile