Google Site Search discontinuation

[Tabita Green]

All,

I just got the notice that Google Site Search is discontinuing its paid service Sept. 2017. Has anybody thought about alternatives?

-Tabita

[Carolyn Zinn]

We at Kalamazoo are using the free CSE.

Carolyn

--
You received this message because you are subscribed to the Google Groups "Reason Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reason-discuss...@googlegroups.com.
To post to this group, send email to reason-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

[Matt Ryan]

We use paid CSE. We use the API and present "sweetened" results from a Reason module, with no ads. We could share what we've got in case anyone wants to see how we're doing it.

Matt

On Thu, Feb 23, 2017 at 2:14 PM, Carolyn Zinn <Caroly...@kzoo.edu> wrote:

We at Kalamazoo are using the free CSE.

Carolyn

 

 

From: reason-discussion@googlegroups.com [mailto:reason-discussion@googlegroups.com] On Behalf Of Tabita Green
Sent: Thursday, February 23, 2017 1:45 PM
To: Reason Discussion <reason-discussion@googlegroups.com>
Subject: [reason-discussion] Google Site Search discontinuation

 

All,

I just got the notice that Google Site Search is discontinuing its paid service Sept. 2017. Has anybody thought about alternatives?

 

-Tabita

--
You received this message because you are subscribed to the Google Groups "Reason Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to reason-discussion@googlegroups.com.

Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Reason Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to reason-discussion@googlegroups.com.

[Tabita Green]

We're using the paid version also. Is that not what they're discontinuing? 

Tabita Green

Director of Web Content 

Luther College

Email: gree...@luther.edu

Office: 563-387-1902

Cell: 563-387-7663

Facebook | Google+ | Twitter | YouTube | LinkedIn

Ask me about Reason CMS, built by and for Higher Ed.

[Matt Ryan]

My understanding is that they are discontinuing CSS. We use CSE, which has two tiers -- free and paid.

Matt

[Tabita Green]

I'm asking support.

[Tabita Green]

I emailed support. Here's what they said:

Your paid GSS subscription "Luther College" expires on August 10, 2017. You will continue to receive support and maintenance through the duration of
your contract i.e August 10, 2017. Please note that this change will not have any impact on your current use of GSS.

Once your search engine expires on August 10, 2017 or exceeds its annual queries before the expiry date, there will be no "Renew/Upgrade" option available to renew/upgrade your Search Engine and your search engine will be downgraded to a Free CSE engine were Ads will be displayed.

When I log in to our custom search dashboard, it goes to "https://cse.google.com/." Is that where you manage your search engine as well, Matt? I think GSS is actually the paid version of CSE.

[David Huyck]

Hi, Tabita -

Before I ramble, here's the link to our search page home:

https://apps.carleton.edu/search/

I did the implementation for Google Custom Search Engine (CSE) early in my time here at Carleton (I'm at 7 months right now, so one might argue I'm still "early in my time at Carleton"...), but I went into the project thinking I was going to implement Google Site Search (GSS).

When I went to do so, all I could get working was CSE, and from the docs it looked like that was the way to get things going before switching up to GSS. Apparently, I stumbled into an internal Google transition process, but the result was that when I went to turn on GSS, there was no way to do so, so I went live with CSE. Rather than use the front-end JavaScripty thing they provide, I am pulling the results as a JSON lump that I unpack and iterate through to display results.

That worked great until our account hit its quota for queries, and started quietly throwing an error, inside which was a URL where I could add payments to turn up our limits. I probably could have figured it out eariler, but I didn't know what to look for.

That story is mostly to illustrate that as much as I dug into the available docs, a lot of it was inaccurate or out of date, so I stumbled about until I found the way to do it "properly."

Cutting to the chase, I'm happy to pass over a cleaned-up version of code that I built to get and then display the results. They are perhaps not riddled with best-practices, especially in the context of Reason, since I was getting to know the codebase at the same time I was implementing CSE.

Some things to note (and maybe this is consistent with GSS?):

• Each query sent to Google returns 10 results, and each query counts against your quota. Here at Carleton, I am caching the result sets so 20 queries for "bookstore hours", for example, only goes to Google once every 6 hours or so. I flush the cache overnight, too, just for housekeeping.
  
• Also, you can only get a max of 10 sets of 10 results, but really, if a person cannot find what they are looking for in 10 pages of search results, their search terms need to be fixed, or the thing they are looking for is probably not there.
  
• ALSO also, sometimes when the initial result set from Google says there are about 130 results, sometimes it really means there are only 84 results, so your link to page 10 of results is going to give you a blank result list. Yay.
  
• We've done some brute-force adjustments to the results using the Promotions (like ads), so when someone searches for our departmental abbreviations or looks for some obscure Carleton abbreviation, the search will still hit an accurate result. Ideally it will return the right things each time (especially if our SEO is all working well), but Google ranking alchemy is imprecise, so this helps us out.
  
• Misspellings or alternate spellings return unpredictable results, so I had to phrase my handling of that bit carefully.
  
• Because of the caching, the analytics in the CSE admin interface are inaccurate (we only ask them for a popular search once every 6 hours, during which time it would be possible for 1000+ queries to only show once in their analytics), so I built a custom hacked-together log thing so we would know what people are actually searching for and how often. I fiddle with that code a lot, so I am tracking a variety of things now: where they searched from, the query they submitted, if the results came from cache or not, whether they were logged in, whether they were on our network (IP address), and after some JavaScript hoop-jumping, we also track the outbound clicks, so we know what they ended up clicking.

I'm sure there is more going on that I'm forgetting or that is too granular for this specific response, but feel free to ask about it. The code will need some love and tidying before I'm ready to release it, but it works really well for the most part, and the upgrade from our previous 2006-era Google appliance was dramatic.

Thanks,

David

[Tabita Green]

Hi David,

Thanks for that! Yes, your new search page looks amazing! We'd love to get your code when you've had a chance to tidy it up. I guess this will be one of our summer projects...

-Tabita

--

You received this message because you are subscribed to the Google Groups "Reason Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discuss...@googlegroups.com.
To post to this group, send email to reason-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

[Carolyn Zinn]

Hi David,

Echoing Tabita – love the integration with the map results and the people directory. Would love to see your code as well!

Carolyn Zinn

Kalamazoo College

[David Huyck]

Cool - I'm glad it looks useful!

The page is set up with a single page type and then the search form, search results, map, and campus directory are four separate modules within the page type. The map and directory are fairly custom to our Carleton network setup, but I can hand-wavy describe what I am doing there.

I will work towards making the code more broadly sharable, and try to document what needs to happen on the Google side of things, too. I don't think I'll be able to get that all going immediately, but I'll try to get it all together this month/next month (meaning March/April - I will be out for a week at Spring Break).

Thanks,

David

--

David Huyck

Senior Web Application Developer

ITS/Web Services Group

Carleton College

One North College Street

Northfield, MN 55057

507-222-4278

http://www.carleton.edu/

I draw in meetings:

"Doodling is a pre-emptive measure to stop you from losing focus."

https://www.ted.com/talks/sunni_brown

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to reason-discussion@googlegroups.com.

Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

--

Tabita Green

Director of Web Content 

Luther College

Email: gree...@luther.edu

Office: 563-387-1902

Cell: 563-387-7663

Facebook | Google+ | Twitter | YouTube | LinkedIn

 

Ask me about Reason CMS, built by and for Higher Ed.

--
You received this message because you are subscribed to the Google Groups "Reason Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to reason-discussion@googlegroups.com.

Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Reason Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to reason-discussion@googlegroups.com.

[David Huyck]

Hi, All -

I am behind on this for a number of reasons, one of which was determining if the JSON/REST interface we use to run our search was going away with the rest of the things (it is NOT going away... yet?) This uncovered a question: it is not clear to me that new accounts can be set to use the JSON API. If you cannot create a new account that has access to the JSON API, my code won't do you any good.

So my mission for you all, if you are still interested, is to find out if you can create a Google Custom Search Engine (CSE) account that has access to the JSON API. The documentation about it is here:

https://developers.google.com/custom-search/json-api/v1/overview

In the "Pricing" section, it still includes links to GSS (which is deprecated), so I'm wary of the current-ness of this information. Regardless, it also information on how to pay for CSE - I missed this in my initial setup, and we had some downtime for a half hour the day of launch while I scrambled to figure out how to pay and where to find a school credit card. You do not need to set up payment to test - you get 100 queries/day for free - but you will quickly exceed that with a live search engine.

Let me know what, if anything, you can uncover. I'm pretty swamped right now, but we can at least figure out whether sharing our code would be useful in the first place.

Thanks,

David

[Tabita Green]

It looks like we can access the JSON API with our current account:

The question is if this will still work once our account expires on Aug. 1, 2017. Worst-case scenario, we may need to switch to the API key for CSE (free version). This forum has a lot of info: https://productforums.google.com/forum/#!topic/customsearch/dXYqNIsfG14, including that non-profits should be able to turn off ads. I'm thinking we'll move ahead with your solution, and then see what happens on Aug. 1.

-Tabita

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discuss...@googlegroups.com.
To post to this group, send email to reason-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

--

Tabita Green

Director of Web Content 

Luther College

Email: gree...@luther.edu

Office: 563-387-1902

Cell: 563-387-7663

Facebook | Google+ | Twitter | YouTube | LinkedIn

 

Ask me about Reason CMS, built by and for Higher Ed.

--
You received this message because you are subscribed to the Google Groups "Reason Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discuss...@googlegroups.com.
To post to this group, send email to reason-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Reason Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discuss...@googlegroups.com.
To post to this group, send email to reason-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

[David Huyck]

Good to know! I'm seeing a hint of light at the end of the tunnel, so I should be able to get this code tidied up soonish. I'll aim for sometime this month, and I'll keep you all posted.

Thanks,

David

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to reason-discussion@googlegroups.com.

Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

[Tabita Green]

Okay, based on the latest email I received from Google on this, I don't think we'll be able to use this solution. "Site Search will continue to function normally for now, but will be downgraded to our free Custom Search Engine product after 2017-06-21, which is ad-supported and doesn't allow XML delivery of results, among other differences." The good news is that we're a non-profit, so we can request to have ads turned off. But it will still be branded as a Google search, as far as I know. We'll see what happens on June 21.

[David Huyck]

You should be able to do a quick, simple test. Put together a query in a browser that looks like this:

https://www.googleapis.com/customsearch/v1?key=[google api key]&cx=[your custom search engine ID]&q=[search term]

If you can do that and get something that looks like actual results, you're in good shape. I just tried it with our credentials, and I got an error because I've set up only specific IPs to have access (so nobody can run up our bill but us), but it was a well-formatted JSON response error. 

You can get the API key from the API manager:

https://console.developers.google.com/apis/credentials

And the CSE ID from your search console:

https://cse.google.com/cse/setup/basic (click the "search engine ID" button)

Try that and see if you can get anything back from the server.

Thanks,

David

To unsubscribe from this group and stop receiving emails from it, send an email to reason-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to reason-discussion@googlegroups.com.

Visit this group at https://groups.google.com/group/reason-discussion.
For more options, visit https://groups.google.com/d/optout.

[Tabita Green]

Hi David,

I wanted to let you know that we were downgraded to Google Custom Search today. We no longer have the XML option. I was able to turn off ads though, so it's really no different than what we had before. So any generalization work can be halted at least from our standpoints.

-Tabita