Build a Site Search with the Google Search API

Published on Wednesday 16th of November, 2005

Google provides several APIs, including one for their web based search, another for their desktop toolbar, and one for their Adwords program. Here we look at creating a custom site search with PHP and the search API.

A quick note before getting into the meat of the article. If you happened to make it thru last week's lengthy instalment, you should be well prepared for this article. I'm going to do my best to make this one a bit more concise, so that readers can get to playing with the API. Apologies if it seems that I gloss over anything. Code is available for downloading at the end of the article.

The formalities: Getting started with the Google APIs

Develop Your Own Applications Using Google is the homepage for Google's APIs. From this page you can get started by registering for an API Key, downloading their developers kit with some example code, and head over to their terms of service and some help and Faqs (their is also a Google Group).

Visions of Dollar Signs, Dancing

Can I develop commercial applications using Google Web APIs?

You can develop any application you want, but you must abide by the Google Web APIs terms of service. One condition is you cannot create a commercial service using Google Web APIs without first obtaining written consent from Google. Another is that you can only create one account for your personal use.

The Google Search API

Much like our look at the Yahoo! Search API, we will be focusing on the Search API to build a site search.

For their main search API, however, Google offers us doGoogleSearch(), doGetCachedPage() and doSpellingSuggestion() - these allow you to do what you would expect:

1. Given Search terms, return results
2. Given a web page, return the cache file
3. Given a word it returns spelling suggestions

Getting creative, you could allow a user to do a search query, then fetch the results and the resulting cache pages, take an automated screenshot of the cached pages (trimming out the Google cache info from the top of the page) and provide a set of linked images with the search results.

Steps to building our site search

Much like the last instalment, we will be taking input, in the form of search terms from a user, and building a request that we will send to the API server. We will then take the response, unserialize it and format the results into some html.

Google uses the SOAP protocol (specs, W3Schools, Wikipedia) for sending data between it's API server and your application. PHP 5 can handle SOAP natively, however for PHP 4 we need to use an external library to send, receive, and unserialize our communications. As you can see, this is much the same process as with the REST powered Yahoo! example from last week.

There are many SOAP classes available for use, and since we'll be using PHP (4) in this example, I've chosen to use NuSOAP. You may also want to check out the PEAR SOAP class.

Note: if you are playing with nusoap on PHP 5, it will throw an error as the nusoap class has the same name as the SOAP client in PHP 5: soapclient.

Step 1: Request and receive

The first thing we need to do is build a request to send to Google. We will do this by setting our search parameters in an array:

1. // Build an array with the parameters we want to use:
2. $params = array(
3. 'key' => 'yourGoogleAPIKeyHere',
4. 'q' => 'Search Terms Here',
5. 'start' => 0,
6. 'maxResults' => 10,
7. 'filter' => true,
8. );
9.  
10. Download this code

Site search

In order to make a site search, a couple of things need to take place.

1. A form that passes the search terms must be used, and when submitted those terms must be passed into the value for q in our array above.
2. As Google doesn't provide a parameter for a site search, a value of site:www.yoursite.com must be embedded into the value for q.

Taking into account those two points, the line for q would be 'q' => 'site:www.yoursite.com search terms'.

Moving forward: request and receive in one easy step

This next part moves quite fast, condensing a few of the steps done in last weeks Yahoo! example into simply a few lines of code. Rather than opening a file with PHP, we will be passing the url and the parameters directly to the nusoap class:

1. // include the class:
2. include('nusoap.php');
3. //
4. // instantiate a new soap client:
5. $soapclient = new soapclient("http://api.google.com/search/beta2");
6. //
7. // send the query off to the server, with our
8. // parameters and using the 'doGoogleSearch' method
9. $searchresults = $soapclient->call("doGoogleSearch", $params,
10. "urn:GoogleSearch", "urn:GoogleSearch");
11.  
12. Download this code

First we include the class, then we instantiate a new 'soapclient', passing it the URI for the API server. From this point we can call the server as outlined in the example.

Nusoap returns the data from the server to us in the form of an array. Compared to the example from last week, this was certainly much simpler to get from the request to an array of data (granted, nusoap is composed of a lot of lines of code, and it did all of the heavy lifting).

A look at the data

If you were to print_r($searchresults) at this point, you would see something similar to the following:

1. Array
2. (
3. [directoryCategories] => Array
4. (
5. )
6. [documentFiltering] =>
7. [endIndex] => 2
8. [estimateIsExact] =>
9. [estimatedTotalResultsCount] => 190
10. [resultElements] => Array
11. (
12. [0] => Array
13. (
14. [URL] => http://www.fiftyfoureleven.com/sandbox/weblog/2004/jun/the-definitive-css-gzip-method/
15. [cachedSize] => 18k
16. [directoryCategory] => Array
17. (
18. [fullViewableName] =>
19. [specialEncoding] =>
20. )
21. [directoryTitle] =>
22. [hostName] =>
23. [relatedInformationPresent] => 1
24. [snippet] => This post is the source for the most definitive/recent/tested version of gzipping
25. your CSS.
26. [summary] =>
27. [title] => The Definitive Post on Gzipping your CSS
28. )
29. [1] => Array
30. (
31. [URL] => http://www.fiftyfoureleven.com/weblog/web-development/css/applied-css-management-and-optimization
32. [cachedSize] => 44k
33. [directoryCategory] => Array
34. (
35. [fullViewableName] =>
36. [specialEncoding] =>
37. )
38. [directoryTitle] =>
39. [hostName] =>
40. [relatedInformationPresent] => 1
41. [snippet] => Building on the previous discussion about managing CSS files, this post looks at
42. the practical solutions in use to help offset the results of some of the ...
43. [summary] =>
44. [title] => Applied CSS Management and Optimization
45. )
46. )
47. [searchComments] =>
48. [searchQuery] => site:www.fiftyfoureleven.com css
49. [searchTime] => 0.093227
50. [searchTips] =>
51. [startIndex] => 1
52. )
53. Download this code

Looking at the array above, we can see that the total number of search results can be taken from $searchresults[estimatedTotalResultsCount], and that our results are held in $searchresults[resultElements]. Each of those elements holds a result, with the associated title, url, file size and s snippet from the page. Further down, you can find the search query and the start index, among some other details.

Step 3: Presenting the results

Now that the data is held in an array, it will be quite easy to loop thru the $searchresults[resultElements] and present the data however you see fit. I won't elaborate an example here, but in the downloadable code at the bottom an example is given.

A short discussion

This is the second search API that has been examined, and we can see some similarities despite the use of REST for Yahoo! and SOAP for Google. The data transfer is done with XML - apart from the initial request when using Yahoo! - and we have unserialized the data to arrays in both instances.

Personally, I prefer to move the data to an array in this case, because after some "array normalization", I could pass either the Yahoo! or Google results array to the same HTML processing function.

Not only would this be an efficient setup if I decide to change the format later, but in this manner I could provide three separate sets of results (Yahoo! + GOOG + MSN) in the site search for a site. Or, I could develop a search website (gada.be anyone?) and have one single function deal with processing the data into a view.

One last note, pagination is possible here just like Yahoo!, and with Google it may be more of a necessity as you can only have 10 results returned at a time.

Download some code!

Here is an example script that pulls this whole article together. When using, remember:

1. To change the extension to php.
2. That you need a server with php installed to run it.
3. That you will need a Google Application Key to use it.

That's it!

As usual, questions or comments invited below, and stay tuned for the MSN search API - which will be similar work to this one - coming up later this week. Next week we move away from search APIs...

Bookmark this link at:
del.icio.us new digg blinklist newsvine