Google 'Search Engine Rankings' App in Groovy

Posted by Stan Carney on Oct 4, 2011 in Programming, World Wide Web | 0 comments

I wanted to see where certain sites rank for a given search term. Normally this is accomplished by heading to Google, entering your search query and potentially paging through N pages using ctrl + f until I find the site I’m looking for. Obviously there is a better way. Searching around I found only questionable sites offering the service so I figured it wouldn’t be that hard to build.

Google.groovy:

@Grapes([ @Grab(group = 'org.codehaus.groovy.modules.http-builder', module = 'http-builder', version = '0.5.1'), @Grab(group = 'org.ccil.cowan.tagsoup', module = 'tagsoup', version = '0.9.7') ]) term = "beefeater" pattern = ~/.*furaffinity.net.*/ google = "www.google.com" //incase you wanted to use country specific googles. i.e. www.google.co.uk //this is pulled from the PREF cookie. googlePrefCookieValue = "ID=7e4fa65beb903a0b:U=f9cff12ac43a7208:FF=0:LD=en:NR=100:TM=1317785603:LM=1317785629:SG=2:S=2TdkM_GcD8NBPuyp" http = new groovyx.net.http.HTTPBuilder("http://${google}") cookie = new org.apache.http.impl.cookie.BasicClientCookie('PREF', googlePrefCookieValue) cookie["domain"] = google cookie["path"] = '/' http.client.cookieStore.addCookie cookie position = 0 http.request(groovyx.net.http.Method.GET, groovyx.net.http.ContentType.TEXT) { req -> uri.path = '/search' uri.query = [q: term] headers.'User-Agent' = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:6.0.2) Gecko/20100101 Firefox/6.0.2" response.success = { resp, reader -> assert resp.statusLine.statusCode == 200 html = new XmlSlurper(new org.ccil.cowan.tagsoup.Parser()).parse(reader) results = html.body.'**'.findAll { e1 -> e1.name() == 'li' && e1.'@class' == 'g' && e1.@id == '' } for (line in results) { position++ match = line.'**'.find { e2 -> e2.@class.text().startsWith('l') && e2.name() == 'a' && pattern.matcher(e2.'@href'.text()).matches() }?.@href if (match) { println match println position break } } } }

It simply posts the search criteria to Google and loops through the results to find the matching URL. If the provided ‘pattern’ is found it dumps out the position and the URL found. If it doesn’t find anything it exits without displaying anything. The PREF cookie is obtained by using your favorite web developer tools such as Chrome or FireBug. Before you grab the value though you need to visit http://www.google.com/preferences and disable Google Instant and increase the number of results if you are interested in more than just the first 10. I like using Google Instant so I did this in a Chrome Incognito window so my preferences are left untouched. I also wanted to see what the general public saw. If I ran the script as myself my +1’s etc… would skew the results to show me what I like.

The app takes a bit of time to run the first time as Grape has to download the dependencies, TagSoup and HttpBuilder. Even subsequent runs aren’t as fast as you would think they should be. I’m guessing it is Grape checking to make sure all the dependencies exist locally.

UPDATE: It appears that sometime after I started to play with this stuff Google released a new version of Analytics that has the ability to associate a Webmaster Tools property to an Analytics property under Traffic Sources –> Search Engine Optimization –> Queries. It allows you to cross reference search queries used to arrive at your site and Country. You are still shown the average position but it has a pretty graph.

Appears to have been added on September 8th/2011: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=1120006

Stan Carney

Interesting Tech

Google ‘Search Engine Rankings’ App in Groovy

Google.groovy:

Post a Reply Cancel reply

About Me

Recent Posts

Recent Comments

Archives

Categories

Sites