At Aqute over the years, we have written various programs to perform competitive intelligence tasks based on querying websites, including search engine APIs. The most common involve extracting search queries in a particular format, or performing the same search query for, say, each of the Fortune 500 and analysing the results.
One of the tools we created ourselves was a way of reading the HTML code (rather than the human-visible content) of web pages and extracting competitive intelligence from that. In ye olde days, it was half-possible to do this kind of thing on Google et al, but that has gotten more difficult.
The most common application of this kind of programming is also available through a third party website, builtwith.com. Here you can enter a target website, and BuiltWith will tell you what technologies that website is using. So, for example, B&Q's website in the UK uses ChannelAdvisor, Omniture SiteCatalyst and Google Analytics as its tracking tools.
The more interesting use of BuiltWith is to find out which websites are using a particular technology. This is a common question among software companies, who want to know their competitors' customer base. So, for example, BuiltWith provides details of 85,000 websites using DoubleClick and 86,000 websites using Akamai.
BuiltWith is useful for front-facing technologies, i.e. those that require some code to be written into the HTML. It works with a subset of websites, but a large subset - for example listing 650,000 AdSense websites. Overall, BuiltWith is a very useful tool.