By ICSI researchers Christian Kreibich, Nicholas Weaver and Vern Paxson, with Peter Eckersley.

UPDATE, 8/25/11: There are a couple of revisions to this post which are marked inline below, and explained further here.

Earlier this year, two research papers reported the observation of strange phenomena in the Domain Name System (DNS) at several US ISPs. On these ISPs' networks, some or all traffic to major search engines, including Bing, Yahoo! and (sometimes) Google, is being directed to mysterious third party proxies.

A report in New Scientist today documents that the traffic is being rerouted through a company called Paxfire. This blog post, coauthored with one of the teams that discovered the phenomenon, will explain the situation in more detail.

Who is rerouting this search traffic?

The published research papers did not identify the controller of the proxy servers that were receiving the traffic, but parallel investigations by the ICSI Networking Group and EFF have since revealed a company called Paxfire as the main actor behind this interception. Paxfire's privacy policy says that it may retain copies of users' "queries", a vague term that could be construed to mean either the domain names that they look up or the searches they conduct, or both. The redirections mostly occur transparently to the user and few if any of the affected ISP customers are likely to have ever heard of Paxfire, let alone consented to this collection rerouting and occasional logging and alteration of their communications with search engines.

The proxies in question are operated either directly by Paxfire, or by the ISPs using web proxies provided by Paxfire. Major users of the Paxfire system include Cavalier, Cogent, Frontier, Fuse, DirecPC, RCN, and Wide Open West. Charter also used Paxfire in the past, but appears to have discontinued this practice.

Why do they do this?

In short, the purpose appears to be monetization of users' searches. ICSI Networking's investigation has revealed that Paxfire's HTTP proxies selectively siphon search requests out of the proxied traffic flows and redirect them through one or more affiliate marketing programs, presumably resulting in commission payments to Paxfire and the ISPs involved. The affiliate programs involved include Commission Junction, the Google Affiliate Network, LinkShare, and Ask.com. When looking up brand names such as "apple", "dell", "groupon", and "wsj", the affiliate programs direct the queries to the corresponding brands' websites or to search assistance pages instead of providing the intended search engine results page.

What can I do about it?

If you want to know if the network you're currently on is subject to this type of traffic redirection, you can run a Netalyzr test. And the best protection against the privacy and security risks created by this type of hijacking is to visit sites using HTTPS rather than HTTP, which can easily be achieved using EFF's HTTPS Everywhere Firefox extension.

More technical details below...

A detailed explanation

For most users of the World Wide Web, visiting a website equals clicking on a link to the site or entering the site's name into their browser, and receiving the corresponding page from the site. Users generally assume that the site's name is identical to the site itself, and essentially trust the site's authenticity if it looks as usual and the browser does not pop up phishing warnings or other signs of trouble. Paxfire's misdirection of search traffic undermines this trust.

The ICSI Networking group develops and operates the ICSI Netalyzr, a tool that tests the characteristics of users' Internet connections. Netalyzr's measurements show that approximately a dozen US Internet Service Providers (ISPs), including DirecPC, Frontier, Hughes, and Wide Open West, deliberately and with no visible indication route thousands of users' entire web search traffic via Paxfire's web proxies.

To explain these redirections further, we first need to delve into the workings of the Internet a bit. Since the Internet does not route traffic to names but to network addresses, contacting a website involves translating the site's name (say "www.google.com") to the IP address (say 74.125.224.49) of a computer that runs Google's web server. It is to this address that the browser actually sends its request. The Domain Name System (DNS) is in charge of facilitating this mapping of names to addresses. It is the Internet's equivalent of telephone books.

Usually, ISPs provide DNS servers (directory assistance, essentially) for their users. When a user's computer asks to map a name to an IP address, the user's system contacts the ISP's DNS server, which looks up the correct IP address for the name and returns it to the user. As currently implemented, this process does not provide any guaranteed correctness. In essence, users must trust their ISP's DNS servers to correctly return IP addresses that indeed belong to the site the user intends to visit. In some instances, however, this trust may not be warranted.

For a while now, a number of ISPs have worked in cooperation with Paxfire and similar businesses like Barefruit and Golog to profit from mistakes that users make when typing names into their browsers. Paxfire provides a product for ISPs that rewrites DNS errors (effectively conveying "the name you asked for doesn't exist") to responses sending users to search pages that host advertisements, for which Paxfire then shares the corresponding ad-related revenue with the ISPs. This practice has already been controversial.

Rerouting of requests to and responses from search engines

Paxfire's product also includes an optional, unadvertised, and more alarming feature that drastically expands Paxfire's window into users' traffic. Instead of activating only upon error, this product redirects the customers' entire web search traffic destined for Yahoo!, Bing, and sometimes Google, to a small number of separate web traffic proxies.

These proxies collect receive, examine and process all search terms and results, but only log a small subset of search queries that were entered into a browser search box and are related to major trademark holders, the users' web searches and the corresponding search results, mostly forwarding them the rest to and from the intended search engines. This allows Paxfire and/or the ISPs to directly monitor all searches made by the ISPs' customers Paxfire's code to examine the queries and responses, selecting out those that are of relevance to its business. and build up corresponding profiles, a process on which Paxfire holds a patent. It also puts Paxfire in a position to modify the underlying traffic if it decides to.

Under specific conditions, the Paxfire proxies do not merely relay traffic to and from the search engines. When the user initiates searches for specific keywords from the browser's URL bar or search bar, the proxy no longer relays the query to the intended search engine, but instead redirects the browser's request through affiliate networks, as the equivalent of a click on advertisements. Using the names of popular websites, we have so far identified 170 brand-related keywords that trigger redirections via affiliate programs and result either on the brands' sites or on search assistance pages unrelated to the intended search engine results page.

The subset of customers affected varies from temporally localized deployments to apparently entire customer bases. The DNS-based redirection operates in a surgical fashion, affecting only search engines but not other services such as Google Maps or Yahoo! Mail, and remains completely invisible to the user. The treatment of Google queries varies. Charter and Cogent appear to redirect only Bing and Yahoo, while DirecPC, Frontier and Wide Open West also used to redirect Google to Paxfire proxies located within their own networks. Google has recently put significant pressure (see the answer to the question) on the ISPs to get them to stop redirecting Google searches. As of August 2011, all major ISPs involved have stopped proxying Google, but they still proxy Yahoo and Bing.