Ajit Balakrishnan’s Blog

It was late night at the end of a long, long day at work. Even the lights in the fishing boats anchored in Bombay harbour for the night were out. I was fishing in another ocean that night, trawling the internet for any research paper that would provide us with a method to detect India-related content on the web.

There is a brute force way to do that - employ hundreds to check every major website in the world… But every time the content changed you'd have to go back and redo the work. Anyway, hiring that many people would cost too much.  There had to be a more elegant way. For example, write a computer program to patiently check sites throughout the world and send back snippets of information that was of interest to Indians.

We understood the first steps in doing this. For instance you can easily get a computer program to tag an article as 'India interest' if it encounters the word "Gandhi" in an article. Generalizing from this rule you could make the program compare every word it encounters against a list of Indian pronouns to detect 'India interest content'. Even more generally you could make the program compare words it encounters with a corpus known to be relevant to Indians, for example the last five years of Business Standard articles.

This much we had figured out on our own. The trick was how to do this economically. Here again the brute force method is to crawl every single English language site in the world and look for words to compare with our chosen corpus. But the elegant way would be to devise a method of inspired guessing as to where to look for first and where to look for next.

That night I was trawling the internet for research papers that described methods of inspired guessing.

Here was one! Accelerated Focused Crawling through

Online Relevance Feedback.  I skimmed through the paper; it pretty much dealt with the problem I had in mind.

            It was past midnight in India, a good time to call the author and ask if he was willing to consult for us, I said to myself.

I ran my eyes back up to the start of the paper to check which American University the paper had came out of. Then came the shock! The research paper was from our own backyard- IIT Bombay. And the lead author was Soumen Chakravarty, a Professor of Computer Science there. I was astounded. Anything to do with web crawling was hot; that's the stuff Search Engines are built on. There was a person from India publishing on such a hot topic in a prestigious international journal?

            I couldn't wait for the sun to rise to call him. The next day, I and some colleagues trudged to IIT Bombay to meet the professor. He was sitting at his computer in an ice cold room in a remote corner of the Computer Science Department which itself was in a remote corner of the IIT Bombay campus. He was very helpful and immediately gave us the computer code that we needed.

            I was curious about how he got interested in this topic. He pulled out a book from the stack in his room:  Mining the Web: Discovering Knowledge from Hypertext Data.

"This is a textbook I have just finished writing for US computer science students", he said nonchalantly tossing it back into the stack. "It will be out in later this year in the US."

It turns out that he was at Stanford at the same time as Brin and Page, the Google founders; they went and found a venture capitalist to fund their search engine efforts  and Soumen came back to India to work at IIT Bombay: "because my mother is old and does not want to leave India".

            Landing the Soumen catch turned out to be the easy part. Getting to engage IIT Bombay in a commercial relationship was to be a near-impossible task. The process for such an engagement is unchartered territory for Indian academic institutions. We settled on a compromise: we hired two of his star graduate students (or more accurately he persuaded them to join us instead of doing what all their classmates did- immigrate to America). Since then, we have been happily working together; whenever we run into a really tough computer science problem, we could get to Soumen through his students.

            I have ever since felt mildly guilty about this arrangement that gave us so much know-how for so little payment. Till I encountered the head of Sarnoff Labs at a conference in Beijing. Sarnoff Labs, based in Princeton New Jersey, is a haloed center for innovation since the 1940's and is responsible for bringing to the world, Colour Television and the Electron Microscope, among other inventions.  He shocked the Beijing audience and me by asserting that "big corporations in the world maintain R&D departments mainly to impress Wall Street."  He said that the era of corporations maintaining large central research labs staffed with Nobel Prize winners (Bell Labs symbolized this) is long gone. The locus of  innovation has almost entirely moved to start-ups. These start-ups staff themselves with recent graduates of top institutions and the alumni who maintain close consultative links to their University professors. 

            I was much relieved to hear this. So, the arrangement we stumbled into: hiring a professor's top students and getting the professor for free seems to be the way R&D is done today. And the irony was that we found it in our own backyard.

END