Crawling at Top Speed
Josh Cohen is the Senior Business Product Manager at Google News, where he manages global product strategy, marketing, and publisher outreach. He was previously vice president of business development for Reuters Media and director of business development for SmartMoney.com.
Question: What exactly is a search algorithm, and how does Google News’s algorithm differ from Google’s?
Josh Cohen: Sure. Yeah, it does. I mean, I think the mission behind Google News is really the same thing that we're trying to do with Web search overall, for the Google mission statement, of organizing the world's information and make it universally accessible and useful. That's what we're trying to do for news as well. We're just trying to sort of apply that to -- in every single different language, every single country, every single source, and ultimately down to every single story. So the process of putting together Google News, I think the easiest way to think about it is in sort of three easy steps of how you get to Google News. So first is the crawl process, where we go off and actually crawl the Web looking for news information. There are -- technically it's the same process as Web search does. There's a Google bot or crawler that will go off and index information.
There are two key differences, though, in the crawling process between how news works and how Web search works. One is that Google News is a closed index, versus an open index for Web search. And what that means is for Web search, if you put up a personal blog, you start a Web site tomorrow, at some point Web search is going to go and pick up that information and index it. Google News is focused just on news sites. So it's a smaller set of sites. So it's -- we're really just looking for news sites, so it's a smaller set of sources in our index. And the second is the speed with which we'll crawl it. So Web search has hundreds and millions of different URLs that they're trying to crawl. And for Google News, we have a smaller set of that, but it's also the speed with which we crawl; it has to be much, much faster. So Web search, you'll have a wide variety of crawl times, because if it's a personal blog, maybe you update it once a month or so. You don't need to crawl it every single second. For news, since we're crawling these news sites, speed just has to be -- it's a matter of minutes. It really has to be as close to real time as possible. So that's one key difference in sort of the crawl part of it.
The second part of this whole clustering, which is again one of the key innovations that Krishna applied to news, which is taking all these different sources and trying to cluster them into similar types of storylines. So it can be trying to understand, obviously, the same languages, same editions, same topics, and those same story clusters together. And then the ranking part of it is completely different from Web search. And you'll hear things -- when people talk about Google Web search, you'll hear about page rank, and inbound links and outbound links.
And it's really a very different process that we're trying to do for Google News. So some of the things that we're looking at is obviously timeliness -- that matters for a news site; location: is it a local source reporting on a local story, doing original reporting? That's something else we're going to take a look at, as well as the nature of the source as well. So it's not just -- you know, it's not a human coming in and saying, well, this is a good source or a bad source, but rather looking at a number of different signals to try and determine the quality of a given source. Are they producing the original content? You know, how do users respond to the brand? Is there a sort of certain value when they see that link, and sort of -- not just the most clicks, because that becomes somewhat self-fulfilling, but rather trying to discern the user's trust for a given source. So those are all things that are really unique to news as opposed to how Web search works.
An inside look at the Google News search algorithm and how it differs from its more famous parent.
These five main food groups are important for your brain's health and likely to boost the production of feel-good chemicals.
We all know eating “healthy” food is good for our physical health and can decrease our risk of developing diabetes, cancer, obesity and heart disease. What is not as well known is that eating healthy food is also good for our mental health and can decrease our risk of depression and anxiety.
Infographics show the classes and anxieties in the supposedly classless U.S. economy.
For those of us who follow politics, we’re used to commentators referring to the President’s low approval rating as a surprise given the U.S.'s “booming” economy. This seeming disconnect, however, should really prompt us to reconsider the measurements by which we assess the health of an economy. With a robust U.S. stock market and GDP and low unemployment figures, it’s easy to see why some think all is well. But looking at real U.S. wages, which have remained stagnant—and have, thus, in effect gone down given rising costs from inflation—a very different picture emerges. For the 1%, the economy is booming. For the rest of us, it’s hard to even know where we stand. A recent study by Porch (a home-improvement company) of blue-collar vs. white-collar workers shows how traditional categories are becoming less distinct—the study references "new-collar" workers, who require technical certifications but not college degrees. And a set of recent infographics from CreditLoan capturing the thoughts of America’s middle class as defined by the Pew Research Center shows how confused we are.
SMARTER FASTER trademarks owned by The Big Think, Inc. All rights reserved.