Crawling at Top Speed

Question: What exactly is a search algorithm, and how does Google News’s algorithm differ from Google’s?

Josh Cohen: Sure. Yeah, it does. I mean, I think the mission behind Google News is really the same thing that we're trying to do with Web search overall, for the Google mission statement, of organizing the world's information and make it universally accessible and useful. That's what we're trying to do for news as well. We're just trying to sort of apply that to -- in every single different language, every single country, every single source, and ultimately down to every single story. So the process of putting together Google News, I think the easiest way to think about it is in sort of three easy steps of how you get to Google News. So first is the crawl process, where we go off and actually crawl the Web looking for news information. There are -- technically it's the same process as Web search does. There's a Google bot or crawler that will go off and index information.

There are two key differences, though, in the crawling process between how news works and how Web search works. One is that Google News is a closed index, versus an open index for Web search. And what that means is for Web search, if you put up a personal blog, you start a Web site tomorrow, at some point Web search is going to go and pick up that information and index it. Google News is focused just on news sites. So it's a smaller set of sites. So it's -- we're really just looking for news sites, so it's a smaller set of sources in our index. And the second is the speed with which we'll crawl it. So Web search has hundreds and millions of different URLs that they're trying to crawl. And for Google News, we have a smaller set of that, but it's also the speed with which we crawl; it has to be much, much faster. So Web search, you'll have a wide variety of crawl times, because if it's a personal blog, maybe you update it once a month or so. You don't need to crawl it every single second. For news, since we're crawling these news sites, speed just has to be -- it's a matter of minutes. It really has to be as close to real time as possible. So that's one key difference in sort of the crawl part of it.

The second part of this whole clustering, which is again one of the key innovations that Krishna applied to news, which is taking all these different sources and trying to cluster them into similar types of storylines. So it can be trying to understand, obviously, the same languages, same editions, same topics, and those same story clusters together. And then the ranking part of it is completely different from Web search. And you'll hear things -- when people talk about Google Web search, you'll hear about page rank, and inbound links and outbound links.

And it's really a very different process that we're trying to do for Google News. So some of the things that we're looking at is obviously timeliness -- that matters for a news site; location: is it a local source reporting on a local story, doing original reporting? That's something else we're going to take a look at, as well as the nature of the source as well. So it's not just -- you know, it's not a human coming in and saying, well, this is a good source or a bad source, but rather looking at a number of different signals to try and determine the quality of a given source. Are they producing the original content? You know, how do users respond to the brand? Is there a sort of certain value when they see that link, and sort of -- not just the most clicks, because that becomes somewhat self-fulfilling, but rather trying to discern the user's trust for a given source. So those are all things that are really unique to news as opposed to how Web search works.

An inside look at the Google News search algorithm and how it differs from its more famous parent.

Live on Monday: Does the US need one billion people?

What would happen if you tripled the US population? Matthew Yglesias and moderator Charles Duhigg explore the idea on Big Think Live.

Big Think LIVE

Is immigration key to bolstering the American economy? Could having one billion Americans secure the US's position as the global superpower?

Keep reading Show less

Landau Genius Scale ranking of the smartest physicists ever

How Nobel Prize winner physicist Lev Landau ranked the best physics minds of his generation.

Photo by: Photo12/Universal Images Group via Getty Images
Surprising Science
  • Nobel-Prize-winning Soviet physicist Lev Landau used a scale to rank the best physicists of the 20th century.
  • The physicist based it on their level of contribution to science.
  • The scale was logarithmic, with each level being 10 times more valuable.
  • Keep reading Show less

    Universe works like a cosmological neural network, argues new paper

    Controversial physics theory says reality around us behaves like a computer neural network.

    Credit: sakkmesterke
    Surprising Science
    • Physicist proposes that the universe behaves like an artificial neural network.
    • The scientist's new paper seeks to reconcile classical physics and quantum mechanics.
    • The theory claims that natural selection produces both atoms and "observers".
    Keep reading Show less

    Mystery anomaly weakens Earth's magnetic field, report scientists

    A strange weakness in the Earth's protective magnetic field is growing and possibly splitting, shows data.

    ESA
    Surprising Science
    • "The South Atlantic Anomaly" in the Earth's magnetic field is growing and possibly splitting, shows data.
    • The information was gathered by the ESA's Swarm Constellation mission satellites.
    • The changes may indicate the coming reversal of the North and South Poles.
    Keep reading Show less

    We studied what happens when guys add their cats to their dating app profiles

    43% of people think they can get a sense of someone's personality by their picture.

    Photo by Luigi Pozzoli on Unsplash
    Sex & Relationships

    If you've used a dating app, you'll know the importance of choosing good profile pics.

    Keep reading Show less
    Quantcast