Self-Motivation
David Goggins
Former Navy Seal
Career Development
Bryan Cranston
Actor
Critical Thinking
Liv Boeree
International Poker Champion
Emotional Intelligence
Amaryllis Fox
Former CIA Clandestine Operative
Management
Chris Hadfield
Retired Canadian Astronaut & Author
Learn
from the world's big
thinkers
Start Learning

Crawling at Top Speed

Question: What exactly is a search algorithm, and how does Google News’s algorithm differ from Google’s?

Josh Cohen: Sure. Yeah, it does. I mean, I think the mission behind Google News is really the same thing that we're trying to do with Web search overall, for the Google mission statement, of organizing the world's information and make it universally accessible and useful. That's what we're trying to do for news as well. We're just trying to sort of apply that to -- in every single different language, every single country, every single source, and ultimately down to every single story. So the process of putting together Google News, I think the easiest way to think about it is in sort of three easy steps of how you get to Google News. So first is the crawl process, where we go off and actually crawl the Web looking for news information. There are -- technically it's the same process as Web search does. There's a Google bot or crawler that will go off and index information.

There are two key differences, though, in the crawling process between how news works and how Web search works. One is that Google News is a closed index, versus an open index for Web search. And what that means is for Web search, if you put up a personal blog, you start a Web site tomorrow, at some point Web search is going to go and pick up that information and index it. Google News is focused just on news sites. So it's a smaller set of sites. So it's -- we're really just looking for news sites, so it's a smaller set of sources in our index. And the second is the speed with which we'll crawl it. So Web search has hundreds and millions of different URLs that they're trying to crawl. And for Google News, we have a smaller set of that, but it's also the speed with which we crawl; it has to be much, much faster. So Web search, you'll have a wide variety of crawl times, because if it's a personal blog, maybe you update it once a month or so. You don't need to crawl it every single second. For news, since we're crawling these news sites, speed just has to be -- it's a matter of minutes. It really has to be as close to real time as possible. So that's one key difference in sort of the crawl part of it.

The second part of this whole clustering, which is again one of the key innovations that Krishna applied to news, which is taking all these different sources and trying to cluster them into similar types of storylines. So it can be trying to understand, obviously, the same languages, same editions, same topics, and those same story clusters together. And then the ranking part of it is completely different from Web search. And you'll hear things -- when people talk about Google Web search, you'll hear about page rank, and inbound links and outbound links.

And it's really a very different process that we're trying to do for Google News. So some of the things that we're looking at is obviously timeliness -- that matters for a news site; location: is it a local source reporting on a local story, doing original reporting? That's something else we're going to take a look at, as well as the nature of the source as well. So it's not just -- you know, it's not a human coming in and saying, well, this is a good source or a bad source, but rather looking at a number of different signals to try and determine the quality of a given source. Are they producing the original content? You know, how do users respond to the brand? Is there a sort of certain value when they see that link, and sort of -- not just the most clicks, because that becomes somewhat self-fulfilling, but rather trying to discern the user's trust for a given source. So those are all things that are really unique to news as opposed to how Web search works.

An inside look at the Google News search algorithm and how it differs from its more famous parent.

Live on Tuesday | Personal finance in the COVID-19 era

Sallie Krawcheck and Bob Kulhan will be talking money, jobs, and how the pandemic will disproportionally affect women's finances.

How often do vaccine trials hit paydirt?

Vaccines find more success in development than any other kind of drug, but have been relatively neglected in recent decades.

Pedro Vilela/Getty Images
Surprising Science

Vaccines are more likely to get through clinical trials than any other type of drug — but have been given relatively little pharmaceutical industry support during the last two decades, according to a new study by MIT scholars.

Keep reading Show less

Consumer advocacy groups are mostly funded by Big Pharma, according to new research

An article in Journal of Bioethical Inquiry raises questions about the goal of these advocacy groups.

Image by Jukka Niittymaa / Pixabay
Politics & Current Affairs
  • Two-thirds of American consumer advocacy groups are funded by pharmaceutical companies.
  • The authors of an article in Journal of Bioethical Inquiry say this compromises their advocacy.
  • Groups like the National Alliance on Mental Illness act more like lobbyists than patient advocates.

Keep reading Show less

Bubonic plague case reported in China

Health officials in China reported that a man was infected with bubonic plague, the infectious disease that caused the Black Death.

(Photo by Centers for Disease Control and Prevention/Getty Images)
Coronavirus
  • The case was reported in the city of Bayannur, which has issued a level-three plague prevention warning.
  • Modern antibiotics can effectively treat bubonic plague, which spreads mainly by fleas.
  • Chinese health officials are also monitoring a newly discovered type of swine flu that has the potential to develop into a pandemic virus.
Keep reading Show less

Women who go to church have more kids—and more help

Want help raising your kids? Spend more time at church, says new study.

Pixabay
Culture & Religion
  • Religious people tend to have more children than secular people, but why remains unknown.
  • A new study suggests that the social circles provided by regular church going make raising kids easier.
  • Conversely, having a large secular social group made women less likely to have children.
Keep reading Show less
Quantcast