Crawling at Top Speed
Josh Cohen is the Senior Business Product Manager at Google News, where he manages global product strategy, marketing, and publisher outreach. He was previously vice president of business development for Reuters Media and director of business development for SmartMoney.com.
Question: What exactly is a search algorithm, and how does Google News’s algorithm differ from Google’s?
Josh Cohen: Sure. Yeah, it does. I mean, I think the mission behind Google News is really the same thing that we're trying to do with Web search overall, for the Google mission statement, of organizing the world's information and make it universally accessible and useful. That's what we're trying to do for news as well. We're just trying to sort of apply that to -- in every single different language, every single country, every single source, and ultimately down to every single story. So the process of putting together Google News, I think the easiest way to think about it is in sort of three easy steps of how you get to Google News. So first is the crawl process, where we go off and actually crawl the Web looking for news information. There are -- technically it's the same process as Web search does. There's a Google bot or crawler that will go off and index information.
There are two key differences, though, in the crawling process between how news works and how Web search works. One is that Google News is a closed index, versus an open index for Web search. And what that means is for Web search, if you put up a personal blog, you start a Web site tomorrow, at some point Web search is going to go and pick up that information and index it. Google News is focused just on news sites. So it's a smaller set of sites. So it's -- we're really just looking for news sites, so it's a smaller set of sources in our index. And the second is the speed with which we'll crawl it. So Web search has hundreds and millions of different URLs that they're trying to crawl. And for Google News, we have a smaller set of that, but it's also the speed with which we crawl; it has to be much, much faster. So Web search, you'll have a wide variety of crawl times, because if it's a personal blog, maybe you update it once a month or so. You don't need to crawl it every single second. For news, since we're crawling these news sites, speed just has to be -- it's a matter of minutes. It really has to be as close to real time as possible. So that's one key difference in sort of the crawl part of it.
The second part of this whole clustering, which is again one of the key innovations that Krishna applied to news, which is taking all these different sources and trying to cluster them into similar types of storylines. So it can be trying to understand, obviously, the same languages, same editions, same topics, and those same story clusters together. And then the ranking part of it is completely different from Web search. And you'll hear things -- when people talk about Google Web search, you'll hear about page rank, and inbound links and outbound links.
And it's really a very different process that we're trying to do for Google News. So some of the things that we're looking at is obviously timeliness -- that matters for a news site; location: is it a local source reporting on a local story, doing original reporting? That's something else we're going to take a look at, as well as the nature of the source as well. So it's not just -- you know, it's not a human coming in and saying, well, this is a good source or a bad source, but rather looking at a number of different signals to try and determine the quality of a given source. Are they producing the original content? You know, how do users respond to the brand? Is there a sort of certain value when they see that link, and sort of -- not just the most clicks, because that becomes somewhat self-fulfilling, but rather trying to discern the user's trust for a given source. So those are all things that are really unique to news as opposed to how Web search works.
An inside look at the Google News search algorithm and how it differs from its more famous parent.
What would happen if you tripled the US population? Matthew Yglesias and moderator Charles Duhigg explore the idea on Big Think Live.
Is immigration key to bolstering the American economy? Could having one billion Americans secure the US's position as the global superpower?
How Nobel Prize winner physicist Lev Landau ranked the best physics minds of his generation.
Rank 0.5 – Albert Einstein<img type="lazy-image" data-runner-src="https://assets.rebelmouse.io/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNDQ0NDY3NS9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTYzNjI2NTU4OH0.FtBYC7oJz-ZOiiGC9y0Z50_JvQChmp-ONa3jhR3SuLA/img.jpg?width=980" id="d6f66" class="rm-shortcode" data-rm-shortcode-id="61288810a4f035ec2af8957fad4e9015" data-rm-shortcode-name="rebelmouse-image" />
Albert Einstein With Displaced Children From Concentration Camps. 1949.
Photo by Keystone-France/Gamma-Keystone via Getty Images
Rank 1<p>The group in this class of the smartest physicists included the top minds that developed the theories of quantum mechanics.</p><p><a href="http://en.wikipedia.org/wiki/Werner_Heisenberg" target="_blank">Werner Heisenberg</a> (1901 - 1976) - a German theoretical physicist, who's achieved pop-culture fame by being the name of Walter White's alter ego in <em>Breaking Bad</em>. He is known for the Heiseinberg Uncertainty Principle and his 1932 Nobel Prize award flatly states it was for nothing less than "the creation of quantum mechanics".</p><p><a href="http://en.wikipedia.org/wiki/Erwin_Schr%C3%B6dinger" target="_blank">Erwin Schrödinger</a> (1887 - 1961) - an Austrian-Irish physicist who gave us the infamous "Schroedinger's Cat" thought experiment and other mind-benders from quantum mechanics. The Nobel-prize-winner's <a href="https://en.wikipedia.org/wiki/Schr%C3%B6dinger_equation" target="_blank" rel="noopener noreferrer">Schrödinger equation</a> calculates the <a href="https://en.wikipedia.org/wiki/Wave_function" target="_blank">wave function</a> of a system and how it changes over time. </p>
Erwin Schrödinger. 1933.
Satyendra Nath Bose. 1930s.
Enrico Fermi. 1950s.
Rank 2.5<img type="lazy-image" data-runner-src="https://assets.rebelmouse.io/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNDQ0NDcwNy9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTY0NDE1MDIxM30.Eg6tca61EredHxjqNH29HY3UeJbgBVa1nA13EhXTooU/img.jpg?width=980" id="90f86" class="rm-shortcode" data-rm-shortcode-id="0f1e6c5e13263a77b2061e1191fd8baf" data-rm-shortcode-name="rebelmouse-image" />
Lev Landau. 1962.<p><strong>Rank 2.5</strong> is where Landau initially ranked himself, rather modestly, thinking he didn't produce any foundational accomplishments. He later moved his prominence, as his achievement mounted, to the higher <strong>1.5.</strong></p>
Controversial physics theory says reality around us behaves like a computer neural network.
- Physicist proposes that the universe behaves like an artificial neural network.
- The scientist's new paper seeks to reconcile classical physics and quantum mechanics.
- The theory claims that natural selection produces both atoms and "observers".
Vanchurin interview:<span style="display:block;position:relative;padding-top:56.25%;" class="rm-shortcode" data-rm-shortcode-id="539759cbfd8fcd5b6ebf14a3b597b3f9"><iframe type="lazy-iframe" data-runner-src="https://www.youtube.com/embed/bmyRy2-UhEE?rel=0" width="100%" height="auto" frameborder="0" scrolling="no" style="position:absolute;top:0;left:0;width:100%;height:100%;"></iframe></span>
Vanchurin on “Hidden Phenomena”:<span style="display:block;position:relative;padding-top:56.25%;" class="rm-shortcode" data-rm-shortcode-id="18886ffd5e5840bb19d4494212f88d82"><iframe type="lazy-iframe" data-runner-src="https://www.youtube.com/embed/2NDVdNwsHCo?rel=0" width="100%" height="auto" frameborder="0" scrolling="no" style="position:absolute;top:0;left:0;width:100%;height:100%;"></iframe></span>Vitaly Vanchurin speaking at the 6th International FQXi Conference, "Mind Matters: Intelligence and Agency in the Physical World." The Foundational Questions...
A strange weakness in the Earth's protective magnetic field is growing and possibly splitting, shows data.
- "The South Atlantic Anomaly" in the Earth's magnetic field is growing and possibly splitting, shows data.
- The information was gathered by the ESA's Swarm Constellation mission satellites.
- The changes may indicate the coming reversal of the North and South Poles.
Is the Magnetic Field Reversing?<span style="display:block;position:relative;padding-top:56.25%;" class="rm-shortcode" data-rm-shortcode-id="e3e0b16dac3b05dab808a4ddf04d198b"><iframe type="lazy-iframe" data-runner-src="https://www.youtube.com/embed/51usJ74pPP8?rel=0" width="100%" height="auto" frameborder="0" scrolling="no" style="position:absolute;top:0;left:0;width:100%;height:100%;"></iframe></span>
43% of people think they can get a sense of someone's personality by their picture.
If you've used a dating app, you'll know the importance of choosing good profile pics.