Jimmy Wales on Wikia Search

Question: What is Wikia Search?

Jimmy Wales: Yeah, so at Wikia Search, what we’re really trying to do is build a freely licensed, or open source software search engine, a general search engine for the web.  We’re building on efforts that have been going on for several years in the Nutch project.  But we’re committing a lot of infrastructure to it and programmer time.  And the basic idea here is to say that, you know, search is the one key piece of infrastructure, the internet that is still very much proprietary, closed, secretive, and we want to try to change that.  We want to make it open.  So we’re publishing all the algorithms, publishing all the software, so people can see it.  But additionally, what we’re trying to do is bring in the element of mass participation, user participation.  The idea is to take the normal editorial decisions that are made within the search engine company, and push those out into the community as much as possible, so that, you know, the search results reflect whatever this community thinks should be ranked in certain ways.  So we just launched a very preliminary alpha version a couple months ago.  We’re currently very hard at work at Version 0.2, which we’ll be releasing within the next month or two.  We haven’t really fixed a date yet.  That’s actually something I need to be working on right now is my release schedule.  But yeah, we’re really looking forward to getting that out, because there’s a lot of cool new tools and things.

Question: What would that mean for the non-coding user?

Jimmy Wales: So in terms of the user experience, we expect it to be really quite a bit similar, I mean, in other words you come and you type something you’re looking for, and you hit “Search,” and you get some stuff back.  The one major difference at that point is that you’ll be able to edit the results, in essence.  So you’ll be able to delete things, add things.  And those are going to be public actions, just as the public, like when you edit Wikipedia that’s a public action.  And there’ll be a whole community of people monitoring and overseeing all that.  That’s the main difference from the end user point of view, but for me, I think the deeper implication here is not about sort of just the user experience, but about the open source nature, the free software nature.  The way I look at that for the average user, obviously, you’re not going to go and download all our source code and read it, that’s really not that important, but what is important-- I view this as being very similar to-- in a free society, in an open society we insist that our court systems operate in a public fashion.  So you can go down to the courthouse today and go in and watch a trial unfold, and that’s done in public.  And that’s a valuable safeguard for human rights, democracy, even though most of us never actually bother going down to the courthouse.  But it’s important that there’re people who can, and people who do.  And they’ll raise the alarm if something untoward is going on, and that’s really the way it works with open source software.  There’s also a lot of practical benefits to open source software which I think we can realize here.  You know, right now, hands down the best web browser is Firefox, which is an open source project.  And you know, that development model for software has proven to be highly effective.  One of the things we’re looking at is that there’re lot of parts of the search business that are duplicated efforts.  Lots of different smaller competitors are wasting a lot of money duplicating a lot of effort to doing some very basic infrastructure things that are really not necessary to do multiple times.  I mean, crawling the web, it’s a big job, but what it means is you have to go and fetch, you know, lots and lots and lots of pages from all over the internet, and it’s a bit of a commodity item.  I mean, anyone can crawl the web, it just takes, well, a lot of engineers and money and time.  And you know, the techniques for doing it are pretty robust and well-known.  So what we’re all hoping to do is using the social model of free licensing, which has already allowed lots of different companies to come together.  You know, IBM contributes a lot to free software, Red Hat, you know, all these different companies are working in this space, and they’re able to do it because the free licensing model creates a level playing field for that.  So we’re hoping to see the same kinds of things start to develop with search data.  In other words having different players share their crawl results with others freely.  Well, we’re working on that.

Another part of it is one of the things-- one of the really interesting things that’s going on right now in search is that all of the-- <coughs> sorry.  One of the interesting things about search that’s going on right now is that all of the research and development, the vast majority of it is going on inside companies in proprietary research, behind closed doors, very secretive.  At Google, Yahoo! Ask and Microsoft and so forth.  And right now if you are a PhD student in computer science and you’re really interested in search algorithms, it’s kind of hard for you to really get involved and to work on things, because you don’t have access to the kind of resources that you would like to have.  And so a big part of what I want to do is say, “Well, look, if we make all this source code open source, any of those people can download it, they can run out on a university cluster.  We’re gonna try to make available research machines for people to run things.  Just to do experimentation.  Just to sort of say, “Let’s try to use the time-honored and tested methods of academic research where things are done publicly under peer review, and really sort of let’s provide some infrastructure for that.  So that’s kind of the idea there as well.

Recorded on: 4/30/08

The revolutionary idea of building an open-source software search engine.

Elizabeth Warren's plan to forgive student loan debt could lead to an economic boom

A plan to forgive almost a trillion dollars in debt would solve the student loan debt crisis, but can it work?

Photo credit: Drew Angerer / Getty Images
Politics & Current Affairs
  • Sen. Elizabeth Warren has just proposed a bold education reform plan that would forgive billions in student debt.
  • The plan would forgive the debt held by more than 30 million Americans.
  • The debt forgiveness program is one part of a larger program to make higher education more accessible.
Keep reading Show less

Banned books: 10 of the most-challenged books in America

America isn't immune to attempts to remove books from libraries and schools, here are ten frequent targets and why you ought to go check them out.

Nazis burn books on a huge bonfire of 'anti-German' literature in the Opernplatz, Berlin. (Photo by Keystone/Getty Images)
Culture & Religion
  • Even in America, books are frequently challenged and removed from schools and public libraries.
  • Every year, the American Library Association puts on Banned Books Week to draw attention to this fact.
  • Some of the books they include on their list of most frequently challenged are some of the greatest, most beloved, and entertaining books there are.
Keep reading Show less

Supreme Court to hear 3 cases on LGBT workplace discrimination

In most states, LGBTQ Americans have no legal protections against discrimination in the workplace.

(Photo by Andres Pantoja/SOPA Images/LightRocket via Getty Images)
Politics & Current Affairs
  • The Supreme Court will decide whether the Civil Rights Act of 1964 also applies to gay and transgender people.
  • The court, which currently has a probable conservative majority, will likely decide on the cases in 2020.
  • Only 21 states and the District of Columbia have passed laws effectively extending the Civil Rights of 1964 to gay and transgender people.
Keep reading Show less