Once a week.
Subscribe to our weekly newsletter.
Collective intelligence out-diagnoses even professionals
The Human Diagnosis Project project is building the world's "open medical intelligence" system.
- The Human Diagnosis Project can develop medical diagnoses with startling accuracy.
- The platform combines the knowledge of medical professionals and artifical intelligence.
- The goal of the project is to provide open, readily available high-level guidance and training to health care professionals across the globe.
The world-class Mayo Clinic is often the place patients go for a second opinion on a medical diagnosis. It's a good thing they do. According to a report issued by the clinic in 2017, 88 percent of them return home with either a completely different diagnosis or a significantly altered one. Only 12 percent receive confirmation of their doctors' original conclusions.
It's hard to overstate the life-and-death importance of medical misdiagnoses, and with all the artificial intelligence and data collection tools out there, you'd think there might be a way to improve on these statistics. This said, the goal of the Human Diagnosis project, or "Human Dx," (a triple pun their site explains) is to create the world's open medical intelligence system, a "collective intelligence" that can produce vastly improved diagnostic accuracy.
In early March, JAMA published the results of an experiment conducted by Human Dx in cooperation with Harvard, and the results were impressive. Where 54 individual human medical specialists correctly diagnosed 156 test cases 66.3 percent of the time, collective intelligence achieved an 85.5 percent accuracy rate. Nine medical professionals contributed to the collective intelligence conclusions.
Human Dx founder Jayanth Komarneni tells Big Think that, "We can get numbers in the 97th, 98th [percentile], and even — if we have sufficiently large numbers of participants — we can get to super intelligent results. That means that it outperforms 100 percent of individual participants."
About Human Dx
The Human Dx project is a partnership between the social, public, and private sectors — in the U.S., it's a 501 (c)(3) not-for-profit/public-benefit corporation. According to Komarneni, Human Dx's business model is as free of cost to users as possible while still generating enough income to be self-sustaining. There are now nearly 20,000 medical professionals in almost 80 countries contributing. Among Human Dx's partners are, as the company states: the American Medical Association, the Association of American Medical Colleges, American Board of Medical Specialties, and the American Board of Internal Medicine. They're also working in collaboration with researchers at Harvard, Johns Hopkins., University of California San Francisco, Berkeley, and MIT.
While diagnoses produced by Human Dx do bring together the opinions of multiple medical professionals, it's far from a simple voting system. It incorporates its own massive data set, machine learning, and artificial intelligence in addition to the input from medical professionals to develop its diagnoses. In designing their collective intelligence, says Komarneni, Human Dx had to first re-think the idea of open intelligence itself.
"We believe that open intelligence is the third form of open knowledge," he explains. The first was open source-protocols such as those on which the internet is based, as well as operating systems such as Linux. These protocols enabled the second form, open content: Wikipedia, data libraries, and so on. Open intelligence combines the first two: "And when you think about A.I. in the context of software," says Komarneni, "it really is code which is smartly delivering content to you based on what you put into the system."
The importance of open intelligence is that without it being available at low cost or free, the cost of A.I. is going to be so prohibitive that it'll "exacerbate, as opposed to close, income, health, and other disparities in society," warns Komarneni. Nowhere will the ramification be more serious than in health care, since "there is nothing we care more about than the well-being of the people we love and ourselves."
How Human Dx collective intelligence works
Collective intelligence in the Human Dx project is not unlike a panel of participants, when are referred to as "agents." Some of these are medical professionals, but they may also include the outputs of other systems. For example, Komarneni mentions that it's entirely possible IBM's Watson could be one of these agents, or even a data set from the National Institutes of Health.
Of course, individual agents, even the human participants, express themselves in their own ways — is a lump "blue" or "blueberry-colored," for example — not to mention that contributions from some agents such as A.I. or datasets may be in the form of raw data. Before any meaningful synthesis of all these opinions can be performed, the first step is to convert them all into a common language of some sort. Human Dx's AI uses natural language processing, text prediction, and medical ontologies to derive these translations as the process's first step.
Human Dx establishes the capability, or CQ ("clinical quotient"), of each agent. To do this they rank agents' skills using test cases with known diagnoses, including "some of the most wickedly complex cases," says Komarneni. This allows Human Dx to determine how accurate agents' diagnoses can be expected to be, and how heavily they should be weighted against other participants' contributions in solving the current case.
A.I. joins the panel
At this point, the agents' inputs are synthesized to derive the most likely diagnosis, and this is combined in an A.I. model with all of the aggregated case data that's ever been captured by Human Dx — interactions in the "tens of millions" — including how "lots of other participants over many other cases have solved these cases." This A.I. model then joins the panel in arriving at the final diagnosis.
"And those [agents] combined," says Komarneni, "are how we can get to results that outperform the vast majority of individual participants."
The Harvard and Johns Hopkins studies
The Harvard study published in JAMA is the first public demonstration of the Human Dx system as a diagnostic tool. Working with an international cohort of medical students and professionals, the results were unquestionably amazing. There were 2069 users working 1572 cases — again, these were cases with known correct answers — from the Human Dx data set. About 60 percent of the participants were residents or fellows, 20 percent were attending physicians, and another 20 percent were medical students. In the study, as more medical professionals were added to the collective intelligence "panel," up to nine individuals, its accuracy consistently rose. Physicians who weren't specialists in their test-case areas achieved just a 62.5 percent accuracy score.
A previous study published in JAMA in January, and done in cooperation with Johns Hopkins, looked at Human Dx as an automatic platform for assessing the diagnostic abilities of health care professionals and students. That the scores of participants looking at 11,023 case simulations were consistent with their training level shows, in Komarneni words, "that we provided a valid, quantitative, scalable measure of medical reasoning." While he admits this doesn't sound like a big deal, it is, since it offers a far more accurate and scalable option to current multiple-choice assessments, which have been shown to correspond poorly to real-world diagnostic skills.
The future of health care and Human Dx
Komarneni says that there are basically only two ways to provide global universal health care, a pressing need since, "Almost half the world has no access to essential health services." One way, he says, would be to create a God-like A.I. system to provide health care to everyone, but, "We know that's not going to happen." God-like AI is just too hard, potentially requiring having to know everything about a patient from the tiniest details — say, the quantum behavior of electrons in mitochondria — to the huge, as in the kind of environment a patient lived in as a child.
In addition, Komarneni says, "In a world where data is locked up in many disparate silos, there isn't going to be a single collective agent. There's going to be a collective of many intelligent agents, both human and machine. The key is how do you integrate intelligence into larger buckets of intelligence than can solve the world's hardest problems."
This is where the Human Dx project, and the second approach, comes in. It actually has two components:
- The first is the expansion of existing medical professionals' diagnostic accuracy skills by providing them access to the Human Dx platform and its collective intelligence as a diagnostic tool.
- The second is helping to train new professionals, and Human Dx Training is already offering this on the Human Dx site.
For those concerned with privacy in a system such as Human Dx, Komarneni says it'll be a non-issue, explaining with an example. When two people converse, "We don't have access to the underlying data of each others' minds. We're agents that are interacting with each other to gain relevant and useful information from each other." Similarly, Human Dx's system of interacting agents doesn't require the exposure of patients' personal data. What's shared with Human Dx are the conclusions agents draw from that data, not the data itself. In the case of a dataset operating as an agent, the data would be anonymized.
Human Dx's interest in all this is developing a platform it hopes others find uses for. "We believe we're just building the enabling technology that many other stakeholders could use." As examples, Komarneni imagines, "The VA could implement their own version of this. Kaiser Permanente could implement their own version. Employers could contract with us or with their own insurers. You could even also have individual and group practices use Human Dx software to serve patients directly."
Human Dx is currently looking at ways to open up as much of the project for non-professionals as possible, and they've already made a start: On their home page is a diagnosis cloud — mouse over the various blue bubbles to see different conditions, and then click for further details. In addition, just beneath the cloud is a search field with which you can look up diseases and symptoms.
- A new day is coming in healthcare, where AI will help diagnose and ... ›
- How deep medicine will revolutionize health care - Big Think ›
Scientists discover what our human ancestors were making inside the Wonderwerk Cave in South Africa 1.8 million years ago.
- Researchers find evidence of early tool-making and fire use inside the Wonderwerk Cave in Africa.
- The scientists date the human activity in the cave to 1.8 million years ago.
- The evidence is the earliest found yet and advances our understanding of human evolution.
One of the oldest activities carried out by humans has been identified in a cave in South Africa. A team of geologists and archaeologists found evidence that our ancestors were making fire and tools in the Wonderwerk Cave in the country's Kalahari Desert some 1.8 million years ago.
A new study published in the journal Quaternary Science Reviews from researchers at the Hebrew University of Jerusalem and the University of Toronto proposes that Wonderwerk — which means "miracle" in Afrikaans — contains the oldest evidence of human activity discovered.
"We can now say with confidence that our human ancestors were making simple Oldowan stone tools inside the Wonderwerk Cave 1.8 million years ago," shared the study's lead author Professor Ron Shaar from Hebrew University.
Oldowan stone tools are the earliest type of tools that date as far back as 2.6 million years ago. An Oldowan tool, which was useful for chopping, was made by chipping flakes off of one stone by hitting it with another stone.
An Oldowan stone toolCredit: Wikimedia / Public domain
Professor Shaar explained that Wonderwerk is different from other ancient sites where tool shards have been found because it is a cave and not in the open air, where sample origins are harder to pinpoint and contamination is possible.
Studying the cave, the researchers were able to pinpoint the time over one million years ago when a shift from Oldowan tools to the earliest handaxes could be observed. Investigating deeper in the cave, the scientists also established that a purposeful use of fire could be dated to one million years back.
This is significant because examples of early fire use usually come from sites in the open air, where there is the possibility that they resulted from wildfires. The remnants of ancient fires in a cave — including burned bones, ash, and tools — contain clear clues as to their purpose.
To precisely date their discovery, the researchers relied on paleomagnetism and burial dating to measure magnetic signals from the remains hidden within a sedimentary rock layer that was 2.5 meters thick. Prehistoric clay particles that settled on the cave floor exhibit magnetization and can show the direction of the ancient earth's magnetic field. Knowing the dates of magnetic field reversals allowed the scientists to narrow down the date range of the cave layers.
The Kalahari desert Wonderwerk CaveCredit: Michael Chazan / Hebrew University of Jerusalem
Professor Ari Matmon of Hebrew University used another dating method to solidify their conclusions, focusing on isotopes within quartz particles in the sand that "have a built-in geological clock that starts ticking when they enter a cave." He elaborated that in their lab, the scientists were "able to measure the concentrations of specific isotopes in those particles and deduce how much time had passed since those grains of sand entered the cave."
Finding the exact dates of human activity in the Wonderwerk Cave could lead to a better understanding of human evolution in Africa as well as the way of life of our early ancestors.
If you ask your maps app to find "restaurants that aren't McDonald's," you won't like the result.
- The Chinese Room thought experiment is designed to show how understanding something cannot be reduced to an "input-process-output" model.
- Artificial intelligence today is becoming increasingly sophisticated thanks to learning algorithms but still fails to demonstrate true understanding.
- All humans demonstrate computational habits when we first learn a new skill, until this somehow becomes understanding.
It's your first day at work, and a new colleague, Kendall, catches you over coffee.
"You watch the game last night?" she says. You're desperate to make friends, but you hate football.
"Sure, I can't believe that result," you say, vaguely, and it works. She nods happily and talks at you for a while. Every day after that, you live a lie. You listen to a football podcast on the weekend and then regurgitate whatever it is you hear. You have no idea what you're saying, but it seems to impress Kendall. You somehow manage to come across as an expert, and soon she won't stop talking football with you.
The question is: do you actually know about football, or are you imitating knowledge? And what's the difference? Welcome to philosopher John Searle's "Chinese Room."
The Chinese Room
Searle's argument was designed as a critique of what's called a "functionalist" view of mind. This is the philosophy that argues that our mind can be explained fully by what role it plays, or in other words, what it does or what "function" it has.
One form of functionalism sees the human mind as following an "input-process-output" model. We have the input of our senses, the process of our brains, and a behavioral output. Searle thought this was at best an oversimplification, and his Chinese Room thought experiment goes to show how human minds are not simply biological computers. It goes like this:
Imagine a room, and inside is John, who can't speak a word of Chinese. Outside the room, a Chinese person sends a message into the room in Chinese. Luckily, John has an "if-then" book for Chinese characters. For instance, if he gets <你好吗>, the proper reply is <我还好>. All John has to do is follow his instruction book.
The Chinese speaker outside of the room thinks they're talking to someone inside who knows Chinese. But in reality, it's just John with his fancy book.
What is understanding?
Does John understand Chinese? The Chinese Room is, by all accounts, a computational view of the mind, yet it seems that something is missing. Truly understanding something is not an "if-then" automated response. John is missing that sinking in feeling, the absorption, the bit of understanding that's so hard to express. Understanding a language doesn't work like this. Humans are not Google Translate.
And yet, this is how AIs are programmed. A computer system is programmed to provide a certain output based on a finite list of certain inputs. If I double click the mouse, I open a file. If you type a letter, your monitor displays tiny black squiggles. If we press the right buttons in order, we win at Mario Kart. Input — Process — Output.
Can imitation become so fluid or competent that it is understanding.
But AIs don't know what they're doing, and Google Translate doesn't really understand what it's saying, does it? They're just following a programmer's orders. If I say, "Will it rain tomorrow?" Siri can look up the weather. But if I ask, "Will water fall from the clouds tomorrow?" it'll be stumped. A human would not (although they might look at you oddly).
A fun way to test just how little an AI understands us is to ask your maps app to find "restaurants that aren't McDonald's." Unsurprisingly, you won't get what you want.
The Future of AI
To be fair, the field of artificial intelligence is just getting started. Yes, it's easy right now to trick our voice assistant apps, and search engines can be frustratingly unhelpful at times. But that doesn't mean AI will always be like that. It might be that the problem is only one of complexity and sophistication, rather than anything else. It might be that the "if-then" rule book just needs work. Things like "the McDonald's test" or AI's inability to respond to original questions reveal only a limitation in programming. Given that language and the list of possible questions is finite, it's quite possible that AI will be able to (at the very least) perfectly mimic a human response in the not too distant future.
What's more, AIs today have increasingly advanced learning capabilities. Algorithms are no longer simply input-process-output but rather allow systems to search for information and adapt anew to what they receive.
A notorious example of this occurred when a Microsoft chat bot started spouting bigotry and racism after "learning" from what it read on Twitter. (Although, this might just say more about Twitter than AI.) Or, more sinister perhaps, two Facebook chat bots were shut down after it was discovered that they were not only talking to each other but were doing so in an invented language. Did they understand what they were doing? Who's to say that, with enough learning and enough practice, an AI "Chinese Room" might not reach understanding?
Can imitation become understanding?
We've all been a "Chinese Room" at times — be it talking about sports at work, cramming for an exam, using a word we didn't entirely know the meaning of, or calculating math problems. We can all mimic understanding, but it also begs the question: can imitation become so fluid or competent that it is understanding.
The old adage "fake it, 'till you make it" has been proven true over and over. If you repeat an action enough times, it becomes easy and habitual. For instance, when you practice a language, musical instrument, or a math calculation, then after a while, it becomes second nature. Our brain changes with repetition.
So, it might just be that we all start off as Chinese Rooms when we learn something new, but this still leaves us with a pertinent question: when, how, and at what point does John actually understand Chinese? More importantly, will Siri or Alexa ever understand you?
With the rise of Big Data, methods used to study the movement of stars or atoms can now reveal the movement of people. This could have important implications for cities.
- A treasure trove of mobility data from devices like smartphones has allowed the field of "city science" to blossom.
- I recently was part of team that compared mobility patterns in Brazilian and American cities.
- We found that, in many cities, low-income and high-income residents rarely travel to the same geographic locations. Such segregation has major implications for urban design.
Almost 55 percent of the world's seven billion people live in cities. And unless the COVID-19 pandemic puts a serious — and I do mean serious — dent in long-term trends, the urban fraction will climb almost to 70 percent by midcentury. Given that our project of civilization is staring down a climate crisis, the massive population shift to urban areas is something that could really use some "sciencing."
Is urbanization going to make things worse? Will it make things better? Will it lead to more human thriving or more grinding poverty and inequality? These questions need answers, and a science of cities, if there was such a thing, could provide answers.
Good news. There already is one!
The science of cities
With the rise of Big Data (for better or worse), scientists from a range of disciplines are getting an unprecedented view into the beating heart of cities and their dynamics. Of course, really smart people have been studying cities scientifically for a long time. But Big Data methods have accelerated what's possible to warp speed. As "exhibit A" for the rise of a new era of city science, let me introduce you to the field of "human mobility" and a new study just published by a team I was on.
Credit: nonnie192 / 405009778 via Adobe Stock
Human mobility is a field that's been amped up by all those location-enabled devices we carry around and the large-scale datasets of our activities, such as credit card purchases, taxi rides, and mobile phone usage. These days, all of us are leaving digital breadcrumbs of our everyday activities, particularly our movements around towns and cities. Using anonymized versions of these datasets (no names please), scientists can look for patterns in how large collections of people engage in daily travel and how these movements correlate with key social factors like income, health, and education.
There have been many studies like this in the recent past. For example, researchers looking at mobility patterns in Louisville, Kentucky found that low-income residents tended to travel further on average than affluent ones. Another study found that mobility patterns across different socioeconomic classes exhibit very similar characteristics in Boston and Singapore. And an analysis of mobility in Bogota, Colombia found that the most mobile population was neither the poorest nor the wealthiest citizens but the upper-middle class.
These were all excellent studies, but it was hard to make general conclusions from them. They seemed to point in different directions. The team I was part of wanted to get a broader, comparative view of human mobility and income. Through a partnership with Google, we were able to compare data from two countries — Brazil and the United States — of relatively equal populations but at different points on the "development spectrum." By comparing mobility patterns both within and between the two countries, we hoped to gain a better understanding of how people at different income levels moved around each day.
Mobility in Brazil vs. United States
Socioeconomic mobility "heatmaps" for selected cities in the U.S. and Brazil. The colors represent destination based on income level. Red depicts destinations traveled by low-income residents, while blue depicts destinations traveled by high-income residents. Overlapping areas are colored purple.Credit: Hugo Barbosa et al., Scientific Reports, 2021.
The results were remarkable. In a figure from our paper (shown above), it's clear that we found two distinct kinds of relationship between income and mobility in cities.
The first was a relatively sharp distinction between where people in lower and higher income brackets traveled each day. For example, in my hometown of Rochester, New York or Detroit, the places visited by the two income groups (e.g., job sites, shopping centers, doctors' offices) were relatively partitioned. In other words, people from low-income and high-income neighborhoods were not mixing very much, meaning they weren't spending time in the same geographical locations. In addition, lower income groups traveled to the city center more often, while upper income groups traveled around the outer suburbs.
The second kind of relationship was exemplified by cities like Boston and Atlanta, which didn't show this kind of partitioning. There was a much higher degree of mixing in terms of travel each day, indicating that income was less of a factor for determining where people lived or traveled.
In Brazil, however, all the cities showed the kind of income-based segregation seen in U.S. cities like Rochester and Detroit. There was a clear separation of regions visited with practically no overlap. And unlike the U.S., visits by the wealthy were strongly concentrated in the city centers, while the poor largely traversed the periphery.
Data-driven urban design
Our results have straightforward implications for city design. As we wrote in the paper, "To the extent that it is undesirable to have cities with residents whose ability to navigate and access resources is dependent on their socioeconomic status, public policy measures to mitigate this phenomenon are the need of the hour." That means we need better housing and public transportation policies.
But while our study shows there are clear links between income disparity and mobility patterns, it also shows something else important. As an astrophysicist who spent decades applying quantitative methods to stars and planets, I am amazed at how deep we can now dive into understanding cities using similar methods. We have truly entered a new era in the study of cities and all human systems. Hopefully, we'll use this new power for good.
A small percentage of people who consume psychedelics experience strange lingering effects, sometimes years after they took the drug.