Machine learning is a powerful and imperfect tool that should not go unmonitored.
- When you harness the power and potential of machine learning, there are also some drastic downsides that you've got to manage.
- Deploying machine learning, you face the risk that it be discriminatory, biased, inequitable, exploitative, or opaque.
- In this article, I cover six ways that machine learning threatens social justice and reach an incisive conclusion: The remedy is to take on machine learning standardization as a form of social activism.
When you harness the power and potential of machine learning, there are also some drastic downsides that you've got to manage. Deploying machine learning, you face the risk that it be discriminatory, biased, inequitable, exploitative, or opaque. In this article, I cover six ways that machine learning threatens social justice and reach an incisive conclusion: The remedy is to take on machine learning standardization as a form of social activism.
When you use machine learning, you aren't just optimizing models and streamlining business. You're governing. In essence, the models embody policies that control access to opportunities and resources for many people. They drive consequential decisions as to whom to investigate, incarcerate, set up on a date, or medicate – or to whom to grant a loan, insurance coverage, housing, or a job.
For the same reason that machine learning is valuable—that it drives operational decisions more effectively—it also wields power in the impact it has on millions of individuals' lives. Threats to social justice arise when that impact is detrimental, when models systematically limit the opportunities of underprivileged or protected groups.
Here are six ways machine learning threatens social justice
Credit: metamorworks via Shutterstock
1) Blatantly discriminatory models are predictive models that base decisions partly or entirely on a protected class. Protected classes include race, religion, national origin, gender, gender identity, sexual orientation, pregnancy, and disability status. By taking one of these characteristics as an input, the model's outputs – and the decisions driven by the model – are based at least in part on membership in a protected class. Although models rarely do so directly, there is precedent and support for doing so.
This would mean that a model could explicitly hinder, for example, black defendants for being black. So, imagine sitting across from a person being evaluated for a job, a loan, or even parole. When they ask you how the decision process works, you inform them, "For one thing, our algorithm penalized your score by seven points because you're black." This may sound shocking and sensationalistic, but I'm only literally describing what the model would do, mechanically, if race were permitted as a model input.
2) Machine bias. Even when protected classes are not provided as a direct model input, we find, in some cases, that model predictions are still inequitable. This is because other variables end up serving as proxies to protected classes. This is a bit complicated, since it turns out that models that are fair in one sense are unfair in another.
For example, some crime risk models succeed in flagging both black and white defendants with equal precision – each flag tells the same probabilistic story, regardless of race – and yet the models falsely flag black defendants more often than white ones. A crime-risk model called COMPAS, which is sold to law enforcement across the US, falsely flags white defendants at a rate of 23.5%, and Black defendants at 44.9%. In other words, black defendants who don't deserve it are erroneously flagged almost twice as much as white defendants who don't deserve it.
3) Inferring sensitive attributes—predicting pregnancy and beyond. Machine learning predicts sensitive information about individuals, such as sexual orientation, whether they're pregnant, whether they'll quit their job, and whether they're going to die. Researchers have shown that it is possible to predict race based on Facebook likes. These predictive models deliver dynamite.
In a particularly extraordinary case, officials in China use facial recognition to identify and track the Uighurs, a minority ethnic group systematically oppressed by the government. This is the first known case of a government using machine learning to profile by ethnicity. One Chinese start-up valued at more than $1 billion said its software could recognize "sensitive groups of people." It's website said, "If originally one Uighur lives in a neighborhood, and within 20 days six Uighurs appear, it immediately sends alarms" to law enforcement.
4) A lack of transparency. A computer can keep you in jail, or deny you a job, a loan, insurance coverage, or housing – and yet you cannot face your accuser. The predictive models generated by machine learning to drive these weighty decisions are generally kept locked up as a secret, unavailable for audit, inspection, or interrogation. Such models, inaccessible to the public, perpetrate a lack of due process and a lack of accountability.
Two ethical standards oppose this shrouding of electronically-assisted decisions: 1) model transparency, the standard that predictive models be accessible, inspectable, and understandable. And 2) the right to explanation, the standard that consequential decisions that are driven or informed by a predictive model are always held up to that standard of transparency. Meeting those standards would mean, for example, that a defendant be told which factors contributed to their crime risk score -- which aspects of their background, circumstances, or past behavior caused the defendant to be penalized. This would provide the defendant the opportunity to respond accordingly, establishing context, explanations, or perspective on these factors.
5) Predatory micro-targeting. Powerlessness begets powerlessness – and that cycle can magnify for consumers when machine learning increases the efficiency of activities designed to maximize profit for companies. Improving the micro-targeting of marketing and the predictive pricing of insurance and credit can magnify the cycle of poverty. For example, highly-targeted ads are more adept than ever at exploiting vulnerable consumers and separating them from their money.
And insurance pricing can lead to the same result. With insurance, the name of the game is to charge more for those at higher risk. Left unchecked, this process can quickly slip into predatory pricing. For example, a churn model may find that elderly policyholders don't tend to shop around and defect to better offers, so there's less of an incentive to keep their policy premiums in check. And pricing premiums based on other life factors also contributes to a cycle of poverty. For example, individuals with poor credit ratings are charged more for car insurance. In fact, a low credit score can increase your premium more than an at-fault car accident.
6) The coded gaze. If a group of people is underrepresented in the data from which the machine learns, the resulting model won't work as well for members of that group. This results in exclusionary experiences and discriminatory practices. This phenomenon can occur for both facial image processing and speech recognition.
Recourse: Establish machine learning standards as a form of social activism
To address these problems, take on machine learning standardization as a form of social activism. We must establish standards that go beyond nice-sounding yet vague platitudes such as "be fair", "avoid bias", and "ensure accountability". Without being precisely defined, these catch phrases are subjective and do little to guide concrete action. Unfortunately, such broad language is fairly common among the principles released by many companies. In so doing, companies protect their public image more than they protect the public.
People involved in initiatives to deploy machine learning have a powerful, influential voice. These relatively small numbers of people mold and set the trajectory for systems that automatically dictate the rights and resources that great numbers of consumers and citizens gain access to.
Famed machine learning leader and educator Andrew Ng drove it home: "AI is a superpower that enables a small team to affect a huge number of people's lives... Make sure the work you do leaves society better off."
And Allan Sammy, Director, Data Science and Audit Analytics at Canada Post, clarified the level of responsibility: "A decision made by an organization's analytic model is a decision made by that entity's senior management team."
Implementing ethical data science is as important as ensuring a self-driving car knows when to put on the breaks.
Establishing well-formed ethical standards for machine learning will be an intensive, ongoing process. For more, watch this short video, in which I provide some specifics meant to kick-start the process.
Eric Siegel, Ph.D., is a leading consultant and former Columbia University professor who makes machine learning understandable and captivating. He is the founder of the long-running Predictive Analytics World and the Deep Learning World conference series and the instructor of the end-to-end, business-oriented Coursera specialization Machine learning for Everyone. Stay in touch with Eric on Twitter @predictanalytic.
In spite of a government mandate, females are often treated as afterthoughts in scientific research.
- A new study finds that though more females are included in experiments, sex-specific data often goes un-analyzed.
- Only about a third of studies analyzed published participant breakdown by sex.
- Some researchers say considering females more fully as research subjects is logistically too challenging.
In 2016, the National Institutes of Health (NIH) issued a directive that scientists receiving NIH funding must consider sex as a biological variable in pre-clinical research on vertebrate animals and human cells and tissues. According to a new study published in eLife that looked at over 700 journal articles, the number of women included as participants in pre-clinical research has jumped from 28 percent in 2009 to 49 percent in 2019. However, it's also unfortunately still the case that few studies actually consider sex as a biological influence that may potentially affect outcomes, and that data from women participants continues to be simply combined with data from men.
Study co-author Nicole C. Woitowich of Northwestern University's Feinberg School of Medicine tells INSIDE Higher Ed, "In the last 10 years, there has been a major in increase in sex inclusion, but it's still not where it's needs to be."
What's missing in current research
Image source: Hush Naidoo/Unsplash
Woitowich and others see two particularly problematic aspects to the continuing disregard of sex as a meaningful biological research variable.
First, female-specific data is rarely considered in study conclusions, despite the fact that it may have implications for women's health. According to L. Syd M Johnson of SUNY Update Medical University, who was not involved with the study, "This becomes highly problematic both scientifically and ethically, because women, children, and the elderly also need medical care, and they shouldn't be treated as if they have adult, male bodies. When they are excluded from research, and from the reported results, treatment for them becomes, effectively, off-label.
Second, Woitowich tells INSIDE Higher Ed it's, "troublesome to me as a scientist [that] a little under one-third [of studies] did not even report the number of males and females used as subjects." This makes it impossible for scientists to replicate the results. "If I don't have all the information," Woitowich says, "I'm left guessing."
On top of that, Woitowich laments that too much of the female-focused research that is undertaken is what's been called "bikini science," research surrounding issues related to female reproductive organs.
Why is this happening?
Image source: Image Point Fr/Shutterstock
"Many scientists, I don't even know if this is on their radar," says Woitowich. She proposes, therefore, that in the short term it may be the research gatekeepers — the funding entities, journal editors, and peer reviewers — who will have to step up and demand more inclusive science. She expresses surprise that they aren't already doing more to enforce the NIH's mandate. In the longer term, training for medical students should include a fuller awareness of the role that can be played by sex differences in research.
In a 2014 letter to the journal Nature, Janine A. Clayton and Francis S. Collins of the NIH admitted the problem even extends to female researchers. Noting that roughly half of the scientists doing NIH-funded research are women: "There has not been a corresponding revolution in experimental design and analyses in cell and animal research — despite multiple calls to action."
Another possible explanation
Image source: Ousa Chea/Unsplash
There are some researchers who feel that a greater inclusion of women and their data in studies would unnecessarily complicate the problems inherent in designing research and getting it funded.
In a 2015 letter to the journal Science, a group of researchers wrote that sex considerations added an additional investigational layer to research, one that was often irrelevant to the purpose of a research project. They asserted that, "nonhypothesis-driven documentation of sex differences in basic laboratory research is more likely to introduce conceptual and empirical problems in research on sex and gender than bring new clarity to differences in men's and women's health outcomes."
The writers also suggested that sex may be less of a biological variable than gender and weight. If, for example, women are more likely to be taking multiple pharmaceuticals than men and tend to be lighter in weight, these factors may be more influential on experiment outcomes than sex. Reluctant to commit to considering sex as a variable, they suggested instead two generalized studies to determine if it should be, writing, "we see a stronger empirical basis for directed funding initiatives in two areas: scientific validation of preclinical models for studying human sex differences, and human studies of the interaction of sex- and gender-related variables in producing health outcomes that vary by sex."
Image source: Valeriy Lebedev/Shutterstock
A 2019 analysis by Harvard University's GenderSci Lab found that basic science researchers, "repeated again and again that their experiments were in large part constrained by practicalities of various sorts. These practicalities were often used to explain why they don't or can't account for sex in their research," says the lab's Annika Gompers. Among the practicalities noted were the acquisition of study materials such as cells from deceased patients, test animals, fat from cosmetic surgery patients, and so on. Gompers said researchers often simply work with what they can get.
She adds, "While my participants recognize that considering sex can be important for the generalizability of results, in practice it is often impractical if not impossible to incorporate sex as a variable into biomedical research. Such a finding is consistent with scholars who have long looked at science as practice and observed how practicalities — as mundane as the availability of materials — are often central to the reduction of complexity into 'doable problems.'"
As far as sample composition goes, the choice of subjects may have to do with researchers wanting to avoid the constraints and costs of the safety regulations that accompany studies of pregnant women, women of child-bearing age whom may become pregnant, children, and the elderly.
Finally, though it may be that having enough females in a sample to draw valid conclusions would likely require larger participant cohorts. Woitowich's co-author, Smith College's Anneliese Beery, says that fears of doubled sample sizes are overblown, asserting that such increases in participant numbers would be "not actually necessary."
Avoiding wasted research opportunities
One of the authors of that Science letter was Harvard's Sarah S. Richardson, who suggests a sort of middle path, though it does give researchers license to ignore the NIH requirement as they see fit. Richardson proposes something she calls "sex contextualism," which is the "simple view that the definition of sex and sex-related variables, and whether they are relevant in biological research, depends on the research context."
Science journalist Angela Saini agrees , saying, "While it's valuable to include a broad spectrum of people in studies, it doesn't necessarily follow that the sex differences will be significant or important. So disaggregating for sex, while useful sometimes, doesn't always matter."
The above points, however, don't seem to acknowledge the potential for findings important specifically to female health, and seem more concerned with protecting the efficacy of studies that benefit males.
In any event, Woitowich finds that things are progressing more slowly than the NIH and others may have hoped. While Beery says it's "exciting to see increased inclusion of female subjects across so many different fields of biology," there are potentially meaningful scientific insights being lost. The disinclination toward fully collecting and analyzing female data for research experiments "means we are still missing out on the opportunity to understand when there are sex differences and losing statistical power when sex differences go unnoticed."
Do you know the implicit biases you have? Here are some ways to find them out.
- A study finds that even becoming aware of your own implicit bias can help you overcome it.
- We all have biases. Some of them are helpful — others not so much.
When we talk about a bias, what we're talking about, as Harvard University social psychologist Mahzarin Banaji puts it, is a shortcut our brain has created so that we don't have spend time and energy thinking about how we feel each time we encounter something — we have an opinion already formed and ready to use.
Many of these shortcuts are useful: A bias against hangovers, for example, has one refusing alcohol without having to think about it. The problem is the brain does a lot of this shortcutting, silently. What's more, it creates shortcuts for people different than ourselves, sometimes based on actual personal experience, but often based on incorrect information we've unknowingly absorbed: other peoples' opinions, media depictions, cultural attitudes, for instance.
Worst of all, this kind of bias may be created and deployed without our even being aware of it — it's implicit in our actions in spite of ourselves and our conscious intentions.
Our brains don't always get things right. We make errors in judgement all of the time. An accurate bias is a great time-saver. An inaccurate bias is a serious problem, especially if it causes us to unknowingly discriminate against others. For instance, the systemic assumptions about women that keep them from advancing in scientific fields.
How we can curb the effects of implicit biases
Image source: Radachynskyi Serhii / Shutterstock / Big Think
New research, published in Nature Human Behavior on August 26, suggests the gender bias, which continues to prevent women from advancing in science, has a lot to do with its hidden underbelly — human blindspots. During the study, French researchers discovered that more women were promoted after the scientists in charge of awarding research positions became consciously aware of the impact of their implicit bias.
When it was no longer being highlighted, their biases discriminatory effect re-asserted itself, with award grants regressing to their traditional, pro-male pattern. Other research suggests that diversity training doesn't really help and may even exacerbate the problem it seeks to address.
We can glean a new approach, though — one that could result in better outcomes — from the new research.
About the study
Image source: Tartila/Shutterstock/Big Think
What the new study encouragingly reveals is that a conscious awareness of one's own hidden bias can mitigate its effect. The mechanism, it would appear, is that awareness may not delete the bias so much as make it less implicit, or unconscious.
The study looked at the awards handed out during annual nationwide competitions for elite French research positions. There were 414 people on the committees altogether, assessing candidates' worthiness across a spectrum of research specialties — "from particle physics to political sciences." The study analyzed committee-level data without digging too deeply into whether a committee was internally gender-balanced. The assumption was that the consensus decision reached by group represented the outcome of its internal makeup, whatever that may be.
The study took place over two years. In the first year, committee members were given Harvard's implicit association test (IAT), which established there was a significant implicit gender biases among them. Nonetheless, that year, the influence of such biases appeared to be significantly suppressed in the awards the committees handed out.
To the researchers, this outcome suggested that simply being aware of one's own implicit biases may take away their invisibility — the callout could make the bias more apparent and, therefore, something that can be more readily over-ridden.
The second year of the study, from the subjects' point of view at least, was quite silent. The researchers were still watching, but the issue of implicit bias wasn't called out. What ended up happening? The committee members returned to awarding more positions to men than women. A regression, it seemed.
It should be said, there are some possible flaws in the study: Perhaps the committee members were simply on their good behavior the first time around — until they thought that they were no longer being observed. Additionally, the study notes that there were more male submissions to the committees than female, which could skew the test. Further studies will need to be done to get a more accurate picture.
Nonetheless, the study's authors do conclude that becoming aware of one's own implicit biases may be the first step — maybe the most essential step — needed to overcome them.
How do I know if implicit bias is affecting my judgement?
Image source: AlexandreNunes / Shutterstock / Big Think
While the study looked at gender bias, of course, it's not the only variety to be concerned about, others pervade our culture: race bias, ethnicity bias, anti-LGBTQ bias, age bias, anti-Muslim bias, and so on. There are a couple of online methods available for sussing out our own. Note that if the researchers are correct, then just making yourself aware of your implicit biases can help you combat them.
The IAT mentioned above is one widely used way to identify your own bias issues. Project Implicit — from psychologists at Harvard, the University of Virginia, and the University of Washington — offers a self-test you can take. Be aware, though, that the IAT requires multiple tests to produce a meaningful result.
If you're willing to invest a little time, there's also the "bias cleanse" offered by MTV in partnership with the Kirwan Institute for the Study of Race and Ethnicity. It's a seven-day program aimed at helping you sort out implicit gender, race, or anti-LGBTQ biases you may be harboring. Each day you receive three eye-opening email thought exercises, one for each type of bias.
Side note: Did you know that more people die in female-named hurricanes because they're typically perceived as less threatening? We didn't.
It's a well-worn bromide that simply acknowledging you have a problem is the first step to solving it, but the new study provides supporting evidence that this is especially true when dealing with implicit biases — a pernicious, stubborn problem in our society. Our brains are clever beasties, silently putting together shortcuts that reduce our cognitive load. We just need to be smarter about seeing and consciously assessing them if we can ever hope to be the people that we hope to be. That may mean, on occasion, being humble enough to receive feedback in the form of callouts.
Stereotyping isn't about "bad people doing bad things." It's about our subconscious biases, and how they sneak into organizational structures.
Psychologist Valerie Purdie Greenaway is the first African American to be tenured in the sciences at Columbia University, in its entire 263 year history. Despite her celebrated position—and, in fact, perhaps because of it—she still struggles with perception, subtle stereotyping, and the enormous stakes of being one of few women of color in a leadership role. Here, Valerie Purdie Greenaway speaks with diversity and inclusion expert Jennifer Brown about being "the only" in a workplace, whether that is along lines of gender, race, culture, or sexual orientation, and how organizations and individuals can do more to recognize and address their biases. That also means letting go of the idea that stereotyping is a malevolent case of "bad people doing bad things." What does discrimination really look like day to day? Most of it is subconscious, subtle, and is deeply embedded into the structure of organizations, which can have an impact on performance, mentorship, and staff turnover. Do you recognize any of your own behavior in this discussion? This live conversation was part of a recent New York panel on diversity, inclusion, and collaboration at work.
AI is leaving human needs and democracy behind in its race to accomplish its current profit-generating goals.
It doesn't have to be this way, but for now it is: AI's primary purpose is to maximize profits. For all of the predictions of its benefits to society, right now, that's just window-dressing—a pie-in-the-sky vision of a world we don't actually inhabit. While some like Elon Musk issue dire warnings against finding ourselves beneath the silicon thumbs of robot overlords, the fact is we're already under threat. As long as AI is dedicated to economic goals and not societal concerns, its tunnel vision is a problem. And as so often seems to be the case these days, the benefits will go to the already wealthy and powerful.
Right now, while artificial intelligence is focusing on profit-generation, natural intelligence has proven to be more than up to the task of manipulating it, as if sneaking up behind someone distracted by a shiny object.
We're coming to understand just how adroitly AI can be played as we learn more and more about Russia's manipulation of social media during the 2016 presidential election. Facebook's much-lauded AI was working to “consume as much of your time and conscious attention as possible," as Facebook's first president Sean Parker recently put it to Mike Allen. After all, as we've often been told, “You're not the customer—you're the product" meant to draw advertisers to the platform. Cleverly parsing our newsfeeds for clues to our most addictive interests and associations, Facebook's AI somehow completely failed to notice it was being gamed by Russia, as noted in this stunning exchange between Senator Al Franken and Facebook General Counsel Colin Stretch:
What neither man explicitly says is that it was not the job of Facebook's AI to do anything but maximize the platform's profits. Democracy? Not Facebook's problem—until it was. Stretch's classic tech-speak/euphemism is that Facebook's algorithms should have had a “broader lens."
This lack of a broader lens is at the root of growing concerns that automation is going to mean the loss of a significant number of jobs. Katherine Dempsey, writing for The Nation, discussed the issue via email with deep-learning expert Yoshua Bengio, and he summed up the end game this way:
“AI will probably exacerbate inequalities, first with job disruptions—a few people will benefit greatly from the wealth created, [while] a large number will suffer because of job loss—and second because wealth created by AI is likely to be concentrated in a few companies and a few countries."
The future currently under construction is frightening if you're not among those few people. Dempsey cites a McKinsey & Company report, 'A Future That Works', describing a time in which fewer actually will. According to that report, 51% of all the work done in the U.S. economy could be automated at a savings for companies—and loss in workers' salaries—of $2.7 trillion. While only about 5% of all occupations could be fully automated, about a third of the work in 60% of them can be taken over by machines.
Dempsey also notes that AI is reinforcing existing biases. Its mistakes may be attributable to the narrowness of programmers' intentions and sensitivities, or not, but the algorithms are just not that smart so far. The New York Times cites Google Photos tagging black people as gorillas, the algorithms in Nikon cameras assuming Asian people are blinking, and a terrifying expose by ProPublica revealing that AI is being used to identify future criminals.
A Princeton study found that a “machine-learning program associated female names more than male names with familial attributes such as 'parents' and 'wedding.' Male names had stronger associations with career-related words such as 'professional' and 'salary.'" No surprise then that, as a Carnegie Mellon study found, Google is targeting ads for high-paying jobs primarily at men. Still, as Michael Carl Tschantz of the International Computer Science Institute admits, “We can't look inside the black box that makes the decisions."
And there's the problem at its basic level. As long as AI is primarily dedicated to advancing economic goals, its workings are likely to remain largely proprietary and thus unavailable for scrutiny—that's assuming its creators even know how it works. Our best—and maybe only—defense against this danger to our society is to educate ourselves and our children about AI and machine-learning technology so we aren't treating AI as some sacred form of modern magic whose workings and effects we're forced to unquestioningly accept. Forget robot overlords for now—it's the short-sighted greed of our human ones that should worry us.