Ever seen someone and immediately thought that they look “shady”? Sure, we all have. Well, it could be based on cultural bias that depends on the density of the place you live. If you live in a rural area you’re more likely to assume that “someone that doesn’t look like [you]” is less trustworthy (take that how you may). City dwellers have less of this bias simply because they’re around so many other people, which is a new thing in human history. Alexander Todorov is the author of Face Value: The Irresistible Influence of First Impressions.

Alexander Todorov: So in the early days, including from the time of Aristotle and later in the 16th and 17th century most of physiognomy consisted of this whimsical comparisons between the physiognomy of humans and animals. 

So for example, you would have a drawing of a human who presumably looks like a cow, and from there you will make all kinds of inferences that perhaps the character of the person matches the character of the cow—whatever that might be. 

In fact if you look at the history of European history most of the 19th century's novels, a standard feature of this analysis that you have physiognomic descriptions of the characters. 

So it was very, very popular and influenced not only marginal writers but big names like Balzac, Stendhal, and many others.

Now, interestingly enough in the early 20th century people no longer talk about physiognomy they talk about character analysis. And in fact a lot of the references are no longer to Lavater's whimsical ideas, but they're to evolutionary ideas. And the so-called character analysts, they were quite influential and they were involved heavily in business and recruitment of employees, but this is exactly the time when the new science of psychology arises and then psychologists are kind of skeptical about the claims of the character analysts—or really the new physiognomists. 

The fact that we agree on this impression was discovered over 100 years ago in psychology, but psychologists at the time were really focused on the accuracy of the impressions and paid very little attention to the extremely interesting psychological fact that we actually agree on these impressions.

Very often in psychology, and generally in the social sciences lately, if you observe that there's a pervasive bias that is: it's something that feels fairly automatic and we can all do it. There is kind of almost immediate assumption that this might be actually wired, that it's something that we are born to be able to do. 

Of course you can easily think of counterexamples like driving, which essentially becomes automatic and there's nothing evolutionary about driving—or reading. 

But nevertheless things that seem to have been present always in our environment like faces, that seems like natural assumptions. 

It's interesting actually, we've done some studies—there are many, many different inputs to impressions. One is emotional expressions, there's stereotypes about gender, there's cues about age and facial maturity, all of this go into our impressions. Another one that is very interesting is typicality. 

So as it turns out we tend to like faces that are typical, that means faces that are closer to what we perceive as typical in our social environment. Now there's an interesting wrinkle because typicality is also culturally specific, especially if the different cultures are linked to different ethnicities and there's distinctive physiognomies, and that makes it worse. 

We've done a study where we created morphs of a typical Japanese face and a typical Israeli face, and then we can interpolate the morph. So we can imagine like a typical Japanese face gradually turning into a typical Israeli face.

Now, if you ask Israeli and Japanese participants to evaluate the faces, what happens is as the face become more Israeli-looking the Israelis believe the face is becoming more trustworthy, and the other way around for the Japanese. 

So in a sense to a large extent what we perceive as typical is shaped by our natural environment. And it's something that we very rapidly extract, we are incredibly good learners about faces. 

And most likely people who live in New York City, with a hugely diverse face, will have a different notion of typicality if you live in a small rural town where there's not so much diversity. 

And in this case this can lead to different kind of suboptimal outcomes because naturally we wouldn't trust people that do not look like us, not having any other information. 

The reason why we will never be able to get rid of first impressions is because they serve important psychological functions. 

That is in the absence of any other information we're trying the best we could to figure out what the other people are thinking. That doesn't mean that we wouldn't change our minds, on the contrary when you have good diagnostic evidence about the person or when you know about past behavior, that would change inferences based on appearance. 

But most of the time if you don't have any other information people will act on this, and that may not to be in their best interest. 

Starting back to Cesare Lombroso who wrote books like The Criminal Man and The Criminal Woman, and he claimed that he can identify these “inferior types” based on their facial features, to Francis Gaulton who invented composite photography, and in fact all of today's morphing methods are based on this method of composite photography. And the first application of the method was to identify the criminal type, so it has a very long history.

I think a very reasonable argument could be that we are kind of hardwired to figure out the intentions of other people, of the people around us, because what is the most important thing in our social life? It's other people! 

And in interactions with strangers you're always trying to figure out what are their intentions? Are they good? Are they bad? What are they going to do? Can they hurt me, whether that’s physical or in a non-physical way? So these are things that have always been a concern for us.

But let's think in terms of evolutionary history. Well, for most of our evolutionary history we've basically lived in extended families, typically between five to eight individuals. All of these changes in the last 50,000—in fact even less—maybe in the last 20,000 years, when you have large societies—that is, if you imagine the human evolution compressed within 24 hours, we have been living with strangers surrounded by strangers in the last five minutes. 

So it's not obvious at all that we are kind of endowed with reading the characters of other faces. I think there's good comparative evidence that in fact we are very good at picking up on social cues in the immediate situations. 

So if you look, for example, at comparative studies, you look across all primates, it turns out that we are the only primate which has whites of the eyes. That is our iris is dark and then you have the white sclera, and then you have darker skin. There are no other primates with this kind of coloration. So why is that interesting or important?

Well, the fact that you have the white of the sclera makes it super easy to detect eye gaze. And eye gaze is very important for sharing social attention, we can communicate from a long distance, similar emotional expressions are very important, the fact that we are kind of the “naked ape” and the fact that our faces are not completely covered by hair makes it very easy to detect changes in skin colorations, which is often an indication of different kinds of emotional or mental states. 

So we are very sensitive to changes, momentary changes in what we are observing, because these momentary changes are indicative of what is important in this situation, what is happening right now. 

But it's hard to make the argument that somehow we are endowed also to read into faces the character of others. I think we have the natural propensity trying to figure out what these other people are thinking or feeling right now. 

And I think the problem with physiognomy in the modern version is this assumption that because you can make these rapid inferences that's also informative about these people across time and situations.