I think that Pinker would say that he enjoyed living in a house like that because he's who he is. He wouldn't say that living in that house made him who he is.
In other words, the genetics made him into a person who loved exchanging ideas and such. If he'd been adopted into a home without books and ideas, he would have been miserable. That's as opposed to saying that he would have grown up to be someone who didn't love books and ideas.
The interview segment was short, so he couldn't go into the overwhelming evidence that backs up what he's saying. For a fuller explanation, check out his book, The Blank Slate.