There are essentially two ways to make money on Wall Street. The first—let's call it the old-fashioned way—is to match people who have money with people who can use it to make or do something. This is how Wall Street really earns its money. By facilitating investment and diversifying risk, the finance sector adds enormous value to the overall economy. When it ceases to perform this function and credit dries up—as it partially has in this recession—the economy grinds almost to a halt. As galling as it was to have to bail Wall Street out, if we hadn't, we would have found out just how vital what Wall Street does is to the economy.
The second way to make money is to inflate the value of something without adding any real value. Instead of betting on which investments are likely to yield the highest return, you gamble on which are likely to attract other investors. Because even if nothing of value is created the price of an asset can go up. All that has to change is our collective estimate of its value. While bubbles are often called "irrational" because they don't reflect any real change in the value of the asset, it isn't necessarily irrational to invest in something that's inflated this way. You can get rich doing it—provided you get out before the bubble bursts.
Bubbles are the market equivalent of Ponzi schemes, although they are generally as much the product of collective enthusiasm as of out-and-out fraud. In Ponzi schemes—like the one Bernie Madoff used to bilk investors out of billions of dollars—no money is actually invested. Instead, money from people who buy into the scheme later is used to pay off the "investments" of the people who buy into it earlier. Because no money is actually made—it's only shifted from one person to another—such schemes are unsustainable. When there is no one left to bring into the scheme, it collapses. Market bubbles work essentially the same way. Fortunes are made as the bubble inflates, but since not much of value is being produced the bubble can't keep growing forever. Without a continuing supply of fresh money it collapses, and whoever is left in the market takes a financial beating.
Until the 1990s, the financial industry never made more than 20% of domestic corporate profits. But in this decade that figure rose to a staggering 41%. It's hard to argue that the financial sector ever actually accounted for that much of our national value-added. Instead, these new profits were earned largely on paper, an artifact of the inflated prices of financial instruments like mortgage-backed securities. The industry was playing a zero-sum game, gambling pure and simple. Much of the apparent growth of our financial sector, in other words—much like in the dot-com bubble of the 1990s—was an illusion created by the money pouring into Wall Street.
Bubbles like this are intrinsic feature of free markets. They have the roots in the early days of capitalism, going back at least as far as the early 17th century, when collapse in the price of tulip bulbs created a panic in the Dutch stock exchange. My own family was involved in a bubble when the collapse of commodity prices following the end of the Seven Years' War in 1763 led to the failure of the highly-leveraged de Neufville Brothers banking house and precipitated a crisis. The problem is that there's no surefire way to distinguish between real investment and market hype, no way to know whether the price of an asset really reflects by its intrinsic value. So it is probably too simple to blame the latest crash on any particular type of financial instrument or institution, since we didn't need credit default swaps to drive up the price of tulips.
The more fundamental issue is that the financial industry had very little incentive to avoid such crises. While many people on Wall Street fooled themselves into believing that the party would never end—perhaps simply to help themselves sleep at night—they were never stupid. When the music stopped some of them would lose money, but it was money which they wouldn't otherwise have made anyway. And before that happened many more would make their fortunes and get out. So it was never that they couldn't evaluate the risk properly if they really wanted to, but rather that the risk was never really to them. The financial industry is simply too important to the economy—not to mention too influential—for us to allow it to collapse. Once we were taken in by the financial markets' scam, we couldn't get our money back.
We may not be able to avoid speculative bubbles. But we can take steps to make sure the people who gamble with our money bear a greater share of the risks. We can diversify our risk by making sure that banks are not too big to fail, and that the collapse of one will not necessarily lead to the collapse of all the others. Part of the reason this bubble did so much damage to our economy is that the deregulation of the banking industry in the eighties and nineties allowed banks to get bigger and to take on greater risks. The truth is that because of its importance to our economy, the financial sector is a public institution in the same way that utilities are. Greater regulation will undoubtedly make it less efficient in the short run. But in the long run regulation may be necessary to make sure the economy runs more smoothly. And without real regulation this will happen again, and sooner rather than later.