Why do the rich take more financial risk and earn higher returns? In this paper I argue that understanding the interdependence of optimal housing decisions, debt taking and portfolio allocation over the wealth distribution is key to explain this robust empirical pattern. Calibrating a rich life-cycle model to match saving and home ownership profiles over age in Swedish administrative data I find that the model predicts portfolio choice patterns over the wealth distribution which provide a good fit to data. A crucial driver is the wedge between borrowing and lending rates: the lower risk premium due to debt costs affects not only indebted households, but also those who lack a large buffer of liquid savings and thus are in the risk of becoming indebted in the future. This mechanism is magnified by life-cycle effects and the lower optimal leverage ratio of the rich. I decompose the effect of different channels and also analyze the response of the economy to shocks in partial equilibrium. Changes in housing market conditions, such as the mortgage rate of house prices have a substantial effect on the demand for risky assets and on wealth inequality.