Workers who are displaced from their job experience a well-documented scarring effect: a large and persistent average earnings loss. However, these average effects mask a substantial amount of heterogeneity along a number of observable dimensions. In this paper, I explore how the scarring effect of job displacement differs by the affected workers' earnings prior to displacement. I use detailed administrative data from Germany to empirically analyze this dimension. I find that earnings losses, relative to pre-displacement earnings, are larger for individuals whose recent earnings situate them at the bottom of the earnings distribution. This seemingly contradicts existing models that can account for the average scarring effect, as these are generally based on the idea of a job ladder, and thus imply that workers at the top of the earnings distribution should suffer from (relatively) larger earnings losses. I then propose a model in which displaced workers do not necessarily move into unemployment, as they may find a job immediately upon being displaced. Furthermore, they do not fall off the ladder completely if they find a new job immediately. I show that this framework enables the model to explain the relatively larger earnings losses at the bottom of the earnings distribution. These larger losses are primarily driven by the incidence of the job-to-job transitions upon displacement and the heterogeneity in separation rates in subsequent jobs, amplified by larger accumulation of human capital loss among workers at the bottom of the earnings distribution.