Job loss comes with large present value earnings losses which elude workhorse models of unemployment and labor market policy. I propose a parsimonious model of a frictional labor market in which jobs differ in terms of unemployment risk and workers search off‐ and on‐the‐job. This gives rise to a job ladder with slippery bottom rungs where unemployment spells beget unemployment spells. I allow for human capital to respond to time spent out of work and estimate the framework on German Social Security data. The model captures the joint response of wages, employment, and unemployment risk to job loss which I measure empirically. The key driver of the “unemployment scar” is the loss in job security and its interaction with the evolution of human capital and, in particular, the search for better employment.