Caffeine is often used to counteract sleepiness generated by sleep deprivation, jet lag, and shift-work, and is consumed at different times of day. Caffeine also has effects on sleep. However, little is known about the interaction between sleep deprivation, circadian timing, and caffeine consumption on sleep. In this study, we compared the effects of caffeine on nocturnal sleep initiated at habitual circadian time and on daytime recovery sleep. Thirty-four moderate caffeine consumers participated in both caffeine (200 mg) and placebo (lactose) conditions in a double-blind crossover design. Seventeen subjects followed their habitual sleep-wake cycle and slept in the laboratory during the night (Night), while 17 subjects were sleep deprived for one night and recovery sleep started in the morning (DayRec). All subjects received a capsule of 100 mg of caffeine (or placebo) 3 h before bedtime, and the remaining dose 1 h before bedtime. Compared to placebo, caffeine lengthened sleep latency, increased stage 1, and reduced stage 2 and slow-wave sleep (SWS) in both groups. However, caffeine reduced sleep efficiency more strongly in the DayRec group, and decreased sleep duration and REM sleep only in that group. The stronger effects of caffeine on daytime recovery sleep compared to nocturnal sleep are probably the consequence of the combined influence of increasing circadian wake propensity drive and the dissipation of homeostatic sleep pressure. We propose that the reduction of SWS by caffeine during daytime sleep increases the impact of the circadian wake signal on sleep. These results have implications for individuals using caffeine during night time.