This study explored how the life expectancy in Japan was affected by the COVID-19 pandemic from 2020–2021, where the epidemic size was relatively small from a global perspective at least until the end of 2021. Our life table analysis of 2019, 2020 and 2021 showed that there was a slight extension of life expectancy in 2020, occurring in 44 out of 47 prefectures. The epidemic increased in size in 2021, and the life expectancy was slightly shortened in 34 out of 47 prefectures.