Green - actual employed. Red - not included in workforce.
Looking at such data it is easy to understand that most unemployment changes are related mostly to number of not included in workforce. For Japan you even can find some logic in this - more people are going to pension in last period. For US it is more complicated as such approach is main technique to hold unemployment numbers in last years.