That is correct. This is by design - the extension's main goal is to limit the time you spend on YouTube which includes scrolling, reading comments, etc. and not just watching a video.

Note that it only tracks time spent if you have YouTube open as the "active" tab. It should not count towards total time spent if you have YouTube opened in the background (there is a setting for this - enabled by default), or not in the active tab.