I think it also helps them figure out which videos keep people on YouTube longer. If I scroll to a section of the page that has 6 videos, and I stare at them for 10 seconds, then scroll down, they'll know that one or two of those videos must have been somewhat interesting. But if I stare at 6 videos, then scroll away 2 seconds later, it knows that nothing in that batch was worthwhile.
The fewer videos they have in focus at a time, the more accurate their algorithms can be.