Currently for every past version that is being used as an update_from package, dependencies are installed and instantly deleted afterwards.

Since composer caches most dependencies, this is almost not noticeable. In earlier versions of 3.2, however, we installed dependencies by referring to their commits. These downloads are not cached by composer and thus have to be re-downloaded every single time. (I'm also talking about nightly builds by our servers)
While on 3.1 I couldn't see almost no build time decrease (4-5%), I surely could see some on 3.2 and 3.3 (Up to 25%. I've measured the times including tests)