The main slowdowns will likely come from device initialization and the bootloader.
Bootloaders need to initialize most of the devices and load the kernel image. Then they hand the control over to Linux which proceeds to re-init these devices again.
The userspace matters, but on recent computers it doesn't matter that much.
You can get to sub-40ms with https://katacontainers.io/ That's a project that uses full VMs to run Docker images boot instead of kernel namespacing.
This _is_ possible with Linux, but not at all trivial and likely impossible with general-purpose distros.
Interrupt handling and (on RP2040) dedicated multicore code is also nice.