Keep in mind that RISC-V is still a new architecture, with limited history, and lots of stuff is still changing in occasionally incompatible ways. I doubt that anything before 4.15 would work, and while this kernel works, it is known to have significant bugs. You need something around 5.8 at least, maybe 5.10, to have a stable system. While you can probably find old RISC-V linux releases that support a 4.x kernel, they probably won't work with a distro built for a 5.x kernel. A lot of stuff has changed, the bootloaders (bbl versus u-boot/opensbi), the syntax and structure of the DTS file, the set of system calls officially and unofficially supported, the vdsos supported, the hardware/errata supported, etc. If you really need a 4.x kernel, then you will need to look for an old and unsupported RISC-V linux distro, and accept that it is too broken for production work. Maybe something like this from 2018
I don't know of any Ubuntu releases old enough to have a 4.x kernel. Fedora was one of the first ones to get working RISC-V support. Debian was also early, and might also have old obsolete broken images with a 4.x kernel, but I don't know where to find them offhand.
ARM and x86 are 30+ years old. Give us another 10 years or so, and we will have a software environment as stable as them, that allows changing linux kernel versions.
Jim