Hi Karolos, and thanks for contacting me (and sorry for the delayed response).
Custom Maps app is designed to display one map image at a time. It does not have built-in zoom level management logic for tiled maps. The map image size limit for the app really comes from the Android OS memory limit for apps. Custom Maps only fails to use a map image when Android does not allow the app to allocate enough memory to load the whole image into memory all at once.
I originally designed it for displaying maps for US state and national parks, as you can often find the map images online either in jpg of PDF format. While the app does not support PDF files, the PDF images can be converted to jpgs easily on laptops or desktops.
I have found that most maps that I want to use in Custom Maps come in PDF format, and they are typically not much larger than a letter size (or A4 size) paper. And rendering those images at 300 dpi results only in about 8 megapixel image which can be easily handled by Custom Maps. Even if the map is double the size (two letter size pages or A3), it would render to little over 17 megapixels image which Custom Maps should be able to handle on most modern smartphones. And 300 dpi is high enough of a resolution that even very small text (3 pt) will be legible. And I don't think there are any printed materials that use text as small as 3 pt (3/72 inch). Basically, you should never need to use higher resolution than 300 dpi.
What is the source of your map images that they are too large for Custom Maps? If at all possible, I'd recommend converting the map images to smaller resolution (~16 megapixels) in a graphics program on a laptop or a desktop. Or if that would make the text illegible on the map image, then break your map images into approximately 16 megapixel sections, and switch between the map images as you travel around the mapped area.