Hi Phil,
The simplest program that I know that can do the job for you is:
MAPC2MAPC
The cost is $25 USD. You can import the image and calibrate (georeference) it there in the software. It runs in two modes depending on what you prefer. Wizard mode has big buttons on the GUI and in advanced mode it has drop-down menus with many more options. If you're in wizard mode, load the image (button - Load an image file and calibrate it) > (button - Calibrate with map coordinates). In the window (view map at xx% shown in title bar at top), choose your points and put in your coordinates, then close the tie-points window (close with "X" in top-right of window). Then click the button "Save the map for a mobile application", then select from list: Garmin Custom Map KMZ.
To work in advanced mode: Load the image: File > Load image for calibration > From Map Coordinates Then export: File > Write Garmin Custom Map (KMZ).
Hopefully with either of these methods, you'll get a kmz file that works for you in the Custom Maps app. However, once when I tried this, I got a folder exported instead of a single kmz file, and if that's the case, follow the instructions below:
For a folder with a .kml file and an image in a subfolder, compress the result into a zip compressed file, with the zip file only containing one folder, (the folder inside is called files or images). The kml file probably will be named: "doc.kml". Change the resulting compressed file extension from .zip to .kmz and it hopefully works in the Custom Maps app.
Otherwise you can use QGIS (free but a big install on your computer and there's some learning of course...)
Or an expensive paid program such as Global Mapper or ArcGIS. The programs will do a lot of GIS functions, however MAPC2MAPC does geo-referencing and conversion of different map formats for mobile app use and that's all it does.
You could probably do it in Google Earth also, however if you try in Google Earth, you would be able to enter your specific tie points, so you'll have the same problem that you had with doing it in the app (Custom Maps).
Another app that will load and display this kind of georeferenced map on Android is called OruxMaps and it has a lot of powerful functions. It will also load georeferenced maps in other formats, such as geospatial PDF or GeoTIFF. Other apps that will load this kmz type of georeferenced map are: Google Earth and QField. If you get into this GIS work more I'd highly recommend ArcGIS on PC and QField on phone. They're both free and quite powerful.
One day if the developer (Marko) has time we'll get more functionality in the Custom Maps app. The ability to define tie-points with coordinates would be a wonderful thing, and as far as I know, that would make it be the only Android app that would have that ability.
Cheers.