Hi,
As you are probably aware
there is a 50x50 limit on the GUI version and a 200x200 limit on the standard Shetran version. See: http://research.ncl.ac.uk/shetran/faq.htm.
If the number of grid squares is greater than the 50x50 limit in the GUI version they need to be reduced. This can be done by using a GIS (either ArcMap or QGIS) and using an aggregate tool. You can keep the finer resolution by using the standard version of Shetran. This has an Easy Setup so you can still use the DEM and Mask files you have produced.
I hope this helps. If not can you give me more details about the grid resolution you are using and the number of rows and columns in the mask and DEM.
Regards,
Steve