Mm, this is kinda a problem intrinsic to raycasting, raycasting works by moving X distance in a certain direction and testing if there's a wall there, then just looping until a target is found, so this problem will occur.
You can however filter the raycast and check if it is actually valid, my idea would be something like this.
var step:Int = tileWidth;
var p:FlxPoint = FlxPoint.get();
var t:FlxPoint = FlxPoint.get();
p.copyFrom(startPoint);
for (i in 0...Std.int(distance/step)) {
p.x += step * cos(angle);
p.y += step * sin(angle);
t.x = Math.round(p.x/tileWidth);
t.y = Math.round(p.y/tileHeight);
if (isTile(t.x + 1, t.y + 1) || isTile(t.x - 1, t.y - 1)) {
trace("The line probably only goes from $startPoint till $p");
break;
}
}
p.put();
t.put();
Obviously untest, but just step through the ray result and check if this instance has occurred.
And now that I think about it I'm surprised this is how FlxTilemap().ray() is implemented. There has to be a way to properly ray cast through a tilemap rather than this step and check method.