Ah, by nx and ny you mean the number of particles in each direction? I was assuming it meant the distance in x or y direction between the particles. Sorry for my confusion.
So, if I call the distance between particles deltax and deltay (such that nx*deltax = x and ny*deltay=y), then I meant to keep deltax = deltay = 1 (or some other number), and calculate nx=x/deltax.
If you want to do it the way you proposed it originally, then nx and ny have to be integers, right?
nx = Math.floor(sqrt(n * x/y))
ny = Math.floor(sqrt(n * y/x))
newn = nx * ny.
Then you have integer nx and ny, and newn < n and it works always somehow.
(It may be that deltax=x/nx does not equal deltay=y/ny. In this case one could adjust x or y slightly so that deltax=deltay. Also if nx=0 or ny=0 one should also adjust properly.)