Brian,
It seems that since I've posted this twice without testing, I'd learn that doesn't work ;-)
I did a complete testing cycle and found two errors which you've pointed out. The import commands were not correct in the serialload.m file. Also, I had not included an update to the newly compiled Serial.dll (the one in the source directory was updated as you observed).
I got out my voltmeter and carefully probed the DTR and RTS lines and observed the successful toggle of the state of those lines from the FreeMat command line:
serialload
serialPortOpen('COM1', '57600,n,8,1')
setPortRTS(1) % Measured 7.62 V on pin 7 (I did this on a laptop with a 9 pin serial port, so the serial voltages are low)
setPortRTS(0) % Measured -5.91 V on pin 7
setPortRTS(1) % Measured 7.62 V on pin 7
setPortDTR(1) % Measured 7.62 V on pin 4
setPortDTR(0) % Measured -5.91 V on pin 4
setPortDTR(1) % Measured 7.62 V on pin 4
serialPortClose
Everything appears to work (at least on Windows7 64bit with FreeMat 4.1), if you will simply download it again.
Sorry, I didn't put forth the appropriate level of effort earlier.
Jonathan