Thank you Oleg and Dnyaneshwar!
I've had some success since my last post. I'm trying to communicate with a VFD, which up until now I've been communicating with over a USB serial connection. I'd like USB on the AI to be dedicated to other things and the device name to be guaranteed (to avoid having to figure out whether I need to communicate with /dev/ttyUSB0 or /dev/ttyUSB1, etc.), so I wanted to wire up communication over the VFD's RS485 port to a dedicated UART device on the AI. I was hoping it would be a drop in replacement, but the half duplex nature of RS485 has complicated the issue more than I had hoped. There was a combination of device tree overlay issues, circuit issues, software issues and examples that took different approaches that was confusing the issue for me. I still have some issues to work out.
My hope for a drop in replacement led me to try and leverage kernel driver level support for RS485 (rather than manually set a GPIO pin when I need to write data). I was able to combine these examples to get some basic communication working:
Various examples configured RTS as a GPIO pin, which affects how it's configured in the device tree overlay. I ultimately went with using the UART3 hardware RTS pin rather than an arbitrary GPIO pin. On the Beaglebone AI, this is P9.17a and I have it configured as PIN_OUTPUT_PULLDOWN | INPUT_EN | MUX_MODE1, the same as the TX and RX pins P9.21b and P9.22b. I'm not sure if it needs to be that exactly (or if it does, why), but it has worked for me.
Using the combination of examples linked above with the device tree overlay properly configured, I was able to monitor the RTS, RX and TX pins on a connected Arduino which showed that RTS was indeed getting pulled low prior to sending data and then being set high again after the data was done sending. To communicate with the VFD, though, I also wanted to leverage libmodbus, which seems to have its own incantations to get RS485 working that don't seem to work properly for me. This issue looks like it has valuable information about it:
https://github.com/stephane/libmodbus/issues/331
In it, there is a table of drivers that libmodbus could work with with some changes. How do I tell which driver I'm using?