It's possible, but for it to work the SOC would need to have usb
hardware capable of operating in device mode, and it would need to
have whatever sort of kernel USB hooks the (device side half of) adb
is used to talking to. IIRC adbd is statically linked (for security,
and because /system may not be mounted when it starts), so there's a
chance that with appropriate kernel support and hardware you might not
even have to rebuild adbd if your arm linux has a compatible abi with
android, but you are likely to end up building a kernel or modules.
This would probably be a better fit for android-porting, even there
it's a bit off-topic as you aren't actually building something for
android.