Yes, true frequency-based trips (exact_times=0) are tricky in GTFS Realtime. Unlike normal or exact_times=1 trips, each real-time trip instance is materialized in real-time because they aren't previously defined in the GTFS schedule data (hence UNSCHEDULED).
Hmmm, that's not good - it's an error.
The canonical spec on GitHub outlines how UNSCHEDULED should be used:
Some relevant sections:
When the trip_id correponds to a frequency-based trip defined in GTFS frequencies.txt, start_time is required and must be specified for trip updates and vehicle positions. If the trip corresponds to exact_times=1 GTFS record, then start_time must be some multiple (including zero) of headway_secs later than frequencies.txt start_time for the corresponding time period. If the trip corresponds to exact_times=0, then its start_time may be arbitrary, and is initially expected to be the first departure of the trip. Once established, the start_time of this frequency-based exact_times=0 trip should be considered immutable, even if the first departure time changes -- that time change may instead be reflected in a StopTimeUpdate. If trip_id is omitted, start_time must be provided. Format and semantics of the field is same as that of GTFS/frequencies.txt/start_time, e.g., 11:15:35 or 25:15:35.
>The trip_id field cannot, by itself or in combination with other TripDescriptor fields, be used to identify multiple trip instances. For example, a TripDescriptor should never specify trip_id by itself for GTFS frequencies.txt exact_times=0 trips because start_time is also required to resolve to a single trip instance starting at a specific time of the day. If the TripDescriptor does not resolve to a single trip instance (i.e., it resolves to zero or multiple trip instances), it is considered an error and the entity containing the erroneous TripDescriptor may be discarded by consumers.
>UNSCHEDULED - A trip that is running with no schedule associated to it - this value is used to identify trips defined in GTFS frequencies.txt with exact_times = 0. It should not be used to describe trips not defined in GTFS frequencies.txt, or trips in GTFS frequencies.txt with exact_times = 1. Trips with schedule_relationship: UNSCHEDULED must also set all StopTimeUpdates schedule_relationship: UNSCHEDULED
UNSCHEDULED should only be used with exact_times=0 trips. And, when providing TripUpdates, you should provide the trip_id, start_time (based on the above highlighted definition - it's materialized on-the-fly and is somewhat arbitrary), and start_date - the combination of these field is your stable identifier and shouldn't change while that same vehicle is serving that same trip.
If you have a loop route, your start_time should change based on the loop iteration that the vehicle is serving when it reaches the end of the GTFS trip. Note that as the vehicle nears the end of one loop iteration, you can (and should) have two simultaneous TripUpdates for loop routes that both refer to the same vehicle but different loop iterations (loop that is finishing and upcoming loop) with two different start_times. This avoids "pop-in" for predictions for travelers waiting at the first few stops in the trip.
Does that make sense?