Jumplist oddity

[Axel Bender]

Given two files A, and B, with the following jump lists:

A:
3: Text A2
2: Text A1
1: <file_B>
>

B:
3: Text in B2
2: Text in B1
1: <file_A>
>

Starting from B, how would I ever be able to reach position "Text in B1" using <C-o>?

And - as this is a rhetorical question - how could we improve this (ping-pong) behavior that - under the specified circumstances - renders the jump lists useless?