What you are asking for isn't really the job of Prometheus.
Having a metric detailing the number of pending orders &
alerting on that is completely within the normal area for
Prometheus & Alertmanager - observing the system and alerting
if there are issues that need investigation. However the next step
of dealing with the individual events/orders is the job for a
different system. If paymentEngine could be a small number of
options (e.g. PayPal, Swipe, Cash) then it would be reasonable to
have that as a label to the pending orders metric (which then
would allow you to alert if one method stops working), but order
ID isn't something you should ever put in the metrics. Instead
once you were alerted about a potential issue you might query your
order database directly or look at log files to dig into the
detail and figure out what is happening.
-- Stuart Clark