Instance status keep STARTING in eureka server even it is UP on Local

[duw...@gmail.com]

More description in github issue 

PR  is here

 I created a btrace script to reproduce it.

In my case, I set the renew interval to 5s.

There is two registration flows during application starting:

1. InstanceInfoReplicator, this starts to register instance triggered by status change (ApplicationInfoManager.StatusChangeListener in DiscoveryClient).
2. DiscoveryClient$HeartbeatThread, this thread starts with delay renewInterval to DiscoveryClient initiated. It sends heartbeat request to eureka server. If response code is 404, it will do registration instead.

Both flows above will invoke AbstractJerseyEurekaHttpClient$register() and use EurekaJacksonCodec$InstanceInfoSerializer to serialize instance object. During serializing, it records status first, then records lastDirtyTimestamp.

In particularly extreme case, HeartbeatThread records status as STARTING and records lastDirtyTimestamp after status changed to UP. But InstanceInfoReplicator thread records status as UP and same lastDirtyTimestamp and send request to eureka server first.

Then registration request of HeartbeatThread overwrites status with STARTING. In subsequent heartbeat request, even status is UP but lastDirtyTimestamp are same, and eureka server won'tchange instance status to UP due to logic in InstanceResource$validateDirtyTimestamp

Below tcp dump shows the flow described above.

1. HeartbeatThread sends heartbeat request, but receive 404 (instance not register yet). then prepare to send registration request.
2. InstanceInfoReplicator sends registration request. (status: UP, lastDirtyTimestamp: a)
3. HeartbeatThread sends registration request. (status: STARTING, lastDirtyTimestamp: a)