How to form jgroups cluster on kubernetes/openshift

955 views
Skip to first unread message

John Deng

unread,
Sep 12, 2017, 7:57:14 AM9/12/17
to Axon Framework Users
Hi, I used axon framework to implement with axon-spring-boot-starter-jgroups, it works perfectly on local machine, when I deploy it to kubernetes/openshift, jgroups does not form the cluster, I found the project https://github.com/jgroups-extras/jgroups-kubernetes, it seems that it will solve my issue. but I not quite sure how to integrate it into axon framework. does any one know how to solve this issue?

Benoît Liessens

unread,
Sep 13, 2017, 7:31:12 AM9/13/17
to Axon Framework Users
Hi John,

I have no experience with kubernetes but we do use Axon and JGroups in combination with docker. Based on that I would suggest:
1) check which JGroups configuration you are using. Possibly it relies on TCP broadcast messages which might are not be supported in you environment.
2) The hosts on which we deploy Axon (and JGroup clusters) have multiple network interfaces. Occasionally, on some hosts, JGroups used a different netework interface (NIC)  to bind to which resulted in nodes not seeing each other. We where able to solve that by telling JGroups which NIC / IP address(range) to bind to. Check the JGroups docs for system property named 'jgroups.udp.bind_addr'

Cheers,
Benoît   

John Deng

unread,
Sep 13, 2017, 12:40:32 PM9/13/17
to Axon Framework Users
Hi Benoît,

Thanks for your suggestion.

I managed to make a jgroups demo works on kubernetes, however, the jgroups version must be 4.0.0 or upper according to the dependency jgroups-kubernetes. I checked the jgroups version is 3.6.0.Final that used inside axon. Could we upgrade jgroups version on axon?

Best regards,
John

Benoît Liessens

unread,
Sep 13, 2017, 12:42:32 PM9/13/17
to Axon Framework Users
John,

Just define an explicit dependency to jgroups 4 in you own build configuration. That will do.

Regards,
Benoît

Steven van Beelen

unread,
Sep 14, 2017, 3:35:28 AM9/14/17
to Axon Framework Users
Hi John/Benoît,

Everybody is always free to make pull request for the Axon Framework.
If you've got time to check out the framework locally, update the version of jgroups and check if everything still works fine, that would obviously be a great candidate for a small PR! :-)

Cheers,
Steven

--
You received this message because you are subscribed to the Google Groups "Axon Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to axonframewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Message has been deleted

John Deng

unread,
Sep 14, 2017, 11:46:37 PM9/14/17
to Axon Framework Users
Hi Steven/Benoît,

I tried to upgrade to jgroups-4.0.4.Final, and then caught the exception caused by: java.io.NotSerializableException: org.jgroups.util.UUID. Please help!

org.springframework.context.ApplicationContextException: Failed to start bean 'jgroupsConnectorFactoryBean'; nested exception is org.axonframework.jgroups.commandhandling.ConnectionFailedException: Could not start JGroups Connector
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:50) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:348) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:151) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:114) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:880) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144) ~[spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) ~[spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.4.RELEASE.jar:1.5.4.RELEASE]
at cn.vpclub.moses.product.command.ProductApplication.main(ProductApplication.java:22) [classes/:na]
Caused by: org.axonframework.jgroups.commandhandling.ConnectionFailedException: Could not start JGroups Connector
at org.axonframework.spring.commandhandling.distributed.jgroups.JGroupsConnectorFactoryBean.start(JGroupsConnectorFactoryBean.java:223) ~[axon-spring-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:175) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
... 14 common frames omitted
Caused by: java.lang.Exception: connecting to channel debug-order-product failed
at org.jgroups.JChannel._connect(JChannel.java:904) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel.connect(JChannel.java:393) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel.connect(JChannel.java:384) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.axonframework.jgroups.commandhandling.JGroupsConnector.connect(JGroupsConnector.java:152) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at org.axonframework.spring.commandhandling.distributed.jgroups.JGroupsConnectorFactoryBean.start(JGroupsConnectorFactoryBean.java:213) ~[axon-spring-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
... 15 common frames omitted
Caused by: org.axonframework.jgroups.commandhandling.MembershipUpdateFailedException: Failed to broadcast my settings
at org.axonframework.jgroups.commandhandling.JGroupsConnector.viewAccepted(JGroupsConnector.java:186) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at org.jgroups.JChannel.invokeCallback(JChannel.java:956) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel.up(JChannel.java:800) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:891) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:122) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.RSVP.up(RSVP.java:158) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FRAG2.up(FRAG2.java:163) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:338) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:338) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:710) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.ClientGmsImpl.becomeSingletonMember(ClientGmsImpl.java:273) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.ClientGmsImpl.joinInternal(ClientGmsImpl.java:83) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.ClientGmsImpl.join(ClientGmsImpl.java:41) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1066) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FlowControl.down(FlowControl.java:300) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FlowControl.down(FlowControl.java:300) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.FRAG2.down(FRAG2.java:141) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.RSVP.down(RSVP.java:102) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:206) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:901) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel.down(JChannel.java:668) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.JChannel._connect(JChannel.java:897) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
... 19 common frames omitted
Caused by: org.axonframework.commandhandling.distributed.ServiceRegistryException: Could not broadcast local membership details to the cluster
at org.axonframework.jgroups.commandhandling.JGroupsConnector.broadCastMembership(JGroupsConnector.java:133) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at org.axonframework.jgroups.commandhandling.JGroupsConnector.viewAccepted(JGroupsConnector.java:183) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
... 40 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.io.NotSerializableException: org.jgroups.util.UUID
at org.jgroups.Message.setObject(Message.java:303) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.Message.<init>(Message.java:145) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.axonframework.jgroups.commandhandling.JGroupsConnector.broadCastMembership(JGroupsConnector.java:128) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
... 41 common frames omitted
Caused by: java.io.NotSerializableException: org.jgroups.util.UUID
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) ~[na:1.8.0_144]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[na:1.8.0_144]
at org.axonframework.jgroups.commandhandling.JoinMessage.writeExternal(JoinMessage.java:77) ~[axon-distributed-commandbus-jgroups-3.1-SNAPSHOT.jar:3.1-SNAPSHOT]
at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1459) ~[na:1.8.0_144]
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1430) ~[na:1.8.0_144]
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) ~[na:1.8.0_144]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[na:1.8.0_144]
at org.jgroups.util.Util.objectToByteBuffer(Util.java:548) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
at org.jgroups.Message.setObject(Message.java:300) ~[jgroups-4.0.4.Final.jar:4.0.4.Final]
... 43 common frames omitted

Best regards,
John

John Deng

unread,
Sep 15, 2017, 12:09:59 AM9/15/17
to Axon Framework Users
Hi Steven/Benoît,

I just found the root cause, jgroups 4 removed Externalizable from io.jgroups.Address

public interface Address extends SizeStreamable, Comparable<Address> {
    // flags used for marshalling
    byte NULL         = 1 << 0;
    byte UUID_ADDR    = 1 << 1;
    byte SITE_UUID    = 1 << 2;
    byte SITE_MASTER  = 1 << 3;
    byte IP_ADDR      = 1 << 4;
    byte IP_ADDR_UUID = 1 << 5;
}

in jgroups 3, the Address is different

public interface Address extends Streamable, Comparable<Address>, Externalizable {
    byte NULL = 1;
    byte UUID_ADDR = 2;
    byte SITE_UUID = 4;
    byte SITE_MASTER = 8;
    byte IP_ADDR = 16;

    int size();
}

So, how do we solve this issue?

Best regards,
John

Benoît Liessens

unread,
Sep 15, 2017, 2:58:23 AM9/15/17
to Axon Framework Users
Hi John,

Check the last lines of the stacktrace. You will notice class org.axonframework.jgroups.commandhandling.JoinMessage is writing an instance of Address to the ObjectStream.
I guess with JGroups v4 the JoinMessage class (which implements Externalizable) shoud no longer reference the Address interface.

The Address instance inside JoinMessage is obtained from a JChannel. The latter also provides methods JChannel.getAddressAsString() and JChannel.getAddressAsUUID() which might do the trick.

Regards,
Benoît

John Deng

unread,
Sep 15, 2017, 8:07:29 AM9/15/17
to Axon Framework Users
Hi Benoît,

You are right, I fixed this issue just now, see the code below, 


@Override
public void writeExternal(ObjectOutput out) throws IOException {

    try {
        Util.writeAddress(address, out);
    } catch (Exception e) {
        logger.error("Serialize exception: {}", e);
    }
    out.writeInt(loadFactor);
    out.writeObject(messageFilter);
}

@SuppressWarnings("unchecked")
@Override
public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException {
    try {
        address = Util.readAddress(in);
    } catch (Exception e) {
        logger.error("Serialize exception: {}", e);
    }
    loadFactor = in.readInt();
    messageFilter = (Predicate<CommandMessage<?>>) in.readObject();
}


1. Start the application.

-------------------------------------------------------------------
GMS: address=order-service-3-4n76b-35634, cluster=moses-product-order, physical address=10.131.2.19:7800
-------------------------------------------------------------------
2017-09-15 19:24:38.896 DEBUG 1 --- [           main] o.j.protocols.kubernetes.KUBE_PING       : read all from Kubernetes Client[masterUrl=https://devops:8443/api/v1, headers={Authorization=#MASKED:874#}, connectTimeout=5000, readTimeout=30000, operationAttempts=3, operationSleep=1000, streamProvider=org.jgroups.protocols.kubernetes.stream.InsecureStreamProvider@19c1820d] for cluster [moses-product-order], namespace [moses-demos-dev], labels [jcluster=moses-product-order]

2. The cluster seems formed on kubernetes,


2017-09-15 19:24:39.714 DEBUG 1 --- [           main] o.j.protocols.kubernetes.KUBE_PING       : findMembers()
2017-09-15 19:24:39.971 DEBUG 1 --- [           main] org.jgroups.protocols.pbcast.GMS         : order-service-3-4n76b-35634: sending JOIN(order-service-3-4n76b-35634) to product-service-2-36mrc-34743
2017-09-15 19:24:40.053  INFO 1 --- [e-3-4n76b-35634] o.a.j.commandhandling.JGroupsConnector   : order-service-3-4n76b-35634 joined with load factor: 100
2017-09-15 19:24:40.053 DEBUG 1 --- [e-3-4n76b-35634] o.a.j.commandhandling.JGroupsConnector   : Got a network of members: [SimpleMember{name=order-service-3-4n76b-35634, endpoint=product-service-2-36mrc-34743}, SimpleMember{name=order-service-3-4n76b-35634, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 19:24:40.061  INFO 1 --- [e-3-4n76b-35634] o.a.j.commandhandling.JGroupsConnector   : order-service-3-4n76b-35634 joined with load factor: 100
2017-09-15 19:24:40.061 DEBUG 1 --- [e-3-4n76b-35634] o.a.j.commandhandling.JGroupsConnector   : Got a network of members: [SimpleMember{name=order-service-3-4n76b-35634, endpoint=product-service-2-36mrc-34743}, SimpleMember{name=order-service-3-4n76b-35634, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 19:24:41.189 DEBUG 1 --- [           main] org.jgroups.protocols.pbcast.NAKACK2     :
[order-service-3-4n76b-35634 setDigest()]
existing digest:  []
new digest:       product-service-2-36mrc-34743: [5 (5)], product-query-1-qqsvn-57792: [2 (2)], order-service-3-4n76b-35634: [0 (0)]
resulting digest: product-service-2-36mrc-34743: [5 (5)], product-query-1-qqsvn-57792: [2 (2)], order-service-3-4n76b-35634: [0 (0)]
2017-09-15 19:24:41.189 DEBUG 1 --- [           main] org.jgroups.protocols.pbcast.GMS         : order-service-3-4n76b-35634: installing view [product-service-2-36mrc-34743|8] (3) [product-service-2-36mrc-34743, product-query-1-qqsvn-57792, order-service-3-4n76b-35634]
2017-09-15 19:24:41.436 DEBUG 1 --- [           main] org.jgroups.protocols.CENTRAL_LOCK       : view=[product-service-2-36mrc-34743|8] (3) [product-service-2-36mrc-34743, product-query-1-qqsvn-57792, order-service-3-4n76b-35634]
2017-09-15 19:24:41.437 DEBUG 1 --- [           main] org.jgroups.protocols.CENTRAL_LOCK       : local_addr=order-service-3-4n76b-35634, coord=product-service-2-36mrc-34743, is_coord=false
2017-09-15 19:24:41.437 DEBUG 1 --- [           main] org.jgroups.protocols.COUNTER            : view=[product-service-2-36mrc-34743|8] (3) [product-service-2-36mrc-34743, product-query-1-qqsvn-57792, order-service-3-4n76b-35634]
2017-09-15 19:24:41.437  INFO 1 --- [           main] o.a.j.commandhandling.JGroupsConnector   : Local segment (order-service-3-4n76b-35634) joined the cluster. Broadcasting configuration.
2017-09-15 19:24:41.494 DEBUG 1 --- [e-3-4n76b-35634] org.jgroups.protocols.FD_SOCK            : order-service-3-4n76b-35634: pingable_mbrs=[product-service-2-36mrc-34743, product-query-1-qqsvn-57792, order-service-3-4n76b-35634], ping_dest=product-service-2-36mrc-34743
2017-09-15 19:24:41.504 DEBUG 1 --- [e-3-4n76b-35634] o.a.j.commandhandling.JGroupsConnector   : Got a network of members: [SimpleMember{name=order-service-3-4n76b-35634, endpoint=product-service-2-36mrc-34743}, SimpleMember{name=order-service-3-4n76b-35634, endpoint=product-query-1-qqsvn-57792}, SimpleMember{name=order-service-3-4n76b-35634, endpoint=order-service-3-4n76b-35634}]
2017-09-15 19:24:41.538 DEBUG 1 --- [e-3-4n76b-35634] o.a.j.commandhandling.JGroupsConnector   : Got a network of members: [SimpleMember{name=order-service-3-4n76b-35634, endpoint=product-service-2-36mrc-34743}, SimpleMember{name=order-service-3-4n76b-35634, endpoint=product-query-1-qqsvn-57792}, SimpleMember{name=order-service-3-4n76b-35634, endpoint=order-service-3-4n76b-35634}]
2017-09-15 19:24:41.995  INFO 1 --- [           main] b.c.e.u.UndertowEmbeddedServletContainer : Undertow started on port(s) 8080 (http)
2017-09-15 19:24:42.015  INFO 1 --- [           main] c.v.m.o.command.OrderCommandApplication  : Started OrderCommandApplication in 272.708 seconds (JVM running for 303.66)
2017-09-15 19:25:05.468 DEBUG 1 --- [e-3-4n76b-35634] o.j.protocols.kubernetes.KUBE_PING       : read all from Kubernetes Client[masterUrl=https://devops:8443/api/v1, headers={Authorization=#MASKED:874#}, connectTimeout=5000, readTimeout=30000, operationAttempts=3, operationSleep=1000, streamProvider=org.jgroups.protocols.kubernetes.stream.InsecureStreamProvider@19c1820d] for cluster [moses-product-order], namespace [moses-demos-dev], labels [jcluster=moses-product-order]

2017-09-15 19:25:53.874 DEBUG 1 --- [e-3-4n76b-35634] o.j.protocols.kubernetes.KUBE_PING       : findMembers()
2017-09-15 19:26:25.850  INFO 1 --- [  XNIO-2 task-1] io.undertow.servlet                      : Initializing Spring FrameworkServlet 'dispatcherServlet'
2017-09-15 19:26:25.851  INFO 1 --- [  XNIO-2 task-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization started
2017-09-15 19:26:26.051  INFO 1 --- [  XNIO-2 task-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization completed in 200 ms
2017-09-15 19:26:30.891 DEBUG 1 --- [  XNIO-2 task-1] o.a.common.IdentifierFactory             : Looking for IdentifierFactory implementation using the context class loader
2017-09-15 19:26:30.896 DEBUG 1 --- [  XNIO-2 task-1] o.a.common.IdentifierFactory             : Looking for IdentifierFactory implementation using the IdentifierFactory class loader.
2017-09-15 19:26:30.904 DEBUG 1 --- [  XNIO-2 task-1] o.a.common.IdentifierFactory             : Using default UUID-based IdentifierFactory
2017-09-15 19:26:31.076 DEBUG 1 --- [e-3-4n76b-35634] o.a.commandhandling.SimpleCommandBus     : Dispatching command [cn.vpclub.moses.common.api.commands.order.CreateOrderCommand]
2017-09-15 19:26:31.082 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.unitofwork.AbstractUnitOfWork      : Starting Unit Of Work
2017-09-15 19:26:31.083 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$383/1078156589 for phase ROLLBACK
2017-09-15 19:26:31.085 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase STARTED
2017-09-15 19:26:31.106 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.SimpleCommandBus$$Lambda$387/1609492861 for phase COMMIT
2017-09-15 19:26:31.120 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.SimpleCommandBus$$Lambda$388/1310282756 for phase ROLLBACK
2017-09-15 19:26:31.157 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$408/603824624 for phase PREPARE_COMMIT
2017-09-15 19:26:31.158 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$409/1507192609 for phase COMMIT
2017-09-15 19:26:31.159 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$410/899428240 for phase AFTER_COMMIT
2017-09-15 19:26:31.159 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$411/1989201974 for phase CLEANUP
2017-09-15 19:26:31.168 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.model.LockingRepository$$Lambda$412/942411061 for phase CLEANUP
2017-09-15 19:26:31.196 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.model.AbstractRepository$$Lambda$418/1154177762 for phase ROLLBACK
2017-09-15 19:26:31.197 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.model.AbstractRepository$$Lambda$420/1938778820 for phase PREPARE_COMMIT
2017-09-15 19:26:31.200 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.unitofwork.AbstractUnitOfWork      : Committing Unit Of Work
2017-09-15 19:26:31.201 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase PREPARE_COMMIT
Hibernate: insert into domain_event_entry (event_identifier, meta_data, payload, payload_revision, payload_type, time_stamp, aggregate_identifier, sequence_number, type) values (?, ?, ?, ?, ?, ?, ?, ?, ?)
2017-09-15 19:26:31.833 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher$$Lambda$430/864548989 for phase COMMIT
2017-09-15 19:26:31.834 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher$$Lambda$431/846785210 for phase AFTER_COMMIT
2017-09-15 19:26:31.835 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher$$Lambda$432/715676850 for phase ROLLBACK
2017-09-15 19:26:32.225 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.unitofwork.AbstractUnitOfWork      : Starting Unit Of Work
2017-09-15 19:26:32.226 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$383/1078156589 for phase ROLLBACK
2017-09-15 19:26:32.227 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$441/417481094 for phase CLEANUP
2017-09-15 19:26:32.227 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase STARTED
2017-09-15 19:26:32.232 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventProcessor$$Lambda$444/245366789 for phase CLEANUP
2017-09-15 19:26:32.244 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.unitofwork.AbstractUnitOfWork      : Committing Unit Of Work
2017-09-15 19:26:32.319 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase PREPARE_COMMIT
2017-09-15 19:26:32.319 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase COMMIT
2017-09-15 19:26:32.322 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$445/79990310 for phase AFTER_COMMIT
2017-09-15 19:26:32.323 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$446/1901862014 for phase ROLLBACK
2017-09-15 19:26:32.324 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase COMMIT
2017-09-15 19:26:32.662 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase AFTER_COMMIT
2017-09-15 19:26:32.662 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase AFTER_COMMIT
2017-09-15 19:26:32.663 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLEANUP
2017-09-15 19:26:32.663 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLEANUP
2017-09-15 19:26:32.663 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLOSED
2017-09-15 19:26:32.663 DEBUG 1 --- [e-3-4n76b-35634] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLOSED
2017-09-15 19:26:32.915 DEBUG 1 --- [cTaskExecutor-1] c.v.m.o.c.config.AmqpConfiguration       : product message received: (Body:'[B@19d27bef(byte[2883])' MessageProperties [headers={axon-message-id=8b8a58e4-fa2a-493b-afe0-68635d85c7f3, axon-message-aggregate-seq=0, axon-metadata-correlationId=0c831edd-b033-49e5-b8c4-adbd6ce5103f, axon-message-aggregate-type=OrderAggregate, axon-message-revision=null, axon-message-timestamp=2017-09-15T11:26:31.150Z, axon-message-type=cn.vpclub.moses.common.api.events.order.OrderCreatedEvent, axon-metadata-traceId=0c831edd-b033-49e5-b8c4-adbd6ce5103f, axon-message-aggregate-id=908653228899155970}, timestamp=null, messageId=null, userId=null, receivedUserId=null, appId=null, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=null, contentEncoding=null, contentLength=0, deliveryMode=null, receivedDeliveryMode=PERSISTENT, expiration=null, priority=null, redelivered=false, receivedExchange=amq.fanout, receivedRoutingKey=cn.vpclub.moses.common.api.events.order, receivedDelay=null, deliveryTag=1, messageCount=0, consumerTag=amq.ctag-82I_LU8ZdtkdEymfd-7xqQ, consumerQueue=moses-product])
2017-09-15 19:26:33.504 DEBUG 1 --- [cTaskExecutor-1] o.a.m.unitofwork.AbstractUnitOfWork      : Starting Unit Of Work
2017-09-15 19:26:33.507 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$383/1078156589 for phase ROLLBACK
2017-09-15 19:26:33.507 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase STARTED
2017-09-15 19:26:33.508 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventProcessor$$Lambda$444/245366789 for phase CLEANUP
2017-09-15 19:26:33.635 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.interceptors.TransactionManagingInterceptor$$Lambda$457/390815680 for phase COMMIT
2017-09-15 19:26:33.637 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.interceptors.TransactionManagingInterceptor$$Lambda$458/1615068535 for phase ROLLBACK
Hibernate: select associatio0_.saga_id as col_0_0_ from association_value_entry associatio0_ where associatio0_.association_key=? and associatio0_.association_value=? and associatio0_.saga_type=?
2017-09-15 19:26:34.073 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.saga.repository.LockingSagaRepository$$Lambda$476/1177575326 for phase CLEANUP
2017-09-15 19:26:34.110 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.saga.repository.AnnotatedSagaRepository$$Lambda$479/653921510 for phase PREPARE_COMMIT
2017-09-15 19:26:34.114 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.saga.repository.AnnotatedSagaRepository$$Lambda$480/2120807475 for phase CLEANUP
2017-09-15 19:26:34.285 ERROR 1 --- [cTaskExecutor-1] o.a.e.saga.AbstractSagaManager           : An exception occurred while a Saga [AnnotatedSaga] was handling an Event [OrderCreatedEvent]:


3. But the connection was removed, I don’t understand why?

2017-09-15 19:39:27.449 DEBUG 1 --- [e-3-4n76b-35634] o.j.protocols.kubernetes.KUBE_PING       : findMembers()
2017-09-15 19:39:47.185 DEBUG 1 --- [e-3-4n76b-35634] org.jgroups.protocols.UNICAST3           : order-service-3-4n76b-35634: removing expired connection for product-service-2-shtws-51213 (240204 ms old) from send_table
2017-09-15 19:39:47.186 DEBUG 1 --- [e-3-4n76b-35634] org.jgroups.protocols.UNICAST3           : order-service-3-4n76b-35634: removing expired connection for product-query-1-qqsvn-57792 (240204 ms old) from send_table
2017-09-15 19:39:47.186 DEBUG 1 --- [e-3-4n76b-35634] org.jgroups.protocols.UNICAST3           : order-service-3-4n76b-35634: removing expired connection for order-service-3-4n76b-35634 (240204 ms old) from send_table
2017-09-15 19:39:47.187 DEBUG 1 --- [e-3-4n76b-35634] org.jgroups.protocols.UNICAST3           : order-service-3-4n76b-35634: removing expired connection for product-query-1-qqsvn-57792 (240204 ms old) from recv_table

4. Of cause it will not work with distributed command, the log shows " No node known to accept cn.vpclub.moses.common.api.commands.order.CreateOrderCommand "

2017-09-15 19:37:27.128 ERROR 1 --- [  XNIO-2 task-3] io.undertow.request                      : UT005023: Exception handling request to /order/create

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.axonframework.commandhandling.distributed.CommandDispatchException: No node known to accept cn.vpclub.moses.common.api.commands.order.CreateOrderCommand
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javaee-api-7.0.jar!/:na]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javaee-api-7.0.jar!/:na]
    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) ~[spring-boot-1.5.4.RELEASE.jar!/:1.5.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) ~[spring-boot-actuator-1.5.4.RELEASE.jar!/:1.5.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) ~[spring-boot-actuator-1.5.4.RELEASE.jar!/:1.5.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) ~[undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:211) [undertow-core-1.4.15.Final.jar!/:1.4.15.Final]
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:809) [undertow-core-1.4.15.Final.jar!/:1.4.15.Final]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: org.axonframework.commandhandling.distributed.CommandDispatchException: No node known to accept cn.vpclub.moses.common.api.commands.order.CreateOrderCommand
    at org.axonframework.commandhandling.distributed.DistributedCommandBus.lambda$dispatch$4(DistributedCommandBus.java:125) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at java.util.Optional.orElseThrow(Optional.java:290) ~[na:1.8.0_121]
    at org.axonframework.commandhandling.distributed.DistributedCommandBus.dispatch(DistributedCommandBus.java:125) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.commandhandling.gateway.AbstractCommandGateway.send(AbstractCommandGateway.java:79) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:95) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.commandhandling.gateway.DefaultCommandGateway.sendAndWait(DefaultCommandGateway.java:113) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at cn.vpclub.moses.order.command.controllers.OrderController.create(OrderController.java:55) ~[classes!/:1.0.0]
    at cn.vpclub.moses.order.command.controllers.OrderController$$FastClassBySpringCGLIB$$521b68c6.invoke(<generated>) ~[classes!/:1.0.0]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at cn.vpclub.moses.web.aop.MessageAspectImpl.execute(MessageAspectImpl.java:28) ~[moses-web-1.0.13.jar!/:1.0.13]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at cn.vpclub.moses.order.command.controllers.OrderController$$EnhancerBySpringCGLIB$$1b73dc60.create(<generated>) ~[classes!/:1.0.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    ... 61 common frames omitted




John Deng

unread,
Sep 15, 2017, 10:09:38 AM9/15/17
to Axon Framework Users

Hi Benoît,

It seems that the my two services joined the same jgroup cluster together, however, I got another issue, org.axonframework.commandhandling.distributed.CommandDispatchException: No node known to accept cn.vpclub.moses.common.api.commands.product.ReserveProductCommand, do you know what the problem is? 


2017-09-15 21:58:23.771 DEBUG 1 --- [           main] org.jgroups.protocols.pbcast.GMS         : order-service-13-6tkb9-31962: sending JOIN(order-service-13-6tkb9-31962) to product-query-1-qqsvn-57792
2017-09-15 21:58:24.410  INFO 1 --- [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector   : order-service-13-6tkb9-31962 joined with load factor: 100
2017-09-15 21:58:24.411 DEBUG 1 --- [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector   : Got a network of members: [SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-service-7-7c17m-20181}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 21:58:24.414  INFO 1 --- [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector   : order-service-13-6tkb9-31962 joined with load factor: 100
2017-09-15 21:58:24.415 DEBUG 1 --- [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector   : Got a network of members: [SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-service-7-7c17m-20181}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 21:58:24.484 DEBUG 1 --- [           main] org.jgroups.protocols.pbcast.NAKACK2     :
[order-service-13-6tkb9-31962 setDigest()]
existing digest:  []
new digest:       product-query-1-qqsvn-57792: [14 (14)], product-service-7-7c17m-20181: [2 (2)], order-service-13-6tkb9-31962: [0 (0)]
resulting digest: product-query-1-qqsvn-57792: [14 (14)], product-service-7-7c17m-20181: [2 (2)], order-service-13-6tkb9-31962: [0 (0)]
2017-09-15 21:58:24.484 DEBUG 1 --- [           main] org.jgroups.protocols.pbcast.GMS         : order-service-13-6tkb9-31962: installing view [product-query-1-qqsvn-57792|20] (3) [product-query-1-qqsvn-57792, product-service-7-7c17m-20181, order-service-13-6tkb9-31962]
2017-09-15 21:58:24.514 DEBUG 1 --- [-13-6tkb9-31962] org.jgroups.protocols.FD_SOCK            : order-service-13-6tkb9-31962: pingable_mbrs=[product-query-1-qqsvn-57792, product-service-7-7c17m-20181, order-service-13-6tkb9-31962], ping_dest=product-query-1-qqsvn-57792
2017-09-15 21:58:24.529 DEBUG 1 --- [           main] org.jgroups.protocols.CENTRAL_LOCK       : view=[product-query-1-qqsvn-57792|20] (3) [product-query-1-qqsvn-57792, product-service-7-7c17m-20181, order-service-13-6tkb9-31962]
2017-09-15 21:58:24.529 DEBUG 1 --- [           main] org.jgroups.protocols.CENTRAL_LOCK       : local_addr=order-service-13-6tkb9-31962, coord=product-query-1-qqsvn-57792, is_coord=false
2017-09-15 21:58:24.529 DEBUG 1 --- [           main] org.jgroups.protocols.COUNTER            : view=[product-query-1-qqsvn-57792|20] (3) [product-query-1-qqsvn-57792, product-service-7-7c17m-20181, order-service-13-6tkb9-31962]
2017-09-15 21:58:24.529  INFO 1 --- [           main] o.a.j.commandhandling.JGroupsConnector   : Local segment (order-service-13-6tkb9-31962) joined the cluster. Broadcasting configuration.
2017-09-15 21:58:24.539 DEBUG 1 --- [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector   : Got a network of members: [SimpleMember{name=order-service-13-6tkb9-31962, endpoint=order-service-13-6tkb9-31962}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-service-7-7c17m-20181}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 21:58:24.543 DEBUG 1 --- [-13-6tkb9-31962] o.a.j.commandhandling.JGroupsConnector   : Got a network of members: [SimpleMember{name=order-service-13-6tkb9-31962, endpoint=order-service-13-6tkb9-31962}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-service-7-7c17m-20181}, SimpleMember{name=order-service-13-6tkb9-31962, endpoint=product-query-1-qqsvn-57792}]
2017-09-15 21:58:24.691  INFO 1 --- [           main] b.c.e.u.UndertowEmbeddedServletContainer : Undertow started on port(s) 8080 (http)
2017-09-15 21:58:24.710  INFO 1 --- [           main] c.v.m.o.command.OrderCommandApplication  : Started OrderCommandApplication in 112.152 seconds (JVM running for 123.284)

2017-09-15 21:58:40.468  INFO 1 --- [  XNIO-2 task-1] io.undertow.servlet                      : Initializing Spring FrameworkServlet 'dispatcherServlet'
2017-09-15 21:58:40.469  INFO 1 --- [  XNIO-2 task-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization started
2017-09-15 21:58:40.582  INFO 1 --- [  XNIO-2 task-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization completed in 113 ms
2017-09-15 21:58:42.253 DEBUG 1 --- [  XNIO-2 task-1] o.a.common.IdentifierFactory             : Looking for IdentifierFactory implementation using the context class loader
2017-09-15 21:58:42.256 DEBUG 1 --- [  XNIO-2 task-1] o.a.common.IdentifierFactory             : Looking for IdentifierFactory implementation using the IdentifierFactory class loader.
2017-09-15 21:58:42.259 DEBUG 1 --- [  XNIO-2 task-1] o.a.common.IdentifierFactory             : Using default UUID-based IdentifierFactory
2017-09-15 21:58:42.452 DEBUG 1 --- [-13-6tkb9-31962] o.a.commandhandling.SimpleCommandBus     : Dispatching command [cn.vpclub.moses.common.api.commands.order.CreateOrderCommand]
2017-09-15 21:58:42.457 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.unitofwork.AbstractUnitOfWork      : Starting Unit Of Work
2017-09-15 21:58:42.459 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$385/587493189 for phase ROLLBACK
2017-09-15 21:58:42.461 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase STARTED
2017-09-15 21:58:42.578 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.SimpleCommandBus$$Lambda$389/1995400911 for phase COMMIT
2017-09-15 21:58:42.579 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.SimpleCommandBus$$Lambda$390/1820925069 for phase ROLLBACK
2017-09-15 21:58:42.674 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$410/443108395 for phase PREPARE_COMMIT
2017-09-15 21:58:42.675 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$411/712595626 for phase COMMIT
2017-09-15 21:58:42.675 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$412/1490749862 for phase AFTER_COMMIT
2017-09-15 21:58:42.679 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventBus$$Lambda$413/746270005 for phase CLEANUP
2017-09-15 21:58:42.685 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.model.LockingRepository$$Lambda$414/2082681989 for phase CLEANUP
2017-09-15 21:58:42.687 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.model.AbstractRepository$$Lambda$420/959407878 for phase ROLLBACK
2017-09-15 21:58:42.688 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.commandhandling.model.AbstractRepository$$Lambda$422/1208575357 for phase PREPARE_COMMIT
2017-09-15 21:58:42.690 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.unitofwork.AbstractUnitOfWork      : Committing Unit Of Work
2017-09-15 21:58:42.691 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase PREPARE_COMMIT
Hibernate: insert into domain_event_entry (event_identifier, meta_data, payload, payload_revision, payload_type, time_stamp, aggregate_identifier, sequence_number, type) values (?, ?, ?, ?, ?, ?, ?, ?, ?)
2017-09-15 21:58:43.143 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher$$Lambda$432/982164527 for phase COMMIT
2017-09-15 21:58:43.165 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher$$Lambda$433/1942719544 for phase AFTER_COMMIT
2017-09-15 21:58:43.168 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.amqp.eventhandling.spring.SpringAMQPPublisher$$Lambda$434/1727754046 for phase ROLLBACK
2017-09-15 21:58:43.192 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.unitofwork.AbstractUnitOfWork      : Starting Unit Of Work
2017-09-15 21:58:43.192 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$385/587493189 for phase ROLLBACK
2017-09-15 21:58:43.193 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$443/1714174130 for phase CLEANUP
2017-09-15 21:58:43.194 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase STARTED
2017-09-15 21:58:43.195 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventProcessor$$Lambda$446/1968958534 for phase CLEANUP
2017-09-15 21:58:43.197 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.unitofwork.AbstractUnitOfWork      : Committing Unit Of Work
2017-09-15 21:58:43.198 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase PREPARE_COMMIT
2017-09-15 21:58:43.198 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase COMMIT
2017-09-15 21:58:43.198 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$447/1924506615 for phase AFTER_COMMIT
2017-09-15 21:58:43.199 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$448/2114481982 for phase ROLLBACK
2017-09-15 21:58:43.200 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase COMMIT
2017-09-15 21:58:43.206 DEBUG 1 --- [cTaskExecutor-1] c.v.m.o.c.config.AmqpConfiguration       : product message received: (Body:'[B@3ed99e34(byte[2883])' MessageProperties [headers={axon-message-id=b8fadcbd-5677-4859-b537-9ff8385aa696, axon-message-aggregate-seq=0, axon-metadata-correlationId=1d7c5e43-2015-4954-812a-d202cee1764d, axon-message-aggregate-type=OrderAggregate, axon-message-revision=null, axon-message-timestamp=2017-09-15T13:58:42.658Z, axon-message-type=cn.vpclub.moses.common.api.events.order.OrderCreatedEvent, axon-metadata-traceId=1d7c5e43-2015-4954-812a-d202cee1764d, axon-message-aggregate-id=908691541454090241}, timestamp=null, messageId=null, userId=null, receivedUserId=null, appId=null, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=null, contentEncoding=null, contentLength=0, deliveryMode=null, receivedDeliveryMode=PERSISTENT, expiration=null, priority=null, redelivered=false, receivedExchange=amq.fanout, receivedRoutingKey=cn.vpclub.moses.common.api.events.order, receivedDelay=null, deliveryTag=1, messageCount=0, consumerTag=amq.ctag-FnAv3dxXcTfNbWYg5Tjk1A, consumerQueue=moses-product])
2017-09-15 21:58:43.232 DEBUG 1 --- [cTaskExecutor-1] o.a.m.unitofwork.AbstractUnitOfWork      : Starting Unit Of Work
2017-09-15 21:58:43.233 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.unitofwork.AbstractUnitOfWork$$Lambda$385/587493189 for phase ROLLBACK
2017-09-15 21:58:43.233 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase STARTED
2017-09-15 21:58:43.233 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.AbstractEventProcessor$$Lambda$446/1968958534 for phase CLEANUP
2017-09-15 21:58:43.235 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase AFTER_COMMIT
2017-09-15 21:58:43.236 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase AFTER_COMMIT
2017-09-15 21:58:43.236 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.interceptors.TransactionManagingInterceptor$$Lambda$458/1489446096 for phase COMMIT
2017-09-15 21:58:43.237 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.messaging.interceptors.TransactionManagingInterceptor$$Lambda$459/270452166 for phase ROLLBACK
2017-09-15 21:58:43.238 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLEANUP
2017-09-15 21:58:43.238 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLEANUP
2017-09-15 21:58:43.239 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLOSED
2017-09-15 21:58:43.239 DEBUG 1 --- [-13-6tkb9-31962] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLOSED
Hibernate: select associatio0_.saga_id as col_0_0_ from association_value_entry associatio0_ where associatio0_.association_key=? and associatio0_.association_value=? and associatio0_.saga_type=?
2017-09-15 21:58:43.296 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.saga.repository.LockingSagaRepository$$Lambda$478/580163873 for phase CLEANUP
2017-09-15 21:58:43.321 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.saga.repository.AnnotatedSagaRepository$$Lambda$481/1161655833 for phase PREPARE_COMMIT
2017-09-15 21:58:43.322 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Adding handler org.axonframework.eventhandling.saga.repository.AnnotatedSagaRepository$$Lambda$482/1198029371 for phase CLEANUP
2017-09-15 21:58:43.339 ERROR 1 --- [cTaskExecutor-1] o.a.e.saga.AbstractSagaManager           : An exception occurred while a Saga [AnnotatedSaga] was handling an Event [OrderCreatedEvent]:

org.axonframework.commandhandling.distributed.CommandDispatchException: No node known to accept cn.vpclub.moses.common.api.commands.product.ReserveProductCommand
    at org.axonframework.commandhandling.distributed.DistributedCommandBus.lambda$dispatch$4(DistributedCommandBus.java:125) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at java.util.Optional.orElseThrow(Optional.java:290) ~[na:1.8.0_121]
    at org.axonframework.commandhandling.distributed.DistributedCommandBus.dispatch(DistributedCommandBus.java:125) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.commandhandling.gateway.AbstractCommandGateway.send(AbstractCommandGateway.java:79) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.commandhandling.gateway.DefaultCommandGateway.send(DefaultCommandGateway.java:95) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at cn.vpclub.moses.order.command.saga.OrderSaga.lambda$null$1(OrderSaga.java:56) ~[classes!/:1.0.0]
    at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_121]
    at cn.vpclub.moses.order.command.saga.OrderSaga.lambda$handle$2(OrderSaga.java:51) ~[classes!/:1.0.0]
    at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_121]
    at cn.vpclub.moses.order.command.saga.OrderSaga.handle(OrderSaga.java:50) ~[classes!/:1.0.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
    at org.axonframework.messaging.annotation.AnnotatedMessageHandlingMember.handle(AnnotatedMessageHandlingMember.java:127) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.saga.SagaMethodMessageHandlingMember.handle(SagaMethodMessageHandlingMember.java:77) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.saga.AnnotatedSaga.lambda$null$4(AnnotatedSaga.java:102) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.saga.SagaLifecycle.executeWithResult(SagaLifecycle.java:138) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.saga.AnnotatedSaga.lambda$handle$5(AnnotatedSaga.java:102) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at java.util.Optional.map(Optional.java:215) ~[na:1.8.0_121]
    at org.axonframework.eventhandling.saga.AnnotatedSaga.handle(AnnotatedSaga.java:100) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.saga.AbstractSagaManager.doInvokeSaga(AbstractSagaManager.java:110) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.saga.AbstractSagaManager.startNewSaga(AbstractSagaManager.java:87) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.saga.AbstractSagaManager.handle(AbstractSagaManager.java:79) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.saga.AbstractSagaManager.handle(AbstractSagaManager.java:40) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:57) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.messaging.interceptors.TransactionManagingInterceptor.handle(TransactionManagingInterceptor.java:50) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.messaging.interceptors.CorrelationDataInterceptor.handle(CorrelationDataInterceptor.java:55) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.messaging.DefaultInterceptorChain.proceed(DefaultInterceptorChain.java:55) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.AbstractEventProcessor.lambda$process$3(AbstractEventProcessor.java:126) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.messaging.unitofwork.BatchingUnitOfWork.executeWithResult(BatchingUnitOfWork.java:76) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.AbstractEventProcessor.process(AbstractEventProcessor.java:116) [axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.SubscribingEventProcessor.process(SubscribingEventProcessor.java:142) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.DirectEventProcessingStrategy.handle(DirectEventProcessingStrategy.java:32) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.eventhandling.SubscribingEventProcessor.lambda$start$0(SubscribingEventProcessor.java:135) ~[axon-core-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at org.axonframework.amqp.eventhandling.spring.SpringAMQPMessageSource.lambda$onMessage$1(SpringAMQPMessageSource.java:90) ~[axon-amqp-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890) ~[na:1.8.0_121]
    at org.axonframework.amqp.eventhandling.spring.SpringAMQPMessageSource.onMessage(SpringAMQPMessageSource.java:90) ~[axon-amqp-3.1-SNAPSHOT.jar!/:3.1-SNAPSHOT]
    at cn.vpclub.moses.order.command.config.AmqpConfiguration$1.onMessage(AmqpConfiguration.java:54) ~[classes!/:1.0.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180) ~[spring-messaging-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112) ~[spring-messaging-4.3.9.RELEASE.jar!/:4.3.9.RELEASE]
    at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421) ~[spring-rabbit-1.7.3.RELEASE.jar!/:na]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]

2017-09-15 21:58:43.340 DEBUG 1 --- [cTaskExecutor-1] o.a.m.unitofwork.AbstractUnitOfWork      : Committing Unit Of Work
2017-09-15 21:58:43.340 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase PREPARE_COMMIT
Hibernate: insert into saga_entry (revision, saga_type, serialized_saga, saga_id) values (?, ?, ?, ?)
Hibernate: insert into association_value_entry (association_key, association_value, saga_id, saga_type) values (?, ?, ?, ?)
2017-09-15 21:58:43.377 DEBUG 1 --- [cTaskExecutor-1] o.a.e.saga.repository.jpa.JpaSagaStore   : Storing saga id 9eda8d28-d790-4548-95cd-3c535958cffa as <cn.vpclub.moses.order.command.saga.OrderSaga><orderIdentifier>908691541454090241</orderIdentifier><toReserve class="concurrent-hash-map"><entry><long>907127376998625280</long><cn.vpclub.moses.common.api.entity.order.OrderItem><appId>100000054</appId><id>908691547393224706</id><mainOrderId>908691541454090241</mainOrderId><orderId>908691547464527873</orderId><skuId>907127376998625280</skuId><productId>907127006415097858</productId><productName>测试2</productName><productCode>8</productCode><merchantId>98140070540607516</merchantId><buyQty>1</buyQty><salePrice>1</salePrice><property></property><refundQty>0</refundQty><coverImage>http://dev.vpclub.cn/group1/M00/00/18/rBAC6lmmnZ2AUb73AAB3Fn4tgUU108.jpg</coverImage><reserved>false</reserved></cn.vpclub.moses.common.api.entity.order.OrderItem></entry></toReserve><toRollback class="concurrent-hash-map"/><toReserveNumber>1</toReserveNumber><needRollback>false</needRollback></cn.vpclub.moses.order.command.saga.OrderSaga>
2017-09-15 21:58:43.379 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase COMMIT
2017-09-15 21:58:43.562 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase AFTER_COMMIT
2017-09-15 21:58:43.562 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLEANUP
2017-09-15 21:58:43.563 DEBUG 1 --- [cTaskExecutor-1] o.a.m.u.MessageProcessingContext         : Notifying handlers for phase CLOSED


...

Benoît Liessens

unread,
Sep 15, 2017, 11:16:25 AM9/15/17
to Axon Framework Users
John,

Can you confirm the CreateOrderCommand is processed as expected in each of these setups:
  1. single node deployment, with SimpleCommandBus (no JGroups) ? If yes then presumably command bus, handlers, aggregate(s), listeners etc are properly configured and wired by spring
  2. multi-node deployment with JGroups (and DistributedCommandBus) but without Kubernetes (e.g. deploy 1 nodes on your own pc and a 2nd node on your colleagues' pc)?
    Fire several CreateOrderCommands te create distinct Order aggregates. Each node should handle roughly 50% of the commands. Is that the case?
Regards,
Benoît 
...

John Deng

unread,
Sep 15, 2017, 12:15:51 PM9/15/17
to Axon Framework Users
Hi Benoît,

I have two services, the product service and order service, the order service send the command ReserveProductCommand to product service to reserve product, 
it works fine when I deployed these two services to standard alone machines (without Kubernetes) with config file jgroups-config-local.xml.
however, when I deploy these two services to Kubernetes with config file jgroups-config.xml, CommandDispatchException occurred, I don't know if 
it's caused by the wrong config file?  I uploaded both config files, please help to check the configurations see if something wrong there.

I checked the logs, both said that the services are joined the cluster, I didn't find any errors.

For your suggestions, Since the services worked on the standard alone machines, do I still need to confirm them? if so, I will test them now.

Best regards,
John
...
jgroups-config-local.xml
jgroups-config.xml

John Deng

unread,
Sep 18, 2017, 1:44:41 AM9/18/17
to Axon Framework Users
Hi All,

I finally got it to work, I already send the pull request, please check it out https://github.com/AxonFramework/AxonFramework/pull/408.
...
Reply all
Reply to author
Forward
0 new messages