Dynamic Shovel AMQP10 with content-type

Stephane Hatoum

unread,

Oct 21, 2019, 6:11:29 AM10/21/19

to rabbitmq-users

Hello,

I defined a dynamic shovel using the pluggin amqp10 with the option content-type at text/plain :

"dest-protocol": "amqp10","dest-add-forward-headers":false, "dest-address": "RABBIT2MQ","prefetch-count":1, "dest-properties": {"to":"RABBIT2MQ", "content-type":"text/plain"}

The shovel is created and RUNNING status, but, when i send a message in the input queue for this shovel, it fails with the following error message :

{"node":"RMQFRRTTST01@sb034tts","timestamp":"2019-10-21 12:06:17","name":"SHOV_LOQ.RABBIT2MQ","vhost":"/vtst","type":"dynamic","state":"terminated","reason":"{function_clause,\n [{amqp10_msg,'-set_properties/2-fun-0-',\n ['content-type',<<\"text/plain\">>,\n {'v1_0.properties',undefined,undefined,undefined,undefined,\n undefined,undefined,undefined,undefined,undefined,undefined,\n undefined,undefined,undefined}],\n [{file,\"src/amqp10_msg.erl\"},{line,333}]},\n {maps,fold_1,3,[{file,\"maps.erl\"},{line,232}]},\n {amqp10_msg,set_proper* Connection #0 to host sb034tts.mc2.renault.fr left intact

ties,2,[{file,\"src/amqp10_msg.erl\"},{line,333}]},\n {rabbit_amqp10_shovel,new_message,3,\n [{file,\"src/rabbit_amqp10_shovel.erl\"},{line,342}]},\n {rabbit_amqp10_shovel,forward,4,\n [{file,\"src/rabbit_amqp10_shovel.erl\"},{line,321}]},\n {rabbit_shovel_worker,handle_info,2,\n [{file,\"src/rabbit_shovel_worker.erl\"},{line,101}]},\n {gen_server2,handle_msg,2,[{file,\"src/gen_server2.erl\"},{line,1056}]},\n {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,249}]}]}"}

Thank you for your help.

Best regards,

Stephane

Wesley Peng

unread,

Oct 21, 2019, 6:15:25 AM10/21/19

to rabbitm...@googlegroups.com

Hi

on 2019/10/21 18:11, Stephane Hatoum wrote:
> I defined a dynamic shovel using the pluggin amqp10 with the option
> content-type at text/plain :
> "dest-protocol": "amqp10","dest-add-forward-headers":false,
> "dest-address": "RABBIT2MQ","prefetch-count":1, "dest-properties":
> {"to":"RABBIT2MQ", "content-type":"text/plain"}

what's the versions of erlang, rabbitmq and plugin?

regards

Karl Nilsson

unread,

Oct 21, 2019, 6:27:28 AM10/21/19

to rabbitm...@googlegroups.com

It needs to be `content_type` with an underscore.

Cheers

Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_85495575-2D7335-2D45f6-2D0e61-2Dba7bd0465f63-2540gmail.com&d=DwIFaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=opi9G2FkNu0cXYiVF-A6mfH0OOvfYv6Ki1WLkdLIPcA&m=mKpDhRrWF7FCb6MWa7V8rr0YOVUWadnK8yxGNiByByo&s=2sDt9t6ro_aw4cCiVCMC3HFGNVhFtF4Eq3WECWIQFhI&e= .

--

Karl Nilsson

Pivotal/RabbitMQ

Stephane Renault

unread,

Oct 21, 2019, 6:30:51 AM10/21/19

to rabbitmq-users

Hi,

Please find below all components and their versions :

sb034tts:/var/lib/rabbitmq/RMQFRRTTST01/log-> rabbitmqctl status

Status of node RMQFRRTTST01@sb034tts ...

[{pid,65411},

{running_applications,

[{rabbitmq_federation_management,"RabbitMQ Federation Management",

"3.7.17"},

{rabbitmq_federation,"RabbitMQ Federation","3.7.17"},

{rabbitmq_shovel_management,

"Management extension for the Shovel plugin","3.7.17"},

{rabbitmq_management,"RabbitMQ Management Console","3.7.17"},

{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.7.17"},

{cowboy,"Small, fast, modern HTTP server.","2.6.1"},

{cowlib,"Support library for manipulating Web protocols.","2.7.0"},

{rabbitmq_management_agent,"RabbitMQ Management Agent","3.7.17"},

{rabbitmq_shovel,"Data Shovel for RabbitMQ","3.7.17"},

{amqp10_client,"AMQP 1.0 client from the RabbitMQ Project","3.7.17"},

{rabbitmq_amqp1_0,"AMQP 1.0 support for RabbitMQ","3.7.17"},

{amqp_client,"RabbitMQ AMQP Client","3.7.17"},

{amqp10_common,

"Modules shared by rabbitmq-amqp1.0 and rabbitmq-amqp1.0-client",

"3.7.17"},

{rabbit,"RabbitMQ","3.7.17"},

{mnesia,"MNESIA CXC 138 12","4.16"},

{rabbit_common,

"Modules shared by rabbitmq-server and rabbitmq-erlang-client",

"3.7.17"},

{ranch,"Socket acceptor pool for TCP protocols.","1.7.1"},

{ssl,"Erlang/OTP SSL application","9.3.5"},

{public_key,"Public key infrastructure","1.6.7"},

{crypto,"CRYPTO","4.5.1"},

{asn1,"The Erlang ASN1 compiler version 5.0.9","5.0.9"},

{xmerl,"XML parser","1.3.21"},

{jsx,"a streaming, evented json parsing toolkit","2.9.0"},

{sysmon_handler,"Rate-limiting system_monitor event handler","1.1.0"},

{os_mon,"CPO CXC 138 46","2.5"},

{observer_cli,"Visualize Erlang Nodes On The Command Line","1.5.0"},

{recon,"Diagnostic tools for production use","2.5.0"},

{stdout_formatter,

"Tools to format paragraphs, lists and tables as plain text",

"0.2.2"},

{inets,"INETS CXC 138 49","7.0.9"},

{lager,"Erlang logging framework","3.6.10"},

{goldrush,"Erlang event stream processor","0.1.9"},

{compiler,"ERTS CXC 138 10","7.4.4"},

{syntax_tools,"Syntax tools","2.2"},

{sasl,"SASL CXC 138 11","3.4"},

{stdlib,"ERTS CXC 138 10","3.9.2"},

{kernel,"ERTS CXC 138 10","6.4.1"}]},

Thank you for your help.

Best regards,

Stéphane

Stephane Renault

unread,

Oct 21, 2019, 11:48:23 AM10/21/19

to rabbitmq-users

Thank you Karl, effectivelly, the content_type is the good key word even if in the trace i found contentType and in the documentation you can found content-type.

Below the trace with different field names.

16.53.03.920 49 @3ea395dc com.ibm.mq.MQXRService.AMQPServerWireContext ----+---- } readNextMessage [Delivery: DeliveryImpl [_tag=[50], _link=org.apache.qpid.proton.engine.impl.ReceiverImpl@53d1c05d, _deliveryState=null, _settled=false, _remoteSettled=false, _remoteDeliveryState=null, _flags=0, _defaultDeliveryState=null, _transportDelivery=org.apache.qpid.proton.engine.impl.TransportDelivery@4e0bf021, _dataSize=0, _complete=true, _updated=false, _done=true, _offset=101]] [Message: Message{header=Header{durable=true, priority=null, ttl=null, firstAcquirer=null, deliveryCount=null}properties=Properties{messageId=null, userId=null, to='RABBIT2MQ', subject='null', replyTo='null', correlationId=null, contentType=text/plain, contentEncoding=null, absoluteExpiryTime=null, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}body=Data{MSG_TST}}] [Address: RABBIT2MQ] [Subject: null] [Properties: Properties{messageId=null, userId=null, to='RABBIT2MQ', subject='null', replyTo='null', correlationId=null, contentType=text/plain, contentEncoding=null, absoluteExpiryTime=null, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}] [ReplyTo: null] [ContentType: text/plain] [Format: DATA] [Body: Data{MSG_TST}]

My message arrives with AMQP format, i would like to have a STRING format, how and with key work can i define the format field ?

Thank you for your help.

Best regards,

Stéphane

Le lundi 21 octobre 2019 12:27:28 UTC+2, Karl Nilsson a écrit :

It needs to be `content_type` with an underscore.

Cheers
Karl

On Mon, 21 Oct 2019 at 11:15, Wesley Peng <wesle...@gmail.com> wrote:

Hi

on 2019/10/21 18:11, Stephane Hatoum wrote:
> I defined a dynamic shovel using the pluggin amqp10 with the option
> content-type at text/plain :
> "dest-protocol": "amqp10","dest-add-forward-headers":false,
> "dest-address": "RABBIT2MQ","prefetch-count":1, "dest-properties":
> {"to":"RABBIT2MQ", "content-type":"text/plain"}

what's the versions of erlang, rabbitmq and plugin?

regards

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_85495575-2D7335-2D45f6-2D0e61-2Dba7bd0465f63-2540gmail.com&d=DwIFaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=opi9G2FkNu0cXYiVF-A6mfH0OOvfYv6Ki1WLkdLIPcA&m=mKpDhRrWF7FCb6MWa7V8rr0YOVUWadnK8yxGNiByByo&s=2sDt9t6ro_aw4cCiVCMC3HFGNVhFtF4Eq3WECWIQFhI&e= .

--
Karl Nilsson

Pivotal/RabbitMQ

Karl Nilsson

unread,

Oct 21, 2019, 12:15:49 PM10/21/19

to rabbitm...@googlegroups.com

The content-type is an indicator for you application and nothing that RabbitMQ will interpret. You still need to decode the message to get the body data.

Cheers

Karl

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/9364af96-4ab7-4ef8-8af2-2189791b2199%40googlegroups.com.

--

Karl Nilsson

Pivotal/RabbitMQ

Stephane Renault

unread,

Oct 21, 2019, 12:22:29 PM10/21/19

to rabbitmq-users

Data in the message are prefixed by an AMQP header because the format value is DATA, i would like to change it at STRING; how can i do it ?

Le lundi 21 octobre 2019 18:15:49 UTC+2, Karl Nilsson a écrit :

The content-type is an indicator for you application and nothing that RabbitMQ will interpret. You still need to decode the message to get the body data.

Cheers
Karl

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/9364af96-4ab7-4ef8-8af2-2189791b2199%40googlegroups.com.

--
Karl Nilsson

Pivotal/RabbitMQ

Karl Nilsson

unread,

Oct 21, 2019, 12:58:31 PM10/21/19

to rabbitm...@googlegroups.com

A Data section contains just an opaque blob of binary data. There are two other options for "body" data but they are structured amqp values which further away from what you want. See §3.2.6 of the AMQP 1.0 spec.

https://www.amqp.org/resources/specifications

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/74ae51af-d2db-4a32-b574-9d89d80fbc4e%40googlegroups.com.

--

Karl Nilsson

Pivotal/RabbitMQ

Message has been deleted

Stephane Renault

unread,

Oct 22, 2019, 3:44:50 AM10/22/19

to rabbitmq-users

Hi,

- Piece of trace of AMQP channel when publishing a message using send.js:

----+---- { receiveMessage [ClientIdentifier:send_4a183d3] [Receiver:org.apache.qpid.proton.engine.impl.ReceiverImpl@d8b1db3a] [receiver.getRemoteSource():Source{address='localhost', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties={}, distributionMode=null, filter={}, defaultOutcome=null, outcomes=null, capabilities=null}] [receiver.getRemoteTarget():Target{address='FOOT', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties={}, capabilities=null}] [Message:Message{header=Header{durable=false, priority=4, ttl=null, firstAcquirer=false, deliveryCount=0}properties=Properties{messageId=null, userId=null, to='amqp://localhost:5672/FOOT', subject='null', replyTo='null', correlationId=null, contentType=text/plain, contentEncoding=null, absoluteExpiryTime=null, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}body=AmqpValue{Hello World!}}]

The message in the target queue has a good format STRING.

- Piece of AMQP trace using a rabbitmq shovel:

12.06.45.561 49 @37e50b1b com.ibm.mq.MQXRService.AMQPServerSessionV10 ----+---- { receiveMessage [ClientIdentifier:RMQFRRTTST01@sb034tts_b023b904e43d1023] [Receiver:org.apache.qpid.proton.engine.impl.ReceiverImpl@fdee8dd0] [receiver.getRemoteSource():Source{address='null', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}] [receiver.getRemoteTarget():Target{address='RABBIT2MQ', durable=UNSETTLED_STATE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}] [Message:Message{header=Header{durable=true, priority=null, ttl=null, firstAcquirer=null, deliveryCount=null}properties=Properties{messageId=null, userId=null, to='RABBIT2MQ', subject='null', replyTo='null', correlationId=null, contentType=text/plain, contentEncoding=null, absoluteExpiryTime=null, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}body=Data{MSG_1}}]

The message in the target queue has an AMQP format and not a STRING format.

Thank you for your help.

Best regards,

Stéphane

Le lundi 21 octobre 2019 18:58:31 UTC+2, Karl Nilsson a écrit :

A Data section contains just an opaque blob of binary data. There are two other options for "body" data but they are structured amqp values which further away from what you want. See §3.2.6 of the AMQP 1.0 spec.

https://www.amqp.org/resources/specifications

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/74ae51af-d2db-4a32-b574-9d89d80fbc4e%40googlegroups.com.

--
Karl Nilsson

Pivotal/RabbitMQ

Karl Nilsson

unread,

Oct 22, 2019, 4:45:27 AM10/22/19

to rabbitm...@googlegroups.com

Right I see. The content-type has no impact as already explained. RabbitMQ's primary protocol is AMQP 0.9.1 and this is what the internal model is based on so all message data needs to be roundtripped through this protocol meaning the final data type may not be returned exactly the same. This is a technical limitation that we currently may not be able to address although we have future plans that may help. The best option for AMQP 1.0 use now would be to use data frames as source as this maps directly and correctly onto AMQP 0.9.1

Cheers

Karl

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/902d1252-08d4-485b-91f8-600b3100cd64%40googlegroups.com.

--

Karl Nilsson

Pivotal/RabbitMQ

Reply all

Reply to author

Forward