Shovel and Message Properties

[Giscard Fernandes Faria]

Hi folks,

Is there a way to overwrite a message property, or part of the routing key using the shovel plugin?

Basically I would like to change a routing key value from "foo.myname.bar" to "foo.othername.bar".

Or, just change a custom property (i.e. not a AMQP Basic one), let's say property name from myname to othername?

Reading the dynamic shovel documentation it says:

publish-properties

Properties to overwrite when shovelling messages. Setting headers this way is not currently supported. Default is {}.

So, if it is not possible, does your guys have any clue on when it will be supported?

Thanks and Regards.

[Michael Klishin]

You can set certain properties to specific (pre-defined) values:

http://www.rabbitmq.com/shovel-dynamic.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Giscard Fernandes Faria]

Hi Michael, thank you for your time.

I trying the following JSON through REST API:

{"value":
    {
        "src-uri"           : "amqp://admin:admin@localhost:5672",
        "src-queue"         : "a.onpremise",
        "dest-uri"          : "amqp://admin:admin@localhost:5672",
        "dest-queue"        : "a.cloud",
        "prefetch-count"    : 100,
        "publish-properties": {"myheader":"myvalue"}
    }
}

and rabbit is replying:

{"error":"bad_request","reason":"Validation failed\n\nUnrecognised terms [{<<\"myheader\">>,<<\"myvalue\">>}] in publish-properties\n"}

 What is the right way to set it?

Regards

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/yUx8aTamv48/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Giscard Fernandes Faria]

Well, looking the source code I could see there a is "fixed" list of acceptable properties, that is why a custom property isn't accepted:

	Name, [{<<"content_type">>, Str, optional},
	{<<"content_encoding">>, Str, optional},
	{<<"delivery_mode">>, Num, optional},
	{<<"priority">>, Num, optional},
	{<<"correlation_id">>, Str, optional},
	{<<"reply_to">>, Str, optional},
	{<<"expiration">>, Str, optional},
	{<<"message_id">>, Str, optional},
	{<<"timestamp">>, Num, optional},
	{<<"type">>, Str, optional},
	{<<"user_id">>, Str, optional},
	{<<"app_id">>, Str, optional},
	{<<"cluster_id">>, Str, optional}], Term).

Such a feature would be great ;-)

Regards.

[Michael Klishin]

Message properties are pre-defined by the AMQP 0-9-1 protocol spec. Clients can also

publish arbitrary dictionaries in the headers property. Shovel doesn't support that

at the moment. Feel free to file an issue for rabbitmq/rabbitmq-shovel on github.

It should be quite easy to add if we can figure out how arbitrary data structures

can be provided in the headers.

[ppat...@arpae.it]

I cannot use shovel whith messages having user_id as properties if shovel have different user in dest-uri. This is right following policy rules.
But I cannot override the user_id properties :

rabbitmqctl set_parameter shovel test '{"src-protocol": "amqp091", "src-uri": "amqp://userout:passwordout@fromhost", "src-queue": "testout", "dest-protocol": "amqp091", "dest-uri": "amqp://userin:passwordin@", "dest-queue": "testin", "dest-publish-properties":{"user_id":"userin"} }'
Setting runtime parameter "shovel" for component "test" to "{"src-protocol": "amqp091", "src-uri": "amqp://userout:passwordout@fromhost", "src-queue": "testout", "dest-protocol": "amqp091", "dest-uri": "amqp://userin:passwordin@", "dest-queue": "testin", "dest-publish-properties":{"user_id":"userin"} }" in vhost "/" ...
Error:
{{:case_clause, {:error, {{:badmatch, []}, [{:mirrored_supervisor, :child, 2, [file: 'src/mirrored_supervisor.erl', line: 260]}, {:mnesia_tm, :apply_fun, 3, [file: 'mnesia_tm.erl', line: 836]}, {:mnesia_tm, :execute_transaction, 5, [file: 'mnesia_tm.erl', line: 812]}, {:rabbit_misc, :"-execute_mnesia_transaction/1-fun-0-", 1, [file: 'src/rabbit_misc.erl', line: 542]}, {:worker_pool_worker, :"-run/2-fun-0-", 3, [file: 'src/worker_pool_worker.erl', line: 78]}]}}}, [{:rabbit_shovel_dyn_worker_sup_sup, :start_child, 2, [file: 'src/rabbit_shovel_dyn_worker_sup_sup.erl', line: 45]}, {:rabbit_runtime_parameters, :set_any0, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 164]}, {:rabbit_runtime_parameters, :set_any, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 143]}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 197]}]}

What is wrong ?

rabbitmqctl list_parameters
Listing runtime parameters for vhost "/" ...
component    name    value
shovel    test    {"dest-protocol":"amqp091","dest-publish-properties":{"user_id":"userin"},"dest-queue":"testin","dest-uri":"amqp://userin:passwordin@","src-protocol":"amqp091","src-queue":"testout","src-uri":"amqp://userout:passwordout@fromhost"}

rabbitmqctl clear_parameter shovel test
Clearing runtime parameter "test" for component "shovel" on vhost "/" ...
Error:
{{:badmatch, {:error, :not_found}}, [{:rabbit_shovel_dyn_worker_sup_sup, :stop_child, 1, [file: 'src/rabbit_shovel_dyn_worker_sup_sup.erl', line: 61]}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 197]}]}

rabbitmqctl list_parameters
Listing runtime parameters for vhost "/" ...
component    name    value

for me it's a very strange behavior

rabbitmq 3.7.9 Erlang 21.1

Thanks in advance

Paolo Patruno

[Michael Klishin]

RabbitMQ generally doesn't allow you to use arbitrary username properties [1], by design.

1. https://www.rabbitmq.com/validated-user-id.html

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ppat...@arpae.it]

Thanks for your answer.
So is not clear to me which user run the shovel plugin; the user I have to add the impersonator tag.

Also I get error setting other dest-publish-properties as delivery_mode I suppose do not have the same restrictions as user_id:

rabbitmqctl set_parameter shovel test '{"src-protocol": "amqp091", "src-uri": "amqp://src-user:src-password@src-server", "src-queue": "testout", "dest-protocol": "amqp091", "dest-uri": "amqp://dest-user:dest-password@", "dest-queue": "testin", "dest-publish-properties":{"delivery_mode":2}  }'
Setting runtime parameter "shovel" for component "test" to "{"src-protocol": "amqp091", "src-uri": "amqp://src-user:src-password@src-server", "src-queue": "testout", "dest-protocol": "amqp091", "dest-uri": "amqp://dest-user:dest-password@", "dest-queue": "testin", "dest-publish-properties":{"delivery_mode":2}  }" in vhost "/" ...
Error:
{{:case_clause, {:error, {{:shutdown, {:failed_to_start_child, {"/", "test"}, {:function_clause, [{:rabbit_shovel_parameters, :"-lookup_indices/2-lc$^0/1-0-", [%{"delivery_mode" => 2}], [file: 'src/rabbit_shovel_parameters.erl', line: 430]}, {:rabbit_shovel_parameters, :parse_amqp091_dest, 4, [file: 'src/rabbit_shovel_parameters.erl', line: 316]}, {:rabbit_shovel_parameters, :parse, 3, [file: 'src/rabbit_shovel_parameters.erl', line: 241]}, {:rabbit_shovel_worker, :init, 1, [file: 'src/rabbit_shovel_worker.erl', line: 53]}, {:gen_server2, :init_it, 6, [file: 'src/gen_server2.erl', line: 548]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}}, {:child, :undefined, {"/", "test"}, {:rabbit_shovel_dyn_worker_sup, :start_link, [{"/", "test"}, [{"dest-protocol", "amqp091"}, {"dest-publish-properties", %{"delivery_mode" => 2}}, {"dest-queue", "testin"}, {"dest-uri", "amqp://dest-user:dest-password@"}, {"src-protocol", "amqp091"}, {"src-queue", "testout"}, {"src-uri", "amqp://src-user:src-password@src-server"}]]}, :transient, 30000, :worker, [:rabbit_shovel_dyn_worker_sup]}}}}, [{:rabbit_shovel_dyn_worker_sup_sup, :start_child, 2, [file: 'src/rabbit_shovel_dyn_worker_sup_sup.erl', line: 45]}, {:rabbit_runtime_parameters, :set_any0, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 164]}, {:rabbit_runtime_parameters, :set_any, 5, [file: 'src/rabbit_runtime_parameters.erl', line: 143]}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 197]}]}

If I omit  dest-publish-properties all works fine.

Thanks in advance
Paolo

[Michael Klishin]

This list uses one question per thread. Please start a new one and at the very least mention what RabbitMQ version is used.

At the first glance this looks like [1] or something similar.

1. https://github.com/rabbitmq/rabbitmq-shovel/issues/46