command commit ordering

[Karthikeyan Natarajan]

Hi All,

       I have entered the configuration commands
in confd shell in the following order:

cmd1 (server <address> key<>)

cmd2  key <num>

cmd3 (server <address> key<>)

cmd4 (key<num>)

commit

    I assumed that confd will commit these commands in the
same order it was entered. But, I am seeing that commit
is done in the following order.

cmd1 (server <address> key<>)
cmd3 (server <address> key<>)
cmd2  key <num>
cmd4 (key<num>)

    Looks like confd is grouping the same type of commands and then

commit each group.

    Could anyone please let me know how to enforce the comamnds
commit ordering?

Thanks,

-karthi

[chriss]

Hi Karthikeyan,

I think we need to first get some basic terminology straight. The "commit" occurs when you type "commit" and it enters the CDB database as an atomic action - you don't really see an "order" per se, it is invisible to you. What I think you are describing is the order of subscription notifications, using diff iterate() Is that correct? In this case, ConfD will traverse the change "tree" in a certain order, with like items grouped as you reported, completely unrelated to the order in which the commands were entered preceding the "commit" command. They could have been entered in a different order. ConfD will sort the changes based on the schema and generate subscription activity.

Is there something in your design which precludes responding to the subscriptions in the order observed? If you need to gather a certain amount of info before applying it to your system backend, there are various tricks and techniques available. You can also specify subscription priority, higher  numbers are delivered later, but I don't think that won't work here because that is normally used to ensure certain namespaces or containers are processed ahead of others. Here you actually want to interleave items from different areas.

If you must process the subscriptions in the order you illustrated, one technique you can use is to subscribe more than one time, or use two-phase subscriptions.Use the first subscription or subscription phase to accumulate config changes; use the second to apply the gathered info in bulk, in the order you desire. The second subscription does not have to traverse the tree, presumably it can just process all the gathered changes and return ITER_STOP .But this technique tricky too, you have to handle every kind of subscription event correctly - modifies, creates, deletes, etc. and make sure the accumulated changes will result in the desired outcome.

Another technique is to do a bulk config fetch using cdb_get() APIs of these interdependent objects in the first subscription callback and process everything at once, applying the entire configuration. then return ITER_STOP. This avoids accumulating changes. It might cause performance issues if the datasets are large, but it is kind of simple and brute-force. This is a good way to handle stuff which ends up being written to .conf files to control things like syslogd, ntpd, sshd, etc. These events don't occur very often, the data sets are modest and you need all of them together anyway in bulk to write a coherent config file.

Feel free to clarify the problem your trying to solve and maybe I can offer more suggestions.

- chris

[Karthikeyan Natarajan]

Hi Chris,

     Thanks for your detailed response.

Yes, I am describing about the subscription notification order.

Basically, I need to do the validation for one command depending
on the value configured in another command.

key <num>
server <addr> key <num>

In this case, 'server command' will be declared invalid, if the key value
used in that command is not configured using 'key command'.

valid server command:

key 10

server 20.20.20.20 key 10

invalid server command:

key 15

server 10.10.10.10 key 20

    Please let me know how to do the validation for server command here.

Thanks,

-karthi

--
You received this message because you are subscribed to the Google Groups "ConfD User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confd-user-gro...@googlegroups.com.
To post to this group, send an email to confd-us...@googlegroups.com.
Visit this group at http://groups.google.com/group/confd-user-group.
For more options, visit https://groups.google.com/groups/opt_out.

[chriss]

Hi Karthi,

Well, as it turns out, you need to do your validation before the configuration gets committed at all! Your problem has nothing do with subscriptions or commit order or command order, it is a validation issue. The situation you describe is actually quite common. What you need to do is define a constraint using a yang "must" statement. This will get evaluated before ConfD even tries to commit it. For example, in the yang model for server you'd have a leaf called "key" of some type, e.g. serverKeyType or int or something. In that leaf you'd put the following must statement (I am using assumed names for some items because I do't have your yang models):

import key {  <== guessing
prefix key;

}

list server {

key index;  <=== guessing

leaf index {...

leaf key {

type int;

must "/key:key[key:index=current()])" {

    error-message "key must exist";

    tailf:dependency "/key:key";

    tailf:dependency ".";

}

What this says is "for the proposed value of server/key, there must be an instance of key{n} where n = the value of this server/key,  which the current() function gives." Behold the power of yang! All this gets evaluated by Confd when you type "validate" or "commit." It occurs in the pending transaction. If it doesn't evaluate to true, it won't commit until you correct it or revert. A disclaimer - the syntax I gave might be inexact, but it illustrates the point. We've written countless statements like this in our code.

To debug this stuff, configure your confdConfig/logs/xpathTraceLog enabled, set a file name, and watch the file output when you try to commit or validate. You may have to refine the expressions with some trial and error, it takes a while to get the hang of it.

I don't know the namespaces or names of your indices (list keys or server) so I am using placeholders.

in general, you should avoid validation in a subscription in less it is unavoidable. If something is invalid in a subscription phase 1 (prepare), you can roll back the transaction. In phase 2, the data is already committed to CDB so even if you reject it in a subscription callback, it's too late, your system is out of balance because you've got data committed which cannot be consumed by the system. Always avoid letting data get committed which is not acceptable.