Some items in leaf-list are missed after ModelConverter.createDataNode

21 views
Skip to first unread message

hir...@opennetworking.org

unread,
May 21, 2018, 11:32:35 AM5/21/18
to Dynamic configuration
Hi all,

When I convert modelObjectData to ResourceData using ModelConverter.createDataNode,
some items in the leaf-list seems to be purged after conversion.

The patch to reproduce this issue is below:

onos> apps -a -s
*   8 org.onosproject.yang                 1.14.0.SNAPSHOT YANG Compiler and Runtime
*   9 org.onosproject.config               1.14.0.SNAPSHOT Dynamic Configuration
*  10 org.onosproject.configsync           1.14.0.SNAPSHOT Dynamic Configuration Synchronizer
*  11 org.onosproject.faultmanagement      1.14.0.SNAPSHOT Fault Management
*  12 org.onosproject.netconf              1.14.0.SNAPSHOT NETCONF Provider
*  13 org.onosproject.configsync-netconf   1.14.0.SNAPSHOT Dynamic Configuration Synchronizer for NETCONF
*  22 org.onosproject.drivers              1.14.0.SNAPSHOT Default Drivers
*  28 org.onosproject.restconf             1.14.0.SNAPSHOT RESTCONF Application Module
*  51 org.onosproject.protocols.restconfserver 1.14.0.SNAPSHOT RESTCONF Server Module
*  87 org.onosproject.models.tapi          1.14.0.SNAPSHOT ONF Transport API YANG Models
*  88 org.onosproject.models.ietf          1.14.0.SNAPSHOT IETF YANG Models
*  89 org.onosproject.models.openconfig    1.14.0.SNAPSHOT OpenConfig YANG Models
*  90 org.onosproject.odtn-api             1.14.0.SNAPSHOT ODTN API & Utilities Application
*  91 org.onosproject.drivers.netconf      1.14.0.SNAPSHOT Generic NETCONF Drivers
*  92 org.onosproject.drivers.odtn-driver  1.14.0.SNAPSHOT ODTN Driver
*  93 org.onosproject.odtn-service         1.14.0.SNAPSHOT ODTN Service Application
onos> onos:reproduce-issue
Model Data DefaultGetServiceInterfacePointListOutput{sip=[DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=358b49d2-8713-428d-aba1-a4ca7c964afc, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}, DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=fddebebf-fdda-4543-911c-5168ba9bd6d6, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}, DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=a8abff7f-7137-4b9e-be8c-7098b8ddbb49, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}, DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=78eab7c1-1378-4d33-8e11-cc02180dab8d, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}, DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=1c36e622-4108-4f46-9292-a5d8d4a59ecd, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}, DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=ed1fdaeb-2162-4c31-9511-750348470200, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}, DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=75e5ab4c-3383-4401-8505-e37eea030826, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}, DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=7fe55491-435b-4784-aa55-e92b549ce19e, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}, DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=f0a35a0d-4520-4fd9-943f-e844f138969a, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}, DefaultSip{administrativeState=null, operationalState=null, lifecycleState=null, uuid=fa2e2d9f-70d7-4599-b95c-6df76534ab03, layerProtocolName=null, valueLeafFlags={4}, totalPotentialCapacity=null, availableCapacity=null, name=null}]}

There are certainly several DefaultSips in the sip leaf-list.
But after conversion,
 
Resource Data DefaultResourceData{resourceId=ResourceId{nodeKeyList=[NodeKey{schemaId=SchemaId{name=/, nameSpace=null}}, NodeKey{schemaId=SchemaId{name=get-service-interface-point-list, nameSpace=urn:onf:otcc:yang:tapi-common}}]}, nodes=[{key=NodeKey{schemaId=SchemaId{name=output, nameSpace=urn:onf:otcc:yang:tapi-common}}, childNodes=[{key=NodeKey{schemaId=SchemaId{name=sip, nameSpace=urn:onf:otcc:yang:tapi-common}}, childNodes=[{key=NodeKey{schemaId=SchemaId{name=uuid, nameSpace=urn:onf:otcc:yang:tapi-common}}, value=fa2e2d9f-70d7-4599-b95c-6df76534ab03valueNamespace=nullleafType=STRING}]}]}]}
XML:
<output xmlns="urn:onf:otcc:yang:tapi-common">
  <sip>
    <uuid>fa2e2d9f-70d7-4599-b95c-6df76534ab03</uuid>
  </sip>
</output>

only 1 sip is remained.


Could you let me know your thoughts?

Thanks,
Hiroki 

Yuta Higuchi

unread,
May 22, 2018, 1:36:37 PM5/22/18
to Hiroki Okui, Dynamic configuration
Hi Gaurav, 

To add on to this issue,

I've slightly modified the reproducer code with the patch attached at the end applied, to see which one is getting lost during the process
and the result of reproduce-issue command after the patch was that
only element which got added last was surviving.

Looking inside the yang runtime code, it seemed like that in the end of
 DataTreeBuilderHelper.processMultiInstanceNode(YangNode, DataTreeNodeInfo, DataTreeNodeInfo, DataTreeNodeInfo)
it calls processChildNode(..) where inside it, 
it always calls, createChildBuilder() which creates a new instance of the Builder.

It seemed like a this could overwrite previous objects when walking over siblings.
without build()-ing it. 

But even if we had a way around to avoid creating of new Builder, 
InnerNode.Builder maintain child as Map<NodeKey, DataNode>
so in the case of list of Sip(s) NodeKey will be equals to each other so, 
it seems it will get lost when multiple Sips got added there.

For the latter, I think we might be able to workaround it by replacing Map with Multimap, 
but we need to assume that 
InnerNode.Builder#addNode was never used with an intention to override and update existing children.
Does that sound like a reasonable assumption for addNode(..) usage?


# last result of reproduce-issue command
XML:
<output xmlns="urn:onf:otcc:yang:tapi-common">
  <sip>
    <uuid>9</uuid>
  </sip>
</output>



# patch: instead of random UUID, just write integer.

diff --git a/apps/odtn/service/src/main/java/org/onosproject/odtn/cli/impl/Reproducer.java b/apps/odtn/service/src/main/java/org/onosproject/odtn/cli/impl/Reproducer.java
index de9d0b8779..2c09c53d48 100644
--- a/apps/odtn/service/src/main/java/org/onosproject/odtn/cli/impl/Reproducer.java
+++ b/apps/odtn/service/src/main/java/org/onosproject/odtn/cli/impl/Reproducer.java
@@ -16,7 +16,6 @@

 package org.onosproject.odtn.cli.impl;

-import java.util.UUID;
 import java.util.regex.Pattern;
 import org.apache.karaf.shell.commands.Command;
 import org.onlab.util.XmlString;
@@ -66,7 +65,7 @@ public class Reproducer extends AbstractShellCommand {
         DefaultGetServiceInterfacePointListOutput output = new DefaultGetServiceInterfacePointListOutput();
         for (int i = 0; i < 10; i++) {
             Sip sip = new DefaultSip();
-            sip.uuid(Uuid.fromString(UUID.randomUUID().toString()));
+            sip.uuid(Uuid.fromString(String.valueOf(i)));
             output.addToSip(sip);
         }
         return output;



--
You received this message because you are subscribed to the Google Groups "Dynamic configuration" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dynconfig+unsubscribe@onosproject.org.
To post to this group, send email to brigade-dynconfig@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/brigade-dynconfig/.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/brigade-dynconfig/6cdac970-8b60-469a-9c93-ed31c16375bd%40onosproject.org.
For more options, visit https://groups.google.com/a/onosproject.org/d/optout.

Gaurav agrawal

unread,
May 24, 2018, 10:56:41 AM5/24/18
to Yuta Higuchi, Hiroki Okui, Vidyashree Rama, Dynamic configuration

@Vidya, please look into the same.

 

Thanks and Regards,

Gaurav Agrawal

Senior System Architect

Network Software Platform Dept, HTIPL [2012 Laboratories]


Huawei Technologies India Pvt. Ltd
Survey No. 37, Next to EPIP Area, Kundanahalli, Whitefield
Benguluru-560066, Karnataka
Tel: +91-80-49160700, Ext 70126 || Mob : 7838700296, Email : gaurav....@huawei.com

--

To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dyncon...@onosproject.org.
To post to this group, send email to brigade-...@onosproject.org.

 

--

You received this message because you are subscribed to the Google Groups "Dynamic configuration" group.

To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dyncon...@onosproject.org.
To post to this group, send email to brigade-...@onosproject.org.

Vidya Bijoor

unread,
May 29, 2018, 9:41:11 AM5/29/18
to Gaurav agrawal, Yuta Higuchi, Hiroki Okui, Vidyashree Rama, Dynamic configuration
rpc get-service-interface-point-list {
    description "none";
    output {
        list sip {
            uses service-interface-point;
            description "none";
        }
    }
}
sip is list without key and is currently not supported in onos-yang-tools. 
This issue is same as the issue reported earlier.
======================================================
I am unsure how to proceed. I understand this is considered a bug that will be addressed in the future, or should lists in yang models be restricted to have a key?

Let me know if I can be of any help.
Thanks
Ramon

On Friday, February 16, 2018 at 7:43:17 AM UTC+1, Vidya Bijoor wrote:
Regarding issue 2, Only one entry of end-point is created in datanode

        rpc create-connectivity-service {
            description "none";
            input {
                list end-point {
                    min-elements 2;
                    uses connectivity-service-end-point;
                    description "none";
                }

end-point is list without key and so list entries are overwritten while creating dataNodes.

Thanks,
Vidya

On Fri, Feb 16, 2018 at 1:40 AM, Yuta Higuchi <y-hi...@opennetworking.org> wrote:
I think I have a fix for Issue 1 and possibly Issue 3 latter half in place.

After the patch, I was able to push OpenConfig interfaces tree to the emulator using that reproducer code.


On Thu, Feb 15, 2018 at 11:17 AM, Yuta Higuchi <y-hi...@opennetworking.org> wrote:
Thanks for summarising.

Regarding issue 1 original report, we've asked Hiroki (or possibly Sean) to see if they can come up with reproducer or reproduction steps.

Regarding issue 1 second report which seems to be LeafNode valueNamespace issue, thanks for the analysis, I will investigate where it was impacted.

Regarding issue 3, I guess this also has same root cause, originating from LeafNode's internal change, 
hopefully fix for issue 1 will resolve both issue 1 and 3.


Off-Topic, but I think these incidents are itself a good data point that we should decouple DCS from yang models, 
and avoid using DataNode as storage format, etc. in DCS 2.0 design,
so that each can evolve independently.







-- 
You received this message because you are subscribed to the Google Groups "Dynamic configuration" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dynconfig+unsubscribe@onosproject.org.
To post to this group, send email to brigade-dynconfig@onosproject.org.
Vidya Bijoor <vidyash...@gmail.com>
Feb 28
to Ramon, Dynamic, Yuta, Gaurav, Hiroki, Henry, Sonu, Appana, Kalyankumar
 Currently yang tools doesn't support list without keys, its new requirement which has to be addressed in future, it is not planned currently.

Thanks,
Vidya
Yuta Higuchi <y-hi...@opennetworking.org>
Mar 2
to me, Ramon, Dynamic, Gaurav, Hiroki, Henry, Sonu, Appana, Kalyankumar
Hi Vidya,

As part of ResourceId to instance-identifier conversion exercise,
  https://gerrit.onosproject.org/17279

I did notice that in ResourceId
there is no corresponding construct for pos (positional identifier?)

Are we talking about the same thing?

   instance-identifier = 1*("/" (node-identifier
                                 [1*key-predicate /
                                  leaf-list-predicate /
                                  pos]))

   key-predicate       = "[" *WSP key-predicate-expr *WSP "]"

   key-predicate-expr  = node-identifier *WSP "=" *WSP quoted-string

   leaf-list-predicate = "[" *WSP leaf-list-predicate-expr *WSP "]"

   leaf-list-predicate-expr = "." *WSP "=" *WSP quoted-string

   pos                 = "[" *WSP positive-integer-value *WSP "]"
Vidya Bijoor <vidyash...@gmail.com>
Mar 2
to Yuta, Ramon, Dynamic, Gaurav, Hiroki, Henry, Sonu, Appana, Kalyankumar
yes, positional identifier is one of the requirement to support list without keys.

Thanks,
Vidya

--

To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dynconfig+unsubscribe@onosproject.org.
To post to this group, send email to brigade-dynconfig@onosproject.org.

--
You received this message because you are subscribed to the Google Groups "Dynamic configuration" group.

To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dynconfig+unsubscribe@onosproject.org.
To post to this group, send email to brigade-dynconfig@onosproject.org.

--
You received this message because you are subscribed to the Google Groups "Dynamic configuration" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dynconfig+unsubscribe@onosproject.org.
To post to this group, send email to brigade-dynconfig@onosproject.org.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/brigade-dynconfig/2F2059F256F9B24F82EAC5EE47F446C6E945108A%40dggemi501-mbx.china.huawei.com.

Ramon Casellas

unread,
May 29, 2018, 10:16:04 AM5/29/18
to Vidya Bijoor, Gaurav agrawal, Yuta Higuchi, Hiroki Okui, Vidyashree Rama, Dynamic configuration
Thank you for the analysis, and sorry this was a duplicate.

We reported this upstream, but I guess we need to make sure that all lists are keyed.

Best regards
Ramon


To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dyncon...@onosproject.org.
To post to this group, send email to brigade-...@onosproject.org.

--

To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dyncon...@onosproject.org.
To post to this group, send email to brigade-...@onosproject.org.

--
You received this message because you are subscribed to the Google Groups "Dynamic configuration" group.

To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dyncon...@onosproject.org.
To post to this group, send email to brigade-...@onosproject.org.

--
You received this message because you are subscribed to the Google Groups "Dynamic configuration" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dyncon...@onosproject.org.
To post to this group, send email to brigade-...@onosproject.org.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/brigade-dynconfig/2F2059F256F9B24F82EAC5EE47F446C6E945108A%40dggemi501-mbx.china.huawei.com.

--
You received this message because you are subscribed to the Google Groups "Dynamic configuration" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dyncon...@onosproject.org.
To post to this group, send email to brigade-...@onosproject.org.

Visit this group at https://groups.google.com/a/onosproject.org/group/brigade-dynconfig/.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/brigade-dynconfig/CAPRmjeuBonbCcmPg7-gQFFcrGfcv4r8C9_18_cwfMVbg0Z%2BuHQ%40mail.gmail.com.

hir...@opennetworking.org

unread,
May 29, 2018, 1:16:30 PM5/29/18
to Dynamic configuration, vidyash...@gmail.com, gaurav....@huawei.com, y-hi...@opennetworking.org, hir...@opennetworking.org, Vidyash...@huawei.com
HI,

Thank you for your comments.
I've forgotten the issue, I'm sorry for duplicating.

But a list without key doesn't violate yang RFCs, so I hope onos-yang-tool will support it in the future.

Thanks,
Hiroki
To unsubscribe from this group and stop receiving emails from it, send an email to brigade-dynconfig+unsub...@onosproject.org.
Reply all
Reply to author
Forward
0 new messages