[snmp-exporter] lookups end up with empty values for new label

1,877 views
Skip to first unread message

Danny Kulchinsky

unread,
Jun 19, 2018, 1:47:08 PM6/19/18
to Prometheus Users
Dear experts!


snmp_exporter v0.11.0
generator from HEAD in master


For test purposes, I'm just trying to get one OID: fwAcceptPcktsIn (1.3.6.1.4.1.2620.1.1.25.5.1.5) and replace the fwIfIndex label value with fwIfName, so I setup the following in generator.yml:

modules:
  cpfw:
    walk:
      - fwAcceptPcktsIn
    auth:
      community: <.....>
    lookups:
      - old_index: fwIfIndex
        new_index: fwIfName


Here's the snmp.yml:

cpfw:
  walk:
  - 1.3.6.1.4.1.2620.1.1.25.5.1.2
  - 1.3.6.1.4.1.2620.1.1.25.5.1.5
  metrics:
  - name: fwAcceptPcktsIn
    oid: 1.3.6.1.4.1.2620.1.1.25.5.1.5
    type: counter
    help: ' - 1.3.6.1.4.1.2620.1.1.25.5.1.5'
    indexes:
    - labelname: fwIfName
      type: counter
    lookups:
    - labels:
      - fwIfName
      labelname: fwIfName
      oid: 1.3.6.1.4.1.2620.1.1.25.5.1.2
      type: DisplayString
  auth:
    community: <....>


However, I'm getting the following error from snmp_exporter:

An error has occurred during metrics gathering:

25 error(s) occurred:
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:62238 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:67028 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:13176 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:1808 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:112693 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:1 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:94716 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:0 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:74566 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:0 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:614348 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:214 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:4971 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:41023 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:5761 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:0 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:0 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:52726 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:3825 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:19324 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:438273 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:28033 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:5621 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:0 >  was collected before with the same name and label values
* collected metric fwAcceptPcktsIn label:<name:"fwIfName" value:"" > counter:<value:7603 >  was collected before with the same name and label values

I can't seem to figure out why above output shows an empty value for fwIfName label (<name:"fwIfName" value:"" >) which I believe what causes the duplicate metrics in this case.

If I include the fwAcceptPcktsIn, fwIfIndex and fwIfName OIDs (without lookups) I get the following metrics.

# HELP fwAcceptPcktsIn  - 1.3.6.1.4.1.2620.1.1.25.5.1.5
# TYPE fwAcceptPcktsIn counter
fwAcceptPcktsIn{fwIfIndex="10"} 0
fwAcceptPcktsIn{fwIfIndex="11"} 0
fwAcceptPcktsIn{fwIfIndex="12"} 5590
fwAcceptPcktsIn{fwIfIndex="13"} 70023
fwAcceptPcktsIn{fwIfIndex="14"} 106555
fwAcceptPcktsIn{fwIfIndex="15"} 8551
fwAcceptPcktsIn{fwIfIndex="16"} 75476
fwAcceptPcktsIn{fwIfIndex="17"} 83832
fwAcceptPcktsIn{fwIfIndex="18"} 57929
fwAcceptPcktsIn{fwIfIndex="19"} 14925
fwAcceptPcktsIn{fwIfIndex="2"} 21792
fwAcceptPcktsIn{fwIfIndex="20"} 45751
fwAcceptPcktsIn{fwIfIndex="21"} 6118
fwAcceptPcktsIn{fwIfIndex="22"} 1999
fwAcceptPcktsIn{fwIfIndex="23"} 239
fwAcceptPcktsIn{fwIfIndex="24"} 6478
fwAcceptPcktsIn{fwIfIndex="25"} 126726
fwAcceptPcktsIn{fwIfIndex="26"} 4099
fwAcceptPcktsIn{fwIfIndex="27"} 0
fwAcceptPcktsIn{fwIfIndex="3"} 0
fwAcceptPcktsIn{fwIfIndex="4"} 690207
fwAcceptPcktsIn{fwIfIndex="5"} 488788
fwAcceptPcktsIn{fwIfIndex="6"} 0
fwAcceptPcktsIn{fwIfIndex="7"} 1
fwAcceptPcktsIn{fwIfIndex="8"} 31512
fwAcceptPcktsIn{fwIfIndex="9"} 5579
# HELP fwIfIndex  - 1.3.6.1.4.1.2620.1.1.25.5.1.1
# TYPE fwIfIndex counter
fwIfIndex{fwIfIndex="10"} 10
fwIfIndex{fwIfIndex="11"} 11
fwIfIndex{fwIfIndex="12"} 12
fwIfIndex{fwIfIndex="13"} 13
fwIfIndex{fwIfIndex="14"} 14
fwIfIndex{fwIfIndex="15"} 15
fwIfIndex{fwIfIndex="16"} 16
fwIfIndex{fwIfIndex="17"} 17
fwIfIndex{fwIfIndex="18"} 18
fwIfIndex{fwIfIndex="19"} 19
fwIfIndex{fwIfIndex="2"} 2
fwIfIndex{fwIfIndex="20"} 20
fwIfIndex{fwIfIndex="21"} 21
fwIfIndex{fwIfIndex="22"} 22
fwIfIndex{fwIfIndex="23"} 23
fwIfIndex{fwIfIndex="24"} 24
fwIfIndex{fwIfIndex="25"} 25
fwIfIndex{fwIfIndex="26"} 26
fwIfIndex{fwIfIndex="27"} 27
fwIfIndex{fwIfIndex="3"} 3
fwIfIndex{fwIfIndex="4"} 4
fwIfIndex{fwIfIndex="5"} 5
fwIfIndex{fwIfIndex="6"} 6
fwIfIndex{fwIfIndex="7"} 7
fwIfIndex{fwIfIndex="8"} 8
fwIfIndex{fwIfIndex="9"} 9
# HELP fwIfName  - 1.3.6.1.4.1.2620.1.1.25.5.1.2
# TYPE fwIfName gauge
fwIfName{fwIfIndex="10",fwIfName="eth1.266"} 1
fwIfName{fwIfIndex="11",fwIfName="eth1.265"} 1
fwIfName{fwIfIndex="12",fwIfName="eth2-02.641"} 1
fwIfName{fwIfIndex="13",fwIfName="eth2-02.273"} 1
fwIfName{fwIfIndex="14",fwIfName="eth2-02.290"} 1
fwIfName{fwIfIndex="15",fwIfName="eth1.246"} 1
fwIfName{fwIfIndex="16",fwIfName="eth2-02.288"} 1
fwIfName{fwIfIndex="17",fwIfName="eth2-02.268"} 1
fwIfName{fwIfIndex="18",fwIfName="eth2-02.610"} 1
fwIfName{fwIfIndex="19",fwIfName="eth1.294"} 1
fwIfName{fwIfIndex="2",fwIfName="eth2-01"} 1
fwIfName{fwIfIndex="20",fwIfName="eth2-02.631"} 1
fwIfName{fwIfIndex="21",fwIfName="eth2-02.621"} 1
fwIfName{fwIfIndex="22",fwIfName="eth1.244"} 1
fwIfName{fwIfIndex="23",fwIfName="eth2-02.269"} 1
fwIfName{fwIfIndex="24",fwIfName="eth1.292"} 1
fwIfName{fwIfIndex="25",fwIfName="eth1.291"} 1
fwIfName{fwIfIndex="26",fwIfName="eth1.240"} 1
fwIfName{fwIfIndex="27",fwIfName="eth1.242"} 1
fwIfName{fwIfIndex="3",fwIfName="eth2-02"} 1
fwIfName{fwIfIndex="4",fwIfName="eth2-03"} 1
fwIfName{fwIfIndex="5",fwIfName="eth2-04"} 1
fwIfName{fwIfIndex="6",fwIfName="eth1"} 1
fwIfName{fwIfIndex="7",fwIfName="Sync"} 1
fwIfName{fwIfIndex="8",fwIfName="eth1.241"} 1
fwIfName{fwIfIndex="9",fwIfName="eth2-02.210"} 1
# HELP snmp_scrape_duration_seconds Total SNMP time scrape took (walk and processing).
# TYPE snmp_scrape_duration_seconds gauge
snmp_scrape_duration_seconds 0.007003819
# HELP snmp_scrape_pdus_returned PDUs returned from walk.
# TYPE snmp_scrape_pdus_returned gauge
snmp_scrape_pdus_returned 78
# HELP snmp_scrape_walk_duration_seconds Time SNMP walk/bulkwalk took.
# TYPE snmp_scrape_walk_duration_seconds gauge
snmp_scrape_walk_duration_seconds 0.006071158



Not quite sure what I'm doing wrong, this same method works fine for me on other devices.


Any help/advise would be very appreciated :)


Regards,
Danny




Ben Kochie

unread,
Jun 19, 2018, 3:38:27 PM6/19/18
to Danny Kulchinsky, Prometheus Users
That should work in theory. Do you see any issues if you turn the snmp_exporter log level up to debug?

What do you get for this snmp.yml config? Maybe you need to explicitly walk the fwIfIndex as well.

cpfw:
  walk:
  - 1.3.6.1.4.1.2620.1.1.25.5.1.1
  - 1.3.6.1.4.1.2620.1.1.25.5.1.2
  - 1.3.6.1.4.1.2620.1.1.25.5.1.5
  - name: fwAcceptPcktsIn
    oid: 1.3.6.1.4.1.2620.1.1.25.5.1.5
    type: gauge
    help: ' - 1.3.6.1.4.1.2620.1.1.25.5.1.5'
    indexes:
    - labelname: fwIfIndex
      type: gauge
    lookups:
    - labels:
      - fwIfIndex
      labelname: fwIfName
      oid: 1.3.6.1.4.1.2620.1.1.25.5.1.2
      type: DisplayString
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/24c4a329-aaa8-4d0b-a68b-025de8c4581a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Brian Brazil

unread,
Jun 19, 2018, 3:49:03 PM6/19/18
to Ben Kochie, Danny Kulchinsky, Prometheus Users
On 19 June 2018 at 20:38, Ben Kochie <sup...@gmail.com> wrote:
That should work in theory. Do you see any issues if you turn the snmp_exporter log level up to debug?

What do you get for this snmp.yml config?


 
Maybe you need to explicitly walk the fwIfIndex as well.

That shouldn't be required, it's that index is already fully in the oid. It's probably that that lookup isn't unique.

Brian 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpsNOaCzA-WSdZ0v_Nga3_57Zj8-yVYt8vjc9Ob7BP%3DpQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--

Ben Kochie

unread,
Jun 19, 2018, 3:51:06 PM6/19/18
to Brian Brazil, Danny Kulchinsky, Prometheus Users
On Tue, Jun 19, 2018 at 9:49 PM Brian Brazil <brian....@robustperception.io> wrote:
On 19 June 2018 at 20:38, Ben Kochie <sup...@gmail.com> wrote:
That should work in theory. Do you see any issues if you turn the snmp_exporter log level up to debug?

What do you get for this snmp.yml config?


 
Maybe you need to explicitly walk the fwIfIndex as well.

That shouldn't be required, it's that index is already fully in the oid. It's probably that that lookup isn't unique.

It looks like the label value is coming back as empty when it tries to apply the lookup. That's very strange.
 

Brian 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--

Brian Brazil

unread,
Jun 19, 2018, 3:55:00 PM6/19/18
to Ben Kochie, Danny Kulchinsky, Prometheus Users
On 19 June 2018 at 20:50, Ben Kochie <sup...@gmail.com> wrote:

On Tue, Jun 19, 2018 at 9:49 PM Brian Brazil <brian.brazil@robustperception.io> wrote:
On 19 June 2018 at 20:38, Ben Kochie <sup...@gmail.com> wrote:
That should work in theory. Do you see any issues if you turn the snmp_exporter log level up to debug?

What do you get for this snmp.yml config?


 
Maybe you need to explicitly walk the fwIfIndex as well.

That shouldn't be required, it's that index is already fully in the oid. It's probably that that lookup isn't unique.

It looks like the label value is coming back as empty when it tries to apply the lookup. That's very strange.

Yes.


Could you share your generator.yml for generating that good /metrics output? It doesn't line up with what you provided.

Brian
 
 

Brian 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.



--



--

Danny Kulchinsky

unread,
Jun 19, 2018, 5:38:28 PM6/19/18
to Prometheus Users
Guys, sorry for the delay.

Here's the generator.yml for the "good" output:

# CheckPoint FW R77.30

modules:
  cpfw:
    walk:
      - fwIfIndex
      - fwIfName
      - fwAcceptPcktsIn
    auth:
      community: <...>



On Tuesday, 19 June 2018 15:55:00 UTC-4, Brian Brazil wrote:
On 19 June 2018 at 20:50, Ben Kochie <sup...@gmail.com> wrote:
 

Brian 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--



--

Brian Brazil

unread,
Jun 19, 2018, 5:41:46 PM6/19/18
to Danny Kulchinsky, Prometheus Users
On 19 June 2018 at 22:38, Danny Kulchinsky <dann...@gmail.com> wrote:
Guys, sorry for the delay.

Here's the generator.yml for the "good" output:

# CheckPoint FW R77.30

modules:
  cpfw:
    walk:
      - fwIfIndex
      - fwIfName
      - fwAcceptPcktsIn
    auth:
      community: <...>


Can you try that without the fwIfIndex?

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/34bcffad-3525-4794-a6d7-3bc1551b8d50%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Danny Kulchinsky

unread,
Jun 19, 2018, 6:23:28 PM6/19/18
to Prometheus Users
Adding the response here as replied only to Brian by mistake :)

Here's the generator.yml (without fwIfIndex):

# CheckPoint FW R77.30
modules:
  cpfw:
    walk:

      - fwIfName
      - fwAcceptPcktsIn
    auth:
      community: <...>



Here's the resulting snmp.yml:

cpfw:
  walk:
  - 1.3.6.1.4.1.2620.1.1.25.5.1.2
  - 1.3.6.1.4.1.2620.1.1.25.5.1.5
  metrics:

  - name: fwIfName
    oid: 1.3.6.1.4.1.2620.1.1.25.5.1.2
    type: DisplayString
    help: ' - 1.3.6.1.4.1.2620.1.1.25.5.1.2'
    indexes:
    - labelname: fwIfIndex
      type: counter
  - name: fwAcceptPcktsIn
    oid: 1.3.6.1.4.1.2620.1.1.25.5.1.5

    type: counter
    help: ' - 1.3.6.1.4.1.2620.1.1.25.5.1.5'
    indexes:

    - labelname: fwIfIndex
      type: counter
  auth:
    community: <....>



and these are the metrics:

# HELP fwAcceptPcktsIn  - 1.3.6.1.4.1.2620.1.1.25.5.1.5
# TYPE fwAcceptPcktsIn counter
fwAcceptPcktsIn{fwIfIndex="10"} 0
fwAcceptPcktsIn{fwIfIndex="11"
} 0
fwAcceptPcktsIn{fwIfIndex="12"} 13904
fwAcceptPcktsIn{fwIfIndex="13"} 184284
fwAcceptPcktsIn{fwIfIndex="14"} 279823
fwAcceptPcktsIn{fwIfIndex="15"} 21976
fwAcceptPcktsIn{fwIfIndex="16"} 199160
fwAcceptPcktsIn{fwIfIndex="17"} 220389
fwAcceptPcktsIn{fwIfIndex="18"} 183565
fwAcceptPcktsIn{fwIfIndex="19"} 39227
fwAcceptPcktsIn{fwIfIndex="2"} 56649
fwAcceptPcktsIn{fwIfIndex="20"} 115641
fwAcceptPcktsIn{fwIfIndex="21"} 15261
fwAcceptPcktsIn{fwIfIndex="22"} 5031
fwAcceptPcktsIn{fwIfIndex="23"} 616
fwAcceptPcktsIn{fwIfIndex="24"} 18003
fwAcceptPcktsIn{fwIfIndex="25"} 333053
fwAcceptPcktsIn{fwIfIndex="26"} 8310
fwAcceptPcktsIn{fwIfIndex="27"} 0
fwAcceptPcktsIn{fwIfIndex="3"} 0
fwAcceptPcktsIn{fwIfIndex="4"} 1.80565e+06
fwAcceptPcktsIn{fwIfIndex="5"} 1.264355e+06
fwAcceptPcktsIn{fwIfIndex="6"} 0
fwAcceptPcktsIn{fwIfIndex="7"} 1
fwAcceptPcktsIn{fwIfIndex="8"} 79532
fwAcceptPcktsIn{fwIfIndex="9"} 14797
# HELP fwIfName  - 1.3.6.1.4.1.2620.1.1.25.5.1.2
# TYPE fwIfName gauge
fwIfName{fwIfIndex="10",fwIfName="eth1.266"} 1
fwIfName{fwIfIndex="11",fwIfName="eth1.265"} 1
fwIfName{fwIfIndex="12",fwIfName="eth2-02.641"} 1
fwIfName{fwIfIndex="13",fwIfName="eth2-02.273"} 1
fwIfName{fwIfIndex="14",fwIfName="eth2-02.290"} 1
fwIfName{fwIfIndex="15",fwIfName="eth1.246"} 1
fwIfName{fwIfIndex="16",fwIfName="eth2-02.288"} 1
fwIfName{fwIfIndex="17",fwIfName="eth2-02.268"} 1
fwIfName{fwIfIndex="18",fwIfName="eth2-02.610"} 1
fwIfName{fwIfIndex="19",fwIfName="eth1.294"} 1
fwIfName{fwIfIndex="2",fwIfName="eth2-01"} 1
fwIfName{fwIfIndex="20",fwIfName="eth2-02.631"} 1
fwIfName{fwIfIndex="21",fwIfName="eth2-02.621"} 1
fwIfName{fwIfIndex="22",fwIfName="eth1.244"} 1
fwIfName{fwIfIndex="23",fwIfName="eth2-02.269"} 1
fwIfName{fwIfIndex="24",fwIfName="eth1.292"} 1
fwIfName{fwIfIndex="25",fwIfName="eth1.291"} 1
fwIfName{fwIfIndex="26",fwIfName="eth1.240"} 1
fwIfName{fwIfIndex="27",fwIfName="eth1.242"} 1
fwIfName{fwIfIndex="3",fwIfName="eth2-02"} 1
fwIfName{fwIfIndex="4",fwIfName="eth2-03"} 1
fwIfName{fwIfIndex="5",fwIfName="eth2-04"} 1
fwIfName{fwIfIndex="6",fwIfName="eth1"} 1
fwIfName{fwIfIndex="7",fwIfName="Sync"} 1
fwIfName{fwIfIndex="8",fwIfName="eth1.241"} 1
fwIfName{fwIfIndex="9",fwIfName="eth2-02.210"} 1
# HELP snmp_scrape_duration_seconds Total SNMP time scrape took (walk and processing).
# TYPE snmp_scrape_duration_seconds gauge
snmp_scrape_duration_seconds 0.00550708
# HELP snmp_scrape_pdus_returned PDUs returned from walk.
# TYPE snmp_scrape_pdus_returned gauge
snmp_scrape_pdus_returned 52
# HELP snmp_scrape_walk_duration_seconds Time SNMP walk/bulkwalk took.
# TYPE snmp_scrape_walk_duration_seconds gauge
snmp_scrape_walk_duration_seconds 0.004812931


I don't see any duplicates in fwIfIndex and it actually looks ok... I'm really not sure what could be the issue :(


Danny




-- 




Brian Brazil
unread,
Jun 19, 2018, 6:41:41 PM6/19/18
to Danny Kulchinsky, Prometheus Users
Yeah, that all looks fine. I was thinking maybe the device had issues with too many repeaters (a not uncommon device bug). This will take some more serious debugging.

Brian
 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.

To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/03be2cc8-b5e6-4185-a375-c057c6bf5b5f%40googlegroups.com.


For more options, visit https://groups.google.com/d/optout.





-- 




Danny Kulchinsky
unread,
Jun 19, 2018, 6:43:08 PM6/19/18
to Brian Brazil, Prometheus Users
I understand, anything I can do to help?

Perhaps run the "bad" config with debugging enabled?

Danny




-- 






Brian Brazil
unread,
Jun 19, 2018, 6:52:31 PM6/19/18
to Danny Kulchinsky, Prometheus Users
On 19 June 2018 at 23:42, Danny Kulchinsky <dann...@gmail.com> wrote:
I understand, anything I can do to help?

Perhaps run the "bad" config with debugging enabled?

Can't hurt, though I doubt it'll provide any useful information. Maybe tcpdump to be sure the device isn't sending dodgy data when you're using lookups (this possibility should already be eliminated, but doesn't hurt to double check).

Do you have the raw oids from an snmpbulkwalk?

Brian
 

Danny


To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.

To post to this group, send email to prometheus-users@googlegroups.com.



-- 









-- 




Danny Kulchinsky
unread,
Jun 19, 2018, 7:29:28 PM6/19/18
to Prometheus Users
Thanks Brian, I sent you the tcpdump in a private message (it's a bit sensitive).

I don't have an output from an snmpbulkwalk, but I can get it if needed.



Some additional info:

generator.yml:
modules:
  cpfw:
    walk:
      - fwAcceptPcktsIn
    auth:
      community: <....>

    lookups:
      - old_index: fwIfIndex
        new_index: fwIfName

snmp.yml:
cpfw:
  walk:
  - 1.3.6.1.4.1.2620.1.1.25.5.1.2
  - 1.3.6.1.4.1.2620.1.1.25.5.1.5
  metrics:
  - name: fwAcceptPcktsIn
    oid: 1.3.6.1.4.1.2620.1.1.25.5.1.5
    type: counter
    help: ' - 1.3.6.1.4.1.2620.1.1.25.5.1.5'
    indexes:
    - labelname: fwIfName
      type: counter
    lookups:
    - labels:
      - fwIfName
      labelname: fwIfName
      oid: 1.3.6.1.4.1.2620.1.1.25.5.1.2
      type: DisplayString
  auth:
    community: <...>


Debug log:

Jun 19 23:24:48 prometheus-02 docker-snmp-exporter[31437]: time="2018-06-19T23:24:48Z" level=debug msg="Scraping target '<target>' with module 'cpfw'" source="main.go:86"
Jun 19 23:24:48 prometheus-02 docker-snmp-exporter[31437]: time="2018-06-19T23:24:48Z" level=debug msg="Walking target \"<target>\" subtree \"1.3.6.1.4.1.2620.1.1.25.5.1.2\"" source="collector.go:124"
Jun 19 23:24:48 prometheus-02 docker-snmp-exporter[31437]: time="2018-06-19T23:24:48Z" level=debug msg="Walk of target \"<target>\" subtree \"1.3.6.1.4.1.2620.1.1.25.5.1.2\" completed in 3.135859ms" source="collector.go:134"
Jun 19 23:24:48 prometheus-02 docker-snmp-exporter[31437]: time="2018-06-19T23:24:48Z" level=debug msg="Walking target \"<target>\" subtree \"1.3.6.1.4.1.2620.1.1.25.5.1.5\"" source="collector.go:124"
Jun 19 23:24:48 prometheus-02 docker-snmp-exporter[31437]: time="2018-06-19T23:24:48Z" level=debug msg="Walk of target \"<target>\" subtree \"1.3.6.1.4.1.2620.1.1.25.5.1.5\" completed in 1.190988ms" source="collector.go:134"
Jun 19 23:24:48 prometheus-02 docker-snmp-exporter[31437]: time="2018-06-19T23:24:48Z" level=debug msg="Scrape of target '<target>' with module 'cpfw' took 0.009878 seconds" source="main.go:97"


Danny

On Tuesday, 19 June 2018 18:52:31 UTC-4, Brian Brazil  wrote:
On 19 June 2018 at 23:42, Danny Kulchinsky <dann...@gmail.com> wrote:
I understand, anything I can do to help?

Perhaps run the "bad" config with debugging enabled?

Can't hurt, though I doubt it'll provide any useful information. Maybe tcpdump to be sure the device isn't sending dodgy data when you're using lookups (this possibility should already be eliminated, but doesn't hurt to double check).

Do you have the raw oids from an snmpbulkwalk?

Brian
 

Danny




-- 









-- 




Danny Kulchinsky
unread,
Jun 19, 2018, 10:34:05 PM6/19/18
to Prometheus Users
snmpbulkwalk results:

fwIfIndex (1.3.6.1.4.1.2620.1.1.25.5.1.1):

$ snmpbulkwalk -v 2c -c <...> <target> 1.3.6.1.4.1.2620.1.1.25.5.1.1
iso.3.6.1.4.1.2620.1.1.25.5.1.1.2.0 = Counter32: 2
iso.3.6.1.4.1.2620.1.1.25.5.1.1.3.0 = Counter32: 3
iso.3.6.1.4.1.2620.1.1.25.5.1.1.4.0 = Counter32: 4
iso.3.6.1.4.1.2620.1.1.25.5.1.1.5.0 = Counter32: 5
iso.3.6.1.4.1.2620.1.1.25.5.1.1.6.0 = Counter32: 6
iso.3.6.1.4.1.2620.1.1.25.5.1.1.7.0 = Counter32: 7
iso.3.6.1.4.1.2620.1.1.25.5.1.1.8.0 = Counter32: 8
iso.3.6.1.4.1.2620.1.1.25.5.1.1.9.0 = Counter32: 9
iso.3.6.1.4.1.2620.1.1.25.5.1.1.10.0 = Counter32: 10
iso.3.6.1.4.1.2620.1.1.25.5.1.1.11.0 = Counter32: 11
iso.3.6.1.4.1.2620.1.1.25.5.1.1.12.0 = Counter32: 12
iso.3.6.1.4.1.2620.1.1.25.5.1.1.13.0 = Counter32: 13
iso.3.6.1.4.1.2620.1.1.25.5.1.1.14.0 = Counter32: 14
iso.3.6.1.4.1.2620.1.1.25.5.1.1.15.0 = Counter32: 15
iso.3.6.1.4.1.2620.1.1.25.5.1.1.16.0 = Counter32: 16
iso.3.6.1.4.1.2620.1.1.25.5.1.1.17.0 = Counter32: 17
iso.3.6.1.4.1.2620.1.1.25.5.1.1.18.0 = Counter32: 18
iso.3.6.1.4.1.2620.1.1.25.5.1.1.19.0 = Counter32: 19
iso.3.6.1.4.1.2620.1.1.25.5.1.1.20.0 = Counter32: 20
iso.3.6.1.4.1.2620.1.1.25.5.1.1.21.0 = Counter32: 21
iso.3.6.1.4.1.2620.1.1.25.5.1.1.22.0 = Counter32: 22
iso.3.6.1.4.1.2620.1.1.25.5.1.1.23.0 = Counter32: 23
iso.3.6.1.4.1.2620.1.1.25.5.1.1.24.0 = Counter32: 24
iso.3.6.1.4.1.2620.1.1.25.5.1.1.25.0 = Counter32: 25
iso.3.6.1.4.1.2620.1.1.25.5.1.1.26.0 = Counter32: 26
iso.3.6.1.4.1.2620.1.1.25.5.1.1.27.0 = Counter32: 27



fwIfName (1.3.6.1.4.1.2620.1.1.25.5.1.2):

$ snmpbulkwalk -v 2c -c <...> <target> 1.3.6.1.4.1.2620.1.1.25.5.1.2
iso.3.6.1.4.1.2620.1.1.25.5.1.2.2.0 = STRING: "eth2-01"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.3.0 = STRING: "eth2-02"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.4.0 = STRING: "eth2-03"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.5.0 = STRING: "eth2-04"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.6.0 = STRING: "eth1"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.7.0 = STRING: "Sync"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.8.0 = STRING: "eth1.241"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.9.0 = STRING: "eth2-02.210"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.10.0 = STRING: "eth1.266"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.11.0 = STRING: "eth1.265"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.12.0 = STRING: "eth2-02.641"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.13.0 = STRING: "eth2-02.273"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.14.0 = STRING: "eth2-02.290"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.15.0 = STRING: "eth1.246"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.16.0 = STRING: "eth2-02.288"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.17.0 = STRING: "eth2-02.268"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.18.0 = STRING: "eth2-02.610"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.19.0 = STRING: "eth1.294"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.20.0 = STRING: "eth2-02.631"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.21.0 = STRING: "eth2-02.621"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.22.0 = STRING: "eth1.244"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.23.0 = STRING: "eth2-02.269"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.24.0 = STRING: "eth1.292"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.25.0 = STRING: "eth1.291"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.26.0 = STRING: "eth1.240"
iso.3.6.1.4.1.2620.1.1.25.5.1.2.27.0 = STRING: "eth1.242"


fwAcceptPcktsIn (1.3.6.1.4.1.2620.1.1.25.5.1.5):

$ snmpbulkwalk -v 2c -c <...> <target> 1.3.6.1.4.1.2620.1.1.25.5.1.5
iso.3.6.1.4.1.2620.1.1.25.5.1.5.2.0 = Counter32: 30371
iso.3.6.1.4.1.2620.1.1.25.5.1.5.3.0 = Counter32: 0
iso.3.6.1.4.1.2620.1.1.25.5.1.5.4.0 = Counter32: 987413
iso.3.6.1.4.1.2620.1.1.25.5.1.5.5.0 = Counter32: 633837
iso.3.6.1.4.1.2620.1.1.25.5.1.5.6.0 = Counter32: 0
iso.3.6.1.4.1.2620.1.1.25.5.1.5.7.0 = Counter32: 1
iso.3.6.1.4.1.2620.1.1.25.5.1.5.8.0 = Counter32: 45015
iso.3.6.1.4.1.2620.1.1.25.5.1.5.9.0 = Counter32: 8368
iso.3.6.1.4.1.2620.1.1.25.5.1.5.10.0 = Counter32: 0
iso.3.6.1.4.1.2620.1.1.25.5.1.5.11.0 = Counter32: 0
iso.3.6.1.4.1.2620.1.1.25.5.1.5.12.0 = Counter32: 7747
iso.3.6.1.4.1.2620.1.1.25.5.1.5.13.0 = Counter32: 100813
iso.3.6.1.4.1.2620.1.1.25.5.1.5.14.0 = Counter32: 151004
iso.3.6.1.4.1.2620.1.1.25.5.1.5.15.0 = Counter32: 12401
iso.3.6.1.4.1.2620.1.1.25.5.1.5.16.0 = Counter32: 108640
iso.3.6.1.4.1.2620.1.1.25.5.1.5.17.0 = Counter32: 120444
iso.3.6.1.4.1.2620.1.1.25.5.1.5.18.0 = Counter32: 74222
iso.3.6.1.4.1.2620.1.1.25.5.1.5.19.0 = Counter32: 21458
iso.3.6.1.4.1.2620.1.1.25.5.1.5.20.0 = Counter32: 62670
iso.3.6.1.4.1.2620.1.1.25.5.1.5.21.0 = Counter32: 9568
iso.3.6.1.4.1.2620.1.1.25.5.1.5.22.0 = Counter32: 2958
iso.3.6.1.4.1.2620.1.1.25.5.1.5.23.0 = Counter32: 336
iso.3.6.1.4.1.2620.1.1.25.5.1.5.24.0 = Counter32: 11240
iso.3.6.1.4.1.2620.1.1.25.5.1.5.25.0 = Counter32: 181703
iso.3.6.1.4.1.2620.1.1.25.5.1.5.26.0 = Counter32: 5842
iso.3.6.1.4.1.2620.1.1.25.5.1.5.27.0 = Counter32: 0



Danny
Brian Brazil
unread,
Jun 20, 2018, 3:35:02 AM6/20/18
to Danny Kulchinsky, Prometheus Users
On 20 June 2018 at 03:34, Danny Kulchinsky <dann...@gmail.com> wrote:
snmpbulkwalk results:

fwIfIndex (1.3.6.1.4.1.2620.1.1.25.5.1.1):

$ snmpbulkwalk -v 2c -c <...> <target> 1.3.6.1.4.1.2620.1.1.25.5.1.1
iso.3.6.1.4.1.2620.1.1.25.5.1.1.2.0 = Counter32: 2


So it is a device bug, that .0 shouldn't be there - that's only for objects outside of a table.

What you can do is add a dummy integer index, and use multi index lookup (which is only supported in snmp.yml at the moment).

Brian

 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.

To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6c844cce-54a4-4e4e-a9f9-0232d23fe471%40googlegroups.com.


For more options, visit https://groups.google.com/d/optout.





-- 




Danny Kulchinsky
unread,
Jun 20, 2018, 3:36:44 AM6/20/18
to Brian Brazil, Prometheus Users
Thanks Brian!

Could you point me to an example how to configure it?

Danny
Ben Kochie
unread,
Jun 20, 2018, 4:36:28 AM6/20/18
to Danny Kulchinsky, Brian Brazil, Prometheus Users
The generator won't handle this, try this snmp.yml:
cpfw:
  walk:
  - 1.3.6.1.4.1.2620.1.1.25.5.1.1
  - 1.3.6.1.4.1.2620.1.1.25.5.1.2
  - 1.3.6.1.4.1.2620.1.1.25.5.1.5
  - name: fwAcceptPcktsIn
    oid: 1.3.6.1.4.1.2620.1.1.25.5.1.5
    type: gauge
    help: ' - 1.3.6.1.4.1.2620.1.1.25.5.1.5'
    indexes:
    - labelname: fwIfIndex
      type: gauge
    - labelname: fwIfDummyIndex
      type: gauge
    lookups:
    - labels:
      - fwIfIndex
      - fwIfDummyIndex

      labelname: fwIfName
      oid: 1.3.6.1.4.1.2620.1.1.25.5.1.2
      type: DisplayString




Danny Kulchinsky
unread,
Jun 20, 2018, 10:01:13 AM6/20/18
to Ben Kochie, Brian Brazil, Prometheus Users
Thanks for the example Ben!

That works really well, going to apply this now.

I see there's a plan to support this in generator (https://github.com/prometheus/snmp_exporter/issues/227)? would be great!


Thanks again guys for the support :)

Danny
Reply all
Reply to author
Forward
0 new messages