Edda not collecting data from all VPC

[Hridyesh Pant]

Hi,
I have multiple VPC per account, when i configure edda , it is only collecting information based on default VPC for that account. i could not see any VPC related setting in edda properties file.

Does Edda scan all VPC for a account?
Am i missing somethings?

--Hridyesh

[Hridyesh Pant]

Also Edda is not storing all data but storing some part of data.

aws.stacks is only showing 100 recodes,but have more then 100 stacks
db.aws.stacks.count()
100

loadBalancers is also not showing all ELBs, there are more then 2000 ELB in account but edda is storing only few.
db.aws.loadBalancers.count()
402

is there any setting where we can store all historical data?

[iain wright]

RE: VPC

Do you still have only data from your default VPC? I didn't do anything special to collect automatically from several VPCs

RE: Not storing all data

Any errors in your logs? Maybe your aws api calls are getting throttled? Are the values not changing over time at all?

RE: Storing of all data

Looks like edda doesn't pull _historical_ data (IE: resources that are no longer live) from AWS (ala cloudtrail), but anything that is live and returned from the aws api, can be stored forever or purged based on a TTL:

# There are three supported purge policies for record retention:

#   NONE           => Keep Forever (Default)

#   LIVE           => Only keep current records (mimics the AWS API)

#   AGE;expiry=N   => Remove records after expiry in milliseconds (N)

#

# edda.collection.purgeFrequency=86400000            # 24 hours in ms

# edda.collection.purgePolicy=AGE;expiry=2678400000  # 31 days in ms

--

Iain Wright

This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.

--
You received this message because you are subscribed to the Google Groups "Edda Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to edda-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hridyesh Pant]

Thanks Iain.

Yes i am getting below exception for most of the crawling call, i enable debug log4j.logger.com.amazonaws.request=DEBUG but not getting any more info.

Any idea how to fix below exception?

--Hridyesh

017-02-24 18:25:35.764 - ERROR - [StateMachine.scala:204] [9f3c21fc-82a2-46b2-9afd-1fa38bbfdeb7 refresh]  failed to handle event Crawl([Collection us-west-2.aws.loadBalancers] refresher)

com.amazonaws.SdkClientException: Unable to unmarshall response (ParseError at [row,col]:[13688,10]

Message: Connection reset). Response Code: 200, Response Text: OK

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1468)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)

at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)

at com.amazonaws.services.elasticloadbalancing.AmazonElasticLoadBalancingClient.doInvoke(AmazonElasticLoadBalancingClient.java:1879)

at com.amazonaws.services.elasticloadbalancing.AmazonElasticLoadBalancingClient.invoke(AmazonElasticLoadBalancingClient.java:1855)

at com.amazonaws.services.elasticloadbalancing.AmazonElasticLoadBalancingClient.describeLoadBalancers(AmazonElasticLoadBalancingClient.java:1236)

at com.netflix.edda.aws.AwsLoadBalancerCrawler$$anon$6$$anonfun$18.apply(AwsCrawlers.scala:344)

at com.netflix.edda.aws.AwsLoadBalancerCrawler$$anon$6$$anonfun$18.apply(AwsCrawlers.scala:344)

at com.netflix.edda.Crawler.backoffRequest(Crawler.scala:209)

at com.netflix.edda.aws.AwsLoadBalancerCrawler$$anon$6.next(AwsCrawlers.scala:344)

at com.netflix.edda.aws.AwsLoadBalancerCrawler$$anon$6.next(AwsCrawlers.scala:342)

at scala.collection.Iterator$class.foreach(Iterator.scala:893)

at com.netflix.edda.aws.AwsIterator.foreach(AwsCrawlers.scala:131)

at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)

at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:183)

at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45)

at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)

at com.netflix.edda.aws.AwsIterator.to(AwsCrawlers.scala:131)

at scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:294)

at com.netflix.edda.aws.AwsIterator.toList(AwsCrawlers.scala:131)

at com.netflix.edda.aws.AwsLoadBalancerCrawler.doCrawl(AwsCrawlers.scala:353)

at com.netflix.edda.aws.AwsLoadBalancerCrawler.doCrawl(AwsCrawlers.scala:336)

at com.netflix.edda.Crawler$$anonfun$localTransitions$1.applyOrElse(Crawler.scala:132)

at com.netflix.edda.Crawler$$anonfun$localTransitions$1.applyOrElse(Crawler.scala:119)

at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)

at com.netflix.edda.StateMachine$$anonfun$act$1$$anonfun$applyOrElse$2$$anonfun$apply$1.applyOrElse(StateMachine.scala:201)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at scala.actors.ReactorTask.run(ReactorTask.scala:31)

at scala.actors.Reactor$class.resumeReceiver(Reactor.scala:130)

at com.netflix.edda.StateMachine.scala$actors$InternalReplyReactor$$super$resumeReceiver(StateMachine.scala:100)

at scala.actors.InternalReplyReactor$class.resumeReceiver(InternalReplyReactor.scala:60)

at com.netflix.edda.StateMachine.resumeReceiver(StateMachine.scala:100)

at scala.actors.InternalActor$class.searchMailbox(InternalActor.scala:76)

at com.netflix.edda.StateMachine.searchMailbox(StateMachine.scala:100)

at scala.actors.Reactor$$anonfun$startSearch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Reactor.scala:118)

at scala.actors.Reactor$$anonfun$startSearch$1$$anonfun$apply$mcV$sp$1.apply(Reactor.scala:115)

at scala.actors.Reactor$$anonfun$startSearch$1$$anonfun$apply$mcV$sp$1.apply(Reactor.scala:115)

at scala.actors.ReactorTask.run(ReactorTask.scala:33)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[13688,10]

Message: Connection reset

at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:596)

at com.sun.xml.internal.stream.XMLEventReaderImpl.peek(XMLEventReaderImpl.java:276)

at com.amazonaws.transform.StaxUnmarshallerContext.nextEvent(StaxUnmarshallerContext.java:220)

at com.amazonaws.services.elasticloadbalancing.model.transform.LoadBalancerDescriptionStaxUnmarshaller.unmarshall(LoadBalancerDescriptionStaxUnmarshaller.java:42)

at com.amazonaws.services.elasticloadbalancing.model.transform.DescribeLoadBalancersResultStaxUnmarshaller.unmarshall(DescribeLoadBalancersResultStaxUnmarshaller.java:49)

at com.amazonaws.services.elasticloadbalancing.model.transform.DescribeLoadBalancersResultStaxUnmarshaller.unmarshall(DescribeLoadBalancersResultStaxUnmarshaller.java:31)

at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:101)

at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:43)

at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444)

... 46 more

To unsubscribe from this group and stop receiving emails from it, send an email to edda-users+...@googlegroups.com.