Data Collector Stopped Collecting?
bmelendy
I've just downloaded and installed FA yesterday. I followed the installation video located here:
http://www.fusion-analytics.com/fa/screencast/fa1/FA1only.html
I can log into the FA application on port 8400 and see data up until yesterday afternoon when I did my initial push of logs from the FRAM (as directed in the video). After that, there is no new data. I configured the collector to send data every 60 minutes (as per the video) but nothing more is showing up.
In the FA Upload Application under Application Summary, my application (named faapp which I copied from the setup video) says it is running with 0 paused, 12 started and 0 processing jobs.
I have gone back into the FRAM and verified that the collector targets are OK. Like the video it is configured to sent to target from FRAM.
It is as if the initial transfer of logs from the FRAM went through, but nothing after that has been sent? Thanks for any help, I'd really like to get this working and see what FA can do for me.
...Brad
Darren Pywell
Can you go into the instance of FusionReactor that you want to send
data from and check the targets look ok. Also verify the targets in
that instance FR verify ok? If so you can kick off log rotation by
pressing on the Rotate Now button on the FusionReactor->Analytics-
>Connector Status. You can see the rotation status on that page and if
the data gets sent you should see No Exception in the FusionAnalytics
Connector Status column.
You should see the files appear soon afterwards in FA. If they don't
then it would seem like the files are not being processed at the FA
side. In that case, please check to see if you have a very large
number of files pending in the Import folder of FusionAnalytics. If
so, move the files somewhere else and then restart FA. Once restarted
you can copy some files back to the import folder. We have seen a case
where a very large number of files pending import can cause the import
to stall. We have a fix coming for that in 1.0.2 soon.
If that's not the issue, can you take a look into the FusionAnalytics
log folder and look for any error reported in the logs. If you could
post any errors, we can take a look.
Kind Regards,
Darren
bmelendy
I found lots of errors in the fusionanalytics-datacollector-errors.log file. I see lots of entries for truncated db entries where the coldfusion error exceeded the column length. Also, there are errors:
"Could not allocate space for object 'dbo.ANNOTATION'.'PK__ANNOTATI__3214EC2725869641' in database 'fadb' because the 'PRIMARY' filegroup is full.
"Could not allocate space for object 'dbo.LOGLINES'.'PK__LOGLINES__3214EC273B75D760' in database 'fadb' because the 'PRIMARY' filegroup is full.
This to me could be the cause, however, I have verified that there is plenty of disk space on the drive where the sql server express data files live and also that the database has autogrowth set to "by 10 percent, unrestricted growth" as detailed in the installation video.
When I go to the Filegroup settings for the database, there are no settings regarding size or space so I'm not sure what more can be done. When I view the settings for the database, the GENERAL section is grey and cannot be changed and the "space available" is only .08 MB?
After going back over all my steps the only thing I think I have done wrong is that I don't think I have SQL Server Express 2008 R2. I believe I have the 64bit NON R2 version. Could this be the problem? Thanks again for your help.
...Brad
Darren Pywell
It looks like the database size limit has been reached. Microsoft
increased the size in Express R2 to 10GB up from 4GB, so I think you
are right. The R2 Express edition limit of 10GB may be too small
depending on how much data you want to keep. In that case you will
have to use data volume management in the FADC application.
In FusionReactor you can decide which logs should be sent to
FusionAnalytics. If you have a lot of JDBC log entries but don't want
to analyze query performance then you can set FusionReactor to only
archive the logs instead or transferring and archiving them. This is
done by going to Plugins->Active Plugins->FusionReactor Log Rotation
Plugin Configuration. Here you can set which logs get sent.
Alternatively you can set the JDBC logs to only log certain JDBC
requests on the JDBC->JDBC Settings. This can cut down the amount of
data being sent to FA. I do think this data is some of the most
interesting data that you can analyze but I'm just sharing that as an
example of how to cut down the data volume through other ways.
Hope that helps,
Darren
bmelendy
how much space do you think I would need to keep about a weeks worth of data in FA from three FR monitored coldfusion servers?
David Stockton
You should check out our data guidelines page - http://www.fusion-analytics.com/fa/dataguidelines.cfm
Best regards,
David Stockton
Fusion Team
bmelendy
Nick Gleason
We recently upgraded FusionReactor from 4.0.1 to 4.0.8 and got FusionAnalytics configured and working on our database server (with help from David Stockton). For the most part FA is great, but since about the time of the install, the memory on the web server is running very high and sending us alerts every minute or two. Looking at task manager in the server, JRUN is the main user of memory with some used by FRAM.
I am guessing this is somehow related to the installation / configuration of FusionAnalytics, but the one strange thing for that theory is that FA is on our db server and the memory problem is happening on the web server (which is not running FA). I know that there is a connection between FusionReactor on the web server and FusionAnalytics on the db server for the purpose of collecting data, so I am guessing that something about that process is spiking memory on the web server. Even over the weekend when we had little traffic on the web server, memory usage was spiking to 80-90%.
Also, the web server is 64bit running IIS 7.5 with 4 GB of RAM. So, it is fairly robust and has never had any memory issues until now.
As a test, I changed the FusionReactor setting “FusionAnalytics Connector Mode” to Disabled just now and this may be having a positive effect on the memory situation on the server.
So, I guess my question is whether there is something about FA and how it would interact with the web server and FusionReactor that would drive up memory usage. And, what would be the best way to address this.
Any ideas are welcome. We’re used FusionReactor for years but we’re just starting with FusionAnalytics and don’t know a lot of details about it yet.
Thanks in advance for your thoughts!
Best,
Nick
Charlie Arehart
That really shouldn’t be necessary. You could just purge the DB, if nothing else. But the features in FA to manage the size really would seem the right (and simpler) answer. Or is there more to your decision than what’s been discussed here? Just curious.
/charlie
Charlie Arehart
While you await an answer from the engineers, I’ll just say that I’ve not seen this problem myself, having worked with a few FA customers.
If I had to guess, I’d wonder how the configuration is set for both the logging management (in the FR instance for CF) and for the FA communication in the FRAM instance.
It may help to note that the default behavior is that your FR instance (for CF) creates logs, the FRAM instance manages those logs (archiving, etc.), and then (if configured to talk to an FA instance) it sends those archives down to the FA server.
So in that you say you have disabled the FA connection, that does suggest that the problem was seemingly in that sending of logs down to the FA server (though not necessarily, I suppose).
I would propose that you look at the logs for the FRAM instance, since that’s where you say the problems are happening. It could be simply that it (like any Java web app) may be having outofmemory errors and needs more RAM allocated to it. That said, I’ve not seen the FRAM instance need more than the defaults (but the FA instance is another matter, as it does a lot more work, both importing the logs sent to it from FRAM and then serving up responses to FA clients).
I’ll propose one other possibility: you say that the “web server, memory usage was spiking to 80-90%”. So that’s across the whole box, which is running CF (and its FR instance within it) and the FRAM instance, right? Is there by any chance anything else on the box? For instance, some people run DBs or have lots of app pools (if on IIS), all of which could be stealing memory from the other instances. I appreciate that you feel that “it was fine before adding the communication from FRAM to FA”. Just suggesting you may want to look more closely (at task manager, if on Windows, for instance) to see what’s using memory on the box. Sometimes problems happen just because one box is being asked to do too much.
Again, all just a guess. The FR/FA engineers may spot something more specific for your particular problem.
/charlie
From: fusiona...@googlegroups.com [mailto:fusiona...@googlegroups.com] On Behalf Of Nick Gleason
Sent: Monday, March 26, 2012 7:25 PM
To: fusiona...@googlegroups.com
--
You received this message because you are subscribed to the Google Groups "FusionAnalytics" group.
To post to this group, send email to fusiona...@googlegroups.com.
To unsubscribe from this group, send email to fusionanalyti...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fusionanalytics?hl=en.
Nick Gleason
Hi Charlie,
Thanks for the thoughtful follow up.
It appears that one factor was that we had wrapped all the DBs in the FusionReactor db wrapper, which hadn’t been previously done. This was done in order to provide richer data to FA. But, it may have had a side impact of driving up the memory usage across the server.
Another factor turned out to be not related to FA, but possibly to FR. We saw that a bot (Google Image bot) was hitting a captcha image repeatedly on our sites and this was strangely leading to memory spikes. We eventually blocked that which seemed to have a positive effect. At the same time, the folks at Integral said they had seen some JRun memory spiking in the latest FR release (1.0.8, I think) and gave us a patch that would presumably be included in the next FR release. We haven’t tried that yet but hopefully it will help when we re-implement FA soon (but more slowly this time).
Also, we are running IIS, but I don’t know a lot about how the app pools from IIS relate to memory allocation. I can do some research on that.
Thanks again!
Nick
Charlie Arehart
Thanks for the update, Nick, and glad to help.
So about the JDBC wrapper, I’d note that if you have lots of requests that have lots of queries, and especially if perhaps very long SQL statements, then I could see an impact from that. In that regard, note that there is an option in the JDBC Settings where you can tell it to track (in the interface) only those queries that exceed more than x milliseconds. (There is a separate setting there also for controlling the logging of requests taking longer than y milliseconds, but that would not affect memory).
Of course, the more that you can see in the FR interface, the more that FR is tracking in the address space (and heap) of the CF Address space it’s monitoring. So if you can tell it to track fewer things, that can have a beneficial impact. (Some think that FR runs “out of process”. It does not. Well, now in FR 4 the new FRAM instance DOES run out of process. But when you open the FR interface on, say, a CF instance, you are seeing info being sent from within that instance, and FR is storing the data that it stores for display in that that instance’s memory address space/java heap.
As for 4.0.8 having an issue, I thought instead it was that 4.0.8 FIXED such an issue. Let’s wait and see if the FR guys clarify either way. Would be good to know if there’s an issue in 4.0.8.
/charlie
From: fusiona...@googlegroups.com [mailto:fusiona...@googlegroups.com] On Behalf Of Nick Gleason
Sent: Wednesday, March 28, 2012 12:42 PM
To: fusiona...@googlegroups.com
Subject: RE: [fusionanalytics] Memory spiking on web server after FA installation
Hi Charlie,
Darren Pywell
Although the JDBC wrapper will need some memory I wouldn't expect it
to use anything like the amount that is being talked about.
FusionReactor's request history is by default 100 requests with each
request recording the slowest 30 JDBC queries (if it runs that many
queries or more). This gives a total of 3000 stored queries; we record
metrics and the query text from the query which typically uses around
1-3Kbytes of data. In total that would be a total varying between
3-9MB's of memory which fits with the results of FR memory profiling
that we've done. In the JDBC settings you can alter the number of
queries recorded and that will affect the amount of memory needed
(approximately) according to the amounts I've outlined. Nick, have you
found that removing the JDBC drivers has made any noticeable impact on
the memory?
The FA connector does very little memory work in its default mode
which is to send via FRAM. In this mode the instance simply rotates
the logs and pings FRAM to say "go send those". FRAM does a little
more work in then sending those to FA but you shouldn't see any memory
usage in the instance for that. If you change the FusionAnalytics
connector mode in the instance to sent by this instance then the
instance does do more work and will need some memory, but again we've
worked very hard to keep that memory footprint down. One thing to note
is that if the FusionAnalytics connector mode is "Sent via FRAM" and
the FRAM server is down then the instance will automatically fallback
to sent via this instance, so it's good idea to have FRAM up if you
are using FusionAnalytics. Nick, is the memory issue resolved with the
FusionAnalytics Connector Mode disabled?
We are currently working on FR4.0.9 which contains a fix for an issue
that we've discovered in the Adobe JRun Connector while working
closely together with a customer. Under certain circumstances the
connector can be made to consume bytes from the next request and this
will cause corrupted requests and serious memory leaks to occur. We'll
be registering the bug with Adobe, but just to be clear this is not a
bug in FusionReactor but something that we are doing to lower the
chances of this happening. We sent FR4.0.9 to Nick on the off chance
that it might be the issue he is running into. Nick, can you try
FusionReactor 4.0.9 and see if it helps?
Thanks,
Darren
bmelendy
Does anyone have any suggestions? Unlike last time where I had errors in my FA Logs indicating that space in the database had been maxed out, I have no such errors this time around, just a blank AIR application (no data).
On Monday, March 26, 2012 12:30:20 PM UTC-7, David Stockton wrote:
Charlie Arehart
Brian, as for your second issue, of “only one of your datacollectors displayed” in the AIR app, note that they are not listed automatically. You have to add the link yourself. (If you may wonder why, it would help to keep in mind that the AIR app could be installed anywhere, including not on the FA server, so it really has no way to detect a) what servers you want to monitor and b) what collectors on those servers you want to monitor.)
But back to the bigger problem, when you say the collector “just stopped”, what do you mean? Are you saying you DID add the collector to the FA Air App and it opens to some date/time but shows no data? And are you saying that the Data Finder (under the first Deep Server Analysis link) also shows nothing, if you zoom all the way out in time (week, month, quarter, etc.)?
You also mention seeing no errors in the logs. Which logs are you looking in, specifically?
Finally, it could be that either the data collector or the dataservices applications are stopped. Is that, perhaps, what you mean when you say they “stopped”? I can’t tell for sure. If you log into the browser apps for FADS and FADC, and view the apps, are they all running?
Hope that helps, while you may await other word from Intergral folks.
/charlie
From: fusiona...@googlegroups.com [mailto:fusiona...@googlegroups.com] On Behalf Of bmelendy
Sent: Tuesday, April 03, 2012 1:21 PM
To: fusiona...@googlegroups.com
Subject: [fusionanalytics] Re: Data Collector Stopped Collecting?
So I have started over with a fresh installation of FA and a new SQL Server running on dedicated server with 50GB allocated for the initial DB size and then set auto growth to unlimited and 10% as per the installation video. I have the exact same problem where FA collected and presented data to me for almost one day, then stopped. Additionally, I have added a second data collector, and can see that it is reporting that logs are being sent to it, however, when I go to view the FA Summary in the AIR Application, only one of my data collectors are displayed?
Does anyone have any suggestions? Unlike last time where I had errors in my FA Logs indicating that space in the database had been maxed out, I have no such errors this time around, just a blank AIR application (no data).
On Monday, March 26, 2012 12:30:20 PM UTC-7, David Stockton wrote:
bmelendy
First my name is Brad, no Brian, but I think Brian might have been the name of the other user who posted their problem on this same thread.
As for the data collectors displayed in the AIR application, I'm speaking of the WEB page you call that launches what I believe to be the AIR application. The WEB page is running on port 8400 of the FA server. I access it with a URL http://mxtest9:8400/fads/faapp/. I have done this installation twice, the first time, I had three data collectors and they all showed up automatically in the AIR application. I had some problems and decided to start from scratch with a new SQL Server (the express server would fill up after 18 hours).
So now I have two problems:
1. If I add a new data collector, I can't see it in the AIR application (web page on port 8400, I don't know what the name of this is) and I can't see a way to add it. Last time all my data collectors were just there automagically.
2. There is a web application where you manage your data collectors. It displays a section called "Current Applications". My application is named "faapp" taken from the install video on the FA website. It says it is revision 1.0.6 and that it is running. It appears to be working but when I go into the AIR application, the only data I can see is from three days ago when I first turned things on with the new database. After April 2nd at 4pm there is no more data? In the FRAM under Connector Status, everything looks to have transferred properly.
As for the log files, I'm looking at the log files located on the FA server in the \FusionAnalytics\Server\logs\ directory. Before I was seeing clear errors regarding the max size limit being reached on the database (when I was using the express edition of the database) but now I am not seeing anything that looks like an error. The only errors I am seeing are from FA truncating log entries that exceed the maximum column width of the SQL Server database.
I have a voice message from the FA folks and they are going to arrange to call me and try to get this ironed out.
...Brad
Charlie Arehart
Thanks, Brad, and sorry for that. To be clear, in the note you wrote, it showed you replying to David Stockton, and his note was replying to a Brian. Since you had no name as a signature, and a “b” to start your email address, I (mis)connected the dots.
Moving on to your real challenges, in case it may help while you await their call, I have some thoughts. Let’s start with your last point, as that may be the quickest solution to your problem.
You say, “In the FRAM under Connector Status, everything looks to have transferred properly.” Well, are you saying that there are logs there for every hour, for today for instance, and the next to last column says “no exception”? And as for other days, note that there’s a widget at the top that lets you go back and forth to different days. If they’re not being sent (or received), that would explain why you see no data.
As for why they may fail to send, it’s possible that what you configured in the FR instance for CF (in its Analytics>FusionAnalytics Targets page) is no longer correct as of your reinstall. It has to name the correct URL for your data collector in the FA app. Note that there’s a status column there and a button to test the target. If that fails, that would explain why it’s failing. Maybe the domain name or port is wrong, or the datacollector.
Here’s another tip: on the server that’s doing the sending, try connecting to whatever link is shown there. If it works, it should return the datacollector name. If it does not, you may get some other error. For instance, if it hangs up for a while, then you may have a firewall problem (where either the FA server is not letting the CF/FRAM server connect to it, or the CF/FRAM server may not be letting you call out to the FA server.) I assume they are on different boxed form each other. Are they on different networks? That’s ok. It just raised the odds of firewall problems.
If all this is right, then I’d question again whether the data collector logs would show a problem. Or it could be that the datacollector has stopped. Go into the FADC app (in your case, http://mxtest9:8400/fadc/), and see if perhaps the datacollector shows “running”. If not, try clicking “start” there.
I have still more thoughts to share, on some of the other questions you ask and how you are describing things (referring to “the AIR app” when it seems you really mean the web interface to the app). I hope that info may help you or help others (in helping you, or if they may have similar confusion). But I know some hate long notes, so I’ll offer that in a second reply. Really the above is most important for your main problem.
Hope it helps. If not, I’m sure they’ll sort you out.
/charlie
From: fusiona...@googlegroups.com [mailto:fusiona...@googlegroups.com] On Behalf Of bmelendy
Sent: Tuesday, April 03, 2012 5:42 PM
To: fusiona...@googlegroups.com
Subject: Re: [fusionanalytics] Re: Data Collector Stopped Collecting?
Charlie Arehart
Brad, following up my last note, I wanted to offer some clarifications, because I sense you’re confusing some terms.
First, the AIR app and the web apps are different (though they share some things, which can add to confusion). I sense that what you’re calling below the “AIR app” is what you mean by “the interface you see to query the data via the graphs and timelines, etc.”. But I don’t think you’re running the “real” AIR app”, but rather you’re launching the web version via the FADS web app, because later you say, “I'm speaking of the WEB page you call that launches what I believe to be the AIR application” and then still later you say, “I can't see it in the AIR application (web page on port 8400, I don't know what the name of this is)”.
So to be clear, you are select an application in the FADS web app, then clicking the “Launch” button. That opens what looks exactly like what you would see in the AIR app, but to be clear it’s running within your browser, so it’s not really an AIR app. Actually, it’s a Flex page, and Flex apps can run either in a browser or in within Adobe AIR, which is a desktop product.
Fair enough. It can be confusing. But now I hope you can see that you don’t want to call what you launch from within the browser “the air app”. It’s the FA interface, viewed within a browser (maybe the Intergral guys can indicate if there’s a better generic term for that interface, that applies whether viewing it in AIR or as a web page).
Moving on, second, you then say “I had three data collectors and they all showed up automatically in the AIR application”. I’ll assume now you’re really talking about the web page, but then now you’ve also changed to talking about data collectors, when I think you still really mean FADS apps. Again, it can be easy to confuse the two when getting started.
The data collectors are responsible for loading the FR logs into the database, and they are accessed/managed in the dedicated FADC app (separate from the FADS, or data services, app), generally accessed via the FADC web app, in your case, http://mxtest9:8400/fadc/. (Technically, you can also access the FADC and FADS admin interfaces within the AIR app, but let’s not confuse matters further.)
The FADS app, on the other hand, is what connects you to a Data Collector to query the DB. If you launch a FADS app, like I discuss above, that’s what displays an interface within the browser which shows how to query your data (and which also can be launched manually in the AIR app).
So you’re saying you reinstalled, but you somehow “had three data collectors and they all showed up automatically in the AIR application”. Again, when you say the AIR app, I’ll assume you really mean the web app (for FADC). Even so, I can’t see that how data collectors would show up automatically. The Intergral guys can correct me if I’m wrong. You generally need to add them, by hand. And even if you meant the FADS app, that too requires that you create a new FADS app, to point to the data collector.
I’m not being pedantic about all this. I just want to make sure we’re all on the same page. Trust me: I appreciate how confusing it can be. There are lots of moving parts. :-) Once it’s all working and you’ve used it successfully for a few days, it will all click and make sense, I think. But until then, I do want to help if I can.
I was about to point out the quick start guide (http://docs.intergral.com/display/FA10/Quick+Start+Guide), linked to off the support page, http://www.fusion-analytics.com/fa/support.cfm, if it may help. I was also going to point out the quickstart video (http://www.fusion-analytics.com/fa/screencast/fa1/FA1only.html, also linked to as “training videos” on the left of that support page), but you say you’ve viewed that. For those who have not, in just a few minutes, it can help clear up a lot of confusion, which again is natural with such a powerful product.
So for your question 1, are you really talking about a data collector (FADC), or do you mean a FADS app? A FADS app does point to datacollector, so it’s a subtle difference. It may help to clarify, when you say in that question “I can't see it in the AIR application (web page on port 8400)”, which do you mean, specifically: http://mxtest9:8400/fads/index.jsp or http://mxtest9:8400/fadc/?
As for your question 2, that’s indeed the FADS app. Are you saying that’s where you didn’t see your 3 datacollectors? So you mean FADS app pointing to the datacollector. Again, still, they do not appear automagically that I can recall. But now you’re switching to the fact that you launch the app for that (again, not really the AIR app, but I hope that’s more clear now), and you don’t see data after a couple of days. As you note, that’s controlled mostly via the servers (FRAM or the instance creating the logs), and that’s why I started with the other note before this.
Hope some of this is helpful.
/charlie
PS Finally, uf you may still be reading this far :-), back to the matter of me referring to you as Brian (for the reason I explained), I’ll make two quick observations on that, if it may help you (or others) in the future: besides the value of ending an email with a signature (personal preference, of course), note also (or instead) that you could also edit your google groups account to indicate your name. I realize that that, too, may be something you prefer not to bother with. I’m just point out the possibility, if it would interest you.
PPS If anyone reading this is the sort of person who hates being taught, I’m sorry. It’s just my nature! And if you ever feel I’m being condescending, please know I’m only ever trying to help. I know that some people read me the wrong way. I never mean offense. I share all I do out of a sincere desire to help. I just don’t think most questions can be answered in a tweet, obviously, so I share what I think may be needed. It doesn’t suit everyone, I realize. I like to point this out once in a while when I offer a long note here. I don’t work for the Intergral folks, so don’t represent them. If you ever have feedback for me that you’d like to share privately, please contact me at cha...@carehart.org.
From: fusiona...@googlegroups.com [mailto:fusiona...@googlegroups.com] On Behalf Of Charlie Arehart
Sent: Tuesday, April 03, 2012 10:01 PM
To: fusiona...@googlegroups.com
--
David Stockton
Just FYI the issue was if you want to change/move the DB you need to
update the settings in both FADC and FADS.
Link to docs on configuring data sources in FADC:
http://docs.intergral.com/pages/viewpage.action?pageId=14320040#ApplicationConfigurationFADC-datasources
Link to docs on configuring data sources in FADS:
http://docs.intergral.com/pages/viewpage.action?pageId=13828146#ApplicationConfigurationFADS-DataSources
Best regards,
David Stockton
Fusion Team