I can log into the FA application on port 8400 and see data up until yesterday afternoon when I did my initial push of logs from the FRAM (as directed in the video). After that, there is no new data. I configured the collector to send data every 60 minutes (as per the video) but nothing more is showing up.
In the FA Upload Application under Application Summary, my application (named faapp which I copied from the setup video) says it is running with 0 paused, 12 started and 0 processing jobs.
I have gone back into the FRAM and verified that the collector targets are OK. Like the video it is configured to sent to target from FRAM.
It is as if the initial transfer of logs from the FRAM went through, but nothing after that has been sent? Thanks for any help, I'd really like to get this working and see what FA can do for me.
We recently upgraded FusionReactor from 4.0.1 to 4.0.8 and got FusionAnalytics configured and working on our database server (with help from David Stockton). For the most part FA is great, but since about the time of the install, the memory on the web server is running very high and sending us alerts every minute or two. Looking at task manager in the server, JRUN is the main user of memory with some used by FRAM.
I am guessing this is somehow related to the installation / configuration of FusionAnalytics, but the one strange thing for that theory is that FA is on our db server and the memory problem is happening on the web server (which is not running FA). I know that there is a connection between FusionReactor on the web server and FusionAnalytics on the db server for the purpose of collecting data, so I am guessing that something about that process is spiking memory on the web server. Even over the weekend when we had little traffic on the web server, memory usage was spiking to 80-90%.
Also, the web server is 64bit running IIS 7.5 with 4 GB of RAM. So, it is fairly robust and has never had any memory issues until now.
As a test, I changed the FusionReactor setting “FusionAnalytics Connector Mode” to Disabled just now and this may be having a positive effect on the memory situation on the server.
So, I guess my question is whether there is something about FA and how it would interact with the web server and FusionReactor that would drive up memory usage. And, what would be the best way to address this.
Any ideas are welcome. We’re used FusionReactor for years but we’re just starting with FusionAnalytics and don’t know a lot of details about it yet.
Thanks in advance for your thoughts!
That really shouldn’t be necessary. You could just purge the DB, if nothing else. But the features in FA to manage the size really would seem the right (and simpler) answer. Or is there more to your decision than what’s been discussed here? Just curious.
While you await an answer from the engineers, I’ll just say that I’ve not seen this problem myself, having worked with a few FA customers.
If I had to guess, I’d wonder how the configuration is set for both the logging management (in the FR instance for CF) and for the FA communication in the FRAM instance.
It may help to note that the default behavior is that your FR instance (for CF) creates logs, the FRAM instance manages those logs (archiving, etc.), and then (if configured to talk to an FA instance) it sends those archives down to the FA server.
So in that you say you have disabled the FA connection, that does suggest that the problem was seemingly in that sending of logs down to the FA server (though not necessarily, I suppose).
I would propose that you look at the logs for the FRAM instance, since that’s where you say the problems are happening. It could be simply that it (like any Java web app) may be having outofmemory errors and needs more RAM allocated to it. That said, I’ve not seen the FRAM instance need more than the defaults (but the FA instance is another matter, as it does a lot more work, both importing the logs sent to it from FRAM and then serving up responses to FA clients).
I’ll propose one other possibility: you say that the “web server, memory usage was spiking to 80-90%”. So that’s across the whole box, which is running CF (and its FR instance within it) and the FRAM instance, right? Is there by any chance anything else on the box? For instance, some people run DBs or have lots of app pools (if on IIS), all of which could be stealing memory from the other instances. I appreciate that you feel that “it was fine before adding the communication from FRAM to FA”. Just suggesting you may want to look more closely (at task manager, if on Windows, for instance) to see what’s using memory on the box. Sometimes problems happen just because one box is being asked to do too much.
Again, all just a guess. The FR/FA engineers may spot something more specific for your particular problem.
You received this message because you are subscribed to the Google Groups "FusionAnalytics" group.
To post to this group, send email to fusiona...@googlegroups.com.
To unsubscribe from this group, send email to fusionanalyti...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fusionanalytics?hl=en.
Thanks for the thoughtful follow up.
It appears that one factor was that we had wrapped all the DBs in the FusionReactor db wrapper, which hadn’t been previously done. This was done in order to provide richer data to FA. But, it may have had a side impact of driving up the memory usage across the server.
Another factor turned out to be not related to FA, but possibly to FR. We saw that a bot (Google Image bot) was hitting a captcha image repeatedly on our sites and this was strangely leading to memory spikes. We eventually blocked that which seemed to have a positive effect. At the same time, the folks at Integral said they had seen some JRun memory spiking in the latest FR release (1.0.8, I think) and gave us a patch that would presumably be included in the next FR release. We haven’t tried that yet but hopefully it will help when we re-implement FA soon (but more slowly this time).
Also, we are running IIS, but I don’t know a lot about how the app pools from IIS relate to memory allocation. I can do some research on that.
Thanks for the update, Nick, and glad to help.
So about the JDBC wrapper, I’d note that if you have lots of requests that have lots of queries, and especially if perhaps very long SQL statements, then I could see an impact from that. In that regard, note that there is an option in the JDBC Settings where you can tell it to track (in the interface) only those queries that exceed more than x milliseconds. (There is a separate setting there also for controlling the logging of requests taking longer than y milliseconds, but that would not affect memory).
Of course, the more that you can see in the FR interface, the more that FR is tracking in the address space (and heap) of the CF Address space it’s monitoring. So if you can tell it to track fewer things, that can have a beneficial impact. (Some think that FR runs “out of process”. It does not. Well, now in FR 4 the new FRAM instance DOES run out of process. But when you open the FR interface on, say, a CF instance, you are seeing info being sent from within that instance, and FR is storing the data that it stores for display in that that instance’s memory address space/java heap.
As for 4.0.8 having an issue, I thought instead it was that 4.0.8 FIXED such an issue. Let’s wait and see if the FR guys clarify either way. Would be good to know if there’s an issue in 4.0.8.
Brian, as for your second issue, of “only one of your datacollectors displayed” in the AIR app, note that they are not listed automatically. You have to add the link yourself. (If you may wonder why, it would help to keep in mind that the AIR app could be installed anywhere, including not on the FA server, so it really has no way to detect a) what servers you want to monitor and b) what collectors on those servers you want to monitor.)
But back to the bigger problem, when you say the collector “just stopped”, what do you mean? Are you saying you DID add the collector to the FA Air App and it opens to some date/time but shows no data? And are you saying that the Data Finder (under the first Deep Server Analysis link) also shows nothing, if you zoom all the way out in time (week, month, quarter, etc.)?
You also mention seeing no errors in the logs. Which logs are you looking in, specifically?
Finally, it could be that either the data collector or the dataservices applications are stopped. Is that, perhaps, what you mean when you say they “stopped”? I can’t tell for sure. If you log into the browser apps for FADS and FADC, and view the apps, are they all running?
Hope that helps, while you may await other word from Intergral folks.
Thanks, Brad, and sorry for that. To be clear, in the note you wrote, it showed you replying to David Stockton, and his note was replying to a Brian. Since you had no name as a signature, and a “b” to start your email address, I (mis)connected the dots.
Moving on to your real challenges, in case it may help while you await their call, I have some thoughts. Let’s start with your last point, as that may be the quickest solution to your problem.
You say, “In the FRAM under Connector Status, everything looks to have transferred properly.” Well, are you saying that there are logs there for every hour, for today for instance, and the next to last column says “no exception”? And as for other days, note that there’s a widget at the top that lets you go back and forth to different days. If they’re not being sent (or received), that would explain why you see no data.
As for why they may fail to send, it’s possible that what you configured in the FR instance for CF (in its Analytics>FusionAnalytics Targets page) is no longer correct as of your reinstall. It has to name the correct URL for your data collector in the FA app. Note that there’s a status column there and a button to test the target. If that fails, that would explain why it’s failing. Maybe the domain name or port is wrong, or the datacollector.
Here’s another tip: on the server that’s doing the sending, try connecting to whatever link is shown there. If it works, it should return the datacollector name. If it does not, you may get some other error. For instance, if it hangs up for a while, then you may have a firewall problem (where either the FA server is not letting the CF/FRAM server connect to it, or the CF/FRAM server may not be letting you call out to the FA server.) I assume they are on different boxed form each other. Are they on different networks? That’s ok. It just raised the odds of firewall problems.
If all this is right, then I’d question again whether the data collector logs would show a problem. Or it could be that the datacollector has stopped. Go into the FADC app (in your case, http://mxtest9:8400/fadc/), and see if perhaps the datacollector shows “running”. If not, try clicking “start” there.
I have still more thoughts to share, on some of the other questions you ask and how you are describing things (referring to “the AIR app” when it seems you really mean the web interface to the app). I hope that info may help you or help others (in helping you, or if they may have similar confusion). But I know some hate long notes, so I’ll offer that in a second reply. Really the above is most important for your main problem.
Hope it helps. If not, I’m sure they’ll sort you out.
Brad, following up my last note, I wanted to offer some clarifications, because I sense you’re confusing some terms.
First, the AIR app and the web apps are different (though they share some things, which can add to confusion). I sense that what you’re calling below the “AIR app” is what you mean by “the interface you see to query the data via the graphs and timelines, etc.”. But I don’t think you’re running the “real” AIR app”, but rather you’re launching the web version via the FADS web app, because later you say, “I'm speaking of the WEB page you call that launches what I believe to be the AIR application” and then still later you say, “I can't see it in the AIR application (web page on port 8400, I don't know what the name of this is)”.
So to be clear, you are select an application in the FADS web app, then clicking the “Launch” button. That opens what looks exactly like what you would see in the AIR app, but to be clear it’s running within your browser, so it’s not really an AIR app. Actually, it’s a Flex page, and Flex apps can run either in a browser or in within Adobe AIR, which is a desktop product.
Fair enough. It can be confusing. But now I hope you can see that you don’t want to call what you launch from within the browser “the air app”. It’s the FA interface, viewed within a browser (maybe the Intergral guys can indicate if there’s a better generic term for that interface, that applies whether viewing it in AIR or as a web page).
Moving on, second, you then say “I had three data collectors and they all showed up automatically in the AIR application”. I’ll assume now you’re really talking about the web page, but then now you’ve also changed to talking about data collectors, when I think you still really mean FADS apps. Again, it can be easy to confuse the two when getting started.
The data collectors are responsible for loading the FR logs into the database, and they are accessed/managed in the dedicated FADC app (separate from the FADS, or data services, app), generally accessed via the FADC web app, in your case, http://mxtest9:8400/fadc/. (Technically, you can also access the FADC and FADS admin interfaces within the AIR app, but let’s not confuse matters further.)
The FADS app, on the other hand, is what connects you to a Data Collector to query the DB. If you launch a FADS app, like I discuss above, that’s what displays an interface within the browser which shows how to query your data (and which also can be launched manually in the AIR app).
So you’re saying you reinstalled, but you somehow “had three data collectors and they all showed up automatically in the AIR application”. Again, when you say the AIR app, I’ll assume you really mean the web app (for FADC). Even so, I can’t see that how data collectors would show up automatically. The Intergral guys can correct me if I’m wrong. You generally need to add them, by hand. And even if you meant the FADS app, that too requires that you create a new FADS app, to point to the data collector.
I’m not being pedantic about all this. I just want to make sure we’re all on the same page. Trust me: I appreciate how confusing it can be. There are lots of moving parts. :-) Once it’s all working and you’ve used it successfully for a few days, it will all click and make sense, I think. But until then, I do want to help if I can.
I was about to point out the quick start guide (http://docs.intergral.com/display/FA10/Quick+Start+Guide), linked to off the support page, http://www.fusion-analytics.com/fa/support.cfm, if it may help. I was also going to point out the quickstart video (http://www.fusion-analytics.com/fa/screencast/fa1/FA1only.html, also linked to as “training videos” on the left of that support page), but you say you’ve viewed that. For those who have not, in just a few minutes, it can help clear up a lot of confusion, which again is natural with such a powerful product.
So for your question 1, are you really talking about a data collector (FADC), or do you mean a FADS app? A FADS app does point to datacollector, so it’s a subtle difference. It may help to clarify, when you say in that question “I can't see it in the AIR application (web page on port 8400)”, which do you mean, specifically: http://mxtest9:8400/fads/index.jsp or http://mxtest9:8400/fadc/?
As for your question 2, that’s indeed the FADS app. Are you saying that’s where you didn’t see your 3 datacollectors? So you mean FADS app pointing to the datacollector. Again, still, they do not appear automagically that I can recall. But now you’re switching to the fact that you launch the app for that (again, not really the AIR app, but I hope that’s more clear now), and you don’t see data after a couple of days. As you note, that’s controlled mostly via the servers (FRAM or the instance creating the logs), and that’s why I started with the other note before this.
Hope some of this is helpful.
PS Finally, uf you may still be reading this far :-), back to the matter of me referring to you as Brian (for the reason I explained), I’ll make two quick observations on that, if it may help you (or others) in the future: besides the value of ending an email with a signature (personal preference, of course), note also (or instead) that you could also edit your google groups account to indicate your name. I realize that that, too, may be something you prefer not to bother with. I’m just point out the possibility, if it would interest you.
PPS If anyone reading this is the sort of person who hates being taught, I’m sorry. It’s just my nature! And if you ever feel I’m being condescending, please know I’m only ever trying to help. I know that some people read me the wrong way. I never mean offense. I share all I do out of a sincere desire to help. I just don’t think most questions can be answered in a tweet, obviously, so I share what I think may be needed. It doesn’t suit everyone, I realize. I like to point this out once in a while when I offer a long note here. I don’t work for the Intergral folks, so don’t represent them. If you ever have feedback for me that you’d like to share privately, please contact me at cha...@carehart.org.