Idea - Memory analysis per scope suggestion.

[Dave]

Hey all,

I'm trying FA for the first time and I got excited because I saw the ColdFusionMonitor -> CFScope sizes report, then I realized that it's data from the ColdFusion Monitor which can't be enabled on a framework oriented environment and I came crashing back to earth.  :)

But it did get me thinking:  CFMemoryCounterAgent is really useful for sizing objects on the heap in CF.  Maybe this style of memory sizing could somehow be incorporated in FR/FA.  Not sure of the locking implications, but I'm thinking of a background thread that wakes up every X minutes and does takes some size measurements of application scope for each application.

Why is this useful for us?

We run a multi-tenanted CF SaaS application with each tenant in their own application scope. It would be handy to keep tabs on the sizes of these application scopes for server management.

Actually, it would be really handy to incorporate some EhCache metrics in FA too :)  

[Charlie Arehart]

You’re right that the memory size info is only available is “start memory tracking” is turned on in CF, and nearly everyone finds that their suffers when they do that, so sadly it’s a feature in FA that doesn’t tend to be of value to many. (Of course, there’s SO much more in FA that is useful, that I hope no one would hold it too much “against” FA. I realize you’re not saying that, Dave.)

But as for the Intergral folks adding some way to gather that info via FR (to pull into FA), I doubt it’s something they ever would/could do. You need to hook into CF too closely to get that, which has not been something FR have ever done. That said, the “FREC” feature (FR Extensions for CF) does get closer to that than any previous FR editions did. But all it does is poll for info that CF is already gathering. It doesn’t “cause CF to gather” new info, so until Adobe adds some way to better “count” memory, I don’t think we can ever expect to see such counting within FR or FA.

As for CFMemoryCounterAgent, I’m not familiar with that. Is it something you find within FA? Even so, given the above, I still don’t have much hope. :-)

Finally, as for ehcache entries, you make a good point there. CF10 did add new ehcache monitoring, and as such, it would be nice if FREC were tweaked to gather that to log in FR (and therefore to import and show within FA.) Hopefully they’ll respond here to say they will track that as a future improvement. :-)

/charlie

 

--
You received this message because you are subscribed to the Google Groups "FusionAnalytics" group.
To view this discussion on the web visit https://groups.google.com/d/msg/fusionanalytics/-/w3d-SfxEXKwJ.
To post to this group, send email to fusiona...@googlegroups.com.
To unsubscribe from this group, send email to fusionanalyti...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fusionanalytics?hl=en.

[Dave]

CFMemoryCounterAgent  is not a part of FA. It was written by the guy that runs the " http://bloginblack.de/ " blog (which does not appear to be up now). Here is a 3rd party reprint of that blog article ( http://watchingthewatchers.org/indepth/1356927/much-memory-does-my-coldfusion-variable )

It's not part of FA. It's installed as a JavaAgent (source and binary available in the net).  

It's very similar to how ehCache's SizeOf operators work for the byte sized caching in ehCache 2.5 upwards, in that you give it a ColdFusion variable and it tells you how much space it occupies on the heap, including size of all of the references that object holds.

For example, I can do a 

oObjectSize = CreateObject("java","nz.co.ventegocreative.coldfusion.memory.CFMemoryCounterAgent");

writeOutput("On-Heap Size of Application scope : #numberFormat(oObjectSize.deepSizeOf(application, false)/1024,"0,00")# kb");

and it tells me that my application scope is using 129mb of heap.

It's probably not perfect in sizing, but it gets pretty close.

dave

To post to this group, send email to fusionanalytics@googlegroups.com.
To unsubscribe from this group, send email to fusionanalytics+unsubscribe@googlegroups.com.

[David Stockton]

Hello David,

I like the idea but I'm not sure of the practicality of it all with respect to keeping a server performing well (the overhead of these checks) and checking memory size on changing objects plus various other considerations that would need to be made (e.g. making sure we don't reference something due for GC) etc.

It's certainly something we'd like to do more of but as I'm sure you can appreciate it's a complex subject and at the end of the day only gives "good estimates" not hard facts - another reason why we've shy'd away from it in the past.

I'll definitely bring your comments to the attention of the team though.

Memory/scope size tracking: FR-2826
EhCache stats tracking: FR-2827

Best regards,
David Stockton
Fusion Support Team

To view this discussion on the web visit https://groups.google.com/d/msg/fusionanalytics/-/Z-Bp51kjlSQJ.

To post to this group, send email to fusiona...@googlegroups.com.
To unsubscribe from this group, send email to fusionanalyti...@googlegroups.com.

[Dave]

Hi David,

I understand what you are saying about the technical difficulties of measuring object heap size.

I also understand that some of the things we want to capture will not have broad appeal to the rest of your user base.

That being said, FR offers a really nice metrics logging (rotation, archiving, etc) solution and FA offers a really nice ETL and reporting solution. From what I read, one of the goals with FA is to make is extensible (in the future?).

So maybe what would really help us is some generic metrics logging functionality in FR.  IE, how about as a developer, I can us a FR API to log whatever metrics I need, potentially via a CF Scheduled task.  

EG: 

var someFrLoggingObect = createObject("java", "someFrClass").init("EhcacheMetrics.log");

someFrLoggingObect.logMetrics([1,2,3,45,6,5])

If I had access to the FR internal requestID, I could even create a Mach-II filter to create a log of {FRrequestID,eventname,ourModuleName} for each request which goes part-way to solving our problem of logging the mach-ii event name for Form Posts.

These logs would form part of FR's normal log rotation system.  I can already use FA to define the extra files/data in the application configuration, and a groovy file to add any custom ETL logic.

The ability to customize/design my own FA reports would make this a really powerful custom metrics gathering system.

There are obviously lots of logging systems out there, but the traditional ETL and reporting process is very time consuming and painful to get right.

Anyway - just a thought.

Dave

  

f 

[David Stockton]

Hi David,

Firstly, I just want to re-iterate that we really value your input - it's fantastic to start seeing the FusionAnalytics community building up and we're starting to see more and more really positive, constructive feedback - **EVERY** comment and piece of feedback we receive from all our users is taken on board. The entire FR+FA team have a weekly meeting where we can discuss feedback / ideas and this community input is really important to us... so thanks!

"From what I read, one of the goals with FA is to make is extensible (in the future?)."

FA is actually currently built as a completely extensible analytics platform. Whilst we haven't got fully fleshed out documentation, every effort has been taken during development that FA is a platform for analytics. You may have noticed that when you install FusionAnalytics you're asked if you want to install the data collector (the thing that gathers data and puts it into a DB and quantizes/aggregates it) the data services (the thing that pushes data out to the clients) and "Analytics for FusionReactor" - this is just an analytics "application". From a marketing perspective we of course (at least initially) need to market the product as a single unit. But in reality you're getting the data collector (FADC) the data services (FADS) which both make up the core (currently v1.0.2) and "Analytics for FusionReactor" (currently v1.0.7). This last one you can really think of as a set of rules for importing FR data and reports to display it. Equally, you could either a) extend this or b) create your own "analytics application" for reading data. It sounds like you've successfully been able to look at route "a" and load your own format log data - GREAT!

FR (since v4) has supported an OSGi style plugin architecture. You can subscribe to OSGi events to run certain code when triggered. The public javadoc described the interface available since v4.0.1. Although v4.5.0 introduces some enhancements, you're probably still best coding to the publicly documented API:
http://www.fusion-reactor.com/fr/frapi/index.html?index-all.html
The intention is that these events are accessed by FusionReactor plugins (eg this is how the FusionReactor extensions for ColdFusion - or FREC for short - works). However, you could access them from a CF page using the FR-API.
Now having said that, I'd actually just say you may be better off keeping things simple and using the FR-API to do something like you suggest - which is already implemented and possible...
Take a look at these log methods...
http://www.fusion-reactor.com/fr/frapi/com/intergral/fusionreactor/api/FRAPI.html#log%28java.lang.String,%20java.lang.String,%20java.lang.String%29
http://www.fusion-reactor.com/fr/frapi/com/intergral/fusionreactor/api/FRAPI.html#log%28java.lang.String,%20java.lang.String%29

So your code would be something like...
<cfset frapiClass = createObject("java", "com.intergral.fusionreactor.api.FRAPI")>
<cfset frapi = frapiClass.getInstance()>
<cfset frapi.log("ehcache", "#metricA# #metricB# #metricC# #metricD#"

These logs would of course be handled within the boundaries of the existing logging/rotation system and also get ZIPed up - and optionally pushed to FusionAnalytics :-)

The only slightly tricky problem is getting a reference to the "current" request. This is because the API doesn't have a "getCurrentRequest()" type method. BUT you can still get it by looping over the result of a "getRunningRequests()" method (which returns an array of all running requests) and then checking the thread name of each in the array to your current request. A little ugly I know but it's very fast string comparisons and it works.
http://www.fusion-reactor.com/fr/frapi/com/intergral/fusionreactor/api/FRAPI.html#getRunningRequests%28%29

So at this point you could now have some new logs which are being managed nicely, contain your custom data and are getting pushed to FA.

For the FA loader I think you've already seen that you can quite easily import new data (by updating/creating new data collector markup language - or DCML).
For the reporting side, that's where you need to write application provider markup language - or APML - which describes the layout of a report (eg chart/table types, etc). The data is provided to these charts/tables through a script. We have support for several different types of script - to give you some history we started off with beanshell but we moved to Groovy and currently everything in "Analytics for FusionReactor" application is groovy. Our hope is that one day we could allow these scripts to be written in CF!

Suffice to say, what you're trying to do is relatively advanced compared to what the majority of users are asking for - we ourselves are trying to write more reports for FA and provide more plugins for FR but it's an evolutionary process - it takes time.
All the code for our current reports are available for you to review and use so the best advice I can offer is to copy one of the existing reports and start customizing it for your needs.


In summary... can you do what you want right now - absolutely! It's always what we've hoped that the community can start contributing useful extensions to the software. There are some areas where you might find documentation needs fleshing out but that also is a learning curve for us and if there's something you need please let us know.

[David Stockton]

Hi All,

One more thing that could be useful to those coming across this thread. The DCML syntax is publicly documented:

http://docs.intergral.com/pages/viewpage.action?pageId=18284720

[Charlie Arehart]

Great stuff, David. Thanks for sharing.

/charlie

 

From: fusiona...@googlegroups.com [mailto:fusiona...@googlegroups.com] On Behalf Of David Stockton
Sent: Thursday, June 28, 2012 5:01 AM
To: fusiona...@googlegroups.com
Subject: Re: [fusionanalytics] Idea - Memory analysis per scope suggestion.

 

Hi David,

Firstly, I just want to re-iterate that we really value your input - it's fantastic to start seeing the FusionAnalytics community building up and we're starting to see more and more really positive, constructive feedback - **EVERY** comment and piece of feedback we receive from all our users is taken on board. The entire FR+FA team have a weekly meeting where we can discuss feedback / ideas and this community input is really important to us... so thanks!

"From what I read, one of the goals with FA is to make is extensible (in the future?)."

FA is actually currently built as a completely extensible analytics platform. Whilst we haven't got fully fleshed out documentation, every effort has been taken during development that FA is a platform for analytics. You <snip>

[Dave]

WOW! That's some really powerful stuff right there. Sounds fantastic.

As part of the current project I'm scoping, I need to capture and ETL the output of some SQL DMVs from a remote site on a bi-daily - it sounds like I can use the same FR framework to achieve this with some simple CF code which is fantastic.

Once of the fields I need to capture would be quite long (query text, which I associate with sys.dm_query_stats), and could contain CF/LFs - I'm assuming that I can convert/replace the CR/LF pairs with something else - but is there a limit to the length that line written with the .log method can be?  

Dave

On Thursday, 28 June 2012 19:01:09 UTC+10, David Stockton wrote:

Hi David,

Firstly, I just want to re-iterate that we really value your input - it's fantastic to start seeing the FusionAnalytics community building up and we're starting to see more and more really positive, constructive feedback - **EVERY** comment and piece of feedback we receive from all our users is taken on board. The entire FR+FA team have a weekly meeting where we can discuss feedback / ideas and this community input is really important to us... so thanks!

"From what I read, one of the goals with FA is to make is extensible (in the future?)."
FA is actually currently built as a completely extensible analytics platform. Whilst we haven't got fully fleshed out documentation, every effort has been taken during development that FA is a platform for analytics. You may have noticed that when you install FusionAnalytics you're asked if you want to install the data collector (the thing that gathers data and puts it into a DB and quantizes/aggregates it) the data services (the thing that pushes data out to the clients) and "Analytics for FusionReactor" - this is just an analytics "application". From a marketing perspective we of course (at least initially) need to market the product as a single unit. But in reality you're getting the data collector (FADC) the data services (FADS) which both make up the core (currently v1.0.2) and "Analytics for FusionReactor" (currently v1.0.7). This last one you can really think of as a set of rules for importing FR data and reports to display it. Equally, you could either a) extend this or b) create your own "analytics application" for reading data. It sounds like you've successfully been able to look at route "a" and load your own format log data - GREAT!

FR (since v4) has supported an OSGi style plugin architecture. You can subscribe to OSGi events to run certain code when triggered. The public javadoc described the interface available since v4.0.1. Although v4.5.0 introduces some enhancements, you're probably still best coding to the publicly documented API:
http://www.fusion-reactor.com/fr/frapi/index.html?index-all.html
The intention is that these events are accessed by FusionReactor plugins (eg this is how the FusionReactor extensions for ColdFusion - or FREC for short - works). However, you could access them from a CF page using the FR-API.

Now having said that, I'd actually just say you may be better off keeping things simple and using the FR-API to do something like you suggest - which is already implemhttp://www.fusion-reactor.com/fr/frapi/com/intergral/fusionreactor/api/FRAPI.html#log%28java.lang.String,%20java.lang.String,%20java.lang.String%29

http://www.fusion-reactor.com/fr/frapi/com/intergral/fusionreactor/api/FRAPI.html#log%28java.lang.String,%20java.lang.String%29ented and possible...

Take a look at these log methods...

[David Stockton]

Hello David,

As far as I know there's no limit on what can be written. However, your DCML would likely include a limit on what can be analyzed in FA (because it creates a corresponding column which likely has a limit on it's length unless you choose a "text" type field).

Best regards,
David Stockton
Fusion Support Team

--
You received this message because you are subscribed to the Google Groups "FusionAnalytics" group.

To view this discussion on the web visit https://groups.google.com/d/msg/fusionanalytics/-/vXOuqh2YmqUJ.

To post to this group, send email to fusiona...@googlegroups.com.
To unsubscribe from this group, send email to fusionanalyti...@googlegroups.com.