Delay log transfer to FusionAnalytics?

[vance...@healthcaresource.com]

Hello, we are using FusionReactor on our web servers, and have FusionAnalytics running on another server. Is there a way in either FR or FA to delay the processing of the FR log files until after hours? I would like to be able to wait until 1am or so for FR to send the log files for the past 24 hours, or for FA to process the received log file, if this is possible. I know I can disable it, and then I can manually send them over, but I do not want to have to do that at 1am myself, and would like that automated.

Any guidance would be helpful. Thanks

Vance Duke

[kane_...@intergral.com]

Hi Vance, 

You are able to change the logging settings so that the logs rotate once per day at a given time. 

Rotating the logs at a given time each day will ensure that only one log file of each type will be sent per day. 

The files will be sent to FusionAnalytics shortly after the given time.

To configure this you can go to Logging -> Settings -> Log Rotation Settings and specify the following options: 

Rotation Mode : Fixed - Rotate at the same time every day

Rotate Every/At : 01:00 

Kane Punter

FusionReactor Support Team 

[kane_...@intergral.com]

On Monday, September 11, 2017 at 2:00:22 AM UTC+2, vance...@healthcaresource.com wrote:

[charlie arehart]

Kane, I wonder if there may be another solution. Do you know if it's possible instead to leave the FR logging on the instance (CF or otherwise) to be hourly, but have FRAM send them instead on such a nightly basis?

(For those not aware, in the "Analytics" page of FR, in its "FusionAnalytics Settings" submenu, it's possible to have the FR logs sent to such an FA instance either directly from the current instance or via the FRAM instance.)

Kane, I appreciate that making the change in the FR instance as you pose COULD work, but it does mean then that the FR logs will now be large (for the instance) as kept on the server, and such as to look at with the Logging>Log Archive page within the FR instance.

So I'm just wondering if there may be the only or even best way to achieve what Vance asks. I'm not sure myself, but maybe one of the FA engineers could have another thought. It's indeed an interesting challenge Vance poses.

/charlie

-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com] On Behalf Of kane_...@intergral.com
Sent: Monday, September 11, 2017 02:54 AM
To: FusionReactor <fusion...@googlegroups.com>
Subject: [fusionreactor] Re: Delay log transfer to FusionAnalytics?

Hi Vance,

You are able to change the logging settings so that the logs rotate once per day at a given time.

<snip>

[David Stockton]

Hi

It is possible to change the "cron" schedule for the tasks in FusionAnalytics of which consuming logs is one. This would allow you to keep your logs in their current state but only process them in FA at the time(s) you desire. Hopefully one of the Intergral guys can tell you where those options live in FA as I can't remember off the top of my head!

Thanks
David

> --
> You received this message because you are subscribed to the Google Groups "FusionReactor" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fusionreacto...@googlegroups.com.
> To post to this group, send email to fusion...@googlegroups.com.
> Visit this group at https://groups.google.com/group/fusionreactor.
> For more options, visit https://groups.google.com/d/optout.

[kane_...@intergral.com]

Hi David,

It would be possible to change the cron scheduler time. However, the cron jobs will only perform the data volume management and quantizer jobs. Changing the schedule of these tasks will not stop the data being imported into FusionAnalytics as the jobs depend on the imported data.

To stop the data being imported, you would have to stop the logs being sent until the desired time. 

Best Regards,

Kane Punter

FusionReactor Support team. 

[kane_...@intergral.com]

Hi Charlie / Vance,

Ultimately, transferring data via FRAM rather than the FusionReactor instance will only move the issue to the FRAM instance. You would still have to ensure that the data sent to FRAM was only being sent once per day. 

I understand that sending 1 large log file, there would be a lot of data. However, to FusionAnalytics, it would make no difference if 1 large log file was sent as opposed to 24 smaller log files simultaneously. 

I have spoken to the FusionAnalytics engineers and we believe editing the FusionReactor log files as suggested in my first response would be the most efficient solution. 

Please be aware that if all data is sent Once per 24 hours there may be a considerable slow down for FusionAnalytics during this period. The FADS may also be unavailable during this time. It may be worth considering data every 12 hours rather than once per day. 

Best Regards,

Kane Punter

FusionReactor Support Team

[David Stockton]

Thanks for the clarification Kane!

Sorry for the false-hope Vance :(

David

To unsubscribe from this group and stop receiving emails from it, send an email to fusionreactor+unsubscribe@googlegroups.com.

[charlie arehart]

Kane, I appreciate that your first solution is “most efficient”, in telling Vance he could a) set the FR instance to rotate only daily and b) have it send them to FRAM itself.

But I was asking about a slight variation of things, and I was asking if it COULD work, for the reasons I’d listed. Could you please reread my last note and let me know if it would even be possible?

After you do reread it (to understand the benefits I’m trying to maintain), I will restate my question more simply: can you (and the engineers) confirm if one COULD a ) have the FR instance itself generating logs hourly and set to “send to analytics via FRAM”, and then b) have the FRAM instance set to send the logs down to FA on a daily basis? I realize it’s not specifically what Vance was asking specifically, but it’s what I am wondering as a potentially beneficial alternative to your initial solution. :-)

Thanks.

/charlie

[kane_...@intergral.com]

Hi Charlie, 

I have spoken with both the engineering team and Mikey about your proposed solution. 

The reason your solution would not be possible is because that forwarding your logs through FRAM to FusionAnalytics would not stop data from being sent every time the logs were rotated. 

All the FRAM instance would do is forward the request, not queue the request until its own log files are sent. 

It may be possible to add a setting to only send data to FusionAnalytics at specific time intervals in a future release of FusionReactor. 

Best Regards,

Kane Punter

[charlie arehart]

OK, it was worth asking, and thanks. :-) I knew it depended on whether the Logging settings in FRAM affected only its own logs (as now it seems) or whether it might have affected the processing of the FR instance’s logs (as now it seems it does not).

So yep, adding a mechanism in FRAM to better control that (the sending of logs to FA) would give someone like Vance the control he wants (send them only nightly) while giving someone like me (who looks at the FR logs) the option to have the logs remain as hourly archives in the FR instance. :-)

But to be clear, Vance, if all you wanted was to make FA get the logs only once a day, the first solution Kane offered will indeed do that for you.

 

/charlie

 

From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com] On Behalf Of kane_...@intergral.com
Sent: Tuesday, September 12, 2017 03:09 PM
To: FusionReactor <fusion...@googlegroups.com>
Subject: Re: [fusionreactor] Re: Delay log transfer to FusionAnalytics?

 

Hi Charlie, 

[vance...@healthcaresource.com]

Thanks everyone for the great replies.

A follow up question... I was able to leave the FA service running, but stop the FADC (data collector) application.  This allowed the FR logs to be transferred from the instance to the FA server each hour, consider them sent, and go through the FR archival process and still work with smaller files. I was then able to manually restart the FADC for the application, and it started processing the FR logs that had been received. I then turned it off a little later.

Is there a way to turn the FADC Application on or off via a powershell command or API call, that could be setup as a local cron job that could replace my current manual activity?

Let me know that you think about this approach.

Thanks again,

Vance

[kane_...@intergral.com]

Hi Vance, 

Unfortunately, at this time it is not possible to enable or disable the FADC Application with a script or an API call. This must be done manually via the user interface.

Just to reiterate, it does not matter if you process 1 large log or 24 smaller logs at 1 am. This will not have a detrimental affect on performance. 

Using the solution we initially suggested should work for you in this case. 

Best Regards,

Kane Punter

FusionReactor Support Team