Journals size are increasing every hour

[Jeff Taylor]

Dear team

we have recently seen that the journals are getting created in every 1 hour of size 4gb which was getting created every 4 hour, I request every one share your view what steps required to check the reason for the growth and how it can impact the application

also please help in how we can control it

--Regards

--Taylor

[K.S. Bhaskar]

Which database are you using?

Regards

– Bhaskar

[Jeff Taylor]

Hi KS 

We are using GT.M database

--
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com

---
You received this message because you are subscribed to the Google Groups "Hardhats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hardhats+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/518a2e9d-4c59-4c6b-b027-fa683dfbffacn%40googlegroups.com.

[Coty Embry]

If a global is getting built with a lot of data and the global isn’t set to disable journaling then that can be an issue

I’ve hosed our server up building up a lot of data before on a global that backed up all the read and writes since the journals were filling up its memory

Hope that helps some

On Sep 18, 2021, at 10:43 AM, Jeff Taylor <jeff.ta...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/CAB1QNhBVTXKu%2BSgfcLjFGLH6Ow7dCoqY2ErCuqTUdXVvkTf90Q%40mail.gmail.com.

[K.S. Bhaskar]

I can't quite understand from your post whether this is a recent change in behavior or whether there has been no change in behavior and you are exploring why the journals are growing faster than you expect.

If this is a recent change, try to find out what changed. If this is old behavior that you want to troubleshoot, let's start with (a) how many database regions there are; (b) which regions are generating the large journal files; and (c) what type of journaling is used.

Is this a VistA application?

Regards

– Bhaskar

[Tim Frazier]

Not to butt in here and I am not familiar with GT.M but I would check the actual journal entries i.e. VIEW the journal and see what the actual SETS and KILLS are. 

In Intersystems Cache, IRIS etc there is usually enough information in the Journal VIEW option to hunt down the offending code and process and discover  the source of the unusual growth and stop it.

In the meantime there is always the option to stop journaling until you find the source if that option is prudent.

my two bits. 

Good luck.

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/99361669-1372-465d-8a40-8264424ebffbn%40googlegroups.com.

[K.S. Bhaskar]

The command to extract journal files is MUPIP JOURNAL EXTRACT (https://docs.yottadb.com/AdminOpsGuide/ydbjournal.html#extract-file-name-stdout is the YottaDB documentation of the command; GT.M would be the same).

If the large journal files are not a new issue, then MUPIP JOURNAL EXTRACT to look at global variable SETs that are causing the journals to grow would be the next step.

Regards

– Bhaskar

[Jeff Taylor]

Dear All

I am looking for  opinion or any Observation on two points below

1. How  Increasing journals impact on database or if it does not impact the database

Our observation: database is not increasing with the speed journals has increased like in our case journals speed has increased by 4 times but database growth is still like before

2. If the journals are getting at higher speed will it impact the application like will it impact on application response

Regards

Taylor

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/b8f78bb1-34db-4907-8cca-7bcc48a5093dn%40googlegroups.com.

[Coty Embry]

Journals only increase the file system storage, not the database storage im pretty sure

On Sep 19, 2021, at 11:27 AM, Jeff Taylor <jeff.ta...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/CAB1QNhAsmHHaHjy6nMow3ZVOJ0%3DdOEQgqo94NKCrdqB%2B9pzdUw%40mail.gmail.com.

[Mike Baker]

Since what goes in the journal file is what goes in your database there is some correlation, but If you are changing a global that already had data, and if the length of the data saved is the same length as the old data there will not be a corresponding increase in size in the database.  So your database may not expand too much or at all just because your journal files are getting bigger. Since you are on the hardhats site I will assume you are running VISTA.  I can tell you from experience that sometimes there are processes that will update the same records over and over again with the same data.  All of those sets go in the journal files but none of them are having an effect on the database.  I would second the suggestion to look inside the journal file, to see what is going on.  You may find a job that is generating a lot of writes to the same global node or a set of global nodes over and over again.  You might be able to look on your system for a job that has an unusually high number of commands or a higher than normal CPU usage but if your journals are behaving abnormally, then that is the place to look.

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/9CFD2153-77BE-4D73-83C2-AFEC68271AE1%40gmail.com.

[rms]

You could take a look at the contents of a journal file using mupip journal -extract command.  Based on what you see you may be able to track down likely causes.

To extract human readable data from a journal file:

$ mupip journal -extract=myFileOutput20210919.txt -detail -forward mumps.mjl

Where "myFileOutput20210919.txt" is the destination file name for your "human readable" output and "mumps.mjl" is the name of the journal file you'd like to see the contents of.  Since the output file is likely to be quite large, you would probably want to view the file either with an editor you know is able to look at very large files without dragging your system down, or use a viewer like the less command.  After the above command you could use:

$ less myFileOutput20210919.txt

On your last two specific questions:

1. How  Increasing journals impact on database or if it does not impact the database

Our observation: database is not increasing with the speed journals has increased like in our case journals speed has increased by 4 times but database growth is still like before

It is likely that the process or processes involved in creating the journals are setting and killing the same or similar nodes such that journals grow quickly but database size is not as much impacted.  Imagine a situation where the ^XTMP global is being set during some processing to track some intermediate values and when processing completes those values are killed -- and the whole process happens in a tight loop, possibly due to a malfunction in the software.  In that case, assuming the ^XTMP global is being journaled,  the journal file would grow quite quickly but there would be no impact to the size of the database.

2. If the journals are getting at higher speed will it impact the application like will it impact on application response

Anything that causes contention for limited IO resources can affect your performance.  Heavy journaling can possibly slow your overall system and application performance, especially if the journal is being written to the same physical disk as the database.  That said, SSDs have mitigated much of the impact of disk IO issues.

Finally, if you determine that the journaling is happening on globals that need not be journaled, you can turn off journaling for each individual global; and if you decide that these journal files are valid and needed, you can start automatically removing them if space is becoming a issue.

[Jeff Taylor]

Hi Team

Thanks for your reply, we have applied the suggestion and monitoring the server will share my observation 

Regards

Taylor

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/809324e8-ab0a-4183-bf4a-1804b3e541b4n%40googlegroups.com.

[K.S. Bhaskar]

Taylor –

I'm glad you have found a path to move forward.

In case you have support for your system, you should really ask this question of your support provider. In case you are running a production VistA on GT.M system, let me make a strong recommendation to procure support with assured service levels.

Regards

– Bhaskar

[Jeff Taylor]

Hi Team

Thanks !!

Issue is resolved, your post helped a lot!!

Regards

Taylor

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/7a71454b-f718-46de-8c1a-f60f5e563fd4n%40googlegroups.com.

[Tim Frazier]

Good to hear Jeff!

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/CAB1QNhAuHCp-XK2RbRj%3D%3DvHugvHmsoGkK%2B5Lb%3D6NdmZEg0ZeYg%40mail.gmail.com.

[K.S. Bhaskar]

Jeff –

Could you summarize what you found and what the resolution was, please? I'm sure it will help others in the future.

Regards

– Bhaskar