[otrs] otrs speed consideration

[Mimiko]

Hello.

Finally I managed to install and set-up a working OTRS v.3.3.6 to work
without any memory leaks. It's working now stable. But the speed is very
poor. For a customer to view any ticket, it must wait more than 10 sec.
This is very much time. I verified that perl and modperl are used. DB is
on separate host with good performance in mysql.

Monitoring with top, htop and iotop does not show very much CPU or HDD
utilization. There is 4 cores and it seems that otrs uses only one core.
Rarely CPU is 100% in use. Also HDD use is less than 20MB/s.

Where to dig to improve performance?
Thanks

--
Mimiko desu.
---------------------------------------------------------------------
OTRS mailing list: otrs - Webpage: http://otrs.org/
Archive: http://lists.otrs.org/pipermail/otrs
To unsubscribe: http://lists.otrs.org/cgi-bin/listinfo/otrs

[Mark Dissington]

>>> On 30/03/2014 at 16:53, Mimiko <vbv...@gmail.com> wrote:
> Hello.
>
> Finally I managed to install and set-up a working OTRS v.3.3.6 to work
> without any memory leaks. It's working now stable. But the speed is very
> poor. For a customer to view any ticket, it must wait more than 10 sec.
> This is very much time. I verified that perl and modperl are used. DB is
> on separate host with good performance in mysql.
>
> Monitoring with top, htop and iotop does not show very much CPU or HDD
> utilization. There is 4 cores and it seems that otrs uses only one core.
> Rarely CPU is 100% in use. Also HDD use is less than 20MB/s.
>
> Where to dig to improve performance?
> Thanks

Are you using virtualisation, specifically hyper-v with linux based hosts running MySQL, and if so, what chips do your NICs run?

If they're Broadcom and all of the above boxes are ticked turn off VMQ on the chips. There is a bug in the Broadcom drivers that causes some major slowdowns. Have a look at these.


http://datacenter-flo.de/?p=1469

http://social.technet.microsoft.com/Forums/windowsserver/en-US/b7e50892-2ff2-4410-a545-c43584f30674/poor-network-when-create-virtual-switch-in-hyperv-2012?forum=winserverhyperv

http://en.community.dell.com/techcenter/virtualization/f/4472/t/19484292.aspx

http://technet.microsoft.com/en-us/library/gg162696%28v=ws.10%29

http://fundamentallygeek.blogspot.co.uk/2012/11/slow-network-access-within-virtual.html

http://www.larmib.com/2013/hyper-v-virtual-machine-very-slow-network-vmq-broadcom/

http://blog.osmicro.org/hyper-v-virtual-machine-very-slow-network-vmq/

BR,
Mark.

Netmania IT Limited
Registered in England No: 4039293
Registered Office: Units E & F, Stowe Court, Stowe Street, Lichfield, Staffordshire, UK, WS13 6AQ
VAT Reg No: 765 6677 74

This electronic message contains information from Netmania IT Ltd which may be privileged and confidential. The information is intended to be for the use of the individual(s) or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic message in error, please notify us by telephone or email (to the number or address above) immediately.

Activity and use of the Netmania IT Ltd's email system is monitored to secure its effective operation and for other lawful business purposes. Communications using this system will also be monitored and may be recorded to secure effective operation and for other lawful business purposes.

--
Scanned by iCritical.

[Mimiko]

On 31.03.2014 15:02, Mark Dissington wrote:
> Are you using virtualisation, specifically hyper-v with linux based hosts running MySQL, and if so, what chips do your NICs run?
>
> If they're Broadcom and all of the above boxes are ticked turn off VMQ on the chips. There is a bug in the Broadcom drivers that causes some major slowdowns. Have a look at these.

The mysql DB is on physical separate server with intel chipsets at base
and NIC.

OTRS is installed in Debian Wheezy x86_64 in a vertial server based on
Hyper-V 2008 R2 host on intel chipset for NIC.

--
Mimiko desu.

[Marty Hillman]

Mimiko - Did you do an upgrade in place of a pre-existing install (for example 3.3.4)? If so, did you clear the cache folders?

shell> bin/otrs.RebuildConfig.pl
shell> bin/otrs.DeleteCache.pl

This helped me when I upgraded to 3.3.4 from a prior version.

Reference: http://doc.otrs.org/3.0/en/html/x907.html

[Mimiko]

On 31.03.2014 20:37, Marty Hillman wrote:
> shell> bin/otrs.RebuildConfig.pl
> shell> bin/otrs.DeleteCache.pl
>
> This helped me when I upgraded to 3.3.4 from a prior version.

Thank you. This helped me too. Although I didn't do any upgrade. I just
clean installed otrs 3.3.6.

How often I must do this? Do I have to create a cron job for this?

[Marty Hillman]

To my knowledge, this should be a one time thing each time you do an upgrade.

-----Original Message-----
From: Mimiko [mailto:vbv...@gmail.com]
Sent: Monday, March 31, 2014 12:48 PM
To: User questions and discussions about OTRS.
Subject: Re: [otrs] otrs speed consideration

[Mimiko]

On 31.03.2014 20:50, Marty Hillman wrote:
> To my knowledge, this should be a one time thing each time you do an upgrade.

Thank you.

But still opening Package Manager takes long time. Installing packages
also takes long time.

[Mimiko]

Oh, and during package's install, all other access to otrs, like
costumers pages, a slow too.

[Sander Goudswaard]

Are you using mod_perl or perl via cgi? The latter tends to be slower.

[Mimiko]

On 01.04.2014 09:07, Sander Goudswaard wrote:
> Are you using mod_perl or perl via cgi? The latter tends to be slower.

I am using mod_perl. I installed following guide on installation.

--
Mimiko desu.

[Bogdan Iosif]

Hi Mimiko. Did you manage to solve the performance issue?

[Mimiko]

On 13.05.2014 11:56, Bogdan Iosif wrote:
> Hi Mimiko. Did you manage to solve the performance issue?

Hello.
Somehow. I did what mhillman sugested - ran
shell> bin/otrs.RebuildConfig.pl
shell> bin/otrs.DeleteCache.pl

but in long terms, still it is visible slow for agents. Dashboard
generation on server for agents takes 0-6 seconds. Accessing SysConfig
or Package Manager takes 5-15 seconds.

In case of customers and agents it seems that if the ticket system is
not accessed for some time, then first to access will wait awhile. Then
others will access pages quicker. Also, when selecting a queue or
service in new ticket page, refreshing other fields takes 1-2 seconds.
And this is a starting project, where there are 2-3 customers and 2-3
agents. I think that perl is slow anyway.

[Bogdan Iosif]

There's probably a bottleneck in your config. Either at the hypervisor level or inside the VM.

On my instance, I have 150 tickets/day, up to ~40 agents and up to ~140 customers simultaneously connected. I'm running Apache 32-bit/MySQL 32-bit/Strawberry Perl 32-bit/Windows 2008 R2 64-bit in a VM with 4 CPUs on Hyper-V (Windows 2008 R2), on 4 year old hardware. I think my access times are slightly better than yours.

Ensure your VM performs optimally. It may be worth to install OTRS in a different VM, on a different hypervisor and see if you get the same performance.

Make sure you follow chapter 6 from the admin manual for performance improvements but really for 2-3 customers and 2-3 agents you shouldn't need to do make any special performance performance tweaks for OTRS to runs decently fast.

[Mimiko]

I did say that I don't run otrs in VM. Its a dedicated physical server
on Debian Wheezy x86-64 4 logical CPU 5GB RAM. Its Xeon 2.8GHz 5 years
old. And only apache and otrs is on this server.

I did read the chapter 6. There are no more than 100 tickets now and no
more than 200 articles. Also, I use preloaded modules and
pre-established database connections. Most of the tips in this chapter
is for more than 60000 tickets.

How this is related to poor performance when accessing SysConfig or
Package Manager?

On 13.05.2014 14:49, Bogdan Iosif wrote:
> There's probably a bottleneck in your config. Either at the hypervisor
> level or inside the VM.
>
> On my instance, I have 150 tickets/day, up to ~40 agents and up to ~140
> customers simultaneously connected. I'm running Apache 32-bit/MySQL
> 32-bit/Strawberry Perl 32-bit/Windows 2008 R2 64-bit in a VM with 4 CPUs
> on Hyper-V (Windows 2008 R2), on 4 year old hardware. I think my access
> times are slightly better than yours.
>
> Ensure your VM performs optimally. It may be worth to install OTRS in a
> different VM, on a different hypervisor and see if you get the same
> performance.
>
> Make sure you follow chapter 6 from the admin manual for performance
> improvements but really for 2-3 customers and 2-3 agents you shouldn't
> need to do make any special performance performance tweaks for OTRS to
> runs decently fast.

[Bogdan Iosif]

I missed the point about you having this issue on a physical machine.

I also noticed on my systems that opening SysConfig, and especially opening Package Manager, is an operation that is much slower than most others.

The Support Package has a builtin benchmarking tool for the database. Did you run that benchmark? Are the results ok?

If everything checks out then the only possible cause that is left standing is the performance of mod_perl + Apache on your Linux distro. It would be worth to try a different, more common distro (such us CentOS or Ubuntu) and trying to run mod_perl and Apache in 32-bit vs 64-bit modes to see if there are any differences.

I'm also tackling some performance issue with my OTRS system as I'm trying to upgrade from 3.2 to 3.3 on Windows and during my investigations I've noticed mod_perl has somewhat fallen out of grace, not just on Windows where it became borderline unusable. You may want to try mod_fastcgi instead.

[Mimiko]

On 13.05.2014 16:05, Bogdan Iosif wrote:
> I also noticed on my systems that opening SysConfig, and especially
> opening Package Manager, is an operation that is much slower than most
> others.

Especially when submitting changes.

>
> The Support Package has a builtin benchmarking tool for the database.
> Did you run that benchmark? Are the results ok?

I did run Performance Log, from where I've found those seconds. Where is
the tool for database?

> If everything checks out then the only possible cause that is left
> standing is the performance of mod_perl + Apache on your Linux distro.
> It would be worth to try a different, more common distro (such us CentOS
> or Ubuntu) and trying to run mod_perl and Apache in 32-bit vs 64-bit
> modes to see if there are any differences.

Trying other distro is not an option. I don't have physical resources
for this. Same Debian handles a VoIP server with database and apache
together on other server without a problem.

>
> I'm also tackling some performance issue with my OTRS system as I'm
> trying to upgrade from 3.2 to 3.3 on Windows and during my
> investigations I've noticed mod_perl has somewhat fallen out of grace,
> not just on Windows where it became borderline unusable. You may want to
> try mod_fastcgi instead.

I've read in manual that fastcgi is slower than modperl that's why I
didn't try it. May be I will try fastcgi.

[Bogdan Iosif]

There's a package named Support that you should install from Package Manager. After it's installed you'll have a new link in the ADMIN page, named Support Assessment. Follow the link to a page presenting a report about the environment OTRS is running on. The report may load VERY slowly and return timeouts. Keep retrying. When it finally succeeds, go through it and beware of any anomalies as they may clue you into why you're having problems.

Additionally, notice that on the Support Assessment page you have a link in the upper left named SQL benchmark. Follow the link to a page from which you can run a benchmark on the database. This will help evidencing if there are issues for OTRS with the connection between the web server and the db server.

If I understood things correctly, you have the db server on a dedicated machine (call it dbsrv-machine) but you run the VoIP server on the same machine as OTRS (call it websrv-machine). In this case, an additional thing to check would be if the Apache instance running OTRS is starved for resources (memory, etc.).

[Mimiko]

SQL Benchmar results:

Result: SQL
KEY VALUE TIME COMMENT
Insert Time: 10000 8 s :-( Should not take more than 5's on an average
system.
Update Time: 10000 7 s Ok
Select Time: 10000 7 s :-( Should not take more than 6's on an average
system.
Delete Time: 10000 4 s :-) Looks fine!
Multiplier: * 1 s

----------------------------------------------------------------
Result: SQL
KEY VALUE TIME COMMENT
Insert Time: 50000 24 s Ok
Update Time: 50000 27 s Ok
Select Time: 50000 30 s Ok
Delete Time: 50000 24 s Ok
Multiplier: * 5 s


The overall report loads fast. Its all green, meaning ok. Except some
log warnings.

Well, server 1: Mysql, VoIP, Apache+PHP.
server 2: otrs, apache+modperl

On 13.05.2014 16:52, Bogdan Iosif wrote:
> There's a package named Support that you should install from Package
> Manager. After it's installed you'll have a new link in the ADMIN page,
> named Support Assessment. Follow the link to a page presenting a report
> about the environment OTRS is running on. The report may load VERY
> slowly and return timeouts. Keep retrying. When it finally succeeds, go
> through it and beware of any anomalies as they may clue you into why
> you're having problems.
>
> Additionally, notice that on the Support Assessment page you have a link
> in the upper left named SQL benchmark. Follow the link to a page from
> which you can run a benchmark on the database. This will help evidencing
> if there are issues for OTRS with the connection between the web server
> and the db server.
>
> If I understood things correctly, you have the db server on a dedicated
> machine (call it dbsrv-machine) but you run the VoIP server on the same
> machine as OTRS (call it websrv-machine). In this case, an additional
> thing to check would be if the Apache instance running OTRS is starved
> for resources (memory, etc.).

[Bogdan Iosif]

There's your problem right there. MySQL performs badly. I'm getting max 3s for these operations where you get ~7-8s and that's in a VM, on Windows.

You need to install a private MySQL instance for OTRS, on the same machine where the rest of OTRS components are installed (server 2: otrs, apache+modperl). I'm quite sure this will solve your performance problems.

[Mimiko]

On 13.05.2014 17:30, Bogdan Iosif wrote:
> You need to install a private MySQL instance for OTRS, on the same
> machine where the rest of OTRS components are installed (server 2: otrs,
> apache+modperl). I'm quite sure this will solve your performance problems.
>

I did some test on this:

--------------------------------------------
Installed mysql on otrs server
Mysql host on otrs configuration is 127.0.0.1

Result: SQL
KEY VALUE TIME COMMENT

Insert Time: 10000 3 s :-) Looks fine!
Update Time: 10000 5 s :-) Looks fine!
Select Time: 10000 5 s :-) Looks fine!

Delete Time: 10000 4 s :-) Looks fine!
Multiplier: * 1 s

KEY VALUE TIME COMMENT
Insert Time: 50000 16 s Ok
Update Time: 50000 25 s :-) Looks fine!
Select Time: 50000 25 s :-) Looks fine!
Delete Time: 50000 21 s Ok
Multiplier: * 5 s

----------------------------------
mysql is on otrs server
Mysql host on otrs configuration is otrshost.com

KEY VALUE TIME COMMENT
Insert Time: 10000 4 s Ok
Update Time: 10000 5 s :-) Looks fine!
Select Time: 10000 4 s :-) Looks fine!
Delete Time: 10000 5 s Ok
Multiplier: * 1 s

KEY VALUE TIME COMMENT
Insert Time: 50000 17 s Ok
Update Time: 50000 24 s :-) Looks fine!
Select Time: 50000 25 s :-) Looks fine!
Delete Time: 50000 21 s Ok
Multiplier: * 5 s


-----------------------------
mysql is on other server
Mysql host on otrs configuration is ip of the mysql server

KEY VALUE TIME COMMENT
Insert Time: 10000 5 s Ok
Update Time: 10000 5 s :-) Looks fine!
Select Time: 10000 6 s Ok
Delete Time: 10000 5 s Ok
Multiplier: * 1 s

KEY VALUE TIME COMMENT
Insert Time: 50000 20 s Ok
Update Time: 50000 25 s :-) Looks fine!

Select Time: 50000 30 s Ok
Delete Time: 50000 24 s Ok
Multiplier: * 5 s

--------------------------
mysql is on other server
Mysql host on otrs configuration is mysqlhost.com

KEY VALUE TIME COMMENT
Insert Time: 10000 4 s Ok
Update Time: 10000 5 s :-) Looks fine!
Select Time: 10000 6 s Ok

Delete Time: 10000 4 s :-) Looks fine!
Multiplier: * 1 s

KEY VALUE TIME COMMENT
Insert Time: 50000 22 s Ok
Update Time: 50000 26 s Ok
Select Time: 50000 29 s Ok
Delete Time: 50000 22 s Ok
Multiplier: * 5 s


----------------------------------
mysql is on another server
Mysql host on otrs configuration is mysqlhost.com
In apache on otrs host following was changed:
keepalive on
UseCanonicalName off
HostnameLookups off

KEY VALUE TIME COMMENT
Insert Time: 10000 4 s Ok
Update Time: 10000 5 s :-) Looks fine!
Select Time: 10000 6 s Ok

Delete Time: 10000 4 s :-) Looks fine!
Multiplier: * 1 s

KEY VALUE TIME COMMENT
Insert Time: 50000 20 s Ok
Update Time: 50000 25 s :-) Looks fine!
Select Time: 50000 29 s Ok
Delete Time: 50000 22 s Ok
Multiplier: * 5 s


In all cases despite of different SQL Benchmark results, accessing
tickets, dashboard, sysconfig remained same speed.

[Bogdan Iosif]

I don't have any other ideas then. Try using a more common Linux distro and try both 32 and 64 bit Apache + mod_perl.

Your hardware seems slow as far as MySQL is concerned and the CPU is probably also slow. Based on what I've seen on my system, the CPU is indeed the bottleneck for OTRS but I thought my hardware is crap and when I saw it's at least twice as fast as yours, I thought that's the problem.

[Mimiko]

On 14.05.2014 12:22, Bogdan Iosif wrote:
> Try using a more common Linux distro and try both 32 and 64 bit Apache +
> mod_perl.

More common linux distributive? Isn't Debian a common linux
distributive? Why use x32 bit and not allowing to spread mysql's or
apache's files in more RAM? x32 can address natively only 3 and a half.

>
> Your hardware seems slow as far as MySQL is concerned and the CPU is
> probably also slow. Based on what I've seen on my system, the CPU is
> indeed the bottleneck for OTRS but I thought my hardware is crap and
> when I saw it's at least twice as fast as yours, I thought that's the
> problem.

Well, I don't know what to say about hardware. Mysql is on IBM System
x3100 M3 with Xeon X3400 CPU, and otrs is on Fujitsu Siemens Primergy
Econel 200 with Intel Xeon 2.8GHz CPU. This isn't enough for normal run
for 20 agents and 100 customers with 2-3 tickets per day?

[Bogdan Iosif]

My opinion is that Redhat/CentOS or Ubuntu LTS (based on Debian, I know) are more likely to have the best performing drivers and the most tested and stable configurations, compared with other distros. Thus, I wouldn't stray from those for prod systems. Then again, I'm no Linux expert and I don't want to start a holly war about which distro is the best. Your choice.

I would care about using 64-bit to access more RAM only after the performance is satisfactory. Thus, I would test 32-bit first. Also, running 32-bit Apache + mod_perl on a 64-bit OS would give you at most 4 GB of RAM for Apache. That's way beyond enough as you shouldn't need more than 1-2 GB for this process.

Face it, your hardware is very old. I'm not one to not understand a lack of resources (believe me) but we're taking a single core CPU that was launched 8-9 years ago http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#.22Irwindale.22_.2890_nm.29 I don't know if you have 2x CPUs but, even then, your CPU power is very very low if I got the CPU model correctly.

[Mimiko]

On 14.05.2014 16:09, Bogdan Iosif wrote:
> My opinion is that Redhat/CentOS or Ubuntu LTS (based on Debian, I know)
> are more likely to have the best performing drivers and the most tested
> and stable configurations, compared with other distros. Thus, I wouldn't
> stray from those for prod systems. Then again, I'm no Linux expert and I
> don't want to start a holly war about which distro is the best. Your choice.

Before going into Linux, I did read about which distro to use. I stopped
at Debian as it is more standard compliant, straight forward and stable.
Redhat is not fully open. CentOS seemed too complicated. And Ubuntu were
performing slower than Debian. But I don't imply on Debian. Its just
main distro I use for now.

> Face it, your hardware is very old. I'm not one to not understand a lack
> of resources (believe me) but we're taking a single core CPU that was
> launched 8-9 years ago
> http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#.22Irwindale.22_.2890_nm.29
> I don't know if you have 2x CPUs but, even then, your CPU power is very
> very low if I got the CPU model correctly.

Well, of course its not modern CPUs I know that. But I use what we have.
Then its just CPU a bottleneck now, not which distro and how much bits.
First - when boss will ask for performance, let him buy new hardware
first. :)

Thanks for guiding. I could find that options in apache to tweak, so
performance in other application increased.

[Florian Edlhuber]

Hi,

13.05.2014 11:06 - Mimiko schrieb:

Somehow. I did what mhillman sugested - ran
shell> bin/otrs.RebuildConfig.pl
shell> bin/otrs.DeleteCache.pl

but in long terms, still it is visible slow for agents. Dashboard
generation on server for agents takes 0-6 seconds. Accessing SysConfig
or Package Manager takes 5-15 seconds.

Are your cron jobs running fine?

Florian

[Florian Edlhuber]

Hi,

>On 14.05.2014 16:09, Bogdan Iosif wrote:
>> My opinion is that Redhat/CentOS or Ubuntu LTS (based on Debian, I know)
>> are more likely to have the best performing drivers and the most tested
>> and stable configurations, compared with other distros.

RedHat, CentOS, Ubuntu, Debian, (Open-)SuSE as well as all Distributions with good perl
integration are performing well. There are/were some issues if you use specific
distributions and their online repository, but if you install from source
or rpm (and follow the install.md) it should work + perform fine!

How many concurrent users do you have, and what is your (virtual) hardware settings?

Florian