Performance degradation upgrading from 2019.2.4 to 3004.2

[Barry Wainstock]

I am upgrading a saltstack environment from 2019.2.4 to 3004.2 and seeing significant performance decreases and I'm having trouble pinpointing where things are going wrong.

For some context, we heavily utilize Saltstack for remote execution by calling the CherryPy API.  Our monitoring around the REST API shows a 2-5x increase in latency for most endpoints.

Digging into what happens in the API a bit, it looks like the NetapiClient.run method gets called, so I looked there for some timings.  Using the same lowstate each time, salt 3004.2 takes 2x to run the test.version function:

{'client': 'local', 'tgt': 'localhost', 'arg': '', 'token': '8625a42f1b3058293d29a6c2260394029593281a5d45fb082b35be7c69ca479c', 'fun': 'test.version'}

>>> list(salt.version.salt_information()) 

[('Salt', '2019.2.4')] 

>>> client = salt.netapi.NetapiClient(opts) 

>>> print(timeit.timeit('client.run(lowstate)', globals=globals(), number=10)/10)

0.2503005139529705

>>> client = salt.netapi.NetapiClient(opts) 

>>> list(salt.version.salt_information()) 

[('Salt', '3004.2')] 

>>> print(timeit.timeit('client.run(lowstate)', globals=globals(), number=10)/10)

0.5183012321591377

On the other hand, timing the LocalClient.run directly shows essentially the same performance:

>>> list(salt.version.salt_information())

[('Salt', '3004.2')]

>>> client = salt.client.get_local_client(mopts=opts)

>>> print(timeit.timeit('client.cmd(**low)', globals=globals(), number=10)/10)

0.22454488202929496

—

>>> list(salt.version.salt_information())

[('Salt', '2019.2.4')]

>>> client = salt.client.get_local_client(mopts=opts)

>>> print(timeit.timeit('client.cmd(**low)', globals=globals(), number=10)/10)

0.23006968647241594

I'm not sure where to look next to figure out what is causing this degradation.  Is there anything I'm overlooking in the upgrade or is there something in the 300x series that would cause this?

Thanks for any help you can provide.