AWX horizontal scaling

[Sinnl]

Hi,

I have a 3 node k8s cluster (1 master and and 2 worker node - all 64Gb or RAM and 16vCPUS on VMware) and latest AWX version deployed with awx-operator using external DB and having 2 replica sets of AWX.

awx.yml

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  deployment_type: awx
  postgres_configuration_secret: awx-postgres-configuration
  secret_key_secret: awx-secret-key

  service_type: NodePort
  nodeport_port: 30080
  ingress_type: ingress
  hostname: awx
  replicas: 2

I need to run automation against ~6K nodes and trying to figure out how I can increase parallel execution. I tried increasing number of forks which seems to work, it does limit the number I put in, but after saving lower value and editing it again I can set it to something that is higher, not sure if this is some sort of a bug in UI or something else. 

Under Instance Groups I can see control plane with 2 instances each reporting capacity of 640 forks. All templates run under default execution group which is a container group providing no information on instance count or capacity.

How is scaling managed in AWX? Will it deploy more task pods if more forks are defined? 

How many task/ee pods are there and can this number be increased? 

Would adding more nodes to the cluster help with ability to run more parallel tasks? 

Can it be done and would it make sene to change the "size" of the task containers and if so how can this be done exactly. 

Finally at which point does it make sense to use slices. With AAP2 you can see execution nodes under instance group and their capacity so it's easier to figure out at which point slicing makes sense. In AWX I'm not even sure what slicing would mean as not exactly sure how many "execution nodes" I do have, what are their specs and how I can change those values (number of instances and their size) 

Any help or insight into scaling with AWX would be of great help.

Lukasz