Pyramid Registry Creates Too many connections To the Same Server with pysimplesoap

[bhe...@uniqueinsuranceco.com]

We are using pyramid for our insured portal at Unique Insurance.

We have had great success with pyramid and still use it to this day!

However we have hit a problem, when we use the pyramid.registry and register an SOAP object to the registry object our server seems to block us because we are making to many connections to the server.

I have tried using a singleton, which I will try again today. Is there a way in the registry to force it to only create one instance of the object?

 

from pyramid.config import Configurator

from pyramid.authentication import AuthTktAuthenticationPolicy

from pyramid.authorization import ACLAuthorizationPolicy

import email.utils

from pyramid.authentication import AuthTktAuthenticationPolicy

from pyramid.authorization import ACLAuthorizationPolicy

from pyramid.session import SignedCookieSessionFactory

from sqlalchemy import engine_from_config

from .models import DBSession, User

from .views.utils import email

from .libpn import PN

def main(global_config, **settings):

""" This function returns a Pyramid WSGI application.

"""

engine = engine_from_config(settings, 'sqlalchemy.')

DBSession.configure(bind=engine)

User.metadata.bind = engine

authn_policy = AuthTktAuthenticationPolicy("REDACTED")

authz_policy = ACLAuthorizationPolicy()

my_session_factory = SignedCookieSessionFactory('REDACTED')

config = Configurator(

settings=settings,

authentication_policy=authn_policy,

authorization_policy=authz_policy,

)

config.set_session_factory(my_session_factory)

config.registry.PN = PN(wsdl_cache_folder=settings['WSDL_CACHE_FOLDER'],

wsdl_baseurl=settings['WSDL_BASEURL'],

company_cache=settings['COMPANY_CACHE'])

import logging

log = logging.getLogger(__name__)

from .pysimplesoap112.client import SoapClient as SoapClient112, SoapFault as SoapFault112, SimpleXMLElement as SimpleXMLElement112

import requests

import datetime

import csv

import os

import base64

import traceback

import boto3

import time

import dateutil.parser

import pickle

import io

from .StaticDataManager import *

# Set working folder

workfolder=os.path.dirname(__file__)

if workfolder:

os.chdir(workfolder)

class Client:

def __init__(self):

self.url = ""

self.cache_path = ""

self.client = ""

def connect(self):

self.client = SoapClient112(wsdl = self.url, cache = self.cache_path)

class PN:

def __init__(self, wsdl_cache_folder='', wsdl_baseurl='',

esign_service_url = '',

company_cache=''):

ts = time.time()

self.security_token = None

self.wsdl_cache_folder = wsdl_cache_folder

self.wsdl_baseurl = wsdl_baseurl

self.company_cache_filename = company_cache

self.TransactionService = Client()

self.TransactionService.url = self.wsdl_baseurl+'XXX.svc?wsdl'

self.TransactionService.cache_path = os.path.join(self.wsdl_cache_folder,'cache_transaction')

self.TransactionService.connect()

self.LoginService = Client()

self.LoginService.url = self.wsdl_baseurl+'XXX.svc?wsdl'

self.LoginService.cache_path = os.path.join(self.wsdl_cache_folder,'XXX')

self.LoginService.connect()

self.BillingService = Client()

self.BillingService.url = self.wsdl_baseurl+'XXX.svc?wsdl'

self.BillingService.cache_path = os.path.join(self.wsdl_cache_folder,'XXX')

self.BillingService.connect()

self.PolicyService = Client()

self.PolicyService.url = self.wsdl_baseurl+'XXX.svc?wsdl'

self.PolicyService.cache_path = os.path.join(self.wsdl_cache_folder,'XXX')

self.PolicyService.connect()

self.NotesService = Client()

self.NotesService.url = self.wsdl_baseurl+'XXX.svc?wsdl'

self.NotesService.cache_path = os.path.join(self.wsdl_cache_folder,'XXX')

self.NotesService.connect()

self.ClaimsService = Client()

self.ClaimsService.url = self.wsdl_baseurl+'XXX.svc?wsdl'

self.ClaimsService.cache_path = os.path.join(self.wsdl_cache_folder,'XXX')

self.ClaimsService.connect()

self.PrintingService = Client()

self.PrintingService.url = self.wsdl_baseurl+'XXX.svc?wsdl'

self.PrintingService.cache_path = os.path.join(self.wsdl_cache_folder,'XXX')

self.PrintingService.connect()

self.AdministrationService = Client()

self.AdministrationService.url = self.wsdl_baseurl+'XXX.svc?wsdl'

self.AdministrationService.cache_path = os.path.join(self.wsdl_cache_folder,'XXX')

self.AdministrationService.connect()

self.SecurityService = Client()

self.SecurityService.url = self.wsdl_baseurl+'XXX.svc?wsdl'

self.SecurityService.cache_path = os.path.join(self.wsdl_cache_folder,'XXX')

self.SecurityService.connect()

self.ESignService = Client()

self.ESignService.url esign_service_url

self.ESignService.cache_path = os.path.join(self.wsdl_cache_folder,'XXX')

self.ESignService.connect()

te = time.time()

with io.open("timings.log", "a+", encoding="utf-8") as f:

f.write('%r %2.2f sec \n' % ('cache_file_load', te-ts))

with io.open("timings.csv", "a+", encoding="utf-8") as f:

f.write('%s %r, %2.2f \n' % (datetime.datetime.now(), 'cache_file_load', te-ts))

  

[bhe...@uniqueinsuranceco.com]

Is there anyway to force the pyramid registry to have one instance of itself or only of one reference of a signal object?

[Jonathan Vanasco]

A few things that jump out at me:

1. it looks like your `PN` class is creating 10 different `Client` objects which each start a persistent(?) connection to the SOAP server when created. that's a huge factor in overloading the connections.

2. I'm not familiar with your SOAP library. are these connection objects threadsafe/forksafe?  If not, you should defer the `connect` to happen after the application forks/threads. that will result in even more connections.

i would look at creating and disposing of the connections as needed, instead of starting up with them.  if you need all of them them on every request, i'd rethink the application design and how all the nodes interact with one other.

[Brian Herman]

Thanks man I appreciate it!

--
You received this message because you are subscribed to a topic in the Google Groups "pylons-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylons-discuss/rbXBsUrCKn4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/2ea804d0-1b40-49f2-944a-2f890d92f307%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,

Brian Herman

Unique Insurance Company

IT Department

Programmer

(773) 696-2511

bhe...@uniqueinsuranceco.com

[Brian Herman]

How do i defer the connection after the application forks? Is there like a application after fork function in pyramid?

[Brian Herman]

Oh cool there is sorry for the spam https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/deployment/forked_threaded_servers.html

[Jonathan Vanasco]

On Thursday, November 15, 2018 at 5:43:56 PM UTC-5, Brian Herman wrote:

Oh cool there is sorry for the spam https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/deployment/forked_threaded_servers.html

it depends on the container you run pyramid in - they each have a different hook.  there's a way of catching it in waitress, but i forget what that is.

[Jens W. Klein]

On 14.11.18 17:42,

bhe...@uniqueinsuranceco.com wrote:
> We are using pyramid for our insured portal at Unique Insurance.
> We have had great success with pyramid and still use it to this day!
> However we have hit a problem, when we use the pyramid.registry and
> register an SOAP object to the registry object our server seems to block
> us because we are making to many connections to the server.
> I have tried using a singleton, which I will try again today. Is there a
> way in the registry to force it to only create one instance of the object?

[...]

I do not know you SOAP Client implementation and can't help in detail here.

But recently I had a similar use-case
I used the Zeep library https://python-zeep.readthedocs.io/ for our SOAP
connections. It is a very cleanly written, well documented and easy to
use SOAP client library. It is using the famous Requests library for all
HTTP connections. So, it offers all session/connection pooling features
Requests has. This really helps getting the number of connections down.

hth Jens
--
Klein & Partner KG, member of BlueDynamics Alliance

[Brian Herman]

We have tried other SOAP libraries in the past pysimplesoap for some reason is the only one that works.

--
You received this message because you are subscribed to a topic in the Google Groups "pylons-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylons-discuss/rbXBsUrCKn4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/psm7o7%2424s%241%40blaine.gmane.org.

For more options, visit https://groups.google.com/d/optout.

[Michael Merickel]

The pyramid registry is effectively a singleton. There is only one per Pyramid app. Thus, from your example code, you're creating 10 persistent soap connections per instance of the app. If you're forking the server or anything then this may explode the way you're describing. You need to come up with some way to pool these connections or, instead of persistent connections, be able to lazily create them on demand if that can be done in a lightweight way. I don't work with SOAP much so I can't help with any specifics there but I have also had success using zeep in the one time I needed it.

You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.

To post to this group, send email to pylons-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CACNnZHarKQoxZ_zyta_xCAqg%2BJULdbHQjD8M%3DuaE%2BiMUZMFReQ%40mail.gmail.com.