Study Import in cBioPortal with redis and keycloak
Leila Kamkar
Dear cBioPortal Team,
I have configured cBioPortal to use Redis for caching (versions: cbioportal/cbioportal:6.4.1, cbioportal/session-service:0.6.4, MongoDB:4.2, MySQL:8.0, Redis:7.4.2).
However, when I attempt to import a study, I encounter the following error while creating the portal_info_folder. The portal is deployed behind Keycloak for authentication.
I would greatly appreciate your guidance on resolving this issue.
Thank you in advance for your support.
Kind regards,
Leila Kamkar
Error message:
root@f4505484fbff:/core/scripts# ./dumpPortalInfo.pl /cbioportal/my_portal_info_folder
ABORTED!
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class: URL [jar:file:/core/core-1.0.11.jar!/org/cbioportal/legacy/persistence/config/RedisConfig.class]
at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.scanCandidateComponents(ClassPathScanningCandidateComponentProvider.java:510)
at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(ClassPathScanningCandidateComponentProvider.java:351)
at org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan(ClassPathBeanDefinitionScanner.java:277)
at org.springframework.context.annotation.ComponentScanBeanDefinitionParser.parse(ComponentScanBeanDefinitionParser.java:91)
at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:74)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1391)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1371)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:179)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:150)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseDefaultElement(DefaultBeanDefinitionDocumentReader.java:201)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:176)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:150)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:96)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:520)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:400)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:347)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:319)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:184)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:220)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:191)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:253)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:130)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:95)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:131)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:716)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:598)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:144)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:85)
at org.mskcc.cbio.portal.util.SpringUtil.initDataSource(SpringUtil.java:62)
at org.mskcc.cbio.portal.scripts.DumpPortalInfo.run(DumpPortalInfo.java:123)
at org.mskcc.cbio.portal.scripts.ConsoleRunnable.runInConsole(ConsoleRunnable.java:145)
at org.mskcc.cbio.portal.scripts.DumpPortalInfo.main(DumpPortalInfo.java:192)
Caused by: java.lang.IllegalStateException: Error processing condition on org.cbioportal.legacy.persistence.config.RedisConfig
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:60)
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:99)
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:88)
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:71)
at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.isConditionMatch(ClassPathScanningCandidateComponentProvider.java:564)
at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.isCandidateComponent(ClassPathScanningCandidateComponentProvider.java:547)
at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.scanCandidateComponents(ClassPathScanningCandidateComponentProvider.java:471)
... 31 more
Caused by: org.springframework.beans.factory.BeanExpressionException: Expression parsing failed
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:185)
at org.springframework.boot.autoconfigure.condition.OnExpressionCondition.evaluateExpression(OnExpressionCondition.java:60)
at org.springframework.boot.autoconfigure.condition.OnExpressionCondition.getMatchOutcome(OnExpressionCondition.java:48)
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47)
... 37 more
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1008E: Property or field 'environment' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext' - maybe not public or not valid?
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:229)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:112)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:100)
at org.springframework.expression.spel.ast.CompoundExpression.getValueRef(CompoundExpression.java:60)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:96)
at org.springframework.expression.spel.ast.OpEQ.getValueInternal(OpEQ.java:42)
at org.springframework.expression.spel.ast.OpEQ.getValueInternal(OpEQ.java:32)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:203)
at org.springframework.expression.spel.ast.OpOr.getBooleanValue(OpOr.java:57)
at org.springframework.expression.spel.ast.OpOr.getValueInternal(OpOr.java:48)
at org.springframework.expression.spel.ast.OpOr.getValueInternal(OpOr.java:38)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:116)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:273)
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:182)
... 40 more
Leila Kamkar
These are my current settings. I would appreciate your guidance in identifying any misconfiguration.
Thank you in advance for your assistance.
Kind regards,
Leila
In application.properties:
# valid cache types are (ehcache-heap, ehcache-disk, ehcache-hybrid, redis), or use 'no-cache' to disable caching
# caution 1: the 'redis' caching option will likely cause a conflict when installing the portal in a tomcat installation which uses redisson for session management
persistence.cache_type=redis
redis.name=redis_cache
# redis service is set in the docker compose
redis.leader_address=redis://redis/6379
redis.follower_address=redis://redis/6379
redis.database=0
redis.password=some_password
redis.ttl_mins=10000
redis.clear_on_startup=true
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.data.redis.RedisAutoConfiguration
spring.data.redis.host=redis
spring.data.redis.port=6379
In Docker-compose file
redis:
image: ${DOCKER_IMAGE_REDIS}
container_name: redis_cache
restart: unless-stopped
ports:
- "6379:6379"
command: ["redis-server", "/etc/redis/redis.conf", "--appendonly", "yes"]
volumes:
- cbioportal_redis_data:/data
- ./data/redis/redis.conf:/etc/redis/redis.conf:ro
networks:
- cbio-net
Redis.conf
requirepass some_password
port 6379
ole...@thehyve.nl
I think I found the issue in the configuration. It looks like the port is currently being separated by a forward slash instead of a colon.
Current (Incorrect):
redis.leader_address=redis://redis/6379
redis.follower_address=redis://redis/6379
Correction:
redis.leader_address=redis://redis:6379
redis.follower_address=redis://redis:6379
Could you try updating the address with the colon and let us know if that fixes the connection?
Best regards,
Leila Kamkar
I even attempted to configure the portal without Redis, yet the same error occurs when creating the portal info folder. These settings worked correctly with the previous version 6.0.17 (without Redis), but in the newer version, they no longer seem to function as expected.
I would greatly appreciate any further guidance on how to address this issue.
Best regards,
Leila
--------------------------------
docker-compose:
services:
cbioportal:
restart: unless-stopped
image: ${DOCKER_IMAGE_CBIOPORTAL}
container_name: cbioportal-container
environment:
SHOW_DEBUG_INFO: "true"
PORTAL_HOME: "/cbioportal-webapp"
HTTP_PROXY: "x:port"
HTTPS_PROXY: "x:port"
NO_PROXY: "localhost,127.0.0.1"
ports:
- "8080:8080"
volumes:
- ./study:/study/
- ${APPLICATION_PROPERTIES_PATH:-./config/application.properties}:/cbioportal-webapp/application.properties:ro
- ${SECURITY_PROPERTIES_PATH:-./config/security.properties}:/cbioportal-webapp/security.properties:ro
depends_on:
- cbioportal-database
- cbioportal-session
networks:
- cbio-net
command: >
/bin/sh -c "rm -rf /cbioportal-webapp/lib/servlet-api-2.5.jar &&
java -Xms4g -Xmx8g -cp '/cbioportal-webapp:/cbioportal-webapp/lib/*'
org.cbioportal.PortalApplication
--spring.config.location=/cbioportal-webapp/application.properties
--authenticate=saml
--connectionTimeout=20000
--enable-compression
--proxy-base-url=https://cbioportal.x.de
--session.service.url=http://cbioportal-session:5001/api/sessions/my_portal/
--clickhouse_mode=$${APP_CLICKHOUSE_MODE:-false}
--spring.profiles.active=$${APP_SPRING_PROFILE:-default}"
cbioportal-database:
restart: unless-stopped
image: ${DOCKER_IMAGE_MYSQL}
container_name: cbioportal-database-container
environment:
MYSQL_DATABASE: ${DB_NAME}
MYSQL_USER: ${DB_USERNAME}
MYSQL_PASSWORD: ${DB_PASSWORD}
MYSQL_ROOT_PASSWORD: ${DB_ROOT}
ports:
- "3306:3306"
volumes:
- ./data/cgds.sql:/docker-entrypoint-initdb.d/cgds.sql:ro
- ./data/seed.sql.gz:/docker-entrypoint-initdb.d/seed.sql.gz:ro
- cbioportal_mysql_data:/var/lib/mysql
networks:
- cbio-net
command: --local-infile=1
healthcheck:
test: [ "CMD-SHELL", "test 853 -eq $$(mysql -hlocalhost -P3306 -u$$DB_MYSQL_USERNAME -p$$DB_MYSQL_PASSWORD -e 'SELECT COUNT(*) FROM type_of_cancer' cbioportal --skip-column-names --silent)" ]
interval: 10s
timeout: 3s
retries: 300
start_period: 3m
cbioportal-session:
restart: unless-stopped
image: ${DOCKER_IMAGE_SESSION_SERVICE}
container_name: cbioportal-session-container
environment:
SERVER_PORT: 5001
JAVA_OPTS: -Dspring.data.mongodb.uri=mongodb://cbioportal-session-database:27017/session-service
depends_on:
- cbioportal-session-database
networks:
- cbio-net
cbioportal-session-database:
restart: unless-stopped
image: mongo:4.2
container_name: cbioportal-session-database-container
environment:
MONGO_INITDB_DATABASE: session_service
volumes:
- cbioportal_mongo_data:/data/db
networks:
- cbio-net
networks:
cbio-net:
volumes:
cbioportal_mysql_data:
cbioportal_mongo_data:
--------------------------------------
Application.properties:
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.data.redis.RedisAutoConfiguration
persistence.cache_type=no-cache
Gaofei Zhao
--
You received this message because you are subscribed to the Google Groups "cBioPortal for Cancer Genomics Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cbioportal+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cbioportal/59eb011b-953b-4e7c-85b7-a1e2a9a64dc3n%40googlegroups.com.
Leila Kamkar
Best regards,
&& ./dumpPortalInfo.pl my_portal_info_folder \
&& ./importer/metaImport.py -p my_portal_info_folder -s "study_directory"' -v
Gaofei Zhao
Thank you for sharing the exact commands you are using.
In general, it is not necessary to run dumpPortalInfo.pl directly when importing a study. Could you try using the standard import command with "-u" as described in our documentation?
Best,
Gaofei
Leila Kamkar
The standard import command does not appear to work in our case because the portal is authenticated using SAML with Keycloak. As a result, the command fails during the authentication process.
Thank you very much for your support.
Kind regards,
$ docker compose exec cbioportal metaImport.py -u https://cbioportal.x.de:8080 -s study/lgg_ucsf_2014 -o
Starting validation...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Error occurred during validation step:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/urllib3/connectionpool.py", line 773, in urlopen
self._prepare_proxy(conn)
File "/usr/local/lib/python3.12/dist-packages/urllib3/connectionpool.py", line 1042, in _prepare_proxy
conn.connect()
File "/usr/local/lib/python3.12/dist-packages/urllib3/connection.py", line 770, in connect
self._tunnel()
File "/usr/lib/python3.12/http/client.py", line 979, in _tunnel
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 403 Forbidden
The above exception was the direct cause of the following exception:
urllib3.exceptions.ProxyError: ('Unable to connect to proxy', OSError('Tunnel connection failed: 403 Forbidden'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cbioportal.x.de', port=8080): Max retries exceeded with url: /api/info (Caused by ProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 403 Forbidden')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/metaImport.py", line 126, in <module>
exitcode = validateData.main_validate(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/core/scripts/importer/validateData.py", line 5707, in main_validate
portal_instance = load_portal_info(server_url, logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/core/scripts/importer/validateData.py", line 5314, in load_portal_info
parsed_json = request_from_portal_api(path, api_name, logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/core/scripts/importer/validateData.py", line 5164, in request_from_portal_api
response = requests.get(service_url)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/requests/adapters.py", line 694, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='cbioportal.x.de', port=8080): Max retries exceeded with url: /api/info (Caused by ProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 403 Forbidden')))
Gaofei Zhao
To view this discussion visit https://groups.google.com/d/msgid/cbioportal/25560e60-2bbd-4b2f-8049-59e0905c01ccn%40googlegroups.com.
Leila Kamkar
Thank you for your help. It is working now.
Best,
Leila