Grid 4 launched from discrete Java artifacts, UI shows '500' status

[Scott Babcock]

I'm trying to launch Selenium Grid 4.9.1 by building a class path with individual JARs and running the Bootstrap "main" method, specifying the "standalone" option. I've gotten enough of the dependencies figured out to get the server to launch without the console emitting any error, but the Grid UI shows:

> There has been an error while loading the Nodes from the Grid.
> Response not successful: Received status code 500

The browser console shows that the immediate issue originates from failed attempts to access the resource at path "/graphql", so this endpoint clearly isn't responsive.

Here's my current list of dependencies:

• selenium-grid: org.openqa.selenium.grid.Bootstrap
• selenium-remote-driver: org.openqa.selenium.grid.config.Config
• selenium-http: org.openqa.selenium.remote.http.Routable
• selenium-api: org.openqa.selenium.internal.Require
• selenium-json: org.openqa.selenium.json.Json
• guava: com.google.common.collect.ImmutableSet
• jcommander: com.beust.jcommander.JCommander
• opentelemetry-sdk-extension-autoconfigure: io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk
• opentelemetry-sdk-extension-autoconfigure-spi: io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizer
• opentelemetry-api: io.opentelemetry.api.metrics.MeterProvider
• opentelemetry-api-logs: io.opentelemetry.api.logs.LoggerProvider
• opentelemetry-api-events: io.opentelemetry.api.events.EventEmitterProvider
• opentelemetry-sdk-trace: io.opentelemetry.sdk.trace.SdkTracerProviderBuilder
• opentelemetry-context: io.opentelemetry.context.propagation.TextMapPropagator
• opentelemetry-sdk-metrics: io.opentelemetry.sdk.metrics.SdkMeterProviderBuilder
• opentelemetry-sdk-logs: io.opentelemetry.sdk.logs.SdkLoggerProviderBuilder
• opentelemetry-sdk-common: io.opentelemetry.sdk.resources.Resource
• opentelemetry-semconv: io.opentelemetry.semconv.resource.attributes.ResourceAttributes
• opentelemetry-sdk: io.opentelemetry.sdk.OpenTelemetrySdk
• jeromq: org.zeromq.ZContext
• failsafe: dev.failsafe.Policy
• graphql-java: graphql.schema.DataFetcher
• slf4j-api: org.slf4j.LoggerFactory
• netty-transport: io.netty.channel.EventLoopGroup
• netty-common: io.netty.util.concurrent.EventExecutorGroup
• netty-handler: io.netty.handler.logging.LoggingHandler
• logback-classic: org.slf4j.impl.StaticLoggerBinder
• logback-core: :ch.qos.logback.core.joran.spi.JoranException
• netty-buffer: io.netty.buffer.ByteBufAllocator
• netty-codec-http: io.netty.handler.codec.http.HttpServerCodec
• netty-codec: io/netty/handler/codec/ByteToMessageDecoder
• failureaccess: com/google/common/util/concurrent/internal/InternalFutureFailureAccess
• graphql-java: graphql.GraphQL
• selenium-chrome-driver
• selenium-chromium-driver

[⇜Krishnan Mahadevan⇝]

Scott,

Would it be possible for you to share a sample project?

Alternatively I think you just need to be using https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-grid/4.9.1/ and you should be able to spawn the Grid via code (if that's what you are trying to do)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/88333632-a14b-4f93-ad97-bfe4afe61418n%40googlegroups.com.

[Scott Babcock]

 Krishnan,

Thanks for the reply! The Bootstrap class from selenium-grid is the core of my attempt to launch a Grid instance programmatically. This is the class that defines the "main" method used by selenium-server. I don't currently have a project assembled for this but am simply building a Java command line that declares the dependencies (class path), the main class (Bootstrap), and the "standalone" option. Here are the dependencies I've identified:

• selenium-grid
• selenium-remote-driver
• selenium-http
• selenium-api
• selenium-json
• guava
• jcommander
• opentelemetry-sdk-extension-autoconfigure
• opentelemetry-sdk-extension-autoconfigure-spi
• opentelemetry-api
• opentelemetry-api-logs
• opentelemetry-api-events
• opentelemetry-sdk-trace
• opentelemetry-context
• opentelemetry-sdk-metrics
• opentelemetry-sdk-logs
• opentelemetry-sdk-common
• opentelemetry-semconv
• opentelemetry-sdk
• jeromq
• failsafe
• graphql-java
• slf4j-api
• netty-transport
• netty-common
• netty-handler
• logback-classic
• logback-core
• netty-buffer
• netty-codec-http
• netty-codec
• failureaccess
  

The server launches without reporting any errors, but the Grid UI shows the '500' status and the browser network tab shows that it gets the '500' from the "/graphql" endpoint.

[⇜Krishnan Mahadevan⇝]

Scott,

If all that you want to do is to build yet another CLI, maybe you could just leverage coursier and use their "resolve" command to resolve artifacts (maven coordinates) which gets downloaded into a directory which you can easily add up to your classpath

https://get-coursier.io/docs/cli-resolve

This is the same tool that we are using in docker selenium to manage the observability dependencies by downloading them on the fly and adding them to the classpath.

Can you please check that ? Its still not clear to me as to what your command line is intending to do, and why the already available CLI options of the selenium grid would not work for you.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/afc4b269-9e55-4dc8-a98f-4ce1a2bd3979n%40googlegroups.com.

[Scott Babcock]

Thanks for the reference to Coursier. I'll check this out to see what it can do for me.

My objective is to launch and manage a local Selenium 4 Grid programmatically, using only the required discrete Jav artifacts required by this configuration. Here are the problems with the available CLI options that I'm aware of: 

•  All of the CLI options I'm aware of are based on the massive `selenium-server` uber-JAR (more than 30 MB).
• This JAR isn't published to the standard Maven Central Java artifact repositories.
• The uber-JAR bakes in all of the dependencies required by every configuration it supports, which is why it's so large.
• It locks in the versions of all of these dependencies, making surgical remediation of bugs and vulnerabilities in each dependency impossible. (I'm not sure how you'd go about even determining which version of each discrete artifact was incorporated into the uber-JAR to figure out which identified vulnerabilities or bugs impact your installation.)

The current state of my efforts in this direction gives me a launched server that doesn't appear to actually work. The GraphQL query feature is clearly not functioning.

[⇜Krishnan Mahadevan⇝]

Scott,

Thanks for sharing the rationale.

Have you tried building your custom CLI standalone java app by adding the required dependencies? In that case, you would be able to start adding the required versions as a maven dependency and move forward no?

Basically you would start by adding a maven dependency on https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-grid/4.9.1/ and start working by adding the required dependencies.

This way, your custom CLI maven project would exactly know what are the maven dependencies that you are bringing in. It also allows you to remediate security vulnerabilities by adding exclusions in your dependency and pinning it to the newer dependency which has the fix.

Have you treaded down this path? If yes, what are the challenges you are facing?

I doubt if you would be able to get your custom CLI work in whatever approach you are using because the whole point of the Selenium Grid was to ensure that a user gets hold of an uber jar (I dont think Selenium publishes the uber jars anymore into Maven central and I believe it's being done so, because the uber jar is NOT expected to be consumed as a maven dependency) and it just works.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/52ac7c25-2a47-4b74-b7d2-16fcc3719b4en%40googlegroups.com.

[Scott Babcock]

Krishnan,

This is essentially what I'm doing from the command line. With a class path composed of the dependencies indicated previously, I get the server to launch without errors. However, the GraphQL feature of the server that's supposed to handle status and configuration queries doesn't respond... probably because the handler for this endpoint didn't get launched. This appears to be the source of the status '500' failure reported in the Grid UI. I need to keep digging to see if I can discover which class declares this handler and why it's missing from my configuration.

[⇜Krishnan Mahadevan⇝]

Scott,

If a handler did not get loaded you should see 404 and not 500 as return code.

All the handlers (which includes `org.openqa.selenium.grid.graphql.GraphqlHandler` responsible for the graphql endpoint and "org.openqa.selenium.grid.web.GridUiRoute#GridUiRoute" which is responsible for the UI) both are part of the selenium-grid jar.

If you can share your project on github, I can perhaps give it a shot and see what is causing the problem.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/1b291ed9-1ce1-48cb-9bc0-ff7d140654dcn%40googlegroups.com.

[Scott Babcock]

Krishnan,

I figured out what was missing. I set the Java command line to enable remote debugging and set a breakpoint on the handler in GraphqlHandler. This revealed that the handler was failing because it had no definition for `org.dataloader.DataLoaderRegistry`. I added the corresponding JAR to my class path, and everything now appears to work as expected.

Thanks for pointing me in the right direction!

= Scott Babcock =

[Scott Babcock]

For anyone who's interested, I pushed the most basic form of this process here:
https://github.com/sbabcoc/GridLauncherV4