Job scheduling?

[iv...@uchicago.edu]

Hello,

How are the jobs scheduled to ES cluster when multiple users run their jobs? 

Does Search Guard do anything for job scheduling or does ES handle it somehow? Or do people integrate ES somehow with a regular scheduler like Slurm?

Thank you,

Igor

[Jochen Kressin]

I don't understand this question. To what jobs are you referring to exactly? 

[Igor Yakushin]

Hi Jochen,

I am coming from a standard HPC environment and not sure that I understand how elastic search behaves when multiple users are trying to query it.

Can, for example, only one query at a time be executed on ES cluster or multiple queries can run simultaneously? Are queries queued until they can run? Can queries be scheduled and request certain amount of resources (like number of nodes, number of threads, memory)?

Thank you,

Igor

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/AhVmQaG6yfw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/e06f4f7f-b73f-4bdd-b4d8-68d206de75a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Igor Yakushin, Ph.D.

Computational Scientist

Kavli Institute for Cosmological Physics, ERC #413

Research Computing Center, room #2

The University of Chicago

773-834-5313

https://rcc.uchicago.edu

[Jochen Kressin]

Ok, got you. Well, that's more of a general Elasticsearch question I guess, not so much a Search Guard one. I think you will get the best answer if you post your question on the Elasticsearch forum: https://discuss.elastic.co/

But to answer your question: Elasticsearch is a highly concurrent distributed system. Depending on the sizing and settings of your cluster it is able to handle  massive amounts of requests, including search, index, delete etc. requests, in parallel. It distributes the requests among the nodes and shards in the cluster automatically. One of the strengths of Elasticsearch is its scalability, so it can grow alongside your data or user growth (hence the name Elastic).

There are a number of configurable circuit breakers that you can use to prevent queries from eating up memory or CPU: https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker.html

But I guess you can't set these on a per-request basis. But like I said, the experts on the Elastic forum can probably answer this question in more depth.