Running Cortex as a service in DCOS Mesosphere is throwing error

[Vikram Yerneni]

Hello Cortex'rs,
I am trying to deploy or run Cortex as a Service in DCOS Mesosphere and its throwing this below error:

`level=info ts=2020-09-03T15:07:09.830798562Z caller=modules.go:488 msg="RulerStorage is not configured in single binary mode and will not be started."

level=error ts=2020-09-03T15:07:09.831042466Z caller=log.go:149 msg="error running cortex" err="schema config file needs to be set\nerror initialising module: table-manager\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices\n\t/go/src/github.com/cortexproject/cortex/pkg/util/modules/modules.go:89\ngithub.com/cortexproject/cortex/pkg/cortex.(*Cortex).Run\n\t/go/src/github.com/cortexproject/cortex/pkg/cortex/cortex.go:290\nmain.main\n\t/go/src/github.com/cortexproject/cortex/cmd/cortex/main.go:164\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"
`

SO based on this error, looks like the Docker container is pulling the data from this official Cortex repo and looking for the Storage information (ruler.storage)
How can I predefine or even define the variable from Cortex within Mesosphere?

Anyone been through this or any help here will be appreciated.

Cheers!
Vikram Y

[Peter Štibraný]

Hello Vikram,

first message is just informative, and unrelated to the second error.

First message says that since ruler storage is not configured, Ruler will not be started. If you want to use ruler, please take a look at "ruler.storage.type" option and other storage-specific options with "ruler.storage." prefix. Ruler options can also be configured via YAML config file.

Second error says that Cortex wasn't given a schema file (required when using chunks storage), which must be done via "-schema-config-file" command line option. I'm not familiar with Mesosphere, but I assume it has a way to pass command line options to program it starts. Note that schema cannot be configured via YAML config file, only command line option is supported.

I hope that helps,

-Peter

--
You received this message because you are subscribed to the Google Groups "Cortex Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cortex-monitor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cortex-monitoring/8359335c-d52d-4f0a-9cdc-dd97c9b88703n%40googlegroups.com.

[Vikram Yerneni]

Hi Peter,

Thanks a lot fr the reply here...

Alright, regarding the informative message, I will try to set the ruler.storage.type through a YAML file. I am guessing the options here are Chunks or Blocks?

Coming to the error, I was chatting about this in the Slack channels and it seems this Schema Config needs to be passed prior to the deployment and Mesosphere has an option as passing the file as a volume/secret during the deployment. And I am little confused about your statement about cannot configured using the YAML config file here Peter. Because I was told that I need to pass the yaml file during the deployment.

Also one more point here, I am planing to use the Blocks storage, since its cheaper than the Chunks. And if I chose to use the block storage then I really don't have to provide this Schema Config file, right? 

Please correct me if I am wrong...

Thanks

Vikram

[Peter Štibraný]

Hi Vikram,

re ruler: this is configuring cloud storage for storing user ruler configuration files (I believe), so basically some bucket.

re schema: I am sorry for confusion here. Cortex uses configuration options or YAML file for main configuration. When using chunks storage (and only for chunks), Cortex also needs a schema file. File itself is a YAML file, but to tell Cortex where to find it, Cortex needs to be given that command-line option I mentioned in my previous email. This specific flag cannot be put into *main* config file (unlike most other CLI options).

You’re correct that you don’t need schema file when running blocks. However since you see error message about missing schema file, your Cortex is not configured correctly for blocks yet... otherwise this error message wouldn’t be generated.

-Peter

On 3 Sep 2020, at 18.59, Vikram Yerneni <vikram....@qomplx.com> wrote:

Hi Peter,

To view this discussion on the web visit https://groups.google.com/d/msgid/cortex-monitoring/81ccea7f-68fc-4154-bf30-1138efbf7163n%40googlegroups.com.

[Vikram Yerneni]

Hi Peter,

Sorry if I confused with my amateur replies here .. This Cortex is brand new for me (this is my second day literally with it)... So I apologize in advance..

Alright

Regarding Ruler: if I want to store the user rules conf files, I will try to provide the s3 bucket info. Since its informative and not mandatory for the deployment, I guess I will skip it for now.

Coming to the Schema portion: This is where I think my deployment is crapping. So I am planning to use the Blocks rather than chunks due to budget constraint. So if I opt for using blocks, how can I tell the deployment to use blocks rather than chunks, also by doing this I don't have to provide an additional schema file for the chunks storage?

Please do let me know.

Thanks

Vikram Y

[Vikram Yerneni]

Also let me ask you one quick question here...

Is there a sample YAML file for the Cortex configuration I can use it for the deployments? I tried to look in GitHub and found nothing over there...

Thanks

Vikram Y

[Peter Štibraný]

Hi Vikram,

there is an example configuration file for running Cortex (in single binary mode) with blocks: https://cortexmetrics.io/docs/configuration/single-process-config-blocks.yaml (It is used in this quick start guide: https://cortexmetrics.io/docs/getting-started/getting-started-blocks-storage/)

Most important settings are:

- in “storage” section, “engine” must be set to “blocks”

- “blocks_storage” section must be present, this is where you can configure details of various services. https://cortexmetrics.io/docs/configuration/configuration-file/#blocks_storage_config

To get a better understanding of individual components when using blocks storage, there is this section on the website: https://cortexmetrics.io/docs/blocks-storage/

These are some pointers to get you started. Don’t hesitate to ask more questions if you get stuck or something is not clear.

-Peter

To view this discussion on the web visit https://groups.google.com/d/msgid/cortex-monitoring/86f2cb17-e3d4-4a07-bcca-8b82f153e201n%40googlegroups.com.

[Vikram Yerneni]

Hi Peter,

Thanks a lot for this info and this is exactly what I need. Perfect response man...!

Let me set my environment up and will reply if I need more info. really appreciate it.

And thanks again...

Thanks

Vikram Y