Confd Download

[Kandy Swartzel]

Confdis a configuration management system that can actively watch a consistent kv store like etcd or zookeeper and change config files based on templates whenever those key change; it can also tell the service depending on said configs to reload its own configuration. It is ideal to coordinate changes across a cluster almost atomically whenever your application doesn't support the kv store as a config source. It will probably also speed up our integration of more and more services easier.

Confd will watch the keys you tell it to watch in the service file, and re-apply your template whenever they change. If the resulting file has changed compared to what you precedingly obtained, confd will change the file on disk, and - if you have given it such information - will first validate the new config file and in case it's valid, it will issue the reload command you indicated. Please note that confd will not watch its own templates directory for changes, so if you change a template, you need to restart confd. No interruption of service will come from confd not running for a short while, and any changes it didn't get earlier will be accounted for. If you are using our own confd puppet module, it will take care of it

Roughly speaking, puppet holds configuration, and etcd holds the state of resources. So, using f.e. varnish, the whole VCL logic should be configured via puppet, but the backends list should probably be generated via confd. Which brings us to the next topic

The main strategy, whenever the same config file holds configuration and state, is to make puppet generate just the confd template files, and let confd create the actual config files and take care of the service.

I've never heard of a way to disable the compilation of the confd-debug log. If this is a paid license, you will want to et a case open with Sophos Support to see what's causing so many entries into that log.

I checked with Sophos paid support, they said disabling it may not be possible but rather suggested adjusting the retention period to 30 or 90 days. But for us, 1 year retention period is a must, to which they suggested to manually deleting older confd-debug logs.

If you've selected "Delete logs after a year" for 'Automatic Log File Deletion', Jesse, then you're correctly configured. Maybe something else is filling the disk that makes confd-debug a problem. What does df -h tell you?

Interesting. If you're not getting notifications that the logging partition is filling up, this shouldn't be a problem. Nonetheless, it's unusual - I wonder what might be causing this. What do you see with the following at the command line?

It seems we have found the issue, so the firewall has been storing thousands of IPs within DNS definitions, this are from the domain names attached with AWS elastic IPs, which are dynamic in nature and changes all the time. Once we've cleaned it up, the issue has been pretty much resolved.

Main task: Programs routes and ACLs, and anything else required on the host to provide desired connectivity for the endpoints on that host. Runs on each machine that hosts endpoints. Runs as an agent daemon. Felix resource.

Programs information about interfaces into the kernel so the kernel can correctly handle the traffic from that endpoint. In particular, it ensures that the host responds to ARP requests from each workload with the MAC of the host, and enables IP forwarding for interfaces that it manages. It also monitors interfaces to ensure that the programming is applied at the appropriate time.

Programs routes to the endpoints on its host into the Linux kernel FIB (Forwarding Information Base). This ensures that packets destined for those endpoints that arrive on at the host are forwarded accordingly.

Provides network health data. In particular, it reports errors and problems when configuring its host. This data is written to the datastore so it visible to other components and operators of the network.

calico/node can be run in policy only mode where Felix runs without BIRD and confd. This provides policy management without route distribution between hosts, and is used for deployments like managed cloud providers. You enable this mode by setting the environment variable, CALICO_NETWORKING_BACKEND=none before starting the node.

Main task: Gets routes from Felix and distributes to BGP peers on the network for inter-host routing. Runs on each node that hosts a Felix agent. Open source, internet routing daemon. BIRD.

BGP route reflectors are often configured for large deployments rather than a standard BGP client. BGP route reflectors acts as a central point for connecting BGP clients. (Standard BGP requires that every BGP client be connected to every other BGP client in a mesh topology, which is difficult to maintain.)

For redundancy, you can seamlessly deploy multiple BGP route reflectors. BGP route reflectors are involved only in control of the network: no endpoint data passes through them. When the Calico BGP client advertises routes from its FIB to the route reflector, the route reflector advertises those routes out to the other nodes in the deployment.

Main task: Monitors Calico datastore for changes to BGP configuration and global defaults such as AS number, logging levels, and IPAM information. Open source, lightweight configuration management tool.

Confd dynamically generates BIRD configuration files based on the updates to data in the datastore. When the configuration file changes, confd triggers BIRD to load the new files. Configure confd, and confd project.

(Optional) Calico enforces network policy for workloads at both the Linux kernel (using iptables, L3-L4), and at L3-L7 using a Envoy sidecar proxy called Dikastes, with cryptographic authentication of requests. Using multiple enforcement points establishes the identity of the remote endpoint based on multiple criteria. The host Linux kernel enforcement protects your workloads even if the workload pod is compromised, and the Envoy proxy is bypassed.

Dikastes can be terminated by issuing an HTTP POST request to /terminate on the socket address specified using environment variables DIKASTES_HTTP_BIND_ADDR and DIKASTES_HTTP_BIND_PORT. This is to allow for graceful termination so that Kubernetes Jobs can complete successfully and is analogous to Envoy's /quitquitquit. eg. curl -XPOST :7777/terminate

The Calico binary that presents this API to Kubernetes is called the CNI plugin, and must be installed on every node in the Kubernetes cluster. The Calico CNI plugin allows you to use Calico networking for any orchestrator that makes use of the CNI networking specification. Configured through the standard CNI configuration mechanism, and Calico CNI plugin.

etcd is a consistent, highly-available distributed key-value store that provides data storage for the Calico network, and for communications between components. etcd is supported for protecting only non-cluster hosts (as of Calico v3.1). For completeness, etcd advantages are:

Main task: Command line interface to create, read, update, and delete Calico objects. calicoctl command line is available on any host with network access to the Calico datastore as either a binary or a container. Requires separate installation. calicoctl.

For cloud providers, Calico has a separate plugin for each major cloud orchestration platform. This allows Calico to tightly bind to the orchestrator, so users can manage the Calico network using their orchestrator tools. When required, the orchestrator plugin provides feedback from the Calico network to the orchestrator. For example, providing information about Felix liveness, and marking specific endpoints as failed if network setup fails.

ConfD is a low-level API that can be used to perform all administrative actions in an Exasol system. You can interact with ConfD using the confd_client command-line tool, which is available on all nodes in a cluster. In AWS deployments, you can also access ConfD through XML-RPC in your own Python programs.

ConfD abstracts the administration and configuration tasks into jobs that can take various parameters. When you submit jobs in ConfD, they are orchestrated into a scheduler that executes them asynchronously. In the ConfD scheduler, each job has its own pipeline in which several stages are defined. The scheduler saves the result for the job at each stage. The client that submitted the job is then free to perform other actions while the pipeline is being executed.

The final stage of a ConfD job depends on whether the job modifies the database or is a read-only job. The job is marked as finished when the final stage has completed. The client can then query the result using its job ID.

Each database node contains a configuration file /exa/etc/EXAConf where most of the configuration settings that are managed by the ConfD jobs are stored. Manual changes to this file are normally not necessary and may cause unexpected behavior.

ConfD supports basic authentication with a user and password. When you create a new Exasol deployment, the user admin is added with a password specified in the CCC_PLAY_ADMIN_PASSWORD parameter in the c4 configuration.

In most cases it does not matter on which node you access ConfD. If you do not specify a node, c4 will connect to the first active node in the deployment. If the cluster is configured with an access node, the first node is the access node (usually n10). The command prompt in COS indicates which node you are connected to. For example:

You can use different syntax formats when submitting jobs in confd_client. Most jobs can be submitted using either of these formats. For more information about the accepted syntax for a specific ConfD job, refer to the respective job description in this reference.

Job Name Description bucket_add This job adds one or more buckets to a BucketFS service. bucket_delete This job deletes a bucket. bucket_modify This job modifies a bucket. bucketfs_add This job adds a BucketFS service. bucketfs_delete This job deletes a BucketFS service. bucketfs_info This job provides information about a BucketFS service. bucketfs_list This job lists BucketFS services. bucketfs_modify This job modifies a BucketFS service.

3a8082e126