OpenSearch
This module deploys a highly-available install of OpenSearch.
Usage
Credentials
For in-cluster applications, credentials can be sourced from the following Kubernetes Secrets named in the module’s outputs:
superuser_creds_secret
: Full access to the cluster, including the ability to manage sensitive settings like replication and permissionsadmin_creds_secret
: Read and write access to all indices including the ability to configure index settingsreader_creds_secret
: Read-only access to all indices
Authenticating with OpenSearch is done via x.509 authentication. Each of the above named Secrets contains the following values:
ca.crt
: The CA certificate used to verify the server-provided certificate.tls.crt
: The certificate that the OpenSearch client should provide to the server for authentication.tls.key
: The TLS key that the OpenSearch client will use for securing communications with the server.
The credentials in each Secret are managed by Vault and rotated automatically before they expire. In the Panfactum Stack, credential rotation will automatically trigger a pod restart for pods that reference the credentials.
The credential lifetime is configured by the vault_credential_lifetime_hours
input (defaults to 16 hours). Credentials are rotated 50% of the way through their lifetime. Thus, in the worst-case, credentials that a pod receives are valid for vault_credential_lifetime_hours
/ 2.
Connecting
The below example show how to connect to the OpenSearch cluster using dynamically rotated admin credentials by mounting the client certificates in our kube_deployment module.
module "opensearch" {
source = "${var.pf_module_source}kube_opensearch${var.pf_module_ref}"
...
}
module "deployment" {
source = "${var.pf_module_source}kube_deployment${var.pf_module_ref}"
...
secret_mounts = {
"${module.opensearch.admin_creds_secret}" = {
mount_path = "/etc/opensearch-certs"
}
}
common_env = {
OPENSEARCH_HOST = module.opensearch.host
OPENSEARCH_PORT = module.opensearch.client_port
}
}
Note that you also must configure the client to use the certificates. For example, if using the opensearch NPM package:
import { Client } from "@opensearch-project/opensearch";
import { readFileSync } from "fs";
const client = new Client({
node:`http://${process.env.OPENSEARCH_HOST}:${process.env.OPENSEARCH_PORT}`,
ssl: {
ca: readFileSync("/etc/opensearch-certs/ca.crt"),
cert: readFileSync("/etc/opensearch-certs/tls.crt"),
key: readFileSync("/etc/opensearch-certs/tls.key")
},
});
Persistence
TODO once upstream issue is merged
Disruptions
TODO: adjust once the flexible topology is enabled
Disruption Windows
Disruption windows provide the ability to confine disruptions to specific time intervals (e.g., periods of low load) if this is needed to meet your stability goals. You can enable this feature by setting voluntary_disruption_window_enabled
to true
.
The disruption windows are scheduled via voluntary_disruption_window_cron_schedule
and the length of time of each window via voluntary_disruption_window_seconds
.
If you use this feature, we strongly recommend that you allow disruptions at least once per day, and ideally more frequently.
For more information on how this works, see the kube_disruption_window_controller submodule.
Completely Disabling Voluntary Disruptions
Allowing the cluster to periodically initiate restarts of OpenSearch nodes is critical to maintaining system health. However, there are rare cases where you want to override the safe behavior and disable voluntary disruptions altogether. Setting the voluntary_disruptions_enabled
to false
will set up PDBs that disallow any voluntary disruption of any OpenSearch pod in this module.
This is strongly discouraged. If limiting any and all potential disruptions is of primary importance you should instead:
- Create a one-hour weekly disruption window to allow some opportunity for automatic maintenance operations
- Ensure that
spot_instances_enabled
andburstable_instances_enabled
are both set tofalse
Note that the above configuration will significantly increase the costs of running the OpenSearch cluster (2.5-5x) versus more flexible settings. In the vast majority of cases, this is entirely unnecessary, so this should only be used as a last resort.
Enabling PDBs either manually or via disruption windows will not prevent all forms of disruption, only voluntary ones. A voluntary disruption is one that is done through the Eviction API and limited by the use of PDBs.
An example of a non-voluntary disruption would be via spot node termination or resource constraints. As a result, you should still implement defensive coding practices in your client code to account for potential disruptions.
Providers
The following providers are needed by this module:
Required Inputs
The following input variables are required:
namespace
Description: The namespace to deploy to the opensearch instances into
Type: string
Optional Inputs
The following input variables are optional (have default values):
arm_nodes_enabled
Description: Whether the opensearch pods can be scheduled on arm64 nodes
Type: bool
Default: true
aws_iam_ip_allow_list
Description: A list of IPs that can use the service account token to authenticate with AWS API
Type: list(string)
Default: []
backups_force_delete
Description: Whether to delete backups on destroy
Type: bool
Default: false
burstable_nodes_enabled
Description: Whether the opensearch pods can be scheduled on burstable nodes
Type: bool
Default: true
controller_nodes_enabled
Description: Whether to allow pods to schedule on EKS Node Group nodes (controller nodes)
Type: bool
Default: false
dashboard_domain
Description: The domain for hosting the opensearch dashboard
Type: string
Default: null
dashboard_enabled
Description: Whether the opensearch dashboard is enabled
Type: bool
Default: false
extra_cluster_settings
Description: Settings that will be added to the opensearch.yml configuration file for the cluster. Can also be used to override existing configuration values. See docs.
Type: map(any)
Default: {}
instance_type_anti_affinity_required
Description: Whether to enable anti-affinity to prevent pods from being scheduled on the same instance type. Defaults to true iff sla_target == 3.
Type: bool
Default: null
log_level
Description: The log level for the opensearch pods
Type: string
Default: "WARN"
minimum_memory_mb
Description: The minimum memory in MB to allocate to each OpenSearch node
Type: number
Default: 1000
monitoring_enabled
Description: Whether to allow monitoring CRs to be deployed in the namespace
Type: bool
Default: false
node_image_cached_enabled
Description: Whether to add the container images to the node image cache for faster startup times
Type: bool
Default: true
opensearch_version
Description: The version of OpenSearch to deploy. Note that we only test this module with the default version.
Type: string
Default: "3.0.0"
panfactum_scheduler_enabled
Description: Whether to use the Panfactum pod scheduler with enhanced bin-packing
Type: bool
Default: true
persistence_backups_enabled
Description: Whether to enable backups of the persistent storage.
Type: bool
Default: true
pull_through_cache_enabled
Description: Whether to use the ECR pull through cache for the deployed images
Type: bool
Default: false
replica_count
Description: The number of OpenSearch replicas to deploy
Type: number
Default: 3
s3_bucket_access_policy
Description: Additional AWS access policy for the remote-storage S3 bucket.
Type: string
Default: null
slow_request_log_thresholds
Description: The thresholds for which slow requests are logged. See docs.
Type:
object({
trace = optional(string, "10m")
debug = optional(string, "2m")
warn = optional(string, "30s")
info = optional(string, "-1")
})
Default: {}
spot_nodes_enabled
Description: Whether the opensearch pods can be scheduled on spot nodes
Type: bool
Default: true
storage_class
Description: The StorageClass to use for the disk storage. Cannot be changed after
Type: string
Default: "ebs-standard"
storage_increase_gb
Description: The amount of GB to increase storage by if free space drops below the threshold
Type: number
Default: 5
storage_increase_threshold_percent
Description: Dropping below this percent of free storage will trigger an automatic increase in storage size
Type: number
Default: 20
storage_initial_gb
Description: How many GB to initially allocate for persistent storage (will grow automatically as needed). Can not be changed after installation.
Type: number
Default: 10
storage_limit_gb
Description: The maximum number of GB of storage to provision for each opensearch node
Type: number
Default: null
vault_credential_lifetime_hours
Description: The lifetime of database credentials generated by Vault
Type: number
Default: 16
vault_internal_url
Description: The url to the vault instance for internal cert issuance
Type: string
Default: "http://vault-active.vault.svc.cluster.local:8200"
voluntary_disruption_window_cron_schedule
Description: The times when disruption windows should start
Type: string
Default: "0 4 * * *"
voluntary_disruption_window_enabled
Description: Whether to confine voluntary disruptions of pods in this module to specific time windows
Type: bool
Default: false
voluntary_disruption_window_seconds
Description: The length of the disruption window in seconds
Type: number
Default: 3600
voluntary_disruptions_enabled
Description: Whether to enable voluntary disruptions of pods in this module.
Type: bool
Default: true
vpa_enabled
Description: Whether the VPA resources should be enabled
Type: bool
Default: true
wait
Description: Wait for resources to be in a ready state before proceeding. Disabling this flag will allow upgrades to proceed faster but will disable automatic rollbacks. As a result, manual intervention may be required for deployment failures.
Type: bool
Default: true
Outputs
The following outputs are exported:
admin_creds_secret
Description: The name of the Kubernetes Secret holding certificate credentials for the admin role in the OpenSearch cluster
client_port
Description: The port that OpenSearch clients should connect to.
dashboard_superuser_password
Description: The password for superuser access to the opensearch dashboard
dashboard_superuser_username
Description: The username for superuser access to the opensearch dashboard
host
Description: The OpenSearch cluster hostname to connect to,
reader_creds_secret
Description: The name of the Kubernetes Secret holding certificate credentials for the reader role in the OpenSearch cluster
superuser_creds_secret
Description: The name of the Kubernetes Secret holding certificate credentials for the superuser role in the OpenSearch cluster
Maintainer Notes
No notes