Deployment of full-scale ELK stack to Kubernetes | IT Svit DevOps Solution

Elasticsearch, Logstash, and Kibana, known as ELK stack or Elastic stack are the tools of the trade for logs aggregation and analysis. As these DevOps services are amongst the most often requested, we automated their deployment with our tool available on Github.

Approximate scheme of ELK:

ELK Stack workflow

These manifests DO NOT include the Filebeat installation! Refer to the official Filebeat configuration documentation.

Configuring a new ELK installation

This installation suits Kubernetes on AWS deployment. Namespace `elasticsearch` is used by default. Elasticsearch comes with 2 endpoints: external and internal. Use both or drop the unnecessary one.

docker run --rm -ti -v ${HOME}/.opsbox -v ${PWD}:/opsbox itsvit/opsbox kubectl apply -f kubernetes-manifests/storageclass-gp2.yml

1. Launch HA Elasticsearch cluster. There will be 2 Elasticsearch masters, 2 Elasticsearch clients, 3 Elasticsearch data nodes.

docker run --rm -ti -v ${HOME}/.opsbox -v ${PWD}:/opsbox itsvit/opsbox kubectl apply -f kubernetes-manifests/elasticsearch

Customizing Logstash

1. If you need to store data in various indices, you should create a new manifest for Logstash. E.g. make a copy from existing manifest logstash-application.yaml
2. Set the required index name in the output section:
   yaml output { elasticsearch { hosts => "elasticsearch-logging.elasticsearch:9200" index => "new_index-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}"
}
}

where new_index is the required index name
3. Run this command to deploy a new Logstash:

docker run --rm -ti -v ${HOME}/.opsbox -v ${PWD}:/opsbox itsvit/opsbox kubectl apply -f kubernetes-manifests/elasticsearch/logstash-application.yaml

We successfully use this DevOps solution as a part of data analysis and processing system. Here is an example of a running solution:

This is yet another neat module from a collection of custom-tailored IT Svit DevOps tools, which ensures quick and simple deployment of a full-cycle ELK stack to Kubernetes.

Originally published at itsvit.com.