Skip to content

Commit

Permalink
Merge branch 'deviantony:main' into master
Browse files Browse the repository at this point in the history
  • Loading branch information
waitesgithub authored Aug 6, 2021
2 parents 2cfd0d5 + 3ed2a5a commit e46ed0b
Show file tree
Hide file tree
Showing 18 changed files with 416 additions and 80 deletions.
2 changes: 1 addition & 1 deletion .env
Original file line number Diff line number Diff line change
@@ -1 +1 @@
ELK_VERSION=7.10.0
ELK_VERSION=7.13.4
33 changes: 0 additions & 33 deletions .github/ISSUE_TEMPLATE.md

This file was deleted.

11 changes: 11 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: Discuss the Elastic Stack
url: https://discuss.elastic.co
about: Please ask questions related to the usage of Elastic products in those forums.
- name: Docker Community Forums
url: https://forums.docker.com
about: Please ask questions related to the usage of Docker products in those forums.
- name: docker-elk Gitter chat room
url: https://gitter.im/deviantony/docker-elk
about: General questions regarding this project can also be asked in the chat.
87 changes: 87 additions & 0 deletions .github/ISSUE_TEMPLATE/issue_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
name: Issue report
about: Report a problem with the docker-elk integration or its documentation.
---

<!--
Thanks for your issue report!
In order for us to be able to reproduce the problem and identify the root cause
quickly, we kindly ask you to include *all* the information requested below in
your issue report. It saves us a lot of effort and allows us to provide you
with a solution with as little delay as possible.
Issues submitted without the requested information will be closed.
Thank you for your understanding.
-->


### Problem description

<!--
Please be as descriptive as possible regarding the encountered issue versus the
expected outcome.
-->

### Extra information

#### Stack configuration

<!--
Please mention all changes performed to the default configuration, including to Dockerfiles.
If possible, provide the output of the `git diff` command.
-->

#### Docker setup

<!--
Please paste the full output of the `docker version` command below.
Example:
Client: Docker Engine - Community
Version: 20.10.2
API version: 1.41
...
-->

```console
$ docker version

[OUTPUT HERE]
```

<!--
Please paste the full output of the `docker-compose version` command below.
Example:
docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
...
-->

```console
$ docker-compose version

[OUTPUT HERE]
```

#### Container logs

<!--
Please paste the full output of the `docker-compose logs` command below.
Example:
elasticsearch_1 | Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
elasticsearch_1 | {"@timestamp":"2021-01-16T21:53:38.331Z", "log.level": "INFO", "message":"version...
kibana_1 | {"type":"log","@timestamp":"2021-01-16T21:54:10+00:00","tags":["info","plugins-system"],...
...
-->

```console
$ docker-compose logs

[OUTPUT HERE]
```
58 changes: 54 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ name: CI

on:
push:
branches: [ master ]
branches: [ main ]
pull_request:
branches: [ master ]
branches: [ main ]

jobs:

Expand Down Expand Up @@ -53,6 +53,7 @@ jobs:
sed -i -e 's/\(elasticsearch.username:\) elastic/\1 kibana_system/g' -e 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' kibana/config/kibana.yml
sed -i -e 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' -e 's/\(secret_management.encryption_keys:\)/\1 [test-encrypt]/g' extensions/enterprise-search/config/enterprise-search.yml
sed -i 's/\(password:\) changeme/\1 testpasswd/g' extensions/apm-server/config/apm-server.yml
sed -i 's/\(password:\) changeme/\1 testpasswd/g' extensions/metricbeat/config/metricbeat.yml
# Run Elasticsearch and wait for its availability
Expand Down Expand Up @@ -85,15 +86,45 @@ jobs:
docker-compose logs elasticsearch
docker-compose logs logstash
docker-compose logs kibana
# next steps don't need Logstash
docker-compose stop logstash
##############################
# #
# Test supported extensions. #
# #
##############################

#
# Logspout
#

- name: Execute Logspout test suite
run: |
# Set mandatory Logstash settings
sed -i '$ a input { udp { port => 5000 codec => json } }' logstash/pipeline/logstash.conf
# Run Logspout and execute tests.
# Logstash will be restarted as a result of building the Logspout
# image, so changes above will automatically take effect.
docker-compose -f docker-compose.yml -f extensions/logspout/logspout-compose.yml up -d logspout
.github/workflows/scripts/run-tests-logspout.sh
# Revert changes to Logstash configuration
sed -i '/input { udp { port => 5000 codec => json } }/d' logstash/pipeline/logstash.conf
- name: 'debug: Display state and logs (Logspout)'
if: always()
run: |
docker-compose -f docker-compose.yml -f extensions/logspout/logspout-compose.yml ps
docker-compose -f docker-compose.yml -f extensions/logspout/logspout-compose.yml logs logspout
# next steps don't need Logspout
docker-compose -f docker-compose.yml -f extensions/logspout/logspout-compose.yml stop logspout
# next steps don't need Logstash
docker-compose stop logstash
#
# Enterprise Search
#
Expand Down Expand Up @@ -144,6 +175,23 @@ jobs:
# next steps don't need APM Server
docker-compose -f docker-compose.yml -f extensions/apm-server/apm-server-compose.yml stop apm-server
#
# Metricbeat
#

- name: Execute Metricbeat test suite
run: |
docker-compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml up -d metricbeat
.github/workflows/scripts/run-tests-metricbeat.sh
- name: 'debug: Display state and logs (Metricbeat)'
if: always()
run: |
docker-compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml ps
docker-compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml logs metricbeat
# next steps don't need Metricbeat
docker-compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml stop metricbeat
##############
# #
# Tear down. #
Expand All @@ -155,8 +203,10 @@ jobs:
run: >-
docker-compose
-f docker-compose.yml
-f extensions/logspout/logspout-compose.yml
-f extensions/enterprise-search/enterprise-search-compose.yml
-f extensions/apm-server/apm-server-compose.yml
-f extensions/metricbeat/metricbeat-compose.yml
down -v
test-swarm:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ on:
schedule:
- cron: '0 0 * * 0' # At 00:00 every Sunday
push:
branches: [ master ]
branches: [ main ]
pull_request:
branches: [ master ]
branches: [ main ]

jobs:

Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/scripts/lib/testing.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ function log {

# Log an error.
function err {
echo -e "\n[x] $1\n"
echo -e "\n[x] $1\n" >&2
}

# Return the ID of the container running the given service.
Expand All @@ -22,6 +22,7 @@ function container_id {
fi

local cid

# retry for max 60s (30*2s)
for _ in $(seq 1 30); do
cid="$(docker container ls -aq -f label="$label")"
Expand All @@ -32,7 +33,6 @@ function container_id {
echo -n '.' >&2
sleep 2
done
echo -e '\n' >&2

if [ -z "${cid:-}" ]; then
err "Timed out waiting for creation of container with label ${label}"
Expand Down Expand Up @@ -107,7 +107,6 @@ function poll_ready {
echo -n 'x' >&2
sleep 5
done
echo -e '\n' >&2

echo -e "\n${output::-3}"

Expand Down
22 changes: 6 additions & 16 deletions .github/workflows/scripts/run-tests-enterprise-search.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,22 +19,12 @@ poll_ready "$cid_es" "http://${ip_es}:9200/" -u 'elastic:testpasswd'
log 'Waiting for readiness of Enterprise Search'
poll_ready "$cid_en" "http://${ip_en}:3002/api/ent/v1/internal/health" -u 'elastic:testpasswd'

log 'Retrieving private key from Elasticsearch'
response="$(curl "http://${ip_es}:9200/.ent-search-actastic-app_search_api_tokens_v2/_search?q=name:private-key" -s -u elastic:testpasswd)"
hits="$(jq -rn --argjson data "${response}" '$data.hits.hits')"
echo "$hits"
count="$(jq -rn --argjson data "${response}" '$data.hits.total.value')"
if [[ $count -ne 1 ]]; then
echo "Private key not found. Expected 1 result, got ${count}"
exit 1
fi
key="$(jq -rn --argjson data "${hits}" '$data[0]._source.authentication_token')"

log 'Creating App Search engine'
response="$(curl "http://${ip_en}:3002/api/as/v1/engines" -s -d '{"name": "dockerelk"}' -H "Authorization: Bearer ${key}")"
log 'Ensuring that App Search API keys were created in Elasticsearch'
response="$(curl "http://${ip_es}:9200/.ent-search-actastic-app_search_api_tokens_v3/_count?pretty" -s -u elastic:testpasswd)"
echo "$response"
name="$(jq -rn --argjson data "${response}" '$data.name')"
if [[ $name != 'dockerelk' ]]; then
echo 'Failed to create engine'
declare -i count
count="$(jq -rn --argjson data "${response}" '$data.count')"
if (( count != 2)); then
echo "Expected search and private keys, got ${count} result(s)"
exit 1
fi
52 changes: 52 additions & 0 deletions .github/workflows/scripts/run-tests-logspout.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
#!/usr/bin/env bash

set -eu
set -o pipefail


source "$(dirname ${BASH_SOURCE[0]})/lib/testing.sh"


cid_es="$(container_id elasticsearch)"
cid_ls="$(container_id logspout)"

ip_es="$(service_ip elasticsearch)"
ip_ls="$(service_ip logspout)"

log 'Waiting for readiness of Elasticsearch'
poll_ready "$cid_es" "http://${ip_es}:9200/" -u 'elastic:testpasswd'

log 'Waiting for readiness of Logspout'
poll_ready "$cid_ls" "http://${ip_ls}/health"

# When Logspout starts, it prints the following log line:
# 2021/01/07 16:14:52 # logspout v3.2.13-custom by gliderlabs
#
# which we expect to find by querying:
# docker.image:"docker-elk_logspout" AND message:"logspout gliderlabs"~3
#
log 'Searching a log entry forwarded by Logspout'

declare response
declare -i count

# retry for max 60s (30*2s)
for _ in $(seq 1 30); do
response="$(curl "http://${ip_es}:9200/_count?q=docker.image:%22docker-elk_logspout%22%20AND%20message:%22logspout%20gliderlabs%22~3&pretty" -s -u elastic:testpasswd)"
count="$(jq -rn --argjson data "${response}" '$data.count')"
if [[ $count -gt 0 ]]; then
break
fi

echo -n 'x' >&2
sleep 2
done
echo -e '\n' >&2

echo "$response"
# Logspout may restart if Logstash isn't ready yet, so we tolerate multiple
# results
if [[ $count -lt 1 ]]; then
echo "Expected at least 1 document, got ${count}"
exit 1
fi
Loading

0 comments on commit e46ed0b

Please sign in to comment.