Adopting ELK Stack for blog -2-

Introduction

As I announced in the last installment, I had a war with Kibana and NGINX's reverse proxy, which was about a week ago now. But why am I posting about it now, because after I won that war, I had a lot of exams and interviews and didn't have time to finish the basic setup of the ELK stack until yesterday. Today I'm going to talk about my experience with the reverse proxy of the two.

Kibana and NGINX Reverse Proxy

After finishing the basic Docker and NGINX setup, I happily connected to example.url/kibana/, but was greeted with a 404 Not Found from NGINX.
By default, I set up the reverse proxy to redirect requests to example.url/kibana/ to example.url/kibana/, which worked fine until I got to the point where I got a 301 Moved Permanently. The problem was that the redirected /kibana/ was still returning 404s.

I couldn't figure out why, so I tried connecting directly to the port locally.

curl -iL localhost:5601/kibana

Funnily enough, I was automatically redirected to localhost:5601/kibana/app/home/ and was able to see the Kibana screen. This is where the never-ending configuration battle began, but here's what I remember trying.

(A) Kibana basePath related settings
- I tried to adjust Kibana settings such as server.basePath, server.rewriteBasePath, server.publicBaseUrl, etc. to reconcile path handling with NGINX.

(B) Handling rewrite on NGINX
- I tried to apply rewrite ^/kibana/(.)$ /$1 break; to the /kibana/ path on NGINX and pass it to Kibana.
- We tried adjusting the response Location header with proxy_redirect / /kibana/;, but the double rewrite caused a mismatch with the path Kibana intended, resulting in infinite redirects.

(C) Trying different redirect paths
- I tried forcing redirects to /kibana/spaces/enter, /kibana/app/home, etc. when accessing /kibana, but my version of Kibana returned a 404 because these routes did not exist on the server side.

I ended up with the following successful setup

Kibana configuration

server.host: 0.0.0.0
server.port: 5601

server.basePath: "/kibana"
server.rewriteBasePath: true
server.publicBaseUrl: "https://example.url/kibana"

• We set server.basePath to /kibana, and rewriteBasePath to true, so that Kibana internally strips the /kibana prefix and maps it to the correct path. This prevents Kibana from including unnecessary basePaths in internal API requests.
• We did the same for server.publicBaseUrl, which ensures that the response Location header uses the correct absolute URL. This also ensures that Kibana can load static resources to the correct URL internally.

NGINX settings

location = /kibana{
    return 301 /kibana/;
}

location = /kibana/ {
    proxy_pass http://localhost:5601;
    proxy_redirect off;
}

• Redirect requests to /kibana (without slashes) to /kibana
• We removed the existing rewrite setting and passed it without the trailing slash, disabling any redirect processing.

This configuration ensures that
1. NGINX redirects incoming requests to /kibana to /kibana/, and then passes the path below /kibana/ to Kibana without rewriting.
2. Kibana's configuration is set to rewriteBasePath: true, so it automatically processes the /kibana prefix for the request URLs it receives and performs the correct internal routing (e.g. /spaces/enter -> /app/home, etc.).
3. proxy_redirect off; prevents the response Location header from changing, so the URL set by Kibana remains the same. This is to allow Kibana to handle the basePath on its own and avoid adding unnecessary redirects from NGINX.

Ultimately, what failed was that both NGINX and Kibana initially tried to rewrite the basePath at the same time, which resulted in incorrectly transformed URLs, resulting in 404s or infinite redirects due to the mismatch between the internal routing Kibana expects and the actual URLs delivered.

Writing this out now, it seems like a pretty simple solution, but it took me about 2 days to solve this problem, and I almost used up my Cursor IDE token and my ChatGPT token. I also consulted the official documentation, but I got even more confused when I saw that rewriteBasePath is False by default.

I'm not kidding, I must have typed sudo nginx -t and sudo systemctl restart nginx hundreds of times over those two days. It was like something out of a dream. I was exhausted but happy to be done with this, so I went to Kibana, only to find that Logstash was busted and FastAPI wasn't getting any logs.

JSON parse error, original data now in message field {:message=>"Unexpected character ('\\' (code 92)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (String)\"\\u0003\\u0004\\u0003\\u0003\\u0003\\u0002\\u0003\\u0001\\u0003\\u0000\\u00003\\u0000&\\u0000$\\u0000\\u001D\\u0000 \\xB2\\xFE\\xD1'\\xDE\\rt\\xB2-:\\x889\\xAB\\xA0\\xFD\\x9A@\\u001C\\u0018\"; line: 1, column: 2]",

I started to wonder why I even bothered to try to roll ELK, and if I should just live with it, but I'd already crossed the river and was ready to fight again. That's a story for another time.