Adopting ELK Stack for blog -1-

Why did you start thinking about log management?

I was checking the logs once in a while, and I was getting a lot of malicious requests from botnets.
The problem was that I had to ssh into the instance and check the logs with docker-compose logs, which I don't have time to do every day, and lately I haven't been doing any feature development because I've been looking for a job.

Also, if you look at the current CD pipeline
1. committing changes
2. Github Actions build Docker image and push to Docker Hub
3. Pull latest image from OCI Instance
4. server Down & Up

This flow had a fatal problem: stdout logs disappear on container restart.
Docker itself leaves logs in the path /var/lib/docker/containers, but I couldn't check them with docker-compose logs after restarting.

First attempt: collecting logs with GitHub Actions

I didn't intend to do a big ELK-like modification from the start, so I tried a relatively simple approach of simply cron-saving the logs at 0:00 every day and saving them as GitHub Actions Artifacts.

 - name: Collect logs

 uses: appleboy/ssh-action@v1.0.0

 with:

 host: ${{ secrets.OCI_HOST }}

 username: ${{ secrets.OCI_USERNAME }}

 key: ${{ secrets.OCI_SSH_KEY }}

 script: |

 cd /home/ubuntu/ && \

 mkdir -p logs && \

 docker-compose logs --no-color > logs/app-$(date +%Y%m%d).log



 - name: Create temporary SSH key file

 run: |

 echo "${{ secrets.OCI_SSH_KEY }}" > /tmp/oci_key

 chmod 600 /tmp/oci_key



 - name: Download logs to GitHub Actions

 run: |

 mkdir -p ~/.ssh

 ssh-keyscan -H ${{ secrets.OCI_HOST }} >> ~/.ssh/known_hosts

 scp -i /tmp/oci_key -r ${{ secrets.OCI_USERNAME }}@${{ secrets.OCI_HOST }}:/home/ubuntu/logs ./logs/



 - name: Upload logs as artifact

 uses: actions/upload-artifact@v4

 with:

 name: app-logs

 path: ./logs/



 - name: Clean up temporary SSH key file

 run: |

 rm /tmp/oci_key

I made this modification and ran it myself, but when I tested it, I had an unexpected problem.

As mentioned above, the moment this workflow is executed, the server goes down for a second and comes back up, and surprisingly, that's all the logs that were saved at that time.

This workflow also added the log collection process to an existing workflow, so I ended up having to run the workflow at least once a day, which I found to be very inefficient.

Second attempt: switching to the ELK stack

Then I remembered that I recently organized my portfolio during a job interview, and I searched for keywords such as Full-Text Search Index, Elasticsearch, etc. to implement a detailed search function when searching data stored in the DB for the MusicLab project, and I remembered that one of the interviewers was in charge of the search part, so I asked the question, and I found that I had to manage logs anyway, and the current blog's search function does not support Korean detailed search.

Adopting the ELK stack not only solves the problem of log loss, but also enables real-time log visualization, searchable log data, and future expansion of system monitoring capabilities. Given these strengths, we decided, "Let's get this done".

In fact, as it says in the portfolio description on my resume, this blog was never intended to be a portfolio, but rather my own sketchbook where I can actually apply the technology stacks I'll be learning in the future.

I immediately modified the existing docker-compose.yml to expose Elasticsearch, Logstash, and Kibana as containers, and converted the existing FastAPI's uvicorn default logs into a JSON format suitable for ELK using python-json-logger, as it is difficult to parse directly in ELK.

After that, I configured NGINX and tried to connect to Kibana with the configured path, and was greeted with

"404 Not Found."

This is where the never-ending battle with NGINX + Kibana began. I'll chronicle this shoveling in my next post.