Skip to content

Overview

Overview

observability is a foundational concept in modern software development and DevOps.

What Is an Observability Stack?

An observability stack is a set of tools and practices that allow you to understand what's happening inside your application or system, especially when things go wrong.

It helps you answer:

  • “Why is my app slow?”
  • “Why did this service fail?”
  • “What caused this spike in traffic?”

📊 Core Pillars of Observability

There are three key types of data in an observability stack, often called the 3 Pillars:

Pillar Description Example Tools
Metrics Numeric time-series data about your system (e.g., CPU usage, requests/sec) Prometheus, InfluxDB
Logs Textual records of events (e.g., error messages, request logs) Loki, ELK (Elasticsearch, Logstash, Kibana), Fluentd
Traces End-to-end request flows through a system (e.g., what microservices were called and how long they took) Jaeger, Zipkin, OpenTelemetry

🧱 Typical Observability Stack

Here’s a commonly used open-source observability stack:

Layer Tool Role
Metrics Prometheus Collects & stores metrics
Visualization Grafana Dashboards & alerts using Prometheus
Logs Loki Stores and queries logs
Tracing Jaeger or Tempo Tracks request paths across services
Instrumentation OpenTelemetry Common API for collecting data

🧠 Why Is It Useful?

  • Detect outages or slowdowns early
  • Pinpoint bottlenecks and errors
  • Provide alerts to engineers
  • Improve incident response time
  • Help with root cause analysis

🚀 FastAPI + Observability Stack (Example)

For your case with FastAPI:

  • Metrics: Use prometheus_client to expose /metrics
  • Prometheus: Scrapes that endpoint
  • Grafana: Visualizes metrics
  • (Optional) Add logs (with Loki) and tracing (with OpenTelemetry)

Setting up Observability Stack for FastAPI

Here's a basic Observability Stack for a FastAPI app using:

  • 🟡 FastAPI
  • 🔵 Prometheus (metrics collector)
  • 🟢 Grafana (dashboard/visualizer)

Later, you can extend it with logging (e.g. Loki) and tracing (e.g. Jaeger).

🐳 Docker Compose Setup

Here's a minimal docker-compose.yml for:

  1. Your FastAPI app
  2. Prometheus
  3. Grafana

📁 Folder Structure

my-observability-stack/
│
├── docker-compose.yml
├── app/
│   ├── main.py
│   └── requirements.txt
├── prometheus/
│   └── prometheus.yml

📦 docker-compose.yml

version: "3.8"

services:
  fastapi:
    build: ./app
    ports:
      - "8000:8000"
    networks:
      - monitor-net

  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    networks:
      - monitor-net

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    networks:
      - monitor-net

networks:
  monitor-net:

🔧 prometheus/prometheus.yml

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "fastapi"
    static_configs:
      - targets: ["fastapi:8000"]

🐍 app/main.py

from fastapi import FastAPI, Request
from prometheus_client import start_http_server, Counter, generate_latest
from prometheus_client import CONTENT_TYPE_LATEST
from starlette.responses import Response
import uvicorn

REQUEST_COUNTER = Counter("http_requests_total", "Total HTTP Requests")

app = FastAPI()

@app.middleware("http")
async def add_metrics_middleware(request: Request, call_next):
    REQUEST_COUNTER.inc()
    response = await call_next(request)
    return response

@app.get("/")
def read_root():
    return {"message": "Hello from FastAPI!"}

@app.get("/metrics")
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

📦 app/requirements.txt

fastapi
uvicorn
prometheus_client

🏃‍♂️ Build and Run

From the project root:

docker-compose up --build

#or

sudo docker-compose up --build

🔗 Access URLs

Tool URL
FastAPI http://localhost:8000
Metrics http://localhost:8000/metrics
Prometheus http://localhost:9090
Grafana http://localhost:3000

Grafana login: admin / admin (change after login)

📈 In Grafana

  1. Go to http://localhost:3000

  2. Add data source:

    • Type: Prometheus
    • URL: http://prometheus:9090

  3. Create a dashboard with a panel.

    • Use query: http_requests_total

✅ Next Steps

Once you get this working:

  • Add Loki for logging
  • Add Tempo or Jaeger for tracing
  • Use OpenTelemetry for standardized instrumentation

Would you like me to add logging (Loki) or tracing (Jaeger) next?

text-based, flowchart-style diagram VISULIZATION

Docker Compose Observability Stack with:

  • FastAPI App
  • Prometheus
  • Grafana
+----------------------------+
|     FastAPI Application    |
|  - Runs on port 8000       |
|  - Exposes /metrics        |
+----------------------------+
              |
              V
+----------------------------+
|        Prometheus          |
|  - Scrapes /metrics        |
|  - Stores time-series data |
|  - Uses prometheus.yml     |
+----------------------------+
              |
              V
+----------------------------+
|          Grafana           |
|  - Connects to Prometheus  |
|  - Visualizes metrics      |
|  - Dashboards & Alerts     |
+----------------------------+
              |
              V
+----------------------------+
|          You (User)        |
|  - View metrics in browser |
|  - Analyze, debug, monitor |
+----------------------------+

🔄 Docker Compose Networking

All services are on the same Docker network, so they can reach each other by container name:

  • Prometheus can scrape FastAPI via http://fastapi:8000/metrics
  • Grafana connects to Prometheus via http://prometheus:9090

📦 Updated Docker Compose Diagram (Text-Based)

+----------------------------+
|     FastAPI Application    |
|  - Exposes:                |
|     • /metrics (Prometheus)|
|     • Logs (Loki)          |
|     • Traces (Jaeger)      |
+----------------------------+
       |       |       |
       |       |       |
       V       V       V
+-----------+ +----------+ +----------+
| Prometheus| |   Loki   | |  Jaeger  |
|  Metrics  | |   Logs   | | Traces   |
+-----------+ +----------+ +----------+
       \        |        /
        \       |       /
         V      V      V
         +------------------------+
         |        Grafana         |
         | - Dashboards (All Data)|
         +------------------------+
                     |
                     V
            +------------------+
            |   You (Developer)|
            +------------------+

🐳 Next Step: Add Loki and Jaeger to docker-compose.yml

Would you like me to:

  1. 🔧 Give you the full updated docker-compose.yml with Loki & Jaeger?
  2. 📂 Provide file structure + config files (e.g. for Loki and Promtail)?
  3. ✅ Or just a minimal Loki & Jaeger config added to your existing stack?

Let me know your preference, and I’ll tailor it for you.