It Works... But It Feels Wrong - The Real Way to Run a Java Monolith on Kubernetes Without Breaking Your Brain

**“It Works… But It Feels Wrong” — The Real Way to Run a Java Monolith on Kubernetes Without Breaking Your Brain** You’ve got: - Ubuntu host - Apache2 handling SSL - `kind` cluster - Moqui + OpenSearch in pods - MySQL on the host - Service type: NodePort - Apache reverse proxying to `172.x.x.x:30083` And it works. But it feels duct-taped together. That instinct? It’s correct. Let’s clean this up properly — production-style, self-managed, no cloud load balancers, no magic. --- # 1️⃣ NodePort + Apache Reverse Proxy — Is It Bad? Short answer: For local dev? Fine. For production? Fragile. Right now Apache is proxying to a `kind` internal IP . That’s brittle because: - `kind` is not meant for production - Control-plane IPs are not stable design boundaries - You’re bypassing Kubernetes’ intended ingress model You said you don’t want cloud-managed load balancers and you’re running everything on your own machine. That’s completely fine. The clean production-style setup on bare metal looks like this: ### Recommended Architecture (Self-Managed) ``` Internet ↓ Apache (or Nginx) — TLS termination ↓ Kubernetes Ingress Controller ↓ Service (ClusterIP) ↓ Pods ``` Instead of NodePort, install an Ingress controller like: - NGINX (nginx-ingress) - Traefik Then: - Apache proxies to the ingress controller (not NodePort) - Or even better: remove Apache entirely and let ingress handle TLS using cert-manager If you're staying fully self-managed on a single machine, the simplest clean version is: - Install nginx-ingress - Use `hostNetwork` or MetalLB if you later scale to multiple nodes - Terminate TLS at ingress NodePort is not “wrong.” It’s just low-level plumbing. Ingress exists so you don’t wire ports manually forever. --- # 2️⃣ Autoscaling a Java Monolith — The Hard Truth You’re seeing: 400–500MB RAM per pod Scaling 1 → 3 replicas = ~1.5GB memory Yes. That’s how horizontal scaling works. Kubernetes does not magically share JVM heap between replicas. Each replica = full JVM. That’s not a Kubernetes issue. That’s Java architecture. But here’s where you *can* optimize. ### 🧠 JVM Tuning If you don’t explicitly set: ``` -XX:MaxRAMPercentage -Xms -Xmx ``` The JVM may over-allocate. Set memory limits in Kubernetes: ```yaml resources: requests: memory: "512Mi" limits: memory: "512Mi" ``` Then tune JVM to respect container memory. Otherwise your pod will use more memory than you think. --- ### ⚙ HPA Strategy HPA works better with CPU than memory. Memory scaling is sticky. CPU scaling reacts faster. Your idea of 90% CPU utilization target is reasonable . But the real optimization question is: How many concurrent requests can one Moqui pod handle? Measure that. That’s your scaling unit. --- ### 🚫 The Bigger Question If you need 3 replicas for availability — fine. If you need 3 replicas for traffic — fine. But if you’re scaling because memory grows due to session storage? That’s architecture smell. Which brings us to your real pain point. --- # 3️⃣ Sessions, Logs, Locks, and PVC Bottlenecks You mentioned: - `logs/` - `txlogs/` - `sessions/` - Multiple replicas - RWX PVC - LOCK issues This is the real blocker. And it’s not Kubernetes’ fault. It’s shared filesystem semantics. Multiple Java pods writing to the same RWX volume will absolutely cause locking contention. Especially for sessions. Here’s the clean way to think about it: --- ## 🔥 Sessions Should NOT Be On Disk Disk-based sessions do not scale horizontally. Proper production setup: Use Redis. - External Redis - Or Redis in Kubernetes - Or Redis cluster Session store becomes centralized in-memory key-value store. No file locking. No RWX problems. No user logout on pod switch. This is the industry standard for Java monolith scaling. --- ## 📁 Logs Should NOT Be On Shared PVC Logs go to stdout. Then use: - Fluent Bit - Loki - ELK stack Kubernetes logging model expects logs on stdout, not shared files. If you need transaction tracing, ship logs to centralized logging — not shared disk. --- ## 📦 Static Files / Uploads Now we separate concerns: - Sessions → Redis - Logs → centralized logging - Uploads → object storage Object storage is ideal because: - It’s replica-safe - No locking - No file corruption - Scales independently If you're fully self-hosted, run: - MinIO (S3-compatible) - Or NFS (but NFS is legacy compared to object storage) RWX PVC is the least scalable option here. --- # 4️⃣ Should MySQL Be Inside Kubernetes? You currently run MySQL on host . That’s actually fine. Running databases inside Kubernetes is possible (StatefulSet), but it adds complexity: - Volume management - Backup orchestration - HA complexity If you're single-machine bare metal, keeping MySQL outside Kubernetes is completely reasonable. If you go multi-node, reconsider. --- # 5️⃣ Managing Kubernetes Through UI Yes. You can use: - Lens - Kubernetes Dashboard - Portainer Lens is by far the cleanest experience for remote cluster management. You just download kubeconfig and connect. No need to rely on fragile NodePort exposure. --- # The Real Production Shape for Your Case Since you're self-hosting on local hardware and want full control: ### Clean Architecture - Bare metal server - Proper Kubernetes distro (not kind) RKE2, Talos, k3s, etc. - nginx-ingress installed - TLS handled at ingress - Moqui pods (no shared filesystem for sessions) - Redis for sessions - Object storage (MinIO) for uploads - MySQL either outside k8s or as StatefulSet - Centralized logging No NodePort. No control-plane IP dependencies. No RWX locks. --- # The Hard but Honest Take You’re not stuck because of Kubernetes. You’re stuck because monoliths expect shared disk. Kubernetes expects stateless pods. Once you move: - Sessions → Redis - Logs → stdout - Uploads → object storage Scaling becomes boring. And boring is good. Right now your architecture works. But it’s tightly coupled to a single-machine mental model. Kubernetes shines when you remove filesystem coupling. That’s the unlock. ---