Blog · November 4, 2026

From Ready Pods to a Working System: Building a Distributed Kafka Chat Architecture

An experimental architecture case study, November 2026. The system described here is a laboratory proof of concept, not a production deployment or a compliance certification.

Source code and reproducible lab: Kafka Chat — Experimental Distributed Messaging Architecture on GitHub. The repository contains the Rust gateway, Helm/deployment manifests, bootstrap scripts, architecture notes, and verification matrix. See also the architecture documentation and verification matrix.

A green Kubernetes dashboard is reassuring. It is not a proof that a system works.

In this project, I saw an identity provider report Ready and serve valid OpenID Connect discovery metadata, yet remain effectively unmanageable because its initial administrator credential had not been made available. I also saw a healthy PostgreSQL high-availability cluster reject a properly configured client: the server certificate identified spilo.example.org, not the Kubernetes Service DNS name the application used. Later, an OAuth project and API application existed, but machine-to-machine authentication still failed with invalid_client.

None of those incidents was a Kubernetes readiness failure. All were application-level integration failures.

I wanted to explore a more interesting question than “Can I get Kafka and PostgreSQL running on Kubernetes?”: Can I bootstrap an isolated, security-conscious tenant stack from scratch, rerun that bootstrap without breaking existing state, and demonstrate authenticated messaging and permission revocation across independent application instances?

The resulting PoC combines Rust/Axum WebSocket gateways, Kafka in KRaft mode, PostgreSQL managed by Patroni and the Zalando Operator, encrypted Longhorn volumes, RustFS, ZITADEL, and a PostgreSQL-backed room access-control model. A final authenticated test delivered messages in both directions between separate gateway Pods and showed that an authenticated principal removed from a room could neither publish nor receive subsequent messages.

This article covers the architecture, the evidence, the costs, and the failures that changed the design. It deliberately separates what was observed from what still needs testing.

1. Goals, assumptions, and boundaries

The starting point was a small Rust/Kafka chat prototype. Its initial terminal-oriented design was useful for learning about producers, consumers, topics, and asynchronous work, but it was not enough to reason about identity, room membership, multi-instance fan-out, recovery, or operational isolation. The experiment therefore evolved into a WebSocket backend running on Kubernetes.

The target was one namespace per tenant, with a dedicated application stack inside it. For this lab, the namespace is kafka-chat. A tenant gets its own Kafka HA deployment, PostgreSQL HA cluster, identity provider, Rust gateways, storage credentials, and namespace-scoped configuration. Operators that manage multiple installations remain cluster-wide or operator-scoped shared components.

This choice prioritizes isolation and an understandable operational boundary over hardware efficiency. It is not the only valid multi-tenant design. A shared Kafka cluster plus tenant-specific topics, or a shared PostgreSQL cluster with separate databases and strict access controls, might be more economical. I wanted to test the stronger dedicated-stack model first.

The success criteria were concrete: a clean bootstrap that brings up the whole tenant infrastructure; a second bootstrap run that preserves credentials and persistent identities; valid internal TLS identities; a real machine-issued OAuth access token; independent JWT verification; two authenticated gateway replicas; cross-instance messaging; and a room membership revocation test on already connected clients.

Several goals were intentionally out of scope. The PoC does not demonstrate availability across independent physical hosts, a throughput or latency service-level objective, complete disaster recovery of the live database, browser-based Authorization Code with PKCE, or compliance with a particular government or industry framework. Three replicas on one physical machine do not create three independent physical failure domains.

2. Architecture: a dedicated stack inside one namespace

The gateway is the public-facing application boundary in the design, although this experiment used internal connectivity rather than a production public ingress. Clients authenticate with ZITADEL-issued JWT access tokens. The Rust/Axum process validates those tokens using OIDC discovery and JWKS, then asks PostgreSQL whether the verified identity may enter a room, publish, or receive a message.

Kafka carries room events between gateways. Each gateway needs to receive events relevant to its locally connected clients. This matters: if gateway replicas simply share one Kafka consumer group, Kafka will divide partitions among them; it will not automatically broadcast every room event to every replica. The implementation uses separate gateway consumer groups for instance-level fan-out, with the acknowledged cost that every gateway may consume events it ultimately does not deliver locally.

PostgreSQL has a different job. It is the authority for room membership and permissions, not the transport for chat events in this PoC. Its primary handles security-sensitive ACL decisions. The read-only HAProxy route is a separate experiment for non-authoritative reads; it is explicitly not placed in the authorization path.

The namespace arrangement also makes naming predictable. A service such as kafka-chat-postgres has a DNS identity derived from its namespace and cluster domain. Kafka advertised broker names, PostgreSQL certificate SANs, Secrets, Helm values, and OIDC endpoints must all refer to the same actual deployment topology. A namespace is more than an organizational label once clients validate server identities.

There is a critical caveat: a Kubernetes namespace is not, by itself, a complete security boundary. A defensible tenant model also needs ServiceAccounts and RBAC, network restrictions, admission and Pod Security policies, resource controls, Secret-management rules, and separation of privileged operators. This lab exercises part of that pattern; it does not prove complete hostile-tenant isolation.

3. A reproducible bootstrap, not a migration exercise

An early iteration placed services in several namespaces, including kafka-ha and a PostgreSQL Service under database. That made the setup harder to reason about and directly contributed to mismatched DNS assumptions. I initially considered preserving and migrating every experimental database, Kafka offset, and PVC to the final namespace. For a disposable lab, that was the wrong optimization.

The real objective was bootstrap, not migration. The final approach discarded verified disposable lab state and rebuilt the dedicated stack under kafka-chat. The reset was narrowly scoped: identify which resources belong to this experiment before deleting anything, and leave shared operators and unrelated workloads alone.

Figure: Dependency gates rather than just Pod readiness. The repeat run tests reuse of existing persistent identities; the arrows show logical prerequisites, not a single serialized command sequence.

The repository acquired two entrypoints:

scripts/reset-kafka-chat-lab.sh
scripts/bootstrap-kafka-chat-lab.sh

The reset command is a separate, guarded operation. Ordinary bootstrap execution does not imply data destruction. The bootstrap checks its prerequisites, creates or reuses tenant resources, provisions credentials and certificates, deploys stateful components in dependency order, and validates resulting connectivity. It is not a general-purpose cloud provisioning framework; it is a reproducible workflow for this lab.

The most important verification was running it twice. The first real clean run produced a healthy three-member PostgreSQL cluster, a three-broker Kafka HA cluster, a Ready RustFS tenant, and an initialized ZITADEL. The second run reused existing credentials, PostgreSQL PVCs, and Kafka cluster identity rather than generating a new cluster or invalidating Secrets. In a later iteration, provisioning also created the OAuth machine identities needed to start authenticated gateways.

Idempotency has limits: observing a successful repeat run does not prove that every imaginable partial failure can be recovered automatically. It does show that the normal rerun does not casually destroy the identity of already-running stateful services. That is substantially stronger evidence than a successful helm template or a manifest dry-run.

A good bootstrap script is also a contract between components. The order matters: cert-manager must be able to issue the service identity before a strict TLS client connects; the PostgreSQL primary and required roles must exist before ZITADEL initializes; the ZITADEL administrator must be usable before automation can create machine accounts; and the machine token must be issued and checked before an auth-required gateway can start. Treating each stage as an explicit dependency produces actionable failures instead of a long sequence of apparently healthy but unusable Pods.

The contract should be testable without publishing Secrets. An operational report can safely include namespace, release names, object readiness, certificate SANs, token claim names, audience-match results, and Kafka topic replication settings. It should not include actual PATs, machine secrets, JWTs, or Longhorn encryption material. That distinction is particularly important if the experiment will be described in a public article.

There is another reproducibility lesson here. If the automation requires a person to create a TLS Secret or copy an OAuth audience into an environment variable before each install, the documented procedure is incomplete. Those steps must either be automated or made explicit, validated external dependencies. The final flow derives its verified audience from a real token rather than inventing an identifier.

4. Kafka HA: KRaft, replication, and cross-instance fan-out

The Kafka portion uses three KRaft combined broker/controller instances. The final lab creates fresh KRaft metadata on a clean install. The chat topic was verified with replication factor 3 and minimum in-sync replicas 2. Kafka communication used TLS and SASL authentication, and the deployment checked broker discovery as well as the bootstrap Service. The latter is easy to miss: after the initial metadata request, a Kafka client contacts advertised broker addresses, not only the hostname provided at startup.

For durability semantics, min.insync.replicas=2 has its intended effect when paired with an appropriate producer acknowledgement configuration such as acks=all. The topic setting alone is not a guarantee that every client uses that acknowledgement policy. Kafka also provides ordering within partitions, not an unconditional total order across an application.

The room-event model is important to the chat design. Keying events by room can keep that room's records together under a stable partitioning scheme, while message IDs and bounded deduplication help manage duplicate observations at the gateway. That still does not establish exactly-once delivery to WebSocket clients. A Kafka produce acknowledgement means the broker accepted the record under its configured durability conditions; it does not prove that every connected browser or CLI client received it.

For cross-gateway delivery, each Rust instance consumes the event stream and distributes eligible room events to its own connected subscribers. The design avoids the common error of expecting a shared consumer group to behave as a broadcast mechanism. It does incur extra consumption and authorization work as the number of gateways grows. At larger scale, room-aware routing, partition ownership, or a different fan-out topology might become attractive; this PoC did not benchmark those alternatives.

Figure: Separate consumer groups deliver each event to both gateway processes; each gateway then filters for locally connected and authorized subscribers. This is not a client delivery-acknowledgement protocol.

Earlier migration experiments also explored message and offset transfer between separate Kafka clusters, including a rollback path. That was useful operational research, but it is not part of the final clean-bootstrap contract. Fresh test data and consumer groups are appropriate for a laboratory reset. The article's core result is the verified new cluster and authenticated event flow, not a zero-loss production migration guarantee.

Kafka HA here means replicated Kafka processes and topic data. It does not imply survival of a physical host outage. The earlier lab evidence placed several replicas on the same worker; the final experiments did not demonstrate independent physical fault domains. Kafka's own documentation also discourages combined broker/controller roles for critical production environments. Both caveats matter when interpreting a three-node topology running on one machine.

5. PostgreSQL HA: Patroni and one primary for authorization

PostgreSQL 16 is managed by the Zalando PostgreSQL Operator with Patroni. The clean bootstrap verified one leader and two streaming replicas. The observed WAL replay lag in that check was zero, and the three instance PVCs were encrypted. A point-in-time observation of zero lag says the replicas were caught up during the check; it does not make asynchronous replication synchronous or prove that every replica read is current at every instant.

The application needs two distinct databases or logical database contexts: ZITADEL's own state and Kafka Chat's room model. Room membership uses a deliberately small schema: rooms and room_members, with an authenticated identity represented by the validated issuer and subject, plus a room role such as owner or member. The backend uses parameterized queries and transactional membership changes.

A separate read-only routing experiment showed why “send reads to a replica” is not a sufficient policy. The authorization path cannot tolerate a stale membership decision after someone has been removed from a room. Even a short replication delay might allow a revoked account to be treated as an active member. This is why ACL reads go to the primary, despite the availability and scaling trade-off.

Strict PostgreSQL TLS became an unexpectedly important part of this work. The running Spilo instance initially presented a certificate associated with spilo.example.org, while the application connected through a Kubernetes Service in another namespace. The Rust client correctly rejected the identity mismatch. Disabling hostname verification would have made the integration test pass for the wrong reason.

The fix was to produce a certificate with SANs matching the actual Service DNS names, configure Zalando's custom TLS Secret, and set the required spiloFSGroup. The clean namespace-local bootstrap then verified the primary Service connection using full hostname and CA verification. That fix addressed the infrastructure naming model rather than simply suppressing a client error.

Figure: The failure was expected behavior from a strict TLS client. The right fix was a certificate matching the real internal Service identity, not disabling verification.

6. Encrypted Longhorn volumes and the boundary of backup evidence

The storage experiment used a Longhorn StorageClass with encryption enabled. The per-PVC key controller creates distinct, PVC-bound encryption Secrets. Earlier checks inspected the node's device mappings and confirmed that a filesystem was mounted over a dm-crypt/LUKS device, rather than merely trusting a Kubernetes annotation. Separate PVC keys were compared and found distinct. The final PostgreSQL HA deployment reported three encrypted PVCs.

This is useful evidence for storage-at-rest encryption, but it must be interpreted precisely. It does not prove resistance to a fully compromised Kubernetes administrator, unrestricted access to encryption Secrets, or a compromised node that already has a volume mounted. Key lifetime, recovery, rotation, and off-host key protection would need their own threat model.

The stack also has an isolated RustFS tenant, managed by an existing RustFS Operator. The S3-compatible endpoint and dedicated bucket were tested in earlier experiments as a target for WAL-G. The most revealing recovery exercise did not start as a backup success. A base-backup attempt initially failed because the restored fixture lacked the necessary checkpoint WAL segment. Another attempt ran into an AppArmor-related signal permission mismatch: a kubectl exec process and the PostgreSQL service process did not share the same confinement context. A corrected disposable fixture ran PostgreSQL and WAL-G in the same appropriate process tree.

The resulting isolated WAL-G experiment passed checkpoint creation, WAL archiving, backup/fetch, recovery startup, and SQL integrity checks. It restored known test records to a consistent point. That does not mean the running PostgreSQL HA cluster completed a disaster-recovery drill. In particular, recovery of the active cluster, promotion after loss of its primary, and replay of additional post-backup WAL beyond the demonstrated restore point were not proven.

There are two distinct claims: encrypted persistent volumes were observed, and a disposable backup/restore fixture worked. A mature DR story would add scheduled backup validation, retention checks, an independent restore environment, tested key recovery, and measurable recovery point and recovery time objectives.

Figure: A consistent isolated restore is meaningful evidence, but it must not be relabeled as verified live HA disaster recovery.

7. Replica-only read routing and its consistency trade-off

The HAProxy proof of concept explored read scaling without silently falling back to the PostgreSQL leader. Its backend discovery used Kubernetes DNS and Patroni's replica health endpoint with a strict zero-lag query, represented in the fixture by /replica?lag=0&replication_state=streaming.

It is easy to overstate what a health check means. A replica can become stale after it was checked and before the next query. A lag threshold of zero at sampling time is not a linearizability guarantee. The goal of this router was narrower: choose nodes that currently advertise the required replica state and reject connections when no eligible replica exists.

A controlled backend fixture exercised actual client connections during health transitions. A failing backend was excluded after two unsuccessful checks, in approximately 2–5 seconds, and re-entered after two successful checks, in approximately 4–8 seconds. When all simulated replicas were unhealthy, the router failed closed. Earlier SQL tests reached both replica IPs and confirmed pg_is_in_recovery()=true, not the primary.

These are useful but bounded observations. The delays describe a controlled fixture, not a statistically significant availability benchmark. Actual WAL lag injection, production-style Patroni failover, and EndpointSlice replacement were not established as end-to-end results. The HAProxy container also needed a higher memory limit after an earlier 64 MiB limit produced exit code 137; the tested configuration used 128 MiB.

This component illustrates a broader trade-off. Replica routing is attractive for eventually consistent reads and reports. It is the wrong default for a security check that must act on freshly committed membership changes. The application therefore keeps its authorization workload on the primary even though an RO router exists.

Figure: The RO route fails closed when no replica is eligible. The health check is sampled; it cannot promise that a replica remains current after the check.

8. ZITADEL: when Ready did not mean usable

For identity, I chose to test ZITADEL as an alternative to Keycloak, not to run a head-to-head benchmark. Keycloak was not deployed into this Kafka Chat lab. There is consequently no measured ZITADEL-versus-Keycloak comparison in the results.

The deployed ZITADEL application version was 4.19.2, using Helm chart 10.4.0, and a dedicated database in the existing PostgreSQL HA cluster. The instance used an internal issuer, originally verified as http://zitadel.kafka-chat.svc.cluster.local:8080. That HTTP issuer was an intentional internal laboratory constraint, not a recommendation for a publicly exposed production IdP.

The first identity-provider experiment reached a deceptive state. Helm setup completed, the application Pod was Ready, and discovery and JWKS returned HTTP 200. Database inspection showed an instance, an iam-admin machine user, and a human administrator. Yet the expected bootstrap PAT or usable administrator credential had not been materialized. The instance could advertise its keys while the automation could not create an OAuth client or test authentication.

The important issue was the first-instance credential lifecycle: rerunning Helm against an already initialized ZITADEL database did not recreate missing initial access. For a disposable lab, rather than writing a complex recovery procedure, I reset only ZITADEL's experiment-specific state and configured first-instance bootstrap correctly. The clean run created the administrative credentials, supported an OAuth client-credentials test, and passed independent JWT signature, issuer, audience, and expiration checks.

The next failure was different. The full single-namespace bootstrap automatically created a ZITADEL project and API applications, but token issuance returned invalid_client. The cause was established: the provisioning step tried to use API-application credentials for a machine-user client_credentials flow. The corrected implementation used ZITADEL User Service v2 machine users with their own persisted secrets. Real tokens were issued, and JWT verification passed. The audience was read and validated from actual token content and passed through Helm as a string rather than allowing its long numeric identifier to be reinterpreted.

This distinction is worth more than a simple “ZITADEL worked” status. An OAuth resource server, an API application, a service-account identity, and an administrator PAT are different objects with different responsibilities. Treating them as interchangeable can create an installation that appears configured but cannot authenticate the client one intends to use.

Figure: Authentication ends at verified token identity; room authorization is a separate, primary-backed decision. Human browser login with PKCE was not part of this test.

In a prior test, idle ZITADEL usage was observed at approximately 7 milliCPU and 104 MiB of memory; earlier observations had different values. These are snapshots from particular runs, not a load benchmark or a claim about predictable tenant operating cost. The database, storage, and auxiliary processes also consume resources.

9. Rust JWT validation and PostgreSQL room authorization

The Rust/Axum gateway does not depend on a vendor-specific ZITADEL SDK. Instead, it uses OIDC discovery and JWKS to verify signed access tokens. The implemented validation path caches discovery metadata and keys, restricts algorithms to RS256, checks issuer and audience, validates expiration and nbf, requires a valid subject and key identifier, and refreshes keys when rotation or an unknown kid requires it.

Authentication only answers who is making this request? It does not answer is this subject allowed to read or publish in this room? Room authorization is therefore stored separately in PostgreSQL. The stable identity key is issuer plus subject, not a display name or a user ID supplied in the WebSocket payload.

The gateway has distinct enforcement points for subscriptions, publishing, membership management, and outbound delivery. That last point was an intentional design decision. If a gateway checks a membership once when a client subscribes, removing that member from PostgreSQL would not necessarily stop messages flowing over an already open socket. The implementation performs revocation-aware authorization before outbound delivery.

Two kinds of consistency meet in this path. Kafka provides event transport and its own delivery/replay semantics; PostgreSQL supplies the current authorization state. A Kafka event can be present in the consumer stream while the recipient is no longer authorized to receive it. The gateway must enforce that distinction at the moment it decides to send the event to its WebSocket client.

This costs database work. Doing an ACL check at delivery time can increase read traffic, latency, and sensitivity to database availability. In-memory caching could improve performance but introduce a revocation window unless carefully invalidated. For a correctness-oriented PoC, primary-backed checks gave a clear model to test before optimizing.

The WebSocket Bearer-token path was tested with machine clients. Browser clients do not universally permit arbitrary Authorization headers in native WebSocket handshakes. A production browser experience would need a deliberately designed login and socket-establishment flow, potentially using a short-lived WebSocket ticket. That work, along with interactive Authorization Code plus PKCE, was outside this experiment.

10. What the end-to-end tests actually proved

The end-to-end test used real ZITADEL-issued JWT access tokens for two distinct machine subjects and connected them to different gateway Pods. The test did not rely on the Kubernetes Service randomly choosing separate replicas. The gateway instances interacted through Kafka HA, while permission decisions were read from the PostgreSQL primary.

The first authenticated rehearsal established successful delivery in both directions. An unauthorized subject was denied room access. The clean bootstrap could then be rerun without rebuilding the Kafka cluster, resetting PostgreSQL PVCs, or rotating machine secrets.

The final targeted test kept both WebSocket clients connected while the room owner removed the other authenticated principal from the membership table. The revoked principal's new publish attempt was rejected; a new message published by the owner was not delivered to the revoked principal. A reconnect and resubscription attempt also failed. When membership was restored, room access worked again.

Experiment	Observation	What it does not prove
Clean bootstrap	PostgreSQL HA, Kafka HA, RustFS and ZITADEL came up successfully	Robust recovery from every interrupted bootstrap step
Second bootstrap	Credentials, PVCs and Kafka cluster identity reused	Exhaustive long-term idempotency under all failure modes
Kafka HA	Three brokers; topic RF=3/min ISR=2	Independent physical-host availability
PostgreSQL HA	One leader, two streaming replicas; observed lag 0	Synchronous consistency for every replica read
PostgreSQL TLS	Correct Service SAN and trusted-CA verification	Complete PKI lifecycle or production certificate rotation tests
OAuth / JWT	Real machine tokens; signature, issuer and audience validated	Human browser login or PKCE
Cross-gateway messaging	Authenticated A→B and B→A delivery passed	Global exactly-once client delivery or throughput targets
Revoked publish and delivery	Both denied after membership removal	Retroactive cancellation of already authorized, in-flight events
Reconnect after revocation	Subscription denied; restoration later passed	Absence of all possible revocation races
WAL-G fixture	Backup/fetch and consistent restore passed	Full recovery drill of the active PostgreSQL HA cluster

The reported membership-change operation on the PostgreSQL primary took approximately 5 ms locally in the final revocation rehearsal. That is not an end-to-end revocation latency measurement. No post-commit race was observed within the bounded test, but “none observed” is not a linearizability proof, and messages already in flight cannot be recalled retroactively. Failure injection for unavailable PostgreSQL ACL queries was not performed in that final rehearsal.

A useful way to state the result is: removing an authenticated principal from a room must prevent tested subsequent publishing and outbound delivery, including across separate gateway instances, once the membership removal has committed. That statement is narrower, and much stronger, than claiming “instant revocation” without a defined consistency model.

11. Failure modes that improved the design

Several problems shaped the final architecture more than the initial successful deployments did.

Namespace drift broke identity assumptions. Services spread across database, kafka-ha, and kafka-chat made internal DNS, TLS SANs, client endpoint configuration, and ownership less predictable. The fix was a single namespace-local bootstrap and explicit exceptions for shared operators. No stateful cross-namespace migration was needed for a disposable experiment.

A successful Helm installation hid failed administrative initialization. ZITADEL looked healthy from the outside but lacked usable bootstrap credentials. The improvement was to treat admin access and OAuth token issuance as functional readiness checks. A Pod becoming Ready is only the beginning.

A configured OAuth application was not a machine identity. Creating projects and API applications did not make the machine client_credentials request valid. The actual fix required the right ZITADEL user type and its persisted credentials, followed by an issued and independently verified token. The distinction belongs in deployment automation, not just in documentation.

Correct TLS verification exposed a naming mistake. PostgreSQL was healthy and reachable, yet the presented certificate did not match the Service DNS. The safe response was to fix certificate issuance and Spilo configuration, not to set a weaker client TLS mode. Strict verification is useful precisely because it makes such mistakes visible.

Backup success required recovery evidence. WAL-G needed the necessary checkpoint WAL and a process arrangement compatible with AppArmor restrictions. A command returning success is not sufficient; an isolated restore with SQL integrity checks supplied stronger proof. Even that proof does not extend automatically to live-cluster DR.

Replica health is a sampled observation. HAProxy excluded failing backends and failed closed in a controlled test, but periodic checks do not establish a consistency barrier. This reinforced the decision to keep security-sensitive ACL reads on the PostgreSQL primary.

Authorization must survive the WebSocket connection lifetime. Authentication at handshake and ACL at subscription were not enough for revocation semantics. The final test specifically verified outbound filtering after a membership change, across two independently connected gateways.

These failures are not isolated incidents. Together, they illustrate a broader engineering principle: deployment readiness must be demonstrated through verified application behavior, not infrastructure health alone.

12. Trade-offs, limits, and what I would do differently

The most obvious benefit of this design is its clear boundary. One tenant namespace contains the tenant's runtime components, Secrets, certificates, storage, and application-specific persistence. The bootstrap can create the stack from scratch, and a rerun can verify that it does not rotate foundational identities. Dedicated infrastructure also reduces the ways one tenant's application-level data and credentials can accidentally be mixed with another's.

That separation is expensive. Three Kafka processes, three PostgreSQL instances, a separate IdP, application gateways, storage volumes, monitoring, and backup infrastructure are substantial overhead for a small tenant. A namespace per tenant does not eliminate shared cluster failure modes or the operational trust placed in cluster-wide controllers. It also multiplies upgrade, certificate, backup, and observability work across tenants. The correct economic answer may be a hybrid: dedicate stateful infrastructure to higher-isolation customers and use carefully designed shared clusters elsewhere.

This also changes the questions to ask before assigning a tenant its own dedicated stack. What is the minimum sustainable footprint when Kafka, PostgreSQL, and an IdP are all allocated separately? How will application and operator upgrades be rolled out across hundreds of namespaces? Which components can share a trusted control plane without accidentally sharing tenant data? What is the plan for tenants with wildly different traffic profiles? The lab answers none of these quantitatively, but it makes the resource and operational questions impossible to ignore.

The Kafka fan-out model has a similar scale trade-off. Separate consumer groups per gateway are straightforward and testable, but they amplify Kafka reads as gateway count rises. PostgreSQL checks on outbound delivery preserve revocation semantics more directly, but they add authorization queries and make the primary an important dependency. Neither approach has been load-tested here, so I cannot attach credible messages-per-second, p95 latency, or tenants-per-node claims.

The lab also had an important physical limit. Logical replication, several Pods, and separate PVCs do not establish host-level HA when the underlying Kubernetes workers, storage replicas, or power/network components share a failure domain. A production interpretation would require fault-domain placement, validated restore and failover procedures, monitoring, capacity management, key recovery, and a more deliberate KRaft controller/broker topology.

For identity, the tested flow was machine-to-machine OAuth. A real product would add browser Authorization Code with PKCE, deliberate WebSocket session establishment, user/session lifecycle rules, abuse prevention, and token revocation or introspection policy where required. Token signature verification and room membership are necessary components of application security, not a complete security assessment.

If starting again, I would keep the same narrow success criteria but enforce them earlier: define the namespace contract first; generate certificates from actual Service identities; make initial IdP administration and real token issuance mandatory deployment checks; reserve primary reads for security decisions; and document the difference between an infrastructure health probe, a functional test, and a recovery drill.

The resulting proof of concept did achieve its central goal: a fresh single-namespace stack, a safe bootstrap rerun, real OAuth machine identity, authenticated cross-instance chat delivery, and demonstrated room membership revocation. The untested areas are not hidden behind green status badges. They are the next design questions, if this laboratory architecture ever needs to support real users and real availability targets.

Verification trail and reference material

The observations above are drawn from recorded lab runs and source-control milestones, rather than a third-party benchmark. Relevant public repository commits include:

6cc9bb7 (WAL-G recovery proof),
a81cc1b and e27214d (ZITADEL PoC and bootstrap correction),
e19cf69 (Rust JWT and room ACL),
9db96e9 and 0447c95 (single-namespace bootstrap),
72a068b (machine OAuth and authenticated cross-gateway delivery),
and c862064 (cross-gateway revocation).

In particular, the final authorization evidence is in kafka-chat-room-revocation-20261103T193000Z.md, and the machine OAuth E2E evidence is in kafka-chat-oidc-machine-e2e-20261103T192000Z.md. These report paths are local experiment artifacts retained locally; they are not included in the public repository or represented here as public download links.

The complete Kafka Chat source repository includes the published architecture and implementation. The public README, architecture documentation, verification matrix, JWT implementation, and Apache-2.0 license provide stable repository references. For specification and implementation context, consult the Apache Kafka KRaft documentation, Zalando PostgreSQL Operator documentation, Longhorn encryption documentation, and ZITADEL API access documentation. Repository configuration and lab reports remain the authoritative source for exact commands and installed version details.

← Back to all posts