A Dual-Gateway Kubernetes Setup for Public and Private Traffic

tldr; Put private and public traffic on different gateway instances. Give the private gateway a Tailscale-managed LoadBalancer, give the public gateway a normal cloud LoadBalancer, and make every HTTPRoute explicitly choose the gateway it is allowed to use. Use cert-manager and normal DNS so both sides still have HTTPS and friendly domain names.

A Kubernetes cluster often starts with one ingress endpoint. That is convenient until the cluster contains things that should never be reachable from the public internet:

an admin dashboard;
Argo CD;
Vault;
internal APIs;
observability tools;
staging applications; or
operational consoles for databases and queues.

The tempting solution is to expose everything through the same public ingress and add authentication later. That creates a larger blast radius. A forgotten route, a bad hostname, or a misconfigured authentication layer can turn an internal service into an internet-facing one.

A safer boundary is physical and declarative:

---
config:
  theme: base
  layout: dagre
  flowchart:
    curve: linear
    nodeSpacing: 50
    rankSpacing: 60
---
flowchart LR
    Internet(("Public<br/>Internet"))
    Tailnet(("Tailscale<br/>Tailnet"))

    subgraph Cluster["Kubernetes Cluster"]
        PublicGateway["Public Gateway<br/>Cloud LoadBalancer"]
        PublicApps["Public Applications<br/>Frontend · API"]

        PrivateGateway["Private Gateway<br/>Tailscale LoadBalancer"]
        PrivateApps["Internal Applications<br/>Argo CD · Vault · Admin"]

        PublicGateway -->|"Public HTTPRoutes"| PublicApps
        PrivateGateway -->|"Private HTTPRoutes"| PrivateApps
    end

    Internet -->|"Public DNS<br/>*.example.com"| PublicGateway
    Tailnet -->|"Private DNS + ACLs<br/>*.internal.example.com"| PrivateGateway

    classDef public fill:#f3f7ff,stroke:#315efb,stroke-width:2px,color:#111;
    classDef private fill:#f5f3ff,stroke:#7657ff,stroke-width:2px,color:#111;
    classDef external fill:#ffffff,stroke:#777777,stroke-width:1.5px,color:#111;

    class PublicGateway,PublicApps public;
    class PrivateGateway,PrivateApps private;
    class Internet,Tailnet external;`

This post shows the Kubernetes pieces behind that arrangement. The examples are sanitized. Replace the example domains, namespaces, service names, cloud annotations, and addresses with values from your own environment.

The design: two gateways, two exposure models

The Kubernetes Gateway API separates infrastructure from application routing. A GatewayClass describes the controller, a Gateway owns listeners and infrastructure, and an HTTPRoute connects a hostname and path to a Kubernetes Service.

That separation gives us a useful rule:

A route is private or public because of the Gateway it references—not because someone remembered to add a special annotation to an application.

The private gateway has these properties:

its Service is type: LoadBalancer;
its loadBalancerClass is tailscale;
it receives a stable name in the tailnet;
it is reachable only by devices allowed onto the tailnet; and
its listeners accept private hostnames such as argo.internal.example.com.

The public gateway has a different Service:

it is also type: LoadBalancer;
it does not use the Tailscale load-balancer class;
the cloud provider assigns or reserves a public IP; and
its listeners accept public hostnames such as app.example.com.

The Tailscale Kubernetes Operator documents this exposure model: a Service can be managed by Tailscale with type: LoadBalancer and loadBalancerClass: tailscale. The operator creates the tailnet-facing proxy path; it does not turn every Kubernetes Service into a public service.

Install and configure the Tailscale operator

Install the Tailscale Kubernetes Operator using your normal GitOps or Helm workflow. The operator needs credentials, but those credentials should come from a secret manager or an ExternalSecret—not from a committed manifest.

A sanitized GitOps-style application might look like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tailscale-operator
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://pkgs.tailscale.com/helmcharts
      chart: tailscale-operator
      targetRevision: <tested-version>
      helm:
        valuesObject:
          installCRDs: true
    - repoURL: https://git.example.invalid/platform/infra.git
      targetRevision: main
      path: platform/tailscale
  destination:
    server: https://kubernetes.default.svc
    namespace: tailscale
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

The secret is intentionally not shown. A safe flow is:

store the operator OAuth client data in Vault or another secret manager;
let External Secrets create the Kubernetes Secret in the operator namespace;
configure the operator to read that Secret; and
restrict the Tailscale ACL policy to the people, groups, or devices that need access.

The operator is only one layer of the boundary. Tailnet ACLs still decide who can connect to the private endpoint.

Create the private gateway

The important part of the private gateway is its Service. This is the point where the Kubernetes load-balancer request is handed to Tailscale.

apiVersion: v1
kind: Service
metadata:
  name: private-gateway
  namespace: private-gateway-system
  annotations:
    tailscale.com/hostname: private-gateway
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  selector:
    app.kubernetes.io/name: gateway-controller
  ports:
    - name: http
      port: 80
      targetPort: 80
    - name: https
      port: 443
      targetPort: 443

The annotation controls the tailnet-facing name. The exact DNS name that users type can be separate. For example, your private DNS zone could map argo.internal.example.com to the private gateway endpoint while Tailscale provides the network path and access control. In a split-horizon setup, public DNS should not publish records for these internal names.

Now define Gateway API listeners. The gateway should own the private hostnames and the private TLS certificate:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: private-gateway
  namespace: private-gateway-system
spec:
  gatewayClassName: caddy
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.internal.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: wildcard-internal-tls
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: private

Using a namespace selector is safer than allowing every namespace to attach routes. If your controller or Gateway API version requires a different cross-namespace authorization pattern, use the documented ReferenceGrant mechanism instead of widening permissions silently.

A separate HTTP listener can redirect port 80 to HTTPS. Do not treat the redirect as the security boundary; the private endpoint is private because it is reachable through Tailscale and protected by your tailnet policy.

Create the public gateway

The public gateway uses the cloud provider’s normal load-balancer integration. The provider-specific annotations vary, so keep them in a platform-owned layer. The important distinction is that this Service does not use loadBalancerClass: tailscale.

apiVersion: v1
kind: Service
metadata:
  name: public-gateway
  namespace: public-gateway-system
  annotations:
    # Example only. Use the annotations for your cloud provider.
    service.beta.kubernetes.io/cloud-load-balancer-internal: "false"
    service.beta.kubernetes.io/cloud-public-ip-name: public-gateway-ip
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app.kubernetes.io/name: gateway-controller
  ports:
    - name: http
      port: 80
      targetPort: 80
    - name: https
      port: 443
      targetPort: 443

The public Gateway owns only public names:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public-gateway
  namespace: public-gateway-system
spec:
  gatewayClassName: caddy
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: wildcard-public-tls
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: public

A wildcard listener is convenient, but it is not mandatory. Explicit listeners for app.example.com, api.example.com, and www.example.com make the intended public surface easier to review. Avoid adding internal names to this gateway even if a route would be protected by an application login.

TLS and domains still work on the private side

Private does not mean HTTP-only. The browser should still see a normal certificate for argo.internal.example.com or vault.internal.example.com.

cert-manager can request certificates from a ClusterIssuer. For private names, DNS-01 is usually the most practical ACME challenge because certificate validation does not require the service to be reachable from the internet. The DNS provider credentials used by the solver must be narrowly scoped and stored as a Secret or ExternalSecret.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-internal
  namespace: private-gateway-system
spec:
  secretName: wildcard-internal-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - "*.internal.example.com"

The cert-manager DNS-01 documentation explains the authoritative DNS challenge flow. If your internal suffix is not publicly delegated, use a certificate authority and DNS strategy that your clients trust. The key point is that network reachability and certificate issuance are separate concerns.

For the public side, use a public certificate Secret in the public gateway namespace:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-public
  namespace: public-gateway-system
spec:
  secretName: wildcard-public-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - "*.example.com"
    - "example.com"

Keep the Secret in the namespace expected by the Gateway listener unless your controller explicitly supports cross-namespace certificate references. Do not copy private keys between namespaces by hand.

Attach routes to the correct boundary

A private service should reference the private Gateway explicitly:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: admin
  namespace: admin
  labels:
    gateway-access: private
spec:
  parentRefs:
    - name: private-gateway
      namespace: private-gateway-system
      sectionName: https
  hostnames:
    - admin.internal.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: admin-dashboard
          port: 8080

The same pattern works for Argo CD, Vault, an internal API, or an operations dashboard. Each route remains in the application’s namespace, while the gateway infrastructure remains platform-owned.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: argocd
  namespace: argocd
  labels:
    gateway-access: private
spec:
  parentRefs:
    - name: private-gateway
      namespace: private-gateway-system
      sectionName: https
  hostnames:
    - argo.internal.example.com
  rules:
    - backendRefs:
        - name: argocd-server
          port: 80

A public application references the public Gateway instead:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: frontend
  namespace: frontend
  labels:
    gateway-access: public
spec:
  parentRefs:
    - name: public-gateway
      namespace: public-gateway-system
      sectionName: https
  hostnames:
    - app.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: frontend
          port: 80

The HTTPRoute reference is worth keeping nearby. parentRefs selects the Gateway, hostnames matches the request host, and backendRefs selects the Service. Those three fields make the exposure decision visible in review.

DNS: the part that makes the browser experience normal

You need two DNS paths:

Private DNS

Internal names should resolve only for clients connected to the tailnet. Depending on your setup, that may be Tailscale DNS, a private DNS zone, or split-horizon DNS. The private name should resolve to the private gateway’s Tailscale endpoint—not to the public gateway IP.

argo.internal.example.com  -> private gateway tailnet endpoint
vault.internal.example.com -> private gateway tailnet endpoint

Public DNS

Public names resolve to the public load balancer’s IP or provider hostname:

app.example.com -> public gateway IP
api.example.com -> public gateway IP

DNS is not an authorization system. A private hostname accidentally published in public DNS is still a leak of topology, and a private endpoint without Tailscale ACLs is not private enough. Treat DNS, load-balancer exposure, Gateway listeners, and tailnet policy as four separate controls.

Useful kubectl checks

The following commands are safe first checks after applying the manifests. The outputs are representative; do not copy the addresses into your environment.

Check both gateway objects and their conditions:

kubectl get gateway -A

NAMESPACE                NAME               CLASS   ADDRESS             PROGRAMMED   AGE
private-gateway-system   private-gateway    caddy   private-gateway     True          12m
public-gateway-system    public-gateway     caddy   203.0.113.20        True          12m

Check that the Services use different exposure models:

kubectl get svc -A -l app.kubernetes.io/component=gateway

NAMESPACE                NAME              TYPE           CLASS       EXTERNAL-IP
private-gateway-system   private-gateway   LoadBalancer   tailscale   100.64.0.12
public-gateway-system    public-gateway    LoadBalancer   <none>      203.0.113.20

The private address above is documentation-only. Tailscale addresses and operator-managed names depend on your tailnet configuration. The important observation is the class and the separation of Services.

Check route attachment. A route that is not accepted is not serving traffic:

kubectl get httproute -A
kubectl describe httproute -n admin admin

Look for conditions similar to:

Type:    Accepted
Status:  True
Reason:  Accepted

Type:    ResolvedRefs
Status:  True
Reason:  ResolvedRefs

Check certificates before debugging the application:

kubectl get certificate,secret -n private-gateway-system
kubectl get certificate,secret -n public-gateway-system

NAME                                              READY   SECRET                 AGE
certificate.cert-manager.io/wildcard-internal    True    wildcard-internal-tls   18m

NAME                           TYPE                DATA   AGE
secret/wildcard-internal-tls   kubernetes.io/tls   2      18m

Finally, test from both network positions. From a device on the tailnet, the private hostname should return a successful TLS response. From an ordinary external network, it should not be reachable. Conversely, the public hostname should work without tailnet access.

curl -I https://argo.internal.example.com/
# Run this from a tailnet-connected device.

curl -I https://app.example.com/
# Run this from a normal external network as well.

Do not use a successful HTTP response from one location as proof of the whole design. Test the negative path deliberately.

Common failure modes

The private hostname resolves to the public IP

The route can be perfectly configured and still be exposed through the wrong path if DNS points at the public load balancer. Check the answer from a tailnet-connected resolver and from an external resolver.

The route is attached to the wrong Gateway

A route pointing at public-gateway is public, regardless of the application’s name or namespace. Review parentRefs before adding authentication workarounds.

The Gateway is programmed but the route is not accepted

Inspect allowedRoutes, namespace labels, sectionName, and cross-namespace references. Gateway API conditions usually tell you whether the parent or backend reference was rejected.

TLS is valid but access is denied

That is often correct. TLS proves that the client reached the endpoint and verified its identity; it does not grant access. Check Tailscale ACLs, DNS visibility, and any application-level authentication separately.

A private service has another public Service

The gateway split does not protect a service that is independently exposed with type: LoadBalancer, NodePort, a cloud ingress, or a host-networked process. Inventory all external entry points.

A practical security checklist

Before calling the setup complete:

Private gateway Service uses loadBalancerClass: tailscale.
Public gateway Service is the only intended public load balancer.
Private and public Gateway listeners use different hostnames and TLS Secrets.
Private DNS records are not published in public DNS.
Tailscale ACLs restrict the private gateway to the intended users or groups.
Routes have explicit parentRefs and namespace authorization.
Secrets, OAuth data, DNS credentials, and private keys come from a secret manager.
No internal application has a second public Service or direct cloud load balancer.
Gateway, HTTPRoute, Certificate, and Service conditions are healthy.
Both positive and negative network tests have been run.

The main idea is simple: make the network boundary a first-class Kubernetes object. A public application and an internal dashboard can live in the same cluster, use the same Gateway API model, and still have very different reachability. Tailscale supplies the private path, the cloud load balancer supplies the public path, cert-manager supplies certificates, and HTTPRoute.parentRefs makes the intended boundary reviewable in Git.