Homestead Foundry

Appearance

SPIKE-28: Networking, ingress, TLS, and storage for Foundry Local on Azure Local ​

Role: foundry-researcher (Opus). Status: research spike complete. No Azure resources created, no spend, no software installed, no cluster touched, no az or kubectl command run. First-party documentation review only. Date: 2026-07-30 Scope: the networking, ingress, certificate, and storage surface of deployment track 3 (Foundry Local on Azure Local). Nine questions covering Gateway API topology, the api.exposure setting, TLS issuance, cert-manager coexistence, install-time and runtime egress, model storage, the disconnected path, multi-replica routing, and the Azure Local IP and logical-network prerequisites. Every factual claim is grounded in a first-party Microsoft Learn source, cited inline. Anything Microsoft has not published is marked UNKNOWN with the test or doc that would resolve it. This spike feeds ADR-0023; it authorizes no deployment and no spend. Depends on: docs/research/SPIKE-19-foundry-local-azure-local-deployment.md (this spike closes its UNKNOWN #6 and extends its questions 2 and 5), docs/research/SPIKE-09-azure-local-foundry.md (the original Azure Local assessment), and docs/adr/ADR-0014-foundry-local-azure-local-deployment-layers.md (the three-layer deployment shape this spike fills in). All three were read in full before any source was fetched. This spike verifies and corrects against Microsoft Learn; it does not restate them.

Headline: SPIKE-19 got the certificate story backwards, and the correction matters. Self-signed is not merely accepted, it is the default and mandatory mechanism for all internal traffic: cert-manager mints a self-signed cluster root CA on first deployment and every model sidecar presents a certificate chained to it. A real CA certificate is required only for the external LoadBalancer Gateway, and only when off-cluster clients cannot be made to trust the cluster CA. Separately, SPIKE-19 UNKNOWN #6 is now closed from documentation: the two cert-managers do not coexist, and Microsoft says so explicitly. And a prerequisite that appears in no Foundry Local document at all: exposure: external needs a working LoadBalancer implementation on an AKS Arc cluster, which does not have one by default.

Question ​

Nine questions, all new, all scoped to the networking, certificate, and storage surface that ADR-0014 deferred:

  1. What is the ingress topology, in Gateway API objects, and how does a request reach a ModelDeployment?
  2. What does api.exposure control, what are its permitted values and default, and what network exposure does each produce?
  3. What is the TLS certificate issuance path, and what must an operator supply?
  4. How does Microsoft.CertManagement coexist with an existing community cert-manager? (SPIKE-19 UNKNOWN #6)
  5. What egress does the cluster require at install time and at runtime?
  6. What storage does a model need, and is the model cache shared or per node?
  7. How does the disconnected path differ, and what must be pre-staged?
  8. What load balancing, scaling, and node-affinity behaviour applies above one replica?
  9. What IP, subnet, and logical-network prerequisites exist on the Azure Local side?

Findings ​

Q1. The ingress topology: one GatewayClass, two Gateways, one HTTPRoute per deployment, and an InferencePool only when EPP is on ​

Foundry Local does not use an Ingress controller. "Foundry Local no longer uses a Kubernetes ingress controller for external inference traffic. External access routes through the Kubernetes Gateway API." Source: Configure TLS for Foundry Local on Azure Local. The objects, with their documented default names, are:

ObjectName / valueCreated byWhen
GatewayClassistio, controller istio.io/gateway-controllerIstio (istiod) at startup, once it sees the Gateway API CRDsPrerequisite layer, before either ARM extension
Gateway (internal)inference-internal-gateway, gatewayClassName: istio, listener httpsInference operatorAt extension install
Gateway (external)inference-external-gateway, gatewayClassName: istio, serviceType: LoadBalancer, listener https, port 443Inference operator, "created on demand when any deployment uses exposure: external"Only when external exposure is requested
HTTPRoute (per model)derived from the deployment, path default /<deployment-name>, PathPrefix matching, rewritePath default /Inference operator per ModelDeploymentWhen endpoint.exposure is internal or external
HTTPRoute (control plane)serves /inference-apiInference operatorWhen api.exposure is internal or external (see Q2)
InferencePoolselects across replicas, targeted by the HTTPRoute in place of the ServiceInference operator, alongside an Endpoint Picker (EPP) podOnly for vLLM deployments where EPP is on (see Q8)
BackendTLSPolicyso "the gateway validates the backend service certificate against the Foundry Local CA bundle"Inference operatorExternal exposure
Service (ClusterIP)per deployment, always createdInference operatorAlways

Sources for the gateway names, classes, listener names, port, and the on-demand external Gateway: ModelDeployment and operator configuration reference, Inference operator configuration. For the HTTPRoute path, rewrite, and PathPrefix semantics: same page, ModelDeployment spec fields. For the GatewayClass name and controller, and the kubectl get gatewayclass istio Accepted check: Deploy Foundry Local as an Azure Arc extension. For BackendTLSPolicy and the external Gateway terminating TLS: Configure TLS. For the always-created child resources (Deployment, Service, Secret, ConfigMap, Certificate when TLS is enabled, Gateway API, StoreModel): Inference operator and model lifecycle.

The request path, end to end, for an externally exposed generative model:

  1. Client resolves the external Gateway's LoadBalancer address and sends POST https://<external-gateway-address>/<deployment-name>/v1/chat/completions with either a Bearer API key or an Entra JWT.
  2. The external Gateway (Istio, listener https, port 443) terminates TLS using either a customer-supplied secret or an auto-issued internal-CA certificate (Q3).
  3. The HTTPRoute for that deployment matches on the PathPrefix (and on endpoint.host if set), rewrites the prefix to / per endpoint.rewritePath, and forwards to its backend.
  4. The backend is the deployment's ClusterIP Service, or, for a vLLM deployment with EPP on, the InferencePool. With EPP, "the selection happens out-of-band via ExtProc gRPC" and Envoy forwards to the pod EPP names, so there is "no extra hop on the data path."
  5. Envoy connects to the pod over HTTPS, validated against the Foundry CA bundle by the BackendTLSPolicy.
  6. The pod's NGINX sidecar terminates TLS on port 443 and "forwards requests to the main container over HTTP on localhost (typically port 8001 or 5000)." The main application "only listens on localhost, so all external communication must go through the sidecar."
  7. When Entra ID authentication is enabled, the Entra Auth SDK sidecar and msi-adapter sidecar validate the JWT and evaluate ARM RBAC before the request reaches the model.

Sources: Inference operator and model lifecycle (the /v1/chat/completions curl example and Bearer token), Configure TLS (NGINX sidecar, localhost-only main container, BackendTLSPolicy), Inference runtimes (EPP, InferencePool, ExtProc), Deploy as an Azure Arc extension (Entra sidecars).

On InferenceModel: the task asked for it by name. No current Microsoft Learn page in the Foundry Local set mentions InferenceModel or its successor InferenceObjective. The only Gateway API Inference Extension object Microsoft names is InferencePool, and the only reason the Inference Extension CRDs are installed at all is stated as "so istiod picks up InferencePool support at startup." Whether the operator also creates an InferenceModel or InferenceObjective resource is not documented. Logged as UNKNOWN #1 rather than assumed either way.

The gap that matters most in this answer: the external Gateway is serviceType: LoadBalancer. An AKS Arc cluster has no LoadBalancer implementation unless one is installed. Microsoft's AKS Arc networking guidance is explicit that you must choose one: "AKS supports the following options to deploy a load balancer for your Kubernetes cluster: Deploy extension for MetalLB for Azure Arc enabled Kubernetes. Bring your own third party load balancer." Source: AKS enabled by Azure Arc network requirements. No page in the Foundry Local documentation set lists a load balancer as a prerequisite, yet exposure: external cannot produce a reachable address without one. This is a real, unstated prerequisite and ADR-0023 should record it as such.

A second, smaller trap in the same area: the operator's default external Gateway annotations are Azure cloud-provider annotations (service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path and the port 443 health probe pair), documented as pointing "the Azure LB health probe at Istio's /healthz/ready on port 15021 so the gateway's 404 on GET / does not mark backends unhealthy." Source: ModelDeployment and operator configuration reference. Those annotation keys are read by the Azure cloud controller manager. Whether they do anything at all on a MetalLB-backed AKS Arc cluster is not documented. Logged as UNKNOWN #2.

Q2. api.exposure controls the operator's own control-plane API, and nothing else ​

This is the single clearest answer in the whole spike, because Microsoft publishes the value table verbatim. api.exposure is a Helm value set at extension install time. It "controls how the Foundry Local Inference API control-plane endpoint is exposed. This value is separate from spec.endpoint.exposure, which controls individual model data-plane endpoints."

ValueDocumented behaviorResulting network exposure
internalDefault. "Routes the Inference API through the internal Gateway only." The operator "creates an HTTPRoute through the cluster-internal Gateway and the API is reachable inside the cluster at the /inference-api path."In-cluster only. No LoadBalancer, no external address. Reachable by pods and by anything that can reach the internal Gateway.
external"Routes the Inference API through both the internal Gateway and external LoadBalancer Gateway." Exposed at https://<external-gateway-address>/inference-api.Reachable off-cluster at the external Gateway's LoadBalancer address, subject to whatever the LoadBalancer implementation exposes it on. Requires the external Gateway, therefore requires a LoadBalancer.
none"Creates no HTTPRoute; the API remains reachable only through the ClusterIP service." Reachable "for example by using kubectl port-forward."ClusterIP only. Not routed through any Gateway.

Source, including the default and the three-row table: Deploy Foundry Local as an Azure Arc extension, Configure Inference API exposure.

The parallel data-plane setting is spec.endpoint.exposure on each ModelDeployment, with the same three values and the same internal default, plus a documented note that it defaults to internal "when exposure is unset or when endpoint is omitted entirely," and that the deprecated endpoint.enabled boolean "is honored only when exposure is not set." Source: ModelDeployment and operator configuration reference.

Two consequences worth carrying into ADR-0023. First, the defaults are already the safe posture: both the control plane and every model endpoint default to internal, so a deployment that sets nothing exposes nothing off-cluster. Second, external on either setting is the moment the LoadBalancer prerequisite from Q1 becomes load-bearing, and it is also the moment the TLS decision in Q3 stops being academic.

Q3. TLS: SPIKE-19 stated this backwards. Self-signed is the mandatory internal default; a real CA is an external-access requirement ​

SPIKE-19 recorded, from the requirements page, that "a real certificate-authority certificate is required and self-signed is not accepted." The requirements page does say a TLS termination certificate "signed by a company-specific certification authority (CA) or a well-known public CA" is an Azure resource prerequisite. Source: Requirements for Foundry Local on Azure Local, Azure resources. But the dedicated TLS page contradicts the blanket reading of that line, and it is the more specific document:

  • Internal TLS is self-signed, by design, and is not optional. "Foundry Local on Azure Local encrypts all internal service communication by using TLS. Each model service uses self-signed certificates that the cluster manages." On first deployment "cert-manager creates a self-signed root Certificate Authority in the Foundry Local namespace and stores it in a Kubernetes secret named root-ca-secret."
  • The issuer is a cert-manager ClusterIssuer of kind ca. Microsoft publishes the manifest: ClusterIssuer named foundry-local-ca-issuer, spec.ca.secretName: root-ca-secret. From it, cert-manager issues a wildcard Certificate (for example *.foundry-local.svc.cluster.local) into inference-service-tls-secret.
  • Rotation is automatic. "cert-manager automatically rotates service certificates before they expire, for example, renewing 30 days before a 90-day expiry. The rotation is seamless: Kubernetes updates the secret and NGINX picks up the new certificate without downtime."
  • trust-manager distributes the root. A Bundle resource publishes root-ca-secret's ca.crt as a foundry-local-ca-bundle ConfigMap "in every namespace," which is how cross-namespace callers trust the internal certificates.
  • The external Gateway also auto-issues, if you let it. "By default, if cluster TLS is enabled and you don't configure a customer certificate, the operator auto-generates a certificate signed by the internal cluster CA. Off-cluster clients must trust that CA to connect successfully."
  • The customer-supplied path is one setting and one secret. "For production external access, provide a customer-managed TLS secret for the external gateway by setting operatorConfig.networking.externalGateway.tls.secretName during extension installation or Helm configuration. The secret must be a Kubernetes TLS secret in the external gateway namespace." The operator config field is networking.externalGateway.tls.secretName, default "", documented as "Empty means auto-issue from the internal CA when global TLS is enabled."

Sources: Configure TLS for Foundry Local on Azure Local for all of the above except the config-field default, which is in ModelDeployment and operator configuration reference.

The corrected statement, which ADR-0023 should adopt:

TrafficCertificateWho supplies itSelf-signed acceptable?
Pod to pod, service to service, cross-namespace in-clusterIssued by foundry-local-ca-issuer from the self-signed root-ca-secret rootcert-manager, automatically, on first deploymentYes. Mandatory. There is no other option documented.
Gateway to model pod backendSame internal CA, validated by a BackendTLSPolicyOperator, automaticallyYes
External Gateway to off-cluster client, exposure: externalEither auto-issued from the internal CA, or a customer-supplied kubernetes.io/tls secretOperator by default; the deployer for the customer-managed pathWorks, but every off-cluster client must be made to trust the cluster CA. Microsoft's production guidance is a company CA or well-known public CA.

Can azure-cert-manager obtain a real certificate automatically? Yes in principle, and this is a genuine capability the Foundry Local docs never connect up. The Arc cert-manager extension "supports both self-signed CA certificates and external enterprise CA certificates. You can let it create a self-signed cluster-local CA certificate for internal certificates, or configure it to request certificates from your organization's existing PKI (for example, an external corporate CA or an ACME provider). In addition, you can create a cluster-local issuer that is either a root self-signed issuer or an intermediate issuer that chains to a key and certificate obtained from your enterprise PKI, which can be useful for air-gapped or offline certificate issuance scenarios." The deploy guide names the issuer types available: "a self-signed CA, an account registered with an Automated Certificate Management Environment (ACME) CA server such as Let's Encrypt (or others), or a CA whose certificate and private key are stored inside the cluster as a Kubernetes Secret." Sources: Cert-manager for Arc-enabled Kubernetes (preview), Deploy cert-manager for Arc-enabled Kubernetes (preview).

So the automatable production pattern is: create an Issuer or ClusterIssuer (ACME, or a CA issuer backed by the enterprise PKI key pair held in a Kubernetes Secret), create a Certificate that writes into a kubernetes.io/tls secret in the external gateway namespace, and set operatorConfig.networking.externalGateway.tls.secretName to that secret. Each of the three steps is first-party documented. What is not documented is the three of them wired together, and specifically whether the operator tolerates the secret not existing at install time and picking it up when cert-manager creates it. Logged as UNKNOWN #3.

What an operator must supply, concretely:

  • Nothing, for a first increment that stays at exposure: internal everywhere. cert-manager mints the whole chain.
  • For external access: a DNS name for the external Gateway address, an issuer of some kind (ACME account details, or an enterprise CA key and certificate loaded into a Kubernetes Secret), and either a pre-created kubernetes.io/tls secret name or a cert-manager Certificate that produces one. The ACME path additionally implies HTTP-01 or DNS-01 solver reachability, which no Microsoft page in this set discusses for an on-premises Azure Local gateway. That is a second face of UNKNOWN #3.
  • One more consequence worth naming: because ACME certificates are typically 90 days and cert-manager's documented default duration is 2160h with renewBefore: 360h, the external certificate is an ongoing rotation obligation, not a one-time procurement. Source for the durations: Deploy cert-manager for Arc-enabled Kubernetes.

One naming detail an implementer will trip over: the Foundry Local install command names the extension instance azure-cert-manager with --extension-type Microsoft.CertManagement, while the Arc cert-manager article names it azure-cert-management with --extension-type "microsoft.certmanagement". The extension type is the same (Kubernetes extension types are case-insensitive in practice); the instance name differs between the two documents. Track 3 automation should use the Foundry Local page's name, azure-cert-manager, since that is the page that also passes the five --config flags Foundry Local needs. Sources: Deploy Foundry Local as an Azure Arc extension, Step 2, Deploy cert-manager for Arc-enabled Kubernetes.

Those five flags are themselves load-bearing and are not the extension's defaults: config.enableGatewayAPI=true, cert-manager.crds.keep=true, trust-manager.defaultPackage.enabled=false, trust-manager.secretTargets.enabled=true, trust-manager.secretTargets.authorizedSecretsAll=true. An azure-cert-manager installed by the generic Arc article's command will not have them. Source: Deploy Foundry Local as an Azure Arc extension, Step 2.

Q4. SPIKE-19 UNKNOWN #6, resolved: they do not coexist. Uninstall the community one first ​

This is now answerable directly from a first-party page, and the answer is unambiguous.

"If you previously installed open source cert-manager or trust-manager manually, uninstall them before deploying the Arc extension to avoid conflicts. Existing custom resources (CRs) will be retained and recognized by the extension, as long as they adhere to the conventions of the open-source projects."

And, under a dedicated migration heading:

"The cert-manager for Azure Arc-enabled Kubernetes extension serves as a replacement for both open source cert-manager and trust-manager. Before you install cert-manager for Arc-enabled Kubernetes, you must uninstall the open source resources."

Source: Deploy cert-manager for Arc-enabled Kubernetes (preview), Prerequisites and Migrate from open source cert-manager and trust-manager.

The migration procedure and its cost, verbatim from the same page:

  • Discover where they are: helm list -A | grep -E 'trust-manager|cert-manager'.
  • Uninstall: helm uninstall cert-manager -n "your namespace" --ignore-not-found and the same for trust-manager. For non-Helm installs, follow the upstream uninstall docs.
  • There is a security gap during the swap, and Microsoft flags it as a warning: "During the time between uninstalling the OS version and installing the Arc extension, certificate rotation doesn't occur and trust bundles aren't distributed to new namespaces. To minimize potential security risks, ensure that this transition period is as short as possible."
  • Nothing you created is destroyed: "Uninstalling the open source cert-manager and trust-manager doesn't remove any existing certificates or related resources you created. These resources remain accessible and usable after the cert-manager for Arc-enabled Kubernetes extension is installed."
  • Removal is symmetric and also non-destructive: az k8s-extension delete "removes the cert-manager and trust-manager deployments. It doesn't remove secrets and configmaps for trust bundles, but it does remove the controllers that keep them updated."

The Foundry Local TLS page reinforces the same direction of travel for this specific product: "For Arc-enabled Kubernetes clusters, use cert-manager for Arc-enabled Kubernetes (CME) as the supported installation path. Generic open-source cert-manager is only required when you deploy Foundry Local by using Helm without the Arc extension." Source: Configure TLS for Foundry Local on Azure Local.

Net on Q4, and this closes SPIKE-19 UNKNOWN #6: there is no supported coexistence. The two installations manage the same CRDs and the same controller responsibilities, and Microsoft's word is "conflicts." The correct sequence for a cluster that already runs community cert-manager is: inventory with helm list -A, uninstall both charts, install azure-cert-manager with the five Foundry Local --config flags, verify, and keep the window short. Existing Issuer, ClusterIssuer, Certificate, and Bundle resources survive and are picked up, so this is a controller swap rather than a certificate re-issue.

Two caveats an implementer needs, and neither is covered by the above. First, cert-manager.crds.keep=true is passed by the Foundry Local install; the interaction between that flag and CRDs left behind by an uninstalled community chart is not documented. Second, the Arc extension's validated distribution list reads "Azure Local version 2602 + Kubernetes versions 1.31.13, 1.32.9, 1.33.5" and "Azure Kubernetes Service (AKS) enabled by Azure Arc: v1.32.7", while Foundry Local's own minimum is Kubernetes 1.29. A cluster at 1.29 or 1.30 satisfies Foundry Local and falls outside the cert-manager extension's validated set, where Microsoft warns "some functionality may not work as expected if your cluster is running a distribution that hasn't been validated." Sources: Cert-manager for Arc-enabled Kubernetes, Validated Arc-enabled Kubernetes distributions, Requirements for Foundry Local on Azure Local, Software requirements. Logged as UNKNOWN #4. The practical mitigation is cheap: build the cluster at 1.32 or later and both constraints are satisfied at once.

Third caveat, regional: the cert-manager extension's supported region list and Foundry Local's supported region list are different lists. Both include East US, the region ADR-0014 selected, so track 3 is unaffected. Sources: Cert-manager for Arc-enabled Kubernetes, Regional support, What is Foundry Local on Azure Local?, Supported regions.

Q5. Egress: four categories, three of them fully documented and one with a real hole ​

Microsoft's own routing for this question is a pointer: "For Azure Local deployments, follow the current Azure Local and AKS on Azure Local network requirements." Source: Requirements for Foundry Local on Azure Local, Network requirements. So the answer assembles from four first-party sources.

Category 1: the Arc agents and the Kubernetes extension control plane. All outbound TCP on port 443 unless noted, all HTTPS. Source for the whole table: Azure Arc-enabled Kubernetes network requirements.

EndpointPurpose
https://management.azure.comAgent connects to Azure and registers the cluster
https://<region>.dp.kubernetesconfiguration.azure.comData plane for the agent to push status and fetch configuration. This is the endpoint the extension install itself rides on.
https://gbl.his.arc.azure.com and https://*.his.arc.azure.comRegional endpoint discovery and pulling system-assigned managed identity certificates
guestnotificationservice.azure.com, *.guestnotificationservice.azure.com, sts.windows.net, *.servicebus.windows.netCluster Connect and Custom Location scenarios. Websockets must be enabled for *.servicebus.windows.net.
*.arc.azure.netManaging connected clusters in the Azure portal
https://<region>.obo.arc.azure.com:8084/Cluster Connect with Azure RBAC. Note the non-443 port.
dl.k8s.iokubectl download during onboarding by the connectedk8s CLI extension

Category 2: extension artifacts (container images and charts).

EndpointPurpose
https://mcr.microsoft.com, https://*.data.mcr.microsoft.comPull container images for the Arc agents. This is also where the operator's default EPP image lives (mcr.microsoft.com/oss/v2/gateway-api-inference-extension/epp).
https://linuxgeneva-microsoft.azurecr.io"Required if using Azure Arc-enabled Kubernetes extensions." Both Microsoft.CertManagement and Microsoft.Foundry are Arc extensions, so this is in scope.
gcs.prod.monitoring.core.windows.net, *.prod.microsoftmetrics.com, *.prod.hot.ingest.monitor.core.windows.net, *.prod.warm.ingest.monitor.core.windows.netExplicitly listed as "Required if using ... cert-manager for Azure Arc-enabled Kubernetes." Telemetry can be reduced with global.telemetry.logs.enabled=false and global.telemetry.metrics.enabled=false.

Sources: Azure Arc-enabled Kubernetes network requirements; the EPP image path from ModelDeployment and operator configuration reference; the telemetry flags from Deploy cert-manager for Arc-enabled Kubernetes, Restricted PSA environments.

Category 2b: the prerequisite layer, which is not a Microsoft endpoint at all. ADR-0014's prerequisite layer downloads from third parties, and this is visible directly in Microsoft's own install commands: https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml, https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/manifests.yaml, and the Istio Helm repository https://istio-release.storage.googleapis.com/charts. Source: Deploy Foundry Local as an Azure Arc extension, Step 1. Two of those are github.com and one is storage.googleapis.com, which many enterprise firewalls do not allow by default and which no Microsoft firewall table lists. The container registry the Istio images are pulled from is not named on any page in this set; it is a property of the upstream chart, not of Microsoft's documentation. Logged as UNKNOWN #5.

Category 3: the model download path. This is the hole. The catalog metadata path is described but not addressed: catalog-sync "fetches model metadata from the Azure AI Foundry catalog API" on a Helm post-install hook and then daily via a CronJob. The artifact path is described as "Pulled from the online Foundry catalog." Source: Inference operator and model lifecycle, How catalog sync works, Requirements, Requirements by environment. No FQDN, no port, and no registry hostname is published for either the catalog API or the model artifact pull. For a firewall operator this is the one category that cannot be turned into a rule from documentation. Logged as UNKNOWN #6, and it is the highest-value unknown in this spike, because a locked-down firewall will fail the StoreModel cache job with no way to pre-approve the endpoint.

Category 4: Entra token acquisition. https://login.microsoftonline.com, https://<region>.login.microsoft.com, and login.windows.net, "required to fetch and update Azure Resource Manager tokens," plus https://graph.microsoft.com/ "required when Azure RBAC is configured." Since ADR-0014 decision 5 enables Entra ID authentication and its authorization "is then evaluated by using Azure role-based access control," both are in scope. Source: Azure Arc-enabled Kubernetes network requirements.

Three platform-level rules that constrain how any of this can be filtered, all from Firewall requirements for Azure Local:

  1. HTTPS inspection is not supported. "Azure Local doesn't support HTTPS inspection. Make sure that HTTPS inspection is disabled along your networking path for Azure Local to prevent any connectivity errors. This includes use of Entra ID tenant restrictions v1."
  2. Arc endpoints must resolve to public IPs. "Azure Arc Private Link Scopes are not supported by Azure Local. Arc endpoints (*.his.arc.azure.com, *.guestconfiguration.azure.com and *.dp.kubernetesconfiguration.azure.com) must always resolve to public IPs from Azure Local nodes, ARB VM and the proxy server if in use."
  3. Ports 80 and 443 outbound are the baseline: "Opening ports 80 and 443 for outbound network traffic on your organization's firewall meets the connectivity requirements."

Microsoft also publishes consolidated per-region endpoint lists covering Azure Local, Arc-enabled servers, Azure Resource Bridge, and AKS together, including one for East US. That list, not this spike's tables, is the artifact to hand a firewall team. Source: the region links on Firewall requirements for Azure Local.

One final proxy warning that belongs in any deployment runbook: "You cannot update incorrect proxy settings after you deploy Azure Local. If the proxy is misconfigured, you must redeploy Azure Local." And AKS Arc cannot override it: "Proxy settings in AKS are inherited from the underlying infrastructure system. The functionality to set individual proxy settings for Kubernetes clusters and change proxy settings isn't supported yet." Source: AKS enabled by Azure Arc network requirements.

Q6. Storage: 100 GiB per model cache PVC by default, and storeModel.cacheJob.resources is about memory, not disk ​

The documented facts.

  • The model cache is a PVC and it defaults to 100 GiB. "Plan your cluster storage around the models you deploy. The model cache persistent volume claim (PVC) defaults to 100 GiB, which is enough for most models. To deploy large models such as magistral, allocate more storage by setting spec.vllm.modelCacheStorageGi on the deployment." Source: Requirements for Foundry Local on Azure Local.
  • The knob is vLLM-only. vllm.modelCacheStorageGi, integer, default 100, minimum 1, "Size of the model cache PVC in GiB. Increase this value for large models that exceed the 100-GiB default." And explicitly: "This field applies only to deployments that use runtime: vllm." Source: ModelDeployment and operator configuration reference, Configure model cache storage.
  • storeModel.cacheJob.resources controls the cache Job's memory, not its disk. "The StoreModel cache job is configured with default memory values of 16Gi for requests and 32Gi for limits. These settings are established due to the typically large size of models, as downloading and caching them requires substantial memory to prevent out-of-memory (OOM) issues." Tunable at install with --config storeModel.cacheJob.resources.requests.memory=... and .limits.memory=..., with the warning "Ensure that the requests value doesn't exceed the limits value." Source: Deploy Foundry Local as an Azure Arc extension, Additional installation parameters. Note the trap SPIKE-19 already flagged and this confirms: a 32 GiB limit against a 32 GiB recommended node.
  • Shared or per node? Shared at the registry, materialized per pod. The cache is "the cluster's local OCI registry," a cluster-scoped component, and StoreModel "tracks whether model artifacts are downloaded and cached" in it, with status.storeRef holding the local OCI path. Crucially, the name is content-addressed rather than node-addressed: "It generates a deterministic StoreModel name from the model source, alias, compute type, framework, and version," then "checks if a StoreModel CR with that name already exists" and reuses it if Available. Delivery to each pod is per pod, not per node: "The inference pods use an init container (the model-store-retriever) that pulls model files from the local registry into the pod's filesystem before the main inference container starts." Source: Model caching and StoreModel lifecycle.

So the answer to "shared across nodes or per node" is: the download is done once per model per cluster (shared, cluster-scoped OCI registry), and the materialisation into a running pod happens once per pod (per pod, not even per node) via an init container. A second replica on a second node does not re-download from the internet; it re-pulls from the in-cluster registry. That is the important operational property, and it is documented.

What is not documented, and it is a lot. For a spike whose job is to tell an operator what to provision, this section has the thinnest first-party coverage of the nine:

  • Which StorageClass the operator requests is not stated anywhere. AKS Arc has a default StorageClass "automatically created" that "uses CSV to create VHDX-backed volumes," is expandable, and deletes the VHDX on PV deletion. Whether the operator asks for default, asks for nothing (and so gets default), or accepts an override is unpublished. Source for the AKS Arc default: Storage options for applications in AKS enabled by Azure Arc. Logged as UNKNOWN #7.
  • The access mode is not stated. This matters more than it looks. On AKS Arc, "AKS disk volumes backed by VHDX ... are mounted as ReadWriteOnce and are accessible to a single node at a time," whereas ReadWriteMany requires "AKS file volumes backed by SMB or NFS file shares." Source: same page. If the model cache PVC is ReadWriteOnce on the default class, a multi-replica deployment spread across nodes has an obvious constraint, and Microsoft's multi-node page discusses replica scheduling without ever mentioning the storage implication. Logged as UNKNOWN #8.
  • The ONNX path has no documented storage knob at all. modelCacheStorageGi is vLLM-only, yet the requirements page states the 100 GiB PVC default as a general planning fact. Whether an onnx-genai CPU deployment (ADR-0014's first increment) provisions a 100 GiB PVC with no way to shrink it is unpublished. On a small proof-of-concept cluster, an unshrinkable 100 GiB claim per model for a roughly 5 to 8 GB model is a meaningful over-allocation. Logged as UNKNOWN #9, and it is the storage question most likely to bite the first increment.
  • The local OCI registry's own backing storage is undescribed. storeModel configuration is summarised as "Local registry URL, cache job timeout, and poll interval" with no capacity or volume field. Source: Inference operator and model lifecycle, Configuration. Something has to hold every cached model in the cluster and its size is not a documented input. Logged as UNKNOWN #10.

For sizing the catalog side of the equation, the operator does publish real sizes: Phi-4-generic-cpu at 8.13 GB and qwen2.5-coder-0.5b-instruct-cpu at 0.43 GB, readable live from the catalog ConfigMap. Source: Inference operator and model lifecycle, Query the catalog.

Q7. The disconnected path: same shape, different source, and three real behavioural differences ​

Microsoft's framing is that the architecture is unchanged: "Foundry Local on Azure Local in disconnected environments uses the same Arc-enabled Kubernetes cluster and operator-based control plane as connected deployments. The key difference is that catalog model artifacts and extension components are imported into the disconnected environment through locally installed expansion packs, rather than pulled from internet-connected registries." Source: Foundry Local on Azure Local in disconnected environments overview. That confirms ADR-0014's claim that the three-layer model holds for both paths.

What must be pre-staged, and the exact mechanism. Download the expansion pack in a connected environment (https://aka.ms/azurelocal-pxp-microsoft-foundrylocal-k8sextension), transfer it, and install it on the Azure Local Disconnected Operations machine with the published PowerShell:

Import-Module "<PATH_TO_ALDO_MODULES>\Azure.Local.ExpansionPack.psm1"
$expansionPackId = Start-AldoExpansionPackUpload -ExpansionPackPath "<PATH_TO_EXPANSION_PACK>"
$result = Start-AldoExpansionPackInstallation -ExpansionPackId $expansionPackId -Wait

On completion: "Container images are imported into the edgeartifacts registry. Model artifacts are published to the registry. The Foundry Local Azure Arc extension becomes available for installation." Verify with Get-ApplianceExpansionPackDetails and confirm the pack is Installed. Source: Prepare to Deploy Foundry Local on Azure Local in a Disconnected Environment.

How artifacts are sourced. The expansion pack supplies both the ARM layer and the whole prerequisite layer: "the Foundry Local expansion pack provides the networking dependencies that connected deployments pull from online sources. During expansion pack installation, these assets are imported into edgeartifacts, including Istio control plane components (istio-base, istiod), Kubernetes Gateway API custom resource definitions (CRDs), Gateway API Inference Extension CRDs (including InferencePool), and the Endpoint Picker (EPP) container image." Consequently "deployment doesn't require outbound internet connectivity to install those networking components." For models, "a cache job pulls model artifacts from the local EdgeArtifacts container registry instead of fetching from the Foundry cloud catalog." BYO models come "from a customer-managed OCI-compatible container registry within the disconnected environment." Sources: Disconnected environments overview, Deploy Foundry Local as an Azure Arc extension, Step 1.

Three differences that are not just "same thing, local source", and each one invalidates a piece of ADR-0014 if disconnected is ever chosen:

  1. azure-cert-manager does not exist in disconnected. "The azure-cert-manager extension isn't available in disconnected environments. Instead, you must install: cert-manager, trust-manager. These Helm charts and container images are included in the Foundry Local expansion pack." So the disconnected path is the one case where community-style cert-manager is the correct choice, exactly inverting Q4's answer. ADR-0014's platform layer shrinks from two ARM extensions to one, and the prerequisite layer grows.
  2. Entra ID authentication is not used. "Authentication doesn't use public Microsoft Entra ID endpoints. Instead, Foundry Local integrates with the Active Directory infrastructure configured in the disconnected Azure Local environment." Authorization instead uses "standard Azure RBAC roles on the Foundry extension resource": Reader for read-only catalog operations, Contributor for control-plane writes and for data-plane inference including predict and chat/completions. Microsoft explicitly contrasts this: "This authorization model differs from connected deployments, which typically use roles such as Cognitive Services OpenAI User." ADR-0014 decision 5 chose Entra ID authentication; that decision simply does not transfer to a disconnected cluster.
  3. Telemetry stops, and so does the normal diagnostic path. "Telemetry isn't transmitted to Microsoft. To collect diagnostics for support, use the az k8s-extension troubleshoot command."

Sources for all three: Disconnected environments overview.

Two disconnected-specific prerequisites in the networking and GPU space that the connected requirements page does not have. The disconnected prerequisite table adds a row the connected one lacks: "Logical network: Reachable from edgeartifacts ACR." And for GPU, "nvidia/k8s-device-plugin:v0.11.0 is mirrored into the edgeartifacts container registry at the path expected by the auto-deployed DaemonSet," plus "NVIDIA mitigation INF is installed on the physical Azure Local node." Source: Prepare to Deploy Foundry Local in a Disconnected Environment. Minimum ALDO version remains 2604.3.0, confirming SPIKE-19.

Q8. Above one replica: Kubernetes scheduling for everything, model-aware routing only for vLLM ​

Replica count and scheduling. replicas is an integer, default 1, range 1 to 100. Placement uses ordinary Kubernetes mechanisms: "Kubernetes schedules workloads to nodes that meet their CPU, memory, and GPU requirements by using standard controls such as resource requests and limits, node selectors, and affinity rules. This architecture supports heterogeneous clusters where some nodes are CPU-only and others are GPU-capable." The ModelDeployment spec exposes nodeSelector (object) and tolerations (array) directly, plus skipGpuResource which "requires nodeSelector when set to true." Sources: ModelDeployment and operator configuration reference, Multi-Node Kubernetes Deployment. Status surfaces replicas.desired, replicas.ready, replicas.available, and deploymentReady when all replicas are ready.

Does the Inference Extension do model-aware routing? Yes, but only for vLLM, and vLLM is GPU-only. This is the important qualification, because ADR-0014's first increment is CPU-only onnx-genai.

  • "For multireplica vLLM deployments, Foundry Local routes requests by using the Gateway API Inference Extension instead of the Gateway's default round-robin. The operator deploys an Endpoint Picker (EPP) alongside the model and binds it to an InferencePool that selects across the replicas."
  • EPP scores each replica on three signals scraped from vLLM: "Queue depth ... KV-cache utilization ... Prefix-cache locality ... EPP picks the highest-scoring pod and instructs Envoy to forward there. The selection happens out-of-band via ExtProc gRPC, there's no extra hop on the data path."
  • The default is replica-count driven and re-evaluates on scaling. With spec.vllm.epp.enabled unset: off at replicas: 1, on at replicas > 1. An explicit value always wins. "The default re-evaluates on scaling: a deployment created at one replica with the field unset brings up the EPP stack the moment it is scaled to two or more, and tears it down again on scale-back."
  • It has three hard dependencies, all in ADR-0014's prerequisite layer: runtime: vllm, the Gateway API Inference Extension CRDs, and "Istio configured with pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true."
  • Microsoft publishes measured numbers on a 3-replica vLLM deployment with output capped at 256 tokens: single-turn at 20 concurrent users, "average TTFT is 14% lower (646 ms vs 753 ms), inter-token latency is 5% lower, and per-user throughput is 6% higher"; multi-turn 3-turn chat, "throughput is 16% higher at one concurrent user and 10% higher at five; inter-token latency is 14% lower; aggregate throughput is 14% higher at five concurrent users."
  • Microsoft also names when to turn it off: "Force enabled: false for throughput-maximizing batch workloads that generate very long, unbounded outputs at high concurrency, where the per-token ExtProc overhead can outweigh the routing benefit."
  • Capacity cost: "one Endpoint Picker (EPP) pod per ModelDeployment (about 512 MiB request and 2 GiB limit)," with operator defaults of 250m CPU / 512Mi memory requests and 2 CPU / 2Gi limits, replicas: 1, target port 8443.

Sources: Inference runtimes in Foundry Local, Inference-aware routing with the Endpoint Picker (EPP), Multi-Node Kubernetes Deployment, Requirements, ModelDeployment and operator configuration reference.

The consequence ADR-0023 must record: a multi-replica onnx-genai deployment gets plain Gateway round-robin. EPP requires runtime: vllm, and vLLM requires compute: gpu. There is no documented inference-aware routing for the CPU path. Scaling the first increment horizontally therefore buys concurrency but no cache locality and no queue-depth awareness, which is precisely the property that makes multi-turn chat efficient. That is an argument for a GPU increment later, stated honestly, rather than an argument against the CPU increment now.

Not documented anywhere in this set: autoscaling. replicas is a static integer. No HorizontalPodAutoscaler support, no scale-to-zero, no queue-based autoscaling is described. Logged as UNKNOWN #11. Also not documented: any anti-affinity default that would spread replicas across nodes rather than stacking them. The multi-node page says Kubernetes uses "affinity rules" but shows none, and the ModelDeployment spec exposes nodeSelector and tolerations but no affinity field. Logged as UNKNOWN #12.

Q9. The Azure Local side: static IPs only, one logical network, three separate IP budgets ​

Before an AKS Arc cluster can exist at all, three distinct pools of IP addresses must be planned, and Microsoft is explicit that mixing them causes failures that are hard to diagnose.

1. A logical network for the node VMs. "AKS Arc uses Azure Local logical networks to provide IP addresses and networking for the underlying VMs of the Kubernetes clusters. You must plan to reserve one IP address per AKS cluster node VM." Critically: "Static IP is the only supported mode for assigning an IP address to AKS Arc VMs. This is because Kubernetes requires the IP address assigned to a Kubernetes node to be constant throughout the lifecycle of the Kubernetes cluster. Software defined virtual networks and SDN related features are currently not supported on AKS on Azure Local."

The required logical-network parameters, as published: --address-prefixes ("Currently only 1 address prefix is supported"), --dns-servers, --gateway ("must be within the scope of the address prefix"), --ip-allocation-method ("Supported values are Static"), and --vm-switch-name. Two further parameters are "Optional, but highly recommended": --ip-pool-start and --ip-pool-end, because "If you use MetalLB or any other third party load balancer in L2/ARP mode, we highly recommend using IP pools to separate AKS Arc IP requirements from load balancer IPs. This recommendation is to help avoid IP address conflicts that can lead to unintended and hard-to-diagnose failures."

2. A control plane IP, one per Kubernetes cluster. "AKS enabled by Arc deploys the KubeVIP load balancer to ensure that the API server IP address of the Kubernetes control plane is always available. This KubeVIP instance requires a single immutable control plane IP address." AKS Arc picks one from the logical network automatically, or you pass one, in which case "the control plane IP must be within the scope of the address prefix of the logical network. You must ensure that the control plane IP address does not overlap with anything else, including Arc VM logical networks, infrastructure network IPs, load balancers, etc. ... You must plan to reserve one IP address per Kubernetes cluster."

3. A load balancer IP pool, for the external Gateway. As covered in Q1, either the MetalLB Arc extension or a third-party load balancer, and "you must provide a set of IP addresses to the load balancer service," either from the same subnet as the AKS Arc VMs or from a different network. "Regardless of the option you choose, you must ensure that the IP addresses allocated to the load balancer don't conflict with the IP addresses in the logical network."

4. And underneath all of it, the Azure Local management allocation. "When you deploy Azure Local, you allocate a contiguous block of at least six static IP addresses on your management network's subnet, omitting addresses already used by the physical machines. These IPs are used by Azure Local and internal infrastructure (Arc Resource Bridge) for Arc VM management and AKS Arc."

DNS. "You need to ensure that the DNS server of the logical network can resolve the FQDN of the Azure Local cluster. DNS name resolution is required for all Azure Local nodes to be able to communicate with the AKS VM nodes."

Cross-VLAN ports, if the management network and the AKS logical network are on different VLANs. All bidirectional:

Destination portDestinationSourcePurpose
22Logical network used for AKS Arc VMsManagement networkLog collection for troubleshooting
6443Logical network used for AKS Arc VMsManagement networkCommunicate with Kubernetes APIs
55000Cluster IP addressAKS Arc VM logical networkCloud Agent gRPC server
65000Cluster IP addressAKS Arc VM logical networkCloud Agent gRPC authentication
40343Cluster IP addressAKS Arc VM logical networkOnly when the Azure Local cluster is configured with Arc gateway for outbound connectivity

All of Q9's citations: AKS enabled by Azure Arc network requirements. The six-address management block cross-references Deploy Azure Local via the Azure portal, Specify network settings. The internal cluster-node ports (ICMP, SMB 445, SMB Direct 5445 for iWARP RDMA, WS-MAN 5985, Azure Local service 30301) are in Firewall requirements for Azure Local.

Net on Q9: the minimum IP budget for ADR-0014's first increment is: 6 or more contiguous static management-network addresses (already consumed by the existing Azure Local deployment), 1 per AKS Arc node VM (control plane VMs plus workers), 1 control plane VIP per Kubernetes cluster, and, only if exposure: external is used, a non-overlapping load balancer pool. No real addresses appear in this spike by design; the numbers above are counts, not addresses.

What is still UNKNOWN ​

#UnknownWhy it is not in the docsWhat resolves it
1Whether the operator creates an InferenceModel or InferenceObjective resource alongside InferencePool.Microsoft names only InferencePool and installs the Inference Extension CRDs "so istiod picks up InferencePool support." The other Inference Extension kinds are never mentioned.kubectl api-resources --api-group=inference.networking.x-k8s.io on a cluster with the CRDs installed, then kubectl get inferencepool,inferencemodel,inferenceobjective -A after creating a multi-replica vLLM deployment. Read-only.
2Whether the operator's default external-Gateway annotations do anything on a MetalLB-backed AKS Arc cluster.The defaults are Azure cloud-provider annotations (service.beta.kubernetes.io/azure-load-balancer-*), read by the Azure cloud controller manager, which is not what an on-premises AKS Arc cluster runs.Deploy with exposure: external on a MetalLB cluster and inspect the resulting Service and its health-probe behavior; or override networking.externalGateway.annotations at install with MetalLB-appropriate values.
3Whether a cert-manager-issued certificate (ACME or enterprise CA) can be wired to externalGateway.tls.secretName end to end, and whether the secret may be absent at install time.Each of the three pieces is documented on a different page; no page joins them. The ACME solver reachability question for an on-premises gateway is not discussed at all.Create a ClusterIssuer plus Certificate producing a kubernetes.io/tls secret, install the extension pointing at that secret name, and confirm the external Gateway listener serves the resulting chain. Also test install-before-secret ordering.
4Whether the Microsoft.CertManagement extension is supported on Kubernetes 1.29 or 1.30, the low end of Foundry Local's stated minimum.The cert-manager extension's validated list starts at AKS Arc v1.32.7 and Azure Local 2602 with 1.31.13 / 1.32.9 / 1.33.5; Foundry Local's minimum is 1.29. The two documents were written independently.Build the cluster at 1.32 or later, which satisfies both, and the question does not need answering. Otherwise, a support question to Microsoft.
5The container registry hostnames the Istio charts pull images from.The prerequisite layer is upstream Istio, and Microsoft documents only the Helm repository URL (istio-release.storage.googleapis.com), not the image registry the chart resolves to.Read the pinned Istio chart's values.yaml global.hub before the firewall change, or run helm template offline and collect the image references.
6The FQDNs and ports for the Foundry catalog API and the model artifact pull. This is the highest-value unknown in this spike.Documented only as "the Azure AI Foundry catalog API" and "Pulled from the online Foundry catalog." No hostname is published on any page in the set, and the Azure Local firewall tables do not carry a Foundry Local row.Ask Microsoft as part of the preview access onboarding, or observe the catalog-sync CronJob and the StoreModel cache Job egress on a permissive-firewall cluster before locking the firewall down. Until then a locked-down firewall will fail the cache job with no way to pre-approve the endpoint.
7Which StorageClass the model cache PVC requests.Neither the requirements page nor the operator configuration reference names a storage class field.kubectl get pvc -n foundry-local-operator -o yaml after a first deployment, and check whether an operator config override exists in the installed Helm values.
8The access mode of the model cache PVC (ReadWriteOnce or ReadWriteMany).Never stated. It matters because AKS Arc's default class is VHDX-backed and ReadWriteOnce, "accessible to a single node at a time," while ReadWriteMany needs SMB or NFS file volumes.Same inspection as #7. If ReadWriteOnce, confirm what actually happens to a multi-replica deployment scheduled across two nodes.
9Whether an onnx-genai deployment provisions the 100 GiB cache PVC, and whether it can be sized down.modelCacheStorageGi is documented as vLLM-only, yet the 100 GiB PVC default is stated as general planning guidance. The ONNX path has no documented knob.Deploy one CPU catalog model and inspect the PVC. If a 100 GiB claim is created with no override, that is a hard sizing input for the first increment on a small cluster.
10The backing storage and capacity planning input for the cluster-local OCI registry that holds every cached model.storeModel configuration is summarised as registry URL, cache job timeout, and poll interval. No volume or capacity field appears.Inspect the registry component's own PVC after install, and multiply catalog model sizes (readable from the catalog ConfigMap, for example 8.13 GB for Phi-4-generic-cpu) by the number of models intended.
11Whether any autoscaling is supported (HPA, scale to zero, queue-based).replicas is a static integer 1 to 100 and no autoscaling mechanism is described anywhere in the set.Test a standard HorizontalPodAutoscaler against the operator-owned Deployment and observe whether the operator's 30-second reconciliation loop reverts the replica count. This is a real risk: an operator that reconciles replicas from the CR will fight an HPA.
12Whether replicas are spread across nodes by default, or may stack on one node.The multi-node page mentions "affinity rules" generically; the ModelDeployment spec exposes nodeSelector and tolerations but no affinity field.Deploy at replicas: 2 on a 2-node cluster and check placement. If they stack, HA requires topology spread constraints the CRD does not expose.
13Real CPU inference latency for a roughly 5 to 8 GB model on a Standard_D8s_v3 worker. Carried unchanged from SPIKE-19 UNKNOWN #7.No latency or throughput table is published for CPU-backed deployments. Microsoft's only published numbers (Q8) are vLLM on GPU.Measure after the first increment deploys.

Carried forward without change from SPIKE-19 and still open: its UNKNOWN #1 (a vision-capable catalog model), #2 (preview to GA timeline and SLA), #3 (Azure Local per-core price), #4 (whether the owner's Azure Local release supports the GPU SKUs in scope), #5 (whether the AKS Arc cluster is Bicep-expressible), #6a (does an AKS Arc cluster exist in the target environment), and #6b (is supported NVIDIA hardware present). SPIKE-19 UNKNOWN #6 (cert-manager coexistence) is closed by Q4 of this spike.

Recommendation ​

  1. Correct the TLS statement in ADR-0014 decision 4, and carry the corrected version into ADR-0023. ADR-0014 currently reads "A TLS termination certificate from a company or well-known public CA. Self-signed is not used for anything beyond throwaway evaluation." That is wrong as written: self-signed is the mandatory, automatic, non-optional mechanism for all internal traffic, minted by cert-manager into root-ca-secret and issued through the foundry-local-ca-issuer ClusterIssuer. The corrected decision is: internal TLS is self-signed and requires no operator action; a company or public CA certificate is required only for the external LoadBalancer Gateway, and only when off-cluster clients cannot be made to trust the cluster CA via the foundry-local-ca-bundle trust bundle.

  2. Keep the first increment entirely at exposure: internal, which is already the default on both settings, and defer the external certificate question. api.exposure defaults to internal and spec.endpoint.exposure defaults to internal. A first increment that sets neither exposes nothing off-cluster, needs no LoadBalancer, needs no DNS name, needs no CA certificate, and can be validated with kubectl port-forward and in-cluster clients. That removes three procurement dependencies from the critical path at zero cost to what the increment proves.

  3. Record the LoadBalancer as an undocumented prerequisite, and choose MetalLB before anything is designed for external access. The external Gateway is serviceType: LoadBalancer and no Foundry Local page lists a load balancer as a prerequisite, but AKS Arc has none by default. The moment exposure: external is used, the MetalLB Arc extension (or a third-party load balancer) and a non-overlapping IP pool become hard requirements, along with the --ip-pool-start / --ip-pool-end separation Microsoft "highly recommends" to avoid conflicts. Treat this as a finding, not a footnote: it is the kind of gap that surfaces as a Gateway stuck without an address on deployment day.

  4. Write the cert-manager sequencing into the wrapper as a hard, ordered precondition with a gap-minimisation note. Before installing azure-cert-manager: run helm list -A filtered for cert-manager and trust-manager, uninstall both if present, then install immediately. Microsoft's warning that "certificate rotation doesn't occur and trust bundles aren't distributed to new namespaces" during the window is a security statement, not a convenience one. Also pin the instance name azure-cert-manager and all five --config flags from the Foundry Local page, because the generic Arc article's command produces an installation that lacks them.

  5. Build the AKS Arc cluster at Kubernetes 1.32 or later, not at the 1.29 minimum. Foundry Local requires 1.29 or later; the Microsoft.CertManagement extension's validated distribution list starts higher. Choosing 1.32 satisfies both and retires UNKNOWN #4 for free. This is a one-word change to a cluster-creation parameter with no downside identified.

  6. Do not treat the storage answer as settled, and inspect the PVC as the first read-only action after the first deployment. Four of thirteen unknowns in this spike are storage. The specific risk for ADR-0014's CPU-only first increment is UNKNOWN #9: modelCacheStorageGi is documented as vLLM-only, so an onnx-genai deployment may create a 100 GiB claim with no documented way to shrink it, for a model of roughly 8 GB. On a small proof-of-concept cluster that is the difference between fitting and not fitting. Add a storage inspection step (kubectl get pvc -n foundry-local-operator -o yaml, plus kubectl get sc) to the first-increment verification, alongside ADR-0014 decision 7's three-layer drift check.

  7. Get the Foundry catalog and model-artifact FQDNs from Microsoft during preview onboarding, and make it an explicit ask. UNKNOWN #6 is the only egress category that cannot be turned into a firewall rule from published documentation, and it gates the StoreModel cache Job, which gates every model deployment. Since ADR-0014 decision 9 already submits the preview access request, attach this question to it. In the meantime, hand the firewall team Microsoft's consolidated East US endpoint list rather than a hand-assembled table, and note the three platform rules: no HTTPS inspection, no Arc Private Link Scopes, ports 80 and 443 outbound.

  8. State plainly in ADR-0023 that the CPU-only increment gets round-robin, not inference-aware routing. EPP requires runtime: vllm, which requires compute: gpu. Every published performance number Microsoft has for this product is vLLM on GPU. Scaling the CPU increment past one replica adds concurrency and adds nothing else: no KV-cache locality, no queue-depth scoring. This is not an argument against the CPU increment, which exists to prove the automation pattern. It is an argument for saying so now, so that a later "why is multi-turn chat slow" question has a documented answer instead of becoming a surprise.

  9. Before relying on replicas at all, test whether an HPA survives the operator's reconciliation loop (UNKNOWN #11). The operator runs a 30-second reconciliation timer over ModelDeployment resources and "updates replica counts in the resource status." If it also enforces spec.replicas against the Deployment, any HorizontalPodAutoscaler will be fought and reverted. That is a five-minute test with a large blast radius if assumed wrong, and it belongs in the same first-increment verification pass as recommendation 6.

  10. Record the disconnected path as a genuinely different decision set, not a sourcing variant. ADR-0014 says the disconnected path "changes the source of the prerequisite layer, not its shape," and for the prerequisite layer that is correct and now confirmed. But two of ADR-0014's own decisions do not transfer: azure-cert-manager does not exist disconnected (community cert-manager and trust-manager from the expansion pack instead, inverting recommendation 4), and Entra ID authentication does not apply (local Active Directory, with Azure RBAC Reader and Contributor on the extension resource, where Contributor is needed even for chat/completions). If disconnected ever comes into scope, it needs its own ADR, not an amendment.

Verdict ​

Track 3's networking, ingress, and certificate surface is well enough documented to design against, and its storage surface is not.

Six of the nine questions have complete, first-party, unambiguous answers: the Gateway API topology and object names (Q1), the api.exposure value table and default (Q2), the certificate issuance path once the internal and external cases are separated (Q3), the cert-manager coexistence rule (Q4), the disconnected pre-staging procedure (Q7), and the Azure Local IP and logical-network prerequisites (Q9). Q8 is answered but with a sharp qualification that reshapes the first increment's expectations. Q5 is three quarters answered, with the model download path missing entirely. Q6 is the weak one: four of the thirteen unknowns sit in storage, and the one most likely to bite is whether a CPU model provisions an unshrinkable 100 GiB claim.

Two findings change what ADR-0023 should say rather than merely filling it in. SPIKE-19's TLS statement was backwards and would have sent a deployer procuring a certificate they do not need for an increment that never leaves the cluster. The external Gateway's LoadBalancer dependency appears in no Foundry Local document, and would have surfaced as a Gateway with no address on the day someone first set exposure: external. Both are cheap to fix now and expensive to discover later.

SPIKE-19 UNKNOWN #6 is closed. The two cert-managers do not coexist; the Arc extension is a documented replacement, the community charts must be uninstalled first, existing custom resources survive the swap, and Microsoft warns that certificate rotation stops during the window. That is a complete answer from documentation, which is what SPIKE-19 asked for and could not find.

Nothing in this spike changes ADR-0014's three-layer shape, its CPU-only first increment, or its Entra ID decision for the connected path. What it changes is the prerequisite list: add a load balancer decision, add a Kubernetes 1.32 floor, remove the certificate procurement from the first increment, and add a storage inspection to the verification pass. This spike authorizes nothing and spends nothing.

Sources ​

All first-party Microsoft Learn. Retrieved 2026-07-30.

Foundry Local on Azure Local

Azure Arc

  • Cert-manager for Arc-enabled Kubernetes (preview) (what the extension installs, self-signed and enterprise PKI and ACME issuer support, intermediate issuers for air-gapped issuance, the validated distribution list, and the regional support list): https://learn.microsoft.com/azure/azure-arc/kubernetes/cert-manager-overview
  • Deploy cert-manager for Arc-enabled Kubernetes (preview) (the coexistence answer: uninstall open source cert-manager and trust-manager first to avoid conflicts, CRs are retained, the helm list -A discovery command, the helm uninstall commands, the rotation-gap warning, the extension-is-a-replacement statement, the issuer types, the Certificate duration and renewBefore defaults, the telemetry disable flags, and the non-destructive delete behavior): https://learn.microsoft.com/azure/azure-arc/kubernetes/cert-manager-deploy
  • Azure Arc-enabled Kubernetes network requirements (the full outbound endpoint table: management.azure.com, <region>.dp.kubernetesconfiguration.azure.com, the login endpoints, mcr.microsoft.com and *.data.mcr.microsoft.com, gbl.his.arc.azure.com and *.his.arc.azure.com, guest notification and service bus, graph.microsoft.com, *.arc.azure.net, <region>.obo.arc.azure.com:8084, linuxgeneva-microsoft.azurecr.io, and the monitoring endpoints called out for cert-manager for Arc): https://learn.microsoft.com/azure/azure-arc/kubernetes/network-requirements

Azure Local and AKS Arc

  • AKS enabled by Azure Arc network requirements (logical networks and the static-IP-only rule, the logical-network parameter table including --ip-pool-start and --ip-pool-end, the KubeVIP control plane IP, the MetalLB or third-party load balancer choice and its IP pool, the proxy inheritance and no-update warning, DNS resolution, the six-address management block, and the cross-VLAN port table plus the Arc gateway port 40343): https://learn.microsoft.com/azure/aks/aksarc/network-system-requirements
  • Firewall requirements for Azure Local (ports 80 and 443 outbound, the no-HTTPS-inspection statement, the Arc Private Link Scopes exclusion and the public-IP resolution requirement, the consolidated per-region endpoint lists including East US, and the internal cluster port tables): https://learn.microsoft.com/azure/azure-local/concepts/firewall-requirements
  • Storage options for applications in AKS enabled by Azure Arc (the default StorageClass, CSV and VHDX backing, expandable volumes and reclaim policy, and the ReadWriteOnce versus ReadWriteMany distinction between disk and file volumes): https://learn.microsoft.com/azure/aks/aksarc/concepts-storage

Related records in this repository

SPIKE-19 (this spike closes its UNKNOWN #6 and extends its questions 2 and 5), SPIKE-09 (the original Azure Local assessment), ADR-0014 (the three-layer deployment shape, whose decision 4 TLS statement recommendation 1 corrects), ADR-0011 (multi-target deployment automation), ADR-0009 (the on-prem reviewer track).