Homestead Foundry

Appearance

SPIKE-29: Lifecycle, upgrade, and drift for the two Foundry Local tracks ​

Role: foundry-researcher (Opus). Status: research spike complete. No Azure resources created, no spend, no software installed, no cluster touched, no az or kubectl command run. First-party documentation review only. Date: 2026-07-30 Scope: lifecycle, upgrade, and drift for deployment track 2 (Foundry Local on Windows Server, ADR-0013) and track 3 (Foundry Local on Azure Local, ADR-0014). Neither track currently documents how anything gets updated after day one. Every factual claim is grounded in a first-party (Microsoft Learn, or Microsoft's own legal terms) source, cited inline. Anything Microsoft has not published is marked UNKNOWN with the test or doc that would resolve it. This spike feeds ADR-0024; it authorizes no deployment and no spend.

Depends on: docs/research/SPIKE-18-foundry-local-windows-server.md (track 2 install mechanism), docs/research/SPIKE-19-foundry-local-azure-local-deployment.md (track 3 layer model, and UNKNOWN #2, the preview-to-GA question this spike is asked to answer), docs/adr/ADR-0013-foundry-local-windows-server-install.md (decision 6, the idempotency contract), and docs/adr/ADR-0014-foundry-local-azure-local-deployment-layers.md (decision 1, the load-bearing install ordering, and decision 7, the split drift-detection problem). This spike verifies and extends those records; it does not restate them.

Headline, in two parts. First: Microsoft documents install ordering for track 3 and documents nothing at all about upgrade ordering, while the one upgrade that certainly will happen underneath the stack, an AKS Arc Kubernetes upgrade, is documented as a rolling node replacement that necessarily restarts istiod, which is the exact event Microsoft's own install warning says is flaky when it happens in the wrong order. Second: there is no documented preview-to-GA migration path, in-place migration is not promised anywhere, and a breaking change has already been delivered inside the preview (three deprecated ModelDeployment endpoint fields plus a published nginx-to-Gateway-API annotation migration table), which is the strongest available evidence of how this preview will behave at GA.

A third finding runs through everything below: latest is the default in three separate places (model.catalog.version, the operator's inference container images.<type>.tag, and the extension's --auto-upgrade-minor-version true). Track 3 therefore already upgrades itself in production, silently, along three axes that no what-if and no manifest diff can see.

Question ​

Ten questions, split across the two tracks plus a shared pair and one design input:

  1. Track 2: how is a machine-wide provisioned Foundry Local MSIX updated? Does Add-AppxProvisionedPackage -Online upgrade in place, what happens to a running service and the model cache, is there an automatic update channel, and can it be disabled?
  2. Track 2: how are cached models updated or pinned? Can a specific model version be held?
  3. Track 2: how does an operator detect that a host has drifted, given that Arc governs the install action rather than the resulting state?
  4. Track 3: how are Microsoft.CertManagement and Microsoft.Foundry versioned? Auto-upgrade, pinning, documented upgrade procedure?
  5. Track 3: what is the upgrade ordering constraint across the three layers, and what happens on a Kubernetes or AKS Arc upgrade underneath a deployed Foundry stack?
  6. Track 3: how are the CRDs upgraded, and what is the compatibility matrix between CRD versions, Istio versions, and the Foundry extension version?
  7. Track 3: what is the documented preview-to-GA migration path? (SPIKE-19 UNKNOWN #2)
  8. Both tracks: what is the backup and restore story, and what actually needs backing up as opposed to being rebuildable?
  9. Both tracks: what does teardown look like, and does it leave anything behind?
  10. Drift detection design input for ADR-0014 decision 7: what tooling detects drift in each layer, and what remains undetectable?

Findings ​

Q1. Track 2 has no documented update path for the install mechanism ADR-0013 selected ​

This is the sharpest gap in track 2, and it is the same shape as the defect SPIKE-18 found in ADR-0011.

The only documented upgrade command is a winget command. The Foundry Local CLI reference gives exactly one upgrade instruction for Windows: winget upgrade --id Microsoft.FoundryLocal. Source: Foundry Local CLI reference, Upgrade Foundry Local. That is the same tool that SPIKE-18 established cannot perform a machine-scope MSIX install at all, because "Winget blocks MSIX machine-scope installs." Source: Best practices and troubleshooting guide for Foundry Local CLI (preview). ADR-0013 decision 1 correctly replaced the install command with Add-AppxProvisionedPackage -Online and did not notice that doing so also orphaned the documented upgrade command. Track 2 currently has an install mechanism with no matching update mechanism.

The Add-AppxProvisionedPackage documentation does not describe upgrade semantics. The cmdlet reference describes only addition: it "adds an app package (.appx) that will install for each new user to a Windows image," and the -PackagePath parameter "specifies the location of the app package (.appx) to add to the Windows image. This package will be added for every new user account." Nothing in the syntax, the description, the parameter table, or the examples addresses what happens when a package of the same identity but a higher version is added to an image that already has one provisioned: whether it replaces the existing provisioning entry, whether it errors, whether users who already have the app installed are updated, or whether the running service is stopped. Source: Add-AppxProvisionedPackage (Dism). The related cmdlets are Get-AppxProvisionedPackage and Remove-AppxProvisionedPackage; there is no Update-AppxProvisionedPackage. This is UNKNOWN #1 and it is resolvable only by test.

What happens to the running service and the model cache is likewise unstated. No first-party page addresses either. Two things are known and bound the risk. The service is a separately controlled object with its own commands (foundry service start | stop | restart | status), and Microsoft's standing advice after an install-time service connection error is foundry service restart, so treating the service as needing an explicit restart after any package operation is consistent with documented behaviour. The cache is a separate on-disk directory with its own commands (foundry cache location, foundry cache list, foundry cache cd <path>, foundry cache remove <model>), which strongly implies it is not inside the package payload, but no page states that an upgrade or uninstall preserves it. Source for both: Foundry Local CLI reference. Carried as UNKNOWN #2.

There is one automatic update channel, it is real, and no way to disable it is documented. The CLI reference states, of the plugin execution providers: "Foundry Local automatically downloads these execution providers on first run. The plugin execution providers automatically update when new versions are available." Source: Foundry Local CLI reference, Execution providers. The plugin set covers NvTensorRTRTXExecutionProvider, OpenVINOExecutionProvider, QNNExecutionProvider, and VitisAIExecutionProvider, each with documented minimum driver versions. For a governed host this matters in two ways: a component of the inference stack updates itself without operator action, and no foundry service set option, configuration file, registry key, or policy for disabling it appears in any first-party page. Note the mitigating detail for this repo's specific target: the CPU, WebGPU, and CUDA execution providers are described as built in, not plugins, and ADR-0013 decision 9 commits track 2 to CPU-only inference, so on a GPU-less Server host the auto-updating plugin set has nothing applicable to update. That is a scope accident, not a control. Whether the channel can be disabled is UNKNOWN #3.

No Azure-side patching service covers this. Azure Update Manager works over Arc-connected machines and applies "updates that are available in Microsoft Update," "updates that are available in Linux packages," and "updates that are published to Windows Server Update Services (WSUS)." Source: Azure Update Manager Overview. MSIX or Microsoft Store packaged applications are not named in that scope. So Update Manager governs the Windows Server host underneath Foundry Local, and does not govern Foundry Local. That distinction belongs in ADR-0024 explicitly, because "the host is patched by Update Manager" is exactly the sentence a reader will over-read.

Net on Q1: the honest answer is that the update path for a machine-wide provisioned Foundry Local is undocumented, the one documented path uses a tool that cannot perform the corresponding install, and the practical fallback (re-provision the newer MSIX over the older one, then foundry service restart, then verify with foundry service status and one inference call) is a plausible design that has no first-party backing and must be labelled as such.

Q2. Track 2 has no model version pinning at all, and the CLI's "model ID" is a hardware variant, not a version ​

The cache surface is complete and simple: foundry cache list enumerates cached models, foundry cache remove <model> deletes one, foundry cache location reports the directory, foundry cache cd <path> moves it. Downloads are foundry model download <model>, and foundry model run <model> "runs a specified model, downloads it if it isn't cached." Source: Foundry Local CLI reference.

No command in that surface takes a version. The <model> argument is documented as "its alias or model ID," where the alias "selects the best model for your available hardware automatically" and the model ID pins hardware, not release: the worked example is "to run the qwen2.5-0.5b on CPU - irrespective of your available hardware - use: foundry model run qwen2.5-0.5b-instruct-generic-cpu." Source: same page. Every documented model ID in the filter examples has the same shape (phi-4-mini-instruct-generic-cpu, deepseek-r1-distill-qwen-1.5b-generic-cpu): publisher, size, tuning, and execution target, with no version segment. foundry model info <model> displays details and --license displays licence information, but no version-selection flag is documented for any command.

The contrast with track 3 is instructive and worth carrying into ADR-0024: the ModelDeployment spec has a first-class model.catalog.version field, type string, not required, default latest. Source: ModelDeployment and operator configuration reference. So the cluster product exposes a version axis and defaults it to latest; the device product does not appear to expose one at all.

Consequences for ADR-0013 decision 6 stage 2. That stage checks foundry cache list for the target model ID and downloads only if absent, which is correct as idempotency but is not version control: it holds whatever bytes were fetched the first time, indefinitely, with no way to ask what they are or to compare them to what the catalog now offers. Conversely a cache miss (a rebuilt host, a foundry cache cd to a new path, an operator running foundry cache remove) re-pulls whatever is current. So the effective policy today is "pinned by accident until it isn't." Whether re-pulling always takes the latest, and whether any version identifier is exposed anywhere (in foundry model info output, in the cache directory layout, or in the catalog API), is UNKNOWN #4, and it is resolvable read-only once anything is installed.

Q3. Track 2 drift: the honest answer is that Arc run-command cannot detect state, and machine configuration is the mechanism ADR-0013 is missing ​

The question is fair and the answer is unflattering to the current design.

What the run-command actually tells you. SPIKE-18 established that Instance View returns InstanceViewExecutionState, ProvisioningState, exit code, stdout, and stderr, with output capped at 4 KB. Source: Run command on Azure Arc-enabled servers (preview). Every one of those describes an execution, at a point in time, of a script the operator chose to send. None of them describes the machine. Between two invocations the Arc control plane has no opinion about, and no visibility into, whether Foundry Local is still installed, still the version that was installed, still running, or still holding the model it was given. ADR-0013 decision 10 already says Arc governs installation and not the endpoint; this is the same truth applied to time rather than to scope, and ADR-0024 should state it in those words.

What a periodic re-run of the decision 6 script would genuinely verify. Exactly its four stage checks, at the moment it runs, and nothing else:

StageCheckDetectsDoes not detect
1. Installprovisioned-package list plus foundry --versionThe package is gone, or a version other than expected if the script compares the string rather than merely asserting non-emptyAnything about how it got there, or a change made and reverted between runs
2. Model cachefoundry cache list for the target model idThe model was removed, or the cache directory was movedWhether the cached bytes are the intended model release (Q2: no version is exposed)
3. Service configfoundry service statusService stopped or unreachableConfiguration set through foundry service set, which has no documented read-back
4. Validateone inference callThe runtime is broken end to endDegradation, quality change, or a swapped execution provider

Two structural limits follow. It is poll-only, so its resolution is its schedule and it is blind between runs. And it is self-reported through a 4 KB channel unless the blob path is enabled, which ADR-0013 decision 4 correctly treats as a secret-bearing opt-in. That is a thin instrument to call drift detection, and calling it that would overstate it.

The mechanism that does detect state is Azure Machine Configuration, and neither ADR-0011 nor ADR-0013 mentions it. It "provides native capability to audit or configure operating system settings as code for machines running in Azure and hybrid Arc-enabled machines," configurations "can include ... application configuration or presence," and it offers three enforcement modes: Audit ("only report on the state of the machine"), Apply and Monitor ("configuration applied to the machine and then monitored for changes"), and Apply and Autocorrect ("configuration applied to the machine and brought back into conformance if drift occurs"). It supports Microsoft Windows Server "2012 - 2025," is orchestrated by Azure Policy or assigned manually, surfaces per-setting results on the guest assignments page, and allows "up to 50 guest assignments per machine." Source: What is Azure Machine Configuration?.

That is a first-party, RBAC-governed, continuously evaluated, Policy-reportable state mechanism for exactly the host class track 2 targets, and it maps onto ADR-0013's four stages almost field for field. It is also the only way track 2 gets a compliance signal in Azure rather than a script result an operator has to go and read. Two honest caveats: Microsoft does not document a built-in configuration for Foundry Local, so this means authoring a custom configuration package, and Autocorrect mode against a multi-GB model download is an aggressive default that ADR-0024 should reject in favour of Audit or Apply and Monitor. What machine configuration can actually assert about an MSIX-provisioned application, as opposed to files, registry, and services, is UNKNOWN #5.

Q4. Track 3 extension versioning: the generic mechanism is fully documented, the product-specific one is not documented at all ​

Both Foundry Local extensions are ordinary Microsoft.KubernetesConfiguration/extensions resources, so the generic Arc cluster-extension lifecycle applies in full.

Auto-upgrade is on by default, and the documented Foundry install command turns it on explicitly. The --auto-upgrade-minor-version parameter is "a Boolean property that sets whether the extension minor version upgrades automatically. The default setting is true. If you set this parameter to true, you can't set the version parameter, because the version is dynamically updated. If you set this parameter to false, the extension isn't automatically upgraded, even for patch versions." Major versions never auto-upgrade: "Because major version upgrades can include breaking changes, there's no automatic upgrade support for new major versions of an extension instance." Source: Deploy and manage an Azure Arc-enabled Kubernetes cluster extension.

The documented Foundry Local install command sets it explicitly:

--auto-upgrade-minor-version true \
--release-train stable \

and the cert-manager command in the same article sets --release-train stable but omits the auto-upgrade flag, which means it takes the true default. Source: Deploy Foundry Local as an Azure Arc extension. So as documented, both of track 3's ARM extensions auto-upgrade their minor and patch versions in place, on Microsoft's schedule, with no operator action and no change to any Bicep file.

Pinning is possible and it is an either-or. Set --auto-upgrade-minor-version false and supply --version, which is "the version of the extension to install (the specific version to pin the extension instance to)." The two are mutually exclusive in both create and update. Source: same page.

The documented upgrade procedure is generic. With auto-upgrade on, "the extension automatically upgrades when a new minor version is released." With it off, "you must upgrade the extension manually," using az k8s-extension update ... --version x.y.z. The same command changes --configuration-settings, with an important trap for any wrapper: for --config-protected-settings, "provide all settings, even if you update only one setting. If you omit any of these settings, the omitted settings are deleted." Source: same page. Since entraAuth.tenantId and entraAuth.clientId are passed as config at install, any partial config update is a live risk to track 3's authentication decision (ADR-0014 decision 5).

Release trains are the third axis. --release-train selects "the release train, if the extension has published versions in different release trains such as Stable or Preview. If you don't set this parameter explicitly, Stable is the default." Source: same page. Both documented Foundry commands set stable explicitly, which is the correct choice and worth locking in ADR-0024 rather than leaving to a default.

What is missing is product-specific and it is significant. "Foundry Local on Azure Local (preview)" and "Cert-manager for Arc-enabled Kubernetes (preview)" are both listed as available extensions, both marked preview, with supported distribution "AKS enabled by Azure Arc" for Foundry Local. Source: Available extensions for Azure Arc-enabled Kubernetes clusters. Neither entry publishes a version list, release notes, or a version-support window. The same article shows what that would look like if it existed: Flux gets an explicit statement, "The most recent version of the Flux v2 extension and the two previous versions (N-2) are supported," plus dated release notes, and the Azure Policy extension gets versioned release notes with dates and component image versions. Foundry Local gets a paragraph of description. So an operator today cannot answer "what version am I on, what is current, what changed, and how long is my version supported" from documentation; only az k8s-extension show (a live read) reports the version field. That is UNKNOWN #6.

Q5. Track 3 upgrade ordering: undocumented, and the one upgrade that is documented restarts istiod by design ​

This is the question the tasking flagged and it deserves a blunt answer.

Install ordering is documented and load-bearing. Microsoft states it directly: "Install the Gateway API CRDs first, then install Istio as the Gateway API provider. Istio's istiod discovers the CRDs at startup and registers the istio GatewayClass once it sees them. Installing the CRDs and Istio in the reverse order forces an istiod restart and is reported as flaky on some clusters." The Inference Extension CRDs go in "before istiod so istiod picks up InferencePool support at startup," and there is a gate: kubectl get gatewayclass istio must show ACCEPTED True. Source: Deploy Foundry Local as an Azure Arc extension.

Upgrade ordering is documented nowhere. There is no upgrade article in the Foundry Local on Azure Local doc set at all (see Q7 for the full table of contents). No page says whether the CRDs must be upgraded before Istio, whether the extensions must be upgraded before or after the prerequisite layer, or whether the GatewayClass gate should be re-checked after any layer changes. UNKNOWN #7. The install warning is about a startup ordering property of istiod, and every mechanism below restarts istiod at a time the operator does not choose, so the same failure shape is plainly reachable on upgrade. Microsoft does not say so, and this spike will not assert it as fact.

What is documented, and consequential, is the AKS Arc upgrade itself. From Upgrade an Azure Kubernetes Service (AKS) cluster (AKS enabled by Azure Arc), which carries "Applies to: AKS on Azure Local":

  • Upgrades are rolling node replacements, not in-place node patches. "All upgrades run in a continuous, rolling manner to ensure uninterrupted availability of workloads. When a new Kubernetes worker node with a newer build joins the cluster, the process moves resources from the old node to the new node. After this step completes successfully, the process decommissions and removes the old node from the cluster." Every pod in the Foundry stack, including istiod, the inference operator, and the model-serving pods with their init-container model pulls, is therefore rescheduled onto a new node during a Kubernetes upgrade.
  • Minor versions cannot be skipped. "You can't skip Kubernetes minor versions. You must perform all upgrades sequentially by major version number." And if you are far behind, Microsoft says start over: "If you upgrade from an unsupported version that skips two or more minor versions, the upgrade might not work properly. If your version is significantly out of date, we recommend you recreate your cluster instead."
  • There is no node-image-only path. "Currently, AKS Arc doesn't support node-image-only updates across all supported Kubernetes versions. If you need to update the node image, you must upgrade the cluster to the latest Kubernetes version." So OS-level currency on the worker nodes forces a Kubernetes version move, which forces the node replacement above. An operator cannot patch nodes and hold Kubernetes still.
  • A failed-looking upgrade may not be failed. "if the operation times out, provisioningState shows Failed, while currentState continues to show Upgrading as the upgrade continues in the background. No action is required." A wrapper that treats Failed as terminal will do the wrong thing.
  • Available targets come from az aksarc get-upgrades; the upgrade is az aksarc upgrade --kubernetes-version <version>.

And the platform underneath has its own cadence, which explicitly excludes the workload. Azure Local follows the Modern Lifecycle policy and "you must stay within six months of the most recent release"; cumulative updates are monthly, feature updates semi-annual; "Azure Arc resource bridge requires solution updates to be applied within one year. This is critical to keep certificates valid." The update package covers the OS, "core agents and services" including the Connected Machine agent and Arc resource bridge, and the Solution Builder Extension. Then the decisive sentence: "Customer workloads aren't covered by this update solution." Also: "Microsoft does not support out of band updates for individual components," and a list of interfaces not to use, including Windows Admin Center and the machine-level Azure Update Manager pane. Source: About updates for Azure Local, version 23H2.

Net on Q5, and this is the answer ADR-0024 needs. Track 3 sits on a platform with a mandatory six-month currency obligation and a monthly update cadence, on a Kubernetes cluster that cannot be patched without a version move and cannot be version-moved without replacing every node, running a stack whose only documented ordering constraint is a startup-order dependency of the component that node replacement necessarily restarts. Microsoft documents each of those facts and connects none of them. The interaction is unowned: the Azure Local update solution explicitly does not cover the workload, and the Foundry Local doc set does not cover being upgraded underneath. That is not a small gap in a preview, it is the central operational risk of the track, and ADR-0024 should record it as such rather than as a footnote.

Q6. CRD upgrades are unaddressed, and there is no compatibility matrix, only floors ​

How the CRDs get installed is by direct kubectl apply --server-side against pinned upstream release URLs: gateway-api/releases/download/v1.4.0/standard-install.yaml and gateway-api-inference-extension/releases/download/v1.5.0/manifests.yaml. Istio arrives by Helm from the upstream chart repository, as istio-base then istiod with --set pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true. Source: Deploy Foundry Local as an Azure Arc extension.

Three consequences follow directly and all of them matter for ADR-0024:

  1. The prerequisite layer is upstream-sourced, not Microsoft-serviced. These are Kubernetes SIG and Istio artifacts fetched from GitHub and a Google Cloud Storage Helm repository. No Microsoft extension, no Microsoft release train, no Microsoft version-support statement covers them. Upgrading them is by definition an out-of-band act against a stack Microsoft says elsewhere it does not support out-of-band updates for.
  2. Microsoft publishes floors, never ceilings, and never a tested combination. The stated requirements are Gateway API CRDs "v1.4.0 or later," Gateway API Inference Extension CRDs "v1.5.0 or later," Istio "version 1.29 or later," Kubernetes "version 1.29 or later." Source: same page. There is no upper bound on any of them, no statement of which combinations were validated together, and the Foundry extension's own version does not appear in that list at all. There is no compatibility matrix. UNKNOWN #8.
  3. cert-manager's CRDs are deliberately retained. The documented install passes --config cert-manager.crds.keep=true. Source: same page. That is an explicit instruction to leave the CRDs behind when the release goes away, which is the correct choice for avoiding data loss and is also a guaranteed teardown residue (Q9).

The --server-side flag is the one upgrade-relevant detail Microsoft does supply, in that server-side apply is the mechanism that makes re-applying a newer CRD manifest a merge rather than a clobber. But no page states that re-applying a later release manifest is the supported upgrade procedure, and none addresses what a CRD version bump does to existing ModelDeployment, Gateway, HTTPRoute, and InferencePool objects. The stack's fragility here is visible in the ModelDeployment reference, which already documents an endpoint.gatewayAnnotations field for "Istio networking.istio.io/* overrides" and an EnvoyFilter-based mechanism for body-size and timeout settings. Source: ModelDeployment and operator configuration reference. Those are Istio implementation details reaching into the product's own API surface, which is precisely the kind of coupling that makes an Istio major upgrade risky and which no compatibility statement covers.

Q7. Preview to GA: no path is published, in-place migration is not promised, and a breaking change has already landed inside the preview ​

SPIKE-19 UNKNOWN #2 asked for the preview-to-GA timeline and SLA. The answer is now firmer than "not published yet," and it is worth stating in three parts.

Part one: nothing is published, and the doc set has no lifecycle articles at all. Every current page carries the same notice: "Foundry Local is available in preview. Preview releases provide early access to features that are in active deployment. Features, approaches, and processes can change or have limited capabilities before general availability (GA)." Access remains by request form. Sources: Deployment overview, Deploy Foundry Local as an Azure Arc extension, Known issues.

The published table of contents for the whole doc set is: overview, known issues, eight concept articles (models, model catalog, bring your own models, inference operator, model caching, inference runtimes, authentication and authorization, GPU inference planner), one quickstart, six deploy and secure how-tos, one run-inference how-to, four disconnected-operations articles, and five reference articles. Source: Foundry Local for Microsoft Sovereign Private Clouds documentation. There is no upgrade article, no uninstall or teardown article, no backup and restore article, no migration article, and no day-2 operations article. That is a complete enumeration, not a sampling, and it is the citation ADR-0024 should use when it says the lifecycle is undocumented.

The omission is conspicuous because the same doc set tells the reader to evaluate the thing it does not document. The planning table's Pilot stage says to check "authentication flow, monitoring, update process, and operational ownership," and the deployment overview describes the control plane as the "Azure Arc extension and inference operator that manage model resources and lifecycle changes." Source: Deployment overview. The update process is named as a gate and never described.

Part two: the closest first-party legal statement points at redeploy, not in-place migration. The Supplemental Terms of Use for Microsoft Azure Previews state that "You may not use Early Access Previews in production or in a 'live' operating environment." On the transition to GA, the only concrete obligation in the terms concerns containers, and it is a replacement obligation, not a migration promise: on general availability, "Customer must discontinue its use of preview containers and is required to update to the relevant GA version(s)." Source: Supplemental Terms of Use for Microsoft Azure Previews. Nothing in those terms promises that preview state, configuration, or deployed resources survive the transition.

Part three, and the most useful evidence: a breaking change has already shipped inside the preview. The ModelDeployment reference now marks three fields as deprecated. endpoint.enabled is "Deprecated. Retained for backward compatibility ... New deployments should use exposure directly." endpoint.ingressClassName is "Deprecated and ignored. Was used to select the nginx IngressClass; no longer in the data path." endpoint.annotations is "Deprecated and ignored. Was used to pass nginx Ingress annotations; replaced by endpoint.gatewayAnnotations." The same article carries an "Annotation migration" table stating that "The nginx Ingress annotations that earlier releases honored on endpoint.annotations are no longer applied," and that one of them, configuration-snippet, is "Not portable." Source: ModelDeployment and operator configuration reference. This is the whole data-path having moved from an nginx Ingress controller to the Gateway API, mid-preview, with fields silently ignored rather than rejected and at least one migration declared non-portable.

Net on Q7, stated as the tasking asks:

  • Is an in-place migration promised? No. No first-party source promises one, and the only GA-transition obligation Microsoft's preview terms state anywhere is a discontinue-and-update-to-GA obligation.
  • Is a redeploy implied? Yes, on the balance of the evidence: the preview terms' container clause, the absence of any migration or upgrade article, and a demonstrated willingness to change the data path and ignore previously honoured fields within the preview. This is an inference from first-party evidence and is labelled as such, not a Microsoft statement.
  • Are preview deployments documented as disposable? No. Microsoft does not say that either. The correct posture is that this project should treat them as disposable by decision, and ADR-0024 should record that as a decision with its reasoning, rather than attributing it to a document that does not exist.

The practical form of "disposable by decision" is testable rather than rhetorical: every input needed to rebuild the deployment (cluster, prerequisite versions, extension configuration, generated ModelDeployment manifests from the registry) must live in source control, so that a rebuild is a pipeline run and not an archaeology exercise. ADR-0014 decision 6 already delivers most of that for the intent layer. What is not yet in source control is the prerequisite layer's pinned versions and the extension configuration, and Q10 argues they should be.

Q8. Backup and restore: almost everything is rebuildable, and the exceptions are the certificate material and the disconnected artifacts ​

Neither track has a first-party backup story. Track 2's doc set has no backup page. Track 3's doc set has none either (Q7's table of contents). The honest analysis is therefore a classification exercise, and the classification is favourable.

Track 2. The state on the host is: the provisioned MSIX (rebuildable from the artifact), the downloaded execution providers (self-fetching and self-updating, per Q1), the model cache (rebuildable by foundry model download, subject to the version caveat in Q2 and to internet access, which the CLI prerequisites require for "first-time downloads"), and whatever service configuration foundry service set was given (see the caveat below). Source: Foundry Local CLI reference. Nothing here warrants a backup product. The correct statement for ADR-0024 is that track 2 needs no backup, it needs a reliable rebuild, which is precisely what ADR-0013 decision 6's idempotent script provides. The one caveat is that foundry service set <options> is documented as a setter with no documented getter and no documented configuration file location, so any configuration applied through it is not obviously readable and therefore not obviously reproducible. That is UNKNOWN #9, and the mitigation is to keep the settings in the script rather than applying them by hand.

Track 3, classified layer by layer:

StateRebuildable?How, and the caveat
ModelDeployment CRsYesRegenerated from the registry, per ADR-0014 decision 6. This is the payoff of that decision.
Model CRs for catalog modelsYesNever authored: lazy registration creates them from the catalog ConfigMap, catalog.lazyRegistrationEnabled default true. Source: ModelDeployment reference.
Catalog ConfigMap foundry-local-catalogYesMaintained by catalog-sync; Model status carries catalogSync.lastSynced and syncStatus. Source: same.
Cached model artifacts in the local OCI registryYes, connectedStoreModel is internal, never authored directly, and re-created by the operator on the next ModelDeployment reconcile; on failure "the operator deletes the StoreModel so the next deployment attempt can retry." Source: Model caching and StoreModel lifecycle. Disconnected is different: dependencies arrive as expansion packs imported into the local edgeartifacts registry, so a rebuild depends on retaining those artifacts, not on re-downloading.
API key Kubernetes SecretYes, by regenerationNot restorable, but regenerable: "Delete the Kubernetes secret for the deployment. The operator recreates it automatically with new keys." Source: Known issues. This is also the documented rotation workaround, since "The inference operator doesn't support automatic rotation of API keys." Moot under ADR-0014 decision 5's Entra choice, but relevant if a key path is ever used.
TLS certificate materialPartly, and this is the real exceptionADR-0014 decision 4 requires a certificate from a company or public CA. cert-manager reissues what it can, and Certificates are visible via kubectl get certificates -n foundry-local-operator. Source: Troubleshoot Foundry Local on Azure Local. But an externally issued certificate and any private CA key are inputs the cluster cannot regenerate.
vLLM model cache PVCNot in first-increment scopespec.vllm.modelCacheStorageGi defaults to 100 GiB and applies only to runtime: vllm. Source: ModelDeployment reference. Relevant when the GPU increment lands.

The cluster-level backup story is weak and should not be leaned on. The AKS Arc backup guidance is a Velero article that carries "Applies to: AKS on Windows Server," not AKS on Azure Local. Velero is described as "an open-source community standard tool," not a first-party service. The article states plainly: "Velero doesn't officially support Microsoft Windows. In testing, the Velero team was able to back up stateless Windows applications only. Restic integration and backups of stateful applications or persistent volumes aren't supported." It also states "AKS Arc currently doesn't support volume snapshots," which is why Restic is needed for persistent volumes at all, and, critically for any restore plan: "To restore a cluster, you must create a new cluster to restore the old cluster to. You can't restore a cluster backup to an existing cluster." The install procedure additionally requires a service principal with a client secret written to a plaintext file. Source: Back up, restore workload clusters using Velero. That last point conflicts with ADR-0005's managed-identity-first stance and with ADR-0014 decision 10's statement that track 3 needs no exception to ADR-0005. Adopting Velero would create one. That is a good reason not to.

Net on Q8: the recommendation is to back up nothing at the cluster level and to guarantee rebuildability instead, with two named exceptions that must be held outside the cluster: the TLS certificate and any private CA material (by name in the tenant Key Vault, never by value in git), and, for a disconnected deployment, the expansion pack artifacts. Everything else is regenerable from source control and the catalog.

Q9. Teardown leaves residue in both tracks, and in track 3 some of it is left behind by design ​

Track 2. The only documented uninstall is winget uninstall Microsoft.FoundryLocal. Source: Foundry Local CLI reference. That has the same mismatch as the upgrade command in Q1: it is the per-user package-manager path, not the counterpart to machine-wide provisioning. The counterpart cmdlet exists and is linked from the install cmdlet's related links as Remove-AppxProvisionedPackage, but no Foundry Local page names it, so using it is an inference from the DISM module rather than a documented procedure. Source: Add-AppxProvisionedPackage (Dism).

Likely residue, none of it documented either way, all of it UNKNOWN #10: the model cache directory (a separate, relocatable path per foundry cache location and foundry cache cd, so there is no reason to expect package removal to touch it, and on this repo's target that is several GB per model), the downloaded execution providers, and the Windows service registration. ADR-0013 decision 6 rule 6 deliberately keeps uninstall out of the install script, which remains right; ADR-0024 should specify a separate, explicitly invoked teardown script whose contract is: remove the provisioned package, stop and remove the service, and, as a distinct and separately confirmed step, empty the model cache via foundry cache remove per model. Splitting the cache deletion from the package removal matters because the cache is the expensive thing to rebuild.

Track 3 residue is better documented, and larger.

  • The ARM delete and the in-cluster delete can diverge. az k8s-extension delete "immediately deletes the Azure extension resource. However, the command deletes the Helm release on the cluster that's associated with this extension only when the agents running on the Kubernetes cluster have network connectivity and can reach Azure services." Source: Deploy and manage an Azure Arc-enabled Kubernetes cluster extension. So a teardown against a disconnected or degraded cluster removes the Azure-side record and leaves the workload running, which is the worst possible half-state: no ARM representation, no what-if visibility, and pods still serving. The related connectivity rule is that protected configuration settings are held for up to 48 hours, after which a pending extension goes to Failed. Source: Cluster extensions in Azure Arc-enabled Kubernetes.
  • cert-manager's CRDs are retained on purpose. cert-manager.crds.keep=true is in the documented install command. Source: Deploy Foundry Local as an Azure Arc extension.
  • The entire prerequisite layer survives any extension delete, by construction. Gateway API CRDs, Inference Extension CRDs, istio-base, and istiod were installed by kubectl and helm, outside the extension mechanism, so no az k8s-extension delete touches them. Nor does anything remove the istio-system namespace, the Gateway resources the operator created, or the LoadBalancer service backing an external Gateway.
  • The cached model artifacts in the cluster's local OCI registry have no documented removal path. StoreModel is internal and the docs say "You never create StoreModel resources directly." Source: Model caching and StoreModel lifecycle. Whether deleting the last ModelDeployment garbage-collects the cached artifacts is UNKNOWN #11.

The clean statement for ADR-0024 is that track 3 teardown has three separate scopes (ARM extensions, prerequisite layer, cluster storage), only the first is a single command, and a teardown wrapper must own the other two explicitly or accept a cluster that cannot be cleanly re-installed onto.

Q10. Drift detection per layer: what each tool sees, and the three things nothing sees ​

ADR-0014 decision 7 is correct in shape. This section adds what each mechanism can and cannot assert, and adds the track 2 row the ADR does not have.

Track 2 (not covered by ADR-0014 decision 7 at all).

MechanismDetectsCannot detect
Scheduled Arc run-command of the idempotent scriptThe four Q3 stage checks, at run time, poll-onlyAnything between runs; anything the script does not check; version, unless the script compares strings
Azure Machine Configuration in Audit or Apply and Monitor modeContinuous, Policy-reportable guest state including "application configuration or presence," with per-setting compliance resultsRequires a custom configuration package; what it can assert about a provisioned MSIX specifically is UNKNOWN #5
Azure Update ManagerOS update compliance on the Arc-connected hostFoundry Local itself: its scope is Microsoft Update, Linux packages, and WSUS

Sources: Run command on Azure Arc-enabled servers, What is Azure Machine Configuration?, Azure Update Manager Overview.

Track 3, per ADR-0014's three layers.

LayerMechanismDetectsCannot detect
Prerequisitekubectl get crd against pinned versions; kubectl get gatewayclass istio for ACCEPTED True; helm list -n istio-system against pinned chart versionsMissing, downgraded, or upgraded CRDs; an unhealthy or unregistered GatewayClass; a chart version changeA Helm --set value change that leaves the chart version identical. Specifically, pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true can be turned off in a helm upgrade at the same chart version and helm list will look unchanged
Platform (ARM)az deployment sub what-if; plus Azure Policy, which can "check for the presence or absence of a specific cluster extension" and lets you "remediate noncompliant resources"Extension removed, config changed, wrong release trainVersion, when autoUpgradeMinorVersion is true: the docs state that in that mode "the version is dynamically updated" and version cannot be set, so there is no desired value for what-if to compare against. The documented install command sets exactly this mode
Intent (Kubernetes)kubectl get modeldeployments -A -o json diffed against generated manifestsA hand-created, deleted, or edited ModelDeploymentWhat latest resolved to. model.catalog.version defaults to latest, and the operator's inference images default to tag latest. The model variant and the container image can both change with no change to any spec

Sources: Cluster extensions in Azure Arc-enabled Kubernetes, Deploy and manage an Azure Arc-enabled Kubernetes cluster extension, ModelDeployment and operator configuration reference.

The three genuinely undetectable things, and what to do about each. All three are the same failure: desired state does not name a version, so a diff has nothing to compare.

  1. Extension version under auto-upgrade. Unfixable while auto-upgrade is on, by design. The choice is between currency without drift detection and pinning with drift detection. Given that both extensions are preview and neither publishes release notes or a support window (Q4), currency has real value and pinning has real cost. ADR-0024 should pick deliberately rather than inherit the documented command's default.
  2. Resolved model variant. Partly fixable at low cost: status.resolvedModel.variant and status.resolvedModel.name are published status fields. Source: ModelDeployment reference. Setting model.catalog.version explicitly in generated manifests turns this from unobservable into a normal spec diff.
  3. Resolved inference container image. Fixable as an observation only: status.resolvedModel.image publishes it. The operator's images.<type>.tag default of latest lives in the operator's own ConfigMap, so it is extension configuration rather than per-deployment spec.

The general rule ADR-0024 should adopt: record observed status as a baseline, not just desired spec. A check that snapshots status.resolvedModel.{name,variant,image} alongside the spec diff converts all three of the above from invisible to at least reportable, without needing any capability Microsoft has not shipped. It also gives the "what changed" answer after an auto-upgrade, which is otherwise unavailable.

One further note for the check script ADR-0014 decision 7 rule 2 mandates: the ModelDeployment status.state enumeration includes Updating alongside Pending, Creating, Running, Error, and Terminating. Source: same reference. A drift check that runs during an operator-initiated update must not report Updating as drift, and a check that runs during an AKS Arc rolling upgrade (Q5) will see pods moving. Both need a documented quiet-period or retry rule.

What is still UNKNOWN ​

#UnknownWhy it is not in the docsWhat resolves it
1Does Add-AppxProvisionedPackage -Online upgrade an already-provisioned package in place? Does it replace the provisioning entry, error, or leave both? What happens to users who already have it installed?The DISM cmdlet reference documents addition only. There is no Update-AppxProvisionedPackage. The Foundry Local docs' only upgrade instruction is a winget command that cannot do machine-scope MSIX.Test on a disposable Windows Server 2025 build VM: provision version N, then provision version N+1, then inspect Get-AppxProvisionedPackage and foundry --version. Requires owner authorization (installs software) and is the same test ADR-0013 decision 11 already gates, extended by one step.
2Does an MSIX upgrade or uninstall preserve the model cache and the service state?Unaddressed by both the Foundry Local and DISM documentation.Same test: record foundry cache location and foundry cache list before and after, and foundry service status after.
3Can the automatic execution-provider update channel be disabled on a governed host?The CLI reference states plugin EPs "automatically update when new versions are available" and documents no control. No foundry service set option, config file, registry key, or policy is published.Read foundry service set --help and foundry --help output once installed; if no control exists, escalate through the preview feedback channel. Low urgency for track 2's CPU-only scope, since the built-in CPU provider is not in the auto-updating plugin set.
4Can a specific model release be pinned or held on the device product, and does a re-pull always take the latest?The CLI exposes alias and model ID only; documented model IDs encode hardware variant, not version. No --version flag on any documented command. Track 3's model.catalog.version has no device counterpart.Read foundry model info <model> output and the cache directory layout for a version identifier once installed. Read-only after install.
5What can Azure Machine Configuration actually assert about a provisioned MSIX application?Machine configuration documents "application configuration or presence" generically and publishes no MSIX-specific resource or built-in configuration for Foundry Local.Author a minimal custom configuration package asserting file, service, and provisioned-package presence, assign it in Audit mode to one Arc-connected test host, and read the guest assignment results.
6What versions of Microsoft.Foundry and Microsoft.CertManagement exist, what changed between them, and how long is a version supported?Both are listed as preview extensions with a description and no version list, no release notes, and no support-window statement. Flux and Azure Policy in the same article publish all three.Read az k8s-extension show --name inference-operator for the live version field once a cluster exists (read-only). A published version history and support window can only come from Microsoft.
7Is there an upgrade ordering constraint across the three layers, and specifically does the CRD-before-istiod rule apply on upgrade?Microsoft documents install ordering and its flakiness failure mode, and publishes no upgrade guidance at all. There is no upgrade article in the doc set.Test on a non-production cluster: upgrade istiod at a fixed CRD version and observe gatewayclass istio and the operator; then upgrade CRDs under a running istiod. Ask the same question through the preview onboarding channel, which is also the channel that supplies the Helm path.
8The compatibility matrix between Gateway API CRD version, Inference Extension CRD version, Istio version, Kubernetes version, and Foundry extension version.Microsoft publishes minimum floors only (v1.4.0+, v1.5.0+, Istio 1.29+, Kubernetes 1.29+), no ceilings, no tested combinations, and does not include the extension version in the list.Only Microsoft can publish this. Interim mitigation: pin all four in source control and treat any change as a tested change on a non-production cluster.
9Is foundry service set configuration readable, and where does it persist?The CLI documents a setter with no getter and names no configuration file.Read foundry service set --help once installed, and check the cache and application data directories.
10What does track 2 teardown leave behind?winget uninstall is the only documented removal and it is the wrong tool for a provisioned package; Remove-AppxProvisionedPackage is never named by the Foundry Local docs.Same disposable-VM test: remove, then check the cache directory, the execution-provider directory, the service registration, and the provisioned-package list.
11Does deleting the last ModelDeployment reclaim cached model artifacts from the cluster's local OCI registry?StoreModel is internal, never user-created, and the caching article documents creation and error paths but not reclamation.kubectl get storemodels and the registry's storage consumption before and after deleting the last referencing ModelDeployment, on a test cluster.
12The preview-to-GA migration path and the GA date. Carried forward from SPIKE-19 UNKNOWN #2, now with a firmer negative.Not published. No migration or upgrade article exists. The Azure preview terms' only GA-transition obligation is a discontinue-and-update-to-GA clause for containers.Only Microsoft can publish this. Ask directly through the preview onboarding channel, which is the same channel the access request opens.

Unknowns 1, 2, 4, 9, and 10 all close in a single extension of the install test ADR-0013 decision 11 already gates: provision, upgrade, remove, inspecting cache and service state at each step. Unknowns 6, 7, 8, 11 need a cluster. Unknowns 3, 5, and 12 need either a live read or Microsoft.

Recommendation ​

  1. Write ADR-0024 to cover lifecycle for both tracks in one record, and open it by stating the gap plainly. Neither track had an update, teardown, or backup story, and for track 3 that is not an omission in this repo's documents but in Microsoft's. The one-line version for the ADR: install is documented for both tracks, lifecycle is documented for neither, and for track 3 the update of the layer underneath is documented in a different doc set that explicitly excludes the workload.

  2. For track 2, add an update path to ADR-0013 rather than leaving decision 1 with an install-only contract. The documented winget upgrade command is unusable for a machine-wide provisioned install, exactly as the documented winget install was. The proposed replacement, and it must be labelled as unverified until UNKNOWN #1 closes, is: stage the newer .msix and its dependency, run Add-AppxProvisionedPackage -Online against it, then foundry service restart, then validate with foundry service status plus one inference call, which is ADR-0013 decision 8's rule applied unchanged to upgrade. Extend the decision 6 script contract with a fifth rule: stage 1 compares the resolved version against an intended version, rather than merely asserting presence. A check that only asks "is it installed" cannot detect an upgrade, a downgrade, or an auto-update, and ADR-0013's current table asks exactly that.

  3. Adopt Azure Machine Configuration as track 2's drift mechanism, and stop describing a scheduled run-command as drift detection. It is the only first-party mechanism that continuously evaluates guest state on Arc-connected Windows Server, reports through Azure Policy, and offers a graduated Audit / Apply and Monitor / Apply and Autocorrect choice. Use Audit or Apply and Monitor, never Autocorrect, because autocorrect against a multi-GB model download is an unbounded action triggered by a transient condition. The idempotent run-command script remains the remediation path, invoked deliberately when the audit reports non-compliance. That split (machine configuration detects, run-command remediates) is a cleaner contract than either mechanism alone and it closes the honest gap in ADR-0013.

  4. For track 3, decide the auto-upgrade posture explicitly, and record the reasoning either way. The documented install command sets --auto-upgrade-minor-version true, which means both ARM extensions upgrade themselves and their version becomes undetectable as drift. The recommendation is to keep auto-upgrade on during preview and pin at GA, because neither extension publishes release notes or a support window, so a pinned preview version is a version nobody can tell you anything about, while an auto-upgraded one at least tracks Microsoft's own testing. Record it as a decision with a named review trigger (GA, or the publication of a version-support statement), not as an inherited default. Set --release-train stable explicitly in both cases, as the documented commands do. And record the --config-protected-settings trap: any partial update deletes omitted settings, which puts the Entra configuration at risk on every extension update.

  5. Make upgrade ordering an explicit, owned deliverable of the wrapper, and treat the AKS Arc upgrade as the primary risk event for track 3. ADR-0014 decision 1 already makes the wrapper own install ordering. Extend it to own an upgrade runbook whose default is conservative because nothing is documented: upgrade one layer at a time, re-run the gatewayclass istio Accepted gate and a live inference call after each, and never change two layers in one window. For the Kubernetes upgrade specifically, record the four documented facts as design constraints: rolling node replacement moves every pod, minor versions cannot be skipped, node images cannot be updated without a version move, and a timed-out upgrade shows Failed while still succeeding. Add a scheduling constraint that Azure Local's six-month currency obligation and monthly cadence make this a recurring event, not a rare one.

  6. Pin the prerequisite layer's versions in source control and check the pinned value, not merely presence. ADR-0014 decision 7 rule 3 already says this. The addition from this spike is why it is not sufficient: helm list sees a chart version and not a --set value, so pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true can be turned off at an unchanged chart version and the check will pass. The drift check must read the istiod deployment's environment, not only the chart version. This is a small change to a script and it closes a real hole.

  7. Set model.catalog.version explicitly in every generated ModelDeployment, and record status.resolvedModel as an observed baseline. The latest default makes the deployed model a moving target that no spec diff can see. Setting it explicitly converts that into ordinary drift. Recording status.resolvedModel.{name,variant,image} alongside the diff catches what remains, including the operator's latest image tag, which is extension configuration rather than per-deployment spec and therefore cannot be fixed the same way. Add a rule that the check tolerates status.state: Updating and the pod churn of a rolling node replacement rather than reporting them as drift.

  8. Decide that backups are not the answer for either track, and name the two exceptions. Both tracks are rebuildable from source control and the catalog, and ADR-0014 decision 6 already delivers most of that. Do not adopt Velero: the only AKS Arc guidance for it targets a different product, Velero is community-maintained rather than first-party, persistent-volume backup on Windows is explicitly unsupported, a restore requires a brand new cluster, and its documented install writes a service principal secret to a plaintext file, which would create the ADR-0005 exception ADR-0014 decision 10 currently and correctly says track 3 does not need. The two things that must be held outside the cluster are the TLS certificate and any private CA material (by name in the tenant Key Vault, never by value) and, for a disconnected deployment, the expansion pack artifacts.

  9. Specify teardown as a separate, explicitly invoked deliverable in both tracks, with the residue enumerated. ADR-0013 decision 6 rule 6 rightly keeps uninstall out of the install script; ADR-0024 should give it its own script rather than leaving it unowned. Track 2: remove the provisioned package, remove the service, and clear the model cache as a distinct confirmed step. Track 3: three scopes, not one, because az k8s-extension delete covers only the ARM layer, cert-manager.crds.keep=true retains CRDs by design, and the whole prerequisite layer was installed outside the extension mechanism. Record the connectivity hazard prominently: deleting an extension against a disconnected cluster removes the Azure record and leaves the workload running, which is the worst half-state available.

  10. Record "preview deployments are disposable" as this project's decision, attributed to this project, and make it testable. Microsoft does not document it, does not promise in-place migration, and has already delivered a breaking data-path change inside the preview with fields silently ignored. That evidence supports the posture; it does not supply the statement. The test of the decision is that every input needed to rebuild is in source control: the cluster definition, the pinned prerequisite versions, the extension configuration, and the generated manifests. Two of those four are not yet, and recommendation 6 puts them there.

Verdict: PROCEED, with lifecycle recorded as an open risk on track 3 and a closable gap on track 2 ​

Stated plainly, because the tasking's pattern asks for a verdict and a hedge is not an answer.

Track 2's lifecycle gap is real and it is closable with work this project controls. The missing update path, the missing drift mechanism, and the missing teardown are all designable now, and four of the five track 2 unknowns close in a single extension of the install test ADR-0013 already gates. Nothing external blocks it. The residual risk is that Add-AppxProvisionedPackage turns out not to upgrade in place, in which case the fallback is remove-then-provision with an explicit cache-preservation step, which is worse operationally but not blocking.

Track 3's lifecycle gap is real and it is not closable by this project. No upgrade documentation exists, no compatibility matrix exists, no version history or support window exists for either extension, no preview-to-GA path exists, and the most likely disruptive event, an AKS Arc rolling upgrade under a running Foundry stack, is documented on one side of the seam and unmentioned on the other. That is not a reason to stop: ADR-0014's first increment is a non-production CPU-only reviewer capability, and ADR-0014 decision 9 already accepted preview-with-no-SLA risk on that basis. It is a reason to name the lifecycle gap explicitly as a production blocker in ADR-0024, so that the decision to go to production later is taken against a stated condition rather than by drift.

The condition to state: track 3 does not carry a production workload until either Microsoft publishes upgrade guidance and a version-support statement for Microsoft.Foundry, or this project has executed a full upgrade cycle (Kubernetes minor version, prerequisite layer, and both extensions) on a non-production cluster and written the runbook itself. The second is achievable without waiting for Microsoft and is the better answer, because it also produces the ordering evidence UNKNOWN #7 needs.

One thing this spike deliberately does not do is guess. Twelve unknowns are recorded rather than resolved, and several of them (the compatibility matrix, the extension version history, the GA path) have no published answer anywhere and cannot be inferred. For a preview product that is the correct outcome, and the correct response is to design for redeploy rather than to design against a migration path that has not been promised.

Sources ​

All first-party Microsoft Learn unless noted. Retrieved 2026-07-30.

Foundry Local, device product (track 2):

Azure Arc, servers (track 2 governance):

Foundry Local on Azure Local (track 3):

Azure Arc, Kubernetes cluster extensions (track 3 platform layer):

Platform lifecycle underneath track 3:

  • Upgrade an Azure Kubernetes Service (AKS) cluster, AKS enabled by Azure Arc (rolling node replacement, no minor-version skipping, the recreate-if-far-behind guidance, no node-image-only updates, the Failed versus Upgrading timeout behaviour, az aksarc get-upgrades and az aksarc upgrade): https://learn.microsoft.com/azure/aks/aksarc/cluster-upgrade
  • About updates for Azure Local, version 23H2 (Modern Lifecycle policy and the six-month currency requirement, the update cadence table, the Arc resource bridge one-year certificate constraint, what the update package covers, the statement that customer workloads are not covered, and the unsupported update interfaces): https://learn.microsoft.com/azure/azure-local/update/about-updates-23h2
  • Back up, restore workload clusters using Velero, AKS enabled by Azure Arc (Velero as an open-source community tool, the "Applies to: AKS on Windows Server" scope, no official Windows support, no volume snapshots in AKS Arc, the requirement for Restic, the inability to restore into an existing cluster, and the service-principal-secret install procedure): https://learn.microsoft.com/azure/aks/aksarc/backup-workload-cluster

Microsoft legal terms (not Learn):

  • Supplemental Terms of Use for Microsoft Azure Previews (no production use for Early Access Previews; the discontinue-and-update-to-GA obligation for preview containers, cited as the closest first-party analogue for a preview-to-GA transition, and explicitly not as a statement about this product): https://azure.microsoft.com/support/legal/preview-supplemental-terms/

No Azure resource was read or written, no az or kubectl command was run, and no software was installed or upgraded in the production of this spike.