DNS is the #1 cause of deploy failures here — read the standard first
Before you publish a web app or add/change any app’s DNS, you MUST read dns-standards — the canonical reference for the public zone, the privatelink two-zone rule (Azure Private DNS and the AD domain controllers), deployment-slot DNS, the FQDN-for-on-prem rule, and the Cisco Umbrella interception gotcha. The command-level how-to lives in §5 and §10 below; the rules live there.
Quick Reference
Need
Solution
Example
Static SPA (React, Vue)
PSI-Wiki-Site
Full-stack app with API
Redbook Dashboard
Internal-only app (VPN)
PSI Explorer
Serverless functions
Azure Functions
API endpoints
Database + API
App Service + SQL
Future ERP apps
0. PSI Web Apps Overview
PSI web applications provide browser-based access to internal tools, replacing or complementing desktop ProApps. All web apps are internal-only (VPN/onsite), Azure-hosted, auto-deployed via GitHub Actions, and share a unified PSI visual identity.
State: React hooks (simple apps) or React Query (API-heavy apps)
React version: React 19 is the standard for new apps. Some existing apps (PSI Explorer, Redbook, Project Explorer, Argo Analytics, ERP Migration) are still on React 18 and migrate opportunistically — don’t assume an existing repo is on 19; check its package.json.
Backend (API): ASP.NET Core 8 + UniData via U2 Toolkit for .NET
Auth: Windows Authentication (NTLM) for API; Swagger/OpenAPI docs
All UI uses the PSI Design System — tokens, components, and patterns live in the psi-design-system repo (assets/ps.css). Consume the --ps-* custom properties rather than hardcoding a palette.
Fonts: the design system self-hosts Inter + JetBrains Mono (ps-fonts.css + vendored .woff2) — no Google Fonts CDN. This matters here: these App Services run behind private endpoints with no public egress, so a CDN font @import would fail silently and fall back to system fonts.
Note: earlier docs referenced a “PSI Blue” #284b63 palette — that predates the green design system and is superseded by #027A54. Don’t anchor to it.
User Access
PSI web apps are registered as Enterprise Applications in Entra ID, appearing in the M365 app launcher (waffle menu) and my.apps.microsoft.com.
App
Entra App ID
PSI Explorer
db5621e9-f3db-495b-ae14-f11d18ba8ad6
PSI Portal
7f929c7f-2483-4206-93b6-11225e07ca85
The PSI Portal serves as the central landing page with app cards, status indicators, and quick stats.
# Ensure publishing creds stay disabled (PSI production baseline)az rest --method put \ --uri "https://management.azure.com/subscriptions/<SUB_ID>/resourceGroups/PS-WEBAPPS/providers/Microsoft.Web/sites/ps-yourapp-dashboard/basicPublishingCredentialsPolicies/scm?api-version=2022-03-01" \ --body '{"properties":{"allow":false}}'az rest --method put \ --uri "https://management.azure.com/subscriptions/<SUB_ID>/resourceGroups/PS-WEBAPPS/providers/Microsoft.Web/sites/ps-yourapp-dashboard/basicPublishingCredentialsPolicies/ftp?api-version=2022-03-01" \ --body '{"properties":{"allow":false}}'# Use GitHub Actions + identity-based deploy (no publish profile)GH_HOST=progressivesurface.ghe.com gh workflow run deploy.yml \ -R ProgressiveSurface/your-repo --ref main
Example deploy step in workflow:
- name: Azure Login (Managed Identity) run: az login --identity- name: Deploy package (async + poll) run: | set -euo pipefail # --async true: az hands the package to OneDeploy and returns. Do NOT # use --async false (or the deprecated `webapp deployment source # config-zip`) for anything but a tiny app — see the warning below. az webapp deploy \ --name ps-yourapp-dashboard --resource-group PS-WEBAPPS \ --src-path server-deploy.zip --type zip --async true TOKEN=$(az account get-access-token --resource https://management.azure.com --query accessToken -o tsv) SCM="https://ps-yourapp-dashboard.scm.azurewebsites.net/api/deployments/latest" for i in $(seq 1 90); do sleep 10 RESP=$(curl -s --max-time 30 -H "Authorization: Bearer $TOKEN" "$SCM" || echo '{}') STATUS=$(echo "$RESP" | python3 -c "import sys,json;print(json.load(sys.stdin).get('status','?'))" 2>/dev/null || echo '?') COMPLETE=$(echo "$RESP" | python3 -c "import sys,json;print(json.load(sys.stdin).get('complete',False))" 2>/dev/null || echo '?') [ "$COMPLETE" = "True" ] && [ "$STATUS" = "4" ] && { echo "Deployed."; exit 0; } [ "$COMPLETE" = "True" ] && [ "$STATUS" = "3" ] && { echo "::error::Kudu deploy failed"; exit 1; } done echo "::error::Deploy did not complete in 15 min"; exit 1- name: Verify site is reachable # Be generous: after Kudu reports deployed, App Service still swaps and # cold-starts the container. 12 minutes covers the swap. run: | for i in $(seq 1 48); do STATUS=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 https://yourapp.progressivesurface.com/api/health || echo "000") case "$STATUS" in 200|401|403) echo "Reachable ($STATUS)"; exit 0;; esac sleep 15 done echo "Health gate failed"; exit 1 timeout-minutes: 14
Do not deploy synchronously — the 4-minute 504
az webapp deploy --async false (and the deprecated webapp deployment source config-zip) hold one synchronous HTTP call open to
*.scm.azurewebsites.net for the entire Kudu extraction + container
restart. Azure Front Door caps that call at ~4 minutes — any app
whose deploy runs longer gets a 504 GatewayTimeout, and worse, the
deploy is left half-applied. PSI DataSync hit this hard on 2026-05-18:
three synchronous deploys in quick succession all 504’d, one left Kudu
mid-extraction, and the site crash-looped for ~18 minutes.
Always use --async true and poll the Kudu deployment record
(/api/deployments/latest: status 4 = success, 3 = failed) as shown
above. The slow phase then runs server-side with nothing waiting
synchronously through it.
Serialise deploys — the 409 that ships a commit behind
Kudu accepts one zip deployment at a time and 409s the loser:
Deployment endpoint responded with status code 409 / There may be an ongoing deployment. Two merges landing inside a deploy window are enough.
psi-portal hit this on 2026-07-30: PRs #114 and #112 merged 14 seconds
apart, both deploys built and applied schemas, and the run carrying both
merges lost the race while the run carrying only the older one won. master
was green, production was a commit behind, and the only signal was a red X
on a run whose build had passed. Every deploy workflow needs:
cancel-in-progress: false is the important half — cancelling a
half-uploaded zip deployment is how wwwroot ends up partially written.
Queueing serialises the runs in merge order, so the newest commit lands
last.
Make the health gate prove the commit, not the status code
curl-ing a health URL for a 200 verifies less than it looks like:
the old container answers 200 for the entire swap window, so the gate
passes before the new build is serving;
on an SPA, if the health route doesn’t exist the catch-all returns
200 text/html and the gate passes with the API completely down. That was
literally true of psi-portal’s /api/health until 2026-07-30 — the route
was never defined.
Stamp the commit into the package at build time and compare it:
# in the packaging stepprintf '{"sha":"%s","builtAt":"%s","run":"%s"}\n' \ "$GITHUB_SHA" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$GITHUB_RUN_ID" > deploy/build-info.json
Serve it from an unauthenticated /api/health (registered above the SPA
catch-all), then poll until sha equals GITHUB_SHA. A boot failure or a
stuck swap then fails the run instead of going green over a 503. Reference
implementation: psi-portal/server/lib/buildInfo.js + its deploy.yml.
Step 3: Trigger Deployment and Watch Run
# Trigger manually when needed (supports one-app-at-a-time validation)GH_HOST=progressivesurface.ghe.com gh workflow run deploy.yml \ -R ProgressiveSurface/your-repo --ref main# Follow the latest runGH_HOST=progressivesurface.ghe.com gh run list \ -R ProgressiveSurface/your-repo --workflow deploy.yml --limit 1GH_HOST=progressivesurface.ghe.com gh run watch <run-id> -R ProgressiveSurface/your-repo
Step 4: Configure Startup Command
For Python/Streamlit apps:
az webapp config set \ --name ps-yourapp-dashboard \ --resource-group PS-WEBAPPS \ --startup-file "python -m streamlit run app.py \ --server.port 8000 \ --server.address 0.0.0.0 \ --server.headless true \ --browser.gatherUsageStats false"
For Node.js apps:
az webapp config set \ --name ps-yourapp-dashboard \ --resource-group PS-WEBAPPS \ --startup-file "npm start"
PSI-specific: When an App Service uses publicNetworkAccess=Disabled (i.e. all PSI internal apps following the private-endpoint pattern in §5), adding a deployment slot requires three additional setup steps that the slot does not inherit from the main slot. PRGJSMES hit all three the first time it added a staging slot — this section captures them so future apps don’t have to.
Why use slots
Slots enable blue-green deploys via slot swap: build → deploy to staging slot → smoke test → atomic swap to production. Rollback = re-swap (the previous code is still alive in the slot). Zero-downtime. Required tier: Standard or higher (Premium recommended for prod). See CD for the full pattern in action.
What a slot does NOT inherit from the main app
When you create a slot with --configuration-source <main-app>, you get a copy of app settings + connection strings only. You do not get:
Resource
Inherited?
Why it matters
Custom domains (psmes.progressivesurface.com)
❌ No
Slot only has <app>-<slot>.azurewebsites.net
Private endpoint
❌ No
Slot is unreachable from inside VNet without its own PE
Privatelink DNS records
❌ No (auto-registered for the new PE only if zone group attached)
Without records, runner can’t resolve <app>-<slot>.scm.azurewebsites.net
DNS records on PS-AZ-DC01 / PS-GR-DC02
❌ No (manual step)
Without DC records, runner queries fail (DCs are the actual resolver, not Azure DNS)
System-assigned Managed Identity
❌ No — slot has its own MI
Slot’s MI starts with zero DB access, app crashes 503 with CanConnectAsync failure
App Service basic publishing creds policy
Yes (inherits)
—
VNet integration
Yes (inherits from plan)
—
Setup checklist for a new slot
When publicNetworkAccess=Disabled, run these in order. Each is required.
The slot’s Web (and SCM) endpoints are a separate sub-resource of the App Service, so they need a separate PE targeting the sites-<slotname> group ID. Use the next sequentially available IP in the PS-ProdData subnet (see Private Endpoint Subnet Allocation).
4. Add DNS records on the AD DCs (PS-AZ-DC01 + PS-GR-DC02)
Slots without a custom domain need two DNS layers on the DC, depending on who’s connecting:
4a. Privatelink zone records — for traffic from inside the VNet
VNet clients (CI runner, App Service, other Azure resources) resolve <app-name>-<slot>.azurewebsites.net by following the public CNAME chain that ends at <app-name>-<slot>.privatelink.azurewebsites.net. The DC’s privatelink.azurewebsites.net primary zone provides the final A record.
4b. Per-app primary zone — for browser access from outside the VNet (PSI office / VPN)
The privatelink CNAME chain only resolves correctly when public DNS sees the privatelink.* CNAME on the slot. For deployment slots (no custom domain bound), Azure’s public CNAME chain does not always end at privatelink — meaning a query from your laptop via the PSI VPN gets the public IP and hits a 403 Forbidden from publicNetworkAccess=Disabled.
Fix: create a per-app primary zone on the DC that short-circuits the lookup directly to the private IP. This matches the existing PSI pattern used for ps-redbook-dashboard.azurewebsites.net (verify with Get-DnsServerZone -Name "ps-redbook-dashboard.azurewebsites.net").
# Run on PS-GR-DC02 (replicates to PS-AZ-DC01)# Apex zone for the slot's web endpointAdd-DnsServerPrimaryZone ` -Name "<app-name>-<slot>.azurewebsites.net" ` -ReplicationScope DomainAdd-DnsServerResourceRecordA ` -ZoneName "<app-name>-<slot>.azurewebsites.net" ` -Name "@" ` -IPv4Address "<PE-IP>" ` -TimeToLive 01:00:00# Apex zone for the slot's SCM (Kudu / deploy) endpointAdd-DnsServerPrimaryZone ` -Name "<app-name>-<slot>.scm.azurewebsites.net" ` -ReplicationScope DomainAdd-DnsServerResourceRecordA ` -ZoneName "<app-name>-<slot>.scm.azurewebsites.net" ` -Name "@" ` -IPv4Address "<PE-IP>" ` -TimeToLive 01:00:00
Verify both layers
Wait for AD replication (15 min within site, up to 3 hours cross-site) or run repadmin /syncall /AeP to force. Verify from any VNet machine and from a PSI VPN-connected laptop:
Resolve-DnsName -Name "<app-name>-<slot>.azurewebsites.net" -Server "10.160.0.5"# Should return <PE-IP>, NOT a public Azure IPResolve-DnsName -Name "<app-name>-<slot>.scm.azurewebsites.net" -Server "10.160.0.5"# Should return <PE-IP>
If the per-app primary zone exists, it takes precedence and the privatelink CNAME chain is bypassed entirely. The privatelink zone records remain useful for VNet-internal CNAME resolution if the per-app zone is ever removed. Both layers can coexist without conflict.
Workstation gotcha: Cisco Umbrella intercepts DNS
Verifying slot DNS from a PSI workstation can mislead you — Cisco Umbrella intercepts the query (even with -Server) and hands back the public chain while the DC has the right answer. Verify from the DC or a VNet machine instead. Full explanation + fix options: workstation-gotcha-cisco-umbrella.
5. Enable system-assigned Managed Identity on the slot
az webapp identity assign \ -g PS-WEBAPPS -n <app-name> --slot staging# Capture the principalId from the output
6. Grant the slot’s MI access to dependencies (SQL, Key Vault, etc.)
For Azure SQL with Authentication=Active Directory Default:
-- Run against your DB as a db_owner / Entra adminCREATE USER [<app-name>/slots/staging] FROM EXTERNAL PROVIDER;ALTER ROLE db_ddladmin ADD MEMBER [<app-name>/slots/staging];ALTER ROLE db_datareader ADD MEMBER [<app-name>/slots/staging];ALTER ROLE db_datawriter ADD MEMBER [<app-name>/slots/staging];
The user name format is literal: <appname>/slots/<slotname>. This is how Azure exposes a slot’s MI to Azure SQL.
For Key Vault, grant the slot’s principalId Reader / Secret User access at the appropriate scope.
7. Verify end-to-end
After all 6 steps, run a deploy via the slot-swap workflow. The deploy should:
Reach the staging SCM endpoint over the VNet (proves DNS + PE)
Land the build (proves slot-level publishing perms)
Smoke test against <app-name>-staging.azurewebsites.net/api/health and get 200 (proves slot’s MI can reach SQL)
Swap into production (proves swap mechanics)
Symptoms when a step is missing
Step missed
Failure mode
2 (PE)
Deploy fails immediately: 403 Forbidden from SCM (slot is unreachable)
3 (DNS zone group)
Records missing in Azure private DNS zone — runner sees Name or service not known
4a (DC privatelink records)
Same Name or service not known — runner can’t resolve via CNAME chain
4b (DC per-app primary zone)
CI deploys work, but browser access from PSI office / VPN gets 403 x-ms-forbidden-ip — your machine hit Azure’s public IP because public DNS didn’t return a privatelink CNAME
5 (MI)
App boots but health endpoint returns 503 (CanConnectAsync fails)
6 (SQL grant)
App boots but health returns 503 (MI exists but can’t auth to SQL)
If you see a 503 from a freshly-deployed slot, check the slot’s MI exists and has the right SQL/KV grants before going deeper.
Auto-registered by the PE’s DNS zone group in ps-rg-01/privatelink.azurewebsites.net
5. PSI Network Architecture (Private Endpoints)
PSI-specific: This section applies to internal-only web applications accessed through the PSI VPN. All production internal apps should use private endpoints to prevent public internet exposure.
Architecture Overview
Internal Users (VPN) --> DNS (10.160.0.5) --> Private Endpoint IP --> Azure App Service
|
VNet: PS-VNMAIN
Subnet: PS-SERVERS (private endpoints)
Subnet: PS-WebApps (VNet integration)
⚠️ Reaching on-prem from Azure: the Meraki route table
A new subnet has no on-prem reachability by default, and nothing about the app’s configuration
hints at it. Budget an hour of confusion if you skip this.
PSI has no Azure VPN gateway. There are no virtualNetworkGateways, connections or
localNetworkGateways in the subscription — checking for them and concluding “there is no VPN” is
wrong. The Azure↔on-prem path is a Cisco Meraki vMX appliance:
Thing
Value
Appliance VM
PS-AZ-VMXM (in the Meraki managed RG mrg-cisco-meraki-vmx-…)
Because those are user-defined routes, they apply only to subnets the table is attached to.
Check before assuming, and attach it if your app must reach on-prem:
# who currently has on-prem reachability?az network route-table show -g PS-RG-01 -n MerakiRouteTableAZ-GR --query "subnets[].id" -o tsv# give a subnet on-prem reachabilityaz network vnet subnet update -g PS-RG-01 --vnet-name PS-VNMAIN -n <subnet> \ --route-table MerakiRouteTableAZ-GR
It takes two changes, one on each side. The route table only decides that packets leave toward the
appliance. The vMX advertises a fixed list of local subnets into Meraki AutoVPN, and on-prem has a
return route only for what is on that list — so a new subnet needs both:
Side
Change
Where
Azure
attach MerakiRouteTableAZ-GR to the subnet
az network vnet subnet update (above)
Meraki
add the subnet to the vMX site’s VPN local networks
Meraki Dashboard → the Azure vMX network → Security & SD-WAN → Site-to-site VPN → Local networks → add the CIDR with “Use VPN: yes”. API equivalent: PUT /networks/{networkId}/appliance/vpn/siteToSiteVpn
Miss the Azure side and the connection never leaves; miss the Meraki side and it leaves but nothing
comes back. Both look identical from the app: a connect timeout.
Advertising the whole VNet (10.160.0.0/16) on the vMX rather than each subnet individually means
future subnets only need the Azure-side attach. Everything in that space is PSI’s own.
Worked example (2026-08-07). IT Helper’s AFTEC lookup called
https://api.progressivesurface.com (PS-PROXY, 10.150.141.9) and hung for 101 seconds, dying on
HttpClient’s 100-second default. DNS was fine — the record and both VNet resolvers returned the right
address. The app was VNet-integrated with vnetRouteAllEnabled. The cause was that
ps-flexfunc (10.160.151.0/26) was not attached to MerakiRouteTableAZ-GR, so packets had no next
hop toward on-prem and were silently blackholed — and the vMX was not advertising that /26, so even
once they reached the appliance there was no return route.
A correct-looking DNS answer proves nothing about reachability here — resolution and routing fail
independently, and this failure mode looks exactly like a DNS problem. Distinguish them: a resolvable
name that times out on connect is a routing symptom.
Service endpoints are a different mechanism for a different job: they cover firewalled Azure
services (Key Vault, SQL). They do nothing for on-prem.
Add Privatelink DNS Records (Azure Private DNS zone and DC primary zone)
Yes
4
Disable Public Access
Yes
5
Add Custom Domain
Yes
6
Add DNS Verification TXT
Yes
7
Create DNS A Record (internal IP)
Yes
8
Enable HTTPS Only
Yes
9
Bind Wildcard SSL Certificate
Yes
10
Configure API CORS
Yes (if calling API)
11
Update Wiki Documentation
Yes
Private Endpoint Subnet Allocation
Private endpoints are allocated from the PS-ProdData subnet (10.160.140.0/24). IPs were assigned sequentially as endpoints were created rather than by service type. The current allocation is:
Key Vault (PE, zone privatelink.vaultcore.azure.net)
10.160.140.33
ps-bom-coverage 🚧
App Service (Web)
10.160.140.34
ps-ach-web
App Service (Web)
10.160.140.35+
Available for new endpoints
—
Convention: Assign the next available IP sequentially — the next free address is 10.160.140.35. ⚠ The allocation is not webapp-only: .23 and .32 are the ps-certificates-kv and psi-finance-kv private endpoints (in the privatelink.vaultcore.azure.net zone, so they aren’t visible from the privatelink.azurewebsites.net records) — always check live NIC allocations across the whole subnet, not just the webapp DNS zone, before pinning an IP. (psi-service hit this on 2026-06-29: .23 looked free in the azurewebsites zone but was already taken by the KV PE.) ⚠ A NIC scan that reads only ipConfigurations[0] will also miss .28, a second ipconfig on ps-iot-bridge — cross-check the allocation table. The legacy PS-SERVERS subnet (10.160.0.0/23) also hosts older private endpoints (psi-portal .6, bom-explorer-web .17, psi-datasync .19, psargostorage .11) — new endpoints should use PS-ProdData.
Re-verified against live NIC ipConfigurations 2026-08-05 (az network nic list, filtered to 10.160.140.*), which is the check this warning asks for. Two corrections came out of it: .8, .19 and .20 were live but undocumented — they are additional ipconfigs on the single AzureIOT_PE endpoint fronting PSTestHub, so one IoT Hub PE holds four addresses (.7, .8, .19, .20). Nothing was allocated above .30, so ps-sop took .31. A private endpoint can carry more than one ipconfig — count addresses, not endpoints, when looking for a free IP.
FQDN Required: When your App Service needs to reach on-premises servers (e.g., PS-PROXY, AFTEC), you must use the fully-qualified domain name (ps-proxy.ad.ptihome.com) — not the short hostname (ps-proxy). App Service Linux containers don’t have the ad.ptihome.com DNS search domain. See PSI-Specific Issues for details.
Step 3: Create Private Endpoint
# Get App Service resource IDAPP_ID=$(az webapp show -n your-app-name -g PS-WEBAPPS --query "id" -o tsv)# Create private endpoint in PS-SERVERS subnet# Note: PS-WebApps subnet is delegated and cannot host private endpointsMSYS_NO_PATHCONV=1 az network private-endpoint create \ --name your-app-name-pe \ --resource-group PS-WEBAPPS \ --subnet "/subscriptions/1f3a4b35-1cf1-4fea-af63-b3fc0d11acdf/resourceGroups/PS-RG-01/providers/Microsoft.Network/virtualNetworks/PS-VNMAIN/subnets/PS-SERVERS" \ --private-connection-resource-id "$APP_ID" \ --group-id sites \ --connection-name your-app-name-connection \ --location "North Central US"
Note the private endpoint IP from the output (customDnsConfigs[0].ipAddresses[0]).
Step 4: Disable Public Network Access
Two methods exist for blocking public access:
Method 1: publicNetworkAccess=Disabled (recommended for private-endpoint apps)
Fully blocks public network at the resource level. No exceptions possible.
MSYS_NO_PATHCONV=1 az resource update \ --ids "$APP_ID" \ --set properties.publicNetworkAccess=Disabled
Method 2: Access restriction rules (when you need selective access)
Uses access restriction rules — blocks by default but allows adding exceptions (e.g., specific IPs).
az webapp config access-restriction set --name your-app-name \ --resource-group PS-WEBAPPS --default-action Deny
Use Method 1 for all private-endpoint apps. Use Method 2 only when you need to allow specific public IPs (e.g., GitHub Actions runners) while blocking general access.
Step 4b: Add Privatelink DNS Records
Required when using publicNetworkAccess=Disabled with self-hosted GitHub Actions runners. Without these records, the runner cannot resolve your-app-name.scm.azurewebsites.net to deploy.
The privatelink.azurewebsites.net private DNS zone (in ps-rg-01) is linked to PS-VNMAIN. When a machine on the VNet resolves your-app-name.azurewebsites.net, Azure DNS appends privatelink and checks this zone. If no record exists, the name resolves to the public IP — which is blocked.
Create two A records: one for the app, one for the SCM (deployment) endpoint:
# App recordaz network private-dns record-set a create \ -g ps-rg-01 -z privatelink.azurewebsites.net \ -n your-app-name --ttl 3600az network private-dns record-set a add-record \ -g ps-rg-01 -z privatelink.azurewebsites.net \ -n your-app-name -a "10.160.0.XX" # Use your private endpoint IP# SCM (deployment) record — REQUIRED for GitHub Actions deploysaz network private-dns record-set a create \ -g ps-rg-01 -z privatelink.azurewebsites.net \ -n your-app-name.scm --ttl 3600az network private-dns record-set a add-record \ -g ps-rg-01 -z privatelink.azurewebsites.net \ -n your-app-name.scm -a "10.160.0.XX" # Same PE IP
Current Privatelink DNS Records
Record
IP
Zone
bom-explorer-web
10.160.0.17
privatelink.azurewebsites.net
bom-explorer-web.scm
10.160.0.17
privatelink.azurewebsites.net
psi-portal
10.160.0.6
privatelink.azurewebsites.net
psi-portal.scm
10.160.0.6
privatelink.azurewebsites.net
ps-project-explorer
10.160.140.10
privatelink.azurewebsites.net
ps-project-explorer.scm
10.160.140.10
privatelink.azurewebsites.net
erp-migration-api
10.160.140.9
privatelink.azurewebsites.net
erp-migration-api.scm
10.160.140.9
privatelink.azurewebsites.net
prgjsmes-prod
10.160.140.11
privatelink.azurewebsites.net
prgjsmes-prod.scm
10.160.140.11
privatelink.azurewebsites.net
redbook-web
10.160.140.12
privatelink.azurewebsites.net
redbook-web.scm
10.160.140.12
privatelink.azurewebsites.net
ps-redbook-dashboard
10.160.140.14
privatelink.azurewebsites.net
ps-redbook-dashboard.scm
10.160.140.14
privatelink.azurewebsites.net
ps-argo-analytics
10.160.140.16
privatelink.azurewebsites.net
ps-argo-analytics.scm
10.160.140.16
privatelink.azurewebsites.net
prgjsmes-prod-staging
10.160.140.18
privatelink.azurewebsites.net
prgjsmes-prod-staging.scm
10.160.140.18
privatelink.azurewebsites.net
psi-service
10.160.140.24
privatelink.azurewebsites.net
psi-service.scm
10.160.140.24
privatelink.azurewebsites.net
ps-docgen
10.160.140.25
privatelink.azurewebsites.net
ps-docgen.scm
10.160.140.25
privatelink.azurewebsites.net
ps-sop
10.160.140.31
privatelink.azurewebsites.net (Azure zone only — DC pending, psi-azure-admin#4)
ps-sop.scm
10.160.140.31
privatelink.azurewebsites.net (Azure zone only — DC pending, psi-azure-admin#4)
⚠ Gap:ps-progressive-view (10.160.140.15) — Azure Private DNS zone records added (2026-03-24). DC primary zone records still needed before runner can deploy.
Note: The PSI Explorer repo was renamed from bom-explorer-web to psi-explorer-web, but the Azure App Service resource name remains bom-explorer-web. All DNS and PE records correctly use bom-explorer-web to match the Azure resource name.
Verify:nslookup your-app-name.azurewebsites.net 10.160.0.5 should return the private endpoint IP when resolved from the VNet.
# Create A record pointing to private endpoint IPaz network dns record-set a create \ -g ps-rg-01 -z progressivesurface.com \ -n "yourapp" --ttl 3600az network dns record-set a add-record \ -g ps-rg-01 -z progressivesurface.com \ -n "yourapp" -a "10.160.0.XX" # Use your private endpoint IP
The runner has no SSH key on file — admin access uses Azure RunCommand, which authenticates via your az login identity. No password, no key.
# Run any shell command as rootMSYS_NO_PATHCONV=1 az vm run-command invoke \ -g PS-RG-01 -n ps-cicd-runner \ --command-id RunShellScript \ --scripts "<your command>"
Example (one-time install of Playwright Chromium system deps for E2E tests, done 2026-05-11). The list below is the complete canonical set — the initial install missed libasound2t64 (ALSA, required even in headless) and several transitively-loaded libs, which caused the first E2E run on PR #69 to fail with libasound.so.2: cannot open shared object file:
To verify after install, the chrome-headless-shell binary under /home/runner/.cache/ms-playwright/ should run --version cleanly and ldd <binary> | grep 'not found' should return nothing.
Runner DNS Configuration
The runner uses systemd-resolved with two drop-in configs in /etc/systemd/resolved.conf.d/: ad-domain.conf (pins ad.ptihome.com / ptihome.com to the DCs) and privatelink.conf (routes privatelink.* to Azure DNS, 168.63.129.16).
The deploy runner resolves through its privatelink.conf drop-in → Azure DNS (168.63.129.16), so it relies on the Azure Private DNS zone record. This is what unblocks deploys.
Other VNet/VPN clients (Windows machines, etc. — no drop-in) resolve the azurewebsites.net hostname via the DCs, so they rely on the DC AD-integrated primary-zone record.
Verified 2026-06-25: with only the Azure-zone record present, the runner resolved ps-buildvsbuy.scm.azurewebsites.net → 10.160.140.5 correctly while the DCs still returned NXDOMAIN (DC record pending replication). So the runner does not depend on the DC record — but other clients do. Create both.
AD domain fallback issue: Some public DNS records (e.g., api.progressivesurface.com) use CNAMEs pointing to internal AD hostnames (e.g., ps-proxy.ad.ptihome.com). When the DC DNS is slow or briefly unreachable, systemd-resolved falls back to Azure DNS (168.63.129.16), which resolves ad.ptihome.com names to incorrect public IPs. This causes intermittent TLS failures.
Fix: A drop-in file pins AD domain queries to the DCs:
# /etc/systemd/resolved.conf.d/ad-domain.conf# Route AD domain queries to internal DCs — never fall back to Azure DNS# Prevents ps-proxy.ad.ptihome.com from resolving to public IPs[Resolve]DNS=10.160.0.5 192.9.200.110Domains=~ad.ptihome.com ~ptihome.com
After changes: sudo systemctl restart systemd-resolved
DNS Resolution Chain (example)
api.progressivesurface.com (public zone)
→ CNAME: ps-proxy.ad.ptihome.com
→ ~ad.ptihome.com routes to DC (10.160.0.5)
→ A: 192.9.201.217 (PS-PROXY internal IP via VPN) ✓
bom-explorer-web.scm.azurewebsites.net
→ CNAME: psi-explorer-web.scm.privatelink.azurewebsites.net
→ DC checks local authoritative zone for privatelink.azurewebsites.net
→ A: 10.160.0.17 (private endpoint IP) ✓
Wildcard Certificate (Key Vault)
The wildcard SSL certificate is stored in Azure Key Vault for centralized management.
Property
Value
Key Vault
ps-certificates-kv
Certificate Name
wildcard-progressivesurface
Subject
*.progressivesurface.com
Thumbprint
8ECD7C39FA4BD44E10D3D89A80EF33F3922A291A
Expires
2027-02-03
Vault URI
https://ps-certificates-kv.vault.azure.net/
Key Vault Access Policies
Principal
Permissions
Azure Web Sites (abfa0a7c-a6b6-4736-8310-5855508787cd)
Important: The CLI commands above fail with Cannot use auth v2 commands when the app is using auth v1 if the app has any existing v1 auth config (even if disabled). Use the REST API to bypass this:
# Replace <SUB_ID>, <APP_NAME>, and <CLIENT_ID> with your valuesaz rest --method PUT \ --url "/subscriptions/<SUB_ID>/resourceGroups/PS-WEBAPPS/providers/Microsoft.Web/sites/<APP_NAME>/config/authsettingsV2?api-version=2021-02-01" \ --body '{ "properties": { "platform": { "enabled": true }, "globalValidation": { "unauthenticatedClientAction": "RedirectToLoginPage", "redirectToProvider": "azureactivedirectory" }, "identityProviders": { "azureActiveDirectory": { "enabled": true, "registration": { "openIdIssuer": "https://login.microsoftonline.com/a83ae943-0a50-49cc-83c3-479b7a44b7fb/v2.0", "clientId": "<CLIENT_ID>", "clientSecretSettingName": "MICROSOFT_PROVIDER_AUTHENTICATION_SECRET" }, "validation": { "allowedAudiences": ["api://<CLIENT_ID>"] } } }, "login": { "tokenStore": { "enabled": true } } } }'# Then set the client secret as an app settingaz webapp config appsettings set \ --name <APP_NAME> \ --resource-group PS-WEBAPPS \ --settings "MICROSOFT_PROVIDER_AUTHENTICATION_SECRET=<CLIENT_SECRET>"
EasyAuth Headers
After authentication, Azure injects these headers into every request reaching your app:
Note: The ERP Migration Tool originally used EasyAuth (Feb 2026) but later migrated to MSAL.js for more control over the auth flow. See the ERP Migration Tool auth docs for the MSAL pattern with global fetch override.
MSAL Configuration (For Pure SPAs Without a Backend)
If your app is a pure SPA (Static Web App, no Express backend), use MSAL instead:
App Service with SPA + API (e.g., React + Express)
MSAL
More control, global fetch override pattern, avoids EasyAuth config issues
Static Web App (pure SPA)
MSAL
No server to read headers from
Need to call Microsoft Graph from client
MSAL
Need access tokens in the browser
EasyAuth Session Defaults
Setting
Default Value
Cookie expiration
8 hours
Token refresh window
72 hours
Token store
Enabled (file system)
Require HTTPS
Yes
7. Secrets Management with Key Vault
ps-certificates-kv uses access policies, not RBAC
Grant its identities with az keyvault set-policy --object-id <oid> --secret-permissions get. Assigning the Key Vault Secrets Userrole on this vault grants control-plane visibility and no secret access at all — the app (or person) can see the vault and read nothing, with no error to explain why. This is the most common Key Vault access ticket at PSI. The diagram below labels the arrow “RBAC Access” and is wrong on that point; the mechanism it shows — managed identity → vault → app setting reference — is correct.
The RBAC model in this section does apply to psi-finance-kv. See key-vaults for which vault to use, both grant syntaxes, and the failure→cause table.
cd C:\GIT\PSI.UniData.APIgit add src/PSI.UniData.API/appsettings.jsongit commit -m "Add CORS origin for yourapp.progressivesurface.com"git push origin master
The GitHub Actions runner on PS-PROXY will automatically deploy the change.
Verifying CORS
# Check if CORS headers are returned for your origincurl -s -H "Origin: https://yourapp.progressivesurface.com" \ "https://api.progressivesurface.com/api/health" -I | grep -i access-control# Should return:# Access-Control-Allow-Credentials: true# Access-Control-Allow-Origin: https://yourapp.progressivesurface.com
App Service deployment hardening baseline (April 2026)
No local-git or publish-profile deploys for production App Services.
SCM/FTP basic publishing credentials disabled (scm=false, ftp=false) unless a time-bound exception is approved.
Identity-based deploy only (self-hosted runner managed identity or Entra federated credential).
Manual workflow_dispatch supported for one-app-at-a-time controlled rollouts and validation.
Health gates for protected endpoints must treat 200, 401, and 403 as reachable depending on endpoint auth posture.
Auto-Versioning
PSI web apps should auto-increment their version number on every deploy so users always know which build they’re running. The pattern used by ERP Migration Tool (the reference implementation):
How It Works
Push to main → Tests pass → npm version patch → Vite injects version → Deploy → Commit version bump back
CI bumps the patch version before building:
- name: Bump patch version run: | cd client npm version patch --no-git-tag-version echo "APP_VERSION=$(node -p "require('./package.json').version")" >> $GITHUB_ENV
Bot commits the bump back to the repo after successful deploy:
- name: Commit version bump run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.progressivesurface.ghe.com" git add package.json package-lock.json git commit -m "v${APP_VERSION} [skip ci]" || echo "No version change to commit" git push
The [skip ci] in the commit message prevents an infinite deploy loop.
Adoption Status
App
Auto-Version
Notes
ERP Migration Tool
Yes
Reference implementation (v1.27.x)
PSI Explorer
No
Hardcoded v1.2 in App.tsx — needs migration
Redbook Dashboard
No
—
PSI Portal
No
—
PSI Wiki
N/A
Static site, no version displayed
Adding Auto-Versioning to an Existing App
Sync package.json version to the current UI version (e.g. 1.2.0)
Add define: { __APP_VERSION__: ... } to vite.config.ts
Add declare const __APP_VERSION__: string to vite-env.d.ts
Replace hardcoded version string in the UI with __APP_VERSION__
Add the “Bump patch version” + “Commit version bump” steps to the GitHub Actions workflow
Ensure the workflow has permissions: contents: write for the commit-back step
Dependabot Configuration
Canonical policy lives in dependency-update-policy — security PRs merge immediately; routine version updates run behind a release-age cooldown. Dependabot is the only tool (the planned Renovate rollout was dropped 2026-07-27), and the cooldown config was rolled out to all active repos on 2026-07-27.
Every new repo needs .github/dependabot.yml copied from the template in dependency-update-policy. Key points:
Security updates (CVE/GHSA) are never delayed — verify Repo Settings → Code security → Dependabot security updates = ENABLED.
Version updates use cooldown (npm: 30/14/7 days for major/minor/patch; other ecosystems: 14 days) with minor+patch bumps grouped into one weekly PR.
Config is per-repo — GitHub has no org-wide dependabot.yml, so copy the template rather than expecting inheritance.
10. Self-Hosted GitHub Actions Runner
Why
GitHub-hosted runners cannot reach apps behind private endpoints (public network access disabled). A self-hosted runner on the PSI internal network can deploy directly.
Setup
The self-hosted GitHub Actions runner is an Azure Linux VM (ps-cicd-runner) on the PS-SERVERS subnet with managed identity.
Property
Value
VM Name
ps-cicd-runner
Resource Group
PS-RG-01
Subnet
PS-SERVERS (10.160.0.9)
OS
Ubuntu Linux
Auth
Azure Managed Identity
Runner Labels
[self-hosted, psi-internal]
Which Apps Use It
App
Why
UniData API
Deployed to App Service with private endpoint; needs internal network for az webapp deploy
Project Explorer
Private endpoint on PS-ProdData subnet
PRGJSMES (MES)
Private endpoint on PS-ProdData subnet
PSI Explorer
Private endpoint, publicNetworkAccess=Disabled
Redbook Web
Private endpoint on PS-ProdData subnet, publicNetworkAccess=Disabled
Redbook Dashboard
Private endpoint on PS-ProdData subnet, publicNetworkAccess=Disabled
ERP Migration Tool
Private endpoint on PS-ProdData subnet, publicNetworkAccess=Disabled
Progressive Data View
Private endpoint on PS-ProdData subnet, publicNetworkAccess=Disabled
ARGO Analytics
Private endpoint on PS-ProdData subnet, publicNetworkAccess=Disabled
DNS Configuration for Private Endpoints
Canonical DNS rules + new-app checklist live in dns-standards. This section is the command-level how-to and PSI-specific background; dns-standards is the authoritative “what records, which zone, and why.”
Critical: When publicNetworkAccess=Disabled, the SCM deployment endpoint is only reachable through the private endpoint. DNS must resolve *.privatelink.azurewebsites.net to private IPs, not public ones.
Why (short): a conditional forwarder for privatelink.azurewebsites.net does not work — Windows DNS trusts the complete public CNAME chain returned for app.azurewebsites.net and never consults the forwarder. Fix (implemented 2026-02-17): AD-integrated primary zones for the privatelink.* domains on the DCs, so the DC is authoritative and resolves the CNAME against its own A records. (Full reasoning + the runner-reads-Azure-zone vs. other-clients-read-DC-zone model is in dns-standards.)
Zones configured on PS-AZ-DC01 (Forest-replicated):
bom-explorer-web.scm.azurewebsites.net
→ CNAME: psi-explorer-web.scm.privatelink.azurewebsites.net
→ DC checks local authoritative zone for privatelink.azurewebsites.net
→ A: 10.160.0.17 (private endpoint IP) ✓
⚠ Maintenance — both zones must be updated:
When adding a new App Service private endpoint, add A records in two places — each serves a different resolver path:
Azure Private DNS zone (privatelink.azurewebsites.net in PS-RG-01) — via Azure CLI or Portal. The deploy runner resolves via this (its privatelink.conf drop-in routes privatelink.* to Azure DNS); missing it = deploy NXDOMAIN/403.
DC primary zone on PS-AZ-DC01 (Forest-replicated) — via DNS Manager or Add-DnsServerResourceRecordA. Other VNet/VPN clients (no drop-in) resolve the azurewebsites.net hostname via the DCs and rely on this.
Add both the app name record (your-app-name) and the SCM record (your-app-name.scm), pointing to the same private endpoint IP. Missing the SCM record is the most common cause of deploy failures.
Adding Records to the DC Primary Zone
# Run on PS-AZ-DC01 or via remote PowerShellAdd-DnsServerResourceRecordA -ZoneName "privatelink.azurewebsites.net" ` -Name "your-app-name" -IPv4Address "10.160.140.XX"Add-DnsServerResourceRecordA -ZoneName "privatelink.azurewebsites.net" ` -Name "your-app-name.scm" -IPv4Address "10.160.140.XX"
Adding a New Privatelink Zone Type
If you create a private endpoint for a new Azure service type (e.g., Blob Storage → privatelink.blob.core.windows.net, Key Vault → privatelink.vaultcore.azure.net), you need to:
Create the Azure Private DNS zone in PS-RG-01 and link it to PS-VNMAIN
Create a matching AD-integrated primary zone on PS-AZ-DC01 (Forest-replicated)
Add A records to both zones for each endpoint
Without the DC primary zone, the same Windows DNS CNAME chain problem applies — the DC will follow the public DNS chain instead of resolving to the private IP. See DNS Configuration for Private Endpoints for background on the root cause.
Workflow Usage
runs-on: [self-hosted, psi-internal]
For detailed setup and maintenance instructions, see the UniData API docs (deployment section).
11. Security Compliance Checklist
Automated verification: The psi-azure-admin agent runs a full compliance scan every Sunday (and on demand) covering all resource types across the subscription. Before launching a new app, confirm it will pass its first automated audit: SCM/FTP disabled, httpsOnly=true, public access disabled (or documented exception), no plain-text secrets, managed identity present.
Pre-Deployment Requirements (S-REQ)
ID
Requirement
How to Verify
Commands
S-REQ-01
Authentication via Entra ID
Check auth config
az webapp auth show --name APP --resource-group RG
S-REQ-02
Secrets in Key Vault
No secrets in code
git log -p | grep -i "password|secret|key"
S-REQ-03
HTTPS Only
Check HTTPS redirect
az webapp show --name APP --resource-group RG --query httpsOnly
S-REQ-04
Input Validation
Code review
Manual review of form handlers
S-REQ-05
Dependency Scanning
Run audit
npm audit or pip-audit
S-REQ-06
Least Privilege
Check RBAC roles
az role assignment list --assignee PRINCIPAL_ID
S-REQ-07
SCM/FTP basic auth disabled
Check publishing policies
az rest --method get --uri ".../basicPublishingCredentialsPolicies/scm?api-version=2022-03-01"
S-REQ-08
Public network access disabled (internal apps)
Check publicNetworkAccess
az webapp show --name APP --resource-group RG --query publicNetworkAccess
Full Checklist
## Security- [ ] No secrets in source code (use Key Vault references)- [ ] Authentication implemented (Entra ID)- [ ] HTTPS enforced (`httpsOnly: true`)- [ ] SCM/FTP basic publishing credentials disabled (`scm=false`, `ftp=false`)- [ ] Input validation on all user inputs- [ ] Dependency vulnerability scan passed (`npm audit` / `pip-audit`)- [ ] Error messages don't expose internal details- [ ] Logging doesn't capture sensitive data (passwords, tokens)## Architecture- [ ] Code in approved repository (GitHub/Azure DevOps)- [ ] README with setup and deploy instructions- [ ] Entry in App Registry- [ ] All external connections documented## Azure Resources- [ ] App Service/Static Web App created- [ ] Application settings configured- [ ] Entra ID redirect URIs updated- [ ] Custom domain configured (if needed)- [ ] SSL certificate in place- [ ] Deployment pipeline working- [ ] Application Insights enabled## Functional- [ ] Authentication flow tested on production URL- [ ] Database/storage connectivity verified- [ ] All environment variables set correctly- [ ] Performance acceptable under expected load- [ ] Error handling tested- [ ] Rollback procedure documented
Verification Commands
# Check HTTPS enforcementaz webapp show \ --name ps-yourapp-dashboard \ --resource-group PS-WEBAPPS \ --query "httpsOnly"# Check authentication statusaz webapp auth show \ --name ps-yourapp-dashboard \ --resource-group PS-WEBAPPS# List app settings (verify no plain-text secrets)az webapp config appsettings list \ --name ps-yourapp-dashboard \ --resource-group PS-WEBAPPS \ --output table# Run dependency audit (npm)npm audit --audit-level=high# Run dependency audit (Python)pip-audit --strict
12. Troubleshooting
Common Issues
Issue
Cause
Solution
502 Bad Gateway
Startup command failing
Check startup logs: az webapp log tail
401 Unauthorized
Endpoint is auth-protected by design
Verify expected auth model before treating as outage
Key Vault access denied
Missing RBAC
Grant “Key Vault Secrets User” role
Deployment fails
Runner identity or private DNS issue
Verify runner MI permissions and privatelink DNS resolution
# View auth configaz webapp auth show \ --name ps-yourapp-dashboard \ --resource-group PS-WEBAPPS# Common auth issues:# 1. Redirect URI mismatch - must match exactly including trailing slash# 2. Client secret expired - regenerate in Azure Portal# 3. Wrong tenant ID - verify in app settings# 4. Token audience mismatch - check allowed-audiences setting
PSI-Specific Issues
“PrivateEndpointCreationNotAllowedAsSubnetIsDelegated”
The PS-WebApps subnet is delegated for VNet integration. Use PS-SERVERS or PS-ProdData for private endpoints.
“Bad Request” when adding custom domain
Add the verification TXT record first: asuid.{subdomain} with the App Service’s customDomainVerificationId.
“Conflict” when importing certificate from Key Vault
A certificate with the same thumbprint already exists in the resource group. This happens if the cert was previously uploaded directly. The existing binding will work; Key Vault import is for new deployments.
Path conversion issues in Git Bash
Prefix commands with MSYS_NO_PATHCONV=1 when using resource IDs starting with /subscriptions/.
App works locally but not in production
Check that .env.production has the correct API URL
Verify the API URL is accessible from the user’s network (VPN required)
Check browser console for CORS errors
Verify DNS resolves correctly: nslookup yourapp.progressivesurface.com 10.160.0.5
“Failed to fetch” / CORS errors in browser
The web app can reach the API, but the browser blocks the response due to missing CORS headers. Solution: Add your app’s origin to the API’s Cors:AllowedOrigins in appsettings.json. See 8. API CORS Configuration above.
Deploy 403 when publicNetworkAccess=Disabledaz webapp deploy hits the SCM endpoint (*.scm.azurewebsites.net) which is blocked when DNS resolves to the public IP. Root cause: Windows DNS returns the complete CNAME chain from public DNS, bypassing privatelink resolution. Solution: See DNS Configuration for Private Endpoints in Section 10. AD-integrated primary zones on PS-AZ-DC01 provide authoritative privatelink A records for all VNet machines. If adding a new app, remember to add both app and app.scm A records to the DC zone.
App Service cannot reach on-premises servers by short hostname
Azure Linux App Services do not have the ad.ptihome.com DNS search domain configured. Short hostnames like ps-proxy will fail to resolve — even though the VNet integration, route table, and VPN tunnel are all working correctly. The symptom is fetch failed or ENOTFOUND errors when the app tries to connect to on-prem services.
Root cause: On a domain-joined Windows machine (or the self-hosted runner with systemd-resolved configured), DNS automatically appends ad.ptihome.com to unqualified hostnames. App Service containers don’t have this search domain, so bare hostnames like ps-proxy go to public DNS and fail.
Solution: Always use the fully-qualified domain name (FQDN) in App Service environment variables:
Wrong
Correct
http://ps-proxy:3100/mcp
http://ps-proxy.ad.ptihome.com:3100/mcp
http://aftec-server:5000
http://aftec-server.ad.ptihome.com:5000
This applies to any App Service configuration that references on-premises servers: environment variables, connection strings, API endpoints, etc. Local development and the self-hosted runner can use short hostnames (they have the search domain), but Azure App Service always requires the FQDN.
Discovered: PSI Explorer “Ask the Fleet” chat was connecting to ps-proxy:3100 locally but failing on Azure App Service. Changing MCP_SERVER_URL to http://ps-proxy.ad.ptihome.com:3100/mcp in the App Service Configuration resolved the issue immediately.