This is the on-call companion to the DataSync overview. If an ARGO stops syncing, if alerts fire, or if the dashboard shows red, start here.
For architecture, data model, and “how does it work” questions, read the overview first. This page is specifically about what to do when something is wrong.
Sync Fidelity Audit — the “is everything actually working?” check
The Sync Fidelity Audit is the named, repeatable end-to-end health check. Run it after any deploy, any install, or any incident to get a definitive yes/no on whether the system is healthy top-to-bottom.
# One-liner from any workstation with curl + jq + the CI secretDATASYNC_CI_UPLOAD_SECRET=xxx scripts/fidelity-audit.sh# Require all agents to be at or above a specific version (post-rollout gate)DATASYNC_CI_UPLOAD_SECRET=xxx scripts/fidelity-audit.sh --min-version 1.0.93.0# Dashboard equivalent: GET /api/admin/fidelity-audit (Bearer or X-CI-Secret auth)
What it checks
Check
Validates
AGENT_ALIVE
Each active runner heartbeated within the last 10 min.
AGENT_VERSION
(When --min-version is set) Every runner is at or above the required version.
JOBS_SCHEDULED
Every enabled job has a sync run within 2× its schedule window.
JOBS_SUCCEEDED
Latest run per job is success, partial, or running — NOT failed.
NO_STALE_ORPHANS
Zero sync runs stuck at running beyond the 30-min orphan threshold.
CHECKSUMS_POPULATED
Recent sync_run_files.checksum rows are populated — proves the P0-1 integrity verification is live on current agents.
STATUS_HEARTBEAT_RICH
At least one heartbeat in the lookback window carried active_jobs — proves P2-1 real-status reporting is live.
NO_ACTIVE_ALERTS
Zero unresolved critical alerts on the dashboard.
BACKUP_AT_RISK
No geometry job has a failed latest sync with an unrecovered backlog. A fail here means source files may be purged by the ARGO before they reach Azure — see .
Exit codes
0 — all checks pass or warn (safe to proceed with deploys)
1 — one or more checks failed (block the deploy, read the report)
2 — audit itself failed (network, auth, server error — this is what the CI deploy gate fails the deploy on; a 1 only warns, because failed checks are usually fleet-side state, not a broken deploy)
When to run
Immediately after any installer rollout (with --min-version set to the new version).
As a post-deploy gate in CI after server changes.
Daily from a scheduled task to catch drift.
During any incident as the definitive “is it fixed?” check.
Quick health check (first 60 seconds)
# 1. Is the server up?curl -s https://datasync.progressivesurface.com/api/health# Expect: {"status":"healthy","database":"connected", ...}# 2. Are all six ARGOs heartbeating?curl -s -H "X-CI-Secret: $DATASYNC_CI_UPLOAD_SECRET" \ https://datasync.progressivesurface.com/api/diag/fleet | jq '.runners[].last_heartbeat'# Expect: ISO timestamps within the last 5 minutes# 3. Any unresolved alerts?# Dashboard → Alerts tab, or via API:# https://datasync.progressivesurface.com → login → Alerts
If (1) is not 200 or database is not connected, go to Server down below.
If an ARGO’s last heartbeat is older than 10 minutes, go to Runner offline.
If there are active critical alerts, the alert detail tells you which ARGO + job.
Incident playbook
no_new_data (source moved or stopped — data-loss risk)
A geometry job has synced zero new files for 6× its schedule while another
job on the same runner is producing normally. The runner and line are alive —
the job’s source is the anomaly. The most likely cause (this is exactly the
2026-08-31 ARGO5 incident this alert was built from): someone changed or
swapped the Intego export path on the source host, so new measurements are
landing where no job is watching — not in Azure and not on the SSD backup,
single-copy on a machine that auto-purges.
RDP to the runner (or ask an operator) and check what the job’s configured
source path actually contains: dir \\<source-host>\d\<folder> — compare
the newest date folder there against today.
Look for a sibling folder the data moved to (dir \\<source-host>\d\).
Fix the job’s Source Paths in the dashboard (Edit Job Settings). The agent
picks up config within ~5 minutes and the catch-up sync recovers the
backlog automatically (2026-08-31: 20,859 files recovered in one cycle).
If the old path may have received stragglers before the fix, run the gap
audit scoped to the affected dates.
The alert resolves itself on the first run that syncs real files.
backup_at_risk (data-loss risk)
This is the highest-priority alert. The ARGO Intego measurement PCs cap
their local storage at a fixed file count and auto-purge the oldest files.
DataSync exists to archive that data to Azure before the ARGO purges it.
A backup_at_risk alert means a geometry job’s most recent sync failed,
left a real backlog of un-synced files, and has not recovered for over a
full schedule cycle — so those files are racing the ARGO’s purge clock.
The alert fires critical (Teams + email) and also shows as a BACKUP_AT_RISK
fail in the Fidelity Audit.
Respond immediately:
Identify the job — the alert message names the ARGO + job and the
pending file count.
Find out why the sync is failing — check that job’s recent runs on
the dashboard, and the agent logs (GET /api/admin/logs?runnerId=<id>&level=Warning,Error).
Common causes: server 5xx (check server health first), the Intego
source share unreachable, or Azure Files auth.
Once the underlying fault is cleared, the agent re-syncs on its next
cycle; the alert auto-resolves when a successful run lands. To force it,
trigger the job from the dashboard (job → Sync now).
Confirm nothing was lost — run the direct source-vs-Azure audit
(scripts/audit-argo-pc.ps1 on the ARGO PC). If the ARGO already purged
un-synced files, they are only recoverable if they still exist anywhere
on the source.
It is idle-safe: a geometry job with no production (e.g. a weekend) produces
no runs at all and is never flagged — only a failed sync with a backlog
trips it.
Server down (health endpoint not 200)
Check App Service is running:
MSYS_NO_PATHCONV=1 az webapp show --name psi-datasync --resource-group PS-WEBAPPS \ --query "{state:state, enabled:enabled}" -o table
Tail live logs for a fatal error on startup:
MSYS_NO_PATHCONV=1 az webapp log tail --name psi-datasync --resource-group PS-WEBAPPS
If the log shows a database connection failure, check the Azure SQL server and the managed-identity auth:
MSYS_NO_PATHCONV=1 az sql db show --name DataSync --server procserv-proddata \ --resource-group ProcServices-Prod-Data --query "status" -o tsv# Expect: Online
If the log shows “database has reached its size quota” → jump to Database full below.
Last resort: restart the App Service. Safe, ~30s downtime:
MSYS_NO_PATHCONV=1 az webapp restart --name psi-datasync --resource-group PS-WEBAPPS
Database full
Known failure mode — the 2 GB Basic-tier cap was hit on 2026-04-23. Tier is now S0 (250 GB cap) with the retention sweeper running daily; this shouldn’t recur organically, but:
Symptoms: Every INSERT returns RequestError: The database 'DataSync' has reached its size quota. Agent log ingestion rolls back. Sync /complete fails so state index drifts.
Immediate mitigation (one-liner to reclaim space fast):
MSYS_NO_PATHCONV=1 az sql db update --name DataSync --server procserv-proddata \ --resource-group ProcServices-Prod-Data --max-size 500GB
Root cause fix — force a retention sweep:
# Connect with sqlcmd using Azure AD auth and delete beyond retention:sqlcmd -S procserv-proddata.database.windows.net -d DataSync -G \ -Q "DELETE FROM agent_logs WHERE timestamp < DATEADD(DAY, -30, GETUTCDATE())"
Then restart the App Service once the quota is cleared:
MSYS_NO_PATHCONV=1 az webapp restart --name psi-datasync --resource-group PS-WEBAPPS
Runner offline (single ARGO not heartbeating)
Confirm the ARGO PC is up — ping it, TeamViewer in, or check the shop-floor UPS monitoring.
Is the Windows service running on that ARGO?
# On the ARGO PCGet-Service PSIDataSync
If Stopped: Start-Service PSIDataSync, then check Event Viewer → Application for why it stopped.
If Running but no heartbeat: check outbound connectivity:
Search for [OrphanSweeper] Closed. Runs that exceed 30 minutes without a last_activity_at touch are closed.
If you see [OrphanSweeper] Sweep failed: Connection is closed. repeating every 5 minutes, the server has hit a transient Azure SQL disconnect and (pre-2026-05-18) was unable to recover. Post-2026-05-18 the schedulers re-fetch the pool on every tick, so this should self-heal — but if you ever see this pattern persisting, restart the App Service to force a fresh boot:
MSYS_NO_PATHCONV=1 az webapp restart --name psi-datasync --resource-group PS-WEBAPPS
Then re-run the Fidelity Audit to confirm NO_STALE_ORPHANS passes.
From the dashboard, check the sync run’s files — a single file stuck retrying can hold up the run.
If Azure Files is the bottleneck (timeouts on upload), check Azure service health and the share’s request metrics:
MSYS_NO_PATHCONV=1 az storage share show --name argodatastore --account-name psargostorage \ --query "properties.quota" -o tsv
Auth expired (Azure Files credentials)
SAS tokens issued to agents rotate automatically, but the underlying storage account key rotates manually.
Symptoms:Access is denied or network path errors in per-file results on the dashboard. azureFilesStatus heartbeat field shows auth_failure.
Azure SQL agent_logs table, queryable via dashboard → Logs
90 days (retention sweeper)
Installer diagnostics (SSD backup task registration, service account reconfig)
Shipped to the same agent_logs table (category Installer) via Report-InstallLog.ps1 at the end of every install, if the machine is already enrolled — dashboard → Logs, filter by category. Local copy at C:\ProgramData\PSI\DataSync\install-diagnostics.log until shipped.
90 days (retention sweeper)
Sync run history
Azure SQL sync_runs + sync_run_files, queryable via dashboard → Runs
180 days (retention sweeper)
Alert history
Azure SQL alerts table, dashboard → Alerts
No auto-purge (low volume)
Routine operations
Seal / unseal a date folder
A geometry date folder is sealed when no new files have been seen in 24 hours. Sealed folders are skipped by agents to save work. If Intego writes to a previously-quiet date folder later, the files are invisible until unsealed.
Check seal status (dashboard): Runner Detail → Job → “Sealed folders” list.
Force-seal a folder manually:
POST /api/admin/runners/:runnerId/jobs/:jobId/seal/:folderDate
(requires dashboard Bearer token)
Unseal a folder so agents rescan it:
POST /api/admin/runners/:runnerId/jobs/:jobId/unseal/:folderDate
Credential rotation (SAS + storage account key)
The storage account key is the master credential. Rotate it every 90 days minimum, or immediately on any compromise signal.
Portal → psargostorage → Access keys → Regenerate key2 (never both at once).
Update the App Service (see Auth expired above).
Wait for all six ARGOs to heartbeat azureFilesStatus: ok.
After 24 hours (well past any cached SAS lifetime), regenerate key1 if you started with key2.
Reconcile an agent’s state index
If a sync failed silently (files in Azure but not recorded in the state index), force a reconciliation:
POST /api/admin/runners/:runnerId/jobs/:jobId/reconcile
This enumerates the agent’s files on Azure and repopulates synced_files. See the reconciliation safeguards — it generates real Azure Files list transactions, so scope it to a single job and watch the progress.
Promote a new agent release
Auto-update works for service-mode runners (fixed 2026-07-09, GHE #23 — the CI publish job now
pins to ps-gr-runner, the only machine with Inno Setup, so every build is a real installer, not a
zip). Confirmed working end-to-end on ARGO1/ARGO2/ARGO6. Promotion is a manual, deliberate step —
new releases never auto-promote:
CI builds PSIDataSync-Setup-{version}.exe on every push touching src/agent/** or scripts/**
and uploads it as isCurrent: false.
Dashboard → Agent Releases lists every build. Promote a version explicitly once you’ve
confirmed it (this is a human click — nothing auto-promotes).
Connected service-mode agents pick up the new current version on their next UpdateWorker poll
(~10 min), download, verify SHA256, schedule a one-time /VERYSILENT install via a SYSTEM-level
Scheduled Task, and restart.
ARGO3/4/5 are still in tray mode and cannot self-update at all — schtasks /Create /RU SYSTEM
requires a privilege a tray-mode login session doesn’t have, so their UpdateWorker retries forever
and never converges (see GHE #41). Until they’re converted to service mode (same install-as-service
.\intego account flow as ARGO1/2/6), those three need the old manual path:
Download the installer from dashboard → Agent Releases (or W:\scripts\PSIDataSync\).
TeamViewer into the ARGO PC, copy the .exe, run as admin.
The installer stops the service/tray process, swaps binaries, preserves DPAPI state (no
re-enrollment needed), restarts.
Not yet formalized. Informal = whoever owns the app.
Must be formalized before contracting any customer SLA — see P4-5
Azure infra
Shared with PSI IT team
For SQL/AppService/Files incidents needing subscription-level access
Customer (data-critical)
TBD per P4-5
TBD
Must be informed within 1 hour of a data-loss-class incident
Security incidents
Follow PSI security contact chain
Storage-account-key leak, credential compromise, etc.
Escalation ladder
Minor / non-urgent (single-ARGO failure, transient alert): ticket in the normal work queue. No page.
Sev-2 / operational (multiple ARGOs down, DB performance, sync stalled > 4h): email the application owner. Follow up in Teams if no response within 30 min.
Sev-1 / customer-impacting (all ARGOs down, data loss risk, DB full, Azure Files share compromised): phone/Teams the application owner immediately. If no response within 15 min, escalate to IT leadership.
Security incident: follow the PSI security incident chain. Do NOT post details in Teams — credentials may be in scope.
Alert channels
Teams webhook — configured on the App Service as TEAMS_WEBHOOK_URL. Destination channel: TBD, confirm in dashboard alert rule config.
Email — recipient list configured per alert rule in the dashboard. TBD confirm the distribution list.
No PagerDuty / Opsgenie today. If customer SLA requires on-call paging (P4-5), integrate one before signing.
Change-control contacts
Production deploys — application owner can push to main directly (deploy is auto from push); no change-advisory board required.
Customer-visible schema changes — coordinate with customer contact first.
Credential rotation — notify the application owner before rotating the storage account key.
Known failure modes
DB size quota (2026-04-23)
Basic-tier 2 GB cap hit; all INSERTs failed for ~3 hours. Resolved by bumping to Standard S0 (250 GB) and adding the daily retention sweeper. Watchlist: database_size in az sql db list-usages should stay well below quota.
Auto-update — fixed for service mode, structurally impossible in tray mode
Superseded 2026-07-09 (GHE #23): the original fragility was the CI publish job randomly landing
on a runner without Inno Setup, silently uploading a zip that UpdateWorker correctly refuses to
auto-install. Pinning publish-agent to ps-gr-runner fixed that — auto-update now works
end-to-end, confirmed on ARGO1, ARGO2, and ARGO6 (all service mode).
What remains (GHE #41): a tray-mode agent can never self-update, full stop — schtasks /Create /RU SYSTEM needs a privilege the login-session process doesn’t have, so UpdateWorker retries
every ~10 min forever and never converges. This isn’t a bug to fix in the update mechanism itself;
it’s a fleet-rollout task (convert ARGO3/4/5 to service mode). Until then those three are manual
installs for every release — see Promote a new agent release above.
SSD backup Scheduled Task silently fails to register (2026-07-13, GHE 40)
Symptoms: service installs correctly, runs under the entered account, but
PSIDataSync-SSDBackup never appears in Task Scheduler — even when the account/password fields
were confirmed filled in and the account is confirmed working (network shares reachable).
Root cause:RegisterSsdBackupTask (installer.iss) built an unescaped nested-quote /TR
value for schtasks /Create. Windows argv parsing closes the quoted argument early, so /RU,
/RP, and /RL got silently swallowed into the Task To Run string instead of being parsed as real
flags — confirmed via schtasks /Query /TN "PSIDataSync-SSDBackup" /V /FO LIST, which showed Run As User defaulted to whoever ran the command interactively and Logon Mode: Interactive only
(no real password ever reached schtasks). The installer’s own ResultCode still read 0 the whole
time, which is why this went unnoticed for as long as it did.
Current state (fixed in PR #40, pending merge + retest on ARGO1):
Don’t trust the GUI for “is the task really there” — taskschd.msc has a known refresh lag after
an external schtasks.exe change. Use schtasks /Query /TN "PSIDataSync-SSDBackup" /V /FO LIST
or PowerShell’s Get-ScheduledTask -TaskName "PSIDataSync-SSDBackup" instead.
The installer now verifies the task with a post-creation /Query (not just the /Create exit
code) and fires it once immediately via /Run — check the dashboard’s Local SSD Backup table
“Scheduled Task” column and Agent Logs (category Installer) for a real, remote answer instead
of relying on whoever’s physically at the machine.
If it’s still not registering after a fresh install with #40 merged, check whether “Install as a
Windows Service” was actually (re-)selected on that install run — InstallService (and therefore
RegisterSsdBackupTask) only ever runs if WantsService.
Lockfile drift in CI
npm ci sometimes fails during the “Reinstall dev dependencies for cache” step after npm prune --omit=dev. Mitigated by continue-on-error: true so the deploy itself isn’t blocked — but it does mean that if the REAL npm ci fails, we won’t catch it as visibly. Watch the actual build phase, not the trailing cache-warming step.
Local SSD backup Scheduled Task fails to register (2026-07-14)
The installer registers the PSIDataSync-SSDBackup Scheduled Task (daily 02:00, runs as the entered service account).
Confirmed root cause (2026-07-14, ARGO6): the .\ account prefix. The installer’s account field defaults to .\intego. sc config obj= .\intego accepts the .\ local-machine shorthand — so the service runs fine under it — but schtasks /RU resolves the name via LookupAccountName, which does not understand .\. It fails with ERROR: No mapping between account names and security IDs was done and schtasks /Create exits 1, so the task is never created. The installer (1.0.167+) now converts a leading .\ to COMPUTERNAME\user for the schtasks call. Manual fix on an affected PC (prompts for password):
An earlier related failure mode was a task that appeared to register (schtasks /Create exit 0) but was mangled (nested-quote bug swallowed /RU//RP into the action), registering with Logon Mode: Interactive only so it would never fire unattended at 02:00. Both are now guarded.
Verifying it exists — don’t trust the Task Scheduler GUI.taskschd.msc caches aggressively and frequently will NOT show a newly-created task even after F5. The task is created at the root of the Task Scheduler Library (\PSIDataSync-SSDBackup), not in a subfolder. To actually confirm, use the command line (authoritative):
schtasks /Query /TN "PSIDataSync-SSDBackup" /V /FO LIST
Check Logon Mode is notInteractive only (should be Interactive/Background).
If the task runs (Get-ScheduledTask / schtasks see it) but is missing from the Task Scheduler GUI: launch taskschd.mscas administrator. The task’s security descriptor (created by the elevated install / SYSTEM) only grants read to admins, so a non-elevated console silently filters it out while still showing operator-created tasks. Confirmed on PRG2202-PC01, 2026-07-14 — Run as administrator makes it appear under Task Scheduler Library. This is cosmetic: the Task Scheduler service runs the task regardless of what a non-elevated console displays, so no reboot is needed.
How to diagnose now (no hands on the machine needed): after a service-mode (re)install, the installer queries the task back with schtasks /Query /V /FO LIST and reports the real state to the dashboard. Check Agent Logs → category Installer for lines like:
RegisterSsdBackupTask: task detail — Run As User: <acct> | Logon Mode: <mode> | Task To Run: <cmd> — the actual registered task.
RegisterSsdBackupTask: WARNING — ... Logon Mode is "Interactive only" ... — broken: the task won’t run at 02:00. Usually means the service-account password didn’t register correctly. Re-run the service-mode install with the correct account/password.
The installer also fires the task once immediately (schtasks /Run), so a real run report should land in the SSD Backup table (ssd_backup_runs) within moments of a good install. If the “Scheduled Task” column shows Registered but no run report ever appears, suspect the mangled/interactive-only case above.
Note: this relies on the installer diagnostics shipping to the server (Report-InstallLog.ps1), which needs the PC already enrolled (state.dat present). A brand-new PC’s very first install won’t report until it’s enrolled.
Verification checklist after any server or agent change
curl .../api/health returns 200 with fresh uptime
active_jobs column in runners shows non-null during active syncs (P2-1)
checksum column in sync_run_files populated for recent geometry runs (P0-1)
All six ARGOs heartbeating within the last 5 minutes
No new unresolved alerts on the dashboard
BUILD_LOG.md entry in C:\git\PSISync\ describing what changed