Designing and Evaluating a Physical AI or AIoT Pilot
A practical framework for testing a bounded utility use case—with a defined workflow, baseline, data requirements, success measures, and decision criteria.
Explore the pilot framework ↘Prove a real workflow under representative conditions
A useful pilot answers a defined operational question under representative conditions. For utilities, that may mean determining whether connected sensing can improve visibility into transformer condition, whether AI-supported analysis can help prioritize pump maintenance, or whether a field inspection workflow can capture more complete and timely asset records. Establish the workflow, baseline, data requirements, success measures, and decision criteria before configuring the system.
A pilot should test a bounded use case, not attempt to prove that an entire utility operation can be automated. It should show whether the proposed approach works with the equipment, operating conditions, information systems, and people involved. It should also identify what remains unverified before a wider deployment is considered.
What should a utility pilot prove?
A Physical AI or Artificial Intelligence of Things (AIoT) pilot should answer four connected questions.
Operational value
Does the approach improve a defined task, such as identifying an abnormal equipment condition, completing an inspection record, locating an asset, or prioritizing a maintenance response?
Technical fit
Can the required sensors, identification devices, edge equipment, software, and interfaces collect and exchange usable information under site conditions?
Operational acceptance
Can operators, field technicians, reliability engineers, and control-room personnel understand the output and incorporate it into an approved workflow?
Risk and readiness
Are data quality, cybersecurity, access permissions, safety boundaries, outage behavior, and human oversight understood well enough to decide what should happen next?
Example objective: “Determine whether condition data from selected distribution transformers can help the maintenance team identify inspection priorities earlier than its current review process.” This is more useful than a broad objective such as “introduce AI to the grid.”
Choose a utility workflow and set the pilot boundary
Start with one asset class, one operational process, and a defined group of users.
Select an appropriate asset and task
A suitable boundary may include selected distribution transformers, substation circuit breakers, feeder equipment, water or wastewater pumps, treatment-process instruments, gas distribution pressure-monitoring points, or a field inspection route. Select equipment and tasks that belong to the organization’s actual operating responsibilities.
Document what is included—and excluded
Identify the assets, locations, shifts, operating modes, users, and decisions included. Record exclusions such as other substations, emergency switching, safety instrumented functions, or autonomous control. Confirm who owns each asset and who can authorize workflow changes.
Define the capability being evaluated
An AIoT pilot may focus on identity, sensing, data capture, event processing, dashboards, or decision support.
Distinguish insight from physical action
A Physical AI pilot adds a defined path from an approved decision to a physical action, together with authorization, protective boundaries, and independent checks of the resulting state. A recommendation or work-order request is not the same as direct equipment control.
Utility Substation AIoT Pilot Design
Select an asset boundary, identify relevant measurement locations, and plan how field observations will reach the monitoring system. Connect every sensor or device to a specific operational question and asset record.

Illustrative AI-generated visual. It does not depict an actual UtilityGrid installation or customer site.
Establish the baseline before introducing the candidate system
A baseline describes how the selected task is performed today. It gives the pilot a fair point of comparison and helps distinguish measurable improvement from changes in workload, operating conditions, or reporting practice.
Transformer condition monitoring
The baseline may include existing inspection intervals, available temperature or load records, alarm history, maintenance findings, and the time required to review an abnormal condition.
Pump monitoring
The baseline may include current vibration or pressure checks, operator rounds, work orders, and the time between a reported issue and a maintenance assessment.
Field inspection
The baseline may include completed inspection records, missing fields, repeat visits, and time spent reconciling asset information.
Record the limitations
Record the baseline source, collection period, exclusions, and known gaps. Use a period and sample reflecting normal operating variation, planned maintenance, relevant seasonal conditions, and the shifts involved. If records are incomplete, decide whether a prospective baseline is needed.
Do not compare a fully instrumented pilot period with an unusually quiet historical period without explaining the difference.
Define measures that match the operational question
Select a small set of primary measures and supporting diagnostic measures. Each metric needs a written definition, unit, denominator, time window, data source, and rule for handling missing or repeated records.
Event completeness
Correctly captured required events divided by the required events established from an agreed reference record. Useful for inspection events, asset transfers, or condition observations.
Alert usefulness
Share of reviewed alerts that lead to a defined, relevant follow-up, with false alerts and missed conditions reported separately.
Task time
Elapsed time or staff effort required to complete a defined inspection, review, reconciliation, or maintenance-prioritization task.
Detection delay
Elapsed time between an agreed reference event and its detection or notification, where reliable event timestamps are available.
Verified action outcomes
Eligible actions confirmed by independent evidence of the resulting physical state divided by attempted eligible actions. A command acknowledgment alone does not prove the equipment reached the requested state.
Set acceptance criteria before testing begins. Criteria may include minimum record completeness, a maximum tolerable false-alert rate, an agreed reduction in manual reconciliation effort, or successful handling of specified failure cases. Values must be selected with operational owners and technical reviewers; there is no universal threshold for every asset class or site.
Prepare data, equipment, and system interfaces
List every data source needed to evaluate the pilot and map the systems that own or consume that information.
Identify required data
Depending on the use case, include sensor readings, equipment identifiers, asset registers, inspection records, maintenance history, alarms, operating modes, work orders, and operator confirmations. Associate each observation with the correct asset, timestamp, unit, source, and quality status. Define handling for missing, duplicated, delayed, or conflicting records.
Map systems and ownership
Utility environments may include Supervisory Control and Data Acquisition (SCADA), operational historians, Enterprise Asset Management (EAM) or Computerized Maintenance Management Systems (CMMS), Geographic Information Systems (GIS), outage-management systems, and other operational applications.
Verify interfaces and permissions
Confirm actual interfaces, data owners, access permissions, update frequency, and acceptance process for each connection. Do not imply an existing connector or integration is available until it has been verified.
Check physical installation
Confirm sensor suitability, measurement range, sampling interval, calibration approach, mounting location, power, environmental rating, communications coverage, and maintenance access. For edge devices and gateways, document local processing responsibilities and behavior during connectivity interruptions.
For enterprise and equipment interfaces, refer to the System Integration and Deployment Guide ↗.
Design test cases for normal operation and failure conditions
A pilot tested only under favorable conditions provides an incomplete view of operational readiness. Include normal operation, routine exceptions, degraded data, and recovery.
Normal operation
Relevant load, flow, pressure, temperature, or environmental range.
Data exceptions
Missing, duplicated, delayed, stale, or out-of-sequence sensor and event records.
Device faults
Sensor drift, implausible readings, device faults, loss of power, or loss of communications.
Identity and records
Asset identity mismatches, incorrect equipment associations, and incomplete maintenance records.
Operating changes
Changes in operating mode, startup and shutdown, planned maintenance, and abnormal but credible site conditions.
Action-enabled cases
Denied authorization, rejected commands, delayed acknowledgments, partial execution, communications loss, manual override, and independent outcome verification.
Define how the team will introduce or observe each condition without creating an unsafe operating state. Use historical records, controlled bench tests, or simulation where appropriate. Any test involving live equipment or control interfaces requires engineering review, approved procedures, and site authorization. Simulated protection behavior does not establish that a real installation is safe or certified.
Assign responsibilities and operating oversight
Name an accountable owner for the pilot and assign responsibilities before installation.
Asset or operations owner
Approves the workflow boundary and operating conditions.
Field personnel
Support site access, installation, inspection, and issue reporting.
Data or IT team
Manages approved access, data transfer, and system interfaces.
Technical team
Configures devices and analytics, records versions, and investigates defects.
Designated reviewer
Checks results and limitations.
Authorized control and safety personnel
Retain authority over any operational action.
Agree on how alerts and exceptions will be reviewed, who can acknowledge them, how urgent conditions are escalated, and where decisions are recorded. Provide a clear route for operators to question an output or return to the existing process. The pilot must not weaken established operating procedures or transfer authority to an unapproved model or application.
Evaluate performance against pre-agreed criteria
At the end of the test period, compare the candidate approach with the baseline using the definitions and acceptance criteria approved at the outset. Report the number of assets, observations, events, operating hours, and test cases included.
Separate measured pilot results from assumptions, estimates, and observations that could not be verified. Review false alerts, missed events, data gaps, operator interventions, rejected commands, recovery behavior, and additional maintenance effort. Explain whether performance changed by asset, operating mode, shift, weather condition, or connectivity state. If a relevant failure condition was not encountered, state that it was not tested rather than implying it passed.
Assess operational impact separately from model or sensor performance. Better detection does not automatically mean avoided downtime, lower costs, or improved reliability. Those outcomes depend on whether a practical intervention was available, whether staff and parts were available, whether the intervention was effective, and whether the result was confirmed. Any financial estimate should state its assumptions and distinguish staff time released from cash savings.
Make a documented expand, revise, or stop decision
A pilot should end with a decision record, not simply a demonstration. Summarize the original question, scope, baseline, test conditions, metric definitions, acceptance criteria, results, exceptions, and unresolved limitations.
Expand
When agreed criteria are met, the operational owner accepts the workflow, required interfaces and controls have been validated, and remaining deployment risks have owners and plans.
Revise
When the use case remains worthwhile but data quality, sensor placement, integration, alert thresholds, user workflow, or test coverage needs improvement. Define additional evidence and repeat relevant tests.
Stop
When the operational need is not strong enough, the candidate approach does not meet essential criteria, risks cannot be controlled, or the evidence does not justify further investment.
Simulation is not a customer pilot. A successful simulation is a reason to test an approach, not a substitute for a customer pilot. Synthetic data can help explore assumptions, test data pipelines, or compare methods under controlled scenarios. It does not establish field performance, equipment compatibility, operator adoption, safety, or customer deployment. Label simulation-based results clearly and validate relevant assumptions with site-specific information before making deployment claims.
Simulation-to-pilot readiness checklist
Use this checklist before moving from a modeled study or technical demonstration into a live utility pilot.
Frequently asked questions
How long should a utility AIoT pilot run?
The duration should reflect the asset, event frequency, operating modes, seasonal variation, and time needed to observe the workflow. A short test may validate connectivity or data capture, while a condition-monitoring evaluation may need a longer observation period to include representative operating variation. Set the period based on the evidence required, not a universal number of weeks.
Should a utility begin with AIoT or Physical AI?
An AIoT pilot is often a practical starting point when the main need is identification, sensing, monitoring, or decision support. A Physical AI pilot is relevant when the scope includes an authorized physical response. It requires additional engineering review of command interfaces, authority, protective functions, fallback behavior, and independent confirmation of outcomes.
Can simulation results be used as pilot evidence?
Simulation results can help define a testable hypothesis and identify important operating conditions. They are not customer-pilot results and do not replace validation with real site data, equipment, users, and approved procedures.
What happens if the pilot misses its acceptance criteria?
Record the shortfall and its likely causes, including data quality, sensor coverage, operating variation, model behavior, integration, and workflow adoption. Decide whether a targeted revision can address the gap. If an essential criterion remains unmet or a material risk cannot be controlled, stop rather than expand on incomplete evidence.
Plan a pilot around your utility workflow
The right pilot begins with a specific asset, a real operational question, and agreement on what evidence will support the next decision. UtilityGrid can help frame evaluation requirements for connected sensing, AI-supported analysis, and authorized physical workflows. Discuss the asset class, current process, available data, and operating constraints with the team before selecting a pilot configuration.
