Designing and Evaluating a Utility AIoT Pilot | UtilityGrid
UtilityGrid · Knowledge Center

Designing and Evaluating a Physical AI or AIoT Pilot

A practical framework for testing a bounded utility use case—with a defined workflow, baseline, data requirements, success measures, and decision criteria.

Explore the pilot framework ↘
◎
Start with an operational question. End with an evidence-based decision.Scope · Measure · Test · Decide
A useful pilot is bounded

Prove a real workflow under representative conditions

A useful pilot answers a defined operational question under representative conditions. For utilities, that may mean determining whether connected sensing can improve visibility into transformer condition, whether AI-supported analysis can help prioritize pump maintenance, or whether a field inspection workflow can capture more complete and timely asset records. Establish the workflow, baseline, data requirements, success measures, and decision criteria before configuring the system.

A pilot should test a bounded use case, not attempt to prove that an entire utility operation can be automated. It should show whether the proposed approach works with the equipment, operating conditions, information systems, and people involved. It should also identify what remains unverified before a wider deployment is considered.

Pilot foundations

What should a utility pilot prove?

A Physical AI or Artificial Intelligence of Things (AIoT) pilot should answer four connected questions.

01

Operational value

Does the approach improve a defined task, such as identifying an abnormal equipment condition, completing an inspection record, locating an asset, or prioritizing a maintenance response?

02

Technical fit

Can the required sensors, identification devices, edge equipment, software, and interfaces collect and exchange usable information under site conditions?

03

Operational acceptance

Can operators, field technicians, reliability engineers, and control-room personnel understand the output and incorporate it into an approved workflow?

04

Risk and readiness

Are data quality, cybersecurity, access permissions, safety boundaries, outage behavior, and human oversight understood well enough to decide what should happen next?

Example objective: “Determine whether condition data from selected distribution transformers can help the maintenance team identify inspection priorities earlier than its current review process.” This is more useful than a broad objective such as “introduce AI to the grid.”

01 · Define the scope

Choose a utility workflow and set the pilot boundary

Start with one asset class, one operational process, and a defined group of users.

01

Select an appropriate asset and task

A suitable boundary may include selected distribution transformers, substation circuit breakers, feeder equipment, water or wastewater pumps, treatment-process instruments, gas distribution pressure-monitoring points, or a field inspection route. Select equipment and tasks that belong to the organization’s actual operating responsibilities.

02

Document what is included—and excluded

Identify the assets, locations, shifts, operating modes, users, and decisions included. Record exclusions such as other substations, emergency switching, safety instrumented functions, or autonomous control. Confirm who owns each asset and who can authorize workflow changes.

03

Define the capability being evaluated

An AIoT pilot may focus on identity, sensing, data capture, event processing, dashboards, or decision support.

04

Distinguish insight from physical action

A Physical AI pilot adds a defined path from an approved decision to a physical action, together with authorization, protective boundaries, and independent checks of the resulting state. A recommendation or work-order request is not the same as direct equipment control.

Pilot design visual

Utility Substation AIoT Pilot Design

Select an asset boundary, identify relevant measurement locations, and plan how field observations will reach the monitoring system. Connect every sensor or device to a specific operational question and asset record.

Utility substation transformer with connected sensor points and an outlined AIoT pilot monitoring area.

Illustrative AI-generated visual. It does not depict an actual UtilityGrid installation or customer site.

02 · Establish a baseline

Establish the baseline before introducing the candidate system

A baseline describes how the selected task is performed today. It gives the pilot a fair point of comparison and helps distinguish measurable improvement from changes in workload, operating conditions, or reporting practice.

Transformer condition monitoring

The baseline may include existing inspection intervals, available temperature or load records, alarm history, maintenance findings, and the time required to review an abnormal condition.

Pump monitoring

The baseline may include current vibration or pressure checks, operator rounds, work orders, and the time between a reported issue and a maintenance assessment.

Field inspection

The baseline may include completed inspection records, missing fields, repeat visits, and time spent reconciling asset information.

Record the limitations

Record the baseline source, collection period, exclusions, and known gaps. Use a period and sample reflecting normal operating variation, planned maintenance, relevant seasonal conditions, and the shifts involved. If records are incomplete, decide whether a prospective baseline is needed.

Do not compare a fully instrumented pilot period with an unusually quiet historical period without explaining the difference.

03 · Define success

Define measures that match the operational question

Select a small set of primary measures and supporting diagnostic measures. Each metric needs a written definition, unit, denominator, time window, data source, and rule for handling missing or repeated records.

Event completeness

Correctly captured required events divided by the required events established from an agreed reference record. Useful for inspection events, asset transfers, or condition observations.

Alert usefulness

Share of reviewed alerts that lead to a defined, relevant follow-up, with false alerts and missed conditions reported separately.

Task time

Elapsed time or staff effort required to complete a defined inspection, review, reconciliation, or maintenance-prioritization task.

Detection delay

Elapsed time between an agreed reference event and its detection or notification, where reliable event timestamps are available.

Verified action outcomes

Eligible actions confirmed by independent evidence of the resulting physical state divided by attempted eligible actions. A command acknowledgment alone does not prove the equipment reached the requested state.

Set acceptance criteria before testing begins. Criteria may include minimum record completeness, a maximum tolerable false-alert rate, an agreed reduction in manual reconciliation effort, or successful handling of specified failure cases. Values must be selected with operational owners and technical reviewers; there is no universal threshold for every asset class or site.

04 · Prepare the environment

Prepare data, equipment, and system interfaces

List every data source needed to evaluate the pilot and map the systems that own or consume that information.

Identify required data

Depending on the use case, include sensor readings, equipment identifiers, asset registers, inspection records, maintenance history, alarms, operating modes, work orders, and operator confirmations. Associate each observation with the correct asset, timestamp, unit, source, and quality status. Define handling for missing, duplicated, delayed, or conflicting records.

Map systems and ownership

Utility environments may include Supervisory Control and Data Acquisition (SCADA), operational historians, Enterprise Asset Management (EAM) or Computerized Maintenance Management Systems (CMMS), Geographic Information Systems (GIS), outage-management systems, and other operational applications.

Verify interfaces and permissions

Confirm actual interfaces, data owners, access permissions, update frequency, and acceptance process for each connection. Do not imply an existing connector or integration is available until it has been verified.

Check physical installation

Confirm sensor suitability, measurement range, sampling interval, calibration approach, mounting location, power, environmental rating, communications coverage, and maintenance access. For edge devices and gateways, document local processing responsibilities and behavior during connectivity interruptions.

For enterprise and equipment interfaces, refer to the System Integration and Deployment Guide ↗.

05 · Test beyond ideal conditions

Design test cases for normal operation and failure conditions

A pilot tested only under favorable conditions provides an incomplete view of operational readiness. Include normal operation, routine exceptions, degraded data, and recovery.

01

Normal operation

Relevant load, flow, pressure, temperature, or environmental range.

02

Data exceptions

Missing, duplicated, delayed, stale, or out-of-sequence sensor and event records.

03

Device faults

Sensor drift, implausible readings, device faults, loss of power, or loss of communications.

04

Identity and records

Asset identity mismatches, incorrect equipment associations, and incomplete maintenance records.

05

Operating changes

Changes in operating mode, startup and shutdown, planned maintenance, and abnormal but credible site conditions.

06

Action-enabled cases

Denied authorization, rejected commands, delayed acknowledgments, partial execution, communications loss, manual override, and independent outcome verification.

Define how the team will introduce or observe each condition without creating an unsafe operating state. Use historical records, controlled bench tests, or simulation where appropriate. Any test involving live equipment or control interfaces requires engineering review, approved procedures, and site authorization. Simulated protection behavior does not establish that a real installation is safe or certified.

06 · People and governance

Assign responsibilities and operating oversight

Name an accountable owner for the pilot and assign responsibilities before installation.

Asset or operations owner

Approves the workflow boundary and operating conditions.

Field personnel

Support site access, installation, inspection, and issue reporting.

Data or IT team

Manages approved access, data transfer, and system interfaces.

Technical team

Configures devices and analytics, records versions, and investigates defects.

Designated reviewer

Checks results and limitations.

Authorized control and safety personnel

Retain authority over any operational action.

Agree on how alerts and exceptions will be reviewed, who can acknowledge them, how urgent conditions are escalated, and where decisions are recorded. Provide a clear route for operators to question an output or return to the existing process. The pilot must not weaken established operating procedures or transfer authority to an unapproved model or application.

07 · Evaluate the evidence

Evaluate performance against pre-agreed criteria

At the end of the test period, compare the candidate approach with the baseline using the definitions and acceptance criteria approved at the outset. Report the number of assets, observations, events, operating hours, and test cases included.

Separate measured pilot results from assumptions, estimates, and observations that could not be verified. Review false alerts, missed events, data gaps, operator interventions, rejected commands, recovery behavior, and additional maintenance effort. Explain whether performance changed by asset, operating mode, shift, weather condition, or connectivity state. If a relevant failure condition was not encountered, state that it was not tested rather than implying it passed.

Assess operational impact separately from model or sensor performance. Better detection does not automatically mean avoided downtime, lower costs, or improved reliability. Those outcomes depend on whether a practical intervention was available, whether staff and parts were available, whether the intervention was effective, and whether the result was confirmed. Any financial estimate should state its assumptions and distinguish staff time released from cash savings.

08 · Make the decision

Make a documented expand, revise, or stop decision

A pilot should end with a decision record, not simply a demonstration. Summarize the original question, scope, baseline, test conditions, metric definitions, acceptance criteria, results, exceptions, and unresolved limitations.

Path 01

Expand

When agreed criteria are met, the operational owner accepts the workflow, required interfaces and controls have been validated, and remaining deployment risks have owners and plans.

Path 02

Revise

When the use case remains worthwhile but data quality, sensor placement, integration, alert thresholds, user workflow, or test coverage needs improvement. Define additional evidence and repeat relevant tests.

Path 03

Stop

When the operational need is not strong enough, the candidate approach does not meet essential criteria, risks cannot be controlled, or the evidence does not justify further investment.

Simulation is not a customer pilot. A successful simulation is a reason to test an approach, not a substitute for a customer pilot. Synthetic data can help explore assumptions, test data pipelines, or compare methods under controlled scenarios. It does not establish field performance, equipment compatibility, operator adoption, safety, or customer deployment. Label simulation-based results clearly and validate relevant assumptions with site-specific information before making deployment claims.

Ready for the field?

Simulation-to-pilot readiness checklist

Use this checklist before moving from a modeled study or technical demonstration into a live utility pilot.

Operational question, target asset class, process boundary, and excluded activities are documented.
Assumptions to verify at the site are listed with sources, units, confidence, and responsible reviewers.
Real measurements, asset records, operating modes, and maintenance history needed for evaluation are identified.
Baseline period, baseline workflow, sample coverage, and known data gaps are recorded.
Metrics, denominators, time windows, acceptance criteria, and failure cases are approved before testing.
Normal, degraded, outage, and recovery conditions are included in the test plan.
Installation, data access, integration, cybersecurity, safety review, and operating responsibilities are assigned.
Human oversight, escalation, override, and stop conditions are defined for every action-enabled workflow.
Results will be independently reviewed, limitations documented, and an expand, revise, or stop decision recorded.
Common questions

Frequently asked questions

How long should a utility AIoT pilot run?

The duration should reflect the asset, event frequency, operating modes, seasonal variation, and time needed to observe the workflow. A short test may validate connectivity or data capture, while a condition-monitoring evaluation may need a longer observation period to include representative operating variation. Set the period based on the evidence required, not a universal number of weeks.

Should a utility begin with AIoT or Physical AI?

An AIoT pilot is often a practical starting point when the main need is identification, sensing, monitoring, or decision support. A Physical AI pilot is relevant when the scope includes an authorized physical response. It requires additional engineering review of command interfaces, authority, protective functions, fallback behavior, and independent confirmation of outcomes.

Can simulation results be used as pilot evidence?

Simulation results can help define a testable hypothesis and identify important operating conditions. They are not customer-pilot results and do not replace validation with real site data, equipment, users, and approved procedures.

What happens if the pilot misses its acceptance criteria?

Record the shortfall and its likely causes, including data quality, sensor coverage, operating variation, model behavior, integration, and workflow adoption. Decide whether a targeted revision can address the gap. If an essential criterion remains unmet or a material risk cannot be controlled, stop rather than expand on incomplete evidence.

Plan your next step

Plan a pilot around your utility workflow

The right pilot begins with a specific asset, a real operational question, and agreement on what evidence will support the next decision. UtilityGrid can help frame evaluation requirements for connected sensing, AI-supported analysis, and authorized physical workflows. Discuss the asset class, current process, available data, and operating constraints with the team before selecting a pilot configuration.

Review the pilot framework ↗