Managing Data Science as an Operational Capability
Decision-centered management of analytical work from problem definition through operational use.
Executive Summary
Organizations often turn to data science when available data may reveal a better way to make a decision or perform work, but the actionable relationship is not yet understood. The investigation may expose a data-quality problem that requires stronger validation, show that additional data should be captured at a specific point in a workflow, or reveal a different workflow that was not previously apparent. Any of these findings may improve a mission result, but the investigation begins with evidence rather than an assumed application.
That investigation needs direction. Without a defined outcome, data science can consume substantial staff capacity while producing analyses that no decision owner can use. Exploratory work expands, candidate patterns multiply, and technical outputs become difficult to evaluate against the operation. A named outcome gives the team a basis for deciding which questions deserve attention and when the evidence is strong enough to justify further investment.
The National Institute of Standards and Technology’s AI Risk Management Framework treats governance as a lifecycle function and organizes risk work around Govern, Map, Measure, and Manage (Tabassi, 2023). Sculley et al. (2015) show how production systems accumulate obligations beyond the model itself as they interact with the environment around them. Figure 1 provides a way for managers to revisit whether the original decision, supporting evidence, and authority for future action remain clear as an analytical finding moves toward operational use.
Figure 1. Decision-centered operating loop. A simple management view: define the decision, establish the evidence, operate the capability, then review and change. Four stages run left to right, from named decision and owner, to fit-for-purpose evidence, to operational use and monitoring, to outcome review and change, with a return path to the decision when operating results require change.
Joe Palaia, ACC3’s Lead of Data Science, provides expertise in method selection, validation, and model limitations. Jamie Campbell contributes executive and technology-delivery judgment on operational value and sustainment. Zach Preman contributes implementation experience with synthetic data and monitoring. Their questionnaire responses supplement this paper’s research as practitioner evidence.
Mining offers a limited analogy. Paydirt is earth or ore containing enough valuable material to justify recovery. Data science looks for comparable value in data and tests whether an apparent insight is strong enough to support action. This is where advisory data-science work can add value: defining the outcome, directing the investigation, and establishing whether the evidence warrants a change. When it does, Alchemy SDLC™ can carry that evidence into the requirements and controls for an application or revised workflow.
Begin With the Decision
Data-science requests often arrive in the form of a solution: dashboards, demand forecasts, and the like. Before choosing a method, management needs to translate the request into the decision the request is meant to support. That discussion clarifies who holds decision authority, how the action may change in practice, and what is at stake if it fails. It also establishes when the action can occur and the limits that govern its use. Joe Palaia recommends settling the desired outcome, success measure, and consequences of error before the team evaluates data or algorithms (personal communication, September 3, 2026).
The current process provides the comparison point: useful baselines should reflect how the work performs before the new capability is introduced. Without that reference, a team may report strong model accuracy and still have little evidence that the capability improved the operation itself. A baseline can also make simpler alternatives easier to recognize. Some problems may be addressed through existing analytical methods or straightforward automation without taking on the additional operational burden of machine learning.
Assign Ownership Before Development
Ownership should be clear before development begins. Figure 2 shows how responsibility is divided across the work.
Figure 2. Ownership and authority map. A simple management view: make ownership explicit before development begins. The mission or process owner defines the outcome and the action, the data owner or steward approves access, use, and constraints, the technical owner owns implementation and reliability, and the model or analytics owner maintains validation and change history. All four connect to the operational decision and use, which sits above a release and change authority that approves release, major change, suspension, and retirement. Responsibilities may be combined on smaller efforts, but ownership and approval authority should remain explicit.
Jamie Campbell’s experience is that leadership needs to define the desired outcome before teams collect data or build a solution; otherwise, a polished technical product can emerge without strong operational demand (personal communication, September 2, 2026).
Determine Whether the Data May Support the Use
Data readiness goes beyond completeness and accuracy. History, timeliness, representativeness, permitted use, privacy, security, and policy limits can all affect whether the data is suitable. Aggregation deserves separate review because combined fields may reveal more than they do individually. NIST’s AI RMF likewise treats availability, representativeness, suitability, and intended use as part of mapping AI risk (Tabassi, 2023).
One practical approach is to use synthetic data to test uncommon scenarios while limiting exposure to operational records. Teams should treat it as development and test evidence, not as a substitute for evidence from the target population. Its usefulness depends on how well it represents the conditions under examination. Zach Preman also notes that synthetic-data generators need to be maintained as underlying schemas and business processes change (personal communication, September 2, 2026).
Make the Requirement Testable
When a validated insight leads to an application or workflow change, release criteria should connect the operating condition to an observable result. Brooks (2026) argues for acceptance criteria that retain actors, preconditions, triggers, negative paths, and expected results. Preyna (2026) extends that trace through source material, owner decisions, approved exceptions, versions, and later evidence. These links allow the analytical findings and their limitations to inform the requirements without treating the investigation itself as application development.
Establish Fitness for Purpose
Fitness for purpose depends on the decision. The consequences of error determine how much review is warranted. Figure 3 shows how validation depth can increase as operational consequence rises.
Figure 3. Validation depth by consequence. A simple management view: review rigor should rise as consequence, scale, and irreversibility increase. Lower-consequence use calls for peer review, task-aligned measures, and owner acceptance. Moderate-consequence use adds error analysis, reproducibility, subgroup checks, and documented limits. Higher-impact or regulated use adds independent or separated review, stronger validation evidence, and human intervention or suspension authority.
Evaluation should match the task and the decision it supports. Different analytical methods require different forms of evidence, and some depend more heavily on domain review or human judgment than others. For generative systems, common text-overlap measures such as BLEU and ROUGE capture only a narrow part of performance. Sai et al. (2022) survey studies showing these metrics correlate poorly with human judgment and do not check factual consistency or faithfulness to the source. Neither do they tell a manager whether an output is useful in operation.
Table 1 gathers the evidence a manager may need at different points in the capability’s life, from the original purpose and data approval through release, operation, and later change.
| Evidence question |
Minimum record |
Decision use |
| Purpose |
Named decision, owner, baseline, action, timing, error consequences |
Authorize discovery and method selection |
| Data |
Provenance, permitted use, quality profile, population and aggregation risks |
Authorize analytical use |
| Validation |
Task-aligned metrics, error analysis, uncertainty, reproducibility, subgroup results |
Approve or reject release |
| Operations |
Workflow integration, human authority, service levels, alerts, fallback and incident path |
Control use and response |
| Change |
Training and release log, drift evidence, approvals, suspension and retirement criteria |
Continue, modify, suspend, or retire |
Table 1. Evidence required to manage an analytical capability.
Interpret Explanations Conservatively
SHAP and related feature-attribution methods can help reviewers inspect how a fitted model behaves under a particular dataset and set of assumptions. That is different from showing that changing a feature would cause the outcome to change. Janzing et al. (2020) show why the distinction matters for Shapley-based attribution by separating observational and interventional distributions. In practice, attribution is useful for error analysis and review; causal claims need a causal design.
Translate Uncertainty Into Operating Guidance
A validation report should show where a capability can be trusted, where the evidence remains limited, and how users should respond when confidence is low. Thresholds also need to work in practice. A cutoff that looks strong statistically may still fail operationally if it creates more work than the review process can absorb.
Reviewers should also be able to reconstruct the basis for release from the retained evidence. For high-impact uses, that record should make clear who has authority to intervene or suspend the capability when operating conditions move beyond what was approved (Tabassi, 2023).
Operationalize the Whole Capability
Deployment exposes conditions that development cannot fully reproduce. Real users, current data, workload, interfaces, policy, and mission conditions now become observable, and any of them may change over time. Palaia summarizes the problem directly: “A model can remain mathematically unchanged while becoming operationally unreliable because the population being scored has changed” (personal communication, September 3, 2026). Monitoring becomes useful when a change can be detected, assigned, acted on, and recorded. Figure 4 shows that operating loop.
Figure 4. Monitoring and response loop. Five stages run left to right: monitor data, model behavior, service health, and workflow outcomes; detect drift, failure, latency, queue effects, or missed mission results; assess whether action is needed, who owns it, and how much it matters; act by continuing, correcting, restricting, routing to human review, or suspending; and record the incident, decision, evidence, and any baseline change. A return path leads back to monitoring after the action or baseline change. Design rule: if the user cannot perform an action to alleviate the alert, they should not see it.
Fallback authority should also be established before release. The organization needs to know who can route a decision to human review, restrict use, or suspend the capability when conditions move outside the approved envelope.
Preserve Evidence Through Change
Operational learning needs to become part of the approved record. When it remains scattered across working tools and informal exchanges, later reviewers are left to reconstruct decisions from fragments. Preyna’s (2026) lifecycle evidence boundary provides a way to preserve the source and status of those decisions, along with the rationale and supporting evidence that carried forward. Brooks (2026) shows how structured acceptance criteria can remain connected to executable tests. Together, those practices help preserve the link between mission intent, verification, and sustainment.
Move Validated Insights Into Delivery
Alchemy SDLC™ becomes relevant when validated analytical evidence points to an application, an enhancement to an existing workflow, or a change in how data is captured and checked. The critical success factors identified through data science then become inputs to requirements. Operating conditions and consequences of error inform acceptance criteria, while the analytical assumptions and limitations remain connected to the delivery record. Human reviewers retain authority over the approved baseline. Software-test evidence and model-validation evidence remain distinct because they answer different questions, but the trace between them shows whether the application implements the insight that justified the change.
Measure Use and Outcome
Usage data can reveal where people engage with the capability, where training or workflow changes may be needed, and where the product may no longer be useful. Whether that use produced value is judged against the mission measure established at intake. The review should also look for side effects, including added review burden, automation bias, delayed decisions, or uneven performance across affected groups.
Close the Decision Loop
Periodic review should return the capability to the decision that originally justified its use. The review asks whether performance still supports that decision, whether the evidence remains valid, and whether the conditions surrounding use have changed. The responsible authority can then determine how the capability should proceed. For consequential uses, a decision to continue without change should also become part of the record.
Implications for Acquisition and Oversight
In acquisition, the same management model can inform statements of work and task orders. The requirement can define the operational decision, the evidence the government expects to review, and the conditions under which the capability will be accepted and sustained while vendors still retain room to choose the method they consider appropriate.
Oversight should distinguish production of an artifact from acceptance of an operational capability. A completed model, dashboard, or interface is a delivery milestone. Acceptance depends on evidence that it meets the approved criteria, and continued authorization depends on what happens in use. Contract data requirements and transition plans can preserve configurations, source references, validation results, change history, and operating procedures for a successor team. That reduces dependence on undocumented knowledge during sustainment.
From Validated Insight to Operational Change
ACC3 can support the work in two connected stages. First, the organization identifies the operational outcome, investigates where useful evidence may exist, and determines whether the finding is strong enough to warrant a change. This establishes that the analytical equivalent of paydirt exists before substantial delivery resources are committed.
When that evidence calls for technology, Alchemy SDLC™ provides the structure for building the application or workflow change that puts the insight to use. The validated finding becomes a source for requirements and critical success factors. Structured acceptance criteria support executable testing (Brooks, 2026), while the lifecycle evidence boundary preserves the source, approval, and rationale across delivery and sustainment (Preyna, 2026). The application puts the validated insight into repeated operational use and leaves evidence showing whether the expected value appears in practice.
References
Brooks, K. (2026, July 19). From acceptance criteria to automated assurance: How the Alchemy SDLC™ feeds Playwright-enabled testing from a traceable requirements baseline. ACC3 International. https://acc3int.com/whitepapers/acceptance-criteria-to-automated-assurance/
Janzing, D., Minorics, L., & Blöbaum, P. (2020). Feature relevance quantification in explainable AI: A causal problem. Proceedings of Machine Learning Research, 108, 2907–2916. https://proceedings.mlr.press/v108/janzing20a.html
Preyna, D. (2026, July 25). The lifecycle evidence boundary: Governing AI-assisted delivery through Alchemy SDLC™. ACC3 International. https://acc3int.com/whitepapers/lifecycle-evidence-boundary/
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2022). A survey of evaluation metrics used for NLG systems. ACM Computing Surveys, 55(2), Article 26. https://doi.org/10.1145/3485766
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems 28 (pp. 2503–2511). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
Tabassi, E. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1
