NetSuite Forecast Accuracy: Backtest Before You Buy an Add-On

Executive Summary

A forecast accuracy number is decision-grade only when four things are fixed before anyone reads the result: which forecast was evaluated, which demand it was compared with, which baseline it had to beat, and which planning decision the error informs. This report, written as of October 2026, gives NetSuite planners, inventory controllers and operations leaders an evaluation design for choosing between a new forecasting method, an approved manual override and a forecasting add-on. It is a methodology reference. The worked series are fictional, and no evaluation on client data was performed.

First, freeze the forecast as it existed on each decision date. NetSuite's Item Demand Forecast vs. Actual report compares forecasted demand with demand over a period, and its historical side follows the Transactions to Consider preference. Under Orders, demand is approved, non-canceled sales orders; under Actual Sales, it is cash sales and invoices ^[1]. These are different populations, so the demand basis belongs in the test plan. The Oracle pages reviewed did not describe saved versions of a demand plan, so teams without stored vintages should measure prospectively. Evaluating decisions on later-revised data can make performance look better than it was in real time ^[2].

Second, report a small set of measures against an explicit baseline. Percentage errors are infinite or undefined when an actual is zero ^[3], and Hyndman cautions that WAPE (weighted absolute percentage error) is undefined for all-zero test windows and unsuitable for trending or seasonal data ^[4]. In the fictional example below, a frozen forecast with a 10.7% WAPE on a seasonal item loses to a seasonal-naive forecast at 6.6%, while a "latest revised" forecast shows 1.5%, which shows how hindsight understates error.

Third, treat error as one input to a gate. A study of more than 60,000 forecasts at four supply-chain companies found judgmental adjustments raised accuracy on average in three of them, while smaller adjustments often damaged it ^[5]. Accurate forecasts by no means ensure desired inventory performance ^[6]. The approval sheet should tie each error measure to service, stock and effort constraints and name a signing owner, and a register should keep failed hypotheses and drift checks after release.

Introduction and Background

NetSuite teams are routinely asked whether the forecast is any good. The answer depends on which forecast, compared with which demand, over which horizon, against which alternative. A single accuracy percentage hides all four choices. This report proposes an evaluation design that a planner can document in a workbook and an owner can sign, so a decision to change a method, retain an override or buy an add-on rests on a fair, reproducible comparison.

The audience is supply-chain leaders, chief operating officers, inventory controllers and planners at distributors, manufacturers and direct-to-consumer brands. An error measure is not a replenishment decision, and neither is forecast consumption, the netting of forecasts against incoming orders, which is a separate topic not covered here. The report does not assess any vendor's quality; it describes how a buyer can test claims on its own data.

Public practitioner coverage of NetSuite demand planning concentrates on setup and method selection. One consultancy guide lists four statistical methods, a demand-versus-actual comparison through saved searches, and a first-year MAPE (mean absolute percentage error) benchmark for fast-moving items (Source: www.anchorgroup.tech); another post walks through a calculate, review and order workflow ^[7]. This report adopts no universal error target, and no public benchmark of NetSuite forecast accuracy was found.

Houseblend, a NetSuite consultancy, helps teams configure the data extraction, planning workflows and integrations that such an evaluation needs. That describes a service, not an outcome: no accuracy improvement or commercial result is measured here.

Freezing the Forecast: Vintages, Decision Dates and Overrides

State the decision and population first

Write down the decision, the item and location population, the horizon and the minimum evidence needed. Retiring an override on stable items is a different test from adopting a seasonal method for promotional items. The horizon should match the lead time the buyer faces: a one-period-ahead error says little about an order placed twelve weeks before receipt. The textbook guidance is that a test set should ideally be at least as large as the maximum horizon required ^[3] and is typically about 20% of the sample ^[3]; short or sparse histories may need a documented alternative.

Why vintage matters

A forecast made after the target period cannot represent the information available at the buying decision. Macroeconomists formalized this with real-time data sets built from vintages, or snapshots, of data as available at the time ^[2]. The St. Louis Fed's ALFRED database retrieves each data release available on a specific date ^[8], and notes that sources revise data at multiple moments ^[9]. A Federal Reserve Bank of Philadelphia working paper reports that a leading-indicator index did a much worse job of predicting output in real time than after data revisions ^[2]. The demand-planning analogy is direct: a plan rebuilt after the month closes is a revision, not a forecast.

For each decision date, retain these records, and store later changes separately:

• Vintage: the as-of timestamp at which the forecast was frozen.
• Model and version: method, parameters and history window.
• Inputs then available: demand history, open orders and calendars as of that date.
• Manual override: overridden quantity, reason code and approver.
• Later revisions: kept apart and never scored against that vintage.

What NetSuite documentation says, and does not say

Oracle's documentation describes monthly, weekly and daily plan types ^[10], lets a plan be entered manually instead of calculated ^[11], and allows the quantity on a plan line to be changed when editing ^[12]. Only the custom form, unit of measure and memo header fields are editable ^[12], and a plan shows the date of its most recent changes. The pages reviewed did not describe stored plan versions. Oracle states that NetSuite does not consider past forecast quantities when estimating demand ^[13]; that describes calculation inputs, not record retention, so it is not a statement about audit history. Confirm in a sandbox whether system notes, saved searches or exports can reconstruct each vintage in your account.

Also verify the planning engine. Oracle states that demand plans can only be created for items whose replenishment method is Time Phased ^[10], that Supply Planning (MRP) replaces Time-Phased Planning ^[14], and that existing customers should plan to move to MRP ^[14]. Demand Planning also processes up to 10,000 items at a time ^[15], which affects how a large population is batched.

When no snapshots exist

Many teams never stored vintages. A past evaluation is then impossible, and the correct response is prospective measurement: start freezing forecasts now and evaluate after enough target periods accumulate. Running today's model on yesterday's data, with today's item mappings and cleansed history, produces a hindsight reconstruction. It can compare methods under identical information, but it must be labeled a reconstruction and never presented as what the business knew.

Defining Actual Demand Consistently

Two Oracle definitions, two populations

Oracle describes the report as showing forecasted demand versus demand across a period ^[1], with historical demand based on the Transactions to Consider preference ^[1]. Orders means approved, non-canceled sales orders ^[1]; Actual Sales means cash sales and invoices ^[1]. The setup documentation points companies that mix sales orders with standalone cash sales or invoices to Actual Sales ^[16]. The report is date-based: it does not support reporting by period even when the Report by Period preference is set to All Reports ^[1]. A mismatch between fiscal periods and report date ranges is a definitional risk to document, not a defect to assume. Oracle's history documentation also says demand calculations consider only transactions that decrease an item's stock level ^[17], so returns, transfers and adjustments need an explicit rule.

Match the basis to the question

An order-based definition measures when customers committed; an invoice-based definition measures when billing occurred, which can follow shipment by days or weeks. Neither is automatically unconstrained customer demand. A paper on forecasting with censored data states that sales represent a constrained demand relative to underlying unconstrained demand ^[18], and that using sales as the proxy tends to give downward-biased forecasts ^[18]. In retailing, unmet demand is often lost and unobserved ^[19]. Choose the basis that matches the decision under test (order-based for a supplier purchase, billed for revenue planning, for example), apply it identically to every candidate and record it in the data contract.

Distinguish what a zero means

A zero actual can be true zero demand, a missing observation, a stockout-censored period, a period before launch or after discontinuation, or an artifact of a changed unit of measure or item mapping. Each needs its own status flag. Treating them all as zero demand biases both training history and test errors.

Backtest Design: Windows, Baselines and Horizons

Rolling origins

Evaluate on genuine forecasts, not on fit to training data ^[3]. The textbook calls time series cross-validation a more sophisticated version of training/test sets ^[3], requires that no future observations be used in constructing the forecast ^[3], and names the procedure evaluation on a rolling forecasting origin ^[20]. Scikit-learn's documentation explains why shuffled cross-validation fails here: it would lead to training on future data and evaluating on past data ^[21]. A cloud forecasting service's documentation notes that increasing the number of backtests generally produces more reliable metrics ^[22]. Choose origins that cover the seasonal cycle and the lead-time horizon, and report accuracy by horizon, because error grows as the horizon lengthens ^[20].

Leakage checks:

• Future inputs: no feature dated after the origin.
• Retroactive cleansing: history as it stood at the origin, including returns posted later.
• Mappings: the item hierarchy and unit of measure valid at the origin.
• Overrides: one entered after the target period began is not part of that vintage.
• Model selection: tune on training windows only, then report untouched test errors.

Choose the baseline explicitly

The textbook defines the naive method as setting all forecasts to the value of the last observation ^[23] and seasonal naive as setting each forecast to the last observed value from the same season ^[23]. Such methods serve as benchmarks rather than the method of choice ^[23], and a new method should be checked against them ^[24]. For a seasonal item the seasonal-naive baseline is the fair opponent. State the choice before the test.

The M5 competition shows why. It asked for point forecasts of 42,840 hierarchical unit-sales series ^[25], validated on a hidden 28-day sample ^[25]. In the working-paper version of the results, 48.4% of teams outperformed the naive benchmark, while 35.8% beat seasonal naive and 7.5% beat the best benchmark ^[25], among 7,092 participants ^[25]. These are public retail data, not a NetSuite benchmark, but they show that a plain baseline is not a trivial opponent.

Score every candidate on the same items, buckets and horizons. Dropping items a candidate cannot forecast, then comparing it with a method that forecasts everything, inflates its apparent accuracy; report coverage as its own column.

Error Measures: Choose a Small Set

Define signed error as actual minus forecast, consistently. The textbook defines a forecast error as the difference between an observed value and its forecast ^[3], but some software reverses the sign: one open-source metrics library defines bias as prediction minus actual ^[26]. Write the convention into the workbook and recheck it whenever a tool's output is imported. Bias matters because a mean residual other than zero indicates biased forecasts ^[27], and bias tends to be under-represented in the literature relative to accuracy ^[28].

Table 1 below lists the recommended candidates with a decision use and failure case. In it, e is the signed error, y is actual demand, and m is the seasonal period (1 for non-seasonal data).

No row is a universal best. The choice depends on the loss the business bears, and the decision record should say which measure was chosen and why. The set recommended here is small: bias, MAE or WAPE for units, and MASE (optionally RMSSE) against a named baseline. The textbook notes that scaled errors scale test errors by the training MAE from a simple forecast method ^[3].

MAPE, WAPE and the debate

MAPE fails with zeros, as the textbook states ^[3], and a paper in the International Journal of Forecasting makes the same point for intermittent demand ^[30]. Its minimization is often not a good idea ^[31]. The symmetric variant, sMAPE, is no clean fix: a 2026 paper by Kolassa concludes that, in the author's opinion, it is either unnecessary or dangerous ^[32].

WAPE is the usual replacement in demand planning. Hyndman notes it was introduced by Kolassa and Schütz (2007) who called it the MAD/Mean ratio ^[4], which they presented as a weighted mean of absolute percentage errors ^[33]. One library documents it as robust to zero actuals and weighting higher-volume points more ^[34]. Hyndman's 2025 analysis is more cautious: the WAPE is undefined when a test window contains only zeros, a case he links to intermittent demand and small test sets ^[4]; it is only consistent for stationary series ^[4]; and it should not be used with trends, seasonality or heteroscedasticity ^[4]. His own preference for a scale-free measure is RMSSE ^[4], whose denominator comes from training data that is much less likely to contain only zeros ^[4].

Handle the disagreement by design. Use WAPE for portfolio communication in units, with a zero-denominator flag and a stated window, and a scaled measure for model comparison. Check formulas, not labels: a cloud forecasting service's documentation notes it previously referred to its WAPE metric as the Mean Absolute Percentage Error ^[22]. One 2016 paper also argues that measures such as MAD, MASE and wMAPE are inherently unsuitable for count data ^[35], another reason to keep the set small and disclose the loss each implies.

Segmenting Items Before Comparing Methods

A pooled score can hide opposite outcomes, so segment first.

• Intermittent items: many zero periods, so percentage measures fail. One classification scheme uses an average demand interval cut-off of 1.32 and a squared coefficient of variation cut-off of 0.49 ^[36]; its authors tested it on 3,000 real intermittent series from the automotive industry ^[37]. A later discussion finds the split between two intermittent-demand methods does not form four distinct areas ^[38], so treat the cut-offs as a starting rule.
• New items: no history and so no training scale. Report them separately with an analogue-item baseline; Oracle's calculation documentation lets another item's history supply the demand calculation ^[13], and the analogue chosen belongs in the vintage record.
• Seasonal items: use a seasonal-naive baseline and a test window covering a full season.
• Promotional items: record the promotion calendar as an input available at the decision date and separate promoted from unpromoted periods.

The M5 organizers state that it was the first M competition whose series display intermittency ^[25]; the University of Nicosia describes the majority of series as intermittent (Source: www.unic.ac.cy). They chose a scaled squared-error measure because it can be safely computed for all series ^[25], noting that absolute errors are optimized for the median ^[25], and computed the scale only over periods when the product was actively sold ^[25]. For a NetSuite data contract the lesson is to scale a series only over its active life and flag pre-launch periods.

Do not impute invented lost demand. Lost demand is mostly unknown ^[18], so any reconstruction rule is a modeling assumption, and applying it to one candidate but not another is a leakage route. Carry an availability flag, report accuracy with and without flagged periods, and let the owner see both.

The Data Contract and the Approval Sheet

An evaluation is reproducible only if the extract has fixed fields. Table 2 below proposes a contract; map your own field names to it.

Table 2 makes two requirements visible. Model and approved quantities are separate fields, so the value of an override can be measured; and the demand definition is a field, not a footnote. Forecast value added, the change in a performance metric attributable to a particular step in the process ^[39], formalizes the first.

What the override evidence suggests

Overrides are not inherently good or bad. In the study of more than 60,000 forecasts and outcomes from four supply-chain companies, judgmental adjustments increased accuracy on average in three companies ^[5], but the smaller adjustments often damaged accuracy ^[5], and the authors point to a general bias towards optimism ^[5]. Judgment can improve accuracy but also suffers from biases and inconsistency ^[40]. These samples are not a prediction for any NetSuite account; they are a reason to score the override separately, with signed error, before retaining or retiring it.

Approval sheet

Table 3 below links evidence to the decision. Thresholds are blank on purpose: the owner sets them before the test from the cost of a miss in that business, not from a borrowed benchmark.

The same error table can support different decisions under different constraints. A method that lowers MAE but increases under-forecast bias on long-lead items may raise stockout risk. The correct choice of accuracy metric depends on the organization's inventory rules ^[41], and in one study traditional safety-stock approaches built on forecast error could leave stocks up to 30 percent too low ^[42]. Link any method change to the separate safety-stock approval, which has its own tests.

Data Analysis and Evidence

Fictional worked example (Hypothetical Example)

The series below are invented for illustration. They are not a benchmark, a client case or a measurement of NetSuite. All quantities share one base unit. The calculations are the report author's, using the definitions in Table 1.

Item A (intermittent, weekly). Twelve training weeks: 0, 3, 0, 0, 5, 0, 2, 0, 0, 4, 0, 1. Six test actuals: 0, 2, 0, 0, 6, 0. The frozen approved forecast was 1.5 per week (model output 1.25 raised by an approved override). Naive is the last training value, 1. A "latest revised" forecast, written after the actuals were known, was 0, 2, 0, 0, 5, 1.

Item B (seasonal, quarterly buckets). Eight training quarters: 100, 140, 90, 200, 110, 150, 95, 215. Four test actuals: 120, 160, 100, 230. Frozen forecast: 125, 150, 110, 190. Seasonal naive (same quarter last year): 110, 150, 95, 215. Naive: 215 repeated. Latest revised, written after the fact: 120, 158, 102, 225.

Scaling uses the training series. For item A the scale is the mean absolute one-step change, 29 / 11 = 2.636. For item B it is the mean absolute lag-4 change, (10 + 10 + 5 + 15) / 4 = 10, so a MASE of 1 equals the training seasonal-naive error. Table 4 below gives absolute errors by horizon for item B, then WAPE and MASE for both items, with the naive comparison and the revised forecast as rows. Item B uses a single origin for exposition; a real test would use several.

Four points follow. First, item B's frozen forecast has a respectable-looking 10.7% WAPE, yet the free seasonal-naive rule scores 6.6% and the frozen MASE is 1.63 against 1.00: an accuracy percentage without a baseline hides that a free rule did better. Second, item A's MAPE is undefined because four of six actuals are zero, and its 137.5% WAPE is valid but of little decision value on a sparse item. Third, on item A absolute error slightly favors naive (MASE 0.63 against 0.70), while the squared-error scaled measure RMSSE slightly favors the frozen forecast (0.70 against 0.71), a reversal that shows how loss preferences differ. Fourth, the revised forecast scores 1.5% and 25.0%, far better than the vintage; that gap is the understatement of historical error from evaluating the latest revision instead of the frozen forecast.

Bias adds what absolute error hides. Item B's frozen forecast averaged 8.75 units too low, concentrated in the last quarter, where actual exceeded forecast by 40. If that quarter is the peak, the under-forecast sits where a stockout costs most.

Portfolio WAPE is not an average of item percentages. The frozen forecasts for A and B have summed absolute errors of 11 and 65 against summed actuals of 8 and 610, so portfolio WAPE is (11 + 65) / (8 + 610) = 12.3%. The simple average of item WAPEs, (137.5% + 10.7%) / 2 = 74.1%, weights an eight-unit item equally with a 610-unit item. In practice the window and unit must be common, and teams may prefer value-weighted aggregation; document the choice.

What public sources can and cannot support

The public sources reviewed support definitions and design rules. They do not quantify how accurate NetSuite's built-in methods are or how much an add-on would change accuracy. NetSuite's product page states that Demand Planning delivers improved forecast accuracy and is available as an add-on ^[43], a vendor statement with no figure on the page. Oracle documents linear regression using ordinary least squares ^[13] and a sales forecast method using forward-looking sales forecast data ^[13].

Competition results show that baselines change conclusions. A University of Nicosia paper reports that M4 had 100,000 series ^[44] and that only a small number of methods outperformed a simple combined baseline ^[44]; in M5, seasonal naive was on average 11.5% more accurate than Croston's method at the product-store level ^[25]. These are public data with competition scoring rules, not forecasts of any NetSuite deployment.

Comparing Native Planning, Add-ons and Overrides

Teams face three options: improve the native process, keep human overrides, or add third-party software. Native demand planning offers four statistical methods (moving average, linear regression, seasonal average and sales forecast), as practitioner guides also describe (Source: www.anchorgroup.tech). Vendors describe their own approaches in their own words. One says its software identifies each product's demand profile and applies the most appropriate model ^[45]. Another says forecasts are corrected for stockouts and promotions ^[46] and reports an accuracy improvement tested on a random sample of 46,000 products ^[46]. A third states that customers can reduce inventory by up to 25% ^[47]. Another publishes the position that MAPE does not apply to probabilistic forecasts ^[48]. A partnership announcement notes that many NetSuite customers still manage inventory with standard ERP functionality ^[49]. These are attributed vendor statements, and none was reproduced for this report.

The design above turns such claims into testable questions. Run the candidate on your frozen vintages and demand basis with the same horizons and items. Ask which metric and baseline stand behind any stated improvement, since a percentage gain in one measure on one population is not a gain on yours. If output is probabilistic, compare it with a probabilistic metric as well as a point forecast, and fix the decision rule in advance. A native-versus-add-on comparison is fair only when both start from identical information and the same override policy; otherwise an override may be credited to software or blamed on it.

Financial forecasting add-ons follow the same logic with different objects: a revenue or expense line against the booked actual, the approved budget or reforecast as the vintage, and last year's actual or a run-rate as the baseline. No NetSuite Planning and Budgeting product page was retrievable for this research, so no vendor claim on it is reported. Implementation effort belongs in the gate: count extraction work, calendar mapping, training, integration and the time to produce each vintage.

Implications and Future Directions

Monitor after release. Keep freezing vintages and compare live results with the backtest. Track bias, scaled error and the share of forecasts that fail to beat the baseline; one study found around 50% of individual forecasts failing its naive benchmark ^[50], so a live baseline check is cheap insurance. Drift appears as a persistent shift in signed error before it moves a headline percentage.

Keep an evaluation register. Each entry records the hypothesis, population, demand basis, vintages, baseline, metrics, result, decision and owner, with failed hypotheses included. A method that lost to seasonal naive saves the next team a quarter of work.

Know the limits. A lower average error does not by itself prove better service, lower stock or a profitable supplier choice. Error measures do not model order quantities, lead-time variability, minimum order quantities or supplier terms. Once the error comparison has narrowed the candidates, evaluate the planning decision itself, for example by simulating the order policy on the frozen vintages.

What remains unknown. No public evidence reviewed here shows how often NetSuite users retain historical vintages, how built-in methods compare with add-ons on a common population, or whether any evaluation design improves service. Until anonymized, method-documented comparisons are published, own-data backtesting and prospective measurement are the defensible approach. Disclosure: the publisher of this article, Houseblend, offers NetSuite consulting services; the framework here does not depend on that or any provider.

Frequently Asked Questions (FAQs)

How do I measure forecast accuracy in NetSuite?

Choose a demand basis (Orders or Actual Sales, per the Transactions to Consider preference), freeze the forecast at each decision date, extract forecast and actual using the fields in Table 2, and compute bias, MAE or WAPE and a scaled error against a named baseline. The native report is a comparison view over a date range, not a vintage store.

What is the difference between MAPE and WAPE?

MAPE averages each observation's percentage error; WAPE divides total absolute error by total actual demand. MAPE is undefined at a zero actual ^[3]; WAPE only when all denominator actuals are zero ^[4], though Hyndman cautions against it for trending or seasonal data ^[4].

How do I backtest a forecast without stored snapshots?

You cannot recover what the business knew. You can run a labeled rolling-origin reconstruction of methods ^[3] and begin freezing vintages now for a prospective test.

Is NetSuite Demand Planning different from third-party forecasting software?

Oracle describes NetSuite Demand Planning as an add-on with methods such as linear regression. Third-party tools describe other approaches in their own materials. Compare them on your frozen vintages, not on stated outcomes.

Conclusion

A forecast accuracy comparison is a design problem before it is a calculation. State the decision and population. Freeze the forecast as it existed at each decision date, with its model, inputs, override and approver. Define actual demand on the Orders or Actual Sales basis that matches the question and keep availability flags instead of imputed lost demand. Run rolling-origin tests against a named naive or seasonal-naive baseline on identical items and horizons. Report a small set of measures, and never average item percentages into a portfolio figure.

The fictional example shows three recurring traps: a respectable WAPE that loses to a seasonal-naive baseline, a metric that is undefined on zero-heavy items, and a latest revised forecast that understates error through hindsight. None is specific to NetSuite or to any add-on.

The approval sheet should connect each result to the planning decision, with thresholds set in advance by the owner and implementation effort counted. After release, keep freezing vintages, watch bias and drift, and record failed hypotheses. Public evidence on NetSuite-specific accuracy is limited, so own-data evaluation is the sound basis for a decision.