Skip to content
TradingTune
TradingView Optimization

How Decimal Precision Affects Search Results

Teaches how decimal steps can create many combinations and make exported values harder to compare.

On this page

Key takeaways

  • A finer decimal step multiplies the number of combinations in a sweep and makes each exported row harder to compare.
  • Decimal steps add floating-point noise to exported values, so two runs that should tie can look like different outcomes.
  • A value that looks best at one very fine step can be a coincidence in one stretch of history, not a stable property of the strategy.
  • The method is a coarse pass first, a finer pass only on the band that showed movement, and an out-of-sample check last.

How Decimal Step Size Shapes Your Parameter Sweep

Shrinking a parameter’s step from a whole number to a decimal multiplies the size of your sweep and makes every exported row harder to line up against the last one. If you sweep a multiplier from 1.0 to 3.0 in steps of 1.0, you test three values. Drop the step to 0.1 and you test twenty-one. Drop it to 0.01 and you test two hundred and one, each producing its own equity curve, its own cycle count, and its own row in the export. The combinatorial growth is not a rounding curiosity; it is the direct mechanical cost of finer precision, and it compounds fast once you are sweeping two or three parameters at once instead of one, since the counts multiply against each other rather than add. Formal treatments of numeric precision in tabular data make the same point about how decimal representation affects how values are stored, compared, and rounded when reconciling large result sets bookdown.org, which is exactly the friction a trader hits when diffing two CSV exports from the same sweep.

Three things follow from this. First, a finer step is not automatically a better sweep; it is a bigger one, and the extra rows need a reason to exist beyond “more precision felt safer.” Second, decimal steps introduce floating-point noise into exported values, so two runs that should tie can differ in the fourth decimal place and look like distinct outcomes when they are not. Third, comparing rows by eye gets unreliable past a few dozen combinations, which pushes you toward sorting and rounding the export before you trust any pattern in it. This is written for traders running parameter sweeps on their own TradingView Pine strategy who have noticed their optimizer runs taking longer, or their exported results getting harder to scan, as they added decimal precision to a range.

Setting Up a Decimal-Stepped Parameter Range

Before you touch the step field in TradingTune’s optimizer, decide what the range is actually testing. Take a simple multiplier that scales a stop-loss distance off average true range. A whole-number sweep from 1 to 5 asks a coarse question: does the stop need to be roughly one, two, three, four, or five times the volatility unit? A decimal sweep from 1.0 to 2.0 in steps of 0.1 asks a finer question: does the strategy care whether the stop sits at 1.3x or 1.4x specifically? That is a legitimate question to ask, but only once the coarse sweep has already told you the answer lives somewhere in that narrower band. Sweeping fine from the start, before you know where the interesting region is, just multiplies your row count without multiplying your insight.

Decimal precision, in this context, is how many digits after the decimal point a swept value is allowed to carry, and step size is the fixed distance between one tested value and the next. A step of 0.01 across a range of 1.0 to 3.0 does not just add more rows: it adds rows that differ from their neighbors by an amount smaller than the noise in a single backtest run, which is a distinction worth sitting with before you commit compute time to it.

Walk-forward testing is the discipline of splitting your historical data into sequential windows, optimizing the parameter on an earlier window (the in-sample segment), then checking that same parameter, unchanged, on the window immediately after it (the out-of-sample segment), and rolling that pair forward through the whole history. TradingTune runs this as a cycle: pick a lookback of, say, 200 to 300 bars for the in-sample fit, then a shorter forward window of 50 to 100 bars to validate, then slide both forward and repeat. Every value in your decimal-stepped range gets run through every cycle, which is why the row count and the cycle count multiply together rather than just adding.

When I ran a stop-multiplier sweep from 1.0 to 2.0 at a 0.05 step against six walk-forward cycles, I watched the exported table balloon from twenty-one combinations to well over a hundred rows once the cycle dimension was folded in, and I noticed that sorting by any single column stopped being useful past the first screen. What helped was picking the step size deliberately: coarse enough to survey the whole plausible range in one pass, fine enough only in the narrow band the coarse pass flagged as worth a second look.

What Happens as Decimal Precision Changes Across a Sweep

Move the same stop multiplier across a decimal-stepped range and two things happen at once: the coarse ends of the range behave predictably, and the exported rows in the middle become nearly impossible to tell apart at a glance. Those are separate problems, and conflating them is how a trader ends up trusting a pattern that is really just export noise.

At the low end of a range, say a multiplier crawling from 1.0 to 1.3 in steps of 0.05, the equity curve barely moves between adjacent values. The cycle count stays flat, the win distribution across walk-forward windows looks nearly identical row to row, and the curve shape does not bend so much as flicker. That flicker is real information: it tells you the strategy is insensitive to the parameter in that band, which is worth knowing, but it also means six or seven of those rows are functionally redundant. At the high end of the same range, past roughly 1.7 or 1.8 in this example, the behavior can flip: the curve starts to bend sharply between one decimal step and the next, cycles that used to validate cleanly start disagreeing with each other, and the out-of-sample segment stops tracking the in-sample fit. Coarse and fine ends of a range rarely behave the same way, and a sweep that only samples densely in one region will miss that the interesting instability is somewhere else entirely.

The second observation sits in the export itself, not in the strategy’s behavior. Once you are comparing a hundred-plus rows across six cycles, small formatting differences in how each run rounds its decimal output start to matter. A value stored as 1.30 in one cycle and 1.3000001 in another, because of how the underlying calculation accumulated floating-point error across bars, reads as a distinct row when it is not. Lining up two exports side by side to check whether a result held from one cycle to the next becomes a manual rounding exercise before it can become an actual comparison, and that manual step is exactly the kind of friction that formal treatments of decimal representation in tabular data describe when they discuss how stored precision affects downstream comparison and reconciliation tulip.co. The practical fix is not clever: round every export to the same number of decimal places before comparing rows, and treat two values within that rounding tolerance as the same result rather than two different ones. Skipping that step is how a trader convinces themselves a parameter matters when the only thing that changed was how the ninth digit got written to disk.

TradingTune for Chrome

Stop tuning your strategy's inputs by hand.

Add to Chrome, it's free

Free tier, no API keys. A free account is required to run. Not affiliated with TradingView, Inc.

Reading an Illustrative Decimal Precision Table

The following table is hypothetical, the founder’s own Smart Momentum Grid, historical and illustrative only.

Parameter Range Step Cycle count Lookback bars
Stop multiplier 1.0 to 2.0 1.0 6 200 bars
Stop multiplier 1.0 to 2.0 0.1 6 200 bars
Stop multiplier 1.0 to 2.0 0.01 6 200 bars
Lookback window 100 to 300 50 6 250 bars
Lookback window 100 to 300 10 6 250 bars
Entry threshold 0.5 to 1.5 0.25 6 300 bars

Reading this table row by row is a lesson in what decimal precision buys you and what it costs, not a lesson in what any single row returned. The first three rows hold everything constant except the step on the same stop-multiplier range, and the point of stacking them like that is to isolate step size as the only variable in play. Going from a step of 1.0 to a step of 0.1 takes the row count for that parameter from two values to eleven, and dropping to 0.01 pushes it past a hundred; fold in six walk-forward cycles per value, the way the lookback bars column pins down, and the export size for that one parameter alone crosses six hundred rows before you have touched a second dimension. That is the mechanical growth the earlier sections described, laid out as an actual count rather than an abstraction, and it is the same relationship formal treatments of decimal representation describe when they discuss how precision choices govern how many distinct stored values a dataset can hold before comparison and reconciliation break down wikipedia.org.

The lookback and entry-threshold rows earn their place in the table for a different reason: they show that the same discipline applies outside the stop-multiplier example. A lookback window swept in steps of 50 bars versus 10 bars is not a decimal question at all, it is a plain integer one, but it produces the identical shape of problem, a coarse pass that surveys the whole plausible range against a fine pass that multiplies row count without necessarily multiplying insight. A practitioner scanning this table would probe the middle rows first, the 0.1-step stop multiplier and the 10-bar lookback step, because those are the resolutions fine enough to catch a real inflection point without yet drowning the export in redundant neighbors. The 0.01-step row is the one worth flagging rather than trusting outright: it is the row most likely to contain values that differ from each other by less than the noise floor of a single backtest, which is exactly the kind of stored-precision question that guidance on setting display precision in structured tables addresses when it recommends rounding a column to a resolution that matches what the underlying measurement can actually support wikipedia.org. Read that way, the table is not a result to memorize, it is a map of where the next round of sweeping should and should not spend compute.

When Fine Decimal Steps Create an Overfit Trap

The clearest failure mode I have hit is a stop multiplier that looked like the best row in the export and then fell apart the moment the walk-forward split moved forward one window. Overfitting, defined plainly, is when a parameter fits the noise of one specific stretch of history so tightly that it stops describing anything general about how the strategy behaves. A decimal step fine enough to isolate a single lucky combination is exactly the mechanism that lets overfitting sneak past you, because the sweep has enough resolution to land on a value that happened to dodge every bad trade in the in-sample window without that dodge meaning anything.

Here is the shape of it. I was sweeping the same stop multiplier from the earlier sections, but this time at a 0.02 step across a narrow band from 1.40 to 1.60, chasing what looked like a promising region from a coarser pass. In the in-sample segment of one walk-forward cycle, a value of 1.52 produced a noticeably smoother equity curve than its neighbors at 1.50 and 1.54. Nothing else about the row stood out: the cycle count matched, the lookback bars matched, only the curve shape looked cleaner. When I tested that same 1.52 value, unchanged, against the out-of-sample segment immediately following it, the curve gave back most of what it had gained in-sample and the shape that looked smooth suddenly looked ordinary, no better than its neighbors at 1.50 or 1.54. The fine step had not found a genuinely better multiplier; it had found the one value narrow enough to fit a handful of specific bars in that particular window, a fit that a coarser 0.1 step would never have surfaced because it would have skipped right past 1.52 entirely.

What fooled me was the assumption that a smoother in-sample curve at a step that fine was itself evidence of something real. It is not, on its own. A parameter that only looks good at one decimal step out of twenty neighboring steps, with no smooth trend connecting it to its neighbors, is describing a coincidence in that specific historical window rather than a stable property of the strategy. The related read from research on decimal representation and precision in stored data makes a parallel point: precision that exceeds what the underlying measurement can support does not add information, it adds combinations that look distinct but are not meaningfully different wikipedia.org. The lesson I took from it is a checking habit, not a rule against decimals: any row that stands out sharply from its immediate step-neighbors, at a fine step, needs to survive at least one full out-of-sample window, unchanged, before it earns any trust.

Rounded Stop Multiplier Input. Constrains a stop-multiplier input to a coarse, fixed decimal resolution so a sweep cannot silently drift into overfit-prone fine steps.

Pine v6
//@version=6
indicator("Rounded Stop Multiplier Example", overlay=true)
rawMult = input.float(1.5, "Stop Multiplier", minval=1.0, maxval=2.0, step=0.1)
// Round to one decimal place regardless of any finer internal adjustment
roundedMult = math.round(rawMult * 10) / 10
atrValue = ta.atr(14)
stopDistance = atrValue * roundedMult
plot(stopDistance, title="Stop Distance")

The snippet itself is deliberately unglamorous: it takes a float input, rounds it to a fixed decimal resolution before it touches the ATR calculation, and plots the resulting distance. That rounding step is the practical guard against the trap above. If the sweep can only ever land on values a tenth apart, no single combination can hide inside a gap narrower than the noise floor of the backtest, which is the same discipline the earlier sections argued for when they described choosing a step deliberately rather than defaulting to the finest one available.

Choosing a Practical Decimal Step for Your Own Sweep

My take is that the starting step on any sweep should be the coarsest one that still lets you see the shape of the curve across the whole range, not the finest one your optimizer allows. Start a stop multiplier at a step of 0.5 or even 1.0 across its full plausible range, let that first pass tell you where the curve flickers and where it bends, and only then narrow the step inside the band that showed movement. Refining everywhere is how a twenty-one-row sweep turns into two hundred rows that mostly repeat each other; refining only where the coarse pass flagged instability keeps the row count tied to actual information instead of to how fine the step field lets you go.

I recommend treating the coarse pass as the whole point of the first sweep, not as a throwaway step before the “real” one. Run the wide range at a wide step, walk it forward through your usual cycle count, and look for exactly two things: a region where the curve stays flat across several neighbors, and a region where it bends or where cycles start disagreeing with each other. The flat region tells you the strategy does not care much about that stretch of values, so it does not need a second, finer pass at all. The bending region is the only place a decimal step earns its cost, because that is where a tenth of a unit might actually separate two genuinely different behaviors rather than two rows that differ only in the ninth digit written to the export, a distinction the literature on stored decimal precision frames the same way when it discusses how finer representation adds distinct values without necessarily adding distinct information wikipedia.org.

The concrete next step, if you are setting this up for your own strategy: pick your parameter’s full plausible range, run it once at a step no finer than a twentieth of that range, note which sub-band shows curve movement across walk-forward cycles rather than flat agreement, then re-run only that sub-band at a step five to ten times finer than the first pass. Whatever row stands out in that second pass, hold it against one full out-of-sample window, unchanged, before you treat it as anything more than a candidate worth watching. That sequence, coarse first, narrow second, verify third, is the whole method; nothing here names a value to trade, only a way to spend sweep time on the part of the range that can actually tell you something.

Decimal Precision as a Design Choice, Not an Afterthought

Step size is not a technical afterthought you set once and forget; it is the single decision that determines whether your export is a readable map of how a parameter behaves or an unreadable pile of near-duplicate rows. Every example in this post traces back to that one choice. A step of 1.0 across a stop multiplier gives you three or five honest rows to reason about. A step of 0.01 across the same range gives you two hundred rows, most of which differ from their neighbors by less than the floating-point noise a single backtest already carries. The row count is not a side effect of precision; it is the direct, mechanical price of it, and it multiplies against every other parameter you sweep and every walk-forward cycle you run alongside it. Treating step size as a deliberate design choice, made before you touch the optimizer, is what keeps that multiplication tied to actual insight instead of to how fine a field happens to let you type. Coarse first to find the shape of the range, narrow only where the shape moved, verify whatever narrow result stands out against an out-of-sample window it has never seen: that sequence is the whole discipline, and it only works if you pick the step on purpose rather than by habit.

That discipline answers the question this post opened with. Decimal precision affects your results not because finer numbers are somehow less true, but because finer steps generate more combinations than any trader can honestly compare by eye, and because the exported values themselves pick up rounding noise that makes two ties look like two different outcomes. The risk it protects against is concrete: a sweep fine enough to land on a single lucky value in one historical window, a value that looks like a discovery and is actually a coincidence the walk-forward split would have caught if you had checked it. Reserve fine decimal steps for the narrow band a coarse pass has already flagged as worth a second look, round your exports before you trust any comparison between rows, and hold every standout result against one full out-of-sample cycle before it earns anything more than curiosity. That habit, checking a fine-step result against data it has not seen, is the one validation step worth carrying out of every sweep you run.

Sources

TradingTune for Chrome

Stop tuning your strategy's inputs by hand.

TradingTune runs systematic parameter searches on your own TradingView strategy, right inside the chart page. You keep the strategy, you read the results.

Free tier, no API keys. A free account is required to run. Not affiliated with TradingView, Inc.

Share this post