Reducing RAM requirements with dynamic dust

[uuowwpevskfcordh]

Given the increasing RAM requirements, due to the increasing UTXO set, I suggest seeing the UTXO set size as a controlled variable. A feedback mechanism sets a dynamic dust level, below from which UTXOs are removed/discarded and thus freeing RAM.

Below is an overview essay better expressed by grok, which can also be seen in here:

https://hackmd.io/P-2lzGb8TiC86IOE3OGiYA?view

# Enhancing Bitcoin's Scalability: A PID-Inspired Approach to Managing UTXO Set Growth

## Abstract

Bitcoin’s UTXO set is currently an unbounded accumulator that risks long-term centralization as node RAM requirements grow without limit. Existing fee incentives have proven insufficient against sustained low-value output creation (e.g., inscriptions, tokenized assets, dust-heavy protocols). This article proposes a soft-fork mechanism that treats UTXO set size as a controlled variable: a slowly rising target size is defined, and a PID-style feedback controller, updated every difficulty epoch, dynamically raises a minimum-value floor beneath which old UTXOs become unspendable. The result is bounded, predictable growth of the UTXO set with ample warning periods, no hard caps on monetary use, and strong resistance to bloat attacks—all while remaining fully compatible with a soft-fork deployment.

## Introduction

Bitcoin, the pioneering decentralized digital currency, operates as a complex dynamic system governed by consensus rules that ensure security, immutability, and permissionless participation. At its core, Bitcoin maintains a distributed ledger known as the blockchain, which records all transactions in a sequence of blocks. Each transaction involves inputs and outputs: inputs reference previously unspent outputs from prior transactions, while outputs create new spendable units called Unspent Transaction Outputs (UTXOs). The UTXO set represents the aggregate state of all currently spendable coins in the network, serving as a critical component for transaction validation and wallet management.

As Bitcoin has evolved, the UTXO set has grown significantly, influenced by increasing adoption and diverse usage patterns. This growth, while indicative of the network's vitality, poses challenges to its long-term scalability and decentralization. Nodes—computers that validate and relay transactions—must store and process the entire UTXO set in memory for efficient operation, which can strain resources such as random access memory (RAM). Unchecked expansion could lead to higher barriers for running full nodes, potentially centralizing control among fewer, well-resourced participants. This article explores a proposed mechanism to address these concerns: an adaptive control system inspired by Proportional-Integral-Derivative (PID) feedback principles, designed to impose bounded growth on the UTXO set while preserving Bitcoin's foundational properties.

## Motivations for UTXO Set Management

To appreciate the need for enhanced UTXO management, it is essential to understand the current status quo and its vulnerabilities. Bitcoin's design prioritizes efficiency and security through mechanisms like block size limits, which cap the amount of data added per block (approximately 1 MB base size, expandable to about 4 MB with Segregated Witness). These limits help control the overall blockchain size, ensuring predictable hardware requirements for storage on hard disk drives (HDDs). Similarly, the difficulty adjustment algorithm maintains a consistent block production rate of roughly one every 10 minutes by dynamically scaling the computational challenge for miners based on network hashrate.

However, the UTXO set lacks comparable built-in constraints. It accumulates as users create new outputs—often in small denominations—without a mandatory mechanism to consolidate or prune them. This can result from various activities, including high-frequency microtransactions, the embedding of non-monetary data (such as through protocols like Ordinals, which inscribe arbitrary information onto satoshis, Bitcoin's smallest unit), or the anchoring of sidechain or tokenized assets that leverage Bitcoin's security but operate externally. While these innovations expand Bitcoin's utility, they can inadvertently increase the UTXO count disproportionately to their economic value, elevating transaction fees during congestion and raising operational costs for nodes.

Existing incentives, such as transaction fees, partially mitigate this by encouraging users to consolidate low-value UTXOs to avoid higher costs. Yet, these market-driven forces are insufficient against sustained patterns of low-value output creation, particularly when driven by external systems that do not bear the full cost of network maintenance. Over time, this leads to bloat: as of recent estimates, the UTXO set exceeds several gigabytes when loaded into memory, complicating node synchronization and validation. Without intervention, projected growth could undermine decentralization, as fewer individuals or entities might afford to participate fully in the network.

The motivation for reform, therefore, stems from a desire to balance innovation with sustainability. An ideal solution would allow gradual UTXO expansion to accommodate genuine monetary usage while introducing feedback to curb excessive accumulation, all without compromising Bitcoin's permissionless nature or requiring a hard fork that could fragment the community.

## A PID-Inspired Feedback Controller for UTXO Control

Drawing from control theory, which studies how systems maintain desired behaviors through feedback, this proposal introduces a dynamic mechanism to regulate UTXO set size. PID controllers, widely used in engineering for processes like temperature regulation or autopilot systems, combine three terms: Proportional \(P\) for immediate response to errors, Integral (I) for correcting persistent deviations, and Derivative (D) for anticipating changes based on trends. In Bitcoin's context, we adapt this framework to compute a minimum value threshold—or "floor"—below which UTXOs become ineligible for spending, effectively deprecating them over time.

Importantly, this deprecation rule constitutes a soft fork: it restricts the set of valid transactions to a subset of those previously allowed, without introducing new capabilities. Upgraded nodes enforce the rule, rejecting deprecated UTXOs as invalid inputs, ensuring backward compatibility and minimizing disruption.

### Mechanism Overview

The controller activates at each difficulty adjustment epoch, approximately every two weeks (2016 blocks), aligning with Bitcoin's existing periodic recalibrations for predictability.

1. **Define a Target UTXO Size Trajectory**: Establish an increasing maximum target for the UTXO set size, denoted as $T(t)$, where $t$ represents the epoch number. This could grow sublinearly with time or blockchain height to reflect organic adoption— for instance, $T(t) = U_0 \cdot (1 + r)^t$, with $U_0$ as the initial size and $r$ a small annual growth rate (e.g., less than 1%). This permits expansion while preventing unbounded divergence.

2. **Measure the Error**: At each epoch, compute the error $e(t) = S(t) - T(t)$, where $S(t)$ is the current UTXO set size (measured by count or aggregate data footprint).

3. **Compute the Floor Adjustment**: Apply the PID formula to adjust the value floor $F(t)$:

   

    $$  \Delta F(t) = K_p \cdot e(t) + K_i \cdot \sum_{k=0}^{t} e(k) + K_d \cdot \frac{e(t) - e(t-1)}{\Delta t} $$

   

   Here, $K_p, K_i, K_d$ are tunable gains, selected conservatively through simulations to ensure stability (e.g., using root locus analysis to avoid oscillatory or unstable roots in the system's characteristic equation). The floor $F(t)$ increases only if $e(t) > 0$, applying to UTXOs below this threshold.

4. **Deprecation Process**: UTXOs with values below $F(t)$ are marked as unspendable in future transactions. To provide fairness and predictability, implement a grace period (e.g., 4-12 months) during which owners can consolidate affected UTXOs.

5. **Safeguards and Tuning**: Incorporate clamps on $\Delta F(t)$ (similar to Bitcoin's difficulty adjustment limits) to prevent abrupt changes that might cause network congestion from mass consolidations. Periodic reviews of gains and targets via community governance would adapt the system to evolving conditions.

### Stability and Dynamics Considerations

In dynamic systems terms, the UTXO set can be modeled as an integrator accumulating outputs minus expenditures. The PID controller introduces negative feedback to dampen this accumulation, driving $S(t)$ toward $T(t)$ with minimal overshoot. Simulations would verify robustness against adversarial scenarios, such as deliberate UTXO spam, ensuring the system's poles remain in the stable region of the complex plane.

Potential benefits include reduced node resource demands, lower fees during normal operation, and enhanced decentralization. Risks, such as over-deprecation affecting small holders, can be mitigated through careful gain selection.

## Conclusion

This PID-inspired approach offers a principled path to managing Bitcoin's UTXO set, fostering sustainable growth while upholding core principles. By integrating feedback control, Bitcoin can evolve as a resilient dynamic system, better equipped for widespread adoption. Further research, including detailed modeling and community discourse, is essential to refine and potentially implement such enhancements.

[Eric Voskuil]

> Given the increasing RAM requirements, due to the increasing UTXO set

I have been assured multiple times recently that this problem does not exist in Bitcoin Core. I'm not sure about that, but I can say for sure that this is not a problem in Libbitcoin and is therefore not problem with the Bitcoin protocol.

Best,
Eric

[Erik Aronesty]

Core: Keep UTXO lookups in RAM as much as possible. This gives predictable validation under adversarial load and trades memory for protection from pathological disk latency.
Libbitcoin: Represent history in append-only, memory-mapped files. Let the operating system decide what stays hot. Accept more disk I/O in return for simpler concurrency and no explicit UTXO map.

To create 200 million outputs as an attack, the cost is roughly 48 BTC based on eight vb per output at low feerates.

Given the operational problems that a very large UTXO set creates, it may be worthwhile to develop a clear storage abstraction layer.

The goal would be to ensure that validation correctness does not depend on the transitional behavior of a specific storage engine.

Once the validator has a well defined storage contract, alternate backends such as a libbitcoin style layout could exist as storage formats while still benefiting from the reliability of the rest of the Bitcoin implementation.

(I would be interested in something like this.)

--
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/049858f4-28ba-4ffb-8966-f8bef09035b7n%40googlegroups.com.

[Eric Voskuil]

Hi Erik,

It's a good idea, that's why we did it.

Libbitcoin is a set of libraries, libbitcoin-database being one of 4 that make up node (system, network, database, and node).  (libbitcoin-server is an additional library that adds comprehensive set of client-server interfaces, making the node useful.) libbitcoin-database is an implementation of a simple query interface (defined as a c++ class) over a backing store. The store is a templated collection of tables. The tables are mmap-based head and body files used to construct multimaps, hashmaps, arraymaps, arrays, and blobs. We mock the tables using std::vector<uint8_t>, mock the store using the the mocked tables, and test most of the query interface using the mock store. The structure is highly relational, and surrogate keys are exposed to the caller for optimized navigation.

This is an isolated and clear storage abstraction layer. All validation is performed within the chain:: classes (e.g. block, header, tx, input, script, etc.) defined in the base level libbitcoin-system. There is no validator coupling to the store. The store retains chain objects, indexes, and validation state for headers/blocks and txs as they progress through a state machine. There is no utxo table, just natural relations between objects, indexed and related. Validation correctness of course requires store fidelity, but is totally decoupled from it. We validate blocks concurrently, queueing up 50,000 blocks at a time by default (e.g. with 50k threads we would validate all at the same time).

The store could be replaced with no impact to the query interface (as we already do in testing). So it's not really accurate to imply that libbitcoin's validation is tied to mmap or even append-only. Pruning could be implemented in the existing model for example. The existing store could be replaced with something simple and light like SQLite, or a full RDBMS. We had some interns working the SQLite approach last summer. That would be more specialized for low performance scenarios, where the custom database targets ultra high performance.

With sufficient RAM there is never SSD access. The store can sync up and just live in RAM, never touching a disk. Since it is append only, it's very low impact on SSDs. As the store builds, no table body byte is ever re-written. Table heads are hashmaps buckets, small and dynamic. It performs live automated/manual snapshotting, automatic fault detection/recovery on restart, automated disk-full pause/restart, and supports hot backup. Query performance is phenomenal. A warm node can query the full 5.2 million output Electrum query (very complex relations) in 15 seconds on my 2.1GHz workstation. But at the low end, an off-the-shelf store is sufficient. A clear interface and swappable store makes a lot of sense.

A large utxo set makes no difference in this design. There are no operational problems associated with it. Given the recent fuss about it, I figured it was worth putting some detail behind it here. This is not theoretical, at this point we are only working on server (client-server interfaces). The utxo set size is a complete non-issue.

e

[uuowwpevskfcordh]

Thanks both Eric and Erik for the replies. I can't answer whether RAM requirements due to UTXO would indeed be a problem or not. I think a worse-case performance could be considered assuming a "no-RAM" environment.

This being said, I wanted to share some insights still related to the dynamic dust mechanism (the essay).

1. This proposal could be named "dynamic dust" plus "dust sweeping". Dynamic dust refers to the dust level being defined by the controller at every epoch, and dust sweeping refers to deprecation of utxos below that dust threshold.

2. The grace period for the sweep, individual for each transaction, allows for a new perspective. This is, again, better portrayed by Grok:

## Annex: Addressing Concerns of Confiscation in UTXO Deprecation

While the proposed PID-inspired mechanism for UTXO set management offers a pathway to enhanced scalability, it has elicited valid criticisms, particularly regarding its confiscatory nature. Deprecating low-value UTXOs could be perceived as an involuntary seizure of assets, undermining Bitcoin's principles of ownership and immutability. This concern merits careful consideration, as any protocol change must preserve user trust and avoid arbitrary interventions.

The proposal maintains a degree of neutrality by evaluating UTXOs solely based on their satoshi (SAT) value, without interpreting their content or purpose. This objective criterion minimizes subjective judgments, applying uniformly across all outputs regardless of their origin or use case.

A key mitigating factor lies in the incorporation of a grace period—typically spanning 6 to 12 months—during which owners of affected UTXOs can consolidate or spend them without direct penalty. This window transforms the deprecation process from outright confiscation into a dynamic incentive structure, favoring resource allocation in a constrained system. In Bitcoin's ecosystem, users already incur ongoing costs for participation, such as transaction fees, which can render low-value UTXOs economically immobile if fees rise persistently. Such "frozen" outputs are not labeled as confiscated but rather as subject to market realities.

From this perspective, low-value (or "dust") UTXOs enter a "renting" state upon falling below the dynamic dust floor. Owners, including those engaging in high-volume, low-value activities (e.g., data inscription or sidechain anchoring), can extend their usability by periodically transacting them—paying the requisite fees to "renew" the grace period. Failure to do so results in expiration and deprecation, mirroring the fee-based prioritization inherent in Bitcoin's design rather than imposing an immediate punitive forfeiture.

This framework may inadvertently benefit miners, as sustained renewal activities could increase block occupancy, elevating average fees and bolstering network security through heightened hashrate incentives. By channeling resources toward productive on-chain operations, the mechanism redirects potential inefficiencies into contributions that enhance overall system resilience.

In summary, while the confiscatory nature and risks are acknowledged, the grace period and value-based neutrality recast deprecation as an extension of Bitcoin's economic model, promoting efficient resource use without compromising core tenets.

Best regards,

> To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/97065ca0-662b-4a5d-90c5-7607db41ba2fn%40googlegroups.com.