Skip to content

Commit 89793d2

Browse files
authored
docs: misc docusaurus improvements + update for v1.2.9 (#478)
* chore: update OpenAPI specification and documentation visual improvements - Updated API title to reflect rebranding to Mesh API. - Enhanced some endpoint descriptions for clarity and accuracy (network status, options, block transaction). - Added detailed descriptions for Cardano-specific transaction and block metadata. - Commented out unimplemented mempool endpoints and related documentation. - Improved Docusaurus configuration for better user experience, including last update timestamps . - Updated CSS for improved layout and styling consistency across documentation pages. * chore: update OpenAPI specification for java implementation - Updated example values for block, account and transaction identifiers to align with current data. - Enhanced operation types and statuses to include Cardano-specific options. - Added valid response examples to improve clarity. - Commented out unimplemented features and provided TODO notes for future implementation. * chore: update Docusaurus configuration and CSS for improved documentation layout - Updated custom CSS to enhance layout consistency, including centering the navbar and main wrapper. - Removed the proxy URL from the configuration. - Adjusted styles for the theme edit button and sidebar to improve user experience. - Deleted the outdated API specification file to streamline documentation. * chore: update dependencies in package.json and yarn.lock * chore: restore SearchTransactionsResponse schema to specs * fix: remove schema 'examples' breaking CI validation Remove multiple 'examples' entries from schema definitions ('Operation', 'NetworkIdentifier'). These were intended solely for enhanced rendering in Scalar docs but caused spec-validation failures during CI builds. Single 'example' per schema retained to ensure OpenAPI compliance and fix build errors. * chore: update OpenAPI generator plugin version and refine example values in API specification - Upgraded the OpenAPI generator Maven plugin from version 6.4.0 to 7.13.0. - Simplified example values in the API specification by removing unnecessary array structures for 'network' and 'status' fields. * chore: drop API spec and pom edits from PR 456 * chore: update Docker compose command and version in documentation * refactor: update documentation for spent UTXO pruning - Renamed "Pruning UTXOs" to "Spent UTXO Pruning" for clarity. - Enhanced explanations regarding the impact and configuration of spent UTXO pruning. - Updated resource requirements in README and intro documentation to specify spent UTXO pruning. * docs: enhance documentation for queryng stake addresses and rename multi-assets section - Added a note on stake addresses to clarify the response from the `/account/balance` endpoint when queried with a stake address (available rewards). - Renamed the "Multi-Assets" section to "Native Tokens" for better alignment with terminology used in Cardano documentation. * docs: update performance measurement documentation and environment variables - Enhanced the performance measurements section with load test results for versions 1.2.6, 1.2.7, and 1.2.9. - Updated environment variables documentation to reflect changes in version 1.2.9, including the introduction of `REMOVE_SPENT_UTXOS` and `REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT`, while deprecating older variables. * docs: update storage requirements and pruning notes in documentation - Revised storage requirements in README and intro documentation to reflect updated values for both pruning enabled and disabled scenarios. - Added clarifications regarding the limitations of searching transactions by address when pruning is enabled in the pruning and cardano-addons documentation.
1 parent 90dbc3f commit 89793d2

File tree

27 files changed

+785
-3530
lines changed

27 files changed

+785
-3530
lines changed

README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ Since [Yaci-Store](https://github.com/bloxbean/yaci-store) is a comparatively li
2929

3030
- 4CPU Cores
3131
- 32GB RAM
32-
- ca. > 1TB of storage (REMOVE_SPENT_UTXOS=false) [default]
33-
- ca > 400GB of storage (REMOVE_SPENT_UTXOS=true)
32+
- ~1.3 TB total storage (node ~250 GB + Rosetta DB ~1 TB) — pruning disabled [default]
33+
- ~750 GB total storage (node ~250 GB + Rosetta DB ~500 GB) — pruning enabled
3434

3535
Better hardware will improve the performance of the indexer and the node, which will result in faster syncing times.
3636

@@ -124,7 +124,7 @@ This will start:
124124
### Entry level hardware profile
125125

126126
```bash
127-
docker-compose --env-file .env.docker-compose --env-file .env.docker-compose-profile-mid-level -f docker-compose.yaml up -d
127+
docker compose --env-file .env.docker-compose --env-file .env.docker-compose-profile-mid-level -f docker-compose.yaml up -d
128128
```
129129

130130
### A complete list of hardware profiles:
Lines changed: 107 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,134 @@
11
---
22
sidebar_position: 1
3-
title: Remove Spent UTXOs
4-
description: Optimizing disk usage with removal of spent UTxOs
3+
title: Spent UTXO Pruning
4+
description: Optimizing disk usage with spent UTXO pruning
55
---
66

7-
# Remove Spent UTXOs
7+
# Spent UTXO Pruning
88

9-
This guide explains how to optimize disk usage in **cardano-rosetta-java** through removal of spent utxo (a special rosetta-java specific form of pruning).
9+
This guide explains how to optimize disk usage in **cardano-rosetta-java** through spent UTXO pruning, including its impact on Rosetta API endpoints and configuration options.
1010

11-
## What is UTxO Spent Pruning (removal of spent UTxOs)?
11+
## Understanding Spent UTXO Pruning
1212

13-
UTxO Spent Pruning removes spent (consumed) UTXOs from local storage, keeping only unspent UTXOs. This can reduce on-disk storage from ~1TB down to ~400GB, but discards historical transaction data.
13+
Spent UTXO pruning is a disk optimization mechanism in `cardano-rosetta-java`, powered by its underlying indexer, Yaci-Store. This feature selectively removes data related to spent UTXOs from the local database.
1414

15-
- Only unspent outputs are preserved.
16-
- You can still validate the chain's current state (and spend tokens), since active UTXOs remain.
17-
- You should be able to build transaction because only spent UTxOs are removed (unspent stay)
15+
**Core Principles:**
16+
17+
- **Targeted Deletion**: Only _spent_ UTXOs are removed. All _current, unspent_ UTXOs are preserved, ensuring the accuracy of the present blockchain state and balances.
18+
- **Distinction from Other Pruning**: This mechanism differs from what is commonly understood as 'pruning' in some other blockchain contexts, including certain descriptions in the Coinbase Mesh API (formerly Rosetta). Unlike methods such as Bitcoin's pruning (which removes entire historical blocks), our approach retains full block history but selectively trims the UTXO set by removing only spent outputs.
19+
20+
**How it Works:**
21+
When enabled, the pruning process operates as follows:
22+
23+
1. New UTXOs are indexed as transactions occur.
24+
2. UTXOs are marked as spent when consumed in subsequent transactions.
25+
3. A background job periodically permanently deletes spent UTXOs that are older than a configurable safety margin (default: 2,160 blocks, ~12 hours on mainnet). This buffer safeguards data integrity against chain rollbacks within Cardano's finality window.
26+
27+
**Impact Summary:**
28+
| Aspect | Effect |
29+
| :------------------------- | :----------------------------------------------------------- |
30+
| **Disk Storage** | ✅ Significantly reduced (e.g., mainnet from ~1TB to ~500GB) |
31+
| **Current UTXO Set** | ✅ Fully preserved; current balances remain accurate |
32+
| **Historical Spent UTXOs** | ⚠️ Permanently deleted beyond the safety margin |
33+
| **Query Performance** | ✅ Improved for queries against the current UTXO set |
34+
35+
## Impact on Rosetta API Endpoints
36+
37+
Spent UTXO pruning affects Rosetta API endpoints differently based on their reliance on historical transaction data. The table below summarizes the impact. Note that "Recent" refers to data within the safety margin (default ~12 hours).
38+
39+
:::info Oldest Block Identifier
40+
When pruning is enabled, the `/network/status` endpoint includes an additional `oldest_block_identifier` object in its response. This identifier corresponds to the latest fully queryable block with complete data. Below this block index, blocks might have missing data due to pruning, making historical queries unreliable.
41+
:::
42+
43+
| **Endpoint** | **Current State** | **Historical Queries** | **Impact & Notes** |
44+
| ---------------------- | ----------------- | ---------------------- | ---------------------------------------------------------------------------- |
45+
| `/account/balance` | ✅ Works | ⚠️ Limited | **Low** - Current balances unaffected |
46+
| `/account/coins` | ✅ Works | ⚠️ Limited | **Low** - Current UTXO lists complete |
47+
| `/block` | ✅ Recent only | ❌ Incomplete | **High** - Missing old transaction inputs |
48+
| `/block/transaction` | ✅ Recent only | ❌ Incomplete | **High** - Missing spent UTXOs operation details |
49+
| `/search/transactions` | ⚠️ Recent only | ❌ Limited | **Medium** - Hash search works, address limited |
50+
| `/network/status` | ✅ Works | ✅ Works | **None** - Returns additional `oldest_block_identifier` when pruning enabled |
51+
| `/network/*` | ✅ Works | ✅ Works | **None** - Independent of UTXO data |
52+
| `/construction/*` | ✅ Works | ✅ Works | **None** - Uses current UTXOs only |
53+
54+
After enabling pruning, searching for transactions by their hash will always work, because transaction records themselves are never pruned. However, searching by address is limited: address-based searches rely on the UTXO set, and once spent UTXOs older than the pruning window are deleted, only transactions involving current or recently spent UTXOs can be found by address. Older history is not returned once pruned.
1855

1956
**Enable Spent UTxO removal **: Set `REMOVE_SPENT_UTXOS=true` in your environment (e.g., in `.env.dockerfile` or `.env.docker-compose`).
2057
**Disable Spent UTxO removal ** (default): Set `REMOVE_SPENT_UTXOS=false`.
2158

2259
## When Spent UTxO Removal should be enabled?
2360

24-
- **Low Disk Environments**: If you need to minimize disk usage and only require UTXO data for current balances.
25-
- **Exploratory / Dev Environments**: If historical queries are not critical.
26-
- **Performance**: if you are running into performance / scalability issues, i.e. especially on /account/balance when working with large addresses
61+
:::tip Recommended Use Cases
62+
Pruning is beneficial in scenarios where optimizing disk space and focusing on current data is prioritized over maintaining a complete historical record. Consider enabling pruning if your use case aligns with the following:
63+
64+
- **Exchange Integrations & Wallet Services**: Primarily for tracking current balances, processing recent deposits/withdrawals, and validating recent transactions.
65+
- **Resource-Constrained Environments**: Ideal when disk space is a significant limitation (e.g., under 1TB available for mainnet data).
66+
- **Tip-of-Chain Operations**: For applications focused on the latest blockchain state rather than deep historical analysis.
67+
- **Development and Testing**: Useful when a full historical dataset is not essential for development or testing purposes.
68+
:::
2769

2870
## When to avoid setting UtxO Removal feature?
2971

30-
- **Full Historical Data Requirements**: If you need the complete transaction history—whether for exchange operations, audit trails, or compliance mandates—do not enable pruning. Pruning discards spent UTXOs, which removes older transaction data and prevents certain types of historical lookups or reporting.
72+
:::warning Not Suitable For
73+
Avoid pruning if your operational or regulatory requirements necessitate access to complete and auditable historical blockchain data. Pruning is generally not suitable if you need:
74+
75+
- **Complete Historical Data & Deep Queries**: For comprehensive auditing, compliance, data analytics, or block explorer-like functionality that requires querying full transaction history from any point in time.
76+
- **Strict Compliance and Audit Trails**: If regulatory mandates demand immutable, complete historical records. Pruned data cannot be recovered without a full resync, and historical queries for `/block` and `/block/transaction` become unreliable beyond the safety window.
77+
:::
78+
79+
:::danger Data Loss Warning
80+
Once data is pruned, it cannot be recovered without a full blockchain resynchronization. Assess your historical data needs carefully before enabling pruning.
81+
:::
3182

32-
## Example Configuration
83+
## Configuration
3384

34-
Below is a snippet of how you might configure `.env.dockerfile` or `.env.docker-compose` for pruning:
85+
Spent UTXO pruning is configured via environment variables, typically set in your `.env.dockerfile` or `.env.docker-compose` file. Below is an example demonstrating the available settings and their default values:
3586

3687
```bash
37-
# --- Remove Spent UTxOs Toggle ---
88+
# --- Spent UTXO Pruning Configuration ---
89+
90+
# Enable or disable spent UTXO pruning.
91+
# Default: false (Pruning is disabled by default)
92+
# To enable, set to: true
3893
REMOVE_SPENT_UTXOS=true
94+
95+
# Safety margin: Number of recent blocks for which spent UTXOs are retained.
96+
# Default: 2160 (approximately 12 hours of blocks on mainnet)
97+
# This value balances safety for rollbacks against storage savings.
98+
# Example: To keep ~24 hours of spent UTXOs, set to 4320.
99+
# Note: Larger REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT values provide longer historical query support
100+
# but use more disk space and delay the realization of storage benefits.
101+
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=2160
39102
```
40103

104+
:::note Configuration Guidelines
105+
106+
- Start with the default settings (`REMOVE_SPENT_UTXOS=false` means pruning is initially off).
107+
- If enabling, the provided defaults (`REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=2160`) are sensible starting points.
108+
- Adjust `REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT` based on your specific needs for recent historical data access. A larger value offers a longer window for historical queries but increases storage.
109+
:::
110+
111+
## Migration and Operational Notes
112+
113+
This section outlines key considerations when changing pruning settings or managing a system with pruning enabled.
114+
115+
### Enabling Pruning on an Existing Deployment
116+
117+
1. **Configuration**: Update the necessary environment variables (e.g., `REMOVE_SPENT_UTXOS=true`) and restart your cardano-rosetta-java services.
118+
2. **Initial Monitoring**: After enabling, closely monitor your application for any errors. Disk usage benefits will typically appear after the first pruning cycle completes.
119+
120+
### Disabling Pruning
121+
122+
:::caution Important Considerations
123+
124+
- **Halts Deletion, No Restoration**: Setting `REMOVE_SPENT_UTXOS=false` and restarting services will stop further deletion of spent UTXOs. However, this action **does not restore** any data already pruned.
125+
- **Full Resync for Complete History**: To regain a complete historical dataset after pruning has been active, a full resynchronization of the blockchain data is required.
126+
- **Storage Growth Resumes**: Once pruning is disabled, expect storage usage to gradually increase again as new UTXOs (both spent and unspent) accumulate.
127+
:::
128+
41129
## Further Reading
42130

43-
- [Rosetta API Reference](https://docs.cdp.coinbase.com/mesh/docs/api-reference/)
131+
- [Environment Variables Reference](../install-and-deploy/env-vars.md)
44132
- [Yaci-Store Repository](https://github.com/bloxbean/yaci-store)
133+
- [Coinbase Mesh (formerly Rosetta) API Specification](https://docs.cdp.coinbase.com/mesh/docs/api-reference/)
134+
- [Cardano UTXO Model Documentation](https://docs.cardano.org/learn/eutxo-explainer/)

docs/docs/core-concepts/cardano-addons.md

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,10 @@ Max amount of transactions allowed to be requested is defined by `PAGE_SIZE` env
7373
`status` and `maxBlock` filters work as excluding filters, if they are set, besides operator value.
7474
:::
7575

76+
:::warning Pruning limitation
77+
Searching for transactions by hash is always possible, because transaction records themselves are never pruned. However, searching by address is limited: address-based searches depend on the UTXO set, and once spent UTXOs older than the pruning window are deleted, only transactions involving current or recently spent UTXOs can be found by address. Older history is not returned once pruned.
78+
:::
79+
7680
### `/block`
7781

7882
The following metadata is also returned, when querying for block information:
@@ -163,7 +167,23 @@ For accounts that have a multi asset balance, these will be returned with the co
163167
}
164168
```
165169

166-
Also, `coins` will be returned with the token bundle list corresponding to each coin as metadata.
170+
When querying `/account/balance`, the result depends on the type of address:
171+
172+
- For a **payment address** (Base or Enterprise), the response contains the sum of all unspent UTXOs for that address (the "spendable" balance).
173+
- For a **stake address** (reward address), the response contains only the available rewards that can be withdrawn from that stake address.
174+
175+
To present a wallet's total value, clients should query both the payment address (for spendable funds) and the stake address (for rewards), then combine the results.
176+
177+
:::info Note on Stake Addresses
178+
A stake address never holds UTXOs and cannot have a spendable balance. The `/account/balance` response for a stake address always reflects only the withdrawable rewards.
179+
:::
180+
181+
Internally, the Rosetta service gathers these two figures from different subsystems:
182+
183+
- **Spendable funds** (ADA and native tokens) are obtained by summing all unspent UTXOs that belong to the payment address(es). These UTXOs are stored in the PostgreSQL database maintained by the **Yaci Indexer**.
184+
- **Rewards balance** is fetched directly from the **Yaci Store** through a lightweight internal HTTP call, which queries the current withdrawable rewards for the stake address.
185+
186+
As a consequence, the spendable portion updates whenever new blocks modify the UTXO set for the payment address, whereas the rewards portion changes only after the protocol distributes staking rewards (typically once per epoch).
167187

168188
### `/account/coins`
169189

docs/docs/development/performance-measurements.md

Lines changed: 75 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -4,36 +4,98 @@ title: Performance Measurements
44
description: Performance measurement methodologies and results
55
---
66

7+
import Tabs from '@theme/Tabs';
8+
import TabItem from '@theme/TabItem';
9+
710
# Performance Measurements
811

9-
This page documents the performance measurements and load test results for different releases of `cardano-rosetta-java`.
12+
This page documents load test results for different releases of `cardano-rosetta-java`, evaluating performance across various deployment scenarios.
13+
14+
### Service Level Agreements (SLAs)
15+
16+
Our performance evaluation is based on the following Service Level Agreements (SLAs), which define the acceptable thresholds for production environments:
17+
18+
- **Response Time**: A **p99 of less than 1 second** is the target for all standard API endpoints.
19+
- **Error Rate**: A **non-2xx response rate of less than 1%** is considered acceptable.
1020

11-
## Load Test Results
21+
In specific high-load scenarios, such as querying addresses with hundreds of thousands of transactions, these SLAs may be adjusted to assess system stability under extreme conditions. Any such adjustments are noted in the relevant test results.
1222

13-
Below you can find the load test results for various releases, comparing different deployment options.
23+
:::note
24+
Load tests are conducted using Apache Bench (ab) with a ramp-up strategy, progressively increasing concurrency to a predefined ceiling of 500 simultaneous requests.
25+
:::
26+
27+
:::tip
28+
To better understand the environments in which these results were obtained, please refer to our [hardware profiles documentation](../install-and-deploy/hardware-profiles).
29+
:::
1430

1531
<details>
1632
<summary>
1733

18-
### Release 1.2.7 (Test Run: 2025-04-16)
34+
### v1.2.9 (Jun 11, 2025)
1935

2036
</summary>
2137

22-
import Tabs from '@theme/Tabs';
23-
import TabItem from '@theme/TabItem';
24-
import SingleDocker from './\_test-results/1.2.6-dev_2025-04-16/single-docker.md';
25-
import DockerCompose from './\_test-results/1.2.6-dev_2025-04-16/docker-compose.md';
38+
- [Release Notes](https://github.com/cardano-foundation/cardano-rosetta-java/releases/tag/1.2.9)
39+
40+
The following tests were conducted on a **mid-level** hardware profile with the following specifications: **8 cores, 8 threads, 47GB RAM, 3.9TB NVMe, QEMU Virtual CPU v2.5+**.
41+
42+
import DockerCompose129 from './test-results/1.2.9/docker-compose.md';
43+
import HugeAddress373k from './test-results/1.2.9/huge-address-373k.md';
44+
import HugeAddress16M from './test-results/1.2.9/huge-address-1.6M.md';
2645

2746
<Tabs>
28-
<TabItem value="single" label="Single Docker" default>
29-
<SingleDocker />
47+
<TabItem value="compose" label="Docker Compose" default>
48+
<DockerCompose129 />
49+
</TabItem>
50+
<TabItem value="huge-373k" label="Huge Address (~373k txs)">
51+
<HugeAddress373k />
52+
</TabItem>
53+
<TabItem value="huge-1.6M" label="Huge Address (~1.6M txs)">
54+
<HugeAddress16M />
3055
</TabItem>
56+
</Tabs>
57+
</details>
58+
59+
<details>
60+
<summary>
61+
62+
### v1.2.7 (Apr 29, 2025)
63+
64+
</summary>
65+
66+
import SingleDocker127 from './test-results/1.2.7/single-docker.md';
67+
import DockerCompose127 from './test-results/1.2.7/docker-compose.md';
68+
69+
- [Release Notes](https://github.com/cardano-foundation/cardano-rosetta-java/releases/tag/1.2.7)
70+
71+
<Tabs>
72+
<!-- <TabItem value="single" label="Single Docker" default>
73+
<SingleDocker127 />
74+
</TabItem> -->
3175
<TabItem value="compose" label="Docker Compose">
32-
<DockerCompose />
76+
<DockerCompose127 />
3377
</TabItem>
3478
</Tabs>
3579
</details>
3680

37-
---
81+
<details>
82+
<summary>
83+
84+
### v1.2.6 (Apr 15, 2025)
85+
86+
</summary>
87+
88+
import SingleDocker126 from './test-results/1.2.6/single-docker.md';
89+
import DockerCompose126 from './test-results/1.2.6/docker-compose.md';
90+
91+
- [Release Notes](https://github.com/cardano-foundation/cardano-rosetta-java/releases/tag/1.2.6)
3892

39-
_Future results will be added here, each as a collapsible section._
93+
<Tabs>
94+
<TabItem value="single" label="Single Docker" default>
95+
<SingleDocker126 />
96+
</TabItem>
97+
<TabItem value="compose" label="Docker Compose">
98+
<DockerCompose126 />
99+
</TabItem>
100+
</Tabs>
101+
</details>

docs/docs/development/qa-regression.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ description: Testing methodology and procedures for releases
88

99
The following steps are performed as part of the QA process for new releases:
1010

11-
1. **E2E Testing**: Run the complete end-to-end test flow on preprod network using the [`e2e-tests`](https://github.com/cardano-foundation/cardano-rosetta-java/tree/main/e2e-tests) suite
11+
1. **E2E Testing**: Run the complete end-to-end test flow on preprod network using the [`e2e_tests`](https://github.com/cardano-foundation/cardano-rosetta-java/tree/main/e2e_tests) suite
1212

1313
- Test on both single docker and docker-compose deployments
1414
- Verify all construction API flows work correctly

docs/docs/development/test-results/1.2.6-dev_2025-04-16/docker-compose.md

Lines changed: 0 additions & 21 deletions
This file was deleted.

0 commit comments

Comments
 (0)