Skip to content

Commit 90a7971

Browse files
ppekrolppekrol
authored
Merge pull request #2019 from reebhub/RDoc-3272_maxItems
Document expiration & refresh updates, SQS ETL page minor fixes
2 parents c2c8942 + 986522f commit 90a7971

File tree

34 files changed

+755
-136
lines changed

34 files changed

+755
-136
lines changed

‎Documentation/5.4/Raven.Documentation.Pages/server/extensions/expiration.dotnet.markdown

Lines changed: 63 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -3,73 +3,104 @@
33

44
{NOTE: }
55

6-
* The Expiration feature deletes expired documents, documents whose time has passed.
7-
Documents that are set with a future expiration time will be automatically deleted.
8-
9-
* The Expiration feature can be turned on and off while the database is already live with data.
6+
* Documents can be given a future expiration time in which they'll be automatically deleted.
7+
* The Expiration feature deletes documents set for expiration, when their time has passed.
8+
* You can enable or disable the expiration feature while the database is already live with data.
109

1110
* In this page:
1211
* [Expiration feature usages](../../server/extensions/expiration#expiration-feature-usages)
1312
* [Configuring the expiration feature](../../server/extensions/expiration#configuring-the-expiration-feature)
13+
* [Configure expiration settings using the client API](../../server/extensions/expiration#configure-expiration-settings-using-the-client-api)
1414
* [Setting the document expiration time](../../server/extensions/expiration#setting-the-document-expiration-time)
1515
* [Eventual consistency considerations](../../server/extensions/expiration#eventual-consistency-considerations)
16-
* [More details](../../server/extensions/expiration#more-details)
1716
{NOTE/}
1817

1918
---
2019

2120
{PANEL: Expiration feature usages}
2221

23-
* Use the Expiration feature when data is needed to be kept for only a temporary time.
22+
Use the Expiration feature when data is needed only for a given time period.
23+
E.g., for -
24+
25+
* Shopping cart data that is kept only for a certain time period
26+
* Email links that need to be expired after a few hours
27+
* A web application login session details
28+
* Cache data from an SQL server
2429

25-
* Examples:
26-
* Shopping cart data that is kept for only a specific time
27-
* Email links that need to be expired after a few hours
28-
* Storing a web application login session details
29-
* When using RavenDB to hold cache data from a SQL server.
3030
{PANEL/}
3131

3232
{PANEL: Configuring the expiration feature}
3333

34-
* By default, the expiration feature is turned off.
34+
Documents expiration settings can be changed via Studio or the API.
35+
It is possible to:
3536

36-
* The delete frequency is configurable, the default value is 60 secs.
37+
* Enable or Disable the deletion of expired documents.
38+
Default value: **Disable** the deletion of expired documents.
39+
* Determine how often RavenDB would look for expired documents and delete them.
40+
Default value: **60 seconds**
41+
* Set the maximal number of documents that RavenDB is allowed to delete per interval.
42+
Default value: **All expired documents**
3743

38-
* The Expiration feature can be turned on and off using **Studio**, see [Setting Document Expiration in Studio](../../studio/database/settings/document-expiration).
44+
---
3945

40-
* The Expiration feature can also be configured using the **Client**:
46+
{INFO: }
47+
[Learn how to configure expiration settings via Studio](../../studio/database/settings/document-expiration)
48+
{INFO/}
4149

42-
{CODE configuration@Server\Expiration\Expiration.cs /}
43-
{PANEL/}
50+
---
4451

45-
{PANEL: Setting the document expiration time}
52+
### Configure expiration settings using the client API
4653

47-
* To set the document expiration time just add the `@expires` property to the document `@metadata` and set it to contain the appropriate expiration time.
54+
Modify the expiration settings using the client API by setting an `ExpirationConfiguration`
55+
object and sending it to RavenDB using a `ConfigureExpirationOperation` operation.
4856

49-
* Once the Expiration feature is enabled, the document will automatically be deleted at the specified time.
57+
#### Example:
5058

51-
* **Note**: The date must be in **UTC** format, not local time.
59+
{CODE configuration@Server\Expiration\Expiration.cs /}
60+
61+
#### `ExpirationConfiguration`
62+
63+
{CODE expirationConfiguration@Server\Expiration\Expiration.cs /}
5264

53-
* The document expiration time can be set using the following code from the client:
65+
| Parameter | Type | Description |
66+
| - | - | - |
67+
| **Disabled** | `bool` | If `true`, deleting expired documents is disabled for the entire database.<BR>Default: `true` |
68+
| **DeleteFrequencyInSec** | `long?` | Determines how often (in seconds) the expiration feature looks for expired documents and deletes them.<BR>Default: `60` |
69+
| **MaxItemsToProcess** | `long?` | Determines the maximal number of documents the feature is allowed to delete in one run. |
5470

55-
{CODE expiration1@Server\Expiration\Expiration.cs /}
5671
{PANEL/}
5772

58-
{PANEL: Eventual consistency considerations}
73+
{PANEL: Setting the document expiration time}
5974

60-
* Once documents are expired, it can take up to the delete frequency interval (60 seconds by default) until these expired documents are actually deleted.
75+
* To set a document expiration time, add the document's `@metadata` an
76+
`@expires` property with the designated expiration time as a value.
77+
Set the time in **UTC** format, not local time. E.g. -
78+
**"@expires": "2025-04-22T08:00:00.0000000Z"**
79+
{WARNING: }
80+
Metadata properties starting with `@` are for internal RavenDB usage only.
81+
Do _not_ use the metadata `@expires` property for any other purpose than
82+
scheduling a document's expiration time for the built-in expiration feature.
83+
{WARNING/}
84+
* If and when the expiration feature is enabled, it will process all documents
85+
carrying the `@expires` flag and automatically delete each document
86+
[by its expiration time](../../server/extensions/expiration#eventual-consistency-considerations).
87+
* To set the document expiration time from the client, use the following code:
88+
{CODE expiration1@Server\Expiration\Expiration.cs /}
6189

62-
* Expired documents are _not_ filtered on load/query/indexing time, so be aware that an expired document might still be there after it has expired up to the 'delete frequency interval' timeframe.
6390
{PANEL/}
6491

65-
{PANEL: More details}
92+
{PANEL: Eventual consistency considerations}
6693

67-
* Internally, each document that has the `@expires` property in the metadata is tracked by the RavenDB server
68-
even if the expiration feature is turned off.
69-
This way, once the expiration feature is turned on we can easily delete all the expired documents.
94+
* Internally, RavenDB tracks all documents carrying the `@expires` flag even if the
95+
expiration feature is disabled. This way, once the expiration feature is enabled expired
96+
documents can be processed without delay.
97+
* Once a document expires, it may take up to the _delete frequency interval_ (60 seconds by default)
98+
until is it actually deleted.
99+
* Deletion may be further delayed if `MaxItemsToProcess` is set, limiting the number
100+
of documents that RavenDB is allowed to delete each time the expiration feature is invoked.
101+
* Expired documents are _not_ filtered out during `load`, `query`, or indexing, so be aware that
102+
as long as an expired document hasn't been actually deleted it may still be included in the results.
70103

71-
* **Note**: Metadata properties starting with `@` are internal for RavenDB usage.
72-
You should _not_ use the `@expires` property in the metadata for any other purpose other than the built-in expiration feature.
73104
{PANEL/}
74105

75106
## Related Articles

‎Documentation/5.4/Raven.Documentation.Pages/server/extensions/expiration.js.markdown

Lines changed: 38 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -3,75 +3,73 @@
33

44
{NOTE: }
55

6-
* The Expiration feature deletes expired documents, documents whose time has passed.
7-
Documents that are set with a future expiration time will be automatically deleted.
8-
9-
* The Expiration feature can be turned on and off while the database is already live with data.
6+
* Documents can be given a future expiration time in which they'll be automatically deleted.
7+
* The Expiration feature deletes documents set for expiration, when their time has passed.
8+
* You can enable or disable the expiration feature while the database is already live with data.
109

1110
* In this page:
1211
* [Expiration feature usages](../../server/extensions/expiration#expiration-feature-usages)
1312
* [Configuring the expiration feature](../../server/extensions/expiration#configuring-the-expiration-feature)
1413
* [Setting the document expiration time](../../server/extensions/expiration#setting-the-document-expiration-time)
1514
* [Eventual consistency considerations](../../server/extensions/expiration#eventual-consistency-considerations)
16-
* [More details](../../server/extensions/expiration#more-details)
1715
{NOTE/}
1816

1917
---
2018

2119
{PANEL: Expiration feature usages}
2220

23-
* Use the Expiration feature when data is needed to be kept for only a temporary time.
24-
25-
* Examples:
26-
* Shopping cart data that is kept for only a specific time
27-
* Email links that need to be expired after a few hours
28-
* Storing a web application login session details
29-
* When using RavenDB to hold cache data from a SQL server.
30-
{PANEL/}
21+
Use the Expiration feature when data is needed only for a given time period.
3122

32-
{PANEL: Configuring the expiration feature}
23+
Examples:
3324

34-
* By default, the expiration feature is turned off.
25+
* Shopping cart data that is kept only for a certain time period
26+
* Email links that need to be expired after a few hours
27+
* A web application login session details
28+
* Cache data from an SQL server
3529

36-
* The delete frequency is configurable, the default value is 60 secs.
30+
{PANEL/}
3731

38-
* The Expiration feature can be turned on and off using **Studio**, see [Setting Document Expiration in Studio](../../studio/database/settings/document-expiration).
32+
{PANEL: Configuring the expiration feature}
3933

34+
* By default, the expiration feature is turned **off**.
35+
* The deletion frequency is configurable.
36+
The default frequency is 60 seconds.
37+
* The Expiration feature can be turned **on** or **off** using **Studio**,
38+
see [Setting Document Expiration in Studio](../../studio/database/settings/document-expiration).
4039
* The Expiration feature can also be configured using the **Client**:
41-
42-
{CODE:nodejs expiration_1@Server\Expiration\expiration.js /}
40+
{CODE:nodejs expiration_1@Server\Expiration\expiration.js /}
4341

4442
{PANEL/}
4543

4644
{PANEL: Setting the document expiration time}
4745

48-
* To set the document expiration time just add the `@expires` property to the document `@metadata` and set it to contain the appropriate expiration time.
49-
50-
* Once the Expiration feature is enabled, the document will automatically be deleted at the specified time.
51-
52-
* **Note**: The date must be in **UTC** format, not local time.
53-
54-
* The document expiration time can be set using the following code from the client:
55-
56-
{CODE:nodejs expiration_2@Server\Expiration\expiration.js /}
46+
* To set document expiration time add an `@expires` property to the document
47+
`@metadata` and set the property's value to the designated expiration time.
48+
{NOTE: }
49+
The date must be in **UTC** format, not local time.
50+
{NOTE/}
51+
{WARNING: }
52+
Metadata properties starting with `@` are internal for RavenDB usage.
53+
Do _not_ use metadata `@expires` property for any other usage but that of the built-in expiration feature.
54+
{WARNING/}
55+
* Once the Expiration feature is enabled, the document will be automatically deleted at the specified time.
56+
{INFO: }
57+
Internally, each document that has the `@expires` property in the metadata is tracked
58+
by the RavenDB server even if the expiration feature is turned off.
59+
This way, once the expiration feature is turned on RavenDB can delete expired documents right away.
60+
{INFO/}
61+
* To set the document expiration time from the client, use the following code:
62+
{CODE:nodejs expiration_2@Server\Expiration\expiration.js /}
5763

5864
{PANEL/}
5965

6066
{PANEL: Eventual consistency considerations}
6167

62-
* Once documents are expired, it can take up to the delete frequency interval (60 seconds by default) until these expired documents are actually deleted.
63-
64-
* Expired documents are _not_ filtered on load/query/indexing time, so be aware that an expired document might still be there after it has expired up to the 'delete frequency interval' timeframe.
65-
{PANEL/}
66-
67-
{PANEL: More details}
68-
69-
* Internally, each document that has the `@expires` property in the metadata is tracked by the RavenDB server
70-
even if the expiration feature is turned off.
71-
This way, once the expiration feature is turned on we can easily delete all the expired documents.
68+
* Once documents expire, it may take up to the _delete frequency interval_ (60 seconds by default)
69+
until they are actually deleted.
70+
* Expired documents are _not_ filtered out during `load`, `query`, or indexing, so be aware that an expired
71+
document may still be included in the results after its expiration, for the period set by _delete frequency interval_.
7272

73-
* **Note**: Metadata properties starting with `@` are internal for RavenDB usage.
74-
You should _not_ use the `@expires` property in the metadata for any other purpose other than the built-in expiration feature.
7573
{PANEL/}
7674

7775
## Related Articles

‎Documentation/5.4/Raven.Documentation.Pages/server/extensions/images/StudioRefresh.png

34.5 KB
Loading

‎Documentation/5.4/Raven.Documentation.Pages/server/extensions/refresh.dotnet.markdown

Lines changed: 61 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -10,76 +10,93 @@
1010

1111
* In this page:
1212
* [Overview](../../server/extensions/refresh#overview)
13-
* [Syntax](../../server/extensions/refresh#syntax)
14-
* [Examples](../../server/extensions/refresh#examples)
13+
* [Examples](../../server/extensions/refresh#examples)
14+
* [Syntax](../../server/extensions/refresh#syntax)
15+
* [Configure from Studio](../../server/extensions/refresh#configure-from-studio)
16+
1517

1618
{NOTE/}
1719

1820
---
1921

2022
{PANEL: Overview}
2123

22-
To refresh a document, add a `@refresh` flag to a document's metadata with a value of
23-
type `DateTime` (this must be in UTC format, i.e. `DateTime.UtcNow`). This specifies
24-
when the document should be refreshed.
25-
26-
This will cause the document to refresh **_only once_**! The refresh operation removes
27-
the `@refresh` flag.
24+
* To set a document refresh time, add the document's `@metadata` a
25+
`@refresh` property with the designated refresh time as a value.
26+
Set the time in **UTC** format, not local time, e.g. -
27+
**"@refresh": "2025-04-22T08:00:00.0000000Z"**
28+
{WARNING: }
29+
Metadata properties starting with `@` are for internal RavenDB usage only.
30+
Do _not_ use the metadata `@refresh` property for any other purpose than
31+
scheduling a document's refresh time for the built-in refresh feature.
32+
{WARNING/}
33+
34+
* This will cause the document to refresh **only once**.
35+
When the refresh operation takes place, it will also remove the `@refresh` property from the document.
36+
{INFO: The exact refresh time is determined by:}
37+
38+
1. The refresh time value set for the `@refresh` property.
39+
2. The way you set the [Refresh Configuration](../../server/extensions/refresh#syntax),
40+
including -
41+
- The interval by which the server refreshes documents (set by default to 60 seconds).
42+
- The way you set **maximal items to process**, potentially limiting the number
43+
of documents that RavenDB is allowed to delete each time the refresh feature is invoked.
44+
45+
{INFO/}
46+
47+
* Refreshing a document causes its [change vector](../../server/clustering/replication/change-vector)
48+
to increment the same way it would after any other kind of update to the document.
49+
This triggers any features that react to document updating, including but not limited to:
50+
- Re-indexing of the document by indexes that cover it
51+
- [Replication](../../server/ongoing-tasks/external-replication),
52+
[Subscriptions](../../client-api/data-subscriptions/what-are-data-subscriptions),
53+
and [ETL](../../server/ongoing-tasks/etl/basics) triggering
54+
- Creation of a [document revision](../../document-extensions/revisions/overview)
2855

29-
The exact time that the document refreshes is not determined by the value of `@refresh`
30-
- rather, the server refreshes documents at regular intervals determined by the Refresh
31-
Configuration. The default interval is 60 seconds.
56+
{PANEL/}
3257

33-
Refreshing a document causes its [change vector](../../server/clustering/replication/change-vector)
34-
to increment the same way it would after any other kind of update to the document.
35-
This triggers any features that react to documents updating, including but not limited
36-
to:
58+
{PANEL: Examples}
3759

38-
* The document is re-indexed by any indexes that cover it.
39-
* [Replication](../../server/ongoing-tasks/external-replication),
40-
[Subscriptions](../../client-api/data-subscriptions/what-are-data-subscriptions),
41-
and [ETL](../../server/ongoing-tasks/etl/basics) are triggered.
42-
* A [revision](../../document-extensions/revisions/overview) of the document is created.
60+
#### Example I
4361

44-
{PANEL/}
62+
How to set refresh configuration for a database:
4563

46-
{PANEL: Syntax}
64+
{CODE refresh_0@Server/Extensions/DocumentRefresh.cs /}
4765

48-
To activate and/or configure document refreshing, the server needs to be sent a
49-
`RefreshConfiguration` object using a `ConfigureRefreshOperation` operation.
66+
This activates document refreshing and sets the interval at 5 minutes.
5067
<br/>
51-
#### RefreshConfiguration Object
68+
#### Example II
5269

53-
{CODE refresh_config@Server/Extensions/DocumentRefresh.cs /}
70+
How to set a document to refresh 1 hour from now:
5471

55-
| Parameter | Type | Description |
56-
| - | - | - |
57-
| **Disabled** | `bool` | If set to true, document refreshing is disabled for the entire database. Default: `true` |
58-
| **RefreshFrequencyInSec** | `long` | Determines how often the server checks for documents that need to be refreshed. Default: `60` |
59-
<br/>
60-
#### Studio
72+
{CODE refresh_1@Server/Extensions/DocumentRefresh.cs /}
6173

62-
Alternatively, document refreshing can also be configured in the studio, under **Settings > Document Refresh**.
74+
{PANEL/}
6375

64-
![NoSQL DB Server - Document Refresh](images/StudioRefresh.png "Document Refresh Settings")
76+
{PANEL: Syntax}
6577

66-
{PANEL/}
78+
To activate and/or configure document refreshing, send the server a
79+
`RefreshConfiguration` object using the `ConfigureRefreshOperation` operation.
6780

68-
{PANEL: Examples}
81+
---
6982

70-
#### Example I
83+
#### `RefreshConfiguration`
7184

72-
How to set refresh configuration for a database:
85+
{CODE refresh_config@Server/Extensions/DocumentRefresh.cs /}
7386

74-
{CODE refresh_0@Server/Extensions/DocumentRefresh.cs /}
87+
| Parameter | Type | Description |
88+
| - | - | - |
89+
| **Disabled** | `bool` | If `true`, refreshing documents is disabled for the entire database.<BR>Default: `true` |
90+
| **RefreshFrequencyInSec** | `long?` | Determines how often (in seconds) the server processes documents that need to be refreshed.<BR>Default: `60` |
91+
| **MaxItemsToProcess** | `long?` | Determines the maximal number of documents the feature is allowed to refresh in one run. |
7592

76-
This activates document refreshing and sets the interval at 5 minutes.
77-
<br/>
78-
#### Example II
93+
{PANEL/}
7994

80-
How to set a document to refresh 1 hour from now:
95+
{PANEL: Configure from Studio}
8196

82-
{CODE refresh_1@Server/Extensions/DocumentRefresh.cs /}
97+
Alternatively, document refreshing can also be configured via Studio, under **Settings > Document Refresh**.
98+
99+
![NoSQL DB Server - Document Refresh](images/StudioRefresh.png "Document Refresh Settings")
83100

84101
{PANEL/}
85102

0 commit comments

Comments
 (0)