You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/connections/reverse-etl/system.md
+14-11Lines changed: 14 additions & 11 deletions
Original file line number
Diff line number
Diff line change
@@ -16,31 +16,34 @@ For Segment to compute the data changes within your warehouse, Segment needs to
16
16
> warning ""
17
17
> There may be cost implications to having Segment query your warehouse tables.
18
18
19
-
## Reverse ETL Schema
19
+
## Reverse ETL schema
20
20
When using Reverse ETL with Segment, several system tables are created within the `__segment_reverse_etl` schema in your warehouse. These tables are crucial for managing the sync process efficiently and tracking state information. Below are the details of the system tables in this schema:
21
21
22
-
**1. Records Table**
22
+
###Records table
23
23
24
-
`records_<subscription_id>` table is located within the` __segment_reverse_etl` schema, this table contains two key columns:
24
+
`records_<subscription_id>` table is located within the` __segment_reverse_etl` schema.
25
25
26
-
`record_id`: A unique identifier for each record.
26
+
This table contains two key columns:
27
27
28
-
`checksum`: A checksum value that is used to detect changes to a record since the last sync.
28
+
-`record_id`: A unique identifier for each record.
29
+
-`checksum`: A checksum value that is used to detect changes to a record since the last sync.
29
30
The records table helps in determining new and updated rows by comparing the checksum values during each sync. If a record’s checksum changes, it indicates that the record has been modified and should be included in the next sync. This ensures that only the necessary updates are processed, reducing the amount of data transferred.
30
31
31
-
**2. Checkpoint Table**
32
+
###Checkpoint table
32
33
33
-
The `checkpoints_<subscription_id>` tables are located within the __segment_reverse_etl schema, this table contains the following columns:
34
+
The `checkpoints_<subscription_id>` tables are located within the __segment_reverse_etl schema.
34
35
35
-
`source_id`: Identifies the source from which the data is being synced.
36
+
This table contains the following columns:
37
+
38
+
-`source_id`: Identifies the source from which the data is being synced.
39
+
-`model_id`: Identifies the specific model or query that is used to pull data.
40
+
-`checkpoint`: Stores a timestamp value that represents the last sync point for a particular model.
36
41
37
-
`model_id`: Identifies the specific model or query that is used to pull data.
38
-
checkpoint: Stores a timestamp value that represents the last sync point for a particular model.
39
42
The checkpoints table is used for timestamp-based checkpointing between syncs. This enables Segment to track the last successful sync for each model and avoid duplicating data when syncing, ensuring incremental and efficient data updates.
40
43
41
44
### Important Considerations
42
45
43
-
Do not modify or delete these tables: Altering or deleting the records and checkpoints tables can cause unpredictable behavior in the sync process. These tables are essential for maintaining the integrity of data during Reverse ETL operations.
46
+
Do not modify or delete these tables. Altering or deleting the records and checkpoints tables can cause unpredictable behavior in the sync process. These tables are essential for maintaining the integrity of data during Reverse ETL operations.
44
47
State management: The `__segment_reverse_etl` schema and its associated tables (records and checkpoints) manage the state of each sync, ensuring that only necessary data changes are synced and that the sync process can resume where it left off.
0 commit comments