fix: Non-random KSUIDs from hashes in folder, dashboard, and alert migrations (#5726)

### Summary

This PR updates the logic that is used to generate KSUID primary keys
for the `folders`, `dashboards`, and `alerts` tables in existing
migrations which transform data from the `meta` table into the
`folders`, `dashboards`, and `alerts` tables.

The new logic generates 160-bit "KSUIDs" which are actually 160-bit
SHA-1 hashes of the records being transformed rather than being truly
random KSUIDs.

### Motivation

In the existing implementation, the migrations generate random KSUIDs
for the records that are transformed from the `meta` table and into the
different `folders`, `dashboards`, and `alerts` tables. When running the
application in a single cluster this is acceptable behavior.

However when running the application in a multi-cluster environment this
causes problems. In a multi-cluster environment, each cluster will have
it's own copy of the application database, and each database should
contain the exact same data in the `meta` table. When we run the
migration scripts in each cluster, the data from the `meta` table will
be transformed into the `folders`, `dashboards`, and `alerts` tables.
However if the migration generates a random KSUID for each new record
inserted into those new tables, then different clusters will each
generate different KSUID `id`s for rows in the `folders`, `dashboards`,
and `alerts` tables which should actually have the same `id`s.

Our solution to this problem is that during the migration from the
`meta` table to the `folders`, `dashboards`, and `alerts` tables, rather
than generating random 160-bit KSUID `id`s for the new records, we
generate 160-bit SHA-1 hashes for each record and we use that SHA-1 hash
as the `id` of the each new record. This ensure that each record that is
transformed from the `meta` table will have an `id` in the `folders`,
`alerts`, or `dashboards` tables that is consistent across all clusters.

Newly generated records (ie, records generated AFTER the migration) in
the `folders`, `alerts`, and `dashboards` tables will continue to be
generated with random KSUID `id`s since these `id`s can be shared with
other clusters via NATS super-cluster events and then records with those
`id`s can be inserted into databases in other clusters, ensuring that
records have consistent `id`s across clusters.

### Notes about KSUID generation

To generate a KSUID this function generates the 160-bit SHA-1 hash of
the alert's `org`, `stream_type`, `stream_name`, and `name` and
interprets that 160-bit hash as a 160-bit KSUID. Therefore two KSUIDs
generated in this manner will always be equal if the alerts have the
same `org`, `stream_type`, `stream_name`, and `name`.

⚠️ It is important to note that although KSUIDs generated in this manner
can be parsed as KSUIDs since they are 160-bit values, the KSUIDs
generated in this manner will have timestamp bits which are effectively
random, meaning that the timestamp in any KSUID generated with this
function will be random. This will render the timestamp-sortability
property of these KSUIDs useless. This probably isn't a big deal since
we can always add and sort by a `created_at` column if we want to, but
we should be aware of this limitation.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added SHA-1 hashing dependency to generate unique identifiers (KSUIDs)
for alerts, folders, and dashboards.
	- Implemented KSUID generation based on specific entity attributes.

- **Refactor**
- Transitioned database tables from integer-based to KSUID-based primary
keys.
	- Updated `StreamType` enum to support more flexible usage.

- **Chores**
	- Updated project dependencies to include SHA-1 hashing library.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: patrickcsullivan

Cargo.lock

@@ -305,6 +305,7 @@ sea-orm-migration = { version = "1.1.0", features = [
 segment = "~0.2.4"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
+sha1 = "0.10.6"
 sha256 = "1.4.0"
 snafu = "0.7.5"
 snap = "1"

Cargo.toml

@@ -25,6 +25,7 @@ once_cell.workspace = true
 parking_lot.workspace = true
 serde.workspace = true
 serde_json.workspace = true
+sha1.workspace = true
 sqlx.workspace = true
 svix-ksuid.workspace = true
 thiserror.workspace = true

src/infra/Cargo.toml

@@ -675,7 +675,7 @@ mod meta_table_alerts {
         DerivedStream,
     }
 
-    #[derive(Default, Deserialize, Serialize)]
+    #[derive(Default, Clone, Copy, Deserialize, Serialize)]
     #[serde(rename_all = "lowercase")]
     pub enum StreamType {
         #[default]
@@ -706,7 +706,7 @@ impl TryFrom<MetaAlertWithFolder> for alerts_table::ActiveModel {
             .ok_or("Alert in meta table references folder that does not exist")?;
         let meta_alert: meta_table_alerts::Alert = serde_json::from_str(m.alert_json())
             .map_err(|_| "Alert in meta table could not be deserialized")?;
-        let id = svix_ksuid::Ksuid::new(None, None).to_string();
+        let id = ksuid_from_hash(&meta_alert).to_string();
 
         // Transform the parsed stream type from the meta table and serialize
         // into string for storage in DB.
@@ -847,6 +847,33 @@ impl TryFrom<MetaAlertWithFolder> for alerts_table::ActiveModel {
     }
 }
 
+/// Generates a KSUID from a hash of the alert's `org`, `stream_type`,
+/// `stream_name`, and `name`.
+///
+/// To generate a KSUID this function generates the 160-bit SHA-1 hash of
+/// the alert's `org`, `stream_type`, `stream_name`, and `name` and interprets
+/// that 160-bit hash as a 160-bit KSUID. Therefore two KSUIDs generated in this
+///  manner will always be equal if the alerts have the same `org`,
+/// `stream_type`, `stream_name`, and `name`.
+///
+/// It is important to note that KSUIDs generated in this manner will have
+/// timestamp bits which are effectively random, meaning that the timestamp
+/// in any KSUID generated with this function will be random.
+fn ksuid_from_hash(alert: &meta_table_alerts::Alert) -> svix_ksuid::Ksuid {
+    use sha1::{Digest, Sha1};
+
+    let stream_type: alerts_table_ser::StreamType = alert.stream_type.into();
+    let stream_type = stream_type.to_string();
+
+    let mut hasher = Sha1::new();
+    hasher.update(alert.org_id.clone());
+    hasher.update(stream_type);
+    hasher.update(alert.stream_name.clone());
+    hasher.update(alert.name.clone());
+    let hash = hasher.finalize();
+    svix_ksuid::Ksuid::from_bytes(hash.into())
+}
+
 impl From<meta_table_alerts::CompareHistoricData> for alerts_table_ser::CompareHistoricData {
     fn from(value: meta_table_alerts::CompareHistoricData) -> Self {
         Self {

src/infra/src/table/migration/m20241217_155000_populate_alerts_table.rs

@@ -175,8 +175,8 @@ mod legacy_folders {
 
         while let Some(folders) = pages.fetch_and_next().await? {
             for folder in folders {
+                let ksuid = ksuid_from_hash(&folder).to_string();
                 let mut am = folder.into_active_model();
-                let ksuid = svix_ksuid::Ksuid::new(None, None).to_string();
                 println!("folder ksuid: {}", ksuid);
                 am.ksuid = Set(Some(ksuid));
                 am.update(conn).await?;
@@ -193,6 +193,26 @@ mod legacy_folders {
     pub fn drop_table() -> TableDropStatement {
         Table::drop().table(Alias::new(NEW_TABLE_NAME)).to_owned()
     }
+
+    /// Generates a KSUID from a hash of the folder's `org` and `folder_id`.
+    ///
+    /// To generate a KSUID this function generates the 160-bit SHA-1 hash of
+    /// the folder's `org` and `folder_id` and interprets that 160-bit hash as
+    /// a 160-bit KSUID. Therefore two KSUIDs generated in this manner will
+    /// always be equal if the folders have the same `org` and `folder_id`.
+    ///
+    /// It is important to note that KSUIDs generated in this manner will have
+    /// timestamp bits which are effectively random, meaning that the timestamp
+    /// in any KSUID generated with this function will be random.
+    fn ksuid_from_hash(folder: &legacy_entities::legacy_folders::Model) -> svix_ksuid::Ksuid {
+        use sha1::{Digest, Sha1};
+        let mut hasher = Sha1::new();
+        hasher.update(folder.org.clone());
+        hasher.update(folder.r#type.to_string());
+        hasher.update(folder.folder_id.clone());
+        let hash = hasher.finalize();
+        svix_ksuid::Ksuid::from_bytes(hash.into())
+    }
 }
 
 /// Data structures and migration statements for the legacy dashboards table.
@@ -240,8 +260,8 @@ mod legacy_dashboards {
 
         while let Some(dashboards) = pages.fetch_and_next().await? {
             for dashboard in dashboards {
+                let ksuid = ksuid_from_hash(&dashboard).to_string();
                 let mut am = dashboard.into_active_model();
-                let ksuid = svix_ksuid::Ksuid::new(None, None).to_string();
                 am.ksuid = Set(Some(ksuid));
                 am.update(conn).await?;
             }
@@ -257,6 +277,24 @@ mod legacy_dashboards {
     pub fn drop_table() -> TableDropStatement {
         Table::drop().table(Alias::new(NEW_TABLE_NAME)).to_owned()
     }
+
+    /// Generates a KSUID from a hash of the dashboards's `dashboard_id`.
+    ///
+    /// To generate a KSUID this function generates the 160-bit SHA-1 hash of
+    /// the dashboard's `dashboard_id` and interprets that 160-bit hash as a
+    /// 160-bit KSUID. Therefore two KSUIDs generated in this manner will always
+    /// be equal if the dashboard's have the same `dashboard_id`.
+    ///
+    /// It is important to note that KSUIDs generated in this manner will have
+    /// timestamp bits which are effectively random, meaning that the timestamp
+    /// in any KSUID generated with this function will be random.
+    fn ksuid_from_hash(dashboard: &legacy_entities::legacy_dashboards::Model) -> svix_ksuid::Ksuid {
+        use sha1::{Digest, Sha1};
+        let mut hasher = Sha1::new();
+        hasher.update(dashboard.dashboard_id.clone());
+        let hash = hasher.finalize();
+        svix_ksuid::Ksuid::from_bytes(hash.into())
+    }
 }
 
 /// Data structures and migration statements for the legacy alerts table.