Merge pull request #6 from rackspace-infrastructure-automation/indexes

Add additional indexes, tags, streams, update docs, tests, examples

Author: John Titus

README.md

@@ -1,21 +1,76 @@
+# aws-terraform-dynamo
+
+This module creates an AWS DynamoDB table.
+
+## Basic Usage
+
+```HCL
+module "basic" {
+  source = "git@github.com:rackspace-infrastructure-automation/aws-terraform-dynamo/?ref=v0.0.3"
+
+  attributes = [
+    {
+      name = "TestHashKey"
+      type = "S"
+    },
+  ]
+
+  environment          = "Test"
+  hash_key             = "MyHashKey"
+  read_capacity_units  = 10
+  table_name           = "myexampletable"
+  tags                 = "${local.tags}"
+  write_capacity_units = 5
+}
+```
+
+## Known Bugs
+
+When using the index maps for GSI's and LSI's there is a bug in the Terraform AWS provider which stores the order of `non_key_attributes` in the state file in an order that may not match what you pass in.
+
+If a subsequent plan shows changes for your indexes look for lines similar to the below in the output:
+
+```
+global_secondary_index.1708383685.non_key_attributes.0: "data2" => ""
+global_secondary_index.1708383685.non_key_attributes.1: "data1" => ""
+global_secondary_index.1708383685.non_key_attributes.2: "data3" => ""
+```
+
+The input data was `["data1", "data2", "data3"]` and we can see that attributes 0, 1, and 2 do not follow this same order.
+
+The resolution at the moment is to edit the list in your map to match the order in the state file. This should result in a clean plan when no other changes are present.
+
+**References:**
+
+[global secondary index always recreated #3828](https://github.com/terraform-providers/terraform-provider-aws/issues/3828)
+
+[DynamoDB Non-Key Attributes Ordering #3807](https://github.com/terraform-providers/terraform-provider-aws/issues/3807)
+
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|:----:|:-----:|:-----:|
-| attributes | List of nested attribute definitions. Only required for hash_key (always) and range_key (if used) attributes. Attributes have name and type. Type must be a scalar type: S, N, or B for (S)tring, (N)umber or (B)inary data. i.e. [{ name=<hash_key> type=<data_type>}] | list | n/a | yes |
+| attributes | List of nested attribute definitions. Only required for hash_key's (always) and range_key's (if used) attributes. Attributes have name and type. Type must be a scalar type: S, N, or B for (S)tring, (N)umber or (B)inary data. i.e. [{ name=<hash_key> type=<data_type>}]. Terraform documentation: [A note about attributes](https://www.terraform.io/docs/providers/aws/r/dynamodb_table.html#a-note-about-attributes) | list | n/a | yes |
 | enable\_pay\_per\_request | Controls how you are charged for read and write throughput and how you manage capacity. If True, DynamoDB charges you for the data reads and writes your application performs on your tables. You do not need to specify how much read and write throughput you expect your application to perform because DynamoDB instantly accommodates your workloads as they ramp up or down. [On-Demand Pricing](https://aws.amazon.com/dynamodb/pricing/on-demand/) If False, you specify the number of `read_capacity_units` and `write_capacity_units` per second that you expect your workload to require. [Provisioned Pricing](https://aws.amazon.com/dynamodb/pricing/provisioned/) | string | `"false"` | no |
-| hash\_key | ** Forces new resource ** Must contain only alphanumberic characters, dash (-), underscore (_) or dot (.). Needs to be defined by type in attributes. | string | n/a | yes |
+| environment | Application environment for which this resource is being created. Preferred values are Development, Integration, PreProduction, Production, QA, Staging, or Test. | string | `"Development"` | no |
+| global\_secondary\_index\_maps | A list of maps for each [global secondary index (GSI)](https://www.terraform.io/docs/providers/aws/r/dynamodb_table.html#global_secondary_index-1). Please see [examples](./examples) for usage. | list | `<list>` | no |
+| hash\_key | **Forces new resource!** Must contain only alphanumberic characters, dash (-), underscore (_) or dot (.). Needs to be defined by type in attributes. | string | n/a | yes |
+| local\_secondary\_index\_maps | A list of maps for each [local secondary index (LSI)](https://www.terraform.io/docs/providers/aws/r/dynamodb_table.html#local_secondary_index-1). Please see [examples](./examples) for usage. | list | `<list>` | no |
 | point\_in\_time\_recovery | Enable point in time recovery for the table. | string | `"false"` | no |
-| range\_key | ** Forces new resource ** RangeType PrimaryKey Name. If used, it will need to be defined by type in attributes | string | `""` | no |
+| range\_key | **Forces new resource!** RangeType PrimaryKey Name. If used, it will need to be defined by type in attributes | string | `""` | no |
 | read\_capacity\_units | Provisioned read throughput. Should be between 5 and 10000. Ignored if `enable_pay_per_request` is set to `true`. | string | `"5"` | no |
-| table\_encryption | Server side table encryption at rest. i.e. true | false | string | `"true"` | no |
+| stream\_enabled | Enable the stream setting on the table. | string | `"false"` | no |
+| stream\_view\_type | If using `stream_enabled, you can specify a valid DynamoDB StreamViewType; must be one of: `KEYS_ONLY`, `NEW_IMAGE`. `OLD_IMAGE`, `NEW_AND_OLD_IMAGES` | string | `""` | no |
+| table\_encryption\_cmk | You may choose to use an [AWS Managed CMK](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-managed-cmk) by setting this to `true`. Otherwise, server side table encryption defaults to an [AWS Owned CMK](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-owned-cmk). | string | `"false"` | no |
 | table\_name | The name of the table, this needs to be unique within a region. | string | n/a | yes |
+| tags | Custom tags to apply to all resources. | map | `<map>` | no |
 | write\_capacity\_units | Provisioned write throughput. Should be between 5 and 10000. Ignored if `enable_pay_per_request` is set to `true`. | string | `"10"` | no |
 
 ## Outputs
 
 | Name | Description |
 |------|-------------|
+| stream\_arn | ARN for the stream if `stream_enabled` was set to `true`, otherwise returns a string of "null". |
 | table\_arn | Table ARN |
 | table\_name | Table Name |
 

examples/example.tf

@@ -4,7 +4,7 @@ provider "aws" {
 }
 
 module "dynamo_table_provisioned" {
-  source = "git@github.com/rackspace-infrastructure-automation/aws-terraform-dynamo//?ref=v0.0.2"
+  source = "git@github.com/rackspace-infrastructure-automation/aws-terraform-dynamo//?ref=v0.0.3"
 
   table_name             = "<TableName>"
   hash_key               = "<HashKeyName>"
@@ -30,7 +30,7 @@ module "dynamo_table_provisioned" {
 }
 
 module "dynamo_table_pay_per_requst" {
-  source = "git@github.com/rackspace-infrastructure-automation/aws-terraform-dynamo//?ref=v0.0.2"
+  source = "git@github.com/rackspace-infrastructure-automation/aws-terraform-dynamo//?ref=v0.0.3"
 
   table_name             = "<TableName>"
   hash_key               = "<HashKeyName>"

main.tf

@@ -1,18 +1,85 @@
-resource "aws_dynamodb_table" "basic-dynamodb-table" {
-  name           = "${var.table_name}"
-  billing_mode   = "${var.enable_pay_per_request ? "PAY_PER_REQUEST":"PROVISIONED"}"
-  read_capacity  = "${var.read_capacity_units}"
-  write_capacity = "${var.write_capacity_units}"
-  hash_key       = "${var.hash_key}"
-  range_key      = "${var.range_key}"
-
-  attribute = "${var.attributes}"
+/**
+ * # aws-terraform-dynamo
+ *
+ * This module creates an AWS DynamoDB table.
+ *
+ * ## Basic Usage
+ *
+ * ```HCL
+ * module "basic" {
+ *   source = "git@github.com:rackspace-infrastructure-automation/aws-terraform-dynamo/?ref=v0.0.3"
+ *
+ *   attributes = [
+ *     {
+ *       name = "TestHashKey"
+ *       type = "S"
+ *     },
+ *   ]
+ *
+ *   environment          = "Test"
+ *   hash_key             = "MyHashKey"
+ *   read_capacity_units  = 10
+ *   table_name           = "myexampletable"
+ *   tags                 = "${local.tags}"
+ *   write_capacity_units = 5
+ * }
+ * ```
+ *
+ * ## Known Bugs
+ *
+ * When using the index maps for GSI's and LSI's there is a bug in the Terraform AWS provider which stores the order of `non_key_attributes` in the state file in an order that may not match what you pass in.
+ *
+ * If a subsequent plan shows changes for your indexes look for lines similar to the below in the output:
+ *
+ * ```
+ * global_secondary_index.1708383685.non_key_attributes.0: "data2" => ""
+ * global_secondary_index.1708383685.non_key_attributes.1: "data1" => ""
+ * global_secondary_index.1708383685.non_key_attributes.2: "data3" => ""
+ * ```
+ *
+ * The input data was `["data1", "data2", "data3"]` and we can see that attributes 0, 1, and 2 do not follow this same order.
+ *
+ * The resolution at the moment is to edit the list in your map to match the order in the state file. This should result in a clean plan when no other changes are present.
+ *
+ * **References:**
+ *
+ * [global secondary index always recreated #3828](https://github.com/terraform-providers/terraform-provider-aws/issues/3828)
+ *
+ * [DynamoDB Non-Key Attributes Ordering #3807](https://github.com/terraform-providers/terraform-provider-aws/issues/3807)
+ */
 
-  server_side_encryption {
-    enabled = "${var.table_encryption}"
+locals {
+  tags {
+    Environment     = "${var.environment}"
+    Name            = "${var.table_name}"
+    ServiceProvider = "Rackspace"
   }
+}
+
+resource "aws_dynamodb_table" "basic-dynamodb-table" {
+  billing_mode     = "${var.enable_pay_per_request ? "PAY_PER_REQUEST" : "PROVISIONED" }"
+  hash_key         = "${var.hash_key}"
+  name             = "${var.table_name}"
+  range_key        = "${var.range_key}"
+  read_capacity    = "${var.read_capacity_units}"
+  stream_enabled   = "${var.stream_enabled}"
+  stream_view_type = "${var.stream_enabled ? var.stream_view_type : "" }"
+  write_capacity   = "${var.write_capacity_units}"
+
+  attribute              = "${var.attributes}"
+  global_secondary_index = "${var.global_secondary_index_maps}"
+  local_secondary_index  = "${var.local_secondary_index_maps}"
 
   point_in_time_recovery {
     enabled = "${var.point_in_time_recovery}"
   }
+
+  server_side_encryption {
+    enabled = "${var.table_encryption_cmk}"
+  }
+
+  tags = "${merge(
+    local.tags,
+    var.tags
+  )}"
 }

outputs.tf

@@ -1,3 +1,8 @@
+output "stream_arn" {
+  description = "ARN for the stream if `stream_enabled` was set to `true`, otherwise returns a string of \"null\"."
+  value       = "${var.stream_enabled == "true" ? aws_dynamodb_table.basic-dynamodb-table.stream_arn : "null" }"
+}
+
 output "table_arn" {
   description = "Table ARN"
   value       = "${aws_dynamodb_table.basic-dynamodb-table.arn}"

tests/default/main.tf

@@ -3,23 +3,145 @@ provider "aws" {
   region  = "us-west-2"
 }
 
+locals {
+  tags = {
+    Terraform = "true"
+  }
+}
+
 resource "random_string" "rstring" {
   length  = 8
   upper   = false
   special = false
 }
 
 module "dynamo" {
-  source               = "../../module"
-  table_name           = "${random_string.rstring.result}-testTable"
+  source = "../../module"
+
+  attributes = [
+    {
+      name = "TestHashKey"
+      type = "S"
+    },
+  ]
+
   hash_key             = "TestHashKey"
   read_capacity_units  = 6
+  table_name           = "${random_string.rstring.result}-testTable"
+  tags                 = "${local.tags}"
   write_capacity_units = 11
+}
+
+module "advanced" {
+  source = "../../module"
 
   attributes = [
     {
       name = "TestHashKey"
       type = "S"
     },
+    {
+      name = "TestRangeKey"
+      type = "S"
+    },
   ]
+
+  environment            = "Test"
+  hash_key               = "TestHashKey"
+  point_in_time_recovery = true
+  range_key              = "TestRangeKey"
+  read_capacity_units    = 20
+  table_encryption_cmk   = true
+  table_name             = "${random_string.rstring.result}-advanced"
+  tags                   = "${local.tags}"
+  write_capacity_units   = 5
+}
+
+module "complex" {
+  source = "../../module"
+
+  attributes = [
+    {
+      name = "TestHashKey"
+      type = "S"
+    },
+    {
+      name = "TestRangeKey"
+      type = "S"
+    },
+    {
+      name = "GsiHashKey01"
+      type = "S"
+    },
+    {
+      name = "GsiHashKey02"
+      type = "S"
+    },
+    {
+      name = "GsiRangeKey01"
+      type = "S"
+    },
+    {
+      name = "GsiRangeKey02"
+      type = "S"
+    },
+  ]
+
+  environment = "Test"
+
+  global_secondary_index_maps = [
+    {
+      name            = "TestGSIProject"
+      write_capacity  = "5"
+      read_capacity   = "10"
+      hash_key        = "GsiHashKey01"
+      range_key       = "GsiRangeKey01"
+      projection_type = "ALL"
+    },
+    {
+      name            = "TestGSIInclude"
+      write_capacity  = "5"
+      read_capacity   = "10"
+      hash_key        = "GsiHashKey02"
+      range_key       = "GsiRangeKey02"
+      projection_type = "INCLUDE"
+
+      non_key_attributes = [
+        "data2",
+        "data1",
+        "data3",
+      ]
+    },
+  ]
+
+  hash_key = "TestHashKey"
+
+  local_secondary_index_maps = [
+    {
+      name            = "TestLSIProjectAll"
+      range_key       = "TestRangeKey"
+      projection_type = "ALL"
+    },
+    {
+      name            = "TestLSIInclude"
+      range_key       = "TestRangeKey"
+      projection_type = "INCLUDE"
+
+      non_key_attributes = [
+        "data1",
+        "data2",
+        "data3",
+      ]
+    },
+  ]
+
+  point_in_time_recovery = true
+  range_key              = "TestRangeKey"
+  read_capacity_units    = 20
+  stream_enabled         = true
+  stream_view_type       = "NEW_AND_OLD_IMAGES"
+  table_encryption_cmk   = true
+  table_name             = "${random_string.rstring.result}-complex"
+  tags                   = "${local.tags}"
+  write_capacity_units   = 5
 }