Engine Release 8.5.0

With our release process bedding in, we've been able to speed up our release process. So here we are with another release with some minor but useful fixes and features.

Key Features

Runbook Tasks for existing processes

In the 8.4 release we added the initial support for RunBooks. They provide a way to run operational tasks on your deployments using the hamlet CLI. Previously this was implemented through dedicated bash scripts for a specific task, in this release we have now replicated these scripts as runbook tasks and included some modules to help with setting them up

• AWS Module runbook_rds_snapshot adds support for a runbook task that will take a snapshot of a db instance.
• AWS Module runbook_run_task starts a container task with a provided command and waits for the task to exit. You can provide environment or command overrides as part of running the runbook or provide fixed values. This is useful when you want to run a configuration task such as database migrations as part of your deployment pipelines
• The baseline AWS module has been introduced which adds runbooks for encrypting and decrypting content using the baseline KMS key.

Each of these modules provides our recommended approaches to using the runbook tasks. You can also use any of the tasks included in these modules as part of your own runbooks

• engine: (tasks): add basic tasks
• engine: (ecs): add support for default task compute provider
• engine: (tasks): extend ssh tasks and add bash command
• engine: (ecs): add initprocess support for containers
• aws: add additional runbook tasks and modules
• aws: add run ecs task support for runbooks
• aws: add baseline encryption module
• aws: (ecs): support for ecs exec
• aws: extended runbooks for access

Backup Components

Backups have now been mapped in to hamlet components with the backupstore and backupsstoreregime components.

The store represents a backup store that will host the backups that have been made. Each store has regimes that define the resources that should have backups in the store. The regime uses link based processing or layer context rules to define your backups.

This allows for a really simple setup for backups if you just want to backup anything that supports backups

Using a component such as

{
    "backup" : {
        "Type" : "backupstore",
        "Lock" : {
            "Enabled" : true,
            "MinRetention" : 30,
            "MaxRetention" : 365
        },
        "Regimes" : {
            "allComponents" : {
                "Targets" : {
                    "All" : {
                        "Enabled" : true
                    }
                },
                "Rules" : {
                    "nightly" : {
                        "Schedule" : {
                            "Expression" : "cron(0 1 * * *)"
                        }
                    }
                },
                "Conditions" : {
                    "MatchesStore" : {
                        "Enabled" : true,
                        "Product" : true,
                        "Environment" : true,
                        "Segment": true,
                        "Tier" : false
                    }
                }
            }
        }
    }
}

I can create a backup store that will backup any resources in my segment, store backups for a minimum of 30 days and a maximum of 365 days. Any new resources added to the segment should be automatically detected and added to the backup schedule. This has been implemented in AWS using AWS Backup. Hamlet helps with the backup setup as we define the tagging structure applied to all components, so we can create simple backup rules using this standardised tagging structure.

• engine: (backup): Backup support (#1921)
• engine: (backup): Configuration options (#1926)
• engine: backup encryption key (#1927)
• aws: (backup): Initial AWS implementation (#507)

Updates to the CI/CD container extensions

The hamlet and dind extensions have been updated to align with the latest configuration environment variables used in the hamlet CLI;

• AWS_AUTOMATION_USER has been replaced by the HAMLET_AWS_AUTH_SOURCE and HAMLET_AWS_AUTH_USER settings

• When using dind as a docker host TLS verification is required at all times

• Use of the rexray/ebs plugin for docker has been removed as the plugin no longer seems to be maintained and is failing on newer ec2 instances. The recommendation now is to use local volumes and extend the size of your docker volume

• add support for using local docker volumes for builds

• refactor hamlet and dind docker extensions

Log Encryption at Rest

Mostly an AWS function but we have added support for enabling log group encryption at rest across all components which create log groups. This is an extension of the LoggingProfile that can be applied across all components to ensure that log encryption is enabled. Adding this in the logging profile makes it an easy set and forget option to enable encryption for logs at rest whenever a new log group is created.

• engine: (legacy): add encryption at rest support for logs
• engine: add account layer control over logging
• engine: add logging profile support for encryption at rest
• aws: (logs): add support for at rest encryption of cw logs

Engine Input Sanitisation

The underlying engine that we use for hamlet is the freemarker engine, it doesn't have great support for null values imported from JSON objects. As part of our input processing we now strip any null values and their keys from the JSON object.

To help understand how inputs are processed within the engine we now have a new entrance called an inputinfo entrance. This lists all of the input stages and how the processing is assembled for different input providers

• add inputinfo entrance
• add null value cleaner to input stages

Bundled Core Engine

The freemarker engine is built as a Java based application that we call as part of the hamlet CLI. This is a bit of a surprise for someone starting with hamlet especially when using our python base CLI. So to help with that the core engine is now distributed with its own Java installation as a bundled command line tool. While this does increase the size of the engines as they are downloaded it does make it a lot easier to install and get going with hamlet.

In this update we have also moved from Java8 to Java11 to catch up with the latest Java releases.

Component Updates

The updates below are the key change log updates from each of the hamlet engine components

Engine: New Features

• max-age control on bucket content (#1953)
• (adaptor): add support for alert configuration
• full names based on district
• (directory): support for log forwarding
• (datafeed): compression control for buckets
• (globaldb): add alerts support
• (ses): add control over IP access policy
• (cdn): add origin connection timeout support

Engine: Fixes

• include component settings in environment

• district type for district lookup

• include account in mock filter layers

• encryption of logs for sms

• use raw path for settings path prefix

• (apigateway): handling of null values in definitions

• prefix handling for shared provider

• (lb): add required logging profile to lb (#1925)

• module lookup process

• (cdn): add logging profile

• (cd): engine install

• domain parent handling

Engine: Refactorings

• asfile ordering (#1955)
• use local engine definition for testing
• update deployment group district handling
• rename district to district type
• remove task containers from shared provider
• consolidate link functions
• remove type based attributes from healthchecks
• (s3): use recommended process for bucket policy
• (district): use attributeset for config
• more specific name part config
• bootstrap clo processing
• Account descriptions and placement profiles (#1916)
• attributesets for components
• move domain and certificate to reference data

Engine: Others

• update runtimes for lambda environment

Bash Executor: New Features

• add support for bundled freemarker wrapper

• handling of bundled vs jar wrapper

• add support for providing the district CLI option to templates

• add docdb support (#312)

Bash Executor: Fixes

• district default handling
• sentry release CLI support
• make output dir for binary only
• macos run_id and expo ouputs

Bash Executor: Refactorings

• district rename to district type
• use weird seperate for user provided paramters

AWS Plugin: New Features

• use local engine setup for testing
• (spa): force max-age for config (#530)
• (adaptor): adaptor alert support
• (mta): add enable/disable handling on rules
• (directory): log forwarding support (#517)
• (s3): backup support (#516)
• (kinesis): compression support for firehose
• (globaldb): cloudwatch alarms (#508)
• (cdn): add support for origin connection timeouts

AWS Plugin: Fixes

• segment seed fixture value (#535)
• region lookup for resources
• (healthcheck): add more entropy to naming of health checks
• (db): secret lookup for engine setup
• typo in module
• (healthceheck): testing changes from type to engine
• remove use of isPresent for AV setup
• ipmatch and geomatch for wafv2 (#518)
• (db): aurora cluster updates
• (task): kms encrypt parameters
• efs mount script formatting
• (db): ingress security group id
• (sns): add support for encrypted topics
• (lb): logging profile for WAF logs (#510)
• (cdn): missing logging profile for waf logging
• (cdn): logging script for wafv1
• clean up old if statement
• (s3): replication validation checking

AWS Plugin: Refactorings

• align the run task module to task
• update ecs task configuration after testing
• move to latest unicycle install process
• update district to district type on group filter (#534)
• move ecs container setup to aws provider
• update iam standard policy name
• (iam): standard policies for app components
• (s3): use references for bucket policy
• backup encryption key (#512)
• (backup): Configuration options (#511)
• attribute sets for global configuration

Azure Plugin: Fixes

• (lambda): align runtimes with latest updates

Refactorings

• use local engine for testing (#294)