Engine Release 8.6

May 27, 2022 · 10 min read

This release has been a pretty standard update with new components, tasks and general improvements, and bug fixes.

Nothing too drastic... but we have started putting together a list of larger changes that will be introduced in our next release.

Key Features

User Defined Tagging

You can now control the tags applied to your resources that have been deployed by hamlet. Tags have been broken up into two categories, Common and Additional.

Common tags are based on details of the occurrence that is being deployed. This includes the tags that we already provide in hamlet, and include things like environment, product, build reference and the generated name of the component
Additional tags are user defined tags. These work as a collection of key value pairs that you can apply as you need to. Using Deployment Profiles you can easily create a tagging policy across all of your deployments.

To use tags all components now have a Tags attribute

{
    "Tags": {
        "Additional": {},
        "Common": {
            "Component": "boolean",
            "CostCentre": "boolean",
            "Deployment": "boolean",
            "Layers": "boolean",
            "Name": "string",
            "Prefix": "string",
            "Solution": "boolean"
        }
    }
}

The additional tags are defined as objects underneath the Additional attribute

```json
{
    "MyKey" : {
        "Value" : "MyValue"
    }
}

Tag configuration has been added to all components, some providers don't support tags for particular resources so they might not be included on all resources.

tagging control for occurrences

Support for Additional Values

Component attributes now support setting additional values on top of the default values they offer. This is handy when values are passed from hamlet components directly to providers. If the provider adds support for a new value, for example a new AWS lambda runtime, you can use it straight away without waiting for us to add the new runtime into hamlet. This is currently enabled on the lambda component and will be added to other components as we identify use cases for this configuration option.

(lambda): chg constraint on runtime to warning
(lambda): remove constraint on runtime

New Components

A few new components made it into this release.

Load balancer backends This is a new subcomponent on load balancers that defines a group of backend servers independent of their Port Mappings. This is useful when you have a set of conditional rules that all use the same backend servers. Now you can define one backend and link the Port Mapping to the Backend.
Log Store Is an instance of log storage standalone from a component. You could use this when you have a lambda function that writes application specific logs and you want to maintain separation between the logs generated by the lambda function and the application logs.
Data Stream Similar to the data feed component which looks after forwarding data to a destination, the data stream handles collecting large amounts of data and queuing it for processing by clients to consume.
DocDB Adds support for deploying Document based NoSQL hosts. It's similar to the db component which provides the same support for SQL based database hosts
DNS Zone Public or private (network attached) dns zones can be defined through hamlet and uses the standard domain based configuration to assign the zone name.
Certificate Authority Used for building private PKI infrastructure, the certificate authority component represents a single authority. Through links between certificate authorities you can define an authority chain in line with best practices for PKI.

These components were implemented in the following changes:

(lb): define backends independent from port mappings
(logstore): add engine parameter support
(logstore): add dedicated log storage component
(datastream): define data stream component
add docdb support (#1934)
(dnszone): add support for network based configuration
add certificate authority component
aws logstore implementation (#573)
(datastream): aws implementation of data stream (#575)
add docdb support (#521)
(dnszone): add support for private vpc zones
(certificateauthority): intial support with ACMPCA

New Tasks

Runbooks are becoming more and more useful for running operational tasks on your deployments. In this release the aws provider has been updated with more tasks;

SES Password Generation This generates a sig4 signature of an AWS_SECRET_ACCESS_KEY that can be used to send emails through SES using the SMTP based API
S3 Empty Bucket Removes all objects and their versions from an S3 bucket
S3 Download Bucket Copies the contents of an S3 bucket to a local directory
Secrets Manager Get Secret Gets the value of a secret stored in AWS Secrets manager
add SES SMTP user module
(task): add ses smtp password generation task
add s3 runbook tasks
(task): add secrets manager get secret task

Policy Chunking for AWS

hamlet generates fine grained IAM permissions policies for each resource it deploys. This is great from a security best practice approach, however AWS does have limits on the size of an IAM policy. Some of our users were having issues with the generated policy hitting the policy size limits with complex deployments. To support this when generating IAM policy documents we split the permissions across multiple policies automatically.

policy chunking (#545)

Build Details

When working with images for application deployments its useful to know which version of an image is being used in a given environment. You can now find this quickly using the cli.

Running hamlet release list-image-references will provide a list of all the occurrences that have an image associated with them along with details of the image

hamlet --profile msw release list-image-references

| Occurrence                       | Deployment Unit   | Build Reference                          | Code Tag   | Image Format   | Image Source   |
|----------------------------------|-------------------|------------------------------------------|------------|----------------|----------------|
| management-lamb-test-function    | lamb              | def45678900000                           |            | lambda         |                |
| management-apphost-myapp-service | myapp             | d66500e2741196b75fc3aed24e35fddad42fa212 |            | docker         |                |
| management-apphost-mytask-task   | myapp             | d66500e2741196b75fc3aed24e35fddad42fa212 |            | docker         |                |

Event Logging

The logging used when hamlet saves commits to your cmdb has been extended to support a wider range of actions. Events are now stored in an event log stored in your local HAMLET_HOME_DIR. When changes to the CMDB are committed to a repository using the hamlet cmdb commit-changes command the events for the CMDB are collected from the log, formatted as commit messages and saved to the CMDB.

add HAMLET_EVENT_DIR
event logging for write events

Changes Required

This release had a couple of updates that will require some minor updates to cmdb solutions;

On firewall rules, the attribute Type has been renamed to RuleType. This is to fix an issue when the Type attribute was added to all components to set their component type
The Solution layer has been removed as it didn't make sense with what layers are intended for. This was also done to make way for being able to define solutions differently in future releases. Any configuration applied on the Solution object in your solutions should be moved to an appropriate layer, most likely the Segment or Environment layer
MultiAZ configuration has been moved from the Layers down to the components. A deployment profile was added to all layers that used the old multiAZ configuration option. You should check that your deployments are using MultiAZ deployment options if they need it
Backup configuration was removed from the datavolume component. To backup a datavolume we now recommend using the backup component which can handle backing up a much wider ranger of resources

Upcoming

Finally in this release we have deprecated direct usage of the bash executor as the cli should now support all the actions that were available from the bash executor. When running a deprecated command you will be given a warning that the command has been deprecated. If you have questions about which command replaces a deprecated command feel free to reach out in our GitHub discussions.

Full Change Log

Engine

New Features

(cdn): add priority to cdn routes
(lambda): versioned lambda retention policy (#1989)
(alerts): add enable attribute on alerts
(lb): define external or internal for lb endpoints
(lambda): provisioned executions (#1980)
(datavolume): zone control and remove backups
(cdn): add support for disabling event handlers
add support for HealthCheck Protocol
(ec2): add support for zone based deploy control

Fixes

add generation contract for unitlist view
(runbook): don't include disabled tasks
only include active layers based on district type
(account): minor fixes for account level aws deployments
support regions in state that use refs
minor version upgrade control
typo
typos in attributes

Refactorings

(apigateway): authorization models (#1995)
testing output handling and test profiles
move functions out of setContext
update testing to remove solution layer

AWS Engine Plugin

New Features

(vpc): support creating security groups without inbound ports
(lb): add support for lb backends
(datafeed): add support for data streams as a source
(cdn): rule priority sorting
(rds): event support (#569)
occurrence level configuration tagging
(lambda): sqs batch control
(lambda): versioned lambda retention policy (#574)
(alertslack): allow setting runtime with param (#570)
(alerts): add filter on alerts for enable status
(ec2): support IPAddress groups and LB on same port
only include enabled occurrences in suboccurrence processing
(lambda): provisioned executions (#559)
remove auto state generation for fixutre testing
(secretsmanager): add read write support fo secrets
(apigateway): mutual TLS attribute (#548)
ec2 resource outputs and replace updates
(cdn): only add enabled event handlers
(cdn): add error for wrong logging region
(lb): add support for alb as a network target (#537)
add role tag to components if present
(ec2): zone based control for instances

Fixes

(ecs): tags handling (#588)
handle missing link
(datastream): typo in attribute name
(globaldb): tag function call
(sqs): add dlqName back into setup routine
handle empty tag sets
add backup tags for dds
spelling in message
(s3): handle notifications for endpoints already deployed
casing for MulitAZ attribute
format json content for run task module
(ec2): fix ordering for cfn init commands
(ecs): ensure subnets are always treated as an array
(ecs): paramter types for templates
typo
(lb): use suboccurrence for static forwardning
testing updates (#556)
various updates from testing
log and account processing
include Value in getReference
(computecluster): general fixes
(iam): inline policy creation from policy set (#547)
(backupstore): tag based conditions
typo in message

Refactorings

(apigateway): authorization models (#581)
(network): remove baseline components that aren't required
network subnet function
move test module loading to product layer
(iam): limits used for policy splitting (#549)
(datavolume): zone filter support for volume mounts

Azure Engine Plugin

move test module loading to product layer

Full set of changes: 8.5.0...8.6.0

Bash Executor

New Features

detect existing images (#326)
add entrance output dir for imagedetails

Fixes

spelling in messages
three level context properties (#327)

Refactorings

sort template objects
scope settings search and remove legacy
remove deprecated integrator commands
remove blueprint doc generation
remove reference to slices
automation setContext updates