Skip to main content

· 5 min read

This has been a pretty quiet release this round, a few helpful updates with bug fixes taking the front of stage. There is also some initial work in our image management process which will be progressed in our next release and includes using links for image sharing along with a more engine based approach to managing images.

Key Features

Extensions for more components

We've found that as solutions grow the need for minor tweaks on components are required. So we have added extensions to more components to allow for updates to resource policies and even the ability to add additional resources to a common component.

  • Core - (baseline): add extension support for data (#2028)
  • Core - add support for topic and queue extensions
  • AWS - (baseline): extension and policy support
  • AWS - queue topic permission extensions (#618)

Dynamic Value Evaluation

When runbooks were introduced they needed the ability to pass value references between steps. The solution to this was to add support for a format for referencing a dynamic value as part of another string. With this release we have extended this support to all components and introduced the ability to add different dynamic look up sources.

The syntax for a dynamic value is __<type>:<attribute>:<attribute>__ where type is the dynamic value provider type and a value provider provides attributes that are used in the lookup process.

Dynamic values can be used on any component string properties defined in the solution. They can be combined with other values in the same string as well. So you can have a url formatted like "https://__attribute:cdn:FQDN__" and when the dynamic value is replaced it will replace everything between __ ( included ) with the value that was looked up.

The initial release supports the following dynamic providers

  • attribute: look up an attribute of linked component
  • setting: use a setting of the component in its env var format
  • aws_secret: (AWS only) use CloudFormation Secrets manager integration to provide a secret value to CloudFormation

Adding more Enable controls across attributes

With support for qualification across the whole solution having the ability to define if something is enabled or not in solution configuration makes it easy to use qualification to control if that setting should be used in each environment. This release has seen the enabled option added to more component configuration options.

Component Updates

Lambda

Added support for defining aliases as fixed name references to a given version of a lambda function

Correspondent

Added support for configuring the channels used by correspondent to communicate with users

LB

AWS - Added support for using a lambda function as a backend to an application load balancer

Full Change Log

Core

New Features

  • (baseline): add extension support for data (#2028)
  • (port): add enable for port healthcheck (#2030)
  • (entrance): stackoutput entrance
  • add support for topic and queue extensions
  • tasks and extensions for images in runbooks
  • (image): add support for the image component
  • (lambda): lambda alias (#2015)
  • add support for dynamic value evaulation
  • (correspondent): add support for channel configuration
  • (lb): add enable support for conditions

Fixes

  • (dn): Test for missing domain name (#2008)
  • (dynamicinput): error message text (#2027)
  • set buildblueprint output file name (#2026)
  • schema entrance output filename (#2025)
  • typo for deployment test
  • (output): output prefix handling
  • typo in dynamicvalue return
  • (dynamicvalue): remove fatal on missing link
  • update naming for source
  • correct private key property name (#2017)
  • (dynamicvalue): fix typo in configuration scope
  • (dynamicvalue): resolve handling for lists (#2013)
  • comment update
  • handle empty values or object references

Refactorings

  • (runbook): support dynamic values in conditions
  • remove occurrence dependency and apply at the occurrence level

Others

  • (lambda): update lambda runtime versions (#2031)

AWS

New Features

  • (baseline): extension and policy support
  • (lb): lambda backend support for lb (#626)
  • queue topic permission extensions (#618)
  • (image): aws support for the image component
  • (runbook): add push_image runbook for images
  • adds extensions and tasks for image management
  • (lambda): lambda aliases (#606)
  • (dyanmicvalues): add support for aws secrets as dynamic values
  • (correspondent): add support for AWS pinpoint channels
  • (lb): add support for enabling/disbaling conditions

Fixes

  • incorporate feedback
  • (cert): raise error on invalid FQDN
  • support dns zone creation without network (#619)
  • (lb): expand permissions for lambda invoke (#629)
  • error messages for port lookups (#628)
  • (lb): protocol checks for nlb
  • (lb): action lookup for network load balancer (#623)
  • (computetask): windows directory creation (#622)
  • (directory): handle missing config connector
  • (healthcheck): add iam service to setup
  • (lb): create alerts across all occurrences
  • (waf): support v1 -> v2 migrations
  • (db): aurora cluster backups (#616)
  • (lambda): size checking (#609)
  • (apigateway): handle open ip address groups (#608)
  • (apigateway): reference versioned lambdas
  • (lb): standard error for invalid port mapping

Refactorings

  • (ecs): lg pass handling (#607)

Azure

Fixes

  • (lb): standard error for invalid port mapping

Bash

New Features

  • (expo): add support for app display name on IOS apps (#340)
  • add stackoutput entrance support
  • (expo): add support for firebase and plugins

Fixes

  • conventional commit support (#342)
  • incorporate feedback
  • incorporate feedback
  • (cli): disable paging for aws cli v2
  • (expo): include project details for plist (#341)

Refactorings

  • set credentials (#343)

· 10 min read

This release has been a pretty standard update with new components, tasks and general improvements, and bug fixes.

Nothing too drastic... but we have started putting together a list of larger changes that will be introduced in our next release.

Key Features

User Defined Tagging

You can now control the tags applied to your resources that have been deployed by hamlet. Tags have been broken up into two categories, Common and Additional.

  • Common tags are based on details of the occurrence that is being deployed. This includes the tags that we already provide in hamlet, and include things like environment, product, build reference and the generated name of the component
  • Additional tags are user defined tags. These work as a collection of key value pairs that you can apply as you need to. Using Deployment Profiles you can easily create a tagging policy across all of your deployments.

To use tags all components now have a Tags attribute

{
"Tags": {
"Additional": {},
"Common": {
"Component": "boolean",
"CostCentre": "boolean",
"Deployment": "boolean",
"Layers": "boolean",
"Name": "string",
"Prefix": "string",
"Solution": "boolean"
}
}
}

The additional tags are defined as objects underneath the Additional attribute

```json
{
"MyKey" : {
"Value" : "MyValue"
}
}

Tag configuration has been added to all components, some providers don't support tags for particular resources so they might not be included on all resources.

  • tagging control for occurrences

Support for Additional Values

Component attributes now support setting additional values on top of the default values they offer. This is handy when values are passed from hamlet components directly to providers. If the provider adds support for a new value, for example a new AWS lambda runtime, you can use it straight away without waiting for us to add the new runtime into hamlet. This is currently enabled on the lambda component and will be added to other components as we identify use cases for this configuration option.

  • (lambda): chg constraint on runtime to warning
  • (lambda): remove constraint on runtime

New Components

A few new components made it into this release.

  • Load balancer backends This is a new subcomponent on load balancers that defines a group of backend servers independent of their Port Mappings. This is useful when you have a set of conditional rules that all use the same backend servers. Now you can define one backend and link the Port Mapping to the Backend.
  • Log Store Is an instance of log storage standalone from a component. You could use this when you have a lambda function that writes application specific logs and you want to maintain separation between the logs generated by the lambda function and the application logs.
  • Data Stream Similar to the data feed component which looks after forwarding data to a destination, the data stream handles collecting large amounts of data and queuing it for processing by clients to consume.
  • DocDB Adds support for deploying Document based NoSQL hosts. It's similar to the db component which provides the same support for SQL based database hosts
  • DNS Zone Public or private (network attached) dns zones can be defined through hamlet and uses the standard domain based configuration to assign the zone name.
  • Certificate Authority Used for building private PKI infrastructure, the certificate authority component represents a single authority. Through links between certificate authorities you can define an authority chain in line with best practices for PKI.

These components were implemented in the following changes:

  • (lb): define backends independent from port mappings
  • (logstore): add engine parameter support
  • (logstore): add dedicated log storage component
  • (datastream): define data stream component
  • add docdb support (#1934)
  • (dnszone): add support for network based configuration
  • add certificate authority component
  • aws logstore implementation (#573)
  • (datastream): aws implementation of data stream (#575)
  • add docdb support (#521)
  • (dnszone): add support for private vpc zones
  • (certificateauthority): intial support with ACMPCA

New Tasks

Runbooks are becoming more and more useful for running operational tasks on your deployments. In this release the aws provider has been updated with more tasks;

  • SES Password Generation This generates a sig4 signature of an AWS_SECRET_ACCESS_KEY that can be used to send emails through SES using the SMTP based API

  • S3 Empty Bucket Removes all objects and their versions from an S3 bucket

  • S3 Download Bucket Copies the contents of an S3 bucket to a local directory

  • Secrets Manager Get Secret Gets the value of a secret stored in AWS Secrets manager

  • add SES SMTP user module

  • (task): add ses smtp password generation task

  • add s3 runbook tasks

  • (task): add secrets manager get secret task

Policy Chunking for AWS

hamlet generates fine grained IAM permissions policies for each resource it deploys. This is great from a security best practice approach, however AWS does have limits on the size of an IAM policy. Some of our users were having issues with the generated policy hitting the policy size limits with complex deployments. To support this when generating IAM policy documents we split the permissions across multiple policies automatically.

  • policy chunking (#545)

Build Details

When working with images for application deployments its useful to know which version of an image is being used in a given environment. You can now find this quickly using the cli.

Running hamlet release list-image-references will provide a list of all the occurrences that have an image associated with them along with details of the image

hamlet --profile msw release list-image-references
| Occurrence                       | Deployment Unit   | Build Reference                          | Code Tag   | Image Format   | Image Source   |
|----------------------------------|-------------------|------------------------------------------|------------|----------------|----------------|
| management-lamb-test-function | lamb | def45678900000 | | lambda | |
| management-apphost-myapp-service | myapp | d66500e2741196b75fc3aed24e35fddad42fa212 | | docker | |
| management-apphost-mytask-task | myapp | d66500e2741196b75fc3aed24e35fddad42fa212 | | docker | |

Event Logging

The logging used when hamlet saves commits to your cmdb has been extended to support a wider range of actions. Events are now stored in an event log stored in your local HAMLET_HOME_DIR. When changes to the CMDB are committed to a repository using the hamlet cmdb commit-changes command the events for the CMDB are collected from the log, formatted as commit messages and saved to the CMDB.

  • add HAMLET_EVENT_DIR
  • event logging for write events

Changes Required

This release had a couple of updates that will require some minor updates to cmdb solutions;

  • On firewall rules, the attribute Type has been renamed to RuleType. This is to fix an issue when the Type attribute was added to all components to set their component type
  • The Solution layer has been removed as it didn't make sense with what layers are intended for. This was also done to make way for being able to define solutions differently in future releases. Any configuration applied on the Solution object in your solutions should be moved to an appropriate layer, most likely the Segment or Environment layer
  • MultiAZ configuration has been moved from the Layers down to the components. A deployment profile was added to all layers that used the old multiAZ configuration option. You should check that your deployments are using MultiAZ deployment options if they need it
  • Backup configuration was removed from the datavolume component. To backup a datavolume we now recommend using the backup component which can handle backing up a much wider ranger of resources

Upcoming

Finally in this release we have deprecated direct usage of the bash executor as the cli should now support all the actions that were available from the bash executor. When running a deprecated command you will be given a warning that the command has been deprecated. If you have questions about which command replaces a deprecated command feel free to reach out in our GitHub discussions.

Full Change Log

Engine

New Features

  • (cdn): add priority to cdn routes
  • (lambda): versioned lambda retention policy (#1989)
  • (alerts): add enable attribute on alerts
  • (lb): define external or internal for lb endpoints
  • (lambda): provisioned executions (#1980)
  • (datavolume): zone control and remove backups
  • (cdn): add support for disabling event handlers
  • add support for HealthCheck Protocol
  • (ec2): add support for zone based deploy control

Fixes

  • add generation contract for unitlist view
  • (runbook): don't include disabled tasks
  • only include active layers based on district type
  • (account): minor fixes for account level aws deployments
  • support regions in state that use refs
  • minor version upgrade control
  • typo
  • typos in attributes

Refactorings

  • (apigateway): authorization models (#1995)
  • testing output handling and test profiles
  • move functions out of setContext
  • update testing to remove solution layer

AWS Engine Plugin

New Features

  • (vpc): support creating security groups without inbound ports
  • (lb): add support for lb backends
  • (datafeed): add support for data streams as a source
  • (cdn): rule priority sorting
  • (rds): event support (#569)
  • occurrence level configuration tagging
  • (lambda): sqs batch control
  • (lambda): versioned lambda retention policy (#574)
  • (alertslack): allow setting runtime with param (#570)
  • (alerts): add filter on alerts for enable status
  • (ec2): support IPAddress groups and LB on same port
  • only include enabled occurrences in suboccurrence processing
  • (lambda): provisioned executions (#559)
  • remove auto state generation for fixutre testing
  • (secretsmanager): add read write support fo secrets
  • (apigateway): mutual TLS attribute (#548)
  • ec2 resource outputs and replace updates
  • (cdn): only add enabled event handlers
  • (cdn): add error for wrong logging region
  • (lb): add support for alb as a network target (#537)
  • add role tag to components if present
  • (ec2): zone based control for instances

Fixes

  • (ecs): tags handling (#588)
  • handle missing link
  • (datastream): typo in attribute name
  • (globaldb): tag function call
  • (sqs): add dlqName back into setup routine
  • handle empty tag sets
  • add backup tags for dds
  • spelling in message
  • (s3): handle notifications for endpoints already deployed
  • casing for MulitAZ attribute
  • format json content for run task module
  • (ec2): fix ordering for cfn init commands
  • (ecs): ensure subnets are always treated as an array
  • (ecs): paramter types for templates
  • typo
  • (lb): use suboccurrence for static forwardning
  • testing updates (#556)
  • various updates from testing
  • log and account processing
  • include Value in getReference
  • (computecluster): general fixes
  • (iam): inline policy creation from policy set (#547)
  • (backupstore): tag based conditions
  • typo in message

Refactorings

  • (apigateway): authorization models (#581)
  • (network): remove baseline components that aren't required
  • network subnet function
  • move test module loading to product layer
  • (iam): limits used for policy splitting (#549)
  • (datavolume): zone filter support for volume mounts

Azure Engine Plugin

  • move test module loading to product layer

Full set of changes: 8.5.0...8.6.0

Bash Executor

New Features

  • detect existing images (#326)
  • add entrance output dir for imagedetails

Fixes

  • spelling in messages
  • three level context properties (#327)

Refactorings

  • sort template objects
  • scope settings search and remove legacy
  • remove deprecated integrator commands
  • remove blueprint doc generation
  • remove reference to slices
  • automation setContext updates

· 9 min read

With our release process bedding in, we've been able to speed up our release process. So here we are with another release with some minor but useful fixes and features.

Key Features

Runbook Tasks for existing processes

In the 8.4 release we added the initial support for RunBooks. They provide a way to run operational tasks on your deployments using the hamlet CLI. Previously this was implemented through dedicated bash scripts for a specific task, in this release we have now replicated these scripts as runbook tasks and included some modules to help with setting them up

  • AWS Module runbook_rds_snapshot adds support for a runbook task that will take a snapshot of a db instance.
  • AWS Module runbook_run_task starts a container task with a provided command and waits for the task to exit. You can provide environment or command overrides as part of running the runbook or provide fixed values. This is useful when you want to run a configuration task such as database migrations as part of your deployment pipelines
  • The baseline AWS module has been introduced which adds runbooks for encrypting and decrypting content using the baseline KMS key.

Each of these modules provides our recommended approaches to using the runbook tasks. You can also use any of the tasks included in these modules as part of your own runbooks

  • engine: (tasks): add basic tasks
  • engine: (ecs): add support for default task compute provider
  • engine: (tasks): extend ssh tasks and add bash command
  • engine: (ecs): add initprocess support for containers
  • aws: add additional runbook tasks and modules
  • aws: add run ecs task support for runbooks
  • aws: add baseline encryption module
  • aws: (ecs): support for ecs exec
  • aws: extended runbooks for access

Backup Components

Backups have now been mapped in to hamlet components with the backupstore and backupsstoreregime components.

The store represents a backup store that will host the backups that have been made. Each store has regimes that define the resources that should have backups in the store. The regime uses link based processing or layer context rules to define your backups.

This allows for a really simple setup for backups if you just want to backup anything that supports backups

Using a component such as

{
"backup" : {
"Type" : "backupstore",
"Lock" : {
"Enabled" : true,
"MinRetention" : 30,
"MaxRetention" : 365
},
"Regimes" : {
"allComponents" : {
"Targets" : {
"All" : {
"Enabled" : true
}
},
"Rules" : {
"nightly" : {
"Schedule" : {
"Expression" : "cron(0 1 * * *)"
}
}
},
"Conditions" : {
"MatchesStore" : {
"Enabled" : true,
"Product" : true,
"Environment" : true,
"Segment": true,
"Tier" : false
}
}
}
}
}
}

I can create a backup store that will backup any resources in my segment, store backups for a minimum of 30 days and a maximum of 365 days. Any new resources added to the segment should be automatically detected and added to the backup schedule. This has been implemented in AWS using AWS Backup. Hamlet helps with the backup setup as we define the tagging structure applied to all components, so we can create simple backup rules using this standardised tagging structure.

  • engine: (backup): Backup support (#1921)
  • engine: (backup): Configuration options (#1926)
  • engine: backup encryption key (#1927)
  • aws: (backup): Initial AWS implementation (#507)

Updates to the CI/CD container extensions

The hamlet and dind extensions have been updated to align with the latest configuration environment variables used in the hamlet CLI;

  • AWS_AUTOMATION_USER has been replaced by the HAMLET_AWS_AUTH_SOURCE and HAMLET_AWS_AUTH_USER settings

  • When using dind as a docker host TLS verification is required at all times

  • Use of the rexray/ebs plugin for docker has been removed as the plugin no longer seems to be maintained and is failing on newer ec2 instances. The recommendation now is to use local volumes and extend the size of your docker volume

  • add support for using local docker volumes for builds

  • refactor hamlet and dind docker extensions

Log Encryption at Rest

Mostly an AWS function but we have added support for enabling log group encryption at rest across all components which create log groups. This is an extension of the LoggingProfile that can be applied across all components to ensure that log encryption is enabled. Adding this in the logging profile makes it an easy set and forget option to enable encryption for logs at rest whenever a new log group is created.

  • engine: (legacy): add encryption at rest support for logs
  • engine: add account layer control over logging
  • engine: add logging profile support for encryption at rest
  • aws: (logs): add support for at rest encryption of cw logs

Engine Input Sanitisation

The underlying engine that we use for hamlet is the freemarker engine, it doesn't have great support for null values imported from JSON objects. As part of our input processing we now strip any null values and their keys from the JSON object.

To help understand how inputs are processed within the engine we now have a new entrance called an inputinfo entrance. This lists all of the input stages and how the processing is assembled for different input providers

  • add inputinfo entrance
  • add null value cleaner to input stages

Bundled Core Engine

The freemarker engine is built as a Java based application that we call as part of the hamlet CLI. This is a bit of a surprise for someone starting with hamlet especially when using our python base CLI. So to help with that the core engine is now distributed with its own Java installation as a bundled command line tool. While this does increase the size of the engines as they are downloaded it does make it a lot easier to install and get going with hamlet.

In this update we have also moved from Java8 to Java11 to catch up with the latest Java releases.

Component Updates

The updates below are the key change log updates from each of the hamlet engine components

Engine: New Features

  • max-age control on bucket content (#1953)
  • (adaptor): add support for alert configuration
  • full names based on district
  • (directory): support for log forwarding
  • (datafeed): compression control for buckets
  • (globaldb): add alerts support
  • (ses): add control over IP access policy
  • (cdn): add origin connection timeout support

Engine: Fixes

  • include component settings in environment

  • district type for district lookup

  • include account in mock filter layers

  • encryption of logs for sms

  • use raw path for settings path prefix

  • (apigateway): handling of null values in definitions

  • prefix handling for shared provider

  • (lb): add required logging profile to lb (#1925)

  • module lookup process

  • (cdn): add logging profile

  • (cd): engine install

  • domain parent handling

Engine: Refactorings

  • asfile ordering (#1955)
  • use local engine definition for testing
  • update deployment group district handling
  • rename district to district type
  • remove task containers from shared provider
  • consolidate link functions
  • remove type based attributes from healthchecks
  • (s3): use recommended process for bucket policy
  • (district): use attributeset for config
  • more specific name part config
  • bootstrap clo processing
  • Account descriptions and placement profiles (#1916)
  • attributesets for components
  • move domain and certificate to reference data

Engine: Others

  • update runtimes for lambda environment

Bash Executor: New Features

  • add support for bundled freemarker wrapper

  • handling of bundled vs jar wrapper

  • add support for providing the district CLI option to templates

  • add docdb support (#312)

Bash Executor: Fixes

  • district default handling
  • sentry release CLI support
  • make output dir for binary only
  • macos run_id and expo ouputs

Bash Executor: Refactorings

  • district rename to district type
  • use weird seperate for user provided paramters

AWS Plugin: New Features

  • use local engine setup for testing
  • (spa): force max-age for config (#530)
  • (adaptor): adaptor alert support
  • (mta): add enable/disable handling on rules
  • (directory): log forwarding support (#517)
  • (s3): backup support (#516)
  • (kinesis): compression support for firehose
  • (globaldb): cloudwatch alarms (#508)
  • (cdn): add support for origin connection timeouts

AWS Plugin: Fixes

  • segment seed fixture value (#535)
  • region lookup for resources
  • (healthcheck): add more entropy to naming of health checks
  • (db): secret lookup for engine setup
  • typo in module
  • (healthceheck): testing changes from type to engine
  • remove use of isPresent for AV setup
  • ipmatch and geomatch for wafv2 (#518)
  • (db): aurora cluster updates
  • (task): kms encrypt parameters
  • efs mount script formatting
  • (db): ingress security group id
  • (sns): add support for encrypted topics
  • (lb): logging profile for WAF logs (#510)
  • (cdn): missing logging profile for waf logging
  • (cdn): logging script for wafv1
  • clean up old if statement
  • (s3): replication validation checking

AWS Plugin: Refactorings

  • align the run task module to task
  • update ecs task configuration after testing
  • move to latest unicycle install process
  • update district to district type on group filter (#534)
  • move ecs container setup to aws provider
  • update iam standard policy name
  • (iam): standard policies for app components
  • (s3): use references for bucket policy
  • backup encryption key (#512)
  • (backup): Configuration options (#511)
  • attribute sets for global configuration

Azure Plugin: Fixes

  • (lambda): align runtimes with latest updates

Refactorings

  • use local engine for testing (#294)

· 11 min read

Another year and another release is here. This release has mostly been housekeeping, with bug fixes and refactoring taking the charge in the changelogs. But it is always nice to have new toys, so we have also included some quality of life features as part of this release post.

· 15 min read

This release post is a long read for a couple of reasons, we missed doing one for the 8.2.0 release, and there have been a lot of new features since our last post.

Similar to our last post this release has focussed on the first 30 minutes of using hamlet, along with a collection of updates to make hamlet easy to use in your existing workflows.

· 15 min read

Another release another stack of changes! The main focus on this release has been on inputs, outputs and the first 30 minutes of using hamlet. The first 30 minutes when working with a new tool is a crucial point for users to see how something works and to get a feel for it. hamlet can be a bit intimidating to start off with so we wanted to reduce this as much as we could.

· 10 min read

Instead of defining the raw infrastructure we need, hamlet works off a solution. The solution outlines the functional components required in the deployment and hamlet then takes this to build out the infrastructure definition that would perform this function.

Combining this with our pluggable architecture we can use the same solution to generate infrastructure definitions in different cloud providers like AWS and Azure. This is an interesting feature itself but in this post we are going to look at another way to use this data, diagram generation.

· 12 min read

Static websites are everywhere nowadays, alongside them on their rise in popularity are the numerous app frameworks like Jekyll, Gatsby and Docusaurus that allow anyone to spin up and deploy documentation, a blog or simple website, often for for little or no cost.

· 11 min read

hamlet has its own opinions on how things should be done. These opinions are formed by our community based on best practice ( including the best way to workaround things ) to get you working in the cloud as efficiently as possible. Sometimes you might want to try out something new that we don't support or a component we offer isn't quite right. In this series of posts, Prototyping Deployments you should get an idea of how hamlet helps with these deployments and integrates with the rest of your solution.

· 17 min read

Welcome to our first release of 2021, we've been a bit late on our releases so this one has a lot of changes. We will touch on the highlights here and you can discover the rest in our new repo changelogs