Skip to main content

· 2 min read

Just a small release this time round. Mostly just focusing on some minor features and fixes for the AWS provider.

At this stage we are announcing that hamlet is entering maintenance mode, future development will focus on include specific features as required for our own purposes and fixes that effect deployments.

With the rise in other deployment tools in the world that have a much larger following than us its difficult to keep up, while its not good bye completely, we are heading in for a nap.

Key Updates

CDN Alias Configuration

Add explicit host name aliases that will be accepted by the CDN

  • (cdn): add support for configuring aliases (#2128)
  • cdn aliases (#789)

Fixed Subscription Configuration

Specify subscriptions to topics without needing an external link. This makes configuring email, phone and url based subscriptions easier.

  • (topic): add fixed subscription config
  • (topic): add fixed endpoint subscriptions (#786)

Extended support for S3 Bucket Replication

Updated the IAM role used in S3 bucket replication so that it can also be used by Batch Replication Jobs. When creating a batch replication job you can use the role that is created by the S3 bucket deployment for replication

  • (s3): batch replication support (#801)

AWS Provider

AWS Provider Features

  • (user): add permissions boundary arn config (#799)
  • add the cluster type when calling snapshot username (#796)
  • (baseline): add sqs extension (#792)
  • (db): add storage type configuration support (#790)
  • instance type support for lb
  • add support for kms replication of objects

AWS Provider Fixes

  • (cdn): restrict CDN region check lookup (#798)
  • (ec2): update ssh key env lookup (#797)
  • (ecs): policy split for ecs tasks (#795)
  • target group sg lookup
  • typo in templates
  • (ecs): memory and lb complex configuration (#794)
  • (correspondent): only deploy for right template type (#793)
  • (topic): kms permissions (#791)

· 5 min read

Key Updates

Removal of bash based client

hamlet started life as a collection of bash scripts with the freemarker template engine handling creating files. Over the years we've evolved the bash scripts into a cli that does still call some of these bash scripts but the main interface has been the cli for a little while now. In this release we have removed the bash scripts that aren't being used by the cli as well as the shim engine in the CLI which allowed you to access the bash scripts directly while still being able to use the engines we have in hamlet.

This shouldn't have an impact on most users of hamlet as most people found the cli the way to go for understanding how to get things going.

Along with that some of the commands in the cli which directly mapped to cli calls have also been removed as they have been replaced by other processes, mostly with the introduction of runbooks.

The following command groups have been removed:

  • hamlet release - Has now been replaced with run books

    hamlet release upload-image

    is now available as a run book. For the aws provider

    hamlet task run-runbook -n management-image_push-runbook -- Tier=app Component=myservice Reference=abc123 DockerImage=myimage:latest

    Will now push a local docker image tag up to the hamlet registry and update the image reference based on the occurrence details provided

    hamlet release update-build-reference

    This is completed as part of the runbook above but if you need to manually change the reference when you don't have an image as well

    hamlet task run-book -n management-image_set_reference-runbook -- Tier=app Component=myservice Reference=def456

    hamlet release transfer-image

    The transfer process has now been removed. Instead we have added a pull image command which you can use to pull an image locally then push back to another registry

    hamlet task run-runbook -n management-image_pull-runbook -- Tier=app Component=myservice

    Will pull down the image and provide you with details of the image that you can use to perform a push

  • hamlet manage - Has now been replaced with runbooks and the hamlet deploy commands

    hamlet manage stack
hamlet manage deployment

    have been replaced with the hamlet deploy run-deployments command

    hamlet manage crypto
hamlet manage file-crypto
hamlet manage credential-crypto

    Have all been replaced with the encrypt and decrypt runbooks

    hamlet task run-runbook -n management-kms_encrypt-runbook -- Value=mysecret
hamlet task run-runbook -n management-kms_decrypt-runbook -- Value=bXlzZWNyZXQK

    The value section will support multiline strings so you can cat a file into the value section and it will look after that for you

Changes

  • remove legacy bash scripts (#370)
  • legacy bash approach removal

Datapipeline component removal

The datapipeline component has now been removed. This was added to support the AWS Data Pipeline service which isn't really used anymore in favour of services like Step Functions. This was only available for AWS

Changes

  • (datapipeline): remove support for AWS data pipeline
  • remove datapipeline script support (#324)
  • remove datapipeline support (#1984)

Overall Updates

Engine

New Features

  • (waf): add support for waf on userpools
  • (user): add user name control
  • (user): source IP filtering
  • (lb): client IP control (#2119)
  • (vpcendpoint): policy support (#2117)
  • (apigateway): private APIs (#2115)
  • (mta): add stop after match action

Fixes

  • missing waf userpool attrs (#2125)
  • (userpool): add waf section that was lost (#2124)
  • don't list disabled suboccurrences (#2118)

Refactorings

  • (backups): default priority (#2123)

AWS

New Features

  • (waf): enable waf and add challenges (#784)
  • (user): name format handling
  • (vpcendpoint): source vpc endpoint extension (#777)
  • (user): source IP filtering
  • SQS and SNS endpoint policies (#775)
  • (ec2): lb fixed target mapping (#774)
  • (lb): client IP control (#773)
  • (iam): extend use of the large policy setup
  • (vpcendpoint): policy support (#764)
  • (apigateway): private APIs (#762)
  • (mta): stop after match

Fixes

  • (firewall): add both log destinations for all
  • (datafeed): support subset passes (#782)
  • (ecs): round max memory when calculated
  • (gateway): route table collection
  • add note for migration
  • ec2 sec groups and ecs ids
  • (gateway): duplicate route table ids (#770)
  • (gateway): duplicate route table ids (#769)
  • (account): disable cloudtrail by default
  • (objectsql): permissions state details
  • docker image tag extension (#765)
  • (cdn): extraneous resources in lg pass (#761)

Refactorings

  • (datafeed): check for undeployed lambda (#781)
  • (datafeed): undeployed lambda functions (#778)
  • remove use of component ids

Azure

Fixes

  • (ci): update to latest shared workflows (#313)
  • (computecluster): bootstraps lookup from global
  • remove use of getRegistryPrefix and EndPoint (#310)

Refactorings

  • replace reference lookups with function
  • standardise github workflows (#309)

CLI

Fixes

  • path to the sh script for set_provider_credentails task

· 12 min read

Key Updates

Image Management

With cloud native resources you often need to have your built application code artefact ready to be deployed with your infrastructure. hamlet provides the registry to manage the images and their versions as part of the deployment process. In this release we have added a new component called the image. Previously the image was part of your other components and you could share these images between your components using settings. The image component aims to make this a bit easier to follow and understand. The image represents the built code artefact and its current version. Other components can link to this image and they will use the code artefact from the image during there deployments.

This has a number of benefits over the current process:

  • You can deploy your image independent of the actual component. This saves having to run through the image deploy process if there is something stopping the component from building
  • You can define the image used by your component in the solution file. This makes it easy to see where the image is used

The image upload process has been moved over to the runbook approach with provider specific implementations. This means you can use the ones that hamlet provides by default or use your own if you need to.

To use the image runbook in your AWS solutions add the baseline runbook into your solution:

{
    "Segment" : {
        "Modules" : {
            "baseline": {
                "Provider": "aws",
                "Name": "baseline"
            }
        }
    }
}

After that you can list the runbooks from your segment to see the names of the available runbooks

hamlet task list-runbooks
| Name                                   | Description                                                          | Engine   |
|----------------------------------------|----------------------------------------------------------------------|----------|
| management-image_push-runbook          | Push an image to the hamlet registry and update the image references | hamlet   |
| management-image_pull-runbook          | Pull an image from the Hamlet registry                               | hamlet   |
| management-image_set_reference-runbook | Override the reference for an image                                  | hamlet   |

The runbooks accept the following inputs

The image component details

  • Tier: The tier of the image component
  • Component: The id of the image component
  • Version (Optional): the Version of the image in the solution
  • Instance (Optional): The instance of the image in the solution

The reference details for the image

  • Reference: The unique reference for this image - generally this is set to the git commit
  • Tag: A friendly reference for the image - generally a git tag

The path to the image locally

  • DockerImage (Optional): The name of a tagged docker image that you have in your local docker host
  • ImagePath (Optional): The path to a directory or zip file that contains the image code artefact

The runbook will determine the path to take for the image, upload it to the appropriate registry and then save the references as outputs to the CMDB

Changes

  • engine - (image): add docker image repository config
  • engine - (image): adds support for images in occurrence
  • engine - remove gen contract from stack outputs (#2089)
  • engine - cmdb write output for occurrences
  • engine - stackoutput generation and image upload runbooks (#2075)
  • engine - (account_s3): remove s3 registry initialisation (#2076)
  • engine - standardise image attributes
  • engine - add image settings setup
  • engine - remove image settings for env
  • aws - (ecr): extended repository configuration
  • aws - (runbooks): shorter names and pull image
  • aws - (images): add support for images on components
  • aws - (image): Adds aws image component
  • aws - (baseline): add image reference update runbook
  • aws - image copying from registry
  • aws - (image): include tag state from output (#745)
  • aws - (image): s3 path when pull image
  • aws - (image): source values (#741)
  • aws - (image): handle single level docker tags
  • aws - (images): output based reference handling (#689)
  • aws - (image): case handling for image sources (#681)
  • aws - (images): remove filename from CODE_SRC_PREFIX config for mobileapp component (#673)
  • aws - (image): update image push runbooks (#679)
  • aws - container image reference (#728)
  • bash - (image): Update scripts for image support

DB Proxy

Adds a subcomponent to the db component known as a db proxy. The DB Proxy is used to provided load balancing of connections across multiple database instances and perform DB specific connection multiplexing and connection handling.

Changes

  • engine - (db): add proxy subcomponent
  • aws - (db): add support for RDS Proxies (#678)

Cloud trail

For AWS we've added support for setting up Cloud Trail at the account level. This allows for logging management activities performed in AWS accounts to the current account level audit bucket.

Changes

  • aws - (cloudtrail): Add support for cloudtrail (#687)
  • engine - (cloudtrail): support for account cloudtrail (#2079)

Reduce Hamlet Cli dependencies

Hamlet has support for using some different cli based tools to test and validate templates before they are deployed. We previously bundled these cli tools into the package dependencies for the hamlet cli. This caused issues with large package sizes and created some conflicts especially around the aws cli.

So to make this easier we have now removed these dependencies from the hamlet cli and instead the testing will raise an error if they can't be found and let you know where you can find them

Changes

Extended WAF Support

Web Application Firewall support has been updated a lot in this release with a core focus on the AWS WAF service.

AWS WAFv2 was released some time ago now and while we had parallel support we wanted to start using some of the newer features that aren't available in WAF classic. We have now removed all support for WAF Classic and migrated everything to WAFv2. You don't need to update your existing WAF configuration to move to WAFv2, hamlet handles it for you.

After the removal of WAF Classic support was added for the following WAFv2 features:

  • Vendor managed rules - These are groups of rules which are managed by an external party ( including AWS). AWS provides a great set of base rules that reduce the work you need to do for basic rules.
  • Custom Responses - By default WAF will return a generic error page when it has blocked a request. Custom responses let you control this and return your own content and status codes. This can be managed for each rule as well
  • Label Matching - Rules when they are triggered can label requests for future processing. This match rule lets you provide customised configuration and overrides for rules added by vendors

Changes

  • engine - (waf): align inbuilt rules config
  • engine - (waf): add extended support for vendor rules
  • engine - (waf): add support for custom responses
  • engine - (waf): remove rule tuples and add error
  • engine - (waf): remove version control option
  • aws - (waf): label matching and vendor overrides
  • aws - support for custom body block responses
  • aws - (waf): add support for regex pattern set
  • aws - oversize handling and method fieldtotest (#685)
  • aws - (waf): remove wafv1 support
  • aws - (lb): remove waf version lookup on lb (#709)
  • aws - remove version on setupWAFRule call (#708)
  • aws - remove version from waf rule lookup (#707)
  • bash - (waf): remove older waf logging setup (#365)

AWS Best Practice Security Controls

In this release we had an active deployment which was deployed to an AWS environment that used AWS Security Hub to monitor aws resource security best practices.

Security hub reviews AWS resources using AWS Config and provides best practice recommendations on how you can improve security for your deployments.

Key focus areas we focussed on to comply with these best practices were:

  • Encryption at rest for ES (OpenSearch), SQS ad SNS
  • Encryption in transit enforcement for ES (OpenSearch), S3 and SQS
  • Logging for ES (OpenSearch) and RDS
  • Policy lockdowns on S3 buckets
  • ECS ReadOnlyRootFileSystem support. This also required supporting sidecar containers with dependencies between containers and control of their startup. This was required to set file/folder permissions when mounting a volume when write access is required

Changes

  • engine - (computeprovider): set available providers
  • engine - (network): support for network profile on net
  • engine - (ecs): task service limits, readonlyds, depend
  • engine - (db): add support for configuring db logging
  • engine - (es): encryption and hostname control
  • engine - (sqs): add support for encryption at rest
  • engine - (audit): enforce https for s3 connections
  • engine - add support for deployment locks
  • engine - (s3): add support for transit encryption
  • engine - (db): make enhanced monitoring the default
  • engine - (account): add aws es log access policy (#2077)
  • aws - (ecs): control if ec2 asg should be created
  • aws - (network): support default sg nacl control
  • aws - (baseline): cmk access for cloudwatch service
  • aws - (ecs): task cpu, depends on, readonly
  • aws - (rds): add support for cloudwatch log export (#719)
  • aws - (es): extend encrypotion config and custom endpoints (#718)
  • aws - (sqs): add support for enabling SSE on sqs
  • aws - (s3): add support for in-transit https policy
  • aws - (cloudtrail): Add support for cloudtrail (#687)
  • aws - (iam): add support for not actions on policies (#688)
  • aws - (baseline): permissions for logging
  • aws - (baseline): data bucket object ownership
  • aws - (sqs): typo in sqs encryption policy
  • aws - else statement for network acl creation
  • aws - handle cmk based encryption at rest (#725)
  • aws - (ecs): handle secrets on ec2 tasks (#692)
  • aws - (s3): allow external policy sharing on public
  • aws - (es): logging configuration (#684)
  • aws - (es): log group setup for occurrence (#683)
  • aws - (baseline): bucket policy extensions (#757)
  • aws - (baseline): bucket policy
  • aws - (s3): object ownership support (#732)
  • aws - update policy for aws service to cmk

Overall Hamlet changes

Below are the minor changes and fixes made across hamlet overall.

Engine

Full set of changes: 8.8.2...latest

Engine New Features

  • private endpoint services (#2106)
  • (userpool): add schema enable attributes
  • (lb): add support for multiple paths (#2102)
  • (mta): add support for bouncing emails
  • (cdn): include body for event handler function
  • add support for allExcept on cachepolicy (#2096)
  • (acct): remove the credentials and code bucket
  • (audit): add support for replication (#2080)
  • (cloudtrail): support for account cloudtrail (#2079)
  • (userpool): control enabling hostedUI
  • (attributeset): add enable for scaling policy

Engine Fixes

  • (runbook): multiple conditions (#2110)
  • (cache): syntax for retention period
  • update release package version

Engine Refactorings

  • use reference lookup methods over global
  • (s3): object ownership support (#2104)
  • (lambda): node 18 run-time (#2092)
  • standardise github actions

AWS

Full set of changes: 8.8.2...latest

AWS New Features

  • (userpool): enable support for schema attr
  • add support for multiple paths on lb (#729)
  • (lb): add support for multi value in lb lambda (#727)
  • (mta): Add support for bounces on receive (#724)
  • (cdn): include body for lambda@edge
  • add support for allExcept on cachepolicy (#721)
  • add smtp endpoint address to send mta (#715)
  • (module): add ses send events to service logs
  • (module): add ses based mail sender with log (#712)
  • (secretstore): add descriptions on secrets (#703)
  • remove the user defined boostrap process
  • (ecs): add support for container insights
  • (userpool): define email from address (#696)
  • (s3): add support for replication v2 (#690)
  • (userpool): control hosted ui setup (#686)
  • (ecs): propagate service tags to tasks (#682)

AWS Fixes

  • (secretstore): handle missing cmk
  • (api): build info sourcing (#753)
  • (s3): inbound link permissions for cdn
  • (cdn): link to cachepolicy
  • (api): image registry type access
  • (api): spec download logic
  • (apigateway): image source type checking
  • (baseline): default file path for image pull
  • map for ipset
  • (networking): handling of missing port on acl
  • (baseline): provide correct image for pull (#733)
  • readonly attribute assignment
  • lb path for state attribute (#730)
  • (elasticache): Use number based logic for retention (#720)
  • (baseline): add extra policies for cmk
  • (lb): backend support for lambda (#716)
  • (module): update link to basline component (#714)
  • (baselinedata): policy lookup on suboccurrence (#695)
  • (s3): add delete marker replication handling (#693)
  • athena s3 policy (#691)
  • (dnszone): add domain configuration if setup (#680)
  • update shared release workflow version
  • bugfix for aurora scaling
  • (lambda): remove env vars for lambda@edge

AWS Refactorings

  • replace reference with function lookups
  • (mta): move to using cfn for config set
  • allow manual trigger of release
  • github actions
  • (images): Update testing

Azure

Full set of changes: 8.8.0...latest

Azure Fixes

  • (ci): update to latest shared workflows (#313)
  • (computecluster): bootstraps lookup from global
  • remove use of getRegistryPrefix and EndPoint (#310)

Azure Refactorings

  • replace reference lookups with function
  • standardise github workflows (#309)

Bash

Full set of changes: 8.8.2...latest

Bash Fixes

  • (expo): remove ITMSTRANSPORTER_FORCE_ITMS_PACKAGE_UPLOAD override (#368)
  • (setup): handle multiple remote ref matches (#367)
  • repo management return code handling (#366)
  • update release package version
  • force ITMS package upload (#362)
  • job name

· 8 min read

Key Updates

Expo Mobile App Updates

With recent changes to both expo and the release of iOS16 we have made the usual annual changes to catch up with the latest changes.

  • Expo SDK46+ changed the way that OTA updates are published and we have updated the build process to support this in hamlet
  • Support has been updated around fastlane and the use of the Transporter app for mobile updates to TestFlight
  • All configuration for signing, publishing and controlling the mobile apps has been moved out of settings into solution based configuration. This makes the configuration available from our docs site and added defaults and validation to make sure it works as expected

bash

  • (expo): update cli usage for SDK 46+
  • (expo): testing updates for expo publish (#350)
  • workaround for fastlane missing support for transporter move
  • make search for signing more specific
  • include symbols disable and bitcode
  • disable signing on 3rd party libraries
  • ensure manual signing enabled on pod project
  • (expo): sdk differences in updates
  • (expo): use right app version when picking cli
  • (expo): align publish script with engine
  • (expo): reduce command line options
  • deprecate turtle and move to prebuild process
  • (expo): reinstate archive upload for sourcemaps (#349)

engine

  • (mobileapp): remove default on expo id override (#2045)

aws

  • (mobileapp): lookup for firebase properties (#641)
  • (mobileapp): testing updates

python

  • remove deprecated expo_app_publish param

AWS S3 Public Access Block By default

Inline with best practice security and those pesky news reports we have now enabled public access blocking across S3 buckets by default.

  • ACL based public access for objects can not be enabled by default within the hamlet aws implementation
  • To use policy based public access you can use the S3 component public access which will allow for bucket policy updates
  • The App Public permission has been removed which created an automatic public share on the ops data bucket. To ensure limited exposure we recommend creating an explicit S3 bucket for public access

engine

  • (s3): block public access on account s3

aws

  • (s3): disable public access by default
  • remove app public permissions attribute
  • remove app public attributes

Data Catalog Component

This new component represents services like AWS Glue which are used to provide a catalog of data sources that used across a system along with a centralised schema reference point. The focus of the initial release was to add support for creating a data catalog of AWS Service logs which are stored in S3.

Details on the module are available in PR #647) but essentially if you need to access any AWS services used in hamlet that log to S3 ( cdn/CloudFront, lb/Application Load Balancer) you can add the module and get the logs added into Glue and can make queries of the data using Athena

core

  • (datacatalog): add new component (#2049)
  • (datacatalog): initial support using Glue

aws

  • (module): add aws s3 service log datacatalog (#647)
  • (catalog): add base testing
  • SerDe naming

WAF Rule Engines

Web Application Firewalls offered by Cloud Providers have been extended to support more functionality including managed rules, rate limiting and more complex matching capabilities. In this release we have extended the support in hamlet to support these different WAF rules with the idea of a Rule engine. The engine is how a given rule is processed.

The engines included in this release are:

  • Rate Based: IP based tracking of requests and the enforcement of a rate limit for a given client IP
  • Vendor Managed: Rules that can be added to your WAF that are managed by a third party vendor. These rules are generally updated to align with the latest security incidents or over collective knowledge and automated processes developed by the vendor

core

  • (waf): add rule engines
  • (wafrule): engine type spelling (#2058)
  • (waf): cleanup of shared provider setup

aws

  • (waf): add support for more wafv2 rules
  • (waf): various fixes in WAF Handling

Extended CDN Support for Complex Scenarios

The CDN component has been updated to support more complex scenarios when dealing with distributions that perform complex routing or have shared caching policies across different parts of a site

This included the addition of some new subcomponents for the CDN:

  • origin is a new subcomponent which represents a single origin that can be consumed by multiple routes
  • response policy adds support for CDN providers which can modify things like response headers as they are returned to a client. For example, AWS CloudFront can now inject common security headers like HSTS or CORS if the origin doesn't include them
  • cache policy CDN Cache Policy defines a caching policy which can be used across multiple routes to standardise how caching is performed

You don't need to use these additional components if you don't want to and all existing support for controlling these options via the route are still in place.

core

  • (cdn): enable/disable error response overrides
  • (cdn): extended support for managing CDNS
  • (cdn): Response Policy Child reference

aws

  • (cdn): origin request policy headers (#657)
  • add default header policy for placeholders
  • (cdn): add enable/disable for error responses
  • (cloudfront): complex cdn scenarios
  • (cdn): use cloudformation to find cdn id
  • (cdn): add type checks and fix resource name
  • (cdn): add testing and fixes
  • (cdn): Redirect processing (#643)

Dynamic Values

IP Address Group resolution to CIDR ranges has been added as a dynamic value source, while we support IPAdressGroups across components that use IP Access control. The dynamic value support lets you provide network details to container env vars for example.

Now that we have dynamic values it is easier to add composite values which use a dynamic value and some fixed content. With that in mind we've added callback and logout url solution level configuration to the userpool

core

  • (dynamicvalue): add IP Address group support
  • (userpool): add support callback urls solution

aws

  • (userpool): add solution callback urls

Logging Enabled by Default

For all components the default behaviour is now to enable logging if we can. While this could add some additional costs when deploying to cloud providers, the cost of regret when you forget to enable logging and need it in an incident is much higher. So we turn it on now to help support your deployments.

core

  • enable logging by default
  • (network): add prefix control for flow logs

aws

  • (network): add control over flowlog prefix (#648)

Overall Hamlet changes

Below are the minor changes and fixes made across hamlet overall.

Bash Executor

Full set of changes: 8.7.0...latest

New Features

  • support cliv2 kms encrypt (#345)

Fixes

  • add an empty suffix for running on MacOS (#353)
  • update sourceMappingURL value after bundle rename
  • adjust sentry url prefix react-native apps (#351)

Engine

Full set of changes: 8.7.0...latest

New Features

  • (globaldb): add enable support for indexes
  • migrate tiers and solution setup into configuration data approach
  • district based solutions
  • add support for base attribute set expansion
  • add support for links as image sources
  • extend missing link detail error message

Fixes

  • (dynamicvalue): correct the output for tasks (#2057)
  • set tier indexes for tiers not in network
  • tier network configuration (#2046)
  • (dynamicvalues): support objects in replace (#2043)
  • nested expansion of attributes (#2040)
  • split expand base configuration (#2039)
  • set fixed index for shared tier
  • index handling for network ordering
  • tier lookup id name
  • configuration loading
  • update component settings for dynamic values

Refactorings

  • dynamic value handling (#2042)
  • add apigw certificate
  • move AV configuration to shared

aws

Full set of changes: 8.7.0...latest

New Features

  • (cw): ensure lambda is deleted with canary
  • (globaldb): add support for enabling indexes

Fixes

  • engine case for rule lookup
  • (network): handle missing subnet lookup
  • (s3): oai permissions (#653)
  • (windows): logContent setup for windows logging
  • (ecs): handle container tagging for ecs version
  • dynamic value setup for aws secrets
  • smtp user permissions in module
  • (filetransfer): log group name for subscription
  • fall through on missing network
  • remove redundant line from script
  • (image): use container repository for images
  • (ecs): skip lb processing when no lb port
  • (efs): correct tag format for access points
  • (cloudwatch): dependencies on subscription
  • (dnszone): add deployment subset check
  • typo in test module
  • (spa): cdn reference for path
  • (cw): update permissions for cw logs to kinesis (#634)
  • (lambda): check deployment units on function

Refactorings

  • (network): checks for networked tiers (#645)
  • (mobileapp): build configuration updates
  • remove public app data prefixes
  • (network): remove use of segmentObject
  • (ec2): av migration to shared provider

Azure

Full set of changes: 8.7.0...latest

New Features

  • (network): manage vpn cipher configuration

Refactorings

  • use base level attribute sets (#307)

Core Engine

Full set of changes: [2.0.0, 8.6.0...latest](https://github.com/hamlet-io/engine-core/compare/2.0.0, 8.6.0...master)

Fixes

  • Dependency updates and updates to the bundled JRE version used

Python

Full set of changes: 9.23.0...master

Fixes

  • remove use of transport from httpx client

· 5 min read

This has been a pretty quiet release this round, a few helpful updates with bug fixes taking the front of stage. There is also some initial work in our image management process which will be progressed in our next release and includes using links for image sharing along with a more engine based approach to managing images.

Key Features

Extensions for more components

We've found that as solutions grow the need for minor tweaks on components are required. So we have added extensions to more components to allow for updates to resource policies and even the ability to add additional resources to a common component.

  • Core - (baseline): add extension support for data (#2028)
  • Core - add support for topic and queue extensions
  • AWS - (baseline): extension and policy support
  • AWS - queue topic permission extensions (#618)

Dynamic Value Evaluation

When runbooks were introduced they needed the ability to pass value references between steps. The solution to this was to add support for a format for referencing a dynamic value as part of another string. With this release we have extended this support to all components and introduced the ability to add different dynamic look up sources.

The syntax for a dynamic value is __<type>:<attribute>:<attribute>__ where type is the dynamic value provider type and a value provider provides attributes that are used in the lookup process.

Dynamic values can be used on any component string properties defined in the solution. They can be combined with other values in the same string as well. So you can have a url formatted like "https://__attribute:cdn:FQDN__" and when the dynamic value is replaced it will replace everything between __ ( included ) with the value that was looked up.

The initial release supports the following dynamic providers

  • attribute: look up an attribute of linked component
  • setting: use a setting of the component in its env var format
  • aws_secret: (AWS only) use CloudFormation Secrets manager integration to provide a secret value to CloudFormation

Adding more Enable controls across attributes

With support for qualification across the whole solution having the ability to define if something is enabled or not in solution configuration makes it easy to use qualification to control if that setting should be used in each environment. This release has seen the enabled option added to more component configuration options.

Component Updates

Lambda

Added support for defining aliases as fixed name references to a given version of a lambda function

Correspondent

Added support for configuring the channels used by correspondent to communicate with users

LB

AWS - Added support for using a lambda function as a backend to an application load balancer

Full Change Log

Core

New Features

  • (baseline): add extension support for data (#2028)
  • (port): add enable for port healthcheck (#2030)
  • (entrance): stackoutput entrance
  • add support for topic and queue extensions
  • tasks and extensions for images in runbooks
  • (image): add support for the image component
  • (lambda): lambda alias (#2015)
  • add support for dynamic value evaulation
  • (correspondent): add support for channel configuration
  • (lb): add enable support for conditions

Fixes

  • (dn): Test for missing domain name (#2008)
  • (dynamicinput): error message text (#2027)
  • set buildblueprint output file name (#2026)
  • schema entrance output filename (#2025)
  • typo for deployment test
  • (output): output prefix handling
  • typo in dynamicvalue return
  • (dynamicvalue): remove fatal on missing link
  • update naming for source
  • correct private key property name (#2017)
  • (dynamicvalue): fix typo in configuration scope
  • (dynamicvalue): resolve handling for lists (#2013)
  • comment update
  • handle empty values or object references

Refactorings

  • (runbook): support dynamic values in conditions
  • remove occurrence dependency and apply at the occurrence level

Others

  • (lambda): update lambda runtime versions (#2031)

AWS

New Features

  • (baseline): extension and policy support
  • (lb): lambda backend support for lb (#626)
  • queue topic permission extensions (#618)
  • (image): aws support for the image component
  • (runbook): add push_image runbook for images
  • adds extensions and tasks for image management
  • (lambda): lambda aliases (#606)
  • (dyanmicvalues): add support for aws secrets as dynamic values
  • (correspondent): add support for AWS pinpoint channels
  • (lb): add support for enabling/disbaling conditions

Fixes

  • incorporate feedback
  • (cert): raise error on invalid FQDN
  • support dns zone creation without network (#619)
  • (lb): expand permissions for lambda invoke (#629)
  • error messages for port lookups (#628)
  • (lb): protocol checks for nlb
  • (lb): action lookup for network load balancer (#623)
  • (computetask): windows directory creation (#622)
  • (directory): handle missing config connector
  • (healthcheck): add iam service to setup
  • (lb): create alerts across all occurrences
  • (waf): support v1 -> v2 migrations
  • (db): aurora cluster backups (#616)
  • (lambda): size checking (#609)
  • (apigateway): handle open ip address groups (#608)
  • (apigateway): reference versioned lambdas
  • (lb): standard error for invalid port mapping

Refactorings

  • (ecs): lg pass handling (#607)

Azure

Fixes

  • (lb): standard error for invalid port mapping

Bash

New Features

  • (expo): add support for app display name on IOS apps (#340)
  • add stackoutput entrance support
  • (expo): add support for firebase and plugins

Fixes

  • conventional commit support (#342)
  • incorporate feedback
  • incorporate feedback
  • (cli): disable paging for aws cli v2
  • (expo): include project details for plist (#341)

Refactorings

  • set credentials (#343)

· 10 min read

This release has been a pretty standard update with new components, tasks and general improvements, and bug fixes.

Nothing too drastic... but we have started putting together a list of larger changes that will be introduced in our next release.

Key Features

User Defined Tagging

You can now control the tags applied to your resources that have been deployed by hamlet. Tags have been broken up into two categories, Common and Additional.

  • Common tags are based on details of the occurrence that is being deployed. This includes the tags that we already provide in hamlet, and include things like environment, product, build reference and the generated name of the component
  • Additional tags are user defined tags. These work as a collection of key value pairs that you can apply as you need to. Using Deployment Profiles you can easily create a tagging policy across all of your deployments.

To use tags all components now have a Tags attribute

{
    "Tags": {
        "Additional": {},
        "Common": {
            "Component": "boolean",
            "CostCentre": "boolean",
            "Deployment": "boolean",
            "Layers": "boolean",
            "Name": "string",
            "Prefix": "string",
            "Solution": "boolean"
        }
    }
}

The additional tags are defined as objects underneath the Additional attribute

```json
{
    "MyKey" : {
        "Value" : "MyValue"
    }
}

Tag configuration has been added to all components, some providers don't support tags for particular resources so they might not be included on all resources.

  • tagging control for occurrences

Support for Additional Values

Component attributes now support setting additional values on top of the default values they offer. This is handy when values are passed from hamlet components directly to providers. If the provider adds support for a new value, for example a new AWS lambda runtime, you can use it straight away without waiting for us to add the new runtime into hamlet. This is currently enabled on the lambda component and will be added to other components as we identify use cases for this configuration option.

  • (lambda): chg constraint on runtime to warning
  • (lambda): remove constraint on runtime

New Components

A few new components made it into this release.

  • Load balancer backends This is a new subcomponent on load balancers that defines a group of backend servers independent of their Port Mappings. This is useful when you have a set of conditional rules that all use the same backend servers. Now you can define one backend and link the Port Mapping to the Backend.
  • Log Store Is an instance of log storage standalone from a component. You could use this when you have a lambda function that writes application specific logs and you want to maintain separation between the logs generated by the lambda function and the application logs.
  • Data Stream Similar to the data feed component which looks after forwarding data to a destination, the data stream handles collecting large amounts of data and queuing it for processing by clients to consume.
  • DocDB Adds support for deploying Document based NoSQL hosts. It's similar to the db component which provides the same support for SQL based database hosts
  • DNS Zone Public or private (network attached) dns zones can be defined through hamlet and uses the standard domain based configuration to assign the zone name.
  • Certificate Authority Used for building private PKI infrastructure, the certificate authority component represents a single authority. Through links between certificate authorities you can define an authority chain in line with best practices for PKI.

These components were implemented in the following changes:

  • (lb): define backends independent from port mappings
  • (logstore): add engine parameter support
  • (logstore): add dedicated log storage component
  • (datastream): define data stream component
  • add docdb support (#1934)
  • (dnszone): add support for network based configuration
  • add certificate authority component
  • aws logstore implementation (#573)
  • (datastream): aws implementation of data stream (#575)
  • add docdb support (#521)
  • (dnszone): add support for private vpc zones
  • (certificateauthority): intial support with ACMPCA

New Tasks

Runbooks are becoming more and more useful for running operational tasks on your deployments. In this release the aws provider has been updated with more tasks;

  • SES Password Generation This generates a sig4 signature of an AWS_SECRET_ACCESS_KEY that can be used to send emails through SES using the SMTP based API

  • S3 Empty Bucket Removes all objects and their versions from an S3 bucket

  • S3 Download Bucket Copies the contents of an S3 bucket to a local directory

  • Secrets Manager Get Secret Gets the value of a secret stored in AWS Secrets manager

  • add SES SMTP user module

  • (task): add ses smtp password generation task

  • add s3 runbook tasks

  • (task): add secrets manager get secret task

Policy Chunking for AWS

hamlet generates fine grained IAM permissions policies for each resource it deploys. This is great from a security best practice approach, however AWS does have limits on the size of an IAM policy. Some of our users were having issues with the generated policy hitting the policy size limits with complex deployments. To support this when generating IAM policy documents we split the permissions across multiple policies automatically.

  • policy chunking (#545)

Build Details

When working with images for application deployments its useful to know which version of an image is being used in a given environment. You can now find this quickly using the cli.

Running hamlet release list-image-references will provide a list of all the occurrences that have an image associated with them along with details of the image

hamlet --profile msw release list-image-references

| Occurrence                       | Deployment Unit   | Build Reference                          | Code Tag   | Image Format   | Image Source   |
|----------------------------------|-------------------|------------------------------------------|------------|----------------|----------------|
| management-lamb-test-function    | lamb              | def45678900000                           |            | lambda         |                |
| management-apphost-myapp-service | myapp             | d66500e2741196b75fc3aed24e35fddad42fa212 |            | docker         |                |
| management-apphost-mytask-task   | myapp             | d66500e2741196b75fc3aed24e35fddad42fa212 |            | docker         |                |

Event Logging

The logging used when hamlet saves commits to your cmdb has been extended to support a wider range of actions. Events are now stored in an event log stored in your local HAMLET_HOME_DIR. When changes to the CMDB are committed to a repository using the hamlet cmdb commit-changes command the events for the CMDB are collected from the log, formatted as commit messages and saved to the CMDB.

  • add HAMLET_EVENT_DIR
  • event logging for write events

Changes Required

This release had a couple of updates that will require some minor updates to cmdb solutions;

  • On firewall rules, the attribute Type has been renamed to RuleType. This is to fix an issue when the Type attribute was added to all components to set their component type
  • The Solution layer has been removed as it didn't make sense with what layers are intended for. This was also done to make way for being able to define solutions differently in future releases. Any configuration applied on the Solution object in your solutions should be moved to an appropriate layer, most likely the Segment or Environment layer
  • MultiAZ configuration has been moved from the Layers down to the components. A deployment profile was added to all layers that used the old multiAZ configuration option. You should check that your deployments are using MultiAZ deployment options if they need it
  • Backup configuration was removed from the datavolume component. To backup a datavolume we now recommend using the backup component which can handle backing up a much wider ranger of resources

Upcoming

Finally in this release we have deprecated direct usage of the bash executor as the cli should now support all the actions that were available from the bash executor. When running a deprecated command you will be given a warning that the command has been deprecated. If you have questions about which command replaces a deprecated command feel free to reach out in our GitHub discussions.

Full Change Log

Engine

New Features

  • (cdn): add priority to cdn routes
  • (lambda): versioned lambda retention policy (#1989)
  • (alerts): add enable attribute on alerts
  • (lb): define external or internal for lb endpoints
  • (lambda): provisioned executions (#1980)
  • (datavolume): zone control and remove backups
  • (cdn): add support for disabling event handlers
  • add support for HealthCheck Protocol
  • (ec2): add support for zone based deploy control

Fixes

  • add generation contract for unitlist view
  • (runbook): don't include disabled tasks
  • only include active layers based on district type
  • (account): minor fixes for account level aws deployments
  • support regions in state that use refs
  • minor version upgrade control
  • typo
  • typos in attributes

Refactorings

  • (apigateway): authorization models (#1995)
  • testing output handling and test profiles
  • move functions out of setContext
  • update testing to remove solution layer

AWS Engine Plugin

New Features

  • (vpc): support creating security groups without inbound ports
  • (lb): add support for lb backends
  • (datafeed): add support for data streams as a source
  • (cdn): rule priority sorting
  • (rds): event support (#569)
  • occurrence level configuration tagging
  • (lambda): sqs batch control
  • (lambda): versioned lambda retention policy (#574)
  • (alertslack): allow setting runtime with param (#570)
  • (alerts): add filter on alerts for enable status
  • (ec2): support IPAddress groups and LB on same port
  • only include enabled occurrences in suboccurrence processing
  • (lambda): provisioned executions (#559)
  • remove auto state generation for fixutre testing
  • (secretsmanager): add read write support fo secrets
  • (apigateway): mutual TLS attribute (#548)
  • ec2 resource outputs and replace updates
  • (cdn): only add enabled event handlers
  • (cdn): add error for wrong logging region
  • (lb): add support for alb as a network target (#537)
  • add role tag to components if present
  • (ec2): zone based control for instances

Fixes

  • (ecs): tags handling (#588)
  • handle missing link
  • (datastream): typo in attribute name
  • (globaldb): tag function call
  • (sqs): add dlqName back into setup routine
  • handle empty tag sets
  • add backup tags for dds
  • spelling in message
  • (s3): handle notifications for endpoints already deployed
  • casing for MulitAZ attribute
  • format json content for run task module
  • (ec2): fix ordering for cfn init commands
  • (ecs): ensure subnets are always treated as an array
  • (ecs): paramter types for templates
  • typo
  • (lb): use suboccurrence for static forwardning
  • testing updates (#556)
  • various updates from testing
  • log and account processing
  • include Value in getReference
  • (computecluster): general fixes
  • (iam): inline policy creation from policy set (#547)
  • (backupstore): tag based conditions
  • typo in message

Refactorings

  • (apigateway): authorization models (#581)
  • (network): remove baseline components that aren't required
  • network subnet function
  • move test module loading to product layer
  • (iam): limits used for policy splitting (#549)
  • (datavolume): zone filter support for volume mounts

Azure Engine Plugin

  • move test module loading to product layer

Full set of changes: 8.5.0...8.6.0

Bash Executor

New Features

  • detect existing images (#326)
  • add entrance output dir for imagedetails

Fixes

  • spelling in messages
  • three level context properties (#327)

Refactorings

  • sort template objects
  • scope settings search and remove legacy
  • remove deprecated integrator commands
  • remove blueprint doc generation
  • remove reference to slices
  • automation setContext updates

· 9 min read

With our release process bedding in, we've been able to speed up our release process. So here we are with another release with some minor but useful fixes and features.

Key Features

Runbook Tasks for existing processes

In the 8.4 release we added the initial support for RunBooks. They provide a way to run operational tasks on your deployments using the hamlet CLI. Previously this was implemented through dedicated bash scripts for a specific task, in this release we have now replicated these scripts as runbook tasks and included some modules to help with setting them up

  • AWS Module runbook_rds_snapshot adds support for a runbook task that will take a snapshot of a db instance.
  • AWS Module runbook_run_task starts a container task with a provided command and waits for the task to exit. You can provide environment or command overrides as part of running the runbook or provide fixed values. This is useful when you want to run a configuration task such as database migrations as part of your deployment pipelines
  • The baseline AWS module has been introduced which adds runbooks for encrypting and decrypting content using the baseline KMS key.

Each of these modules provides our recommended approaches to using the runbook tasks. You can also use any of the tasks included in these modules as part of your own runbooks

  • engine: (tasks): add basic tasks
  • engine: (ecs): add support for default task compute provider
  • engine: (tasks): extend ssh tasks and add bash command
  • engine: (ecs): add initprocess support for containers
  • aws: add additional runbook tasks and modules
  • aws: add run ecs task support for runbooks
  • aws: add baseline encryption module
  • aws: (ecs): support for ecs exec
  • aws: extended runbooks for access

Backup Components

Backups have now been mapped in to hamlet components with the backupstore and backupsstoreregime components.

The store represents a backup store that will host the backups that have been made. Each store has regimes that define the resources that should have backups in the store. The regime uses link based processing or layer context rules to define your backups.

This allows for a really simple setup for backups if you just want to backup anything that supports backups

Using a component such as

{
    "backup" : {
        "Type" : "backupstore",
        "Lock" : {
            "Enabled" : true,
            "MinRetention" : 30,
            "MaxRetention" : 365
        },
        "Regimes" : {
            "allComponents" : {
                "Targets" : {
                    "All" : {
                        "Enabled" : true
                    }
                },
                "Rules" : {
                    "nightly" : {
                        "Schedule" : {
                            "Expression" : "cron(0 1 * * *)"
                        }
                    }
                },
                "Conditions" : {
                    "MatchesStore" : {
                        "Enabled" : true,
                        "Product" : true,
                        "Environment" : true,
                        "Segment": true,
                        "Tier" : false
                    }
                }
            }
        }
    }
}

I can create a backup store that will backup any resources in my segment, store backups for a minimum of 30 days and a maximum of 365 days. Any new resources added to the segment should be automatically detected and added to the backup schedule. This has been implemented in AWS using AWS Backup. Hamlet helps with the backup setup as we define the tagging structure applied to all components, so we can create simple backup rules using this standardised tagging structure.

  • engine: (backup): Backup support (#1921)
  • engine: (backup): Configuration options (#1926)
  • engine: backup encryption key (#1927)
  • aws: (backup): Initial AWS implementation (#507)

Updates to the CI/CD container extensions

The hamlet and dind extensions have been updated to align with the latest configuration environment variables used in the hamlet CLI;

  • AWS_AUTOMATION_USER has been replaced by the HAMLET_AWS_AUTH_SOURCE and HAMLET_AWS_AUTH_USER settings

  • When using dind as a docker host TLS verification is required at all times

  • Use of the rexray/ebs plugin for docker has been removed as the plugin no longer seems to be maintained and is failing on newer ec2 instances. The recommendation now is to use local volumes and extend the size of your docker volume

  • add support for using local docker volumes for builds

  • refactor hamlet and dind docker extensions

Log Encryption at Rest

Mostly an AWS function but we have added support for enabling log group encryption at rest across all components which create log groups. This is an extension of the LoggingProfile that can be applied across all components to ensure that log encryption is enabled. Adding this in the logging profile makes it an easy set and forget option to enable encryption for logs at rest whenever a new log group is created.

  • engine: (legacy): add encryption at rest support for logs
  • engine: add account layer control over logging
  • engine: add logging profile support for encryption at rest
  • aws: (logs): add support for at rest encryption of cw logs

Engine Input Sanitisation

The underlying engine that we use for hamlet is the freemarker engine, it doesn't have great support for null values imported from JSON objects. As part of our input processing we now strip any null values and their keys from the JSON object.

To help understand how inputs are processed within the engine we now have a new entrance called an inputinfo entrance. This lists all of the input stages and how the processing is assembled for different input providers

  • add inputinfo entrance
  • add null value cleaner to input stages

Bundled Core Engine

The freemarker engine is built as a Java based application that we call as part of the hamlet CLI. This is a bit of a surprise for someone starting with hamlet especially when using our python base CLI. So to help with that the core engine is now distributed with its own Java installation as a bundled command line tool. While this does increase the size of the engines as they are downloaded it does make it a lot easier to install and get going with hamlet.

In this update we have also moved from Java8 to Java11 to catch up with the latest Java releases.

Component Updates

The updates below are the key change log updates from each of the hamlet engine components

Engine: New Features

  • max-age control on bucket content (#1953)
  • (adaptor): add support for alert configuration
  • full names based on district
  • (directory): support for log forwarding
  • (datafeed): compression control for buckets
  • (globaldb): add alerts support
  • (ses): add control over IP access policy
  • (cdn): add origin connection timeout support

Engine: Fixes

  • include component settings in environment

  • district type for district lookup

  • include account in mock filter layers

  • encryption of logs for sms

  • use raw path for settings path prefix

  • (apigateway): handling of null values in definitions

  • prefix handling for shared provider

  • (lb): add required logging profile to lb (#1925)

  • module lookup process

  • (cdn): add logging profile

  • (cd): engine install

  • domain parent handling

Engine: Refactorings

  • asfile ordering (#1955)
  • use local engine definition for testing
  • update deployment group district handling
  • rename district to district type
  • remove task containers from shared provider
  • consolidate link functions
  • remove type based attributes from healthchecks
  • (s3): use recommended process for bucket policy
  • (district): use attributeset for config
  • more specific name part config
  • bootstrap clo processing
  • Account descriptions and placement profiles (#1916)
  • attributesets for components
  • move domain and certificate to reference data

Engine: Others

  • update runtimes for lambda environment

Bash Executor: New Features

  • add support for bundled freemarker wrapper

  • handling of bundled vs jar wrapper

  • add support for providing the district CLI option to templates

  • add docdb support (#312)

Bash Executor: Fixes

  • district default handling
  • sentry release CLI support
  • make output dir for binary only
  • macos run_id and expo ouputs

Bash Executor: Refactorings

  • district rename to district type
  • use weird seperate for user provided paramters

AWS Plugin: New Features

  • use local engine setup for testing
  • (spa): force max-age for config (#530)
  • (adaptor): adaptor alert support
  • (mta): add enable/disable handling on rules
  • (directory): log forwarding support (#517)
  • (s3): backup support (#516)
  • (kinesis): compression support for firehose
  • (globaldb): cloudwatch alarms (#508)
  • (cdn): add support for origin connection timeouts

AWS Plugin: Fixes

  • segment seed fixture value (#535)
  • region lookup for resources
  • (healthcheck): add more entropy to naming of health checks
  • (db): secret lookup for engine setup
  • typo in module
  • (healthceheck): testing changes from type to engine
  • remove use of isPresent for AV setup
  • ipmatch and geomatch for wafv2 (#518)
  • (db): aurora cluster updates
  • (task): kms encrypt parameters
  • efs mount script formatting
  • (db): ingress security group id
  • (sns): add support for encrypted topics
  • (lb): logging profile for WAF logs (#510)
  • (cdn): missing logging profile for waf logging
  • (cdn): logging script for wafv1
  • clean up old if statement
  • (s3): replication validation checking

AWS Plugin: Refactorings

  • align the run task module to task
  • update ecs task configuration after testing
  • move to latest unicycle install process
  • update district to district type on group filter (#534)
  • move ecs container setup to aws provider
  • update iam standard policy name
  • (iam): standard policies for app components
  • (s3): use references for bucket policy
  • backup encryption key (#512)
  • (backup): Configuration options (#511)
  • attribute sets for global configuration

Azure Plugin: Fixes

  • (lambda): align runtimes with latest updates

Refactorings

  • use local engine for testing (#294)

· 11 min read

Another year and another release is here. This release has mostly been housekeeping, with bug fixes and refactoring taking the charge in the changelogs. But it is always nice to have new toys, so we have also included some quality of life features as part of this release post.

· 15 min read

This release post is a long read for a couple of reasons, we missed doing one for the 8.2.0 release, and there have been a lot of new features since our last post.

Similar to our last post this release has focussed on the first 30 minutes of using hamlet, along with a collection of updates to make hamlet easy to use in your existing workflows.

· 15 min read

Another release another stack of changes! The main focus on this release has been on inputs, outputs and the first 30 minutes of using hamlet. The first 30 minutes when working with a new tool is a crucial point for users to see how something works and to get a feel for it. hamlet can be a bit intimidating to start off with so we wanted to reduce this as much as we could.