Skip to main content

Engine Release 8.9

ยท 12 min read

Key Updates

Image Management

With cloud native resources you often need to have your built application code artefact ready to be deployed with your infrastructure. hamlet provides the registry to manage the images and their versions as part of the deployment process. In this release we have added a new component called the image. Previously the image was part of your other components and you could share these images between your components using settings. The image component aims to make this a bit easier to follow and understand. The image represents the built code artefact and its current version. Other components can link to this image and they will use the code artefact from the image during there deployments.

This has a number of benefits over the current process:

  • You can deploy your image independent of the actual component. This saves having to run through the image deploy process if there is something stopping the component from building
  • You can define the image used by your component in the solution file. This makes it easy to see where the image is used

The image upload process has been moved over to the runbook approach with provider specific implementations. This means you can use the ones that hamlet provides by default or use your own if you need to.

To use the image runbook in your AWS solutions add the baseline runbook into your solution:

{
    "Segment" : {
        "Modules" : {
            "baseline": {
                "Provider": "aws",
                "Name": "baseline"
            }
        }
    }
}

After that you can list the runbooks from your segment to see the names of the available runbooks

hamlet task list-runbooks
| Name                                   | Description                                                          | Engine   |
|----------------------------------------|----------------------------------------------------------------------|----------|
| management-image_push-runbook          | Push an image to the hamlet registry and update the image references | hamlet   |
| management-image_pull-runbook          | Pull an image from the Hamlet registry                               | hamlet   |
| management-image_set_reference-runbook | Override the reference for an image                                  | hamlet   |

The runbooks accept the following inputs

The image component details

  • Tier: The tier of the image component
  • Component: The id of the image component
  • Version (Optional): the Version of the image in the solution
  • Instance (Optional): The instance of the image in the solution

The reference details for the image

  • Reference: The unique reference for this image - generally this is set to the git commit
  • Tag: A friendly reference for the image - generally a git tag

The path to the image locally

  • DockerImage (Optional): The name of a tagged docker image that you have in your local docker host
  • ImagePath (Optional): The path to a directory or zip file that contains the image code artefact

The runbook will determine the path to take for the image, upload it to the appropriate registry and then save the references as outputs to the CMDB

Changes

  • engine - (image): add docker image repository config
  • engine - (image): adds support for images in occurrence
  • engine - remove gen contract from stack outputs (#2089)
  • engine - cmdb write output for occurrences
  • engine - stackoutput generation and image upload runbooks (#2075)
  • engine - (account_s3): remove s3 registry initialisation (#2076)
  • engine - standardise image attributes
  • engine - add image settings setup
  • engine - remove image settings for env
  • aws - (ecr): extended repository configuration
  • aws - (runbooks): shorter names and pull image
  • aws - (images): add support for images on components
  • aws - (image): Adds aws image component
  • aws - (baseline): add image reference update runbook
  • aws - image copying from registry
  • aws - (image): include tag state from output (#745)
  • aws - (image): s3 path when pull image
  • aws - (image): source values (#741)
  • aws - (image): handle single level docker tags
  • aws - (images): output based reference handling (#689)
  • aws - (image): case handling for image sources (#681)
  • aws - (images): remove filename from CODE_SRC_PREFIX config for mobileapp component (#673)
  • aws - (image): update image push runbooks (#679)
  • aws - container image reference (#728)
  • bash - (image): Update scripts for image support

DB Proxy

Adds a subcomponent to the db component known as a db proxy. The DB Proxy is used to provided load balancing of connections across multiple database instances and perform DB specific connection multiplexing and connection handling.

Changes

  • engine - (db): add proxy subcomponent
  • aws - (db): add support for RDS Proxies (#678)

Cloud trail

For AWS we've added support for setting up Cloud Trail at the account level. This allows for logging management activities performed in AWS accounts to the current account level audit bucket.

Changes

  • aws - (cloudtrail): Add support for cloudtrail (#687)
  • engine - (cloudtrail): support for account cloudtrail (#2079)

Reduce Hamlet Cli dependencies

Hamlet has support for using some different cli based tools to test and validate templates before they are deployed. We previously bundled these cli tools into the package dependencies for the hamlet cli. This caused issues with large package sizes and created some conflicts especially around the aws cli.

So to make this easier we have now removed these dependencies from the hamlet cli and instead the testing will raise an error if they can't be found and let you know where you can find them

Changes

Extended WAF Support

Web Application Firewall support has been updated a lot in this release with a core focus on the AWS WAF service.

AWS WAFv2 was released some time ago now and while we had parallel support we wanted to start using some of the newer features that aren't available in WAF classic. We have now removed all support for WAF Classic and migrated everything to WAFv2. You don't need to update your existing WAF configuration to move to WAFv2, hamlet handles it for you.

After the removal of WAF Classic support was added for the following WAFv2 features:

  • Vendor managed rules - These are groups of rules which are managed by an external party ( including AWS). AWS provides a great set of base rules that reduce the work you need to do for basic rules.
  • Custom Responses - By default WAF will return a generic error page when it has blocked a request. Custom responses let you control this and return your own content and status codes. This can be managed for each rule as well
  • Label Matching - Rules when they are triggered can label requests for future processing. This match rule lets you provide customised configuration and overrides for rules added by vendors

Changes

  • engine - (waf): align inbuilt rules config
  • engine - (waf): add extended support for vendor rules
  • engine - (waf): add support for custom responses
  • engine - (waf): remove rule tuples and add error
  • engine - (waf): remove version control option
  • aws - (waf): label matching and vendor overrides
  • aws - support for custom body block responses
  • aws - (waf): add support for regex pattern set
  • aws - oversize handling and method fieldtotest (#685)
  • aws - (waf): remove wafv1 support
  • aws - (lb): remove waf version lookup on lb (#709)
  • aws - remove version on setupWAFRule call (#708)
  • aws - remove version from waf rule lookup (#707)
  • bash - (waf): remove older waf logging setup (#365)

AWS Best Practice Security Controls

In this release we had an active deployment which was deployed to an AWS environment that used AWS Security Hub to monitor aws resource security best practices.

Security hub reviews AWS resources using AWS Config and provides best practice recommendations on how you can improve security for your deployments.

Key focus areas we focussed on to comply with these best practices were:

  • Encryption at rest for ES (OpenSearch), SQS ad SNS
  • Encryption in transit enforcement for ES (OpenSearch), S3 and SQS
  • Logging for ES (OpenSearch) and RDS
  • Policy lockdowns on S3 buckets
  • ECS ReadOnlyRootFileSystem support. This also required supporting sidecar containers with dependencies between containers and control of their startup. This was required to set file/folder permissions when mounting a volume when write access is required

Changes

  • engine - (computeprovider): set available providers
  • engine - (network): support for network profile on net
  • engine - (ecs): task service limits, readonlyds, depend
  • engine - (db): add support for configuring db logging
  • engine - (es): encryption and hostname control
  • engine - (sqs): add support for encryption at rest
  • engine - (audit): enforce https for s3 connections
  • engine - add support for deployment locks
  • engine - (s3): add support for transit encryption
  • engine - (db): make enhanced monitoring the default
  • engine - (account): add aws es log access policy (#2077)
  • aws - (ecs): control if ec2 asg should be created
  • aws - (network): support default sg nacl control
  • aws - (baseline): cmk access for cloudwatch service
  • aws - (ecs): task cpu, depends on, readonly
  • aws - (rds): add support for cloudwatch log export (#719)
  • aws - (es): extend encrypotion config and custom endpoints (#718)
  • aws - (sqs): add support for enabling SSE on sqs
  • aws - (s3): add support for in-transit https policy
  • aws - (cloudtrail): Add support for cloudtrail (#687)
  • aws - (iam): add support for not actions on policies (#688)
  • aws - (baseline): permissions for logging
  • aws - (baseline): data bucket object ownership
  • aws - (sqs): typo in sqs encryption policy
  • aws - else statement for network acl creation
  • aws - handle cmk based encryption at rest (#725)
  • aws - (ecs): handle secrets on ec2 tasks (#692)
  • aws - (s3): allow external policy sharing on public
  • aws - (es): logging configuration (#684)
  • aws - (es): log group setup for occurrence (#683)
  • aws - (baseline): bucket policy extensions (#757)
  • aws - (baseline): bucket policy
  • aws - (s3): object ownership support (#732)
  • aws - update policy for aws service to cmk

Overall Hamlet changes

Below are the minor changes and fixes made across hamlet overall.

Engine

Full set of changes: 8.8.2...latest

Engine New Features

  • private endpoint services (#2106)
  • (userpool): add schema enable attributes
  • (lb): add support for multiple paths (#2102)
  • (mta): add support for bouncing emails
  • (cdn): include body for event handler function
  • add support for allExcept on cachepolicy (#2096)
  • (acct): remove the credentials and code bucket
  • (audit): add support for replication (#2080)
  • (cloudtrail): support for account cloudtrail (#2079)
  • (userpool): control enabling hostedUI
  • (attributeset): add enable for scaling policy

Engine Fixes

  • (runbook): multiple conditions (#2110)
  • (cache): syntax for retention period
  • update release package version

Engine Refactorings

  • use reference lookup methods over global
  • (s3): object ownership support (#2104)
  • (lambda): node 18 run-time (#2092)
  • standardise github actions

AWS

Full set of changes: 8.8.2...latest

AWS New Features

  • (userpool): enable support for schema attr
  • add support for multiple paths on lb (#729)
  • (lb): add support for multi value in lb lambda (#727)
  • (mta): Add support for bounces on receive (#724)
  • (cdn): include body for lambda@edge
  • add support for allExcept on cachepolicy (#721)
  • add smtp endpoint address to send mta (#715)
  • (module): add ses send events to service logs
  • (module): add ses based mail sender with log (#712)
  • (secretstore): add descriptions on secrets (#703)
  • remove the user defined boostrap process
  • (ecs): add support for container insights
  • (userpool): define email from address (#696)
  • (s3): add support for replication v2 (#690)
  • (userpool): control hosted ui setup (#686)
  • (ecs): propagate service tags to tasks (#682)

AWS Fixes

  • (secretstore): handle missing cmk
  • (api): build info sourcing (#753)
  • (s3): inbound link permissions for cdn
  • (cdn): link to cachepolicy
  • (api): image registry type access
  • (api): spec download logic
  • (apigateway): image source type checking
  • (baseline): default file path for image pull
  • map for ipset
  • (networking): handling of missing port on acl
  • (baseline): provide correct image for pull (#733)
  • readonly attribute assignment
  • lb path for state attribute (#730)
  • (elasticache): Use number based logic for retention (#720)
  • (baseline): add extra policies for cmk
  • (lb): backend support for lambda (#716)
  • (module): update link to basline component (#714)
  • (baselinedata): policy lookup on suboccurrence (#695)
  • (s3): add delete marker replication handling (#693)
  • athena s3 policy (#691)
  • (dnszone): add domain configuration if setup (#680)
  • update shared release workflow version
  • bugfix for aurora scaling
  • (lambda): remove env vars for lambda@edge

AWS Refactorings

  • replace reference with function lookups
  • (mta): move to using cfn for config set
  • allow manual trigger of release
  • github actions
  • (images): Update testing

Azure

Full set of changes: 8.8.0...latest

Azure Fixes

  • (ci): update to latest shared workflows (#313)
  • (computecluster): bootstraps lookup from global
  • remove use of getRegistryPrefix and EndPoint (#310)

Azure Refactorings

  • replace reference lookups with function
  • standardise github workflows (#309)

Bash

Full set of changes: 8.8.2...latest

Bash Fixes

  • (expo): remove ITMSTRANSPORTER_FORCE_ITMS_PACKAGE_UPLOAD override (#368)
  • (setup): handle multiple remote ref matches (#367)
  • repo management return code handling (#366)
  • update release package version
  • force ITMS package upload (#362)
  • job name