This release post is a long read for a couple of reasons, we missed doing one for the 8.2.0 release, and there have been a lot of new features since our last post.
Similar to our last post this release has focussed on the first 30 minutes of using hamlet, along with a collection of updates to make hamlet easy to use in your existing workflows.
Getting Started Guide
Our biggest features this release is a getting started guide.
This has been something we have needed for some time in hamlet as we want people to discover what hamlet does and how it works by themselves and to see the benefit of using hamlet.
There are a lot of deployment tools around the place and people don't have an endless amount of time to try out and adapt to new tools so we set some rules on our guide
- The whole guide should take 30 minutes on average to complete
- Each task in the guide could have no more than 10 steps
- All of the work had to be completed from the users local workstation
- The guide had to cover:
- Installing hamlet on the users local workstation
- Bootstrapping the account the user wants to deploy into
- Running a common deployment pattern that was relatable to a real world scenario
- The deployment in the guide needed to be cheap to run but produce a tangible result at the end
- Cover off the basic concepts of hamlet
Based on these rules we chose to walk through the deployment of a container based API and a Single Page Application that talks to the API. This would be deployed to AWS with the container available as a public image and the single page app needed to built by the user.
So after second breakfast was done, we set off to create the guide....
And ran into some dragons, our install process was a bit more than 10 steps, not all of the tasks could be easily completed locally and there was quite a bit of jumping between different workflows to get where you wanted. So in this release we focussed on a number of features to make the getting started guide going.
CLI Engines
The hamlet install process is a bit complicated and can take a bit of getting used to. There are number of repos to clone and quite a few environment variables that need to be set to get everything lined up. This is one of the reasons for releasing hamlet as a prebuilt docker image. It meant that we could handle all of the setup and you could get going, as long as you wanted to use that container.
With this release we've now moved this setup process into the hamlet cli with the introduction of engines. Engines are versioned installations of all repositories which the cli looks after. You can have multiple versions of the hamlet engine installed locally and switch between them. This is really useful when you want to test a new version of hamlet before updating your CD process or working across a range of deployments.
Lets have a look at how it works. In the hamlet cli there is now a new command group called engines;
hamlet engine list-engines
| Name | Description | Installed | Global | Update Available |
|----------|--------------------------------------------------|-------------|----------|--------------------|
| unicycle | Latest build of the official engine parts | True | False | True |
| dev | local development engine | True | True | False |
| tram | Nightly build of the official engine | True | False | False |
| train | The latest release of the official hamlet engine | False | False | None |
| 8.2.1 | Stable release of the official hamlet engine | False | False | None |
This command shows us all the engines the cli knows about :-
- unicycle, tram, train - are dynamic releases which align with our scheduled release process. Engine updates will be pushed to each of these engines as we publish new releases
- 8.2.1 - Is a fixed stable release of hamlet. Using this engine means that you don't get updates but can lock to a known release for your deployments
- dev - Is an engine that I've configured for my local cli. This allows me to have a clone of all the code repositories that I can tell the cli to use when developing new features for hamlet
Engines support updates and you can configure automatic installation of updates if you want to. By default we add a notification to the next cli command you run if an update is available
To install an engine
hamlet engine install-engine train
This pulls down the hamlet images as required and saves them as a local copy for the engine
Finally to use the engine
hamlet engine set-engine train
hamlet engine set-engine train
[*] global engine set to train
The engine setup uses a shim style approach similar to pyenv or nodenv - when you set the engine, we update the links to the engine that you've configured. Then when you run a hamlet cli command we set the environment variables that hamlet requires to line up with the engine that's in use.
If you have existing pipelines that call our bash scripts directly you can still use the hamlet cli to install the engine you want.
In your pipeline run the following to set the engine and then set the required environment variables for your shell
hamlet engine set-engine 8.2.1
eval $(hamlet engine env)
[*] installing engine
[*] global engine set to 8.2.1
Running set-engine will automatically install the engine if its not available and then we export all the hamlet environment variables that are required.
Now, having read this you might be thinking, how have you made this easier?. Well managing engines is pretty advanced, so we have embedded engine management into the hamlet cli. When you install the hamlet cli and run your first command the cli automatically installs and configures the train release. So if you are just using the cli, all you need to is install the cli ( plus some OS dependencies ) and you are off and running.
To see this in action run through the install guide
Provider Authentication updates
Within hamlet we had a couple of different ways of authenticating with AWS during deployments. For commands that could run locally we used your local AWS config. When running CD tasks in a pipeline we would either use a user namespaced AWS access key and secret key or an assumed role. In this release we have standardised the authentication process for both AWS or Azure logins.
For AWS we now support configuring an authentication source which manages where credentials are collected from. hamlet will also validate that the account you've asked to deploy into matches the account that hamlet has permissions for. If it doesn't hamlet will fail to run the deployment task.
You can select how you want to authenticate using the environment variable
HAMLET_AWS_AUTH_SOURCE
Which supports the following values
ENV- Use the standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variablesUSER- Works like the ENV source but supports a prefix on the AWS keys that you can set with HAMLET_AWS_AUTH_USER so if you set HAMLET_AWS_AUTH_USER=SEC hamlet would look for SEC_AWS_ACCESS_KEY_ID for your aws authentication keysINSTANCE|INSTANCE:EC2|INSTANCE:ECS- Uses the ec2 or ecs instance iam roles. If you specify INSTANCE hamlet will prefer ECS over EC2, but use either one.CONFIG- Uses an aws cli config file to determine credentials. hamlet will search for a profile in your config file based on the hamlet account id or the AWS account id derived from the hamlet account detailsNONE- This bypasses authentication and disables the account check. This is useful when you are testing deployment generation locally and don't have access to the AWS account you are working with.
After configuring the source you can optionally configure a role and MFA token
HAMLET_AWS_AUTH_ROLE- Is either an role ARN or name that would be used to switch into the account that you are deploying intoAWS_AUTH_MFA_SERIAL- If you need MFA you can set the MFA_SERIAL arn here and hamlet will use it on any of the authentication sources. If you are using the config provider it should be set in your config file
The Azure Provider has a similar configuration option inline with their authentication support
AZ_AUTH_METHOD
Which supports the following values
SERVICE- Uses a service principal supplied via HAMLET_AZ_USERNAMEMANAGED- Uses a managed identityINTERACTIVE- Uses the interactive login process which will ask you to browse to a URL to loginNONE- This bypasses authentication and disables the account check. This is useful when you are testing deployment generation locally and don't have access to the Azure account you are working with.
For both providers you can override the source to use for a specific account by adding the hamlet account id as a suffix on any of these environment keys.
Release via CLI
hamlet manages application code as images, which are artefacts built based on your code for a particular deployment model. You can now provide these images to hamlet through the cli using the hamlet release command group.
hamlet release upload-image is the command to upload your images into the hamlet staging registry. You can provide a number of formats for the images
--dockerfile --docker-context- Will build a docker image based on these values and upload the image to the container registry--docker-image- Will look for a image in your local docker image store with this name and upload it to the container registry--image-path- The path to a file based image - this can either be a directory or a zip file. If you provide a directory it will be zipped up. The zip file with then be uploaded to the file registry ( s3 on aws)
hamlet release transfer-image allows you to transfer images between your accounts. This is useful for promoting code images between environments.
hamlet release update-image-reference updates the image reference that will be used in an environment.
This makes it easy to integrate hamlet with your existing build processes. You run the build processes and let hamlet know where the results are and it will look after adding it the hamlet registries.
Deploy handling
The deploy commands have had some updates to get more output and details of what is happening during a deployment.
- The output has been condensed to only include information about the deployment and how its going
- A check is performed before running any commands against a cloud provider to make sure the command is being run within the right account/subscription
- If a cloudformation deployment fails in AWS we now provide the events from that particular cloudformation update. This allows you to see why a failure occurred from your console
- The dry run command is now supported for create and update deployments. When a new deployment is run you can add the
--dryrunflag to run-deployments and it will show you what the deployment will create - Account level deployments which are used to setup an AWS account for use with hamlet are now available through the cli. Use
--district accounton any deployment command to see and run the deployments
Engine Updates
Within the engine we have extended the support in some of our key functions
Links Refs
Link refs allow you to create named links which can be overridden or shared throughout a CMDB. They can be defined as part of a solution or can be set in your account or tenant level configuration and shared across all solutions.
Qualification on all attributes
The qualification syntax is now available on any attribute defined in your cmdb. Instead of performing overrides based on district details like environment or segment by providing an override file, you can now define these rules in shared configuration files. This can significantly reduce your CMDB configuration and make it easy to understand where configuration is sourced and applied.
New Components
Two new components were added in this release
- Firewall which provides network level filtering of traffic in private networks. It can be configured per tier based on their route table configuration.
- Directory which represents a directory service such as an LDAP user store or a Microsoft Active Directory deployment. They are expected to be managed offerings during deployment.
Maintenance Windows
For managed services based on instances deployed per user there is often a need to perform maintenance like patching or hardware updates. In hamlet setting the maintenance window across all components has now been standardised. This includes timezones to make it easier to set up the windows at times that won't impact your application and a common pattern to set when the window occurs.
AWS Provider
The AWS provider has seen a collection of updates and new features added. This is mostly in line with our day to day work.
Windows EC2 Support
The AWS provider now supports Ec2 instances running the Windows Server Operating system. This was a proof of concept based on the introduction of compute tasks into the provider last release.
The Windows support covers all of the bootstrap activities that we perform to setup load balancers, ecs configuration and disk configuration.
The Directory component, which was added into the engine this release, has been implemented as part of this support based on the AWS Directory Services for Active Directory implementation.
Tagging updates
Tagging support has been added to a number of resources inline with the support added to CloudFormation.
This includes the AWS services:
- IAM
- CloudFront
- SNS
- ECS
- SQS
- API Gateway
Test Coverage
Testing coverage has increased significantly across the AWS components in this release to ensure that we can catch issues and identify breaking changes as soon as possible in our deployment process.
CLI Updates
The hamlet package has now been moved in pypi to hamlet. With the hamlet cli becoming our primary interface for working with hamlet this made sense and makes it easy to install.
The query command group has now been replaced with the component command group. This aligns with the scope of the queries available in the command. The component command can now list the available occurrences along with allowing custom queries of a single occurrence or all occurrences.
Development Changes
The development and release processes for hamlet have seen a number of significant changes in this release.
Docker build process
A lot of the hamlet software is based on scripts or files called at run time across a couple of different tools ( bash and Freemarker mostly). Our current release process was based on building up a general purpose container and cloning the various hamlet code repositories into the container. This gave us a collection of all the latest changes to hamlet as container image but it required the download of a big container image and meant that you couldn't really use your own images or run hamlet locally.
There was also an issue with our latest container builds where we would always clone the default branch of each repo. This meant that if one of the plugins had an issue, it would be introduced into the latest container build and cause failures in hamlet.
Each hamlet code repository is now built as its own container and the image is only created once testing of the code has been completed. The container images don't have an OS. Instead, we use the scratch image as the base and copy the required code into the image.
This process is what drives our new engine support and makes it simple to understand and version our hamlet installations.
Github Actions Migration
Along with the setup of docker build process we have moved our CI and release processes over to Github Actions. We used to run these processes on a Jenkins server provided by GoSource ( our primary sponsor ) however we also used hamlet as the agent container. This was a bit of a problem when we found a bug during hamlet development as it could stop the CI server from running ... not so great.
Moving to GitHub Actions has simplified our CI processes and also made it easy to move to the Docker based hamlet build process as we could use the GitHub Packages repository to manage our docker images.
Anyone contributing to hamlet can also see their action results and get details on why an action had issues during CI.
CLI Improvements
- Code is now formatted using the black code formatting tool.
- Sphinx is now used to generate documentation on the cli include a command line reference document which will be added into our documentation site in the next release
Full Change Log
So this has been a pretty big set of changes that we think help using hamlet and provide a lot of flexibility in what you can do. To see the full change list, head to the change logs in each repo.