Welcome to hamlet! One big part of this release is our rename from codeontap to hamlet. Along with the rename we have also introduced deployment profiles and a new CMDB layout
hamlet and Code Restructure
CodeOnTap is now hamlet - the new name reflects the idea of building a city from many self-contained hamlets which are the smallest level of development in a city. This also fits with our idea of components and solutions forming your overall product which is part of your tenancy. With the new name we've also give our site a big facelift and merged the documentation into the site as well. Head to hamlet.io to check it out.
While everything was moved around, we also took the chance to restructure our code as well. This was mainly to reflect some recent changes (mostly the introduction of multiple providers) but also to reflect some future changes we have on the road map.
The code has been broken up into two key types;
-
engine The engine is our deployment document generation process which takes your CMDB and turns it into a collection of deployment documents. It is now divided into separate repositories based on providers :-
engine-plugin-awsprovides support for aws deploymentsengine-plugin-azureprovides support for azure deploymentsengineprovides the core shared provider.
-
executor Executors are responsible for executing the deployment documents generated by the engine. Each executor implementation has its own repository :-
executor-bashprovides the bash based executorexecutor-pythonprovides the python based executor which currently relies on the bash executor.
As a result of the restructure the environment variables which are used to reference a hamlet installation have changed, along with the repositories required for each of the functions.
The two guides below outline the current state and we recommend removing the old codeontap installation once you have migrated
Our new docker images also reflect these changes, and have been renamed to align with the hamlet name. See our Docker Image Guide for more information
Core Engine Changes
Deployment Profiles
Deployment profiles provide a convenient way to standardise configuration across all instances of a component. Prior to this release, deployment profiles were always applied AFTER all other configuration.
With this release, deployment profiles can now be applied BEFORE (still referred to as deployment profiles) and AFTER (referred as policy profiles) product specific configuration.
With this arrangement, deployment profiles continue to provide convenient defaults to the solution designer, but now they can be overridden in the product configuration.
On the other hand, policy profiles provide a mechanism for tenant wide policies to be employed.
An example might be where you want to define that all components which support encryption at rest always have the setting enabled.
Let's assume we have the following profile;
{
"DeploymentProfiles" : {
"encrypt" : {
"Modes" : {
"*" : {
"*" : {
"Encrypt" : true
}
}
}
}
}
}
This also shows another new deployment profile feature - wild card component type - which will apply the setting across any component type. Previously only the mode supported a wild card type (as shown above).
If applied as a deployment profile, encryption would be turned on if not overridden in the product configuration. On the other hand, if defined as shown below as a policy profile, it would be enforced regardless of the product configuration.
{
"Tenant": {
"Id": "msw",
"Title": "Ministry of Silly Walks",
"Domain": "msw",
"Profiles" : {
"Policy" : [ "encrypt " ]
}
},
}
6bc0e4d4 - <feature> Policy profiles (#1216) (Michael Leditschke)
cb556418 - <feature> Permit loading of a dedicated file for profiles (Michael Leditschke)
6a3df258 - <Feature> Component type wildcard support in profiles (#1099) (Michael Leditschke)
CMDB Layout
A new version of the CMDB layout has been added, and will be the default from our next release.
The high level product CMDB structure is now
- config
- settings
contains settings for the application
- infrastructure
- builds
the build reference registry for each application unit
- solutions
contains the solution outlining your deployment
- operations
- settings
an alternative settings location generally for sensitive settings
- state
- stores the state of the deployment ( templates, stack outputs etc.)
the state of each unit is now stored in its own subtree
To start using the new CMDB structure you can set the following environment variables to upgrade your current CMDB
GENERATION_MAX_CMDB_UPGRADE_VERSION=v2.0.1
GENERATION_MAX_CMDB_CLEANUP_VERSION=v2.0.0
The upgrade copies the solutionsv2 subtree under the config directory to the solutions directory under the infrastructure directory. The cleanup removes the solutionsv2 directory. If relying on the working directory to determine the product, environment and segment, the application of the upgrade needs to be separate to the application of the cleanup to avoid issues with removing the working directory. With this release, the infrastructure/solutions tree should be used if the working directory is being used to provide the product, environment and segment. The change of working directory should be applied, if used, between the upgrade and the cleanup.
CMDB's can also be pinned to a specific CMDB version to control automatic upgrades. In your .cmdb file you can add the following
{
"Pin" : {
"Upgrade": "v1.3.2",
"Cleanup": "v1.1.1"
}
}
This will pin the CMDB upgrade process to v1.3.2 and the cleanup version process to v1.1.1
fc945699 - <feature> Don't check all versions of pinned repo (Michael Leditschke)
dc6ab03f - <feature> CMDB v2.0.1 - Add du/placement subdirectories to the state tree (#1169) (Michael Leditschke)
5496da0b - <feature> Optimise cmdb version checks (#1113) (Michael Leditschke)
9acd1222 - <Feature> Add 1.1.1 cleanup to rerun 1.1.0 cleanup (#1106) (Michael Leditschke)
ef5f3094 - <Feature> Constrain cmdb upgrades (Michael Leditschke)
ea248567 - <Feature> semver range checking (Michael Leditschke)
Dynamic CMDB Loading
This release continues the transition to use of dynamically CMDB loading within the hamlet engine as much as possible. All input data is now loaded and validated during the engine's bootstrapping process. StackOutput processing is also handled in the engine so that we can handle multiple stack output formats and control how they work in the engine.
A set of new functions in the freemarker engine also move the loading of the freemarker templates themselves into the engine. This allows for the discovery of templates added by plugins and improves startup performance of the engine significantly.
3e62047f - <feature> Plugin Introspection (#1230) (Michael Leditschke)
5ea01739 - <feature> Dynamic CMDB loading (#1159) (Michael Leditschke)
8580ab5d - <feature> Dyanmic Load - Settings, Masterdata, Blueprint, Definitions (#1010) (Michael Foley)
e4f1368a - <feature> Reference Data Validation (roleyfoley)
7dfb93fb - <feature> StackOutput dynamic loading and input sources (#1007) (Michael Foley)
Deployment Document Testing
Deployment Document testing allows you to confirm that the deployment documents generated by hamlet match what you expect. You can define a test case in your solution which tests the contents of the documents. For example, you can test that the name of resource matches what you expect or test the value of a specific parameter. This facility is now being used in hamlet development to provide a method of unit testing of the deployment documents that we generate.
To help support testing, we've also introduced the concept of inputsources. The default input source composite uses your CMDB to generate a composite blueprint which is then processed to generate your documents. The mock input source is a new input source which uses templates in the engine itself ( including plugins ) to generate the blueprint which will be used by the engine to generate deployment documents.
When using the mock input source, scenarios give you the ability to load all of our input sources through engine freemarker files.
See the awstest library as part of our AWS plugin for some examples.
6c06cf8d - <feature> Create test case templates without a cmdb (#1087) (Michael Foley)
7d94833e - <feature> testplan generation (#1030) (Michael Foley)
dfa28e4a - <feature>Jenkinsfile for template testing (#1233) (Michael Foley)
Command line updates
The engine now has support for a new output level, the unitlist. This provides a list of all the deployment units found in your segment. This is useful in big solutions when you want to know everything that can be deployed.
The provider and resource group inputs allow you to define which provider you want to generate templates for.
eb1c1956 - <feature> Deployment Unit list (roleyfoley)
4a239232 - <feature> Provider switch (#1017) (Michael Foley)
f4347012 - <feature> Add resource group selection (Michael Leditschke)
Contracts
A contract provide a declarative agreement between the engine and the executor to achieve an outcome. This release migrates genplans to generation contracts, which express the steps the executor should perform to generate the necessary deployment documents for a unit.
A future release will introduce deployment contracts, which will express the steps needed to use the deployment documents to perform a deployment of a unit.
97d5101f - <feature> Contract output generation support (#1240) (Michael Foley)
External Integrations
A collection of new components types have been added to support integration with external services which might be managed by 3rd parties or other deployment processes.
externalserviceallows you to define a component in your solution which has no deployment but provides attributes to other components. The attributes of the component are defined using the settings process that we use for application settings and you can modify these settings using fragments as well. When another component links to an external service, it can use these attributes as part of its own processing. This component is a more natural alternative to an external link as the settings can easily be shared via conventional links, rather than having to be defined on every instance of the external link.templateallows you to write your own declarative provider templates, such as CloudFormation Templates, which hamlet will treat as a code artefact. In the solution or fragment you can configure the parameters that will be given to the template and also map the template outputs to hamlet stack outputs. This allows you integrate this template within the hamlet solutionadaptoris similar to the template component but instead of using a declarative template the adaptor component allows you to use the fragment template to define a scripted process for configuring an external resource. The adaptor supports the scripts the code unit type which you can use to feed in source code for your adaptor deployment. hamlet will generate a config file which you can use to provide environment contextual information.
797e0f89 - <feature> adaptor generic component (#1292) (Michael Foley)
3efc6f5b - <feature> external service links (roleyfoley)
af43a03e - <feature> Template component (#1281) (Michael Foley)
3384c37f - <Feature> Add external tier for external services (Michael Leditschke)
9da67160 - <feature> ExternalService Fragment and Env control Adds support for external services to use fragments Which also adds support for _context environment control (roleyfoley)
ed26c1c1 - <feature> External Serivce Component (#1068) (Michael Foley)
Other Changes
ed628b9a - <feature> Disable resource type lookups for alerts (roleyfoley)
c111bf35 - <Feature> Permit escaping of characters in envvar names (Michael Leditschke)
f438d4ee - <feature> Add help and args for blueprint generation (#1076) (Michael Foley)
e151af88 - <feature> jenkins docker build (#1243) (Michael Foley)
69c16a5e - <feature> Registry scopes (#1235) (Michael Leditschke)
2af1c551 - <feature> Null stripping for freemarker (roleyfoley)
05f07e5e - <feature> Control link attributes included in context (#1112) (Michael Leditschke)
433545fa - <feature> Namespaced provider specific configuration attributes (#1295) (Michael Leditschke)
AWS Provider
RDS
Adds Support for AWS Aurora cluster and minor version control support
4c8062d6 - <feature> rds minor version support (#1278) (Michael Foley)
ab3358e7 - <feature> Aurora read replica scaling (#1095) (Michael Foley)
5c57b63c - <feature> Add Aurora cluster support (#1088) (Michael Foley)
756eef32 - <feature> RDS Advanced Monitoring (#1077) (Michael Foley)
ECS
Adds support for the ECS capacity provider to implement an Ec2 Ondemand container scaling
bd54e762 - <feature> container run mode explicit configuration (#1302) (Michael Foley)
c61153f8 - <feature> ECS OnDemand Capacity Provider (roleyfoley)
e36f6ee7 - <feature> ECS Subcomponent links (roleyfoley)
Cognito Userpools
Updates cognito to use CloudFormation and also adds support for a collection of extra features in cognito
- Resource Servers
- Security enhanced features and username management
02055f85 - <feature> userpool resources and advanced sec (#1301) (Michael Foley)
82289f0f - <feature> userpool client external service support (#1189) (Michael Foley)
API Gateway
a3cd58f5 - <feature> Allow API publshers to be disabled (#1249) (Michael Leditschke)
34637eb8 - <feature> APIGW add option security control (#1204) (Michael Foley)
98339e7b - <feature> APIGateway - error on missing cert (#1188) (Michael Foley)
da6e7bc0 - <Feature> Make API Gateway viewer policy configurable" (Michael Leditschke)
Lambda
13c6f40d - <feature> Supported lambda run-times (Michael Leditschke)
2a6cc8b7 - <feature> lambda reserved executions (roleyfoley)
818dde2a - <feature> Lambda Create new version on deploy (#1065) (Michael Foley)
f52f4d34 - <feature> Add node10 support for lambda (Michael Leditschke)
ElasticSearch
084c893f - <feature> Add support for ES S3 restoration (#1186) (Michael Foley)
4f20782b - <feature> es dataset support (#1177) (Michael Foley)
581df5b7 - <feature> ES Improvements (#1156) (Michael Foley)
Other AWS Changes
ac3043d5 - <feature> WAF GeoMatch Support (#1291) (Michael Leditschke)
c838fbcc - <feature> export method configuration in fastlane (#1272) (Michael Foley)
1e07a02b - <feature> CodeOnTap Agent switch role policy (#1236) (Michael Foley)
333ef5b6 - <feature> X-Ray control (#1142) (Michael Leditschke)
010c2334 - <Feature> Add asFile support for user component (#1098) (Michael Leditschke)
6182ecc6 - <feature> SNS topic Subscriptions for AWS services Adds support for Lambda and SQS to be used for topic subscriptions (roleyfoley)
Azure Provider
This is first version of the Azure provider, it includes a collection of our existing components and support for a wide range of resources
968fabf - <feature> Github templates (Michael Leditschke)
91e22c8 - <feature> jenkinsfile build process (#96) (Michael Foley)
e61e566 - Feature - New Component - Userpools (#111) (Ross Murray)
865e7fe - new service - microsoft.dbforpostgresql (#113) (Ross Murray)
dbd256f - new component - adaptor (#109) (Ross Murray)
7a5b0b3 - Component - LB (#103) (Michael Foley)
0f8c821 - new service - microsoft.apimanagement (#105) (Ross Murray)
c0123cf - Component - Bastion Host (#95) (Ross Murray)
55d138f - new Service for role definitions and assignments (#100) (Ross Murray)
ecd9ddf - New Component - Lambda (#89) (Ross Murray)
d0a7264 - create sqs component and new utility functions (#87) (Ross Murray)
83a48a3 - Create cdn component /w route subcomponent and WAF (#80) (Ross Murray)
fcbcdd0 - Feature keyvault reference (#73) (Ross Murray)
a4d4a34 - Include NetworkEndpoints in Subnet Creation (#51) (Ross Murray)
9071c26 - Implementing VM ScaleSets Resource (#50) (Ross Murray)
c679608 - implimenting autoscale settings. (#49) (Ross Murray)
6eaf479 - Component - Network (rework) (#47) (Ross Murray)
baa93a0 - Implimenting zones attribute for use on armResource as it is available to the vm-scalesets azure resource. (#48) (Ross Murray)
b00f6d9 - Feature - Network Component (#46) (Ross Murray)
45836d2 - Feat component baseline (#45) (Ross Murray)
d3f50d0 - added resources frontDoor + frontDoorWAFPolicy (#71) (Ross Murray)
915fa92 - Feat resource alb (#58) (Ross Murray)
edaa4de - Feat component gateway (#53) (Ross Murray)
6089b2d - created spa component sans cdn interactions. (#77) (Ross Murray)
2e0e7a7 - Azure ARM Template Generation (#23) (Ross Murray)
adf6475 - Feat deploy (#19) (Ross Murray)
170e2fe - Feat bootstrap (#3) (Ross Murray)
4f18f2a - Feat bootstrap (#2) (Ross Murray)
39ee476 - Feat resource profile conditions (#40) (Ross Murray)s