Welcome to the v6.0.0 release of codeontap which includes support for dynamically loading hamlet implementations through plugins
Breaking Changes
RDS Component renamed to align with vendor agnostic naming
As part of our support for multiple cloud providers for template generation we have changed the type name for the rds component to db. If you are using deployment profiles to manage configuration options on your db component you will need to update the type of all rds components in your solution to the db type in both the deployment profile and the component itself. This should not impact the RDS instances themselves but can override configuration applied using the deployment profile
- 224833c - Rename RDS component to DB (#881) (Michael Leditschke)
CDN/SPA Updates - Breaking Change
This change introduces a new CDN component. CDN components replace the solution level SPA component which provisioned a Content Distribution Network responsible for making the SPA code available over HTTPS. The CDN component adds a Route subcomponent which allows you to route specific path patterns to different origins. For example you could have an SPA available for all paths by default along with a /api/* pattern which would redirect API calls to a load balancer.
For AWS deployments this transition will require a breaking change for existing SPA deployments. This will require running a manageEnvironment stop on the solution SPA component, once the template has finished the deletion process the CDN component can be deployed and the DNS entry updated. AWS does not allow the same hostname to be used on two distributions so you can't build one then swap over the DNS.
- 0192be2 - CDN Component (#938) (Michael Foley)
- 39985ad - CDN Error Response config (#957) (Michael Foley)
- eddbc80 - SPA config path control (#964) (Michael Foley)
- 76e09d0 - CDN support for APIGateway (roleyfoley)
Userpool updates and federated roles
These changes split the Federated identity pool functionality which provides access to a cloud provider using external identities away from the user pool component that we have now. In AWS these functions are provided by the same AWS service however they are quite different in what they do.
The federated role component allows you to use the identities from a third party authentication provider ( including userpools ) to access cloud provider resources which require native authentication methods e.g. AWS IAM for S3. User attributes can be used to map different permissions to provide role based access control as well.
Along with the federated role changes we've also added support for userpool federation with an OIDC based authentication provider
Finally we have updated the required setting values when using an external userpool to align with the changes introduced for clients in the last round of hamlet changes
- 2283841 - Federated Roles (#828) (Michael Foley)
- e1aad61 - Federated Role Attributes (#937) (Michael Foley)
- 2e9a244 - OIDC userpool federation (#970) (Michael Foley)
- 210d835 - userpools: align ext and int config (roleyfoley)
- 6795a0f - userpool only validate cognito config during cli pass (#976) (Michael Foley)
Alert Actions
This change adds to our existing alarms functionality with the ability to specify where alerts are sent to when they are triggered. Alerts must now be configured with a severity and the new AlertRules master configuration option allows you to distribute alarms to different components based on their severity. Initially we only support topics in the Alert Rule distributions.
With the introduction of alert actions and topics we have removed the baseline SNS topic from the base line components. Instead you now need to create a new topic along with alert rules which match your requirements
For example you can have two SNS topics setup
- business-hours-topic
- after-hours-support
You could then have two AlertRules setup
{
"AlertRules" : {
"GetOutOfBedThings" : {
"Severity" : "error",
"Destinations" : {
"Links" : {
"Tier" : "msg",
"Component" : "after-hours-support"
}
}
},
"WaitToAfterCoffeeThings" : {
"Severity" : "info",
"Destinations" : {
"Links" : {
"Tier" : "msg",
"Component" : "business-hours-topic"
}
}
}
}
}
When we deploy the alerts you have configured on your component, these rules route alerts to the right topics. The severity lookup works in a similar fashion to log message levels which act as a limit on the log level that you want to receive.
- c9b4475 - Add support for Alert Action controls (roleyfoley)
Features
Core engine rewrite and plugins
The biggest change is this release has been to our core template engine with the introduction of a modular architecture which uses dynamic loading for all components, resources and reference data
Dynamic Loading of component configuration
Dynamic loading allows for freemarker files to be dynamically imported when they have been invoked rather than our existing process which combined all of our template files into a single file and used case statements to separate out different components. This process forms the foundation of the modular architecture while increasing the performance of the template engine. More details on the freemarker approach that we are using is available here
- d09dd12 - Dynamic output formatting (#838) (Michael Leditschke)
- ac0d9c6 - Dynamic loading of seg/sol/app templates (#809) (Michael Leditschke)
- beff719 - Template processing (#836) (Michael Leditschke)
- 809e0fc - Use macros to create component config (#817) (Michael Leditschke)
- 54272a3 - Add dynamic state routine lookup (#802) (Michael Leditschke)
Core engine improvements
A collection of changes to our core engine to improve the development experience
Generic Logging replaces the existing GENERATION_DEBUG with the GENERATION_LOG_LEVEL variable. This allows for different log types to be used during template generation. To replace GENERATION_DEBUG use export GENERATION_LOG_LEVEL=debug
Timings have been introduced to provide a better understanding on what is taking up the time in generating templates
Updating the directory layout within the template engine to fully support the idea of plugins and core services
- aab5271 - Processing timing (#841) (Michael Leditschke)
- aac1463 - Generic logging (#830) (Michael Leditschke)
- 343363a - Separate occurrence/link processing from common (#846) (Michael Leditschke)
- fec5dea - Remove commonSegment.ftl (#852) (Michael Leditschke)
- 13efb0a - Remove commonSolution.ftl (#854) (Michael Leditschke)
Generation Plans
Generation plans allow for a component to specify the templates that need to be processed and generated. Generation plans are generated through the same template engine and provide the combinations of templates which need to be generated. This plan is then processed to generate the actual templates you need. This significantly reduces the processing time for template generation and reduces the number of passes requried to get all of the scripts setup.
- 7bba831 - Generation plans for all components (#811) (Michael Leditschke)
- d78a9a0 - Generation plans (#810) (Michael Leditschke)
Multi provider - Plugin
This set of changes introduces the idea of plugins. We currently have two plugins
- shared - which contains the standard component definitions which are vendor specific
- aws - which contains the existing component, services and resources to deploy to AWS
Plugins are loaded by the engine is responsible for the load process along with defining the core hamlet concepts like occurrences, components etc.
To add a new plugin add the path to your plugin code to the GENERATION_PLUGIN_DIRS environment variable
- 362c269 - Split id files (#806) (Michael Leditschke)
- c366bf6 - Move master data under respective provider (#844) (Michael Leditschke)
- 7617b71 - Provider subdirectories (#832) (Michael Leditschke)
- c3ed2dd - Reorganise component files under aws provider (#815) (Michael Leditschke)
- 95fea42 - Resource Groups and placements (#827) (Michael Leditschke)
- 1b35507 - Component service dependencies (#833) (Michael Leditschke)
- 2155e15 - Static component dependencies (#819) (Michael Leditschke)
Context Management for Account and product level solutions
These changes are still in progress and introduce the idea of contexts. Contexts provide the foundation for cross product and links to account/tenant level components. This is an important step in our work to allow for deployments to multiple provider accounts and to multiple providers combined with our plugin support.
- 242c75a - Dynamic model control. (#870) (Michael Leditschke)
- 2447b1e - Round out context based model support (#953) (Michael Leditschke)
- 930cd2f - First cut of context based in-memory model (#859) (Michael Leditschke)
- eced907 - Transitional arangements for composites (#804) (Michael Leditschke)
Setting Namespaces
Adds support for settings namespaces. Namespaces allow for custom settings structures for components. At the moment settings are looked up based on fixed values based on the name of occurrence details. SettingNamespaces work in a similar way to domain naming where different occurrence name sections are combined to generate a new namespace.
Combining Namespaces with deployment profiles offers a way to provide settings to groups of components
- 57fa2bb - Core setting namespaces (#697) (roleyfoley)
Expo Fastlane support and Sentry Releases
This continues on from our previous release to add ExpoKit build and deploy support using fastlane. ExpoKit builds can not use the standard expo build process as they generally contain native application libraries, When using ExpoKit you need to build the binary app images your self which can be automated with tools such as fastlane.
Along with ExpoKit builds we also add support for uploading sourcemaps created using expo to Sentry for exception monitoring and reporting
- 3b7414e - expo app bundle id overrides (#845) (Michael Foley)
- c73bf1c - Expo Fastlane build support (#865) (Michael Foley)
- 3d149a6 - fastlane builds for expoKit apps (#850) (Michael Foley)
- 0eab8e2 - upload source maps for the current expo sdk version only (#897) (Kseniya Shychko)
- 6260312 - expo app id override (#894) (Michael Foley)
- 7c8a16a - sentry releases (#888) (Kseniya Shychko)
- f71cb3c - Expo build - skip OTA (#891) (Michael Foley)
- c3783ee - Expo Testflight submission (#879) (Michael Foley)
- 814ada1 - use BUILD_REFERENCE for sentry release name (#915) (Kseniya Shychko)
- 3149509 - override expo revisionId after export (#909) (Kseniya Shychko)
Topics
Adds a new component for Pub/Sub services called topics. Topics are used to distribute messages to different subscribers that need to know about a particular event which has occurred. For AWS we also added support for S3 events to trigger topics to notify of changes to an S3 bucket, along with allowing topics to invoke lambda functions when an event is received by the topic
Solution
{
"alerts" : {
"topic" : {
"Instances" : {
"default" : {
"DeploymentUnits" : [ "alerts"]
}
},
"Subscriptions" : {
"fixit" : {
"Links" : {
"fixit" : {
"Tier" : "external",
"Component" : "fixiit"
},
"fixitemail" : {
"Tier" : "external",
"Component" : "fixitemail"
}
}
}
}
}
}
}
Settings
The setting below map to the two external links outlined in the solution
{
"FIXIT_SUBSCRIPTION_PROTOCOL" : "https",
"FIXIT_SUBSCRIPTION_ENDPOINT" : "https://webhook.msw.io/fixit",
"FIXITEMAIL_SUBSCRIPTION_PROTOCOL" : "email",
"FIXITEMAIL_SUBSCRIPTION_ENDPOINT" : "fixit@msw.io"
}
- 8d661df - add SNS as an inbound link for lambda (roleyfoley)
- 648d8e2 - S3 extend notifications for sns lambda (roleyfoley)
- a2196a3 - Add topic component (roleyfoley)
- a020b15 - s3 service support for notifications (roleyfoley)
- 9b66361 - topic add monitoring and permissions (roleyfoley)
Scaling Policies
Scaling policies allow you to add metric based autoscaling policies to components which have horizontal scaling capabilities. Scaling policies use our metric and alarm configuration to apply either stepped or tracked based scaling policies.
Stepped scaling policies use an alarm to set a watermark level. You can the apply step policies relative to alarm threshold to scaling resources in and out Tracked scaling policy will automatically adjust the resource counts to keep a metric at as specific target value
Scheduled policies are also available to schedule the maximum and minimum resource counts based on time based schedules. This allows you to restrict how many resources are deployed to meet the scaling policy. For example you might want to scale further during business hours than after hours.
{
"squid" : {
"Engine" : "fargate",
"DeploymentUnits" : [ "squid" ],
"Profiles" : {
"Processor" : "default"
},
"Containers" : {
"_squid" : {
"Version" : "latest",
"Cpu" : 256,
"Memory" : 512,
"MaximumMemory" : 512,
"Ports" : {
"squid" : {
"IPAddressGroups" : [ "_global" ]
}
}
}
},
"NetworkMode" : "awsvpc",
"ScalingPolicies" : {
"TrackMemoryUsage" : {
"Type" : "Tracked",
"TrackingResource" : {
"Link" : {
"Tier" : "app",
"Component" : "apphost",
"Instance" : "",
"Version" : ""
},
"MetricTrigger" : {
"Name" : "MemoryUsage",
"Metric" : "MemoryUtilization",
"Threshold" : 2,
"Severity" : "info",
"Statistic" : "Average",
"Unit" : "Percent"
}
},
"Tracked" : {
"TargetValue" : 80
}
},
"BusinessHours" : {
"Type" : "Scheduled",
"Scheduled" : {
"ProcessorProfile" : "businesshours",
"Schedule" : "cron(0 9 ? * MON-FRI *)"
}
},
"AfterHours" : {
"Type" : "Scheduled",
"Scheduled" : {
"ProcessorProfile" : "afterhours",
"Schedule" : "cron(0 18 ? * MON-FRI *)"
}
}
}
}
}
This example shows how you might want to scale an ecs service. We have a tracking metric on memory utilisation so that we can ensure we aren't over provisioning services. The scheduled policies change the processor profile for the service to restrict how many service instances can be deployed during after hours when we don't expect a lot of traffic.
Scaling Policies are availble for ecs services, ecs solution clusters, compute clusters, and ec2 components
- 10d3201 - Autoscaling ECS & Ec2 (#952) ( Michael Foley )
Object SQL
Adds a new component which deploys the resources required for object store sql services ( like AWS athena ). The component doesn't deploy an actual database table as these are generally highly dependent on application code and are best left to an admin lambda function, instead this deploys the resources and permissions required to use the service
- 73e8737 - Object SQL component (#969) ( Michael Foley )
Azure Deployment Support
These changes add the intial support for Azure as a provider. They are temporary workaround while we work through how to define the process for deploying resources in pluggable way. Stay tuned for big changes in this space next release
- 7a486c3 - feature - manageDeployment (#951) (Ross Murray)
- b94f373 - format composite outputs for azure (Ross Murray)
Others
- 3fc8ae7 - DB: Major version update support (#954) (Michael Foley)
- 79079c0 - datafeed s3 destination (#944) (Michael Foley)
- 1901d78 - APIGW: publish with filename control (#902) (Michael Foley)
- 2c9fbe0 - Pipelines add concurrency saftey and logging (roleyfoley)
Refactor
- a9d2453 - APIGW: move deployment details in to swagger update (roleyfoley)
- 46b6885 - Use occurrence to determine baseline links (#939) (Michael Leditschke)
- 672be66 - Remove id/name defaulting (#918) (Michael Leditschke)
- c7a9be5 - Revert change to OWASP payload limit (Michael Leditschke)
- f73af93 - Align default OWASP size limit to API limit (Michael Leditschke)
- 709b796 - remove product deployment components (roleyfoley)
- 8d5ed24 - remove product specifc id formating (roleyfoley)
- 40ce947 - baseline: Migrate OAI to cf for all deployments (#840) (Michael Foley)
- 7bffb3c - tidyup aws use of dynamic outputs (#839) (Michael Leditschke)
- ded2abd - use baselineprofile for all components (roleyfoley)
- 5e12005 - baseline component lookups (roleyfoley)
- 70c1f22 - Locate server role via occurrence (#803) (Michael Leditschke)
Fixes
- 317e1b7 - SPA should link to CDN route instead of CDN (#966) (Michael Foley)
- 60099f8 - cache alert actions (#965) (Michael Foley)
- 5f601a9 - Add dummy dataset deployment (#962) (Michael Foley)
- 6921ee8 - AWS Pipeline activation logic (#961) (Michael Foley)
- fd8b073 - RDS note typo (#960) (Michael Foley)
- dba2aba - pipeline error reporting and cleanup naming for better understanding (roleyfoley)
- 40128e0 - dataset - update to support new db component type (#958) (Michael Foley)
- 2f6c9cb - Minor fixes (#955) (Michael Foley)
- 1709411 - add VolumeMounts into ec2 setup (#950) (Michael Foley)
- 3e716cf - datafeed S3 permissions (roleyfoley)
- d94c3d7 - userpool don't show error in resources (#949) (Michael Foley)
- 3fad30a - Allow resources to be always marked as deployed (roleyfoley)
- 79e0972 - ECS task parameters in wrong code section (#946) (Michael Leditschke)
- 07c930d - swagger paths and schemes for swaggerv2 (#942) (Michael Foley)
- 55e2f54 - SPA generate solution level genplan with warning about deprecation (#941) (Michael Foley)
- 43b7963 - base link lookup for ecs components (#940) (Kseniya Shychko)
- eb77168 - identity pools fixes for rule based rules (#936) (Michael Foley)
- f32dde1 - Federated role validation rules (roleyfoley)
- a3448d8 - SPA links should be subobjects (Michael Leditschke)
- 4dbafbc - ECS Volumes and LB Base Path redirects (roleyfoley)
- 95e7fd1 - Add id/name defaulting to blueprints (Michael Leditschke)
- 90b5fcc - Add id/name defaulting for domains and certificates (#922) (Michael Leditschke)
- 4f8a30e - Correct spelling of Unkown (#921) (Michael Leditschke)
- 7c905a3 - Use log level in cleanup logic (#920) (Michael Leditschke)
- 0cf6461 - regenerate AWS nmasterdata ftl (Michael Leditschke)
- 8cffd63 - APIGw Swagger dist copy (#914) (Michael Foley)
- b8a1817 - APIGw: fix for publisher copying (#913) (Michael Foley)
- ad3b2ee - lb baseline lookup for lb logging (#912) (Michael Foley)
- 381406d - Lb Port Id name lookup (#911) (Michael Foley)
- 12bd1b9 - hibernation - fix for determinig deployment (roleyfoley)
- 5427eba - taskContainers remove old debug (#908) (Michael Foley)
- 001d960 - cleanup loop config from contentnode (#907) (Michael Foley)
- 27b9a25 - typo in contentnode setup (#906) (Michael Foley)
- 6b03bdf - contentndoe sync modes (#904) (Michael Foley)
- c372085 - contentnode file copying (#903) (Michael Foley)
- ebe0544 - contenthub deployed detection (#901) (Michael Foley)
- b1ebede - contenthub typo (#900) (Michael Foley)
- a0a543b - expo id override (#898) (Michael Foley)
- 1d2f04d - SPA: primary domain lookup fixes (#896) (Michael Foley)
- 6fa15ab - SPA certificate manager and manageCrypto fix (#895) (Michael Foley)
- 3d62b05 - Expo typo in build checks (#893) (Michael Foley)
- 3f40c1c - Expo publish syntax (#892) (Michael Foley)
- 6d3344a - fastlane fix app id for upload (#890) (Michael Foley)
- b04afac - bunlde hosting location and remove cleanup (#889) (Michael Foley)
- 3e97c7a - Userpool role creation resource dependency (#887) (Michael Leditschke)
- 0ddaab9 - userpool role id lookup (roleyfoley)
- b97caa0 - expo testflight fix up naming of variable (#885) (Michael Foley)
- 2ef102c - Default id/name in suboccurrence processing (Michael Leditschke)
- b74adb4 - Correct fetching of baseline component ids (Michael Leditschke)
- 6dab650 - SPA baseline processing (Michael Leditschke)
- a6d0a52 - baseline parameter lookup (#880) (Michael Foley)
- 9b10b2e - Ignore role resources in deployment state (#878) (Michael Leditschke)
- 94c1aa7 - Ignore iam/lg resources for deployment state (#877) (Michael Leditschke)
- c0f9733 - Default id and name for subobjects (#876) (Michael Leditschke)
- eb63d32 - fastlane certificate import (#875) (Michael Foley)
- 30b60cf - fastlane certificate lookup (#874) (Michael Foley)
- c1de14d - force identity lookup for codesigning (#873) (Michael Foley)
- f898af3 - fastlane - fixup parameter name (#872) (Michael Foley)
- 5b3a40f - update code sign identity for iphone (#871) (Michael Foley)
- 152db29 - Merge pull request #869 from ml019/fix-sqs-uses-cw-service (Ross Murray)
- 0476d37 - SQS has a dependency on cloudwatch for alarms (Michael Leditschke)
- 4263394 - Expo fastlane build updates (#867) (Michael Foley)
- 93ce9ea - expo fastlane verbose update logging (#866) (Michael Foley)
- ad5e877 - instance and version name for cfredirect (#864) (Michael Foley)
- aa3ee1b - master data file naming (#863) (Michael Foley)
- fdcf4d7 - Add id and name config for child components (#862) (Michael Foley)
- 87434fc - Baseline Attributes in default Env (roleyfoley)
- 5d94dc4 - Remove redundant mode parameter (Michael Leditschke)
- 7a509d4 - use sdk specifc manifest for binary (#858) (Michael Foley)
- 42d5c15 - change severity for missing macro (roleyfoley)
- 948de0e - Merge branch 'master' of https://github.com/hamlet-io/engine (roleyfoley)
- b7fd23a - Bump expo build tool versions (roleyfoley)
- 54a58d1 - Minor fixes to publish job for expo (#856) (Michael Foley)
- 8fa0739 - baseline bucketId Lookup (roleyfoley)
- 6c77ee5 - remove extra add to buildConfig (roleyfoley)
- 71492b0 - Update Expo config setup to support baseline (roleyfoley)
- 4a80a36 - Application level templates (#851) (Michael Leditschke)
- 4d66a68 - typo in expo build (roleyfoley)
- d9c7359 - Align blueprint generation with output framework (#847) (Michael Leditschke)
- 24b9984 - mobile app typo fix (#849) (Michael Foley)
- 7c1b1dd - typo in alertchild config (#848) (Michael Foley)
- c6ee4a6 - CMK Lookup when creating ssh keys uses profile (roleyfoley)
- 61a8140 - remove diff in ES (roleyfoley)
- 307da4d - userpool remove identifier option (roleyfoley)
- 0862244 - add opsdata and appdata envvars (roleyfoley)
- 8a32c32 - add ref output to ssh key to make sure links are strict (roleyfoley)
- 474f85d - minor fixes (roleyfoley)
- a8bc55d - apigateway userpool role permissions (#835) (Michael Foley)
- 61bcea0 - cleanup legacy oai support (#831) (Michael Foley)
- 4477484 - baseline: fix legacy oai lookup for bucket policy (#829) (Michael Foley)
- 2ae4c88 - servreg: network link lookup (#826) (Michael Foley)
- 03604fe - servicereg: network lookup (#825) (Michael Foley)
- a23650f - ES: external vs local state lookup (#824) (Michael Foley)
- 07fde93 - basline: oai key lookup for bucket policy (#823) (Michael Foley)
- d472626 - apiusageplan: typo (#822) (Michael Foley)
- cc33249 - External link template inclusion (#821) (Michael Leditschke)
- 760b210 - apigw dependency on userpool component (#820) (Michael Foley)
- 40736b6 - user: resource lookup (#818) (Michael Foley)
- cf5224f - baseline legacy handling (#816) (Michael Foley)
- 255acf5 - fix: baseline oai legacy lookup (roleyfoley)
- 7ac67ba - fragment mode global (#814) (Michael Foley)
- 57bbd32 - update _context variable to use global scope (#813) (Michael Foley)
- c709e8d - ECS task variable scope (#812) (Michael Foley)
- 91eff40 - userpool: fix role lookup for iam permissions (#808) (Michael Foley)
- 1001f35 - api gateway userpool auth header (#807) (Michael Foley)