Unified Alerting in Grafana: Codifying Your Monitoring Stack with Helm and GitFlow

Infrastructure has shifted. You don’t manually click your way to production anymore—everything’s code. But somehow, alerting often lags behind. Grafana’s Unified Alerting model, introduced in version 8, changes that. Now, with proper use of values.yaml and Helm, you can manage alert rules, routing logic, contact points, and data sources like you would any other critical system component: version-controlled, peer-reviewed, and auto-deployed.

This post walks through how to declaratively manage your entire Grafana alerting pipeline with Helm—tying it into a GitFlow CI/CD strategy to lock in reliability, reproducibility, and observability from day one.

Automating AWS Cost Retrieval Using Python and Boto3

Effectively managing AWS costs demands accurate and timely data. Python, combined with AWS’s powerful boto3 library, makes it straightforward to programmatically access up-to-date pricing data directly from AWS. Below you'll find a clear, practical guide on retrieving AWS service pricing, with Amazon EBS snapshot costs provided as an illustrative example.

Grafana and Prometheus: Why Prometheus and Grafana and how to install it.

Grafana is an open-source platform used for monitoring, visualization, and alerting on metrics and log data. It allows users to create interactive and customizable dashboards that integrate data from various sources, such as Prometheus, InfluxDB, Elasticsearch, and many others. Grafana is widely recognized for its powerful visualization capabilities, enabling teams to gain real-time insights into the performance and health of their systems.

Sending Automated AWS Findings to Microsoft Teams Using Python

In fast-moving DevOps and FinOps environments, it’s easy to lose track of stopped or non-compliant cloud resources. While Lambda or cron jobs can detect these resources, what matters just as much is where those results go. I don’t want alerts buried in email or tucked away in an S3 bucket—I want actionable messages delivered straight to my team’s Microsoft Teams channel.

This post focuses specifically on using Python to craft and send structured messages to Microsoft Teams using an Incoming Webhook.

FinOps, The next step in "operational" development

In several of my recent posts, I’ve discussed using Lambda scripting to identify and clean up unused resources in AWS environments. While these tasks traditionally fell under DevOps, they are now part of a broader discipline known as FinOps. Short for Financial Operations, FinOps merges financial management with operational efficiencies to maximize the value organizations derive from cloud computing.

Although the FinOps Foundation formally established the concept in 2019, its principles date back to the early 2010s. During this time, businesses began focusing on managing cloud costs as the shift from capital expenditure (CapEx) to operational expenditure (OpEx) models made cost efficiency a priority.

The Importance of Tagging

Tagging in cloud environments, particularly in development settings, is a foundational practice that can transform the way organizations manage and optimize their resources. Beyond simple organization, tagging serves as a critical tool for financial operations, resource accountability, and operational efficiency. By implementing a robust tagging strategy, teams can address common challenges in cloud resource management, such as uncontrolled costs, unclear ownership, and untracked manual processes.

Managing EBS Snapshots with Lambda functions

Recently, we faced a situation where we found an account with over 25 TB of EBS snapshots, some of which dated back to 2017. These old snapshots had been piling up, creating substantial, unnecessary costs. We realized that without cleanup, costs would only increase, especially in our dev environment, where frequent changes to snapshots were generating excess storage overhead. This Lambda function was developed as a solution to automate the cleanup of outdated snapshots and refine our volume snapshot policy, allowing us to regain control over storage costs effectively.

Schedule Lambda Functions Using AWS EventBridge

Cleaning up stopped EIPs instances is a crucial maintenance task for AWS accounts to avoid unnecessary costs associated with EIP instances not attached to a resource. To streamline this process, I’ve set up two versions of a Lambda function to automate the identification and deletion of stopped instances.

Each week, one version of the Lambda function runs on Thursday to inspect stopped instances and log them for review, while another version runs on Saturday to delete the identified instances. This two-phase approach allows time to verify what instances are flagged for deletion before executing the cleanup.

Amazon EventBridge

To run the Lambda function on a specific day, you can use Amazon EventBridge (formerly CloudWatch Events). EventBridge allows you to create a scheduled rule that triggers the Lambda function at a specific time.

1. Navigate to EventBridge in the AWS Management Console.
2. Create a Rule:
   • Set the rule type to Schedule.
3. Define the cron expression or rate expression for the desired schedule. For exam
• To run at 7 AM every Thursday: cron(0 7 ? * 5 *)
• This cron expression means: "At 07:00 AM UTC on every Thursday."
4. Set Target to your Lambda function.

Step 2: Pass a Specific Set of Environment Variables

To use specific environment variables for a particular run:

1. Use AWS Lambda Versions and Aliases:

   • You can create different versions of the Lambda function, each with its own set of environment variables.
   • For example, you can create a version with inspection variables (DELETE_QUEUES=False) and another with deletion variables (DELETE_QUEUES=True).
   • Assign an alias to each version (e.g., inspection and deletion).

2. EventBridge Rule Target Configuration:

   • In the target configuration of the EventBridge rule, specify the alias for the Lambda version you want to run.
   • This allows you to run different versions of the Lambda function based on the schedule.

Step 3: Use Code Variables

If you need to dynamically set environment variables for each run:

1. Update Environment Variables in Code:

   • Modify the Lambda function code to accept environment variable overrides via the event payload.
   import os

   def lambda_handler(event, context): # Override environment variables if provided in the event delete_queues = event.get('DELETE_QUEUES', os.getenv('DELETE_QUEUES', 'True')).lower() == 'true' send_slack_message = event.get('SEND_SLACK_MESSAGE', os.getenv('SEND_SLACK_MESSAGE', 'True')).lower() == 'true' # Your logic here...

2. Create implementations of EventBridge with different versions of the code and variables:
1. Assuming you already have a Lambda function that checks for non-running EC2 instances and deletes them if required, you’ll need to create two separate versions:
1. Version 1: For inspection (running every Thursday, without deleting).
2. Version 2: For deletion (running every Saturday, with deletion enabled).

Version 1 of my code ONLY sends a slack notification

Version 2 of my code sends slack notifications AND deletes the instances

Step 4: Use Environmental Variables

These changes can also be done through the environment variables within the lambda job. (within the lambda function go to configuration-> environment variables)

By managing environment variables at the Lambda function level, you maintain a clear separation between inspection and deletion tasks, making it easy to configure and schedule them appropriately.

Stuck ArgoCD Deployment: How to Fix Finalizers causing Stuck Pods

When using ArgoCD to automate your application deployments, you might encounter an issue where a deployment gets stuck in the Progressing state. This often happens due to kubernetes finalizers not being released properly, preventing the related pods and resources from being destroyed. 

Finalizers are Kubernetes resources that ensure certain clean-up tasks are completed before an object is fully deleted, but if they malfunction or don’t get removed, they can block deletion and cause your deployment to hang indefinitely.

I’ll walk you through a step-by-step process to resolve this issue by manually removing the finalizers and then successfully deleting the stuck ArgoCD app.

Step-by-Step Guide

Step 1: Identify the Problem

When an ArgoCD app gets stuck in the "Progressing" state, you can confirm the issue by inspecting the status of the app and looking for finalizers that are preventing deletion.

You can do this by running the following command:

kubectl get app APP_NAME -o yaml

Look for the metadata.finalizers field. If you see finalizers listed but the app cannot progress to completion, that’s the cause of the problem.

Step 2: Patch the App to Remove Finalizers

To resolve the stuck state, you need to remove the finalizers from the ArgoCD app. You can do this by running the following **kubectl** command:

kubectl patch app APP_NAME -p '{"metadata": {"finalizers": null}}' --type merge

This command removes the finalizers from the app, allowing Kubernetes to bypass the finalizer logic and proceed with the app deletion.

Step 3: Patch the CRD (If Necessary)

In some cases, the Custom Resource Definition (CRD) associated with the app may also have finalizers that are causing the stuck state. To remove the finalizers from the CRD, use the following command:

kubectl patch crd CRD_NAME -p '{"metadata": {"finalizers": null}}' --type merge

This ensures that any finalizer present in the CRD is also removed, allowing for complete deletion of the resources.

Step 4: Delete the Stuck App

After patching the finalizers, you can safely delete the stuck app by using the following command:

kubectl delete app APP_NAME

Step 5: Delete the CRD

If needed, you can also delete the CRD after patching its finalizers:

kubectl delete crd CRD_NAME

This should fully remove the app and any related resources, resolving the stuck progressing state.

Deployments getting stuck in the Progressing state in ArgoCD can often be traced back to finalizers not being properly removed, which blocks resource deletion. By manually patching the app and any related CRDs to remove finalizers, you can resolve this issue and successfully delete the resources.

If you’re facing this issue frequently, consider reviewing the finalizer behavior in your environment and ArgoCD configurations to ensure that finalizers are being handled correctly during normal operations. Properly configured finalizers will help avoid these kinds of issues in the future.

Account you added is not fully configured (Outlook 2010 style)

If you’re encountering the error message, “The account you added is not fully configured. It might not work properly until re-configured correctly,” when trying to open Microsoft Outlook 2010, you’re not alone. This issue, along with related errors, can prevent Outlook from functioning correctly. Below, we’ll walk through the symptoms, causes, and steps to resolve this problem.

Symptom

When attempting to open Microsoft Outlook 2010, you might receive the following error message: 

Error: "The account you added is not fully configured. It might not work properly until re-configured correctly."

Additionally, you may encounter a secondary error that reads:

Error: "The file C:\Users\USERID\AppData\Local\Microsoft\Outlook\PROFILE.ost is in use and cannot be accessed. Close any application that is using this file, and then try again. You might need to restart your computer."

Causes and Solutions

Here are the possible causes of these errors and the steps you can take to resolve them:

MS Lync Interference:

Solution: Start by closing Microsoft Lync (now known as Skype for Business) and then attempt to open Outlook again. 

Corrupted mapisvc.inf File:

Solution: If closing Lync does not work, ensure that it is completely closed. Then, navigate to `C:\Windows\System32` and locate the file named `mapisvc.inf`. Delete this file and restart Microsoft Outlook. This step should help resolve the issue if the `mapisvc.inf` file was causing the conflict.

Corrupt Windows Profile:

Solution: If the issue persists after following the above steps, it’s possible that the user’s Windows profile is corrupt. In this case, the profile may need to be recreated. 

Final Thoughts

These steps should help you resolve the "Account Not Fully Configured" error in Microsoft Outlook 2010. By addressing potential conflicts with Microsoft Lync, removing problematic files, or recreating the user’s profile, you can get Outlook running smoothly again.

If you continue to experience issues, consider consulting the referenced knowledge base articles for more in-depth troubleshooting or reaching out to your IT support team for assistance.

Moving Files from OneDrive to Sharepoint Drive

With the recent implementation of Microsoft Office 365 for the city of Bertram, IA, we have a great opportunity to consolidate and centralize all of our documents in SharePoint. Moving files to a SharePoint site allows everyone in the organization to have easy access to the most up-to-date resources. This article provides a step-by-step guide on how to move your files from OneDrive to a SharePoint site using the "Move to" function, ensuring a smooth transition.

Why Move Files to SharePoint?

As city employees, we often need to share documents across different departments. SharePoint offers a centralized document management system that allows multiple people to collaborate on the same files, making it easier for everyone to access important documents. Additionally, SharePoint’s version control and permissions features help maintain document integrity and security.

Step-by-Step Guide

1. Select the Files or Folders

• Start by navigating to your OneDrive account, where your current files and folders are stored. 
• Select the files or folders you want to move to SharePoint by clicking on the checkbox next to each item.
• Once the items are selected, click on the Move to option located in the toolbar.

2. Choose Your Destination Site

•  After selecting Move to, a dialog box will appear showing available destinations. You’ll see a list of SharePoint sites under the Quick Access section.
• Select the SharePoint site where you want to move the files. If you're currently on a SharePoint site, you'll also see options like your OneDrive and other sites.

 If you don't see your desired SharePoint site listed, it could be due to organizational restrictions on cross-site movement. Contact your SharePoint or global admin to check settings that might restrict access to certain sites.

3. Pin Your SharePoint Site for Quick Access

•  If you frequently move files to the same SharePoint location, consider pinning the site to Quick Access. Navigate to the SharePoint site, click on the **More places...** link, and use the **Pin to Quick Access** feature to add the site to your left-hand navigation for easier future use.

4. Move the Files

•  Once you've selected your destination, click **Move here** to start transferring the items to the chosen SharePoint site.

Depending on the size of the files and your network connection, this process could take a few moments. 

This migration to SharePoint will enhance collaboration, streamline document management, and ensure that everyone can quickly find the information they need. If you have any questions or run into issues during the file transfer process, feel free to reach out to the IT department for assistance.

How to take ownership of a file or folder in Windows 7

When transferring files or folders from an old machine to a new one, you may encounter issues with permissions that prevent you from editing, deleting, or even opening certain files. This often happens when documents or favorites are moved from a reimaged computer. If you're not the owner of these files, you won't be able to change their permissions either. Fortunately, Windows allows you to take ownership of files and folders, enabling you to regain control.

Why Taking Ownership is Important

If you’re unable to access or modify a file or folder, it’s likely because the ownership is assigned to another user or system account. Without ownership, you can't edit the permissions, which restricts your ability to make changes. This can be especially frustrating when dealing with important documents or personal files that you need to access or modify.

How to Take Ownership of a File or Folder

Follow these steps to take ownership of a file or folder in Windows:

1. Navigate to the Folder or File:

   - Browse to the file or folder you want to take ownership of. Right-click on it and select Properties.

2. Access Security Settings:

   - In the Properties window, click on the Security tab at the top.

3. Open Advanced Security Settings:

   - Click on the Advanced button to open the Advanced Security Settings window.

4. Select the Owner Tab:

   - In the Advanced Security Settings window, select the Owner tab.

5. Edit Ownership:

   - Click on the Edit button. This will allow you to change the owner of the file or folder.

6. Assign Ownership:

   - Select your user ID from the list or choose the desired user. Ensure you check the option Replace owner on subcontainers and objects if you’re taking ownership of a folder with subfolders and files.

   - Click OK to apply the changes.

7. Confirm and Close:

   - After confirming the ownership change, click OK on the confirmation window. Then, close all open properties windows.

How to Access Outlook Mail from the Outlook 365 Welcome Screen: Three Simple Methods

If you're new to Outlook 365 or just need a quick guide on how to access your email, this post will walk you through three ways to reach your inbox—whether you're using the web version, installing Outlook on your computer, or setting it up on your mobile device.

Method 1: Access Outlook from the Outlook 365 Welcome Screen

Step 1: Navigate to the App Menu

When you first sign in to Outlook 365, you’ll be on the welcome screen. To access your email, locate the grid of squares in the top-left corner of the screen. This is your app launcher.

Step 2: Open More Apps

Click on the grid icon, and a menu will appear with several apps. If you don’t see Outlook right away, scroll down and click on **"More Apps"** to reveal additional options.

Step 3: Select Outlook

From the list of apps, click the Outlook* icon. This will open the Outlook application and take you directly to your email inbox. Now you’re ready to read, compose, and organize your messages.

Method 2: Install Outlook on Your Computer

If you prefer accessing your email directly from your computer, you can install Outlook as part of the Microsoft 365 suite. Here’s how:

Step 1: Open the Installation Menu

From the Outlook 365 home screen, look toward the upper right-hand corner for the Install and more button.

Step 2: Choose Microsoft 365 Apps

On the drop down, choose "Install Microsoft 365 Apps" to access the suite of apps included in your license.

Step 3: Install Office

Under the section labeled Apps and Devices, click "Install Office" This will download an installable file or a `.exe` to your computer.

Step 4: Run the Installer

Once the download is complete, locate the file in your downloads folder and double-click to run it. Follow the on-screen instructions to install Outlook and other Microsoft Office products on your device.

Once installed, you can access Outlook directly from your desktop for an easier way to stay connected to your email.

Method 3: Install Outlook on Your Mobile Device

Prefer to check your email on the go? You can easily install the mobile version of Outlook through the Microsoft 365 mobile app. Here’s how:

Step 1: Open the Installation Menu

As with the previous method, start by clicking the Install and more button in the upper-right corner of the Outlook 365 home screen. 

Step 2: Select Microsoft 365 Mobile App

In the dropdown menu, you’ll see an option to "Install Microsoft 365 Mobile App". Click this option to proceed.

Step 3: Scan the QR Code

A new page will appear with a QR code. Open the camera app on your phone and scan the QR code displayed on your screen. This will direct you to the app store for your device, where you can download the Microsoft 365 mobile app.

Step 4: Install the App

Once redirected, install the app and log in with your Microsoft 365 credentials. You’ll now have access to Outlook and all your Office apps directly on your mobile device.

Now you know three ways to access your Outlook email—via the Outlook 365 web app, installing Outlook on your computer, or setting it up on your mobile device. Whether you’re at home or on the go, you’ll always be connected to your inbox!

Disable Automatic Deletion of Meeting Requests

When using Microsoft Outlook, you might notice that meeting requests and notifications are automatically deleted from your inbox once you've responded to them. This is the default setting, and while it helps keep your inbox clean, it can also lead to the loss of important information. If you prefer to keep these meeting requests for reference, you can easily disable this automatic deletion feature.

Steps to Disable Automatic Deletion of Meeting Requests

Follow these simple steps to turn off the automatic deletion of meeting requests in Outlook:

1. Open Outlook and click on the File tab.
2. Select Options from the menu.
3. In the Outlook Options window, click on the Mail tab.
4. Scroll down to the Send messages section.
5. Deselect the checkbox labeled "Delete meeting requests and notifications from Inbox after responding."
6. Click OK to save your changes.

What Happens Next?

With this setting disabled, any meeting requests or notifications you accept will remain in your inbox instead of being automatically moved to the Deleted Items folder. This allows you to keep track of all your meeting-related communications and have easy access to them when needed.

Final Thoughts

Customizing Outlook to fit your workflow can greatly enhance your productivity. By disabling the automatic deletion of meeting requests, you ensure that all important communications are easily accessible and not accidentally lost. If you ever need to revert to the default setting, simply follow the steps above and re-enable the checkbox.

This small adjustment can make a big difference in how you manage your meetings and notifications in Outlook, giving you more control over your inbox.

Enable/Disable automatic download of ALL pictures in MS Outlook for Windows

When it comes to managing your email settings in Outlook, particularly regarding the automatic download of pictures in emails, it's essential to proceed with caution. Changing these settings can expose you to potential risks, including malware, viruses, and an increase in targeted junk emails. Here's a guide on how to adjust these settings safely and why the default settings are there to protect you.

Why You Should Be Cautious

Spammers often use embedded pictures in emails to confirm that your email address is active. When you automatically download these images, you signal to the spammer that your address is valid, leading to more targeted junk emails in your inbox. Additionally, downloading pictures automatically can open the door to malware or viruses that might be embedded within these images.

Recommended Approach

The safest method to manage picture downloads in Outlook is to keep the default settings, which are designed to protect you. These settings block automatic downloads, allowing you to enable pictures on a per-message basis if you trust the sender. 

How to Manually Enable Picture Downloads

If you still decide that you want to change the default settings, here’s how you can do it:

1. Open Outlook.
2. Navigate to File > Options > Trust Center.
3. Click on the Trust Center Settings button.
4. Select Automatic Download** from the left-hand pane.
5. Uncheck the box that says, "Don't download pictures automatically in HTML e-mail messages or RSS items."
6. Click OK to save your changes.

Final Thoughts

While it might be tempting to enable automatic picture downloads for convenience, remember that the default settings are in place for your protection. Always exercise caution when adjusting these settings, and only download pictures from trusted sources. 

By keeping your Outlook settings secure, you can minimize your exposure to potential threats and keep your inbox safer from unwanted spam and malware.

Outlook Cannot Search Email on VDI

If you’re working on a Virtual Desktop Infrastructure (VDI) and experiencing issues with searching emails in Outlook, you're not alone. It's a common problem with a few specific causes and solutions. Below, we'll walk through the common symptoms, causes, and actions required to resolve these issues.

Symptom 1: Outlook Cannot Search Emails on VDI

Problem: You’re unable to search through emails in Outlook while on a VDI session.

Cause: Outlook is attempting to index files in order to search them, but the indexing process never completes. This is a common issue on VDIs due to the nature of their storage and session management.

Action Required:

Disable Cached Exchange Mode:

 Cached Exchange Mode is not supported on standard or dynamic VDI sessions because it requires writing and storing the cached data locally, which isn't preserved from session to session. Typically, VDI sessions are set to run in online mode to avoid this issue.

   

   Here’s how to disable Cached Exchange Mode:

1.    Open Outlook and click the File tab at the top.
2.    Select Account Settings and click on Account Settings from the dropdown.
3.    Under the E-mail tab, double-click the user’s email address.
4.    Uncheck the option Use Cached Exchange Mode under the Server settings.
5.    Click Next and then Finish.
6.    Restart Outlook.
7.    Verify that the search functionality is now working as expected.

   Note: This process might take longer than anticipated to apply.

Symptom 2: Windows Search Engine Disabled Warning

Problem: When you click into the Search bar, you receive a message saying, “The Windows Search Engine is currently disabled. Outlook will not be able to provide fast search results using the Instant Search functionality unless this service is running.”

Cause: The Windows Search function has been disabled on standard VDI sessions by the system administrators due to its heavy usage of processor and network bandwidth, which can slow down performance. 

Action Required:

Clear Windows Search Warning:

Simply click OK on the pop-up message. The search might take slightly longer than usual, but it will still work. The message should only appear once per session, but it may reappear in future sessions or after a VDI profile rebuild.

You can also check the box "Do not show this message again", though this may not prevent it from reappearing after a session reset.

Symptom 3: Search Words Are Not Highlighted

Problem: When searching emails, the search terms are not highlighted within the email content.

Cause: Highlighting search words requires indexing, which is disabled on standard VDI sessions due to the aforementioned resource concerns.

Action Required:

Search Individual Emails:

You can manually search and highlight words within individual emails. Here’s how:

1.    Open the email from the search results.
2.    Click **Find** (or press **F4**) under the **Editing** section on the **Message** menu.
3.    This will allow you to search for the specific word within the email, and it will be highlighted for easier viewing.

Final Note:

As of March 11, 2015, Windows Search has been re-enabled for all pooled VDI sessions, which should resolve some of the search-related issues. However, symptoms 2 and 3 still apply if the search service is disabled.

By following these steps, you should be able to troubleshoot and resolve most search-related issues within Outlook on a VDI session. If problems persist, it might be necessary to consult with your IT department for further assistance.

Creating a Chef Automate Workflow Pipeline

My company's Chef Automate workflow pipelines were designed as a part of a middleware infrastructure project. The project required three auto scaled instances each sitting behind their own AWS ELB. The project enlisted the services of five teams, each with their own specialization for the project. The Infrastructure team created AWS Cloudformation Templates (CFTs), CFT cookbooks, VPCs, security groups and ELBs. The middleware team created the cookbooks for the respective instances including the underlying base cookbooks which will be utilized by or company for future projects. The QA team created and iterated upon smoke and functional testing for single instances and their communication between other instances. Finally, the Security team determined the compliance testing necessary for instances' and helped create proper security testing which would stop pipelines should servers fall out of compliance.

When designing the infrastructure and procedures for my company's Chef Automate workflow pipelines we came across a number of hurdles.

First, when provisioning instances via our CFT cookbook, the nodes are bootstrapped with chef client with a user data script. After chef client is installed, via the script, the nodes will run their first-boot.json. This contains the name of the cookbook for the the current project pipeline. If the recipe fails, however, during the initial bootstrapping process the node will not be attached appropriately to chef server.

This bootstrapping process is a necessary component for AutoScaled instances. If new instances are booted, as a part of an AutoScale procedure, those nodes will require a bootstrap procedure be run with the latest cookbooks. Therefore, testing of the cookbook will need to be independent of the CFT deployment steps.

In order to bypass this issue my company developed a pipeline that calls on, not only, our internal CFT provisioning cookbook but also test kitchen for our acceptance nodes.

By using kitchen-ec2 we are able to converge and destroy our cookbooks in acceptance to verify their viability before passing them to our user data script. This is made easier with the inclusion of the delivery-sugar cookbook. Delivery-sugar contains resources that allow for the creation, convergence and destruction of EC2, Azure, Docker and Vsphere instances using the delivery_test_kitchen resource.

My company is currently calling on kitchen-ec2 for instance creation. EC2 currently requires ALL of the following components to run successfully.

Test Kitchen Setup (Acceptance Stage Provisioning):

In order to enable this functionality please perform the following prerequisite steps.

Add ALL of the following items to the appropriate data bag within your Chef Server 

You can convert the private key content to a JSON-compatible string with the following command. 

chef exec ruby -e 'p ARGF.read' automate_kitchen.pem >> automate_kitchen.json 

Since the private key should be secured this data bag should be encrypted. In order to add an encrypted databag to the chef server you must first have proper access to the chef server, which is necessary for a knife command to be run. After this permission is in place you must run the following command.

knife data bag create delivery-secrets -- --secret-file encrypted_data_bag_secret

Where  is the name of your enterprise,  is the name of your organization and  is the current name of the pipeline you are creating. 

In order to decrypt this data the encrypted_data_bag_secret file, used to encrypt the data bag, must be added to your Chef Build servers at the following location.

/etc/chef/

Once these components are deployed Customize your kitchen YAML file with all the required information needed by the kitchen-ec2 driver driver.

NOTE  This kitchen.yml file will be the one found in your .delivery/build_cookbook and not the one found under your project cookbook

Delivery-sugar will expose the following ENV variables for use by kitchen.

• KITCHEN_INSTANCE_NAME - set to the - values provided by delivery-cli 
• KITCHEN_EC2_SSH_KEY_PATH - path to the SSH private key created from the delivery-secrets data bag created from the step above. 

These variable may be used in your kitchen YAML like the following example:



Once the prerequisites are in place you can use delivery_test_kitchen within your .delivery/build_cookbook/provision.rb to deploy instances through test kitchen.




Trigger a kitchen converge and destroy action using Ec2 driver and pointing it to .kitchen.ec2.yml in delivery.

NOTE: When adding a repo_path my companychooses #{workflow_workspace_repo}/.delivery/build_cookbook/. This is by preference and the location of the .yml file can sit wherever the user requires.



Trigger a kitchen create passing extra options for debugging 



Trigger a kitchen create extending the timeout to 20 minutes.

 

Since we are only using kitchen in our acceptance node my company must add the following logic to verify test kitchen is not used outside of the acceptance stage. (workflow_stage is a resource provided by delivery-sugar)

Version Pinning

The second issue we were presented with in creating our workflow pipelines came in the pinning of our environments. 

If using base cookbooks for multiple projects, pinning should not be done on the base cookbook itself. Since cookbooks are pinned at an environment level if the base cookbook is pinned at the environment and then updated, that base cookbook update will in effect alter all projects using it in that environment (acceptance, union, rehearsal delivered. To prevent this pinning from taking place, through workflow, under

.delivery/build-cookbook/provision.rb 

comment out

delivery-truck::provision

In turn if we version pin only the role cookbook at the environment level, being project specific, any changes made to the role cookbook should not have an effect on any other project.

This does mean that in order for a base cookbook to be updated in a project its version must be changed in the role cookbook. So for every underlying cookbook change the role cookbook will need to be version bumped. This is a much more manual process, but it will provide protection from projects breaking with a change to one base cookbook.

This also has the added benefit of version controlling any version bumps we have in our environments for a given projects node. Since the only version pins in an environment fall on the role cookbook, all other changes to versions should be controlled through the role cookbooks metadata and delivery cli commands. These commits can be tied back to individual users and version changes which will better stabilize the environments.

The leading measure in Workflow, if base cookbooks are not project specific, should sit with role cookbooks. These cookbooks should be used to provision servers, and version pin underlying cookbooks, when going through the Union, Rehearsal and Delivered stages of the Chef Automate Workflow to separate project version pinning. 

Setting up Metadata.rb, Provision.rb, Kitchen.yml and Berksfile in .delivery/build_cookbook

NOTE: before adding the workflow provisioning steps to the build_cookbook please add the project cookbook to the chef server, Either through automate workflow or through a knife command. If the project cookbook is not available upon the first run of the pipeline it will fail when trying to download cookbooks

With these two problems resolved, and explained, it is now time to setup the rest of our workflow pipeline.

We will start by modifying our Berksfile within .delivery/build_cookbook/. Since we will be calling on cookbooks that are currently stored in the chef server we will need to make sure that the workflow pipeline can reach out to it to find cookbooks. We do this by adding the chef server source

Next, we will modify our metadata.rb file. We need to make sure we are calling in delivery-sugar, delivery-truck, the current project cookbook for the pipeline and the cookbook we are using to provision our servers. 

NOTE: We only need to call the provisioning cookbook here if this is the role cookbook

We will also configure our kitchen.yml (which we have named here as kitchen.ec2.yml) as we described in the steps above. This file is used for our kitchen converge and destroy in our acceptance provisioning stage. 

NOTE: do not forget to change the cookbook we are calling in the kitchen.yml to reflect the current cookbook we are sitting in. (See run_list) 

Finally, we will modify our provision.rb file. Depending on our environment (role cookbook vs base cookbook or wrapper cookbook. Please see the documentation for version pinning for further explanation).

In a ROLE cookbook We will call upon the provisioning cookbook if we are in the union, rehearsal or delivered stage. This check can be made using the delivery-sugar resource workflow_stage which will call the current stage the pipeline is currently running in.

We will also call on the delivery-truck::provision cookbook to pin our environment. 

NOTE: the delivery-truck::provision recipe is included AFTER the run of our provisioning cookbook, See the section on versioning for more information)

If we are NOT IN A ROLE COOKBOOK delivery-truck::provision will not be called. We will also not need to include the recipe for provisioning in union, rehearsal or delivered. To keep things simple, and to prevent us from having to make too many modifications to our code, we will simply add a warning message in place of the provisioning cookbook includes. 

Once these changes are saved we can version bump our project cookbook, either through the metadata.rb file or the delivery command, and run delivery review.

NOTE: this version bump is done in the PROJECT COOKBOOK not the build cookbook. 

This will push the cookbook into Automate and start the Chef Automate Workflow Pipeline.