with special guest: Ian McKay, Former2 Developer
While working on AWS services, I always wondered why there was no tool I could use to convert existing services/configurations to CloudFormation templates.
It would be so practical to have this as an option on any given AWS service console, so I asked AWS’ awesome support and our TAM several times. Unfortunately, I always received the same reply – it is a great idea – and that was the end of it.
So, when I finally found such a tool, my joy knew no bounds. Behold “Former2”. I was so impressed by the solution, created by Ian McKay, that I caught up with him for a quick chat:
Q&A with Former2 Developer
Q: Tell me a little bit about yourself:
A: I am working in Sydney as the Cloud Lead at Kablamo, and also as an AWS Hero. I enjoy speaking at meetups, co-hosting podcasts, and writing my thoughts about AWS on my blog at onecloudplease.com.
Q What can you tell me about the project?
A: Former2 allows you to generate Infrastructure as Code outputs from your existing resources within your AWS account (that’s 124 services, mind you).
By making the relevant calls using the AWS JavaScript SDK, Former2 will scan across your infrastructure and generate an output to your liking: CloudFormation, Terraform, CDK Cfn Primitives/Terraform, and other languages such as Python, Java, C#, & Pulumi – TypeScript, and finally, even as a draw.io diagram.
I worked previously on a couple of other projects, but really took this project to heart once I saw its usability. Former2 is actually a fork of an old project used by AWS back in the day, about 10 years ago, so I took it, and continued with it as an open source project.
It can be used via a web interface as well as a command line tool, or you can even host your own http server. Former2 does not create any resources within your AWS account and will keep your AWS credentials locally (in memory) without sending it to the cloud or somewhere else. You can exclude services, sort outputted data or filter by searching for a specific application.
Former2 takes advantage of the AWS SDK for JavaScript in the browser. It makes Get/List calls to determine the resources within your AWS account and maps those SDK responses to the various template formats. Former2 derives its name from the previous CloudFormer tool.
Former2 has evolved to fully support all AWS Services. Both inexperienced and experienced AWS developers can use Former2 to quickly construct Infrastructure as Code templates for their companies or customers. You can also add an option to add a deletion policy to your CF (delete/retain)
Q: What does it cost?
A: Since it is an open source tool, it will not incur any charges beside the API calls to AWS.
Q: Why open source?
A: Besides being an awesome concept, as an open source software, anyone can contribute to Former2, meaning enhancements or bug fixes can be shared by the community.
Former2 has more than 30 pull requests and more than 100 issues raised, which has significantly contributed to shaping the project. Many of these contributions are raising or fixing bugs that I otherwise wouldn’t find due to the sheer scale of the AWS products and services. I thank everyone who has contributed directly or indirectly to the project.
Building trust is also important. Handling AWS credentials is a high-risk activity, and Former2 is designed to never send these credentials to an external server that isn’t an AWS API endpoint. Being fully transparent about the code you are running is essential to show that you understand the sensitivity of the data.
I thank Ian for his time, responsiveness on GitHub, and for creating this awesome solution. Now let us take a closer look.
CloudFormation
Before we start, a couple of words about CloudFormation.
AWS CloudFormation is an Infrastructure as Code (IaC) service, which means you can use text files (JSON or YAML) to define a set of resources to be deployed without worrying about semantics of how they are deployed.
Using AWS CloudFormation truly helps with consistency and repeatability.
What is Former 2?
Former2 has a couple of uses, depending on how you look at it:
- It can be a replication tool that you can create or move resources from one environment to another.
- It can be a backup tool, but not in its traditional sense.
While Former2 provides a current snapshot of your AWS infrastructure, when redeploying in many services and configurations it may not be relevant the second time.
Security groups, local IPs, AMI IDs or any resources created by Stackset are randomly assigned, and as such, it is not likely you will get the same ones when restoring. Former2 actually provides you with a list of resources you had and how they interacted with other services/resources.
This precious information can be a godsend for those who lost access to their account, or if the account configuration is destroyed by a rough actor, especially if you don’t really know what you had inside your account.
So, to recap, Former2 provides you with a backup/list of your infrastructure-service connections and configuration, but not a backup of your data (the content of the EC2 server, for example).
Collaboration
While debugging some issues I noticed on Former2, I worked with Ian and we managed to solve those issues (different behavior on edge scenarios) by communicating via his GitHub account. This is the power of open source.
While working on the Former2 CLI, I noticed that it takes the AWS user identity from the ~/.aws/credentials file. This means that if you have one user or a couple of local users, you can use it without any issue.
But what if you are using assume role on the ~/.aws/config file? What if you are getting your accounts from your organization? With this setup you are stuck, and this great adventure comes to a screeching halt before you even had a chance to check out this solution.
Fret not, young grasshopper – we have the power of the code. I looked into the Former2 source code (NodeJs) and found that you can actually use 3 different options:
- Using the default configuration in the ‘AWS_SHARED_CREDENTIALS_FILE’ environment variable.
- Add a custom profile using the ‘AWS_PROFILE’ environment variable.
- Directly load the AWS.SharedIniFileCredentials provider.
var creds = new AWS.SharedIniFileCredentials({profile: 'myprofile'});
AWS.config.credentials = creds;
More details can be found here.
Former2 GitHub page.
Use case
My use case was a bit more complicated: I had a couple of hundred accounts that obviously cannot and will not be configured with an access key and secret key, so I needed to get temporary credentials for each account.
The next thing was to understand where and how to store them:
Where was obvious – on the ~/.aws/credentials file.
How needed more brainstorming. This is the main file, undoubtedly one of the most important files, which I can’t use to store hundreds of temporary credentials. It will get real messy, real fast.
Let’s not forget that the default duration of IAM role session is 900 seconds (15 minutes), and can be up to 12 hours.
So in the course of an hour, this file can potentially contain thousands of active redundant credentials. Searching through this file will become a long and wasteful task, memory wise, and can pose a security risk if used inappropriately.
So, after putting my thinking cap on, I found a clever solution (if you don’t mind me saying so):
- I will loop all my accounts IDs, keep them in a list and request an STS for each account.
- I will then rewrite the default credentials file with the default profile and a temporary profile that holds the STS details I received previously as a parameter.
- The default profile (which holds sensitive data that can’t be placed on Git), will be placed on a vault (by Hashicorp, which is a cool project on its own, but it’s another story for another time) so the details will not be visible at all.
- This way, the file is nice and clear, dynamic, only contains one assumed credential every time it is called and will not be alive more than a few minutes, until the Former2 finishes its turn.
- At the end of the process, the file is cleaned for the next run, and eventually will contain only the default profile as started. No messy credentials file, no old details to clean, no info that can be abused.
STS
For those who are wondering what STS means:
STS is a Security Token Service that can provide you, as a user, the ability to assume roles via API calls. You can use many SDKs to perform the task you need to with either Python, Java, Node.js, GO, and many more other languages. More information can be found here.
Running Former2 CLI
There are a couple of ways to use this tool, either by installing it from brew.io for mac os users, as a Docker container, or by getting it from a NPM repository.
I used the NPM method. If you don’t already have NPM installed, go ahead and check out your preferred install method.
To install the CLI tool, run this command:
npm install -g former2
The g flag will install packages globally onto your npm cache folder on your machine.
If you want to update, run this command for the latest version:
npm update -g former2
or
npm update -g former2@0.2.67
for a specific version.
To run the former2 command, you would use (for a single service):
former2 generate --services "EC2" --region "eu-west-1" --profile “profile_name” --output-cloudformation "cfn.yml" --proxy protocol://Host_url:port
Former2 can scan services which are region-specific, globally, or backup all AWS services. Mind you this can take some time, depending on how many resources you have on each service.
For information on additional options that can be used, see the help section:
Here is an example of the output:
