Create your own developer environment in AWS with the help of Pulumi

Introduction

Let’s say you want to create a developer environment in AWS for your team or for a workshop. You want to have a fully functional IDE in the cloud that you can access from anywhere. You want to be able to easily create multiple developer environments for different projects or teams. How would you do that? You can do that by just using the AWS console and create there different develop environment. A better solution would probably be to use a tool that allows you to define your infrastructure as code and then deploy it to your cloud provider. A well known tool for that is Terraform, with Terraform you can define your infrastructure using a declarative language and then deploy it to your cloud provider. But there is also another tool that allows you to do that, Pulumi. Pulumi is an open-source infrastructure as code tool that allows you to define and manage your cloud infrastructure using familiar programming languages like Python, JavaScript, and Go.

I think this is a big advantage because you can use the same programming language that you use for your application code to define your infrastructure. This makes it easier to understand and maintain your infrastructure code, especially for developers who are already familiar with the programming language.

In this blog post, we will create a developer environment in AWS using Pulumi. The idea is that in the end you will have a fully functional developer environment in AWS based on the open source project code-server. This project allows you to run Visual Studio Code in the browser and is perfect for remote development.

As stated before, we will use Pulumi to create the necessary infrastructure in AWS. Pulumi allows you to define your infrastructure using code and then deploy it to your cloud provider. This way you can easily create, update, and delete your infrastructure in a reproducible way. In this blog post, we will use Python to define our infrastructure, but you can also use other programming languages like JavaScript or Go.

To create our developer environment in AWS, we basically need to do the following steps:

1. Create a new EC2 instance
2. Install the required tools
3. Configure the instance

Prerequisites

Before we start, you need to have the following prerequisites:

• An AWS account (you can create one for free at https://aws.amazon.com/)
• IAM user with the necessary permissions to create the infrastructure (note: you should never use your root account for this)
• Token for the IAM user to authenticate with AWS, see https://www.pulumi.com/registry/packages/aws/installation-configuration/#option-2-create-by-hand
• Pulumi installed https://www.pulumi.com/docs/iac/download-install/
• Python installed (a recent version)

If you don’t have these prerequisites yet, please install them before continuing.
When you have all the prerequisites, you can continue with the next section.

Creating the developer environment

Creating the Pulumi project

First, we need to create a new Pulumi project. You can do this by running the following command in your terminal:

pulumi new aws-python

If you want to verify that the project was created successfully, you can run the following command:

pulumi up

This will show you the resources that will be created in your AWS account. If everything looks good, you can confirm the creation by typing yes and hitting enter.
If you encounter any issues, please refer to the Pulumi documentation for help: https://www.pulumi.com/docs/. Also make sure your AWS credentials are set up correctly.

Creating the AWS EC2 instance

Now that we have our Pulumi project set up, we can start creating the infrastructure. The first thing we need to do is create a new EC2 instance in AWS. To do that we need to modify the file __main__.py. This file contains the Python code that defines our infrastructure.

If you open the file __main__.py in your favorite code editor, you will see that it contains some boilerplate code that creates a new s3 bucket. We can remove all those lines because we don’t need the S3 bucket anymore. We need to create a EC2 instance, but how do we do that?

Luckily, Pulumi has excellent documentation that explains how to create an EC2 instance in AWS using Python. You can find the documentation here: https://www.pulumi.com/docs/reference/pkg/aws/ec2/instance/. On this page, you can see all the available options that you can use to create an EC2 instance and also some examples. Let’s use the basic example as a starting point. We can copy the example code and paste it into our __main__.py file. It will look like this:

import pulumi
import pulumi_aws as aws

ubuntu = aws.ec2.get_ami(most_recent=True,
    filters=[
        {
            "name": "name",
            "values": ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"],
        },
        {
            "name": "virtualization-type",
            "values": ["hvm"],
        },
    ],
    owners=["099720109477"])
web = aws.ec2.Instance("web",
    ami=ubuntu.id,
    instance_type=aws.ec2.InstanceType.T3_MICRO,
    tags={
        "Name": "HelloWorld",
    })

Let me explain this code quickly, so you understand what it does:

1. We import the necessary modules from Pulumi and the AWS provider.
2. We get the latest Ubuntu AMI using the get_ami function.
3. We create a new EC2 instance using the Instance function.
4. We specify the AMI, instance type, and tags for the EC2 instance.
5. We save the EC2 instance in a variable called web.
6. That’s it!

Now we can deploy our infrastructure by running the following command:

pulumi up

Let’s do that now and see what happens.

If you now go to the AWS console and navigate to the EC2 dashboard, you should see a new EC2 instance running. Congratulations, you have created your first EC2 instance using Pulumi!
But we don’t want to create a simple EC2 instance, we want to create a developer environment. So let’s continue with the next steps.

Running the install script

The next step is to run the install script that will install the necessary software on the EC2 instance. This script will install Docker, Docker Compose, and the code-server. We can do this by using the UserData property of the EC2 instance. The UserData property allows you to specify a script that will be executed when the instance is launched. We can use this to run our install script.

Let’s modify our __main__.py file to include the install script. We can do this by adding the following lines of code after the import statements:

with open("boot_script.sh", "r") as file:
    boot_script = file.read()

This will read the contents of the boot_script.sh file into a variable called boot_script. Now we can add the UserData property to our EC2 instance like this:

web = aws.ec2.Instance("web",
    ami=ubuntu.id,
    instance_type=aws.ec2.InstanceType.T3_MICRO,
    user_data=boot_script,
    user_data_replace_on_change=True,
    tags={
        "Name": "HelloWorld",
    })

This script will update the package list, install Docker, Docker Compose, and curl, start the Docker service, and then install the code-server using the install script from the code-server website.

Now we can deploy our infrastructure again by running pulumi up again. Let’s do that now and see what happens.

It should stop the current instance and restart it with the new UserData script. If you now go to the AWS console and navigate to the EC2 dashboard, you should see that the instance is stopped and then started again.

If you face any problems, you can find the code up to now on Gitlab

Opening the firewall

The next step is to open the firewall so that we can access the code-server from the browser. We can do this by adding a security group to the EC2 instance. A security group acts as a virtual firewall that controls the traffic to and from the EC2 instance. We can specify the inbound and outbound rules of the security group to allow or deny traffic.

Let’s modify our __main__.py file to include the security group. We can do this by adding the following lines of code before the web variable:

# Create a security group allowing HTTP access
security_group = aws.ec2.SecurityGroup("web-sg",
    description="Allow HTTP access",
    ingress=[
        {
            "protocol": "tcp",
            "from_port": 443,
            "to_port": 443,
            "cidr_blocks": ["0.0.0.0/0"],
        },
    ],
    egress=[
        {
            "protocol": "-1",
            "from_port": 0,
            "to_port": 0,
            "cidr_blocks": ["0.0.0.0/0"],
        },
    ]
)

This will create a new security group called web-sg that allows inbound traffic on port 443 from any IP address. Now we need to associate this security group with our EC2 instance. We can do this by adding the vpc_security_group_ids property to the EC2 instance like this:

# Create the EC2 instance
web = aws.ec2.Instance("web",
    ami=ubuntu.id,
    instance_type=aws.ec2.InstanceType.T3_MICRO,
    user_data=boot_script,
    vpc_security_group_ids=[security_group.id],
    tags={
        "Name": "HelloWorld",
    })

Now let’s also add a pulumi export to the end of our code so we can print the public IP address of the EC2 instance:

pulumi.export("public_ip", web.public_ip)

Now we can deploy our infrastructure again by running pulumi up again. After Pulumi is done applying your changes you can go to https://<<instance-ip>>:443 in your browser, you should see the code-server login page. It will ask for a password, which is MyNotSoSecretPassword34789# by default. This is not really secure, we can of course change the password in the boot_script.sh file but why not use the power of Pulumi to make the password configurable? Let’s do that in the next section.

If you face any problems, you can find the code up to now on Gitlab

Making the password configurable

Because we use Python to define our infrastructure, we can use Python to make the password configurable. We can do this by modifying the boot_script.sh file at runtime. We can use the pulumi.Config object to get the password from the Pulumi configuration and then replace the default password in the boot_script.sh file.

Let’s modify our __main__.py file to include the password configuration. We can do this by adding the following lines of code after the import statements:

config = pulumi.Config()
password = config.require_secret("password")

Now add the password as a Pulumi configuration like this:

pulumi config set password MySuperSecret

This will set the password in the Pulumi configuration. Now we need to modify our __main__.py file to replace the default password in the boot_script.sh file with the password from the Pulumi configuration. We can do this by adding the following lines of codes where the boot_script is read:

# read boot_script from file
with open("boot_script.sh", "r") as file:
    boot_script = file.read()

# Function to update the boot script with the dynamic password
def update_boot_script(boot_script, password):
    return boot_script.replace("MyNotSoSecretPassword34789#", password)

# Apply the password to the boot script
boot_script_with_password = password.apply(lambda pwd: update_boot_script(boot_script, pwd))

This maybe looks complex but let me explain here what happens, so you understand it better:

1. We define a function called update_boot_script that takes the boot_script and the password as arguments and returns the boot_script with the default password replaced by the password.
2. We use the apply method of the password object to apply the update_boot_script function to the boot_script and the password. This will return a new boot_script with the default password replaced by the password.
3. We save the new boot_script in a variable called boot_script_with_password.
4. That’s it!

We need to use this apply method because the password is a pulumi.Output object that is not available at runtime. The apply method allows us to apply a function to the password and get the result at runtime.

The last thing we now need to do is update the code that creates the EC2 instance to use the boot_script_with_password instead of the boot_script:

# Create the EC2 instance
web = aws.ec2.Instance("web",
    ami=ubuntu.id,
    instance_type=aws.ec2.InstanceType.T3_MICRO,
    user_data=boot_script_with_password,
    user_data_replace_on_change=True,
    vpc_security_group_ids=[security_group.id],
    tags={
        "Name": "HelloWorld",
    })

Now we can deploy our infrastructure again by running pulumi up again.

If you face any problems, you can find the code up to now on Gitlab

Configuring a Elastic IP

To use a static IP address we need to use the service in AWS called Elastic IP. This will provide you with a static IP address which you then can assign to your EC2 instance. Let’s modify our __main__.py file to include the Elastic IP. We can do this by adding the following lines of code after the `web` variable:

elastic_ip = aws.ec2.Eip(f"elasticIp")
aws.ec2.EipAssociation(f"eipAssoc",
        instance_id=web.id,
        allocation_id=elastic_ip.id)

The first line will initialize a new Elastic IP address and the second line will associate this IP address with the EC2 instance.

Make sure to also change the Pulumi export, from exporting the web.public_ip to elastic_ip.public_ip. This way Pulumi will print the public IP from the Elastic IP instead of the EC2 instance.
If you now run pulumi up again then it should create a Elastic IP and assign it to the EC2 instance. You can now use this Elastic IP to access the code-server in your browser.

config = pulumi.Config()
password = config.require_secret("password")
number_of_instances = config.get_int("number_of_instances") or 1

# Create a list to store the instances and the Elastic IPs
instances = []
elastic_ips = []

# Create the EC2 instances
for i in range(number_of_instances):
    instance = aws.ec2.Instance(f"web-{i}",
        ami=ubuntu.id,
        instance_type=aws.ec2.InstanceType.T3_MICRO,
        user_data=boot_script_with_password,
        user_data_replace_on_change=True,
        vpc_security_group_ids=[security_group.id],
        tags={
            "Name": f"HelloWorld-{i}",
        })
    instances.append(instance)

    # Allocate an Elastic IP for each instance
    elastic_ip = aws.ec2.Eip(f"elasticIp-{i}")
    elastic_ips.append(elastic_ip)

    # Associate the Elastic IP with the instance
    aws.ec2.EipAssociation(f"eipAssoc-{i}",
        instance_id=instance.id,
        allocation_id=elastic_ip.id)

# Export the public IP addresses and DNS names of all instances
public_ips = [elastic_ip.public_ip for elastic_ip in elastic_ips]
public_dns = [elastic_ip.public_dns for elastic_ip in elastic_ips]

# Construct the full URLs
urls = pulumi.Output.all(*public_dns).apply(lambda dns_list: [f"https://{dns}" for dns in dns_list])

pulumi.export("public_ips", pulumi.Output.all(*public_ips))
pulumi.export("public_dns", pulumi.Output.all(*public_dns))
pulumi.export("urls", urls)