Create your own developer environment in AWS with the help of Pulumi

Introduction

Let’s say you want to create a developer environment in AWS for your team or for a workshop. You want to have a fully functional IDE in the cloud that you can access from anywhere. You want to be able to easily create multiple developer environments for different projects or teams. How would you do that? You can do that by just using the AWS console and create there different develop environment. A better solution would probably be to use a tool that allows you to define your infrastructure as code and then deploy it to your cloud provider. A well known tool for that is Terraform, with Terraform you can define your infrastructure using a declarative language and then deploy it to your cloud provider. But there is also another tool that allows you to do that, Pulumi. Pulumi is an open-source infrastructure as code tool that allows you to define and manage your cloud infrastructure using familiar programming languages like Python, JavaScript, and Go.

I think this is a big advantage because you can use the same programming language that you use for your application code to define your infrastructure. This makes it easier to understand and maintain your infrastructure code, especially for developers who are already familiar with the programming language.

In this blog post, we will create a developer environment in AWS using Pulumi. The idea is that in the end you will have a fully functional developer environment in AWS based on the open source project code-server. This project allows you to run Visual Studio Code in the browser and is perfect for remote development.

As stated before, we will use Pulumi to create the necessary infrastructure in AWS. Pulumi allows you to define your infrastructure using code and then deploy it to your cloud provider. This way you can easily create, update, and delete your infrastructure in a reproducible way. In this blog post, we will use Python to define our infrastructure, but you can also use other programming languages like JavaScript or Go.

To create our developer environment in AWS, we basically need to do the following steps:

  1. Create a new EC2 instance
  2. Install the required tools
  3. Configure the instance

Prerequisites

Before we start, you need to have the following prerequisites:

If you don’t have these prerequisites yet, please install them before continuing.
When you have all the prerequisites, you can continue with the next section.

Creating the developer environment

Creating the Pulumi project

First, we need to create a new Pulumi project. You can do this by running the following command in your terminal:

pulumi new aws-python

If you want to verify that the project was created successfully, you can run the following command:

pulumi up

This will show you the resources that will be created in your AWS account. If everything looks good, you can confirm the creation by typing yes and hitting enter.
If you encounter any issues, please refer to the Pulumi documentation for help: https://www.pulumi.com/docs/. Also make sure your AWS credentials are set up correctly.

Creating the AWS EC2 instance

Now that we have our Pulumi project set up, we can start creating the infrastructure. The first thing we need to do is create a new EC2 instance in AWS. To do that we need to modify the file __main__.py. This file contains the Python code that defines our infrastructure.

If you open the file __main__.py in your favorite code editor, you will see that it contains some boilerplate code that creates a new s3 bucket. We can remove all those lines because we don’t need the S3 bucket anymore. We need to create a EC2 instance, but how do we do that?

Luckily, Pulumi has excellent documentation that explains how to create an EC2 instance in AWS using Python. You can find the documentation here: https://www.pulumi.com/docs/reference/pkg/aws/ec2/instance/. On this page, you can see all the available options that you can use to create an EC2 instance and also some examples. Let’s use the basic example as a starting point. We can copy the example code and paste it into our __main__.py file. It will look like this:

import pulumi
import pulumi_aws as aws

ubuntu = aws.ec2.get_ami(most_recent=True,
    filters=[
        {
            "name": "name",
            "values": ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"],
        },
        {
            "name": "virtualization-type",
            "values": ["hvm"],
        },
    ],
    owners=["099720109477"])
web = aws.ec2.Instance("web",
    ami=ubuntu.id,
    instance_type=aws.ec2.InstanceType.T3_MICRO,
    tags={
        "Name": "HelloWorld",
    })

Let me explain this code quickly, so you understand what it does:

  1. We import the necessary modules from Pulumi and the AWS provider.
  2. We get the latest Ubuntu AMI using the get_ami function.
  3. We create a new EC2 instance using the Instance function.
  4. We specify the AMI, instance type, and tags for the EC2 instance.
  5. We save the EC2 instance in a variable called web.
  6. That’s it!

Now we can deploy our infrastructure by running the following command:

pulumi up

Let’s do that now and see what happens.

If you now go to the AWS console and navigate to the EC2 dashboard, you should see a new EC2 instance running. Congratulations, you have created your first EC2 instance using Pulumi!
But we don’t want to create a simple EC2 instance, we want to create a developer environment. So let’s continue with the next steps.

Running the install script

The next step is to run the install script that will install the necessary software on the EC2 instance. This script will install Docker, Docker Compose, and the code-server. We can do this by using the UserData property of the EC2 instance. The UserData property allows you to specify a script that will be executed when the instance is launched. We can use this to run our install script.

Let’s modify our __main__.py file to include the install script. We can do this by adding the following lines of code after the import statements:

with open("boot_script.sh", "r") as file:
    boot_script = file.read()

This will read the contents of the boot_script.sh file into a variable called boot_script. Now we can add the UserData property to our EC2 instance like this:

web = aws.ec2.Instance("web",
    ami=ubuntu.id,
    instance_type=aws.ec2.InstanceType.T3_MICRO,
    user_data=boot_script,
    user_data_replace_on_change=True,
    tags={
        "Name": "HelloWorld",
    })
This will run the contents of the boot_script.sh file when the EC2 instance is launched. Now we need to create the boot_script.sh file and add the install script to it. You can download the content of the file from Gitlab, make sure you save this file or the content to a file called boot_script.sh in the same directory as the __main__.py file.

This script will update the package list, install Docker, Docker Compose, and curl, start the Docker service, and then install the code-server using the install script from the code-server website.

Now we can deploy our infrastructure again by running pulumi up again. Let’s do that now and see what happens.

It should stop the current instance and restart it with the new UserData script. If you now go to the AWS console and navigate to the EC2 dashboard, you should see that the instance is stopped and then started again.

If you face any problems, you can find the code up to now on Gitlab

Opening the firewall

The next step is to open the firewall so that we can access the code-server from the browser. We can do this by adding a security group to the EC2 instance. A security group acts as a virtual firewall that controls the traffic to and from the EC2 instance. We can specify the inbound and outbound rules of the security group to allow or deny traffic.

Let’s modify our __main__.py file to include the security group. We can do this by adding the following lines of code before the web variable:

# Create a security group allowing HTTP access
security_group = aws.ec2.SecurityGroup("web-sg",
    description="Allow HTTP access",
    ingress=[
        {
            "protocol": "tcp",
            "from_port": 443,
            "to_port": 443,
            "cidr_blocks": ["0.0.0.0/0"],
        },
    ],
    egress=[
        {
            "protocol": "-1",
            "from_port": 0,
            "to_port": 0,
            "cidr_blocks": ["0.0.0.0/0"],
        },
    ]
)

This will create a new security group called web-sg that allows inbound traffic on port 443 from any IP address. Now we need to associate this security group with our EC2 instance. We can do this by adding the vpc_security_group_ids property to the EC2 instance like this:

# Create the EC2 instance
web = aws.ec2.Instance("web",
    ami=ubuntu.id,
    instance_type=aws.ec2.InstanceType.T3_MICRO,
    user_data=boot_script,
    vpc_security_group_ids=[security_group.id],
    tags={
        "Name": "HelloWorld",
    })

Now let’s also add a pulumi export to the end of our code so we can print the public IP address of the EC2 instance:

pulumi.export("public_ip", web.public_ip)

Now we can deploy our infrastructure again by running pulumi up again. After Pulumi is done applying your changes you can go to https://<<instance-ip>>:443 in your browser, you should see the code-server login page. It will ask for a password, which is MyNotSoSecretPassword34789# by default. This is not really secure, we can of course change the password in the boot_script.sh file but why not use the power of Pulumi to make the password configurable? Let’s do that in the next section.

If you face any problems, you can find the code up to now on Gitlab

Making the password configurable

Because we use Python to define our infrastructure, we can use Python to make the password configurable. We can do this by modifying the boot_script.sh file at runtime. We can use the pulumi.Config object to get the password from the Pulumi configuration and then replace the default password in the boot_script.sh file.

Let’s modify our __main__.py file to include the password configuration. We can do this by adding the following lines of code after the import statements:

config = pulumi.Config()
password = config.require_secret("password")

Now add the password as a Pulumi configuration like this:

pulumi config set password MySuperSecret

This will set the password in the Pulumi configuration. Now we need to modify our __main__.py file to replace the default password in the boot_script.sh file with the password from the Pulumi configuration. We can do this by adding the following lines of codes where the boot_script is read:

# read boot_script from file
with open("boot_script.sh", "r") as file:
    boot_script = file.read()

# Function to update the boot script with the dynamic password
def update_boot_script(boot_script, password):
    return boot_script.replace("MyNotSoSecretPassword34789#", password)

# Apply the password to the boot script
boot_script_with_password = password.apply(lambda pwd: update_boot_script(boot_script, pwd))

This maybe looks complex but let me explain here what happens, so you understand it better:

  1. We define a function called update_boot_script that takes the boot_script and the password as arguments and returns the boot_script with the default password replaced by the password.
  2. We use the apply method of the password object to apply the update_boot_script function to the boot_script and the password. This will return a new boot_script with the default password replaced by the password.
  3. We save the new boot_script in a variable called boot_script_with_password.
  4. That’s it!

We need to use this apply method because the password is a pulumi.Output object that is not available at runtime. The apply method allows us to apply a function to the password and get the result at runtime.

The last thing we now need to do is update the code that creates the EC2 instance to use the boot_script_with_password instead of the boot_script:

# Create the EC2 instance
web = aws.ec2.Instance("web",
    ami=ubuntu.id,
    instance_type=aws.ec2.InstanceType.T3_MICRO,
    user_data=boot_script_with_password,
    user_data_replace_on_change=True,
    vpc_security_group_ids=[security_group.id],
    tags={
        "Name": "HelloWorld",
    })

Now we can deploy our infrastructure again by running pulumi up again.

If you now go to https://<public_ip>:443 in your browser, you should see the code-server login page. It will ask for a password, which is now the password you set in the Pulumi configuration. Congratulations, you have created a developer environment in AWS using Pulumi! But accessing the developer environment directly via the public IP address of the EC2 instance is not very convenient. Because if the instance for some reason is restarted you will get a different IP address, to prevent that we can use an static IP address using Amazon Elestic IP. Let’s see how we can do that in the next section.

If you face any problems, you can find the code up to now on Gitlab

Configuring a Elastic IP

To use a static IP address we need to use the service in AWS called Elastic IP. This will provide you with a static IP address which you then can assign to your EC2 instance. Let’s modify our __main__.py file to include the Elastic IP. We can do this by adding the following lines of code after the `web` variable:

elastic_ip = aws.ec2.Eip(f"elasticIp")
aws.ec2.EipAssociation(f"eipAssoc",
        instance_id=web.id,
        allocation_id=elastic_ip.id)

The first line will initialize a new Elastic IP address and the second line will associate this IP address with the EC2 instance.

Make sure to also change the Pulumi export, from exporting the web.public_ip to elastic_ip.public_ip. This way Pulumi will print the public IP from the Elastic IP instead of the EC2 instance.
If you now run pulumi up again then it should create a Elastic IP and assign it to the EC2 instance. You can now use this Elastic IP to access the code-server in your browser.

If you want to connect to the developer environment using a DNS name then you can use the automatically genereated DNS name of the Elastic IP. You can create a new export for that to display the DNS name in the Pulumi output:

pulumi.export("public_dns", elastic_ip.public_dns)If you run pulumi up again then you should see the DNS name in the Pulumi output. You can now use this DNS name to access the code-server in your browser. Congratulations, you have created a developer environment in AWS.

But what if you need more than one developer environment? For example if you are giving a workshop and you need to have an environment available for all the student or you want to provide this developer environment to all of your team members. Let’s see how we can do this in the next section.

If you face any problems, you can find the code up to now on Gitlab

Creating multiple developer environments

Because we use Python to write our Pulumi code, we can leverage all the functionality Python has to offer to make our Pulumi code more flexible. Normally, we would just duplicate the existing code to create multiple EC2 instances. But that is not very efficient and not very Pythonic. Instead we can just use a simple loop to create multiple EC2 instances. Let’s see how we can do that.

First let’s add a new configuration to our __main__.py file to specify the number of developer environments we want to create:

config = pulumi.Config()
password = config.require_secret("password")
number_of_instances = config.get_int("number_of_instances") or 1

Now we can create a loop that creates the specified number of developer environments:

# Create a list to store the instances and the Elastic IPs
instances = []
elastic_ips = []

# Create the EC2 instances
for i in range(number_of_instances):
    instance = aws.ec2.Instance(f"web-{i}",
        ami=ubuntu.id,
        instance_type=aws.ec2.InstanceType.T3_MICRO,
        user_data=boot_script_with_password,
        user_data_replace_on_change=True,
        vpc_security_group_ids=[security_group.id],
        tags={
            "Name": f"HelloWorld-{i}",
        })
    instances.append(instance)

    # Allocate an Elastic IP for each instance
    elastic_ip = aws.ec2.Eip(f"elasticIp-{i}")
    elastic_ips.append(elastic_ip)

    # Associate the Elastic IP with the instance
    aws.ec2.EipAssociation(f"eipAssoc-{i}",
        instance_id=instance.id,
        allocation_id=elastic_ip.id)

# Export the public IP addresses and DNS names of all instances
public_ips = [elastic_ip.public_ip for elastic_ip in elastic_ips]
public_dns = [elastic_ip.public_dns for elastic_ip in elastic_ips]

# Construct the full URLs
urls = pulumi.Output.all(*public_dns).apply(lambda dns_list: [f"https://{dns}" for dns in dns_list])

pulumi.export("public_ips", pulumi.Output.all(*public_ips))
pulumi.export("public_dns", pulumi.Output.all(*public_dns))
pulumi.export("urls", urls)

What we basically do here is that we put the code, where we create the instance, in a for Python for loop. This way we can create multiple instances instead of one. To make it possible to export the IP’s and DNS names of the created instances we need to save the created instances to a list (instances = []) so we can iterate over that list later when we do the exports.

If you now run pulumi up it should create the specified number of EC2 instances with the specified password and export the public IP addresses and the public DNS of all instances.

After you run pulumi up it will probably will recreate one instance, this is because the name is changed from web to web-0. If you now set the Pulumi configuration number_of_instances to 2 (pulumi config set number_of_instances 2) and run pulumi up again, you should see two EC2 instances running in the AWS console. As you can see it is very easy to use simple Python code to extend the functionality of Pulumi.

Conclusion

In this blog post, we have created a developer environment in AWS using Pulumi. We have used Python to define our infrastructure and create multiple developer environments. But this is just the beginning. There are many more things you can do with Pulumi, like creating more complex infrastructure, using different cloud providers, or integrating with other tools. I encourage you to explore the Pulumi documentation and experiment with different features. You can find the complete code for this blog post on GitHub at https://gitlab.com/techforce1/public/create-a-developer-environment-blog

Comparison with Terraform

As mentioned in the introduction, Pulumi is an alternative to Terraform. Both tools allow you to define your infrastructure as code and deploy it to your cloud provider. After using Pulumi for a while now I really like that you write your infrastructure code in a programming language like Python, JavaScript, or Go. The big advantage of this is that you can use all kind of function from the programming language to make your infrastructure code more flexible and reusable. As we did with the for loop to create multiple instances. With Terraform this would have been more difficult to do.

What can be a disadvantage of Pulumi is that different teams can use different programming languages to define their infrastructure. This can make it harder to maintain the infrastructure code because you need to know multiple programming languages. With Terraform you only need to know HashiCorp Configuration Language (HCL) which is a declarative language that is specifically designed for defining infrastructure. So I would recommend to always use the same programming language for defining your infrastructure to make it easier to maintain and understand the infrastructure code between different teams.

Next steps

If you want to go further with this example, then probably the next step would be to create a domain name for your developer environments and use a SSL certificate to secure the connection. Also it is a good idea to generate the password dynamically instead of using the same password for all developer environments. And also from a security perspective it is a good idea to run the code-server instance not as a root user.