HTTPS with Let's Encrypt On AWS EC2 To Serve Flask Behind Nginx

2023-07-03 1000 words 5 minutes

Contents

I originally tried doing this with Apache. Getting the certificate with Apache wasn’t too bad, but getting the Flask setup served behind Apache was rough. Turns out Nginx was a lot simpler, so I’ve rewritten the instructions below for Nginx, and preserved the raw notes for Apache further down.

Cert With Certbot

Install nginx on your ec2 instance

1

sudo amazon-linux-extras install nginx1

Just using nginx to deploy flask to production
You’ll want to use Let’s Encrypt to create a cert. This will point you to use Certbot, which wants to use Snap, but that doesn’t seem to work on this Amazon Linux distro, so you’ll want to use the pip instructions instead.
The pip instructions will have you create a venv, but there’s some OpenSSL compatibility issue. A workaround seems to be to pin the requests module at an older version, per GitHub

1
2
3
4
5


cd ~/ScrapeDebugServer
python3 -m venv certbot
source certbot/bin/activate
echo "requests==2.29.0" > certbot/requirements.txt
python -m pip install -r certbot/requirements.txt

You’ll also have to install augeas from yum instead of dnf, probably:

1
2


sudo yum update -y
sudo yum install augeas-libs

Then you should be able to install certbot, as described in the instructions linked above:

1

python -m pip install certbot certbot-nginx

Note that I’ve done it slightly differently than the instructions call for. They have you put the venv somewhere in /opt/certbot, but I’d like the venv to just stay in the ~/ScrapeDebugServer folder. This doesn’t end up causing problems, it just means you’ll need to do the crontab slightly differently. I also skip the step to symlink certbot, and just execute directly from the venv bin folder.

1
2


cd certbot/bin
sudo ./certbot --nginx

Did you know that cron has a few different files? /etc/crontab is system-wide, and takes in a user argument. sudo crontab -e will edit a different, root-user crontab file that doesn’t take a user argument. The typical crontab -e will edit a user-specific crontab file. Details here. Seems like you also don’t need to reload the cron service after editing.

Here’s what I ended up using in my /etc/crontab.

1

0 0,12 * * * root /home/ec2-user/ScrapeDebugServer/certbot/bin/python -c 'import random; import time; time.sleep(random.random() * 3600)' && sudo /home/ec2-user/ScrapeDebugServer/certbot/bin/certbot renew -q

Step 3 here goes into detail on testing the SSL quality.
You’re supposed to tell flask it’s behind a proxy, but it doesn’t seem to be necessary.
Make sure to enable the nginx service:

1
2


sudo systemctl enable nginx
sudo systemctl start nginx

Adjusting final config:

I might need to update some of these proxy_set_header commands in the configuration later, but I’ve just left it as recommended for now.

Here are the final server blocks I end up with. Note that I specifically turn off the automatic HTTPS redirection in this case, because I want to be able to test both HTTP and HTTPS traffic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


    server {
        server_name scrapedebug.vasuagrawal.com scrapedebugproxy.vasuagrawal.com;

        location / {
            proxy_pass http://127.0.0.1:5000/;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Prefix /;
        }

        listen 443 ssl; # managed by Certbot
        ssl_certificate /etc/letsencrypt/live/scrapedebug.vasuagrawal.com/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/scrapedebug.vasuagrawal.com/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    }   
    
    server {
        listen 80;
        server_name scrapedebug.vasuagrawal.com scrapedebugproxy.vasuagrawal.com;

        location / {
            proxy_pass http://127.0.0.1:5000/;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Prefix /;
        }

# Don't need redirection in this case
#        if ($host = scrapedebugproxy.vasuagrawal.com) {
#            return 301 https://$host$request_uri;
#        } # managed by Certbot
#
#
#        if ($host = scrapedebug.vasuagrawal.com) {
#            return 301 https://$host$request_uri;
#        } # managed by Certbot
#
#
#        return 404; # managed by Certbot
    }  

Then, you can start a gunicorn instance to serve the Flask app. I’ve bound it to all interfaces so that I can also access it via port 5000 directly (bypassing Nginx) to see if there’s any differences in behavior.

1

gunicorn -w 1 -b 0.0.0.0:5000 scrape_debug_server:app

Leftover Raw Apache Notes

NOTE: Numbering is off and some of the steps have been cannibalized for the above version. This all mostly worked - the thing I ran into trouble with was the mod_wsgi package seemingly only being available for Python 2, and installing one yourself didn’t seem to install the right Apache commands for WSGI. Some of these problems might be related to EC2 specifically.

Follow the instructions here to install apache, which boil down to:

1

sudo yum install -y httpd

Follow step 1 here, to enable TLS on the server
You’ll also have to configure Apache to provide a Virtual Host on port 80, as described here. You can put multiple lines for ServerAlias, mine (initially) looks like this:

1
2
3
4
5
6
7
8


$ cat /etc/httpd/conf.d/scrapedebug.conf 
<VirtualHost *:80>
    ServerName scrapedebug.vasuagrawal.com
    DocumentRoot /var/www/html
    ServerAlias www.scrapedebug.vasuagrawal.com scrapedebugproxy.vasuagrawal.com www.scrapedebugproxy.vasuagrawal.com
    ErrorLog /var/www/error.log
    CustomLog /var/www/requests.log combined
</VirtualHost>

It seems certbot adds some extra lines, so after configuration mine actually looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


$ cat /etc/httpd/conf.d/scrapedebug.conf 
<VirtualHost *:80>
    ServerName scrapedebug.vasuagrawal.com
    DocumentRoot /var/www/html
    ServerAlias www.scrapedebug.vasuagrawal.com scrapedebugproxy.vasuagrawal.com www.scrapedebugproxy.vasuagrawal.com
    ErrorLog /var/www/error.log
    CustomLog /var/www/requests.log combined
RewriteEngine on
RewriteCond %{SERVER_NAME} =scrapedebugproxy.vasuagrawal.com [OR]
RewriteCond %{SERVER_NAME} =scrapedebug.vasuagrawal.com [OR]
RewriteCond %{SERVER_NAME} =www.scrapedebug.vasuagrawal.com [OR]
RewriteCond %{SERVER_NAME} =www.scrapedebugproxy.vasuagrawal.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

And then there’s also a new file that got created:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


cat scrapedebug-le-ssl.conf 
<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName scrapedebug.vasuagrawal.com
    DocumentRoot /var/www/html
    ServerAlias www.scrapedebug.vasuagrawal.com scrapedebugproxy.vasuagrawal.com www.scrapedebugproxy.vasuagrawal.com
    ErrorLog /var/www/error.log
    CustomLog /var/www/requests.log combined
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/scrapedebug.vasuagrawal.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/scrapedebug.vasuagrawal.com/privkey.pem
</VirtualHost>
</IfModule>

Using Apache for WSGI for Flask following these instructions

1
2
3


sudo yum install httpd-devel
sudo yum groupinstall "Development Tools"
sudo yum install python3-devel.x86_64

Amazon Linux might be deprecated?