Storing Authentication Information Outside of Source Code

“An environment variable is a variable whose value is set outside the program, typically through functionality built into the operating system or microservice. An environment variable is made up of a name/value pair, and any number may be created and available for reference at a point in time.” - An Introduction to Environment Variables and How to Use Them by Jim Medlock on Medium

Summary

Working with credentials when programming, wether username & password in BASE64, an API key or one of the many, many other forms, causes us to face a problem - where to store them for my code to access? Writing them down inside your code, which may be copied/cloned/shared, is no way acceptable due to obvious risks. Of course, you could choose to enter them every time your code runs, which is secure to an extent but time consuming. What’s the middle ground? In production environments, it’s common to have a back end server that provides information to a front end server without exposing the API key to any third party. What about test/dev environments? In this post I’m going to detail a few options.

My Environment

Coffee: House Roast, Original Blend from Union Hand-Roasted Coffee
Music: Funk & Soul Classics Playlist by Various
OS: Windows 10 Pro v20H2 x64.
Python: v3.9.2
IDE: Visual Studio Code v1.56.2

Tip o’ the Hat

Rod’s answer on Stack Overflow
Python OS documentation on python.org
Malwarebytes blog post on Windows environmental variables
Raivat Shah’s blog post on Medium
Steve Griffith’s video on YouTube.

Let’s Begin

<NOTE>: Instead of re-inventing the wheel and explaining things that have been well defined by someone else, I have included links next to some words/technologies/acronyms/protocols that I feel could proove useful to those not yet ‘in the know’. </NOTE>

For production, processes differ than in development - simply put, a front end server sends a request to a backend server, which in turn sends a request to a final destination server using an authentication medium that it has access to. Once the back end server receives data from the final destination server, it sends to the front end server. Essentially the front end server never has access to that authentication information and uses the back end server as a proxy. The back end server may store the authentication information in environmental variables, in a local config file or in some sort of secure repository/server that it has access to. This YouTube video by PortEXE explains this process with React. Backend servers can be behind strict firewall rules to permit/deny all but relevant & secure traffic, usually from only the front end server and management infrastructure.

In local development, it really depends on your environment, if you’re using shared source code, etc. If the aim of the game is to simply want to store credentials outside of the source code, we can use something like an environmental variable. An environmental variable is a variable whose value is set outside of the program (like Python), typically through functionality built into the OS (like Windows). In this instance, I have a single Python file - app.py - which needs access to an API key. The following process explains how to store both temporary (cleared after the session/application is terminated) and persistent (remains after the session/application is terminated) on both Windows and macOS and access them in Python.

Temporary Environmental Variables in Windows, macOS & Python

As mentioned, temporary environmental variables are only stored for the remainder of that session - looking at the following GIF of me displaying this in Command Prompt, if I were to close and re-open the application, I would not be able to re-call the variable. For me, in most applications these are next to useless - I’ve included them in this post to serve as contrast to persistent ones.

You can set and show envrionmental variables with the following syntax -

Command Prompt

::Set the environmental variables - 
set user=jonny
set pass=123456789

::Display the environmental variables - 
echo %user%'s password is %pass%
::Output = jonny's password is 123456789

PowerShell

#Set the environmental variables - 
$env:user = "jonny"
$env:pass = 123456789

#Display the environmental variables - 
Write-Host $env:user"'"s password is $env:pass
#Output = jonny's password is 123456789

Terminal/Bash

#Set the environmental variables - 
export user="jonny"
export pass="123456789"

#Display the environmental variables - 
echo $user"'"s password is $pass
#Output = jonny's password is 123456789

Python (for completeness, but missing the point)

import os

#Set the environmental variables - 
os.environ['user'] = "jonny"
os.environ['pass'] = "123456789"

#Display the environmental variables -
print(os.environ['user'] + "'s password is " + os.environ['pass'])
#Output = jonny's password is 123456789

Persistent Environmental Variables in Windows & macOS and Retrieving Them in Python

This Help Desk Geek blog post written by Aseem Kishore goes into creating this in depth, but from a high level -

1. Open System Properties by going to Settings > System > About and clicking Advanced system settings.
2. On the Advanced tab, click Environmental Variables….
3. Here, click New under User variables for %username%.
4. Give the variable a name, for example user followed by a value, like jonnywinter
5. Click OK to save.

That’s really it for Windows. You can now open up either Command Prompt or PowerShell and display the list of variables or that specific variable by -

Command Prompt

::Display the list of environmental variables
set
::Display a specific environmental variable
echo %user%

PowerShell

#Display the list of environmental variables
Get-ChildItem -path env:
#Display a specific environmental variable
$env:user

For macOS, the process is a little different; the environmental variables are stored in a file that we must edit in a text editor (and if using nano press CTRL+W to quit, selecting Y to save). The easiset way to do this is to open up Terminal and -

Terminal/Bash

#Navigate to your home directory
cd ~
#Open up the hidden file, .bash_profile (or .zprofile if your using ZSH and not Bash)
nano .bash_profile

Nano

export user="jonny"
export pass="123456789"

The following image, taken from Alexander Fox’s post on AppleGazette shows you what this would look like in nano -

Once saved, the variables will be persistent if you close & re-open Terminal. Like with the above, you can call the variables in Terminal by using echo. With the above steps complete, we can now go about calling those variables from within Python. It’s the same process for macOS or Windows.

Python

import os

user = os.environ.get("user")
pass = os.environ.get("pass")

print(user + "'s password is " + pass)
#Output = jonny's password is 123456789

In the above code, you are importing the os module and using the environ.get method to retrieve the environmental variables, setting them as python variables user and pass respectively.

I’ve detailed another option for storing authentication information outside of source code below. In the following examople I’m not using environmental variables, but instead creating a JSON file to store and retrieve that data. Immediately following that section I’ve included another section on how to exclude that file from commits in the Git VCS system.

Storing Authentication Information Outside of Source Code in JSON

As an alternative to environmental variables, storing something like an API key in a file in a local directory or secure server can be a great option. This could easily be YAML or XML, but I like working with JSON so I’m using that in this example. The directory could be an absolute path (i.e. c:\foo\bar.json) or simply relying on the working directory (i.e. bar.json). The great thing here is that your source code, once again, doesn’t have your authentication information within it but instead references another place that does -

Python

import os
import json

config = json.load(open('C:\\app\\config.json', 'r'))

print(config['apikey'])
#Output = 123-456-789

JSON

{
  "apikey":"123-456-789"
}

Using GitHub & .gitignore

GitHub is fantastic, and using it is great. However, if you start to commit directories & files to the cloud and share them with others then you’re going to need to exclude files & folders from being synced. Luckily the creators of Git thought of this and came up with the .gitignore file. Although the .gitignore file is synced to the cloud, the files and folders that the file references will not be. This YouTube video by The Coding Train (~06:10 mins in) goes into creating this file and specifically not syncing a .env file which contains an API key. To do this -

1. The .gitignore file must exist in the root directory and will be created manually.
2. You can specify files (i.e. config.json) & folders (i.e. project_notes/).
3. You can use the * wildcard to specifiy files or folders that start/end with any string (i.e. *.json will ignore any .json files)

The below picture taken from Raul Guarini’s answer on Stack Overflow shows what this should/could look like in practice -

Cloud Key Management & Finishing Up

As a final mention, there are various cloud systems, including GitHub, Azure, AWS, etc. that will allow you to safely store and retrieve keys, connection strings and authentication information which can be both used in production & dev/testing. Although this post doesn’t delve into them, they are very valuable and widely used tools. Some paid for, some not - often they really rely on where you are storing your code.

So, there we have it - a few options for hiding/obscuring secrets in both development/test environments as well as in production.

Happy scripting!