# Managing Configurations in Python Projects - **URL:** https://isaacfei.com/posts/py-config - **Date:** 2025-01-22 - **Tags:** Python - **Description:** Learn to manage Python configurations with dotenv and pydantic. Load environment-specific settings (dev, test, prod) using the ENV variable, keeping your app secure and scalable. --- When building a Python project, managing configurations can quickly become a headache. You have different environments like development, testing, and production, each with its own set of configurations. You want to keep things organized, avoid hardcoding sensitive information, and make it easy to switch between environments. In this post, I'll walk you through how I tackled this problem using Pydantic and dotenv files. Let's dive in! ## The Problem: Managing Multiple Environments In any non-trivial project, you'll likely have at least three environments: - Development (dev): Where you write and test your code locally - Testing (test): Where automated tests run - Production (prod): Where your application runs in the real world Each environment has its own configuration. For example, database credentials, API keys, and file paths might differ between development and production. Hardcoding these values is a no-go—it's error-prone and insecure. Instead, we want to manage these configurations externally, typically using environment variables. ```dotenv ENV=dev DEEPSEEK_API_KEY=your_api_key_here FILE_UPLOAD_DIR=~/uploads POSTGRES_USER=dev_user POSTGRES_PASSWORD=dev_password QDRANT_HTTP_PORT=6333 ``` You should NEVER commit these dotenv files to Git. They often contain sensitive information like API keys and passwords. However, you should include a `.env.example` file with placeholder values. This way, other developers know what environment variables they need to set up. ## Determining the Current Environment To decide which environment to use, I introduced a special environment variable called `ENV`. This variable can be set to `dev`, `test`, or `prod`. Based on its value, the application will load the corresponding dotenv file (e.g., `.env.dev`, `.env.test`, or `.env.prod`). Here's how I implemented this in the `find_dotenv.py` file: ```py # project/config/find_dotenv.py import os import dotenv def find_dotenv() -> str: # Get the value of ENV from the environment variables env = os.getenv("ENV") # Default to dev if env is None: env = "dev" # Select different env files based on ENV match env: case "dev": env_filename = ".env.dev" case "test": env_filename = ".env.test" case "prod": env_filename = ".env.prod" case _: raise ValueError(f"unknown env: {env}") # Find the dotenv file env_filepath = dotenv.find_dotenv(env_filename) return env_filepath ``` This function checks the ENV variable and selects the appropriate dotenv file. If ENV is not set, it defaults to the development environment. ## Grouping Configurations with Pydantic Dotenv files are great, but they don't support nested fields. For example, you can't directly group all PostgreSQL-related configurations under a `POSTGRES` key. To solve this, I used Pydantic, a powerful library for data validation and settings management. Pydantic allows you to define configuration classes that can load values from environment variables. Here's how I grouped PostgreSQL configurations: ```py # project/config/postgres.py from pydantic_settings import BaseSettings, SettingsConfigDict class PostgresConfig(BaseSettings): user: str password: str db: str port: int data_dir: str uri: str model_config = SettingsConfigDict( env_file_encoding="utf-8", env_prefix="POSTGRES_", extra="ignore", ) ``` Notice the `env_prefix="POSTGRES_"` line. This tells Pydantic to look for environment variables that start with `POSTGRES_`. For example, `POSTGRES_USER`, `POSTGRES_PASSWORD`, etc. This way, you can keep all PostgreSQL-related configurations neatly grouped together. Similarly, I created a QdrantConfig class for Qdrant-related settings: ```py # project/config/qdrant.py from pydantic_settings import BaseSettings, SettingsConfigDict class QdrantConfig(BaseSettings): http_port: int grpc_port: int uri: str model_config = SettingsConfigDict( env_file_encoding="utf-8", env_prefix="QDRANT_", extra="ignore", ) ``` ## Loading the Configuration Now that we have our configuration classes, we need a way to load them. This is where the `load_config` function comes in. It loads the appropriate dotenv file based on the `ENV` variable and initializes the configuration classes. Here's the `load_config` function from the `config.py` file: ```py # project/config/config.py def load_config() -> Config: # Find the dotenv file based on the ENV env_filepath = find_dotenv() # Load the qdrant configuration qdrant_config = load_qdrant_config() # Load the postgres configuration postgres_config = load_postgres_config() # Load the configuration config = Config( qdrant=qdrant_config, postgres=postgres_config, _env_file=env_filepath, ) return config ``` This function does the following: - Finds the correct dotenv file using the `find_dotenv` function from `python-dotenv` package - Loads the Qdrant and PostgreSQL configurations using their respective `load_*_config` functions - Initializes the main `Config` class with these configurations ## Putting It All Together Finally, here's the main Config class that ties everything together: ```py # project/config/config.py from pydantic_settings import BaseSettings, SettingsConfigDict from pydantic import Field, field_validator from pathlib import Path class Config(BaseSettings): env: Env = Field(default=Env.DEV) file_upload_dir: Path deepseek_api_key: str qdrant: QdrantConfig = Field(default_factory=QdrantConfig) postgres: PostgresConfig = Field(default_factory=PostgresConfig) model_config = SettingsConfigDict( env_file_encoding="utf-8", extra="ignore", ) @field_validator("file_upload_dir", mode="after") @classmethod def resolve_path(cls, path: Path) -> Path: return path.expanduser().resolve() ``` This class includes: - A default environment (`Env.DEV`) - A `file_upload_dir` field that resolves to an absolute path - Nested configurations for Qdrant and PostgreSQL ## Why This Design? This design has several advantages: - Environment-Specific Configurations: By using the `ENV` variable, you can easily switch between different environments without changing any code. - Security: Sensitive information is stored in dotenv files, which are not committed to Git. - Organization: Configurations are grouped logically using Pydantic classes, making the code easier to maintain. - Flexibility: You can add more configurations by simply creating new Pydantic classes and adding them to the `Config` class. ## Final Thoughts By using dotenv files and Pydantic, you can keep your configurations organized, secure, and environment-specific. This approach has worked well for me, and I hope it helps you too!