atmos.GomplateDatasource

Stop hardcoding values in your infrastructure configurations. The atmos.GomplateDatasource function lets you pull live data from anywhere - APIs, cloud services, databases, files - directly into your Atmos stacks. Instead of manually updating IP addresses, account IDs, or configuration values, your infrastructure can fetch them automatically at runtime. Plus, with built-in caching, it's lightning fast even when you reference the same data hundreds of times.

What are Datasources?

Think of datasources as live connections to your data, wherever it lives. Instead of copying and pasting values into your configurations (and keeping them updated), datasources let Atmos fetch the latest data automatically when it runs.

The Problem: Traditional infrastructure configurations require you to:

• Hardcode values that change (IP addresses, account IDs, API keys)
• Manually update configurations when external systems change
• Duplicate data across multiple files
• Risk using outdated or incorrect values

The Solution: Datasources let you:

• Pull live data from any API, database, or cloud service
• Stay synchronized with external systems automatically
• Centralize configuration in its proper home (not in your IaC)
• Access secrets securely from vaults without exposing them
• Query cloud metadata like account IDs, regions, or resource tags dynamically

Real-World Example

Without Datasources (Manual, Error-Prone):

components:
  terraform:
    vpc:
      vars:
        account_id: "123456789012"  # Hardcoded - wrong in other environments!
        allowed_ip: "203.0.113.45"  # Bob's IP - outdated when he moves
        db_password: "prod-pass-123" # SECURITY RISK! Password in git

With Datasources (Dynamic, Secure):

components:
  terraform:
    vpc:
      vars:
        account_id: '{{ (atmos.GomplateDatasource "aws").account_id }}'
        allowed_ip: '{{ (atmos.GomplateDatasource "myip").ip }}'
        db_password: '{{ (atmos.GomplateDatasource "vault").database.password }}'

Common Use Cases

• Service Discovery: Query current IP addresses, endpoints, or service locations
• Dynamic Configuration: Fetch configuration values from external systems
• Secret Management: Retrieve secrets from HashiCorp Vault, AWS Secrets Manager, etc.
• Metadata Queries: Get AWS account info, region data, or resource tags
• External Data Import: Load data from CSV files, JSON APIs, or databases

Usage

  {{ (atmos.GomplateDatasource "<alias>").<attribute> }}

Arguments

alias

The datasource alias defined in your Atmos configuration

attribute

Attribute name (field) to extract from the datasource response

Supported Datasource Types

Gomplate supports numerous datasource types:

• HTTP/HTTPS: REST APIs and web services
• File: Local files (JSON, YAML, TOML, CSV, etc.)
• AWS: Systems Manager Parameter Store, Secrets Manager, S3
• Cloud: Azure Key Vault, Google Cloud Storage
• Vault: HashiCorp Vault secrets
• Environment: Environment variables
• Git: Git repository data
• Consul: HashiCorp Consul key-value store

How it Works

Automatic Caching

The atmos.GomplateDatasource function automatically caches results in memory during stack processing:

1. First call: Fetches data from the external source and stores in cache
2. Subsequent calls: Returns cached data without additional external calls
3. Cache scope: Per Atmos execution (cache is cleared between runs)

This caching mechanism provides several benefits:

• Performance: Dramatically faster stack processing with multiple datasource references
• Reliability: Reduces failures from rate limiting or network issues
• Cost savings: Fewer API calls to metered services
• Consistency: All references get the same data within a single execution

Comparison with Direct Gomplate Datasources

Feature	datasource (Gomplate)	atmos.GomplateDatasource
External calls	Every invocation	Once per alias
Performance	Slower with multiple uses	Fast (cached)
Rate limiting risk	High	Low
Network failures	Each call can fail	Single point of failure
Use case	Single reference	Multiple references

Examples

Basic API Call

settings:
  templates:
    settings:
      gomplate:
        timeout: 5
        datasources:
          # Define a datasource to get current IP
          ip:
            url: "https://api.ipify.org?format=json"
            headers:
              accept:
                - "application/json"

components:
  terraform:
    vpc:
      vars:
        # This will be cached after first call
        public_ip: '{{ (atmos.GomplateDatasource "ip").ip }}'

    firewall:
      vars:
        # Reuses cached result - no additional API call
        allowed_ip: '{{ (atmos.GomplateDatasource "ip").ip }}'

AWS Systems Manager Parameter

settings:
  templates:
    settings:
      gomplate:
        datasources:
          # Fetch from AWS SSM Parameter Store
          database:
            url: "aws+smp:///myapp/database/config?region=us-east-1"

components:
  terraform:
    app:
      vars:
        # All these use cached result after first fetch
        db_host: '{{ (atmos.GomplateDatasource "database").host }}'
        db_port: '{{ (atmos.GomplateDatasource "database").port }}'
        db_name: '{{ (atmos.GomplateDatasource "database").name }}'

Loading External Configuration

settings:
  templates:
    settings:
      gomplate:
        datasources:
          # Load configuration from external YAML file
          config:
            url: "file:///configs/app-config.yaml"

components:
  terraform:
    application:
      vars:
        # Access nested configuration values
        feature_flags: '{{ (atmos.GomplateDatasource "config").features }}'
        api_limits: '{{ (atmos.GomplateDatasource "config").limits.api }}'

Performance Example

Consider this configuration that references the same datasource multiple times:

settings:
  templates:
    settings:
      gomplate:
        timeout: 5
        datasources:
          metadata:
            url: "https://api.example.com/metadata"

components:
  terraform:
    component1:
      vars:
        region: '{{ (datasource "metadata").region }}'           # API call #1
        account: '{{ (atmos.GomplateDatasource "metadata").account }}'  # API call #2 (cached)

    component2:
      vars:
        region: '{{ (atmos.GomplateDatasource "metadata").region }}'    # Cached
        env: '{{ (atmos.GomplateDatasource "metadata").environment }}'  # Cached

    component3:
      vars:
        cluster: '{{ (atmos.GomplateDatasource "metadata").cluster }}'  # Cached

Without caching: 5 API calls (potential for timeouts, rate limiting) With atmos.GomplateDatasource: 1 API call (4 cache hits)

Best Practices

1. Always use atmos.GomplateDatasource for repeated references to the same external data
2. Configure appropriate timeouts in the gomplate.timeout setting
3. Handle errors gracefully - external sources can fail
4. Use caching strategically - cache expensive or rate-limited APIs
5. Document your datasources - explain what external data is being fetched and why

See Also

• Gomplate Datasources Documentation
• Atmos Template Settings
• Template Functions