Bulk Enrich Companies

This guide covers how to efficiently enrich large datasets — from thousands to hundreds of thousands of companies — using the HG API.

How It Works

The /companies/enrich endpoint accepts up to 25 companies per request. Your throughput depends on your plan's rate limit — see Rate Limiting. On the Standard plan (25 requests/second), that's up to 625 companies per second, or roughly 100,000 companies in under 3 minutes.

Dataset Size	Estimated Time (Standard plan)
1,000 companies	~2 seconds
10,000 companies	~16 seconds
100,000 companies	~3 minutes
500,000 companies	~14 minutes

Before You Start

1. Check your credit balance

Large enrichments consume significant credits. Use the Credits API to verify you have enough before starting:

curl https://api.hginsights.com/data-api/v2/credits \
  -H "Authorization: Bearer $HG_API_KEY"

2. Only request what you need

Each field group adds to your credit consumption. If you only need firmographics, don't request technographics and spend:

{
  "companies": {"domains": ["walmart.com"]},
  "fields": ["firmographics"]
}

3. Use domains or HG IDs

If you have company names but no domains or IDs, use the Company Match endpoint first to resolve them, then enrich using the returned IDs.

Implementation

Step 1: Split into batches of 25

import requests

API_KEY = "your_api_key"
BASE_URL = "https://api.hginsights.com/data-api/v2/companies/enrich"
HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

domains = ["walmart.com", "google.com", ...]  # your full list

# Split into chunks of 25
batches = [domains[i:i+25] for i in range(0, len(domains), 25)]

Step 2: Send requests with rate limiting

Stay within the 25 requests/second limit. A simple approach is to add a small delay between requests:

import time

results = []

for batch in batches:
    payload = {
        "companies": {"domains": batch},
        "fields": ["firmographics", "technographics", "spend"]
    }

    response = requests.post(BASE_URL, json=payload, headers=HEADERS)

    if response.status_code == 200:
        results.extend(response.json()["companies"])
    elif response.status_code == 429:
        # Rate limited — wait and retry
        time.sleep(2)
        response = requests.post(BASE_URL, json=payload, headers=HEADERS)
        results.extend(response.json()["companies"])

    time.sleep(0.05)  # ~20 requests/second, safely under the limit

Step 3: Handle rate limits gracefully

If you hit a 429 (Too Many Requests) response, back off and retry:

def enrich_with_retry(payload, max_retries=3):
    for attempt in range(max_retries):
        response = requests.post(BASE_URL, json=payload, headers=HEADERS)

        if response.status_code == 200:
            return response.json()["companies"]
        elif response.status_code == 429:
            wait_time = 2 ** attempt  # 1s, 2s, 4s
            time.sleep(wait_time)
        else:
            response.raise_for_status()

    raise Exception("Max retries exceeded")

Parallel Requests

For maximum throughput, send requests in parallel using a thread pool. This example sends up to 20 concurrent requests:

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def enrich_batch(batch):
    payload = {
        "companies": {"domains": batch},
        "fields": ["firmographics", "technographics"]
    }
    response = requests.post(BASE_URL, json=payload, headers=HEADERS)
    if response.status_code == 429:
        time.sleep(2)
        response = requests.post(BASE_URL, json=payload, headers=HEADERS)
    response.raise_for_status()
    return response.json()["companies"]

results = []
with ThreadPoolExecutor(max_workers=20) as executor:
    futures = {executor.submit(enrich_batch, batch): batch for batch in batches}
    for future in as_completed(futures):
        results.extend(future.result())

Handling Large Technographics Responses

Some companies have thousands of technology installs. The API paginates technographics results — use the pagination parameter to retrieve all installs:

{
  "companies": {"domains": ["walmart.com"]},
  "fields": ["technographics"],
  "pagination": {
    "technographics": {"limit": 100, "offset": 0}
  }
}

Increment offset by limit until you've retrieved all installs (check installs_count in the response).

Tips

Start small — Test with 100 companies first to validate your pipeline and estimate credit usage
Log failures — Track which batches fail so you can retry them without re-processing the entire dataset
Store results — Write results to a database or file as you go, rather than holding everything in memory
Filter technographics — If you only need specific vendors or products, use filters to reduce response size and credit consumption

Need Recurring Large-Scale Data?

If you're enriching large datasets on a regular basis, the API may not be the most cost-effective option. HG Insights also offers bulk data deliveries tailored to the companies you care about. Contact your account manager to learn about dataset licensing options.