Software Engineer, Data Acquisition

Job summary

Build and operate Rust-based web scrapers and resilient data pipelines, including systems that handle anti-bot defenses, proxies, and browser fingerprinting. Maintain distributed infrastructure, diagnose production issues, onboard data sources through reverse engineering, and develop AI-agent workflows for discovery, integration, and troubleshooting.

Job details

I’m working with a firm looking for a hands-on Software Engineer to build and operate the systems responsible for sourcing the data that powers its trading and research strategies. This is a role for someone who enjoys both building production software and getting into the weeds when things break. You’ll work across web scraping, distributed infrastructure, networking, and AI-powered automation, with real ownership over systems running 24/7. What you'll be working on: Building web scrapers and resilient data pipelines, primarily in Rust. Tackling anti-bot systems, proxy management, browser fingerprinting, and evolving website defenses. Operating distributed infrastructure across multiple cloud providers, bare metal, and mixed operating systems. Diagnosing production issues through logs and telemetry, improving observability, uptime, and reliability. Reverse-engineering web applications and APIs to onboard new data sources. Building AI-agent workflows to discover sources, automate integrations, and troubleshoot pipeline failures. What they're looking for: 2–5 years of professional software engineering experience. Strong backend or systems fundamentals and production engineering in Rust. Networking knowledge across TCP/IP, DNS, HTTP, proxies, and connection handling. Comfort operating distributed infrastructure and debugging issues beyond the application code. Hands-on experience with LLM-based agents and automation workflows. A builder's mindset: you take ownership of what you ship and care about how it performs in production.

What you’ll do

Build and operate Rust-based web scrapers and resilient data pipelines, including systems that handle anti-bot defenses, proxies, and browser fingerprinting. Maintain distributed infrastructure, diagnose production issues, onboard data sources through reverse engineering, and develop AI-agent workflows for discovery, integration, and troubleshooting.

Requirements

Candidates should have 2–5 years of professional software engineering experience, strong backend or systems fundamentals, and production engineering experience in Rust. They should also understand networking, be comfortable operating and debugging distributed infrastructure, and have hands-on experience with LLM-based agents and automation workflows.

Listed skills

• Rust · Preferred
• TCP/IP · Preferred
• DNS · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Rust
• Backend Engineering
• Systems Engineering
• Web Scraping
• Data Pipelines
• Networking
• TCP/IP
• DNS
• HTTP
• Proxy Management
• Distributed Infrastructure
• Production Debugging
• Observability
• Reverse Engineering
• LLM-Based Agents
• Automation Workflows

Job areas

• Software
• Technology
• Data & Analytics
• Engineering
• Finance & Accounting

More jobs from Lawrence Harvey