Data Engineer

Big Data Engineer (Large-Scale Text Data)

Worcestershire (Hybrid – 1 day per week)
£80,000 - £85,000 + Bonus + Excellent Pension
SC Clearance Required (DV eligible)

The Opportunity

We are supporting a specialist technology consultancy delivering advanced data and AI solutions within highly secure government environments.

This is a genuinely unique engineering role, focused on solving complex challenges at scale by working with extremely large, text-heavy datasets to support real-world, high-impact outcomes.

If you enjoy tackling complex data problems in distributed systems rather than standard reporting or dashboarding, this role will be a strong fit.

The Role

• Work with multi-terabyte to large-scale datasets (100TB+), primarily text-based
• Build and optimise data ingestion and processing pipelines
• Improve batch processing performance and throughput across large datasets
• Solve challenges related to scalability, reliability, and data flow
• Contribute to the development of distributed data platforms
• Collaborate within a highly technical, research-driven environment

What We're Looking For

• Experience working with large-scale datasets (multi-terabyte minimum)
• Strong background in data engineering within distributed systems
• Proven ability to build and optimise data pipelines at scale
• Solid Python skills with exposure to tools such as Spark, PySpark, and Hadoop
• Strong understanding of data ingestion and batch processing methodologies
• Experience with text or unstructured data is advantageous but not essential
• A hands-on, problem-solving mindset with a focus on engineering quality

Environment & Culture

• Calm, focused, and highly technical – not a high-pressure commercial setting
• Strong emphasis on autonomy, trust, and long-term thinking
• Work alongside intelligent, down-to-earth engineers solving complex problems
• Clear opportunities for progression and technical growth

Package

• £80,000 - £85,000 base salary
• Approximately 10% bonus
• Approximately 10% pension contribution
• Hybrid working – one day per week on-site
• Strong benefits package and long-term career development

Security Requirements

Due to the nature of the work, candidates must hold active SC clearance and be willing to undergo DV clearance.