Data Engineer
Big Data Engineer (Large-Scale Text Data)
Worcestershire (Hybrid – 1 day per week)
£80,000 - £85,000 + Bonus + Excellent Pension
SC Clearance Required (DV eligible)
The Opportunity
We are supporting a specialist technology consultancy delivering advanced data and AI solutions within highly secure government environments.
This is a genuinely unique engineering role, focused on solving complex challenges at scale by working with extremely large, text-heavy datasets to support real-world, high-impact outcomes.
If you enjoy tackling complex data problems in distributed systems rather than standard reporting or dashboarding, this role will be a strong fit.
The Role
- Work with multi-terabyte to large-scale datasets (100TB+), primarily text-based
- Build and optimise data ingestion and processing pipelines
- Improve batch processing performance and throughput across large datasets
- Solve challenges related to scalability, reliability, and data flow
- Contribute to the development of distributed data platforms
- Collaborate within a highly technical, research-driven environment
What We're Looking For
- Experience working with large-scale datasets (multi-terabyte minimum)
- Strong background in data engineering within distributed systems
- Proven ability to build and optimise data pipelines at scale
- Solid Python skills with exposure to tools such as Spark, PySpark, and Hadoop
- Strong understanding of data ingestion and batch processing methodologies
- Experience with text or unstructured data is advantageous but not essential
- A hands-on, problem-solving mindset with a focus on engineering quality
Environment & Culture
- Calm, focused, and highly technical – not a high-pressure commercial setting
- Strong emphasis on autonomy, trust, and long-term thinking
- Work alongside intelligent, down-to-earth engineers solving complex problems
- Clear opportunities for progression and technical growth
Package
- £80,000 - £85,000 base salary
- Approximately 10% bonus
- Approximately 10% pension contribution
- Hybrid working – one day per week on-site
- Strong benefits package and long-term career development
Security Requirements
Due to the nature of the work, candidates must hold active SC clearance and be willing to undergo DV clearance.
