Applied LLM Engineer
Job Description
Wordsmith AI is revolutionizing how legal teams operate by creating an AI command center for in-house legal operations. Backed by Index Ventures, they're rapidly expanding across London, New York, and beyond, serving both fast-growth tech companies and public enterprises.
As an Applied LLM Engineer, you'll be at the forefront of developing AI-native experiences powered by Large Language Models. The role combines technical expertise with practical application, requiring you to build and integrate LLM-powered features while maintaining high standards for security, scalability, and performance. You'll work on everything from backend systems to evaluation frameworks and agentic LLM workflows.
The ideal candidate brings 4+ years of software engineering experience, with strong Python skills and practical experience in building with LLMs. You'll need to demonstrate expertise in creating and maintaining evaluation datasets, running experiments, and implementing observability solutions for LLM systems. The role demands both technical excellence and product thinking, as you'll be collaborating closely with design, product, and legal teams.
What makes this opportunity unique is the chance to work on challenging problems at the intersection of law, language, and artificial intelligence. You'll be part of a collaborative team building something truly innovative in the legal tech space. The company offers competitive compensation including meaningful equity, and you'll have significant influence in shaping both the product and company culture.
Working in either Edinburgh or London in a hybrid setup (minimum 3 days in office), you'll be part of a high-trust environment where you'll have ownership over mission-critical AI systems. If you're passionate about applying cutting-edge AI technology to solve real-world problems in the legal sector, this role offers the perfect blend of technical challenge and practical impact.
Responsibilities For Applied LLM Engineer
- Build and integrate LLM-powered features in production
- Define success metrics (task success, grounding, latency, cost) and set practical SLOs
- Create and maintain evaluation datasets
- Automate quality checks with offline eval harness, CI gates, and safe online tests
- Instrument tracing and logs for prompts, responses, errors, and costs
- Improve quality end-to-end: prompt design, retrieval (RAG), model selection/fine-tuning, and post-processing
- Work across APIs, data pipelines, and backend services
- Collaborate with product and design teams
- Maintain privacy, reliability, and performance standards
Requirements For Applied LLM Engineer
- 4+ years building user-facing software (B2B SaaS is a plus)
- Strong Python skills and comfort with at least one other backend language
- Production experience building with LLMs or similar AI services
- Solid understanding of APIs, cloud infrastructure, and scalable backend design
- Experience with building/maintaining eval datasets and regression suites
- Experience with running offline evals and online experiments
- Knowledge of observability for LLM systems
- Product sense and curiosity
Benefits For Applied LLM Engineer
- Competitive salary
- Meaningful equity
- High-trust, high-impact environment
- Opportunity to work on challenging product problems
- Collaborative team environment
- Minimum 3 days in office
Related Jobs
