Lead Data Engineer
Responsibilities
Lead the design, build, and optimisation of the organisation’s Microsoft Fabric-based data platform, implementing a medallion architecture (bronze, silver, gold) to ensure data is structured, governed, and analytics-ready.
Architect and manage scalable data pipelines within Microsoft Fabric, including Data Factory pipelines with REST API integrations, to automate and orchestrate multi-source ingestion workflows.
Own the end-to-end delivery of enterprise-level Power BI reporting solutions, including advanced DAX modelling, Power Query transformations, and performance tuning across large semantic models.
Design and maintain Power BI Embedded integrations, ensuring reliable report rendering, token management, row-level security, and seamless application-level embedding for internal and external users.
Establish and execute robust processes for rolling out Power BI reports into production, including structured Dev/QA/Prod workspace strategies, deployment pipelines, and automated validation gates.
Define and enforce data quality standards across the platform, building testing frameworks that span ingestion, transformation, and semantic model layers to ensure reporting accuracy and trust.
Use Python and PySpark notebooks within Fabric for data transformation, automation, data quality checks, and AI-related scripting in collaboration with the data science team.
Translate complex business requirements into scalable, maintainable reporting and data platform solutions, acting as a bridge between technical delivery and stakeholder objectives.
Monitor and optimise platform and report performance to ensure reliability, cost-efficiency, and scalability across the Microsoft Fabric and Power BI environments.
Mentor and guide data engineers, establishing best practices for development standards, code review, documentation, and deployment workflows.
Maintain comprehensive documentation for data architecture, pipeline logic, semantic models, and operational processes, supporting team collaboration and business continuity.
Manage and prioritise ad hoc data requests through structured intake processes, ensuring high accuracy, timely turnaround, and clarity of deliverables.
Required Skills & Experience
Strong hands-on experience with Microsoft Fabric, including Lakehouses, Warehouses, Data Factory pipelines, and Notebooks.
Demonstrated experience designing and implementing medallion architecture (bronze/silver/gold) patterns in a modern cloud data platform.
5+ years of hands-on development experience with Power BI, including advanced DAX, Power Query, semantic model design, and Power BI Service administration.
Proven track record of building and executing production rollout processes for Power BI reports, including deployment pipelines, workspace strategies, and validation gates.
Strong experience with Power BI Embedded, including capacity management, token-based authentication, RLS enforcement, and troubleshooting embedding scenarios.
Proficiency in T-SQL and SQL-based querying for cloud data platforms and relational databases, with deep understanding of schema design, indexing, and optimisation best practices.
Experience with SQL analytics endpoints within Microsoft Fabric or equivalent cloud-native SQL tooling.
Proficiency in Python and/or PySpark for data transformation, automation, testing, and AI-related development.
Strong understanding of Row-Level Security (RLS), object-level security, workspace governance, and access control in enterprise Power BI environments.
Experience with Azure DevOps or equivalent CI/CD tooling to automate testing, deployment, and release management of data solutions.
Excellent analytical thinking, problem-solving, and stakeholder communication skills with a track record of influencing technical decisions.
Experience leading or mentoring junior engineers in a data platform or BI team context.
Key Competencies
Data Platform Architecture: Demonstrates ability to design, build, and govern a Microsoft Fabric-based data platform using medallion architecture principles.
SQL & Data Modelling: Demonstrates advanced proficiency in SQL and T-SQL for querying, modelling, and optimising complex datasets within cloud-native environments.
Power BI Mastery: Demonstrates deep expertise in DAX, Power Query, semantic model design, Power BI Embedded, and enterprise reporting governance including RLS.
Production Delivery: Demonstrates proven ability to establish and execute structured rollout processes for Power BI reports, from development through to production.
CI/CD & Deployment: Demonstrates ability to manage and automate deployments using Azure DevOps Pipelines, Fabric deployment pipelines, and Power BI deployment workflows.
Python & Notebook Development: Demonstrates capability in using Python and PySpark within Fabric for data transformation, quality assurance, and AI team support.
Data Quality Ownership: Demonstrates a proactive approach to data quality, building validation frameworks and testing processes across the full data lifecycle.
Performance Engineering: Demonstrates attention to performance tuning, capacity planning, cost-efficiency, and reliability across platform and reporting infrastructure.
Technical Leadership: Demonstrates ability to mentor junior engineers, set development standards, and drive best practices across the data team.
Cross-Team Collaboration: Demonstrates strong communication and delivery alignment with both technical and non-technical stakeholders at all levels.
Lifecycle Ownership: Demonstrates end-to-end ownership across the full data lifecycle from ingestion and modelling to visualisation, deployment, and ongoing governance.
Qualifications
Degree or equivalent qualification in Data Science, Computer Science, Information Systems, Mathematics, or a related field.
Microsoft certifications such as Fabric Analytics Engineer Associate (DP-600), Power BI Data Analyst Associate (PL-300), or Azure Data Engineer Associate (DP-203) preferred or in progress.
Proven experience leading data platform or BI initiatives in a professional environment, with demonstrable production delivery outcomes.
What's in it for You?
EMI company share scheme, meaning you would own a percentage of shares, turning into significant monetary value in time.
Competitive salary.
Westfield Health cash plan.
1-2-1 Sanctus coaching sessions to support personal and professional development with proactive mental wellbeing support.
Flexible working.
A paid day off on your birthday.
Aviva Life insurance.
Workplace Nursery Benefit.
Long service awards to celebrate key employment anniversaries.
Excellent discounts/wider wallet.
5% matched contribution pension scheme
Pension salary exchange scheme
25 days holiday + bank holidays (33 days).
Paid wellbeing days, volunteer days, and study days.
Quarterly All Star award.
Annual company Summer event.
Modern homeworking tech kit.
Enhanced maternity & paternity pay.
And being part of something AMAZING!
Cognassist is a Disability Confident Committed employer, and we welcome your application even if you believe you do not meet all of the above criteria - your unique skills, and experiences are valued, and your contribution could be exactly what we need to grow together.
- Department
- Engineering
- Locations
- Remote
- Remote status
- Fully Remote
Uniqueness is a superpower
- Our culture really matters to us at Cognassist.
- We know our vision of creating a more inclusive world, starts with us.
- Our commitment to inclusion across race, gender, age, religion, identity, and experience drives us forward every day. We welcome candidates from all walks of life.
- We’re committed to authentically investing in our people.
- We provide our teams with the best possible benefits, the latest tools, and nurturing environments to work in.
- We always strive to empower people to do their best work and deliver real impact.