Senior Data Engineer

Who we are We are on a mission to make all science open, enabling the research community to develop the solutions we all need to live healthy lives on a healthy planet. Frontiers is one of the world's largest and most impactful research publishers, dedicated to making peer-reviewed, quality-certified science openly accessible. Our articles have been viewed 4 billion times, reflecting the power of research that is open for all. Follow the links below to learn more about our work. www.frontiersin.org/about/annual-reports www.youtube.com/watch?v=87ejFfnQzko What we can offer you At Frontiers, our working model for new roles includes a balanced approach that fosters collaboration through regular engagement at our office hubs in key locations. While we value flexibility, we also believe that shared in-person time strengthens team culture, trust, and productivity. We provide a range of benefits across our global locations, including but not limited to: Private Medical Insurance (Luxmed) fully covered by Frontiers. Multisport Plus basic plan funded by Frontiers. 26 vacation days max , plus 4 Wellbeing Days . Employee Assistance Programme (24/7 mental health & wellbeing support). Volunteering Days Work From Another Country Annual Bonus About Us We are one of the world's leading open access publishers, dedicated to advancing knowledge and making research accessible to everyone. Our engineering team builds and maintains high-quality data systems that are the backbone of our publishing and peer review pipelines. Our data products are used by hundreds of internal users across editorial, publishing, and review workflows. These systems manage complex, heterogeneous, and high-volume datasets, enabling smooth operations and supporting data-driven decisions across the organization. By joining us, you'll have a direct impact on advancing open science and accelerating research discovery. Role Overview As a Senior Data Engineer , you will design, build, and optimize large-scale data pipelines and systems on Azure. You'll be responsible not only for delivery, but also for architecture, scalability, testing, and cost efficiency of our data platform. You will ensure our data is accurate, secure, and reliable while playing a strategic role in influencing technical direction. Beyond hands-on engineering, you will mentor peers, collaborate with data scientists and analysts, and contribute to shaping the future of our data ecosystem. Responsibilities Architecture & Design: Shape data models, define engineering standards, and design pipelines optimized for performance, scalability, and cost efficiency. Pipeline Development: Build and maintain scalable data pipelines using Databricks, Delta Lake, Python, SQL, and Spark. Performance Tuning: Optimize pipelines for high-throughput workloads (billions of records) and explore near-real-time streaming where relevant. Cloud Cost Optimization: Monitor and optimize Azure costs to ensure efficient resource usage. Workflow Orchestration: Build and optimize pipelines in Azure Data Factory. Quality & Testing: Apply dbt for data quality checks and implement testing frameworks (unit, integration, data validation) to ensure robust pipelines. CI/CD: Implement CI/CD for data pipelines with Azure DevOps. Collaboration: Work closely with data scientists and analysts to deliver pipelines that power advanced analytics and machine learning. Strategic Input: Contribute to technical direction and roadmap planning beyond execution. Documentation & Mentorship: Document complex systems, mentor junior engineers, and foster a collaborative culture. Trade-off Evaluation: Weigh complexity, costs, scalability, and coordination overhead when proposing solutions. Requirements 4–8 years of experience in Data Engineering or related fields. Strong expertise in SQL and Apache Spark. Proven hands-on experience with Databricks and Delta Lake (most critical skills). Experience with BigQuery (nice to have). Experience with dbt (for data quality checks). Good understanding of Docker and containerized environments. Experience with CI/CD pipelines (Azure DevOps). Experience with testing data pipelines (e.g., Great Expectations, dbt tests, unit/integration testing in Spark or Python). Deep understanding of data engineering principles, data governance, and quality standards. Strong problem-solving and analytical skills, with attention to detail. Ability to collaborate across teams and communicate with both technical and non-technical stakeholders. Leadership and mentoring experience. Ability to balance trade-offs and apply judgment in complex technical decisions. Nice to have: Understanding of statistics for data validation and analysis; familiarity with graph databases (Neo4j). Apply now Please click the 'apply' button on this page and submit your application in English. Please inform us if you require any special accommodations to participate fully in our recruitment experience. You can contact us at careers@frontiersin.org for any additional support. Equal opportunity statement Frontiers actively embraces diversity and is a safe and welcoming workplace. Recruitment is free from discrimination – including based on race, national or ethnic origin, age, religion, disability, sex, gender identity or sexual orientation. With employees from more than 50 different nations, our diversity creates vibrant teams and constantly challenges us to appreciate multiple perspectives.