Lead Data Engineer – Data Ingestion

6 dni temu

Gdańsk, Pomerania, Polska Hapag-Lloyd Pełny etat 60 000 zł - 120 000 zł rocznie
Description

We are looking for a hands-on Lead Data Engineer – Data Ingestion to guide the development and execution of our data ingestion processes into the Data Lakehouse. In this role, you will be responsible for designing scalable and reliable data pipelines, transforming source inputs into trusted Bronze-layer assets. You'll work closely with data owners, source system teams, and the Product Manager to ensure successful onboarding of new sources and to drive continuous improvements in data quality, automation, and performance. You'll lead a small team of engineers by example—actively coding, reviewing solutions, and setting engineering standards. This is a role for someone who is passionate about data integration and wants to shape the way data flows through our platform. 

Responsibilities

Design and maintain scalable data ingestion pipelines from internal and external sources including Kafka, APIs, SFTP, and file-based systems. 

Establish and maintain best practices for ingesting and processing structured and semi-structured data (e.g., JSON, Avro, CSV). 

Align ingestion pipelines with enterprise data architecture and naming conventions. 

Manage batch and micro-batch processing; contribute to future transition towards streaming ingestion. 

Transform and validate ingested data into the Bronze layer within the Data Lakehouse. 

Follow standardized onboarding scenarios and integrate new data sources in a consistent and governed way. 

Apply schema and metadata standards using Unity Catalog and Collibra, ensure proper lineage tracking. 

Collaborate with Product Manager and domain teams to scope and deliver ingestion use cases. 

Ensure data quality and validation as part of the ingestion process. 

Implement improvements in pipeline performance, cost efficiency, and automation.  

Collaborate closely with teams using the ingested data to ensure usability and traceability. 

Mentor team members, perform code reviews, and contribute to internal engineering standards and documentation. 

Collaborate with the Data Governance team to ensure traceability, cataloging, and access control across new data domains. 

Define and maintain onboarding playbooks and reusable ingestion templates. 

Actively participate in backlog grooming, planning sessions, and technical refinement. 

Qualifications

Minimum 6 years of experience in data engineering with a focus on ingestion and transformation. 

Hands-on experience with Databricks (Spark), Python, Kafka, and cloud data processing on AWS (e.g., S3). 

Experience with orchestration and workflow management (Airflow on Astronomer). 

Strong SQL and Spark (preferably PySpark) skills. 

Working knowledge of metadata governance via Unity Catalog and Collibra. 

Familiarity with infrastructure-as-code tools (Terraform) and version control (GitLab). 

Proven experience building reliable and performant ingestion pipelines. 

Ability to collaborate with stakeholders, drive improvements, and document processes clearly. 

Comfortable in an agile, product-oriented environment. 

Experience working with semi-structured data and schema evolution techniques. 

Knowledge of distributed data systems and challenges related to ingestion at scale. 

Ability to lead technical conversations with external vendors or source system teams. 

Understanding of testing frameworks for data pipelines and CI/CD validation. 

Familiarity with tools for data profiling, schema validation, and automated lineage capture. 

  • Lead Data Engineer – Data Ingestion

    1 tydzień temu

    Gdańsk, Pomerania, Polska Hapag-Lloyd AG Pełny etat 80 000 zł - 120 000 zł rocznie

    We are looking for a hands-on Lead Data Engineer – Data Ingestion to guide the development and execution of our data ingestion processes into the Data Lakehouse. In this role, you will be responsible for designing scalable and reliable data pipelines, transforming source inputs into trusted Bronze-layer assets. You'll work closely with data owners, source...

  • Senior Data Engineer

    1 tydzień temu

    Gdańsk, Pomerania, Polska EPAM Systems Pełny etat 90 000 zł - 120 000 zł rocznie

    We are looking for a dynamic, hands-onSenior Data Engineerfor a full-stack AI native Data Eng team for building a cutting-edge quantitative investment platform.This role requires 5 days of work per week from the office in Krakow or Gdansk.ResponsibilitiesImplement data pipelines for structured, unstructured, and operational data sourcesEnsure efficient data...

  • Lead Data Engineer – Core Data Platform

    6 dni temu

    Gdańsk, Pomerania, Polska Hapag-Lloyd Pełny etat 80 000 zł - 120 000 zł rocznie

    DescriptionWe are seeking an experienced Lead Data Engineer– Core Data Platform to drive the development and stability of our cloud-native data infrastructure. This role is crucial in ensuring our data platform is secure, reliable, and scalable. You'll be responsible for the provisioning and automation of key components including Databricks, Airflow...

  • Global GTM Data Integration and Fusion Lead

    1 tydzień temu

    Gdańsk, Pomerania, Polska KINESSO Poland Pełny etat 60 000 zł - 120 000 zł rocznie

    At KINESSO, we offer a unique perspective on the marketing landscape. We're building the future of performance marketing, fueled by a dynamic data ecosystem. This includes the breadth and depth of consumer data – encompassing demographics, lifestyle, purchase behavior, and more – combined with first-party client data and the rich, real-time signals from...

  • Data Platform Lead

    1 tydzień temu

    Gdańsk, Pomerania, Polska TF Bank Pełny etat 90 000 zł - 120 000 zł rocznie

    We are seeking a highly skilled Data Platform Lead to drive the design and development of Avarda Bank's next-generation data platform. This is a pivotal role where you will own the architecture, governance, and build of a cloud-native, event-driven platform that powers both advanced analytics and the consumer journey across Europe. You will combine strategic...

  • Data Engineer

    6 dni temu

    Gdańsk, Pomerania, Polska Sanoma Learning Pełny etat 40 000 zł - 80 000 zł rocznie

    Important:This position isonly for candidates legally residing in Polandand operating under a registered sole proprietorship (JDG).As part of onboarding, we kindly invite new joiners to our Warsaw office on the first day for a short introduction and identity verification.Please note that applications from outside Poland cannot be considered.About UsSanoma...

  • AI Validation engineer

    1 tydzień temu

    Gdańsk, Pomerania, Polska KBC Technologies Group Pełny etat 55 000 zł - 85 000 zł rocznie

    Job description: AI Validation engineerValidation Engineer to ensure the quality, reliability, and compliance of our dual-pipeline data ingestion systems and full RAG application deployments. This critical role involves designing, executing, and documenting validation protocols to verify that all data processes and AI applications meet established...

  • Data DevOps Engineer

    6 dni temu

    Gdańsk, Pomerania, Polska Hard Rock Digital Pełny etat 40 000 zł - 80 000 zł rocznie

    What are we building?Hard Rock Digital is a team focused on becoming the best online sportsbook, casino, and social casino company in the world. We're building a team that resonates passion for learning, operating, and building new products and technologies for millions of consumers. We care about each customer's interaction, experience, behaviour, and...

  • Senior Data Engineer

    1 tydzień temu

    Gdańsk, Pomerania, Polska B2B S.A Pełny etat 9 600 zł - 14 400 zł rocznie

    Jako Senior Data Engineer / ML będziesz odpowiedzialny za kompleksowe zarządzanie cyklem życia modeli uczenia maszynowego — od etapu rozwoju, poprzez wdrożenie i monitorowanie, aż po zapewnienie ich optymalnej wydajności i niezawodności.Hybrydowo (3 dni w tygodniu z biura)Senior Data Engineer / MLTwój zakres obowiązkówImplementacja zasad MLOps...

  • engineer, product data management

    1 tydzień temu

    Gdańsk, Pomerania, Polska Rockwell Automation Pełny etat 40 000 zł - 80 000 zł rocznie

    Rockwell Automation is a global technology leader focused on helping the world's manufacturers be more productive, sustainable, and agile. With more than 28,000 employees who make the world better every day, we know we have something special. Behind our customers - amazing companies that help feed the world, provide life-saving medicine on a global scale,...