Senior Data Engineer

2 tygodni temu

Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

Project description

We are embarking on an exciting project that requires expertise in a diverse range of technologies. This project aims to enhance our systems' efficiency and performance through the implementation of cutting-edge tools and methodologies. As part of our team, you will be responsible for deploying and maintaining critical infrastructure components, streamlining development workflows, and optimizing system performance. This project offers an opportunity to work with state-of-the-art technologies and contribute to the evolution of our IT landscape.

Responsibilities

  • 1. Data Engineering and Processing:
  • Utilize Databricks (or equivalent platforms) to develop and maintain data engineering solutions.
  • Write and optimize SQL queries and Python scripts for data processing and analysis.
    2. Data Modeling and Architecture:
  • Design and implement data models, ensuring they align with data warehousing principles and Lakehouse architecture.
  • Develop and maintain data warehouses to support business intelligence and analytics.
    3. ETL Processes:
  • Design, develop, and manage ETL pipelines using Azure Databricks (Spark, Spark SQL, Python, SQL).
  • Implement ETL/ELT design patterns to ensure efficient data extraction, transformation, and loading.
    4. Spark-Based Data Processing:
  • Leverage Spark and Python for large-scale data processing tasks.
  • Develop and optimize data processing workflows using Spark.
    5. Real-Time Data Processing:
  • Implement real-time data processing solutions using Delta Live Tables and Spark Streaming.
  • Ensure timely and accurate processing of streaming data for real-time analytics and applications.
    6. Workflow Design and Optimization:
  • Design data processing workflows that are scalable and maintainable.
  • Continuously monitor and optimize workflows to improve performance and efficiency.
    7. Collaboration and Communication:
  • Work closely with cross-functional teams to understand data requirements and deliver solutions that meet business needs.
  • Communicate effectively with stakeholders to ensure alignment on data engineering initiatives.

SKILLS

Must have

  • 1) Hands on Data Engineering Experience using Databricks (or equivalent) , SQL and Python skills(as a script)
    2) Solid understanding of data modelling, data warehousing principles and Lakehouse architecture
    3) Expert knowledge of ETL using Azure Databricks (Spark, Spark SQL, Python, SQL) and understand the ETL/ETL design patterns
    4) Proficiency in Spark-based Python for data processing.
    5) Strong experience in data modeling and designing data processing workflows using Spark.
    6) Familiarity with Delta Live Tables and Spark Streaming for real-time data processing.

Nice to have

a) Understanding of ETD (Extract, Transform, Load) product and lifecycle.
b) Reconciliation experience


#J-18808-Ljbffr

  • Senior Data Engineer Flink

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska GetInData | Part of Xebia Pełny etat

    Senior Data Engineer Flink Wrocław, Województwo dolnośląskie Senior Data Engineer Flink Wrocław Wrocław, Lower Silesian Voivodeship, Polska Informacje: Lokalizacja: Wrocław Wrocław, Lower Silesian Voivodeship, Polska Dodano: Praca na stanowisku - aktualna Praca Wrocław - Ciekawe oferty pracy w okolicy:WrocławOferty pracy w okolicznych...

  • Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Scalo Sp. z o.o. Pełny etat

    Senior Data Engineer (DataBricks + SQL + Python) Wrocław, Województwo dolnośląskie Senior Data Engineer (DataBricks + SQL + Python) Wrocław Wrocław, Lower Silesian Voivodeship, Polska uczestnictwo w międzynarodowym projekcie z branży automotive, w odniesieniu do usprawnienia systemu do zarządzania materiałami, przygotowania raportu Power BI, który...

  • Senior Data Engineer @

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska SNI Pełny etat

    SNI is serving as a trusted IT Outsourcing partner in line with the needs of World's most prestigious firms and carried out successful projects worldwide.We're on the lookout for a Senior Data Engineer for one of our key clients.Job details:Full-time 1st contract till the end of 2024 Location: pref. Wroclaw area or close locations but open for 100% remote...

  • Senior Backend Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Kalepa Pełny etat

    Who are we?Kalepa is a fully remote startup dedicated to enhancing underwriting performance and driving profitable premium growth for the commercial insurance industry. Kalepa's AI-powered Copilot software enables underwriters to focus on the highest ROI opportunities and quickly evaluate and optimally select risk. Kalepa is backed by IA Ventures, Inspired...

  • Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Rite NRG Pełny etat

    Senior Data Engineer Wrocław Wrocław, Lower Silesian Voivodeship, Polska Dołącz do naszej ekipy Rite NRG jako Senior Data Engineer i stwórz z nami w Polsce centrum dostaw dla lidera branży iGaming. Będziesz miał możliwość rozwijania się w obszarze analizy danych i zarządzania danymi.Jakimi umiejętnościami możesz się pochwalić?Jesteś...

  • Senior Data Engineer z jniemieckim

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska dmTECH Polska Pełny etat

    Senior Data Engineer z jniemieckim Wrocław, Lower Silesian Voivodeship Senior Data Engineer z jniemieckim Wrocław Wrocław, Dolnośląskie, Polska dmTECH to międzynarodowy zespół ponad tysiąca ekspertek i ekspertów z różnych obszarów IT i technologii wspierających e-commerce pracujących dla dm – jednej z największych sieci drogerii w Europie....

  • Mid Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska TransferGo Pełny etat

    TransferGo is a growing fintech scale-up on a mission to make the world a fairer place for migrants. We strive to provide tailored, more affordable financial services that make people's hard-earned money go further. Now in our 12th year, with over 370 employees in offices across Europe, we're crafting a brilliant, relevant product that makes a difference in...

  • Senior Data Engineer Analyst

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Sunscrapers Pełny etat

    Are you ready to take on the challenge?We're looking for a Senior Data Engineer/Analyst to join our team in Warsaw or remotely.Advance your career with Sunscrapers, a leading force in software development, now expanding its presence in a data-centric environment. Join us in our mission to help clients grow and innovate through a comprehensive tech stack and...

  • Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Shaped Thoughts Sp. Z O.o. Pełny etat

    Technologies-expected : Amazon Redshift Snowflake Google Big Query SQL Python Java technologies-optional : AWS Glue Apache Spark Apache Flink Kinesis Kafka about-project : We are looking for a Senior Data Engineer who wants to join an outstanding engineering team that will develop a cutting-edge data platform in the insurance sector.responsibilities : help...

  • Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska j-labs Pełny etat

    Senior Data Engineer Wrocław Wrocław, Lower Silesian Voivodeship, Polska Dołącz do zespołu odpowiedzialnego za przetwarzanie danych potrzebnych do generowania raportów.Masz okazję zostać częścią projektu, który ma na celu zrewolucjonizowanie branży zarządzania inwestycjami.Platforma pozwala użytkownikom na pełną dywersyfikację oraz...

  • Senior Data Engineer Python Pyspark Cloud

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Crestt Pełny etat

    Senior Data Engineer Python Pyspark Cloud Wrocław Wrocław, Lower Silesian Voivodeship, Polska CześćJako Senior/Lead Data Engineer będziesz miał możliwość pracy nad kluczowym projektem naszego klienta, który obejmuje budowanie mechanizmu składowania danych decyzyjnych. Projekt ten oferuje unikalną szansę na zaangażowanie w tworzenie i...

  • Senior Data Virtualization Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Epam Pełny etat

    Description Are you a seasoned Data Virtualization Engineer skilled in Databricks and Denodo? Join our team as a Senior Data Virtualization Engineer, where you'll be at the forefront of an exciting project in the Client's Asset Management division. Leverage your expertise in data virtualization, storage, processing, and streaming data at scale to develop...

  • Mid Senior Data Analytics Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska TransferGo Pełny etat

    TransferGo is a growing fintech scale-up on a mission to make the world a fairer place for migrants. We strive to provide tailored, more affordable financial services that make people's hard-earned money go further. Now in our 12th year, with over 370 employees in offices across Europe, we're crafting a brilliant, relevant product that makes a difference in...

  • Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Limango Pełny etat

    Senior Data Engineer At limango we have a long history specializing in e-commerce. As part of the OTTO Group, one of Europe's top e-commerce companies, we offer the largest product selection for families across platforms in the Netherlands, Poland, Austria, and Germany. Your role and main tasks Development of Central Lakehouse Data Platform based on AWS /...

  • Senior Big Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Scalac Pełny etat

    We are looking for a Senior Big Data Engineer with Spark who will be working on an external project in the fintech area. But what's more important is that you will still be an integral part of the Scalac team and work with the best specialists As a Big Data Engineer, you will:Work on an external project and develop an application that is based on the Hadoop...

  • Senior Data Virtualization Engineer Wroclaw, Poland or Remote

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska EPAM Systems Pełny etat

    Senior Data Virtualization Engineer Wroclaw, Poland or Remote Wrocław, Lower Silesian Voivodeship Senior Data Virtualization Engineer Wroclaw, Poland or Remote 96206Engineer solutions focused on data virtualization, distributed processing, and cloud-based large data storage architectureCreate and implement highly scalable and reliable data distribution...

  • Senior Python Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska EPAM Systems Pełny etat

    Senior Python Engineer (with Pandas and AWS) Poland Wrocław, Lower Silesian Voivodeship Senior Python Engineer (with Pandas and AWS) Poland 95353Build and enhance a highly concurrent backend system for processing large volumes of data, using Python and packages like Pandas, Numpy, SQLAlchemyWork with high performance computingWork with bigdata storage and...

  • Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Scalo Sp. Z O.o. Pełny etat

    Technologies-expected : SQL Data Bricks Data Factory Data Lake Azure SQL DB Azure Dev Ops responsibilities : uczestnictwo w międzynarodowym projekcie z branży automotive, w odniesieniu do usprawnienia systemu do zarządzania materiałami, przygotowania raportu Power BI, który obejmuje pomiar śladu węglowego, w dążeniu do osiągnięcia zerowego poziomu...

  • Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Link Group Pełny etat

    Job Description:As a member of our Finance team, you will be an integral part of our "Financial Transformation" project, aimed at providing the Finance department with accurate and well-structured data. This role involves integrating seamlessly with source systems and establishing data layers that meet business requirements. Your primary responsibilities...

  • Senior Azure Data Engineer

    4 tygodni temu

    Wrocław, dolnośląskie, Polska C&F S.A. Pełny etat

    C&F is a technology services company providing innovative solutions in Data Management, Big Data, Advanced Analytics and Compliance. With a highly integrated team of 400+ experts, for over 20 years C&F has been delivering superior engineering and premium operations to Fortune 500 companies, primarily in the Life Science industry.Senior Azure Data...