Big Data Engineer with Apache Spark Expertise

1 miesiąc temu

Warszawa, Polska Link Group Pełny etat
Job Title: Big Data Engineer with Apache Spark Expertise

**Company Overview**

We are a leading global business solutions provider, Link Group. Our company is dedicated to delivering innovative solutions that transform the way businesses operate.

**Salary:** We offer a highly competitive salary of $120,000 - $180,000 per annum, depending on experience and qualifications.

**Job Description**

As a Big Data Engineer with Apache Spark expertise, you will play a key role in designing, developing, and maintaining large-scale data processing pipelines using PySpark. Your primary responsibility will be to work closely with cross-functional teams to understand data requirements and design optimal solutions.

**Key Responsibilities**

  • Design, develop, and maintain ETL pipelines using PySpark to process and analyze large datasets.
  • Collaborate with data engineers, data scientists, and business stakeholders to understand data requirements and design optimal solutions.
  • Optimize PySpark applications for performance and scalability in distributed computing environments.
  • Work with Hadoop-based data platforms and integrate with other tools like Hive, HDFS, or Kafka.
  • Ensure data quality and integrity through robust validation and monitoring practices.
  • Debug and resolve issues in production and pre-production environments.
  • Document technical solutions and best practices.

**Technical Requirements**

  • 5+ years of experience in data engineering or big data development, with a strong focus on PySpark.
  • Proficiency in Python programming, with experience in libraries commonly used in data processing (e.g., Pandas, NumPy).
  • Strong understanding of Apache Spark concepts: Spark Core, Spark SQL, and Spark Streaming.
  • Experience with distributed data processing frameworks and working in cloud-based environments (e.g., AWS, Azure, GCP).
  • Solid knowledge of big data technologies like Hadoop, Hive, HDFS, Kafka, or Airflow.
  • Hands-on experience with relational and NoSQL databases (e.g., PostgreSQL, Cassandra).
  • Familiarity with CI/CD pipelines and version control (e.g., Git).

**Preferred Qualifications**

  • Experience with streaming data using Spark Streaming or Kafka.
  • Knowledge of machine learning workflows and integration with big data pipelines.
  • Understanding of containerization tools like Docker or orchestration with Kubernetes.

  • Big Data Engineer with Cloud Expertise

    2 tygodni temu

    Warszawa, Mazovia, Polska Datumo Pełny etat

    About DatumoDatumo is a leading Big Data and Cloud consulting services provider, serving clients globally, particularly in Western Europe, Poland, and the USA. Our core industries include e-commerce, telecommunications, and life science.Our team boasts exceptional individuals committed to delivering high-demanding projects. Team members tend to stay for more...

  • Senior Big Data Engineer

    2 tygodni temu

    Aleje Jerozolimskie A, Warszawa, Polska Link Group Pełny etat

    Job DescriptionWe are seeking a highly skilled Senior Big Data Engineer with at least 5 years of experience in designing, implementing, and optimizing large-scale data processing pipelines. The ideal candidate will leverage Apache Spark and Python to develop efficient ETL pipelines for big data analysis.Key Responsibilities:Develop, test, and maintain...

  • Big Data Ecosystem Manager

    2 tygodni temu

    Warszawa, Polska Kevin Edward Pełny etat

    About Kevin EdwardWe are a technology-driven organization seeking an experienced Big Data Ecosystem Manager to lead our big data infrastructure. If you have expertise in Hadoop, Spark, and Kafka, we want to hear from you.Estimated Salary: $120,000 - $180,000 per yearJob Description:Hadoop, Spark, and Kafka AdministrationManage and support the deployment,...

  • Senior Big Data Engineer

    2 tygodni temu

    Warszawa, Mazovia, Polska DCV Technologies Pełny etat

    Job Title: Senior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to join our team at DCV Technologies in Warsaw, Poland. This is an exciting opportunity to work with a multinational client in the financial sector, performing Big Data Administration and Engineering tasks on cutting-edge open-source platforms.About the Job:The...

  • Big Data Reliability Engineer

    4 tygodni temu

    Warszawa, Polska DCV Technologies Pełny etat

    DCV Technologies is seeking a highly skilled Big Data Reliability Engineer to join our team in Poland.We offer a competitive daily rate of up to 1,250 PLN/day + VAT for this role.About the RoleAs a Big Data Reliability Engineer, you will be responsible for ensuring the reliability and performance of our Big Data platforms, including Hadoop, Spark, and HBASE....

  • Data Engineer with Quantexa Expertise

    4 tygodni temu

    Warszawa, Polska B2B.net S.A Pełny etat

    We are seeking a highly skilled and certified Data Engineer with expertise in Quantexa. The ideal candidate will demonstrate the ability to work independently on complex and well-defined challenges within their area of specialization. He should also possess strong experience in managing technically demanding projects and an in-depth...

  • Data Analytics Engineer for Product Insights

    7 dni temu

    Warszawa, Polska ING Hubs Poland Pełny etat

    At ING Hubs Poland, we're looking to hire a Data Analytics Engineer for Product Insights. This role is perfect for someone with hands-on experience building complex data pipelines using Apache Spark and strong knowledge of data processing and optimization.We're aiming to create a data-driven digital leader and bring tangible value to our organization through...

  • Big Data Ecosystem Architect

    2 tygodni temu

    centrum, Warszawa, Polska Kevin Edward Pełny etat

    Job DescriptionWe are seeking a seasoned Big Data Ecosystem Architect to spearhead the design and deployment of large-scale big data environments at Kevin Edward.The ideal candidate will have a deep understanding of distributed computing principles, data storage systems, and expertise in Hadoop, Spark, and Kafka administration.Key Responsibilities:Hadoop,...

  • Senior Big Data Architect

    2 tygodni temu

    Warszawa, Mazovia, Polska Jit Team Pełny etat

    **Job Description:**We are seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will work with the Hadoop ecosystem to develop and manage big data solutions.**Key Responsibilities:**Develop and manage big data solutions using Hadoop ecosystemSet up CI/CD pipelines and work with Docker and Kubernetes platformsManage relational...

  • Big Data Solutions Engineer

    1 miesiąc temu

    Warszawa, Mazovia, Polska DCV Technologies Pełny etat

    We are seeking a highly skilled Hadoop Administrator to join our client team on a B2B contract basis.This is an exciting opportunity to work with a multinational client headquarted in the US and a leader in the financial sector, performing Big Data Administration and Engineering tasks on cutting-edge open-source platforms.Job Opportunity: Hadoop...

  • Big Data Engineer

    2 miesięcy temu

    Warszawa, Polska Grid Dynamics Poland Sp. z o.o. Pełny etat

    We are looking for a skilled Big Data Engineer to join our team and contribute to the Cassandra Ingestion Service project. You’ll be working with a highly selective client who values quality and precision in every aspect of development. This role requires expertise in handling large-scale data using technologies such as Cassandra,...

  • Big Data Engineer

    1 tydzień temu

    Warszawa, Polska DCG Pełny etat

    Job Description:We are seeking a highly skilled Big Data Engineer to join our team as an Architecture Specialist.About the Role:The successful candidate will be responsible for designing and implementing scalable, reliable, and secure data platform architectures using Azure services and Azure Databricks.You will work closely with Product Owners, Data...

  • Senior Data Engineering Specialist

    1 dzień temu

    Warszawa, Polska Link Group Pełny etat

    Job Title: Senior Data Engineering SpecialistAt Link Group, we are seeking a highly skilled Senior Data Engineering Specialist to join our team. This is a challenging and rewarding role that requires expertise in big data processing and analytics.Salary: $140,000 - $180,000 per annumJob Description:We are looking for a senior-level data engineer with at...

  • Senior Quantexa Engineer with Advanced Data Expertise

    2 tygodni temu

    Warszawa, Mazovia, Polska B2B S.A Pełny etat

    We are seeking a highly skilled and certified Data Engineer with expertise in Quanetxa. The ideal candidate will demonstrate the ability to work independently on complex and well-defined challenges within their area of specialization. He should also possess strong experience in managing technically demanding projects and an in-depth understanding of relevant...

  • Data Engineer in Product Analytics

    2 miesięcy temu

    Warszawa, mazowieckie, Polska ING Hubs Poland Pełny etat

    We are looking for you, if you have:hands-on experience building complex data pipelines,strong experience in data processing and optimization using Apache Spark, good knowledge of data manipulation and transformation,experience with Apache Airflow, Python, SQL, knowledge of software engineering best practices, deployment and provisioning automation tools...

  • Data Engineer in Product Analytics Specialist

    1 miesiąc temu

    al. Jana Pawła II , Warszawa, Polska ING Hubs Poland Pełny etat

    About ING Hubs PolandING Hubs Poland is a major driving force in ING's transformation, aiming to help us become a data-driven digital leader and create tangible value for ING and its customers through world-class analytics models, products, and services.Job DescriptionWe are looking for a skilled Data Engineer to join our Product Analytics squad. As a Data...

  • Data Engineer

    1 miesiąc temu

    Warszawa, Mazovia, Polska 7N Pełny etat

    We are seeking experienced professionals to join our team as a Cloud Data Engineer - Data Lake Development. As a key member of our global container liner shipping company, you will play a crucial role in building and managing our Data Lake infrastructure.About the Role:You will design and implement robust data processing pipelines using internal databases,...

  • Big Data Reliability Engineer

    1 miesiąc temu

    City, Warszawa, Polska DCV Technologies Pełny etat

    Big Data Reliability Engineer (Remote in Poland, occasional office visits in Warsaw)We are seeking an experienced Hadoop Admin/Site Reliability Engineer to join our team driving innovation in Big Data and ensuring platform reliability for a global leader in the digital payments space.About Our ClientOur client is a world-leading digital payments network that...

  • Data Engineer for Banking Solutions

    5 dni temu

    Warszawa, Mazovia, Polska emagine Polska Pełny etat

    At emagine Polska, we are seeking an experienced Data Engineer to join our team in Warsaw, Gdynia or Gdańsk. The successful candidate will be responsible for driving release activities, designing and maintaining applications built on the Quantexa platform.We offer a competitive salary of 170 PLN per hour and a hybrid work model with one day per week spent...

  • Senior Data Solutions Engineer

    2 tygodni temu

    Warszawa, Polska Jit Team Pełny etat

    About the RoleWe are seeking a skilled Data Engineer to join our team as a Senior Data Solutions Engineer. This role will involve designing, developing, and implementing big data solutions using Hadoop ecosystem.The ideal candidate will have expertise in Hadoop Cloudera distribution, proficiency in Docker and Kubernetes platforms, and strong knowledge of...