Big Data Engineer with Apache Spark Expertise

15 godzin temu

Warszawa, Polska Link Group Pełny etat
Job Title: Big Data Engineer with Apache Spark Expertise

**Company Overview**

We are a leading global business solutions provider, Link Group. Our company is dedicated to delivering innovative solutions that transform the way businesses operate.

**Salary:** We offer a highly competitive salary of $120,000 - $180,000 per annum, depending on experience and qualifications.

**Job Description**

As a Big Data Engineer with Apache Spark expertise, you will play a key role in designing, developing, and maintaining large-scale data processing pipelines using PySpark. Your primary responsibility will be to work closely with cross-functional teams to understand data requirements and design optimal solutions.

**Key Responsibilities**

  • Design, develop, and maintain ETL pipelines using PySpark to process and analyze large datasets.
  • Collaborate with data engineers, data scientists, and business stakeholders to understand data requirements and design optimal solutions.
  • Optimize PySpark applications for performance and scalability in distributed computing environments.
  • Work with Hadoop-based data platforms and integrate with other tools like Hive, HDFS, or Kafka.
  • Ensure data quality and integrity through robust validation and monitoring practices.
  • Debug and resolve issues in production and pre-production environments.
  • Document technical solutions and best practices.

**Technical Requirements**

  • 5+ years of experience in data engineering or big data development, with a strong focus on PySpark.
  • Proficiency in Python programming, with experience in libraries commonly used in data processing (e.g., Pandas, NumPy).
  • Strong understanding of Apache Spark concepts: Spark Core, Spark SQL, and Spark Streaming.
  • Experience with distributed data processing frameworks and working in cloud-based environments (e.g., AWS, Azure, GCP).
  • Solid knowledge of big data technologies like Hadoop, Hive, HDFS, Kafka, or Airflow.
  • Hands-on experience with relational and NoSQL databases (e.g., PostgreSQL, Cassandra).
  • Familiarity with CI/CD pipelines and version control (e.g., Git).

**Preferred Qualifications**

  • Experience with streaming data using Spark Streaming or Kafka.
  • Knowledge of machine learning workflows and integration with big data pipelines.
  • Understanding of containerization tools like Docker or orchestration with Kubernetes.

  • Big Data Engineer Position for Large-Scale Data Systems

    1 tydzień temu

    Warszawa, Mazovia, Polska EndySoft Pełny etat

    We are seeking a highly skilled Big Data Engineer to join our data engineering team at EndySoft. This role offers an opportunity to work on complex, large-scale data projects and contribute to shaping the future of data-driven decision-making.Job SummaryThis is a challenging position that requires extensive experience in building and managing large-scale...

  • Big Data Engineer

    1 miesiąc temu

    Warszawa, Mazovia, Polska Jit Team Pełny etat

    Job DescriptionAs a Junior Big Data Engineer at Jit Team, you will be part of a dynamic team working on various projects, including providing first-level support and technical expertise to the DROC team.Key Responsibilities:Executing ETL processes and refreshing UIs for data validationAssisting with Spark/Hive queries for dataset analysisClassifying and...

  • Scala Developer with Big Data Expertise

    4 tygodni temu

    Warszawa, Mazovia, Polska SCALAC Sp. Z O.o. Pełny etat

    About the RoleWe are seeking an experienced Scala developer to join our team as a Big Data Engineer. The ideal candidate will have a strong background in Scala, Hadoop, and Spark, with experience in developing analytics pipelines and data transformations.

  • Big Data Analytics Expert

    2 tygodni temu

    Aleje Jerozolimskie A, Warszawa, Polska Link Group Pełny etat

    Link Group - Job DescriptionWe are seeking a highly skilled PySpark Developer to join our team in big data processing and analytics.Salary:$150,000 - $200,000 per annumAbout the RoleAs a Big Data Analytics Expert, you will design, implement, and optimize large-scale data processing pipelines, leveraging the capabilities of Apache Spark and Python.Key...

  • Big Data Engineering Lead

    1 tydzień temu

    Warszawa, Mazovia, Polska B2B S.A Pełny etat

    Senior Data EngineerYour Key Responsibilities:Data Ingestion and Transformation: Collaborate with cross-functional teams to design, develop, and maintain large-scale data ingestion and transformation processes using Spark and Python.Distributed Data Processing: Build high-performance, parallelized big data processing pipelines to process massive amounts of...

  • Data Engineer with Cloud Expertise

    4 tygodni temu

    Warszawa, Mazovia, Polska Endava Pełny etat

    Data Engineering Role at EndavaWe are seeking a Senior Data Engineer with expertise in cloud technologies to join our team at Endava. As a key member of our data team, you will be responsible for designing and implementing cloud data strategies, including security posture, target operating model, and disaster recovery strategy.Key...

  • Big Data Engineer

    4 tygodni temu

    Warszawa, Mazovia, Polska Jit Team Pełny etat

    About the RoleWe are seeking an experienced Big Data Engineer to join our team in Warsaw. This is a 100% remote or hybrid opportunity, offering a flexible work organization and international collaboration.Key ResponsibilitiesProvide first-level support and technical expertise to the DROC team, handling queries related to data engineering tasks.Execute ETL...

  • Big Data Solutions Engineer

    14 godzin temu

    Warszawa, Mazovia, Polska DCV Technologies Pełny etat

    We are seeking a highly skilled Hadoop Administrator to join our client team on a B2B contract basis.This is an exciting opportunity to work with a multinational client headquarted in the US and a leader in the financial sector, performing Big Data Administration and Engineering tasks on cutting-edge open-source platforms.Job Opportunity: Hadoop...

  • Senior Big Data Developer in Real Estate

    7 dni temu

    Aleje Jerozolimskie A, Warszawa, Polska Link Group Pełny etat

    Job DescriptionWe are seeking a highly skilled Big Data Developer to join our team at Link Group.About the RoleThis is a 12-month contract with 100% remote work option.Primarily Polish hours (9 AM - 5 PM) with occasional flexibility required.English Language proficiency: B2/C1 level.Project OverviewThis challenging project involves working on large-scale...

  • Big Data Engineer

    4 tygodni temu

    Warszawa, Polska Grid Dynamics Poland Sp. z o.o. Pełny etat

    We are looking for a skilled Big Data Engineer to join our team and contribute to the Cassandra Ingestion Service project. You’ll be working with a highly selective client who values quality and precision in every aspect of development. This role requires expertise in handling large-scale data using technologies such as Cassandra,...

  • Senior Big Data Solutions Architect

    4 tygodni temu

    Warszawa, Mazovia, Polska DATUMO Sp. Z O.o. Pełny etat

    At DATUMO Sp. Z O.o., we are seeking a highly skilled Senior Big Data Solutions Architect to join our team.About the RoleWe offer an estimated salary of 120,000 - 150,000 PLN per year, depending on your experience and qualifications.Job DescriptionThe ideal candidate will have at least 3 years of commercial experience in Big Data, with a proven record of...

  • Data Engineer Domestic Bank Fraud Specialist

    4 tygodni temu

    Warszawa, Mazovia, Polska ING Pełny etat

    Fraud Detection Expertise in Data Engineering:As a Fraud Detection Data Engineer, you will play a crucial role in our team at ING Bank Netherlands, leveraging state-of-the-art machine learning models and processing algorithms to identify potential fraud activities within our payment chain.With your expertise in data processing and optimization using Apache...

  • Data Engineering Expert with Cloud Experience

    3 tygodni temu

    Warszawa, Mazovia, Polska AIDA projektai, MB Pełny etat

    We are seeking a highly skilled Big Data Engineer to enhance our data capabilities and contribute to the growth of AIDA projektai, MB.With a focus on innovation and collaboration, this role involves designing and implementing large-scale data pipelines using AWS, leveraging expertise in Apache Spark, Airflow, and Databricks.The ideal candidate will possess...

  • Cloud Data Engineer

    3 tygodni temu

    Warszawa, Mazovia, Polska kdrhumanresourcesconsulting Pełny etat

    We are seeking a skilled Cloud Data Engineer to join our team at kdrhumanresourcesconsulting as a Big Data Reliability Expert. This exciting role offers a unique opportunity to work with cutting-edge technologies and contribute to the success of our organization.**Job Description:**As a Cloud Data Engineer, you will be responsible for ensuring the...

  • Data Engineer in Product Analytics

    1 miesiąc temu

    Warszawa, mazowieckie, Polska ING Hubs Poland Pełny etat

    We are looking for you, if you have:hands-on experience building complex data pipelines,strong experience in data processing and optimization using Apache Spark, good knowledge of data manipulation and transformation,experience with Apache Airflow, Python, SQL, knowledge of software engineering best practices, deployment and provisioning automation tools...

  • Data Engineer in Product Analytics Specialist

    5 dni temu

    al. Jana Pawła II , Warszawa, Polska ING Hubs Poland Pełny etat

    About ING Hubs PolandING Hubs Poland is a major driving force in ING's transformation, aiming to help us become a data-driven digital leader and create tangible value for ING and its customers through world-class analytics models, products, and services.Job DescriptionWe are looking for a skilled Data Engineer to join our Product Analytics squad. As a Data...

  • Product Analytics Engineer

    3 tygodni temu

    Warszawa, Polska ING Hubs Poland Pełny etat

    At ING Hubs Poland, we're seeking a talented Product Analytics Engineer to join our team. As a Product Analytics Engineer, you'll play a vital role in helping us become a data-driven digital leader and create tangible value for ING and its customers through world-class analytics models, products, and services.Job DescriptionWe are looking for an experienced...

  • Data Engineer

    1 tydzień temu

    Warszawa, Mazovia, Polska 7N Pełny etat

    We are seeking experienced professionals to join our team as a Cloud Data Engineer - Data Lake Development. As a key member of our global container liner shipping company, you will play a crucial role in building and managing our Data Lake infrastructure.About the Role:You will design and implement robust data processing pipelines using internal databases,...

  • Big Data Engineer

    1 miesiąc temu

    Warszawa, Mazovia, Polska B2B S.A Pełny etat

    Big Data Job DescriptionWe are seeking an experienced Big Data Developer to work on a data-driven initiative focused on enhancing group risk reporting and supporting various use cases through the development of new and optimization of existing data pipelines.Job ResponsibilitiesThe role involves re-modeling and aggregating data inputs into a common format,...

  • Big Data Engineer for Real-Time Analytics

    3 tygodni temu

    Warszawa, Mazovia, Polska emagine Consulting Pełny etat

    Job Overview:We are seeking a skilled Big Data Engineer to join our team at emagine Consulting in Warsaw. This is an exciting opportunity to work on implementing real-time analytics solutions using Scala and Spark.About the Role:The ideal candidate will have experience in building distributed and highly parallelized BigData processing pipelines that process...