Senior Data Engineer Spark AWS

4 tygodni temu

Jaworska Wrocław, Polska Addepto Pełny etat
Senior Data Engineer Spark AWS

Addepto is a leading consulting and technology company specializing in AI and Big Data, helping clients deliver innovative data projects. We partner with top-tier global enterprises and pioneering startups, including Rolls Royce, Continental, Porsche, ABB, and WGU. Our exclusive focus on AI and Big Data has earned us recognition by Forbes as one of the top 10 AI companies.

As a Senior Data Engineer (Spark, AWS) you will have the exciting opportunity to work with a team of technology experts on challenging projects across various industries, leveraging cutting-edge technologies. Here are some of the projects we are seeking talented individuals to join:

  • Design and development of a universal data platform for global aerospace companies. This Azure and Databricks-powered initiative combines diverse enterprise and public data sources. The data platform is in the early stages of development, covering the design of architecture and processes and allowing freedom for technology selection.
  • Design and development of the data platform for managing electric and hybrid vehicle data. This project involves building a data pipeline for electric vehicle data, processing thousands of signals efficiently through streaming and batch services. The data powers IoT applications for business intelligence, customer support, maintenance, and AI insights, offering a chance to work with cutting-edge technology in electric mobility.
  • Design of the data transformation and following data ops pipelines for a global car manufacturer. This project aims to build a data processing system for both real-time streaming and batch data. We'll handle data for business uses like process monitoring, analysis, and reporting, while also exploring LLMs for chatbots and data analysis. Key tasks include data cleaning, normalization, and optimizing the data model for performance and accuracy.

Your main responsibilities:

  • Design and develop scalable data management architectures, infrastructure, and platform solutions for streaming and batch processing using Big Data technologies like Apache Spark, Hadoop, Databricks, Snowflake.
  • Design streaming pipelines using Apache Spark and Kafka.
  • Design and implement data management and data governance processes and best practices.
  • Contribute to the development of CI/CD and MLOps processes
  • Develop applications to aggregate, process, and analyze data from diverse sources.
  • Collaborate with the Data Science team on Machine Learning projects, including text/image analysis and predictive model building.
  • Develop and organize data transformations using DBT and Apache Airflow.
  • Translate business requirements into technical solutions and ensure optimal performance and quality.

What you'll need to succeed in this role:

  • 5+ years of proven commercial experience in implementing, developing, or maintaining Big Data systems.
  • Strong programming skills in Python: writing a clean code, OOP design.
  • Experience in designing and implementing data governance and data management processes.
  • Familiarity with Big Data technologies like Spark, Cloudera, Kafka, Airflow, NiFi, Docker, Kubernetes, Iceberg, Trino or Hudi.
  • Proven expertise in implementing and deploying solutions in cloud environments (with a preference for Azure and AWS).
  • Experience with Databricks and / or Snowflake is an asset.
  • Excellent understanding of dimensional data and data modeling techniques.
  • Excellent communication skills and consulting experience with direct interaction with clients.
  • Ability to work independently and take ownership of project deliverables.
  • Master's or Ph.D. in Computer Science, Data Science, Mathematics, Physics, or a related field.

Discover our perks & benefits:

  • Work in a supportive team of passionate enthusiasts of AI & Big Data.
  • Engage with top-tier global enterprises and cutting-edge startups on international projects.
  • Enjoy flexible work arrangements, allowing you to work remotely or from modern offices and coworking spaces.
  • Accelerate your professional growth through career paths, knowledge-sharing initiatives, language classes, and sponsored training or conferences.
  • Choose from various employment options: B2B, employment contracts, or contracts of mandate.
  • Make use of 20 fully paid days off available for B2B contractors and individuals under contracts of mandate.
  • Participate in team-building events and utilize the integration budget.
  • Celebrate work anniversaries, birthdays, and milestones.
  • Access medical and sports packages, eye care, and well-being support services, including psychotherapy and coaching.
  • Get full work equipment for optimal productivity, including a laptop and other necessary devices.
  • With our backing, you can boost your personal brand by speaking at conferences, writing for our blog, or participating in meetups.
  • Experience a smooth onboarding with a dedicated buddy, and start your journey in our friendly, supportive, and autonomous culture.

Are you interested in Addepto and would like to join us?

Get in touch We are looking forward to receiving your application. Would you like to know more about us?

Visit our website (career page) and social media (Facebook, LinkedIn, Instagram).

  • Senior Data Platform Engineer

    3 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Limango Pełny etat

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Limango, a leading e-commerce company. As a key member of our data engineering team, you will be responsible for designing and implementing a Central Lakehouse Data Platform based on the AWS/Databricks ecosystem.Your ResponsibilitiesDesign and implement data infrastructure...

  • Senior Data Engineer

    4 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Limango Pełny etat

    About the RoleWe are seeking a Senior Data Engineer to join our team at Limango, a leading e-commerce company in Europe. As a Senior Data Engineer, you will be responsible for developing and managing our Central Lakehouse Data Platform based on the AWS/Databricks ecosystem.Main ResponsibilitiesDesign and implement data infrastructure using Terraform and...

  • Data Engineer

    1 tydzień temu

    Wrocław, Polska SCALO Sp. z o.o. Pełny etat

    W Scalo zajmujemy się dostarczaniem projektów software"owych i wspieraniem naszych partnerów w rozwijaniu ich biznesu. Tworzymy oprogramowanie, które umożliwia ludziom dokonywanie zmian, działanie w szybszym tempie oraz osiąganie lepszych rezultatów. Jesteśmy firmą, która wykorzystuje szerokie spektrum usług IT, żeby pomagać klientom. Obszary...

  • Senior Data Engineer

    6 miesięcy temu

    Wrocław, Polska Limango Pełny etat

    Senior Data Engineer As limango we create a company which has been specializing in e-commerce for 17 years. Together with platforms in the Netherlands, Poland, Austria and Germany, we are part of the OTTO Group, one of Europe's leading e-commerce companies. We are the shopping platform with the largest selection of products for the whole family! We work...

  • Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Limango Pełny etat

    As Limango, we strive to be a pioneering e-commerce company, leveraging innovative technologies to drive business growth. We are seeking a skilled Senior Data Engineer to join our team and contribute to the development of our Central Lakehouse Data Platform.Key Responsibilities:Design and implement data infrastructure using Terraform, AWS, and...

  • AWS Cloud Data Solutions Expert

    4 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Engenious Pełny etat

    pWe are seeking an experienced AWS Cloud Data Solutions Expert to join our team at Engenious. The ideal candidate will have a strong background in designing and developing streaming data solutions using Databricks, Spark, PySpark, and Pandas./ppKey Responsibilities:/pulliDesign and develop scalable data pipelines using AWS services such as Lambda, Kinesis,...

  • Senior Data Engineer

    4 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Limango Pełny etat

    About LimangoLimango is a leading e-commerce company with a strong presence in Europe. We are part of the OTTO Group, one of the largest e-commerce companies in Europe. Our company culture values work-life balance, respect, trust, and equality. We are looking for a talented Data Engineer to join our team.Your Role and ResponsibilitiesDesign and develop a...

  • Data Engineer

    4 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Ciklum Pełny etat

    About the RoleCiklum is seeking a skilled Data Engineer to join our team in Poland. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems that support our business operations.ResponsibilitiesDesign and implement data pipelines to extract, transform, and load data from various sourcesDevelop and maintain...

  • Data Engineer

    1 tydzień temu

    Wrocław, Polska SCALO Pełny etat

    W Scalo zajmujemy się dostarczaniem projektów software'owych i wspieraniem naszych partnerów w rozwijaniu ich biznesu. Tworzymy oprogramowanie, które umożliwia ludziom dokonywanie zmian, działanie w szybszym tempie oraz osiąganie lepszych rezultatów. Jesteśmy firmą, która wykorzystuje szerokie spektrum usług IT, żeby pomagać klientom. Obszary...

  • Senior Data Engineer

    1 tydzień temu

    Wrocław, Polska SCALO Sp. z o.o. Pełny etat

    W Scalo zajmujemy się dostarczaniem projektów software"owych i wspieraniem naszych partnerów w rozwijaniu ich biznesu. Tworzymy oprogramowanie, które umożliwia ludziom dokonywanie zmian, działanie w szybszym tempie oraz osiąganie lepszych rezultatów. Jesteśmy firmą, która wykorzystuje szerokie spektrum usług IT, żeby pomagać klientom. Obszary...

  • Data Engineer Blockchain Innovator

    1 tydzień temu

    Wrocław, Polska BCF Software Sp. z o.o. Pełny etat

    We are seeking a highly skilled Data Engineer to contribute to the development of an innovative blockchain crypto solution at BCF Software Sp. z o.o.**Estimated Salary: 120,000 - 180,000 PLN per year**About the RoleThis exciting opportunity offers the chance to work on cutting-edge technology that helps audit teams verify client assets, including...

  • Data Scientist/ML Engineer

    4 tygodni temu

    Jaworska -, Wrocław, Polska Addepto Pełny etat

    About AddeptoAddepto is a leading consulting and technology company specializing in AI and Big Data, helping clients deliver innovative data projects. We partner with top-tier global enterprises and pioneering startups, including Rolls Royce, Continental, Porsche, ABB, and WGU. Our exclusive focus on AI and Big Data has earned us recognition by Forbes as one...

  • Data Engineer

    4 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Pathway Pełny etat

    About PathwayWe are a deeptech start-up founded in March 2020, dedicated to developing innovative data processing solutions. Our primary offering is an ultra-performant Data Processing Framework, featuring a Python API, distributed Rust engine, and capabilities for data source integration & transformation at scale. This framework is designed to handle...

  • Chief Data Architect

    2 dni temu

    Wrocław, Województwo dolnośląskie, Polska Limango Pełny etat

    Company OverviewLimango is a leading e-commerce company with 17 years of experience in the industry. We are part of the OTTO Group, one of Europe's largest e-commerce companies, operating across multiple platforms in several countries.Job DescriptionWe are seeking a Senior Data Engineer to join our dynamic team and play a key role in shaping our data...

  • Senior Data Engineer

    4 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Aptitude Software Pełny etat

    About Aptitude SoftwareWe are a leading provider of financial control and insights to empower our clients to achieve their strategies and ambitions.Our team of smart, motivated, and passionate people believe diverse experiences and perspectives build stronger teams and better solutions.Headquartered in London, we have seven office locations around the world...

  • Senior Data Architect for Cloud Data Platforms

    3 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska RITS Professional Services Pełny etat

    We are seeking an experienced Senior Data Architect to join our global team at RITS Professional Services, a leading provider of innovative data solutions to the financial industry. Our client, a global, American, market-leading, NYSE-listed trading platform, is looking for a highly skilled professional to design and implement a scalable cloud data...

  • Senior Data Engineer

    4 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Limango Pełny etat

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Limango. As a Senior Data Engineer, you will be responsible for developing and maintaining our Central Lakehouse Data Platform based on the AWS/Databricks ecosystem.Main ResponsibilitiesDesign and implement data infrastructure using Terraform and AWS/DatabricksDevelop and...

  • Senior Data Engineer

    4 tygodni temu

    Wrocław, Polska Link Group Pełny etat

    The Link Group is committed to becoming a data-driven organization, leveraging analytics to drive business decisions and customer value. To achieve this goal, we need a skilled Data Engineer to design and implement scalable data pipelines for machine learning models using GCP tools like Dataflow, Dataproc, and BigQuery.Key Responsibilities:Design and Develop...

  • Senior Data Engineer

    2 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Link Group Pełny etat

    The Link Group is seeking a skilled Data Engineer to join its team and contribute to the development of world-class analytics products and services. The ideal candidate will have expertise in designing and implementing scalable data pipelines using GCP tools like Dataflow, Dataproc, and BigQuery.Key Responsibilities:Data Pipeline Design and Development:...

  • Data Engineer

    3 tygodni temu

    Wrocław, Województwo dolnośląskie, Polska Ciklum Pełny etat

    About the RoleWe are seeking a skilled Data Engineer - Architectural Specialist to join our team in Poland.As a key member of our development team, you will be responsible for designing, developing, and implementing data processing solutions using a range of technologies, including Hadoop, MapReduce, Pig, Hive, Impala, Spark, Kafka, Storm, SQL, and NoSQL...