Data engineer

3 miesięcy temu


Gdańsk, Polska b2bnetwork Pełny etat

Detailed description of work task to be carried out

 

We are seeking a skilled and motivated Data Engineer with extensive experience in Hadoop and HDFS as well as with deep understanding of relational databases to our team. In this role you will be responsible for designing, building and maintaining our data infrastructure to ensure efficient, reliable and scalable data processing and storage solutions.

You will collaborate with cross-functional teams to acquire, transform and optimize data from various sources.

The ideal candidate will have a deep understanding of big data technologies, scalable data pipelines and ensuring data quality and integrity.

 

Key Responsibilities:

  • design, develop and implement data solutions that acquire, transform and optimize data from diverse sources
  • design and build a scalable and optimize data pipelines to enable for data ingestion, transformation, and storage
  • ensure data quality and integrity by implementing data validation and cleansing processes
  • contribute to the development of data models and data architecture
  • collaborate with cross-functional teams to understand business requirements and translate them into technical solutions
  • manage and maintain HDFS for distributed storage of large datasets
  • perform data transformations and aggregations to support analytical and reporting requirements
  • identify and resolve issues related to data processing and storage

 

Tasks this person could cover:

  • Development related to provisioning data from Hadoop (either for historical FACP data provisioning or any other integration we’d like to do in the future using Hadoop as a source)
  • Ownership of data structures on ODS – design, trouble-shooting, executing changes on lower-environments (currently not centralized, different teams are responsible for creating different structures, sometimes not taking data modelling principles into consideration)
  • Data requirements and oversight on downstream integration with Hadoop and communication with GDM/Hadoop Ops

 

Description of knowledge and experience

Qualifications:

  • Bachelor’s degree in Computer Science, Information Technology or a related field. Master's degree preferred
  • Proven experience as a Data Engineer with a strong focus on Hadoop and HDFS
  • Proficiency in Hadoop ecosystem tools such as Hive, Pig, Flume, Oozie, and Sqoop
  • Experience with data processing frameworks like Apache Spark
  • Strong skills in relational databases with experience in SQL development
  • Knowledge of data warehousing concepts and database technologies
  • Excellent problem-solving skills and the ability to troubleshoot complex data issues

Req. number 36124


  • Data Engineer

    3 tygodni temu


    Gdańsk, Pomerania, Polska b2bnetwork Pełny etat

    Data Engineer Job DescriptionWe are seeking a skilled and motivated Data Engineer to join our team at b2bnetwork. In this role, you will be responsible for designing, building, and maintaining our data infrastructure to ensure efficient, reliable, and scalable data processing and storage solutions.Key Responsibilities:Design and develop data solutions that...

  • Data Engineer

    3 tygodni temu


    Gdańsk, Pomerania, Polska Acoustic Pełny etat

    About AcousticAcoustic is a leading marketing cloud that empowers businesses to succeed in a dynamic world. We're reimagining marketing technology by automating repetitive tasks and equipping marketers with intuitive tools that simplify complex processes.Job DescriptionWe're seeking a skilled Data Engineer to join our team. As a Data Engineer, you will play...

  • Data Engineer

    3 tygodni temu


    Gdańsk, Pomerania, Polska Epam Pełny etat

    About the RoleWe are seeking a talented Data Engineer to join our team at EPAM Data Competency Center. As a Data Engineer, you will play a key role in designing, building, and maintaining large-scale data systems and pipelines.ResponsibilitiesDesign and implement data pipelines and architectures to support business needsCollaborate with cross-functional...

  • Data Engineer

    3 tygodni temu


    Gdańsk, Polska RITS Professional Services Pełny etat

    Job Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at RITS Professional Services. As a Data Engineer, you will play a crucial role in designing, building, and maintaining our data platforms using cutting-edge technologies.Job Responsibilities:Design and implement data pipelines using public cloud infrastructure (AWS and...

  • Data Engineer

    1 dzień temu


    Gdańsk, Pomerania, Polska b2bnetwork Pełny etat

    About the RoleWe are seeking an experienced Data Engineer to join our team at b2bnetwork.Key ResponsibilitiesDevelop data visualization solutions using PowerBI.Analyze and process data from various databases using SQL.Design and implement tabular in-memory data models in SQL Server Analysis Services.Set up data connectors and refresh jobs in SQL Server...

  • Data Engineer

    12 godzin temu


    Gdańsk, Pomerania, Polska b2bnetwork Pełny etat

    Unlock New Opportunities with b2bnetwork as our Big Data Engineer!About the Role:We are seeking a skilled Big Data Engineer to join our dynamic team at b2bnetwork. As a key member of our data engineering team, you will be responsible for designing and developing highly scalable and efficient data pipelines using Spark and Python.Key Responsibilities:Design...

  • Data Engineer

    2 miesięcy temu


    Gdańsk, Polska BiLLWERK+ Pełny etat

    BiLLWERK+ is looking for a Data Engineer who wants to work in a high growth international environment with a world class team.BiLLWERK+ is a fast-growing company and having proven our ability to attract and retain customers we are now at the exciting phase of scaling up all aspects of the company as we look to expand into new product lines and enter new...

  • Senior Data Engineer

    3 tygodni temu


    centrum, Gdańsk, Polska Altimetrik Poland Pełny etat

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at Altimetrik Poland. As a Senior Data Engineer, you will play a crucial role in designing, developing, and maintaining our data pipelines and data warehouses.Responsibilities:Collaborate with cross-functional teams to understand business requirements and...

  • Cloud Data Engineer

    3 tygodni temu


    Gdańsk, Pomerania, Polska Kolomolo Pełny etat

    Cloud Data EngineerWe are seeking a skilled Cloud Data Engineer to craft and maintain intricate data pipelines within our state-of-the-art Snowflake data warehouse. As a key member of our dynamic team at DW & Analytics, you will empower your team of BI developers to generate insightful Power BI reports by seamlessly loading and extracting data from source to...

  • Cloud Data Engineer

    3 tygodni temu


    Gdańsk, Pomerania, Polska Kolomolo Pełny etat

    Cloud Data EngineerWe are seeking a skilled Cloud Data Engineer to craft and maintain intricate data pipelines within our state-of-the-art Snowflake data warehouse. As a key member of our dynamic team at DW & Analytics, you will empower your team of BI developers to generate insightful Power BI reports by seamlessly loading and extracting data from source to...

  • Senior Data Engineer

    3 tygodni temu


    centrum, Gdańsk, Polska Altimetrik Poland Pełny etat

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at Altimetrik Poland. As a Senior Data Engineer, you will play a key role in designing, developing, and maintaining our data pipelines and architectures.Responsibilities:Collaborate with cross-functional teams to understand business requirements and design...

  • Senior Data Engineer

    3 tygodni temu


    centrum, Gdańsk, Polska Altimetrik Poland Pełny etat

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at Altimetrik Poland. As a Senior Data Engineer, you will play a key role in designing, developing, and maintaining our data pipelines and architectures.Responsibilities:Collaborate with cross-functional teams to understand requirements and develop data...

  • Data Engineer

    2 tygodni temu


    Gdańsk, Pomerania, Polska Ciklum Pełny etat

    About the RoleCiklum is seeking a highly skilled Data Engineer to join our team in Poland. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems that support our clients' business needs.ResponsibilitiesDesign and implement data pipelines using Hadoop, MapReduce, Pig, Hive, Impala, Spark, Kafka, and...

  • Senior Data Engineer

    3 tygodni temu


    centrum, Gdańsk, Polska Altimetrik Poland Pełny etat

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at Altimetrik Poland. As a Senior Data Engineer, you will play a key role in designing, developing, and maintaining our data pipelines and architectures.Responsibilities:Collaborate with cross-functional teams to understand business requirements and develop...

  • Senior Data Engineer

    3 tygodni temu


    centrum, Gdańsk, Polska Altimetrik Poland Pełny etat

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at Altimetrik Poland. As a Senior Data Engineer, you will play a key role in designing, developing, and maintaining our data pipelines and data warehouses.Responsibilities:Collaborate with cross-functional teams to understand business requirements and develop...

  • Data Engineer

    1 miesiąc temu


    Gdańsk, Polska SCALO Sp. z o.o. Pełny etat

    W Scalo zajmujemy się dostarczaniem projektów software'owych i wspieraniem naszych partnerów w rozwijaniu ich biznesu. Tworzymy oprogramowanie, które umożliwia ludziom dokonywanie zmian, działanie w szybszym tempie oraz osiąganie lepszych rezultatów. Jesteśmy firmą, która wykorzystuje szerokie spektrum usług IT, żeby pomagać...

  • Data Engineer

    5 miesięcy temu


    Gdańsk, Polska Lyreco Group Pełny etat

    Responsible for building systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret. The data engineer aims to make data accessible so that the organization can use it to evaluate and optimize its performance Are you excited for your new career adventure? At Lyreco, we offer more than...

  • Data Engineer

    3 tygodni temu


    Gdańsk, Pomerania, Polska Transition Technologies MS Pełny etat

    We are looking for a highly skilled Data Engineer to join our team at Transition Technologies MS. As a Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data pipelines using cloud technologies such as Airflow, Glue, and Dataflow.Key Responsibilities:Design and develop data pipelines using cloud technologiesMaintain...


  • Gdańsk, Polska N-iX Pełny etat

    N-iX is seeking a Senior Data Platform Engineer to contribute to the development of a scalable and efficient data platform.About the project:The European online car market, with over 30 million monthly users across 18 countries, requires a data platform that can handle large volumes of data and provide real-time insights. As a Senior Data Platform Engineer,...

  • Middle Data Engineer

    3 tygodni temu


    Gdańsk, Pomerania, Polska N-iX Pełny etat

    Key Responsibilities:We are seeking a proactive Middle Data Engineer to join our vibrant team. In this role, you will take on a pivotal position, collaborating closely with senior data engineers to architect, develop, and maintain sophisticated data pipelines, Ontology Objects, and Foundry Functions within Palantir Foundry.Key Responsibilities:Design,...