Lead BigData Lead Engineer

2 dni temu

Kraków, Lesser Poland Luxoft Poland Pełny etat

The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to support strategic business decisions.

Responsibilities:

  1. Engineer reliable data pipelines for sourcing, processing, distributing, and storing data in different ways, using cloud (Azure) data platform infrastructure effectively.
  2. Transform data into valuable insights that inform business decisions, making use of our internal data platforms and applying appropriate analytical techniques.
  3. Develop, train, and apply data engineering techniques to automate manual processes, and solve challenging business problems.
  4. Ensure the quality, security, reliability, and compliance of our solutions by applying our digital principles and implementing both functional and non-functional requirements.
  5. Build observability into our solutions, monitor production health, help to resolve incidents, and remediate the root cause of risks and issues.
  6. Understand, represent, and advocate for client needs.
  7. Codify best practices, methodology and share knowledge with other engineers in UBS.
  8. Shape the Data and Distribution architecture and technology stack within our new cloud-based datalake-house, being a hands-on contributor, senior lead in the big data and data lake space, capable to collaborate and influence architectural and design principles across batch and real-time flows.
  9. Have a continuous improvement mindset, who is always on the lookout for ways to automate and reduce time to market for deliveries.

Mandatory Skills Description:

Your Expertise

  1. Experience in building Data Processing pipeline using various ETL/ELT design patterns and methodologies to Azure data solution, building solutions using ADLSv2, Azure Data Factory, Databricks, Python, and PySpark.
  2. Experience with at least one of the following technologies: Scala/Java or Python.
  3. Deep understanding of the software development craft, with a focus on cloud-based (Azure), event-driven solutions and architectures, with key focus on Apache Spark batch and streaming, Datalakehouses using medallion architecture. Knowledge of DataMesh principles is an added plus.
  4. Ability to debug using tools like Ganglia UI, expertise in optimizing Spark Jobs.
  5. The ability to work across structured, semi-structured, and unstructured data, extracting information and identifying linkages across disparate datasets.
  6. Expert in creating data structures optimized for storage and various query patterns for e.g. Parquet and Delta Lake.
  7. Experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD) / ETL tools (Azure Data Factory, Informatica).
  8. Experience in data modeling at least one database technology such as:
  • Traditional RDBMS (MS SQL Server, Oracle, PostgreSQL).
Understanding of Information Security principles to ensure compliant handling and management of data.

Nice-to-Have Skills Description:

  1. Ability to clearly communicate complex solutions.
  2. Strong problem-solving and analytical skills.
  3. Working experience in Agile methodologies (SCRUM).

Languages:

What do we offer our employees?

  1. Tax-Deductible Costs on a contract of employment for all development roles.
  2. Stable employment based on an employment contract (or B2B).
  3. Special offer of banking services.
  4. Equipment such as a laptop and monitor.
  5. LuxTalent platform (webinars, training, courses with certificates).
  6. ...and even more
Seniority level

Associate

Employment type

Full-time

Job function

Information Technology

Industries

IT Services and IT Consulting

#J-18808-Ljbffr

  • Lead Engineer

    4 tygodni temu

    Kraków, Lesser Poland TN Poland Pełny etat

    Lead Engineer (Production Engineering) Position OverviewWe are committed to driving real change in the energy industry. Our ambition is to build a sustainable future through relentless innovation and global collaboration.You will join a culture that values curiosity, expertise, and ideas as well as diversity, inclusion, and authenticity. Bring your unique...

  • Subsystem Lead Engineer

    4 tygodni temu

    Kraków, Lesser Poland TN Poland Pełny etat

    Subsystem Lead EngineerJob Title: Subsystem Lead Engineer, Location: Onshore, Employment Type: Employee, Place of work: HybridWe are seeking a highly skilled and experienced Subsystem Lead Engineer to join our team at TechnipFMC. As a global leader in the energy industry, we value innovation, collaboration, and diversity.Job PurposeThe Subsystem Lead...

  • Technical Lead Engineer

    2 godzin temu

    Kraków, Lesser Poland TN Poland Pełny etat

    About the RoleThe Product Lead Engineer will be part of a team responsible for delivering high-quality engineering solutions. This individual will lead the development of technical-technological solutions and ensure their implementation aligns with company standards.Responsibilities Include:Develop and implement engineering solutions.Lead a team of engineers...

  • Electrical Lead Engineer

    1 tydzień temu

    Kraków, Lesser Poland HAYS POLAND Sp. z o.o. Pełny etat

    Electrical Lead Engineerlokalizacja: Kraków (małopolskie)numer referencyjny: 1189337/jobs.plDue to the request of our Client, an extended company from the petrochemical industry with an established position on the market, we are looking for people in the position of Electrical Lead Engineer.- Design electrical systems for petrochemical and refinery...

  • Backend Engineer Lead

    6 dni temu

    Kraków, Lesser Poland TN Poland Pełny etat

    The position of Backend Engineer Lead at TN Poland requires a highly skilled professional with experience in designing, developing, and securing the backend infrastructure to support millions of concurrent API calls.Job Summary:We are looking for a seasoned system architect to lead our Server Engineering team and collaborate with mobile operators to build a...

  • Lead Android Software Engineer

    2 dni temu

    Kraków, Lesser Poland Tbwa ChiatDay Inc Pełny etat

    We are looking for an experienced Lead Android Software Engineer to join our team at SpotOn. The successful candidate will have a strong background in Android development and will be able to lead the design and implementation of new features.About the Job:You will be part of a strong Android team working in Scrum methodology.Key Responsibilities:Lead the...

  • Electrical Lead Engineer

    5 godzin temu

    Kraków, Lesser Poland NES Fircroft Pełny etat

    We are seeking an experienced Electrical Lead Engineer to contribute to cutting-edge Electrolysis technology and key Oil & Gas projects worldwide.Job SummaryThe successful candidate will play a key role in designing High Voltage (HV) stations and HV electrical systems for major EPC Electrolyzer projects across Europe, as well as supporting global initiatives...

  • Lead Mechanical Development Engineer

    6 dni temu

    Kraków, Lesser Poland TN Poland Pełny etat

    About the JobWe are looking for a Lead Mechanical Development Engineer to join our team at TN Poland. The successful candidate will have a strong background in mechanical engineering, with experience in designing and developing eDrive systems.You will be responsible for leading a team of mechanical engineers and collaborating with cross-functional teams to...

  • Lead Java Engineer @

    3 tygodni temu

    Kraków, Lesser Poland N-iX Pełny etat

    N-iX is looking for a Lead Java Engineer to join our team.One of the world's largest fashion retailers selling in 215 markets through its online platform or its 6,477 stores in 95 markets. By working closely together as a single company globally focused on the key elements of fashion production – design, manufacture, distribution, and retail.All the...

  • Electrical Lead Engineer

    7 godzin temu

    Kraków, Lesser Poland NES Fircroft Pełny etat

    We are seeking an experienced Electrical Lead Engineer to join our Clients team, contributing to cutting-edge Electrolysis technology and key Oil & Gas projects worldwide. You will play a key role in designing High Voltage (HV) stations and HV electrical systems for major EPC Electrolyzer projects across Europe, as well as supporting global initiatives like...

  • Lead MLOps Engineer @

    3 tygodni temu

    Kraków, Lesser Poland Mindbox S.A. Pełny etat

    Obecnie na potrzeby mojego brytyjskiego Klienta z branży bankowej poszukuję Lead MLOps Engineer, który będzie będzie ściśle współpracować z zespołem Data Scientistów. Głównym zadanie będzie budowanie wydajnych i skalowalnych pipelineów uczenia maszynowego na platformie Google Cloud Platform, z jednoczesnym zapewnieniem integracji ze...

  • Senior Data Engineer Lead

    3 dni temu

    Kraków, Lesser Poland Autodesk, Inc. Pełny etat

    Job SummaryAutodesk, Inc. is seeking a seasoned Principal Data Engineer/Prompt Engineer to lead the development of our data engineering capabilities. The successful candidate will be responsible for designing, developing, and implementing scalable data pipelines and architectures to support business growth and innovation.Main ResponsibilitiesDevelop and...

  • Lead UX Engineer

    4 tygodni temu

    Kraków, Lesser Poland TN Poland Pełny etat

    Job DescriptionWe are seeking a Lead UX Engineer to build and maintain reusable React components for our design system.Key Responsibilities:Develop and maintain React components with a focus on performance, accessibility, and scalability.Implement and manage design tokens to maintain consistent theming and styling across platforms.Create and maintain CI/CD...

  • Lead Engineer, Kraków

    4 tygodni temu

    Kraków, Lesser Poland TN Poland Pełny etat

    Lead EngineerA highly motivated engineering professional is sought to join our Controls & Automation Department in Krakow Engineering Center as a Lead Engineer.About the JobThis is an exciting opportunity for someone with strong technical expertise and leadership skills to work on deliveries of subsea control units and auxiliaries in close cooperation with...

  • Software Engineer Lead

    6 dni temu

    Kraków, Lesser Poland Hitachi Automotive Systems Americas, Inc. Pełny etat

    Role OverviewHitachi Energy seeks a Senior Software Engineer to lead the development, implementation, and maintenance of its Energy Market Management system platforms and environments.This position involves working closely with the team to deliver complex system solutions and contributing to the growth and success of the company.Analyzing and understanding...

  • Embedded Software Engineer Technical Lead., Kraków

    4 tygodni temu

    Kraków, Lesser Poland TN Poland Pełny etat

    Technical Lead Software EngineerCisco is seeking a skilled engineer to join our Distributed System Engineering team. As a Technical Lead, you will be responsible for developing features for routers that accelerate customer success.Key Responsibilities:Collaborate with multi-functional teams to develop and deploy container-based infrastructure and...

  • Lead Web Application Engineer

    2 dni temu

    Kraków, Lesser Poland GlobalLogic Pełny etat

    About the Position: As a Lead Web Application Engineer at GlobalLogic, you will lead the development of high-quality web applications that drive business growth and customer satisfaction. We are seeking an experienced engineer to join our team and help us deliver innovative solutions.Key Responsibilities:Develop new user-facing features, optimize the...

  • Lead ML Engineer, Kraków

    4 tygodni temu

    Kraków, Lesser Poland TN Poland Pełny etat

    Job Title: Lead Machine Learning EngineerWe are seeking a highly skilled and experienced Lead Machine Learning Engineer to join our team. The successful candidate will have strong software engineering skills and a creative mindset for ML-focused problem-solving.Key Responsibilities:Lead large-scale ML projects and products from inception to production,...

  • Product Lead Engineer

    3 godzin temu

    Kraków, Lesser Poland TN Poland Pełny etat

    Product Lead EngineerLocation: Houston, TX, USEmployment type: EmployeePlace of work: HybridOffshore/Onshore: OnshoreTechnipFMC is committed to driving real change in the energy industry. Our ambition is to build a sustainable future through relentless innovation and global collaboration – and we want you to be part of it. You'll be joining a culture that...

  • Lead Software Engineer

    7 dni temu

    Kraków, Lesser Poland DCG Poland Pełny etat

    DCG Poland is a recruitment company dedicated to meeting the needs of its partners through continuous support and a management style that fosters growth and success.Job RequirementsWe are currently looking for a Lead Software Engineer to join our team. The successful candidate will be responsible for leading the design and development of application...