Lead BigData Lead Engineer

5 dni temu


Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to support strategic business decisions.

Responsibilities:

  1. Engineer reliable data pipelines for sourcing, processing, distributing, and storing data in different ways, using cloud (Azure) data platform infrastructure effectively.
  2. Transform data into valuable insights that inform business decisions, making use of our internal data platforms and applying appropriate analytical techniques.
  3. Develop, train, and apply data engineering techniques to automate manual processes, and solve challenging business problems.
  4. Ensure the quality, security, reliability, and compliance of our solutions by applying our digital principles and implementing both functional and non-functional requirements.
  5. Build observability into our solutions, monitor production health, help to resolve incidents, and remediate the root cause of risks and issues.
  6. Understand, represent, and advocate for client needs.
  7. Codify best practices, methodology and share knowledge with other engineers in UBS.
  8. Shape the Data and Distribution architecture and technology stack within our new cloud-based datalake-house, being a hands-on contributor and senior lead in the big data and data lake space, capable to collaborate and influence architectural and design principles across batch and real-time flows.
  9. Have a continuous improvement mindset, always looking for ways to automate and reduce time to market for deliveries.

Mandatory Skills Description:

Your Expertise

  1. Experience in building Data Processing pipeline using various ETL/ELT design patterns and methodologies to Azure data solution, building solutions using ADLSv2, Azure Data factory, Databricks, Python and PySpark.
  2. Experience with at least one of the following technologies: Scala/Java or Python.
  3. Deep understanding of the software development craft, focusing on cloud-based (Azure), event-driven solutions and architectures, with key focus on Apache Spark batch and streaming, Datalakehouses using medallion architecture. Knowledge of DataMesh principles is an added plus.
  4. Ability to debug using tools like Ganglia UI, expertise in Optimizing Spark Jobs.
  5. The ability to work across structured, semi-structured, and unstructured data, extracting information and identifying linkages across disparate datasets.
  6. Expert in creating data structures optimized for storage and various query patterns for e.g. Parquet and Delta Lake.
  7. Experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD) / ETL tools (Azure Data factory, Informatica).
  8. Experience in data modelling at least one database technology such as: Traditional RDBMS (MS SQL Server, Oracle, PostgreSQL).
  9. Understanding of Information Security principles to ensure compliant handling and management of data.

Nice-to-Have Skills Description:

  1. Ability to clearly communicate complex solutions.
  2. Strong problem-solving and analytical skills.
  3. Working experience in Agile methodologies (SCRUM).

Languages:

What do we offer our employees?

  1. Tax-Deductible Costs on a contract of employment for all development roles.
  2. Stable employment based on an employment contract (or B2B).
  3. Special offer of banking services.
  4. Equipment such as a laptop and monitor.
  5. LuxTalent platform (webinars, training, courses with certificates).
  6. ...and even more
Seniority level
  • Associate
Employment type
  • Full-time
Job function
  • Information Technology
  • Industries: IT Services and IT Consulting
#J-18808-Ljbffr

  • Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

    The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to...

  • Data Engineer Lead

    4 dni temu


    Wrocław, Województwo dolnośląskie, Polska Addepto Pełny etat

    We are looking for a highly skilled Data Engineer Lead to join our team at Addepto. Our company specializes in AI and Big Data, helping clients deliver innovative data projects. As a Data Engineer Lead, you will have the opportunity to work with a team of technology experts on challenging projects across various industries, leveraging cutting-edge...


  • Wrocław, Województwo dolnośląskie, Polska Cyclad Pełny etat

    At Cyclad, we're looking for an experienced Lead Java Software Engineer to lead our Java team. As a Lead Java Software Engineer, you will oversee the development and maintenance of our critical global component within the customer ecosystem.Key responsibilities include:Leading and mentoring junior engineers to develop high-quality Java-based solutions using...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    We are seeking a seasoned Azure Data Engineer Lead to lead the design and implementation of innovative data engineering solutions using Azure. The successful candidate will be responsible for developing and maintaining large-scale data processing systems, ensuring high performance, scalability, and reliability.The ideal candidate will have hands-on...


  • Wrocław, Województwo dolnośląskie, Polska N-iX Pełny etat

    We are looking for a Lead Full Stack Engineer comfortable with C# and Angular, with a proven history of building high-performance apps. This is a great opportunity for an experienced Lead .NET Full Stack Engineer to join our super-motivated Software Development team, being part of one of our existing Scrum teams. Each Agile team includes technical...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    About the RoleWe are seeking an experienced Lead Big Data Engineer to join our team. In this role, you will lead the development of cloud-based data processing platforms, enabling efficient management and real-time analysis of large datasets.Responsibilities:Data Pipeline Development: Engineer reliable data pipelines for sourcing, processing, distributing,...


  • Wrocław, Województwo dolnośląskie, Polska JobFinder Spain Pełny etat

    Job SummaryAs a Lead Software Engineer (Backend), you will be responsible for leading the development of complex backend systems using Ruby on Rails and React. You will collaborate with cross-functional teams to design and implement features that meet business requirements.Key Responsibilities:Design and develop scalable backend systems using Ruby on Rails...


  • Wrocław, Województwo dolnośląskie, Polska N-iX Pełny etat

    Our Client is a global investment advisor organization that utilizes advanced technologies and data analytics to improve the operational performance of their portfolio companies. Key objectives include delivering strong financial returns, offering diverse investment opportunities, enhancing portfolio company operations, and proactively managing risks. They...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

    Job RequirementsWe are seeking an experienced Lead Cloud Data Engineer to join our team at Luxoft Poland. In this role, you will be responsible for designing and implementing data processing pipelines using various ETL/ELT design patterns and methodologies to Azure data solution.Your expertise should include experience in building solutions using ADLSv2,...


  • Wrocław, Województwo dolnośląskie, Polska TN Poland Pełny etat

    TN Poland is a leading player in the digital transformation of the logistics industry. We are seeking an experienced AWS Cloud Engineer Lead to join our Cloud Competence Centre as a key member of our public cloud services team.ResponsibilitiesThe successful candidate will be responsible for designing, building, and managing cloud landing zone services,...


  • Wrocław, Województwo dolnośląskie, Polska Softjourn, Inc Pełny etat

    About the Role:As a seasoned Software Engineer, you will be responsible for architecting, designing, and implementing robust full-stack applications using React on the frontend and Node.js on the backend. You will lead the technical implementation of complex features, collaborating with other team members to break down tasks into manageable pieces.About the...


  • Wrocław, Województwo dolnośląskie, Polska TN Poland Pełny etat

    **Job Title:** Lead Full-Stack Engineer – SmartBear Platform ServicesKey Responsibilities:Drive the future of the SmartBear Platform Services.Collaborate with cross-functional teams.Engage in dynamic technical discussions.Benefits of Working at SmartBear


  • Wrocław, Województwo dolnośląskie, Polska SmartBear Software Pełny etat

    At SmartBear, we deliver the complete visibility developers need to make each release better than the last. Our award-winning and industry favorite tools TestComplete, Swagger, Cucumber, ReadyAPI, and Zephyr are trusted by over 16 million developers, testers, and software engineers at 32,000+ organizations – including world-renowned innovators like Adobe,...

  • SAP LE

    4 tygodni temu


    Wrocław, Województwo dolnośląskie, Polska Michael Page Pełny etat

    technologies-expected : SAP responsibilities : Lead the SAP LE / WM / eWM modules implementation across the organisation. Provide expert business process capability and SAP product knowledge. Collaborate with stakeholders to understand and document business requirements. Design and implement SAP solutions that meet business needs and deliver value. Conduct...

  • Lead Power BI

    4 dni temu


    Wrocław, Województwo dolnośląskie, Polska GlobalLogic Pełny etat

    Lead Power BI (Business Intelligence Lead) IRC254808Job: IRC254808Location: Poland - Gdansk, Krakow, Warsaw, WroclawDesignation: Lead Software EngineerExperience: 5-10 yearsFunction: EngineeringSkills: Power BIWork Model: RemoteClient is a leading global Contract Research Organization (CRO) that provides Phase I-IV clinical trial management, market access,...


  • Wrocław, Województwo dolnośląskie, Polska Correctcontext Pełny etat

    Creative Context is a global leader in media analytics, delivering accurate data to businesses across industries. As a Snowflake Lead Engineer, you'll play a key role in driving business outcomes by designing and implementing scalable data pipelines using Snowflake.We're seeking an experienced professional with expertise in Snowflake, Python, and Java to...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    Lead Big Data Engineer (with Azure) - Wroclaw or KrakowLocation: Wroclaw or Krakow - Hybrid work (3 days from office)Project Description:The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data...

  • SAP LE

    2 dni temu


    Wrocław, Województwo dolnośląskie, Polska Michael Page Pełny etat

    Impactful Role in a Large OrganizationAttractive Compensation & Career GrowthAbout Our ClientOur client is a respected player in the industrial / manufacturing industry. This organisation is known for its rigorous standards, commitment to innovation, and a workforce that is dedicated to achieving the company's vision.Job DescriptionLead the SAP LE / WM / eWM...

  • Lead Big Data Engineer

    1 tydzień temu


    Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    Are you ready to become one of the Luxofters?Who we are:Luxoft DXC Technology Company is an established company focusing on consulting and implementation of complex projects in the financial industry. At the interface between technology and business, we convince with our know-how, well-founded methodology and pleasure in success. As a reliable partner to our...


  • Wrocław, Województwo dolnośląskie, Polska Infor Pełny etat

    About the RoleWe are seeking a skilled Software Engineer to join our team as an ERP Application Development Lead. As a key member of our team, you will be responsible for designing, developing, and documenting Infor COM application business logic and framework changes. You will work closely with Technical and Functional software engineers, Quality Assurance,...