Lead Big Data Engineer

4 dni temu


Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to support strategic business decisions.

Responsibilities:

  1. Engineer reliable data pipelines for sourcing, processing, distributing, and storing data in different ways, using cloud (Azure) data platform infrastructure effectively.
  2. Transform data into valuable insights that inform business decisions, making use of our internal data platforms and applying appropriate analytical techniques.
  3. Develop, train, and apply data engineering techniques to automate manual processes, and solve challenging business problems.
  4. Ensure the quality, security, reliability, and compliance of our solutions by applying our digital principles and implementing both functional and non-functional requirements.
  5. Build observability into our solutions, monitor production health, help to resolve incidents, and remediate the root cause of risks and issues.
  6. Understand, represent, and advocate for client needs.
  7. Codify best practices, methodology and share knowledge with other engineers in UBS.
  8. Shape the Data and Distribution architecture and technology stack within our new cloud-based datalake-house, being a hands-on contributor, senior lead in the big data and data lake space, capable to collaborate and influence architectural and design principles across batch and real-time flows.
  9. Have a continuous improvement mindset, who is always on the lookout for ways to automate and reduce time to market for deliveries.

Mandatory Skills Description:

Your Expertise

  1. Experience in building Data Processing pipeline using various ETL/ELT design patterns and methodologies to Azure data solution, building solutions using ADLSv2, Azure Data factory, Databricks, Python and PySpark.
  2. Experience with at least one of the following technologies: Scala/Java or Python.
  3. Deep understanding of the software development craft, with focus on cloud based (Azure), event driven solutions and architectures, with key focus on Apache Spark batch and streaming, Datalakehouses using medallion architecture. Knowledge of DataMesh principles is an added plus.
  4. Ability to debug using tools Ganglia UI, expertise in Optimizing Spark Jobs.
  5. The ability to work across structured, semi-structured, and unstructured data, extracting information and identifying linkages across disparate datasets.
  6. Expert in creating data structures optimized for storage and various query patterns for e.g. Parquet and Delta Lake.
  7. Experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD) / ETL tools (Azure Data factory, Informatica).
  8. Experience in data modelling at least one database technology such as Traditional RDBMS (MS SQL Server, Oracle, PostgreSQL).
  9. Understanding of Information Security principles to ensure compliant handling and management of data.

Nice-to-Have Skills Description:

  1. Ability to clearly communicate complex solutions.
  2. Strong problem solving and analytical skills.
  3. Working experience in Agile methodologies (SCRUM).
Seniority level

Mid-Senior level

Employment type

Full-time

Job function

Information Technology

Industries

IT Services and IT Consulting

#J-18808-Ljbffr

  • Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    We are looking for a highly skilled Big Data Engineer Lead to join our team at Luxoft. As a key member of our data engineering team, you will be responsible for leading the development of our modern cloud-based data processing platform.The ideal candidate will have extensive experience in building data processing pipelines using various ETL/ELT design...

  • Lead Big Data Engineer

    1 dzień temu


    Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    Lead Big Data Engineer (with Azure) - Wroclaw or KrakowLocation: Wroclaw or Krakow - Hybrid work (3 days from office)Project Description:The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    Are you ready to become one of the Luxofters?Who we are:Luxoft DXC Technology Company is an established company focusing on consulting and implementation of complex projects in the financial industry. At the interface between technology and business, we convince with our know-how, well-founded methodology and pleasure in success. As a reliable partner to our...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    About the RoleWe are seeking an experienced Lead Big Data Engineer to join our team. In this role, you will lead the development of cloud-based data processing platforms, enabling efficient management and real-time analysis of large datasets.Responsibilities:Data Pipeline Development: Engineer reliable data pipelines for sourcing, processing, distributing,...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    We are seeking an experienced Big Data Solutions Architect to lead the design and implementation of innovative big data solutions using Azure. The successful candidate will be responsible for developing and maintaining large-scale data processing systems, ensuring high performance, scalability, and reliability.The ideal candidate will have hands-on...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

    About the RoleThis is an exciting opportunity for a seasoned Senior Big Data Solutions Architect to lead the design and implementation of big data solutions using Azure technologies.You will work closely with cross-functional teams to develop cloud-based data processing platforms, ensuring high-quality, secure, and scalable solutions.Leading the design and...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    About the JobThis is a mid-senior level position in our information technology department. The job involves working with Azure, Apache Spark, and Data Lakehouse architecture to support strategic business decisions.Responsibilities:Data Platform Development: Engineer reliable data pipelines for sourcing, processing, distributing, and storing data in different...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

    About the RoleLuxoft Poland is looking for a highly skilled Big Data Engineering Manager to lead our data engineering team. As a manager, you will be responsible for ensuring the quality, security, reliability, and compliance of our data solutions.You should have experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD) /...


  • Wrocław, Województwo dolnośląskie, Polska Correctcontext Pełny etat

    Are you passionate about leveraging cutting-edge technology to drive business growth? Do you have expertise in Scala and Linux development? We're seeking an experienced Scala/Linux Big Data Developer to join our team at Correct Context.In this role, you'll design, implement, and maintain complex big data pipelines using Scala, Kubernetes, and other...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

    We are seeking a highly skilled Lead Data Processing Specialist to join our data engineering team. As a Lead Data Processing Specialist, you will be responsible for designing and implementing modern data processing platforms using big data technologies.About the RoleStrong expertise in big data technologies such as Apache Spark, Hadoop, and NoSQL databases...


  • Wrocław, Województwo dolnośląskie, Polska Experis ManpowerGroup Sp. z o.o. Pełny etat

    About the TeamWe are looking for a highly skilled Big Data Infrastructure Manager to join our team. In this role, you will be responsible for designing, developing, and deploying big data infrastructure solutions that meet the needs of our clients.Key Responsibilities:Design and develop scalable big data infrastructure solutions using Hadoop, Spark, and...

  • Azure Data Engineer Lead

    23 godzin temu


    Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etat

    We are seeking a seasoned Azure Data Engineer Lead to lead the design and implementation of innovative data engineering solutions using Azure. The successful candidate will be responsible for developing and maintaining large-scale data processing systems, ensuring high performance, scalability, and reliability.The ideal candidate will have hands-on...


  • Wrocław, Województwo dolnośląskie, Polska DCV Technologies Pełny etat

    An exciting opportunity has arisen for an experienced Enterprise Data Hub (EDH) Architect to join our team at DCV Technologies. As a key member of our data engineering team, you will play a critical role in the design and implementation of a centralized data platform that drives business growth and innovation.About the RoleThe EDH Architect will be...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

    As a Senior Big Data Solutions Architect at Luxoft Poland, you will play a crucial role in designing and developing innovative big data solutions for our clients. You will work closely with cross-functional teams to deliver high-quality solutions that meet the evolving needs of our customers.About the JobWe are looking for an experienced architect who can...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

    ResponsibilitiesThe Big Data and Data Lake Architect will play a critical role in designing and implementing big data solutions using Azure technologies, ensuring high-quality, secure, and scalable data processing platforms.Your key responsibilities will include:Designing and implementing ETL/ELT pipelines using cloud infrastructure effectively.Developing...


  • Wrocław, Województwo dolnośląskie, Polska Correctcontext Pełny etat

    At Correct Context, we're on a mission to revolutionize insights into consumer behavior, media consumption, and digital engagement. As a key member of our team, you'll have the opportunity to work with cutting-edge technology and contribute to empowering businesses in the digital era across various sectors.We're looking for a highly skilled Scala/Linux Big...


  • Wrocław, Województwo dolnośląskie, Polska Comscore (via CC) Pełny etat

    Correct Context is looking for a  Scala/Linux Big Data Developer for Comscore in Poland and around. Comscore is a global leader in media analytics, revolutionizing insights into consumer behavior, media consumption, and digital engagement. Comscore leads in measuring and analyzing audiences across diverse digital platforms. Thrive on using cutting-edge...


  • Wrocław, Województwo dolnośląskie, Polska Xebia sp. z o.o. Pełny etat

    Job SummaryWe are seeking an experienced Azure Big Data Architect to join our team at Xebia sp. z o.o. This is an exciting opportunity for a talented individual with extensive experience in big data, cloud computing, and Azure technologies.About the RoleThe successful candidate will be responsible for leading the design, build, and deployment of large-scale...


  • Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etat

    The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to...