Lead Big Data Engineer
4 dni temu
The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to support strategic business decisions.
Responsibilities:
- Engineer reliable data pipelines for sourcing, processing, distributing, and storing data in different ways, using cloud (Azure) data platform infrastructure effectively.
- Transform data into valuable insights that inform business decisions, making use of our internal data platforms and applying appropriate analytical techniques.
- Develop, train, and apply data engineering techniques to automate manual processes, and solve challenging business problems.
- Ensure the quality, security, reliability, and compliance of our solutions by applying our digital principles and implementing both functional and non-functional requirements.
- Build observability into our solutions, monitor production health, help to resolve incidents, and remediate the root cause of risks and issues.
- Understand, represent, and advocate for client needs.
- Codify best practices, methodology and share knowledge with other engineers in UBS.
- Shape the Data and Distribution architecture and technology stack within our new cloud-based datalake-house, being a hands-on contributor, senior lead in the big data and data lake space, capable to collaborate and influence architectural and design principles across batch and real-time flows.
- Have a continuous improvement mindset, who is always on the lookout for ways to automate and reduce time to market for deliveries.
Mandatory Skills Description:
Your Expertise
- Experience in building Data Processing pipeline using various ETL/ELT design patterns and methodologies to Azure data solution, building solutions using ADLSv2, Azure Data factory, Databricks, Python and PySpark.
- Experience with at least one of the following technologies: Scala/Java or Python.
- Deep understanding of the software development craft, with focus on cloud based (Azure), event driven solutions and architectures, with key focus on Apache Spark batch and streaming, Datalakehouses using medallion architecture. Knowledge of DataMesh principles is an added plus.
- Ability to debug using tools Ganglia UI, expertise in Optimizing Spark Jobs.
- The ability to work across structured, semi-structured, and unstructured data, extracting information and identifying linkages across disparate datasets.
- Expert in creating data structures optimized for storage and various query patterns for e.g. Parquet and Delta Lake.
- Experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD) / ETL tools (Azure Data factory, Informatica).
- Experience in data modelling at least one database technology such as Traditional RDBMS (MS SQL Server, Oracle, PostgreSQL).
- Understanding of Information Security principles to ensure compliant handling and management of data.
Nice-to-Have Skills Description:
- Ability to clearly communicate complex solutions.
- Strong problem solving and analytical skills.
- Working experience in Agile methodologies (SCRUM).
Mid-Senior level
Employment typeFull-time
Job functionInformation Technology
IndustriesIT Services and IT Consulting
#J-18808-Ljbffr-
Big Data Engineer Lead
4 dni temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etatWe are looking for a highly skilled Big Data Engineer Lead to join our team at Luxoft. As a key member of our data engineering team, you will be responsible for leading the development of our modern cloud-based data processing platform.The ideal candidate will have extensive experience in building data processing pipelines using various ETL/ELT design...
-
Lead Big Data Engineer
1 dzień temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etatLead Big Data Engineer (with Azure) - Wroclaw or KrakowLocation: Wroclaw or Krakow - Hybrid work (3 days from office)Project Description:The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data...
-
Lead Big Data Engineer
4 dni temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etatAre you ready to become one of the Luxofters?Who we are:Luxoft DXC Technology Company is an established company focusing on consulting and implementation of complex projects in the financial industry. At the interface between technology and business, we convince with our know-how, well-founded methodology and pleasure in success. As a reliable partner to our...
-
Cloud-Based Data Engineer Lead
23 godzin temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etatAbout the RoleWe are seeking an experienced Lead Big Data Engineer to join our team. In this role, you will lead the development of cloud-based data processing platforms, enabling efficient management and real-time analysis of large datasets.Responsibilities:Data Pipeline Development: Engineer reliable data pipelines for sourcing, processing, distributing,...
-
Big Data Solutions Architect
23 godzin temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etatWe are seeking an experienced Big Data Solutions Architect to lead the design and implementation of innovative big data solutions using Azure. The successful candidate will be responsible for developing and maintaining large-scale data processing systems, ensuring high performance, scalability, and reliability.The ideal candidate will have hands-on...
-
Lead Big Data Engineer with Azure
1 dzień temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etatThe project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to...
-
Senior Big Data Solutions Architect
7 dni temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etatAbout the RoleThis is an exciting opportunity for a seasoned Senior Big Data Solutions Architect to lead the design and implementation of big data solutions using Azure technologies.You will work closely with cross-functional teams to develop cloud-based data processing platforms, ensuring high-quality, secure, and scalable solutions.Leading the design and...
-
Big Data Solution Developer
23 godzin temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etatAbout the JobThis is a mid-senior level position in our information technology department. The job involves working with Azure, Apache Spark, and Data Lakehouse architecture to support strategic business decisions.Responsibilities:Data Platform Development: Engineer reliable data pipelines for sourcing, processing, distributing, and storing data in different...
-
Big Data Engineering Manager
11 godzin temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etatAbout the RoleLuxoft Poland is looking for a highly skilled Big Data Engineering Manager to lead our data engineering team. As a manager, you will be responsible for ensuring the quality, security, reliability, and compliance of our data solutions.You should have experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD) /...
-
Linux Big Data Engineer Position
6 dni temu
Wrocław, Województwo dolnośląskie, Polska Correctcontext Pełny etatAre you passionate about leveraging cutting-edge technology to drive business growth? Do you have expertise in Scala and Linux development? We're seeking an experienced Scala/Linux Big Data Developer to join our team at Correct Context.In this role, you'll design, implement, and maintain complex big data pipelines using Scala, Kubernetes, and other...
-
Lead Data Processing Specialist
4 dni temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etatWe are seeking a highly skilled Lead Data Processing Specialist to join our data engineering team. As a Lead Data Processing Specialist, you will be responsible for designing and implementing modern data processing platforms using big data technologies.About the RoleStrong expertise in big data technologies such as Apache Spark, Hadoop, and NoSQL databases...
-
Big Data Infrastructure Manager
2 dni temu
Wrocław, Województwo dolnośląskie, Polska Experis ManpowerGroup Sp. z o.o. Pełny etatAbout the TeamWe are looking for a highly skilled Big Data Infrastructure Manager to join our team. In this role, you will be responsible for designing, developing, and deploying big data infrastructure solutions that meet the needs of our clients.Key Responsibilities:Design and develop scalable big data infrastructure solutions using Hadoop, Spark, and...
-
Azure Data Engineer Lead
23 godzin temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Pełny etatWe are seeking a seasoned Azure Data Engineer Lead to lead the design and implementation of innovative data engineering solutions using Azure. The successful candidate will be responsible for developing and maintaining large-scale data processing systems, ensuring high performance, scalability, and reliability.The ideal candidate will have hands-on...
-
Big Data Solutions Designer
2 dni temu
Wrocław, Województwo dolnośląskie, Polska DCV Technologies Pełny etatAn exciting opportunity has arisen for an experienced Enterprise Data Hub (EDH) Architect to join our team at DCV Technologies. As a key member of our data engineering team, you will play a critical role in the design and implementation of a centralized data platform that drives business growth and innovation.About the RoleThe EDH Architect will be...
-
Senior Big Data Solutions Architect
4 dni temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etatAs a Senior Big Data Solutions Architect at Luxoft Poland, you will play a crucial role in designing and developing innovative big data solutions for our clients. You will work closely with cross-functional teams to deliver high-quality solutions that meet the evolving needs of our customers.About the JobWe are looking for an experienced architect who can...
-
Big Data and Data Lake Architect
7 dni temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etatResponsibilitiesThe Big Data and Data Lake Architect will play a critical role in designing and implementing big data solutions using Azure technologies, ensuring high-quality, secure, and scalable data processing platforms.Your key responsibilities will include:Designing and implementing ETL/ELT pipelines using cloud infrastructure effectively.Developing...
-
Big Data Developer Scala Specialist
6 dni temu
Wrocław, Województwo dolnośląskie, Polska Correctcontext Pełny etatAt Correct Context, we're on a mission to revolutionize insights into consumer behavior, media consumption, and digital engagement. As a key member of our team, you'll have the opportunity to work with cutting-edge technology and contribute to empowering businesses in the digital era across various sectors.We're looking for a highly skilled Scala/Linux Big...
-
Scala/Linux Big Data Developer @
2 tygodni temu
Wrocław, Województwo dolnośląskie, Polska Comscore (via CC) Pełny etatCorrect Context is looking for a Scala/Linux Big Data Developer for Comscore in Poland and around. Comscore is a global leader in media analytics, revolutionizing insights into consumer behavior, media consumption, and digital engagement. Comscore leads in measuring and analyzing audiences across diverse digital platforms. Thrive on using cutting-edge...
-
Azure Big Data Architect
7 dni temu
Wrocław, Województwo dolnośląskie, Polska Xebia sp. z o.o. Pełny etatJob SummaryWe are seeking an experienced Azure Big Data Architect to join our team at Xebia sp. z o.o. This is an exciting opportunity for a talented individual with extensive experience in big data, cloud computing, and Azure technologies.About the RoleThe successful candidate will be responsible for leading the design, build, and deployment of large-scale...
-
Lead BigData Engineer
3 dni temu
Wrocław, Województwo dolnośląskie, Polska Luxoft Poland Pełny etatThe project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to...