Big Data Engineer, Warsaw
4 tygodni temu
Job Summary:
Our company is seeking experienced developers to contribute to the development and expansion of our data processing platform. The ideal candidate will have expertise in building and maintaining distributed systems that process large amounts of data in real-time.
Key Responsibilities:
- Design, develop, and maintain scalable and efficient distributed systems for real-time data processing
- Optimize software performance and resource utilization
- Ensure system reliability, scalability, and fault tolerance
- Develop performance and correctness tests for new system components
- Assess new technologies for their applicability in production environments
- Develop tools for monitoring and analyzing production system operation
- Continuously improve existing tools and processes
Technologies Used:
- Java and Python programming languages
- Hadoop and Kafka frameworks
- Docker, Jenkins, and Graphite for containerization and deployment
- Google Big Query and Elastic search engines
Recent Challenges:
- Migrating from Storm to Kafka Streams in the data processing component
- Creating a data stream merger using the Kafka Client API
- Implementing a user profile synchronizer between data centers using Kafka Streams
- Developing an aggregate calculator based on the Kafka Client API and Bloom filters
- Transitioning from Flume and Solr to Logstash and Elastic
- Establishing end-to-end monitoring for data correctness and delay
- Replacing the data streaming component with BigQuery and HDFS
Requirements:
- Proficiency in programming
- Deep understanding of complex IT systems (hardware, software, and algorithmics)
- Knowledge of concurrent programming and distributed systems methodologies
- Ability to analyze and optimize production system performance
- Readiness to work in a DevOps environment
Preferred Qualifications:
- Experience with Big Data technologies such as Hadoop, Kafka, Storm, Spark, or Flink
- Knowledge of application profiling methods and tools
We Offer:
- A collaborative team environment with knowledge-sharing opportunities
- Flexible work arrangements with remote work options
- Access to the latest technologies and the opportunity to apply them in a large-scale project
-
Big Data Scientist and Software Engineer
21 godzin temu
Warszawa, Mazovia, Polska Tietoevry Pełny etatTechnical RequirementsWe are seeking a highly skilled Senior Software Engineer-Machine Learning Engineer/Data Engineer to join our team in Warsaw. As a senior engineer, you will play a crucial role in designing, developing, and deploying machine learning models using Python frameworks.Develop and maintain ETL/ELT pipelines using Python and SQLWork with...
-
Big Data Engineer
14 godzin temu
Warszawa, Mazovia, Polska Capgemini Pełny etatCapgemini is seeking a talented Big Data Engineer to join our team. As a Big Data Engineer, you will design, implement, and maintain big data platforms, data architecture, and design techniques to work with large data sets. You will be empowered to shape your career in the way you'd like and supported by a collaborative community of colleagues around the...
-
Big Data Analyst and Engineer
2 dni temu
Warszawa, Mazovia, Polska OPTIVEUM sp. z o.o. Pełny etatAbout OPTIVEUM sp. z o.o.We are a forward-thinking SaaS provider driving innovation in the insurance technology space, renowned for transformative software solutions enhancing critical functions like policy management, claims, and reinsurance.Our team is growing, and we're looking for a talented Big Data Analyst and Engineer to join our dynamic...
-
Big Data Engineer Opportunity
3 dni temu
Warszawa, Mazovia, Polska SALESTUBE sp. z o.o. Pełny etatWe are looking for a highly skilled Big Data Engineer to join our team at SALESTUBE sp. z o.o. As a Big Data Engineer, you will be responsible for building and maintaining data warehouse and data lake architectures, implementing streaming and batch data processing solutions, and developing automated data processing solutions in Python.The ideal candidate...
-
Big Data/Hadoop Engineer
5 dni temu
Warszawa, Mazovia, Polska TN Poland Pełny etatCompany Overview:Visa's Technology Organization is a community of problem solvers and innovators reshaping the future of commerce.We operate the world's most sophisticated processing networks capable of handling massive scale problems centered on new payment flows, business and data solutions, cyber security, and B2C platforms.The OpportunityAs a Staff Site...
-
Big Data Engineer for AI Innovations
4 tygodni temu
Warszawa, Mazovia, Polska TN Poland Pełny etatJob Title: Big Data Engineer for AI InnovationsWe are seeking an experienced Big Data Engineer to join our team and contribute to the development of innovative AI solutions.About the Role:Design and develop scalable Big Data infrastructure to support the development of AI models.Create efficient processes for data ingestion, processing, and integration from...
-
Big Data Engineer Leader
4 dni temu
Warszawa, Mazovia, Polska ACAISOFT POLAND Sp. z o.o. Pełny etatWe're seeking a highly skilled Data Engineer to lead our big data initiatives and drive innovation in the field of artificial intelligence.About the JobThe ideal candidate will have a strong background in distributed data systems, with a focus on data mesh principles and real-time data processing.Key Responsibilities:Developing and implementing data...
-
DevOps Engineer for Big Data
7 dni temu
Warszawa, Mazovia, Polska Capco Poland Pełny etatJob DescriptionWe are seeking a seasoned Big Data Administrator/Engineer with 5+ years of experience in Hadoop administration and expertise in automation tools like Ansible, shell scripting, or Python scripting. The ideal candidate will have strong DevOps skills and proficiency in coding, particularly in Python. This dynamic role focuses on managing and...
-
Senior Big Data Engineer
6 dni temu
Warszawa, Mazovia, Polska Addepto Pełny etatAbout AddeptoWe are a leading consulting and technology company specializing in AI and Big Data. Our mission is to help clients deliver innovative data projects by leveraging cutting-edge technologies.Data Engineer RoleAs a Data Engineer, you will work on challenging projects across various industries, utilizing Azure and Databricks-powered solutions.You...
-
Big Data Developer
6 dni temu
Warszawa, Mazovia, Polska TN Poland Pełny etatJob SummaryWe are seeking a talented Big Data Developer to join our big data team in Warsaw. The successful candidate will have experience working with Oracle RDBMS, including tables, views, indexing, tuning, and data types.Develop high-performance solutions using best practices for the tools utilized.Complete thorough testing cycles before deploying...
-
Data Engineer with Big Data Expertise
14 godzin temu
Warszawa, Mazovia, Polska HireForYou Pełny etatAbout the JobWe are seeking an experienced Data Engineer to join our team at HireForYou, a leading programmatic media company. Our client is expanding rapidly, and we need someone with expertise in big data processing to help us develop scalable solutions.Job DescriptionThe successful candidate will be responsible for designing, developing, and maintaining...
-
Big Data Engineer – M/F/D
2 godzin temu
Warszawa, Mazovia, Polska SGS Société Générale de Surveillance SA Pełny etatResponsibilitiesJoin our data-centric team at SG Group's Corporate and Investment Banking (CIB) division, working at the heart of the information system. We are responsible for building the new data platform that supports various business processes and interacts with multiple stakeholders. Our team is dedicated to implementing data solutions using the latest...
-
Senior Big Data Engineer
5 dni temu
Warszawa, Mazovia, Polska TN Poland Pełny etatAbout the RoleWe are seeking a highly skilled Senior Big Data Engineer to lead the migration of data and applications from Microsoft Fabric to Databricks. The ideal candidate will have extensive experience with Databricks, PySpark, and SQL, with a proven track record in executing large-scale migrations.Key ResponsibilitiesMigrate data and applications from...
-
Data Engineer for Scalable Big Data Solutions
1 dzień temu
Warszawa, Mazovia, Polska Datumo Pełny etatAt Datumo, we specialize in providing cutting-edge Big Data and Cloud consulting services to clients worldwide. Our team consists of exceptional professionals who thrive in a dynamic environment.We're looking for a talented Data Engineer to join our team and contribute to the development of scalable Big Data solutions. As a Data Engineer, you'll work closely...
-
Big Data Technology Expert
5 dni temu
Warszawa, Mazovia, Polska Grid Dynamics International, Inc. Pełny etatWe are looking for an experienced Big Data Engineer to join our team at Grid Dynamics International, Inc.Job DescriptionIn this project, we are working on the bleeding edge of Big Data technology to develop a high-performance data analytics platform, which handles petabytes datasets.The ideal candidate will have solid programming skills in Python and...
-
Senior Big Data Solutions Architect
5 dni temu
Warszawa, Mazovia, Polska Grid Dynamics International, Inc. Pełny etatAs a Senior Big Data Engineer, you will design and develop Big Data solutions that meet our clients' needs.ResponsibilitiesDesign and Develop Big Data SolutionsData Ingestion and ProcessingReal-Time Data ProcessingData Quality and Governance
-
Big Data Technology Expert
5 dni temu
Warszawa, Mazovia, Polska Wipro Technologies Pełny etatIn today's fast-paced digital landscape, businesses are looking for ways to gain a competitive edge through data-driven decision-making. As a Senior Data Engineer at Wipro Technologies, you'll play a critical role in helping clients achieve this goal by designing and implementing cutting-edge big data solutions.About the RoleWe're seeking an experienced...
-
Big Data Engineer Specialist
2 dni temu
Warszawa, Mazovia, Polska Cyclad Pełny etatCyclad works with top international IT companies to boost their potential in delivering outstanding technologies. We are seeking a Senior Data Engineer to collaborate with project teams responsible for developing advanced data lakehouse, business intelligence, and advanced analytics solutions.Job Description:Designing new solutions for data...
-
Data Engineer @
1 tydzień temu
Warszawa, Mazovia, Polska Quontex Ltd Pełny etatOur partner is a leading programmatic media company, specializing in ingesting large volumes of data, modeling insights, and offering a range of products and services across Media, Analytics, and Technology. Among their clients are well-known brands such as Walmart, Barclaycard, and Ford. The company has expanded to over 700 employees, with 15 global offices...
-
Data Engineer @
3 tygodni temu
Warszawa, Mazovia, Polska Quontex Ltd Pełny etatOur partner is a leading programmatic media company, specializing in ingesting large volumes of data, modeling insights, and offering a range of products and services across Media, Analytics, and Technology. Among their clients are well-known brands such as Walmart, Barclaycard, and Ford.The company has expanded to over 700 employees, with 15 global offices...