Big Data Ecosystem Manager

1 tydzień temu

Warszawa, Polska Kevin Edward Pełny etat

About Kevin Edward

We are a technology-driven organization seeking an experienced Big Data Ecosystem Manager to lead our big data infrastructure. If you have expertise in Hadoop, Spark, and Kafka, we want to hear from you.

Estimated Salary: $120,000 - $180,000 per year

Job Description:

Hadoop, Spark, and Kafka Administration
  • Manage and support the deployment, configuration, and scaling of Hadoop, Spark, and Kafka clusters in production environments.
  • Administer distributed storage systems and ensure that Hadoop clusters are highly available and perform optimally.
  • Configure, monitor, and troubleshoot issues related to Apache Spark jobs and Kafka streams.
  • Perform regular maintenance of big data platforms and associated components to ensure data integrity and availability.
Deployment & Patching
  • Lead the deployment of new services and solutions within the big data infrastructure.
  • Ensure the timely installation of patches and updates across hosts and systems to maintain security and performance.
  • Develop and automate deployment pipelines for big data services using tools like Ansible, Terraform, or Jenkins.
Troubleshooting & Performance Tuning
  • Troubleshoot and resolve issues related to the big data infrastructure, including performance bottlenecks, data loss, system outages, and other operational incidents.
  • Monitor the health of Hadoop, Spark, and Kafka systems using appropriate tools (e.g., Cloudera Manager, Grafana, etc.) and take corrective action as needed.
  • Conduct performance tuning for data workflows, ensuring efficient and reliable data processing in the environment.
Monitoring & Maintenance
  • Set up and configure monitoring and alerting tools to track system performance, data quality, and usage metrics.
  • Proactively monitor system logs, disk usage, network throughput, and other resources to avoid system failures.
  • Perform regular backup and recovery tasks to ensure the integrity and availability of data stored in Hadoop, Spark, and Kafka.
Capacity Planning & Scaling
  • Plan and manage the capacity and growth of the big data ecosystem, including scaling clusters as data volume increases.
  • Ensure that all systems are optimized for high-performance data processing and can handle large-scale, high-volume workloads.
Security & Compliance
  • Implement and maintain appropriate security measures across big data platforms, including user access control, data encryption, and compliance with data protection regulations.
  • Work with security teams to conduct audits and maintain access controls across systems.
Documentation & Knowledge Sharing
  • Maintain thorough documentation on system architecture, deployment procedures, troubleshooting guides, and maintenance protocols.
  • Share knowledge and provide support to other team members, developers, and stakeholders in optimizing big data solutions.

Required Skills and Qualifications:

  • Proven experience as a Big Data Administrator or System Administrator with strong expertise in Hadoop, Spark, and Kafka.
  • Hands-on experience deploying, managing, and scaling Hadoop clusters, Apache Spark jobs, and Kafka brokers in production environments.
  • Strong knowledge of Linux/Unix system administration, with experience in configuring and maintaining data infrastructure.
  • Expertise in troubleshooting, resolving issues, and optimizing the performance of big data environments.
  • Experience with deployment automation tools (e.g., Terraform, Jenkins).
  • Familiarity with data ingestion and ETL processes in a big data context.
  • Knowledge of cloud-based environments (e.g., AWS, GCP, Azure) for managing big data clusters.
  • Familiarity with monitoring tools like Ambari, Grafana.
  • Strong scripting skills (e.g., Bash, Python, or other shell scripting languages) for automation and troubleshooting.
  • Solid understanding of distributed computing principles and data storage systems (e.g., HDFS, S3, etc.).
  • Strong communication skills and the ability to collaborate with engineering, development, and operations teams.

  • Big Data Ecosystem Architect

    7 dni temu

    centrum, Warszawa, Polska Kevin Edward Pełny etat

    Job DescriptionWe are seeking a seasoned Big Data Ecosystem Architect to spearhead the design and deployment of large-scale big data environments at Kevin Edward.The ideal candidate will have a deep understanding of distributed computing principles, data storage systems, and expertise in Hadoop, Spark, and Kafka administration.Key Responsibilities:Hadoop,...

  • Senior Big Data Architect

    1 tydzień temu

    Warszawa, Mazovia, Polska Jit Team Pełny etat

    **Job Description:**We are seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will work with the Hadoop ecosystem to develop and manage big data solutions.**Key Responsibilities:**Develop and manage big data solutions using Hadoop ecosystemSet up CI/CD pipelines and work with Docker and Kubernetes platformsManage relational...

  • Big Data Engineering Lead

    4 tygodni temu

    Warszawa, Mazovia, Polska B2B S.A Pełny etat

    Senior Data EngineerYour Key Responsibilities:Data Ingestion and Transformation: Collaborate with cross-functional teams to design, develop, and maintain large-scale data ingestion and transformation processes using Spark and Python.Distributed Data Processing: Build high-performance, parallelized big data processing pipelines to process massive amounts of...

  • Hadoop Ecosystem Architect

    5 dni temu

    aleje Jerozolimskie , Warszawa, Polska Kevin Edward Pełny etat

    About Kevin EdwardWe are a forward-thinking company that values innovation and excellence. At Kevin Edward, we strive to deliver cutting-edge solutions that drive business success.Job Title: Hadoop Ecosystem ArchitectJob Summary:We are seeking an experienced Hadoop Ecosystem Architect to join our team. As a key member of our big data infrastructure team, you...

  • Big Data Engineer

    2 miesięcy temu

    Warszawa, Mazovia, Polska B2B S.A Pełny etat

    Big Data Job DescriptionWe are seeking an experienced Big Data Developer to work on a data-driven initiative focused on enhancing group risk reporting and supporting various use cases through the development of new and optimization of existing data pipelines.Job ResponsibilitiesThe role involves re-modeling and aggregating data inputs into a common format,...

  • Big Data Developer

    2 miesięcy temu

    Warszawa, Polska B2B.net S.A Pełny etat

    We are looking for an experienced Big Data Developer for a data-driven initiative focused on enhancing group risk reporting and supporting various use cases through the development of new and optimization of existing data pipelines.Big Data DeveloperYour responsibilities The role involves re-modeling and aggregating data inputs...

  • Big Data Engineer

    1 miesiąc temu

    Daimlera, Warszawa, Polska DCG Pełny etat

    About UsAt DCG, we specialize in providing experienced professionals to businesses across various sectors. Our unique approach enables us to understand the specific needs of our clients and provide tailored support.About the RoleWe are currently looking for a talented Big Data Engineer to join our team. As a Big Data Engineer - Risk Reporting and Analytics,...

  • Senior Big Data Engineer

    5 dni temu

    Warszawa, Mazovia, Polska DCV Technologies Pełny etat

    Job Title: Senior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to join our team at DCV Technologies in Warsaw, Poland. This is an exciting opportunity to work with a multinational client in the financial sector, performing Big Data Administration and Engineering tasks on cutting-edge open-source platforms.About the Job:The...

  • Big Data Reliability Engineer

    3 tygodni temu

    Warszawa, Polska DCV Technologies Pełny etat

    DCV Technologies is seeking a highly skilled Big Data Reliability Engineer to join our team in Poland.We offer a competitive daily rate of up to 1,250 PLN/day + VAT for this role.About the RoleAs a Big Data Reliability Engineer, you will be responsible for ensuring the reliability and performance of our Big Data platforms, including Hadoop, Spark, and HBASE....

  • Big Data Analyst

    2 miesięcy temu

    Warszawa, mazowieckie, Polska InPost Pełny etat

    Role Overview:As a Big Data Analyst, you will be at the forefront of analyzing vast volumes of real-time data from diverse systems, including parcel lockers, package lifecycle, transportation, user communication and similar topics relative to InPost Group products. This role spans across Poland and extends to 7 international markets, offering a unique...

  • Senior Big Data Developer

    3 tygodni temu

    Domaniewska A, Warszawa, Polska emagine Polska Pełny etat

    Job DescriptionWe are looking for an experienced Senior Big Data Developer to join our team at emagine Polska. As a key member of our data engineering team, you will be responsible for designing, implementing, and maintaining large-scale data processing systems using Hadoop and related technologies.The ideal candidate will have a strong background in big...

  • Big Data Analytics Expert

    1 miesiąc temu

    Aleje Jerozolimskie A, Warszawa, Polska Link Group Pełny etat

    Link Group - Job DescriptionWe are seeking a highly skilled PySpark Developer to join our team in big data processing and analytics.Salary:$150,000 - $200,000 per annumAbout the RoleAs a Big Data Analytics Expert, you will design, implement, and optimize large-scale data processing pipelines, leveraging the capabilities of Apache Spark and Python.Key...

  • Scala&Big Data Developer

    2 miesięcy temu

    Warszawa, Polska Square One Resources Sp. z o.o. Pełny etat

    This data-driven initiative supports group risk reporting and other Use Cases by implementing new or improving existing data pipelines.Data requirements cover re-modeling and aggregating defined data input into common company format.Developer hired for this position will be working on upstream source integrations, efficient data...

  • Senior Big Data Engineer

    7 dni temu

    Aleje Jerozolimskie A, Warszawa, Polska Link Group Pełny etat

    Job DescriptionWe are seeking a highly skilled Senior Big Data Engineer with at least 5 years of experience in designing, implementing, and optimizing large-scale data processing pipelines. The ideal candidate will leverage Apache Spark and Python to develop efficient ETL pipelines for big data analysis.Key Responsibilities:Develop, test, and maintain...

  • Senior Data Solutions Engineer

    5 dni temu

    Warszawa, Polska Jit Team Pełny etat

    About the RoleWe are seeking a skilled Data Engineer to join our team as a Senior Data Solutions Engineer. This role will involve designing, developing, and implementing big data solutions using Hadoop ecosystem.The ideal candidate will have expertise in Hadoop Cloudera distribution, proficiency in Docker and Kubernetes platforms, and strong knowledge of...

  • Big Data Developer

    3 tygodni temu

    Warszawa, mazowieckie, Polska Grupa PZU Pełny etat

    Twój zakres obowiązkówProjektowanie i implementacja nowoczesnych rozwiązań w oparciu o technologie Big DataDefiniowanie i rozwijanie dobrych praktyk i standardów programistycznych w PZUZapewnianie właściwej jakości wytwarzanego oprogramowaniaWspółpraca z członkami zespołu zwinnego oraz projektowego w celu efektywnej realizacji wymagań...

  • Big Data Solutions Engineer

    3 tygodni temu

    Warszawa, Mazovia, Polska DCV Technologies Pełny etat

    We are seeking a highly skilled Hadoop Administrator to join our client team on a B2B contract basis.This is an exciting opportunity to work with a multinational client headquarted in the US and a leader in the financial sector, performing Big Data Administration and Engineering tasks on cutting-edge open-source platforms.Job Opportunity: Hadoop...

  • Big Data Expert

    3 dni temu

    Warszawa, Mazovia, Polska Datumo Pełny etat

    Unlock Your Potential as a Big Data ExpertAre you passionate about driving innovation and growth in the field of big data? Do you have a strong background in Java and a desire to transition into the exciting world of big data?About DatumoDatumo is a leading provider of big data and cloud consulting services, serving clients across Western Europe, Poland, and...

  • Big Data Analysis Platform Developer

    1 miesiąc temu

    Warszawa, Mazovia, Polska Jit Team Pełny etat

    We are seeking a highly skilled Big Data Analysis Platform Developer to join our team in Poland. This is an exciting opportunity to contribute to the development of an advanced Big Data analysis platform utilizing Hadoop, Scala/Spark, and HDFS technologies.Job DescriptionThis project focuses on creating a cutting-edge Big Data analysis platform that manages...

  • Big Data Solutions Architect

    2 miesięcy temu

    Warszawa, mazowieckie, Polska BlueSoft Pełny etat

    Job Summary:Data Solutions Architects play a key role in designing and delivering advanced data architectures and analytics solutions. Focused on both cloud and on-premise environments, this role ensures robust data management and security. You will lead architectural sessions, create proof-of-concepts, and collaborate with clients to build scalable, secure...