Data Engineer - Elastic

Who We Are

Kyndryl is a market leader that thinks and acts like a start-up. We design, build, manage, and modernize the mission-critical technology systems that the world depends on every day. So why work with Kyndryl? Because we are always moving forward–always pushing ourselves to go further in our efforts to build a more equitable, inclusive world for our employees, our customers, and our communities.

The Role

• Job Description:
  An ELK(Elastic, Logstash & Kibana) Data Engineer is responsible for developing, implementing, and maintaining ELK stack-based solutions within an organization. This engineer plays a crucial role in developing efficient and effective log processing, indexing, and visualization for monitoring, troubleshooting, and analysis purposes.
  Key Responsibilities:
  1. Installation and Configuration:
     - Deploy Elasticsearch, Logstash, and Kibana on virtual machines, containerized environments (e.g., Docker).
     - Set up and configure Elasticsearch clusters, ensuring high availability, fault tolerance, and scalability.
     - Configure Logstash to receive, filter, and transform logs from diverse sources (e.g., servers, applications, AppDynamics, Storage, Databases, etc.) before sending them to Elasticsearch.
     - Configure Kibana for data visualization, including creating visualizations, dashboards, and alerts.
     - Configure ILM policies, Index templates, etc.
  2. Data Ingestion and Processing:
     - Develop Logstash configuration files to parse, enrich, and filter log data from various input sources (e.g., AppDynamics, Database, Storage, etc.).
     - Implement techniques like grok patterns, regular expressions, and plugins to handle complex log formats and structures.
     - Ensure efficient and reliable data ingestion by optimizing Logstash performance, handling high data volumes, and managing throughput.
  3. Data Transformation and Enrichment:
  - Define and implement data transformation and enrichment processes to cleanse, enrich, and prepare data for analysis and visualization.
  - Use Logstash pipelines for data manipulation and transformation.
  4. Data visualization and reporting:
  - Use Kibana to create visually appealing dashboards, reports, and custom visualizations.
  - Collaborate with business users to understand their data visualization needs and translate them into actionable insights.
  - Establish correlations within the data and develop visualizations to detect the root cause of issues.
  - Integration with ticketing tools such as Service Now.
  - Hands-on with ML and Watcher functionalities.
  4. Cluster Monitoring:
     - Monitor Elasticsearch clusters for health, performance, and resource utilization.
   5. Security and Access Control:
     - Implement security measures for the ELK stack, including secure transport, authentication, and authorization.
     - Configure role-based access control (RBAC) to manage user access privileges within Kibana and Elasticsearch.
  6. Documentation and Collaboration:
     - Create and maintain technical documentation, including system diagrams, deployment procedures, and troubleshooting guides.
     - Collaborate with cross-functional teams, such as developers, system administrators, and security teams, to integrate ELK stack functionalities into existing systems and workflows.
     - Stay up-to-date with industry trends and best practices related to log management, Elasticsearch, Logstash, and Kibana.

Who You Are

• The candidate should have a minimum of 6+ years of experience.
• Implement Data-related Development Methods.
• Design Databases.
• Apply Data Modelling Techniques to IT Solutions.
• Apply Knowledge of Data Mining Techniques & Tools.
• Applications, Data and AI.

Being You

Diversity is so much more than what we look like, or where we come from. It's how we think, and who we are. We welcome people of all cultures, backgrounds, and experiences. But we're not doing it alone: Our Kyndryl Inclusion Networks are just one of the ways we are creating a workplace where all Kyndryls can find support and advice. This dedication to welcoming everyone into our company means that Kyndryl gives you–and everyone next to you–the ability to bring your whole self to work, individually and collectively, and support the activation of our equitable culture. That’s the Kyndryl Way.

What You Can Expect

With state-of-the-art resources and Fortune 100 clients, every day is an opportunity to innovate, build new capabilities, new relationships, new processes, and new value. Our employee learning hub gives you access to the best learning in the industry to receive certifications and accreditations, including Microsoft University, AWS Cloud Center of Excellence, Udemy, and the Harvard Business Review. Through our company-wide volunteering and giving platform, you can donate, start fundraisers, volunteer, and search over 2 million non-profit organizations.  At Kyndryl, we invest heavily in you! We want you to succeed so that together, we will all succeed.