Senior Data Engineer/ Lead Data Engineer

The Upstox Story

Upstox is one of India's premier Fin-Tech companies, passionate about making trading and investing accessible to everyone. Our intuitive trade and investment platform, combined with commission-free pricing, empowers both new investors and seasoned traders to invest across a variety of asset categories including Stocks, IPOs, and Mutual Funds.

With a customer-centric approach and powerful tools, we experienced an 800% growth in our customer base, expanding from 25,000 in 2017 to 200,000 in 2019. Our commitment to simplicity, affordability, and accessibility has led over 10 million customers to trust us with their investments as of 2020, heralding our path to becoming an industry leader.

Founded in 2009 by Ravi Kumar and Shrinivas Viswanath, and joined by Kavitha Subramanian in 2016, Upstox is backed by industry stalwart Ratan Tata. With over $29 million raised in Series A and B funding rounds from Kalaari Capital and Tiger Global Management respectively, our growth trajectory has been exponential.

We boast a team of highly skilled technology and finance professionals and are seeking motivated experts to join our dynamic team.

Job Opening: Senior Data Engineer / Lead Data Engineer

Who You Will Work With

The Data Engineering team at Upstox is tasked with architecting, developing, and maintaining a unified Lake House (Data Lake + Data Warehouse). This serves as the organization's single source of truth, while also governing data assets, generating business and customer insights, and securely exposing data to downstream systems, services, and users.

Key Responsibilities

• Create and maintain scalable Big-Data ETL pipelines feeding data into the Upstox Data Platform (UDP).
• Develop modular and scalable Big-Data processors to derive business and customer insights, enhance operational efficiency, and measure key business metrics.
• Build scalable connectors to securely expose data for consumption by downstream systems and services in near real-time.
• Collaborate with the DevOps and Infrastructure teams to build optimal data extraction, transformation, and loading infrastructure.
• Work with the DevOps team to monitor and maintain data platform components, ensuring internal SLA uptime is met.
• Collaborate with the DBA and BI teams to establish data quality control processes.

Qualifications

• Hands-on experience with Python / Scala, Spark or similar frameworks, Airflow, Apache HUDI/Delta/Iceberg (OpenTableFormat), AWS Athena, Big-Data ETL pipelines, Hadoop, advanced SQL, and SQL query tuning.
• Exposure to streaming frameworks (Spark Streaming / Apache Flink), workflow management tools (Apache Airflow / NiFi), DBT (data build tool), dimensional data modeling, and cloud-native solutions (preferably on AWS) is a plus.
• Understanding of AWS Services like Redshift, DynamoDB, Lambda, Glue, Athena, Lake Formation, IAM, SQS & SNS, along with experience creating CloudFormation / Terraform templates and knowledge of DevOps & SRE will earn you extra points.
• At least 7 years of experience in a Data Engineering role handling large data volumes.
• Bachelor's or Master's degree in Computer Science or equivalent.
• Resilience, determination, meticulous problem-solving skills, and attention to detail.
• A strong desire to work in a fast-paced dynamic environment.
• Technical expertise to deliver scalable enterprise-grade solutions independently.
• Exceptional communication skills and the ability to collaborate with multiple stakeholders effectively.

Location: Bengaluru

Company Name: Upstox

Job Title: Senior Data Engineer/ Lead Data Engineer