Big Data Automation Lead
WHAT IS THE OPPORTUNITY?
We are seeking a highly skilled Big Data Automation Lead with 4-5+ years of experience to drive automation initiatives across our big data platforms. The ideal candidate will have a strong background in Python, Pandas, PySpark, and automation frameworks, with hands-on experience in big data technologies and data engineering best practices.
WHAT WILL YOU DO?
- Lead the design, development, and implementation of automation solutions for big data pipelines and workflows.
- Develop and maintain robust automation scripts using Python, Pandas, and PySpark.
- Implement and manage test automation using Robot Framework and BDD frameworks.
- Design and execute automated API testing and validation.
- Write and optimize complex SQL queries for data validation and transformation.
- Work with UNIX-based systems for scripting, automation, and job scheduling.
- Manage and automate data processes on HDFS, Hive, Hadoop, and Spark environments.
- Collaborate with data engineers, QA, and business stakeholders to ensure data quality and process efficiency.
- Troubleshoot and resolve issues in data pipelines and automation frameworks.
WHAT DO YOU NEED TO SUCCEED?
Must have:
- Strong programming skills in Python, with experience in Pandas and PySpark.
- Hands-on experience with Robot Framework and BDD (Behavior Driven Development) frameworks.
- Experience with API testing and automation.
- Advanced SQL skills for data manipulation and validation.
- Experience working in UNIX environments (shell scripting, process automation).
- Solid understanding of HDFS, Hive, Hadoop, and Spark.
- Strong problem-solving and analytical skills.
- Excellent communication and leadership abilities.
Nice-to-have:
- Experience in leading automation teams or projects.
- Exposure to CI/CD tools and cloud-based big data platforms.
- Experience with GitHub Actions for CI/CD automation.
- Experience with Ansible Automation Platform for infrastructure and workflow automation.
WHAT’s IN FOR YOU?
We thrive on the challenge to be our best, progressive thinking to keep growing, and working together to deliver trusted advice to help our clients thrive and communities prosper. We care about each other, reaching our potential, making a difference to our communities, and achieving success that is mutual.
- A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable
- Leaders who support your development through coaching and managing opportunities
- Ability to make a difference and lasting impact
- Work in a dynamic, collaborative, progressive, and high-performing team
- A world-class training program in financial services
- Flexible work/life balance options
- Opportunities to do challenging work
#LI-HYBRID
#LI-POST
Job Skills
Critical Thinking, Decision Making, Detail-Oriented, DevOps, IT Quality Assurance, Long Term Planning, Process Management, Risk Management, Software Product Testing, Test Automation