Lead Data Engineer

What You’ll Do

  • Work at all levels of the stack. – infrastructure (AWS, ECS, EKS, Lambda, Kinesis), database (Postgres, Snowflake), services (dbt, Cube.js)
  • Architect robust reusable systems that are not only performant and scalable but thoughtfully crafted
  • Contribute to the core design of data architecture, data models and schemas, and implementation plan.
  • Design, build and maintain optimal data pipeline architecture for optimal extraction, transformation, and loading of data from a wide variety of data sources, including external APIs, data streams, and data stores.
  • Design, create and maintain the infrastructure for ingesting data into our data lake and data warehouse and providing frameworks and services for operating on that data.
  • Design, create and maintain the infrastructure for real-time streaming analytics, big data analytics, and machine learning analytics capabilities.
  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
  • Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
  • Work with security to implement data privacy and data security requirements to ensure solutions stay compliant to security standards and frameworks.

Key Qualifications

  • 6+ Years of experience with python and micro-services architecture.
  • Expertise with relational and non-relational data modeling.
  • Proficient with message queuing and stream processing.
  • Experience integrating 3rd Party APIs and authentication services.
  • Deep knowledge and hands-on experience in areas such as data structures, database table design, algorithm design, runtime complexity, system architecture (scalable, reliable, redundant design), API design, security and privacy best practices, at-scale monitoring, logging & alerting and testing best practices.
  • Demonstrated success in owning projects end-to-end, including working with technical and non-technical stakeholders and making decisions with minimal supervision.
  • Ability to collaborate and empathize with a variety of individuals. Experience iterating with users and non-technical stakeholders with a strong understanding of how technical decisions impact them.

Preferred Qualifications

  • Strong knowledge of AWS including managed AWS services
  • Experience with build tools like Jenkins, Terraform and Cloud Formation
  • Experience at an early-stage startup
  • Experience working on a financial application
  • Prior experience scaling data pipelines to handle 10’s of billions of records daily.