Lead Data Engineer
What You’ll Do
- Work at all levels of the stack. – infrastructure (AWS, ECS, EKS, Lambda, Kinesis), database (Postgres, Snowflake), services (dbt, Cube.js)
- Architect robust reusable systems that are not only performant and scalable but thoughtfully crafted
- Contribute to the core design of data architecture, data models and schemas, and implementation plan.
- Design, build and maintain optimal data pipeline architecture for optimal extraction, transformation, and loading of data from a wide variety of data sources, including external APIs, data streams, and data stores.
- Design, create and maintain the infrastructure for ingesting data into our data lake and data warehouse and providing frameworks and services for operating on that data.
- Design, create and maintain the infrastructure for real-time streaming analytics, big data analytics, and machine learning analytics capabilities.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
- Work with security to implement data privacy and data security requirements to ensure solutions stay compliant to security standards and frameworks.
- 6+ Years of experience with python and micro-services architecture.
- Expertise with relational and non-relational data modeling.
- Proficient with message queuing and stream processing.
- Experience integrating 3rd Party APIs and authentication services.
- Deep knowledge and hands-on experience in areas such as data structures, database table design, algorithm design, runtime complexity, system architecture (scalable, reliable, redundant design), API design, security and privacy best practices, at-scale monitoring, logging & alerting and testing best practices.
- Demonstrated success in owning projects end-to-end, including working with technical and non-technical stakeholders and making decisions with minimal supervision.
- Ability to collaborate and empathize with a variety of individuals. Experience iterating with users and non-technical stakeholders with a strong understanding of how technical decisions impact them.
- Strong knowledge of AWS including managed AWS services
- Experience with build tools like Jenkins, Terraform and Cloud Formation
- Experience at an early-stage startup
- Experience working on a financial application
- Prior experience scaling data pipelines to handle 10’s of billions of records daily.