Background

In large and fast-growing companies, when the data warehouse becomes filled with an overwhelming amount of data, both producers and consumers struggle to determine which datasets can be trusted. This lack of data quality checks leads to various issues for data owners, requiring time and resources to detect and respond to incidents. Similarly, data consumers waste valuable time writing validation queries to ensure the reliability of datasets and attributes.

Verity is a data quality tool that enables trustable data, thus ensuring accurate data-driven decisions. It evolved from a system to assert and monitor the semantic correctness of offline data through static checks to cover many more use cases like table diff, table landing SLOs, templated checks, and Real-time checks using online events.

Role

I was part of the founding team tasked with creating the project from the ground up. As the Frontend tech lead of a group of up to six members, my responsibilities included:

• Building from scratch Verity UI, the data quality check tracking and debugging web application, using Lyft's Frontend stack: NextJS, React.js, TypeScript, Styled Components, and the LPL Design System.
• Collaborating on setting the UX of the project with our product designer.
• Preparing and reviewing technical design specifications based on functional requirements, user requests, and usage data.
• Delivering code that encouraged the team regarding best practices, readability, simple design, reusability, and testing.
• Increasing the team productivity and focus by leading Frontend planning, milestones, and product and technical roadmaps.
• Facilitating cross-org data quality information sharing and optimal web performance by owning Verity's Golang GraphQL server integration.
• Owning the user documentation, including planning and prioritization for the team.
• Enhancing shipping velocity, developer experience, observability, and code quality standards by adopting tooling and agile processes (sprint planning, grooming, story mapping).

Results

Verity was a solid success at Lyft, growing adoption from zero to 65 teams, 9600 weekly data quality checks over 2500+ data sources, and verifying 25TB of data daily.

These numbers impacted the business by reducing operational costs via SEVs reduction and avoiding data-driven decision mistakes, enabling trustworthy financial business metrics tracking and early detection of data regressions to avoid expensive and error-prone data backfills.