Foundational gives us visibility to column-level lineage during development; Team leads can better review PRs that are more sensitive; It helps us avoid breaking queries that were previously unknown to us. Everyone understands the entire data flow when making changes.
Company: Lightricks
Industry: Consumer Apps
Lightricks is a fast-growing technology company who is disrupting photo and video editing in mobile, known for its super popular app, Facetune. Across its app portfolio, Lightricks has millions of monthly active users (MAUs) and has had over 700M users install its apps.
This case study demonstrates how Foundational — a leader in source-code lineage, data quality and governance — has helped Lightricks get end-to-end visibility with data lineage, improved data development thanks to its and downstream impact analysis, and improved data and code quality across the data organization
Lightricks relies on BigQuery as its primary data warehouse, along with Tableau as the business intelligence tool. It uses dbt Cloud for modeling and transformation, across a 500+ model system that powers hundreds of tables and Tableau dashboards. There are multiple dbt projects as well as hundreds of queries that run outside of dbt. Multiple groups rely on mission-critical data at Lightricks, between analytics, data science, and marketing.
Lightricks has multiple groups that leverage data on an ongoing basis to drive insights. While it runs dbt Cloud, data lineage has remained a challenge, in particular around the actual usage by different groups, which is often outside of dbt, and the potential impact upstream changes have on downstream reports
With multiple people touching data at different stages of the data lifecycle, the Lightricks organization looked into ways of continuing to scale effectively while ensuring high quality standards and making it easy to manage data changes.
Lightricks has several parts to its data stack: BigQuery, which runs multiple dbt Cloud projects, Tableau, which is used by business analysts, and additional queries and notebooks used by Data Science teams for exploration and research–a critical component in Lightricks’ innovative data journey.
Given the sensitivity to data quality and downstream impact from code changes, Lightricks’ data organization was looking for solutions to improve visibility during development and proving different stakeholders and in particular managers, for better controls.
The team has found Foundational as a perfect partner to provide lineage and downstream impact, as well as preventative data quality, for its data stack.
Foundational offers a unique approach to lineage and downstream impact analysis during code development, in a fragmented world of data tools.
Our proprietary solution to source-code based data lineage allows for easy integration across every part of the data stack, and in particular–the ability for developers to understand the downstream impact of code changes pre-deploy while the cost of mistakes is still small.
Foundational does not require data access, which gives IT and Security teams greater piece of mind, knowing that no 3rd party can access sensitive data and PII.
while keeping data security
The Foundational team are world-class experts in reverse engineering and source code analysis, who put customer satisfaction and user value as the top priority.
Our expertise is not just in handling the tools and technology but also in understanding real-world deployments and constraints, being able to adapt to various challenges when analyzing complex data environments.
With Foundational, Lightricks found a partner who iterated quickly around requests and customization to unique use cases, working closely with various stakeholders.
Foundational partnered with Lightricks data organization to provide a better development experience around data, which finds and checks every downstream element that’s impacted across BigQuery, dbt Cloud, and Tableau. Our holistic solution incorporated two key components:
Our integrations with multiple dbt projects, BigQuery and Tableau allowed Foundational to automate data lineage across all of these tools – being always updated to the latest commit. In Lightricks’ case, Foundational is also extracting BigQuery tags to identify which team is potentially impacted by a new pull request.
While traditional lineage tools rely on query logs, Foundational’s source-code based approach allows for real-time lineage, better accuracy, and improved visibility to non-warehouse flows.
The lineage provided by dbt Cloud is useful for Tableau dashboards, but is falling short in the case of BigQuery queries that are non-Tableau and non-git, which in this case are heavily used by Data Science.
Foundational’s integration with GitHub enables Downstream Impact Analysis for every pull request, providing visibility to data engineers to the cross-team impact of their code change, before deploying it. This visibility provides confidence and replaces the need for manual checks, ultimately costing time and money to Lightricks.
Foundational enables Lightricks to leverage BigQuery tags to identify data consumers across the company, providing a simple mechanism for data contracts between data producers and consumers who wish to be kept in the loop for the case of upstream changes.
With Foundational, Lightricks was able to completely remove the risk of unexpectedly impacting data for downstream consumers, as any critical path can be registered through a tag, and benefit from visibility and data contract enforcement.
Foundational uses a connector-based setup and does not process or extract any data or user records. This means that the only access we get is to code and configurations, which provides for better data security and access controls.
In Lightricks’ case, Foundational is also leveraging an on-prem Tableau connector to ensure that API requests to Tableau do not leave the company’s cloud environment.
Foundational natively integrates with git, which in Lightricks’ case is provided by GitHub, to power the developer experience directly in the interface developers use, regardless of their department in the organization.
The deployment was highly successful, marked by clear improvements in KPIs and positive outcomes:
Lastly, but most importantly, the team at Lightricks is satisfied. When asked about the partnership, Eyal El-Bahar, the VP of Data at Lightricks, has said, “Foundational is critical for ensuring we can ship code without negatively impacting downstream consumers. This was previously a big problem and is simply no longer the case. We also appreciate the team’s responsiveness to our specific use cases and generally to product feedback.“
