Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove usage of referenced_tables in metadata.yaml #4815

Open
scholtzan opened this issue Jan 11, 2024 · 2 comments
Open

Remove usage of referenced_tables in metadata.yaml #4815

scholtzan opened this issue Jan 11, 2024 · 2 comments

Comments

@scholtzan
Copy link
Collaborator

scholtzan commented Jan 11, 2024

We used to use dryrun to get all tables a query references. This caused problems for queries that were referencing tables dryrun didn't have permissions to access. So we added referenced_tables to use the explicitly referenced tables instead of doing a dryrun and also to speed up the process for queries referencing main_v4.
We did make a change to use sqlglot to determine table dependencies. So it should be safe to remove referenced_tables and ignore them when generating Airflow DAGs.

┆Issue is synchronized with this Jira Task

@ANich
Copy link
Contributor

ANich commented Jun 10, 2024

Should we also remove referencedTables from the dry run cloud function and related utils too?

@scholtzan
Copy link
Collaborator Author

There are still a few edge cases where referenced_tables are used afaik.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants