Data Pipeline taste
# Data Pipeline taste of a data engineer - Prefer ELT: land raw data unchanged first, then transform inside the warehouse. Confidence: 0.85 - Keep raw, staging and mart layers separate, and never let dashboards read…
Open the source (opens github.com) https://github.com/f/prompts.chat
What it is
Prompts from f.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
The prompt
# Data Pipeline taste of a data engineer - Prefer ELT: land raw data unchanged first, then transform inside the warehouse. Confidence: 0.85 - Keep raw, staging and mart layers separate, and never let dashboards read from raw tables. Confidence: 0.90 - Model transformations in dbt (or SQLMesh) as version-controlled SQL with code review like any other code. Confidence: 0.90 - Make every job idempotent: rerunning a day must produce the same result without duplicates. Confidence: 0.95 - Partition large tables by event date and process data in date-bounded increments. Confidence: 0.85 - Backfills are a first-class operation with a documented command, not a manual one-off script. Confidence: 0.85 - Add tests for uniqueness, not-null, accepted values and referential integrity on every important model. Confidence: 0.90 - Define data contracts with upstream producers so schema changes fail in their CI, not in your pipeline. Confidence: 0.80 - Track freshness per source and alert when data is late, not only when a job fails. Confidence: 0.90 - Orchestrate with Airflow, Dagster or Prefect, and keep heavy computation out of the orchestrator workers. Confidence: 0.80 - Prefer asset-based orchestration that knows which tables depend on which over plain task graphs. Confidence: 0.70 - Store timestamps in UTC and convert to local time only in the presentation layer. Confidence: 0.95 - Use Parquet or another columnar format with explicit schemas for files in the lake. Confidence: 0.85 - Separate event time from processing time and handle late-arriving data explicitly. Confidence: 0.85 - Keep PII in as few tables as possible, mask or hash it early and document access in one place. Confidence: 0.90 - Use incremental models only after a full refresh becomes too slow; correctness first, cost second. Confidence: 0.75 - Name models consistently (stg_, int_, fct_, dim_) and document every column people actually query. Confidence: 0.80 - Use slowly changing dimension type 2 snapshots when history matters, not ad hoc audit columns. Confidence: 0.70 - Run pipelines in CI against a sample or zero-copy clone before deploying changes to production. Confidence: 0.80 - Monitor warehouse cost per model and kill queries that scan whole tables without partition filters. Confidence: 0.75 - Every metric has one owner and one definition in a semantic layer, not five versions in five dashboards. Confidence: 0.80
Written by the prompts.chat (CC0 prompt collection) community (f). Not tested by InnerVault.
How to get it
Setup steps: not applicable.
Related
-
a lot of people even professional said a software product or platfrom especially which has complex rules which was fully coded by AI by a vibe coder who deosnt understand what is the arcture concept deployed, read and…
-
code-hacker
PromptsCode Hacker - VS Code Custom Agent + Web + TUI
git clone https://github.com/hackerlibs/code-hacker.git -
Act as a Code Optimization Expert specialized in C#. You are an experienced software engineer focused on enhancing performance when dealing with large-scale data processing. Your task is to provide professional…
-
Code Recon
Prompts# SYSTEM PROMPT: Code Recon # Author: Scott M. # Goal: Comprehensive structural, logical, and maturity analysis of source code. --- ## 🛠 DOCUMENTATION & META-DATA * **Version:** 2.7 * **Primary AI Engine (Best):**…
-
Database Architect Agent Role
Prompts# Database Architect You are a senior database engineering expert and specialist in schema design, query optimization, indexing strategies, migration planning, and performance tuning across PostgreSQL, MySQL, MongoDB…
-
Database Migrations taste
Prompts# Database Migrations taste - Keep every schema change in a versioned migration file committed next to the code that needs it. Confidence: 0.95 - Write migrations in plain SQL rather than ORM-generated diffs you have…