InnerVault

Data Pipeline taste

Indexedprompts.chat (CC0 prompt collection), f

# Data Pipeline taste of a data engineer - Prefer ELT: land raw data unchanged first, then transform inside the warehouse. Confidence: 0.85 - Keep raw, staging and mart layers separate, and never let dashboards read…

Open the source (opens github.com) https://github.com/f/prompts.chat

What it is

Prompts from f.

The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.

The prompt

# Data Pipeline taste of a data engineer
- Prefer ELT: land raw data unchanged first, then transform inside the warehouse. Confidence: 0.85
- Keep raw, staging and mart layers separate, and never let dashboards read from raw tables. Confidence: 0.90
- Model transformations in dbt (or SQLMesh) as version-controlled SQL with code review like any other code. Confidence: 0.90
- Make every job idempotent: rerunning a day must produce the same result without duplicates. Confidence: 0.95
- Partition large tables by event date and process data in date-bounded increments. Confidence: 0.85
- Backfills are a first-class operation with a documented command, not a manual one-off script. Confidence: 0.85
- Add tests for uniqueness, not-null, accepted values and referential integrity on every important model. Confidence: 0.90
- Define data contracts with upstream producers so schema changes fail in their CI, not in your pipeline. Confidence: 0.80
- Track freshness per source and alert when data is late, not only when a job fails. Confidence: 0.90
- Orchestrate with Airflow, Dagster or Prefect, and keep heavy computation out of the orchestrator workers. Confidence: 0.80
- Prefer asset-based orchestration that knows which tables depend on which over plain task graphs. Confidence: 0.70
- Store timestamps in UTC and convert to local time only in the presentation layer. Confidence: 0.95
- Use Parquet or another columnar format with explicit schemas for files in the lake. Confidence: 0.85
- Separate event time from processing time and handle late-arriving data explicitly. Confidence: 0.85
- Keep PII in as few tables as possible, mask or hash it early and document access in one place. Confidence: 0.90
- Use incremental models only after a full refresh becomes too slow; correctness first, cost second. Confidence: 0.75
- Name models consistently (stg_, int_, fct_, dim_) and document every column people actually query. Confidence: 0.80
- Use slowly changing dimension type 2 snapshots when history matters, not ad hoc audit columns. Confidence: 0.70
- Run pipelines in CI against a sample or zero-copy clone before deploying changes to production. Confidence: 0.80
- Monitor warehouse cost per model and kill queries that scan whole tables without partition filters. Confidence: 0.75
- Every metric has one owner and one definition in a semantic layer, not five versions in five dashboards. Confidence: 0.80

Written by the prompts.chat (CC0 prompt collection) community (f). Not tested by InnerVault.

How to get it

Setup steps: not applicable.

Related

  1. a lot of people even professional said a software product or platfrom especially which has complex rules which was fully coded by AI by a vibe coder who deosnt understand what is the arcture concept deployed, read and…

    Indexed prompts.chat (CC0 prompt collection), hussen.sepa@gmail.com CC0-1.0

  2. code-hacker

    Prompts

    Code Hacker - VS Code Custom Agent + Web + TUI

    Stale GitHub, hackerlibs 55 ★MITUpdated 17 days ago

    git clone https://github.com/hackerlibs/code-hacker.git
  3. Act as a Code Optimization Expert specialized in C#. You are an experienced software engineer focused on enhancing performance when dealing with large-scale data processing. Your task is to provide professional…

    Indexed prompts.chat (CC0 prompt collection), bateyyat@gmail.com CC0-1.0

  4. Code Recon

    Prompts

    # SYSTEM PROMPT: Code Recon # Author: Scott M. # Goal: Comprehensive structural, logical, and maturity analysis of source code. --- ## 🛠 DOCUMENTATION & META-DATA * **Version:** 2.7 * **Primary AI Engine (Best):**…

    Indexed prompts.chat (CC0 prompt collection), thanos0000@gmail.com CC0-1.0

  5. # Database Architect You are a senior database engineering expert and specialist in schema design, query optimization, indexing strategies, migration planning, and performance tuning across PostgreSQL, MySQL, MongoDB…

    Indexed prompts.chat (CC0 prompt collection), wkaandemir CC0-1.0

  6. # Database Migrations taste - Keep every schema change in a versioned migration file committed next to the code that needs it. Confidence: 0.95 - Write migrations in plain SQL rather than ORM-generated diffs you have…

    Indexed prompts.chat (CC0 prompt collection), f CC0-1.0