AI for Data Engineering and Analytics

Use assistants to write, explain, and debug analytical code while keeping results verifiable.

• Giving assistants schema, sample rows, and expected output
• Generating pandas and SQL transformations
• Explaining and optimizing inherited queries
• Debugging errors and unexpected results with assistant help
• Checking AI-written code against known totals
• Handling sensitive data and what not to paste into prompts

Hands-on lab: Solve six analytical questions on a sample database with an assistant, then verify each answer with an independently written query.

Build a controlled workflow that turns business questions into reviewed SQL.

• How text-to-SQL works and common failure modes
• Providing schema, descriptions, and business definitions as context
• Read-only access, query limits, and validation before execution
• Ambiguous questions and metric definitions
• Evaluating generated queries against a gold-answer set
• When to use a semantic layer instead

Hands-on lab: Build a small text-to-SQL script over DuckDB with schema context and a read-only guard, then score it on 20 test questions.

Detect bad data early with explicit rules, profiling, and AI-suggested checks.

• Dimensions of quality: completeness, uniqueness, validity, consistency, timeliness
• Profiling datasets with pandas
• Writing expectations with Great Expectations
• Using an assistant to propose checks, then reviewing them
• Anomaly detection basics for metrics
• Alerting and ownership when checks fail

Hands-on lab: Profile a messy dataset, create a suite of at least 12 validation checks, and report which ones fail and why.

Design ingest, transform, and load flows that are testable, observable, and recoverable.

• Batch versus streaming at a conceptual level
• Structuring extract, transform, and load steps
• Idempotency, retries, and logging
• Orchestration concepts with Apache Airflow
• Testing transformations with small fixtures
• Reviewing assistant-generated pipeline code for hidden assumptions

Hands-on lab: Build an idempotent pipeline that loads raw files into a DuckDB table, validates them, and logs row counts at each step.

Prepare data for modeling and train baseline models with sound evaluation habits.

• Encoding categories, scaling, and handling missing values
• Creating date, aggregate, and ratio features
• Train and test splits and data leakage
• Pipelines and preprocessing in scikit-learn
• Baseline models: logistic regression and decision trees
• Choosing metrics: precision, recall, and ROC AUC

Hands-on lab: Engineer features for a churn dataset, train two baseline models in a scikit-learn pipeline, and report metrics while identifying one source of possible leakage.

Turn unstructured text into structured, measurable data using language models.

• Prompting for structured JSON output
• Schema validation of model responses
• Zero-shot versus few-shot classification
• Handling errors, retries, and inconsistent outputs
• Measuring accuracy against a labeled sample
• Batch processing, cost, and privacy considerations

Hands-on lab: Extract fields and topic labels from 50 sample support tickets with an LLM, validate the output schema, and compute accuracy against hand labels.

Present results clearly and responsibly, and bring the course work together.

• Choosing charts that fit the question
• Building interactive dashboards with Streamlit
• AI-generated narrative summaries and how to verify them
• Documenting metric definitions and data lineage
• Communicating uncertainty and data limitations
• Presenting analysis to non-technical stakeholders

Hands-on lab: Build a Streamlit dashboard on your pipeline output and present three verified insights with notes on limitations.