ELT Mini-Framework

Lightweight, declarative ELT framework with DuckDB destination, TOML specs, and agent skills.

code View on GitHub

⚡ The Prototype // Idea #3

ELT Mini-Framework

Built in 1.5h with Google Antigravity

code View on GitHub

tips_and_updates The 5 Morning Sparks

Based on the inspiration from the Big Data LDN Conference

IDEA #1 lightbulb

Meta-Wiki

Wiki for all the projects managed in one place

Factory.com allow maintaining a meta-wiki for all the projects, besides doing other awesome stuff. I wish I worked there, but they are US based. Anyway, the idea would be to create a plugin into agentic SDLC that supports maintaining a meta-wiki.

Dev Tools, AI

IDEA #2 lightbulb

Token efficient skill stack

Reduce token usage through skills

There are a lot of skills that promise making LLM token usage more efficient, often while maintaining high efficiency. Today I heard about Code-Graph-RAG, which reminded me of another project called treesitter. There is also Ponytail, Caveman, and probably some others. I want a stack based on the latest community benchmaxxed, token efficient skills.

Dev Tools, AI

IDEA #3 lightbulb

ELT Mini-Framework

Data Engineering Framework with small footprint

I want to build a small framework that has few built in tools and a schema for defining spec for building a ELT tool, I imagine being able to define input (lets say API source + parameters), defines the target schema, write disposition, backfill date, natural key, and then uploads it to target (for now DuckDB). It should have very small spec and skills to create it first based on the source, then implement it, and validate it.

Dev Tools, Data SHIPPED

IDEA #4 lightbulb

AFDocs pre-commit hook

Make the docs here better

AF Docs promises making AI suitable documentation. I want a loop to make these docs both human and AI readable with AFDocs evalutation

Dev Tools, AI

IDEA #5 lightbulb

Benchmarking Framework

Data Engineering needs more AI benchmarks for evals

Everything is benchmaxxxed. We need an eval framework for Data Engineering in the industry. I need one for myself. I want as et of tasks in some Project Management framework, so I can test AI + Harness performance on achieving them, with a test suite matching the Acceptance Criteria to evaluate the output on one-shot execution.

Dev Tools, AI, Data

construction

How It Was Built With AI in 1.5 Hours

Step 1

Build an ETL mini pipeline constrained to one source type (API) and one destination (DuckDB) with some nice additions

Step 2

Verify it works as expected in general with new pipelines

Step 3

Ask to install evaluation tools for the skills, run validation for code quality

thumb_up

What Rocked

✓We have an actual working ETL based on a spec and with skills that are validated
✓It worked for new specs after implementation

bug_report

What Broke & Fixed

✗Nothing major, but the skill quality was not perfect.

smart_toy AI Interaction Summary

Genesis Prompt: "/spec-implementation"
Interaction Turns: 8 human turns across 231 execution steps

Key Guidance Turns:
  • Turn 2: This should be in new standalone repositroty. Lets use toml for config. We should have one sample pipeline
  • Turn 3: prceed
  • Turn 4: can you make 2 more API specs based on some free APIs like the openstreetmaps or Elexon BMRS and run it to verify
  • Turn 5: how are the skills? Can we install some dev dependencies to evaluate the skill quality?
  • Turn 6: ok sounds decent. Make a repo for the new codebase
  • Turn 7: /record-implementation Suggest the steps
  • Turn 8: Nah, Okay so lets do steps: 1. Build am ETL mini pipeline constrained to one source type (API) and one destination (DuckDB) wit...

Tools Executed: run_command (60), view_file (20), write_to_file (14), replace_file_content (10), find_by_name (3), ask_question (1)
Files Modified: .gitignore, README.md, SKILL.md, __init__.py, app.py, cli.py, elexon_frequency.toml, evaluate_skills.py