Power Desk
Minimalist analyst dashboard for UK power & energy data with DuckDB, dlt ELT pipeline, and DAG lineage.
Power Desk
tips_and_updates The 5 Morning Sparks
Projects to do with Energy DataDuckDB hoard
Store a bunch of data from open sources in DuckDB and explore it using notebook-embedded skills. I want to use a good modern notebook + AI solution to do this. The idea is to build a lightweight DBT catalog and a full pipeline project. Maybe starting with Elexon BMRS data, and energy price data (UK) building vocabulary, etc. over the data and getting some visualisations
Map of electric power plants
I want to see Power plants in Europe + UK on a map. It would be good to get a clear representation, some way of representing important dimensions, like production volumes, type, active/inactive, etc. in an intuitive, visual way that is clear, not too busy (think clustering points that overlap, unclustering when zooming in)
Weather and renewable production
Weather affects renewables and their production. I want a model that given weather in various places, can predict the production based on historical data. We want to probably create a dataset from weather data and renewable production data with some geo-location specific data, and then do the classic ML stuff to predict the future (with validation dataset). Would be good to understand feature importance. Optimiser too
Energy price compare
Do this over time, for the UK market
Energy trading primer
An Energy trading guide, with vivid examples, condensed from authoritative resources. It should be clear, with some examples of what types of strategies are possible, with a possible simulation of outcomes, to show the expectation for each different one
How It Was Built With AI in 4.0 Hours
Spec Refinement
Did few rounds of spec refinement to inject some of my preexisting knoledge and avoid generic look
Dashboard Creation
One shot created a dashboards, but the data was for 2024 for some reason
ELT Refinement
Refined the ELT bit of the codebase, to backfill the data
Schema Debugging
Few rounds of debugging schema issues. All good in the end
What Rocked
- ✓Genuine one shot of a working PoC on a quite complicated, multi-fasceted task
What Broke & Fixed
- ✗Not so unique, this design. Some color issues. Some data never gets materialised in marts
smart_toy AI Interaction Summary
Genesis Prompt: "/spec-implementation" Interaction Turns: 11 human turns across 517 execution steps Key Guidance Turns: • Turn 2: the design should be minimalist analyst dashboard. Don't use this repo. Create a new repo in parent folder, named 'power-desk' ... • Turn 3: for the dlt, I want the raw data values to be preserved, no custom row processing. That happens in DBT. It should be incrementa... • Turn 4: ok that sounds good. Make sure to record the progress in an MD planning file, so you don't get lost (since this task is best to... • Turn 5: how do I run a dlt backfill? • Turn 6: the source should be incremental, which means it should run until the data ends and store the pointer to the latest value (for ... • Turn 7: dlt cannot update tables ['_dlt_version', 'raw_demand_forecast', 'raw_fuelinst', 'raw_itsdo', 'raw_windfor', '_dlt_pipeline_sta... • Turn 8: can we do full refresh for the new schema then? • Turn 9: doesn't dlt just have a --refresh drop_source option for the pipelines? • Turn 10: /record-implementation • Turn 11: Don't change these: 1. * Did few rounds of spec refinement to inject some of my preexisting knoledge and avoid generic look * O... Tools Executed: run_command (136), view_file (43), write_to_file (38), replace_file_content (20), find_by_name (3), ask_question (2) Files Modified: .gitignore, .python-version, PLAN.md, README.md, SKILL.md, __init__.py, app.py, backfill.py