From bdda04705fd302646da94b09ea395eee58f8426a Mon Sep 17 00:00:00 2001 From: kbt-devops Date: Wed, 11 Feb 2026 01:21:39 +0700 Subject: [PATCH] Update README for transmission datasets and mining data changes - Add Transmission Capacity and Transmission Cost to overview - Update mining description to reflect EUR pricing and power metrics - Update script descriptions to include transmission data - Add transmission parameters to configuration section - Update data specifications table with actual values and 2 new datasets --- README.md | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 83f4b43..f7329d4 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,9 @@ This project generates and processes realistic test data for energy trading stra - **Conventional Generation**: Gas, coal, and nuclear plant outputs - **Load Profiles**: Regional electricity demand with weather correlations - **Data Centers**: Power demand profiles including mining client -- **Mining**: Hashrate, price, and profitability data (from mempool.space) +- **Mining**: Hashrate, price (EUR), power efficiency, demand, revenue, and profit per MWh +- **Transmission Capacity**: Region-to-region interconnector limits and efficiency +- **Transmission Cost**: Transmission costs including losses, congestion charges, and fees ## Project Structure @@ -64,9 +66,9 @@ python scripts/04_validate.py ### Individual scripts -**01_generate_synthetic.py**: Creates synthetic data for battery systems, renewable generation, conventional generation, and data centers. +**01_generate_synthetic.py**: Creates synthetic data for battery systems, renewable generation, conventional generation, data centers, and transmission capacity/cost. -**02_fetch_historical.py**: Fetches electricity prices, mining data, and load profiles from public APIs (or generates realistic synthetic data when APIs are unavailable). +**02_fetch_historical.py**: Fetches electricity prices, mining data (with EUR pricing and power metrics), and load profiles from public APIs (or generates realistic synthetic data when APIs are unavailable). **03_process_merge.py**: Merges datasets, optimizes memory usage, and saves to compressed Parquet format. @@ -82,20 +84,23 @@ Edit `config/data_config.yaml` to customize: - **Generation parameters**: Noise levels, outlier rates, missing value rates - **Battery parameters**: Capacity ranges, efficiency, degradation - **Generation parameters**: Plant capacities, marginal costs -- **Mining parameters**: Hashrate ranges, mining efficiency +- **Mining parameters**: Hashrate ranges, power efficiency +- **Transmission parameters**: Capacity ranges, efficiency, congestion surcharges, fees ## Data Specifications -| Dataset | Time Range | Rows (10d × 1min) | Est. Size | -|---------|-----------|-------------------|-----------| -| electricity_prices | 10 days | 72,000 | ~40MB | -| battery_capacity | 10 days | 144,000 | ~20MB | -| renewable_generation | 10 days | 216,000 | ~35MB | -| conventional_generation | 10 days | 144,000 | ~25MB | -| load_profiles | 10 days | 72,000 | ~30MB | -| data_centers | 10 days | 72,000 | ~15MB | -| mining | 10 days | 14,400 | ~20MB | -| **Total** | | | **~185MB** | +| Dataset | Rows | Actual Size | +|---------|------|-------------| +| electricity_prices | 72,005 | ~2.0 MB | +| battery_capacity | 144,010 | ~4.0 MB | +| renewable_generation | 216,015 | ~5.4 MB | +| conventional_generation | 144,010 | ~3.0 MB | +| load_profiles | 72,005 | ~1.7 MB | +| data_centers | 72,005 | ~1.0 MB | +| mining | 14,401 | ~0.5 MB | +| transmission_capacity | 20 | ~0.01 MB | +| transmission_cost | 20 | ~0.01 MB | +| **Total** | **734,491** | **~17.9 MB** | ## Output Format