mirror of
https://github.com/acamarata/pray-calc-ml.git
synced 2026-07-01 03:14:27 +00:00
ada08e7ec4
Add 6 new data collection pipelines and their processed outputs: Sources added: - TESS/Stars4All photometer network: 37 months (Jun 2017-Aug 2020), ~40k raw events from 100+ European stations via Zenodo archives - Globe at Night citizen science: 26k twilight observations (2006-2024), filtered from 308k total observations for solar depression 6-22 deg - GaN-MN continuous monitoring: 45 months (Jan 2022-Sep 2025), ~12.5k twilight events from 88 stations across 20+ countries - Galicia SQM network: 14 stations, 1-min resolution, 7.5k events - Madrid/Majadahonda SQM: multi-year continuous monitoring, 3.1k events - washetdonker.nl Netherlands: 7 stations, 3.3k morning events - Academic papers: Jordan (Abed 2015), Fayum Egypt, India photometer Pipeline changes: - ingest.py: add all new files to APPROVED_RAW_CSVS allowlist, fix filter to use allowlist instead of hardcoded exclusions - .gitignore: exclude bulk raw data directories (BSRN, TESS, GaN-MN, washetdonker, Globe at Night downloads) Final dataset: 56,668 Fajr + 34,763 Isha = 91,431 total records Previous: 5,871 Fajr + 46 Isha = 5,917 total records
88 lines
1.6 KiB
Text
88 lines
1.6 KiB
Text
__pycache__/
|
|
*.py[cod]
|
|
*.egg-info/
|
|
.eggs/
|
|
dist/
|
|
build/
|
|
.venv/
|
|
venv/
|
|
env/
|
|
.env
|
|
*.log
|
|
.DS_Store
|
|
.ipynb_checkpoints/
|
|
.jupyter/
|
|
# IDE directories
|
|
.vscode/
|
|
.idea/
|
|
*.suo
|
|
*.user
|
|
*.swp
|
|
*.swo
|
|
|
|
# AI agent working directories
|
|
.claude/
|
|
.codex/
|
|
.cursor/
|
|
.aider/
|
|
.aider.chat.history.md
|
|
.continue/
|
|
.windsurf/
|
|
.gemini/
|
|
.codeium/
|
|
|
|
# Raw scraped/downloaded files
|
|
data/raw/*.pdf
|
|
research/downloaded_papers/
|
|
research/raw_pdfs/
|
|
nohup.out
|
|
|
|
# Generated notebook outputs
|
|
data/processed/*.png
|
|
data/processed/*.svg
|
|
|
|
# Bulk data directories (too large for git, not curated)
|
|
data/cache/
|
|
data/raw/crawled/
|
|
data/raw/excluded/
|
|
data/raw/brin_multistation_raw/
|
|
data/raw/bsrn/
|
|
data/raw/tess/
|
|
data/raw/gan_mn/
|
|
data/raw/washetdonker/
|
|
data/raw/globe_at_night/
|
|
data/raw/surfrad/
|
|
data/raw/galicia_night_sky/
|
|
data/raw/india_twilight_photometer/
|
|
data/raw/madrid_sqm_evol/
|
|
data/raw/majadahonda_sqm_2019/
|
|
|
|
# Collection session logs and scratch
|
|
data/raw/collection_log*.txt
|
|
data/raw/sources_crawled.md
|
|
|
|
# Research scratch files (superseded by aggregate_d0_values.csv)
|
|
research/aggregate_d0_database.csv
|
|
research/aggregate_analysis.md
|
|
research/candidate_papers.json
|
|
research/mine_*.py
|
|
|
|
# Experimental collection scripts (not part of core pipeline)
|
|
src/autonomous_collect.py
|
|
src/compute_aggregate_times.py
|
|
src/collect/aladhan.py
|
|
src/collect/autonomous_collector.py
|
|
src/collect/aggregate_to_records.py
|
|
src/collect/bulk_generator.py
|
|
src/collect/bulk_runner.py
|
|
src/collect/cities.py
|
|
src/collect/collect_agent.py
|
|
src/collect/harvest.py
|
|
src/collect/jakim.py
|
|
src/collect/morocco.py
|
|
src/collect/muis_singapore.py
|
|
src/collect/openalex_harvester.py
|
|
src/collect/source_tracker.py
|
|
src/collect/waktusolat.py
|
|
src/collect/web_harvester.py
|
|
.vscode/*
|