pray-calc-ml/data/raw/raw_sightings/fayum_egypt_2022_sqm.csv at ad8f1c4bc7f09ea9e47d12f1cb53442e771b7708

mirror of https://github.com/acamarata/pray-calc-ml.git synced 2026-07-01 11:24:26 +00:00

Aric Camarata ada08e7ec4 data: expand dataset from 5.9k to 91k records via 6 new SQM sources

Add 6 new data collection pipelines and their processed outputs:

Sources added:
- TESS/Stars4All photometer network: 37 months (Jun 2017-Aug 2020),
  ~40k raw events from 100+ European stations via Zenodo archives
- Globe at Night citizen science: 26k twilight observations (2006-2024),
  filtered from 308k total observations for solar depression 6-22 deg
- GaN-MN continuous monitoring: 45 months (Jan 2022-Sep 2025),
  ~12.5k twilight events from 88 stations across 20+ countries
- Galicia SQM network: 14 stations, 1-min resolution, 7.5k events
- Madrid/Majadahonda SQM: multi-year continuous monitoring, 3.1k events
- washetdonker.nl Netherlands: 7 stations, 3.3k morning events
- Academic papers: Jordan (Abed 2015), Fayum Egypt, India photometer

Pipeline changes:
- ingest.py: add all new files to APPROVED_RAW_CSVS allowlist,
  fix filter to use allowlist instead of hardcoded exclusions
- .gitignore: exclude bulk raw data directories (BSRN, TESS, GaN-MN,
  washetdonker, Globe at Night downloads)

Final dataset: 56,668 Fajr + 34,763 Isha = 91,431 total records
Previous: 5,871 Fajr + 46 Isha = 5,917 total records

2026-03-22 16:39:29 -04:00

759 B

Raw Blame History

1	prayer	date_local	time_local	utc_offset	lat	lng	elevation_m	source	notes
2	fajr	2018-12-09	05:05	2.0	29.283	30.05	50	Rashed et al. 2022, IJMET 13(10), Wadi al Hitan Fayum Egypt	SQM measurement; eye threshold 0.83mag from full night gives D0=14.6; time estimated from D0=14.6 at this location/date; clear night
3	fajr	2018-12-10	05:04	2.0	29.283	30.05	50	Rashed et al. 2022, IJMET 13(10), Wadi al Hitan Fayum Egypt	SQM measurement; mesopic threshold 0.015cd/m2 gives D0=13.75; clear night; time estimated from D0=13.75
4	fajr	2019-12-19	05:08	2.0	29.283	30.05	50	Rashed et al. 2022, IJMET 13(10), Wadi al Hitan Fayum Egypt	SQM H+Z measurement; Kocifaj H=piZ gives D0=14.0; naked eye group D0=14.8; time estimated from D0=14.0 (conservative); clear night

759 B Raw Blame History

759 B

Raw Blame History