aric/pray-calc-ml
1
0
Fork 0
mirror of https://github.com/acamarata/pray-calc-ml.git synced 2026-07-01 03:14:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
pray-calc-ml/data/raw/raw_sightings
History
Aric Camarata ada08e7ec4 data: expand dataset from 5.9k to 91k records via 6 new SQM sources 
Add 6 new data collection pipelines and their processed outputs:

Sources added:
- TESS/Stars4All photometer network: 37 months (Jun 2017-Aug 2020),
  ~40k raw events from 100+ European stations via Zenodo archives
- Globe at Night citizen science: 26k twilight observations (2006-2024),
  filtered from 308k total observations for solar depression 6-22 deg
- GaN-MN continuous monitoring: 45 months (Jan 2022-Sep 2025),
  ~12.5k twilight events from 88 stations across 20+ countries
- Galicia SQM network: 14 stations, 1-min resolution, 7.5k events
- Madrid/Majadahonda SQM: multi-year continuous monitoring, 3.1k events
- washetdonker.nl Netherlands: 7 stations, 3.3k morning events
- Academic papers: Jordan (Abed 2015), Fayum Egypt, India photometer

Pipeline changes:
- ingest.py: add all new files to APPROVED_RAW_CSVS allowlist,
  fix filter to use allowlist instead of hardcoded exclusions
- .gitignore: exclude bulk raw data directories (BSRN, TESS, GaN-MN,
  washetdonker, Globe at Night downloads)

Final dataset: 56,668 Fajr + 34,763 Isha = 91,431 total records
Previous: 5,871 Fajr + 46 Isha = 5,917 total records
2026-03-22 16:39:29 -04:00
..
abdelhadi_2022_malaysia_sqm.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
abed_2015_jordan.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
brin_multistation_fajr.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
brin_multistation_isha.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
fayum_egypt_2022_sqm.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
galicia_sqm_2015.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_apr_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_apr_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_apr_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_apr_2025.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_aug_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_aug_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_aug_2025.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_august_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_dec_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_dec_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_dec_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_feb_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_feb_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_feb_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_feb_2025.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jan_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jan_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jan_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_january_2025.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jul_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jul_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jul_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jul_2025.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jun_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jun_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_jun_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_june_2025.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_mar_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_mar_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_mar_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_mar_2025.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_may_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_may_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_may_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_may_2025.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_nov_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_nov_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_nov_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_oct_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_oct_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_oct_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_sep_2022.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_sep_2023.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_sep_2024.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
gan_mn_sep_2025.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
globe_at_night_twilight.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
india_twilight_photometer.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
kassim_bahali_2017_malaysia.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
kassim_bahali_2019_ijmet.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
khalifa_2018_saudi_desert.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
madrid_sqm_evol.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
majadahonda_2019_sqm.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
moonsighting_com_sightings.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
shaukat_2015_blackburn_uk.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
shaukat_2015_other_sites.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
tess_apr2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_apr2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_apr2020.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_aug2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_aug2020.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_dec2017.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_dec2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_dec2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_feb2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_feb2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_feb2020.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jan2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jan2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jan2020.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jul2017.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jul2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jul2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jul2020.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jun2017.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jun2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jun2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_jun2020.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_mar2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_mar2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_mar2020.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_may2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_may2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_may2020.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_nov2017.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_nov2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_nov2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_oct2017.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_oct2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_oct2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_sep2017.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_sep2018.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
tess_sep2019.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00
walisongo_2022_sulawesi_sqm.csv Expand dataset to 5,871 Fajr / 46 Isha across 114 locations 2026-02-28 10:51:01 -05:00
washetdonker_morning.csv data: expand dataset from 5.9k to 91k records via 6 new SQM sources 2026-03-22 16:39:29 -04:00