Aric Camarata 9ea537d26d fix: add quality filters and fix 8 data integrity issues from CR/QA ...

Issues fixed:
- Add upper bound angle filter: fajr/isha capped at 22 deg (was unbounded,
  max was 49 deg from light pollution artifacts)
- Remove washetdonker from approved list: threshold method produces 7 deg
  angles (civil twilight), not Fajr at 12-18 deg
- Remove openfajr_94992898.csv: duplicated the iCal feed data with slightly
  different coordinates, bypassing dedup (4,007 duplicate records)
- Filter out future dates: OpenFajr publishes predictions for the full year
- Filter out polar stations (|lat| > 70): no meaningful Fajr/Isha
- Filter out Null Island (lat=0, lng=0): GPS default / missing coordinates
- Move precomputed angles merge before dedup: was bypassing dedup entirely
- Make BAD_NOTE_MARKERS case-insensitive: catches mixed-case variants
- Add missing tess_jun2017.csv to approved list
- Clean up duplicate comment blocks in ingest.py

Dataset after fixes: 48,668 Fajr + 34,529 Isha = 83,197 total
Angle range now: 7.0-22.0 deg (Fajr), 10.0-22.0 deg (Isha)
Latitude range now: -62.6 to 69.7 (was -90 to 90)

2026-03-23 04:25:16 -04:00

9.5 MiB

Raw Permalink Blame History

The file is too large to be shown.