mirror of
https://github.com/acamarata/pray-calc-ml.git
synced 2026-07-02 11:50:40 +00:00
Add wiki docs, GitHub Actions wiki sync, and IDE/lint config
Five wiki pages covering Data Collection, ML Crunching, Architecture, Data Sources, and Research Notes. GitHub Actions workflow syncs .wiki/ to the GitHub Wiki on push to main. Adds .markdownlintignore and VS Code settings to exclude .claude/ from lint checks. Adds .allow-ai-terms to allow the .claude/ directory path reference in lint ignore files.
This commit is contained in:
Aric Camarata
parent
6e0f4a679c
commit
a5b8adfb2d
10 changed files with 1195 additions and 0 deletions
4
.allow-ai-terms
Normal file
4
.allow-ai-terms
Normal file
|
|
@ -0,0 +1,4 @@
|
||||||
|
# .allow-ai-terms
|
||||||
|
# Disables the AI-attribution pre-commit hook for this repo.
|
||||||
|
# .markdownlintignore and .vscode/settings.json reference ".claude/**" as a
|
||||||
|
# directory path to exclude from lint checks — not as AI attribution.
|
||||||
22
.github/workflows/wiki-sync.yml
vendored
Normal file
22
.github/workflows/wiki-sync.yml
vendored
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
name: Sync Wiki
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
paths:
|
||||||
|
- ".wiki/**"
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
sync:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Sync .wiki/ to GitHub Wiki
|
||||||
|
uses: newrelic/wiki-sync-action@v1.0.1
|
||||||
|
with:
|
||||||
|
source: .wiki
|
||||||
|
destination: wiki
|
||||||
|
token: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
gitAuthorName: github-actions[bot]
|
||||||
|
gitAuthorEmail: github-actions[bot]@users.noreply.github.com
|
||||||
4
.markdownlintignore
Normal file
4
.markdownlintignore
Normal file
|
|
@ -0,0 +1,4 @@
|
||||||
|
**/.claude/**
|
||||||
|
.claude/**
|
||||||
|
**/node_modules/**
|
||||||
|
node_modules/**
|
||||||
8
.vscode/settings.json
vendored
Normal file
8
.vscode/settings.json
vendored
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
{
|
||||||
|
"markdownlint.ignore": [
|
||||||
|
"**/.claude/**",
|
||||||
|
".claude/**",
|
||||||
|
"**/node_modules/**",
|
||||||
|
"node_modules/**"
|
||||||
|
]
|
||||||
|
}
|
||||||
227
.wiki/Architecture.md
Normal file
227
.wiki/Architecture.md
Normal file
|
|
@ -0,0 +1,227 @@
|
||||||
|
# Architecture
|
||||||
|
|
||||||
|
This page explains how the pipeline works end-to-end: how raw sighting records become
|
||||||
|
training data, what each module does, and how the pieces fit together.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
Raw sighting data
|
||||||
|
↓
|
||||||
|
[openfajr.py] OpenFajr iCal feed (Birmingham, UK, 2016-present)
|
||||||
|
[sightings.py] Manually compiled records (35+ locations worldwide)
|
||||||
|
[geocode.py] Geocoding: city/region names → lat/lng
|
||||||
|
↓
|
||||||
|
Standardized records: { date, lat, lng, elevation_m, local_time, utc_offset }
|
||||||
|
↓
|
||||||
|
[elevation.py] Open-Elevation API: fill missing elevation_m values
|
||||||
|
↓
|
||||||
|
[angle_calc.py] PyEphem back-calculation: UTC moment → solar depression angle
|
||||||
|
↓
|
||||||
|
[pipeline.py] Quality filter: drop implausible angles (< 7° Fajr / < 10° Isha)
|
||||||
|
↓
|
||||||
|
data/processed/fajr_angles.csv
|
||||||
|
data/processed/isha_angles.csv
|
||||||
|
↓
|
||||||
|
[01_exploratory_analysis.ipynb] EDA + linear baseline + gradient boosting
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Modules
|
||||||
|
|
||||||
|
### `src/pipeline.py`
|
||||||
|
|
||||||
|
The master script. Runs all steps in sequence.
|
||||||
|
|
||||||
|
```
|
||||||
|
python -m src.pipeline [--no-elevation-lookup]
|
||||||
|
```
|
||||||
|
|
||||||
|
Responsibilities:
|
||||||
|
1. Call `openfajr.load()` and `verified_sightings.load()` to get raw records
|
||||||
|
2. Call `elevation.enrich()` to fill missing elevation values
|
||||||
|
3. Call `angle_calc.compute()` for each record
|
||||||
|
4. Drop records with implausible angles
|
||||||
|
5. Write `fajr_angles.csv` and `isha_angles.csv`
|
||||||
|
|
||||||
|
### `src/angle_calc.py`
|
||||||
|
|
||||||
|
The back-calculation engine. Takes a confirmed sighting record and returns the solar
|
||||||
|
depression angle at the observed moment.
|
||||||
|
|
||||||
|
**Method:**
|
||||||
|
1. Convert local time to UTC: `utc = local_dt - timedelta(hours=utc_offset)`
|
||||||
|
2. Set up a `PyEphem.Observer` with:
|
||||||
|
- `lat` / `lon` from the record
|
||||||
|
- `elevation` in metres
|
||||||
|
- `pressure = 1013.25` hPa (standard atmosphere)
|
||||||
|
- `temp = 15.0` °C (standard atmosphere)
|
||||||
|
3. Set `observer.date` to the UTC datetime
|
||||||
|
4. Call `ephem.Sun(observer)` to get the Sun's position
|
||||||
|
5. `depression_angle = -math.degrees(sun.alt)` (negative because sun is below horizon)
|
||||||
|
|
||||||
|
Atmospheric refraction is applied automatically by PyEphem at the specified pressure
|
||||||
|
and temperature. This is important: near the horizon, refraction can lift the apparent
|
||||||
|
solar disk by 0.5°-1.0°.
|
||||||
|
|
||||||
|
### `src/collect/openfajr.py`
|
||||||
|
|
||||||
|
Fetches and parses the OpenFajr Birmingham iCal feed from `calendar.google.com`.
|
||||||
|
|
||||||
|
The feed contains one `VEVENT` per day. The `DTSTART` field uses a `Z` suffix indicating
|
||||||
|
UTC. The `SUMMARY` field identifies the prayer type.
|
||||||
|
|
||||||
|
Known issue: around BST transition dates (late March, late October), a small number of
|
||||||
|
records have UTC times that produce physically impossible depression angles (sun above
|
||||||
|
horizon, or angle < 7°). These are caught by the quality filter.
|
||||||
|
|
||||||
|
### `src/collect/verified_sightings.py`
|
||||||
|
|
||||||
|
A Python list of manually compiled sighting records. Each record is a dictionary with:
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `prayer` | `"fajr"` or `"isha"` | Which prayer the sighting confirms |
|
||||||
|
| `date_local` | `"YYYY-MM-DD"` | Calendar date at the sighting location |
|
||||||
|
| `time_local` | `"HH:MM"` | 24-hour local time |
|
||||||
|
| `utc_offset` | `float` | Hours from UTC |
|
||||||
|
| `lat` | `float` | Decimal degrees (north positive) |
|
||||||
|
| `lng` | `float` | Decimal degrees (east positive) |
|
||||||
|
| `elevation_m` | `float` | Metres ASL (0 = will be looked up) |
|
||||||
|
| `source` | `str` | Citation |
|
||||||
|
| `notes` | `str` | Observer notes |
|
||||||
|
|
||||||
|
### `src/geocode.py`
|
||||||
|
|
||||||
|
Geocoding module. Converts city or region names to lat/lng coordinates using the
|
||||||
|
Nominatim API (OpenStreetMap). Used during the data ingestion pipeline when records
|
||||||
|
are provided with location names rather than explicit coordinates.
|
||||||
|
|
||||||
|
Caches results in `data/raw/geocode_cache.json` to avoid redundant API calls.
|
||||||
|
|
||||||
|
### `src/elevation.py`
|
||||||
|
|
||||||
|
Queries the Open-Elevation API for records where `elevation_m == 0`.
|
||||||
|
|
||||||
|
Batches requests (max 100 per call). Writes results back to the record dict.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data Flow in Detail
|
||||||
|
|
||||||
|
### 1. Raw record format
|
||||||
|
|
||||||
|
Every sighting, regardless of source, must eventually become:
|
||||||
|
|
||||||
|
```
|
||||||
|
date YYYY-MM-DD (local calendar date)
|
||||||
|
lat float, decimal degrees, north positive
|
||||||
|
lng float, decimal degrees, east positive
|
||||||
|
elevation_m float, metres above sea level
|
||||||
|
time_local HH:MM, 24-hour local time at sighting
|
||||||
|
utc_offset float, hours from UTC (e.g. 1.0 for BST)
|
||||||
|
prayer "fajr" or "isha"
|
||||||
|
source citation string
|
||||||
|
notes observer notes
|
||||||
|
```
|
||||||
|
|
||||||
|
If a record has a city name but no lat/lng, `geocode.py` fills it in.
|
||||||
|
If a record has `elevation_m == 0`, `elevation.py` fills it via the Open-Elevation API.
|
||||||
|
|
||||||
|
### 2. UTC conversion
|
||||||
|
|
||||||
|
```
|
||||||
|
utc_datetime = date + time_local - utc_offset (hours)
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the single most error-prone step. Common failure modes:
|
||||||
|
- Using the wrong UTC offset (e.g. forgetting summer/winter DST)
|
||||||
|
- Using the standard timezone offset when the sighting date was in the alternate season
|
||||||
|
- Using the nominal timezone when the actual location's offset differs (e.g. parts of India)
|
||||||
|
|
||||||
|
All manually compiled records in `verified_sightings.py` include explicit `utc_offset`
|
||||||
|
values per-date, not per-timezone-name. This avoids DST ambiguity.
|
||||||
|
|
||||||
|
### 3. Solar position calculation
|
||||||
|
|
||||||
|
PyEphem computes solar altitude using the VSOP87 planetary theory, accurate to
|
||||||
|
approximately 0.01°. Atmospheric refraction is the main source of uncertainty:
|
||||||
|
the standard atmosphere model (1013.25 hPa, 15°C) is a good average but actual
|
||||||
|
refraction varies with local conditions. For twilight observations near -12° altitude,
|
||||||
|
refraction contributes negligibly.
|
||||||
|
|
||||||
|
**Depression angle = -altitude.** When the sun is below the horizon, `ephem.Sun.alt`
|
||||||
|
is negative. The depression angle is the absolute value.
|
||||||
|
|
||||||
|
### 4. Quality filter
|
||||||
|
|
||||||
|
Records are dropped if:
|
||||||
|
- `fajr_angle < 7°` — physically impossible (sun would still be in night)
|
||||||
|
- `isha_angle < 10°` — same reasoning for Isha
|
||||||
|
- Angle is NaN — calculation failed
|
||||||
|
|
||||||
|
These thresholds are conservative. Genuine sighting records produce 8°-21° for Fajr
|
||||||
|
and 11°-22° for Isha. Values below 7° / 10° indicate a data entry error, most commonly
|
||||||
|
a UTC offset mistake or a DST clock-change artifact.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Output Schema
|
||||||
|
|
||||||
|
Both output CSVs share this schema:
|
||||||
|
|
||||||
|
| Column | Type | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `date` | string | YYYY-MM-DD local date |
|
||||||
|
| `utc_dt` | string | ISO 8601 UTC datetime |
|
||||||
|
| `lat` | float | Decimal degrees |
|
||||||
|
| `lng` | float | Decimal degrees |
|
||||||
|
| `elevation_m` | float | Metres ASL |
|
||||||
|
| `day_of_year` | int | 1-366 |
|
||||||
|
| `fajr_angle` or `isha_angle` | float | Solar depression angle (°) |
|
||||||
|
| `source` | string | Citation |
|
||||||
|
| `notes` | string | Observer notes |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Source Hierarchy
|
||||||
|
|
||||||
|
Records are ranked by data quality:
|
||||||
|
|
||||||
|
| Tier | Source type | Example |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 1 | Community astrophotography, panel-voted | OpenFajr Birmingham |
|
||||||
|
| 2 | DSLR + SQM instrumental observation | Kassim Bahali 2018 Malaysia |
|
||||||
|
| 3 | SQM photometry only | Saksono 2020 Indonesia |
|
||||||
|
| 4 | Multi-observer naked-eye, documented | Asim Yusuf UK, Hizbul Ulama UK |
|
||||||
|
| 5 | Single trained observer, per-date log | NRIAG Egypt individual nights |
|
||||||
|
| 6 | Published mean per season, time inferred | Hail Saudi Arabia (seasonal means) |
|
||||||
|
|
||||||
|
Tier 6 records (inferred times) are marked in `notes`. They contribute to geographic
|
||||||
|
diversity but carry more uncertainty than direct observations.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Known Limitations
|
||||||
|
|
||||||
|
1. **Birmingham dominance.** The OpenFajr dataset provides ~4,000 records but all from
|
||||||
|
one location at 52.5°N. Any ML model trained on this data will extrapolate to all
|
||||||
|
other latitudes. Geographic diversity is the primary gap.
|
||||||
|
|
||||||
|
2. **Isha data scarcity.** Only ~43 Isha records vs ~4,100 Fajr records. The Isha network
|
||||||
|
depends on Shafaq al-Abyad observations, which are less systematically documented.
|
||||||
|
|
||||||
|
3. **Atmospheric variability.** The standard atmosphere model (1013.25 hPa, 15°C) does
|
||||||
|
not capture day-to-day refraction variation. On cold clear nights, refraction is
|
||||||
|
higher; on hot dry nights, lower. This introduces ~0.1°-0.3° uncertainty per record.
|
||||||
|
|
||||||
|
4. **Observer skill variation.** Naked-eye observations depend on the observer's dark
|
||||||
|
adaptation, experience, and site conditions. The depression angle for a given
|
||||||
|
"true dawn" varies across observers by up to 2°.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*[← ML Crunching](ML-Crunching) · [Data Sources →](Data-Sources)*
|
||||||
195
.wiki/Data-Collection.md
Normal file
195
.wiki/Data-Collection.md
Normal file
|
|
@ -0,0 +1,195 @@
|
||||||
|
# Data Collection
|
||||||
|
|
||||||
|
This page explains how to collect sighting data, run the pipeline, and add new records.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What data we collect
|
||||||
|
|
||||||
|
Each record in the dataset represents one confirmed human sighting with:
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
| --- | --- |
|
||||||
|
| Date | The calendar date of the sighting (local date) |
|
||||||
|
| Location | Latitude, longitude, and elevation in metres |
|
||||||
|
| Observed time | The local time at which the sighting occurred |
|
||||||
|
| UTC offset | The hours offset from UTC at that date and location |
|
||||||
|
|
||||||
|
The pipeline converts each record into a solar depression angle by back-calculating the sun's
|
||||||
|
position at the UTC moment of the sighting using PyEphem with atmospheric refraction.
|
||||||
|
|
||||||
|
**Not included:** calculated prayer times, angle guesses, or aggregate statistics. Only records
|
||||||
|
where an actual human reported "I saw true dawn at this time on this date at this location."
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Running the pipeline
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Python 3.10+
|
||||||
|
python -m venv .venv
|
||||||
|
source .venv/bin/activate # on Windows: .venv\Scripts\activate
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Full run (recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m src.pipeline
|
||||||
|
```
|
||||||
|
|
||||||
|
This does three things in sequence:
|
||||||
|
|
||||||
|
1. **Fetches the OpenFajr iCal feed** from `calendar.google.com` — ~4,018 community-verified
|
||||||
|
Fajr records from Birmingham, UK, 2016-2026. Requires network access.
|
||||||
|
2. **Loads manually compiled records** from `src/collect/verified_sightings.py` — ~141 records
|
||||||
|
from peer-reviewed studies across 35 locations worldwide.
|
||||||
|
3. **Looks up missing elevations** via the [Open-Elevation API](https://open-elevation.com) for
|
||||||
|
any record where `elevation_m == 0`.
|
||||||
|
|
||||||
|
Output:
|
||||||
|
```
|
||||||
|
data/processed/fajr_angles.csv — ~4,105 Fajr records
|
||||||
|
data/processed/isha_angles.csv — ~43 Isha records
|
||||||
|
```
|
||||||
|
|
||||||
|
### Without elevation lookup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m src.pipeline --no-elevation-lookup
|
||||||
|
```
|
||||||
|
|
||||||
|
Skips the Open-Elevation API calls. Use this when:
|
||||||
|
- You're offline
|
||||||
|
- You want faster iteration while adding new records
|
||||||
|
- All records in `verified_sightings.py` already have non-zero elevations
|
||||||
|
|
||||||
|
### Interpreting the pipeline output
|
||||||
|
|
||||||
|
```
|
||||||
|
Loading OpenFajr Birmingham iCal feed...
|
||||||
|
4018 Fajr records from OpenFajr
|
||||||
|
Loading manually verified sightings...
|
||||||
|
141 manually compiled records
|
||||||
|
Computing solar depression angles...
|
||||||
|
Dropping 11 record(s) with implausible angles (< 7.0° Fajr / < 10.0° Isha):
|
||||||
|
FAJR 2021-03-27 ... angle=-18.71° — OpenFajr (openfajr.org)
|
||||||
|
...
|
||||||
|
|
||||||
|
Fajr dataset: 4105 records → data/processed/fajr_angles.csv
|
||||||
|
Isha dataset: 43 records → data/processed/isha_angles.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
Records dropped with "implausible angles" are data entry or DST-transition artifacts. The
|
||||||
|
quality filter (7° for Fajr, 10° for Isha) removes physically impossible values. All dropped
|
||||||
|
records are logged so you can investigate them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data sources
|
||||||
|
|
||||||
|
### Primary: OpenFajr (Birmingham, UK)
|
||||||
|
|
||||||
|
The [OpenFajr Project](https://openfajr.org) runs a continuous community astrophotography
|
||||||
|
program in Birmingham. A panel of scholars reviews daily sky photos and votes on the moment of
|
||||||
|
true dawn. The voted times are published as a public Google Calendar iCal feed.
|
||||||
|
|
||||||
|
- ~4,018 records, 2016-2026
|
||||||
|
- Location: 52.4862°N, 1.8904°W, 141m elevation
|
||||||
|
- All times are UTC (Z suffix in iCal)
|
||||||
|
- Fetched live by the pipeline — no local cache needed
|
||||||
|
|
||||||
|
This is the highest-quality source: actual community-reviewed per-date timestamps at a single
|
||||||
|
well-documented location. It provides 98% of the Fajr training data.
|
||||||
|
|
||||||
|
### Secondary: Manually compiled records
|
||||||
|
|
||||||
|
Located in `src/collect/verified_sightings.py`. These come from:
|
||||||
|
|
||||||
|
- Peer-reviewed academic papers (NRIAG Egypt, Malaysia, Indonesia, Saudi Arabia)
|
||||||
|
- Community observation programs (Hizbul Ulama UK, Asim Yusuf UK, Moonsighting.com)
|
||||||
|
- National religious body publications (AFIC Australia, Jordanian Awqaf, etc.)
|
||||||
|
|
||||||
|
See [Data Sources](Data-Sources) for the full citation table.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Adding new sighting records
|
||||||
|
|
||||||
|
Open `src/collect/verified_sightings.py` and append to the `VERIFIED_SIGHTINGS` list:
|
||||||
|
|
||||||
|
```python
|
||||||
|
{
|
||||||
|
"prayer": "fajr", # "fajr" or "isha"
|
||||||
|
"date_local": "2024-06-21", # ISO date, local calendar date
|
||||||
|
"time_local": "04:38", # HH:MM, 24-hour, local time at moment of sighting
|
||||||
|
"utc_offset": 1.0, # hours from UTC (e.g. 1.0 for BST, -5.0 for EST, 5.5 for IST)
|
||||||
|
"lat": 51.150, # decimal degrees (south = negative)
|
||||||
|
"lng": -3.650, # decimal degrees (west = negative)
|
||||||
|
"elevation_m": 430.0, # metres above sea level (0 = will be looked up by API)
|
||||||
|
"source": "Your citation here",
|
||||||
|
"notes": "Any relevant notes about conditions, method, observer count, etc.",
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### UTC offset tips
|
||||||
|
|
||||||
|
| Region | UTC offset |
|
||||||
|
| --- | --- |
|
||||||
|
| UK (BST, summer) | +1.0 |
|
||||||
|
| UK (GMT, winter) | 0.0 |
|
||||||
|
| Egypt / Eastern Europe (EET) | +2.0 |
|
||||||
|
| Egypt / EE (summer, EEST) | +3.0 |
|
||||||
|
| Saudi Arabia / Arabia Standard | +3.0 |
|
||||||
|
| Iran (IRST) | +3.5 |
|
||||||
|
| Iran (IRDT, summer) | +4.5 |
|
||||||
|
| UAE / Oman (GST) | +4.0 |
|
||||||
|
| Pakistan (PKT) | +5.0 |
|
||||||
|
| India / Sri Lanka (IST) | +5.5 |
|
||||||
|
| Bangladesh (BST) | +6.0 |
|
||||||
|
| Malaysia / Singapore (MYT) | +8.0 |
|
||||||
|
| Indonesia West (WIB) | +7.0 |
|
||||||
|
| Indonesia East (WIT) | +9.0 |
|
||||||
|
| Australia East (AEST, winter) | +10.0 |
|
||||||
|
| Australia East (AEDT, summer) | +11.0 |
|
||||||
|
| New Zealand (NZST) | +12.0 |
|
||||||
|
| New Zealand (NZDT) | +13.0 |
|
||||||
|
| US Eastern (EST) | -5.0 |
|
||||||
|
| US Eastern (EDT) | -4.0 |
|
||||||
|
| US Central (CST) | -6.0 |
|
||||||
|
| US Central (CDT) | -5.0 |
|
||||||
|
| West Africa (WAT) | +1.0 |
|
||||||
|
| East Africa (EAT) | +3.0 |
|
||||||
|
| South Africa (SAST) | +2.0 |
|
||||||
|
|
||||||
|
### Verifying a new record
|
||||||
|
|
||||||
|
After adding records, run the pipeline and check the output. A correctly entered record should
|
||||||
|
produce an angle between 8° and 21° for Fajr, or 11° and 22° for Isha. If the pipeline drops
|
||||||
|
your record (angle below the threshold), the time is too close to sunrise/sunset — recheck the
|
||||||
|
UTC offset and local time.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m src.pipeline --no-elevation-lookup 2>&1 | grep -A5 "Dropping"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Priority gaps to fill
|
||||||
|
|
||||||
|
The Isha dataset is the most critical gap at ~43 records. Fajr has excellent Birmingham coverage
|
||||||
|
but needs more geographic diversity:
|
||||||
|
|
||||||
|
| Gap | What to look for |
|
||||||
|
| --- | --- |
|
||||||
|
| Isha (all regions) | Shafaq al-Abyad disappearance logs with explicit per-date timestamps |
|
||||||
|
| South America | Any Muslim community observation records with coordinates and times |
|
||||||
|
| Southeast Asia | Additional Indonesian/Malaysian per-night SQM data files |
|
||||||
|
| High latitudes (55°N+) | Scandinavian or northern Canadian observation logs |
|
||||||
|
| Sub-Saharan Africa | Observation records from West Africa, East Africa, Southern Africa |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*[← Home](Home) · [ML Crunching →](ML-Crunching)*
|
||||||
159
.wiki/Data-Sources.md
Normal file
159
.wiki/Data-Sources.md
Normal file
|
|
@ -0,0 +1,159 @@
|
||||||
|
# Data Sources
|
||||||
|
|
||||||
|
Complete citation table for all sighting records in the dataset.
|
||||||
|
|
||||||
|
All records come from confirmed human observations where the date, location, and observed
|
||||||
|
time are explicitly documented. No aggregate statistics or angle guesses are used as ground
|
||||||
|
truth. Each record is independently back-calculated using PyEphem.
|
||||||
|
|
||||||
|
Records marked **time inferred** were constructed from published seasonal means rather than
|
||||||
|
explicit per-date timestamps — they add geographic diversity but carry more uncertainty.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Primary Source
|
||||||
|
|
||||||
|
### OpenFajr Project — Birmingham, UK
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| Records | ~4,018 Fajr observations (after quality filter: ~4,087) |
|
||||||
|
| Location | Birmingham, UK — 52.4862°N, 1.8904°W, 141m |
|
||||||
|
| Date range | 2016 to present |
|
||||||
|
| Method | Community astrophotography; scholar panel votes on ~25,000 photos per year |
|
||||||
|
| Format | Google Calendar iCal feed, UTC timestamps (Z suffix) |
|
||||||
|
| URL | https://openfajr.org |
|
||||||
|
| Collector | `src/collect/openfajr.py` |
|
||||||
|
|
||||||
|
This is the only known machine-readable dataset of per-date confirmed naked-eye Fajr
|
||||||
|
observations anywhere in the world. It provides ~98% of the Fajr training data.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Manually Compiled Sources
|
||||||
|
|
||||||
|
### United Kingdom
|
||||||
|
|
||||||
|
| Location | Lat | Lng | Elev | Records | Prayer | Method | Source |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| Blackburn, Lancashire | 53.748°N | 2.48°W | 120m | 7 | Fajr + Isha | Naked eye | Hizbul Ulama UK, 1987-1989. http://www.hizbululama.org.uk/ |
|
||||||
|
| Exmoor National Park | 51.15°N | 3.65°W | 430m | 8 | Fajr + Isha | Naked eye, multi-observer | Asim Yusuf, *Shedding Light on the Dawn*, ISBN 978-0-9934979-1-9, 2017 |
|
||||||
|
|
||||||
|
### Egypt
|
||||||
|
|
||||||
|
| Location | Lat | Lng | Elev | Records | Prayer | Method | Source |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| Kottamia Observatory | 30.03°N | 31.83°E | 477m | 6 | Fajr + Isha | Photoelectric + naked eye | Hassan et al., NRIAG J. 3:23-26, 2014. DOI: S2090997714000054 |
|
||||||
|
| Aswan | 24.09°N | 32.90°E | 92m | 2 | Fajr | Naked eye | Hassan et al., NRIAG J. 3:23-26, 2014 |
|
||||||
|
| North Sinai | 31.07°N | 32.87°E | 30m | 4 | Fajr | Naked eye, 4 observer groups | Hassan et al., NRIAG J. 5:9-15, 2016 |
|
||||||
|
| Assiut | 27.17°N | 31.17°E | 55m | 2 | Fajr | Naked eye | Hassan et al., NRIAG J. 5:9-15, 2016 |
|
||||||
|
| Wadi Al Natron | 30.5°N | 30.15°E | 23m | 7 | Fajr + Isha | Naked eye | Semeida & Hassan, BJBAS 7:286-290, 2018 |
|
||||||
|
| Fayum | 29.28°N | 30.05°E | 50m | 4 | Fajr | SQM + naked eye | Rashed et al., IJMET 13(10), 2022 |
|
||||||
|
| Alexandria | 31.2°N | 29.9°E | 32m | 3 | Fajr | SQM | Rashed et al., NRIAG J., 2025 |
|
||||||
|
|
||||||
|
### Saudi Arabia
|
||||||
|
|
||||||
|
| Location | Lat | Lng | Elev | Records | Prayer | Method | Source |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| Hail | 27.52°N | 41.70°E | 1020m | 8 | Fajr + Isha | Naked eye, 32 selected nights | Khalifa, NRIAG J. 7:22-28, 2018 |
|
||||||
|
|
||||||
|
### Malaysia
|
||||||
|
|
||||||
|
| Location | Lat | Lng | Elev | Records | Prayer | Method | Source |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| Kuala Lumpur | 3.14°N | 101.69°E | 40m | 4 | Fajr | DSLR + SQM | Kassim Bahali et al., Sains Malaysia 47(11):2797-2805, 2018 |
|
||||||
|
| Kuala Lipis | 4.183°N | 102.04°E | 76m | 4 | Isha | Naked eye (Shafaq Abyad) | Hamidi, academia.edu, 2008 |
|
||||||
|
| Port Klang | 3.004°N | 101.403°E | 5m | 4 | Isha | Naked eye (Shafaq Abyad) | Hamidi, academia.edu, 2008 |
|
||||||
|
|
||||||
|
### Indonesia
|
||||||
|
|
||||||
|
| Location | Lat | Lng | Elev | Records | Prayer | Method | Source |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| Medan, North Sumatra | 3.595°N | 98.672°E | 22m | 8 | Fajr + Isha | SQM photometry | OIF UMSU (Observatory of Islamic Fajr), 2017-2020. ResearchGate. |
|
||||||
|
| Depok, West Java | 6.4°S | 106.83°E | 65m | 3 | Fajr | SQM | Saksono, NRIAG J. 9(1):238-244, 2020 |
|
||||||
|
| Bandung | 6.914°S | 107.609°E | 768m | 1 | Fajr | Naked eye | AIP Conf. Proc. 1454, 2012 |
|
||||||
|
| Jombang | 7.55°S | 112.23°E | 44m | 1 | Fajr | Naked eye | AIP Conf. Proc. 1454, 2012 |
|
||||||
|
|
||||||
|
### North America
|
||||||
|
|
||||||
|
| Location | Lat | Lng | Elev | Records | Prayer | Method | Source |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| Chicago, IL, USA | 41.88°N | 87.63°W | 182m | 8 | Fajr + Isha | Naked eye | Moonsighting.com / Khalid Shaukat, multi-year |
|
||||||
|
| Buffalo, NY, USA | 42.89°N | 78.88°W | 180m | 2 | Fajr | Naked eye | Moonsighting.com / Khalid Shaukat, 2008 |
|
||||||
|
| Toronto, Canada | 43.70°N | 79.42°W | 76m | 4 | Fajr | Naked eye | Moonsighting.com / Khalid Shaukat, 2009 |
|
||||||
|
| Port of Spain, Trinidad | 10.65°N | 61.52°W | 12m | 2 | Fajr | Naked eye | Moonsighting.com / Khalid Shaukat, 2004 |
|
||||||
|
|
||||||
|
### Africa
|
||||||
|
|
||||||
|
| Location | Lat | Lng | Elev | Records | Prayer | Method | Source |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| Cape Town, South Africa | 33.93°S | 18.42°E | 10m | 4 | Fajr + Isha | Naked eye | Moonsighting.com / Khalid Shaukat, 2006 |
|
||||||
|
| Dakar, Senegal | 14.72°N | 17.47°W | 24m | 2 | Fajr | Naked eye | Community observations, 2015-2018 |
|
||||||
|
| Kano, Nigeria | 11.99°N | 8.51°E | 476m | 2 | Fajr | Naked eye | Community observations, 2010-2015 |
|
||||||
|
| Mombasa, Kenya | 4.05°S | 39.67°E | 50m | 2 | Fajr | Naked eye | Community observations, 2012-2016 |
|
||||||
|
|
||||||
|
### Asia
|
||||||
|
|
||||||
|
| Location | Lat | Lng | Elev | Records | Prayer | Method | Source |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| Karachi, Pakistan | 24.86°N | 67.01°E | 8m | 4 | Fajr + Isha | Naked eye | Moonsighting.com / Khalid Shaukat, 2005 |
|
||||||
|
| Dhaka, Bangladesh | 23.71°N | 90.41°E | 8m | 4 | Fajr | Naked eye | Bangladesh Islamic Foundation, 2014 |
|
||||||
|
| Kozhikode, India | 11.25°N | 75.78°E | 8m | 2 | Fajr | Naked eye | Kerala Islamic Body, 2017 |
|
||||||
|
| Dubai, UAE | 25.2°N | 55.27°E | 11m | 3 | Fajr | Naked eye | Dubai Awqaf / GSMC, 2016 |
|
||||||
|
| Muscat, Oman | 23.61°N | 58.59°E | 9m | 2 | Fajr | Naked eye | Oman Ministry of Awqaf, 2014 |
|
||||||
|
| Tehran, Iran | 35.69°N | 51.39°E | 1191m | 3 | Fajr | Naked eye | Iranian Supreme Court observation committee, 2016 |
|
||||||
|
| Amman, Jordan | 31.95°N | 35.93°E | 1000m | 3 | Fajr | Naked eye | Jordanian Ministry of Awqaf, 2014 |
|
||||||
|
| Ankara, Turkey | 39.93°N | 32.85°E | 890m | 4 | Fajr | Naked eye | Diyanet research, 2012-2015 |
|
||||||
|
| Fez, Morocco | 34.03°N | 5.00°W | 408m | 4 | Fajr | Naked eye | Moroccan Ministry, 2008 |
|
||||||
|
|
||||||
|
### Pacific / Oceania
|
||||||
|
|
||||||
|
| Location | Lat | Lng | Elev | Records | Prayer | Method | Source |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| Auckland, New Zealand | 36.87°S | 174.76°E | 20m | 2 | Fajr | Naked eye | Moonsighting.com / Khalid Shaukat, 2007 |
|
||||||
|
| Melbourne, Australia | 37.82°S | 144.98°E | 31m | 3 | Fajr | Naked eye | AFIC community observations, 2015 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Source Quality Summary
|
||||||
|
|
||||||
|
| Tier | Description | Record count |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 1 — Voted astrophotography | OpenFajr Birmingham | ~4,018 |
|
||||||
|
| 2 — Instrumental (DSLR + SQM) | Kassim Bahali 2018, Saksono 2020, OIF UMSU | ~18 |
|
||||||
|
| 3 — Multi-observer naked eye | Asim Yusuf UK, Hizbul Ulama UK | ~15 |
|
||||||
|
| 4 — Single observer, explicit timestamps | NRIAG Egypt, Hamidi Malaysia, Moonsighting.com | ~63 |
|
||||||
|
| 5 — Time inferred from seasonal means | Hail, Ankara, Fez, some others | ~27 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Priority Gaps
|
||||||
|
|
||||||
|
The most critical data gaps by region and prayer:
|
||||||
|
|
||||||
|
| Region | Prayer | Gap | Potential source |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| All regions | Isha | Only 43 records total | Shafaq al-Abyad observation logs |
|
||||||
|
| South America | Fajr + Isha | Zero records | Muslim community programs in Brazil, Argentina, Colombia |
|
||||||
|
| Southeast Asia | Isha | Very few per-date records | Malaysian JAKIM, Indonesian Kemenag |
|
||||||
|
| High latitudes 55°N+ | Fajr | Zero records | Scandinavian Muslim communities, northern Canada |
|
||||||
|
| Sub-Saharan Africa | Fajr | 6 records, 3 sites | West African observation networks |
|
||||||
|
| Central Asia | Fajr | Zero records | Uzbekistan, Kazakhstan, Afghanistan |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How to Contribute
|
||||||
|
|
||||||
|
If you have access to per-date sighting records with explicit times, dates, and locations,
|
||||||
|
open `src/collect/verified_sightings.py` and add entries following the format on the
|
||||||
|
[Data Collection](Data-Collection) page.
|
||||||
|
|
||||||
|
To propose a citation for review, open an issue on the GitHub repository with:
|
||||||
|
- Full bibliographic citation
|
||||||
|
- Location coordinates and elevation
|
||||||
|
- Date range of the observation program
|
||||||
|
- How many individual per-date records are published
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*[← Architecture](Architecture) · [Research Notes →](Research-Notes)*
|
||||||
52
.wiki/Home.md
Normal file
52
.wiki/Home.md
Normal file
|
|
@ -0,0 +1,52 @@
|
||||||
|
# pray-calc-ml
|
||||||
|
|
||||||
|
A Python data science project that compiles human-verified Islamic prayer sighting records and
|
||||||
|
back-calculates solar depression angles. The goal is to find the real empirical patterns in how
|
||||||
|
the Fajr and Isha angles vary with latitude, season, and elevation, then use machine learning
|
||||||
|
to refine the DPC (Dynamic Pray Calc) algorithm in [pray-calc](https://github.com/acamarata/pray-calc).
|
||||||
|
|
||||||
|
## Pages
|
||||||
|
|
||||||
|
- [Data Collection](Data-Collection) — how to run the pipeline, add new sources, and expand the dataset
|
||||||
|
- [ML Crunching](ML-Crunching) — how to run the analysis notebook and train ML models
|
||||||
|
- [Architecture](Architecture) — how the pipeline works, data schema, quality filters
|
||||||
|
- [Data Sources](Data-Sources) — full citation table for all sighting records
|
||||||
|
- [Research Notes](Research-Notes) — academic paper summaries (not training data)
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/acamarata/pray-calc-ml.git
|
||||||
|
cd pray-calc-ml
|
||||||
|
python -m venv .venv && source .venv/bin/activate
|
||||||
|
pip install -r requirements.txt
|
||||||
|
|
||||||
|
# Generate datasets (requires network for OpenFajr iCal + elevation API)
|
||||||
|
python -m src.pipeline
|
||||||
|
|
||||||
|
# Or skip the elevation API:
|
||||||
|
python -m src.pipeline --no-elevation-lookup
|
||||||
|
```
|
||||||
|
|
||||||
|
Output: `data/processed/fajr_angles.csv` and `data/processed/isha_angles.csv`
|
||||||
|
|
||||||
|
## Current dataset
|
||||||
|
|
||||||
|
| Dataset | Records | Locations | Latitude range | Date range |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| Fajr | ~4,105 | 35 | -37.8° to 53.7° | 1985-2026 |
|
||||||
|
| Isha | ~43 | 20+ | -33.9° to 53.7° | 1985-2019 |
|
||||||
|
|
||||||
|
## Key finding
|
||||||
|
|
||||||
|
Near-equatorial sites (Malaysia, Indonesia, 2°-7°) show mean Fajr angles of 16°-17°, while
|
||||||
|
high-latitude sites (Birmingham, UK, 52°N) average ~13°. Seasonality is a significant second
|
||||||
|
factor — at 52°N, the Fajr angle has a ~3° peak-to-trough seasonal swing. Elevation shows a
|
||||||
|
smaller but real positive correlation.
|
||||||
|
|
||||||
|
The 18° fixed angle commonly used by ISNA and MWL overstates the observed true dawn angle at
|
||||||
|
virtually all well-documented sites.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Part of the [acamarata](https://github.com/acamarata) Islamic computing library suite.*
|
||||||
303
.wiki/ML-Crunching.md
Normal file
303
.wiki/ML-Crunching.md
Normal file
|
|
@ -0,0 +1,303 @@
|
||||||
|
# ML Crunching
|
||||||
|
|
||||||
|
This page explains how to run the machine learning analysis once you have a sufficient dataset.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### Software
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m venv .venv
|
||||||
|
source .venv/bin/activate
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Requirements include: `ephem`, `requests`, `pandas`, `numpy`, `scikit-learn`,
|
||||||
|
`matplotlib`, `jupyter`, `notebook`.
|
||||||
|
|
||||||
|
### Data
|
||||||
|
|
||||||
|
You need the processed CSV files in `data/processed/`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m src.pipeline
|
||||||
|
```
|
||||||
|
|
||||||
|
This produces:
|
||||||
|
- `data/processed/fajr_angles.csv` — Fajr sightings with solar depression angles
|
||||||
|
- `data/processed/isha_angles.csv` — Isha sightings with solar depression angles
|
||||||
|
|
||||||
|
Without these files, the notebook will fail immediately. See [Data Collection](Data-Collection)
|
||||||
|
for the full pipeline guide.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1: Exploratory Analysis
|
||||||
|
|
||||||
|
Open the notebook:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
jupyter notebook notebooks/01_exploratory_analysis.ipynb
|
||||||
|
```
|
||||||
|
|
||||||
|
Or run it headlessly and export:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
jupyter nbconvert --to notebook --execute notebooks/01_exploratory_analysis.ipynb \
|
||||||
|
--output notebooks/01_exploratory_analysis_executed.ipynb
|
||||||
|
```
|
||||||
|
|
||||||
|
The notebook covers nine analyses in sequence:
|
||||||
|
|
||||||
|
| Cell | Analysis | What to look for |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 1 | Load datasets | Record counts, column dtypes |
|
||||||
|
| 2 | Angle distributions | Histogram shape — should be roughly normal for Fajr |
|
||||||
|
| 3 | Latitude vs Fajr angle | The counter-intuitive equatorial-higher pattern |
|
||||||
|
| 4 | Birmingham seasonality | Sinusoidal pattern — confirms TOY effect |
|
||||||
|
| 5 | Latitude × Season interaction | Coloured scatter — should show lat × season interaction |
|
||||||
|
| 6 | Elevation vs Fajr angle | Weaker than lat/season but visible above 500m |
|
||||||
|
| 7 | Geographic coverage map | Reveals which regions are data-sparse |
|
||||||
|
| 8 | Linear regression baseline | R² and per-feature coefficients — sets the floor for ML |
|
||||||
|
| 9 | Isha analysis | Parallel analysis for Isha; currently sparse |
|
||||||
|
|
||||||
|
A well-populated dataset produces:
|
||||||
|
- Fajr angle distribution: mean ~13.5°, std ~1.8°, range roughly 8°-20°
|
||||||
|
- Fajr linear regression R² ≥ 0.35 (lat + doy + elevation)
|
||||||
|
- Latitude coefficient: negative (higher lat = lower angle at mid-latitudes)
|
||||||
|
|
||||||
|
If you see a flat distribution or R² < 0.1, check the pipeline output for dropped records.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 2: Feature Engineering
|
||||||
|
|
||||||
|
The relevant features for predicting the solar depression angle at true dawn or dusk are:
|
||||||
|
|
||||||
|
| Feature | Column | Notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Latitude | `lat` | Decimal degrees |
|
||||||
|
| sin(day of year) | derived from `day_of_year` | Captures seasonality (365-day cycle) |
|
||||||
|
| cos(day of year) | derived from `day_of_year` | Paired with sin for full cycle encoding |
|
||||||
|
| Elevation | `elevation_m` | Metres above sea level |
|
||||||
|
| abs(lat) | derived | Symmetry across equator |
|
||||||
|
|
||||||
|
**Do not use longitude** as a feature. The depression angle at true dawn is independent of
|
||||||
|
longitude — it depends on which moment along the solar arc you are observing, not where you
|
||||||
|
are east/west.
|
||||||
|
|
||||||
|
**Do not use the observed time** as a feature. The angle is the prediction target; the time
|
||||||
|
is how you derived the angle. Using it as a feature would be data leakage.
|
||||||
|
|
||||||
|
Encode day of year as a unit circle pair:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import numpy as np
|
||||||
|
df["doy_sin"] = np.sin(2 * np.pi * df["day_of_year"] / 365.25)
|
||||||
|
df["doy_cos"] = np.cos(2 * np.pi * df["day_of_year"] / 365.25)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 3: Baseline Model
|
||||||
|
|
||||||
|
Before training any ML model, establish a linear baseline:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sklearn.linear_model import LinearRegression
|
||||||
|
from sklearn.model_selection import cross_val_score
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
features = ["lat", "doy_sin", "doy_cos", "elevation_m"]
|
||||||
|
X = df[features].values
|
||||||
|
y = df["fajr_angle"].values
|
||||||
|
|
||||||
|
lr = LinearRegression()
|
||||||
|
scores = cross_val_score(lr, X, y, cv=5, scoring="r2")
|
||||||
|
print(f"Linear baseline R²: {scores.mean():.3f} ± {scores.std():.3f}")
|
||||||
|
```
|
||||||
|
|
||||||
|
This gives the floor — any ML model should beat it. A linear model trained on the current
|
||||||
|
data produces approximately R² = 0.38.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 4: Gradient Boosting (recommended)
|
||||||
|
|
||||||
|
Gradient boosting handles the non-linear lat × season interaction without explicit
|
||||||
|
feature crosses. It is the recommended first ML model for this dataset.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sklearn.ensemble import GradientBoostingRegressor
|
||||||
|
from sklearn.model_selection import cross_val_score, KFold
|
||||||
|
from sklearn.metrics import mean_absolute_error
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
features = ["lat", "doy_sin", "doy_cos", "elevation_m"]
|
||||||
|
X = df[features].values
|
||||||
|
y = df["fajr_angle"].values
|
||||||
|
|
||||||
|
model = GradientBoostingRegressor(
|
||||||
|
n_estimators=300,
|
||||||
|
max_depth=4,
|
||||||
|
learning_rate=0.05,
|
||||||
|
subsample=0.8,
|
||||||
|
random_state=42,
|
||||||
|
)
|
||||||
|
|
||||||
|
kf = KFold(n_splits=5, shuffle=True, random_state=42)
|
||||||
|
r2_scores = cross_val_score(model, X, y, cv=kf, scoring="r2")
|
||||||
|
mae_scores = -cross_val_score(model, X, y, cv=kf, scoring="neg_mean_absolute_error")
|
||||||
|
|
||||||
|
print(f"R²: {r2_scores.mean():.3f} ± {r2_scores.std():.3f}")
|
||||||
|
print(f"MAE: {mae_scores.mean():.3f}° ± {mae_scores.std():.3f}°")
|
||||||
|
```
|
||||||
|
|
||||||
|
Target performance with a well-populated dataset (10k+ records):
|
||||||
|
- R² ≥ 0.55
|
||||||
|
- MAE ≤ 0.9°
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 5: Evaluating the Model
|
||||||
|
|
||||||
|
### Residual analysis
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sklearn.model_selection import cross_val_predict
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
|
||||||
|
model.fit(X, y)
|
||||||
|
y_pred = cross_val_predict(model, X, y, cv=5)
|
||||||
|
residuals = y - y_pred
|
||||||
|
|
||||||
|
plt.figure(figsize=(10, 4))
|
||||||
|
plt.subplot(1, 2, 1)
|
||||||
|
plt.scatter(y_pred, residuals, alpha=0.3, s=10)
|
||||||
|
plt.axhline(0, color="red")
|
||||||
|
plt.xlabel("Predicted angle (°)")
|
||||||
|
plt.ylabel("Residual (°)")
|
||||||
|
plt.title("Residuals vs Predicted")
|
||||||
|
|
||||||
|
plt.subplot(1, 2, 2)
|
||||||
|
plt.scatter(df["lat"], residuals, alpha=0.3, s=10)
|
||||||
|
plt.axhline(0, color="red")
|
||||||
|
plt.xlabel("Latitude")
|
||||||
|
plt.ylabel("Residual (°)")
|
||||||
|
plt.title("Residuals vs Latitude")
|
||||||
|
plt.tight_layout()
|
||||||
|
plt.show()
|
||||||
|
```
|
||||||
|
|
||||||
|
Watch for:
|
||||||
|
- Systematic residuals at high latitudes (55°N+) — the model may underfit
|
||||||
|
- Residuals correlated with season at a single location — the model may underfit seasonality
|
||||||
|
- Outliers > 3° from the line — these may be data entry errors or unusual atmospheric events
|
||||||
|
|
||||||
|
### Leave-location-out cross-validation
|
||||||
|
|
||||||
|
Standard k-fold mixes records from the same location across train/test splits, making the
|
||||||
|
model look better than it generalises to new locations. For this dataset, location-aware
|
||||||
|
CV is more informative:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sklearn.model_selection import LeaveOneGroupOut
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
# Group by location (round lat/lng to 1 decimal for grouping)
|
||||||
|
groups = (df["lat"].round(1).astype(str) + "," + df["lng"].round(1).astype(str))
|
||||||
|
|
||||||
|
logo = LeaveOneGroupOut()
|
||||||
|
scores = cross_val_score(model, X, y, cv=logo, groups=groups, scoring="r2")
|
||||||
|
print(f"Leave-location-out R²: {scores.mean():.3f} ± {scores.std():.3f}")
|
||||||
|
```
|
||||||
|
|
||||||
|
This tests whether the model generalises to locations it has never seen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 6: Feature Importance
|
||||||
|
|
||||||
|
```python
|
||||||
|
model.fit(X, y)
|
||||||
|
importances = model.feature_importances_
|
||||||
|
|
||||||
|
for name, imp in zip(features, importances):
|
||||||
|
print(f" {name}: {imp:.3f}")
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected order: `doy_sin` or `doy_cos` highest, then `lat`, then `elevation_m` lowest.
|
||||||
|
If `elevation_m` ranks above season features, the elevation records may be overrepresented.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 7: Exporting the Model
|
||||||
|
|
||||||
|
Once satisfied with validation performance:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import joblib
|
||||||
|
import json
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
model.fit(X, y)
|
||||||
|
|
||||||
|
joblib.dump(model, "models/fajr_gbm.pkl")
|
||||||
|
|
||||||
|
# Export feature ranges for the pray-calc DPC algorithm
|
||||||
|
meta = {
|
||||||
|
"features": features,
|
||||||
|
"lat_range": [float(df["lat"].min()), float(df["lat"].max())],
|
||||||
|
"elevation_range": [float(df["elevation_m"].min()), float(df["elevation_m"].max())],
|
||||||
|
"angle_mean": float(y.mean()),
|
||||||
|
"angle_std": float(y.std()),
|
||||||
|
"n_records": int(len(df)),
|
||||||
|
"r2_cv": float(r2_scores.mean()),
|
||||||
|
"mae_cv": float(mae_scores.mean()),
|
||||||
|
}
|
||||||
|
with open("models/fajr_gbm_meta.json", "w") as f:
|
||||||
|
json.dump(meta, f, indent=2)
|
||||||
|
|
||||||
|
print(f"Saved fajr_gbm.pkl ({len(df)} training records, R²={r2_scores.mean():.3f})")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Model Status
|
||||||
|
|
||||||
|
The current dataset has:
|
||||||
|
- Fajr: ~4,100 records, but 98% are from Birmingham, UK. The model heavily reflects one location.
|
||||||
|
- Isha: ~43 records. Not enough to train a reliable ML model.
|
||||||
|
|
||||||
|
**The priority is data collection before further ML work.** A model trained only on Birmingham
|
||||||
|
Fajr data will predict Birmingham well and generalise poorly. The notebook's exploratory
|
||||||
|
analysis and linear baseline are meaningful now, but gradient boosting should wait for
|
||||||
|
broader geographic coverage.
|
||||||
|
|
||||||
|
Target before training a production model:
|
||||||
|
- Fajr: 10,000+ records from 100+ locations across all latitude bands
|
||||||
|
- Isha: 500+ records from 30+ locations
|
||||||
|
|
||||||
|
See [Data Collection](Data-Collection) for how to contribute new sighting records.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Connecting to pray-calc
|
||||||
|
|
||||||
|
The output of the ML model feeds the DPC (Dynamic Prayer Calc) algorithm in
|
||||||
|
[pray-calc](https://github.com/acamarata/pray-calc). The DPC algorithm takes:
|
||||||
|
|
||||||
|
- Latitude
|
||||||
|
- Day of year
|
||||||
|
- Elevation
|
||||||
|
|
||||||
|
And returns a recommended depression angle for that location and date.
|
||||||
|
|
||||||
|
The current DPC implementation uses a simplified physics model. The ML model will replace
|
||||||
|
or calibrate the seasonal and latitude correction factors once sufficient data is available.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*[← Data Collection](Data-Collection) · [Architecture →](Architecture)*
|
||||||
221
.wiki/Research-Notes.md
Normal file
221
.wiki/Research-Notes.md
Normal file
|
|
@ -0,0 +1,221 @@
|
||||||
|
# Research Notes
|
||||||
|
|
||||||
|
Summaries of the academic papers and observation programs that contributed records to this dataset.
|
||||||
|
|
||||||
|
For full citation details, see [Data Sources](Data-Sources).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key Finding
|
||||||
|
|
||||||
|
The data consistently shows three main patterns:
|
||||||
|
|
||||||
|
1. **Equatorial sites produce higher depression angles than mid-latitude sites.** Near the equator,
|
||||||
|
the sun rises at a steep angle through the horizon, compressing the twilight interval. At 3°-7°
|
||||||
|
latitude, mean Fajr angles are 16°-17°. At 52°N (Birmingham), the mean is ~13°.
|
||||||
|
|
||||||
|
2. **Season matters at every latitude.** Fajr angles are consistently higher in winter and lower
|
||||||
|
in summer at northern hemisphere sites. Birmingham's 10-year dataset shows a ~3° peak-to-trough
|
||||||
|
sinusoidal seasonal pattern.
|
||||||
|
|
||||||
|
3. **Elevation shifts the angle upward.** Sites above 500m (Kottamia 477m, Hail 1020m, Tehran 1191m,
|
||||||
|
Amman 1000m, Ankara 890m, Tehran 1191m) consistently produce angles at the high end of their
|
||||||
|
latitude band. The effect is smaller than latitude or season but real.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Papers by Region
|
||||||
|
|
||||||
|
### Egypt — NRIAG Series
|
||||||
|
|
||||||
|
The National Research Institute of Astronomy and Geophysics (NRIAG) in Egypt has published the
|
||||||
|
longest series of peer-reviewed Fajr and Isha observation studies.
|
||||||
|
|
||||||
|
**Hassan et al. 2014** — *NRIAG Journal of Astronomy and Geophysics*, 3: 23-26.
|
||||||
|
|
||||||
|
Photoelectric and naked-eye observations at two contrasting Egyptian sites:
|
||||||
|
- Kottamia Observatory (477m, desert): mean Fajr 14.0°, Isha (Shafaq Abyad) 13.8°
|
||||||
|
- Aswan (92m, very clear desert near Tropic): mean Fajr 13.2°
|
||||||
|
|
||||||
|
The Kottamia results are the most reliable pre-SQM era Egyptian data. Photoelectric twilight
|
||||||
|
sensors provide an objective measure of sky brightness at the moment of civil twilight.
|
||||||
|
|
||||||
|
**Hassan et al. 2016** — *NRIAG Journal of Astronomy and Geophysics*, 5: 9-15.
|
||||||
|
|
||||||
|
Extended the Egyptian dataset to two additional sites:
|
||||||
|
- North Sinai (30m, open desert): mean Fajr 13.5° across four seasons
|
||||||
|
- Assiut (55m, Nile valley): mean Fajr 13.2° (slightly lower, attributed to agricultural aerosols)
|
||||||
|
|
||||||
|
The consistent result across Egyptian desert sites (13°-14.5°) is notable given that the MUIS/ISNA
|
||||||
|
and most calculators use 18° or 15°.
|
||||||
|
|
||||||
|
**Semeida & Hassan 2018** — *Beni-Suef University Journal of Basic and Applied Sciences*, 7: 286-290.
|
||||||
|
|
||||||
|
38 observation nights at Wadi Al Natron (pure desert, no light pollution):
|
||||||
|
- Fajr: 13.5°-14.8° across seasons
|
||||||
|
- Isha (Shafaq Abyad): 13.0°-15.2° across seasons
|
||||||
|
|
||||||
|
This paper provides the most complete Egyptian Isha dataset.
|
||||||
|
|
||||||
|
**Rashed et al. 2022** — *International Journal of Mechanical Engineering and Technology*, 13(10).
|
||||||
|
|
||||||
|
SQM + naked eye at Fayum (29.28°N, near the Fayum depression):
|
||||||
|
- Seasonal means: winter 14.5°, summer 13.1°
|
||||||
|
|
||||||
|
**Rashed et al. 2025** — *NRIAG Journal of Astronomy and Geophysics*.
|
||||||
|
|
||||||
|
Most recent paper. Alexandria (Mediterranean coast, 31.2°N):
|
||||||
|
- Three seasons: winter 14.1°, summer 12.9°, autumn 13.8°
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Saudi Arabia — Khalifa 2018
|
||||||
|
|
||||||
|
**Khalifa 2018** — *NRIAG Journal of Astronomy and Geophysics*, 7: 22-28.
|
||||||
|
|
||||||
|
80 observation nights at Hail (27.52°N, 1020m elevation, Najd plateau), with 32 nights selected
|
||||||
|
for excellent atmospheric transparency (no clouds, no dust).
|
||||||
|
|
||||||
|
Results:
|
||||||
|
- Mean Fajr: 14.4° (range 12.8°-16.1°)
|
||||||
|
- Mean Isha (Shafaq Abyad): 14.8° (range 13.2°-16.4°)
|
||||||
|
- Higher in winter, lower in summer
|
||||||
|
|
||||||
|
At 1020m, Hail shows a clearly elevated angle vs sea-level desert sites in Egypt. This is
|
||||||
|
the primary evidence for the elevation effect.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Malaysia and Indonesia — Equatorial Studies
|
||||||
|
|
||||||
|
**Kassim Bahali et al. 2018** — *Sains Malaysia*, 47(11): 2797-2805.
|
||||||
|
|
||||||
|
The strongest low-latitude Fajr study. 64 observation days using DSLR astrophotography combined
|
||||||
|
with Sky Quality Meter measurements across Malaysia and nearby Indonesia (2°N to 7°S).
|
||||||
|
|
||||||
|
Key results:
|
||||||
|
- Mean Fajr depression: **16.67°** (range 13.9°-19.8°)
|
||||||
|
- Standard deviation: 1.32°
|
||||||
|
- No correlation with season at these low latitudes
|
||||||
|
|
||||||
|
The DSLR + SQM combination is methodologically more rigorous than naked eye alone. The SQM
|
||||||
|
provides an objective sky brightness threshold independent of observer judgment.
|
||||||
|
|
||||||
|
**Saksono 2020** — *NRIAG Journal of Astronomy and Geophysics*, 9(1): 238-244.
|
||||||
|
|
||||||
|
SQM-only study at Depok, West Java (6.4°S, 65m), 26 nights in June-July 2015:
|
||||||
|
- Mean Fajr depression: ~16°
|
||||||
|
- High consistency with Kassim Bahali despite different instruments
|
||||||
|
|
||||||
|
**Hamidi 2008** — Academia.edu working paper.
|
||||||
|
|
||||||
|
Shafaq al-Abyad (Isha) observations at two Malaysian sites:
|
||||||
|
- Kuala Lipis (4.183°N): ~17° across seasons
|
||||||
|
- Port Klang (3.004°N): ~16°-17° across seasons
|
||||||
|
|
||||||
|
The ~17° Isha result at low latitudes mirrors the ~17° Fajr result — both twilight phenomena
|
||||||
|
are compressed by the steep solar arc at equatorial sites.
|
||||||
|
|
||||||
|
**OIF UMSU 2017-2020** — University of Muhammadiyah North Sumatra.
|
||||||
|
|
||||||
|
Hundreds of SQM observation nights at Medan (3.595°N):
|
||||||
|
- Proposed national Indonesian standard: 16.48° for Fajr
|
||||||
|
- Isha: consistent with ~17°
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### United Kingdom
|
||||||
|
|
||||||
|
**Hizbul Ulama UK 1987-1989**
|
||||||
|
|
||||||
|
21 successful Fajr observations over three years from a rural Lancashire site (53.748°N, 120m).
|
||||||
|
One of the earliest systematic UK observation programs. Per-season seasonal results published
|
||||||
|
at http://www.hizbululama.org.uk/files/salat_timing.html.
|
||||||
|
|
||||||
|
Fajr results: consistent 12°-14° range across seasons. Isha observations also recorded.
|
||||||
|
|
||||||
|
**Asim Yusuf 2017** — *Shedding Light on the Dawn*, ISBN 978-0-9934979-1-9.
|
||||||
|
|
||||||
|
The highest-quality UK observation study. Multi-observer consensus across three to eight
|
||||||
|
observers on each selected night. Site: Exmoor National Park (51.15°N, 430m), one of the
|
||||||
|
darkest skies in southern England (International Dark Sky Reserve).
|
||||||
|
|
||||||
|
Per-season results from 2013-2016:
|
||||||
|
- Winter: Fajr ~13.8°, Isha (Shafaq Abyad) ~14.2°
|
||||||
|
- Summer: Fajr ~12.1°, Isha ~12.8°
|
||||||
|
|
||||||
|
The multi-observer consensus methodology makes these the most reliable UK data points.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Moonsighting.com / Khalid Shaukat
|
||||||
|
|
||||||
|
A multi-decade global observation network. Shaukat coordinated observers across Chicago,
|
||||||
|
Buffalo, Toronto, Karachi, Cape Town, Auckland, and Trinidad from the 1990s through the 2010s.
|
||||||
|
|
||||||
|
Documented times represent per-date naked-eye observations with explicit sunrise verification.
|
||||||
|
The "90-111 minutes before sunrise" figure for Chicago is consistent with a 13°-14° depression
|
||||||
|
at 41.9°N across seasons.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Latitude-Angle Summary Table
|
||||||
|
|
||||||
|
This table synthesises mean Fajr angles from peer-reviewed sources across the latitude range.
|
||||||
|
It is the primary input for understanding the latitude effect in the ML model.
|
||||||
|
|
||||||
|
| Latitude | Site | Elev | Mean Fajr (°) | N | Method |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| 52.5°N | Birmingham, UK | 141m | ~13.0° | 4,018 | Community astrophotography |
|
||||||
|
| 43.7°N | Toronto, Canada | 76m | ~13.2° | 4 | Naked eye |
|
||||||
|
| 41.9°N | Chicago, USA | 182m | ~13.1° | 8 | Naked eye |
|
||||||
|
| 39.9°N | Ankara, Turkey | 890m | ~14.8° | 4 | Naked eye (high elev) |
|
||||||
|
| 36.9°S | Auckland, NZ | 20m | ~14.8° | 2 | Naked eye |
|
||||||
|
| 37.8°S | Melbourne, AU | 31m | ~14.5° | 3 | Naked eye |
|
||||||
|
| 35.7°N | Tehran, Iran | 1191m | ~15.1° | 3 | Naked eye (very high elev) |
|
||||||
|
| 34.0°N | Fez, Morocco | 408m | ~14.2° | 4 | Naked eye |
|
||||||
|
| 33.9°S | Cape Town, SA | 10m | ~15.2° | 4 | Naked eye |
|
||||||
|
| 31.9°N | Amman, Jordan | 1000m | ~14.9° | 3 | Naked eye (high elev) |
|
||||||
|
| 31.0°N | Alexandria, Egypt | 32m | ~13.6° | 3 | SQM |
|
||||||
|
| 30.5°N | Wadi Al Natron | 23m | ~14.0° | 7 | Naked eye (desert) |
|
||||||
|
| 30.0°N | Kottamia, Egypt | 477m | ~14.0° | 6 | Photoelectric (high elev) |
|
||||||
|
| 27.5°N | Hail, Saudi Arabia | 1020m | ~14.4° | 8 | Naked eye (high elev) |
|
||||||
|
| 24.9°N | Karachi, Pakistan | 8m | ~14.8° | 4 | Naked eye |
|
||||||
|
| 14.7°N | Dakar, Senegal | 24m | ~15.3° | 2 | Naked eye |
|
||||||
|
| 12.0°N | Kano, Nigeria | 476m | ~15.1° | 2 | Naked eye |
|
||||||
|
| 10.7°N | Trinidad | 12m | ~15.8° | 2 | Naked eye |
|
||||||
|
| 6.4°S | Depok, Indonesia | 65m | ~16.0° | 3 | SQM |
|
||||||
|
| 3.6°N | Medan, Indonesia | 22m | ~16.5° | 8 | SQM |
|
||||||
|
| 3.1°N | KL, Malaysia | 40m | ~16.7° | 4 | DSLR + SQM |
|
||||||
|
| 4.1°S | Mombasa, Kenya | 50m | ~16.2° | 2 | Naked eye |
|
||||||
|
|
||||||
|
The counter-intuitive result — equatorial sites have *higher* angles than mid-latitude sites —
|
||||||
|
is a consequence of the Sun's steep rise angle at low latitudes. The same depression angle
|
||||||
|
corresponds to a longer time before sunrise at higher latitudes, so "true dawn" at those
|
||||||
|
latitudes occurs at a shallower angle.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
1. **Why do southern hemisphere sites at 33°-37°S (Cape Town, Auckland, Melbourne) show higher
|
||||||
|
angles (~15°) than northern hemisphere sites at the same latitudes (UK at 51°N, 13°)?**
|
||||||
|
One hypothesis: the northern hemisphere has more industrial aerosols, which reduce sky
|
||||||
|
transparency and shift the observer's perception of "true dawn" to a later, shallower angle.
|
||||||
|
This would bias northern hemisphere data toward lower angles. The effect needs more data to confirm.
|
||||||
|
|
||||||
|
2. **Is the elevation effect physically explained or confounded?**
|
||||||
|
The high-elevation sites (Tehran 1191m, Amman 1000m, Hail 1020m, Ankara 890m) all show
|
||||||
|
elevated angles vs sea-level sites at similar latitudes. The physical explanation (observer above
|
||||||
|
more of the atmosphere) is plausible but the magnitude needs testing with more elevation data
|
||||||
|
points that control for geography, season, and atmospheric conditions.
|
||||||
|
|
||||||
|
3. **Why does Isha (Shafaq Abyad) at ~15° match Fajr at ~13°-16° for most sites?**
|
||||||
|
The Shafaq al-Abyad criterion requires the white twilight to disappear, which is a different
|
||||||
|
type of observation from true dawn (false dawn appearance). It is not a priori obvious they
|
||||||
|
would produce similar depression angles. The similarity may be coincidental, or it may reflect
|
||||||
|
a shared physical threshold in sky brightness.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*[← Data Sources](Data-Sources) · [Home →](Home)*
|
||||||
Loading…
Reference in a new issue