================================================================================
  SITE DOWN ALERT — Server Monitor Setup Guide
  local_downalert v1.1.2
================================================================================

This guide is for anyone who manages a Linux server. Follow every step exactly
and you will have automatic Moodle downtime monitoring running in ~10 minutes.

The plugin was just upgraded in v1.1.1 — you no longer need a config.php file
on the server. All settings come from your Moodle plugin settings page. Just
give the script two arguments and it reads everything automatically.

================================================================================
  HOW IT WORKS (read this first — it's short)
================================================================================

  1. You install this Moodle plugin on EACH of your Moodle sites.
     The plugin automatically tells the AI Grader platform "I exist, here is
     my URL and health token." It does this every night at 3:30am.

  2. On EACH SERVER that hosts your Moodle sites, you set up the monitor script
     once. The script runs every 3 minutes via crontab.

  3. The script fetches its settings directly from Moodle's database using your
     health token — alert email, API key, thresholds. Nothing to edit on the
     server.

  4. It checks each site by visiting its URL. If a site is unreachable it:
       - Diagnoses the exact cause (disk full, web server crashed, SSL expired...)
       - Emails you with the cause AND the exact SSH commands to fix it

  IMPORTANT — if your Moodles are on MULTIPLE SERVERS:
  Run this setup on EACH server separately. Each server's monitor only watches
  the Moodles on its own disk.

================================================================================
  WHAT YOU NEED BEFORE STARTING
================================================================================

  - SSH access to your server
  - Your AI Grader health token
    → In Moodle: Site Admin → Plugins → Local plugins → Site Down Alert
    → Copy the "Health token" value from that page
    → It looks like: f1a8bd9a6be77720fc40a60e0d927c86

  You do NOT need your API key on the server — it is read automatically from
  your Moodle plugin settings.

================================================================================
  STEP 1 — SSH INTO YOUR SERVER
================================================================================

  On Windows: open "Command Prompt" or "PowerShell" and type:
    ssh root@YOUR.SERVER.IP.ADDRESS

  On Mac or Linux: open "Terminal" and type:
    ssh root@YOUR.SERVER.IP.ADDRESS

  Replace YOUR.SERVER.IP.ADDRESS with your actual server IP.
  You will be asked for your password. Type it and press Enter.
  (The cursor won't move while you type — that's normal.)

  Once connected you will see:  root@server:~#

================================================================================
  STEP 2 — FIND YOUR MOODLE INSTALLATION PATH
================================================================================

  Type this and press Enter:
    find /var/www -name "config.php" -path "*/moodle*" 2>/dev/null | head -5

  Example output:
    /var/www/moodle/mymoodle.mcgill.edu.au/config.php

  The folder CONTAINING config.php is your Moodle root.
  Write it down — you will use it below.

================================================================================
  STEP 3 — TEST THE MONITOR MANUALLY
================================================================================

  Run this command (replacing the path and token with your own values):

    php8.2 /var/www/moodle/mymoodle.mcgill.edu.au/local/downalert/server-monitor/monitor.php \
      --moodle-url=https://mymoodle.mcgill.edu.au \
      --token=YOUR_HEALTH_TOKEN_HERE

  What each part means:
    php8.2             = use PHP 8.2 (use "php" if this doesn't work)
    --moodle-url=...   = the public URL of your Moodle site
    --token=...        = the health token from your plugin settings page

  You should see output like:
    [2026-05-30 07:30:12] Fetching config from Moodle: https://mymoodle.mcgill.edu.au
    [2026-05-30 07:30:12] Remote config loaded (alert_email=admin@school.edu.au, api_key=***set***).
    [2026-05-30 07:30:12] Server stack detected: nginx | php8.2-fpm | mysqld
    [2026-05-30 07:30:12] Auto-discovery: 8 local site(s) added from platform.
    [2026-05-30 07:30:12] Checking: https://mymoodle.mcgill.edu.au
      OK (342ms, HTTP 200)
    ...
    [2026-05-30 07:30:45] Done.

  If you see "HTTP 500" fetching the config:
    → Update the plugin to v1.1.2 (the path bug was fixed in this version).

  If you see "0 local site(s)" from the platform:
    → The Moodle site may not have registered yet. Force it now:
      php8.2 /var/www/moodle/mymoodle.mcgill.edu.au/admin/cli/scheduled_task.php \
        --execute="\local_downalert\task\register_with_platform"
    → Then re-run the monitor script.

  If you see "Invalid token":
    → Go back to Moodle → Site Admin → Plugins → Site Down Alert and copy the
      token again — make sure there are no extra spaces.

================================================================================
  STEP 4 — SET UP THE AUTOMATIC SCHEDULE (CRONTAB)
================================================================================

  A crontab makes the script run automatically every 3 minutes.

  Open the crontab editor:
    crontab -e

  The FIRST TIME it may ask which editor to use — type 1 and press Enter
  to choose nano (the easiest).

  Move to the VERY LAST LINE and add this (all on one line — no line breaks):

    */3 * * * * php8.2 /var/www/moodle/mymoodle.mcgill.edu.au/local/downalert/server-monitor/monitor.php --moodle-url=https://mymoodle.mcgill.edu.au --token=YOUR_HEALTH_TOKEN_HERE >> /var/log/sitedown.log 2>&1

  Replace:
    /var/www/moodle/mymoodle.mcgill.edu.au  → your actual Moodle root
    https://mymoodle.mcgill.edu.au          → your actual Moodle URL
    YOUR_HEALTH_TOKEN_HERE                  → your actual health token

  SAVE AND EXIT nano:
    Press  Ctrl + X
    Press  Y
    Press  Enter

  Confirm it saved:
    crontab -l

  You should see your line listed.

================================================================================
  STEP 5 — VERIFY IT IS RUNNING AUTOMATICALLY
================================================================================

  Wait 5 minutes, then check the log:
    tail -30 /var/log/sitedown.log

  You should see timestamped lines showing sites being checked.
  If the file is empty after 5 minutes:
    - Run the manual test from Step 3 again to confirm it works
    - Check crontab -l still shows your line

================================================================================
  MULTIPLE SERVERS — REPEAT FOR EACH ONE
================================================================================

  If your Moodle sites are on different servers:
    - Repeat Steps 1-5 on EACH server
    - Each server uses the token for the Moodle installed on that server
    - The monitor auto-detects which Moodles are on each server's disk

================================================================================
  WHAT HAPPENS WHEN A SITE GOES DOWN
================================================================================

  You will receive an email like:

    SITE DOWN ALERT -- moodle.clientname.edu.au
    ============================================================
    Site:      https://moodle.clientname.edu.au
    Time:      Thu, 30 May 2026 07:35:01 AEST
    Failure:   TIMEOUT -- No response after 10 seconds

    PRIMARY CAUSE:
    ============================================================
      PHP-FPM Worker Pool Exhausted

      All 20 PHP workers are busy. Requests are queuing and timing out.

    IMMEDIATE ACTION — SSH TO SERVER AND RUN:
    ------------------------------------------------------------
      1. sudo systemctl restart php8.2-fpm   # Immediate fix

    SERVER STATUS:
    ------------------------------------------------------------
      [OK]   Webserver    Running
      [WARN] Php_fpm      Running — pool saturated
      [OK]   Database     Running
      [WARN] Memory       88% used (210MB free of 2048MB)

================================================================================
  COMMON QUESTIONS
================================================================================

  Q: Do I need to set this up inside each Moodle?
  A: No. Install the plugin once per Moodle (it self-registers automatically).
     Set up the monitor once per SERVER. One server with 10 Moodles only needs
     one crontab line.

  Q: I have 3 servers. How many crontab entries do I need?
  A: Three — one on each server, each pointing to the Moodle on that server.

  Q: Can I test it without a real outage?
  A: Yes. Run the manual test in Step 3 pointing at a fake URL:
       --moodle-url=https://thisdomaindoesnotexist12345.com --token=...
     You won't get config (it will fail), but pointing it at a real Moodle and
     temporarily blocking port 443 on that site will trigger an alert.

  Q: How do I check the script is healthy?
  A: tail -50 /var/log/sitedown.log — all output lands there.

  Q: What if the whole server goes down?
  A: If the server is completely offline the monitor also goes offline and cannot
     send alerts. For full-server failure coverage, set up a secondary monitor on
     a DIFFERENT server with a manual 'sites' list in a local config.php pointing
     to the sites on the other server.

================================================================================
  NEED HELP?
================================================================================

  AI Grader support: https://lms-labs.com
  Plugin version: local_downalert v1.1.2

================================================================================
