/var/www/meran-kauf/AGENTS.md
21.9 KB
Agents.md
Agent instructions
⚠️ COMMIT RULE (mandatory): After every feature addition or change, make a commit on
mainwith a clear message that says what was implemented, e.g.Objekttyp: DB column + PATCH API + Typ dropdown. Use no feature branches. Never force-push, amend pushed commits, rebase, or otherwise rewrite history. The repo is/var/www/meran-kauf/.git(branchmain, no remote). History is visible in the app under Settings → Commits (/settings?section=commits).
🔌 VPS ACCESS RULE: From Grok Bot's Computer use
ssh netcup(direct, primary; details/home/box/shared-ssh/README.mdon that computer). Only if it fails (e.g. "Connection closed by <proxy-ip>" or "Outbound SSH is turned off for this assistant") fall back to the Mac route (SSH aliasnetcup_db_serveron Andreas's Mac).
⏱️ SCRAPER ON, EMAILS OFF (since 2026-11-11). Daily scrape continues; new-listings mail is disabled (
MERAN_NOTIFY_EMAIL_ENABLED=0). Cutoff still turns scraping OFF on 2027-01-01 00:00 Atlantic/Madeira. Switch:meran-scrape-ctl status|on|offandmeran-scrape-ctl email on|off. Full instructions:memory/automation-control.md.
👉 Start here:
memory/PROJECT.md— compact handover (purpose, hosting, sources, data model, core logic, ops, workflow, open issues, memory index). Also visible in the app under Settings → Agents / Memory .md (/settings?section=docs).
Git workflow
-
After implementing a new feature, always commit the changes to
mainormaster(use whichever default branch exists; prefermainif creating a new repo). -
The repository already exists (
/var/www/meran-kauf/.git, branchmain). Do not re-init it. - Commit on the default branch directly. Do not create feature branches for completed work unless the user explicitly asks.
- Never use other git worktrees. Stay in this workspace’s primary working tree only.
-
If no
.gitfolder is present, initialize a repository first (git init), then commit tomain.
Commit expectations
-
After a feature is implemented and working, stage relevant files and create a commit on
main/master. - Do not leave finished feature work uncommitted.
-
Never use additional worktrees (
git worktree addor similar).
Project memory (memory/)
Long-lived topic notes live in /var/www/meran-kauf/memory/ — one .md file per topic. Read the relevant file before continuing work on that topic; update it when you learn something durable. Do not dump large histories into this AGENTS.md; keep only the short index below.
| File | Topic |
|---|---|
memory/PROJECT.md | Handover overview — read first |
memory/project-overview.md | What Meran-Kauf is, live URL, stack, deploy habit |
memory/vps-ops.md | VPS paths, systemd, env, git location |
memory/gentle-scraping.md | Ramp-up playbook for new URLs / methods |
memory/idealista-scraper.md | Idealista / DataDome: tries, failures, pause, next tests |
memory/immoreal-scraper.md | Immoreal search scrape, stub hydration, lazy galleries |
memory/image-thumbs.md | Thumb vs full image URLs, hover-delay, Immoweb/Immoreal |
memory/listings-infinite-scroll.md | Paginated listings table, /api/listings, infinite scroll |
memory/listings-search.md | Free-text search over listings table + /api/listings?q= |
memory/wohnen-scraper.md | Wohnen in Südtirol AJAX scrape, thumbs, lazy gallery |
memory/duplicates.md | Cross-source duplicate candidates, merge/unmerge, canonical rows |
memory/url-clusters.md | Manual URL clusters (user-curated collections, not duplicates) |
memory/settings.md | Settings shell + configurable status funnel |
memory/backups.md | DB+code backups path, retention, timer, Settings UI |
memory/daily-scrape.md | Daily random scrape timer + new-listings email toggle (MERAN_NOTIFY_EMAIL_ENABLED, NOTIFY_*) |
memory/image-storage.md | Local copies of all listing images (listing_images, /media/, gentle downloader, Settings > Speicher) |
memory/objekttyp.md | Objekttyp column (Wohnung/Haus/Grundstück): DB, API, dropdown, merge carry |
memory/note.md | Manual note per listing: DB, API, inline editor |
memory/listings-sort.md | Header click-to-sort (?sort=&dir=), sort expressions, NULLs last |
memory/offline-listings.md | Background manual scrape (systemd + lock + /api/scrape/status), offline URL check (offline_at), not-seen marker |
Related scratch/todos (not the memory index): todos/idealist-scraper.md (Idealista pause notes; prefer the memory/ copy as canonical going forward).
Daily scrape + email
-
Scrape runs once per day at a random time between 06:00 and midnight Europe/Berlin (
meran-kauf-scrape.timer). -
When a scrape adds new listings, one email is sent from
grok@liftt.deonly ifMERAN_NOTIFY_EMAIL_ENABLEDis on (default ON when unset; currently OFF since 2026-11-11). SMTP viaNOTIFY_*in VPS.envonly — never commit secrets. Toggle:meran-scrape-ctl email on|off. -
Idealista remains paused. Details:
memory/daily-scrape.md. -
One result per unique URL:
db.upsert_listingkeys on(source, external_id)with a(source, url_key)fallback (normalized host+path), so regenerated portal ids never duplicate a known URL. Backfill viadb.dedupe_urls(). Details:memory/duplicates.md.
Gentle scraping (mandatory)
When scraping or testing a new URL / new method (especially Idealista or any site with bot protection):
- Propose first in chat — describe a gentle, human-like plan (one method, one URL, long pauses). Do not blast probes.
- Start tiny — one request, inspect result, then stop. Increase volume step by step only after something clearly worked.
- Behave like a human — low frequency, single client path, random short delays if paging, no parallel hammering, abort immediately on 403 / captcha / empty block pages.
- Never run “try every UA / tool / fingerprint” matrices from the VPS IP in one session; that soft-bans the IP and blocks even good methods.
-
Follow the ramp-up detail in
memory/gentle-scraping.md. For Idealista specifics and current pause state, seememory/idealista-scraper.md.
Images in the listings table (Bild)
Be careful how many images the UI loads at once. Full-resolution photos (especially Immoreal) are large; loading every gallery image for every row makes the table slow.
-
Bild column: small thumbs only (
image_thumb_url/ Immoweb_pre/ Immoreal_processed_). -
Hover preview: set full-res
srconly on hover — never preload big images for every row. -
Do not wire the table to download whole galleries on page load. Details:
memory/image-thumbs.md. -
Local copies (
/media/…) are preferred when stored (display_images/display_thumb); originals are the fallback. Details:memory/image-storage.md.
UI reference IDs (data-ref)
When adding or changing UI:
-
Give main page landmarks a stable
idand matchingdata-ref(kebab-case), e.g.id="filters" data-ref="filters". -
Always give every button (and filter chip / primary control that acts like a button) a
data-ref. -
Always give main elements inside each listing row a
data-ref, including: the row itself, status cell/matrix, each status button, gallery + gallery link, title link, meta, and the main value columns (price, sizes, first-seen). -
For repeated row elements, include the listing DB id in the ref so it is unique and copyable, e.g.
data-ref="status-merken-42",data-ref="listing-42",data-ref="gallery-42". Status slug viaconfig.status_ref_slug: spaces///()→ hyphens (in kontakt→in-kontakt,gesendet / waiting→gesendet-waiting,OUT (war in kontakt)→out-war-in-kontakt);OUT→out(status-out-42). -
Filter chips:
filter-ALL,filter-neu,filter-gesendet-waiting,filter-in-kontakt,filter-out-war-in-kontakt,filter-out, etc. -
Do not tag tiny decorative bits (raw
<img>inside gallery, count badges, table header cells) unless they become interactive controls. -
Prefer the selector form agents and humans copy in chat:
[data-ref="status-merken-42"]. - Keep names stable across deploys; rename only when the UI meaning changes.
-
Local inspect mode: press
eto enter/exit; thedata-reflabel appears only on the hovered element; click copies[data-ref="…"]to the clipboard. (Ignored while typing in inputs.)
Page landmarks: header, brand, header-path (VPS path, main + settings), header-drive-link (Google Drive „Kauf“ folder, new tab), btn-sources-toggle (header path row, right of Drive link; toggles sources-panel as a full-width row below the header, aria-expanded, state in localStorage meranSourcesOpen; auto-opens while a scrape reports status), header-actions, neu-badge, btn-settings, btn-url-check („URLs prüfen“, detached offline check via POST /api/url-check, polls /api/url-check/status), btn-scrape (starts a background scrape, shows „Scrape läuft…“ and polls /api/scrape/status; progress + summary go to scrape-status), sources-panel, source-urls, last-runs, scrape-status, filters, manual-merge-panel, merge-manual, merge-ids-input, btn-merge-load, merge-primary-picker, btn-merge-confirm, listings-main, listings-table-wrap, listings-table, listings-tbody, listings-load-more, listings-end, search-wrap, listings-search, search-result-count, content-wrap (wraps filters + manual-merge-panel + listings-main on /, incl. Duplikate view; width = CSS var --content-width in :root of app/static/app.css, default 1400px; settings page uses --settings-width, default 1100px), duplicates-view, dup-toolbar, btn-dup-refresh, ref-toast, settings-layout, settings-nav, settings-funnel, funnel-editor, btn-funnel-save, btn-funnel-add, status-rules, status-rule-0, rule-0-enabled, rule-0-value, rule-0-target, btn-rules-save, rules-save-status, settings-backups, btn-backup-run, backup-table, settings-storage, storage-path, storage-example-path, storage-stats, btn-storage-fetch, btn-storage-refresh, storage-fetch-status, storage-last-fetch, status-context-menu, funnel-config.
Settings → Agents / Memory .md (/settings?section=docs, nav settings-nav-docs): panel settings-docs, blocks doc-agents / doc-project with doc-path-agents / doc-path-project (absolute path), doc-copy-{key} (copy path), doc-toggle-{key} (rendered ↔ raw), doc-content-agents / doc-content-project (rendered Markdown), doc-raw-{key}. Server: app/docs.py, fixed whitelist DOCS (no path parameter), escape-first mini Markdown renderer (raw HTML never passes through; only http(s)/mailto links become <a>).
Settings → Commits (/settings?section=commits, nav settings-nav-commits): panel settings-commits, table commits-table / commits-tbody, rows commit-row-{shortHash} (data-hash = full hash), body <details> commit-body-{shortHash}, header info commits-branch / commits-total, paging btn-commits-more + commits-more-status, error commits-error. API GET /api/settings/commits?offset=&limit= (200 per page). Server: app/gitlog.py runs git log with an 8 s timeout and safe.directory only via the subprocess env (GIT_CONFIG_COUNT), because .git is root-owned while gunicorn runs as www-data.
Filter chips include filter-duplicates (Duplikate).
Duplikate members: dup-member-{id}, dup-primary-{id}, dup-img-{id} (plus shared gallery-{id} / gallery-link-{id}). Candidate groups: btn-dup-merge-{n}, btn-dup-dismiss-{n} („Keine Dublette“ — whole group), reason chips dup-reason-{n}-{k}. Section filters in dup-toolbar: dup-filters, dup-filter-A … dup-filter-E with counts dup-filter-count-A … -E (URL ?status=duplicates&dup=C, localStorage meranDupSection); empty state empty-dup-section. Groups carry data-sections. Popup (all pages, bottom-right, grey bg + amber border/stripe): dup-popup, dup-popup-count, dup-popup-link, btn-dup-popup-close. Merge popup (bottom-center, same styling language): merge-popup, merge-popup-title, inputs merge-popup-primary (single primary ID) + merge-popup-others (other IDs, comma/space separated), btn-merge-popup-save, btn-merge-popup-close, merge-popup-status, btn-merge-popup-open (reopen pill, shown while popup is closed). Save POSTs {ids:[primary,…others], primary_id, enrich:true} to /api/duplicates/merge (reuses parseIdList), then reloads. URL clusters (2026-09-29, manual collections, NOT duplicates): bottom popup url-cluster-popup (teal) with target select url-cluster-target, name url-cluster-name, URL textarea url-cluster-urls, btn-url-cluster-popup-save, url-cluster-popup-status, btn-url-cluster-popup-close, reopen pill btn-url-cluster-popup-open; filters button btn-url-clusters (count url-clusters-count) toggles the cluster table view (groups in #listings-table, A-Z, drag & drop; group heads cluster-group-head-{id|none} with btn-cluster-del-{id}); regular table has trailing Cluster column col-head-cluster with per-row dropdown cluster-select-{id} (name cluster-name-{id} in cluster view). APIs: GET/POST /api/url-clusters, POST /api/url-clusters/move, DELETE /api/url-clusters/<id>. One URL per cluster max (UNIQUE url_key). Details: memory/url-clusters.md. Merged groups: collapsed <details>, btn-dup-set-primary-{n} to change primary, btn-dup-unmerge-{n}. Selective merge (2026-09-29): per-group btn-dup-edit-select-{n} toggles selection mode (group class is-selecting; checkboxes dup-select-{id}; sticky bar dup-select-bar-{n} with dup-select-count-{n}, primary select dup-select-primary-{n}, enrich dup-select-enrich-{n}, btn-dup-select-merge-{n}, btn-dup-select-cancel-{n}); merges only the checked listings (POST /api/duplicates/selective-merge {ids, selected, primary_id, enrich}) and dismisses every other pair among the group (kept-vs-kept included). Card right-click menu dup-card-ctx: dup-ctx-select-{id} / dup-ctx-unique-{id} / dup-ctx-primary-{id} / dup-ctx-open-{id}. Dismissed pairs section dup-dismissed with dup-dismissed-list, dup-dismissed-item-{a}-{b}, btn-dup-undismiss-{a}-{b}; notice toast dup-notice with dup-notice-undo / dup-notice-close. POST /api/duplicates/dismiss also accepts {pairs:[[a,b],...]} (max 200); GET /api/duplicates/dismissals. Group status sync (2026-09-29): the primary's status is the group's status — merge_listings pushes the primary's final status onto all siblings (result field siblings_status_updated); PATCH /api/listings/{id}/status also syncs merged siblings (response field siblings_updated); Duplikate-view JS mirrors the primary's new status onto sibling dup-status-{id} selects + dup-meta-status-{id} text. Details: memory/duplicates.md. Tags (2026-09-29, sub-filter under the status funnel): settings section tags (nav settings-nav-tags, panel settings-tags), API /api/tags + /api/listings/{id}/tags, ?tag=<id> on / and /api/listings, pills tag-pills-{id} + editor tag-edit-{id} in the title column, filter row tag-filter-row in #filters. Details: memory/tags.md. Row context menu (2026-09-29, right-click on tr[data-id] in #listings-table): row-ctx with Delete (row-ctx-delete-<id>, SOFT delete since 2026-10-01: sets listings.deleted_at on that one id only (not relevant anymore); row stays in DB, hidden from all table views; DELETE /api/listings/<id>; merge/dup logic untouched; details: memory/soft-delete.md) and Tag (row-ctx-tag-<id>) with a hover submenu (row-ctx-sub) listing all tags for toggle (row-ctx-tagopt-<tagId>). Sources status badges in sources-panel inside header.
Per listing {id}: listing-{id}, col-id-{id}, col-status-{id}, status-cell-{id}, status-matrix-{id}, status-out-{id}, status-out-war-in-kontakt-{id}, status-neu-{id}, status-merken-{id}, status-kontaktieren-{id}, status-gesendet-waiting-{id}, status-in-kontakt-{id}, status-gesehen-{id}, status-pill-{id}, col-img-{id}, gallery-{id}, gallery-link-{id}, gallery-prev-{id}, gallery-next-{id}, col-title-{id}, listing-title-{id}, listing-meta-{id}, offline-badge-{id} (red „offline“ in the meta line: URL answered 404/410, tooltip with HTTP status + dates; see memory/offline-listings.md), not-seen-badge-{id} (grey „nicht mehr gesehen seit DD.MM.“: last_seen_at older than the latest error-free run of the source; not for paused sources; shown next to offline too), also-on-{id}, col-source-{id}, listing-source-{id}, source-search-{id}, source-sibling-{id}, source-sibling-link-{id}, col-first-seen-{id}, col-price-{id}, col-grundstueck-{id}, col-wohnflaeche-{id}, dup-count-btn-{id} (Titel column, number of merged siblings; hover/click opens dup-popover-{id} with dup-popover-item-{sid} + dup-popover-link-{sid} → in-app link /?status=duplicates&dup=A&merged={id}&focus={sid}#merged-{id}; merged groups on the Duplikate page carry id="merged-{primaryId}", members data-member-id, deep link highlights .dup-focus / .dup-focus-member), col-drive-{id} (Drive cell: link drive-{id}, edit/add button drive-edit-{id}, inline editor drive-input-{id} / drive-save-{id} / drive-cancel-{id}; header col-head-drive). Duplikate cards: dup-drive-cell-{id} with dup-drive-{id}, dup-drive-edit-{id}, dup-drive-input-{id}. API PATCH /api/listings/{id}/drive {drive_url} ("" clears). col-objekttyp-{id} (column „Typ“, far right after Drive; header col-head-objekttyp) with dropdown objekttyp-select-{id} (— / Wohnung / Haus / Grundstück, saves on change; feedback classes .is-saving / .is-saved / .is-error). DB listings.objekttyp ENUM NULL; API PATCH /api/listings/{id}/objekttyp {objekttyp} ("" or null clears, other values → 400). Merge copies a sibling objekttyp onto a NULL primary. Details: memory/objekttyp.md. col-note-{id} (column „Notiz“, far right after Typ; header col-head-note) with text note-{id} (title = full text, CSS clamps to 3 lines) + edit/add button note-edit-{id}, inline editor note-input-{id} (textarea) / note-save-{id} / note-cancel-{id} (save: Ctrl+Enter or ✓; empty clears). DB listings.note TEXT NULL; API PATCH /api/listings/{id}/note {note} ("" or null clears, max 2000 chars). Details: memory/note.md. col-rolle-{id} (column „Rolle“, far right after Notiz; header col-head-rolle) with badge rolle-{id}: „Primär“ (green, merged_into_id IS NULL) or „Sekundär → #<id>“ (amber, merged_into_id set). Main table shows canonical rows only, so currently all rows are Primär; the badge also covers merged rows if they ever appear. JS twin: meranRolle.cellHtml() in app.js. Header DB-column badges (thead of #listings-main, part A, no sorting yet): every th.th-db has data-db-col, a label span col-label-{key} (keep free for future click-to-sort) and a hover/focus-only badge col-dbname-{key} with a copy button col-dbname-copy-{key} (clipboard API, textarea/execCommand fallback, "kopiert" feedback; data-copied-via = clipboard|fallback|failed). Badge clicks and mousedowns stop propagation. Click-to-sort (part B): sortable headers render the label as a link a.th-sort (col-label-{key}, data-sort-key) with indicator sort-ind-{key} (▲ asc / ▼ desc); cycle asc → desc → default. Server-side via ?sort={key}&dir=asc|desc (kept in status-chip hrefs, infinite scroll and live search via #listings-table[data-sort][data-dir]; /api/listings accepts the same params). NULLs/empty always last. Sortable: id, title, source, status (alphabetical), first_seen_at (date), price_text (sorts by price_num), wohnflaeche (parsed numerically from the VARCHAR), gesamtflaeche, grundstueck, objekttyp. Not sortable: image_thumb_url (Bild), drive_url (Drive), note (Notiz), merged_into_id (Rolle, derived badge). Details: memory/listings-sort.md. Keys = real DB columns: id, status, image_thumb_url (Bild; the displayed local copy is listing_images.local_thumb_path), title, source (Quelle; search URL = search_url), first_seen_at, price_text (numeric: price_num), wohnflaeche, gesamtflaeche, grundstueck, drive_url, objekttyp, note, merged_into_id (Rolle badge, derived from merged_into_id IS NULL). Header data-refs: col-head-id, col-head-status, col-head-bild, col-head-titel, col-head-quelle, col-head-first-seen, col-head-preis, col-head-note, col-head-rolle (+ the ones below). Table headers: col-head-grundstueck (Grundstück, numeric DB column grundstueck), col-head-gesamtflaeche (cells col-gesamtflaeche-{id}; DB gesamtflaeche = Wohnfläche + Nebenflächen, without Grundstück), col-head-wohnflaeche. Duplikate cards: dup-grundstueck-{id}, dup-gesamtflaeche-{id} (next to dup-meta-{id}). Per-source area label mapping: app/source_fields.py (SOURCE_FIELD_MAP). Immoweb areas (Netto → Wohnfläche, Grundstück, Gesamtfläche) are documented in memory/immoweb-areas.md.
# Agents.md
# Agent instructions
> **⚠️ COMMIT RULE (mandatory):** After **every** feature addition or change, make a commit on **`main`** with a clear message that says **what was implemented**, e.g. `Objekttyp: DB column + PATCH API + Typ dropdown`. Use **no feature branches**. **Never force-push, amend pushed commits, rebase, or otherwise rewrite history.** The repo is `/var/www/meran-kauf/.git` (branch `main`, no remote). History is visible in the app under **Settings → Commits** (`/settings?section=commits`).
> **🔌 VPS ACCESS RULE:** From Grok Bot's Computer use **`ssh netcup`** (direct, primary; details `/home/box/shared-ssh/README.md` on that computer). Only if it fails (e.g. "Connection closed by <proxy-ip>" or "Outbound SSH is turned off for this assistant") fall back to the Mac route (SSH alias `netcup_db_server` on Andreas's Mac).
> **⏱️ SCRAPER ON, EMAILS OFF (since 2026-11-11).** Daily scrape continues; new-listings mail is disabled (`MERAN_NOTIFY_EMAIL_ENABLED=0`). Cutoff still turns scraping OFF on 2027-01-01 00:00 Atlantic/Madeira. Switch: `meran-scrape-ctl status|on|off` and `meran-scrape-ctl email on|off`. Full instructions: [`memory/automation-control.md`](memory/automation-control.md).
> **👉 Start here: [`memory/PROJECT.md`](memory/PROJECT.md)** — compact handover (purpose, hosting, sources, data model, core logic, ops, workflow, open issues, memory index). Also visible in the app under **Settings → Agents / Memory .md** (`/settings?section=docs`).
## Git workflow
- After implementing a new feature, always commit the changes to `main` or `master` (use whichever default branch exists; prefer `main` if creating a new repo).
- The repository already exists (`/var/www/meran-kauf/.git`, branch `main`). Do **not** re-init it.
- Commit on the default branch directly. Do not create feature branches for completed work unless the user explicitly asks.
- Never use other git worktrees. Stay in this workspace’s primary working tree only.
- If no `.git` folder is present, initialize a repository first (`git init`), then commit to `main`.
## Commit expectations
- After a feature is implemented and working, stage relevant files and create a commit on `main`/`master`.
- Do not leave finished feature work uncommitted.
- Never use additional worktrees (`git worktree add` or similar).
## Project memory (`memory/`)
Long-lived topic notes live in `/var/www/meran-kauf/memory/` — **one `.md` file per topic**. Read the relevant file before continuing work on that topic; update it when you learn something durable. Do not dump large histories into this `AGENTS.md`; keep only the short index below.
| File | Topic |
|------|--------|
| [`memory/PROJECT.md`](memory/PROJECT.md) | **Handover overview — read first** |
| [`memory/project-overview.md`](memory/project-overview.md) | What Meran-Kauf is, live URL, stack, deploy habit |
| [`memory/vps-ops.md`](memory/vps-ops.md) | VPS paths, systemd, env, git location |
| [`memory/gentle-scraping.md`](memory/gentle-scraping.md) | Ramp-up playbook for new URLs / methods |
| [`memory/idealista-scraper.md`](memory/idealista-scraper.md) | Idealista / DataDome: tries, failures, pause, next tests |
| [`memory/immoreal-scraper.md`](memory/immoreal-scraper.md) | Immoreal search scrape, stub hydration, lazy galleries |
| [`memory/image-thumbs.md`](memory/image-thumbs.md) | Thumb vs full image URLs, hover-delay, Immoweb/Immoreal |
| [`memory/listings-infinite-scroll.md`](memory/listings-infinite-scroll.md) | Paginated listings table, `/api/listings`, infinite scroll |
| [`memory/listings-search.md`](memory/listings-search.md) | Free-text search over listings table + `/api/listings?q=` |
| [`memory/wohnen-scraper.md`](memory/wohnen-scraper.md) | Wohnen in Südtirol AJAX scrape, thumbs, lazy gallery |
| [`memory/duplicates.md`](memory/duplicates.md) | Cross-source duplicate candidates, merge/unmerge, canonical rows |
| [`memory/url-clusters.md`](memory/url-clusters.md) | Manual URL clusters (user-curated collections, not duplicates) |
| [`memory/settings.md`](memory/settings.md) | Settings shell + configurable status funnel |
| [`memory/backups.md`](memory/backups.md) | DB+code backups path, retention, timer, Settings UI |
| [`memory/daily-scrape.md`](memory/daily-scrape.md) | Daily random scrape timer + new-listings email toggle (`MERAN_NOTIFY_EMAIL_ENABLED`, NOTIFY_*) |
| [`memory/image-storage.md`](memory/image-storage.md) | Local copies of all listing images (`listing_images`, `/media/`, gentle downloader, Settings > Speicher) |
| [`memory/objekttyp.md`](memory/objekttyp.md) | Objekttyp column (Wohnung/Haus/Grundstück): DB, API, dropdown, merge carry |
| [`memory/note.md`](memory/note.md) | Manual note per listing: DB, API, inline editor |
| [`memory/listings-sort.md`](memory/listings-sort.md) | Header click-to-sort (`?sort=&dir=`), sort expressions, NULLs last |
| [`memory/offline-listings.md`](memory/offline-listings.md) | Background manual scrape (systemd + lock + `/api/scrape/status`), offline URL check (`offline_at`), not-seen marker |
Related scratch/todos (not the memory index): `todos/idealist-scraper.md` (Idealista pause notes; prefer the `memory/` copy as canonical going forward).
## Daily scrape + email
- Scrape runs **once per day** at a **random** time between **06:00 and midnight** Europe/Berlin (`meran-kauf-scrape.timer`).
- When a scrape **adds** new listings, one email is sent from `grok@liftt.de` **only if** `MERAN_NOTIFY_EMAIL_ENABLED` is on (default ON when unset; currently OFF since 2026-11-11). SMTP via `NOTIFY_*` in VPS `.env` only — never commit secrets. Toggle: `meran-scrape-ctl email on|off`.
- Idealista remains **paused**. Details: [`memory/daily-scrape.md`](memory/daily-scrape.md).
- One result per unique URL: `db.upsert_listing` keys on `(source, external_id)` with a `(source, url_key)` fallback (normalized host+path), so regenerated portal ids never duplicate a known URL. Backfill via `db.dedupe_urls()`. Details: [`memory/duplicates.md`](memory/duplicates.md).
## Gentle scraping (mandatory)
When scraping or **testing a new URL / new method** (especially Idealista or any site with bot protection):
1. **Propose first in chat** — describe a gentle, human-like plan (one method, one URL, long pauses). Do not blast probes.
2. **Start tiny** — one request, inspect result, then stop. Increase volume **step by step** only after something clearly worked.
3. **Behave like a human** — low frequency, single client path, random short delays if paging, no parallel hammering, **abort immediately on 403 / captcha / empty block pages**.
4. **Never** run “try every UA / tool / fingerprint” matrices from the VPS IP in one session; that soft-bans the IP and blocks even good methods.
5. Follow the ramp-up detail in [`memory/gentle-scraping.md`](memory/gentle-scraping.md). For Idealista specifics and current pause state, see [`memory/idealista-scraper.md`](memory/idealista-scraper.md).
## Images in the listings table (Bild)
Be careful how many images the UI loads at once. Full-resolution photos (especially Immoreal) are large; loading every gallery image for every row makes the table slow.
- **Bild column:** small thumbs only (`image_thumb_url` / Immoweb `_pre` / Immoreal `_processed_`).
- **Hover preview:** set full-res `src` only on hover — never preload big images for every row.
- Do not wire the table to download whole galleries on page load. Details: [`memory/image-thumbs.md`](memory/image-thumbs.md).
- Local copies (`/media/…`) are preferred when stored (`display_images` / `display_thumb`); originals are the fallback. Details: [`memory/image-storage.md`](memory/image-storage.md).
## UI reference IDs (`data-ref`)
When adding or changing UI:
1. Give **main page landmarks** a stable `id` **and** matching `data-ref` (kebab-case), e.g. `id="filters" data-ref="filters"`.
2. **Always** give every **button** (and filter chip / primary control that acts like a button) a `data-ref`.
3. **Always** give **main elements inside each listing row** a `data-ref`, including: the row itself, status cell/matrix, each status button, gallery + gallery link, title link, meta, and the main value columns (price, sizes, first-seen).
4. For repeated row elements, include the listing DB id in the ref so it is unique and copyable, e.g. `data-ref="status-merken-42"`, `data-ref="listing-42"`, `data-ref="gallery-42"`. Status slug via `config.status_ref_slug`: spaces/`/`/`()` → hyphens (`in kontakt` → `in-kontakt`, `gesendet / waiting` → `gesendet-waiting`, `OUT (war in kontakt)` → `out-war-in-kontakt`); `OUT` → `out` (`status-out-42`).
5. Filter chips: `filter-ALL`, `filter-neu`, `filter-gesendet-waiting`, `filter-in-kontakt`, `filter-out-war-in-kontakt`, `filter-out`, etc.
6. Do **not** tag tiny decorative bits (raw `<img>` inside gallery, count badges, table header cells) unless they become interactive controls.
7. Prefer the selector form agents and humans copy in chat: `[data-ref="status-merken-42"]`.
8. Keep names stable across deploys; rename only when the UI meaning changes.
9. Local inspect mode: press **`e`** to enter/exit; the `data-ref` label appears **only on the hovered** element; click copies `[data-ref="…"]` to the clipboard. (Ignored while typing in inputs.)
Page landmarks: `header`, `brand`, `header-path` (VPS path, main + settings), `header-drive-link` (Google Drive „Kauf“ folder, new tab), `btn-sources-toggle` (header path row, right of Drive link; toggles `sources-panel` as a full-width row below the header, `aria-expanded`, state in localStorage `meranSourcesOpen`; auto-opens while a scrape reports status), `header-actions`, `neu-badge`, `btn-settings`, `btn-url-check` („URLs prüfen“, detached offline check via `POST /api/url-check`, polls `/api/url-check/status`), `btn-scrape` (starts a background scrape, shows „Scrape läuft…“ and polls `/api/scrape/status`; progress + summary go to `scrape-status`), `sources-panel`, `source-urls`, `last-runs`, `scrape-status`, `filters`, `manual-merge-panel`, `merge-manual`, `merge-ids-input`, `btn-merge-load`, `merge-primary-picker`, `btn-merge-confirm`, `listings-main`, `listings-table-wrap`, `listings-table`, `listings-tbody`, `listings-load-more`, `listings-end`, `search-wrap`, `listings-search`, `search-result-count`, `content-wrap` (wraps `filters` + `manual-merge-panel` + `listings-main` on `/`, incl. Duplikate view; width = CSS var `--content-width` in `:root` of `app/static/app.css`, default 1400px; settings page uses `--settings-width`, default 1100px), `duplicates-view`, `dup-toolbar`, `btn-dup-refresh`, `ref-toast`, `settings-layout`, `settings-nav`, `settings-funnel`, `funnel-editor`, `btn-funnel-save`, `btn-funnel-add`, `status-rules`, `status-rule-0`, `rule-0-enabled`, `rule-0-value`, `rule-0-target`, `btn-rules-save`, `rules-save-status`, `settings-backups`, `btn-backup-run`, `backup-table`, `settings-storage`, `storage-path`, `storage-example-path`, `storage-stats`, `btn-storage-fetch`, `btn-storage-refresh`, `storage-fetch-status`, `storage-last-fetch`, `status-context-menu`, `funnel-config`.
Settings → Agents / Memory .md (`/settings?section=docs`, nav `settings-nav-docs`): panel `settings-docs`, blocks `doc-agents` / `doc-project` with `doc-path-agents` / `doc-path-project` (absolute path), `doc-copy-{key}` (copy path), `doc-toggle-{key}` (rendered ↔ raw), `doc-content-agents` / `doc-content-project` (rendered Markdown), `doc-raw-{key}`. Server: `app/docs.py`, fixed whitelist `DOCS` (no path parameter), escape-first mini Markdown renderer (raw HTML never passes through; only http(s)/mailto links become `<a>`).
Settings → Commits (`/settings?section=commits`, nav `settings-nav-commits`): panel `settings-commits`, table `commits-table` / `commits-tbody`, rows `commit-row-{shortHash}` (`data-hash` = full hash), body `<details>` `commit-body-{shortHash}`, header info `commits-branch` / `commits-total`, paging `btn-commits-more` + `commits-more-status`, error `commits-error`. API `GET /api/settings/commits?offset=&limit=` (200 per page). Server: `app/gitlog.py` runs `git log` with an 8 s timeout and `safe.directory` only via the subprocess env (`GIT_CONFIG_COUNT`), because `.git` is root-owned while gunicorn runs as `www-data`.
Filter chips include `filter-duplicates` (Duplikate).
Duplikate members: `dup-member-{id}`, `dup-primary-{id}`, `dup-img-{id}` (plus shared `gallery-{id}` / `gallery-link-{id}`). Candidate groups: `btn-dup-merge-{n}`, `btn-dup-dismiss-{n}` („Keine Dublette“ — whole group), reason chips `dup-reason-{n}-{k}`. Section filters in `dup-toolbar`: `dup-filters`, `dup-filter-A` … `dup-filter-E` with counts `dup-filter-count-A` … `-E` (URL `?status=duplicates&dup=C`, localStorage `meranDupSection`); empty state `empty-dup-section`. Groups carry `data-sections`. Popup (all pages, bottom-right, grey bg + amber border/stripe): `dup-popup`, `dup-popup-count`, `dup-popup-link`, `btn-dup-popup-close`. Merge popup (bottom-center, same styling language): `merge-popup`, `merge-popup-title`, inputs `merge-popup-primary` (single primary ID) + `merge-popup-others` (other IDs, comma/space separated), `btn-merge-popup-save`, `btn-merge-popup-close`, `merge-popup-status`, `btn-merge-popup-open` (reopen pill, shown while popup is closed). Save POSTs `{ids:[primary,…others], primary_id, enrich:true}` to `/api/duplicates/merge` (reuses `parseIdList`), then reloads. URL clusters (2026-09-29, manual collections, NOT duplicates): bottom popup `url-cluster-popup` (teal) with target select `url-cluster-target`, name `url-cluster-name`, URL textarea `url-cluster-urls`, `btn-url-cluster-popup-save`, `url-cluster-popup-status`, `btn-url-cluster-popup-close`, reopen pill `btn-url-cluster-popup-open`; filters button `btn-url-clusters` (count `url-clusters-count`) toggles the cluster table view (groups in `#listings-table`, A-Z, drag & drop; group heads `cluster-group-head-{id|none}` with `btn-cluster-del-{id}`); regular table has trailing Cluster column `col-head-cluster` with per-row dropdown `cluster-select-{id}` (name `cluster-name-{id}` in cluster view). APIs: `GET`/`POST /api/url-clusters`, `POST /api/url-clusters/move`, `DELETE /api/url-clusters/<id>`. One URL per cluster max (UNIQUE url_key). Details: [`memory/url-clusters.md`](memory/url-clusters.md). Merged groups: collapsed `<details>`, `btn-dup-set-primary-{n}` to change primary, `btn-dup-unmerge-{n}`. Selective merge (2026-09-29): per-group `btn-dup-edit-select-{n}` toggles selection mode (group class `is-selecting`; checkboxes `dup-select-{id}`; sticky bar `dup-select-bar-{n}` with `dup-select-count-{n}`, primary select `dup-select-primary-{n}`, enrich `dup-select-enrich-{n}`, `btn-dup-select-merge-{n}`, `btn-dup-select-cancel-{n}`); merges only the checked listings (`POST /api/duplicates/selective-merge` `{ids, selected, primary_id, enrich}`) and dismisses every other pair among the group (kept-vs-kept included). Card right-click menu `dup-card-ctx`: `dup-ctx-select-{id}` / `dup-ctx-unique-{id}` / `dup-ctx-primary-{id}` / `dup-ctx-open-{id}`. Dismissed pairs section `dup-dismissed` with `dup-dismissed-list`, `dup-dismissed-item-{a}-{b}`, `btn-dup-undismiss-{a}-{b}`; notice toast `dup-notice` with `dup-notice-undo` / `dup-notice-close`. `POST /api/duplicates/dismiss` also accepts `{pairs:[[a,b],...]}` (max 200); `GET /api/duplicates/dismissals`. Group status sync (2026-09-29): the primary's status is the group's status — `merge_listings` pushes the primary's final status onto all siblings (result field `siblings_status_updated`); `PATCH /api/listings/{id}/status` also syncs merged siblings (response field `siblings_updated`); Duplikate-view JS mirrors the primary's new status onto sibling `dup-status-{id}` selects + `dup-meta-status-{id}` text. Details: [`memory/duplicates.md`](memory/duplicates.md). Tags (2026-09-29, sub-filter under the status funnel): settings section `tags` (nav `settings-nav-tags`, panel `settings-tags`), API `/api/tags` + `/api/listings/{id}/tags`, `?tag=<id>` on `/` and `/api/listings`, pills `tag-pills-{id}` + editor `tag-edit-{id}` in the title column, filter row `tag-filter-row` in `#filters`. Details: [`memory/tags.md`](memory/tags.md). Row context menu (2026-09-29, right-click on `tr[data-id]` in `#listings-table`): `row-ctx` with **Delete** (`row-ctx-delete-<id>`, SOFT delete since 2026-10-01: sets `listings.deleted_at` on that one id only (not relevant anymore); row stays in DB, hidden from all table views; `DELETE /api/listings/<id>`; merge/dup logic untouched; details: [`memory/soft-delete.md`](memory/soft-delete.md)) and **Tag** (`row-ctx-tag-<id>`) with a hover submenu (`row-ctx-sub`) listing all tags for toggle (`row-ctx-tagopt-<tagId>`). Sources status badges in `sources-panel` inside `header`.
Per listing `{id}`: `listing-{id}`, `col-id-{id}`, `col-status-{id}`, `status-cell-{id}`, `status-matrix-{id}`, `status-out-{id}`, `status-out-war-in-kontakt-{id}`, `status-neu-{id}`, `status-merken-{id}`, `status-kontaktieren-{id}`, `status-gesendet-waiting-{id}`, `status-in-kontakt-{id}`, `status-gesehen-{id}`, `status-pill-{id}`, `col-img-{id}`, `gallery-{id}`, `gallery-link-{id}`, `gallery-prev-{id}`, `gallery-next-{id}`, `col-title-{id}`, `listing-title-{id}`, `listing-meta-{id}`, `offline-badge-{id}` (red „offline“ in the meta line: URL answered 404/410, tooltip with HTTP status + dates; see [`memory/offline-listings.md`](memory/offline-listings.md)), `not-seen-badge-{id}` (grey „nicht mehr gesehen seit DD.MM.“: `last_seen_at` older than the latest error-free run of the source; not for paused sources; shown next to `offline` too), `also-on-{id}`, `col-source-{id}`, `listing-source-{id}`, `source-search-{id}`, `source-sibling-{id}`, `source-sibling-link-{id}`, `col-first-seen-{id}`, `col-price-{id}`, `col-grundstueck-{id}`, `col-wohnflaeche-{id}`, `dup-count-btn-{id}` (Titel column, number of merged siblings; hover/click opens `dup-popover-{id}` with `dup-popover-item-{sid}` + `dup-popover-link-{sid}` → in-app link `/?status=duplicates&dup=A&merged={id}&focus={sid}#merged-{id}`; merged groups on the Duplikate page carry `id="merged-{primaryId}"`, members `data-member-id`, deep link highlights `.dup-focus` / `.dup-focus-member`), `col-drive-{id}` (Drive cell: link `drive-{id}`, edit/add button `drive-edit-{id}`, inline editor `drive-input-{id}` / `drive-save-{id}` / `drive-cancel-{id}`; header `col-head-drive`). Duplikate cards: `dup-drive-cell-{id}` with `dup-drive-{id}`, `dup-drive-edit-{id}`, `dup-drive-input-{id}`. API `PATCH /api/listings/{id}/drive` `{drive_url}` ("" clears). `col-objekttyp-{id}` (column „Typ“, far right after Drive; header `col-head-objekttyp`) with dropdown `objekttyp-select-{id}` (— / Wohnung / Haus / Grundstück, saves on change; feedback classes `.is-saving` / `.is-saved` / `.is-error`). DB `listings.objekttyp` ENUM NULL; API `PATCH /api/listings/{id}/objekttyp` `{objekttyp}` ("" or null clears, other values → 400). Merge copies a sibling objekttyp onto a NULL primary. Details: [`memory/objekttyp.md`](memory/objekttyp.md). `col-note-{id}` (column „Notiz“, far right after Typ; header `col-head-note`) with text `note-{id}` (title = full text, CSS clamps to 3 lines) + edit/add button `note-edit-{id}`, inline editor `note-input-{id}` (textarea) / `note-save-{id}` / `note-cancel-{id}` (save: Ctrl+Enter or ✓; empty clears). DB `listings.note` TEXT NULL; API `PATCH /api/listings/{id}/note` `{note}` ("" or null clears, max 2000 chars). Details: [`memory/note.md`](memory/note.md). `col-rolle-{id}` (column „Rolle“, far right after Notiz; header `col-head-rolle`) with badge `rolle-{id}`: „Primär“ (green, `merged_into_id IS NULL`) or „Sekundär → #<id>“ (amber, `merged_into_id` set). Main table shows canonical rows only, so currently all rows are Primär; the badge also covers merged rows if they ever appear. JS twin: `meranRolle.cellHtml()` in app.js. Header DB-column badges (thead of `#listings-main`, part A, no sorting yet): every `th.th-db` has `data-db-col`, a label span `col-label-{key}` (keep free for future click-to-sort) and a hover/focus-only badge `col-dbname-{key}` with a copy button `col-dbname-copy-{key}` (clipboard API, textarea/execCommand fallback, "kopiert" feedback; `data-copied-via` = clipboard|fallback|failed). Badge clicks and mousedowns stop propagation. Click-to-sort (part B): sortable headers render the label as a link `a.th-sort` (`col-label-{key}`, `data-sort-key`) with indicator `sort-ind-{key}` (▲ asc / ▼ desc); cycle asc → desc → default. Server-side via `?sort={key}&dir=asc|desc` (kept in status-chip hrefs, infinite scroll and live search via `#listings-table[data-sort][data-dir]`; `/api/listings` accepts the same params). NULLs/empty always last. Sortable: `id`, `title`, `source`, `status` (alphabetical), `first_seen_at` (date), `price_text` (sorts by `price_num`), `wohnflaeche` (parsed numerically from the VARCHAR), `gesamtflaeche`, `grundstueck`, `objekttyp`. Not sortable: `image_thumb_url` (Bild), `drive_url` (Drive), `note` (Notiz), `merged_into_id` (Rolle, derived badge). Details: [`memory/listings-sort.md`](memory/listings-sort.md). Keys = real DB columns: `id`, `status`, `image_thumb_url` (Bild; the displayed local copy is `listing_images.local_thumb_path`), `title`, `source` (Quelle; search URL = `search_url`), `first_seen_at`, `price_text` (numeric: `price_num`), `wohnflaeche`, `gesamtflaeche`, `grundstueck`, `drive_url`, `objekttyp`, `note`, `merged_into_id` (Rolle badge, derived from `merged_into_id IS NULL`). Header data-refs: `col-head-id`, `col-head-status`, `col-head-bild`, `col-head-titel`, `col-head-quelle`, `col-head-first-seen`, `col-head-preis`, `col-head-note`, `col-head-rolle` (+ the ones below). Table headers: `col-head-grundstueck` (Grundstück, numeric DB column `grundstueck`), `col-head-gesamtflaeche` (cells `col-gesamtflaeche-{id}`; DB `gesamtflaeche` = Wohnfläche + Nebenflächen, without Grundstück), `col-head-wohnflaeche`. Duplikate cards: `dup-grundstueck-{id}`, `dup-gesamtflaeche-{id}` (next to `dup-meta-{id}`). Per-source area label mapping: `app/source_fields.py` (`SOURCE_FIELD_MAP`). Immoweb areas (Netto → Wohnfläche, Grundstück, Gesamtfläche) are documented in [`memory/immoweb-areas.md`](memory/immoweb-areas.md).