Agents / Memory .md

Anleitung für Agents (AGENTS.md) und Übergabe-Überblick (memory/PROJECT.md) — live vom Server gelesen, nur lesend. Neue Agents: zuerst PROJECT.md.

/var/www/meran-kauf/AGENTS.md 21.9 KB

Agents.md

Agent instructions

⚠️ COMMIT RULE (mandatory): After every feature addition or change, make a commit on main with a clear message that says what was implemented, e.g. Objekttyp: DB column + PATCH API + Typ dropdown. Use no feature branches. Never force-push, amend pushed commits, rebase, or otherwise rewrite history. The repo is /var/www/meran-kauf/.git (branch main, no remote). History is visible in the app under Settings → Commits (/settings?section=commits).

🔌 VPS ACCESS RULE: From Grok Bot's Computer use ssh netcup (direct, primary; details /home/box/shared-ssh/README.md on that computer). Only if it fails (e.g. "Connection closed by <proxy-ip>" or "Outbound SSH is turned off for this assistant") fall back to the Mac route (SSH alias netcup_db_server on Andreas's Mac).

⏱️ SCRAPER ON, EMAILS OFF (since 2026-11-11). Daily scrape continues; new-listings mail is disabled (MERAN_NOTIFY_EMAIL_ENABLED=0). Cutoff still turns scraping OFF on 2027-01-01 00:00 Atlantic/Madeira. Switch: meran-scrape-ctl status|on|off and meran-scrape-ctl email on|off. Full instructions: memory/automation-control.md.

👉 Start here: memory/PROJECT.md — compact handover (purpose, hosting, sources, data model, core logic, ops, workflow, open issues, memory index). Also visible in the app under Settings → Agents / Memory .md (/settings?section=docs).

Git workflow

  • After implementing a new feature, always commit the changes to main or master (use whichever default branch exists; prefer main if creating a new repo).
  • The repository already exists (/var/www/meran-kauf/.git, branch main). Do not re-init it.
  • Commit on the default branch directly. Do not create feature branches for completed work unless the user explicitly asks.
  • Never use other git worktrees. Stay in this workspace’s primary working tree only.
  • If no .git folder is present, initialize a repository first (git init), then commit to main.

Commit expectations

  • After a feature is implemented and working, stage relevant files and create a commit on main/master.
  • Do not leave finished feature work uncommitted.
  • Never use additional worktrees (git worktree add or similar).

Project memory (memory/)

Long-lived topic notes live in /var/www/meran-kauf/memory/ — one .md file per topic. Read the relevant file before continuing work on that topic; update it when you learn something durable. Do not dump large histories into this AGENTS.md; keep only the short index below.

FileTopic
memory/PROJECT.mdHandover overview — read first
memory/project-overview.mdWhat Meran-Kauf is, live URL, stack, deploy habit
memory/vps-ops.mdVPS paths, systemd, env, git location
memory/gentle-scraping.mdRamp-up playbook for new URLs / methods
memory/idealista-scraper.mdIdealista / DataDome: tries, failures, pause, next tests
memory/immoreal-scraper.mdImmoreal search scrape, stub hydration, lazy galleries
memory/image-thumbs.mdThumb vs full image URLs, hover-delay, Immoweb/Immoreal
memory/listings-infinite-scroll.mdPaginated listings table, /api/listings, infinite scroll
memory/listings-search.mdFree-text search over listings table + /api/listings?q=
memory/wohnen-scraper.mdWohnen in Südtirol AJAX scrape, thumbs, lazy gallery
memory/duplicates.mdCross-source duplicate candidates, merge/unmerge, canonical rows
memory/url-clusters.mdManual URL clusters (user-curated collections, not duplicates)
memory/settings.mdSettings shell + configurable status funnel
memory/backups.mdDB+code backups path, retention, timer, Settings UI
memory/daily-scrape.mdDaily random scrape timer + new-listings email toggle (MERAN_NOTIFY_EMAIL_ENABLED, NOTIFY_*)
memory/image-storage.mdLocal copies of all listing images (listing_images, /media/, gentle downloader, Settings > Speicher)
memory/objekttyp.mdObjekttyp column (Wohnung/Haus/Grundstück): DB, API, dropdown, merge carry
memory/note.mdManual note per listing: DB, API, inline editor
memory/listings-sort.mdHeader click-to-sort (?sort=&dir=), sort expressions, NULLs last
memory/offline-listings.mdBackground manual scrape (systemd + lock + /api/scrape/status), offline URL check (offline_at), not-seen marker

Related scratch/todos (not the memory index): todos/idealist-scraper.md (Idealista pause notes; prefer the memory/ copy as canonical going forward).

Daily scrape + email

  • Scrape runs once per day at a random time between 06:00 and midnight Europe/Berlin (meran-kauf-scrape.timer).
  • When a scrape adds new listings, one email is sent from grok@liftt.de only if MERAN_NOTIFY_EMAIL_ENABLED is on (default ON when unset; currently OFF since 2026-11-11). SMTP via NOTIFY_* in VPS .env only — never commit secrets. Toggle: meran-scrape-ctl email on|off.
  • Idealista remains paused. Details: memory/daily-scrape.md.
  • One result per unique URL: db.upsert_listing keys on (source, external_id) with a (source, url_key) fallback (normalized host+path), so regenerated portal ids never duplicate a known URL. Backfill via db.dedupe_urls(). Details: memory/duplicates.md.

Gentle scraping (mandatory)

When scraping or testing a new URL / new method (especially Idealista or any site with bot protection):

  1. Propose first in chat — describe a gentle, human-like plan (one method, one URL, long pauses). Do not blast probes.
  2. Start tiny — one request, inspect result, then stop. Increase volume step by step only after something clearly worked.
  3. Behave like a human — low frequency, single client path, random short delays if paging, no parallel hammering, abort immediately on 403 / captcha / empty block pages.
  4. Never run “try every UA / tool / fingerprint” matrices from the VPS IP in one session; that soft-bans the IP and blocks even good methods.
  5. Follow the ramp-up detail in memory/gentle-scraping.md. For Idealista specifics and current pause state, see memory/idealista-scraper.md.

Images in the listings table (Bild)

Be careful how many images the UI loads at once. Full-resolution photos (especially Immoreal) are large; loading every gallery image for every row makes the table slow.

  • Bild column: small thumbs only (image_thumb_url / Immoweb _pre / Immoreal _processed_).
  • Hover preview: set full-res src only on hover — never preload big images for every row.
  • Do not wire the table to download whole galleries on page load. Details: memory/image-thumbs.md.
  • Local copies (/media/…) are preferred when stored (display_images / display_thumb); originals are the fallback. Details: memory/image-storage.md.

UI reference IDs (data-ref)

When adding or changing UI:

  1. Give main page landmarks a stable id and matching data-ref (kebab-case), e.g. id="filters" data-ref="filters".
  2. Always give every button (and filter chip / primary control that acts like a button) a data-ref.
  3. Always give main elements inside each listing row a data-ref, including: the row itself, status cell/matrix, each status button, gallery + gallery link, title link, meta, and the main value columns (price, sizes, first-seen).
  4. For repeated row elements, include the listing DB id in the ref so it is unique and copyable, e.g. data-ref="status-merken-42", data-ref="listing-42", data-ref="gallery-42". Status slug via config.status_ref_slug: spaces///() → hyphens (in kontakt → in-kontakt, gesendet / waiting → gesendet-waiting, OUT (war in kontakt) → out-war-in-kontakt); OUT → out (status-out-42).
  5. Filter chips: filter-ALL, filter-neu, filter-gesendet-waiting, filter-in-kontakt, filter-out-war-in-kontakt, filter-out, etc.
  6. Do not tag tiny decorative bits (raw <img> inside gallery, count badges, table header cells) unless they become interactive controls.
  7. Prefer the selector form agents and humans copy in chat: [data-ref="status-merken-42"].
  8. Keep names stable across deploys; rename only when the UI meaning changes.
  9. Local inspect mode: press e to enter/exit; the data-ref label appears only on the hovered element; click copies [data-ref="…"] to the clipboard. (Ignored while typing in inputs.)

Page landmarks: header, brand, header-path (VPS path, main + settings), header-drive-link (Google Drive „Kauf“ folder, new tab), btn-sources-toggle (header path row, right of Drive link; toggles sources-panel as a full-width row below the header, aria-expanded, state in localStorage meranSourcesOpen; auto-opens while a scrape reports status), header-actions, neu-badge, btn-settings, btn-url-check („URLs prüfen“, detached offline check via POST /api/url-check, polls /api/url-check/status), btn-scrape (starts a background scrape, shows „Scrape läuft…“ and polls /api/scrape/status; progress + summary go to scrape-status), sources-panel, source-urls, last-runs, scrape-status, filters, manual-merge-panel, merge-manual, merge-ids-input, btn-merge-load, merge-primary-picker, btn-merge-confirm, listings-main, listings-table-wrap, listings-table, listings-tbody, listings-load-more, listings-end, search-wrap, listings-search, search-result-count, content-wrap (wraps filters + manual-merge-panel + listings-main on /, incl. Duplikate view; width = CSS var --content-width in :root of app/static/app.css, default 1400px; settings page uses --settings-width, default 1100px), duplicates-view, dup-toolbar, btn-dup-refresh, ref-toast, settings-layout, settings-nav, settings-funnel, funnel-editor, btn-funnel-save, btn-funnel-add, status-rules, status-rule-0, rule-0-enabled, rule-0-value, rule-0-target, btn-rules-save, rules-save-status, settings-backups, btn-backup-run, backup-table, settings-storage, storage-path, storage-example-path, storage-stats, btn-storage-fetch, btn-storage-refresh, storage-fetch-status, storage-last-fetch, status-context-menu, funnel-config.

Settings → Agents / Memory .md (/settings?section=docs, nav settings-nav-docs): panel settings-docs, blocks doc-agents / doc-project with doc-path-agents / doc-path-project (absolute path), doc-copy-{key} (copy path), doc-toggle-{key} (rendered ↔ raw), doc-content-agents / doc-content-project (rendered Markdown), doc-raw-{key}. Server: app/docs.py, fixed whitelist DOCS (no path parameter), escape-first mini Markdown renderer (raw HTML never passes through; only http(s)/mailto links become <a>).
Settings → Commits (/settings?section=commits, nav settings-nav-commits): panel settings-commits, table commits-table / commits-tbody, rows commit-row-{shortHash} (data-hash = full hash), body <details> commit-body-{shortHash}, header info commits-branch / commits-total, paging btn-commits-more + commits-more-status, error commits-error. API GET /api/settings/commits?offset=&limit= (200 per page). Server: app/gitlog.py runs git log with an 8 s timeout and safe.directory only via the subprocess env (GIT_CONFIG_COUNT), because .git is root-owned while gunicorn runs as www-data.

Filter chips include filter-duplicates (Duplikate).

Duplikate members: dup-member-{id}, dup-primary-{id}, dup-img-{id} (plus shared gallery-{id} / gallery-link-{id}). Candidate groups: btn-dup-merge-{n}, btn-dup-dismiss-{n} („Keine Dublette“ — whole group), reason chips dup-reason-{n}-{k}. Section filters in dup-toolbar: dup-filters, dup-filter-A … dup-filter-E with counts dup-filter-count-A … -E (URL ?status=duplicates&dup=C, localStorage meranDupSection); empty state empty-dup-section. Groups carry data-sections. Popup (all pages, bottom-right, grey bg + amber border/stripe): dup-popup, dup-popup-count, dup-popup-link, btn-dup-popup-close. Merge popup (bottom-center, same styling language): merge-popup, merge-popup-title, inputs merge-popup-primary (single primary ID) + merge-popup-others (other IDs, comma/space separated), btn-merge-popup-save, btn-merge-popup-close, merge-popup-status, btn-merge-popup-open (reopen pill, shown while popup is closed). Save POSTs {ids:[primary,…others], primary_id, enrich:true} to /api/duplicates/merge (reuses parseIdList), then reloads. URL clusters (2026-09-29, manual collections, NOT duplicates): bottom popup url-cluster-popup (teal) with target select url-cluster-target, name url-cluster-name, URL textarea url-cluster-urls, btn-url-cluster-popup-save, url-cluster-popup-status, btn-url-cluster-popup-close, reopen pill btn-url-cluster-popup-open; filters button btn-url-clusters (count url-clusters-count) toggles the cluster table view (groups in #listings-table, A-Z, drag & drop; group heads cluster-group-head-{id|none} with btn-cluster-del-{id}); regular table has trailing Cluster column col-head-cluster with per-row dropdown cluster-select-{id} (name cluster-name-{id} in cluster view). APIs: GET/POST /api/url-clusters, POST /api/url-clusters/move, DELETE /api/url-clusters/<id>. One URL per cluster max (UNIQUE url_key). Details: memory/url-clusters.md. Merged groups: collapsed <details>, btn-dup-set-primary-{n} to change primary, btn-dup-unmerge-{n}. Selective merge (2026-09-29): per-group btn-dup-edit-select-{n} toggles selection mode (group class is-selecting; checkboxes dup-select-{id}; sticky bar dup-select-bar-{n} with dup-select-count-{n}, primary select dup-select-primary-{n}, enrich dup-select-enrich-{n}, btn-dup-select-merge-{n}, btn-dup-select-cancel-{n}); merges only the checked listings (POST /api/duplicates/selective-merge {ids, selected, primary_id, enrich}) and dismisses every other pair among the group (kept-vs-kept included). Card right-click menu dup-card-ctx: dup-ctx-select-{id} / dup-ctx-unique-{id} / dup-ctx-primary-{id} / dup-ctx-open-{id}. Dismissed pairs section dup-dismissed with dup-dismissed-list, dup-dismissed-item-{a}-{b}, btn-dup-undismiss-{a}-{b}; notice toast dup-notice with dup-notice-undo / dup-notice-close. POST /api/duplicates/dismiss also accepts {pairs:[[a,b],...]} (max 200); GET /api/duplicates/dismissals. Group status sync (2026-09-29): the primary's status is the group's status — merge_listings pushes the primary's final status onto all siblings (result field siblings_status_updated); PATCH /api/listings/{id}/status also syncs merged siblings (response field siblings_updated); Duplikate-view JS mirrors the primary's new status onto sibling dup-status-{id} selects + dup-meta-status-{id} text. Details: memory/duplicates.md. Tags (2026-09-29, sub-filter under the status funnel): settings section tags (nav settings-nav-tags, panel settings-tags), API /api/tags + /api/listings/{id}/tags, ?tag=<id> on / and /api/listings, pills tag-pills-{id} + editor tag-edit-{id} in the title column, filter row tag-filter-row in #filters. Details: memory/tags.md. Row context menu (2026-09-29, right-click on tr[data-id] in #listings-table): row-ctx with Delete (row-ctx-delete-<id>, SOFT delete since 2026-10-01: sets listings.deleted_at on that one id only (not relevant anymore); row stays in DB, hidden from all table views; DELETE /api/listings/<id>; merge/dup logic untouched; details: memory/soft-delete.md) and Tag (row-ctx-tag-<id>) with a hover submenu (row-ctx-sub) listing all tags for toggle (row-ctx-tagopt-<tagId>). Sources status badges in sources-panel inside header.

Per listing {id}: listing-{id}, col-id-{id}, col-status-{id}, status-cell-{id}, status-matrix-{id}, status-out-{id}, status-out-war-in-kontakt-{id}, status-neu-{id}, status-merken-{id}, status-kontaktieren-{id}, status-gesendet-waiting-{id}, status-in-kontakt-{id}, status-gesehen-{id}, status-pill-{id}, col-img-{id}, gallery-{id}, gallery-link-{id}, gallery-prev-{id}, gallery-next-{id}, col-title-{id}, listing-title-{id}, listing-meta-{id}, offline-badge-{id} (red „offline“ in the meta line: URL answered 404/410, tooltip with HTTP status + dates; see memory/offline-listings.md), not-seen-badge-{id} (grey „nicht mehr gesehen seit DD.MM.“: last_seen_at older than the latest error-free run of the source; not for paused sources; shown next to offline too), also-on-{id}, col-source-{id}, listing-source-{id}, source-search-{id}, source-sibling-{id}, source-sibling-link-{id}, col-first-seen-{id}, col-price-{id}, col-grundstueck-{id}, col-wohnflaeche-{id}, dup-count-btn-{id} (Titel column, number of merged siblings; hover/click opens dup-popover-{id} with dup-popover-item-{sid} + dup-popover-link-{sid} → in-app link /?status=duplicates&dup=A&merged={id}&focus={sid}#merged-{id}; merged groups on the Duplikate page carry id="merged-{primaryId}", members data-member-id, deep link highlights .dup-focus / .dup-focus-member), col-drive-{id} (Drive cell: link drive-{id}, edit/add button drive-edit-{id}, inline editor drive-input-{id} / drive-save-{id} / drive-cancel-{id}; header col-head-drive). Duplikate cards: dup-drive-cell-{id} with dup-drive-{id}, dup-drive-edit-{id}, dup-drive-input-{id}. API PATCH /api/listings/{id}/drive {drive_url} ("" clears). col-objekttyp-{id} (column „Typ“, far right after Drive; header col-head-objekttyp) with dropdown objekttyp-select-{id} (— / Wohnung / Haus / Grundstück, saves on change; feedback classes .is-saving / .is-saved / .is-error). DB listings.objekttyp ENUM NULL; API PATCH /api/listings/{id}/objekttyp {objekttyp} ("" or null clears, other values → 400). Merge copies a sibling objekttyp onto a NULL primary. Details: memory/objekttyp.md. col-note-{id} (column „Notiz“, far right after Typ; header col-head-note) with text note-{id} (title = full text, CSS clamps to 3 lines) + edit/add button note-edit-{id}, inline editor note-input-{id} (textarea) / note-save-{id} / note-cancel-{id} (save: Ctrl+Enter or ✓; empty clears). DB listings.note TEXT NULL; API PATCH /api/listings/{id}/note {note} ("" or null clears, max 2000 chars). Details: memory/note.md. col-rolle-{id} (column „Rolle“, far right after Notiz; header col-head-rolle) with badge rolle-{id}: „Primär“ (green, merged_into_id IS NULL) or „Sekundär → #<id>“ (amber, merged_into_id set). Main table shows canonical rows only, so currently all rows are Primär; the badge also covers merged rows if they ever appear. JS twin: meranRolle.cellHtml() in app.js. Header DB-column badges (thead of #listings-main, part A, no sorting yet): every th.th-db has data-db-col, a label span col-label-{key} (keep free for future click-to-sort) and a hover/focus-only badge col-dbname-{key} with a copy button col-dbname-copy-{key} (clipboard API, textarea/execCommand fallback, "kopiert" feedback; data-copied-via = clipboard|fallback|failed). Badge clicks and mousedowns stop propagation. Click-to-sort (part B): sortable headers render the label as a link a.th-sort (col-label-{key}, data-sort-key) with indicator sort-ind-{key} (▲ asc / ▼ desc); cycle asc → desc → default. Server-side via ?sort={key}&dir=asc|desc (kept in status-chip hrefs, infinite scroll and live search via #listings-table[data-sort][data-dir]; /api/listings accepts the same params). NULLs/empty always last. Sortable: id, title, source, status (alphabetical), first_seen_at (date), price_text (sorts by price_num), wohnflaeche (parsed numerically from the VARCHAR), gesamtflaeche, grundstueck, objekttyp. Not sortable: image_thumb_url (Bild), drive_url (Drive), note (Notiz), merged_into_id (Rolle, derived badge). Details: memory/listings-sort.md. Keys = real DB columns: id, status, image_thumb_url (Bild; the displayed local copy is listing_images.local_thumb_path), title, source (Quelle; search URL = search_url), first_seen_at, price_text (numeric: price_num), wohnflaeche, gesamtflaeche, grundstueck, drive_url, objekttyp, note, merged_into_id (Rolle badge, derived from merged_into_id IS NULL). Header data-refs: col-head-id, col-head-status, col-head-bild, col-head-titel, col-head-quelle, col-head-first-seen, col-head-preis, col-head-note, col-head-rolle (+ the ones below). Table headers: col-head-grundstueck (Grundstück, numeric DB column grundstueck), col-head-gesamtflaeche (cells col-gesamtflaeche-{id}; DB gesamtflaeche = Wohnfläche + Nebenflächen, without Grundstück), col-head-wohnflaeche. Duplikate cards: dup-grundstueck-{id}, dup-gesamtflaeche-{id} (next to dup-meta-{id}). Per-source area label mapping: app/source_fields.py (SOURCE_FIELD_MAP). Immoweb areas (Netto → Wohnfläche, Grundstück, Gesamtfläche) are documented in memory/immoweb-areas.md.

/var/www/meran-kauf/memory/PROJECT.md 7.3 KB

Meran-Kauf — handover (start here)

A compact overview for a new agent. Details are in the topic notes indexed at the bottom. No secrets in this file. Secrets live only in /var/www/meran-kauf/.env on the VPS (never commit it; it is excluded from backups).

Purpose

Private tool for Andreas to track property purchases in Meran/Merano. It scrapes listings from several portals into a single MariaDB, detects cross-portal duplicates and manages each listing through a German status funnel (neu → kontaktieren → … → OUT). The UI is German.

Where things run

  • Live: https://scraper-staatenlos.devotron.de (no login, so treat everything as public-ish)
  • Host: Netcup VPS. Primary access: ssh netcup directly from Grok Bot's Computer (root, key-based). Fallback only if that fails: SSH alias netcup_db_server from the Mac.
  • Code + git: /var/www/meran-kauf (branch main, no remote), venv .venv, env .env
  • Stack: Flask + MariaDB + gunicorn on 127.0.0.1:3127 as meran-kauf.service (user www-data). nginx site scraper-staatenlos.devotron.de proxies to it.
  • Images: local copies in /var/lib/meran-kauf/images/, served by nginx at /media/
  • Unit files (repo copies): deploy/ (app, scrape + backup timers, nginx conf)

Sources and scrape rules

  • Active: Immoweb, Immoreal, Wohnen in Südtirol (plain requests). Idealista is paused (DataDome anti-bot), see idealista-scraper.md.
  • Gentle scraping is mandatory (AGENTS.md + gentle-scraping.md): low frequency, one client path, abort immediately on 403 / 429 / captcha / empty page, cooldowns, no retry loops.
  • Field mapping per portal (area labels → Wohnfläche / Gesamtfläche / Grundstück) is in app/source_fields.py (SOURCE_FIELD_MAP).

Data model (MariaDB)

  • listings (one row per portal listing):
    • source, external_id, url, title, price_text + price_num
    • wohnflaeche (VARCHAR, portal text), gesamtflaeche / grundstueck (DECIMAL)
    • status (funnel id), merged_into_id (NULL = canonical/primary)
    • drive_url (Google Drive link), objekttyp (ENUM Wohnung / Haus / Grundstück, NULL)
    • first_seen_at, last_seen_at, image_url / image_thumb_url / image_urls
  • listing_images: local image copies (paths, status, attempts), see image-storage.md.
  • app_settings (JSON values):
    • status_funnel: steps, labels, colours
    • status_rules: auto-status for new inserts
    • image_fetch: downloader state
  • Also: duplicate_candidates, duplicate_dismissals, scrape_runs.

Core logic

  • Duplicates: candidate groups come from matching rules (same title/size, price, …; sections A–E on the Duplikate page). Merge sets merged_into_id on the siblings. Only the primary is visible in lists, filters and counts. Merge carries drive_url, objekttyp and non-neu status onto the primary. Unmerge and re-primary are supported. See duplicates.md.
  • Funnel: configurable in Settings. Status ids are stable (stored in listings.status); only labels and colours change.
  • Auto-rule: new inserts with Wohnfläche < 120 m² get neu (unter 120 m²) instead of neu (app/status_rules.py). Existing rows are never touched.
  • Table: server-side search (?q=), infinite scroll, header click-to-sort ?sort=&dir= with NULLs last (listings-sort.md), DB-name badges on the headers.

Operations

  • Daily scrape: meran-kauf-scrape.timer at a random time between 06:00 and 24:00 Berlin (run_scrape.py). A new-listings email is sent only when the email toggle is on (currently OFF since 2026-11-11; see automation-control.md). "Jetzt scrapen" starts the same systemd unit in the background (GET /api/scrape/status); every scrape ends with the offline URL check (offline-listings.md).
  • Backups: meran-kauf-backup.timer at 03:15 writes to /var/backups/meran-kauf/: db.sql.gz + code.tar.gz. It keeps 7 daily + 1 per month for 6 months. It contains no images and no .env. A manual backup can be started in Settings → Backups.
  • Restart / logs:
    • systemctl restart meran-kauf.service
    • journalctl -u meran-kauf.service -n 100 (same for meran-kauf-scrape.service, meran-kauf-backup.service)
    • systemctl list-timers | grep meran

Workflow

  • Commit rule: commit on main after every change, with a clear message. No feature branches. Never force-push or rewrite history. History is visible in Settings → Commits.
  • Deploy:
    1. Edit a mirror on the agent box.
    2. tar czf, then scp it to netcup:/tmp/ directly (fallback if direct SSH fails: copy to the Mac, then scp to netcup_db_server:/tmp/).
    3. Extract in /var/www/meran-kauf.
    4. systemctl restart meran-kauf.service.
    5. git commit on the VPS.
  • Data-refs: every UI element carries a data-ref (listed in AGENTS.md). Keep that list updated.
  • Verification: headless Chrome screenshots at about 1600px width. The table must fit --content-width: 1400px.

Open issues

  • SMTP blocked: the VPS can't send mail (Netcup blocks outbound 25/465/587), so the daily notify mail fails. It needs an HTTPS mail API or unblocking (daily-scrape.md).
  • Images not backed up: /var/lib/meran-kauf/images is not in the backups and can only be re-downloaded while the portals still host the images.
  • Questionable Wohnfläche: #12, #36 and #104 (#104 is a building plot, objekttyp = Grundstück).
  • #193: shows a placeholder image ("Objekt nicht mehr verfügbar").
  • No auth: the manual "Jetzt backup" button and all write APIs are unprotected. Dump downloads are deliberately not offered in the UI.

Contacts

  • Notify mails go to andreas.rueping@gmail.com.
  • Language: always English (docs, code comments, commit messages, agent replies about this project)

Memory index (memory/)

FileTopic
project-overview.mdShort overview, runtime, deploy habit
vps-ops.mdVPS paths, services, git, backups, images
gentle-scraping.mdRamp-up and abort rules for scraping
idealista-scraper.mdIdealista / DataDome attempts, pause
immoreal-scraper.mdImmoreal search + detail scrape
immoweb-areas.mdImmoweb area fields → Wohn/Gesamt/Grundstück
wohnen-scraper.mdWohnen in Südtirol AJAX scrape
image-thumbs.mdThumb vs. full image URLs, hover
image-storage.mdLocal image copies, /media/, downloader
listings-infinite-scroll.mdPaged table, /api/listings
listings-search.mdFree-text search
listings-sort.mdHeader click-to-sort
duplicates.mdDuplicate candidates, merge / unmerge, popup, count button
drive-links.mdDrive column
objekttyp.mdObjekttyp column
settings.mdSettings shell + status funnel
backups.mdBackups, retention, timer
daily-scrape.mdDaily scrape timer + notify mail toggle
offline-listings.mdBackground manual scrape + status API, offline URL check, not-seen marker

Scraper + email on/off

Scraping is ON; new-listings emails are OFF since 2026-11-11. The cutoff still turns scraping OFF on 2027-01-01 00:00 Atlantic/Madeira. Commands and details: automation-control.md (/var/www/meran-kauf/memory/automation-control.md). The 2026-11-08 pause was undone the same day.