Add uploader scripts and docs
This commit is contained in:
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"permissions": {
|
||||||
|
"allow": [
|
||||||
|
"Bash(brew search:*)"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
+18969
File diff suppressed because it is too large
Load Diff
+124
@@ -0,0 +1,124 @@
|
|||||||
|
# Installation
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
### 1. Python 3
|
||||||
|
|
||||||
|
macOS includes Python 3. Verify with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 --version
|
||||||
|
```
|
||||||
|
|
||||||
|
If missing: `brew install python`
|
||||||
|
|
||||||
|
### 2. rsync
|
||||||
|
|
||||||
|
macOS ships rsync 2.6.9 (2006), which has bugs with path handling. Install the current version:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
brew install rsync
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. SSH key access to the remote host
|
||||||
|
|
||||||
|
The uploader runs unattended, so it must be able to reach the remote host without a password. This means SSH key-based authentication needs to be in place before you install.
|
||||||
|
|
||||||
|
If you don't already have an SSH key, generate one:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh-keygen -t ed25519
|
||||||
|
```
|
||||||
|
|
||||||
|
Then authorize it on the remote host:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh-copy-id filez@10.10.8.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Confirm the connection works without any password prompt before continuing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh filez@10.10.8.1 echo ok
|
||||||
|
```
|
||||||
|
|
||||||
|
If your key lives somewhere other than the default location (`~/.ssh/id_rsa` or `~/.ssh/id_ed25519`), set the `ssh_key_path` field in `downloads_uploader_config.json` to its full path.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/Downloads/uploader
|
||||||
|
./downloads_uploader_install.sh install
|
||||||
|
```
|
||||||
|
|
||||||
|
This will:
|
||||||
|
1. Create `~/.local/bin/downloads-uploader` (a named symlink to `python3` so macOS permission dialogs show a friendly name)
|
||||||
|
2. Write `~/Library/LaunchAgents/com.user.downloads-uploader.plist`
|
||||||
|
3. Load the agent — it runs immediately and then every 5 minutes
|
||||||
|
|
||||||
|
**macOS may prompt for network/file access permissions** the first time. Allow them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Configure
|
||||||
|
|
||||||
|
Open `downloads_uploader_config.json` and set your remote destination:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"remote_destination": "user@your-host:/path/to/remote/dir/",
|
||||||
|
"ssh_key_path": "",
|
||||||
|
"ssh_port": 22
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Then trigger a fresh run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./downloads_uploader_install.sh run
|
||||||
|
```
|
||||||
|
|
||||||
|
Check the output:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./downloads_uploader_install.sh status
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Uninstall
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./downloads_uploader_install.sh uninstall
|
||||||
|
```
|
||||||
|
|
||||||
|
This stops and removes the launchd agent and the symlink. Config, logs, and state files are left in place.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
**"rsync not found"**
|
||||||
|
```bash
|
||||||
|
brew install rsync
|
||||||
|
```
|
||||||
|
|
||||||
|
**SSH connects but `find` fails on remote**
|
||||||
|
The script uses GNU `find -printf`, which requires Linux. BSD/macOS remotes are not supported on the remote side.
|
||||||
|
|
||||||
|
**Files keep re-uploading**
|
||||||
|
Size mismatch between local and remote. Check that the remote filesystem isn't silently modifying files (e.g. a torrent client moving them). You can inspect `.uploader_state.json` for the last recorded sizes.
|
||||||
|
|
||||||
|
**Agent not running after reboot**
|
||||||
|
Verify the plist is in `~/Library/LaunchAgents/` and loaded:
|
||||||
|
```bash
|
||||||
|
launchctl list | grep downloads-uploader
|
||||||
|
```
|
||||||
|
If missing, re-run `./downloads_uploader_install.sh install`.
|
||||||
|
|
||||||
|
**See full logs**
|
||||||
|
```bash
|
||||||
|
tail -f ~/Downloads/uploader/uploader_log.txt
|
||||||
|
```
|
||||||
Executable
+463
@@ -0,0 +1,463 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Downloads Auto-Uploader
|
||||||
|
=======================
|
||||||
|
Scans the Downloads folder, fetches the remote file tree via SSH,
|
||||||
|
and uploads — file by file — only what is missing or has changed.
|
||||||
|
|
||||||
|
Config file : downloads_uploader_config.json (same folder as this script)
|
||||||
|
State file : .uploader_state.json (audit log of uploads)
|
||||||
|
Log file : uploader_log.txt
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# ── Safety: never scan these paths ────────────────────────────────────────────
|
||||||
|
FORBIDDEN_SCAN_ROOTS = {
|
||||||
|
"/", "/private", "/private/var",
|
||||||
|
"/usr", "/bin", "/sbin", "/etc", "/var",
|
||||||
|
"/System", "/Library", "/Applications",
|
||||||
|
"/home", "/opt", "/cores",
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── Fixed paths (relative to this script's directory) ─────────────────────────
|
||||||
|
SCRIPT_DIR = Path(__file__).parent.resolve()
|
||||||
|
SCAN_DEFAULT = SCRIPT_DIR.parent # one level up: Downloads/
|
||||||
|
CONFIG_FILE = SCRIPT_DIR / "downloads_uploader_config.json"
|
||||||
|
STATE_FILE = SCRIPT_DIR / ".uploader_state.json"
|
||||||
|
LOG_FILE = SCRIPT_DIR / "uploader_log.txt"
|
||||||
|
|
||||||
|
# ── Defaults ───────────────────────────────────────────────────────────────────
|
||||||
|
DEFAULT_CONFIG = {
|
||||||
|
"remote_destination": "filez@10.10.8.1:/mnt/ssd2t/shared/torrent/Downloads/",
|
||||||
|
"ssh_key_path": "", # empty = use ~/.ssh/id_rsa (system default)
|
||||||
|
"ssh_port": 22,
|
||||||
|
# Empty = auto-detect (parent of the uploader/ folder = Downloads/).
|
||||||
|
# Set an absolute path to watch a different directory.
|
||||||
|
"scan_directory": "",
|
||||||
|
"skip_patterns": [
|
||||||
|
".*", # hidden / dot-files
|
||||||
|
"*.crdownload", # incomplete Chrome downloads
|
||||||
|
"*.part", # incomplete Firefox downloads
|
||||||
|
"*.tmp",
|
||||||
|
"uploader", # this tool's own folder
|
||||||
|
"ORGANIZER_LOG.txt",
|
||||||
|
"organizer_log*.txt",
|
||||||
|
"__pycache__",
|
||||||
|
],
|
||||||
|
"min_file_age_seconds": 30, # ignore files newer than this (still being written)
|
||||||
|
"timeout_seconds": 7200, # per-file rsync timeout (2 h)
|
||||||
|
"remote_scan_timeout": 120, # SSH find timeout
|
||||||
|
"dry_run": False, # True = log only, no actual transfer
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Config / State helpers
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def load_config() -> dict:
|
||||||
|
if CONFIG_FILE.exists():
|
||||||
|
with open(CONFIG_FILE, "r", encoding="utf-8") as f:
|
||||||
|
user_cfg = json.load(f)
|
||||||
|
return {**DEFAULT_CONFIG, **user_cfg}
|
||||||
|
cfg = DEFAULT_CONFIG.copy()
|
||||||
|
save_config(cfg)
|
||||||
|
log("INFO", f"Created default config: {CONFIG_FILE}")
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
def save_config(cfg: dict):
|
||||||
|
with open(CONFIG_FILE, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
|
def load_state() -> dict:
|
||||||
|
if STATE_FILE.exists():
|
||||||
|
with open(STATE_FILE, "r", encoding="utf-8") as f:
|
||||||
|
return json.load(f)
|
||||||
|
return {"uploaded": {}}
|
||||||
|
|
||||||
|
|
||||||
|
def save_state(state: dict):
|
||||||
|
with open(STATE_FILE, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(state, f, indent=2, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Logging
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def log(level: str, message: str):
|
||||||
|
ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
line = f"[{ts}] {level:7s} {message}"
|
||||||
|
print(line)
|
||||||
|
with open(LOG_FILE, "a", encoding="utf-8") as f:
|
||||||
|
f.write(line + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Utilities
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def matches_skip_pattern(name: str, patterns: list) -> bool:
|
||||||
|
import fnmatch
|
||||||
|
return any(fnmatch.fnmatch(name, p) for p in patterns)
|
||||||
|
|
||||||
|
|
||||||
|
def _ssh_opts(cfg: dict) -> list[str]:
|
||||||
|
"""Return ssh CLI option list from config."""
|
||||||
|
opts = [
|
||||||
|
"-p", str(cfg["ssh_port"]),
|
||||||
|
"-o", "StrictHostKeyChecking=accept-new",
|
||||||
|
"-o", "BatchMode=yes", # never prompt for a password
|
||||||
|
"-o", "ConnectTimeout=15",
|
||||||
|
]
|
||||||
|
if cfg.get("ssh_key_path"):
|
||||||
|
opts += ["-i", cfg["ssh_key_path"]]
|
||||||
|
return opts
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_remote(destination: str) -> tuple[str, str]:
|
||||||
|
"""Split 'user@host:/path' into ('user@host', '/path')."""
|
||||||
|
host, rpath = destination.split(":", 1)
|
||||||
|
return host, rpath.rstrip("/")
|
||||||
|
|
||||||
|
|
||||||
|
def format_bytes(n: int) -> str:
|
||||||
|
for unit in ("B", "KB", "MB", "GB"):
|
||||||
|
if n < 1024:
|
||||||
|
return f"{n:.1f} {unit}"
|
||||||
|
n /= 1024
|
||||||
|
return f"{n:.1f} TB"
|
||||||
|
|
||||||
|
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Scan directory validation
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def resolve_scan_dir(cfg: dict) -> Path:
|
||||||
|
raw = cfg.get("scan_directory", "").strip()
|
||||||
|
scan_dir = Path(raw).resolve() if raw else SCAN_DEFAULT.resolve()
|
||||||
|
|
||||||
|
log("INFO", f"Script dir : {SCRIPT_DIR}")
|
||||||
|
log("INFO", f"Scan dir : {scan_dir}")
|
||||||
|
|
||||||
|
if str(scan_dir) in FORBIDDEN_SCAN_ROOTS or scan_dir == Path("/"):
|
||||||
|
log("FATAL",
|
||||||
|
f"Refusing to scan a system/root path: {scan_dir}\n"
|
||||||
|
" → Set 'scan_directory' in downloads_uploader_config.json\n"
|
||||||
|
" → Or re-run: ./downloads_uploader_install.sh install")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if not scan_dir.is_dir():
|
||||||
|
log("FATAL", f"scan_directory does not exist: {scan_dir}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
return scan_dir
|
||||||
|
|
||||||
|
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Remote file tree
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def get_remote_file_tree(cfg: dict) -> dict[str, int]:
|
||||||
|
"""
|
||||||
|
SSH into the remote host, run `find`, and return a dict of
|
||||||
|
{ relative_path : size_in_bytes } for every file under remote_destination.
|
||||||
|
|
||||||
|
Uses GNU find's -printf '%P\\t%s\\n' (Linux servers have this).
|
||||||
|
If the remote directory doesn't exist yet, returns an empty dict.
|
||||||
|
On any error, logs a warning and returns {} (safe: everything gets uploaded).
|
||||||
|
"""
|
||||||
|
remote = cfg["remote_destination"]
|
||||||
|
host, rpath = remote.split(":", 1)
|
||||||
|
rpath = rpath.rstrip("/")
|
||||||
|
timeout = cfg.get("remote_scan_timeout", 120)
|
||||||
|
|
||||||
|
# find with -printf (GNU / Linux); || true so SSH exits 0 even if dir missing
|
||||||
|
find_cmd = (
|
||||||
|
f'find "{rpath}" -type f -printf "%P\\t%s\\n" 2>/dev/null || true'
|
||||||
|
)
|
||||||
|
cmd = ["ssh"] + _ssh_opts(cfg) + [host, find_cmd]
|
||||||
|
|
||||||
|
log("INFO", f"Fetching remote file tree: {host}:{rpath} ...")
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
cmd,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
stderr = result.stderr.decode("utf-8", errors="replace").strip()
|
||||||
|
if result.returncode != 0 and stderr:
|
||||||
|
log("WARN", f"SSH stderr: {stderr.splitlines()[0]}")
|
||||||
|
|
||||||
|
remote_files: dict[str, int] = {}
|
||||||
|
for line in result.stdout.decode("utf-8", errors="replace").splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
parts = line.split("\t", 1)
|
||||||
|
if len(parts) == 2:
|
||||||
|
rel, size_str = parts
|
||||||
|
try:
|
||||||
|
remote_files[rel] = int(size_str)
|
||||||
|
except ValueError:
|
||||||
|
remote_files[rel] = -1 # unknown size → will compare by name only
|
||||||
|
else:
|
||||||
|
remote_files[line] = -1
|
||||||
|
|
||||||
|
log("INFO", f"Remote tree: {len(remote_files):,} file(s) indexed")
|
||||||
|
return remote_files
|
||||||
|
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
log("WARN",
|
||||||
|
f"Timeout ({timeout}s) fetching remote tree. "
|
||||||
|
"Treating remote as empty — all local files will be uploaded.")
|
||||||
|
return {}
|
||||||
|
except Exception as exc:
|
||||||
|
log("WARN", f"Could not fetch remote tree: {exc}. Treating remote as empty.")
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Local ↔ Remote diff
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def collect_files_to_upload(
|
||||||
|
scan_dir: Path,
|
||||||
|
remote_files: dict[str, int],
|
||||||
|
cfg: dict,
|
||||||
|
) -> list[tuple[Path, str]]:
|
||||||
|
"""
|
||||||
|
Walk every candidate under scan_dir and compare with remote_files.
|
||||||
|
Returns [(local_abs_path, remote_relative_path), ...] for files to upload.
|
||||||
|
remote_relative_path uses forward slashes, relative to scan_dir.
|
||||||
|
"""
|
||||||
|
patterns = cfg["skip_patterns"]
|
||||||
|
min_age = cfg["min_file_age_seconds"]
|
||||||
|
now = datetime.now().timestamp()
|
||||||
|
to_upload = []
|
||||||
|
synced_count = 0
|
||||||
|
skipped_count = 0
|
||||||
|
|
||||||
|
for top in sorted(scan_dir.iterdir()):
|
||||||
|
if matches_skip_pattern(top.name, patterns):
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
age = now - top.stat().st_mtime
|
||||||
|
if age < min_age:
|
||||||
|
log("SKIP", f"{top.name} (age={age:.0f}s, threshold={min_age}s)")
|
||||||
|
skipped_count += 1
|
||||||
|
continue
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Enumerate all files: a single file, or every file inside a directory
|
||||||
|
if top.is_file():
|
||||||
|
local_files = [top]
|
||||||
|
elif top.is_dir():
|
||||||
|
local_files = sorted(f for f in top.rglob("*") if f.is_file())
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
|
||||||
|
for lf in local_files:
|
||||||
|
rel = lf.relative_to(scan_dir).as_posix() # forward-slash path
|
||||||
|
local_size = lf.stat().st_size
|
||||||
|
remote_size = remote_files.get(rel)
|
||||||
|
|
||||||
|
if remote_size is None:
|
||||||
|
# File is not on remote at all
|
||||||
|
to_upload.append((lf, rel))
|
||||||
|
elif remote_size == -1:
|
||||||
|
# Remote size unknown (find didn't return it) — upload to be safe
|
||||||
|
to_upload.append((lf, rel))
|
||||||
|
elif remote_size != local_size:
|
||||||
|
log("DIFF",
|
||||||
|
f"{rel} "
|
||||||
|
f"local={format_bytes(local_size)} "
|
||||||
|
f"remote={format_bytes(remote_size)} → will re-upload")
|
||||||
|
to_upload.append((lf, rel))
|
||||||
|
else:
|
||||||
|
synced_count += 1 # identical size → skip
|
||||||
|
|
||||||
|
log("INFO",
|
||||||
|
f"Diff complete: "
|
||||||
|
f"{len(to_upload)} to upload, "
|
||||||
|
f"{synced_count} already in sync, "
|
||||||
|
f"{skipped_count} skipped (too recent)")
|
||||||
|
return to_upload
|
||||||
|
|
||||||
|
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Per-file upload
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def _ensure_remote_dir(remote_dir: str, host: str, cfg: dict,
|
||||||
|
created_dirs: set) -> bool:
|
||||||
|
"""
|
||||||
|
Create *remote_dir* on *host* via SSH mkdir -p (once per session per dir).
|
||||||
|
Returns True on success, False on error.
|
||||||
|
"""
|
||||||
|
if remote_dir in created_dirs:
|
||||||
|
return True
|
||||||
|
cmd = ["ssh"] + _ssh_opts(cfg) + [host, f"mkdir -p '{remote_dir}'"]
|
||||||
|
try:
|
||||||
|
result = subprocess.run(cmd, stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE, timeout=30)
|
||||||
|
if result.returncode == 0:
|
||||||
|
created_dirs.add(remote_dir)
|
||||||
|
return True
|
||||||
|
err = result.stderr.decode("utf-8", errors="replace").strip()
|
||||||
|
log("ERROR", f"mkdir -p failed for {remote_dir}: {err}")
|
||||||
|
return False
|
||||||
|
except Exception as exc:
|
||||||
|
log("ERROR", f"mkdir -p exception: {exc}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def upload_file(
|
||||||
|
local_path: Path,
|
||||||
|
remote_rel: str,
|
||||||
|
cfg: dict,
|
||||||
|
created_dirs: set,
|
||||||
|
) -> bool:
|
||||||
|
"""
|
||||||
|
Upload a single file to the exact remote path, preserving the
|
||||||
|
relative directory structure.
|
||||||
|
|
||||||
|
Strategy (no --relative, no ./ anchor tricks):
|
||||||
|
1. SSH mkdir -p the remote parent directory (cached per session).
|
||||||
|
2. rsync the file into that directory.
|
||||||
|
|
||||||
|
This avoids the macOS rsync 2.6.9 bug where --relative recreates the
|
||||||
|
entire absolute local path (/Users/mcer323/Downloads/…) on the remote.
|
||||||
|
"""
|
||||||
|
host, rbase = _parse_remote(cfg["remote_destination"])
|
||||||
|
timeout = cfg.get("timeout_seconds", 7200)
|
||||||
|
ssh_args = " ".join(_ssh_opts(cfg))
|
||||||
|
|
||||||
|
# Compute the remote directory for this file
|
||||||
|
parent = Path(remote_rel).parent
|
||||||
|
if str(parent) == ".":
|
||||||
|
remote_dir = rbase # file sits directly in Downloads
|
||||||
|
else:
|
||||||
|
remote_dir = rbase + "/" + parent.as_posix()
|
||||||
|
|
||||||
|
remote_target = f"{host}:{remote_dir}/" # trailing / = place file inside dir
|
||||||
|
|
||||||
|
log("UPLOAD", f"[{remote_rel}] ({format_bytes(local_path.stat().st_size)})")
|
||||||
|
|
||||||
|
# Step 1 — ensure the remote directory exists
|
||||||
|
if not cfg.get("dry_run"):
|
||||||
|
if not _ensure_remote_dir(remote_dir, host, cfg, created_dirs):
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Step 2 — rsync the file (no --relative needed)
|
||||||
|
cmd = [
|
||||||
|
"rsync",
|
||||||
|
"--archive", # preserve perms / timestamps
|
||||||
|
"--compress", # compress during transfer
|
||||||
|
"--partial", # resume if interrupted
|
||||||
|
"-e", f"ssh {ssh_args}",
|
||||||
|
]
|
||||||
|
if cfg.get("dry_run"):
|
||||||
|
cmd.append("--dry-run")
|
||||||
|
cmd += [str(local_path), remote_target]
|
||||||
|
|
||||||
|
log("CMD", " ".join(cmd))
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
cmd,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
stderr = result.stderr.decode("latin-1", errors="replace").strip()
|
||||||
|
|
||||||
|
if result.returncode == 0:
|
||||||
|
log("OK", f"✓ {remote_rel}")
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
log("ERROR", f"✗ {remote_rel} (rc={result.returncode})")
|
||||||
|
if stderr:
|
||||||
|
log("ERROR", f" {stderr.splitlines()[0]}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
log("ERROR",
|
||||||
|
f"Timeout ({timeout}s) uploading: {remote_rel}. "
|
||||||
|
"Increase 'timeout_seconds' in config for large files.")
|
||||||
|
return False
|
||||||
|
except FileNotFoundError:
|
||||||
|
log("ERROR", "rsync not found. Install: brew install rsync")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# Main
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
def main():
|
||||||
|
log("─" * 64, "")
|
||||||
|
log("START", "Downloads uploader scan triggered")
|
||||||
|
|
||||||
|
cfg = load_config()
|
||||||
|
state = load_state()
|
||||||
|
|
||||||
|
if cfg.get("dry_run"):
|
||||||
|
log("INFO", "DRY RUN — no files will actually be transferred")
|
||||||
|
|
||||||
|
scan_dir = resolve_scan_dir(cfg)
|
||||||
|
|
||||||
|
# ── 1. Fetch remote file tree ──────────────────────────────────────────────
|
||||||
|
remote_files = get_remote_file_tree(cfg)
|
||||||
|
|
||||||
|
# ── 2. Diff local vs remote ────────────────────────────────────────────────
|
||||||
|
to_upload = collect_files_to_upload(scan_dir, remote_files, cfg)
|
||||||
|
|
||||||
|
if not to_upload:
|
||||||
|
log("DONE", "Everything is already in sync. Nothing to upload.")
|
||||||
|
return
|
||||||
|
|
||||||
|
log("INFO",
|
||||||
|
f"Starting upload of {len(to_upload)} file(s) "
|
||||||
|
f"({format_bytes(sum(p.stat().st_size for p, _ in to_upload))} total)")
|
||||||
|
|
||||||
|
# ── 3. Upload file by file ─────────────────────────────────────────────────
|
||||||
|
uploaded = 0
|
||||||
|
failed = 0
|
||||||
|
created_dirs: set[str] = set() # remote dirs created this session (cache)
|
||||||
|
|
||||||
|
for idx, (local_path, remote_rel) in enumerate(to_upload, start=1):
|
||||||
|
log("INFO", f"── [{idx}/{len(to_upload)}] {remote_rel}")
|
||||||
|
|
||||||
|
success = upload_file(local_path, remote_rel, cfg, created_dirs)
|
||||||
|
|
||||||
|
if success:
|
||||||
|
uploaded += 1
|
||||||
|
state["uploaded"][remote_rel] = {
|
||||||
|
"size": local_path.stat().st_size,
|
||||||
|
"mtime": local_path.stat().st_mtime,
|
||||||
|
"uploaded_at": datetime.now().isoformat(timespec="seconds"),
|
||||||
|
}
|
||||||
|
save_state(state) # persist after every successful file
|
||||||
|
else:
|
||||||
|
failed += 1
|
||||||
|
|
||||||
|
log("DONE",
|
||||||
|
f"Session complete — "
|
||||||
|
f"uploaded: {uploaded}, failed: {failed}, total: {len(to_upload)}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
{
|
||||||
|
"remote_destination": "filez@10.10.8.1:/mnt/ssd2t/shared/torrent/Downloads/",
|
||||||
|
"ssh_key_path": "",
|
||||||
|
"ssh_port": 22,
|
||||||
|
"scan_directory": "/Users/mcer323/Downloads",
|
||||||
|
"skip_patterns": [
|
||||||
|
".*",
|
||||||
|
"*.crdownload",
|
||||||
|
"*.part",
|
||||||
|
"*.tmp",
|
||||||
|
"uploader",
|
||||||
|
"organizer",
|
||||||
|
"ORGANIZER_LOG.txt",
|
||||||
|
"organizer_log*.txt",
|
||||||
|
"__pycache__"
|
||||||
|
],
|
||||||
|
"min_file_age_seconds": 30,
|
||||||
|
"timeout_seconds": 7200,
|
||||||
|
"remote_scan_timeout": 120,
|
||||||
|
"dry_run": false
|
||||||
|
}
|
||||||
Executable
+180
@@ -0,0 +1,180 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# ═══════════════════════════════════════════════════════════════════
|
||||||
|
# Downloads Auto-Uploader — macOS launchd installer
|
||||||
|
# Runs downloads_uploader.py every 5 minutes in the background.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./downloads_uploader_install.sh install # register & start
|
||||||
|
# ./downloads_uploader_install.sh uninstall # stop & remove
|
||||||
|
# ./downloads_uploader_install.sh status # show current state
|
||||||
|
# ./downloads_uploader_install.sh run # manual one-shot run
|
||||||
|
# ═══════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
LABEL="com.user.downloads-uploader"
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
SCAN_DIR="$(dirname "$SCRIPT_DIR")" # parent of uploader/ = Downloads/
|
||||||
|
PYTHON_SCRIPT="$SCRIPT_DIR/downloads_uploader.py"
|
||||||
|
PLIST_DIR="$HOME/Library/LaunchAgents"
|
||||||
|
PLIST_FILE="$PLIST_DIR/$LABEL.plist"
|
||||||
|
LOG_OUT="$SCRIPT_DIR/uploader_stdout.log"
|
||||||
|
LOG_ERR="$SCRIPT_DIR/uploader_stderr.log"
|
||||||
|
INTERVAL=300 # seconds (5 minutes)
|
||||||
|
|
||||||
|
# macOS shows the executable name in permission dialogs, not the script name.
|
||||||
|
# To display "downloads-uploader" instead of "python3.13", we create a symlink
|
||||||
|
# named "downloads-uploader" pointing to the real python3 binary, and use that
|
||||||
|
# symlink in the launchd plist.
|
||||||
|
SYMLINK_DIR="$HOME/.local/bin"
|
||||||
|
SYMLINK="$SYMLINK_DIR/downloads-uploader"
|
||||||
|
|
||||||
|
# Detect python3
|
||||||
|
PYTHON=$(command -v python3 || command -v python || echo "")
|
||||||
|
if [[ -z "$PYTHON" ]]; then
|
||||||
|
echo "❌ python3 not found. Install it: brew install python"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ───────────────────────────────────────────────
|
||||||
|
generate_plist() {
|
||||||
|
cat <<PLIST
|
||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
|
||||||
|
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||||
|
<plist version="1.0">
|
||||||
|
<dict>
|
||||||
|
<key>Label</key>
|
||||||
|
<string>$LABEL</string>
|
||||||
|
|
||||||
|
<key>ProgramArguments</key>
|
||||||
|
<array>
|
||||||
|
<string>$SYMLINK</string>
|
||||||
|
<string>$PYTHON_SCRIPT</string>
|
||||||
|
</array>
|
||||||
|
|
||||||
|
<key>StartInterval</key>
|
||||||
|
<integer>$INTERVAL</integer>
|
||||||
|
|
||||||
|
<key>RunAtLoad</key>
|
||||||
|
<true/>
|
||||||
|
|
||||||
|
<!-- Pin the working directory so __file__ resolves correctly -->
|
||||||
|
<key>WorkingDirectory</key>
|
||||||
|
<string>$SCRIPT_DIR</string>
|
||||||
|
|
||||||
|
<key>StandardOutPath</key>
|
||||||
|
<string>$LOG_OUT</string>
|
||||||
|
|
||||||
|
<key>StandardErrorPath</key>
|
||||||
|
<string>$LOG_ERR</string>
|
||||||
|
|
||||||
|
<key>KeepAlive</key>
|
||||||
|
<false/>
|
||||||
|
</dict>
|
||||||
|
</plist>
|
||||||
|
PLIST
|
||||||
|
}
|
||||||
|
|
||||||
|
# ───────────────────────────────────────────────
|
||||||
|
cmd_install() {
|
||||||
|
echo "📦 Installing Downloads Auto-Uploader..."
|
||||||
|
|
||||||
|
if [[ ! -f "$PYTHON_SCRIPT" ]]; then
|
||||||
|
echo "❌ Script not found: $PYTHON_SCRIPT"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create named symlink so macOS shows "downloads-uploader" in permission dialogs
|
||||||
|
mkdir -p "$SYMLINK_DIR"
|
||||||
|
if [[ -L "$SYMLINK" || -e "$SYMLINK" ]]; then
|
||||||
|
rm -f "$SYMLINK"
|
||||||
|
fi
|
||||||
|
ln -s "$PYTHON" "$SYMLINK"
|
||||||
|
echo "✅ Symlink created: $SYMLINK → $PYTHON"
|
||||||
|
echo " (macOS permission dialogs will now show 'downloads-uploader')"
|
||||||
|
|
||||||
|
# Write the absolute scan_directory into the config so launchd's cwd=/
|
||||||
|
# never affects path resolution inside the Python script.
|
||||||
|
"$PYTHON" - "$SCRIPT_DIR" "$SCAN_DIR" <<'PYEOF'
|
||||||
|
import json, sys
|
||||||
|
cfg_path = sys.argv[1] + "/downloads_uploader_config.json"
|
||||||
|
scan_dir = sys.argv[2]
|
||||||
|
with open(cfg_path) as f:
|
||||||
|
cfg = json.load(f)
|
||||||
|
cfg["scan_directory"] = scan_dir
|
||||||
|
with open(cfg_path, "w") as f:
|
||||||
|
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||||
|
print(f" scan_directory set to: {scan_dir}")
|
||||||
|
PYEOF
|
||||||
|
echo "✅ Config updated with absolute scan_directory"
|
||||||
|
|
||||||
|
mkdir -p "$PLIST_DIR"
|
||||||
|
generate_plist > "$PLIST_FILE"
|
||||||
|
echo "✅ Plist written: $PLIST_FILE"
|
||||||
|
|
||||||
|
# Unload first if already loaded
|
||||||
|
launchctl unload "$PLIST_FILE" 2>/dev/null || true
|
||||||
|
launchctl load -w "$PLIST_FILE"
|
||||||
|
echo "✅ Agent loaded. Script will run every $((INTERVAL / 60)) minutes."
|
||||||
|
echo " Logs → $LOG_OUT"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_uninstall() {
|
||||||
|
echo "🗑 Uninstalling Downloads Auto-Uploader..."
|
||||||
|
if [[ -f "$PLIST_FILE" ]]; then
|
||||||
|
launchctl unload -w "$PLIST_FILE" 2>/dev/null || true
|
||||||
|
rm -f "$PLIST_FILE"
|
||||||
|
echo "✅ Removed $PLIST_FILE"
|
||||||
|
else
|
||||||
|
echo "ℹ️ Plist not found — already uninstalled."
|
||||||
|
fi
|
||||||
|
if [[ -L "$SYMLINK" ]]; then
|
||||||
|
rm -f "$SYMLINK"
|
||||||
|
echo "✅ Removed symlink $SYMLINK"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_status() {
|
||||||
|
echo "── Status ────────────────────────────────"
|
||||||
|
if launchctl list | grep -q "$LABEL"; then
|
||||||
|
echo "✅ Agent is RUNNING (label: $LABEL)"
|
||||||
|
launchctl list "$LABEL" 2>/dev/null || true
|
||||||
|
else
|
||||||
|
echo "⏹ Agent is NOT loaded."
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
echo "── Files ─────────────────────────────────"
|
||||||
|
echo " Script dir : $SCRIPT_DIR"
|
||||||
|
echo " Scan dir : $(dirname "$SCRIPT_DIR")"
|
||||||
|
echo " Config : $SCRIPT_DIR/downloads_uploader_config.json"
|
||||||
|
echo " State : $SCRIPT_DIR/.uploader_state.json"
|
||||||
|
echo " App log : $SCRIPT_DIR/uploader_log.txt"
|
||||||
|
echo " Stdout log : $LOG_OUT"
|
||||||
|
echo " Stderr log : $LOG_ERR"
|
||||||
|
echo ""
|
||||||
|
echo "── Recent activity (uploader_log.txt) ────"
|
||||||
|
LOG_MAIN="$SCRIPT_DIR/uploader_log.txt"
|
||||||
|
[[ -f "$LOG_MAIN" ]] && tail -25 "$LOG_MAIN" || echo "(no log yet)"
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_run() {
|
||||||
|
echo "▶ Running uploader now (one-shot)..."
|
||||||
|
"$PYTHON" "$PYTHON_SCRIPT"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ───────────────────────────────────────────────
|
||||||
|
case "${1:-help}" in
|
||||||
|
install) cmd_install ;;
|
||||||
|
uninstall) cmd_uninstall ;;
|
||||||
|
status) cmd_status ;;
|
||||||
|
run) cmd_run ;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 {install|uninstall|status|run}"
|
||||||
|
echo ""
|
||||||
|
echo " install Register the launchd agent (runs every 5 min)"
|
||||||
|
echo " uninstall Remove the launchd agent"
|
||||||
|
echo " status Show if the agent is running + recent log"
|
||||||
|
echo " run Run the uploader once right now"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
+2036
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,13 @@
|
|||||||
|
Traceback (most recent call last):
|
||||||
|
File "/Users/mcer323/Downloads/uploader/downloads_uploader.py", line 463, in <module>
|
||||||
|
main()
|
||||||
|
~~~~^^
|
||||||
|
File "/Users/mcer323/Downloads/uploader/downloads_uploader.py", line 444, in main
|
||||||
|
success = upload_file(local_path, remote_rel, cfg, created_dirs)
|
||||||
|
File "/Users/mcer323/Downloads/uploader/downloads_uploader.py", line 357, in upload_file
|
||||||
|
log("UPLOAD", f"[{remote_rel}] ({format_bytes(local_path.stat().st_size)})")
|
||||||
|
~~~~~~~~~~~~~~~^^
|
||||||
|
File "/opt/homebrew/Cellar/python@3.14/3.14.4/Frameworks/Python.framework/Versions/3.14/lib/python3.14/pathlib/__init__.py", line 654, in stat
|
||||||
|
return os.stat(self, follow_symlinks=follow_symlinks)
|
||||||
|
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
FileNotFoundError: [Errno 2] No such file or directory: '/Users/mcer323/Downloads/Архивы/painter_website.zip'
|
||||||
+2036
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user