Add uploader scripts and docs
This commit is contained in:
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(brew search:*)"
|
||||
]
|
||||
}
|
||||
}
|
||||
+18969
File diff suppressed because it is too large
Load Diff
+124
@@ -0,0 +1,124 @@
|
||||
# Installation
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Python 3
|
||||
|
||||
macOS includes Python 3. Verify with:
|
||||
|
||||
```bash
|
||||
python3 --version
|
||||
```
|
||||
|
||||
If missing: `brew install python`
|
||||
|
||||
### 2. rsync
|
||||
|
||||
macOS ships rsync 2.6.9 (2006), which has bugs with path handling. Install the current version:
|
||||
|
||||
```bash
|
||||
brew install rsync
|
||||
```
|
||||
|
||||
### 3. SSH key access to the remote host
|
||||
|
||||
The uploader runs unattended, so it must be able to reach the remote host without a password. This means SSH key-based authentication needs to be in place before you install.
|
||||
|
||||
If you don't already have an SSH key, generate one:
|
||||
|
||||
```bash
|
||||
ssh-keygen -t ed25519
|
||||
```
|
||||
|
||||
Then authorize it on the remote host:
|
||||
|
||||
```bash
|
||||
ssh-copy-id filez@10.10.8.1
|
||||
```
|
||||
|
||||
Confirm the connection works without any password prompt before continuing:
|
||||
|
||||
```bash
|
||||
ssh filez@10.10.8.1 echo ok
|
||||
```
|
||||
|
||||
If your key lives somewhere other than the default location (`~/.ssh/id_rsa` or `~/.ssh/id_ed25519`), set the `ssh_key_path` field in `downloads_uploader_config.json` to its full path.
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
cd ~/Downloads/uploader
|
||||
./downloads_uploader_install.sh install
|
||||
```
|
||||
|
||||
This will:
|
||||
1. Create `~/.local/bin/downloads-uploader` (a named symlink to `python3` so macOS permission dialogs show a friendly name)
|
||||
2. Write `~/Library/LaunchAgents/com.user.downloads-uploader.plist`
|
||||
3. Load the agent — it runs immediately and then every 5 minutes
|
||||
|
||||
**macOS may prompt for network/file access permissions** the first time. Allow them.
|
||||
|
||||
---
|
||||
|
||||
## Configure
|
||||
|
||||
Open `downloads_uploader_config.json` and set your remote destination:
|
||||
|
||||
```json
|
||||
{
|
||||
"remote_destination": "user@your-host:/path/to/remote/dir/",
|
||||
"ssh_key_path": "",
|
||||
"ssh_port": 22
|
||||
}
|
||||
```
|
||||
|
||||
Then trigger a fresh run:
|
||||
|
||||
```bash
|
||||
./downloads_uploader_install.sh run
|
||||
```
|
||||
|
||||
Check the output:
|
||||
|
||||
```bash
|
||||
./downloads_uploader_install.sh status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Uninstall
|
||||
|
||||
```bash
|
||||
./downloads_uploader_install.sh uninstall
|
||||
```
|
||||
|
||||
This stops and removes the launchd agent and the symlink. Config, logs, and state files are left in place.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"rsync not found"**
|
||||
```bash
|
||||
brew install rsync
|
||||
```
|
||||
|
||||
**SSH connects but `find` fails on remote**
|
||||
The script uses GNU `find -printf`, which requires Linux. BSD/macOS remotes are not supported on the remote side.
|
||||
|
||||
**Files keep re-uploading**
|
||||
Size mismatch between local and remote. Check that the remote filesystem isn't silently modifying files (e.g. a torrent client moving them). You can inspect `.uploader_state.json` for the last recorded sizes.
|
||||
|
||||
**Agent not running after reboot**
|
||||
Verify the plist is in `~/Library/LaunchAgents/` and loaded:
|
||||
```bash
|
||||
launchctl list | grep downloads-uploader
|
||||
```
|
||||
If missing, re-run `./downloads_uploader_install.sh install`.
|
||||
|
||||
**See full logs**
|
||||
```bash
|
||||
tail -f ~/Downloads/uploader/uploader_log.txt
|
||||
```
|
||||
Executable
+463
@@ -0,0 +1,463 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Downloads Auto-Uploader
|
||||
=======================
|
||||
Scans the Downloads folder, fetches the remote file tree via SSH,
|
||||
and uploads — file by file — only what is missing or has changed.
|
||||
|
||||
Config file : downloads_uploader_config.json (same folder as this script)
|
||||
State file : .uploader_state.json (audit log of uploads)
|
||||
Log file : uploader_log.txt
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import subprocess
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# ── Safety: never scan these paths ────────────────────────────────────────────
|
||||
FORBIDDEN_SCAN_ROOTS = {
|
||||
"/", "/private", "/private/var",
|
||||
"/usr", "/bin", "/sbin", "/etc", "/var",
|
||||
"/System", "/Library", "/Applications",
|
||||
"/home", "/opt", "/cores",
|
||||
}
|
||||
|
||||
# ── Fixed paths (relative to this script's directory) ─────────────────────────
|
||||
SCRIPT_DIR = Path(__file__).parent.resolve()
|
||||
SCAN_DEFAULT = SCRIPT_DIR.parent # one level up: Downloads/
|
||||
CONFIG_FILE = SCRIPT_DIR / "downloads_uploader_config.json"
|
||||
STATE_FILE = SCRIPT_DIR / ".uploader_state.json"
|
||||
LOG_FILE = SCRIPT_DIR / "uploader_log.txt"
|
||||
|
||||
# ── Defaults ───────────────────────────────────────────────────────────────────
|
||||
DEFAULT_CONFIG = {
|
||||
"remote_destination": "filez@10.10.8.1:/mnt/ssd2t/shared/torrent/Downloads/",
|
||||
"ssh_key_path": "", # empty = use ~/.ssh/id_rsa (system default)
|
||||
"ssh_port": 22,
|
||||
# Empty = auto-detect (parent of the uploader/ folder = Downloads/).
|
||||
# Set an absolute path to watch a different directory.
|
||||
"scan_directory": "",
|
||||
"skip_patterns": [
|
||||
".*", # hidden / dot-files
|
||||
"*.crdownload", # incomplete Chrome downloads
|
||||
"*.part", # incomplete Firefox downloads
|
||||
"*.tmp",
|
||||
"uploader", # this tool's own folder
|
||||
"ORGANIZER_LOG.txt",
|
||||
"organizer_log*.txt",
|
||||
"__pycache__",
|
||||
],
|
||||
"min_file_age_seconds": 30, # ignore files newer than this (still being written)
|
||||
"timeout_seconds": 7200, # per-file rsync timeout (2 h)
|
||||
"remote_scan_timeout": 120, # SSH find timeout
|
||||
"dry_run": False, # True = log only, no actual transfer
|
||||
}
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# Config / State helpers
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def load_config() -> dict:
|
||||
if CONFIG_FILE.exists():
|
||||
with open(CONFIG_FILE, "r", encoding="utf-8") as f:
|
||||
user_cfg = json.load(f)
|
||||
return {**DEFAULT_CONFIG, **user_cfg}
|
||||
cfg = DEFAULT_CONFIG.copy()
|
||||
save_config(cfg)
|
||||
log("INFO", f"Created default config: {CONFIG_FILE}")
|
||||
return cfg
|
||||
|
||||
|
||||
def save_config(cfg: dict):
|
||||
with open(CONFIG_FILE, "w", encoding="utf-8") as f:
|
||||
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
def load_state() -> dict:
|
||||
if STATE_FILE.exists():
|
||||
with open(STATE_FILE, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
return {"uploaded": {}}
|
||||
|
||||
|
||||
def save_state(state: dict):
|
||||
with open(STATE_FILE, "w", encoding="utf-8") as f:
|
||||
json.dump(state, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# Logging
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def log(level: str, message: str):
|
||||
ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
line = f"[{ts}] {level:7s} {message}"
|
||||
print(line)
|
||||
with open(LOG_FILE, "a", encoding="utf-8") as f:
|
||||
f.write(line + "\n")
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# Utilities
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def matches_skip_pattern(name: str, patterns: list) -> bool:
|
||||
import fnmatch
|
||||
return any(fnmatch.fnmatch(name, p) for p in patterns)
|
||||
|
||||
|
||||
def _ssh_opts(cfg: dict) -> list[str]:
|
||||
"""Return ssh CLI option list from config."""
|
||||
opts = [
|
||||
"-p", str(cfg["ssh_port"]),
|
||||
"-o", "StrictHostKeyChecking=accept-new",
|
||||
"-o", "BatchMode=yes", # never prompt for a password
|
||||
"-o", "ConnectTimeout=15",
|
||||
]
|
||||
if cfg.get("ssh_key_path"):
|
||||
opts += ["-i", cfg["ssh_key_path"]]
|
||||
return opts
|
||||
|
||||
|
||||
def _parse_remote(destination: str) -> tuple[str, str]:
|
||||
"""Split 'user@host:/path' into ('user@host', '/path')."""
|
||||
host, rpath = destination.split(":", 1)
|
||||
return host, rpath.rstrip("/")
|
||||
|
||||
|
||||
def format_bytes(n: int) -> str:
|
||||
for unit in ("B", "KB", "MB", "GB"):
|
||||
if n < 1024:
|
||||
return f"{n:.1f} {unit}"
|
||||
n /= 1024
|
||||
return f"{n:.1f} TB"
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# Scan directory validation
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def resolve_scan_dir(cfg: dict) -> Path:
|
||||
raw = cfg.get("scan_directory", "").strip()
|
||||
scan_dir = Path(raw).resolve() if raw else SCAN_DEFAULT.resolve()
|
||||
|
||||
log("INFO", f"Script dir : {SCRIPT_DIR}")
|
||||
log("INFO", f"Scan dir : {scan_dir}")
|
||||
|
||||
if str(scan_dir) in FORBIDDEN_SCAN_ROOTS or scan_dir == Path("/"):
|
||||
log("FATAL",
|
||||
f"Refusing to scan a system/root path: {scan_dir}\n"
|
||||
" → Set 'scan_directory' in downloads_uploader_config.json\n"
|
||||
" → Or re-run: ./downloads_uploader_install.sh install")
|
||||
sys.exit(1)
|
||||
|
||||
if not scan_dir.is_dir():
|
||||
log("FATAL", f"scan_directory does not exist: {scan_dir}")
|
||||
sys.exit(1)
|
||||
|
||||
return scan_dir
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# Remote file tree
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def get_remote_file_tree(cfg: dict) -> dict[str, int]:
|
||||
"""
|
||||
SSH into the remote host, run `find`, and return a dict of
|
||||
{ relative_path : size_in_bytes } for every file under remote_destination.
|
||||
|
||||
Uses GNU find's -printf '%P\\t%s\\n' (Linux servers have this).
|
||||
If the remote directory doesn't exist yet, returns an empty dict.
|
||||
On any error, logs a warning and returns {} (safe: everything gets uploaded).
|
||||
"""
|
||||
remote = cfg["remote_destination"]
|
||||
host, rpath = remote.split(":", 1)
|
||||
rpath = rpath.rstrip("/")
|
||||
timeout = cfg.get("remote_scan_timeout", 120)
|
||||
|
||||
# find with -printf (GNU / Linux); || true so SSH exits 0 even if dir missing
|
||||
find_cmd = (
|
||||
f'find "{rpath}" -type f -printf "%P\\t%s\\n" 2>/dev/null || true'
|
||||
)
|
||||
cmd = ["ssh"] + _ssh_opts(cfg) + [host, find_cmd]
|
||||
|
||||
log("INFO", f"Fetching remote file tree: {host}:{rpath} ...")
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
timeout=timeout,
|
||||
)
|
||||
stderr = result.stderr.decode("utf-8", errors="replace").strip()
|
||||
if result.returncode != 0 and stderr:
|
||||
log("WARN", f"SSH stderr: {stderr.splitlines()[0]}")
|
||||
|
||||
remote_files: dict[str, int] = {}
|
||||
for line in result.stdout.decode("utf-8", errors="replace").splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
parts = line.split("\t", 1)
|
||||
if len(parts) == 2:
|
||||
rel, size_str = parts
|
||||
try:
|
||||
remote_files[rel] = int(size_str)
|
||||
except ValueError:
|
||||
remote_files[rel] = -1 # unknown size → will compare by name only
|
||||
else:
|
||||
remote_files[line] = -1
|
||||
|
||||
log("INFO", f"Remote tree: {len(remote_files):,} file(s) indexed")
|
||||
return remote_files
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
log("WARN",
|
||||
f"Timeout ({timeout}s) fetching remote tree. "
|
||||
"Treating remote as empty — all local files will be uploaded.")
|
||||
return {}
|
||||
except Exception as exc:
|
||||
log("WARN", f"Could not fetch remote tree: {exc}. Treating remote as empty.")
|
||||
return {}
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# Local ↔ Remote diff
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def collect_files_to_upload(
|
||||
scan_dir: Path,
|
||||
remote_files: dict[str, int],
|
||||
cfg: dict,
|
||||
) -> list[tuple[Path, str]]:
|
||||
"""
|
||||
Walk every candidate under scan_dir and compare with remote_files.
|
||||
Returns [(local_abs_path, remote_relative_path), ...] for files to upload.
|
||||
remote_relative_path uses forward slashes, relative to scan_dir.
|
||||
"""
|
||||
patterns = cfg["skip_patterns"]
|
||||
min_age = cfg["min_file_age_seconds"]
|
||||
now = datetime.now().timestamp()
|
||||
to_upload = []
|
||||
synced_count = 0
|
||||
skipped_count = 0
|
||||
|
||||
for top in sorted(scan_dir.iterdir()):
|
||||
if matches_skip_pattern(top.name, patterns):
|
||||
continue
|
||||
|
||||
try:
|
||||
age = now - top.stat().st_mtime
|
||||
if age < min_age:
|
||||
log("SKIP", f"{top.name} (age={age:.0f}s, threshold={min_age}s)")
|
||||
skipped_count += 1
|
||||
continue
|
||||
except OSError:
|
||||
continue
|
||||
|
||||
# Enumerate all files: a single file, or every file inside a directory
|
||||
if top.is_file():
|
||||
local_files = [top]
|
||||
elif top.is_dir():
|
||||
local_files = sorted(f for f in top.rglob("*") if f.is_file())
|
||||
else:
|
||||
continue
|
||||
|
||||
for lf in local_files:
|
||||
rel = lf.relative_to(scan_dir).as_posix() # forward-slash path
|
||||
local_size = lf.stat().st_size
|
||||
remote_size = remote_files.get(rel)
|
||||
|
||||
if remote_size is None:
|
||||
# File is not on remote at all
|
||||
to_upload.append((lf, rel))
|
||||
elif remote_size == -1:
|
||||
# Remote size unknown (find didn't return it) — upload to be safe
|
||||
to_upload.append((lf, rel))
|
||||
elif remote_size != local_size:
|
||||
log("DIFF",
|
||||
f"{rel} "
|
||||
f"local={format_bytes(local_size)} "
|
||||
f"remote={format_bytes(remote_size)} → will re-upload")
|
||||
to_upload.append((lf, rel))
|
||||
else:
|
||||
synced_count += 1 # identical size → skip
|
||||
|
||||
log("INFO",
|
||||
f"Diff complete: "
|
||||
f"{len(to_upload)} to upload, "
|
||||
f"{synced_count} already in sync, "
|
||||
f"{skipped_count} skipped (too recent)")
|
||||
return to_upload
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# Per-file upload
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def _ensure_remote_dir(remote_dir: str, host: str, cfg: dict,
|
||||
created_dirs: set) -> bool:
|
||||
"""
|
||||
Create *remote_dir* on *host* via SSH mkdir -p (once per session per dir).
|
||||
Returns True on success, False on error.
|
||||
"""
|
||||
if remote_dir in created_dirs:
|
||||
return True
|
||||
cmd = ["ssh"] + _ssh_opts(cfg) + [host, f"mkdir -p '{remote_dir}'"]
|
||||
try:
|
||||
result = subprocess.run(cmd, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE, timeout=30)
|
||||
if result.returncode == 0:
|
||||
created_dirs.add(remote_dir)
|
||||
return True
|
||||
err = result.stderr.decode("utf-8", errors="replace").strip()
|
||||
log("ERROR", f"mkdir -p failed for {remote_dir}: {err}")
|
||||
return False
|
||||
except Exception as exc:
|
||||
log("ERROR", f"mkdir -p exception: {exc}")
|
||||
return False
|
||||
|
||||
|
||||
def upload_file(
|
||||
local_path: Path,
|
||||
remote_rel: str,
|
||||
cfg: dict,
|
||||
created_dirs: set,
|
||||
) -> bool:
|
||||
"""
|
||||
Upload a single file to the exact remote path, preserving the
|
||||
relative directory structure.
|
||||
|
||||
Strategy (no --relative, no ./ anchor tricks):
|
||||
1. SSH mkdir -p the remote parent directory (cached per session).
|
||||
2. rsync the file into that directory.
|
||||
|
||||
This avoids the macOS rsync 2.6.9 bug where --relative recreates the
|
||||
entire absolute local path (/Users/mcer323/Downloads/…) on the remote.
|
||||
"""
|
||||
host, rbase = _parse_remote(cfg["remote_destination"])
|
||||
timeout = cfg.get("timeout_seconds", 7200)
|
||||
ssh_args = " ".join(_ssh_opts(cfg))
|
||||
|
||||
# Compute the remote directory for this file
|
||||
parent = Path(remote_rel).parent
|
||||
if str(parent) == ".":
|
||||
remote_dir = rbase # file sits directly in Downloads
|
||||
else:
|
||||
remote_dir = rbase + "/" + parent.as_posix()
|
||||
|
||||
remote_target = f"{host}:{remote_dir}/" # trailing / = place file inside dir
|
||||
|
||||
log("UPLOAD", f"[{remote_rel}] ({format_bytes(local_path.stat().st_size)})")
|
||||
|
||||
# Step 1 — ensure the remote directory exists
|
||||
if not cfg.get("dry_run"):
|
||||
if not _ensure_remote_dir(remote_dir, host, cfg, created_dirs):
|
||||
return False
|
||||
|
||||
# Step 2 — rsync the file (no --relative needed)
|
||||
cmd = [
|
||||
"rsync",
|
||||
"--archive", # preserve perms / timestamps
|
||||
"--compress", # compress during transfer
|
||||
"--partial", # resume if interrupted
|
||||
"-e", f"ssh {ssh_args}",
|
||||
]
|
||||
if cfg.get("dry_run"):
|
||||
cmd.append("--dry-run")
|
||||
cmd += [str(local_path), remote_target]
|
||||
|
||||
log("CMD", " ".join(cmd))
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
timeout=timeout,
|
||||
)
|
||||
stderr = result.stderr.decode("latin-1", errors="replace").strip()
|
||||
|
||||
if result.returncode == 0:
|
||||
log("OK", f"✓ {remote_rel}")
|
||||
return True
|
||||
else:
|
||||
log("ERROR", f"✗ {remote_rel} (rc={result.returncode})")
|
||||
if stderr:
|
||||
log("ERROR", f" {stderr.splitlines()[0]}")
|
||||
return False
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
log("ERROR",
|
||||
f"Timeout ({timeout}s) uploading: {remote_rel}. "
|
||||
"Increase 'timeout_seconds' in config for large files.")
|
||||
return False
|
||||
except FileNotFoundError:
|
||||
log("ERROR", "rsync not found. Install: brew install rsync")
|
||||
return False
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# Main
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
def main():
|
||||
log("─" * 64, "")
|
||||
log("START", "Downloads uploader scan triggered")
|
||||
|
||||
cfg = load_config()
|
||||
state = load_state()
|
||||
|
||||
if cfg.get("dry_run"):
|
||||
log("INFO", "DRY RUN — no files will actually be transferred")
|
||||
|
||||
scan_dir = resolve_scan_dir(cfg)
|
||||
|
||||
# ── 1. Fetch remote file tree ──────────────────────────────────────────────
|
||||
remote_files = get_remote_file_tree(cfg)
|
||||
|
||||
# ── 2. Diff local vs remote ────────────────────────────────────────────────
|
||||
to_upload = collect_files_to_upload(scan_dir, remote_files, cfg)
|
||||
|
||||
if not to_upload:
|
||||
log("DONE", "Everything is already in sync. Nothing to upload.")
|
||||
return
|
||||
|
||||
log("INFO",
|
||||
f"Starting upload of {len(to_upload)} file(s) "
|
||||
f"({format_bytes(sum(p.stat().st_size for p, _ in to_upload))} total)")
|
||||
|
||||
# ── 3. Upload file by file ─────────────────────────────────────────────────
|
||||
uploaded = 0
|
||||
failed = 0
|
||||
created_dirs: set[str] = set() # remote dirs created this session (cache)
|
||||
|
||||
for idx, (local_path, remote_rel) in enumerate(to_upload, start=1):
|
||||
log("INFO", f"── [{idx}/{len(to_upload)}] {remote_rel}")
|
||||
|
||||
success = upload_file(local_path, remote_rel, cfg, created_dirs)
|
||||
|
||||
if success:
|
||||
uploaded += 1
|
||||
state["uploaded"][remote_rel] = {
|
||||
"size": local_path.stat().st_size,
|
||||
"mtime": local_path.stat().st_mtime,
|
||||
"uploaded_at": datetime.now().isoformat(timespec="seconds"),
|
||||
}
|
||||
save_state(state) # persist after every successful file
|
||||
else:
|
||||
failed += 1
|
||||
|
||||
log("DONE",
|
||||
f"Session complete — "
|
||||
f"uploaded: {uploaded}, failed: {failed}, total: {len(to_upload)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"remote_destination": "filez@10.10.8.1:/mnt/ssd2t/shared/torrent/Downloads/",
|
||||
"ssh_key_path": "",
|
||||
"ssh_port": 22,
|
||||
"scan_directory": "/Users/mcer323/Downloads",
|
||||
"skip_patterns": [
|
||||
".*",
|
||||
"*.crdownload",
|
||||
"*.part",
|
||||
"*.tmp",
|
||||
"uploader",
|
||||
"organizer",
|
||||
"ORGANIZER_LOG.txt",
|
||||
"organizer_log*.txt",
|
||||
"__pycache__"
|
||||
],
|
||||
"min_file_age_seconds": 30,
|
||||
"timeout_seconds": 7200,
|
||||
"remote_scan_timeout": 120,
|
||||
"dry_run": false
|
||||
}
|
||||
Executable
+180
@@ -0,0 +1,180 @@
|
||||
#!/usr/bin/env bash
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
# Downloads Auto-Uploader — macOS launchd installer
|
||||
# Runs downloads_uploader.py every 5 minutes in the background.
|
||||
#
|
||||
# Usage:
|
||||
# ./downloads_uploader_install.sh install # register & start
|
||||
# ./downloads_uploader_install.sh uninstall # stop & remove
|
||||
# ./downloads_uploader_install.sh status # show current state
|
||||
# ./downloads_uploader_install.sh run # manual one-shot run
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
LABEL="com.user.downloads-uploader"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SCAN_DIR="$(dirname "$SCRIPT_DIR")" # parent of uploader/ = Downloads/
|
||||
PYTHON_SCRIPT="$SCRIPT_DIR/downloads_uploader.py"
|
||||
PLIST_DIR="$HOME/Library/LaunchAgents"
|
||||
PLIST_FILE="$PLIST_DIR/$LABEL.plist"
|
||||
LOG_OUT="$SCRIPT_DIR/uploader_stdout.log"
|
||||
LOG_ERR="$SCRIPT_DIR/uploader_stderr.log"
|
||||
INTERVAL=300 # seconds (5 minutes)
|
||||
|
||||
# macOS shows the executable name in permission dialogs, not the script name.
|
||||
# To display "downloads-uploader" instead of "python3.13", we create a symlink
|
||||
# named "downloads-uploader" pointing to the real python3 binary, and use that
|
||||
# symlink in the launchd plist.
|
||||
SYMLINK_DIR="$HOME/.local/bin"
|
||||
SYMLINK="$SYMLINK_DIR/downloads-uploader"
|
||||
|
||||
# Detect python3
|
||||
PYTHON=$(command -v python3 || command -v python || echo "")
|
||||
if [[ -z "$PYTHON" ]]; then
|
||||
echo "❌ python3 not found. Install it: brew install python"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ───────────────────────────────────────────────
|
||||
generate_plist() {
|
||||
cat <<PLIST
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
|
||||
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>$LABEL</string>
|
||||
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>$SYMLINK</string>
|
||||
<string>$PYTHON_SCRIPT</string>
|
||||
</array>
|
||||
|
||||
<key>StartInterval</key>
|
||||
<integer>$INTERVAL</integer>
|
||||
|
||||
<key>RunAtLoad</key>
|
||||
<true/>
|
||||
|
||||
<!-- Pin the working directory so __file__ resolves correctly -->
|
||||
<key>WorkingDirectory</key>
|
||||
<string>$SCRIPT_DIR</string>
|
||||
|
||||
<key>StandardOutPath</key>
|
||||
<string>$LOG_OUT</string>
|
||||
|
||||
<key>StandardErrorPath</key>
|
||||
<string>$LOG_ERR</string>
|
||||
|
||||
<key>KeepAlive</key>
|
||||
<false/>
|
||||
</dict>
|
||||
</plist>
|
||||
PLIST
|
||||
}
|
||||
|
||||
# ───────────────────────────────────────────────
|
||||
cmd_install() {
|
||||
echo "📦 Installing Downloads Auto-Uploader..."
|
||||
|
||||
if [[ ! -f "$PYTHON_SCRIPT" ]]; then
|
||||
echo "❌ Script not found: $PYTHON_SCRIPT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create named symlink so macOS shows "downloads-uploader" in permission dialogs
|
||||
mkdir -p "$SYMLINK_DIR"
|
||||
if [[ -L "$SYMLINK" || -e "$SYMLINK" ]]; then
|
||||
rm -f "$SYMLINK"
|
||||
fi
|
||||
ln -s "$PYTHON" "$SYMLINK"
|
||||
echo "✅ Symlink created: $SYMLINK → $PYTHON"
|
||||
echo " (macOS permission dialogs will now show 'downloads-uploader')"
|
||||
|
||||
# Write the absolute scan_directory into the config so launchd's cwd=/
|
||||
# never affects path resolution inside the Python script.
|
||||
"$PYTHON" - "$SCRIPT_DIR" "$SCAN_DIR" <<'PYEOF'
|
||||
import json, sys
|
||||
cfg_path = sys.argv[1] + "/downloads_uploader_config.json"
|
||||
scan_dir = sys.argv[2]
|
||||
with open(cfg_path) as f:
|
||||
cfg = json.load(f)
|
||||
cfg["scan_directory"] = scan_dir
|
||||
with open(cfg_path, "w") as f:
|
||||
json.dump(cfg, f, indent=2, ensure_ascii=False)
|
||||
print(f" scan_directory set to: {scan_dir}")
|
||||
PYEOF
|
||||
echo "✅ Config updated with absolute scan_directory"
|
||||
|
||||
mkdir -p "$PLIST_DIR"
|
||||
generate_plist > "$PLIST_FILE"
|
||||
echo "✅ Plist written: $PLIST_FILE"
|
||||
|
||||
# Unload first if already loaded
|
||||
launchctl unload "$PLIST_FILE" 2>/dev/null || true
|
||||
launchctl load -w "$PLIST_FILE"
|
||||
echo "✅ Agent loaded. Script will run every $((INTERVAL / 60)) minutes."
|
||||
echo " Logs → $LOG_OUT"
|
||||
}
|
||||
|
||||
cmd_uninstall() {
|
||||
echo "🗑 Uninstalling Downloads Auto-Uploader..."
|
||||
if [[ -f "$PLIST_FILE" ]]; then
|
||||
launchctl unload -w "$PLIST_FILE" 2>/dev/null || true
|
||||
rm -f "$PLIST_FILE"
|
||||
echo "✅ Removed $PLIST_FILE"
|
||||
else
|
||||
echo "ℹ️ Plist not found — already uninstalled."
|
||||
fi
|
||||
if [[ -L "$SYMLINK" ]]; then
|
||||
rm -f "$SYMLINK"
|
||||
echo "✅ Removed symlink $SYMLINK"
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_status() {
|
||||
echo "── Status ────────────────────────────────"
|
||||
if launchctl list | grep -q "$LABEL"; then
|
||||
echo "✅ Agent is RUNNING (label: $LABEL)"
|
||||
launchctl list "$LABEL" 2>/dev/null || true
|
||||
else
|
||||
echo "⏹ Agent is NOT loaded."
|
||||
fi
|
||||
echo ""
|
||||
echo "── Files ─────────────────────────────────"
|
||||
echo " Script dir : $SCRIPT_DIR"
|
||||
echo " Scan dir : $(dirname "$SCRIPT_DIR")"
|
||||
echo " Config : $SCRIPT_DIR/downloads_uploader_config.json"
|
||||
echo " State : $SCRIPT_DIR/.uploader_state.json"
|
||||
echo " App log : $SCRIPT_DIR/uploader_log.txt"
|
||||
echo " Stdout log : $LOG_OUT"
|
||||
echo " Stderr log : $LOG_ERR"
|
||||
echo ""
|
||||
echo "── Recent activity (uploader_log.txt) ────"
|
||||
LOG_MAIN="$SCRIPT_DIR/uploader_log.txt"
|
||||
[[ -f "$LOG_MAIN" ]] && tail -25 "$LOG_MAIN" || echo "(no log yet)"
|
||||
}
|
||||
|
||||
cmd_run() {
|
||||
echo "▶ Running uploader now (one-shot)..."
|
||||
"$PYTHON" "$PYTHON_SCRIPT"
|
||||
}
|
||||
|
||||
# ───────────────────────────────────────────────
|
||||
case "${1:-help}" in
|
||||
install) cmd_install ;;
|
||||
uninstall) cmd_uninstall ;;
|
||||
status) cmd_status ;;
|
||||
run) cmd_run ;;
|
||||
*)
|
||||
echo "Usage: $0 {install|uninstall|status|run}"
|
||||
echo ""
|
||||
echo " install Register the launchd agent (runs every 5 min)"
|
||||
echo " uninstall Remove the launchd agent"
|
||||
echo " status Show if the agent is running + recent log"
|
||||
echo " run Run the uploader once right now"
|
||||
;;
|
||||
esac
|
||||
+2036
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,13 @@
|
||||
Traceback (most recent call last):
|
||||
File "/Users/mcer323/Downloads/uploader/downloads_uploader.py", line 463, in <module>
|
||||
main()
|
||||
~~~~^^
|
||||
File "/Users/mcer323/Downloads/uploader/downloads_uploader.py", line 444, in main
|
||||
success = upload_file(local_path, remote_rel, cfg, created_dirs)
|
||||
File "/Users/mcer323/Downloads/uploader/downloads_uploader.py", line 357, in upload_file
|
||||
log("UPLOAD", f"[{remote_rel}] ({format_bytes(local_path.stat().st_size)})")
|
||||
~~~~~~~~~~~~~~~^^
|
||||
File "/opt/homebrew/Cellar/python@3.14/3.14.4/Frameworks/Python.framework/Versions/3.14/lib/python3.14/pathlib/__init__.py", line 654, in stat
|
||||
return os.stat(self, follow_symlinks=follow_symlinks)
|
||||
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
FileNotFoundError: [Errno 2] No such file or directory: '/Users/mcer323/Downloads/Архивы/painter_website.zip'
|
||||
+2036
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user