docs: document the CIFS/forceuid utime gotcha for FileWorker.move (shutil.copy2 path) #27

New issue

Open

opened 2026-05-13 16:04:52 +00:00 by gravityfargo · 0 comments

gravityfargo commented

2026-05-13 16:04:52 +00:00

Owner

Context

`FileWorker.move()` in `athena_file/main.py:156` uses `shutil.copy2` (via `self.copy()` at `main.py:136`). `copy2` is `copyfile` + `copystat`; the latter calls `os.utime(dst, ns=(...))` with explicit times.

Linux requires the calling process to own the file (or hold `CAP_FOWNER`) for `utime(path, times=non-NULL)`. Group-write permission is not sufficient.

The footgun

When the destination directory is on a CIFS / SMB mount with `uid=…,forceuid` options (very common with Hetzner StorageBox, Synology shares, etc.), the kernel forces every file's owner to the mount-specified UID regardless of who actually wrote it. If the writing process runs as a different UID, the `utime(non-NULL)` call inside `copystat` fails with `PermissionError: [Errno 1] Operation not permitted`.

I just hit this in `athena-archive-api`'s media upload route (`POST /media/upload/`). The file lands on disk successfully (the `copyfile` step inside `copy2` succeeds), then `copystat` raises and `FileWorker.move` propagates the exception → the route 500s. From the user's perspective: "uploads broke."

Traceback excerpt (full traceback in the BE repo's deploy logs):

```
File "athena_file/main.py:136" in copy:
shutil.copy2(self.path, new_path)
File "shutil.py:530" in copy2:
copystat(src, dst, follow_symlinks=follow)
File "shutil.py:446" in copystat:
lookup("utime")(dst, ns=(st.st_atime_ns, st.st_mtime_ns), …)
PermissionError: [Errno 1] Operation not permitted: '/media/storage/4.webp'
```

Reproduction

Mount a CIFS share with `uid=33,forceuid,gid=GGG,forcegid,file_mode=0774,dir_mode=0775` (e.g. a Hetzner StorageBox).
Run a Python process as uid != 33 (e.g. the `athena` uid in athena-archive-api's container, uid 1000).
`FileWorker(src).move(dst)` where `dst` is on the share.
→ `PermissionError: [Errno 1] Operation not permitted: ` from inside `shutil.copystat`.

Docs ask

A README or `FileWorker.move` docstring note along these lines:

Filesystem requirements

`FileWorker.move()` and `FileWorker.copy()` use `shutil.copy2`, which preserves file timestamps via `os.utime(path, times=…)`. The Linux kernel requires the writing process to own the destination file for that call. This is fine on local filesystems where the writer becomes the owner. It can fail on shares with owner-forcing mount options — most commonly:

CIFS / SMB with `uid=…,forceuid`

FUSE filesystems with `--force-user=…` (bindfs, mergerfs, …)

NFS with `all_squash` / squashed UIDs

Workaround (CIFS): add `noperm` to the mount options to delegate permission checks to the SMB server's ACL.
Workaround (FUSE/bindfs): set `--force-user` to match the writing process's UID.
Workaround (NFS): map the writer's UID via `no_root_squash` / explicit `anonuid` matching.

Optional follow-up: drop `copystat`

`shutil.copy2` → `shutil.copyfile` in `FileWorker.copy()` would resolve this library-side at the cost of not preserving src timestamps on the destination. For archive-storage use cases (`athena-archive-api`'s primary consumer) that trade-off is fine — the destination's creation time is the move time, not the temp file's mtime. Happy to PR if that's acceptable.

Surfaced from https://code.modernleft.org/ModernLeft/athena-archive-api debugging.

## Context \`FileWorker.move()\` in \`athena_file/main.py:156\` uses \`shutil.copy2\` (via \`self.copy()\` at \`main.py:136\`). \`copy2\` is \`copyfile\` + \`copystat\`; the latter calls \`os.utime(dst, ns=(...))\` with explicit times. Linux requires the calling process to **own the file** (or hold \`CAP_FOWNER\`) for \`utime(path, times=non-NULL)\`. Group-write permission is not sufficient. ## The footgun When the destination directory is on a **CIFS / SMB mount** with \`uid=…,forceuid\` options (very common with Hetzner StorageBox, Synology shares, etc.), the kernel forces every file's owner to the mount-specified UID **regardless of who actually wrote it**. If the writing process runs as a different UID, the \`utime(non-NULL)\` call inside \`copystat\` fails with \`PermissionError: [Errno 1] Operation not permitted\`. I just hit this in \`athena-archive-api\`'s media upload route (\`POST /media/upload/\`). The file lands on disk successfully (the \`copyfile\` step inside \`copy2\` succeeds), then \`copystat\` raises and \`FileWorker.move\` propagates the exception → the route 500s. From the user's perspective: "uploads broke." Traceback excerpt (full traceback in the BE repo's deploy logs): \`\`\` File \"athena_file/main.py:136\" in copy: shutil.copy2(self.path, new_path) File \"shutil.py:530\" in copy2: copystat(src, dst, follow_symlinks=follow) File \"shutil.py:446\" in copystat: lookup(\"utime\")(dst, ns=(st.st_atime_ns, st.st_mtime_ns), …) PermissionError: [Errno 1] Operation not permitted: '/media/storage/4.webp' \`\`\` ## Reproduction 1. Mount a CIFS share with \`uid=33,forceuid,gid=GGG,forcegid,file_mode=0774,dir_mode=0775\` (e.g. a Hetzner StorageBox). 2. Run a Python process as uid != 33 (e.g. the \`athena\` uid in athena-archive-api's container, uid 1000). 3. \`FileWorker(src).move(dst)\` where \`dst\` is on the share. 4. → \`PermissionError: [Errno 1] Operation not permitted: <dst>\` from inside \`shutil.copystat\`. ## Docs ask A README or \`FileWorker.move\` docstring note along these lines: > ### Filesystem requirements > > \`FileWorker.move()\` and \`FileWorker.copy()\` use \`shutil.copy2\`, which preserves file timestamps via \`os.utime(path, times=…)\`. The Linux kernel requires the **writing process to own the destination file** for that call. This is fine on local filesystems where the writer becomes the owner. It can fail on shares with **owner-forcing mount options** — most commonly: > > - CIFS / SMB with \`uid=…,forceuid\` > - FUSE filesystems with \`--force-user=…\` (bindfs, mergerfs, …) > - NFS with \`all_squash\` / squashed UIDs > > **Workaround (CIFS):** add \`noperm\` to the mount options to delegate permission checks to the SMB server's ACL. > **Workaround (FUSE/bindfs):** set \`--force-user\` to match the writing process's UID. > **Workaround (NFS):** map the writer's UID via \`no_root_squash\` / explicit \`anonuid\` matching. ## Optional follow-up: drop \`copystat\` \`shutil.copy2\` → \`shutil.copyfile\` in \`FileWorker.copy()\` would resolve this library-side at the cost of not preserving src timestamps on the destination. For archive-storage use cases (\`athena-archive-api\`'s primary consumer) that trade-off is fine — the destination's creation time is the move time, not the temp file's mtime. Happy to PR if that's acceptable. --- Surfaced from <https://code.modernleft.org/ModernLeft/athena-archive-api> debugging.