Atomic File Writes

Deep dive · part of Python File Handling

If a program crashes mid-write, the file on disk can be left half-written. The fix is to write to a temporary file in the same directory, fsync, then atomically os.replace it over the real one.

Crash mid-write leaves truncated files—config parsers then fail mysteriously. Atomic write pattern: create temp file in target directory, fsync, os.replace into place so readers see old or new content, never half-written JSON.

Same-directory requirement matters: os.replace cannot atomically move across filesystems; temp must share mount point with final path.

Production code combines this topic with logging, tests, and clear module boundaries so refactors stay safe when requirements grow.

tempfile.mkstemp in target dir avoids cross-device replace failures.

os.fsync flushes kernel buffers before replace on critical data.

os.replace is atomic on POSIX; Windows 10+ overwrites destination atomically.

Read-modify-write still needs file locking for multi-process writers.

pathlib write_text is not atomic—wrap with helper for production configs.

Backup rotation can copy previous file before replace for rollback.

Databases and SQLite have their own transaction semantics—do not double-implement unless storing auxiliary JSON sidecars. For large files, stream to temp then replace; memory-mapped edits are a different pattern.

Windows antivirus may briefly lock files—retry replace with exponential backoff in installers.

Configuration tools should validate JSON on the temp path with json.loads before replace—invalid JSON must never reach the live filename.

Validate JSON on the temp file before os.replace; corrupt partial writes should never become the live config path.

Read the parent tutorial on pythondeck.com for runnable snippets, then reproduce them locally in a virtual environment with pinned dependency versions matching your deployment target.

When pairing with teammates, agree on one idiomatic pattern per concern—mixed styles in one repo slow reviews and invite subtle integration bugs during merges.

Writing directly to final path from long-running processes without temp.

Temp files on /tmp different mount than /var/lib/app config.

Skipping fsync and losing data on power loss despite atomic rename.

Unlinking temp on failure without re-raising, hiding corruption.

Centralize atomic_write_text in one utility module.

Include file version or checksum in written JSON for validation.

Test kill -9 simulation by mocking os.replace failure paths.

Document single-writer assumption or add portalocker for multi-writer.

Re-read the examples below with these ideas in mind; change variable names and inputs to match your own project.

The program below demonstrates atomic write helper. Read the comments on each line, run the code, then change names or values to see how the output shifts.

# Example: Atomic write helper
# Run in the REPL or save as a .py file and execute with python.
import os, tempfile

def atomic_write_text(path, data, encoding="utf-8"):
    d = os.path.dirname(os.path.abspath(path)) or "."
    fd, tmp = tempfile.mkstemp(dir=d, prefix=".tmp_", suffix=".part")
    try:
        with os.fdopen(fd, "w", encoding=encoding) as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, path)
    except Exception:
        os.unlink(tmp)
        raise

atomic_write_text("config.json", '{"ok": true}')

This sample walks through read-modify-write in a small, runnable script. Paste it into the REPL or save it as a .py file before you continue to the next block.

# Example: Read-modify-write
# Run in the REPL or save as a .py file and execute with python.
import json
from pathlib import Path

def bump(path):
    p = Path(path)
    data = json.loads(p.read_text()) if p.exists() else {}
    data["count"] = data.get("count", 0) + 1
    p.write_text(json.dumps(data))

bump("counter.json"); bump("counter.json")
print(Path("counter.json").read_text())

Here is a hands-on illustration of atomic replace. Follow the inline comments first; only then execute the snippet and compare the result with what you expected.

# Write temp file then os.replace for crash safety
import os, tempfile  # stdlib
from pathlib import Path  # paths

def atomic_write(path, data):  # helper
    path = Path(path)  # coerce
    path.parent.mkdir(parents=True, exist_ok=True)  # ensure dir
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=".tmp_")  # temp name
    try:  # write
        with os.fdopen(fd, "w", encoding="utf-8") as fh:  # text mode
            fh.write(data)  # payload
            fh.flush(); os.fsync(fh.fileno())  # push to disk
        os.replace(tmp, path)  # atomic on POSIX/Windows
    except Exception:  # failure
        os.unlink(tmp); raise  # cleanup temp

atomic_write("cfg.json", "{}")  # demo
print(Path("cfg.json").read_text())  # {}

The program below demonstrates json bump. Read the comments on each line, run the code, then change names or values to see how the output shifts.

# Read-modify-write should still use atomic writer in production
import json  # json
from pathlib import Path  # Path
p = Path("counter.json")  # counter file
data = json.loads(p.read_text()) if p.exists() else {}  # load or empty
data["count"] = data.get("count", 0) + 1  # increment
p.write_text(json.dumps(data))  # write (use atomic helper in apps)
print(p.read_text())  # show json

Related deep dives on Python File Handling:

Pathlib Patterns

« back to Python File Handling All tutorials