Startup errors: finding the cause

Find the last stage and complete error: build, SSH, database, migrations, backend, or HTTPS. This determines the next step.

Check logs

Work from the project root: Source on the VPS as the deploy user, Archive/Registry on the controlling computer, and prod-local on your computer.

bash ./deploy/control status --env local --json
bash ./deploy/control logs --env local --service backend
bash ./deploy/control logs --env local --service postgres
bash ./deploy/control logs --env local --service frontend

For a server, replace local with the environment name. With make dev, backend and Vite write to its terminal; these are not Docker services. Remove secrets and password-bearing connection strings before sharing logs.

Build did not complete

DNS or network timeout

lookup ...: i/o timeout means the address of the named service could not be resolved. It does not yet prove a code error, regional blocking, or a particular provider issue.

Check the address from the message where the stage ran: Source builds on the VPS; Archive/Registry builds on your computer or runner. Docker may use DNS/proxy settings different from the terminal. Distinguish registry, Go/npm, and public HTTPS. After resolving the cause, retry build; if an operation ID exists, investigate the operation first. Increasing timeout and disabling checks do not fix DNS.

Deploy stopped

Certbot did not issue a certificate

Certbot failed is the outcome; the cause is in certbot.log, whose path the command prints. Open that exact file on the VPS:

tail -n 80 /ПУТЬ_ИЗ_ОШИБКИ/certbot.log

If the certificate authority received HTTP 403 for /.well-known/acme-challenge/, the request reached the HTTP server, but the challenge file was not served. Compare the logged IP with the VPS and open frontend logs:

bash ./deploy/control logs --env production --service frontend

Permission denied for /var/www/acme/ indicates challenge directory permissions. Current tools prepare it for Nginx access. Do not change permissions across runtime storage: secrets and database data are nearby. If an external proxy blocks the request, fix its rule for the challenge path.

The Nginx warning can not modify ... default.conf (read-only file system?) alone does not mean failure: the configuration check result and next error matter.

After fixing the cause, resume the original operation. Do not start another provision over it.

Old tools after a generator update

configuration must be a regular file with a new UI profile, No such image: sha256:... despite an existing Docker 29 image, and repeated secrets.json upload on resume were fixed in generator tools. First run Generate with the current generator, send project changes to the VPS through Git, and investigate the current operation.

Whether rebuilding is needed depends on the file: deploy/control and the controlling production.sh are read from the checkout; application code and tools inside the ops image need a new build. An updated checkout does not automatically replace saved tools of an operation already started. Do not manually edit image IDs or release JSON.

Backend runs but /readyz = 503

503 means the application is not ready. Check migrator/backend logs and whether migrations match the release. Early 0.8.1 checks looked for old system migration names; the corrected template uses the manifest. This case needs Generate and a new image build, rather than fake schema_migrations records. If first startup did not record current, start with recovery.

HTTPS or sign-in fails

If curl with local --cacert .../pki/ui-ca.crt works but the browser does not, check CA trust, hosts, address, and port from JSON. There is no default login: create the first superadministrator with bootstrap-superadmin. For 401/403, check accounts and permissions in this specific environment; Generator UI sign-in is independent.

Images do not open

Check that files moved with the database and that directory and backend permissions are correct. After changing a local path, bind to the new location after restoring files. Do not change the binding to an empty folder or create marker files manually.

The admin interface needs access to the source persistent entity; virtual access does not replace it. Direct size links from the external API use separate link-access rules and caching — check your actual loading method.

After fixing the cause

Code changes do not update an already running image. After a model/template change, run Generate, then build and deploy. If the previous operation is unfinished, determine its result first. Do not delete the database or .state as a universal fix.

Disk full after several builds

Check df -h / and docker system df on the VPS itself. Images, build cache, and *.images.tar files consume space independently. Detailed commands are in disk cleanup. Do not manually delete /var/lib/docker, /var/lib/containerd, or environment state.

Docker cannot download a base image

Source does not need your own registry, but Docker still downloads base images and dependencies. lookup auth.docker.io ... timeout occurs during DNS/registry access. Check access from the VPS and repeat docker pull for the image named in the error. With IPv6 network is unreachable, check server networking. Do not delete working images or the database to retry a build. Build uses existing service images locally; source=local and source=pull show their origins.

Ops was deleted and control no longer works

Ops remains necessary after release. For Source/Archive, locate the current release archive using state/current.json and load it with docker load -i /полный/путь/к/архиву.images.tar. This restores images in Docker without restarting the application. For Registry, restore the exact image with docker pull using the .ops reference in the release record. Do not build a different image under the old name as a replacement.