Why Your Deployment Script Breaks at 3AM and How to Actually Fix It
I shipped a deployment script that took down production at 3AM on a Tuesday. The pager went off, I rolled out of bed, and spent two hours rolling back changes while half-asleep and fully panicking. The worst part was that the script worked perfectly on my laptop. I had tested it three times in staging. But production decided to teach me a lesson about assumptions, and that lesson cost the company money and cost me sleep I will never get back. If you have been there, you know the feeling. If you have not been there yet, just wait.
The Environment Gap Nobody Talks About
Your deployment script runs in a completely different universe when it hits production servers. The shell version is different. The environment variables you depend on do not exist. The paths you hardcoded point to directories that are not there. The user running the script does not have the permissions you assumed it would have. Production does not care about your laptop environment, and it will punish you for assuming otherwise.
Let me give you a real example. You write a script that uses bash 4.0 features like associative arrays because that is what you have on your development machine. Production is running bash 3.2 because it is some ancient Linux distribution that has not been updated since 2014. Your script does not just fail gracefully. It explodes with cryptic error messages that make zero sense at 3AM. Now you are grepping through stack overflow posts while your boss is asking for an ETA on the fix.
Environment variables are another landmine. You reference DATABASE_URL in your script because it is in your local .env file and it works great. Production does not have that variable set because the previous deployment method used a different naming convention. Your script tries to connect to an undefined database URL and just hangs there waiting for a connection that will never come. You do not find out until the deployment times out fifteen minutes later.
Path assumptions will burn you every single time. You write something like "cd /home/deploy/app" because that is where the app lives on your staging server. Production has a different username or a different directory structure. The cd command fails, but your script keeps running in the wrong directory and starts overwriting files it should not touch. This is how you accidentally delete your application config at 3AM.
The Exit Code Lie
Most deployment scripts are built on a foundation of lies, and the biggest lie is that commands will tell you when they fail. You run a command, it returns exit code 0, and you assume everything is fine. Except that command did not actually do what you think it did. It failed silently, returned success anyway, and now your deployment is in an undefined state that will cause mysterious failures three steps later.
Everyone tells you to use "set -e" at the top of your bash script to exit on errors. That sounds great until you realize that "set -e" has about fifty edge cases where it does not actually work. Pipelines do not trigger it unless you also set pipefail. Commands in conditionals do not trigger it. Background processes do not trigger it. You end up with a false sense of security and a script that keeps running after critical failures.
Here is the pattern you see everywhere: someone runs curl to download a deployment artifact, the curl command times out or gets a 404, but curl returns exit code 0 because it successfully made the request even though it got nothing useful back. Your script proceeds to try to extract a corrupted or empty tar file and everything goes sideways. You need to check curl with the --fail flag and actually validate that what you downloaded is what you expected.
The correct pattern is to check return codes explicitly for anything that matters. After you run a database migration, check that it actually succeeded. After you copy files, verify they exist at the destination. After you restart a service, check that it is actually running. This feels tedious and paranoid, but paranoid is exactly what you need to be at 3AM when you are trying to figure out which step failed.
Idempotency Is Not Optional
You know what makes a bad deployment worse? Running the deployment script twice because you were not sure if it finished the first time. If your script is not idempotent, that second run will break things that were working and create a cascading failure that is even harder to debug. Idempotency is not some theoretical computer science concept. It is the difference between fixing a deployment in five minutes versus spending an hour untangling a mess.
Idempotency means you can run the script multiple times and get the same result. If you create a directory, check if it exists first. If you append to a configuration file, check if that line is already there. If you start a service, check if it is already running. Every operation needs to ask "what if I already did this?" before doing it again.
The classic mistake is something like "mkdir /var/app/config" without the -p flag and without checking if the directory exists. The first run works fine. The second run fails because the directory already exists, and now your script exits with an error even though nothing is actually wrong. Use "mkdir -p" everywhere. Check before you append to files. Use systemctl restart instead of systemctl start so it works whether the service is running or not.
This matters most when deployments fail halfway through. Let's say your script gets to step 7 out of 10 and then hits a network timeout. You fix the network issue and run the script again. If steps 1 through 6 are not idempotent, they are going to fail or create duplicate state. Now you are manually cleaning up before you can retry, and you are doing all of this while production is down. Idempotent scripts let you hit the retry button without thinking.
Logging That Actually Helps You Debug
Your deployment logs are useless. I know they are useless because I have seen a thousand deployment logs that just say "deploying application..." followed by "deployment complete" with nothing in between. When that deployment breaks production, those logs tell you absolutely nothing about what happened. You need logs that answer questions, not logs that make you feel busy.
Log the commands you are about to run before you run them. Log the output of commands that matter. Log timestamps so you can see where the script spent its time. Log the values of critical variables so you know what the script thought it was doing. This is not about creating gigabytes of log spam. This is about having enough information to reconstruct what happened when you are investigating a failure.
Here is what good logging looks like. Before you run a database migration, log "Running migration: apply_schema_v2.sql at 2023-11-15 03:24:18". Run the migration and capture the output. Log whether it succeeded or failed and what the exit code was. Now when you are looking at logs later, you can see exactly which migration ran, when it ran, and what it said. You can trace the timeline of the deployment and find the exact moment things went wrong.
The balance is between verbosity and signal. You do not need to log every single line of a bash script. You need to log decision points, external commands, and anything that touches production state. If you are deploying a web application, log when you stop the old version, when you start the new version, when you run migrations, when you clear caches. Skip logging things like variable assignments unless those variables are critical to the deployment logic.
The Rollback Plan You Hope to Never Use
Perfect deployments are a myth sold by people who do not deploy often enough. Real deployments fail. Servers run out of disk space. Dependencies conflict. Configuration is wrong. Network issues happen. You need a rollback plan, and that plan cannot be "figure it out when it breaks" because you will not figure it out fast enough at 3AM.
Every deployment script should have a rollback strategy built in. Before you overwrite a configuration file, copy the old one to a backup location with a timestamp. Before you deploy a new version of code, tag the current version so you can get back to it. Before you run a destructive database migration, take a snapshot if your database supports it. The goal is to create breadcrumbs you can follow backward when things go wrong.
The trick is knowing when to abort versus when to proceed. Some failures are fatal and you should stop immediately. If your database migration fails, do not proceed to start the new version of the application that depends on that migration. Stop, roll back, and figure out what went wrong. Other failures are warnings. If clearing a cache fails, maybe that is okay and you can proceed. Build your script to distinguish between critical failures and recoverable issues.
Rollback is not just about technology. It is about having the discipline to actually use it. When production is on fire and people are asking for updates, the temptation is to keep trying forward fixes. Sometimes the right answer is to roll back to the last known good state, stop the bleeding, and then figure out the real fix in a calm environment. Your rollback plan is only useful if you are willing to execute it.
Testing Deployment Scripts Without Breaking Production
You cannot fully test deployment scripts without running them in production, and that is a frustrating reality you just have to accept. Staging environments are not production. They do not have the same data volume, the same traffic patterns, the same weird configuration edge cases that have accumulated over years. You can get close, but you cannot get perfect.
That does not mean you should skip testing entirely. Build a dry-run mode into your deployment script that shows what would happen without actually doing it. Use flags like --dry-run or --check to print out the commands that would execute and the state changes that would occur. This catches obvious problems like path errors or missing variables before you run the script for real.
Staging environments are still valuable even if they are not perfect replicas. Deploy to staging first. Run your tests. Let the deployment sit for a few hours or a few days and see if anything weird happens. This catches a lot of issues, just not all of them. The goal is to reduce risk, not eliminate it, because you cannot eliminate it.
The hard truth is that some issues only appear in production. Production has that one server with a slightly different configuration that nobody documented. Production has network latency to external services that staging does not have. Production has data edge cases that your test data does not cover. You will find these issues when you deploy to production, and that is why you need good logging and rollback plans. Testing reduces failure frequency. Logging and rollback reduce failure impact.
Closing: Accepting Imperfection
Deployment scripts will never be perfect, and you need to make peace with that. The goal is not zero failures. The goal is faster recovery when failures happen. The goal is turning a two-hour outage into a ten-minute rollback. The goal is having enough information in your logs to understand what went wrong without having to reproduce it.
Defensive scripting feels like over-engineering when you are writing it. All those checks for existing directories and validation of downloaded files and logging of timestamps seem excessive. Then your script fails at 3AM and every single one of those defensive patterns saves you time and stress. You stop thinking of it as over-engineering and start thinking of it as survival.
This is not a problem you solve once and forget about. Deployment scripts need maintenance. Production environments change. New edge cases appear. Dependencies get updated and break assumptions you made six months ago. You will come back to these scripts over and over, usually when they break, and each time you will add a little more defensive code based on what just failed. That is how it works.
The deployment script that took down my production at 3AM taught me more than any tutorial or best practices document ever did. It taught me to assume everything will go wrong. It taught me to check exit codes explicitly. It taught me to log the things that matter. It taught me that idempotent operations are not optional. I still write deployment scripts that break sometimes, but now they break less often and I fix them faster. That is the best you can hope for.