# Zero-Downtime ECS Service Restarts: A Fully AWS-Native Orchestration Solution
## Introduction
In modern cloud-native architectures, **Amazon ECS (Elastic Container Service)** is a popular choice for running containerized applications at scale. While ECS provides high availability, scalability, and fault tolerance out of the box, there are operational scenarios where **automating ECS service restarts** becomes essential—**without causing any downtime**.
Whether you're dealing with memory bloat, stale connections, periodic resource refresh, or specific application lifecycle needs, you may need to restart services **on a schedule** or in response to **operational triggers**. I recently work on one such use case involves containerized sidecars—like log shippers—that need a controlled restart to function optimally.
**📌 My Real-World Example: Restarting CloudWatch Agent Sidecar Containers**
Consider a scenario where each ECS task runs:
* A **main application container**, and
* A **CloudWatch Agent container** as a **sidecar**, responsible for shipping logs to Amazon CloudWatch.
\*\* The **sidecar** is chosen to avoid or minimize application code changes.
The requirement is to:
* **Rotate log files daily**, so each new file is timestamped.
* The **CloudWatch Agent only generates a new log file** on task start or container restart.
* Hence, a **daily restart** of ECS tasks is necessary—**but without affecting application availability**.
This blog post walks you through an elegant, fully **AWS-native, low-code solution** to:
* Automatically restart ECS services daily (e.g., at **12:01 AM EST**),
* Avoid **application downtime** through rolling deployments,
* And **minimize complexity and cost** using tools like **Amazon EventBridge**, **AWS Lambda**, and **ECS UpdateService API**.
Let’s dive into the design and step-by-step implementation.
---
## **Options Explored**
### **Option 1:** **CloudWatch Agent's Built-in Log Rotation :**
Naturally the best solutions would be the Built-in Log Rotation as it requires **No service restarts.** But in this specific scenario (sidecars) log rotation can’t use dynamic file names with dates unless container is restarted. So this opting is and deliver the expected outcome.
### **Option 2: Manually Rotate Logs in Container :**
This needs custom agent which complicate the setup and deviate the purpose of pre-build sidecar selection for simplicity and low operational overhead.
* **Pros**: Fine-grained control.
* **Cons**: High operational overhead and requires custom code
### **Option 3: Restart Specific Containers via SSM Exec :**
This sounds great initially, considering the advantage that we can target just the CloudWatch agent and no interruption to actual application. But the major drawback is it’s **More Complex Setup**
* **Pros**: More targeted solution with
* **Cons**:
* Requires ECS Exec setup, custom command logic, container introspection
* ✖ **Not Natively Automated**: Unlike ECS deployments, SSM does not have a built-in rolling update mechanism.
* ✖ **Potential Execution Failures**: If the CloudWatch Agent crashes unexpectedly, SSM may fail to restart it.
* ✖ **Potential loss of data:** prone to miss data generate while agent restarting.
### **Option 4: Restart Entire Service via ECS API :**
The key advantage of this approach is, ECS performs a **rolling restart**, ensuring **zero downtime** while forcing CloudWatch Agent to create a new log file with a timestamp. This is simple, can be achieved with native tools: EventBridge Scheduler + Lambda and can be scaled to address complex scenarios if required.
* **Pros**: Best for simplicity, reliability, and scalability.
* **Cons**: A rolling restart causes the creation of new tasks, which momentarily increases resource utilization.
---
## **My Final Choice**
I chose **Option 4**: Trigger an **ECS service restart using UpdateService with forceNewDeployment: true**, orchestrated by **EventBridge Scheduler + Lambda**.
### Why?
* **Fully AWS-native and serverless:** A fully AWS-managed solution with minimal manual intervention.
* **AWS Best Practice**: ECS rolling restarts are the **recommended approach** for long-running tasks.
* **Zero-downtime by design**: Thanks to autoscaling, it ensures that at least 1 container is always available.
* **Supports multiple services :** Simpler setup, avoiding unnecessary IAM permissions, agent & service dependencies.
* **Easy to monitor and extend :** Add CloudWatch Alarms or SNS alerts for failures. Extend Lambda to support dry-run or Slack notifications
* **EventBridge Scheduler** is better than EventBridge Rules because:
* Supports one-time and recurring schedules
* Supports timezones
* Allows per-schedule flexibility without needing multiple rules
* Provides execution logs for better monitoring
* Easier to modify via API/Console
* Visualize with new UI
---
## **High-Level Architecture**
![](https://cdn.hashnode.com/res/hashnode/image/upload/v1743976861107/d4194906-8017-4cc2-adce-5c539e997149.png align="center")
1. **EventBridge Scheduler** triggers Lambda daily at 12:01 AM EST
2. **Lambda Function**:
* Accepts a list of ECS clusters/services as input
* Invokes ECS update\_service API with forceNewDeployment
* Logs success/failure per service
3. **ECS Deployment:**
* Service configured with autoscaling, and rolling deployments at least minimum 1 desired task.
---
## **Implementation Steps**
> For full Terraform project check my Git repo here [ecs-restart-automation-terraform](https://github.com/sthallapelly/ecs-restart-automation-terraform)
### **Step 1: Create IAM Role for Lambda**
* Go to **IAM Console** → Click **Roles** → Click **Create role**.
* Select **AWS Service** → Choose **Lambda** → Click **Next**.
* Attach the following permissions:
* `AmazonECS_FullAccess`
* `AWSLambdaBasicExecutionRole`
* Click **Next** → Name the role: `LambdaECSRestartRole`
* Click **Create role**
or Alternatively Attach the following permissions:
```bash
{
"Effect": "Allow",
"Action": [
"ecs:UpdateService",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
```
### **Step 2: Deploy Lambda Function**
* Go to **AWS Lambda Console** → Click **Create function**
* Select **Author from scratch**
* Name it: `ecs-rolling-restart`
* Runtime: **Python** 3.13
* Select **Execution Role** → Choose **LambdaECSRestartRole** created in step 1.
* Click **Create function**
* In the function editor, replace the default code with:
```python
import boto3, json, logging
from botocore.exceptions import ClientError
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
services = event.get("services", [])
if not services:
logger.warning("No services provided.")
return {"statusCode": 400, "body": json.dumps({"error": "No services provided."})}
ecs = boto3.client("ecs")
results = []
for svc in services:
cluster = svc.get("cluster")
service = svc.get("service")
if not cluster or not service:
results.append({"status": "skipped", "reason": "Missing cluster/service"})
continue
try:
ecs.update_service(cluster=cluster, service=service, forceNewDeployment=True)
results.append({"cluster": cluster, "service": service, "status": "success"})
except Exception as e:
logger.error("exception on %s/%s: %s", cluster, service, e)
results.append({"service": service, "cluster": cluster, "status": "failed", "error": str(e)})
return {"statusCode": 200, "body": json.dumps(results)}
```
* Click **Deploy**
### **Step 3: Create EventBridge Scheduler**
* Navigate to **EventBridge Scheduler**
* Click **Create Schedule**
* Choose **Recurring Schedule**
* Select Time zone
* Set **cron expression**: cron(1 0 \* \* ? \*) for 12:01 AM EST
* Select **Lambda Function** as target and provide Lambda function create in Step 2
* Create new default Execution Role or select if one exist.
* Provide input Payload.
* **Input Example:**
```json
{
"services": [
{ "cluster": "prod-cluster", "service": "orders-service" },
{ "cluster": "prod-cluster", "service": "billing-service" }
]
}
```
---
### **Optional: Test with AWS CLI**
```bash
aws lambda invoke \
--function-name ecs-daily-restart \
--payload file://input.json \
output.json
```
Where input.json contains:
```json
{
"services": [
{"cluster": "prod-cluster", "service": "orders-service"}
]
}
```
---
## **Monitoring and Troubleshooting**
* Check CloudWatch Logs under: /aws/lambda/<your-function-name>
* Add structured logging (logger.info, logger.error)
* Validate ECS task restarts under ECS service -> Events tab
---
## **Final Thoughts**
### **This pattern gives you:**
* Zero-downtime, daily ECS service rolling restarts
* Daily log file rotation via CloudWatch Agent
* Dynamic, multi-service support with a single Lambda
* Fully serverless and scalable design
### **Next Steps**
1. Add CloudWatch Alarms or SNS alerts for failures
2. Extend Lambda to support dry-run or Slack notifications
3. Use Parameter Store or DynamoDB to store service metadata
4. Visualize with EventBridge Scheduler (new UI)
### **Summary**
By combining **Amazon EventBridge Scheduler**, **AWS Lambda**, and **Amazon ECS**, we built a reliable, serverless orchestration for ECS task restarts tailored to log rotation needs. This approach balances low-code simplicity with enterprise-grade flexibility.
---
Thank you for taking the time to read my post! 🙌 If you found it insightful, I’d truly appreciate a like and share to help others benefit as well. 🚀