A comprehensive collection of DevOps and Site Reliability Engineering (SRE) scripts for monitoring, logging, backup, and deployment automation across AWS, Azure, GCP, and Kubernetes.
- Multi-cloud Support: AWS, Azure, GCP
- Container Orchestration: Kubernetes, Docker, ECS
- Zero-downtime Deployments: Blue-green deployment strategies
- Comprehensive Monitoring: Health checks, uptime monitoring, service monitoring
- Log Analysis: Parse, filter, and analyze logs from multiple sources
- Disaster Recovery: Automated backup and recovery procedures
- CI/CD Integration: Pipeline helpers and build automation
# Clone this toolbox to your preferred location
git clone <repository-url> devops-toolbox
cd devops-toolbox
# Make scripts executable
chmod +x monitoring/*.sh monitoring/*.py
chmod +x logs/*.sh logs/*.py
chmod +x backup/*.sh backup/*.py
chmod +x deployment/*.sh deployment/*.pyPython Requirements:
pip install requests # For HTTP health checksGo Requirements (for uptime monitor):
cd monitoring
go build uptime_monitor.goSystem Requirements:
- bash 4.0+
- Python 3.7+
- Docker (for container operations)
- kubectl (for Kubernetes operations)
- AWS CLI (for AWS operations)
- Azure CLI (for Azure operations)
- gcloud/gsutil (for GCP operations)
# Check HTTP endpoint
./monitoring/health_check.py --http https://example.com
# Check SSL certificate
./monitoring/health_check.py --ssl example.com
# Check TCP port
./monitoring/health_check.py --tcp example.com:443# Monitor Docker containers
./monitoring/service_monitor.sh docker web-app db-container
# Monitor Kubernetes deployments
./monitoring/service_monitor.sh k8s production/my-app# Parse and filter logs
./logs/log_analyzer.py /var/log/app.log --level ERROR --grep "timeout"
# Tail multiple sources
./logs/multitail.sh /var/log/app.log docker:web k8s:default/api# Backup PostgreSQL to S3
./backup/db_backup.sh --s3 my-bucket backup postgres mydb# Deploy to Kubernetes
./deployment/deploy.sh k8s deployment.yaml production
# Blue-green deployment
./deployment/blue_green.sh deploy k8s myapp deployment.yaml productionMulti-purpose health check script supporting HTTP, TCP, and SSL certificate checks.
Features:
- HTTP/HTTPS endpoint monitoring
- TCP port connectivity checks
- SSL certificate expiration monitoring
- Batch mode with JSON configuration
- JSON output for integration with monitoring systems
Usage:
# Single HTTP check
./monitoring/health_check.py --http https://api.example.com
# Multiple checks with batch file
cat > checks.json << EOF
{
"checks": [
{"type": "http", "url": "https://api.example.com"},
{"type": "tcp", "target": "db.example.com:5432"},
{"type": "ssl", "hostname": "example.com"}
]
}
EOF
./monitoring/health_check.py --batch checks.json --jsonMonitor systemd services, Docker containers, Kubernetes deployments, and processes.
Features:
- State tracking with notifications
- Webhook and Slack integration
- Continuous monitoring mode
- Support for multiple service types
Usage:
# Monitor systemd services
./monitoring/service_monitor.sh systemd nginx postgresql
# Monitor with Slack notifications
export SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
./monitoring/service_monitor.sh -c 60 docker web-app
# Monitor Kubernetes deployments
./monitoring/service_monitor.sh k8s production/api production/workerConcurrent uptime monitor with statistics tracking.
Features:
- Concurrent monitoring of multiple targets
- Configurable check intervals
- Uptime statistics and reporting
- Response time tracking
Usage:
# Build
go build monitoring/uptime_monitor.go
# Monitor single URL
./uptime_monitor -url https://example.com -interval 60
# Monitor with config file
cat > monitor-config.json << EOF
{
"targets": [
{"name": "API", "url": "https://api.example.com", "interval": 30},
{"name": "Web", "url": "https://example.com", "interval": 60}
]
}
EOF
./uptime_monitor -config monitor-config.jsonParse and analyze logs from various formats with filtering and statistics.
Features:
- Multiple log format support (nginx, apache, syslog, JSON, etc.)
- Pattern filtering and searching
- Time-range filtering
- Statistics generation
- JSON output
Usage:
# Show errors from last 24 hours
./logs/log_analyzer.py /var/log/app.log --level ERROR --since 24
# Filter by pattern with regex
./logs/log_analyzer.py /var/log/app.log --grep "timeout|error" --regex
# Generate statistics
./logs/log_analyzer.py /var/log/app.log --stats
# JSON output for integration
./logs/log_analyzer.py /var/log/app.log --level ERROR --jsonRotate, compress, and manage log files with cloud backup support.
Features:
- Size and age-based rotation
- Compression with pigz support
- Cloud backup (S3, Azure)
- Configurable retention
Usage:
# Rotate if size exceeds 100MB
./logs/log_rotator.sh -s 100 check-size /var/log/app.log
# Rotate with S3 backup
./logs/log_rotator.sh --s3 my-bucket rotate /var/log/app.log
# Clean old logs
./logs/log_rotator.sh clean /var/log
# Continuous monitoring
./logs/log_rotator.sh monitor /var/log/app.logTail multiple log sources with colored output and filtering.
Features:
- Multiple file sources
- Docker container logs
- Kubernetes pod logs
- systemd journal logs
- Pattern filtering and highlighting
Usage:
# Tail multiple files
./logs/multitail.sh /var/log/app.log /var/log/error.log
# Tail Docker containers
./logs/multitail.sh docker:web docker:api docker:db
# Tail Kubernetes pods
./logs/multitail.sh k8s:production/api k8s:production/worker
# Mix sources with filtering
./logs/multitail.sh -f "ERROR|WARN" -H "ERROR" \
/var/log/app.log docker:web k8s:default/apiMulti-cloud backup tool with encryption and verification.
Features:
- Archive creation with compression
- SHA256 checksum verification
- Multi-cloud upload (S3, Azure, GCS)
- Exclude patterns
- Dry-run mode
Usage:
# Backup to S3
./backup/cloud_backup.py /var/www /etc/nginx \
--s3-bucket my-backups \
--exclude "*.log" "*.tmp"
# Backup to multiple clouds
./backup/cloud_backup.py /data \
--s3-bucket aws-backups \
--azure-container azure-backups \
--gcp-bucket gcp-backups
# Dry run
./backup/cloud_backup.py /data --s3-bucket my-backups --dry-runBackup PostgreSQL, MySQL, MongoDB, and Redis databases.
Features:
- Multiple database support
- Compression and encryption
- Cloud upload (S3, Azure)
- Automated cleanup
Usage:
# Backup PostgreSQL
export PGPASSWORD="secret"
./backup/db_backup.sh backup postgres mydb localhost 5432 postgres
# Backup MySQL with S3 upload
export MYSQL_PASSWORD="secret"
./backup/db_backup.sh --s3 my-bucket backup mysql mydb
# Backup MongoDB
./backup/db_backup.sh backup mongodb mydb localhost 27017
# Cleanup old backups
./backup/db_backup.sh cleanup 30Restore backups and test recovery procedures.
Features:
- Multi-cloud download
- Checksum verification
- Database restoration
- Recovery testing
- Recovery plan generation
Usage:
# Download backup from S3
./backup/disaster_recovery.sh download \
s3://my-bucket/backups/db.sql.gz /tmp/db.sql.gz
# Verify backup integrity
./backup/disaster_recovery.sh verify /tmp/db.sql.gz
# Restore database
./backup/disaster_recovery.sh decompress /tmp/db.sql.gz
./backup/disaster_recovery.sh restore postgres /tmp/db.sql mydb
# Test recovery procedure
./backup/disaster_recovery.sh test /var/backups/db.sql.gz
# Show recovery plan
./backup/disaster_recovery.sh planDeploy applications to Kubernetes, Docker, AWS ECS, and systemd.
Features:
- Multi-platform deployment
- Health checks
- Webhook notifications
- Deployment logging
Usage:
# Deploy to Kubernetes
./deployment/deploy.sh k8s deployment.yaml production
# Deploy Docker container
./deployment/deploy.sh docker myapp:latest myapp-container 8080:80 .env
# Deploy to AWS ECS
./deployment/deploy.sh ecs production myapp task-definition.json
# Deploy systemd service
./deployment/deploy.sh systemd myapp /etc/systemd/system/myapp.service
# With Slack notifications
./deployment/deploy.sh --slack "$SLACK_WEBHOOK" k8s deployment.yaml prodZero-downtime blue-green deployments.
Features:
- Kubernetes support
- Docker support
- AWS ECS support
- Automatic rollback capability
- Smoke testing
Usage:
# Deploy with blue-green strategy
./deployment/blue_green.sh deploy k8s myapp deployment.yaml production
# Rollback deployment
./deployment/blue_green.sh rollback k8s myapp production
# Check status
./deployment/blue_green.sh status
# Generate nginx config for Docker
./deployment/blue_green.sh nginx-config myapp 8080 9080 > nginx.confQuick rollback for failed deployments.
Features:
- Platform-specific rollback
- Deployment history tracking
- Automatic state management
Usage:
# Rollback Kubernetes deployment
./deployment/rollback.sh k8s myapp production
# Rollback to specific revision
./deployment/rollback.sh k8s myapp production 5
# Rollback Docker container
./deployment/rollback.sh docker myapp-container myapp:v1.0
# Rollback AWS ECS
./deployment/rollback.sh ecs production myapp
# View deployment history
./deployment/rollback.sh history
./deployment/rollback.sh history k8s myappCI/CD pipeline utilities for build info, artifacts, and notifications.
Features:
- Git information extraction
- Build info generation
- Checksum calculation
- Release notes generation
- Docker image tagging
- URL health checks
- Webhook notifications
Usage:
# Generate build info
./deployment/ci_helper.py build-info -o build-info.json
# Calculate checksums
./deployment/ci_helper.py checksums dist/*.tar.gz -o checksums.json
# Generate release notes
./deployment/ci_helper.py release-notes v1.0.0 HEAD -o RELEASE_NOTES.md
# Tag and push Docker image
./deployment/ci_helper.py docker-tag myapp:latest latest v1.2.3
./deployment/ci_helper.py docker-push myapp latest v1.2.3
# Wait for deployment
./deployment/ci_helper.py wait-for-url https://example.com/health -t 300
# Send notification
./deployment/ci_helper.py notify "$WEBHOOK_URL" "Deployment successful" -s success#!/bin/bash
set -euo pipefail
# Generate build info
./deployment/ci_helper.py build-info
# Run tests (your test commands here)
npm test
# Build Docker image
docker build -t myapp:latest .
# Tag image
GIT_SHA=$(git rev-parse --short HEAD)
./deployment/ci_helper.py docker-tag myapp:latest latest "$GIT_SHA"
# Push to registry
./deployment/ci_helper.py docker-push myapp latest "$GIT_SHA"
# Deploy with blue-green strategy
./deployment/blue_green.sh deploy k8s myapp k8s/deployment.yaml production
# Wait for health check
./deployment/ci_helper.py wait-for-url https://myapp.com/health -t 300
# Send success notification
./deployment/ci_helper.py notify "$SLACK_WEBHOOK" \
"Deployment of myapp@$GIT_SHA successful" -s success#!/bin/bash
set -euo pipefail
# Backup databases
./backup/db_backup.sh --s3 my-backups backup postgres prod_db
./backup/db_backup.sh --s3 my-backups backup redis
# Backup application files
./backup/cloud_backup.py /var/www /etc/nginx \
--s3-bucket my-backups \
--exclude "*.log" "cache/*"
# Test recovery procedure
./backup/disaster_recovery.sh test /var/backups/postgres-prod_db-*.sql.gz
# Cleanup old backups
./backup/db_backup.sh cleanup 7#!/bin/bash
# Create monitoring config
cat > monitors.json << EOF
{
"checks": [
{"type": "http", "url": "https://api.example.com/health", "expected_status": 200},
{"type": "http", "url": "https://example.com", "expected_status": 200},
{"type": "tcp", "target": "db.example.com:5432"},
{"type": "ssl", "hostname": "example.com"}
]
}
EOF
# Run health checks
./monitoring/health_check.py --batch monitors.json --json > health-report.json
# Monitor services continuously
./monitoring/service_monitor.sh -c 60 \
--slack "$SLACK_WEBHOOK" \
docker web api worker- bash 4.0+
- Python 3.7+
- curl
- jq (for JSON processing)
- AWS CLI (for AWS operations)
- Azure CLI (for Azure operations)
- gcloud/gsutil (for GCP operations)
- Docker (for container operations)
- docker-compose (optional)
- kubectl (for Kubernetes operations)
- pg_dump/psql (for PostgreSQL)
- mysqldump/mysql (for MySQL)
- mongodump/mongorestore (for MongoDB)
- redis-cli (for Redis)
Most scripts support configuration via environment variables:
# Monitoring
export SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK"
export NOTIFICATION_WEBHOOK="https://your-webhook.com/notify"
# Backup
export S3_BUCKET="my-backups"
export AZURE_CONTAINER="backups"
export BACKUP_DIR="/var/backups"
export KEEP_DAYS="30"
# Deployment
export DEPLOYMENT_DIR="/opt/deployments"
export LOG_FILE="/var/log/deployments.log"
# Database credentials
export PGPASSWORD="postgres-password"
export MYSQL_PASSWORD="mysql-password"
export MONGODB_PASSWORD="mongo-password"
export REDIS_PASSWORD="redis-password"Many scripts support JSON configuration files for batch operations and complex setups.
Contributions are welcome! Please feel free to submit pull requests or open issues for bugs and feature requests.
This toolbox is provided as-is for DevOps and SRE work. Customize and extend as needed for your infrastructure.
For issues, questions, or contributions:
- Check the script's
--helpoutput for detailed usage - Review the examples in this README
- Open an issue in the repository
Built with ❤️ for DevOps and SRE teams