Data and Software Engineer with 8 years of technical experience in data-oriented projects. I specialize in establishing a DataOps culture through CI/CD pipelines, deployment automation, and Infrastructure as Code (IaC).
Beyond my core focus on data platforms, I am highly interested and actively engaged in Software Engineering (building robust backend services and platform tools) and Data Science (leveraging analytical workflows and structured modeling to solve complex problems).
I am a DevOps enthusiast who genuinely loves building efficient, automated, and self-healing systems. I enjoy creating seamless developer experiences, designing scalable CI/CD pipelines, and treating infrastructure-as-code as a core engineering discipline.
Currently, I work as a Senior Data Engineer (DataOps Focus), building scalable, robust, and secure data platforms across multi-cloud environments (AWS and GCP).
- Languages: Go, Python, SQL, Shell Scripting
- DataOps & IaC: Terraform, CloudFormation, Helm, CI/CD (GitHub Actions, ArgoCD), Docker, Kubernetes (EKS, GKE)
- Orchestration & Processing: Apache Airflow, dbt, Apache Kafka, Debezium, Google Cloud Dataflow
- Cloud Providers: Amazon Web Services (AWS), Google Cloud Platform (GCP)
- Databases & Data Warehouses: BigQuery, PostgreSQL, S3/Cloud Storage (Lakehouse Architectures)
- Observability & Quality: Prometheus, Grafana, Cloud Monitoring, Soda, OpenTelemetry
- DevOps & DataOps Automation: Implementing robust CI/CD pipelines (GitHub Actions, ArgoCD), containerized environments (Docker, Kubernetes), and managing infrastructure as code (Terraform) to automate and scale development and deployment workflows.
- Scalable Go APIs & Clean Architecture: Designing high-performance RESTful and gRPC APIs in Go using Hexagonal Architecture to fully decouple core domain logic from infrastructure, databases, and transport layers.
- Robust Testing Strategies: Implementing comprehensive Unit Tests for core domains alongside Integration Tests using Docker/Testcontainers to validate API endpoints and real database interactions.
- High-Scale Ingestion Systems: Architecting distributed and resilient pipelines for real-time (streaming) and batch financial operations, integrating external data sources, transactional databases, and third-party APIs.
- Platform Abstractions: Developing internal tools, microservices, and modular components in Go and Python to simplify the deployment, monitoring, and orchestration of API workflows on Kubernetes.
- Data Lakehouses & Governance: Structuring analytical foundations in BigQuery, securing API credentials, and integrating modern solutions for governance and secret management (HashiCorp Vault).
- LinkedIn: linkedin.com/in/ribeiro-rafael-junior