Understanding Circuit Breaker Patterns
Before diving into implementation details, it’s crucial to understand the core concepts behind circuit breakers and how they fit into a broader resilience strategy.
The Circuit Breaker State Machine
At its core, a circuit breaker is a state machine with three distinct states:
package circuitbreaker
// State represents the current state of the circuit breaker
type State int
const (
// Closed means the circuit breaker is allowing requests through
Closed State = iota
// Open means the circuit breaker is preventing requests from going through
Open
// HalfOpen means the circuit breaker is allowing a limited number of test requests
HalfOpen
)
// CircuitBreaker represents a basic circuit breaker
type CircuitBreaker struct {
state State
failureCount int
failureThreshold int
resetTimeout time.Duration
lastFailureTime time.Time
mutex sync.RWMutex
}
The circuit breaker operates as follows:
- Closed State: The default state where requests flow normally. The circuit breaker monitors for failures.
- Open State: When failures exceed a threshold, the circuit “trips” to the open state, fast-failing requests without attempting to execute them.
- Half-Open State: After a timeout period, the circuit transitions to half-open, allowing a limited number of test requests to check if the underlying system has recovered.
Failure Detection Strategies
Effective circuit breakers must accurately detect failures, which can be more nuanced than simple error returns:
package circuitbreaker
// FailureDetector determines if a response should be considered a failure
type FailureDetector interface {
IsFailure(err error, response interface{}, duration time.Duration) bool
}
// StandardFailureDetector considers any error as a failure
type StandardFailureDetector struct{}
func (d *StandardFailureDetector) IsFailure(err error, _ interface{}, _ time.Duration) bool {
return err != nil
}
// AdvancedFailureDetector considers errors, status codes, and timeouts
type AdvancedFailureDetector struct {
TimeoutThreshold time.Duration
ErrorCodes map[int]bool // HTTP status codes considered failures
}
func (d *AdvancedFailureDetector) IsFailure(err error, response interface{}, duration time.Duration) bool {
// Check for timeout
if duration > d.TimeoutThreshold {
return true
}
// Check for error
if err != nil {
return true
}
// Check for HTTP status code if response is an HTTP response
if httpResp, ok := response.(*http.Response); ok {
if d.ErrorCodes[httpResp.StatusCode] {
return true
}
}
return false
}
Circuit Breaker in the Resilience Stack
Circuit breakers don’t operate in isolation but form part of a comprehensive resilience strategy:
package resilience
// ResilienceStack represents a complete resilience strategy
type ResilienceStack struct {
// Timeout prevents requests from hanging indefinitely
Timeout time.Duration
// Retry attempts to recover from transient failures
RetryStrategy RetryStrategy
// CircuitBreaker prevents overwhelming failing services
CircuitBreaker *circuitbreaker.CircuitBreaker
// Bulkhead limits concurrent requests
Bulkhead *bulkhead.Bulkhead
// Fallback provides alternative response when all else fails
Fallback FallbackFunc
}
// Execute runs a function with the complete resilience stack
func (s *ResilienceStack) Execute(ctx context.Context, operation func(context.Context) (interface{}, error)) (interface{}, error) {
// Apply timeout
timeoutCtx, cancel := context.WithTimeout(ctx, s.Timeout)
defer cancel()
// Check circuit breaker
if !s.CircuitBreaker.AllowRequest() {
return s.Fallback(ctx, circuitbreaker.ErrCircuitOpen)
}
// Apply bulkhead
if !s.Bulkhead.Acquire() {
return s.Fallback(ctx, bulkhead.ErrBulkheadFull)
}
defer s.Bulkhead.Release()
// Execute with retry
result, err := s.RetryStrategy.Execute(timeoutCtx, operation)
// Record result in circuit breaker
s.CircuitBreaker.RecordResult(err == nil)
// Apply fallback if needed
if err != nil {
return s.Fallback(ctx, err)
}
return result, nil
}
This layered approach provides defense in depth, with each mechanism addressing different failure modes.