Mock HTTP Dependencies in Go with Dependency Injection

You will learn why directly calling package-level functions such as http.Get makes tests harder, and how dependency injection makes Go code easy to test. You will use a small HTTP interface and a fake implementation to test parsing logic without network access.

Go does not have built-in runtime monkey patching for ordinary functions. A direct call such as http.Get(url) is fixed when the program is compiled, so replacing it in a test usually requires a special tool and fragile build-time rewriting.

The idiomatic Go solution is dependency injection: make the outside dependency available to your code as a value. In this case, the dependency is the component that performs HTTP requests.

Instead of tying application logic directly to http.Get, accept an object that can make requests. Production code receives a real *http.Client; tests receive a fake client that returns a prepared response.

This separation matters because it keeps tests:

• Fast: no real network calls.
• Deterministic: tests do not depend on a remote server or changing HTML.
• Focused: a downloader test verifies downloader behavior, not the internet.
• Safe: tests can simulate failures, malformed responses, and status codes reliably.

A useful design rule is: keep I/O at the edges of the program, and pass its results or dependencies into the logic that needs them.

Think of downloader as a kitchen that needs water.

If the kitchen is permanently connected to one specific outdoor pipe (http.Get), you cannot test it when the pipe is unavailable. Instead, give the kitchen a faucet connection (an injected dependency). In production, connect it to the city water supply (http.Client). In a test, connect it to a small container of clean, predictable water (a fake client).

The kitchen still knows how to cook; it simply does not need to know where the water comes from.

Define the smallest interface your code needs. http.Client has many methods and fields, but the downloader only needs Get.

type HTTPGetter interface {
	Get(url string) (*http.Response, error)
}

A real *http.Client already has a compatible Get method, so it satisfies this interface automatically:

client := &http.Client{}
var getter HTTPGetter = client

Now make the page-loading function receive that dependency:

func getPage(client HTTPGetter, url string) (string, error) {
	resp, err := client.Get(url)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		return "", err
	}

	return string(body), nil
}

A test can supply a fake:

type fakeClient struct {
	body string
	err  error
}

func (f fakeClient) Get(url string) (*http.Response, error) {
	if f.err != nil {
		return nil, f.err
	}

	return &http.Response{
		StatusCode: http.StatusOK,
		Body:       io.NopCloser(strings.NewReader(f.body)),
	}, nil
}

The fake does not contact a server. It returns exactly the body the test asks for. Notice that its response body is wrapped with io.NopCloser: http.Response.Body must implement both io.Reader and io.Closer.

io.ReadAll is used in modern Go. ioutil.ReadAll is still available for compatibility, but the io package is preferred.

Consider this small test:

func TestGetPageReturnsFakeBody(t *testing.T) {
	client := fakeClient{body: "<html>test page</html>"}

	got, err := getPage(client, "https://example.test/list")
	if err != nil {
		t.Fatalf("getPage returned an error: %v", err)
	}

	want := "<html>test page</html>"
	if got != want {
		t.Fatalf("getPage() = %q, want %q", got, want)
	}
}

Execution flow:

1. The test creates fakeClient with a known HTML string.
2. The test calls getPage, passing the fake as the HTTPGetter argument.
3. getPage calls client.Get(...).
4. Because client is a fakeClient, Go runs fakeClient.Get, not a real HTTP request.
5. The fake returns an in-memory response body.
6. getPage reads and returns that body.
7. The test compares the returned result with the expected string.

No DNS lookup, server, or internet connection is involved.

Dependency injection for HTTP is useful whenever code talks to systems outside the current process:

• API clients: return fixed JSON while testing code that decodes API responses.
• Web scrapers: supply sample HTML pages to test selectors and regular expressions.
• Payment or email integrations: simulate approved, rejected, and failed responses without contacting a provider.
• Background workers: test retry behavior by returning temporary errors from a fake.
• Command-line tools: test download and upload behavior without modifying real remote data.

For a downloader specifically, a test can provide HTML containing known links and then verify that the code identifies the expected links.

In production Go code, prefer clear dependency boundaries over a general-purpose function mocking framework.

Pass a dependency into a constructor

type Downloader struct {
	client HTTPGetter
}

func NewDownloader(client HTTPGetter) *Downloader {
	return &Downloader{client: client}
}

Production setup supplies the real client:

d := NewDownloader(&http.Client{Timeout: 10 * time.Second})

Tests supply a fake client.

Return errors instead of silently returning an empty string

An empty page may be valid, while an HTTP failure is a different situation. Returning (string, error) lets callers decide whether to retry, log, or stop.

Separate fetching from processing

Keep parsing and link extraction in a function that accepts a string or []byte:

func extractMatches(content string, re *regexp.Regexp) [][]string {
	return re.FindAllStringSubmatch(content, -1)
}

This pure function needs no mock at all.

Make concurrent work observable

Starting goroutines inside downloader can make a test finish before serieDL runs. In real projects, inject a handler function, use a sync.WaitGroup, or have the method return work items that another layer executes. This makes behavior testable and avoids timing-dependent tests.

Trying to replace http.Get directly

This is difficult because it is a package-level function, not a variable passed to your code. Prefer injecting a client or transport.

Mocking too much

Avoid creating a large interface that duplicates all of http.Client.

// Too broad for code that only calls Get.
type HugeHTTPClient interface {
	Get(string) (*http.Response, error)
	Post(string, string, io.Reader) (*http.Response, error)
	Do(*http.Request) (*http.Response, error)
	// many more methods...
}

Use the smallest interface required by the consumer.

Returning a response with a nil body

This broken fake panics when the production code calls resp.Body.Close() or reads the body:

func (f fakeClient) Get(url string) (*http.Response, error) {
	return &http.Response{StatusCode: http.StatusOK}, nil
}

Provide a non-nil body:

Body: io.NopCloser(strings.NewReader("test content")),

Ignoring HTTP status codes

A request can succeed at the transport level while returning 404 or 500. Check the status code when your application requires a successful response.

Testing goroutines with time.Sleep

Sleeping makes tests slow and flaky. Coordinate goroutines with channels or sync.WaitGroup instead.

Making real requests in unit tests

A real request creates an integration test, not a focused unit test. It can fail due to unavailable internet, rate limits, server changes, or slow responses.

A function dependency can be especially concise:

type FetchPage func(url string) (string, error)

func downloader(fetch FetchPage) error {
	content, err := fetch("https://example.test")
	if err != nil {
		return err
	}
	_ = content
	return nil
}

Use a small interface when the dependency represents a service with one or more related operations. Use an injected function when there is only one operation.

// 1. Describe only what the consumer needs.
type HTTPGetter interface {
	Get(string) (*http.Response, error)
}

// 2. Receive the dependency.
func getPage(client HTTPGetter, url string) (string, error) {
	resp, err := client.Get(url)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		return "", err
	}
	return string(body), nil
}

// 3. Production dependency.
client := &http.Client{Timeout: 10 * time.Second}

// 4. Fake response body in tests.
resp := &http.Response{
	StatusCode: http.StatusOK,
	Body: io.NopCloser(strings.NewReader("fake response")),
}

• Define interfaces near the code that uses them.
• Keep interfaces small.
• Prefer (value, error) over using a special value such as "" to signal failure.
• Always close a non-nil HTTP response body.
• Use httptest when you need a local server; use a fake interface for focused unit tests.
• Set a timeout on real http.Client instances.

Can Go mock functions directly?

Not idiomatically. Go generally favors passing dependencies as interfaces or functions rather than replacing package-level functions at runtime.

Should I mock http.Get or getPage?

Usually neither directly. Refactor getPage to accept an HTTP dependency, then test it with a fake. Keep parsing separate so it can be tested without HTTP.

Is *http.Client compatible with an interface containing Get?

Yes. *http.Client has Get(url string) (*http.Response, error), so it satisfies an interface with that exact method.

When should I use httptest.NewServer instead of a fake client?

Use httptest.NewServer when you want to verify HTTP details such as paths, headers, request methods, redirects, or server behavior. Use a fake client for a smaller, faster unit test.

Why should getPage return an error?

Returning "" hides the reason for failure and makes a failed request indistinguishable from a successful empty page. An error preserves that information.

How do I test code that launches goroutines?

Avoid relying on delays. Have the goroutines signal completion through a channel or sync.WaitGroup, then wait for that signal in the test.

Is Withmock the recommended solution for modern Go?

No. It relies on rewriting imports and is not the usual Go approach. Refactoring toward dependency injection is clearer, safer, and easier to maintain.

• Interfaces in Go — interfaces let real and fake implementations satisfy the same dependency contract.
• Dependency injection — pass required services into code instead of constructing or calling them directly.
• Table-driven tests — test several fake responses and expected outcomes with one test structure.
• net/http/httptest — create local HTTP servers for HTTP-level integration tests.
• http.RoundTripper — customize request transport when testing http.Client.Do calls.
• Error handling in Go — return and check errors so callers can handle failures intentionally.
• Goroutines and sync.WaitGroup — coordinate concurrent work and write deterministic tests.
• Separation of concerns — isolate fetching, parsing, and processing so each part is easier to test.