Remove Duplicate Values from a Go Slice Without Losing First-Seen Order

If the output must follow the input’s order, walk the slice once and append each value the first time you see it. For comparable values, a map makes that check straightforward. If you want sorted output instead, sort the slice and use slices.Compact to remove the now-adjacent duplicates. Choose the output order before choosing the helper: sorting does not preserve first-seen order.

Keep first-seen order with a map

This generic helper accepts comparable element types, including the integers in the example. It records each value in seen and appends only values that have not appeared earlier:

package main

import "fmt"

func uniqueInOrder[T comparable](in []T) []T {
    seen := make(map[T]struct{})
    out := make([]T, 0, len(in))

    for _, value := range in {
        if _, exists := seen[value]; exists {
            continue
        }
        seen[value] = struct{}{}
        out = append(out, value)
    }
    return out
}

func main() {
    values := []int{8, 7, 5, 8, 0, 5, 2}
    fmt.Println(uniqueInOrder(values))
}

Expected output:

[8 7 5 0 2]

The second 8 and 5 are skipped; every remaining value appears where its first occurrence placed it. The result is a separate slice, while the map holds the values already encountered. Giving the result capacity equal to the input length leaves room for the case where nothing is duplicated; its length still starts at zero so append builds only the selected values. This is the map-backed pattern shown in the ordered deduplication example.

Supply an equality rule for non-comparable values

A struct containing a slice cannot serve as the key type for the helper above. When such a value needs custom equality, compare each input value with the values already selected. Here, two records are duplicates when their labels and every element of their Codes slices match:

package main

import "fmt"

type Record struct {
    Label string
    Codes []int
}

func sameRecord(a, b Record) bool {
    if a.Label != b.Label || len(a.Codes) != len(b.Codes) {
        return false
    }
    for i := range a.Codes {
        if a.Codes[i] != b.Codes[i] {
            return false
        }
    }
    return true
}

func uniqueBy[T any](in []T, equal func(T, T) bool) []T {
    out := make([]T, 0, len(in))
    for _, value := range in {
        found := false
        for _, kept := range out {
            if equal(value, kept) {
                found = true
                break
            }
        }
        if !found {
            out = append(out, value)
        }
    }
    return out
}

func main() {
    records := []Record{
        {"api", []int{2, 1}},
        {"cli", []int{3}},
        {"api", []int{2, 1}},
        {"api", []int{1, 2}},
    }
    fmt.Println(uniqueBy(records, sameRecord))
}

Expected output:

[{api [2 1]} {cli [3]} {api [1 2]}]

The third record matches the first under sameRecord, so it is omitted. The fourth has the same label but different code order and remains. Because each candidate may need checking against many previously kept values, this approach can require many comparisons as the result grows. Its advantage is that the caller defines what counts as a duplicate without rearranging the input order. The standard library’s slice API makes a similar type distinction: Compact takes comparable elements, while CompactFunc accepts an equality function for any element type. Neither changes the fact that compacting alone addresses consecutive duplicates.

Sort, then compact, when sorted output is the goal

For ordered values, sorting puts equal values next to one another. slices.Compact can then remove each consecutive repeat:

package main

import (
    "fmt"
    "slices"
)

func main() {
    values := []int{4, 2, 4, 1, 2, 3}
    slices.Sort(values)
    values = slices.Compact(values)
    fmt.Println(values)
}

Expected output:

[1 2 3 4]

Assign the return value of Compact: it is the slice containing the retained prefix. Sorting rearranges values, and the output no longer follows the input’s first-seen order (4 came first). Calling Compact on the unsorted input would not remove duplicates separated by other values; sorting is what groups them. The sort-and-compact example illustrates the same distinction.

Use the map pass when first-seen order matters and the elements are comparable. Use the equality-function pass when your definition of a duplicate needs a custom comparison, accepting its potentially large number of checks. If sorted output is useful, sorting followed by compacting avoids a seen-value map, but deliberately trades away the original order.