Removing duplicates in lists

There are a few ways to remove duplicates from a list in Python. One of the simplest ways is to convert the list to a set, and then back to a list. This works because sets only allow unique elements, so any duplicates will be removed automatically when you convert the list to a set. Here's an example:

Removing duplicates in lists in Python using sets

original_list = [1, 2, 3, 4, 2, 3, 4, 5]
unique_list = list(set(original_list))
print(unique_list)

This will print [1, 2, 3, 4, 5], which is the original list with duplicates removed.

Another approach to remove duplicates from a list is to create an empty list, and then iterate through the original list, adding each element to the new list only if it has not already been added. Here's an example:

Removing duplicates in lists in Python using a for loop and a new empty list

original_list = [1, 2, 3, 4, 2, 3, 4, 5]
unique_list = []
for element in original_list:
    if element not in unique_list:
        unique_list.append(element)
print(unique_list)

This will also print [1, 2, 3, 4, 5].

Another approach is to use the dict.fromkeys() method, which can often make your code more concise:

Removing duplicates in lists in Python using a temporary dictionary

original_list = [1, 2, 3, 4, 2, 3, 4, 5]
unique_list = list(dict.fromkeys(original_list))
print(unique_list)

This code uses the dict.fromkeys() method to construct a dictionary from the original list with all values set to None. Because dictionaries preserve insertion order and only allow unique keys, this will give you the same output, with no duplicates.

It's worth noting that only the set() approach does not preserve the original order of elements. The dict.fromkeys() method already preserves order in Python 3.7+, but if you need to support older versions, you can use the collections.OrderedDict.fromkeys() method:

Removing duplicates in lists in Python using an OrderedDict

from collections import OrderedDict
original_list = [1, 2, 3, 4, 2, 3, 4, 5]
unique_list = list(OrderedDict.fromkeys(original_list))
print(unique_list)

Keep in mind that while the above methods work great for small lists and simple types, if you are dealing with large lists or complex types this may not be the best approach; you might prefer using something like pandas or numpy to handle such cases.