What is making this Python code so slow? How can I modify it to run faster?

I am writing a program in Python for a data analytics project involving advertisement performance data matched to advertisement characteristics aimed at identifying high performing groups of ads that share n similar characteristics. The dataset I am using has individual ads as rows, and characteristic, summary, and performance data as columns. Below is my current code - the actual dataset I am using has 51 columns, 4 are excluded, so it is running with 47 C 4, or 178365 iterations in the outer loop.

Currently, this code takes ~2 hours to execute. I know that nested for loops can be the source of such a problem, but I do not know why it is taking so long to run, and am not sure how I can modify the inner/outer for loops to improve performance. Any feedback on either of these topics would be greatly appreciated.

import itertools
import pandas as pd
import numpy as np

# Identify Clusters of Rows (Ads) that have a KPI value above a certain threshold
def set_groups(df, n):
    """This function takes a dataframe and a number n, and returns a list of lists. Each list is a group of n columns.
    The list of lists will hold all size n combinations of the columns in the dataframe.
    """
    # Create a list of all relevant column names
    columns = list(df.columns[4:]) # exclude first 4 summary columns
    # Create a list of lists, where each list is a group of n columns
    groups = []
    vals_lst = list(map(list, itertools.product([True, False], repeat=n))) # Create a list of all possible combinations of 0s and 1s
    for comb in itertools.combinations(columns, n): # itertools.combinations returns a list of tuples
        groups.append([comb, vals_lst])
    groups = np.array(groups,dtype=object)
    return groups  # len(groups) = len(columns(df)) choose n

def identify_clusters(df, KPI, KPI_threshhold, max_size, min_size, groups):
    """
    This function takes in a dataframe, a KPI, a threshhold value, a max and min size, and a list of lists of groupings.
    The function will identify groups of rows in the dataframe that have the same values for each column in each list of groupings.
    The function will return a list of lists with each list of groups, the values list, and the ad_ids in the cluster.
    """
    # Create a dictionary to hold the results
    output = []
    # Iterate through each list of groups
    for group in groups:
        for vals_lst in group[1]:  # for each pair of groups and associated value matrices
            # Create a temporary dataframe to hold the group of rows with matching values for columns in group
            temp_df = df
            for i in range(len(group[0])):
                temp_df = temp_df[(temp_df[group[0][i]] == vals_lst[i])]  # reduce the temp_df to only rows that match the values in vals_lst for each combination of values
            if temp_df[KPI].mean() > KPI_threshhold:  # if the mean of the KPI for the temp_df is above the threshhold
                output.append([group, vals_lst, temp_df['ad_id'].values])  # append the group, vals_lst, and ad_ids to the output list
    print(output)
    return output

## Main
df = pd.read_excel('data.xlsx', sheet_name='name')
groups = set_groups(df, 4)
print(len(groups))
identify_clusters(df, 'KPI_var', 0.0015, 6, 4, groups)

Any insight into why the code is taking such a long time to run, and/or any advice on improving the performance of this code would be extremely helpful.

source https://stackoverflow.com/questions/74158253/what-is-making-this-python-code-so-slow-how-can-i-modify-it-to-run-faster