Skip to main content

Simplifying/collapsing blocks of df.iterrows() where another row is selected based on a value in the current row

I need help :) I've got a python script in which a snippet that repeats df.iterrows() a couple of times takes a good half hour to run. I've seen a lot of examples using lambda functions but this is where apparently I meet my limitations. It's too complex for me to figure out I guess.

Current code (simplified snippet):

            for index, row in df.iterrows():
                if row['series'] == -1 and row['sort'] == 1 and row['amount_t-2'] != 0:
                    index_match = (df[(df['sort']  == 4) & (df['year'] == (row['year']-1)) & (df['amount_t-1'] == (row['amount_t-2'])) & (df['cola'] == (row['cola'])) & (df_int['colb'] == (row['colb']))].index.tolist())
                    if len(index_match) == 1:
                        df.loc[index, 'series'] = df.loc[index_match[0],'series']
                        df.loc[index, 'colc'] = 'done'
                        df.loc[index, 'cold'] = ''

This snippet is repeated a number of times for different configurations/values. What's done here is looping through the entire dataset, creating a time series by setting the 'series' column to the same (index) number if certain conditions are met (i.e. if the values are linked)

Little background: the dataset consists of budgetary tables in which each row represents the state of a budget article for a certain budget phase/time (sort field: 1=budget, 2=first supplement, 3=second supplement, 4=actuals). In order to be able to trace a certain budget article through time, I want to link all relevant rows through a common value for the field 'series'.

When searching for a row to link to the current row, I filter the df by: 'previous phase' (sort-1 or in this case 4 if sort=1), 'year of previous phase' (same year for sort=2,3,4, previous if sort=1), matching value/amounts and matching article name etc.

So far, I've been meddling with something like the following code, but cannot get it to work. Am I at least close?

df['series'] = df.apply(lambda x: df.query("sort == 4 and year == @x['year']-1 and `amount_t-1` == @x['amount_t-2'] and cola == @x['cola'] and colb == @x['colb']").index[0] if x['series'] == -1 and x['sort'] == 1 and x['amount_t-2'] != 0, axis=1)

Thanks in advance,

Chris



source https://stackoverflow.com/questions/71071125/simplifying-collapsing-blocks-of-df-iterrows-where-another-row-is-selected-bas
Labels:

Comments

Post a Comment

Popular posts from this blog

Why is my reports service not connecting?

I am trying to pull some data from a Postgres database using Node.js and node-postures but I can't figure out why my service isn't connecting. my routes/index.js file: const express = require('express'); const router = express.Router(); const ordersCountController = require('../controllers/ordersCountController'); const ordersController = require('../controllers/ordersController'); const weeklyReportsController = require('../controllers/weeklyReportsController'); router.get('/orders_count', ordersCountController); router.get('/orders', ordersController); router.get('/weekly_reports', weeklyReportsController); module.exports = router; My controllers/weeklyReportsController.js file: const weeklyReportsService = require('../services/weeklyReportsService'); const weeklyReportsController = async (req, res) => { try { const data = await weeklyReportsService; res.json({data}) console...
Post a Comment
Read more

How to show number of registered users in Laravel based on usertype?

i'm trying to display data from the database in the admin dashboard i used this: <?php use Illuminate\Support\Facades\DB; $users = DB::table('users')->count(); echo $users; ?> and i have successfully get the correct data from the database but what if i want to display a specific data for example in this user table there is "usertype" that specify if the user is normal user or admin i want to user the same code above but to display a specific usertype i tried this: <?php use Illuminate\Support\Facades\DB; $users = DB::table('users')->count()->WHERE usertype =admin; echo $users; ?> but it didn't work, what am i doing wrong? source https://stackoverflow.com/questions/68199726/how-to-show-number-of-registered-users-in-laravel-based-on-usertype
Post a Comment
Read more

The number of nodes and edges I calculated manually and by a method provided by networkx library dont match

I am trying to analyse a graph using networkx library in Python. I have implemented a function that returns the nodes and edges of that graph. After that I counted the number of nodes and edges by implementing the len() function for nodes and edges, which gave me 17 for both nodes and edges. However, when I used the method graph.number_of_nodes() of networkx library it gave me 53. My code: import pandas as pd import networkx as nx import matplotlib.pyplot as plt header_list = ["a","b","w"] E =pd.read_csv("aves-barn-swallow-contact-network.edges",sep=" ",header= None,names=header_list) # print(E.head()) # prints first few rows of the csv file G = nx.from_pandas_edgelist(E,"a","b",["w"]) nx.draw(G) plt.show() class GraphAnalyse: #finding nodes and edges def nodeEdge(self,Graph): nodes = Graph.nodes() edges = Graph.edges() ...
Post a Comment
Read more