1,150 questions
0
votes
0
answers
150
views
How to efficiently get the last row of a rolling aggregation group without .last()?
I'm working with a large Polars LazyFrame and computing rolling aggregations grouped by customer (Cusid). I need to find the "front" of the rolling window (last Tts_date) for each group to ...
- 37
3
votes
1
answer
66
views
Unexpected behaviour of some Polars rolling functions when NaN's and Nulls are together
I recently came across some behaviour in the way that some of the Polars rolling functions work that I don't understand. The problem seems to only present itself when there is a NaN (np.nan) as well ...
- 95
0
votes
0
answers
63
views
In Algotrading, How to Incrementally Calculate Features for New Live Candles, Ensuring Full-Backtest Consistency (Pandas/TA/ML)
I'm developing a live trading bot in Python that fetches OHLCV data (e.g., 15m candles) and computes a large number of features—rolling indicators (VWAP/Volume-ADI,SMA/EMA/ATR/RSI), price action, ...
0
votes
1
answer
49
views
Why is this pandas rolling function exhibiting nonlinear slowdown when run on the cloud?
I moved a library for stock market analysis to the cloud and found strange performance with a key pandas function. It runs reasonably quickly on my low-performance Dell XPS laptop and exhibits ...
- 1,044
2
votes
0
answers
68
views
Why does bottleneck move_mean give different answers for different size arrays but using the same values
This may be similar to bottleneck.move_mean get different result for the same ndarray, but that 'question' was ill-posed and so attracted no answers.
I have found that for a single array, bottleneck ...
- 67
0
votes
2
answers
75
views
Multiple scrolling text in a label
I need help with this code, My purpose is to show multiple Message one after one, this one is working fine but I need several messages to be shown one after one,
I tryied different Method but allways ...
- 13
3
votes
2
answers
179
views
Rolling Time Window Average in Polars with Left-Edge Expansion
The goal is to compute a rolling average over 5-minute windows using Polars, where:
the window ends at each timestamp t
the left edge of the window is not strict — if there are no values exactly at t ...
- 31
3
votes
2
answers
81
views
How to filter sequential event data according to whether record is followed by specific event within X minutes?
I have some data with a timestamp column t, an event category column cat, and a user_id column. cat can take n values, including value A.
I want to select records which are followed (not necessarily ...
- 31
0
votes
2
answers
127
views
Function that calculates rolling MAD while ignoring outlier values [closed]
I am trying to write a function to compute the rolling Median Absolute Deviation (MAD) for outlier detection in a time series dataset. The goal is to:
Detect outliers based on a rolling MAD algorithm ...
- 533
0
votes
1
answer
98
views
How to do "rolling grouping" in R?
I have a dataset with daily data with the following structure (with many firms and for a very long time frame) in R:
date month_id company_id value
2024-01-02 1 1 2
2024-...
- 133
2
votes
1
answer
107
views
Create rolling windows with custom overlap
I'd like to analyse time-series data along multiple rolling windows. As a first step, I'd like to extract the start and end-times of said windows. Sadly, using zoo:rollapply seems to be missing a ...
- 177
1
vote
1
answer
85
views
Aggregate 3-month rolling dates with overlap [closed]
Suppose I have below dataset:
date Value
01-Jul-24 37
01-Aug-24 76
01-Sep-24 25
01-Oct-24 85
01-Nov-24 27
01-Dec-24 28
And I want to aggregate by 3 months rolling:...
- 13
2
votes
2
answers
89
views
How to group by on multiple columns and retain the original index in a pandas dataframe?
I need to group by multiple columns on a dataframe and calculate the rolling mean in the group. But the original index needs to be preserved.
Simple python code below :
data = {'values': [1,2,3,4,5,6,...
- 173
1
vote
0
answers
53
views
Unique IDs per 3 month rolling period in SQL
I have an Appointments table with the columns MemberID and DateOfConsultation. Each member may have more than 1 consultation, with the same or different dates. I want to get a rolling 3 month period, ...
4
votes
1
answer
109
views
Grouped Rolling Mean in Polars
Similar question is asked here
However it didn't seem to work in my case.
I have a dataframe with 3 columns, date, groups, prob. What I want is to create a 3 day rolling mean of the prob column values ...
- 327
5
votes
2
answers
166
views
Perform a rolling operation on indices without using `with_row_index()`?
I have a DataFrame like this:
import polars as pl
df = pl.DataFrame({"x": [1.2, 1.3, 3.4, 3.5]})
df
# shape: (3, 1)
# ┌─────┐
# │ a │
# │ --- │
# │ f64 │
# ╞═════╡
# │ 1.2 │
# │ 1.3 │
# │...
- 8,865
2
votes
1
answer
71
views
R calculate timestamp of earliest sequential prior message based on user ID of who sent message
I have a dataset of messages being sent between two people (user A and user B) on different chats over time. I need to figure out how to track the datetime of the last but earliest user A message ...
- 327
2
votes
2
answers
63
views
R calculate timestamp of last message based on user ID of who sent message
I have a dataset of messages being sent between two people (user A and user B) on different chats over time. I need to figure out how to track the datetime of the last, most recent user A message ...
- 327
3
votes
1
answer
106
views
Rolling Absolute Median Deviation Efficient Algorithm?
Is there an efficient algo for Rolling Absolute Median Deviation (MAD)? The Moving Window sliding over the Array X, and computing MAD for every array item.
It's possible to efficiently compute Rolling ...
- 15.8k
0
votes
0
answers
21
views
querying from json rolling data files
I have a series of json files. Each file has the information for the last 30 days. Within each file, the data is broken down by date. So for example,
File_Mar31 has data for Mar 1 - Mar 30 (broken ...
- 567
0
votes
0
answers
26
views
Pandas Dataframe rolling mean of last 50 daily values differs from rolling("50D").mean() [duplicate]
I'm trying to find how the "50D" rolling mean is being calculated in the following example because really I cannot find the way.
import pandas as pd
values = [np.nan, -0.00076194, -0....
4
votes
1
answer
449
views
How to find the last non-null value before the current row in Polars?
I'd like to perform the following:
Input:
import polars as pl
df = pl.DataFrame({
"a": [1,15,None,20,None]
})
Output:
expected = pl.from_repr("""
┌──────┬──────┐
│ a ┆...
- 301
1
vote
1
answer
78
views
Why does Pandas rolling method return a series with a different dtype to the original?
Just curious why the Pandas Series rolling window method doesn't preserve the data-type of the original series:
import numpy as np
import pandas as pd
x = pd.Series(np.ones(6), dtype='float32')
x....
- 11.8k
0
votes
0
answers
62
views
Pandas: how to perform groupby rolling deduplicate sum in an efficient manner?
We have a dataframe indexed by time with identifier id_col, a group columns group_col and value_col="value". The value_col is updated at random intervals with different frequencies per ...
- 131
0
votes
2
answers
197
views
How can I perform a SUM window function with a time range but handle duplicate timestamps row-wise in SQL
I have a scenario where I need to calculate a running total using the SUM window function in SQL. The issue arises because some rows have duplicate timestamps, and the RANGE clause in the window ...
1
vote
1
answer
39
views
Fill Nans with rolling mean which is a combination of actual data + calculated rolling mean
I have tried to find the solution to this but havent been able to. So asking.
I have a dataset for which I wish to forecast values for future dates for groups within the dataset. The goal would be to ...
- 11
1
vote
1
answer
41
views
Optimize Persistence Analysis with Rolling Windows in Pandas, Handling Overlaps Between Months
I’m relatively new to Python and coding in general. I’m working on a project to perform persistence analysis in a large dataset and am trying to avoid using for loops for efficiency. I’d appreciate ...
- 11
0
votes
2
answers
36
views
R calculating Decreasing Rolling Percentage with fixed start point across data frame
I have a data frame below, would like to add a column call Percentage.
Sort by each Trade
In CofQYr == 0 row, Percentage would be row Sum across all columns (from 2014 5o 2024) divide by the row sum ...
- 93
0
votes
2
answers
55
views
Groupby and rolling by monthly mean and stdev
data = {‘A’:list(range(250)), ‘B’: list(range(251,501))}
dates = pd.date_range(‘2023-01-17’, period=250, freq=‘D’)
df = pd.DataFrame(data, index=dates)
I want rolling 2 calendar month mean as follows:...
- 636
0
votes
1
answer
35
views
Calculate the average of a window of n elements with a minimum period of 1
I have this time serie:
var numNames = new SeriesBuilder<int, string>() {
{ 1, "one" }, { 2, "two" },{ 3, "three" }, { 4, "four" }, { 5, "five"...
-1
votes
1
answer
102
views
Apply expanding calculations on SQL Server
I have this table in SQL server :
date
var
val
2022-2-1
A
1.1
2022-3-1
A
2.3
2022-4-1
A
1.5
2022-5-1
A
1.7
2022-09-1
B
1.8
2022-10-1
B
1.9
2022-11-1
B
2.1
2022-12-1
B
2.22
I want to group by column ...
- 1,348
2
votes
2
answers
103
views
most frequent *rolling* value on a object/str pandas dataframe column
I would like to have a new column with the most frequent rolling value from another str/object column.
date name state
0 2024-02-29 Alice CA
1 2024-02-27 Bob HI
2 ...
- 79
1
vote
1
answer
51
views
A question for the Rolling Cross Validation where the tsCV function in R is used
I have one data set consisting of a total of 16 data (quarterly data from 2012 to 2015) as sample data set.
I want to predict the 15th data (3rd quarter of 2015) from the model learned with an arima ...
- 11
3
votes
2
answers
203
views
Polars Rolling Mean, fill start of window with null instead of shortened window
My question is whether there is a way to have null until the full window can be filled at the start of a rolling window in polars. For example:
dates = [
"2020-01-01",
"2020-01-...
- 327
3
votes
1
answer
95
views
Rolling mode in Polars
I have a ~100M rows long data frame containing IDs in different groups. Some of them are wrong (indicated by the 99). I am trying to correct them with a rolling mode window, similar to the code ...
- 353
4
votes
1
answer
179
views
Rolling linear regression slope over n rows by group
My data are at the ID-month level with a payment at every month. It is sorted by id and dt. What I'd like to do is a create a new column that, for each group, holds the linear slope for the next N ...
- 871
2
votes
2
answers
136
views
Apply multiple window sizes to rolling aggregation functions in polars dataframe
In a number of aggregation function, such as rolling_mean, rolling_max, rolling_min, etc, the input argument window_size is supposed to be of type int
I am wondering how to efficiently compute results ...
- 5,167
0
votes
1
answer
75
views
rolling average across grouped dates
I have a time series of half-hourly observations. I want a running average of the range of each day's measurements. I group by date and get the correct daily ranges, but then everything I can think ...
- 147
0
votes
1
answer
65
views
Create columns based on a rolling condition
I have a dataframe of 2 asset prices and the Z score of their ratio (A/B) and am trying to do a simple backtest with signal generation of a pairs ratio trade of entering when Z score < -2 (long ...
- 33
4
votes
1
answer
78
views
Rolling KPI calculations in polars, index not visible
How to add rolling KPI's to original dataframe in polars? when I do group by, I am not seeing an index and so cant join? I want to keep all original columns in dataframe intact but add rolling kpi to ...
- 290
1
vote
1
answer
76
views
Apply sklearn logloss with rolling on pandas dataframe
My function call looks something like
loss = log_loss(y_true=validate_d['y'], y_pred=validate_probs, sample_weight=validate_df['weight'], normalize=True)
Is there any way to combine this with pandas ...
- 4,355
0
votes
1
answer
31
views
DAX Rolling Averages Totals on table doesnt show
On this SQLBI video rolling average doesnt calculate anything to totals row? Why is that? Would like to have snapshot value also for rolling value on card etc
https://www.youtube.com/watch?v=...
- 45
0
votes
1
answer
137
views
Calculating 5 Year Moving Average in R with inflation-adjusted exchange rates [closed]
I tried to calculate a 5 Year Moving Average of Exchange Rate in R but I can't get the code to work properly. I want to calculate a Moving Average similar to Atlas Conversion Factor used by World Bank ...
- 15
2
votes
1
answer
224
views
How to apply rolling_map on polars and create two columns?
See the code below, it's a stand-in for what I need. ChatGPT and perplexity aren't helpful here. In practice I can compute col1 and col2 one at a time, but it is unnecessarily slow.
import numpy as np
...
- 1,176
0
votes
1
answer
63
views
Pandas hybrid rolling mean
I want to consider a window of size 3, in which I take the last value from the b column, and the other 2 values from the a column, like in this example:
df = pd.DataFrame.from_dict({'a':[1, 2, 3, 4, 5]...
- 4,355
1
vote
1
answer
46
views
Create columns based on a rolling condition without looping
Background
I have a dataframe of stock prices with close price, day high, day low prices and am trying to backtest potential short sell entries.
The logic is if there has been any 30% downwards ...
- 33
1
vote
1
answer
74
views
Vega Lite Percent of a count for a rolling year,
This is what I am trying to accomplish codelink
I want a rolling percent for a rolling year that looks back either 365 days or 11 months, I am just not sure how the frames work. If a hover over a ...
- 97
6
votes
3
answers
770
views
Rolling aggregation in polars and also get the original column back without join or using .over
Using polars .rolling and .agg, how do I get the original column back, without having to join back with the original column, or without having to use .over?
Example:
import polars as pl
dates = [
...
- 5,232
0
votes
1
answer
184
views
How to use (Rolling/Sliding) Window Linear Regression to Predict Future Data
I am trying to implement a sliding window regression to predict stock prices and I realized I didn't understand how sliding window regression works for prediction. I have some training data (the ...
- 101
2
votes
4
answers
98
views
How to conditionally count no. of occurrences in sliding window?
Take the following example data set
structure(list(id = c("AA2A", "AA2A", "AA2A", "AA2A", "AA2A",
"AA2A", "XX7H", "XX7H"...
- 131