Using Pipeline with custom classes in sklearn

Question

I have an issue during predict inside of Pipelines flow with custom classes per each pipe step.

class MyFeatureSelector():
    def __init__(self, features=5, method='pca'):
        self.features = features
        self.method = method

    def fit(self, X, Y):
        return self

    def transform(self, X, Y=None):
        try:
            if self.features < X.shape[1]:
                if self.method == 'pca':
                    selector = PCA(n_components=self.features)
                elif self.method == 'rfe':
                    selector = RFE(estimator=LinearRegression(n_jobs=-1),
                                   n_features_to_select=self.features,
                                   step=1)
                selector.fit(X, Y)
                return selector.transform(X)
        except Exception as err:
            print('MyFeatureSelector.transform(): {}'.format(err))
        return X

    def fit_transform(self, X, Y=None):
        self.fit(X, Y)
        return self.transform(X, Y)


model = Pipeline([
    ("DATA_CLEANER", MyDataCleaner(demo='', mode='strict')),
    ("DATA_ENCODING", MyEncoder(encoder_name='code')),
    ("FEATURE_SELECTION", MyFeatureSelector(features=15, method='rfe')),
    ("HUBER_MODELLING", HuberRegressor())
])

So, code above works very good here:

 model.fit(X, _Y)

But I have an error here

 prediction = model.predict(XT)

ERROR: shapes (672,107) and (15,) not aligned: 107 (dim 1) != 15 (dim 0)

Debug shows that issue here: selector.fit(X, Y) because new instance of MyFeatureSelector was created during predict() step and Y is not exists at that moment.

Where was I wrong?

Yes. Because the pipeline will call fit() during training and only transform() during predict. And you are calling selector.fit() inside transform(), which will change the data — Vivek Kumar
– Vivek Kumar, Commented Apr 5, 2017 at 13:42
First check the shapes of X, XT and Y_. Show the full stack trace of error. Then I may able to suggest changes in your custom class. — Vivek Kumar
– Vivek Kumar, Commented Apr 5, 2017 at 13:43

SpanishBoy · Accepted Answer · 2017-04-06 09:45:04Z

Working version posted below:

class MyFeatureSelector():
    def __init__(self, features=5, method='pca'):
        self.features = features
        self.method = method
        self.selector = None
        self.init_selector()


    def init_selector():
        if self.method == 'pca':
            self.selector = PCA(n_components=self.features)
        elif self.method == 'rfe':
        self.selector = RFE(estimator=LinearRegression(n_jobs=-1),
                               n_features_to_select=self.features,
                               step=1)

    def fit(self, X, Y):
       return self

    def transform(self, X, Y=None):
        try:
            if self.features < X.shape[1]:
                if Y is not None:
                    self.selector.fit(X, Y)
                return selector.transform(X)
        except Exception as err:
            print('MyFeatureSelector.transform(): {}'.format(err))
       return X

def fit_transform(self, X, Y=None):
    self.fit(X, Y)
    return self.transform(X, Y)

Collectives™ on Stack Overflow

Using Pipeline with custom classes in sklearn

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related