col function error type mismatch: found string required Int

Question

I am attempting to programmatically remove specific columns/fields from a dataframe (anything that starts with _), whether the field is in the root or in a struct, using the dropFields method.

For example, if I had "foo._baz", the syntax would be:

df.withColumn("foo",col("foo").dropFields("_baz"))

Hard coded this works fine. When I try to do this in a loop, generating the strings "foo" and "_baz", I get a type mismatch.

I've got a function that parses the columns to find any starting with _, that's working fine. Here's the (hopefully) relevant bit of my code:

var (baseCol, dropCol) = getColumnNodes(col)
df2 = df2.withColumn(baseCol,col(baseCol).dropFields(dropCol)

That results in:

error: type mismatch;
 found   : String
 required: Int
    df2 = df2.withColumn(baseCol,col(baseCol).dropFields(dropCol))

Where am I going wrong? ^

Andrew · Accepted Answer · 2025-11-10 16:48:08Z

1

I've got this resolved, will post this here in case it helps anyone. This bit here was the problem:

col(baseCol)

I don't understand exactly why this doesn't work, but it doesn't. Using the $ notation like so does work:

 df2 = df2.withColumn(baseCol,( $"$baseCol".dropFields(dropCol)))

answered Nov 10 at 16:48

Andrew

8,9133 gold badges31 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

col function error type mismatch: found string required Int

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related