Palantir Foundry Incremental decorator: does changing the Column-Ordering require a Semantic versioning update?

Question

In Palantir Foundry I was running a transformation with @incremental decorator for a long. Now I have to change the ordering of columns (no change in schema though).

Does that require a semantic-version update?

I have followed this palatir foundry documentation but not very clear.

Answer 1

This works. You don't need to bump the semantic version and the column will "get aligned".

Testing code, so that you can validate yourself:

from pyspark.sql import functions as F
from pyspark.sql import types as T
from transforms.api import transform, Input, Output, incremental
import random


# Generation of empty dataframe, that just appends the current timestamp
def get_empty_df(ctx, other_order=False):
    if other_order:
        schema = T.StructType([T.StructField("value", T.StringType(), True), T.StructField("key", T.StringType(), True)])
        df = ctx.spark_session.createDataFrame([("dummy_value_left", "dummy_key")], schema)

    else:
        schema = T.StructType([T.StructField("key", T.StringType(), True), T.StructField("value", T.StringType(), True)])
        df = ctx.spark_session.createDataFrame([("dummy_key", "dummy_value_right")], schema)

    df = df.withColumn('when', F.current_timestamp())
    return df


@incremental(semantic_version=2)
@transform(
    out=Output("/path/output_df")
)
def out_1(ctx, out):
    random_value = random.random()  # random number in range [0.0,1.0)

    df = get_empty_df(ctx, random_value < 0.5)

    # build all the time and append the timestamp
    # out.set_mode("modify") # rely on default behavior
    out.write_dataframe(df)