Skip to content

Add support for TypedDeltaTable #727

@pgrandjean

Description

@pgrandjean

Frameless introduced the TypedDataset on top of Apache Spark's Dataset API. The aim of this ticket is to create a TypedDeltaTable on top of delta.io's DeltaTable API. This would allow to have strongly typed merge actions, something that would look like the following:

val spark: SparkSession = ...
val updates: TypedDataset[U] = ...
val target: TypedDeltaTable[T] = TypedDataTable.unsafeForPath[T](spark, path)

val sq = target
  .merge[U](updates, target('id) === updates('id))
  .whenMatched(target('someSymbol) != updates('anotherSymbol))
  .update(target('someSymbol) := updates('anotherSymbol))
  .execute()

sq.awaitTermination()

This could be included in a new module, called frameless-delta.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions