Question

Reading a parquet file using parquet-tool converting string values to scientific notation

score 95 · Answer 1 · 2024-02-02T15:18:28.853000

95

Views

Reading a parquet file using parquet-tool converting string values to scientific notation

Published on 02 February 2024 at 15:18

score 46 · Answer 2 · 2024-01-23T16:20:30.933000

46

Views

Is there a painless way to migrate existing Parquet files to a new schema? I wish to update an AWS Glue table column data type

Published on 23 January 2024 at 16:20

score 13 · Answer 3 · 2023-12-21T16:40:13.670000

13

Views

pyspark structural streaming one column dynamic json save to elasticsearch as json object

Published on 21 December 2023 at 16:40

score 124 · Answer 4 · 2023-12-02T09:27:56.617000

124

Views

pyspark error while writing data to existing hive table

Published on 02 December 2023 at 09:27

score 197 · Answer 5 · 2023-11-21T10:36:40.120000

197

Views

ValueError: Unable to parse datatype from schema. Could not parse datatype: interval year

Published on 21 November 2023 at 10:36

score 187 · Answer 6 · 2023-11-08T20:14:58.447000

187

Views

pyspark - how to add a new element to ArrayType column

Published on 08 November 2023 at 20:14

score 91 · Answer 7 · 2023-10-25T06:32:38.003000

91

Views

Need to add headers in existing data frame

Published on 25 October 2023 at 06:32

score 63 · Answer 8 · 2023-10-07T13:02:52.217000

63

Views

o1566.showString error persists even with os env variables defined with PySpark in jupyter notebook

Published on 07 October 2023 at 13:02

score 56 · Answer 9 · 2023-10-04T06:00:41.937000

56

Views

can we merge two parquet files into one which has different datatypes to the columns using the pyspark

Published on 04 October 2023 at 06:00

score 169 · Answer 10 · 2023-09-25T18:10:34.487000

169

Views

check if rows are already present in pyspark dataset

Published on 25 September 2023 at 18:10

score 343 · Answer 11 · 2023-09-17T20:28:08.457000

343

Views

PySpark DataFrame creation is throwing PySparkTypeError

Published on 17 September 2023 at 20:28

score 55 · Answer 12 · 2023-09-14T08:31:11.033000

55

Views

PySpark: Get Number of Columns from DataSchema

Published on 14 September 2023 at 08:31

score 81 · Answer 13 · 2023-09-10T10:02:18.600000

81

Views

How to get a timestamp data type column without the seconds in Pyspark?

Published on 10 September 2023 at 10:02

score 124 · Answer 14 · 2023-09-09T20:30:32.497000

124

Views

Pyspark calculate new rows based on previous rows from current and other multiple columns

Published on 09 September 2023 at 20:30

score 32 · Answer 15 · 2023-09-04T08:26:08.200000

32

Views

pyspark - making a new column lookup_l that contains a list and its elements are values from other columns from same df from current row

Published on 04 September 2023 at 08:26

score 270 · Answer 16 · 2023-09-03T17:39:13.570000

270

Views

How do I grab all data from multiple tables found in a certain table in PySpark?

Published on 03 September 2023 at 17:39

score 161 · Answer 17 · 2023-08-27T17:47:11.203000

161

Views

Azure Synapse PySpark - Load Schema from a Schema Definition File

Published on 27 August 2023 at 17:47

score 46 · Answer 18 · 2023-08-23T17:43:35.820000

46

Views

Mulitple Row in spark dataframe using schema

Published on 23 August 2023 at 17:43

score 186 · Answer 19 · 2023-08-04T12:53:28.177000

186

Views

Selecting a column with backtick in its name - AnalysisException: cannot resolve Column

Published on 04 August 2023 at 12:53

score 270 · Answer 20 · 2023-07-31T06:07:30.890000

270

Views

How to replace null value with some value using coalesce in pyspark

Published on 31 July 2023 at 06:07

List Question

Reading a parquet file using parquet-tool converting string values to scientific notation

Is there a painless way to migrate existing Parquet files to a new schema? I wish to update an AWS Glue table column data type

pyspark structural streaming one column dynamic json save to elasticsearch as json object

pyspark error while writing data to existing hive table

ValueError: Unable to parse datatype from schema. Could not parse datatype: interval year

pyspark - how to add a new element to ArrayType column

Need to add headers in existing data frame

o1566.showString error persists even with os env variables defined with PySpark in jupyter notebook

can we merge two parquet files into one which has different datatypes to the columns using the pyspark

check if rows are already present in pyspark dataset

PySpark DataFrame creation is throwing PySparkTypeError

PySpark: Get Number of Columns from DataSchema

How to get a timestamp data type column without the seconds in Pyspark?

Pyspark calculate new rows based on previous rows from current and other multiple columns

pyspark - making a new column lookup_l that contains a list and its elements are values from other columns from same df from current row

How do I grab all data from multiple tables found in a certain table in PySpark?

Azure Synapse PySpark - Load Schema from a Schema Definition File

Mulitple Row in spark dataframe using schema

Selecting a column with backtick in its name - AnalysisException: cannot resolve Column

How to replace null value with some value using coalesce in pyspark

Trending Questions

Popular # Hahtags

Popular Questions