These functions are commonly used with groupBy(), agg(), or select() to compute things like sum, average, max, min, count, etc. PySpark functions come from pyspark.sql.functions, which includes a wide variety of operations like aggregation, date/time, string, and more.
🔹 1. Aggregation Functions
These are used to perform calculations on a group of rows.
| Function | Description | Example |
|---|---|---|
count() | Count number of rows | df.select(count("*")) |
sum() | Sum of column values | df.select(sum("salary")) |
avg() | Average of column values | df.select(avg("salary")) |
max() | Maximum value | df.select(max("salary")) |
min() | Minimum value | df.select(min("salary")) |
mean() | Alias for avg | df.select(mean("salary")) |
from pyspark.sql.functions import count, sum, avg, max, min
df.select(count("*"), sum("salary"), avg("salary")).show()
🔹 2. String Functions
Manipulate string columns.
| Function | Description | Example |
|---|---|---|
lower() | Convert to lowercase | df.select(lower("name")) |
upper() | Convert to uppercase | df.select(upper("name")) |
length() | String length | df.select(length("name")) |
substr() | Extract substring | df.select(substr("name", 1, 3)) |
concat() | Concatenate strings | df.select(concat(col("fname"), col("lname"))) |
trim() | Remove spaces | df.select(trim("name")) |
lpad() / rpad() | Pad strings | df.select(lpad("id", 5, "0")) |
from pyspark.sql.functions import lower, upper, length, concat, lit
df.select(lower("name"), upper("name"), length("name")).show()
df.select(concat(col("fname"), lit(" "), col("lname"))).show()
🔹 3. Date and Time Functions
| Function | Description | Example |
|---|---|---|
current_date() | Current date | df.select(current_date()) |
current_timestamp() | Current timestamp | df.select(current_timestamp()) |
date_add() | Add days to date | df.select(date_add(col("start_date"), 10)) |
date_sub() | Subtract days | df.select(date_sub(col("start_date"), 5)) |
datediff() | Difference in days | df.select(datediff(col("end_date"), col("start_date"))) |
year(), month(), dayofmonth() | Extract parts of date | df.select(year("date")) |
to_date() | Convert string to date | df.select(to_date("date_string")) |
from pyspark.sql.functions import current_date, datediff, year
df.select(current_date(), datediff(col("end_date"), col("start_date")), year("start_date")).show()
🔹 4. Null Handling Functions
| Function | Description | Example |
|---|---|---|
isnull() | Check for null | df.filter(col("salary").isNull()) |
fillna() | Replace null | df.fillna(0) |
na.drop() | Drop null rows | df.na.drop() |
coalesce() | First non-null value | df.select(coalesce(col("col1"), col("col2"))) |
df.fillna({'salary': 0}).show()
df.select(coalesce(col("bonus"), col("salary"))).show()
🔹 5. Conditional Functions
| Function | Description | Example |
|---|---|---|
when() | Like SQL CASE WHEN | df.select(when(col("age") > 18, "Adult").otherwise("Child")) |
lit() | Add constant value | df.select(lit("Hello")) |
from pyspark.sql.functions import when, lit
df.select(when(col("age") >= 18, "Adult").otherwise("Minor")).show()
🔹 6. Window Functions (Used with Window spec)
| Function | Description | Example |
|---|---|---|
row_number() | Row number in window | row_number().over(Window.partitionBy("dept")) |
rank() | Rank within window | rank().over(Window.partitionBy("dept")) |
dense_rank() | Dense rank | dense_rank().over(Window.partitionBy("dept")) |
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number
windowSpec = Window.partitionBy("department").orderBy("salary")
df.withColumn("row_num", row_number().over(windowSpec)).show()
🔹 7. Collection Functions
| Function | Description | Example |
|---|---|---|
array() | Create array | df.select(array("col1", "col2")) |
explode() | Explode array into rows | df.select(explode("hobbies")) |
size() | Get array size | df.select(size("hobbies")) |
from pyspark.sql.functions import array, explode, size
df.select(array("col1", "col2")).show()
df.select(explode("hobbies")).show()
🔹 8. JSON Functions
| Function | Description | Example |
|---|---|---|
get_json_object() | Extract JSON field | get_json_object(col("json_col"), "$.field") |
from_json() | Parse JSON string to struct | from_json(col("json_col"), schema) |
to_json() | Struct to JSON string | to_json(struct("col1", "col2")) |
🔹 9. Others
| Function | Description | Example |
|---|---|---|
col() | Reference a column | col("name") |
expr() | SQL expression | df.select(expr("salary * 0.1")) |
monotonically_increasing_id() | Generate unique ID | df.withColumn("id", monotonically_increasing_id()) |
simpleData =
schema =
df = spark.createDataFrame(data=simpleData, schema = schema)
df.printSchema()
df.show(truncate=False)
