You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: rules/S7469/python/rule.adoc
+16-2Lines changed: 16 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ In PySpark, a `DataFrame` with duplicate column names can cause ambiguous and un
8
8
* Joins with other DataFrames may produce unexpected results or errors
9
9
* Saving to external data sources may fail
10
10
11
-
Case-insensitive duplicates, for example a column named "name" and "Name", are also flagged. This is because having column names that differ only in casing creates confusion when referencing columns and makes code harder to understand and maintain. This can lead to subtle bugs that are difficult to detect and fix.
11
+
Case-insensitive duplicates, for example a column named "name" and "Name", are also flagged. This is because having column names that differ only in casing creates confusion when referencing columns and makes code harder to understand and maintain leading to subtle bugs that are difficult to detect and fix.
12
12
13
13
== How to fix it
14
14
To fix this issue, remove or rename the duplicate columns.
In addition to that, parts can be passed as variable, instead of literals. This seems to be especially common for schemas.
91
91
92
+
This rule could also apply to pandas `DataFrame`s, as well as the `DataFrame`s from Pandas API on Spark. However, this would increase the scope of an already big rule. Depending if the implementation raises on pandas (or pandas on spark) `DataFrame`s or not, the rule description should be updated to reflect this. Below are examples of how to construct a pandas `DataFrame` with duplicate column names.
Documentation for best practices for pandas on spark: https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/best_practices.html#do-not-use-duplicated-column-names[Best Practices]
0 commit comments