重复标签#

Index 对象不需要是唯一的;您可以拥有重复的行或列标签。这可能一开始有点令人困惑。如果您熟悉 SQL,您知道行标签类似于表上的主键,并且您永远不希望在 SQL 表中出现重复项。但 pandas 的作用之一是在数据进入某些下游系统之前清理混乱的现实世界数据。现实世界的数据有重复项,即使是在应该唯一的字段中也是如此。

本节介绍重复标签如何改变某些操作的行为,以及如何在操作过程中防止重复项出现,或者在重复项出现时检测它们。

In [1]: import pandas as pd

In [2]: import numpy as np

重复标签的后果#

某些 pandas 方法(例如 Series.reindex())在存在重复项的情况下无法正常工作。输出无法确定,因此 pandas 会引发异常。

In [3]: s1 = pd.Series([0, 1, 2], index=["a", "b", "b"])

In [4]: s1.reindex(["a", "b", "c"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 1
----> 1 s1.reindex(["a", "b", "c"])

File ~/work/pandas/pandas/pandas/core/series.py:5144, in Series.reindex(self, index, axis, method, copy, level, fill_value, limit, tolerance)
   5127 @doc(
   5128     NDFrame.reindex,  # type: ignore[has-type]
   5129     klass=_shared_doc_kwargs["klass"],
   (...)
   5142     tolerance=None,
   5143 ) -> Series:
-> 5144     return super().reindex(
   5145         index=index,
   5146         method=method,
   5147         copy=copy,
   5148         level=level,
   5149         fill_value=fill_value,
   5150         limit=limit,
   5151         tolerance=tolerance,
   5152     )

File ~/work/pandas/pandas/pandas/core/generic.py:5607, in NDFrame.reindex(self, labels, index, columns, axis, method, copy, level, fill_value, limit, tolerance)
   5604     return self._reindex_multi(axes, copy, fill_value)
   5606 # perform the reindex on the axes
-> 5607 return self._reindex_axes(
   5608     axes, level, limit, tolerance, method, fill_value, copy
   5609 ).__finalize__(self, method="reindex")

File ~/work/pandas/pandas/pandas/core/generic.py:5630, in NDFrame._reindex_axes(self, axes, level, limit, tolerance, method, fill_value, copy)
   5627     continue
   5629 ax = self._get_axis(a)
-> 5630 new_index, indexer = ax.reindex(
   5631     labels, level=level, limit=limit, tolerance=tolerance, method=method
   5632 )
   5634 axis = self._get_axis_number(a)
   5635 obj = obj._reindex_with_indexers(
   5636     {axis: [new_index, indexer]},
   5637     fill_value=fill_value,
   5638     copy=copy,
   5639     allow_dups=False,
   5640 )

File ~/work/pandas/pandas/pandas/core/indexes/base.py:4429, in Index.reindex(self, target, method, level, limit, tolerance)
   4426     raise ValueError("cannot handle a non-unique multi-index!")
   4427 elif not self.is_unique:
   4428     # GH#42568
-> 4429     raise ValueError("cannot reindex on an axis with duplicate labels")
   4430 else:
   4431     indexer, _ = self.get_indexer_non_unique(target)

ValueError: cannot reindex on an axis with duplicate labels

其他方法,例如索引,可能会产生非常令人惊讶的结果。通常,用标量索引会降低维度。用标量切片DataFrame将返回一个Series。用标量切片Series将返回一个标量。但是,对于重复项,情况并非如此。

In [5]: df1 = pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=["A", "A", "B"])

In [6]: df1
Out[6]: 
   A  A  B
0  0  1  2
1  3  4  5

我们在列中存在重复项。如果我们切片'B',我们将得到一个Series

In [7]: df1["B"]  # a series
Out[7]: 
0    2
1    5
Name: B, dtype: int64

但是切片'A'会返回一个DataFrame

In [8]: df1["A"]  # a DataFrame
Out[8]: 
   A  A
0  0  1
1  3  4

这也适用于行标签

In [9]: df2 = pd.DataFrame({"A": [0, 1, 2]}, index=["a", "a", "b"])

In [10]: df2
Out[10]: 
   A
a  0
a  1
b  2

In [11]: df2.loc["b", "A"]  # a scalar
Out[11]: 2

In [12]: df2.loc["a", "A"]  # a Series
Out[12]: 
a    0
a    1
Name: A, dtype: int64

重复标签检测#

您可以使用Index.is_unique检查Index(存储行或列标签)是否唯一

In [13]: df2
Out[13]: 
   A
a  0
a  1
b  2

In [14]: df2.index.is_unique
Out[14]: False

In [15]: df2.columns.is_unique
Out[15]: True

注意

检查索引是否唯一对于大型数据集来说成本较高。pandas 会缓存此结果,因此对同一索引进行重新检查非常快。

Index.duplicated()将返回一个布尔型 ndarray,指示标签是否重复。

In [16]: df2.index.duplicated()
Out[16]: array([False,  True, False])

这可以用作布尔型过滤器来删除重复行。

In [17]: df2.loc[~df2.index.duplicated(), :]
Out[17]: 
   A
a  0
b  2

如果您需要额外的逻辑来处理重复标签,而不是仅仅删除重复项,那么对索引使用groupby()是一个常见的技巧。例如,我们将通过对具有相同标签的所有行的平均值来解决重复项。

In [18]: df2.groupby(level=0).mean()
Out[18]: 
     A
a  0.5
b  2.0

禁止重复标签#

版本 1.2.0 中的新增功能。

如上所述,处理重复项是在读取原始数据时的一项重要功能。也就是说,您可能希望避免在数据处理管道(来自pandas.concat()rename()等方法)中引入重复项。SeriesDataFrame通过调用.set_flags(allows_duplicate_labels=False)禁止重复标签。(默认情况下允许它们)。如果存在重复标签,将引发异常。

In [19]: pd.Series([0, 1, 2], index=["a", "b", "b"]).set_flags(allows_duplicate_labels=False)
---------------------------------------------------------------------------
DuplicateLabelError                       Traceback (most recent call last)
Cell In[19], line 1
----> 1 pd.Series([0, 1, 2], index=["a", "b", "b"]).set_flags(allows_duplicate_labels=False)

File ~/work/pandas/pandas/pandas/core/generic.py:507, in NDFrame.set_flags(self, copy, allows_duplicate_labels)
    505 df = self.copy(deep=copy and not using_copy_on_write())
    506 if allows_duplicate_labels is not None:
--> 507     df.flags["allows_duplicate_labels"] = allows_duplicate_labels
    508 return df

File ~/work/pandas/pandas/pandas/core/flags.py:109, in Flags.__setitem__(self, key, value)
    107 if key not in self._keys:
    108     raise ValueError(f"Unknown flag {key}. Must be one of {self._keys}")
--> 109 setattr(self, key, value)

File ~/work/pandas/pandas/pandas/core/flags.py:96, in Flags.allows_duplicate_labels(self, value)
     94 if not value:
     95     for ax in obj.axes:
---> 96         ax._maybe_check_unique()
     98 self._allows_duplicate_labels = value

File ~/work/pandas/pandas/pandas/core/indexes/base.py:715, in Index._maybe_check_unique(self)
    712 duplicates = self._format_duplicate_message()
    713 msg += f"\n{duplicates}"
--> 715 raise DuplicateLabelError(msg)

DuplicateLabelError: Index has duplicates.
      positions
label          
b        [1, 2]

这适用于 DataFrame 的行标签和列标签。

In [20]: pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=["A", "B", "C"],).set_flags(
   ....:     allows_duplicate_labels=False
   ....: )
   ....: 
Out[20]: 
   A  B  C
0  0  1  2
1  3  4  5

此属性可以通过 allows_duplicate_labels 进行检查或设置,它指示该对象是否可以具有重复标签。

In [21]: df = pd.DataFrame({"A": [0, 1, 2, 3]}, index=["x", "y", "X", "Y"]).set_flags(
   ....:     allows_duplicate_labels=False
   ....: )
   ....: 

In [22]: df
Out[22]: 
   A
x  0
y  1
X  2
Y  3

In [23]: df.flags.allows_duplicate_labels
Out[23]: False

DataFrame.set_flags() 可用于返回一个新的 DataFrame,其属性(如 allows_duplicate_labels)设置为某个值。

In [24]: df2 = df.set_flags(allows_duplicate_labels=True)

In [25]: df2.flags.allows_duplicate_labels
Out[25]: True

返回的新 DataFrame 是对旧 DataFrame 中相同数据的视图。或者,可以直接在同一个对象上设置该属性。

In [26]: df2.flags.allows_duplicate_labels = False

In [27]: df2.flags.allows_duplicate_labels
Out[27]: False

在处理原始的、杂乱无章的数据时,您可能最初会读取杂乱无章的数据(可能包含重复标签),进行去重,然后禁止以后出现重复,以确保您的数据管道不会引入重复。

>>> raw = pd.read_csv("...")
>>> deduplicated = raw.groupby(level=0).first()  # remove duplicates
>>> deduplicated.flags.allows_duplicate_labels = False  # disallow going forward

在具有重复标签的 SeriesDataFrame 上设置 allows_duplicate_labels=False,或者对禁止重复的 SeriesDataFrame 执行引入重复标签的操作,将引发 errors.DuplicateLabelError

In [28]: df.rename(str.upper)
---------------------------------------------------------------------------
DuplicateLabelError                       Traceback (most recent call last)
Cell In[28], line 1
----> 1 df.rename(str.upper)

File ~/work/pandas/pandas/pandas/core/frame.py:5754, in DataFrame.rename(self, mapper, index, columns, axis, copy, inplace, level, errors)
   5623 def rename(
   5624     self,
   5625     mapper: Renamer | None = None,
   (...)
   5633     errors: IgnoreRaise = "ignore",
   5634 ) -> DataFrame | None:
   5635     """
   5636     Rename columns or index labels.
   5637 
   (...)
   5752     4  3  6
   5753     """
-> 5754     return super()._rename(
   5755         mapper=mapper,
   5756         index=index,
   5757         columns=columns,
   5758         axis=axis,
   5759         copy=copy,
   5760         inplace=inplace,
   5761         level=level,
   5762         errors=errors,
   5763     )

File ~/work/pandas/pandas/pandas/core/generic.py:1139, in NDFrame._rename(self, mapper, index, columns, axis, copy, inplace, level, errors)
   1137     return None
   1138 else:
-> 1139     return result.__finalize__(self, method="rename")

File ~/work/pandas/pandas/pandas/core/generic.py:6259, in NDFrame.__finalize__(self, other, method, **kwargs)
   6252 if other.attrs:
   6253     # We want attrs propagation to have minimal performance
   6254     # impact if attrs are not used; i.e. attrs is an empty dict.
   6255     # One could make the deepcopy unconditionally, but a deepcopy
   6256     # of an empty dict is 50x more expensive than the empty check.
   6257     self.attrs = deepcopy(other.attrs)
-> 6259 self.flags.allows_duplicate_labels = other.flags.allows_duplicate_labels
   6260 # For subclasses using _metadata.
   6261 for name in set(self._metadata) & set(other._metadata):

File ~/work/pandas/pandas/pandas/core/flags.py:96, in Flags.allows_duplicate_labels(self, value)
     94 if not value:
     95     for ax in obj.axes:
---> 96         ax._maybe_check_unique()
     98 self._allows_duplicate_labels = value

File ~/work/pandas/pandas/pandas/core/indexes/base.py:715, in Index._maybe_check_unique(self)
    712 duplicates = self._format_duplicate_message()
    713 msg += f"\n{duplicates}"
--> 715 raise DuplicateLabelError(msg)

DuplicateLabelError: Index has duplicates.
      positions
label          
X        [0, 2]
Y        [1, 3]

此错误消息包含重复的标签,以及 SeriesDataFrame 中所有重复项(包括“原始”项)的数字位置。

重复标签传播#

通常,禁止重复是“粘性”的。它在操作中得以保留。

In [29]: s1 = pd.Series(0, index=["a", "b"]).set_flags(allows_duplicate_labels=False)

In [30]: s1
Out[30]: 
a    0
b    0
dtype: int64

In [31]: s1.head().rename({"a": "b"})
---------------------------------------------------------------------------
DuplicateLabelError                       Traceback (most recent call last)
Cell In[31], line 1
----> 1 s1.head().rename({"a": "b"})

File ~/work/pandas/pandas/pandas/core/series.py:5081, in Series.rename(self, index, axis, copy, inplace, level, errors)
   5074     axis = self._get_axis_number(axis)
   5076 if callable(index) or is_dict_like(index):
   5077     # error: Argument 1 to "_rename" of "NDFrame" has incompatible
   5078     # type "Union[Union[Mapping[Any, Hashable], Callable[[Any],
   5079     # Hashable]], Hashable, None]"; expected "Union[Mapping[Any,
   5080     # Hashable], Callable[[Any], Hashable], None]"
-> 5081     return super()._rename(
   5082         index,  # type: ignore[arg-type]
   5083         copy=copy,
   5084         inplace=inplace,
   5085         level=level,
   5086         errors=errors,
   5087     )
   5088 else:
   5089     return self._set_name(index, inplace=inplace, deep=copy)

File ~/work/pandas/pandas/pandas/core/generic.py:1139, in NDFrame._rename(self, mapper, index, columns, axis, copy, inplace, level, errors)
   1137     return None
   1138 else:
-> 1139     return result.__finalize__(self, method="rename")

File ~/work/pandas/pandas/pandas/core/generic.py:6259, in NDFrame.__finalize__(self, other, method, **kwargs)
   6252 if other.attrs:
   6253     # We want attrs propagation to have minimal performance
   6254     # impact if attrs are not used; i.e. attrs is an empty dict.
   6255     # One could make the deepcopy unconditionally, but a deepcopy
   6256     # of an empty dict is 50x more expensive than the empty check.
   6257     self.attrs = deepcopy(other.attrs)
-> 6259 self.flags.allows_duplicate_labels = other.flags.allows_duplicate_labels
   6260 # For subclasses using _metadata.
   6261 for name in set(self._metadata) & set(other._metadata):

File ~/work/pandas/pandas/pandas/core/flags.py:96, in Flags.allows_duplicate_labels(self, value)
     94 if not value:
     95     for ax in obj.axes:
---> 96         ax._maybe_check_unique()
     98 self._allows_duplicate_labels = value

File ~/work/pandas/pandas/pandas/core/indexes/base.py:715, in Index._maybe_check_unique(self)
    712 duplicates = self._format_duplicate_message()
    713 msg += f"\n{duplicates}"
--> 715 raise DuplicateLabelError(msg)

DuplicateLabelError: Index has duplicates.
      positions
label          
b        [0, 1]

警告

这是一个实验性功能。目前,许多方法无法传播 allows_duplicate_labels 值。在未来的版本中,预计每个接受或返回一个或多个 DataFrame 或 Series 对象的方法都将传播 allows_duplicate_labels