These are new features and improvements of note in each release.

v0.17.1 (November 21, 2015)¶ Note We are proud to announce that pandas has become a sponsored project of the (NUMFocus organization). This will help ensure the success of development of pandas as a world-class open-source project. This is a minor bug-fix release from 0.17.0 and includes a large number of bug fixes along several new features, enhancements, and performance improvements. We recommend that all users upgrade to this version. Highlights include: Support for Conditional HTML Formatting, see here

Releasing the GIL on the csv reader & other ops, see here

Fixed regression in DataFrame.drop_duplicates from 0.16.2, causing incorrect results on integer values (GH11376) What’s new in v0.17.1 New features Conditional HTML Formatting

Enhancements

API changes Deprecations

Performance Improvements

Bug Fixes New features¶ Conditional HTML Formatting¶ Warning This is a new feature and is under active development. We’ll be adding features an possibly making breaking changes in future releases. Feedback is welcome. We’ve added experimental support for conditional HTML formatting: the visual styling of a DataFrame based on the data. The styling is accomplished with HTML and CSS. Acesses the styler class with the pandas.DataFrame.style , attribute, an instance of Styler with your data attached. Here’s a quick example: In [1]: np . random . seed ( 123 ) In [2]: df = DataFrame ( np . random . randn ( 10 , 5 ), columns = list ( 'abcde' )) In [3]: html = df . style . background_gradient ( cmap = 'viridis' , low =. 5 ) We can render the HTML to get the following table. a b c d e 0 -1.085631 0.997345 0.282978 -1.506295 -0.5786 1 1.651437 -2.426679 -0.428913 1.265936 -0.86674 2 -0.678886 -0.094709 1.49139 -0.638902 -0.443982 3 -0.434351 2.20593 2.186786 1.004054 0.386186 4 0.737369 1.490732 -0.935834 1.175829 -1.253881 5 -0.637752 0.907105 -1.428681 -0.140069 -0.861755 6 -0.255619 -2.798589 -1.771533 -0.699877 0.927462 7 -0.173636 0.002846 0.688223 -0.879536 0.283627 8 -0.805367 -1.727669 -0.3909 0.573806 0.338589 9 -0.01183 2.392365 0.412912 0.978736 2.238143 Styler interacts nicely with the Jupyter Notebook. See the documentation for more. Enhancements¶ DatetimeIndex now supports conversion to strings with astype(str) (GH10442)

Support for compression (gzip/bz2) in pandas.DataFrame.to_csv() (GH7615)

pd.read_* functions can now also accept pathlib.Path , or py._path.local.LocalPath objects for the filepath_or_buffer argument. (GH11033) - The DataFrame and Series functions .to_csv() , .to_html() and .to_latex() can now handle paths beginning with tildes (e.g. ~/Documents/ ) (GH11438)

DataFrame now uses the fields of a namedtuple as columns, if columns are not supplied (GH11181)

DataFrame.itertuples() now returns namedtuple objects, when possible. (GH11269, GH11625)

Added axvlines_kwds to parallel coordinates plot (GH10709)

Option to .info() and .memory_usage() to provide for deep introspection of memory consumption. Note that this can be expensive to compute and therefore is an optional parameter. (GH11595) In [4]: df = DataFrame ({ 'A' : [ 'foo' ] * 1000 }) In [5]: df [ 'B' ] = df [ 'A' ] . astype ( 'category' ) # shows the '+' as we have object dtypes In [6]: df . info () <class 'pandas.core.frame.DataFrame'> RangeIndex: 1000 entries, 0 to 999 Data columns (total 2 columns): A 1000 non-null object B 1000 non-null category dtypes: category(1), object(1) memory usage: 8.9+ KB # we have an accurate memory assessment (but can be expensive to compute this) In [7]: df . info ( memory_usage = 'deep' ) <class 'pandas.core.frame.DataFrame'> RangeIndex: 1000 entries, 0 to 999 Data columns (total 2 columns): A 1000 non-null object B 1000 non-null category dtypes: category(1), object(1) memory usage: 48.0 KB

Index now has a fillna method (GH10089) In [8]: pd . Index ([ 1 , np . nan , 3 ]) . fillna ( 2 ) Out[8]: Float64Index ([ 1.0 , 2.0 , 3.0 ], dtype = 'float64' )

Series of type category now make .str.<...> and .dt.<...> accessor methods / properties available, if the categories are of that type. (GH10661) In [9]: s = pd . Series ( list ( 'aabb' )) . astype ( 'category' ) In [10]: s Out[10]: 0 a 1 a 2 b 3 b dtype: category Categories (2, object): [a, b] In [11]: s . str . contains ( "a" ) Out[11]: 0 True 1 True 2 False 3 False dtype: bool In [12]: date = pd . Series ( pd . date_range ( '1/1/2015' , periods = 5 )) . astype ( 'category' ) In [13]: date Out[13]: 0 2015-01-01 1 2015-01-02 2 2015-01-03 3 2015-01-04 4 2015-01-05 dtype: category Categories (5, datetime64[ns]): [2015-01-01, 2015-01-02, 2015-01-03, 2015-01-04, 2015-01-05] In [14]: date . dt . day Out[14]: 0 1 1 2 2 3 3 4 4 5 dtype: int64

pivot_table now has a margins_name argument so you can use something other than the default of ‘All’ (GH3335)

Implement export of datetime64[ns, tz] dtypes with a fixed HDF5 store (GH11411)

Pretty printing sets (e.g. in DataFrame cells) now uses set literal syntax ( {x, y} ) instead of Legacy Python syntax ( set([x, y]) ) (GH11215)

Improve the error message in pandas.io.gbq.to_gbq() when a streaming insert fails (GH11285) and when the DataFrame does not match the schema of the destination table (GH11359) API changes¶ raise NotImplementedError in Index.shift for non-supported index types (GH8038)

in for non-supported index types (GH8038) min and max reductions on datetime64 and timedelta64 dtyped series now result in NaT and not nan (GH11245).

and reductions on and dtyped series now result in and not (GH11245). Indexing with a null key will raise a TypeError , instead of a ValueError (GH11356)

, instead of a (GH11356) Series.ptp will now ignore missing values by default (GH11163) Deprecations¶ The pandas.io.ga module which implements google-analytics support is deprecated and will be removed in a future version (GH11308)

module which implements support is deprecated and will be removed in a future version (GH11308) Deprecate the engine keyword in .to_csv() , which will be removed in a future version (GH11274) Performance Improvements¶ Checking monotonic-ness before sorting on an index (GH11080)

Series.dropna performance improvement when its dtype can’t contain NaN (GH11159)

performance improvement when its dtype can’t contain (GH11159) Release the GIL on most datetime field operations (e.g. DatetimeIndex.year , Series.dt.year ), normalization, and conversion to and from Period , DatetimeIndex.to_period and PeriodIndex.to_timestamp (GH11263)

, ), normalization, and conversion to and from , and (GH11263) Release the GIL on some rolling algos: rolling_median , rolling_mean , rolling_max , rolling_min , rolling_var , rolling_kurt , rolling_skew (GH11450)

, , , , , , (GH11450) Release the GIL when reading and parsing text files in read_csv , read_table (GH11272)

, (GH11272) Improved performance of rolling_median (GH11450)

(GH11450) Improved performance of to_excel (GH11352)

(GH11352) Performance bug in repr of Categorical categories, which was rendering the strings before chopping them for display (GH11305)

categories, which was rendering the strings before chopping them for display (GH11305) Performance improvement in Categorical.remove_unused_categories , (GH11643).

, (GH11643). Improved performance of Series constructor with no data and DatetimeIndex (GH11433)

constructor with no data and (GH11433) Improved performance of shift , cumprod , and cumsum with groupby (GH4095) Bug Fixes¶ SparseArray.__iter__() now does not cause PendingDeprecationWarning in Python 3.5 (GH11622)

now does not cause in Python 3.5 (GH11622) Regression from 0.16.2 for output formatting of long floats/nan, restored in (GH11302)

Series.sort_index() now correctly handles the inplace option (GH11402)

now correctly handles the option (GH11402) Incorrectly distributed .c file in the build on PyPi when reading a csv of floats and passing na_values=<a scalar> would show an exception (GH11374)

when reading a csv of floats and passing would show an exception (GH11374) Bug in .to_latex() output broken when the index has a name (GH10660)

output broken when the index has a name (GH10660) Bug in HDFStore.append with strings whose encoded length exceded the max unencoded length (GH11234)

with strings whose encoded length exceded the max unencoded length (GH11234) Bug in merging datetime64[ns, tz] dtypes (GH11405)

dtypes (GH11405) Bug in HDFStore.select when comparing with a numpy scalar in a where clause (GH11283)

when comparing with a numpy scalar in a where clause (GH11283) Bug in using DataFrame.ix with a multi-index indexer (GH11372)

with a multi-index indexer (GH11372) Bug in date_range with ambigous endpoints (GH11626)

with ambigous endpoints (GH11626) Prevent adding new attributes to the accessors .str , .dt and .cat . Retrieving such a value was not possible, so error out on setting it. (GH10673)

, and . Retrieving such a value was not possible, so error out on setting it. (GH10673) Bug in tz-conversions with an ambiguous time and .dt accessors (GH11295)

accessors (GH11295) Bug in output formatting when using an index of ambiguous times (GH11619)

Bug in comparisons of Series vs list-likes (GH11339)

Bug in DataFrame.replace with a datetime64[ns, tz] and a non-compat to_replace (GH11326, GH11153)

with a and a non-compat to_replace (GH11326, GH11153) Bug in isnull where numpy.datetime64('NaT') in a numpy.array was not determined to be null(GH11206)

where in a was not determined to be null(GH11206) Bug in list-like indexing with a mixed-integer Index (GH11320)

Bug in pivot_table with margins=True when indexes are of Categorical dtype (GH10993)

with when indexes are of dtype (GH10993) Bug in DataFrame.plot cannot use hex strings colors (GH10299)

cannot use hex strings colors (GH10299) Regression in DataFrame.drop_duplicates from 0.16.2, causing incorrect results on integer values (GH11376)

from 0.16.2, causing incorrect results on integer values (GH11376) Bug in pd.eval where unary ops in a list error (GH11235)

where unary ops in a list error (GH11235) Bug in squeeze() with zero length arrays (GH11230, GH8999)

with zero length arrays (GH11230, GH8999) Bug in describe() dropping column names for hierarchical indexes (GH11517)

dropping column names for hierarchical indexes (GH11517) Bug in DataFrame.pct_change() not propagating axis keyword on .fillna method (GH11150)

not propagating keyword on method (GH11150) Bug in .to_csv() when a mix of integer and string column names are passed as the columns parameter (GH11637)

when a mix of integer and string column names are passed as the parameter (GH11637) Bug in indexing with a range , (GH11652)

, (GH11652) Bug in inference of numpy scalars and preserving dtype when setting columns (GH11638)

Bug in to_sql using unicode column names giving UnicodeEncodeError with (GH11431).

using unicode column names giving UnicodeEncodeError with (GH11431). Fix regression in setting of xticks in plot (GH11529).

in (GH11529). Bug in holiday.dates where observance rules could not be applied to holiday and doc enhancement (GH11477, GH11533)

where observance rules could not be applied to holiday and doc enhancement (GH11477, GH11533) Fix plotting issues when having plain Axes instances instead of SubplotAxes (GH11520, GH11556).

instances instead of (GH11520, GH11556). Bug in DataFrame.to_latex() produces an extra rule when header=False (GH7124)

produces an extra rule when (GH7124) Bug in df.groupby(...).apply(func) when a func returns a Series containing a new datetimelike column (GH11324)

when a func returns a containing a new datetimelike column (GH11324) Bug in pandas.json when file to load is big (GH11344)

when file to load is big (GH11344) Bugs in to_excel with duplicate columns (GH11007, GH10982, GH10970)

with duplicate columns (GH11007, GH10982, GH10970) Fixed a bug that prevented the construction of an empty series of dtype datetime64[ns, tz] (GH11245).

(GH11245). Bug in read_excel with multi-index containing integers (GH11317)

with multi-index containing integers (GH11317) Bug in to_excel with openpyxl 2.2+ and merging (GH11408)

with openpyxl 2.2+ and merging (GH11408) Bug in DataFrame.to_dict() produces a np.datetime64 object instead of Timestamp when only datetime is present in data (GH11327)

produces a object instead of when only datetime is present in data (GH11327) Bug in DataFrame.corr() raises exception when computes Kendall correlation for DataFrames with boolean and not boolean columns (GH11560)

raises exception when computes Kendall correlation for DataFrames with boolean and not boolean columns (GH11560) Bug in the link-time error caused by C inline functions on FreeBSD 10+ (with clang ) (GH10510)

functions on FreeBSD 10+ (with ) (GH10510) Bug in DataFrame.to_csv in passing through arguments for formatting MultiIndexes , including date_format (GH7791)

in passing through arguments for formatting , including (GH7791) Bug in DataFrame.join() with how='right' producing a TypeError (GH11519)

with producing a (GH11519) Bug in Series.quantile with empty list results has Index with object dtype (GH11588)

with empty list results has with dtype (GH11588) Bug in pd.merge results in empty Int64Index rather than Index(dtype=object) when the merge result is empty (GH11588)

results in empty rather than when the merge result is empty (GH11588) Bug in Categorical.remove_unused_categories when having NaN values (GH11599)

when having values (GH11599) Bug in DataFrame.to_sparse() loses column names for MultiIndexes (GH11600)

loses column names for MultiIndexes (GH11600) Bug in DataFrame.round() with non-unique column index producing a Fatal Python error (GH11611)

with non-unique column index producing a Fatal Python error (GH11611) Bug in DataFrame.round() with decimals being a non-unique indexed Series producing extra columns (GH11618)

v0.16.2 (June 12, 2015)¶ This is a minor bug-fix release from 0.16.1 and includes a a large number of bug fixes along some new features ( pipe() method), enhancements, and performance improvements. We recommend that all users upgrade to this version. Highlights include: A new pipe method, see here

method, see here Documentation on how to use numba with pandas, see here What’s new in v0.16.2 New features Pipe Other Enhancements

API Changes

Performance Improvements

Bug Fixes New features¶ Pipe¶ We’ve introduced a new method DataFrame.pipe() . As suggested by the name, pipe should be used to pipe data through a chain of function calls. The goal is to avoid confusing nested function calls like # df is a DataFrame # f, g, and h are functions that take and return DataFrames f ( g ( h ( df ), arg1 = 1 ), arg2 = 2 , arg3 = 3 ) The logic flows from inside out, and function names are separated from their keyword arguments. This can be rewritten as ( df . pipe ( h ) . pipe ( g , arg1 = 1 ) . pipe ( f , arg2 = 2 , arg3 = 3 ) ) Now both the code and the logic flow from top to bottom. Keyword arguments are next to their functions. Overall the code is much more readable. In the example above, the functions f , g , and h each expected the DataFrame as the first positional argument. When the function you wish to apply takes its data anywhere other than the first argument, pass a tuple of (function, keyword) indicating where the DataFrame should flow. For example: In [1]: import statsmodels.formula.api as sm In [2]: bb = pd . read_csv ( 'data/baseball.csv' , index_col = 'id' ) # sm.poisson takes (formula, data) In [3]: ( bb . query ( 'h > 0' ) ...: . assign ( ln_h = lambda df : np . log ( df . h )) ...: . pipe (( sm . poisson , 'data' ), 'hr ~ ln_h + year + g + C(lg)' ) ...: . fit () ...: . summary () ...: ) ...: Optimization terminated successfully. Current function value: 2.116284 Iterations 24 Out[3]: <class 'statsmodels.iolib.summary.Summary'> """ Poisson Regression Results ============================================================================== Dep. Variable: hr No. Observations: 68 Model: Poisson Df Residuals: 63 Method: MLE Df Model: 4 Date: Thu, 17 Mar 2016 Pseudo R-squ.: 0.6878 Time: 16:26:41 Log-Likelihood: -143.91 converged: True LL-Null: -460.91 LLR p-value: 6.774e-136 =============================================================================== coef std err z P>|z| [95.0% Conf. Int.] ------------------------------------------------------------------------------- Intercept -1267.3636 457.867 -2.768 0.006 -2164.767 -369.960 C(lg)[T.NL] -0.2057 0.101 -2.044 0.041 -0.403 -0.008 ln_h 0.9280 0.191 4.866 0.000 0.554 1.302 year 0.6301 0.228 2.762 0.006 0.183 1.077 g 0.0099 0.004 2.754 0.006 0.003 0.017 =============================================================================== """ The pipe method is inspired by unix pipes, which stream text through processes. More recently dplyr and magrittr have introduced the popular (%>%) pipe operator for R. See the documentation for more. (GH10129) Other Enhancements¶ Added rsplit to Index/Series StringMethods (GH10303)

Removed the hard-coded size limits on the DataFrame HTML representation in the IPython notebook, and leave this to IPython itself (only for IPython v3.0 or greater). This eliminates the duplicate scroll bars that appeared in the notebook with large frames (GH10231). Note that the notebook has a toggle output scrolling feature to limit the display of very large frames (by clicking left of the output). You can also configure the way DataFrames are displayed using the pandas options, see here here.

axis parameter of DataFrame.quantile now accepts also index and column . (GH9543) API Changes¶ Holiday now raises NotImplementedError if both offset and observance are used in the constructor instead of returning an incorrect result (GH10217). Performance Improvements¶ Improved Series.resample performance with dtype=datetime64[ns] (GH7754)

performance with (GH7754) Increase performance of str.split when expand=True (GH10081) Bug Fixes¶ Bug in Series.hist raises an error when a one row Series was given (GH10214)

raises an error when a one row was given (GH10214) Bug where HDFStore.select modifies the passed columns list (GH7212)

modifies the passed columns list (GH7212) Bug in Categorical repr with display.width of None in Python 3 (GH10087)

repr with of in Python 3 (GH10087) Bug in to_json with certain orients and a CategoricalIndex would segfault (GH10317)

with certain orients and a would segfault (GH10317) Bug where some of the nan funcs do not have consistent return dtypes (GH10251)

Bug in DataFrame.quantile on checking that a valid axis was passed (GH9543)

on checking that a valid axis was passed (GH9543) Bug in groupby.apply aggregation for Categorical not preserving categories (GH10138)

aggregation for not preserving categories (GH10138) Bug in to_csv where date_format is ignored if the datetime is fractional (GH10209)

where is ignored if the is fractional (GH10209) Bug in DataFrame.to_json with mixed data types (GH10289)

with mixed data types (GH10289) Bug in cache updating when consolidating (GH10264)

Bug in mean() where integer dtypes can overflow (GH10172)

where integer dtypes can overflow (GH10172) Bug where Panel.from_dict does not set dtype when specified (GH10058)

does not set dtype when specified (GH10058) Bug in Index.union raises AttributeError when passing array-likes. (GH10149)

raises when passing array-likes. (GH10149) Bug in Timestamp ‘s’ microsecond , quarter , dayofyear , week and daysinmonth properties return np.int type, not built-in int . (GH10050)

‘s’ , , , and properties return type, not built-in . (GH10050) Bug in NaT raises AttributeError when accessing to daysinmonth , dayofweek properties. (GH10096)

raises when accessing to , properties. (GH10096) Bug in Index repr when using the max_seq_items=None setting (GH10182).

setting (GH10182). Bug in getting timezone data with dateutil on various platforms ( GH9059, GH8639, GH9663, GH10121)

on various platforms ( GH9059, GH8639, GH9663, GH10121) Bug in displaying datetimes with mixed frequencies; display ‘ms’ datetimes to the proper precision. (GH10170)

Bug in setitem where type promotion is applied to the entire block (GH10280)

where type promotion is applied to the entire block (GH10280) Bug in Series arithmetic methods may incorrectly hold names (GH10068)

arithmetic methods may incorrectly hold names (GH10068) Bug in GroupBy.get_group when grouping on multiple keys, one of which is categorical. (GH10132)

when grouping on multiple keys, one of which is categorical. (GH10132) Bug in DatetimeIndex and TimedeltaIndex names are lost after timedelta arithmetics ( GH9926)

and names are lost after timedelta arithmetics ( GH9926) Bug in DataFrame construction from nested dict with datetime64 (GH10160)

construction from nested with (GH10160) Bug in Series construction from dict with datetime64 keys (GH9456)

construction from with keys (GH9456) Bug in Series.plot(label="LABEL") not correctly setting the label (GH10119)

not correctly setting the label (GH10119) Bug in plot not defaulting to matplotlib axes.grid setting (GH9792)

not defaulting to matplotlib setting (GH9792) Bug causing strings containing an exponent, but no decimal to be parsed as int instead of float in engine='python' for the read_csv parser (GH9565)

instead of in for the parser (GH9565) Bug in Series.align resets name when fill_value is specified (GH10067)

resets when is specified (GH10067) Bug in read_csv causing index name not to be set on an empty DataFrame (GH10184)

causing index name not to be set on an empty DataFrame (GH10184) Bug in SparseSeries.abs resets name (GH10241)

resets (GH10241) Bug in TimedeltaIndex slicing may reset freq (GH10292)

slicing may reset freq (GH10292) Bug in GroupBy.get_group raises ValueError when group key contains NaT (GH6992)

raises when group key contains (GH6992) Bug in SparseSeries constructor ignores input data name (GH10258)

constructor ignores input data name (GH10258) Bug in Categorical.remove_categories causing a ValueError when removing the NaN category if underlying dtype is floating-point (GH10156)

causing a when removing the category if underlying dtype is floating-point (GH10156) Bug where infer_freq infers timerule (WOM-5XXX) unsupported by to_offset (GH9425)

Bug in DataFrame.to_hdf() where table format would raise a seemingly unrelated error for invalid (non-string) column names. This is now explicitly forbidden. (GH9057)

where table format would raise a seemingly unrelated error for invalid (non-string) column names. This is now explicitly forbidden. (GH9057) Bug to handle masking empty DataFrame (GH10126).

(GH10126). Bug where MySQL interface could not handle numeric table/column names (GH10255)

Bug in read_csv with a date_parser that returned a datetime64 array of other time resolution than [ns] (GH10245)

with a that returned a array of other time resolution than (GH10245) Bug in Panel.apply when the result has ndim=0 (GH10332)

when the result has ndim=0 (GH10332) Bug in read_hdf where auto_close could not be passed (GH9327).

where could not be passed (GH9327). Bug in read_hdf where open stores could not be used (GH10330).

where open stores could not be used (GH10330). Bug in adding empty DataFrame``s, now results in a ``DataFrame that .equals an empty DataFrame (GH10181).

that an empty (GH10181). Bug in to_hdf and HDFStore which did not check that complib choices were valid (GH4582, GH8874).

v0.16.1 (May 11, 2015)¶ This is a minor bug-fix release from 0.16.0 and includes a a large number of bug fixes along several new features, enhancements, and performance improvements. We recommend that all users upgrade to this version. Highlights include: Support for a CategoricalIndex , a category based index, see here

, a category based index, see here New section on how-to-contribute to pandas, see here

Revised “Merge, join, and concatenate” documentation, including graphical examples to make it easier to understand each operations, see here

New method sample for drawing random samples from Series, DataFrames and Panels. See here

for drawing random samples from Series, DataFrames and Panels. See here The default Index printing has changed to a more uniform format, see here

printing has changed to a more uniform format, see here BusinessHour datetime-offset is now supported, see here

datetime-offset is now supported, see here Further enhancement to the .str accessor to make string operations easier, see here What’s new in v0.16.1 Enhancements CategoricalIndex Sample String Methods Enhancements Other Enhancements

API changes Deprecations

Index Representation

Performance Improvements

Bug Fixes Warning In pandas 0.17.0, the sub-package pandas.io.data will be removed in favor of a separately installable package. See here for details (GH8961) Enhancements¶ CategoricalIndex¶ We introduce a CategoricalIndex , a new type of index object that is useful for supporting indexing with duplicates. This is a container around a Categorical (introduced in v0.15.0) and allows efficient indexing and storage of an index with a large number of duplicated elements. Prior to 0.16.1, setting the index of a DataFrame/Series with a category dtype would convert this to regular object-based Index . In [1]: df = DataFrame ({ 'A' : np . arange ( 6 ), ...: 'B' : Series ( list ( 'aabbca' )) . astype ( 'category' , ...: categories = list ( 'cab' )) ...: }) ...: In [2]: df Out[2]: A B 0 0 a 1 1 a 2 2 b 3 3 b 4 4 c 5 5 a In [3]: df . dtypes Out[3]: A int64 B category dtype: object In [4]: df . B . cat . categories Out[4]: Index ([ u'c' , u'a' , u'b' ], dtype = 'object' ) setting the index, will create create a CategoricalIndex In [5]: df2 = df . set_index ( 'B' ) In [6]: df2 . index Out[6]: CategoricalIndex ([ u'a' , u'a' , u'b' , u'b' , u'c' , u'a' ], categories = [ u'c' , u'a' , u'b' ], ordered = False , name = u'B' , dtype = 'category' ) indexing with __getitem__/.iloc/.loc/.ix works similarly to an Index with duplicates. The indexers MUST be in the category or the operation will raise. In [7]: df2 . loc [ 'a' ] Out[7]: A B a 0 a 1 a 5 and preserves the CategoricalIndex In [8]: df2 . loc [ 'a' ] . index Out[8]: CategoricalIndex ([ u'a' , u'a' , u'a' ], categories = [ u'c' , u'a' , u'b' ], ordered = False , name = u'B' , dtype = 'category' ) sorting will order by the order of the categories In [9]: df2 . sort_index () Out[9]: A B c 4 a 0 a 1 a 5 b 2 b 3 groupby operations on the index will preserve the index nature as well In [10]: df2 . groupby ( level = 0 ) . sum () Out[10]: A B c 4 a 6 b 5 In [11]: df2 . groupby ( level = 0 ) . sum () . index Out[11]: CategoricalIndex ([ u'c' , u'a' , u'b' ], categories = [ u'c' , u'a' , u'b' ], ordered = False , name = u'B' , dtype = 'category' ) reindexing operations, will return a resulting index based on the type of the passed indexer, meaning that passing a list will return a plain-old- Index ; indexing with a Categorical will return a CategoricalIndex , indexed according to the categories of the PASSED Categorical dtype. This allows one to arbitrarly index these even with values NOT in the categories, similarly to how you can reindex ANY pandas index. In [12]: df2 . reindex ([ 'a' , 'e' ]) Out[12]: A B a 0.0 a 1.0 a 5.0 e NaN In [13]: df2 . reindex ([ 'a' , 'e' ]) . index Out[13]: Index ([ u'a' , u'a' , u'a' , u'e' ], dtype = 'object' , name = u'B' ) In [14]: df2 . reindex ( pd . Categorical ([ 'a' , 'e' ], categories = list ( 'abcde' ))) Out[14]: A B a 0.0 a 1.0 a 5.0 e NaN In [15]: df2 . reindex ( pd . Categorical ([ 'a' , 'e' ], categories = list ( 'abcde' ))) . index Out[15]: CategoricalIndex ([ u'a' , u'a' , u'a' , u'e' ], categories = [ u'a' , u'e' ], ordered = False , name = u'B' , dtype = 'category' ) See the documentation for more. (GH7629, GH10038, GH10039) Sample¶ Series, DataFrames, and Panels now have a new method: sample() . The method accepts a specific number of rows or columns to return, or a fraction of the total number or rows or columns. It also has options for sampling with or without replacement, for passing in a column for weights for non-uniform sampling, and for setting seed values to facilitate replication. (GH2419) In [16]: example_series = Series ([ 0 , 1 , 2 , 3 , 4 , 5 ]) # When no arguments are passed, returns 1 In [17]: example_series . sample () Out[17]: 5 5 dtype: int64 # One may specify either a number of rows: In [18]: example_series . sample ( n = 3 ) Out[18]: 5 5 3 3 2 2 dtype: int64 # Or a fraction of the rows: In [19]: example_series . sample ( frac = 0.5 ) Out[19]: 5 5 0 0 3 3 dtype: int64 # weights are accepted. In [20]: example_weights = [ 0 , 0 , 0.2 , 0.2 , 0.2 , 0.4 ] In [21]: example_series . sample ( n = 3 , weights = example_weights ) Out[21]: 5 5 2 2 4 4 dtype: int64 # weights will also be normalized if they do not sum to one, # and missing values will be treated as zeros. In [22]: example_weights2 = [ 0.5 , 0 , 0 , 0 , None , np . nan ] In [23]: example_series . sample ( n = 1 , weights = example_weights2 ) Out[23]: 0 0 dtype: int64 When applied to a DataFrame, one may pass the name of a column to specify sampling weights when sampling from rows. In [24]: df = DataFrame ({ 'col1' :[ 9 , 8 , 7 , 6 ], 'weight_column' :[ 0.5 , 0.4 , 0.1 , 0 ]}) In [25]: df . sample ( n = 3 , weights = 'weight_column' ) Out[25]: col1 weight_column 1 8 0.4 0 9 0.5 2 7 0.1 String Methods Enhancements¶ Continuing from v0.16.0, the following enhancements make string operations easier and more consistent with standard python string operations. Added StringMethods ( .str accessor) to Index (GH9068) The .str accessor is now available for both Series and Index . In [26]: idx = Index ([ ' jack' , 'jill ' , ' jesse ' , 'frank' ]) In [27]: idx . str . strip () Out[27]: Index ([ u'jack' , u'jill' , u'jesse' , u'frank' ], dtype = 'object' ) One special case for the .str accessor on Index is that if a string method returns bool , the .str accessor will return a np.array instead of a boolean Index (GH8875). This enables the following expression to work naturally: In [28]: idx = Index ([ 'a1' , 'a2' , 'b1' , 'b2' ]) In [29]: s = Series ( range ( 4 ), index = idx ) In [30]: s Out[30]: a1 0 a2 1 b1 2 b2 3 dtype: int64 In [31]: idx . str . startswith ( 'a' ) Out[31]: array ([ True , True , False , False ], dtype = bool ) In [32]: s [ s . index . str . startswith ( 'a' )] Out[32]: a1 0 a2 1 dtype: int64

The following new methods are accesible via .str accessor to apply the function to each values. (GH9766, GH9773, GH10031, GH10045, GH10052) Methods capitalize() swapcase() normalize() partition() rpartition() index() rindex() translate()

split now takes expand keyword to specify whether to expand dimensionality. return_type is deprecated. (GH9847) In [33]: s = Series ([ 'a,b' , 'a,c' , 'b,c' ]) # return Series In [34]: s . str . split ( ',' ) Out[34]: 0 [a, b] 1 [a, c] 2 [b, c] dtype: object # return DataFrame In [35]: s . str . split ( ',' , expand = True ) Out[35]: 0 1 0 a b 1 a c 2 b c In [36]: idx = Index ([ 'a,b' , 'a,c' , 'b,c' ]) # return Index In [37]: idx . str . split ( ',' ) Out[37]: Index ([[ u'a' , u'b' ], [ u'a' , u'c' ], [ u'b' , u'c' ]], dtype = 'object' ) # return MultiIndex In [38]: idx . str . split ( ',' , expand = True ) Out[38]: MultiIndex(levels=[[u'a', u'b'], [u'b', u'c']], labels=[[0, 0, 1], [0, 1, 1]])

Improved extract and get_dummies methods for Index.str (GH9980) Other Enhancements¶ BusinessHour offset is now supported, which represents business hours starting from 09:00 - 17:00 on BusinessDay by default. See Here for details. (GH7905) In [39]: from pandas.tseries.offsets import BusinessHour In [40]: Timestamp ( '2014-08-01 09:00' ) + BusinessHour () Out[40]: Timestamp ( '2014-08-01 10:00:00' ) In [41]: Timestamp ( '2014-08-01 07:00' ) + BusinessHour () Out[41]: Timestamp ( '2014-08-01 10:00:00' ) In [42]: Timestamp ( '2014-08-01 16:30' ) + BusinessHour () Out[42]: Timestamp ( '2014-08-04 09:30:00' )

DataFrame.diff now takes an axis parameter that determines the direction of differencing (GH9727)

Allow clip , clip_lower , and clip_upper to accept array-like arguments as thresholds (This is a regression from 0.11.0). These methods now have an axis parameter which determines how the Series or DataFrame will be aligned with the threshold(s). (GH6966)

DataFrame.mask() and Series.mask() now support same keywords as where (GH8801)

drop function can now accept errors keyword to suppress ValueError raised when any of label does not exist in the target data. (GH6736) In [43]: df = DataFrame ( np . random . randn ( 3 , 3 ), columns = [ 'A' , 'B' , 'C' ]) In [44]: df . drop ([ 'A' , 'X' ], axis = 1 , errors = 'ignore' ) Out[44]: B C 0 0.991946 0.953324 1 -0.334077 0.002118 2 0.289092 1.321158

Add support for separating years and quarters using dashes, for example 2014-Q1. (GH9688)

Allow conversion of values with dtype datetime64 or timedelta64 to strings using astype(str) (GH9757)

get_dummies function now accepts sparse keyword. If set to True , the return DataFrame is sparse, e.g. SparseDataFrame . (GH8823)

Period now accepts datetime64 as value input. (GH9054)

Allow timedelta string conversion when leading zero is missing from time definition, ie 0:00:00 vs 00:00:00 . (GH9570)

Allow Panel.shift with axis='items' (GH9890)

Trying to write an excel file now raises NotImplementedError if the DataFrame has a MultiIndex instead of writing a broken Excel file. (GH9794)

Allow Categorical.add_categories to accept Series or np.array . (GH9927)

Add/delete str/dt/cat accessors dynamically from __dir__ . (GH9910)

Add normalize as a dt accessor method. (GH10047)

DataFrame and Series now have _constructor_expanddim property as overridable constructor for one higher dimensionality data. This should be used only when it is really needed, see here

pd.lib.infer_dtype now returns 'bytes' in Python 3 where appropriate. (GH10032) API changes¶ When passing in an ax to df.plot( ..., ax=ax) , the sharex kwarg will now default to False . The result is that the visibility of xlabels and xticklabels will not anymore be changed. You have to do that by yourself for the right axes in your figure or set sharex=True explicitly (but this changes the visible for all axes in the figure, not only the one which is passed in!). If pandas creates the subplots itself (e.g. no passed in ax kwarg), then the default is still sharex=True and the visibility changes are applied.

, the kwarg will now default to . The result is that the visibility of xlabels and xticklabels will not anymore be changed. You have to do that by yourself for the right axes in your figure or set explicitly (but this changes the visible for all axes in the figure, not only the one which is passed in!). If pandas creates the subplots itself (e.g. no passed in kwarg), then the default is still and the visibility changes are applied. assign() now inserts new columns in alphabetical order. Previously the order was arbitrary. (GH9777)

now inserts new columns in alphabetical order. Previously the order was arbitrary. (GH9777) By default, read_csv and read_table will now try to infer the compression type based on the file extension. Set compression=None to restore the previous behavior (no decompression). (GH9770) Deprecations¶ Series.str.split ‘s return_type keyword was removed in favor of expand (GH9847) Index Representation¶ The string representation of Index and its sub-classes have now been unified. These will show a single-line display if there are few values; a wrapped multi-line display for a lot of values (but less than display.max_seq_items ; if lots of items (> display.max_seq_items ) will show a truncated display (the head and tail of the data). The formatting for MultiIndex is unchanges (a multi-line wrapped display). The display width responds to the option display.max_seq_items , which is defaulted to 100. (GH6482) Previous Behavior In [2]: pd.Index(range(4),name='foo') Out[2]: Int64Index([0, 1, 2, 3], dtype='int64') In [3]: pd.Index(range(104),name='foo') Out[3]: Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, ...], dtype='int64') In [4]: pd.date_range('20130101',periods=4,name='foo',tz='US/Eastern') Out[4]: <class 'pandas.tseries.index.DatetimeIndex'> [2013-01-01 00:00:00-05:00, ..., 2013-01-04 00:00:00-05:00] Length: 4, Freq: D, Timezone: US/Eastern In [5]: pd.date_range('20130101',periods=104,name='foo',tz='US/Eastern') Out[5]: <class 'pandas.tseries.index.DatetimeIndex'> [2013-01-01 00:00:00-05:00, ..., 2013-04-14 00:00:00-04:00] Length: 104, Freq: D, Timezone: US/Eastern New Behavior In [45]: pd . set_option ( 'display.width' , 80 ) In [46]: pd . Index ( range ( 4 ), name = 'foo' ) Out[46]: Int64Index ([ 0 , 1 , 2 , 3 ], dtype = 'int64' , name = u'foo' ) In [47]: pd . Index ( range ( 30 ), name = 'foo' ) Out[47]: Int64Index([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], dtype='int64', name=u'foo') In [48]: pd . Index ( range ( 104 ), name = 'foo' ) Out[48]: Int64Index([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ... 94, 95, 96, 97, 98, 99, 100, 101, 102, 103], dtype='int64', name=u'foo', length=104) In [49]: pd . CategoricalIndex ([ 'a' , 'bb' , 'ccc' , 'dddd' ], ordered = True , name = 'foobar' ) Out[49]: CategoricalIndex ([ u'a' , u'bb' , u'ccc' , u'dddd' ], categories = [ u'a' , u'bb' , u'ccc' , u'dddd' ], ordered = True , name = u'foobar' , dtype = 'category' ) In [50]: pd . CategoricalIndex ([ 'a' , 'bb' , 'ccc' , 'dddd' ] * 10 , ordered = True , name = 'foobar' ) Out[50]: CategoricalIndex([u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd'], categories=[u'a', u'bb', u'ccc', u'dddd'], ordered=True, name=u'foobar', dtype='category') In [51]: pd . CategoricalIndex ([ 'a' , 'bb' , 'ccc' , 'dddd' ] * 100 , ordered = True , name = 'foobar' ) Out[51]: CategoricalIndex([u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', ... u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd'], categories=[u'a', u'bb', u'ccc', u'dddd'], ordered=True, name=u'foobar', dtype='category', length=400) In [52]: pd . date_range ( '20130101' , periods = 4 , name = 'foo' , tz = 'US/Eastern' ) Out[52]: DatetimeIndex(['2013-01-01 00:00:00-05:00', '2013-01-02 00:00:00-05:00', '2013-01-03 00:00:00-05:00', '2013-01-04 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', name=u'foo', freq='D') In [53]: pd . date_range ( '20130101' , periods = 25 , freq = 'D' ) Out[53]: DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04', '2013-01-05', '2013-01-06', '2013-01-07', '2013-01-08', '2013-01-09', '2013-01-10', '2013-01-11', '2013-01-12', '2013-01-13', '2013-01-14', '2013-01-15', '2013-01-16', '2013-01-17', '2013-01-18', '2013-01-19', '2013-01-20', '2013-01-21', '2013-01-22', '2013-01-23', '2013-01-24', '2013-01-25'], dtype='datetime64[ns]', freq='D') In [54]: pd . date_range ( '20130101' , periods = 104 , name = 'foo' , tz = 'US/Eastern' ) Out[54]: DatetimeIndex(['2013-01-01 00:00:00-05:00', '2013-01-02 00:00:00-05:00', '2013-01-03 00:00:00-05:00', '2013-01-04 00:00:00-05:00', '2013-01-05 00:00:00-05:00', '2013-01-06 00:00:00-05:00', '2013-01-07 00:00:00-05:00', '2013-01-08 00:00:00-05:00', '2013-01-09 00:00:00-05:00', '2013-01-10 00:00:00-05:00', ... '2013-04-05 00:00:00-04:00', '2013-04-06 00:00:00-04:00', '2013-04-07 00:00:00-04:00', '2013-04-08 00:00:00-04:00', '2013-04-09 00:00:00-04:00', '2013-04-10 00:00:00-04:00', '2013-04-11 00:00:00-04:00', '2013-04-12 00:00:00-04:00', '2013-04-13 00:00:00-04:00', '2013-04-14 00:00:00-04:00'], dtype='datetime64[ns, US/Eastern]', name=u'foo', length=104, freq='D') Performance Improvements¶ Improved csv write performance with mixed dtypes, including datetimes by up to 5x (GH9940)

Improved csv write performance generally by 2x (GH9940)

Improved the performance of pd.lib.max_len_string_array by 5-7x (GH10024) Bug Fixes¶ Bug where labels did not appear properly in the legend of DataFrame.plot() , passing label= arguments works, and Series indices are no longer mutated. (GH9542)

, passing arguments works, and Series indices are no longer mutated. (GH9542) Bug in json serialization causing a segfault when a frame had zero length. (GH9805)

Bug in read_csv where missing trailing delimiters would cause segfault. (GH5664)

where missing trailing delimiters would cause segfault. (GH5664) Bug in retaining index name on appending (GH9862)

Bug in scatter_matrix draws unexpected axis ticklabels (GH5662)

draws unexpected axis ticklabels (GH5662) Fixed bug in StataWriter resulting in changes to input DataFrame upon save (GH9795).

resulting in changes to input upon save (GH9795). Bug in transform causing length mismatch when null entries were present and a fast aggregator was being used (GH9697)

causing length mismatch when null entries were present and a fast aggregator was being used (GH9697) Bug in equals causing false negatives when block order differed (GH9330)

causing false negatives when block order differed (GH9330) Bug in grouping with multiple pd.Grouper where one is non-time based (GH10063)

where one is non-time based (GH10063) Bug in read_sql_table error when reading postgres table with timezone (GH7139)

error when reading postgres table with timezone (GH7139) Bug in DataFrame slicing may not retain metadata (GH9776)

slicing may not retain metadata (GH9776) Bug where TimdeltaIndex were not properly serialized in fixed HDFStore (GH9635)

were not properly serialized in fixed (GH9635) Bug with TimedeltaIndex constructor ignoring name when given another TimedeltaIndex as data (GH10025).

constructor ignoring when given another as data (GH10025). Bug in DataFrameFormatter._get_formatted_index with not applying max_colwidth to the DataFrame index (GH7856)

with not applying to the index (GH7856) Bug in .loc with a read-only ndarray data source (GH10043)

with a read-only ndarray data source (GH10043) Bug in groupby.apply() that would raise if a passed user defined function either returned only None (for all input). (GH9685)

that would raise if a passed user defined function either returned only (for all input). (GH9685) Always use temporary files in pytables tests (GH9992)

Bug in plotting continuously using secondary_y may not show legend properly. (GH9610, GH9779)

may not show legend properly. (GH9610, GH9779) Bug in DataFrame.plot(kind="hist") results in TypeError when DataFrame contains non-numeric columns (GH9853)

results in when contains non-numeric columns (GH9853) Bug where repeated plotting of DataFrame with a DatetimeIndex may raise TypeError (GH9852)

with a may raise (GH9852) Bug in setup.py that would allow an incompat cython version to build (GH9827)

that would allow an incompat cython version to build (GH9827) Bug in plotting secondary_y incorrectly attaches right_ax property to secondary axes specifying itself recursively. (GH9861)

incorrectly attaches property to secondary axes specifying itself recursively. (GH9861) Bug in Series.quantile on empty Series of type Datetime or Timedelta (GH9675)

on empty Series of type or (GH9675) Bug in where causing incorrect results when upcasting was required (GH9731)

causing incorrect results when upcasting was required (GH9731) Bug in FloatArrayFormatter where decision boundary for displaying “small” floats in decimal format is off by one order of magnitude for a given display.precision (GH9764)

where decision boundary for displaying “small” floats in decimal format is off by one order of magnitude for a given display.precision (GH9764) Fixed bug where DataFrame.plot() raised an error when both color and style keywords were passed and there was no color symbol in the style strings (GH9671)

raised an error when both and keywords were passed and there was no color symbol in the style strings (GH9671) Not showing a DeprecationWarning on combining list-likes with an Index (GH10083)

on combining list-likes with an (GH10083) Bug in read_csv and read_table when using skip_rows parameter if blank lines are present. (GH9832)

and when using parameter if blank lines are present. (GH9832) Bug in read_csv() interprets index_col=True as 1 (GH9798)

interprets as (GH9798) Bug in index equality comparisons using == failing on Index/MultiIndex type incompatibility (GH9785)

failing on Index/MultiIndex type incompatibility (GH9785) Bug in which SparseDataFrame could not take nan as a column name (GH8822)

could not take as a column name (GH8822) Bug in to_msgpack and read_msgpack zlib and blosc compression support (GH9783)

and zlib and blosc compression support (GH9783) Bug GroupBy.size doesn’t attach index name properly if grouped by TimeGrouper (GH9925)

doesn’t attach index name properly if grouped by (GH9925) Bug causing an exception in slice assignments because length_of_indexer returns wrong results (GH9995)

returns wrong results (GH9995) Bug in csv parser causing lines with initial whitespace plus one non-space character to be skipped. (GH9710)

Bug in C csv parser causing spurious NaNs when data started with newline followed by whitespace. (GH10022)

Bug causing elements with a null group to spill into the final group when grouping by a Categorical (GH9603)

(GH9603) Bug where .iloc and .loc behavior is not consistent on empty dataframes (GH9964)

Bug in invalid attribute access on a TimedeltaIndex incorrectly raised ValueError instead of AttributeError (GH9680)

incorrectly raised instead of (GH9680) Bug in unequal comparisons between categorical data and a scalar, which was not in the categories (e.g. Series(Categorical(list("abc"), ordered=True)) > "d" . This returned False for all elements, but now raises a TypeError . Equality comparisons also now return False for == and True for != . (GH9848)

. This returned for all elements, but now raises a . Equality comparisons also now return for and for . (GH9848) Bug in DataFrame __setitem__ when right hand side is a dictionary (GH9874)

when right hand side is a dictionary (GH9874) Bug in where when dtype is datetime64/timedelta64 , but dtype of other is not (GH9804)

when dtype is , but dtype of other is not (GH9804) Bug in MultiIndex.sortlevel() results in unicode level name breaks (GH9856)

results in unicode level name breaks (GH9856) Bug in which groupby.transform incorrectly enforced output dtypes to match input dtypes. (GH9807)

incorrectly enforced output dtypes to match input dtypes. (GH9807) Bug in DataFrame constructor when columns parameter is set, and data is an empty list (GH9939)

constructor when parameter is set, and is an empty list (GH9939) Bug in bar plot with log=True raises TypeError if all values are less than 1 (GH9905)

raises if all values are less than 1 (GH9905) Bug in horizontal bar plot ignores log=True (GH9905)

(GH9905) Bug in PyTables queries that did not return proper results using the index (GH8265, GH9676)

Bug where dividing a dataframe containing values of type Decimal by another Decimal would raise. (GH9787)

by another would raise. (GH9787) Bug where using DataFrames asfreq would remove the name of the index. (GH9885)

Bug causing extra index point when resample BM/BQ (GH9756)

Changed caching in AbstractHolidayCalendar to be at the instance level rather than at the class level as the latter can result in unexpected behaviour. (GH9552)

to be at the instance level rather than at the class level as the latter can result in unexpected behaviour. (GH9552) Fixed latex output for multi-indexed dataframes (GH9778)

Bug causing an exception when setting an empty range using DataFrame.loc (GH9596)

(GH9596) Bug in hiding ticklabels with subplots and shared axes when adding a new plot to an existing grid of axes (GH9158)

Bug in transform and filter when grouping on a categorical variable (GH9921)

and when grouping on a categorical variable (GH9921) Bug in transform when groups are equal in number and dtype to the input index (GH9700)

when groups are equal in number and dtype to the input index (GH9700) Google BigQuery connector now imports dependencies on a per-method basis.(GH9713)

Updated BigQuery connector to no longer use deprecated oauth2client.tools.run() (GH8327)

(GH8327) Bug in subclassed DataFrame . It may not return the correct class, when slicing or subsetting it. (GH9632)

. It may not return the correct class, when slicing or subsetting it. (GH9632) Bug in .median() where non-float null values are not handled correctly (GH10040)

where non-float null values are not handled correctly (GH10040) Bug in Series.fillna() where it raises if a numerically convertible string is given (GH10092)

v0.15.2 (December 12, 2014)¶ This is a minor release from 0.15.1 and includes a large number of bug fixes along with several new features, enhancements, and performance improvements. A small number of API changes were necessary to fix existing bugs. We recommend that all users upgrade to this version. Enhancements

API Changes

Performance Improvements

Bug Fixes API changes¶ Indexing in MultiIndex beyond lex-sort depth is now supported, though a lexically sorted index will have a better performance. (GH2646) In [1]: df = pd . DataFrame ({ 'jim' :[ 0 , 0 , 1 , 1 ], ...: 'joe' :[ 'x' , 'x' , 'z' , 'y' ], ...: 'jolie' : np . random . rand ( 4 )}) . set_index ([ 'jim' , 'joe' ]) ...: In [2]: df Out[2]: jolie jim joe 0 x 0.043324 x 0.561433 1 z 0.329668 y 0.502967 In [3]: df . index . lexsort_depth Out[3]: 1 # in prior versions this would raise a KeyError # will now show a PerformanceWarning In [4]: df . loc [( 1 , 'z' )] Out[4]: jolie jim joe 1 z 0.329668 # lexically sorting In [5]: df2 = df . sortlevel () In [6]: df2 Out[6]: jolie jim joe 0 x 0.043324 x 0.561433 1 y 0.502967 z 0.329668 In [7]: df2 . index . lexsort_depth Out[7]: 2 In [8]: df2 . loc [( 1 , 'z' )] Out[8]: jolie jim joe 1 z 0.329668

Bug in unique of Series with category dtype, which returned all categories regardless whether they were “used” or not (see GH8559 for the discussion). Previous behaviour was to return all categories: In [ 3 ]: cat = pd . Categorical ([ 'a' , 'b' , 'a' ], categories = [ 'a' , 'b' , 'c' ]) In [ 4 ]: cat Out [ 4 ]: [ a , b , a ] Categories ( 3 , object ): [ a < b < c ] In [ 5 ]: cat . unique () Out [ 5 ]: array ([ 'a' , 'b' , 'c' ], dtype = object ) Now, only the categories that do effectively occur in the array are returned: In [9]: cat = pd . Categorical ([ 'a' , 'b' , 'a' ], categories = [ 'a' , 'b' , 'c' ]) In [10]: cat . unique () Out[10]: [a, b] Categories (2, object): [a, b]

Series.all and Series.any now support the level and skipna parameters. Series.all , Series.any , Index.all , and Index.any no longer support the out and keepdims parameters, which existed for compatibility with ndarray. Various index types no longer support the all and any aggregation functions and will now raise TypeError . (GH8302).

Allow equality comparisons of Series with a categorical dtype and object dtype; previously these would raise TypeError (GH8938)

Bug in NDFrame : conflicting attribute/column names now behave consistently between getting and setting. Previously, when both a column and attribute named y existed, data.y would return the attribute, while data.y = z would update the column (GH8994) In [11]: data = pd . DataFrame ({ 'x' :[ 1 , 2 , 3 ]}) In [12]: data . y = 2 In [13]: data [ 'y' ] = [ 2 , 4 , 6 ] In [14]: data Out[14]: x y 0 1 2 1 2 4 2 3 6 # this assignment was inconsistent In [15]: data . y = 5 Old behavior: In [ 6 ]: data . y Out [ 6 ]: 2 In [ 7 ]: data [ 'y' ] . values Out [ 7 ]: array ([ 5 , 5 , 5 ]) New behavior: In [16]: data . y Out[16]: 5 In [17]: data [ 'y' ] . values Out[17]: array ([ 2 , 4 , 6 ])

Timestamp('now') is now equivalent to Timestamp.now() in that it returns the local time rather than UTC. Also, Timestamp('today') is now equivalent to Timestamp.today() and both have tz as a possible argument. (GH9000)

Fix negative step support for label-based slices (GH8753) Old behavior: In [ 1 ]: s = pd . Series ( np . arange ( 3 ), [ 'a' , 'b' , 'c' ]) Out [ 1 ]: a 0 b 1 c 2 dtype : int64 In [ 2 ]: s . loc [ 'c' : 'a' : - 1 ] Out [ 2 ]: c 2 dtype : int64 New behavior: In [18]: s = pd . Series ( np . arange ( 3 ), [ 'a' , 'b' , 'c' ]) In [19]: s . loc [ 'c' : 'a' : - 1 ] Out[19]: c 2 b 1 a 0 dtype: int64 Enhancements¶ Categorical enhancements: Added ability to export Categorical data to Stata (GH8633). See here for limitations of categorical variables exported to Stata data files.

Added flag order_categoricals to StataReader and read_stata to select whether to order imported categorical data (GH8836). See here for more information on importing categorical variables from Stata data files.

to and to select whether to order imported categorical data (GH8836). See here for more information on importing categorical variables from Stata data files. Added ability to export Categorical data to to/from HDF5 (GH7621). Queries work the same as if it was an object array. However, the category dtyped data is stored in a more efficient manner. See here for an example and caveats w.r.t. prior versions of pandas.

dtyped data is stored in a more efficient manner. See here for an example and caveats w.r.t. prior versions of pandas. Added support for searchsorted() on Categorical class (GH8420). Other enhancements: Added the ability to specify the SQL type of columns when writing a DataFrame to a database (GH8778). For example, specifying to use the sqlalchemy String type instead of the default Text type for string columns: from sqlalchemy.types import String data . to_sql ( 'data_dtype' , engine , dtype = { 'Col_1' : String })

Series.all and Series.any now support the level and skipna parameters (GH8302): In [20]: s = pd . Series ([ False , True , False ], index = [ 0 , 0 , 1 ]) In [21]: s . any ( level = 0 ) Out[21]: 0 True 1 False dtype: bool

Panel now supports the all and any aggregation functions. (GH8302): In [22]: p = pd . Panel ( np . random . rand ( 2 , 5 , 4 ) > 0.1 ) In [23]: p . all () Out[23]: 0 1 0 True False 1 True True 2 True True 3 False True

Added support for utcfromtimestamp() , fromtimestamp() , and combine() on Timestamp class (GH5351).

Added Google Analytics ( pandas.io.ga ) basic documentation (GH8835). See here.

Timedelta arithmetic returns NotImplemented in unknown cases, allowing extensions by custom classes (GH8813).

Timedelta now supports arithemtic with numpy.ndarray objects of the appropriate dtype (numpy 1.8 or newer only) (GH8884).

Added Timedelta.to_timedelta64() method to the public API (GH8884).

Added gbq.generate_bq_schema() function to the gbq module (GH8325).

Series now works with map objects the same way as generators (GH8909).

Added context manager to HDFStore for automatic closing (GH8791).

to_datetime gains an exact keyword to allow for a format to not require an exact match for a provided format string (if its False ). exact defaults to True (meaning that exact matching is still the default) (GH8904)

Added axvlines boolean option to parallel_coordinates plot function, determines whether vertical lines will be printed, default is True

Added ability to read table footers to read_html (GH8552)

to_sql now infers datatypes of non-NA values for columns that contain NA values and have dtype object (GH8778). Performance¶ Reduce memory usage when skiprows is an integer in read_csv (GH8681)

Performance boost for to_datetime conversions with a passed format= , and the exact=False (GH8904) Bug Fixes¶ Bug in concat of Series with category dtype which were coercing to object . (GH8641)

dtype which were coercing to . (GH8641) Bug in Timestamp-Timestamp not returning a Timedelta type and datelike-datelike ops with timezones (GH8865)

Made consistent a timezone mismatch exception (either tz operated with None or incompatible timezone), will now return TypeError rather than ValueError (a couple of edge cases only), (GH8865)

rather than (a couple of edge cases only), (GH8865) Bug in using a pd.Grouper(key=...) with no level/axis or level only (GH8795, GH8866)

with no level/axis or level only (GH8795, GH8866) Report a TypeError when invalid/no parameters are passed in a groupby (GH8015)

when invalid/no parameters are passed in a groupby (GH8015) Bug in packaging pandas with py2app/cx_Freeze (GH8602, GH8831)

(GH8602, GH8831) Bug in groupby signatures that didn’t include *args or **kwargs (GH8733).

signatures that didn’t include *args or **kwargs (GH8733). io.data.Options now raises RemoteDataError when no expiry dates are available from Yahoo and when it receives no data from Yahoo (GH8761), (GH8783).

now raises when no expiry dates are available from Yahoo and when it receives no data from Yahoo (GH8761), (GH8783). Unclear error message in csv parsing when passing dtype and names and the parsed data is a different data type (GH8833)

Bug in slicing a multi-index with an empty list and at least one boolean indexer (GH8781)

io.data.Options now raises RemoteDataError when no expiry dates are available from Yahoo (GH8761).

now raises when no expiry dates are available from Yahoo (GH8761). Timedelta kwargs may now be numpy ints and floats (GH8757).

kwargs may now be numpy ints and floats (GH8757). Fixed several outstanding bugs for Timedelta arithmetic and comparisons (GH8813, GH5963, GH5436).

arithmetic and comparisons (GH8813, GH5963, GH5436). sql_schema now generates dialect appropriate CREATE TABLE statements (GH8697)

now generates dialect appropriate statements (GH8697) slice string method now takes step into account (GH8754)

string method now takes step into account (GH8754) Bug in BlockManager where setting values with different type would break block integrity (GH8850)

where setting values with different type would break block integrity (GH8850) Bug in DatetimeIndex when using time object as key (GH8667)

when using object as key (GH8667) Bug in merge where how='left' and sort=False would not preserve left frame order (GH7331)

where and would not preserve left frame order (GH7331) Bug in MultiIndex.reindex where reindexing at level would not reorder labels (GH4088)

where reindexing at level would not reorder labels (GH4088) Bug in certain operations with dateutil timezones, manifesting with dateutil 2.3 (GH8639)

Regression in DatetimeIndex iteration with a Fixed/Local offset timezone (GH8890)

Bug in to_datetime when parsing a nanoseconds using the %f format (GH8989)

when parsing a nanoseconds using the format (GH8989) io.data.Options now raises RemoteDataError when no expiry dates are available from Yahoo and when it receives no data from Yahoo (GH8761), (GH8783).

now raises when no expiry dates are available from Yahoo and when it receives no data from Yahoo (GH8761), (GH8783). Fix: The font size was only set on x axis if vertical or the y axis if horizontal. (GH8765)

Fixed division by 0 when reading big csv files in python 3 (GH8621)

Bug in outputing a Multindex with to_html,index=False which would add an extra column (GH8452)

which would add an extra column (GH8452) Imported categorical variables from Stata files retain the ordinal information in the underlying data (GH8836).

Defined .size attribute across NDFrame objects to provide compat with numpy >= 1.9.1; buggy with np.array_split (GH8846)

attribute across objects to provide compat with numpy >= 1.9.1; buggy with (GH8846) Skip testing of histogram plots for matplotlib <= 1.2 (GH8648).

Bug where get_data_google returned object dtypes (GH3995)

returned object dtypes (GH3995) Bug in DataFrame.stack(..., dropna=False) when the DataFrame’s columns is a MultiIndex whose labels do not reference all its levels . (GH8844)

when the DataFrame’s is a whose do not reference all its . (GH8844) Bug in that Option context applied on __enter__ (GH8514)

(GH8514) Bug in resample that causes a ValueError when resampling across multiple days and the last offset is not calculated from the start of the range (GH8683)

Bug where DataFrame.plot(kind='scatter') fails when checking if an np.array is in the DataFrame (GH8852)

fails when checking if an np.array is in the DataFrame (GH8852) Bug in pd.infer_freq/DataFrame.inferred_freq that prevented proper sub-daily frequency inference when the index contained DST days (GH8772).

that prevented proper sub-daily frequency inference when the index contained DST days (GH8772). Bug where index name was still used when plotting a series with use_index=False (GH8558).

(GH8558). Bugs when trying to stack multiple columns, when some (or all) of the level names are numbers (GH8584).

Bug in MultiIndex where __contains__ returns wrong result if index is not lexically sorted or unique (GH7724)

where returns wrong result if index is not lexically sorted or unique (GH7724) BUG CSV: fix problem with trailing whitespace in skipped rows, (GH8679), (GH8661), (GH8983)

Regression in Timestamp does not parse ‘Z’ zone designator for UTC (GH8771)

does not parse ‘Z’ zone designator for UTC (GH8771) Bug in StataWriter the produces writes strings with 244 characters irrespective of actual size (GH8969)

the produces writes strings with 244 characters irrespective of actual size (GH8969) Fixed ValueError raised by cummin/cummax when datetime64 Series contains NaT. (GH8965)

Bug in Datareader returns object dtype if there are missing values (GH8980)

Bug in plotting if sharex was enabled and index was a timeseries, would show labels on multiple axes (GH3964).

Bug where passing a unit to the TimedeltaIndex constructor applied the to nano-second conversion twice. (GH9011).

Bug in plotting of a period-like array (GH9012)

v0.15.1 (November 9, 2014)¶ This is a minor bug-fix release from 0.15.0 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. Enhancements

API Changes

Bug Fixes API changes¶ s.dt.hour and other .dt accessors will now return np.nan for missing values (rather than previously -1), (GH8689) In [1]: s = Series ( date_range ( '20130101' , periods = 5 , freq = 'D' )) In [2]: s . iloc [ 2 ] = np . nan In [3]: s Out[3]: 0 2013-01-01 1 2013-01-02 2 NaT 3 2013-01-04 4 2013-01-05 dtype: datetime64[ns] previous behavior: In [ 6 ]: s . dt . hour Out [ 6 ]: 0 0 1 0 2 - 1 3 0 4 0 dtype : int64 current behavior: In [4]: s . dt . hour Out[4]: 0 0.0 1 0.0 2 NaN 3 0.0 4 0.0 dtype: float64

groupby with as_index=False will not add erroneous extra columns to result (GH8582): In [5]: np . random . seed ( 2718281 ) In [6]: df = pd . DataFrame ( np . random . randint ( 0 , 100 , ( 10 , 2 )), ...: columns = [ 'jim' , 'joe' ]) ...: In [7]: df . head () Out[7]: jim joe 0 61 81 1 96 49 2 55 65 3 72 51 4 77 12 In [8]: ts = pd . Series ( 5 * np . random . randint ( 0 , 3 , 10 )) previous behavior: In [ 4 ]: df . groupby ( ts , as_index = False ) . max () Out [ 4 ]: NaN jim joe 0 0 72 83 1 5 77 84 2 10 96 65 current behavior: In [9]: df . groupby ( ts , as_index = False ) . max () Out[9]: jim joe 0 72 83 1 77 84 2 96 65

groupby will not erroneously exclude columns if the column name conflics with the grouper name (GH8112): In [10]: df = pd . DataFrame ({ 'jim' : range ( 5 ), 'joe' : range ( 5 , 10 )}) In [11]: df Out[11]: jim joe 0 0 5 1 1 6 2 2 7 3 3 8 4 4 9 In [12]: gr = df . groupby ( df [ 'jim' ] < 2 ) previous behavior (excludes 1st column from output): In [ 4 ]: gr . apply ( sum ) Out [ 4 ]: joe jim False 24 True 11 current behavior: In [13]: gr . apply ( sum ) Out[13]: jim joe jim False 9 24 True 1 11

Support for slicing with monotonic decreasing indexes, even if start or stop is not found in the index (GH7860): In [14]: s = pd . Series ([ 'a' , 'b' , 'c' , 'd' ], [ 4 , 3 , 2 , 1 ]) In [15]: s Out[15]: 4 a 3 b 2 c 1 d dtype: object previous behavior: In [ 8 ]: s . loc [ 3.5 : 1.5 ] KeyError : 3.5 current behavior: In [16]: s . loc [ 3.5 : 1.5 ] Out[16]: 3 b 2 c dtype: object

io.data.Options has been fixed for a change in the format of the Yahoo Options page (GH8612), (GH8741) Note As a result of a change in Yahoo’s option page layout, when an expiry date is given, Options methods now return data for a single expiry date. Previously, methods returned all data for the selected month. The month and year parameters have been undeprecated and can be used to get all options data for a given month. If an expiry date that is not valid is given, data for the next expiry after the given date is returned. Option data frames are now saved on the instance as callsYYMMDD or putsYYMMDD . Previously they were saved as callsMMYY and putsMMYY . The next expiry is saved as calls and puts . New features: The expiry parameter can now be a single date or a list-like object containing dates. A new property expiry_dates was added, which returns all available expiry dates. Current behavior: In [17]: from pandas.io.data import Options In [18]: aapl = Options ( 'aapl' , 'yahoo' ) In [19]: aapl . get_call_data () . iloc [ 0 : 5 , 0 : 1 ] Out[19]: Last Strike Expiry Type Symbol 50.0 2016-03-18 call AAPL160318C00050000 46.75 55.0 2016-03-18 call AAPL160318C00055000 45.40 60.0 2016-03-18 call AAPL160318C00060000 44.82 65.0 2016-03-18 call AAPL160318C00065000 28.75 70.0 2016-03-18 call AAPL160318C00070000 32.40 In [20]: aapl . expiry_dates Out[20]: [datetime.date(2016, 3, 18), datetime.date(2016, 3, 24), datetime.date(2016, 4, 1), datetime.date(2016, 4, 8), datetime.date(2016, 4, 15), datetime.date(2016, 4, 22), datetime.date(2016, 4, 29), datetime.date(2016, 5, 20), datetime.date(2016, 6, 17), datetime.date(2016, 7, 15), datetime.date(2016, 10, 21), datetime.date(2017, 1, 20), datetime.date(2017, 6, 16), datetime.date(2018, 1, 19)] In [21]: aapl . get_near_stock_price ( expiry = aapl . expiry_dates [ 0 : 3 ]) . iloc [ 0 : 5 , 0 : 1 ] Out[21]: Last Strike Expiry Type Symbol 105.0 2016-03-24 call AAPL160324C00105000 1.79 2016-04-01 call AAPL160401C00105000 2.23 106.0 2016-03-18 call AAPL160318C00106000 0.37 2016-03-24 call AAPL160324C00106000 1.25 2016-04-01 call AAPL160401C00106000 1.67 See the Options documentation in Remote Data

Enhancements¶ concat permits a wider variety of iterables of pandas objects to be passed as the first parameter (GH8645): In [22]: from collections import deque In [23]: df1 = pd . DataFrame ([ 1 , 2 , 3 ]) In [24]: df2 = pd . DataFrame ([ 4 , 5 , 6 ]) previous behavior: In [ 7 ]: pd . concat ( deque (( df1 , df2 ))) TypeError : first argument must be a list - like of pandas objects , you passed an object of type "deque" current behavior: In [25]: pd . concat ( deque (( df1 , df2 ))) Out[25]: 0 0 1 1 2 2 3 0 4 1 5 2 6

Represent MultiIndex labels with a dtype that utilizes memory based on the level size. In prior versions, the memory usage was a constant 8 bytes per element in each level. In addition, in prior versions, the reported memory usage was incorrect as it didn’t show the usage for the memory occupied by the underling data array. (GH8456) In [26]: dfi = DataFrame ( 1 , index = pd . MultiIndex . from_product ([[ 'a' ], range ( 1000 )]), columns = [ 'A' ]) previous behavior: # this was underreported in prior versions In [ 1 ]: dfi . memory_usage ( index = True ) Out [ 1 ]: Index 8000 # took about 24008 bytes in < 0.15.1 A 8000 dtype : int64 current behavior: In [27]: dfi . memory_usage ( index = True ) Out[27]: Index 8000 A 8000 dtype: int64

Added Index properties is_monotonic_increasing and is_monotonic_decreasing (GH8680).

Added option to select columns when importing Stata files (GH7935)

Qualify memory usage in DataFrame.info() by adding + if it is a lower bound (GH8578)

Raise errors in certain aggregation cases where an argument such as numeric_only is not handled (GH8592).

Added support for 3-character ISO and non-standard country codes in io.wb.download() (GH8482)

World Bank data requests now will warn/raise based on an errors argument, as well as a list of hard-coded country codes and the World Bank’s JSON response. In prior versions, the error messages didn’t look at the World Bank’s JSON response. Problem-inducing input were simply dropped prior to the request. The issue was that many good countries were cropped in the hard-coded approach. All countries will work now, but some bad countries will raise exceptions because some edge cases break the entire response. (GH8482)

Added option to Series.str.split() to return a DataFrame rather than a Series (GH8428)

Added option to df.info(null_counts=None|True|False) to override the default display options and force showing of the null-counts (GH8701) Bug Fixes¶ Bug in unpickling of a CustomBusinessDay object (GH8591)

object (GH8591) Bug in coercing Categorical to a records array, e.g. df.to_records() (GH8626)

to a records array, e.g. (GH8626) Bug in Categorical not created properly with Series.to_frame() (GH8626)

not created properly with (GH8626) Bug in coercing in astype of a Categorical of a passed pd.Categorical (this now raises TypeError correctly), (GH8626)

of a passed (this now raises correctly), (GH8626) Bug in cut / qcut when using Series and retbins=True (GH8589)

/ when using and (GH8589) Bug in writing Categorical columns to an SQL database with to_sql (GH8624).

(GH8624). Bug in comparing Categorical of datetime raising when being compared to a scalar datetime (GH8687)

of datetime raising when being compared to a scalar datetime (GH8687) Bug in selecting from a Categorical with .iloc (GH8623)

with (GH8623) Bug in groupby-transform with a Categorical (GH8623)

Bug in duplicated/drop_duplicates with a Categorical (GH8623)

Bug in Categorical reflected comparison operator raising if the first argument was a numpy array scalar (e.g. np.int64) (GH8658)

reflected comparison operator raising if the first argument was a numpy array scalar (e.g. np.int64) (GH8658) Bug in Panel indexing with a list-like (GH8710)

Compat issue is DataFrame.dtypes when options.mode.use_inf_as_null is True (GH8722)

when is True (GH8722) Bug in read_csv , dialect parameter would not take a string (:issue: 8703 )

, parameter would not take a string (:issue: ) Bug in slicing a multi-index level with an empty-list (GH8737)

Bug in numeric index operations of add/sub with Float/Index Index with numpy arrays (GH8608)

Bug in setitem with empty indexer and unwanted coercion of dtypes (GH8669)

Bug in ix/loc block splitting on setitem (manifests with integer-like dtypes, e.g. datetime64) (GH8607)

Bug when doing label based indexing with integers not found in the index for non-unique but monotonic indexes (GH8680).

Bug when indexing a Float64Index with np.nan on numpy 1.7 (GH8980).

on numpy 1.7 (GH8980). Fix shape attribute for MultiIndex (GH8609)

attribute for (GH8609) Bug in GroupBy where a name conflict between the grouper and columns would break groupby operations (GH7115, GH8112)

where a name conflict between the grouper and columns would break operations (GH7115, GH8112) Fixed a bug where plotting a column y and specifying a label would mutate the index name of the original DataFrame (GH8494)

and specifying a label would mutate the index name of the original DataFrame (GH8494) Fix regression in plotting of a DatetimeIndex directly with matplotlib (GH8614).

Bug in date_range where partially-specified dates would incorporate current date (GH6961)

where partially-specified dates would incorporate current date (GH6961) Bug in Setting by indexer to a scalar value with a mixed-dtype Panel4d was failing (GH8702)

was failing (GH8702) Bug where DataReader ‘s would fail if one of the symbols passed was invalid. Now returns data for valid symbols and np.nan for invalid (GH8494)

‘s would fail if one of the symbols passed was invalid. Now returns data for valid symbols and np.nan for invalid (GH8494) Bug in get_quote_yahoo that wouldn’t allow non-float return values (GH5229).

v0.14.1 (July 11, 2014)¶ This is a minor release from 0.14.0 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. Highlights include: New methods select_dtypes() to select columns based on the dtype and sem() to calculate the standard error of the mean. Support for dateutil timezones (see docs). Support for ignoring full line comments in the read_csv() text parser. New documentation section on Options and Settings. Lots of bug fixes.

Enhancements

API Changes

Performance Improvements

Experimental Changes

Bug Fixes API changes¶ Openpyxl now raises a ValueError on construction of the openpyxl writer instead of warning on pandas import (GH7284).

For StringMethods.extract , when no match is found, the result - only containing NaN values - now also has dtype=object instead of float (GH7242)

Period objects no longer raise a TypeError when compared using == with another object that isn’t a Period . Instead when comparing a Period with another object using == if the other object isn’t a Period False is returned. (GH7376)

Previously, the behaviour on resetting the time or not in offsets.apply , rollforward and rollback operations differed between offsets. With the support of the normalize keyword for all offsets(see below) with a default value of False (preserve time), the behaviour changed for certain offsets (BusinessMonthBegin, MonthEnd, BusinessMonthEnd, CustomBusinessMonthEnd, BusinessYearBegin, LastWeekOfMonth, FY5253Quarter, LastWeekOfMonth, Easter): In [ 6 ]: from pandas.tseries import offsets In [ 7 ]: d = pd . Timestamp ( '2014-01-01 09:00' ) # old behaviour < 0.14.1 In [ 8 ]: d + offsets . MonthEnd () Out [ 8 ]: Timestamp ( '2014-01-31 00:00:00' ) Starting from 0.14.1 all offsets preserve time by default. The old behaviour can be obtained with normalize=True # new behaviour In [1]: d + offsets . MonthEnd () Out[1]: Timestamp ( '2014-01-31 09:00:00' ) In [2]: d + offsets . MonthEnd ( normalize = True ) Out[2]: Timestamp ( '2014-01-31 00:00:00' ) Note that for the other offsets the default behaviour did not change.

Add back #N/A N/A as a default NA value in text parsing, (regresion from 0.12) (GH5521)

Raise a TypeError on inplace-setting with a .where and a non np.nan value as this is inconsistent with a set-item expression like df[mask] = None (GH7656) Enhancements¶ Add dropna argument to value_counts and nunique (GH5569).

Add select_dtypes() method to allow selection of columns based on dtype (GH7316). See the docs.

All offsets suppports the normalize keyword to specify whether offsets.apply , rollforward and rollback resets the time (hour, minute, etc) or not (default False , preserves time) (GH7156): In [3]: import pandas.tseries.offsets as offsets In [4]: day = offsets . Day () In [5]: day . apply ( Timestamp ( '2014-01-01 09:00' )) Out[5]: Timestamp ( '2014-01-02 09:00:00' ) In [6]: day = offsets . Day ( normalize = True ) In [7]: day . apply ( Timestamp ( '2014-01-01 09:00' )) Out[7]: Timestamp ( '2014-01-02 00:00:00' )

PeriodIndex is represented as the same format as DatetimeIndex (GH7601)

StringMethods now work on empty Series (GH7242)

The file parsers read_csv and read_table now ignore line comments provided by the parameter comment , which accepts only a single character for the C reader. In particular, they allow for comments before file data begins (GH2685)

Add NotImplementedError for simultaneous use of chunksize and nrows for read_csv() (GH6774).

Tests for basic reading of public S3 buckets now exist (GH7281).

read_html now sports an encoding argument that is passed to the underlying parser library. You can use this to read non-ascii encoded web pages (GH7323).

read_excel now supports reading from URLs in the same way that read_csv does. (GH6809)

Support for dateutil timezones, which can now be used in the same way as pytz timezones across pandas. (GH4688) In [8]: rng = date_range ( '3/6/2012 00:00' , periods = 10 , freq = 'D' , ...: tz = 'dateutil/Europe/London' ) ...: In [9]: rng . tz Out[9]: tzfile ( '/usr/share/zoneinfo/Europe/London' ) See the docs.

Implemented sem (standard error of the mean) operation for Series , DataFrame , Panel , and Groupby (GH6897)

Add nlargest and nsmallest to the Series groupby whitelist, which means you can now use these methods on a SeriesGroupBy object (GH7053).

All offsets apply , rollforward and rollback can now handle np.datetime64 , previously results in ApplyTypeError (GH7452)

Period and PeriodIndex can contain NaT in its values (GH7485)

Support pickling Series , DataFrame and Panel objects with non-unique labels along item axis ( index , columns and items respectively) (GH7370).

Improved inference of datetime/timedelta with mixed null objects. Regression from 0.13.1 in interpretation of an object Index with all null elements (GH7431) Performance¶ Improvements in dtype inference for numeric operations involving yielding performance gains for dtypes: int64 , timedelta64 , datetime64 (GH7223)

, , (GH7223) Improvements in Series.transform for significant performance gains (GH6496)

Improvements in DataFrame.transform with ufuncs and built-in grouper functions for signifcant performance gains (GH7383)

Regression in groupby aggregation of datetime64 dtypes (GH7555)

Improvements in MultiIndex.from_product for large iterables (GH7627) Experimental¶ pandas.io.data.Options has a new method, get_all_data method, and now consistently returns a multi-indexed DataFrame , see the docs. (GH5602)

has a new method, method, and now consistently returns a multi-indexed , see the docs. (GH5602) io.gbq.read_gbq and io.gbq.to_gbq were refactored to remove the dependency on the Google bq.py command line client. This submodule now uses httplib2 and the Google apiclient and oauth2client API client libraries which should be more stable and, therefore, reliable than bq.py . See the docs. (GH6937). Bug Fixes¶ Bug in DataFrame.where with a symmetric shaped frame and a passed other of a DataFrame (GH7506)

with a symmetric shaped frame and a passed other of a DataFrame (GH7506) Bug in Panel indexing with a multi-index axis (GH7516)

Regression in datetimelike slice indexing with a duplicated index and non-exact end-points (GH7523)

Bug in setitem with list-of-lists and single vs mixed types (GH7551:)

Bug in timeops with non-aligned Series (GH7500)

Bug in timedelta inference when assigning an incomplete Series (GH7592)

Bug in groupby .nth with a Series and integer-like column name (GH7559)

with a Series and integer-like column name (GH7559) Bug in Series.get with a boolean accessor (GH7407)

with a boolean accessor (GH7407) Bug in value_counts where NaT did not qualify as missing ( NaN ) (GH7423)

where did not qualify as missing ( ) (GH7423) Bug in to_timedelta that accepted invalid units and misinterpreted ‘m/h’ (GH7611, GH6423)

that accepted invalid units and misinterpreted ‘m/h’ (GH7611, GH6423) Bug in line plot doesn’t set correct xlim if secondary_y=True (GH7459)

if (GH7459) Bug in grouped hist and scatter plots use old figsize default (GH7394)

and plots use old default (GH7394) Bug in plotting subplots with DataFrame.plot , hist clears passed ax even if the number of subplots is one (GH7391).

, clears passed even if the number of subplots is one (GH7391). Bug in plotting subplots with DataFrame.boxplot with by kw raises ValueError if the number of subplots exceeds 1 (GH7391).

with kw raises if the number of subplots exceeds 1 (GH7391). Bug in subplots displays ticklabels and labels in different rule (GH5897)

and in different rule (GH5897) Bug in Panel.apply with a multi-index as an axis (GH7469)

with a multi-index as an axis (GH7469) Bug in DatetimeIndex.insert doesn’t preserve name and tz (GH7299)

doesn’t preserve and (GH7299) Bug in DatetimeIndex.asobject doesn’t preserve name (GH7299)

doesn’t preserve (GH7299) Bug in multi-index slicing with datetimelike ranges (strings and Timestamps), (GH7429)

Bug in Index.min and max doesn’t handle nan and NaT properly (GH7261)

and doesn’t handle and properly (GH7261) Bug in PeriodIndex.min/max results in int (GH7609)

results in (GH7609) Bug in resample where fill_method was ignored if you passed how (GH2073)

where was ignored if you passed (GH2073) Bug in TimeGrouper doesn’t exclude column specified by key (GH7227)

doesn’t exclude column specified by (GH7227) Bug in DataFrame and Series bar and barh plot raises TypeError when bottom and left keyword is specified (GH7226)

and bar and barh plot raises when and keyword is specified (GH7226) Bug in DataFrame.hist raises TypeError when it contains non numeric column (GH7277)

raises when it contains non numeric column (GH7277) Bug in Index.delete does not preserve name and freq attributes (GH7302)

does not preserve and attributes (GH7302) Bug in DataFrame.query() / eval where local string variables with the @ sign were being treated as temporaries attempting to be deleted (GH7300).

/ where local string variables with the @ sign were being treated as temporaries attempting to be deleted (GH7300). Bug in Float64Index which didn’t allow duplicates (GH7149).

which didn’t allow duplicates (GH7149). Bug in DataFrame.replace() where truthy values were being replaced (GH7140).

where truthy values were being replaced (GH7140). Bug in StringMethods.extract() where a single match group Series would use the matcher’s name instead of the group name (GH7313).

where a single match group Series would use the matcher’s name instead of the group name (GH7313). Bug in isnull() when mode.use_inf_as_null == True where isnull wouldn’t test True when it encountered an inf / -inf (GH7315).

when where isnull wouldn’t test when it encountered an / (GH7315). Bug in inferred_freq results in None for eastern hemisphere timezones (GH7310)

Bug in Easter returns incorrect date when offset is negative (GH7195)

returns incorrect date when offset is negative (GH7195) Bug in broadcasting with .div , integer dtypes and divide-by-zero (GH7325)

, integer dtypes and divide-by-zero (GH7325) Bug in CustomBusinessDay.apply raiases NameError when np.datetime64 object is passed (GH7196)

raiases when object is passed (GH7196) Bug in MultiIndex.append , concat and pivot_table don’t preserve timezone (GH6606)

, and don’t preserve timezone (GH6606) Bug in .loc with a list of indexers on a single-multi index level (that is not nested) (GH7349)

with a list of indexers on a single-multi index level (that is not nested) (GH7349) Bug in Series.map when mapping a dict with tuple keys of different lengths (GH7333)

when mapping a dict with tuple keys of different lengths (GH7333) Bug all StringMethods now work on empty Series (GH7242)

now work on empty Series (GH7242) Fix delegation of read_sql to read_sql_query when query does not contain ‘select’ (GH7324).

to when query does not contain ‘select’ (GH7324). Bug where a string column name assignment to a DataFrame with a Float64Index raised a TypeError during a call to np.isnan (GH7366).

with a raised a during a call to (GH7366). Bug where NDFrame.replace() didn’t correctly replace objects with Period values (GH7379).

didn’t correctly replace objects with values (GH7379). Bug in .ix getitem should always return a Series (GH7150)

getitem should always return a Series (GH7150) Bug in multi-index slicing with incomplete indexers (GH7399)

Bug in multi-index slicing with a step in a sliced level (GH7400)

Bug where negative indexers in DatetimeIndex were not correctly sliced (GH7408)

were not correctly sliced (GH7408) Bug where NaT wasn’t repr’d correctly in a MultiIndex (GH7406, GH7409).

wasn’t repr’d correctly in a (GH7406, GH7409). Bug where bool objects were converted to nan in convert_objects (GH7416).

in (GH7416). Bug in quantile ignoring the axis keyword argument (:issue`7306`)

ignoring the axis keyword argument (:issue`7306`) Bug where nanops._maybe_null_out doesn’t work with complex numbers (GH7353)

doesn’t work with complex numbers (GH7353) Bug in several nanops functions when axis==0 for 1-dimensional nan arrays (GH7354)

functions when for 1-dimensional arrays (GH7354) Bug where nanops.nanmedian doesn’t work when axis==None (GH7352)

doesn’t work when (GH7352) Bug where nanops._has_infs doesn’t work with many dtypes (GH7357)

doesn’t work with many dtypes (GH7357) Bug in StataReader.data where reading a 0-observation dta failed (GH7369)

where reading a 0-observation dta failed (GH7369) Bug in StataReader when reading Stata 13 (117) files containing fixed width strings (GH7360)

when reading Stata 13 (117) files containing fixed width strings (GH7360) Bug in StataWriter where encoding was ignored (GH7286)

where encoding was ignored (GH7286) Bug in DatetimeIndex comparison doesn’t handle NaT properly (GH7529)

comparison doesn’t handle properly (GH7529) Bug in passing input with tzinfo to some offsets apply , rollforward or rollback resets tzinfo or raises ValueError (GH7465)

to some offsets , or resets or raises (GH7465) Bug in DatetimeIndex.to_period , PeriodIndex.asobject , PeriodIndex.to_timestamp doesn’t preserve name (GH7485)

, , doesn’t preserve (GH7485) Bug in DatetimeIndex.to_period and PeriodIndex.to_timestanp handle NaT incorrectly (GH7228)

and handle incorrectly (GH7228) Bug in offsets.apply , rollforward and rollback may return normal datetime (GH7502)

, and may return normal (GH7502) Bug in resample raises ValueError when target contains NaT (GH7227)

raises when target contains (GH7227) Bug in Timestamp.tz_localize resets nanosecond info (GH7534)

resets info (GH7534) Bug in DatetimeIndex.asobject raises ValueError when it contains NaT (GH7539)

raises when it contains (GH7539) Bug in Timestamp.__new__ doesn’t preserve nanosecond properly (GH7610)

doesn’t preserve nanosecond properly (GH7610) Bug in Index.astype(float) where it would return an object dtype Index (GH7464).

where it would return an dtype (GH7464). Bug in DataFrame.reset_index loses tz (GH3950)

loses (GH3950) Bug in DatetimeIndex.freqstr raises AttributeError when freq is None (GH7606)

raises when is (GH7606) Bug in GroupBy.size created by TimeGrouper raises AttributeError (GH7453)

created by raises (GH7453) Bug in single column bar plot is misaligned (GH7498).

Bug in area plot with tz-aware time series raises ValueError (GH7471)

(GH7471) Bug in non-monotonic Index.union may preserve name incorrectly (GH7458)

may preserve incorrectly (GH7458) Bug in DatetimeIndex.intersection doesn’t preserve timezone (GH4690)

doesn’t preserve timezone (GH4690) Bug in rolling_var where a window larger than the array would raise an error(GH7297)

where a window larger than the array would raise an error(GH7297) Bug with last plotted timeseries dictating xlim (GH2960)

(GH2960) Bug with secondary_y axis not being considered for timeseries xlim (GH3490)

axis not being considered for timeseries (GH3490) Bug in Float64Index assignment with a non scalar indexer (GH7586)

assignment with a non scalar indexer (GH7586) Bug in pandas.core.strings.str_contains does not properly match in a case insensitive fashion when regex=False and case=False (GH7505)

does not properly match in a case insensitive fashion when and (GH7505) Bug in expanding_cov , expanding_corr , rolling_cov , and rolling_corr for two arguments with mismatched index (GH7512)

, , , and for two arguments with mismatched index (GH7512) Bug in to_sql taking the boolean column as text column (GH7678)

taking the boolean column as text column (GH7678) Bug in grouped hist doesn’t handle rot kw and sharex kw properly (GH7234)

doesn’t handle kw and kw properly (GH7234) Bug in .loc performing fallback integer indexing with object dtype indices (GH7496)

performing fallback integer indexing with dtype indices (GH7496) Bug (regression) in PeriodIndex constructor when passed Series objects (GH7701).

v0.13.1 (February 3, 2014)¶ This is a minor release from 0.13.0 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. Highlights include: Added infer_datetime_format keyword to read_csv/to_datetime to allow speedups for homogeneously formatted datetimes.

keyword to to allow speedups for homogeneously formatted datetimes. Will intelligently limit display precision for datetime/timedelta formats.

Enhanced Panel apply() method.

method. Suggested tutorials in new Tutorials section.

Our pandas ecosystem is growing, We now feature related projects in a new Pandas Ecosystem section.

Much work has been taking place on improving the docs, and a new Contributing section has been added.

Even though it may only be of interest to devs, we <3 our new CI status page: ScatterCI. Warning 0.13.1 fixes a bug that was caused by a combination of having numpy < 1.8, and doing chained assignment on a string-like array. Please review the docs, chained indexing can have unexpected results and should generally be avoided. This would previously segfault: In [1]: df = DataFrame ( dict ( A = np . array ([ 'foo' , 'bar' , 'bah' , 'foo' , 'bar' ]))) In [2]: df [ 'A' ] . iloc [ 0 ] = np . nan In [3]: df Out[3]: A 0 NaN 1 bar 2 bah 3 foo 4 bar The recommended way to do this type of assignment is: In [4]: df = DataFrame ( dict ( A = np . array ([ 'foo' , 'bar' , 'bah' , 'foo' , 'bar' ]))) In [5]: df . ix [ 0 , 'A' ] = np . nan In [6]: df Out[6]: A 0 NaN 1 bar 2 bah 3 foo 4 bar Output Formatting Enhancements¶ df.info() view now display dtype info per column (GH5682)

df.info() now honors the option max_info_rows , to disable null counts for large frames (GH5974) In [7]: max_info_rows = pd . get_option ( 'max_info_rows' ) In [8]: df = DataFrame ( dict ( A = np . random . randn ( 10 ), ...: B = np . random . randn ( 10 ), ...: C = date_range ( '20130101' , periods = 10 ))) ...: In [9]: df . iloc [ 3 : 6 ,[ 0 , 2 ]] = np . nan # set to not display the null counts In [10]: pd . set_option ( 'max_info_rows' , 0 ) In [11]: df . info () <class 'pandas.core.frame.DataFrame'> RangeIndex: 10 entries, 0 to 9 Data columns (total 3 columns): A float64 B float64 C datetime64[ns] dtypes: datetime64[ns](1), float64(2) memory usage: 312.0 bytes # this is the default (same as in 0.13.0) In [12]: pd . set_option ( 'max_info_rows' , max_info_rows ) In [13]: df . info () <class 'pandas.core.frame.DataFrame'> RangeIndex: 10 entries, 0 to 9 Data columns (total 3 columns): A 7 non-null float64 B 10 non-null float64 C 7 non-null datetime64[ns] dtypes: datetime64[ns](1), float64(2) memory usage: 312.0 bytes

Add show_dimensions display option for the new DataFrame repr to control whether the dimensions print. In [14]: df = DataFrame ([[ 1 , 2 ], [ 3 , 4 ]]) In [15]: pd . set_option ( 'show_dimensions' , False ) In [16]: df Out[16]: 0 1 0 1 2 1 3 4 In [17]: pd . set_option ( 'show_dimensions' , True ) In [18]: df Out[18]: 0 1 0 1 2 1 3 4 [2 rows x 2 columns]

The ArrayFormatter for datetime and timedelta64 now intelligently limit precision based on the values in the array (GH3401) Previously output might look like: age today diff 0 2001 - 01 - 01 00 : 00 : 00 2013 - 04 - 19 00 : 00 : 00 4491 days , 00 : 00 : 00 1 2004 - 06 - 01 00 : 00 : 00 2013 - 04 - 19 00 : 00 : 00 3244 days , 00 : 00 : 00 Now the output looks like: In [19]: df = DataFrame ([ Timestamp ( '20010101' ), ....: Timestamp ( '20040601' ) ], columns = [ 'age' ]) ....: In [20]: df [ 'today' ] = Timestamp ( '20130419' ) In [21]: df [ 'diff' ] = df [ 'today' ] - df [ 'age' ] In [22]: df Out[22]: age today diff 0 2001-01-01 2013-04-19 4491 days 1 2004-06-01 2013-04-19 3244 days [2 rows x 3 columns] API changes¶ Add -NaN and -nan to the default set of NA values (GH5952). See NA Values.

Added Series.str.get_dummies vectorized string method (GH6021), to extract dummy/indicator variables for separated string columns: In [23]: s = Series ([ 'a' , 'a|b' , np . nan , 'a|c' ]) In [24]: s . str . get_dummies ( sep = '|' ) Out[24]: a b c 0 1 0 0 1 1 1 0 2 0 0 0 3 1 0 1 [4 rows x 3 columns]

Added the NDFrame.equals() method to compare if two NDFrames are equal have equal axes, dtypes, and values. Added the array_equivalent function to compare if two ndarrays are equal. NaNs in identical locations are treated as equal. (GH5283) See also the docs for a motivating example. In [25]: df = DataFrame ({ 'col' :[ 'foo' , 0 , np . nan ]}) In [26]: df2 = DataFrame ({ 'col' :[ np . nan , 0 , 'foo' ]}, index = [ 2 , 1 , 0 ]) In [27]: df . equals ( df2 ) Out[27]: False In [28]: df . equals ( df2 . sort ()) Out[28]: True In [29]: import pandas.core.common as com In [30]: com . array_equivalent ( np . array ([ 0 , np . nan ]), np . array ([ 0 , np . nan ])) Out[30]: True In [31]: np . array_equal ( np . array ([ 0 , np . nan ]), np . array ([ 0 , np . nan ])) Out[31]: False

DataFrame.apply will use the reduce argument to determine whether a Series or a DataFrame should be returned when the DataFrame is empty (GH6007). Previously, calling DataFrame.apply an empty DataFrame would return either a DataFrame if there were no columns, or the function being applied would be called with an empty Series to guess whether a Series or DataFrame should be returned: In [32]: def applied_func ( col ): ....: print ( "Apply function being called with: " , col ) ....: return col . sum () ....: In [33]: empty = DataFrame ( columns = [ 'a' , 'b' ]) In [34]: empty . apply ( applied_func ) ('Apply function being called with: ', Series([], dtype: float64)) Out[34]: a NaN b NaN dtype: float64 Now, when apply is called on an empty DataFrame : if the reduce argument is True a Series will returned, if it is False a DataFrame will be returned, and if it is None (the default) the function being applied will be called with an empty series to try and guess the return type. In [35]: empty . apply ( applied_func , reduce = True ) Out[35]: a NaN b NaN dtype: float64 In [36]: empty . apply ( applied_func , reduce = False ) Out[36]: Empty DataFrame Columns: [a, b] Index: [] [0 rows x 2 columns] Prior Version Deprecations/Changes¶ There are no announced changes in 0.13 or prior that are taking effect as of 0.13.1 Deprecations¶ There are no deprecations of prior behavior in 0.13.1 Enhancements¶ pd.read_csv and pd.to_datetime learned a new infer_datetime_format keyword which greatly improves parsing perf in many cases. Thanks to @lexual for suggesting and @danbirken for rapidly implementing. (GH5490, GH6021) If parse_dates is enabled and this flag is set, pandas will attempt to infer the format of the datetime strings in the columns, and if it can be inferred, switch to a faster method of parsing them. In some cases this can increase the parsing speed by ~5-10x. # Try to infer the format for the index column df = pd . read_csv ( 'foo.csv' , index_col = 0 , parse_dates = True , infer_datetime_format = True )

date_format and datetime_format keywords can now be specified when writing to excel files (GH4133)

MultiIndex.from_product convenience function for creating a MultiIndex from the cartesian product of a set of iterables (GH6055): In [37]: shades = [ 'light' , 'dark' ] In [38]: colors = [ 'red' , 'green' , 'blue' ] In [39]: MultiIndex . from_product ([ shades , colors ], names = [ 'shade' , 'color' ]) Out[39]: MultiIndex(levels=[[u'dark', u'light'], [u'blue', u'green', u'red']], labels=[[1, 1, 1, 0, 0, 0], [2, 1, 0, 2, 1, 0]], names=[u'shade', u'color'])

Panel apply() will work on non-ufuncs. See the docs. In [40]: import pandas.util.testing as tm In [41]: panel = tm . makePanel ( 5 ) In [42]: panel Out[42]: <class 'pandas.core.panel.Panel'> Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis) Items axis: ItemA to ItemC Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00 Minor_axis axis: A to D In [43]: panel [ 'ItemA' ] Out[43]: A B C D 2000-01-03 0.694103 1.893534 -1.735349 -0.850346 2000-01-04 0.678630 0.639633 1.210384 1.176812 2000-01-05 0.239556 -0.962029 0.797435 -0.524336 2000-01-06 0.151227 -2.085266 -0.379811 0.700908 2000-01-07 0.816127 1.930247 0.702562 0.984188 [5 rows x 4 columns] Specifying an apply that operates on a Series (to return a single element) In [44]: panel . apply ( lambda x : x . dtype , axis = 'items' ) Out[44]: A B C D 2000-01-03 float64 float64 float64 float64 2000-01-04 float64 float64 float64 float64 2000-01-05 float64 float64 float64 float64 2000-01-06 float64 float64 float64 float64 2000-01-07 float64 float64 float64 float64 [5 rows x 4 columns] A similar reduction type operation In [45]: panel . apply ( lambda x : x . sum (), axis = 'major_axis' ) Out[45]: ItemA ItemB ItemC A 2.579643 3.062757 0.379252 B 1.416120 -1.960855 0.923558 C 0.595222 -1.079772 -3.118269 D 1.487226 -0.734611 -1.979310 [4 rows x 3 columns] This is equivalent to In [46]: panel . sum ( 'major_axis' ) Out[46]: ItemA ItemB ItemC A 2.579643 3.062757 0.379252 B 1.416120 -1.960855 0.923558 C 0.595222 -1.079772 -3.118269 D 1.487226 -0.734611 -1.979310 [4 rows x 3 columns] A transformation operation that returns a Panel, but is computing the z-score across the major_axis In [47]: result = panel . apply ( ....: lambda x : ( x - x . mean ()) / x . std (), ....: axis = 'major_axis' ) ....: In [48]: result Out[48]: <class 'pandas.core.panel.Panel'> Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis) Items axis: ItemA to ItemC Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00 Minor_axis axis: A to D In [49]: result [ 'ItemA' ] Out[49]: A B C D 2000-01-03 0.595800 0.907552 -1.556260 -1.244875 2000-01-04 0.544058 0.200868 0.915883 0.953747 2000-01-05 -0.924165 -0.701810 0.569325 -0.891290 2000-01-06 -1.219530 -1.334852 -0.418654 0.437589 2000-01-07 1.003837 0.928242 0.489705 0.744830 [5 rows x 4 columns]

Panel apply() operating on cross-sectional slabs. (GH1148) In [50]: f = lambda x : (( x . T - x . mean ( 1 )) / x . std ( 1 )) . T In [51]: result = panel . apply ( f , axis = [ 'items' , 'major_axis' ]) In [52]: result Out[52]: <class 'pandas.core.panel.Panel'> Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis) Items axis: A to D Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00 Minor_axis axis: ItemA to ItemC In [53]: result . loc [:,:, 'ItemA' ] Out[53]: A B C D 2000-01-03 0.331409 1.071034 -0.914540 -0.510587 2000-01-04 -0.741017 -0.118794 0.383277 0.537212 2000-01-05 0.065042 -0.767353 0.655436 0.069467 2000-01-06 0.027932 -0.569477 0.908202 0.610585 2000-01-07 1.116434 1.133591 0.871287 1.004064 [5 rows x 4 columns] This is equivalent to the following In [54]: result = Panel ( dict ([ ( ax , f ( panel . loc [:,:, ax ])) ....: for ax in panel . minor_axis ])) ....: In [55]: result Out[55]: <class 'pandas.core.panel.Panel'> Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis) Items axis: A to D Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00 Minor_axis axis: ItemA to ItemC In [56]: result . loc [:,:, 'ItemA' ] Out[56]: A B C D 2000-01-03 0.331409 1.071034 -0.914540 -0.510587 2000-01-04 -0.741017 -0.118794 0.383277 0.537212 2000-01-05 0.065042 -0.767353 0.655436 0.069467 2000-01-06 0.027932 -0.569477 0.908202 0.610585 2000-01-07 1.116434 1.133591 0.871287 1.004064 [5 rows x 4 columns] Performance¶ Performance improvements for 0.13.1 Series datetime/timedelta binary operations (GH5801)

DataFrame count/dropna for axis=1

for Series.str.contains now has a regex=False keyword which can be faster for plain (non-regex) string patterns