Pandas: Powerful Python Data Analysis Toolkit Pandas User Guide
User Manual:
Open the PDF directly: View PDF 
.
Page Count: 1661
pandas: powerful Python data analysis
toolkit
Release 0.16.1
Wes McKinney & PyData Development Team
May 11, 2015
CONTENTS
1
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
13
29
35
40
67
73
100
107
130
141
150
156
167
171
173
173
178
182
183
183
188
189
191
192
2
Installation
2.1 Python version support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Installing pandas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
195
195
195
198
3
Contributing to pandas
3.1 Where to start? . . . . . . . . . . . .
3.2 Bug Reports/Enhancement Requests .
3.3 Working with the code . . . . . . . .
3.4 Contributing to the documentation . .
3.5 Contributing to the code base . . . .
3.6 Contributing your changes to pandas
201
201
202
202
205
207
209
4
What’s New
1.1 v0.16.1 (May 11, 2015) . . . . . . . . . . . . . . . . .
1.2 v0.16.0 (March 22, 2015) . . . . . . . . . . . . . . . .
1.3 v0.15.2 (December 12, 2014) . . . . . . . . . . . . . .
1.4 v0.15.1 (November 9, 2014) . . . . . . . . . . . . . . .
1.5 v0.15.0 (October 18, 2014) . . . . . . . . . . . . . . . .
1.6 v0.14.1 (July 11, 2014) . . . . . . . . . . . . . . . . . .
1.7 v0.14.0 (May 31 , 2014) . . . . . . . . . . . . . . . . .
1.8 v0.13.1 (February 3, 2014) . . . . . . . . . . . . . . . .
1.9 v0.13.0 (January 3, 2014) . . . . . . . . . . . . . . . .
1.10 v0.12.0 (July 24, 2013) . . . . . . . . . . . . . . . . . .
1.11 v0.11.0 (April 22, 2013) . . . . . . . . . . . . . . . . .
1.12 v0.10.1 (January 22, 2013) . . . . . . . . . . . . . . . .
1.13 v0.10.0 (December 17, 2012) . . . . . . . . . . . . . .
1.14 v0.9.1 (November 14, 2012) . . . . . . . . . . . . . . .
1.15 v0.9.0 (October 7, 2012) . . . . . . . . . . . . . . . . .
1.16 v0.8.1 (July 22, 2012) . . . . . . . . . . . . . . . . . .
1.17 v0.8.0 (June 29, 2012) . . . . . . . . . . . . . . . . . .
1.18 v.0.7.3 (April 12, 2012) . . . . . . . . . . . . . . . . .
1.19 v.0.7.2 (March 16, 2012) . . . . . . . . . . . . . . . . .
1.20 v.0.7.1 (February 29, 2012) . . . . . . . . . . . . . . .
1.21 v.0.7.0 (February 9, 2012) . . . . . . . . . . . . . . . .
1.22 v.0.6.1 (December 13, 2011) . . . . . . . . . . . . . . .
1.23 v.0.6.0 (November 25, 2011) . . . . . . . . . . . . . . .
1.24 v.0.5.0 (October 24, 2011) . . . . . . . . . . . . . . . .
1.25 v.0.4.3 through v0.4.1 (September 25 - October 9, 2011)
Frequently Asked Questions (FAQ)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
213
i
4.1
4.2
4.3
4.4
4.5
5
6
7
8
9
ii
DataFrame memory usage . . . . . . . . . . . . . .
Adding Features to your pandas Installation . . . . .
Migrating from scikits.timeseries to pandas >= 0.8.0
Byte-Ordering Issues . . . . . . . . . . . . . . . . .
Visualizing Data in Qt applications . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
213
214
215
219
220
Package overview
5.1 Data structures at a glance . . .
5.2 Mutability and copying of data .
5.3 Getting Support . . . . . . . .
5.4 Credits . . . . . . . . . . . . .
5.5 Development Team . . . . . . .
5.6 License . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
223
223
224
224
224
224
224
10 Minutes to pandas
6.1 Object Creation . . .
6.2 Viewing Data . . . .
6.3 Selection . . . . . .
6.4 Missing Data . . . .
6.5 Operations . . . . .
6.6 Merge . . . . . . . .
6.7 Grouping . . . . . .
6.8 Reshaping . . . . .
6.9 Time Series . . . . .
6.10 Categoricals . . . .
6.11 Plotting . . . . . . .
6.12 Getting Data In/Out
6.13 Gotchas . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
227
227
229
230
235
235
238
240
241
243
244
246
247
249
Tutorials
7.1 Internal Guides . . . . . . . . . . . . . . . . . .
7.2 pandas Cookbook . . . . . . . . . . . . . . . .
7.3 Lessons for New pandas Users . . . . . . . . . .
7.4 Practical data analysis with Python . . . . . . .
7.5 Excel charts with pandas, vincent and xlsxwriter
7.6 Various Tutorials . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
251
251
251
252
252
252
252
Cookbook
8.1 Idioms . . . . . . . . .
8.2 Selection . . . . . . . .
8.3 MultiIndexing . . . . .
8.4 Missing Data . . . . . .
8.5 Grouping . . . . . . . .
8.6 Timeseries . . . . . . .
8.7 Merge . . . . . . . . . .
8.8 Plotting . . . . . . . . .
8.9 Data In/Out . . . . . . .
8.10 Computation . . . . . .
8.11 Timedeltas . . . . . . .
8.12 Aliasing Axis Names . .
8.13 Creating Example Data .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
255
255
258
262
266
267
275
276
277
278
281
281
283
284
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Intro to Data Structures
285
9.1 Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
9.2 DataFrame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
9.3
9.4
9.5
Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
Panel4D (Experimental) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
PanelND (Experimental) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
10 Essential Basic Functionality
10.1 Head and Tail . . . . . . . . . . . .
10.2 Attributes and the raw ndarray(s) .
10.3 Accelerated operations . . . . . . .
10.4 Flexible binary operations . . . . .
10.5 Descriptive statistics . . . . . . . .
10.6 Function application . . . . . . . .
10.7 Reindexing and altering labels . . .
10.8 Iteration . . . . . . . . . . . . . . .
10.9 Vectorized string methods . . . . .
10.10 Sorting by index and value . . . . .
10.11 Copying . . . . . . . . . . . . . .
10.12 dtypes . . . . . . . . . . . . . . . .
10.13 Selecting columns based on dtype
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
313
313
314
315
315
321
329
334
341
345
346
349
349
356
11 Working with Text Data
11.1 Splitting and Replacing Strings
11.2 Indexing with .str . . . . . .
11.3 Extracting Substrings . . . . . .
11.4 Method Summary . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
359
360
362
363
365
12 Options and Settings
12.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . .
12.2 Getting and Setting Options . . . . . . . . . . . . . . .
12.3 Setting Startup Options in python/ipython Environment
12.4 Frequently Used Options . . . . . . . . . . . . . . . . .
12.5 List of Options . . . . . . . . . . . . . . . . . . . . . .
12.6 Number Formatting . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
367
367
368
369
369
375
376
13 Indexing and Selecting Data
13.1 Different Choices for Indexing . . . . .
13.2 Deprecations . . . . . . . . . . . . . .
13.3 Basics . . . . . . . . . . . . . . . . . .
13.4 Attribute Access . . . . . . . . . . . .
13.5 Slicing ranges . . . . . . . . . . . . .
13.6 Selection By Label . . . . . . . . . . .
13.7 Selection By Position . . . . . . . . .
13.8 Selecting Random Samples . . . . . .
13.9 Setting With Enlargement . . . . . . .
13.10 Fast scalar value getting and setting . .
13.11 Boolean indexing . . . . . . . . . . . .
13.12 Indexing with isin . . . . . . . . . . .
13.13 The where() Method and Masking .
13.14 The query() Method (Experimental)
13.15 Duplicate Data . . . . . . . . . . . . .
13.16 Dictionary-like get() method . . . .
13.17 The select() Method . . . . . . . .
13.18 The lookup() Method . . . . . . . .
13.19 Index objects . . . . . . . . . . . . . .
13.20 Set / Reset Index . . . . . . . . . . . .
13.21 Returning a view versus a copy . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
377
377
378
379
380
382
383
386
390
392
393
394
395
397
401
411
412
413
413
413
415
418
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
iii
14 MultiIndex / Advanced Indexing
14.1 Hierarchical indexing (MultiIndex) . . . . .
14.2 Advanced indexing with hierarchical index .
14.3 The need for sortedness with MultiIndex
14.4 Take Methods . . . . . . . . . . . . . . . .
14.5 CategoricalIndex . . . . . . . . . . . . . . .
14.6 Float64Index . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
421
421
426
435
437
439
441
15 Computational tools
15.1 Statistical functions . . . . . . . . . . . .
15.2 Moving (rolling) statistics / moments . . .
15.3 Expanding window moment functions . . .
15.4 Exponentially weighted moment functions
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
445
445
449
457
458
16 Working with missing data
16.1 Missing data basics . . . . . . . . . . .
16.2 Datetimes . . . . . . . . . . . . . . . .
16.3 Inserting missing data . . . . . . . . .
16.4 Calculations with missing data . . . . .
16.5 Cleaning / filling missing data . . . . .
16.6 Missing data casting rules and indexing
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
463
463
465
465
466
467
480
17 Group By: split-apply-combine
17.1 Splitting an object into groups .
17.2 Iterating through groups . . . .
17.3 Selecting a group . . . . . . . .
17.4 Aggregation . . . . . . . . . .
17.5 Transformation . . . . . . . . .
17.6 Filtration . . . . . . . . . . . .
17.7 Dispatching to instance methods
17.8 Flexible apply . . . . . . . .
17.9 Other useful features . . . . . .
17.10 Examples . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
483
484
488
489
489
493
497
498
500
502
509
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
18 Merge, join, and concatenate
511
18.1 Concatenating objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
18.2 Database-style DataFrame joining/merging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
19 Reshaping and Pivot Tables
19.1 Reshaping by pivoting DataFrame objects
19.2 Reshaping by stacking and unstacking . .
19.3 Reshaping by Melt . . . . . . . . . . . .
19.4 Combining with stats and GroupBy . . .
19.5 Pivot tables and cross-tabulations . . . .
19.6 Tiling . . . . . . . . . . . . . . . . . . .
19.7 Computing indicator / dummy variables .
19.8 Factorizing values . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
535
535
536
541
542
543
547
547
550
20 Time Series / Date functionality
20.1 Time Stamps vs. Time Spans . . . .
20.2 Converting to Timestamps . . . . . .
20.3 Generating Ranges of Timestamps . .
20.4 DatetimeIndex . . . . . . . . . . . .
20.5 DateOffset objects . . . . . . . . . .
20.6 Time series-related instance methods
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
551
552
553
554
556
563
573
iv
.
.
.
.
.
.
.
.
.
.
.
.
20.7
20.8
20.9
20.10
20.11
Up- and downsampling . . . . . . . .
Time Span Representation . . . . . .
Converting between Representations
Representing out-of-bounds spans . .
Time Zone Handling . . . . . . . . .
21 Time Deltas
21.1 Parsing . . . . . . . .
21.2 Operations . . . . . .
21.3 Reductions . . . . . .
21.4 Frequency Conversion
21.5 Attributes . . . . . . .
21.6 TimedeltaIndex . . . .
21.7 Resampling . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
575
577
583
584
585
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
593
593
594
598
599
600
601
604
22 Categorical Data
22.1 Object Creation . . . . .
22.2 Description . . . . . . .
22.3 Working with categories
22.4 Sorting and Order . . .
22.5 Comparisons . . . . . .
22.6 Operations . . . . . . .
22.7 Data munging . . . . .
22.8 Getting Data In/Out . .
22.9 Missing Data . . . . . .
22.10 Gotchas . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
605
605
608
609
612
614
616
617
621
622
624
23 Plotting
23.1 Basic Plotting: plot . . . . . .
23.2 Other Plots . . . . . . . . . . .
23.3 Plotting with Missing Data . . .
23.4 Plotting Tools . . . . . . . . . .
23.5 Plot Formatting . . . . . . . . .
23.6 Plotting directly with matplotlib
23.7 Trellis plotting interface . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
629
629
632
663
664
672
695
696
24 IO Tools (Text, CSV, HDF5, ...)
24.1 CSV & Text files . . . . . . . . .
24.2 JSON . . . . . . . . . . . . . . .
24.3 HTML . . . . . . . . . . . . . .
24.4 Excel files . . . . . . . . . . . .
24.5 Clipboard . . . . . . . . . . . . .
24.6 Pickling . . . . . . . . . . . . . .
24.7 msgpack (experimental) . . . . .
24.8 HDF5 (PyTables) . . . . . . . . .
24.9 SQL Queries . . . . . . . . . . .
24.10 Google BigQuery (Experimental)
24.11 Stata Format . . . . . . . . . . .
24.12 Other file formats . . . . . . . . .
24.13 Performance Considerations . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
711
712
735
743
751
754
755
756
758
785
793
794
796
797
25 Remote Data Access
799
25.1 Yahoo! Finance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 799
25.2 Yahoo! Finance Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 800
25.3 Google Finance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 802
v
25.4
25.5
25.6
25.7
FRED . . . . . .
Fama/French . .
World Bank . . .
Google Analytics
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
802
803
803
806
26 Enhancing Performance
809
26.1 Cython (Writing C extensions for pandas) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 809
26.2 Expression Evaluation via eval() (Experimental) . . . . . . . . . . . . . . . . . . . . . . . . . . . 813
27 Sparse data structures
27.1 SparseArray . . . . . . . .
27.2 SparseList . . . . . . . . .
27.3 SparseIndex objects . . . .
27.4 Interaction with scipy.sparse
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
821
823
823
824
824
28 Caveats and Gotchas
28.1 Using If/Truth Statements with pandas . . . . .
28.2 NaN, Integer NA values and NA type promotions
28.3 Integer indexing . . . . . . . . . . . . . . . . .
28.4 Label-based slicing conventions . . . . . . . . .
28.5 Miscellaneous indexing gotchas . . . . . . . . .
28.6 Timestamp limitations . . . . . . . . . . . . . .
28.7 Parsing Dates from Text Files . . . . . . . . . .
28.8 Differences with NumPy . . . . . . . . . . . . .
28.9 Thread-safety . . . . . . . . . . . . . . . . . . .
28.10 HTML Table Parsing . . . . . . . . . . . . . . .
28.11 Byte-Ordering Issues . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
829
829
830
832
832
833
835
835
836
836
836
837
29 rpy2 / R interface
29.1 Updating your code to use rpy2 functions
29.2 R interface with rpy2 . . . . . . . . . . .
29.3 Transferring R data sets into Python . . .
29.4 Converting DataFrames into R objects . .
29.5 Calling R functions with pandas objects .
29.6 High-level interface to R estimators . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
839
839
840
840
840
841
841
30 pandas Ecosystem
30.1 Statistics and Machine Learning
30.2 Visualization . . . . . . . . . .
30.3 IDE . . . . . . . . . . . . . . .
30.4 API . . . . . . . . . . . . . . .
30.5 Domain Specific . . . . . . . .
30.6 Out-of-core . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
843
843
843
844
844
845
845
31 Comparison with R / R libraries
31.1 Base R . . . . . . . . . . .
31.2 zoo . . . . . . . . . . . . .
31.3 xts . . . . . . . . . . . . .
31.4 plyr . . . . . . . . . . . . .
31.5 reshape / reshape2 . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
847
847
853
853
853
854
32 Comparison with SQL
32.1 SELECT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32.2 WHERE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32.3 GROUP BY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
859
859
860
862
vi
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
32.4
32.5
32.6
32.7
JOIN . .
UNION .
UPDATE
DELETE
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
864
865
867
867
33 API Reference
33.1 Input/Output . . . . . .
33.2 General functions . . . .
33.3 Series . . . . . . . . . .
33.4 DataFrame . . . . . . .
33.5 Panel . . . . . . . . . .
33.6 Panel4D . . . . . . . .
33.7 Index . . . . . . . . . .
33.8 CategoricalIndex . . . .
33.9 DatetimeIndex . . . . .
33.10 TimedeltaIndex . . . . .
33.11 GroupBy . . . . . . . .
33.12 General utility functions
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
869
869
896
938
1097
1270
1355
1405
1436
1462
1492
1512
1536
34 Internals
1549
34.1 Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1549
34.2 Subclassing pandas Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1550
35 Release Notes
35.1 pandas 0.16.1
35.2 pandas 0.16.0
35.3 pandas 0.15.2
35.4 pandas 0.15.1
35.5 pandas 0.15.0
35.6 pandas 0.14.1
35.7 pandas 0.14.0
35.8 pandas 0.13.1
35.9 pandas 0.13.0
35.10 pandas 0.12.0
35.11 pandas 0.11.0
35.12 pandas 0.10.1
35.13 pandas 0.10.0
35.14 pandas 0.9.1
35.15 pandas 0.9.0
35.16 pandas 0.8.1
35.17 pandas 0.8.0
35.18 pandas 0.7.3
35.19 pandas 0.7.2
35.20 pandas 0.7.1
35.21 pandas 0.7.0
35.22 pandas 0.6.1
35.23 pandas 0.6.0
35.24 pandas 0.5.0
35.25 pandas 0.4.3
35.26 pandas 0.4.2
35.27 pandas 0.4.1
35.28 pandas 0.4.0
35.29 pandas 0.3.0
Python Module Index
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1555
1555
1557
1559
1561
1562
1564
1566
1569
1572
1586
1593
1599
1601
1606
1608
1613
1615
1620
1621
1623
1624
1630
1632
1636
1640
1641
1642
1644
1649
1651
vii
viii
pandas: powerful Python data analysis toolkit, Release 0.16.1
PDF Version
Zipped HTML Date: May 11, 2015 Version: 0.16.1
Binary Installers: http://pypi.python.org/pypi/pandas
Source Repository: http://github.com/pydata/pandas
Issues & Ideas: https://github.com/pydata/pandas/issues
Q&A Support: http://stackoverflow.com/questions/tagged/pandas
Developer Mailing List: http://groups.google.com/group/pydata
pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with
“relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing
practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful
and flexible open source data analysis / manipulation tool available in any language. It is already well on its way
toward this goal.
pandas is well suited for many different kinds of data:
• Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet
• Ordered and unordered (not necessarily fixed-frequency) time series data.
• Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels
• Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placed
into a pandas data structure
The two primary data structures of pandas, Series (1-dimensional) and DataFrame (2-dimensional), handle the
vast majority of typical use cases in finance, statistics, social science, and many areas of engineering. For R users,
DataFrame provides everything that R’s data.frame provides and much more. pandas is built on top of NumPy
and is intended to integrate well within a scientific computing environment with many other 3rd party libraries.
Here are just a few of the things that pandas does well:
• Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data
• Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects
• Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can
simply ignore the labels and let Series, DataFrame, etc. automatically align the data for you in computations
• Powerful, flexible group by functionality to perform split-apply-combine operations on data sets, for both aggregating and transforming data
• Make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures into
DataFrame objects
• Intelligent label-based slicing, fancy indexing, and subsetting of large data sets
• Intuitive merging and joining data sets
• Flexible reshaping and pivoting of data sets
• Hierarchical labeling of axes (possible to have multiple labels per tick)
• Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving / loading
data from the ultrafast HDF5 format
• Time series-specific functionality: date range generation and frequency conversion, moving window statistics,
moving window linear regressions, date shifting and lagging, etc.
CONTENTS
1
pandas: powerful Python data analysis toolkit, Release 0.16.1
Many of these principles are here to address the shortcomings frequently experienced using other languages / scientific
research environments. For data scientists, working with data is typically divided into multiple stages: munging and
cleaning data, analyzing / modeling it, then organizing the results of the analysis into a form suitable for plotting or
tabular display. pandas is the ideal tool for all of these tasks.
Some other notes
• pandas is fast. Many of the low-level algorithmic bits have been extensively tweaked in Cython code. However,
as with anything else generalization usually sacrifices performance. So if you focus on one feature for your
application you may be able to create a faster specialized tool.
• pandas is a dependency of statsmodels, making it an important part of the statistical computing ecosystem in
Python.
• pandas has been used extensively in production in financial applications.
Note: This documentation assumes general familiarity with NumPy. If you haven’t used NumPy much or at all, do
invest some time in learning about NumPy first.
See the package overview for more detail about what’s in the library.
2
CONTENTS
CHAPTER
ONE
WHAT’S NEW
These are new features and improvements of note in each release.
1.1 v0.16.1 (May 11, 2015)
This is a minor bug-fix release from 0.16.0 and includes a a large number of bug fixes along several new features,
enhancements, and performance improvements. We recommend that all users upgrade to this version.
Highlights include:
• Support for a CategoricalIndex, a category based index, see here
• New section on how-to-contribute to pandas, see here
• Revised “Merge, join, and concatenate” documentation, including graphical examples to make it easier to understand each operations, see here
• New method sample for drawing random samples from Series, DataFrames and Panels. See here
• The default Index printing has changed to a more uniform format, see here
• BusinessHour datetime-offset is now supported, see here
• Further enhancement to the .str accessor to make string operations easier, see here
What’s new in v0.16.1
• Enhancements
– CategoricalIndex
– Sample
– String Methods Enhancements
– Other Enhancements
• API changes
– Deprecations
• Index Representation
• Performance Improvements
• Bug Fixes
Warning: In pandas 0.17.0, the sub-package pandas.io.data will be removed in favor of a separately
installable package. See here for details (GH8961)
3
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.1.1 Enhancements
CategoricalIndex
We introduce a CategoricalIndex, a new type of index object that is useful for supporting indexing with duplicates. This is a container around a Categorical (introduced in v0.15.0) and allows efficient indexing and storage
of an index with a large number of duplicated elements. Prior to 0.16.1, setting the index of a DataFrame/Series
with a category dtype would convert this to regular object-based Index.
In [1]: df = DataFrame({'A' : np.arange(6),
...:
'B' : Series(list('aabbca')).astype('category',
...:
categories=list('cab'))
...:
})
...:
In [2]: df
Out[2]:
A B
0 0 a
1 1 a
2 2 b
3 3 b
4 4 c
5 5 a
In [3]: df.dtypes
Out[3]:
A
int32
B
category
dtype: object
In [4]: df.B.cat.categories
Out[4]: Index([u'c', u'a', u'b'], dtype='object')
setting the index, will create create a CategoricalIndex
In [5]: df2 = df.set_index('B')
In [6]: df2.index
Out[6]: CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a'], categories=[u'c', u'a', u'b'], ordered
indexing with __getitem__/.iloc/.loc/.ix works similarly to an Index with duplicates. The indexers
MUST be in the category or the operation will raise.
In [7]: df2.loc['a']
Out[7]:
A
B
a 0
a 1
a 5
and preserves the CategoricalIndex
In [8]: df2.loc['a'].index
Out[8]: CategoricalIndex([u'a', u'a', u'a'], categories=[u'c', u'a', u'b'], ordered=False, name=u'B',
sorting will order by the order of the categories
4
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [9]: df2.sort_index()
Out[9]:
A
B
c 4
a 0
a 1
a 5
b 2
b 3
groupby operations on the index will preserve the index nature as well
In [10]: df2.groupby(level=0).sum()
Out[10]:
A
B
c 4
a 6
b 5
In [11]: df2.groupby(level=0).sum().index
Out[11]: CategoricalIndex([u'c', u'a', u'b'], categories=[u'c', u'a', u'b'], ordered=False, name=u'B'
reindexing operations, will return a resulting index based on the type of the passed indexer, meaning that passing a
list will return a plain-old-Index; indexing with a Categorical will return a CategoricalIndex, indexed
according to the categories of the PASSED Categorical dtype. This allows one to arbitrarly index these even with
values NOT in the categories, similarly to how you can reindex ANY pandas index.
In [12]: df2.reindex(['a','e'])
Out[12]:
A
B
a
0
a
1
a
5
e NaN
In [13]: df2.reindex(['a','e']).index
Out[13]: Index([u'a', u'a', u'a', u'e'], dtype='object', name=u'B')
In [14]: df2.reindex(pd.Categorical(['a','e'],categories=list('abcde')))
Out[14]:
A
B
a
0
a
1
a
5
e NaN
In [15]: df2.reindex(pd.Categorical(['a','e'],categories=list('abcde'))).index
Out[15]: CategoricalIndex([u'a', u'a', u'a', u'e'], categories=[u'a', u'b', u'c', u'd', u'e'], ordere
See the documentation for more. (GH7629, GH10038, GH10039)
1.1. v0.16.1 (May 11, 2015)
5
pandas: powerful Python data analysis toolkit, Release 0.16.1
Sample
Series, DataFrames, and Panels now have a new method: sample(). The method accepts a specific number of rows
or columns to return, or a fraction of the total number or rows or columns. It also has options for sampling with or
without replacement, for passing in a column for weights for non-uniform sampling, and for setting seed values to
facilitate replication. (GH2419)
In [16]: example_series = Series([0,1,2,3,4,5])
# When no arguments are passed, returns 1
In [17]: example_series.sample()
Out[17]:
2
2
dtype: int64
# One may specify either a number of rows:
In [18]: example_series.sample(n=3)
Out[18]:
4
4
2
2
5
5
dtype: int64
# Or a fraction of the rows:
In [19]: example_series.sample(frac=0.5)
Out[19]:
0
0
4
4
3
3
dtype: int64
# weights are accepted.
In [20]: example_weights = [0, 0, 0.2, 0.2, 0.2, 0.4]
In [21]: example_series.sample(n=3, weights=example_weights)
Out[21]:
5
5
2
2
3
3
dtype: int64
# weights will also be normalized if they do not sum to one,
# and missing values will be treated as zeros.
In [22]: example_weights2 = [0.5, 0, 0, 0, None, np.nan]
In [23]: example_series.sample(n=1, weights=example_weights2)
Out[23]:
0
0
dtype: int64
When applied to a DataFrame, one may pass the name of a column to specify sampling weights when sampling from
rows.
In [24]: df = DataFrame({'col1':[9,8,7,6], 'weight_column':[0.5, 0.4, 0.1, 0]})
In [25]: df.sample(n=3, weights='weight_column')
Out[25]:
col1 weight_column
1
8
0.4
6
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
0
7
9
0.1
0.5
String Methods Enhancements
Continuing from v0.16.0, the following enhancements make string operations easier and more consistent with standard
python string operations.
• Added StringMethods (.str accessor) to Index (GH9068)
The .str accessor is now available for both Series and Index.
In [26]: idx = Index([' jack', 'jill ', ' jesse ', 'frank'])
In [27]: idx.str.strip()
Out[27]: Index([u'jack', u'jill', u'jesse', u'frank'], dtype='object')
One special case for the .str accessor on Index is that if a string method returns bool, the .str accessor
will return a np.array instead of a boolean Index (GH8875). This enables the following expression to work
naturally:
In [28]: idx = Index(['a1', 'a2', 'b1', 'b2'])
In [29]: s = Series(range(4), index=idx)
In [30]: s
Out[30]:
a1
0
a2
1
b1
2
b2
3
dtype: int64
In [31]: idx.str.startswith('a')
Out[31]: array([ True, True, False, False], dtype=bool)
In [32]: s[s.index.str.startswith('a')]
Out[32]:
a1
0
a2
1
dtype: int64
• The following new methods are accesible via .str accessor to apply the function to each values. (GH9766,
GH9773, GH10031, GH10045, GH10052)
capitalize()
index()
swapcase()
rindex()
Methods
normalize()
translate()
partition()
rpartition()
• split now takes expand keyword to specify whether to expand dimensionality. return_type is deprecated. (GH9847)
In [33]: s = Series(['a,b', 'a,c', 'b,c'])
# return
In [34]:
Out[34]:
0
[a,
1
[a,
Series
s.str.split(',')
b]
c]
1.1. v0.16.1 (May 11, 2015)
7
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
[b, c]
dtype: object
# return DataFrame
In [35]: s.str.split(',', expand=True)
Out[35]:
0 1
0 a b
1 a c
2 b c
In [36]: idx = Index(['a,b', 'a,c', 'b,c'])
# return Index
In [37]: idx.str.split(',')
Out[37]: Index([[u'a', u'b'], [u'a', u'c'], [u'b', u'c']], dtype='object')
# return MultiIndex
In [38]: idx.str.split(',', expand=True)
Out[38]:
MultiIndex(levels=[[u'a', u'b'], [u'b', u'c']],
labels=[[0, 0, 1], [0, 1, 1]])
• Improved extract and get_dummies methods for Index.str (GH9980)
Other Enhancements
• BusinessHour offset is now supported, which represents business hours starting from 09:00 - 17:00 on
BusinessDay by default. See Here for details. (GH7905)
In [39]: from pandas.tseries.offsets import BusinessHour
In [40]: Timestamp('2014-08-01 09:00') + BusinessHour()
Out[40]: Timestamp('2014-08-01 10:00:00')
In [41]: Timestamp('2014-08-01 07:00') + BusinessHour()
Out[41]: Timestamp('2014-08-01 10:00:00')
In [42]: Timestamp('2014-08-01 16:30') + BusinessHour()
Out[42]: Timestamp('2014-08-04 09:30:00')
• DataFrame.diff now takes an axis parameter that determines the direction of differencing (GH9727)
• Allow clip, clip_lower, and clip_upper to accept array-like arguments as thresholds (This is a regression from 0.11.0). These methods now have an axis parameter which determines how the Series or DataFrame
will be aligned with the threshold(s). (GH6966)
• DataFrame.mask() and Series.mask() now support same keywords as where (GH8801)
• drop function can now accept errors keyword to suppress ValueError raised when any of label does not
exist in the target data. (GH6736)
In [43]: df = DataFrame(np.random.randn(3, 3), columns=['A', 'B', 'C'])
In [44]: df.drop(['A', 'X'], axis=1, errors='ignore')
Out[44]:
B
C
0 -0.064034 -1.282782
8
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1 -1.071357
2 0.583787
0.441153
0.221471
• Add support for separating years and quarters using dashes, for example 2014-Q1. (GH9688)
• Allow conversion of values with dtype datetime64 or timedelta64 to strings using astype(str)
(GH9757)
• get_dummies function now accepts sparse keyword. If set to True, the return DataFrame is sparse, e.g.
SparseDataFrame. (GH8823)
• Period now accepts datetime64 as value input. (GH9054)
• Allow timedelta string conversion when leading zero is missing from time definition, ie 0:00:00 vs 00:00:00.
(GH9570)
• Allow Panel.shift with axis=’items’ (GH9890)
• Trying to write an excel file now raises NotImplementedError if the DataFrame has a MultiIndex
instead of writing a broken Excel file. (GH9794)
• Allow Categorical.add_categories to accept Series or np.array. (GH9927)
• Add/delete str/dt/cat accessors dynamically from __dir__. (GH9910)
• Add normalize as a dt accessor method. (GH10047)
• DataFrame and Series now have _constructor_expanddim property as overridable constructor for
one higher dimensionality data. This should be used only when it is really needed, see here
• pd.lib.infer_dtype now returns ’bytes’ in Python 3 where appropriate. (GH10032)
1.1.2 API changes
• When passing in an ax to df.plot( ..., ax=ax), the sharex kwarg will now default to False. The result
is that the visibility of xlabels and xticklabels will not anymore be changed. You have to do that by yourself
for the right axes in your figure or set sharex=True explicitly (but this changes the visible for all axes in the
figure, not only the one which is passed in!). If pandas creates the subplots itself (e.g. no passed in ax kwarg),
then the default is still sharex=True and the visibility changes are applied.
• assign() now inserts new columns in alphabetical order. Previously the order was arbitrary. (GH9777)
• By default, read_csv and read_table will now try to infer the compression type based on the file extension. Set compression=None to restore the previous behavior (no decompression). (GH9770)
Deprecations
• Series.str.split‘s return_type keyword was removed in favor of expand (GH9847)
1.1.3 Index Representation
The string representation of Index and its sub-classes have now been unified. These will show a single-line display
if there are few values; a wrapped multi-line display for a lot of values (but less than display.max_seq_items;
if lots of items (> display.max_seq_items) will show a truncated display (the head and tail of the data). The
formatting for MultiIndex is unchanges (a multi-line wrapped display). The display width responds to the option
display.max_seq_items, which is defaulted to 100. (GH6482)
Previous Behavior
1.1. v0.16.1 (May 11, 2015)
9
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [2]: pd.Index(range(4),name='foo')
Out[2]: Int64Index([0, 1, 2, 3], dtype='int64')
In [3]: pd.Index(range(104),name='foo')
Out[3]: Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
In [4]: pd.date_range('20130101',periods=4,name='foo',tz='US/Eastern')
Out[4]:
[2013-01-01 00:00:00-05:00, ..., 2013-01-04 00:00:00-05:00]
Length: 4, Freq: D, Timezone: US/Eastern
In [5]: pd.date_range('20130101',periods=104,name='foo',tz='US/Eastern')
Out[5]:
[2013-01-01 00:00:00-05:00, ..., 2013-04-14 00:00:00-04:00]
Length: 104, Freq: D, Timezone: US/Eastern
New Behavior
In [45]: pd.set_option('display.width', 80)
In [46]: pd.Index(range(4), name='foo')
Out[46]: Int64Index([0, 1, 2, 3], dtype='int64', name=u'foo')
In [47]: pd.Index(range(30), name='foo')
Out[47]:
Int64Index([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
dtype='int64', name=u'foo')
In [48]: pd.Index(range(104), name='foo')
Out[48]:
Int64Index([ 0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
...
94, 95, 96, 97, 98, 99, 100, 101, 102, 103],
dtype='int64', name=u'foo', length=104)
In [49]: pd.CategoricalIndex(['a','bb','ccc','dddd'], ordered=True, name='foobar')
Out[49]: CategoricalIndex([u'a', u'bb', u'ccc', u'dddd'], categories=[u'a', u'bb', u'ccc', u'dddd'],
In [50]: pd.CategoricalIndex(['a','bb','ccc','dddd']*10, ordered=True, name='foobar')
Out[50]:
CategoricalIndex([u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd',
u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd',
u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd',
u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd',
u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd'],
categories=[u'a', u'bb', u'ccc', u'dddd'], ordered=True, name=u'foobar', dtype='cate
In [51]: pd.CategoricalIndex(['a','bb','ccc','dddd']*100, ordered=True, name='foobar')
Out[51]:
CategoricalIndex([u'a', u'bb', u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd',
u'a', u'bb',
...
u'ccc', u'dddd', u'a', u'bb', u'ccc', u'dddd', u'a', u'bb',
u'ccc', u'dddd'],
categories=[u'a', u'bb', u'ccc', u'dddd'], ordered=True, name=u'foobar', dtype='cate
10
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [52]: pd.date_range('20130101',periods=4, name='foo', tz='US/Eastern')
Out[52]:
DatetimeIndex(['2013-01-01 00:00:00-05:00', '2013-01-02 00:00:00-05:00',
'2013-01-03 00:00:00-05:00', '2013-01-04 00:00:00-05:00'],
dtype='datetime64[ns]', name=u'foo', freq='D', tz='US/Eastern')
In [53]: pd.date_range('20130101',periods=25, freq='D')
Out[53]:
DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03',
'2013-01-05', '2013-01-06', '2013-01-07',
'2013-01-09', '2013-01-10', '2013-01-11',
'2013-01-13', '2013-01-14', '2013-01-15',
'2013-01-17', '2013-01-18', '2013-01-19',
'2013-01-21', '2013-01-22', '2013-01-23',
'2013-01-25'],
dtype='datetime64[ns]', freq='D', tz=None)
'2013-01-04',
'2013-01-08',
'2013-01-12',
'2013-01-16',
'2013-01-20',
'2013-01-24',
In [54]: pd.date_range('20130101',periods=104, name='foo', tz='US/Eastern')
Out[54]:
DatetimeIndex(['2013-01-01 00:00:00-05:00', '2013-01-02 00:00:00-05:00',
'2013-01-03 00:00:00-05:00', '2013-01-04 00:00:00-05:00',
'2013-01-05 00:00:00-05:00', '2013-01-06 00:00:00-05:00',
'2013-01-07 00:00:00-05:00', '2013-01-08 00:00:00-05:00',
'2013-01-09 00:00:00-05:00', '2013-01-10 00:00:00-05:00',
...
'2013-04-05 00:00:00-04:00', '2013-04-06 00:00:00-04:00',
'2013-04-07 00:00:00-04:00', '2013-04-08 00:00:00-04:00',
'2013-04-09 00:00:00-04:00', '2013-04-10 00:00:00-04:00',
'2013-04-11 00:00:00-04:00', '2013-04-12 00:00:00-04:00',
'2013-04-13 00:00:00-04:00', '2013-04-14 00:00:00-04:00'],
dtype='datetime64[ns]', name=u'foo', length=104, freq='D', tz='US/Eastern')
1.1.4 Performance Improvements
• Improved csv write performance with mixed dtypes, including datetimes by up to 5x (GH9940)
• Improved csv write performance generally by 2x (GH9940)
• Improved the performance of pd.lib.max_len_string_array by 5-7x (GH10024)
1.1.5 Bug Fixes
• Bug where labels did not appear properly in the legend of DataFrame.plot(), passing label= arguments
works, and Series indices are no longer mutated. (GH9542)
• Bug in json serialization causing a segfault when a frame had zero length. (GH9805)
• Bug in read_csv where missing trailing delimiters would cause segfault. (GH5664)
• Bug in retaining index name on appending (GH9862)
• Bug in scatter_matrix draws unexpected axis ticklabels (GH5662)
• Fixed bug in StataWriter resulting in changes to input DataFrame upon save (GH9795).
• Bug in transform causing length mismatch when null entries were present and a fast aggregator was being
used (GH9697)
1.1. v0.16.1 (May 11, 2015)
11
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in equals causing false negatives when block order differed (GH9330)
• Bug in grouping with multiple pd.Grouper where one is non-time based (GH10063)
• Bug in read_sql_table error when reading postgres table with timezone (GH7139)
• Bug in DataFrame slicing may not retain metadata (GH9776)
• Bug where TimdeltaIndex were not properly serialized in fixed HDFStore (GH9635)
• Bug with TimedeltaIndex constructor ignoring name when given another TimedeltaIndex as data
(GH10025).
• Bug in DataFrameFormatter._get_formatted_index with not applying max_colwidth to the
DataFrame index (GH7856)
• Bug in .loc with a read-only ndarray data source (GH10043)
• Bug in groupby.apply() that would raise if a passed user defined function either returned only None (for
all input). (GH9685)
• Always use temporary files in pytables tests (GH9992)
• Bug in plotting continuously using secondary_y may not show legend properly. (GH9610, GH9779)
• Bug in DataFrame.plot(kind="hist") results in TypeError when DataFrame contains nonnumeric columns (GH9853)
• Bug where repeated plotting of DataFrame with a DatetimeIndex may raise TypeError (GH9852)
• Bug in setup.py that would allow an incompat cython version to build (GH9827)
• Bug in plotting secondary_y incorrectly attaches right_ax property to secondary axes specifying itself
recursively. (GH9861)
• Bug in Series.quantile on empty Series of type Datetime or Timedelta (GH9675)
• Bug in where causing incorrect results when upcasting was required (GH9731)
• Bug in FloatArrayFormatter where decision boundary for displaying “small” floats in decimal format is
off by one order of magnitude for a given display.precision (GH9764)
• Fixed bug where DataFrame.plot() raised an error when both color and style keywords were passed
and there was no color symbol in the style strings (GH9671)
• Not showing a DeprecationWarning on combining list-likes with an Index (GH10083)
• Bug in read_csv and read_table when using skip_rows parameter if blank lines are present. (GH9832)
• Bug in read_csv() interprets index_col=True as 1 (GH9798)
• Bug in index equality comparisons using == failing on Index/MultiIndex type incompatibility (GH9785)
• Bug in which SparseDataFrame could not take nan as a column name (GH8822)
• Bug in to_msgpack and read_msgpack zlib and blosc compression support (GH9783)
• Bug GroupBy.size doesn’t attach index name properly if grouped by TimeGrouper (GH9925)
• Bug causing an exception in slice assignments because length_of_indexer returns wrong results
(GH9995)
• Bug in csv parser causing lines with initial whitespace plus one non-space character to be skipped. (GH9710)
• Bug in C csv parser causing spurious NaNs when data started with newline followed by whitespace. (GH10022)
• Bug causing elements with a null group to spill into the final group when grouping by a Categorical
(GH9603)
12
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug where .iloc and .loc behavior is not consistent on empty dataframes (GH9964)
• Bug in invalid attribute access on a TimedeltaIndex incorrectly raised ValueError instead of
AttributeError (GH9680)
• Bug in unequal comparisons between categorical data and a scalar, which was not in the categories (e.g.
Series(Categorical(list("abc"), ordered=True)) > "d". This returned False for all elements, but now raises a TypeError. Equality comparisons also now return False for == and True for !=.
(GH9848)
• Bug in DataFrame __setitem__ when right hand side is a dictionary (GH9874)
• Bug in where when dtype is datetime64/timedelta64, but dtype of other is not (GH9804)
• Bug in MultiIndex.sortlevel() results in unicode level name breaks (GH9856)
• Bug in which groupby.transform incorrectly enforced output dtypes to match input dtypes. (GH9807)
• Bug in DataFrame constructor when columns parameter is set, and data is an empty list (GH9939)
• Bug in bar plot with log=True raises TypeError if all values are less than 1 (GH9905)
• Bug in horizontal bar plot ignores log=True (GH9905)
• Bug in PyTables queries that did not return proper results using the index (GH8265, GH9676)
• Bug where dividing a dataframe containing values of type Decimal by another Decimal would raise.
(GH9787)
• Bug where using DataFrames asfreq would remove the name of the index. (GH9885)
• Bug causing extra index point when resample BM/BQ (GH9756)
• Changed caching in AbstractHolidayCalendar to be at the instance level rather than at the class level as
the latter can result in unexpected behaviour. (GH9552)
• Fixed latex output for multi-indexed dataframes (GH9778)
• Bug causing an exception when setting an empty range using DataFrame.loc (GH9596)
• Bug in hiding ticklabels with subplots and shared axes when adding a new plot to an existing grid of axes
(GH9158)
• Bug in transform and filter when grouping on a categorical variable (GH9921)
• Bug in transform when groups are equal in number and dtype to the input index (GH9700)
• Google BigQuery connector now imports dependencies on a per-method basis.(GH9713)
• Updated BigQuery connector to no longer use deprecated oauth2client.tools.run() (GH8327)
• Bug in subclassed DataFrame. It may not return the correct class, when slicing or subsetting it. (GH9632)
• Bug in .median() where non-float null values are not handled correctly (GH10040)
• Bug in Series.fillna() where it raises if a numerically convertible string is given (GH10092)
1.2 v0.16.0 (March 22, 2015)
This is a major release from 0.15.2 and includes a small number of API changes, several new features, enhancements,
and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this
version.
Highlights include:
1.2. v0.16.0 (March 22, 2015)
13
pandas: powerful Python data analysis toolkit, Release 0.16.1
• DataFrame.assign method, see here
• Series.to_coo/from_coo methods to interact with scipy.sparse, see here
• Backwards incompatible change
datetime.timedelta, see here
to
Timedelta
to
conform
the
.seconds
attribute
with
• Changes to the .loc slicing API to conform with the behavior of .ix see here
• Changes to the default for ordering in the Categorical constructor, see here
• Enhancement to the .str accessor to make string operations easier, see here
• The pandas.tools.rplot, pandas.sandbox.qtpandas and pandas.rpy modules are deprecated.
We refer users to external packages like seaborn, pandas-qt and rpy2 for similar or equivalent functionality, see
here
Check the API Changes and deprecations before updating.
What’s new in v0.16.0
• New features
– DataFrame Assign
– Interaction with scipy.sparse
– String Methods Enhancements
– Other enhancements
• Backwards incompatible API changes
– Changes in Timedelta
– Indexing Changes
– Categorical Changes
– Other API Changes
– Deprecations
– Removal of prior version deprecations/changes
• Performance Improvements
• Bug Fixes
1.2.1 New features
DataFrame Assign
Inspired by dplyr’s mutate verb, DataFrame has a new assign() method. The function signature for assign is
simply **kwargs. The keys are the column names for the new fields, and the values are either a value to be inserted
(for example, a Series or NumPy array), or a function of one argument to be called on the DataFrame. The new
values are inserted, and the entire DataFrame (with all original and new columns) is returned.
In [1]: iris = read_csv('data/iris.data')
In [2]: iris.head()
Out[2]:
SepalLength SepalWidth
0
5.1
3.5
1
4.9
3.0
2
4.7
3.2
3
4.6
3.1
4
5.0
3.6
14
PetalLength
1.4
1.4
1.3
1.5
1.4
PetalWidth
0.2
0.2
0.2
0.2
0.2
Name
Iris-setosa
Iris-setosa
Iris-setosa
Iris-setosa
Iris-setosa
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [3]: iris.assign(sepal_ratio=iris['SepalWidth'] /
Out[3]:
SepalLength SepalWidth PetalLength PetalWidth
0
5.1
3.5
1.4
0.2
1
4.9
3.0
1.4
0.2
2
4.7
3.2
1.3
0.2
3
4.6
3.1
1.5
0.2
4
5.0
3.6
1.4
0.2
iris['SepalLength']).head()
Name
Iris-setosa
Iris-setosa
Iris-setosa
Iris-setosa
Iris-setosa
sepal_ratio
0.686275
0.612245
0.680851
0.673913
0.720000
Above was an example of inserting a precomputed value. We can also pass in a function to be evalutated.
In [4]: iris.assign(sepal_ratio = lambda x: (x['SepalWidth'] /
...:
x['SepalLength'])).head()
...:
Out[4]:
SepalLength SepalWidth PetalLength PetalWidth
Name sepal_ratio
0
5.1
3.5
1.4
0.2 Iris-setosa
0.686275
1
4.9
3.0
1.4
0.2 Iris-setosa
0.612245
2
4.7
3.2
1.3
0.2 Iris-setosa
0.680851
3
4.6
3.1
1.5
0.2 Iris-setosa
0.673913
4
5.0
3.6
1.4
0.2 Iris-setosa
0.720000
The power of assign comes when used in chains of operations. For example, we can limit the DataFrame to just
those with a Sepal Length greater than 5, calculate the ratio, and plot
In [5]: (iris.query('SepalLength > 5')
...:
.assign(SepalRatio = lambda x: x.SepalWidth / x.SepalLength,
...:
PetalRatio = lambda x: x.PetalWidth / x.PetalLength)
...:
.plot(kind='scatter', x='SepalRatio', y='PetalRatio'))
...:
Out[5]:
See the documentation for more. (GH9229)
Interaction with scipy.sparse
Added SparseSeries.to_coo() and SparseSeries.from_coo() methods (GH8048) for converting to
and from scipy.sparse.coo_matrix instances (see here). For example, given a SparseSeries with MultiIndex
we can convert to a scipy.sparse.coo_matrix by specifying the row and column labels as index levels:
1.2. v0.16.0 (March 22, 2015)
15
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [6]: from numpy import nan
In [7]: s = Series([3.0, nan, 1.0, 3.0, nan, nan])
In [8]: s.index = MultiIndex.from_tuples([(1, 2, 'a', 0),
...:
(1, 2, 'a', 1),
...:
(1, 1, 'b', 0),
...:
(1, 1, 'b', 1),
...:
(2, 1, 'b', 0),
...:
(2, 1, 'b', 1)],
...:
names=['A', 'B', 'C', 'D'])
...:
In [9]: s
Out[9]:
A B C D
1 2 a 0
3
1
NaN
1 b 0
1
1
3
2 1 b 0
NaN
1
NaN
dtype: float64
# SparseSeries
In [10]: ss = s.to_sparse()
In [11]: ss
Out[11]:
A B C D
1 2 a 0
3
1
NaN
1 b 0
1
1
3
2 1 b 0
NaN
1
NaN
dtype: float64
BlockIndex
Block locations: array([0, 2])
Block lengths: array([1, 2])
In [12]: A, rows, columns = ss.to_coo(row_levels=['A', 'B'],
....:
column_levels=['C', 'D'],
....:
sort_labels=False)
....:
In [13]: A
Out[13]:
<3x4 sparse matrix of type ''
with 3 stored elements in COOrdinate format>
In [14]: A.todense()
Out[14]:
matrix([[ 3., 0., 0.,
[ 0., 0., 1.,
[ 0., 0., 0.,
0.],
3.],
0.]])
In [15]: rows
16
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[15]: [(1L, 2L), (1L, 1L), (2L, 1L)]
In [16]: columns
Out[16]: [('a', 0L), ('a', 1L), ('b', 0L), ('b', 1L)]
The from_coo method is a
scipy.sparse.coo_matrix:
convenience
method
for
creating
a
SparseSeries
from
a
In [17]: from scipy import sparse
In [18]: A = sparse.coo_matrix(([3.0, 1.0, 2.0], ([1, 0, 0], [0, 2, 3])),
....:
shape=(3, 4))
....:
In [19]: A
Out[19]:
<3x4 sparse matrix of type ''
with 3 stored elements in COOrdinate format>
In [20]: A.todense()
Out[20]:
matrix([[ 0., 0., 1.,
[ 3., 0., 0.,
[ 0., 0., 0.,
2.],
0.],
0.]])
In [21]: ss = SparseSeries.from_coo(A)
In [22]: ss
Out[22]:
0 2
1
3
2
1 0
3
dtype: float64
BlockIndex
Block locations: array([0])
Block lengths: array([3])
String Methods Enhancements
• Following new methods are accesible via .str accessor to apply the function to each values. This is intended
to make it more consistent with standard methods on strings. (GH9282, GH9352, GH9386, GH9387, GH9439)
isalnum()
islower()
find()
isalpha()
isupper()
rfind()
Methods
isdigit()
istitle()
ljust()
isdigit()
isnumeric()
rjust()
isspace()
isdecimal()
zfill()
In [23]: s = Series(['abcd', '3456', 'EFGH'])
In [24]: s.str.isalpha()
Out[24]:
0
True
1
False
2
True
dtype: bool
In [25]: s.str.find('ab')
1.2. v0.16.0 (March 22, 2015)
17
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[25]:
0
0
1
-1
2
-1
dtype: int64
• Series.str.pad() and Series.str.center() now accept fillchar option to specify filling character (GH9352)
In [26]: s = Series(['12', '300', '25'])
In [27]: s.str.pad(5, fillchar='_')
Out[27]:
0
___12
1
__300
2
___25
dtype: object
• Added Series.str.slice_replace(), which previously raised NotImplementedError (GH8888)
In [28]: s = Series(['ABCD', 'EFGH', 'IJK'])
In [29]: s.str.slice_replace(1, 3, 'X')
Out[29]:
0
AXD
1
EXH
2
IX
dtype: object
# replaced with empty char
In [30]: s.str.slice_replace(0, 1)
Out[30]:
0
BCD
1
FGH
2
JK
dtype: object
Other enhancements
• Reindex now supports method=’nearest’ for frames or series with a monotonic increasing or decreasing
index (GH9258):
In [31]: df = pd.DataFrame({'x': range(5)})
In [32]: df.reindex([0.2, 1.8, 3.5], method='nearest')
Out[32]:
x
0.2 0
1.8 2
3.5 4
This method is also exposed by the lower level Index.get_indexer and Index.get_loc methods.
• The read_excel() function’s sheetname argument now accepts a list and None, to get multiple or all sheets
respectively. If more than one sheet is specified, a dictionary is returned. (GH9450)
# Returns the 1st and 4th sheet, as a dictionary of DataFrames.
pd.read_excel('path_to_file.xls',sheetname=['Sheet1',3])
18
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Allow Stata files to be read incrementally with an iterator; support for long strings in Stata files. See the docs
here (GH9493:).
• Paths beginning with ~ will now be expanded to begin with the user’s home directory (GH9066)
• Added time interval selection in get_data_yahoo (GH9071)
• Added Timestamp.to_datetime64() to complement Timedelta.to_timedelta64() (GH9255)
• tseries.frequencies.to_offset() now accepts Timedelta as input (GH9064)
• Lag parameter was added to the autocorrelation method of Series, defaults to lag-1 autocorrelation (GH9192)
• Timedelta will now accept nanoseconds keyword in constructor (GH9273)
• SQL code now safely escapes table and column names (GH8986)
• Added auto-complete for Series.str., Series.dt. and Series.cat.
(GH9322)
• Index.get_indexer now supports method=’pad’ and method=’backfill’ even for any target array, not just monotonic targets. These methods also work for monotonic decreasing as well as monotonic
increasing indexes (GH9258).
• Index.asof now works on all index types (GH9258).
• A verbose argument has been augmented in io.read_excel(), defaults to False. Set to True to print
sheet names as they are parsed. (GH9450)
• Added days_in_month (compatibility alias daysinmonth) property to Timestamp, DatetimeIndex,
Period, PeriodIndex, and Series.dt (GH9572)
• Added decimal option in to_csv to provide formatting for non-‘.’ decimal separators (GH781)
• Added normalize option for Timestamp to normalized to midnight (GH8794)
• Added example for DataFrame import to R using HDF5 file and rhdf5 library. See the documentation for
more (GH9636).
1.2.2 Backwards incompatible API changes
Changes in Timedelta
In v0.15.0 a new scalar type Timedelta was introduced, that is a sub-class of datetime.timedelta. Mentioned
here was a notice of an API change w.r.t. the .seconds accessor. The intent was to provide a user-friendly set of
accessors that give the ‘natural’ value for that unit, e.g. if you had a Timedelta(’1 day, 10:11:12’), then
.seconds would return 12. However, this is at odds with the definition of datetime.timedelta, which defines
.seconds as 10 * 3600 + 11 * 60 + 12 == 36672.
So in v0.16.0, we are restoring the API to match that of datetime.timedelta. Further, the component values are
still available through the .components accessor. This affects the .seconds and .microseconds accessors,
and removes the .hours, .minutes, .milliseconds accessors. These changes affect TimedeltaIndex and
the Series .dt accessor as well. (GH9185, GH9139)
Previous Behavior
In [2]: t = pd.Timedelta('1 day, 10:11:12.100123')
In [3]: t.days
Out[3]: 1
In [4]: t.seconds
1.2. v0.16.0 (March 22, 2015)
19
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[4]: 12
In [5]: t.microseconds
Out[5]: 123
New Behavior
In [33]: t = pd.Timedelta('1 day, 10:11:12.100123')
In [34]: t.days
Out[34]: 1L
In [35]: t.seconds
Out[35]: 36672L
In [36]: t.microseconds
Out[36]: 100123L
Using .components allows the full component access
In [37]: t.components
Out[37]: Components(days=1L, hours=10L, minutes=11L, seconds=12L, milliseconds=100L, microseconds=123
In [38]: t.components.seconds
Out[38]: 12L
Indexing Changes
The behavior of a small sub-set of edge cases for using .loc have changed (GH8613). Furthermore we have improved
the content of the error messages that are raised:
• Slicing with .loc where the start and/or stop bound is not found in the index is now allowed; this previously
would raise a KeyError. This makes the behavior the same as .ix in this case. This change is only for
slicing, not when indexing with a single label.
In [39]: df = DataFrame(np.random.randn(5,4),
....:
columns=list('ABCD'),
....:
index=date_range('20130101',periods=5))
....:
In [40]: df
Out[40]:
A
B
C
D
2013-01-01 -0.744471 0.758527 1.729689 -0.964980
2013-01-02 -0.845696 -1.340896 1.846883 -1.328865
2013-01-03 1.682706 -1.717693 0.888782 0.228440
2013-01-04 0.901805 1.171216 0.520260 -1.197071
2013-01-05 -1.066969 -0.303421 -0.858447 0.306996
In [41]: s = Series(range(5),[-2,-1,1,2,3])
In [42]: s
Out[42]:
-2
0
-1
1
1
2
2
3
20
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
3
4
dtype: int64
Previous Behavior
In [4]: df.loc['2013-01-02':'2013-01-10']
KeyError: 'stop bound [2013-01-10] is not in the [index]'
In [6]: s.loc[-10:3]
KeyError: 'start bound [-10] is not the [index]'
New Behavior
In [43]: df.loc['2013-01-02':'2013-01-10']
Out[43]:
A
B
C
D
2013-01-02 -0.845696 -1.340896 1.846883 -1.328865
2013-01-03 1.682706 -1.717693 0.888782 0.228440
2013-01-04 0.901805 1.171216 0.520260 -1.197071
2013-01-05 -1.066969 -0.303421 -0.858447 0.306996
In [44]: s.loc[-10:3]
Out[44]:
-2
0
-1
1
1
2
2
3
3
4
dtype: int64
• Allow slicing with float-like values on an integer index for .ix. Previously this was only enabled for .loc:
Previous Behavior
In [8]: s.ix[-1.0:2]
TypeError: the slice start value [-1.0] is not a proper indexer for this index type (Int64Index)
New Behavior
In [45]: s.ix[-1.0:2]
Out[45]:
-1
1
1
2
2
3
dtype: int64
• Provide a useful exception for indexing with an invalid type for that index when using .loc. For example
trying to use .loc on an index of type DatetimeIndex or PeriodIndex or TimedeltaIndex, with an
integer (or a float).
Previous Behavior
In [4]: df.loc[2:3]
KeyError: 'start bound [2] is not the [index]'
New Behavior
In [4]: df.loc[2:3]
TypeError: Cannot do slice indexing on with 0.1)
In [23]: p.all()
Out[23]:
0
1
0
True False
1
True False
2 False False
3
True
True
• Added support for utcfromtimestamp(), fromtimestamp(), and combine() on Timestamp class
(GH5351).
• Added Google Analytics (pandas.io.ga) basic documentation (GH8835). See here.
• Timedelta arithmetic returns NotImplemented in unknown cases, allowing extensions by custom classes
(GH8813).
• Timedelta now supports arithemtic with numpy.ndarray objects of the appropriate dtype (numpy 1.8 or
newer only) (GH8884).
32
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Added Timedelta.to_timedelta64() method to the public API (GH8884).
• Added gbq.generate_bq_schema() function to the gbq module (GH8325).
• Series now works with map objects the same way as generators (GH8909).
• Added context manager to HDFStore for automatic closing (GH8791).
• to_datetime gains an exact keyword to allow for a format to not require an exact match for a provided format string (if its False). exact defaults to True (meaning that exact matching is still the default) (GH8904)
• Added axvlines boolean option to parallel_coordinates plot function, determines whether vertical lines will
be printed, default is True
• Added ability to read table footers to read_html (GH8552)
• to_sql now infers datatypes of non-NA values for columns that contain NA values and have dtype object
(GH8778).
1.3.3 Performance
• Reduce memory usage when skiprows is an integer in read_csv (GH8681)
• Performance boost for to_datetime conversions with a passed format=, and the exact=False
(GH8904)
1.3.4 Bug Fixes
• Bug in concat of Series with category dtype which were coercing to object. (GH8641)
• Bug in Timestamp-Timestamp not returning a Timedelta type and datelike-datelike ops with timezones
(GH8865)
• Made consistent a timezone mismatch exception (either tz operated with None or incompatible timezone), will
now return TypeError rather than ValueError (a couple of edge cases only), (GH8865)
• Bug in using a pd.Grouper(key=...) with no level/axis or level only (GH8795, GH8866)
• Report a TypeError when invalid/no paramaters are passed in a groupby (GH8015)
• Bug in packaging pandas with py2app/cx_Freeze (GH8602, GH8831)
• Bug in groupby signatures that didn’t include *args or **kwargs (GH8733).
• io.data.Options now raises RemoteDataError when no expiry dates are available from Yahoo and
when it receives no data from Yahoo (GH8761), (GH8783).
• Unclear error message in csv parsing when passing dtype and names and the parsed data is a different data type
(GH8833)
• Bug in slicing a multi-index with an empty list and at least one boolean indexer (GH8781)
• io.data.Options now raises RemoteDataError when no expiry dates are available from Yahoo
(GH8761).
• Timedelta kwargs may now be numpy ints and floats (GH8757).
• Fixed several outstanding bugs for Timedelta arithmetic and comparisons (GH8813, GH5963, GH5436).
• sql_schema now generates dialect appropriate CREATE TABLE statements (GH8697)
• slice string method now takes step into account (GH8754)
• Bug in BlockManager where setting values with different type would break block integrity (GH8850)
1.3. v0.15.2 (December 12, 2014)
33
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in DatetimeIndex when using time object as key (GH8667)
• Bug in merge where how=’left’ and sort=False would not preserve left frame order (GH7331)
• Bug in MultiIndex.reindex where reindexing at level would not reorder labels (GH4088)
• Bug in certain operations with dateutil timezones, manifesting with dateutil 2.3 (GH8639)
• Regression in DatetimeIndex iteration with a Fixed/Local offset timezone (GH8890)
• Bug in to_datetime when parsing a nanoseconds using the %f format (GH8989)
• io.data.Options now raises RemoteDataError when no expiry dates are available from Yahoo and
when it receives no data from Yahoo (GH8761), (GH8783).
• Fix: The font size was only set on x axis if vertical or the y axis if horizontal. (GH8765)
• Fixed division by 0 when reading big csv files in python 3 (GH8621)
• Bug in outputing a Multindex with to_html,index=False which would add an extra column (GH8452)
• Imported categorical variables from Stata files retain the ordinal information in the underlying data (GH8836).
• Defined .size attribute across NDFrame objects to provide compat with numpy >= 1.9.1; buggy with
np.array_split (GH8846)
• Skip testing of histogram plots for matplotlib <= 1.2 (GH8648).
• Bug where get_data_google returned object dtypes (GH3995)
• Bug in DataFrame.stack(..., dropna=False) when the DataFrame’s columns is a MultiIndex
whose labels do not reference all its levels. (GH8844)
• Bug in that Option context applied on __enter__ (GH8514)
• Bug in resample that causes a ValueError when resampling across multiple days and the last offset is not calculated from the start of the range (GH8683)
• Bug where DataFrame.plot(kind=’scatter’) fails when checking if an np.array is in the DataFrame
(GH8852)
• Bug in pd.infer_freq/DataFrame.inferred_freq that prevented proper sub-daily frequency inference when the index contained DST days (GH8772).
• Bug where index name was still used when plotting a series with use_index=False (GH8558).
• Bugs when trying to stack multiple columns, when some (or all) of the level names are numbers (GH8584).
• Bug in MultiIndex where __contains__ returns wrong result if index is not lexically sorted or unique
(GH7724)
• BUG CSV: fix problem with trailing whitespace in skipped rows, (GH8679), (GH8661), (GH8983)
• Regression in Timestamp does not parse ‘Z’ zone designator for UTC (GH8771)
• Bug in StataWriter the produces writes strings with 244 characters irrespective of actual size (GH8969)
• Fixed ValueError raised by cummin/cummax when datetime64 Series contains NaT. (GH8965)
• Bug in Datareader returns object dtype if there are missing values (GH8980)
• Bug in plotting if sharex was enabled and index was a timeseries, would show labels on multiple axes (GH3964).
• Bug where passing a unit to the TimedeltaIndex constructor applied the to nano-second conversion twice.
(GH9011).
• Bug in plotting of a period-like array (GH9012)
34
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.4 v0.15.1 (November 9, 2014)
This is a minor bug-fix release from 0.15.0 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users
upgrade to this version.
• Enhancements
• API Changes
• Bug Fixes
1.4.1 API changes
• s.dt.hour and other .dt accessors will now return np.nan for missing values (rather than previously -1),
(GH8689)
In [1]: s = Series(date_range('20130101',periods=5,freq='D'))
In [2]: s.iloc[2] = np.nan
In [3]: s
Out[3]:
0
2013-01-01
1
2013-01-02
2
NaT
3
2013-01-04
4
2013-01-05
dtype: datetime64[ns]
previous behavior:
In [6]: s.dt.hour
Out[6]:
0
0
1
0
2
-1
3
0
4
0
dtype: int64
current behavior:
In [4]: s.dt.hour
Out[4]:
0
0
1
0
2
NaN
3
0
4
0
dtype: float64
• groupby with as_index=False will not add erroneous extra columns to result (GH8582):
In [5]: np.random.seed(2718281)
In [6]: df = pd.DataFrame(np.random.randint(0, 100, (10, 2)),
...:
columns=['jim', 'joe'])
1.4. v0.15.1 (November 9, 2014)
35
pandas: powerful Python data analysis toolkit, Release 0.16.1
...:
In [7]: df.head()
Out[7]:
jim joe
0
61
81
1
96
49
2
55
65
3
72
51
4
77
12
In [8]: ts = pd.Series(5 * np.random.randint(0, 3, 10))
previous behavior:
In [4]: df.groupby(ts, as_index=False).max()
Out[4]:
NaN jim joe
0
0
72
83
1
5
77
84
2
10
96
65
current behavior:
In [9]: df.groupby(ts, as_index=False).max()
Out[9]:
jim joe
0
72
83
1
77
84
2
96
65
• groupby will not erroneously exclude columns if the column name conflics with the grouper name (GH8112):
In [10]: df = pd.DataFrame({'jim': range(5), 'joe': range(5, 10)})
In [11]: df
Out[11]:
jim joe
0
0
5
1
1
6
2
2
7
3
3
8
4
4
9
In [12]: gr = df.groupby(df['jim'] < 2)
previous behavior (excludes 1st column from output):
In [4]: gr.apply(sum)
Out[4]:
joe
jim
False
24
True
11
current behavior:
In [13]: gr.apply(sum)
Out[13]:
jim joe
36
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
jim
False
True
9
1
24
11
• Support for slicing with monotonic decreasing indexes, even if start or stop is not found in the index
(GH7860):
In [14]: s = pd.Series(['a', 'b', 'c', 'd'], [4, 3, 2, 1])
In [15]: s
Out[15]:
4
a
3
b
2
c
1
d
dtype: object
previous behavior:
In [8]: s.loc[3.5:1.5]
KeyError: 3.5
current behavior:
In [16]: s.loc[3.5:1.5]
Out[16]:
3
b
2
c
dtype: object
• io.data.Options has been fixed for a change in the format of the Yahoo Options page (GH8612),
(GH8741)
Note: As a result of a change in Yahoo’s option page layout, when an expiry date is given, Options methods
now return data for a single expiry date. Previously, methods returned all data for the selected month.
The month and year parameters have been undeprecated and can be used to get all options data for a given
month.
If an expiry date that is not valid is given, data for the next expiry after the given date is returned.
Option data frames are now saved on the instance as callsYYMMDD or putsYYMMDD. Previously they were
saved as callsMMYY and putsMMYY. The next expiry is saved as calls and puts.
New features:
– The expiry parameter can now be a single date or a list-like object containing dates.
– A new property expiry_dates was added, which returns all available expiry dates.
Current behavior:
In [17]: from pandas.io.data import Options
In [18]: aapl = Options('aapl','yahoo')
In [19]: aapl.get_call_data().iloc[0:5,0:1]
Out[19]:
Last
Strike Expiry
Type Symbol
70
2015-05-15 call AAPL150515C00070000
1.4. v0.15.1 (November 9, 2014)
53.88
37
pandas: powerful Python data analysis toolkit, Release 0.16.1
75
80
85
90
2015-05-15
2015-05-15
2015-05-15
2015-05-15
call
call
call
call
AAPL150515C00075000
AAPL150515C00080000
AAPL150515C00085000
AAPL150515C00090000
52.42
45.60
39.80
37.45
In [20]: aapl.expiry_dates
Out[20]:
[datetime.date(2015, 5, 15),
datetime.date(2015, 5, 22),
datetime.date(2015, 5, 29),
datetime.date(2015, 6, 5),
datetime.date(2015, 6, 12),
datetime.date(2015, 6, 19),
datetime.date(2015, 6, 26),
datetime.date(2015, 7, 17),
datetime.date(2015, 8, 21),
datetime.date(2015, 10, 16),
datetime.date(2016, 1, 15),
datetime.date(2017, 1, 20)]
In [21]: aapl.get_near_stock_price(expiry=aapl.expiry_dates[0:3]).iloc[0:5,0:1]
Out[21]:
Last
Strike Expiry
Type Symbol
127
2015-05-22 call AAPL150522C00127000 2.45
2015-05-29 call AAPL150529C00127000 2.93
128
2015-05-15 call AAPL150515C00128000 1.21
2015-05-22 call AAPL150522C00128000 1.90
2015-05-29 call AAPL150529C00128000 2.38
See the Options documentation in Remote Data
• pandas now also registers the datetime64 dtype in matplotlib’s units registry to plot such values as datetimes. This is activated once pandas is imported. In previous versions, plotting an array of datetime64
values will have resulted in plotted integer values. To keep the previous behaviour, you can do del
matplotlib.units.registry[np.datetime64] (GH8614).
1.4.2 Enhancements
• concat permits a wider variety of iterables of pandas objects to be passed as the first parameter (GH8645):
In [22]: from collections import deque
In [23]: df1 = pd.DataFrame([1, 2, 3])
In [24]: df2 = pd.DataFrame([4, 5, 6])
previous behavior:
In [7]: pd.concat(deque((df1, df2)))
TypeError: first argument must be a list-like of pandas objects, you passed an object of type "d
current behavior:
In [25]: pd.concat(deque((df1, df2)))
Out[25]:
0
0 1
38
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
2
0
1
2
2
3
4
5
6
• Represent MultiIndex labels with a dtype that utilizes memory based on the level size. In prior versions,
the memory usage was a constant 8 bytes per element in each level. In addition, in prior versions, the reported
memory usage was incorrect as it didn’t show the usage for the memory occupied by the underling data array.
(GH8456)
In [26]: dfi = DataFrame(1,index=pd.MultiIndex.from_product([['a'],range(1000)]),columns=['A'])
previous behavior:
# this was underreported in prior versions
In [1]: dfi.memory_usage(index=True)
Out[1]:
Index
8000 # took about 24008 bytes in < 0.15.1
A
8000
dtype: int64
current behavior:
In [27]: dfi.memory_usage(index=True)
Out[27]:
Index
11020
A
8000
dtype: int64
• Added Index properties is_monotonic_increasing and is_monotonic_decreasing (GH8680).
• Added option to select columns when importing Stata files (GH7935)
• Qualify memory usage in DataFrame.info() by adding + if it is a lower bound (GH8578)
• Raise errors in certain aggregation cases where an argument such as numeric_only is not handled (GH8592).
• Added support for 3-character ISO and non-standard country codes in io.wb.download() (GH8482)
• World Bank data requests now will warn/raise based on an errors argument, as well as a list of hard-coded
country codes and the World Bank’s JSON response. In prior versions, the error messages didn’t look at the
World Bank’s JSON response. Problem-inducing input were simply dropped prior to the request. The issue was
that many good countries were cropped in the hard-coded approach. All countries will work now, but some bad
countries will raise exceptions because some edge cases break the entire response. (GH8482)
• Added option to Series.str.split() to return a DataFrame rather than a Series (GH8428)
• Added option to df.info(null_counts=None|True|False) to override the default display options
and force showing of the null-counts (GH8701)
1.4.3 Bug Fixes
• Bug in unpickling of a CustomBusinessDay object (GH8591)
• Bug in coercing Categorical to a records array, e.g. df.to_records() (GH8626)
• Bug in Categorical not created properly with Series.to_frame() (GH8626)
• Bug in coercing in astype of a Categorical of a passed pd.Categorical (this now raises TypeError
correctly), (GH8626)
1.4. v0.15.1 (November 9, 2014)
39
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in cut/qcut when using Series and retbins=True (GH8589)
• Bug in writing Categorical columns to an SQL database with to_sql (GH8624).
• Bug in comparing Categorical of datetime raising when being compared to a scalar datetime (GH8687)
• Bug in selecting from a Categorical with .iloc (GH8623)
• Bug in groupby-transform with a Categorical (GH8623)
• Bug in duplicated/drop_duplicates with a Categorical (GH8623)
• Bug in Categorical reflected comparison operator raising if the first argument was a numpy array scalar
(e.g. np.int64) (GH8658)
• Bug in Panel indexing with a list-like (GH8710)
• Compat issue is DataFrame.dtypes when options.mode.use_inf_as_null is True (GH8722)
• Bug in read_csv, dialect parameter would not take a string (:issue: 8703)
• Bug in slicing a multi-index level with an empty-list (GH8737)
• Bug in numeric index operations of add/sub with Float/Index Index with numpy arrays (GH8608)
• Bug in setitem with empty indexer and unwanted coercion of dtypes (GH8669)
• Bug in ix/loc block splitting on setitem (manifests with integer-like dtypes, e.g. datetime64) (GH8607)
• Bug when doing label based indexing with integers not found in the index for non-unique but monotonic indexes
(GH8680).
• Bug when indexing a Float64Index with np.nan on numpy 1.7 (GH8980).
• Fix shape attribute for MultiIndex (GH8609)
• Bug in GroupBy where a name conflict between the grouper and columns would break groupby operations
(GH7115, GH8112)
• Fixed a bug where plotting a column y and specifying a label would mutate the index name of the original
DataFrame (GH8494)
• Fix regression in plotting of a DatetimeIndex directly with matplotlib (GH8614).
• Bug in date_range where partially-specified dates would incorporate current date (GH6961)
• Bug in Setting by indexer to a scalar value with a mixed-dtype Panel4d was failing (GH8702)
• Bug where DataReader‘s would fail if one of the symbols passed was invalid. Now returns data for valid
symbols and np.nan for invalid (GH8494)
• Bug in get_quote_yahoo that wouldn’t allow non-float return values (GH5229).
1.5 v0.15.0 (October 18, 2014)
This is a major release from 0.14.1 and includes a small number of API changes, several new features, enhancements,
and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this
version.
Warning: pandas >= 0.15.0 will no longer support compatibility with NumPy versions < 1.7.0. If you want to
use the latest versions of pandas, please upgrade to NumPy >= 1.7.0 (GH7711)
• Highlights include:
– The Categorical type was integrated as a first-class pandas type, see here
40
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
– New scalar type Timedelta, and a new index type TimedeltaIndex, see here
– New datetimelike properties accessor .dt for Series, see Datetimelike Properties
– New DataFrame default display for df.info() to include memory usage, see Memory Usage
– read_csv will now by default ignore blank lines when parsing, see here
– API change in using Indexes in set operations, see here
– Enhancements in the handling of timezones, see here
– A lot of improvements to the rolling and expanding moment funtions, see here
– Internal refactoring of the Index class to no longer sub-class ndarray, see Internal Refactoring
– dropping support for PyTables less than version 3.0.0, and numexpr less than version 2.1 (GH7990)
– Split indexing documentation into Indexing and Selecting Data and MultiIndex / Advanced Indexing
– Split out string methods documentation into Working with Text Data
• Check the API Changes and deprecations before updating
• Other Enhancements
• Performance Improvements
• Bug Fixes
Warning: In 0.15.0 Index has internally been refactored to no longer sub-class ndarray but instead subclass
PandasObject, similarly to the rest of the pandas objects. This change allows very easy sub-classing and
creation of new index types. This should be a transparent change with only very limited API implications (See the
Internal Refactoring)
Warning: The refactorings in Categorical changed the two argument constructor from “codes/labels and
levels” to “values and levels (now called ‘categories’)”. This can lead to subtle bugs. If you use Categorical
directly, please audit your code before updating to this pandas version and change it to use the from_codes()
constructor. See more on Categorical here
1.5.1 New features
Categoricals in Series/DataFrame
Categorical can now be included in Series and DataFrames and gained new methods to manipulate. Thanks to Jan
Schulz for much of this API/implementation. (GH3943, GH5313, GH5314, GH7444, GH7839, GH7848, GH7864,
GH7914, GH7768, GH8006, GH3678, GH8075, GH8076, GH8143, GH8453, GH8518).
For full docs, see the categorical introduction and the API documentation.
In [1]: df = DataFrame({"id":[1,2,3,4,5,6], "raw_grade":['a', 'b', 'b', 'a', 'a', 'e']})
In [2]: df["grade"] = df["raw_grade"].astype("category")
In [3]: df["grade"]
Out[3]:
0
a
1
b
2
b
3
a
1.5. v0.15.0 (October 18, 2014)
41
pandas: powerful Python data analysis toolkit, Release 0.16.1
4
a
5
e
Name: grade, dtype: category
Categories (3, object): [a, b, e]
# Rename the categories
In [4]: df["grade"].cat.categories = ["very good", "good", "very bad"]
# Reorder the categories and simultaneously add the missing categories
In [5]: df["grade"] = df["grade"].cat.set_categories(["very bad", "bad", "medium", "good", "very good
In [6]: df["grade"]
Out[6]:
0
very good
1
good
2
good
3
very good
4
very good
5
very bad
Name: grade, dtype: category
Categories (5, object): [very bad, bad, medium, good, very good]
In [7]: df.sort("grade")
Out[7]:
id raw_grade
grade
5
6
e
very bad
1
2
b
good
2
3
b
good
0
1
a very good
3
4
a very good
4
5
a very good
In [8]: df.groupby("grade").size()
Out[8]:
grade
very bad
1
bad
NaN
medium
NaN
good
2
very good
3
dtype: float64
• pandas.core.group_agg and pandas.core.factor_agg were removed. As an alternative, construct a dataframe and use df.groupby().agg().
• Supplying “codes/labels and levels” to the Categorical constructor is not supported anymore. Supplying
two arguments to the constructor is now interpreted as “values and levels (now called ‘categories’)”. Please
change your code to use the from_codes() constructor.
• The Categorical.labels attribute was renamed to Categorical.codes and is read only. If you want
to manipulate codes, please use one of the API methods on Categoricals.
• The Categorical.levels attribute is renamed to Categorical.categories.
TimedeltaIndex/Scalar
We introduce a new scalar type Timedelta, which is a subclass of datetime.timedelta, and behaves in a
similar manner, but allows compatibility with np.timedelta64 types as well as a host of custom representation,
42
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
parsing, and attributes. This type is very similar to how Timestamp works for datetimes. It is a nice-API box for
the type. See the docs. (GH3009, GH4533, GH8209, GH8187, GH8190, GH7869, GH7661, GH8345, GH8471)
Warning: Timedelta scalars (and TimedeltaIndex) component fields are not the same as the component
fields on a datetime.timedelta object. For example, .seconds on a datetime.timedelta object
returns the total number of seconds combined between hours, minutes and seconds. In contrast, the pandas
Timedelta breaks out hours, minutes, microseconds and nanoseconds separately.
# Timedelta accessor
In [9]: tds = Timedelta('31 days 5 min 3 sec')
In [10]: tds.minutes
Out[10]: 5L
In [11]: tds.seconds
Out[11]: 3L
# datetime.timedelta accessor
# this is 5 minutes * 60 + 3 seconds
In [12]: tds.to_pytimedelta().seconds
Out[12]: 303
Note: this is no longer true starting from v0.16.0, where full compatibility with datetime.timedelta is
introduced. See the 0.16.0 whatsnew entry
Warning: Prior to 0.15.0 pd.to_timedelta would return a Series for list-like/Series input, and a
np.timedelta64 for scalar input. It will now return a TimedeltaIndex for list-like input, Series for
Series input, and Timedelta for scalar input.
The arguments to pd.to_timedelta are now (arg,unit=’ns’,box=True,coerce=False), previously were (arg,box=True,unit=’ns’) as these are more logical.
Consruct a scalar
In [9]: Timedelta('1 days 06:05:01.00003')
Out[9]: Timedelta('1 days 06:05:01.000030')
In [10]: Timedelta('15.5us')
Out[10]: Timedelta('0 days 00:00:00.000015')
In [11]: Timedelta('1 hour 15.5us')
Out[11]: Timedelta('0 days 01:00:00.000015')
# negative Timedeltas have this string repr
# to be more consistent with datetime.timedelta conventions
In [12]: Timedelta('-1us')
Out[12]: Timedelta('-1 days +23:59:59.999999')
# a NaT
In [13]: Timedelta('nan')
Out[13]: NaT
Access fields for a Timedelta
In [14]: td = Timedelta('1 hour 3m 15.5us')
In [15]: td.seconds
Out[15]: 3780L
1.5. v0.15.0 (October 18, 2014)
43
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [16]: td.microseconds
Out[16]: 15L
In [17]: td.nanoseconds
Out[17]: 500L
Construct a TimedeltaIndex
In [18]: TimedeltaIndex(['1 days','1 days, 00:00:05',
....:
np.timedelta64(2,'D'),timedelta(days=2,seconds=2)])
....:
Out[18]:
TimedeltaIndex(['1 days 00:00:00', '1 days 00:00:05', '2 days 00:00:00',
'2 days 00:00:02'],
dtype='timedelta64[ns]', freq=None)
Constructing a TimedeltaIndex with a regular range
In [19]: timedelta_range('1 days',periods=5,freq='D')
Out[19]: TimedeltaIndex(['1 days', '2 days', '3 days', '4 days', '5 days'], dtype='timedelta64[ns]',
In [20]: timedelta_range(start='1 days',end='2 days',freq='30T')
Out[20]:
TimedeltaIndex(['1 days 00:00:00', '1 days 00:30:00', '1 days 01:00:00',
'1 days 01:30:00', '1 days 02:00:00', '1 days 02:30:00',
'1 days 03:00:00', '1 days 03:30:00', '1 days 04:00:00',
'1 days 04:30:00', '1 days 05:00:00', '1 days 05:30:00',
'1 days 06:00:00', '1 days 06:30:00', '1 days 07:00:00',
'1 days 07:30:00', '1 days 08:00:00', '1 days 08:30:00',
'1 days 09:00:00', '1 days 09:30:00', '1 days 10:00:00',
'1 days 10:30:00', '1 days 11:00:00', '1 days 11:30:00',
'1 days 12:00:00', '1 days 12:30:00', '1 days 13:00:00',
'1 days 13:30:00', '1 days 14:00:00', '1 days 14:30:00',
'1 days 15:00:00', '1 days 15:30:00', '1 days 16:00:00',
'1 days 16:30:00', '1 days 17:00:00', '1 days 17:30:00',
'1 days 18:00:00', '1 days 18:30:00', '1 days 19:00:00',
'1 days 19:30:00', '1 days 20:00:00', '1 days 20:30:00',
'1 days 21:00:00', '1 days 21:30:00', '1 days 22:00:00',
'1 days 22:30:00', '1 days 23:00:00', '1 days 23:30:00',
'2 days 00:00:00'],
dtype='timedelta64[ns]', freq='30T')
You can now use a TimedeltaIndex as the index of a pandas object
In [21]: s = Series(np.arange(5),
....:
index=timedelta_range('1 days',periods=5,freq='s'))
....:
In [22]: s
Out[22]:
1 days 00:00:00
0
1 days 00:00:01
1
1 days 00:00:02
2
1 days 00:00:03
3
1 days 00:00:04
4
Freq: S, dtype: int32
You can select with partial string selections
44
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [23]: s['1 day 00:00:02']
Out[23]: 2
In [24]: s['1 day':'1 day 00:00:02']
Out[24]:
1 days 00:00:00
0
1 days 00:00:01
1
1 days 00:00:02
2
dtype: int32
Finally, the combination of TimedeltaIndex with DatetimeIndex allow certain combination operations that
are NaT preserving:
In [25]: tdi = TimedeltaIndex(['1 days',pd.NaT,'2 days'])
In [26]: tdi.tolist()
Out[26]: [Timedelta('1 days 00:00:00'), NaT, Timedelta('2 days 00:00:00')]
In [27]: dti = date_range('20130101',periods=3)
In [28]: dti.tolist()
Out[28]:
[Timestamp('2013-01-01 00:00:00', offset='D'),
Timestamp('2013-01-02 00:00:00', offset='D'),
Timestamp('2013-01-03 00:00:00', offset='D')]
In [29]: (dti + tdi).tolist()
Out[29]: [Timestamp('2013-01-02 00:00:00'), NaT, Timestamp('2013-01-05 00:00:00')]
In [30]: (dti - tdi).tolist()
Out[30]: [Timestamp('2012-12-31 00:00:00'), NaT, Timestamp('2013-01-01 00:00:00')]
• iteration of a Series e.g. list(Series(...)) of timedelta64[ns] would prior to v0.15.0 return
np.timedelta64 for each element. These will now be wrapped in Timedelta.
Memory Usage
Implemented methods to find memory usage of a DataFrame. See the FAQ for more. (GH6852).
A new display option display.memory_usage (see Options and Settings) sets the default behavior of the
memory_usage argument in the df.info() method. By default display.memory_usage is True.
In [31]: dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',
....:
'complex128', 'object', 'bool']
....:
In [32]: n = 5000
In [33]: data = dict([ (t, np.random.randint(100, size=n).astype(t))
....:
for t in dtypes])
....:
In [34]: df = DataFrame(data)
In [35]: df['categorical'] = df['object'].astype('category')
In [36]: df.info()
1.5. v0.15.0 (October 18, 2014)
45
pandas: powerful Python data analysis toolkit, Release 0.16.1
Int64Index: 5000 entries, 0 to 4999
Data columns (total 8 columns):
bool
5000 non-null bool
complex128
5000 non-null complex128
datetime64[ns]
5000 non-null datetime64[ns]
float64
5000 non-null float64
int64
5000 non-null int64
object
5000 non-null object
timedelta64[ns]
5000 non-null timedelta64[ns]
categorical
5000 non-null category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), time
memory usage: 303.5+ KB
Additionally memory_usage() is an available method for a dataframe object which returns the memory usage of
each column.
In [37]: df.memory_usage(index=True)
Out[37]:
Index
40000
bool
5000
complex128
80000
datetime64[ns]
40000
float64
40000
int64
40000
object
20000
timedelta64[ns]
40000
categorical
5800
dtype: int64
.dt accessor
Series has gained an accessor to succinctly return datetime like properties for the values of the Series, if its a
datetime/period like Series. (GH7207) This will return a Series, indexed like the existing Series. See the docs
# datetime
In [38]: s = Series(date_range('20130101 09:10:12',periods=4))
In [39]: s
Out[39]:
0
2013-01-01 09:10:12
1
2013-01-02 09:10:12
2
2013-01-03 09:10:12
3
2013-01-04 09:10:12
dtype: datetime64[ns]
In [40]: s.dt.hour
Out[40]:
0
9
1
9
2
9
3
9
dtype: int64
In [41]: s.dt.second
Out[41]:
0
12
1
12
46
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
12
3
12
dtype: int64
In [42]: s.dt.day
Out[42]:
0
1
1
2
2
3
3
4
dtype: int64
In [43]: s.dt.freq
Out[43]:
This enables nice expressions like this:
In [44]: s[s.dt.day==2]
Out[44]:
1
2013-01-02 09:10:12
dtype: datetime64[ns]
You can easily produce tz aware transformations:
In [45]: stz = s.dt.tz_localize('US/Eastern')
In [46]: stz
Out[46]:
0
2013-01-01
1
2013-01-02
2
2013-01-03
3
2013-01-04
dtype: object
09:10:12-05:00
09:10:12-05:00
09:10:12-05:00
09:10:12-05:00
In [47]: stz.dt.tz
Out[47]:
You can also chain these types of operations:
In [48]: s.dt.tz_localize('UTC').dt.tz_convert('US/Eastern')
Out[48]:
0
2013-01-01 04:10:12-05:00
1
2013-01-02 04:10:12-05:00
2
2013-01-03 04:10:12-05:00
3
2013-01-04 04:10:12-05:00
dtype: object
The .dt accessor works for period and timedelta dtypes.
# period
In [49]: s = Series(period_range('20130101',periods=4,freq='D'))
In [50]: s
Out[50]:
0
2013-01-01
1
2013-01-02
2
2013-01-03
3
2013-01-04
dtype: object
1.5. v0.15.0 (October 18, 2014)
47
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [51]: s.dt.year
Out[51]:
0
2013
1
2013
2
2013
3
2013
dtype: int64
In [52]: s.dt.day
Out[52]:
0
1
1
2
2
3
3
4
dtype: int64
# timedelta
In [53]: s = Series(timedelta_range('1 day 00:00:05',periods=4,freq='s'))
In [54]: s
Out[54]:
0
1 days 00:00:05
1
1 days 00:00:06
2
1 days 00:00:07
3
1 days 00:00:08
dtype: timedelta64[ns]
In [55]: s.dt.days
Out[55]:
0
1
1
1
2
1
3
1
dtype: int64
In [56]: s.dt.seconds
Out[56]:
0
5
1
6
2
7
3
8
dtype: int64
In [57]: s.dt.components
Out[57]:
days hours minutes seconds
0
1
0
0
5
1
1
0
0
6
2
1
0
0
7
3
1
0
0
8
milliseconds
0
0
0
0
microseconds
0
0
0
0
nanoseconds
0
0
0
0
Timezone handling improvements
• tz_localize(None) for tz-aware Timestamp and DatetimeIndex now removes timezone holding
local time, previously this resulted in Exception or TypeError (GH7812)
48
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [58]: ts = Timestamp('2014-08-01 09:00', tz='US/Eastern')
In [59]: ts
Out[59]: Timestamp('2014-08-01 09:00:00-0400', tz='US/Eastern')
In [60]: ts.tz_localize(None)
Out[60]: Timestamp('2014-08-01 09:00:00')
In [61]: didx = DatetimeIndex(start='2014-08-01 09:00', freq='H', periods=10, tz='US/Eastern')
In [62]: didx
Out[62]:
DatetimeIndex(['2014-08-01 09:00:00-04:00', '2014-08-01 10:00:00-04:00',
'2014-08-01 11:00:00-04:00', '2014-08-01 12:00:00-04:00',
'2014-08-01 13:00:00-04:00', '2014-08-01 14:00:00-04:00',
'2014-08-01 15:00:00-04:00', '2014-08-01 16:00:00-04:00',
'2014-08-01 17:00:00-04:00', '2014-08-01 18:00:00-04:00'],
dtype='datetime64[ns]', freq='H', tz='US/Eastern')
In [63]: didx.tz_localize(None)
Out[63]:
DatetimeIndex(['2014-08-01 09:00:00',
'2014-08-01 11:00:00',
'2014-08-01 13:00:00',
'2014-08-01 15:00:00',
'2014-08-01 17:00:00',
dtype='datetime64[ns]',
'2014-08-01 10:00:00',
'2014-08-01 12:00:00',
'2014-08-01 14:00:00',
'2014-08-01 16:00:00',
'2014-08-01 18:00:00'],
freq='H', tz=None)
• tz_localize now accepts the ambiguous keyword which allows for passing an array of bools indicating
whether the date belongs in DST or not, ‘NaT’ for setting transition times to NaT, ‘infer’ for inferring DST/nonDST, and ‘raise’ (default) for an AmbiguousTimeError to be raised. See the docs for more details (GH7943)
• DataFrame.tz_localize and DataFrame.tz_convert now accepts an optional level argument
for localizing a specific level of a MultiIndex (GH7846)
• Timestamp.tz_localize and Timestamp.tz_convert now raise TypeError in error cases, rather
than Exception (GH8025)
• a timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone
(rather than being a naive datetime64[ns]) as object dtype (GH8411)
• Timestamp.__repr__ displays dateutil.tz.tzoffset info (GH7907)
Rolling/Expanding Moments improvements
• rolling_min(), rolling_max(), rolling_cov(), and rolling_corr() now return objects with
all NaN when len(arg) < min_periods <= window rather than raising. (This makes all rolling functions consistent in this behavior). (GH7766)
Prior to 0.15.0
In [64]: s = Series([10, 11, 12, 13])
In [15]: rolling_min(s, window=10, min_periods=5)
ValueError: min_periods (5) must be <= window (4)
New behavior
1.5. v0.15.0 (October 18, 2014)
49
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [65]: rolling_min(s, window=10, min_periods=5)
Out[65]:
0
NaN
1
NaN
2
NaN
3
NaN
dtype: float64
• rolling_max(), rolling_min(), rolling_sum(), rolling_mean(), rolling_median(),
rolling_std(),
rolling_var(),
rolling_skew(),
rolling_kurt(),
rolling_quantile(), rolling_cov(), rolling_corr(), rolling_corr_pairwise(),
rolling_window(), and rolling_apply() with center=True previously would return a result of
the same structure as the input arg with NaN in the final (window-1)/2 entries.
Now the final (window-1)/2 entries of the result are calculated as if the input arg were followed by
(window-1)/2 NaN values (or with shrinking windows, in the case of rolling_apply()). (GH7925,
GH8269)
Prior behavior (note final value is NaN):
In [7]: rolling_sum(Series(range(4)), window=3, min_periods=0, center=True)
Out[7]:
0
1
1
3
2
6
3
NaN
dtype: float64
New behavior (note final value is 5 = sum([2, 3, NaN])):
In [66]: rolling_sum(Series(range(4)), window=3, min_periods=0, center=True)
Out[66]:
0
1
1
3
2
6
3
5
dtype: float64
• rolling_window() now normalizes the weights properly in rolling mean mode (mean=True) so that the
calculated weighted means (e.g. ‘triang’, ‘gaussian’) are distributed about the same means as those calculated
without weighting (i.e. ‘boxcar’). See the note on normalization for further details. (GH7618)
In [67]: s = Series([10.5, 8.8, 11.4, 9.7, 9.3])
Behavior prior to 0.15.0:
In [39]: rolling_window(s, window=3, win_type='triang', center=True)
Out[39]:
0
NaN
1
6.583333
2
6.883333
3
6.683333
4
NaN
dtype: float64
New behavior
In [68]: rolling_window(s, window=3, win_type='triang', center=True)
Out[68]:
0
NaN
50
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
9.875
2
10.325
3
10.025
4
NaN
dtype: float64
• Removed center argument from all expanding_ functions (see list), as the results produced when
center=True did not make much sense. (GH7925)
• Added optional ddof argument to expanding_cov() and rolling_cov(). The default value of 1 is
backwards-compatible. (GH8279)
• Documented the ddof argument to expanding_var(), expanding_std(), rolling_var(), and
rolling_std(). These functions’ support of a ddof argument (with a default value of 1) was previously
undocumented. (GH8064)
• ewma(), ewmstd(), ewmvol(), ewmvar(), ewmcov(), and ewmcorr() now interpret min_periods
in the same manner that the rolling_*() and expanding_*() functions do: a given result entry will be
NaN if the (expanding, in this case) window does not contain at least min_periods values. The previous
behavior was to set to NaN the min_periods entries starting with the first non- NaN value. (GH7977)
Prior behavior (note values start at index 2, which is min_periods after index 0 (the index of the first nonempty value)):
In [69]: s
= Series([1, None, None, None, 2, 3])
In [51]: ewma(s, com=3., min_periods=2)
Out[51]:
0
NaN
1
NaN
2
1.000000
3
1.000000
4
1.571429
5
2.189189
dtype: float64
New behavior (note values start at index 4, the location of the 2nd (since min_periods=2) non-empty value):
In [70]: ewma(s, com=3., min_periods=2)
Out[70]:
0
NaN
1
NaN
2
NaN
3
NaN
4
1.759644
5
2.383784
dtype: float64
• ewmstd(), ewmvol(), ewmvar(), ewmcov(), and ewmcorr() now have an optional adjust argument, just like ewma() does, affecting how the weights are calculated. The default value of adjust is True,
which is backwards-compatible. See Exponentially weighted moment functions for details. (GH7911)
• ewma(), ewmstd(), ewmvol(), ewmvar(), ewmcov(), and ewmcorr() now have an optional
ignore_na argument. When ignore_na=False (the default), missing values are taken into account in
the weights calculation. When ignore_na=True (which reproduces the pre-0.15.0 behavior), missing values
are ignored in the weights calculation. (GH7543)
In [71]: ewma(Series([None, 1., 8.]), com=2.)
Out[71]:
0
NaN
1.5. v0.15.0 (October 18, 2014)
51
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
1.0
2
5.2
dtype: float64
In [72]: ewma(Series([1., None, 8.]), com=2., ignore_na=True)
Out[72]:
0
1.0
1
1.0
2
5.2
dtype: float64
# pre-0.15.0 behavior
In [73]: ewma(Series([1., None, 8.]), com=2., ignore_na=False)
Out[73]:
0
1.000000
1
1.000000
2
5.846154
dtype: float64
# new default
Warning: By default (ignore_na=False) the ewm*() functions’ weights calculation in the presence
of missing values is different than in pre-0.15.0 versions. To reproduce the pre-0.15.0 calculation of weights
in the presence of missing values one must specify explicitly ignore_na=True.
• Bug in expanding_cov(), expanding_corr(), rolling_cov(), rolling_cor(), ewmcov(),
and ewmcorr() returning results with columns sorted by name and producing an error for non-unique
columns; now handles non-unique columns and returns columns in original order (except for the case of two
DataFrames with pairwise=False, where behavior is unchanged) (GH7542)
• Bug in rolling_count() and expanding_*() functions unnecessarily producing error message for
zero-length data (GH8056)
• Bug in rolling_apply()
min_periods=1 (GH8080)
and
expanding_apply()
interpreting
min_periods=0
as
• Bug in expanding_std() and expanding_var() for a single value producing a confusing error message
(GH7900)
• Bug in rolling_std() and rolling_var() for a single value producing 0 rather than NaN (GH7900)
• Bug in ewmstd(), ewmvol(), ewmvar(), and ewmcov() calculation of de-biasing factors when
bias=False (the default). Previously an incorrect constant factor was used, based on adjust=True,
ignore_na=True, and an infinite number of observations. Now a different factor is used for each entry,
based on the actual weights (analogous to the usual N/(N-1) factor). In particular, for a single point a value of
NaN is returned when bias=False, whereas previously a value of (approximately) 0 was returned.
For example, consider the following pre-0.15.0 results for ewmvar(..., bias=False), and the corresponding debiasing factors:
In [74]: s = Series([1., 2., 0., 4.])
In [89]: ewmvar(s, com=2., bias=False)
Out[89]:
0
-2.775558e-16
1
3.000000e-01
2
9.556787e-01
3
3.585799e+00
dtype: float64
In [90]: ewmvar(s, com=2., bias=False) / ewmvar(s, com=2., bias=True)
Out[90]:
52
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1.25
1
1.25
2
1.25
3
1.25
dtype: float64
Note that entry 0 is approximately 0, and the debiasing factors are a constant 1.25. By comparison, the following
0.15.0 results have a NaN for entry 0, and the debiasing factors are decreasing (towards 1.25):
In [75]: ewmvar(s, com=2., bias=False)
Out[75]:
0
NaN
1
0.500000
2
1.210526
3
4.089069
dtype: float64
In [76]: ewmvar(s, com=2., bias=False) / ewmvar(s, com=2., bias=True)
Out[76]:
0
NaN
1
2.083333
2
1.583333
3
1.425439
dtype: float64
See Exponentially weighted moment functions for details. (GH7912)
Improvements in the sql io module
• Added support for a chunksize parameter to to_sql function. This allows DataFrame to be written in
chunks and avoid packet-size overflow errors (GH8062).
• Added support for a chunksize parameter to read_sql function. Specifying this argument will return an
iterator through chunks of the query result (GH2908).
• Added support for writing datetime.date and datetime.time object columns with to_sql (GH6932).
• Added support for specifying a schema to read from/write to with read_sql_table and to_sql
(GH7441, GH7952). For example:
df.to_sql('table', engine, schema='other_schema')
pd.read_sql_table('table', engine, schema='other_schema')
• Added support for writing NaN values with to_sql (GH2754).
• Added support for writing datetime64 columns with to_sql for all database flavors (GH7103).
1.5.2 Backwards incompatible API changes
Breaking changes
API changes related to Categorical (see here for more details):
• The Categorical constructor with two arguments changed from “codes/labels and levels” to “values and
levels (now called ‘categories’)”. This can lead to subtle bugs. If you use Categorical directly, please audit
your code by changing it to use the from_codes() constructor.
An old function call like (prior to 0.15.0):
1.5. v0.15.0 (October 18, 2014)
53
pandas: powerful Python data analysis toolkit, Release 0.16.1
pd.Categorical([0,1,0,2,1], levels=['a', 'b', 'c'])
will have to adapted to the following to keep the same behaviour:
In [2]: pd.Categorical.from_codes([0,1,0,2,1], categories=['a', 'b', 'c'])
Out[2]:
[a, b, a, c, b]
Categories (3, object): [a, b, c]
API changes related to the introduction of the Timedelta scalar (see above for more details):
• Prior to 0.15.0 to_timedelta() would return a Series for list-like/Series input, and a
np.timedelta64 for scalar input. It will now return a TimedeltaIndex for list-like input, Series
for Series input, and Timedelta for scalar input.
For API changes related to the rolling and expanding functions, see detailed overview above.
Other notable API changes:
• Consistency when indexing with .loc and a list-like indexer when no values are found.
In [77]: df = DataFrame([['a'],['b']],index=[1,2])
In [78]: df
Out[78]:
0
1 a
2 b
In prior versions there was a difference in these two constructs:
– df.loc[[3]] would return a frame reindexed by 3 (with all np.nan values)
– df.loc[[3],:] would raise KeyError.
Both will now raise a KeyError. The rule is that at least 1 indexer must be found when using a list-like and
.loc (GH7999)
Furthermore in prior versions these were also different:
– df.loc[[1,3]] would return a frame reindexed by [1,3]
– df.loc[[1,3],:] would raise KeyError.
Both will now return a frame reindex by [1,3]. E.g.
In [79]: df.loc[[1,3]]
Out[79]:
0
1
a
3 NaN
In [80]: df.loc[[1,3],:]
Out[80]:
0
1
a
3 NaN
This can also be seen in multi-axis indexing with a Panel.
In [81]: p = Panel(np.arange(2*3*4).reshape(2,3,4),
....:
items=['ItemA','ItemB'],
....:
major_axis=[1,2,3],
54
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
....:
....:
minor_axis=['A','B','C','D'])
In [82]: p
Out[82]:
Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemB
Major_axis axis: 1 to 3
Minor_axis axis: A to D
The following would raise KeyError prior to 0.15.0:
In [83]: p.loc[['ItemA','ItemD'],:,'D']
Out[83]:
ItemA ItemD
1
3
NaN
2
7
NaN
3
11
NaN
Furthermore, .loc will raise If no values are found in a multi-index with a list-like indexer:
In [84]: s = Series(np.arange(3,dtype='int64'),
....:
index=MultiIndex.from_product([['A'],['foo','bar','baz']],
....:
names=['one','two'])
....:
).sortlevel()
....:
In [85]: s
Out[85]:
one two
A
bar
1
baz
2
foo
0
dtype: int64
In [86]: try:
....:
s.loc[['D']]
....: except KeyError as e:
....:
print("KeyError: " + str(e))
....:
KeyError: 'cannot index a multi-index axis with these keys'
• Assigning values to None now considers the dtype when choosing an ‘empty’ value (GH7941).
Previously, assigning to None in numeric containers changed the dtype to object (or errored, depending on the
call). It now uses NaN:
In [87]: s = Series([1, 2, 3])
In [88]: s.loc[0] = None
In [89]: s
Out[89]:
0
NaN
1
2
2
3
dtype: float64
NaT is now used similarly for datetime containers.
1.5. v0.15.0 (October 18, 2014)
55
pandas: powerful Python data analysis toolkit, Release 0.16.1
For object containers, we now preserve None values (previously these were converted to NaN values).
In [90]: s = Series(["a", "b", "c"])
In [91]: s.loc[0] = None
In [92]: s
Out[92]:
0
None
1
b
2
c
dtype: object
To insert a NaN, you must explicitly use np.nan. See the docs.
• In prior versions, updating a pandas object inplace would not reflect in other python references to this object.
(GH8511, GH5104)
In [93]: s = Series([1, 2, 3])
In [94]: s2 = s
In [95]: s += 1.5
Behavior prior to v0.15.0
# the original object
In [5]: s
Out[5]:
0
2.5
1
3.5
2
4.5
dtype: float64
# a reference to the original object
In [7]: s2
Out[7]:
0
1
1
2
2
3
dtype: int64
This is now the correct behavior
# the original object
In [96]: s
Out[96]:
0
2.5
1
3.5
2
4.5
dtype: float64
# a reference to the original object
In [97]: s2
Out[97]:
0
2.5
1
3.5
2
4.5
dtype: float64
56
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Made both the C-based and Python engines for read_csv and read_table ignore empty lines in input as well as
whitespace-filled lines, as long as sep is not whitespace. This is an API change that can be controlled by the
keyword parameter skip_blank_lines. See the docs (GH4466)
• A timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone
and inserted as object dtype rather than being converted to a naive datetime64[ns] (GH8411).
• Bug in passing a DatetimeIndex with a timezone that was not being retained in DataFrame construction
from a dict (GH7822)
In prior versions this would drop the timezone, now it retains the timezone, but gives a column of object
dtype:
In [98]: i = date_range('1/1/2011', periods=3, freq='10s', tz = 'US/Eastern')
In [99]: i
Out[99]:
DatetimeIndex(['2011-01-01 00:00:00-05:00', '2011-01-01 00:00:10-05:00',
'2011-01-01 00:00:20-05:00'],
dtype='datetime64[ns]', freq='10S', tz='US/Eastern')
In [100]: df = DataFrame( {'a' : i } )
In [101]: df
Out[101]:
0
1
2
a
2011-01-01 00:00:00-05:00
2011-01-01 00:00:10-05:00
2011-01-01 00:00:20-05:00
In [102]: df.dtypes
Out[102]:
a
object
dtype: object
Previously this would have yielded a column of datetime64 dtype, but without timezone info.
The behaviour of assigning a column to an existing dataframe as df[’a’] = i remains unchanged (this already
returned an object column with a timezone).
• When passing multiple levels to stack(), it will now raise a ValueError when the levels aren’t all level
names or all level numbers (GH7660). See Reshaping by stacking and unstacking.
• Raise a ValueError in df.to_hdf with ‘fixed’ format, if df has non-unique columns as the resulting file
will be broken (GH7761)
• SettingWithCopy raise/warnings (according to the option mode.chained_assignment) will now be
issued when setting a value on a sliced mixed-dtype DataFrame using chained-assignment. (GH7845, GH7950)
In [1]: df = DataFrame(np.arange(0,9), columns=['count'])
In [2]: df['group'] = 'b'
In [3]: df.iloc[0:5]['group'] = 'a'
/usr/local/bin/ipython:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.h
1.5. v0.15.0 (October 18, 2014)
57
pandas: powerful Python data analysis toolkit, Release 0.16.1
• merge, DataFrame.merge, and ordered_merge now return the same type as the left argument
(GH7737).
• Previously an enlargement with a mixed-dtype frame would act unlike .append which will preserve dtypes
(related GH2578, GH8176):
In [103]: df = DataFrame([[True, 1],[False, 2]],
.....:
columns=["female","fitness"])
.....:
In [104]: df
Out[104]:
female fitness
0
True
1
1 False
2
In [105]: df.dtypes
Out[105]:
female
bool
fitness
int64
dtype: object
# dtypes are now preserved
In [106]: df.loc[2] = df.loc[1]
In [107]: df
Out[107]:
female fitness
0
True
1
1 False
2
2 False
2
In [108]: df.dtypes
Out[108]:
female
bool
fitness
int64
dtype: object
• Series.to_csv() now returns
DataFrame.to_csv() (GH8215).
a
string
when
path=None,
matching
the
behaviour
of
• read_hdf now raises IOError when a file that doesn’t exist is passed in. Previously, a new, empty file was
created, and a KeyError raised (GH7715).
• DataFrame.info() now ends its output with a newline character (GH8114)
• Concatenating no objects will now raise a ValueError rather than a bare Exception.
• Merge errors will now be sub-classes of ValueError rather than raw Exception (GH8501)
• DataFrame.plot and Series.plot keywords are now have consistent orders (GH8037)
Internal Refactoring
In 0.15.0 Index has internally been refactored to no longer sub-class ndarray but instead subclass
PandasObject, similarly to the rest of the pandas objects. This change allows very easy sub-classing and creation of new index types. This should be a transparent change with only very limited API implications (GH5080,
GH7439, GH7796, GH8024, GH8367, GH7997, GH8522):
58
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• you may need to unpickle pandas version < 0.15.0 pickles using pd.read_pickle rather than
pickle.load. See pickle docs
• when plotting with a PeriodIndex, the matplotlib internal axes will now be arrays of Period rather than a
PeriodIndex (this is similar to how a DatetimeIndex passes arrays of datetimes now)
• MultiIndexes will now raise similary to other pandas objects w.r.t. truth testing, see here (GH7897).
• When plotting a DatetimeIndex directly with matplotlib’s plot function, the axis labels will no longer be formatted as dates but as integers (the internal representation of a datetime64). UPDATE This is fixed in 0.15.1,
see here.
Deprecations
• The attributes Categorical labels and levels attributes are deprecated and renamed to codes and
categories.
• The outtype argument to pd.DataFrame.to_dict has been deprecated in favor of orient. (GH7840)
• The convert_dummies method has been deprecated in favor of get_dummies (GH8140)
• The infer_dst argument in tz_localize will be deprecated in favor of ambiguous to allow for more
flexibility in dealing with DST transitions. Replace infer_dst=True with ambiguous=’infer’ for the
same behavior (GH7943). See the docs for more details.
• The top-level pd.value_range has been deprecated and can be replaced by .describe() (GH8481)
• The Index set operations + and - were deprecated in order to provide these for numeric type operations on
certain index types. + can be replaced by .union() or |, and - by .difference(). Further the method
name Index.diff() is deprecated and can be replaced by Index.difference() (GH8226)
# +
Index(['a','b','c']) + Index(['b','c','d'])
# should be replaced by
Index(['a','b','c']).union(Index(['b','c','d']))
# Index(['a','b','c']) - Index(['b','c','d'])
# should be replaced by
Index(['a','b','c']).difference(Index(['b','c','d']))
• The infer_types argument to read_html() now has no effect and is deprecated (GH7762, GH7032).
Removal of prior version deprecations/changes
• Remove DataFrame.delevel method in favor of DataFrame.reset_index
1.5.3 Enhancements
Enhancements in the importing/exporting of Stata files:
• Added support for bool, uint8, uint16 and uint32 datatypes in to_stata (GH7097, GH7365)
• Added conversion option when importing Stata files (GH8527)
1.5. v0.15.0 (October 18, 2014)
59
pandas: powerful Python data analysis toolkit, Release 0.16.1
• DataFrame.to_stata and StataWriter check string length for compatibility with limitations imposed
in dta files where fixed-width strings must contain 244 or fewer characters. Attempting to write Stata dta files
with strings longer than 244 characters raises a ValueError. (GH7858)
• read_stata and StataReader can import missing data information into a DataFrame by setting
the argument convert_missing to True. When using this options, missing values are returned as
StataMissingValue objects and columns containing missing values have object data type. (GH8045)
Enhancements in the plotting functions:
• Added layout keyword to DataFrame.plot. You can pass a tuple of (rows, columns), one of which
can be -1 to automatically infer (GH6667, GH8071).
• Allow to pass multiple axes to DataFrame.plot, hist and boxplot (GH5353, GH6970, GH7069)
• Added support for c, colormap
kind=’scatter’ (GH7780)
and
colorbar
arguments
for
DataFrame.plot
with
• Histogram from DataFrame.plot with kind=’hist’ (GH7809), See the docs.
• Boxplot from DataFrame.plot with kind=’box’ (GH7998), See the docs.
Other:
• read_csv now has a keyword parameter float_precision which specifies which floating-point converter
the C engine should use during parsing, see here (GH8002, GH8044)
• Added searchsorted method to Series objects (GH7447)
• describe() on mixed-types DataFrames is more flexible. Type-based column filtering is now possible via
the include/exclude arguments. See the docs (GH8164).
In [109]: df = DataFrame({'catA': ['foo', 'foo', 'bar'] * 8,
.....:
'catB': ['a', 'b', 'c', 'd'] * 6,
.....:
'numC': np.arange(24),
.....:
'numD': np.arange(24.) + .5})
.....:
In [110]: df.describe(include=["object"])
Out[110]:
catA catB
count
24
24
unique
2
4
top
foo
d
freq
16
6
In [111]: df.describe(include=["number", "object"], exclude=["float"])
Out[111]:
catA catB
numC
count
24
24 24.000000
unique
2
4
NaN
top
foo
d
NaN
freq
16
6
NaN
mean
NaN NaN 11.500000
std
NaN NaN
7.071068
min
NaN NaN
0.000000
25%
NaN NaN
5.750000
50%
NaN NaN 11.500000
75%
NaN NaN 17.250000
max
NaN NaN 23.000000
Requesting all columns is possible with the shorthand ‘all’
60
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [112]: df.describe(include='all')
Out[112]:
catA catB
numC
numD
count
24
24 24.000000 24.000000
unique
2
4
NaN
NaN
top
foo
d
NaN
NaN
freq
16
6
NaN
NaN
mean
NaN NaN 11.500000 12.000000
std
NaN NaN
7.071068
7.071068
min
NaN NaN
0.000000
0.500000
25%
NaN NaN
5.750000
6.250000
50%
NaN NaN 11.500000 12.000000
75%
NaN NaN 17.250000 17.750000
max
NaN NaN 23.000000 23.500000
Without those arguments, ‘describe‘ will behave as before, including only numerical columns or, if none are,
only categorical columns. See also the docs
• Added split as an option to the orient argument in pd.DataFrame.to_dict. (GH7840)
• The get_dummies method can now be used on DataFrames. By default only catagorical columns are encoded
as 0’s and 1’s, while other columns are left untouched.
In [113]: df = DataFrame({'A': ['a', 'b', 'a'], 'B': ['c', 'c', 'b'],
.....:
'C': [1, 2, 3]})
.....:
In [114]: pd.get_dummies(df)
Out[114]:
C A_a A_b B_b B_c
0 1
1
0
0
1
1 2
0
1
0
1
2 3
1
0
1
0
• PeriodIndex supports resolution as the same as DatetimeIndex (GH7708)
• pandas.tseries.holiday has added support for additional holidays and ways to observe holidays
(GH7070)
• pandas.tseries.holiday.Holiday now supports a list of offsets in Python3 (GH7070)
• pandas.tseries.holiday.Holiday now supports a days_of_week parameter (GH7070)
• GroupBy.nth() now supports selecting multiple nth values (GH7910)
In [115]: business_dates = date_range(start='4/1/2014', end='6/30/2014', freq='B')
In [116]: df = DataFrame(1, index=business_dates, columns=['a', 'b'])
# get the first, 4th, and last date index for each month
In [117]: df.groupby((df.index.year, df.index.month)).nth([0, 3, -1])
Out[117]:
a b
2014-04-01 1 1
2014-04-04 1 1
2014-04-30 1 1
2014-05-01 1 1
2014-05-06 1 1
2014-05-30 1 1
2014-06-02 1 1
1.5. v0.15.0 (October 18, 2014)
61
pandas: powerful Python data analysis toolkit, Release 0.16.1
2014-06-05
2014-06-30
1
1
1
1
• Period and PeriodIndex supports addition/subtraction with timedelta-likes (GH7966)
If Period freq is D, H, T, S, L, U, N, Timedelta-like can be added if the result can have same freq. Otherwise,
only the same offsets can be added.
In [118]: idx = pd.period_range('2014-07-01 09:00', periods=5, freq='H')
In [119]: idx
Out[119]:
PeriodIndex(['2014-07-01 09:00', '2014-07-01 10:00', '2014-07-01 11:00',
'2014-07-01 12:00', '2014-07-01 13:00'],
dtype='int64', freq='H')
In [120]: idx + pd.offsets.Hour(2)
Out[120]:
PeriodIndex(['2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00',
'2014-07-01 14:00', '2014-07-01 15:00'],
dtype='int64', freq='H')
In [121]: idx + Timedelta('120m')
Out[121]:
PeriodIndex(['2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00',
'2014-07-01 14:00', '2014-07-01 15:00'],
dtype='int64', freq='H')
In [122]: idx = pd.period_range('2014-07', periods=5, freq='M')
In [123]: idx
Out[123]: PeriodIndex(['2014-07', '2014-08', '2014-09', '2014-10', '2014-11'], dtype='int64', fr
In [124]: idx + pd.offsets.MonthEnd(3)
Out[124]: PeriodIndex(['2014-10', '2014-11', '2014-12', '2015-01', '2015-02'], dtype='int64', fr
• Added experimental compatibility with openpyxl for versions >= 2.0. The DataFrame.to_excel method
engine keyword now recognizes openpyxl1 and openpyxl2 which will explicitly require openpyxl v1
and v2 respectively, failing if the requested version is not available. The openpyxl engine is a now a metaengine that automatically uses whichever version of openpyxl is installed. (GH7177)
• DataFrame.fillna can now accept a DataFrame as a fill value (GH8377)
• Passing multiple levels to stack() will now work when multiple level numbers are passed (GH7660). See
Reshaping by stacking and unstacking.
• set_names(), set_labels(), and set_levels() methods now take an optional level keyword argument to all modification of specific level(s) of a MultiIndex. Additionally set_names() now accepts a
scalar string value when operating on an Index or on a specific level of a MultiIndex (GH7792)
In [125]: idx = MultiIndex.from_product([['a'], range(3), list("pqr")], names=['foo', 'bar', 'ba
In [126]: idx.set_names('qux', level=0)
Out[126]:
MultiIndex(levels=[[u'a'], [0, 1, 2], [u'p', u'q', u'r']],
labels=[[0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 2, 2, 2], [0, 1, 2, 0, 1, 2,
names=[u'qux', u'bar', u'baz'])
In [127]: idx.set_names(['qux','baz'], level=[0,1])
Out[127]:
62
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
MultiIndex(levels=[[u'a'], [0, 1, 2], [u'p', u'q', u'r']],
labels=[[0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 2, 2, 2], [0, 1, 2, 0, 1, 2,
names=[u'qux', u'baz', u'baz'])
In [128]: idx.set_levels(['a','b','c'], level='bar')
Out[128]:
MultiIndex(levels=[[u'a'], [u'a', u'b', u'c'], [u'p', u'q', u'r']],
labels=[[0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 2, 2, 2], [0, 1, 2, 0, 1, 2,
names=[u'foo', u'bar', u'baz'])
In [129]: idx.set_levels([['a','b','c'],[1,2,3]], level=[1,2])
Out[129]:
MultiIndex(levels=[[u'a'], [u'a', u'b', u'c'], [1, 2, 3]],
labels=[[0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 2, 2, 2], [0, 1, 2, 0, 1, 2,
names=[u'foo', u'bar', u'baz'])
• Index.isin now supports a level argument to specify which index level to use for membership tests
(GH7892, GH7890)
In [1]: idx = MultiIndex.from_product([[0, 1], ['a', 'b', 'c']])
In [2]: idx.values
Out[2]: array([(0, 'a'), (0, 'b'), (0, 'c'), (1, 'a'), (1, 'b'), (1, 'c')], dtype=object)
In [3]: idx.isin(['a', 'c', 'e'], level=1)
Out[3]: array([ True, False, True, True, False,
True], dtype=bool)
• Index now supports duplicated and drop_duplicates. (GH4060)
In [130]: idx = Index([1, 2, 3, 4, 1, 2])
In [131]: idx
Out[131]: Int64Index([1, 2, 3, 4, 1, 2], dtype='int64')
In [132]: idx.duplicated()
Out[132]: array([False, False, False, False,
True,
True], dtype=bool)
In [133]: idx.drop_duplicates()
Out[133]: Int64Index([1, 2, 3, 4], dtype='int64')
• add copy=True argument to pd.concat to enable pass thru of complete blocks (GH8252)
• Added support for numpy 1.8+ data types (bool_, int_, float_, string_) for conversion to R dataframe
(GH8400)
1.5.4 Performance
• Performance improvements in DatetimeIndex.__iter__ to allow faster iteration (GH7683)
• Performance improvements in Period creation (and PeriodIndex setitem) (GH5155)
• Improvements in Series.transform for significant performance gains (revised) (GH6496)
• Performance improvements in StataReader when reading large files (GH8040, GH8073)
• Performance improvements in StataWriter when writing large files (GH8079)
• Performance and memory usage improvements in multi-key groupby (GH8128)
1.5. v0.15.0 (October 18, 2014)
63
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Performance improvements in groupby .agg and .apply where builtins max/min were not mapped to
numpy/cythonized versions (GH7722)
• Performance improvement in writing to sql (to_sql) of up to 50% (GH8208).
• Performance benchmarking of groupby for large value of ngroups (GH6787)
• Performance improvement in CustomBusinessDay, CustomBusinessMonth (GH8236)
• Performance improvement for MultiIndex.values for multi-level indexes containing datetimes (GH8543)
1.5.5 Bug Fixes
• Bug in pivot_table, when using margins and a dict aggfunc (GH8349)
• Bug in read_csv where squeeze=True would return a view (GH8217)
• Bug in checking of table name in read_sql in certain cases (GH7826).
• Bug in DataFrame.groupby where Grouper does not recognize level when frequency is specified
(GH7885)
• Bug in multiindexes dtypes getting mixed up when DataFrame is saved to SQL table (GH8021)
• Bug in Series 0-division with a float and integer operand dtypes (GH7785)
• Bug in Series.astype("unicode") not calling unicode on the values correctly (GH7758)
• Bug in DataFrame.as_matrix() with mixed datetime64[ns] and timedelta64[ns] dtypes
(GH7778)
• Bug in HDFStore.select_column() not preserving UTC timezone info when selecting a
DatetimeIndex (GH7777)
• Bug in to_datetime when format=’%Y%m%d’ and coerce=True are specified, where previously an
object array was returned (rather than a coerced time-series with NaT), (GH7930)
• Bug in DatetimeIndex and PeriodIndex in-place addition and subtraction cause different result from
normal one (GH6527)
• Bug in adding and subtracting PeriodIndex with PeriodIndex raise TypeError (GH7741)
• Bug in combine_first with PeriodIndex data raises TypeError (GH3367)
• Bug in multi-index slicing with missing indexers (GH7866)
• Bug in multi-index slicing with various edge cases (GH8132)
• Regression in multi-index indexing with a non-scalar type object (GH7914)
• Bug in Timestamp comparisons with == and int64 dtype (GH8058)
• Bug in pickles contains DateOffset may raise AttributeError when normalize attribute is reffered
internally (GH7748)
• Bug in Panel when using major_xs and copy=False is passed (deprecation warning fails because of
missing warnings) (GH8152).
• Bug in pickle deserialization that failed for pre-0.14.1 containers with dup items trying to avoid ambiguity when
matching block and manager items, when there’s only one block there’s no ambiguity (GH7794)
• Bug in putting a PeriodIndex into a Series would convert to int64 dtype, rather than object of
Periods (GH7932)
• Bug in HDFStore iteration when passing a where (GH8014)
64
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in DataFrameGroupby.transform when transforming with a passed non-sorted key (GH8046,
GH8430)
• Bug in repeated timeseries line and area plot may result in ValueError or incorrect kind (GH7733)
• Bug in inference in a MultiIndex with datetime.date inputs (GH7888)
• Bug in get where an IndexError would not cause the default value to be returned (GH7725)
• Bug in offsets.apply, rollforward and rollback may reset nanosecond (GH7697)
• Bug in offsets.apply, rollforward and rollback may raise AttributeError if Timestamp
has dateutil tzinfo (GH7697)
• Bug in sorting a multi-index frame with a Float64Index (GH8017)
• Bug in inconsistent panel setitem with a rhs of a DataFrame for alignment (GH7763)
• Bug in is_superperiod and is_subperiod cannot handle higher frequencies than S (GH7760, GH7772,
GH7803)
• Bug in 32-bit platforms with Series.shift (GH8129)
• Bug in PeriodIndex.unique returns int64 np.ndarray (GH7540)
• Bug in groupby.apply with a non-affecting mutation in the function (GH8467)
• Bug in DataFrame.reset_index which has MultiIndex
DatetimeIndex with tz raises ValueError (GH7746, GH7793)
contains
PeriodIndex
or
• Bug in DataFrame.plot with subplots=True may draw unnecessary minor xticks and yticks (GH7801)
• Bug in StataReader which did not read variable labels in 117 files due to difference between Stata documentation and implementation (GH7816)
• Bug in StataReader where strings were always converted to 244 characters-fixed width irrespective of underlying string size (GH7858)
• Bug in DataFrame.plot and Series.plot may ignore rot and fontsize keywords (GH7844)
• Bug in DatetimeIndex.value_counts doesn’t preserve tz (GH7735)
• Bug in PeriodIndex.value_counts results in Int64Index (GH7735)
• Bug in DataFrame.join when doing left join on index and there are multiple matches (GH5391)
• Bug in GroupBy.transform() where int groups with a transform that didn’t preserve the index were incorrectly truncated (GH7972).
• Bug in groupby where callable objects without name attributes would take the wrong path, and produce a
DataFrame instead of a Series (GH7929)
• Bug in groupby error message when a DataFrame grouping column is duplicated (GH7511)
• Bug in read_html where the infer_types argument forced coercion of date-likes incorrectly (GH7762,
GH7032).
• Bug in Series.str.cat with an index which was filtered as to not include the first item (GH7857)
• Bug in Timestamp cannot parse nanosecond from string (GH7878)
• Bug in Timestamp with string offset and tz results incorrect (GH7833)
• Bug in tslib.tz_convert and tslib.tz_convert_single may return different results (GH7798)
• Bug in DatetimeIndex.intersection of non-overlapping timestamps with tz raises IndexError
(GH7880)
1.5. v0.15.0 (October 18, 2014)
65
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in alignment with TimeOps and non-unique indexes (GH8363)
• Bug in GroupBy.filter() where fast path vs. slow path made the filter return a non scalar value that
appeared valid but wasn’t (GH7870).
• Bug in date_range()/DatetimeIndex() when the timezone was inferred from input dates yet incorrect
times were returned when crossing DST boundaries (GH7835, GH7901).
• Bug in to_excel() where a negative sign was being prepended to positive infinity and was absent for negative
infinity (GH7949)
• Bug in area plot draws legend with incorrect alpha when stacked=True (GH8027)
• Period and PeriodIndex addition/subtraction with np.timedelta64 results in incorrect internal representations (GH7740)
• Bug in Holiday with no offset or observance (GH7987)
• Bug in DataFrame.to_latex formatting when columns or index is a MultiIndex (GH7982).
• Bug in DateOffset around Daylight Savings Time produces unexpected results (GH5175).
• Bug in DataFrame.shift where empty columns would throw ZeroDivisionError on numpy 1.7
(GH8019)
• Bug in installation where html_encoding/*.html wasn’t installed and therefore some tests were not running correctly (GH7927).
• Bug in read_html where bytes objects were not tested for in _read (GH7927).
• Bug in DataFrame.stack() when one of the column levels was a datelike (GH8039)
• Bug in broadcasting numpy scalars with DataFrame (GH8116)
• Bug in pivot_table performed with nameless index and columns raises KeyError (GH8103)
• Bug in DataFrame.plot(kind=’scatter’) draws points and errorbars with different colors when the
color is specified by c keyword (GH8081)
• Bug in Float64Index where iat and at were not testing and were failing (GH8092).
• Bug in DataFrame.boxplot() where y-limits were not set correctly when producing multiple axes
(GH7528, GH5517).
• Bug in read_csv where line comments were not handled correctly given a custom line terminator or
delim_whitespace=True (GH8122).
• Bug in read_html where empty tables caused a StopIteration (GH7575)
• Bug in casting when setting a column in a same-dtype block (GH7704)
• Bug in accessing groups from a GroupBy when the original grouper was a tuple (GH8121).
• Bug in .at that would accept integer indexers on a non-integer index and do fallback (GH7814)
• Bug with kde plot and NaNs (GH8182)
• Bug in GroupBy.count with float32 data type were nan values were not excluded (GH8169).
• Bug with stacked barplots and NaNs (GH8175).
• Bug in resample with non evenly divisible offsets (e.g. ‘7s’) (GH8371)
• Bug in interpolation methods with the limit keyword when no values needed interpolating (GH7173).
• Bug where col_space was ignored in DataFrame.to_string() when header=False (GH8230).
66
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug with DatetimeIndex.asof incorrectly matching partial strings and returning the wrong date
(GH8245).
• Bug in plotting methods modifying the global matplotlib rcParams (GH8242).
• Bug in DataFrame.__setitem__ that caused errors when setting a dataframe column to a sparse array
(GH8131)
• Bug where Dataframe.boxplot() failed when entire column was empty (GH8181).
• Bug with messed variables in radviz visualization (GH8199).
• Bug in interpolation methods with the limit keyword when no values needed interpolating (GH7173).
• Bug where col_space was ignored in DataFrame.to_string() when header=False (GH8230).
• Bug in to_clipboard that would clip long column data (GH8305)
• Bug in DataFrame terminal display: Setting max_column/max_rows to zero did not trigger auto-resizing of
dfs to fit terminal width/height (GH7180).
• Bug in OLS where running with “cluster” and “nw_lags” parameters did not work correctly, but also did not
throw an error (GH5884).
• Bug in DataFrame.dropna that interpreted non-existent columns in the subset argument as the ‘last column’
(GH8303)
• Bug in Index.intersection on non-monotonic non-unique indexes (GH8362).
• Bug in masked series assignment where mismatching types would break alignment (GH8387)
• Bug in NDFrame.equals gives false negatives with dtype=object (GH8437)
• Bug in assignment with indexer where type diversity would break alignment (GH8258)
• Bug in NDFrame.loc indexing when row/column names were lost when target was a list/ndarray (GH6552)
• Regression in NDFrame.loc indexing when rows/columns were converted to Float64Index if target was an
empty list/ndarray (GH7774)
• Bug in Series that allows it to be indexed by a DataFrame which has unexpected results. Such indexing is
no longer permitted (GH8444)
• Bug in item assignment of a DataFrame with multi-index columns where right-hand-side columns were not
aligned (GH7655)
• Suppress FutureWarning generated by NumPy when comparing object arrays containing NaN for equality
(GH7065)
• Bug in DataFrame.eval() where the dtype of the not operator (~) was not correctly inferred as bool.
1.6 v0.14.1 (July 11, 2014)
This is a minor release from 0.14.0 and includes a small number of API changes, several new features, enhancements,
and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this
version.
• Highlights include:
– New methods select_dtypes() to select columns based on the dtype and sem() to calculate the
standard error of the mean.
– Support for dateutil timezones (see docs).
1.6. v0.14.1 (July 11, 2014)
67
pandas: powerful Python data analysis toolkit, Release 0.16.1
– Support for ignoring full line comments in the read_csv() text parser.
– New documentation section on Options and Settings.
– Lots of bug fixes.
• Enhancements
• API Changes
• Performance Improvements
• Experimental Changes
• Bug Fixes
1.6.1 API changes
• Openpyxl now raises a ValueError on construction of the openpyxl writer instead of warning on pandas import
(GH7284).
• For StringMethods.extract, when no match is found, the result - only containing NaN values - now also
has dtype=object instead of float (GH7242)
• Period objects no longer raise a TypeError when compared using == with another object that isn’t a
Period. Instead when comparing a Period with another object using == if the other object isn’t a Period
False is returned. (GH7376)
• Previously, the behaviour on resetting the time or not in offsets.apply, rollforward and rollback
operations differed between offsets. With the support of the normalize keyword for all offsets(see below) with a default value of False (preserve time), the behaviour changed for certain offsets (BusinessMonthBegin, MonthEnd, BusinessMonthEnd, CustomBusinessMonthEnd, BusinessYearBegin, LastWeekOfMonth,
FY5253Quarter, LastWeekOfMonth, Easter):
In [6]: from pandas.tseries import offsets
In [7]: d = pd.Timestamp('2014-01-01 09:00')
# old behaviour < 0.14.1
In [8]: d + offsets.MonthEnd()
Out[8]: Timestamp('2014-01-31 00:00:00')
Starting from 0.14.1 all offsets preserve time by default.
normalize=True
The old behaviour can be obtained with
# new behaviour
In [1]: d + offsets.MonthEnd()
Out[1]: Timestamp('2014-01-31 09:00:00')
In [2]: d + offsets.MonthEnd(normalize=True)
Out[2]: Timestamp('2014-01-31 00:00:00')
Note that for the other offsets the default behaviour did not change.
• Add back #N/A N/A as a default NA value in text parsing, (regresion from 0.12) (GH5521)
• Raise a TypeError on inplace-setting with a .where and a non np.nan value as this is inconsistent with a
set-item expression like df[mask] = None (GH7656)
68
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.6.2 Enhancements
• Add dropna argument to value_counts and nunique (GH5569).
• Add select_dtypes() method to allow selection of columns based on dtype (GH7316). See the docs.
• All offsets suppports the normalize keyword to specify whether offsets.apply, rollforward
and rollback resets the time (hour, minute, etc) or not (default False, preserves time) (GH7156):
In [3]: import pandas.tseries.offsets as offsets
In [4]: day = offsets.Day()
In [5]: day.apply(Timestamp('2014-01-01 09:00'))
Out[5]: Timestamp('2014-01-02 09:00:00')
In [6]: day = offsets.Day(normalize=True)
In [7]: day.apply(Timestamp('2014-01-01 09:00'))
Out[7]: Timestamp('2014-01-02 00:00:00')
• PeriodIndex is represented as the same format as DatetimeIndex (GH7601)
• StringMethods now work on empty Series (GH7242)
• The file parsers read_csv and read_table now ignore line comments provided by the parameter comment,
which accepts only a single character for the C reader. In particular, they allow for comments before file data
begins (GH2685)
• Add NotImplementedError for simultaneous use of chunksize and nrows for read_csv() (GH6774).
• Tests for basic reading of public S3 buckets now exist (GH7281).
• read_html now sports an encoding argument that is passed to the underlying parser library. You can use
this to read non-ascii encoded web pages (GH7323).
• read_excel now supports reading from URLs in the same way that read_csv does. (GH6809)
• Support for dateutil timezones, which can now be used in the same way as pytz timezones across pandas.
(GH4688)
In [8]: rng = date_range('3/6/2012 00:00', periods=10, freq='D',
...:
tz='dateutil/Europe/London')
...:
In [9]: rng.tz
Out[9]: tzfile('Europe/Belfast')
See the docs.
• Implemented sem (standard error of the mean) operation for Series, DataFrame, Panel, and Groupby
(GH6897)
• Add nlargest and nsmallest to the Series groupby whitelist, which means you can now use these
methods on a SeriesGroupBy object (GH7053).
• All offsets apply, rollforward and rollback can now handle np.datetime64, previously results in
ApplyTypeError (GH7452)
• Period and PeriodIndex can contain NaT in its values (GH7485)
• Support pickling Series, DataFrame and Panel objects with non-unique labels along item axis (index,
columns and items respectively) (GH7370).
1.6. v0.14.1 (July 11, 2014)
69
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Improved inference of datetime/timedelta with mixed null objects. Regression from 0.13.1 in interpretation of
an object Index with all null elements (GH7431)
1.6.3 Performance
• Improvements in dtype inference for numeric operations involving yielding performance gains for dtypes:
int64, timedelta64, datetime64 (GH7223)
• Improvements in Series.transform for significant performance gains (GH6496)
• Improvements in DataFrame.transform with ufuncs and built-in grouper functions for signifcant performance
gains (GH7383)
• Regression in groupby aggregation of datetime64 dtypes (GH7555)
• Improvements in MultiIndex.from_product for large iterables (GH7627)
1.6.4 Experimental
• pandas.io.data.Options has a new method, get_all_data method, and now consistently returns a
multi-indexed DataFrame, see the docs. (GH5602)
• io.gbq.read_gbq and io.gbq.to_gbq were refactored to remove the dependency on the Google
bq.py command line client. This submodule now uses httplib2 and the Google apiclient and
oauth2client API client libraries which should be more stable and, therefore, reliable than bq.py. See the
docs. (GH6937).
1.6.5 Bug Fixes
• Bug in DataFrame.where with a symmetric shaped frame and a passed other of a DataFrame (GH7506)
• Bug in Panel indexing with a multi-index axis (GH7516)
• Regression in datetimelike slice indexing with a duplicated index and non-exact end-points (GH7523)
• Bug in setitem with list-of-lists and single vs mixed types (GH7551:)
• Bug in timeops with non-aligned Series (GH7500)
• Bug in timedelta inference when assigning an incomplete Series (GH7592)
• Bug in groupby .nth with a Series and integer-like column name (GH7559)
• Bug in Series.get with a boolean accessor (GH7407)
• Bug in value_counts where NaT did not qualify as missing (NaN) (GH7423)
• Bug in to_timedelta that accepted invalid units and misinterpreted ‘m/h’ (GH7611, GH6423)
• Bug in line plot doesn’t set correct xlim if secondary_y=True (GH7459)
• Bug in grouped hist and scatter plots use old figsize default (GH7394)
• Bug in plotting subplots with DataFrame.plot, hist clears passed ax even if the number of subplots is
one (GH7391).
• Bug in plotting subplots with DataFrame.boxplot with by kw raises ValueError if the number of
subplots exceeds 1 (GH7391).
• Bug in subplots displays ticklabels and labels in different rule (GH5897)
70
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in Panel.apply with a multi-index as an axis (GH7469)
• Bug in DatetimeIndex.insert doesn’t preserve name and tz (GH7299)
• Bug in DatetimeIndex.asobject doesn’t preserve name (GH7299)
• Bug in multi-index slicing with datetimelike ranges (strings and Timestamps), (GH7429)
• Bug in Index.min and max doesn’t handle nan and NaT properly (GH7261)
• Bug in PeriodIndex.min/max results in int (GH7609)
• Bug in resample where fill_method was ignored if you passed how (GH2073)
• Bug in TimeGrouper doesn’t exclude column specified by key (GH7227)
• Bug in DataFrame and Series bar and barh plot raises TypeError when bottom and left keyword is
specified (GH7226)
• Bug in DataFrame.hist raises TypeError when it contains non numeric column (GH7277)
• Bug in Index.delete does not preserve name and freq attributes (GH7302)
• Bug in DataFrame.query()/eval where local string variables with the @ sign were being treated as
temporaries attempting to be deleted (GH7300).
• Bug in Float64Index which didn’t allow duplicates (GH7149).
• Bug in DataFrame.replace() where truthy values were being replaced (GH7140).
• Bug in StringMethods.extract() where a single match group Series would use the matcher’s name
instead of the group name (GH7313).
• Bug in isnull() when mode.use_inf_as_null == True where isnull wouldn’t test True when it
encountered an inf/-inf (GH7315).
• Bug in inferred_freq results in None for eastern hemisphere timezones (GH7310)
• Bug in Easter returns incorrect date when offset is negative (GH7195)
• Bug in broadcasting with .div, integer dtypes and divide-by-zero (GH7325)
• Bug in CustomBusinessDay.apply raiases NameError when np.datetime64 object is passed
(GH7196)
• Bug in MultiIndex.append, concat and pivot_table don’t preserve timezone (GH6606)
• Bug in .loc with a list of indexers on a single-multi index level (that is not nested) (GH7349)
• Bug in Series.map when mapping a dict with tuple keys of different lengths (GH7333)
• Bug all StringMethods now work on empty Series (GH7242)
• Fix delegation of read_sql to read_sql_query when query does not contain ‘select’ (GH7324).
• Bug where a string column name assignment to a DataFrame with a Float64Index raised a TypeError
during a call to np.isnan (GH7366).
• Bug where NDFrame.replace() didn’t correctly replace objects with Period values (GH7379).
• Bug in .ix getitem should always return a Series (GH7150)
• Bug in multi-index slicing with incomplete indexers (GH7399)
• Bug in multi-index slicing with a step in a sliced level (GH7400)
• Bug where negative indexers in DatetimeIndex were not correctly sliced (GH7408)
• Bug where NaT wasn’t repr’d correctly in a MultiIndex (GH7406, GH7409).
1.6. v0.14.1 (July 11, 2014)
71
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug where bool objects were converted to nan in convert_objects (GH7416).
• Bug in quantile ignoring the axis keyword argument (:issue‘7306‘)
• Bug where nanops._maybe_null_out doesn’t work with complex numbers (GH7353)
• Bug in several nanops functions when axis==0 for 1-dimensional nan arrays (GH7354)
• Bug where nanops.nanmedian doesn’t work when axis==None (GH7352)
• Bug where nanops._has_infs doesn’t work with many dtypes (GH7357)
• Bug in StataReader.data where reading a 0-observation dta failed (GH7369)
• Bug in StataReader when reading Stata 13 (117) files containing fixed width strings (GH7360)
• Bug in StataWriter where encoding was ignored (GH7286)
• Bug in DatetimeIndex comparison doesn’t handle NaT properly (GH7529)
• Bug in passing input with tzinfo to some offsets apply, rollforward or rollback resets tzinfo or
raises ValueError (GH7465)
• Bug in DatetimeIndex.to_period, PeriodIndex.asobject, PeriodIndex.to_timestamp
doesn’t preserve name (GH7485)
• Bug in DatetimeIndex.to_period and PeriodIndex.to_timestanp handle NaT incorrectly
(GH7228)
• Bug in offsets.apply, rollforward and rollback may return normal datetime (GH7502)
• Bug in resample raises ValueError when target contains NaT (GH7227)
• Bug in Timestamp.tz_localize resets nanosecond info (GH7534)
• Bug in DatetimeIndex.asobject raises ValueError when it contains NaT (GH7539)
• Bug in Timestamp.__new__ doesn’t preserve nanosecond properly (GH7610)
• Bug in Index.astype(float) where it would return an object dtype Index (GH7464).
• Bug in DataFrame.reset_index loses tz (GH3950)
• Bug in DatetimeIndex.freqstr raises AttributeError when freq is None (GH7606)
• Bug in GroupBy.size created by TimeGrouper raises AttributeError (GH7453)
• Bug in single column bar plot is misaligned (GH7498).
• Bug in area plot with tz-aware time series raises ValueError (GH7471)
• Bug in non-monotonic Index.union may preserve name incorrectly (GH7458)
• Bug in DatetimeIndex.intersection doesn’t preserve timezone (GH4690)
• Bug in rolling_var where a window larger than the array would raise an error(GH7297)
• Bug with last plotted timeseries dictating xlim (GH2960)
• Bug with secondary_y axis not being considered for timeseries xlim (GH3490)
• Bug in Float64Index assignment with a non scalar indexer (GH7586)
• Bug in pandas.core.strings.str_contains does not properly match in a case insensitive fashion
when regex=False and case=False (GH7505)
• Bug in expanding_cov, expanding_corr, rolling_cov, and rolling_corr for two arguments
with mismatched index (GH7512)
• Bug in to_sql taking the boolean column as text column (GH7678)
72
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in grouped hist doesn’t handle rot kw and sharex kw properly (GH7234)
• Bug in .loc performing fallback integer indexing with object dtype indices (GH7496)
• Bug (regression) in PeriodIndex constructor when passed Series objects (GH7701).
1.7 v0.14.0 (May 31 , 2014)
This is a major release from 0.13.1 and includes a small number of API changes, several new features, enhancements,
and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this
version.
• Highlights include:
– Officially support Python 3.4
– SQL interfaces updated to use sqlalchemy, See Here.
– Display interface changes, See Here
– MultiIndexing Using Slicers, See Here.
– Ability to join a singly-indexed DataFrame with a multi-indexed DataFrame, see Here
– More consistency in groupby results and more flexible groupby specifications, See Here
– Holiday calendars are now supported in CustomBusinessDay, see Here
– Several improvements in plotting functions, including: hexbin, area and pie plots, see Here.
– Performance doc section on I/O operations, See Here
• Other Enhancements
• API Changes
• Text Parsing API Changes
• Groupby API Changes
• Performance Improvements
• Prior Deprecations
• Deprecations
• Known Issues
• Bug Fixes
Warning: In 0.14.0 all NDFrame based containers have undergone significant internal refactoring. Before that
each block of homogeneous data had its own labels and extra care was necessary to keep those in sync with the
parent container’s labels. This should not have any visible user/API behavior changes (GH6745)
1.7.1 API changes
• read_excel uses 0 as the default sheet (GH6573)
• iloc will now accept out-of-bounds indexers for slices, e.g. a value that exceeds the length of the object
being indexed. These will be excluded. This will make pandas conform more with python/numpy indexing of
out-of-bounds values. A single indexer that is out-of-bounds and drops the dimensions of the object will still
1.7. v0.14.0 (May 31 , 2014)
73
pandas: powerful Python data analysis toolkit, Release 0.16.1
raise IndexError (GH6296, GH6299). This could result in an empty axis (e.g. an empty DataFrame being
returned)
In [1]: dfl = DataFrame(np.random.randn(5,2),columns=list('AB'))
In [2]: dfl
Out[2]:
A
B
0 1.583584 -0.438313
1 -0.402537 -0.780572
2 -0.141685 0.542241
3 0.370966 -0.251642
4 0.787484 1.666563
In [3]: dfl.iloc[:,2:3]
Out[3]:
Empty DataFrame
Columns: []
Index: [0, 1, 2, 3, 4]
In [4]: dfl.iloc[:,1:3]
Out[4]:
B
0 -0.438313
1 -0.780572
2 0.542241
3 -0.251642
4 1.666563
In [5]: dfl.iloc[4:6]
Out[5]:
A
B
4 0.787484 1.666563
These are out-of-bounds selections
dfl.iloc[[4,5,6]]
IndexError: positional indexers are out-of-bounds
dfl.iloc[:,4]
IndexError: single positional indexer is out-of-bounds
• Slicing with negative start, stop & step values handles corner cases better (GH6531):
– df.iloc[:-len(df)] is now empty
– df.iloc[len(df)::-1] now enumerates all elements in reverse
• The DataFrame.interpolate() keyword downcast default has been changed from infer to None.
This is to preseve the original dtype unless explicitly requested otherwise (GH6290).
• When converting a dataframe to HTML it used to return Empty DataFrame. This special case has been removed,
instead a header with the column names is returned (GH6062).
• Series
and
Index
now
internall
share
more
common
operations,
e.g.
factorize(),nunique(),value_counts() are now supported on Index types as well.
The Series.weekday property from is removed from Series for API consistency.
Using a
DatetimeIndex/PeriodIndex method on a Series will now raise a TypeError. (GH4551, GH4056,
GH5519, GH6380, GH7206).
• Add
74
is_month_start,
is_month_end,
is_quarter_start,
is_quarter_end,
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
is_year_start, is_year_end accessors for DateTimeIndex / Timestamp which return a
boolean array of whether the timestamp(s) are at the start/end of the month/quarter/year defined by the
frequency of the DateTimeIndex / Timestamp (GH4565, GH6998)
• Local variable usage has changed in pandas.eval()/DataFrame.eval()/DataFrame.query()
(GH5987). For the DataFrame methods, two things have changed
– Column names are now given precedence over locals
– Local variables must be referred to explicitly. This means that even if you have a local variable that is not
a column you must still refer to it with the ’@’ prefix.
– You can have an expression like df.query(’@a < a’) with no complaints from pandas about ambiguity of the name a.
– The top-level pandas.eval() function does not allow you use the ’@’ prefix and provides you with
an error message telling you so.
– NameResolutionError was removed because it isn’t necessary anymore.
• Define and document the order of column vs index names in query/eval (GH6676)
• concat will now concatenate mixed Series and DataFrames using the Series name or numbering columns as
needed (GH2385). See the docs
• Slicing and advanced/boolean indexing operations on Index classes as well as Index.delete() and
Index.drop() methods will no longer change the type of the resulting index (GH6440, GH7040)
In [6]: i = pd.Index([1, 2, 3, 'a' , 'b', 'c'])
In [7]: i[[0,1,2]]
Out[7]: Index([1, 2, 3], dtype='object')
In [8]: i.drop(['a', 'b', 'c'])
Out[8]: Int64Index([1, 2, 3], dtype='int64')
Previously, the above operation would return Int64Index.
Index.astype()
If you’d like to do this manually, use
In [9]: i[[0,1,2]].astype(np.int_)
Out[9]: Int64Index([1, 2, 3], dtype='int32')
• set_index no longer converts MultiIndexes to an Index of tuples. For example, the old behavior returned an
Index in this case (GH6459):
# Old behavior, casted MultiIndex to an Index
In [10]: tuple_ind
Out[10]: Index([(u'a', u'c'), (u'a', u'd'), (u'b', u'c'), (u'b', u'd')], dtype='object')
In [11]: df_multi.set_index(tuple_ind)
Out[11]:
0
1
(a, c) 0.471435 -1.190976
(a, d) 1.432707 -0.312652
(b, c) -0.720589 0.887163
(b, d) 0.859588 -0.636524
# New behavior
In [12]: mi
Out[12]:
MultiIndex(levels=[[u'a', u'b'], [u'c', u'd']],
labels=[[0, 0, 1, 1], [0, 1, 0, 1]])
1.7. v0.14.0 (May 31 , 2014)
75
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [13]: df_multi.set_index(mi)
Out[13]:
0
1
a c 0.471435 -1.190976
d 1.432707 -0.312652
b c -0.720589 0.887163
d 0.859588 -0.636524
This also applies when passing multiple indices to set_index:
# Old output, 2-level MultiIndex of tuples
In [14]: df_multi.set_index([df_multi.index, df_multi.index])
Out[14]:
0
1
(a, c) (a, c) 0.471435 -1.190976
(a, d) (a, d) 1.432707 -0.312652
(b, c) (b, c) -0.720589 0.887163
(b, d) (b, d) 0.859588 -0.636524
# New output, 4-level MultiIndex
In [15]: df_multi.set_index([df_multi.index, df_multi.index])
Out[15]:
0
1
a c a c 0.471435 -1.190976
d a d 1.432707 -0.312652
b c b c -0.720589 0.887163
d b d 0.859588 -0.636524
• pairwise keyword was added to the statistical moment functions rolling_cov, rolling_corr,
ewmcov, ewmcorr, expanding_cov, expanding_corr to allow the calculation of moving window
covariance and correlation matrices (GH4950). See Computing rolling pairwise covariances and correlations
in the docs.
In [16]: df = DataFrame(np.random.randn(10,4),columns=list('ABCD'))
In [17]: covs = rolling_cov(df[['A','B','C']], df[['B','C','D']], 5, pairwise=True)
In [18]: covs[df.index[-1]]
Out[18]:
B
C
D
A 0.128104 0.183628 -0.047358
B 0.856265 0.058945 0.145447
C 0.058945 0.335350 0.390637
• Series.iteritems() is now lazy (returns an iterator rather than a list). This was the documented behavior
prior to 0.14. (GH6760)
• Added nunique and value_counts functions to Index for counting unique elements. (GH6734)
• stack and unstack now raise a ValueError when the level keyword refers to a non-unique item in the
Index (previously raised a KeyError). (GH6738)
• drop unused order argument from Series.sort; args now are in the same order as Series.order; add
na_position arg to conform to Series.order (GH6847)
• default sorting algorithm for Series.order is now quicksort, to conform with Series.sort (and
numpy defaults)
• add inplace keyword to Series.order/sort to make them inverses (GH6859)
76
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• DataFrame.sort now places NaNs at the beginning or end of the sort according to the na_position
parameter. (GH3917)
• accept TextFileReader in concat, which was affecting a common user idiom (GH6583), this was a
regression from 0.13.1
• Added factorize functions to Index and Series to get indexer and unique values (GH7090)
• describe on a DataFrame with a mix of Timestamp and string like objects returns a different Index (GH7088).
Previously the index was unintentionally sorted.
• Arithmetic operations with only bool dtypes now give a warning indicating that they are evaluated in Python
space for +, -, and * operations and raise for all others (GH7011, GH6762, GH7015, GH7210)
x
y
x
x
=
=
+
/
pd.Series(np.random.rand(10) > 0.5)
True
y # warning generated: should do x | y instead
y # this raises because it doesn't make sense
NotImplementedError: operator '/' not implemented for bool dtypes
• In HDFStore, select_as_multiple will always raise a KeyError, when a key or the selector is not
found (GH6177)
• df[’col’] = value and df.loc[:,’col’] = value are now completely equivalent; previously the
.loc would not necessarily coerce the dtype of the resultant series (GH6149)
• dtypes and ftypes now return a series with dtype=object on empty containers (GH5740)
• df.to_csv will now return a string of the CSV data if neither a target path nor a buffer is provided (GH6061)
• pd.infer_freq() will now raise a TypeError if given an invalid Series/Index type (GH6407,
GH6463)
• A tuple passed to DataFame.sort_index will be interpreted as the levels of the index, rather than requiring
a list of tuple (GH4370)
• all offset operations now return Timestamp types (rather than datetime), Business/Week frequencies were
incorrect (GH4069)
• to_excel now converts np.inf into a string representation, customizable by the inf_rep keyword argument (Excel has no native inf representation) (GH6782)
• Replace pandas.compat.scipy.scoreatpercentile with numpy.percentile (GH6810)
• .quantile on a datetime[ns] series now returns Timestamp instead of np.datetime64 objects
(GH6810)
• change AssertionError to TypeError for invalid types passed to concat (GH6583)
• Raise a TypeError when DataFrame is passed an iterator as the data argument (GH5357)
1.7.2 Display Changes
• The default way of printing large DataFrames has changed. DataFrames exceeding max_rows and/or
max_columns are now displayed in a centrally truncated view, consistent with the printing of a
pandas.Series (GH5603).
In previous versions, a DataFrame was truncated once the dimension constraints were reached and an ellipse
(...) signaled that part of the data was cut off.
1.7. v0.14.0 (May 31 , 2014)
77
pandas: powerful Python data analysis toolkit, Release 0.16.1
In the current version, large DataFrames are centrally truncated, showing a preview of head and tail in both
dimensions.
• allow option ’truncate’ for display.show_dimensions to only show the dimensions if the frame is
truncated (GH6547).
The default for display.show_dimensions will now be truncate. This is consistent with how Series
display length.
In [19]: dfd = pd.DataFrame(np.arange(25).reshape(-1,5), index=[0,1,2,3,4], columns=[0,1,2,3,4])
# show dimensions since this is truncated
In [20]: with pd.option_context('display.max_rows', 2, 'display.max_columns', 2,
....:
'display.show_dimensions', 'truncate'):
....:
print(dfd)
....:
0 ...
4
0
0 ...
4
78
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
..
4
.. ...
20 ...
..
24
[5 rows x 5 columns]
# will not show dimensions since it is not truncated
In [21]: with pd.option_context('display.max_rows', 10, 'display.max_columns', 40,
....:
'display.show_dimensions', 'truncate'):
....:
print(dfd)
....:
0
1
2
3
4
0
0
1
2
3
4
1
5
6
7
8
9
2 10 11 12 13 14
3 15 16 17 18 19
4 20 21 22 23 24
• Regression in the display of a MultiIndexed Series with display.max_rows is less than the length of the
series (GH7101)
• Fixed a bug in the HTML repr of a truncated Series or DataFrame not showing the class name with the large_repr
set to ‘info’ (GH7105)
• The verbose keyword in DataFrame.info(), which controls whether to shorten the info representation,
is now None by default. This will follow the global setting in display.max_info_columns. The global
setting can be overriden with verbose=True or verbose=False.
• Fixed a bug with the info repr not honoring the display.max_info_columns setting (GH6939)
• Offset/freq info now in Timestamp __repr__ (GH4553)
1.7.3 Text Parsing API Changes
read_csv()/read_table() will now be noiser w.r.t invalid options rather than falling back to the
PythonParser.
• Raise
ValueError
when
sep
read_csv()/read_table() (GH6607)
specified
• Raise
ValueError
when
engine=’c’
read_csv()/read_table() (GH6607)
with
specified
delim_whitespace=True
with
unsupported
options
in
in
• Raise ValueError when fallback to python parser causes options to be ignored (GH6607)
• Produce ParserWarning on fallback to python parser when no options are ignored (GH6607)
• Translate sep=’\s+’ to delim_whitespace=True in read_csv()/read_table() if no other Cunsupported options specified (GH6607)
1.7.4 Groupby API Changes
More consistent behaviour for some groupby methods:
• groupby head and tail now act more like filter rather than an aggregation:
In [22]: df = pd.DataFrame([[1, 2], [1, 4], [5, 6]], columns=['A', 'B'])
In [23]: g = df.groupby('A')
1.7. v0.14.0 (May 31 , 2014)
79
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [24]: g.head(1)
Out[24]:
A B
0 1 2
2 5 6
# filters DataFrame
In [25]: g.apply(lambda x: x.head(1))
Out[25]:
A B
A
1 0 1 2
5 2 5 6
# used to simply fall-through
• groupby head and tail respect column selection:
In [26]: g[['B']].head(1)
Out[26]:
B
0 2
2 6
• groupby nth now reduces by default; filtering can be achieved by passing as_index=False. With an
optional dropna argument to ignore NaN. See the docs.
Reducing
In [27]: df = DataFrame([[1, np.nan], [1, 4], [5, 6]], columns=['A', 'B'])
In [28]: g = df.groupby('A')
In [29]: g.nth(0)
Out[29]:
B
A
1 NaN
5
6
# this is equivalent to g.first()
In [30]: g.nth(0, dropna='any')
Out[30]:
B
A
1 4
5 6
# this is equivalent to g.last()
In [31]: g.nth(-1, dropna='any')
Out[31]:
B
A
1 4
5 6
Filtering
In [32]: gf = df.groupby('A',as_index=False)
In [33]: gf.nth(0)
Out[33]:
A
B
80
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
2
1 NaN
5
6
In [34]: gf.nth(0, dropna='any')
Out[34]:
B
A
1 4
5 6
• groupby will now not return the grouped column for non-cython functions (GH5610, GH5614, GH6732), as its
already the index
In [35]: df = DataFrame([[1, np.nan], [1, 4], [5, 6], [5, 8]], columns=['A', 'B'])
In [36]: g = df.groupby('A')
In [37]: g.count()
Out[37]:
B
A
1 1
5 2
In [38]: g.describe()
Out[38]:
B
A
1 count 1.000000
mean
4.000000
std
NaN
min
4.000000
25%
4.000000
50%
4.000000
75%
4.000000
...
...
5 mean
7.000000
std
1.414214
min
6.000000
25%
6.500000
50%
7.000000
75%
7.500000
max
8.000000
[16 rows x 1 columns]
• passing as_index will leave the grouped column in-place (this is not change in 0.14.0)
In [39]: df = DataFrame([[1, np.nan], [1, 4], [5, 6], [5, 8]], columns=['A', 'B'])
In [40]: g = df.groupby('A',as_index=False)
In [41]: g.count()
Out[41]:
A B
0 1 1
1 5 2
In [42]: g.describe()
1.7. v0.14.0 (May 31 , 2014)
81
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[42]:
A
0 count 2
mean
1
std
0
min
1
25%
1
50%
1
75%
1
...
..
1 mean
5
std
0
min
5
25%
5
50%
5
75%
5
max
5
B
1.000000
4.000000
NaN
4.000000
4.000000
4.000000
4.000000
...
7.000000
1.414214
6.000000
6.500000
7.000000
7.500000
8.000000
[16 rows x 2 columns]
• Allow specification of a more complex groupby via pd.Grouper, such as grouping by a Time and a string
field simultaneously. See the docs. (GH3794)
• Better propagation/preservation of Series names when performing groupby operations:
– SeriesGroupBy.agg will ensure that the name attribute of the original series is propagated to the
result (GH6265).
– If the function provided to GroupBy.apply returns a named series, the name of the series will be kept as
the name of the column index of the DataFrame returned by GroupBy.apply (GH6124). This facilitates
DataFrame.stack operations where the name of the column index is used as the name of the inserted
column containing the pivoted data.
1.7.5 SQL
The SQL reading and writing functions now support more database flavors through SQLAlchemy (GH2717, GH4163,
GH5950, GH6292). All databases supported by SQLAlchemy can be used, such as PostgreSQL, MySQL, Oracle,
Microsoft SQL server (see documentation of SQLAlchemy on included dialects).
The functionality of providing DBAPI connection objects will only be supported for sqlite3 in the future. The
’mysql’ flavor is deprecated.
The new functions read_sql_query() and read_sql_table() are introduced. The function read_sql()
is kept as a convenience wrapper around the other two and will delegate to specific function depending on the provided
input (database table name or sql query).
In practice, you have to provide a SQLAlchemy engine to the sql functions. To connect with SQLAlchemy you use
the create_engine() function to create an engine object from database URI. You only need to create the engine
once per database you are connecting to. For an in-memory sqlite database:
In [43]: from sqlalchemy import create_engine
# Create your connection.
In [44]: engine = create_engine('sqlite:///:memory:')
This engine can then be used to write or read data to/from this database:
82
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [45]: df = pd.DataFrame({'A': [1,2,3], 'B': ['a', 'b', 'c']})
In [46]: df.to_sql('db_table', engine, index=False)
You can read data from a database by specifying the table name:
In [47]: pd.read_sql_table('db_table', engine)
Out[47]:
A B
0 1 a
1 2 b
2 3 c
or by specifying a sql query:
In [48]: pd.read_sql_query('SELECT * FROM db_table', engine)
Out[48]:
A B
0 1 a
1 2 b
2 3 c
Some other enhancements to the sql functions include:
• support for writing the index. This can be controlled with the index keyword (default is True).
• specify the column label to use when writing the index with index_label.
• specify string columns to parse as datetimes withh the parse_dates keyword in read_sql_query() and
read_sql_table().
Warning: Some of the existing functions or function aliases have been deprecated and will be removed in future
versions. This includes: tquery, uquery, read_frame, frame_query, write_frame.
Warning: The support for the ‘mysql’ flavor when using DBAPI connection objects has been deprecated. MySQL
will be further supported with SQLAlchemy engines (GH6900).
1.7.6 MultiIndexing Using Slicers
In 0.14.0 we added a new way to slice multi-indexed objects. You can slice a multi-index by providing multiple
indexers.
You can provide any of the selectors as if you are indexing by label, see Selection by Label, including slices, lists of
labels, labels, and boolean indexers.
You can use slice(None) to select all the contents of that level. You do not need to specify all the deeper levels,
they will be implied as slice(None).
As usual, both sides of the slicers are included as this is label indexing.
See the docs See also issues (GH6134, GH4036, GH3057, GH2598, GH5641, GH7106)
1.7. v0.14.0 (May 31 , 2014)
83
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning: You should specify all axes in the .loc specifier, meaning the indexer for the index and for the
columns. Their are some ambiguous cases where the passed indexer could be mis-interpreted as indexing both
axes, rather than into say the MuliIndex for the rows.
You should do this:
df.loc[(slice('A1','A3'),.....),:]
rather than this:
df.loc[(slice('A1','A3'),.....)]
Warning: You will need to make sure that the selection axes are fully lexsorted!
In [49]: def mklbl(prefix,n):
....:
return ["%s%s" % (prefix,i)
....:
for i in range(n)]
In [50]: index = MultiIndex.from_product([mklbl('A',4),
....:
mklbl('B',2),
....:
mklbl('C',4),
....:
mklbl('D',2)])
....:
In [51]: columns = MultiIndex.from_tuples([('a','foo'),('a','bar'),
....:
('b','foo'),('b','bah')],
....:
names=['lvl0', 'lvl1'])
....:
In [52]: df = DataFrame(np.arange(len(index)*len(columns)).reshape((len(index),len(columns))),
....:
index=index,
....:
columns=columns).sortlevel().sortlevel(axis=1)
....:
In [53]: df
Out[53]:
lvl0
lvl1
A0 B0 C0 D0
D1
C1 D0
D1
C2 D0
D1
C3 D0
...
A3 B1 C0 D1
C1 D0
D1
C2 D0
D1
C3 D0
D1
a
bar
1
5
9
13
17
21
25
...
229
233
237
241
245
249
253
foo
0
4
8
12
16
20
24
...
228
232
236
240
244
248
252
b
bah
3
7
11
15
19
23
27
...
231
235
239
243
247
251
255
foo
2
6
10
14
18
22
26
...
230
234
238
242
246
250
254
[64 rows x 4 columns]
Basic multi-index slicing using slices, lists, and labels.
84
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [54]: df.loc[(slice('A1','A3'),slice(None), ['C1','C3']),:]
Out[54]:
lvl0
a
b
lvl1
bar foo bah foo
A1 B0 C1 D0
73
72
75
74
D1
77
76
79
78
C3 D0
89
88
91
90
D1
93
92
95
94
B1 C1 D0 105 104 107 106
D1 109 108 111 110
C3 D0 121 120 123 122
...
... ... ... ...
A3 B0 C1 D1 205 204 207 206
C3 D0 217 216 219 218
D1 221 220 223 222
B1 C1 D0 233 232 235 234
D1 237 236 239 238
C3 D0 249 248 251 250
D1 253 252 255 254
[24 rows x 4 columns]
You can use a pd.IndexSlice to shortcut the creation of these slices
In [55]: idx = pd.IndexSlice
In [56]: df.loc[idx[:,:,['C1','C3']],idx[:,'foo']]
Out[56]:
lvl0
a
b
lvl1
foo foo
A0 B0 C1 D0
8
10
D1
12
14
C3 D0
24
26
D1
28
30
B1 C1 D0
40
42
D1
44
46
C3 D0
56
58
...
... ...
A3 B0 C1 D1 204 206
C3 D0 216 218
D1 220 222
B1 C1 D0 232 234
D1 236 238
C3 D0 248 250
D1 252 254
[32 rows x 2 columns]
It is possible to perform quite complicated selections using this method on multiple axes at the same time.
In [57]: df.loc['A1',(slice(None),'foo')]
Out[57]:
lvl0
a
b
lvl1
foo foo
B0 C0 D0
64
66
D1
68
70
C1 D0
72
74
D1
76
78
C2 D0
80
82
1.7. v0.14.0 (May 31 , 2014)
85
pandas: powerful Python data analysis toolkit, Release 0.16.1
D1
C3 D0
...
B1 C0 D1
C1 D0
D1
C2 D0
D1
C3 D0
D1
84
88
...
100
104
108
112
116
120
124
86
90
...
102
106
110
114
118
122
126
[16 rows x 2 columns]
In [58]: df.loc[idx[:,:,['C1','C3']],idx[:,'foo']]
Out[58]:
lvl0
a
b
lvl1
foo foo
A0 B0 C1 D0
8
10
D1
12
14
C3 D0
24
26
D1
28
30
B1 C1 D0
40
42
D1
44
46
C3 D0
56
58
...
... ...
A3 B0 C1 D1 204 206
C3 D0 216 218
D1 220 222
B1 C1 D0 232 234
D1 236 238
C3 D0 248 250
D1 252 254
[32 rows x 2 columns]
Using a boolean indexer you can provide selection related to the values.
In [59]: mask = df[('a','foo')]>200
In [60]: df.loc[idx[mask,:,['C1','C3']],idx[:,'foo']]
Out[60]:
lvl0
a
b
lvl1
foo foo
A3 B0 C1 D1 204 206
C3 D0 216 218
D1 220 222
B1 C1 D0 232 234
D1 236 238
C3 D0 248 250
D1 252 254
You can also specify the axis argument to .loc to interpret the passed slicers on a single axis.
In [61]: df.loc(axis=0)[:,:,['C1','C3']]
Out[61]:
lvl0
a
b
lvl1
bar foo bah foo
A0 B0 C1 D0
9
8
11
10
D1
13
12
15
14
86
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
C3 D0
D1
B1 C1 D0
D1
C3 D0
...
A3 B0 C1 D1
C3 D0
D1
B1 C1 D0
D1
C3 D0
D1
25
29
41
45
57
...
205
217
221
233
237
249
253
24
28
40
44
56
...
204
216
220
232
236
248
252
27
31
43
47
59
...
207
219
223
235
239
251
255
26
30
42
46
58
...
206
218
222
234
238
250
254
[32 rows x 4 columns]
Furthermore you can set the values using these methods
In [62]: df2 = df.copy()
In [63]: df2.loc(axis=0)[:,:,['C1','C3']] = -10
In [64]: df2
Out[64]:
lvl0
a
lvl1
bar
A0 B0 C0 D0
1
D1
5
C1 D0 -10
D1 -10
C2 D0
17
D1
21
C3 D0 -10
...
...
A3 B1 C0 D1 229
C1 D0 -10
D1 -10
C2 D0 241
D1 245
C3 D0 -10
D1 -10
foo
0
4
-10
-10
16
20
-10
...
228
-10
-10
240
244
-10
-10
b
bah
3
7
-10
-10
19
23
-10
...
231
-10
-10
243
247
-10
-10
foo
2
6
-10
-10
18
22
-10
...
230
-10
-10
242
246
-10
-10
[64 rows x 4 columns]
You can use a right-hand-side of an alignable object as well.
In [65]: df2 = df.copy()
In [66]: df2.loc[idx[:,:,['C1','C3']],:] = df2*1000
In [67]: df2
Out[67]:
lvl0
lvl1
A0 B0 C0 D0
D1
C1 D0
D1
C2 D0
a
bar
1
5
9000
13000
17
foo
0
4
8000
12000
16
1.7. v0.14.0 (May 31 , 2014)
b
bah
3
7
11000
15000
19
foo
2
6
10000
14000
18
87
pandas: powerful Python data analysis toolkit, Release 0.16.1
D1
C3 D0
...
A3 B1 C0 D1
C1 D0
D1
C2 D0
D1
C3 D0
D1
21
25000
...
229
233000
237000
241
245
249000
253000
20
24000
...
228
232000
236000
240
244
248000
252000
23
27000
...
231
235000
239000
243
247
251000
255000
22
26000
...
230
234000
238000
242
246
250000
254000
[64 rows x 4 columns]
1.7.7 Plotting
• Hexagonal bin plots from DataFrame.plot with kind=’hexbin’ (GH5478), See the docs.
• DataFrame.plot and Series.plot now supports area plot with specifying kind=’area’ (GH6656),
See the docs
• Pie plots from Series.plot and DataFrame.plot with kind=’pie’ (GH6976), See the docs.
• Plotting with Error Bars is now supported in the .plot method of DataFrame and Series objects (GH3796,
GH6834), See the docs.
• DataFrame.plot and Series.plot now support a table keyword for plotting matplotlib.Table,
See the docs. The table keyword can receive the following values.
– False: Do nothing (default).
– True: Draw a table using the DataFrame or Series called plot method. Data will be transposed to
meet matplotlib’s default layout.
– DataFrame or Series: Draw matplotlib.table using the passed data.
The data will be
drawn as displayed in print method (not transposed automatically).
Also, helper function
pandas.tools.plotting.table is added to create a table from DataFrame and Series, and
add it to an matplotlib.Axes.
• plot(legend=’reverse’) will now reverse the order of legend labels for most plot kinds. (GH6014)
• Line plot and area plot can be stacked by stacked=True (GH6656)
• Following keywords are now acceptable for DataFrame.plot() with kind=’bar’ and kind=’barh’:
– width: Specify the bar width. In previous versions, static value 0.5 was passed to matplotlib and it cannot
be overwritten. (GH6604)
– align: Specify the bar alignment. Default is center (different from matplotlib). In previous versions,
pandas passes align=’edge’ to matplotlib and adjust the location to center by itself, and it results align
keyword is not applied as expected. (GH4525)
– position: Specify relative alignments for bar plot layout. From 0 (left/bottom-end) to 1(right/top-end).
Default is 0.5 (center). (GH6604)
Because of the default align value changes, coordinates of bar plots are now located on integer values (0.0, 1.0,
2.0 ...). This is intended to make bar plot be located on the same coodinates as line plot. However, bar plot
may differs unexpectedly when you manually adjust the bar location or drawing area, such as using set_xlim,
set_ylim, etc. In this cases, please modify your script to meet with new coordinates.
88
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• The parallel_coordinates() function now takes argument color instead of colors.
A
FutureWarning is raised to alert that the old colors argument will not be supported in a future release.
(GH6956)
• The parallel_coordinates() and andrews_curves() functions now take positional argument
frame instead of data. A FutureWarning is raised if the old data argument is used by name. (GH6956)
• DataFrame.boxplot() now supports layout keyword (GH6769)
• DataFrame.boxplot() has a new keyword argument, return_type. It accepts ’dict’, ’axes’, or
’both’, in which case a namedtuple with the matplotlib axes and a dict of matplotlib Lines is returned.
1.7.8 Prior Version Deprecations/Changes
There are prior version deprecations that are taking effect as of 0.14.0.
• Remove DateRange in favor of DatetimeIndex (GH6816)
• Remove column keyword from DataFrame.sort (GH4370)
• Remove precision keyword from set_eng_float_format() (GH395)
• Remove force_unicode keyword from DataFrame.to_string(), DataFrame.to_latex(), and
DataFrame.to_html(); these function encode in unicode by default (GH2224, GH2225)
• Remove nanRep keyword from DataFrame.to_csv() and DataFrame.to_string() (GH275)
• Remove unique keyword from HDFStore.select_column() (GH3256)
• Remove inferTimeRule keyword from Timestamp.offset() (GH391)
• Remove name keyword from get_data_yahoo() and get_data_google() ( commit b921d1a )
• Remove offset keyword from DatetimeIndex constructor ( commit 3136390 )
• Remove time_rule from several rolling-moment statistical functions, such as rolling_sum() (GH1042)
• Removed neg - boolean operations on numpy arrays in favor of inv ~, as this is going to be deprecated in numpy
1.9 (GH6960)
1.7.9 Deprecations
• The pivot_table()/DataFrame.pivot_table() and crosstab() functions now take arguments
index and columns instead of rows and cols. A FutureWarning is raised to alert that the old rows
and cols arguments will not be supported in a future release (GH5505)
• The DataFrame.drop_duplicates() and DataFrame.duplicated() methods now take argument
subset instead of cols to better align with DataFrame.dropna(). A FutureWarning is raised to
alert that the old cols arguments will not be supported in a future release (GH6680)
• The DataFrame.to_csv() and DataFrame.to_excel() functions now takes argument columns instead of cols. A FutureWarning is raised to alert that the old cols arguments will not be supported in a
future release (GH6645)
• Indexers will warn FutureWarning when used with a scalar indexer and a non-floating point Index (GH4892,
GH6960)
# non-floating point indexes can only be indexed by integers / labels
In [1]: Series(1,np.arange(5))[3.0]
pandas/core/index.py:469: FutureWarning: scalar indexers for index type Int64Index shoul
Out[1]: 1
1.7. v0.14.0 (May 31 , 2014)
89
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [2]: Series(1,np.arange(5)).iloc[3.0]
pandas/core/index.py:469: FutureWarning: scalar indexers for index type Int64Index shoul
Out[2]: 1
In [3]: Series(1,np.arange(5)).iloc[3.0:4]
pandas/core/index.py:527: FutureWarning: slice indexers when using iloc should be intege
Out[3]:
3
1
dtype: int64
# these are Float64Indexes, so integer or floating point is acceptable
In [4]: Series(1,np.arange(5.))[3]
Out[4]: 1
In [5]: Series(1,np.arange(5.))[3.0]
Out[6]: 1
• Numpy 1.9 compat w.r.t. deprecation warnings (GH6960)
• Panel.shift() now has a function signature that matches DataFrame.shift(). The old positional argument lags has been changed to a keyword argument periods with a default value of 1. A
FutureWarning is raised if the old argument lags is used by name. (GH6910)
• The order keyword argument of factorize() will be removed. (GH6926).
• Remove the copy keyword from DataFrame.xs(), Panel.major_xs(), Panel.minor_xs(). A
view will be returned if possible, otherwise a copy will be made. Previously the user could think that
copy=False would ALWAYS return a view. (GH6894)
• The parallel_coordinates() function now takes argument color instead of colors.
A
FutureWarning is raised to alert that the old colors argument will not be supported in a future release.
(GH6956)
• The parallel_coordinates() and andrews_curves() functions now take positional argument
frame instead of data. A FutureWarning is raised if the old data argument is used by name. (GH6956)
• The support for the ‘mysql’ flavor when using DBAPI connection objects has been deprecated. MySQL will be
further supported with SQLAlchemy engines (GH6900).
• The following io.sql functions have been deprecated: tquery, uquery, read_frame, frame_query,
write_frame.
• The percentile_width keyword argument in describe() has been deprecated. Use the percentiles keyword
instead, which takes a list of percentiles to display. The default output is unchanged.
• The default return type of boxplot() will change from a dict to a matpltolib Axes in a future release. You
can use the future behavior now by passing return_type=’axes’ to boxplot.
1.7.10 Known Issues
• OpenPyXL 2.0.0 breaks backwards compatibility (GH7169)
1.7.11 Enhancements
• DataFrame and Series will create a MultiIndex object if passed a tuples dict, See the docs (GH3323)
90
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [68]: Series({('a', 'b'): 1, ('a', 'a'): 0,
....:
('a', 'c'): 2, ('b', 'a'): 3, ('b', 'b'): 4})
....:
Out[68]:
a a
0
b
1
c
2
b a
3
b
4
dtype: int64
In [69]: DataFrame({('a',
....:
('a',
....:
('a',
....:
('b',
....:
('b',
....:
Out[69]:
a
b
a
b
c
a
b
A B
4
1
5
8 10
C
3
2
6
7 NaN
D NaN NaN NaN NaN
9
'b'):
'a'):
'c'):
'a'):
'b'):
{('A',
{('A',
{('A',
{('A',
{('A',
'B'):
'C'):
'B'):
'C'):
'D'):
1,
3,
5,
7,
9,
('A',
('A',
('A',
('A',
('A',
'C'):
'B'):
'C'):
'B'):
'B'):
2},
4},
6},
8},
10}})
• Added the sym_diff method to Index (GH5543)
• DataFrame.to_latex now takes a longtable keyword, which if True will return a table in a longtable
environment. (GH6617)
• Add option to turn off escaping in DataFrame.to_latex (GH6472)
• pd.read_clipboard will, if the keyword sep is unspecified, try to detect data copied from a spreadsheet
and parse accordingly. (GH6223)
• Joining a singly-indexed DataFrame with a multi-indexed DataFrame (GH3662)
See the docs. Joining multi-index DataFrames on both the left and right is not yet supported ATM.
In [70]: household = DataFrame(dict(household_id = [1,2,3],
....:
male = [0,1,0],
....:
wealth = [196087.3,316478.7,294750]),
....:
columns = ['household_id','male','wealth']
....:
).set_index('household_id')
....:
In [71]: household
Out[71]:
male
household_id
1
0
2
1
3
0
wealth
196087.3
316478.7
294750.0
In [72]: portfolio = DataFrame(dict(household_id = [1,2,2,3,3,3,4],
....:
asset_id = ["nl0000301109","nl0000289783","gb00b03mlx29",
....:
"gb00b03mlx29","lu0197800237","nl0000289965",np.
....:
name = ["ABN Amro","Robeco","Royal Dutch Shell","Royal Dutch
....:
"AAB Eastern Europe Equity Fund","Postbank BioTech F
....:
share = [1.0,0.4,0.6,0.15,0.6,0.25,1.0]),
....:
columns = ['household_id','asset_id','name','share']
1.7. v0.14.0 (May 31 , 2014)
91
pandas: powerful Python data analysis toolkit, Release 0.16.1
....:
....:
).set_index(['household_id','asset_id'])
In [73]: portfolio
Out[73]:
household_id asset_id
1
nl0000301109
2
nl0000289783
gb00b03mlx29
3
gb00b03mlx29
lu0197800237
nl0000289965
4
NaN
name
share
ABN Amro
Robeco
Royal Dutch Shell
Royal Dutch Shell
AAB Eastern Europe Equity Fund
Postbank BioTech Fonds
NaN
1.00
0.40
0.60
0.15
0.60
0.25
1.00
In [74]: household.join(portfolio, how='inner')
Out[74]:
male
wealth
name
household_id asset_id
1
nl0000301109
0 196087.3
ABN Amro
2
nl0000289783
1 316478.7
Robeco
gb00b03mlx29
1 316478.7
Royal Dutch Shell
3
gb00b03mlx29
0 294750.0
Royal Dutch Shell
lu0197800237
0 294750.0 AAB Eastern Europe Equity Fund
nl0000289965
0 294750.0
Postbank BioTech Fonds
\
share
household_id asset_id
1
nl0000301109
2
nl0000289783
gb00b03mlx29
3
gb00b03mlx29
lu0197800237
nl0000289965
1.00
0.40
0.60
0.15
0.60
0.25
• quotechar, doublequote, and escapechar can now be specified when using DataFrame.to_csv
(GH5414, GH4528)
• Partially sort by only the specified levels of a MultiIndex with the sort_remaining boolean kwarg.
(GH3984)
• Added to_julian_date to TimeStamp and DatetimeIndex. The Julian Date is used primarily in
astronomy and represents the number of days from noon, January 1, 4713 BC. Because nanoseconds are used
to define the time in pandas the actual range of dates that you can use is 1678 AD to 2262 AD. (GH4041)
• DataFrame.to_stata will now check data for compatibility with Stata data types and will upcast when
needed. When it is not possible to losslessly upcast, a warning is issued (GH6327)
• DataFrame.to_stata and StataWriter will accept keyword arguments time_stamp and data_label
which allow the time stamp and dataset label to be set when creating a file. (GH6545)
• pandas.io.gbq now handles reading unicode strings properly. (GH5940)
• Holidays Calendars are now available and can be used with the CustomBusinessDay offset (GH6719)
• Float64Index is now backed by a float64 dtype ndarray instead of an object dtype array (GH6471).
• Implemented Panel.pct_change (GH6904)
• Added how option to rolling-moment functions to dictate how to handle resampling; rolling_max() defaults to max, rolling_min() defaults to min, and all others default to mean (GH6297)
92
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• CustomBuisnessMonthBegin and CustomBusinessMonthEnd are now available (GH6866)
• Series.quantile() and DataFrame.quantile() now accept an array of quantiles.
• describe() now accepts an array of percentiles to include in the summary statistics (GH4196)
• pivot_table can now accept Grouper by index and columns keywords (GH6913)
In [75]: import datetime
In [76]: df = DataFrame({
....:
'Branch' : 'A A A A A B'.split(),
....:
'Buyer': 'Carl Mark Carl Carl Joe Joe'.split(),
....:
'Quantity': [1, 3, 5, 1, 8, 1],
....:
'Date' : [datetime.datetime(2013,11,1,13,0), datetime.datetime(2013,9,1,13,5),
....:
datetime.datetime(2013,10,1,20,0), datetime.datetime(2013,10,2,10,0),
....:
datetime.datetime(2013,11,1,20,0), datetime.datetime(2013,10,2,10,0)],
....:
'PayDay' : [datetime.datetime(2013,10,4,0,0), datetime.datetime(2013,10,15,13,5),
....:
datetime.datetime(2013,9,5,20,0), datetime.datetime(2013,11,2,10,0),
....:
datetime.datetime(2013,10,7,20,0), datetime.datetime(2013,9,5,10,0)]})
....:
In [77]: df
Out[77]:
Branch Buyer
0
A Carl 2013-11-01
1
A Mark 2013-09-01
2
A Carl 2013-10-01
3
A Carl 2013-10-02
4
A
Joe 2013-11-01
5
B
Joe 2013-10-02
Date
13:00:00
13:05:00
20:00:00
10:00:00
20:00:00
10:00:00
2013-10-04
2013-10-15
2013-09-05
2013-11-02
2013-10-07
2013-09-05
PayDay
00:00:00
13:05:00
20:00:00
10:00:00
20:00:00
10:00:00
Quantity
1
3
5
1
8
1
In [78]: pivot_table(df, index=Grouper(freq='M', key='Date'),
....:
columns=Grouper(freq='M', key='PayDay'),
....:
values='Quantity', aggfunc=np.sum)
....:
Out[78]:
PayDay
2013-09-30 2013-10-31 2013-11-30
Date
2013-09-30
NaN
3
NaN
2013-10-31
6
NaN
1
2013-11-30
NaN
9
NaN
• Arrays of strings can be wrapped to a specified width (str.wrap) (GH6999)
• Add nsmallest() and Series.nlargest() methods to Series, See the docs (GH3960)
• PeriodIndex fully supports partial string indexing like DatetimeIndex (GH7043)
In [79]: prng = period_range('2013-01-01 09:00', periods=100, freq='H')
In [80]: ps = Series(np.random.randn(len(prng)), index=prng)
In [81]: ps
Out[81]:
2013-01-01 09:00
2013-01-01 10:00
2013-01-01 11:00
2013-01-01 12:00
2013-01-01 13:00
2013-01-01 14:00
0.755414
0.215269
0.841009
-1.445810
-1.401973
-0.100918
1.7. v0.14.0 (May 31 , 2014)
93
pandas: powerful Python data analysis toolkit, Release 0.16.1
2013-01-01 15:00
-0.548242
...
2013-01-05 06:00
-0.379811
2013-01-05 07:00
0.702562
2013-01-05 08:00
-0.850346
2013-01-05 09:00
1.176812
2013-01-05 10:00
-0.524336
2013-01-05 11:00
0.700908
2013-01-05 12:00
0.984188
Freq: H, dtype: float64
In [82]: ps['2013-01-02']
Out[82]:
2013-01-02 00:00
-0.208499
2013-01-02 01:00
1.033801
2013-01-02 02:00
-2.400454
2013-01-02 03:00
2.030604
2013-01-02 04:00
-1.142631
2013-01-02 05:00
0.211883
2013-01-02 06:00
0.704721
...
2013-01-02 17:00
0.464392
2013-01-02 18:00
-3.563517
2013-01-02 19:00
1.321106
2013-01-02 20:00
0.152631
2013-01-02 21:00
0.164530
2013-01-02 22:00
-0.430096
2013-01-02 23:00
0.767369
Freq: H, dtype: float64
• read_excel can now read milliseconds in Excel dates and times with xlrd >= 0.9.3. (GH5945)
• pd.stats.moments.rolling_var now uses Welford’s method for increased numerical stability
(GH6817)
• pd.expanding_apply and pd.rolling_apply now take args and kwargs that are passed on to the func (GH6289)
• DataFrame.rank() now has a percentage rank option (GH5971)
• Series.rank() now has a percentage rank option (GH5971)
• Series.rank() and DataFrame.rank() now accept method=’dense’ for ranks without gaps
(GH6514)
• Support passing encoding with xlwt (GH3710)
• Refactor Block classes removing Block.items attributes to avoid duplication in item handling (GH6745,
GH6988).
• Testing statements updated to use specialized asserts (GH6175)
1.7.12 Performance
• Performance improvement when
DatetimeConverter (GH6636)
converting
DatetimeIndex
to
floating
ordinals
using
• Performance improvement for DataFrame.shift (GH5609)
• Performance improvement in indexing into a multi-indexed Series (GH5567)
• Performance improvements in single-dtyped indexing (GH6484)
94
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Improve performance of DataFrame construction with certain offsets, by removing faulty caching (e.g. MonthEnd,BusinessMonthEnd), (GH6479)
• Improve performance of CustomBusinessDay (GH6584)
• improve performance of slice indexing on Series with string keys (GH6341, GH6372)
• Performance improvement for DataFrame.from_records when reading a specified number of rows from
an iterable (GH6700)
• Performance improvements in timedelta conversions for integer dtypes (GH6754)
• Improved performance of compatible pickles (GH6899)
• Improve performance in certain reindexing operations by optimizing take_2d (GH6749)
• GroupBy.count() is now implemented in Cython and is much faster for large numbers of groups (GH7016).
1.7.13 Experimental
There are no experimental changes in 0.14.0
1.7.14 Bug Fixes
• Bug in Series ValueError when index doesn’t match data (GH6532)
• Prevent segfault due to MultiIndex not being supported in HDFStore table format (GH1848)
• Bug in pd.DataFrame.sort_index where mergesort wasn’t stable when ascending=False
(GH6399)
• Bug in pd.tseries.frequencies.to_offset when argument has leading zeroes (GH6391)
• Bug in version string gen. for dev versions with shallow clones / install from tarball (GH6127)
• Inconsistent tz parsing Timestamp / to_datetime for current year (GH5958)
• Indexing bugs with reordered indexes (GH6252, GH6254)
• Bug in .xs with a Series multiindex (GH6258, GH5684)
• Bug in conversion of a string types to a DatetimeIndex with a specified frequency (GH6273, GH6274)
• Bug in eval where type-promotion failed for large expressions (GH6205)
• Bug in interpolate with inplace=True (GH6281)
• HDFStore.remove now handles start and stop (GH6177)
• HDFStore.select_as_multiple handles start and stop the same way as select (GH6177)
• HDFStore.select_as_coordinates and select_column works with a where clause that results in
filters (GH6177)
• Regression in join of non_unique_indexes (GH6329)
• Issue with groupby agg with a single function and a a mixed-type frame (GH6337)
• Bug in DataFrame.replace() when passing a non- bool to_replace argument (GH6332)
• Raise when trying to align on different levels of a multi-index assignment (GH3738)
• Bug in setting complex dtypes via boolean indexing (GH6345)
1.7. v0.14.0 (May 31 , 2014)
95
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in TimeGrouper/resample when presented with a non-monotonic DatetimeIndex that would return invalid
results. (GH4161)
• Bug in index name propogation in TimeGrouper/resample (GH4161)
• TimeGrouper has a more compatible API to the rest of the groupers (e.g. groups was missing) (GH3881)
• Bug in multiple grouping with a TimeGrouper depending on target column order (GH6764)
• Bug in pd.eval when parsing strings with possible tokens like ’&’ (GH6351)
• Bug correctly handle placements of -inf in Panels when dividing by integer 0 (GH6178)
• DataFrame.shift with axis=1 was raising (GH6371)
• Disabled clipboard tests until release time (run locally with nosetests -A disabled) (GH6048).
• Bug in DataFrame.replace() when passing a nested dict that contained keys not in the values to be
replaced (GH6342)
• str.match ignored the na flag (GH6609).
• Bug in take with duplicate columns that were not consolidated (GH6240)
• Bug in interpolate changing dtypes (GH6290)
• Bug in Series.get, was using a buggy access method (GH6383)
• Bug in hdfstore queries of the form where=[(’date’, ’>=’, datetime(2013,1,1)),
(’date’, ’<=’, datetime(2014,1,1))] (GH6313)
• Bug in DataFrame.dropna with duplicate indices (GH6355)
• Regression in chained getitem indexing with embedded list-like from 0.12 (GH6394)
• Float64Index with nans not comparing correctly (GH6401)
• eval/query expressions with strings containing the @ character will now work (GH6366).
• Bug in Series.reindex when specifying a method with some nan values was inconsistent (noted on a
resample) (GH6418)
• Bug in DataFrame.replace() where nested dicts were erroneously depending on the order of dictionary
keys and values (GH5338).
• Perf issue in concatting with empty objects (GH3259)
• Clarify sorting of sym_diff on Index objects with NaN values (GH6444)
• Regression in MultiIndex.from_product with a DatetimeIndex as input (GH6439)
• Bug in str.extract when passed a non-default index (GH6348)
• Bug in str.split when passed pat=None and n=1 (GH6466)
• Bug
in
io.data.DataReader
when
data_source="famafrench" (GH6460)
passed
"F-F_Momentum_Factor"
and
• Bug in sum of a timedelta64[ns] series (GH6462)
• Bug in resample with a timezone and certain offsets (GH6397)
• Bug in iat/iloc with duplicate indices on a Series (GH6493)
• Bug in read_html where nan’s were incorrectly being used to indicate missing values in text. Should use the
empty string for consistency with the rest of pandas (GH5129).
• Bug in read_html tests where redirected invalid URLs would make one test fail (GH6445).
96
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in multi-axis indexing using .loc on non-unique indices (GH6504)
• Bug that caused _ref_locs corruption when slice indexing across columns axis of a DataFrame (GH6525)
• Regression from 0.13 in the treatment of numpy datetime64 non-ns dtypes in Series creation (GH6529)
• .names attribute of MultiIndexes passed to set_index are now preserved (GH6459).
• Bug in setitem with a duplicate index and an alignable rhs (GH6541)
• Bug in setitem with .loc on mixed integer Indexes (GH6546)
• Bug in pd.read_stata which would use the wrong data types and missing values (GH6327)
• Bug in DataFrame.to_stata that lead to data loss in certain cases, and could be exported using the wrong
data types and missing values (GH6335)
• StataWriter replaces missing values in string columns by empty string (GH6802)
• Inconsistent types in Timestamp addition/subtraction (GH6543)
• Bug in preserving frequency across Timestamp addition/subtraction (GH4547)
• Bug in empty list lookup caused IndexError exceptions (GH6536, GH6551)
• Series.quantile raising on an object dtype (GH6555)
• Bug in .xs with a nan in level when dropped (GH6574)
• Bug in fillna with method=’bfill/ffill’ and datetime64[ns] dtype (GH6587)
• Bug in sql writing with mixed dtypes possibly leading to data loss (GH6509)
• Bug in Series.pop (GH6600)
• Bug in iloc indexing when positional indexer matched Int64Index of the corresponding axis and no reordering happened (GH6612)
• Bug in fillna with limit and value specified
• Bug in DataFrame.to_stata when columns have non-string names (GH4558)
• Bug in compat with np.compress, surfaced in (GH6658)
• Bug in binary operations with a rhs of a Series not aligning (GH6681)
• Bug in DataFrame.to_stata which incorrectly handles nan values and ignores with_index keyword
argument (GH6685)
• Bug in resample with extra bins when using an evenly divisible frequency (GH4076)
• Bug in consistency of groupby aggregation when passing a custom function (GH6715)
• Bug in resample when how=None resample freq is the same as the axis frequency (GH5955)
• Bug in downcasting inference with empty arrays (GH6733)
• Bug in obj.blocks on sparse containers dropping all but the last items of same for dtype (GH6748)
• Bug in unpickling NaT (NaTType) (GH4606)
• Bug in DataFrame.replace() where regex metacharacters were being treated as regexs even when
regex=False (GH6777).
• Bug in timedelta ops on 32-bit platforms (GH6808)
• Bug in setting a tz-aware index directly via .index (GH6785)
• Bug in expressions.py where numexpr would try to evaluate arithmetic ops (GH6762).
1.7. v0.14.0 (May 31 , 2014)
97
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Bug in Makefile where it didn’t remove Cython generated C files with make clean (GH6768)
• Bug with numpy < 1.7.2 when reading long strings from HDFStore (GH6166)
• Bug in DataFrame._reduce where non bool-like (0/1) integers were being coverted into bools. (GH6806)
• Regression from 0.13 with fillna and a Series on datetime-like (GH6344)
• Bug in adding np.timedelta64 to DatetimeIndex with timezone outputs incorrect results (GH6818)
• Bug in DataFrame.replace() where changing a dtype through replacement would only replace the first
occurrence of a value (GH6689)
• Better error message when passing a frequency of ‘MS’ in Period construction (GH5332)
• Bug in Series.__unicode__ when max_rows=None and the Series has more than 1000 rows. (GH6863)
• Bug in groupby.get_group where a datetlike wasn’t always accepted (GH5267)
• Bug in groupBy.get_group created by TimeGrouper raises AttributeError (GH6914)
• Bug in DatetimeIndex.tz_localize and DatetimeIndex.tz_convert converting NaT incorrectly (GH5546)
• Bug in arithmetic operations affecting NaT (GH6873)
• Bug in Series.str.extract where the resulting Series from a single group match wasn’t renamed to
the group name
• Bug in DataFrame.to_csv where setting index=False ignored the header kwarg (GH6186)
• Bug in DataFrame.plot and Series.plot, where the legend behave inconsistently when plotting to the
same axes repeatedly (GH6678)
• Internal tests for patching __finalize__ / bug in merge not finalizing (GH6923, GH6927)
• accept TextFileReader in concat, which was affecting a common user idiom (GH6583)
• Bug in C parser with leading whitespace (GH3374)
• Bug in C parser with delim_whitespace=True and \r-delimited lines
• Bug in python parser with explicit multi-index in row following column header (GH6893)
• Bug in Series.rank and DataFrame.rank that caused small floats (<1e-13) to all receive the same rank
(GH6886)
• Bug in DataFrame.apply with functions that used *args‘‘ or **kwargs and returned an empty result
(GH6952)
• Bug in sum/mean on 32-bit platforms on overflows (GH6915)
• Moved Panel.shift to NDFrame.slice_shift and fixed to respect multiple dtypes. (GH6959)
• Bug in enabling subplots=True in DataFrame.plot only has single column raises TypeError, and
Series.plot raises AttributeError (GH6951)
• Bug in DataFrame.plot draws unnecessary axes when enabling subplots and kind=scatter
(GH6951)
• Bug in read_csv from a filesystem with non-utf-8 encoding (GH6807)
• Bug in iloc when setting / aligning (GH6766)
• Bug causing UnicodeEncodeError when get_dummies called with unicode values and a prefix (GH6885)
• Bug in timeseries-with-frequency plot cursor display (GH5453)
• Bug surfaced in groupby.plot when using a Float64Index (GH7025)
98
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Stopped tests from failing if options data isn’t able to be downloaded from Yahoo (GH7034)
• Bug in parallel_coordinates and radviz where reordering of class column caused possible color/class
mismatch (GH6956)
• Bug in radviz and andrews_curves where multiple values of ‘color’ were being passed to plotting method
(GH6956)
• Bug in Float64Index.isin() where containing nan s would make indices claim that they contained all
the things (GH7066).
• Bug in DataFrame.boxplot where it failed to use the axis passed as the ax argument (GH3578)
• Bug in the XlsxWriter and XlwtWriter implementations that resulted in datetime columns being formatted without the time (GH7075) were being passed to plotting method
• read_fwf() treats None in colspec like regular python slices. It now reads from the beginning or until the
end of the line when colspec contains a None (previously raised a TypeError)
• Bug in cache coherence with chained indexing and slicing; add _is_view property to NDFrame to correctly
predict views; mark is_copy on xs only if its an actual copy (and not a view) (GH7084)
• Bug in DatetimeIndex creation from string ndarray with dayfirst=True (GH5917)
• Bug in MultiIndex.from_arrays created from DatetimeIndex doesn’t preserve freq and tz
(GH7090)
• Bug in unstack raises ValueError when MultiIndex contains PeriodIndex (GH4342)
• Bug in boxplot and hist draws unnecessary axes (GH6769)
• Regression in groupby.nth() for out-of-bounds indexers (GH6621)
• Bug in quantile with datetime values (GH6965)
• Bug in Dataframe.set_index, reindex and pivot don’t preserve DatetimeIndex and
PeriodIndex attributes (GH3950, GH5878, GH6631)
• Bug in MultiIndex.get_level_values doesn’t preserve DatetimeIndex and PeriodIndex attributes (GH7092)
• Bug in Groupby doesn’t preserve tz (GH3950)
• Bug in PeriodIndex partial string slicing (GH6716)
• Bug in the HTML repr of a truncated Series or DataFrame not showing the class name with the large_repr set
to ‘info’ (GH7105)
• Bug in DatetimeIndex specifying freq raises ValueError when passed value is too short (GH7098)
• Fixed a bug with the info repr not honoring the display.max_info_columns setting (GH6939)
• Bug PeriodIndex string slicing with out of bounds values (GH5407)
• Fixed a memory error in the hashtable implementation/factorizer on resizing of large tables (GH7157)
• Bug in isnull when applied to 0-dimensional object arrays (GH7176)
• Bug in query/eval where global constants were not looked up correctly (GH7178)
• Bug in recognizing out-of-bounds positional list indexers with iloc and a multi-axis tuple indexer (GH7189)
• Bug in setitem with a single value, multi-index and integer indices (GH7190, GH7218)
• Bug in expressions evaluation with reversed ops, showing in series-dataframe ops (GH7198, GH7192)
• Bug in multi-axis indexing with > 2 ndim and a multi-index (GH7199)
1.7. v0.14.0 (May 31 , 2014)
99
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Fix a bug where invalid eval/query operations would blow the stack (GH5198)
1.8 v0.13.1 (February 3, 2014)
This is a minor release from 0.13.0 and includes a small number of API changes, several new features, enhancements,
and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this
version.
Highlights include:
• Added infer_datetime_format keyword to read_csv/to_datetime to allow speedups for homogeneously formatted datetimes.
• Will intelligently limit display precision for datetime/timedelta formats.
• Enhanced Panel apply() method.
• Suggested tutorials in new Tutorials section.
• Our pandas ecosystem is growing, We now feature related projects in a new Pandas Ecosystem section.
• Much work has been taking place on improving the docs, and a new Contributing section has been added.
• Even though it may only be of interest to devs, we <3 our new CI status page: ScatterCI.
Warning: 0.13.1 fixes a bug that was caused by a combination of having numpy < 1.8, and doing chained
assignment on a string-like array. Please review the docs, chained indexing can have unexpected results and should
generally be avoided.
This would previously segfault:
In [1]: df = DataFrame(dict(A = np.array(['foo','bar','bah','foo','bar'])))
In [2]: df['A'].iloc[0] = np.nan
In [3]: df
Out[3]:
A
0 NaN
1 bar
2 bah
3 foo
4 bar
The recommended way to do this type of assignment is:
In [4]: df = DataFrame(dict(A = np.array(['foo','bar','bah','foo','bar'])))
In [5]: df.ix[0,'A'] = np.nan
In [6]: df
Out[6]:
A
0 NaN
1 bar
2 bah
3 foo
4 bar
100
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.8.1 Output Formatting Enhancements
• df.info() view now display dtype info per column (GH5682)
• df.info() now honors the option max_info_rows, to disable null counts for large frames (GH5974)
In [7]: max_info_rows = pd.get_option('max_info_rows')
In [8]: df = DataFrame(dict(A = np.random.randn(10),
...:
B = np.random.randn(10),
...:
C = date_range('20130101',periods=10)))
...:
In [9]: df.iloc[3:6,[0,2]] = np.nan
# set to not display the null counts
In [10]: pd.set_option('max_info_rows',0)
In [11]: df.info()
Int64Index: 10 entries, 0 to 9
Data columns (total 3 columns):
A
float64
B
float64
C
datetime64[ns]
dtypes: datetime64[ns](1), float64(2)
memory usage: 320.0 bytes
# this is the default (same as in 0.13.0)
In [12]: pd.set_option('max_info_rows',max_info_rows)
In [13]: df.info()
Int64Index: 10 entries, 0 to 9
Data columns (total 3 columns):
A
7 non-null float64
B
10 non-null float64
C
7 non-null datetime64[ns]
dtypes: datetime64[ns](1), float64(2)
memory usage: 320.0 bytes
• Add show_dimensions display option for the new DataFrame repr to control whether the dimensions print.
In [14]: df = DataFrame([[1, 2], [3, 4]])
In [15]: pd.set_option('show_dimensions', False)
In [16]: df
Out[16]:
0 1
0 1 2
1 3 4
In [17]: pd.set_option('show_dimensions', True)
In [18]: df
Out[18]:
0 1
0 1 2
1.8. v0.13.1 (February 3, 2014)
101
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
3
4
[2 rows x 2 columns]
• The ArrayFormatter for datetime and timedelta64 now intelligently limit precision based on the
values in the array (GH3401)
Previously output might look like:
age
today
diff
0 2001-01-01 00:00:00 2013-04-19 00:00:00 4491 days, 00:00:00
1 2004-06-01 00:00:00 2013-04-19 00:00:00 3244 days, 00:00:00
Now the output looks like:
In [19]: df = DataFrame([ Timestamp('20010101'),
....:
Timestamp('20040601') ], columns=['age'])
....:
In [20]: df['today'] = Timestamp('20130419')
In [21]: df['diff'] = df['today']-df['age']
In [22]: df
Out[22]:
age
today
diff
0 2001-01-01 2013-04-19 4491 days
1 2004-06-01 2013-04-19 3244 days
[2 rows x 3 columns]
1.8.2 API changes
• Add -NaN and -nan to the default set of NA values (GH5952). See NA Values.
• Added Series.str.get_dummies vectorized string method (GH6021), to extract dummy/indicator variables for separated string columns:
In [23]: s = Series(['a', 'a|b', np.nan, 'a|c'])
In [24]:
Out[24]:
a b
0 1 0
1 1 1
2 0 0
3 1 0
s.str.get_dummies(sep='|')
c
0
0
0
1
[4 rows x 3 columns]
• Added the NDFrame.equals() method to compare if two NDFrames are equal have equal axes, dtypes, and
values. Added the array_equivalent function to compare if two ndarrays are equal. NaNs in identical
locations are treated as equal. (GH5283) See also the docs for a motivating example.
In [25]: df = DataFrame({'col':['foo', 0, np.nan]})
In [26]: df2 = DataFrame({'col':[np.nan, 0, 'foo']}, index=[2,1,0])
102
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [27]: df.equals(df2)
Out[27]: False
In [28]: df.equals(df2.sort())
Out[28]: True
In [29]: import pandas.core.common as com
In [30]: com.array_equivalent(np.array([0, np.nan]), np.array([0, np.nan]))
Out[30]: True
In [31]: np.array_equal(np.array([0, np.nan]), np.array([0, np.nan]))
Out[31]: False
• DataFrame.apply will use the reduce argument to determine whether a Series or a DataFrame
should be returned when the DataFrame is empty (GH6007).
Previously, calling DataFrame.apply an empty DataFrame would return either a DataFrame if there
were no columns, or the function being applied would be called with an empty Series to guess whether a
Series or DataFrame should be returned:
In [32]: def applied_func(col):
....:
print("Apply function being called with: ", col)
....:
return col.sum()
....:
In [33]: empty = DataFrame(columns=['a', 'b'])
In [34]: empty.apply(applied_func)
('Apply function being called with: ', Series([], dtype: float64))
Out[34]:
a
NaN
b
NaN
dtype: float64
Now, when apply is called on an empty DataFrame: if the reduce argument is True a Series will
returned, if it is False a DataFrame will be returned, and if it is None (the default) the function being
applied will be called with an empty series to try and guess the return type.
In [35]: empty.apply(applied_func, reduce=True)
Out[35]:
a
NaN
b
NaN
dtype: float64
In [36]: empty.apply(applied_func, reduce=False)
Out[36]:
Empty DataFrame
Columns: [a, b]
Index: []
[0 rows x 2 columns]
1.8.3 Prior Version Deprecations/Changes
There are no announced changes in 0.13 or prior that are taking effect as of 0.13.1
1.8. v0.13.1 (February 3, 2014)
103
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.8.4 Deprecations
There are no deprecations of prior behavior in 0.13.1
1.8.5 Enhancements
• pd.read_csv and pd.to_datetime learned a new infer_datetime_format keyword which greatly
improves parsing perf in many cases. Thanks to @lexual for suggesting and @danbirken for rapidly implementing. (GH5490, GH6021)
If parse_dates is enabled and this flag is set, pandas will attempt to infer the format of the datetime strings
in the columns, and if it can be inferred, switch to a faster method of parsing them. In some cases this can
increase the parsing speed by ~5-10x.
# Try to infer the format for the index column
df = pd.read_csv('foo.csv', index_col=0, parse_dates=True,
infer_datetime_format=True)
• date_format and datetime_format keywords can now be specified when writing to excel files
(GH4133)
• MultiIndex.from_product convenience function for creating a MultiIndex from the cartesian product of
a set of iterables (GH6055):
In [37]: shades = ['light', 'dark']
In [38]: colors = ['red', 'green', 'blue']
In [39]: MultiIndex.from_product([shades, colors], names=['shade', 'color'])
Out[39]:
MultiIndex(levels=[[u'dark', u'light'], [u'blue', u'green', u'red']],
labels=[[1, 1, 1, 0, 0, 0], [2, 1, 0, 2, 1, 0]],
names=[u'shade', u'color'])
• Panel apply() will work on non-ufuncs. See the docs.
In [40]: import pandas.util.testing as tm
In [41]: panel = tm.makePanel(5)
In [42]: panel
Out[42]:
Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: A to D
In [43]: panel['ItemA']
Out[43]:
A
B
C
D
2000-01-03 0.952478 -1.239072 -1.409432 -0.014752
2000-01-04 0.988138 0.139683 1.422986 1.272395
2000-01-05 -0.072608 -0.223019 -2.147855 -1.449567
2000-01-06 -0.550603 2.123692 -1.347533 -1.195524
2000-01-07 -0.938153 0.122273 0.363565 -0.591863
[5 rows x 4 columns]
104
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
Specifying an apply that operates on a Series (to return a single element)
In [44]: panel.apply(lambda x: x.dtype, axis='items')
Out[44]:
A
B
C
D
2000-01-03 float64 float64 float64 float64
2000-01-04 float64 float64 float64 float64
2000-01-05 float64 float64 float64 float64
2000-01-06 float64 float64 float64 float64
2000-01-07 float64 float64 float64 float64
[5 rows x 4 columns]
A similar reduction type operation
In [45]: panel.apply(lambda x: x.sum(), axis='major_axis')
Out[45]:
ItemA
ItemB
ItemC
A 0.379252 -3.696907 3.709335
B 0.923558 0.504242 4.656781
C -3.118269 -1.545718 3.188329
D -1.979310 -0.758060 -1.436483
[4 rows x 3 columns]
This is equivalent to
In [46]: panel.sum('major_axis')
Out[46]:
ItemA
ItemB
ItemC
A 0.379252 -3.696907 3.709335
B 0.923558 0.504242 4.656781
C -3.118269 -1.545718 3.188329
D -1.979310 -0.758060 -1.436483
[4 rows x 3 columns]
A transformation operation that returns a Panel, but is computing the z-score across the major_axis
In [47]: result = panel.apply(
....:
lambda x: (x-x.mean())/x.std(),
....:
axis='major_axis')
....:
In [48]: result
Out[48]:
Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: A to D
In [49]: result['ItemA']
Out[49]:
A
B
C
D
2000-01-03 1.004994 -1.166509 -0.535027 0.350970
2000-01-04 1.045875 -0.036892 1.393532 1.536326
2000-01-05 -0.170198 -0.334055 -1.037810 -0.970374
2000-01-06 -0.718186 1.588611 -0.492880 -0.736422
2000-01-07 -1.162486 -0.051156 0.672185 -0.180500
1.8. v0.13.1 (February 3, 2014)
105
pandas: powerful Python data analysis toolkit, Release 0.16.1
[5 rows x 4 columns]
• Panel apply() operating on cross-sectional slabs. (GH1148)
In [50]: f = lambda x: ((x.T-x.mean(1))/x.std(1)).T
In [51]: result = panel.apply(f, axis = ['items','major_axis'])
In [52]: result
Out[52]:
Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis)
Items axis: A to D
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: ItemA to ItemC
In [53]: result.loc[:,:,'ItemA']
Out[53]:
A
B
C
2000-01-03 0.116579 -0.667845 -1.151538
2000-01-04 0.650448 -1.114910 0.841527
2000-01-05 -0.987433 -0.438897 -1.154468
2000-01-06 0.494000 1.060450 -0.775993
2000-01-07 -0.363770 0.013169 0.392036
D
-0.157547
0.760706
-0.015033
-1.140165
-1.123913
[5 rows x 4 columns]
This is equivalent to the following
In [54]: result = Panel(dict([ (ax,f(panel.loc[:,:,ax]))
....:
for ax in panel.minor_axis ]))
....:
In [55]: result
Out[55]:
Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis)
Items axis: A to D
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: ItemA to ItemC
In [56]: result.loc[:,:,'ItemA']
Out[56]:
A
B
C
2000-01-03 0.116579 -0.667845 -1.151538
2000-01-04 0.650448 -1.114910 0.841527
2000-01-05 -0.987433 -0.438897 -1.154468
2000-01-06 0.494000 1.060450 -0.775993
2000-01-07 -0.363770 0.013169 0.392036
D
-0.157547
0.760706
-0.015033
-1.140165
-1.123913
[5 rows x 4 columns]
1.8.6 Performance
Performance improvements for 0.13.1
• Series datetime/timedelta binary operations (GH5801)
106
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• DataFrame count/dropna for axis=1
• Series.str.contains now has a regex=False keyword which can be faster for plain (non-regex) string patterns.
(GH5879)
• Series.str.extract (GH5944)
• dtypes/ftypes methods (GH5968)
• indexing with object dtypes (GH5968)
• DataFrame.apply (GH6013)
• Regression in JSON IO (GH5765)
• Index construction from Series (GH6150)
1.8.7 Experimental
There are no experimental changes in 0.13.1
1.8.8 Bug Fixes
See V0.13.1 Bug Fixes for an extensive list of bugs that have been fixed in 0.13.1.
See the full release notes or issue tracker on GitHub for a complete list of all API changes, Enhancements and Bug
Fixes.
1.9 v0.13.0 (January 3, 2014)
This is a major release from 0.12.0 and includes a number of API changes, several new features and enhancements
along with a large number of bug fixes.
Highlights include:
• support for a new index type Float64Index, and other Indexing enhancements
• HDFStore has a new string based syntax for query specification
• support for new methods of interpolation
• updated timedelta operations
• a new string manipulation method extract
• Nanosecond support for Offsets
• isin for DataFrames
Several experimental features are added, including:
• new eval/query methods for expression evaluation
• support for msgpack serialization
• an i/o interface to Google’s BigQuery
Their are several new or updated docs sections including:
• Comparison with SQL, which should be useful for those familiar with SQL but still learning pandas.
• Comparison with R, idiom translations from R to pandas.
1.9. v0.13.0 (January 3, 2014)
107
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Enhancing Performance, ways to enhance pandas performance with eval/query.
Warning: In 0.13.0 Series has internally been refactored to no longer sub-class ndarray but instead subclass
NDFrame, similar to the rest of the pandas containers. This should be a transparent change with only very limited
API implications. See Internal Refactoring
1.9.1 API changes
• read_excel now supports an integer in its sheetname argument giving the index of the sheet to read in
(GH4301).
• Text parser now treats anything that reads like inf (“inf”, “Inf”, “-Inf”, “iNf”, etc.) as infinity. (GH4220,
GH4219), affecting read_table, read_csv, etc.
• pandas now is Python 2/3 compatible without the need for 2to3 thanks to @jtratner. As a result, pandas now
uses iterators more extensively. This also led to the introduction of substantive parts of the Benjamin Peterson’s
six library into compat. (GH4384, GH4375, GH4372)
• pandas.util.compat and pandas.util.py3compat have been merged into pandas.compat.
pandas.compat now includes many functions allowing 2/3 compatibility. It contains both list and iterator versions of range, filter, map and zip, plus other necessary elements for Python 3 compatibility. lmap,
lzip, lrange and lfilter all produce lists instead of iterators, for compatibility with numpy, subscripting
and pandas constructors.(GH4384, GH4375, GH4372)
• Series.get with negative indexers now returns the same as [] (GH4390)
• Changes to how Index and MultiIndex handle metadata (levels, labels, and names) (GH4039):
# previously, you would have set levels or labels directly
index.levels = [[1, 2, 3, 4], [1, 2, 4, 4]]
# now, you use the set_levels or set_labels methods
index = index.set_levels([[1, 2, 3, 4], [1, 2, 4, 4]])
# similarly, for names, you can rename the object
# but setting names is not deprecated
index = index.set_names(["bob", "cranberry"])
# and all methods take an inplace kwarg - but return None
index.set_names(["bob", "cranberry"], inplace=True)
• All division with NDFrame objects is now truedivision, regardless of the future import. This means that operating on pandas objects will by default use floating point division, and return a floating point dtype. You can use
// and floordiv to do integer division.
Integer division
In [3]: arr = np.array([1, 2, 3, 4])
In [4]: arr2 = np.array([5, 3, 2, 1])
In [5]: arr / arr2
Out[5]: array([0, 0, 1, 4])
In [6]: Series(arr) // Series(arr2)
Out[6]:
0
0
1
0
108
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
1
3
4
dtype: int64
True Division
In [7]: pd.Series(arr) / pd.Series(arr2) # no future import required
Out[7]:
0
0.200000
1
0.666667
2
1.500000
3
4.000000
dtype: float64
• Infer and downcast dtype if downcast=’infer’ is passed to fillna/ffill/bfill (GH4604)
• __nonzero__ for all NDFrame objects, will now raise a ValueError, this reverts back to (GH1073,
GH4633) behavior. See gotchas for a more detailed discussion.
This prevents doing boolean comparison on entire pandas objects, which is inherently ambiguous. These all
will raise a ValueError.
if df:
....
df1 and df2
s1 and s2
Added the .bool() method to NDFrame objects to facilitate evaluating of single-element boolean Series:
In [1]: Series([True]).bool()
Out[1]: True
In [2]: Series([False]).bool()
Out[2]: False
In [3]: DataFrame([[True]]).bool()
Out[3]: True
In [4]: DataFrame([[False]]).bool()
Out[4]: False
• All non-Index NDFrames (Series, DataFrame, Panel, Panel4D, SparsePanel, etc.), now support the
entire set of arithmetic operators and arithmetic flex methods (add, sub, mul, etc.). SparsePanel does not
support pow or mod with non-scalars. (GH3765)
• Series and DataFrame now have a mode() method to calculate the statistical mode(s) by axis/Series.
(GH5367)
• Chained assignment will now by default warn if the user is assigning to a copy. This can be changed with the
option mode.chained_assignment, allowed options are raise/warn/None. See the docs.
In [5]: dfc = DataFrame({'A':['aaa','bbb','ccc'],'B':[1,2,3]})
In [6]: pd.set_option('chained_assignment','warn')
The following warning / exception will show if this is attempted.
In [7]: dfc.loc[0]['A'] = 1111
1.9. v0.13.0 (January 3, 2014)
109
pandas: powerful Python data analysis toolkit, Release 0.16.1
Traceback (most recent call last)
...
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
Here is the correct method of assignment.
In [8]: dfc.loc[0,'A'] = 11
In [9]:
Out[9]:
A
0
11
1 bbb
2 ccc
dfc
B
1
2
3
[3 rows x 2 columns]
• Panel.reindex has the following call signature Panel.reindex(items=None, major_axis=None, minor_ax
to conform with other NDFrame objects. See Internal Refactoring for more information.
• Series.argmin and Series.argmax are now aliased to Series.idxmin and Series.idxmax. These return the i
min or max element respectively. Prior to 0.13.0 these would return the position of the min / max element.
(GH6214)
1.9.2 Prior Version Deprecations/Changes
These were announced changes in 0.12 or prior that are taking effect as of 0.13.0
• Remove deprecated Factor (GH3650)
• Remove deprecated set_printoptions/reset_printoptions (GH3046)
• Remove deprecated _verbose_info (GH3215)
• Remove
deprecated
read_clipboard/to_clipboard/ExcelFile/ExcelWriter
from
pandas.io.parsers (GH3717) These are available as functions in the main pandas namespace (e.g.
pd.read_clipboard)
• default for tupleize_cols is now False for both to_csv and read_csv. Fair warning in 0.12
(GH3604)
• default for display.max_seq_len is now 100 rather then None. This activates truncated display (”...”) of long
sequences in various places. (GH3391)
1.9.3 Deprecations
Deprecated in 0.13.0
• deprecated iterkv, which will be removed in a future release (this was an alias of iteritems used to bypass
2to3‘s changes). (GH4384, GH4375, GH4372)
• deprecated the string method match, whose role is now performed more idiomatically by extract. In a
future release, the default behavior of match will change to become analogous to contains, which returns
a boolean indexer. (Their distinction is strictness: match relies on re.match while contains relies on
re.search.) In this release, the deprecated behavior is the default, but the new behavior is available through
the keyword argument as_indexer=True.
110
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.9.4 Indexing API Changes
Prior to 0.13, it was impossible to use a label indexer (.loc/.ix) to set a value that was not contained in the index
of a particular axis. (GH2578). See the docs
In the Series case this is effectively an appending operation
In [10]: s = Series([1,2,3])
In [11]: s
Out[11]:
0
1
1
2
2
3
dtype: int64
In [12]: s[5] = 5.
In [13]: s
Out[13]:
0
1
1
2
2
3
5
5
dtype: float64
In [14]: dfi = DataFrame(np.arange(6).reshape(3,2),
....:
columns=['A','B'])
....:
In [15]: dfi
Out[15]:
A B
0 0 1
1 2 3
2 4 5
[3 rows x 2 columns]
This would previously KeyError
In [16]: dfi.loc[:,'C'] = dfi.loc[:,'A']
In [17]:
Out[17]:
A B
0 0 1
1 2 3
2 4 5
dfi
C
0
2
4
[3 rows x 3 columns]
This is like an append operation.
In [18]: dfi.loc[3] = 5
In [19]: dfi
Out[19]:
A B C
0 0 1 0
1.9. v0.13.0 (January 3, 2014)
111
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
2
3
2
4
5
3
5
5
2
4
5
[4 rows x 3 columns]
A Panel setting operation on an arbitrary axis aligns the input to the Panel
In [20]: p = pd.Panel(np.arange(16).reshape(2,4,2),
....:
items=['Item1','Item2'],
....:
major_axis=pd.date_range('2001/1/12',periods=4),
....:
minor_axis=['A','B'],dtype='float64')
....:
In [21]: p
Out[21]:
Dimensions: 2 (items) x 4 (major_axis) x 2 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2001-01-12 00:00:00 to 2001-01-15 00:00:00
Minor_axis axis: A to B
In [22]: p.loc[:,:,'C'] = Series([30,32],index=p.items)
In [23]: p
Out[23]:
Dimensions: 2 (items) x 4 (major_axis) x 3 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2001-01-12 00:00:00 to 2001-01-15 00:00:00
Minor_axis axis: A to C
In [24]: p.loc[:,:,'C']
Out[24]:
Item1 Item2
2001-01-12
30
32
2001-01-13
30
32
2001-01-14
30
32
2001-01-15
30
32
[4 rows x 2 columns]
1.9.5 Float64Index API Change
• Added a new index type, Float64Index. This will be automatically created when passing floating values in
index creation. This enables a pure label-based slicing paradigm that makes [],ix,loc for scalar indexing
and slicing work exactly the same. See the docs, (GH263)
Construction is by default for floating type values.
In [25]: index = Index([1.5, 2, 3, 4.5, 5])
In [26]: index
Out[26]: Float64Index([1.5, 2.0, 3.0, 4.5, 5.0], dtype='float64')
In [27]: s = Series(range(5),index=index)
112
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [28]: s
Out[28]:
1.5
0
2.0
1
3.0
2
4.5
3
5.0
4
dtype: int64
Scalar selection for [],.ix,.loc will always be label based. An integer will match an equal float index (e.g.
3 is equivalent to 3.0)
In [29]: s[3]
Out[29]: 2
In [30]: s.ix[3]
Out[30]: 2
In [31]: s.loc[3]
Out[31]: 2
The only positional indexing is via iloc
In [32]: s.iloc[3]
Out[32]: 3
A scalar index that is not found will raise KeyError
Slicing is ALWAYS on the values of the index, for [],ix,loc and ALWAYS positional with iloc
In [33]: s[2:4]
Out[33]:
2
1
3
2
dtype: int64
In [34]: s.ix[2:4]
Out[34]:
2
1
3
2
dtype: int64
In [35]: s.loc[2:4]
Out[35]:
2
1
3
2
dtype: int64
In [36]: s.iloc[2:4]
Out[36]:
3.0
2
4.5
3
dtype: int64
In float indexes, slicing using floats are allowed
In [37]: s[2.1:4.6]
Out[37]:
3.0
2
4.5
3
1.9. v0.13.0 (January 3, 2014)
113
pandas: powerful Python data analysis toolkit, Release 0.16.1
dtype: int64
In [38]: s.loc[2.1:4.6]
Out[38]:
3.0
2
4.5
3
dtype: int64
• Indexing on other index types are preserved (and positional fallback for [],ix), with the exception, that floating
point slicing on indexes on non Float64Index will now raise a TypeError.
In [1]: Series(range(5))[3.5]
TypeError: the label [3.5] is not a proper indexer for this index type (Int64Index)
In [1]: Series(range(5))[3.5:4.5]
TypeError: the slice start [3.5] is not a proper indexer for this index type (Int64Index)
Using a scalar float indexer will be deprecated in a future version, but is allowed for now.
In [3]: Series(range(5))[3.0]
Out[3]: 3
1.9.6 HDFStore API Changes
• Query Format Changes. A much more string-like query format is now supported. See the docs.
In [39]: path = 'test.h5'
In [40]: dfq = DataFrame(randn(10,4),
....:
columns=list('ABCD'),
....:
index=date_range('20130101',periods=10))
....:
In [41]: dfq.to_hdf(path,'dfq',format='table',data_columns=True)
Use boolean expressions, with in-line function evaluation.
In [42]: read_hdf(path,'dfq',
....:
where="index>Timestamp('20130104') & columns=['A', 'B']")
....:
Out[42]:
A
B
2013-01-05 -1.392054 1.153922
2013-01-06 -0.881047 0.295080
2013-01-07 -1.407085 0.126781
2013-01-08 -0.838843 0.553921
2013-01-09 1.529401 0.205455
2013-01-10 0.299071 1.076541
[6 rows x 2 columns]
Use an inline column reference
In [43]: read_hdf(path,'dfq',
....:
where="A>0 or C>0")
....:
Out[43]:
A
B
114
C
D
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
2013-01-01
2013-01-03
2013-01-04
2013-01-05
2013-01-06
2013-01-07
2013-01-09
2013-01-10
1.126386
0.581073
-0.275774
-1.392054
-0.881047
-1.407085
1.529401
0.299071
0.247112
2.763844
0.500483
1.153922
0.295080
0.126781
0.205455
1.076541
0.121172 0.298984
0.399325 0.668488
0.863065 -1.051628
1.181944 0.391371
1.863801 -1.712274
0.003760 -1.268994
0.313013 0.866521
0.363177 1.893680
[8 rows x 4 columns]
• the format keyword now replaces the table keyword; allowed values are fixed(f) or table(t) the
same defaults as prior < 0.13.0 remain, e.g. put implies fixed format and append implies table format.
This default format can be set as an option by setting io.hdf.default_format.
In [44]: path = 'test.h5'
In [45]: df = DataFrame(randn(10,2), columns=['a','b'])
In [46]: df.to_hdf(path,'df_table',format='table')
In [47]: df.to_hdf(path,'df_table2',append=True)
In [48]: df.to_hdf(path,'df_fixed')
In [49]: with get_store(path) as store:
....:
print(store)
....:
File path: test.h5
/df_fixed
frame
(shape->[10,2])
/df_table
frame_table (typ->appendable,nrows->10,ncols->2,indexers->[index])
/df_table2
frame_table (typ->appendable,nrows->10,ncols->2,indexers->[index])
• Significant table writing performance improvements
• handle a passed Series in table format (GH4330)
• can now serialize a timedelta64[ns] dtype in a table (GH3577), See the docs.
• added an is_open property to indicate if the underlying file handle is_open; a closed store will now report
‘CLOSED’ when viewing the store (rather than raising an error) (GH4409)
• a close of a HDFStore now will close that instance of the HDFStore but will only close the actual file if
the ref count (by PyTables) w.r.t. all of the open handles are 0. Essentially you have a local instance of
HDFStore referenced by a variable. Once you close it, it will report closed. Other references (to the same
file) will continue to operate until they themselves are closed. Performing an action on a closed file will raise
ClosedFileError
In [50]: path = 'test.h5'
In [51]: df = DataFrame(randn(10,2))
In [52]: store1 = HDFStore(path)
In [53]: store2 = HDFStore(path)
In [54]: store1.append('df',df)
In [55]: store2.append('df2',df)
1.9. v0.13.0 (January 3, 2014)
115
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [56]: store1
Out[56]:
File path: test.h5
/df
frame_table (typ->appendable,nrows->10,ncols->2,indexers->[index])
In [57]: store2
Out[57]:
File path: test.h5
/df
frame_table (typ->appendable,nrows->10,ncols->2,indexers->[index])
/df2
frame_table (typ->appendable,nrows->10,ncols->2,indexers->[index])
In [58]: store1.close()
In [59]: store2
Out[59]:
File path: test.h5
/df
frame_table (typ->appendable,nrows->10,ncols->2,indexers->[index])
/df2
frame_table (typ->appendable,nrows->10,ncols->2,indexers->[index])
In [60]: store2.close()
In [61]: store2
Out[61]:
File path: test.h5
File is CLOSED
• removed the _quiet attribute, replace by a DuplicateWarning if retrieving duplicate rows from a table
(GH4367)
• removed the warn argument from open. Instead a PossibleDataLossError exception will be raised if
you try to use mode=’w’ with an OPEN file handle (GH4367)
• allow a passed locations array or mask as a where condition (GH4467). See the docs for an example.
• add the keyword dropna=True to append to change whether ALL nan rows are not written to the store
(default is True, ALL nan rows are NOT written), also settable via the option io.hdf.dropna_table
(GH4625)
• pass thru store creation arguments; can be used to support in-memory stores
1.9.7 DataFrame repr Changes
The HTML and plain text representations of DataFrame now show a truncated view of the table once it exceeds
a certain size, rather than switching to the short info view (GH4886, GH5550). This makes the representation more
consistent as small DataFrames get larger.
116
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
To get the info view, call DataFrame.info(). If you prefer the info view as the repr for large DataFrames, you
can set this by running set_option(’display.large_repr’, ’info’).
1.9.8 Enhancements
• df.to_clipboard() learned a new excel keyword that let’s you paste df data directly into excel (enabled
by default). (GH5070).
• read_html now raises a URLError instead of catching and raising a ValueError (GH4303, GH4305)
• Added a test for read_clipboard() and to_clipboard() (GH4282)
• Clipboard functionality now works with PySide (GH4282)
• Added a more informative error message when plot arguments contain overlapping color and style arguments
(GH4402)
• to_dict now takes records as a possible outtype. Returns an array of column-keyed dictionaries. (GH4936)
• NaN handing in get_dummies (GH4446) with dummy_na
# previously, nan was erroneously counted as 2 here
# now it is not counted at all
In [62]: get_dummies([1, 2, np.nan])
Out[62]:
1 2
0 1 0
1 0 1
2 0 0
[3 rows x 2 columns]
# unless requested
In [63]: get_dummies([1, 2, np.nan], dummy_na=True)
Out[63]:
1
2
NaN
0
1
0
0
1
0
1
0
2
0
0
1
[3 rows x 3 columns]
• timedelta64[ns] operations. See the docs.
Warning: Most of these operations require numpy >= 1.7
1.9. v0.13.0 (January 3, 2014)
117
pandas: powerful Python data analysis toolkit, Release 0.16.1
Using the new top-level to_timedelta, you can convert a scalar or array from the standard timedelta format
(produced by to_csv) into a timedelta type (np.timedelta64 in nanoseconds).
In [64]: to_timedelta('1 days 06:05:01.00003')
Out[64]: Timedelta('1 days 06:05:01.000030')
In [65]: to_timedelta('15.5us')
Out[65]: Timedelta('0 days 00:00:00.000015')
In [66]: to_timedelta(['1 days 06:05:01.00003','15.5us','nan'])
Out[66]: TimedeltaIndex(['1 days 06:05:01.000030', '0 days 00:00:00.000015', NaT], dtype='timede
In [67]: to_timedelta(np.arange(5),unit='s')
Out[67]: TimedeltaIndex(['00:00:00', '00:00:01', '00:00:02', '00:00:03', '00:00:04'], dtype='tim
In [68]: to_timedelta(np.arange(5),unit='d')
Out[68]: TimedeltaIndex(['0 days', '1 days', '2 days', '3 days', '4 days'], dtype='timedelta64[n
A Series of dtype timedelta64[ns] can now be divided by another timedelta64[ns] object, or
astyped to yield a float64 dtyped Series. This is frequency conversion. See the docs for the docs.
In [69]: from datetime import timedelta
In [70]: td = Series(date_range('20130101',periods=4))-Series(date_range('20121201',periods=4))
In [71]: td[2] += np.timedelta64(timedelta(minutes=5,seconds=3))
In [72]: td[3] = np.nan
In [73]: td
Out[73]:
0
31 days 00:00:00
1
31 days 00:00:00
2
31 days 00:05:03
3
NaT
dtype: timedelta64[ns]
# to days
In [74]: td / np.timedelta64(1,'D')
Out[74]:
0
31.000000
1
31.000000
2
31.003507
3
NaN
dtype: float64
In [75]: td.astype('timedelta64[D]')
Out[75]:
0
31
1
31
2
31
3
NaN
dtype: float64
# to seconds
In [76]: td / np.timedelta64(1,'s')
Out[76]:
0
2678400
1
2678400
118
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
2678703
3
NaN
dtype: float64
In [77]: td.astype('timedelta64[s]')
Out[77]:
0
2678400
1
2678400
2
2678703
3
NaN
dtype: float64
Dividing or multiplying a timedelta64[ns] Series by an integer or integer Series
In [78]: td * -1
Out[78]:
0
-31 days +00:00:00
1
-31 days +00:00:00
2
-32 days +23:54:57
3
NaT
dtype: timedelta64[ns]
In [79]: td * Series([1,2,3,4])
Out[79]:
0
31 days 00:00:00
1
62 days 00:00:00
2
93 days 00:15:09
3
NaT
dtype: timedelta64[ns]
Absolute DateOffset objects can act equivalently to timedeltas
In [80]: from pandas import offsets
In [81]: td + offsets.Minute(5) + offsets.Milli(5)
Out[81]:
0
31 days 00:05:00.005000
1
31 days 00:05:00.005000
2
31 days 00:10:03.005000
3
NaT
dtype: timedelta64[ns]
Fillna is now supported for timedeltas
In [82]: td.fillna(0)
Out[82]:
0
31 days 00:00:00
1
31 days 00:00:00
2
31 days 00:05:03
3
0 days 00:00:00
dtype: timedelta64[ns]
In [83]: td.fillna(timedelta(days=1,seconds=5))
Out[83]:
0
31 days 00:00:00
1
31 days 00:00:00
2
31 days 00:05:03
3
1 days 00:00:05
dtype: timedelta64[ns]
1.9. v0.13.0 (January 3, 2014)
119
pandas: powerful Python data analysis toolkit, Release 0.16.1
You can do numeric reduction operations on timedeltas.
In [84]: td.mean()
Out[84]: Timedelta('31 days 00:01:41')
In [85]: td.quantile(.1)
Out[85]: Timedelta('31 days 00:00:00')
• plot(kind=’kde’) now accepts the optional parameters bw_method and ind, passed to
scipy.stats.gaussian_kde() (for scipy >= 0.11.0) to set the bandwidth, and to gkde.evaluate() to specify the indices at which it is evaluated, respectively. See scipy docs. (GH4298)
• DataFrame constructor now accepts a numpy masked record array (GH3478)
• The new vectorized string method extract return regular expression matches more conveniently.
In [86]: Series(['a1', 'b2', 'c3']).str.extract('[ab](\d)')
Out[86]:
0
1
1
2
2
NaN
dtype: object
Elements that do not match return NaN. Extracting a regular expression with more than one group returns a
DataFrame with one column per group.
In [87]: Series(['a1', 'b2', 'c3']).str.extract('([ab])(\d)')
Out[87]:
0
1
0
a
1
1
b
2
2 NaN NaN
[3 rows x 2 columns]
Elements that do not match return a row of NaN. Thus, a Series of messy strings can be converted into a likeindexed Series or DataFrame of cleaned-up or more useful strings, without necessitating get() to access tuples
or re.match objects.
Named groups like
In [88]: Series(['a1', 'b2', 'c3']).str.extract(
....:
'(?P[ab])(?P\d)')
....:
Out[88]:
letter digit
0
a
1
1
b
2
2
NaN
NaN
[3 rows x 2 columns]
and optional groups can also be used.
In [89]: Series(['a1', 'b2', '3']).str.extract(
....:
'(?P[ab])?(?P\d)')
....:
Out[89]:
letter digit
0
a
1
1
b
2
120
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
NaN
3
[3 rows x 2 columns]
• read_stata now accepts Stata 13 format (GH4291)
• read_fwf now infers the column specifications from the first 100 rows of the file if the data has correctly
separated and properly aligned columns using the delimiter provided to the function (GH4488).
• support for nanosecond times as an offset
Warning: These operations require numpy >= 1.7
Period conversions in the range of seconds and below were reworked and extended up to nanoseconds. Periods
in the nanosecond range are now available.
In [90]: date_range('2013-01-01', periods=5, freq='5N')
Out[90]:
DatetimeIndex(['2013-01-01', '2013-01-01', '2013-01-01', '2013-01-01',
'2013-01-01'],
dtype='datetime64[ns]', freq='5N', tz=None)
or with frequency as offset
In [91]: date_range('2013-01-01', periods=5, freq=pd.offsets.Nano(5))
Out[91]:
DatetimeIndex(['2013-01-01', '2013-01-01', '2013-01-01', '2013-01-01',
'2013-01-01'],
dtype='datetime64[ns]', freq='5N', tz=None)
Timestamps can be modified in the nanosecond range
In [92]: t = Timestamp('20130101 09:01:02')
In [93]: t + pd.datetools.Nano(123)
Out[93]: Timestamp('2013-01-01 09:01:02.000000123')
• A new method, isin for DataFrames, which plays nicely with boolean indexing. The argument to isin, what
we’re comparing the DataFrame to, can be a DataFrame, Series, dict, or array of values. See the docs for more.
To get the rows where any of the conditions are met:
In [94]: dfi = DataFrame({'A': [1, 2, 3, 4], 'B': ['a', 'b', 'f', 'n']})
In [95]: dfi
Out[95]:
A B
0 1 a
1 2 b
2 3 f
3 4 n
[4 rows x 2 columns]
In [96]: other = DataFrame({'A': [1, 3, 3, 7], 'B': ['e', 'f', 'f', 'e']})
In [97]: mask = dfi.isin(other)
In [98]: mask
1.9. v0.13.0 (January 3, 2014)
121
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[98]:
A
0
True
1 False
2
True
3 False
B
False
False
True
False
[4 rows x 2 columns]
In [99]: dfi[mask.any(1)]
Out[99]:
A B
0 1 a
2 3 f
[2 rows x 2 columns]
• Series now supports a to_frame method to convert it to a single-column DataFrame (GH5164)
• All R datasets listed here http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/00Index.html can now be
loaded into Pandas objects
# note that pandas.rpy was deprecated in v0.16.0
import pandas.rpy.common as com
com.load_data('Titanic')
• tz_localize can infer a fall daylight savings transition based on the structure of the unlocalized data
(GH4230), see the docs
• DatetimeIndex is now in the API documentation, see the docs
• json_normalize() is a new method to allow you to create a flat table from semi-structured JSON data. See
the docs (GH1067)
• Added PySide support for the qtpandas DataFrameModel and DataFrameWidget.
• Python csv parser now supports usecols (GH4335)
• Frequencies gained several new offsets:
– LastWeekOfMonth (GH4637)
– FY5253, and FY5253Quarter (GH4511)
• DataFrame has a new interpolate method, similar to Series (GH4434, GH1892)
In [100]: df = DataFrame({'A': [1, 2.1, np.nan, 4.7, 5.6, 6.8],
.....:
'B': [.25, np.nan, np.nan, 4, 12.2, 14.4]})
.....:
In [101]: df.interpolate()
Out[101]:
A
B
0 1.0
0.25
1 2.1
1.50
2 3.4
2.75
3 4.7
4.00
4 5.6 12.20
5 6.8 14.40
[6 rows x 2 columns]
122
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
Additionally, the method argument to interpolate has been expanded to include ’nearest’,
’zero’, ’slinear’, ’quadratic’, ’cubic’, ’barycentric’, ’krogh’,
’piecewise_polynomial’, ’pchip’, ‘polynomial‘, ’spline’ The new methods require scipy. Consult the Scipy reference guide and documentation for more information about when the various
methods are appropriate. See the docs.
Interpolate now also accepts a limit keyword argument. This works similar to fillna‘s limit:
In [102]: ser = Series([1, 3, np.nan, np.nan, np.nan, 11])
In [103]: ser.interpolate(limit=2)
Out[103]:
0
1
1
3
2
5
3
7
4
NaN
5
11
dtype: float64
• Added wide_to_long panel data convenience function. See the docs.
In [104]: np.random.seed(123)
In [105]: df = pd.DataFrame({"A1970" : {0 : "a", 1 : "b",
.....:
"A1980" : {0 : "d", 1 : "e",
.....:
"B1970" : {0 : 2.5, 1 : 1.2,
.....:
"B1980" : {0 : 3.2, 1 : 1.3,
.....:
"X"
: dict(zip(range(3),
.....:
})
.....:
2 : "c"},
2 : "f"},
2 : .7},
2 : .1},
np.random.randn(3)))
In [106]: df["id"] = df.index
In [107]: df
Out[107]:
A1970 A1980
0
a
d
1
b
e
2
c
f
B1970
2.5
1.2
0.7
B1980
X
3.2 -1.085631
1.3 0.997345
0.1 0.282978
id
0
1
2
[3 rows x 6 columns]
In [108]: wide_to_long(df, ["A", "B"], i="id", j="year")
Out[108]:
X A
B
id year
0 1970 -1.085631 a 2.5
1 1970 0.997345 b 1.2
2 1970 0.282978 c 0.7
0 1980 -1.085631 d 3.2
1 1980 0.997345 e 1.3
2 1980 0.282978 f 0.1
[6 rows x 3 columns]
• to_csv now takes a date_format keyword argument that specifies how output datetime objects should
be formatted. Datetimes encountered in the index, columns, and values will all have this formatting applied.
(GH4313)
1.9. v0.13.0 (January 3, 2014)
123
pandas: powerful Python data analysis toolkit, Release 0.16.1
• DataFrame.plot will scatter plot x versus y by passing kind=’scatter’ (GH2215)
• Added support for Google Analytics v3 API segment IDs that also supports v2 IDs. (GH5271)
1.9.9 Experimental
• The new eval() function implements expression evaluation using numexpr behind the scenes. This results
in large speedups for complicated expressions involving large DataFrames/Series. For example,
In [109]: nrows, ncols = 20000, 100
In [110]: df1, df2, df3, df4 = [DataFrame(randn(nrows, ncols))
.....:
for _ in range(4)]
.....:
# eval with NumExpr backend
In [111]: %timeit pd.eval('df1 + df2 + df3 + df4')
100 loops, best of 3: 14.2 ms per loop
# pure Python evaluation
In [112]: %timeit df1 + df2 + df3 + df4
10 loops, best of 3: 23.4 ms per loop
For more details, see the the docs
• Similar to pandas.eval, DataFrame has a new DataFrame.eval method that evaluates an expression
in the context of the DataFrame. For example,
In [113]: df = DataFrame(randn(10, 2), columns=['a', 'b'])
In [114]: df.eval('a + b')
Out[114]:
0
-0.685204
1
1.589745
2
0.325441
3
-1.784153
4
-0.432893
5
0.171850
6
1.895919
7
3.065587
8
-0.092759
9
1.391365
dtype: float64
• query() method has been added that allows you to select elements of a DataFrame using a natural query
syntax nearly identical to Python syntax. For example,
In [115]: n = 20
In [116]: df = DataFrame(np.random.randint(n, size=(n, 3)), columns=['a', 'b', 'c'])
In [117]: df.query('a < b < c')
Out[117]:
a
b
c
11 1
5
8
15 8 16 19
[2 rows x 3 columns]
124
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
selects all the rows of df where a < b < c evaluates to True. For more details see the the docs.
• pd.read_msgpack() and pd.to_msgpack() are now a supported method of serialization of arbitrary
pandas (and python objects) in a lightweight portable binary format. See the docs
Warning: Since this is an EXPERIMENTAL LIBRARY, the storage format may not be stable until a future
release.
In [118]: df = DataFrame(np.random.rand(5,2),columns=list('AB'))
In [119]: df.to_msgpack('foo.msg')
In [120]: pd.read_msgpack('foo.msg')
Out[120]:
A
B
0 0.251082 0.017357
1 0.347915 0.929879
2 0.546233 0.203368
3 0.064942 0.031722
4 0.355309 0.524575
[5 rows x 2 columns]
In [121]: s = Series(np.random.rand(5),index=date_range('20130101',periods=5))
In [122]: pd.to_msgpack('foo.msg', df, s)
In [123]: pd.read_msgpack('foo.msg')
Out[123]:
[
A
B
0 0.251082 0.017357
1 0.347915 0.929879
2 0.546233 0.203368
3 0.064942 0.031722
4 0.355309 0.524575
[5 rows x 2 columns], 2013-01-01
2013-01-02
0.227025
2013-01-03
0.383282
2013-01-04
0.193225
2013-01-05
0.110977
Freq: D, dtype: float64]
0.022321
You can pass iterator=True to iterator over the unpacked results
In [124]: for o in pd.read_msgpack('foo.msg',iterator=True):
.....:
print o
.....:
A
B
0 0.251082 0.017357
1 0.347915 0.929879
2 0.546233 0.203368
3 0.064942 0.031722
4 0.355309 0.524575
[5 rows x 2 columns]
2013-01-01
0.022321
2013-01-02
0.227025
2013-01-03
0.383282
1.9. v0.13.0 (January 3, 2014)
125
pandas: powerful Python data analysis toolkit, Release 0.16.1
2013-01-04
0.193225
2013-01-05
0.110977
Freq: D, dtype: float64
• pandas.io.gbq provides a simple way to extract from, and load data into, Google’s BigQuery Data Sets by
way of pandas DataFrames. BigQuery is a high performance SQL-like database service, useful for performing
ad-hoc queries against extremely large datasets. See the docs
from pandas.io import gbq
#
#
#
#
A query to select the average monthly temperatures in the
in the year 2000 across the USA. The dataset,
publicata:samples.gsod, is available on all BigQuery accounts,
and is based on NOAA gsod data.
query = """SELECT station_number as STATION,
month as MONTH, AVG(mean_temp) as MEAN_TEMP
FROM publicdata:samples.gsod
WHERE YEAR = 2000
GROUP BY STATION, MONTH
ORDER BY STATION, MONTH ASC"""
# Fetch the result set for this query
# Your Google BigQuery Project ID
# To find this, see your dashboard:
# https://code.google.com/apis/console/b/0/?noredirect
projectid = xxxxxxxxx;
df = gbq.read_gbq(query, project_id = projectid)
# Use pandas to process and reshape the dataset
df2 = df.pivot(index='STATION', columns='MONTH', values='MEAN_TEMP')
df3 = pandas.concat([df2.min(), df2.mean(), df2.max()],
axis=1,keys=["Min Tem", "Mean Temp", "Max Temp"])
The resulting DataFrame is:
> df3
Min Tem
MONTH
1
2
3
4
5
6
7
8
9
10
11
12
126
-53.336667
-49.837500
-77.926087
-82.892858
-92.378261
-77.703334
-87.821428
-89.431999
-86.611112
-78.209677
-50.125000
-50.332258
Mean Temp
39.827892
43.685219
48.708355
55.070087
61.428117
65.858888
68.169663
68.614215
63.436935
56.880838
48.861228
42.286879
Max Temp
89.770968
93.437932
96.099998
97.317240
102.042856
102.900000
106.510714
105.500000
107.142856
92.103333
94.996428
94.396774
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning:
To use this module, you will need a BigQuery account.
See
for details.
As of 10/10/13, there is a bug in Google’s API preventing result sets from being larger than 100,000 rows.
A patch is scheduled for the week of 10/14/13.
1.9.10 Internal Refactoring
In 0.13.0 there is a major refactor primarily to subclass Series from NDFrame, which is the base class currently
for DataFrame and Panel, to unify methods and behaviors. Series formerly subclassed directly from ndarray.
(GH4080, GH3862, GH816)
1.9. v0.13.0 (January 3, 2014)
127
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning: There are two potential incompatibilities from < 0.13.0
• Using certain numpy functions would previously return a Series if passed a Series as an argument. This
seems only to affect np.ones_like, np.empty_like, np.diff and np.where. These now return
ndarrays.
In [125]: s = Series([1,2,3,4])
Numpy Usage
In [126]: np.ones_like(s)
Out[126]: array([1, 1, 1, 1], dtype=int64)
In [127]: np.diff(s)
Out[127]: array([1, 1, 1], dtype=int64)
In [128]: np.where(s>1,s,np.nan)
Out[128]: array([ nan,
2.,
3.,
4.])
Pandonic Usage
In [129]: Series(1,index=s.index)
Out[129]:
0
1
1
1
2
1
3
1
dtype: int64
In [130]: s.diff()
Out[130]:
0
NaN
1
1
2
1
3
1
dtype: float64
In [131]: s.where(s>1)
Out[131]:
0
NaN
1
2
2
3
3
4
dtype: float64
• Passing a Series directly to a cython function expecting an ndarray type will no long work directly, you
must pass Series.values, See Enhancing Performance
• Series(0.5) would previously return the scalar 0.5, instead this will return a 1-element Series
• This change breaks rpy2<=2.3.8. an Issue has been opened against rpy2 and a workaround is detailed in
GH5698. Thanks @JanSchulz.
• Pickle compatibility is preserved for pickles created prior to 0.13.
pd.read_pickle, see Pickling.
These must be unpickled with
• Refactor of series.py/frame.py/panel.py to move common code to generic.py
– added _setup_axes to created generic NDFrame structures
– moved methods
* from_axes,_wrap_array,axes,ix,loc,iloc,shape,empty,swapaxes,transpose,pop
128
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
* __iter__,keys,__contains__,__len__,__neg__,__invert__
* convert_objects,as_blocks,as_matrix,values
* __getstate__,__setstate__ (compat remains in frame/panel)
* __getattr__,__setattr__
* _indexed_same,reindex_like,align,where,mask
* fillna,replace (Series replace is now consistent with DataFrame)
* filter (also added axis argument to selectively filter on a different axis)
* reindex,reindex_axis,take
* truncate (moved to become part of NDFrame)
• These are API changes which make Panel more consistent with DataFrame
– swapaxes on a Panel with the same axes specified now return a copy
– support attribute access for setting
– filter supports the same API as the original DataFrame filter
• Reindex called with no arguments will now return a copy of the input object
• TimeSeries is now an alias for Series. the property is_time_series can be used to distinguish (if
desired)
• Refactor of Sparse objects to use BlockManager
– Created a new block type in internals, SparseBlock, which can hold multi-dtypes and is nonconsolidatable. SparseSeries and SparseDataFrame now inherit more methods from there hierarchy (Series/DataFrame), and no longer inherit from SparseArray (which instead is the object of
the SparseBlock)
– Sparse suite now supports integration with non-sparse data. Non-float sparse data is supportable (partially
implemented)
– Operations on sparse structures within DataFrames should preserve sparseness, merging type operations
will convert to dense (and back to sparse), so might be somewhat inefficient
– enable setitem on SparseSeries for boolean/integer/slices
– SparsePanels implementation is unchanged (e.g. not using BlockManager, needs work)
• added ftypes method to Series/DataFrame, similar to dtypes, but indicates if the underlying is sparse/dense
(as well as the dtype)
• All NDFrame objects can now use __finalize__() to specify various values to propagate to new objects
from an existing one (e.g. name in Series will follow more automatically now)
• Internal type checking is now done via a suite of generated classes, allowing isinstance(value, klass)
without having to directly import the klass, courtesy of @jtratner
• Bug in Series update where the parent frame is not updating its cache based on changes (GH4080) or types
(GH3217), fillna (GH3386)
• Indexing with dtype conversions fixed (GH4463, GH4204)
• Refactor Series.reindex to core/generic.py (GH4604, GH4618), allow method= in reindexing on a Series to work
• Series.copy no longer accepts the order parameter and is now consistent with NDFrame copy
1.9. v0.13.0 (January 3, 2014)
129
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Refactor rename methods to core/generic.py; fixes Series.rename for (GH4605), and adds rename with
the same signature for Panel
• Refactor clip methods to core/generic.py (GH4798)
• Refactor of _get_numeric_data/_get_bool_data to core/generic.py, allowing Series/Panel functionality
• Series (for index) / Panel (for items) now allow attribute access to its elements (GH1903)
In [132]: s = Series([1,2,3],index=list('abc'))
In [133]: s.b
Out[133]: 2
In [134]: s.a = 5
In [135]: s
Out[135]:
a
5
b
2
c
3
dtype: int64
1.9.11 Bug Fixes
See V0.13.0 Bug Fixes for an extensive list of bugs that have been fixed in 0.13.0.
See the full release notes or issue tracker on GitHub for a complete list of all API changes, Enhancements and Bug
Fixes.
1.10 v0.12.0 (July 24, 2013)
This is a major release from 0.11.0 and includes several new features and enhancements along with a large number of
bug fixes.
Highlights include a consistent I/O API naming scheme, routines to read html, write multi-indexes to csv files, read
& write STATA data files, read & write JSON format files, Python 3 support for HDFStore, filtering of groupby
expressions via filter, and a revamped replace routine that accepts regular expressions.
1.10.1 API changes
• The I/O API is now much more consistent with a set of top level reader functions accessed like
pd.read_csv() that generally return a pandas object.
– read_csv
– read_excel
– read_hdf
– read_sql
– read_json
– read_html
– read_stata
130
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
– read_clipboard
The corresponding writer functions are object methods that are accessed like df.to_csv()
– to_csv
– to_excel
– to_hdf
– to_sql
– to_json
– to_html
– to_stata
– to_clipboard
• Fix modulo and integer division on Series,DataFrames to act similary to float dtypes to return np.nan
or np.inf as appropriate (GH3590). This correct a numpy bug that treats integer and float dtypes
differently.
In [1]: p = DataFrame({ 'first' : [4,5,8], 'second' : [0,0,3] })
In [2]: p % 0
Out[2]:
first second
0
NaN
NaN
1
NaN
NaN
2
NaN
NaN
[3 rows x 2 columns]
In [3]: p % p
Out[3]:
first second
0
0
NaN
1
0
NaN
2
0
0
[3 rows x 2 columns]
In [4]: p / p
Out[4]:
first second
0
1
NaN
1
1
NaN
2
1
1
[3 rows x 2 columns]
In [5]: p / 0
Out[5]:
first second
0
inf
NaN
1
inf
NaN
2
inf
inf
[3 rows x 2 columns]
1.10. v0.12.0 (July 24, 2013)
131
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Add squeeze keyword to groupby to allow reduction from DataFrame -> Series if groups are unique. This
is a Regression from 0.10.1. We are reverting back to the prior behavior. This means groupby will return the
same shaped objects whether the groups are unique or not. Revert this issue (GH2893) with (GH3596).
In [6]: df2 = DataFrame([{"val1": 1, "val2" : 20}, {"val1":1, "val2": 19},
...:
{"val1":1, "val2": 27}, {"val1":1, "val2": 12}])
...:
In [7]: def func(dataf):
...:
return dataf["val2"]
...:
- dataf["val2"].mean()
# squeezing the result frame to a series (because we have unique groups)
In [8]: df2.groupby("val1", squeeze=True).apply(func)
Out[8]:
0
0.5
1
-0.5
2
7.5
3
-7.5
Name: 1, dtype: float64
# no squeezing (the default, and behavior in 0.10.1)
In [9]: df2.groupby("val1").apply(func)
Out[9]:
val2
0
1
2
3
val1
1
0.5 -0.5 7.5 -7.5
[1 rows x 4 columns]
• Raise on iloc when boolean indexing with a label based indexer mask e.g. a boolean Series, even with integer
labels, will raise. Since iloc is purely positional based, the labels on the Series are not alignable (GH3631)
This case is rarely used, and there are plently of alternatives. This preserves the iloc API to be purely positional
based.
In [10]: df = DataFrame(lrange(5), list('ABCDE'), columns=['a'])
In [11]: mask = (df.a%2 == 0)
In [12]: mask
Out[12]:
A
True
B
False
C
True
D
False
E
True
Name: a, dtype: bool
# this is what you should use
In [13]: df.loc[mask]
Out[13]:
a
A 0
C 2
E 4
[3 rows x 1 columns]
132
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
# this will work as well
In [14]: df.iloc[mask.values]
Out[14]:
a
A 0
C 2
E 4
[3 rows x 1 columns]
df.iloc[mask] will raise a ValueError
• The raise_on_error argument to plotting functions is removed. Instead, plotting functions raise a
TypeError when the dtype of the object is object to remind you to avoid object arrays whenever
possible and thus you should cast to an appropriate numeric dtype if you need to plot something.
• Add colormap keyword to DataFrame plotting methods. Accepts either a matplotlib colormap object (ie,
matplotlib.cm.jet) or a string name of such an object (ie, ‘jet’). The colormap is sampled to select the color for
each column. Please see Colormaps for more information. (GH3860)
• DataFrame.interpolate() is now deprecated.
Please use DataFrame.fillna() and
DataFrame.replace() instead. (GH3582, GH3675, GH3676)
• the method and axis arguments of DataFrame.replace() are deprecated
• DataFrame.replace ‘s infer_types parameter is removed and now performs conversion by default.
(GH3907)
• Add the keyword allow_duplicates to DataFrame.insert to allow a duplicate column to be inserted
if True, default is False (same as prior to 0.12) (GH3679)
• Implement __nonzero__ for NDFrame objects (GH3691, GH3696)
• IO api
– added top-level function read_excel to replace the following, The original API is deprecated and will
be removed in a future version
from pandas.io.parsers import ExcelFile
xls = ExcelFile('path_to_file.xls')
xls.parse('Sheet1', index_col=None, na_values=['NA'])
With
import pandas as pd
pd.read_excel('path_to_file.xls', 'Sheet1', index_col=None, na_values=['NA'])
– added top-level function read_sql that is equivalent to the following
from pandas.io.sql import read_frame
read_frame(....)
• DataFrame.to_html and DataFrame.to_latex now accept a path for their first argument (GH3702)
• Do not allow astypes on datetime64[ns] except to object, and timedelta64[ns] to object/int
(GH3425)
• The behavior of datetime64 dtypes has changed with respect to certain so-called reduction operations
(GH3726). The following operations now raise a TypeError when perfomed on a Series and return an
empty Series when performed on a DataFrame similar to performing these operations on, for example, a
DataFrame of slice objects:
– sum, prod, mean, std, var, skew, kurt, corr, and cov
1.10. v0.12.0 (July 24, 2013)
133
pandas: powerful Python data analysis toolkit, Release 0.16.1
• read_html now defaults to None when reading, and falls back on bs4 + html5lib when lxml fails to
parse. a list of parsers to try until success is also valid
• The internal pandas class hierarchy has changed (slightly). The previous PandasObject now is called
PandasContainer and a new PandasObject has become the baseclass for PandasContainer as well
as Index, Categorical, GroupBy, SparseList, and SparseArray (+ their base classes). Currently,
PandasObject provides string methods (from StringMixin). (GH4090, GH4092)
• New StringMixin that, given a __unicode__ method, gets python 2 and python 3 compatible string
methods (__str__, __bytes__, and __repr__). Plus string safety throughout. Now employed in many
places throughout the pandas library. (GH4090, GH4092)
1.10.2 I/O Enhancements
• pd.read_html() can now parse HTML strings, files or urls and return DataFrames, courtesy of @cpcloud.
(GH3477, GH3605, GH3606, GH3616). It works with a single parser backend: BeautifulSoup4 + html5lib See
the docs
You can use pd.read_html() to read the output from DataFrame.to_html() like so
In [15]: df = DataFrame({'a': range(3), 'b': list('abc')})
In [16]: print(df)
a b
0 0 a
1 1 b
2 2 c
[3 rows x 2 columns]
In [17]: html = df.to_html()
In [18]: alist = pd.read_html(html, infer_types=True, index_col=0)
In [19]:
a
0 True
1 True
2 True
print(df == alist[0])
b
True
True
True
[3 rows x 2 columns]
Note that alist here is a Python list so pd.read_html() and DataFrame.to_html() are not inverses.
– pd.read_html() no longer performs hard conversion of date strings (GH3656).
Warning: You may have to install an older version of BeautifulSoup4, See the installation docs
• Added module for reading and writing Stata files: pandas.io.stata (GH1512) accessable via
read_stata top-level function for reading, and to_stata DataFrame method for writing, See the docs
• Added module for reading and writing json format files: pandas.io.json accessable via read_json toplevel function for reading, and to_json DataFrame method for writing, See the docs various issues (GH1226,
GH3804, GH3876, GH3867, GH1305)
• MultiIndex column support for reading and writing csv format files
134
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
– The header option in read_csv now accepts a list of the rows from which to read the index.
– The option, tupleize_cols can now be specified in both to_csv and read_csv, to provide compatiblity for the pre 0.12 behavior of writing and reading MultIndex columns via a list of tuples. The
default in 0.12 is to write lists of tuples and not interpret list of tuples as a MultiIndex column.
Note: The default behavior in 0.12 remains unchanged from prior versions, but starting with 0.13, the
default to write and read MultiIndex columns will be in the new format. (GH3571, GH1651, GH3141)
– If an index_col is not specified (e.g. you don’t have an index, or wrote it with df.to_csv(...,
index=False), then any names on the columns index will be lost.
In [20]: from pandas.util.testing import makeCustomDataframe as mkdf
In [21]: df = mkdf(5,3,r_idx_nlevels=2,c_idx_nlevels=4)
In [22]: df.to_csv('mi.csv',tupleize_cols=False)
In [23]: print(open('mi.csv').read())
C0,,C_l0_g0,C_l0_g1,C_l0_g2
C1,,C_l1_g0,C_l1_g1,C_l1_g2
C2,,C_l2_g0,C_l2_g1,C_l2_g2
C3,,C_l3_g0,C_l3_g1,C_l3_g2
R0,R1,,,
R_l0_g0,R_l1_g0,R0C0,R0C1,R0C2
R_l0_g1,R_l1_g1,R1C0,R1C1,R1C2
R_l0_g2,R_l1_g2,R2C0,R2C1,R2C2
R_l0_g3,R_l1_g3,R3C0,R3C1,R3C2
R_l0_g4,R_l1_g4,R4C0,R4C1,R4C2
In [24]: pd.read_csv('mi.csv',header=[0,1,2,3],index_col=[0,1],tupleize_cols=False)
Out[24]:
C0
C_l0_g0 C_l0_g1 C_l0_g2
C1
C_l1_g0 C_l1_g1 C_l1_g2
C2
C_l2_g0 C_l2_g1 C_l2_g2
C3
C_l3_g0 C_l3_g1 C_l3_g2
R0
R1
R_l0_g0 R_l1_g0
R0C0
R0C1
R0C2
R_l0_g1 R_l1_g1
R1C0
R1C1
R1C2
R_l0_g2 R_l1_g2
R2C0
R2C1
R2C2
R_l0_g3 R_l1_g3
R3C0
R3C1
R3C2
R_l0_g4 R_l1_g4
R4C0
R4C1
R4C2
[5 rows x 3 columns]
• Support for HDFStore (via PyTables 3.0.0) on Python3
• Iterator support via read_hdf that automatically opens and closes the store when iteration is finished. This is
only for tables
In [25]: path = 'store_iterator.h5'
In [26]: DataFrame(randn(10,2)).to_hdf(path,'df',table=True)
In [27]: for df in read_hdf(path,'df', chunksize=3):
....:
print(df)
....:
0
1
0 1.392665 -0.123497
1.10. v0.12.0 (July 24, 2013)
135
pandas: powerful Python data analysis toolkit, Release 0.16.1
1 -0.402761 -0.246604
2 -0.288433 -0.763434
[3 rows x 2 columns]
0
1
3 2.069526 -1.203569
4 0.591830 0.841159
5 -0.501083 -0.816561
[3 rows x 2
0
6 -0.207082
7 0.580411
8 -0.038605
columns]
1
-0.664112
-0.965628
-0.460478
[3 rows x 2 columns]
0
1
9 -0.310458 0.866493
[1 rows x 2 columns]
• read_csv will now throw a more informative error message when a file contains no columns, e.g., all newline
characters
1.10.3 Other Enhancements
• DataFrame.replace() now allows regular expressions on contained Series with object dtype. See the
examples section in the regular docs Replacing via String Expression
For example you can do
In [28]: df = DataFrame({'a': list('ab..'), 'b': [1, 2, 3, 4]})
In [29]: df.replace(regex=r'\s*\.\s*', value=np.nan)
Out[29]:
a b
0
a 1
1
b 2
2 NaN 3
3 NaN 4
[4 rows x 2 columns]
to replace all occurrences of the string ’.’ with zero or more instances of surrounding whitespace with NaN.
Regular string replacement still works as expected. For example, you can do
In [30]: df.replace('.', np.nan)
Out[30]:
a b
0
a 1
1
b 2
2 NaN 3
3 NaN 4
[4 rows x 2 columns]
to replace all occurrences of the string ’.’ with NaN.
136
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• pd.melt() now accepts the optional parameters var_name and value_name to specify custom column
names of the returned DataFrame.
• pd.set_option() now allows N option, value pairs (GH3667).
Let’s say that we had an option ’a.b’ and another option ’b.c’. We can set them at the same
time:
In [31]: pd.get_option('a.b')
Out[31]: 2
In [32]: pd.get_option('b.c')
Out[32]: 3
In [33]: pd.set_option('a.b', 1, 'b.c', 4)
In [34]: pd.get_option('a.b')
Out[34]: 1
In [35]: pd.get_option('b.c')
Out[35]: 4
• The filter method for group objects returns a subset of the original object. Suppose we want to take only
elements that belong to groups with a group sum greater than 2.
In [36]: sf = Series([1, 1, 2, 3, 3, 3])
In [37]: sf.groupby(sf).filter(lambda x: x.sum() > 2)
Out[37]:
3
3
4
3
5
3
dtype: int64
The argument of filter must a function that, applied to the group as a whole, returns True or False.
Another useful operation is filtering out elements that belong to groups with only a couple members.
In [38]: dff = DataFrame({'A': np.arange(8), 'B': list('aabbbbcc')})
In [39]: dff.groupby('B').filter(lambda x: len(x) > 2)
Out[39]:
A B
2 2 b
3 3 b
4 4 b
5 5 b
[4 rows x 2 columns]
Alternatively, instead of dropping the offending groups, we can return a like-indexed objects where the groups
that do not pass the filter are filled with NaNs.
In [40]: dff.groupby('B').filter(lambda x: len(x) > 2, dropna=False)
Out[40]:
A
B
0 NaN NaN
1 NaN NaN
2
2
b
3
3
b
4
4
b
1.10. v0.12.0 (July 24, 2013)
137
pandas: powerful Python data analysis toolkit, Release 0.16.1
5
5
6 NaN
7 NaN
b
NaN
NaN
[8 rows x 2 columns]
• Series and DataFrame hist methods now take a figsize argument (GH3834)
• DatetimeIndexes no longer try to convert mixed-integer indexes during join operations (GH3877)
• Timestamp.min and Timestamp.max now represent valid Timestamp instances instead of the default datetime.min and datetime.max (respectively), thanks @SleepingPills
• read_html now raises when no tables are found and BeautifulSoup==4.2.0 is detected (GH4214)
1.10.4 Experimental Features
• Added experimental CustomBusinessDay class to support DateOffsets with custom holiday calendars
and custom weekmasks. (GH2301)
Note: This uses the numpy.busdaycalendar API introduced in Numpy 1.7 and therefore requires Numpy
1.7.0 or newer.
In [41]: from pandas.tseries.offsets import CustomBusinessDay
In [42]: from datetime import datetime
# As an interesting example, let's look at Egypt where
# a Friday-Saturday weekend is observed.
In [43]: weekmask_egypt = 'Sun Mon Tue Wed Thu'
# They also observe International Workers' Day so let's
# add that for a couple of years
In [44]: holidays = ['2012-05-01', datetime(2013, 5, 1), np.datetime64('2014-05-01')]
In [45]: bday_egypt = CustomBusinessDay(holidays=holidays, weekmask=weekmask_egypt)
In [46]: dt = datetime(2013, 4, 30)
In [47]: print(dt + 2 * bday_egypt)
2013-05-05 00:00:00
In [48]: dts = date_range(dt, periods=5, freq=bday_egypt)
In [49]: print(Series(dts.weekday, dts).map(Series('Mon Tue Wed Thu Fri Sat Sun'.split())))
2013-04-30
Tue
2013-05-02
Thu
2013-05-05
Sun
2013-05-06
Mon
2013-05-07
Tue
Freq: C, dtype: object
1.10.5 Bug Fixes
• Plotting functions now raise a TypeError before trying to plot anything if the associated objects have have a
dtype of object (GH1818, GH3572, GH3911, GH3912), but they will try to convert object arrays to numeric
138
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
arrays if possible so that you can still plot, for example, an object array with floats. This happens before any
drawing takes place which elimnates any spurious plots from showing up.
• fillna methods now raise a TypeError if the value parameter is a list or tuple.
• Series.str now supports iteration (GH3638). You can iterate over the individual elements of each string in
the Series. Each iteration yields yields a Series with either a single character at each index of the original
Series or NaN. For example,
In [50]: strs = 'go', 'bow', 'joe', 'slow'
In [51]: ds = Series(strs)
In [52]: for s in ds.str:
....:
print(s)
....:
0
g
1
b
2
j
3
s
dtype: object
0
o
1
o
2
o
3
l
dtype: object
0
NaN
1
w
2
e
3
o
dtype: object
0
NaN
1
NaN
2
NaN
3
w
dtype: object
In [53]: s
Out[53]:
0
NaN
1
NaN
2
NaN
3
w
dtype: object
In [54]: s.dropna().values.item() == 'w'
Out[54]: True
The last element yielded by the iterator will be a Series containing the last element of the longest string in
the Series with all other elements being NaN. Here since ’slow’ is the longest string and there are no other
strings with the same length ’w’ is the only non-null string in the yielded Series.
• HDFStore
– will retain index attributes (freq,tz,name) on recreation (GH3499)
– will warn with a AttributeConflictWarning if you are attempting to append an index with a
different frequency than the existing, or attempting to append an index with a different name than the
existing
– support datelike columns with a timezone as data_columns (GH2852)
1.10. v0.12.0 (July 24, 2013)
139
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Non-unique index support clarified (GH3468).
– Fix assigning a new index to a duplicate index in a DataFrame would fail (GH3468)
– Fix construction of a DataFrame with a duplicate index
– ref_locs support to allow duplicative indices across dtypes, allows iget support to always find the index
(even across dtypes) (GH2194)
– applymap on a DataFrame with a non-unique index now works (removed warning) (GH2786), and fix
(GH3230)
– Fix to_csv to handle non-unique columns (GH3495)
– Duplicate indexes with getitem will return items in the correct order (GH3455, GH3457) and handle missing elements like unique indices (GH3561)
– Duplicate indexes with and empty DataFrame.from_records will return a correct frame (GH3562)
– Concat to produce a non-unique columns when duplicates are across dtypes is fixed (GH3602)
– Allow insert/delete to non-unique columns (GH3679)
– Non-unique indexing with a slice via loc and friends fixed (GH3659)
– Allow insert/delete to non-unique columns (GH3679)
– Extend reindex to correctly deal with non-unique indices (GH3679)
– DataFrame.itertuples() now works with frames with duplicate column names (GH3873)
– Bug in non-unique indexing via iloc (GH4017); added takeable argument to reindex for locationbased taking
– Allow non-unique indexing in series via .ix/.loc and __getitem__ (GH4246)
– Fixed non-unique indexing memory allocation issue with .ix/.loc (GH4280)
• DataFrame.from_records did not accept empty recarrays (GH3682)
• read_html now correctly skips tests (GH3741)
• Fixed a bug where DataFrame.replace with a compiled regular expression in the to_replace argument
wasn’t working (GH3907)
• Improved network test decorator to catch IOError (and therefore URLError as well). Added
with_connectivity_check decorator to allow explicitly checking a website as a proxy for seeing if there
is network connectivity. Plus, new optional_args decorator factory for decorators. (GH3910, GH3914)
• Fixed testing issue where too many sockets where open thus leading to a connection reset issue (GH3982,
GH3985, GH4028, GH4054)
• Fixed failing tests in test_yahoo, test_google where symbols were not retrieved but were being accessed
(GH3982, GH3985, GH4028, GH4054)
• Series.hist will now take the figure from the current environment if one is not passed
• Fixed bug where a 1xN DataFrame would barf on a 1xN mask (GH4071)
• Fixed running of tox under python3 where the pickle import was getting rewritten in an incompatible way
(GH4062, GH4063)
• Fixed bug where sharex and sharey were not being passed to grouped_hist (GH4089)
• Fixed bug in DataFrame.replace where a nested dict wasn’t being iterated over when regex=False
(GH4115)
• Fixed bug in the parsing of microseconds when using the format argument in to_datetime (GH4152)
140
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Fixed bug in PandasAutoDateLocator
MilliSecondLocator (GH3990)
where
invert_xaxis
triggered
incorrectly
• Fixed bug in plotting that wasn’t raising on invalid colormap for matplotlib 1.1.1 (GH4215)
• Fixed the legend displaying in DataFrame.plot(kind=’kde’) (GH4216)
• Fixed bug where Index slices weren’t carrying the name attribute (GH4226)
• Fixed bug in initializing DatetimeIndex with an array of strings in a certain time zone (GH4229)
• Fixed bug where html5lib wasn’t being properly skipped (GH4265)
• Fixed bug where get_data_famafrench wasn’t using the correct file edges (GH4281)
See the full release notes or issue tracker on GitHub for a complete list.
1.11 v0.11.0 (April 22, 2013)
This is a major release from 0.10.1 and includes many new features and enhancements along with a large number of
bug fixes. The methods of Selecting Data have had quite a number of additions, and Dtype support is now full-fledged.
There are also a number of important API changes that long-time pandas users should pay close attention to.
There is a new section in the documentation, 10 Minutes to Pandas, primarily geared to new users.
There is a new section in the documentation, Cookbook, a collection of useful recipes in pandas (and that we want
contributions!).
There are several libraries that are now Recommended Dependencies
1.11.1 Selection Choices
Starting in 0.11.0, object selection has had a number of user-requested additions in order to support more explicit
location based indexing. Pandas now supports three types of multi-axis indexing.
• .loc is strictly label based, will raise KeyError when the items are not found, allowed inputs are:
– A single label, e.g. 5 or ’a’, (note that 5 is interpreted as a label of the index. This use is not an integer
position along the index)
– A list or array of labels [’a’, ’b’, ’c’]
– A slice object with labels ’a’:’f’, (note that contrary to usual python slices, both the start and the stop
are included!)
– A boolean array
See more at Selection by Label
• .iloc is strictly integer position based (from 0 to length-1 of the axis), will raise IndexError when the
requested indicies are out of bounds. Allowed inputs are:
– An integer e.g. 5
– A list or array of integers [4, 3, 0]
– A slice object with ints 1:7
– A boolean array
See more at Selection by Position
1.11. v0.11.0 (April 22, 2013)
141
pandas: powerful Python data analysis toolkit, Release 0.16.1
• .ix supports mixed integer and label based access. It is primarily label based, but will fallback to integer
positional access. .ix is the most general and will support any of the inputs to .loc and .iloc, as well as
support for floating point label schemes. .ix is especially useful when dealing with mixed positional and label
based hierarchial indexes.
As using integer slices with .ix have different behavior depending on whether the slice is interpreted as position
based or label based, it’s usually better to be explicit and use .iloc or .loc.
See more at Advanced Indexing and Advanced Hierarchical.
1.11.2 Selection Deprecations
Starting in version 0.11.0, these methods may be deprecated in future versions.
• irow
• icol
• iget_value
See the section Selection by Position for substitutes.
1.11.3 Dtypes
Numeric dtypes will propagate and can coexist in DataFrames. If a dtype is passed (either directly via the dtype
keyword, a passed ndarray, or a passed Series, then it will be preserved in DataFrame operations. Furthermore,
different numeric dtypes will NOT be combined. The following example will give you a taste.
In [1]: df1 = DataFrame(randn(8, 1), columns = ['A'], dtype = 'float32')
In [2]: df1
Out[2]:
A
0 0.245972
1 0.319442
2 1.378512
3 0.292502
4 0.329791
5 1.392047
6 0.769914
7 -2.472300
[8 rows x 1 columns]
In [3]: df1.dtypes
Out[3]:
A
float32
dtype: object
In [4]: df2 = DataFrame(dict( A = Series(randn(8),dtype='float16'),
...:
B = Series(randn(8)),
...:
C = Series(randn(8),dtype='uint8') ))
...:
In [5]: df2
Out[5]:
A
B
0 -0.611328 -0.270630
142
C
255
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1 1.044922 -1.685677
2 1.503906 -0.440747
3 -1.328125 -0.115070
4 1.024414 -0.632102
5 0.660156 -0.585977
6 1.236328 -1.444787
7 -2.169922 -0.201135
0
0
1
0
0
0
0
[8 rows x 3 columns]
In [6]: df2.dtypes
Out[6]:
A
float16
B
float64
C
uint8
dtype: object
# here you get some upcasting
In [7]: df3 = df1.reindex_like(df2).fillna(value=0.0) + df2
In [8]: df3
Out[8]:
A
0 -0.365356
1 1.364364
2 2.882418
3 -1.035623
4 1.354205
5 2.052203
6 2.006243
7 -4.642221
B
-0.270630
-1.685677
-0.440747
-0.115070
-0.632102
-0.585977
-1.444787
-0.201135
C
255
0
0
1
0
0
0
0
[8 rows x 3 columns]
In [9]: df3.dtypes
Out[9]:
A
float32
B
float64
C
float64
dtype: object
1.11.4 Dtype Conversion
This is lower-common-denomicator upcasting, meaning you get the dtype which can accomodate all of the types
In [10]: df3.values.dtype
Out[10]: dtype('float64')
Conversion
In [11]: df3.astype('float32').dtypes
Out[11]:
A
float32
B
float32
C
float32
dtype: object
Mixed Conversion
1.11. v0.11.0 (April 22, 2013)
143
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [12]: df3['D'] = '1.'
In [13]: df3['E'] = '1'
In [14]: df3.convert_objects(convert_numeric=True).dtypes
Out[14]:
A
float32
B
float64
C
float64
D
float64
E
int64
dtype: object
# same, but specific dtype conversion
In [15]: df3['D'] = df3['D'].astype('float16')
In [16]: df3['E'] = df3['E'].astype('int32')
In [17]: df3.dtypes
Out[17]:
A
float32
B
float64
C
float64
D
float16
E
int32
dtype: object
Forcing Date coercion (and setting NaT when not datelike)
In [18]: from datetime import datetime
In [19]: s = Series([datetime(2001,1,1,0,0), 'foo', 1.0, 1,
....:
Timestamp('20010104'), '20010105'],dtype='O')
....:
In [20]: s.convert_objects(convert_dates='coerce')
Out[20]:
0
2001-01-01
1
NaT
2
NaT
3
NaT
4
2001-01-04
5
2001-01-05
dtype: datetime64[ns]
1.11.5 Dtype Gotchas
Platform Gotchas
Starting in 0.11.0, construction of DataFrame/Series will use default dtypes of int64 and float64, regardless
of platform. This is not an apparent change from earlier versions of pandas. If you specify dtypes, they WILL be
respected, however (GH2837)
The following will all result in int64 dtypes
In [21]: DataFrame([1,2],columns=['a']).dtypes
Out[21]:
a
int64
144
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
dtype: object
In [22]: DataFrame({'a' : [1,2] }).dtypes
Out[22]:
a
int64
dtype: object
In [23]: DataFrame({'a' : 1 }, index=range(2)).dtypes
Out[23]:
a
int64
dtype: object
Keep in mind that DataFrame(np.array([1,2])) WILL result in int32 on 32-bit platforms!
Upcasting Gotchas
Performing indexing operations on integer type data can easily upcast the data. The dtype of the input data will be
preserved in cases where nans are not introduced.
In [24]: dfi = df3.astype('int32')
In [25]: dfi['D'] = dfi['D'].astype('int64')
In [26]: dfi
Out[26]:
A B
C
0 0 0 255
1 1 -1
0
2 2 0
0
3 -1 0
1
4 1 0
0
5 2 0
0
6 2 -1
0
7 -4 0
0
D
1
1
1
1
1
1
1
1
E
1
1
1
1
1
1
1
1
[8 rows x 5 columns]
In [27]: dfi.dtypes
Out[27]:
A
int32
B
int32
C
int32
D
int64
E
int32
dtype: object
In [28]: casted = dfi[dfi>0]
In [29]: casted
Out[29]:
A
B
C
0 NaN NaN 255
1
1 NaN NaN
2
2 NaN NaN
3 NaN NaN
1
4
1 NaN NaN
5
2 NaN NaN
6
2 NaN NaN
7 NaN NaN NaN
D
1
1
1
1
1
1
1
1
E
1
1
1
1
1
1
1
1
1.11. v0.11.0 (April 22, 2013)
145
pandas: powerful Python data analysis toolkit, Release 0.16.1
[8 rows x 5 columns]
In [30]: casted.dtypes
Out[30]:
A
float64
B
float64
C
float64
D
int64
E
int32
dtype: object
While float dtypes are unchanged.
In [31]: df4 = df3.copy()
In [32]: df4['A'] = df4['A'].astype('float32')
In [33]: df4.dtypes
Out[33]:
A
float32
B
float64
C
float64
D
float16
E
int32
dtype: object
In [34]: casted = df4[df4>0]
In [35]: casted
Out[35]:
A
B
0
NaN NaN
1 1.364364 NaN
2 2.882418 NaN
3
NaN NaN
4 1.354205 NaN
5 2.052203 NaN
6 2.006243 NaN
7
NaN NaN
C
255
NaN
NaN
1
NaN
NaN
NaN
NaN
D
1
1
1
1
1
1
1
1
E
1
1
1
1
1
1
1
1
[8 rows x 5 columns]
In [36]: casted.dtypes
Out[36]:
A
float32
B
float64
C
float64
D
float16
E
int32
dtype: object
1.11.6 Datetimes Conversion
Datetime64[ns] columns in a DataFrame (or a Series) allow the use of np.nan to indicate a nan value, in addition to the traditional NaT, or not-a-time. This allows convenient nan setting in a generic way. Furthermore
datetime64[ns] columns are created by default, when passed datetimelike objects (this change was introduced in
146
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
0.10.1) (GH2809, GH2810)
In [37]: df = DataFrame(randn(6,2),date_range('20010102',periods=6),columns=['A','B'])
In [38]: df['timestamp'] = Timestamp('20010103')
In [39]: df
Out[39]:
2001-01-02
2001-01-03
2001-01-04
2001-01-05
2001-01-06
2001-01-07
A
B timestamp
-1.448835 0.153437 2001-01-03
-1.123570 -0.791498 2001-01-03
0.105400 1.262401 2001-01-03
-0.721844 -0.647645 2001-01-03
-0.830631 0.761823 2001-01-03
0.597819 1.045558 2001-01-03
[6 rows x 3 columns]
# datetime64[ns] out of the box
In [40]: df.get_dtype_counts()
Out[40]:
datetime64[ns]
1
float64
2
dtype: int64
# use the traditional nan, which is mapped to NaT internally
In [41]: df.ix[2:4,['A','timestamp']] = np.nan
In [42]: df
Out[42]:
A
B timestamp
2001-01-02 -1.448835 0.153437 2001-01-03
2001-01-03 -1.123570 -0.791498 2001-01-03
2001-01-04
NaN 1.262401
NaT
2001-01-05
NaN -0.647645
NaT
2001-01-06 -0.830631 0.761823 2001-01-03
2001-01-07 0.597819 1.045558 2001-01-03
[6 rows x 3 columns]
Astype conversion on datetime64[ns] to object, implicity converts NaT to np.nan
In [43]: import datetime
In [44]: s = Series([datetime.datetime(2001, 1, 2, 0, 0) for i in range(3)])
In [45]: s.dtype
Out[45]: dtype('2'])
Out[54]:
A B
3 3 3
4 4 4
[2 rows x 2 columns]
– provide dotted attribute access to get from stores, e.g. store.df == store[’df’]
– new keywords iterator=boolean, and chunksize=number_in_a_chunk are provided to support iteration on select and select_as_multiple (GH3076)
• You can now select timestamps from an unordered timeseries similarly to an ordered timeseries (GH2437)
• You can now select with a string from a DataFrame with a datelike index, in a similar way to a Series (GH3070)
148
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [55]: idx = date_range("2001-10-1", periods=5, freq='M')
In [56]: ts = Series(np.random.rand(len(idx)),index=idx)
In [57]: ts['2001']
Out[57]:
2001-10-31
0.483450
2001-11-30
0.407530
2001-12-31
0.965096
Freq: M, dtype: float64
In [58]: df = DataFrame(dict(A = ts))
In [59]: df['2001']
Out[59]:
A
2001-10-31 0.483450
2001-11-30 0.407530
2001-12-31 0.965096
[3 rows x 1 columns]
• Squeeze to possibly remove length 1 dimensions from an object.
In [60]: p = Panel(randn(3,4,4),items=['ItemA','ItemB','ItemC'],
....:
major_axis=date_range('20010102',periods=4),
....:
minor_axis=['A','B','C','D'])
....:
In [61]: p
Out[61]:
Dimensions: 3 (items) x 4 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2001-01-02 00:00:00 to 2001-01-05 00:00:00
Minor_axis axis: A to D
In [62]: p.reindex(items=['ItemA']).squeeze()
Out[62]:
A
B
C
D
2001-01-02 0.396537 0.534880 -0.488797 -1.539385
2001-01-03 -0.829037 0.306681 -0.331032 1.544977
2001-01-04 -0.621754 1.026208 -0.413106 -1.490869
2001-01-05 -1.253235 -0.538879 -1.487449 -1.426475
[4 rows x 4 columns]
In [63]: p.reindex(items=['ItemA'],minor=['B']).squeeze()
Out[63]:
2001-01-02
0.534880
2001-01-03
0.306681
2001-01-04
1.026208
2001-01-05
-0.538879
Freq: D, Name: B, dtype: float64
• In pd.io.data.Options,
– Fix bug when trying to fetch data for the current month when already past expiry.
1.11. v0.11.0 (April 22, 2013)
149
pandas: powerful Python data analysis toolkit, Release 0.16.1
– Now using lxml to scrape html instead of BeautifulSoup (lxml was faster).
– New instance variables for calls and puts are automatically created when a method that creates them is
called. This works for current month where the instance variables are simply calls and puts. Also
works for future expiry months and save the instance variable as callsMMYY or putsMMYY, where
MMYY are, respectively, the month and year of the option’s expiry.
– Options.get_near_stock_price now allows the user to specify the month for which to get relevant options data.
– Options.get_forward_data now has optional kwargs near and above_below. This allows the
user to specify if they would like to only return forward looking data for options near the current stock
price. This just obtains the data from Options.get_near_stock_price instead of Options.get_xxx_data()
(GH2758).
• Cursor coordinate information is now displayed in time-series plots.
• added option display.max_seq_items to control the number of elements printed per sequence pprinting it.
(GH2979)
• added option display.chop_threshold to control display of small numerical values. (GH2739)
• added option display.max_info_rows to prevent verbose_info from being calculated for frames above 1M rows
(configurable). (GH2807, GH2918)
• value_counts() now accepts a “normalize” argument, for normalized histograms. (GH2710).
• DataFrame.from_records now accepts not only dicts but any instance of the collections.Mapping ABC.
• added option display.mpl_style providing a
https://gist.github.com/huyng/816622 (GH3075).
sleeker
visual
style
for
plots.
Based
on
• Treat boolean values as integers (values 1 and 0) for numeric operations. (GH2641)
• to_html() now accepts an optional “escape” argument to control reserved HTML character escaping (enabled
by default) and escapes &, in addition to < and >. (GH2919)
See the full release notes or issue tracker on GitHub for a complete list.
1.12 v0.10.1 (January 22, 2013)
This is a minor release from 0.10.0 and includes new features, enhancements, and bug fixes. In particular, there is
substantial new HDFStore functionality contributed by Jeff Reback.
An undesired API breakage with functions taking the inplace option has been reverted and deprecation warnings
added.
1.12.1 API changes
• Functions taking an inplace option return the calling object as before. A deprecation message has been added
• Groupby aggregations Max/Min no longer exclude non-numeric data (GH2700)
• Resampling an empty DataFrame now returns an empty DataFrame instead of raising an exception (GH2640)
• The file reader will now raise an exception when NA values are found in an explicitly specified integer column
instead of converting the column to float (GH2631)
• DatetimeIndex.unique now returns a DatetimeIndex with the same name and
• timezone instead of an array (GH2563)
150
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.12.2 New features
• MySQL support for database (contribution from Dan Allan)
1.12.3 HDFStore
You may need to upgrade your existing data files. Please visit the compatibility section in the main docs.
You can designate (and index) certain columns that you want to be able to perform queries on a table, by passing a list
to data_columns
In [1]: store = HDFStore('store.h5')
In [2]: df = DataFrame(randn(8, 3), index=date_range('1/1/2000', periods=8),
...:
columns=['A', 'B', 'C'])
...:
In [3]: df['string'] = 'foo'
In [4]: df.ix[4:6,'string'] = np.nan
In [5]: df.ix[7:9,'string'] = 'bar'
In [6]: df['string2'] = 'cool'
In [7]: df
Out[7]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
-1.601262
0.174122
0.980347
-0.761218
-0.862613
1.498195
1.511487
-0.007364
B
-0.256718
-1.131794
-0.674429
1.768215
-0.210968
0.462413
-0.727189
1.427674
C string string2
0.239369
foo
cool
-1.948006
foo
cool
-0.361633
foo
cool
0.152288
foo
cool
-0.859278
NaN
cool
-0.647604
NaN
cool
-0.342928
foo
cool
0.104020
bar
cool
[8 rows x 5 columns]
# on-disk operations
In [8]: store.append('df', df, data_columns = ['B','C','string','string2'])
In [9]: store.select('df',[ 'B > 0', 'string == foo' ])
Out[9]:
A
B
C string string2
2000-01-04 -0.761218 1.768215 0.152288
foo
cool
[1 rows x 5 columns]
# this is in-memory version of this type of selection
In [10]: df[(df.B > 0) & (df.string == 'foo')]
Out[10]:
A
B
C string string2
2000-01-04 -0.761218 1.768215 0.152288
foo
cool
[1 rows x 5 columns]
Retrieving unique values in an indexable or data column.
1.12. v0.10.1 (January 22, 2013)
151
pandas: powerful Python data analysis toolkit, Release 0.16.1
# note that this is deprecated as of 0.14.0
# can be replicated by: store.select_column('df','index').unique()
store.unique('df','index')
store.unique('df','string')
You can now store datetime64 in data columns
In [11]: df_mixed
= df.copy()
In [12]: df_mixed['datetime64'] = Timestamp('20010102')
In [13]: df_mixed.ix[3:4,['A','B']] = np.nan
In [14]: store.append('df_mixed', df_mixed)
In [15]: df_mixed1 = store.select('df_mixed')
In [16]: df_mixed1
Out[16]:
A
B
C string string2 datetime64
2000-01-01 -1.601262 -0.256718 0.239369
foo
cool 2001-01-02
2000-01-02 0.174122 -1.131794 -1.948006
foo
cool 2001-01-02
2000-01-03 0.980347 -0.674429 -0.361633
foo
cool 2001-01-02
2000-01-04
NaN
NaN 0.152288
foo
cool 2001-01-02
2000-01-05 -0.862613 -0.210968 -0.859278
NaN
cool 2001-01-02
2000-01-06 1.498195 0.462413 -0.647604
NaN
cool 2001-01-02
2000-01-07 1.511487 -0.727189 -0.342928
foo
cool 2001-01-02
2000-01-08 -0.007364 1.427674 0.104020
bar
cool 2001-01-02
[8 rows x 6 columns]
In [17]: df_mixed1.get_dtype_counts()
Out[17]:
datetime64[ns]
1
float64
3
object
2
dtype: int64
You can pass columns keyword to select to filter a list of the return columns, this is equivalent to passing a
Term(’columns’,list_of_columns_to_filter)
In [18]: store.select('df',columns = ['A','B'])
Out[18]:
A
B
2000-01-01 -1.601262 -0.256718
2000-01-02 0.174122 -1.131794
2000-01-03 0.980347 -0.674429
2000-01-04 -0.761218 1.768215
2000-01-05 -0.862613 -0.210968
2000-01-06 1.498195 0.462413
2000-01-07 1.511487 -0.727189
2000-01-08 -0.007364 1.427674
[8 rows x 2 columns]
HDFStore now serializes multi-index dataframes when appending tables.
In [19]: index = MultiIndex(levels=[['foo', 'bar', 'baz', 'qux'],
....:
['one', 'two', 'three']],
152
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
....:
....:
....:
....:
labels=[[0, 0, 0, 1, 1, 2, 2, 3, 3, 3],
[0, 1, 2, 0, 1, 1, 2, 0, 1, 2]],
names=['foo', 'bar'])
In [20]: df = DataFrame(np.random.randn(10, 3), index=index,
....:
columns=['A', 'B', 'C'])
....:
In [21]: df
Out[21]:
A
B
C
foo bar
foo one
2.052171 -1.230963 -0.019240
two
-1.713238 0.838912 -0.637855
three 0.215109 -1.515362 1.586924
bar one
-0.447974 -1.573998 0.630925
two
-0.071659 -1.277640 -0.102206
baz two
0.870302 1.275280 -1.199212
three 1.060780 1.673018 1.249874
qux one
1.458210 -0.710542 0.825392
two
1.557329 1.993441 -0.616293
three 0.150468 0.132104 0.580923
[10 rows x 3 columns]
In [22]: store.append('mi',df)
In [23]: store.select('mi')
Out[23]:
A
B
foo bar
foo one
2.052171 -1.230963
two
-1.713238 0.838912
three 0.215109 -1.515362
bar one
-0.447974 -1.573998
two
-0.071659 -1.277640
baz two
0.870302 1.275280
three 1.060780 1.673018
qux one
1.458210 -0.710542
two
1.557329 1.993441
three 0.150468 0.132104
C
-0.019240
-0.637855
1.586924
0.630925
-0.102206
-1.199212
1.249874
0.825392
-0.616293
0.580923
[10 rows x 3 columns]
# the levels are automatically included as data columns
In [24]: store.select('mi', Term('foo=bar'))
Out[24]:
A
B
C
foo bar
bar one -0.447974 -1.573998 0.630925
two -0.071659 -1.277640 -0.102206
[2 rows x 3 columns]
Multi-table creation via append_to_multiple and selection via select_as_multiple can create/select from
multiple tables and return a combined result, by using where on a selector table.
1.12. v0.10.1 (January 22, 2013)
153
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [25]: df_mt = DataFrame(randn(8, 6), index=date_range('1/1/2000', periods=8),
....:
columns=['A', 'B', 'C', 'D', 'E', 'F'])
....:
In [26]: df_mt['foo'] = 'bar'
# you can also create the tables individually
In [27]: store.append_to_multiple({ 'df1_mt' : ['A','B'], 'df2_mt' : None }, df_mt, selector = 'df1_m
In [28]: store
Out[28]:
File path: store.h5
/df
frame_table (typ->appendable,nrows->8,ncols->5,indexers->[index],dc->[B,C,strin
/df1_mt
frame_table (typ->appendable,nrows->8,ncols->2,indexers->[index],dc->[A,B])
/df2_mt
frame_table (typ->appendable,nrows->8,ncols->5,indexers->[index])
/df_mixed
frame_table (typ->appendable,nrows->8,ncols->6,indexers->[index])
/mi
frame_table (typ->appendable_multi,nrows->10,ncols->5,indexers->[index],dc->[ba
# indiviual tables were created
In [29]: store.select('df1_mt')
Out[29]:
A
B
2000-01-01 -0.128750 1.445964
2000-01-02 -0.688741 0.228006
2000-01-03 0.932498 -2.200069
2000-01-04 1.298390 1.662964
2000-01-05 -0.462446 -0.112019
2000-01-06 -1.626124 0.982041
2000-01-07 0.942864 2.502156
2000-01-08 0.268766 -1.225092
[8 rows x 2 columns]
In [30]: store.select('df2_mt')
Out[30]:
C
D
E
2000-01-01 -0.431163 0.016640 0.904578
2000-01-02 0.800353 -0.451572 0.831767
2000-01-03 1.239198 0.185437 -0.540770
2000-01-04 -0.040863 0.290110 -0.096145
2000-01-05 -0.134024 -0.205969 1.348944
2000-01-06 0.059493 -0.460111 -1.565401
2000-01-07 -0.302741 0.261551 -0.066342
2000-01-08 0.582752 -1.490764 -0.639757
F
-1.645852
0.228760
-0.370038
1.717830
-1.198246
-0.025706
0.897097
-0.952750
foo
bar
bar
bar
bar
bar
bar
bar
bar
[8 rows x 5 columns]
# as a multiple
In [31]: store.select_as_multiple(['df1_mt','df2_mt'], where = [ 'A>0','B>0' ], selector = 'df1_mt')
Out[31]:
A
B
C
D
E
F foo
2000-01-04 1.298390 1.662964 -0.040863 0.290110 -0.096145 1.717830 bar
2000-01-07 0.942864 2.502156 -0.302741 0.261551 -0.066342 0.897097 bar
[2 rows x 7 columns]
Enhancements
154
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• HDFStore now can read native PyTables table format tables
• You can pass nan_rep = ’my_nan_rep’ to append, to change the default nan representation on disk
(which converts to/from np.nan), this defaults to nan.
• You can pass index to append. This defaults to True. This will automagically create indicies on the
indexables and data columns of the table
• You can pass chunksize=an integer to append, to change the writing chunksize (default is 50000).
This will signficantly lower your memory usage on writing.
• You can pass expectedrows=an integer to the first append, to set the TOTAL number of expectedrows
that PyTables will expected. This will optimize read/write performance.
• Select now supports passing start and stop to provide selection space limiting in selection.
• Greatly improved ISO8601 (e.g., yyyy-mm-dd) date parsing for file parsers (GH2698)
• Allow DataFrame.merge to handle combinatorial sizes too large for 64-bit integer (GH2690)
• Series now has unary negation (-series) and inversion (~series) operators (GH2686)
• DataFrame.plot now includes a logx parameter to change the x-axis to log scale (GH2327)
• Series arithmetic operators can now handle constant and ndarray input (GH2574)
• ExcelFile now takes a kind argument to specify the file type (GH2613)
• A faster implementation for Series.str methods (GH2602)
Bug Fixes
• HDFStore tables can now store float32 types correctly (cannot be mixed with float64 however)
• Fixed Google Analytics prefix when specifying request segment (GH2713).
• Function to reset Google Analytics token store so users can recover from improperly setup client secrets
(GH2687).
• Fixed groupby bug resulting in segfault when passing in MultiIndex (GH2706)
• Fixed bug where passing a Series with datetime64 values into to_datetime results in bogus output values
(GH2699)
• Fixed bug in pattern in HDFStore expressions when pattern is not a valid regex (GH2694)
• Fixed performance issues while aggregating boolean data (GH2692)
• When given a boolean mask key and a Series of new values, Series __setitem__ will now align the incoming
values with the original Series (GH2686)
• Fixed MemoryError caused by performing counting sort on sorting MultiIndex levels with a very large number
of combinatorial values (GH2684)
• Fixed bug that causes plotting to fail when the index is a DatetimeIndex with a fixed-offset timezone (GH2683)
• Corrected businessday subtraction logic when the offset is more than 5 bdays and the starting date is on a
weekend (GH2680)
• Fixed C file parser behavior when the file has more columns than data (GH2668)
• Fixed file reader bug that misaligned columns with data in the presence of an implicit column and a specified
usecols value
• DataFrames with numerical or datetime indices are now sorted prior to plotting (GH2609)
• Fixed DataFrame.from_records error when passed columns, index, but empty records (GH2633)
1.12. v0.10.1 (January 22, 2013)
155
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Several bug fixed for Series operations when dtype is datetime64 (GH2689, GH2629, GH2626)
See the full release notes or issue tracker on GitHub for a complete list.
1.13 v0.10.0 (December 17, 2012)
This is a major release from 0.9.1 and includes many new features and enhancements along with a large number of
bug fixes. There are also a number of important API changes that long-time pandas users should pay close attention
to.
1.13.1 File parsing new features
The delimited file parsing engine (the guts of read_csv and read_table) has been rewritten from the ground up
and now uses a fraction the amount of memory while parsing, while being 40% or more faster in most use cases (in
some cases much faster).
There are also many new features:
• Much-improved Unicode handling via the encoding option.
• Column filtering (usecols)
• Dtype specification (dtype argument)
• Ability to specify strings to be recognized as True/False
• Ability to yield NumPy record arrays (as_recarray)
• High performance delim_whitespace option
• Decimal format (e.g. European format) specification
• Easier CSV dialect options: escapechar, lineterminator, quotechar, etc.
• More robust handling of many exceptional kinds of files observed in the wild
1.13.2 API changes
Deprecated DataFrame BINOP TimeSeries special case behavior
The default behavior of binary operations between a DataFrame and a Series has always been to align on the
DataFrame’s columns and broadcast down the rows, except in the special case that the DataFrame contains time
series. Since there are now method for each binary operator enabling you to specify how you want to broadcast, we
are phasing out this special case (Zen of Python: Special cases aren’t special enough to break the rules). Here’s what
I’m talking about:
In [1]: import pandas as pd
In [2]: df = pd.DataFrame(np.random.randn(6, 4),
...:
index=pd.date_range('1/1/2000', periods=6))
...:
In [3]: df
Out[3]:
0
2000-01-01 -0.892402
2000-01-02 0.586586
2000-01-03 0.408279
156
1
2
3
0.505987 -0.681624 0.850162
1.175843 -0.160391 0.481679
1.641246 0.383888 -1.495227
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-04 1.166096 -0.802272 -0.275253 0.517938
2000-01-05 -0.750872 1.216537 -0.910343 -0.606534
2000-01-06 -0.410659 0.264024 -0.069315 -1.814768
[6 rows x 4 columns]
# deprecated now
In [4]: df - df[0]
Out[4]:
0
1
2000-01-01 0 1.398389
2000-01-02 0 0.589256
2000-01-03 0 1.232968
2000-01-04 0 -1.968368
2000-01-05 0 1.967410
2000-01-06 0 0.674682
2
0.210778
-0.746978
-0.024391
-1.441350
-0.159471
0.341344
3
1.742564
-0.104908
-1.903505
-0.648158
0.144338
-1.404109
[6 rows x 4 columns]
# Change your code to
In [5]: df.sub(df[0], axis=0) # align on axis 0 (rows)
Out[5]:
0
1
2
3
2000-01-01 0 1.398389 0.210778 1.742564
2000-01-02 0 0.589256 -0.746978 -0.104908
2000-01-03 0 1.232968 -0.024391 -1.903505
2000-01-04 0 -1.968368 -1.441350 -0.648158
2000-01-05 0 1.967410 -0.159471 0.144338
2000-01-06 0 0.674682 0.341344 -1.404109
[6 rows x 4 columns]
You will get a deprecation warning in the 0.10.x series, and the deprecated functionality will be removed in 0.11 or
later.
Altered resample default behavior
The default time series resample binning behavior of daily D and higher frequencies has been changed to
closed=’left’, label=’left’. Lower nfrequencies are unaffected. The prior defaults were causing a great
deal of confusion for users, especially resampling data to daily frequency (which labeled the aggregated group with
the end of the interval: the next day).
Note:
In [6]: dates = pd.date_range('1/1/2000', '1/5/2000', freq='4h')
In [7]: series = Series(np.arange(len(dates)), index=dates)
In [8]: series
Out[8]:
2000-01-01 00:00:00
2000-01-01 04:00:00
2000-01-01 08:00:00
2000-01-01 12:00:00
2000-01-01 16:00:00
2000-01-01 20:00:00
2000-01-02 00:00:00
2000-01-04 00:00:00
0
1
2
3
4
5
6
..
18
1.13. v0.10.0 (December 17, 2012)
157
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-04 04:00:00
2000-01-04 08:00:00
2000-01-04 12:00:00
2000-01-04 16:00:00
2000-01-04 20:00:00
2000-01-05 00:00:00
Freq: 4H, dtype: int32
19
20
21
22
23
24
In [9]: series.resample('D', how='sum')
Out[9]:
2000-01-01
15
2000-01-02
51
2000-01-03
87
2000-01-04
123
2000-01-05
24
Freq: D, dtype: int32
# old behavior
In [10]: series.resample('D', how='sum', closed='right', label='right')
Out[10]:
2000-01-01
0
2000-01-02
21
2000-01-03
57
2000-01-04
93
2000-01-05
129
Freq: D, dtype: int32
• Infinity and negative infinity are no longer treated as NA by isnull and notnull. That they every were was
a relic of early pandas. This behavior can be re-enabled globally by the mode.use_inf_as_null option:
In [11]: s = pd.Series([1.5, np.inf, 3.4, -np.inf])
In [12]: pd.isnull(s)
Out[12]:
0
False
1
False
2
False
3
False
dtype: bool
In [13]: s.fillna(0)
Out[13]:
0
1.500000
1
inf
2
3.400000
3
-inf
dtype: float64
In [14]: pd.set_option('use_inf_as_null', True)
In [15]: pd.isnull(s)
Out[15]:
0
False
1
True
2
False
3
True
dtype: bool
158
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [16]: s.fillna(0)
Out[16]:
0
1.5
1
0.0
2
3.4
3
0.0
dtype: float64
In [17]: pd.reset_option('use_inf_as_null')
• Methods with the inplace option now all return None instead of the calling object. E.g. code written like
df = df.fillna(0, inplace=True) may stop working. To fix, simply delete the unnecessary variable
assignment.
• pandas.merge no longer sorts the group keys (sort=False) by default. This was done for performance
reasons: the group-key sorting is often one of the more expensive parts of the computation and is often unnecessary.
• The default column names for a file with no header have been changed to the integers 0 through N - 1. This
is to create consistency with the DataFrame constructor with no columns specified. The v0.9.0 behavior (names
X0, X1, ...) can be reproduced by specifying prefix=’X’:
In [18]: data= 'a,b,c\n1,Yes,2\n3,No,4'
In [19]: print(data)
a,b,c
1,Yes,2
3,No,4
In [20]: pd.read_csv(StringIO(data), header=None)
Out[20]:
0
1 2
0 a
b c
1 1 Yes 2
2 3
No 4
[3 rows x 3 columns]
In [21]: pd.read_csv(StringIO(data), header=None, prefix='X')
Out[21]:
X0
X1 X2
0 a
b c
1 1 Yes 2
2 3
No 4
[3 rows x 3 columns]
• Values like ’Yes’ and ’No’ are not interpreted as boolean by default, though this can be controlled by new
true_values and false_values arguments:
In [22]: print(data)
a,b,c
1,Yes,2
3,No,4
In [23]: pd.read_csv(StringIO(data))
Out[23]:
a
b c
0 1 Yes 2
1.13. v0.10.0 (December 17, 2012)
159
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
3
No
4
[2 rows x 3 columns]
In [24]: pd.read_csv(StringIO(data), true_values=['Yes'], false_values=['No'])
Out[24]:
a
b c
0 1
True 2
1 3 False 4
[2 rows x 3 columns]
• The file parsers will not recognize non-string values arising from a converter function as NA if passed in the
na_values argument. It’s better to do post-processing using the replace function instead.
• Calling fillna on Series or DataFrame with no arguments is no longer valid code. You must either specify a
fill value or an interpolation method:
In [25]: s = Series([np.nan, 1., 2., np.nan, 4])
In [26]: s
Out[26]:
0
NaN
1
1
2
2
3
NaN
4
4
dtype: float64
In [27]: s.fillna(0)
Out[27]:
0
0
1
1
2
2
3
0
4
4
dtype: float64
In [28]: s.fillna(method='pad')
Out[28]:
0
NaN
1
1
2
2
3
2
4
4
dtype: float64
Convenience methods ffill and bfill have been added:
In [29]: s.ffill()
Out[29]:
0
NaN
1
1
2
2
3
2
4
4
dtype: float64
• Series.apply will now operate on a returned value from the applied function, that is itself a series, and
160
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
possibly upcast the result to a DataFrame
In [30]: def f(x):
....:
return Series([ x, x**2 ], index = ['x', 'x^2'])
....:
In [31]: s = Series(np.random.rand(5))
In [32]: s
Out[32]:
0
0.013135
1
0.909855
2
0.098093
3
0.023540
4
0.141354
dtype: float64
In [33]: s.apply(f)
Out[33]:
x
x^2
0 0.013135 0.000173
1 0.909855 0.827836
2 0.098093 0.009622
3 0.023540 0.000554
4 0.141354 0.019981
[5 rows x 2 columns]
• New API functions for working with pandas options (GH2097):
– get_option / set_option - get/set the value of an option. Partial names are accepted. reset_option - reset one or more options to their default value. Partial names are accepted. describe_option - print a description of one or more options. When called with no arguments.
print all registered options.
Note: set_printoptions/ reset_printoptions are now deprecated (but functioning), the print options now live under “display.XYZ”. For example:
In [34]: get_option("display.max_rows")
Out[34]: 15
• to_string() methods now always return unicode strings (GH2224).
1.13.3 New features
1.13.4 Wide DataFrame Printing
Instead of printing the summary information, pandas now splits the string representation across multiple rows by
default:
In [35]: wide_frame = DataFrame(randn(5, 16))
In [36]: wide_frame
Out[36]:
0
1
2
3
4
0 2.520045 1.570114 -0.360875 -0.880096 0.235532
1 0.422194 0.288403 -0.487393 -0.777639 0.055865
2 0.585174 -0.568825 -0.719412 1.191340 -0.456362
1.13. v0.10.0 (December 17, 2012)
5
6
0.207232 -1.983857
1.383381 0.085638
0.089931 0.776079
\
161
pandas: powerful Python data analysis toolkit, Release 0.16.1
3 1.218080 -0.564705 -0.581790 0.286071 0.048725
4 -0.376280 0.511936 -0.116412 -0.625256 -0.550627
1.002440 1.276582
1.261433 -0.552429
7
8
9
10
11
12
13
0 -1.702547 -1.621234 -0.906840 1.014601 -0.475108 -0.358944 1.262942
1 0.246392 0.965887 0.246354 -0.727728 -0.094414 -0.276854 0.158399
2 0.752889 -1.195795 -1.425911 -0.548829 0.774225 0.740501 1.510263
3 0.054399 0.241963 -0.471786 0.314510 -0.059986 -2.069319 -1.115104
4 1.695803 -1.025917 -0.910942 0.426805 -0.131749 0.432600 0.044671
0
1
2
3
4
\
14
15
-0.412451 -0.462580
-0.277255 1.331263
-1.642511 0.432560
-0.369325 -1.502617
-0.341265 1.844536
[5 rows x 16 columns]
The old behavior of printing out summary information can be achieved via the ‘expand_frame_repr’ print option:
In [37]: pd.set_option('expand_frame_repr', False)
In [38]: wide_frame
Out[38]:
0
1
0 2.520045 1.570114
1 0.422194 0.288403
2 0.585174 -0.568825
3 1.218080 -0.564705
4 -0.376280 0.511936
2
3
4
-0.360875 -0.880096 0.235532
-0.487393 -0.777639 0.055865
-0.719412 1.191340 -0.456362
-0.581790 0.286071 0.048725
-0.116412 -0.625256 -0.550627
5
6
7
8
9
0.207232 -1.983857 -1.702547 -1.621234 -0.906840
1.383381 0.085638 0.246392 0.965887 0.246354
0.089931 0.776079 0.752889 -1.195795 -1.425911
1.002440 1.276582 0.054399 0.241963 -0.471786
1.261433 -0.552429 1.695803 -1.025917 -0.910942
[5 rows x 16 columns]
The width of each line can be changed via ‘line_width’ (80 by default):
In [39]: pd.set_option('line_width', 40)
line_width has been deprecated, use display.width instead (currently both are
identical)
In [40]: wide_frame
Out[40]:
0
1
0 2.520045 1.570114
1 0.422194 0.288403
2 0.585174 -0.568825
3 1.218080 -0.564705
4 -0.376280 0.511936
2
-0.360875
-0.487393
-0.719412
-0.581790
-0.116412
\
3
4
0 -0.880096 0.235532
1 -0.777639 0.055865
2 1.191340 -0.456362
3 0.286071 0.048725
4 -0.625256 -0.550627
5
0.207232
1.383381
0.089931
1.002440
1.261433
\
6
7
8
0 -1.983857 -1.702547 -1.621234
\
162
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1 0.085638
2 0.776079
3 1.276582
4 -0.552429
0.246392 0.965887
0.752889 -1.195795
0.054399 0.241963
1.695803 -1.025917
9
10
11
-0.906840 1.014601 -0.475108
0.246354 -0.727728 -0.094414
-1.425911 -0.548829 0.774225
-0.471786 0.314510 -0.059986
-0.910942 0.426805 -0.131749
\
12
13
14
0 -0.358944 1.262942 -0.412451
1 -0.276854 0.158399 -0.277255
2 0.740501 1.510263 -1.642511
3 -2.069319 -1.115104 -0.369325
4 0.432600 0.044671 -0.341265
\
0
1
2
3
4
15
0 -0.462580
1 1.331263
2 0.432560
3 -1.502617
4 1.844536
[5 rows x 16 columns]
1.13.5 Updated PyTables Support
Docs for PyTables Table format & several enhancements to the api. Here is a taste of what to expect.
In [41]: store = HDFStore('store.h5')
In [42]: df = DataFrame(randn(8, 3), index=date_range('1/1/2000', periods=8),
....:
columns=['A', 'B', 'C'])
....:
In [43]: df
Out[43]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
B
C
-2.036047 0.000830 -0.955697
-0.898872 -0.725411 0.059904
-0.449644 1.082900 -1.221265
0.361078 1.330704 0.855932
-1.216718 1.488887 0.018993
-0.877046 0.045976 0.437274
-0.567182 -0.888657 -0.556383
0.655457 1.117949 -2.782376
[8 rows x 3 columns]
# appending data frames
In [44]: df1 = df[0:4]
In [45]: df2 = df[4:]
1.13. v0.10.0 (December 17, 2012)
163
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [46]: store.append('df', df1)
In [47]: store.append('df', df2)
In [48]: store
Out[48]:
File path: store.h5
/df
frame_table (typ->appendable,nrows->8,ncols->3,indexers->[index])
# selecting the entire store
In [49]: store.select('df')
Out[49]:
A
B
2000-01-01 -2.036047 0.000830
2000-01-02 -0.898872 -0.725411
2000-01-03 -0.449644 1.082900
2000-01-04 0.361078 1.330704
2000-01-05 -1.216718 1.488887
2000-01-06 -0.877046 0.045976
2000-01-07 -0.567182 -0.888657
2000-01-08 0.655457 1.117949
C
-0.955697
0.059904
-1.221265
0.855932
0.018993
0.437274
-0.556383
-2.782376
[8 rows x 3 columns]
In [50]: wp = Panel(randn(2, 5, 4), items=['Item1', 'Item2'],
....:
major_axis=date_range('1/1/2000', periods=5),
....:
minor_axis=['A', 'B', 'C', 'D'])
....:
In [51]: wp
Out[51]:
Dimensions: 2 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to D
# storing a panel
In [52]: store.append('wp',wp)
# selecting via A QUERY
In [53]: store.select('wp',
....:
[ Term('major_axis>20000102'), Term('minor_axis', '=', ['A','B']) ])
....:
Out[53]:
Dimensions: 2 (items) x 3 (major_axis) x 2 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to B
# removing data from tables
In [54]: store.remove('wp', Term('major_axis>20000103'))
Out[54]: 8
In [55]: store.select('wp')
Out[55]:
164
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-03 00:00:00
Minor_axis axis: A to D
# deleting a store
In [56]: del store['df']
In [57]: store
Out[57]:
File path: store.h5
/wp
wide_table
(typ->appendable,nrows->12,ncols->2,indexers->[major_axis,minor_axis])
Enhancements
• added ability to hierarchical keys
In [58]: store.put('foo/bar/bah', df)
In [59]: store.append('food/orange', df)
In [60]: store.append('food/apple',
df)
In [61]: store
Out[61]:
File path: store.h5
/foo/bar/bah
frame
(shape->[8,3])
/food/apple
frame_table (typ->appendable,nrows->8,ncols->3,indexers->[index])
/food/orange
frame_table (typ->appendable,nrows->8,ncols->3,indexers->[index])
/wp
wide_table
(typ->appendable,nrows->12,ncols->2,indexers->[major_ax
# remove all nodes under this level
In [62]: store.remove('food')
In [63]: store
Out[63]:
File path: store.h5
/foo/bar/bah
frame
(shape->[8,3])
/wp
wide_table
(typ->appendable,nrows->12,ncols->2,indexers->[major_ax
• added mixed-dtype support!
In [64]: df['string'] = 'string'
In [65]: df['int']
= 1
In [66]: store.append('df',df)
In [67]: df1 = store.select('df')
In [68]: df1
Out[68]:
A
B
C
2000-01-01 -2.036047 0.000830 -0.955697
2000-01-02 -0.898872 -0.725411 0.059904
1.13. v0.10.0 (December 17, 2012)
string
string
string
int
1
1
165
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
-0.449644 1.082900 -1.221265
0.361078 1.330704 0.855932
-1.216718 1.488887 0.018993
-0.877046 0.045976 0.437274
-0.567182 -0.888657 -0.556383
0.655457 1.117949 -2.782376
string
string
string
string
string
string
1
1
1
1
1
1
[8 rows x 5 columns]
In [69]: df1.get_dtype_counts()
Out[69]:
float64
3
int64
1
object
1
dtype: int64
• performance improvments on table writing
• support for arbitrarily indexed dimensions
• SparseSeries now has a density property (GH2384)
• enable Series.str.strip/lstrip/rstrip methods to take an input argument to strip arbitrary characters (GH2411)
• implement value_vars in melt to limit values to certain columns and add melt to pandas namespace
(GH2412)
Bug Fixes
• added Term method of specifying where conditions (GH1996).
• del store[’df’] now call store.remove(’df’) for store deletion
• deleting of consecutive rows is much faster than before
• min_itemsize parameter can be specified in table creation to force a minimum size for indexing columns
(the previous implementation would set the column size based on the first append)
• indexing support via create_table_index (requires PyTables >= 2.3) (GH698).
• appending on a store would fail if the table was not first created via put
• fixed issue with missing attributes after loading a pickled dataframe (GH2431)
• minor change to select and remove: require a table ONLY if where is also provided (and not None)
Compatibility
0.10 of HDFStore is backwards compatible for reading tables created in a prior version of pandas, however, query
terms using the prior (undocumented) methodology are unsupported. You must read in the entire file and write it out
using the new format to take advantage of the updates.
1.13.6 N Dimensional Panels (Experimental)
Adding experimental support for Panel4D and factory functions to create n-dimensional named panels. Docs for
NDim. Here is a taste of what to expect.
In [70]: p4d = Panel4D(randn(2, 2, 5, 4),
....:
labels=['Label1','Label2'],
....:
items=['Item1', 'Item2'],
....:
major_axis=date_range('1/1/2000', periods=5),
166
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
....:
....:
minor_axis=['A', 'B', 'C', 'D'])
In [71]: p4d
Out[71]:
Dimensions: 2 (labels) x 2 (items) x 5 (major_axis) x 4 (minor_axis)
Labels axis: Label1 to Label2
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to D
See the full release notes or issue tracker on GitHub for a complete list.
1.14 v0.9.1 (November 14, 2012)
This is a bugfix release from 0.9.0 and includes several new features and enhancements along with a large number of
bug fixes. The new features include by-column sort order for DataFrame and Series, improved NA handling for the
rank method, masking functions for DataFrame, and intraday time-series filtering for DataFrame.
1.14.1 New features
• Series.sort, DataFrame.sort, and DataFrame.sort_index can now be specified in a per-column manner to support
multiple sort orders (GH928)
In [1]: df = DataFrame(np.random.randint(0, 2, (6, 3)), columns=['A', 'B', 'C'])
In [2]: df.sort(['A', 'B'], ascending=[1, 0])
Out[2]:
A B C
2 0 1 1
3 0 1 1
4 0 0 1
0 1 1 0
1 1 0 1
5 1 0 1
[6 rows x 3 columns]
• DataFrame.rank now supports additional argument values for the na_option parameter so missing values can
be assigned either the largest or the smallest rank (GH1508, GH2159)
In [3]: df = DataFrame(np.random.randn(6, 3), columns=['A', 'B', 'C'])
In [4]: df.ix[2:4] = np.nan
In [5]: df.rank()
Out[5]:
A
B
C
0
3
2
1
1
2
1
3
2 NaN NaN NaN
3 NaN NaN NaN
4 NaN NaN NaN
5
1
3
2
1.14. v0.9.1 (November 14, 2012)
167
pandas: powerful Python data analysis toolkit, Release 0.16.1
[6 rows x 3 columns]
In [6]: df.rank(na_option='top')
Out[6]:
A B C
0 6 5 4
1 5 4 6
2 2 2 2
3 2 2 2
4 2 2 2
5 4 6 5
[6 rows x 3 columns]
In [7]: df.rank(na_option='bottom')
Out[7]:
A B C
0 3 2 1
1 2 1 3
2 5 5 5
3 5 5 5
4 5 5 5
5 1 3 2
[6 rows x 3 columns]
• DataFrame has new where and mask methods to select values according to a given boolean mask (GH2109,
GH2151)
DataFrame currently supports slicing via a boolean vector the same length as the DataFrame (inside
the []). The returned DataFrame has the same number of columns as the original, but is sliced on its
index.
In [8]: df = DataFrame(np.random.randn(5, 3), columns = ['A','B','C'])
In [9]: df
Out[9]:
A
B
C
0 0.706220 -1.130744 -0.690308
1 -0.885387 0.246004 1.986687
2 0.212595 -1.189832 -0.344258
3 0.816335 -1.514102 1.298184
4 0.089527 0.576687 -0.737750
[5 rows x 3 columns]
In [10]: df[df['A'] >
Out[10]:
A
B
0 0.706220 -1.130744
2 0.212595 -1.189832
3 0.816335 -1.514102
4 0.089527 0.576687
0]
C
-0.690308
-0.344258
1.298184
-0.737750
[4 rows x 3 columns]
If a DataFrame is sliced with a DataFrame based boolean condition (with the same size as the original
DataFrame), then a DataFrame the same size (index and columns) as the original is returned, with
168
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
elements that do not meet the boolean condition as NaN. This is accomplished via the new method
DataFrame.where. In addition, where takes an optional other argument for replacement.
In [11]: df[df>0]
Out[11]:
A
B
0 0.706220
NaN
1
NaN 0.246004
2 0.212595
NaN
3 0.816335
NaN
4 0.089527 0.576687
C
NaN
1.986687
NaN
1.298184
NaN
[5 rows x 3 columns]
In [12]: df.where(df>0)
Out[12]:
A
B
C
0 0.706220
NaN
NaN
1
NaN 0.246004 1.986687
2 0.212595
NaN
NaN
3 0.816335
NaN 1.298184
4 0.089527 0.576687
NaN
[5 rows x 3 columns]
In [13]: df.where(df>0,-df)
Out[13]:
A
B
C
0 0.706220 1.130744 0.690308
1 0.885387 0.246004 1.986687
2 0.212595 1.189832 0.344258
3 0.816335 1.514102 1.298184
4 0.089527 0.576687 0.737750
[5 rows x 3 columns]
Furthermore, where now aligns the input boolean condition (ndarray or DataFrame), such that partial
selection with setting is possible. This is analagous to partial setting via .ix (but on the contents rather
than the axis labels)
In [14]: df2 = df.copy()
In [15]: df2[ df2[1:4] > 0 ] = 3
In [16]: df2
Out[16]:
A
B
C
0 0.706220 -1.130744 -0.690308
1 -0.885387 3.000000 3.000000
2 3.000000 -1.189832 -0.344258
3 3.000000 -1.514102 3.000000
4 0.089527 0.576687 -0.737750
[5 rows x 3 columns]
DataFrame.mask is the inverse boolean operation of where.
In [17]: df.mask(df<=0)
Out[17]:
1.14. v0.9.1 (November 14, 2012)
169
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1
2
3
4
A
0.706220
NaN
0.212595
0.816335
0.089527
B
NaN
0.246004
NaN
NaN
0.576687
C
NaN
1.986687
NaN
1.298184
NaN
[5 rows x 3 columns]
• Enable referencing of Excel columns by their column names (GH1936)
In [18]: xl = ExcelFile('data/test.xls')
In [19]: xl.parse('Sheet1', index_col=0, parse_dates=True,
....:
parse_cols='A:D')
....:
Out[19]:
A
B
C
2000-01-03 0.980269 3.685731 -0.364217
2000-01-04 1.047916 -0.041232 -0.161812
2000-01-05 0.498581 0.731168 -0.537677
2000-01-06 1.120202 1.567621 0.003641
2000-01-07 -0.487094 0.571455 -1.611639
2000-01-10 0.836649 0.246462 0.588543
2000-01-11 -0.157161 1.340307 1.195778
[7 rows x 3 columns]
• Added option to disable pandas-style tick locators and formatters using series.plot(x_compat=True) or pandas.plot_params[’x_compat’] = True (GH2205)
• Existing TimeSeries methods at_time and between_time were added to DataFrame (GH2149)
• DataFrame.dot can now accept ndarrays (GH2042)
• DataFrame.drop now supports non-unique indexes (GH2101)
• Panel.shift now supports negative periods (GH2164)
• DataFrame now support unary ~ operator (GH2110)
1.14.2 API changes
• Upsampling data with a PeriodIndex will result in a higher frequency TimeSeries that spans the original time
window
In [20]: prng = period_range('2012Q1', periods=2, freq='Q')
In [21]: s = Series(np.random.randn(len(prng)), prng)
In [22]: s.resample('M')
Out[22]:
2012-01
0.194513
2012-02
NaN
2012-03
NaN
2012-04
-0.854246
2012-05
NaN
2012-06
NaN
Freq: M, dtype: float64
170
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Period.end_time now returns the last nanosecond in the time interval (GH2124, GH2125, GH1764)
In [23]: p = Period('2012')
In [24]: p.end_time
Out[24]: Timestamp('2012-12-31 23:59:59.999999999')
• File parsers no longer coerce to float or bool for columns that have custom converters specified (GH2184)
In [25]: data = 'A,B,C\n00001,001,5\n00002,002,6'
In [26]: read_csv(StringIO(data), converters={'A' : lambda x: x.strip()})
Out[26]:
A B C
0 00001 1 5
1 00002 2 6
[2 rows x 3 columns]
See the full release notes or issue tracker on GitHub for a complete list.
1.15 v0.9.0 (October 7, 2012)
This is a major release from 0.8.1 and includes several new features and enhancements along with a large number of
bug fixes. New features include vectorized unicode encoding/decoding for Series.str, to_latex method to DataFrame,
more flexible parsing of boolean values, and enabling the download of options data from Yahoo! Finance.
1.15.1 New features
• Add encode and decode for unicode handling to vectorized string processing methods in Series.str (GH1706)
• Add DataFrame.to_latex method (GH1735)
• Add convenient expanding window equivalents of all rolling_* ops (GH1785)
• Add Options class to pandas.io.data for fetching options data from Yahoo! Finance (GH1748, GH1739)
• More flexible parsing of boolean values (Yes, No, TRUE, FALSE, etc) (GH1691, GH1295)
• Add level parameter to Series.reset_index
• TimeSeries.between_time can now select times across midnight (GH1871)
• Series constructor can now handle generator as input (GH1679)
• DataFrame.dropna can now take multiple axes (tuple/list) as input (GH924)
• Enable skip_footer parameter in ExcelFile.parse (GH1843)
1.15.2 API changes
• The default column names when header=None and no columns names passed to functions like read_csv
has changed to be more Pythonic and amenable to attribute access:
In [1]: data = '0,0,1\n1,1,0\n0,1,0'
In [2]: df = read_csv(StringIO(data), header=None)
1.15. v0.9.0 (October 7, 2012)
171
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [3]: df
Out[3]:
0 1 2
0 0 0 1
1 1 1 0
2 0 1 0
[3 rows x 3 columns]
• Creating a Series from another Series, passing an index, will cause reindexing to happen inside rather than treating the Series like an ndarray. Technically improper usages like Series(df[col1], index=df[col2])
that worked before “by accident” (this was never intended) will lead to all NA Series in some cases. To be perfectly clear:
In [4]: s1 = Series([1, 2, 3])
In [5]: s1
Out[5]:
0
1
1
2
2
3
dtype: int64
In [6]: s2 = Series(s1, index=['foo', 'bar', 'baz'])
In [7]: s2
Out[7]:
foo
NaN
bar
NaN
baz
NaN
dtype: float64
• Deprecated day_of_year API removed from PeriodIndex, use dayofyear (GH1723)
• Don’t modify NumPy suppress printoption to True at import time
• The internal HDF5 data arrangement for DataFrames has been transposed. Legacy files will still be readable by
HDFStore (GH1834, GH1824)
• Legacy cruft removed: pandas.stats.misc.quantileTS
• Use ISO8601 format for Period repr: monthly, daily, and on down (GH1776)
• Empty DataFrame columns are now created as object dtype. This will prevent a class of TypeErrors that was
occurring in code where the dtype of a column would depend on the presence of data or not (e.g. a SQL query
having results) (GH1783)
• Setting parts of DataFrame/Panel using ix now aligns input Series/DataFrame (GH1630)
• first and last methods in GroupBy no longer drop non-numeric columns (GH1809)
• Resolved inconsistencies in specifying custom NA values in text parser. na_values of type dict no longer
override default NAs unless keep_default_na is set to false explicitly (GH1657)
• DataFrame.dot will not do data alignment, and also work with Series (GH1915)
See the full release notes or issue tracker on GitHub for a complete list.
172
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.16 v0.8.1 (July 22, 2012)
This release includes a few new features, performance enhancements, and over 30 bug fixes from 0.8.0. New features
include notably NA friendly string processing functionality and a series of new plot types and options.
1.16.1 New features
• Add vectorized string processing methods accessible via Series.str (GH620)
• Add option to disable adjustment in EWMA (GH1584)
• Radviz plot (GH1566)
• Parallel coordinates plot
• Bootstrap plot
• Per column styles and secondary y-axis plotting (GH1559)
• New datetime converters millisecond plotting (GH1599)
• Add option to disable “sparse” display of hierarchical indexes (GH1538)
• Series/DataFrame’s set_index method can append levels to an existing Index/MultiIndex (GH1569,
GH1577)
1.16.2 Performance improvements
• Improved implementation of rolling min and max (thanks to Bottleneck !)
• Add accelerated ’median’ GroupBy option (GH1358)
• Significantly improve the performance of parsing ISO8601-format date strings with DatetimeIndex or
to_datetime (GH1571)
• Improve the performance of GroupBy on single-key aggregations and use with Categorical types
• Significant datetime parsing performance improvments
1.17 v0.8.0 (June 29, 2012)
This is a major release from 0.7.3 and includes extensive work on the time series handling and processing infrastructure
as well as a great deal of new functionality throughout the library. It includes over 700 commits from more than 20
distinct authors. Most pandas 0.7.3 and earlier users should not experience any issues upgrading, but due to the
migration to the NumPy datetime64 dtype, there may be a number of bugs and incompatibilities lurking. Lingering
incompatibilities will be fixed ASAP in a 0.8.1 release if necessary. See the full release notes or issue tracker on
GitHub for a complete list.
1.17.1 Support for non-unique indexes
All objects can now work with non-unique indexes. Data alignment / join operations work according to SQL join
semantics (including, if application, index duplication in many-to-many joins)
1.16. v0.8.1 (July 22, 2012)
173
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.17.2 NumPy datetime64 dtype and 1.6 dependency
Time series data are now represented using NumPy’s datetime64 dtype; thus, pandas 0.8.0 now requires at least NumPy
1.6. It has been tested and verified to work with the development version (1.7+) of NumPy as well which includes some
significant user-facing API changes. NumPy 1.6 also has a number of bugs having to do with nanosecond resolution
data, so I recommend that you steer clear of NumPy 1.6’s datetime64 API functions (though limited as they are) and
only interact with this data using the interface that pandas provides.
See the end of the 0.8.0 section for a “porting” guide listing potential issues for users migrating legacy codebases from
pandas 0.7 or earlier to 0.8.0.
Bug fixes to the 0.7.x series for legacy NumPy < 1.6 users will be provided as they arise. There will be no more further
development in 0.7.x beyond bug fixes.
1.17.3 Time series changes and improvements
Note: With this release, legacy scikits.timeseries users should be able to port their code to use pandas.
Note: See documentation for overview of pandas timeseries API.
• New datetime64 representation speeds up join operations and data alignment, reduces memory usage, and
improve serialization / deserialization performance significantly over datetime.datetime
• High performance and flexible resample method for converting from high-to-low and low-to-high frequency.
Supports interpolation, user-defined aggregation functions, and control over how the intervals and result labeling
are defined. A suite of high performance Cython/C-based resampling functions (including Open-High-LowClose) have also been implemented.
• Revamp of frequency aliases and support for frequency shortcuts like ‘15min’, or ‘1h30min’
• New DatetimeIndex class supports both fixed frequency and irregular time series. Replaces now deprecated
DateRange class
• New PeriodIndex and Period classes for representing time spans and performing calendar logic, including the 12 fiscal quarterly frequencies . This is a partial port of, and a substantial
enhancement to, elements of the scikits.timeseries codebase. Support for conversion between PeriodIndex and
DatetimeIndex
• New Timestamp data type subclasses datetime.datetime, providing the same interface while enabling working
with nanosecond-resolution data. Also provides easy time zone conversions.
• Enhanced support for time zones. Add tz_convert and tz_lcoalize methods to TimeSeries and DataFrame.
All timestamps are stored as UTC; Timestamps from DatetimeIndex objects with time zone set will be localized
to localtime. Time zone conversions are therefore essentially free. User needs to know very little about pytz
library now; only time zone names as as strings are required. Time zone-aware timestamps are equal if and only
if their UTC timestamps match. Operations between time zone-aware time series with different time zones will
result in a UTC-indexed time series.
• Time series string indexing conveniences / shortcuts: slice years, year and month, and index values with strings
• Enhanced time series plotting; adaptation of scikits.timeseries matplotlib-based plotting code
• New date_range, bdate_range, and period_range factory functions
• Robust frequency inference function infer_freq and inferred_freq property of DatetimeIndex, with option
to infer frequency on construction of DatetimeIndex
174
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• to_datetime function efficiently parses array of strings to DatetimeIndex. DatetimeIndex will parse array or
list of strings to datetime64
• Optimized support for datetime64-dtype data in Series and DataFrame columns
• New NaT (Not-a-Time) type to represent NA in timestamp arrays
• Optimize Series.asof for looking up “as of” values for arrays of timestamps
• Milli, Micro, Nano date offset objects
• Can index time series with datetime.time objects to select all data at particular time of day
(TimeSeries.at_time) or between two times (TimeSeries.between_time)
• Add tshift method for leading/lagging using the frequency (if any) of the index, as opposed to a naive lead/lag
using shift
1.17.4 Other new features
• New cut and qcut functions (like R’s cut function) for computing a categorical variable from a continuous
variable by binning values either into value-based (cut) or quantile-based (qcut) bins
• Rename Factor to Categorical and add a number of usability features
• Add limit argument to fillna/reindex
• More flexible multiple function application in GroupBy, and can pass list (name, function) tuples to get result in
particular order with given names
• Add flexible replace method for efficiently substituting values
• Enhanced read_csv/read_table for reading time series data and converting multiple columns to dates
• Add comments option to parser functions: read_csv, etc.
• Add :ref‘dayfirst ‘ option to parser functions for parsing international DD/MM/YYYY dates
• Allow the user to specify the CSV reader dialect to control quoting etc.
• Handling thousands separators in read_csv to improve integer parsing.
• Enable unstacking of multiple levels in one shot. Alleviate pivot_table bugs (empty columns being introduced)
• Move to klib-based hash tables for indexing; better performance and less memory usage than Python’s dict
• Add first, last, min, max, and prod optimized GroupBy functions
• New ordered_merge function
• Add flexible comparison instance methods eq, ne, lt, gt, etc. to DataFrame, Series
• Improve scatter_matrix plotting function and add histogram or kernel density estimates to diagonal
• Add ‘kde’ plot option for density plots
• Support for converting DataFrame to R data.frame through rpy2
• Improved support for complex numbers in Series and DataFrame
• Add pct_change method to all data structures
• Add max_colwidth configuration option for DataFrame console output
• Interpolate Series values using index values
• Can select multiple columns from GroupBy
1.17. v0.8.0 (June 29, 2012)
175
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Add update methods to Series/DataFrame for updating values in place
• Add any and all method to DataFrame
1.17.5 New plotting methods
Series.plot now supports a secondary_y option:
In [1]: plt.figure()
Out[1]:
In [2]: fx['FR'].plot(style='g')
Out[2]:
In [3]: fx['IT'].plot(style='k--', secondary_y=True)
Out[3]:
../_static/whatsnew_secondary_y.png
Vytautas Jancauskas, the 2012 GSOC participant, has added many new plot types. For example, ’kde’ is a new
option:
In [4]: s = Series(np.concatenate((np.random.randn(1000),
...:
np.random.randn(1000) * 0.5 + 3)))
...:
In [5]: plt.figure()
Out[5]:
In [6]: s.hist(normed=True, alpha=0.2)
Out[6]:
In [7]: s.plot(kind='kde')
Out[7]:
../_static/whatsnew_kde.png
See the plotting page for much more.
1.17.6 Other API changes
• Deprecation of offset, time_rule, and timeRule arguments names in time series functions. Warnings
will be printed until pandas 0.9 or 1.0.
176
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.17.7 Potential porting issues for pandas <= 0.7.3 users
The major change that may affect you in pandas 0.8.0 is that time series indexes use NumPy’s datetime64 data
type instead of dtype=object arrays of Python’s built-in datetime.datetime objects. DateRange has been
replaced by DatetimeIndex but otherwise behaved identically. But, if you have code that converts DateRange
or Index objects that used to contain datetime.datetime values to plain NumPy arrays, you may have bugs
lurking with code using scalar values because you are handing control over to NumPy:
In [8]: import datetime
In [9]: rng = date_range('1/1/2000', periods=10)
In [10]: rng[5]
Out[10]: Timestamp('2000-01-06 00:00:00', offset='D')
In [11]: isinstance(rng[5], datetime.datetime)
Out[11]: True
In [12]: rng_asarray = np.asarray(rng)
In [13]: scalar_val = rng_asarray[5]
In [14]: type(scalar_val)
Out[14]: numpy.datetime64
pandas’s Timestamp object is a subclass of datetime.datetime that has nanosecond support (the
nanosecond field store the nanosecond value between 0 and 999). It should substitute directly into any code that
used datetime.datetime values before. Thus, I recommend not casting DatetimeIndex to regular NumPy
arrays.
If you have code that requires an array of datetime.datetime objects, you have a couple of options. First, the
asobject property of DatetimeIndex produces an array of Timestamp objects:
In [15]: stamp_array = rng.asobject
In [16]: stamp_array
Out[16]:
Index([2000-01-01 00:00:00, 2000-01-02 00:00:00, 2000-01-03 00:00:00,
2000-01-04 00:00:00, 2000-01-05 00:00:00, 2000-01-06 00:00:00,
2000-01-07 00:00:00, 2000-01-08 00:00:00, 2000-01-09 00:00:00,
2000-01-10 00:00:00],
dtype='object')
In [17]: stamp_array[5]
Out[17]: Timestamp('2000-01-06 00:00:00', offset='D')
To get an array of proper datetime.datetime objects, use the to_pydatetime method:
In [18]: dt_array = rng.to_pydatetime()
In [19]: dt_array
Out[19]:
array([datetime.datetime(2000,
datetime.datetime(2000,
datetime.datetime(2000,
datetime.datetime(2000,
datetime.datetime(2000,
datetime.datetime(2000,
datetime.datetime(2000,
1.17. v0.8.0 (June 29, 2012)
1,
1,
1,
1,
1,
1,
1,
1,
2,
3,
4,
5,
6,
7,
0,
0,
0,
0,
0,
0,
0,
0),
0),
0),
0),
0),
0),
0),
177
pandas: powerful Python data analysis toolkit, Release 0.16.1
datetime.datetime(2000, 1, 8, 0, 0),
datetime.datetime(2000, 1, 9, 0, 0),
datetime.datetime(2000, 1, 10, 0, 0)], dtype=object)
In [20]: dt_array[5]
Out[20]: datetime.datetime(2000, 1, 6, 0, 0)
matplotlib knows how to handle datetime.datetime but not Timestamp objects. While I recommend that you
plot time series using TimeSeries.plot, you can either use to_pydatetime or register a converter for the
Timestamp type. See matplotlib documentation for more on this.
Warning: There are bugs in the user-facing API with the nanosecond datetime64 unit in NumPy 1.6. In particular,
the string version of the array shows garbage values, and conversion to dtype=object is similarly broken.
In [21]: rng = date_range('1/1/2000', periods=10)
In [22]: rng
Out[22]:
DatetimeIndex(['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-04',
'2000-01-05', '2000-01-06', '2000-01-07', '2000-01-08',
'2000-01-09', '2000-01-10'],
dtype='datetime64[ns]', freq='D', tz=None)
In [23]: np.asarray(rng)
Out[23]:
array(['2000-01-01T01:00:00.000000000+0100',
'2000-01-02T01:00:00.000000000+0100',
'2000-01-03T01:00:00.000000000+0100',
'2000-01-04T01:00:00.000000000+0100',
'2000-01-05T01:00:00.000000000+0100',
'2000-01-06T01:00:00.000000000+0100',
'2000-01-07T01:00:00.000000000+0100',
'2000-01-08T01:00:00.000000000+0100',
'2000-01-09T01:00:00.000000000+0100',
'2000-01-10T01:00:00.000000000+0100'], dtype='datetime64[ns]')
In [24]: converted = np.asarray(rng, dtype=object)
In [25]: converted[5]
Out[25]: 947116800000000000L
Trust me: don’t panic. If you are using NumPy 1.6 and restrict your interaction with datetime64 values to
pandas’s API you will be just fine. There is nothing wrong with the data-type (a 64-bit integer internally); all of the
important data processing happens in pandas and is heavily tested. I strongly recommend that you do not work
directly with datetime64 arrays in NumPy 1.6 and only use the pandas API.
Support for non-unique indexes: In the latter case, you may have code inside a try:... catch: block that
failed due to the index not being unique. In many cases it will no longer fail (some method like append still check for
uniqueness unless disabled). However, all is not lost: you can inspect index.is_unique and raise an exception
explicitly if it is False or go to a different code branch.
1.18 v.0.7.3 (April 12, 2012)
This is a minor release from 0.7.2 and fixes many minor bugs and adds a number of nice new features. There are
also a couple of API changes to note; these should not affect very many users, and we are inclined to call them “bug
178
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
fixes” even though they do constitute a change in behavior. See the full release notes or issue tracker on GitHub for a
complete list.
1.18.1 New features
• New fixed width file reader, read_fwf
• New scatter_matrix function for making a scatter plot matrix
from pandas.tools.plotting import scatter_matrix
scatter_matrix(df, alpha=0.2)
• Add stacked argument to Series and DataFrame’s plot method for stacked bar plots.
df.plot(kind='bar', stacked=True)
1.18. v.0.7.3 (April 12, 2012)
179
pandas: powerful Python data analysis toolkit, Release 0.16.1
df.plot(kind='barh', stacked=True)
• Add log x and y scaling options to DataFrame.plot and Series.plot
• Add kurt methods to Series and DataFrame for computing kurtosis
1.18.2 NA Boolean Comparison API Change
Reverted some changes to how NA values (represented typically as NaN or None) are handled in non-numeric Series:
In [1]: series = Series(['Steve', np.nan, 'Joe'])
In [2]: series == 'Steve'
Out[2]:
0
True
180
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
False
2
False
dtype: bool
In [3]: series != 'Steve'
Out[3]:
0
False
1
True
2
True
dtype: bool
In comparisons, NA / NaN will always come through as False except with != which is True. Be very careful with
boolean arithmetic, especially negation, in the presence of NA data. You may wish to add an explicit NA filter into
boolean array operations if you are worried about this:
In [4]: mask = series == 'Steve'
In [5]: series[mask & series.notnull()]
Out[5]:
0
Steve
dtype: object
While propagating NA in comparisons may seem like the right behavior to some users (and you could argue on purely
technical grounds that this is the right thing to do), the evaluation was made that propagating NA everywhere, including
in numerical arrays, would cause a large amount of problems for users. Thus, a “practicality beats purity” approach
was taken. This issue may be revisited at some point in the future.
1.18.3 Other API Changes
When calling apply on a grouped Series, the return value will also be a Series, to be more consistent with the
groupby behavior with DataFrame:
In [1]: df = DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
...:
'foo', 'bar', 'foo', 'foo'],
...:
'B' : ['one', 'one', 'two', 'three',
...:
'two', 'two', 'one', 'three'],
...:
'C' : np.random.randn(8), 'D' : np.random.randn(8)})
...:
In [2]: df
Out[2]:
A
B
0 foo
one
1 bar
one
2 foo
two
3 bar three
4 foo
two
5 bar
two
6 foo
one
7 foo three
C
D
0.144909 1.387310
-1.033812 0.063490
0.197333 1.437656
-0.059730 -0.814844
0.087205 -0.482060
-1.607906 1.521442
-1.275249 0.882182
-0.054460 -0.108020
[8 rows x 4 columns]
In [3]: grouped = df.groupby('A')['C']
In [4]: grouped.describe()
Out[4]:
1.18. v.0.7.3 (April 12, 2012)
181
pandas: powerful Python data analysis toolkit, Release 0.16.1
A
bar
count
mean
std
min
25%
50%
75%
3.000000
-0.900483
0.782652
-1.607906
-1.320859
-1.033812
-0.546771
...
foo mean
-0.180052
std
0.619410
min
-1.275249
25%
-0.054460
50%
0.087205
75%
0.144909
max
0.197333
dtype: float64
In [5]: grouped.apply(lambda x: x.order()[-2:]) # top 2 values
Out[5]:
A
bar 1
-1.033812
3
-0.059730
foo 0
0.144909
2
0.197333
dtype: float64
1.19 v.0.7.2 (March 16, 2012)
This release targets bugs in 0.7.1, and adds a few minor features.
1.19.1 New features
• Add additional tie-breaking methods in DataFrame.rank (GH874)
• Add ascending parameter to rank in Series, DataFrame (GH875)
• Add coerce_float option to DataFrame.from_records (GH893)
• Add sort_columns parameter to allow unsorted plots (GH918)
• Enable column access via attributes on GroupBy (GH882)
• Can pass dict of values to DataFrame.fillna (GH661)
• Can select multiple hierarchical groups by passing list of values in .ix (GH134)
• Add axis option to DataFrame.fillna (GH174)
• Add level keyword to drop for dropping values from a level (GH159)
1.19.2 Performance improvements
• Use khash for Series.value_counts, add raw function to algorithms.py (GH861)
• Intercept __builtin__.sum in groupby (GH885)
182
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.20 v.0.7.1 (February 29, 2012)
This release includes a few new features and addresses over a dozen bugs in 0.7.0.
1.20.1 New features
• Add to_clipboard function to pandas namespace for writing objects to the system clipboard (GH774)
• Add itertuples method to DataFrame for iterating through the rows of a dataframe as tuples (GH818)
• Add ability to pass fill_value and method to DataFrame and Series align method (GH806, GH807)
• Add fill_value option to reindex, align methods (GH784)
• Enable concat to produce DataFrame from Series (GH787)
• Add between method to Series (GH802)
• Add HTML representation hook to DataFrame for the IPython HTML notebook (GH773)
• Support for reading Excel 2007 XML documents using openpyxl
1.20.2 Performance improvements
• Improve performance and memory usage of fillna on DataFrame
• Can concatenate a list of Series along axis=1 to obtain a DataFrame (GH787)
1.21 v.0.7.0 (February 9, 2012)
1.21.1 New features
• New unified merge function for efficiently performing full gamut of database / relational-algebra operations.
Refactored existing join methods to use the new infrastructure, resulting in substantial performance gains
(GH220, GH249, GH267)
• New unified concatenation function for concatenating Series, DataFrame or Panel objects along an axis.
Can form union or intersection of the other axes. Improves performance of Series.append and
DataFrame.append (GH468, GH479, GH273)
• Can pass multiple DataFrames to DataFrame.append to concatenate (stack) and multiple Series to
Series.append too
• Can pass list of dicts (e.g., a list of JSON objects) to DataFrame constructor (GH526)
• You can now set multiple columns in a DataFrame via __getitem__, useful for transformation (GH342)
• Handle differently-indexed output values in DataFrame.apply (GH498)
In [1]: df = DataFrame(randn(10, 4))
In [2]: df.apply(lambda x: x.describe())
Out[2]:
0
1
2
3
count 10.000000 10.000000 10.000000 10.000000
mean
0.119046
0.455043 -0.093701 -0.330828
std
0.814006
0.972606
0.948124
0.814913
1.20. v.0.7.1 (February 29, 2012)
183
pandas: powerful Python data analysis toolkit, Release 0.16.1
min
25%
50%
75%
max
-0.964456
-0.512550
0.013691
0.616168
1.507974
-0.790943
-0.462622
0.415879
1.351857
1.755240
-1.921164
-0.683389
-0.061961
0.671847
1.183075
-1.578003
-0.934434
-0.343709
0.150746
1.051356
[8 rows x 4 columns]
• Add reorder_levels method to Series and DataFrame (GH534)
• Add dict-like get function to DataFrame and Panel (GH521)
• Add DataFrame.iterrows method for efficiently iterating through the rows of a DataFrame
• Add DataFrame.to_panel with code adapted from LongPanel.to_long
• Add reindex_axis method added to DataFrame
• Add level option to binary arithmetic functions on DataFrame and Series
• Add level option to the reindex and align methods on Series and DataFrame for broadcasting values
across a level (GH542, GH552, others)
• Add attribute-based item access to Panel and add IPython completion (GH563)
• Add logy option to Series.plot for log-scaling on the Y axis
• Add index and header options to DataFrame.to_string
• Can pass multiple DataFrames to DataFrame.join to join on index (GH115)
• Can pass multiple Panels to Panel.join (GH115)
• Added justify argument to DataFrame.to_string to allow different alignment of column headers
• Add sort option to GroupBy to allow disabling sorting of the group keys for potential speedups (GH595)
• Can pass MaskedArray to Series constructor (GH563)
• Add Panel item access via attributes and IPython completion (GH554)
• Implement DataFrame.lookup, fancy-indexing analogue for retrieving values given a sequence of row and
column labels (GH338)
• Can pass a list of functions to aggregate with groupby on a DataFrame, yielding an aggregated result with
hierarchical columns (GH166)
• Can call cummin and cummax on Series and DataFrame to get cumulative minimum and maximum, respectively (GH647)
• value_range added as utility function to get min and max of a dataframe (GH288)
• Added encoding argument to read_csv, read_table, to_csv and from_csv for non-ascii text
(GH717)
• Added abs method to pandas objects
• Added crosstab function for easily computing frequency tables
• Added isin method to index objects
• Added level argument to xs method of DataFrame.
184
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.21.2 API Changes to integer indexing
One of the potentially riskiest API changes in 0.7.0, but also one of the most important, was a complete review of how
integer indexes are handled with regard to label-based indexing. Here is an example:
In [3]: s = Series(randn(10), index=range(0, 20, 2))
In [4]: s
Out[4]:
0
-0.392051
2
-0.189537
4
0.886170
6
-1.125894
8
0.319635
10
0.998222
12
0.091743
14
-2.032047
16
-0.448560
18
0.730510
dtype: float64
In [5]: s[0]
Out[5]: -0.39205110783730307
In [6]: s[2]
Out[6]: -0.18953739573269113
In [7]: s[4]
Out[7]: 0.88617008348573789
This is all exactly identical to the behavior before. However, if you ask for a key not contained in the Series, in
versions 0.6.1 and prior, Series would fall back on a location-based lookup. This now raises a KeyError:
In [2]: s[1]
KeyError: 1
This change also has the same impact on DataFrame:
In [3]: df = DataFrame(randn(8, 4), index=range(0, 16, 2))
In [4]: df
0
0
0.88427
2
0.14451
4 -1.44779
6 -0.26598
8 -0.58776
10 0.10940
12 -1.16919
14 -0.07337
1
0.3363
-0.1415
-0.9186
-2.4184
0.3144
-0.7175
-0.3087
0.3410
2
3
-0.1787 0.03162
0.2504 0.58374
-1.4996 0.27163
-0.2658 0.11503
-0.8566 0.61941
-1.0108 0.47990
-0.6049 -0.43544
0.0424 -0.16037
In [5]: df.ix[3]
KeyError: 3
In order to support purely integer-based indexing, the following methods have been added:
1.21. v.0.7.0 (February 9, 2012)
185
pandas: powerful Python data analysis toolkit, Release 0.16.1
Method
Series.iget_value(i)
Series.iget(i)
DataFrame.irow(i)
DataFrame.icol(j)
DataFrame.iget_value(i, j)
Description
Retrieve value stored at location i
Alias for iget_value
Retrieve the i-th row
Retrieve the j-th column
Retrieve the value at row i and column j
1.21.3 API tweaks regarding label-based slicing
Label-based slicing using ix now requires that the index be sorted (monotonic) unless both the start and endpoint are
contained in the index:
In [8]: s = Series(randn(6), index=list('gmkaec'))
In [9]: s
Out[9]:
g
1.269713
m
1.209524
k
2.160843
a
0.533532
e
-2.371548
c
0.562726
dtype: float64
Then this is OK:
In [10]: s.ix['k':'e']
Out[10]:
k
2.160843
a
0.533532
e
-2.371548
dtype: float64
But this is not:
In [12]: s.ix['b':'h']
KeyError 'b'
If the index had been sorted, the “range selection” would have been possible:
In [11]: s2 = s.sort_index()
In [12]: s2
Out[12]:
a
0.533532
c
0.562726
e
-2.371548
g
1.269713
k
2.160843
m
1.209524
dtype: float64
In [13]: s2.ix['b':'h']
Out[13]:
c
0.562726
e
-2.371548
g
1.269713
dtype: float64
186
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
1.21.4 Changes to Series [] operator
As as notational convenience, you can pass a sequence of labels or a label slice to a Series when getting and setting
values via [] (i.e. the __getitem__ and __setitem__ methods). The behavior will be the same as passing
similar input to ix except in the case of integer indexing:
In [14]: s = Series(randn(6), index=list('acegkm'))
In [15]: s
Out[15]:
a
2.031757
c
0.851077
e
0.660056
g
-1.662471
k
0.571380
m
0.945588
dtype: float64
In [16]: s[['m', 'a', 'c', 'e']]
Out[16]:
m
0.945588
a
2.031757
c
0.851077
e
0.660056
dtype: float64
In [17]: s['b':'l']
Out[17]:
c
0.851077
e
0.660056
g
-1.662471
k
0.571380
dtype: float64
In [18]: s['c':'k']
Out[18]:
c
0.851077
e
0.660056
g
-1.662471
k
0.571380
dtype: float64
In the case of integer indexes, the behavior will be exactly as before (shadowing ndarray):
In [19]: s = Series(randn(6), index=range(0, 12, 2))
In [20]: s[[4, 0, 2]]
Out[20]:
4
-1.263534
0
-0.414691
2
2.108285
dtype: float64
In [21]: s[1:5]
Out[21]:
2
2.108285
4
-1.263534
6
2.617801
8
1.967592
1.21. v.0.7.0 (February 9, 2012)
187
pandas: powerful Python data analysis toolkit, Release 0.16.1
dtype: float64
If you wish to do indexing with sequences and slicing on an integer index with label semantics, use ix.
1.21.5 Other API Changes
• The deprecated LongPanel class has been completely removed
• If Series.sort is called on a column of a DataFrame, an exception will now be raised. Before it was
possible to accidentally mutate a DataFrame’s column by doing df[col].sort() instead of the side-effect
free method df[col].order() (GH316)
• Miscellaneous renames and deprecations which will (harmlessly) raise FutureWarning
• drop added as an optional parameter to DataFrame.reset_index (GH699)
1.21.6 Performance improvements
• Cythonized GroupBy aggregations no longer presort the data, thus achieving a significant speedup (GH93).
GroupBy aggregations with Python functions significantly sped up by clever manipulation of the ndarray data
type in Cython (GH496).
• Better error message in DataFrame constructor when passed column labels don’t match data (GH497)
• Substantially improve performance of multi-GroupBy aggregation when a Python function is passed, reuse
ndarray object in Cython (GH496)
• Can store objects indexed by tuples and floats in HDFStore (GH492)
• Don’t print length by default in Series.to_string, add length option (GH489)
• Improve Cython code for multi-groupby to aggregate without having to sort the data (GH93)
• Improve MultiIndex reindexing speed by storing tuples in the MultiIndex, test for backwards unpickling compatibility
• Improve column reindexing performance by using specialized Cython take function
• Further performance tweaking of Series.__getitem__ for standard use cases
• Avoid Index dict creation in some cases (i.e. when getting slices, etc.), regression from prior versions
• Friendlier error message in setup.py if NumPy not installed
• Use common set of NA-handling operations (sum, mean, etc.) in Panel class also (GH536)
• Default name assignment when calling reset_index on DataFrame with a regular (non-hierarchical) index
(GH476)
• Use Cythonized groupers when possible in Series/DataFrame stat ops with level parameter passed (GH545)
• Ported skiplist data structure to C to speed up rolling_median by about 5-10x in most typical use cases
(GH374)
1.22 v.0.6.1 (December 13, 2011)
1.22.1 New features
• Can append single rows (as Series) to a DataFrame
188
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Add Spearman and Kendall rank correlation options to Series.corr and DataFrame.corr (GH428)
• Added get_value and set_value methods to Series, DataFrame, and Panel for very low-overhead access
(>2x faster in many cases) to scalar elements (GH437, GH438). set_value is capable of producing an
enlarged object.
• Add PyQt table widget to sandbox (GH435)
• DataFrame.align can accept Series arguments and an axis option (GH461)
• Implement new SparseArray and SparseList data structures. SparseSeries now derives from SparseArray
(GH463)
• Better console printing options (GH453)
• Implement fast data ranking for Series and DataFrame, fast versions of scipy.stats.rankdata (GH428)
• Implement DataFrame.from_items alternate constructor (GH444)
• DataFrame.convert_objects method for inferring better dtypes for object columns (GH302)
• Add rolling_corr_pairwise function for computing Panel of correlation matrices (GH189)
• Add margins option to pivot_table for computing subgroup aggregates (GH114)
• Add Series.from_csv function (GH482)
• Can pass DataFrame/DataFrame and DataFrame/Series to rolling_corr/rolling_cov (GH #462)
• MultiIndex.get_level_values can accept the level name
1.22.2 Performance improvements
• Improve memory usage of DataFrame.describe (do not copy data unnecessarily) (PR #425)
• Optimize scalar value lookups in the general case by 25% or more in Series and DataFrame
• Fix performance regression in cross-sectional count in DataFrame, affecting DataFrame.dropna speed
• Column deletion in DataFrame copies no data (computes views on blocks) (GH #158)
1.23 v.0.6.0 (November 25, 2011)
1.23.1 New Features
• Added melt function to pandas.core.reshape
• Added level parameter to group by level in Series and DataFrame descriptive statistics (GH313)
• Added head and tail methods to Series, analogous to to DataFrame (GH296)
• Added Series.isin function which checks if each value is contained in a passed sequence (GH289)
• Added float_format option to Series.to_string
• Added skip_footer (GH291) and converters (GH343) options to read_csv and read_table
• Added drop_duplicates and duplicated functions for removing duplicate DataFrame rows and checking for duplicate rows, respectively (GH319)
• Implemented operators ‘&’, ‘|’, ‘^’, ‘-‘ on DataFrame (GH347)
• Added Series.mad, mean absolute deviation
1.23. v.0.6.0 (November 25, 2011)
189
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Added QuarterEnd DateOffset (GH321)
• Added dot to DataFrame (GH65)
• Added orient option to Panel.from_dict (GH359, GH301)
• Added orient option to DataFrame.from_dict
• Added passing list of tuples or list of lists to DataFrame.from_records (GH357)
• Added multiple levels to groupby (GH103)
• Allow multiple columns in by argument of DataFrame.sort_index (GH92, GH362)
• Added fast get_value and put_value methods to DataFrame (GH360)
• Added cov instance methods to Series and DataFrame (GH194, GH362)
• Added kind=’bar’ option to DataFrame.plot (GH348)
• Added idxmin and idxmax to Series and DataFrame (GH286)
• Added read_clipboard function to parse DataFrame from clipboard (GH300)
• Added nunique function to Series for counting unique elements (GH297)
• Made DataFrame constructor use Series name if no columns passed (GH373)
• Support regular expressions in read_table/read_csv (GH364)
• Added DataFrame.to_html for writing DataFrame to HTML (GH387)
• Added support for MaskedArray data in DataFrame, masked values converted to NaN (GH396)
• Added DataFrame.boxplot function (GH368)
• Can pass extra args, kwds to DataFrame.apply (GH376)
• Implement DataFrame.join with vector on argument (GH312)
• Added legend boolean flag to DataFrame.plot (GH324)
• Can pass multiple levels to stack and unstack (GH370)
• Can pass multiple values columns to pivot_table (GH381)
• Use Series name in GroupBy for result index (GH363)
• Added raw option to DataFrame.apply for performance if only need ndarray (GH309)
• Added proper, tested weighted least squares to standard and panel OLS (GH303)
1.23.2 Performance Enhancements
• VBENCH Cythonized cache_readonly, resulting in substantial micro-performance enhancements throughout the codebase (GH361)
• VBENCH Special Cython matrix iterator for applying arbitrary reduction operations with 3-5x better performance than np.apply_along_axis (GH309)
• VBENCH Improved performance of MultiIndex.from_tuples
• VBENCH Special Cython matrix iterator for applying arbitrary reduction operations
• VBENCH + DOCUMENT Add raw option to DataFrame.apply for getting better performance when
• VBENCH Faster cythonized count by level in Series and DataFrame (GH341)
190
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• VBENCH? Significant GroupBy performance enhancement with multiple keys with many “empty” combinations
• VBENCH New Cython vectorized function map_infer speeds up Series.apply and Series.map significantly when passed elementwise Python function, motivated by (GH355)
• VBENCH Significantly improved performance of Series.order, which also makes np.unique called on a
Series faster (GH327)
• VBENCH Vastly improved performance of GroupBy on axes with a MultiIndex (GH299)
1.24 v.0.5.0 (October 24, 2011)
1.24.1 New Features
• Added DataFrame.align method with standard join options
• Added parse_dates option to read_csv and read_table methods to optionally try to parse dates in the
index columns
• Added nrows, chunksize, and iterator arguments to read_csv and read_table. The last two
return a new TextParser class capable of lazily iterating through chunks of a flat file (GH242)
• Added ability to join on multiple columns in DataFrame.join (GH214)
• Added private _get_duplicates function to Index for identifying duplicate values more easily (ENH5c)
• Added column attribute access to DataFrame.
• Added Python tab completion hook for DataFrame columns. (GH233, GH230)
• Implemented Series.describe for Series containing objects (GH241)
• Added inner join option to DataFrame.join when joining on key(s) (GH248)
• Implemented selecting DataFrame columns by passing a list to __getitem__ (GH253)
• Implemented & and | to intersect / union Index objects, respectively (GH261)
• Added pivot_table convenience function to pandas namespace (GH234)
• Implemented Panel.rename_axis function (GH243)
• DataFrame will show index level names in console output (GH334)
• Implemented Panel.take
• Added set_eng_float_format for alternate DataFrame floating point string formatting (ENH61)
• Added convenience set_index function for creating a DataFrame index from its existing columns
• Implemented groupby hierarchical index level name (GH223)
• Added support for different delimiters in DataFrame.to_csv (GH244)
• TODO: DOCS ABOUT TAKE METHODS
1.24.2 Performance Enhancements
• VBENCH Major performance improvements in file parsing functions read_csv and read_table
• VBENCH Added Cython function for converting tuples to ndarray very fast. Speeds up many MultiIndex-related
operations
1.24. v.0.5.0 (October 24, 2011)
191
pandas: powerful Python data analysis toolkit, Release 0.16.1
• VBENCH Refactored merging / joining code into a tidy class and disabled unnecessary computations in the
float/object case, thus getting about 10% better performance (GH211)
• VBENCH Improved speed of DataFrame.xs on mixed-type DataFrame objects by about 5x, regression from
0.3.0 (GH215)
• VBENCH With new DataFrame.align method, speeding up binary operations between differently-indexed
DataFrame objects by 10-25%.
• VBENCH Significantly sped up conversion of nested dict into DataFrame (GH212)
• VBENCH Significantly speed up DataFrame __repr__ and count on large mixed-type DataFrame objects
1.25 v.0.4.3 through v0.4.1 (September 25 - October 9, 2011)
1.25.1 New Features
• Added Python 3 support using 2to3 (GH200)
• Added name attribute to Series, now prints as part of Series.__repr__
• Added instance methods isnull and notnull to Series (GH209, GH203)
• Added Series.align method for aligning two series with choice of join method (ENH56)
• Added method get_level_values to MultiIndex (GH188)
• Set values in mixed-type DataFrame objects via .ix indexing attribute (GH135)
• Added new DataFrame methods get_dtype_counts and property dtypes (ENHdc)
• Added ignore_index option to DataFrame.append to stack DataFrames (ENH1b)
• read_csv tries to sniff delimiters using csv.Sniffer (GH146)
• read_csv can read multiple columns into a MultiIndex; DataFrame’s to_csv method writes out a corresponding MultiIndex (GH151)
• DataFrame.rename has a new copy parameter to rename a DataFrame in place (ENHed)
• Enable unstacking by name (GH142)
• Enable sortlevel to work by level (GH141)
1.25.2 Performance Enhancements
• Altered binary operations on differently-indexed SparseSeries objects to use the integer-based (dense) alignment
logic which is faster with a larger number of blocks (GH205)
• Wrote faster Cython data alignment / merging routines resulting in substantial speed increases
• Improved performance of isnull and notnull, a regression from v0.3.0 (GH187)
• Refactored code related to DataFrame.join so that intermediate aligned copies of the data in each
DataFrame argument do not need to be created. Substantial performance increases result (GH176)
• Substantially improved performance of generic Index.intersection and Index.union
• Implemented BlockManager.take resulting in significantly faster take performance on mixed-type
DataFrame objects (GH104)
• Improved performance of Series.sort_index
192
Chapter 1. What’s New
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Significant groupby performance enhancement: removed unnecessary integrity checks in DataFrame internals
that were slowing down slicing operations to retrieve groups
• Optimized _ensure_index function resulting in performance savings in type-checking Index objects
• Wrote fast time series merging / joining methods in Cython. Will be integrated later into DataFrame.join and
related functions
1.25. v.0.4.3 through v0.4.1 (September 25 - October 9, 2011)
193
pandas: powerful Python data analysis toolkit, Release 0.16.1
194
Chapter 1. What’s New
CHAPTER
TWO
INSTALLATION
The easiest way for the majority of users to install pandas is to install it as part of the Anaconda distribution, a cross
platform distribution for data analysis and scientific computing. This is the recommended installation method for most
users.
Instructions for installing from source, PyPI, various Linux distributions, or a development version are also provided.
2.1 Python version support
Officially Python 2.6, 2.7, 3.2, 3.3, and 3.4.
2.2 Installing pandas
2.2.1 Trying out pandas, no installation required!
The easiest way to start experimenting with pandas doesn’t involve installing pandas at all.
Wakari is a free service that provides a hosted IPython Notebook service in the cloud.
Simply create an account, and have access to pandas from within your brower via an IPython Notebook in a few
minutes.
2.2.2 Installing pandas with Anaconda
Installing pandas and the rest of the NumPy and SciPy stack can be a little difficult for inexperienced users.
The simplest way to install not only pandas, but Python and the most popular packages that make up the SciPy
stack (IPython, NumPy, Matplotlib, ...) is with Anaconda, a cross-platform (Linux, Mac OS X, Windows) Python
distribution for data analytics and scientific computing.
After running a simple installer, the user will have access to pandas and the rest of the SciPy stack without needing to
install anything else, and without needing to wait for any software to be compiled.
Installation instructions for Anaconda can be found here.
A full list of the packages available as part of the Anaconda distribution can be found here.
An additional advantage of installing with Anaconda is that you don’t require admin rights to install it, it will install
in the user’s home directory, and this also makes it trivial to delete Anaconda at a later date (just delete that folder).
195
pandas: powerful Python data analysis toolkit, Release 0.16.1
2.2.3 Installing pandas with Miniconda
The previous section outlined how to get pandas installed as part of the Anaconda distribution. However this approach
means you will install well over one hundred packages and involves downloading the installer which is a few hundred
megabytes in size.
If you want to have more control on which packages, or have a limited internet bandwidth, then installing pandas with
Miniconda may be a better solution.
Conda is the package manager that the Anaconda distribution is built upon. It is a package manager that is both
cross-platform and language agnostic (it can play a similar role to a pip and virtualenv combination).
Miniconda allows you to create a minimal self contained Python installation, and then use the Conda command to
install additional packages.
First you will need Conda to be installed and downloading and running the Miniconda will do this for you. The
installer can be found here
The next step is to create a new conda environment (these are analogous to a virtualenv but they also allow you to
specify precisely which Python version to install also). Run the following commands from a terminal window:
conda create -n name_of_my_env python
This will create a minimal environment with only Python installed in it. To put your self inside this environment run:
source activate name_of_my_env
On Windows the command is:
activate name_of_my_env
The final step required is to install pandas. This can be done with the following command:
conda install pandas
To install a specific pandas version:
conda install pandas=0.13.1
To install other packages, IPython for example:
conda install ipython
To install the full Anaconda distribution:
conda install anaconda
If you require any packages that are available to pip but not conda, simply install pip, and use pip to install these
packages:
conda install pip
pip install django
2.2.4 Installing from PyPI
pandas can be installed via pip from PyPI.
pip install pandas
This will likely require the installation of a number of dependencies, including NumPy, will require a compiler to
compile required bits of code, and can take a few minutes to complete.
196
Chapter 2. Installation
pandas: powerful Python data analysis toolkit, Release 0.16.1
2.2.5 Installing using your Linux distribution’s package manager.
Distribution
Debian
Status
Download / Repository Link
Install method
stable
official Debian repository
Debian
&
Ubuntu
Ubuntu
unstable
(latest
packages)
stable
NeuroDebian
Ubuntu
unstable
(daily
builds)
OpenSuse &
Fedora
stable
PythonXY PPA; activate by: sudo
add-apt-repository
ppa:pythonxy/pythonxy-devel && sudo
apt-get update
OpenSuse Repository
sudo apt-get
install
python-pandas
sudo apt-get
install
python-pandas
sudo apt-get
install
python-pandas
sudo apt-get
install
python-pandas
official Ubuntu repository
zypper in
python-pandas
2.2.6 Installing from source
See the contributing documentation for complete instructions on building from the git source tree. Further, see creating
a devevlopment environment if you wish to create a pandas development environment.
2.2.7 Running the test suite
pandas is equipped with an exhaustive set of unit tests covering about 97% of the codebase as of this writing. To run it
on your machine to verify that everything is working (and you have all of the dependencies, soft and hard, installed),
make sure you have nose and run:
$ nosetests pandas
..........................................................................
.......................S..................................................
..........................................................................
..........................................................................
..........................................................................
..........................................................................
..........................................................................
..........................................................................
..........................................................................
..........................................................................
.................S........................................................
....
---------------------------------------------------------------------Ran 818 tests in 21.631s
OK (SKIP=2)
2.2. Installing pandas
197
pandas: powerful Python data analysis toolkit, Release 0.16.1
2.3 Dependencies
• NumPy: 1.7.0 or higher
• python-dateutil 1.5 or higher
• pytz
– Needed for time zone support
2.3.1 Recommended Dependencies
• numexpr: for accelerating certain numerical operations. numexpr uses multiple cores as well as smart chunking and caching to achieve large speedups. If installed, must be Version 2.1 or higher.
• bottleneck: for accelerating certain types of nan evaluations. bottleneck uses specialized cython routines
to achieve large speedups.
Note: You are highly encouraged to install these libraries, as they provide large speedups, especially if working with
large data sets.
2.3.2 Optional Dependencies
• Cython: Only necessary to build development version. Version 0.19.1 or higher.
• SciPy: miscellaneous statistical functions
• PyTables: necessary for HDF5-based storage. Version 3.0.0 or higher required, Version 3.2.0 or higher highly
recommended.
• SQLAlchemy: for SQL database support. Version 0.8.1 or higher recommended.
• matplotlib: for plotting
• statsmodels
– Needed for parts of pandas.stats
• openpyxl, xlrd/xlwt
– openpyxl version 1.6.1 or higher, but lower than 2.0.0
– Needed for Excel I/O
• XlsxWriter
– Alternative Excel writer.
• boto: necessary for Amazon S3 access.
• ‘blosc >> from pandas import DataFrame
>>> df = DataFrame(...)
...
```
2. Include the full version string of pandas and its dependencies. In recent (>0.12) versions of pandas you can use
a built in function:
>>> from pandas.util.print_versions import show_versions
>>> show_versions()
and in 0.13.1 onwards:
>>> pd.show_versions()
3. Explain why the current behavior is wrong/not desired and what you expect instead.
The issue will then show up to the pandas community and be open to comments/ideas from others.
3.3 Working with the code
Now that you have an issue you want to fix, enhancement to add, or documentation to improve, you need to learn how
to work with GitHub and the pandas code base.
3.3.1 Version Control, Git, and GitHub
To the new user, working with Git is one of the more daunting aspects of contributing to pandas. It can very quickly
become overwhelming, but sticking to the guidelines below will make the process straightforward and will work
without much trouble. As always, if you are having difficulties please feel free to ask for help.
The code is hosted on GitHub. To contribute you will need to sign up for a free GitHub account. We use Git for
version control to allow many people to work together on the project.
Some great resources for learning git:
• the GitHub help pages.
202
Chapter 3. Contributing to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
• the NumPy’s documentation.
• Matthew Brett’s Pydagogue.
3.3.2 Getting Started with Git
GitHub has instructions for installing git, setting up your SSH key, and configuring git. All these steps need to be
completed before working seamlessly with your local repository and GitHub.
3.3.3 Forking
You will need your own fork to work on the code. Go to the pandas project page and hit the fork button. You will want
to clone your fork to your machine:
git clone git@github.com:your-user-name/pandas.git pandas-yourname
cd pandas-yourname
git remote add upstream git://github.com/pydata/pandas.git
This creates the directory pandas-yourname and connects your repository to the upstream (main project) pandas
repository.
The testing suite will run automatically on Travis-CI once your Pull Request is submitted. However, if you wish to run
the test suite on a branch prior to submitting the Pull Request, then Travis-CI needs to be hooked up to your GitHub
repository. Instructions are for doing so are here.
3.3.4 Creating a Branch
You want your master branch to reflect only production-ready code, so create a feature branch for making your changes.
For example:
git branch shiny-new-feature
git checkout shiny-new-feature
The above can be simplified to:
git checkout -b shiny-new-feature
This changes your working directory to the shiny-new-feature branch. Keep any changes in this branch specific to one
bug or feature so it is clear what the branch brings to pandas. You can have many shiny-new-features and switch in
between them using the git checkout command.
To update this branch, you need to retrieve the changes from the master branch:
git fetch upstream
git rebase upstream/master
This will replay your commits on top of the lastest pandas git master. If this leads to merge conflicts, you must resolve
these before submitting your Pull Request. If you have uncommitted changes, you will need to stash them prior to
updating. This will effectively store your changes and they can be reapplied after updating.
3.3.5 Creating a Development Environment
An easy way to create a pandas development environment is as follows.
• Install either Install Anaconda or Install miniconda
3.3. Working with the code
203
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Make sure that you have cloned the repository
• cd to the pandas source directory
Tell conda to create a new environment, named pandas_dev, or any name you would like for this environment by
running:
conda create -n pandas_dev --file ci/requirements_dev.txt
For a python 3 environment
conda create -n pandas_dev python=3 --file ci/requirements_dev.txt
If you are on windows, then you will need to install the compiler linkages:
conda install -n pandas_dev libpython
This will create the new environment, and not touch any of your existing environments, nor any existing python
installation. It will install all of the basic dependencies of pandas, as well as the development and testing tools. If you
would like to install other dependencies, you can install them as follows:
conda install -n pandas_dev -c pandas pytables scipy
To install all pandas dependencies you can do the following:
conda install -n pandas_dev -c pandas --file ci/requirements_all.txt
To work in this environment, activate it as follows:
activate pandas_dev
At which point, the prompt will change to indicate you are in the new development environment.
Note: The above syntax is for windows environments. To work on macosx/linux, use:
source activate pandas_dev
To view your environments:
conda info -e
To return to you home root environment:
deactivate
See the full conda docs here.
At this point you can easily do an in-place install, as detailed in the next section.
3.3.6 Making changes
Before making your code changes, it is often necessary to build the code that was just checked out. There are two
primary methods of doing this.
1. The best way to develop pandas is to build the C extensions in-place by running:
python setup.py build_ext --inplace
If you startup the Python interpreter in the pandas source directory you will call the built C extensions
2. Another very common option is to do a develop install of pandas:
204
Chapter 3. Contributing to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
python setup.py develop
This makes a symbolic link that tells the Python interpreter to import pandas from your development directory.
Thus, you can always be using the development version on your system without being inside the clone directory.
3.4 Contributing to the documentation
If you’re not the developer type, contributing to the documentation is still of huge value. You don’t even have to be an
expert on pandas to do so! Something as simple as rewriting small passages for clarity as you reference the docs is a
simple but effective way to contribute. The next person to read that passage will be in your debt!
Actually, there are sections of the docs that are worse off by being written by experts. If something in the docs doesn’t
make sense to you, updating the relevant section after you figure it out is a simple way to ensure it will help the next
person.
Documentation:
• About the pandas documentation
• How to build the pandas documentation
– Requirements
– Building the documentation
– Built Master Branch Documentation
3.4.1 About the pandas documentation
The documentation is written in reStructuredText, which is almost like writing in plain English, and built using
Sphinx. The Sphinx Documentation has an excellent introduction to reST. Review the Sphinx docs to perform more
complex changes to the documentation as well.
Some other important things to know about the docs:
• The pandas documentation consists of two parts: the docstrings in the code itself and the docs in this folder
pandas/doc/.
The docstrings provide a clear explanation of the usage of the individual functions, while the documentation
in this folder consists of tutorial-like overviews per topic together with some other information (what’s new,
installation, etc).
• The docstrings follow the Numpy Docstring Standard which is used widely in the Scientific Python community. This standard specifies the format of the different sections of the docstring. See this document for a detailed
explanation, or look at some of the existing functions to extend it in a similar manner.
• The tutorials make heavy use of the ipython directive sphinx extension. This directive lets you put code in the
documentation which will be run during the doc build. For example:
.. ipython:: python
x = 2
x**3
will be rendered as
In [1]: x = 2
3.4. Contributing to the documentation
205
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [2]: x**3
Out[2]: 8
This means that almost all code examples in the docs are always run (and the output saved) during the doc build.
This way, they will always be up to date, but it makes the doc building a bit more complex.
3.4.2 How to build the pandas documentation
Requirements
To build the pandas docs there are some extra requirements: you will need to have sphinx and ipython installed.
numpydoc is used to parse the docstrings that follow the Numpy Docstring Standard (see above), but you don’t need
to install this because a local copy of numpydoc is included in the pandas source code.
It is easiest to create a development environment, then install:
conda install -n pandas_dev sphinx ipython
Furthermore, it is recommended to have all optional dependencies installed. This is not strictly necessary, but be aware
that you will see some error messages. Because all the code in the documentation is executed during the doc build, the
examples using this optional dependencies will generate errors. Run pd.show_versions() to get an overview of
the installed version of all dependencies.
Warning: Sphinx version >= 1.2.2 or the older 1.1.3 is required.
Building the documentation
So how do you build the docs? Navigate to your local the folder pandas/doc/ directory in the console and run:
python make.py html
And then you can find the html output in the folder pandas/doc/build/html/.
The first time it will take quite a while, because it has to run all the code examples in the documentation and build all
generated docstring pages. In subsequent evocations, sphinx will try to only build the pages that have been modified.
If you want to do a full clean build, do:
python make.py clean
python make.py build
Starting with 0.13.1 you can tell make.py to compile only a single section of the docs, greatly reducing the turnaround time for checking your changes. You will be prompted to delete .rst files that aren’t required. This is okay
since the prior version can be checked out from git, but make sure to not commit the file deletions.
#omit autosummary and API section
python make.py clean
python make.py --no-api
# compile the docs with only a single
# section, that which is in indexing.rst
python make.py clean
python make.py --single indexing
206
Chapter 3. Contributing to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
For comparison, a full documentation build may take 10 minutes. a -no-api build may take 3 minutes and a single
section may take 15 seconds. However, subsequent builds only process portions you changed. Now, open the following
file in a web browser to see the full documentation you just built:
pandas/docs/build/html/index.html
And you’ll have the satisfaction of seeing your new and improved documentation!
Built Master Branch Documentation
When pull-requests are merged into the pandas master branch, the main parts of the documentation are also built by
Travis-CI. These docs are then hosted here.
3.5 Contributing to the code base
Code Base:
• Code Standards
• Test-driven Development/Writing Code
– Writing tests
– Running the test suite
– Running the performance test suite
• Documenting your code
3.5.1 Code Standards
pandas uses the PEP8 standard. There are several tools to ensure you abide by this standard.
We’ve written a tool to check that your commits are PEP8 great, pip install pep8radius. Look at PEP8 fixes in your
branch vs master with:
pep8radius master --diff` and make these changes with `pep8radius master --diff --in-place`
Alternatively, use flake8 tool for checking the style of your code. Additional standards are outlined on the code style
wiki page.
Please try to maintain backward-compatibility. Pandas has lots of users with lots of existing code, so don’t break it if
at all possible. If you think breakage is required clearly state why as part of the Pull Request. Also, be careful when
changing method signatures and add deprecation warnings where needed.
3.5.2 Test-driven Development/Writing Code
Pandas is serious about testing and strongly encourages individuals to embrace Test-driven Development (TDD). This
development process “relies on the repetition of a very short development cycle: first the developer writes an (initially
failing) automated test case that defines a desired improvement or new function, then produces the minimum amount
of code to pass that test.” So, before actually writing any code, you should write your tests. Often the test can
be taken from the original GitHub issue. However, it is always worth considering additional use cases and writing
corresponding tests.
Adding tests is one of the most common requests after code is pushed to pandas. It is worth getting in the habit of
writing tests ahead of time so this is never an issue.
3.5. Contributing to the code base
207
pandas: powerful Python data analysis toolkit, Release 0.16.1
Like many packages, pandas uses the Nose testing system and the convenient extensions in numpy.testing.
Writing tests
All tests should go into the tests subdirectory of the specific package. There are probably many examples already there
and looking to these for inspiration is suggested. If you test requires working with files or network connectivity there
is more information on the testing page of the wiki.
The pandas.util.testing module has many special assert functions that make it easier to make statements
about whether Series or DataFrame objects are equivalent. The easiest way to verify that your code is correct is to
explicitly construct the result you expect, then compare the actual result to the expected correct result:
def test_pivot(self):
data = {
'index' : ['A', 'B', 'C', 'C', 'B', 'A'],
'columns' : ['One', 'One', 'One', 'Two', 'Two', 'Two'],
'values' : [1., 2., 3., 3., 2., 1.]
}
frame = DataFrame(data)
pivoted = frame.pivot(index='index', columns='columns', values='values')
expected = DataFrame({
'One' : {'A' : 1., 'B' : 2., 'C' : 3.},
'Two' : {'A' : 1., 'B' : 2., 'C' : 3.}
})
assert_frame_equal(pivoted, expected)
Running the test suite
The tests can then be run directly inside your git clone (without having to install pandas) by typing::
nosetests pandas
The tests suite is exhaustive and takes around 20 minutes to run. Often it is worth running only a subset of tests first
around your changes before running the entire suite. This is done using one of the following constructs:
nosetests pandas/tests/[test-module].py
nosetests pandas/tests/[test-module].py:[TestClass]
nosetests pandas/tests/[test-module].py:[TestClass].[test_method]
Running the performance test suite
Performance matters and it is worth considering that your code has not introduced performance regressions. Currently
pandas uses the vbench library to enable easy monitoring of the performance of critical pandas operations. These
benchmarks are all found in the pandas/vb_suite directory. vbench currently only works on python2.
To install vbench:
pip install git+https://github.com/pydata/vbench
Vbench also requires sqlalchemy, gitpython, and psutil which can all be installed using pip. If you need to run a
benchmark, change your directory to the pandas root and run:
208
Chapter 3. Contributing to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
./test_perf.sh -b master -t HEAD
This will checkout the master revision and run the suite on both master and your commit. Running the full test suite
can take up to one hour and use up to 3GB of RAM. Usually it is sufficient to past a subset of the results in to the Pull
Request to show that the committed changes do not cause unexpected performance regressions.
You can run specific benchmarks using the -r flag which takes a regular expression.
See the performance testing wiki for information on how to write a benchmark.
3.5.3 Documenting your code
Changes should be reflected in the release notes located in doc/source/whatsnew/vx.y.z.txt. This file contains an ongoing change log for each release. Add an entry to this file to document your fix, enhancement or (unavoidable) breaking
change. Make sure to include the GitHub issue number when adding your entry.
If your code is an enhancement, it is most likely necessary to add usage examples to the existing documentation. This
can be done following the section regarding documentation.
3.6 Contributing your changes to pandas
3.6.1 Committing your code
Keep style fixes to a separate commit to make your PR more readable.
Once you’ve made changes, you can see them by typing:
git status
If you’ve created a new file, it is not being tracked by git. Add it by typing
git add path/to/file-to-be-added.py
Doing ‘git status’ again should give something like
# On branch shiny-new-feature
#
#
modified:
/relative/path/to/file-you-added.py
#
Finally, commit your changes to your local repository with an explanatory message. Pandas uses a convention for
commit message prefixes and layout. Here are some common prefixes along with general guidelines for when to use
them:
• ENH: Enhancement, new functionality
• BUG: Bug fix
• DOC: Additions/updates to documentation
• TST: Additions/updates to tests
• BLD: Updates to the build process/scripts
• PERF: Performance improvement
• CLN: Code cleanup
3.6. Contributing your changes to pandas
209
pandas: powerful Python data analysis toolkit, Release 0.16.1
The following defines how a commit message should be structured. Please reference the relevant GitHub issues in
your commit message using GH1234 or #1234. Either style is fine, but the former is generally preferred:
• a subject line with < 80 chars.
• One blank line.
• Optionally, a commit message body.
Now you can commit your changes in your local repository:
git commit -m
If you have multiple commits, it is common to want to combine them into one commit, often referred to as “squashing”
or “rebasing”. This is a common request by package maintainers when submitting a Pull Request as it maintains a
more compact commit history. To rebase your commits:
git rebase -i HEAD~#
Where # is the number of commits you want to combine. Then you can pick the relevant commit message and discard
others.
3.6.2 Pushing your changes
When you want your changes to appear publicly on your GitHub page, push your forked feature branch’s commits
git push origin shiny-new-feature
Here origin is the default name given to your remote repository on GitHub. You can see the remote repositories
git remote -v
If you added the upstream repository as described above you will see something like
origin git@github.com:yourname/pandas.git (fetch)
origin git@github.com:yourname/pandas.git (push)
upstream
git://github.com/pydata/pandas.git (fetch)
upstream
git://github.com/pydata/pandas.git (push)
Now your code is on GitHub, but it is not yet a part of the pandas project. For that to happen, a Pull Request needs to
be submitted on GitHub.
3.6.3 Review your code
When you’re ready to ask for a code review, you will file a Pull Request. Before you do, again make sure you’ve
followed all the guidelines outlined in this document regarding code style, tests, performance tests, and documentation.
You should also double check your branch changes against the branch it was based off of:
1. Navigate to your repository on GitHub–https://github.com/your-user-name/pandas.
2. Click on Branches.
3. Click on the Compare button for your feature branch.
4. Select the base and compare branches, if necessary. This will be master and shiny-new-feature, respectively.
210
Chapter 3. Contributing to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
3.6.4 Finally, make the Pull Request
If everything looks good you are ready to make a Pull Request. A Pull Request is how code from a local repository
becomes available to the GitHub community and can be looked at and eventually merged into the master version. This
Pull Request and its associated changes will eventually be committed to the master branch and available in the next
release. To submit a Pull Request:
1. Navigate to your repository on GitHub.
2. Click on the Pull Request button.
3. You can then click on Commits and Files Changed to make sure everything looks okay one last time.
4. Write a description of your changes in the Preview Discussion tab.
5. Click Send Pull Request.
This request then appears to the repository maintainers, and they will review the code. If you need to make more
changes, you can make them in your branch, push them to GitHub, and the pull request will be automatically updated.
Pushing them to GitHub again is done by:
git push -f origin shiny-new-feature
This will automatically update your Pull Request with the latest code and restart the Travis-CI tests.
3.6.5 Delete your merged branch (optional)
Once your feature branch is accepted into upstream, you’ll probably want to get rid of the branch. First, merge
upstream master into your branch so git knows it is safe to delete your branch
git fetch upstream
git checkout master
git merge upstream/master
Then you can just do:
git branch -d shiny-new-feature
Make sure you use a lower-case -d, or else git won’t warn you if your feature branch has not actually been merged.
The branch will still exist on GitHub, so to delete it there do
git push origin --delete shiny-new-feature
3.6. Contributing your changes to pandas
211
pandas: powerful Python data analysis toolkit, Release 0.16.1
212
Chapter 3. Contributing to pandas
CHAPTER
FOUR
FREQUENTLY ASKED QUESTIONS (FAQ)
4.1 DataFrame memory usage
As of pandas version 0.15.0, the memory usage of a dataframe (including the index) is shown when accessing the info
method of a dataframe. A configuration option, display.memory_usage (see Options and Settings), specifies if
the dataframe’s memory usage will be displayed when invoking the df.info() method.
For example, the memory usage of the dataframe below is shown when calling df.info():
In [1]: dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',
...:
'complex128', 'object', 'bool']
...:
In [2]: n = 5000
In [3]: data = dict([ (t, np.random.randint(100, size=n).astype(t))
...:
for t in dtypes])
...:
In [4]: df = DataFrame(data)
In [5]: df['categorical'] = df['object'].astype('category')
In [6]: df.info()
Int64Index: 5000 entries, 0 to 4999
Data columns (total 8 columns):
bool
5000 non-null bool
complex128
5000 non-null complex128
datetime64[ns]
5000 non-null datetime64[ns]
float64
5000 non-null float64
int64
5000 non-null int64
object
5000 non-null object
timedelta64[ns]
5000 non-null timedelta64[ns]
categorical
5000 non-null category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), time
memory usage: 303.5+ KB
The + symbol indicates that the true memory usage could be higher, because pandas does not count the memory used
by values in columns with dtype=object.
By default the display option is set to True but can be explicitly overridden by passing the memory_usage argument
when invoking df.info().
The memory usage of each column can be found by calling the memory_usage method. This returns a Series with
213
pandas: powerful Python data analysis toolkit, Release 0.16.1
an index represented by column names and memory usage of each column shown in bytes. For the dataframe above,
the memory usage of each column and the total memory usage of the dataframe can be found with the memory_usage
method:
In [7]: df.memory_usage()
Out[7]:
bool
5000
complex128
80000
datetime64[ns]
40000
float64
40000
int64
40000
object
20000
timedelta64[ns]
40000
categorical
5800
dtype: int64
# total memory usage of dataframe
In [8]: df.memory_usage().sum()
Out[8]: 270800
By default the memory usage of the dataframe’s index is not shown in the returned Series, the memory usage of the
index can be shown by passing the index=True argument:
In [9]: df.memory_usage(index=True)
Out[9]:
Index
40000
bool
5000
complex128
80000
datetime64[ns]
40000
float64
40000
int64
40000
object
20000
timedelta64[ns]
40000
categorical
5800
dtype: int64
The memory usage displayed by the info method utilizes the memory_usage method to determine the memory
usage of a dataframe while also formatting the output in human-readable units (base-2 representation; i.e., 1KB = 1024
bytes).
See also Categorical Memory Usage.
4.2 Adding Features to your pandas Installation
pandas is a powerful tool and already has a plethora of data manipulation operations implemented, most of them are
very fast as well. It’s very possible however that certain functionality that would make your life easier is missing. In
that case you have several options:
1. Open an issue on Github , explain your need and the sort of functionality you would like to see implemented.
2. Fork the repo, Implement the functionality yourself and open a PR on Github.
3. Write a method that performs the operation you are interested in and Monkey-patch the pandas class as part of
your IPython profile startup or PYTHONSTARTUP file.
For example, here is an example of adding an just_foo_cols() method to the dataframe class:
214
Chapter 4. Frequently Asked Questions (FAQ)
pandas: powerful Python data analysis toolkit, Release 0.16.1
import pandas as pd
def just_foo_cols(self):
"""Get a list of column names containing the string 'foo'
"""
return [x for x in self.columns if 'foo' in x]
pd.DataFrame.just_foo_cols = just_foo_cols # monkey-patch the DataFrame class
df = pd.DataFrame([list(range(4))], columns=["A","foo","foozball","bar"])
df.just_foo_cols()
del pd.DataFrame.just_foo_cols # you can also remove the new method
Monkey-patching is usually frowned upon because it makes your code less portable and can cause subtle bugs in some
circumstances. Monkey-patching existing methods is usually a bad idea in that respect. When used with proper care,
however, it’s a very useful tool to have.
4.3 Migrating from scikits.timeseries to pandas >= 0.8.0
Starting with pandas 0.8.0, users of scikits.timeseries should have all of the features that they need to migrate their code
to use pandas. Portions of the scikits.timeseries codebase for implementing calendar logic and timespan frequency
conversions (but not resampling, that has all been implemented from scratch from the ground up) have been ported to
the pandas codebase.
The scikits.timeseries notions of Date and DateArray are responsible for implementing calendar logic:
In [16]: dt = ts.Date('Q', '1984Q3')
# sic
In [17]: dt
Out[17]:
In [18]: dt.asfreq('D', 'start')
Out[18]:
In [19]: dt.asfreq('D', 'end')
Out[19]:
In [20]: dt + 3
Out[20]:
Date and DateArray from scikits.timeseries have been reincarnated in pandas Period and PeriodIndex:
In [10]: pnow('D') # scikits.timeseries.now()
Out[10]: Period('2015-05-11', 'D')
In [11]: Period(year=2007, month=3, day=15, freq='D')
Out[11]: Period('2007-03-15', 'D')
In [12]: p = Period('1984Q3')
In [13]: p
Out[13]: Period('1984Q3', 'Q-DEC')
In [14]: p.asfreq('D', 'start')
Out[14]: Period('1984-07-01', 'D')
In [15]: p.asfreq('D', 'end')
4.3. Migrating from scikits.timeseries to pandas >= 0.8.0
215
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[15]: Period('1984-09-30', 'D')
In [16]: (p + 3).asfreq('T') + 6 * 60 + 30
Out[16]: Period('1985-07-01 06:29', 'T')
In [17]: rng = period_range('1990', '2010', freq='A')
In [18]: rng
Out[18]:
PeriodIndex(['1990', '1991', '1992', '1993', '1994', '1995', '1996', '1997',
'1998', '1999', '2000', '2001', '2002', '2003', '2004', '2005',
'2006', '2007', '2008', '2009', '2010'],
dtype='int64', freq='A-DEC')
In [19]: rng.asfreq('B', 'end') - 3
Out[19]:
PeriodIndex(['1990-12-26', '1991-12-26',
'1994-12-27', '1995-12-26',
'1998-12-28', '1999-12-28',
'2002-12-26', '2003-12-26',
'2006-12-26', '2007-12-26',
'2010-12-28'],
dtype='int64', freq='B')
scikits.timeseries
Date
DateArray
convert
convert_to_annual
pandas
Period
PeriodIndex
resample
pivot_annual
'1992-12-28',
'1996-12-26',
'2000-12-26',
'2004-12-28',
'2008-12-26',
'1993-12-28',
'1997-12-26',
'2001-12-26',
'2005-12-27',
'2009-12-28',
Notes
A span of time, from yearly through to secondly
An array of timespans
Frequency conversion in scikits.timeseries
currently supports up to daily frequency, see GH736
4.3.1 PeriodIndex / DateArray properties and functions
The scikits.timeseries DateArray had a number of information properties. Here are the pandas equivalents:
scikits.timeseries
get_steps
has_missing_dates
is_full
is_valid
is_chronological
arr.sort_chronologically()
pandas
np.diff(idx.values)
not idx.is_full
idx.is_full
idx.is_monotonic and idx.is_unique
is_monotonic
idx.order()
Notes
4.3.2 Frequency conversion
Frequency conversion is implemented using the resample method on TimeSeries and DataFrame objects (multiple
time series). resample also works on panels (3D). Here is some code that resamples daily data to monthly:
In [20]: rng = period_range('Jan-2000', periods=50, freq='M')
In [21]: data = Series(np.random.randn(50), index=rng)
In [22]: data
Out[22]:
2000-01
1.544821
2000-02
-1.708552
216
Chapter 4. Frequently Asked Questions (FAQ)
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-03
2000-04
2000-05
2000-06
2000-07
1.545458
-0.735738
-0.649091
-0.403878
-2.474932
...
2003-08
1.034493
2003-09
1.269838
2003-10
0.606166
2003-11
-0.827409
2003-12
-0.943863
2004-01
1.041569
2004-02
0.701815
Freq: M, dtype: float64
In [23]: data.resample('A', how=np.mean)
Out[23]:
2000
0.102447
2001
-0.204847
2002
0.210840
2003
0.300564
2004
0.871692
Freq: A-DEC, dtype: float64
4.3.3 Plotting
Much of the plotting functionality of scikits.timeseries has been ported and adopted to pandas’s data structures. For
example:
In [24]: rng = period_range('1987Q2', periods=10, freq='Q-DEC')
In [25]: data = Series(np.random.randn(10), index=rng)
In [26]: plt.figure(); data.plot()
Out[26]:
4.3. Migrating from scikits.timeseries to pandas >= 0.8.0
217
pandas: powerful Python data analysis toolkit, Release 0.16.1
4.3.4 Converting to and from period format
Use the to_timestamp and to_period instance methods.
4.3.5 Treatment of missing data
Unlike scikits.timeseries, pandas data structures are not based on NumPy’s MaskedArray object. Missing data is
represented as NaN in numerical arrays and either as None or NaN in non-numerical arrays. Implementing a version of
pandas’s data structures that use MaskedArray is possible but would require the involvement of a dedicated maintainer.
Active pandas developers are not interested in this.
4.3.6 Resampling with timestamps and periods
resample has a kind argument which allows you to resample time series with a DatetimeIndex to PeriodIndex:
In [27]: rng = date_range('1/1/2000', periods=200, freq='D')
In [28]: data = Series(np.random.randn(200), index=rng)
In [29]: data[:10]
218
Chapter 4. Frequently Asked Questions (FAQ)
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[29]:
2000-01-01
-0.197661
2000-01-02
0.507155
2000-01-03
-0.493913
2000-01-04
-0.994339
2000-01-05
-0.581662
2000-01-06
-0.855251
2000-01-07
-0.256469
2000-01-08
-0.454868
2000-01-09
0.519612
2000-01-10
0.764490
Freq: D, dtype: float64
In [30]: data.index
Out[30]:
DatetimeIndex(['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-04',
'2000-01-05', '2000-01-06', '2000-01-07', '2000-01-08',
'2000-01-09', '2000-01-10',
...
'2000-07-09', '2000-07-10', '2000-07-11', '2000-07-12',
'2000-07-13', '2000-07-14', '2000-07-15', '2000-07-16',
'2000-07-17', '2000-07-18'],
dtype='datetime64[ns]', length=200, freq='D', tz=None)
In [31]: data.resample('M', kind='period')
Out[31]:
2000-01
-0.226155
2000-02
0.056704
2000-03
-0.132553
2000-04
-0.064003
2000-05
0.233736
2000-06
-0.301008
2000-07
-0.584631
Freq: M, dtype: float64
Similarly, resampling from periods to timestamps is possible with an optional interval (’start’ or ’end’) convention:
In [32]: rng = period_range('Jan-2000', periods=50, freq='M')
In [33]: data = Series(np.random.randn(50), index=rng)
In [34]: resampled = data.resample('A', kind='timestamp', convention='end')
In [35]: resampled.index
Out[35]:
DatetimeIndex(['2000-12-31', '2001-12-31', '2002-12-31', '2003-12-31',
'2004-12-31'],
dtype='datetime64[ns]', freq='A-DEC', tz=None)
4.4 Byte-Ordering Issues
Occasionally you may have to deal with data that were created on a machine with a different byte order than the one
on which you are running Python. To deal with this issue you should convert the underlying NumPy array to the native
system byte order before passing it to Series/DataFrame/Panel constructors using something similar to the following:
4.4. Byte-Ordering Issues
219
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [36]: x = np.array(list(range(10)), '>i4') # big endian
In [37]: newx = x.byteswap().newbyteorder() # force native byteorder
In [38]: s = Series(newx)
See the NumPy documentation on byte order for more details.
4.5 Visualizing Data in Qt applications
Warning: The qt support is deprecated and will be removed in a future version. We refer users to the external
package pandas-qt.
There is experimental support for visualizing DataFrames in PyQt4 and PySide applications. At the moment you
can display and edit the values of the cells in the DataFrame. Qt will take care of displaying just the portion of the
DataFrame that is currently visible and the edits will be immediately saved to the underlying DataFrame
To demonstrate this we will create a simple PySide application that will switch between two editable DataFrames. For
this will use the DataFrameModel class that handles the access to the DataFrame, and the DataFrameWidget,
which is just a thin layer around the QTableView.
import numpy as np
import pandas as pd
from pandas.sandbox.qtpandas import DataFrameModel, DataFrameWidget
from PySide import QtGui, QtCore
# Or if you use PyQt4:
# from PyQt4 import QtGui, QtCore
class MainWidget(QtGui.QWidget):
def __init__(self, parent=None):
super(MainWidget, self).__init__(parent)
# Create two DataFrames
self.df1 = pd.DataFrame(np.arange(9).reshape(3, 3),
columns=['foo', 'bar', 'baz'])
self.df2 = pd.DataFrame({
'int': [1, 2, 3],
'float': [1.5, 2.5, 3.5],
'string': ['a', 'b', 'c'],
'nan': [np.nan, np.nan, np.nan]
}, index=['AAA', 'BBB', 'CCC'],
columns=['int', 'float', 'string', 'nan'])
# Create the widget and set the first DataFrame
self.widget = DataFrameWidget(self.df1)
# Create the buttons for changing DataFrames
self.button_first = QtGui.QPushButton('First')
self.button_first.clicked.connect(self.on_first_click)
self.button_second = QtGui.QPushButton('Second')
self.button_second.clicked.connect(self.on_second_click)
# Set the layout
vbox = QtGui.QVBoxLayout()
220
Chapter 4. Frequently Asked Questions (FAQ)
pandas: powerful Python data analysis toolkit, Release 0.16.1
vbox.addWidget(self.widget)
hbox = QtGui.QHBoxLayout()
hbox.addWidget(self.button_first)
hbox.addWidget(self.button_second)
vbox.addLayout(hbox)
self.setLayout(vbox)
def on_first_click(self):
'''Sets the first DataFrame'''
self.widget.setDataFrame(self.df1)
def on_second_click(self):
'''Sets the second DataFrame'''
self.widget.setDataFrame(self.df2)
if __name__ == '__main__':
import sys
# Initialize the application
app = QtGui.QApplication(sys.argv)
mw = MainWidget()
mw.show()
app.exec_()
4.5. Visualizing Data in Qt applications
221
pandas: powerful Python data analysis toolkit, Release 0.16.1
222
Chapter 4. Frequently Asked Questions (FAQ)
CHAPTER
FIVE
PACKAGE OVERVIEW
pandas consists of the following things
• A set of labeled array data structures, the primary of which are Series/TimeSeries and DataFrame
• Index objects enabling both simple axis indexing and multi-level / hierarchical axis indexing
• An integrated group by engine for aggregating and transforming data sets
• Date range generation (date_range) and custom date offsets enabling the implementation of customized frequencies
• Input/Output tools: loading tabular data from flat files (CSV, delimited, Excel 2003), and saving and loading
pandas objects from the fast and efficient PyTables/HDF5 format.
• Memory-efficient “sparse” versions of the standard data structures for storing data that is mostly missing or
mostly constant (some fixed value)
• Moving window statistics (rolling mean, rolling standard deviation, etc.)
• Static and moving window linear and panel regression
5.1 Data structures at a glance
Dimensions
1
1
2
3
Name
Description
Series
1D labeled homogeneously-typed array
TimeSeries with index containing datetimes
Series
DataFrame General 2D labeled, size-mutable tabular structure with potentially
heterogeneously-typed columns
Panel
General 3D labeled, also size-mutable array
5.1.1 Why more than 1 data structure?
The best way to think about the pandas data structures is as flexible containers for lower dimensional data. For
example, DataFrame is a container for Series, and Panel is a container for DataFrame objects. We would like to be
able to insert and remove objects from these containers in a dictionary-like fashion.
Also, we would like sensible default behaviors for the common API functions which take into account the typical
orientation of time series and cross-sectional data sets. When using ndarrays to store 2- and 3-dimensional data, a
burden is placed on the user to consider the orientation of the data set when writing functions; axes are considered
more or less equivalent (except when C- or Fortran-contiguousness matters for performance). In pandas, the axes are
intended to lend more semantic meaning to the data; i.e., for a particular data set there is likely to be a “right” way to
223
pandas: powerful Python data analysis toolkit, Release 0.16.1
orient the data. The goal, then, is to reduce the amount of mental effort required to code up data transformations in
downstream functions.
For example, with tabular data (DataFrame) it is more semantically helpful to think of the index (the rows) and the
columns rather than axis 0 and axis 1. And iterating through the columns of the DataFrame thus results in more
readable code:
for col in df.columns:
series = df[col]
# do something with series
5.2 Mutability and copying of data
All pandas data structures are value-mutable (the values they contain can be altered) but not always size-mutable. The
length of a Series cannot be changed, but, for example, columns can be inserted into a DataFrame. However, the vast
majority of methods produce new objects and leave the input data untouched. In general, though, we like to favor
immutability where sensible.
5.3 Getting Support
The first stop for pandas issues and ideas is the Github Issue Tracker. If you have a general question, pandas community
experts can answer through Stack Overflow.
Longer discussions occur on the developer mailing list, and commercial support inquiries for Lambda Foundry should
be sent to: support@lambdafoundry.com
5.4 Credits
pandas development began at AQR Capital Management in April 2008. It was open-sourced at the end of 2009. AQR
continued to provide resources for development through the end of 2011, and continues to contribute bug reports today.
Since January 2012, Lambda Foundry, has been providing development resources, as well as commercial support,
training, and consulting for pandas.
pandas is only made possible by a group of people around the world like you who have contributed new code, bug
reports, fixes, comments and ideas. A complete list can be found on Github.
5.5 Development Team
pandas is a part of the PyData project. The PyData Development Team is a collection of developers focused on the
improvement of Python’s data libraries. The core team that coordinates development can be found on Github. If you’re
interested in contributing, please visit the project website.
5.6 License
224
Chapter 5. Package overview
pandas: powerful Python data analysis toolkit, Release 0.16.1
=======
License
=======
pandas is distributed under a 3-clause ("Simplified" or "New") BSD
license. Parts of NumPy, SciPy, numpydoc, bottleneck, which all have
BSD-compatible licenses, are included. Their licenses follow the pandas
license.
pandas license
==============
Copyright (c) 2011-2012, Lambda Foundry, Inc. and PyData Development Team
All rights reserved.
Copyright (c) 2008-2011 AQR Capital Management, LLC
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided
with the distribution.
* Neither the name of the copyright holder nor the names of any
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
About the Copyright Holders
===========================
AQR Capital Management began pandas development in 2008. Development was
led by Wes McKinney. AQR released the source under this license in 2009.
Wes is now an employee of Lambda Foundry, and remains the pandas project
lead.
The PyData Development Team is the collection of developers of the PyData
project. This includes all of the PyData sub-projects, including pandas. The
core team that coordinates development on GitHub can be found here:
http://github.com/pydata.
5.6. License
225
pandas: powerful Python data analysis toolkit, Release 0.16.1
Full credits for pandas contributors can be found in the documentation.
Our Copyright Policy
====================
PyData uses a shared copyright model. Each contributor maintains copyright
over their contributions to PyData. However, it is important to note that
these contributions are typically only changes to the repositories. Thus,
the PyData source code, in its entirety, is not the copyright of any single
person or institution. Instead, it is the collective copyright of the
entire PyData Development Team. If individual contributors want to maintain
a record of what changes/contributions they have specific copyright on,
they should indicate their copyright in the commit message of the change
when they commit the change to one of the PyData repositories.
With this in mind, the following banner should be used in any source code
file to indicate the copyright and license terms:
#----------------------------------------------------------------------------# Copyright (c) 2012, PyData Development Team
# All rights reserved.
#
# Distributed under the terms of the BSD Simplified License.
#
# The full license is in the LICENSE file, distributed with this software.
#----------------------------------------------------------------------------Other licenses can be found in the LICENSES directory.
226
Chapter 5. Package overview
CHAPTER
SIX
10 MINUTES TO PANDAS
This is a short introduction to pandas, geared mainly for new users. You can see more complex recipes in the Cookbook
Customarily, we import as follows
In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: import matplotlib.pyplot as plt
6.1 Object Creation
See the Data Structure Intro section
Creating a Series by passing a list of values, letting pandas create a default integer index:
In [4]: s = pd.Series([1,3,5,np.nan,6,8])
In [5]: s
Out[5]:
0
1
1
3
2
5
3
NaN
4
6
5
8
dtype: float64
Creating a DataFrame by passing a numpy array, with a datetime index and labeled columns:
In [6]: dates = pd.date_range('20130101', periods=6)
In [7]: dates
Out[7]:
DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04',
'2013-01-05', '2013-01-06'],
dtype='datetime64[ns]', freq='D', tz=None)
In [8]: df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))
In [9]: df
Out[9]:
A
B
C
D
227
pandas: powerful Python data analysis toolkit, Release 0.16.1
2013-01-01 0.469112 -0.282863 -1.509059 -1.135632
2013-01-02 1.212112 -0.173215 0.119209 -1.044236
2013-01-03 -0.861849 -2.104569 -0.494929 1.071804
2013-01-04 0.721555 -0.706771 -1.039575 0.271860
2013-01-05 -0.424972 0.567020 0.276232 -1.087401
2013-01-06 -0.673690 0.113648 -1.478427 0.524988
Creating a DataFrame by passing a dict of objects that can be converted to series-like.
In [10]: df2 = pd.DataFrame({ 'A' : 1.,
....:
'B' : pd.Timestamp('20130102'),
....:
'C' : pd.Series(1,index=list(range(4)),dtype='float32'),
....:
'D' : np.array([3] * 4,dtype='int32'),
....:
'E' : pd.Categorical(["test","train","test","train"]),
....:
'F' : 'foo' })
....:
In [11]: df2
Out[11]:
A
B
0 1 2013-01-02
1 1 2013-01-02
2 1 2013-01-02
3 1 2013-01-02
C
1
1
1
1
D
3
3
3
3
E
test
train
test
train
F
foo
foo
foo
foo
Having specific dtypes
In [12]: df2.dtypes
Out[12]:
A
float64
B
datetime64[ns]
C
float32
D
int32
E
category
F
object
dtype: object
If you’re using IPython, tab completion for column names (as well as public attributes) is automatically enabled.
Here’s a subset of the attributes that will be completed:
In [13]: df2.
df2.A
df2.abs
df2.add
df2.add_prefix
df2.add_suffix
df2.align
df2.all
df2.any
df2.append
df2.apply
df2.applymap
df2.as_blocks
df2.asfreq
df2.as_matrix
df2.astype
df2.at
df2.at_time
df2.axes
228
df2.boxplot
df2.C
df2.clip
df2.clip_lower
df2.clip_upper
df2.columns
df2.combine
df2.combineAdd
df2.combine_first
df2.combineMult
df2.compound
df2.consolidate
df2.convert_objects
df2.copy
df2.corr
df2.corrwith
df2.count
df2.cov
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
df2.B
df2.between_time
df2.bfill
df2.blocks
df2.bool
df2.cummax
df2.cummin
df2.cumprod
df2.cumsum
df2.D
As you can see, the columns A, B, C, and D are automatically tab completed. E is there as well; the rest of the attributes
have been truncated for brevity.
6.2 Viewing Data
See the Basics section
See the top & bottom rows of the frame
In [14]: df.head()
Out[14]:
A
B
C
D
2013-01-01 0.469112 -0.282863 -1.509059 -1.135632
2013-01-02 1.212112 -0.173215 0.119209 -1.044236
2013-01-03 -0.861849 -2.104569 -0.494929 1.071804
2013-01-04 0.721555 -0.706771 -1.039575 0.271860
2013-01-05 -0.424972 0.567020 0.276232 -1.087401
In [15]: df.tail(3)
Out[15]:
A
B
C
D
2013-01-04 0.721555 -0.706771 -1.039575 0.271860
2013-01-05 -0.424972 0.567020 0.276232 -1.087401
2013-01-06 -0.673690 0.113648 -1.478427 0.524988
Display the index, columns, and the underlying numpy data
In [16]: df.index
Out[16]:
DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04',
'2013-01-05', '2013-01-06'],
dtype='datetime64[ns]', freq='D', tz=None)
In [17]: df.columns
Out[17]: Index([u'A', u'B', u'C', u'D'], dtype='object')
In [18]: df.values
Out[18]:
array([[ 0.4691, -0.2829,
[ 1.2121, -0.1732,
[-0.8618, -2.1046,
[ 0.7216, -0.7068,
[-0.425 , 0.567 ,
[-0.6737, 0.1136,
-1.5091, -1.1356],
0.1192, -1.0442],
-0.4949, 1.0718],
-1.0396, 0.2719],
0.2762, -1.0874],
-1.4784, 0.525 ]])
Describe shows a quick statistic summary of your data
In [19]: df.describe()
Out[19]:
A
B
count 6.000000 6.000000
6.2. Viewing Data
C
6.000000
D
6.000000
229
pandas: powerful Python data analysis toolkit, Release 0.16.1
mean
std
min
25%
50%
75%
max
0.073711
0.843157
-0.861849
-0.611510
0.022070
0.658444
1.212112
-0.431125
0.922818
-2.104569
-0.600794
-0.228039
0.041933
0.567020
-0.687758
0.779887
-1.509059
-1.368714
-0.767252
-0.034326
0.276232
-0.233103
0.973118
-1.135632
-1.076610
-0.386188
0.461706
1.071804
Transposing your data
In [20]: df.T
Out[20]:
2013-01-01
A
0.469112
B
-0.282863
C
-1.509059
D
-1.135632
2013-01-02
1.212112
-0.173215
0.119209
-1.044236
2013-01-03
-0.861849
-2.104569
-0.494929
1.071804
2013-01-04
0.721555
-0.706771
-1.039575
0.271860
2013-01-05
-0.424972
0.567020
0.276232
-1.087401
2013-01-06
-0.673690
0.113648
-1.478427
0.524988
Sorting by an axis
In [21]: df.sort_index(axis=1,
Out[21]:
D
C
2013-01-01 -1.135632 -1.509059
2013-01-02 -1.044236 0.119209
2013-01-03 1.071804 -0.494929
2013-01-04 0.271860 -1.039575
2013-01-05 -1.087401 0.276232
2013-01-06 0.524988 -1.478427
ascending=False)
B
A
-0.282863 0.469112
-0.173215 1.212112
-2.104569 -0.861849
-0.706771 0.721555
0.567020 -0.424972
0.113648 -0.673690
Sorting by values
In [22]: df.sort(columns='B')
Out[22]:
A
B
2013-01-03 -0.861849 -2.104569
2013-01-04 0.721555 -0.706771
2013-01-01 0.469112 -0.282863
2013-01-02 1.212112 -0.173215
2013-01-06 -0.673690 0.113648
2013-01-05 -0.424972 0.567020
C
D
-0.494929 1.071804
-1.039575 0.271860
-1.509059 -1.135632
0.119209 -1.044236
-1.478427 0.524988
0.276232 -1.087401
6.3 Selection
Note: While standard Python / Numpy expressions for selecting and setting are intuitive and come in handy for
interactive work, for production code, we recommend the optimized pandas data access methods, .at, .iat, .loc,
.iloc and .ix.
See the indexing documentation Indexing and Selecing Data and MultiIndex / Advanced Indexing
6.3.1 Getting
Selecting a single column, which yields a Series, equivalent to df.A
230
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [23]: df['A']
Out[23]:
2013-01-01
0.469112
2013-01-02
1.212112
2013-01-03
-0.861849
2013-01-04
0.721555
2013-01-05
-0.424972
2013-01-06
-0.673690
Freq: D, Name: A, dtype: float64
Selecting via [], which slices the rows.
In [24]: df[0:3]
Out[24]:
A
B
C
D
2013-01-01 0.469112 -0.282863 -1.509059 -1.135632
2013-01-02 1.212112 -0.173215 0.119209 -1.044236
2013-01-03 -0.861849 -2.104569 -0.494929 1.071804
In [25]: df['20130102':'20130104']
Out[25]:
A
B
C
D
2013-01-02 1.212112 -0.173215 0.119209 -1.044236
2013-01-03 -0.861849 -2.104569 -0.494929 1.071804
2013-01-04 0.721555 -0.706771 -1.039575 0.271860
6.3.2 Selection by Label
See more in Selection by Label
For getting a cross section using a label
In [26]: df.loc[dates[0]]
Out[26]:
A
0.469112
B
-0.282863
C
-1.509059
D
-1.135632
Name: 2013-01-01 00:00:00, dtype: float64
Selecting on a multi-axis by label
In [27]: df.loc[:,['A','B']]
Out[27]:
A
B
2013-01-01 0.469112 -0.282863
2013-01-02 1.212112 -0.173215
2013-01-03 -0.861849 -2.104569
2013-01-04 0.721555 -0.706771
2013-01-05 -0.424972 0.567020
2013-01-06 -0.673690 0.113648
Showing label slicing, both endpoints are included
In [28]: df.loc['20130102':'20130104',['A','B']]
Out[28]:
A
B
2013-01-02 1.212112 -0.173215
6.3. Selection
231
pandas: powerful Python data analysis toolkit, Release 0.16.1
2013-01-03 -0.861849 -2.104569
2013-01-04 0.721555 -0.706771
Reduction in the dimensions of the returned object
In [29]: df.loc['20130102',['A','B']]
Out[29]:
A
1.212112
B
-0.173215
Name: 2013-01-02 00:00:00, dtype: float64
For getting a scalar value
In [30]: df.loc[dates[0],'A']
Out[30]: 0.46911229990718628
For getting fast access to a scalar (equiv to the prior method)
In [31]: df.at[dates[0],'A']
Out[31]: 0.46911229990718628
6.3.3 Selection by Position
See more in Selection by Position
Select via the position of the passed integers
In [32]: df.iloc[3]
Out[32]:
A
0.721555
B
-0.706771
C
-1.039575
D
0.271860
Name: 2013-01-04 00:00:00, dtype: float64
By integer slices, acting similar to numpy/python
In [33]: df.iloc[3:5,0:2]
Out[33]:
A
B
2013-01-04 0.721555 -0.706771
2013-01-05 -0.424972 0.567020
By lists of integer position locations, similar to the numpy/python style
In [34]: df.iloc[[1,2,4],[0,2]]
Out[34]:
A
C
2013-01-02 1.212112 0.119209
2013-01-03 -0.861849 -0.494929
2013-01-05 -0.424972 0.276232
For slicing rows explicitly
In [35]: df.iloc[1:3,:]
Out[35]:
A
B
C
D
2013-01-02 1.212112 -0.173215 0.119209 -1.044236
2013-01-03 -0.861849 -2.104569 -0.494929 1.071804
232
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
For slicing columns explicitly
In [36]: df.iloc[:,1:3]
Out[36]:
B
C
2013-01-01 -0.282863 -1.509059
2013-01-02 -0.173215 0.119209
2013-01-03 -2.104569 -0.494929
2013-01-04 -0.706771 -1.039575
2013-01-05 0.567020 0.276232
2013-01-06 0.113648 -1.478427
For getting a value explicitly
In [37]: df.iloc[1,1]
Out[37]: -0.17321464905330858
For getting fast access to a scalar (equiv to the prior method)
In [38]: df.iat[1,1]
Out[38]: -0.17321464905330858
6.3.4 Boolean Indexing
Using a single column’s values to select data.
In [39]: df[df.A > 0]
Out[39]:
A
B
C
D
2013-01-01 0.469112 -0.282863 -1.509059 -1.135632
2013-01-02 1.212112 -0.173215 0.119209 -1.044236
2013-01-04 0.721555 -0.706771 -1.039575 0.271860
A where operation for getting.
In [40]: df[df > 0]
Out[40]:
2013-01-01
2013-01-02
2013-01-03
2013-01-04
2013-01-05
2013-01-06
A
0.469112
1.212112
NaN
0.721555
NaN
NaN
B
NaN
NaN
NaN
NaN
0.567020
0.113648
C
NaN
0.119209
NaN
NaN
0.276232
NaN
D
NaN
NaN
1.071804
0.271860
NaN
0.524988
Using the isin() method for filtering:
In [41]: df2 = df.copy()
In [42]: df2['E']=['one', 'one','two','three','four','three']
In [43]: df2
Out[43]:
A
B
C
D
2013-01-01 0.469112 -0.282863 -1.509059 -1.135632
2013-01-02 1.212112 -0.173215 0.119209 -1.044236
2013-01-03 -0.861849 -2.104569 -0.494929 1.071804
2013-01-04 0.721555 -0.706771 -1.039575 0.271860
2013-01-05 -0.424972 0.567020 0.276232 -1.087401
6.3. Selection
E
one
one
two
three
four
233
pandas: powerful Python data analysis toolkit, Release 0.16.1
2013-01-06 -0.673690
0.113648 -1.478427
0.524988
In [44]: df2[df2['E'].isin(['two','four'])]
Out[44]:
A
B
C
D
2013-01-03 -0.861849 -2.104569 -0.494929 1.071804
2013-01-05 -0.424972 0.567020 0.276232 -1.087401
three
E
two
four
6.3.5 Setting
Setting a new column automatically aligns the data by the indexes
In [45]: s1 = pd.Series([1,2,3,4,5,6],index=pd.date_range('20130102',periods=6))
In [46]: s1
Out[46]:
2013-01-02
1
2013-01-03
2
2013-01-04
3
2013-01-05
4
2013-01-06
5
2013-01-07
6
Freq: D, dtype: int64
In [47]: df['F'] = s1
Setting values by label
In [48]: df.at[dates[0],'A'] = 0
Setting values by position
In [49]: df.iat[0,1] = 0
Setting by assigning with a numpy array
In [50]: df.loc[:,'D'] = np.array([5] * len(df))
The result of the prior setting operations
In [51]: df
Out[51]:
A
B
C
2013-01-01 0.000000 0.000000 -1.509059
2013-01-02 1.212112 -0.173215 0.119209
2013-01-03 -0.861849 -2.104569 -0.494929
2013-01-04 0.721555 -0.706771 -1.039575
2013-01-05 -0.424972 0.567020 0.276232
2013-01-06 -0.673690 0.113648 -1.478427
D
F
5 NaN
5
1
5
2
5
3
5
4
5
5
A where operation with setting.
In [52]: df2 = df.copy()
In [53]: df2[df2 > 0] = -df2
In [54]: df2
Out[54]:
A
234
B
C
D
F
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
2013-01-01
2013-01-02
2013-01-03
2013-01-04
2013-01-05
2013-01-06
0.000000
-1.212112
-0.861849
-0.721555
-0.424972
-0.673690
0.000000
-0.173215
-2.104569
-0.706771
-0.567020
-0.113648
-1.509059
-0.119209
-0.494929
-1.039575
-0.276232
-1.478427
-5 NaN
-5 -1
-5 -2
-5 -3
-5 -4
-5 -5
6.4 Missing Data
pandas primarily uses the value np.nan to represent missing data. It is by default not included in computations. See
the Missing Data section
Reindexing allows you to change/add/delete the index on a specified axis. This returns a copy of the data.
In [55]: df1 = df.reindex(index=dates[0:4],columns=list(df.columns) + ['E'])
In [56]: df1.loc[dates[0]:dates[1],'E'] = 1
In [57]: df1
Out[57]:
A
B
C
2013-01-01 0.000000 0.000000 -1.509059
2013-01-02 1.212112 -0.173215 0.119209
2013-01-03 -0.861849 -2.104569 -0.494929
2013-01-04 0.721555 -0.706771 -1.039575
D
F
E
5 NaN
1
5
1
1
5
2 NaN
5
3 NaN
To drop any rows that have missing data.
In [58]: df1.dropna(how='any')
Out[58]:
A
B
2013-01-02 1.212112 -0.173215
C
0.119209
D
5
F
1
E
1
In [59]: df1.fillna(value=5)
Out[59]:
A
B
C
2013-01-01 0.000000 0.000000 -1.509059
2013-01-02 1.212112 -0.173215 0.119209
2013-01-03 -0.861849 -2.104569 -0.494929
2013-01-04 0.721555 -0.706771 -1.039575
D
5
5
5
5
F
5
1
2
3
E
1
1
5
5
Filling missing data
To get the boolean mask where values are nan
In [60]: pd.isnull(df1)
Out[60]:
A
B
2013-01-01 False False
2013-01-02 False False
2013-01-03 False False
2013-01-04 False False
C
False
False
False
False
D
False
False
False
False
F
True
False
False
False
E
False
False
True
True
6.5 Operations
See the Basic section on Binary Ops
6.4. Missing Data
235
pandas: powerful Python data analysis toolkit, Release 0.16.1
6.5.1 Stats
Operations in general exclude missing data.
Performing a descriptive statistic
In [61]: df.mean()
Out[61]:
A
-0.004474
B
-0.383981
C
-0.687758
D
5.000000
F
3.000000
dtype: float64
Same operation on the other axis
In [62]: df.mean(1)
Out[62]:
2013-01-01
0.872735
2013-01-02
1.431621
2013-01-03
0.707731
2013-01-04
1.395042
2013-01-05
1.883656
2013-01-06
1.592306
Freq: D, dtype: float64
Operating with objects that have different dimensionality and need alignment. In addition, pandas automatically
broadcasts along the specified dimension.
In [63]: s = pd.Series([1,3,5,np.nan,6,8],index=dates).shift(2)
In [64]: s
Out[64]:
2013-01-01
NaN
2013-01-02
NaN
2013-01-03
1
2013-01-04
3
2013-01-05
5
2013-01-06
NaN
Freq: D, dtype: float64
In [65]: df.sub(s,axis='index')
Out[65]:
A
B
C
D
F
2013-01-01
NaN
NaN
NaN NaN NaN
2013-01-02
NaN
NaN
NaN NaN NaN
2013-01-03 -1.861849 -3.104569 -1.494929
4
1
2013-01-04 -2.278445 -3.706771 -4.039575
2
0
2013-01-05 -5.424972 -4.432980 -4.723768
0 -1
2013-01-06
NaN
NaN
NaN NaN NaN
6.5.2 Apply
Applying functions to the data
In [66]: df.apply(np.cumsum)
Out[66]:
236
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
A
2013-01-01 0.000000
2013-01-02 1.212112
2013-01-03 0.350263
2013-01-04 1.071818
2013-01-05 0.646846
2013-01-06 -0.026844
B
0.000000
-0.173215
-2.277784
-2.984555
-2.417535
-2.303886
C
-1.509059
-1.389850
-1.884779
-2.924354
-2.648122
-4.126549
D
F
5 NaN
10
1
15
3
20
6
25 10
30 15
In [67]: df.apply(lambda x: x.max() - x.min())
Out[67]:
A
2.073961
B
2.671590
C
1.785291
D
0.000000
F
4.000000
dtype: float64
6.5.3 Histogramming
See more at Histogramming and Discretization
In [68]: s = pd.Series(np.random.randint(0,7,size=10))
In [69]: s
Out[69]:
0
4
1
2
2
1
3
2
4
6
5
4
6
4
7
6
8
4
9
4
dtype: int32
In [70]: s.value_counts()
Out[70]:
4
5
6
2
2
2
1
1
dtype: int64
6.5.4 String Methods
Series is equipped with a set of string processing methods in the str attribute that make it easy to operate on each
element of the array, as in the code snippet below. Note that pattern-matching in str generally uses regular expressions
by default (and in some cases always uses them). See more at Vectorized String Methods.
In [71]: s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])
In [72]: s.str.lower()
Out[72]:
6.5. Operations
237
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
a
1
b
2
c
3
aaba
4
baca
5
NaN
6
caba
7
dog
8
cat
dtype: object
6.6 Merge
6.6.1 Concat
pandas provides various facilities for easily combining together Series, DataFrame, and Panel objects with various
kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations.
See the Merging section
Concatenating pandas objects together with concat():
In [73]: df = pd.DataFrame(np.random.randn(10, 4))
In [74]: df
Out[74]:
0
0 -0.548702
1 1.637550
2 -0.263952
3 -0.709661
4 -0.919854
5 0.290213
6 -1.131345
7 -0.932132
8 -0.575247
9 1.193555
1
1.467327
-1.217659
0.991460
1.669052
-0.042379
0.495767
-0.089329
1.956030
0.254161
-0.077118
2
-1.015962
-0.291519
-0.919069
1.037882
1.247642
0.362949
0.337863
0.017587
-1.143704
-0.408530
3
-0.483075
-1.745505
0.266046
-1.705775
-0.009920
1.548106
-0.945867
-0.016692
0.215897
-0.862495
# break it into pieces
In [75]: pieces = [df[:3], df[3:7], df[7:]]
In [76]: pd.concat(pieces)
Out[76]:
0
1
2
0 -0.548702 1.467327 -1.015962
1 1.637550 -1.217659 -0.291519
2 -0.263952 0.991460 -0.919069
3 -0.709661 1.669052 1.037882
4 -0.919854 -0.042379 1.247642
5 0.290213 0.495767 0.362949
6 -1.131345 -0.089329 0.337863
7 -0.932132 1.956030 0.017587
8 -0.575247 0.254161 -1.143704
9 1.193555 -0.077118 -0.408530
238
3
-0.483075
-1.745505
0.266046
-1.705775
-0.009920
1.548106
-0.945867
-0.016692
0.215897
-0.862495
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
6.6.2 Join
SQL style merges. See the Database style joining
In [77]: left = pd.DataFrame({'key': ['foo', 'foo'], 'lval': [1, 2]})
In [78]: right = pd.DataFrame({'key': ['foo', 'foo'], 'rval': [4, 5]})
In [79]: left
Out[79]:
key lval
0 foo
1
1 foo
2
In [80]: right
Out[80]:
key rval
0 foo
4
1 foo
5
In [81]: pd.merge(left, right, on='key')
Out[81]:
key lval rval
0 foo
1
4
1 foo
1
5
2 foo
2
4
3 foo
2
5
6.6.3 Append
Append rows to a dataframe. See the Appending
In [82]: df = pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'])
In [83]: df
Out[83]:
A
B
C
D
0 1.346061 1.511763 1.627081 -0.990582
1 -0.441652 1.211526 0.268520 0.024580
2 -1.577585 0.396823 -0.105381 -0.532532
3 1.453749 1.208843 -0.080952 -0.264610
4 -0.727965 -0.589346 0.339969 -0.693205
5 -0.339355 0.593616 0.884345 1.591431
6 0.141809 0.220390 0.435589 0.192451
7 -0.096701 0.803351 1.715071 -0.708758
In [84]: s = df.iloc[3]
In [85]: df.append(s, ignore_index=True)
Out[85]:
A
B
C
D
0 1.346061 1.511763 1.627081 -0.990582
1 -0.441652 1.211526 0.268520 0.024580
2 -1.577585 0.396823 -0.105381 -0.532532
3 1.453749 1.208843 -0.080952 -0.264610
4 -0.727965 -0.589346 0.339969 -0.693205
5 -0.339355 0.593616 0.884345 1.591431
6.6. Merge
239
pandas: powerful Python data analysis toolkit, Release 0.16.1
6 0.141809
7 -0.096701
8 1.453749
0.220390 0.435589 0.192451
0.803351 1.715071 -0.708758
1.208843 -0.080952 -0.264610
6.7 Grouping
By “group by” we are referring to a process involving one or more of the following steps
• Splitting the data into groups based on some criteria
• Applying a function to each group independently
• Combining the results into a data structure
See the Grouping section
In [86]: df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
....:
'foo', 'bar', 'foo', 'foo'],
....:
'B' : ['one', 'one', 'two', 'three',
....:
'two', 'two', 'one', 'three'],
....:
'C' : np.random.randn(8),
....:
'D' : np.random.randn(8)})
....:
In [87]: df
Out[87]:
A
B
0 foo
one
1 bar
one
2 foo
two
3 bar three
4 foo
two
5 bar
two
6 foo
one
7 foo three
C
-1.202872
-1.814470
1.018601
-0.595447
1.395433
-0.392670
0.007207
1.928123
D
-0.055224
2.395985
1.552825
0.166599
0.047609
-0.136473
-0.561757
-1.623033
Grouping and then applying a function sum to the resulting groups.
In [88]: df.groupby('A').sum()
Out[88]:
C
D
A
bar -2.802588 2.42611
foo 3.146492 -0.63958
Grouping by multiple columns forms a hierarchical index, which we then apply the function.
In [89]: df.groupby(['A','B']).sum()
Out[89]:
C
D
A
B
bar one
-1.814470 2.395985
three -0.595447 0.166599
two
-0.392670 -0.136473
foo one
-1.195665 -0.616981
three 1.928123 -1.623033
two
2.414034 1.600434
240
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
6.8 Reshaping
See the sections on Hierarchical Indexing and Reshaping.
6.8.1 Stack
In [90]: tuples = list(zip(*[['bar',
....:
'foo',
....:
['one',
....:
'one',
....:
'bar',
'foo',
'two',
'two',
'baz',
'qux',
'one',
'one',
'baz',
'qux'],
'two',
'two']]))
In [91]: index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second'])
In [92]: df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=['A', 'B'])
In [93]: df2 = df[:4]
In [94]: df2
Out[94]:
A
B
first second
bar
one
0.029399 -0.542108
two
0.282696 -0.087302
baz
one
-1.575170 1.771208
two
0.816482 1.100230
The stack() method “compresses” a level in the DataFrame’s columns.
In [95]: stacked = df2.stack()
In [96]: stacked
Out[96]:
first second
bar
one
A
B
two
A
B
baz
one
A
B
two
A
B
dtype: float64
0.029399
-0.542108
0.282696
-0.087302
-1.575170
1.771208
0.816482
1.100230
With a “stacked” DataFrame or Series (having a MultiIndex as the index), the inverse operation of stack() is
unstack(), which by default unstacks the last level:
In [97]: stacked.unstack()
Out[97]:
A
B
first second
bar
one
0.029399 -0.542108
two
0.282696 -0.087302
baz
one
-1.575170 1.771208
two
0.816482 1.100230
In [98]: stacked.unstack(1)
6.8. Reshaping
241
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[98]:
second
one
two
first
bar
A 0.029399 0.282696
B -0.542108 -0.087302
baz
A -1.575170 0.816482
B 1.771208 1.100230
In [99]: stacked.unstack(0)
Out[99]:
first
bar
baz
second
one
A 0.029399 -1.575170
B -0.542108 1.771208
two
A 0.282696 0.816482
B -0.087302 1.100230
6.8.2 Pivot Tables
See the section on Pivot Tables.
In [100]: df = pd.DataFrame({'A' : ['one', 'one', 'two', 'three'] * 3,
.....:
'B' : ['A', 'B', 'C'] * 4,
.....:
'C' : ['foo', 'foo', 'foo', 'bar', 'bar', 'bar'] * 2,
.....:
'D' : np.random.randn(12),
.....:
'E' : np.random.randn(12)})
.....:
In [101]: df
Out[101]:
A B
0
one A
1
one B
2
two C
3
three A
4
one B
5
one C
6
two A
7
three B
8
one C
9
one A
10
two B
11 three C
C
foo
foo
foo
bar
bar
bar
foo
foo
foo
bar
bar
bar
D
1.418757
-1.879024
0.536826
1.006160
-0.029716
-1.146178
0.100900
-1.035018
0.314665
-0.773723
-1.170653
0.648740
E
-0.179666
1.291836
-0.009614
0.392149
0.264599
-0.057409
-1.425638
1.024098
-0.106062
1.824375
0.595974
1.167115
We can produce pivot tables from this data very easily:
In [102]: pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'])
Out[102]:
C
bar
foo
A
B
one
A -0.773723 1.418757
B -0.029716 -1.879024
C -1.146178 0.314665
three A 1.006160
NaN
B
NaN -1.035018
C 0.648740
NaN
two
A
NaN 0.100900
242
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
B -1.170653
C
NaN
NaN
0.536826
6.9 Time Series
pandas has simple, powerful, and efficient functionality for performing resampling operations during frequency conversion (e.g., converting secondly data into 5-minutely data). This is extremely common in, but not limited to, financial
applications. See the Time Series section
In [103]: rng = pd.date_range('1/1/2012', periods=100, freq='S')
In [104]: ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)
In [105]: ts.resample('5Min', how='sum')
Out[105]:
2012-01-01
25083
Freq: 5T, dtype: int32
Time zone representation
In [106]: rng = pd.date_range('3/6/2012 00:00', periods=5, freq='D')
In [107]: ts = pd.Series(np.random.randn(len(rng)), rng)
In [108]: ts
Out[108]:
2012-03-06
0.464000
2012-03-07
0.227371
2012-03-08
-0.496922
2012-03-09
0.306389
2012-03-10
-2.290613
Freq: D, dtype: float64
In [109]: ts_utc = ts.tz_localize('UTC')
In [110]: ts_utc
Out[110]:
2012-03-06 00:00:00+00:00
2012-03-07 00:00:00+00:00
2012-03-08 00:00:00+00:00
2012-03-09 00:00:00+00:00
2012-03-10 00:00:00+00:00
Freq: D, dtype: float64
0.464000
0.227371
-0.496922
0.306389
-2.290613
Convert to another time zone
In [111]: ts_utc.tz_convert('US/Eastern')
Out[111]:
2012-03-05 19:00:00-05:00
0.464000
2012-03-06 19:00:00-05:00
0.227371
2012-03-07 19:00:00-05:00
-0.496922
2012-03-08 19:00:00-05:00
0.306389
2012-03-09 19:00:00-05:00
-2.290613
Freq: D, dtype: float64
Converting between time span representations
6.9. Time Series
243
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [112]: rng = pd.date_range('1/1/2012', periods=5, freq='M')
In [113]: ts = pd.Series(np.random.randn(len(rng)), index=rng)
In [114]: ts
Out[114]:
2012-01-31
-1.134623
2012-02-29
-1.561819
2012-03-31
-0.260838
2012-04-30
0.281957
2012-05-31
1.523962
Freq: M, dtype: float64
In [115]: ps = ts.to_period()
In [116]: ps
Out[116]:
2012-01
-1.134623
2012-02
-1.561819
2012-03
-0.260838
2012-04
0.281957
2012-05
1.523962
Freq: M, dtype: float64
In [117]: ps.to_timestamp()
Out[117]:
2012-01-01
-1.134623
2012-02-01
-1.561819
2012-03-01
-0.260838
2012-04-01
0.281957
2012-05-01
1.523962
Freq: MS, dtype: float64
Converting between period and timestamp enables some convenient arithmetic functions to be used. In the following
example, we convert a quarterly frequency with year ending in November to 9am of the end of the month following
the quarter end:
In [118]: prng = pd.period_range('1990Q1', '2000Q4', freq='Q-NOV')
In [119]: ts = pd.Series(np.random.randn(len(prng)), prng)
In [120]: ts.index = (prng.asfreq('M', 'e') + 1).asfreq('H', 's') + 9
In [121]: ts.head()
Out[121]:
1990-03-01 09:00
-0.902937
1990-06-01 09:00
0.068159
1990-09-01 09:00
-0.057873
1990-12-01 09:00
-0.368204
1991-03-01 09:00
-1.144073
Freq: H, dtype: float64
6.10 Categoricals
Since version 0.15, pandas can include categorical data in a DataFrame. For full docs, see the categorical introduction and the API documentation.
244
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [122]: df = pd.DataFrame({"id":[1,2,3,4,5,6], "raw_grade":['a', 'b', 'b', 'a', 'a', 'e']})
Convert the raw grades to a categorical data type.
In [123]: df["grade"] = df["raw_grade"].astype("category")
In [124]: df["grade"]
Out[124]:
0
a
1
b
2
b
3
a
4
a
5
e
Name: grade, dtype: category
Categories (3, object): [a, b, e]
Rename the categories to more meaningful names (assigning to Series.cat.categories is inplace!)
In [125]: df["grade"].cat.categories = ["very good", "good", "very bad"]
Reorder the categories and simultaneously add the missing categories (methods under Series .cat return a new
Series per default).
In [126]: df["grade"] = df["grade"].cat.set_categories(["very bad", "bad", "medium", "good", "very go
In [127]: df["grade"]
Out[127]:
0
very good
1
good
2
good
3
very good
4
very good
5
very bad
Name: grade, dtype: category
Categories (5, object): [very bad, bad, medium, good, very good]
Sorting is per order in the categories, not lexical order.
In [128]: df.sort("grade")
Out[128]:
id raw_grade
grade
5
6
e
very bad
1
2
b
good
2
3
b
good
0
1
a very good
3
4
a very good
4
5
a very good
Grouping by a categorical column shows also empty categories.
In [129]: df.groupby("grade").size()
Out[129]:
grade
very bad
1
bad
NaN
medium
NaN
good
2
very good
3
dtype: float64
6.10. Categoricals
245
pandas: powerful Python data analysis toolkit, Release 0.16.1
6.11 Plotting
Plotting docs.
In [130]: ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
In [131]: ts = ts.cumsum()
In [132]: ts.plot()
Out[132]:
On DataFrame, plot() is a convenience to plot all of the columns with labels:
In [133]: df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,
.....:
columns=['A', 'B', 'C', 'D'])
.....:
In [134]: df = df.cumsum()
In [135]: plt.figure(); df.plot(); plt.legend(loc='best')
Out[135]:
246
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
6.12 Getting Data In/Out
6.12.1 CSV
Writing to a csv file
In [136]: df.to_csv('foo.csv')
Reading from a csv file
In [137]: pd.read_csv('foo.csv')
Out[137]:
Unnamed: 0
A
B
C
0
2000-01-01
0.266457 -0.399641 -0.219582
1
2000-01-02 -1.170732 -0.345873 1.653061
2
2000-01-03 -1.734933
0.530468 2.060811
3
2000-01-04 -1.555121
1.452620 0.239859
4
2000-01-05
0.578117
0.511371 0.103552
5
2000-01-06
0.478344
0.449933 -0.741620
6
2000-01-07
1.235339 -0.091757 -1.543861
..
...
...
...
...
6.12. Getting Data In/Out
D
1.186860
-0.282953
-0.515536
-1.156896
-2.428202
-1.962409
-1.084753
...
247
pandas: powerful Python data analysis toolkit, Release 0.16.1
993
994
995
996
997
998
999
2002-09-20
2002-09-21
2002-09-22
2002-09-23
2002-09-24
2002-09-25
2002-09-26
-10.628548 -9.153563
-10.390377 -8.727491
-8.985362 -8.485624
-9.558560 -8.781216
-9.902058 -9.340490
-10.216020 -9.480682
-11.856774 -10.671012
-7.883146
-6.399645
-4.669462
-4.499815
-4.386639
-3.933802
-3.216025
28.313940
30.914107
31.367740
30.518439
30.105593
29.758560
29.369368
[1000 rows x 5 columns]
6.12.2 HDF5
Reading and writing to HDFStores
Writing to a HDF5 Store
In [138]: df.to_hdf('foo.h5','df')
Reading from a HDF5 Store
In [139]: pd.read_hdf('foo.h5','df')
Out[139]:
A
B
C
2000-01-01
0.266457 -0.399641 -0.219582
2000-01-02 -1.170732 -0.345873 1.653061
2000-01-03 -1.734933
0.530468 2.060811
2000-01-04 -1.555121
1.452620 0.239859
2000-01-05
0.578117
0.511371 0.103552
2000-01-06
0.478344
0.449933 -0.741620
2000-01-07
1.235339 -0.091757 -1.543861
...
...
...
...
2002-09-20 -10.628548 -9.153563 -7.883146
2002-09-21 -10.390377 -8.727491 -6.399645
2002-09-22 -8.985362 -8.485624 -4.669462
2002-09-23 -9.558560 -8.781216 -4.499815
2002-09-24 -9.902058 -9.340490 -4.386639
2002-09-25 -10.216020 -9.480682 -3.933802
2002-09-26 -11.856774 -10.671012 -3.216025
D
1.186860
-0.282953
-0.515536
-1.156896
-2.428202
-1.962409
-1.084753
...
28.313940
30.914107
31.367740
30.518439
30.105593
29.758560
29.369368
[1000 rows x 4 columns]
6.12.3 Excel
Reading and writing to MS Excel
Writing to an excel file
In [140]: df.to_excel('foo.xlsx', sheet_name='Sheet1')
Reading from an excel file
In [141]: pd.read_excel('foo.xlsx', 'Sheet1', index_col=None, na_values=['NA'])
Out[141]:
A
B
C
D
2000-01-01
0.266457 -0.399641 -0.219582
1.186860
2000-01-02 -1.170732 -0.345873 1.653061 -0.282953
2000-01-03 -1.734933
0.530468 2.060811 -0.515536
248
Chapter 6. 10 Minutes to pandas
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-04
2000-01-05
2000-01-06
2000-01-07
...
2002-09-20
2002-09-21
2002-09-22
2002-09-23
2002-09-24
2002-09-25
2002-09-26
-1.555121
1.452620
0.578117
0.511371
0.478344
0.449933
1.235339 -0.091757
...
...
-10.628548 -9.153563
-10.390377 -8.727491
-8.985362 -8.485624
-9.558560 -8.781216
-9.902058 -9.340490
-10.216020 -9.480682
-11.856774 -10.671012
0.239859
0.103552
-0.741620
-1.543861
...
-7.883146
-6.399645
-4.669462
-4.499815
-4.386639
-3.933802
-3.216025
-1.156896
-2.428202
-1.962409
-1.084753
...
28.313940
30.914107
31.367740
30.518439
30.105593
29.758560
29.369368
[1000 rows x 4 columns]
6.13 Gotchas
If you are trying an operation and you see an exception like:
>>> if pd.Series([False, True, False]):
print("I was true")
Traceback
...
ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().
See Comparisons for an explanation and what to do.
See Gotchas as well.
6.13. Gotchas
249
pandas: powerful Python data analysis toolkit, Release 0.16.1
250
Chapter 6. 10 Minutes to pandas
CHAPTER
SEVEN
TUTORIALS
This is a guide to many pandas tutorials, geared mainly for new users.
7.1 Internal Guides
pandas own 10 Minutes to pandas
More complex recipes are in the Cookbook
7.2 pandas Cookbook
The goal of this cookbook (by Julia Evans) is to give you some concrete examples for getting started with pandas.
These are examples with real-world data, and all the bugs and weirdness that that entails.
Here are links to the v0.1 release. For an up-to-date table of contents, see the pandas-cookbook GitHub repository. To
run the examples in this tutorial, you’ll need to clone the GitHub repository and get IPython Notebook running. See
How to use this cookbook.
• A quick tour of the IPython Notebook: Shows off IPython’s awesome tab completion and magic functions.
• Chapter 1: Reading your data into pandas is pretty much the easiest thing. Even when the encoding is wrong!
• Chapter 2: It’s not totally obvious how to select data from a pandas dataframe. Here we explain the basics (how
to take slices and get columns)
• Chapter 3: Here we get into serious slicing and dicing and learn how to filter dataframes in complicated ways,
really fast.
• Chapter 4: Groupby/aggregate is seriously my favorite thing about pandas and I use it all the time. You should
probably read this.
• Chapter 5: Here you get to find out if it’s cold in Montreal in the winter (spoiler: yes). Web scraping with pandas
is fun! Here we combine dataframes.
• Chapter 6: Strings with pandas are great. It has all these vectorized string operations and they’re the best. We
will turn a bunch of strings containing “Snow” into vectors of numbers in a trice.
• Chapter 7: Cleaning up messy data is never a joy, but with pandas it’s easier.
• Chapter 8: Parsing Unix timestamps is confusing at first but it turns out to be really easy.
251
pandas: powerful Python data analysis toolkit, Release 0.16.1
7.3 Lessons for New pandas Users
For more resources, please visit the main repository.
• 01 - Lesson: - Importing libraries - Creating data sets - Creating data frames - Reading from CSV - Exporting
to CSV - Finding maximums - Plotting data
• 02 - Lesson: - Reading from TXT - Exporting to TXT - Selecting top/bottom records - Descriptive statistics Grouping/sorting data
• 03 - Lesson: - Creating functions - Reading from EXCEL - Exporting to EXCEL - Outliers - Lambda functions
- Slice and dice data
• 04 - Lesson: - Adding/deleting columns - Index operations
• 05 - Lesson: - Stack/Unstack/Transpose functions
• 06 - Lesson: - GroupBy function
• 07 - Lesson: - Ways to calculate outliers
• 08 - Lesson: - Read from Microsoft SQL databases
• 09 - Lesson: - Export to CSV/EXCEL/TXT
• 10 - Lesson: - Converting between different kinds of formats
• 11 - Lesson: - Combining data from various sources
7.4 Practical data analysis with Python
This guide is a comprehensive introduction to the data analysis process using the Python data ecosystem and an
interesting open dataset. There are four sections covering selected topics as follows:
• Munging Data
• Aggregating Data
• Visualizing Data
• Time Series
7.5 Excel charts with pandas, vincent and xlsxwriter
• Using Pandas and XlsxWriter to create Excel charts
7.6 Various Tutorials
• Wes McKinney’s (pandas BDFL) blog
• Statistical analysis made easy in Python with SciPy and pandas DataFrames, by Randal Olson
• Statistical Data Analysis in Python, tutorial videos, by Christopher Fonnesbeck from SciPy 2013
• Financial analysis in python, by Thomas Wiecki
• Intro to pandas data structures, by Greg Reda
252
Chapter 7. Tutorials
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Pandas and Python: Top 10, by Manish Amde
• Pandas Tutorial, by Mikhail Semeniuk
7.6. Various Tutorials
253
pandas: powerful Python data analysis toolkit, Release 0.16.1
254
Chapter 7. Tutorials
CHAPTER
EIGHT
COOKBOOK
This is a repository for short and sweet examples and links for useful pandas recipes. We encourage users to add to
this documentation.
Adding interesting links and/or inline examples to this section is a great First Pull Request.
Simplified, condensed, new-user friendly, in-line examples have been inserted where possible to augment the StackOverflow and GitHub links. Many of the links contain expanded information, above what the in-line examples offer.
Pandas (pd) and Numpy (np) are the only two abbreviated imported modules. The rest are kept explicitly imported for
newer users.
These examples are written for python 3.4. Minor tweaks might be necessary for earlier python versions.
8.1 Idioms
These are some neat pandas idioms
if-then/if-then-else on one column, and assignment to another one or more columns:
In [1]: df =
...:
...:
Out[1]:
AAA BBB
0
4
10
1
5
20
2
6
30
3
7
40
pd.DataFrame(
{'AAA' : [4,5,6,7], 'BBB' : [10,20,30,40],'CCC' : [100,50,-30,-50]}); df
CCC
100
50
-30
-50
8.1.1 if-then...
An if-then on one column
In [2]: df.ix[df.AAA >= 5,'BBB'] = -1; df
Out[2]:
AAA BBB CCC
0
4
10 100
1
5
-1
50
2
6
-1 -30
3
7
-1 -50
An if-then with assignment to 2 columns:
255
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [3]:
Out[3]:
AAA
0
4
1
5
2
6
3
7
df.ix[df.AAA >= 5,['BBB','CCC']] = 555; df
BBB
10
555
555
555
CCC
100
555
555
555
Add another line with different logic, to do the -else
In [4]: df.ix[df.AAA < 5,['BBB','CCC']] = 2000; df
Out[4]:
AAA
BBB
CCC
0
4 2000 2000
1
5
555
555
2
6
555
555
3
7
555
555
Or use pandas where after you’ve set up a mask
In [5]: df_mask = pd.DataFrame({'AAA' : [True] * 4, 'BBB' : [False] * 4,'CCC' : [True,False] * 2})
In [6]: df.where(df_mask,-1000)
Out[6]:
AAA
BBB
CCC
0
4 -1000 2000
1
5 -1000 -1000
2
6 -1000
555
3
7 -1000 -1000
if-then-else using numpy’s where()
In [7]: df =
...:
...:
Out[7]:
AAA BBB
0
4
10
1
5
20
2
6
30
3
7
40
pd.DataFrame(
{'AAA' : [4,5,6,7], 'BBB' : [10,20,30,40],'CCC' : [100,50,-30,-50]}); df
CCC
100
50
-30
-50
In [8]: df['logic'] = np.where(df['AAA'] > 5,'high','low'); df
Out[8]:
AAA BBB CCC logic
0
4
10 100
low
1
5
20
50
low
2
6
30 -30 high
3
7
40 -50 high
8.1.2 Splitting
Split a frame with a boolean criterion
In [9]: df = pd.DataFrame(
...:
{'AAA' : [4,5,6,7], 'BBB' : [10,20,30,40],'CCC' : [100,50,-30,-50]}); df
...:
Out[9]:
256
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1
2
3
AAA
4
5
6
7
BBB
10
20
30
40
CCC
100
50
-30
-50
In [10]: dflow = df[df.AAA <= 5]
In [11]: dfhigh = df[df.AAA > 5]
In [12]: dflow; dfhigh
Out[12]:
AAA BBB CCC
2
6
30 -30
3
7
40 -50
8.1.3 Building Criteria
Select with multi-column criteria
In [13]: df = pd.DataFrame(
....:
{'AAA' : [4,5,6,7], 'BBB' : [10,20,30,40],'CCC' : [100,50,-30,-50]}); df
....:
Out[13]:
AAA BBB CCC
0
4
10 100
1
5
20
50
2
6
30 -30
3
7
40 -50
...and (without assignment returns a Series)
In [14]: newseries = df.loc[(df['BBB'] < 25) & (df['CCC'] >= -40), 'AAA']; newseries
Out[14]:
0
4
1
5
Name: AAA, dtype: int64
...or (without assignment returns a Series)
In [15]: newseries = df.loc[(df['BBB'] > 25) | (df['CCC'] >= -40), 'AAA']; newseries;
...or (with assignment modifies the DataFrame.)
In [16]: df.loc[(df['BBB'] > 25) | (df['CCC'] >= 75), 'AAA'] = 0.1; df
Out[16]:
AAA BBB CCC
0 0.1
10 100
1 5.0
20
50
2 0.1
30 -30
3 0.1
40 -50
Select rows with data closest to certain value using argsort
In [17]: df = pd.DataFrame(
....:
{'AAA' : [4,5,6,7], 'BBB' : [10,20,30,40],'CCC' : [100,50,-30,-50]}); df
....:
Out[17]:
8.1. Idioms
257
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1
2
3
AAA
4
5
6
7
BBB
10
20
30
40
CCC
100
50
-30
-50
In [18]: aValue = 43.0
In [19]: df.ix[(df.CCC-aValue).abs().argsort()]
Out[19]:
AAA BBB CCC
1
5
20
50
0
4
10 100
2
6
30 -30
3
7
40 -50
Dynamically reduce a list of criteria using a binary operators
In [20]: df = pd.DataFrame(
....:
{'AAA' : [4,5,6,7], 'BBB' : [10,20,30,40],'CCC' : [100,50,-30,-50]}); df
....:
Out[20]:
AAA BBB CCC
0
4
10 100
1
5
20
50
2
6
30 -30
3
7
40 -50
In [21]: Crit1 = df.AAA <= 5.5
In [22]: Crit2 = df.BBB == 10.0
In [23]: Crit3 = df.CCC > -40.0
One could hard code:
In [24]: AllCrit = Crit1 & Crit2 & Crit3
...Or it can be done with a list of dynamically built criteria
In [25]: CritList = [Crit1,Crit2,Crit3]
In [26]: AllCrit = functools.reduce(lambda x,y: x & y, CritList)
In [27]: df[AllCrit]
Out[27]:
AAA BBB CCC
0
4
10 100
8.2 Selection
8.2.1 DataFrames
The indexing docs.
Using both row labels and value conditionals
258
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [28]: df = pd.DataFrame(
....:
{'AAA' : [4,5,6,7], 'BBB' : [10,20,30,40],'CCC' : [100,50,-30,-50]}); df
....:
Out[28]:
AAA BBB CCC
0
4
10 100
1
5
20
50
2
6
30 -30
3
7
40 -50
In [29]: df[(df.AAA <= 6) & (df.index.isin([0,2,4]))]
Out[29]:
AAA BBB CCC
0
4
10 100
2
6
30 -30
Use loc for label-oriented slicing and iloc positional slicing
In [30]: data = {'AAA' : [4,5,6,7], 'BBB' : [10,20,30,40],'CCC' : [100,50,-30,-50]}
In [31]: df = pd.DataFrame(data=data,index=['foo','bar','boo','kar']); df
Out[31]:
AAA BBB CCC
foo
4
10 100
bar
5
20
50
boo
6
30 -30
kar
7
40 -50
There are 2 explicit slicing methods, with a third general case
1. Positional-oriented (Python slicing style : exclusive of end)
2. Label-oriented (Non-Python slicing style : inclusive of end)
3. General (Either slicing style : depends on if the slice contains labels or positions)
In [32]: df.loc['bar':'kar'] #Label
Out[32]:
AAA BBB CCC
bar
5
20
50
boo
6
30 -30
kar
7
40 -50
#Generic
In [33]: df.ix[0:3] #Same as .iloc[0:3]
Out[33]:
AAA BBB CCC
foo
4
10 100
bar
5
20
50
boo
6
30 -30
In [34]: df.ix['bar':'kar'] #Same as .loc['bar':'kar']
Out[34]:
AAA BBB CCC
bar
5
20
50
boo
6
30 -30
kar
7
40 -50
Ambiguity arises when an index consists of integers with a non-zero start or non-unit increment.
8.2. Selection
259
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [35]: df2 = pd.DataFrame(data=data,index=[1,2,3,4]); #Note index starts at 1.
In [36]: df2.iloc[1:3] #Position-oriented
Out[36]:
AAA BBB CCC
2
5
20
50
3
6
30 -30
In [37]: df2.loc[1:3] #Label-oriented
Out[37]:
AAA BBB CCC
1
4
10 100
2
5
20
50
3
6
30 -30
In [38]: df2.ix[1:3] #General, will mimic loc (label-oriented)
Out[38]:
AAA BBB CCC
1
4
10 100
2
5
20
50
3
6
30 -30
In [39]: df2.ix[0:3] #General, will mimic iloc (position-oriented), as loc[0:3] would raise a KeyErro
Out[39]:
AAA BBB CCC
1
4
10 100
2
5
20
50
3
6
30 -30
Using inverse operator (~) to take the complement of a mask
In [40]: df = pd.DataFrame(
....:
{'AAA' : [4,5,6,7], 'BBB' : [10,20,30,40], 'CCC' : [100,50,-30,-50]}); df
....:
Out[40]:
AAA BBB CCC
0
4
10 100
1
5
20
50
2
6
30 -30
3
7
40 -50
In [41]: df[~((df.AAA <= 6) & (df.index.isin([0,2,4])))]
Out[41]:
AAA BBB CCC
1
5
20
50
3
7
40 -50
8.2.2 Panels
Extend a panel frame by transposing, adding a new dimension, and transposing back to the original dimensions
In [42]: rng = pd.date_range('1/1/2013',periods=100,freq='D')
In [43]: data = np.random.randn(100, 4)
In [44]: cols = ['A','B','C','D']
260
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [45]: df1, df2, df3 = pd.DataFrame(data, rng, cols), pd.DataFrame(data, rng, cols), pd.DataFrame(d
In [46]: pf = pd.Panel({'df1':df1,'df2':df2,'df3':df3});pf
Out[46]:
Dimensions: 3 (items) x 100 (major_axis) x 4 (minor_axis)
Items axis: df1 to df3
Major_axis axis: 2013-01-01 00:00:00 to 2013-04-10 00:00:00
Minor_axis axis: A to D
#Assignment using Transpose (pandas < 0.15)
In [47]: pf = pf.transpose(2,0,1)
In [48]: pf['E'] = pd.DataFrame(data, rng, cols)
In [49]: pf = pf.transpose(1,2,0);pf
Out[49]:
Dimensions: 3 (items) x 100 (major_axis) x 5 (minor_axis)
Items axis: df1 to df3
Major_axis axis: 2013-01-01 00:00:00 to 2013-04-10 00:00:00
Minor_axis axis: A to E
#Direct assignment (pandas > 0.15)
In [50]: pf.loc[:,:,'F'] = pd.DataFrame(data, rng, cols);pf
Out[50]:
Dimensions: 3 (items) x 100 (major_axis) x 6 (minor_axis)
Items axis: df1 to df3
Major_axis axis: 2013-01-01 00:00:00 to 2013-04-10 00:00:00
Minor_axis axis: A to F
Mask a panel by using np.where and then reconstructing the panel with the new masked values
8.2.3 New Columns
Efficiently and dynamically creating new columns using applymap
In [51]: df = pd.DataFrame(
....:
{'AAA' : [1,2,1,3], 'BBB' : [1,1,2,2], 'CCC' : [2,1,3,1]}); df
....:
Out[51]:
AAA BBB CCC
0
1
1
2
1
2
1
1
2
1
2
3
3
3
2
1
In [52]: source_cols = df.columns # or some subset would work too.
In [53]: new_cols = [str(x) + "_cat" for x in source_cols]
In [54]: categories = {1 : 'Alpha', 2 : 'Beta', 3 : 'Charlie' }
In [55]: df[new_cols] = df[source_cols].applymap(categories.get);df
Out[55]:
AAA BBB CCC AAA_cat BBB_cat CCC_cat
8.2. Selection
261
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1
2
3
1
2
1
3
1
1
2
2
2
1
3
1
Alpha
Beta
Alpha
Charlie
Alpha
Alpha
Beta
Beta
Beta
Alpha
Charlie
Alpha
Keep other columns when using min() with groupby
In [56]: df = pd.DataFrame(
....:
{'AAA' : [1,1,1,2,2,2,3,3], 'BBB' : [2,1,3,4,5,1,2,3]}); df
....:
Out[56]:
AAA BBB
0
1
2
1
1
1
2
1
3
3
2
4
4
2
5
5
2
1
6
3
2
7
3
3
Method 1 : idxmin() to get the index of the mins
In [57]: df.loc[df.groupby("AAA")["BBB"].idxmin()]
Out[57]:
AAA BBB
1
1
1
5
2
1
6
3
2
Method 2 : sort then take first of each
In [58]: df.sort("BBB").groupby("AAA", as_index=False).first()
Out[58]:
AAA BBB
0
1
1
1
2
1
2
3
2
Notice the same results, with the exception of the index.
8.3 MultiIndexing
The multindexing docs.
Creating a multi-index from a labeled frame
In [59]: df = pd.DataFrame({'row' :
....:
'One_X'
....:
'One_Y'
....:
'Two_X'
....:
'Two_Y'
....:
Out[59]:
One_X One_Y Two_X Two_Y row
0
1.1
1.2
1.11
1.22
0
1
1.1
1.2
1.11
1.22
1
2
1.1
1.2
1.11
1.22
2
262
[0,1,2],
: [1.1,1.1,1.1],
: [1.2,1.2,1.2],
: [1.11,1.11,1.11],
: [1.22,1.22,1.22]}); df
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
# As Labelled Index
In [60]: df = df.set_index('row');df
Out[60]:
One_X One_Y Two_X Two_Y
row
0
1.1
1.2
1.11
1.22
1
1.1
1.2
1.11
1.22
2
1.1
1.2
1.11
1.22
# With Heirarchical Columns
In [61]: df.columns = pd.MultiIndex.from_tuples([tuple(c.split('_')) for c in df.columns]);df
Out[61]:
One
Two
X
Y
X
Y
row
0
1.1 1.2 1.11 1.22
1
1.1 1.2 1.11 1.22
2
1.1 1.2 1.11 1.22
# Now stack & Reset
In [62]: df = df.stack(0).reset_index(1);df
Out[62]:
level_1
X
Y
row
0
One 1.10 1.20
0
Two 1.11 1.22
1
One 1.10 1.20
1
Two 1.11 1.22
2
One 1.10 1.20
2
Two 1.11 1.22
# And fix the labels (Notice the label 'level_1' got added automatically)
In [63]: df.columns = ['Sample','All_X','All_Y'];df
Out[63]:
Sample All_X All_Y
row
0
One
1.10
1.20
0
Two
1.11
1.22
1
One
1.10
1.20
1
Two
1.11
1.22
2
One
1.10
1.20
2
Two
1.11
1.22
8.3.1 Arithmetic
Performing arithmetic with a multi-index that needs broadcasting
In [64]: cols = pd.MultiIndex.from_tuples([ (x,y) for x in ['A','B','C'] for y in ['O','I']])
In [65]: df = pd.DataFrame(np.random.randn(2,6),index=['n','m'],columns=cols); df
Out[65]:
A
B
C
O
I
O
I
O
I
n 1.920906 -0.388231 -2.314394 0.665508 0.402562 0.399555
m -1.765956 0.850423 0.388054 0.992312 0.744086 -0.739776
8.3. MultiIndexing
263
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [66]: df = df.div(df['C'],level=1); df
Out[66]:
A
B
O
I
O
I
n 4.771702 -0.971660 -5.749162 1.665625
m -2.373321 -1.149568 0.521518 -1.341367
C
O
1
1
I
1
1
8.3.2 Slicing
Slicing a multi-index with xs
In [67]: coords = [('AA','one'),('AA','six'),('BB','one'),('BB','two'),('BB','six')]
In [68]: index = pd.MultiIndex.from_tuples(coords)
In [69]: df = pd.DataFrame([11,22,33,44,55],index,['MyData']); df
Out[69]:
MyData
AA one
11
six
22
BB one
33
two
44
six
55
To take the cross section of the 1st level and 1st axis the index:
In [70]: df.xs('BB',level=0,axis=0)
Out[70]:
MyData
one
33
two
44
six
55
#Note : level and axis are optional, and default to zero
...and now the 2nd level of the 1st axis.
In [71]: df.xs('six',level=1,axis=0)
Out[71]:
MyData
AA
22
BB
55
Slicing a multi-index with xs, method #2
In [72]: index = list(itertools.product(['Ada','Quinn','Violet'],['Comp','Math','Sci']))
In [73]: headr = list(itertools.product(['Exams','Labs'],['I','II']))
In [74]: indx = pd.MultiIndex.from_tuples(index,names=['Student','Course'])
In [75]: cols = pd.MultiIndex.from_tuples(headr) #Notice these are un-named
In [76]: data = [[70+x+y+(x*y)%3 for x in range(4)] for y in range(9)]
In [77]: df = pd.DataFrame(data,indx,cols); df
Out[77]:
Exams
Labs
I II
I II
Student Course
264
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
Ada
Quinn
Violet
Comp
Math
Sci
Comp
Math
Sci
Comp
Math
Sci
70
71
72
73
74
75
76
77
78
71
73
75
74
76
78
77
79
81
72
75
75
75
78
78
78
81
81
73
74
75
76
77
78
79
80
81
In [78]: All = slice(None)
In [79]: df.loc['Violet']
Out[79]:
Exams
Labs
I II
I II
Course
Comp
76 77
78 79
Math
77 79
81 80
Sci
78 81
81 81
In [80]: df.loc[(All,'Math'),All]
Out[80]:
Exams
Labs
I II
I II
Student Course
Ada
Math
71 73
75 74
Quinn
Math
74 76
78 77
Violet Math
77 79
81 80
In [81]: df.loc[(slice('Ada','Quinn'),'Math'),All]
Out[81]:
Exams
Labs
I II
I II
Student Course
Ada
Math
71 73
75 74
Quinn
Math
74 76
78 77
In [82]: df.loc[(All,'Math'),('Exams')]
Out[82]:
I II
Student Course
Ada
Math
71 73
Quinn
Math
74 76
Violet Math
77 79
In [83]: df.loc[(All,'Math'),(All,'II')]
Out[83]:
Exams Labs
II
II
Student Course
Ada
Math
73
74
Quinn
Math
76
77
Violet Math
79
80
Setting portions of a multi-index with xs
8.3. MultiIndexing
265
pandas: powerful Python data analysis toolkit, Release 0.16.1
8.3.3 Sorting
Sort by specific column or an ordered list of columns, with a multi-index
In [84]: df.sort(('Labs', 'II'), ascending=False)
Out[84]:
Exams
Labs
I II
I II
Student Course
Violet Sci
78 81
81 81
Math
77 79
81 80
Comp
76 77
78 79
Quinn
Sci
75 78
78 78
Math
74 76
78 77
Comp
73 74
75 76
Ada
Sci
72 75
75 75
Math
71 73
75 74
Comp
70 71
72 73
Partial Selection, the need for sortedness;
8.3.4 Levels
Prepending a level to a multiindex
Flatten Hierarchical columns
8.3.5 panelnd
The panelnd docs.
Construct a 5D panelnd
8.4 Missing Data
The missing data docs.
Fill forward a reversed timeseries
In [85]: df = pd.DataFrame(np.random.randn(6,1), index=pd.date_range('2013-08-01', periods=6, freq='B
In [86]: df.ix[3,'A'] = np.nan
In [87]: df
Out[87]:
A
2013-08-01 -1.054874
2013-08-02 -0.179642
2013-08-05 0.639589
2013-08-06
NaN
2013-08-07 1.906684
2013-08-08 0.104050
In [88]: df.reindex(df.index[::-1]).ffill()
Out[88]:
266
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
A
2013-08-08 0.104050
2013-08-07 1.906684
2013-08-06 1.906684
2013-08-05 0.639589
2013-08-02 -0.179642
2013-08-01 -1.054874
cumsum reset at NaN values
8.4.1 Replace
Using replace with backrefs
8.5 Grouping
The grouping docs.
Basic grouping with apply
Unlike agg, apply’s callable is passed a sub-DataFrame which gives you access to all the columns
In [89]: df = pd.DataFrame({'animal': 'cat dog cat fish dog cat cat'.split(),
....:
'size': list('SSMMMLL'),
....:
'weight': [8, 10, 11, 1, 20, 12, 12],
....:
'adult' : [False] * 5 + [True] * 2}); df
....:
Out[89]:
adult animal size weight
0 False
cat
S
8
1 False
dog
S
10
2 False
cat
M
11
3 False
fish
M
1
4 False
dog
M
20
5
True
cat
L
12
6
True
cat
L
12
#List the size of the animals with the highest weight.
In [90]: df.groupby('animal').apply(lambda subf: subf['size'][subf['weight'].idxmax()])
Out[90]:
animal
cat
L
dog
M
fish
M
dtype: object
Using get_group
In [91]: gb = df.groupby(['animal'])
In [92]: gb.get_group('cat')
Out[92]:
adult animal size weight
0 False
cat
S
8
2 False
cat
M
11
8.5. Grouping
267
pandas: powerful Python data analysis toolkit, Release 0.16.1
5
6
True
True
cat
cat
L
L
12
12
Apply to different items in a group
In [93]: def GrowUp(x):
....:
avg_weight = sum(x[x['size'] == 'S'].weight * 1.5)
....:
avg_weight += sum(x[x['size'] == 'M'].weight * 1.25)
....:
avg_weight += sum(x[x['size'] == 'L'].weight)
....:
avg_weight /= len(x)
....:
return pd.Series(['L',avg_weight,True], index=['size', 'weight', 'adult'])
....:
In [94]: expected_df = gb.apply(GrowUp)
In [95]: expected_df
Out[95]:
size
weight adult
animal
cat
L 12.4375 True
dog
L 20.0000 True
fish
L
1.2500 True
Expanding Apply
In [96]: S = pd.Series([i / 100.0 for i in range(1,11)])
In [97]: def CumRet(x,y):
....:
return x * (1 + y)
....:
In [98]: def Red(x):
....:
return functools.reduce(CumRet,x,1.0)
....:
In [99]: pd.expanding_apply(S, Red)
Out[99]:
0
1.010000
1
1.030200
2
1.061106
3
1.103550
4
1.158728
5
1.228251
6
1.314229
7
1.419367
8
1.547110
9
1.701821
dtype: float64
Replacing some values with mean of the rest of a group
In [100]: df = pd.DataFrame({'A' : [1, 1, 2, 2], 'B' : [1, -1, 1, 2]})
In [101]: gb = df.groupby('A')
In [102]: def replace(g):
.....:
mask = g < 0
.....:
g.loc[mask] = g[~mask].mean()
.....:
return g
268
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
.....:
In [103]: gb.transform(replace)
Out[103]:
B
0 1
1 1
2 1
3 2
Sort groups by aggregated data
In [104]: df = pd.DataFrame({'code': ['foo', 'bar', 'baz'] * 2,
.....:
'data': [0.16, -0.21, 0.33, 0.45, -0.59, 0.62],
.....:
'flag': [False, True] * 3})
.....:
In [105]: code_groups = df.groupby('code')
In [106]: agg_n_sort_order = code_groups[['data']].transform(sum).sort('data')
In [107]: sorted_df = df.ix[agg_n_sort_order.index]
In [108]: sorted_df
Out[108]:
code data
flag
1 bar -0.21
True
4 bar -0.59 False
0 foo 0.16 False
3 foo 0.45
True
2 baz 0.33 False
5 baz 0.62
True
Create multiple aggregated columns
In [109]: rng = pd.date_range(start="2014-10-07",periods=10,freq='2min')
In [110]: ts = pd.Series(data = list(range(10)), index = rng)
In [111]: def MyCust(x):
.....:
if len(x) > 2:
.....:
return x[1] * 1.234
.....:
return pd.NaT
.....:
In [112]: mhc = {'Mean' : np.mean, 'Max' : np.max, 'Custom' : MyCust}
In [113]: ts.resample("5min",how = mhc)
Out[113]:
Max Custom Mean
2014-10-07 00:00:00
2 1.234
1.0
2014-10-07 00:05:00
4
NaN
3.5
2014-10-07 00:10:00
7 7.404
6.0
2014-10-07 00:15:00
9
NaN
8.5
In [114]: ts
Out[114]:
2014-10-07 00:00:00
2014-10-07 00:02:00
8.5. Grouping
0
1
269
pandas: powerful Python data analysis toolkit, Release 0.16.1
2014-10-07 00:04:00
2014-10-07 00:06:00
2014-10-07 00:08:00
2014-10-07 00:10:00
2014-10-07 00:12:00
2014-10-07 00:14:00
2014-10-07 00:16:00
2014-10-07 00:18:00
Freq: 2T, dtype: int64
2
3
4
5
6
7
8
9
Create a value counts column and reassign back to the DataFrame
In [115]: df = pd.DataFrame({'Color': 'Red Red Red Blue'.split(),
.....:
'Value': [100, 150, 50, 50]}); df
.....:
Out[115]:
Color Value
0
Red
100
1
Red
150
2
Red
50
3 Blue
50
In [116]: df['Counts'] = df.groupby(['Color']).transform(len)
In [117]: df
Out[117]:
Color Value
0
Red
100
1
Red
150
2
Red
50
3 Blue
50
Counts
3
3
3
1
Shift groups of the values in a column based on the index
In [118]: df = pd.DataFrame(
.....:
{u'line_race': [10, 10, 8, 10, 10, 8],
.....:
u'beyer': [99, 102, 103, 103, 88, 100]},
.....:
index=[u'Last Gunfighter', u'Last Gunfighter', u'Last Gunfighter',
.....:
u'Paynter', u'Paynter', u'Paynter']); df
.....:
Out[118]:
beyer line_race
Last Gunfighter
99
10
Last Gunfighter
102
10
Last Gunfighter
103
8
Paynter
103
10
Paynter
88
10
Paynter
100
8
In [119]: df['beyer_shifted'] = df.groupby(level=0)['beyer'].shift(1)
In [120]: df
Out[120]:
Last Gunfighter
Last Gunfighter
Last Gunfighter
Paynter
Paynter
270
beyer
99
102
103
103
88
line_race
10
10
8
10
10
beyer_shifted
NaN
99
102
NaN
103
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
Paynter
100
8
88
Select row with maximum value from each group
In [121]: df = pd.DataFrame({'host':['other','other','that','this','this'],
.....:
'service':['mail','web','mail','mail','web'],
.....:
'no':[1, 2, 1, 2, 1]}).set_index(['host', 'service'])
.....:
In [122]: mask = df.groupby(level=0).agg('idxmax')
In [123]: df_count = df.loc[mask['no']].reset_index()
In [124]: df_count
Out[124]:
host service no
0 other
web
2
1
that
mail
1
2
this
mail
2
Grouping like Python’s itertools.groupby
In [125]: df = pd.DataFrame([0, 1, 0, 1, 1, 1, 0, 1, 1], columns=['A'])
In [126]: df.A.groupby((df.A != df.A.shift()).cumsum()).groups
Out[126]: {1: [0L], 2: [1L], 3: [2L], 4: [3L, 4L, 5L], 5: [6L], 6: [7L, 8L]}
In [127]: df.A.groupby((df.A != df.A.shift()).cumsum()).cumsum()
Out[127]:
0
0
1
1
2
0
3
1
4
2
5
3
6
0
7
1
8
2
dtype: int64
8.5.1 Expanding Data
Alignment and to-date
Rolling Computation window based on values instead of counts
Rolling Mean by Time Interval
8.5.2 Splitting
Splitting a frame
Create a list of dataframes, split using a delineation based on logic included in rows.
In [128]: df = pd.DataFrame(data={'Case' : ['A','A','A','B','A','A','B','A','A'],
.....:
'Data' : np.random.randn(9)})
.....:
8.5. Grouping
271
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [129]: dfs = list(zip(*df.groupby(pd.rolling_median((1*(df['Case']=='B')).cumsum(),3,True))))[-1]
In [130]: dfs[0]
Out[130]:
Case
Data
0
A 0.174068
1
A -0.439461
2
A -0.741343
3
B -0.079673
In [131]: dfs[1]
Out[131]:
Case
Data
4
A -0.922875
5
A 0.303638
6
B -0.917368
In [132]: dfs[2]
Out[132]:
Case
Data
7
A -1.624062
8
A -0.758514
8.5.3 Pivot
The Pivot docs.
Partial sums and subtotals
In [133]: df = pd.DataFrame(data={'Province' : ['ON','QC','BC','AL','AL','MN','ON'],
.....:
'City' : ['Toronto','Montreal','Vancouver','Calgary','Edmonton','W
.....:
'Sales' : [13,6,16,8,4,3,1]})
.....:
In [134]: table = pd.pivot_table(df,values=['Sales'],index=['Province'],columns=['City'],aggfunc=np.s
In [135]: table.stack('City')
Out[135]:
Sales
Province City
AL
All
12
Calgary
8
Edmonton
4
BC
All
16
Vancouver
16
MN
All
3
Winnipeg
3
...
...
All
Calgary
8
Edmonton
4
Montreal
6
Toronto
13
Vancouver
16
Windsor
1
Winnipeg
3
[20 rows x 1 columns]
272
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
Frequency table like plyr in R
In [136]: grades = [48,99,75,80,42,80,72,68,36,78]
In [137]: df = pd.DataFrame( {'ID': ["x%d" % r for r in range(10)],
.....:
'Gender' : ['F', 'M', 'F', 'M', 'F', 'M', 'F', 'M', 'M', 'M'],
.....:
'ExamYear': ['2007','2007','2007','2008','2008','2008','2008','2009','2
.....:
'Class': ['algebra', 'stats', 'bio', 'algebra', 'algebra', 'stats', 'st
.....:
'Participated': ['yes','yes','yes','yes','no','yes','yes','yes','yes','
.....:
'Passed': ['yes' if x > 50 else 'no' for x in grades],
.....:
'Employed': [True,True,True,False,False,False,False,True,True,False],
.....:
'Grade': grades})
.....:
In [138]: df.groupby('ExamYear').agg({'Participated': lambda x: x.value_counts()['yes'],
.....:
'Passed': lambda x: sum(x == 'yes'),
.....:
'Employed' : lambda x : sum(x),
.....:
'Grade' : lambda x : sum(x) / len(x)})
.....:
Out[138]:
Grade Employed Participated Passed
ExamYear
2007
74
3
3
2
2008
68
0
3
3
2009
60
2
3
2
8.5.4 Apply
Rolling Apply to Organize - Turning embedded lists into a multi-index frame
In [139]: df = pd.DataFrame(data={'A' : [[2,4,8,16],[100,200],[10,20,30]], 'B' : [['a','b','c'],['jj'
In [140]: def SeriesFromSubList(aList):
.....:
return pd.Series(aList)
.....:
In [141]: df_orgz = pd.concat(dict([ (ind,row.apply(SeriesFromSubList)) for ind,row in df.iterrows()
Rolling Apply with a DataFrame returning a Series
Rolling Apply to multiple columns where function calculates a Series before a Scalar from the Series is returned
In [142]: df = pd.DataFrame(data=np.random.randn(2000,2)/10000,
.....:
index=pd.date_range('2001-01-01',periods=2000),
.....:
columns=['A','B']); df
.....:
Out[142]:
A
B
2001-01-01 -0.000056 -0.000059
2001-01-02 -0.000107 -0.000168
2001-01-03 0.000040 0.000061
2001-01-04 0.000039 0.000182
2001-01-05 0.000071 -0.000067
2001-01-06 0.000024 0.000031
2001-01-07 0.000012 -0.000021
...
...
...
2006-06-17 0.000129 0.000094
2006-06-18 0.000059 0.000216
8.5. Grouping
273
pandas: powerful Python data analysis toolkit, Release 0.16.1
2006-06-19 -0.000069 0.000283
2006-06-20 0.000089 0.000084
2006-06-21 0.000075 0.000041
2006-06-22 -0.000037 -0.000011
2006-06-23 -0.000070 -0.000048
[2000 rows x 2 columns]
In [143]: def gm(aDF,Const):
.....:
v = ((((aDF.A+aDF.B)+1).cumprod())-1)*Const
.....:
return (aDF.index[0],v.iloc[-1])
.....:
In [144]: S = pd.Series(dict([ gm(df.iloc[i:min(i+51,len(df)-1)],5) for i in range(len(df)-50) ])); S
Out[144]:
2001-01-01
-0.003108
2001-01-02
-0.001787
2001-01-03
0.000204
2001-01-04
-0.000166
2001-01-05
-0.002148
2001-01-06
-0.001831
2001-01-07
-0.001663
...
2006-04-28
-0.009152
2006-04-29
-0.006728
2006-04-30
-0.005840
2006-05-01
-0.003650
2006-05-02
-0.003801
2006-05-03
-0.004272
2006-05-04
-0.003839
dtype: float64
Rolling apply with a DataFrame returning a Scalar
Rolling Apply to multiple columns where function returns a Scalar (Volume Weighted Average Price)
In [145]: rng = pd.date_range(start = '2014-01-01',periods = 100)
In [146]: df = pd.DataFrame({'Open' : np.random.randn(len(rng)),
.....:
'Close' : np.random.randn(len(rng)),
.....:
'Volume' : np.random.randint(100,2000,len(rng))}, index=rng); df
.....:
Out[146]:
Close
Open Volume
2014-01-01 1.550590 0.458513
1371
2014-01-02 -0.818812 -0.508850
1433
2014-01-03 1.160619 0.257610
645
2014-01-04 0.081521 -1.773393
878
2014-01-05 1.083284 -0.560676
1143
2014-01-06 -0.518721 0.284174
1088
2014-01-07 0.140661 1.146889
1722
...
...
...
...
2014-04-04 0.458193 -0.669474
1768
2014-04-05 0.108502 -1.616315
836
2014-04-06 1.418082 -1.294906
694
2014-04-07 0.486530 1.171647
796
2014-04-08 0.181885 0.501639
265
2014-04-09 -0.707238 -0.361868
1293
2014-04-10 1.211432 1.564429
1088
274
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
[100 rows x 3 columns]
In [147]: def vwap(bars): return ((bars.Close*bars.Volume).sum()/bars.Volume.sum()).round(2)
In [148]: window = 5
In [149]: s = pd.concat([ (pd.Series(vwap(df.iloc[i:i+window]), index=[df.index[i+window]])) for i in
Out[149]:
2014-01-06
0.55
2014-01-07
0.06
2014-01-08
0.32
2014-01-09
0.03
2014-01-10
0.08
2014-01-11
-0.50
2014-01-12
-0.26
...
2014-04-04
0.36
2014-04-05
0.48
2014-04-06
0.54
2014-04-07
0.46
2014-04-08
0.45
2014-04-09
0.53
2014-04-10
0.15
dtype: float64
8.6 Timeseries
Between times
Using indexer between time
Constructing a datetime range that excludes weekends and includes only certain times
Vectorized Lookup
Turn a matrix with hours in columns and days in rows into a continuous row sequence in the form of a time series.
How to rearrange a python pandas DataFrame?
Dealing with duplicates when reindexing a timeseries to a specified frequency
Calculate the first day of the month for each entry in a DatetimeIndex
In [150]: dates = pd.date_range('2000-01-01', periods=5)
In [151]: dates.to_period(freq='M').to_timestamp()
Out[151]:
DatetimeIndex(['2000-01-01', '2000-01-01', '2000-01-01', '2000-01-01',
'2000-01-01'],
dtype='datetime64[ns]', freq=None, tz=None)
8.6.1 Resampling
The Resample docs.
TimeGrouping of values grouped across time
TimeGrouping #2
8.6. Timeseries
275
pandas: powerful Python data analysis toolkit, Release 0.16.1
Using TimeGrouper and another grouping to create subgroups, then apply a custom function
Resampling with custom periods
Resample intraday frame without adding new days
Resample minute data
Resample with groupby
8.7 Merge
The Concat docs. The Join docs.
Append two dataframes with overlapping index (emulate R rbind)
In [152]: rng = pd.date_range('2000-01-01', periods=6)
In [153]: df1 = pd.DataFrame(np.random.randn(6, 3), index=rng, columns=['A', 'B', 'C'])
In [154]: df2 = df1.copy()
ignore_index is needed in pandas < v0.13, and depending on df construction
In [155]: df
Out[155]:
A
0 -0.174202
1 -0.654455
2
0.351578
3
0.565398
4
0.446473
5
1.710685
6 -0.174202
7 -0.654455
8
0.351578
9
0.565398
10 0.446473
11 1.710685
= df1.append(df2,ignore_index=True); df
B
-0.477257
-1.411456
0.307871
-0.185821
0.566368
-0.667054
-0.477257
-1.411456
0.307871
-0.185821
0.566368
-0.667054
C
0.239870
-1.778457
-0.286865
0.937593
0.721476
-0.651191
0.239870
-1.778457
-0.286865
0.937593
0.721476
-0.651191
Self Join of a DataFrame
In [156]: df
.....:
.....:
.....:
.....:
Out[156]:
Area Bins
0
A
110
1
A
110
2
A
160
3
A
160
4
A
160
5
C
40
6
C
40
= pd.DataFrame(data={'Area' :
'Bins' :
'Test_0'
'Data' :
Data
-0.399974
-1.519206
1.678487
0.005345
-0.534461
0.255077
1.093310
['A'] * 5 + ['C'] * 2,
[110] * 2 + [160] * 3 + [40] * 2,
: [0, 1, 0, 1, 2, 0, 1],
np.random.randn(7)});df
Test_0
0
1
0
1
2
0
1
In [157]: df['Test_1'] = df['Test_0'] - 1
276
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [158]: pd.merge(df, df, left_on=['Bins', 'Area','Test_0'], right_on=['Bins', 'Area','Test_1'],suff
Out[158]:
Area Bins
Data_L Test_0_L Test_1_L
Data_R Test_0_R Test_1_R
0
A
110 -0.399974
0
-1 -1.519206
1
0
1
A
160 1.678487
0
-1 0.005345
1
0
2
A
160 0.005345
1
0 -0.534461
2
1
3
C
40 0.255077
0
-1 1.093310
1
0
How to set the index and join
KDB like asof join
Join with a criteria based on the values
8.8 Plotting
The Plotting docs.
Make Matplotlib look like R
Setting x-axis major and minor labels
Plotting multiple charts in an ipython notebook
Creating a multi-line plot
Plotting a heatmap
Annotate a time-series plot
Annotate a time-series plot #2
Generate Embedded plots in excel files using Pandas, Vincent and xlsxwriter
Boxplot for each quartile of a stratifying variable
In [159]: df = pd.DataFrame(
.....:
{u'stratifying_var': np.random.uniform(0, 100, 20),
.....:
u'price': np.random.normal(100, 5, 20)})
.....:
In [160]: df[u'quartiles'] = pd.qcut(
.....:
df[u'stratifying_var'],
.....:
4,
.....:
labels=[u'0-25%', u'25-50%', u'50-75%', u'75-100%'])
.....:
In [161]: df.boxplot(column=u'price', by=u'quartiles')
Out[161]:
8.8. Plotting
277
pandas: powerful Python data analysis toolkit, Release 0.16.1
8.9 Data In/Out
Performance comparison of SQL vs HDF5
8.9.1 CSV
The CSV docs
read_csv in action
appending to a csv
how to read in multiple files, appending to create a single dataframe
Reading a csv chunk-by-chunk
Reading only certain rows of a csv chunk-by-chunk
Reading the first few lines of a frame
278
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
Reading a file that is compressed but not by gzip/bz2 (the native compressed formats which read_csv understands). This example shows a WinZipped file, but is a general application of opening the file within a context
manager and using that handle to read. See here
Inferring dtypes from a file
Dealing with bad lines
Dealing with bad lines II
Reading CSV with Unix timestamps and converting to local timezone
Write a multi-row index CSV without writing duplicates
Parsing date components in multi-columns is faster with a format
In [30]: i = pd.date_range('20000101',periods=10000)
In [31]: df = pd.DataFrame(dict(year = i.year, month = i.month, day = i.day))
In [32]: df.head()
Out[32]:
day month year
0
1
1 2000
1
2
1 2000
2
3
1 2000
3
4
1 2000
4
5
1 2000
In [33]: %timeit pd.to_datetime(df.year*10000+df.month*100+df.day,format='%Y%m%d')
100 loops, best of 3: 7.08 ms per loop
# simulate combinging into a string, then parsing
In [34]: ds = df.apply(lambda x: "%04d%02d%02d" % (x['year'],x['month'],x['day']),axis=1)
In [35]: ds.head()
Out[35]:
0
20000101
1
20000102
2
20000103
3
20000104
4
20000105
dtype: object
In [36]: %timeit pd.to_datetime(ds)
1 loops, best of 3: 488 ms per loop
8.9.2 SQL
The SQL docs
Reading from databases with SQL
8.9.3 Excel
The Excel docs
Reading from a filelike handle Reading HTML tables from a server that cannot handle the default request header
8.9. Data In/Out
279
pandas: powerful Python data analysis toolkit, Release 0.16.1
8.9.4 HDFStore
The HDFStores docs
Simple Queries with a Timestamp Index
Managing heterogeneous data using a linked multiple table hierarchy
Merging on-disk tables with millions of rows
Avoiding inconsistencies when writing to a store from multiple processes/threads
De-duplicating a large store by chunks, essentially a recursive reduction operation. Shows a function for taking in data
from csv file and creating a store by chunks, with date parsing as well. See here
Creating a store chunk-by-chunk from a csv file
Appending to a store, while creating a unique index
Large Data work flows
Reading in a sequence of files, then providing a global unique index to a store while appending
Groupby on a HDFStore with low group density
Groupby on a HDFStore with high group density
Hierarchical queries on a HDFStore
Counting with a HDFStore
Troubleshoot HDFStore exceptions
Setting min_itemsize with strings
Using ptrepack to create a completely-sorted-index on a store
Storing Attributes to a group node
In [162]: df = pd.DataFrame(np.random.randn(8,3))
In [163]: store = pd.HDFStore('test.h5')
In [164]: store.put('df',df)
# you can store an arbitrary python object via pickle
In [165]: store.get_storer('df').attrs.my_attribute = dict(A = 10)
In [166]: store.get_storer('df').attrs.my_attribute
Out[166]: {'A': 10}
8.9.5 Binary Files
pandas readily accepts numpy record arrays, if you need to read in a binary file consisting of an array of C structs.
For example, given this C program in a file called main.c compiled with gcc main.c -std=gnu99 on a 64-bit
machine,
#include
#include
typedef struct _Data
{
int32_t count;
280
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
double avg;
float scale;
} Data;
int main(int argc, const char *argv[])
{
size_t n = 10;
Data d[n];
for (int i = 0; i < n; ++i)
{
d[i].count = i;
d[i].avg = i + 1.0;
d[i].scale = (float) i + 2.0f;
}
FILE *file = fopen("binary.dat", "wb");
fwrite(&d, sizeof(Data), n, file);
fclose(file);
return 0;
}
the following Python code will read the binary file ’binary.dat’ into a pandas DataFrame, where each element
of the struct corresponds to a column in the frame:
names = 'count', 'avg', 'scale'
# note that the offsets are larger than the size of the type because of
# struct padding
offsets = 0, 8, 16
formats = 'i4', 'f8', 'f4'
dt = np.dtype({'names': names, 'offsets': offsets, 'formats': formats},
align=True)
df = pd.DataFrame(np.fromfile('binary.dat', dt))
Note: The offsets of the structure elements may be different depending on the architecture of the machine on which
the file was created. Using a raw binary file format like this for general data storage is not recommended, as it is not
cross platform. We recommended either HDF5 or msgpack, both of which are supported by pandas’ IO facilities.
8.10 Computation
Numerical integration (sample-based) of a time series
8.11 Timedeltas
The Timedeltas docs.
Using timedeltas
In [167]: s
= pd.Series(pd.date_range('2012-1-1', periods=3, freq='D'))
In [168]: s - s.max()
8.10. Computation
281
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[168]:
0
-2 days
1
-1 days
2
0 days
dtype: timedelta64[ns]
In [169]: s.max() - s
Out[169]:
0
2 days
1
1 days
2
0 days
dtype: timedelta64[ns]
In [170]: s - datetime.datetime(2011,1,1,3,5)
Out[170]:
0
364 days 20:55:00
1
365 days 20:55:00
2
366 days 20:55:00
dtype: timedelta64[ns]
In [171]: s + datetime.timedelta(minutes=5)
Out[171]:
0
2012-01-01 00:05:00
1
2012-01-02 00:05:00
2
2012-01-03 00:05:00
dtype: datetime64[ns]
In [172]: datetime.datetime(2011,1,1,3,5) - s
Out[172]:
0
-365 days +03:05:00
1
-366 days +03:05:00
2
-367 days +03:05:00
dtype: timedelta64[ns]
In [173]: datetime.timedelta(minutes=5) + s
Out[173]:
0
2012-01-01 00:05:00
1
2012-01-02 00:05:00
2
2012-01-03 00:05:00
dtype: datetime64[ns]
Adding and subtracting deltas and dates
In [174]: deltas = pd.Series([ datetime.timedelta(days=i) for i in range(3) ])
In [175]: df
Out[175]:
A
0 2012-01-01
1 2012-01-02
2 2012-01-03
= pd.DataFrame(dict(A = s, B = deltas)); df
B
0 days
1 days
2 days
In [176]: df['New Dates'] = df['A'] + df['B'];
In [177]: df['Delta'] = df['A'] - df['New Dates']; df
Out[177]:
A
B New Dates
Delta
0 2012-01-01 0 days 2012-01-01 0 days
282
Chapter 8. Cookbook
pandas: powerful Python data analysis toolkit, Release 0.16.1
1 2012-01-02 1 days 2012-01-03 -1 days
2 2012-01-03 2 days 2012-01-05 -2 days
In [178]: df.dtypes
Out[178]:
A
datetime64[ns]
B
timedelta64[ns]
New Dates
datetime64[ns]
Delta
timedelta64[ns]
dtype: object
Another example
Values can be set to NaT using np.nan, similar to datetime
In [179]: y = s - s.shift(); y
Out[179]:
0
NaT
1
1 days
2
1 days
dtype: timedelta64[ns]
In [180]: y[1] = np.nan; y
Out[180]:
0
NaT
1
NaT
2
1 days
dtype: timedelta64[ns]
8.12 Aliasing Axis Names
To globally provide aliases for axis names, one can define these 2 functions:
In [181]: def set_axis_alias(cls, axis, alias):
.....:
if axis not in cls._AXIS_NUMBERS:
.....:
raise Exception("invalid axis [%s] for alias [%s]" % (axis, alias))
.....:
cls._AXIS_ALIASES[alias] = axis
.....:
In [182]: def clear_axis_alias(cls, axis, alias):
.....:
if axis not in cls._AXIS_NUMBERS:
.....:
raise Exception("invalid axis [%s] for alias [%s]" % (axis, alias))
.....:
cls._AXIS_ALIASES.pop(alias,None)
.....:
In [183]: set_axis_alias(pd.DataFrame,'columns', 'myaxis2')
In [184]: df2 = pd.DataFrame(np.random.randn(3,2),columns=['c1','c2'],index=['i1','i2','i3'])
In [185]: df2.sum(axis='myaxis2')
Out[185]:
i1
0.239786
i2
0.259018
i3
0.163470
dtype: float64
In [186]: clear_axis_alias(pd.DataFrame,'columns', 'myaxis2')
8.12. Aliasing Axis Names
283
pandas: powerful Python data analysis toolkit, Release 0.16.1
8.13 Creating Example Data
To create a dataframe from every combination of some given values, like R’s expand.grid() function, we can
create a dict where the keys are column names and the values are lists of the data values:
In [187]: def expand_grid(data_dict):
.....:
rows = itertools.product(*data_dict.values())
.....:
return pd.DataFrame.from_records(rows, columns=data_dict.keys())
.....:
In [188]: df = expand_grid(
.....:
{'height': [60, 70],
.....:
'weight': [100, 140, 180],
.....:
'sex': ['Male', 'Female']})
.....:
In [189]: df
Out[189]:
sex weight
0
Male
100
1
Male
100
2
Male
140
3
Male
140
4
Male
180
5
Male
180
6
Female
100
7
Female
100
8
Female
140
9
Female
140
10 Female
180
11 Female
180
284
height
60
70
60
70
60
70
60
70
60
70
60
70
Chapter 8. Cookbook
CHAPTER
NINE
INTRO TO DATA STRUCTURES
We’ll start with a quick, non-comprehensive overview of the fundamental data structures in pandas to get you started.
The fundamental behavior about data types, indexing, and axis labeling / alignment apply across all of the objects. To
get started, import numpy and load pandas into your namespace:
In [1]: import numpy as np
# will use a lot in examples
In [2]: randn = np.random.randn
In [3]: from pandas import *
Here is a basic tenet to keep in mind: data alignment is intrinsic. The link between labels and data will not be broken
unless done so explicitly by you.
We’ll give a brief intro to the data structures, then consider all of the broad categories of functionality and methods in
separate sections.
When using pandas, we recommend the following import convention:
import pandas as pd
9.1 Series
Warning: In 0.13.0 Series has internaly been refactored to no longer sub-class ndarray but instead subclass
NDFrame, similarly to the rest of the pandas containers. This should be a transparent change with only very
limited API implications (See the Internal Refactoring)
Series is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers,
Python objects, etc.). The axis labels are collectively referred to as the index. The basic method to create a Series is
to call:
>>> s = Series(data, index=index)
Here, data can be many different things:
• a Python dict
• an ndarray
• a scalar value (like 5)
The passed index is a list of axis labels. Thus, this separates into a few cases depending on what data is:
From ndarray
285
pandas: powerful Python data analysis toolkit, Release 0.16.1
If data is an ndarray, index must be the same length as data. If no index is passed, one will be created having values
[0, ..., len(data) - 1].
In [4]: s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])
In [5]: s
Out[5]:
a
-2.783
b
0.426
c
-0.650
d
1.146
e
-0.663
dtype: float64
In [6]: s.index
Out[6]: Index([u'a', u'b', u'c', u'd', u'e'], dtype='object')
In [7]: Series(randn(5))
Out[7]:
0
0.294
1
-0.405
2
1.167
3
0.842
4
0.540
dtype: float64
Note: Starting in v0.8.0, pandas supports non-unique index values. If an operation that does not support duplicate
index values is attempted, an exception will be raised at that time. The reason for being lazy is nearly all performancebased (there are many instances in computations, like parts of GroupBy, where the index is not used).
From dict
If data is a dict, if index is passed the values in data corresponding to the labels in the index will be pulled out.
Otherwise, an index will be constructed from the sorted keys of the dict, if possible.
In [8]: d = {'a' : 0., 'b' : 1., 'c' : 2.}
In [9]: Series(d)
Out[9]:
a
0
b
1
c
2
dtype: float64
In [10]: Series(d, index=['b', 'c', 'd', 'a'])
Out[10]:
b
1
c
2
d
NaN
a
0
dtype: float64
Note: NaN (not a number) is the standard missing data marker used in pandas
From scalar value If data is a scalar value, an index must be provided. The value will be repeated to match the
length of index
286
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [11]: Series(5., index=['a', 'b', 'c', 'd', 'e'])
Out[11]:
a
5
b
5
c
5
d
5
e
5
dtype: float64
9.1.1 Series is ndarray-like
Series acts very similarly to a ndarray, and is a valid argument to most NumPy functions. However, things like
slicing also slice the index.
In [12]: s[0]
Out[12]: -2.7827595933769937
In [13]: s[:3]
Out[13]:
a
-2.783
b
0.426
c
-0.650
dtype: float64
In [14]: s[s > s.median()]
Out[14]:
b
0.426
d
1.146
dtype: float64
In [15]: s[[4, 3, 1]]
Out[15]:
e
-0.663
d
1.146
b
0.426
dtype: float64
In [16]: np.exp(s)
Out[16]:
a
0.062
b
1.532
c
0.522
d
3.147
e
0.515
dtype: float64
We will address array-based indexing in a separate section.
9.1.2 Series is dict-like
A Series is like a fixed-size dict in that you can get and set values by index label:
In [17]: s['a']
Out[17]: -2.7827595933769937
In [18]: s['e'] = 12.
9.1. Series
287
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [19]: s
Out[19]:
a
-2.783
b
0.426
c
-0.650
d
1.146
e
12.000
dtype: float64
In [20]: 'e' in s
Out[20]: True
In [21]: 'f' in s
Out[21]: False
If a label is not contained, an exception is raised:
>>> s['f']
KeyError: 'f'
Using the get method, a missing label will return None or specified default:
In [22]: s.get('f')
In [23]: s.get('f', np.nan)
Out[23]: nan
See also the section on attribute access.
9.1.3 Vectorized operations and label alignment with Series
When doing data analysis, as with raw NumPy arrays looping through Series value-by-value is usually not necessary.
Series can be also be passed into most NumPy methods expecting an ndarray.
In [24]: s + s
Out[24]:
a
-5.566
b
0.853
c
-1.301
d
2.293
e
24.000
dtype: float64
In [25]: s * 2
Out[25]:
a
-5.566
b
0.853
c
-1.301
d
2.293
e
24.000
dtype: float64
In [26]: np.exp(s)
Out[26]:
a
0.062
b
1.532
c
0.522
288
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
d
3.147
e
162754.791
dtype: float64
A key difference between Series and ndarray is that operations between Series automatically align the data based on
label. Thus, you can write computations without giving consideration to whether the Series involved have the same
labels.
In [27]: s[1:] + s[:-1]
Out[27]:
a
NaN
b
0.853
c
-1.301
d
2.293
e
NaN
dtype: float64
The result of an operation between unaligned Series will have the union of the indexes involved. If a label is not found
in one Series or the other, the result will be marked as missing NaN. Being able to write code without doing any explicit
data alignment grants immense freedom and flexibility in interactive data analysis and research. The integrated data
alignment features of the pandas data structures set pandas apart from the majority of related tools for working with
labeled data.
Note: In general, we chose to make the default result of operations between differently indexed objects yield the
union of the indexes in order to avoid loss of information. Having an index label, though the data is missing, is
typically important information as part of a computation. You of course have the option of dropping labels with
missing data via the dropna function.
9.1.4 Name attribute
Series can also have a name attribute:
In [28]: s = Series(np.random.randn(5), name='something')
In [29]: s
Out[29]:
0
0.541
1
-1.175
2
0.129
3
0.043
4
-0.429
Name: something, dtype: float64
In [30]: s.name
Out[30]: 'something'
The Series name will be assigned automatically in many cases, in particular when taking 1D slices of DataFrame as
you will see below.
9.2 DataFrame
DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it
like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object.
Like Series, DataFrame accepts many different kinds of input:
9.2. DataFrame
289
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Dict of 1D ndarrays, lists, dicts, or Series
• 2-D numpy.ndarray
• Structured or record ndarray
• A Series
• Another DataFrame
Along with the data, you can optionally pass index (row labels) and columns (column labels) arguments. If you pass
an index and / or columns, you are guaranteeing the index and / or columns of the resulting DataFrame. Thus, a dict
of Series plus a specific index will discard all data not matching up to the passed index.
If axis labels are not passed, they will be constructed from the input data based on common sense rules.
9.2.1 From dict of Series or dicts
The result index will be the union of the indexes of the various Series. If there are any nested dicts, these will be first
converted to Series. If no columns are passed, the columns will be the sorted list of dict keys.
In [31]: d = {'one' : Series([1., 2., 3.], index=['a', 'b', 'c']),
....:
'two' : Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
....:
In [32]: df = DataFrame(d)
In [33]: df
Out[33]:
one two
a
1
1
b
2
2
c
3
3
d NaN
4
In [34]: DataFrame(d, index=['d', 'b', 'a'])
Out[34]:
one two
d NaN
4
b
2
2
a
1
1
In [35]: DataFrame(d, index=['d', 'b', 'a'], columns=['two', 'three'])
Out[35]:
two three
d
4
NaN
b
2
NaN
a
1
NaN
The row and column labels can be accessed respectively by accessing the index and columns attributes:
Note: When a particular set of columns is passed along with a dict of data, the passed columns override the keys in
the dict.
In [36]: df.index
Out[36]: Index([u'a', u'b', u'c', u'd'], dtype='object')
In [37]: df.columns
Out[37]: Index([u'one', u'two'], dtype='object')
290
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
9.2.2 From dict of ndarrays / lists
The ndarrays must all be the same length. If an index is passed, it must clearly also be the same length as the arrays.
If no index is passed, the result will be range(n), where n is the array length.
In [38]: d = {'one' : [1., 2., 3., 4.],
....:
'two' : [4., 3., 2., 1.]}
....:
In [39]: DataFrame(d)
Out[39]:
one two
0
1
4
1
2
3
2
3
2
3
4
1
In [40]: DataFrame(d, index=['a', 'b', 'c', 'd'])
Out[40]:
one two
a
1
4
b
2
3
c
3
2
d
4
1
9.2.3 From structured or record array
This case is handled identically to a dict of arrays.
In [41]: data = np.zeros((2,),dtype=[('A', 'i4'),('B', 'f4'),('C', 'a10')])
In [42]: data[:] = [(1,2.,'Hello'),(2,3.,"World")]
In [43]: DataFrame(data)
Out[43]:
A B
C
0 1 2 Hello
1 2 3 World
In [44]: DataFrame(data, index=['first', 'second'])
Out[44]:
A B
C
first
1 2 Hello
second 2 3 World
In [45]: DataFrame(data, columns=['C', 'A', 'B'])
Out[45]:
C A B
0 Hello 1 2
1 World 2 3
Note: DataFrame is not intended to work exactly like a 2-dimensional NumPy ndarray.
9.2. DataFrame
291
pandas: powerful Python data analysis toolkit, Release 0.16.1
9.2.4 From a list of dicts
In [46]: data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]
In [47]: DataFrame(data2)
Out[47]:
a
b
c
0 1
2 NaN
1 5 10 20
In [48]: DataFrame(data2, index=['first', 'second'])
Out[48]:
a
b
c
first
1
2 NaN
second 5 10 20
In [49]: DataFrame(data2, columns=['a', 'b'])
Out[49]:
a
b
0 1
2
1 5 10
9.2.5 From a dict of tuples
You can automatically create a multi-indexed frame by passing a tuples dictionary
In [50]: DataFrame({('a',
....:
('a',
....:
('a',
....:
('b',
....:
('b',
....:
Out[50]:
a
b
a
b
c
a
b
A B
4
1
5
8 10
C
3
2
6
7 NaN
D NaN NaN NaN NaN
9
'b'):
'a'):
'c'):
'a'):
'b'):
{('A',
{('A',
{('A',
{('A',
{('A',
'B'):
'C'):
'B'):
'C'):
'D'):
1,
3,
5,
7,
9,
('A',
('A',
('A',
('A',
('A',
'C'):
'B'):
'C'):
'B'):
'B'):
2},
4},
6},
8},
10}})
9.2.6 From a Series
The result will be a DataFrame with the same index as the input Series, and with one column whose name is the
original name of the Series (only if no other column name provided).
Missing Data
Much more will be said on this topic in the Missing data section. To construct a DataFrame with missing data, use
np.nan for those values which are missing. Alternatively, you may pass a numpy.MaskedArray as the data
argument to the DataFrame constructor, and its masked entries will be considered missing.
9.2.7 Alternate Constructors
DataFrame.from_dict
292
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
DataFrame.from_dict takes a dict of dicts or a dict of array-like sequences and returns a DataFrame. It operates
like the DataFrame constructor except for the orient parameter which is ’columns’ by default, but which can
be set to ’index’ in order to use the dict keys as row labels. DataFrame.from_records
DataFrame.from_records takes a list of tuples or an ndarray with structured dtype. Works analogously to the
normal DataFrame constructor, except that index maybe be a specific field of the structured dtype to use as the
index. For example:
In [51]: data
Out[51]:
array([(1, 2.0, 'Hello'), (2, 3.0, 'World')],
dtype=[('A', ' 2
In [58]: df
Out[58]:
one two
a
1
1
b
2
2
c
3
3
d NaN
4
three
1
4
9
NaN
flag
False
False
True
False
Columns can be deleted or popped like with a dict:
In [59]: del df['two']
In [60]: three = df.pop('three')
In [61]: df
Out[61]:
one
flag
a
1 False
b
2 False
c
3
True
d NaN False
When inserting a scalar value, it will naturally be propagated to fill the column:
In [62]: df['foo'] = 'bar'
In [63]: df
Out[63]:
one
flag
a
1 False
b
2 False
c
3
True
d NaN False
foo
bar
bar
bar
bar
When inserting a Series that does not have the same index as the DataFrame, it will be conformed to the DataFrame’s
index:
In [64]: df['one_trunc'] = df['one'][:2]
In [65]: df
Out[65]:
one
flag
a
1 False
b
2 False
c
3
True
d NaN False
foo
bar
bar
bar
bar
one_trunc
1
2
NaN
NaN
You can insert raw ndarrays but their length must match the length of the DataFrame’s index.
By default, columns get inserted at the end. The insert function is available to insert at a particular location in the
columns:
In [66]: df.insert(1, 'bar', df['one'])
In [67]: df
Out[67]:
294
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
a
b
c
d
one
1
2
3
NaN
bar
1
2
3
NaN
flag
False
False
True
False
foo
bar
bar
bar
bar
one_trunc
1
2
NaN
NaN
9.2.9 Assigning New Columns in Method Chains
New in version 0.16.0.
Inspired by dplyr’s mutate verb, DataFrame has an assign() method that allows you to easily create new columns
that are potentially derived from existing columns.
In [68]: iris = read_csv('data/iris.data')
In [69]: iris.head()
Out[69]:
SepalLength SepalWidth
0
5.1
3.5
1
4.9
3.0
2
4.7
3.2
3
4.6
3.1
4
5.0
3.6
PetalLength
1.4
1.4
1.3
1.5
1.4
PetalWidth
0.2
0.2
0.2
0.2
0.2
Name
Iris-setosa
Iris-setosa
Iris-setosa
Iris-setosa
Iris-setosa
In [70]: (iris.assign(sepal_ratio = iris['SepalWidth'] / iris['SepalLength'])
....:
.head())
....:
Out[70]:
SepalLength SepalWidth PetalLength PetalWidth
Name sepal_ratio
0
5.1
3.5
1.4
0.2 Iris-setosa
0.686
1
4.9
3.0
1.4
0.2 Iris-setosa
0.612
2
4.7
3.2
1.3
0.2 Iris-setosa
0.681
3
4.6
3.1
1.5
0.2 Iris-setosa
0.674
4
5.0
3.6
1.4
0.2 Iris-setosa
0.720
Above was an example of inserting a precomputed value. We can also pass in a function of one argument to be
evalutated on the DataFrame being assigned to.
In [71]: iris.assign(sepal_ratio = lambda x: (x['SepalWidth'] /
....:
x['SepalLength'])).head()
....:
Out[71]:
SepalLength SepalWidth PetalLength PetalWidth
Name sepal_ratio
0
5.1
3.5
1.4
0.2 Iris-setosa
0.686
1
4.9
3.0
1.4
0.2 Iris-setosa
0.612
2
4.7
3.2
1.3
0.2 Iris-setosa
0.681
3
4.6
3.1
1.5
0.2 Iris-setosa
0.674
4
5.0
3.6
1.4
0.2 Iris-setosa
0.720
assign always returns a copy of the data, leaving the original DataFrame untouched.
Passing a callable, as opposed to an actual value to be inserted, is useful when you don’t have a reference to the
DataFrame at hand. This is common when using assign in chains of operations. For example, we can limit the
DataFrame to just those observations with a Sepal Length greater than 5, calculate the ratio, and plot:
In [72]: (iris.query('SepalLength > 5')
....:
.assign(SepalRatio = lambda x: x.SepalWidth / x.SepalLength,
....:
PetalRatio = lambda x: x.PetalWidth / x.PetalLength)
9.2. DataFrame
295
pandas: powerful Python data analysis toolkit, Release 0.16.1
....:
.plot(kind='scatter', x='SepalRatio', y='PetalRatio'))
....:
Out[72]:
Since a function is passed in, the function is computed on the DataFrame being assigned to. Importantly, this is the
DataFrame that’s been filtered to those rows with sepal length greater than 5. The filtering happens first, and then the
ratio calculations. This is an example where we didn’t have a reference to the filtered DataFrame available.
The function signature for assign is simply **kwargs. The keys are the column names for the new fields, and the
values are either a value to be inserted (for example, a Series or NumPy array), or a function of one argument to be
called on the DataFrame. A copy of the original DataFrame is returned, with the new values inserted.
296
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning: Since the function signature of assign is **kwargs, a dictionary, the order of the new columns in
the resulting DataFrame cannot be guaranteed to match the order you pass in. To make things predictable, items
are inserted alphabetically (by key) at the end of the DataFrame.
All expressions are computed first, and then assigned. So you can’t refer to another column being assigned in the
same call to assign. For example:
In [73]: # Don't do this, bad reference to `C`
df.assign(C = lambda x: x['A'] + x['B'],
D = lambda x: x['A'] + x['C'])
In [2]: # Instead, break it into two assigns
(df.assign(C = lambda x: x['A'] + x['B'])
.assign(D = lambda x: x['A'] + x['C']))
9.2.10 Indexing / Selection
The basics of indexing are as follows:
Operation
Select column
Select row by label
Select row by integer location
Slice rows
Select rows by boolean vector
Syntax
df[col]
df.loc[label]
df.iloc[loc]
df[5:10]
df[bool_vec]
Result
Series
Series
Series
DataFrame
DataFrame
Row selection, for example, returns a Series whose index is the columns of the DataFrame:
In [74]: df.loc['b']
Out[74]:
one
2
bar
2
flag
False
foo
bar
one_trunc
2
Name: b, dtype: object
In [75]: df.iloc[2]
Out[75]:
one
3
bar
3
flag
True
foo
bar
one_trunc
NaN
Name: c, dtype: object
For a more exhaustive treatment of more sophisticated label-based indexing and slicing, see the section on indexing.
We will address the fundamentals of reindexing / conforming to new sets of labels in the section on reindexing.
9.2.11 Data alignment and arithmetic
Data alignment between DataFrame objects automatically align on both the columns and the index (row labels).
Again, the resulting object will have the union of the column and row labels.
In [76]: df = DataFrame(randn(10, 4), columns=['A', 'B', 'C', 'D'])
9.2. DataFrame
297
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [77]: df2 = DataFrame(randn(7, 3), columns=['A', 'B', 'C'])
In [78]: df + df2
Out[78]:
A
B
C
0 -1.916 -0.986 -2.421e+00
1 0.965 1.677 3.298e-01
2 -1.662 2.197 -1.917e+00
3 -0.189 0.765 -9.522e-04
4 -1.076 0.397 -1.177e+00
5 2.810 -0.179 -5.705e-01
6 -1.227 0.196 5.312e-01
7
NaN
NaN
NaN
8
NaN
NaN
NaN
9
NaN
NaN
NaN
D
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
When doing an operation between DataFrame and Series, the default behavior is to align the Series index on the
DataFrame columns, thus broadcasting row-wise. For example:
In [79]: df - df.iloc[0]
Out[79]:
A
B
C
D
0 0.000 0.000 0.000 0.000
1 2.386 1.358 1.223 -2.107
2 2.105 1.700 1.327 -0.689
3 1.874 2.718 2.382 -0.760
4 2.199 0.966 0.826 0.093
5 4.997 1.197 1.330 -0.285
6 1.263 0.578 1.071 -0.525
7 3.463 0.632 1.063 -0.443
8 2.680 3.163 1.298 -1.818
9 1.304 0.196 3.590 -0.867
In the special case of working with time series data, if the Series is a TimeSeries (which it will be automatically if the
index contains datetime objects), and the DataFrame index also contains dates, the broadcasting will be column-wise:
In [80]: index = date_range('1/1/2000', periods=8)
In [81]: df = DataFrame(randn(8, 3), index=index, columns=list('ABC'))
In [82]: df
Out[82]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
0.063
-0.269
0.638
-0.511
1.664
0.029
-0.729
-0.048
B
-0.028
-1.578
-0.557
0.156
-0.438
0.179
-0.898
-0.876
C
0.444
1.850
-0.071
-1.076
-0.077
1.740
-0.314
0.169
In [83]: type(df['A'])
Out[83]: pandas.core.series.Series
In [84]: df - df['A']
Out[84]:
A
B
C
2000-01-01 0 -0.091 0.381
298
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
0
0
0
0
0
0
0
-1.309 2.119
-1.195 -0.709
0.668 -0.564
-2.101 -1.741
0.150 1.711
-0.169 0.415
-0.828 0.217
Warning:
df - df['A']
is now deprecated and will be removed in a future release. The preferred way to replicate this behavior is
df.sub(df['A'], axis=0)
For explicit control over the matching and broadcasting behavior, see the section on flexible binary operations.
Operations with scalars are just as you would expect:
In [85]: df * 5 + 2
Out[85]:
A
B
2000-01-01
2.314 1.858
2000-01-02
0.656 -5.888
2000-01-03
5.190 -0.783
2000-01-04 -0.557 2.781
2000-01-05 10.318 -0.189
2000-01-06
2.146 2.895
2000-01-07 -1.645 -2.490
2000-01-08
1.760 -2.378
C
4.218
11.251
1.644
-3.378
1.613
10.700
0.429
2.846
In [86]: 1 / df
Out[86]:
A
B
C
2000-01-01 15.948 -35.193
2.255
2000-01-02 -3.721 -0.634
0.540
2000-01-03
1.567 -1.797 -14.039
2000-01-04 -1.955
6.398 -0.930
2000-01-05
0.601 -2.285 -12.936
2000-01-06 34.257
5.586
0.575
2000-01-07 -1.372 -1.114 -3.183
2000-01-08 -20.802 -1.142
5.913
In [87]: df ** 4
Out[87]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
1.546e-05
5.219e-03
1.657e-01
6.841e-02
7.660e+00
7.261e-07
2.825e-01
5.341e-06
B
6.519e-07
6.195e+00
9.598e-02
5.966e-04
3.671e-02
1.027e-03
6.503e-01
5.878e-01
C
3.871e-02
1.172e+01
2.574e-05
1.339e+00
3.571e-05
9.168e+00
9.747e-03
8.178e-04
Boolean operators work as well:
9.2. DataFrame
299
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [88]: df1 = DataFrame({'a' : [1, 0, 1], 'b' : [0, 1, 1] }, dtype=bool)
In [89]: df2 = DataFrame({'a' : [0, 1, 1], 'b' : [1, 1, 0] }, dtype=bool)
In [90]: df1 & df2
Out[90]:
a
b
0 False False
1 False
True
2
True False
In [91]:
Out[91]:
a
0 True
1 True
2 True
df1 | df2
b
True
True
True
In [92]: df1 ^ df2
Out[92]:
a
b
0
True
True
1
True False
2 False
True
In [93]: -df1
Out[93]:
a
b
0 False
True
1
True False
2 False False
9.2.12 Transposing
To transpose, access the T attribute (also the transpose function), similar to an ndarray:
# only show the first 5 rows
In [94]: df[:5].T
Out[94]:
2000-01-01 2000-01-02 2000-01-03
A
0.063
-0.269
0.638
B
-0.028
-1.578
-0.557
C
0.444
1.850
-0.071
2000-01-04
-0.511
0.156
-1.076
2000-01-05
1.664
-0.438
-0.077
9.2.13 DataFrame interoperability with NumPy functions
Elementwise NumPy ufuncs (log, exp, sqrt, ...) and various other NumPy functions can be used with no issues on
DataFrame, assuming the data within are numeric:
In [95]: np.exp(df)
Out[95]:
A
B
2000-01-01 1.065 0.972
2000-01-02 0.764 0.206
2000-01-03 1.893 0.573
2000-01-04 0.600 1.169
300
C
1.558
6.361
0.931
0.341
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-05
2000-01-06
2000-01-07
2000-01-08
5.278
1.030
0.482
0.953
0.646
1.196
0.407
0.417
0.926
5.698
0.730
1.184
In [96]: np.asarray(df)
Out[96]:
array([[ 0.0627, -0.0284,
[-0.2688, -1.5776,
[ 0.6381, -0.5566,
[-0.5114, 0.1563,
[ 1.6636, -0.4377,
[ 0.0292, 0.179 ,
[-0.729 , -0.898 ,
[-0.0481, -0.8756,
0.4436],
1.8502],
-0.0712],
-1.0756],
-0.0773],
1.7401],
-0.3142],
0.1691]])
The dot method on DataFrame implements matrix multiplication:
In [97]: df.T.dot(df)
Out[97]:
A
B
C
A 4.047 -0.039 0.178
B -0.039 4.621 -2.581
C 0.178 -2.581 7.943
Similarly, the dot method on Series implements dot product:
In [98]: s1 = Series(np.arange(5,10))
In [99]: s1.dot(s1)
Out[99]: 255
DataFrame is not intended to be a drop-in replacement for ndarray as its indexing semantics are quite different in
places from a matrix.
9.2.14 Console display
Very large DataFrames will be truncated to display them in the console. You can also get a summary using info().
(Here I am reading a CSV version of the baseball dataset from the plyr R package):
In [100]: baseball = read_csv('data/baseball.csv')
In [101]: print(baseball)
id
player year
0
88641 womacto01 2006
1
88643 schilcu01 2006
..
...
...
...
98 89533
aloumo01 2007
99 89534 alomasa02 2007
stint
2
1
...
1
1
...
...
...
...
...
...
hbp sh
0 3
0 0
.. ..
2 0
0 0
sf
0
0
..
3
0
gidp
0
0
...
13
0
[100 rows x 23 columns]
In [102]: baseball.info()
Int64Index: 100 entries, 0 to 99
Data columns (total 23 columns):
id
100 non-null int64
player
100 non-null object
9.2. DataFrame
301
pandas: powerful Python data analysis toolkit, Release 0.16.1
year
100 non-null int64
stint
100 non-null int64
team
100 non-null object
lg
100 non-null object
g
100 non-null int64
ab
100 non-null int64
r
100 non-null int64
h
100 non-null int64
X2b
100 non-null int64
X3b
100 non-null int64
hr
100 non-null int64
rbi
100 non-null float64
sb
100 non-null float64
cs
100 non-null float64
bb
100 non-null int64
so
100 non-null float64
ibb
100 non-null float64
hbp
100 non-null float64
sh
100 non-null float64
sf
100 non-null float64
gidp
100 non-null float64
dtypes: float64(9), int64(11), object(3)
memory usage: 17.6+ KB
However, using to_string will return a string representation of the DataFrame in tabular form, though it won’t
always fit the console width:
In [103]: print(baseball.iloc[-20:, :12].to_string())
id
player year stint team lg
g
ab
80 89474 finlest01 2007
1 COL NL
43
94
81 89480 embreal01 2007
1 OAK AL
4
0
82 89481 edmonji01 2007
1 SLN NL 117 365
83 89482 easleda01 2007
1 NYN NL
76 193
84 89489 delgaca01 2007
1 NYN NL 139 538
85 89493 cormirh01 2007
1 CIN NL
6
0
86 89494 coninje01 2007
2 NYN NL
21
41
87 89495 coninje01 2007
1 CIN NL
80 215
88 89497 clemero02 2007
1 NYA AL
2
2
89 89498 claytro01 2007
2 BOS AL
8
6
90 89499 claytro01 2007
1 TOR AL
69 189
91 89501 cirilje01 2007
2 ARI NL
28
40
92 89502 cirilje01 2007
1 MIN AL
50 153
93 89521 bondsba01 2007
1 SFN NL 126 340
94 89523 biggicr01 2007
1 HOU NL 141 517
95 89525 benitar01 2007
2 FLO NL
34
0
96 89526 benitar01 2007
1 SFN NL
19
0
97 89530 ausmubr01 2007
1 HOU NL 117 349
98 89533
aloumo01 2007
1 NYN NL
87 328
99 89534 alomasa02 2007
1 NYN NL
8
22
r
9
0
39
24
71
0
2
23
0
1
23
6
18
75
68
0
0
38
51
1
h
17
0
92
54
139
0
8
57
1
0
48
8
40
94
130
0
0
82
112
3
X2b
3
0
15
6
30
0
2
11
0
0
14
4
9
14
31
0
0
16
19
1
X3b
0
0
2
0
0
0
0
1
0
0
0
0
2
0
3
0
0
3
1
0
New since 0.10.0, wide DataFrames will now be printed across multiple rows by default:
In [104]: DataFrame(randn(3, 12))
Out[104]:
0
1
2
3
4
5
6
0 1.225021 -0.528620 0.448676 0.619107 -1.199110 -0.949097 2.169523
1 -1.753617 0.992384 -0.505601 -0.599848 0.133585 0.008836 -1.767710
2 -0.461585 -1.321106 1.745476 1.445100 0.991037 -0.860733 -0.870661
302
\
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
7
8
9
10
11
0 0.302230 0.919516 0.657436 0.262574 -0.804798
1 0.700112 -0.020773 -0.302481 0.347869 0.179123
2 -0.117845 -0.046266 2.095649 -0.524324 -0.610555
You can change how much to print on a single row by setting the display.width option:
In [105]: set_option('display.width', 40) # default is 80
In [106]: DataFrame(randn(3, 12))
Out[106]:
0
1
2
\
0 -1.280951 1.472585 -1.001914
1 0.130529 -1.603771 -0.128830
2 -1.084566 -0.515272 1.367586
3
4
5
0 1.044770 -0.050668 -0.013289
1 -1.869301 -0.232977 -0.139801
2 0.963500 0.224105 -0.020051
\
6
7
8
0 -0.291893 2.029038 -1.117195
1 -1.083341 -0.357234 -0.818199
2 0.524663 0.351081 -1.574209
\
9
10
11
0 1.598577 -0.397325 0.151653
1 -0.886885 1.238885 -1.639274
2 -0.486856 -0.545888 -0.927076
You can also disable this feature via the expand_frame_repr option. This will print the table in one block.
9.2.15 DataFrame column attribute access and IPython completion
If a DataFrame column label is a valid Python variable name, the column can be accessed like attributes:
In [107]: df = DataFrame({'foo1' : np.random.randn(5),
.....:
'foo2' : np.random.randn(5)})
.....:
In [108]: df
Out[108]:
foo1
foo2
0 0.909160 1.360298
1 -0.667763 -1.603624
2 -0.101656 -1.648929
3 1.189682 0.145121
4 -0.090648 -2.536359
In [109]: df.foo1
Out[109]:
0
0.909160
1
-0.667763
2
-0.101656
3
1.189682
4
-0.090648
Name: foo1, dtype: float64
9.2. DataFrame
303
pandas: powerful Python data analysis toolkit, Release 0.16.1
The columns are also connected to the IPython completion mechanism so they can be tab-completed:
In [5]: df.fo
df.foo1 df.foo2
9.3 Panel
Panel is a somewhat less-used, but still important container for 3-dimensional data. The term panel data is derived
from econometrics and is partially responsible for the name pandas: pan(el)-da(ta)-s. The names for the 3 axes are
intended to give some semantic meaning to describing operations involving panel data and, in particular, econometric
analysis of panel data. However, for the strict purposes of slicing and dicing a collection of DataFrame objects, you
may find the axis names slightly arbitrary:
• items: axis 0, each item corresponds to a DataFrame contained inside
• major_axis: axis 1, it is the index (rows) of each of the DataFrames
• minor_axis: axis 2, it is the columns of each of the DataFrames
Construction of Panels works about like you would expect:
9.3.1 From 3D ndarray with optional axis labels
In [110]: wp = Panel(randn(2, 5, 4), items=['Item1', 'Item2'],
.....:
major_axis=date_range('1/1/2000', periods=5),
.....:
minor_axis=['A', 'B', 'C', 'D'])
.....:
In [111]: wp
Out[111]:
Dimensions: 2 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to D
9.3.2 From dict of DataFrame objects
In [112]: data = {'Item1' : DataFrame(randn(4, 3)),
.....:
'Item2' : DataFrame(randn(4, 2))}
.....:
In [113]: Panel(data)
Out[113]:
Dimensions: 2 (items) x 4 (major_axis) x 3 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 0 to 3
Minor_axis axis: 0 to 2
Note that the values in the dict need only be convertible to DataFrame. Thus, they can be any of the other valid
inputs to DataFrame as per above.
One helpful factory method is Panel.from_dict, which takes a dictionary of DataFrames as above, and the
following named parameters:
304
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
Parameter
intersect
orient
Default
False
items
Description
drops elements whose indices do not align
use minor to use DataFrames’ columns as panel items
For example, compare to the construction above:
In [114]: Panel.from_dict(data, orient='minor')
Out[114]:
Dimensions: 3 (items) x 4 (major_axis) x 2 (minor_axis)
Items axis: 0 to 2
Major_axis axis: 0 to 3
Minor_axis axis: Item1 to Item2
Orient is especially useful for mixed-type DataFrames. If you pass a dict of DataFrame objects with mixed-type
columns, all of the data will get upcasted to dtype=object unless you pass orient=’minor’:
In [115]: df = DataFrame({'a': ['foo', 'bar', 'baz'],
.....:
'b': np.random.randn(3)})
.....:
In [116]: df
Out[116]:
a
b
0 foo -1.264356
1 bar -0.497629
2 baz 1.789719
In [117]: data = {'item1': df, 'item2': df}
In [118]: panel = Panel.from_dict(data, orient='minor')
In [119]: panel['a']
Out[119]:
item1 item2
0
foo
foo
1
bar
bar
2
baz
baz
In [120]: panel['b']
Out[120]:
item1
item2
0 -1.264356 -1.264356
1 -0.497629 -0.497629
2 1.789719 1.789719
In [121]: panel['b'].dtypes
Out[121]:
item1
float64
item2
float64
dtype: object
Note: Unfortunately Panel, being less commonly used than Series and DataFrame, has been slightly neglected featurewise. A number of methods and options available in DataFrame are not available in Panel. This will get worked on,
of course, in future releases. And faster if you join me in working on the codebase.
9.3. Panel
305
pandas: powerful Python data analysis toolkit, Release 0.16.1
9.3.3 From DataFrame using to_panel method
This method was introduced in v0.7 to replace LongPanel.to_long, and converts a DataFrame with a two-level
index to a Panel.
In [122]: midx = MultiIndex(levels=[['one', 'two'], ['x','y']], labels=[[1,1,0,0],[1,0,1,0]])
In [123]: df = DataFrame({'A' : [1, 2, 3, 4], 'B': [5, 6, 7, 8]}, index=midx)
In [124]: df.to_panel()
Out[124]:
Dimensions: 2 (items) x 2 (major_axis) x 2 (minor_axis)
Items axis: A to B
Major_axis axis: one to two
Minor_axis axis: x to y
9.3.4 Item selection / addition / deletion
Similar to DataFrame functioning as a dict of Series, Panel is like a dict of DataFrames:
In [125]: wp['Item1']
Out[125]:
A
B
C
D
2000-01-01 0.835993 -0.621868 -0.173710 -0.174326
2000-01-02 -0.354356 2.090183 -0.736019 -1.250412
2000-01-03 -0.581326 -0.244477 0.917119 0.611695
2000-01-04 -1.576078 -0.528562 -0.704643 -0.481453
2000-01-05 1.085093 -1.229749 2.295679 -1.016910
In [126]: wp['Item3'] = wp['Item1'] / wp['Item2']
The API for insertion and deletion is the same as for DataFrame. And as with DataFrame, if the item is a valid python
identifier, you can access it as an attribute and tab-complete it in IPython.
9.3.5 Transposing
A Panel can be rearranged using its transpose method (which does not make a copy by default unless the data are
heterogeneous):
In [127]: wp.transpose(2, 0, 1)
Out[127]:
Dimensions: 4 (items) x 3 (major_axis) x 5 (minor_axis)
Items axis: A to D
Major_axis axis: Item1 to Item3
Minor_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
9.3.6 Indexing / Selection
Operation
Select item
Get slice at major_axis label
Get slice at minor_axis label
306
Syntax
wp[item]
wp.major_xs(val)
wp.minor_xs(val)
Result
DataFrame
DataFrame
DataFrame
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
For example, using the earlier example data, we could do:
In [128]: wp['Item1']
Out[128]:
A
B
C
D
2000-01-01 0.835993 -0.621868 -0.173710 -0.174326
2000-01-02 -0.354356 2.090183 -0.736019 -1.250412
2000-01-03 -0.581326 -0.244477 0.917119 0.611695
2000-01-04 -1.576078 -0.528562 -0.704643 -0.481453
2000-01-05 1.085093 -1.229749 2.295679 -1.016910
In [129]: wp.major_xs(wp.major_axis[2])
Out[129]:
Item1
Item2
Item3
A -0.581326 -1.271582 0.457167
B -0.244477 -0.861256 0.283861
C 0.917119 -0.597879 -1.533955
D 0.611695 -0.118700 -5.153265
In [130]: wp.minor_axis
Out[130]: Index([u'A', u'B', u'C', u'D'], dtype='object')
In [131]: wp.minor_xs('C')
Out[131]:
Item1
Item2
2000-01-01 -0.173710 2.381645
2000-01-02 -0.736019 -2.413161
2000-01-03 0.917119 -0.597879
2000-01-04 -0.704643 -1.536019
2000-01-05 2.295679 0.181524
Item3
-0.072937
0.305002
-1.533955
0.458746
12.646732
9.3.7 Squeezing
Another way to change the dimensionality of an object is to squeeze a 1-len object, similar to wp[’Item1’]
In [132]: wp.reindex(items=['Item1']).squeeze()
Out[132]:
A
B
C
D
2000-01-01 0.835993 -0.621868 -0.173710 -0.174326
2000-01-02 -0.354356 2.090183 -0.736019 -1.250412
2000-01-03 -0.581326 -0.244477 0.917119 0.611695
2000-01-04 -1.576078 -0.528562 -0.704643 -0.481453
2000-01-05 1.085093 -1.229749 2.295679 -1.016910
In [133]: wp.reindex(items=['Item1'],minor=['B']).squeeze()
Out[133]:
2000-01-01
-0.621868
2000-01-02
2.090183
2000-01-03
-0.244477
2000-01-04
-0.528562
2000-01-05
-1.229749
Freq: D, Name: B, dtype: float64
9.3. Panel
307
pandas: powerful Python data analysis toolkit, Release 0.16.1
9.3.8 Conversion to DataFrame
A Panel can be represented in 2D form as a hierarchically indexed DataFrame. See the section hierarchical indexing
for more on this. To convert a Panel to a DataFrame, use the to_frame method:
In [134]: panel = Panel(np.random.randn(3, 5, 4), items=['one', 'two', 'three'],
.....:
major_axis=date_range('1/1/2000', periods=5),
.....:
minor_axis=['a', 'b', 'c', 'd'])
.....:
In [135]: panel.to_frame()
Out[135]:
one
major
minor
2000-01-01 a
0.445900
b
-0.574496
c
0.872979
d
0.297255
2000-01-02 a
-1.022617
b
1.091870
c
1.831444
d
1.271808
2000-01-03 a
-0.472876
b
-0.279340
c
0.495966
d
0.367858
2000-01-04 a
-1.530917
b
-0.285890
c
0.943062
d
1.361752
2000-01-05 a
0.210373
b
-1.945608
c
2.532409
d
0.373819
two
three
-1.286198
-0.407154
0.068084
-2.157051
-0.443982
-0.881639
0.851834
-1.352515
0.228761
0.416858
0.301709
0.569010
-0.047619
0.413370
0.573056
-0.154419
0.987044
0.063191
0.439086
1.657475
-1.023189
0.591682
-0.008919
-0.415572
-0.772683
-0.516197
0.626655
0.269623
1.709250
-0.830728
-0.290244
-1.588782
0.639406
1.055533
-0.260898
-0.289725
0.279621
0.454423
-0.065750
1.465709
9.4 Panel4D (Experimental)
Panel4D is a 4-Dimensional named container very much like a Panel, but having 4 named dimensions. It is
intended as a test bed for more N-Dimensional named containers.
• labels: axis 0, each item corresponds to a Panel contained inside
• items: axis 1, each item corresponds to a DataFrame contained inside
• major_axis: axis 2, it is the index (rows) of each of the DataFrames
• minor_axis: axis 3, it is the columns of each of the DataFrames
Panel4D is a sub-class of Panel, so most methods that work on Panels are applicable to Panel4D. The following
methods are disabled:
• join , to_frame , to_excel , to_sparse , groupby
Construction of Panel4D works in a very similar manner to a Panel
308
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
9.4.1 From 4D ndarray with optional axis labels
In [136]: p4d = Panel4D(randn(2, 2, 5, 4),
.....:
labels=['Label1','Label2'],
.....:
items=['Item1', 'Item2'],
.....:
major_axis=date_range('1/1/2000', periods=5),
.....:
minor_axis=['A', 'B', 'C', 'D'])
.....:
In [137]: p4d
Out[137]:
Dimensions: 2 (labels) x 2 (items) x 5 (major_axis) x 4 (minor_axis)
Labels axis: Label1 to Label2
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to D
9.4.2 From dict of Panel objects
In [138]: data = { 'Label1' : Panel({ 'Item1' : DataFrame(randn(4, 3)) }),
.....:
'Label2' : Panel({ 'Item2' : DataFrame(randn(4, 2)) }) }
.....:
In [139]: Panel4D(data)
Out[139]:
Dimensions: 2 (labels) x 2 (items) x 4 (major_axis) x 3 (minor_axis)
Labels axis: Label1 to Label2
Items axis: Item1 to Item2
Major_axis axis: 0 to 3
Minor_axis axis: 0 to 2
Note that the values in the dict need only be convertible to Panels. Thus, they can be any of the other valid inputs to
Panel as per above.
9.4.3 Slicing
Slicing works in a similar manner to a Panel. [] slices the first dimension. .ix allows you to slice arbitrarily and get
back lower dimensional objects
In [140]: p4d['Label1']
Out[140]:
Dimensions: 2 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to D
4D -> Panel
In [141]: p4d.ix[:,:,:,'A']
Out[141]:
Dimensions: 2 (items) x 2 (major_axis) x 5 (minor_axis)
Items axis: Label1 to Label2
9.4. Panel4D (Experimental)
309
pandas: powerful Python data analysis toolkit, Release 0.16.1
Major_axis axis: Item1 to Item2
Minor_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
4D -> DataFrame
In [142]: p4d.ix[:,:,0,'A']
Out[142]:
Label1
Label2
Item1 1.127489 0.015494
Item2 -1.650400 0.130533
4D -> Series
In [143]: p4d.ix[:,0,0,'A']
Out[143]:
Label1
1.127489
Label2
0.015494
Name: A, dtype: float64
9.4.4 Transposing
A Panel4D can be rearranged using its transpose method (which does not make a copy by default unless the data
are heterogeneous):
In [144]: p4d.transpose(3, 2, 1, 0)
Out[144]:
Dimensions: 4 (labels) x 5 (items) x 2 (major_axis) x 2 (minor_axis)
Labels axis: A to D
Items axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Major_axis axis: Item1 to Item2
Minor_axis axis: Label1 to Label2
9.5 PanelND (Experimental)
PanelND is a module with a set of factory functions to enable a user to construct N-dimensional named containers like
Panel4D, with a custom set of axis labels. Thus a domain-specific container can easily be created.
The following creates a Panel5D. A new panel type object must be sliceable into a lower dimensional object. Here we
slice to a Panel4D.
In [145]: from pandas.core import panelnd
In [146]: Panel5D = panelnd.create_nd_panel_factory(
.....:
klass_name
= 'Panel5D',
.....:
orders = [ 'cool', 'labels','items','major_axis','minor_axis'],
.....:
slices = { 'labels' : 'labels', 'items' : 'items',
.....:
'major_axis' : 'major_axis', 'minor_axis' : 'minor_axis' },
.....:
slicer = Panel4D,
.....:
aliases = { 'major' : 'major_axis', 'minor' : 'minor_axis' },
.....:
stat_axis
= 2)
.....:
In [147]: p5d = Panel5D(dict(C1 = p4d))
310
Chapter 9. Intro to Data Structures
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [148]: p5d
Out[148]:
Dimensions: 1 (cool) x 2 (labels) x 2 (items) x 5 (major_axis) x 4 (minor_axis)
Cool axis: C1 to C1
Labels axis: Label1 to Label2
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to D
# print a slice of our 5D
In [149]: p5d.ix['C1',:,:,0:3,:]
Out[149]:
Dimensions: 2 (labels) x 2 (items) x 3 (major_axis) x 4 (minor_axis)
Labels axis: Label1 to Label2
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-03 00:00:00
Minor_axis axis: A to D
# transpose it
In [150]: p5d.transpose(1,2,3,4,0)
Out[150]:
Dimensions: 2 (cool) x 2 (labels) x 5 (items) x 4 (major_axis) x 1 (minor_axis)
Cool axis: Label1 to Label2
Labels axis: Item1 to Item2
Items axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Major_axis axis: A to D
Minor_axis axis: C1 to C1
# look at the shape & dim
In [151]: p5d.shape
Out[151]: (1, 2, 2, 5, 4)
In [152]: p5d.ndim
Out[152]: 5
9.5. PanelND (Experimental)
311
pandas: powerful Python data analysis toolkit, Release 0.16.1
312
Chapter 9. Intro to Data Structures
CHAPTER
TEN
ESSENTIAL BASIC FUNCTIONALITY
Here we discuss a lot of the essential functionality common to the pandas data structures. Here’s how to create some
of the objects used in the examples from the previous section:
In [1]: index = date_range('1/1/2000', periods=8)
In [2]: s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])
In [3]: df = DataFrame(randn(8, 3), index=index,
...:
columns=['A', 'B', 'C'])
...:
In [4]: wp = Panel(randn(2, 5, 4), items=['Item1', 'Item2'],
...:
major_axis=date_range('1/1/2000', periods=5),
...:
minor_axis=['A', 'B', 'C', 'D'])
...:
10.1 Head and Tail
To view a small sample of a Series or DataFrame object, use the head() and tail() methods. The default number
of elements to display is five, but you may pass a custom number.
In [5]: long_series = Series(randn(1000))
In [6]: long_series.head()
Out[6]:
0
-0.305384
1
-0.479195
2
0.095031
3
-0.270099
4
-0.707140
dtype: float64
In [7]: long_series.tail(3)
Out[7]:
997
0.588446
998
0.026465
999
-1.728222
dtype: float64
313
pandas: powerful Python data analysis toolkit, Release 0.16.1
10.2 Attributes and the raw ndarray(s)
pandas objects have a number of attributes enabling you to access the metadata
• shape: gives the axis dimensions of the object, consistent with ndarray
• Axis labels
– Series: index (only axis)
– DataFrame: index (rows) and columns
– Panel: items, major_axis, and minor_axis
Note, these attributes can be safely assigned to!
In [8]: df[:2]
Out[8]:
2000-01-01
2000-01-02
A
B
C
0.187483 -1.933946 0.377312
0.734122 2.141616 -0.011225
In [9]: df.columns = [x.lower() for x in df.columns]
In [10]: df
Out[10]:
a
b
2000-01-01 0.187483 -1.933946
2000-01-02 0.734122 2.141616
2000-01-03 0.048869 -1.360687
2000-01-04 -0.859661 -0.231595
2000-01-05 -1.296337 0.150680
2000-01-06 0.571764 1.555563
2000-01-07 0.535420 -1.032853
2000-01-08 1.304124 1.449735
c
0.377312
-0.011225
-0.479010
-0.527750
0.123836
-0.823761
1.469725
0.203109
To get the actual data inside a data structure, one need only access the values property:
In [11]: s.values
Out[11]: array([ 0.1122,
0.8717, -0.8161, -0.7849,
In [12]: df.values
Out[12]:
array([[ 0.1875, -1.9339,
[ 0.7341, 2.1416,
[ 0.0489, -1.3607,
[-0.8597, -0.2316,
[-1.2963, 0.1507,
[ 0.5718, 1.5556,
[ 0.5354, -1.0329,
[ 1.3041, 1.4497,
0.3773],
-0.0112],
-0.479 ],
-0.5278],
0.1238],
-0.8238],
1.4697],
0.2031]])
In [13]: wp.values
Out[13]:
array([[[-1.032 , 0.9698,
[-0.9388, 0.6691,
[ 0.6804, -0.3084,
[-1.9936, -1.9274,
[ 0.5511, 3.0593,
[[ 0.9357,
314
1.0307])
-0.9627, 1.3821],
-0.4336, -0.2736],
-0.2761, -1.8212],
-2.0279, 1.625 ],
0.4553, -0.0307]],
1.0612, -2.1079,
0.1999],
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
[ 0.3236, -0.6416, -0.5875, 0.0539],
[ 0.1949, -0.382 , 0.3186, 2.0891],
[-0.7283, -0.0903, -0.7482, 1.3189],
[-2.0298, 0.7927, 0.461 , -0.5427]]])
If a DataFrame or Panel contains homogeneously-typed data, the ndarray can actually be modified in-place, and the
changes will be reflected in the data structure. For heterogeneous data (e.g. some of the DataFrame’s columns are not
all the same dtype), this will not be the case. The values attribute itself, unlike the axis labels, cannot be assigned to.
Note: When working with heterogeneous data, the dtype of the resulting ndarray will be chosen to accommodate all
of the data involved. For example, if strings are involved, the result will be of object dtype. If there are only floats and
integers, the resulting array will be of float dtype.
10.3 Accelerated operations
pandas has support for accelerating certain types of binary numerical and boolean operations using the numexpr
library (starting in 0.11.0) and the bottleneck libraries.
These libraries are especially useful when dealing with large data sets, and provide large speedups. numexpr uses
smart chunking, caching, and multiple cores. bottleneck is a set of specialized cython routines that are especially
fast when dealing with arrays that have nans.
Here is a sample (using 100 column x 100,000 row DataFrames):
Operation
df1 > df2
df1 * df2
df1 + df2
0.11.0 (ms)
13.32
21.71
22.04
Prior Version (ms)
125.35
36.63
36.50
Ratio to Prior
0.1063
0.5928
0.6039
You are highly encouraged to install both libraries. See the section Recommended Dependencies for more installation
info.
10.4 Flexible binary operations
With binary operations between pandas data structures, there are two key points of interest:
• Broadcasting behavior between higher- (e.g. DataFrame) and lower-dimensional (e.g. Series) objects.
• Missing data in computations
We will demonstrate how to manage these issues independently, though they can be handled simultaneously.
10.4.1 Matching / broadcasting behavior
DataFrame has the methods add(), sub(), mul(), div() and related functions radd(), rsub(), ... for carrying out binary operations. For broadcasting behavior, Series input is of primary interest. Using these functions, you
can use to either match on the index or columns via the axis keyword:
In [14]: df = DataFrame({'one' : Series(randn(3), index=['a', 'b', 'c']),
....:
'two' : Series(randn(4), index=['a', 'b', 'c', 'd']),
....:
'three' : Series(randn(3), index=['b', 'c', 'd'])})
....:
10.3. Accelerated operations
315
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [15]: df
Out[15]:
one
three
two
a -0.626544
NaN -0.351587
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
d
NaN 1.124472 -1.101558
In [16]: row = df.ix[1]
In [17]: column = df['two']
In [18]: df.sub(row, axis='columns')
Out[18]:
one
three
two
a -0.487650
NaN -1.487837
b 0.000000 0.000000 0.000000
c 0.150512 0.639504 -1.585038
d
NaN 1.301762 -2.237808
In [19]: df.sub(row, axis=1)
Out[19]:
one
three
two
a -0.487650
NaN -1.487837
b 0.000000 0.000000 0.000000
c 0.150512 0.639504 -1.585038
d
NaN 1.301762 -2.237808
In [20]: df.sub(column, axis='index')
Out[20]:
one
three two
a -0.274957
NaN
0
b -1.275144 -1.313539
0
c 0.460406 0.911003
0
d
NaN 2.226031
0
In [21]: df.sub(column, axis=0)
Out[21]:
one
three two
a -0.274957
NaN
0
b -1.275144 -1.313539
0
c 0.460406 0.911003
0
d
NaN 2.226031
0
Furthermore you can align a level of a multi-indexed DataFrame with a Series.
In [22]: dfmi = df.copy()
In [23]: dfmi.index = MultiIndex.from_tuples([(1,'a'),(1,'b'),(1,'c'),(2,'a')],
....:
names=['first','second'])
....:
In [24]: dfmi.sub(column, axis=0,
Out[24]:
one
three
first second
1
a
-0.274957
NaN
b
-1.275144 -1.313539
c
0.460406 0.911003
316
level='second')
two
0.000000
0.000000
0.000000
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
a
NaN
1.476060 -0.749971
With Panel, describing the matching behavior is a bit more difficult, so the arithmetic methods instead (and perhaps
confusingly?) give you the option to specify the broadcast axis. For example, suppose we wished to demean the data
over a particular axis. This can be accomplished by taking the mean over an axis and broadcasting over the same axis:
In [25]: major_mean = wp.mean(axis='major')
In [26]: major_mean
Out[26]:
Item1
Item2
A -0.546569 -0.260774
B 0.492478 0.147993
C -0.649010 -0.532794
D 0.176307 0.623812
In [27]: wp.sub(major_mean, axis='major')
Out[27]:
Dimensions: 2 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to D
And similarly for axis="items" and axis="minor".
Note: I could be convinced to make the axis argument in the DataFrame methods match the broadcasting behavior of
Panel. Though it would require a transition period so users can change their code...
10.4.2 Missing data / operations with fill values
In Series and DataFrame (though not yet in Panel), the arithmetic functions have the option of inputting a fill_value,
namely a value to substitute when at most one of the values at a location are missing. For example, when adding two
DataFrame objects, you may wish to treat NaN as 0 unless both DataFrames are missing that value, in which case the
result will be NaN (you can later replace NaN with some other value using fillna if you wish).
In [28]: df
Out[28]:
one
three
two
a -0.626544
NaN -0.351587
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
d
NaN 1.124472 -1.101558
In [29]: df2
Out[29]:
one
three
two
a -0.626544 1.000000 -0.351587
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
d
NaN 1.124472 -1.101558
In [30]: df + df2
Out[30]:
one
three
two
a -1.253088
NaN -0.703174
10.4. Flexible binary operations
317
pandas: powerful Python data analysis toolkit, Release 0.16.1
b -0.277789 -0.354579 2.272499
c 0.023235 0.924429 -0.897577
d
NaN 2.248945 -2.203116
In [31]: df.add(df2, fill_value=0)
Out[31]:
one
three
two
a -1.253088 1.000000 -0.703174
b -0.277789 -0.354579 2.272499
c 0.023235 0.924429 -0.897577
d
NaN 2.248945 -2.203116
10.4.3 Flexible Comparisons
Starting in v0.8, pandas introduced binary comparison methods eq, ne, lt, gt, le, and ge to Series and DataFrame whose
behavior is analogous to the binary arithmetic operations described above:
In [32]: df.gt(df2)
Out[32]:
one three
two
a False False False
b False False False
c False False False
d False False False
In [33]: df2.ne(df)
Out[33]:
one three
two
a False
True False
b False False False
c False False False
d
True False False
These operations produce a pandas object the same type as the left-hand-side input that if of dtype bool. These
boolean objects can be used in indexing operations, see here
10.4.4 Boolean Reductions
You can apply the reductions: empty, any(), all(), and bool() to provide a way to summarize a boolean result.
In [34]: (df>0).all()
Out[34]:
one
False
three
False
two
False
dtype: bool
In [35]: (df>0).any()
Out[35]:
one
True
three
True
two
True
dtype: bool
You can reduce to a final boolean value.
318
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [36]: (df>0).any().any()
Out[36]: True
You can test if a pandas object is empty, via the empty property.
In [37]: df.empty
Out[37]: False
In [38]: DataFrame(columns=list('ABC')).empty
Out[38]: True
To evaluate single-element pandas objects in a boolean context, use the method bool():
In [39]: Series([True]).bool()
Out[39]: True
In [40]: Series([False]).bool()
Out[40]: False
In [41]: DataFrame([[True]]).bool()
Out[41]: True
In [42]: DataFrame([[False]]).bool()
Out[42]: False
Warning: You might be tempted to do the following:
>>>if df:
...
Or
>>> df and df2
These both will raise as you are trying to compare multiple values.
ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().
See gotchas for a more detailed discussion.
10.4.5 Comparing if objects are equivalent
Often you may find there is more than one way to compute the same result. As a simple example, consider df+df and
df*2. To test that these two computations produce the same result, given the tools shown above, you might imagine
using (df+df == df*2).all(). But in fact, this expression is False:
In [43]: df+df == df*2
Out[43]:
one three
two
a
True False True
b
True
True True
c
True
True True
d False
True True
In [44]: (df+df == df*2).all()
Out[44]:
one
False
10.4. Flexible binary operations
319
pandas: powerful Python data analysis toolkit, Release 0.16.1
three
False
two
True
dtype: bool
Notice that the boolean DataFrame df+df == df*2 contains some False values! That is because NaNs do not
compare as equals:
In [45]: np.nan == np.nan
Out[45]: False
So, as of v0.13.1, NDFrames (such as Series, DataFrames, and Panels) have an equals() method for testing equality,
with NaNs in corresponding locations treated as equal.
In [46]: (df+df).equals(df*2)
Out[46]: True
Note that the Series or DataFrame index needs to be in the same order for equality to be True:
In [47]: df1 = DataFrame({'col':['foo', 0, np.nan]})
In [48]: df2 = DataFrame({'col':[np.nan, 0, 'foo']}, index=[2,1,0])
In [49]: df1.equals(df2)
Out[49]: False
In [50]: df1.equals(df2.sort())
Out[50]: True
10.4.6 Combining overlapping data sets
A problem occasionally arising is the combination of two similar data sets where values in one are preferred over the
other. An example would be two data series representing a particular economic indicator where one is considered to
be of “higher quality”. However, the lower quality series might extend further back in history or have more complete
data coverage. As such, we would like to combine two DataFrame objects where missing values in one DataFrame
are conditionally filled with like-labeled values from the other DataFrame. The function implementing this operation
is combine_first(), which we illustrate:
In [51]: df1 = DataFrame({'A' : [1., np.nan, 3., 5., np.nan],
....:
'B' : [np.nan, 2., 3., np.nan, 6.]})
....:
In [52]: df2 = DataFrame({'A' : [5., 2., 4., np.nan, 3., 7.],
....:
'B' : [np.nan, np.nan, 3., 4., 6., 8.]})
....:
In [53]: df1
Out[53]:
A
B
0
1 NaN
1 NaN
2
2
3
3
3
5 NaN
4 NaN
6
In [54]: df2
Out[54]:
A
B
320
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
5 NaN
1
2 NaN
2
4
3
3 NaN
4
4
3
6
5
7
8
In [55]: df1.combine_first(df2)
Out[55]:
A
B
0 1 NaN
1 2
2
2 3
3
3 5
4
4 3
6
5 7
8
10.4.7 General DataFrame Combine
The combine_first() method above calls the more general DataFrame method combine(). This method takes
another DataFrame and a combiner function, aligns the input DataFrame and then passes the combiner function pairs
of Series (i.e., columns whose names are the same).
So, for instance, to reproduce combine_first() as above:
In [56]: combiner = lambda x, y: np.where(isnull(x), y, x)
In [57]: df1.combine(df2, combiner)
Out[57]:
A
B
0 1 NaN
1 2
2
2 3
3
3 5
4
4 3
6
5 7
8
10.5 Descriptive statistics
A large number of methods for computing descriptive statistics and other related operations on Series, DataFrame,
and Panel. Most of these are aggregations (hence producing a lower-dimensional result) like sum(), mean(), and
quantile(), but some of them, like cumsum() and cumprod(), produce an object of the same size. Generally
speaking, these methods take an axis argument, just like ndarray.{sum, std, ...}, but the axis can be specified by name
or integer:
• Series: no axis argument needed
• DataFrame: “index” (axis=0, default), “columns” (axis=1)
• Panel: “items” (axis=0), “major” (axis=1, default), “minor” (axis=2)
For example:
In [58]: df
Out[58]:
one
three
10.5. Descriptive statistics
two
321
pandas: powerful Python data analysis toolkit, Release 0.16.1
a -0.626544
NaN -0.351587
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
d
NaN 1.124472 -1.101558
In [59]: df.mean(0)
Out[59]:
one
-0.251274
three
0.469799
two
-0.191421
dtype: float64
In [60]: df.mean(1)
Out[60]:
a
-0.489066
b
0.273355
c
0.008348
d
0.011457
dtype: float64
All such methods have a skipna option signaling whether to exclude missing data (True by default):
In [61]: df.sum(0, skipna=False)
Out[61]:
one
NaN
three
NaN
two
-0.765684
dtype: float64
In [62]: df.sum(axis=1, skipna=True)
Out[62]:
a
-0.978131
b
0.820066
c
0.025044
d
0.022914
dtype: float64
Combined with the broadcasting / arithmetic behavior, one can describe various statistical procedures, like standardization (rendering data zero mean and standard deviation 1), very concisely:
In [63]: ts_stand = (df - df.mean()) / df.std()
In [64]: ts_stand.std()
Out[64]:
one
1
three
1
two
1
dtype: float64
In [65]: xs_stand = df.sub(df.mean(1), axis=0).div(df.std(1), axis=0)
In [66]: xs_stand.std(1)
Out[66]:
a
1
b
1
c
1
d
1
dtype: float64
322
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
Note that methods like cumsum() and cumprod() preserve the location of NA values:
In [67]: df.cumsum()
Out[67]:
one
three
two
a -0.626544
NaN -0.351587
b -0.765438 -0.177289 0.784662
c -0.753821 0.284925 0.335874
d
NaN 1.409398 -0.765684
Here is a quick reference summary table of common functions. Each also takes an optional level parameter which
applies only if the object has a hierarchical index.
Function
count
sum
mean
mad
median
min
max
mode
abs
prod
std
var
sem
skew
kurt
quantile
cumsum
cumprod
cummax
cummin
Description
Number of non-null observations
Sum of values
Mean of values
Mean absolute deviation
Arithmetic median of values
Minimum
Maximum
Mode
Absolute Value
Product of values
Unbiased standard deviation
Unbiased variance
Unbiased standard error of the mean
Unbiased skewness (3rd moment)
Unbiased kurtosis (4th moment)
Sample quantile (value at %)
Cumulative sum
Cumulative product
Cumulative maximum
Cumulative minimum
Note that by chance some NumPy methods, like mean, std, and sum, will exclude NAs on Series input by default:
In [68]: np.mean(df['one'])
Out[68]: -0.25127365175839511
In [69]: np.mean(df['one'].values)
Out[69]: nan
Series also has a method nunique() which will return the number of unique non-null values:
In [70]: series = Series(randn(500))
In [71]: series[20:500] = np.nan
In [72]: series[10:20]
= 5
In [73]: series.nunique()
Out[73]: 11
10.5. Descriptive statistics
323
pandas: powerful Python data analysis toolkit, Release 0.16.1
10.5.1 Summarizing data: describe
There is a convenient describe() function which computes a variety of summary statistics about a Series or the
columns of a DataFrame (excluding NAs of course):
In [74]: series = Series(randn(1000))
In [75]: series[::2] = np.nan
In [76]: series.describe()
Out[76]:
count
500.000000
mean
-0.039663
std
1.069371
min
-3.463789
25%
-0.731101
50%
-0.058918
75%
0.672758
max
3.120271
dtype: float64
In [77]: frame = DataFrame(randn(1000, 5), columns=['a', 'b', 'c', 'd', 'e'])
In [78]: frame.ix[::2] = np.nan
In [79]: frame.describe()
Out[79]:
a
b
count 500.000000 500.000000
mean
0.000954
-0.044014
std
1.005133
0.974882
min
-3.010899
-2.782760
25%
-0.682900
-0.681161
50%
-0.001651
-0.006279
75%
0.656439
0.632852
max
3.007143
2.627688
c
500.000000
0.075936
0.967432
-3.401252
-0.528190
0.040098
0.717919
2.702490
d
500.000000
-0.003679
1.004732
-2.944925
-0.663503
-0.003378
0.687214
2.850852
e
500.000000
0.020751
0.963812
-3.794127
-0.615717
0.006282
0.653423
3.072117
You can select specific percentiles to include in the output:
In [80]: series.describe(percentiles=[.05, .25, .75, .95])
Out[80]:
count
500.000000
mean
-0.039663
std
1.069371
min
-3.463789
5%
-1.741334
25%
-0.731101
50%
-0.058918
75%
0.672758
95%
1.854383
max
3.120271
dtype: float64
By default, the median is always included.
For a non-numerical Series object, describe() will give a simple summary of the number of unique values and
most frequently occurring values:
324
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [81]: s = Series(['a', 'a', 'b', 'b', 'a', 'a', np.nan, 'c', 'd', 'a'])
In [82]: s.describe()
Out[82]:
count
9
unique
4
top
a
freq
5
dtype: object
Note that on a mixed-type DataFrame object, describe() will restrict the summary to include only numerical
columns or, if none are, only categorical columns:
In [83]: frame = DataFrame({'a': ['Yes', 'Yes', 'No', 'No'], 'b': range(4)})
In [84]: frame.describe()
Out[84]:
b
count 4.000000
mean
1.500000
std
1.290994
min
0.000000
25%
0.750000
50%
1.500000
75%
2.250000
max
3.000000
This behaviour can be controlled by providing a list of types as include/exclude arguments. The special value
all can also be used:
In [85]: frame.describe(include=['object'])
Out[85]:
a
count
4
unique
2
top
No
freq
2
In [86]: frame.describe(include=['number'])
Out[86]:
b
count 4.000000
mean
1.500000
std
1.290994
min
0.000000
25%
0.750000
50%
1.500000
75%
2.250000
max
3.000000
In [87]: frame.describe(include='all')
Out[87]:
a
b
count
4 4.000000
unique
2
NaN
top
No
NaN
freq
2
NaN
mean
NaN 1.500000
std
NaN 1.290994
10.5. Descriptive statistics
325
pandas: powerful Python data analysis toolkit, Release 0.16.1
min
25%
50%
75%
max
NaN
NaN
NaN
NaN
NaN
0.000000
0.750000
1.500000
2.250000
3.000000
That feature relies on select_dtypes. Refer to there for details about accepted inputs.
10.5.2 Index of Min/Max Values
The idxmin() and idxmax() functions on Series and DataFrame compute the index labels with the minimum and
maximum corresponding values:
In [88]: s1 = Series(randn(5))
In [89]: s1
Out[89]:
0
-0.872725
1
1.522411
2
0.080594
3
-1.676067
4
0.435804
dtype: float64
In [90]: s1.idxmin(), s1.idxmax()
Out[90]: (3, 1)
In [91]: df1 = DataFrame(randn(5,3), columns=['A','B','C'])
In [92]: df1
Out[92]:
A
B
C
0 0.445734 -1.649461 0.169660
1 1.246181 0.131682 -2.001988
2 -1.273023 0.870502 0.214583
3 0.088452 -0.173364 1.207466
4 0.546121 0.409515 -0.310515
In [93]: df1.idxmin(axis=0)
Out[93]:
A
2
B
0
C
1
dtype: int64
In [94]: df1.idxmax(axis=1)
Out[94]:
0
A
1
A
2
B
3
C
4
A
dtype: object
When there are multiple rows (or columns) matching the minimum or maximum value, idxmin() and idxmax()
return the first matching index:
326
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [95]: df3 = DataFrame([2, 1, 1, 3, np.nan], columns=['A'], index=list('edcba'))
In [96]: df3
Out[96]:
A
e
2
d
1
c
1
b
3
a NaN
In [97]: df3['A'].idxmin()
Out[97]: 'd'
Note: idxmin and idxmax are called argmin and argmax in NumPy.
10.5.3 Value counts (histogramming) / Mode
The value_counts() Series method and top-level function computes a histogram of a 1D array of values. It can
also be used as a function on regular arrays:
In [98]: data = np.random.randint(0, 7, size=50)
In [99]: data
Out[99]:
array([5, 3, 2, 2, 1, 4, 0, 4, 0, 2, 0, 6, 4, 1, 6, 3, 3, 0, 2, 1, 0, 5, 5,
3, 6, 1, 5, 6, 2, 0, 0, 6, 3, 3, 5, 0, 4, 3, 3, 3, 0, 6, 1, 3, 5, 5,
0, 4, 0, 6])
In [100]: s = Series(data)
In [101]: s.value_counts()
Out[101]:
0
11
3
10
6
7
5
7
4
5
2
5
1
5
dtype: int64
In [102]: value_counts(data)
Out[102]:
0
11
3
10
6
7
5
7
4
5
2
5
1
5
dtype: int64
Similarly, you can get the most frequently occurring value(s) (the mode) of the values in a Series or DataFrame:
10.5. Descriptive statistics
327
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [103]: s5 = Series([1, 1, 3, 3, 3, 5, 5, 7, 7, 7])
In [104]: s5.mode()
Out[104]:
0
3
1
7
dtype: int64
In [105]: df5 = DataFrame({"A": np.random.randint(0, 7, size=50),
.....:
"B": np.random.randint(-10, 15, size=50)})
.....:
In [106]: df5.mode()
Out[106]:
A B
0 1 -5
10.5.4 Discretization and quantiling
Continuous values can be discretized using the cut() (bins based on values) and qcut() (bins based on sample
quantiles) functions:
In [107]: arr = np.random.randn(20)
In [108]: factor = cut(arr, 4)
In [109]: factor
Out[109]:
[(-0.645, 0.336], (-2.61, -1.626], (-1.626, -0.645], (-1.626, -0.645], (-1.626, -0.645], ..., (0.336,
Length: 20
Categories (4, object): [(-2.61, -1.626] < (-1.626, -0.645] < (-0.645, 0.336] < (0.336, 1.316]]
In [110]: factor = cut(arr, [-5, -1, 0, 1, 5])
In [111]: factor
Out[111]:
[(-1, 0], (-5, -1], (-1, 0], (-5, -1], (-1, 0], ..., (0, 1], (1, 5], (0, 1], (0, 1], (-5, -1]]
Length: 20
Categories (4, object): [(-5, -1] < (-1, 0] < (0, 1] < (1, 5]]
qcut() computes sample quantiles. For example, we could slice up some normally distributed data into equal-size
quartiles like so:
In [112]: arr = np.random.randn(30)
In [113]: factor = qcut(arr, [0, .25, .5, .75, 1])
In [114]: factor
Out[114]:
[(-0.139, 1.00736], (1.00736, 1.976], (1.00736, 1.976], [-1.0705, -0.439], [-1.0705, -0.439], ..., (1
Length: 30
Categories (4, object): [[-1.0705, -0.439] < (-0.439, -0.139] < (-0.139, 1.00736] < (1.00736, 1.976]]
In [115]: value_counts(factor)
Out[115]:
(1.00736, 1.976]
8
[-1.0705, -0.439]
8
328
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
(-0.139, 1.00736]
(-0.439, -0.139]
dtype: int64
7
7
We can also pass infinite values to define the bins:
In [116]: arr = np.random.randn(20)
In [117]: factor = cut(arr, [-np.inf, 0, np.inf])
In [118]: factor
Out[118]:
[(-inf, 0], (0, inf], (0, inf], (0, inf], (-inf, 0], ..., (-inf, 0], (0, inf], (-inf, 0], (-inf, 0],
Length: 20
Categories (2, object): [(-inf, 0] < (0, inf]]
10.6 Function application
Arbitrary functions can be applied along the axes of a DataFrame or Panel using the apply() method, which, like
the descriptive statistics methods, take an optional axis argument:
In [119]: df.apply(np.mean)
Out[119]:
one
-0.251274
three
0.469799
two
-0.191421
dtype: float64
In [120]: df.apply(np.mean, axis=1)
Out[120]:
a
-0.489066
b
0.273355
c
0.008348
d
0.011457
dtype: float64
In [121]: df.apply(lambda x: x.max() - x.min())
Out[121]:
one
0.638161
three
1.301762
two
2.237808
dtype: float64
In [122]: df.apply(np.cumsum)
Out[122]:
one
three
two
a -0.626544
NaN -0.351587
b -0.765438 -0.177289 0.784662
c -0.753821 0.284925 0.335874
d
NaN 1.409398 -0.765684
In [123]: df.apply(np.exp)
Out[123]:
one
three
two
a 0.534436
NaN 0.703570
b 0.870320 0.837537 3.115063
10.6. Function application
329
pandas: powerful Python data analysis toolkit, Release 0.16.1
c
d
1.011685
NaN
1.587586
3.078592
0.638401
0.332353
Depending on the return type of the function passed to apply(), the result will either be of lower dimension or the
same dimension.
apply() combined with some cleverness can be used to answer many questions about a data set. For example,
suppose we wanted to extract the date where the maximum value for each column occurred:
In [124]: tsdf = DataFrame(randn(1000, 3), columns=['A', 'B', 'C'],
.....:
index=date_range('1/1/2000', periods=1000))
.....:
In [125]: tsdf.apply(lambda x: x.idxmax())
Out[125]:
A
2001-04-27
B
2002-06-02
C
2000-04-02
dtype: datetime64[ns]
You may also pass additional arguments and keyword arguments to the apply() method. For instance, consider the
following function you would like to apply:
def subtract_and_divide(x, sub, divide=1):
return (x - sub) / divide
You may then apply this function as follows:
df.apply(subtract_and_divide, args=(5,), divide=3)
Another useful feature is the ability to pass Series methods to carry out some Series operation on each column or row:
In [126]: tsdf
Out[126]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
2000-01-09
2000-01-10
A
B
C
1.796883 -0.930690 3.542846
-1.242888 -0.695279 -1.000884
-0.720299 0.546303 -0.082042
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
-0.527402 0.933507 0.129646
-0.338903 -1.265452 -1.969004
0.532566 0.341548 0.150493
In [127]: tsdf.apply(Series.interpolate)
Out[127]:
A
B
C
2000-01-01 1.796883 -0.930690 3.542846
2000-01-02 -1.242888 -0.695279 -1.000884
2000-01-03 -0.720299 0.546303 -0.082042
2000-01-04 -0.681720 0.623743 -0.039704
2000-01-05 -0.643140 0.701184 0.002633
2000-01-06 -0.604561 0.778625 0.044971
2000-01-07 -0.565982 0.856066 0.087309
2000-01-08 -0.527402 0.933507 0.129646
2000-01-09 -0.338903 -1.265452 -1.969004
2000-01-10 0.532566 0.341548 0.150493
330
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
Finally, apply() takes an argument raw which is False by default, which converts each row or column into a Series
before applying the function. When set to True, the passed function will instead receive an ndarray object, which has
positive performance implications if you do not need the indexing functionality.
See also:
The section on GroupBy demonstrates related, flexible functionality for grouping by some criterion, applying, and
combining the results into a Series, DataFrame, etc.
10.6.1 Applying elementwise Python functions
Since not all functions can be vectorized (accept NumPy arrays and return another array or value), the methods
applymap() on DataFrame and analogously map() on Series accept any Python function taking a single value and
returning a single value. For example:
In [128]: df4
Out[128]:
one
three
two
a -0.626544
NaN -0.351587
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
d
NaN 1.124472 -1.101558
In [129]: f = lambda x: len(str(x))
In [130]: df4['one'].map(f)
Out[130]:
a
14
b
15
c
15
d
3
Name: one, dtype: int64
In [131]: df4.applymap(f)
Out[131]:
one three two
a
14
3
15
b
15
15
11
c
15
14
15
d
3
13
14
Series.map() has an additional feature which is that it can be used to easily “link” or “map” values defined by a
secondary series. This is closely related to merging/joining functionality:
In [132]: s = Series(['six', 'seven', 'six', 'seven', 'six'],
.....:
index=['a', 'b', 'c', 'd', 'e'])
.....:
In [133]: t = Series({'six' : 6., 'seven' : 7.})
In [134]: s
Out[134]:
a
six
b
seven
c
six
d
seven
e
six
dtype: object
10.6. Function application
331
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [135]: s.map(t)
Out[135]:
a
6
b
7
c
6
d
7
e
6
dtype: float64
10.6.2 Applying with a Panel
Applying with a Panel will pass a Series to the applied function. If the applied function returns a Series, the
result of the application will be a Panel. If the applied function reduces to a scalar, the result of the application will
be a DataFrame.
Note: Prior to 0.13.1 apply on a Panel would only work on ufuncs (e.g. np.sum/np.max).
In [136]: import pandas.util.testing as tm
In [137]: panel = tm.makePanel(5)
In [138]: panel
Out[138]:
Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: A to D
In [139]: panel['ItemA']
Out[139]:
A
B
2000-01-03 0.330418 1.893177
2000-01-04 1.761200 0.170247
2000-01-05 0.567133 -0.916844
2000-01-06 -0.251020 0.835024
2000-01-07 1.020099 1.259919
C
0.801111
0.445614
1.453046
2.430373
0.653093
D
0.528154
-0.029371
-0.631117
-0.172441
-1.020485
A transformational apply.
In [140]: result = panel.apply(lambda x: x*2, axis='items')
In [141]: result
Out[141]:
Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: A to D
In [142]: result['ItemA']
Out[142]:
A
B
2000-01-03 0.660836 3.786354
2000-01-04 3.522400 0.340494
2000-01-05 1.134266 -1.833689
332
C
D
1.602222 1.056308
0.891228 -0.058742
2.906092 -1.262234
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-06 -0.502039
2000-01-07 2.040199
1.670047
2.519838
4.860747 -0.344882
1.306185 -2.040969
A reduction operation.
In [143]: panel.apply(lambda x: x.dtype, axis='items')
Out[143]:
A
B
C
D
2000-01-03 float64 float64 float64 float64
2000-01-04 float64 float64 float64 float64
2000-01-05 float64 float64 float64 float64
2000-01-06 float64 float64 float64 float64
2000-01-07 float64 float64 float64 float64
A similar reduction type operation
In [144]: panel.apply(lambda x: x.sum(), axis='major_axis')
Out[144]:
ItemA
ItemB
ItemC
A 3.427831 -2.581431 0.840809
B 3.241522 -1.409935 -1.114512
C 5.783237 0.319672 -0.431906
D -1.325260 -2.914834 0.857043
This last reduction is equivalent to
In [145]: panel.sum('major_axis')
Out[145]:
ItemA
ItemB
ItemC
A 3.427831 -2.581431 0.840809
B 3.241522 -1.409935 -1.114512
C 5.783237 0.319672 -0.431906
D -1.325260 -2.914834 0.857043
A transformation operation that returns a Panel, but is computing the z-score across the major_axis.
In [146]: result = panel.apply(
.....:
lambda x: (x-x.mean())/x.std(),
.....:
axis='major_axis')
.....:
In [147]: result
Out[147]:
Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: A to D
In [148]: result['ItemA']
Out[148]:
A
B
C
D
2000-01-03 -0.469761 1.156225 -0.441347 1.341731
2000-01-04 1.422763 -0.444015 -0.882647 0.398661
2000-01-05 -0.156654 -1.453694 0.367936 -0.619210
2000-01-06 -1.238841 0.173423 1.581149 0.156654
2000-01-07 0.442494 0.568061 -0.625091 -1.277837
Apply can also accept multiple axes in the axis argument. This will pass a DataFrame of the cross-section to the
applied function.
10.6. Function application
333
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [149]: f = lambda x: ((x.T-x.mean(1))/x.std(1)).T
In [150]: result = panel.apply(f, axis = ['items','major_axis'])
In [151]: result
Out[151]:
Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis)
Items axis: A to D
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: ItemA to ItemC
In [152]: result.loc[:,:,'ItemA']
Out[152]:
A
B
C
D
2000-01-03 0.864236 1.132969 0.557316 0.575106
2000-01-04 0.795745 0.652527 0.534808 -0.070674
2000-01-05 -0.310864 0.558627 1.086688 -1.051477
2000-01-06 -0.001065 0.832460 0.846006 0.043602
2000-01-07 1.128946 1.152469 -0.218186 -0.891680
This is equivalent to the following
In [153]: result = Panel(dict([ (ax,f(panel.loc[:,:,ax]))
.....:
for ax in panel.minor_axis ]))
.....:
In [154]: result
Out[154]:
Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis)
Items axis: A to D
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00
Minor_axis axis: ItemA to ItemC
In [155]: result.loc[:,:,'ItemA']
Out[155]:
A
B
C
D
2000-01-03 0.864236 1.132969 0.557316 0.575106
2000-01-04 0.795745 0.652527 0.534808 -0.070674
2000-01-05 -0.310864 0.558627 1.086688 -1.051477
2000-01-06 -0.001065 0.832460 0.846006 0.043602
2000-01-07 1.128946 1.152469 -0.218186 -0.891680
10.7 Reindexing and altering labels
reindex() is the fundamental data alignment method in pandas. It is used to implement nearly all other features
relying on label-alignment functionality. To reindex means to conform the data to match a given set of labels along a
particular axis. This accomplishes several things:
• Reorders the existing data to match a new set of labels
• Inserts missing value (NA) markers in label locations where no data for that label existed
• If specified, fill data for missing labels using logic (highly relevant to working with time series data)
Here is a simple example:
334
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [156]: s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])
In [157]: s
Out[157]:
a
-1.010924
b
-0.672504
c
-1.139222
d
0.354653
e
0.563622
dtype: float64
In [158]: s.reindex(['e', 'b', 'f', 'd'])
Out[158]:
e
0.563622
b
-0.672504
f
NaN
d
0.354653
dtype: float64
Here, the f label was not contained in the Series and hence appears as NaN in the result.
With a DataFrame, you can simultaneously reindex the index and columns:
In [159]: df
Out[159]:
one
three
two
a -0.626544
NaN -0.351587
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
d
NaN 1.124472 -1.101558
In [160]: df.reindex(index=['c', 'f', 'b'], columns=['three', 'two', 'one'])
Out[160]:
three
two
one
c 0.462215 -0.448789 0.011617
f
NaN
NaN
NaN
b -0.177289 1.136249 -0.138894
For convenience, you may utilize the reindex_axis() method, which takes the labels and a keyword axis
parameter.
Note that the Index objects containing the actual axis labels can be shared between objects. So if we have a Series
and a DataFrame, the following can be done:
In [161]: rs = s.reindex(df.index)
In [162]: rs
Out[162]:
a
-1.010924
b
-0.672504
c
-1.139222
d
0.354653
dtype: float64
In [163]: rs.index is df.index
Out[163]: True
This means that the reindexed Series’s index is the same Python object as the DataFrame’s index.
See also:
10.7. Reindexing and altering labels
335
pandas: powerful Python data analysis toolkit, Release 0.16.1
MultiIndex / Advanced Indexing is an even more concise way of doing reindexing.
Note: When writing performance-sensitive code, there is a good reason to spend some time becoming a reindexing
ninja: many operations are faster on pre-aligned data. Adding two unaligned DataFrames internally triggers a
reindexing step. For exploratory analysis you will hardly notice the difference (because reindex has been heavily
optimized), but when CPU cycles matter sprinkling a few explicit reindex calls here and there can have an impact.
10.7.1 Reindexing to align with another object
You may wish to take an object and reindex its axes to be labeled the same as another object. While the syntax for this
is straightforward albeit verbose, it is a common enough operation that the reindex_like() method is available
to make this simpler:
In [164]: df2
Out[164]:
one
two
a -0.626544 -0.351587
b -0.138894 1.136249
c 0.011617 -0.448789
In [165]: df3
Out[165]:
one
two
a -0.375270 -0.463545
b 0.112379 1.024292
c 0.262891 -0.560746
In [166]: df.reindex_like(df2)
Out[166]:
one
two
a -0.626544 -0.351587
b -0.138894 1.136249
c 0.011617 -0.448789
10.7.2 Aligning objects with each other with align
The align() method is the fastest way to simultaneously align two objects. It supports a join argument (related to
joining and merging):
• join=’outer’: take the union of the indexes (default)
• join=’left’: use the calling object’s index
• join=’right’: use the passed object’s index
• join=’inner’: intersect the indexes
It returns a tuple with both of the reindexed Series:
In [167]: s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])
In [168]: s1 = s[:4]
In [169]: s2 = s[1:]
In [170]: s1.align(s2)
Out[170]:
336
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
(a
-0.365106
b
1.092702
c
-1.481449
d
1.781190
e
NaN
dtype: float64, a
b
1.092702
c
-1.481449
d
1.781190
e
-0.031543
dtype: float64)
NaN
In [171]: s1.align(s2, join='inner')
Out[171]:
(b
1.092702
c
-1.481449
d
1.781190
dtype: float64, b
1.092702
c
-1.481449
d
1.781190
dtype: float64)
In [172]: s1.align(s2, join='left')
Out[172]:
(a
-0.365106
b
1.092702
c
-1.481449
d
1.781190
dtype: float64, a
NaN
b
1.092702
c
-1.481449
d
1.781190
dtype: float64)
For DataFrames, the join method will be applied to both the index and the columns by default:
In [173]: df.align(df2, join='inner')
Out[173]:
(
one
two
a -0.626544 -0.351587
b -0.138894 1.136249
c 0.011617 -0.448789,
one
a -0.626544 -0.351587
b -0.138894 1.136249
c 0.011617 -0.448789)
two
You can also pass an axis option to only align on the specified axis:
In [174]: df.align(df2, join='inner', axis=0)
Out[174]:
(
one
three
two
a -0.626544
NaN -0.351587
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789,
one
a -0.626544 -0.351587
b -0.138894 1.136249
c 0.011617 -0.448789)
10.7. Reindexing and altering labels
two
337
pandas: powerful Python data analysis toolkit, Release 0.16.1
If you pass a Series to DataFrame.align(), you can choose to align both objects either on the DataFrame’s index
or columns using the axis argument:
In [175]: df.align(df2.ix[0], axis=1)
Out[175]:
(
one
three
two
a -0.626544
NaN -0.351587
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
d
NaN 1.124472 -1.101558, one
three
NaN
two
-0.351587
Name: a, dtype: float64)
-0.626544
10.7.3 Filling while reindexing
reindex() takes an optional parameter method which is a filling method chosen from the following table:
Method
pad / ffill
bfill / backfill
nearest
Action
Fill values forward
Fill values backward
Fill from the nearest index value
We illustrate these fill methods on a simple Series:
In [176]: rng = date_range('1/3/2000', periods=8)
In [177]: ts = Series(randn(8), index=rng)
In [178]: ts2 = ts[[0, 3, 6]]
In [179]: ts
Out[179]:
2000-01-03
0.480993
2000-01-04
0.604244
2000-01-05
-0.487265
2000-01-06
1.990533
2000-01-07
0.327007
2000-01-08
1.053639
2000-01-09
-2.927808
2000-01-10
0.082065
Freq: D, dtype: float64
In [180]: ts2
Out[180]:
2000-01-03
0.480993
2000-01-06
1.990533
2000-01-09
-2.927808
dtype: float64
In [181]: ts2.reindex(ts.index)
Out[181]:
2000-01-03
0.480993
2000-01-04
NaN
2000-01-05
NaN
2000-01-06
1.990533
2000-01-07
NaN
2000-01-08
NaN
338
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-09
-2.927808
2000-01-10
NaN
Freq: D, dtype: float64
In [182]: ts2.reindex(ts.index, method='ffill')
Out[182]:
2000-01-03
0.480993
2000-01-04
0.480993
2000-01-05
0.480993
2000-01-06
1.990533
2000-01-07
1.990533
2000-01-08
1.990533
2000-01-09
-2.927808
2000-01-10
-2.927808
Freq: D, dtype: float64
In [183]: ts2.reindex(ts.index, method='bfill')
Out[183]:
2000-01-03
0.480993
2000-01-04
1.990533
2000-01-05
1.990533
2000-01-06
1.990533
2000-01-07
-2.927808
2000-01-08
-2.927808
2000-01-09
-2.927808
2000-01-10
NaN
Freq: D, dtype: float64
In [184]: ts2.reindex(ts.index, method='nearest')
Out[184]:
2000-01-03
0.480993
2000-01-04
0.480993
2000-01-05
1.990533
2000-01-06
1.990533
2000-01-07
1.990533
2000-01-08
-2.927808
2000-01-09
-2.927808
2000-01-10
-2.927808
Freq: D, dtype: float64
These methods require that the indexes are ordered increasing or decreasing.
Note that the same result could have been achieved using fillna (except for method=’nearest’) or interpolate:
In [185]: ts2.reindex(ts.index).fillna(method='ffill')
Out[185]:
2000-01-03
0.480993
2000-01-04
0.480993
2000-01-05
0.480993
2000-01-06
1.990533
2000-01-07
1.990533
2000-01-08
1.990533
2000-01-09
-2.927808
2000-01-10
-2.927808
Freq: D, dtype: float64
reindex() will raise a ValueError if the index is not monotonic increasing or descreasing. fillna() and
interpolate() will not make any checks on the order of the index.
10.7. Reindexing and altering labels
339
pandas: powerful Python data analysis toolkit, Release 0.16.1
10.7.4 Dropping labels from an axis
A method closely related to reindex is the drop() function. It removes a set of labels from an axis:
In [186]: df
Out[186]:
one
three
two
a -0.626544
NaN -0.351587
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
d
NaN 1.124472 -1.101558
In [187]: df.drop(['a', 'd'], axis=0)
Out[187]:
one
three
two
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
In [188]: df.drop(['one'], axis=1)
Out[188]:
three
two
a
NaN -0.351587
b -0.177289 1.136249
c 0.462215 -0.448789
d 1.124472 -1.101558
Note that the following also works, but is a bit less obvious / clean:
In [189]: df.reindex(df.index.difference(['a', 'd']))
Out[189]:
one
three
two
b -0.138894 -0.177289 1.136249
c 0.011617 0.462215 -0.448789
10.7.5 Renaming / mapping labels
The rename() method allows you to relabel an axis based on some mapping (a dict or Series) or an arbitrary function.
In [190]: s
Out[190]:
a
-0.365106
b
1.092702
c
-1.481449
d
1.781190
e
-0.031543
dtype: float64
In [191]: s.rename(str.upper)
Out[191]:
A
-0.365106
B
1.092702
C
-1.481449
D
1.781190
E
-0.031543
dtype: float64
If you pass a function, it must return a value when called with any of the labels (and must produce a set of unique
values). But if you pass a dict or Series, it need only contain a subset of the labels as keys:
340
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [192]: df.rename(columns={'one' : 'foo', 'two' : 'bar'},
.....:
index={'a' : 'apple', 'b' : 'banana', 'd' : 'durian'})
.....:
Out[192]:
foo
three
bar
apple -0.626544
NaN -0.351587
banana -0.138894 -0.177289 1.136249
c
0.011617 0.462215 -0.448789
durian
NaN 1.124472 -1.101558
The rename() method also provides an inplace named parameter that is by default False and copies the underlying data. Pass inplace=True to rename the data in place. The Panel class has a related rename_axis() class
which can rename any of its three axes.
10.8 Iteration
Because Series is array-like, basic iteration produces the values. Other data structures follow the dict-like convention
of iterating over the “keys” of the objects. In short:
• Series: values
• DataFrame: column labels
• Panel: item labels
Thus, for example:
In [193]: for col in df:
.....:
print(col)
.....:
one
three
two
10.8.1 iteritems
Consistent with the dict-like interface, iteritems() iterates through key-value pairs:
• Series: (index, scalar value) pairs
• DataFrame: (column, Series) pairs
• Panel: (item, DataFrame) pairs
For example:
In [194]: for item, frame in wp.iteritems():
.....:
print(item)
.....:
print(frame)
.....:
Item1
A
B
C
D
2000-01-01 -1.032011 0.969818 -0.962723 1.382083
2000-01-02 -0.938794 0.669142 -0.433567 -0.273610
2000-01-03 0.680433 -0.308450 -0.276099 -1.821168
2000-01-04 -1.993606 -1.927385 -2.027924 1.624972
2000-01-05 0.551135 3.059267 0.455264 -0.030740
Item2
10.8. Iteration
341
pandas: powerful Python data analysis toolkit, Release 0.16.1
A
B
C
D
2000-01-01 0.935716 1.061192 -2.107852 0.199905
2000-01-02 0.323586 -0.641630 -0.587514 0.053897
2000-01-03 0.194889 -0.381994 0.318587 2.089075
2000-01-04 -0.728293 -0.090255 -0.748199 1.318931
2000-01-05 -2.029766 0.792652 0.461007 -0.542749
10.8.2 iterrows
New in v0.7 is the ability to iterate efficiently through rows of a DataFrame with iterrows(). It returns an iterator
yielding each index value along with a Series containing the data in each row:
In [195]: for row_index, row in df2.iterrows():
.....:
print('%s\n%s' % (row_index, row))
.....:
a
one
-0.626544
two
-0.351587
Name: a, dtype: float64
b
one
-0.138894
two
1.136249
Name: b, dtype: float64
c
one
0.011617
two
-0.448789
Name: c, dtype: float64
For instance, a contrived way to transpose the DataFrame would be:
In [196]: df2 = DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})
In [197]: print(df2)
x y
0 1 4
1 2 5
2 3 6
In [198]: print(df2.T)
0 1 2
x 1 2 3
y 4 5 6
In [199]: df2_t = DataFrame(dict((idx,values) for idx, values in df2.iterrows()))
In [200]: print(df2_t)
0 1 2
x 1 2 3
y 4 5 6
Note: iterrows does not preserve dtypes across the rows (dtypes are preserved across columns for DataFrames).
For example,
In [201]: df_iter = DataFrame([[1, 1.0]], columns=['x', 'y'])
In [202]: row = next(df_iter.iterrows())[1]
342
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [203]: print(row['x'].dtype)
float64
In [204]: print(df_iter['x'].dtype)
int64
10.8.3 itertuples
The itertuples() method will return an iterator yielding a tuple for each row in the DataFrame. The first element
of the tuple will be the row’s corresponding index value, while the remaining values are the row values proper.
For instance,
In [205]: for r in df2.itertuples():
.....:
print(r)
.....:
(0, 1, 4)
(1, 2, 5)
(2, 3, 6)
10.8.4 .dt accessor
Series has an accessor to succinctly return datetime like properties for the values of the Series, if its a datetime/period like Series. This will return a Series, indexed like the existing Series.
# datetime
In [206]: s = Series(date_range('20130101 09:10:12',periods=4))
In [207]: s
Out[207]:
0
2013-01-01 09:10:12
1
2013-01-02 09:10:12
2
2013-01-03 09:10:12
3
2013-01-04 09:10:12
dtype: datetime64[ns]
In [208]: s.dt.hour
Out[208]:
0
9
1
9
2
9
3
9
dtype: int64
In [209]: s.dt.second
Out[209]:
0
12
1
12
2
12
3
12
dtype: int64
In [210]: s.dt.day
Out[210]:
0
1
10.8. Iteration
343
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
2
2
3
3
4
dtype: int64
This enables nice expressions like this:
In [211]: s[s.dt.day==2]
Out[211]:
1
2013-01-02 09:10:12
dtype: datetime64[ns]
You can easily produces tz aware transformations:
In [212]: stz = s.dt.tz_localize('US/Eastern')
In [213]: stz
Out[213]:
0
2013-01-01
1
2013-01-02
2
2013-01-03
3
2013-01-04
dtype: object
09:10:12-05:00
09:10:12-05:00
09:10:12-05:00
09:10:12-05:00
In [214]: stz.dt.tz
Out[214]:
You can also chain these types of operations:
In [215]: s.dt.tz_localize('UTC').dt.tz_convert('US/Eastern')
Out[215]:
0
2013-01-01 04:10:12-05:00
1
2013-01-02 04:10:12-05:00
2
2013-01-03 04:10:12-05:00
3
2013-01-04 04:10:12-05:00
dtype: object
The .dt accessor works for period and timedelta dtypes.
# period
In [216]: s = Series(period_range('20130101',periods=4,freq='D'))
In [217]: s
Out[217]:
0
2013-01-01
1
2013-01-02
2
2013-01-03
3
2013-01-04
dtype: object
In [218]: s.dt.year
Out[218]:
0
2013
1
2013
2
2013
3
2013
dtype: int64
In [219]: s.dt.day
344
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[219]:
0
1
1
2
2
3
3
4
dtype: int64
# timedelta
In [220]: s = Series(timedelta_range('1 day 00:00:05',periods=4,freq='s'))
In [221]: s
Out[221]:
0
1 days 00:00:05
1
1 days 00:00:06
2
1 days 00:00:07
3
1 days 00:00:08
dtype: timedelta64[ns]
In [222]: s.dt.days
Out[222]:
0
1
1
1
2
1
3
1
dtype: int64
In [223]: s.dt.seconds
Out[223]:
0
5
1
6
2
7
3
8
dtype: int64
In [224]: s.dt.components
Out[224]:
days hours minutes seconds
0
1
0
0
5
1
1
0
0
6
2
1
0
0
7
3
1
0
0
8
milliseconds
0
0
0
0
microseconds
0
0
0
0
nanoseconds
0
0
0
0
Note: Series.dt will raise a TypeError if you access with a non-datetimelike values
10.9 Vectorized string methods
Series is equipped with a set of string processing methods that make it easy to operate on each element of the array.
Perhaps most importantly, these methods exclude missing/NA values automatically. These are accessed via the Series’s
str attribute and generally have names matching the equivalent (scalar) built-in string methods. For example:
In [225]: s = Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])
In [226]: s.str.lower()
Out[226]:
0
a
10.9. Vectorized string methods
345
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
b
2
c
3
aaba
4
baca
5
NaN
6
caba
7
dog
8
cat
dtype: object
Powerful pattern-matching methods are provided as well, but note that pattern-matching generally uses regular expressions by default (and in some cases always uses them).
Please see Vectorized String Methods for a complete description.
10.10 Sorting by index and value
There are two obvious kinds of sorting that you may be interested in: sorting by label and sorting by actual values.
The primary method for sorting axis labels (indexes) across data structures is the sort_index() method.
In [227]: unsorted_df = df.reindex(index=['a', 'd', 'c', 'b'],
.....:
columns=['three', 'two', 'one'])
.....:
In [228]: unsorted_df.sort_index()
Out[228]:
three
two
one
a
NaN -0.351587 -0.626544
b -0.177289 1.136249 -0.138894
c 0.462215 -0.448789 0.011617
d 1.124472 -1.101558
NaN
In [229]: unsorted_df.sort_index(ascending=False)
Out[229]:
three
two
one
d 1.124472 -1.101558
NaN
c 0.462215 -0.448789 0.011617
b -0.177289 1.136249 -0.138894
a
NaN -0.351587 -0.626544
In [230]: unsorted_df.sort_index(axis=1)
Out[230]:
one
three
two
a -0.626544
NaN -0.351587
d
NaN 1.124472 -1.101558
c 0.011617 0.462215 -0.448789
b -0.138894 -0.177289 1.136249
DataFrame.sort_index() can accept an optional by argument for axis=0 which will use an arbitrary vector
or a column name of the DataFrame to determine the sort order:
In [231]: df1 = DataFrame({'one':[2,1,1,1],'two':[1,3,2,4],'three':[5,4,3,2]})
In [232]: df1.sort_index(by='two')
Out[232]:
one three two
0
2
5
1
346
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
1
3
1
1
1
3
4
2
2
3
4
The by argument can take a list of column names, e.g.:
In [233]: df1[['one', 'two', 'three']].sort_index(by=['one','two'])
Out[233]:
one two three
2
1
2
3
1
1
3
4
3
1
4
2
0
2
1
5
Series has the method order() (analogous to R’s order function) which sorts by value, with special treatment of NA
values via the na_position argument:
In [234]: s[2] = np.nan
In [235]: s.order()
Out[235]:
0
A
3
Aaba
1
B
4
Baca
6
CABA
8
cat
7
dog
2
NaN
5
NaN
dtype: object
In [236]: s.order(na_position='first')
Out[236]:
2
NaN
5
NaN
0
A
3
Aaba
1
B
4
Baca
6
CABA
8
cat
7
dog
dtype: object
Note: Series.sort() sorts a Series by value in-place. This is to provide compatibility with NumPy methods
which expect the ndarray.sort behavior. Series.order() returns a copy of the sorted data.
Series has the searchsorted() method, which works similar to numpy.ndarray.searchsorted().
In [237]: ser = Series([1, 2, 3])
In [238]: ser.searchsorted([0, 3])
Out[238]: array([0, 2])
In [239]: ser.searchsorted([0, 4])
Out[239]: array([0, 3])
10.10. Sorting by index and value
347
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [240]: ser.searchsorted([1, 3], side='right')
Out[240]: array([1, 3])
In [241]: ser.searchsorted([1, 3], side='left')
Out[241]: array([0, 2])
In [242]: ser = Series([3, 1, 2])
In [243]: ser.searchsorted([0, 3], sorter=np.argsort(ser))
Out[243]: array([0, 2])
10.10.1 smallest / largest values
New in version 0.14.0.
Series has the nsmallest() and nlargest() methods which return the smallest or largest 𝑛 values. For a
large Series this can be much faster than sorting the entire Series and calling head(n) on the result.
In [244]: s = Series(np.random.permutation(10))
In [245]: s
Out[245]:
0
7
1
5
2
4
3
6
4
1
5
8
6
9
7
2
8
0
9
3
dtype: int32
In [246]: s.order()
Out[246]:
8
0
4
1
7
2
9
3
2
4
1
5
3
6
0
7
5
8
6
9
dtype: int32
In [247]: s.nsmallest(3)
Out[247]:
8
0
4
1
7
2
dtype: int32
In [248]: s.nlargest(3)
Out[248]:
348
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
6
9
5
8
0
7
dtype: int32
10.10.2 Sorting by a multi-index column
You must be explicit about sorting when the column is a multi-index, and fully specify all levels to by.
In [249]: df1.columns = MultiIndex.from_tuples([('a','one'),('a','two'),('b','three')])
In [250]: df1.sort_index(by=('a','two'))
Out[250]:
a
b
one two three
3
1
2
4
2
1
3
2
1
1
4
3
0
2
5
1
10.11 Copying
The copy() method on pandas objects copies the underlying data (though not the axis indexes, since they are immutable) and returns a new object. Note that it is seldom necessary to copy objects. For example, there are only a
handful of ways to alter a DataFrame in-place:
• Inserting, deleting, or modifying a column
• Assigning to the index or columns attributes
• For homogeneous data, directly modifying the values via the values attribute or advanced indexing
To be clear, no pandas methods have the side effect of modifying your data; almost all methods return new objects,
leaving the original object untouched. If data is modified, it is because you did so explicitly.
10.12 dtypes
The main types stored in pandas objects are float, int, bool, datetime64[ns], timedelta[ns] and
object. In addition these dtypes have item sizes, e.g. int64 and int32. A convenient dtypes‘ attribute
for DataFrames returns a Series with the data type of each column.
In [251]: dft = DataFrame(dict( A = np.random.rand(3),
.....:
B = 1,
.....:
C = 'foo',
.....:
D = Timestamp('20010102'),
.....:
E = Series([1.0]*3).astype('float32'),
.....:
F = False,
.....:
G = Series([1]*3,dtype='int8')))
.....:
In [252]: dft
Out[252]:
A B
10.11. Copying
C
D
E
F
G
349
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1
2
0.028931
0.936706
0.831782
1
1
1
foo 2001-01-02
foo 2001-01-02
foo 2001-01-02
1
1
1
False
False
False
1
1
1
In [253]: dft.dtypes
Out[253]:
A
float64
B
int64
C
object
D
datetime64[ns]
E
float32
F
bool
G
int8
dtype: object
On a Series use the dtype attribute.
In [254]: dft['A'].dtype
Out[254]: dtype('float64')
If a pandas object contains data multiple dtypes IN A SINGLE COLUMN, the dtype of the column will be chosen to
accommodate all of the data types (object is the most general).
# these ints are coerced to floats
In [255]: Series([1, 2, 3, 4, 5, 6.])
Out[255]:
0
1
1
2
2
3
3
4
4
5
5
6
dtype: float64
# string data forces an ``object`` dtype
In [256]: Series([1, 2, 3, 6., 'foo'])
Out[256]:
0
1
1
2
2
3
3
6
4
foo
dtype: object
The method get_dtype_counts() will return the number of columns of each type in a DataFrame:
In [257]: dft.get_dtype_counts()
Out[257]:
bool
1
datetime64[ns]
1
float32
1
float64
1
int64
1
int8
1
object
1
dtype: int64
Numeric dtypes will propagate and can coexist in DataFrames (starting in v0.11.0). If a dtype is passed (either directly
via the dtype keyword, a passed ndarray, or a passed Series, then it will be preserved in DataFrame operations.
350
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
Furthermore, different numeric dtypes will NOT be combined. The following example will give you a taste.
In [258]: df1 = DataFrame(randn(8, 1), columns = ['A'], dtype = 'float32')
In [259]: df1
Out[259]:
A
0 1.213978
1 -0.505425
2 0.254678
3 -0.744834
4 0.647650
5 0.822993
6 1.778703
7 -1.543048
In [260]: df1.dtypes
Out[260]:
A
float32
dtype: object
In [261]: df2 = DataFrame(dict( A = Series(randn(8),dtype='float16'),
.....:
B = Series(randn(8)),
.....:
C = Series(np.array(randn(8),dtype='uint8')) ))
.....:
In [262]: df2
Out[262]:
A
B
0 -0.123230 -1.508174
1 2.240234 -0.502623
2 -0.143799 0.529008
3 -2.884766 0.590536
4 0.027588 0.296947
5 -1.150391 0.007045
6 0.246460 0.707877
7 -0.455078 0.950661
C
0
0
0
1
0
255
1
0
In [263]: df2.dtypes
Out[263]:
A
float16
B
float64
C
uint8
dtype: object
10.12.1 defaults
By default integer types are int64 and float types are float64, REGARDLESS of platform (32-bit or 64-bit). The
following will all result in int64 dtypes.
In [264]: DataFrame([1, 2], columns=['a']).dtypes
Out[264]:
a
int64
dtype: object
In [265]: DataFrame({'a': [1, 2]}).dtypes
Out[265]:
10.12. dtypes
351
pandas: powerful Python data analysis toolkit, Release 0.16.1
a
int64
dtype: object
In [266]: DataFrame({'a': 1 }, index=list(range(2))).dtypes
Out[266]:
a
int64
dtype: object
Numpy, however will choose platform-dependent types when creating arrays. The following WILL result in int32
on 32-bit platform.
In [267]: frame = DataFrame(np.array([1, 2]))
10.12.2 upcasting
Types can potentially be upcasted when combined with other types, meaning they are promoted from the current type
(say int to float)
In [268]: df3 = df1.reindex_like(df2).fillna(value=0.0) + df2
In [269]: df3
Out[269]:
A
B
0 1.090748 -1.508174
1 1.734810 -0.502623
2 0.110879 0.529008
3 -3.629600 0.590536
4 0.675238 0.296947
5 -0.327398 0.007045
6 2.025163 0.707877
7 -1.998126 0.950661
C
0
0
0
1
0
255
1
0
In [270]: df3.dtypes
Out[270]:
A
float32
B
float64
C
float64
dtype: object
The values attribute on a DataFrame return the lower-common-denominator of the dtypes, meaning the dtype that
can accommodate ALL of the types in the resulting homogeneous dtyped numpy array. This can force some upcasting.
In [271]: df3.values.dtype
Out[271]: dtype('float64')
10.12.3 astype
You can use the astype() method to explicitly convert dtypes from one to another. These will by default return a
copy, even if the dtype was unchanged (pass copy=False to change this behavior). In addition, they will raise an
exception if the astype operation is invalid.
Upcasting is always according to the numpy rules. If two different dtypes are involved in an operation, then the more
general one will be used as the result of the operation.
In [272]: df3
Out[272]:
352
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
A
B
0 1.090748 -1.508174
1 1.734810 -0.502623
2 0.110879 0.529008
3 -3.629600 0.590536
4 0.675238 0.296947
5 -0.327398 0.007045
6 2.025163 0.707877
7 -1.998126 0.950661
C
0
0
0
1
0
255
1
0
In [273]: df3.dtypes
Out[273]:
A
float32
B
float64
C
float64
dtype: object
# conversion of dtypes
In [274]: df3.astype('float32').dtypes
Out[274]:
A
float32
B
float32
C
float32
dtype: object
10.12.4 object conversion
convert_objects() is a method to try to force conversion of types from the object dtype to other types.
To force conversion of specific types that are number like, e.g. could be a string that represents a number, pass
convert_numeric=True. This will force strings and numbers alike to be numbers if possible, otherwise they will
be set to np.nan.
In [275]: df3['D'] = '1.'
In [276]: df3['E'] = '1'
In [277]: df3.convert_objects(convert_numeric=True).dtypes
Out[277]:
A
float32
B
float64
C
float64
D
float64
E
int64
dtype: object
# same, but specific dtype conversion
In [278]: df3['D'] = df3['D'].astype('float16')
In [279]: df3['E'] = df3['E'].astype('int32')
In [280]: df3.dtypes
Out[280]:
A
float32
B
float64
C
float64
D
float16
10.12. dtypes
353
pandas: powerful Python data analysis toolkit, Release 0.16.1
E
int32
dtype: object
To force conversion to datetime64[ns], pass convert_dates=’coerce’. This will convert any datetimelike object to dates, forcing other values to NaT. This might be useful if you are reading in data which is mostly dates,
but occasionally has non-dates intermixed and you want to represent as missing.
In [281]: s = Series([datetime(2001,1,1,0,0),
.....:
'foo', 1.0, 1, Timestamp('20010104'),
.....:
'20010105'],dtype='O')
.....:
In [282]: s
Out[282]:
0
2001-01-01 00:00:00
1
foo
2
1
3
1
4
2001-01-04 00:00:00
5
20010105
dtype: object
In [283]: s.convert_objects(convert_dates='coerce')
Out[283]:
0
2001-01-01
1
NaT
2
NaT
3
NaT
4
2001-01-04
5
2001-01-05
dtype: datetime64[ns]
In addition, convert_objects() will attempt the soft conversion of any object dtypes, meaning that if all the
objects in a Series are of the same type, the Series will have that dtype.
10.12.5 gotchas
Performing selection operations on integer type data can easily upcast the data to floating. The dtype of the
input data will be preserved in cases where nans are not introduced (starting in 0.11.0) See also integer na gotchas
In [284]: dfi = df3.astype('int32')
In [285]: dfi['E'] = 1
In [286]: dfi
Out[286]:
A B
C
0 1 -1
0
1 1 0
0
2 0 0
0
3 -3 0
1
4 0 0
0
5 0 0 255
6 2 0
1
7 -1 0
0
D
1
1
1
1
1
1
1
1
E
1
1
1
1
1
1
1
1
In [287]: dfi.dtypes
354
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[287]:
A
int32
B
int32
C
int32
D
int32
E
int64
dtype: object
In [288]: casted = dfi[dfi>0]
In [289]: casted
Out[289]:
A
B
C D
0
1 NaN NaN 1
1
1 NaN NaN 1
2 NaN NaN NaN 1
3 NaN NaN
1 1
4 NaN NaN NaN 1
5 NaN NaN 255 1
6
2 NaN
1 1
7 NaN NaN NaN 1
E
1
1
1
1
1
1
1
1
In [290]: casted.dtypes
Out[290]:
A
float64
B
float64
C
float64
D
int32
E
int64
dtype: object
While float dtypes are unchanged.
In [291]: dfa = df3.copy()
In [292]: dfa['A'] = dfa['A'].astype('float32')
In [293]: dfa.dtypes
Out[293]:
A
float32
B
float64
C
float64
D
float16
E
int32
dtype: object
In [294]: casted = dfa[df2>0]
In [295]: casted
Out[295]:
A
B
0
NaN
NaN
1 1.734810
NaN
2
NaN 0.529008
3
NaN 0.590536
4 0.675238 0.296947
5
NaN 0.007045
6 2.025163 0.707877
10.12. dtypes
C
NaN
NaN
NaN
1
NaN
255
1
D
NaN
NaN
NaN
NaN
NaN
NaN
NaN
E
NaN
NaN
NaN
NaN
NaN
NaN
NaN
355
pandas: powerful Python data analysis toolkit, Release 0.16.1
7
NaN
0.950661
NaN NaN NaN
In [296]: casted.dtypes
Out[296]:
A
float32
B
float64
C
float64
D
float16
E
float64
dtype: object
10.13 Selecting columns based on dtype
New in version 0.14.1.
The select_dtypes() method implements subsetting of columns based on their dtype.
First, let’s create a DataFrame with a slew of different dtypes:
In [297]: df = DataFrame({'string': list('abc'),
.....:
'int64': list(range(1, 4)),
.....:
'uint8': np.arange(3, 6).astype('u1'),
.....:
'float64': np.arange(4.0, 7.0),
.....:
'bool1': [True, False, True],
.....:
'bool2': [False, True, False],
.....:
'dates': pd.date_range('now', periods=3).values,
.....:
'category': pd.Categorical(list("ABC"))})
.....:
In [298]: df['tdeltas'] = df.dates.diff()
In [299]: df['uint64'] = np.arange(3, 6).astype('u8')
In [300]: df['other_dates'] = pd.date_range('20130101', periods=3).values
In [301]:
Out[301]:
bool1
0
True
1 False
2
True
0
1
2
uint8
3
4
5
df
bool2 category
dates
False
A 2015-05-11 11:00:20.785134
True
B 2015-05-12 11:00:20.785134
False
C 2015-05-13 11:00:20.785134
tdeltas
NaT
1 days
1 days
float64
4
5
6
int64 string
1
a
2
b
3
c
\
uint64 other_dates
3 2013-01-01
4 2013-01-02
5 2013-01-03
select_dtypes() has two parameters include and exclude that allow you to say “give me the columns
WITH these dtypes” (include) and/or “give the columns WITHOUT these dtypes” (exclude).
For example, to select bool columns
In [302]: df.select_dtypes(include=[bool])
Out[302]:
bool1 bool2
0
True False
356
Chapter 10. Essential Basic Functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
2
False
True
True
False
You can also pass the name of a dtype in the numpy dtype hierarchy:
In [303]:
Out[303]:
bool1
0
True
1 False
2
True
df.select_dtypes(include=['bool'])
bool2
False
True
False
select_dtypes() also works with generic dtypes as well.
For example, to select all numeric and boolean columns while excluding unsigned integers
In [304]:
Out[304]:
bool1
0
True
1 False
2
True
df.select_dtypes(include=['number', 'bool'], exclude=['unsignedinteger'])
bool2
False
True
False
float64
4
5
6
int64
1
2
3
tdeltas
NaT
1 days
1 days
To select string columns you must use the object dtype:
In [305]: df.select_dtypes(include=['object'])
Out[305]:
string
0
a
1
b
2
c
To see all the child dtypes of a generic dtype like numpy.number you can define a function that returns a tree of
child dtypes:
In [306]: def subdtypes(dtype):
.....:
subs = dtype.__subclasses__()
.....:
if not subs:
.....:
return dtype
.....:
return [dtype, [subdtypes(dt) for dt in subs]]
.....:
All numpy dtypes are subclasses of numpy.generic:
In [307]: subdtypes(np.generic)
Out[307]:
[numpy.generic,
[[numpy.number,
[[numpy.integer,
[[numpy.signedinteger,
[numpy.int8,
numpy.int16,
numpy.int32,
numpy.int32,
numpy.int64,
numpy.timedelta64]],
[numpy.unsignedinteger,
[numpy.uint8,
numpy.uint16,
numpy.uint32,
10.13. Selecting columns based on dtype
357
pandas: powerful Python data analysis toolkit, Release 0.16.1
numpy.uint32,
numpy.uint64]]]],
[numpy.inexact,
[[numpy.floating,
[numpy.float16, numpy.float32, numpy.float64, numpy.float96]],
[numpy.complexfloating,
[numpy.complex64, numpy.complex128, numpy.complex192]]]]]],
[numpy.flexible,
[[numpy.character, [numpy.string_, numpy.unicode_]],
[numpy.void, [numpy.core.records.record]]]],
numpy.bool_,
numpy.datetime64,
numpy.object_]]
Note: Pandas also defines an additional category dtype, which is not integrated into the normal numpy hierarchy
and wont show up with the above function.
Note: The include and exclude parameters must be non-string sequences.
358
Chapter 10. Essential Basic Functionality
CHAPTER
ELEVEN
WORKING WITH TEXT DATA
Series and Index are equipped with a set of string processing methods that make it easy to operate on each element of
the array. Perhaps most importantly, these methods exclude missing/NA values automatically. These are accessed via
the str attribute and generally have names matching the equivalent (scalar) built-in string methods:
In [1]: s = Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])
In [2]: s.str.lower()
Out[2]:
0
a
1
b
2
c
3
aaba
4
baca
5
NaN
6
caba
7
dog
8
cat
dtype: object
In [3]: s.str.upper()
Out[3]:
0
A
1
B
2
C
3
AABA
4
BACA
5
NaN
6
CABA
7
DOG
8
CAT
dtype: object
In [4]: s.str.len()
Out[4]:
0
1
1
1
2
1
3
4
4
4
5
NaN
6
4
7
3
8
3
dtype: float64
359
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [5]: idx = Index([' jack', 'jill ', ' jesse ', 'frank'])
In [6]: idx.str.strip()
Out[6]: Index([u'jack', u'jill', u'jesse', u'frank'], dtype='object')
In [7]: idx.str.lstrip()
Out[7]: Index([u'jack', u'jill ', u'jesse ', u'frank'], dtype='object')
In [8]: idx.str.rstrip()
Out[8]: Index([u' jack', u'jill', u' jesse', u'frank'], dtype='object')
The string methods on Index are especially useful for cleaning up or transforming DataFrame columns. For instance,
you may have columns with leading or trailing whitespace:
In [9]: df = DataFrame(randn(3, 2), columns=[' Column A ', ' Column B '],
...:
index=range(3))
...:
In [10]: df
Out[10]:
Column A
0
0.017428
1
-2.240248
2
-1.342107
Column B
0.039049
0.847859
0.368828
Since df.columns is an Index object, we can use the .str accessor
In [11]: df.columns.str.strip()
Out[11]: Index([u'Column A', u'Column B'], dtype='object')
In [12]: df.columns.str.lower()
Out[12]: Index([u' column a ', u' column b '], dtype='object')
These string methods can then be used to clean up the columns as needed. Here we are removing leading and trailing
whitespaces, lowercasing all names, and replacing any remaining whitespaces with underscores:
In [13]: df.columns = df.columns.str.strip().str.lower().str.replace(' ', '_')
In [14]: df
Out[14]:
column_a
0 0.017428
1 -2.240248
2 -1.342107
column_b
0.039049
0.847859
0.368828
11.1 Splitting and Replacing Strings
Methods like split return a Series of lists:
In [15]: s2 = Series(['a_b_c', 'c_d_e', np.nan, 'f_g_h'])
In [16]: s2.str.split('_')
Out[16]:
0
[a, b, c]
1
[c, d, e]
2
NaN
360
Chapter 11. Working with Text Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
3
[f, g, h]
dtype: object
Elements in the split lists can be accessed using get or [] notation:
In [17]: s2.str.split('_').str.get(1)
Out[17]:
0
b
1
d
2
NaN
3
g
dtype: object
In [18]: s2.str.split('_').str[1]
Out[18]:
0
b
1
d
2
NaN
3
g
dtype: object
Easy to expand this to return a DataFrame using expand.
In [19]: s2.str.split('_', expand=True)
Out[19]:
0
1
2
0
a
b
c
1
c
d
e
2 NaN NaN NaN
3
f
g
h
Methods like replace and findall take regular expressions, too:
In [20]: s3 = Series(['A', 'B', 'C', 'Aaba', 'Baca',
....:
'', np.nan, 'CABA', 'dog', 'cat'])
....:
In [21]: s3
Out[21]:
0
A
1
B
2
C
3
Aaba
4
Baca
5
6
NaN
7
CABA
8
dog
9
cat
dtype: object
In [22]: s3.str.replace('^.a|dog', 'XX-XX ', case=False)
Out[22]:
0
A
1
B
2
C
3
XX-XX ba
4
XX-XX ca
5
11.1. Splitting and Replacing Strings
361
pandas: powerful Python data analysis toolkit, Release 0.16.1
6
NaN
7
XX-XX BA
8
XX-XX
9
XX-XX t
dtype: object
Some caution must be taken to keep regular expressions in mind! For example, the following code will cause trouble
because of the regular expression meaning of $:
# Consider the following badly formatted financial data
In [23]: dollars = Series(['12', '-$10', '$10,000'])
# This does what you'd naively expect:
In [24]: dollars.str.replace('$', '')
Out[24]:
0
12
1
-10
2
10,000
dtype: object
# But this doesn't:
In [25]: dollars.str.replace('-$', '-')
Out[25]:
0
12
1
-$10
2
$10,000
dtype: object
# We need to escape the special character (for >1 len patterns)
In [26]: dollars.str.replace(r'-\$', '-')
Out[26]:
0
12
1
-10
2
$10,000
dtype: object
11.2 Indexing with .str
You can use [] notation to directly index by position locations. If you index past the end of the string, the result will
be a NaN.
In [27]: s = Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan,
....:
'CABA', 'dog', 'cat'])
....:
In [28]: s.str[0]
Out[28]:
0
A
1
B
2
C
3
A
4
B
5
NaN
6
C
7
d
8
c
362
Chapter 11. Working with Text Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
dtype: object
In [29]: s.str[1]
Out[29]:
0
NaN
1
NaN
2
NaN
3
a
4
a
5
NaN
6
A
7
o
8
a
dtype: object
11.3 Extracting Substrings
The method extract (introduced in version 0.13) accepts regular expressions with match groups. Extracting a
regular expression with one group returns a Series of strings.
In [30]: Series(['a1', 'b2', 'c3']).str.extract('[ab](\d)')
Out[30]:
0
1
1
2
2
NaN
dtype: object
Elements that do not match return NaN. Extracting a regular expression with more than one group returns a DataFrame
with one column per group.
In [31]: Series(['a1', 'b2', 'c3']).str.extract('([ab])(\d)')
Out[31]:
0
1
0
a
1
1
b
2
2 NaN NaN
Elements that do not match return a row filled with NaN. Thus, a Series of messy strings can be “converted” into a
like-indexed Series or DataFrame of cleaned-up or more useful strings, without necessitating get() to access tuples
or re.match objects.
The results dtype always is object, even if no match is found and the result only contains NaN.
Named groups like
In [32]: Series(['a1', 'b2', 'c3']).str.extract('(?P[ab])(?P\d)')
Out[32]:
letter digit
0
a
1
1
b
2
2
NaN
NaN
and optional groups like
In [33]: Series(['a1', 'b2', '3']).str.extract('(?P[ab])?(?P\d)')
Out[33]:
letter digit
11.3. Extracting Substrings
363
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1
2
a
b
NaN
1
2
3
can also be used.
11.3.1 Testing for Strings that Match or Contain a Pattern
You can check whether elements contain a pattern:
In [34]: pattern = r'[a-z][0-9]'
In [35]: Series(['1', '2', '3a', '3b', '03c']).str.contains(pattern)
Out[35]:
0
False
1
False
2
False
3
False
4
False
dtype: bool
or match a pattern:
In [36]: Series(['1', '2', '3a', '3b', '03c']).str.match(pattern, as_indexer=True)
Out[36]:
0
False
1
False
2
False
3
False
4
False
dtype: bool
The distinction between match and contains is strictness: match relies on strict re.match, while contains
relies on re.search.
Warning: In previous versions, match was for extracting groups, returning a not-so-convenient Series of tuples.
The new method extract (described in the previous section) is now preferred.
This old, deprecated behavior of match is still the default. As demonstrated above, use the new behavior by
setting as_indexer=True. In this mode, match is analogous to contains, returning a boolean Series. The
new behavior will become the default behavior in a future release.
Methods like match, contains, startswith, and endswith take an extra na argument so missing values
can be considered True or False:
In [37]: s4 = Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])
In [38]: s4.str.contains('A', na=False)
Out[38]:
0
True
1
False
2
False
3
True
4
False
5
False
6
True
7
False
8
False
dtype: bool
364
Chapter 11. Working with Text Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
11.3.2 Creating Indicator Variables
You can extract dummy variables from string columns. For example if they are separated by a ’|’:
In [39]: s = Series(['a', 'a|b', np.nan, 'a|c'])
In [40]:
Out[40]:
a b
0 1 0
1 1 1
2 0 0
3 1 0
s.str.get_dummies(sep='|')
c
0
0
0
1
See also get_dummies().
11.4 Method Summary
Method
cat()
split()
get()
join()
contains()
replace()
repeat()
pad()
center()
ljust()
rjust()
zfill()
wrap()
slice()
slice_replace()
count()
startswith()
endswith()
findall()
match()
extract()
len()
strip()
rstrip()
lstrip()
partition()
rpartition()
lower()
upper()
find()
rfind()
index()
rindex()
Description
Concatenate strings
Split strings on delimiter
Index into each element (retrieve i-th element)
Join strings in each element of the Series with passed separator
Return boolean array if each string contains pattern/regex
Replace occurrences of pattern/regex with some other string
Duplicate values (s.str.repeat(3) equivalent to x * 3)
Add whitespace to left, right, or both sides of strings
Equivalent to str.center
Equivalent to str.ljust
Equivalent to str.rjust
Equivalent to str.zfill
Split long strings into lines with length less than a given width
Slice each string in the Series
Replace slice in each string with passed value
Count occurrences of pattern
Equivalent to str.startswith(pat) for each element
Equivalent to str.endswith(pat) for each element
Compute list of all occurrences of pattern/regex for each string
Call re.match on each element, returning matched groups as list
Call re.match on each element, as match does, but return matched groups as strings for convenience.
Compute string lengths
Equivalent to str.strip
Equivalent to str.rstrip
Equivalent to str.lstrip
Equivalent to str.partition
Equivalent to str.rpartition
Equivalent to str.lower
Equivalent to str.upper
Equivalent to str.find
Equivalent to str.rfind
Equivalent to str.index
Equivalent to str.rindex
Continued on next page
11.4. Method Summary
365
pandas: powerful Python data analysis toolkit, Release 0.16.1
Method
capitalize()
swapcase()
normalize()
translate()
isalnum()
isalpha()
isdigit()
isspace()
islower()
isupper()
istitle()
isnumeric()
isdecimal()
366
Table 11.1 – continued from previous page
Description
Equivalent to str.capitalize
Equivalent to str.swapcase
Return Unicode normal form. Equivalent to unicodedata.normalize
Equivalent to str.translate
Equivalent to str.isalnum
Equivalent to str.isalpha
Equivalent to str.isdigit
Equivalent to str.isspace
Equivalent to str.islower
Equivalent to str.isupper
Equivalent to str.istitle
Equivalent to str.isnumeric
Equivalent to str.isdecimal
Chapter 11. Working with Text Data
CHAPTER
TWELVE
OPTIONS AND SETTINGS
12.1 Overview
pandas has an options system that lets you customize some aspects of its behaviour, display-related options being those
the user is most likely to adjust.
Options have a full “dotted-style”, case-insensitive name (e.g. display.max_rows). You can get/set options
directly as attributes of the top-level options attribute:
In [1]: import pandas as pd
In [2]: pd.options.display.max_rows
Out[2]: 15
In [3]: pd.options.display.max_rows = 999
In [4]: pd.options.display.max_rows
Out[4]: 999
There is also an API composed of 5 relevant functions, available directly from the pandas namespace:
• get_option() / set_option() - get/set the value of a single option.
• reset_option() - reset one or more options to their default value.
• describe_option() - print the descriptions of one or more options.
• option_context() - execute a codeblock with a set of options that revert to prior settings after execution.
Note: developers can check out pandas/core/config.py for more info.
All of the functions above accept a regexp pattern (re.search style) as an argument, and so passing in a substring
will work - as long as it is unambiguous :
In [5]: pd.get_option("display.max_rows")
Out[5]: 999
In [6]: pd.set_option("display.max_rows",101)
In [7]: pd.get_option("display.max_rows")
Out[7]: 101
In [8]: pd.set_option("max_r",102)
In [9]: pd.get_option("display.max_rows")
Out[9]: 102
367
pandas: powerful Python data analysis toolkit, Release 0.16.1
The following will not work because it matches multiple option names, e.g.
display.max_rows, display.max_columns:
display.max_colwidth,
In [10]: try:
....:
pd.get_option("column")
....: except KeyError as e:
....:
print(e)
....:
'Pattern matched multiple keys'
Note: Using this form of shorthand may cause your code to break if new options with similar names are added in
future versions.
You can get a list of available options and their descriptions with describe_option. When called with no argument describe_option will print out the descriptions for all available options.
12.2 Getting and Setting Options
As described above, get_option() and set_option() are available from the pandas namespace. To change an
option, call set_option(’option regex’, new_value)
In [11]: pd.get_option('mode.sim_interactive')
Out[11]: False
In [12]: pd.set_option('mode.sim_interactive', True)
In [13]: pd.get_option('mode.sim_interactive')
Out[13]: True
Note: that the option ‘mode.sim_interactive’ is mostly used for debugging purposes.
All options also have a default value, and you can use reset_option to do just that:
In [14]: pd.get_option("display.max_rows")
Out[14]: 60
In [15]: pd.set_option("display.max_rows",999)
In [16]: pd.get_option("display.max_rows")
Out[16]: 999
In [17]: pd.reset_option("display.max_rows")
In [18]: pd.get_option("display.max_rows")
Out[18]: 60
It’s also possible to reset multiple options at once (using a regex):
In [19]: pd.reset_option("^display")
height has been deprecated.
line_width has been deprecated, use display.width instead (currently both are
identical)
option_context context manager has been exposed through the top-level API, allowing you to execute code with
given option values. Option values are restored automatically when you exit the with block:
368
Chapter 12. Options and Settings
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [20]: with pd.option_context("display.max_rows",10,"display.max_columns", 5):
....:
print(pd.get_option("display.max_rows"))
....:
print(pd.get_option("display.max_columns"))
....:
10
5
In [21]: print(pd.get_option("display.max_rows"))
60
In [22]: print(pd.get_option("display.max_columns"))
20
12.3 Setting Startup Options in python/ipython Environment
Using startup scripts for the python/ipython environment to import pandas and set options makes working with pandas
more efficient. To do this, create a .py or .ipy script in the startup directory of the desired profile. An example where
the startup folder is in a default ipython profile can be found at:
$IPYTHONDIR/profile_default/startup
More information can be found in the ipython documentation. An example startup script for pandas is displayed
below:
import pandas as pd
pd.set_option('display.max_rows', 999)
pd.set_option('precision', 5)
12.4 Frequently Used Options
The following is a walkthrough of the more frequently used display options.
display.max_rows and display.max_columns sets the maximum number of rows and columns displayed
when a frame is pretty-printed. Truncated lines are replaced by an ellipsis.
In [23]: df=pd.DataFrame(np.random.randn(7,2))
In [24]: pd.set_option('max_rows', 7)
In [25]: df
Out[25]:
0
0 0.469112
1 -1.509059
2 1.212112
3 0.119209
4 -0.861849
5 -0.494929
6 0.721555
1
-0.282863
-1.135632
-0.173215
-1.044236
-2.104569
1.071804
-0.706771
In [26]: pd.set_option('max_rows', 5)
In [27]: df
Out[27]:
12.3. Setting Startup Options in python/ipython Environment
369
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1
0
0.469112 -0.282863
1 -1.509059 -1.135632
..
...
...
5 -0.494929 1.071804
6
0.721555 -0.706771
[7 rows x 2 columns]
In [28]: pd.reset_option('max_rows')
display.expand_frame_repr allows for the the representation of dataframes to stretch across pages, wrapped
over the full column vs row-wise.
In [29]: df=pd.DataFrame(np.random.randn(5,10))
In [30]: pd.set_option('expand_frame_repr', True)
In [31]: df
Out[31]:
0
1
2
3
4
5
0 -1.039575 0.271860 -0.424972 0.567020 0.276232 -1.087401
1 0.404705 0.577046 -1.715002 -1.039268 -0.370647 -1.157892
2 1.643563 -1.469388 0.357021 -0.674600 -1.776904 -0.968914
3 -0.013960 -0.362543 -0.006154 -0.923061 0.895717 0.805244
4 -1.170299 -0.226169 0.410835 0.813850 0.132003 -0.827317
6
-0.673690
-1.344312
-1.294524
-1.206412
-0.076467
\
7
8
9
0 0.113648 -1.478427 0.524988
1 0.844885 1.075770 -0.109050
2 0.413738 0.276662 -0.472035
3 2.565646 1.431256 1.340309
4 -1.187678 1.130127 -1.436737
In [32]: pd.set_option('expand_frame_repr', False)
In [33]: df
Out[33]:
0
1
2
3
4
5
0 -1.039575 0.271860 -0.424972 0.567020 0.276232 -1.087401
1 0.404705 0.577046 -1.715002 -1.039268 -0.370647 -1.157892
2 1.643563 -1.469388 0.357021 -0.674600 -1.776904 -0.968914
3 -0.013960 -0.362543 -0.006154 -0.923061 0.895717 0.805244
4 -1.170299 -0.226169 0.410835 0.813850 0.132003 -0.827317
6
7
8
9
-0.673690 0.113648 -1.478427 0.524988
-1.344312 0.844885 1.075770 -0.109050
-1.294524 0.413738 0.276662 -0.472035
-1.206412 2.565646 1.431256 1.340309
-0.076467 -1.187678 1.130127 -1.436737
In [34]: pd.reset_option('expand_frame_repr')
display.large_repr lets you select whether to display dataframes that exceed max_columns or max_rows
as a truncated frame, or as a summary.
In [35]: df=pd.DataFrame(np.random.randn(10,10))
In [36]: pd.set_option('max_rows', 5)
In [37]: pd.set_option('large_repr', 'truncate')
In [38]: df
Out[38]:
0
370
1
2
3
4
5
6
\
Chapter 12. Options and Settings
pandas: powerful Python data analysis toolkit, Release 0.16.1
0 -1.413681 1.607920 1.024180
1
0.545952 -1.219217 -1.226825
..
...
...
...
8 -2.484478 -0.281461 0.030711
9 -1.071357 0.441153 2.353925
0.569605 0.875906 -2.211372 0.974466
0.769804 -1.281247 -0.727707 -0.121306
...
...
...
...
0.109121 1.126203 -0.977349 1.474071
0.583787 0.221471 -0.744471 0.758527
7
8
9
0 -2.006747 -0.410001 -0.078638
1 -0.097883 0.695775 0.341734
..
...
...
...
8 -0.064034 -1.282782 0.781836
9
1.729689 -0.964980 -0.845696
[10 rows x 10 columns]
In [39]: pd.set_option('large_repr', 'info')
In [40]: df
Out[40]:
Int64Index: 10 entries, 0 to 9
Data columns (total 10 columns):
0
10 non-null float64
1
10 non-null float64
2
10 non-null float64
3
10 non-null float64
4
10 non-null float64
5
10 non-null float64
6
10 non-null float64
7
10 non-null float64
8
10 non-null float64
9
10 non-null float64
dtypes: float64(10)
memory usage: 880.0 bytes
In [41]: pd.reset_option('large_repr')
In [42]: pd.reset_option('max_rows')
display.max_columnwidth sets the maximum width of columns. Cells of this length or longer will be truncated
with an ellipsis.
In [43]: df=pd.DataFrame(np.array([['foo', 'bar', 'bim', 'uncomfortably long string'],
....:
['horse', 'cow', 'banana', 'apple']]))
....:
In [44]: pd.set_option('max_colwidth',40)
In [45]: df
Out[45]:
0
1
0
foo bar
1 horse cow
2
bim
banana
3
uncomfortably long string
apple
In [46]: pd.set_option('max_colwidth', 6)
In [47]: df
Out[47]:
12.4. Frequently Used Options
371
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1
0
foo
horse
1
bar
cow
2
bim
ba...
3
un...
apple
In [48]: pd.reset_option('max_colwidth')
display.max_info_columns sets a threshold for when by-column info will be given.
In [49]: df=pd.DataFrame(np.random.randn(10,10))
In [50]: pd.set_option('max_info_columns', 11)
In [51]: df.info()
Int64Index: 10 entries, 0 to 9
Data columns (total 10 columns):
0
10 non-null float64
1
10 non-null float64
2
10 non-null float64
3
10 non-null float64
4
10 non-null float64
5
10 non-null float64
6
10 non-null float64
7
10 non-null float64
8
10 non-null float64
9
10 non-null float64
dtypes: float64(10)
memory usage: 880.0 bytes
In [52]: pd.set_option('max_info_columns', 5)
In [53]: df.info()
Int64Index: 10 entries, 0 to 9
Columns: 10 entries, 0 to 9
dtypes: float64(10)
memory usage: 880.0 bytes
In [54]: pd.reset_option('max_info_columns')
display.max_info_rows: df.info() will usually show null-counts for each column. For large frames this
can be quite slow. max_info_rows and max_info_cols limit this null check only to frames with smaller
dimensions then specified. Note that you can specify the option df.info(null_counts=True) to override on
showing a particular frame.
In [55]: df=pd.DataFrame(np.random.choice([0,1,np.nan],size=(10,10)))
In [56]: df
Out[56]:
0
1
2
3
4
5
6
7
8
0
0
1
1
0
1
1
0 NaN
1
1
1 NaN
0
0
1
1 NaN
1
0
2 NaN NaN NaN
1
1
0 NaN
0
1
3
0
1
1 NaN
0 NaN
1 NaN NaN
4
0
1
0
0
1
0
0 NaN
0
5
0 NaN
1 NaN NaN NaN NaN
0
1
6
0
1
0
0 NaN
1 NaN NaN
0
7
0 NaN
1
1 NaN
1
1
1
1
8
0
0 NaN
0 NaN
1
0
0 NaN
372
9
NaN
1
NaN
0
0
NaN
NaN
NaN
NaN
Chapter 12. Options and Settings
pandas: powerful Python data analysis toolkit, Release 0.16.1
9 NaN NaN
0 NaN NaN NaN
0
1
1 NaN
In [57]: pd.set_option('max_info_rows', 11)
In [58]: df.info()
Int64Index: 10 entries, 0 to 9
Data columns (total 10 columns):
0
8 non-null float64
1
5 non-null float64
2
8 non-null float64
3
7 non-null float64
4
5 non-null float64
5
7 non-null float64
6
6 non-null float64
7
6 non-null float64
8
8 non-null float64
9
3 non-null float64
dtypes: float64(10)
memory usage: 880.0 bytes
In [59]: pd.set_option('max_info_rows', 5)
In [60]: df.info()
Int64Index: 10 entries, 0 to 9
Data columns (total 10 columns):
0
float64
1
float64
2
float64
3
float64
4
float64
5
float64
6
float64
7
float64
8
float64
9
float64
dtypes: float64(10)
memory usage: 880.0 bytes
In [61]: pd.reset_option('max_info_rows')
display.precision sets the output display precision. This is only a suggestion.
In [62]: df=pd.DataFrame(np.random.randn(5,5))
In [63]: pd.set_option('precision',7)
In [64]: df
Out[64]:
0
1
2
3
0 -2.049028 2.846612 -1.208049 -0.450392
1 0.121108 0.266916 0.843826 -0.222540
2 -0.716789 -2.224485 -1.061137 -0.232825
3 -0.665478 1.829807 -1.406509 1.078248
4 0.200324 0.890024 0.194813 0.351633
4
2.423905
2.021981
0.430793
0.322774
0.448881
In [65]: pd.set_option('precision',4)
12.4. Frequently Used Options
373
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [66]: df
Out[66]:
0
1
2
3
0 -2.049 2.847 -1.208 -0.450
1 0.121 0.267 0.844 -0.223
2 -0.717 -2.224 -1.061 -0.233
3 -0.665 1.830 -1.407 1.078
4 0.200 0.890 0.195 0.352
4
2.424
2.022
0.431
0.323
0.449
display.chop_threshold sets at what level pandas rounds to zero when it displays a Series of DataFrame.
Note, this does not effect the precision at which the number is stored.
In [67]: df=pd.DataFrame(np.random.randn(6,6))
In [68]: pd.set_option('chop_threshold', 0)
In [69]:
Out[69]:
0
0 -0.198
1 1.641
2 0.925
3 -0.824
4 0.432
5 -0.673
df
1
0.966
1.906
-0.006
-0.338
-0.461
-0.741
2
-1.523
2.772
-0.820
-0.928
0.337
-0.111
3
4
5
-0.117 0.296 -1.048
0.089 -1.144 -0.633
-0.601 -1.039 0.825
-0.840 0.249 -0.109
-3.208 -1.536 0.410
-2.673 0.864 0.061
In [70]: pd.set_option('chop_threshold', .5)
In [71]: df
Out[71]:
0
1
2
3
4
5
0 0.000 0.966 -1.523 0.000 0.000 -1.048
1 1.641 1.906 2.772 0.000 -1.144 -0.633
2 0.925 0.000 -0.820 -0.601 -1.039 0.825
3 -0.824 0.000 -0.928 -0.840 0.000 0.000
4 0.000 0.000 0.000 -3.208 -1.536 0.000
5 -0.673 -0.741 0.000 -2.673 0.864 0.000
In [72]: pd.reset_option('chop_threshold')
display.colheader_justify controls the justification of the headers. Options are ‘right’, and ‘left’.
In [73]: df=pd.DataFrame(np.array([np.random.randn(6), np.random.randint(1,9,6)*.1, np.zeros(6)]).T,
In [74]: pd.set_option('colheader_justify', 'right')
In [75]: df
Out[75]:
A
B
0 0.933 0.3
1 0.289 0.2
2 1.325 0.2
3 0.589 0.7
4 0.531 0.1
5 -1.199 0.7
C
0
0
0
0
0
0
In [76]: pd.set_option('colheader_justify', 'left')
374
Chapter 12. Options and Settings
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [77]: df
Out[77]:
A
B
0 0.933 0.3
1 0.289 0.2
2 1.325 0.2
3 0.589 0.7
4 0.531 0.1
5 -1.199 0.7
C
0
0
0
0
0
0
In [78]: pd.reset_option('colheader_justify')
12.5 List of Options
Option
display.chop_threshold
display.colheader_justify
display.column_space
display.date_dayfirst
display.date_yearfirst
display.encoding
display.expand_frame_repr
display.float_format
display.height
display.large_repr
display.line_width
display.max_columns
display.max_colwidth
display.max_info_columns
display.max_info_rows
display.max_rows
display.max_seq_items
display.memory_usage
display.mpl_style
display.multi_sparse
display.notebook_repr_html
display.pprint_nest_depth
display.precision
display.show_dimensions
display.width
io.excel.xls.writer
io.excel.xlsm.writer
io.excel.xlsx.writer
io.hdf.default_format
io.hdf.dropna_table
mode.chained_assignment
mode.sim_interactive
mode.use_inf_as_null
12.5. List of Options
Default
None
right
12
False
False
UTF-8
True
None
60
truncate
80
20
50
100
1690785
60
100
True
None
True
True
3
7
truncate
80
xlwt
openpyxl
openpyxl
None
True
warn
False
False
Function
If set to a float value, all float values smaller then the given threshold will be displayed as e
Controls the justification of column headers. used by DataFrameFormatter.
No description available.
When True, prints and parses dates with the day first, eg 20/01/2005
When True, prints and parses dates with the year first, eg 2005/01/20
Defaults to the detected encoding of the console. Specifies the encoding to be used for strin
Whether to print out the full DataFrame repr for wide DataFrames across multiple lines, m
The callable should accept a floating point number and return a string with the desired form
Deprecated. Use display.max_rows instead.
For DataFrames exceeding max_rows/max_cols, the repr (and HTML repr) can show a tru
Deprecated. Use display.width instead.
max_rows and max_columns are used in __repr__() methods to decide if to_string() or info
The maximum width in characters of a column in the repr of a pandas data structure. When
max_info_columns is used in DataFrame.info method to decide if per column information
df.info() will usually show null-counts for each column. For large frames this can be quite
This sets the maximum number of rows pandas should output when printing out various ou
when pretty-printing a long sequence, no more then max_seq_items will be printed. If item
This specifies if the memory usage of a DataFrame should be displayed when the df.info()
Setting this to ‘default’ will modify the rcParams used by matplotlib to give plots a more p
“Sparsify” MultiIndex display (don’t display repeated elements in outer levels within group
When True, IPython notebook will use html representation for pandas objects (if it is availa
Controls the number of nested levels to process when pretty-printing
Floating point output precision (number of significant digits). This is only a suggestion
Whether to print out dimensions at the end of DataFrame repr. If ‘truncate’ is specified, on
Width of the display in characters. In case python/IPython is running in a terminal this can
The default Excel writer engine for ‘xls’ files.
The default Excel writer engine for ‘xlsm’ files. Available options: ‘openpyxl’ (the default
The default Excel writer engine for ‘xlsx’ files.
default format writing format, if None, then put will default to ‘fixed’ and append will defa
drop ALL nan rows when appending to a table
Raise an exception, warn, or no action if trying to use chained assignment, The default is w
Whether to simulate interactive mode for purposes of testing
True means treat None, NaN, -INF, INF as null (old way), False means None and NaN are
375
pandas: powerful Python data analysis toolkit, Release 0.16.1
12.6 Number Formatting
pandas also allows you to set how numbers are displayed in the console. This option is not set through the
set_options API.
Use the set_eng_float_format function to alter the floating-point formatting of pandas objects to produce a
particular format.
For instance:
In [79]: import numpy as np
In [80]: pd.set_eng_float_format(accuracy=3, use_eng_prefix=True)
In [81]: s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])
In [82]: s/1.e3
Out[82]:
a
-236.866u
b
846.974u
c
-685.597u
d
609.099u
e
-303.961u
dtype: float64
In [83]: s/1.e6
Out[83]:
a
-236.866n
b
846.974n
c
-685.597n
d
609.099n
e
-303.961n
dtype: float64
376
Chapter 12. Options and Settings
CHAPTER
THIRTEEN
INDEXING AND SELECTING DATA
The axis labeling information in pandas objects serves many purposes:
• Identifies data (i.e. provides metadata) using known indicators, important for analysis, visualization, and interactive console display
• Enables automatic and explicit data alignment
• Allows intuitive getting and setting of subsets of the data set
In this section, we will focus on the final point: namely, how to slice, dice, and generally get and set subsets of pandas
objects. The primary focus will be on Series and DataFrame as they have received more development attention in
this area. Expect more work to be invested in higher-dimensional data structures (including Panel) in the future,
especially in label-based advanced indexing.
Note: The Python and NumPy indexing operators [] and attribute operator . provide quick and easy access to
pandas data structures across a wide range of use cases. This makes interactive work intuitive, as there’s little new to
learn if you already know how to deal with Python dictionaries and NumPy arrays. However, since the type of the data
to be accessed isn’t known in advance, directly using standard operators has some optimization limits. For production
code, we recommended that you take advantage of the optimized pandas data access methods exposed in this chapter.
Warning: Whether a copy or a reference is returned for a setting operation, may depend on the context. This is
sometimes called chained assignment and should be avoided. See Returning a View versus Copy
Warning: In 0.15.0 Index has internally been refactored to no longer subclass ndarray but instead subclass
PandasObject, similarly to the rest of the pandas objects. This should be a transparent change with only very
limited API implications (See the Internal Refactoring)
See the MultiIndex / Advanced Indexing for MultiIndex and more advanced indexing documentation.
See the cookbook for some advanced strategies
13.1 Different Choices for Indexing
New in version 0.11.0.
Object selection has had a number of user-requested additions in order to support more explicit location based indexing. pandas now supports three types of multi-axis indexing.
• .loc is primarily label based, but may also be used with a boolean array. .loc will raise KeyError when
the items are not found. Allowed inputs are:
377
pandas: powerful Python data analysis toolkit, Release 0.16.1
– A single label, e.g. 5 or ’a’, (note that 5 is interpreted as a label of the index. This use is not an integer
position along the index)
– A list or array of labels [’a’, ’b’, ’c’]
– A slice object with labels ’a’:’f’, (note that contrary to usual python slices, both the start and the stop
are included!)
– A boolean array
See more at Selection by Label
• .iloc is primarily integer position based (from 0 to length-1 of the axis), but may also be used with a
boolean array. .iloc will raise IndexError if a requested indexer is out-of-bounds, except slice indexers
which allow out-of-bounds indexing. (this conforms with python/numpy slice semantics). Allowed inputs are:
– An integer e.g. 5
– A list or array of integers [4, 3, 0]
– A slice object with ints 1:7
– A boolean array
See more at Selection by Position
• .ix supports mixed integer and label based access. It is primarily label based, but will fall back to integer
positional access unless the corresponding axis is of integer type. .ix is the most general and will support any
of the inputs in .loc and .iloc. .ix also supports floating point label schemes. .ix is exceptionally useful
when dealing with mixed positional and label based hierachical indexes.
However, when an axis is integer based, ONLY label based access and not positional access is supported. Thus,
in such cases, it’s usually better to be explicit and use .iloc or .loc.
See more at Advanced Indexing and Advanced Hierarchical.
Getting values from an object with multi-axes selection uses the following notation (using .loc as an example, but
applies to .iloc and .ix as well). Any of the axes accessors may be the null slice :. Axes left out of the specification
are assumed to be :. (e.g. p.loc[’a’] is equiv to p.loc[’a’, :, :])
Object Type
Series
DataFrame
Panel
Indexers
s.loc[indexer]
df.loc[row_indexer,column_indexer]
p.loc[item_indexer,major_indexer,minor_indexer]
13.2 Deprecations
Beginning with version 0.11.0, it’s recommended that you transition away from the following methods as they may be
deprecated in future versions.
• irow
• icol
• iget_value
See the section Selection by Position for substitutes.
378
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
13.3 Basics
As mentioned when introducing the data structures in the last section, the primary function of indexing with [] (a.k.a.
__getitem__ for those familiar with implementing class behavior in Python) is selecting out lower-dimensional
slices. Thus,
Object Type
Series
DataFrame
Panel
Selection
series[label]
frame[colname]
panel[itemname]
Return Value Type
scalar value
Series corresponding to colname
DataFrame corresponing to the itemname
Here we construct a simple time series data set to use for illustrating the indexing functionality:
In [1]: dates = date_range('1/1/2000', periods=8)
In [2]: df = DataFrame(randn(8, 4), index=dates, columns=['A', 'B', 'C', 'D'])
In [3]: df
Out[3]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
0.469112
1.212112
-0.861849
0.721555
-0.424972
-0.673690
0.404705
-0.370647
B
-0.282863
-0.173215
-2.104569
-0.706771
0.567020
0.113648
0.577046
-1.157892
C
-1.509059
0.119209
-0.494929
-1.039575
0.276232
-1.478427
-1.715002
-1.344312
D
-1.135632
-1.044236
1.071804
0.271860
-1.087401
0.524988
-1.039268
0.844885
In [4]: panel = Panel({'one' : df, 'two' : df - df.mean()})
In [5]: panel
Out[5]:
Dimensions: 2 (items) x 8 (major_axis) x 4 (minor_axis)
Items axis: one to two
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-08 00:00:00
Minor_axis axis: A to D
Note: None of the indexing functionality is time series specific unless specifically stated.
Thus, as per above, we have the most basic indexing using []:
In [6]: s = df['A']
In [7]: s[dates[5]]
Out[7]: -0.67368970808837025
In [8]: panel['two']
Out[8]:
A
B
C
D
2000-01-01 0.409571 0.113086 -0.610826 -0.936507
2000-01-02 1.152571 0.222735 1.017442 -0.845111
2000-01-03 -0.921390 -1.708620 0.403304 1.270929
2000-01-04 0.662014 -0.310822 -0.141342 0.470985
2000-01-05 -0.484513 0.962970 1.174465 -0.888276
2000-01-06 -0.733231 0.509598 -0.580194 0.724113
2000-01-07 0.345164 0.972995 -0.816769 -0.840143
13.3. Basics
379
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-08 -0.430188 -0.761943 -0.446079
1.044010
You can pass a list of columns to [] to select columns in that order. If a column is not contained in the DataFrame, an
exception will be raised. Multiple columns can also be set in this manner:
In [9]: df
Out[9]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
0.469112
1.212112
-0.861849
0.721555
-0.424972
-0.673690
0.404705
-0.370647
B
-0.282863
-0.173215
-2.104569
-0.706771
0.567020
0.113648
0.577046
-1.157892
C
-1.509059
0.119209
-0.494929
-1.039575
0.276232
-1.478427
-1.715002
-1.344312
D
-1.135632
-1.044236
1.071804
0.271860
-1.087401
0.524988
-1.039268
0.844885
In [10]: df[['B', 'A']] = df[['A', 'B']]
In [11]: df
Out[11]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
-0.282863
-0.173215
-2.104569
-0.706771
0.567020
0.113648
0.577046
-1.157892
B
0.469112
1.212112
-0.861849
0.721555
-0.424972
-0.673690
0.404705
-0.370647
C
-1.509059
0.119209
-0.494929
-1.039575
0.276232
-1.478427
-1.715002
-1.344312
D
-1.135632
-1.044236
1.071804
0.271860
-1.087401
0.524988
-1.039268
0.844885
You may find this useful for applying a transform (in-place) to a subset of the columns.
13.4 Attribute Access
You may access an index on a Series, column on a DataFrame, and a item on a Panel directly as an attribute:
In [12]: sa = Series([1,2,3],index=list('abc'))
In [13]: dfa = df.copy()
In [14]: sa.b
Out[14]: 2
In [15]: dfa.A
Out[15]:
2000-01-01
-0.282863
2000-01-02
-0.173215
2000-01-03
-2.104569
2000-01-04
-0.706771
2000-01-05
0.567020
2000-01-06
0.113648
2000-01-07
0.577046
2000-01-08
-1.157892
Freq: D, Name: A, dtype: float64
In [16]: panel.one
380
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[16]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
0.469112
1.212112
-0.861849
0.721555
-0.424972
-0.673690
0.404705
-0.370647
B
-0.282863
-0.173215
-2.104569
-0.706771
0.567020
0.113648
0.577046
-1.157892
C
-1.509059
0.119209
-0.494929
-1.039575
0.276232
-1.478427
-1.715002
-1.344312
D
-1.135632
-1.044236
1.071804
0.271860
-1.087401
0.524988
-1.039268
0.844885
You can use attribute access to modify an existing element of a Series or column of a DataFrame, but be careful; if you
try to use attribute access to create a new column, it fails silently, creating a new attribute rather than a new column.
In [17]: sa.a = 5
In [18]: sa
Out[18]:
a
5
b
2
c
3
dtype: int64
In [19]: dfa.A = list(range(len(dfa.index)))
# ok if A already exists
In [20]: dfa
Out[20]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
0
1
2
3
4
5
6
7
B
0.469112
1.212112
-0.861849
0.721555
-0.424972
-0.673690
0.404705
-0.370647
C
-1.509059
0.119209
-0.494929
-1.039575
0.276232
-1.478427
-1.715002
-1.344312
D
-1.135632
-1.044236
1.071804
0.271860
-1.087401
0.524988
-1.039268
0.844885
In [21]: dfa['A'] = list(range(len(dfa.index)))
# use this form to create a new column
In [22]: dfa
Out[22]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
0
1
2
3
4
5
6
7
B
0.469112
1.212112
-0.861849
0.721555
-0.424972
-0.673690
0.404705
-0.370647
13.4. Attribute Access
C
-1.509059
0.119209
-0.494929
-1.039575
0.276232
-1.478427
-1.715002
-1.344312
D
-1.135632
-1.044236
1.071804
0.271860
-1.087401
0.524988
-1.039268
0.844885
381
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning:
• You can use this access only if the index element is a valid python identifier, e.g. s.1 is not allowed. See
here for an explanation of valid identifiers.
• The attribute will not be available if it conflicts with an existing method name, e.g. s.min is not allowed.
• Similarly, the attribute will not be available if it conflicts with any of the following list: index,
major_axis, minor_axis, items, labels.
• In any of these cases, standard indexing will still work, e.g. s[’1’], s[’min’], and s[’index’] will
access the corresponding element or column.
• The Series/Panel accesses are available starting in 0.13.0.
If you are using the IPython environment, you may also use tab-completion to see these accessible attributes.
You can also assign a dict to a row of a DataFrame:
In [23]: x = pd.DataFrame({'x': [1, 2, 3], 'y': [3, 4, 5]})
In [24]: x.iloc[1] = dict(x=9, y=99)
In [25]: x
Out[25]:
x
y
0 1
3
1 9 99
2 3
5
13.5 Slicing ranges
The most robust and consistent way of slicing ranges along arbitrary axes is described in the Selection by Position
section detailing the .iloc method. For now, we explain the semantics of slicing using the [] operator.
With Series, the syntax works exactly as with an ndarray, returning a slice of the values and the corresponding labels:
In [26]: s[:5]
Out[26]:
2000-01-01
-0.282863
2000-01-02
-0.173215
2000-01-03
-2.104569
2000-01-04
-0.706771
2000-01-05
0.567020
Freq: D, Name: A, dtype: float64
In [27]: s[::2]
Out[27]:
2000-01-01
-0.282863
2000-01-03
-2.104569
2000-01-05
0.567020
2000-01-07
0.577046
Freq: 2D, Name: A, dtype: float64
In [28]: s[::-1]
Out[28]:
2000-01-08
-1.157892
2000-01-07
0.577046
2000-01-06
0.113648
2000-01-05
0.567020
2000-01-04
-0.706771
382
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-03
-2.104569
2000-01-02
-0.173215
2000-01-01
-0.282863
Freq: -1D, Name: A, dtype: float64
Note that setting works as well:
In [29]: s2 = s.copy()
In [30]: s2[:5] = 0
In [31]: s2
Out[31]:
2000-01-01
0.000000
2000-01-02
0.000000
2000-01-03
0.000000
2000-01-04
0.000000
2000-01-05
0.000000
2000-01-06
0.113648
2000-01-07
0.577046
2000-01-08
-1.157892
Freq: D, Name: A, dtype: float64
With DataFrame, slicing inside of [] slices the rows. This is provided largely as a convenience since it is such a
common operation.
In [32]: df[:3]
Out[32]:
A
B
C
D
2000-01-01 -0.282863 0.469112 -1.509059 -1.135632
2000-01-02 -0.173215 1.212112 0.119209 -1.044236
2000-01-03 -2.104569 -0.861849 -0.494929 1.071804
In [33]: df[::-1]
Out[33]:
2000-01-08
2000-01-07
2000-01-06
2000-01-05
2000-01-04
2000-01-03
2000-01-02
2000-01-01
A
-1.157892
0.577046
0.113648
0.567020
-0.706771
-2.104569
-0.173215
-0.282863
B
-0.370647
0.404705
-0.673690
-0.424972
0.721555
-0.861849
1.212112
0.469112
C
-1.344312
-1.715002
-1.478427
0.276232
-1.039575
-0.494929
0.119209
-1.509059
D
0.844885
-1.039268
0.524988
-1.087401
0.271860
1.071804
-1.044236
-1.135632
13.6 Selection By Label
Warning: Whether a copy or a reference is returned for a setting operation, may depend on the context. This is
sometimes called chained assignment and should be avoided. See Returning a View versus Copy
13.6. Selection By Label
383
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning:
.loc is strict when you present slicers that are not compatible (or convertible) with the index type.
For example using integers in a DatetimeIndex. These will raise a TypeError.
In [34]: dfl = DataFrame(np.random.randn(5,4), columns=list('ABCD'), index=date_range('20130101',per
In [35]: dfl
Out[35]:
A
B
C
2013-01-01 1.075770 -0.109050 1.643563
2013-01-02 0.357021 -0.674600 -1.776904
2013-01-03 -1.294524 0.413738 0.276662
2013-01-04 -0.013960 -0.362543 -0.006154
2013-01-05 0.895717 0.805244 -1.206412
D
-1.469388
-0.968914
-0.472035
-0.923061
2.565646
In [4]: dfl.loc[2:3]
TypeError: cannot do slice indexing on with these index
String likes in slicing can be convertible to the type of the index and lead to natural slicing.
In [36]: dfl.loc['20130102':'20130104']
Out[36]:
A
B
C
D
2013-01-02 0.357021 -0.674600 -1.776904 -0.968914
2013-01-03 -1.294524 0.413738 0.276662 -0.472035
2013-01-04 -0.013960 -0.362543 -0.006154 -0.923061
pandas provides a suite of methods in order to have purely label based indexing. This is a strict inclusion based
protocol. At least 1 of the labels for which you ask, must be in the index or a KeyError will be raised! When
slicing, the start bound is included, AND the stop bound is included. Integers are valid labels, but they refer to the
label and not the position.
The .loc attribute is the primary access method. The following are valid inputs:
• A single label, e.g. 5 or ’a’, (note that 5 is interpreted as a label of the index. This use is not an integer
position along the index)
• A list or array of labels [’a’, ’b’, ’c’]
• A slice object with labels ’a’:’f’ (note that contrary to usual python slices, both the start and the stop are
included!)
• A boolean array
In [37]: s1 = Series(np.random.randn(6),index=list('abcdef'))
In [38]: s1
Out[38]:
a
1.431256
b
1.340309
c
-1.170299
d
-0.226169
e
0.410835
f
0.813850
dtype: float64
In [39]: s1.loc['c':]
Out[39]:
384
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
c
-1.170299
d
-0.226169
e
0.410835
f
0.813850
dtype: float64
In [40]: s1.loc['b']
Out[40]: 1.3403088497993827
Note that setting works as well:
In [41]: s1.loc['c':] = 0
In [42]: s1
Out[42]:
a
1.431256
b
1.340309
c
0.000000
d
0.000000
e
0.000000
f
0.000000
dtype: float64
With a DataFrame
In [43]: df1 = DataFrame(np.random.randn(6,4),
....:
index=list('abcdef'),
....:
columns=list('ABCD'))
....:
In [44]: df1
Out[44]:
A
B
a 0.132003 -0.827317
b 1.130127 -1.436737
c 1.024180 0.569605
d 0.974466 -2.006747
e 0.545952 -1.219217
f -1.281247 -0.727707
C
-0.076467
-1.413681
0.875906
-0.410001
-1.226825
-0.121306
D
-1.187678
1.607920
-2.211372
-0.078638
0.769804
-0.097883
In [45]: df1.loc[['a','b','d'],:]
Out[45]:
A
B
C
D
a 0.132003 -0.827317 -0.076467 -1.187678
b 1.130127 -1.436737 -1.413681 1.607920
d 0.974466 -2.006747 -0.410001 -0.078638
Accessing via label slices
In [46]: df1.loc['d':,'A':'C']
Out[46]:
A
B
C
d 0.974466 -2.006747 -0.410001
e 0.545952 -1.219217 -1.226825
f -1.281247 -0.727707 -0.121306
For getting a cross section using a label (equiv to df.xs(’a’))
13.6. Selection By Label
385
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [47]: df1.loc['a']
Out[47]:
A
0.132003
B
-0.827317
C
-0.076467
D
-1.187678
Name: a, dtype: float64
For getting values with a boolean array
In [48]: df1.loc['a']>0
Out[48]:
A
True
B
False
C
False
D
False
Name: a, dtype: bool
In [49]: df1.loc[:,df1.loc['a']>0]
Out[49]:
A
a 0.132003
b 1.130127
c 1.024180
d 0.974466
e 0.545952
f -1.281247
For getting a value explicitly (equiv to deprecated df.get_value(’a’,’A’))
# this is also equivalent to ``df1.at['a','A']``
In [50]: df1.loc['a','A']
Out[50]: 0.13200317033032927
13.7 Selection By Position
Warning: Whether a copy or a reference is returned for a setting operation, may depend on the context. This is
sometimes called chained assignment and should be avoided. See Returning a View versus Copy
pandas provides a suite of methods in order to get purely integer based indexing. The semantics follow closely
python and numpy slicing. These are 0-based indexing. When slicing, the start bounds is included, while the upper
bound is excluded. Trying to use a non-integer, even a valid label will raise a IndexError.
The .iloc attribute is the primary access method. The following are valid inputs:
• An integer e.g. 5
• A list or array of integers [4, 3, 0]
• A slice object with ints 1:7
• A boolean array
In [51]: s1 = Series(np.random.randn(5),index=list(range(0,10,2)))
In [52]: s1
Out[52]:
386
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
0.695775
2
0.341734
4
0.959726
6
-1.110336
8
-0.619976
dtype: float64
In [53]: s1.iloc[:3]
Out[53]:
0
0.695775
2
0.341734
4
0.959726
dtype: float64
In [54]: s1.iloc[3]
Out[54]: -1.1103361028911667
Note that setting works as well:
In [55]: s1.iloc[:3] = 0
In [56]: s1
Out[56]:
0
0.000000
2
0.000000
4
0.000000
6
-1.110336
8
-0.619976
dtype: float64
With a DataFrame
In [57]: df1 = DataFrame(np.random.randn(6,4),
....:
index=list(range(0,12,2)),
....:
columns=list(range(0,8,2)))
....:
In [58]: df1
Out[58]:
0
0
0.149748
2
0.403310
4 -1.369849
6 -0.826591
8
0.995761
10 -0.317441
2
-0.732339
-0.154951
-0.954208
-0.345352
2.396780
-1.236269
4
6
0.687738 0.176444
0.301624 -2.179861
1.462696 -1.743161
1.314232 0.690579
0.014871 3.357427
0.896171 -0.487602
Select via integer slicing
In [59]: df1.iloc[:3]
Out[59]:
0
2
0 0.149748 -0.732339
2 0.403310 -0.154951
4 -1.369849 -0.954208
4
6
0.687738 0.176444
0.301624 -2.179861
1.462696 -1.743161
In [60]: df1.iloc[1:5,2:4]
Out[60]:
4
6
13.7. Selection By Position
387
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
4
6
8
0.301624 -2.179861
1.462696 -1.743161
1.314232 0.690579
0.014871 3.357427
Select via integer list
In [61]: df1.iloc[[1,3,5],[1,3]]
Out[61]:
2
6
2 -0.154951 -2.179861
6 -0.345352 0.690579
10 -1.236269 -0.487602
For slicing rows explicitly (equiv to deprecated df.irow(slice(1,3))).
In [62]: df1.iloc[1:3,:]
Out[62]:
0
2
4
6
2 0.403310 -0.154951 0.301624 -2.179861
4 -1.369849 -0.954208 1.462696 -1.743161
For slicing columns explicitly (equiv to deprecated df.icol(slice(1,3))).
In [63]: df1.iloc[:,1:3]
Out[63]:
2
4
0 -0.732339 0.687738
2 -0.154951 0.301624
4 -0.954208 1.462696
6 -0.345352 1.314232
8
2.396780 0.014871
10 -1.236269 0.896171
For getting a scalar via integer position (equiv to deprecated df.get_value(1,1))
# this is also equivalent to ``df1.iat[1,1]``
In [64]: df1.iloc[1,1]
Out[64]: -0.15495077442490321
For getting a cross section using an integer position (equiv to df.xs(1))
In [65]: df1.iloc[1]
Out[65]:
0
0.403310
2
-0.154951
4
0.301624
6
-2.179861
Name: 2, dtype: float64
Out of range slice indexes are handled gracefully just as in Python/Numpy.
# these are allowed in python/numpy.
# Only works in Pandas starting from v0.14.0.
In [66]: x = list('abcdef')
In [67]: x
Out[67]: ['a', 'b', 'c', 'd', 'e', 'f']
In [68]: x[4:10]
388
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[68]: ['e', 'f']
In [69]: x[8:10]
Out[69]: []
In [70]: s = Series(x)
In [71]: s
Out[71]:
0
a
1
b
2
c
3
d
4
e
5
f
dtype: object
In [72]: s.iloc[4:10]
Out[72]:
4
e
5
f
dtype: object
In [73]: s.iloc[8:10]
Out[73]: Series([], dtype: object)
Note: Prior to v0.14.0, iloc would not accept out of bounds indexers for slices, e.g. a value that exceeds the length
of the object being indexed.
Note that this could result in an empty axis (e.g. an empty DataFrame being returned)
In [74]: dfl = DataFrame(np.random.randn(5,2),columns=list('AB'))
In [75]: dfl
Out[75]:
A
B
0 -0.082240 -2.182937
1 0.380396 0.084844
2 0.432390 1.519970
3 -0.493662 0.600178
4 0.274230 0.132885
In [76]: dfl.iloc[:,2:3]
Out[76]:
Empty DataFrame
Columns: []
Index: [0, 1, 2, 3, 4]
In [77]: dfl.iloc[:,1:3]
Out[77]:
B
0 -2.182937
1 0.084844
2 1.519970
3 0.600178
4 0.132885
In [78]: dfl.iloc[4:6]
13.7. Selection By Position
389
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[78]:
4
A
0.27423
B
0.132885
A single indexer that is out of bounds will raise an IndexError. A list of indexers where any element is out of
bounds will raise an IndexError
dfl.iloc[[4,5,6]]
IndexError: positional indexers are out-of-bounds
dfl.iloc[:,4]
IndexError: single positional indexer is out-of-bounds
13.8 Selecting Random Samples
A random selection of rows or columns from a Series, DataFrame, or Panel with the sample() method. The method
will sample rows by default, and accepts a specific number of rows/columns to return, or a fraction of rows.
In [79]: s = Series([0,1,2,3,4,5])
# When no arguments are passed, returns 1 row.
In [80]: s.sample()
Out[80]:
3
3
dtype: int64
# One may specify either a number of rows:
In [81]: s.sample(n=3)
Out[81]:
5
5
1
1
0
0
dtype: int64
# Or a fraction of the rows:
In [82]: s.sample(frac=0.5)
Out[82]:
1
1
3
3
0
0
dtype: int64
By default, sample will return each row at most once, but one can also sample with replacement using the replace
option:
In [83]: s = Series([0,1,2,3,4,5])
# Without replacement (default):
In [84]: s.sample(n=6, replace=False)
Out[84]:
0
0
5
5
4
4
3
3
2
2
1
1
390
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
dtype: int64
# With replacement:
In [85]: s.sample(n=6, replace=True)
Out[85]:
3
3
4
4
1
1
0
0
4
4
0
0
dtype: int64
By default, each row has an equal probability of being selected, but if you want rows to have different probabilities,
you can pass the sample function sampling weights as weights. These weights can be a list, a numpy array, or a
Series, but they must be of the same length as the object you are sampling. Missing values will be treated as a weight
of zero, and inf values are not allowed. If weights do not sum to 1, they will be re-normalized by dividing all weights
by the sum of the weights. For example:
In [86]: s = Series([0,1,2,3,4,5])
In [87]: example_weights = [0, 0, 0.2, 0.2, 0.2, 0.4]
In [88]: s.sample(n=3, weights=example_weights)
Out[88]:
4
4
5
5
3
3
dtype: int64
# Weights will be re-normalized automatically
In [89]: example_weights2 = [0.5, 0, 0, 0, 0, 0]
In [90]: s.sample(n=1, weights=example_weights2)
Out[90]:
0
0
dtype: int64
When applied to a DataFrame, you can use a column of the DataFrame as sampling weights (provided you are sampling
rows and not columns) by simply passing the name of the column as a string.
In [91]: df2 = DataFrame({'col1':[9,8,7,6], 'weight_column':[0.5, 0.4, 0.1, 0]})
In [92]: df2.sample(n = 3, weights = 'weight_column')
Out[92]:
col1 weight_column
2
7
0.1
0
9
0.5
1
8
0.4
sample also allows users to sample columns instead of rows using the axis argument.
In [93]: df3 = DataFrame({'col1':[1,2,3], 'col2':[2,3,4]})
In [94]: df3.sample(n=1, axis=1)
Out[94]:
col2
0
2
1
3
13.8. Selecting Random Samples
391
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
4
Finally, one can also set a seed for sample‘s random number generator using the random_state argument, which
will accept either an integer (as a seed) or a numpy RandomState object.
In [95]: df4 = DataFrame({'col1':[1,2,3], 'col2':[2,3,4]})
# With a given seed, the sample will always draw the same rows.
In [96]: df4.sample(n=2, random_state=2)
Out[96]:
col1 col2
2
3
4
1
2
3
In [97]: df4.sample(n=2, random_state=2)
Out[97]:
col1 col2
2
3
4
1
2
3
13.9 Setting With Enlargement
New in version 0.13.
The .loc/.ix/[] operations can perform enlargement when setting a non-existant key for that axis.
In the Series case this is effectively an appending operation
In [98]: se = Series([1,2,3])
In [99]: se
Out[99]:
0
1
1
2
2
3
dtype: int64
In [100]: se[5] = 5.
In [101]: se
Out[101]:
0
1
1
2
2
3
5
5
dtype: float64
A DataFrame can be enlarged on either axis via .loc
In [102]: dfi = DataFrame(np.arange(6).reshape(3,2),
.....:
columns=['A','B'])
.....:
In [103]: dfi
Out[103]:
A B
0 0 1
392
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
2
2
4
3
5
In [104]: dfi.loc[:,'C'] = dfi.loc[:,'A']
In [105]: dfi
Out[105]:
A B C
0 0 1 0
1 2 3 2
2 4 5 4
This is like an append operation on the DataFrame.
In [106]: dfi.loc[3] = 5
In [107]: dfi
Out[107]:
A B C
0 0 1 0
1 2 3 2
2 4 5 4
3 5 5 5
13.10 Fast scalar value getting and setting
Since indexing with [] must handle a lot of cases (single-label access, slicing, boolean indexing, etc.), it has a bit of
overhead in order to figure out what you’re asking for. If you only want to access a scalar value, the fastest way is to
use the at and iat methods, which are implemented on all of the data structures.
Similarly to loc, at provides label based scalar lookups, while, iat provides integer based lookups analogously to
iloc
In [108]: s.iat[5]
Out[108]: 5
In [109]: df.at[dates[5], 'A']
Out[109]: 0.11364840968888545
In [110]: df.iat[3, 0]
Out[110]: -0.70677113363008437
You can also set using these same indexers.
In [111]: df.at[dates[5], 'E'] = 7
In [112]: df.iat[3, 0] = 7
at may enlarge the object in-place as above if the indexer is missing.
In [113]: df.at[dates[-1]+1, 0] = 7
In [114]: df
Out[114]:
A
2000-01-01 -0.282863
2000-01-02 -0.173215
B
C
D
E
0
0.469112 -1.509059 -1.135632 NaN NaN
1.212112 0.119209 -1.044236 NaN NaN
13.10. Fast scalar value getting and setting
393
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-03 -2.104569 -0.861849 -0.494929 1.071804 NaN
2000-01-04 7.000000 0.721555 -1.039575 0.271860 NaN
2000-01-05 0.567020 -0.424972 0.276232 -1.087401 NaN
2000-01-06 0.113648 -0.673690 -1.478427 0.524988
7
2000-01-07 0.577046 0.404705 -1.715002 -1.039268 NaN
2000-01-08 -1.157892 -0.370647 -1.344312 0.844885 NaN
2000-01-09
NaN
NaN
NaN
NaN NaN
NaN
NaN
NaN
NaN
NaN
NaN
7
13.11 Boolean indexing
Another common operation is the use of boolean vectors to filter the data. The operators are: | for or, & for and, and
~ for not. These must be grouped by using parentheses.
Using a boolean vector to index a Series works exactly as in a numpy ndarray:
In [115]: s = Series(range(-3, 4))
In [116]: s
Out[116]:
0
-3
1
-2
2
-1
3
0
4
1
5
2
6
3
dtype: int32
In [117]: s[s > 0]
Out[117]:
4
1
5
2
6
3
dtype: int32
In [118]: s[(s < -1) | (s > 0.5)]
Out[118]:
0
-3
1
-2
4
1
5
2
6
3
dtype: int32
In [119]: s[~(s < 0)]
Out[119]:
3
0
4
1
5
2
6
3
dtype: int32
You may select rows from a DataFrame using a boolean vector the same length as the DataFrame’s index (for example,
something derived from one of the columns of the DataFrame):
394
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [120]: df[df['A'] > 0]
Out[120]:
A
B
C
D
E
0
2000-01-04 7.000000 0.721555 -1.039575 0.271860 NaN NaN
2000-01-05 0.567020 -0.424972 0.276232 -1.087401 NaN NaN
2000-01-06 0.113648 -0.673690 -1.478427 0.524988
7 NaN
2000-01-07 0.577046 0.404705 -1.715002 -1.039268 NaN NaN
List comprehensions and map method of Series can also be used to produce more complex criteria:
In [121]: df2 = DataFrame({'a' : ['one', 'one', 'two', 'three', 'two', 'one', 'six'],
.....:
'b' : ['x', 'y', 'y', 'x', 'y', 'x', 'x'],
.....:
'c' : randn(7)})
.....:
# only want 'two' or 'three'
In [122]: criterion = df2['a'].map(lambda x: x.startswith('t'))
In [123]:
Out[123]:
a
2
two
3 three
4
two
df2[criterion]
b
c
y 1.450520
x 0.206053
y -0.251905
# equivalent but slower
In [124]: df2[[x.startswith('t') for x in df2['a']]]
Out[124]:
a b
c
2
two y 1.450520
3 three x 0.206053
4
two y -0.251905
# Multiple criteria
In [125]: df2[criterion & (df2['b'] == 'x')]
Out[125]:
a b
c
3 three x 0.206053
Note, with the choice methods Selection by Label, Selection by Position, and Advanced Indexing you may select along
more than one axis using boolean vectors combined with other indexing expressions.
In [126]: df2.loc[criterion & (df2['b'] == 'x'),'b':'c']
Out[126]:
b
c
3 x 0.206053
13.12 Indexing with isin
Consider the isin method of Series, which returns a boolean vector that is true wherever the Series elements exist in
the passed list. This allows you to select rows where one or more columns have values you want:
In [127]: s = Series(np.arange(5),index=np.arange(5)[::-1],dtype='int64')
In [128]: s
Out[128]:
4
0
13.12. Indexing with isin
395
pandas: powerful Python data analysis toolkit, Release 0.16.1
3
1
2
2
1
3
0
4
dtype: int64
In [129]: s.isin([2, 4, 6])
Out[129]:
4
False
3
False
2
True
1
False
0
True
dtype: bool
In [130]: s[s.isin([2, 4, 6])]
Out[130]:
2
2
0
4
dtype: int64
The same method is available for Index objects and is useful for the cases when you don’t know which of the sought
labels are in fact present:
In [131]: s[s.index.isin([2, 4, 6])]
Out[131]:
4
0
2
2
dtype: int64
# compare it to the following
In [132]: s[[2, 4, 6]]
Out[132]:
2
2
4
0
6
NaN
dtype: float64
In addition to that, MultiIndex allows selecting a separate level to use in the membership check:
In [133]: s_mi = Series(np.arange(6),
.....:
index=pd.MultiIndex.from_product([[0, 1], ['a', 'b', 'c']]))
.....:
In [134]: s_mi
Out[134]:
0 a
0
b
1
c
2
1 a
3
b
4
c
5
dtype: int32
In [135]: s_mi.iloc[s_mi.index.isin([(1, 'a'), (2, 'b'), (0, 'c')])]
Out[135]:
0 c
2
1 a
3
dtype: int32
396
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [136]: s_mi.iloc[s_mi.index.isin(['a', 'c', 'e'], level=1)]
Out[136]:
0 a
0
c
2
1 a
3
c
5
dtype: int32
DataFrame also has an isin method. When calling isin, pass a set of values as either an array or dict. If values is
an array, isin returns a DataFrame of booleans that is the same shape as the original DataFrame, with True wherever
the element is in the sequence of values.
In [137]: df = DataFrame({'vals': [1, 2, 3, 4], 'ids': ['a', 'b', 'f', 'n'],
.....:
'ids2': ['a', 'n', 'c', 'n']})
.....:
In [138]: values = ['a', 'b', 1, 3]
In [139]:
Out[139]:
ids
0
True
1
True
2 False
3 False
df.isin(values)
ids2
True
False
False
False
vals
True
False
True
False
Oftentimes you’ll want to match certain values with certain columns. Just make values a dict where the key is the
column, and the value is a list of items you want to check for.
In [140]: values = {'ids': ['a', 'b'], 'vals': [1, 3]}
In [141]:
Out[141]:
ids
0
True
1
True
2 False
3 False
df.isin(values)
ids2
False
False
False
False
vals
True
False
True
False
Combine DataFrame’s isin with the any() and all() methods to quickly select subsets of your data that meet a
given criteria. To select a row where each column meets its own criterion:
In [142]: values = {'ids': ['a', 'b'], 'ids2': ['a', 'c'], 'vals': [1, 3]}
In [143]: row_mask = df.isin(values).all(1)
In [144]: df[row_mask]
Out[144]:
ids ids2 vals
0
a
a
1
13.13 The where() Method and Masking
Selecting values from a Series with a boolean vector generally returns a subset of the data. To guarantee that selection
output has the same shape as the original data, you can use the where method in Series and DataFrame.
To return only the selected rows
13.13. The where() Method and Masking
397
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [145]: s[s > 0]
Out[145]:
3
1
2
2
1
3
0
4
dtype: int64
To return a Series of the same shape as the original
In [146]: s.where(s > 0)
Out[146]:
4
NaN
3
1
2
2
1
3
0
4
dtype: float64
Selecting values from a DataFrame with a boolean criterion now also preserves input data shape. where is used under
the hood as the implementation. Equivalent is df.where(df < 0)
In [147]: df[df < 0]
Out[147]:
A
2000-01-01
NaN
2000-01-02 -1.048089
2000-01-03
NaN
2000-01-04
NaN
2000-01-05 -2.484478
2000-01-06
NaN
2000-01-07 -1.282782
2000-01-08
NaN
B
C
D
NaN -0.863838
NaN
-0.025747 -0.988387
NaN
NaN
NaN -0.055758
-0.489682
NaN -0.034571
-0.281461
NaN
NaN
-0.977349
NaN -0.064034
NaN -1.071357
NaN
NaN
NaN -0.744471
In addition, where takes an optional other argument for replacement of values where the condition is False, in the
returned copy.
In [148]: df.where(df < 0, -df)
Out[148]:
A
B
C
2000-01-01 -1.266143 -0.299368 -0.863838
2000-01-02 -1.048089 -0.025747 -0.988387
2000-01-03 -1.262731 -1.289997 -0.082423
2000-01-04 -0.536580 -0.489682 -0.369374
2000-01-05 -2.484478 -0.281461 -0.030711
2000-01-06 -1.126203 -0.977349 -1.474071
2000-01-07 -1.282782 -0.781836 -1.071357
2000-01-08 -2.353925 -0.583787 -0.221471
D
-0.408204
-0.094055
-0.055758
-0.034571
-0.109121
-0.064034
-0.441153
-0.744471
You may wish to set values based on some boolean criteria. This can be done intuitively like so:
In [149]: s2 = s.copy()
In [150]: s2[s2 < 0] = 0
In [151]: s2
Out[151]:
4
0
3
1
398
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
2
1
3
0
4
dtype: int64
In [152]: df2 = df.copy()
In [153]: df2[df2 < 0] = 0
In [154]: df2
Out[154]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
1.266143
0.000000
1.262731
0.536580
0.000000
1.126203
0.000000
2.353925
B
0.299368
0.000000
1.289997
0.000000
0.000000
0.000000
0.781836
0.583787
C
0.000000
0.000000
0.082423
0.369374
0.030711
1.474071
0.000000
0.221471
D
0.408204
0.094055
0.000000
0.000000
0.109121
0.000000
0.441153
0.000000
By default, where returns a modified copy of the data. There is an optional parameter inplace so that the original
data can be modified without creating a copy:
In [155]: df_orig = df.copy()
In [156]: df_orig.where(df > 0, -df, inplace=True);
In [157]: df_orig
Out[157]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
A
1.266143
1.048089
1.262731
0.536580
2.484478
1.126203
1.282782
2.353925
B
0.299368
0.025747
1.289997
0.489682
0.281461
0.977349
0.781836
0.583787
C
0.863838
0.988387
0.082423
0.369374
0.030711
1.474071
1.071357
0.221471
D
0.408204
0.094055
0.055758
0.034571
0.109121
0.064034
0.441153
0.744471
alignment
Furthermore, where aligns the input boolean condition (ndarray or DataFrame), such that partial selection with setting
is possible. This is analogous to partial setting via .ix (but on the contents rather than the axis labels)
In [158]: df2 = df.copy()
In [159]: df2[ df2[1:4] > 0 ] = 3
In [160]: df2
Out[160]:
A
B
C
D
2000-01-01 1.266143 0.299368 -0.863838 0.408204
2000-01-02 -1.048089 -0.025747 -0.988387 3.000000
2000-01-03 3.000000 3.000000 3.000000 -0.055758
2000-01-04 3.000000 -0.489682 3.000000 -0.034571
2000-01-05 -2.484478 -0.281461 0.030711 0.109121
2000-01-06 1.126203 -0.977349 1.474071 -0.064034
2000-01-07 -1.282782 0.781836 -1.071357 0.441153
13.13. The where() Method and Masking
399
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-08
2.353925
0.583787
0.221471 -0.744471
New in version 0.13.
Where can also accept axis and level parameters to align the input when performing the where.
In [161]: df2 = df.copy()
In [162]: df2.where(df2>0,df2['A'],axis='index')
Out[162]:
A
B
C
D
2000-01-01 1.266143 0.299368 1.266143 0.408204
2000-01-02 -1.048089 -1.048089 -1.048089 0.094055
2000-01-03 1.262731 1.289997 0.082423 1.262731
2000-01-04 0.536580 0.536580 0.369374 0.536580
2000-01-05 -2.484478 -2.484478 0.030711 0.109121
2000-01-06 1.126203 1.126203 1.474071 1.126203
2000-01-07 -1.282782 0.781836 -1.282782 0.441153
2000-01-08 2.353925 0.583787 0.221471 2.353925
This is equivalent (but faster than) the following.
In [163]: df2 = df.copy()
In [164]: df.apply(lambda x, y: x.where(x>0,y), y=df['A'])
Out[164]:
A
B
C
D
2000-01-01 1.266143 0.299368 1.266143 0.408204
2000-01-02 -1.048089 -1.048089 -1.048089 0.094055
2000-01-03 1.262731 1.289997 0.082423 1.262731
2000-01-04 0.536580 0.536580 0.369374 0.536580
2000-01-05 -2.484478 -2.484478 0.030711 0.109121
2000-01-06 1.126203 1.126203 1.474071 1.126203
2000-01-07 -1.282782 0.781836 -1.282782 0.441153
2000-01-08 2.353925 0.583787 0.221471 2.353925
mask
mask is the inverse boolean operation of where.
In [165]: s.mask(s >= 0)
Out[165]:
4
NaN
3
NaN
2
NaN
1
NaN
0
NaN
dtype: float64
In [166]: df.mask(df
Out[166]:
A
2000-01-01
NaN
2000-01-02 -1.048089
2000-01-03
NaN
2000-01-04
NaN
2000-01-05 -2.484478
2000-01-06
NaN
2000-01-07 -1.282782
2000-01-08
NaN
400
>= 0)
B
C
D
NaN -0.863838
NaN
-0.025747 -0.988387
NaN
NaN
NaN -0.055758
-0.489682
NaN -0.034571
-0.281461
NaN
NaN
-0.977349
NaN -0.064034
NaN -1.071357
NaN
NaN
NaN -0.744471
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
13.14 The query() Method (Experimental)
New in version 0.13.
DataFrame objects have a query() method that allows selection using an expression.
You can get the value of the frame where column b has values between the values of columns a and c. For example:
In [167]: n = 10
In [168]: df = DataFrame(rand(n, 3), columns=list('abc'))
In [169]: df
Out[169]:
a
0 0.191519
1 0.785359
2 0.276464
3 0.875933
4 0.683463
5 0.561196
6 0.772827
7 0.615396
8 0.933140
9 0.788730
b
0.622109
0.779976
0.801872
0.357817
0.712702
0.503083
0.882641
0.075381
0.651378
0.316836
c
0.437728
0.272593
0.958139
0.500995
0.370251
0.013768
0.364886
0.368824
0.397203
0.568099
# pure python
In [170]: df[(df.a < df.b) & (df.b < df.c)]
Out[170]:
a
b
c
2 0.276464 0.801872 0.958139
# query
In [171]: df.query('(a < b) & (b < c)')
Out[171]:
a
b
c
2 0.276464 0.801872 0.958139
Do the same thing but fall back on a named index if there is no column with the name a.
In [172]: df = DataFrame(randint(n / 2, size=(n, 2)), columns=list('bc'))
In [173]: df.index.name = 'a'
In [174]: df
Out[174]:
b c
a
0 2 3
1 4 1
2 4 0
3 4 1
4 1 4
5 1 4
6 0 1
7 0 0
8 4 0
9 4 2
13.14. The query() Method (Experimental)
401
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [175]: df.query('a < b and b < c')
Out[175]:
b c
a
0 2 3
If instead you don’t want to or cannot name your index, you can use the name index in your query expression:
In [176]: df = DataFrame(randint(n, size=(n, 2)), columns=list('bc'))
In [177]: df
Out[177]:
b c
0 3 1
1 2 5
2 2 5
3 6 7
4 4 3
5 5 6
6 4 6
7 2 4
8 2 7
9 9 7
In [178]: df.query('index < b < c')
Out[178]:
b c
1 2 5
3 6 7
Note: If the name of your index overlaps with a column name, the column name is given precedence. For example,
In [179]: df = DataFrame({'a': randint(5, size=5)})
In [180]: df.index.name = 'a'
In [181]: df.query('a > 2') # uses the column 'a', not the index
Out[181]:
a
a
0 3
3 4
You can still use the index in a query expression by using the special identifier ‘index’:
In [182]: df.query('index > 2')
Out[182]:
a
a
3 4
4 1
If for some reason you have a column named index, then you can refer to the index as ilevel_0 as well, but at
this point you should consider renaming your columns to something less ambiguous.
402
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
13.14.1 MultiIndex query() Syntax
You can also use the levels of a DataFrame with a MultiIndex as if they were columns in the frame:
In [183]: import pandas.util.testing as tm
In [184]: n = 10
In [185]: colors = tm.choice(['red', 'green'], size=n)
In [186]: foods = tm.choice(['eggs', 'ham'], size=n)
In [187]: colors
Out[187]:
array(['red', 'green', 'red', 'green', 'red', 'green', 'red', 'green',
'green', 'green'],
dtype='|S5')
In [188]: foods
Out[188]:
array(['ham', 'eggs', 'ham', 'ham', 'ham', 'eggs', 'eggs', 'eggs', 'ham',
'eggs'],
dtype='|S4')
In [189]: index = MultiIndex.from_arrays([colors, foods], names=['color', 'food'])
In [190]: df = DataFrame(randn(n, 2), index=index)
In [191]: df
Out[191]:
color
red
green
red
green
red
green
red
green
food
ham
eggs
ham
ham
ham
eggs
eggs
eggs
ham
eggs
0
1
0.157622
0.111560
-1.270093
-0.193898
-0.234694
-0.171520
-0.363095
1.444721
-0.855732
-0.276134
-0.293555
0.597679
0.120949
1.804172
0.939908
-0.153055
-0.067318
0.325771
-0.697595
-1.258759
In [192]: df.query('color == "red"')
Out[192]:
0
1
color food
red
ham
0.157622 -0.293555
ham -1.270093 0.120949
ham -0.234694 0.939908
eggs -0.363095 -0.067318
If the levels of the MultiIndex are unnamed, you can refer to them using special names:
In [193]: df.index.names = [None, None]
In [194]: df
Out[194]:
0
1
13.14. The query() Method (Experimental)
403
pandas: powerful Python data analysis toolkit, Release 0.16.1
red
green
red
green
red
green
red
green
ham
eggs
ham
ham
ham
eggs
eggs
eggs
ham
eggs
0.157622
0.111560
-1.270093
-0.193898
-0.234694
-0.171520
-0.363095
1.444721
-0.855732
-0.276134
-0.293555
0.597679
0.120949
1.804172
0.939908
-0.153055
-0.067318
0.325771
-0.697595
-1.258759
In [195]: df.query('ilevel_0 == "red"')
Out[195]:
0
1
red ham
0.157622 -0.293555
ham -1.270093 0.120949
ham -0.234694 0.939908
eggs -0.363095 -0.067318
The convention is ilevel_0, which means “index level 0” for the 0th level of the index.
13.14.2 query() Use Cases
A use case for query() is when you have a collection of DataFrame objects that have a subset of column names
(or index levels/names) in common. You can pass the same query to both frames without having to specify which
frame you’re interested in querying
In [196]: df = DataFrame(rand(n, 3), columns=list('abc'))
In [197]: df
Out[197]:
a
0 0.972113
1 0.158930
2 0.053878
3 0.838312
4 0.366946
5 0.699350
6 0.134386
7 0.457034
8 0.933636
9 0.572485
b
0.046532
0.943383
0.254082
0.156925
0.937473
0.502946
0.828932
0.079103
0.418725
0.572111
c
0.917354
0.763162
0.927973
0.690776
0.613365
0.711111
0.742846
0.373047
0.234212
0.416893
In [198]: df2 = DataFrame(rand(n + 2, 3), columns=df.columns)
In [199]: df2
Out[199]:
a
0
0.625883
1
0.477672
2
0.027139
3
0.175274
4
0.565899
5
0.368558
6
0.849930
7
0.330936
8
0.181795
404
b
0.220362
0.974342
0.221022
0.429462
0.569035
0.952385
0.960458
0.260923
0.376800
c
0.622059
0.772985
0.120328
0.657769
0.654196
0.196770
0.381118
0.665491
0.014259
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
9
10
11
0.339135
0.652106
0.403612
0.401351
0.997192
0.058447
0.467574
0.517462
0.045196
In [200]: expr = '0.0 <= a <= c <= 0.5'
In [201]: map(lambda frame: frame.query(expr), [df, df2])
Out[201]:
[Empty DataFrame
Columns: [a, b, c]
Index: [],
a
b
c
2 0.027139 0.221022 0.120328
9 0.339135 0.401351 0.467574]
13.14.3 query() Python versus pandas Syntax Comparison
Full numpy-like syntax
In [202]: df = DataFrame(randint(n, size=(n, 3)), columns=list('abc'))
In [203]: df
Out[203]:
a b c
0 5 3 8
1 8 8 1
2 3 6 8
3 9 1 5
4 8 4 1
5 1 1 2
6 3 4 2
7 1 9 4
8 0 0 2
9 1 2 5
In [204]: df.query('(a < b) & (b < c)')
Out[204]:
a b c
2 3 6 8
9 1 2 5
In [205]: df[(df.a < df.b) & (df.b < df.c)]
Out[205]:
a b c
2 3 6 8
9 1 2 5
Slightly nicer by removing the parentheses (by binding making comparison operators bind tighter than &/|)
In [206]: df.query('a < b & b < c')
Out[206]:
a b c
2 3 6 8
9 1 2 5
Use English instead of symbols
In [207]: df.query('a < b and b < c')
Out[207]:
13.14. The query() Method (Experimental)
405
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
9
a
3
1
b
6
2
c
8
5
Pretty close to how you might write it on paper
In [208]: df.query('a < b < c')
Out[208]:
a b c
2 3 6 8
9 1 2 5
13.14.4 The in and not in operators
query() also supports special use of Python’s in and not in comparison operators, providing a succinct syntax
for calling the isin method of a Series or DataFrame.
# get all rows where columns "a" and "b" have overlapping values
In [209]: df = DataFrame({'a': list('aabbccddeeff'), 'b': list('aaaabbbbcccc'),
.....:
'c': randint(5, size=12), 'd': randint(9, size=12)})
.....:
In [210]:
Out[210]:
a b
0
a a
1
a a
2
b a
3
b a
4
c b
5
c b
6
d b
7
d b
8
e c
9
e c
10 f c
11 f c
df
c
1
0
0
2
0
0
1
1
4
3
2
0
d
7
0
2
8
4
8
3
2
4
7
7
0
In [211]: df.query('a in b')
Out[211]:
a b c d
0 a a 1 7
1 a a 0 0
2 b a 0 2
3 b a 2 8
4 c b 0 4
5 c b 0 8
# How you'd do it in pure Python
In [212]: df[df.a.isin(df.b)]
Out[212]:
a b c d
0 a a 1 7
1 a a 0 0
2 b a 0 2
3 b a 2 8
4 c b 0 4
406
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
5
c
b
In [213]:
Out[213]:
a b
6
d b
7
d b
8
e c
9
e c
10 f c
11 f c
0
8
df.query('a not in b')
c
1
1
4
3
2
0
d
3
2
4
7
7
0
# pure Python
In [214]: df[~df.a.isin(df.b)]
Out[214]:
a b c d
6
d b 1 3
7
d b 1 2
8
e c 4 4
9
e c 3 7
10 f c 2 7
11 f c 0 0
You can combine this with other expressions for very succinct queries:
# rows where cols a and b have overlapping values and col c's values are less than col d's
In [215]: df.query('a in b and c < d')
Out[215]:
a b c d
0 a a 1 7
2 b a 0 2
3 b a 2 8
4 c b 0 4
5 c b 0 8
# pure Python
In [216]: df[df.b.isin(df.a) & (df.c < df.d)]
Out[216]:
a b c d
0
a a 1 7
2
b a 0 2
3
b a 2 8
4
c b 0 4
5
c b 0 8
6
d b 1 3
7
d b 1 2
9
e c 3 7
10 f c 2 7
Note: Note that in and not in are evaluated in Python, since numexpr has no equivalent of this operation.
However, only the in/not in expression itself is evaluated in vanilla Python. For example, in the expression
df.query('a in b + c + d')
(b + c + d) is evaluated by numexpr and then the in operation is evaluated in plain Python. In general, any
operations that can be evaluated using numexpr will be.
13.14. The query() Method (Experimental)
407
pandas: powerful Python data analysis toolkit, Release 0.16.1
13.14.5 Special use of the == operator with list objects
Comparing a list of values to a column using ==/!= works similarly to in/not in
In [217]:
Out[217]:
a b
0
a a
1
a a
2
b a
3
b a
4
c b
5
c b
6
d b
7
d b
8
e c
9
e c
10 f c
11 f c
df.query('b == ["a", "b", "c"]')
c
1
0
0
2
0
0
1
1
4
3
2
0
d
7
0
2
8
4
8
3
2
4
7
7
0
# pure Python
In [218]: df[df.b.isin(["a", "b", "c"])]
Out[218]:
a b c d
0
a a 1 7
1
a a 0 0
2
b a 0 2
3
b a 2 8
4
c b 0 4
5
c b 0 8
6
d b 1 3
7
d b 1 2
8
e c 4 4
9
e c 3 7
10 f c 2 7
11 f c 0 0
In [219]:
Out[219]:
a b
0
a a
3
b a
6
d b
7
d b
10 f c
df.query('c == [1, 2]')
In [220]:
Out[220]:
a b
1
a a
2
b a
4
c b
5
c b
8
e c
9
e c
11 f c
df.query('c != [1, 2]')
c
1
2
1
1
2
c
0
0
0
0
4
3
0
d
7
8
3
2
7
d
0
2
4
8
4
7
0
# using in/not in
In [221]: df.query('[1, 2] in c')
408
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[221]:
a b
0
a a
3
b a
6
d b
7
d b
10 f c
In [222]:
Out[222]:
a b
1
a a
2
b a
4
c b
5
c b
8
e c
9
e c
11 f c
c
1
2
1
1
2
d
7
8
3
2
7
df.query('[1, 2] not in c')
c
0
0
0
0
4
3
0
d
0
2
4
8
4
7
0
# pure Python
In [223]: df[df.c.isin([1, 2])]
Out[223]:
a b c d
0
a a 1 7
3
b a 2 8
6
d b 1 3
7
d b 1 2
10 f c 2 7
13.14.6 Boolean Operators
You can negate boolean expressions with the word not or the ~ operator.
In [224]: df = DataFrame(rand(n, 3), columns=list('abc'))
In [225]: df['bools'] = rand(len(df)) > 0.5
In [226]: df.query('~bools')
Out[226]:
a
b
c
0 0.395827 0.035597 0.171689
2 0.582329 0.898831 0.435002
3 0.078368 0.224708 0.697626
5 0.877177 0.221076 0.287379
6 0.993264 0.861585 0.108845
bools
False
False
False
False
False
In [227]: df.query('not bools')
Out[227]:
a
b
c
0 0.395827 0.035597 0.171689
2 0.582329 0.898831 0.435002
3 0.078368 0.224708 0.697626
5 0.877177 0.221076 0.287379
6 0.993264 0.861585 0.108845
bools
False
False
False
False
False
In [228]: df.query('not bools') == df[~df.bools]
Out[228]:
13.14. The query() Method (Experimental)
409
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
2
3
5
6
a
True
True
True
True
True
b
True
True
True
True
True
c bools
True True
True True
True True
True True
True True
Of course, expressions can be arbitrarily complex too
# short query syntax
In [229]: shorter = df.query('a < b < c and (not bools) or bools > 2')
# equivalent in pure Python
In [230]: longer = df[(df.a < df.b) & (df.b < df.c) & (~df.bools) | (df.bools > 2)]
In [231]: shorter
Out[231]:
a
b
3 0.078368 0.224708
c
0.697626
bools
False
In [232]: longer
Out[232]:
a
b
3 0.078368 0.224708
c
0.697626
bools
False
In [233]: shorter == longer
Out[233]:
a
b
c bools
3 True True True True
13.14.7 Performance of query()
DataFrame.query() using numexpr is slightly faster than Python for large frames
Note: You will only see the performance benefits of using the numexpr engine with DataFrame.query() if
your frame has more than approximately 200,000 rows
410
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
This plot was created using a DataFrame with 3 columns each containing floating point values generated using
numpy.random.randn().
13.15 Duplicate Data
If you want to identify and remove duplicate rows in a DataFrame, there are two methods that will help: duplicated
and drop_duplicates. Each takes as an argument the columns to use to identify duplicated rows.
• duplicated returns a boolean vector whose length is the number of rows, and which indicates whether a row
is duplicated.
• drop_duplicates removes duplicate rows.
By default, the first observed row of a duplicate set is considered unique, but each method has a take_last parameter that indicates the last observed row should be taken instead.
In [234]: df2 = DataFrame({'a' : ['one', 'one', 'two', 'three', 'two', 'one', 'six'],
.....:
'b' : ['x', 'y', 'y', 'x', 'y', 'x', 'x'],
.....:
'c' : np.random.randn(7)})
.....:
In [235]: df2.duplicated(['a','b'])
Out[235]:
0
False
1
False
2
False
3
False
4
True
5
True
6
False
dtype: bool
In [236]: df2.drop_duplicates(['a','b'])
Out[236]:
a b
c
0
one x 0.932713
13.15. Duplicate Data
411
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
2
3
6
one
two
three
six
In [237]:
Out[237]:
a
1
one
3 three
4
two
5
one
6
six
y -0.393510
y -0.548454
x 1.130736
x -1.233298
df2.drop_duplicates(['a','b'], take_last=True)
b
c
y -0.393510
x 1.130736
y -0.447217
x 1.043921
x -1.233298
An alternative way to drop duplicates on the index is .groupby(level=0) combined with first() or last().
In [238]: df3 = df2.set_index('b')
In [239]: df3
Out[239]:
a
c
b
x
one 0.932713
y
one -0.393510
y
two -0.548454
x three 1.130736
y
two -0.447217
x
one 1.043921
x
six -1.233298
In [240]: df3.groupby(level=0).first()
Out[240]:
a
c
b
x one 0.932713
y one -0.393510
# a bit more verbose
In [241]: df3.reset_index().drop_duplicates(subset='b', take_last=False).set_index('b')
Out[241]:
a
c
b
x one 0.932713
y one -0.393510
13.16 Dictionary-like get() method
Each of Series, DataFrame, and Panel have a get method which can return a default value.
In [242]: s = Series([1,2,3], index=['a','b','c'])
In [243]: s.get('a')
Out[243]: 1
# equivalent to s['a']
In [244]: s.get('x', default=-1)
Out[244]: -1
412
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
13.17 The select() Method
Another way to extract slices from an object is with the select method of Series, DataFrame, and Panel. This
method should be used only when there is no more direct way. select takes a function which operates on labels
along axis and returns a boolean. For instance:
In [245]: df.select(lambda x: x == 'A', axis=1)
Out[245]:
A
2000-01-01 0.454389
2000-01-02 0.036249
2000-01-03 0.378125
2000-01-04 0.075871
2000-01-05 -0.677097
2000-01-06 1.482845
2000-01-07 0.272681
2000-01-08 -0.459059
13.18 The lookup() Method
Sometimes you want to extract a set of values given a sequence of row labels and column labels, and the lookup
method allows for this and returns a numpy array. For instance,
In [246]: dflookup = DataFrame(np.random.rand(20,4), columns = ['A','B','C','D'])
In [247]: dflookup.lookup(list(range(0,10,2)), ['B','C','A','B','D'])
Out[247]: array([ 0.012 , 0.3551, 0.3261, 0.4702, 0.3107])
13.19 Index objects
The pandas Index class and its subclasses can be viewed as implementing an ordered multiset. Duplicates are
allowed. However, if you try to convert an Index object with duplicate entries into a set, an exception will be
raised.
Index also provides the infrastructure necessary for lookups, data alignment, and reindexing. The easiest way to
create an Index directly is to pass a list or other sequence to Index:
In [248]: index = Index(['e', 'd', 'a', 'b'])
In [249]: index
Out[249]: Index([u'e', u'd', u'a', u'b'], dtype='object')
In [250]: 'd' in index
Out[250]: True
You can also pass a name to be stored in the index:
In [251]: index = Index(['e', 'd', 'a', 'b'], name='something')
In [252]: index.name
Out[252]: 'something'
The name, if set, will be shown in the console display:
13.17. The select() Method
413
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [253]: index = Index(list(range(5)), name='rows')
In [254]: columns = Index(['A', 'B', 'C'], name='cols')
In [255]: df = DataFrame(np.random.randn(5, 3), index=index, columns=columns)
In [256]: df
Out[256]:
cols
A
B
C
rows
0
0.603791 0.388713 0.544331
1
-0.152978 1.929541 0.202138
2
0.024972 0.117533 -0.184740
3
1.054144 -0.736061 -0.785352
4
-1.362549 -0.063514 0.487562
In [257]: df['A']
Out[257]:
rows
0
0.603791
1
-0.152978
2
0.024972
3
1.054144
4
-1.362549
Name: A, dtype: float64
13.19.1 Setting metadata
New in version 0.13.0. Indexes are “mostly immutable”, but it is possible to set and change their metadata, like the
index name (or, for MultiIndex, levels and labels).
You can use the rename, set_names, set_levels, and set_labels to set these attributes directly. They
default to returning a copy; however, you can specify inplace=True to have the data change in place.
See Advanced Indexing for usage of MultiIndexes.
In [258]: ind = Index([1, 2, 3])
In [259]: ind.rename("apple")
Out[259]: Int64Index([1, 2, 3], dtype='int64', name=u'apple')
In [260]: ind
Out[260]: Int64Index([1, 2, 3], dtype='int64')
In [261]: ind.set_names(["apple"], inplace=True)
In [262]: ind.name = "bob"
In [263]: ind
Out[263]: Int64Index([1, 2, 3], dtype='int64', name=u'bob')
New in version 0.15.0.
set_names, set_levels, and set_labels also take an optional level‘ argument
In [264]: index = MultiIndex.from_product([range(3), ['one', 'two']], names=['first', 'second'])
In [265]: index
414
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[265]:
MultiIndex(levels=[[0, 1, 2], [u'one', u'two']],
labels=[[0, 0, 1, 1, 2, 2], [0, 1, 0, 1, 0, 1]],
names=[u'first', u'second'])
In [266]: index.levels[1]
Out[266]: Index([u'one', u'two'], dtype='object', name=u'second')
In [267]: index.set_levels(["a", "b"], level=1)
Out[267]:
MultiIndex(levels=[[0, 1, 2], [u'a', u'b']],
labels=[[0, 0, 1, 1, 2, 2], [0, 1, 0, 1, 0, 1]],
names=[u'first', u'second'])
13.19.2 Set operations on Index objects
Warning: In 0.15.0. the set operations + and - were deprecated in order to provide these for numeric type
operations on certain index types. + can be replace by .union() or |, and - by .difference().
The two main operations are union (|), intersection (&) These can be directly called as instance methods
or used via overloaded operators. Difference is provided via the .difference() method.
In [268]: a = Index(['c', 'b', 'a'])
In [269]: b = Index(['c', 'e', 'd'])
In [270]: a | b
Out[270]: Index([u'a', u'b', u'c', u'd', u'e'], dtype='object')
In [271]: a & b
Out[271]: Index([u'c'], dtype='object')
In [272]: a.difference(b)
Out[272]: Index([u'a', u'b'], dtype='object')
Also
available
is
the
sym_diff (^)
operation,
which
returns
elements
that
appear
in either idx1 or idx2 but not both.
This is equivalent to the Index created by
idx1.difference(idx2).union(idx2.difference(idx1)), with duplicates dropped.
In [273]: idx1 = Index([1, 2, 3, 4])
In [274]: idx2 = Index([2, 3, 4, 5])
In [275]: idx1.sym_diff(idx2)
Out[275]: Int64Index([1, 5], dtype='int64')
In [276]: idx1 ^ idx2
Out[276]: Int64Index([1, 5], dtype='int64')
13.20 Set / Reset Index
Occasionally you will load or create a data set into a DataFrame and want to add an index after you’ve already done
so. There are a couple of different ways.
13.20. Set / Reset Index
415
pandas: powerful Python data analysis toolkit, Release 0.16.1
13.20.1 Set an index
DataFrame has a set_index method which takes a column name (for a regular Index) or a list of column names
(for a MultiIndex), to create a new, indexed DataFrame:
In [277]: data
Out[277]:
a
b c
0 bar one z
1 bar two y
2 foo one x
3 foo two w
d
1
2
3
4
In [278]: indexed1 = data.set_index('c')
In [279]: indexed1
Out[279]:
a
b d
c
z bar one 1
y bar two 2
x foo one 3
w foo two 4
In [280]: indexed2 = data.set_index(['a', 'b'])
In [281]: indexed2
Out[281]:
c d
a
b
bar one z 1
two y 2
foo one x 3
two w 4
The append keyword option allow you to keep the existing index and append the given columns to a MultiIndex:
In [282]: frame = data.set_index('c', drop=False)
In [283]: frame = frame.set_index(['a', 'b'], append=True)
In [284]: frame
Out[284]:
c d
c a
b
z bar one z 1
y bar two y 2
x foo one x 3
w foo two w 4
Other options in set_index allow you not drop the index columns or to add the index in-place (without creating a
new object):
In [285]: data.set_index('c', drop=False)
Out[285]:
a
b c d
c
z bar one z 1
y bar two y 2
416
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
x
w
foo
foo
one
two
x
w
3
4
In [286]: data.set_index(['a', 'b'], inplace=True)
In [287]: data
Out[287]:
c d
a
b
bar one z 1
two y 2
foo one x 3
two w 4
13.20.2 Reset the index
As a convenience, there is a new function on DataFrame called reset_index which transfers the index values into
the DataFrame’s columns and sets a simple integer index. This is the inverse operation to set_index
In [288]: data
Out[288]:
c d
a
b
bar one z 1
two y 2
foo one x 3
two w 4
In [289]: data.reset_index()
Out[289]:
a
b c d
0 bar one z 1
1 bar two y 2
2 foo one x 3
3 foo two w 4
The output is more similar to a SQL table or a record array. The names for the columns derived from the index are the
ones stored in the names attribute.
You can use the level keyword to remove only a portion of the index:
In [290]: frame
Out[290]:
c d
c a
b
z bar one z 1
y bar two y 2
x foo one x 3
w foo two w 4
In [291]: frame.reset_index(level=1)
Out[291]:
a c d
c b
z one bar z 1
y two bar y 2
x one foo x 3
w two foo w 4
13.20. Set / Reset Index
417
pandas: powerful Python data analysis toolkit, Release 0.16.1
reset_index takes an optional parameter drop which if true simply discards the index, instead of putting index
values in the DataFrame’s columns.
Note: The reset_index method used to be called delevel which is now deprecated.
13.20.3 Adding an ad hoc index
If you create an index yourself, you can just assign it to the index field:
data.index = index
13.21 Returning a view versus a copy
When setting values in a pandas object, care must be taken to avoid what is called chained indexing. Here is an
example.
In [292]: dfmi = DataFrame([list('abcd'),
.....:
list('efgh'),
.....:
list('ijkl'),
.....:
list('mnop')],
.....:
columns=MultiIndex.from_product([['one','two'],
.....:
['first','second']]))
.....:
In [293]: dfmi
Out[293]:
one
two
first second first second
0
a
b
c
d
1
e
f
g
h
2
i
j
k
l
3
m
n
o
p
Compare these two access methods:
In [294]: dfmi['one']['second']
Out[294]:
0
b
1
f
2
j
3
n
Name: second, dtype: object
In [295]: dfmi.loc[:,('one','second')]
Out[295]:
0
b
1
f
2
j
3
n
Name: (one, second), dtype: object
These both yield the same results, so which should you use? It is instructive to understand the order of operations on
these and why method 2 (.loc) is much preferred over method 1 (chained [])
418
Chapter 13. Indexing and Selecting Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
dfmi[’one’] selects the first level of the columns and returns a data frame that is singly-indexed. Then another
python operation dfmi_with_one[’second’] selects the series indexed by ’second’ happens. This is indicated by the variable dfmi_with_one because pandas sees these operations as separate events. e.g. separate calls
to __getitem__, so it has to treat them as linear operations, they happen one after another.
Contrast
this
to
df.loc[:,(’one’,’second’)]
which
passes
a
nested
tuple
of
(slice(None),(’one’,’second’)) to a single call to __getitem__. This allows pandas to deal
with this as a single entity. Furthermore this order of operations can be significantly faster, and allows one to index
both axes if so desired.
13.21.1 Why does the assignment when using chained indexing fail!
So, why does this show the SettingWithCopy warning / and possibly not work when you do chained indexing and
assignment:
dfmi['one']['second'] = value
Since the chained indexing is 2 calls, it is possible that either call may return a copy of the data because of the way
it is sliced. Thus when setting, you are actually setting a copy, and not the original frame data. It is impossible for
pandas to figure this out because their are 2 separate python operations that are not connected.
The SettingWithCopy warning is a ‘heuristic’ to detect this (meaning it tends to catch most cases but is simply a
lightweight check). Figuring this out for real is way complicated.
The .loc operation is a single python operation, and thus can select a slice (which still may be a copy), but allows
pandas to assign that slice back into the frame after it is modified, thus setting the values as you would think.
The reason for having the SettingWithCopy warning is this. Sometimes when you slice an array you will simply
get a view back, which means you can set it no problem. However, even a single dtyped array can generate a copy if
it is sliced in a particular way. A multi-dtyped DataFrame (meaning it has say float and object data), will almost
always yield a copy. Whether a view is created is dependent on the memory layout of the array.
13.21.2 Evaluation order matters
Furthermore, in chained expressions, the order may determine whether a copy is returned or not. If an expression will
set values on a copy of a slice, then a SettingWithCopy exception will be raised (this raise/warn behavior is new
starting in 0.13.0)
You can control the action of a chained assignment via the option mode.chained_assignment, which can take
the values [’raise’,’warn’,None], where showing a warning is the default.
In [296]: dfb = DataFrame({'a' : ['one', 'one', 'two',
.....:
'three', 'two', 'one', 'six'],
.....:
'c' : np.arange(7)})
.....:
# This will show the SettingWithCopyWarning
# but the frame values will be set
In [297]: dfb['c'][dfb.a.str.startswith('o')] = 42
This however is operating on a copy and will not work.
>>> pd.set_option('mode.chained_assignment','warn')
>>> dfb[dfb.a.str.startswith('o')]['c'] = 42
Traceback (most recent call last)
...
SettingWithCopyWarning:
13.21. Returning a view versus a copy
419
pandas: powerful Python data analysis toolkit, Release 0.16.1
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
A chained assignment can also crop up in setting in a mixed dtype frame.
Note: These setting rules apply to all of .loc/.iloc/.ix
This is the correct access method
In [298]: dfc = DataFrame({'A':['aaa','bbb','ccc'],'B':[1,2,3]})
In [299]: dfc.loc[0,'A'] = 11
In [300]: dfc
Out[300]:
A B
0
11 1
1 bbb 2
2 ccc 3
This can work at times, but is not guaranteed, and so should be avoided
In [301]: dfc = dfc.copy()
In [302]: dfc['A'][0] = 111
In [303]: dfc
Out[303]:
A B
0 111 1
1 bbb 2
2 ccc 3
This will not work at all, and so should be avoided
>>> pd.set_option('mode.chained_assignment','raise')
>>> dfc.loc[0]['A'] = 1111
Traceback (most recent call last)
...
SettingWithCopyException:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
Warning: The chained assignment warnings / exceptions are aiming to inform the user of a possibly invalid
assignment. There may be false positives; situations where a chained assignment is inadvertantly reported.
420
Chapter 13. Indexing and Selecting Data
CHAPTER
FOURTEEN
MULTIINDEX / ADVANCED INDEXING
This section covers indexing with a MultiIndex and more advanced indexing features.
See the Indexing and Selecting Data for general indexing documentation.
Warning: Whether a copy or a reference is returned for a setting operation, may depend on the context. This is
sometimes called chained assignment and should be avoided. See Returning a View versus Copy
Warning: In 0.15.0 Index has internally been refactored to no longer sub-class ndarray but instead subclass
PandasObject, similarly to the rest of the pandas objects. This should be a transparent change with only very
limited API implications (See the Internal Refactoring)
See the cookbook for some advanced strategies
14.1 Hierarchical indexing (MultiIndex)
Hierarchical / Multi-level indexing is very exciting as it opens the door to some quite sophisticated data analysis and
manipulation, especially for working with higher dimensional data. In essence, it enables you to store and manipulate
data with an arbitrary number of dimensions in lower dimensional data structures like Series (1d) and DataFrame (2d).
In this section, we will show what exactly we mean by “hierarchical” indexing and how it integrates with the all of
the pandas indexing functionality described above and in prior sections. Later, when discussing group by and pivoting
and reshaping data, we’ll show non-trivial applications to illustrate how it aids in structuring data for analysis.
See the cookbook for some advanced strategies
14.1.1 Creating a MultiIndex (hierarchical index) object
The MultiIndex object is the hierarchical analogue of the standard Index object which typically stores the
axis labels in pandas objects. You can think of MultiIndex an array of tuples where each tuple is unique. A
MultiIndex can be created from a list of arrays (using MultiIndex.from_arrays), an array of tuples (using
MultiIndex.from_tuples), or a crossed set of iterables (using MultiIndex.from_product). The Index
constructor will attempt to return a MultiIndex when it is passed a list of tuples. The following examples demo
different ways to initialize MultiIndexes.
In [1]: arrays = [['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux'],
...:
['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two']]
...:
In [2]: tuples = list(zip(*arrays))
421
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [3]: tuples
Out[3]:
[('bar', 'one'),
('bar', 'two'),
('baz', 'one'),
('baz', 'two'),
('foo', 'one'),
('foo', 'two'),
('qux', 'one'),
('qux', 'two')]
In [4]: index = MultiIndex.from_tuples(tuples, names=['first', 'second'])
In [5]: index
Out[5]:
MultiIndex(levels=[[u'bar', u'baz', u'foo', u'qux'], [u'one', u'two']],
labels=[[0, 0, 1, 1, 2, 2, 3, 3], [0, 1, 0, 1, 0, 1, 0, 1]],
names=[u'first', u'second'])
In [6]: s = Series(randn(8), index=index)
In [7]: s
Out[7]:
first second
bar
one
two
baz
one
two
foo
one
two
qux
one
two
dtype: float64
0.469112
-0.282863
-1.509059
-1.135632
1.212112
-0.173215
0.119209
-1.044236
When you want every pairing of the elements in two iterables,
MultiIndex.from_product function:
it can be easier to use the
In [8]: iterables = [['bar', 'baz', 'foo', 'qux'], ['one', 'two']]
In [9]: MultiIndex.from_product(iterables, names=['first', 'second'])
Out[9]:
MultiIndex(levels=[[u'bar', u'baz', u'foo', u'qux'], [u'one', u'two']],
labels=[[0, 0, 1, 1, 2, 2, 3, 3], [0, 1, 0, 1, 0, 1, 0, 1]],
names=[u'first', u'second'])
As a convenience, you can pass a list of arrays directly into Series or DataFrame to construct a MultiIndex automatically:
In [10]: arrays = [np.array(['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux']),
....:
np.array(['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two'])]
....:
In [11]: s = Series(randn(8), index=arrays)
In [12]: s
Out[12]:
bar one
-0.861849
two
-2.104569
baz one
-0.494929
422
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
two
1.071804
one
0.721555
two
-0.706771
qux one
-1.039575
two
0.271860
dtype: float64
foo
In [13]: df = DataFrame(randn(8, 4), index=arrays)
In [14]: df
Out[14]:
bar one
two
baz one
two
foo one
two
qux one
two
0
-0.424972
-0.673690
0.404705
-0.370647
1.075770
0.357021
-1.294524
-0.013960
1
0.567020
0.113648
0.577046
-1.157892
-0.109050
-0.674600
0.413738
-0.362543
2
0.276232
-1.478427
-1.715002
-1.344312
1.643563
-1.776904
0.276662
-0.006154
3
-1.087401
0.524988
-1.039268
0.844885
-1.469388
-0.968914
-0.472035
-0.923061
All of the MultiIndex constructors accept a names argument which stores string names for the levels themselves.
If no names are provided, None will be assigned:
In [15]: df.index.names
Out[15]: FrozenList([None, None])
This index can back any axis of a pandas object, and the number of levels of the index is up to you:
In [16]: df = DataFrame(randn(3, 8), index=['A', 'B', 'C'], columns=index)
In [17]: df
Out[17]:
first
bar
second
one
A
0.895717
B
0.410835
C
-1.413681
baz
foo
qux
two
one
two
one
two
one
0.805244 -1.206412 2.565646 1.431256 1.340309 -1.170299
0.813850 0.132003 -0.827317 -0.076467 -1.187678 1.130127
1.607920 1.024180 0.569605 0.875906 -2.211372 0.974466
\
first
second
two
A
-0.226169
B
-1.436737
C
-2.006747
In [18]: DataFrame(randn(6, 6), index=index[:6], columns=index[:6])
Out[18]:
first
bar
baz
foo
second
one
two
one
two
one
two
first second
bar
one
-0.410001 -0.078638 0.545952 -1.219217 -1.226825 0.769804
two
-1.281247 -0.727707 -0.121306 -0.097883 0.695775 0.341734
baz
one
0.959726 -1.110336 -0.619976 0.149748 -0.732339 0.687738
two
0.176444 0.403310 -0.154951 0.301624 -2.179861 -1.369849
foo
one
-0.954208 1.462696 -1.743161 -0.826591 -0.345352 1.314232
two
0.690579 0.995761 2.396780 0.014871 3.357427 -0.317441
We’ve “sparsified” the higher levels of the indexes to make the console output a bit easier on the eyes.
It’s worth keeping in mind that there’s nothing preventing you from using tuples as atomic labels on an axis:
14.1. Hierarchical indexing (MultiIndex)
423
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [19]: Series(randn(8), index=tuples)
Out[19]:
(bar, one)
-1.236269
(bar, two)
0.896171
(baz, one)
-0.487602
(baz, two)
-0.082240
(foo, one)
-2.182937
(foo, two)
0.380396
(qux, one)
0.084844
(qux, two)
0.432390
dtype: float64
The reason that the MultiIndex matters is that it can allow you to do grouping, selection, and reshaping operations
as we will describe below and in subsequent areas of the documentation. As you will see in later sections, you can find
yourself working with hierarchically-indexed data without creating a MultiIndex explicitly yourself. However,
when loading data from a file, you may wish to generate your own MultiIndex when preparing the data set.
Note that how the index is displayed by be controlled using the multi_sparse option in
pandas.set_printoptions:
In [20]: pd.set_option('display.multi_sparse', False)
In [21]: df
Out[21]:
first
bar
second
one
A
0.895717
B
0.410835
C
-1.413681
bar
baz
baz
foo
foo
qux
two
one
two
one
two
one
0.805244 -1.206412 2.565646 1.431256 1.340309 -1.170299
0.813850 0.132003 -0.827317 -0.076467 -1.187678 1.130127
1.607920 1.024180 0.569605 0.875906 -2.211372 0.974466
\
first
qux
second
two
A
-0.226169
B
-1.436737
C
-2.006747
In [22]: pd.set_option('display.multi_sparse', True)
14.1.2 Reconstructing the level labels
The method get_level_values will return a vector of the labels for each location at a particular level:
In [23]: index.get_level_values(0)
Out[23]: Index([u'bar', u'bar', u'baz', u'baz', u'foo', u'foo', u'qux', u'qux'], dtype='object', name
In [24]: index.get_level_values('second')
Out[24]: Index([u'one', u'two', u'one', u'two', u'one', u'two', u'one', u'two'], dtype='object', name
14.1.3 Basic indexing on axis with MultiIndex
One of the important features of hierarchical indexing is that you can select data by a “partial” label identifying a
subgroup in the data. Partial selection “drops” levels of the hierarchical index in the result in a completely analogous
way to selecting a column in a regular DataFrame:
424
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [25]: df['bar']
Out[25]:
second
one
two
A
0.895717 0.805244
B
0.410835 0.813850
C
-1.413681 1.607920
In [26]: df['bar', 'one']
Out[26]:
A
0.895717
B
0.410835
C
-1.413681
Name: (bar, one), dtype: float64
In [27]: df['bar']['one']
Out[27]:
A
0.895717
B
0.410835
C
-1.413681
Name: one, dtype: float64
In [28]: s['qux']
Out[28]:
one
-1.039575
two
0.271860
dtype: float64
See Cross-section with hierarchical index for how to select on a deeper level.
14.1.4 Data alignment and using reindex
Operations between differently-indexed objects having MultiIndex on the axes will work as you expect; data
alignment will work the same as an Index of tuples:
In [29]: s + s[:-2]
Out[29]:
bar one
-1.723698
two
-4.209138
baz one
-0.989859
two
2.143608
foo one
1.443110
two
-1.413542
qux one
NaN
two
NaN
dtype: float64
In [30]: s + s[::2]
Out[30]:
bar one
-1.723698
two
NaN
baz one
-0.989859
two
NaN
foo one
1.443110
two
NaN
qux one
-2.079150
two
NaN
dtype: float64
14.1. Hierarchical indexing (MultiIndex)
425
pandas: powerful Python data analysis toolkit, Release 0.16.1
reindex can be called with another MultiIndex or even a list or array of tuples:
In [31]: s.reindex(index[:3])
Out[31]:
first second
bar
one
-0.861849
two
-2.104569
baz
one
-0.494929
dtype: float64
In [32]: s.reindex([('foo', 'two'), ('bar', 'one'), ('qux', 'one'), ('baz', 'one')])
Out[32]:
foo two
-0.706771
bar one
-0.861849
qux one
-1.039575
baz one
-0.494929
dtype: float64
14.2 Advanced indexing with hierarchical index
Syntactically integrating MultiIndex in advanced indexing with .loc/.ix is a bit challenging, but we’ve made
every effort to do so. for example the following works as you would expect:
In [33]: df = df.T
In [34]: df
Out[34]:
A
B
C
first second
bar
one
0.895717 0.410835 -1.413681
two
0.805244 0.813850 1.607920
baz
one
-1.206412 0.132003 1.024180
two
2.565646 -0.827317 0.569605
foo
one
1.431256 -0.076467 0.875906
two
1.340309 -1.187678 -2.211372
qux
one
-1.170299 1.130127 0.974466
two
-0.226169 -1.436737 -2.006747
In [35]: df.loc['bar']
Out[35]:
A
B
C
second
one
0.895717 0.410835 -1.413681
two
0.805244 0.813850 1.607920
In [36]: df.loc['bar', 'two']
Out[36]:
A
0.805244
B
0.813850
C
1.607920
Name: (bar, two), dtype: float64
“Partial” slicing also works quite nicely.
In [37]: df.loc['baz':'foo']
Out[37]:
A
426
B
C
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
first second
baz
one
-1.206412 0.132003 1.024180
two
2.565646 -0.827317 0.569605
foo
one
1.431256 -0.076467 0.875906
two
1.340309 -1.187678 -2.211372
You can slice with a ‘range’ of values, by providing a slice of tuples.
In [38]: df.loc[('baz', 'two'):('qux', 'one')]
Out[38]:
A
B
C
first second
baz
two
2.565646 -0.827317 0.569605
foo
one
1.431256 -0.076467 0.875906
two
1.340309 -1.187678 -2.211372
qux
one
-1.170299 1.130127 0.974466
In [39]: df.loc[('baz', 'two'):'foo']
Out[39]:
A
B
C
first second
baz
two
2.565646 -0.827317 0.569605
foo
one
1.431256 -0.076467 0.875906
two
1.340309 -1.187678 -2.211372
Passing a list of labels or tuples works similar to reindexing:
In [40]: df.ix[[('bar', 'two'), ('qux', 'one')]]
Out[40]:
A
B
C
first second
bar
two
0.805244 0.813850 1.607920
qux
one
-1.170299 1.130127 0.974466
14.2.1 Using slicers
New in version 0.14.0.
In 0.14.0 we added a new way to slice multi-indexed objects. You can slice a multi-index by providing multiple
indexers.
You can provide any of the selectors as if you are indexing by label, see Selection by Label, including slices, lists of
labels, labels, and boolean indexers.
You can use slice(None) to select all the contents of that level. You do not need to specify all the deeper levels,
they will be implied as slice(None).
As usual, both sides of the slicers are included as this is label indexing.
14.2. Advanced indexing with hierarchical index
427
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning: You should specify all axes in the .loc specifier, meaning the indexer for the index and for the
columns. Their are some ambiguous cases where the passed indexer could be mis-interpreted as indexing both
axes, rather than into say the MuliIndex for the rows.
You should do this:
df.loc[(slice('A1','A3'),.....),:]
rather than this:
df.loc[(slice('A1','A3'),.....)]
Warning: You will need to make sure that the selection axes are fully lexsorted!
In [41]: def mklbl(prefix,n):
....:
return ["%s%s" % (prefix,i)
....:
for i in range(n)]
In [42]: miindex = MultiIndex.from_product([mklbl('A',4),
....:
mklbl('B',2),
....:
mklbl('C',4),
....:
mklbl('D',2)])
....:
In [43]: micolumns = MultiIndex.from_tuples([('a','foo'),('a','bar'),
....:
('b','foo'),('b','bah')],
....:
names=['lvl0', 'lvl1'])
....:
In [44]: dfmi = DataFrame(np.arange(len(miindex)*len(micolumns)).reshape((len(miindex),len(micolumns)
....:
index=miindex,
....:
columns=micolumns).sortlevel().sortlevel(axis=1)
....:
In [45]: dfmi
Out[45]:
lvl0
a
lvl1
bar
A0 B0 C0 D0
1
D1
5
C1 D0
9
D1
13
C2 D0
17
D1
21
C3 D0
25
...
...
A3 B1 C0 D1 229
C1 D0 233
D1 237
C2 D0 241
D1 245
C3 D0 249
D1 253
foo
0
4
8
12
16
20
24
...
228
232
236
240
244
248
252
b
bah
3
7
11
15
19
23
27
...
231
235
239
243
247
251
255
foo
2
6
10
14
18
22
26
...
230
234
238
242
246
250
254
[64 rows x 4 columns]
Basic multi-index slicing using slices, lists, and labels.
428
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [46]: dfmi.loc[(slice('A1','A3'),slice(None), ['C1','C3']),:]
Out[46]:
lvl0
a
b
lvl1
bar foo bah foo
A1 B0 C1 D0
73
72
75
74
D1
77
76
79
78
C3 D0
89
88
91
90
D1
93
92
95
94
B1 C1 D0 105 104 107 106
D1 109 108 111 110
C3 D0 121 120 123 122
...
... ... ... ...
A3 B0 C1 D1 205 204 207 206
C3 D0 217 216 219 218
D1 221 220 223 222
B1 C1 D0 233 232 235 234
D1 237 236 239 238
C3 D0 249 248 251 250
D1 253 252 255 254
[24 rows x 4 columns]
You can use a pd.IndexSlice to have a more natural syntax using : rather than using slice(None)
In [47]: idx = pd.IndexSlice
In [48]: dfmi.loc[idx[:,:,['C1','C3']],idx[:,'foo']]
Out[48]:
lvl0
a
b
lvl1
foo foo
A0 B0 C1 D0
8
10
D1
12
14
C3 D0
24
26
D1
28
30
B1 C1 D0
40
42
D1
44
46
C3 D0
56
58
...
... ...
A3 B0 C1 D1 204 206
C3 D0 216 218
D1 220 222
B1 C1 D0 232 234
D1 236 238
C3 D0 248 250
D1 252 254
[32 rows x 2 columns]
It is possible to perform quite complicated selections using this method on multiple axes at the same time.
In [49]: dfmi.loc['A1',(slice(None),'foo')]
Out[49]:
lvl0
a
b
lvl1
foo foo
B0 C0 D0
64
66
D1
68
70
C1 D0
72
74
D1
76
78
C2 D0
80
82
14.2. Advanced indexing with hierarchical index
429
pandas: powerful Python data analysis toolkit, Release 0.16.1
D1
C3 D0
...
B1 C0 D1
C1 D0
D1
C2 D0
D1
C3 D0
D1
84
88
...
100
104
108
112
116
120
124
86
90
...
102
106
110
114
118
122
126
[16 rows x 2 columns]
In [50]: dfmi.loc[idx[:,:,['C1','C3']],idx[:,'foo']]
Out[50]:
lvl0
a
b
lvl1
foo foo
A0 B0 C1 D0
8
10
D1
12
14
C3 D0
24
26
D1
28
30
B1 C1 D0
40
42
D1
44
46
C3 D0
56
58
...
... ...
A3 B0 C1 D1 204 206
C3 D0 216 218
D1 220 222
B1 C1 D0 232 234
D1 236 238
C3 D0 248 250
D1 252 254
[32 rows x 2 columns]
Using a boolean indexer you can provide selection related to the values.
In [51]: mask = dfmi[('a','foo')]>200
In [52]: dfmi.loc[idx[mask,:,['C1','C3']],idx[:,'foo']]
Out[52]:
lvl0
a
b
lvl1
foo foo
A3 B0 C1 D1 204 206
C3 D0 216 218
D1 220 222
B1 C1 D0 232 234
D1 236 238
C3 D0 248 250
D1 252 254
You can also specify the axis argument to .loc to interpret the passed slicers on a single axis.
In [53]: dfmi.loc(axis=0)[:,:,['C1','C3']]
Out[53]:
lvl0
a
b
lvl1
bar foo bah foo
A0 B0 C1 D0
9
8
11
10
D1
13
12
15
14
430
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
C3 D0
D1
B1 C1 D0
D1
C3 D0
...
A3 B0 C1 D1
C3 D0
D1
B1 C1 D0
D1
C3 D0
D1
25
29
41
45
57
...
205
217
221
233
237
249
253
24
28
40
44
56
...
204
216
220
232
236
248
252
27
31
43
47
59
...
207
219
223
235
239
251
255
26
30
42
46
58
...
206
218
222
234
238
250
254
[32 rows x 4 columns]
Furthermore you can set the values using these methods
In [54]: df2 = dfmi.copy()
In [55]: df2.loc(axis=0)[:,:,['C1','C3']] = -10
In [56]: df2
Out[56]:
lvl0
a
lvl1
bar
A0 B0 C0 D0
1
D1
5
C1 D0 -10
D1 -10
C2 D0
17
D1
21
C3 D0 -10
...
...
A3 B1 C0 D1 229
C1 D0 -10
D1 -10
C2 D0 241
D1 245
C3 D0 -10
D1 -10
foo
0
4
-10
-10
16
20
-10
...
228
-10
-10
240
244
-10
-10
b
bah
3
7
-10
-10
19
23
-10
...
231
-10
-10
243
247
-10
-10
foo
2
6
-10
-10
18
22
-10
...
230
-10
-10
242
246
-10
-10
[64 rows x 4 columns]
You can use a right-hand-side of an alignable object as well.
In [57]: df2 = dfmi.copy()
In [58]: df2.loc[idx[:,:,['C1','C3']],:] = df2*1000
In [59]: df2
Out[59]:
lvl0
lvl1
A0 B0 C0 D0
D1
C1 D0
D1
C2 D0
a
bar
1
5
9000
13000
17
foo
0
4
8000
12000
16
b
bah
3
7
11000
15000
19
foo
2
6
10000
14000
18
14.2. Advanced indexing with hierarchical index
431
pandas: powerful Python data analysis toolkit, Release 0.16.1
D1
C3 D0
...
A3 B1 C0 D1
C1 D0
D1
C2 D0
D1
C3 D0
D1
21
25000
...
229
233000
237000
241
245
249000
253000
20
24000
...
228
232000
236000
240
244
248000
252000
23
27000
...
231
235000
239000
243
247
251000
255000
22
26000
...
230
234000
238000
242
246
250000
254000
[64 rows x 4 columns]
14.2.2 Cross-section
The xs method of DataFrame additionally takes a level argument to make selecting data at a particular level of a
MultiIndex easier.
In [60]: df
Out[60]:
A
B
C
first second
bar
one
0.895717 0.410835 -1.413681
two
0.805244 0.813850 1.607920
baz
one
-1.206412 0.132003 1.024180
two
2.565646 -0.827317 0.569605
foo
one
1.431256 -0.076467 0.875906
two
1.340309 -1.187678 -2.211372
qux
one
-1.170299 1.130127 0.974466
two
-0.226169 -1.436737 -2.006747
In [61]: df.xs('one', level='second')
Out[61]:
A
B
C
first
bar
0.895717 0.410835 -1.413681
baz
-1.206412 0.132003 1.024180
foo
1.431256 -0.076467 0.875906
qux
-1.170299 1.130127 0.974466
# using the slicers (new in 0.14.0)
In [62]: df.loc[(slice(None),'one'),:]
Out[62]:
A
B
C
first second
bar
one
0.895717 0.410835 -1.413681
baz
one
-1.206412 0.132003 1.024180
foo
one
1.431256 -0.076467 0.875906
qux
one
-1.170299 1.130127 0.974466
You can also select on the columns with xs(), by providing the axis argument
In [63]: df = df.T
In [64]: df.xs('one', level='second', axis=1)
Out[64]:
first
bar
baz
foo
qux
432
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
A
B
C
0.895717 -1.206412 1.431256 -1.170299
0.410835 0.132003 -0.076467 1.130127
-1.413681 1.024180 0.875906 0.974466
# using the slicers (new in 0.14.0)
In [65]: df.loc[:,(slice(None),'one')]
Out[65]:
first
bar
baz
foo
qux
second
one
one
one
one
A
0.895717 -1.206412 1.431256 -1.170299
B
0.410835 0.132003 -0.076467 1.130127
C
-1.413681 1.024180 0.875906 0.974466
xs() also allows selection with multiple keys
In [66]: df.xs(('one', 'bar'), level=('second', 'first'), axis=1)
Out[66]:
first
bar
second
one
A
0.895717
B
0.410835
C
-1.413681
# using the slicers (new in 0.14.0)
In [67]: df.loc[:,('bar','one')]
Out[67]:
A
0.895717
B
0.410835
C
-1.413681
Name: (bar, one), dtype: float64
New in version 0.13.0.
You can pass drop_level=False to xs() to retain the level that was selected
In [68]: df.xs('one', level='second', axis=1, drop_level=False)
Out[68]:
first
bar
baz
foo
qux
second
one
one
one
one
A
0.895717 -1.206412 1.431256 -1.170299
B
0.410835 0.132003 -0.076467 1.130127
C
-1.413681 1.024180 0.875906 0.974466
versus the result with drop_level=True (the default value)
In [69]: df.xs('one', level='second', axis=1, drop_level=True)
Out[69]:
first
bar
baz
foo
qux
A
0.895717 -1.206412 1.431256 -1.170299
B
0.410835 0.132003 -0.076467 1.130127
C
-1.413681 1.024180 0.875906 0.974466
14.2.3 Advanced reindexing and alignment
The parameter level has been added to the reindex and align methods of pandas objects. This is useful to
broadcast values across a level. For instance:
14.2. Advanced indexing with hierarchical index
433
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [70]: midx = MultiIndex(levels=[['zero', 'one'], ['x','y']],
....:
labels=[[1,1,0,0],[1,0,1,0]])
....:
In [71]: df = DataFrame(randn(4,2), index=midx)
In [72]: df
Out[72]:
one
y
x
zero y
x
0
1
1.519970 -0.493662
0.600178 0.274230
0.132885 -0.023688
2.410179 1.450520
In [73]: df2 = df.mean(level=0)
In [74]: df2
Out[74]:
zero
one
0
1
1.271532 0.713416
1.060074 -0.109716
In [75]: df2.reindex(df.index, level=0)
Out[75]:
0
1
one y 1.060074 -0.109716
x 1.060074 -0.109716
zero y 1.271532 0.713416
x 1.271532 0.713416
# aligning
In [76]: df_aligned, df2_aligned = df.align(df2, level=0)
In [77]: df_aligned
Out[77]:
0
1
one y 1.519970 -0.493662
x 0.600178 0.274230
zero y 0.132885 -0.023688
x 2.410179 1.450520
In [78]: df2_aligned
Out[78]:
0
1
one y 1.060074 -0.109716
x 1.060074 -0.109716
zero y 1.271532 0.713416
x 1.271532 0.713416
14.2.4 Swapping levels with swaplevel()
The swaplevel function can switch the order of two levels:
In [79]: df[:5]
Out[79]:
0
1
one y 1.519970 -0.493662
434
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
x
zero y
x
0.600178 0.274230
0.132885 -0.023688
2.410179 1.450520
In [80]: df[:5].swaplevel(0, 1, axis=0)
Out[80]:
0
1
y one
1.519970 -0.493662
x one
0.600178 0.274230
y zero 0.132885 -0.023688
x zero 2.410179 1.450520
14.2.5 Reordering levels with reorder_levels()
The reorder_levels function generalizes the swaplevel function, allowing you to permute the hierarchical
index levels in one step:
In [81]: df[:5].reorder_levels([1,0], axis=0)
Out[81]:
0
1
y one
1.519970 -0.493662
x one
0.600178 0.274230
y zero 0.132885 -0.023688
x zero 2.410179 1.450520
14.3 The need for sortedness with MultiIndex
Caveat emptor: the present implementation of MultiIndex requires that the labels be sorted for some of the
slicing / indexing routines to work correctly. You can think about breaking the axis into unique groups, where at
the hierarchical level of interest, each distinct group shares a label, but no two have the same label. However, the
MultiIndex does not enforce this: you are responsible for ensuring that things are properly sorted. There is an
important new method sortlevel to sort an axis within a MultiIndex so that its labels are grouped and sorted
by the original ordering of the associated factor at that level. Note that this does not necessarily mean the labels will
be sorted lexicographically!
In [82]: import random; random.shuffle(tuples)
In [83]: s = Series(randn(8), index=MultiIndex.from_tuples(tuples))
In [84]: s
Out[84]:
bar two
0.206053
foo two
-0.251905
bar one
-2.213588
qux two
1.063327
foo one
1.266143
baz two
0.299368
one
-0.863838
qux one
0.408204
dtype: float64
In [85]: s.sortlevel(0)
Out[85]:
bar one
-2.213588
14.3. The need for sortedness with MultiIndex
435
pandas: powerful Python data analysis toolkit, Release 0.16.1
two
0.206053
one
-0.863838
two
0.299368
foo one
1.266143
two
-0.251905
qux one
0.408204
two
1.063327
dtype: float64
baz
In [86]: s.sortlevel(1)
Out[86]:
bar one
-2.213588
baz one
-0.863838
foo one
1.266143
qux one
0.408204
bar two
0.206053
baz two
0.299368
foo two
-0.251905
qux two
1.063327
dtype: float64
Note, you may also pass a level name to sortlevel if the MultiIndex levels are named.
In [87]: s.index.set_names(['L1', 'L2'], inplace=True)
In [88]: s.sortlevel(level='L1')
Out[88]:
L1
L2
bar one
-2.213588
two
0.206053
baz one
-0.863838
two
0.299368
foo one
1.266143
two
-0.251905
qux one
0.408204
two
1.063327
dtype: float64
In [89]: s.sortlevel(level='L2')
Out[89]:
L1
L2
bar one
-2.213588
baz one
-0.863838
foo one
1.266143
qux one
0.408204
bar two
0.206053
baz two
0.299368
foo two
-0.251905
qux two
1.063327
dtype: float64
Some indexing will work even if the data are not sorted, but will be rather inefficient and will also return a copy of the
data rather than a view:
In [90]: s['qux']
Out[90]:
L2
two
1.063327
one
0.408204
436
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
dtype: float64
In [91]: s.sortlevel(1)['qux']
Out[91]:
L2
one
0.408204
two
1.063327
dtype: float64
On higher dimensional objects, you can sort any of the other axes by level if they have a MultiIndex:
In [92]: df.T.sortlevel(1, axis=1)
Out[92]:
zero
one
zero
one
x
x
y
y
0 2.410179 0.600178 0.132885 1.519970
1 1.450520 0.274230 -0.023688 -0.493662
The MultiIndex object has code to explicity check the sort depth. Thus, if you try to index at a depth at which
the index is not sorted, it will raise an exception. Here is a concrete example to illustrate this:
In [93]: tuples = [('a', 'a'), ('a', 'b'), ('b', 'a'), ('b', 'b')]
In [94]: idx = MultiIndex.from_tuples(tuples)
In [95]: idx.lexsort_depth
Out[95]: 2
In [96]: reordered = idx[[1, 0, 3, 2]]
In [97]: reordered.lexsort_depth
Out[97]: 1
In [98]: s = Series(randn(4), index=reordered)
In [99]: s.ix['a':'a']
Out[99]:
a b
-1.048089
a
-0.025747
dtype: float64
However:
>>> s.ix[('a', 'b'):('b', 'a')]
Traceback (most recent call last)
...
KeyError: Key length (3) was greater than MultiIndex lexsort depth (2)
14.4 Take Methods
Similar to numpy ndarrays, pandas Index, Series, and DataFrame also provides the take method that retrieves elements along a given axis at the given indices. The given indices must be either a list or an ndarray of integer index
positions. take will also accept negative integers as relative positions to the end of the object.
In [100]: index = Index(randint(0, 1000, 10))
In [101]: index
14.4. Take Methods
437
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[101]: Int64Index([214, 502, 712, 567, 786, 175, 993, 133, 758, 329], dtype='int64')
In [102]: positions = [0, 9, 3]
In [103]: index[positions]
Out[103]: Int64Index([214, 329, 567], dtype='int64')
In [104]: index.take(positions)
Out[104]: Int64Index([214, 329, 567], dtype='int64')
In [105]: ser = Series(randn(10))
In [106]: ser.iloc[positions]
Out[106]:
0
-0.179666
9
1.824375
3
0.392149
dtype: float64
In [107]: ser.take(positions)
Out[107]:
0
-0.179666
9
1.824375
3
0.392149
dtype: float64
For DataFrames, the given indices should be a 1d list or ndarray that specifies row or column positions.
In [108]: frm = DataFrame(randn(5, 3))
In [109]: frm.take([1, 4, 3])
Out[109]:
0
1
2
1 -1.237881 0.106854 -1.276829
4 0.629675 -1.425966 1.857704
3 0.979542 -1.633678 0.615855
In [110]: frm.take([0, 2], axis=1)
Out[110]:
0
2
0 0.595974 0.601544
1 -1.237881 -1.276829
2 -0.767101 1.499591
3 0.979542 0.615855
4 0.629675 1.857704
It is important to note that the take method on pandas objects are not intended to work on boolean indices and may
return unexpected results.
In [111]: arr = randn(10)
In [112]: arr.take([False, False, True, True])
Out[112]: array([-1.1935, -1.1935, 0.6775, 0.6775])
In [113]: arr[[0, 1]]
Out[113]: array([-1.1935,
0.6775])
In [114]: ser = Series(randn(10))
438
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [115]: ser.take([False, False, True, True])
Out[115]:
0
0.233141
0
0.233141
1
-0.223540
1
-0.223540
dtype: float64
In [116]: ser.ix[[0, 1]]
Out[116]:
0
0.233141
1
-0.223540
dtype: float64
Finally, as a small note on performance, because the take method handles a narrower range of inputs, it can offer
performance that is a good deal faster than fancy indexing.
14.5 CategoricalIndex
New in version 0.16.1.
We introduce a CategoricalIndex, a new type of index object that is useful for supporting indexing with duplicates. This is a container around a Categorical (introduced in v0.15.0) and allows efficient indexing and storage
of an index with a large number of duplicated elements. Prior to 0.16.1, setting the index of a DataFrame/Series
with a category dtype would convert this to regular object-based Index.
In [117]: df = DataFrame({'A' : np.arange(6),
.....:
'B' : Series(list('aabbca')).astype('category',
.....:
categories=list('cab'))
.....:
})
.....:
In [118]: df
Out[118]:
A B
0 0 a
1 1 a
2 2 b
3 3 b
4 4 c
5 5 a
In [119]: df.dtypes
Out[119]:
A
int32
B
category
dtype: object
In [120]: df.B.cat.categories
Out[120]: Index([u'c', u'a', u'b'], dtype='object')
Setting the index, will create create a CategoricalIndex
In [121]: df2 = df.set_index('B')
In [122]: df2.index
Out[122]: CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a'], categories=[u'c', u'a', u'b'], order
14.5. CategoricalIndex
439
pandas: powerful Python data analysis toolkit, Release 0.16.1
Indexing with __getitem__/.iloc/.loc/.ix works similarly to an Index with duplicates. The indexers
MUST be in the category or the operation will raise.
In [123]: df2.loc['a']
Out[123]:
A
B
a 0
a 1
a 5
These PRESERVE the CategoricalIndex
In [124]: df2.loc['a'].index
Out[124]: CategoricalIndex([u'a', u'a', u'a'], categories=[u'c', u'a', u'b'], ordered=False, name=u'B
Sorting will order by the order of the categories
In [125]: df2.sort_index()
Out[125]:
A
B
c 4
a 0
a 1
a 5
b 2
b 3
Groupby operations on the index will preserve the index nature as well
In [126]: df2.groupby(level=0).sum()
Out[126]:
A
B
c 4
a 6
b 5
In [127]: df2.groupby(level=0).sum().index
Out[127]: CategoricalIndex([u'c', u'a', u'b'], categories=[u'c', u'a', u'b'], ordered=False, name=u'B
Reindexing operations, will return a resulting index based on the type of the passed indexer, meaning that passing
a list will return a plain-old-Index; indexing with a Categorical will return a CategoricalIndex, indexed
according to the categories of the PASSED Categorical dtype. This allows one to arbitrarly index these even with
values NOT in the categories, similarly to how you can reindex ANY pandas index.
In [128]: df2.reindex(['a','e'])
Out[128]:
A
B
a
0
a
1
a
5
e NaN
In [129]: df2.reindex(['a','e']).index
Out[129]: Index([u'a', u'a', u'a', u'e'], dtype='object', name=u'B')
In [130]: df2.reindex(pd.Categorical(['a','e'],categories=list('abcde')))
440
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[130]:
A
B
a
0
a
1
a
5
e NaN
In [131]: df2.reindex(pd.Categorical(['a','e'],categories=list('abcde'))).index
Out[131]: CategoricalIndex([u'a', u'a', u'a', u'e'], categories=[u'a', u'b', u'c', u'd', u'e'], order
Warning: Reshaping and Comparision operations on a CategoricalIndex must have the same categories or
a TypeError will be raised.
In [10]: df3 = DataFrame({'A' : np.arange(6),
'B' : Series(list('aabbca')).astype('category',
categories=list('abc'))
}).set_index('B')
In [11]: df3.index
Out[11]: CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a'], categories=[u'a', u'b', u'c'], order
In [12]: pd.concat([df2,df3]
TypeError: categories must match existing categories when appending
14.6 Float64Index
Note: As of 0.14.0, Float64Index is backed by a native float64 dtype array. Prior to 0.14.0, Float64Index
was backed by an object dtype array. Using a float64 dtype in the backend speeds up arithmetic operations by
about 30x and boolean indexing operations on the Float64Index itself are about 2x as fast.
New in version 0.13.0.
By default a Float64Index will be automatically created when passing floating, or mixed-integer-floating values
in index creation. This enables a pure label-based slicing paradigm that makes [],ix,loc for scalar indexing and
slicing work exactly the same.
In [132]: indexf = Index([1.5, 2, 3, 4.5, 5])
In [133]: indexf
Out[133]: Float64Index([1.5, 2.0, 3.0, 4.5, 5.0], dtype='float64')
In [134]: sf = Series(range(5),index=indexf)
In [135]: sf
Out[135]:
1.5
0
2.0
1
3.0
2
4.5
3
5.0
4
dtype: int32
Scalar selection for [],.ix,.loc will always be label based. An integer will match an equal float index (e.g. 3 is
14.6. Float64Index
441
pandas: powerful Python data analysis toolkit, Release 0.16.1
equivalent to 3.0)
In [136]: sf[3]
Out[136]: 2
In [137]: sf[3.0]
Out[137]: 2
In [138]: sf.ix[3]
Out[138]: 2
In [139]: sf.ix[3.0]
Out[139]: 2
In [140]: sf.loc[3]
Out[140]: 2
In [141]: sf.loc[3.0]
Out[141]: 2
The only positional indexing is via iloc
In [142]: sf.iloc[3]
Out[142]: 3
A scalar index that is not found will raise KeyError
Slicing is ALWAYS on the values of the index, for [],ix,loc and ALWAYS positional with iloc
In [143]: sf[2:4]
Out[143]:
2
1
3
2
dtype: int32
In [144]: sf.ix[2:4]
Out[144]:
2
1
3
2
dtype: int32
In [145]: sf.loc[2:4]
Out[145]:
2
1
3
2
dtype: int32
In [146]: sf.iloc[2:4]
Out[146]:
3.0
2
4.5
3
dtype: int32
In float indexes, slicing using floats is allowed
In [147]: sf[2.1:4.6]
Out[147]:
3.0
2
4.5
3
dtype: int32
442
Chapter 14. MultiIndex / Advanced Indexing
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [148]: sf.loc[2.1:4.6]
Out[148]:
3.0
2
4.5
3
dtype: int32
In non-float indexes, slicing using floats will raise a TypeError
In [1]: Series(range(5))[3.5]
TypeError: the label [3.5] is not a proper indexer for this index type (Int64Index)
In [1]: Series(range(5))[3.5:4.5]
TypeError: the slice start [3.5] is not a proper indexer for this index type (Int64Index)
Using a scalar float indexer will be deprecated in a future version, but is allowed for now.
In [3]: Series(range(5))[3.0]
Out[3]: 3
Here is a typical use-case for using this type of indexing. Imagine that you have a somewhat irregular timedelta-like
indexing scheme, but the data is recorded as floats. This could for example be millisecond offsets.
In [149]: dfir = concat([DataFrame(randn(5,2),
.....:
index=np.arange(5) * 250.0,
.....:
columns=list('AB')),
.....:
DataFrame(randn(6,2),
.....:
index=np.arange(4,10) * 250.1,
.....:
columns=list('AB'))])
.....:
In [150]: dfir
Out[150]:
0.0
250.0
500.0
750.0
1000.0
1000.4
1250.5
1500.6
1750.7
2000.8
2250.9
A
0.997289
-0.179129
0.936914
-1.003401
-0.724626
0.310610
-0.974226
-2.281374
-0.742532
2.495362
-0.068954
B
-1.693316
-1.598062
0.912560
1.632781
0.178219
-0.108002
-1.147708
0.760010
1.533318
-0.432771
0.043520
Selection operations then will always work on a value basis, for all selection operators.
In [151]: dfir[0:1000.4]
Out[151]:
A
B
0.0
0.997289 -1.693316
250.0 -0.179129 -1.598062
500.0
0.936914 0.912560
750.0 -1.003401 1.632781
1000.0 -0.724626 0.178219
1000.4 0.310610 -0.108002
In [152]: dfir.loc[0:1001,'A']
Out[152]:
14.6. Float64Index
443
pandas: powerful Python data analysis toolkit, Release 0.16.1
0.0
250.0
500.0
750.0
1000.0
1000.4
Name: A,
0.997289
-0.179129
0.936914
-1.003401
-0.724626
0.310610
dtype: float64
In [153]: dfir.loc[1000.4]
Out[153]:
A
0.310610
B
-0.108002
Name: 1000.4, dtype: float64
You could then easily pick out the first 1 second (1000 ms) of data then.
In [154]: dfir[0:1000]
Out[154]:
A
B
0
0.997289 -1.693316
250 -0.179129 -1.598062
500
0.936914 0.912560
750 -1.003401 1.632781
1000 -0.724626 0.178219
Of course if you need integer based selection, then use iloc
In [155]: dfir.iloc[0:5]
Out[155]:
A
B
0
0.997289 -1.693316
250 -0.179129 -1.598062
500
0.936914 0.912560
750 -1.003401 1.632781
1000 -0.724626 0.178219
444
Chapter 14. MultiIndex / Advanced Indexing
CHAPTER
FIFTEEN
COMPUTATIONAL TOOLS
15.1 Statistical functions
15.1.1 Percent Change
Series, DataFrame, and Panel all have a method pct_change to compute the percent change over a given
number of periods (using fill_method to fill NA/null values before computing the percent change).
In [1]: ser = pd.Series(np.random.randn(8))
In [2]: ser.pct_change()
Out[2]:
0
NaN
1
-1.602976
2
4.334938
3
-0.247456
4
-2.067345
5
-1.142903
6
-1.688214
7
-9.759729
dtype: float64
In [3]: df = pd.DataFrame(np.random.randn(10, 4))
In [4]: df.pct_change(periods=3)
Out[4]:
0
1
2
3
0
NaN
NaN
NaN
NaN
1
NaN
NaN
NaN
NaN
2
NaN
NaN
NaN
NaN
3 -0.218320 -1.054001 1.987147 -0.510183
4 -0.439121 -1.816454 0.649715 -4.822809
5 -0.127833 -3.042065 -5.866604 -1.776977
6 -2.596833 -1.959538 -2.111697 -3.798900
7 -0.117826 -2.169058 0.036094 -0.067696
8 2.492606 -1.357320 -1.205802 -1.558697
9 -1.012977 2.324558 -1.003744 -0.371806
15.1.2 Covariance
The Series object has a method cov to compute covariance between series (excluding NA/null values).
445
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [5]: s1 = pd.Series(np.random.randn(1000))
In [6]: s2 = pd.Series(np.random.randn(1000))
In [7]: s1.cov(s2)
Out[7]: 0.00068010881743109993
Analogously, DataFrame has a method cov to compute pairwise covariances among the series in the DataFrame,
also excluding NA/null values.
Note: Assuming the missing data are missing at random this results in an estimate for the covariance matrix which
is unbiased. However, for many applications this estimate may not be acceptable because the estimated covariance
matrix is not guaranteed to be positive semi-definite. This could lead to estimated correlations having absolute values
which are greater than one, and/or a non-invertible covariance matrix. See Estimation of covariance matrices for more
details.
In [8]: frame = pd.DataFrame(np.random.randn(1000, 5), columns=['a', 'b', 'c', 'd', 'e'])
In [9]: frame.cov()
Out[9]:
a
b
c
d
e
a 1.000882 -0.003177 -0.002698 -0.006889 0.031912
b -0.003177 1.024721 0.000191 0.009212 0.000857
c -0.002698 0.000191 0.950735 -0.031743 -0.005087
d -0.006889 0.009212 -0.031743 1.002983 -0.047952
e 0.031912 0.000857 -0.005087 -0.047952 1.042487
DataFrame.cov also supports an optional min_periods keyword that specifies the required minimum number
of observations for each column pair in order to have a valid result.
In [10]: frame = pd.DataFrame(np.random.randn(20, 3), columns=['a', 'b', 'c'])
In [11]: frame.ix[:5, 'a'] = np.nan
In [12]: frame.ix[5:10, 'b'] = np.nan
In [13]: frame.cov()
Out[13]:
a
b
a 1.210090 -0.430629
b -0.430629 1.240960
c 0.018002 0.347188
c
0.018002
0.347188
1.301149
In [14]: frame.cov(min_periods=12)
Out[14]:
a
b
c
a 1.210090
NaN 0.018002
b
NaN 1.240960 0.347188
c 0.018002 0.347188 1.301149
15.1.3 Correlation
Several methods for computing correlations are provided:
446
Chapter 15. Computational tools
pandas: powerful Python data analysis toolkit, Release 0.16.1
Method name
pearson (default)
kendall
spearman
Description
Standard correlation coefficient
Kendall Tau correlation coefficient
Spearman rank correlation coefficient
All of these are currently computed using pairwise complete observations.
Note: Please see the caveats associated with this method of calculating correlation matrices in the covariance section.
In [15]: frame = pd.DataFrame(np.random.randn(1000, 5), columns=['a', 'b', 'c', 'd', 'e'])
In [16]: frame.ix[::2] = np.nan
# Series with Series
In [17]: frame['a'].corr(frame['b'])
Out[17]: 0.013479040400098794
In [18]: frame['a'].corr(frame['b'], method='spearman')
Out[18]: -0.0072898851595406388
# Pairwise correlation of DataFrame columns
In [19]: frame.corr()
Out[19]:
a
b
c
d
e
a 1.000000 0.013479 -0.049269 -0.042239 -0.028525
b 0.013479 1.000000 -0.020433 -0.011139 0.005654
c -0.049269 -0.020433 1.000000 0.018587 -0.054269
d -0.042239 -0.011139 0.018587 1.000000 -0.017060
e -0.028525 0.005654 -0.054269 -0.017060 1.000000
Note that non-numeric columns will be automatically excluded from the correlation calculation.
Like cov, corr also supports the optional min_periods keyword:
In [20]: frame = pd.DataFrame(np.random.randn(20, 3), columns=['a', 'b', 'c'])
In [21]: frame.ix[:5, 'a'] = np.nan
In [22]: frame.ix[5:10, 'b'] = np.nan
In [23]: frame.corr()
Out[23]:
a
b
a 1.000000 -0.076520
b -0.076520 1.000000
c 0.160092 0.135967
c
0.160092
0.135967
1.000000
In [24]: frame.corr(min_periods=12)
Out[24]:
a
b
c
a 1.000000
NaN 0.160092
b
NaN 1.000000 0.135967
c 0.160092 0.135967 1.000000
A related method corrwith is implemented on DataFrame to compute the correlation between like-labeled Series
contained in different DataFrame objects.
In [25]: index = ['a', 'b', 'c', 'd', 'e']
15.1. Statistical functions
447
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [26]: columns = ['one', 'two', 'three', 'four']
In [27]: df1 = pd.DataFrame(np.random.randn(5, 4), index=index, columns=columns)
In [28]: df2 = pd.DataFrame(np.random.randn(4, 4), index=index[:4], columns=columns)
In [29]: df1.corrwith(df2)
Out[29]:
one
-0.125501
two
-0.493244
three
0.344056
four
0.004183
dtype: float64
In [30]: df2.corrwith(df1, axis=1)
Out[30]:
a
-0.675817
b
0.458296
c
0.190809
d
-0.186275
e
NaN
dtype: float64
15.1.4 Data ranking
The rank method produces a data ranking with ties being assigned the mean of the ranks (by default) for the group:
In [31]: s = pd.Series(np.random.np.random.randn(5), index=list('abcde'))
In [32]: s['d'] = s['b'] # so there's a tie
In [33]: s.rank()
Out[33]:
a
5.0
b
2.5
c
1.0
d
2.5
e
4.0
dtype: float64
rank is also a DataFrame method and can rank either the rows (axis=0) or the columns (axis=1). NaN values are
excluded from the ranking.
In [34]: df = pd.DataFrame(np.random.np.random.randn(10, 6))
In [35]: df[4] = df[2][:5] # some ties
In [36]: df
Out[36]:
0
0 -0.904948
1 -0.976288
2 0.401965
3 0.205954
4 -0.477586
5 -1.092970
6 0.376892
448
1
-1.163537
-0.244652
1.460840
0.369552
-0.730705
-0.689246
0.959292
2
3
4
5
-1.457187 0.135463 -1.457187 0.294650
-0.748406 -0.999601 -0.748406 -0.800809
1.256057 1.308127 1.256057 0.876004
-0.669304 0.038378 -0.669304 1.140296
-1.129149 -0.601463 -1.129149 -0.211196
0.908114 0.204848
NaN 0.463347
0.095572 -0.593740
NaN -0.069180
Chapter 15. Computational tools
pandas: powerful Python data analysis toolkit, Release 0.16.1
7 -1.002601 1.957794 -0.120708 0.094214
8 -0.547231 0.664402 -0.519424 -0.073254
9 -0.250277 -0.237428 -1.056443 0.419477
In [37]:
Out[37]:
0 1
0 4 3
1 2 6
2 1 6
3 4 5
4 5 3
5 1 2
6 4 5
7 2 5
8 2 5
9 2 3
NaN -1.467422
NaN -1.263544
NaN 1.375064
df.rank(1)
2
1.5
4.5
3.5
1.5
1.5
5.0
3.0
3.0
3.0
1.0
3
5
1
5
3
4
3
1
4
4
4
4
1.5
4.5
3.5
1.5
1.5
NaN
NaN
NaN
NaN
NaN
5
6
3
2
6
6
4
2
1
1
5
rank optionally takes a parameter ascending which by default is true; when false, data is reverse-ranked, with
larger values assigned a smaller rank.
rank supports different tie-breaking methods, specified with the method parameter:
• average : average rank of tied group
• min : lowest rank in the group
• max : highest rank in the group
• first : ranks assigned in the order they appear in the array
15.2 Moving (rolling) statistics / moments
For working with time series data, a number of functions are provided for computing common moving or rolling
statistics. Among these are count, sum, mean, median, correlation, variance, covariance, standard deviation, skewness, and kurtosis. All of these methods are in the pandas namespace, but otherwise they can be found in
pandas.stats.moments.
Function
rolling_count
rolling_sum
rolling_mean
rolling_median
rolling_min
rolling_max
rolling_std
rolling_var
rolling_skew
rolling_kurt
rolling_quantile
rolling_apply
rolling_cov
rolling_corr
rolling_window
Description
Number of non-null observations
Sum of values
Mean of values
Arithmetic median of values
Minimum
Maximum
Unbiased standard deviation
Unbiased variance
Unbiased skewness (3rd moment)
Unbiased kurtosis (4th moment)
Sample quantile (value at %)
Generic apply
Unbiased covariance (binary)
Correlation (binary)
Moving window function
Generally these methods all have the same interface. The binary operators (e.g. rolling_corr) take two Series or
DataFrames. Otherwise, they all accept the following arguments:
15.2. Moving (rolling) statistics / moments
449
pandas: powerful Python data analysis toolkit, Release 0.16.1
• window: size of moving window
• min_periods: threshold of non-null data points to require (otherwise result is NA)
• freq: optionally specify a frequency string or DateOffset to pre-conform the data to. Note that prior to pandas v0.8.0, a keyword argument time_rule was used instead of freq that referred to the legacy time rule
constants
• how: optionally specify method for down or re-sampling. Default is is min for rolling_min, max
for rolling_max, median for rolling_median, and mean for all other rolling functions. See
DataFrame.resample()‘s how argument for more information.
These functions can be applied to ndarrays or Series objects:
In [38]: ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
In [39]: ts = ts.cumsum()
In [40]: ts.plot(style='k--')
Out[40]:
In [41]: rolling_mean(ts, 60).plot(style='k')
Out[41]:
They can also be applied to DataFrame objects. This is really just syntactic sugar for applying the moving window
450
Chapter 15. Computational tools
pandas: powerful Python data analysis toolkit, Release 0.16.1
operator to all of the DataFrame’s columns:
In [42]: df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,
....:
columns=['A', 'B', 'C', 'D'])
....:
In [43]: df = df.cumsum()
In [44]: rolling_sum(df, 60).plot(subplots=True)
Out[44]:
array([,
0xab33f5cc>,
0xab3ad3ac>,
0xab297fcc>], dtype=object)
The rolling_apply function takes an extra func argument and performs generic rolling computations. The
func argument should be a single function that produces a single value from an ndarray input. Suppose we wanted
to compute the mean absolute deviation on a rolling basis:
In [45]: mad = lambda x: np.fabs(x - x.mean()).mean()
In [46]: rolling_apply(ts, 60, mad).plot(style='k')
Out[46]:
15.2. Moving (rolling) statistics / moments
451
pandas: powerful Python data analysis toolkit, Release 0.16.1
The rolling_window function performs a generic rolling window computation on the input data. The weights
used in the window are specified by the win_type keyword. The list of recognized types are:
• boxcar
• triang
• blackman
• hamming
• bartlett
• parzen
• bohman
• blackmanharris
• nuttall
• barthann
• kaiser (needs beta)
• gaussian (needs std)
• general_gaussian (needs power, width)
• slepian (needs width).
452
Chapter 15. Computational tools
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [47]: ser = pd.Series(np.random.randn(10), index=pd.date_range('1/1/2000', periods=10))
In [48]: rolling_window(ser, 5, 'triang')
Out[48]:
2000-01-01
NaN
2000-01-02
NaN
2000-01-03
NaN
2000-01-04
NaN
2000-01-05
-1.037870
2000-01-06
-0.767705
2000-01-07
-0.383197
2000-01-08
-0.395513
2000-01-09
-0.558440
2000-01-10
-0.672416
Freq: D, dtype: float64
Note that the boxcar window is equivalent to rolling_mean.
In [49]: rolling_window(ser, 5, 'boxcar')
Out[49]:
2000-01-01
NaN
2000-01-02
NaN
2000-01-03
NaN
2000-01-04
NaN
2000-01-05
-0.841164
2000-01-06
-0.779948
2000-01-07
-0.565487
2000-01-08
-0.502815
2000-01-09
-0.553755
2000-01-10
-0.472211
Freq: D, dtype: float64
In [50]: rolling_mean(ser, 5)
Out[50]:
2000-01-01
NaN
2000-01-02
NaN
2000-01-03
NaN
2000-01-04
NaN
2000-01-05
-0.841164
2000-01-06
-0.779948
2000-01-07
-0.565487
2000-01-08
-0.502815
2000-01-09
-0.553755
2000-01-10
-0.472211
Freq: D, dtype: float64
For some windowing functions, additional parameters must be specified:
In [51]: rolling_window(ser, 5, 'gaussian', std=0.1)
Out[51]:
2000-01-01
NaN
2000-01-02
NaN
2000-01-03
NaN
2000-01-04
NaN
2000-01-05
-1.309989
2000-01-06
-1.153000
2000-01-07
0.606382
2000-01-08
-0.681101
2000-01-09
-0.289724
15.2. Moving (rolling) statistics / moments
453
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-10
-0.996632
Freq: D, dtype: float64
By default the labels are set to the right edge of the window, but a center keyword is available so the labels can be
set at the center. This keyword is available in other rolling functions as well.
In [52]: rolling_window(ser, 5, 'boxcar')
Out[52]:
2000-01-01
NaN
2000-01-02
NaN
2000-01-03
NaN
2000-01-04
NaN
2000-01-05
-0.841164
2000-01-06
-0.779948
2000-01-07
-0.565487
2000-01-08
-0.502815
2000-01-09
-0.553755
2000-01-10
-0.472211
Freq: D, dtype: float64
In [53]: rolling_window(ser, 5, 'boxcar', center=True)
Out[53]:
2000-01-01
NaN
2000-01-02
NaN
2000-01-03
-0.841164
2000-01-04
-0.779948
2000-01-05
-0.565487
2000-01-06
-0.502815
2000-01-07
-0.553755
2000-01-08
-0.472211
2000-01-09
NaN
2000-01-10
NaN
Freq: D, dtype: float64
In [54]: rolling_mean(ser, 5, center=True)
Out[54]:
2000-01-01
NaN
2000-01-02
NaN
2000-01-03
-0.841164
2000-01-04
-0.779948
2000-01-05
-0.565487
2000-01-06
-0.502815
2000-01-07
-0.553755
2000-01-08
-0.472211
2000-01-09
NaN
2000-01-10
NaN
Freq: D, dtype: float64
Note: In rolling sum mode (mean=False) there is no normalization done to the weights. Passing custom weights
of [1, 1, 1] will yield a different result than passing weights of [2, 2, 2], for example. When passing a
win_type instead of explicitly specifying the weights, the weights are already normalized so that the largest weight
is 1.
In contrast, the nature of the rolling mean calculation (mean=True)is such that the weights are normalized with
respect to each other. Weights of [1, 1, 1] and [2, 2, 2] yield the same result.
454
Chapter 15. Computational tools
pandas: powerful Python data analysis toolkit, Release 0.16.1
15.2.1 Binary rolling moments
rolling_cov and rolling_corr can compute moving window statistics about two Series or any combination
of DataFrame/Series or DataFrame/DataFrame. Here is the behavior in each case:
• two Series: compute the statistic for the pairing.
• DataFrame/Series: compute the statistics for each column of the DataFrame with the passed Series, thus
returning a DataFrame.
• DataFrame/DataFrame: by default compute the statistic for matching column names, returning a
DataFrame. If the keyword argument pairwise=True is passed then computes the statistic for each pair
of columns, returning a Panel whose items are the dates in question (see the next section).
For example:
In [55]: df2 = df[:20]
In [56]: rolling_corr(df2, df2['B'], window=5)
Out[56]:
A
B
C
D
2000-01-01
NaN NaN
NaN
NaN
2000-01-02
NaN NaN
NaN
NaN
2000-01-03
NaN NaN
NaN
NaN
2000-01-04
NaN NaN
NaN
NaN
2000-01-05 -0.262853
1 0.334449 0.193380
2000-01-06 -0.083745
1 -0.521587 -0.556126
2000-01-07 -0.292940
1 -0.658532 -0.458128
...
... ..
...
...
2000-01-14 0.519499
1 -0.687277 0.192822
2000-01-15 0.048982
1 0.167669 -0.061463
2000-01-16 0.217190
1 0.167564 -0.326034
2000-01-17 0.641180
1 -0.164780 -0.111487
2000-01-18 0.130422
1 0.322833 0.632383
2000-01-19 0.317278
1 0.384528 0.813656
2000-01-20 0.293598
1 0.159538 0.742381
[20 rows x 4 columns]
15.2.2 Computing rolling pairwise covariances and correlations
In financial data analysis and other fields it’s common to compute covariance and correlation matrices for a collection
of time series. Often one is also interested in moving-window covariance and correlation matrices. This can be done
by passing the pairwise keyword argument, which in the case of DataFrame inputs will yield a Panel whose
items are the dates in question. In the case of a single DataFrame argument the pairwise argument can even be
omitted:
Note: Missing values are ignored and each entry is computed using the pairwise complete observations. Please see
the covariance section for caveats associated with this method of calculating covariance and correlation matrices.
In [57]: covs = rolling_cov(df[['B','C','D']], df[['A','B','C']], 50, pairwise=True)
In [58]: covs[df.index[-50]]
Out[58]:
A
B
C
B 2.667506 1.671711
1.938634
15.2. Moving (rolling) statistics / moments
455
pandas: powerful Python data analysis toolkit, Release 0.16.1
C 8.513843 1.938634
D -7.714737 -1.434529
10.556436
-7.082653
In [59]: correls = rolling_corr(df, 50)
In [60]: correls[df.index[-50]]
Out[60]:
A
B
C
D
A 1.000000 0.604221 0.767429 -0.776170
B 0.604221 1.000000 0.461484 -0.381148
C 0.767429 0.461484 1.000000 -0.748863
D -0.776170 -0.381148 -0.748863 1.000000
Note: Prior to version 0.14 this was available through rolling_corr_pairwise which is now simply syntactic
sugar for calling rolling_corr(..., pairwise=True) and deprecated. This is likely to be removed in a
future release.
You can efficiently retrieve the time series of correlations between two columns using ix indexing:
In [61]: correls.ix[:, 'A', 'C'].plot()
Out[61]:
456
Chapter 15. Computational tools
pandas: powerful Python data analysis toolkit, Release 0.16.1
15.3 Expanding window moment functions
A common alternative to rolling statistics is to use an expanding window, which yields the value of the statistic with
all the data available up to that point in time. As these calculations are a special case of rolling statistics, they are
implemented in pandas such that the following two calls are equivalent:
In [62]: rolling_mean(df, window=len(df), min_periods=1)[:5]
Out[62]:
A
B
C
D
2000-01-01 -1.388345 3.317290 0.344542 -0.036968
2000-01-02 -1.123132 3.622300 1.675867 0.595300
2000-01-03 -0.628502 3.626503 2.455240 1.060158
2000-01-04 -0.768740 3.888917 2.451354 1.281874
2000-01-05 -0.824034 4.108035 2.556112 1.140723
In [63]: expanding_mean(df)[:5]
Out[63]:
A
B
2000-01-01 -1.388345 3.317290
2000-01-02 -1.123132 3.622300
2000-01-03 -0.628502 3.626503
2000-01-04 -0.768740 3.888917
2000-01-05 -0.824034 4.108035
C
D
0.344542 -0.036968
1.675867 0.595300
2.455240 1.060158
2.451354 1.281874
2.556112 1.140723
Like the rolling_ functions, the following methods are included in the pandas namespace or can be located in
pandas.stats.moments.
Function
expanding_count
expanding_sum
expanding_mean
expanding_median
expanding_min
expanding_max
expanding_std
expanding_var
expanding_skew
expanding_kurt
expanding_quantile
expanding_apply
expanding_cov
expanding_corr
Description
Number of non-null observations
Sum of values
Mean of values
Arithmetic median of values
Minimum
Maximum
Unbiased standard deviation
Unbiased variance
Unbiased skewness (3rd moment)
Unbiased kurtosis (4th moment)
Sample quantile (value at %)
Generic apply
Unbiased covariance (binary)
Correlation (binary)
Aside from not having a window parameter, these functions have the same interfaces as their rolling_ counterpart.
Like above, the parameters they all accept are:
• min_periods: threshold of non-null data points to require. Defaults to minimum needed to compute statistic.
No NaNs will be output once min_periods non-null data points have been seen.
• freq: optionally specify a frequency string or DateOffset to pre-conform the data to. Note that prior to pandas v0.8.0, a keyword argument time_rule was used instead of freq that referred to the legacy time rule
constants
Note: The output of the rolling_ and expanding_ functions do not return a NaN if there are at least
min_periods non-null values in the current window. This differs from cumsum, cumprod, cummax, and
cummin, which return NaN in the output wherever a NaN is encountered in the input.
15.3. Expanding window moment functions
457
pandas: powerful Python data analysis toolkit, Release 0.16.1
An expanding window statistic will be more stable (and less responsive) than its rolling window counterpart as
the increasing window size decreases the relative impact of an individual data point. As an example, here is the
expanding_mean output for the previous time series dataset:
In [64]: ts.plot(style='k--')
Out[64]:
In [65]: expanding_mean(ts).plot(style='k')
Out[65]:
15.4 Exponentially weighted moment functions
A related set of functions are exponentially weighted versions of several of the above statistics. A number of expanding
EW (exponentially weighted) functions are provided:
Function
ewma
ewmvar
ewmstd
ewmcorr
ewmcov
458
Description
EW moving average
EW moving variance
EW moving standard deviation
EW moving correlation
EW moving covariance
Chapter 15. Computational tools
pandas: powerful Python data analysis toolkit, Release 0.16.1
In general, a weighted moving average is calculated as
∑︀𝑡
𝑖=0 𝑤𝑖 𝑥𝑡−𝑖
,
𝑦𝑡 = ∑︀
𝑡
𝑖=0 𝑤𝑖
where 𝑥𝑡 is the input at 𝑦𝑡 is the result.
The EW functions support two variants of exponential weights: The default, adjust=True, uses the weights 𝑤𝑖 =
(1 − 𝛼)𝑖 . When adjust=False is specified, moving averages are calculated as
𝑦0 = 𝑥0
𝑦𝑡 = (1 − 𝛼)𝑦𝑡−1 + 𝛼𝑥𝑡 ,
which is equivalent to using weights
{︃
𝑤𝑖 =
𝛼(1 − 𝛼)𝑖
(1 − 𝛼)𝑖
if 𝑖 < 𝑡
if 𝑖 = 𝑡.
Note: These equations are sometimes written in terms of 𝛼′ = 1 − 𝛼, e.g.
𝑦𝑡 = 𝛼′ 𝑦𝑡−1 + (1 − 𝛼′ )𝑥𝑡 .
One must have 0 < 𝛼 ≤ 1, but rather than pass 𝛼 directly, it’s easier to think about either the span, center of mass
(com) or halflife of an EW moment:
⎧
2
⎪
𝑠 = span
⎨ 𝑠+1 ,
1
𝛼 = 1+𝑐 ,
𝑐 = center of mass
⎪
log 0.5
⎩
1 − exp ℎ , ℎ = half life
One must specify precisely one of the three to the EW functions. Span corresponds to what is commonly called a
“20-day EW moving average” for example. Center of mass has a more physical interpretation. For example, span =
20 corresponds to com = 9.5. Halflife is the period of time for the exponential weight to reduce to one half.
Here is an example for a univariate time series:
In [66]: plt.close('all')
In [67]: ts.plot(style='k--')
Out[67]:
In [68]: ewma(ts, span=20).plot(style='k')
Out[68]:
15.4. Exponentially weighted moment functions
459
pandas: powerful Python data analysis toolkit, Release 0.16.1
All the EW functions have a min_periods argument, which has the same meaning it does for all the expanding_
and rolling_ functions: no output values will be set until at least min_periods non-null values are encountered
in the (expanding) window. (This is a change from versions prior to 0.15.0, in which the min_periods argument
affected only the min_periods consecutive entries starting at the first non-null value.)
All the EW functions also have an ignore_na argument, which deterines how intermediate null values affect the
calculation of the weights. When ignore_na=False (the default), weights are calculated based on absolute positions, so that intermediate null values affect the result. When ignore_na=True (which reproduces the behavior
in versions prior to 0.15.0), weights are calculated by ignoring intermediate null values. For example, assuming
adjust=True, if ignore_na=False, the weighted average of 3, NaN, 5 would be calculated as
(1 − 𝛼)2 · 3 + 1 · 5
(1 − 𝛼)2 + 1
Whereas if ignore_na=True, the weighted average would be calculated as
(1 − 𝛼) · 3 + 1 · 5
.
(1 − 𝛼) + 1
The ewmvar, ewmstd, and ewmcov functions have a bias argument, specifying whether the result should contain biased or unbiased statistics. For example, if bias=True, ewmvar(x) is calculated as ewmvar(x) =
ewma(x**2) - ewma(x)**2; whereas if bias=False (the default), the biased variance statistics are scaled
460
Chapter 15. Computational tools
pandas: powerful Python data analysis toolkit, Release 0.16.1
by debiasing factors
(︁∑︀
𝑡
𝑖=0
(︁∑︀
𝑡
𝑖=0
𝑤𝑖
)︁2
𝑤𝑖
)︁2
.
−
∑︀𝑡
2
𝑖=0 𝑤𝑖
(For 𝑤𝑖
= 1, this reduces to the usual 𝑁/(𝑁 − 1) factor, with 𝑁
= 𝑡 + 1.)
http://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_variance for further details.
See
15.4. Exponentially weighted moment functions
461
pandas: powerful Python data analysis toolkit, Release 0.16.1
462
Chapter 15. Computational tools
CHAPTER
SIXTEEN
WORKING WITH MISSING DATA
In this section, we will discuss missing (also referred to as NA) values in pandas.
Note: The choice of using NaN internally to denote missing data was largely for simplicity and performance reasons.
It differs from the MaskedArray approach of, for example, scikits.timeseries. We are hopeful that NumPy
will soon be able to provide a native NA type solution (similar to R) performant enough to be used in pandas.
See the cookbook for some advanced strategies
16.1 Missing data basics
16.1.1 When / why does data become missing?
Some might quibble over our usage of missing. By “missing” we simply mean null or “not present for whatever
reason”. Many data sets simply arrive with missing data, either because it exists and was not collected or it never
existed. For example, in a collection of financial time series, some of the time series might start on different dates.
Thus, values prior to the start date would generally be marked as missing.
In pandas, one of the most common ways that missing data is introduced into a data set is by reindexing. For example
In [1]: df = pd.DataFrame(np.random.randn(5, 3), index=['a', 'c', 'e', 'f', 'h'],
...:
columns=['one', 'two', 'three'])
...:
In [2]: df['four'] = 'bar'
In [3]: df['five'] = df['one'] > 0
In [4]: df
Out[4]:
one
a 0.469112
c -1.135632
e 0.119209
f -2.104569
h 0.721555
two
-0.282863
1.212112
-1.044236
-0.494929
-0.706771
three four
-1.509059 bar
-0.173215 bar
-0.861849 bar
1.071804 bar
-1.039575 bar
five
True
False
True
False
True
In [5]: df2 = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])
In [6]: df2
Out[6]:
one
two
three four
five
463
pandas: powerful Python data analysis toolkit, Release 0.16.1
a 0.469112 -0.282863 -1.509059
b
NaN
NaN
NaN
c -1.135632 1.212112 -0.173215
d
NaN
NaN
NaN
e 0.119209 -1.044236 -0.861849
f -2.104569 -0.494929 1.071804
g
NaN
NaN
NaN
h 0.721555 -0.706771 -1.039575
bar
NaN
bar
NaN
bar
bar
NaN
bar
True
NaN
False
NaN
True
False
NaN
True
16.1.2 Values considered “missing”
As data comes in many shapes and forms, pandas aims to be flexible with regard to handling missing data. While
NaN is the default missing value marker for reasons of computational speed and convenience, we need to be able to
easily detect this value with data of different types: floating point, integer, boolean, and general object. In many cases,
however, the Python None will arise and we wish to also consider that “missing” or “null”.
Prior to version v0.10.0 inf and -inf were also considered to be “null” in computations. This is no longer the case
by default; use the mode.use_inf_as_null option to recover it. To make detecting missing values easier (and
across different array dtypes), pandas provides the isnull() and notnull() functions, which are also methods
on Series objects:
In [7]: df2['one']
Out[7]:
a
0.469112
b
NaN
c
-1.135632
d
NaN
e
0.119209
f
-2.104569
g
NaN
h
0.721555
Name: one, dtype: float64
In [8]: isnull(df2['one'])
Out[8]:
a
False
b
True
c
False
d
True
e
False
f
False
g
True
h
False
Name: one, dtype: bool
In [9]: df2['four'].notnull()
Out[9]:
a
True
b
False
c
True
d
False
e
True
f
True
g
False
h
True
Name: four, dtype: bool
464
Chapter 16. Working with missing data
pandas: powerful Python data analysis toolkit, Release 0.16.1
Summary: NaN and None (in object arrays) are considered missing by the isnull and notnull functions. inf
and -inf are no longer considered missing by default.
16.2 Datetimes
For datetime64[ns] types, NaT represents missing values. This is a pseudo-native sentinel value that can be represented
by numpy in a singular dtype (datetime64[ns]). pandas objects provide intercompatibility between NaT and NaN.
In [10]: df2 = df.copy()
In [11]: df2['timestamp'] = Timestamp('20120101')
In [12]: df2
Out[12]:
one
two
a 0.469112 -0.282863
c -1.135632 1.212112
e 0.119209 -1.044236
f -2.104569 -0.494929
h 0.721555 -0.706771
three four
-1.509059 bar
-0.173215 bar
-0.861849 bar
1.071804 bar
-1.039575 bar
five
True
False
True
False
True
timestamp
2012-01-01
2012-01-01
2012-01-01
2012-01-01
2012-01-01
In [13]: df2.ix[['a','c','h'],['one','timestamp']] = np.nan
In [14]: df2
Out[14]:
one
two
a
NaN -0.282863
c
NaN 1.212112
e 0.119209 -1.044236
f -2.104569 -0.494929
h
NaN -0.706771
three four
-1.509059 bar
-0.173215 bar
-0.861849 bar
1.071804 bar
-1.039575 bar
five timestamp
True
NaT
False
NaT
True 2012-01-01
False 2012-01-01
True
NaT
In [15]: df2.get_dtype_counts()
Out[15]:
bool
1
datetime64[ns]
1
float64
3
object
1
dtype: int64
16.3 Inserting missing data
You can insert missing values by simply assigning to containers. The actual missing value used will be chosen based
on the dtype.
For example, numeric containers will always use NaN regardless of the missing value type chosen:
In [16]: s = pd.Series([1, 2, 3])
In [17]: s.loc[0] = None
In [18]: s
Out[18]:
0
NaN
16.2. Datetimes
465
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
2
2
3
dtype: float64
Likewise, datetime containers will always use NaT.
For object containers, pandas will use the value given:
In [19]: s = pd.Series(["a", "b", "c"])
In [20]: s.loc[0] = None
In [21]: s.loc[1] = np.nan
In [22]: s
Out[22]:
0
None
1
NaN
2
c
dtype: object
16.4 Calculations with missing data
Missing values propagate naturally through arithmetic operations between pandas objects.
In [23]: a
Out[23]:
one
a
NaN
c
NaN
e 0.119209
f -2.104569
h -2.104569
two
-0.282863
1.212112
-1.044236
-0.494929
-0.706771
In [24]: b
Out[24]:
one
a
NaN
c
NaN
e 0.119209
f -2.104569
h
NaN
two
-0.282863
1.212112
-1.044236
-0.494929
-0.706771
three
-1.509059
-0.173215
-0.861849
1.071804
-1.039575
In [25]: a + b
Out[25]:
one three
two
a
NaN
NaN -0.565727
c
NaN
NaN 2.424224
e 0.238417
NaN -2.088472
f -4.209138
NaN -0.989859
h
NaN
NaN -1.413542
The descriptive statistics and computational methods discussed in the data structure overview (and listed here and
here) are all written to account for missing data. For example:
• When summing data, NA (missing) values will be treated as zero
• If the data are all NA, the result will be NA
466
Chapter 16. Working with missing data
pandas: powerful Python data analysis toolkit, Release 0.16.1
• Methods like cumsum and cumprod ignore NA values, but preserve them in the resulting arrays
In [26]: df
Out[26]:
one
a
NaN
c
NaN
e 0.119209
f -2.104569
h
NaN
two
-0.282863
1.212112
-1.044236
-0.494929
-0.706771
three
-1.509059
-0.173215
-0.861849
1.071804
-1.039575
In [27]: df['one'].sum()
Out[27]: -1.9853605075978744
In [28]: df.mean(1)
Out[28]:
a
-0.895961
c
0.519449
e
-0.595625
f
-0.509232
h
-0.873173
dtype: float64
In [29]: df.cumsum()
Out[29]:
one
two
a
NaN -0.282863
c
NaN 0.929249
e 0.119209 -0.114987
f -1.985361 -0.609917
h
NaN -1.316688
three
-1.509059
-1.682273
-2.544122
-1.472318
-2.511893
16.4.1 NA values in GroupBy
NA groups in GroupBy are automatically excluded. This behavior is consistent with R, for example.
16.5 Cleaning / filling missing data
pandas objects are equipped with various data manipulation methods for dealing with missing data.
16.5.1 Filling missing values: fillna
The fillna function can “fill in” NA values with non-null data in a couple of ways, which we illustrate:
Replace NA with a scalar value
In [30]: df2
Out[30]:
one
two
a
NaN -0.282863
c
NaN 1.212112
e 0.119209 -1.044236
f -2.104569 -0.494929
h
NaN -0.706771
three four
-1.509059 bar
-0.173215 bar
-0.861849 bar
1.071804 bar
-1.039575 bar
16.5. Cleaning / filling missing data
five timestamp
True
NaT
False
NaT
True 2012-01-01
False 2012-01-01
True
NaT
467
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [31]: df2.fillna(0)
Out[31]:
one
two
three four
a 0.000000 -0.282863 -1.509059 bar
c 0.000000 1.212112 -0.173215 bar
e 0.119209 -1.044236 -0.861849 bar
f -2.104569 -0.494929 1.071804 bar
h 0.000000 -0.706771 -1.039575 bar
five
True
False
True
False
True
timestamp
1970-01-01
1970-01-01
2012-01-01
2012-01-01
1970-01-01
In [32]: df2['four'].fillna('missing')
Out[32]:
a
bar
c
bar
e
bar
f
bar
h
bar
Name: four, dtype: object
Fill gaps forward or backward
Using the same filling arguments as reindexing, we can propagate non-null values forward or backward:
In [33]: df
Out[33]:
one
a
NaN
c
NaN
e 0.119209
f -2.104569
h
NaN
two
-0.282863
1.212112
-1.044236
-0.494929
-0.706771
three
-1.509059
-0.173215
-0.861849
1.071804
-1.039575
In [34]: df.fillna(method='pad')
Out[34]:
one
two
three
a
NaN -0.282863 -1.509059
c
NaN 1.212112 -0.173215
e 0.119209 -1.044236 -0.861849
f -2.104569 -0.494929 1.071804
h -2.104569 -0.706771 -1.039575
Limit the amount of filling
If we only want consecutive gaps filled up to a certain number of data points, we can use the limit keyword:
In [35]: df
Out[35]:
one
two
three
a NaN -0.282863 -1.509059
c NaN 1.212112 -0.173215
e NaN
NaN
NaN
f NaN
NaN
NaN
h NaN -0.706771 -1.039575
In [36]: df.fillna(method='pad', limit=1)
Out[36]:
one
two
three
a NaN -0.282863 -1.509059
c NaN 1.212112 -0.173215
e NaN 1.212112 -0.173215
468
Chapter 16. Working with missing data
pandas: powerful Python data analysis toolkit, Release 0.16.1
f
h
NaN
NaN
NaN
NaN -0.706771 -1.039575
To remind you, these are the available filling methods:
Method
pad / ffill
bfill / backfill
Action
Fill values forward
Fill values backward
With time series data, using pad/ffill is extremely common so that the “last known value” is available at every time
point.
The ffill() function is equivalent to fillna(method=’ffill’) and bfill() is equivalent to
fillna(method=’bfill’)
16.5.2 Filling with a PandasObject
New in version 0.12.
You can also fillna using a dict or Series that is alignable. The labels of the dict or index of the Series must match the
columns of the frame you wish to fill. The use case of this is to fill a DataFrame with the mean of that column.
In [37]: dff = pd.DataFrame(np.random.randn(10,3),columns=list('ABC'))
In [38]: dff.iloc[3:5,0] = np.nan
In [39]: dff.iloc[4:6,1] = np.nan
In [40]: dff.iloc[5:8,2] = np.nan
In [41]: dff
Out[41]:
A
B
0 0.271860 -0.424972
1 0.276232 -1.087401
2 0.113648 -1.478427
3
NaN 0.577046
4
NaN
NaN
5 -1.344312
NaN
6 -0.109050 1.643563
7 0.357021 -0.674600
8 -0.968914 -1.294524
9 0.276662 -0.472035
C
0.567020
-0.673690
0.524988
-1.715002
-1.157892
NaN
NaN
NaN
0.413738
-0.013960
In [42]: dff.fillna(dff.mean())
Out[42]:
A
B
C
0 0.271860 -0.424972 0.567020
1 0.276232 -1.087401 -0.673690
2 0.113648 -1.478427 0.524988
3 -0.140857 0.577046 -1.715002
4 -0.140857 -0.401419 -1.157892
5 -1.344312 -0.401419 -0.293543
6 -0.109050 1.643563 -0.293543
7 0.357021 -0.674600 -0.293543
8 -0.968914 -1.294524 0.413738
9 0.276662 -0.472035 -0.013960
16.5. Cleaning / filling missing data
469
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [43]: dff.fillna(dff.mean()['B':'C'])
Out[43]:
A
B
C
0 0.271860 -0.424972 0.567020
1 0.276232 -1.087401 -0.673690
2 0.113648 -1.478427 0.524988
3
NaN 0.577046 -1.715002
4
NaN -0.401419 -1.157892
5 -1.344312 -0.401419 -0.293543
6 -0.109050 1.643563 -0.293543
7 0.357021 -0.674600 -0.293543
8 -0.968914 -1.294524 0.413738
9 0.276662 -0.472035 -0.013960
New in version 0.13.
Same result as above, but is aligning the ‘fill’ value which is a Series in this case.
In [44]: dff.where(notnull(dff),dff.mean(),axis='columns')
Out[44]:
A
B
C
0 0.271860 -0.424972 0.567020
1 0.276232 -1.087401 -0.673690
2 0.113648 -1.478427 0.524988
3 -0.140857 0.577046 -1.715002
4 -0.140857 -0.401419 -1.157892
5 -1.344312 -0.401419 -0.293543
6 -0.109050 1.643563 -0.293543
7 0.357021 -0.674600 -0.293543
8 -0.968914 -1.294524 0.413738
9 0.276662 -0.472035 -0.013960
16.5.3 Dropping axis labels with missing data: dropna
You may wish to simply exclude labels from a data set which refer to missing data. To do this, use the dropna method:
In [45]: df
Out[45]:
one
two
three
a NaN -0.282863 -1.509059
c NaN 1.212112 -0.173215
e NaN 0.000000 0.000000
f NaN 0.000000 0.000000
h NaN -0.706771 -1.039575
In [46]: df.dropna(axis=0)
Out[46]:
Empty DataFrame
Columns: [one, two, three]
Index: []
In [47]: df.dropna(axis=1)
Out[47]:
two
three
a -0.282863 -1.509059
c 1.212112 -0.173215
e 0.000000 0.000000
f 0.000000 0.000000
470
Chapter 16. Working with missing data
pandas: powerful Python data analysis toolkit, Release 0.16.1
h -0.706771 -1.039575
In [48]: df['one'].dropna()
Out[48]: Series([], Name: one, dtype: float64)
Series.dropna is a simpler method as it only has one axis to consider. DataFrame.dropna has considerably more options
than Series.dropna, which can be examined in the API.
16.5.4 Interpolation
New in version 0.13.0: interpolate(), and interpolate() have revamped interpolation methods and functionality.
Both Series and Dataframe objects have an interpolate method that, by default, performs linear interpolation at
missing datapoints.
In [49]: ts
Out[49]:
2000-01-31
2000-02-29
2000-03-31
2000-04-28
2000-05-31
2000-06-30
2000-07-31
0.469112
NaN
NaN
NaN
NaN
NaN
NaN
...
2007-10-31
-3.305259
2007-11-30
-5.485119
2007-12-31
-6.854968
2008-01-31
-7.809176
2008-02-29
-6.346480
2008-03-31
-8.089641
2008-04-30
-8.916232
Freq: BM, dtype: float64
In [50]: ts.count()
Out[50]: 61
In [51]: ts.interpolate().count()
Out[51]: 100
In [52]: ts.interpolate().plot()
Out[52]:
16.5. Cleaning / filling missing data
471
pandas: powerful Python data analysis toolkit, Release 0.16.1
Index aware interpolation is available via the method keyword:
In [53]: ts2
Out[53]:
2000-01-31
0.469112
2000-02-29
NaN
2002-07-31
-5.689738
2005-01-31
NaN
2008-04-30
-8.916232
dtype: float64
In [54]: ts2.interpolate()
Out[54]:
2000-01-31
0.469112
2000-02-29
-2.610313
2002-07-31
-5.689738
2005-01-31
-7.302985
2008-04-30
-8.916232
dtype: float64
In [55]: ts2.interpolate(method='time')
Out[55]:
2000-01-31
0.469112
2000-02-29
0.273272
2002-07-31
-5.689738
472
Chapter 16. Working with missing data
pandas: powerful Python data analysis toolkit, Release 0.16.1
2005-01-31
-7.095568
2008-04-30
-8.916232
dtype: float64
For a floating-point index, use method=’values’:
In [56]: ser
Out[56]:
0
0
1
NaN
10
10
dtype: float64
In [57]: ser.interpolate()
Out[57]:
0
0
1
5
10
10
dtype: float64
In [58]: ser.interpolate(method='values')
Out[58]:
0
0
1
1
10
10
dtype: float64
You can also interpolate with a DataFrame:
In [59]: df = pd.DataFrame({'A': [1, 2.1, np.nan, 4.7, 5.6, 6.8],
....:
'B': [.25, np.nan, np.nan, 4, 12.2, 14.4]})
....:
In [60]: df
Out[60]:
A
B
0 1.0
0.25
1 2.1
NaN
2 NaN
NaN
3 4.7
4.00
4 5.6 12.20
5 6.8 14.40
In [61]: df.interpolate()
Out[61]:
A
B
0 1.0
0.25
1 2.1
1.50
2 3.4
2.75
3 4.7
4.00
4 5.6 12.20
5 6.8 14.40
The method argument gives access to fancier interpolation methods. If you have scipy installed, you can set pass the
name of a 1-d interpolation routine to method. You’ll want to consult the full scipy interpolation documentation and
reference guide for details. The appropriate interpolation method will depend on the type of data you are working with.
For example, if you are dealing with a time series that is growing at an increasing rate, method=’quadratic’ may
be appropriate. If you have values approximating a cumulative distribution function, then method=’pchip’ should
work well.
16.5. Cleaning / filling missing data
473
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning: These methods require scipy.
In [62]:
Out[62]:
A
0 1.00
1 2.10
2 3.53
3 4.70
4 5.60
5 6.80
df.interpolate(method='barycentric')
B
0.250
-7.660
-4.515
4.000
12.200
14.400
In [63]: df.interpolate(method='pchip')
Out[63]:
A
B
0 1.000000
0.250000
1 2.100000
1.130135
2 3.429309
2.337586
3 4.700000
4.000000
4 5.600000 12.200000
5 6.800000 14.400000
When interpolating via a polynomial or spline approximation, you must also specify the degree or order of the approximation:
In [64]: df.interpolate(method='spline', order=2)
Out[64]:
A
B
0 1.000000
0.250000
1 2.100000 -0.428598
2 3.404545
1.206900
3 4.700000
4.000000
4 5.600000 12.200000
5 6.800000 14.400000
In [65]: df.interpolate(method='polynomial', order=2)
Out[65]:
A
B
0 1.000000
0.250000
1 2.100000 -4.161538
2 3.547059 -2.911538
3 4.700000
4.000000
4 5.600000 12.200000
5 6.800000 14.400000
Compare several methods:
In [66]: np.random.seed(2)
In [67]: ser = pd.Series(np.arange(1, 10.1, .25)**2 + np.random.randn(37))
In [68]: bad = np.array([4, 13, 14, 15, 16, 17, 18, 20, 29])
In [69]: ser[bad] = np.nan
In [70]: methods = ['linear', 'quadratic', 'cubic']
In [71]: df = pd.DataFrame({m: ser.interpolate(method=m) for m in methods})
474
Chapter 16. Working with missing data
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [72]: df.plot()
Out[72]:
Another use case is interpolation at new values. Suppose you have 100 observations from some distribution. And let’s
suppose that you’re particularly interested in what’s happening around the middle. You can mix pandas’ reindex
and interpolate methods to interpolate at the new values.
In [73]: ser = pd.Series(np.sort(np.random.uniform(size=100)))
# interpolate at new_index
In [74]: new_index = ser.index | Index([49.25, 49.5, 49.75, 50.25, 50.5, 50.75])
In [75]: interp_s = ser.reindex(new_index).interpolate(method='pchip')
In [76]:
Out[76]:
49.00
49.25
49.50
49.75
50.00
50.25
50.50
50.75
interp_s[49:51]
0.471410
0.476841
0.481780
0.485998
0.489266
0.491814
0.493995
0.495763
16.5. Cleaning / filling missing data
475
pandas: powerful Python data analysis toolkit, Release 0.16.1
51.00
0.497074
dtype: float64
Like other pandas fill methods, interpolate accepts a limit keyword argument. Use this to limit the number of
consecutive interpolations, keeping NaN values for interpolations that are too far from the last valid observation:
In [77]: ser = pd.Series([1, 3, np.nan, np.nan, np.nan, 11])
In [78]: ser.interpolate(limit=2)
Out[78]:
0
1
1
3
2
5
3
7
4
NaN
5
11
dtype: float64
16.5.5 Replacing Generic Values
Often times we want to replace arbitrary values with other values. New in v0.8 is the replace method in Series/DataFrame that provides an efficient yet flexible way to perform such replacements.
For a Series, you can replace a single value or a list of values by another value:
In [79]: ser = pd.Series([0., 1., 2., 3., 4.])
In [80]: ser.replace(0, 5)
Out[80]:
0
5
1
1
2
2
3
3
4
4
dtype: float64
You can replace a list of values by a list of other values:
In [81]: ser.replace([0, 1, 2, 3, 4], [4, 3, 2, 1, 0])
Out[81]:
0
4
1
3
2
2
3
1
4
0
dtype: float64
You can also specify a mapping dict:
In [82]: ser.replace({0: 10, 1: 100})
Out[82]:
0
10
1
100
2
2
3
3
4
4
dtype: float64
476
Chapter 16. Working with missing data
pandas: powerful Python data analysis toolkit, Release 0.16.1
For a DataFrame, you can specify individual values by column:
In [83]: df = pd.DataFrame({'a': [0, 1, 2, 3, 4], 'b': [5, 6, 7, 8, 9]})
In [84]: df.replace({'a': 0, 'b': 5}, 100)
Out[84]:
a
b
0 100 100
1
1
6
2
2
7
3
3
8
4
4
9
Instead of replacing with specified values, you can treat all given values as missing and interpolate over them:
In [85]: ser.replace([1, 2, 3], method='pad')
Out[85]:
0
0
1
0
2
0
3
0
4
4
dtype: float64
16.5.6 String/Regular Expression Replacement
Note: Python strings prefixed with the r character such as r’hello world’ are so-called “raw” strings. They
have different semantics regarding backslashes than strings without this prefix. Backslashes in raw strings will be
interpreted as an escaped backslash, e.g., r’\’ == ’\\’. You should read about them if this is unclear.
Replace the ‘.’ with nan (str -> str)
In [86]: d = {'a': list(range(4)), 'b': list('ab..'), 'c': ['a', 'b', np.nan, 'd']}
In [87]: df = pd.DataFrame(d)
In [88]: df.replace('.', np.nan)
Out[88]:
a
b
c
0 0
a
a
1 1
b
b
2 2 NaN NaN
3 3 NaN
d
Now do it with a regular expression that removes surrounding whitespace (regex -> regex)
In [89]: df.replace(r'\s*\.\s*', np.nan, regex=True)
Out[89]:
a
b
c
0 0
a
a
1 1
b
b
2 2 NaN NaN
3 3 NaN
d
Replace a few different values (list -> list)
16.5. Cleaning / filling missing data
477
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [90]: df.replace(['a', '.'], ['b', np.nan])
Out[90]:
a
b
c
0 0
b
b
1 1
b
b
2 2 NaN NaN
3 3 NaN
d
list of regex -> list of regex
In [91]: df.replace([r'\.', r'(a)'], ['dot', '\1stuff'], regex=True)
Out[91]:
a
b
c
0 0 {stuff {stuff
1 1
b
b
2 2
dot
NaN
3 3
dot
d
Only search in column ’b’ (dict -> dict)
In [92]: df.replace({'b': '.'}, {'b': np.nan})
Out[92]:
a
b
c
0 0
a
a
1 1
b
b
2 2 NaN NaN
3 3 NaN
d
Same as the previous example, but use a regular expression for searching instead (dict of regex -> dict)
In [93]: df.replace({'b': r'\s*\.\s*'}, {'b': np.nan}, regex=True)
Out[93]:
a
b
c
0 0
a
a
1 1
b
b
2 2 NaN NaN
3 3 NaN
d
You can pass nested dictionaries of regular expressions that use regex=True
In [94]: df.replace({'b': {'b': r''}}, regex=True)
Out[94]:
a b
c
0 0 a
a
1 1
b
2 2 . NaN
3 3 .
d
or you can pass the nested dictionary like so
In [95]: df.replace(regex={'b': {r'\s*\.\s*': np.nan}})
Out[95]:
a
b
c
0 0
a
a
1 1
b
b
2 2 NaN NaN
3 3 NaN
d
You can also use the group of a regular expression match when replacing (dict of regex -> dict of regex), this works
for lists as well
478
Chapter 16. Working with missing data
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [96]: df.replace({'b': r'\s*(\.)\s*'}, {'b': r'\1ty'}, regex=True)
Out[96]:
a
b
c
0 0
a
a
1 1
b
b
2 2 .ty NaN
3 3 .ty
d
You can pass a list of regular expressions, of which those that match will be replaced with a scalar (list of regex ->
regex)
In [97]: df.replace([r'\s*\.\s*', r'a|b'], np.nan, regex=True)
Out[97]:
a
b
c
0 0 NaN NaN
1 1 NaN NaN
2 2 NaN NaN
3 3 NaN
d
All of the regular expression examples can also be passed with the to_replace argument as the regex argument.
In this case the value argument must be passed explicitly by name or regex must be a nested dictionary. The
previous example, in this case, would then be
In [98]: df.replace(regex=[r'\s*\.\s*', r'a|b'], value=np.nan)
Out[98]:
a
b
c
0 0 NaN NaN
1 1 NaN NaN
2 2 NaN NaN
3 3 NaN
d
This can be convenient if you do not want to pass regex=True every time you want to use a regular expression.
Note: Anywhere in the above replace examples that you see a regular expression a compiled regular expression is
valid as well.
16.5.7 Numeric Replacement
Similar to DataFrame.fillna
In [99]: df = pd.DataFrame(np.random.randn(10, 2))
In [100]: df[np.random.rand(df.shape[0]) > 0.5] = 1.5
In [101]: df.replace(1.5, np.nan)
Out[101]:
0
1
0 -0.844214 -1.021415
1 0.432396 -0.323580
2 0.423825 0.799180
3 1.262614 0.751965
4
NaN
NaN
5
NaN
NaN
6 -0.498174 -1.060799
7 0.591667 -0.183257
8 1.019855 -1.482465
9
NaN
NaN
16.5. Cleaning / filling missing data
479
pandas: powerful Python data analysis toolkit, Release 0.16.1
Replacing more than one value via lists works as well
In [102]: df00 = df.values[0, 0]
In [103]: df.replace([1.5, df00], [np.nan, 'a'])
Out[103]:
0
1
0
a -1.021415
1 0.4323957 -0.323580
2 0.4238247 0.799180
3
1.262614 0.751965
4
NaN
NaN
5
NaN
NaN
6 -0.4981742 -1.060799
7 0.5916665 -0.183257
8
1.019855 -1.482465
9
NaN
NaN
In [104]: df[1].dtype
Out[104]: dtype('float64')
You can also operate on the DataFrame in place
In [105]: df.replace(1.5, np.nan, inplace=True)
Warning:
When replacing multiple bool or datetime64 objects, the first argument to replace
(to_replace) must match the type of the value being replaced type. For example,
s = pd.Series([True, False, True])
s.replace({'a string': 'new value', True: False})
# raises
TypeError: Cannot compare types 'ndarray(dtype=bool)' and 'str'
will raise a TypeError because one of the dict keys is not of the correct type for replacement.
However, when replacing a single object such as,
In [106]: s = pd.Series([True, False, True])
In [107]: s.replace('a string', 'another string')
Out[107]:
0
True
1
False
2
True
dtype: bool
the original NDFrame object will be returned untouched. We’re working on unifying this API, but for backwards
compatibility reasons we cannot break the latter behavior. See GH6354 for more details.
16.6 Missing data casting rules and indexing
While pandas supports storing arrays of integer and boolean type, these types are not capable of storing missing data.
Until we can switch to using a native NA type in NumPy, we’ve established some “casting rules” when reindexing will
cause missing data to be introduced into, say, a Series or DataFrame. Here they are:
480
Chapter 16. Working with missing data
pandas: powerful Python data analysis toolkit, Release 0.16.1
data type
integer
boolean
float
object
Cast to
float
object
no cast
no cast
For example:
In [108]: s = pd.Series(np.random.randn(5), index=[0, 2, 4, 6, 7])
In [109]: s > 0
Out[109]:
0
True
2
True
4
True
6
True
7
True
dtype: bool
In [110]: (s > 0).dtype
Out[110]: dtype('bool')
In [111]: crit = (s > 0).reindex(list(range(8)))
In [112]: crit
Out[112]:
0
True
1
NaN
2
True
3
NaN
4
True
5
NaN
6
True
7
True
dtype: object
In [113]: crit.dtype
Out[113]: dtype('O')
Ordinarily NumPy will complain if you try to use an object array (even if it contains boolean values) instead of a
boolean array to get or set values from an ndarray (e.g. selecting values based on some criteria). If a boolean vector
contains NAs, an exception will be generated:
In [114]: reindexed = s.reindex(list(range(8))).fillna(0)
In [115]: reindexed[crit]
--------------------------------------------------------------------------ValueError
Traceback (most recent call last)
in ()
----> 1 reindexed[crit]
/home/joris/scipy/pandas/pandas/core/series.pyc in __getitem__(self, key)
554
key = list(key)
555
--> 556
if is_bool_indexer(key):
557
key = check_bool_indexer(self.index, key)
558
/home/joris/scipy/pandas/pandas/core/common.pyc in is_bool_indexer(key)
16.6. Missing data casting rules and indexing
481
pandas: powerful Python data analysis toolkit, Release 0.16.1
2126
2127
-> 2128
2129
2130
if not lib.is_bool_array(key):
if isnull(key).any():
raise ValueError('cannot index with vector containing '
'NA / NaN values')
return False
ValueError: cannot index with vector containing NA / NaN values
However, these can be filled in using fillna and it will work fine:
In [116]: reindexed[crit.fillna(False)]
Out[116]:
0
0.126504
2
0.696198
4
0.697416
6
0.601516
7
0.003659
dtype: float64
In [117]: reindexed[crit.fillna(True)]
Out[117]:
0
0.126504
1
0.000000
2
0.696198
3
0.000000
4
0.697416
5
0.000000
6
0.601516
7
0.003659
dtype: float64
482
Chapter 16. Working with missing data
CHAPTER
SEVENTEEN
GROUP BY: SPLIT-APPLY-COMBINE
By “group by” we are referring to a process involving one or more of the following steps
• Splitting the data into groups based on some criteria
• Applying a function to each group independently
• Combining the results into a data structure
Of these, the split step is the most straightforward. In fact, in many situations you may wish to split the data set into
groups and do something with those groups yourself. In the apply step, we might wish to one of the following:
• Aggregation: computing a summary statistic (or statistics) about each group. Some examples:
– Compute group sums or means
– Compute group sizes / counts
• Transformation: perform some group-specific computations and return a like-indexed. Some examples:
– Standardizing data (zscore) within group
– Filling NAs within groups with a value derived from each group
• Filtration: discard some groups, according to a group-wise computation that evaluates True or False. Some
examples:
– Discarding data that belongs to groups with only a few members
– Filtering out data based on the group sum or mean
• Some combination of the above: GroupBy will examine the results of the apply step and try to return a sensibly
combined result if it doesn’t fit into either of the above two categories
Since the set of object instance method on pandas data structures are generally rich and expressive, we often simply
want to invoke, say, a DataFrame function on each group. The name GroupBy should be quite familiar to those who
have used a SQL-based tool (or itertools), in which you can write code like:
SELECT Column1, Column2, mean(Column3), sum(Column4)
FROM SomeTable
GROUP BY Column1, Column2
We aim to make operations like this natural and easy to express using pandas. We’ll address each area of GroupBy
functionality then provide some non-trivial examples / use cases.
See the cookbook for some advanced strategies
483
pandas: powerful Python data analysis toolkit, Release 0.16.1
17.1 Splitting an object into groups
pandas objects can be split on any of their axes. The abstract definition of grouping is to provide a mapping of labels
to group names. To create a GroupBy object (more on what the GroupBy object is later), you do the following:
>>> grouped = obj.groupby(key)
>>> grouped = obj.groupby(key, axis=1)
>>> grouped = obj.groupby([key1, key2])
The mapping can be specified many different ways:
• A Python function, to be called on each of the axis labels
• A list or NumPy array of the same length as the selected axis
• A dict or Series, providing a label -> group name mapping
• For DataFrame objects, a string indicating a column to be used to group. Of course df.groupby(’A’) is
just syntactic sugar for df.groupby(df[’A’]), but it makes life simpler
• A list of any of the above things
Collectively we refer to the grouping objects as the keys. For example, consider the following DataFrame:
In [1]: df = DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
...:
'foo', 'bar', 'foo', 'foo'],
...:
'B' : ['one', 'one', 'two', 'three',
...:
'two', 'two', 'one', 'three'],
...:
'C' : randn(8), 'D' : randn(8)})
...:
In [2]: df
Out[2]:
A
B
0 foo
one
1 bar
one
2 foo
two
3 bar three
4 foo
two
5 bar
two
6 foo
one
7 foo three
C
0.469112
-0.282863
-1.509059
-1.135632
1.212112
-0.173215
0.119209
-1.044236
D
-0.861849
-2.104569
-0.494929
1.071804
0.721555
-0.706771
-1.039575
0.271860
We could naturally group by either the A or B columns or both:
In [3]: grouped = df.groupby('A')
In [4]: grouped = df.groupby(['A', 'B'])
These will split the DataFrame on its index (rows). We could also split by the columns:
In [5]: def get_letter_type(letter):
...:
if letter.lower() in 'aeiou':
...:
return 'vowel'
...:
else:
...:
return 'consonant'
...:
In [6]: grouped = df.groupby(get_letter_type, axis=1)
484
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
Starting with 0.8, pandas Index objects now supports duplicate values. If a non-unique index is used as the group key
in a groupby operation, all values for the same index value will be considered to be in one group and thus the output
of aggregation functions will only contain unique index values:
In [7]: lst = [1, 2, 3, 1, 2, 3]
In [8]: s = Series([1, 2, 3, 10, 20, 30], lst)
In [9]: grouped = s.groupby(level=0)
In [10]: grouped.first()
Out[10]:
1
1
2
2
3
3
dtype: int64
In [11]: grouped.last()
Out[11]:
1
10
2
20
3
30
dtype: int64
In [12]: grouped.sum()
Out[12]:
1
11
2
22
3
33
dtype: int64
Note that no splitting occurs until it’s needed. Creating the GroupBy object only verifies that you’ve passed a valid
mapping.
Note: Many kinds of complicated data manipulations can be expressed in terms of GroupBy operations (though can’t
be guaranteed to be the most efficient). You can get quite creative with the label mapping functions.
17.1.1 GroupBy object attributes
The groups attribute is a dict whose keys are the computed unique groups and corresponding values being the axis
labels belonging to each group. In the above example we have:
In [13]: df.groupby('A').groups
Out[13]: {'bar': [1L, 3L, 5L], 'foo': [0L, 2L, 4L, 6L, 7L]}
In [14]: df.groupby(get_letter_type, axis=1).groups
Out[14]: {'consonant': ['B', 'C', 'D'], 'vowel': ['A']}
Calling the standard Python len function on the GroupBy object just returns the length of the groups dict, so it is
largely just a convenience:
In [15]: grouped = df.groupby(['A', 'B'])
In [16]: grouped.groups
Out[16]:
{('bar', 'one'): [1L],
('bar', 'three'): [3L],
17.1. Splitting an object into groups
485
pandas: powerful Python data analysis toolkit, Release 0.16.1
('bar',
('foo',
('foo',
('foo',
'two'): [5L],
'one'): [0L, 6L],
'three'): [7L],
'two'): [2L, 4L]}
In [17]: len(grouped)
Out[17]: 6
By default the group keys are sorted during the groupby operation. You may however pass sort=False for potential
speedups:
In [18]: df2 = DataFrame({'X' : ['B', 'B', 'A', 'A'], 'Y' : [1, 2, 3, 4]})
In [19]: df2.groupby(['X'], sort=True).sum()
Out[19]:
Y
X
A 7
B 3
In [20]: df2.groupby(['X'], sort=False).sum()
Out[20]:
Y
X
B 3
A 7
GroupBy will tab complete column names (and other attributes)
In [21]: df
Out[21]:
2000-01-01
2000-01-02
2000-01-03
2000-01-04
2000-01-05
2000-01-06
2000-01-07
2000-01-08
2000-01-09
2000-01-10
gender
male
male
male
female
male
female
male
female
female
male
height
42.849980
49.607315
56.293531
48.421077
46.556882
68.448851
70.757698
58.909500
76.435631
45.306120
weight
157.500553
177.340407
171.524640
144.251986
152.526206
168.272968
136.431469
176.499753
174.094104
177.540920
In [22]: gb = df.groupby('gender')
In [23]: gb.
gb.agg
gb.boxplot
gb.aggregate gb.count
gb.apply
gb.cummax
gb.cummin
gb.cumprod
gb.cumsum
gb.describe
gb.dtype
gb.fillna
gb.filter
gb.first
gb.gender
gb.get_group
gb.groups
gb.head
gb.height
gb.hist
gb.indices
17.1.2 GroupBy with MultiIndex
With hierarchically-indexed data, it’s quite natural to group by one of the levels of the hierarchy.
In [24]: s
Out[24]:
first second
486
Chapter 17. Group By: split-apply-combine
gb.
gb.
gb.
pandas: powerful Python data analysis toolkit, Release 0.16.1
bar
one
two
baz
one
two
foo
one
two
qux
one
two
dtype: float64
-0.575247
0.254161
-1.143704
0.215897
1.193555
-0.077118
-0.408530
-0.862495
In [25]: grouped = s.groupby(level=0)
In [26]: grouped.sum()
Out[26]:
first
bar
-0.321085
baz
-0.927807
foo
1.116437
qux
-1.271025
dtype: float64
If the MultiIndex has names specified, these can be passed instead of the level number:
In [27]: s.groupby(level='second').sum()
Out[27]:
second
one
-0.933926
two
-0.469555
dtype: float64
The aggregation functions such as sum will take the level parameter directly. Additionally, the resulting index will be
named according to the chosen level:
In [28]: s.sum(level='second')
Out[28]:
second
one
-0.933926
two
-0.469555
dtype: float64
Also as of v0.6, grouping with multiple levels is supported.
In [29]: s
Out[29]:
first second
bar
doo
baz
bee
foo
bop
qux
bop
third
one
two
one
two
one
two
one
two
1.346061
1.511763
1.627081
-0.990582
-0.441652
1.211526
0.268520
0.024580
dtype: float64
In [30]: s.groupby(level=['first','second']).sum()
Out[30]:
first second
bar
doo
2.857824
17.1. Splitting an object into groups
487
pandas: powerful Python data analysis toolkit, Release 0.16.1
baz
foo
qux
dtype:
bee
bop
bop
float64
0.636499
0.769873
0.293100
More on the sum function and aggregation later.
17.1.3 DataFrame column selection in GroupBy
Once you have created the GroupBy object from a DataFrame, for example, you might want to do something different
for each of the columns. Thus, using [] similar to getting a column from a DataFrame, you can do:
In [31]: grouped = df.groupby(['A'])
In [32]: grouped_C = grouped['C']
In [33]: grouped_D = grouped['D']
This is mainly syntactic sugar for the alternative and much more verbose:
In [34]: df['C'].groupby(df['A'])
Out[34]:
Additionally this method avoids recomputing the internal grouping information derived from the passed key.
17.2 Iterating through groups
With the GroupBy object in hand, iterating through the grouped data is very natural and functions similarly to
itertools.groupby:
In [35]: grouped = df.groupby('A')
In [36]: for name, group in grouped:
....:
print(name)
....:
print(group)
....:
bar
A
B
C
D
1 bar
one -0.042379 -0.089329
3 bar three -0.009920 -0.945867
5 bar
two 0.495767 1.956030
foo
A
B
C
D
0 foo
one -0.919854 -1.131345
2 foo
two 1.247642 0.337863
4 foo
two 0.290213 -0.932132
6 foo
one 0.362949 0.017587
7 foo three 1.548106 -0.016692
In the case of grouping by multiple keys, the group name will be a tuple:
In [37]: for name, group in df.groupby(['A', 'B']):
....:
print(name)
....:
print(group)
....:
('bar', 'one')
488
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
A
1 bar
('bar',
A
3 bar
('bar',
A
5 bar
('foo',
A
0 foo
6 foo
('foo',
A
7 foo
('foo',
A
2 foo
4 foo
B
C
D
one -0.042379 -0.089329
'three')
B
C
D
three -0.00992 -0.945867
'two')
B
C
D
two 0.495767 1.95603
'one')
B
C
D
one -0.919854 -1.131345
one 0.362949 0.017587
'three')
B
C
D
three 1.548106 -0.016692
'two')
B
C
D
two 1.247642 0.337863
two 0.290213 -0.932132
It’s standard Python-fu but remember you can unpack the tuple in the for loop statement if you wish: for (k1,
k2), group in grouped:.
17.3 Selecting a group
A single group can be selected using GroupBy.get_group():
In [38]: grouped.get_group('bar')
Out[38]:
A
B
C
D
1 bar
one -0.042379 -0.089329
3 bar three -0.009920 -0.945867
5 bar
two 0.495767 1.956030
Or for an object grouped on multiple columns:
In [39]: df.groupby(['A', 'B']).get_group(('bar', 'one'))
Out[39]:
A
B
C
D
1 bar one -0.042379 -0.089329
17.4 Aggregation
Once the GroupBy object has been created, several methods are available to perform a computation on the grouped
data.
An obvious one is aggregation via the aggregate or equivalently agg method:
In [40]: grouped = df.groupby('A')
In [41]: grouped.aggregate(np.sum)
Out[41]:
C
D
A
bar 0.443469 0.920834
17.3. Selecting a group
489
pandas: powerful Python data analysis toolkit, Release 0.16.1
foo
2.529056 -1.724719
In [42]: grouped = df.groupby(['A', 'B'])
In [43]: grouped.aggregate(np.sum)
Out[43]:
C
D
A
B
bar one
-0.042379 -0.089329
three -0.009920 -0.945867
two
0.495767 1.956030
foo one
-0.556905 -1.113758
three 1.548106 -0.016692
two
1.537855 -0.594269
As you can see, the result of the aggregation will have the group names as the new index along the grouped axis. In
the case of multiple keys, the result is a MultiIndex by default, though this can be changed by using the as_index
option:
In [44]: grouped = df.groupby(['A', 'B'], as_index=False)
In [45]: grouped.aggregate(np.sum)
Out[45]:
A
B
C
D
0 bar
one -0.042379 -0.089329
1 bar three -0.009920 -0.945867
2 bar
two 0.495767 1.956030
3 foo
one -0.556905 -1.113758
4 foo three 1.548106 -0.016692
5 foo
two 1.537855 -0.594269
In [46]: df.groupby('A', as_index=False).sum()
Out[46]:
A
C
D
0 bar 0.443469 0.920834
1 foo 2.529056 -1.724719
Note that you could use the reset_index DataFrame function to achieve the same result as the column names are
stored in the resulting MultiIndex:
In [47]: df.groupby(['A', 'B']).sum().reset_index()
Out[47]:
A
B
C
D
0 bar
one -0.042379 -0.089329
1 bar three -0.009920 -0.945867
2 bar
two 0.495767 1.956030
3 foo
one -0.556905 -1.113758
4 foo three 1.548106 -0.016692
5 foo
two 1.537855 -0.594269
Another simple aggregation example is to compute the size of each group. This is included in GroupBy as the size
method. It returns a Series whose index are the group names and whose values are the sizes of each group.
In [48]: grouped.size()
Out[48]:
A
B
bar one
1
three
1
two
1
490
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
foo
one
three
two
dtype: int64
2
1
2
In [49]: grouped.describe()
Out[49]:
C
D
0 count 1.000000 1.000000
mean -0.042379 -0.089329
std
NaN
NaN
min
-0.042379 -0.089329
25%
-0.042379 -0.089329
50%
-0.042379 -0.089329
75%
-0.042379 -0.089329
...
...
...
5 mean
0.768928 -0.297134
std
0.677005 0.898022
min
0.290213 -0.932132
25%
0.529570 -0.614633
50%
0.768928 -0.297134
75%
1.008285 0.020364
max
1.247642 0.337863
[48 rows x 2 columns]
Note: Aggregation functions will not return the groups that you are aggregating over if they are named columns,
when as_index=True, the default. The grouped columns will be the indices of the returned object.
Passing as_index=False will return the groups that you are aggregating over, if they are named columns.
Aggregating functions are ones that reduce the dimension of the returned objects, for example: mean, sum, size,
count, std, var, sem, describe, first, last, nth, min, max. This is what happens when
you do for example DataFrame.sum() and get back a Series.
nth can act as a reducer or a filter, see here
17.4.1 Applying multiple functions at once
With grouped Series you can also pass a list or dict of functions to do aggregation with, outputting a DataFrame:
In [50]: grouped = df.groupby('A')
In [51]: grouped['C'].agg([np.sum, np.mean, np.std])
Out[51]:
sum
mean
std
A
bar 0.443469 0.147823 0.301765
foo 2.529056 0.505811 0.966450
If a dict is passed, the keys will be used to name the columns. Otherwise the function’s name (stored in the function
object) will be used.
In [52]: grouped['D'].agg({'result1' : np.sum,
....:
'result2' : np.mean})
....:
Out[52]:
17.4. Aggregation
491
pandas: powerful Python data analysis toolkit, Release 0.16.1
result2
result1
A
bar 0.306945 0.920834
foo -0.344944 -1.724719
On a grouped DataFrame, you can pass a list of functions to apply to each column, which produces an aggregated
result with a hierarchical index:
In [53]: grouped.agg([np.sum, np.mean, np.std])
Out[53]:
C
D
sum
mean
std
sum
mean
A
bar 0.443469 0.147823 0.301765 0.920834 0.306945
foo 2.529056 0.505811 0.966450 -1.724719 -0.344944
std
1.490982
0.645875
Passing a dict of functions has different behavior by default, see the next section.
17.4.2 Applying different functions to DataFrame columns
By passing a dict to aggregate you can apply a different aggregation to the columns of a DataFrame:
In [54]: grouped.agg({'C' : np.sum,
....:
'D' : lambda x: np.std(x, ddof=1)})
....:
Out[54]:
C
D
A
bar 0.443469 1.490982
foo 2.529056 0.645875
The function names can also be strings. In order for a string to be valid it must be either implemented on GroupBy or
available via dispatching:
In [55]: grouped.agg({'C' : 'sum', 'D' : 'std'})
Out[55]:
C
D
A
bar 0.443469 1.490982
foo 2.529056 0.645875
17.4.3 Cython-optimized aggregation functions
Some common aggregations, currently only sum, mean, std, and sem, have optimized Cython implementations:
In [56]: df.groupby('A').sum()
Out[56]:
C
D
A
bar 0.443469 0.920834
foo 2.529056 -1.724719
In [57]: df.groupby(['A', 'B']).mean()
Out[57]:
C
D
A
B
bar one
-0.042379 -0.089329
492
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
three -0.009920 -0.945867
two
0.495767 1.956030
foo one
-0.278452 -0.556879
three 1.548106 -0.016692
two
0.768928 -0.297134
Of course sum and mean are implemented on pandas objects, so the above code would work even without the special
versions via dispatching (see below).
17.5 Transformation
The transform method returns an object that is indexed the same (same size) as the one being grouped. Thus, the
passed transform function should return a result that is the same size as the group chunk. For example, suppose we
wished to standardize the data within each group:
In [58]: index = date_range('10/1/1999', periods=1100)
In [59]: ts = Series(np.random.normal(0.5, 2, 1100), index)
In [60]: ts = rolling_mean(ts, 100, 100).dropna()
In [61]: ts.head()
Out[61]:
2000-01-08
0.779333
2000-01-09
0.778852
2000-01-10
0.786476
2000-01-11
0.782797
2000-01-12
0.798110
Freq: D, dtype: float64
In [62]: ts.tail()
Out[62]:
2002-09-30
0.660294
2002-10-01
0.631095
2002-10-02
0.673601
2002-10-03
0.709213
2002-10-04
0.719369
Freq: D, dtype: float64
In [63]: key = lambda x: x.year
In [64]: zscore = lambda x: (x - x.mean()) / x.std()
In [65]: transformed = ts.groupby(key).transform(zscore)
We would expect the result to now have mean 0 and standard deviation 1 within each group, which we can easily
check:
# Original Data
In [66]: grouped = ts.groupby(key)
In [67]: grouped.mean()
Out[67]:
2000
0.442441
2001
0.526246
2002
0.459365
17.5. Transformation
493
pandas: powerful Python data analysis toolkit, Release 0.16.1
dtype: float64
In [68]: grouped.std()
Out[68]:
2000
0.131752
2001
0.210945
2002
0.128753
dtype: float64
# Transformed Data
In [69]: grouped_trans = transformed.groupby(key)
In [70]: grouped_trans.mean()
Out[70]:
2000
-1.250934e-16
2001
-4.291848e-16
2002
2.404815e-17
dtype: float64
In [71]: grouped_trans.std()
Out[71]:
2000
1
2001
1
2002
1
dtype: float64
We can also visually compare the original and transformed data sets.
In [72]: compare = DataFrame({'Original': ts, 'Transformed': transformed})
In [73]: compare.plot()
Out[73]:
494
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
Another common data transform is to replace missing data with the group mean.
In [74]: data_df
Out[74]:
A
B
0
1.539708 -1.166480
1
1.302092 -0.505754
2
-0.371983 1.104803
3
-1.309622 1.118697
4
-1.924296 0.396437
5
0.815643 0.367816
6
-0.030651 1.376106
..
...
...
993 0.012359 0.554602
994 0.042312 -1.628835
995 -0.093110 0.683847
996 -0.185043 1.438572
997 -0.394469 -0.642343
998 -1.174126 1.857148
999 0.234564 0.517098
C
0.533026
NaN
-0.651520
-1.161657
0.812436
-0.469478
-0.645129
...
-1.976159
1.013822
-0.774753
NaN
0.011374
NaN
0.393534
[1000 rows x 3 columns]
17.5. Transformation
495
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [75]: countries = np.array(['US', 'UK', 'GR', 'JP'])
In [76]: key = countries[np.random.randint(0, 4, 1000)]
In [77]: grouped = data_df.groupby(key)
# Non-NA
In [78]:
Out[78]:
A
GR 209
JP 240
UK 216
US 239
count in each group
grouped.count()
B
217
255
231
250
C
189
217
193
217
In [79]: f = lambda x: x.fillna(x.mean())
In [80]: transformed = grouped.transform(f)
We can verify that the group means have not changed in the transformed data and that the transformed data contains
no NAs.
In [81]: grouped_trans = transformed.groupby(key)
In [82]: grouped.mean() # original group means
Out[82]:
A
B
C
GR -0.098371 -0.015420 0.068053
JP 0.069025 0.023100 -0.077324
UK 0.034069 -0.052580 -0.116525
US 0.058664 -0.020399 0.028603
In [83]: grouped_trans.mean() # transformation did not change group means
Out[83]:
A
B
C
GR -0.098371 -0.015420 0.068053
JP 0.069025 0.023100 -0.077324
UK 0.034069 -0.052580 -0.116525
US 0.058664 -0.020399 0.028603
In [84]:
Out[84]:
A
GR 209
JP 240
UK 216
US 239
grouped.count() # original has some missing data points
In [85]:
Out[85]:
A
GR 228
JP 267
UK 247
US 258
grouped_trans.count() # counts after transformation
B
217
255
231
250
B
228
267
247
258
C
189
217
193
217
C
228
267
247
258
In [86]: grouped_trans.size() # Verify non-NA count equals group size
Out[86]:
496
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
GR
228
JP
267
UK
247
US
258
dtype: int64
Note: Some functions when applied to a groupby object will automatically transform the input, returning an object
of the same shape as the original. Passing as_index=False will not affect these transformation methods.
For example: fillna, ffill, bfill, shift.
In [87]: grouped.ffill()
Out[87]:
A
B
C
0
1.539708 -1.166480 0.533026
1
1.302092 -0.505754 0.533026
2
-0.371983 1.104803 -0.651520
3
-1.309622 1.118697 -1.161657
4
-1.924296 0.396437 0.812436
5
0.815643 0.367816 -0.469478
6
-0.030651 1.376106 -0.645129
..
...
...
...
993 0.012359 0.554602 -1.976159
994 0.042312 -1.628835 1.013822
995 -0.093110 0.683847 -0.774753
996 -0.185043 1.438572 -0.774753
997 -0.394469 -0.642343 0.011374
998 -1.174126 1.857148 -0.774753
999 0.234564 0.517098 0.393534
[1000 rows x 3 columns]
17.6 Filtration
New in version 0.12.
The filter method returns a subset of the original object. Suppose we want to take only elements that belong to
groups with a group sum greater than 2.
In [88]: sf = Series([1, 1, 2, 3, 3, 3])
In [89]: sf.groupby(sf).filter(lambda x: x.sum() > 2)
Out[89]:
3
3
4
3
5
3
dtype: int64
The argument of filter must be a function that, applied to the group as a whole, returns True or False.
Another useful operation is filtering out elements that belong to groups with only a couple members.
In [90]: dff = DataFrame({'A': np.arange(8), 'B': list('aabbbbcc')})
In [91]: dff.groupby('B').filter(lambda x: len(x) > 2)
Out[91]:
17.6. Filtration
497
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
3
4
5
A
2
3
4
5
B
b
b
b
b
Alternatively, instead of dropping the offending groups, we can return a like-indexed objects where the groups that do
not pass the filter are filled with NaNs.
In [92]: dff.groupby('B').filter(lambda x: len(x) > 2, dropna=False)
Out[92]:
A
B
0 NaN NaN
1 NaN NaN
2
2
b
3
3
b
4
4
b
5
5
b
6 NaN NaN
7 NaN NaN
For dataframes with multiple columns, filters should explicitly specify a column as the filter criterion.
In [93]: dff['C'] = np.arange(8)
In [94]:
Out[94]:
A B
2 2 b
3 3 b
4 4 b
5 5 b
dff.groupby('B').filter(lambda x: len(x['C']) > 2)
C
2
3
4
5
Note: Some functions when applied to a groupby object will act as a filter on the input, returning a reduced shape
of the original (and potentitally eliminating groups), but with the index unchanged. Passing as_index=False will
not affect these transformation methods.
For example: head, tail.
In [95]:
Out[95]:
A B
0 0 a
1 1 a
2 2 b
3 3 b
6 6 c
7 7 c
dff.groupby('B').head(2)
C
0
1
2
3
6
7
17.7 Dispatching to instance methods
When doing an aggregation or transformation, you might just want to call an instance method on each data group.
This is pretty easy to do by passing lambda functions:
In [96]: grouped = df.groupby('A')
498
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [97]: grouped.agg(lambda x: x.std())
Out[97]:
C
D
A
bar 0.301765 1.490982
foo 0.966450 0.645875
But, it’s rather verbose and can be untidy if you need to pass additional arguments. Using a bit of metaprogramming
cleverness, GroupBy now has the ability to “dispatch” method calls to the groups:
In [98]: grouped.std()
Out[98]:
C
D
A
bar 0.301765 1.490982
foo 0.966450 0.645875
What is actually happening here is that a function wrapper is being generated. When invoked, it takes any passed
arguments and invokes the function with any arguments on each group (in the above example, the std function). The
results are then combined together much in the style of agg and transform (it actually uses apply to infer the
gluing, documented next). This enables some operations to be carried out rather succinctly:
In [99]: tsdf = DataFrame(randn(1000, 3),
....:
index=date_range('1/1/2000', periods=1000),
....:
columns=['A', 'B', 'C'])
....:
In [100]: tsdf.ix[::2] = np.nan
In [101]: grouped = tsdf.groupby(lambda x: x.year)
In [102]: grouped.fillna(method='pad')
Out[102]:
A
B
C
2000-01-01
NaN
NaN
NaN
2000-01-02 -0.353501 -0.080957 -0.876864
2000-01-03 -0.353501 -0.080957 -0.876864
2000-01-04 0.050976 0.044273 -0.559849
2000-01-05 0.050976 0.044273 -0.559849
2000-01-06 0.030091 0.186460 -0.680149
2000-01-07 0.030091 0.186460 -0.680149
...
...
...
...
2002-09-20 2.310215 0.157482 -0.064476
2002-09-21 2.310215 0.157482 -0.064476
2002-09-22 0.005011 0.053897 -1.026922
2002-09-23 0.005011 0.053897 -1.026922
2002-09-24 -0.456542 -1.849051 1.559856
2002-09-25 -0.456542 -1.849051 1.559856
2002-09-26 1.123162 0.354660 1.128135
[1000 rows x 3 columns]
In this example, we chopped the collection of time series into yearly chunks then independently called fillna on the
groups.
New in version 0.14.1.
The nlargest and nsmallest methods work on Series style groupbys:
17.7. Dispatching to instance methods
499
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [103]: s = Series([9, 8, 7, 5, 19, 1, 4.2, 3.3])
In [104]: g = Series(list('abababab'))
In [105]: gb = s.groupby(g)
In [106]: gb.nlargest(3)
Out[106]:
a 4
19.0
0
9.0
2
7.0
b 1
8.0
3
5.0
7
3.3
dtype: float64
In [107]: gb.nsmallest(3)
Out[107]:
a 6
4.2
2
7.0
0
9.0
b 5
1.0
7
3.3
3
5.0
dtype: float64
17.8 Flexible apply
Some operations on the grouped data might not fit into either the aggregate or transform categories. Or, you may simply
want GroupBy to infer how to combine the results. For these, use the apply function, which can be substituted for
both aggregate and transform in many standard use cases. However, apply can handle some exceptional use
cases, for example:
In [108]: df
Out[108]:
A
B
C
D
0 foo
one -0.919854 -1.131345
1 bar
one -0.042379 -0.089329
2 foo
two 1.247642 0.337863
3 bar three -0.009920 -0.945867
4 foo
two 0.290213 -0.932132
5 bar
two 0.495767 1.956030
6 foo
one 0.362949 0.017587
7 foo three 1.548106 -0.016692
In [109]: grouped = df.groupby('A')
# could also just call .describe()
In [110]: grouped['C'].apply(lambda x: x.describe())
Out[110]:
A
bar count
3.000000
mean
0.147823
std
0.301765
min
-0.042379
25%
-0.026149
500
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
50%
75%
-0.009920
0.242924
...
foo mean
0.505811
std
0.966450
min
-0.919854
25%
0.290213
50%
0.362949
75%
1.247642
max
1.548106
dtype: float64
The dimension of the returned result can also change:
In [111]: grouped = df.groupby('A')['C']
In [112]: def f(group):
.....:
return DataFrame({'original' : group,
.....:
'demeaned' : group - group.mean()})
.....:
In [113]: grouped.apply(f)
Out[113]:
demeaned original
0 -1.425665 -0.919854
1 -0.190202 -0.042379
2 0.741831 1.247642
3 -0.157743 -0.009920
4 -0.215598 0.290213
5 0.347944 0.495767
6 -0.142862 0.362949
7 1.042295 1.548106
apply on a Series can operate on a returned value from the applied function, that is itself a series, and possibly upcast
the result to a DataFrame
In [114]: def f(x):
.....:
return Series([ x, x**2 ], index = ['x', 'x^s'])
.....:
In [115]: s
Out[115]:
0
9.0
1
8.0
2
7.0
3
5.0
4
19.0
5
1.0
6
4.2
7
3.3
dtype: float64
In [116]:
Out[116]:
x
0
9.0
1
8.0
2
7.0
3
5.0
s.apply(f)
x^s
81.00
64.00
49.00
25.00
17.8. Flexible apply
501
pandas: powerful Python data analysis toolkit, Release 0.16.1
4
5
6
7
19.0
1.0
4.2
3.3
361.00
1.00
17.64
10.89
Note: apply can act as a reducer, transformer, or filter function, depending on exactly what is passed to apply. So
depending on the path taken, and exactly what you are grouping. Thus the grouped columns(s) may be included in the
output as well as set the indices.
Warning: In the current implementation apply calls func twice on the first group to decide whether it can take a
fast or slow code path. This can lead to unexpected behavior if func has side-effects, as they will take effect twice
for the first group.
In [117]: d = DataFrame({"a":["x", "y"], "b":[1,2]})
In [118]: def identity(df):
.....:
print df
.....:
return df
.....:
In [119]: d.groupby("a").apply(identity)
a b
0 x 1
a b
0 x 1
a b
1 y 2
Out[119]:
a b
0 x 1
1 y 2
17.9 Other useful features
17.9.1 Automatic exclusion of “nuisance” columns
Again consider the example DataFrame we’ve been looking at:
In [120]: df
Out[120]:
A
B
C
D
0 foo
one -0.919854 -1.131345
1 bar
one -0.042379 -0.089329
2 foo
two 1.247642 0.337863
3 bar three -0.009920 -0.945867
4 foo
two 0.290213 -0.932132
5 bar
two 0.495767 1.956030
6 foo
one 0.362949 0.017587
7 foo three 1.548106 -0.016692
Supposed we wished to compute the standard deviation grouped by the A column. There is a slight problem, namely
that we don’t care about the data in column B. We refer to this as a “nuisance” column. If the passed aggregation
502
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
function can’t be applied to some columns, the troublesome columns will be (silently) dropped. Thus, this does not
pose any problems:
In [121]: df.groupby('A').std()
Out[121]:
C
D
A
bar 0.301765 1.490982
foo 0.966450 0.645875
17.9.2 NA group handling
If there are any NaN values in the grouping key, these will be automatically excluded. So there will never be an “NA
group”. This was not the case in older versions of pandas, but users were generally discarding the NA group anyway
(and supporting it was an implementation headache).
17.9.3 Grouping with ordered factors
Categorical variables represented as instance of pandas’s Categorical class can be used as group keys. If so, the
order of the levels will be preserved:
In [122]: data = Series(np.random.randn(100))
In [123]: factor = qcut(data, [0, .25, .5, .75, 1.])
In [124]: data.groupby(factor).mean()
Out[124]:
[-2.617, -0.684]
-1.331461
(-0.684, -0.0232]
-0.272816
(-0.0232, 0.541]
0.263607
(0.541, 2.369]
1.166038
dtype: float64
17.9.4 Grouping with a Grouper specification
Your may need to specify a bit more data to properly group. You can use the pd.Grouper to provide this local
control.
In [125]: import datetime as DT
In [126]: df = DataFrame({
.....:
'Branch' : 'A A A A A A A B'.split(),
.....:
'Buyer': 'Carl Mark Carl Carl Joe Joe Joe Carl'.split(),
.....:
'Quantity': [1,3,5,1,8,1,9,3],
.....:
'Date' : [
.....:
DT.datetime(2013,1,1,13,0),
.....:
DT.datetime(2013,1,1,13,5),
.....:
DT.datetime(2013,10,1,20,0),
.....:
DT.datetime(2013,10,2,10,0),
.....:
DT.datetime(2013,10,1,20,0),
.....:
DT.datetime(2013,10,2,10,0),
.....:
DT.datetime(2013,12,2,12,0),
.....:
DT.datetime(2013,12,2,14,0),
.....:
]})
17.9. Other useful features
503
pandas: powerful Python data analysis toolkit, Release 0.16.1
.....:
In [127]: df
Out[127]:
Branch Buyer
0
A Carl
1
A Mark
2
A Carl
3
A Carl
4
A
Joe
5
A
Joe
6
A
Joe
7
B Carl
2013-01-01
2013-01-01
2013-10-01
2013-10-02
2013-10-01
2013-10-02
2013-12-02
2013-12-02
Date
13:00:00
13:05:00
20:00:00
10:00:00
20:00:00
10:00:00
12:00:00
14:00:00
Quantity
1
3
5
1
8
1
9
3
Groupby a specific column with the desired frequency. This is like resampling.
In [128]: df.groupby([pd.Grouper(freq='1M',key='Date'),'Buyer']).sum()
Out[128]:
Quantity
Date
Buyer
2013-01-31 Carl
1
Mark
3
2013-10-31 Carl
6
Joe
9
2013-12-31 Carl
3
Joe
9
You have an ambiguous specification in that you have a named index and a column that could be potential groupers.
In [129]: df = df.set_index('Date')
In [130]: df['Date'] = df.index + pd.offsets.MonthEnd(2)
In [131]: df.groupby([pd.Grouper(freq='6M',key='Date'),'Buyer']).sum()
Out[131]:
Quantity
Date
Buyer
2013-02-28 Carl
1
Mark
3
2014-02-28 Carl
9
Joe
18
In [132]: df.groupby([pd.Grouper(freq='6M',level='Date'),'Buyer']).sum()
Out[132]:
Quantity
Date
Buyer
2013-01-31 Carl
1
Mark
3
2014-01-31 Carl
9
Joe
18
17.9.5 Taking the first rows of each group
Just like for a DataFrame or Series you can call head and tail on a groupby:
In [133]: df = DataFrame([[1, 2], [1, 4], [5, 6]], columns=['A', 'B'])
In [134]: df
504
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[134]:
A B
0 1 2
1 1 4
2 5 6
In [135]: g = df.groupby('A')
In [136]: g.head(1)
Out[136]:
A B
0 1 2
2 5 6
In [137]: g.tail(1)
Out[137]:
A B
1 1 4
2 5 6
This shows the first or last n rows from each group.
Warning: Before 0.14.0 this was implemented with a fall-through apply, so the result would incorrectly respect
the as_index flag:
>>> g.head(1):
A B
A
1 0 1 2
5 2 5 6
# was equivalent to g.apply(lambda x: x.head(1))
17.9.6 Taking the nth row of each group
To select from a DataFrame or Series the nth item, use the nth method. This is a reduction method, and will return a
single row (or no row) per group if you pass an int for n:
In [138]: df = DataFrame([[1, np.nan], [1, 4], [5, 6]], columns=['A', 'B'])
In [139]: g = df.groupby('A')
In [140]: g.nth(0)
Out[140]:
B
A
1 NaN
5
6
In [141]: g.nth(-1)
Out[141]:
B
A
1 4
5 6
In [142]: g.nth(1)
Out[142]:
17.9. Other useful features
505
pandas: powerful Python data analysis toolkit, Release 0.16.1
B
A
1
4
If you want to select the nth not-null item, use the dropna kwarg. For a DataFrame this should be either ’any’ or
’all’ just like you would pass to dropna, for a Series this just needs to be truthy.
# nth(0) is the same as g.first()
In [143]: g.nth(0, dropna='any')
Out[143]:
B
A
1 4
5 6
In [144]: g.first()
Out[144]:
B
A
1 4
5 6
# nth(-1) is the same as g.last()
In [145]: g.nth(-1, dropna='any')
Out[145]:
B
A
1 4
5 6
# NaNs denote group exhausted when using dropna
In [146]: g.last()
Out[146]:
B
A
1 4
5 6
In [147]: g.B.nth(0, dropna=True)
Out[147]:
A
1
4
5
6
Name: B, dtype: float64
As with other methods, passing as_index=False, will achieve a filtration, which returns the grouped row.
In [148]: df = DataFrame([[1, np.nan], [1, 4], [5, 6]], columns=['A', 'B'])
In [149]: g = df.groupby('A',as_index=False)
In [150]: g.nth(0)
Out[150]:
A
B
0 1 NaN
2 5
6
In [151]: g.nth(-1)
Out[151]:
A B
506
Chapter 17. Group By: split-apply-combine
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
2
1
5
4
6
You can also select multiple rows from each group by specifying multiple nth values as a list of ints.
In [152]: business_dates = date_range(start='4/1/2014', end='6/30/2014', freq='B')
In [153]: df = DataFrame(1, index=business_dates, columns=['a', 'b'])
# get the first, 4th, and last date index for each month
In [154]: df.groupby((df.index.year, df.index.month)).nth([0, 3, -1])
Out[154]:
a b
2014-04-01 1 1
2014-04-04 1 1
2014-04-30 1 1
2014-05-01 1 1
2014-05-06 1 1
2014-05-30 1 1
2014-06-02 1 1
2014-06-05 1 1
2014-06-30 1 1
17.9.7 Enumerate group items
New in version 0.13.0.
To see the order in which each row appears within its group, use the cumcount method:
In [155]: df = pd.DataFrame(list('aaabba'), columns=['A'])
In [156]: df
Out[156]:
A
0 a
1 a
2 a
3 b
4 b
5 a
In [157]: df.groupby('A').cumcount()
Out[157]:
0
0
1
1
2
2
3
0
4
1
5
3
dtype: int64
In [158]: df.groupby('A').cumcount(ascending=False)
Out[158]:
0
3
1
2
2
1
3
1
4
0
17.9. Other useful features
# kwarg only
507
pandas: powerful Python data analysis toolkit, Release 0.16.1
5
0
dtype: int64
17.9.8 Plotting
Groupby also works with some plotting methods. For example, suppose we suspect that some features in a DataFrame
my differ by group, in this case, the values in column 1 where the group is “B” are 3 higher on average.
In [159]: np.random.seed(1234)
In [160]: df = DataFrame(np.random.randn(50, 2))
In [161]: df['g'] = np.random.choice(['A', 'B'], size=50)
In [162]: df.loc[df['g'] == 'B', 1] += 3
We can easily visualize this with a boxplot:
In [163]: df.groupby('g').boxplot()
Out[163]: OrderedDict([('A', {'boxes': [,
# 2014-08-01 is Friday
In [116]: Timestamp('2014-08-01 10:00').weekday()
Out[116]: 4
In [117]: Timestamp('2014-08-01 10:00') + bh
Out[117]: Timestamp('2014-08-01 11:00:00')
# Below example is the same as Timestamp('2014-08-01 09:00') + bh
In [118]: Timestamp('2014-08-01 08:00') + bh
Out[118]: Timestamp('2014-08-01 10:00:00')
# If the results is on the end time, move to the next business day
In [119]: Timestamp('2014-08-01 16:00') + bh
Out[119]: Timestamp('2014-08-04 09:00:00')
# Remainings are added to the next day
In [120]: Timestamp('2014-08-01 16:30') + bh
Out[120]: Timestamp('2014-08-04 09:30:00')
# Adding 2 business hours
In [121]: Timestamp('2014-08-01 10:00') + BusinessHour(2)
Out[121]: Timestamp('2014-08-01 12:00:00')
# Subtracting 3 business hours
In [122]: Timestamp('2014-08-01 10:00') + BusinessHour(-3)
Out[122]: Timestamp('2014-07-31 15:00:00')
Also, you can specify start and end time by keywords. Argument must be str which has hour:minute representation or datetime.time instance. Specifying seconds, microseconds and nanoseconds as business hour results
in ValueError.
20.5. DateOffset objects
567
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [123]: bh = BusinessHour(start='11:00', end=time(20, 0))
In [124]: bh
Out[124]:
In [125]: Timestamp('2014-08-01 13:00') + bh
Out[125]: Timestamp('2014-08-01 14:00:00')
In [126]: Timestamp('2014-08-01 09:00') + bh
Out[126]: Timestamp('2014-08-01 12:00:00')
In [127]: Timestamp('2014-08-01 18:00') + bh
Out[127]: Timestamp('2014-08-01 19:00:00')
Passing start time later than end represents midnight business hour. In this case, business hour exceeds midnight
and overlap to the next day. Valid business hours are distinguished by whether it started from valid BusinessDay.
In [128]: bh = BusinessHour(start='17:00', end='09:00')
In [129]: bh
Out[129]:
In [130]: Timestamp('2014-08-01 17:00') + bh
Out[130]: Timestamp('2014-08-01 18:00:00')
In [131]: Timestamp('2014-08-01 23:00') + bh
Out[131]: Timestamp('2014-08-02 00:00:00')
# Although 2014-08-02 is Satuaday,
# it is valid because it starts from 08-01 (Friday).
In [132]: Timestamp('2014-08-02 04:00') + bh
Out[132]: Timestamp('2014-08-02 05:00:00')
# Although 2014-08-04 is Monday,
# it is out of business hours because it starts from 08-03 (Sunday).
In [133]: Timestamp('2014-08-04 04:00') + bh
Out[133]: Timestamp('2014-08-04 18:00:00')
Applying BusinessHour.rollforward and rollback to out of business hours results in the next business
hour start or previous day’s end. Different from other offsets, BusinessHour.rollforward may output different
results from apply by definition.
This is because one day’s business hour end is equal to next day’s business hour start. For example, under the default business hours (9:00 - 17:00), there is no gap (0 minutes) between 2014-08-01 17:00 and 2014-08-04
09:00.
# This adjusts a Timestamp to business hour edge
In [134]: BusinessHour().rollback(Timestamp('2014-08-02 15:00'))
Out[134]: Timestamp('2014-08-01 17:00:00')
In [135]: BusinessHour().rollforward(Timestamp('2014-08-02 15:00'))
Out[135]: Timestamp('2014-08-04 09:00:00')
# It is the same as BusinessHour().apply(Timestamp('2014-08-01 17:00')).
# And it is the same as BusinessHour().apply(Timestamp('2014-08-04 09:00'))
In [136]: BusinessHour().apply(Timestamp('2014-08-02 15:00'))
Out[136]: Timestamp('2014-08-04 10:00:00')
# BusinessDay results (for reference)
568
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [137]: BusinessHour().rollforward(Timestamp('2014-08-02'))
Out[137]: Timestamp('2014-08-04 09:00:00')
# It is the same as BusinessDay().apply(Timestamp('2014-08-01'))
# The result is the same as rollworward because BusinessDay never overlap.
In [138]: BusinessHour().apply(Timestamp('2014-08-02'))
Out[138]: Timestamp('2014-08-04 10:00:00')
20.5.4 Offset Aliases
A number of string aliases are given to useful common time series frequencies. We will refer to these aliases as offset
aliases (referred to as time rules prior to v0.8.0).
Alias
B
C
D
W
M
BM
CBM
MS
BMS
CBMS
Q
BQ
QS
BQS
A
BA
AS
BAS
BH
H
T
S
L
U
N
Description
business day frequency
custom business day frequency (experimental)
calendar day frequency
weekly frequency
month end frequency
business month end frequency
custom business month end frequency
month start frequency
business month start frequency
custom business month start frequency
quarter end frequency
business quarter endfrequency
quarter start frequency
business quarter start frequency
year end frequency
business year end frequency
year start frequency
business year start frequency
business hour frequency
hourly frequency
minutely frequency
secondly frequency
milliseonds
microseconds
nanoseconds
20.5.5 Combining Aliases
As we have seen previously, the alias and the offset instance are fungible in most functions:
In [139]: date_range(start, periods=5, freq='B')
Out[139]:
DatetimeIndex(['2011-01-03', '2011-01-04', '2011-01-05', '2011-01-06',
'2011-01-07'],
dtype='datetime64[ns]', freq='B', tz=None)
In [140]: date_range(start, periods=5, freq=BDay())
Out[140]:
DatetimeIndex(['2011-01-03', '2011-01-04', '2011-01-05', '2011-01-06',
20.5. DateOffset objects
569
pandas: powerful Python data analysis toolkit, Release 0.16.1
'2011-01-07'],
dtype='datetime64[ns]', freq='B', tz=None)
You can combine together day and intraday offsets:
In [141]: date_range(start, periods=10, freq='2h20min')
Out[141]:
DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 02:20:00',
'2011-01-01 04:40:00', '2011-01-01 07:00:00',
'2011-01-01 09:20:00', '2011-01-01 11:40:00',
'2011-01-01 14:00:00', '2011-01-01 16:20:00',
'2011-01-01 18:40:00', '2011-01-01 21:00:00'],
dtype='datetime64[ns]', freq='140T', tz=None)
In [142]: date_range(start, periods=10, freq='1D10U')
Out[142]:
DatetimeIndex([
'2011-01-01 00:00:00', '2011-01-02 00:00:00.000010',
'2011-01-03 00:00:00.000020', '2011-01-04 00:00:00.000030',
'2011-01-05 00:00:00.000040', '2011-01-06 00:00:00.000050',
'2011-01-07 00:00:00.000060', '2011-01-08 00:00:00.000070',
'2011-01-09 00:00:00.000080', '2011-01-10 00:00:00.000090'],
dtype='datetime64[ns]', freq='86400000010U', tz=None)
20.5.6 Anchored Offsets
For some frequencies you can specify an anchoring suffix:
Alias
W-SUN
W-MON
W-TUE
W-WED
W-THU
W-FRI
W-SAT
(B)Q(S)-DEC
(B)Q(S)-JAN
(B)Q(S)-FEB
(B)Q(S)-MAR
(B)Q(S)-APR
(B)Q(S)-MAY
(B)Q(S)-JUN
(B)Q(S)-JUL
(B)Q(S)-AUG
(B)Q(S)-SEP
(B)Q(S)-OCT
(B)Q(S)-NOV
(B)A(S)-DEC
(B)A(S)-JAN
(B)A(S)-FEB
(B)A(S)-MAR
(B)A(S)-APR
(B)A(S)-MAY
(B)A(S)-JUN
570
Description
weekly frequency (sundays). Same as ‘W’
weekly frequency (mondays)
weekly frequency (tuesdays)
weekly frequency (wednesdays)
weekly frequency (thursdays)
weekly frequency (fridays)
weekly frequency (saturdays)
quarterly frequency, year ends in December. Same as ‘Q’
quarterly frequency, year ends in January
quarterly frequency, year ends in February
quarterly frequency, year ends in March
quarterly frequency, year ends in April
quarterly frequency, year ends in May
quarterly frequency, year ends in June
quarterly frequency, year ends in July
quarterly frequency, year ends in August
quarterly frequency, year ends in September
quarterly frequency, year ends in October
quarterly frequency, year ends in November
annual frequency, anchored end of December. Same as ‘A’
annual frequency, anchored end of January
annual frequency, anchored end of February
annual frequency, anchored end of March
annual frequency, anchored end of April
annual frequency, anchored end of May
annual frequency, anchored end of June
Continued on next page
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
Alias
(B)A(S)-JUL
(B)A(S)-AUG
(B)A(S)-SEP
(B)A(S)-OCT
(B)A(S)-NOV
Table 20.1 – continued from previous page
Description
annual frequency, anchored end of July
annual frequency, anchored end of August
annual frequency, anchored end of September
annual frequency, anchored end of October
annual frequency, anchored end of November
These can be used as arguments to date_range, bdate_range, constructors for DatetimeIndex, as well as
various other timeseries-related functions in pandas.
20.5.7 Legacy Aliases
Note that prior to v0.8.0, time rules had a slightly different look. pandas will continue to support the legacy time rules
for the time being but it is strongly recommended that you switch to using the new offset aliases.
Legacy Time Rule
WEEKDAY
EOM
W@MON
W@TUE
W@WED
W@THU
W@FRI
W@SAT
W@SUN
Q@JAN
Q@FEB
Q@MAR
A@JAN
A@FEB
A@MAR
A@APR
A@MAY
A@JUN
A@JUL
A@AUG
A@SEP
A@OCT
A@NOV
A@DEC
min
ms
us
Offset Alias
B
BM
W-MON
W-TUE
W-WED
W-THU
W-FRI
W-SAT
W-SUN
BQ-JAN
BQ-FEB
BQ-MAR
BA-JAN
BA-FEB
BA-MAR
BA-APR
BA-MAY
BA-JUN
BA-JUL
BA-AUG
BA-SEP
BA-OCT
BA-NOV
BA-DEC
T
L
U
As you can see, legacy quarterly and annual frequencies are business quarters and business year ends. Please also note
the legacy time rule for milliseconds ms versus the new offset alias for month start MS. This means that offset alias
parsing is case sensitive.
20.5.8 Holidays / Holiday Calendars
Holidays and calendars provide a simple way to define holiday rules to be used with CustomBusinessDay or
in other analysis that requires a predefined set of holidays. The AbstractHolidayCalendar class provides all
20.5. DateOffset objects
571
pandas: powerful Python data analysis toolkit, Release 0.16.1
the necessary methods to return a list of holidays and only rules need to be defined in a specific holiday calendar
class. Further, start_date and end_date class attributes determine over what date range holidays are generated.
These should be overwritten on the AbstractHolidayCalendar class to have the range apply to all calendar
subclasses. USFederalHolidayCalendar is the only calendar that exists and primarily serves as an example for
developing other calendars.
For holidays that occur on fixed dates (e.g., US Memorial Day or July 4th) an observance rule determines when that
holiday is observed if it falls on a weekend or some other non-observed day. Defined observance rules are:
Rule
nearest_workday
sunday_to_monday
next_monday_or_tuesday
previous_friday
next_monday
Description
move Saturday to Friday and Sunday to Monday
move Sunday to following Monday
move Saturday to Monday and Sunday/Monday to Tuesday
move Saturday and Sunday to previous Friday”
move Saturday and Sunday to following Monday
An example of how holidays and holiday calendars are defined:
In [143]: from pandas.tseries.holiday import Holiday, USMemorialDay,\
.....:
AbstractHolidayCalendar, nearest_workday, MO
.....:
In [144]: class ExampleCalendar(AbstractHolidayCalendar):
.....:
rules = [
.....:
USMemorialDay,
.....:
Holiday('July 4th', month=7, day=4, observance=nearest_workday),
.....:
Holiday('Columbus Day', month=10, day=1,
.....:
offset=DateOffset(weekday=MO(2))), #same as 2*Week(weekday=2)
.....:
]
.....:
In [145]: cal = ExampleCalendar()
In [146]: cal.holidays(datetime(2012, 1, 1), datetime(2012, 12, 31))
Out[146]: DatetimeIndex(['2012-05-28', '2012-07-04', '2012-10-08'], dtype='datetime64[ns]', freq=None
Using this calendar, creating an index or doing offset arithmetic skips weekends and holidays (i.e., Memorial Day/July
4th).
In [147]: DatetimeIndex(start='7/1/2012', end='7/10/2012',
.....:
freq=CDay(calendar=cal)).to_pydatetime()
.....:
Out[147]:
array([datetime.datetime(2012, 7, 2, 0, 0),
datetime.datetime(2012, 7, 3, 0, 0),
datetime.datetime(2012, 7, 5, 0, 0),
datetime.datetime(2012, 7, 6, 0, 0),
datetime.datetime(2012, 7, 9, 0, 0),
datetime.datetime(2012, 7, 10, 0, 0)], dtype=object)
In [148]: offset = CustomBusinessDay(calendar=cal)
In [149]: datetime(2012, 5, 25) + offset
Out[149]: Timestamp('2012-05-29 00:00:00')
In [150]: datetime(2012, 7, 3) + offset
Out[150]: Timestamp('2012-07-05 00:00:00')
In [151]: datetime(2012, 7, 3) + 2 * offset
572
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[151]: Timestamp('2012-07-06 00:00:00')
In [152]: datetime(2012, 7, 6) + offset
Out[152]: Timestamp('2012-07-09 00:00:00')
Ranges are defined by the start_date and end_date class attributes of AbstractHolidayCalendar. The
defaults are below.
In [153]: AbstractHolidayCalendar.start_date
Out[153]: Timestamp('1970-01-01 00:00:00')
In [154]: AbstractHolidayCalendar.end_date
Out[154]: Timestamp('2030-12-31 00:00:00')
These dates can be overwritten by setting the attributes as datetime/Timestamp/string.
In [155]: AbstractHolidayCalendar.start_date = datetime(2012, 1, 1)
In [156]: AbstractHolidayCalendar.end_date = datetime(2012, 12, 31)
In [157]: cal.holidays()
Out[157]: DatetimeIndex(['2012-05-28', '2012-07-04', '2012-10-08'], dtype='datetime64[ns]', freq=None
Every calendar class is accessible by name using the get_calendar function which returns a holiday class instance.
Any imported calendar class will automatically be available by this function. Also, HolidayCalendarFactory
provides an easy interface to create calendars that are combinations of calendars or calendars with additional rules.
In [158]: from pandas.tseries.holiday import get_calendar, HolidayCalendarFactory,\
.....:
USLaborDay
.....:
In [159]: cal = get_calendar('ExampleCalendar')
In [160]:
Out[160]:
[Holiday:
Holiday:
Holiday:
cal.rules
MemorialDay (month=5, day=24, offset=),
July 4th (month=7, day=4, observance=),
Columbus Day (month=10, day=1, offset=)]
In [161]: new_cal = HolidayCalendarFactory('NewExampleCalendar', cal, USLaborDay)
In [162]:
Out[162]:
[Holiday:
Holiday:
Holiday:
Holiday:
new_cal.rules
Labor Day (month=9, day=1, offset=),
Columbus Day (month=10, day=1, offset=),
July 4th (month=7, day=4, observance=),
MemorialDay (month=5, day=24, offset=)]
20.6 Time series-related instance methods
20.6.1 Shifting / lagging
One may want to shift or lag the values in a TimeSeries back and forward in time. The method for this is shift,
which is available on all of the pandas objects.
20.6. Time series-related instance methods
573
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [163]: ts = ts[:5]
In [164]: ts.shift(1)
Out[164]:
2011-01-31
NaN
2011-02-28
-1.281247
2011-03-31
-0.727707
2011-04-29
-0.121306
2011-05-31
-0.097883
Freq: BM, dtype: float64
The shift method accepts an freq argument which can accept a DateOffset class or other timedelta-like object
or also a offset alias:
In [165]: ts.shift(5, freq=datetools.bday)
Out[165]:
2011-02-07
-1.281247
2011-03-07
-0.727707
2011-04-07
-0.121306
2011-05-06
-0.097883
2011-06-07
0.695775
dtype: float64
In [166]: ts.shift(5, freq='BM')
Out[166]:
2011-06-30
-1.281247
2011-07-29
-0.727707
2011-08-31
-0.121306
2011-09-30
-0.097883
2011-10-31
0.695775
Freq: BM, dtype: float64
Rather than changing the alignment of the data and the index, DataFrame and TimeSeries objects also have a
tshift convenience method that changes all the dates in the index by a specified number of offsets:
In [167]: ts.tshift(5, freq='D')
Out[167]:
2011-02-05
-1.281247
2011-03-05
-0.727707
2011-04-05
-0.121306
2011-05-04
-0.097883
2011-06-05
0.695775
dtype: float64
Note that with tshift, the leading entry is no longer NaN because the data is not being realigned.
20.6.2 Frequency conversion
The primary function for changing frequencies is the asfreq function. For a DatetimeIndex, this is basically
just a thin, but convenient wrapper around reindex which generates a date_range and calls reindex.
In [168]: dr = date_range('1/1/2010', periods=3, freq=3 * datetools.bday)
In [169]: ts = Series(randn(3), index=dr)
In [170]: ts
Out[170]:
2010-01-01
-0.659574
574
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
2010-01-06
1.494522
2010-01-11
-0.778425
Freq: 3B, dtype: float64
In [171]: ts.asfreq(BDay())
Out[171]:
2010-01-01
-0.659574
2010-01-04
NaN
2010-01-05
NaN
2010-01-06
1.494522
2010-01-07
NaN
2010-01-08
NaN
2010-01-11
-0.778425
Freq: B, dtype: float64
asfreq provides a further convenience so you can specify an interpolation method for any gaps that may appear after
the frequency conversion
In [172]: ts.asfreq(BDay(), method='pad')
Out[172]:
2010-01-01
-0.659574
2010-01-04
-0.659574
2010-01-05
-0.659574
2010-01-06
1.494522
2010-01-07
1.494522
2010-01-08
1.494522
2010-01-11
-0.778425
Freq: B, dtype: float64
20.6.3 Filling forward / backward
Related to asfreq and reindex is the fillna function documented in the missing data section.
20.6.4 Converting to Python datetimes
DatetimeIndex can be converted to an array of Python native datetime.datetime objects using the
to_pydatetime method.
20.7 Up- and downsampling
With 0.8, pandas introduces simple, powerful, and efficient functionality for performing resampling operations during
frequency conversion (e.g., converting secondly data into 5-minutely data). This is extremely common in, but not
limited to, financial applications.
See some cookbook examples for some advanced strategies
In [173]: rng = date_range('1/1/2012', periods=100, freq='S')
In [174]: ts = Series(randint(0, 500, len(rng)), index=rng)
In [175]: ts.resample('5Min', how='sum')
Out[175]:
2012-01-01
25103
Freq: 5T, dtype: int32
20.7. Up- and downsampling
575
pandas: powerful Python data analysis toolkit, Release 0.16.1
The resample function is very flexible and allows you to specify many different parameters to control the frequency
conversion and resampling operation.
The how parameter can be a function name or numpy array function that takes an array and produces aggregated
values:
In [176]: ts.resample('5Min') # default is mean
Out[176]:
2012-01-01
251.03
Freq: 5T, dtype: float64
In [177]: ts.resample('5Min', how='ohlc')
Out[177]:
open high low close
2012-01-01
308
460
9
205
In [178]: ts.resample('5Min', how=np.max)
Out[178]:
2012-01-01
460
Freq: 5T, dtype: int32
Any function available via dispatching can be given to the how parameter by name, including sum, mean, std, sem,
max, min, median, first, last, ohlc.
For downsampling, closed can be set to ‘left’ or ‘right’ to specify which end of the interval is closed:
In [179]: ts.resample('5Min', closed='right')
Out[179]:
2011-12-31 23:55:00
308.000000
2012-01-01 00:00:00
250.454545
Freq: 5T, dtype: float64
In [180]: ts.resample('5Min', closed='left')
Out[180]:
2012-01-01
251.03
Freq: 5T, dtype: float64
For upsampling, the fill_method and limit parameters can be specified to interpolate over the gaps that are
created:
# from secondly to every 250 milliseconds
In [181]: ts[:2].resample('250L')
Out[181]:
2012-01-01 00:00:00
308
2012-01-01 00:00:00.250000
NaN
2012-01-01 00:00:00.500000
NaN
2012-01-01 00:00:00.750000
NaN
2012-01-01 00:00:01
204
Freq: 250L, dtype: float64
In [182]: ts[:2].resample('250L', fill_method='pad')
Out[182]:
2012-01-01 00:00:00
308
2012-01-01 00:00:00.250000
308
2012-01-01 00:00:00.500000
308
2012-01-01 00:00:00.750000
308
2012-01-01 00:00:01
204
Freq: 250L, dtype: int32
In [183]: ts[:2].resample('250L', fill_method='pad', limit=2)
576
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[183]:
2012-01-01 00:00:00
2012-01-01 00:00:00.250000
2012-01-01 00:00:00.500000
2012-01-01 00:00:00.750000
2012-01-01 00:00:01
Freq: 250L, dtype: float64
308
308
308
NaN
204
Parameters like label and loffset are used to manipulate the resulting labels. label specifies whether the result
is labeled with the beginning or the end of the interval. loffset performs a time adjustment on the output labels.
In [184]: ts.resample('5Min') # by default label='right'
Out[184]:
2012-01-01
251.03
Freq: 5T, dtype: float64
In [185]: ts.resample('5Min', label='left')
Out[185]:
2012-01-01
251.03
Freq: 5T, dtype: float64
In [186]: ts.resample('5Min', label='left', loffset='1s')
Out[186]:
2012-01-01 00:00:01
251.03
dtype: float64
The axis parameter can be set to 0 or 1 and allows you to resample the specified axis for a DataFrame.
kind can be set to ‘timestamp’ or ‘period’ to convert the resulting index to/from time-stamp and time-span representations. By default resample retains the input representation.
convention can be set to ‘start’ or ‘end’ when resampling period data (detail below). It specifies how low frequency
periods are converted to higher frequency periods.
Note that 0.8 marks a watershed in the timeseries functionality in pandas. In previous versions, resampling had to be
done using a combination of date_range, groupby with asof, and then calling an aggregation function on the
grouped object. This was not nearly as convenient or performant as the new pandas timeseries API.
20.8 Time Span Representation
Regular intervals of time are represented by Period objects in pandas while sequences of Period objects are
collected in a PeriodIndex, which can be created with the convenience function period_range.
20.8.1 Period
A Period represents a span of time (e.g., a day, a month, a quarter, etc). It can be created using a frequency alias:
In [187]: Period('2012', freq='A-DEC')
Out[187]: Period('2012', 'A-DEC')
In [188]: Period('2012-1-1', freq='D')
Out[188]: Period('2012-01-01', 'D')
In [189]: Period('2012-1-1 19:00', freq='H')
Out[189]: Period('2012-01-01 19:00', 'H')
20.8. Time Span Representation
577
pandas: powerful Python data analysis toolkit, Release 0.16.1
Unlike time stamped data, pandas does not support frequencies at multiples of DateOffsets (e.g., ‘3Min’) for periods.
Adding and subtracting integers from periods shifts the period by its own frequency.
In [190]: p = Period('2012', freq='A-DEC')
In [191]: p + 1
Out[191]: Period('2013', 'A-DEC')
In [192]: p - 3
Out[192]: Period('2009', 'A-DEC')
If Period freq is daily or higher (D, H, T, S, L, U, N), offsets and timedelta-like can be added if the result can
have the same freq. Otherise, ValueError will be raised.
In [193]: p = Period('2014-07-01 09:00', freq='H')
In [194]: p + Hour(2)
Out[194]: Period('2014-07-01 11:00', 'H')
In [195]: p + timedelta(minutes=120)
Out[195]: Period('2014-07-01 11:00', 'H')
In [196]: p + np.timedelta64(7200, 's')
Out[196]: Period('2014-07-01 11:00', 'H')
In [1]: p + Minute(5)
Traceback
...
ValueError: Input has different freq from Period(freq=H)
If Period has other freqs, only the same offsets can be added. Otherwise, ValueError will be raised.
In [197]: p = Period('2014-07', freq='M')
In [198]: p + MonthEnd(3)
Out[198]: Period('2014-10', 'M')
In [1]: p + MonthBegin(3)
Traceback
...
ValueError: Input has different freq from Period(freq=M)
Taking the difference of Period instances with the same frequency will return the number of frequency units between
them:
In [199]: Period('2012', freq='A-DEC') - Period('2002', freq='A-DEC')
Out[199]: 10L
20.8.2 PeriodIndex and period_range
Regular sequences of Period objects can be collected in a PeriodIndex, which can be constructed using the
period_range convenience function:
In [200]: prng = period_range('1/1/2011', '1/1/2012', freq='M')
In [201]: prng
Out[201]:
578
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
PeriodIndex(['2011-01', '2011-02', '2011-03', '2011-04', '2011-05', '2011-06',
'2011-07', '2011-08', '2011-09', '2011-10', '2011-11', '2011-12',
'2012-01'],
dtype='int64', freq='M')
The PeriodIndex constructor can also be used directly:
In [202]: PeriodIndex(['2011-1', '2011-2', '2011-3'], freq='M')
Out[202]: PeriodIndex(['2011-01', '2011-02', '2011-03'], dtype='int64', freq='M')
Just like DatetimeIndex, a PeriodIndex can also be used to index pandas objects:
In [203]: ps = Series(randn(len(prng)), prng)
In [204]: ps
Out[204]:
2011-01
-0.253355
2011-02
-1.426908
2011-03
1.548971
2011-04
-0.088718
2011-05
-1.771348
2011-06
-0.989328
2011-07
-1.584789
2011-08
-0.288786
2011-09
-2.029806
2011-10
-0.761200
2011-11
-1.603608
2011-12
1.756171
2012-01
0.256502
Freq: M, dtype: float64
PeriodIndex supports addition and subtraction with the same rule as Period.
In [205]: idx = period_range('2014-07-01 09:00', periods=5, freq='H')
In [206]: idx
Out[206]:
PeriodIndex(['2014-07-01 09:00', '2014-07-01 10:00', '2014-07-01 11:00',
'2014-07-01 12:00', '2014-07-01 13:00'],
dtype='int64', freq='H')
In [207]: idx + Hour(2)
Out[207]:
PeriodIndex(['2014-07-01 11:00', '2014-07-01 12:00', '2014-07-01 13:00',
'2014-07-01 14:00', '2014-07-01 15:00'],
dtype='int64', freq='H')
In [208]: idx = period_range('2014-07', periods=5, freq='M')
In [209]: idx
Out[209]: PeriodIndex(['2014-07', '2014-08', '2014-09', '2014-10', '2014-11'], dtype='int64', freq='M
In [210]: idx + MonthEnd(3)
Out[210]: PeriodIndex(['2014-10', '2014-11', '2014-12', '2015-01', '2015-02'], dtype='int64', freq='M
20.8. Time Span Representation
579
pandas: powerful Python data analysis toolkit, Release 0.16.1
20.8.3 PeriodIndex Partial String Indexing
You can pass in dates and strings to Series and DataFrame with PeriodIndex, in the same manner as
DatetimeIndex. For details, refer to DatetimeIndex Partial String Indexing.
In [211]: ps['2011-01']
Out[211]: -0.25335528290092818
In [212]: ps[datetime(2011, 12, 25):]
Out[212]:
2011-12
1.756171
2012-01
0.256502
Freq: M, dtype: float64
In [213]: ps['10/31/2011':'12/31/2011']
Out[213]:
2011-10
-0.761200
2011-11
-1.603608
2011-12
1.756171
Freq: M, dtype: float64
Passing a string representing a lower frequency than PeriodIndex returns partial sliced data.
In [214]: ps['2011']
Out[214]:
2011-01
-0.253355
2011-02
-1.426908
2011-03
1.548971
2011-04
-0.088718
2011-05
-1.771348
2011-06
-0.989328
2011-07
-1.584789
2011-08
-0.288786
2011-09
-2.029806
2011-10
-0.761200
2011-11
-1.603608
2011-12
1.756171
Freq: M, dtype: float64
In [215]: dfp = DataFrame(randn(600,1), columns=['A'],
.....:
index=period_range('2013-01-01 9:00', periods=600, freq='T'))
.....:
In [216]: dfp
Out[216]:
2013-01-01
2013-01-01
2013-01-01
2013-01-01
2013-01-01
2013-01-01
2013-01-01
...
2013-01-01
2013-01-01
2013-01-01
2013-01-01
2013-01-01
580
A
09:00 0.020601
09:01 -0.411719
09:02 2.079413
09:03 -1.077911
09:04 0.099258
09:05 -0.089851
09:06 0.711329
...
18:53 -1.340038
18:54 1.315461
18:55 2.396188
18:56 -0.501527
18:57 -3.171938
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
2013-01-01 18:58
2013-01-01 18:59
0.142019
0.606998
[600 rows x 1 columns]
In [217]: dfp['2013-01-01 10H']
Out[217]:
A
2013-01-01 10:00 -0.745396
2013-01-01 10:01 0.141880
2013-01-01 10:02 -1.077754
2013-01-01 10:03 -1.301174
2013-01-01 10:04 -0.269628
2013-01-01 10:05 -0.456347
2013-01-01 10:06 0.157766
...
...
2013-01-01 10:53 0.168057
2013-01-01 10:54 -0.214306
2013-01-01 10:55 -0.069739
2013-01-01 10:56 -1.511809
2013-01-01 10:57 0.307021
2013-01-01 10:58 1.449776
2013-01-01 10:59 0.782537
[60 rows x 1 columns]
As with DatetimeIndex, the endpoints will be included in the result. The example below slices data starting from
10:00 to 11:59.
In [218]: dfp['2013-01-01 10H':'2013-01-01 11H']
Out[218]:
A
2013-01-01 10:00 -0.745396
2013-01-01 10:01 0.141880
2013-01-01 10:02 -1.077754
2013-01-01 10:03 -1.301174
2013-01-01 10:04 -0.269628
2013-01-01 10:05 -0.456347
2013-01-01 10:06 0.157766
...
...
2013-01-01 11:53 -0.064395
2013-01-01 11:54 0.350193
2013-01-01 11:55 1.336433
2013-01-01 11:56 -0.438701
2013-01-01 11:57 -0.915841
2013-01-01 11:58 0.294215
2013-01-01 11:59 0.040959
[120 rows x 1 columns]
20.8.4 Frequency Conversion and Resampling with PeriodIndex
The frequency of Period and PeriodIndex can be converted via the asfreq method. Let’s start with the fiscal
year 2011, ending in December:
In [219]: p = Period('2011', freq='A-DEC')
20.8. Time Span Representation
581
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [220]: p
Out[220]: Period('2011', 'A-DEC')
We can convert it to a monthly frequency. Using the how parameter, we can specify whether to return the starting or
ending month:
In [221]: p.asfreq('M', how='start')
Out[221]: Period('2011-01', 'M')
In [222]: p.asfreq('M', how='end')
Out[222]: Period('2011-12', 'M')
The shorthands ‘s’ and ‘e’ are provided for convenience:
In [223]: p.asfreq('M', 's')
Out[223]: Period('2011-01', 'M')
In [224]: p.asfreq('M', 'e')
Out[224]: Period('2011-12', 'M')
Converting to a “super-period” (e.g., annual frequency is a super-period of quarterly frequency) automatically returns
the super-period that includes the input period:
In [225]: p = Period('2011-12', freq='M')
In [226]: p.asfreq('A-NOV')
Out[226]: Period('2012', 'A-NOV')
Note that since we converted to an annual frequency that ends the year in November, the monthly period of December
2011 is actually in the 2012 A-NOV period. Period conversions with anchored frequencies are particularly useful
for working with various quarterly data common to economics, business, and other fields. Many organizations define
quarters relative to the month in which their fiscal year starts and ends. Thus, first quarter of 2011 could start in 2010
or a few months into 2011. Via anchored frequencies, pandas works for all quarterly frequencies Q-JAN through
Q-DEC.
Q-DEC define regular calendar quarters:
In [227]: p = Period('2012Q1', freq='Q-DEC')
In [228]: p.asfreq('D', 's')
Out[228]: Period('2012-01-01', 'D')
In [229]: p.asfreq('D', 'e')
Out[229]: Period('2012-03-31', 'D')
Q-MAR defines fiscal year end in March:
In [230]: p = Period('2011Q4', freq='Q-MAR')
In [231]: p.asfreq('D', 's')
Out[231]: Period('2011-01-01', 'D')
In [232]: p.asfreq('D', 'e')
Out[232]: Period('2011-03-31', 'D')
582
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
20.9 Converting between Representations
Timestamped data can be converted to PeriodIndex-ed data using to_period and vice-versa using
to_timestamp:
In [233]: rng = date_range('1/1/2012', periods=5, freq='M')
In [234]: ts = Series(randn(len(rng)), index=rng)
In [235]: ts
Out[235]:
2012-01-31
-0.016142
2012-02-29
0.865782
2012-03-31
0.246439
2012-04-30
-1.199736
2012-05-31
0.407620
Freq: M, dtype: float64
In [236]: ps = ts.to_period()
In [237]: ps
Out[237]:
2012-01
-0.016142
2012-02
0.865782
2012-03
0.246439
2012-04
-1.199736
2012-05
0.407620
Freq: M, dtype: float64
In [238]: ps.to_timestamp()
Out[238]:
2012-01-01
-0.016142
2012-02-01
0.865782
2012-03-01
0.246439
2012-04-01
-1.199736
2012-05-01
0.407620
Freq: MS, dtype: float64
Remember that ‘s’ and ‘e’ can be used to return the timestamps at the start or end of the period:
In [239]: ps.to_timestamp('D', how='s')
Out[239]:
2012-01-01
-0.016142
2012-02-01
0.865782
2012-03-01
0.246439
2012-04-01
-1.199736
2012-05-01
0.407620
Freq: MS, dtype: float64
Converting between period and timestamp enables some convenient arithmetic functions to be used. In the following
example, we convert a quarterly frequency with year ending in November to 9am of the end of the month following
the quarter end:
In [240]: prng = period_range('1990Q1', '2000Q4', freq='Q-NOV')
In [241]: ts = Series(randn(len(prng)), prng)
In [242]: ts.index = (prng.asfreq('M', 'e') + 1).asfreq('H', 's') + 9
20.9. Converting between Representations
583
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [243]: ts.head()
Out[243]:
1990-03-01 09:00
-2.470970
1990-06-01 09:00
-0.929915
1990-09-01 09:00
1.385889
1990-12-01 09:00
-1.830966
1991-03-01 09:00
-0.328505
Freq: H, dtype: float64
20.10 Representing out-of-bounds spans
If you have data that is outside of the Timestamp bounds, see Timestamp limitations, then you can use a
PeriodIndex and/or Series of Periods to do computations.
In [244]: span = period_range('1215-01-01', '1381-01-01', freq='D')
In [245]: span
Out[245]:
PeriodIndex(['1215-01-01',
'1215-01-05',
'1215-01-09',
...
'1380-12-23',
'1380-12-27',
'1380-12-31',
dtype='int64',
'1215-01-02', '1215-01-03', '1215-01-04',
'1215-01-06', '1215-01-07', '1215-01-08',
'1215-01-10',
'1380-12-24', '1380-12-25', '1380-12-26',
'1380-12-28', '1380-12-29', '1380-12-30',
'1381-01-01'],
length=60632, freq='D')
To convert from a int64 based YYYYMMDD representation.
In [246]: s = Series([20121231, 20141130, 99991231])
In [247]: s
Out[247]:
0
20121231
1
20141130
2
99991231
dtype: int64
In [248]: def conv(x):
.....:
return Period(year = x // 10000, month = x//100 % 100, day = x%100, freq='D')
.....:
In [249]: s.apply(conv)
Out[249]:
0
2012-12-31
1
2014-11-30
2
9999-12-31
dtype: object
In [250]: s.apply(conv)[2]
Out[250]: Period('9999-12-31', 'D')
These can easily be converted to a PeriodIndex
In [251]: span = PeriodIndex(s.apply(conv))
584
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [252]: span
Out[252]: PeriodIndex(['2012-12-31', '2014-11-30', '9999-12-31'], dtype='int64', freq='D')
20.11 Time Zone Handling
Pandas provides rich support for working with timestamps in different time zones using pytz and dateutil libraries. dateutil support is new in 0.14.1 and currently only supported for fixed offset and tzfile zones. The
default library is pytz. Support for dateutil is provided for compatibility with other applications e.g. if you use
dateutil in other python packages.
20.11.1 Working with Time Zones
By default, pandas objects are time zone unaware:
In [253]: rng = date_range('3/6/2012 00:00', periods=15, freq='D')
In [254]: rng.tz is None
Out[254]: True
To supply the time zone, you can use the tz keyword to date_range and other functions. Dateutil time zone strings
are distinguished from pytz time zones by starting with dateutil/.
• In pytz you can find a list of common (and less common) time zones using from pytz import
common_timezones, all_timezones.
• dateutil uses the OS timezones so there isn’t a fixed list available. For common zones, the names are the
same as pytz.
# pytz
In [255]: rng_pytz = date_range('3/6/2012 00:00', periods=10, freq='D',
.....:
tz='Europe/London')
.....:
In [256]: rng_pytz.tz
Out[256]:
# dateutil
In [257]: rng_dateutil = date_range('3/6/2012 00:00', periods=10, freq='D',
.....:
tz='dateutil/Europe/London')
.....:
In [258]: rng_dateutil.tz
Out[258]: tzfile('Europe/Belfast')
# dateutil - utc special case
In [259]: rng_utc = date_range('3/6/2012 00:00', periods=10, freq='D',
.....:
tz=dateutil.tz.tzutc())
.....:
In [260]: rng_utc.tz
Out[260]: tzutc()
Note that the UTC timezone is a special case in dateutil and should be constructed explicitly as an instance of
dateutil.tz.tzutc. You can also construct other timezones explicitly first, which gives you more control over
which time zone is used:
20.11. Time Zone Handling
585
pandas: powerful Python data analysis toolkit, Release 0.16.1
# pytz
In [261]: tz_pytz = pytz.timezone('Europe/London')
In [262]: rng_pytz = date_range('3/6/2012 00:00', periods=10, freq='D',
.....:
tz=tz_pytz)
.....:
In [263]: rng_pytz.tz == tz_pytz
Out[263]: True
# dateutil
In [264]: tz_dateutil = dateutil.tz.gettz('Europe/London')
In [265]: rng_dateutil = date_range('3/6/2012 00:00', periods=10, freq='D',
.....:
tz=tz_dateutil)
.....:
In [266]: rng_dateutil.tz == tz_dateutil
Out[266]: True
Timestamps, like Python’s datetime.datetime object can be either time zone naive or time zone aware. Naive
time series and DatetimeIndex objects can be localized using tz_localize:
In [267]: ts = Series(randn(len(rng)), rng)
In [268]: ts_utc = ts.tz_localize('UTC')
In [269]: ts_utc
Out[269]:
2012-03-06 00:00:00+00:00
2012-03-07 00:00:00+00:00
2012-03-08 00:00:00+00:00
2012-03-09 00:00:00+00:00
2012-03-10 00:00:00+00:00
2012-03-11 00:00:00+00:00
2012-03-12 00:00:00+00:00
2012-03-13 00:00:00+00:00
2012-03-14 00:00:00+00:00
2012-03-15 00:00:00+00:00
2012-03-16 00:00:00+00:00
2012-03-17 00:00:00+00:00
2012-03-18 00:00:00+00:00
2012-03-19 00:00:00+00:00
2012-03-20 00:00:00+00:00
Freq: D, dtype: float64
0.758606
2.190827
0.706087
1.798831
1.228481
-0.179494
0.634073
0.262123
1.928233
0.322573
-0.711113
1.444272
-0.352268
0.213008
-0.619340
Again, you can explicitly construct the timezone object first. You can use the tz_convert method to convert pandas
objects to convert tz-aware data to another time zone:
In [270]: ts_utc.tz_convert('US/Eastern')
Out[270]:
2012-03-05 19:00:00-05:00
0.758606
2012-03-06 19:00:00-05:00
2.190827
2012-03-07 19:00:00-05:00
0.706087
2012-03-08 19:00:00-05:00
1.798831
2012-03-09 19:00:00-05:00
1.228481
2012-03-10 19:00:00-05:00
-0.179494
2012-03-11 20:00:00-04:00
0.634073
2012-03-12 20:00:00-04:00
0.262123
586
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
2012-03-13 20:00:00-04:00
2012-03-14 20:00:00-04:00
2012-03-15 20:00:00-04:00
2012-03-16 20:00:00-04:00
2012-03-17 20:00:00-04:00
2012-03-18 20:00:00-04:00
2012-03-19 20:00:00-04:00
Freq: D, dtype: float64
1.928233
0.322573
-0.711113
1.444272
-0.352268
0.213008
-0.619340
Warning: Be wary of conversions between libraries. For some zones pytz and dateutil have different definitions of the zone. This is more of a problem for unusual timezones than for ‘standard’ zones like US/Eastern.
Warning: Be aware that a timezone definition across versions of timezone libraries may not be considered equal.
This may cause problems when working with stored data that is localized using one version and operated on with
a different version. See here for how to handle such a situation.
Warning:
It is incorrect to pass a timezone directly into the datetime.datetime constructor (e.g.,
datetime.datetime(2011, 1, 1, tz=timezone(’US/Eastern’)). Instead, the datetime needs
to be localized using the the localize method on the timezone.
Under the hood, all timestamps are stored in UTC. Scalar values from a DatetimeIndex with a time zone will have
their fields (day, hour, minute) localized to the time zone. However, timestamps with the same UTC value are still
considered to be equal even if they are in different time zones:
In [271]: rng_eastern = rng_utc.tz_convert('US/Eastern')
In [272]: rng_berlin = rng_utc.tz_convert('Europe/Berlin')
In [273]: rng_eastern[5]
Out[273]: Timestamp('2012-03-10 19:00:00-0500', tz='US/Eastern', offset='D')
In [274]: rng_berlin[5]
Out[274]: Timestamp('2012-03-11 01:00:00+0100', tz='Europe/Berlin', offset='D')
In [275]: rng_eastern[5] == rng_berlin[5]
Out[275]: True
Like Series, DataFrame, and DatetimeIndex, Timestamps can be converted to other time zones using tz_convert:
In [276]: rng_eastern[5]
Out[276]: Timestamp('2012-03-10 19:00:00-0500', tz='US/Eastern', offset='D')
In [277]: rng_berlin[5]
Out[277]: Timestamp('2012-03-11 01:00:00+0100', tz='Europe/Berlin', offset='D')
In [278]: rng_eastern[5].tz_convert('Europe/Berlin')
Out[278]: Timestamp('2012-03-11 01:00:00+0100', tz='Europe/Berlin')
Localization of Timestamps functions just like DatetimeIndex and TimeSeries:
In [279]: rng[5]
Out[279]: Timestamp('2012-03-11 00:00:00', offset='D')
In [280]: rng[5].tz_localize('Asia/Shanghai')
Out[280]: Timestamp('2012-03-11 00:00:00+0800', tz='Asia/Shanghai')
20.11. Time Zone Handling
587
pandas: powerful Python data analysis toolkit, Release 0.16.1
Operations between TimeSeries in different time zones will yield UTC TimeSeries, aligning the data on the UTC
timestamps:
In [281]: eastern = ts_utc.tz_convert('US/Eastern')
In [282]: berlin = ts_utc.tz_convert('Europe/Berlin')
In [283]: result = eastern + berlin
In [284]: result
Out[284]:
2012-03-06 00:00:00+00:00
2012-03-07 00:00:00+00:00
2012-03-08 00:00:00+00:00
2012-03-09 00:00:00+00:00
2012-03-10 00:00:00+00:00
2012-03-11 00:00:00+00:00
2012-03-12 00:00:00+00:00
2012-03-13 00:00:00+00:00
2012-03-14 00:00:00+00:00
2012-03-15 00:00:00+00:00
2012-03-16 00:00:00+00:00
2012-03-17 00:00:00+00:00
2012-03-18 00:00:00+00:00
2012-03-19 00:00:00+00:00
2012-03-20 00:00:00+00:00
Freq: D, dtype: float64
1.517212
4.381654
1.412174
3.597662
2.456962
-0.358988
1.268146
0.524245
3.856466
0.645146
-1.422226
2.888544
-0.704537
0.426017
-1.238679
In [285]: result.index
Out[285]:
DatetimeIndex(['2012-03-06', '2012-03-07', '2012-03-08', '2012-03-09',
'2012-03-10', '2012-03-11', '2012-03-12', '2012-03-13',
'2012-03-14', '2012-03-15', '2012-03-16', '2012-03-17',
'2012-03-18', '2012-03-19', '2012-03-20'],
dtype='datetime64[ns]', freq='D', tz='UTC')
To remove timezone from tz-aware DatetimeIndex, use tz_localize(None) or tz_convert(None).
tz_localize(None) will remove timezone holding local time representations. tz_convert(None) will remove timezone after converting to UTC time.
In [286]: didx = DatetimeIndex(start='2014-08-01 09:00', freq='H', periods=10, tz='US/Eastern')
In [287]: didx
Out[287]:
DatetimeIndex(['2014-08-01 09:00:00-04:00', '2014-08-01 10:00:00-04:00',
'2014-08-01 11:00:00-04:00', '2014-08-01 12:00:00-04:00',
'2014-08-01 13:00:00-04:00', '2014-08-01 14:00:00-04:00',
'2014-08-01 15:00:00-04:00', '2014-08-01 16:00:00-04:00',
'2014-08-01 17:00:00-04:00', '2014-08-01 18:00:00-04:00'],
dtype='datetime64[ns]', freq='H', tz='US/Eastern')
In [288]: didx.tz_localize(None)
Out[288]:
DatetimeIndex(['2014-08-01 09:00:00',
'2014-08-01 11:00:00',
'2014-08-01 13:00:00',
'2014-08-01 15:00:00',
'2014-08-01 17:00:00',
dtype='datetime64[ns]',
588
'2014-08-01 10:00:00',
'2014-08-01 12:00:00',
'2014-08-01 14:00:00',
'2014-08-01 16:00:00',
'2014-08-01 18:00:00'],
freq='H', tz=None)
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [289]: didx.tz_convert(None)
Out[289]:
DatetimeIndex(['2014-08-01 13:00:00',
'2014-08-01 15:00:00',
'2014-08-01 17:00:00',
'2014-08-01 19:00:00',
'2014-08-01 21:00:00',
dtype='datetime64[ns]',
'2014-08-01 14:00:00',
'2014-08-01 16:00:00',
'2014-08-01 18:00:00',
'2014-08-01 20:00:00',
'2014-08-01 22:00:00'],
freq='H', tz=None)
# tz_convert(None) is identical with tz_convert('UTC').tz_localize(None)
In [290]: didx.tz_convert('UCT').tz_localize(None)
Out[290]:
DatetimeIndex(['2014-08-01 13:00:00', '2014-08-01 14:00:00',
'2014-08-01 15:00:00', '2014-08-01 16:00:00',
'2014-08-01 17:00:00', '2014-08-01 18:00:00',
'2014-08-01 19:00:00', '2014-08-01 20:00:00',
'2014-08-01 21:00:00', '2014-08-01 22:00:00'],
dtype='datetime64[ns]', freq='H', tz=None)
20.11.2 Ambiguous Times when Localizing
In some cases, localize cannot determine the DST and non-DST hours when there are duplicates. This often happens when reading files or database records that simply duplicate the hours. Passing ambiguous=’infer’
(infer_dst argument in prior releases) into tz_localize will attempt to determine the right offset. Below
the top example will fail as it contains ambiguous times and the bottom will infer the right offset.
In [291]: rng_hourly = DatetimeIndex(['11/06/2011 00:00', '11/06/2011 01:00',
.....:
'11/06/2011 01:00', '11/06/2011 02:00',
.....:
'11/06/2011 03:00'])
.....:
# This will fail as there are ambiguous times
In [292]: rng_hourly.tz_localize('US/Eastern')
--------------------------------------------------------------------------AmbiguousTimeError
Traceback (most recent call last)
in ()
----> 1 rng_hourly.tz_localize('US/Eastern')
/home/joris/scipy/pandas/pandas/util/decorators.pyc in wrapper(*args, **kwargs)
86
else:
87
kwargs[new_arg_name] = new_arg_value
---> 88
return func(*args, **kwargs)
89
return wrapper
90
return _deprecate_kwarg
/home/joris/scipy/pandas/pandas/tseries/index.pyc in tz_localize(self, tz, ambiguous)
1644
1645
new_dates = tslib.tz_localize_to_utc(self.asi8, tz,
-> 1646
ambiguous=ambiguous)
1647
new_dates = new_dates.view(_NS_DTYPE)
1648
return self._shallow_copy(new_dates, tz=tz)
/home/joris/scipy/pandas/pandas/tslib.so in pandas.tslib.tz_localize_to_utc (pandas/tslib.c:46871)()
AmbiguousTimeError: Cannot infer dst time from Timestamp('2011-11-06 01:00:00'), try using the 'ambig
20.11. Time Zone Handling
589
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [293]: rng_hourly_eastern = rng_hourly.tz_localize('US/Eastern', ambiguous='infer')
In [294]: rng_hourly_eastern.tolist()
Out[294]:
[Timestamp('2011-11-06 00:00:00-0400',
Timestamp('2011-11-06 01:00:00-0400',
Timestamp('2011-11-06 01:00:00-0500',
Timestamp('2011-11-06 02:00:00-0500',
Timestamp('2011-11-06 03:00:00-0500',
tz='US/Eastern'),
tz='US/Eastern'),
tz='US/Eastern'),
tz='US/Eastern'),
tz='US/Eastern')]
In addition to ‘infer’, there are several other arguments supported. Passing an array-like of bools or 0s/1s where True
represents a DST hour and False a non-DST hour, allows for distinguishing more than one DST transition (e.g., if you
have multiple records in a database each with their own DST transition). Or passing ‘NaT’ will fill in transition times
with not-a-time values. These methods are available in the DatetimeIndex constructor as well as tz_localize.
In [295]: rng_hourly_dst = np.array([1, 1, 0, 0, 0])
In [296]: rng_hourly.tz_localize('US/Eastern', ambiguous=rng_hourly_dst).tolist()
Out[296]:
[Timestamp('2011-11-06 00:00:00-0400', tz='US/Eastern'),
Timestamp('2011-11-06 01:00:00-0400', tz='US/Eastern'),
Timestamp('2011-11-06 01:00:00-0500', tz='US/Eastern'),
Timestamp('2011-11-06 02:00:00-0500', tz='US/Eastern'),
Timestamp('2011-11-06 03:00:00-0500', tz='US/Eastern')]
In [297]: rng_hourly.tz_localize('US/Eastern', ambiguous='NaT').tolist()
Out[297]:
[Timestamp('2011-11-06 00:00:00-0400', tz='US/Eastern'),
NaT,
NaT,
Timestamp('2011-11-06 02:00:00-0500', tz='US/Eastern'),
Timestamp('2011-11-06 03:00:00-0500', tz='US/Eastern')]
In [298]: didx = DatetimeIndex(start='2014-08-01 09:00', freq='H', periods=10, tz='US/Eastern')
In [299]: didx
Out[299]:
DatetimeIndex(['2014-08-01 09:00:00-04:00', '2014-08-01 10:00:00-04:00',
'2014-08-01 11:00:00-04:00', '2014-08-01 12:00:00-04:00',
'2014-08-01 13:00:00-04:00', '2014-08-01 14:00:00-04:00',
'2014-08-01 15:00:00-04:00', '2014-08-01 16:00:00-04:00',
'2014-08-01 17:00:00-04:00', '2014-08-01 18:00:00-04:00'],
dtype='datetime64[ns]', freq='H', tz='US/Eastern')
In [300]: didx.tz_localize(None)
Out[300]:
DatetimeIndex(['2014-08-01 09:00:00',
'2014-08-01 11:00:00',
'2014-08-01 13:00:00',
'2014-08-01 15:00:00',
'2014-08-01 17:00:00',
dtype='datetime64[ns]',
'2014-08-01 10:00:00',
'2014-08-01 12:00:00',
'2014-08-01 14:00:00',
'2014-08-01 16:00:00',
'2014-08-01 18:00:00'],
freq='H', tz=None)
In [301]: didx.tz_convert(None)
Out[301]:
DatetimeIndex(['2014-08-01 13:00:00', '2014-08-01 14:00:00',
'2014-08-01 15:00:00', '2014-08-01 16:00:00',
'2014-08-01 17:00:00', '2014-08-01 18:00:00',
590
Chapter 20. Time Series / Date functionality
pandas: powerful Python data analysis toolkit, Release 0.16.1
'2014-08-01 19:00:00', '2014-08-01 20:00:00',
'2014-08-01 21:00:00', '2014-08-01 22:00:00'],
dtype='datetime64[ns]', freq='H', tz=None)
# tz_convert(None) is identical with tz_convert('UTC').tz_localize(None)
In [302]: didx.tz_convert('UCT').tz_localize(None)
Out[302]:
DatetimeIndex(['2014-08-01 13:00:00', '2014-08-01 14:00:00',
'2014-08-01 15:00:00', '2014-08-01 16:00:00',
'2014-08-01 17:00:00', '2014-08-01 18:00:00',
'2014-08-01 19:00:00', '2014-08-01 20:00:00',
'2014-08-01 21:00:00', '2014-08-01 22:00:00'],
dtype='datetime64[ns]', freq='H', tz=None)
20.11. Time Zone Handling
591
pandas: powerful Python data analysis toolkit, Release 0.16.1
592
Chapter 20. Time Series / Date functionality
CHAPTER
TWENTYONE
TIME DELTAS
Note:
Starting in v0.15.0, we introduce a new scalar type Timedelta, which is a subclass of
datetime.timedelta, and behaves in a similar manner, but allows compatibility with np.timedelta64 types
as well as a host of custom representation, parsing, and attributes.
Timedeltas are differences in times, expressed in difference units, e.g. days, hours, minutes, seconds. They can be
both positive and negative.
21.1 Parsing
You can construct a Timedelta scalar through various arguments:
# strings
In [1]: Timedelta('1 days')
Out[1]: Timedelta('1 days 00:00:00')
In [2]: Timedelta('1 days 00:00:00')
Out[2]: Timedelta('1 days 00:00:00')
In [3]: Timedelta('1 days 2 hours')
Out[3]: Timedelta('1 days 02:00:00')
In [4]: Timedelta('-1 days 2 min 3us')
Out[4]: Timedelta('-2 days +23:57:59.999997')
# like datetime.timedelta
# note: these MUST be specified as keyword arguments
In [5]: Timedelta(days=1,seconds=1)
Out[5]: Timedelta('1 days 00:00:01')
# integers with a unit
In [6]: Timedelta(1,unit='d')
Out[6]: Timedelta('1 days 00:00:00')
# from a timedelta/np.timedelta64
In [7]: Timedelta(timedelta(days=1,seconds=1))
Out[7]: Timedelta('1 days 00:00:01')
In [8]: Timedelta(np.timedelta64(1,'ms'))
Out[8]: Timedelta('0 days 00:00:00.001000')
# negative Timedeltas have this string repr
593
pandas: powerful Python data analysis toolkit, Release 0.16.1
# to be more consistent with datetime.timedelta conventions
In [9]: Timedelta('-1us')
Out[9]: Timedelta('-1 days +23:59:59.999999')
# a NaT
In [10]: Timedelta('nan')
Out[10]: NaT
In [11]: Timedelta('nat')
Out[11]: NaT
DateOffsets (Day, Hour, Minute, Second, Milli, Micro, Nano) can also be used in construction.
In [12]: Timedelta(Second(2))
Out[12]: Timedelta('0 days 00:00:02')
Further, operations among the scalars yield another scalar Timedelta
In [13]: Timedelta(Day(2)) + Timedelta(Second(2)) + Timedelta('00:00:00.000123')
Out[13]: Timedelta('2 days 00:00:02.000123')
21.1.1 to_timedelta
Warning: Prior to 0.15.0 pd.to_timedelta would return a Series for list-like/Series input, and a
np.timedelta64 for scalar input. It will now return a TimedeltaIndex for list-like input, Series for
Series input, and Timedelta for scalar input.
The arguments to pd.to_timedelta are now (arg,unit=’ns’,box=True), previously were
(arg,box=True,unit=’ns’) as these are more logical.
Using the top-level pd.to_timedelta, you can convert a scalar, array, list, or Series from a recognized timedelta
format / value into a Timedelta type. It will construct Series if the input is a Series, a scalar if the input is scalar-like,
otherwise will output a TimedeltaIndex
In [14]: to_timedelta('1 days 06:05:01.00003')
Out[14]: Timedelta('1 days 06:05:01.000030')
In [15]: to_timedelta('15.5us')
Out[15]: Timedelta('0 days 00:00:00.000015')
In [16]: to_timedelta(['1 days 06:05:01.00003','15.5us','nan'])
Out[16]: TimedeltaIndex(['1 days 06:05:01.000030', '0 days 00:00:00.000015', NaT], dtype='timedelta64
In [17]: to_timedelta(np.arange(5),unit='s')
Out[17]: TimedeltaIndex(['00:00:00', '00:00:01', '00:00:02', '00:00:03', '00:00:04'], dtype='timedelt
In [18]: to_timedelta(np.arange(5),unit='d')
Out[18]: TimedeltaIndex(['0 days', '1 days', '2 days', '3 days', '4 days'], dtype='timedelta64[ns]',
21.2 Operations
You can operate on Series/DataFrames and construct timedelta64[ns] Series through subtraction operations on
datetime64[ns] Series, or Timestamps.
594
Chapter 21. Time Deltas
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [19]: s = Series(date_range('2012-1-1', periods=3, freq='D'))
In [20]: td = Series([ Timedelta(days=i) for i in range(3) ])
In [21]: df = DataFrame(dict(A = s, B = td))
In [22]: df
Out[22]:
A
B
0 2012-01-01 0 days
1 2012-01-02 1 days
2 2012-01-03 2 days
In [23]: df['C'] = df['A'] + df['B']
In [24]: df
Out[24]:
A
B
C
0 2012-01-01 0 days 2012-01-01
1 2012-01-02 1 days 2012-01-03
2 2012-01-03 2 days 2012-01-05
In [25]: df.dtypes
Out[25]:
A
datetime64[ns]
B
timedelta64[ns]
C
datetime64[ns]
dtype: object
In [26]: s - s.max()
Out[26]:
0
-2 days
1
-1 days
2
0 days
dtype: timedelta64[ns]
In [27]: s - datetime(2011,1,1,3,5)
Out[27]:
0
364 days 20:55:00
1
365 days 20:55:00
2
366 days 20:55:00
dtype: timedelta64[ns]
In [28]: s + timedelta(minutes=5)
Out[28]:
0
2012-01-01 00:05:00
1
2012-01-02 00:05:00
2
2012-01-03 00:05:00
dtype: datetime64[ns]
In [29]: s + Minute(5)
Out[29]:
0
2012-01-01 00:05:00
1
2012-01-02 00:05:00
2
2012-01-03 00:05:00
dtype: datetime64[ns]
In [30]: s + Minute(5) + Milli(5)
21.2. Operations
595
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[30]:
0
2012-01-01 00:05:00.005000
1
2012-01-02 00:05:00.005000
2
2012-01-03 00:05:00.005000
dtype: datetime64[ns]
Operations with scalars from a timedelta64[ns] series
In [31]: y = s - s[0]
In [32]: y
Out[32]:
0
0 days
1
1 days
2
2 days
dtype: timedelta64[ns]
Series of timedeltas with NaT values are supported
In [33]: y = s - s.shift()
In [34]: y
Out[34]:
0
NaT
1
1 days
2
1 days
dtype: timedelta64[ns]
Elements can be set to NaT using np.nan analogously to datetimes
In [35]: y[1] = np.nan
In [36]: y
Out[36]:
0
NaT
1
NaT
2
1 days
dtype: timedelta64[ns]
Operands can also appear in a reversed order (a singular object operated with a Series)
In [37]: s.max() - s
Out[37]:
0
2 days
1
1 days
2
0 days
dtype: timedelta64[ns]
In [38]: datetime(2011,1,1,3,5) - s
Out[38]:
0
-365 days +03:05:00
1
-366 days +03:05:00
2
-367 days +03:05:00
dtype: timedelta64[ns]
In [39]: timedelta(minutes=5) + s
Out[39]:
0
2012-01-01 00:05:00
1
2012-01-02 00:05:00
596
Chapter 21. Time Deltas
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
2012-01-03 00:05:00
dtype: datetime64[ns]
min, max and the corresponding idxmin, idxmax operations are supported on frames
In [40]: A = s - Timestamp('20120101') - Timedelta('00:05:05')
In [41]: B = s - Series(date_range('2012-1-2', periods=3, freq='D'))
In [42]: df = DataFrame(dict(A=A, B=B))
In [43]: df
Out[43]:
A
B
0 -1 days +23:54:55 -1 days
1
0 days 23:54:55 -1 days
2
1 days 23:54:55 -1 days
In [44]: df.min()
Out[44]:
A
-1 days +23:54:55
B
-1 days +00:00:00
dtype: timedelta64[ns]
In [45]: df.min(axis=1)
Out[45]:
0
-1 days
1
-1 days
2
-1 days
dtype: timedelta64[ns]
In [46]: df.idxmin()
Out[46]:
A
0
B
0
dtype: int64
In [47]: df.idxmax()
Out[47]:
A
2
B
0
dtype: int64
min, max, idxmin, idxmax operations are supported on Series as well. A scalar result will be a Timedelta.
In [48]: df.min().max()
Out[48]: Timedelta('-1 days +23:54:55')
In [49]: df.min(axis=1).min()
Out[49]: Timedelta('-1 days +00:00:00')
In [50]: df.min().idxmax()
Out[50]: 'A'
In [51]: df.min(axis=1).idxmin()
Out[51]: 0
You can fillna on timedeltas. Integers will be interpreted as seconds. You can pass a timedelta to get a particular value.
21.2. Operations
597
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [52]: y.fillna(0)
Out[52]:
0
0 days
1
0 days
2
1 days
dtype: timedelta64[ns]
In [53]: y.fillna(10)
Out[53]:
0
0 days 00:00:10
1
0 days 00:00:10
2
1 days 00:00:00
dtype: timedelta64[ns]
In [54]: y.fillna(Timedelta('-1 days, 00:00:05'))
Out[54]:
0
-1 days +00:00:05
1
-1 days +00:00:05
2
1 days 00:00:00
dtype: timedelta64[ns]
You can also negate, multiply and use abs on Timedeltas
In [55]: td1 = Timedelta('-1 days 2 hours 3 seconds')
In [56]: td1
Out[56]: Timedelta('-2 days +21:59:57')
In [57]: -1 * td1
Out[57]: Timedelta('1 days 02:00:03')
In [58]: - td1
Out[58]: Timedelta('1 days 02:00:03')
In [59]: abs(td1)
Out[59]: Timedelta('1 days 02:00:03')
21.3 Reductions
Numeric reduction operation for timedelta64[ns] will return Timedelta objects. As usual NaT are skipped
during evaluation.
In [60]: y2 = Series(to_timedelta(['-1 days +00:00:05','nat','-1 days +00:00:05','1 days']))
In [61]: y2
Out[61]:
0
-1 days +00:00:05
1
NaT
2
-1 days +00:00:05
3
1 days 00:00:00
dtype: timedelta64[ns]
In [62]: y2.mean()
Out[62]: Timedelta('-1 days +16:00:03.333333')
In [63]: y2.median()
598
Chapter 21. Time Deltas
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[63]: Timedelta('-1 days +00:00:05')
In [64]: y2.quantile(.1)
Out[64]: Timedelta('-1 days +00:00:05')
In [65]: y2.sum()
Out[65]: Timedelta('-1 days +00:00:10')
21.4 Frequency Conversion
New in version 0.13.
Timedelta Series, TimedeltaIndex, and Timedelta scalars can be converted to other ‘frequencies’ by dividing
by another timedelta, or by astyping to a specific timedelta type. These operations yield Series and propogate NaT ->
nan. Note that division by the numpy scalar is true division, while astyping is equivalent of floor division.
In [66]: td = Series(date_range('20130101',periods=4)) - \
....:
Series(date_range('20121201',periods=4))
....:
In [67]: td[2] += timedelta(minutes=5,seconds=3)
In [68]: td[3] = np.nan
In [69]: td
Out[69]:
0
31 days 00:00:00
1
31 days 00:00:00
2
31 days 00:05:03
3
NaT
dtype: timedelta64[ns]
# to days
In [70]: td / np.timedelta64(1,'D')
Out[70]:
0
31.000000
1
31.000000
2
31.003507
3
NaN
dtype: float64
In [71]: td.astype('timedelta64[D]')
Out[71]:
0
31
1
31
2
31
3
NaN
dtype: float64
# to seconds
In [72]: td / np.timedelta64(1,'s')
Out[72]:
0
2678400
1
2678400
2
2678703
3
NaN
21.4. Frequency Conversion
599
pandas: powerful Python data analysis toolkit, Release 0.16.1
dtype: float64
In [73]: td.astype('timedelta64[s]')
Out[73]:
0
2678400
1
2678400
2
2678703
3
NaN
dtype: float64
# to months (these are constant months)
In [74]: td / np.timedelta64(1,'M')
Out[74]:
0
1.018501
1
1.018501
2
1.018617
3
NaN
dtype: float64
Dividing or multiplying a timedelta64[ns] Series by an integer or integer Series yields another
timedelta64[ns] dtypes Series.
In [75]: td * -1
Out[75]:
0
-31 days +00:00:00
1
-31 days +00:00:00
2
-32 days +23:54:57
3
NaT
dtype: timedelta64[ns]
In [76]: td * Series([1,2,3,4])
Out[76]:
0
31 days 00:00:00
1
62 days 00:00:00
2
93 days 00:15:09
3
NaT
dtype: timedelta64[ns]
21.5 Attributes
You can access various components of the Timedelta or TimedeltaIndex directly using the attributes days,seconds,microseconds,nanoseconds. These are identical to the values returned by
datetime.timedelta, in that, for example, the .seconds attribute represents the number of seconds >= 0
and < 1 day. These are signed according to whether the Timedelta is signed.
These operations can also be directly accessed via the .dt property of the Series as well.
Note: Note that the attributes are NOT the displayed values of the Timedelta. Use .components to retrieve the
displayed values.
For a Series
In [77]: td.dt.days
Out[77]:
0
31
1
31
600
Chapter 21. Time Deltas
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
31
3
NaN
dtype: float64
In [78]: td.dt.seconds
Out[78]:
0
0
1
0
2
303
3
NaN
dtype: float64
You can access the value of the fields for a scalar Timedelta directly.
In [79]: tds = Timedelta('31 days 5 min 3 sec')
In [80]: tds.days
Out[80]: 31L
In [81]: tds.seconds
Out[81]: 303L
In [82]: (-tds).seconds
Out[82]: 86097L
You can use the .components property to access a reduced form of the timedelta. This returns a DataFrame
indexed similarly to the Series. These are the displayed values of the Timedelta.
In [83]: td.dt.components
Out[83]:
days hours minutes seconds
0
31
0
0
0
1
31
0
0
0
2
31
0
5
3
3
NaN
NaN
NaN
NaN
milliseconds
0
0
0
NaN
microseconds
0
0
0
NaN
nanoseconds
0
0
0
NaN
In [84]: td.dt.components.seconds
Out[84]:
0
0
1
0
2
3
3
NaN
Name: seconds, dtype: float64
21.6 TimedeltaIndex
New in version 0.15.0.
To generate an index with time delta, you can use either the TimedeltaIndex or the timedelta_range constructor.
Using TimedeltaIndex you can pass string-like, Timedelta, timedelta, or np.timedelta64 objects.
Passing np.nan/pd.NaT/nat will represent missing values.
In [85]: TimedeltaIndex(['1 days','1 days, 00:00:05',
....:
np.timedelta64(2,'D'),timedelta(days=2,seconds=2)])
....:
21.6. TimedeltaIndex
601
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[85]:
TimedeltaIndex(['1 days 00:00:00', '1 days 00:00:05', '2 days 00:00:00',
'2 days 00:00:02'],
dtype='timedelta64[ns]', freq=None)
Similarly to date_range, you can construct regular ranges of a TimedeltaIndex:
In [86]: timedelta_range(start='1 days',periods=5,freq='D')
Out[86]: TimedeltaIndex(['1 days', '2 days', '3 days', '4 days', '5 days'], dtype='timedelta64[ns]',
In [87]: timedelta_range(start='1 days',end='2 days',freq='30T')
Out[87]:
TimedeltaIndex(['1 days 00:00:00', '1 days 00:30:00', '1 days 01:00:00',
'1 days 01:30:00', '1 days 02:00:00', '1 days 02:30:00',
'1 days 03:00:00', '1 days 03:30:00', '1 days 04:00:00',
'1 days 04:30:00', '1 days 05:00:00', '1 days 05:30:00',
'1 days 06:00:00', '1 days 06:30:00', '1 days 07:00:00',
'1 days 07:30:00', '1 days 08:00:00', '1 days 08:30:00',
'1 days 09:00:00', '1 days 09:30:00', '1 days 10:00:00',
'1 days 10:30:00', '1 days 11:00:00', '1 days 11:30:00',
'1 days 12:00:00', '1 days 12:30:00', '1 days 13:00:00',
'1 days 13:30:00', '1 days 14:00:00', '1 days 14:30:00',
'1 days 15:00:00', '1 days 15:30:00', '1 days 16:00:00',
'1 days 16:30:00', '1 days 17:00:00', '1 days 17:30:00',
'1 days 18:00:00', '1 days 18:30:00', '1 days 19:00:00',
'1 days 19:30:00', '1 days 20:00:00', '1 days 20:30:00',
'1 days 21:00:00', '1 days 21:30:00', '1 days 22:00:00',
'1 days 22:30:00', '1 days 23:00:00', '1 days 23:30:00',
'2 days 00:00:00'],
dtype='timedelta64[ns]', freq='30T')
21.6.1 Using the TimedeltaIndex
Similarly to other of the datetime-like indices, DatetimeIndex and PeriodIndex, you can use
TimedeltaIndex as the index of pandas objects.
In [88]: s = Series(np.arange(100),
....:
index=timedelta_range('1 days',periods=100,freq='h'))
....:
In [89]: s
Out[89]:
1 days 00:00:00
1 days 01:00:00
1 days 02:00:00
1 days 03:00:00
1 days 04:00:00
1 days 05:00:00
1 days 06:00:00
4
4
4
5
5
5
5
days
days
days
days
days
days
days
602
21:00:00
22:00:00
23:00:00
00:00:00
01:00:00
02:00:00
03:00:00
0
1
2
3
4
5
6
..
93
94
95
96
97
98
99
Chapter 21. Time Deltas
pandas: powerful Python data analysis toolkit, Release 0.16.1
Freq: H, dtype: int32
Selections work similary, with coercion on string-likes and slices:
In [90]: s['1 day':'2 day']
Out[90]:
1 days 00:00:00
0
1 days 01:00:00
1
1 days 02:00:00
2
1 days 03:00:00
3
1 days 04:00:00
4
1 days 05:00:00
5
1 days 06:00:00
6
..
2 days 17:00:00
41
2 days 18:00:00
42
2 days 19:00:00
43
2 days 20:00:00
44
2 days 21:00:00
45
2 days 22:00:00
46
2 days 23:00:00
47
dtype: int32
In [91]: s['1 day 01:00:00']
Out[91]: 1
In [92]: s[Timedelta('1 day 1h')]
Out[92]: 1
Furthermore you can use partial string selection and the range will be inferred:
In [93]: s['1 day':'1 day 5 hours']
Out[93]:
1 days 00:00:00
0
1 days 01:00:00
1
1 days 02:00:00
2
1 days 03:00:00
3
1 days 04:00:00
4
1 days 05:00:00
5
dtype: int32
21.6.2 Operations
Finally, the combination of TimedeltaIndex with DatetimeIndex allow certain combination operations that
are NaT preserving:
In [94]: tdi = TimedeltaIndex(['1 days',pd.NaT,'2 days'])
In [95]: tdi.tolist()
Out[95]: [Timedelta('1 days 00:00:00'), NaT, Timedelta('2 days 00:00:00')]
In [96]: dti = date_range('20130101',periods=3)
In [97]: dti.tolist()
Out[97]:
[Timestamp('2013-01-01 00:00:00', offset='D'),
Timestamp('2013-01-02 00:00:00', offset='D'),
21.6. TimedeltaIndex
603
pandas: powerful Python data analysis toolkit, Release 0.16.1
Timestamp('2013-01-03 00:00:00', offset='D')]
In [98]: (dti + tdi).tolist()
Out[98]: [Timestamp('2013-01-02 00:00:00'), NaT, Timestamp('2013-01-05 00:00:00')]
In [99]: (dti - tdi).tolist()
Out[99]: [Timestamp('2012-12-31 00:00:00'), NaT, Timestamp('2013-01-01 00:00:00')]
21.6.3 Conversions
Similarly to frequency conversion on a Series above, you can convert these indices to yield another Index.
In [100]: tdi / np.timedelta64(1,'s')
Out[100]: Float64Index([86400.0, nan, 172800.0], dtype='float64')
In [101]: tdi.astype('timedelta64[s]')
Out[101]: Float64Index([86400.0, nan, 172800.0], dtype='float64')
Scalars type ops work as well. These can potentially return a different type of index.
# adding or timedelta and date -> datelike
In [102]: tdi + Timestamp('20130101')
Out[102]: DatetimeIndex(['2013-01-02', 'NaT', '2013-01-03'], dtype='datetime64[ns]', freq=None, tz=No
# subtraction of a date and a timedelta -> datelike
# note that trying to subtract a date from a Timedelta will raise an exception
In [103]: (Timestamp('20130101') - tdi).tolist()
Out[103]: [Timestamp('2012-12-31 00:00:00'), NaT, Timestamp('2012-12-30 00:00:00')]
# timedelta + timedelta -> timedelta
In [104]: tdi + Timedelta('10 days')
Out[104]: TimedeltaIndex(['11 days', NaT, '12 days'], dtype='timedelta64[ns]', freq=None)
# division can result in a Timedelta if the divisor is an integer
In [105]: tdi / 2
Out[105]: TimedeltaIndex(['0 days 12:00:00', NaT, '1 days 00:00:00'], dtype='timedelta64[ns]', freq=N
# or a Float64Index if the divisor is a Timedelta
In [106]: tdi / tdi[0]
Out[106]: Float64Index([1.0, nan, 2.0], dtype='float64')
21.7 Resampling
Similar to timeseries resampling, we can resample with a TimedeltaIndex.
In [107]: s.resample('D')
Out[107]:
1 days
11.5
2 days
35.5
3 days
59.5
4 days
83.5
5 days
97.5
dtype: float64
604
Chapter 21. Time Deltas
CHAPTER
TWENTYTWO
CATEGORICAL DATA
New in version 0.15.
Note: While there was pandas.Categorical in earlier versions, the ability to use categorical data in Series and
DataFrame is new.
This is an introduction to pandas categorical data type, including a short comparison with R’s factor.
Categoricals are a pandas data type, which correspond to categorical variables in statistics: a variable, which can take
on only a limited, and usually fixed, number of possible values (categories; levels in R). Examples are gender, social
class, blood types, country affiliations, observation time or ratings via Likert scales.
In contrast to statistical categorical variables, categorical data might have an order (e.g. ‘strongly agree’ vs ‘agree’ or
‘first observation’ vs. ‘second observation’), but numerical operations (additions, divisions, ...) are not possible.
All values of categorical data are either in categories or np.nan. Order is defined by the order of categories, not lexical
order of the values. Internally, the data structure consists of a categories array and an integer array of codes which
point to the real value in the categories array.
The categorical data type is useful in the following cases:
• A string variable consisting of only a few different values. Converting such a string variable to a categorical
variable will save some memory, see here.
• The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”). By converting to a
categorical and specifying an order on the categories, sorting and min/max will use the logical order instead of
the lexical order, see here.
• As a signal to other python libraries that this column should be treated as a categorical variable (e.g. to use
suitable statistical methods or plot types).
See also the API docs on categoricals.
22.1 Object Creation
Categorical Series or columns in a DataFrame can be created in several ways:
By specifying dtype="category" when constructing a Series:
In [1]: s = Series(["a","b","c","a"], dtype="category")
In [2]: s
Out[2]:
0
a
1
b
605
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
c
3
a
dtype: category
Categories (3, object): [a, b, c]
By converting an existing Series or column to a category dtype:
In [3]: df = DataFrame({"A":["a","b","c","a"]})
In [4]: df["B"] = df["A"].astype('category')
In [5]: df
Out[5]:
A B
0 a a
1 b b
2 c c
3 a a
By using some special functions:
In [6]: df = DataFrame({'value': np.random.randint(0, 100, 20)})
In [7]: labels = [ "{0} - {1}".format(i, i + 9) for i in range(0, 100, 10) ]
In [8]: df['group'] = pd.cut(df.value, range(0, 105, 10), right=False, labels=labels)
In [9]: df.head(10)
Out[9]:
value
group
0
65 60 - 69
1
49 40 - 49
2
56 50 - 59
3
43 40 - 49
4
43 40 - 49
5
91 90 - 99
6
32 30 - 39
7
87 80 - 89
8
36 30 - 39
9
8
0 - 9
See documentation for cut().
By passing a pandas.Categorical object to a Series or assigning it to a DataFrame.
In [10]: raw_cat = Categorical(["a","b","c","a"], categories=["b","c","d"],
....:
ordered=False)
....:
In [11]: s = Series(raw_cat)
In [12]: s
Out[12]:
0
NaN
1
b
2
c
3
NaN
dtype: category
Categories (3, object): [b, c, d]
606
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [13]: df = DataFrame({"A":["a","b","c","a"]})
In [14]: df["B"] = raw_cat
In [15]: df
Out[15]:
A
B
0 a NaN
1 b
b
2 c
c
3 a NaN
You can also specify differently ordered categories or make the resulting data ordered, by passing these arguments to
astype():
In [16]: s = Series(["a","b","c","a"])
In [17]: s_cat = s.astype("category", categories=["b","c","d"], ordered=False)
In [18]: s_cat
Out[18]:
0
NaN
1
b
2
c
3
NaN
dtype: category
Categories (3, object): [b, c, d]
Categorical data has a specific category dtype:
In [19]: df.dtypes
Out[19]:
A
object
B
category
dtype: object
Note: In contrast to R’s factor function, categorical data is not converting input values to strings and categories will
end up the same data type as the original values.
Note: In contrast to R’s factor function, there is currently no way to assign/change labels at creation time. Use
categories to change the categories after creation time.
To get back to the original Series or numpy array, use Series.astype(original_dtype) or
np.asarray(categorical):
In [20]: s = Series(["a","b","c","a"])
In [21]: s
Out[21]:
0
a
1
b
2
c
3
a
dtype: object
In [22]: s2 = s.astype('category')
22.1. Object Creation
607
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [23]: s2
Out[23]:
0
a
1
b
2
c
3
a
dtype: category
Categories (3, object): [a, b, c]
In [24]: s3 = s2.astype('string')
In [25]: s3
Out[25]:
0
a
1
b
2
c
3
a
dtype: object
In [26]: np.asarray(s2)
Out[26]: array(['a', 'b', 'c', 'a'], dtype=object)
If you have already codes and categories, you can use the from_codes() constructor to save the factorize step
during normal constructor mode:
In [27]: splitter = np.random.choice([0,1], 5, p=[0.5,0.5])
In [28]: s = Series(Categorical.from_codes(splitter, categories=["train", "test"]))
22.2 Description
Using .describe() on categorical data will produce similar output to a Series or DataFrame of type string.
In [29]: cat = Categorical(["a","c","c",np.nan], categories=["b","a","c",np.nan] )
In [30]: df = DataFrame({"cat":cat, "s":["a","c","c",np.nan]})
In [31]: df.describe()
Out[31]:
cat s
count
3 3
unique
2 2
top
c c
freq
2 2
In [32]: df["cat"].describe()
Out[32]:
count
3
unique
2
top
c
freq
2
Name: cat, dtype: object
608
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
22.3 Working with categories
Categorical data has a categories and a ordered property, which list their possible values and whether the ordering
matters or not. These properties are exposed as s.cat.categories and s.cat.ordered. If you don’t manually
specify categories and ordering, they are inferred from the passed in values.
In [33]: s = Series(["a","b","c","a"], dtype="category")
In [34]: s.cat.categories
Out[34]: Index([u'a', u'b', u'c'], dtype='object')
In [35]: s.cat.ordered
Out[35]: False
It’s also possible to pass in the categories in a specific order:
In [36]: s = Series(Categorical(["a","b","c","a"], categories=["c","b","a"]))
In [37]: s.cat.categories
Out[37]: Index([u'c', u'b', u'a'], dtype='object')
In [38]: s.cat.ordered
Out[38]: False
Note: New categorical data are NOT automatically ordered. You must explicity pass ordered=True to indicate an
ordered Categorical.
22.3.1 Renaming categories
Renaming categories is done by assigning new values to the Series.cat.categories property or by using the
Categorical.rename_categories() method:
In [39]: s = Series(["a","b","c","a"], dtype="category")
In [40]: s
Out[40]:
0
a
1
b
2
c
3
a
dtype: category
Categories (3, object): [a, b, c]
In [41]: s.cat.categories = ["Group %s" % g for g in s.cat.categories]
In [42]: s
Out[42]:
0
Group a
1
Group b
2
Group c
3
Group a
dtype: category
Categories (3, object): [Group a, Group b, Group c]
In [43]: s.cat.rename_categories([1,2,3])
Out[43]:
22.3. Working with categories
609
pandas: powerful Python data analysis toolkit, Release 0.16.1
0
1
1
2
2
3
3
1
dtype: category
Categories (3, int64): [1, 2, 3]
Note: In contrast to R’s factor, categorical data can have categories of other types than string.
Note:
Be aware that assigning new categories is an inplace operations, while most other operation under
Series.cat per default return a new Series of dtype category.
Categories must be unique or a ValueError is raised:
In [44]: try:
....:
s.cat.categories = [1,1,1]
....: except ValueError as e:
....:
print("ValueError: " + str(e))
....:
ValueError: Categorical categories must be unique
22.3.2 Appending new categories
Appending categories can be done by using the Categorical.add_categories() method:
In [45]: s = s.cat.add_categories([4])
In [46]: s.cat.categories
Out[46]: Index([u'Group a', u'Group b', u'Group c', 4], dtype='object')
In [47]: s
Out[47]:
0
Group a
1
Group b
2
Group c
3
Group a
dtype: category
Categories (4, object): [Group a, Group b, Group c, 4]
22.3.3 Removing categories
Removing categories can be done by using the Categorical.remove_categories() method. Values which
are removed are replaced by np.nan.:
In [48]: s = s.cat.remove_categories([4])
In [49]: s
Out[49]:
0
Group a
1
Group b
2
Group c
3
Group a
dtype: category
Categories (3, object): [Group a, Group b, Group c]
610
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
22.3.4 Removing unused categories
Removing unused categories can also be done:
In [50]: s = Series(Categorical(["a","b","a"], categories=["a","b","c","d"]))
In [51]: s
Out[51]:
0
a
1
b
2
a
dtype: category
Categories (4, object): [a, b, c, d]
In [52]: s.cat.remove_unused_categories()
Out[52]:
0
a
1
b
2
a
dtype: category
Categories (2, object): [a, b]
22.3.5 Setting categories
If you want to do remove and add new categories in one step (which has some speed advantage), or simply set the
categories to a predefined scale, use Categorical.set_categories().
In [53]: s = Series(["one","two","four", "-"], dtype="category")
In [54]: s
Out[54]:
0
one
1
two
2
four
3
dtype: category
Categories (4, object): [-, four, one, two]
In [55]: s = s.cat.set_categories(["one","two","three","four"])
In [56]: s
Out[56]:
0
one
1
two
2
four
3
NaN
dtype: category
Categories (4, object): [one, two, three, four]
Note: Be aware that Categorical.set_categories() cannot know whether some category is omitted intentionally or because it is misspelled or (under Python3) due to a type difference (e.g., numpys S1 dtype and python
strings). This can result in surprising behaviour!
22.3. Working with categories
611
pandas: powerful Python data analysis toolkit, Release 0.16.1
22.4 Sorting and Order
Warning: The default for construction has changed in v0.16.0 to ordered=False, from the prior implicit
ordered=True
If categorical data is ordered (s.cat.ordered == True), then the order of the categories has a meaning and
certain operations are possible. If the categorical is unordered, .min()/.max() will raise a TypeError.
In [57]: s = Series(Categorical(["a","b","c","a"], ordered=False))
In [58]: s.sort()
In [59]: s = Series(["a","b","c","a"]).astype('category', ordered=True)
In [60]: s.sort()
In [61]: s
Out[61]:
0
a
3
a
1
b
2
c
dtype: category
Categories (3, object): [a < b < c]
In [62]: s.min(), s.max()
Out[62]: ('a', 'c')
You can set categorical data to be ordered by using as_ordered() or unordered by using as_unordered().
These will by default return a new object.
In [63]: s.cat.as_ordered()
Out[63]:
0
a
3
a
1
b
2
c
dtype: category
Categories (3, object): [a < b < c]
In [64]: s.cat.as_unordered()
Out[64]:
0
a
3
a
1
b
2
c
dtype: category
Categories (3, object): [a, b, c]
Sorting will use the order defined by categories, not any lexical order present on the data type. This is even true for
strings and numeric data:
In [65]: s = Series([1,2,3,1], dtype="category")
In [66]: s = s.cat.set_categories([2,3,1], ordered=True)
In [67]: s
612
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[67]:
0
1
1
2
2
3
3
1
dtype: category
Categories (3, int64): [2 < 3 < 1]
In [68]: s.sort()
In [69]: s
Out[69]:
1
2
2
3
0
1
3
1
dtype: category
Categories (3, int64): [2 < 3 < 1]
In [70]: s.min(), s.max()
Out[70]: (2, 1)
22.4.1 Reordering
Reordering the categories is possible via the Categorical.reorder_categories() and the
Categorical.set_categories() methods. For Categorical.reorder_categories(), all old
categories must be included in the new categories and no new categories are allowed. This will necessarily make the
sort order the same as the categories order.
In [71]: s = Series([1,2,3,1], dtype="category")
In [72]: s = s.cat.reorder_categories([2,3,1], ordered=True)
In [73]: s
Out[73]:
0
1
1
2
2
3
3
1
dtype: category
Categories (3, int64): [2 < 3 < 1]
In [74]: s.sort()
In [75]: s
Out[75]:
1
2
2
3
0
1
3
1
dtype: category
Categories (3, int64): [2 < 3 < 1]
In [76]: s.min(), s.max()
Out[76]: (2, 1)
22.4. Sorting and Order
613
pandas: powerful Python data analysis toolkit, Release 0.16.1
Note: Note the difference between assigning new categories and reordering the categories: the first renames categories
and therefore the individual values in the Series, but if the first position was sorted last, the renamed value will still be
sorted last. Reordering means that the way values are sorted is different afterwards, but not that individual values in
the Series are changed.
Note: If the Categorical is not ordered, Series.min() and Series.max() will raise TypeError. Numeric
operations like +, -, *, / and operations based on them (e.g.‘‘Series.median()‘‘, which would need to compute the
mean between two values if the length of an array is even) do not work and raise a TypeError.
22.4.2 Multi Column Sorting
A categorical dtyped column will partcipate in a multi-column sort in a similar manner to other columns. The ordering
of the categorical is determined by the categories of that column.
In [77]: dfs = DataFrame({'A' : Categorical(list('bbeebbaa'), categories=['e','a','b'], ordered=True)
....:
'B' : [1,2,1,2,2,1,2,1] })
....:
In [78]: dfs.sort(['A', 'B'])
Out[78]:
A B
2 e 1
3 e 2
7 a 1
6 a 2
0 b 1
5 b 1
1 b 2
4 b 2
Reordering the categories changes a future sort.
In [79]: dfs['A'] = dfs['A'].cat.reorder_categories(['a','b','e'])
In [80]: dfs.sort(['A','B'])
Out[80]:
A B
7 a 1
6 a 2
0 b 1
5 b 1
1 b 2
4 b 2
2 e 1
3 e 2
22.5 Comparisons
Comparing categorical data with other objects is possible in three cases:
• comparing equality (== and !=) to a list-like object (list, Series, array, ...) of the same length as the categorical
data.
614
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
• all comparisons (==, !=, >, >=, <, and <=) of categorical data to another categorical Series, when
ordered==True and the categories are the same.
• all comparisons of a categorical data to a scalar.
All other comparisons, especially “non-equality” comparisons of two categoricals with different categories or a categorical with any list-like object, will raise a TypeError.
Note: Any “non-equality” comparisons of categorical data with a Series, np.array, list or categorical data with
different categories or ordering will raise an TypeError because custom categories ordering could be interpreted in two
ways: one with taking into account the ordering and one without.
In [81]: cat = Series([1,2,3]).astype("category", categories=[3,2,1], ordered=True)
In [82]: cat_base = Series([2,2,2]).astype("category", categories=[3,2,1], ordered=True)
In [83]: cat_base2 = Series([2,2,2]).astype("category", ordered=True)
In [84]: cat
Out[84]:
0
1
1
2
2
3
dtype: category
Categories (3, int64): [3 < 2 < 1]
In [85]: cat_base
Out[85]:
0
2
1
2
2
2
dtype: category
Categories (3, int64): [3 < 2 < 1]
In [86]: cat_base2
Out[86]:
0
2
1
2
2
2
dtype: category
Categories (1, int64): [2]
Comparing to a categorical with the same categories and ordering or to a scalar works:
In [87]: cat > cat_base
Out[87]:
0
True
1
False
2
False
dtype: bool
In [88]: cat > 2
Out[88]:
0
True
1
False
2
False
dtype: bool
Equality comparisons work with any list-like object of same length and scalars:
22.5. Comparisons
615
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [89]: cat == cat_base
Out[89]:
0
False
1
True
2
False
dtype: bool
In [90]: cat == np.array([1,2,3])
Out[90]:
0
True
1
True
2
True
dtype: bool
In [91]: cat == 2
Out[91]:
0
False
1
True
2
False
dtype: bool
This doesn’t work because the categories are not the same:
In [92]: try:
....:
cat > cat_base2
....: except TypeError as e:
....:
print("TypeError: " + str(e))
....:
TypeError: Categoricals can only be compared if 'categories' are the same
If you want to do a “non-equality” comparison of a categorical series with a list-like object which is not categorical
data, you need to be explicit and convert the categorical data back to the original values:
In [93]: base = np.array([1,2,3])
In [94]: try:
....:
cat > base
....: except TypeError as e:
....:
print("TypeError: " + str(e))
....:
TypeError: Cannot compare a Categorical for op __gt__ with type . If you want t
compare values, use 'np.asarray(cat) other'.
In [95]: np.asarray(cat) > base
Out[95]: array([False, False, False], dtype=bool)
22.6 Operations
Apart from Series.min(), Series.max() and Series.mode(), the following operations are possible with
categorical data:
Series methods like Series.value_counts() will use all categories, even if some categories are not present in the data:
In [96]: s = Series(Categorical(["a","b","c","c"], categories=["c","a","b","d"]))
In [97]: s.value_counts()
Out[97]:
616
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
c
2
b
1
a
1
d
0
dtype: int64
Groupby will also show “unused” categories:
In [98]: cats = Categorical(["a","b","b","b","c","c","c"], categories=["a","b","c","d"])
In [99]: df = DataFrame({"cats":cats,"values":[1,2,2,2,3,4,5]})
In [100]: df.groupby("cats").mean()
Out[100]:
values
cats
a
1
b
2
c
4
d
NaN
In [101]: cats2 = Categorical(["a","a","b","b"], categories=["a","b","c"])
In [102]: df2 = DataFrame({"cats":cats2,"B":["c","d","c","d"], "values":[1,2,3,4]})
In [103]: df2.groupby(["cats","B"]).mean()
Out[103]:
values
cats B
a
c
1
d
2
b
c
3
d
4
c
c
NaN
d
NaN
Pivot tables:
In [104]: raw_cat = Categorical(["a","a","b","b"], categories=["a","b","c"])
In [105]: df = DataFrame({"A":raw_cat,"B":["c","d","c","d"], "values":[1,2,3,4]})
In [106]: pd.pivot_table(df, values='values', index=['A', 'B'])
Out[106]:
A B
a c
1
d
2
b c
3
d
4
c c
NaN
d
NaN
Name: values, dtype: float64
22.7 Data munging
The optimized pandas data access methods .loc, .iloc, .ix .at, and .iat, work as normal. The only difference
is the return type (for getting) and that only values already in categories can be assigned.
22.7. Data munging
617
pandas: powerful Python data analysis toolkit, Release 0.16.1
22.7.1 Getting
If the slicing operation returns either a DataFrame or a column of type Series, the category dtype is preserved.
In [107]: idx = Index(["h","i","j","k","l","m","n",])
In [108]: cats = Series(["a","b","b","b","c","c","c"], dtype="category", index=idx)
In [109]: values= [1,2,2,2,3,4,5]
In [110]: df = DataFrame({"cats":cats,"values":values}, index=idx)
In [111]: df.iloc[2:4,:]
Out[111]:
cats values
j
b
2
k
b
2
In [112]: df.iloc[2:4,:].dtypes
Out[112]:
cats
category
values
int64
dtype: object
In [113]: df.loc["h":"j","cats"]
Out[113]:
h
a
i
b
j
b
Name: cats, dtype: category
Categories (3, object): [a, b, c]
In [114]: df.ix["h":"j",0:1]
Out[114]:
cats
h
a
i
b
j
b
In [115]: df[df["cats"] == "b"]
Out[115]:
cats values
i
b
2
j
b
2
k
b
2
An example where the category type is not preserved is if you take one single row: the resulting Series is of dtype
object:
# get the complete "h" row as a Series
In [116]: df.loc["h", :]
Out[116]:
cats
a
values
1
Name: h, dtype: object
Returning a single item from categorical data will also return the value, not a categorical of length “1”.
618
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [117]: df.iat[0,0]
Out[117]: 'a'
In [118]: df["cats"].cat.categories = ["x","y","z"]
In [119]: df.at["h","cats"] # returns a string
Out[119]: 'x'
Note: This is a difference to R’s factor function, where factor(c(1,2,3))[1] returns a single value factor.
To get a single value Series of type category pass in a list with a single value:
In [120]: df.loc[["h"],"cats"]
Out[120]:
h
x
Name: cats, dtype: category
Categories (3, object): [x, y, z]
22.7.2 Setting
Setting values in a categorical column (or Series) works as long as the value is included in the categories:
In [121]: idx = Index(["h","i","j","k","l","m","n"])
In [122]: cats = Categorical(["a","a","a","a","a","a","a"], categories=["a","b"])
In [123]: values = [1,1,1,1,1,1,1]
In [124]: df = DataFrame({"cats":cats,"values":values}, index=idx)
In [125]: df.iloc[2:4,:] = [["b",2],["b",2]]
In [126]: df
Out[126]:
cats values
h
a
1
i
a
1
j
b
2
k
b
2
l
a
1
m
a
1
n
a
1
In [127]: try:
.....:
df.iloc[2:4,:] = [["c",3],["c",3]]
.....: except ValueError as e:
.....:
print("ValueError: " + str(e))
.....:
ValueError: cannot setitem on a Categorical with a new category, set the categories first
Setting values by assigning categorical data will also check that the categories match:
In [128]: df.loc["j":"k","cats"] = Categorical(["a","a"], categories=["a","b"])
In [129]: df
Out[129]:
cats values
22.7. Data munging
619
pandas: powerful Python data analysis toolkit, Release 0.16.1
h
i
j
k
l
m
n
a
a
a
a
a
a
a
1
1
2
2
1
1
1
In [130]: try:
.....:
df.loc["j":"k","cats"] = Categorical(["b","b"], categories=["a","b","c"])
.....: except ValueError as e:
.....:
print("ValueError: " + str(e))
.....:
ValueError: Cannot set a Categorical with another, without identical categories
Assigning a Categorical to parts of a column of other types will use the values:
In [131]: df = DataFrame({"a":[1,1,1,1,1], "b":["a","a","a","a","a"]})
In [132]: df.loc[1:2,"a"] = Categorical(["b","b"], categories=["a","b"])
In [133]: df.loc[2:3,"b"] = Categorical(["b","b"], categories=["a","b"])
In [134]: df
Out[134]:
a b
0 1 a
1 b a
2 b b
3 1 b
4 1 a
In [135]: df.dtypes
Out[135]:
a
object
b
object
dtype: object
22.7.3 Merging
You can concat two DataFrames containing categorical data together, but the categories of these categoricals need to
be the same:
In [136]: cat = Series(["a","b"], dtype="category")
In [137]: vals = [1,2]
In [138]: df = DataFrame({"cats":cat, "vals":vals})
In [139]: res = pd.concat([df,df])
In [140]: res
Out[140]:
cats vals
0
a
1
1
b
2
0
a
1
620
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
b
2
In [141]: res.dtypes
Out[141]:
cats
category
vals
int64
dtype: object
In this case the categories are not the same and so an error is raised:
In [142]: df_different = df.copy()
In [143]: df_different["cats"].cat.categories = ["c","d"]
In [144]: try:
.....:
pd.concat([df,df_different])
.....: except ValueError as e:
.....:
print("ValueError: " + str(e))
.....:
ValueError: incompatible categories in categorical concat
The same applies to df.append(df_different).
22.8 Getting Data In/Out
New in version 0.15.2.
Writing data (Series, Frames) to a HDF store that contains a category dtype was implemented in 0.15.2. See here
for an example and caveats.
Writing data to and reading data from Stata format files was implemented in 0.15.2. See here for an example and
caveats.
Writing to a CSV file will convert the data, effectively removing any information about the categorical (categories and
ordering). So if you read back the CSV file you have to convert the relevant columns back to category and assign the
right categories and categories ordering.
In [145]: s = Series(Categorical(['a', 'b', 'b', 'a', 'a', 'd']))
# rename the categories
In [146]: s.cat.categories = ["very good", "good", "bad"]
# reorder the categories and add missing categories
In [147]: s = s.cat.set_categories(["very bad", "bad", "medium", "good", "very good"])
In [148]: df = DataFrame({"cats":s, "vals":[1,2,3,4,5,6]})
In [149]: csv = StringIO()
In [150]: df.to_csv(csv)
In [151]: df2 = pd.read_csv(StringIO(csv.getvalue()))
In [152]: df2.dtypes
Out[152]:
Unnamed: 0
int64
cats
object
22.8. Getting Data In/Out
621
pandas: powerful Python data analysis toolkit, Release 0.16.1
vals
dtype: object
int64
In [153]: df2["cats"]
Out[153]:
0
very good
1
good
2
good
3
very good
4
very good
5
bad
Name: cats, dtype: object
# Redo the category
In [154]: df2["cats"] = df2["cats"].astype("category")
In [155]: df2["cats"].cat.set_categories(["very bad", "bad", "medium", "good", "very good"],
.....:
inplace=True)
.....:
In [156]: df2.dtypes
Out[156]:
Unnamed: 0
int64
cats
category
vals
int64
dtype: object
In [157]: df2["cats"]
Out[157]:
0
very good
1
good
2
good
3
very good
4
very good
5
bad
Name: cats, dtype: category
Categories (5, object): [very bad, bad, medium, good, very good]
The same holds for writing to a SQL database with to_sql.
22.9 Missing Data
pandas primarily uses the value np.nan to represent missing data. It is by default not included in computations. See
the Missing Data section
There are two ways a np.nan can be represented in categorical data: either the value is not available (“missing value”)
or np.nan is a valid category.
In [158]: s = Series(["a","b",np.nan,"a"], dtype="category")
# only two categories
In [159]: s
Out[159]:
0
a
1
b
2
NaN
622
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
3
a
dtype: category
Categories (2, object): [a, b]
In [160]: s2 = Series(["a","b","c","a"], dtype="category")
In [161]: s2.cat.categories = [1,2,np.nan]
# three categories, np.nan included
In [162]: s2
Out[162]:
0
1
1
2
2
NaN
3
1
dtype: category
Categories (3, object): [1, 2, NaN]
Note: As integer Series can’t include NaN, the categories were converted to object.
Note: Missing value methods like isnull and fillna will take both missing values as well as np.nan categories
into account:
In [163]: c = Series(["a","b",np.nan], dtype="category")
In [164]: c.cat.set_categories(["a","b",np.nan], inplace=True)
# will be inserted as a NA category:
In [165]: c[0] = np.nan
In [166]: s = Series(c)
In [167]: s
Out[167]:
0
NaN
1
b
2
NaN
dtype: category
Categories (3, object): [a, b, NaN]
In [168]: pd.isnull(s)
Out[168]:
0
True
1
False
2
True
dtype: bool
In [169]: s.fillna("a")
Out[169]:
0
a
1
b
2
a
dtype: category
Categories (3, object): [a, b, NaN]
22.9. Missing Data
623
pandas: powerful Python data analysis toolkit, Release 0.16.1
22.9.1 Differences to R’s factor
The following differences to R’s factor functions can be observed:
• R’s levels are named categories
• R’s levels are always of type string, while categories in pandas can be of any dtype.
• It’s not possible to specify labels at creation time. Use s.cat.rename_categories(new_labels)
afterwards.
• In contrast to R’s factor function, using categorical data as the sole input to create a new categorical series will
not remove unused categories but create a new categorical series which is equal to the passed in one!
22.10 Gotchas
22.10.1 Memory Usage
The memory usage of a Categorical is proportional to the number of categories times the length of the data. In
contrast, an object dtype is a constant times the length of the data.
In [170]: s = Series(['foo','bar']*1000)
# object dtype
In [171]: s.nbytes
Out[171]: 8000
# category dtype
In [172]: s.astype('category').nbytes
Out[172]: 2008
Note: If the number of categories approaches the length of the data, the Categorical will use nearly the same or
more memory than an equivalent object dtype representation.
In [173]: s = Series(['foo%04d' % i for i in range(2000)])
# object dtype
In [174]: s.nbytes
Out[174]: 8000
# category dtype
In [175]: s.astype('category').nbytes
Out[175]: 12000
22.10.2 Old style constructor usage
In earlier versions than pandas 0.15, a Categorical could be constructed by passing in precomputed codes (called then
labels) instead of values with categories. The codes were interpreted as pointers to the categories with -1 as NaN. This
type of constructor useage is replaced by the special constructor Categorical.from_codes().
Unfortunately, in some special cases, using code which assumes the old style constructor usage will work with the
current pandas version, resulting in subtle bugs:
624
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
>>> cat = Categorical([1,2], [1,2,3])
>>> # old version
>>> cat.get_values()
array([2, 3], dtype=int64)
>>> # new version
>>> cat.get_values()
array([1, 2], dtype=int64)
Warning: If you used Categoricals with older versions of pandas, please audit your code before upgrading and
change your code to use the from_codes() constructor.
22.10.3 Categorical is not a numpy array
Currently, categorical data and the underlying Categorical is implemented as a python object and not as a low-level
numpy array dtype. This leads to some problems.
numpy itself doesn’t know about the new dtype:
In [176]: try:
.....:
np.dtype("category")
.....: except TypeError as e:
.....:
print("TypeError: " + str(e))
.....:
TypeError: data type "category" not understood
In [177]: dtype = Categorical(["a"]).dtype
In [178]: try:
.....:
np.dtype(dtype)
.....: except TypeError as e:
.....:
print("TypeError: " + str(e))
.....:
TypeError: data type not understood
Dtype comparisons work:
In [179]: dtype == np.str_
Out[179]: False
In [180]: np.str_ == dtype
Out[180]: False
To check if a Series contains Categorical data, with pandas 0.16 or later, use hasattr(s, ’cat’):
In [181]: hasattr(Series(['a'], dtype='category'), 'cat')
Out[181]: True
In [182]: hasattr(Series(['a']), 'cat')
Out[182]: False
Using numpy functions on a Series of type category should not work as Categoricals are not numeric data (even in
the case that .categories is numeric).
In [183]: s = Series(Categorical([1,2,3,4]))
In [184]: try:
.....:
np.sum(s)
22.10. Gotchas
625
pandas: powerful Python data analysis toolkit, Release 0.16.1
.....: except TypeError as e:
.....:
print("TypeError: " + str(e))
.....:
TypeError: Categorical cannot perform the operation sum
Note: If such a function works, please file a bug at https://github.com/pydata/pandas!
22.10.4 dtype in apply
Pandas currently does not preserve the dtype in apply functions: If you apply along rows you get a Series of object
dtype (same as getting a row -> getting one element will return a basic type) and applying along columns will also
convert to object.
In [185]: df = DataFrame({"a":[1,2,3,4],
.....:
"b":["a","b","c","d"],
.....:
"cats":Categorical([1,2,3,2])})
.....:
In [186]: df.apply(lambda row: type(row["cats"]), axis=1)
Out[186]:
0
1
2
3
dtype: object
In [187]: df.apply(lambda col: col.dtype, axis=0)
Out[187]:
a
object
b
object
cats
object
dtype: object
22.10.5 Categorical Index
New in version 0.16.1.
A new CategoricalIndex index type is introduced in version 0.16.1. See the advanced indexing docs for a more
detailed explanation.
Setting the index, will create create a CategoricalIndex
In [188]: cats = Categorical([1,2,3,4], categories=[4,2,3,1])
In [189]: strings = ["a","b","c","d"]
In [190]: values = [4,2,3,1]
In [191]: df = DataFrame({"strings":strings, "values":values}, index=cats)
In [192]: df.index
Out[192]: CategoricalIndex([1, 2, 3, 4], categories=[4, 2, 3, 1], ordered=False, dtype='category')
# This now sorts by the categories order
In [193]: df.sort_index()
626
Chapter 22. Categorical Data
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[193]:
strings
4
d
2
b
3
c
1
a
values
1
2
3
4
In previous versions (<0.16.1) there is no index of type category, so setting the index to categorical column will
convert the categorical data to a “normal” dtype first and therefore remove any custom ordering of the categories.
22.10.6 Side Effects
Constructing a Series from a Categorical will not copy the input Categorical. This means that changes to the Series
will in most cases change the original Categorical:
In [194]: cat = Categorical([1,2,3,10], categories=[1,2,3,4,10])
In [195]: s = Series(cat, name="cat")
In [196]: cat
Out[196]:
[1, 2, 3, 10]
Categories (5, int64): [1, 2, 3, 4, 10]
In [197]: s.iloc[0:2] = 10
In [198]: cat
Out[198]:
[10, 10, 3, 10]
Categories (5, int64): [1, 2, 3, 4, 10]
In [199]: df = DataFrame(s)
In [200]: df["cat"].cat.categories = [1,2,3,4,5]
In [201]: cat
Out[201]:
[5, 5, 3, 5]
Categories (5, int64): [1, 2, 3, 4, 5]
Use copy=True to prevent such a behaviour or simply don’t reuse Categoricals:
In [202]: cat = Categorical([1,2,3,10], categories=[1,2,3,4,10])
In [203]: s = Series(cat, name="cat", copy=True)
In [204]: cat
Out[204]:
[1, 2, 3, 10]
Categories (5, int64): [1, 2, 3, 4, 10]
In [205]: s.iloc[0:2] = 10
In [206]: cat
Out[206]:
[1, 2, 3, 10]
Categories (5, int64): [1, 2, 3, 4, 10]
22.10. Gotchas
627
pandas: powerful Python data analysis toolkit, Release 0.16.1
Note: This also happens in some cases when you supply a numpy array instead of a Categorical: using an
int array (e.g. np.array([1,2,3,4])) will exhibit the same behaviour, while using a string array (e.g.
np.array(["a","b","c","a"])) will not.
628
Chapter 22. Categorical Data
CHAPTER
TWENTYTHREE
PLOTTING
We use the standard convention for referencing the matplotlib API:
In [1]: import matplotlib.pyplot as plt
The plots in this document are made using matplotlib’s ggplot style (new in version 1.4):
import matplotlib
matplotlib.style.use('ggplot')
If your version of matplotlib is 1.3 or lower, you can set display.mpl_style to ’default’ with
pd.options.display.mpl_style = ’default’ to produce more appealing plots. When set, matplotlib’s
rcParams are changed (globally!) to nicer-looking settings.
We provide the basics in pandas to easily create decent looking plots. See the ecosystem section for visualization
libraries that go beyond the basics documented here.
Note: All calls to np.random are seeded with 123456.
23.1 Basic Plotting: plot
See the cookbook for some advanced strategies
The plot method on Series and DataFrame is just a simple wrapper around plt.plot():
In [2]: ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
In [3]: ts = ts.cumsum()
In [4]: ts.plot()
Out[4]:
629
pandas: powerful Python data analysis toolkit, Release 0.16.1
If the index consists of dates, it calls gcf().autofmt_xdate() to try to format the x-axis nicely as per above.
On DataFrame, plot() is a convenience to plot all of the columns with labels:
In [5]: df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index, columns=list('ABCD'))
In [6]: df = df.cumsum()
In [7]: plt.figure(); df.plot();
630
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
You can plot one column versus another using the x and y keywords in plot():
In [8]: df3 = pd.DataFrame(np.random.randn(1000, 2), columns=['B', 'C']).cumsum()
In [9]: df3['A'] = pd.Series(list(range(len(df))))
In [10]: df3.plot(x='A', y='B')
Out[10]:
23.1. Basic Plotting: plot
631
pandas: powerful Python data analysis toolkit, Release 0.16.1
Note: For more formatting and sytling options, see below.
23.2 Other Plots
The kind keyword argument of plot() accepts a handful of values for plots other than the default Line plot. These
include:
• ‘bar’ or ‘barh’ for bar plots
• ‘hist’ for histogram
• ‘box’ for boxplot
• ‘kde’ or ’density’ for density plots
• ‘area’ for area plots
• ‘scatter’ for scatter plots
• ‘hexbin’ for hexagonal bin plots
• ‘pie’ for pie plots
632
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
In addition to these kind s, there are the DataFrame.hist(), and DataFrame.boxplot() methods, which use a separate
interface.
Finally, there are several plotting functions in pandas.tools.plotting that take a Series or DataFrame as
an argument. These include
• Scatter Matrix
• Andrews Curves
• Parallel Coordinates
• Lag Plot
• Autocorrelation Plot
• Bootstrap Plot
• RadViz
Plots may also be adorned with errorbars or tables.
23.2.1 Bar plots
For labeled, non-time series data, you may wish to produce a bar plot:
In [11]: plt.figure();
In [12]: df.ix[5].plot(kind='bar'); plt.axhline(0, color='k')
Out[12]:
23.2. Other Plots
633
pandas: powerful Python data analysis toolkit, Release 0.16.1
Calling a DataFrame’s plot() method with kind=’bar’ produces a multiple bar plot:
In [13]: df2 = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
In [14]: df2.plot(kind='bar');
634
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
To produce a stacked bar plot, pass stacked=True:
In [15]: df2.plot(kind='bar', stacked=True);
23.2. Other Plots
635
pandas: powerful Python data analysis toolkit, Release 0.16.1
To get horizontal bar plots, pass kind=’barh’:
In [16]: df2.plot(kind='barh', stacked=True);
636
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.2.2 Histograms
New in version 0.15.0.
Histogram can be drawn specifying kind=’hist’.
In [17]: df4 = pd.DataFrame({'a': np.random.randn(1000) + 1, 'b': np.random.randn(1000),
....:
'c': np.random.randn(1000) - 1}, columns=['a', 'b', 'c'])
....:
In [18]: plt.figure();
In [19]: df4.plot(kind='hist', alpha=0.5)
Out[19]:
23.2. Other Plots
637
pandas: powerful Python data analysis toolkit, Release 0.16.1
Histogram can be stacked by stacked=True. Bin size can be changed by bins keyword.
In [20]: plt.figure();
In [21]: df4.plot(kind='hist', stacked=True, bins=20)
Out[21]:
638
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
You can pass other keywords supported by matplotlib hist. For example, horizontal and cumulative histgram can be
drawn by orientation=’horizontal’ and cumulative=’True’.
In [22]: plt.figure();
In [23]: df4['a'].plot(kind='hist', orientation='horizontal', cumulative=True)
Out[23]:
23.2. Other Plots
639
pandas: powerful Python data analysis toolkit, Release 0.16.1
See the hist method and the matplotlib hist documentation for more.
The existing interface DataFrame.hist to plot histogram still can be used.
In [24]: plt.figure();
In [25]: df['A'].diff().hist()
Out[25]:
640
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
DataFrame.hist() plots the histograms of the columns on multiple subplots:
In [26]: plt.figure()
Out[26]:
In [27]: df.diff().hist(color='k', alpha=0.5, bins=50)
Out[27]:
array([[,
0xaf16990c>],
0xaecdea4c>,
0xaec921cc>]], dtype=object)
641
pandas: powerful Python data analysis toolkit, Release 0.16.1
New in version 0.10.0.
The by keyword can be specified to plot grouped histograms:
In [28]: data = pd.Series(np.random.randn(1000))
In [29]: data.hist(by=np.random.randint(0, 4, 1000), figsize=(6, 4))
Out[29]:
array([[,
],
[,
]], dtype=object)
642
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.2.3 Box Plots
Boxplot can be drawn calling a Series and DataFrame.plot with kind=’box’, or DataFrame.boxplot
to visualize the distribution of values within each column.
New in version 0.15.0.
plot method now supports kind=’box’ to draw boxplot.
For instance, here is a boxplot representing five trials of 10 observations of a uniform random variable on [0,1).
In [30]: df = pd.DataFrame(np.random.rand(10, 5), columns=['A', 'B', 'C', 'D', 'E'])
In [31]: df.plot(kind='box')
Out[31]:
23.2. Other Plots
643
pandas: powerful Python data analysis toolkit, Release 0.16.1
Boxplot can be colorized by passing color keyword. You can pass a dict whose keys are boxes, whiskers,
medians and caps. If some keys are missing in the dict, default colors are used for the corresponding artists.
Also, boxplot has sym keyword to specify fliers style.
When you pass other type of arguments via color keyword, it will be directly passed to matplotlib for all the boxes,
whiskers, medians and caps colorization.
The colors are applied to every boxes to be drawn. If you want more complicated colorization, you can get each drawn
artists by passing return_type.
In [32]: color = dict(boxes='DarkGreen', whiskers='DarkOrange',
....:
medians='DarkBlue', caps='Gray')
....:
In [33]: df.plot(kind='box', color=color, sym='r+')
Out[33]:
644
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
Also, you can pass other keywords supported by matplotlib boxplot. For example, horizontal and custom-positioned
boxplot can be drawn by vert=False and positions keywords.
In [34]: df.plot(kind='box', vert=False, positions=[1, 4, 5, 6, 8])
Out[34]:
23.2. Other Plots
645
pandas: powerful Python data analysis toolkit, Release 0.16.1
See the boxplot method and the matplotlib boxplot documenation for more.
The existing interface DataFrame.boxplot to plot boxplot still can be used.
In [35]: df = pd.DataFrame(np.random.rand(10,5))
In [36]: plt.figure();
In [37]: bp = df.boxplot()
646
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
You can create a stratified boxplot using the by keyword argument to create groupings. For instance,
In [38]: df = pd.DataFrame(np.random.rand(10,2), columns=['Col1', 'Col2'] )
In [39]: df['X'] = pd.Series(['A','A','A','A','A','B','B','B','B','B'])
In [40]: plt.figure();
In [41]: bp = df.boxplot(by='X')
23.2. Other Plots
647
pandas: powerful Python data analysis toolkit, Release 0.16.1
You can also pass a subset of columns to plot, as well as group by multiple columns:
In [42]: df = pd.DataFrame(np.random.rand(10,3), columns=['Col1', 'Col2', 'Col3'])
In [43]: df['X'] = pd.Series(['A','A','A','A','A','B','B','B','B','B'])
In [44]: df['Y'] = pd.Series(['A','B','A','B','A','B','A','B','A','B'])
In [45]: plt.figure();
In [46]: bp = df.boxplot(column=['Col1','Col2'], by=['X','Y'])
648
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
Basically, plot functions return matplotlib Axes as a return value. In boxplot, the return type can be changed
by argument return_type, and whether the subplots is enabled (subplots=True in plot or by is specified in
boxplot).
When subplots=False / by is None:
• if return_type is ’dict’, a dictionary containing the matplotlib Lines is returned. The keys are “boxes”, “caps”
This is the default of boxplot in historical reason. Note that plot(kind=’box’) returns Axes as
default as the same as other plots.
• if return_type is ’axes’, a matplotlib Axes containing the boxplot is returned.
• if return_type is ’both’ a namedtuple containging the matplotlib Axes and
Lines is returned
matplotlib
When subplots=True / by is some column of the DataFrame:
• A dict of return_type is returned, where the keys are the columns of the DataFrame. The plot has a facet
for each column of the DataFrame, with a separate box for each value of by.
Finally, when calling boxplot on a Groupby object, a dict of return_type is returned, where the keys are the
same as the Groupby object. The plot has a facet for each key, with each facet containing a box for each column of the
DataFrame.
23.2. Other Plots
649
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [47]: np.random.seed(1234)
In [48]: df_box = pd.DataFrame(np.random.randn(50, 2))
In [49]: df_box['g'] = np.random.choice(['A', 'B'], size=50)
In [50]: df_box.loc[df_box['g'] == 'B', 1] += 3
In [51]: bp = df_box.boxplot(by='g')
Compare to:
In [52]: bp = df_box.groupby('g').boxplot()
650
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.2.4 Area Plot
New in version 0.14.
You can create area plots with Series.plot and DataFrame.plot by passing kind=’area’. Area plots are
stacked by default. To produce stacked area plot, each column must be either all positive or all negative values.
When input data contains NaN, it will be automatically filled by 0. If you want to drop or fill by different values, use
dataframe.dropna() or dataframe.fillna() before calling plot.
In [53]: df = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
In [54]: df.plot(kind='area');
23.2. Other Plots
651
pandas: powerful Python data analysis toolkit, Release 0.16.1
To produce an unstacked plot, pass stacked=False. Alpha value is set to 0.5 unless otherwise specified:
In [55]: df.plot(kind='area', stacked=False);
652
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.2.5 Scatter Plot
New in version 0.13.
You can create scatter plots with DataFrame.plot by passing kind=’scatter’. Scatter plot requires numeric
columns for x and y axis. These can be specified by x and y keywords each.
In [56]: df = pd.DataFrame(np.random.rand(50, 4), columns=['a', 'b', 'c', 'd'])
In [57]: df.plot(kind='scatter', x='a', y='b');
23.2. Other Plots
653
pandas: powerful Python data analysis toolkit, Release 0.16.1
To plot multiple column groups in a single axes, repeat plot method specifying target ax. It is recommended to
specify color and label keywords to distinguish each groups.
In [58]: ax = df.plot(kind='scatter', x='a', y='b',
....:
color='DarkBlue', label='Group 1');
....:
In [59]: df.plot(kind='scatter', x='c', y='d',
....:
color='DarkGreen', label='Group 2', ax=ax);
....:
654
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
The keyword c may be given as the name of a column to provide colors for each point:
In [60]: df.plot(kind='scatter', x='a', y='b', c='c', s=50);
23.2. Other Plots
655
pandas: powerful Python data analysis toolkit, Release 0.16.1
You can pass other keywords supported by matplotlib scatter. Below example shows a bubble chart using a
dataframe column values as bubble size.
In [61]: df.plot(kind='scatter', x='a', y='b', s=df['c']*200);
656
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
See the scatter method and the matplotlib scatter documenation for more.
23.2.6 Hexagonal Bin Plot
New in version 0.14.
You can create hexagonal bin plots with DataFrame.plot() and kind=’hexbin’. Hexbin plots can be a useful
alternative to scatter plots if your data are too dense to plot each point individually.
In [62]: df = pd.DataFrame(np.random.randn(1000, 2), columns=['a', 'b'])
In [63]: df['b'] = df['b'] + np.arange(1000)
In [64]: df.plot(kind='hexbin', x='a', y='b', gridsize=25)
Out[64]:
23.2. Other Plots
657
pandas: powerful Python data analysis toolkit, Release 0.16.1
A useful keyword argument is gridsize; it controls the number of hexagons in the x-direction, and defaults to 100.
A larger gridsize means more, smaller bins.
By default, a histogram of the counts around each (x, y) point is computed. You can specify alternative aggregations
by passing values to the C and reduce_C_function arguments. C specifies the value at each (x, y) point and
reduce_C_function is a function of one argument that reduces all the values in a bin to a single number (e.g.
mean, max, sum, std). In this example the positions are given by columns a and b, while the value is given by
column z. The bins are aggregated with numpy’s max function.
In [65]: df = pd.DataFrame(np.random.randn(1000, 2), columns=['a', 'b'])
In [66]: df['b'] = df['b'] = df['b'] + np.arange(1000)
In [67]: df['z'] = np.random.uniform(0, 3, 1000)
In [68]: df.plot(kind='hexbin', x='a', y='b', C='z', reduce_C_function=np.max,
....:
gridsize=25)
....:
Out[68]:
658
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
See the hexbin method and the matplotlib hexbin documenation for more.
23.2.7 Pie plot
New in version 0.14.
You can create a pie plot with DataFrame.plot() or Series.plot() with kind=’pie’. If your data includes any NaN, they will be automatically filled with 0. A ValueError will be raised if there are any negative
values in your data.
In [69]: series = pd.Series(3 * np.random.rand(4), index=['a', 'b', 'c', 'd'], name='series')
In [70]: series.plot(kind='pie', figsize=(6, 6))
Out[70]:
23.2. Other Plots
659
pandas: powerful Python data analysis toolkit, Release 0.16.1
For pie plots it’s best to use square figures, one’s with an equal aspect ratio. You can create the figure with equal
width and height, or force the aspect ratio to be equal after plotting by calling ax.set_aspect(’equal’) on the
returned axes object.
Note that pie plot with DataFrame requires that you either specify a target column by the y argument or
subplots=True. When y is specified, pie plot of selected column will be drawn. If subplots=True is specified, pie plots for each column are drawn as subplots. A legend will be drawn in each pie plots by default; specify
legend=False to hide it.
In [71]: df = pd.DataFrame(3 * np.random.rand(4, 2), index=['a', 'b', 'c', 'd'], columns=['x', 'y'])
In [72]: df.plot(kind='pie', subplots=True, figsize=(8, 4))
Out[72]:
array([,
], dtype=object)
660
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
You can use the labels and colors keywords to specify the labels and colors of each wedge.
Warning: Most pandas plots use the the label and color arguments (note the lack of “s” on those). To be
consistent with matplotlib.pyplot.pie() you must use labels and colors.
If you want to hide wedge labels, specify labels=None. If fontsize is specified, the value will be applied to
wedge labels. Also, other keywords supported by matplotlib.pyplot.pie() can be used.
In [73]: series.plot(kind='pie', labels=['AA', 'BB', 'CC', 'DD'], colors=['r', 'g', 'b', 'c'],
....:
autopct='%.2f', fontsize=20, figsize=(6, 6))
....:
Out[73]:
23.2. Other Plots
661
pandas: powerful Python data analysis toolkit, Release 0.16.1
If you pass values whose sum total is less than 1.0, matplotlib draws a semicircle.
In [74]: series = pd.Series([0.1] * 4, index=['a', 'b', 'c', 'd'], name='series2')
In [75]: series.plot(kind='pie', figsize=(6, 6))
Out[75]:
662
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
See the matplotlib pie documenation for more.
23.3 Plotting with Missing Data
Pandas tries to be pragmatic about plotting DataFrames or Series that contain missing data. Missing values are
dropped, left out, or filled depending on the plot type.
Plot Type
Line
Line (stacked)
Bar
Scatter
Histogram
Box
Area
KDE
Hexbin
Pie
NaN Handling
Leave gaps at NaNs
Fill 0’s
Fill 0’s
Drop NaNs
Drop NaNs (column-wise)
Drop NaNs (column-wise)
Fill 0’s
Drop NaNs (column-wise)
Drop NaNs
Fill 0’s
If any of these defaults are not what you want, or if you want to be explicit about how missing values are handled,
consider using fillna() or dropna() before plotting.
23.3. Plotting with Missing Data
663
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.4 Plotting Tools
These functions can be imported from pandas.tools.plotting and take a Series or DataFrame as an
argument.
23.4.1 Scatter Matrix Plot
New in version 0.7.3.
You can create a scatter plot matrix using the scatter_matrix method in pandas.tools.plotting:
In [76]: from pandas.tools.plotting import scatter_matrix
In [77]: df = pd.DataFrame(np.random.randn(1000, 4), columns=['a', 'b', 'c', 'd'])
In [78]: scatter_matrix(df, alpha=0.2, figsize=(6, 6), diagonal='kde')
Out[78]:
array([[,
,
,
],
[,
,
,
],
[,
,
,
],
[,
,
,
]], dtype=object)
664
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.4.2 Density Plot
New in version 0.8.0.
You can create density plots using the Series/DataFrame.plot and setting kind=’kde’:
In [79]: ser = pd.Series(np.random.randn(1000))
In [80]: ser.plot(kind='kde')
Out[80]:
23.4. Plotting Tools
665
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.4.3 Andrews Curves
Andrews curves allow one to plot multivariate data as a large number of curves that are created using the attributes
of samples as coefficients for Fourier series. By coloring these curves differently for each class it is possible to
visualize data clustering. Curves belonging to samples of the same class will usually be closer together and form
larger structures.
Note: The “Iris” dataset is available here.
In [81]: from pandas.tools.plotting import andrews_curves
In [82]: data = pd.read_csv('data/iris.data')
In [83]: plt.figure()
Out[83]:
In [84]: andrews_curves(data, 'Name')
Out[84]:
666
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.4.4 Parallel Coordinates
Parallel coordinates is a plotting technique for plotting multivariate data. It allows one to see clusters in data and to
estimate other statistics visually. Using parallel coordinates points are represented as connected line segments. Each
vertical line represents one attribute. One set of connected line segments represents one data point. Points that tend to
cluster will appear closer together.
In [85]: from pandas.tools.plotting import parallel_coordinates
In [86]: data = pd.read_csv('data/iris.data')
In [87]: plt.figure()
Out[87]:
In [88]: parallel_coordinates(data, 'Name')
Out[88]:
23.4. Plotting Tools
667
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.4.5 Lag Plot
Lag plots are used to check if a data set or time series is random. Random data should not exhibit any structure in the
lag plot. Non-random structure implies that the underlying data are not random.
In [89]: from pandas.tools.plotting import lag_plot
In [90]: plt.figure()
Out[90]:
In [91]: data = pd.Series(0.1 * np.random.rand(1000) +
....:
0.9 * np.sin(np.linspace(-99 * np.pi, 99 * np.pi, num=1000)))
....:
In [92]: lag_plot(data)
Out[92]:
668
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.4.6 Autocorrelation Plot
Autocorrelation plots are often used for checking randomness in time series. This is done by computing autocorrelations for data values at varying time lags. If time series is random, such autocorrelations should be near zero for any
and all time-lag separations. If time series is non-random then one or more of the autocorrelations will be significantly
non-zero. The horizontal lines displayed in the plot correspond to 95% and 99% confidence bands. The dashed line is
99% confidence band.
In [93]: from pandas.tools.plotting import autocorrelation_plot
In [94]: plt.figure()
Out[94]:
In [95]: data = pd.Series(0.7 * np.random.rand(1000) +
....:
0.3 * np.sin(np.linspace(-9 * np.pi, 9 * np.pi, num=1000)))
....:
In [96]: autocorrelation_plot(data)
Out[96]:
23.4. Plotting Tools
669
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.4.7 Bootstrap Plot
Bootstrap plots are used to visually assess the uncertainty of a statistic, such as mean, median, midrange, etc. A
random subset of a specified size is selected from a data set, the statistic in question is computed for this subset and
the process is repeated a specified number of times. Resulting plots and histograms are what constitutes the bootstrap
plot.
In [97]: from pandas.tools.plotting import bootstrap_plot
In [98]: data = pd.Series(np.random.rand(1000))
In [99]: bootstrap_plot(data, size=50, samples=500, color='grey')
Out[99]:
670
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.4.8 RadViz
RadViz is a way of visualizing multi-variate data. It is based on a simple spring tension minimization algorithm.
Basically you set up a bunch of points in a plane. In our case they are equally spaced on a unit circle. Each point
represents a single attribute. You then pretend that each sample in the data set is attached to each of these points
by a spring, the stiffness of which is proportional to the numerical value of that attribute (they are normalized to
unit interval). The point in the plane, where our sample settles to (where the forces acting on our sample are at an
equilibrium) is where a dot representing our sample will be drawn. Depending on which class that sample belongs it
will be colored differently.
Note: The “Iris” dataset is available here.
In [100]: from pandas.tools.plotting import radviz
In [101]: data = pd.read_csv('data/iris.data')
In [102]: plt.figure()
Out[102]:
In [103]: radviz(data, 'Name')
Out[103]:
23.4. Plotting Tools
671
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.5 Plot Formatting
Most plotting methods have a set of keyword arguments that control the layout and formatting of the returned plot:
In [104]: plt.figure(); ts.plot(style='k--', label='Series');
672
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
For each kind of plot (e.g. line, bar, scatter) any additional arguments keywords are passed along to the corresponding
matplotlib function (ax.plot(), ax.bar(), ax.scatter()). These can be used to control additional styling,
beyond what pandas provides.
23.5.1 Controlling the Legend
You may set the legend argument to False to hide the legend, which is shown by default.
In [105]: df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index, columns=list('ABCD'))
In [106]: df = df.cumsum()
In [107]: df.plot(legend=False)
Out[107]:
23.5. Plot Formatting
673
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.5.2 Scales
You may pass logy to get a log-scale Y axis.
In [108]: ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
In [109]: ts = np.exp(ts.cumsum())
In [110]: ts.plot(logy=True)
Out[110]:
674
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
See also the logx and loglog keyword arguments.
23.5.3 Plotting on a Secondary Y-axis
To plot data on a secondary y-axis, use the secondary_y keyword:
In [111]: df.A.plot()
Out[111]:
In [112]: df.B.plot(secondary_y=True, style='g')
Out[112]:
23.5. Plot Formatting
675
pandas: powerful Python data analysis toolkit, Release 0.16.1
To plot some columns in a DataFrame, give the column names to the secondary_y keyword:
In [113]: plt.figure()
Out[113]:
In [114]: ax = df.plot(secondary_y=['A', 'B'])
In [115]: ax.set_ylabel('CD scale')
Out[115]:
In [116]: ax.right_ax.set_ylabel('AB scale')
Out[116]:
676
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
Note that the columns plotted on the secondary y-axis is automatically marked with “(right)” in the legend. To turn off
the automatic marking, use the mark_right=False keyword:
In [117]: plt.figure()
Out[117]:
In [118]: df.plot(secondary_y=['A', 'B'], mark_right=False)
Out[118]:
23.5. Plot Formatting
677
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.5.4 Suppressing Tick Resolution Adjustment
pandas includes automatic tick resolution adjustment for regular frequency time-series data. For limited cases where
pandas cannot infer the frequency information (e.g., in an externally created twinx), you can choose to suppress this
behavior for alignment purposes.
Here is the default behavior, notice how the x-axis tick labelling is performed:
In [119]: plt.figure()
Out[119]:
In [120]: df.A.plot()
Out[120]:
678
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
Using the x_compat parameter, you can suppress this behavior:
In [121]: plt.figure()
Out[121]:
In [122]: df.A.plot(x_compat=True)
Out[122]:
23.5. Plot Formatting
679
pandas: powerful Python data analysis toolkit, Release 0.16.1
If you have more than one plot that needs to be suppressed, the use method in pandas.plot_params can be used
in a with statement:
In [123]: plt.figure()
Out[123]:
In [124]: with pd.plot_params.use('x_compat', True):
.....:
df.A.plot(color='r')
.....:
df.B.plot(color='g')
.....:
df.C.plot(color='b')
.....:
680
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.5.5 Subplots
Each Series in a DataFrame can be plotted on a different axis with the subplots keyword:
In [125]: df.plot(subplots=True, figsize=(6, 6));
23.5. Plot Formatting
681
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.5.6 Using Layout and Targetting Multiple Axes
The layout of subplots can be specified by layout keyword. It can accept (rows, columns). The layout
keyword can be used in hist and boxplot also. If input is invalid, ValueError will be raised.
The number of axes which can be contained by rows x columns specified by layout must be larger than the number
of required subplots. If layout can contain more axes than required, blank axes are not drawn. Similar to a numpy
array’s reshape method, you can use -1 for one dimension to automatically calculate the number of rows or columns
needed, given the other.
In [126]: df.plot(subplots=True, layout=(2, 3), figsize=(6, 6), sharex=False);
682
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
The above example is identical to using
In [127]: df.plot(subplots=True, layout=(2, -1), figsize=(6, 6), sharex=False);
The required number of columns (3) is inferred from the number of series to plot and the given number of rows (2).
Also, you can pass multiple axes created beforehand as list-like via ax keyword. This allows to use more complicated
layout. The passed axes must be the same number as the subplots being drawn.
When multiple axes are passed via ax keyword, layout, sharex and sharey keywords don’t affect to the output.
You should explicitly pass sharex=False and sharey=False, otherwise you will see a warning.
In [128]: fig, axes = plt.subplots(4, 4, figsize=(6, 6));
In [129]: plt.subplots_adjust(wspace=0.5, hspace=0.5);
In [130]: target1 = [axes[0][0], axes[1][1], axes[2][2], axes[3][3]]
In [131]: target2 = [axes[3][0], axes[2][1], axes[1][2], axes[0][3]]
In [132]: df.plot(subplots=True, ax=target1, legend=False, sharex=False, sharey=False);
In [133]: (-df).plot(subplots=True, ax=target2, legend=False, sharex=False, sharey=False);
23.5. Plot Formatting
683
pandas: powerful Python data analysis toolkit, Release 0.16.1
Another option is passing an ax argument to Series.plot() to plot on a particular axis:
In [134]: fig, axes = plt.subplots(nrows=2, ncols=2)
In [135]: df['A'].plot(ax=axes[0,0]); axes[0,0].set_title('A');
In [136]: df['B'].plot(ax=axes[0,1]); axes[0,1].set_title('B');
In [137]: df['C'].plot(ax=axes[1,0]); axes[1,0].set_title('C');
In [138]: df['D'].plot(ax=axes[1,1]); axes[1,1].set_title('D');
684
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.5.7 Plotting With Error Bars
New in version 0.14.
Plotting with error bars is now supported in the DataFrame.plot() and Series.plot()
Horizontal and vertical errorbars can be supplied to the xerr and yerr keyword arguments to plot(). The error
values can be specified using a variety of formats.
• As a DataFrame or dict of errors with column names matching the columns attribute of the plotting
DataFrame or matching the name attribute of the Series
• As a str indicating which of the columns of plotting DataFrame contain the error values
• As raw values (list, tuple, or np.ndarray).
DataFrame/Series
Must be the same length as the plotting
Asymmetrical error bars are also supported, however raw error values must be provided in this case. For a M
length Series, a Mx2 array should be provided indicating lower and upper (or left and right) errors. For a MxN
DataFrame, asymmetrical errors should be in a Mx2xN array.
Here is an example of one way to easily plot group means with standard deviations from the raw data.
23.5. Plot Formatting
685
pandas: powerful Python data analysis toolkit, Release 0.16.1
# Generate the data
In [139]: ix3 = pd.MultiIndex.from_arrays([['a', 'a', 'a', 'a', 'b', 'b', 'b', 'b'], ['foo', 'foo', '
In [140]: df3 = pd.DataFrame({'data1': [3, 2, 4, 3, 2, 4, 3, 2], 'data2': [6, 5, 7, 5, 4, 5, 6, 5]},
# Group by index labels and take the means and standard deviations for each group
In [141]: gp3 = df3.groupby(level=('letter', 'word'))
In [142]: means = gp3.mean()
In [143]: errors = gp3.std()
In [144]: means
Out[144]:
data1
letter word
a
bar
3.5
foo
2.5
b
bar
2.5
foo
3.0
data2
In [145]: errors
Out[145]:
data1
letter word
a
bar
0.707107
foo
0.707107
b
bar
0.707107
foo
1.414214
6.0
5.5
5.5
4.5
data2
1.414214
0.707107
0.707107
0.707107
# Plot
In [146]: fig, ax = plt.subplots()
In [147]: means.plot(yerr=errors, ax=ax, kind='bar')
Out[147]:
686
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.5.8 Plotting Tables
New in version 0.14.
Plotting with matplotlib table is now supported in DataFrame.plot() and Series.plot() with a table
keyword. The table keyword can accept bool, DataFrame or Series. The simple way to draw a table is to
specify table=True. Data will be transposed to meet matplotlib’s default layout.
In [148]: fig, ax = plt.subplots(1, 1)
In [149]: df = pd.DataFrame(np.random.rand(5, 3), columns=['a', 'b', 'c'])
In [150]: ax.get_xaxis().set_visible(False)
# Hide Ticks
In [151]: df.plot(table=True, ax=ax)
Out[151]:
23.5. Plot Formatting
687
pandas: powerful Python data analysis toolkit, Release 0.16.1
Also, you can pass different DataFrame or Series for table keyword. The data will be drawn as displayed in
print method (not transposed automatically). If required, it should be transposed manually as below example.
In [152]: fig, ax = plt.subplots(1, 1)
In [153]: ax.get_xaxis().set_visible(False)
# Hide Ticks
In [154]: df.plot(table=np.round(df.T, 2), ax=ax)
Out[154]:
688
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
Finally, there is a helper function pandas.tools.plotting.table to create a table from DataFrame and
Series, and add it to an matplotlib.Axes. This function can accept keywords which matplotlib table has.
In [155]: from pandas.tools.plotting import table
In [156]: fig, ax = plt.subplots(1, 1)
In [157]: table(ax, np.round(df.describe(), 2),
.....:
loc='upper right', colWidths=[0.2, 0.2, 0.2])
.....:
Out[157]:
In [158]: df.plot(ax=ax, ylim=(0, 2), legend=None)
Out[158]:
23.5. Plot Formatting
689
pandas: powerful Python data analysis toolkit, Release 0.16.1
Note: You can get table instances on the axes using axes.tables property for further decorations. See the matplotlib table documenation for more.
23.5.9 Colormaps
A potential issue when plotting a large number of columns is that it can be difficult to distinguish some series due to
repetition in the default colors. To remedy this, DataFrame plotting supports the use of the colormap= argument,
which accepts either a Matplotlib colormap or a string that is a name of a colormap registered with Matplotlib. A
visualization of the default matplotlib colormaps is available here.
As matplotlib does not directly support colormaps for line-based plots, the colors are selected based on an even spacing
determined by the number of columns in the DataFrame. There is no consideration made for background color, so
some colormaps will produce lines that are not easily visible.
To use the cubehelix colormap, we can simply pass ’cubehelix’ to colormap=
In [159]: df = pd.DataFrame(np.random.randn(1000, 10), index=ts.index)
In [160]: df = df.cumsum()
In [161]: plt.figure()
Out[161]:
690
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [162]: df.plot(colormap='cubehelix')
Out[162]:
or we can pass the colormap itself
In [163]: from matplotlib import cm
In [164]: plt.figure()
Out[164]:
In [165]: df.plot(colormap=cm.cubehelix)
Out[165]:
23.5. Plot Formatting
691
pandas: powerful Python data analysis toolkit, Release 0.16.1
Colormaps can also be used other plot types, like bar charts:
In [166]: dd = pd.DataFrame(np.random.randn(10, 10)).applymap(abs)
In [167]: dd = dd.cumsum()
In [168]: plt.figure()
Out[168]:
In [169]: dd.plot(kind='bar', colormap='Greens')
Out[169]:
692
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
Parallel coordinates charts:
In [170]: plt.figure()
Out[170]:
In [171]: parallel_coordinates(data, 'Name', colormap='gist_rainbow')
Out[171]:
23.5. Plot Formatting
693
pandas: powerful Python data analysis toolkit, Release 0.16.1
Andrews curves charts:
In [172]: plt.figure()
Out[172]:
In [173]: andrews_curves(data, 'Name', colormap='winter')
Out[173]:
694
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
23.6 Plotting directly with matplotlib
In some situations it may still be preferable or necessary to prepare plots directly with matplotlib, for instance when
a certain type of plot or customization is not (yet) supported by pandas. Series and DataFrame objects behave like
arrays and can therefore be passed directly to matplotlib functions without explicit casts.
pandas also automatically registers formatters and locators that recognize date indices, thereby extending date and
time support to practically all plot types available in matplotlib. Although this formatting does not provide the same
level of refinement you would get when plotting via pandas, it can be faster when plotting a large number of points.
Note: The speed up for large data sets only applies to pandas 0.14.0 and later.
In [174]: price = pd.Series(np.random.randn(150).cumsum(),
.....:
index=pd.date_range('2000-1-1', periods=150, freq='B'))
.....:
In [175]: ma = pd.rolling_mean(price, 20)
In [176]: mstd = pd.rolling_std(price, 20)
In [177]: plt.figure()
23.6. Plotting directly with matplotlib
695
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[177]:
In [178]: plt.plot(price.index, price, 'k')
Out[178]: []
In [179]: plt.plot(ma.index, ma, 'b')
Out[179]: []
In [180]: plt.fill_between(mstd.index, ma-2*mstd, ma+2*mstd, color='b', alpha=0.2)
Out[180]:
23.7 Trellis plotting interface
Warning: The rplot trellis plotting interface is deprecated and will be removed in a future version. We refer
to external packages like seaborn for similar but more refined functionality.
The docs below include some example on how to convert your existing code to seaborn.
Note: The tips data set can be downloaded here. Once you download it execute
696
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
tips_data = pd.read_csv('tips.csv')
from the directory where you downloaded the file.
We import the rplot API:
In [181]: import pandas.tools.rplot as rplot
23.7.1 Examples
RPlot was an API for producing Trellis plots. These plots allow you toµ arrange data in a rectangular grid by values
of certain attributes. In the example below, data from the tips data set is arranged by the attributes ‘sex’ and ‘smoker’.
Since both of those attributes can take on one of two values, the resulting grid has two columns and two rows. A
histogram is displayed for each cell of the grid.
In [182]: plt.figure()
Out[182]:
In [183]: plot = rplot.RPlot(tips_data, x='total_bill', y='tip')
In [184]: plot.add(rplot.TrellisGrid(['sex', 'smoker']))
In [185]: plot.add(rplot.GeomHistogram())
In [186]: plot.render(plt.gcf())
Out[186]:
23.7. Trellis plotting interface
697
pandas: powerful Python data analysis toolkit, Release 0.16.1
A similar plot can be made with seaborn using the FacetGrid object, resulting in the following image:
import seaborn as sns
g = sns.FacetGrid(tips_data, row="sex", col="smoker")
g.map(plt.hist, "total_bill")
698
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
Example below is the same as previous except the plot is set to kernel density estimation. A seaborn example is
included beneath.
In [187]: plt.figure()
Out[187]:
In [188]: plot = rplot.RPlot(tips_data, x='total_bill', y='tip')
In [189]: plot.add(rplot.TrellisGrid(['sex', 'smoker']))
In [190]: plot.add(rplot.GeomDensity())
In [191]: plot.render(plt.gcf())
Out[191]:
23.7. Trellis plotting interface
699
pandas: powerful Python data analysis toolkit, Release 0.16.1
g = sns.FacetGrid(tips_data, row="sex", col="smoker")
g.map(sns.kdeplot, "total_bill")
700
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
The plot below shows that it is possible to have two or more plots for the same data displayed on the same Trellis grid
cell.
In [192]: plt.figure()
Out[192]:
In [193]: plot = rplot.RPlot(tips_data, x='total_bill', y='tip')
In [194]: plot.add(rplot.TrellisGrid(['sex', 'smoker']))
In [195]: plot.add(rplot.GeomScatter())
In [196]: plot.add(rplot.GeomPolyFit(degree=2))
In [197]: plot.render(plt.gcf())
Out[197]:
23.7. Trellis plotting interface
701
pandas: powerful Python data analysis toolkit, Release 0.16.1
A seaborn equivalent for a simple scatter plot:
g = sns.FacetGrid(tips_data, row="sex", col="smoker")
g.map(plt.scatter, "total_bill", "tip")
702
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
and with a regression line, using the dedicated seaborn regplot function:
g = sns.FacetGrid(tips_data, row="sex", col="smoker", margin_titles=True)
g.map(sns.regplot, "total_bill", "tip", order=2)
23.7. Trellis plotting interface
703
pandas: powerful Python data analysis toolkit, Release 0.16.1
Below is a similar plot but with 2D kernel density estimation plot superimposed, followed by a seaborn equivalent:
In [198]: plt.figure()
Out[198]:
In [199]: plot = rplot.RPlot(tips_data, x='total_bill', y='tip')
In [200]: plot.add(rplot.TrellisGrid(['sex', 'smoker']))
In [201]: plot.add(rplot.GeomScatter())
In [202]: plot.add(rplot.GeomDensity2D())
In [203]: plot.render(plt.gcf())
Out[203]:
704
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
g = sns.FacetGrid(tips_data, row="sex", col="smoker")
g.map(plt.scatter, "total_bill", "tip")
g.map(sns.kdeplot, "total_bill", "tip")
23.7. Trellis plotting interface
705
pandas: powerful Python data analysis toolkit, Release 0.16.1
It is possible to only use one attribute for grouping data. The example above only uses ‘sex’ attribute. If the second
grouping attribute is not specified, the plots will be arranged in a column.
In [204]: plt.figure()
Out[204]:
In [205]: plot = rplot.RPlot(tips_data, x='total_bill', y='tip')
In [206]: plot.add(rplot.TrellisGrid(['sex', '.']))
In [207]: plot.add(rplot.GeomHistogram())
In [208]: plot.render(plt.gcf())
Out[208]:
706
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
If the first grouping attribute is not specified the plots will be arranged in a row.
In [209]: plt.figure()
Out[209]:
In [210]: plot = rplot.RPlot(tips_data, x='total_bill', y='tip')
In [211]: plot.add(rplot.TrellisGrid(['.', 'smoker']))
In [212]: plot.add(rplot.GeomHistogram())
In [213]: plot.render(plt.gcf())
Out[213]:
23.7. Trellis plotting interface
707
pandas: powerful Python data analysis toolkit, Release 0.16.1
In seaborn, this can also be done by only specifying one of the row and col arguments.
In the example below the colour and shape of the scatter plot graphical objects is mapped to ‘day’ and ‘size’ attributes
respectively. You use scale objects to specify these mappings. The list of scale classes is given below with initialization
arguments for quick reference.
In [214]: plt.figure()
Out[214]:
In [215]: plot = rplot.RPlot(tips_data, x='tip', y='total_bill')
In [216]: plot.add(rplot.TrellisGrid(['sex', 'smoker']))
In [217]: plot.add(rplot.GeomPoint(size=80.0, colour=rplot.ScaleRandomColour('day'), shape=rplot.Scal
In [218]: plot.render(plt.gcf())
Out[218]:
708
Chapter 23. Plotting
pandas: powerful Python data analysis toolkit, Release 0.16.1
This can also be done in seaborn, at least for 3 variables:
g = sns.FacetGrid(tips_data, row="sex", col="smoker", hue="day")
g.map(plt.scatter, "tip", "total_bill")
g.add_legend()
23.7. Trellis plotting interface
709
pandas: powerful Python data analysis toolkit, Release 0.16.1
710
Chapter 23. Plotting
CHAPTER
TWENTYFOUR
IO TOOLS (TEXT, CSV, HDF5, ...)
The pandas I/O API is a set of top level reader functions accessed like pd.read_csv() that generally return a
pandas object.
• read_csv
• read_excel
• read_hdf
• read_sql
• read_json
• read_msgpack (experimental)
• read_html
• read_gbq (experimental)
• read_stata
• read_clipboard
• read_pickle
The corresponding writer functions are object methods that are accessed like df.to_csv()
• to_csv
• to_excel
• to_hdf
• to_sql
• to_json
• to_msgpack (experimental)
• to_html
• to_gbq (experimental)
• to_stata
• to_clipboard
• to_pickle
Here is an informal performance comparison for some of these IO methods.
Note: For examples that use the StringIO class, make sure you import it according to your Python version, i.e.
from StringIO import StringIO for Python 2 and from io import StringIO for Python 3.
711
pandas: powerful Python data analysis toolkit, Release 0.16.1
24.1 CSV & Text files
The two workhorse functions for reading text files (a.k.a. flat files) are read_csv() and read_table(). They
both use the same parsing code to intelligently convert tabular data into a DataFrame object. See the cookbook for
some advanced strategies
They can take a number of arguments:
• filepath_or_buffer: Either a string path to a file, URL (including http, ftp, and S3 locations), or any
object with a read method (such as an open file or StringIO).
• sep or delimiter: A delimiter / separator to split fields on. With sep=None, read_csv will try to infer
the delimiter automatically in some cases by “sniffing”. The separator may be specified as a regular expression;
for instance you may use ‘|\s*’ to indicate a pipe plus arbitrary whitespace.
• delim_whitespace: Parse whitespace-delimited (spaces or tabs) file (much faster than using a regular
expression)
• compression: decompress ’gzip’ and ’bz2’ formats on the fly. Set to ’infer’ (the default) to guess a
format based on the file extension.
• dialect: string or csv.Dialect instance to expose more ways to specify the file format
• dtype: A data type name or a dict of column name to data type. If not specified, data types will be inferred.
(Unsupported with engine=’python’)
• header: row number(s) to use as the column names, and the start of the data. Defaults to 0 if no names
passed, otherwise None. Explicitly pass header=0 to be able to replace existing names. The header can be a
list of integers that specify row locations for a multi-index on the columns E.g. [0,1,3]. Intervening rows that are
not specified will be skipped (e.g. 2 in this example are skipped). Note that this parameter ignores commented
lines and empty lines if skip_blank_lines=True (the default), so header=0 denotes the first line of data
rather than the first line of the file.
• skip_blank_lines: whether to skip over blank lines rather than interpreting them as NaN values
• skiprows: A collection of numbers for rows in the file to skip. Can also be an integer to skip the first n rows
• index_col: column number, column name, or list of column numbers/names, to use as the index (row
labels) of the resulting DataFrame. By default, it will number the rows without using any column, unless there
is one more data column than there are headers, in which case the first column is taken as the index.
• names: List of column names to use as column names. To replace header existing in file, explicitly pass
header=0.
• na_values: optional list of strings to recognize as NaN (missing values), either in addition to or in lieu of the
default set.
• true_values: list of strings to recognize as True
• false_values: list of strings to recognize as False
• keep_default_na: whether to include the default set of missing values in addition to the ones specified in
na_values
• parse_dates: if True then index will be parsed as dates (False by default). You can specify more complicated
options to parse a subset of columns or a combination of columns into a single date column (list of ints or names,
list of lists, or dict) [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column [[1, 3]] -> combine
columns 1 and 3 and parse as a single date column {‘foo’ : [1, 3]} -> parse columns 1, 3 as date and call result
‘foo’
712
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
• keep_date_col: if True, then date component columns passed into parse_dates will be retained in the
output (False by default).
• date_parser: function to use to parse strings into datetime objects. If parse_dates is True, it defaults
to the very robust dateutil.parser. Specifying this implicitly sets parse_dates as True. You can also
use functions from community supported date converters from date_converters.py
• dayfirst: if True then uses the DD/MM international/European date format (This is False by default)
• thousands: specifies the thousands separator. If not None, this character will be stripped from numeric
dtypes. However, if it is the first character in a field, that column will be imported as a string. In the PythonParser,
if not None, then parser will try to look for it in the output and parse relevant data to numeric dtypes. Because it
has to essentially scan through the data again, this causes a significant performance hit so only use if necessary.
• lineterminator : string (length 1), default None, Character to break file into lines. Only valid with C
parser
• quotechar : string, The character to used to denote the start and end of a quoted item. Quoted items can
include the delimiter and it will be ignored.
• quoting : int, Controls whether quotes should be recognized. Values are taken from csv.QUOTE_* values. Acceptable values are 0, 1, 2, and 3 for QUOTE_MINIMAL, QUOTE_ALL, QUOTE_NONE, and
QUOTE_NONNUMERIC, respectively.
• skipinitialspace : boolean, default False, Skip spaces after delimiter
• escapechar : string, to specify how to escape quoted data
• comment: Indicates remainder of line should not be parsed. If found at the beginning of a line, the line will be
ignored altogether. This parameter must be a single character. Like empty lines, fully commented lines are ignored by the parameter header but not by skiprows. For example, if comment=’#’, parsing ‘#emptyn1,2,3na,b,c’
with header=0 will result in ‘1,2,3’ being treated as the header.
• nrows: Number of rows to read out of the file. Useful to only read a small portion of a large file
• iterator: If True, return a TextFileReader to enable reading a file into memory piece by piece
• chunksize: An number of rows to be used to “chunk” a file into pieces. Will cause an TextFileReader
object to be returned. More on this below in the section on iterating and chunking
• skip_footer: number of lines to skip at bottom of file (default 0) (Unsupported with engine=’c’)
• converters: a dictionary of functions for converting values in certain columns, where keys are either integers
or column labels
• encoding: a string representing the encoding to use for decoding unicode data, e.g.
’latin-1’. Full list of Python standard encodings
’utf-8‘ or
• verbose: show number of NA values inserted in non-numeric columns
• squeeze: if True then output with only one column is turned into Series
• error_bad_lines: if False then any lines causing an error will be skipped bad lines
• usecols: a subset of columns to return, results in much faster parsing time and lower memory usage.
• mangle_dupe_cols: boolean, default True, then duplicate columns will be specified as ‘X.0’...’X.N’, rather
than ‘X’...’X’
• tupleize_cols: boolean, default False, if False, convert a list of tuples to a multi-index of columns, otherwise, leave the column index as a list of tuples
• float_precision : string, default None. Specifies which converter the C engine should use for floatingpoint values. The options are None for the ordinary converter, ‘high’ for the high-precision converter, and
‘round_trip’ for the round-trip converter.
24.1. CSV & Text files
713
pandas: powerful Python data analysis toolkit, Release 0.16.1
Consider a typical CSV file containing, in this case, some time series data:
In [1]: print(open('foo.csv').read())
date,A,B,C
20090101,a,1,2
20090102,b,3,4
20090103,c,4,5
The default for read_csv is to create a DataFrame with simple numbered rows:
In [2]: pd.read_csv('foo.csv')
Out[2]:
date A B C
0 20090101 a 1 2
1 20090102 b 3 4
2 20090103 c 4 5
In the case of indexed data, you can pass the column number or column name you wish to use as the index:
In [3]: pd.read_csv('foo.csv', index_col=0)
Out[3]:
A B C
date
20090101 a 1 2
20090102 b 3 4
20090103 c 4 5
In [4]: pd.read_csv('foo.csv', index_col='date')
Out[4]:
A B C
date
20090101 a 1 2
20090102 b 3 4
20090103 c 4 5
You can also use a list of columns to create a hierarchical index:
In [5]: pd.read_csv('foo.csv', index_col=[0, 'A'])
Out[5]:
B C
date
A
20090101 a 1 2
20090102 b 3 4
20090103 c 4 5
The dialect keyword gives greater flexibility in specifying the file format. By default it uses the Excel dialect but
you can specify either the dialect name or a csv.Dialect instance.
Suppose you had data with unenclosed quotes:
In [6]: print(data)
label1,label2,label3
index1,"a,c,e
index2,b,d,f
By default, read_csv uses the Excel dialect and treats the double quote as the quote character, which causes it to
fail when it finds a newline before it finds the closing double quote.
We can get around this using dialect
714
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [7]: dia = csv.excel()
In [8]: dia.quoting = csv.QUOTE_NONE
In [9]: pd.read_csv(StringIO(data), dialect=dia)
Out[9]:
label1 label2 label3
index1
"a
c
e
index2
b
d
f
All of the dialect options can be specified separately by keyword arguments:
In [10]: data = 'a,b,c~1,2,3~4,5,6'
In [11]:
Out[11]:
a b
0 1 2
1 4 5
pd.read_csv(StringIO(data), lineterminator='~')
c
3
6
Another common dialect option is skipinitialspace, to skip any whitespace after a delimiter:
In [12]: data = 'a, b, c\n1, 2, 3\n4, 5, 6'
In
a,
1,
4,
[13]: print(data)
b, c
2, 3
5, 6
In [14]:
Out[14]:
a b
0 1 2
1 4 5
pd.read_csv(StringIO(data), skipinitialspace=True)
c
3
6
The parsers make every attempt to “do the right thing” and not be very fragile. Type inference is a pretty big deal. So
if a column can be coerced to integer dtype without altering the contents, it will do so. Any non-numeric columns will
come through as object dtype as with the rest of pandas objects.
24.1.1 Specifying column data types
Starting with v0.10, you can indicate the data type for the whole DataFrame or individual columns:
In [15]: data = 'a,b,c\n1,2,3\n4,5,6\n7,8,9'
In [16]: print(data)
a,b,c
1,2,3
4,5,6
7,8,9
In [17]: df = pd.read_csv(StringIO(data), dtype=object)
In [18]:
Out[18]:
a b
0 1 2
1 4 5
df
c
3
6
24.1. CSV & Text files
715
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
7
8
9
In [19]: df['a'][0]
Out[19]: '1'
In [20]: df = pd.read_csv(StringIO(data), dtype={'b': object, 'c': np.float64})
In [21]: df.dtypes
Out[21]:
a
int64
b
object
c
float64
dtype: object
Note: The dtype option is currently only supported by the C engine. Specifying dtype with engine other than
‘c’ raises a ValueError.
24.1.2 Handling column names
A file may or may not have a header row. pandas assumes the first row should be used as the column names:
In [22]: data = 'a,b,c\n1,2,3\n4,5,6\n7,8,9'
In [23]: print(data)
a,b,c
1,2,3
4,5,6
7,8,9
In [24]:
Out[24]:
a b
0 1 2
1 4 5
2 7 8
pd.read_csv(StringIO(data))
c
3
6
9
By specifying the names argument in conjunction with header you can indicate other names to use and whether or
not to throw away the header row (if any):
In [25]: print(data)
a,b,c
1,2,3
4,5,6
7,8,9
In [26]: pd.read_csv(StringIO(data), names=['foo', 'bar', 'baz'], header=0)
Out[26]:
foo bar baz
0
1
2
3
1
4
5
6
2
7
8
9
In [27]: pd.read_csv(StringIO(data), names=['foo', 'bar', 'baz'], header=None)
Out[27]:
foo bar baz
0
a
b
c
716
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
1
2
3
1
4
7
2
5
8
3
6
9
If the header is in a row other than the first, pass the row number to header. This will skip the preceding rows:
In [28]: data = 'skip this skip it\na,b,c\n1,2,3\n4,5,6\n7,8,9'
In [29]:
Out[29]:
a b
0 1 2
1 4 5
2 7 8
pd.read_csv(StringIO(data), header=1)
c
3
6
9
24.1.3 Filtering columns (usecols)
The usecols argument allows you to select any subset of the columns in a file, either using the column names or
position numbers:
In [30]: data = 'a,b,c,d\n1,2,3,foo\n4,5,6,bar\n7,8,9,baz'
In [31]:
Out[31]:
a b
0 1 2
1 4 5
2 7 8
pd.read_csv(StringIO(data))
c
3
6
9
d
foo
bar
baz
In [32]: pd.read_csv(StringIO(data), usecols=['b', 'd'])
Out[32]:
b
d
0 2 foo
1 5 bar
2 8 baz
In [33]:
Out[33]:
a c
0 1 3
1 4 6
2 7 9
pd.read_csv(StringIO(data), usecols=[0, 2, 3])
d
foo
bar
baz
24.1.4 Ignoring line comments and empty lines
If the comment parameter is specified, then completely commented lines will be ignored. By default, completely
blank lines will be ignored as well. Both of these are API changes introduced in version 0.15.
In [34]: data = '\na,b,c\n
\n# commented line\n1,2,3\n\n4,5,6'
In [35]: print(data)
a,b,c
1,2,3
24.1. CSV & Text files
717
pandas: powerful Python data analysis toolkit, Release 0.16.1
4,5,6
# commented line
In [36]: pd.read_csv(StringIO(data), comment='#')
Out[36]:
a b c
0 1 2 3
1 4 5 6
If skip_blank_lines=False, then read_csv will not ignore blank lines:
In [37]: data = 'a,b,c\n\n1,2,3\n\n\n4,5,6'
In [38]: pd.read_csv(StringIO(data), skip_blank_lines=False)
Out[38]:
a
b
c
0 NaN NaN NaN
1
1
2
3
2 NaN NaN NaN
3 NaN NaN NaN
4
4
5
6
718
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning: The presence of ignored lines might create ambiguities involving line numbers; the parameter header
uses row numbers (ignoring commented/empty lines), while skiprows uses line numbers (including commented/empty lines):
In [39]: data = '#comment\na,b,c\nA,B,C\n1,2,3'
In [40]: pd.read_csv(StringIO(data), comment='#', header=1)
Out[40]:
A B C
0 1 2 3
In [41]: data = 'A,B,C\n#comment\na,b,c\n1,2,3'
In [42]: pd.read_csv(StringIO(data), comment='#', skiprows=2)
Out[42]:
a b c
0 1 2 3
If both header and skiprows are specified, header will be relative to the end of skiprows. For example:
In [43]: data = '# empty\n# second empty line\n# third empty' \
In [43]: 'line\nX,Y,Z\n1,2,3\nA,B,C\n1,2.,4.\n5.,NaN,10.0'
In [44]: print(data)
# empty
# second empty line
# third emptyline
X,Y,Z
1,2,3
A,B,C
1,2.,4.
5.,NaN,10.0
In [45]: pd.read_csv(StringIO(data), comment='#', skiprows=4, header=1)
Out[45]:
A
B
C
0 1
2
4
1 5 NaN 10
24.1.5 Dealing with Unicode Data
The encoding argument should be used for encoded unicode data, which will result in byte strings being decoded
to unicode in the result:
In [46]: data = b'word,length\nTr\xc3\xa4umen,7\nGr\xc3\xbc\xc3\x9fe,5'.decode('utf8').encode('latinIn [47]: df = pd.read_csv(BytesIO(data), encoding='latin-1')
In [48]: df
Out[48]:
word length
0 Träumen
7
1
Grüße
5
In [49]: df['word'][1]
24.1. CSV & Text files
719
pandas: powerful Python data analysis toolkit, Release 0.16.1
Out[49]: u'Gr\xfc\xdfe'
Some formats which encode all characters as multiple bytes, like UTF-16, won’t parse correctly at all without specifying the encoding. Full list of Python standard encodings
24.1.6 Index columns and trailing delimiters
If a file has one more column of data than the number of column names, the first column will be used as the
DataFrame’s row names:
In [50]: data = 'a,b,c\n4,apple,bat,5.7\n8,orange,cow,10'
In [51]: pd.read_csv(StringIO(data))
Out[51]:
a
b
c
4
apple bat
5.7
8 orange cow 10.0
In [52]: data = 'index,a,b,c\n4,apple,bat,5.7\n8,orange,cow,10'
In [53]: pd.read_csv(StringIO(data), index_col=0)
Out[53]:
a
b
c
index
4
apple bat
5.7
8
orange cow 10.0
Ordinarily, you can achieve this behavior using the index_col option.
There are some exception cases when a file has been prepared with delimiters at the end of each data line, confusing
the parser. To explicitly disable the index column inference and discard the last column, pass index_col=False:
In [54]: data = 'a,b,c\n4,apple,bat,\n8,orange,cow,'
In [55]: print(data)
a,b,c
4,apple,bat,
8,orange,cow,
In [56]: pd.read_csv(StringIO(data))
Out[56]:
a
b
c
4
apple bat NaN
8 orange cow NaN
In [57]: pd.read_csv(StringIO(data), index_col=False)
Out[57]:
a
b
c
0 4
apple bat
1 8 orange cow
24.1.7 Specifying Date Columns
To better facilitate working with datetime data, read_csv() and read_table() uses the keyword arguments
parse_dates and date_parser to allow users to specify a variety of columns and date/time formats to turn the
input text data into datetime objects.
720
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
The simplest case is to just pass in parse_dates=True:
# Use a column as an index, and parse it as dates.
In [58]: df = pd.read_csv('foo.csv', index_col=0, parse_dates=True)
In [59]: df
Out[59]:
A
B
C
a
b
c
1
3
4
2
4
5
date
2009-01-01
2009-01-02
2009-01-03
# These are python datetime objects
In [60]: df.index
Out[60]: DatetimeIndex(['2009-01-01', '2009-01-02', '2009-01-03'], dtype='datetime64[ns]', name=u'dat
It is often the case that we may want to store date and time data separately, or store various date fields separately. the
parse_dates keyword can be used to specify a combination of columns to parse the dates and/or times from.
You can specify a list of column lists to parse_dates, the resulting date columns will be prepended to the output
(so as to not affect the existing column order) and the new column names will be the concatenation of the component
column names:
In [61]: print(open('tmp.csv').read())
KORD,19990127, 19:00:00, 18:56:00, 0.8100
KORD,19990127, 20:00:00, 19:56:00, 0.0100
KORD,19990127, 21:00:00, 20:56:00, -0.5900
KORD,19990127, 21:00:00, 21:18:00, -0.9900
KORD,19990127, 22:00:00, 21:56:00, -0.5900
KORD,19990127, 23:00:00, 22:56:00, -0.5900
In [62]: df = pd.read_csv('tmp.csv', header=None, parse_dates=[[1, 2], [1, 3]])
In [63]: df
Out[63]:
0
1
2
3
4
5
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1_2
19:00:00
20:00:00
21:00:00
21:00:00
22:00:00
23:00:00
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1_3
18:56:00
19:56:00
20:56:00
21:18:00
21:56:00
22:56:00
0
KORD
KORD
KORD
KORD
KORD
KORD
4
0.81
0.01
-0.59
-0.99
-0.59
-0.59
By default the parser removes the component date columns, but you can choose to retain them via the
keep_date_col keyword:
In [64]: df = pd.read_csv('tmp.csv', header=None, parse_dates=[[1, 2], [1, 3]],
....:
keep_date_col=True)
....:
In [65]: df
Out[65]:
0
1
2
3
4
5
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1_2
19:00:00
20:00:00
21:00:00
21:00:00
22:00:00
23:00:00
24.1. CSV & Text files
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1_3
18:56:00
19:56:00
20:56:00
21:18:00
21:56:00
22:56:00
0
KORD
KORD
KORD
KORD
KORD
KORD
1
19990127
19990127
19990127
19990127
19990127
19990127
2
19:00:00
20:00:00
21:00:00
21:00:00
22:00:00
23:00:00
\
721
pandas: powerful Python data analysis toolkit, Release 0.16.1
3
18:56:00
19:56:00
20:56:00
21:18:00
21:56:00
22:56:00
0
1
2
3
4
5
4
0.81
0.01
-0.59
-0.99
-0.59
-0.59
Note that if you wish to combine multiple columns into a single date column, a nested list must be used. In other
words, parse_dates=[1, 2] indicates that the second and third columns should each be parsed as separate date
columns while parse_dates=[[1, 2]] means the two columns should be parsed into a single column.
You can also use a dict to specify custom name columns:
In [66]: date_spec = {'nominal': [1, 2], 'actual': [1, 3]}
In [67]: df = pd.read_csv('tmp.csv', header=None, parse_dates=date_spec)
In [68]: df
Out[68]:
0
1
2
3
4
5
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
nominal
19:00:00
20:00:00
21:00:00
21:00:00
22:00:00
23:00:00
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
actual
18:56:00
19:56:00
20:56:00
21:18:00
21:56:00
22:56:00
0
KORD
KORD
KORD
KORD
KORD
KORD
4
0.81
0.01
-0.59
-0.99
-0.59
-0.59
It is important to remember that if multiple text columns are to be parsed into a single date column, then a new column
is prepended to the data. The index_col specification is based off of this new set of columns rather than the original
data columns:
In [69]: date_spec = {'nominal': [1, 2], 'actual': [1, 3]}
In [70]: df = pd.read_csv('tmp.csv', header=None, parse_dates=date_spec,
....:
index_col=0) #index is the nominal column
....:
In [71]: df
Out[71]:
nominal
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
19:00:00
20:00:00
21:00:00
21:00:00
22:00:00
23:00:00
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
actual
0
4
18:56:00
19:56:00
20:56:00
21:18:00
21:56:00
22:56:00
KORD
KORD
KORD
KORD
KORD
KORD
0.81
0.01
-0.59
-0.99
-0.59
-0.59
Note: read_csv has a fast_path for parsing datetime strings in iso8601 format, e.g “2000-01-01T00:01:02+00:00” and
similar variations. If you can arrange for your data to store datetimes in this format, load times will be significantly
faster, ~20x has been observed.
Note: When passing a dict as the parse_dates argument, the order of the columns prepended is not guaranteed,
because dict objects do not impose an ordering on their keys. On Python 2.7+ you may use collections.OrderedDict
instead of a regular dict if this matters to you. Because of this, when using a dict for ‘parse_dates’ in conjunction with
the index_col argument, it’s best to specify index_col as a column label rather then as an index on the resulting frame.
722
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
24.1.8 Specifying method for floating-point conversion
The parameter float_precision can be specified in order to use a specific floating-point converter during parsing
with the C engine. The options are the ordinary converter, the high-precision converter, and the round-trip converter
(which is guaranteed to round-trip values after writing to a file). For example:
In [72]: val = '0.3066101993807095471566981359501369297504425048828125'
In [73]: data = 'a,b,c\n1,2,{0}'.format(val)
In [74]: abs(pd.read_csv(StringIO(data), engine='c', float_precision=None)['c'][0] - float(val))
Out[74]: 0.0
In [75]: abs(pd.read_csv(StringIO(data), engine='c', float_precision='high')['c'][0] - float(val))
Out[75]: 5.5511151231257827e-17
In [76]: abs(pd.read_csv(StringIO(data), engine='c', float_precision='round_trip')['c'][0] - float(va
Out[76]: 0.0
24.1.9 Date Parsing Functions
Finally, the parser allows you to specify a custom date_parser function to take full advantage of the flexibility of
the date parsing API:
In [77]: import pandas.io.date_converters as conv
In [78]: df = pd.read_csv('tmp.csv', header=None, parse_dates=date_spec,
....:
date_parser=conv.parse_date_time)
....:
In [79]: df
Out[79]:
0
1
2
3
4
5
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
nominal
19:00:00
20:00:00
21:00:00
21:00:00
22:00:00
23:00:00
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
1999-01-27
actual
18:56:00
19:56:00
20:56:00
21:18:00
21:56:00
22:56:00
0
KORD
KORD
KORD
KORD
KORD
KORD
4
0.81
0.01
-0.59
-0.99
-0.59
-0.59
Pandas will try to call the date_parser function in three different ways. If an exception is raised, the next one is
tried:
1. date_parser is first called with one or more arrays as arguments, as defined using parse_dates (e.g.,
date_parser([’2013’, ’2013’], [’1’, ’2’]))
2. If #1 fails, date_parser is called with all the columns concatenated row-wise into a single array (e.g.,
date_parser([’2013 1’, ’2013 2’]))
3. If #2 fails, date_parser is called once for every row with one or more string arguments from
the columns indicated with parse_dates (e.g., date_parser(’2013’, ’1’) for the first row,
date_parser(’2013’, ’2’) for the second, etc.)
Note that performance-wise, you should try these methods of parsing dates in order:
1. Try to infer the format using infer_datetime_format=True (see section below)
24.1. CSV & Text files
723
pandas: powerful Python data analysis toolkit, Release 0.16.1
2. If
you
know
the
format,
use
pd.to_datetime(x, format=...)
pd.to_datetime():
date_parser=lambda x:
3. If you have a really non-standard format, use a custom date_parser function. For optimal performance, this
should be vectorized, i.e., it should accept arrays as arguments.
You can explore the date parsing functionality in date_converters.py and add your own. We would love to turn
this module into a community supported set of date/time parsers. To get you started, date_converters.py contains functions to parse dual date and time columns, year/month/day columns, and year/month/day/hour/minute/second
columns. It also contains a generic_parser function so you can curry it with a function that deals with a single
date rather than the entire array.
24.1.10 Inferring Datetime Format
If you have parse_dates enabled for some or all of your columns, and your datetime strings are all formatted the
same way, you may get a large speed up by setting infer_datetime_format=True. If set, pandas will attempt
to guess the format of your datetime strings, and then use a faster means of parsing the strings. 5-10x parsing speeds
have been observed. pandas will fallback to the usual parsing if either the format cannot be guessed or the format that
was guessed cannot properly parse the entire column of strings. So in general, infer_datetime_format should
not have any negative consequences if enabled.
Here are some examples of datetime strings that can be guessed (All representing December 30th, 2011 at 00:00:00)
• “20111230”
• “2011/12/30”
• “20111230 00:00:00”
• “12/30/2011 00:00:00”
• “30/Dec/2011 00:00:00”
• “30/December/2011 00:00:00”
infer_datetime_format is sensitive to dayfirst. With dayfirst=True, it will guess “01/12/2011” to be
December 1st. With dayfirst=False (default) it will guess “01/12/2011” to be January 12th.
# Try to infer the format for the index column
In [80]: df = pd.read_csv('foo.csv', index_col=0, parse_dates=True,
....:
infer_datetime_format=True)
....:
In [81]: df
Out[81]:
date
2009-01-01
2009-01-02
2009-01-03
A
B
C
a
b
c
1
3
4
2
4
5
24.1.11 International Date Formats
While US date formats tend to be MM/DD/YYYY, many international formats use DD/MM/YYYY instead. For
convenience, a dayfirst keyword is provided:
In [82]: print(open('tmp.csv').read())
date,value,cat
1/6/2000,5,a
724
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
2/6/2000,10,b
3/6/2000,15,c
In [83]: pd.read_csv('tmp.csv', parse_dates=[0])
Out[83]:
date value cat
0 2000-01-06
5
a
1 2000-02-06
10
b
2 2000-03-06
15
c
In [84]: pd.read_csv('tmp.csv', dayfirst=True, parse_dates=[0])
Out[84]:
date value cat
0 2000-06-01
5
a
1 2000-06-02
10
b
2 2000-06-03
15
c
24.1.12 Thousand Separators
For large numbers that have been written with a thousands separator, you can set the thousands keyword to a string
of length 1 so that integers will be parsed correctly:
By default, numbers with a thousands separator will be parsed as strings
In [85]: print(open('tmp.csv').read())
ID|level|category
Patient1|123,000|x
Patient2|23,000|y
Patient3|1,234,018|z
In [86]: df = pd.read_csv('tmp.csv', sep='|')
In [87]: df
Out[87]:
ID
0 Patient1
1 Patient2
2 Patient3
level category
123,000
x
23,000
y
1,234,018
z
In [88]: df.level.dtype
Out[88]: dtype('O')
The thousands keyword allows integers to be parsed correctly
In [89]: print(open('tmp.csv').read())
ID|level|category
Patient1|123,000|x
Patient2|23,000|y
Patient3|1,234,018|z
In [90]: df = pd.read_csv('tmp.csv', sep='|', thousands=',')
In [91]: df
Out[91]:
ID
0 Patient1
1 Patient2
level category
123000
x
23000
y
24.1. CSV & Text files
725
pandas: powerful Python data analysis toolkit, Release 0.16.1
2
Patient3
1234018
z
In [92]: df.level.dtype
Out[92]: dtype('int64')
24.1.13 NA Values
To control which values are parsed as missing values (which are signified by NaN), specifiy a list of strings in
na_values. If you specify a number (a float, like 5.0 or an integer like 5), the corresponding equivalent
values will also imply a missing value (in this case effectively [5.0,5] are recognized as NaN.
To completely override the default values that are recognized as missing, specify keep_default_na=False.
The default NaN recognized values are [’-1.#IND’, ’1.#QNAN’, ’1.#IND’, ’-1.#QNAN’,
’#N/A’,’N/A’, ’NA’, ’#NA’, ’NULL’, ’NaN’, ’-NaN’, ’nan’, ’-nan’].
read_csv(path, na_values=[5])
the default values, in addition to 5 , 5.0 when interpreted as numbers are recognized as NaN
read_csv(path, keep_default_na=False, na_values=[""])
only an empty field will be NaN
read_csv(path, keep_default_na=False, na_values=["NA", "0"])
only NA and 0 as strings are NaN
read_csv(path, na_values=["Nope"])
the default values, in addition to the string "Nope" are recognized as NaN
24.1.14 Infinity
inf like values will be parsed as np.inf (positive infinity), and -inf as -np.inf (negative infinity). These will
ignore the case of the value, meaning Inf, will also be parsed as np.inf.
24.1.15 Comments
Sometimes comments or meta data may be included in a file:
In [93]: print(open('tmp.csv').read())
ID,level,category
Patient1,123000,x # really unpleasant
Patient2,23000,y # wouldn't take his medicine
Patient3,1234018,z # awesome
By default, the parse includes the comments in the output:
In [94]: df = pd.read_csv('tmp.csv')
In [95]: df
Out[95]:
ID
0 Patient1
1 Patient2
2 Patient3
726
level
123000
23000
1234018
category
x # really unpleasant
y # wouldn't take his medicine
z # awesome
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
We can suppress the comments using the comment keyword:
In [96]: df = pd.read_csv('tmp.csv', comment='#')
In [97]: df
Out[97]:
ID
0 Patient1
1 Patient2
2 Patient3
level category
123000
x
23000
y
1234018
z
24.1.16 Returning Series
Using the squeeze keyword, the parser will return output with a single column as a Series:
In [98]: print(open('tmp.csv').read())
level
Patient1,123000
Patient2,23000
Patient3,1234018
In [99]: output =
pd.read_csv('tmp.csv', squeeze=True)
In [100]: output
Out[100]:
Patient1
123000
Patient2
23000
Patient3
1234018
Name: level, dtype: int64
In [101]: type(output)
Out[101]: pandas.core.series.Series
24.1.17 Boolean values
The common values True, False, TRUE, and FALSE are all recognized as boolean. Sometime you would want to
recognize some other values as being boolean. To do this use the true_values and false_values options:
In [102]: data= 'a,b,c\n1,Yes,2\n3,No,4'
In [103]: print(data)
a,b,c
1,Yes,2
3,No,4
In [104]: pd.read_csv(StringIO(data))
Out[104]:
a
b c
0 1 Yes 2
1 3
No 4
In [105]: pd.read_csv(StringIO(data), true_values=['Yes'], false_values=['No'])
Out[105]:
a
b c
0 1
True 2
1 3 False 4
24.1. CSV & Text files
727
pandas: powerful Python data analysis toolkit, Release 0.16.1
24.1.18 Handling “bad” lines
Some files may have malformed lines with too few fields or too many. Lines with too few fields will have NA values
filled in the trailing fields. Lines with too many will cause an error by default:
In [27]: data = 'a,b,c\n1,2,3\n4,5,6,7\n8,9,10'
In [28]: pd.read_csv(StringIO(data))
--------------------------------------------------------------------------CParserError
Traceback (most recent call last)
CParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 4
You can elect to skip bad lines:
In [29]: pd.read_csv(StringIO(data), error_bad_lines=False)
Skipping line 3: expected 3 fields, saw 4
Out[29]:
a b
c
0 1 2
3
1 8 9 10
24.1.19 Quoting and Escape Characters
Quotes (and other escape characters) in embedded fields can be handled in any number of ways. One way is to use
backslashes; to properly parse this data, you should pass the escapechar option:
In [106]: data = 'a,b\n"hello, \\"Bob\\", nice to see you",5'
In [107]: print(data)
a,b
"hello, \"Bob\", nice to see you",5
In [108]: pd.read_csv(StringIO(data), escapechar='\\')
Out[108]:
a b
0 hello, "Bob", nice to see you 5
24.1.20 Files with Fixed Width Columns
While read_csv reads delimited data, the read_fwf() function works with data files that have known and fixed
column widths. The function parameters to read_fwf are largely the same as read_csv with two extra parameters:
• colspecs: A list of pairs (tuples) giving the extents of the fixed-width fields of each line as half-open intervals
(i.e., [from, to[ ). String value ‘infer’ can be used to instruct the parser to try detecting the column specifications
from the first 100 rows of the data. Default behaviour, if not specified, is to infer.
• widths: A list of field widths which can be used instead of ‘colspecs’ if the intervals are contiguous.
Consider a typical fixed-width data file:
In [109]:
id8141
id1594
id1849
id1230
id1948
728
print(open('bar.csv').read())
360.242940
149.910199
11950.7
444.953632
166.985655
11788.4
364.136849
183.628767
11806.2
413.836124
184.375703
11916.8
502.953953
173.237159
12468.3
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
In order to parse this file into a DataFrame, we simply need to supply the column specifications to the read_fwf
function along with the file name:
#Column specifications are a list of half-intervals
In [110]: colspecs = [(0, 6), (8, 20), (21, 33), (34, 43)]
In [111]: df = pd.read_fwf('bar.csv', colspecs=colspecs, header=None, index_col=0)
In [112]: df
Out[112]:
0
id8141
id1594
id1849
id1230
id1948
1
2
3
360.242940
444.953632
364.136849
413.836124
502.953953
149.910199
166.985655
183.628767
184.375703
173.237159
11950.7
11788.4
11806.2
11916.8
12468.3
Note how the parser automatically picks column names X. when header=None argument is specified. Alternatively, you can supply just the column widths for contiguous columns:
#Widths are a list of integers
In [113]: widths = [6, 14, 13, 10]
In [114]: df = pd.read_fwf('bar.csv', widths=widths, header=None)
In [115]: df
Out[115]:
0
1
0 id8141 360.242940
1 id1594 444.953632
2 id1849 364.136849
3 id1230 413.836124
4 id1948 502.953953
2
149.910199
166.985655
183.628767
184.375703
173.237159
3
11950.7
11788.4
11806.2
11916.8
12468.3
The parser will take care of extra white spaces around the columns so it’s ok to have extra separation between the
columns in the file.
New in version 0.13.0.
By default, read_fwf will try to infer the file’s colspecs by using the first 100 rows of the file. It can do it
only in cases when the columns are aligned and correctly separated by the provided delimiter (default delimiter is
whitespace).
In [116]: df = pd.read_fwf('bar.csv', header=None, index_col=0)
In [117]: df
Out[117]:
0
id8141
id1594
id1849
id1230
id1948
1
2
3
360.242940
444.953632
364.136849
413.836124
502.953953
149.910199
166.985655
183.628767
184.375703
173.237159
11950.7
11788.4
11806.2
11916.8
12468.3
24.1.21 Files with an “implicit” index column
Consider a file with one less entry in the header than the number of data column:
24.1. CSV & Text files
729
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [118]: print(open('foo.csv').read())
A,B,C
20090101,a,1,2
20090102,b,3,4
20090103,c,4,5
In this special case, read_csv assumes that the first column is to be used as the index of the DataFrame:
In [119]: pd.read_csv('foo.csv')
Out[119]:
A B C
20090101 a 1 2
20090102 b 3 4
20090103 c 4 5
Note that the dates weren’t automatically parsed. In that case you would need to do as before:
In [120]: df = pd.read_csv('foo.csv', parse_dates=True)
In [121]: df.index
Out[121]: DatetimeIndex(['2009-01-01', '2009-01-02', '2009-01-03'], dtype='datetime64[ns]', freq=None
24.1.22 Reading an index with a MultiIndex
Suppose you have data indexed by two columns:
In [122]: print(open('data/mindex_ex.csv').read())
year,indiv,zit,xit
1977,"A",1.2,.6
1977,"B",1.5,.5
1977,"C",1.7,.8
1978,"A",.2,.06
1978,"B",.7,.2
1978,"C",.8,.3
1978,"D",.9,.5
1978,"E",1.4,.9
1979,"C",.2,.15
1979,"D",.14,.05
1979,"E",.5,.15
1979,"F",1.2,.5
1979,"G",3.4,1.9
1979,"H",5.4,2.7
1979,"I",6.4,1.2
The index_col argument to read_csv and read_table can take a list of column numbers to turn multiple
columns into a MultiIndex for the index of the returned object:
In [123]: df = pd.read_csv("data/mindex_ex.csv", index_col=[0,1])
In [124]: df
Out[124]:
year indiv
1977 A
B
C
1978 A
B
730
zit
xit
1.20
1.50
1.70
0.20
0.70
0.60
0.50
0.80
0.06
0.20
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
C
D
E
1979 C
D
E
F
G
H
I
0.80
0.90
1.40
0.20
0.14
0.50
1.20
3.40
5.40
6.40
0.30
0.50
0.90
0.15
0.05
0.15
0.50
1.90
2.70
1.20
In [125]: df.ix[1978]
Out[125]:
zit
xit
indiv
A
0.2 0.06
B
0.7 0.20
C
0.8 0.30
D
0.9 0.50
E
1.4 0.90
24.1.23 Reading columns with a MultiIndex
By specifying list of row locations for the header argument, you can read in a MultiIndex for the columns.
Specifying non-consecutive rows will skip the intervening rows. In order to have the pre-0.13 behavior of tupleizing
columns, specify tupleize_cols=True.
In [126]: from pandas.util.testing import makeCustomDataframe as mkdf
In [127]: df = mkdf(5,3,r_idx_nlevels=2,c_idx_nlevels=4)
In [128]: df.to_csv('mi.csv')
In [129]: print(open('mi.csv').read())
C0,,C_l0_g0,C_l0_g1,C_l0_g2
C1,,C_l1_g0,C_l1_g1,C_l1_g2
C2,,C_l2_g0,C_l2_g1,C_l2_g2
C3,,C_l3_g0,C_l3_g1,C_l3_g2
R0,R1,,,
R_l0_g0,R_l1_g0,R0C0,R0C1,R0C2
R_l0_g1,R_l1_g1,R1C0,R1C1,R1C2
R_l0_g2,R_l1_g2,R2C0,R2C1,R2C2
R_l0_g3,R_l1_g3,R3C0,R3C1,R3C2
R_l0_g4,R_l1_g4,R4C0,R4C1,R4C2
In [130]: pd.read_csv('mi.csv',header=[0,1,2,3],index_col=[0,1])
Out[130]:
C0
C_l0_g0 C_l0_g1 C_l0_g2
C1
C_l1_g0 C_l1_g1 C_l1_g2
C2
C_l2_g0 C_l2_g1 C_l2_g2
C3
C_l3_g0 C_l3_g1 C_l3_g2
R0
R1
R_l0_g0 R_l1_g0
R0C0
R0C1
R0C2
R_l0_g1 R_l1_g1
R1C0
R1C1
R1C2
R_l0_g2 R_l1_g2
R2C0
R2C1
R2C2
R_l0_g3 R_l1_g3
R3C0
R3C1
R3C2
24.1. CSV & Text files
731
pandas: powerful Python data analysis toolkit, Release 0.16.1
R_l0_g4 R_l1_g4
R4C0
R4C1
R4C2
Starting in 0.13.0, read_csv will be able to interpret a more common format of multi-columns indices.
In [131]: print(open('mi2.csv').read())
,a,a,a,b,c,c
,q,r,s,t,u,v
one,1,2,3,4,5,6
two,7,8,9,10,11,12
In [132]: pd.read_csv('mi2.csv',header=[0,1],index_col=0)
Out[132]:
a
b
c
q r s
t
u
v
one 1 2 3
4
5
6
two 7 8 9 10 11 12
Note: If an index_col is not specified (e.g. you don’t have an index, or wrote it with df.to_csv(...,
index=False), then any names on the columns index will be lost.
24.1.24 Automatically “sniffing” the delimiter
read_csv is capable of inferring delimited (not necessarily comma-separated) files, as pandas uses the
csv.Sniffer class of the csv module. For this, you have to specify sep=None.
In [133]: print(open('tmp2.sv').read())
:0:1:2:3
0:0.469112299907:-0.282863344329:-1.50905850317:-1.13563237102
1:1.21211202502:-0.173214649053:0.119208711297:-1.04423596628
2:-0.861848963348:-2.10456921889:-0.494929274069:1.07180380704
3:0.721555162244:-0.70677113363:-1.03957498511:0.271859885543
4:-0.424972329789:0.567020349794:0.276232019278:-1.08740069129
5:-0.673689708088:0.113648409689:-1.47842655244:0.524987667115
6:0.40470521868:0.57704598592:-1.71500201611:-1.03926848351
7:-0.370646858236:-1.15789225064:-1.34431181273:0.844885141425
8:1.07576978372:-0.10904997528:1.64356307036:-1.46938795954
9:0.357020564133:-0.67460010373:-1.77690371697:-0.968913812447
In [134]: pd.read_csv('tmp2.sv', sep=None, engine='python')
Out[134]:
Unnamed: 0
0
1
2
3
0
0 0.469112 -0.282863 -1.509059 -1.135632
1
1 1.212112 -0.173215 0.119209 -1.044236
2
2 -0.861849 -2.104569 -0.494929 1.071804
3
3 0.721555 -0.706771 -1.039575 0.271860
4
4 -0.424972 0.567020 0.276232 -1.087401
5
5 -0.673690 0.113648 -1.478427 0.524988
6
6 0.404705 0.577046 -1.715002 -1.039268
7
7 -0.370647 -1.157892 -1.344312 0.844885
8
8 1.075770 -0.109050 1.643563 -1.469388
9
9 0.357021 -0.674600 -1.776904 -0.968914
732
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
24.1.25 Iterating through files chunk by chunk
Suppose you wish to iterate through a (potentially very large) file lazily rather than reading the entire file into memory,
such as the following:
In [135]: print(open('tmp.sv').read())
|0|1|2|3
0|0.469112299907|-0.282863344329|-1.50905850317|-1.13563237102
1|1.21211202502|-0.173214649053|0.119208711297|-1.04423596628
2|-0.861848963348|-2.10456921889|-0.494929274069|1.07180380704
3|0.721555162244|-0.70677113363|-1.03957498511|0.271859885543
4|-0.424972329789|0.567020349794|0.276232019278|-1.08740069129
5|-0.673689708088|0.113648409689|-1.47842655244|0.524987667115
6|0.40470521868|0.57704598592|-1.71500201611|-1.03926848351
7|-0.370646858236|-1.15789225064|-1.34431181273|0.844885141425
8|1.07576978372|-0.10904997528|1.64356307036|-1.46938795954
9|0.357020564133|-0.67460010373|-1.77690371697|-0.968913812447
In [136]: table = pd.read_table('tmp.sv', sep='|')
In [137]: table
Out[137]:
Unnamed: 0
0
0
0 0.469112
1
1 1.212112
2
2 -0.861849
3
3 0.721555
4
4 -0.424972
5
5 -0.673690
6
6 0.404705
7
7 -0.370647
8
8 1.075770
9
9 0.357021
1
-0.282863
-0.173215
-2.104569
-0.706771
0.567020
0.113648
0.577046
-1.157892
-0.109050
-0.674600
2
-1.509059
0.119209
-0.494929
-1.039575
0.276232
-1.478427
-1.715002
-1.344312
1.643563
-1.776904
3
-1.135632
-1.044236
1.071804
0.271860
-1.087401
0.524988
-1.039268
0.844885
-1.469388
-0.968914
By specifying a chunksize to read_csv or read_table, the return value will be an iterable object of type
TextFileReader:
In [138]: reader = pd.read_table('tmp.sv', sep='|', chunksize=4)
In [139]: reader
Out[139]:
In [140]: for
.....:
.....:
Unnamed: 0
0
0
1
1
2
2
3
3
Unnamed: 0
0
4
1
5
2
6
3
7
Unnamed: 0
0
8
1
9
chunk in reader:
print(chunk)
0
0.469112
1.212112
-0.861849
0.721555
0
-0.424972
-0.673690
0.404705
-0.370647
0
1.075770
0.357021
24.1. CSV & Text files
1
2
3
-0.282863 -1.509059 -1.135632
-0.173215 0.119209 -1.044236
-2.104569 -0.494929 1.071804
-0.706771 -1.039575 0.271860
1
2
3
0.567020 0.276232 -1.087401
0.113648 -1.478427 0.524988
0.577046 -1.715002 -1.039268
-1.157892 -1.344312 0.844885
1
2
3
-0.10905 1.643563 -1.469388
-0.67460 -1.776904 -0.968914
733
pandas: powerful Python data analysis toolkit, Release 0.16.1
Specifying iterator=True will also return the TextFileReader object:
In [141]: reader = pd.read_table('tmp.sv', sep='|', iterator=True)
In [142]: reader.get_chunk(5)
Out[142]:
Unnamed: 0
0
1
2
3
0
0 0.469112 -0.282863 -1.509059 -1.135632
1
1 1.212112 -0.173215 0.119209 -1.044236
2
2 -0.861849 -2.104569 -0.494929 1.071804
3
3 0.721555 -0.706771 -1.039575 0.271860
4
4 -0.424972 0.567020 0.276232 -1.087401
24.1.26 Specifying the parser engine
Under the hood pandas uses a fast and efficient parser implemented in C as well as a python implementation which is
currently more feature-complete. Where possible pandas uses the C parser (specified as engine=’c’), but may fall
back to python if C-unsupported options are specified. Currently, C-unsupported options include:
• sep other than a single character (e.g. regex separators)
• skip_footer
• sep=None with delim_whitespace=False
Specifying any of the above options will produce a ParserWarning unless the python engine is selected explicitly
using engine=’python’.
24.1.27 Writing to CSV format
The Series and DataFrame objects have an instance method to_csv which allows storing the contents of the object
as a comma-separated-values file. The function takes a number of arguments. Only the first is required.
• path_or_buf: A string path to the file to write or a StringIO
• sep : Field delimiter for the output file (default ”,”)
• na_rep: A string representation of a missing value (default ‘’)
• float_format: Format string for floating point numbers
• cols: Columns to write (default None)
• header: Whether to write out the column names (default True)
• index: whether to write row (index) names (default True)
• index_label: Column label(s) for index column(s) if desired. If None (default), and header and index are
True, then the index names are used. (A sequence should be given if the DataFrame uses MultiIndex).
• mode : Python write mode, default ‘w’
• encoding: a string representing the encoding to use if the contents are non-ASCII, for python versions prior
to 3
• line_terminator: Character sequence denoting line end (default ‘\n’)
• quoting: Set quoting rules as in csv module (default csv.QUOTE_MINIMAL)
• quotechar: Character used to quote fields (default ‘”’)
• doublequote: Control quoting of quotechar in fields (default True)
734
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
• escapechar: Character used to escape sep and quotechar when appropriate (default None)
• chunksize: Number of rows to write at a time
• tupleize_cols: If False (default), write as a list of tuples, otherwise write in an expanded line format
suitable for read_csv
• date_format: Format string for datetime objects
24.1.28 Writing a formatted string
The DataFrame object has an instance method to_string which allows control over the string representation of the
object. All arguments are optional:
• buf default None, for example a StringIO object
• columns default None, which columns to write
• col_space default None, minimum width of each column.
• na_rep default NaN, representation of NA value
• formatters default None, a dictionary (by column) of functions each of which takes a single argument and
returns a formatted string
• float_format default None, a function which takes a single (float) argument and returns a formatted string;
to be applied to floats in the DataFrame.
• sparsify default True, set to False for a DataFrame with a hierarchical index to print every multiindex key at
each row.
• index_names default True, will print the names of the indices
• index default True, will print the index (ie, row labels)
• header default True, will print the column labels
• justify default left, will print column headers left- or right-justified
The Series object also has a to_string method, but with only the buf, na_rep, float_format arguments.
There is also a length argument which, if set to True, will additionally output the length of the Series.
24.2 JSON
Read and write JSON format files and strings.
24.2.1 Writing JSON
A Series or DataFrame can be converted to a valid JSON string. Use to_json with optional parameters:
• path_or_buf : the pathname or buffer to write the output This can be None in which case a JSON string is
returned
• orient :
Series :
– default is index
– allowed values are {split, records, index}
24.2. JSON
735
pandas: powerful Python data analysis toolkit, Release 0.16.1
DataFrame
– default is columns
– allowed values are {split, records, index, columns, values}
The format of the JSON string
split
records
index
columns
values
dict like {index -> [index], columns -> [columns], data -> [values]}
list like [{column -> value}, ... , {column -> value}]
dict like {index -> {column -> value}}
dict like {column -> {index -> value}}
just the values array
• date_format : string, type of date conversion, ‘epoch’ for timestamp, ‘iso’ for ISO8601.
• double_precision : The number of decimal places to use when encoding floating point values, default 10.
• force_ascii : force encoded string to be ASCII, default True.
• date_unit : The time unit to encode to, governs timestamp and ISO8601 precision. One of ‘s’, ‘ms’, ‘us’ or
‘ns’ for seconds, milliseconds, microseconds and nanoseconds respectively. Default ‘ms’.
• default_handler : The handler to call if an object cannot otherwise be converted to a suitable format for
JSON. Takes a single argument, which is the object to convert, and returns a serializable object.
Note NaN‘s, NaT‘s and None will be converted to null and datetime objects will be converted based on the
date_format and date_unit parameters.
In [143]: dfj = DataFrame(randn(5, 2), columns=list('AB'))
In [144]: json = dfj.to_json()
In [145]: json
Out[145]: '{"A":{"0":-1.2945235903,"1":0.2766617129,"2":-0.0139597524,"3":-0.0061535699,"4":0.8957173
Orient Options
There are a number of different options for the format of the resulting JSON file / string. Consider the following
DataFrame and Series:
In [146]: dfjo = DataFrame(dict(A=range(1, 4), B=range(4, 7), C=range(7, 10)),
.....:
columns=list('ABC'), index=list('xyz'))
.....:
In [147]: dfjo
Out[147]:
A B C
x 1 4 7
y 2 5 8
z 3 6 9
In [148]: sjo = Series(dict(x=15, y=16, z=17), name='D')
In [149]: sjo
Out[149]:
x
15
y
16
z
17
Name: D, dtype: int64
736
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
Column oriented (the default for DataFrame) serializes the data as nested JSON objects with column labels acting
as the primary index:
In [150]: dfjo.to_json(orient="columns")
Out[150]: '{"A":{"x":1,"y":2,"z":3},"B":{"x":4,"y":5,"z":6},"C":{"x":7,"y":8,"z":9}}'
Index oriented (the default for Series) similar to column oriented but the index labels are now primary:
In [151]: dfjo.to_json(orient="index")
Out[151]: '{"x":{"A":1,"B":4,"C":7},"y":{"A":2,"B":5,"C":8},"z":{"A":3,"B":6,"C":9}}'
In [152]: sjo.to_json(orient="index")
Out[152]: '{"x":15,"y":16,"z":17}'
Record oriented serializes the data to a JSON array of column -> value records, index labels are not included. This is
useful for passing DataFrame data to plotting libraries, for example the JavaScript library d3.js:
In [153]: dfjo.to_json(orient="records")
Out[153]: '[{"A":1,"B":4,"C":7},{"A":2,"B":5,"C":8},{"A":3,"B":6,"C":9}]'
In [154]: sjo.to_json(orient="records")
Out[154]: '[15,16,17]'
Value oriented is a bare-bones option which serializes to nested JSON arrays of values only, column and index labels
are not included:
In [155]: dfjo.to_json(orient="values")
Out[155]: '[[1,4,7],[2,5,8],[3,6,9]]'
Split oriented serializes to a JSON object containing separate entries for values, index and columns. Name is also
included for Series:
In [156]: dfjo.to_json(orient="split")
Out[156]: '{"columns":["A","B","C"],"index":["x","y","z"],"data":[[1,4,7],[2,5,8],[3,6,9]]}'
In [157]: sjo.to_json(orient="split")
Out[157]: '{"name":"D","index":["x","y","z"],"data":[15,16,17]}'
Note: Any orient option that encodes to a JSON object will not preserve the ordering of index and column labels
during round-trip serialization. If you wish to preserve label ordering use the split option as it uses ordered containers.
Date Handling
Writing in ISO date format
In [158]: dfd = DataFrame(randn(5, 2), columns=list('AB'))
In [159]: dfd['date'] = Timestamp('20130101')
In [160]: dfd = dfd.sort_index(1, ascending=False)
In [161]: json = dfd.to_json(date_format='iso')
In [162]: json
Out[162]: '{"date":{"0":"2013-01-01T00:00:00.000Z","1":"2013-01-01T00:00:00.000Z","2":"2013-01-01T00:
Writing in ISO date format, with microseconds
24.2. JSON
737
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [163]: json = dfd.to_json(date_format='iso', date_unit='us')
In [164]: json
Out[164]: '{"date":{"0":"2013-01-01T00:00:00.000000Z","1":"2013-01-01T00:00:00.000000Z","2":"2013-01-
Epoch timestamps, in seconds
In [165]: json = dfd.to_json(date_format='epoch', date_unit='s')
In [166]: json
Out[166]: '{"date":{"0":1356998400,"1":1356998400,"2":1356998400,"3":1356998400,"4":1356998400},"B":{
Writing to a file, with a date index and a date column
In [167]: dfj2 = dfj.copy()
In [168]: dfj2['date'] = Timestamp('20130101')
In [169]: dfj2['ints'] = list(range(5))
In [170]: dfj2['bools'] = True
In [171]: dfj2.index = date_range('20130101', periods=5)
In [172]: dfj2.to_json('test.json')
In [173]: open('test.json').read()
Out[173]: '{"A":{"1356998400000":-1.2945235903,"1357084800000":0.2766617129,"1357171200000":-0.013959
Fallback Behavior
If the JSON serializer cannot handle the container contents directly it will fallback in the following manner:
• if a toDict method is defined by the unrecognised object then that will be called and its returned dict will
be JSON serialized.
• if a default_handler has been passed to to_json that will be called to convert the object.
• otherwise an attempt is made to convert the object to a dict by parsing its contents. However if the object is
complex this will often fail with an OverflowError.
Your best bet when encountering OverflowError during serialization is to specify a default_handler. For
example timedelta can cause problems:
In [141]: from datetime import timedelta
In [142]: dftd = DataFrame([timedelta(23), timedelta(seconds=5), 42])
In [143]: dftd.to_json()
--------------------------------------------------------------------------OverflowError
Traceback (most recent call last)
OverflowError: Maximum recursion level reached
which can be dealt with by specifying a simple default_handler:
In [174]: dftd.to_json(default_handler=str)
Out[174]: '{"0":{"0":1987200000,"1":5000,"2":42}}'
738
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [175]: def my_handler(obj):
.....:
return obj.total_seconds()
.....:
24.2.2 Reading JSON
Reading a JSON string to pandas object can take a number of parameters. The parser will try to parse a DataFrame
if typ is not supplied or is None. To explicitly force Series parsing, pass typ=series
• filepath_or_buffer : a VALID JSON string or file handle / StringIO. The string could be a URL. Valid
URL schemes include http, ftp, S3, and file. For file URLs, a host is expected. For instance, a local file could be
file ://localhost/path/to/table.json
• typ : type of object to recover (series or frame), default ‘frame’
• orient :
Series :
– default is index
– allowed values are {split, records, index}
DataFrame
– default is columns
– allowed values are {split, records, index, columns, values}
The format of the JSON string
split
records
index
columns
values
dict like {index -> [index], columns -> [columns], data -> [values]}
list like [{column -> value}, ... , {column -> value}]
dict like {index -> {column -> value}}
dict like {column -> {index -> value}}
just the values array
• dtype : if True, infer dtypes, if a dict of column to dtype, then use those, if False, then don’t infer dtypes at all,
default is True, apply only to the data
• convert_axes : boolean, try to convert the axes to the proper dtypes, default is True
• convert_dates : a list of columns to parse for dates; If True, then try to parse date-like columns, default is
True
• keep_default_dates : boolean, default True. If parsing dates, then parse the default date-like columns
• numpy : direct decoding to numpy arrays. default is False; Supports numeric data only, although labels may be
non-numeric. Also note that the JSON ordering MUST be the same for each term if numpy=True
• precise_float : boolean, default False. Set to enable usage of higher precision (strtod) function when
decoding string to double values. Default (False) is to use fast but less precise builtin functionality
• date_unit : string, the timestamp unit to detect if converting dates. Default None. By default the timestamp
precision will be detected, if this is not desired then pass one of ‘s’, ‘ms’, ‘us’ or ‘ns’ to force timestamp
precision to seconds, milliseconds, microseconds or nanoseconds respectively.
The parser will raise one of ValueError/TypeError/AssertionError if the JSON is not parseable.
If a non-default orient was used when encoding to JSON be sure to pass the same option here so that decoding
produces sensible results, see Orient Options for an overview.
24.2. JSON
739
pandas: powerful Python data analysis toolkit, Release 0.16.1
Data Conversion
The default of convert_axes=True, dtype=True, and convert_dates=True will try to parse the axes, and
all of the data into appropriate types, including dates. If you need to override specific dtypes, pass a dict to dtype.
convert_axes should only be set to False if you need to preserve string-like numbers (e.g. ‘1’, ‘2’) in an axes.
Note: Large integer values may be converted to dates if convert_dates=True and the data and / or column
labels appear ‘date-like’. The exact threshold depends on the date_unit specified.
Warning: When reading JSON data, automatic coercing into dtypes has some quirks:
• an index can be reconstructed in a different order from serialization, that is, the returned order is not guaranteed to be the same as before serialization
• a column that was float data will be converted to integer if it can be done safely, e.g. a column of 1.
• bool columns will be converted to integer on reconstruction
Thus there are times where you may want to specify specific dtypes via the dtype keyword argument.
Reading from a JSON string:
In [176]: pd.read_json(json)
Out[176]:
A
B
date
0 -1.206412 2.565646 2013-01-01
1 1.431256 1.340309 2013-01-01
2 -1.170299 -0.226169 2013-01-01
3 0.410835 0.813850 2013-01-01
4 0.132003 -0.827317 2013-01-01
Reading from a file:
In [177]: pd.read_json('test.json')
Out[177]:
A
B bools
2013-01-01 -1.294524 0.413738 True
2013-01-02 0.276662 -0.472035 True
2013-01-03 -0.013960 -0.362543 True
2013-01-04 -0.006154 -0.923061 True
2013-01-05 0.895717 0.805244 True
date
2013-01-01
2013-01-01
2013-01-01
2013-01-01
2013-01-01
ints
0
1
2
3
4
Don’t convert any data (but still convert axes and dates):
In [178]: pd.read_json('test.json', dtype=object).dtypes
Out[178]:
A
object
B
object
bools
object
date
object
ints
object
dtype: object
Specify dtypes for conversion:
In [179]: pd.read_json('test.json', dtype={'A' : 'float32', 'bools' : 'int8'}).dtypes
Out[179]:
A
float32
B
float64
bools
int8
date
datetime64[ns]
740
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
ints
dtype: object
int64
Preserve string indices:
In [180]: si = DataFrame(np.zeros((4, 4)),
.....:
columns=list(range(4)),
.....:
index=[str(i) for i in range(4)])
.....:
In [181]: si
Out[181]:
0 1 2 3
0 0 0 0 0
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
In [182]: si.index
Out[182]: Index([u'0', u'1', u'2', u'3'], dtype='object')
In [183]: si.columns
Out[183]: Int64Index([0, 1, 2, 3], dtype='int64')
In [184]: json = si.to_json()
In [185]: sij = pd.read_json(json, convert_axes=False)
In [186]: sij
Out[186]:
0 1 2 3
0 0 0 0 0
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
In [187]: sij.index
Out[187]: Index([u'0', u'1', u'2', u'3'], dtype='object')
In [188]: sij.columns
Out[188]: Index([u'0', u'1', u'2', u'3'], dtype='object')
Dates written in nanoseconds need to be read back in nanoseconds:
In [189]: json = dfj2.to_json(date_unit='ns')
# Try to parse timestamps as millseconds -> Won't Work
In [190]: dfju = pd.read_json(json, date_unit='ms')
In [191]: dfju
Out[191]:
A
B bools
1.356998e+18 -1.294524 0.413738 True
1.357085e+18 0.276662 -0.472035 True
1.357171e+18 -0.013960 -0.362543 True
1.357258e+18 -0.006154 -0.923061 True
1.357344e+18 0.895717 0.805244 True
date
1356998400000000000
1356998400000000000
1356998400000000000
1356998400000000000
1356998400000000000
ints
0
1
2
3
4
# Let pandas detect the correct precision
24.2. JSON
741
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [192]: dfju = pd.read_json(json)
In [193]: dfju
Out[193]:
A
B bools
date
2013-01-01 -1.294524 0.413738 True 2013-01-01
2013-01-02 0.276662 -0.472035 True 2013-01-01
2013-01-03 -0.013960 -0.362543 True 2013-01-01
2013-01-04 -0.006154 -0.923061 True 2013-01-01
2013-01-05 0.895717 0.805244 True 2013-01-01
ints
0
1
2
3
4
# Or specify that all timestamps are in nanoseconds
In [194]: dfju = pd.read_json(json, date_unit='ns')
In [195]: dfju
Out[195]:
A
B bools
date
2013-01-01 -1.294524 0.413738 True 2013-01-01
2013-01-02 0.276662 -0.472035 True 2013-01-01
2013-01-03 -0.013960 -0.362543 True 2013-01-01
2013-01-04 -0.006154 -0.923061 True 2013-01-01
2013-01-05 0.895717 0.805244 True 2013-01-01
ints
0
1
2
3
4
The Numpy Parameter
Note: This supports numeric data only. Index and columns labels may be non-numeric, e.g. strings, dates etc.
If numpy=True is passed to read_json an attempt will be made to sniff an appropriate dtype during deserialization
and to subsequently decode directly to numpy arrays, bypassing the need for intermediate Python objects.
This can provide speedups if you are deserialising a large amount of numeric data:
In [196]: randfloats = np.random.uniform(-100, 1000, 10000)
In [197]: randfloats.shape = (1000, 10)
In [198]: dffloats = DataFrame(randfloats, columns=list('ABCDEFGHIJ'))
In [199]: jsonfloats = dffloats.to_json()
In [200]: timeit read_json(jsonfloats)
100 loops, best of 3: 11.1 ms per loop
In [201]: timeit read_json(jsonfloats, numpy=True)
100 loops, best of 3: 7.06 ms per loop
The speedup is less noticeable for smaller datasets:
In [202]: jsonfloats = dffloats.head(100).to_json()
In [203]: timeit read_json(jsonfloats)
100 loops, best of 3: 4.49 ms per loop
In [204]: timeit read_json(jsonfloats, numpy=True)
100 loops, best of 3: 3.45 ms per loop
742
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
Warning: Direct numpy decoding makes a number of assumptions and may fail or produce unexpected output if
these assumptions are not satisfied:
• data is numeric.
• data is uniform. The dtype is sniffed from the first value decoded. A ValueError may be raised, or
incorrect output may be produced if this condition is not satisfied.
• labels are ordered. Labels are only read from the first container, it is assumed that each subsequent row /
column has been encoded in the same order. This should be satisfied if the data was encoded using to_json
but may not be the case if the JSON is from another source.
24.2.3 Normalization
New in version 0.13.0.
pandas provides a utility function to take a dict or list of dicts and normalize this semi-structured data into a flat table.
In [205]: from pandas.io.json import json_normalize
In [206]: data = [{'state': 'Florida',
.....:
'shortname': 'FL',
.....:
'info': {
.....:
'governor': 'Rick Scott'
.....:
},
.....:
'counties': [{'name': 'Dade', 'population': 12345},
.....:
{'name': 'Broward', 'population': 40000},
.....:
{'name': 'Palm Beach', 'population': 60000}]},
.....:
{'state': 'Ohio',
.....:
'shortname': 'OH',
.....:
'info': {
.....:
'governor': 'John Kasich'
.....:
},
.....:
'counties': [{'name': 'Summit', 'population': 1234},
.....:
{'name': 'Cuyahoga', 'population': 1337}]}]
.....:
In [207]: json_normalize(data, 'counties', ['state', 'shortname', ['info', 'governor']])
Out[207]:
name population info.governor
state shortname
0
Dade
12345
Rick Scott Florida
FL
1
Broward
40000
Rick Scott Florida
FL
2 Palm Beach
60000
Rick Scott Florida
FL
3
Summit
1234
John Kasich
Ohio
OH
4
Cuyahoga
1337
John Kasich
Ohio
OH
24.3 HTML
24.3.1 Reading HTML Content
Warning: We highly encourage you to read the HTML parsing gotchas regarding the issues surrounding the
BeautifulSoup4/html5lib/lxml parsers.
New in version 0.12.0.
24.3. HTML
743
pandas: powerful Python data analysis toolkit, Release 0.16.1
The top-level read_html() function can accept an HTML string/file/URL and will parse HTML tables into list of
pandas DataFrames. Let’s look at a few examples.
Note: read_html returns a list of DataFrame objects, even if there is only a single table contained in the
HTML content
Read a URL with no options
In [208]: url = 'http://www.fdic.gov/bank/individual/failed/banklist.html'
In [209]: dfs = read_html(url)
In [210]: dfs
Out[210]:
[
Bank Name
0
Edgebrook Bank
1
Doral BankEn Espanol
2
Capitol City Bank & Trust Company
3
Highland Community Bank
4
First National Bank of Crestview
5
Northern Star Bank
6
Frontier Bank, FSB D/B/A El Paseo Bank
..
...
532
Hamilton Bank, NAEn Espanol
533
Sinclair National Bank
534
Superior Bank, FSB
535
Malta National Bank
536
First Alliance Bank & Trust Co.
537
National State Bank of Metropolis
538
Bank of Honolulu
0
1
2
3
4
5
6
..
532
533
534
535
536
537
538
0
1
2
3
4
5
6
..
532
744
Acquiring Institution
Republic Bank of Chicago
Banco Popular de Puerto Rico
First-Citizens Bank & Trust Company
United Fidelity Bank, fsb
First NBC Bank
BankVista
Bank of Southern California, N.A.
...
Israel Discount Bank of New York
Delta Trust & Bank
Superior Federal, FSB
North Valley Bank
Southern New Hampshire Bank & Trust
Banterra Bank of Marion
Bank of the Orient
City
Chicago
San Juan
Atlanta
Chicago
Crestview
Mankato
Palm Desert
...
Miami
Gravette
Hinsdale
Malta
Manchester
Metropolis
Honolulu
Closing
May 8,
February 27,
February 13,
January 23,
January 16,
December 19,
November 7,
January 11,
September 7,
July 27,
May 3,
February 2,
December 14,
October 13,
ST
IL
PR
GA
IL
FL
MN
CA
..
FL
AR
IL
OH
NH
IL
HI
Date
2015
2015
2015
2015
2015
2014
2014
...
2002
2001
2001
2001
2001
2000
2000
CERT
57772
32102
33938
20290
17557
34983
34738
...
24382
34248
32646
6629
34264
3815
21029
\
\
Updated
May 8,
April 21,
April 21,
April 21,
April 21,
March 26,
March 26,
Date Loss Share Type Agreement Terminated Termination Date
2015
NaN
NaN
NaN
2015
NaN
NaN
NaN
2015
none
NaN
NaN
2015
none
NaN
NaN
2015
none
NaN
NaN
2015
none
NaN
NaN
2015
none
NaN
NaN
...
...
...
...
June 5, 2012
none
NaN
NaN
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
533
534
535
536
537
538
February
August
November
February
March
March
10,
19,
18,
18,
17,
17,
2004
2014
2002
2003
2005
2005
none
none
none
none
none
none
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
NaN
[539 rows x 10 columns]]
Note: The data from the above URL changes every Monday so the resulting data above and the data below may be
slightly different.
Read in the content of the file from the above URL and pass it to read_html as a string
In [211]: with open(file_path, 'r') as f:
.....:
dfs = read_html(f.read())
.....:
In [212]: dfs
Out[212]:
[
Bank Name
0
Banks of Wisconsin d/b/a Bank of Kenosha
1
Central Arizona Bank
2
Sunrise Bank
3
Pisgah Community Bank
4
Douglas County Bank
5
Parkway Bank
6
Chipola Community Bank
..
...
499
Hamilton Bank, NAEn Espanol
500
Sinclair National Bank
501
Superior Bank, FSB
502
Malta National Bank
503
First Alliance Bank & Trust Co.
504
National State Bank of Metropolis
505
Bank of Honolulu
0
1
2
3
4
5
6
..
499
500
501
502
503
504
505
Acquiring Institution
North Shore Bank, FSB
Western State Bank
Synovus Bank
Capital Bank, N.A.
Hamilton State Bank
CertusBank, National Association
First Federal Bank of Florida
...
Israel Discount Bank of New York
Delta Trust & Bank
Superior Federal, FSB
North Valley Bank
Southern New Hampshire Bank & Trust
Banterra Bank of Marion
Bank of the Orient
City
Kenosha
Scottsdale
Valdosta
Asheville
Douglasville
Lenoir
Marianna
...
Miami
Gravette
Hinsdale
Malta
Manchester
Metropolis
Honolulu
Closing
May 31,
May 14,
May 10,
May 10,
April 26,
April 26,
April 19,
January 11,
September 7,
July 27,
May 3,
February 2,
December 14,
October 13,
Date
2013
2013
2013
2013
2013
2013
2013
...
2002
2001
2001
2001
2001
2000
2000
ST
WI
AZ
GA
NC
GA
NC
FL
..
FL
AR
IL
OH
NH
IL
HI
CERT
35386
34527
58185
58701
21649
57158
58034
...
24382
34248
32646
6629
34264
3815
21029
\
Updated
May 31,
May 20,
May 21,
May 14,
May 16,
May 17,
May 16,
June 5,
February 10,
June 5,
November 18,
February 18,
March 17,
March 17,
Date
2013
2013
2013
2013
2013
2013
2013
...
2012
2004
2012
2002
2003
2005
2005
[506 rows x 7 columns]]
You can even pass in an instance of StringIO if you so desire
24.3. HTML
745
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [213]: with open(file_path, 'r') as f:
.....:
sio = StringIO(f.read())
.....:
In [214]: dfs = read_html(sio)
In [215]: dfs
Out[215]:
[
Bank Name
0
Banks of Wisconsin d/b/a Bank of Kenosha
1
Central Arizona Bank
2
Sunrise Bank
3
Pisgah Community Bank
4
Douglas County Bank
5
Parkway Bank
6
Chipola Community Bank
..
...
499
Hamilton Bank, NAEn Espanol
500
Sinclair National Bank
501
Superior Bank, FSB
502
Malta National Bank
503
First Alliance Bank & Trust Co.
504
National State Bank of Metropolis
505
Bank of Honolulu
0
1
2
3
4
5
6
..
499
500
501
502
503
504
505
Acquiring Institution
North Shore Bank, FSB
Western State Bank
Synovus Bank
Capital Bank, N.A.
Hamilton State Bank
CertusBank, National Association
First Federal Bank of Florida
...
Israel Discount Bank of New York
Delta Trust & Bank
Superior Federal, FSB
North Valley Bank
Southern New Hampshire Bank & Trust
Banterra Bank of Marion
Bank of the Orient
City
Kenosha
Scottsdale
Valdosta
Asheville
Douglasville
Lenoir
Marianna
...
Miami
Gravette
Hinsdale
Malta
Manchester
Metropolis
Honolulu
Closing
May 31,
May 14,
May 10,
May 10,
April 26,
April 26,
April 19,
January 11,
September 7,
July 27,
May 3,
February 2,
December 14,
October 13,
Date
2013
2013
2013
2013
2013
2013
2013
...
2002
2001
2001
2001
2001
2000
2000
ST
WI
AZ
GA
NC
GA
NC
FL
..
FL
AR
IL
OH
NH
IL
HI
CERT
35386
34527
58185
58701
21649
57158
58034
...
24382
34248
32646
6629
34264
3815
21029
\
Updated
May 31,
May 20,
May 21,
May 14,
May 16,
May 17,
May 16,
June 5,
February 10,
June 5,
November 18,
February 18,
March 17,
March 17,
Date
2013
2013
2013
2013
2013
2013
2013
...
2012
2004
2012
2002
2003
2005
2005
[506 rows x 7 columns]]
Note: The following examples are not run by the IPython evaluator due to the fact that having so many networkaccessing functions slows down the documentation build. If you spot an error or an example that doesn’t run, please
do not hesitate to report it over on pandas GitHub issues page.
Read a URL and match a table that contains specific text
match = 'Metcalf Bank'
df_list = read_html(url, match=match)
Specify a header row (by default | elements are used to form the column index); if specified, the header row is
taken from the data minus the parsed header elements ( | elements).
746
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
dfs = read_html(url, header=0)
Specify an index column
dfs = read_html(url, index_col=0)
Specify a number of rows to skip
dfs = read_html(url, skiprows=0)
Specify a number of rows to skip using a list (xrange (Python 2 only) works as well)
dfs = read_html(url, skiprows=range(2))
Don’t infer numeric and date types
dfs = read_html(url, infer_types=False)
Specify an HTML attribute
dfs1 = read_html(url, attrs={'id': 'table'})
dfs2 = read_html(url, attrs={'class': 'sortable'})
print(np.array_equal(dfs1[0], dfs2[0])) # Should be True
Use some combination of the above
dfs = read_html(url, match='Metcalf Bank', index_col=0)
Read in pandas to_html output (with some loss of floating point precision)
df = DataFrame(randn(2, 2))
s = df.to_html(float_format='{0:.40g}'.format)
dfin = read_html(s, index_col=0)
The lxml backend will raise an error on a failed parse if that is the only parser you provide (if you only have a single
parser you can provide just a string, but it is considered good practice to pass a list with one string if, for example, the
function expects a sequence of strings)
dfs = read_html(url, 'Metcalf Bank', index_col=0, flavor=['lxml'])
or
dfs = read_html(url, 'Metcalf Bank', index_col=0, flavor='lxml')
However, if you have bs4 and html5lib installed and pass None or [’lxml’, ’bs4’] then the parse will most
likely succeed. Note that as soon as a parse succeeds, the function will return.
dfs = read_html(url, 'Metcalf Bank', index_col=0, flavor=['lxml', 'bs4'])
24.3.2 Writing to HTML files
DataFrame objects have an instance method to_html which renders the contents of the DataFrame as an HTML
table. The function arguments are as in the method to_string described above.
Note:
Not all of the possible options for DataFrame.to_html are shown here for brevity’s sake.
to_html() for the full set of options.
See
24.3. HTML
747
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [216]: df = DataFrame(randn(2, 2))
In [217]: df
Out[217]:
0
1
0 -0.184744 0.496971
1 -0.856240 1.857977
In [218]: print(df.to_html()) # raw html
|
0 |
1 |
| 0 |
-0.184744 |
0.496971 |
| 1 |
-0.856240 |
1.857977 |
HTML:
The columns argument will limit the columns shown
In [219]: print(df.to_html(columns=[0]))
|
0 |
| 0 |
-0.184744 |
| 1 |
-0.856240 |
HTML:
float_format takes a Python callable to control the precision of floating point values
748
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [220]: print(df.to_html(float_format='{0:.10f}'.format))
|
0 |
1 |
| 0 |
-0.1847438576 |
0.4969711327 |
| 1 |
-0.8562396763 |
1.8579766508 |
HTML:
bold_rows will make the row labels bold by default, but you can turn that off
In [221]: print(df.to_html(bold_rows=False))
|
0 |
1 |
| 0 |
-0.184744 |
0.496971 |
| 1 |
-0.856240 |
1.857977 |
The classes argument provides the ability to give the resulting HTML table CSS classes. Note that these classes
are appended to the existing ’dataframe’ class.
In [222]: print(df.to_html(classes=['awesome_table_class', 'even_more_awesome_class']))
|
0 |
24.3. HTML
749
pandas: powerful Python data analysis toolkit, Release 0.16.1
1 |
| 0 |
-0.184744 |
0.496971 |
| 1 |
-0.856240 |
1.857977 |
Finally, the escape argument allows you to control whether the “<”, “>” and “&” characters escaped in the resulting
HTML (by default it is True). So to get the HTML without escaped characters pass escape=False
In [223]: df = DataFrame({'a': list('&<>'), 'b': randn(3)})
Escaped:
In [224]: print(df.to_html())
|
a |
b |
| 0 |
& |
-0.474063 |
| 1 |
< |
-0.230305 |
| 2 |
> |
-0.400654 |
Not escaped:
In [225]: print(df.to_html(escape=False))
|
750
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
a |
b |
| 0 |
& |
-0.474063 |
| 1 |
< |
-0.230305 |
| 2 |
> |
-0.400654 |
Note: Some browsers may not show a difference in the rendering of the previous two HTML tables.
24.4 Excel files
The read_excel() method can read Excel 2003 (.xls) and Excel 2007 (.xlsx) files using the xlrd Python
module and use the same parsing code as the above to convert tabular data into a DataFrame. See the cookbook for
some advanced strategies
24.4.1 Reading Excel Files
New in version 0.16.
read_excel can read more than one sheet, by setting sheetname to either a list of sheet names, a list of sheet
positions, or None to read all sheets.
New in version 0.13.
Sheets can be specified by sheet index or sheet name, using an integer or string, respectively.
New in version 0.12.
ExcelFile has been moved to the top level namespace.
There are two approaches to reading an excel file. The read_excel function and the ExcelFile class.
read_excel is for reading one file with file-specific arguments (ie. identical data formats across sheets).
ExcelFile is for reading one file with sheet-specific arguments (ie. various data formats across sheets). Choosing
the approach is largely a question of code readability and execution speed.
Equivalent class and function approaches to read a single sheet:
# using the ExcelFile class
xls = pd.ExcelFile('path_to_file.xls')
data = xls.parse('Sheet1', index_col=None, na_values=['NA'])
24.4. Excel files
751
pandas: powerful Python data analysis toolkit, Release 0.16.1
# using the read_excel function
data = read_excel('path_to_file.xls', 'Sheet1', index_col=None, na_values=['NA'])
Equivalent class and function approaches to read multiple sheets:
data = {}
# For when Sheet1's format differs from Sheet2
xls = pd.ExcelFile('path_to_file.xls')
data['Sheet1'] = xls.parse('Sheet1', index_col=None, na_values=['NA'])
data['Sheet2'] = xls.parse('Sheet2', index_col=1)
# For when Sheet1's format is identical to Sheet2
data = read_excel('path_to_file.xls', ['Sheet1','Sheet2'], index_col=None, na_values=['NA'])
Specifying Sheets
Note: The second argument is sheetname, not to be confused with ExcelFile.sheet_names
Note: An ExcelFile’s attribute sheet_names provides access to a list of sheets.
• The arguments sheetname allows specifying the sheet or sheets to read.
• The default value for sheetname is 0, indicating to read the first sheet
• Pass a string to refer to the name of a particular sheet in the workbook.
• Pass an integer to refer to the index of a sheet. Indices follow Python convention, beginning at 0.
• Pass a list of either strings or integers, to return a dictionary of specified sheets.
• Pass a None to return a dictionary of all available sheets.
# Returns a DataFrame
read_excel('path_to_file.xls', 'Sheet1', index_col=None, na_values=['NA'])
Using the sheet index:
# Returns a DataFrame
read_excel('path_to_file.xls', 0, index_col=None, na_values=['NA'])
Using all default values:
# Returns a DataFrame
read_excel('path_to_file.xls')
Using None to get all sheets:
# Returns a dictionary of DataFrames
read_excel('path_to_file.xls',sheetname=None)
Using a list to get multiple sheets:
# Returns the 1st and 4th sheet, as a dictionary of DataFrames.
read_excel('path_to_file.xls',sheetname=['Sheet1',3])
752
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
Parsing Specific Columns
It is often the case that users will insert columns to do temporary computations in Excel and you may not want to read
in those columns. read_excel takes a parse_cols keyword to allow you to specify a subset of columns to parse.
If parse_cols is an integer, then it is assumed to indicate the last column to be parsed.
read_excel('path_to_file.xls', 'Sheet1', parse_cols=2)
If parse_cols is a list of integers, then it is assumed to be the file column indices to be parsed.
read_excel('path_to_file.xls', 'Sheet1', parse_cols=[0, 2, 3])
Cell Converters
It is possible to transform the contents of Excel cells via the converters option. For instance, to convert a column to
boolean:
read_excel('path_to_file.xls', 'Sheet1', converters={'MyBools': bool})
This options handles missing values and treats exceptions in the converters as missing data. Transformations are
applied cell by cell rather than to the column as a whole, so the array dtype is not guaranteed. For instance, a column
of integers with missing values cannot be transformed to an array with integer dtype, because NaN is strictly a float.
You can manually mask missing data to recover integer dtype:
cfun = lambda x: int(x) if x else -1
read_excel('path_to_file.xls', 'Sheet1', converters={'MyInts': cfun})
24.4.2 Writing Excel Files
To write a DataFrame object to a sheet of an Excel file, you can use the to_excel instance method. The arguments
are largely the same as to_csv described above, the first argument being the name of the excel file, and the optional
second argument the name of the sheet to which the DataFrame should be written. For example:
df.to_excel('path_to_file.xlsx', sheet_name='Sheet1')
Files with a .xls extension will be written using xlwt and those with a .xlsx extension will be written using
xlsxwriter (if available) or openpyxl.
The DataFrame will be written in a way that tries to mimic the REPL output. One difference from 0.12.0 is that the
index_label will be placed in the second row instead of the first. You can get the previous behaviour by setting
the merge_cells option in to_excel() to False:
df.to_excel('path_to_file.xlsx', index_label='label', merge_cells=False)
The Panel class also has a to_excel instance method, which writes each DataFrame in the Panel to a separate sheet.
In order to write separate DataFrames to separate sheets in a single Excel file, one can pass an ExcelWriter.
with ExcelWriter('path_to_file.xlsx') as writer:
df1.to_excel(writer, sheet_name='Sheet1')
df2.to_excel(writer, sheet_name='Sheet2')
Note: Wringing a little more performance out of read_excel Internally, Excel stores all numeric data as floats.
Because this can produce unexpected behavior when reading in data, pandas defaults to trying to convert integers to
floats if it doesn’t lose information (1.0 --> 1). You can pass convert_float=False to disable this behavior,
which may give a slight performance improvement.
24.4. Excel files
753
pandas: powerful Python data analysis toolkit, Release 0.16.1
24.4.3 Excel writer engines
New in version 0.13.
pandas chooses an Excel writer via two methods:
1. the engine keyword argument
2. the filename extension (via the default specified in config options)
By default, pandas uses the XlsxWriter for .xlsx and openpyxl for .xlsm files and xlwt for .xls
files. If you have multiple engines installed, you can set the default engine through setting the config options
io.excel.xlsx.writer and io.excel.xls.writer. pandas will fall back on openpyxl for .xlsx files
if Xlsxwriter is not available.
To specify which writer you want to use, you can pass an engine keyword argument to to_excel and to
ExcelWriter. The built-in engines are:
• openpyxl: This includes stable support for OpenPyxl 1.6.1 up to but not including 2.0.0, and experimental
support for OpenPyxl 2.0.0 and later.
• xlsxwriter
• xlwt
# By setting the 'engine' in the DataFrame and Panel 'to_excel()' methods.
df.to_excel('path_to_file.xlsx', sheet_name='Sheet1', engine='xlsxwriter')
# By setting the 'engine' in the ExcelWriter constructor.
writer = ExcelWriter('path_to_file.xlsx', engine='xlsxwriter')
# Or via pandas configuration.
from pandas import options
options.io.excel.xlsx.writer = 'xlsxwriter'
df.to_excel('path_to_file.xlsx', sheet_name='Sheet1')
24.5 Clipboard
A handy way to grab data is to use the read_clipboard method, which takes the contents of the clipboard buffer
and passes them to the read_table method. For instance, you can copy the following text to the clipboard (CTRL-C
on many operating systems):
A B C
x 1 4 p
y 2 5 q
z 3 6 r
And then import the data directly to a DataFrame by calling:
clipdf = pd.read_clipboard()
In [226]: clipdf
Out[226]:
A B C
x 1 4 p
y 2 5 q
z 3 6 r
754
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
The to_clipboard method can be used to write the contents of a DataFrame to the clipboard. Following which
you can paste the clipboard contents into other applications (CTRL-V on many operating systems). Here we illustrate
writing a DataFrame into clipboard and reading it back.
In [227]: df=pd.DataFrame(randn(5,3))
In [228]: df
Out[228]:
0
1
2
0 -0.288267 -0.084905 0.004772
1 1.382989 0.343635 -1.253994
2 -0.124925 0.212244 0.496654
3 0.525417 1.238640 -1.210543
4 -1.175743 -0.172372 -0.734129
In [229]: df.to_clipboard()
In [230]: pd.read_clipboard()
Out[230]:
0
1
2
0 -0.288267 -0.084905 0.004772
1 1.382989 0.343635 -1.253994
2 -0.124925 0.212244 0.496654
3 0.525417 1.238640 -1.210543
4 -1.175743 -0.172372 -0.734129
We can see that we got the same content back, which we had earlier written to the clipboard.
Note: You may need to install xclip or xsel (with gtk or PyQt4 modules) on Linux to use these methods.
24.6 Pickling
All pandas objects are equipped with to_pickle methods which use Python’s cPickle module to save data
structures to disk using the pickle format.
In [231]: df
Out[231]:
0
1
2
0 -0.288267 -0.084905 0.004772
1 1.382989 0.343635 -1.253994
2 -0.124925 0.212244 0.496654
3 0.525417 1.238640 -1.210543
4 -1.175743 -0.172372 -0.734129
In [232]: df.to_pickle('foo.pkl')
The read_pickle function in the pandas namespace can be used to load any pickled pandas object (or any other
pickled object) from file:
In [233]: read_pickle('foo.pkl')
Out[233]:
0
1
2
0 -0.288267 -0.084905 0.004772
1 1.382989 0.343635 -1.253994
2 -0.124925 0.212244 0.496654
24.6. Pickling
755
pandas: powerful Python data analysis toolkit, Release 0.16.1
3 0.525417 1.238640 -1.210543
4 -1.175743 -0.172372 -0.734129
Warning: Loading pickled data received from untrusted sources can be unsafe.
See: http://docs.python.org/2.7/library/pickle.html
Warning: Several internal refactorings, 0.13 (Series Refactoring), and 0.15 (Index Refactoring), preserve compatibility with pickles created prior to these versions. However, these must be read with pd.read_pickle, rather
than the default python pickle.load. See this question for a detailed explanation.
Note: These methods were previously pd.save and pd.load, prior to 0.12.0, and are now deprecated.
24.7 msgpack (experimental)
New in version 0.13.0.
Starting in 0.13.0, pandas is supporting the msgpack format for object serialization. This is a lightweight portable
binary format, similar to binary JSON, that is highly space efficient, and provides good performance both on the
writing (serialization), and reading (deserialization).
Warning: This is a very new feature of pandas. We intend to provide certain optimizations in the io of the
msgpack data. Since this is marked as an EXPERIMENTAL LIBRARY, the storage format may not be stable
until a future release.
In [234]: df = DataFrame(np.random.rand(5,2),columns=list('AB'))
In [235]: df.to_msgpack('foo.msg')
In [236]: pd.read_msgpack('foo.msg')
Out[236]:
A
B
0 0.154336 0.710999
1 0.398096 0.765220
2 0.586749 0.293052
3 0.290293 0.710783
4 0.988593 0.062106
In [237]: s = Series(np.random.rand(5),index=date_range('20130101',periods=5))
You can pass a list of objects and you will receive them back on deserialization.
In [238]: pd.to_msgpack('foo.msg', df, 'foo', np.array([1,2,3]), s)
In [239]: pd.read_msgpack('foo.msg')
Out[239]:
[
A
B
0 0.154336 0.710999
1 0.398096 0.765220
2 0.586749 0.293052
3 0.290293 0.710783
4 0.988593 0.062106, u'foo', array([1, 2, 3]), 2013-01-01
2013-01-02
0.235907
2013-01-03
0.712756
756
0.690810
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
2013-01-04
0.119599
2013-01-05
0.023493
Freq: D, dtype: float64]
You can pass iterator=True to iterate over the unpacked results
In [240]: for o in pd.read_msgpack('foo.msg',iterator=True):
.....:
print o
.....:
A
B
0 0.154336 0.710999
1 0.398096 0.765220
2 0.586749 0.293052
3 0.290293 0.710783
4 0.988593 0.062106
foo
[1 2 3]
2013-01-01
0.690810
2013-01-02
0.235907
2013-01-03
0.712756
2013-01-04
0.119599
2013-01-05
0.023493
Freq: D, dtype: float64
You can pass append=True to the writer to append to an existing pack
In [241]: df.to_msgpack('foo.msg',append=True)
In [242]: pd.read_msgpack('foo.msg')
Out[242]:
[
A
B
0 0.154336 0.710999
1 0.398096 0.765220
2 0.586749 0.293052
3 0.290293 0.710783
4 0.988593 0.062106, u'foo', array([1, 2, 3]), 2013-01-01
2013-01-02
0.235907
2013-01-03
0.712756
2013-01-04
0.119599
2013-01-05
0.023493
Freq: D, dtype: float64,
A
B
0 0.154336 0.710999
1 0.398096 0.765220
2 0.586749 0.293052
3 0.290293 0.710783
4 0.988593 0.062106]
0.690810
Unlike other io methods, to_msgpack is available on both a per-object basis, df.to_msgpack() and using the
top-level pd.to_msgpack(...) where you can pack arbitrary collections of python lists, dicts, scalars, while
intermixing pandas objects.
In [243]: pd.to_msgpack('foo2.msg', { 'dict' : [ { 'df' : df }, { 'string' : 'foo' }, { 'scalar' : 1.
In [244]: pd.read_msgpack('foo2.msg')
Out[244]:
{u'dict': ({u'df':
A
0 0.154336 0.710999
1 0.398096 0.765220
2 0.586749 0.293052
24.7. msgpack (experimental)
B
757
pandas: powerful Python data analysis toolkit, Release 0.16.1
3 0.290293 0.710783
4 0.988593 0.062106},
{u'string': u'foo'},
{u'scalar': 1.0},
{u's': 2013-01-01
0.690810
2013-01-02
0.235907
2013-01-03
0.712756
2013-01-04
0.119599
2013-01-05
0.023493
Freq: D, dtype: float64})}
24.7.1 Read/Write API
Msgpacks can also be read from and written to strings.
In [245]: df.to_msgpack()
Out[245]: '\x84\xa6blocks\x91\x86\xa5items\x86\xa4name\xc0\xa5dtype\x11\xa8compress\xc0\xa4data\x92\x
Furthermore you can concatenate the strings to produce a list of the original objects.
In [246]: pd.read_msgpack(df.to_msgpack() + s.to_msgpack())
Out[246]:
[
A
B
0 0.154336 0.710999
1 0.398096 0.765220
2 0.586749 0.293052
3 0.290293 0.710783
4 0.988593 0.062106, 2013-01-01
0.690810
2013-01-02
0.235907
2013-01-03
0.712756
2013-01-04
0.119599
2013-01-05
0.023493
Freq: D, dtype: float64]
24.8 HDF5 (PyTables)
HDFStore is a dict-like object which reads and writes pandas using the high performance HDF5 format using the
excellent PyTables library. See the cookbook for some advanced strategies
Warning: As of version 0.15.0, pandas requires PyTables >= 3.0.0. Stores written with prior versions of
pandas / PyTables >= 2.3 are fully compatible (this was the previous minimum PyTables required version).
Warning: There is a PyTables indexing bug which may appear when querying stores using an index. If you
see a subset of results being returned, upgrade to PyTables >= 3.2. Stores created previously will need to be
rewritten using the updated version.
In [247]: store = HDFStore('store.h5')
In [248]: print(store)
File path: store.h5
Empty
758
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
Objects can be written to the file just like adding key-value pairs to a dict:
In [249]: np.random.seed(1234)
In [250]: index = date_range('1/1/2000', periods=8)
In [251]: s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])
In [252]: df = DataFrame(randn(8, 3), index=index,
.....:
columns=['A', 'B', 'C'])
.....:
In [253]: wp = Panel(randn(2, 5, 4), items=['Item1', 'Item2'],
.....:
major_axis=date_range('1/1/2000', periods=5),
.....:
minor_axis=['A', 'B', 'C', 'D'])
.....:
# store.put('s', s) is an equivalent method
In [254]: store['s'] = s
In [255]: store['df'] = df
In [256]: store['wp'] = wp
# the type of stored data
In [257]: store.root.wp._v_attrs.pandas_type
Out[257]: 'wide'
In [258]: store
Out[258]:
File path: store.h5
/df
frame
(shape->[8,3])
/s
series
(shape->[5])
/wp
wide
(shape->[2,5,4])
In a current or later Python session, you can retrieve stored objects:
# store.get('df') is an equivalent method
In [259]: store['df']
Out[259]:
A
B
C
2000-01-01 0.887163 0.859588 -0.636524
2000-01-02 0.015696 -2.242685 1.150036
2000-01-03 0.991946 0.953324 -2.021255
2000-01-04 -0.334077 0.002118 0.405453
2000-01-05 0.289092 1.321158 -1.546906
2000-01-06 -0.202646 -0.655969 0.193421
2000-01-07 0.553439 1.318152 -0.469305
2000-01-08 0.675554 -1.817027 -0.183109
# dotted (attribute) access provides get as well
In [260]: store.df
Out[260]:
A
B
C
2000-01-01 0.887163 0.859588 -0.636524
2000-01-02 0.015696 -2.242685 1.150036
2000-01-03 0.991946 0.953324 -2.021255
2000-01-04 -0.334077 0.002118 0.405453
24.8. HDF5 (PyTables)
759
pandas: powerful Python data analysis toolkit, Release 0.16.1
2000-01-05 0.289092 1.321158 -1.546906
2000-01-06 -0.202646 -0.655969 0.193421
2000-01-07 0.553439 1.318152 -0.469305
2000-01-08 0.675554 -1.817027 -0.183109
Deletion of the object specified by the key
# store.remove('wp') is an equivalent method
In [261]: del store['wp']
In [262]: store
Out[262]:
File path: store.h5
/df
frame
(shape->[8,3])
/s
series
(shape->[5])
Closing a Store, Context Manager
In [263]: store.close()
In [264]: store
Out[264]:
File path: store.h5
File is CLOSED
In [265]: store.is_open
Out[265]: False
# Working with, and automatically closing the store with the context
# manager
In [266]: with HDFStore('store.h5') as store:
.....:
store.keys()
.....:
24.8.1 Read/Write API
HDFStore supports an top-level API using read_hdf for reading and to_hdf for writing, similar to how
read_csv and to_csv work. (new in 0.11.0)
In [267]: df_tl = DataFrame(dict(A=list(range(5)), B=list(range(5))))
In [268]: df_tl.to_hdf('store_tl.h5','table',append=True)
In [269]: read_hdf('store_tl.h5', 'table', where = ['index>2'])
Out[269]:
A B
3 3 3
4 4 4
24.8.2 Fixed Format
Note: This was prior to 0.13.0 the Storer format.
760
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
The examples above show storing using put, which write the HDF5 to PyTables in a fixed array format, called the
fixed format. These types of stores are are not appendable once written (though you can simply remove them and
rewrite). Nor are they queryable; they must be retrieved in their entirety. They also do not support dataframes with
non-unique column names. The fixed format stores offer very fast writing and slightly faster reading than table
stores. This format is specified by default when using put or to_hdf or by format=’fixed’ or format=’f’
Warning: A fixed format will raise a TypeError if you try to retrieve using a where .
DataFrame(randn(10,2)).to_hdf('test_fixed.h5','df')
pd.read_hdf('test_fixed.h5','df',where='index>5')
TypeError: cannot pass a where specification when reading a fixed format.
this store must be selected in its entirety
24.8.3 Table Format
HDFStore supports another PyTables format on disk, the table format. Conceptually a table is shaped very
much like a DataFrame, with rows and columns. A table may be appended to in the same or other sessions. In addition, delete & query type operations are supported. This format is specified by format=’table’ or format=’t’
to append or put or to_hdf
New in version 0.13.
This format can be set as an option as well pd.set_option(’io.hdf.default_format’,’table’) to
enable put/append/to_hdf to by default store in the table format.
In [270]: store = HDFStore('store.h5')
In [271]: df1 = df[0:4]
In [272]: df2 = df[4:]
# append data (creates a table automatically)
In [273]: store.append('df', df1)
In [274]: store.append('df', df2)
In [275]: store
Out[275]:
File path: store.h5
/df
frame_table (typ->appendable,nrows->8,ncols->3,indexers->[index])
# select the entire object
In [276]: store.select('df')
Out[276]:
A
B
2000-01-01 0.887163 0.859588
2000-01-02 0.015696 -2.242685
2000-01-03 0.991946 0.953324
2000-01-04 -0.334077 0.002118
2000-01-05 0.289092 1.321158
2000-01-06 -0.202646 -0.655969
2000-01-07 0.553439 1.318152
2000-01-08 0.675554 -1.817027
24.8. HDF5 (PyTables)
C
-0.636524
1.150036
-2.021255
0.405453
-1.546906
0.193421
-0.469305
-0.183109
761
pandas: powerful Python data analysis toolkit, Release 0.16.1
# the type of stored data
In [277]: store.root.df._v_attrs.pandas_type
Out[277]: 'frame_table'
Note: You can also create a table by passing format=’table’ or format=’t’ to a put operation.
24.8.4 Hierarchical Keys
Keys to a store can be specified as a string. These can be in a hierarchical path-name like format (e.g. foo/bar/bah),
which will generate a hierarchy of sub-stores (or Groups in PyTables parlance). Keys can be specified with out the
leading ‘/’ and are ALWAYS absolute (e.g. ‘foo’ refers to ‘/foo’). Removal operations can remove everything in the
sub-store and BELOW, so be careful.
In [278]: store.put('foo/bar/bah', df)
In [279]: store.append('food/orange', df)
In [280]: store.append('food/apple',
df)
In [281]: store
Out[281]:
File path: store.h5
/df
frame_table (typ->appendable,nrows->8,ncols->3,indexers->[index])
/foo/bar/bah
frame
(shape->[8,3])
/food/apple
frame_table (typ->appendable,nrows->8,ncols->3,indexers->[index])
/food/orange
frame_table (typ->appendable,nrows->8,ncols->3,indexers->[index])
# a list of keys are returned
In [282]: store.keys()
Out[282]: ['/df', '/food/apple', '/food/orange', '/foo/bar/bah']
# remove all nodes under this level
In [283]: store.remove('food')
In [284]: store
Out[284]:
File path: store.h5
/df
frame_table (typ->appendable,nrows->8,ncols->3,indexers->[index])
/foo/bar/bah
frame
(shape->[8,3])
24.8.5 Storing Mixed Types in a Table
Storing mixed-dtype data is supported. Strings are stored as a fixed-width using the maximum size of the appended
column. Subsequent appends will truncate strings at this length.
Passing min_itemsize={‘values‘: size} as a parameter to append will set a larger minimum for the string
columns. Storing floats, strings, ints, bools, datetime64 are currently supported. For string
columns, passing nan_rep = ’nan’ to append will change the default nan representation on disk (which converts to/from np.nan), this defaults to nan.
In [285]: df_mixed = DataFrame({ 'A' : randn(8),
.....:
'B' : randn(8),
762
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
.....:
.....:
.....:
.....:
.....:
.....:
.....:
'C' : np.array(randn(8),dtype='float32'),
'string' :'string',
'int' : 1,
'bool' : True,
'datetime64' : Timestamp('20010102')},
index=list(range(8)))
In [286]: df_mixed.ix[3:5,['A', 'B', 'string', 'datetime64']] = np.nan
In [287]: store.append('df_mixed', df_mixed, min_itemsize = {'values': 50})
In [288]: df_mixed1 = store.select('df_mixed')
In [289]: df_mixed1
Out[289]:
A
B
C
0 0.704721 -1.152659 -0.430096
1 -0.785435 0.631979 0.767369
2 0.462060 0.039513 0.984920
3
NaN
NaN 0.270836
4
NaN
NaN 1.391986
5
NaN
NaN 0.079842
6 2.007843 0.152631 -0.399965
7 0.226963 0.164530 -1.027851
bool
True
True
True
True
True
True
True
True
datetime64
2001-01-02
2001-01-02
2001-01-02
NaT
NaT
NaT
2001-01-02
2001-01-02
int
1
1
1
1
1
1
1
1
string
string
string
string
NaN
NaN
NaN
string
string
In [290]: df_mixed1.get_dtype_counts()
Out[290]:
bool
1
datetime64[ns]
1
float32
1
float64
2
int64
1
object
1
dtype: int64
# we have provided a minimum string column size
In [291]: store.root.df_mixed.table
Out[291]:
/df_mixed/table (Table(8,)) ''
description := {
"index": Int64Col(shape=(), dflt=0, pos=0),
"values_block_0": Float64Col(shape=(2,), dflt=0.0, pos=1),
"values_block_1": Float32Col(shape=(1,), dflt=0.0, pos=2),
"values_block_2": Int64Col(shape=(1,), dflt=0, pos=3),
"values_block_3": Int64Col(shape=(1,), dflt=0, pos=4),
"values_block_4": BoolCol(shape=(1,), dflt=False, pos=5),
"values_block_5": StringCol(itemsize=50, shape=(1,), dflt='', pos=6)}
byteorder := 'little'
chunkshape := (689,)
autoindex := True
colindexes := {
"index": Index(6, medium, shuffle, zlib(1)).is_csi=False}
24.8.6 Storing Multi-Index DataFrames
Storing multi-index dataframes as tables is very similar to storing/selecting from homogeneous index DataFrames.
24.8. HDF5 (PyTables)
763
pandas: powerful Python data analysis toolkit, Release 0.16.1
In [292]: index = MultiIndex(levels=[['foo', 'bar', 'baz', 'qux'],
.....:
['one', 'two', 'three']],
.....:
labels=[[0, 0, 0, 1, 1, 2, 2, 3, 3, 3],
.....:
[0, 1, 2, 0, 1, 1, 2, 0, 1, 2]],
.....:
names=['foo', 'bar'])
.....:
In [293]: df_mi = DataFrame(np.random.randn(10, 3), index=index,
.....:
columns=['A', 'B', 'C'])
.....:
In [294]: df_mi
Out[294]:
foo bar
foo one
two
three
bar one
two
baz two
three
qux one
two
three
A
B
C
-0.584718
-0.344766
-0.511881
0.503592
1.363482
1.224574
-1.710715
-0.203933
-1.818499
-0.248432
0.816594
0.528288
0.291205
0.285296
-0.781105
-1.281108
-0.450765
-0.182175
0.047072
-0.617707
-0.081947
-1.068989
0.566534
0.484288
-0.468018
0.875476
0.749164
0.680656
0.394844
-0.682884
In [295]: store.append('df_mi',df_mi)
In [296]: store.select('df_mi')
Out[296]:
A
B
C
foo bar
foo one
-0.584718 0.816594 -0.081947
two
-0.344766 0.528288 -1.068989
three -0.511881 0.291205 0.566534
bar one
0.503592 0.285296 0.484288
two
1.363482 -0.781105 -0.468018
baz two
1.224574 -1.281108 0.875476
three -1.710715 -0.450765 0.749164
qux one
-0.203933 -0.182175 0.680656
two
-1.818499 0.047072 0.394844
three -0.248432 -0.617707 -0.682884
# the levels are automatically included as data columns
In [297]: store.select('df_mi', 'foo=bar')
Out[297]:
A
B
C
foo bar
bar one 0.503592 0.285296 0.484288
two 1.363482 -0.781105 -0.468018
24.8.7 Querying a Table
Warning: This query capabilities have changed substantially starting in 0.13.0. Queries from prior version are
accepted (with a DeprecationWarning) printed if its not string-like.
764
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
select and delete operations have an optional criterion that can be specified to select/delete only a subset of the
data. This allows one to have a very large on-disk table and retrieve only a portion of the data.
A query is specified using the Term class under the hood, as a boolean expression.
• index and columns are supported indexers of a DataFrame
• major_axis, minor_axis, and items are supported indexers of the Panel
• if data_columns are specified, these can be used as additional indexers
Valid comparison operators are:
• =, ==, !=, >, >=, <, <=
Valid boolean expressions are combined with:
• | : or
• & : and
• ( and ) : for grouping
These rules are similar to how boolean expressions are used in pandas for indexing.
Note:
• = will be automatically expanded to the comparison operator ==
• ~ is the not operator, but can only be used in very limited circumstances
• If a list/tuple of expressions is passed they will be combined via &
The following are valid expressions:
• ’index>=date’
• "columns=[’A’, ’D’]"
• "columns in [’A’, ’D’]"
• ’columns=A’
• ’columns==A’
• "~(columns=[’A’,’B’])"
• ’index>df.index[3] & string="bar"’
• ’(index>df.index[3] & index<=df.index[6]) | string="bar"’
• "ts>=Timestamp(’2012-02-01’)"
• "major_axis>=20130101"
The indexers are on the left-hand side of the sub-expression:
• columns, major_axis, ts
The right-hand side of the sub-expression (after a comparison operator) can be:
• functions that will be evaluated, e.g. Timestamp(’2012-02-01’)
• strings, e.g. "bar"
• date-like, e.g. 20130101, or "20130101"
• lists, e.g. "[’A’,’B’]"
• variables that are defined in the local names space, e.g. date
24.8. HDF5 (PyTables)
765
pandas: powerful Python data analysis toolkit, Release 0.16.1
Note: Passing a string to a query by interpolating it into the query expression is not recommended. Simply assign the
string of interest to a variable and use that variable in an expression. For example, do this
string = "HolyMoly'"
store.select('df', 'index == string')
instead of this
string = "HolyMoly'"
store.select('df', 'index == %s' % string)
The latter will not work and will raise a SyntaxError.Note that there’s a single quote followed by a double quote
in the string variable.
If you must interpolate, use the ’%r’ format specifier
store.select('df', 'index == %r' % string)
which will quote string.
Here are some examples:
In [298]: dfq = DataFrame(randn(10,4),columns=list('ABCD'),index=date_range('20130101',periods=10))
In [299]: store.append('dfq',dfq,format='table',data_columns=True)
Use boolean expressions, with in-line function evaluation.
In [300]: store.select('dfq',"index>Timestamp('20130104') & columns=['A', 'B']")
Out[300]:
A
B
2013-01-05 1.210384 0.797435
2013-01-06 -0.850346 1.176812
2013-01-07 0.984188 -0.121728
2013-01-08 0.796595 -0.474021
2013-01-09 -0.804834 -2.123620
2013-01-10 0.334198 0.536784
Use and inline column reference
In [301]: store.select('dfq',where="A>0 or C>0")
Out[301]:
A
B
C
D
2013-01-01 0.436258 -1.703013 0.393711 -0.479324
2013-01-02 -0.299016 0.694103 0.678630 0.239556
2013-01-03 0.151227 0.816127 1.893534 0.639633
2013-01-04 -0.962029 -2.085266 1.930247 -1.735349
2013-01-05 1.210384 0.797435 -0.379811 0.702562
2013-01-07 0.984188 -0.121728 2.365769 0.496143
2013-01-08 0.796595 -0.474021 -0.056696 1.357797
2013-01-10 0.334198 0.536784 -0.743830 -0.320204
Works with a Panel as well.
In [302]: store.append('wp',wp)
In [303]: store
Out[303]:
File path: store.h5
766
Chapter 24. IO Tools (Text, CSV, HDF5, ...)
pandas: powerful Python data analysis toolkit, Release 0.16.1
/df
/df_mi
/df_mixed
/dfq
/foo/bar/bah
/wp
frame_table
frame_table
frame_table
frame_table
frame
wide_table
(typ->appendable,nrows->8,ncols->3,indexers->[index])
(typ->appendable_multi,nrows->10,ncols->5,indexers->[index],dc->
(typ->appendable,nrows->8,ncols->7,indexers->[index])
(typ->appendable,nrows->10,ncols->4,indexers->[index],dc->[A,B,C
(shape->[8,3])
(typ->appendable,nrows->20,ncols->2,indexers->[major_axis,minor_
In [304]: store.select('wp', "major_axis>Timestamp('20000102') & minor_axis=['A', 'B']")
Out[304]:
Dimensions: 2 (items) x 3 (major_axis) x 2 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2000-01-03 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to B
The columns keyword can be supplied to select a list of columns to be returned, this is equivalent to passing a
’columns=list_of_columns_to_filter’:
In [305]: store.select('df', "columns=['A', 'B']")
Out[305]:
A
B
2000-01-01 0.887163 0.859588
2000-01-02 0.015696 -2.242685
2000-01-03 0.991946 0.953324
2000-01-04 -0.334077 0.002118
2000-01-05 0.289092 1.321158
2000-01-06 -0.202646 -0.655969
2000-01-07 0.553439 1.318152
2000-01-08 0.675554 -1.817027
start and stop parameters can be specified to limit the total search space. These are in terms of the total number
of rows in a table.
# this is effectively what
In [306]: wp.to_frame()
Out[306]:
Item1
major
minor
2000-01-01 A
1.058969
B
-0.397840
C
0.337438
D
1.047579
2000-01-02 A
1.045938
B
0.863717
C
-0.122092
...
...
2000-01-04 B
0.036142
C
-2.074978
D
0.247792
2000-01-05 A
-0.897157
B
-0.136795
C
0.018289
D
0.755414
the storage of a Panel looks like
Item2
0.215269
0.841009
-1.445810
-1.401973
-0.100918
-0.548242
-0.144620
...
0.307969
-0.208499
1.033801
-2.400454
2.030604
-1.142631
0.211883
[20 rows x 2 columns]
# limiting the search
In [307]: store.select('wp',"major_axis>20000102 & minor_axis=['A','B']",
.....:
start=0, stop=10)
24.8. HDF5 (PyTables)
767
pandas: powerful Python data analysis toolkit, Release 0.16.1
.....:
Out[307]:
|