Time and time zones

Dealing with time and time zones can be a frustrating experience in any programming language and for any application. pvlib-python relies on pandas and pytz to handle time and time zones. Therefore, the vast majority of the information in this document applies to any time series analysis using pandas and is not specific to pvlib-python.

General functionality

pvlib makes extensive use of pandas due to its excellent time series functionality. Take the time to become familiar with pandas’ Time Series / Date functionality page. It is also worthwhile to become familiar with pure Python’s datetime module, although we usually recommend using the corresponding pandas functionality where possible.

First, we’ll import the libraries that we’ll use to explore the basic time and time zone functionality in python and pvlib.

In [1]: import datetime

In [2]: import pandas as pd

In [3]: import pytz

Finding a time zone

pytz is based on the Olson time zone database. You can obtain a list of all valid time zone strings with pytz.all_timezones. It’s a long list, so we only print every 20th time zone.

In [4]: len(pytz.all_timezones)
Out[4]: 593

In [5]: pytz.all_timezones[::20]
Out[5]: 
['Africa/Abidjan',
 'Africa/Douala',
 'Africa/Mbabane',
 'America/Argentina/Catamarca',
 'America/Belize',
 'America/Curacao',
 'America/Guatemala',
 'America/Kentucky/Louisville',
 'America/Mexico_City',
 'America/Port-au-Prince',
 'America/Sitka',
 'Antarctica/Casey',
 'Asia/Ashkhabad',
 'Asia/Dubai',
 'Asia/Khandyga',
 'Asia/Qatar',
 'Asia/Ujung_Pandang',
 'Atlantic/South_Georgia',
 'Australia/South',
 'Canada/Yukon',
 'Etc/GMT+7',
 'Etc/UCT',
 'Europe/Guernsey',
 'Europe/Paris',
 'Europe/Vienna',
 'Indian/Cocos',
 'NZ',
 'Pacific/Honolulu',
 'Pacific/Samoa',
 'US/Eastern']

Wikipedia’s List of tz database time zones is also good reference.

The pytz.country_timezones function is useful, too.

In [6]: pytz.country_timezones('US')
Out[6]: 
['America/New_York',
 'America/Detroit',
 'America/Kentucky/Louisville',
 'America/Kentucky/Monticello',
 'America/Indiana/Indianapolis',
 'America/Indiana/Vincennes',
 'America/Indiana/Winamac',
 'America/Indiana/Marengo',
 'America/Indiana/Petersburg',
 'America/Indiana/Vevay',
 'America/Chicago',
 'America/Indiana/Tell_City',
 'America/Indiana/Knox',
 'America/Menominee',
 'America/North_Dakota/Center',
 'America/North_Dakota/New_Salem',
 'America/North_Dakota/Beulah',
 'America/Denver',
 'America/Boise',
 'America/Phoenix',
 'America/Los_Angeles',
 'America/Anchorage',
 'America/Juneau',
 'America/Sitka',
 'America/Metlakatla',
 'America/Yakutat',
 'America/Nome',
 'America/Adak',
 'Pacific/Honolulu']

And don’t forget about Python’s filter() function.

In [7]: list(filter(lambda x: 'GMT' in x, pytz.all_timezones))
Out[7]: 
['Etc/GMT',
 'Etc/GMT+0',
 'Etc/GMT+1',
 'Etc/GMT+10',
 'Etc/GMT+11',
 'Etc/GMT+12',
 'Etc/GMT+2',
 'Etc/GMT+3',
 'Etc/GMT+4',
 'Etc/GMT+5',
 'Etc/GMT+6',
 'Etc/GMT+7',
 'Etc/GMT+8',
 'Etc/GMT+9',
 'Etc/GMT-0',
 'Etc/GMT-1',
 'Etc/GMT-10',
 'Etc/GMT-11',
 'Etc/GMT-12',
 'Etc/GMT-13',
 'Etc/GMT-14',
 'Etc/GMT-2',
 'Etc/GMT-3',
 'Etc/GMT-4',
 'Etc/GMT-5',
 'Etc/GMT-6',
 'Etc/GMT-7',
 'Etc/GMT-8',
 'Etc/GMT-9',
 'Etc/GMT0',
 'GMT',
 'GMT+0',
 'GMT-0',
 'GMT0']

Note that while pytz has 'EST' and 'MST', it does not have 'PST'. Use 'Etc/GMT+8' instead, or see Fixed offsets.

Timestamps

pandas.Timestamp and pandas.DatetimeIndex can be created in many ways. Here we focus on the time zone issues surrounding them; see the pandas documentation for more information.

First, create a time zone naive pandas.Timestamp.

In [8]: pd.Timestamp('2015-1-1 00:00')
Out[8]: Timestamp('2015-01-01 00:00:00')

You can specify the time zone using the tz keyword argument or the tz_localize method of Timestamp and DatetimeIndex objects.

In [9]: pd.Timestamp('2015-1-1 00:00', tz='America/Denver')
Out[9]: Timestamp('2015-01-01 00:00:00-0700', tz='America/Denver')

In [10]: pd.Timestamp('2015-1-1 00:00').tz_localize('America/Denver')
Out[10]: Timestamp('2015-01-01 00:00:00-0700', tz='America/Denver')

Localized Timestamps can be converted from one time zone to another.

In [11]: midnight_mst = pd.Timestamp('2015-1-1 00:00', tz='America/Denver')

In [12]: corresponding_utc = midnight_mst.tz_convert('UTC')  # returns a new Timestamp

In [13]: corresponding_utc
Out[13]: Timestamp('2015-01-01 07:00:00+0000', tz='UTC')

It does not make sense to convert a time stamp that has not been localized, and pandas will raise an exception if you try to do so.

In [14]: midnight = pd.Timestamp('2015-1-1 00:00')

In [15]: midnight.tz_convert('UTC')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-d106ce42c2eb> in <module>()
----> 1 midnight.tz_convert('UTC')

pandas/tslib.pyx in pandas.tslib.Timestamp.tz_convert (pandas/tslib.c:14012)()

TypeError: Cannot convert tz-naive Timestamp, use tz_localize to localize

The difference between tz_localize and tz_convert is a common source of confusion for new users. Just remember: localize first, convert later.

Daylight savings time

Some time zones are aware of daylight savings time and some are not. For example the winter time results are the same for US/Mountain and MST, but the summer time results are not.

Note the UTC offset in winter…

In [16]: pd.Timestamp('2015-1-1 00:00').tz_localize('US/Mountain')
Out[16]: Timestamp('2015-01-01 00:00:00-0700', tz='US/Mountain')

In [17]: pd.Timestamp('2015-1-1 00:00').tz_localize('Etc/GMT+7')
Out[17]: Timestamp('2015-01-01 00:00:00-0700', tz='Etc/GMT+7')

vs. the UTC offset in summer…

In [18]: pd.Timestamp('2015-6-1 00:00').tz_localize('US/Mountain')
Out[18]: Timestamp('2015-06-01 00:00:00-0600', tz='US/Mountain')

In [19]: pd.Timestamp('2015-6-1 00:00').tz_localize('Etc/GMT+7')
Out[19]: Timestamp('2015-06-01 00:00:00-0700', tz='Etc/GMT+7')

pandas and pytz make this time zone handling possible because pandas stores all times as integer nanoseconds since January 1, 1970. Here is the pandas time representation of the integers 1 and 1e9.

In [20]: pd.Timestamp(1)
Out[20]: Timestamp('1970-01-01 00:00:00.000000001')

In [21]: pd.Timestamp(1e9)
Out[21]: Timestamp('1970-01-01 00:00:01')

So if we specify times consistent with the specified time zone, pandas will use the same integer to represent them.

# US/Mountain
In [22]: pd.Timestamp('2015-6-1 01:00', tz='US/Mountain').value
Out[22]: 1433142000000000000

# MST
In [23]: pd.Timestamp('2015-6-1 00:00', tz='Etc/GMT+7').value
Out[23]: 1433142000000000000

# Europe/Berlin
In [24]: pd.Timestamp('2015-6-1 09:00', tz='Europe/Berlin').value
Out[24]: 1433142000000000000

# UTC
In [25]: pd.Timestamp('2015-6-1 07:00', tz='UTC').value
Out[25]: 1433142000000000000

# UTC
In [26]: pd.Timestamp('2015-6-1 07:00').value
Out[26]: 1433142000000000000

It’s ultimately these integers that are used when calculating quantities in pvlib such as solar position.

As stated above, pandas will assume UTC if you do not specify a time zone. This is dangerous, and we recommend using localized timeseries, even if it is UTC.

Fixed offsets

The 'Etc/GMT*' time zones mentioned above provide fixed offset specifications, but watch out for the counter-intuitive sign convention.

In [27]: pd.Timestamp('2015-1-1 00:00', tz='Etc/GMT-2')
Out[27]: Timestamp('2015-01-01 00:00:00+0200', tz='Etc/GMT-2')

Fixed offset time zones can also be specified as offset minutes from UTC using pytz.FixedOffset.

In [28]: pd.Timestamp('2015-1-1 00:00', tz=pytz.FixedOffset(120))
Out[28]: Timestamp('2015-01-01 00:00:00+0200', tz='pytz.FixedOffset(120)')

You can also specify the fixed offset directly in the tz_localize method, however, be aware that this is not documented and that the offset must be in seconds, not minutes.

In [29]: pd.Timestamp('2015-1-1 00:00', tz=7200)
Out[29]: Timestamp('2015-01-01 00:00:00+0200', tz='pytz.FixedOffset(120)')

Yet another way to specify a time zone with a fixed offset is by using the string formulation.

In [30]: pd.Timestamp('2015-1-1 00:00+0200')
Out[30]: Timestamp('2015-01-01 00:00:00+0200', tz='pytz.FixedOffset(120)')

Native Python objects

Sometimes it’s convenient to use native Python datetime.date and datetime.datetime objects, so we demonstrate their use next. pandas Timestamp objects can also be created from time zone aware or naive datetime.datetime objects. The behavior is as expected.

# tz naive python datetime.datetime object
In [31]: naive_python_dt = datetime.datetime(2015, 6, 1, 0)

# tz naive pandas Timestamp object
In [32]: pd.Timestamp(naive_python_dt)
Out[32]: Timestamp('2015-06-01 00:00:00')

# tz aware python datetime.datetime object
In [33]: aware_python_dt = pytz.timezone('US/Mountain').localize(naive_python_dt)

# tz aware pandas Timestamp object
In [34]: pd.Timestamp(aware_python_dt)
Out[34]: Timestamp('2015-06-01 00:00:00-0600', tz='US/Mountain')

One thing to watch out for is that python datetime.date objects gain time information when passed to Timestamp.

# tz naive python datetime.date object (no time info)
In [35]: naive_python_date = datetime.date(2015, 6, 1)

# tz naive pandas Timestamp object (time=midnight)
In [36]: pd.Timestamp(naive_python_date)
Out[36]: Timestamp('2015-06-01 00:00:00')

You cannot localize a native Python date object.

# fail
In [37]: pytz.timezone('US/Mountain').localize(naive_python_date)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-37-46064241d938> in <module>()
----> 1 pytz.timezone('US/Mountain').localize(naive_python_date)

/home/docs/checkouts/readthedocs.org/user_builds/pvlib-python/conda/latest/lib/python3.5/site-packages/pytz/tzinfo.py in localize(self, dt, is_dst)
    301         Non-existent
    302         '''
--> 303         if dt.tzinfo is not None:
    304             raise ValueError('Not naive datetime (tzinfo is already set)')
    305 

AttributeError: 'datetime.date' object has no attribute 'tzinfo'

pvlib-specific functionality

Note

This section applies to pvlib >= 0.3. Version 0.2 of pvlib used a Location object’s tz attribute to auto-magically correct for some time zone issues. This behavior was counter-intuitive to many users and was removed in version 0.3.

How does this general functionality interact with pvlib? Perhaps the two most common places to get tripped up with time and time zone issues in solar power analysis occur during data import and solar position calculations.

Data import

Let’s first examine how pvlib handles time when it imports a TMY3 file.

In [38]: import os

In [39]: import inspect

In [40]: import pvlib

# some gymnastics to find the example file
In [41]: pvlib_abspath = os.path.dirname(os.path.abspath(inspect.getfile(pvlib)))

In [42]: file_abspath = os.path.join(pvlib_abspath, 'data', '703165TY.csv')

In [43]: tmy3_data, tmy3_metadata = pvlib.tmy.readtmy3(file_abspath)

In [44]: tmy3_metadata
Out[44]: 
{'Name': '"SAND POINT"',
 'State': 'AK',
 'TZ': -9.0,
 'USAF': 703165,
 'altitude': 7.0,
 'latitude': 55.317,
 'longitude': -160.517}

The metadata has a 'TZ' key with a value of -9.0. This is the UTC offset in hours in which the data has been recorded. The readtmy3() function read the data in the file, created a DataFrame with that data, and then localized the DataFrame’s index to have this fixed offset. Here, we print just a few of the rows and columns of the large dataframe.

In [45]: tmy3_data.index.tz
Out[45]: pytz.FixedOffset(-540)

In [46]: tmy3_data.loc[tmy3_data.index[0:3], ['GHI', 'DNI', 'AOD']]
Out[46]: 
                           GHI  DNI    AOD
datetime                                  
1997-01-01 01:00:00-09:00    0    0  0.051
1997-01-01 02:00:00-09:00    0    0  0.051
1997-01-01 03:00:00-09:00    0    0  0.051

The readtmy2() function also returns a DataFrame with a localized DatetimeIndex.

Solar position

The correct solar position can be immediately calculated from the DataFrame’s index since the index has been localized.

In [47]: solar_position = pvlib.solarposition.get_solarposition(tmy3_data.index,
   ....:                                                        tmy3_metadata['latitude'],
   ....:                                                        tmy3_metadata['longitude'])
   ....: 

In [48]: ax = solar_position.loc[solar_position.index[0:24], ['apparent_zenith', 'apparent_elevation', 'azimuth']].plot()

In [49]: ax.legend(loc=1);

In [50]: ax.axhline(0, color='darkgray');  # add 0 deg line for sunrise/sunset

In [51]: ax.axhline(180, color='darkgray');  # add 180 deg line for azimuth at solar noon

In [52]: ax.set_ylim(-60, 200);  # zoom in, but cuts off full azimuth range

In [53]: ax.set_xlabel('Local time ({})'.format(solar_position.index.tz));

In [54]: ax.set_ylabel('(degrees)');
_images/solar-position.png

According to the US Navy, on January 1, 1997 at Sand Point, Alaska, sunrise was at 10:09 am, solar noon was at 1:46 pm, and sunset was at 5:23 pm. This is consistent with the data plotted above (and depressing).

Solar position (assumed UTC)

What if we had a DatetimeIndex that was not localized, such as the one below? The solar position calculator will assume UTC time.

In [55]: index = pd.DatetimeIndex(start='1997-01-01 01:00', freq='1h', periods=24)

In [56]: index
Out[56]: 
DatetimeIndex(['1997-01-01 01:00:00', '1997-01-01 02:00:00',
               '1997-01-01 03:00:00', '1997-01-01 04:00:00',
               '1997-01-01 05:00:00', '1997-01-01 06:00:00',
               '1997-01-01 07:00:00', '1997-01-01 08:00:00',
               '1997-01-01 09:00:00', '1997-01-01 10:00:00',
               '1997-01-01 11:00:00', '1997-01-01 12:00:00',
               '1997-01-01 13:00:00', '1997-01-01 14:00:00',
               '1997-01-01 15:00:00', '1997-01-01 16:00:00',
               '1997-01-01 17:00:00', '1997-01-01 18:00:00',
               '1997-01-01 19:00:00', '1997-01-01 20:00:00',
               '1997-01-01 21:00:00', '1997-01-01 22:00:00',
               '1997-01-01 23:00:00', '1997-01-02 00:00:00'],
              dtype='datetime64[ns]', freq='H')

In [57]: solar_position_notz = pvlib.solarposition.get_solarposition(index,
   ....:                                                             tmy3_metadata['latitude'],
   ....:                                                             tmy3_metadata['longitude'])
   ....: 

In [58]: ax = solar_position_notz.loc[solar_position_notz.index[0:24], ['apparent_zenith', 'apparent_elevation', 'azimuth']].plot()

In [59]: ax.legend(loc=1);

In [60]: ax.axhline(0, color='darkgray');  # add 0 deg line for sunrise/sunset

In [61]: ax.axhline(180, color='darkgray');  # add 180 deg line for azimuth at solar noon

In [62]: ax.set_ylim(-60, 200);  # zoom in, but cuts off full azimuth range

In [63]: ax.set_xlabel('Time (UTC)');

In [64]: ax.set_ylabel('(degrees)');
_images/solar-position-nolocal.png

This looks like the plot above, but shifted by 9 hours.

Solar position (calculate and convert)

In principle, one could localize the tz-naive solar position data to UTC, and then convert it to the desired time zone.

In [65]: fixed_tz = pytz.FixedOffset(tmy3_metadata['TZ'] * 60)

In [66]: solar_position_hack = solar_position_notz.tz_localize('UTC').tz_convert(fixed_tz)

In [67]: solar_position_hack.index
Out[67]: 
DatetimeIndex(['1996-12-31 16:00:00-09:00', '1996-12-31 17:00:00-09:00',
               '1996-12-31 18:00:00-09:00', '1996-12-31 19:00:00-09:00',
               '1996-12-31 20:00:00-09:00', '1996-12-31 21:00:00-09:00',
               '1996-12-31 22:00:00-09:00', '1996-12-31 23:00:00-09:00',
               '1997-01-01 00:00:00-09:00', '1997-01-01 01:00:00-09:00',
               '1997-01-01 02:00:00-09:00', '1997-01-01 03:00:00-09:00',
               '1997-01-01 04:00:00-09:00', '1997-01-01 05:00:00-09:00',
               '1997-01-01 06:00:00-09:00', '1997-01-01 07:00:00-09:00',
               '1997-01-01 08:00:00-09:00', '1997-01-01 09:00:00-09:00',
               '1997-01-01 10:00:00-09:00', '1997-01-01 11:00:00-09:00',
               '1997-01-01 12:00:00-09:00', '1997-01-01 13:00:00-09:00',
               '1997-01-01 14:00:00-09:00', '1997-01-01 15:00:00-09:00'],
              dtype='datetime64[ns, pytz.FixedOffset(-540)]', freq='H')

In [68]: ax = solar_position_hack.loc[solar_position_hack.index[0:24], ['apparent_zenith', 'apparent_elevation', 'azimuth']].plot()

In [69]: ax.legend(loc=1);

In [70]: ax.axhline(0, color='darkgray');  # add 0 deg line for sunrise/sunset

In [71]: ax.axhline(180, color='darkgray');  # add 180 deg line for azimuth at solar noon

In [72]: ax.set_ylim(-60, 200);  # zoom in, but cuts off full azimuth range

In [73]: ax.set_xlabel('Local time ({})'.format(solar_position_hack.index.tz));

In [74]: ax.set_ylabel('(degrees)');
_images/solar-position-hack.png

Note that the time has been correctly localized and converted, however, the calculation bounds still correspond to the original assumed-UTC range.

For this and other reasons, we recommend that users supply time zone information at the beginning of a calculation rather than localizing and converting the results at the end of a calculation.