Time and time zones#
Dealing with time and time zones can be a frustrating experience in any
programming language and for any application. pvlib-python relies on
pandas
and pytz to handle
time and time zones. Therefore, the vast majority of the information in
this document applies to any time series analysis using pandas and is
not specific to pvlib-python.
General functionality#
pvlib makes extensive use of pandas due to its excellent time series
functionality. Take the time to become familiar with pandas’ Time
Series / Date functionality page.
It is also worthwhile to become familiar with pure Python’s
datetime
module, although we usually recommend
using the corresponding pandas functionality where possible.
First, we’ll import the libraries that we’ll use to explore the basic time and time zone functionality in python and pvlib.
In [1]: import datetime
In [2]: import pandas as pd
In [3]: import pytz
Finding a time zone#
pytz is based on the Olson time zone database. You can obtain a list of
all valid time zone strings with pytz.all_timezones
. It’s a long
list, so we only print every 20th time zone.
In [4]: len(pytz.all_timezones)
Out[4]: 596
In [5]: pytz.all_timezones[::20]
Out[5]:
['Africa/Abidjan',
'Africa/Douala',
'Africa/Mbabane',
'America/Argentina/Catamarca',
'America/Belize',
'America/Cuiaba',
'America/Guadeloupe',
'America/Juneau',
'America/Metlakatla',
'America/Paramaribo',
'America/Scoresbysund',
'America/Yakutat',
'Asia/Aqtobe',
'Asia/Dhaka',
'Asia/Kathmandu',
'Asia/Pontianak',
'Asia/Thimphu',
'Atlantic/Jan_Mayen',
'Australia/North',
'Canada/Pacific',
'Etc/GMT+5',
'Etc/GMT0',
'Europe/Dublin',
'Europe/Moscow',
'Europe/Uzhgorod',
'Indian/Antananarivo',
'Mexico/BajaNorte',
'Pacific/Gambier',
'Pacific/Ponape',
'US/Aleutian']
Wikipedia’s List of tz database time zones is also good reference.
The pytz.country_timezones
function is useful, too.
In [6]: pytz.country_timezones('US')
Out[6]:
['America/New_York',
'America/Detroit',
'America/Kentucky/Louisville',
'America/Kentucky/Monticello',
'America/Indiana/Indianapolis',
'America/Indiana/Vincennes',
'America/Indiana/Winamac',
'America/Indiana/Marengo',
'America/Indiana/Petersburg',
'America/Indiana/Vevay',
'America/Chicago',
'America/Indiana/Tell_City',
'America/Indiana/Knox',
'America/Menominee',
'America/North_Dakota/Center',
'America/North_Dakota/New_Salem',
'America/North_Dakota/Beulah',
'America/Denver',
'America/Boise',
'America/Phoenix',
'America/Los_Angeles',
'America/Anchorage',
'America/Juneau',
'America/Sitka',
'America/Metlakatla',
'America/Yakutat',
'America/Nome',
'America/Adak',
'Pacific/Honolulu']
And don’t forget about Python’s filter()
function.
In [7]: list(filter(lambda x: 'GMT' in x, pytz.all_timezones))
Out[7]:
['Etc/GMT',
'Etc/GMT+0',
'Etc/GMT+1',
'Etc/GMT+10',
'Etc/GMT+11',
'Etc/GMT+12',
'Etc/GMT+2',
'Etc/GMT+3',
'Etc/GMT+4',
'Etc/GMT+5',
'Etc/GMT+6',
'Etc/GMT+7',
'Etc/GMT+8',
'Etc/GMT+9',
'Etc/GMT-0',
'Etc/GMT-1',
'Etc/GMT-10',
'Etc/GMT-11',
'Etc/GMT-12',
'Etc/GMT-13',
'Etc/GMT-14',
'Etc/GMT-2',
'Etc/GMT-3',
'Etc/GMT-4',
'Etc/GMT-5',
'Etc/GMT-6',
'Etc/GMT-7',
'Etc/GMT-8',
'Etc/GMT-9',
'Etc/GMT0',
'GMT',
'GMT+0',
'GMT-0',
'GMT0']
Note that while pytz has 'EST'
and 'MST'
, it does not have
'PST'
. Use 'Etc/GMT+8'
instead, or see Fixed offsets.
Timestamps#
pandas.Timestamp
and pandas.DatetimeIndex
can be created in many ways. Here we focus on the time zone issues
surrounding them; see the pandas documentation for more information.
First, create a time zone naive pandas.Timestamp.
In [8]: pd.Timestamp('2015-1-1 00:00')
Out[8]: Timestamp('2015-01-01 00:00:00')
You can specify the time zone using the tz
keyword argument or the
tz_localize
method of Timestamp and DatetimeIndex objects.
In [9]: pd.Timestamp('2015-1-1 00:00', tz='America/Denver')
Out[9]: Timestamp('2015-01-01 00:00:00-0700', tz='America/Denver')
In [10]: pd.Timestamp('2015-1-1 00:00').tz_localize('America/Denver')
Out[10]: Timestamp('2015-01-01 00:00:00-0700', tz='America/Denver')
Localized Timestamps can be converted from one time zone to another.
In [11]: midnight_mst = pd.Timestamp('2015-1-1 00:00', tz='America/Denver')
In [12]: corresponding_utc = midnight_mst.tz_convert('UTC') # returns a new Timestamp
In [13]: corresponding_utc
Out[13]: Timestamp('2015-01-01 07:00:00+0000', tz='UTC')
It does not make sense to convert a time stamp that has not been localized, and pandas will raise an exception if you try to do so.
In [14]: midnight = pd.Timestamp('2015-1-1 00:00')
In [15]: midnight.tz_convert('UTC')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[15], line 1
----> 1 midnight.tz_convert('UTC')
File timestamps.pyx:2350, in pandas._libs.tslibs.timestamps.Timestamp.tz_convert()
TypeError: Cannot convert tz-naive Timestamp, use tz_localize to localize
The difference between tz_localize
and tz_convert
is a common
source of confusion for new users. Just remember: localize first,
convert later.
Daylight savings time#
Some time zones are aware of daylight savings time and some are not. For example the winter time results are the same for US/Mountain and MST, but the summer time results are not.
Note the UTC offset in winter…
In [16]: pd.Timestamp('2015-1-1 00:00').tz_localize('US/Mountain')
Out[16]: Timestamp('2015-01-01 00:00:00-0700', tz='US/Mountain')
In [17]: pd.Timestamp('2015-1-1 00:00').tz_localize('Etc/GMT+7')
Out[17]: Timestamp('2015-01-01 00:00:00-0700', tz='Etc/GMT+7')
vs. the UTC offset in summer…
In [18]: pd.Timestamp('2015-6-1 00:00').tz_localize('US/Mountain')
Out[18]: Timestamp('2015-06-01 00:00:00-0600', tz='US/Mountain')
In [19]: pd.Timestamp('2015-6-1 00:00').tz_localize('Etc/GMT+7')
Out[19]: Timestamp('2015-06-01 00:00:00-0700', tz='Etc/GMT+7')
pandas and pytz make this time zone handling possible because pandas stores all times as integer nanoseconds since January 1, 1970. Here is the pandas time representation of the integers 1 and 1e9.
In [20]: pd.Timestamp(1)
Out[20]: Timestamp('1970-01-01 00:00:00.000000001')
In [21]: pd.Timestamp(1e9)
Out[21]: Timestamp('1970-01-01 00:00:01')
So if we specify times consistent with the specified time zone, pandas will use the same integer to represent them.
# US/Mountain
In [22]: pd.Timestamp('2015-6-1 01:00', tz='US/Mountain').value
Out[22]: 1433142000000000000
# MST
In [23]: pd.Timestamp('2015-6-1 00:00', tz='Etc/GMT+7').value
Out[23]: 1433142000000000000
# Europe/Berlin
In [24]: pd.Timestamp('2015-6-1 09:00', tz='Europe/Berlin').value
Out[24]: 1433142000000000000
# UTC
In [25]: pd.Timestamp('2015-6-1 07:00', tz='UTC').value
Out[25]: 1433142000000000000
# UTC
In [26]: pd.Timestamp('2015-6-1 07:00').value
Out[26]: 1433142000000000000
It’s ultimately these integers that are used when calculating quantities in pvlib such as solar position.
As stated above, pandas will assume UTC if you do not specify a time zone. This is dangerous, and we recommend using localized timeseries, even if it is UTC.
Fixed offsets#
The 'Etc/GMT*'
time zones mentioned above provide fixed offset
specifications, but watch out for the counter-intuitive sign convention.
In [27]: pd.Timestamp('2015-1-1 00:00', tz='Etc/GMT-2')
Out[27]: Timestamp('2015-01-01 00:00:00+0200', tz='Etc/GMT-2')
Fixed offset time zones can also be specified as offset minutes
from UTC using pytz.FixedOffset
.
In [28]: pd.Timestamp('2015-1-1 00:00', tz=pytz.FixedOffset(120))
Out[28]: Timestamp('2015-01-01 00:00:00+0200', tz='pytz.FixedOffset(120)')
You can also specify the fixed offset directly in the tz_localize
method, however, be aware that this is not documented and that the
offset must be in seconds, not minutes.
In [29]: pd.Timestamp('2015-1-1 00:00', tz=7200)
Out[29]: Timestamp('2015-01-01 00:00:00+0200', tz='UTC+02:00')
Yet another way to specify a time zone with a fixed offset is by using the string formulation.
In [30]: pd.Timestamp('2015-1-1 00:00+0200')
Out[30]: Timestamp('2015-01-01 00:00:00+0200', tz='UTC+02:00')
Native Python objects#
Sometimes it’s convenient to use native Python
datetime.date
and
datetime.datetime
objects, so we demonstrate their
use next. pandas Timestamp objects can also be created from time zone
aware or naive
datetime.datetime
objects. The behavior is as
expected.
# tz naive python datetime.datetime object
In [31]: naive_python_dt = datetime.datetime(2015, 6, 1, 0)
# tz naive pandas Timestamp object
In [32]: pd.Timestamp(naive_python_dt)
Out[32]: Timestamp('2015-06-01 00:00:00')
# tz aware python datetime.datetime object
In [33]: aware_python_dt = pytz.timezone('US/Mountain').localize(naive_python_dt)
# tz aware pandas Timestamp object
In [34]: pd.Timestamp(aware_python_dt)
Out[34]: Timestamp('2015-06-01 00:00:00-0600', tz='US/Mountain')
One thing to watch out for is that python
datetime.date
objects gain time information when
passed to Timestamp
.
# tz naive python datetime.date object (no time info)
In [35]: naive_python_date = datetime.date(2015, 6, 1)
# tz naive pandas Timestamp object (time=midnight)
In [36]: pd.Timestamp(naive_python_date)
Out[36]: Timestamp('2015-06-01 00:00:00')
You cannot localize a native Python date object.
# fail
In [37]: pytz.timezone('US/Mountain').localize(naive_python_date)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[37], line 1
----> 1 pytz.timezone('US/Mountain').localize(naive_python_date)
File ~/checkouts/readthedocs.org/user_builds/pvlib-python/envs/latest/lib/python3.11/site-packages/pytz/tzinfo.py:320, in DstTzInfo.localize(self, dt, is_dst)
261 def localize(self, dt, is_dst=False):
262 '''Convert naive time to local time.
263
264 This method should be used to construct localtimes, rather
(...)
318 Non-existent
319 '''
--> 320 if dt.tzinfo is not None:
321 raise ValueError('Not naive datetime (tzinfo is already set)')
323 # Find the two best possibilities.
AttributeError: 'datetime.date' object has no attribute 'tzinfo'
pvlib-specific functionality#
How does this general functionality interact with pvlib? Perhaps the two most common places to get tripped up with time and time zone issues in solar power analysis occur during data import and solar position calculations.
Data import#
Let’s first examine how pvlib handles time when it imports a TMY3 file.
In [38]: import os
In [39]: import inspect
In [40]: import pvlib
# some gymnastics to find the example file
In [41]: pvlib_abspath = os.path.dirname(os.path.abspath(inspect.getfile(pvlib)))
In [42]: file_abspath = os.path.join(pvlib_abspath, 'data', '703165TY.csv')
In [43]: tmy3_data, tmy3_metadata = pvlib.iotools.read_tmy3(file_abspath, map_variables=True)
In [44]: tmy3_metadata
Out[44]:
{'USAF': 703165,
'Name': '"SAND POINT"',
'State': 'AK',
'TZ': -9.0,
'latitude': 55.317,
'longitude': -160.517,
'altitude': 7.0}
The metadata has a 'TZ'
key with a value of -9.0
. This is the
UTC offset in hours in which the data has been recorded. The
read_tmy3()
function read the data in the file,
created a DataFrame
with that data, and then
localized the DataFrame’s index to have this fixed offset. Here, we
print just a few of the rows and columns of the large dataframe.
In [45]: tmy3_data.index.tz
Out[45]: datetime.timezone(datetime.timedelta(days=-1, seconds=54000))
In [46]: tmy3_data.loc[tmy3_data.index[0:3], ['ghi', 'dni', 'AOD (unitless)']]
Out[46]:
ghi dni AOD (unitless)
1997-01-01 01:00:00-09:00 0 0 0.051
1997-01-01 02:00:00-09:00 0 0 0.051
1997-01-01 03:00:00-09:00 0 0 0.051
The read_tmy2()
function also returns a DataFrame
with a localized DatetimeIndex.
Solar position#
The correct solar position can be immediately calculated from the DataFrame’s index since the index has been localized.
In [47]: solar_position = pvlib.solarposition.get_solarposition(tmy3_data.index,
....: tmy3_metadata['latitude'],
....: tmy3_metadata['longitude'])
....:
In [48]: ax = solar_position.loc[solar_position.index[0:24], ['apparent_zenith', 'apparent_elevation', 'azimuth']].plot()
In [49]: ax.legend(loc=1);
In [50]: ax.axhline(0, color='darkgray'); # add 0 deg line for sunrise/sunset
In [51]: ax.axhline(180, color='darkgray'); # add 180 deg line for azimuth at solar noon
In [52]: ax.set_ylim(-60, 200); # zoom in, but cuts off full azimuth range
In [53]: ax.set_xlabel('Local time ({})'.format(solar_position.index.tz));
In [54]: ax.set_ylabel('(degrees)');
According to the US Navy, on January 1, 2024 at Sand Point, Alaska (55.34N, -160.5W), sunrise was at 10:09 am, solar noon was at 1:46 pm, and sunset was at 5:22 pm. This is consistent with the data plotted above (and depressing).
Solar position (assumed UTC)#
What if we had a DatetimeIndex that was not localized, such as the one below? The solar position calculator will assume UTC time.
In [55]: index = pd.date_range(start='1997-01-01 01:00', freq='1h', periods=24)
In [56]: index
Out[56]:
DatetimeIndex(['1997-01-01 01:00:00', '1997-01-01 02:00:00',
'1997-01-01 03:00:00', '1997-01-01 04:00:00',
'1997-01-01 05:00:00', '1997-01-01 06:00:00',
'1997-01-01 07:00:00', '1997-01-01 08:00:00',
'1997-01-01 09:00:00', '1997-01-01 10:00:00',
'1997-01-01 11:00:00', '1997-01-01 12:00:00',
'1997-01-01 13:00:00', '1997-01-01 14:00:00',
'1997-01-01 15:00:00', '1997-01-01 16:00:00',
'1997-01-01 17:00:00', '1997-01-01 18:00:00',
'1997-01-01 19:00:00', '1997-01-01 20:00:00',
'1997-01-01 21:00:00', '1997-01-01 22:00:00',
'1997-01-01 23:00:00', '1997-01-02 00:00:00'],
dtype='datetime64[ns]', freq='h')
In [57]: solar_position_notz = pvlib.solarposition.get_solarposition(index,
....: tmy3_metadata['latitude'],
....: tmy3_metadata['longitude'])
....:
In [58]: ax = solar_position_notz.loc[solar_position_notz.index[0:24], ['apparent_zenith', 'apparent_elevation', 'azimuth']].plot()
In [59]: ax.legend(loc=1);
In [60]: ax.axhline(0, color='darkgray'); # add 0 deg line for sunrise/sunset
In [61]: ax.axhline(180, color='darkgray'); # add 180 deg line for azimuth at solar noon
In [62]: ax.set_ylim(-60, 200); # zoom in, but cuts off full azimuth range
In [63]: ax.set_xlabel('Time (UTC)');
In [64]: ax.set_ylabel('(degrees)');
This looks like the plot above, but shifted by 9 hours.
Solar position (calculate and convert)#
In principle, one could localize the tz-naive solar position data to UTC, and then convert it to the desired time zone.
In [65]: fixed_tz = pytz.FixedOffset(tmy3_metadata['TZ'] * 60)
In [66]: solar_position_hack = solar_position_notz.tz_localize('UTC').tz_convert(fixed_tz)
In [67]: solar_position_hack.index
Out[67]:
DatetimeIndex(['1996-12-31 16:00:00-09:00', '1996-12-31 17:00:00-09:00',
'1996-12-31 18:00:00-09:00', '1996-12-31 19:00:00-09:00',
'1996-12-31 20:00:00-09:00', '1996-12-31 21:00:00-09:00',
'1996-12-31 22:00:00-09:00', '1996-12-31 23:00:00-09:00',
'1997-01-01 00:00:00-09:00', '1997-01-01 01:00:00-09:00',
'1997-01-01 02:00:00-09:00', '1997-01-01 03:00:00-09:00',
'1997-01-01 04:00:00-09:00', '1997-01-01 05:00:00-09:00',
'1997-01-01 06:00:00-09:00', '1997-01-01 07:00:00-09:00',
'1997-01-01 08:00:00-09:00', '1997-01-01 09:00:00-09:00',
'1997-01-01 10:00:00-09:00', '1997-01-01 11:00:00-09:00',
'1997-01-01 12:00:00-09:00', '1997-01-01 13:00:00-09:00',
'1997-01-01 14:00:00-09:00', '1997-01-01 15:00:00-09:00'],
dtype='datetime64[ns, pytz.FixedOffset(-540)]', freq='h')
In [68]: ax = solar_position_hack.loc[solar_position_hack.index[0:24], ['apparent_zenith', 'apparent_elevation', 'azimuth']].plot()
In [69]: ax.legend(loc=1);
In [70]: ax.axhline(0, color='darkgray'); # add 0 deg line for sunrise/sunset
In [71]: ax.axhline(180, color='darkgray'); # add 180 deg line for azimuth at solar noon
In [72]: ax.set_ylim(-60, 200); # zoom in, but cuts off full azimuth range
In [73]: ax.set_xlabel('Local time ({})'.format(solar_position_hack.index.tz));
In [74]: ax.set_ylabel('(degrees)');
Note that the time has been correctly localized and converted, however, the calculation bounds still correspond to the original assumed-UTC range.
For this and other reasons, we recommend that users supply time zone information at the beginning of a calculation rather than localizing and converting the results at the end of a calculation.