NetCDFTimeIndex

[x] closes #1084
[x] passes git diff upstream/master | flake8 --diff
[x] tests added / passed
[x] whatsnew entry

This work in progress PR is a start on implementing a NetCDFTimeIndex, a subclass of pandas.Index, which closely mimics pandas.DatetimeIndex, but uses netcdftime._netcdftime.datetime objects.  Currently implemented in the new index are:

Partial datetime-string indexing (using strictly ISO8601-format strings, using a date parser implemented by @shoyer in https://github.com/pydata/xarray/issues/1084#issuecomment-274372547)
Field-accessors for year, month, day, hour, minute, second, and microsecond, to enable groupby operations on attributes of date objects

This index is meant as a step towards improving the handling of non-standard calendars and dates outside the range Timestamp('1677-09-21 00:12:43.145225') to Timestamp('2262-04-11 23:47:16.854775807').

For now I have pushed only the code and some tests for the new index; I want to make sure the index is solid and well-tested before we consider integrating it into any of xarray's existing logic or writing any documentation.
Regarding the index, there are a couple remaining outstanding issues (that at least I'm aware of):

Currently one can create non-sensical datetimes using netcdftime._netcdftime.datetime objects.  This means one can attempt to index with an out-of-bounds string or datetime without raising an error.  Could this possibly be addressed upstream?  For example:

In [1]: from netcdftime import DatetimeNoLeap

In [2]: DatetimeNoLeap(2000, 45, 45)
Out[2]: netcdftime._netcdftime.DatetimeNoLeap(2000, 45, 45, 0, 0, 0, 0, -1, 1)

I am looking to enable this index to be used in pandas.Series and pandas.DataFrame objects as well;  this requires implementing a get_value method.  I have taken @shoyer's suggested simplified approach from https://github.com/pydata/xarray/issues/1084#issuecomment-275963433, and tweaked it to also allow for slice indexing, so I think this is most of the way there.  A remaining to-do for me, however, is to implement something to allow for integer-indexing outside of iloc, e.g.  if you have a pandas.Series series, indexing with the syntax series[1] or series[1:3].

Hopefully this is a decent start; in particular I'm not an expert in writing tests so please let me know if there are improvements I can make to the structure and / or style I've used so far.  I'm happy to make changes.  I appreciate your help.

Thanks for the quick feedback on the tests @MaximilianR.  Is this on the right track for doing things a little more idiomatically with pytest?
import pytest

import numpy as np

def netcdftime_date_types():
    pytest.importorskip('netCDF4')

    from netcdftime import (
        DatetimeNoLeap, DatetimeJulian, DatetimeAllLeap,
        DatetimeGregorian, DatetimeProlepticGregorian, Datetime360Day)
    return [DatetimeNoLeap, DatetimeJulian, DatetimeAllLeap,
            DatetimeGregorian, DatetimeProlepticGregorian, Datetime360Day]

@pytest.fixture(params=[])
def index(request):
    from xarray.core.netcdftimeindex import NetCDFTimeIndex

    date_type = request.param
    dates = [date_type(1, 1, 1), date_type(1, 2, 1),
             date_type(2, 1, 1), date_type(2, 2, 1)]
    return NetCDFTimeIndex(dates)

@pytest.mark.parametrize('index', netcdftime_date_types(), indirect=True)
@pytest.mark.parametrize(('field', 'expected'), [
    ('year', [1, 1, 2, 2]),
    ('month', [1, 2, 1, 2]),
    ('day', [1, 1, 1, 1]),
    ('hour', [0, 0, 0, 0]),
    ('minute', [0, 0, 0, 0]),
    ('second', [0, 0, 0, 0]),
    ('microsecond', [0, 0, 0, 0])
], ids=['year', 'month', 'day', 'hour', 'minute', 'second', 'microsecond'])
def test_netcdftimeindex_field_accessors(index, field, expected):
    result = getattr(index, field)
    np.testing.assert_array_equal(result, expected)

@MaximilianR I think I'm getting the hang of it; ignore the above.  I'll push a new update to the PR in a bit.

Not far off, although no need to use things like ids (or even indirect, although you can if you want for that).
More than happy to offer any guidance on tests if helpful - post an example. pytest is really nice, even if it takes a bit of time to get used to

@MaximilianR 6496458 contains an updated version of the file containing the tests, updated to use pytest.   I ended up using the ids keyword in places to clean up the test names that are output when running the tests in verbose mode, but I agree it's not super necessary.
Overall pytest seems to clean things up pretty nicely.  One problem that I wish I had a better solution for happens when I am testing indexing operations in DataArrays, Series, and DataFrames.  There are multiple ways of getting the same answer, which makes these tests a good candidate for using pytest.mark.parametrize; however, one of those ways, using netcdftime._netcdftime.datetime objects directly, depends on the date_type used.
For instance, it would be great if I could write something like:
@pytest.mark.parametrize('sel_arg', [
    '0001',
    slice('0001-01-01', '0001-12-30'),
    [True, True, False, False],
    slice(date_type(1, 1, 1), date_type(1, 12, 30)),
    [date_type(1, 1, 1), date_type(1, 2, 1)]
], ids=['string', 'string-slice', 'bool-list', 'date-slice', 'date-list'])
def test_sel(da, index, sel_arg):
    expected = xr.DataArray([1, 2], coords=[index[:2]], dims=['time'])
    result = da.sel(time=sel_arg)
    assert_identical(result, expected)

But I can't use date_type, which is a fixture in my current setup, in an argument to parametrize.        Right now I've worked around this by resorting back to manually iterating over the cases by writing separate methods, but that's pretty verbose; might you happen to know of a cleaner way of setting things up in this case?
In any event, when you get a chance, please let me know if you have any comments / suggestions on my latest push.  Thanks again for your help.

That looks awesome! Quite a turn around there.
Yes, good point on the date_type. Let me think for a bit on the best way

Currently one can create non-sensical datetimes using netcdftime._netcdftime.datetime objects. This means one can attempt to index with an out-of-bounds string or datetime without raising an error. Could this possibly be addressed upstream?

Yes, this would be best addressed upstream in netcdftime.

@shoyer thanks for your initial review comments.  I'll try and push an update in the next few days.

@shoyer when you get a chance, things are ready for another review.  I think the AppVeyor issues may be due to the version of netCDF4 used.  Should we switch to the conda-forge channel to set up the environment there?

Okay this seems like a netcdftime bug. Can you report this upstream?
On Sat, Feb 11, 2017 at 4:18 PM Spencer Clark<EMAIL_ADDRESS>wrote:

@spencerkclark commented on this pull request.
In xarray/conventions/netcdftimeindex.py
https://github.com/pydata/xarray/pull/1252:

@@ -120,23 +118,31 @@ def get_date_type(self):
return type(self._data[0])

-def assert_all_same_netcdftime_datetimes(data):

from netcdftime._netcdftime import datetime
+def assert_all_valid_date_type(data):

from netcdftime import (

Sorry, I buried this in a comment (#1252 (comment)
https://github.com/pydata/xarray/pull/1252#discussion_r100678111)
above. Confusingly, netcdftime.datetime does not refer to the super class:
In [1]: from netcdftime import datetime, DatetimeAllLeap
In [2]: datetime(1, 1, 1)
Out[2]: netcdftime._netcdftime.DatetimeProlepticGregorian(1, 1, 1, 0, 0, 0, 0, -1, 1)
In [3]: test = DatetimeAllLeap(1, 1, 1)
In [4]: isinstance(test, datetime)
Out[4]: False
In [5]: from netcdftime._netcdftime import datetime as super_datetime
In [6]: isinstance(test, super_datetime)
Out[6]: True
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/pull/1252, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABKS1m9Egh8sElE3KoU08DEv-TvRn7KEks5rbk_JgaJpZM4L3tsR
.

@spencerkclark @shoyer I got some more concrete information c.f. on my previous comment on negative and/or 5 digit dates.  The TRACE simulation outputs netCDF files uses units of thousands of years relative to 1950, and therefore doesn't use 5 integers.  But it does use negative and positive floats...negative for <1950, positive for >1950.
Re: 5 digits, there is growing research interest in very long climate model integrations, e.g. http://www.longrunmip.org/, but even those for now appear <10k yr in duration.
But there are also so called EMICs (Earth Models of Intermediate Complexity) that are cheap to run for 1000s of years.  Although I couldn't immediately find any published results using them for >10k yr duration...
So ultimately I'd say there definitely exists a use-case for negative times (albeit with odd format) and there likely exists a use-case for 5 digit years.  IMHO these are not must-haves for the initial netcdftime implementation but should at least be kept in mind, i.e. code design that makes them not-overly-difficult to introduce eventually.
I suspect there are xarray users with more direct experience with these cases...feel free to chime in

@spencerahill many thanks for looking into this further.  I agree that simulations of tens of thousands of years are beginning to appear on the research landscape, and shouldn't necessarily be a use-case we dismiss.

I got some more concrete information c.f. on my previous comment on negative and/or 5 digit dates. The TRACE simulation outputs netCDF files uses units of thousands of years relative to 1950, and therefore doesn't use 5 integers. But it does use negative and positive floats...negative for <1950, positive for >1950.

I think it's important to note that it doesn't matter how the date information is stored in the original files; what matters is what it looks like once it's decoded.  If someone analyzing this simulation would like to use a NetCDFTimeIndex, it would require that they decode these floats into actual datetimes (which would need five-digit years to express).  So they would need support for five digit (possibly negative) years in partial datetime string indexing.   As a side note though, this decoding is currently not possible with netCDF4.num2date (since "thousands of years" is not an accepted unit there), but I could see a way someone could do the decoding manually.
I'll need to think more about options for date parsing; ideally I would like there to be at least one common string format that one could use to specify dates for both a NetCDFTimeIndex and a DatetimeIndex (with the ISO8601 parser we have now, supporting only 4-digit, positive years, I believe that is currently the case).  If we can achieve that while adding support for five-digit and negative years, that would be great, but perhaps it's best to leave that for a future PR.

Sorry for letting this slip these last couple months — I should have time to make some more meaningful progress on this over the next few weeks.  @shoyer when you get a chance, could you let me know what you think the next important steps are for this PR?  Pending more cleanup on the NetCDFTimeIndex portion (both the index and its tests) are we ready to start thinking about how to include this in xarray's existing logic for decoding and encoding dates?
As a side note, I have an initial attempt at addressing Unidata/netcdftime#8 and Unidata/netcdftime#9 in Unidata/netcdftime#12.

Pending more cleanup on the NetCDFTimeIndex portion (both the index and its tests) are we ready to start thinking about how to include this in xarray's existing logic for decoding and encoding dates?

Yes, this seems about right to me. We'll want to save this for a major release (v0.10), since it will be backwards incompatible.

We'll want to save this for a major release (v0.10), since it will be backwards incompatible.

Right.  To what extent do we want to preserve the current behavior?  As I understand it, what is currently done is regardless of the calendar type, every effort is made to convert decoded datetimes into np.datetime64 objects.  What this means is that the only time one gets netcdftime.datetime objects in an xarray object is when either (or both):

A date present in an array does not exist in the standard calendar (e.g. say '2000-02-30' in a '360_day' calendar).
The dates do not fit into the range of years 1678 to 2262.

The main advantage of doing this is that it enables, wherever possible, 1D arrays of datetimes to be converted to DatetimeIndexes and all the nice things that comes with them:

Field accessors (for use in groupby operations)
Partial datetime string indexing
Resample for downsampling only (accurate upsampling would need to be calendar-aware)
"Not a time" support (i.e. support for missing values).

This practice has one disadvantage -- the calendar type of the datetimes is not preserved.  Therefore if one tries to do timedelta arithmetic between values in the array, (e.g. Mar. 1st, 2001 minus Feb. 1st, 2001) one might get an inaccurate answer depending on what the original calendar type was (as was noted when this was originally implemented, and to be fair, I don't think timedelta arithmetic was even possible on netcdftime.datetime objects back then).
Options
I can think of two ways to proceed in integrating NetCDFTimeIndex into xarray:

Only use a NetCDFTimeIndex where a DatetimeIndex currently cannot be used
Always use netcdftime.datetime objects (and hence NetCDFTimeIndex) for representing dates with non-standard calendars

Tradeoffs

Only use a NetCDFTimeIndex where a DatetimeIndex currently cannot be used

This is perhaps the the least dramatic change one could make.   It would involve not modifying the decoding logic at all (i.e. continuing to be aggressive in attempting to convert netcdftime.datetime objects to np.datetime64 types) and only using a NetCDFTimeIndex when dates were DatetimeIndex incompatible (in other words only when an array of netcdftime.datetime objects arrived at utils.safe_cast_to_index under the existing decoding logic).  The timedelta issue would still remain, but this route would not change the behavior for time arrays with non-standard calendars that could be cast as ordinary DatetimeIndexes (so wherever you could use resample for non-standard calendars before, you could still do so here, and of course wherever you couldn't before, you still couldn't).

Always use netcdftime.datetime objects (and hence NetCDFTimeIndex) for representing dates with non-standard calendars

As has been discussed, the current implementation of NetCDFTimeIndex enables (1) and (2), as well as eliminates the timedelta problem, but it does not enable (3) or (4).  Therefore, for instance, for those who used xarray to downsample arrays indexed by datetimes with a non-standard calendar, whose data did not violate the two bulleted specifications at the top, deciding to unilaterally use netcdftime.datetime objects for all non-standard calendar types would be a regression (though it would preserve the original calendar type).
I suppose this comes down to weighing the importance of addressing the timedelta issue (perhaps more generally preserving calendar types) versus preserving existing behavior that allows (3) and (4) for some cases with non-standard calendars.
Is this an accurate summary of the considerations we should make here?  What are folks' opinions on these tradeoffs?  What might be the preferred route to take?

I'm happy to defer to whatever non-standard calendar users think about be most useful. Possibly @jhamman has an opinion here.

My two cents. I think we want:

Always use netcdftime.datetime objects (and hence NetCDFTimeIndex) for representing dates with non-standard calendars

Presumably, we can address many of the negatives with this approach with follow-on PRs. The fallback logic in xarray right now for non-standard calendars is not something we want to keep and was mostly meant as a stop-gap until this feature came along.

Thanks for jumping in @jhamman -- that is my inclination as well.  I will start along this path unless anyone else has any objections.

@spencerkclark - two questions.

How is this going? Could you use some help with anything?
Pandas provides a convenience function to create arrays of datetime objects (pd.date_range). This function just passes its arguments directly on to DatetimeIndex. I'm wondering if we can provide an xr.date_range function that mimics the pandas version with an additional calendar argument?

How is this going? Could you use some help with anything?

@jhamman sure thing -- I just pushed some updates to show where I'm at.
I've made a first pass at the following:

Resetting the logic for decoding datetimes such that np.datetime64 objects are never used for non-standard calendars
Adding logic to use a NetCDFTimeIndex whenever netcdftime.datetime objects are used in an array being cast as an index (so if one reads in a Dataset from a netCDF file or creates one in Python, which is indexed by a time coordinate that uses netcdftime.datetime objects a NetCDFTimeIndex will be used rather than a generic object-based index)
Adding logic to encode netcdftime.datetime objects when saving out to netCDF files

I've added and updated some tests, but it's possible some may be redundant, and others could still be needed.
Hopefully with some guidance from you and @shoyer I can clean things up.  I'll add a couple comments to the latest diff to highlight a few questions that came up as I was making these changes.

Pandas provides a convenience function to create arrays of datetime objects (pd.date_range). This function just passes its arguments directly on to DatetimeIndex. I'm wondering if we can provide an xr.date_range function that mimics the pandas version with an additional calendar argument?

I agree this would be quite useful.  pandas' implementation is pretty complicated, but perhaps we could do something, possibly more limited, but simpler.

From the CI it also looks like I need to accommodate some updates in pandas in NetCDFTimeIndex.  I will try and look into that tomorrow.

I think your test failures may actually be related to https://github.com/pydata/xarray/pull/1356 in xarray

Pinging folks on this.  Summer is upon us and, generally speaking for the academics among us, is a good time for projects like this.
@spencerkclark looks like you're still looking for guidance w/r/t the last batch of comments from May 10.

From the CI it also looks like I need to accommodate some updates in pandas in NetCDFTimeIndex. I will try and look into that tomorrow.

Did your subsequent commits resolve this?

@spencerahill yes, my subsequent commits resolved the test failures associated both with the pandas update and with #1356.
On a broader note it would be great to make some more progress on this.  I've noted a few specific questions above, but obviously would appreciate feedback on any aspect of these changes.

@jhamman thanks a lot for the comments and for your work towards splitting netcdftime out of netCDF4; I'll try and address them by the end of this week.

@jhamman @shoyer with netcdftime now on its way to becoming a standalone package, this might be at the point where it is ready for some more serious review; there are some failing tests for the zarr backend, but I think those are unrelated (I fixed some that were).
I've held off on writing a what's new entry for now, because I know that we'd like to wait for a major release to merge this (as this contains a breaking change for handling of dates with non-standard calendars).

there are some failing tests for the zarr backend, but I think those are unrelated (I fixed some that were).

Right. Those will be fixed once I merge #1793.

Can we potentially merge this behind a flag for now, e.g., you only get the new behavior if you explicitly set an option, e.g., xr.set_options(enable_netcdftimeindex=True)?
There's something to be said for getting code merged when it's ready, regardless of whether we are currently issuing a breaking release. This would also let the new index get some testing in before we turn it on by default.

@jhamman thanks, those are good questions.

This has some breaking behavior in it (e.g. resample). Do we want a way to enable the user to choose whether or not to use this new index?

Yes, I like @shoyer's idea here, and I'll see what I can do over the next few days.

Tests. Can we make sure we have tests that cover the raised errors for all the things we know won't work (e.g. resample).

Agreed -- I'll make sure to add these as well.

With these latest commits, I think the new behavior should now be completely behind the enable_netcdftimeindex option (which by default is set to False); I also added a test to make sure that calling resample on an xarray object that is indexed by a NetCDFTimeIndex raises a TypeError.
Note that because I wanted the convenience of using pytest.mark.parametrize for some tests in test_coding_times.py, I took the liberty of eliminating the class-based structure in that file.  In hindsight I realize I probably could have waited to complete that job in another PR, but I got a little carried away.  My apologies for adding some noise to this already large diff.

I'm sufficiently pleased with this that I think we can merge this in more or less its current state. Given the size of the PR though, I'd like to have one more person review it in detail before it goes in. Any takers from @pydata/xarray? Maybe @spencerahill?

Just seeing this.  I'm tied up the next couple days but would be happy to review it on Wednesday. Although I doubt I'll find anything you or @shoyer wouldn't, so feel free to merge if you'd rather not hold it up another few days.

It occurred to me that netcdftime.num2date can return datetime.datetime objects in certain circumstances (not just objects that subclass netcdftime.datetime).  5e1c4a8 provides fixes to allow things to work with datetime.datetime objects as well as more test coverage.
One thing that makes the logic a bit messier here is that determining the calendar type for encoding datetime.datetime objects for faithful roundtripping is a bit complicated.  As far as I can tell, there are two scenarios, each with their own branches of logic:

Arrays where the minimum date is before the start of the Gregorian calendar (1582-10-15)

If the dates have the type datetime.datetime we encode them using the 'gregorian' calendar.
If the dates have the type netcdftime.DatetimeGregorian we encode them using the 'standard' calendar type.

Arrays where the minimum date is on or after the start of the Gregorian calendar (1582-10-15)

In this scenario, I'm not sure if it is possible to get num2date to return objects of type netcdftime.DatetimeGregorian.   Therefore, regardless of whether the dates are of type datetime.datetime or netcdftime.DatetimeGregorian, we encode them using the 'standard' calendar type.

The rationale behind this more complicated logic has to do with how it appears netcdftime.num2date behaves.  Here are some minimal examples:
>>> from netcdftime import date2num, num2date, DatetimeGregorian
>>> from datetime import datetime

>>> units = 'days since 1582-01-01'
>>> # Case with minimum date before start of Gregorian calendar
>>> arr = date2num([datetime(1582, 10, 1), datetime(1582, 10, 15)], units=units)
>>> # To recover datetime.datetime objects, we need to use calendar='gregorian'
>>> num2date(arr, units=units, calendar='gregorian')
array([datetime.datetime(1582, 10, 1, 0, 0),
       datetime.datetime(1582, 10, 15, 0, 0)], dtype=object)
>>> num2date(arr, units=units, calendar='standard')
>>> # Using calendar='standard' results in netcdftime.DatetimeGregorian objects
array([netcdftime._netcdftime.DatetimeGregorian(1582, 10, 1, 0, 0, 0, 0, -1, 1),
       netcdftime._netcdftime.DatetimeGregorian(1582, 10, 15, 0, 0, 0, 0, -1, 1)],
      dtype=object)

>>> # Case with minimum date after start of Gregorian calendar
>>> arr = date2num([datetime(1582, 10, 15), datetime(1582, 10, 16)], units=units)
>>> # Regardless of the calendar type used for decoding, we get datetime.datetime objects
>>> num2date(arr, units=units, calendar='gregorian')
array([datetime.datetime(1582, 10, 15, 0, 0),
       datetime.datetime(1582, 10, 16, 0, 0)], dtype=object)
>>> num2date(arr, units=units, calendar='standard')
array([datetime.datetime(1582, 10, 15, 0, 0),
       datetime.datetime(1582, 10, 16, 0, 0)], dtype=object)

Given the fact that it does not seem possible to decode times after the start of the Gregorian calendar into dates of type netcdftime.DatetimeGregorian it is surprising that test_roundtrip_netcdftime_datetime_data_post_gregorian passes for DatetimeGregorian objects.  It turns out it works, because timedelta arithmetic is valid between DatetimeGregorian and datetime.datetime objects for dates after the start of the Gregorian calendar:
>>> DatetimeGregorian(1, 1, 1) - datetime(1, 1, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "netcdftime/_netcdftime.pyx", line 1624, in netcdftime._netcdftime.datetime.__sub__
ValueError: cannot compute the time difference between dates with different calendars
>>> DatetimeGregorian(1582, 10, 30) - datetime(1582, 10, 16)
datetime.timedelta(14)

(though maybe I should add an assert statement to check the date types, which would cause it to fail as currently constructed).
@jswhit, @jhamman -- is this behavior expected for netcdftime?  Is there a recommended robust way for determining the calendar type for roundtripping between date2num and num2date, or is what I have in times.py#L248-L268 sufficient?

I think netcdftime should have a version of num2date that always returns
netcdftime.datetime objects. It's really error prone to functions that turn
different types dependent on input values.
On Tue, Mar 20, 2018 at 11:39 AM Spencer Clark<EMAIL_ADDRESS>wrote:

It occurred to me that netcdftime.num2date can return datetime.datetime
objects in certain circumstances (not just objects that subclass
netcdftime.datetime). 5e1c4a8
https://github.com/pydata/xarray/commit/5e1c4a891c22f3c6b54d31ece8c29306483b36a0
provides fixes to allow things to work with datetime.datetime objects as
well as more test coverage.
One thing that makes the logic a bit messier here is that determining the
calendar type for encoding datetime.datetime objects for faithful
roundtripping
https://github.com/spencerkclark/xarray/blob/257f08607c3b0cb975a5114948d2f95f941543db/xarray/coding/times.py#L248-L268
is a bit complicated. As far as I can tell, there are two scenarios, each
with their own branches of logic:

Arrays where the minimum date is before the start of the Gregorian
calendar (1582-10-15)

If the dates have the type datetime.datetime we encode them using
the 'gregorian' calendar.
If the dates have the type netcdftime.DatetimeGregorian we encode
them using the 'standard' calendar type.

Arrays where the minimum date is on or after the start of the
Gregorian calendar (1582-10-15)

In this scenario, I'm not sure if it is possible to get num2date
to return objects of type netcdftime.DatetimeGregorian. Therefore,
regardless of whether the dates are of type datetime.datetime or
netcdftime.DatetimeGregorian, we encode them using the 'standard'
calendar type.

The rationale behind this more complicated logic has to do with how it
appears netcdftime.num2date behaves. Here are some minimal examples:

from netcdftime import date2num, num2date, DatetimeGregorian>>> from datetime import datetime
units = 'days since 1582-01-01'>>> # Case with minimum date before start of Gregorian calendar>>> arr = date2num([datetime(1582, 10, 1), datetime(1582, 10, 15)], units=units)>>> # To recover datetime.datetime objects, we need to use calendar='gregorian'>>> num2date(arr, units=units, calendar='gregorian')
array([datetime.datetime(1582, 10, 1, 0, 0),
datetime.datetime(1582, 10, 15, 0, 0)], dtype=object)>>> num2date(arr, units=units, calendar='standard')>>> # Using calendar='standard' results in netcdftime.DatetimeGregorian objects
array([netcdftime._netcdftime.DatetimeGregorian(1582, 10, 1, 0, 0, 0, 0, -1, 1),
netcdftime._netcdftime.DatetimeGregorian(1582, 10, 15, 0, 0, 0, 0, -1, 1)],
dtype=object)
Case with minimum date after start of Gregorian calendar>>> arr = date2num([datetime(1582, 10, 15), datetime(1582, 10, 16)], units=units)>>> # Regardless of the calendar type used for decoding, we get datetime.datetime objects>>> num2date(arr, units=units, calendar='gregorian')

array([datetime.datetime(1582, 10, 15, 0, 0),
datetime.datetime(1582, 10, 16, 0, 0)], dtype=object)>>> num2date(arr, units=units, calendar='standard')
array([datetime.datetime(1582, 10, 15, 0, 0),
datetime.datetime(1582, 10, 16, 0, 0)], dtype=object)
Given the fact that it does not seem possible to decode times after the
start of the Gregorian calendar into dates of type
netcdftime.DatetimeGregorian it is surprising that
test_roundtrip_netcdftime_datetime_data_post_gregorian
https://github.com/spencerkclark/xarray/blob/257f08607c3b0cb975a5114948d2f95f941543db/xarray/tests/test_backends.py#L383-L408
passes for DatetimeGregorian objects. It turns out it works, because
timedelta arithmetic is valid between DatetimeGregorian and
datetime.datetime objects for dates after the start of the Gregorian
calendar:

DatetimeGregorian(1, 1, 1) - datetime(1, 1, 1)
Traceback (most recent call last):
File "", line 1, in
File "netcdftime/_netcdftime.pyx", line 1624, in netcdftime._netcdftime.datetime.__sub__ValueError: cannot compute the time difference between dates with different calendars>>> DatetimeGregorian(1582, 10, 30) - datetime(1582, 10, 16)
datetime.timedelta(14)

(though maybe I should add an assert statement to check the date types,
which would cause it to fail as currently constructed).
@jswhit https://github.com/jswhit, @jhamman https://github.com/jhamman
-- is this behavior expected for netcdftime? Is there a recommended robust
way for determining the calendar type for roundtripping between date2num
and num2date, or is what I have in times.py#L248-L268
https://github.com/spencerkclark/xarray/blob/257f08607c3b0cb975a5114948d2f95f941543db/xarray/coding/times.py#L248-L268
sufficient?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/pull/1252#issuecomment-374711420, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABKS1vmkPkfyIf8NHPw_RLJxC_L8DlEEks5tgUz9gaJpZM4L3tsR
.

Thanks @shoyer, I agree that would make our life easier here.  I created an issue upstream: Unidata/netcdftime#37.

Hi all, any updates on the current status for this? This will be a big help for me as well in particular processing daily CMIP5 netcdf files. I have been following this thread as well as the original issue and really appreciate this work. One other question: This PR doesn't allow for resampling on non-standard calendars as is, but I remember @shoyer mentioning that a workaround using pandas Grouper objects will exist. Would someone be able to explain to me how this would work? Thanks!

I think the code is in pretty good shape here.
My main concern is about the stability of the netcdftime package, which it looks like might need to be renamed? https://github.com/Unidata/netcdftime/issues/36
As for resampling, this would indeed require custom logic for netcdf datetimes. But I think it would be relatively doable. The key thing would dividing an array of datetimes into frequency groups. Then we could reuse xarray's existing logic for resampling, e.g.,
https://github.com/pydata/xarray/blob/9b76f219ec314dcb0c9a310c097a34f5c751fdd6/xarray/core/groupby.py#L234-L235
For example, if using freq='1AS' (annual frequency, start), we would need a function that maps netcdftime.datetime objects onto a netcdftime.datetime object corresponding to the start of a year, e.g.,
datetime(2000, 1, 1) -> datetime(2000, 1, 1)
datetime(2000, 1, 2) -> datetime(2000, 1, 1)
datetime(2000, 1, 3) -> datetime(2000, 1, 1)
...
datetime(2000, 12, 31) -> datetime(2000, 1, 1)
datetime(2001, 1, 1) -> datetime(2001, 1, 1)
...
datetime(2001, 12, 31) -> datetime(2001, 1, 1)
Pandas does this logic with offset classes. These are somewhat complex because pandas handles complex business day logic. For netcdftime, we could potentially start from scratch and only handle the important cases for climate science (e.g., round to start for year, quarter, month, day, hour, second).

My main concern is about the stability of the netcdftime package, which it looks like might need to be renamed? Unidata/netcdftime#36

Yes, I agree we should wait until that issue is addressed so we can adapt things accordingly.

I can push the name change on cftime forward over the next week.

Thanks @jhamman!  Once that's done, hopefully we can sure up Unidata/netcdftime#39.  I think those are the last outstanding issues here.

One clean option would be to require the new cftime module (with your change) for this functionality. That way we can rely on standard behavior here.

I will wait to implement this until the name change and Unidata/netcdftime#39 go through.  With Unidata/netcdftime#39 we can make the one failing test here, test_roundtrip_netcdftime_datetime_data, pass for all cftime date types.

@spencerkclark - things are well underway fixing up cftime. We'll need some namespace changes in xarray before things will work here. Do you want to do those or should I?

@jhamman great, I can handle those if you'd like.  It's basically just updating what we did in #1920, #1929, and #1933, correct?  I'll do that in separate PR, which we can merge before this one.

I think this is close to being ready.  I've updated things for the renaming of netcdftime to cftime, and added logic to only allow for the use of a CFTimeIndex or serialization of cftime.datetime objects when the standalone cftime package is installed.
The last piece is upstream in Unidata/cftime#39.  With that we should be able to slightly modify the logic here to allow us to roundtrip all dates produced via cftime.num2date (and make all the roundtrip tests pass).

ping @spencerkclark to finish this up. I just released a new version of cftime so this should be ready to wrap up.

@spencerkclark, I would like to test our changes on some netCDF files I am processing and which need to handle data in year 2300, thank you so much for enabling it! is there a github url I can get these code changes from? I understand that they might not be absolutely final.

@ritviksahajpal by all means, give it a try!  You should be able to install my branch via pip:
$ pip install git+https://github.com/spencerkclark/xarray.git@NetCDFTimeIndex

For the new functionality you will also need to install the latest version of the cftime library:
$ conda install -c conda-forge cftime

and in your code you will need to opt-in via xarray.set_options(enable_cftimeindex=True).  Please let me know if you run into any issues as we work towards finishing this up.

works beautifully, thanks so much!

@spencerkclark , one question. The xarray version from git+https://github.com/spencerkclark/xarray.git@NetCDFTimeIndex seems to be 0.8.2, because of this some things seem to be missing e.g. open_rasterio had a new argument called nodatavals intrroduced in 0.10.x. This is not available in your code. Is there a reason why your xarray version is 0.8.2?

It looks like one of the appveyor builds is failing due to something related to cftime missing -- it would be good to fix that

Thanks for the review @shoyer!  I'll try and address your comments tomorrow.

It looks like one of the appveyor builds is failing due to something related to cftime missing -- it would be good to fix that

Right, see my comment above:

The failures in Appveyor are due to the fact that the 32-bit Python 2.7 build is not using the latest version of cftime (instead it is using 1.0.0a3). @jhamman should we expect the version 1.0.0b1 to be available on that platform?

With regards to CI failures, I think it would make sense to add an explicit dependency on cftime rather than relying on an implicit dependency through the netCDF4 package. You can do this by editing the requirements.yml files in the ci/ directory. This would presumably resolve the issue with inadvertently using an old version of cftime.

With regards to CI failures, I think it would make sense to add an explicit dependency on cftime rather than relying on an implicit dependency through the netCDF4 package.

We did this already in #2054, which is why I suspect this is an upstream packaging issue.  Right now on conda-forge the win-32 version of cftime is still v1.0.0a3, while the other platforms' versions have been updated to v1.0.0b1, which has the updates we need for cftime.num2date.

@ritviksahajpal I see what you mean regarding the version, e.g. this is what I get if I install via my suggestion above:
>>> import xarray as xr
>>> xr.__version__
'0.8.2+dev648.gca4d7dd'

That said, I don't think I would read very much into that.  I've been merging the master branch continually to keep things up to date.

open_rasterio had a new argument called nodatavals intrroduced in 0.10.x. This is not available in your code.

Note that nodatavals is not an argument to open_rasterio; it is an attribute that is added to the DataArray returned by open_rasterio:
https://github.com/pydata/xarray/blob/c42cbe787d530db48575d73fe7a3910b0a0e4fd8/xarray/backends/rasterio_.py#L260-L263

We did this already in #2054, which is why I suspect this is an upstream packaging issue. Right now on conda-forge the win-32 version of cftime is still v1.0.0a3, while the other platforms' versions have been updated to v1.0.0b1, which has the updates we need for cftime.num2date.

OK. In that case, we should probably add a temporary "pip" clause to the requirements file for windows, to install cftime from pypi instead for now. I don't want to merge this when it will break CI.

@spencerkclark - mind reporting the conda issue upstream at https://github.com/conda-forge/cftime-feedstock?

@jhamman sure thing, see https://github.com/conda-forge/cftime-feedstock/issues/2.

In that case, we should probably add a temporary "pip" clause to the requirements file for windows, to install cftime from pypi instead for now.

Thanks @shoyer, it looks like that did the trick.  I addressed your recent comments; let me know if you have any further feedback.

See also https://github.com/pydata/xarray/pull/2098, which should fix failing builds on master

Tests are green here now. @shoyer and @spencerkclark - are we waiting on anything else before merging?

This could be ready. I'm happy to address any further concerns if anyone has them.

A couple other things to think about from a usability perspective:

What happens when you try to resample along CFTimeIndex?
What happens when you try to plot a DataArray with a CFTimeIndex?

These should at least raise informative errors (use NotImplementedError).

Thanks @shoyer, those are good questions.  I addressed your inline comments.  Let me know if you have anything else.

What happens when you try to resample along CFTimeIndex?

Through pandas, this raises an informative TypeError:
In [1]: from cftime import num2date

In [2]: import numpy as np

In [3]: import xarray as xr

In [4]: xr.set_options(enable_cftimeindex=True)
Out[4]: <xarray.core.options.set_options at 0x10c3a99d0>

In [5]: time = num2date(np.arange(5), units='days since 0001-01-01', calendar='noleap')

In [6]: data = xr.DataArray(np.arange(5), coords=[time], dims=['time'])

In [7]: data.resample(time='2D').mean()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-577095ede89a> in <module>()
----> 1 data.resample(time='2D').mean()

/Users/spencerclark/xarray-dev/xarray/xarray/core/common.pyc in resample(self, freq, dim, how, skipna, closed, label, base, keep_attrs, **indexer)
    616         resampler = self._resample_cls(self, group=group, dim=dim_name,
    617                                        grouper=grouper,
--> 618                                        resample_dim=RESAMPLE_DIM)
    619
    620         return resampler

/Users/spencerclark/xarray-dev/xarray/xarray/core/resample.pyc in __init__(self, *args, **kwargs)
    128                              "cannot have the same name as actual dimension "
    129                              "('{}')! ".format(self._resample_dim, self._dim))
--> 130         super(DataArrayResample, self).__init__(*args, **kwargs)
    131
    132     def apply(self, func, shortcut=False, **kwargs):

/Users/spencerclark/xarray-dev/xarray/xarray/core/groupby.pyc in __init__(self, obj, group, squeeze, grouper, bins, cut_kwargs)
    233                 raise ValueError('index must be monotonic for resampling')
    234             s = pd.Series(np.arange(index.size), index)
--> 235             first_items = s.groupby(grouper).first()
    236             full_index = first_items.index
    237             if first_items.isnull().any():

//anaconda/envs/xarray-dev/lib/python2.7/site-packages/pandas/core/generic.pyc in groupby(self, by, axis, level, as_index, sort, group_keys, squeeze, **kwargs)
   5160         return groupby(self, by=by, axis=axis, level=level, as_index=as_index,
   5161                        sort=sort, group_keys=group_keys, squeeze=squeeze,
-> 5162                        **kwargs)
   5163
   5164     def asfreq(self, freq, method=None, how=None, normalize=False,

//anaconda/envs/xarray-dev/lib/python2.7/site-packages/pandas/core/groupby.pyc in groupby(obj, by, **kwds)
   1846         raise TypeError('invalid type: %s' % type(obj))
   1847
-> 1848     return klass(obj, by, **kwds)
   1849
   1850

//anaconda/envs/xarray-dev/lib/python2.7/site-packages/pandas/core/groupby.pyc in __init__(self, obj, keys, axis, level, grouper, exclusions, selection, as_index, sort, group_keys, squeeze, **kwargs)
    514                                                     level=level,
    515                                                     sort=sort,
--> 516                                                     mutated=self.mutated)
    517
    518         self.obj = obj

//anaconda/envs/xarray-dev/lib/python2.7/site-packages/pandas/core/groupby.pyc in _get_grouper(obj, key, axis, level, sort, mutated, validate)
   2848     # a passed-in Grouper, directly convert
   2849     if isinstance(key, Grouper):
-> 2850         binner, grouper, obj = key._get_grouper(obj, validate=False)
   2851         if key.key is None:
   2852             return grouper, [], obj

//anaconda/envs/xarray-dev/lib/python2.7/site-packages/pandas/core/resample.pyc in _get_grouper(self, obj, validate)
   1118     def _get_grouper(self, obj, validate=True):
   1119         # create the resampler and return our binner
-> 1120         r = self._get_resampler(obj)
   1121         r._set_binner()
   1122         return r.binner, r.grouper, r.obj

//anaconda/envs/xarray-dev/lib/python2.7/site-packages/pandas/core/resample.pyc in _get_resampler(self, obj, kind)
   1114         raise TypeError("Only valid with DatetimeIndex, "
   1115                         "TimedeltaIndex or PeriodIndex, "
-> 1116                         "but got an instance of %r" % type(ax).__name__)
   1117
   1118     def _get_grouper(self, obj, validate=True):

TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'CFTimeIndex'

What happens when you try to plot a DataArray with a CFTimeIndex?

I updated things such that if cftime.datetime objects are used as a coordinate when plotting, the error message looks like:
In [1]: from cftime import num2date

In [2]: import numpy as np

In [3]: import xarray as xr

In [4]: xr.set_options(enable_cftimeindex=True)
Out[4]: <xarray.core.options.set_options at 0x10af58850>

In [5]: time = num2date(np.arange(5), units='days since 0001-01-01', calendar='noleap')

In [6]: data = xr.DataArray(np.arange(5), coords=[time], dims=['time'])

In [7]: data.plot()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-118f55e0b3d0> in <module>()
----> 1 data.plot()

/Users/spencerclark/xarray-dev/xarray/xarray/plot/plot.pyc in __call__(self, **kwargs)
    357
    358     def __call__(self, **kwargs):
--> 359         return plot(self._da, **kwargs)
    360
    361     @functools.wraps(hist)

/Users/spencerclark/xarray-dev/xarray/xarray/plot/plot.pyc in plot(darray, row, col, col_wrap, ax, rtol, subplot_kws, **kwargs)
    155     kwargs['ax'] = ax
    156
--> 157     return plotfunc(darray, **kwargs)
    158
    159

/Users/spencerclark/xarray-dev/xarray/xarray/plot/plot.pyc in line(darray, *args, **kwargs)
    258             yplt = darray.coords[ylabel]
    259
--> 260     _ensure_plottable(xplt)
    261
    262     primitive = ax.plot(xplt, yplt, *args, **kwargs)

/Users/spencerclark/xarray-dev/xarray/xarray/plot/plot.pyc in _ensure_plottable(*args)
     54         if not (_valid_numpy_subdtype(np.array(x), numpy_types) or
     55                 _valid_other_type(np.array(x), other_types)):
---> 56             raise TypeError('Plotting requires coordinates to be numeric '
     57                             'or dates of type np.datetime64 or '
     58                             'datetime.datetime.')

TypeError: Plotting requires coordinates to be numeric or dates of type np.datetime64 or datetime.datetime.

If cftime.datetime objects are the data requested to be plotted, the following error message results:
In [8]: data = xr.DataArray(time, coords=[np.arange(5)], dims=['x'])

In [9]: data.plot()
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-9-118f55e0b3d0> in <module>()
----> 1 data.plot()

/Users/spencerclark/xarray-dev/xarray/xarray/plot/plot.pyc in __call__(self, **kwargs)
    357
    358     def __call__(self, **kwargs):
--> 359         return plot(self._da, **kwargs)
    360
    361     @functools.wraps(hist)

/Users/spencerclark/xarray-dev/xarray/xarray/plot/plot.pyc in plot(darray, row, col, col_wrap, ax, rtol, subplot_kws, **kwargs)
    124
    125     if contains_cftime_datetimes(darray):
--> 126         raise NotImplementedError('Plotting arrays of cftime.datetime objects '
    127                                   'is currently not possible.')
    128

NotImplementedError: Plotting arrays of cftime.datetime objects is currently not possible.

OK, I'm happy with this. Time to merge I guess?

Congrats! This is a great piece of work and will be very useful to the climate community.

Okay, I'm going to merge now. Hopefully a few of us can stress test this a bit more prior to the next release. Thanks @spencerkclark for all the work here over the past 15 months!!!

@shoyer, @jhamman, @maxim-lian, @spencerahill many thanks for the substantial feedback, help, and encouragement here.  You guys are great!

Congrats to everyone who made this happen, especially @spencerclark. This feature is going to make so many people happy!

Indeed that meeting played an important role here.  Thank you @rabernat!

I have some data like
2000-01-01 00:00:00, 2000-01-01 12:00:00,
2000-01-02 00:00:00, 2000-02-01 12:00:00.
And I want to take the average of the same date.
I am wondering if it is possible to resample them at a daily level (e.g., the results will be 2000-01-01 00:00:00 and 2000-01-02 00:00:00)?


proteins generated from esm-msa sometimes end in

In a set of 1000 generated sequences with settings:
{'num_iters': 30, 'burnin': 10, 'mask': True, 'in_order':False, 'num_positions_percent': 10}
I saw about 3 where the last amino acid was <mask>
I'll see if I can reproduce this.
Kind of a second issue, but to make bugs more reproducible, adding the possibility to define random seeds for the samplers to make them deterministic would be nice.

I think this is fixed, I haven't seen it crop up in a long time.


Use Slash Commands with Cogs

Summary
Use Slash Commands with Cogs
What is the feature request for?
The core library
The Problem
You can't use slash commands with Cogs.
The Ideal Solution
I think the optimal solution is to add the option to declare slash commands like a simple command.
Example:
from nextcord.ext import commands
from nextcord import Interaction

class Example(commands.Cog):
    def __init__(self, bot: commands.Bot):
        self.bot = bot

    @commands.slash_command(
        name="example",
        description="Cantami, o Diva, del Pelìde Achille l'ira funesta che infiniti addusse lutti agli Achei",
    )
    async def example(self, interaction: Interaction, arg0: int, arg1: int):
        await interaction.response.send_message(f"{arg0} + {arg1} = {arg0 + arg1}")

The Current Solution
The current solution is simply to not use cogs.
Additional Context
No response

You can use @nextcord.slash_command(...) for now.

@MRvillager commands.slash_command doesn't exist only nextcord.slash_command

@MRvillager commands.slash_command doesn't exist only nextcord.slash_command

Yeah, I know. It's a suggestion for the implementation of slash commands inside cogs.

It makes no sense for it to be inside ext.commands.

It makes sense to me because to declare commands inside cogs you use @commands.command(), which is inside nextcord.ext. So to maintain consistency with the old commands it should be inside ext.commands.

As a side note, ext exists for extensions which are outside of the core discord api. So slash commands, being in the api and not an extension we add for convenience come under just nextcord

In that case, the solution would be to implement a new Cog class dedicated for slash commands.

It already exists?

ClientCog and commands.Cog subclasses it so it works in cogs

see what soheab said at the start of this thread

My bad, I just checked it. Given that, I think there should be an example on how to use it, just for completeness.


Include typeParameter location in the arrow function expression

This fixes #111

Current coverage is 94.13% (diff: 100%)

Merging #126 into master will increase coverage by <.01%

@@             master       #126   diff @@
==========================================
  Files            19         19
  Lines          3068       3070     +2
  Methods         320        320
  Messages          0          0
  Branches        802        802
==========================================
+ Hits           2888       2890     +2
  Misses          100        100
  Partials         80         80

Powered by Codecov. Last update ddbda7d...ee788f2


file: 5.42 -> 5.43

https://mailman.astron.com/pipermail/file/2022-September/000857.html
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Thanks!

pkgsMusl.file fails to build since this change due to a test failure. Upstream bug report


Should isApprovedForAll throw when _owner == _operator?

I don't think so. Standard itself does not define it and I don't actually see real word need for it.
Actually I am considering removing all checks here to lower GAS costs since they are unnecessary.

I generally agree. I even tested if gas cost is lower if you switch from single line functions returning value from mapping to public field and guess what... it is not only for deployment, but also for calls from other contracts.
I can think of one case where throwing would be useful. When other contracts call getApproved or isApprovedForAll and they have a bug in that they pass parameters that don't make sense.
I was just thinking out loud as there are differences between your implementation of getApproved/isApprovedForAll, the one from OZ and mine. You have some low-gast-cost (under 10 gas) checks there while others don't.


handle variable not existing more gracefully

If I start nvim with nvim +commnd +q to update it from the cli for example, the global variable might be not yet set when we reach the exit function so it will cause an error: undefined variable codeium_server_job
Using get handle this case, it returns 0 when the variable is not set yet instead of an error
My actual use case is updating lunarvim from the cli with lvim +LvimUpdate +q

Thanks!


Password-less sudo?

I'm on OS X. Is there a way to stop Vagrant from asking my password in order to add/remove the host entries?

The working answer could be captured as a shell script in the repo to make life easier for users...
See: https://gist.github.com/EHLOVader/606f7c9f8717608527b8
For example, here is my add_vagrant_hostupdater_to_sudoers.sh:
#!/bin/bash

# Add to sudoers, for `vagrant up` without a password

# Stage updated sudoers in a temporary file for syntax checking
TMP=$(mktemp -t vagrant_sudoers)
cat /etc/sudoers > $TMP

# Remove any previous declarations
sed -e '/Cmnd_Alias VAGRANT_*/ d' -n $TMP

cat >> $TMP <<EOF

# Allow passwordless startup of Vagrant when using NFS.
# based on https://gist.github.com/joemaller/6764700

Cmnd_Alias VAGRANT_EXPORTS_ADD = /usr/bin/tee -a /etc/exports
Cmnd_Alias VAGRANT_NFSD = /sbin/nfsd restart
Cmnd_Alias VAGRANT_EXPORTS_REMOVE = /usr/bin/sed -E -e /*/ d -ibak /etc/exports
Cmnd_Alias VAGRANT_HOSTSUPDATER_ADD = /bin/sh -c echo "*" >> /etc/hosts
Cmnd_Alias VAGRANT_HOSTSUPDATER_REMOVE = /usr/bin/sed -i -e /*/ d /etc/hosts
%staff ALL=(root) NOPASSWD: VAGRANT_EXPORTS_ADD, VAGRANT_NFSD, VAGRANT_EXPORTS_REMOVE, VAGRANT_HOSTSUPDATER_ADD, VAGRANT_HOSTSUPDATER_REMOVE #vagrant
EOF

# Check syntax and overwrite sudoers if clean
visudo -c -f $TMP
if [ $? -eq 0 ]; then
  echo "Adding vagrant commands to sudoers"
  cat $TMP > /etc/sudoers
else
  cat $TMP
  echo "sudoers syntax wasn't valid. Aborting!"
fi

rm -f $TMP

@ngConsulti
The script didn't work on my Ubuntu 14.10 machine. I used your script as starting point and made some improvements to make it work with Ubuntu 14.10.
GitHub: https://gist.github.com/a90120282a3757ea1b9d.git
#!/bin/bash
# Add Vagrant's hostupdater commands to sudoers, for `vagrant up` without a password

# force sudo on self.
if [ $( id -u ) -ne 0 ]; then
    exec sudo -p "Login password for %p: " "$0" "$@"
    exit $?
fi

# Stage updated sudoers in a temporary file for syntax checking
TMP="$(mktemp)"
cat /etc/sudoers > "${TMP}"

# Remove any previous declarations
sed -i -e '/Cmnd_Alias VAGRANT_*/ d' "${TMP}"

sed -i -e '/# Allow passwordless startup of Vagrant when using NFS\./ d' $TMP
sed -i -e '/# based on https:\/\/gist.github.com\/joemaller\/6764700/ d' $TMP

sed -i -e '/NOPASSWD: VAGRANT_*/ d' "${TMP}"

cat >> "${TMP}" <<EOF

# Allow passwordless startup of Vagrant when using NFS.
# based on https://gist.github.com/joemaller/6764700
Cmnd_Alias VAGRANT_HOSTMANAGER_UPDATE = /bin/cp $HOME/.vagrant.d/tmp/hosts.local /etc/hosts
Cmnd_Alias VAGRANT_EXPORTS_ADD = /bin/bash -c echo '*' >> /etc/exports
Cmnd_Alias VAGRANT_NFSD = /sbin/nfsd restart
Cmnd_Alias VAGRANT_EXPORTS_REMOVE = /usr/bin/sed -E -e /*/ d -ibak /etc/exports
%staff ALL=(root) NOPASSWD: VAGRANT_EXPORTS_ADD, VAGRANT_NFSD, VAGRANT_EXPORTS_REMOVE, VAGRANT_HOSTMANAGER_UPDATE #vagrant
EOF

# Check syntax and overwrite sudoers if clean
visudo -c -f $TMP
if [ $? -eq 0 ]; then
  echo "Adding vagrant commands to sudoers"
  cat $TMP > /etc/sudoers
else
  echo "sudoers syntax wasn't valid. Aborting!"
fi

rm -f $TMP


Lazy Load Keymaps Not Set Properly

Describe the bug
Lazy's setting of keymaps does not work properly. For plugins lazy loaded by key, they are set to the wrong key(of other lazy loaded plugins).
Which version of Neovim are you using?
TUI(v0.8.0)
To Reproduce
Make sure to read https://github.com/folke/lazy.nvim/wiki/Minimal-`init.lua`-to-Reproduce-an-Issue
Steps to reproduce the behavior:

Install vim-surround
Try searching for text and using n to go to the next search result

Expected Behavior
Typing n should go to the next search result.
I ran :verbose map to find out what specifically is wrongly mapping the keys. It appears that it is this function that is causing the problem:
https://github.com/folke/lazy.nvim/blob/7f10386e2335d1a479e77a4060fff3aac3097599/lua/lazy/core/handler/keys.lua#L65

repro.lua
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
  vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({
    "git",
    "clone",
    "--filter=blob:none",
    "--single-branch",
    "https://github.com/folke/lazy.nvim.git",
    lazypath,
  })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
  -- do not remove the colorscheme!
  "folke/tokyonight.nvim",
   {
    'tpope/vim-surround',
    keys = { { "n", "ds" }, { "n", "cs" }, { "n", "ys" }, { "v", "S" }, { "v", "gS" } }
  },

}
require("lazy").setup(plugins, {
  root = root .. "/plugins",
})

-- add anything else here
vim.opt.termguicolors = true
-- do not remove the colorscheme!
vim.cmd([[colorscheme tokyonight]])

Log
Please include any related errors from the Noice log file. (open with :Lazy log)

Lazy log

  Total: 28 plugins
Breaking Changes (1)
● lazy.nvim 13.13ms   init.lua
3594bfd docs: todo (73 minutes ago)
7dfb9c1 feat(ui): added options to sort/filter profiling data (73 minutes ago)
fde5fee refactor: refactored ui code (2 hours ago)
cd023dc style: remove setting colors on startup (3 hours ago)
e2bbf3d refactor: better annotations to make sumneko faster (3 hours ago)
65e9036 fix: added error message to debug failing extmarks #117 (4 hours ago)
040aeb6 chore(build): auto-generate vimdoc (4 hours ago)
c8da1c1 docs: fixed typos and capitalization (#118) (4 hours ago)
8bd8e6c chore(build): auto-generate vimdoc (4 hours ago)
cc6276e refactor!: default lazy cache path is now under cache instead of state (4 hours ago)
004b11d docs: improved docs on build. Fixes #119 (4 hours ago)
8251c23 fix(checker): dont report updates on install during startup (4 hours ago)
7b9b476
fix(install): make sure to setup loaders before attempting install so colorscheme can load. Fixes #122
(4 hours ago)
e5dcc87 chore(build): auto-generate vimdoc (13 hours ago)
8112640 feat!: init() no longer implies lazy-loading. Add lazy=false for affected plugins
(13 hours ago)
6f9845e chore(build): auto-generate vimdoc (13 hours ago)
2a7b004
feat(spec): config can be true or a table that will be passed to require("plugin").setup(config)
(13 hours ago)
9e98389 fix(ui): fix buffer being properly deleted (#112) (14 hours ago)
3f60f2d fix(keys): key handlers were not working after reload (14 hours ago)
b5d6afc fix(manage): do not reload pugins on clear (14 hours ago)
037973c chore(build): auto-generate vimdoc (15 hours ago)
b440b3a fix(keys): don't create with remap! (15 hours ago)
9e869a4 docs: migration (15 hours ago)
72b3899 fix(keys): dont delete handlers manually. Let loader do that (17 hours ago)
08d458c fix: remove debug print (17 hours ago)
455a1f4 chore(build): auto-generate vimdoc (17 hours ago)
2756a6f fix!: run init() before loading start plugins. Fixes #107 (17 hours ago)
fb8287c chore(build): auto-generate vimdoc (18 hours ago)
4b4ccdb docs: Add migration for tag = '*' from packer (#105) (18 hours ago)
56890ce fix: remove lazy keymaps with the correct mode. Fixes #97 (18 hours ago)
42f5aa7 fix: pass plugins instead of plugin names to command. Fixes #103 (19 hours ago)
4bf771a feat(spec): allow using plugin names in dependencies (19 hours ago)
eb01b6d chore(build): auto-generate vimdoc (22 hours ago)
07b4677 feat(loader): automatically lazy-load colorschemes (22 hours ago)
210d170 chore(main): release 6.0.0 (#76) (25 hours ago)
3415a61 fix(ui): properly wrap ui elements on small screens. Fixes #92 (25 hours ago)
c0c2e1b fix(cmd): allow ranges. Fixes #93 (25 hours ago)
a8fe63e chore(build): auto-generate vimdoc (25 hours ago)
1c07ea1 feat(keys): more advanced options for setting lazy key mappings (25 hours ago)
28f1511 fix: show mapleader warning with vim.schedule. Fixes #91 (27 hours ago)
2200284 fix(cache): overwrite cache entry with new modpath when loading a file. Fixes #90
(27 hours ago)
6304231 feat(ui): show modpaths in debug (27 hours ago)
dd9648f fix(clean): update lockfile on clean (#88) (28 hours ago)
63418e8 chore(build): auto-generate vimdoc (28 hours ago)
488b487 fix(git): remove --also-filter-submodules. Fixes #86 #83 (28 hours ago)
dc150df test(lockfile): create config dir if it does not exist (2 days ago)
4cf176b fix(install): update lockfile also on install (2 days ago)
7f6f31d fix(git): make sure we properly fetch git submodules. Fixes #72 (2 days ago)
a39d37b chore(build): auto-generate vimdoc (2 days ago)
2e14a2f feat: added support for nvim --headless "+Lazy! sync" +qa (2 days ago)
bc61747 feat!: lazy api commands now take an opts table instead of a list of plugins (2 days ago)
fd1fbef
feat(checker): defer checker to after VeryLazy to make sure nvim-notify and others are loaded
(2 days ago)
58f0876 fix: removed spell again from site. not needed. can download in config/spell (2 days ago)
94d0125 fix(rtp): keep site in rtp (2 days ago)
6dbcdce chore(build): auto-generate vimdoc (2 days ago)
1fb10b9 docs: 'cmd' repeat twice (#75) (2 days ago)
472a091 chore(main): release 5.2.0 (#55) (2 days ago)
f0e1b85 feat: make hover easy to override (2 days ago)
2ab6518 fix(keys): feedkeys should include pending keys. Fixes #71 (2 days ago)
d34a02d ci: Update issue templates (2 days ago)
2fd78fb fix(help): sort tags files for readmes so tags work properly. Fixes #67 (2 days ago)
eab449b build: added .repro and debug to .gitignore (2 days ago)
b7c489b fix(loader): lua modules can be links instead of files. Fixes #66 (2 days ago)
7a57ea2 chore(build): auto-generate vimdoc (2 days ago)
c228908 fix(health): don't show warning on module=false (2 days ago)
b68f94b ci: Update issue templates (2 days ago)
bbebb67 ci: Update issue templates (2 days ago)
95fc814 chore(build): auto-generate vimdoc (2 days ago)
876f7bd feat(loader): allow to add extra paths to rtp reset. Fixes #64 (2 days ago)
a345649
fix(cache): if we can't load from the cache modpath, find path again instead of erroring right away
(2 days ago)
bbace14 fix(git): only mark a plugin as dirty if an update changed the commit HEAD. Fixes #62
(2 days ago)
a939243 fix(checker): allow git checks only for non-pinned plugins (#61) (2 days ago)
57bea32 chore(build): auto-generate vimdoc (2 days ago)
ff24f49 fix(loader): source rtp /plugin files after loading start plugins. Fixes (2 days ago)
b802729 chore(build): auto-generate vimdoc (2 days ago)
9dfefac docs: fixed indentation of auto-generated code blocks (2 days ago)
71b2e2f docs: lazy works on all OSs now (2 days ago)
232232d fix(ui): install command can have plugins as a parameter (2 days ago)
4ca3039 feat(loader): warn when mapleader is changed after init (2 days ago)
540847b fix: strip / from dirs. Fixes #60 (2 days ago)
86eaa11 fix(git): dereference tag refs. Fixes #54 (2 days ago)
e95da35 feat(util): utility method to get sync process output (2 days ago)
3c3a711 chore(build): auto-generate vimdoc (2 days ago)
44f80a7 feat(plugin): allow plugin files only without a main plugin module. Fixes #53 (2 days ago)
f5734f5 chore(build): auto-generate vimdoc (2 days ago)
3814883 fix(ui): set current win only when its valid (2 days ago)
3a7b8c8 chore(main): release 5.1.0 (#30) (3 days ago)
3606d62 fix: add after directories to rtp to make after/ftplugin and others work. Fixes #47
(3 days ago)
b193f96 fix(spec): only process a spec once (3 days ago)
7be46bc style: removed unused requires (3 days ago)
897d6df fix: add filetype to window buffer. (#41) (3 days ago)
14300b3 chore(build): auto-generate vimdoc (3 days ago)
78e9d6c docs: add a note about mapleader (3 days ago)
06ac8bd perf(ui): clear existing extmarks before rendering (3 days ago)
ffcd0ab fix(loader): source filetype.lua before plugins. Fixes #35 (3 days ago)
9d12cdc fix(git): don't run git log for submodules. Fixes #33 (3 days ago)
06ffcf5 chore(build): auto-generate vimdoc (3 days ago)
7fb0652 docs: added docs on update checker (3 days ago)
1f86cb3 chore(build): auto-generate vimdoc (3 days ago)
3cffb2a docs: added change detection to the readme (3 days ago)
6c767a6 feat: added options to configure change detection. Fixes #32 (3 days ago)
cd162f3 chore(build): auto-generate vimdoc (3 days ago)
941df31 feat(ui): make the windoww size configurable. Fixes #34 (3 days ago)
5298441 fix: use nvim_feekeys instead of nvim_input for keys handler. Fixes #28 (3 days ago)
2927b05 docs: added lincense (3 days ago)
4d78203 chore(main): release 5.0.1 (#17) (3 days ago)
1371a14 fix(build): use the shell to execute build commands (3 days ago)
ffabe91 fix(cache): if mod is loaded already in the loader, then return that (3 days ago)
316503f fix: dont autoload cached modules when module=false (3 days ago)
992c679 fix: always set Config.me regardless of reset rtp (3 days ago)
df6c986 fix: add neovim libs to rtp for treesitter parsers etc (3 days ago)
49b69b7 chore(build): auto-generate vimdoc (3 days ago)
e9d3a73 fix: default logs are now since 3 days ago to be in line with the docs (3 days ago)
4234322 chore(build): auto-generate vimdoc (3 days ago)
6e32759 fix: deepcopy lazyspec before processing (3 days ago)
ec0f8d0 docs: added config.dev.path to the example (3 days ago)
6404d42 fix: move re-sourcing check to the top (3 days ago)
ddf36d7 fix: checker should not error on non-existing dirs (3 days ago)
50ba619 test: fix tests (3 days ago)
f78d8bf fix: show error when merging, but continue (3 days ago)
b8a0055 chore(build): auto-generate vimdoc (3 days ago)
1754056 fix: use jobstart instead of system to open urls (3 days ago)
5ecc988 docs: use https to bootstrap lazy (3 days ago)
ae644a6 fix: only run updated checker for installed plugins. Fixes #16 (3 days ago)
Log (4)
● coc.nvim 48.27ms  UIEnter
28feeff commit<PHONE_NUMBER>8f1344efa26c27d0e01c7ef05ffa32 (7 hours ago)
9d47e34 commit 7e391609347658dce6a50d72ccf42efc60e04221 (13 hours ago)
2fc5b8b commit 4ccdb97f6869064ebb5cbd8260dbba8049efe829 (20 hours ago)
1a61ab5 commit a492de819bf00f2b06622e2945c7fef06b85b4db (2 days ago)
● lualine.nvim 17.34ms  start
    fffbcb8 chore: autogen (vimdocs+formating) (22 hours ago)
    3497c6c Passing self as second arg to self.options.fmt (#915) (22 hours ago)
    afb8bfb enhance: explicitly prefer user config dir (28 hours ago)

● nvim-treesitter 93.06ms   telescope.nvim
    82cff67 julia: update parser and highlights (#4016) (82 minutes ago)
    895d4a1 Update parsers: c_sharp (13 hours ago)
    eaeedfb Fix vue injection queries (18 hours ago)
    6f9e5cd Update parsers: ada, c_sharp (21 hours ago)
    23906e8 feat: add Ada (21 hours ago)
    779b422 injections(go): inject regex into some regexp functions (26 hours ago)
    3884544 Update parsers: c_sharp, t32 (26 hours ago)
    9efc2df Update parsers: c_sharp (2 days ago)
    2d48cbc feat(t32): initial support (2 days ago)
    51df421 Update parsers: c_sharp (2 days ago)
    d2d8980 injections(yaml): fix injections (2 days ago)
    e8b765b ci: use builtin jq instead of downloading it (2 days ago)
    770c66d Update parsers: v (2 days ago)
    87dd482 fix(health): align the list of parsers when doing checkhealth (2 days ago)
    ca09752 Update parsers: c_sharp, gdscript, sql (2 days ago)

● telescope.nvim 102.27ms  start
    e960efa fix: jumplist picker indexing the line after (#2273) (2 days ago)
    d16581e fix: misidentification invert and files_with_matches (#2240) (#2262) (2 days ago)
    278c797 fix(builtin.live_grep): add spacer ":" even when coordinates disabled (#2275) (2 days ago)

Loaded (14)
● coc-sources 0.2ms   coc.nvim
● gitsigns.nvim 36.19ms  BufReadPre
● impatient.nvim 19.68ms  start
● indent-blankline.nvim 5.46ms  BufReadPre
● kanagawa.nvim 9.32ms  start
● notifier.nvim 3.67ms  UIEnter
● nvim-web-devicons 0.5ms  start
● plenary.nvim 0.57ms   telescope.nvim
● rainbow_parentheses.vim 19.51ms  BufReadPre
● telescope-fzf-native.nvim 0.46ms   telescope.nvim
● vim-fugitive 3.9ms  start
● vim-sensible 1.11ms  start
● vim-snippets 0.77ms   coc.nvim
● vim-vinegar 1ms  start
Installed (9)
○ Comment.nvim
○ glow.nvim
○ gv.vim
○ leap.nvim
○ nvim-autopairs
○ vim-cool
○ vim-surround
○ vim-unimpaired
○ zen-mode.nvim

your keys spec is all off. Please check the docs

Ah I see, my mistake for not checking that part of the README. Wouldn't it be more convenient if Lazy automatically called the commands instead of you needing to explicitly state them(like Packer)? It would be tedious for plugins that have several keymaps based on the same prefix.

what do you mean? You just need to specify the list you had as:
{"ds",  "cs",  "ys", {"S", mode = "v"}, {"gS", mode = "v"}}

Ah I see. My apologies, I made my deleted comment off of this snippet in the README. Thank you for the clarification! From my understanding, normal mode keymaps do not require you to specify the mode but keymaps for other modes do. Correct?

It's all literally in the docs. I write docs so people can lookup all that inforamation.
I even already mentioned it's in the docs. Just read them! Especially if you encounter an issue. First step should ALWAYS be to read the docs

I thought you by docs you meant the wiki, not the help files. My bad. Though very good suggestion!

Why would I mean the wiki? There's nothing there.
Docs can refer to the readme or vim docs. Both contain the same information.
Also you did refer to the snippet, so you did at least glance at that part.

Yeah, that's why I was a little bit confused. In the README I did. Just an fyi, it is not possible to look up the docs in Neovim using :help.

It would be great if you provided a snippet similar to the one you showed me in the docs. That might clarify this for other users that make the same mistake as I did.

it is possible to look up the docs in neovim with :help

It would be great if you provided a snippet similar to the one you showed me in the docs. That might clarify this for other users that make the same mistake as I did.

"the one you showed me in the docs"
all the other users could and should also look at those docs

It would be great if you provided a snippet similar to the one you showed me in the docs. That might clarify this for other users that make the same mistake as I did.

"the one you showed me in the docs" all the other users could and should also look at those docs

That was a typo, I meant the one he showed me in this comment thread. I will fix that.

What did you supplement :help with? I get this:

for me :help lazy.nvim.txt works

it's already clear in the docs that you should create the mappings the way folke showed you

there was a bug that after the initial install, lazy docs would not be available. Only after updating lazy.
Just pushed a fix for that

Yep, it works now. Perhaps shorten the name to just lazy.nvim?

all the files end with .txt

Not the actual help file name but the name used to call it. E.g :help lazy.nvim instead of :help lazy.nvim.txt.

that's already possible


Add wrap and unwrap

Coverage increased (+0.09%) to 96.667% when pulling 55891b067d7ed5907d49b6dd88d2285780f074c9 on add-wrap-and-unwrap into 5bf600f287da09ddc776cba42b6ed08f3f8b4425 on master.


Remove extra dot from breadcrumbs after entity name

WHY
Need to remove this extra dot from breadcrumbs in create and edit view.
BEFORE - What was wrong? What was happening before this PR?
There is an extra dot(.) after entity name on breadcrumbs in create and edit view.

AFTER - What is happening after this PR?
Removed the dot from breadcrumbs.
HOW
Just remove the dot(.) from create.blade.php and edit.blade.php
How did you achieve that, in technical terms?
??
Is it a breaking change?
No
How can we test the before & after?
Just check the breadcrumbs in create and edit page.
If the PR has changes in multiple repos please provide the command to checkout all branches, eg.:
git checkout "dev-branch-name" &&
cd vendor/backpack/crud && git checkout crud-branch-name &&
cd ../pro && git checkout pro-branch-name &&
cd ../../..

Hey @kamrul-bd
The "." is not the part of the breadcrumb. I can see, it is placed after the action description which looks fine to me.
I don't see any issue there.
Thanks for the PR.


[AIRFLOW-1294] Backfills can loose tasks to execute

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[X] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-1294

Description

[X] Here are some details about my PR, including screenshots of any UI changes:

In backfills we can loose tasks to execute due to a task
setting its own state to NONE if concurrency limits are reached,
this makes them fall outside of the scope the backfill is
managing hence they will not be executed.
Tests

[X] My PR adds the following unit tests OR does not need testing for this extremely good reason:
Should be covered by current tests, will adjust if required.

Commits

[X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@mistercrunch @aoen @saguziel This is a simplified fix that should be easier to digest in 1.8.2. It does not address all underlying issues as in https://github.com/apache/incubator-airflow/pull/2356 , but those can be addressed separately and in smaller bits.

Codecov Report

Merging #2360 into master will decrease coverage by <.01%.
The diff coverage is 53.33%.

@@            Coverage Diff             @@
##           master    #2360      +/-   ##
==========================================
- Coverage   69.43%   69.43%   -0.01%
==========================================
  Files         145      145
  Lines       11070    11079       +9
==========================================
+ Hits         7687     7693       +6
- Misses       3383     3386       +3

Impacted Files
Coverage Δ

airflow/jobs.py
74.67% <53.33%> (-0.08%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 355fc92...2678574. Read the comment docs.

@mistercrunch Done.

I like this change a lot more than the old one, thank you!

@aoen no problem. I was tackling two issues at once in the previous version. I do still think we need to look at the logic underpinning the executor, but that is something else.

when do you think a release might be cut with this fix?


[android][WEEX-474] when the app's WXInstance destroy, but Weex page call toast Toast ,crash

[android][WEEX-474] when the app's WXInstance destroy, but Weex page call toast Toast ,crash

Seems like already merged


BigDecimal constructor throws OverflowException for large decimal values

The following constructor throws an exception...
public BigDecimal(decimal d, int precision, int scale)
      {
         OriginalValue = d;
         Precision = precision;
         Scale = scale;

         decimal value = d;
         while (scale-- > 0) value *= 10m;
         Value = value;
      }

If d =<PHONE_NUMBER>7282.54 and precision = 38 and scale = 16 then the scaled version of the decimal value cannot be calculated and throws an exception at System.Decimal.FCallMultiply().
My dataset holds the value in a column of type DecimalSchemaElement.
Easy to replicate with other values of d greater than<PHONE_NUMBER>0000 when scale is 16.

I took a zip of the current master branch and copied it to my local and referenced that in my code.
To fix the problem (and probably a few other potential bugs) I altered the BigDecimal class to the following:
``
struct BigDecimal
{
///
/// Contains a Decimal value that is the big integer
///
public decimal DecimalValue { get; private set; }
    public BigInteger UnscaledValue { get; private set; }

    /// <summary>
    /// The scale of the decimal value
    /// </summary>
    public int Scale { get; set; }

    /// <summary>
    /// The precision of the decimal value
    /// </summary>
    public int Precision { get; set; }

    public BigDecimal(byte[] data, Thrift.SchemaElement schema)
    {
        data = data.Reverse().ToArray();

        UnscaledValue = new BigInteger(data);
        Precision = schema.Precision;
        Scale = schema.Scale;

        var scaleMultiplier = BigInteger.Pow(10, Scale);
        var fpUnscaled = default(BigInteger);
        var ipScaled = (decimal)BigInteger.DivRem(UnscaledValue, scaleMultiplier, out fpUnscaled);
        var fpScaled = (decimal)fpUnscaled / (decimal)scaleMultiplier;

        DecimalValue = ipScaled + fpScaled;
    }

    public BigDecimal(decimal d)
    {
        uint[] bits = (uint[])(object)decimal.GetBits(d);

        decimal mantissa =
           (bits[2] *<PHONE_NUMBER>m *<PHONE_NUMBER>m) +
           (bits[1] *<PHONE_NUMBER>m) +
           bits[0];

        uint scale = (bits[3] >> 16) & 31;

        uint precision = 0;
        if (d != 0m)
        {
            for (decimal tmp = mantissa; tmp >= 1; tmp /= 10)
            {
                precision++;
            }
        }
        else
        {
            // Handle zero differently. It's odd.
            precision = scale + 1;
        }

        DecimalValue = d;
        Precision = (int)precision;
        Scale = (int)scale;

        var scaleMultiplier = BigInteger.Pow(10, Scale);
        var ipScaled = new BigInteger(DecimalValue);
        var fpScaled = DecimalValue - (decimal)ipScaled;
        var fpUnscaled = fpScaled * (decimal)scaleMultiplier;

        UnscaledValue = (ipScaled * scaleMultiplier) + new BigInteger(fpUnscaled);
    }

    public BigDecimal(decimal d, int precision, int scale)
    {
        DecimalValue = d;
        Precision = precision;
        Scale = scale;

        var scaleMultiplier = BigInteger.Pow(10, Scale);
        var ipScaled = new BigInteger(DecimalValue);
        var fpScaled = DecimalValue - (decimal)ipScaled;
        var fpUnscaled = fpScaled * (decimal)scaleMultiplier;

        UnscaledValue = (ipScaled * scaleMultiplier) + new BigInteger(fpUnscaled);
    }

    /// <summary>
    /// Converts a BigDecimal to a decimal
    /// </summary>
    /// <param name="bd">The BigDecimal value</param>
    public static implicit operator decimal(BigDecimal bd)
    {
        return bd.DecimalValue;
    }

    private byte[] AllocateResult()
    {
        int size;

        //according to impala: https://www.cloudera.com/documentation/enterprise/5-6-x/topics/impala_decimal.html

        if (Precision <= 9)
            size = 4;
        else if (Precision <= 18)
            size = 8;
        else
            size = 16;

        return new byte[size];
    }

    public byte[] ToByteArray()
    {
        byte[] result = AllocateResult();

        byte[] data = UnscaledValue.ToByteArray();

        if (data.Length > result.Length) throw new NotSupportedException($"decimal data buffer is {data.Length} but result must fit into {result.Length} bytes");

        Array.Copy(data, result, data.Length);

        result = result.Reverse().ToArray();
        return result;
    }
}

Hi @moggy001 can you generate a PR for this it's easier for us to diff and credit you with the changes. Can you also ensure that you have an up-to-date unit test in the tests project. Thanks for the contribution!

I've integrated this in the latest alpha. Shame PR wasn't raised.

Thanks Ivan. Unfortunately, I have never contributed to an open project before and I simply didn't get time to figure how to do so in between my regular work. I have other changes/ fixes that I think should be applied to this library so I shall try to raise a pull request for those.

@moggy001 no worries, i understand. You code sample is brilliant and was easy to integrate ;)


Feature request : Custom search extension

Hello,
I've tried searching for that issue but I couldn't find it.
I'm trying to write a vscode extension that would create another search view. The syntax for searching/replacing would be using Comby.
I've tried playing with view containers / views, but they only allow tree views for now. I cannot create a view that has input boxes.
Is there a way of doing that ? If not I think it would be a nice enhancement.
Thank you !

Duplicate of https://github.com/microsoft/vscode/issues/83911


Pinpoint PHP 8.2 compatibility

Hello,
I have an issue while compiling Pinpoint on PHP 8.2:
 /pinpoint-c-agent/src/PHP/pinpoint_php.cpp: In function 'void zif_pinpoint_get_func_ref_args(zend_execute_data*, zval*)':
 /pinpoint-c-agent/src/PHP/pinpoint_php.cpp:559:64: error: too many arguments to function 'zend_result zend_forbid_dynamic_call()'
   559 |     if (zend_forbid_dynamic_call("pinpoint_get_func_ref_args()") == FAILURE)
       |                                                                ^
 In file included from /usr/local/include/php/main/php.h:35,
                  from /pinpoint-c-agent/src/PHP/pinpoint_php.cpp:40:
 /usr/local/include/php/Zend/zend_API.h:782:39: note: declared here
   782 | static zend_always_inline zend_result zend_forbid_dynamic_call(void)
       |                                       ^~~~~~~~~~~~~~~~~~~~~~~~

I see that function zend_forbid_dynamic_call is no longer need param from PHP 8.2 (changed in commit).
May I get any help or a new release for this issue to support PHP 8.2?
Thank you.

@ctykhoa
Please check the latest version of pinpoint-php


srun: error: node25: task 11: Killed

I encounter this problem when testing different bucket number when doing bucket sort. When testing n=100,000,000 and bucket num = 600,000 or 800,000.. It returns this:
srun: error: node25: task 11: Killed
srun: error: node25: task 19: Killed
srun: error: node25: task 9: Killed
srun: error: node25: task 17: Killed
srun: error: node25: task 5: Killed
srun: error: node25: task 23: Killed
srun: error: node25: task 15: Killed
srun: error: node25: task 1: Killed
srun: error: node25: task 21: Killed
srun: error: node25: task 29: Killed
srun: error: node25: task 14: Killed
What could be the possible reason? I have checked smaller n and bucket number, it runs normally.

Hello! The reason that your running time is too long so that your task is killed. My suggestion is that you could test your program locally to ensure that its performance is close to the baseline, then all the tests with srun could be finished in time


docker pull amundsen-metadata : repository does not exist

I am not able to get this working via docker
getting following error
pull access denied for amundsen-metadata, repository does not exist or may require 'docker login'

@ashitabh , we haven't push the repo to docker hub. Currently we need to clone the repo and run https://github.com/lyft/amundsenmetadatalibrary/blob/master/public.Dockerfile. We will update the readme to be more clear :)

On that same container, has anyone else seen this error where it says npm not found and issues with "no files found matching ..." when running the command docker build -f public.Dockerfile -t amundsen-metadata:latest .
Step 5/7 : RUN python3 setup.py install
 ---> Running in 291cf382651d
/bin/sh: 1: npm: not found
ERROR:root:npm must be available
/bin/sh: 1: npm: not found
setup.py:27: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
  logging.warn('Installation of npm dependencies failed')
WARNING:root:Installation of npm dependencies failed
setup.py:28: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
  logging.warn(str(e))
WARNING:root:Command '['npm install']' returned non-zero exit status 127.
running install
running bdist_egg
running egg_info
writing amundsen_frontend.egg-info/PKG-INFO
writing dependency_links to amundsen_frontend.egg-info/dependency_links.txt
writing entry points to amundsen_frontend.egg-info/entry_points.txt
writing requirements to amundsen_frontend.egg-info/requires.txt
writing top-level names to amundsen_frontend.egg-info/top_level.txt
reading manifest file 'amundsen_frontend.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'amundsen_application/static/dist'
warning: no files found matching '*' under directory 'amundsen_application/.*/static/dist'
warning: no files found matching '*' under directory 'amundsen_application/.*/static/fonts'
warning: no files found matching '*' under directory 'amundsen_application/.*/static/images'
writing manifest file 'amundsen_frontend.egg-info/SOURCES.txt'

I also saw this zip_safe flag warning:
zip_safe flag not set; analyzing archive contents...
amundsen_application.__pycache__.__init__.cpython-37: module references __file__

@rosejcday not sure why it shows NPM missing, but metadata doesn't include any NPM package. So I assume the warning is safe to ignore.

@rosejcday , @ashitabh I just published all 3 Amundsen repos images to dockerhub. Now you could take a at this pr(https://github.com/lyft/amundsenfrontendlibrary/pull/115).

@feng-tao thanks! I will use those now.

@feng-tao  and ...

@rosejcday not sure why it shows NPM missing, but metadata doesn't include any NPM package. So I assume the warning is safe to ignore.

I think I have spotted why that error appears in the run above. Somewhere in the output there’s a reference to amundsen_frontend which does depend on npm for building. So it’s safe to ignore in the context of this Amundsen Metadata repo - though still a bug. 😉 I’ll open an issue (or PR) in that other frontend repo.

thanks @jornh


Should we still be shipping the built Cesium Viewer?

We currently ship with a built copy of the Cesium Viewer and even mention it as "ready for deployment" on our website's download page.  This is misleading since the Cesium Viewer contains the default Bing key and can't be deployed as-is without violating usage policy.
I'm not even sure how many people actually ever use the built version either (or why you would use it other than for development).  Should we just stop shipping the built version?  It would probably shrink our overall zip file size a bit as well.  If we want to continue to ship the built version, we'll need a way to specify the Bing key via a configuration file that Cesium Viewer looks for.

Removing it is OK with me.

Sounds good.


Fix bug of service instance refetching when repeating health check is activated

Fixes gh-899
I'm not sure it is a proper solution to this issue but it works.

Fixes gh-899.

@OlgaMaciaszek I've just created a new PR against 2.2.x branch.

@rchigvintsev Thanks :)

Closing in favour of gh-904.


Update acli and acquia provider integration

acli has had some major updates, including

acquia/cli#741)

So we can probably allow it to upgrade now, and make sure the acquia integration works with it.

Tried to work on this in https://github.com/rfay/ddev/tree/20220112_acli_maintenance
But got blocked because you can't use acli in non-interactive any more.

https://github.com/acquia/cli/issues/790

We don't have access to Acquia any more; they promise to get it for us someday. But this will have to be postponed.


PHP 7 Errors + PSR-3 exception logging

Hey Chris, thanks for building php-resque! In this PR I've added support for catching PHP 7 errors when performing jobs in Resque_Worker. This should not break compatibility with PHP 5.
I've also changed the key name name used to log the exception from stack to exception. According to PSR-3:

If an Exception object is passed in the context data, it MUST be in the 'exception' key.

I assume this won't work in php versions < 7.0 because the Error class is unknown there.
To make this work you should include a \Error  class as polyfill.
See https://github.com/symfony/polyfill/blob/master/README.md#compatibility-notes as an example of php5 polyfills for php7 functionality. The Error class is included too.

Great point. This change should work in PHP 5 without a polyfill. When catching an exception, if the Error class does not exist then it will fall through to the original exception handler. Here's a demo that shows this principle working for PHP 5.1 - 7.1.

@chrisboulton Any chance on getting this merged? As PHP 5 reaches end of life this PR can help php-resque stay relevant. :)


Fix crash when event Id is less than 12 chars

Was sometimes getting "slice bounds out of range".
May as well print out the whole thing in debug mode. This is
sometimes names instead of IDs depending on the event.
Signed-off-by: Ben Firshman<EMAIL_ADDRESS>

LGTM


fix: Upstream chartmuseum can't be upgraded to from our variant

Signed-off-by: Andrew Bayer<EMAIL_ADDRESS>

@abayer Hi Andrew, where I can change chartmuseum charts for k8s 1.16 support for chartmuseum.jenkins-x.io?


Fix logging problem

Fixes https://github.com/Dr-Gigavolt/dbus-aggregate-batteries/issues/65
Not tested. Please test it first.

P.S.: @Dr-Gigavolt I recommend you to use Flake8 and Black Lint to properly format your code. It will do it for you automatically and you will save a lot of time.
Here is a short instruction on how to set up Flake8 and Black Lint checks in VS Code. This will save you a lot of time.


Added C++ elastic lib

Might be a good idea to use JNI with the java version if it is planned to move this into a vendordep

Hi, I've been trying to use this for my team and I've been running into the error fatal error: networktables/StringPublisher.h: No such file or directory. on line 6 What WPILib version does this work on? Thanks for your work on this.

Hi, I've been trying to use this for my team and I've been running into the error fatal error: networktables/StringPublisher.h: No such file or directory. on line 6 What WPILib version does this work on? Thanks for your work on this.

StringPublisher.h doesn't exist. If you remove it this should work

I don't have write access to this pr, but line 12 should be         nt::StringPublisher publisher = topic.Publish({.sendAll = true, .keepDuplicates = true}); to prevent issues about not finding the nt::PubSubOptions class

I'll see what I did wrong when I get the chance, probably will get it done by the end of the day

Depends on #133


[MAS4.2.11][Screen Reader- Create new bot configuration] After "Save and connect", the focus goes on the hidden element and announced as "secret copied to the clipboard".

Actual Result:
After "Save and Connect" button i.e. the last element of the dialog, the focus goes on the hidden element and announced "secret copied to the clipboard".
Expected Result:
The voiceover focus shouldn't lands on the hidden element or it shouldn't announced the status message again and again.
Repro Steps:

Open the Bot Framework V4 Emulator.
Welcome tab screen gets opened.
Navigate to the "Create new bot configuration" button and activate it.
Open a "Create new bot configuration" dialog box gets opened.
Navigate to the "encrypt keys stored in your bot configuration." checkbox and check it.
Navigate to "copy" button and activate it.
Now, navigate through the pane and observe the issue after "Save and Connect" button.

User Impact:
If unnecessary information is announced after the control, then voiceover dependent users will get confused.
Test Environment:
OS: MacOs Version 10.15
Application: Bot Framework V4 Emulator (Version 4.5.2-80829)
Impacted MAS Link:
https://microsoft.sharepoint.com/:w:/r/teams/msenable/_layouts/15/WopiFrame.aspx?sourcedoc={eedc2941-f754-4739-b483-3f8180c5aaa5}
Attachment:
Focus goes on hidden element.zip

Verified ,issue is fixed now so closing the bug.


WIP: Configurable container killing

Current coverage is 50.31% (diff: 85.71%)

Merging #977 into master will increase coverage by <.01%

@@             master       #977   diff @@
==========================================
  Files           277        277
  Lines         12970      12979     +9
  Methods           0          0
  Messages          0          0
  Branches       1673       1674     +1
==========================================
+ Hits           6526       6531     +5
- Misses         5954       5955     +1
- Partials        490        493     +3

Powered by Codecov. Last update 6c46ee3...4b56c52

LGTM 👍


XY Grid - Add kitchen sink/flex utilties visual tests

This should hopefully provide a single visual test that shows all use cases in one go.
Let me know if you think there's any other examples I should add.

I've also added a flex utilities visual test so we can confirm these cases work OK too.

I'm going to merge this in myself as I need it for testing.


Silent failure on malformed yaml

Parser fails silently on malformed yaml.
Code in question is:
Swagger20Parser.java, line 98: exception is silently swallowed
    } catch (Exception e) {
        if (System.getProperty("debugParser") != null) {
            e.printStackTrace();
        }

I find it quite unbelievable...

In the current extensions implementation, returning a "null" spec would signify the inability to process a definition (YAML in your bug report).  This would allow another extension to try.
Agreeing that it's not elegant, this is being changed in 1.1.x.


Include note about Azure Stack Hub support

Making change per @rtibi's request

#sign-off

Thanks, all. Publishing in this repository is suspended until Tuesday, November 2. Merging will resume Tuesday morning, and we'll merge this approved pull request at that time. Thanks.


Similar script on redmi note

Hey,
I have an app I cannot run on my redmi note 7, because it is jailbroken (the app says).
Redmi note runs on LineageOS 18.1
Is there anyway to make a similar script, so this above stated app would work?
thanks for enlightenment
-Martin

Wrong platform. This is for iOS only

Use Magisk denylist + Universal safetynet fix

Hi there, this script is made for iOS. If you mean you’re trying to run the script on your Android device for your iOS device, that is not officially supported and my not work:


⚠ Bofrost.at has degraded performance

In 97176cc, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1412 ms

Resolved: Bofrost.at performance has improved in 316a9cf.


ClientInfo must Implement Serializable so that ADAL/AuthenticationResult contract is honored

ClientInfo is now a property of AuthenticationResult. Because AuthenticationResult implements Serializable, objects which are its properties must be declared either transient or themselves Serializable

Resolved by https://github.com/AzureAD/microsoft-authentication-library-common-for-android/pull/435

Dupe of https://github.com/AzureAD/azure-activedirectory-library-for-android/issues/1376


envelope refactor

@h1alexbel take a look, please
some redesign, what do you think?

Codecov Report
Patch coverage: 96.15% and project coverage change: +4.58 :tada:

Comparison is base (faff32b) 73.41% compared to head (1e433e2) 78.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@             Coverage Diff              @@
##             master     #131      +/-   ##
============================================
+ Coverage     73.41%   78.00%   +4.58%
- Complexity       22       31       +9
============================================
  Files             9       11       +2
  Lines            79      100      +21
============================================
+ Hits             58       78      +20
- Misses           21       22       +1

Impacted Files
Coverage Δ

...n/java/io/github/eocqrs/kafka/fake/FkConsumer.java
0.00% <0.00%> (ø)

...va/io/github/eocqrs/kafka/producer/KfProducer.java
64.70% <ø> (ø)

...va/io/github/eocqrs/kafka/consumer/KfConsumer.java
57.14% <100.00%> (+4.51%)
:arrow_up:

...rs/kafka/consumer/settings/KfConsumerSettings.java
100.00% <100.00%> (ø)

.../consumer/settings/KfConsumerSettingsEnvelope.java
100.00% <100.00%> (ø)

...rs/kafka/producer/settings/KfProducerSettings.java
100.00% <100.00%> (ø)

.../producer/settings/KfProducerSettingsEnvelope.java
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@rultor merge

@rultor merge

@h1alexbel OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@h1alexbel Done! FYI, the full log is here (took me 2min)

@l3r8yJ thanks!


machine operators and transporters not functioning properly

Describe the bug
When trying to tie a spirit to an inventory or storage actuator with the book of calling it just tells you that this spirit has left this plane of existence
To Reproduce
Steps to reproduce the behavior:

Place spirit
shift right click on inventory to attach spirit to said inventory
get told no

Expected behavior
it should work
Screenshots
https://gyazo.com/8f2a5619f4d0ca5ec64eeeb1dc6ab825
.
System (please complete the following information):

Occultism Version: 0.2.6

Does this bug still reproduce on the latest version 0.2.8?
I did not add any fixes for it but want to ensure it repros before going deeper into it.

I haven't messed with the mod much since you've been away, I'll check when I get home from work tonight!

can confirm same behavior in 0.2.8 https://gyazo.com/622eff6da172944597207fbc9abdd800

Thank you! I'll dive into it given time then

It seems the fix used in #32, when applied to the books of calling ( 5fce82e0635e70033c5e3625946e984b0e12d8a4 ) fixed this issue, I cannot currently reproduce it.  Please re-open if it remains in 0.2.10.