Chi-square fails on categorical data

I tried using the chi-squared test, but I observed that it only works on numerical columns. When I try using categoricals, it fails. Here's how to reproduce.
def synthetic_interaction_data(n, yintercept = 10):
    df = pd.DataFrame()
    df["id"] = range(n)
    df[f'x1'] = np.random.random(size=n)*10
    df[f'x2'] = np.random.random(size=n)*10
    df[f'x3'] = np.random.random(size=n)*10
    df['target'] = (df['x1']**2 + df['x1']*df['x2'] + 5*df['x1']*np.sin(3*df['x2'])  + yintercept).astype("int16") % 2
    return df

df = synthetic_interaction_data(2000)
df["odd"] = "even"
df.loc[np.floor(df["x1"]) % 2 == 1, "odd"] = "odd"
df['target'] = df['target'].astype(np.object)

df2 = synthetic_interaction_data(2000)
df2["odd"] = "even"
df2.loc[np.floor(df2["x1"]) % 2 == 1, "odd"] = "odd"
df2['target'] = df2['target'].astype(np.object)

ddp = Profile(sections=[DataDriftProfileSection])
ddp.calculate(df, df2, column_mapping = None)

Shows the error below:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-70-92e6117fd55f> in <module>
      1 ddp = Profile(sections=[DataDriftProfileSection])
----> 2 ddp.calculate(df, df2, column_mapping = None)

/opt/conda/lib/python3.8/site-packages/evidently/model_profile/model_profile.py in calculate(self, reference_data, current_data, column_mapping)
     20                   current_data: pandas.DataFrame,
     21                   column_mapping: dict = None):
---> 22         self.execute(reference_data, current_data, column_mapping)
     23
     24     def get_analyzers(self):

/opt/conda/lib/python3.8/site-packages/evidently/pipeline/pipeline.py in execute(self, reference_data, current_data, column_mapping)
     16                 column_mapping: dict = None):
     17         for analyzer in self.get_analyzers():
---> 18             self.analyzers_results[analyzer] = analyzer().calculate(reference_data, current_data, column_mapping)

/opt/conda/lib/python3.8/site-packages/evidently/analyzers/data_drift_analyzer.py in calculate(self, reference_data, current_data, column_mapping)
     57
     58         for feature_name in cat_feature_names:
---> 59             ref_feature_vc = reference_data[feature_name][np.isfinite(reference_data[feature_name])].value_counts()
     60             current_feature_vc = current_data[feature_name][np.isfinite(current_data[feature_name])].value_counts()
     61

/opt/conda/lib/python3.8/site-packages/pandas/core/generic.py in __array_ufunc__(self, ufunc, method, *inputs, **kwargs)
   1934         self, ufunc: Callable, method: str, *inputs: Any, **kwargs: Any
   1935     ):
-> 1936         return arraylike.array_ufunc(self, ufunc, method, *inputs, **kwargs)
   1937
   1938     # ideally we would define this to avoid the getattr checks, but

/opt/conda/lib/python3.8/site-packages/pandas/core/arraylike.py in array_ufunc(self, ufunc, method, *inputs, **kwargs)
    356         # ufunc(series, ...)
    357         inputs = tuple(extract_array(x, extract_numpy=True) for x in inputs)
--> 358         result = getattr(ufunc, method)(*inputs, **kwargs)
    359     else:
    360         # ufunc(dataframe)

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Hope you can help me with this, or if there's something I did incorrectly. Thanks.

One thing I tried is encoding them to codes. And it worked.

Hi @mabreyes thanks for noting, and glad you solved the issue! For now, that is exactly how this feature works: one needs to encode categorical features with numerical labels. We expect to adapt the tool to work with categorical data at a later point. Here is a step by step guide that covers some of the pre-processing steps one might need to perform for some reports/data types: https://docs.evidentlyai.com/step-by-step-guide-for-jupyter-notebooks

@elenasamuylova Thanks so much. Closing this issue for now.


Added be_u24 method

In TLS1.0+ there are payload fields who's length is set via a 24bit unsigned integer value.
Kind of a strange method I agree. I'm not sure if you prefer usize or u32

Coverage remained the same at 81.049% when pulling 58c58efc408367d580b8c9d551855257869edc35 on valarauca:master into da873207a75306069b17c144c9fa68d20d586935 on Geal:master.

u32 should be fine. Can you add the little endian method as well?

Coverage decreased (-81.5%) to 0.0% when pulling 5481bc094aa84776db87721c7bd4f9ab1ec8e64a on valarauca:master into 3196f1a74a8d638f99f1396b8c99215f3f02d49b on Geal:master.

Where should I commit the tests btw?

the tests should be in the same file with the others: https://github.com/Geal/nom/blob/master/src/nom.rs#L871-L956

ping :)

Hey sorry in the middle of a move for work. I'll follow up on this when I
get the chance
On Sun, Mar 19, 2017 at 7:18 AM Geoffroy Couprie<EMAIL_ADDRESS>wrote:

ping :)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Geal/nom/pull/430#issuecomment-287609613, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ASViDNR5DcdetV7tCFSilD1Mmg7drsIeks5rnQ8dgaJpZM4LreTJ
.

Thanks!


Is there a Docker Compose API?

I'm sorry but I can't seem to find it... even in the Docker API Reference, there's nothing about Docker Compose. Can somebody point out a useful link or two?

I asked myself the same thing when I was starting with Docker. The answer is No, unfortunately.
docker-compose is a client thing and as much as we are going more and more deeper on Swarm and Services/Tasks, I don't see IMHO that compose will ever hit the servers.

Ok, I'm not that deep into Docker to completely understand your answer.
But at least I understood the short answer 😃
Thanks mate

@sebagomez sorry. What I meant is that, docker-compose command line and the .yml is a Docker CLI feature so it run on the client side, not in Docker engine itself. Hence why Docker API and Docker.Dotnet don't have support for it.
IMHO, I see that docker-compose is not something that will stay around for much longer.

Oh ok, I see.
And do you see anything at the engine level that will replace docker-compose ?

Well, you may always define a single Dockerfile and create a Swarm Service from it. It will define replicas and you can use overlay networks to communicate with other services inside a Swarm cluster.

Ok, one more question since you've been so nice, say I need the aspnet image and the sql-server-express images... how do I integrate them in a single file. Do I have to create one single dockerfile with what these two have? Like FROM windowsservercore and then add IIS, and install SQL Server, which would replicate exactly what their Dockerfiles have. Or is it a better way I'm not familiar with?

Yeah, AFAIK, there is no merge operation on images unfortunately. You will have to get the base common image and make yours FROM it, and then reapply the layers manually on your dockerfile.

Your too kind... thanks for all the tips and help


Small doc change

This has also been fixed in another PR. Thanks once again for putting this up and if you see more mistakes please keep bringing them up.
Thank you kindly.


SouthernCo just introduced Hourly Usage

Hello,
It looks like SouthernCo, or in my case, Georgia Power, just introduced hourly usage.  At least on the graph it looks to be delayed by 2 days, but still much more useful than daily data for tracking individual appliances.
This is less a bug and more a feature request for daily usage.  I'm trying to find a way to pull this data into Home Assistant.

Hi @mcondren,
I would love to add that feature but I might need your help as my accounts haven't had that feature enabled yet.
Let me know if you're interesting in helping out.

Andrew Pearson

I've done a bit of digging on this topic..
The API for hourly data is: ```
https://customerservice2api.southerncompany.com/api/MyPowerUsage/MPUData/<ACCOUNT_ID>/Hourly?EndDate=01%2F27%2F2022&OPCO=GPC&ServicePointNumber=<METER_ID?>&StartDate=01%2F26%2F2022&intervalBehavior=Automatic

And the response object looks like:

{
"StatusCode": 200,
"Message": "Successfully retrieved My Power Usage data for Hourly Graph",
"MessageType": 0,
"Data": {
"contractriderDesc": null,
"Data": "{"xAxis":{"labels":["2022-01-26T00:00:00","2022-01-26T01:00:00","2022-01-26T02:00:00","2022-01-26T03:00:00","2022-01-26T04:00:00","2022-01-26T05:00:00","2022-01-26T06:00:00","2022-01-26T07:00:00","2022-01-26T08:00:00","2022-01-26T09:00:00","2022-01-26T10:00:00","2022-01-26T11:00:00","2022-01-26T12:00:00","2022-01-26T13:00:00","2022-01-26T14:00:00","2022-01-26T15:00:00","2022-01-26T16:00:00","2022-01-26T17:00:00","2022-01-26T18:00:00","2022-01-26T19:00:00","2022-01-26T20:00:00","2022-01-26T21:00:00","2022-01-26T22:00:00","2022-01-26T23:00:00"]},"series":{"cost":{"data":[{"x":0,"y":0.0,"name":"2022-01-26T00:00:00","resolution":"hourly"},{"x":1,"y":0.0,"name":"2022-01-26T01:00:00","resolution":"hourly"},{"x":2,"y":0.13,"name":"2022-01-26T02:00:00","resolution":"hourly"},{"x":3,"y":0.13,"name":"2022-01-26T03:00:00","resolution":"hourly"},{"x":4,"y":0.13,"name":"2022-01-26T04:00:00","resolution":"hourly"},{"x":5,"y":0.0,"name":"2022-01-26T05:00:00","resolution":"hourly"},{"x":6,"y":0.13,"name":"2022-01-26T06:00:00","resolution":"hourly"},{"x":7,"y":0.13,"name":"2022-01-26T07:00:00","resolution":"hourly"},{"x":8,"y":0.13,"name":"2022-01-26T08:00:00","resolution":"hourly"},{"x":9,"y":0.13,"name":"2022-01-26T09:00:00","resolution":"hourly"},{"x":10,"y":0.13,"name":"2022-01-26T10:00:00","resolution":"hourly"},{"x":11,"y":0.39,"name":"2022-01-26T11:00:00","resolution":"hourly"},{"x":12,"y":0.13,"name":"2022-01-26T12:00:00","resolution":"hourly"},{"x":13,"y":0.13,"name":"2022-01-26T13:00:00","resolution":"hourly"},{"x":14,"y":0.0,"name":"2022-01-26T14:00:00","resolution":"hourly"},{"x":15,"y":0.26,"name":"2022-01-26T15:00:00","resolution":"hourly"},{"x":16,"y":0.0,"name":"2022-01-26T16:00:00","resolution":"hourly"},{"x":17,"y":0.52,"name":"2022-01-26T17:00:00","resolution":"hourly"},{"x":18,"y":0.26,"name":"2022-01-26T18:00:00","resolution":"hourly"},{"x":19,"y":0.13,"name":"2022-01-26T19:00:00","resolution":"hourly"},{"x":20,"y":0.13,"name":"2022-01-26T20:00:00","resolution":"hourly"},{"x":21,"y":0.0,"name":"2022-01-26T21:00:00","resolution":"hourly"},{"x":22,"y":0.13,"name":"2022-01-26T22:00:00","resolution":"hourly"},{"x":23,"y":0.13,"name":"2022-01-26T23:00:00","resolution":"hourly"}]},"usage":{"data":[{"x":0,"y":0.0,"name":"2022-01-26T00:00:00","resolution":"hourly"},{"x":1,"y":0.0,"name":"2022-01-26T01:00:00","resolution":"hourly"},{"x":2,"y":1.0,"name":"2022-01-26T02:00:00","resolution":"hourly"},{"x":3,"y":1.0,"name":"2022-01-26T03:00:00","resolution":"hourly"},{"x":4,"y":1.0,"name":"2022-01-26T04:00:00","resolution":"hourly"},{"x":5,"y":0.0,"name":"2022-01-26T05:00:00","resolution":"hourly"},{"x":6,"y":1.0,"name":"2022-01-26T06:00:00","resolution":"hourly"},{"x":7,"y":1.0,"name":"2022-01-26T07:00:00","resolution":"hourly"},{"x":8,"y":1.0,"name":"2022-01-26T08:00:00","resolution":"hourly"},{"x":9,"y":1.0,"name":"2022-01-26T09:00:00","resolution":"hourly"},{"x":10,"y":1.0,"name":"2022-01-26T10:00:00","resolution":"hourly"},{"x":11,"y":3.0,"name":"2022-01-26T11:00:00","resolution":"hourly"},{"x":12,"y":1.0,"name":"2022-01-26T12:00:00","resolution":"hourly"},{"x":13,"y":1.0,"name":"2022-01-26T13:00:00","resolution":"hourly"},{"x":14,"y":0.0,"name":"2022-01-26T14:00:00","resolution":"hourly"},{"x":15,"y":2.0,"name":"2022-01-26T15:00:00","resolution":"hourly"},{"x":16,"y":0.0,"name":"2022-01-26T16:00:00","resolution":"hourly"},{"x":17,"y":4.0,"name":"2022-01-26T17:00:00","resolution":"hourly"},{"x":18,"y":2.0,"name":"2022-01-26T18:00:00","resolution":"hourly"},{"x":19,"y":1.0,"name":"2022-01-26T19:00:00","resolution":"hourly"},{"x":20,"y":1.0,"name":"2022-01-26T20:00:00","resolution":"hourly"},{"x":21,"y":0.0,"name":"2022-01-26T21:00:00","resolution":"hourly"},{"x":22,"y":1.0,"name":"2022-01-26T22:00:00","resolution":"hourly"},{"x":23,"y":1.0,"name":"2022-01-26T23:00:00","resolution":"hourly"}]},"costDelayed":{"data":[]},"usageDelayed":{"data":[]},"temp":{"data":[{"x":0,"y":40.0,"name":"2022-01-26T00:00:00","resolution":"hourly"},{"x":1,"y":38.0,"name":"2022-01-26T01:00:00","resolution":"hourly"},{"x":2,"y":36.0,"name":"2022-01-26T02:00:00","resolution":"hourly"},{"x":3,"y":34.0,"name":"2022-01-26T03:00:00","resolution":"hourly"},{"x":4,"y":33.0,"name":"2022-01-26T04:00:00","resolution":"hourly"},{"x":5,"y":32.0,"name":"2022-01-26T05:00:00","resolution":"hourly"},{"x":6,"y":31.0,"name":"2022-01-26T06:00:00","resolution":"hourly"},{"x":7,"y":31.0,"name":"2022-01-26T07:00:00","resolution":"hourly"},{"x":8,"y":32.0,"name":"2022-01-26T08:00:00","resolution":"hourly"},{"x":9,"y":37.0,"name":"2022-01-26T09:00:00","resolution":"hourly"},{"x":10,"y":41.0,"name":"2022-01-26T10:00:00","resolution":"hourly"},{"x":11,"y":45.0,"name":"2022-01-26T11:00:00","resolution":"hourly"},{"x":12,"y":49.0,"name":"2022-01-26T12:00:00","resolution":"hourly"},{"x":13,"y":50.0,"name":"2022-01-26T13:00:00","resolution":"hourly"},{"x":14,"y":51.0,"name":"2022-01-26T14:00:00","resolution":"hourly"},{"x":15,"y":51.0,"name":"2022-01-26T15:00:00","resolution":"hourly"},{"x":16,"y":49.0,"name":"2022-01-26T16:00:00","resolution":"hourly"},{"x":17,"y":47.0,"name":"2022-01-26T17:00:00","resolution":"hourly"},{"x":18,"y":44.0,"name":"2022-01-26T18:00:00","resolution":"hourly"},{"x":19,"y":41.0,"name":"2022-01-26T19:00:00","resolution":"hourly"},{"x":20,"y":39.0,"name":"2022-01-26T20:00:00","resolution":"hourly"},{"x":21,"y":37.0,"name":"2022-01-26T21:00:00","resolution":"hourly"},{"x":22,"y":36.0,"name":"2022-01-26T22:00:00","resolution":"hourly"},{"x":23,"y":35.0,"name":"2022-01-26T23:00:00","resolution":"hourly"}]}}}",
"HighTempIndex": -1,
"LowTempIndex": -1,
"TemperatureIndex": -1,
"NotificationIndex": -1,
"AboveAlertIndex": -1,
"DailyAlertIndex": -1,
"WeekendIndex": -1,
"ReminderIndex": -1,
"ProjectedBillAmountHigh": null,
"ProjectedBillAmountLow": null,
"AverageDailyCost": null,
"Days": 0.0,
"DollarsToDate": null,
"TotalkWhUsed": 0.0,
"HasData": true,
"HasEstimatedBill": false,
"IsPartialMonth": false,
"RemainingDays": 0,
"DaysToDate": 0,
"AlertThreshold": 0,
"AlertThresholdExceeded": false,
"UsageTable": null
},
"ModelErrors": [],
"IsScApiResult": true
}

The data object in the middle is prettified as:

{
"xAxis": {
"labels": [
"2022-01-26T00:00:00",
"2022-01-26T01:00:00",
"2022-01-26T02:00:00",
"2022-01-26T03:00:00",
"2022-01-26T04:00:00",
"2022-01-26T05:00:00",
"2022-01-26T06:00:00",
"2022-01-26T07:00:00",
"2022-01-26T08:00:00",
"2022-01-26T09:00:00",
"2022-01-26T10:00:00",
"2022-01-26T11:00:00",
"2022-01-26T12:00:00",
"2022-01-26T13:00:00",
"2022-01-26T14:00:00",
"2022-01-26T15:00:00",
"2022-01-26T16:00:00",
"2022-01-26T17:00:00",
"2022-01-26T18:00:00",
"2022-01-26T19:00:00",
"2022-01-26T20:00:00",
"2022-01-26T21:00:00",
"2022-01-26T22:00:00",
"2022-01-26T23:00:00"
]
},
"series": {
"cost": {
"data": [
{
"x": 0,
"y": 0.0,
"name": "2022-01-26T00:00:00",
"resolution": "hourly"
},
{
"x": 1,
"y": 0.0,
"name": "2022-01-26T01:00:00",
"resolution": "hourly"
},
{
"x": 2,
"y": 0.13,
"name": "2022-01-26T02:00:00",
"resolution": "hourly"
},
{
"x": 3,
"y": 0.13,
"name": "2022-01-26T03:00:00",
"resolution": "hourly"
},
{
"x": 4,
"y": 0.13,
"name": "2022-01-26T04:00:00",
"resolution": "hourly"
},
{
"x": 5,
"y": 0.0,
"name": "2022-01-26T05:00:00",
"resolution": "hourly"
},
{
"x": 6,
"y": 0.13,
"name": "2022-01-26T06:00:00",
"resolution": "hourly"
},
{
"x": 7,
"y": 0.13,
"name": "2022-01-26T07:00:00",
"resolution": "hourly"
},
{
"x": 8,
"y": 0.13,
"name": "2022-01-26T08:00:00",
"resolution": "hourly"
},
{
"x": 9,
"y": 0.13,
"name": "2022-01-26T09:00:00",
"resolution": "hourly"
},
{
"x": 10,
"y": 0.13,
"name": "2022-01-26T10:00:00",
"resolution": "hourly"
},
{
"x": 11,
"y": 0.39,
"name": "2022-01-26T11:00:00",
"resolution": "hourly"
},
{
"x": 12,
"y": 0.13,
"name": "2022-01-26T12:00:00",
"resolution": "hourly"
},
{
"x": 13,
"y": 0.13,
"name": "2022-01-26T13:00:00",
"resolution": "hourly"
},
{
"x": 14,
"y": 0.0,
"name": "2022-01-26T14:00:00",
"resolution": "hourly"
},
{
"x": 15,
"y": 0.26,
"name": "2022-01-26T15:00:00",
"resolution": "hourly"
},
{
"x": 16,
"y": 0.0,
"name": "2022-01-26T16:00:00",
"resolution": "hourly"
},
{
"x": 17,
"y": 0.52,
"name": "2022-01-26T17:00:00",
"resolution": "hourly"
},
{
"x": 18,
"y": 0.26,
"name": "2022-01-26T18:00:00",
"resolution": "hourly"
},
{
"x": 19,
"y": 0.13,
"name": "2022-01-26T19:00:00",
"resolution": "hourly"
},
{
"x": 20,
"y": 0.13,
"name": "2022-01-26T20:00:00",
"resolution": "hourly"
},
{
"x": 21,
"y": 0.0,
"name": "2022-01-26T21:00:00",
"resolution": "hourly"
},
{
"x": 22,
"y": 0.13,
"name": "2022-01-26T22:00:00",
"resolution": "hourly"
},
{
"x": 23,
"y": 0.13,
"name": "2022-01-26T23:00:00",
"resolution": "hourly"
}
]
},
"usage": {
"data": [
{
"x": 0,
"y": 0.0,
"name": "2022-01-26T00:00:00",
"resolution": "hourly"
},
{
"x": 1,
"y": 0.0,
"name": "2022-01-26T01:00:00",
"resolution": "hourly"
},
{
"x": 2,
"y": 1.0,
"name": "2022-01-26T02:00:00",
"resolution": "hourly"
},
{
"x": 3,
"y": 1.0,
"name": "2022-01-26T03:00:00",
"resolution": "hourly"
},
{
"x": 4,
"y": 1.0,
"name": "2022-01-26T04:00:00",
"resolution": "hourly"
},
{
"x": 5,
"y": 0.0,
"name": "2022-01-26T05:00:00",
"resolution": "hourly"
},
{
"x": 6,
"y": 1.0,
"name": "2022-01-26T06:00:00",
"resolution": "hourly"
},
{
"x": 7,
"y": 1.0,
"name": "2022-01-26T07:00:00",
"resolution": "hourly"
},
{
"x": 8,
"y": 1.0,
"name": "2022-01-26T08:00:00",
"resolution": "hourly"
},
{
"x": 9,
"y": 1.0,
"name": "2022-01-26T09:00:00",
"resolution": "hourly"
},
{
"x": 10,
"y": 1.0,
"name": "2022-01-26T10:00:00",
"resolution": "hourly"
},
{
"x": 11,
"y": 3.0,
"name": "2022-01-26T11:00:00",
"resolution": "hourly"
},
{
"x": 12,
"y": 1.0,
"name": "2022-01-26T12:00:00",
"resolution": "hourly"
},
{
"x": 13,
"y": 1.0,
"name": "2022-01-26T13:00:00",
"resolution": "hourly"
},
{
"x": 14,
"y": 0.0,
"name": "2022-01-26T14:00:00",
"resolution": "hourly"
},
{
"x": 15,
"y": 2.0,
"name": "2022-01-26T15:00:00",
"resolution": "hourly"
},
{
"x": 16,
"y": 0.0,
"name": "2022-01-26T16:00:00",
"resolution": "hourly"
},
{
"x": 17,
"y": 4.0,
"name": "2022-01-26T17:00:00",
"resolution": "hourly"
},
{
"x": 18,
"y": 2.0,
"name": "2022-01-26T18:00:00",
"resolution": "hourly"
},
{
"x": 19,
"y": 1.0,
"name": "2022-01-26T19:00:00",
"resolution": "hourly"
},
{
"x": 20,
"y": 1.0,
"name": "2022-01-26T20:00:00",
"resolution": "hourly"
},
{
"x": 21,
"y": 0.0,
"name": "2022-01-26T21:00:00",
"resolution": "hourly"
},
{
"x": 22,
"y": 1.0,
"name": "2022-01-26T22:00:00",
"resolution": "hourly"
},
{
"x": 23,
"y": 1.0,
"name": "2022-01-26T23:00:00",
"resolution": "hourly"
}
]
},
"costDelayed": {
"data": []
},
"usageDelayed": {
"data": []
},
"temp": {
"data": [
{
"x": 0,
"y": 40.0,
"name": "2022-01-26T00:00:00",
"resolution": "hourly"
},
{
"x": 1,
"y": 38.0,
"name": "2022-01-26T01:00:00",
"resolution": "hourly"
},
{
"x": 2,
"y": 36.0,
"name": "2022-01-26T02:00:00",
"resolution": "hourly"
},
{
"x": 3,
"y": 34.0,
"name": "2022-01-26T03:00:00",
"resolution": "hourly"
},
{
"x": 4,
"y": 33.0,
"name": "2022-01-26T04:00:00",
"resolution": "hourly"
},
{
"x": 5,
"y": 32.0,
"name": "2022-01-26T05:00:00",
"resolution": "hourly"
},
{
"x": 6,
"y": 31.0,
"name": "2022-01-26T06:00:00",
"resolution": "hourly"
},
{
"x": 7,
"y": 31.0,
"name": "2022-01-26T07:00:00",
"resolution": "hourly"
},
{
"x": 8,
"y": 32.0,
"name": "2022-01-26T08:00:00",
"resolution": "hourly"
},
{
"x": 9,
"y": 37.0,
"name": "2022-01-26T09:00:00",
"resolution": "hourly"
},
{
"x": 10,
"y": 41.0,
"name": "2022-01-26T10:00:00",
"resolution": "hourly"
},
{
"x": 11,
"y": 45.0,
"name": "2022-01-26T11:00:00",
"resolution": "hourly"
},
{
"x": 12,
"y": 49.0,
"name": "2022-01-26T12:00:00",
"resolution": "hourly"
},
{
"x": 13,
"y": 50.0,
"name": "2022-01-26T13:00:00",
"resolution": "hourly"
},
{
"x": 14,
"y": 51.0,
"name": "2022-01-26T14:00:00",
"resolution": "hourly"
},
{
"x": 15,
"y": 51.0,
"name": "2022-01-26T15:00:00",
"resolution": "hourly"
},
{
"x": 16,
"y": 49.0,
"name": "2022-01-26T16:00:00",
"resolution": "hourly"
},
{
"x": 17,
"y": 47.0,
"name": "2022-01-26T17:00:00",
"resolution": "hourly"
},
{
"x": 18,
"y": 44.0,
"name": "2022-01-26T18:00:00",
"resolution": "hourly"
},
{
"x": 19,
"y": 41.0,
"name": "2022-01-26T19:00:00",
"resolution": "hourly"
},
{
"x": 20,
"y": 39.0,
"name": "2022-01-26T20:00:00",
"resolution": "hourly"
},
{
"x": 21,
"y": 37.0,
"name": "2022-01-26T21:00:00",
"resolution": "hourly"
},
{
"x": 22,
"y": 36.0,
"name": "2022-01-26T22:00:00",
"resolution": "hourly"
},
{
"x": 23,
"y": 35.0,
"name": "2022-01-26T23:00:00",
"resolution": "hourly"
}
]
}
}
}

PR here #27 @apearson @mcondren

Thanks for your work @alex-hall !

Thanks for all the work and effort @alex-hall (thank you @mcondren for pointing it out).  I'm going to take a look at the PR and see how we can get this integrated into the library.  Might take me a couple of days or so but know I'm looking at it!
Yes, they stringify the json for some reason.  I'm assuming it's being passed by an internal system and they aren't parsing before sending to the frontend.

2.2.0 includes the changes in #27
Going to close for now but feel free to open back up and let me know if you run into any issues @alex-hall @mcondren


Rename DAO to Model

What is this pull request?
This PR makes rename DAO to Model.
Why do you implement it? Why do we need this pull request?
In my opinion, the name DAO is not good because it creates misleading.
I thought DAO has a function to operate the data when I seen it. In fact, it is a representation of data, it has not a responsibility of data's operations.

This is good point that dao is misleading. For now we need focus on https://github.com/hyperledger/iroha/pull/417, because it create general pipeline of transactions, proposals and blocks. After adoption this PR we may discuss changing name from dao to another one.


Pre-PR discussion: JSON Schema additions (to conform to spec)

Feature Request | Pre-Pull Request discussion
First, thanks for Pydantic. "Just types". It's awesome. :tada: :taco: :cake:
Summary
I want to update the schema generation in JSON to implement the JSON Schema spec, which is in the process of becoming an IETF RFC.
But first, I want to discuss what would be the most appropriate way of doing it to keep it aligned with the project's direction.
Sorry for the long post. I want to be as explicit as possible and demonstrate the motivation clearly.
Current state
I know the great work done at https://github.com/samuelcolvin/pydantic/pull/190 which implements the current schema generation in a JSON format. It was first discussed at https://github.com/samuelcolvin/pydantic/issues/129.
The current implementation generates a schema in JSON format. But is not the same as the one described in the JSON Schema specification.
The changes/additions required are relatively small, as most of the work is already done. But they are spread across the project. But the benefits could be quite big for the ecosystem around Pydantic.
Motivation
First, some motivation for having the "JSON Schema" spec instead of the current schema generation.
There is currently quite some support, tools and "ecosystem" for "JSON Schema" (as defined in the spec).
As a simple example, easy to test, Visual Studio Code has good support for JSON Schema, including auto-completion and type checking.

Take the schema from the docs:

{
  "type": "object",
  "title": "Main",
  "description": "This is the description of the main model",
  "properties": {
    "foo_bar": {
      "type": "object",
      "title": "FooBar",
      "properties": {
        "count": {
          "type": "int",
          "title": "Count",
          "required": true
        },
        "size": {
          "type": "float",
          "title": "Size",
          "required": false
        }
      },
      "required": true
    },
    "Gender": {
      "type": "int",
      "title": "Gender",
      "required": false,
      "choices": [
        [1, "Male"],
        [2, "Female"],
        [3, "Other"],
        [4, "I'd rather not say"]
      ]
    },
    "snap": {
      "type": "int",
      "title": "The Snap",
      "required": false,
      "default": 42,
      "description": "this is the value of snap"
    }
  }
}

Save it to a file schema.json
Open it in VS Code.
Add a single extra field (only for this test purposes) declaring the $schema of this JSON document, to define it as a "JSON Schema", with: "$schema": "http://json-schema.org/draft-07/schema" (VS Code will auto-complete that field schema declaration too). That field, added there only for this test, tells VS Code that this file is itself a JSON Schema (because the JSON Schema spec itself is declared using JSON Schema).
Then, just hover over the "error highlighted" sections, the editor shows some of the things that would need to be changed.

Then, for example, editing one of the type values that has float, removing it and hitting Ctrl + Space to trigger the auto-complete, shows the types defined by the spec.

Now, let's say that the generated schema was using the JSON Schema spec format, note how the changes are small and simple:

{
    "$schema": "http://json-schema.org/draft-07/schema",
    "type": "object",
    "title": "Main",
    "description": "This is the description of the main model",
    "required": [
      "foo_bar"
    ],
    "properties": {
      "foo_bar": {
        "type": "object",
        "title": "FooBar",
        "properties": {
          "count": {
            "type": "integer",
            "title": "Count"
          },
          "size": {
            "type": "number",
            "title": "Size"
          }
        },
        "required": [
          "count"
        ]
      },
      "Gender": {
        "type": "string",
        "title": "Gender",
        "enum": [
          "Male",
          "Female",
          "Other",
          "I'd rather not say"
        ]
      },
      "snap": {
        "type": "integer",
        "title": "The Snap",
        "default": 42,
        "description": "this is the value of snap"
      }
    }
  }

Save that file to ./schema2.json
Now create a new JSON file. For this example, to get VS Code completion, create a field of "$schema": "./schema2.json". It defines this JSON file as using the $schema defined above in the previous file:

{
    "$schema": "./schema2.json"
}

Now start creating a new JSON field / key, VS Code will provide automatic completion, based on the schema.

It has type error detection.

And it has completion even for enums.

VS Code is just an example of the current support for JSON Schema (as in the spec) given by tools and the ecosystem. There are many other examples, but this one seemed quite simple and explicit.

Additionally, OpenAPI (previously known as Swagger), which is now part of the Linux foundation, is based on JSON Schema up to some extent.
My ultimate plan is to use Pydantic to generate OpenAPI schemas from Python web APIs. Those schemas can then be used directly with systems like Swagger UI: https://petstore.swagger.io to create interactive and explorable documentation for APIs. And to generate Open API clients in different programing languages, etc.
But all that would come afterwards, as my personal experiments. What I want to add to Pydantic now, which would be the first step for my future plans, is support for JSON Schema (as in the spec).
What I propose
I see different ways to achieve what I want:
A. Modify the current schema related code
This would probably be the most straightforward way. It would minimize code duplication and confusion, as there are already schema_json methods.
It would be a breaking change for anyone that already requires the currently generated schema. So, I guess a version bump would be required.
B. Add additional JSON Schema (spec) code
This wouldn't disrupt implementations based on the current functionality, wouldn't required a version bump.
But it would add quite a bunch of code to maintain, as it would have to touch a lot of the methods in several points, so instead of touching them, it would be duplicating all that functionality with little changes.
C. Add some minimal changes to expose data and create an additional module
I could implement it all as a set of functions, without touching the methods from the original classes. But for example, I need to access BaseModel.__fields__, which is a private property. The one I see available is BaseModel.fields, but that's an instance @property method, not a class method. And all the schema generation is done at the class level, not the instance level.
So, I could add little changes to be able to access the data and process it with additional isolated functions in a separated module, part of the same package.
D. Add additional functions that access private properties
Implement it as functions that use BaseModel.__fields__ and related properties, even though most of them are private properties. It woudn't be very clean, but wouldn't touch the current actual code.
E. Expose minimal data and create separate package
Add just the minimum changes to access the data. And create a new package that uses that data and generates JSON Schema from that, but not part of Pydantic.
F.

I already started with some local tests, and already tried starting with some of the possible paths above, but I see that I'll end up touching a lot of code to implement it in one of those ways, and it might not be the preferred way. So I decided to better just ask first what seems more viable for the project and start the discussion here.
What do you think? How should I proceed?

First of all, sorry for the delayed response, I was on holiday, then life got in the way of code.
In general, my answer is very simple: yes. Let's go for it.
Let's do A., frankly I'm not that bothered by backwards compatibility on things like this, I'd rather get it right. @Gr1N I think uses schema a fair bit, would this be ok way you?
The reason I shied away from JSON Schema :tm: are explained in #129, and also that implementing a schema seemed pretty complicated at the time, I wanted to start with something simple.
Questions/concerns:

what happens about errors? Presumably JSON Schema stipulates what error responses should look like? I'm happy to support "JSON Schema formated error respones", but I wouldn't want to change the standard error details format since I suspect this is used a lot more than schema.
do pydantic's types (and custom types) map sensibly to JSON Schema types? If not what's the work around.
I'm happy to review code and make some changes, but @tiangolo are you proposing to implement this yourself or asking me to? (hint: it would be great if you could do the lion's share of the work :wink:)

[...] delayed response [...]

No worries, it happens to me all the time :sweat_smile: I hope you had a nice holiday.

[...] Let's go for it. [...]

Awesome!
Actually, I already started :smile: .
...I have been reading, re-reading and studying the specifications for JSON Schema and OpenAPI, there are a couple of differences but are minimal. My plan is to update the implementation for JSON Schema in a way that is compatible with OpenAPI schemas too, taking into account the minor differences. That way it can be used with JSON Schema directly, but also others (me) can develop OpenAPI stuff based on it.

[...] Let's do A [...]

Awesome again!

The reason I shied away from JSON Schema [...]

Yep. I understand.

what happens about errors? [...]

In short: we don't change them.
There's no need to change the errors. JSON Schema actually doesn't stipulate how to return the errors.
OpenAPI allows defining error shapes using the same schema format. It's optional and doesn't force any shape on them.

do pydantic's types (and custom types) map sensibly to JSON Schema types? [...]

Yes, in most of the cases. Pydantic maps very well from JSON, and serializes to JSON very well too. JSON Schema just declares JSON shapes, so I think it matches Pydantic very well. Specially custom types and sub-types/sub-models (what is referred to "recursive models" in the docs), all those get declared as sub schemas, very similar to what the actual Pydantic based code looks like.
Limitations: JSON Schema defines shapes for JSON content, and JSON itself has limitations compared to all the Pyhton data structures.
My plan is to update/implement JSON Schema support on top of Pydantic, of course, without imposing any limitation on the rest of Pydantic.
There will be some limitations for those (me) who want to have JSON Schema based on Pydantic, but it won't limit Pydantic's general usage.
Specific examples I've found so far (I don't think there's much more):

In JSON, object's keys must be strings. So, it's not possible to declare Python dicts with keys other than strings as JSON. E.g. dicts with keys of numbers, True, False (other objects or classes wouldn't even make much sense as keys in JSON). This is not a limitation of JSON Schema, but JSON itself. In that case, we, developers using Pydantic, should make sure not to declare dicts with keys other than str.

Python Enum would map to enum in JSON Schema, but in JSON Schema it is a predefined list of values, not of keys that map to values. But it is a very straightforward simple change from the currently generated schema.

...in most of the other cases is just a change in names, e.g. from list to array, from set to array with a uniqueItems set to true, etc.

[...] the workaround [...]

The "workaround" is mainly: those of us who want to use Pydantic to generate JSON Schema should not declare models with types that are not compatible (dicts with keys other than str). But it doesn't affect any other use or functionality of Pydantic.

are you proposing to implement this yourself or asking me to?

Yes! Of course, I'll do it. I already started :nerd_face: ...and this is what I have planned for this whole weekend, at least to start :joy: .

Here are a couple things I have in mind and some questions:

While generating one schema, I want to also define sub-schemas as top-level schemas with references. The end result for JSON Schema consumers wouldn't change. But this will help, for example, automatic code generation tools to re-use the generated code for one schema referencing it on the others, avoiding code and schema duplication.

I had already started creating a set of functions to do the work, I based them on the current methods of the BaseModel and Field classes. Having the schema logic in separated functions in a separate module would also allow using them independently of a specific model class. Specifically, it would allow passing a set of classes and getting a single JSON schema that defines them as several sub-schemas (and this can then be re-used by other tools, like OpenAPI stuff). And We can call these functions from inside the existing methods for Field and BaseModel. What do you think?

Is there any reason why you wanted to have Field._schema as a "protected member" (because of the leading _)? I would like to change it from Field._schema to Field.schema, to use it from those functions (described above). What do you think?

Later, probably in a different PR after this one, I would like to extend the Schema class to receive as keyword arguments several (most) of the validation keywords from JSON Schema. Then use them to do the schema declaration but also to set the corresponding validations at the same time.

The first advantage would be to be able to define validations and the declaration of those validations in the same place, avoiding double declaration.
The second advantage would be with IDEs and editors, when using Pydantic's Exotic Types, using functions as constr as type declarations, PyCharm gets the correct type hint from the return of the function. But VS Code with the Python "language server" or editors based on Jedi (VS Code too, Vim, Atom, Sublime, etc) don't get the type hint from the function's return type (the str type returned by constr). By setting the equivalent of min_length=2, max_length=10 from constr as parameters in the Schema and declaring the type as str, these editors (and PyCharm too) would be able to get the hint that the type is str, to provide completion, etc. This on top of the previous advantage.

It would allow, instead of declaring:
short_str: constr(min_length=2, max_length=10) = Schema(None, minLength=2, maxLength=10)

...declaring:
short_str: str = Schema(None, min_length=2, max_length=10)

What do you think? Would it seem acceptable? ...although this is probably a question for a subsequent PR.

I'm using schema only for documentation, so I would be happy for any improvements on this!

While generating one schema, I want to also define sub-schemas as top-level schemas with references.

Not sure what you mean by this. Perhaps easiest to submit a PR and we can discuss from there.

I had already started creating a set of functions to do the work

Yes, moving as much schema logic to a separate module as possible would be good

Is there any reason why you wanted to have Field._schema as a "protected member"

Because I assume that in general users wouldn't want to access _schema directly. Much of python (eg. _asdict on named tuples) uses _ as a more of a warning than a hard barrier to external access. I don't mind changing it though.

Schema(None, minLength=2, maxLength=10)

I'm happy with that as long as constr(min_length=2, max_length=10) continues to work too.

[...] submit a PR and we can discuss from there.

Sure, I'm on it.

[...] moving as much schema logic to a separate module [...]

Cool.

[...] general users wouldn't want to access _schema directly. [...] _asdict on named tuples [...]

Good point, thanks for the reference to the named tuples. I'll just leave it as is and don't pay attention to Pylint's complaints about that.

I'm happy with that as long as constr(min_length=2, max_length=10) continues to work too.

Awesome. I'll do that next.

I'll close this issue as the PR directly related to it is already merged into master. :tada:

thank you @tiangolo


drivers: slip: Consistently log pkt/buf allocation failures

There are 3 cases of possible allocation failures, only 1 of them
was logged. Now, all the cases are logged: 1) failure to allocate
net_pkt; 2) failure to allocate very first net_buf for it; 3)
failure to allocate additional net_buf for it (this latter was
the only one logged previously).
Signed-off-by: Paul Sokolovsky<EMAIL_ADDRESS>

This comes from investigating #4022.


Discussion opencascade (DLR-SC) vs occt (conda-forge)

This is not so much an issue but rather a discussion I'ld like to have with people involved with opencascade and its python wrapper.
@rainman110, @looooo you both packaged opencascade 7.4 for conda. But you did it differently.
opencascade 7.4.0 has the following recipe:
source:
  fn: {{ name }}-{{ version }}.tar.gz
  url: http://git.dev.opencascade.org/gitweb/?p=occt.git;a=snapshot;h=V7_4_0;sf=tgz
  sha256: 9614c2075cef53e30b25317fc8fcfb71d60a69181d562e1c56469ed8e54ad758
  patches:
    - no-xmu.patch     # [linux]
    - fix-private-linking.patch
    - fix_brepblend.patch

build:
  number: 4
  detect_binary_files_with_prefix: True

requirements:
  build:
    - {{ compiler('cxx') }}
    - {{ cdt('mesa-libgl-devel') }}     # [linux]
    - {{ cdt('libxcb') }}               # [linux]
    - {{ cdt('libxi-devel') }}          # [linux]
    - {{ cdt('libx11-devel') }}         # [linux]
    - {{ cdt('libxau-devel') }}         # [linux]
    - {{ cdt('libxext-devel') }}        # [linux]
    - {{ cdt('xorg-x11-proto-devel') }} # [linux]
    - cmake
    - ninja
  host:
    - libglu # [linux]
    - freeimageplus
    - tbb
    - tbb-devel
    - python
    - freetype
    - fontconfig # [unix]

  run:
    - freeimageplus
    - tbb
    - freetype
    - six

Versus this package recipe
source:
  fn: {{ name }}-{{ version }}.tar.gz
  url: http://git.dev.opencascade.org/gitweb/?p=occt.git;a=snapshot;h=fd47711d682be943f0e0a13d1fb54911b0499c31;sf=tgz
  sha256: af73c89ade0467ca7351dc445e4bd20832387da1e205f6955bc9356ab7d9f22b

build:
  number: 1

requirements:
  build:
    - {{ compiler('cxx') }}
    - {{ cdt('mesa-libgl-devel') }}  # [linux]
    - {{ cdt('libxi-devel') }}       # [linux]
    - cmake
    - ninja
  host:
    - freetype
    - fontconfig
    - tbb-devel
    - xorg-libxt  # [linux]
    - xorg-libxmu  # [linux]
  run:
    - freetype
    - fontconfig
    - xorg-libxt  # [linux]
    - xorg-libxmu  # [linux]

Questions:

What are the difference? It seems that in the DLR package, the dependency to XMU has been dropped (why?) and that TBB has been made a private dependency to not make it apparent for instance when packaging pythonocc-core.
Would it be possible to converge on a common recipe?
Note: it seems that you are not using the same commit of opencascade.

Note: it seems that you are not using the same commit of opencascade.

It should be the same commit, but we should change the url used for the conda-forge package.
http://git.dev.opencascade.org/gitweb/?p=occt.git;a=commit;h=fd47711d682be943f0e0a13d1fb54911b0499c31

What are the difference? It seems that in the DLR package, the dependency to XMU has been dropped (why?) and that TBB has been made a private dependency to not make it apparent for instance when packaging pythonocc-core.

Difficult to say. Maybe comparing the linux build scripts:
dlr-sc
cmake -G "Ninja" ..

Freeimage is disabled. We had an issue with that module in freecad, but I can't remember what that was. Does anybody need freeimage support?

Would it be possible to converge on a common recipe?

The question is if there is anything missing for the conda-forge package which is provided in dlr-sc. ?

@fcollonval The dlr-sc opencascade package is derived from the occt feedstock.
There are the following changes:

Removed xmu (it is actually not required to build occt). For me it is important to have no dependencies to conda-forge. the dlr-sc repo should be used without any other repository.
Changed Xorg packages to use the system dev libraries. I don't like the approach the have a second operating system inside conda (like the X11 libraries). I often had problems with it and I think it is better to use the user's libraries instead.
Added Freeimage: We need it to e.g. enable background images or screen dumps
The commit hash: We use the release tag of opencascade 7.4.0.
Private linking: Other software using occt should not be forced to link in TBB. Instead, TBB is technically a private dependency of occt. In the case of the package it means, that we don't have any hard-coded lib paths in the cmake config files, which are PITA when relocating the package.

I did some testing and installed dlr-sc/opencascade instead of occt and it seems like these packages are compatible.

Removed xmu (it is actually not required to build occt). For me it is important to have no dependencies to conda-forge. the dlr-sc repo should be used without any other repository.

We should apply this for the conda-forge recipe.

Changed Xorg packages to use the system dev libraries. I don't like the approach the have a second operating system inside conda (like the X11 libraries). I often had problems with it and I think it is better to use the user's libraries instead

Conda-forge had different approaches how to deal with graphics libraries. As I never used the gui parts of occt I am open to any changes here.

Added Freeimage: We need it to e.g. enable background images or screen dumps

I would like to keep the occt-package independent from freeimage as long as nobody requests this dependency.

The commit hash: We use the release tag of opencascade 7.4.0.

should be applied on the occt-recipe

Private linking: Other software using occt should not be forced to link in TBB. Instead, TBB is technically a private dependency of occt. In the case of the package it means, that we don't have any hard-coded lib paths in the cmake config files, which are PITA when relocating the package.

This avoids some issues. I think it's fine to apply this patch for this repo.
Btw.: Out of curiosity: what's the reason for not using any conda-forge packages?

@looooo We often had issues with the conda-forge ecosystem which had to many alternatives to the official packages. Still, many package recipes are from conda forge and might only have small or no modifications at all.
As we (at DLR) don't want to risk any of these issues, we chose to have a own conda channel for which we have full control. This allows us to have e.g. custom patches to opencascade that are required for our software to work properly.

@looooo By the way, I just added a new cmake defintion to the build scripts:
-D BUILD_RELEASE_DISABLE_EXCEPTIONS=OFF

These enable exceptions in release mode. Otherwise, we experienced nasty segfaults instead of nice exceptions. See here:
https://github.com/tpaviot/pythonocc-core/issues/778

Thank you both of you for the complete and detailed answer.
So it indeed makes sense to have two different version. I should probably  prefer the conda-forge then.
Thanks for sharing your experience @rainman110
@looooo you have the best view on how to improve the recipe for the comments made here. But if you need help let me know, I help if I can. Can I close this issue or do you want to push a PR linked to this issue?

@looooo We often had issues with the conda-forge ecosystem which had to many alternatives to the official packages. Still, many package recipes are from conda forge and might only have small or no modifications at all.

For freecad we use the freecad channel on top of conda-forge. This way it's easy to replace any package from conda-forge in a fast way.

@looooo you have the best view on how to improve the recipe for the comments made here. But if you need help let me know, I help if I can. Can I close this issue or do you want to push a PR linked to this issue?

Best to keep this issue open. I would like to apply some of the points mentioned with the next Iteration.

https://github.com/conda-forge/occt-feedstock/pull/35
@rainman110 Do you think the patches could be applied upstream (tbb, xmu)?

@looooo What do you mean with upstream? The OCCT feedstock? If yes, they can be upstream. But if you mean OCCT itself, these patches are probably not good enough to be included into the OCCT code base. They are a quick and dirty solution for us.

But if you mean OCCT itself, these patches are probably not good enough to be included into the OCCT code base. They are a quick and dirty solution for us.

At least they should know about the issues...

Sure, you can post an issue at https://tracker.dev.opencascade.org/.  It might take a long time though until they'll address the issue (at least in my experience).

The issues are similar to the one here:
https://tracker.dev.opencascade.org/view.php?id=30977

It might take a long time though until they'll address the issue (at least in my experience).

You are right. FreeCAD also encountered this problem in the past...

I guess we can close this one now @fcollonval


Issue #40 - fixed detachable checkbox value in admin/gall

setGallDetails in admin/gall was translating the fp-ts Some's 1/0 into 'yes'/'no' when the checkbox expects true/false

Superseded by  #85


chore: refactor cpuset in package for code reuse

Signed-off-by: Joseph<EMAIL_ADDRESS>Ⅰ. Describe what this PR does
There are multiple implementations of cpuset in the current code, which are now refactored and integrated.

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it
Ⅳ. Special notes for reviews
V. Checklist

[x] I have written necessary docs and comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in make test

/lgtm

/lgtm


Regression tests for find() and aggregate() using primary RP within transaction

See: https://github.com/mongodb/mongo-php-library/issues/562
This is a regression test for CDRIVER-2744. It builds upon https://github.com/mongodb/mongo-php-library/pull/563 (for PHPLIB-373) and may be fixed by either:

PHPLIB-375: omitting read preference option for commands and queries sent in non-sharded topologies
PHPC-1236: Bumping libmongoc to 1.11.1 (once released)

I don't think we should merge this while failing and would suggest leaving it open until one of the solutions can be implemented.

@derickr: This has been rebased on master, so please review at your leisure. I don't expect the ext-mongodb bump is controversial, since the master branch will likely become PHPLIB 1.5.0 and depend on PHPC 1.6.0.

@derickr: I created generic create/drop collection helpers that apply a majority WC when supported by the server. Testing locally, there was no problem using a majority WC with a standalone, but we'll see what Travis says.
I left some explicit uses of CreateCollection and DropCollection operations as-is, particularly when they were using custom collection names.


Local invoke fails when trying to use Layers locally

Description:
When trying to use the new Layers feature locally it fails due to not finding a Dockerfile which seems to be created at runtime -- Haven't dig too much into it yet as I just bumped into that.

Steps to reproduce the issue:

Initialize a boilerplate app: sam init -r python; cd sam-app
Install dependencies into a lib folder: pip install -r hello_world/requirements.txt -t lib/
Create a Layer in template.yaml:

    RequestsLayer:
        Type: 'AWS::Serverless::LayerVersion'
        Properties:
            ContentUri: lib/

Reference that Layer in the HelloWorldFunction resource:

    HelloWorldFunction:
        Type: AWS::Serverless::Function
        Properties:
            CodeUri: hello_world/
            Handler: app.lambda_handler
            Runtime: python3.6 # also tried python3.7 (default)
            Layers:
                - !Ref RequestsLayer

Run function locally: sam local invoke HelloWorldFunction --no-event

Observed result:
Local invocation fails quickly and returning the trace data at the bottom.
Expected result:
Build Layer locally and local invocation to succeed.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc) OSX Sierra 10.12.6 (16G1618)
Output of sam --version: SAM CLI, version 0.8.0
Optional Debug logs:
Add --debug flag to command you are running
sam local invoke HelloWorldFunction --no-event
2018-11-30 08:42:46 Found credentials in shared credentials file: ~/.aws/credentials
2018-11-30 08:42:46 Invoking app.lambda_handler (python3.6)
2018-11-30 08:42:46 RequestsLayer is a local Layer in the template
2018-11-30 08:42:46 Image was not found.
2018-11-30 08:42:46 Building image...
Traceback (most recent call last):
  File "/Users/lessa/.pyenv/versions/work/bin/sam", line 11, in <module>
    sys.exit(cli())
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/samcli/commands/local/invoke/cli.py", line 56, in cli
    parameter_overrides)  # pragma: no cover
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/samcli/commands/local/invoke/cli.py", line 99, in do_cli
    stderr=context.stderr)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/samcli/commands/local/lib/local_lambda.py", line 75, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/samcli/local/lambdafn/runtime.py", line 81, in invoke
    debug_options=debug_context)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/samcli/local/docker/lambda_container.py", line 66, in __init__
    image = LambdaContainer._get_image(image_builder, runtime, layers)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/samcli/local/docker/lambda_container.py", line 157, in _get_image
    return image_builder.build(runtime, layers)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/samcli/local/docker/lambda_image.py", line 113, in build
    self._build_image(base_image, image_tag, downloaded_layers)
  File "/Users/lessa/.pyenv/versions/3.6.2/envs/work/lib/python3.6/site-packages/samcli/local/docker/lambda_image.py", line 173, in _build_image
    with open(str(full_dockerfile_path), "w") as dockerfile:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/lessa/.aws-sam/layers-pkg/dockerfile_3a7f092d-0b92-42c9-b823-19c2401f44c5'

I can confirm this bug using Runtime: ruby2.5.
I even have a problem packaging this setup. The packaging didn't uploads a layer to S3 and didn't changes the ContentUri property.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
Resources:
  Lambda:
    Properties:
      CodeUri: s3://sam-applications/12ed7a722e38966db52f8b5d9bd0687d
      Events:
        Callback:
          Properties:
            Method: post
            Path: /
          Type: Api
      Handler: app.lambda_handler
      Layers:
      - Ref: LibLayer
      Runtime: ruby2.5
    Type: AWS::Serverless::Function
  LibLayer:
    Properties:
      ContentUri: lib/
    Type: AWS::Serverless::LayerVersion
Transform: AWS::Serverless-2016-10-31

Adding a bit more to this - I ran into the same issue and manually created the ~/.aws-sam/layers-pkg directory.   The next build fails with
16:55 $ sam local invoke -e test-event.json test
2018-12-02 16:57:44 Found credentials in shared credentials file: ~/.aws/credentials
2018-12-02 16:57:44 Invoking test.handler (nodejs8.10)
2018-12-02 16:57:44 layerCore is a local Layer in the template
2018-12-02 16:57:44 Image was not found.
2018-12-02 16:57:44 Building image...
2018-12-02 16:57:45 Failed to build Docker Image
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/samcli/local/docker/lambda_image.py", line 187, in _build_image
    pull=not self.skip_pull_image)
  File "/usr/lib/python2.7/site-packages/docker/models/images.py", line 283, in build
    raise BuildError(chunk['error'], result_stream)
BuildError: Unknown flag: chown
Error: Building Image failed.

I did grab the dockerfile generated by sam prior to the process cleaning up:
FROM lambci/lambda:nodejs8.10
ADD --chown=sbx_user1051:495 layerCore /opt

Please DO NOT manually create ~/.aws-sam/layers-pkg for any reason! This directory is the cache for layers and other directories SAM CLI could create in the future. SAM CLI is downloading code to that directory and you are running the risk of having the directory improperly scoped.
We only create that directory on the downloading of a layer. We need to do this if you are referencing a layer. This bug will only impact you, if you are first trying to use layers without downloading a layer.
@timoschilling What version of AWS CLI are you running? Make sure you have upgraded to the latest version otherwise you will not be able to package LayerVersions.
@PrivoAdam What version of docker are you running?

@jfuss Thanks for the clarification on the ~/.aws-sam/layers-pkg directory.
I'm running Fedora 28 with the default docker version 1.13.1.  I've moved to the docker CE repo now with 18.09.0 which supports --chown in ADD.  It looks like anyone using under docker 17.09.0 may run into this.

@jfuss I'm on 0.8.1

I'm have the same bug of @timoschilling.
My version is 0.8.0

@timoschilling That is SAM CLI's version not AWS CLI. SAM CLI shells out to AWS CLI for package and deploy. Therefore you need to update AWS CLI to a min version of 1.16.67. If there is some other issue that does not relate to the layer_cache being corrected, please cut a new issue so we can keep this issue focused on the original bug.
@rodrigomaia: Update AWS CLI to min version of 1.16.67.

@jfuss ok, works now, thanks. Why SAM CLI didn't requires an actual version of the AWS CLI?

Works... tks.

@jfuss Hi i still have the same issue with sam local invoke and local layers. With python3.6 and the most current CLIs
SAM CLI, version 0.8.1
aws-cli/1.16.74 Python/3.6.5 Darwin/18.2.0 botocore/1.12.64
Docker version 18.09.0, build 4d60db4
Traceback (most recent call last):
  File "/Users/db/Library/Python/3.6/bin/sam", line 11, in <module>
    sys.exit(cli())
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/samcli/commands/local/invoke/cli.py", line 56, in cli
    parameter_overrides)  # pragma: no cover
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/samcli/commands/local/invoke/cli.py", line 99, in do_cli
    stderr=context.stderr)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/samcli/commands/local/lib/local_lambda.py", line 75, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/samcli/local/lambdafn/runtime.py", line 81, in invoke
    debug_options=debug_context)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/samcli/local/docker/lambda_container.py", line 66, in __init__
    image = LambdaContainer._get_image(image_builder, runtime, layers)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/samcli/local/docker/lambda_container.py", line 157, in _get_image
    return image_builder.build(runtime, layers)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/samcli/local/docker/lambda_image.py", line 113, in build
    self._build_image(base_image, image_tag, downloaded_layers)
  File "/Users/db/Library/Python/3.6/lib/python/site-packages/samcli/local/docker/lambda_image.py", line 173, in _build_image
    with open(str(full_dockerfile_path), "w") as dockerfile:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/db/.aws-sam/layers-pkg/dockerfile_9c562307-7ace-47e6-977f-9f285dbb6d8c'

This was released in v0.9.0
Closing

I had the same Error: Invalid value for '--layer-cache-basedir'
Solved it with changing the permissions of my /.aws-sam/layers-pkg directory


files-cli: 2.13.209 -> 2.13.213

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/files-com/files-cli/releases.
meta.description for files-cli is: Files.com Command Line App for Windows, Linux, and macOS
meta.homepage for files-cli is: https://developers.files.com
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.13.213 with grep in /nix/store/xm4rgsrsvcwn4x3dmrp1mn6xmgg3ys5j-files-cli-2.13.213
found 2.13.213 in filename of file in /nix/store/xm4rgsrsvcwn4x3dmrp1mn6xmgg3ys5j-files-cli-2.13.213

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

files-cli

Instructions to test this update (click to expand)

Either download from the cache:
nix-store -r /nix/store/xm4rgsrsvcwn4x3dmrp1mn6xmgg3ys5j-files-cli-2.13.213 \
  --option binary-caches 'https://cache.nixos.org/ https://nixpkgs-update-cache.nix-community.org/' \
  --option trusted-public-keys '
  nixpkgs-update-cache.nix-community.org-1:U8d6wiQecHUPJFSqHN9GSSmNkmdiFW7GW7WNAnHW0SM=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The nixpkgs-update cache is only trusted for this store-path realization.)
For the cached download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A files-cli https://github.com/r-ryantm/nixpkgs/archive/a53e4b614459c384d004c128fa1fd7bf5c4c14f3.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/a53e4b614459c384d004c128fa1fd7bf5c4c14f3#files-cli

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/xm4rgsrsvcwn4x3dmrp1mn6xmgg3ys5j-files-cli-2.13.213
ls -la /nix/store/xm4rgsrsvcwn4x3dmrp1mn6xmgg3ys5j-files-cli-2.13.213/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review

x86_64-linux

:white_check_mark: 1 package built:

files-cli

Maintainer pings
cc @kashw2 for testing.

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.

@NixOS/nixpkgs-merge-bot merge


Reasonable numbers in TXT records

Right now, a TXT record looks a lot like this (two examples):
<IP_ADDRESS>.in-addr.arpa.rearview.m3047.net. 600 IN TXT "depth=2,first=1681183806.243363,last=1681408935.7175434,count=4,trend=20764.805014878275,update=1681440677.1266115,score=1.989519303716306"
<IP_ADDRESS>.in-addr.arpa.rearview.m3047.net. 600 IN TXT "depth=5,first=1680501488.6834428,last=1680501488.6837842,count=2,trend=3.404617309570313e-05,update=1681415738.623454,score=1.690577477672348"

There are two problems with this:

The floating points are obnoxiously (and unnecessarily) long.
The second trend value is in scientific notation because it's so small.

Some observations:

first, last, update are (*nix) datetimes. One second (integer) resolution is fine for historical purposes.
trend is a number of seconds. Likewise one second (integer) resolution is fine for historical purposes.
score probably doesn't need more than one decimal point of precision. Three decimal points is plenty of overkill.

I hope this is backwards compatible with whatever you using (outside of RVR itself) to parse TXT records.
I anticipate completing this change sometime in the next few weeks. I'll let it "soak" in the fwm branch for at least a few days, possibly as long as a week.

The update is on the fwm branch and I'm running it locally.

fwm merged to main at 18b384a

Verified the KNIME workflow still loads the data correctly.
Verified that the data loads correctly internally after a restart.

Examples:
<IP_ADDRESS>.in-addr.arpa.rearview.m3047.net. 600 IN TXT "depth=1,first=1682063117,last=1682063117,count=1,trend=0,update=1682063118,score=0.333"
<IP_ADDRESS>.in-addr.arpa.rearview.m3047.net. 600 IN TXT "depth=1,first=1682019224,last=1682019224,count=1,trend=0,update=1682019225,score=0.333"


Wrong result while Iceberg table read with Positional Delete

Bug description
When we have a data file & position delete file with multiple rowgroups then the result count is not coming right.
So if I try to DELETE all the rows from the original table, and try to select count from the resulted table it returns a non-zero count.
Steps To Reproduce -
Execution Engine - Presto
Data Format - PARQUET
presto> set session iceberg.parquet_writer_block_size='2MB';

presto> CREATE TABLE customer_v2 WITH (format = 'PARQUET') AS SELECT * FROM tpch.sf1.customer;
presto> select count(*), max(custkey), min(custkey) from  customer_v2;
 _col0  | _col1  | _col2
--------+--------+-------
 150000 | 150000 |     1
(1 row)

presto> DELETE FROM customer_v2 WHERE custkey >=1 AND custkey<=150000;

Current Query Output -
presto> select count(*) from customer_v2;
 _col0
-------
 57413
(1 row)

Expected Query Output -
presto> select count(*) from customer_v2;
 _col0
-------
0
(1 row)

System information
NA
Relevant logs
No response

@yingsu00 Can you take a look?
The count should be subtracted here, make sure the bitmask is created correctly:
https://github.com/facebookincubator/velox/blob/e8244c2e0fd0fcdb80372b126859493c57382ef0/velox/dwio/common/SelectiveStructColumnReader.cpp#L64

Thanks @Yuhta for looking into this. I will work on the fix.


net: network is unreachable on Amazon Linux 2

What version of Go are you using (go version)?
1.20
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
Amazon Linux 2 (arm64) (linux/arm64)
What did you do?
I'm trying to connect to my PostgreSQL cluster on Amazon RDS. The cluster has dual-stack mode enabled.
What did you expect to see?
Successful connection to the cluster.
What did you see instead?
 failed to connect to `host=collector.cluster-c3tdcjzw8by1.us-east-1.rds.amazonaws.com <redacted>`: dial error (dial tcp [2600:1f18:4769:c800:c1c6:ffd:fdbb:2045]:5432: connect: network is unreachable)

When running dig collector.cluster-c3tdcjzw8by1.us-east-1.rds.amazonaws.com, I get the following result:
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.5.2 <<>> collector.cluster-c3tdcjzw8by1.us-east-1.rds.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24492
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;collector.cluster-c3tdcjzw8by1.us-east-1.rds.amazonaws.com. IN A

;; ANSWER SECTION:
collector.cluster-c3tdcjzw8by1.us-east-1.rds.amazonaws.com. 5 IN CNAME collector-instance-1.c3tdcjzw8by1.us-east-1.rds.amazonaws.com.
collector-instance-1.c3tdcjzw8by1.us-east-1.rds.amazonaws.com. 5 IN A <IP_ADDRESS>

;; Query time: 2 msec
;; SERVER: <IP_ADDRESS>#53(<IP_ADDRESS>)
;; WHEN: Wed Feb 08 06:19:18 UTC 2023
;; MSG SIZE  rcvd: 151

Which gives the IPv4 <IP_ADDRESS>. When using this IP address instead of the hostname (host=<IP_ADDRESS>) the connection is successful.
There seems to be a similar issue here: https://groups.google.com/g/golang-nuts/c/j7ylMOg1qc4
Before anyone says "you've clearly misconfigured your AWS setup"; my setup is correct and I have tested it. The RDS security group allows IPv6 connections and the connector (EC2 instance in this case) has a security group which allows outbound access to port 5432 for IPv6. I've even added "all traffic" rules to both security groups to be 100% sure. My route tables are also correctly setup. Reachability Analyzer also confirms the network is reachable.

Turns out I forgot to enable "Enable auto-assign IPv6 address" in the connector's subnets (the instances connecting to the cluster), so the instances didn't have an IPv6 address, therefor being unable to connect to an IPv6 host.
Still, this error message is very vague. Shouldn't there be a more clear error message if you try to connect to an IPv6 host on a system with no IPv6 address instead of just "network is unreachable"? Or is this standard?

you will always have ipv6 addresses unless turned off at the kernel. What you might not have are addresses that can route to your intended destination.
choosing ipv6 over ipv4 is standard.


integrated protobuf into qpush build (not yet using for commands)

Putting @lsarrazi 's protobuf objects into the build for https://github.com/GrumpyOldTroll/quiche/issues/1

Note: also requires https://github.com/GrumpyOldTroll/chromium/pull/2 to build.
protobuf objects not actually used, just got it built and linking with a dummy usage in qpush_bin.cc.


Edge and Chrome throw a "the spinedata param is required" error

I don't know what else I should say about this...
Using pixi v3.0.10
It appears to work with Firefox.

Well, first - its about pixi-spine. Second - You didn't even make it past example. http://pixijs.github.io/examples/index.html?s=basics&f=container.js&title=Container&v=dev#/spine/dragon.js works just fine

added some examples for you

...
Yeah, that this is about pixi-spine is probably a pretty crucial detail.
And it would probably help to explain what exactly I'm doing:
I'm trying to get a copy of the example to run on it's own page, (locally, on my machine).
I think I implemented it correctly, but I also updated the Pixi.js to v4 before testing the code that was posted. And also I didn't recognize that the line of code was an example of what not to do, in the first place.
The example code pretty much resembles your examples, and changing it look closer to yours has no effect. (unsurprisingly, as I'm pretty such using the example javascript code. )
...Sorry I wasn't clear before.

Fortunately, pixi-spine works with both v3 and v4. What web-server do you use to host the app?

...web-server?
...I think I found my problem: I'm a total noob, and I didn't use a web server, and thus Edge and Chrome deny access to the JSON files.
tch.

Yeah, I'm a telepath :)
Next time please look if there's something strange in the console. For web-server i recommend install node.js, then do this thing in command line:
#one time
npm install -g http-server
#each time you want to launch a server in a folder, "-c-1" is "expire:no-cache"
http-server -c-1
#if you want another port, not 8080
http-server -c-1 -p8081


Use latest schema

Description of change
Store dependants answer
Link to relevant ticket
CRIMAPP-757
Notes for reviewer / how to test

Updated in #247


TEDs

i use text dect model as you supply,get PSENet detection results, TableMASTER structure recognition results and MASTER textline recognition results, and use match.py to get TEDS score.
PSENet + MASTER + TableMASTER_maxlength_500:
In the validation set, the accuracy is 0.56. also is not 0.9658
i  do valiation step by step as readme refer to, can't not recurrent the results: here is my cfg in  table_inference.py:
cfg = {
    'pse_config':'/configs/textdet/psenet/psenet_r50_fpnf_600e_pubtabnet.py',
    'master_config':'/configs/textrecog/master/master_lmdb_ResnetExtra_tableRec_dataset_dynamic_mmfp16.py',
    'structure_master_config':/configs/textrecog/master/table_master_lmdb_ResnetExtract_Ranger_0930.py',
    'pse_ckpt':'./pse_epoch_600.pth',
    'master_ckpt':'./master_epoch_6.pth',
    'structure_master_ckpt':'./epoch_16_0.7767.pth',
    'end2end_result_folder':'/mmocr_pubtabnet_recognition_0917val/work_dirs/end2end_val_result',
    'structure_master_result_folder':'/mmocr_pubtabnet_recognition_0917val/work_dirs/structure_val_result',

    'test_folder':'/pubtabnet/pubtabnet/val',
    # 'test_folder':'./smallVal10'
    'chunks_nums':chunk_nums
}

Visualize result of each step. Check if the text line detection model, the accuracy of the table structure recognition, the text line detection are all right.

i use text dect model as you supply,get PSENet detection results, TableMASTER structure recognition results and MASTER textline recognition results, and use match.py to get TEDS score. PSENet + MASTER + TableMASTER_maxlength_500: In the validation set, the accuracy is 0.56. also is not 0.9658
i do valiation step by step as readme refer to, can't not recurrent the results: here is my cfg in table_inference.py:
cfg = {
    'pse_config':'/configs/textdet/psenet/psenet_r50_fpnf_600e_pubtabnet.py',
    'master_config':'/configs/textrecog/master/master_lmdb_ResnetExtra_tableRec_dataset_dynamic_mmfp16.py',
    'structure_master_config':/configs/textrecog/master/table_master_lmdb_ResnetExtract_Ranger_0930.py',
    'pse_ckpt':'./pse_epoch_600.pth',
    'master_ckpt':'./master_epoch_6.pth',
    'structure_master_ckpt':'./epoch_16_0.7767.pth',
    'end2end_result_folder':'/mmocr_pubtabnet_recognition_0917val/work_dirs/end2end_val_result',
    'structure_master_result_folder':'/mmocr_pubtabnet_recognition_0917val/work_dirs/structure_val_result',

    'test_folder':'/pubtabnet/pubtabnet/val',
    # 'test_folder':'./smallVal10'
    'chunks_nums':chunk_nums
}

We have released the TEDS evaluation log, you can download the log file by this link, and check each val image's TEDS score.

@JiaquanYe  i find detect is right

@JiaquanYe  i find detect is right ,but text recon is wired

Maybe your alphabet is different with the text-line pretrained model. We have released the alphabet in “table_recognition/demo/alphabet”


Obtaining user ID from database and using that ID in the database reference for FirebaseRecyclerView

I understand this isn't particularly an issue, and I can't find anything online to help me with this.
I have a MainActivity class that signs the user in with Firebase and gets the current user there, and assigns it to a global static clientUser variable. I have a fragment in another class (that the MainActivity uses) that contains the FirebaseRecyclerView and uses the clientUser  object to get its firebase ID.
How would I add that user's firebase ID into a database reference so that my FirebaseRecyclerView loads up that particular user's groups from the database into the RecyclerView?
Basically, I get a null ptr exception at the initialization of mRef here because clientUser has not been created yet, because Firebase didnt log in yet. Any help/advice would be appreciated!
       DatabaseReference mRef = FirebaseDatabase.getInstance().getReference().child("users").child(clientUser.get_fb_uid()).child(groups);

        groupRecyclerViewAdapter = new FirebaseRecyclerAdapter<Group, GroupHolder>(Group.class, R.layout.grouptask_item, GroupHolder.class, mRef) {
            @Override
            protected void populateViewHolder(GroupHolder viewHolder, Group group, int position) {
                viewHolder.setCardGroupName(group.getGroupName());
                viewHolder.setGroupMemberCount(group.getMembers().size());
                viewHolder.setCardAuthorProfileImg(group.getAuthorProfilePictureUrl());
            }
        };

Never mind! After hours of experimenting I realized I can just add the following in my fragment class!
@Override
 public View onCreateView(LayoutInflater inflater, ViewGroup container,
                          Bundle savedInstanceState) {

     View view = inflater.inflate(R.layout.grouptask_item_list, container, false);

     itemTouchHelper = new ItemTouchHelper(simpleCallbackItemTouchHelper);

     Context context = view.getContext();
     recyclerView = (RecyclerView) view;
     recyclerView.setHasFixedSize(false);

     recyclerView.setLayoutManager(new LinearLayoutManager(context));

     final DatabaseReference mRef = FirebaseDatabase.getInstance().getReference().child("users");

     mAuth = FirebaseAuth.getInstance();

     mAuthListener = new FirebaseAuth.AuthStateListener() {
         @Override
         public void onAuthStateChanged(@NonNull FirebaseAuth firebaseAuth) {
             FirebaseUser user = firebaseAuth.getCurrentUser();
             if (user != null) {

                 Log.d(TAG, "User signed in... Collecting FireBase currentUser info");
                 // Currently signed in

                 groupRecyclerViewAdapter = new FirebaseRecyclerAdapter<Group, GroupHolder>(Group.class, R.layout.grouptask_item, GroupHolder.class, mRef.child(user.getUid()).child("groups")) {
                     @Override
                     protected void populateViewHolder(GroupHolder viewHolder, Group group, int position) {
                         viewHolder.setCardGroupName(group.getGroupName());
                         viewHolder.setGroupMemberCount(group.getMembers().size());
                         viewHolder.setCardAuthorProfileImg(group.getAuthorProfilePictureUrl());
                     }
                 };

                 recyclerView.setAdapter(groupRecyclerViewAdapter);

             } else {
                 Log.d(TAG, "User is currently signed out");
             }
         }
     };

     itemTouchHelper.attachToRecyclerView(recyclerView);

     return view;
 }

@Joroze You should probably wait for the user to sign in... Firebase Auth provides a nice callback for this:
@Override
public void onAuthStateChanged(@NonNull FirebaseAuth auth) {
    if (auth.getCurrentUser() == null) {
        // Not signed in
    } else {
        // Signed in, init your Firebase recyclerview here
    }
}

In your onCreate and onDestroy:
FirebaseAuth.getInstance().addAuthStateListener(this); // In onCreate

FirebaseAuth.getInstance().removeAuthStateListener(this); // In onDestroy

PS: You really shouldn't be storing the uid in a static variable. That will be full of bugs and weirdness when Android tries to restore your activities from a saved state. Use FirebaseAuth.getInstance().getCurrentUser().getUid() insead.

@SUPERCILEX Hi Alex!
That's what I've exactly figured out to do! Thanks for the advice on how I shouldn't use the static clientUser variable. Definitely makes sense!

@Joroze Awesome, glad you figured it out! 😄 Just FYI, the FirebaseUser class also had some other nifty methods like getEmail and 	 getDisplayName which you'll probably find useful as you go along.

@SUPERCILEX I see you know Firebase pretty well! I'm fairly new to Android development and Firebase, but I'm trying to impress my professor with these API's for my term project! For some silly reason I thought it would be a good idea to use getCurrentUser and create a new clientUser(getDisplayName, getEmail, etc.); at first.
By the way, maybe I can get some more of your opinions on these two ideas I'm doing:

Is it acceptable if I use getCurrentUser.getUID() to write and identify users on my database?
Should I use the OnAuthStateChanged listener for other activities that I'd need the getCurrentUser in?

Thanks again!!!

@Joroze I'm happy to help!
For your first idea, that's totally right. You'll want to identify users with getUid() because the uid will never change over the lifetime of the user. This means that if a user signs out and then back in again, you can retrieve all of their data using the uid from Firebase Auth (because it will always be the same for that specific user).
For your second idea, it depends. For example I have an activity that can only be opened by tapping a list item in my main activity. Since there will be no list items in the main activity if the user is not signed in, I can safely assume that there will always be a logged in user when the second activity is opened. However, if the user can have both the activity where they sign out and the one that requires them to be signed in open at the same time, you'll definitely want an onAuthStateChanged listener to know when to close the activity that requires a sign in. For example, this is a simplified version of my code inside the activity that requires a signed in user:
@Override
public void onAuthStateChanged(@NonNull FirebaseAuth auth) {
    if (auth.getCurrentUser() == null) finish(); // This activity requires a sign in so we need to close it if the user signs out
}

Hope that helps! 😄

@SUPERCILEX That does! Thank you so much for all the advice. You've saved me HOURS of trouble-shooting!

@Joroze You're welcome! 😄

@SUPERCILEX PS: I emailed you about another Firebase RecyclerView related question, (if you don't mind)!


Lab automation module update

CONTRIBUTING.md update
README.md update: Links to examples and Jupyter installation
Lab automation module update

Hi Peter, I have merged the new changes from you. It seems like the failed checks were from the genbabel module. I will debug that

Included in PR #25


Copy To physical plan

Description
This one likely to be tricky.

We'll do this one after #1599

here's a checklist of all of the possible paths for copy to.

[x] copy local to local - https://github.com/GlareDB/glaredb/pull/1633
[x] copy local to external (s3,gcs,...) - https://github.com/GlareDB/glaredb/pull/1633
[ ] copy local to remote
[ ] copy remote to local
[ ] copy remote to remote
[ ] copy remote to external
[ ] copy external to external

To keep the conditional logic simpler inside the exec, what are your thoughts @scsmithr  @vrongmeal on having individual execs for each of these paths? CopyLocalToLocalExec, CopyLocalToRemoteExec, ....
I think it'd make the flow a bit easier to follow.

except for copy remote to local, all other cases should work 🤔

As far as copy execs are concerned, we basically need something to wrap our final physical plan inside something custom (conditionally). Not sure if having different execs helps there, but the part we need to figure out is wrapping the SendRecvJoinExec inside a CopyToLocalExec.


Include Merkle receipt and other validation to schema

In the first release, we created a transaction for each certificate (hash). We stored the transaction id along with each certificate for validation.
The next release will batch the certificate (hashes) into a Merkle tree, so we additionally need to provide the Merkle path to prove the existence of a certificate in a batch of certificates.

Addressed by PR #11


Updated sponsors for fall 2017

Title says it all!

So I understand the reducing file size part, but not the size. Am I supposed to make a version of the logo for mobile vs. bigger screen? Looking at last year's logos it seemed like they were mostly around 300 pixels long

David wanted to merge now cause sponsors were asking when it would be up. I agree we should go back and optimize later if we have time though.


Replace guava's Optional with java.util's one

This is great, thank you!  Any feedback from @dropwizard/committers about this?  This may break some things, but probably worthwhile to get into 1.0.0-rc2.

Coverage increased (+0.02%) to 80.877% when pulling becdf367a17265d3a99502f5f55a54b74cd8779f on patrox:j_u_optional into 1cb97790a18f29e1b476e4fe8dec0184c37fd7e7 on dropwizard:master.

Coverage increased (+0.02%) to 80.877% when pulling becdf367a17265d3a99502f5f55a54b74cd8779f on patrox:j_u_optional into 1cb97790a18f29e1b476e4fe8dec0184c37fd7e7 on dropwizard:master.

LGTM except for that minor comment on HibernateBundle. :+1:

Coverage increased (+0.02%) to 80.877% when pulling 7eb1da41abe9d4a23c5ca819a180af9c144aeed6 on patrox:j_u_optional into 1cb97790a18f29e1b476e4fe8dec0184c37fd7e7 on dropwizard:master.

Thanks!


Refining filter mode

General information

Operating system + version: Arch Linux
Browser + version: Chrome 70
Information about the host app:

How did you install it? AUR
If installed an official release, put a version: 2.0.21

Information about the browser extension:

How did you install it?

Installed via webstore

Browserpass extension version as reported by your browser: 2.0.22

Exact steps to reproduce the problem

Name your password "kde bugs.gpg"
Go to https://bugs.kde.org
No matching passwords found for bugs.kde.org.

What should happen?
Unless the file name explicitly say bugs.kde.org the extension wouldn't recognize it outside of search mode.
Filter mode should be quick and easy but it requires to name all the password as the exact domain they're for which sound to me a little bit too much.
I think it should at least detect "KDE Bugtracker", "KDE", "KDE Bugs" etc...
If it wasn't clear, I was using KDE Bugtracker as an example. This happens with all websites.

Hi there, this is by design, one of primary focuses for browserpass is to protect you against phishing attacks. That's why, the entries that you see in filter mode are only those that explicitly match the current domain.
If a malicious person makes you open "https://bugs.kde.com", you don't want browserpass to show you the entry "kde bugs".
If you create an entry "kde.org.gpg", it will show up on "kde.org" and "bugs.kde.org", if you create an entry "bugs.kde.org.gpg" it will show up on "bugs.kde.org" but not "kde.org".
If you press Backspace and switch back to the search mode, in the eyes of browserpass you officially wave your protection against phishing attacks and you are on your own, browserpass will happily show you all possible entries you have in your password store.

Just a heads up, the next major version of browserpass will remember which passwords you used on which domains and show them next time in the popup.
So if you have a password entry called KDE Bugs.gpg and you navigate to bugs.kde.org, at first just like today browserpass will not show KDE Bugs among the list of passwords (to protect you from phishing attacks), however if you specifically search for KDE Bugs and use it to login, next time you open bugs.kde.org the entry KDE Bugs will be present in the popup.


Removed skip failing E2E perf test on to run on pipeline

Test case: "can sucesfully copy document from a specific version" is failing
Example failing run: https://dev.azure.com/fluidframework/internal/_build/results?buildId=146762&view=logs&j=0ab14b9f-e499-56d5-97b1-fd98b70ea339&t=b3b126df-85da-521c-c1c9-bb5b4719d797&l=43
Temporarily skipped the test case to unblock pipeline: Skip failing E2E perf test by ms-avocado · Pull Request #14940 · microsoft/FluidFramework (github.com)
AB#3934

We tested this on local and in CI, the broken test is working in both environments without changing the test


Model cards for Keras models

Keras models (non-Transformers ones) don't have generated model cards.  I'd like to generate model cards for them that include:

model hyperparameters
metrics from training
name of dataset and splits (can be filled by user later on as well)
versions of libraries (like TF model cards)

I think it would be super cool to actually have KerasTuner outputs in the model cards as well (output of results_summary())

I think it would be a big flex if we would write model.summary() output. This doesn't work for models with subclassing AFAIK, we can get it for Functional and Sequential models. This tends to be a large section but we can put it under toggle (pointed out by @osanseviero in spaCy models)
The rest of the parameters I will get from model.
We can later introduce history object (created with history = model.fit() that contains model training history even if the user doesn't log loss, accuracy etc, we could introduce this as another parameter in push_to_hub_keras() since this is something we can't obtain from the model but we need user to create history while fitting the model.
I want to cc @Rocketknight1 since he's experienced with the TF model card.

@Rocketknight1 it currently looks very shifted when I save it with IO so I'm plotting the model instead :(


Cleanup

Big monster cleanup. No impact on functionality or UI, all Typescript and config cleanup.

tested briefly just in case, but changes shouldn't have any impacts

tested briefly just in case, but changes shouldn't have any impacts


Firefox: calling resolve() don't resolve then()

I'm using BlueBird version 3.3.3.
Platform: Ubuntu with Firefox 45 / Chromium 48
When I call resolve(), the promise listener registered with .then() is never called on Firefox, but it is on Chromium.
Relevant code :
// Used to suppress the click event when the promise is resolved().
const makeSilent = (resolve) => {
    return () => resolve(); // This is called, after that I don't know what happens
};

Dialog = class Dialog {
    static confirm(opts) {
        return new Promise((resolve) => {
            let $confirm = findDialog('confirm');
            refreshDialog($confirm, opts);

            // We call resolve when the 'OK' button is clicked.
            getOk($confirm).click(makeSilent(resolve));

            showDialog($confirm);
        });
    }
};

// The promise consumer :
Dialog.confirm(options).then(() => thisFunctionIsNeverCalledOnFirefox());

By tracing the code with some console.logs, I see that the resolve in makeSilent is called, but after that, nothing more happens. No errors in the console.
I also tried to remove makeSilent by replacing .click(makeSilent(resolve)) with .click(resolve), but it's doesn't work too.

I just wanted to update you that although no one commented yet we're debating how to fix this.
You can work around it by calling setScheduler. The problem is that the low latency scheduler (that uses MutationObservers under the hood) probably doesn't work in that particular case.

Thank you, it works. I'm using Promise.setScheduler(task => task()); now. Is Bluebird exposing better asynchronous schedulers somewhere? Or is a synchronous scheduler, like mine, acceptable ?

@toverux a synchronus scheduler is problematic since it can break some invariants about promises like the order of things. You can use setTimeout as a scheduler or even NativePromise.prototype.resolve.

Ok, thank you for your help :)

Hello, I'm affected by this issue as well. I've adopted Bluebird because it was cross browser and I'd hate to have this be an issue. I was able to workaround by the above or using setTimeout, which I am not in favor of. I would like to see a fix for this in the library, so my code is not having to use workarounds everywhere. I understand bluebird is fast, but fast means nothing if not reliable. Workarounds throughout my codebase means i might as well use alternative libraries.

I cannot reproduce on firefox 45

I just want to add I'm also seeing a weird bug that just started with iOS 9.3 mobile iPhone that may be related.
As soon as i updated code that was working find before stopped working.  Tracing it down it seems like there is some race condition or something causing promises to not return.
Here's my basic code:
console.log("before");
return Promise.resolve().then(function() {
console.log("after");
});

I can get the "before" to consistently log, but sometimes the "after" doesn't.  And as soon as it fails, there is no recovery.
In my specific situation, i'm actually using HammerJS with a swipe and taps in a Cordova App that triggers this race condition.  I don't know if that related but I thought it might help.
Again, as soon as I updated to iOS9.3 it just basically stopped working correctly.

Just a followup,
I changed my code to new Promise(function(resolve, reject) { resolve() }).then(...) but was still having issues with the promise not resolving.
As suggested above I'm not wrapping all resolve()'s in timeouts like
new Promise(function(resolve, reject) {
setTimeout(function() { resolve(); }, 0);
}).then(...)
This seems to fix the issue, but now i have wrap all the all the resolve / rejects to make sure this race condition does't happen.

Will try that asap. Thank you!


Deploy hooks, with Sentry deploy hook.

These hooks are run immediately around the critical section of the
upgrade.  If the upgrade fails for preparatory reasons, the pre-deploy
hook may not be run; if it fails during the upgrade, the post-deploy
hook will not be run.  Hooks are called from the CWD of the new
deploy, with arguments of the old version and the new version.  If
they exit with non-0 exit code, the deploy aborts.
Fixes: #23620
(in passing)

Tested using the Sentry hook.  This needs documentation, both for the hook system in general, as well as the Sentry hook specifically.  I'm not entirely convinced at the old-version/new-version arguments -- but, well, we have them, so why not.
We may also want another hook earlier in the process, but around the critical portion seems the most useful and clearest boundary to have one.
Screenshots and screen captures:
2023-01-30 21:16:13,680 upgrade-zulip-from-git: Fetching the latest commits
Updating files: 100% (5909/5909), done.
2023-01-30 21:16:16,609 upgrade-zulip-stage-2: Caching Zulip Git version...
2023-01-30 21:16:16,804 upgrade-zulip-stage-2: Upgrading from 6.0-548-g8f81c8063e to 6.0-552-g9887b9a3b1, in /home/zulip/deployments/2023-01-30-21-16-13
2023-01-30 21:16:16,825 upgrade-zulip-stage-2: Upgrading system packages...
[...]
2023-01-30 21:18:41,891 upgrade-zulip-stage-2: Pre-checking for puppet changes...
sentry: Creating release<EMAIL_ADDRESS>Created release<EMAIL_ADDRESS>sentry: Setting commit range based on merge-base to upstream of e2e39acbc441a1b5429b691592ad09c2d747a8d2
+-------------+--------------+
| Repository  | Revision     |
+-------------+--------------+
| zulip/zulip | e2e39acbc441 |
+-------------+--------------+
2023-01-30 21:18:52,777 upgrade-zulip-stage-2: Stopping Zulip...
[...]
2023-01-30 21:19:41,799 restart-server: Done!
Zulip started successfully!
2023-01-30 21:19:41,825 upgrade-zulip-stage-2: Upgrade complete!
sentry: Adding deploy of 'production' and finalizing release
Created new deploy unnamed for 'production'
Finalized release<EMAIL_ADDRESS>2023-01-30 21:19:42,926 upgrade-zulip-stage-2: Purging old deployments...
Deployments cleaned successfully...
Cleaning orphaned/unused caches...
Cleaning unused venv caches...
Cleaning unused node modules caches...
Cleaning unused emoji caches...
Done!

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

Added documentation.

Adding to my review queue for Monday AM, thanks for adding that!

This lgtm aside from https://github.com/zulip/zulip/pull/24225#discussion_r1099458959 and the CI failure.

Looks great, queued to merge. I posted a few comments on tweaks to keep on our minds as we try this out. Thanks @alexmv!

The merge queue isn't actually running yet, merging directly.


Snow removal deletes non-snow blocks

The improved snow removal replaces blocks that have been placed on snow during winter.
Steps to reproduce:

Set season to winter
Set weather to snow and wait for some snow to build up
Place something on a snow tile (and thus replace the tile with a new block)
Go to sleep

The now placed block gets replaced with air
Happens on the latest HF version (0.5.10) using Forge <IP_ADDRESS>4

Woops and fixed. Guess this is an important enough bug to release .11 \o/

Thanks for that very fast fix. My friend thought he was going crazy because all his stuff kept disappearing xD


Refactor app.py for Improved Configuration and Security

This pull request refactors the app.py file to enhance the configuration and security of the Flask application. The following changes have been made:

Updated database URI to use environment variables for better security and flexibility.

Set the secret key to be retrieved from environment variables, providing a safer default mechanism.

Organized the import statements for clarity.

Ensured adherence to best practices in Flask application structure.

These modifications aim to make the application more maintainable and secure for production environments.

You havent raised an issue for this

So you are contributing via GSSOC or on your own

You havent raised an issue for this

I'm a bit new to this, cant i direct make the change and raise a pull request instead of raising an issue followed by a pull request.

So you are contributing via GSSOC or on your own

i'm contributing via GSSoC

You havent raised an issue for this

I'm a bit new to this, cant i directly make the change and raise a pull request instead of raising an issue followed by a pull request.

No first you have to get an issue assigned then start working and raise pr. Now you raise the issue for this and I will assign it and give proper labels as well. After that I will merge this pr

You havent raised an issue for this

I'm a bit new to this, cant i directly make the change and raise a pull request instead of raising an issue followed by a pull request.

No first you have to get an issue assigned then start working and raise pr. Now you raise the issue for this and I will assign it and give proper labels as well. After that I will merge this pr

I have submitted the issue, kindly please check.

Closes #15


Fix APK signature

@MathewSachin Please fix the keystore passwords. https://github.com/MathewSachin/Fate-Grand-Automata/pull/25/checks?check_run_id=598764803#step:5:570
Before, it was not using your keystore but instead used a newly generated one because of a typo.

@MathewSachin So because of a typo, the pipeline always used a new keystore for signing.
Additionally, there was a communication error between us. There are up to 3 passwords in this pipeline regarding the keystore:

Keystore password

password for opening the decrypted keystore
currently $KEY_PASSWORD

Key password

password for using the key inside the keystore
currently $KEY_PASSWORD

GPG decryption password

password for decrypting the keystore
currently $KEYSTORE_PASSWORD

I got the pipeline to work again, but I'm not totally happy with the current variable names.

I'm sorry, I messed up really bad here.
Thank you for fixing.
I'll change the secret names. Let me know what you want them to be.

The names before my review in #19 were better. I just didn't understand the pipeline script well enough.
I would ultimately change them to GPG_PASS (or GPG_KEYSTORE_PASS) and KEYSTORE_PASS

I'm changing them back to KEYSTORE_PASS and GPG_KEYSTORE then.
GPG_KEYSTORE sounds better to me = GPG for KEYSTORE and the convention can be followed if we include any other GPG encrypted files.