Pinot query ArrayIndexOutOfBounds for some multi-value column queries

When we run the below query on our cluster via Pinot UI, we get "errorCode": 200,     "message": "QueryExecutionError:\njava.lang.ArrayIndexOutOfBoundsException"
The query is simple as
select account_key from point_entry where account_key = 'xxxxx' and account_fields_values not in ('intercompany_clearing') limit 1 option(timeoutMs=10000)
Both of the columns in the query are multi-value columns.
The Server error log is below:
Caught exception while executing operator of index: 4 (query: QueryContext{_tableName='point_entry_OFFLINE', _selectExpressions=[account_key], _aliasList=[null], _filter=(account_key = 'xxxx' AND account_fields_values NOT IN ('intercompany_clearing','cash','merchant_balance','fx_clearing') AND created_at <= '1634813999'), _groupByExpressions=null, _havingFilter=null, _orderByExpressions=null, _limit=100, _offset=0, _queryOptions={responseFormat=sql, groupByMode=sql, timeoutMs=9863}, _debugOptions=null, _brokerRequest=BrokerRequest(querySource:QuerySource(tableName:point_entry_OFFLINE), pinotQuery:PinotQuery(dataSource:DataSource(tableName:point_entry_OFFLINE), selectList:[Expression(type:IDENTIFIER, identifier:Identifier(name:account_key))], filterExpression:Expression(type:FUNCTION, functionCall:Function(operator:AND, operands:[Expression(type:FUNCTION, functionCall:Function(operator:EQUALS, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:account_key)), Expression(type:LITERAL, literal:<Literal stringValue:xxxxx>)])), Expression(type:FUNCTION, functionCall:Function(operator:NOT_IN, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:account_fields_values)), Expression(type:LITERAL, literal:<Literal stringValue:intercompany_clearing>), Expression(type:LITERAL, literal:<Literal stringValue:cash>), Expression(type:LITERAL, literal:<Literal stringValue:merchant_balance>), Expression(type:LITERAL, literal:<Literal stringValue:fx_clearing>)])), Expression(type:FUNCTION, functionCall:Function(operator:LESS_THAN_OR_EQUAL, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:created_at)), Expression(type:LITERAL, literal:<Literal stringValue:1634813999>)]))])), orderByList:[], limit:100, queryOptions:{responseFormat=sql, groupByMode=sql, timeoutMs=9863}))})

@dongxiaoman can this issue be closed ? Can you update the ticket with the resolution steps as well

I think this ticket can be closed.
The root cause was that one of the very old segment wasn't properly conformed with the new indexed segments. thus the query always fails when hitting that specific segment. (@dongxiaoman plz correct me if I remembered wrong).

Forgot to close it. Happens only to one segment that is probably generated by some version of library that may have issues.


primitives get methods for SquidCursor to prevent autoboxing overhead

Hi!
Currently, there's only one SquidCursor#get() implementation, which implies autoboxing for every property you get, which implies an overhead as this video explains. Could you add primitives version of these methods please?
Thanks, and always remember, perfs matters! :)

Hi @LouisCAD, thanks for opening the issue! We are already aware of the performance implications of autoboxing in SquidCursor, and have made this tradeoff by design. This is partially due to the implementation details of model objects. Model values are stored in ContentValues, to be passed to Android framework methods like SQLiteDatabase#update. ContentValues are just maps under the hood, so primitive values are going to be boxed sooner or later anyways. Primitive values also can't be null, so we designed the API to avoid forcing the user to make a call to cursor.isNull for every field accessed. In our benchmarks and testing, the autoboxing has not been much of an issue in most real-world usage.
That being said, SquidCursor is an instance of CursorWrapper, so all the standard android Cursor methods that work with primitives are still available to you. These are good alternatives if you're not working with model objects and want to squeeze every bit of performance out of the cursor itself. Are you just asking for type safe versions of these methods that work with Property objects rather than column indexes? We can consider that as an API enhancement; it should be very straightforward -- although the caveat about boxing inside model objects would still apply.

Hi @sbosley, first, thanks for replying!
I just saw that ContentValues implies autoboxing as it only returns Object versions of primitives, even in getAsInteger(), so Google doesn't listen to Colt McAnlis advices… I think that it doesn't worth it to add primitives methods with this under the hood now, it would add more not that useful code.
Thank you, and have a nice day!

Ok, thanks for the feedback @LouisCAD! It's always possible that someday down the road we'd change the underlying implementation of model objects, so if we ever do we'll definitely revisit this then. Thanks again for opening the discussion!


When uploading multiple images the image license selector implies singular

Summary:
When uploading multiple images and selecting a license you get a message like:
"This image will be licensed under "
but it should be
"These images will be licensed under "
Screen-shots:

Hello, I've started working on this.

@ravi-kishan @domdomegg
Hi, I am also working on this. I use the plurals string way (https://developer.android.com/guide/topics/resources/string-resource#Plurals). I think that'd be good for i18n.
Will send a PR soon.

Hi @zhao-gang and @ravi-kishan , if one of you is just starting and the other has some progress, I suggest the one who just started can pick another issue, two prevent both of you work on same task.
Did you both already made some progress?

@neslihanturan I have made it working by using plurals string. But need some clean up to submit a PR.

@ravi-kishan might be working for two hours also, it is always better to comment before you start. Lets wait for @ravi-kishan 's answer. If he is also close to finalize PR, I would let him send PR and pick another task. But if he is just started, then he can let you to send PR. We need some communication here:) Thanks both of you for your previous and current contributions and help though:)

As @neslihanturan has pointed out that he has a better way of implementing it, I think he should also send the PR and the better one will be merged.

@zhao-gang , maybe it can be better to adding your reviews under already ready PR of @ravi-kishan , instead of sending another PR . By this way, he can either improve his solution or leave it as is. Since we are all cooperating here. Having two different PR for same task puts reviewer in a hard situation though.

Well, so if you are okay with this @ravi-kishan , waiting for your PR @zhao-gang . Thanks both:)

@ravi-kishan @domdomegg @neslihanturan Hi all, I have submitted my PR. I was not familiar with this project's development process so I didn't comment on this issue before working on it. Sorry for the inconvenience.


Refinement on arrow type crashes the typechecker

The following code
module Bug

type mapping = int -> Tot int

let f : mapping = fun x -> x // Works

let g:h:mapping{forall x. h x = x} = fun x -> x // Crashes F*

crashes F* on the g declaration with
Error: Unexpected error; please file a bug report, ideally with a minimized version of the source program that triggered the error.
Failure("Impossible! let-bound lambda Bug.g = (fun x -> x@0) has a type that's not a function: Bug.mapping\n")

Note that declaring a type abbreviation works:
type mapping = int -> Tot int
let f : mapping = fun x -> x
type identity = iden:mapping{forall x. iden x = x}
let h : identity = fun x -> x

typechecks.

Seems like a limitation of the SMT encoding:
https://github.com/FStarLang/FStar/blob/385e9e809eaf35f316491ddd02f532acf03d9aab/src/smtencoding/encode.fs#L1395

Also, could reproduce it even without the funny let syntax:
val g : (h:mapping{forall x. h x = x})
let g x = x

causes exactly the same error
Error: Unexpected error; please file a bug report, ideally with a minimized version of the source program that triggered the error.
Failure("Impossible! let-bound lambda Bug552.g = (fun x -> x@0) has a type that's not a function: Bug552.mapping\n")


docker can't connect to amqp

os: macOS high sierra
postgres server and rabbitmq server run on localhost, pg-amqp-bridge run in docker
git checkout 0.0.6
POSTGRESQL_URI="postgres://postgres:postgres@<IP_ADDRESS>:5432"  \
<EMAIL_ADDRESS>\ BRIDGE_CHANNELS="pgchannel1:task_queue,pgchannel2:direct_exchange,pgchannel3:topic_exchange" \
cargo run

works.
but
  pg_amqp_bridge:
    image: subzerocloud/pg-amqp-bridge:0.0.6
    environment:
      - POSTGRESQL_URI=postgres://postgres:postgres@<IP_ADDRESS>:5432
      -<EMAIL_ADDRESS>      - BRIDGE_CHANNELS=pgchannel1:task_queue,pgchannel2:direct_exchange,pgchannel3:topic_exchange
    restart: always

but docker-compose up got errors:
Attempting to obtain connection on AMQP server for pgchannel2 channel..
IoError(ConnectionRefused)
Retrying the AMQP connection for pgchannel2 channel in 1 seconds..
Attempting to obtain connection on AMQP server for pgchannel3 channel..
IoError(ConnectionRefused)
Retrying the AMQP connection for pgchannel3 channel in 1 seconds..
IoError(ConnectionRefused)
Retrying the AMQP connection for pgchannel1 channel in 2 seconds..
IoError(ConnectionRefused)

i think this is more of a docker question.
your bridge container does not seem to be able to access your host ip/port
try adding the --network host parameter/property
see if this
https://dev.to/bufferings/access-host-from-a-docker-container-4099
and this
https://docs.docker.com/network/network-tutorial-host/#prerequisites
point you in the right directon


Plugin not getting installed on new nodes

When adding new nodes to an existing cluster, I'm getting  failed to find plugin "ovs" in path [/opt/cni/bin /opt/cni/bin] when trying to use the plugin for a pod running on the new node.
When looking into the indicated folder, I see that the old nodes do have the ovs file on the folder, but the new nodes do not have it.
I'm not sure if this is responsibility of the plugin, but letting you know anyway.
If I find a workaround I let you know.
Thanks.

Hello @axelwass. Deployment of OVS CNI across cluster is not handled by this project. I'd say you should figure out how was OVS CNI deployed originally and then adjust that method to cover newly added nodes too.

ok, thanks @phoracek. I will close the issue them.

Leaving this just in case someone else faces the same issue. The problem was with https://github.com/kubevirt/cluster-network-addons-operator, that it was failing to start because the image of the version I was using ceased to exist.


Fixes build statuses, corrects image pointers, and ...

... removes outdated sponsorship info.

fixes #4899


4.2.0 packages

./tools/new_release.sh 4.2.0 ../osquery ../osquery-site

TODO: Sign and upload the MSI, update the hash.

Heads up @muffins

I'll get the packages built and shipped upstream tonight/tomorrow when I have access to my keys


How to apply this to a pure relationship resource?

For example I have a table stores user's "liked" items. It's a m-to-n relationship. Then I need to expose APIs so client side can get user's likes, as well as adding, deleting, checking, etc.

How to design the path? should it be like /likes/:uid/items/:item_id, or just /likes/:item_id and get uid from session for instance.
The table doesn't have an ID field as it's only simple relationship - 2 fields: uid and item_id. But ID seems to be important in JsonAPI, is it?

We don't make requirements on URL structure, but we do have reccomendations now: http://jsonapi.org/recommendations/#urls
We only require strings for IDs, so UUIDs are just fine.


Unify HttpConnectionOptions and HttpOptions

I'm not sure why we have 2 very similar types on the client side by regardless, we need to unify them. We might also want to consider adding Client to the name HttpClientOptions since it'll conflict with the server if you use them in the same app.
/cc @JamesNK do you know why we have both?

I'll remove HttpConnectionOptions, rename HttpOptions to HttpClientOptions, and have the factory use it.


A MERGE Statement Must be Terminated by a Semi-colon" Error with MSSQLSERVER

Search first

[X] I searched and no similar issues were found

Description
I'm having trouble using MSSQLSERVER (Linux container). I created a change set that includes a MERGE INTO statement, but when I try to run it, Liquibase returns an error message: "A MERGE statement must be terminated by a semi-colon (;)."
I have tried terminating my change set with a semi-colon both in line and on a new line. I am using a SQL-based change set.
Steps To Reproduce

You can use the following docker-compose file to setup MSSQLSERVER and a liquibase container

docker run -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Temp123@' -p 1433:1433 --name sql1 -h sql1 mcr.microsoft.com/mssql/server:2022-latest

Create a test database

Run liquibase with the following changeset

--liquibase formatted sql

--changeset dtorres:0
CREATE TABLE Gender (
   Id INT PRIMARY KEY,
   Gender TEXT NOT NULL
);

--changeset dtorres:1 runOnChange:true
MERGE INTO Gender AS t
USING (
   VALUES
      (1, 'Male'),
      (2, 'Female'),
      (3, 'Non-Binary'),
      (4, 'Unknown')
) AS s (id, Gender)
ON t.id = s.id
WHEN MATCHED THEN
   UPDATE SET t.Gender = s.Gender
WHEN NOT MATCHED THEN
   INSERT (id, Gender) VALUES (s.id, s.Gender);

You will see the following error:
Unexpected error running Liquibase: Migration failed for changeset changelog/changelog.sql::1::dtorres:
liquibase2_1  |      Reason: liquibase.exception.DatabaseException: A MERGE statement must be terminated by a semi-colon (;). [Failed SQL: (10713) MERGE INTO Gender AS t
liquibase2_1  | USING (
liquibase2_1  |    VALUES
liquibase2_1  |       (1, 'Male'),
liquibase2_1  |       (2, 'Female'),
liquibase2_1  |       (3, 'Non-Binary'),
liquibase2_1  |       (4, 'Unknown')
liquibase2_1  | ) AS s (id, Gender)
liquibase2_1  | ON t.id = s.id
liquibase2_1  | WHEN MATCHED THEN
liquibase2_1  |    UPDATE SET t.Gender = s.Gender
liquibase2_1  | WHEN NOT MATCHED THEN
liquibase2_1  |    INSERT (id, Gender) VALUES (s.id, s.Gender)]
liquibase2_1  |
liquibase2_1  | For more information, please use the --log-level flag

Expected/Desired Behavior
changeset should be applied successfully.
Liquibase Version
4.20.0
Database Vendor & Version
Microsoft SQL Server 2022 (RTM-CU1) (KB5022375) - 16.0.4003.1 (X64)  (LINUX CONTAINER)
Liquibase Integration
CLI (Docker container)
Liquibase Extensions
No response
OS and/or Infrastructure Type/Provider
Docker containers
Additional Context
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR (Thank you!)

Note, I did find a workaround (actually I got it from your discord)
Set splitStatements to be false in your change set. Then you just need to ensure that your merge statement is terminated with a semi-colon.

Confirmed to be an issue for liquibase 4.20. Thanks for the submission. If you can submit a pr with a PR to fix this, it would  help speeding up the process. Thanks!

Hello @dtorres-srga ! I managed to reproduce your error using the example on Liquibase 4.20.0 and 4.21.1 . But it has been fixed on version 4.23.0:
Starting Liquibase at 14:50:33 (version 4.23.0 #10709 built at 2023-06-23 21:18+0000)
Liquibase Version: 4.23.0
Liquibase Open Source 4.23.0 by Liquibase

Running Changeset: 3941/changelog.sql::0::dtorres
Running Changeset: 3941/changelog.sql::1::dtorres

UPDATE SUMMARY
Run:                          2
Previously run:               0
Filtered out:                 0
-------------------------------
Total change sets:            2

Liquibase: Update has been successful.
Liquibase command 'update' was executed successfully.

Seems it was fixed here -> https://github.com/liquibase/liquibase/pull/4202 .
Could you confirm that it works for you?


more correctly distinct sound(DB, Frequency)

Using Decibel, Frequency to distinct sound

more correctly distinct sound(DB, Frequency)


Only process levels needed when user passes query points

Current processing:

if a user requests raytracing at some query points (e.g. lat/lon files or station file) then all height levels in the weather model get processed.

Goal:

Only process the height levels needed for 3D interpolation.

Adding optimization label, as the code currently works as is.


osu-lazer: update description to mention osu-lazer-bin

Description of changes
This patch was split off from #208236.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

oops forgot to actually mention osu-lazer-bin, done in b39e27694be5d0188f933d4f75495b673ffb4374

squashed in c60a8f3fbae08f80a4daac5a7088513406a85061

Well, you didn't need to do this in a separate PR 😅. But this is fine too.

Well, you didn't need to do this in a separate PR

Ohh, good to know! Next time ^.^;;


App size increasing after getting pictures/videos

App size increasing after getting pictures/videos.
Every time I use YPImagePicker to get videos/images, it increases my app size. It keeps on increasing and increasing. I have to delete the app and reinstall to get it back to normal size. Is there any way to clear the cache?

Hi @farhan101,
Could you elaborate a bit more on how you measure the App size increase so that we can try to reproduce on our end?
Also, are you experiencing the same using the provided Example project ?

Hi @farhan101,
Could you elaborate a bit more on how you measure the App size increase so that we can try to reproduce on our end?
Also, are you experiencing the same using the provided Example project ?

Upload some pictures/videos to accumulate the size. Then go to iPhone's  Settings -> General -> iPhone Storage -> Select your app -> check app and documents & data size. This keeps increasing after each uploads.
PS: I printed the app's storage contents and I can see images and videos lying in "temp" folder. I guess they are not being removed from there OR I am missing a some configuration???
I shall also try this with your sample projects as well and update you with the results.

 How can this be resolved? Thanks

There is 2022 year outside🎉 Old outdated issues are closing. Please search if there are already existing issues with the same problem before opening new ☮

anyone fixed it anyhow ?


Use "index_name" for Weaviate in README

Fixes #252

Thanks to @therealrodk for reporting #252 :)


Update aliwangwang to 7.06.12-8579

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} left no offenses.
[x] The commit message includes the cask’s name and version.

Add appcast stanza this time.


sub-collaborations are not being pinned

The root collaboration is being pinned, but sub-collaborations are not.
To reproduce it, try dev.peerpad.net on 2 different browser on the same document, one at a time.

Fixed by 1ec3df41827918c1f3f718590857f2aedbd0c17b

Landed on v0.10.4

Hopefully fixed in #185


[SPARK-41051][CORE] Optimize ProcfsMetrics file acquisition

What changes were proposed in this pull request?
Reuse variables from declared procfs files instead of duplicate code
Why are the changes needed?
The cost of looking up the config is often insignificant, but there
reduce some duplicate code
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing unit tests.
Closes #41051 from sus/SPARK-41051.

What if your id @Narcasserun  ? To assign jira to you.

Merged to master, thanks for fixing this @Narcasserun !
And welcome to Apache Spark community :-)


Tool freezes after constructing identity 4-cell

From a framed invertible 1-cell, I constructed a "swallowtail" 3-cell containing two "cusp" 3-cells. I then pressed "Identity" to make an identity 4-cell. Finally, I pressed the left arrow key (or clicked the star) and the tool froze. The workspace is attached.
swallowtail.zip
.

I'm still getting the same problem. Is this just because I'm not using the fixed version?

Just tried it again and it works fine!


]OpenProject prints superfluous stuff to ⎕SE when it shouldn't

]Cider.OpenProject /path/2/project -import -batch
should not print anything at all but does:
Would you like to change the current directory to ../Test-Data/Fire ? (y/N)
Project successfully imported and established in "..."

Fixed in 0.40.0


Copy task needs to support incrementality

Because it lacks @outputs or @outputDirectories, J2objcCopy task will always be executed.

will fix in #114 as well

I fixed this a while ago but this is a better fix.
On Sun, Jun 7, 2015 at 4:42 PM Advay Mengle<EMAIL_ADDRESS>wrote:

will fix in #114
https://github.com/j2objc-contrib/j2objc-gradle/pull/114 as well
—
Reply to this email directly or view it on GitHub
https://github.com/j2objc-contrib/j2objc-gradle/issues/115#issuecomment-109814267
.

I don't see how you fixed it?  No @Output is marked in CopyTask, therefore the task will always be xecuted.

I had something working before... maybe it was just a local fix that never
got pushed. Either wya, this is a much better way of doing it.
On Sun, Jun 7, 2015 at 5:25 PM Advay Mengle<EMAIL_ADDRESS>wrote:

I don't see how you fixed it? No @Output https://github.com/Output is
marked in CopyTask, therefore the task will always be xecuted.
—
Reply to this email directly or view it on GitHub
https://github.com/j2objc-contrib/j2objc-gradle/issues/115#issuecomment-109817266
.


Handling null exceptions

In my application, I got two cases of null exceptions with the latest release of the library from nuget. This PR fixes that. Note: I did not look into the details of the causes of the exceptions; for my purposes, the lib works fine with these changes.

Thanks!

https://www.nuget.org/packages/Gu.Wpf.DataGrid2D/<IP_ADDRESS>
Published an update.


HHH-13611 Restore EntityMetamodel constructor to take SessionFactoryImplementor argument instead of PersisterCreationContext.

https://hibernate.atlassian.net/browse/HHH-13611
PR for 5.3 branch

Merged. Thanks @dreab8!


feat: Ability to create snapshots of results & scores for comparison

Feature

Ability to create snapshots of results & scores for comparison

Working on this.


[Bug] Failing to export HTML: Cannot convert undefined or null to object

Reporting a bug?
I'm getting several of these types of errors after running this command react-static build --staging --debug:
...
  - asyncToGenerator.js:TypeError: Failed exporting HTML for URL ylc_website/about (__react_static_root__/src/About): C  annot convert undefined or null to object

  - Function.keys

  - Routes.js:72
    [static-ylc]/[react-static]/src/browser/components/Routes.js:72:14

  - react-dom-server.node.development.js:1194 useReducer
    [static-ylc]/[react-dom]/cjs/react-dom-server.node.development.js:1194:57

  - react-dom-server.node.development.js:1132 Object.useState
    [static-ylc]/[react-dom]/cjs/react-dom-server.node.development.js:1132:10
...

I've gone through many iterations of trying to get my project to build this week, but I'm really not sure what to do. Amongst other things, I thought it was a react-router issue so I refactored to use @reach-router which didn't seem to help.
You can see the repo here: https://github.com/Yukon-Literacy-Coalition/ylc_website
Environment
  System:
    OS: macOS Mojave 10.14.6
    CPU: (4) x64 Intel(R) Core(TM) i5-4258U CPU @ 2.40GHz
    Memory: 1.69 GB / 8.00 GB
    Shell: 5.0.11 - /usr/local/bin/bash
  Binaries:
    Node: 10.15.1 - ~/.nvm/versions/node/v10.15.1/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.5 - ~/.nvm/versions/node/v10.15.1/bin/npm
    Watchman: 4.9.0 - /usr/local/bin/watchman
  Browsers:
    Chrome: 85.0.4183.83
    Firefox: 76.0.1
    Firefox Developer Edition: 65.0
    Safari: 13.1.2
  npmPackages:
    react: ^16.13.1 => 16.13.1
    react-dom: ^16.13.1 => 16.13.1
    react-elastic-carousel: ^0.6.6 => 0.6.6
    react-markdown: ^4.3.1 => 4.3.1
    react-moment: ^0.9.7 => 0.9.7
    react-multi-carousel: ^2.5.5 => 2.5.5
    react-router: ^5.2.0 => 5.2.0
    react-router-dom: ^5.2.0 => 5.2.0
    react-scroll: ^1.7.16 => 1.7.16
    react-static: ^7.2.0 => 7.4.0
    react-static-plugin-emotion: ^7.2.2 => 7.2.2
    react-static-plugin-favicons: ^2.0.0 => 2.0.0
    react-static-plugin-reach-router: ^7.4.2 => 7.4.2
    react-static-plugin-react-router: ^7.4.0 => 7.4.0
    react-static-plugin-sitemap: ^7.2.0 => 7.4.0
    react-static-plugin-source-filesystem: ^7.2.0 => 7.4.0
    react-static-plugin-styled-components: ^7.3.0 => 7.3.0
    react-text-loop: ^2.3.0 => 2.3.0
  npmGlobalPackages:
    react-static: 7.4.2

Steps to Reproduce the problem

clone my project at https://github.com/Yukon-Literacy-Coalition/ylc_website
run yarn
then run react-static build --staging --debug to see the error reports
To run it locally, make sure to navigate to http://localhost:3000/ylc_website/

Expected Behavior
I believe it should just build.

So, I tried creating a new project using the CLI running react-static create and creating a new project using the basic template. I imported my src files and whatever else was necessary to get it running locally.
Running react-static build leads to these type of errors:
  Error: Failed exporting HTML for URL ylc_website/about/opportunities (__react_static_root__/src/About/Opp  ortunities): Minified React error #294; visit https://reactjs.org/docs/error-decoder.html?invariant=294 f  or the full message or use the non-minified dev environment for full errors and additional helpful warnin  gs.

  - react-dom-server.node.production.min.js:44 a.b.read
    [ylc-site-2]/[react-dom]/cjs/react-dom-server.node.production.min.js:44:107

  - react-dom-server.node.production.min.js:54 renderToString
    [ylc-site-2]/[react-dom]/cjs/react-dom-server.node.production.min.js:54:364

  - exportRoute.js:167 renderToStringAndExtract
    [ylc-site-2]/[react-static]/src/static/exportRoute.js:167:21
...
...

Running `react-static build --debug also leads to the original error type:
  TypeError: Failed exporting HTML for URL / (__react_static_root__/src/Home): Cannot convert undefined or   null to object

  - Function.keys

  - Routes.js:72
    [ylc-site-2]/[react-static]/src/browser/components/Routes.js:72:14

  - react-dom-server.node.development.js:1194 useReducer
    [ylc-site-2]/[react-dom]/cjs/react-dom-server.node.development.js:1194:57

  - react-dom-server.node.development.js:1132 Object.useState
    [ylc-site-2]/[react-dom]/cjs/react-dom-server.node.development.js:1132:10
...
...

receiving the same errors with the empty template

I'm going to close this since I was able to get rid or this error but I'm not sure why.
I believe it might have had to do something with this:
      {
        path: basePath + "/annual_statements",
        template: "src/About/Statements",
      },

Once I removed this basePath it no longer yelled it me... 🤷♂

That would mean that basePath isn't actually defined there.
Good that you figured it out!

This does actually seem to still be an issue though. I just got the same error on first build, no changes.
  "dependencies": {
    "@reach/router": "^1.2.1",
    "axios": "^0.19.0",
    "react": "^16.9.0",
    "react-dom": "^16.9.0",
    "react-static": "^7.2.0",
    "react-static-plugin-reach-router": "^7.2.0",
    "react-static-plugin-sitemap": "^7.2.0",
    "react-static-plugin-source-filesystem": "^7.2.0",
    "react-static-plugin-typescript": "^7.2.0"
  },
  "devDependencies": {
    "@types/node": "^12.7.2",
    "@types/reach__router": "^1.2",
    "@types/react": "^16.9.1",
    "@types/react-dom": "^16.8.5",
    "@types/react-hot-loader": "^4.1.0",
    "@types/webpack-env": "^1.14.0",
    "react-hot-loader": "^4.12.11",
    "serve": "^11.1.0",
    "typescript": "^3.5.3"
  }

This almost always indicates something on the page that renders the routes is actually failing, and not react-static itself.

Yeah you're good mate. Builds fine on a fresh DO dropplett. My system is trash, as I suspected. Cheers!


Incorrect info fixes (both English & Russian)

All contribution in this PR was done with <3 by @AkciumKicum

Is this ready for review, or you are still working on it?

@Sominemo Ready for review.


Package.loadSettings is deprecated.

Store package settings files in the settings/ directory instead of scoped-properties/
Package.loadSettings (/Applications/Atom.app/Contents/Resources/app/src/package.js:565:9)
<unknown> (/Applications/Atom.app/Contents/Resources/app/src/package.js:187:43)

have absolutely no experience with packages, but will look into when I have a little time -- I gather this should be a tirivial fix.
ps.  thanks for the package by the way, this is what brought me to atom!

thanks for reporting. i think i will fix this tomorrow =)...

wow, thanks man! I'm very excited to see this  go away^^

done ;)...

thanks, tried it and it's working wonderfully!


feat: Cloudflare R2 provider

Talked about this already on Twitter.
I think I could fix the bug I  mentioned there.
Let me know if there is something that needs work or is unclear :)

Codecov Report
Attention: Patch coverage is 57.18954% with 131 lines in your changes missing coverage. Please review.

Project coverage is 86.91%. Comparing base (7c0e3ab) to head (60b7d0c).
Report is 12 commits behind head on main.

Files
Patch %
Lines

src/providers/cloudflare-r2/provider.ts
54.34%
105 Missing :warning:

src/utils/r2.ts
60.00%
18 Missing :warning:

src/providers/cloudflare-r2/transformer.ts
50.00%
8 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #277      +/-   ##
==========================================
- Coverage   90.67%   86.91%   -3.76%
==========================================
  Files          30       33       +3
  Lines        2423     2729     +306
  Branches      347      348       +1
==========================================
+ Hits         2197     2372     +175
- Misses        226      356     +130
- Partials        0        1       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

sorry for my late comments. Let me know if there is anything else :)


using OpenAi to Create Automatic Documentation

OpenAI enables automatic creation of documentation using the davinci model. An example can be found here: https://beta.openai.com/examples/default-explain-code
The goal is to write an API that takes lines of Code of interests and generates documentations.
#linesOfCode an array of code
#ModelParameters a dict defining characteristics of the model.
***({ModelName: ".."; Param1: "...", Param2: "...", Param3: "..."}
CreateDocumentation(linesOfCode, ModelParameters)

Input of function info can be a file in this format:  https://github.com/sdsen/opened_extraction/blob/main/op/cilium/cilium.function_file_list.json
Output can be same JSON extended with information regarding, model-params used for open-ai, open-ai op, date-time of invoking open-ai

Closing as tracked here: https://github.com/eBPFDevSecTools/ebpf-projects-annotations/issues/16


Keep getting "Error : Client side error"

In the provided sample scene I keep getting "Error : Client side error", I still get the recognized text though...
Furthermore I can't get it to work in another scene, I don't know if it's linked to the issue above or not.
Also documentation would be very appreciated to speed up integration and making it easier ^^'

"Client side error" here: https://developer.android.com/reference/android/speech/SpeechRecognizer#ERROR_CLIENT.
I don't know why. It is error code form STT of google (ERROR_CLIENT)
For Android. if you want to skip the default popup. pls merge file AndroidManifest with yours.


chore: update eslint to check codeblocks in all markdown files

We had an eslint setup in place for the academy only, now its merged with the root eslint and polished a bit, plus the apify eslint config is updated and the code is modified to comply with them.
The main change that applies to the markdown files is that many of them were incomplete or contained "pseudo JSON", e.g. with trailing commas or with some placeholders. Those are mostly moved to json5 which is more loose and works fine in this regard.

@TC-MO since this modifies a lot of files, it triggers a lot of unrelated vale checks, I'd ignore them and just merge it, I hope you are fine with that :]

LGTM


new build is failed to be triggered by kpack-controller when previous build has somethind wrong

Description
Kpack version: 0.7.2
After creating a cnb image with error env, the pod can't be triggered because the error configuration. But after correct the cnb image with correct env configuration, a new build can't be triggered too.
Reproduce step

Create a build with error env

apiVersion: kpack.io/v1alpha2
kind: Image
metadata:
  name: default.test1-deployment
  namespace: build-service
spec:
  additionalTags:
  - xxxxxxx-default-test1-deployment:result-2
  build:
    env:
    - name: '{BP_JVM_VERSION:8}'  // Error env configuration
      value: ""
    ...
  builder:
    apiVersion: kpack.io/v1alpha2
    kind: Builder
    name: default.default.1
  ...
  source:
    blob:
      url: xxx
  tag: - xxxxxxx-default-test1-deployment:result

Then kpack-controller will report errors:
{
   "level":"error",
   "ts":"2022-12-08T04:48:57.193459241Z",
   "logger":"controller",
   "caller":"controller/controller.go:566",
   "msg":"Reconcile error",
   "commit":"9343a26-dirty",
   "knative.dev/kind":"builds.kpack.io",
   "knative.dev/traceid":"9a908342-0d22-4e9a-a5c8-d0ff5afa199a",
   "knative.dev/key":"build-service/default.test1-deployment-build-1",
   "duration":0.324729574,
   "error":"Pod \"default.test1-deployment-build-1-build-pod\" is invalid: spec.initContainers[0].env[2].name: Invalid value: \"PLATFORM_ENV_{BP_JVM_VERSION:8}\": a valid environment variable name must consist of alphabetic characters, digits, '_', '-', or '.', and must not start with a digit (e.g. 'my.env-name',  or 'MY_ENV.NAME',  or 'MyEnvName1', regex used for validation is '[-._a-zA-Z][-._a-zA-Z0-9]*')",
   "stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20220816153547-f78a00694307/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20220816153547-f78a00694307/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20220816153547-f78a00694307/controller/controller.go:491"
}

patch the cnb image with correct configuration

apiVersion: kpack.io/v1alpha2
kind: Image
metadata:
  name: default.test1-deployment
  namespace: build-service
spec:
  additionalTags:
  - xxxxxxx-default-test1-deployment:result-3
  build:
    env:
    - name: BP_JVM_VERSION  // correct env configuration
      value: 8.*
    ...
  builder:
    apiVersion: kpack.io/v1alpha2
    kind: Builder
    name: default.default.1
  ...
  source:
    blob:
      url: xxx
  tag: - xxxxxxx-default-test1-deployment:result

The new build can't be triggered, there is still only previous build resource exist and kpack will continues report the previous build failed information

Why update the cnbimage can't trigger a new build? is it a bug?

Hi @matthewmcnew  can you take a look? Event the first trigger build has something wrong with the ENV, if the next build updates the env, kpack-controller should trigger a pod successfully, otherwise, all next pods will be blocked. Some of our customers meet the problem

Not sure if it is related to this commit: https://github.com/pivotal/kpack/commit/ac4ad3520f2e0fbdfedc103a9f157d62b1267970

Hi @zhoufenqin, We did some digging and it appears that the issue occurs when a build pod cannot be created. The build reconciler goes into a backoff loop and never updates its status to report a failed build, so the Image resource assumes a build is still in progress and does not schedule a new one with the fixed env. I will submit a pr to fix this, but short term you can delete the latest Build resource if you get into this state and the Image will create a new one with the fixed env var

thanks @tomkennedy513,  which kpack version will fix the bug? will you add a patch version in 0.7.x, like 0.7.3? Now we are using 0.7.2, next kpack version is 0.8.0, we need to do more investigation into it and decide to upgrade to 0.8.0 in our system. We hope to fix it ASAP in our prod env so fix with a patch version is good for short term


nit fix in README

Looks an interesting project. Going through it :)
cc @ryanmccann1024 et.al

Although we greatly appreciate someone going over our project, I am going to close this since the suggestion provided is not sufficient enough to improve the project to be considered for a valid PR. Also, there are typos in the alleged fix.


Variable "base_template" does not exist.

Twig_Error_Runtime
in vendor\redcode\tree-bundle\RedCode\TreeBundle\Resources\views\CRUD\tree.html.twig (line 1)
{% extends base_template %}
http://prntscr.com/hd74pg

downgrade sonata admin bundle to 3.23 version fix thiw bug... but what to do with latest updates?

Hi, Can you share you package.json to reproduce this issue?

https://gist.github.com/TaPTaK/709ddf8ae3571ef7260e905dda608230

i see this error "Variable "base_template" does not exist" in sonata issues... there is something with sonata crud controller "render" method... yesterday was something merged, today it was reverted...

Ok, I'm going to try the latest Sonata and release new version soon

Thanks
can you add some confirmation alert when push "delete" button in tree view?


A little bit more flexibility on urls for handle-function...

In my case i just added the RestServer to a selfwritten API which already does rewrites and routing based on urls...
Therefor i just needed to be able to use the handle-Method with a given url (not the REQUEST_URI)...
i did it that way:
public function handle($path = null) {
        if (!$path) {
            $this->url = $this->getPath();
        } else {
            $this->url = $path;
        }

That way i can do my own routing and authorization and pass the rest of the url directly to the RestServer...
Perhaps somebody may need that also...

RestServer is designed to be a standalone library, not integrated with another. It looks like you are making this more as an informational post, but this will not be implemented without a significant refactor to reduce the ugliness factor of such a modification.

Didn't realize that being a "standalone library" should make it impossible to be flexible to external vars for example... but ok, ack...

It doesn't necessarily mean that it's impossible. This library was not designed in a way that was easily modularizable. Basically, it was designed with the assumption that it handles routing, calling, loading, and processing. Implementing a bypass for any of these is certainly possible, but in order to properly accommodate this usage type, it would be better to properly modularize each of these components - something which would require more significant effort than a simple bypass.
Additionally, the way that the issue report was written looked more to me like an informational post than a feature request - if I've misinterpreted this, please feel free to clarify and reopen.
Hopefully that makes more sense.


Auto-generate pretty cover

Hi. First of all, thanks for the amazing plugin!
I was wondering about the cover shown in the cards like "Marketing" and "Indie Hacker", which are composed by the page's title + solid color.

I guess (mainly based on note's render logic) that these covers were somewhat (manually?) generated beforehand.
Do you have an easy way to generate these "title+background color" covers?
Would it be possible to have this automatically generated by this plugin?
And then have it offered as an option to replace "No cover" or even the default markdown preview? (a little bit like Card preview option in Notion)

Yes. These covers are generated manually. It would be a good idea to provide a way to automate them.🤔But I'm not sure when I'll be free to work on it. If anyone interested, feel free to submit a PR.


Create an operator of API Testing

Operator is a very popular way to install applications. Please feel free to help us to create an operator if you want to have a try.
References

https://operatorhub.io/

Done. See also:

https://github.com/k8s-operatorhub/community-operators/pull/3061
https://github.com/LinuxSuRen/api-testing/pull/158


Sak 29027 - Regression: TA's with site.upd can see announcements of any group.

Just an idea to change the permission for instructor check in announcement to annc.all.groups instead of site.upd. Was mentioned that this caused a regression otherwise.

@jonespm I think you've got 2 SAK's in here.

Yeah, I noticed, something messed up with my git workflow.


Pad code segment with EBREAKs

Currently libgpc linker creates the .text to be just as big as the compiled binary requires. When loaded into I_MEM, the rest of it will end up as garbage. To improve application debugability, the .text segment should be padded with EBREAKs, so execution will stop if PC goes out of bounds.
Open: is it applicable to non-debug builds as well?
Open: should D_MEM be padded as well?

This is very good practice for I_MEM - let's figure out how to integrate this to the linker/crt0

I got into strange troubles when trying to implement this padding in linkage. However, I found that zero is also good for this purpose: it has the structure of a compressed instruction (C extension), but it doesn't map to any real instruction (in disassembly it's shown as "unimp", which is not a real instruction). Bits [1:0] are used to indicate whether the instruction is compressed or not, so it won't be possible to get a valid opcode even if C extension is not supported. Therefore, this is good for this purpose (and much easier to implement).
See also:
https://github.com/riscv/riscv-asm-manual/pull/20


How to resolve SomaErrror

A few weeks ago when trying to extract distributions from the morphologies I had I ran into a SomaError because the soma was already defined. Intially I thought the problem was because of tmd as the call was traced back to it but now I'm not sure. I think the traceback is because of an error in line 81 input_distributions.py as that is where the traceback begins.
Any help would be greatly appreciated.
Source Code:
import json
from pathlib import Path
import neurots
from neurots import extract_input

def run(distr_dir, params_dir, neuron_dir, data_dir):
    for i in range(0,50,1):
        """Run the example for extracting inputs for synthesis."""
        # Generate distribution from directory of neurons

        distr_name = "test_distr_{}.json".format(i)
        params_name = "test_params_{}.json".format(i)
        neuron_name = "test_neuron_{}.swc".format(i)

        distr = extract_input.distributions(
            data_dir / "neurons",
            neurite_types = ["basal_dendrite"],
            feature="path_distances",
            diameter_model="default"
        )

if __name__ == "__main__":
    data_dir = Path("C:\\Users\\user\\Documents\\Experiment_01\\Data\\Human\\Neocortex_interneurons")
    params_dir = Path("C:\\Users\\user\\Documents\\Experiment_01\\Parameters\\Neocortex_interneuron")
    distr_dir = Path("C:\\Users\\user\\Documents\\Experiment_01\\Distributions\\Neocortex_interneuron")
    neuron_dir = Path("C:\\Users\\user\\Documents\\Experiment_01\\Synthesized\\Neurons\\Neocortex_interneuron")

    #result_dir.mkdir(parents=True, exist_ok=True)

    run(distr_dir, params_dir, neuron_dir, data_dir)
    print("DONE!!!")

Morphologies:
morphologies.zip

Hi @riddick-the-furyan
I checked the morphologies that are failing, and it appears that these files contain several somata, which makes no sense. Do they come from NeuroMorpho?
These files are asc files, so if you look inside them there are several blocks that start with "Cell Body", each of them defining a contour of the soma. If you keep only one of these blocks it should work properly. The issue is that it's hard to know which block should be kept, but I think it does not really matter in your case (since you are mainly interested in the neurite data, right?).

H, thanks I appreciate it a lot and yes, you're right.

My pleasure!


Retain empty variables when creating term env

Fixes #174330

Verified this works on mac/linux, Windows doesn't seem to allow empty env vars.


Missing argument in sprintf at ./pgcompacttable line 180.

[Thu Nov 26 08:12:16 2015] (dbname:tablename) Skipping index indexname:
[Thu Nov 26 08:12:16 2015] (dbname:tablename) SQL Error: ERROR:  relation "pgcompact_index_25734" already exists
Missing argument in sprintf at ./pgcompacttable line 180.

where 25734 is pid of ./pgcompacttable,
(used current master version,  122a6d2 as: ./pgcompacttable --host/var/run/postgresql/ --dbname stagingdbname --verbose)
perl:
 /usr/bin/perl --version

This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-linux-gnu-thread-multi
(with 41 registered patches, see perl -V for more detail)

Copyright 1987-2013, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

on ubuntu:
lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.3 LTS
Release:	14.04
Codename:	trusty

trying first time against db cluster:
psql -c 'show server_version';
 server_version
----------------
 9.3.10
(1 row)

Hi,
Try remove the index pgcompact_index_25734 and start the process again.

as errors feels to be only related to outputing info, i will for "clean exit" first (around hour probably), will try dropping index before next attempt though.

| Try remove the index pgcompact_index_25734 and start the process again.
worked. thanks.
any insight how this could have happened,and can any harm come out of this?

This problem is related that before running your process fails and the index remained.
Next, you try to re-start the process , the process generated an error (SQL Error: ERROR:  relation ....) on the existence of an index.

thanks, it did indeed fail on first try, due to attempt connecting to localhost instead of socket by default.
Closing.
PS great script.


Implement pruning of backups

In the backup script, delete backups that are older than 100 days.
Suggested algorithm:

List objects in backup bucket (listObjects)
For each object that has a modification date older than 100 days, delete it (deleteObject)

import boto3
import datetime as dt

s3 = boto3.resource('s3')
retention_period = 100

bucket = s3.Bucket('bucket-name')

# check each file if it is expired or not
for object in bucket.objects.all():
    gap = dt.datetime.now(dt.timezone.utc) - object.last_modified

    if gap.days > retention_period:
        object.delete()


Istio.service in mesh expansion crashes and fails to restart

Bug description
I'm using the sidecar.deb of istio mesh expansion.
After some time the istio.service fails and cannot be restarted.
To fix the service, one has to restore the shared memory with the following command:
sudo rm /dev/shm/envoy_shared_memory_0

The service goes up immediately after.
This is a referenced issue in Envoy: #4191
I include the following stacktrace from /var/log/istio/istio.err.log but I'm not sure if there is a link with the Envoy problem.
 cat /var/log/istio/istio.err.log
[Envoy (Epoch 0)] [2020-02-25 09:24:29.059][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 09:29:29.605][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 09:34:29.748][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 09:39:30.040][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 09:44:30.394][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 09:49:30.451][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 09:54:30.873][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 09:59:30.989][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 10:04:31.194][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 10:16:13.379][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 10:21:13.660][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 10:21:58.607][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 10:26:58.806][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,
[Envoy (Epoch 0)] [2020-02-25 10:31:59.355][6365][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 13,

Expected behavior
Istio.service should not crash.
Steps to reproduce the bug
Unclear. Happens randomly.
Version
Istio sidecar.deb 1.4.3/1.4.4
Environment where bug was observed (cloud vendor, OS, etc)
On premise.
Mesh expansion VM : Ubuntu 14.04 / Debian stretch

The logs do not show any crash, but expected disconnects due to keepalives. Is there any logs of crash?

Unfortunately /var/log/istio/istio.err.log was empty.  I guess that when the service went back up after I cleaned the shared memory, the logs were erased by systemd and the reboot of the service.
The corruption of the shared memory happend 3 times to me. The last one happended over night...
Next time I'll check the logs BEFORE rebooting the service :-/

Wait... It happend again...
I copied the complete logs in here
The main issue at hand seems to be the following:
[warning][config] [external/envoy/source/server/options_impl.cc:196] --allow-unknown-fields is deprecated, use --allow-unknown-static-fields instead.
[critical][assert] [external/envoy/source/server/hot_restart_impl.cc:45]panic: cannot open shared memory region /envoy_shared_memory_0 check user permissions. Error: File exists

And here is the logs from /var/log/istio/istio.log
The main issue seems to be:
warn    Epoch 0 terminated with an error: signal: aborted (core dumped)
warn    Aborted all epochs
error   Permanent error: budget exhausted trying to fulfill the desired configuration
error   cannot start the e with the desired configuration

Have you check and make sure there is only one envoy running? Now the istio.service will be auto-restarted when it has failure so envoy will be restarted as well.

sudo rm /dev/shm/envoy_shared_memory_0 should fix this

I moved on so I can't check anymore.
But as I explained in the original issue:

After some time the istio.service fails and cannot be restarted.

Unfortunatly , auto-restarting the service probably won't fix the error.

And yes, removing the shared memory works as I also mentionned earlier :

To fix the service, one has to restore the shared memory with the following command:
sudo rm /dev/shm/envoy_shared_memory_0

But this is only treating the symptoms and does not fix the corruption itself.

It looks like this is breaking during hot restart. Since we no longer do hot restart,  I think this bug is not present. If I misunderstood please reopen. thanks


Move Caching Cluster Client to java streams and allow parallel intermediate merges

We are seeing severe broker bottlenecks for certain types of queries. The existing Sequence paradigm does a folding reduction whereby the results from all the historicals and real-time nodes are folded into a single accumulator. This accumulation can be a major bottleneck in terms of performance for these large queries. TopN queries with results on order 10k returning from hundreds of nodes can take many seconds (in excess of 40s in some cases) to do the merge on the broker! A cpu-time flame graph of such a scenario can be seen below:

In an attempt to "fix" this bottleneck, I was having a hard time following the logic in CachingClusteredClient.  I also added many more comments as to why different sections were doing what they are doing. As a killer addition, a new query context intermediateMergeBatchThreshold is added which allows parallel intermediate merges
Intermediate merges
The Sequences produced by the druid client connections are collected in a Stream. If the query context intermediateMergeBatchThreshold is specified, then Spliterators are forked off of the Sequence Stream in batches according to the threshold if possible. Failure to do so reverts to prior merge behavior. These Sequences are fully materialized by working in a ForkJoinPool, and the results are Accumulated (reduce-fold-left kind of operation) on a first-available basis. All of the results fetched from cache are also another Sequence that gets merged in.
Why Fork Join Pool?
The "work" for the intermediate merges is done in a dedicated ForkJoinPool. A fork-join-pool was used instead of an Executor service in order to accommodate dynamic expansion of the worker pool. Basically to use a worker setup slightly better than running all the folding accumulation in the http server thread. The forking and joining semantics are used, but any effects of potentially blocking operations (like QTL) are not attempted to be accounted for. As such, having a specific pool of resources was chosen with the hope that eventually it can be more intelligent about being kind when doing potentially blocking operations. This is no worse than a fixed pool executor service and, when done properly, can handle new threads to circumvent blocked threads in a better manner. Future enhancements can include merge work task prioritization based on query priority, as well as enhancements to account for potentially blocking code.
Additionally, with how the fork and join items are intended to work, a ListenableFuture style workflow can be attained by doing a join of a task, and performing a task based on the result, so long as the join is executed in the FJP.
Other changes
Streaming methods are added to the Cache interface in order to accommodate changes in the caching cluster client.
CombiningSequence is moved out of the common module into java-util with the other Sequence classes
Additionally, small changes in the way the per-druid-node Sequences are initiated are present. The proposed implementation causes the direct druid client to fire off queries to all the nodes regardless of intermediate merges being used or not. The cached and uncached results now feed through the same "Stream" and are simply resolved as either a Merge Sequence (for cached results) or a Sequence from the direct druid client (for cache misses). This caused some unit tests to be changed to allow timeline lookups when previously there were none.
Results
Initial internal results show that query speeds were improved by 85% for large queries with intermediateMergeBatchThreshold set. Impact on smaller queries have not been fully investigated, and impact when intermediateMergeBatchThreshold is not set (just the new Stream workflow) is not fully exercised.
Work can clearly be seen being done in the ForkJoinPool as per:

With the final merge in the QueryResource.doPost under the jetty servlet stack.
Not included here
There are other items of interest for larger clusters, such as broker-brokers (or cross-broker merges). Hopefully the migrating of CachingClusterClient onto java streams makes such a possibility easier in the future.
 Since this feature is intended to help large clusters with large results, this could be a major problem if not taken care.
Handling back-pressure better would be a nice improvement. For example, if druid-client request returns could be controlled in a better manner to prevent an arbitrary number of them from returning an arbitrary size of result, then that would help larger results to be handled by the brokers.
TODO

[x] I'm still missing some unit tests for the cache changes in the hybrid cache

Conclusion
This is a huge performance boon (85%) for one of our core use cases. It would be interesting to see if this helps with large group-by queries as well. Since this is an opt-in feature that is very close to prior behavior when disabled, hopefully the risk for the change is very small, while the clarity of how the data stream is handled is clearer (or at least documented more).

nice PR description :)

Ok, I think this is ready now

@drcrallen thanks for this PR. I'll test it in our test cluster.

Thanks @jihoonson , please note there should be NO change in query performance unless intermediateMergeBatchThreshold is set to a non-zero value

Fixing linter problems

ah! copyrights, fixing

@drcrallen thanks. Would you add a document for intermediateMergeBatchThreshold? Is there anything else not documented yet?

@jihoonson I didn't document it yet because it is a highly experimental feature. There are still risks as explained a bit in the master comment. If it is still desired to document it but label it as experimental then I'm happy to do so, but it is in that weird middle ground where I don't feel comfortable recommending it for general use, but it has a very targeted case that it seems to help a lot

I didn't document it yet because it is a highly experimental feature. There are still risks as explained a bit in the master comment. If it is still desired to document it but label it as experimental then I'm happy to do so, but it is in that weird middle ground where I don't feel comfortable recommending it for general use, but it has a very targeted case that it seems to help a lot

@drcrallen what I'm concerned with is, lack of documentation makes difficult to test the users who may benefit from using this feature or even for developers. If it's experimental, I think we should also document a caveat for users to be able to know what are the expectations or not before using it. What do you think?

@jihoonson I can do that. One of the big caveats I've found in testing so far is that it does have a risk of blowing up heap for large results, so concurrent request considerations must be handled delicately.

Tests run: 3990, Failures: 0, Errors: 1, Skipped: 60, Time elapsed: 69.407 sec <<< FAILURE! - in io.druid.query.groupby.GroupByQueryRunnerTest
testBySegmentResultsWithAllFiltersWithExtractionFns[config=v2ParallelCombine, runner=rtIndex](io.druid.query.groupby.GroupByQueryRunnerTest)  Time elapsed: 0.979 sec  <<< ERROR!
java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:694)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
	at io.druid.query.groupby.GroupByQueryRunnerTest$8.get(GroupByQueryRunnerTest.java:358)
	at io.druid.query.groupby.GroupByQueryRunnerTest$8.get(GroupByQueryRunnerTest.java:354)
	at io.druid.collections.StupidPool.makeObjectWithHandler(StupidPool.java:112)
	at io.druid.collections.StupidPool.take(StupidPool.java:103)
	at io.druid.query.groupby.epinephelinae.GroupByQueryEngineV2.process(GroupByQueryEngineV2.java:125)
	at io.druid.query.groupby.strategy.GroupByStrategyV2.process(GroupByStrategyV2.java:354)
	at io.druid.query.groupby.GroupByQueryRunnerFactory$GroupByQueryRunner.run(GroupByQueryRunnerFactory.java:108)
	at io.druid.query.BySegmentQueryRunner.run(BySegmentQueryRunner.java:54)
	at io.druid.query.FinalizeResultsQueryRunner.run(FinalizeResultsQueryRunner.java:104)
	at io.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:119)
	at io.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:114)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
	at io.druid.query.ChainedExecutionQueryRunner$1.lambda$make$0(ChainedExecutionQueryRunner.java:112)
	at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
	at com.google.common.collect.Iterators.addAll(Iterators.java:357)
	at com.google.common.collect.Lists.newArrayList(Lists.java:147)
	at com.google.common.collect.Lists.newArrayList(Lists.java:129)
	at io.druid.query.ChainedExecutionQueryRunner$1.make(ChainedExecutionQueryRunner.java:104)
	at io.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:42)
	at io.druid.java.util.common.guava.Sequence.toList(Sequence.java:76)
	at io.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:124)
	at io.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:114)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

trying to merge with master to see if that fixes

lots of unrelated test failures

This will require some changes before going in. A huge portion of merge work is not accounted for in a parallel mechanism. I'm working on a fix

fixed to use chained execution query runners to execute parallel apply fn.
Looks like this now:

The overall CPU load is drastically reduced compared to the original version of this PR. The cpu flame graph is a bit lop sided because the overall cpu time this accounts for is actually very small.

failing on
[INFO] Compiling 9 source files to /home/travis/build/apache/incubator-druid/extensions-core/kafka-indexing-service/target/test-classes
error: error reading /home/travis/.m2/repository/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar; error in opening zip file
An exception has occurred in the compiler (9-internal). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.util.zip.ZipError: zip END header not found
	at com.sun.nio.zipfs.ZipFileSystem.zerror(ZipFileSystem.java:1605)
	at com.sun.nio.zipfs.ZipFileSystem.findEND(ZipFileSystem.java:1021)
	at com.sun.nio.zipfs.ZipFileSystem.initCEN(ZipFileSystem.java:1030)
	at com.sun.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:130)
	at com.sun.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:139)
	at java.nio.file.FileSystems.newFileSystem(FileSystems.java:390)
	at com.sun.tools.javac.file.JavacFileManager$ArchiveContainer.<init>(JavacFileManager.java:501)
	at com.sun.tools.javac.file.JavacFileManager.getContainer(JavacFileManager.java:310)
	at com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:692)
	at com.sun.tools.javac.code.ClassFinder.scanUserPaths(ClassFinder.java:572)
	at com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:512)
	at com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:294)
	at com.sun.tools.javac.code.ClassFinder.access$000(ClassFinder.java:73)
	at com.sun.tools.javac.code.ClassFinder$1.complete(ClassFinder.java:165)
	at com.sun.tools.javac.code.Symbol.complete(Symbol.java:590)
	at com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:299)
	at com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:509)
	at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:255)
	at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:270)
	at com.sun.tools.javac.comp.Enter.complete(Enter.java:485)
	at com.sun.tools.javac.comp.Enter.main(Enter.java:469)
	at com.sun.tools.javac.main.JavaCompiler.enterTrees(JavaCompiler.java:932)
	at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:815)
	at com.sun.tools.javac.api.JavacTaskImpl$1.call(JavacTaskImpl.java:97)
	at com.sun.tools.javac.api.JavacTaskImpl$1.call(JavacTaskImpl.java:93)
	at com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:128)
	at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:93)
	at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:87)
	at com.google.errorprone.BaseErrorProneJavaCompiler$1.call(BaseErrorProneJavaCompiler.java:99)
	at com.google.errorprone.BaseErrorProneCompiler.run(BaseErrorProneCompiler.java:137)
	at com.google.errorprone.BaseErrorProneCompiler.run(BaseErrorProneCompiler.java:108)
	at com.google.errorprone.ErrorProneCompiler.run(ErrorProneCompiler.java:119)
	at com.google.errorprone.ErrorProneCompiler.compile(ErrorProneCompiler.java:66)
	at org.codehaus.plexus.compiler.javac.errorprone.JavacCompilerWithErrorProne$CompilerInvoker.compile(JavacCompilerWithErrorProne.java:219)
	at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.compiler.javac.errorprone.JavacCompilerWithErrorProne.performCompile(JavacCompilerWithErrorProne.java:91)
	at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:823)
	at org.apache.maven.plugin.compiler.TestCompilerMojo.execute(TestCompilerMojo.java:153)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:955)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:290)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:194)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

Updating for https://github.com/apache/incubator-druid/pull/5108

Test failures:
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 61.769 sec <<< FAILURE! - in io.druid.server.coordinator.CuratorDruidCoordinatorTest
testMoveSegment(io.druid.server.coordinator.CuratorDruidCoordinatorTest)  Time elapsed: 61.767 sec  <<< ERROR!
java.lang.Exception: test timed out after 60000 milliseconds
	at io.druid.server.coordinator.CuratorDruidCoordinatorTest.testMoveSegment(CuratorDruidCoordinatorTest.java:369)

Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec <<< FAILURE! - in io.druid.java.util.emitter.core.HttpEmitterConfigTest
testDefaultsLegacy(io.druid.java.util.emitter.core.HttpEmitterConfigTest)  Time elapsed: 0.011 sec  <<< FAILURE!
java.lang.AssertionError: expected:<14> but was:<13>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.junit.Assert.assertEquals(Assert.java:542)
	at io.druid.java.util.emitter.core.HttpEmitterConfigTest.testDefaultsLegacy(HttpEmitterConfigTest.java:76)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
testDefaults(io.druid.java.util.emitter.core.HttpEmitterConfigTest)  Time elapsed: 0.007 sec  <<< FAILURE!
java.lang.AssertionError: expected:<14> but was:<13>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.junit.Assert.assertEquals(Assert.java:542)
	at io.druid.java.util.emitter.core.HttpEmitterConfigTest.testDefaults(HttpEmitterConfigTest.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)

Seemingly unrelated

@leventov any other comments here?

@drcrallen there are several unanswered comment from my last review round

@leventov / @jihoonson looks like some of the comments are unable to be responded to directly for some reason, maybe they are on a commit instead of the PR? I'll see if I can figure out what's goin on and get responses to them.

@leventov I'm worried some of your comments might be on commits instead of the PR, and might be lost in the UI for me to discover. I think I responded to all the comments. Can you please call out in a fresh comment in this PR any that are missing responses so I can figure out how they were missed?

phew package rename is pushed, fixing unit tests and then I'll get to roman's comments

got clobbered by https://github.com/apache/incubator-druid/pull/6313 , fixing

This Travis fail looks legit. Please check.
testTopNOnPostAggMetricCaching[0](org.apache.druid.client.CachingClusteredClientTest)  Time elapsed: 0.095 sec  <<< ERROR!
java.lang.ArithmeticException: / by zero
	at org.apache.druid.client.CachingClusteredClientTest.lambda$getDefaultQueryRunner$4(CachingClusteredClientTest.java:3114)
	at org.apache.druid.client.CachingClusteredClientTest.lambda$testQueryCaching$3(CachingClusteredClientTest.java:2134)
	at org.apache.druid.client.CachingClusteredClientTest.runWithMocks(CachingClusteredClientTest.java:2625)
	at org.apache.druid.client.CachingClusteredClientTest.testQueryCaching(CachingClusteredClientTest.java:2110)
	at org.apache.druid.client.CachingClusteredClientTest.testQueryCaching(CachingClusteredClientTest.java:1971)
	at org.apache.druid.client.CachingClusteredClientTest.testTopNOnPostAggMetricCaching(CachingClusteredClientTest.java:1059)

Also, are there unit tests for this feature?

I'm digging more. Trying to get this pr back up to the top of my priority list and address Roman's comments

@drcrallen thanks. Also please add some JMH benchmark.

@jihoonson what kind of info are you wanting for JMH stuff?

@drcrallen basically I want to check how large performance benefit the new parallel merge algorithm can give us. Even though you mentioned that it shows about 85% of performance improvement in some internal tests, it shows the overall performance benefit not the algorithm itself. Of course, the overall performance is more important, but the later one is also important because it gives us an insight about what we can expect exactly with this feature, I think.
Also the query performance is affected by many factors like dataSource size, query filter selectivity, # of aggregators and their types, cluster size, and so on. JMH would be useful because we can easily replicate the performance benchmark with the same query and the same data which makes us easy to maintain or improve it in the future.
I think the JMH should include the below:

Simplifying historical part to check the performance of only the broker merge algorithm. For example, there's no HTTP communication between CachingClusteredClient and actual query runners of historicals. Also, the query runners of historicals can just do aggregation for a single segment (or even return just some pre-aggregated values).
Benchmarking with varying intermediateMergeBatchThreshold and # of streams to be merged.
Maybe need testing against different query types (timeseries, topN, groupBy).

intermediateMergeBatchThreshold is a purely experimental feature and as such is completely opt in. Doing full benchmarking on it at this point is overkill unless it is being considered to replace the default.

For those finding this later, there are problems with GroupBy queries that are not appropriately addressed here. I think the split up between handling CachingClusterClient changes (which may or may not be needed) and enabling the GroupBy results merging to work better in parallel are really two different PRs and need addressed separately. I'm leaving the PR open for now as a reference, but it is unlikely to have any more development in favor of pursuing a more sustainable (and groupby compatible) approach


Re-sampling strategy

Our 3 data products scoped for v0 training (https://github.com/Clay-foundation/model/issues/19, https://github.com/Clay-foundation/model/issues/18, https://github.com/Clay-foundation/model/issues/20) have a median spatial resolution of 20 meters. We'll assume a single resolution model for v0, using 20m resolution to standardize inputs. We'll need to select an appropriate interpolation method for specifically the DEM and S1 inputs.

Just a note that in #27, we've went with resampling to a pixel resolution of 10m for Sentinel-2 and the DEM (Sentinel-1 RTC already has a pixel resolution of 10m, though see https://github.com/Clay-foundation/model/issues/19#issuecomment-1797316063). The resampling is using stackstac.stack's default resampling which is Nearest Neighbour (rasterio.enums.Resampling.nearest).
For Sentinel-2 optical imagery, nearest neighbor resampling is fine, but for the Copernicus DEM, we might want to consider bilinear resampling which compromises between pixel value preservation (nearest neighbor) and smoothing (cubic resampling), see e.g. answers like https://gis.stackexchange.com/questions/102868/best-resampling-method-for-dem-reprojection-in-qgis.

Originally posted by @MaxLenormand in https://github.com/Clay-foundation/model/issues/200

2. Copernicus DEM nearest neighbour resampling (?)

I followed the basic tutorial to get MGRS tiles over Puri India and checked the tiles in QGIS, and noticed the Copernicus DEM, band 13, seems to be interpolated to nearest neighbour, i.e. that pixels are resampled from 30m to identical values of 3x3 pixels of 10x10 meters, with all the same value. Here using QGIS's Value Tool plugin I can see there are 3 pixels, with the DEM raster not changing value on a 3x3 grid

I was wondering if there was a reason for not interpolating using something like a bilinear that might smooth the DEM more? I feel like this might weight the edges of these 3x3 grids more heavily in changes compared to the central pixel, when that's not really how the terrain is? I didn't find the code that resamples all datasets to the 10m, so may have gotten that wrong, this is only based on my visual interpretation in QGIS.

@yellowcap, can you make sure for the v1 data pipeline (https://github.com/Clay-foundation/model/issues/178) that we use bilinear interpolation for Copernicus DEM? Thanks!

Thanks for the hint! Definitively something to integrate - and wish we had caught that for v0.2 😄  V1 might not get the DEM anymore at first, but this is good to keep the resampling method at top of mind in the future.

Closing as out of date, feel free to reopen if appropriate.


Fix TypeScript typings

I'm not sure why it wasn't like this but typescript needs it to be like this in order to work. Without this change it just gives errors.

Thanks! The typings came from another PR

Thanks, could perhaps push this to the npm package because it's driving crazy having the package manager remove this fix every time I install another module?

Hey @Gerharddc, I'll release it on NPM once the PR for issue #71 gets tested a bit more.
In the meanwhile, you could just pull the package directly from git instead of npm:
// package.json
"react-native-modal-datetime-picker": "git+https://github.com/mmazzarolo/react-native-modal-datetime-picker.git"

Thanks!

@Gerharddc, landed on v4.10.0


fix: remove inline GC and schedule a background task instead

What type of PR is this?

Which issue does this PR fix:
What does this PR do / Why do we need it:
If an issue # is not available please add repro steps and logs showing the issue:
Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?
Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

"Test - gc with short interval / Test - gc with short interval"
Call this "GC stress test"


New chat creation popup displayed when hitting the <- button

Subject: New chat creation popup displayed when hitting the <- button
Description:

Start a new chat with a member
Click the <- button

Results:
Expected: I'm taken back to the message list
Actual: I'm taken to the new chat creation popup
Backend version: develop-1416d148
Frontend version: develop-1416d148
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36
Page: https://app.couchers.org/messages/chats
User: ccosta (77)

Is this a good first issue @lucaslcode?

Hey guys. I joined the project a while ago but couldn't contribute code so far. I can work on this as a first issue.

@dhovart great! Let me know if you need any guidance.


PCA throws exception no matter what

Attempting to run the pca() method of the PCA class results in the following exception:
Exception in thread "main" java.lang.UnsupportedOperationException
	at org.nd4j.linalg.factory.BaseBlasWrapper.geev(BaseBlasWrapper.java:467)
	at org.nd4j.linalg.eigen.Eigen.eigenvectors(Eigen.java:100)
	at org.nd4j.linalg.dimensionalityreduction.PCA.pca(PCA.java:62)

The same exception occurred with x86, jblas, and netlib backends.

Same here (OS X).

Note: same underlying issue as this: https://github.com/deeplearning4j/deeplearning4j/issues/927

Just a note: for computing PCA, I would prefer SVD (DGESVD, http://www.netlib.org/lapack/lug/node32.html) over Eigen...

I am closing this out in a week if no one objects.


findspam.py: bad_link_text(): fix regex

Refactor and (I hope) fix a couple of apparent errors in the long regex.

Codecov Report

Merging #1052 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #1052   +/-   ##
======================================
  Coverage    63.8%   63.8%
======================================
  Files           6       6
  Lines        1768    1768
======================================
  Hits         1128    1128
  Misses        640     640

Impacted Files
Coverage Δ

findspam.py
82.58% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e9f234...25ad570. Read the comment docs.

Besides wrapping the regex into more logical chunks (don't wrap related phrases across line breaks in the source), I changed a couple of (^| ) and ( |$) into \b (which should also match e.g. when preceded or followed by punctuation) and replaced a "wild" wildcard .* with a slightly less FP_prone (?:\b.{1,20}\b)?

!!/approve

This is waiting for a final merge.  Anyone ...?


Totals show NaN when a null value is in the data set

Whenever there is a null value in a column, the totals end up NaN. It would be great to have a null/undefined check in the totaling routine.

Interesting.   I will look at this -- I thought I had code in their to prevent that.

Sorry I forgot to respond to say this was fixed in the last commit.


Remove :memory file and fix in-memory SQLite setup.

:memory file makes it problematic on Windows

fixes #120

Thanks @simi


fix hygiene of temp_vars (use a Span::mixed_site located at the param span)

Re-attempt 177130db, 29c07fd8

Merged as https://github.com/verus-lang/verus/commit/96882b698c14315cc39050e0f1ed132b3807ed31


Improve the Langchainrb documentation

Problem
We have outgrown the single-file README. It is time to migrate to a more scalable solution.
Auto-generated documention RDoc-style may not be ideal as it'll read as a dry technical manual.
Ideas

Move documentation to Wiki
Create a dedicated a website (@rickychilcott shared https://www.phlex.fun/ as an example)

@ZhenhangTung I know you had mentioned that the documentation was lacking. Would this be a better approach? Do you have any feedback on how to improve it further?

@andreibondarev I do think it's better than the previous one. It's more structured and guide uses with concepts and code. I'll provide feedback if I have any in the future. For now, I believe we should prioritize something like user investigations or feature implementations.

Move documentation to Wiki

The main downsides are that you can't do PRs against them, and that they are totally separate from the main repo. That means it's an extra step to update if code changes means docs changes.

Auto-generated documention RDoc-style may not be ideal as it'll read as a dry technical manual.

There is definitely value in it though. I've had to go to the langchain(.py) docs, and found them hard to read, and not super useful. Example: https://python.langchain.com/en/latest/reference/modules/vectorstores.html
Rails itself is a good example of thorough API reference (https://api.rubyonrails.org) , while there is still room for more general guides (https://guides.rubyonrails.org/)

https://python.langchain.com/en/latest/reference/modules/vectorstores.html
Yeah, that's exactly the technical manual look and feel that I was referring to.

I really like how Rails is documented. And not just the docs but the code itself!
And you're right, I think we'll need to create a /docs directory with a bunch markdown files so they can be modified along with actual code changes. And then auto-deploy the docs to Github Pages or something.

A few changes landed since last update: https://github.com/andreibondarev/langchainrb/pull/112 https://github.com/andreibondarev/langchainrb/pull/152
https://rubydoc.info/gems/langchainrb now contains nearly all of the contents of the wiki, but located on the relevant class's rdoc.


Decompiler removes leading underscores from param names

Bicep version
1.23.0
Describe the bug
ARM templates associated with Visual Studio deployment projects (*.deployproj) have a couple of variables that point to the artefacts.  These are typically defined like this
 "_artifactsLocation": {
      "type": "string"
    },
    "_artifactsLocationSasToken": {
      "type": "securestring"
    },

When decompiled the leading underscore is removed...
param artifactsLocation string

@secure()
param artifactsLocationSasToken string

This breaks the standard Deploy-AzureResourceGroup.ps1 script.

I just got this bug today. I decompiled a json to make a bicep one and all _ were removed. I add them again, rebuild and it's ok but it should be solved or include in a "known issues". @alex-frankel


DataclassJSONField implementation

as discussed in https://github.com/bhch/django-jsonform/issues/75
WIP WIP WIP
Example
@dataclass
class MyDataclass(JsonSchemaMixin):
    id: int
    whatever: str

class MyModel(TimestampedModel):
    single = DataclassJSONField(MyDataclass, null=True)
    many = DataclassJSONField(MyDataclass, many=True, blank=True, default=list)

In [1]: m = MyModel.objects.first()

In [2]: m.single
Out[2]: MyDataclass(id=123, whatever='123')

In [3]: m.many
Out[3]: [MyDataclass(id=123, whatever='123'), MyDataclass(id=456, whatever='456')]

In [4]: m.single.whatever='hello'

In [5]: m.many[0].whatever='world'

In [6]: m.many.pop()
Out[6]: MyDataclass(id=456, whatever='456')

In [7]: m.save()

In [8]: m = MyModel.objects.first()

In [9]: m.single
Out[9]: MyDataclass(id=123, whatever='hello')

In [10]: m.many
Out[10]: [MyDataclass(id=123, whatever='world')]

FYI: I had to delete my other account for logistics reasons.
I'm starting to lean towards making a separate package because I don't have time to investigate and support older Python/Django versions.
My next step is to fix the "required" fields, because on Admin, I can't submit empty for fields like x: int | None

I think it has to do with the generated JSON schema.
@bhch  does your library support json schema with anyOf? https://stackoverflow.com/questions/35450405/how-to-represent-sum-union-types-in-in-json-schema

Currently, anyOf doesn't work but it will be added (#39). I can't seem come up with a good approach to  create forms for anyOf/allOf/oneOf.

I'm starting to lean towards making a separate package because I don't have time to investigate and support older Python/Django versions.

That's understandable. I'd like to avoid breaking changes until the next major release (v3).

Another question @bhch , I noticed that the Django admin form sends Empty String for empty integer field, rather than null
I know this is probably because it's hard to tell the difference between null and no input and empty string.
One thing Swagger UI does is they have a tick box [] Send empty value to indicate sending a null.
Thoughts on adding this to the Form?

@dongyuzheng Okay, I'll add something similar to that. I've just created an issue for this: #78


Add the option to automatically stick link nodes to aligned pixels when close

The idea is to implement the method existent in most graphical editors.
When dragging a link node, and the mouse is close to a point that is horizontally or vertically aligned with other nodes, the mouse with the node is sticked to that point.
Priority rules to select points when multiple options are available.

aligned with the next or previous node in the same link.
align with the closest node in the same link.
align with the closes node.

Add a checkbox to enable disable this option


Update the title of the sign in view

Pre-Submission Checklist

[x] Your pull request targets the staging branch of FreeCodeCamp.
[x] Branch starts with either fix/, feature/, or translate/ (e.g. fix/signin-issue)
[x] You have only one commit (if not, squash them into one commit).
[x] All new and existing tests pass the command npm run test-challenges. Use git commit --amend to amend any fixes.

Type of Change

[x] Small bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds new functionality)
[ ] Breaking change (fix or feature that would change existing functionality)
[ ] Add new translation (feature adding new translations)

Checklist:

[x] Tested changes locally.
[ ] Closes currently open issue (replace XXXX with an issue no): Closes #XXXX

Description

LGTM :+1:
Thanks @raisedadead :sparkles: :tada:


Fix removal of all instances of X item from inv if amount == 1

If a player has multiple instances of the same item with an amount of exactly 1 within their inventory, /hat will remove all of those instances from the inventory upon setting that block as the hat.

Happy to merge this as is, but it might also be worth using the removeItems in Bukkit method instead of your own removeExact.
Let me know your thoughts

I definitely agree with that, I just used what worked for my own personal Hats plugin. You can modify it as you deem best to your scenario.


fix: make revision_id not show in schema

Fix #418
This is my first ever pull request (will hopefully not be the last), so please be kind!  :sweat_smile:
I basically found this solution and applied it to the Document class, and it worked at first try. So, thanks @Xenthys!
Also, @roman-right, correct me if I am wrong but I believe the previous two lines:
            for field_name in model._hidden_fields:
                schema.get("properties", {}).pop(field_name, None)

become redundant. I believe they were trying to accomplish the same thing, but it seems that it only works once the doc is initialized.
So, if you agree @roman-right, I can remove the redundancy before merging.

Hey! Thank you for the PR. I'll take a look tomorrow

I'm running into this issue as well, any timeline on when this will be merged in?

Hi guys,
Sorry for the delay.
Could you please provide an example of when the revision_id appears in the schema? I need it to add a test for this case.
@ivan-gj , @NickRafRGA

No worries, @roman-right, here's a small test written for pytest:
from beanie import Document

class Foo(Document):
    """Dummy document."""
    bar: int = 3

def test_revision_id_not_in_schema():
    """Check if there is a `revision_id` slipping into the schema."""
    schema = Foo.schema()
    assert "revision_id" not in schema["properties"]

Actually, here's a version with an extra check (just in case) where we actually specify that the test is only valid if the collection is not initialized (if the collection is initialized then the issue is gone):
from beanie import Document
from beanie.exceptions import CollecctionWasNotInitialized

import pytest

class Foo(Document):
    """Dummy document."""
    bar: int = 3

def test_revision_id():
    """Check if there is a `revision_id` slipping into the schema."""
    schema = Foo.schema()
    assert "revision_id" not in schema["properties"]

    # check that the document has not been initialized,
    # as otherwise the `revision_id` is normally gone from the schema.
    with pytest.raises(CollecctionWasNotInitialized):
        Foo.get_settings()

Hi! Sorry for the delay again. Thank you! I'll pick this up by this weekend

Ah, I see now. It covers the case when the model was not initialized. I tried to reproduce this with inited models and could not

Thank you for the PR! Merged.
It will be published in the next releases (beta or normal) today-tomorrow

Thank you for merging it!
Happy to contribute to the project.
Looking forward to the upcoming release.


Per-language scopes?

For discussion...
Currently, cursorless has (I believe):

one set of scopes for all languages
per-language scope definitions
one spoken form to scope mapping for all languages

We could generalize the first and third a bit. We could allow each language to have its own set of scopes, ideally with a large shared core of common scopes. And then we could let each language have its own spoken form mapping.
Advantages:

Each language will have its own set of conceptually appropriate scopes. For example, the distinction between map and list scopes do not apply cleanly into Go. And Go has no notion of a class or a class name. And most programming languages have no notion of a chapter or a cell.

Disadvantages:

Complicates mixed language support.

Unsure:

It is unclear to me whether this would make learning harder or easier. On the one hand, things now change between languages. On the other hand, there would still be a large shared core, and there would be fewer confounders in the grammar like "chapter" (which I have disabled). And as it stands, unless you are working in the original target languages for cursorless, you have to learn to mentally translate some scopes to your active language.

This would be a very large change, but it seemed worth at least raising and discussing.

Maybe let's discuss in a meet-up; it's definitely something we've considered.  See also https://github.com/cursorless-dev/cursorless/issues/554

From Slack:

In Clojure I often use “keywords” like :foo. Cursorless shows 2 hat marks, one for the word and one for the colon. Is there a direct way to select (take) the whole keyword?

From Slack:

In Clojure I often use “keywords” like :foo. Cursorless shows 2 hat marks, one for the word and one for the colon. Is there a direct way to select (take) the whole keyword?

Pretty sure that's just "key" but could be wrong

Response from Slack:

“key” does work if the key is inside a map.
But I also often use keywords other places, like an array ex: [:foo :bar] . In that case I get “Couldn’t find containing collectionKey.”

(My apologies for forking the conversation.)


Crates can be opened once without a key

https://www.youtube.com/watch?v=JPkxp41IC1M&feature=youtu.be
(Chests are invisible due to a bug on this players mc client idk)
Tested this with older versions and also confirmed it worked on those too.
No errors in console. Really strange though since I am using the plugin on another gamemode without issues.
Using PaperSpigot 1.8.8
Plugins: http://imgur.com/a/BVTpm

Was happening because new players got 1 free key for every crate.