Dashboard results are very slow when pointed to Postgres Database

We have pointed the SQLALCHEMY_DATABASE_URI to POSTGRES database in superset and created the slices using Oracle database connection.
When we accessing the slices from the Charts menu, getting the results in less than 5 seconds.
But when we place the slices in dashboard and access it from the "Dashboards" menu reports are loading very slow and taking more than 50 seconds to see the data.
Are there any pre-requisites to follow to point Superset to Postgres database or any other configurations needs to perform to solve the server slowness issue.
Expected results
Results should show in the dashboard screen less than 10 seconds.
Actual results
Report performance is very slow when we place it in dashboard and taking more than 50 sec to load.
Environment
(please complete the following information):

OS Details : Redhat Linux - 7
superset version: unofficial-superset:0.33.010402
python version: 3.7.3
postgres version: 9.2.24

Checklist
Make sure these boxes are checked before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Hi @SyamprasadNakka,
I'm not totally understanding your issue. First to clarify, SQLALCHEMY_DATABASE_URI configures the connection string for superset's metadata DB.
Have you changed the metadata DB from Oracle to PG?

Hi @dpgaspar ,

Hi @SyamprasadNakka,
I'm not totally understanding your issue. First to clarify, SQLALCHEMY_DATABASE_URI configures the connection string for superset's metadata DB.
Have you changed the metadata DB from Oracle to PG?

We have changed SQLALCHEMY_DATABASE_URI = 'postgresql://superset:superset@localhost/superset' in config.py to point Postgres database for metadata and running the server.
After we installed cx_oracle module and creating reports through the Oracle database connection as below.

Create Oracle database Connection: Sources --> Databases --> SQLAlchemy URI = oracle://username:password@hostname:port/SID
RUN Query: SQL Lab --> SQL Editor --> execute query --> Explore.
Save Report: Explore --> Save report.
Pull the reports on to Dashboards.

After pulling the reports on to dashboards the results are getting very slow and taking 50 sec to load.
Kindly let me know if any questions.

Could you please provide any inputs/suggestions on the issue.
Regards,
Syam

Sorry to insist, should I assume that this is a first time installation? or have you changed your metadata DB from ORACLE (or some other db engine) to Postgres?
If the dashboards are rendering data from ORACLE what is your reasoning for assuming the poor performance comes from Postgres?

Sorry to insist, should I assume that this is a first time installation? or have you changed your metadata DB from ORACLE (or some other db engine) to Postgres?
If the dashboards are rendering data from ORACLE what is your reasoning for assuming the poor performance comes from Postgres?
(could this be related with #8568 ? )

Hello @dpgaspar ,
Please find the below details.

Using Postgres database for superset metadata.
Using Oracle database for reports development.

Kindly let me know if any questions.
Regards,
Syam

Ok, you can:

Troubleshoot your ORACLE engine, look for IO, network, CPU, memory. Identify the bottleneck if it's related to the engine
Troubleshoot your superset deploy look for high memory usage or CPU.
Verify what is the amount of data requested by the dashboard, if it's too big, consider decreasing (for example using smaller time slices)
Consider setting up cache: https://superset.incubator.apache.org/installation.html#caching

Ok, you can:
* Troubleshoot your ORACLE engine, look for IO, network, CPU, memory. Identify the bottleneck if it's related to the engine

* Troubleshoot your superset deploy look for high memory usage or CPU.

* Verify what is the amount of data requested by the dashboard, if it's too big, consider decreasing (for example using smaller time slices)

* Consider setting up cache: https://superset.incubator.apache.org/installation.html#caching

Hello @dpgaspar ,
We can't set the CACHE as users should see the up-to-date data from the database.
We have a dashboard with 10 slices in it but takes 40 seconds to show up the result in the screen. Our metadata sits on PostgreSQL and our slice data resides on Oracle database.
We noticed that the request from Superset to the Oracle database happens only after 30 seconds.
Why is there a delay in sending the queries to the Oracle DB from Superset application?
Is there any setting we need to modify to have the query requests sent immediately to the Oracle database from Superset application?
Please let me know if any inputs/suggestions.
Regards,
Syam

Hello,
Any inputs/suggestion on the issue really appreciated.
Regards,
Syam

That is unexpected to me, can anyone provide further insights?
Can you post the server logs for this, showing the 30 seconds delay?
Thank you

That is unexpected to me, can anyone provide further insights?
Can you post the server logs for this, showing the 30 seconds delay?
Thank you

Hello @dpgaspar ,
Please find the below log and provide your inputs if any. Have accessed the dashboard @18/Nov/2019:06:22:39 and you can find the same in logs.
Due to some security constraints have removed the report queries and marked the URL's with custom names.

[2019-11-18 06:22:18 -0500] [37356] [INFO] Starting gunicorn 19.8.0
[2019-11-18 06:22:18 -0500] [37356] [INFO] Listening at: https://<IP_ADDRESS>:8088 (37356)
[2019-11-18 06:22:18 -0500] [37356] [INFO] Using worker: threads
[2019-11-18 06:22:18 -0500] [37356] [INFO] Server is ready.  Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:22:53,102:INFO:root:Database.get_sqla_engine().  Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:22:57,445:INFO:root:
SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 1000
2019-11-18 06:22:57,452:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:22:57,525:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:22:57,601:INFO:root:Cache key: c06e2d38845633af72a38326a6aaf327
2019-11-18 06:22:57,613:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:22:57,615:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query for as this is sensitive data ##############
WHERE ROWNUM <= 512345678
2019-11-18 06:22:57,645:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:22:57,830:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:22:57,896:DEBUG:root:[stats_logger] (incr) loaded_from_source
2019-11-18 06:22:57,897:INFO:root:Cache key: 9c207fe9fa22522893958347be568d5d
2019-11-18 06:22:57,901:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 1000
2019-11-18 06:22:57,930:INFO:root:Cache key: acc8b8ca33f74db8f985755ade233c20
2019-11-18 06:22:57,945:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:22:57,946:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 512345678
2019-11-18 06:22:57,979:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:22:58,324:DEBUG:root:[stats_logger] (incr) loaded_from_source
2019-11-18 06:22:58,324:INFO:root:Cache key: None
VPD Successfully Applied for the User : username
VPD Successfully Applied for the User : username
<IP_ADDRESS> - - [18/Nov/2019:06:22:58 -0500] "POST /superset/explore_json/?form_data=%7B%22slice_id%22%3A1000198%7D HTTP/1.1" 200 625 "https://hostname:8088/superset/dashboard/11234567804/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019-11-18 06:22:58,589:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:22:58,659:INFO:root:Cache key: 03d9562c997db8e94dd86889cf8d6d92
2019-11-18 06:22:58,672:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:22:58,673:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 512345678
2019-11-18 06:22:58,702:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:00,322:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:23:00,378:INFO:root:Cache key: ecece617269112a0e00465cae57ca87b
2019-11-18 06:23:00,392:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:00,393:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 512345678
2019-11-18 06:23:00,418:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:06,755:DEBUG:root:[stats_logger] (incr) loaded_from_source
VPD Successfully Applied for the User : username
VPD Successfully Applied for the User : username
<IP_ADDRESS> - - [18/Nov/2019:06:23:06 -0500] "POST /superset/explore_json/?form_data=%7B%22slice_id%22%3A1000217%7D HTTP/1.1" 200 1037 "https://hostname:8088/superset/dashboard/11234567804/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019-11-18 06:23:07,010:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:23:07,076:INFO:root:Cache key: 9d99ee4ac42b004572a78b84c427806e
2019-11-18 06:23:07,088:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:07,090:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 512345678
2019-11-18 06:23:07,115:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:08,472:DEBUG:root:[stats_logger] (incr) loaded_from_source
VPD Successfully Applied for the User : username
<IP_ADDRESS> - - [18/Nov/2019:06:23:08 -0500] "POST /superset/explore_json/?form_data=%7B%22slice_id%22%3A1000227%7D HTTP/1.1" 200 976 "https://hostname:8088/superset/dashboard/11234567804/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019-11-18 06:23:08,736:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:23:08,803:INFO:root:Cache key: 9ee8fe1150feb03553a2584b808153de
2019-11-18 06:23:08,835:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:08,837:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 112345678
2019-11-18 06:23:08,913:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:09,576:DEBUG:root:[stats_logger] (incr) loaded_from_source
VPD Successfully Applied for the User : username
VPD Successfully Applied for the User : username
<IP_ADDRESS> - - [18/Nov/2019:06:23:09 -0500] "POST /superset/explore_json/?form_data=%7B%22slice_id%22%3A1000215%7D HTTP/1.1" 200 1175 "https://hostname:8088/superset/dashboard/11234567804/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019-11-18 06:23:09,594:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:23:09,647:DEBUG:root:[stats_logger] (incr) loaded_from_source
VPD Successfully Applied for the User : username
<IP_ADDRESS> - - [18/Nov/2019:06:23:09 -0500] "POST /superset/explore_json/?form_data=%7B%22slice_id%22%3A1000216%7D HTTP/1.1" 200 1110 "https://hostname:8088/superset/dashboard/11234567804/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019-11-18 06:23:09,669:INFO:root:Cache key: a3fd1a6f4f865841976f75b7d67b7eb8
2019-11-18 06:23:09,689:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:09,691:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 1000
2019-11-18 06:23:09,733:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:09,913:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:23:09,980:INFO:root:Cache key: a4c4e35bfafe1f20ff79edb3ec80342a
2019-11-18 06:23:09,989:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:09,991:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 1000
2019-11-18 06:23:10,011:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:10,820:DEBUG:root:[stats_logger] (incr) loaded_from_source
VPD Successfully Applied for the User : username
<IP_ADDRESS> - - [18/Nov/2019:06:23:10 -0500] "POST /superset/explore_json/?form_data=%7B%22slice_id%22%3A1000230%7D HTTP/1.1" 200 11855 "https://hostname:8088/superset/dashboard/11234567804/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019-11-18 06:23:11,037:DEBUG:root:[stats_logger] (incr) loaded_from_source
<IP_ADDRESS> - - [18/Nov/2019:06:23:11 -0500] "POST /superset/explore_json/?form_data=%7B%22slice_id%22%3A1000214%7D HTTP/1.1" 200 2607 "https://hostname:8088/superset/dashboard/11234567804/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019-11-18 06:23:11,109:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:23:11,123:DEBUG:root:[stats_logger] (incr) loaded_from_source
VPD Successfully Applied for the User : username
<IP_ADDRESS> - - [18/Nov/2019:06:23:11 -0500] "POST /superset/explore_json/?form_data=%7B%22slice_id%22%3A1000225%7D HTTP/1.1" 200 1028 "https://hostname:8088/superset/dashboard/11234567804/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019-11-18 06:23:11,183:INFO:root:Cache key: 011921c76c9878320f597b15302bc630
2019-11-18 06:23:11,193:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:11,195:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 1000
2019-11-18 06:23:11,216:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
<IP_ADDRESS> - - [18/Nov/2019:06:23:11 -0500] "GET /superset/favstar/Dashboard/11234567804/count/ HTTP/1.1" 200 12 "https://hostname:8088/superset/dashboard/11234567804/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019-11-18 06:23:11,397:DEBUG:root:[stats_logger] (incr) explore_json
2019-11-18 06:23:11,466:INFO:root:Cache key: 0bc52b5111d00fdfccb0c66ac43ddd59
2019-11-18 06:23:11,481:INFO:root:Database.get_sqla_engine(). Masked URL: oracle://username:password@hostname:port/dbname
2019-11-18 06:23:11,483:INFO:root:SELECT
##################### Database Query Generated ########### Removed the query, as this is sensitive data ##############
WHERE ROWNUM <= 1000
2019-11-18 06:23:11,509:INFO:root:Database.get_sqla_engine().

To clarify is this the delay:
><IP_ADDRESS> - - [18/Nov/2019:06:22:39 -0500] "GET /dashboardasync/api/read?_oc_DashboardModelViewAsync=changed_on&_od_DashboardModelViewAsync=desc HTTP/1.1" 200 908 "https://hostname:8088/superset/welcome" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
>2019-11-18 06:22:53,010:DEBUG:root:�[36m[stats_logger] (incr) dashboard�[0m

If yes, can you make the HTTP GET and time it: /dashboardasync/api/read?_oc_DashboardModelViewAsync=changed_on&_od_DashboardModelViewAsync=desc

To clarify is this the delay:
><IP_ADDRESS> - - [18/Nov/2019:06:22:39 -0500] "GET /dashboardasync/api/read?_oc_DashboardModelViewAsync=changed_on&_od_DashboardModelViewAsync=desc HTTP/1.1" 200 908 "https://hostname:8088/superset/welcome" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
>2019-11-18 06:22:53,010:DEBUG:root:�[36m[stats_logger] (incr) dashboard�[0m

If yes, can you make the HTTP GET and time it: /dashboardasync/api/read?_oc_DashboardModelViewAsync=changed_on&_od_DashboardModelViewAsync=desc

Hello @dpgaspar ,
I meant to say, on 18/Nov/2019:06:22:39 have accessed the dashboard and it's took more than 30 sec to load the completed dashboard page. Not sure the root cause of the delay.
Can you please elaborate to what needs to be done from our side?
Regards,
Syam

My previous post with a selection of 2 log entries is just to check if this is where you experience the 30s delay. If yes, toubleshoot the API call and the statsd event

My previous post with a selection of 2 log entries is just to check if this is where you experience the 30s delay. If yes, toubleshoot the API call and the statsd event

We tried to debug the issue and couldn't found any root cause to overcome the dashboard response delay.
Regards,
Syam


refactoring

What is the necessity of this update? What is updated?
refactoring code using functional map
How did you do the test? (Not required for updating documents)
path through default maven test

LGTM


[Feature Request] Battery Percentage

It would be great for the battery status to be shown on the lock screen with your WebKit greeter.

Web greeter doesn't give me access to such information, i'll leave this open for a future GTK version of the theme

Thabks, that will be awesome.


Problems installing dependencies

Hi! Amazing paper and results. I'm running into some trouble installing the dependencies to reproduce the results.
Both python-apt and pygobject don't seem to be installable via pip install -r requirements.txt.
ERROR: No matching distribution found for pygobject==3.26.1
ERROR: No matching distribution found for python-apt==1.6.3

Any idea what I might be doing wrong? Thanks!!

btw, I've followed the README instructions exactly!

Maybe try installing a different version? Not sure what the issue is, @ramanans1 might know.

I've met such error when I used conda to create a virtual environment. Then I simply switch to use virtualenv, everything goes well.


Wrong reaction-path degeneracies in PopulateReactions

In comparing RMG-Py and RMG-Java kinetics for the propyl + O2 reaction network, I discovered many reactions which differ only by the reaction-path degeneracy. For example, the R_Recombination reaction
CC[CH2] + [O][O] -> CCCO[O]
should have Arrhenius kinetics such that A = 4.52e12 cm^3/mol*s. When PopulateReactions is run without pressure dependence, this is indeed the case. However, when pressure dependence is turned on, the A factor for the corresponding path reaction is only 2.26e12, and is therefore missing the factor of two coming from the reaction-path degeneracy.
Interestingly, as far as I can tell normal RMG jobs are fine, both with and without pressure dependence.

We also noticed that PopulateReactions.writeOutputString() has code to handle this problem (by rerunning the react() function using the reverse direction). Unfortunately, this is long after the FAME call that generates the k(T,P) values. Perhaps we could move this earlier in the PopulateReactions process so that it happens before addReactionToNetworks()?


floating point operations and test

I met a nasty bug that will now wake me up at night... It concerns floating point operations and I feel like it is not really included in your list. Note that one could argue this problem is not python specific but it still caused some nasty bug and people might want to be aware of this. (This blog reminded me of the bug and helped me formulate it more clearly here). It boils down to:
1 + 2 == 3
True
(1 + 2)/10 == 3/10
True
1/10 + 2/10 == 3/10
False
.1 + .2 == .3
False
And of course :
0.3 - 0.2 - 0.1 == 0
False

That's not a bug, that's how floating point works


feat(fox lp/farming): remove hardcoded accountNumber bip44Param

Description
Does what it says on the box - removes the hardcoded accountNumber in favor of the bip44Params.accountNumber from the relevant account metadata field.
Notice

[x] Have you followed the guidelines in our Contributing guide?
[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?

Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[x] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

tackles #2609
tackles #2610

Risk

There shouldn't be any as safeguards have been added against loading accountNumber, however, this has only been tested with MM, and a cross-wallet FOX-ETH/LP regression test should be done to ensure we are consistently getting and accountNumber, and the FOX-ETH LP/Farming features still work.

Testing

Refer to the risks section

Engineering

Refer to the top-level testing section
console.log / debug the accountNumber memoized value to ensure it is null while the metadata is loading / the correct account number after it's been derived

Operations

Ensure the FOX-ETH Farming/LP feature is still working both with the MultiAccount flag on and off (this assumes FoxLP and FoxFarming flags on for the two features, respectively)

Screenshots (if applicable)

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

PR #2651

PR #2652
PR #2661

PR #2680
PR #2669

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

PR #2651

PR #2652
PR #2661

PR #2680
PR #2669

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

PR #2651

PR #2652
PR #2661

PR #2680
PR #2669

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

PR #2651

PR #2652
PR #2661

PR #2680
PR #2669

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

PR #2651

PR #2652
PR #2661

PR #2680
PR #2669

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

PR #2651

PR #2652
PR #2661

PR #2680
PR #2669

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

PR #2651

PR #2652
PR #2661

PR #2680
PR #2669

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

PR #2651

PR #2652
PR #2661

PR #2680
PR #2669

This comment was auto-generated by Graphite.

Current dependencies on/for this PR:

develop

PR #2642

PR #2644   👈

PR #2651

PR #2652
PR #2661

PR #2680
PR #2669

This comment was auto-generated by Graphite.


JDA throws random stream closed IOException

General Troubleshooting

[X] I have checked for similar issues on the Issue-tracker.
[X] I have updated to the latest JDA version
[X] I have checked the branches or the maintainers' PRs for upcoming bug fixes.

Expected Behaviour
I've been trying to resolve this problem for days now, I looked Stackoverflow, pull requests, issue reports, but couldn't get quite satisfied answer from them. So I built a bot which provides information about one mobile game.
For some commands, it needs to generate image files to improve visuals. Since these generated files don't need to be cached or stored, meaning that they are just temporary files, I delete them whenever bot could send message successfully. Now problem here appears, I really can't give good example to reproduce this, but when it gets rate-limited by Discord, if message sent contains file, JDA randomly throws IOException with stream closed error.
I can't get exact way to reproduce this problem, stack trace gives me something related with RateLimiter, so excuse me that above statements are purely guessing, and my lack of deep knowledge about JDA structure or Discord data... Code example below is one of basic format that I use in my codes; I send message (sendMessage) with file (addFile), and call queue() message. Since queue message can provide listener for when message successfully got sent, or failed to, I add file deletion methods in there, so that bot can delete temporary files only when message sending task is fully done.
This error happens also when I obtain Message instance by calling complete() as well. I thought it could be problem related with these temporary files, so I made bot delete them with 5 seconds delay. It could be just that 5 seconds weren't enough, but such delayed deletion couldn't solve problem.
My bot is open source
Adapter that I implemented is here
One of command which caused this problem is here
or here from line 225
I... can't suggest or provide what should happen because I actually don't know if I'm using API incorrectly, but I can guess that such problem shouldn't be appeared, or handled by JDA API itself. I will wait for answer, thanks for reading these
Code Example for Reproduction Steps
//Use this method in "onMessageReceived" for "ListnerAdapter" class
public void tryToReplicate(GenericMessageEvent event) {
    MessageChannel ch = event.getChannel();

    //This is merely example
    //My actual code doesn't spam command like this, this is just for trying to replicate problem
    //Basically spamming message with file to trigger rate-limit
    for(int i = 0; i < 20; i++) {
        File f = generateFile(); //Generate any temporary file

        ch.sendMessage("File")
            .addFile(f, "Example.txt")
            .queue(m -> {
                if(f.exist() && !f.delete()) {
                    System.out.println("failed to delete file : "+f.getAbsolutePath());
                }
            }, e -> {
                e.printStackTrace();

                if(f.exist() && !f.delete()) {
                    System.out.println("failed to delete file : "+f.getAbsolutePath());
                }
            });
    }
}

Code for JDABuilder or DefaultShardManagerBuilder used
JDABuilder builder = JDABuilder.createDefault(TOKEN);

builder.setEnabledIntents(
        GatewayIntent.GUILD_MEMBERS, GatewayIntent.GUILD_EMOJIS, GatewayIntent.GUILD_MESSAGES,
        GatewayIntent.GUILD_MESSAGE_REACTIONS, GatewayIntent.DIRECT_MESSAGES, GatewayIntent.DIRECT_MESSAGE_REACTIONS
);
builder.disableCache(CacheFlag.VOICE_STATE);
builder.setActivity(Activity.playing(develop ? dev : normal));
builder.addEventListeners(new AllEventAdapter());

JDA client = builder.build();

//Check my github source link below, line 48 for detailed further codes
//https://github.com/battlecatsultimate/PackPack/blob/main/src/main/java/mandarin/packpack/PackBot.java

Exception or Error
net.dv8tion.jda.api.exceptions.ErrorResponseException: -1: java.io.IOException
	at net.dv8tion.jda.api.exceptions.ErrorResponseException.create(ErrorResponseException.java:185)
	at net.dv8tion.jda.api.requests.Request.onFailure(Request.java:117)
	at net.dv8tion.jda.internal.requests.RestActionImpl.handleResponse(RestActionImpl.java:273)
	at net.dv8tion.jda.api.requests.Request.handleResponse(Request.java:259)
	at net.dv8tion.jda.internal.requests.Requester.execute(Requester.java:255)
	at net.dv8tion.jda.internal.requests.Requester.execute(Requester.java:142)
	at net.dv8tion.jda.internal.requests.Requester.execute(Requester.java:125)
	at net.dv8tion.jda.internal.requests.ratelimit.BotRateLimiter$Bucket.run(BotRateLimiter.java:481)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: java.io.IOException: Stream Closed
	at java.base/java.io.FileInputStream.readBytes(Native Method)
	at java.base/java.io.FileInputStream.read(FileInputStream.java:273)
	at okio.InputStreamSource.read(JvmOkio.kt:90)
	at okio.Buffer.writeAll(Buffer.kt:1642)
	at okio.RealBufferedSource.readByteArray(RealBufferedSource.kt:236)
	at net.dv8tion.jda.internal.utils.BufferedRequestBody.writeTo(BufferedRequestBody.java:60)
	at okhttp3.MultipartBody.writeOrCountBytes(MultipartBody.kt:157)
	at okhttp3.MultipartBody.writeTo(MultipartBody.kt:93)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at net.dv8tion.jda.internal.requests.Requester.execute(Requester.java:202)
	... 9 more
Caused by: net.dv8tion.jda.api.exceptions.ContextException
	at net.dv8tion.jda.api.exceptions.ContextException.here(ContextException.java:54)
	at net.dv8tion.jda.api.requests.Request.<init>(Request.java:71)
	at net.dv8tion.jda.internal.requests.RestActionImpl.queue(RestActionImpl.java:198)
	at mandarin.packpack.commands.data.StageImage.handleLast(StageImage.java:126)
	at mandarin.packpack.commands.data.StageImage.doSomething(StageImage.java:83)
	at mandarin.packpack.commands.ConstraintCommand.lambda$execute$1(ConstraintCommand.java:129)
	... 1 more

You should store your images in memory instead of creating temporary files and having to manage them. The exception points to a FileInputStream being closed, so probably your file got deleted or whatever.
For that you can use a ByteArrayOutputStream, tell ImageIO to write in it, get the byte array from the ByteArrayOutputStream and pass the byte array to JDA
If it doesn't solve the issue i invite you to join the discord server

Is it not safe to delete file in lambda funtion of queue? Doesn't it mean that when this lambda function is called, it's fine to delete generated file because it means JDA has done sending message task successfully? It doesn't delete files outside of this lambda function. If it's related with caching, is there any way to listen that full task including this caching is done?
Also I don't prefer sending raw byte data because usually this gives critical effect on memory side, and the hosting server that I'm running doesn't have that large RAM size because I'm developing/hosting this bot just for experiences as developer, with the fact that image usually has large file size, not like 1~10KB just like usual text files, etc.

Multiple things here:

Your stacktrace does not match the latest version
We always read the entire file into memory, before sending it. Otherwise, retrying is impossible.
The file can be safely deleted after the request is done in the queue callback, since we already made a copy into memory.

Okay, for the first point, I now noticed that alpha 13 was released 10 days ago, let me use it, and I will notice you if problem persists as soon as possible.
For third point, so it means that it's mostly safe to delete file as long as deletion is done in callbacks, noted. Is it also safe to delete file after calling complete() as well? For example,
File f = generateFile();

try {
    Message m = ch.sendMessage("File")
        .addFile(f, "file.txt")
        .complete();

    //Doing something...
} catch(Exception e) {
    e.printStacktrace();
} finally {
    if(f.exists() && !f.delete() {
        System.out.println("Failed to delete file");
    }
}

Is it also safe to delete file after calling complete() as well?

The complete() method essentially just blocks the current thread until the queue callback is run. So doing something after complete() returns, is identical to doing something in the queue callback, in terms of timing.
I do not think that the issue here could be caused by the delete in the queue callback, unless the files share the same name. This could be possible, if you have an unsafe way of generating unique filenames. I saw in your code, that you simply have some static method which loops until a file that doesn't exist is created, which is in theory unsafe.
As it was already recommended, you should simply store the file in memory using a byte[], since that is ultimately what JDA will do either way. If you don't want to do it that way, you should use a more appropriate way of generating temp files, such as Files.createTemporaryFile(...).

I honestly felt that way, it's unlikely that my bot is in that many servers, so threads that I created caused conflicts among file names, but it's theoretically still possible. I know Files.createTempFile() method, but I need to store files in this temp folder though, so that I can manage files which got generated by bot easily for debugging, or whatever purposes are. It's just my preference, I will try to improve safety of file handling instead.
For now, bot stopped sending stream closed error after I updating API. But since there are still problem that there can be name conflict among files due to how I'm generating temp files, I will let you know after solving this problem

I know Files.createTempFile() method, but I need to store files in this temp folder

The method I linked lets you specify a directory as the first parameter, that should work. Files.createTempFile(Paths.get("./temp"), "image", ".png")
Which creates a file like this:
jshell> Files.createTempFile(Paths.get("./temp"), "image", ".png")
$1 ==> ./temp/image6700668970453732818.png

I think this is the bug that was fixed here. We used to close the stream too early in an older version, this shouldn't be happening anymore on the current release.


Do not rewrite on all possible destinations

Capturing from https://github.com/mapbox/directions-feedback/issues/575#issuecomment-312885363 and https://github.com/mapbox/directions-feedback/issues/575#issuecomment-312891683:
We currently rewrite exit_to tags into destination tags, if no destination tag is already present (see here).
OSRM treats destination and destination:ref as a combined field, just as we did a while back with name and ref.
This results in combinations as this:

It combines this node and this way into CA 82: SR-82 as an announcement.
We have multiple choices: handling destination and destination:ref separately in OSRM or prevent rewriting on all destination* flags.
For now, the simple fix would be to don't rewrite, if any destination is already present on the target way.
/cc @daniel-j-h @willwhite

Urgh, technically this is a limitation in osrm and our fault for combining these two tags.
Not re-writing exit_to tags if there is any destination* tag on the way present seems a conservative way to handle this but if it will help with our guidance instructions I can run this out.

Urgh, technically this is a limitation in osrm and our fault for combining these two tags.

In an ideal world, how would OSRM's behavior change?

Instead of strings have Json arrays in the API for properties such as ref, destination, exits
Instead of combining destination, destination:ref into a single destinations response have dedicated properties in the response to let the user decide what to use

=> https://github.com/mapbox/rewrite-exit-destination-signage/pull/5

Seems like we could do point 2 without a major API change since we'd just be adding a field.

This landed with https://github.com/mapbox/rewrite-exit-destination-signage/pull/5 - downstream integration is tracked elsewhere.

@willwhite in a perfect world yes - but passing new fields through the full osrm pipeline increases memory and disk requirements quite a bit.


Latest version of smocks nuget package 0.4.15 not working

I downloaded the latest Smocks nuget package 0.4.15 and tried to run the tests.
However, even the existing tests (which were working properly with previous version of smocks) also stopped working and the below error message is displayed.
"An exception of type 'System.NotSupportedException' occurred in Smocks.dll but was not handled in user code
Additional information: An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information."
Kindly check and resolve it asap.
Thanks

I cannot seem to reproduce this issue. It sounds as if the assemblies appear to be loaded from a network (or otherwise "insecure") location. Nuget should put the assemblies in a "packages" directory in the solution directory, which is typically a secure location. How did you update Smocks to the 0.4.15 version? If you select the "Smocks" assembly reference in visual studio, does its "path" property point to a local directory? Is your solution on a local disk?
Please try to reinstall Smocks by removing the nuget package and any remaining references and installing it using the nuget package manager. If that does not help, create a new solution and test if Smocks works there: this works for me. Updating smocks ("Update-Package Smocks" in the package manager console) from 0.4.5 to 0.4.15 also seems to work correctly.

its working now...thanks :)

@jeetendrajaiswal: in case anyone runs into the same problem: what did you do to resolve the issue?

@jeetendrajaiswal I am experiencing the same issue. I tried adding this to my app.config to no avail:
 <runtime>
    <legacyCasPolicy enabled="true" />
    <loadFromRemoteSources enabled="true"/>
  </runtime>

Perhaps one or both these properties need to be added in the project? Any other workarounds?

I remove all the existing references added by smocks and then added the smocks through nuget and the issue was resolved.


Add diagram of workflows

Following on from #361 and the discussion last night's call, here is a PR that adds a diagram of the platform to service broker workflows to the repository.
The diagram is a Google Drawing hosted in the OSBAPI Google Drive folder which all PMC members should be able to edit. The new diagram.md file automatically renders the live version of the linked drawing. I'm hoping this is a sensible approach as it means nobody has to learn any new 'drawing languages'.
I also updated CONTRIBUTING.md to ensure the workflow diagram is updated on a spec release (which I presume is the most sensible time that we will do this).
Feedback is welcome!

you could use a relative URL for the diagram so that Travis doesn't fail?

LGTM

LGTM

LGTM

LGTM

LGTM

LGTM


nuttx-apps/examples/: Fix mistakes in comments

Summary
Fix mistakes in comments
Impact
none
Testing

let's fix nxstyle warning in pwm.h


Added Spotify authentication

Added Spotify (oauth) authentication including test and sample code

Hi @jeroenslor I think the focus for this repo is to just have a few very common authentication providers that are most commonly used on web site. I think a Spotify provider would be great to ship as your own NuGet package.

There's also a simpler way to implement providers now, see:
https://github.com/aspnet/Security/blob/dev/samples/SocialSample/Startup.cs#L116
50 lines vs 15 files.

Hi @Eilon Ok I will create my own nuget, thanks for the quick feedback. Is there any other provider I can help you with that is commonly used, maybe linkedin?

@Tratcher yeah my first attempt was to only create an extensions method that hides the code you just showed but it turned out that Spotify had some extra auth options. To support that I had to create a custom options class and override a methode on the handers see: https://github.com/jeroenslor/Security/blob/SpotifyAuth/src/Microsoft.AspNet.Security.Spotify/SpotifyAuthenticationHandler.cs#L72
I do really like the pattern you guys used with the notifications, it's pretty flexibel and avoids extra implementations!

@jeroenslor I don't think there are currently any other auth providers that we are looking to add outside of Active Directory related scenarios, but for those we have works in progress.

FYI: Here is a related community project for Katana. I hear they plan something similar for ASP.NET 5.
https://github.com/owin-middleware/OwinOAuthProviders/tree/master/Owin.Security.Providers


Switch from _third_party to vendor.

This switches all dependencies that were versioned with party to be
instead included as submodules. The respective projects are checked out
to the same commit at which they were initially vendored with party, so
this should have no functional change.

Thanks


[Ignore - Do Not Merge] Testing CI with Publishing API for Schemas

⚠ This repo is Continuously Deployed: make sure you follow the guidance ⚠
Follow these steps if you are doing a Rails upgrade.

Works 🎉


LLVM Caching save-state deprecated

I'm getting a warning saying:
The `save-state` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Deprecation Notice
From my understanding this is something that needs to be fixed as part of the action, and not something I can configured.
Though I could be wrong.

Context of my build action in case it influences this
# ....
jobs:
  build-linux:

    runs-on: ubuntu-latest
    timeout-minutes: 10

    steps:
      - uses: actions/checkout@v3

      # ...

      - name: Cache LLVM and Clang
        id: cache-llvm
        uses: actions/cache@v2
        with:
          path: "./lib/install"
          key: llvm-14.0

      - name: "Install LLVM and Clang"
        uses: KyleMayes/install-llvm-action@v1
        with:
          version: "14.0"
          directory: "./lib/install"

      - name: Symlink libclang.so (Linux)
        run: sudo ln -s libclang-14.so.1 /lib/x86_64-linux-gnu/libclang.so
        working-directory: ${{ env.LLVM_PATH }}/lib
      # ....

I'm pretty sure this was fixed at some point when I upgraded the @actions/core version used by this action.

Is that change live?
Because it doesn't appear to have completely fixed the issue.
Previously my Github actions wouldn't even load from cache, which they are at least now, however I'm still getting the deprecation warning. (action log)

Oh, maybe this is caused by the caching.
I'll take another look.

The issue specifically appears to be in relation to using actions/cache@v2, if you replace it with actions/cache@v3 it appears to work perfectly.
However it appears that install-llvm-action isn't properly detecting when the cache is found with v3. i.e. 165 -> 166

It appears v2 was also possibly not detecting the cache load correctly either 164

Looking at your workflow file (https://github.com/qupa-project/uniview-lang/actions/runs/4320715678/workflow), you are missing the cached option for this action.
I updated the example in the README.md to use v3, it now looks like this:
- name: Cache LLVM and Clang
  id: cache-llvm
  uses: actions/cache@v3
  with:
    path: |
      C:/Program Files/LLVM
      ./llvm
    key: llvm-3.5
- name: Install LLVM and Clang
  uses: KyleMayes/install-llvm-action@v1
  with:
    version: "3.5"
    cached: ${{ steps.cache-llvm.outputs.cache-hit }}

The cached option is necessary to inform this action that it should use the cached values.
I added a workflow to this repository to check the caching and it seems to work fine with actions/cache@v3. For example, you can see this action using the cache here: https://github.com/KyleMayes/install-llvm-action/actions/runs/4548947844/jobs/8020511525

Confirmed this solved the problem on uniview-lang/action-#173


Using gnu = true; fails due to services.xserver.startGnuPGAgent no longer being an option.

Describe the bug
When I set gnu = true; in configuration.nix, the rebuild fails as services.xserver.startGnuPGAgent is no longer a valid option.
Steps To Reproduce
Steps to reproduce the behavior:

Set gnu = true;
sudo nixos-rebuild switch
Error thrown:

Failed assertions:
- The option definition services.xserver.startGnuPGAgent' in /nix/store/bfg7xzgvbhxhs7vnl7lynaxmjhmvc4j0-nixos/nixos/nixos/modules/config/gnu.nix' no longer has any effect; please remove it.
See the 16.09 release notes for more information.

Expected behavior
gnu = true; should not fail.
Screenshots
-
Additional context
-
Notify maintainers
@ajs124
@pennae
@prusnak
@Ericson2314
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.55, NixOS, 23.11 (Tapir), 23.11pre530560.f5892ddac112`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - channels(ashvith): `""`
 - channels(root): `"nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Having had a look at that module (https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/config/gnu.nix), IMO it should just be removed.
lsh? really?

Other question is when you look at the history they removed lsh 7 years ago because it was unmaintained, but then the commit got reverted later? 🤔

This module doesn't make sense at all. GNU stuff and non-GNU ones usually can't fully replace each other, and it's up to the users to decide which one to use.


Changing the Batch-Size does not show effect when it is not in the TensorNet.cpp

Hello,
When changing:
MAX_BATCH_SIZE
in any of the examples it will not have an effect, it has to be changed in the TensorNet.cpp to have an effect.
Greetings

Thank you, I have modified the TensorNet class and vision primitive APIs to accept the MAX_BATCH_SIZE as an argument, so it should take effect more easily now.

Changing the batch size in tensor net.h alone doesnt worked.Please may i know the locations to  change batch size.

Like written above in TensorNet.cpp

I have changed in tensorNet.h  --> LINE:bool LoadNetwork( const char* prototxt, const char* model, const char* mean=NULL,
const char* input_blob="data", const char* output_blob="prob",
uint32_t maxBatchSize=1 );.Please point me to the locations if iam wrong.Also,i have recompiled.But,i think few more locations are yet to be changed.Please help me.

Sry I was mistaken, that was on an older version.
Anyhow, I think this should do the trick since you are changing the default parameter of the create function. Anytime the create function is called without specifying this statement the default is used. If you wish to specify an other Batchsize you must provide it.
e.g. line 33 of imagenet-console.cpp
// create imageNet
imageNet* net = imageNet::Create(argc, argv);

You could change to
 imageNet* net = imageNet::Create(argc, argv,maxBatchSize=4);
Anyhow remember, that the batchsizes always must be dividable by 2. So setting it to "1" will not make any change

Thankyou.I tried it but haven't succeeded.error: 'maxBatchSize' was not declared in this scope
detectNet* net = detectNet::Create(argc, argv,maxBatchSize=1);
^

I am not quite sure if it possible in this C++ version to write it like this. maybe you have to write evey function argument, till you come to the maxbatchsize argument.
e.g.:
detectNet* net = detectNet::Create(argc,argv,maxBatchSize=1,NULL,NULL,IMAGENET_DEFAULT_INPUT,IMAGENET_DEFAULT_OUTPUT,4);
I am not sure if it will work with the NULL operators, though.
P.S.: Please use the formatting tools for code when you submit code, to make it easier to read

Yes i got it changed when i modified it in detectNet.h.Thankyou for your effort and time.


drop external_doc feature gate usage

The feature no longer exists on nightly.

I've stumbled on this when trying the simd_backend feature of curve25519-dalek, which is currently fails to compile.

Note that has of now, the new doc = include_str!("path") syntax does not work in stable, even when hidden behind features, so this patch does not compile on stable (1.53). However it does compile on beta (1.54), which will be promoted to stable on 29th of July, at which point using cfg_attr won't make much sens as the feature will be entirely stable.

I suppose I should include this patch for releases of 2.3.x and 2.2.x since people seem to still be using those.

C.f. #86 where I describe and fix an issue, also note that currently I've only patched this to the 2.4.x series (but I'm perfectly happy to go back to other versions if anyone needs it).


set_index datetime in dataframe composed of multiple HDF file keys fails

Hi,
I am unable to set a datetime  index in a dask dataframe, that I have read out of multiple pandas dataframes,  originally saved in a single HDF file, under different keys.
The pandas dataframes contain same fields, are mostly categoricals but a couple of fields, including a date field.
`df = dd.read_hdf(store_file,"/ISE_week_22")
pddf=pd.read_hdf(store_file,"/ISE_week_22")
df2=dd.from_pandas(pddf,npartitions=10)
pddf.date.head()
0   2017-06-01 12:52:54.329
1   2017-06-01 12:52:56.280
2   2017-06-01 12:52:56.306
3   2017-06-01 12:52:57.317
4   2017-06-01 12:52:58.320
Name: date, dtype: datetime64[ns]
df2.date.head()
0   2017-06-01 12:52:54.329
0   2017-06-02 17:02:54.352
0   2017-05-28 00:00:08.493
0   2017-05-28 00:00:08.494
0   2017-05-28 00:00:00.131
Name: date, dtype: datetime64[ns]
df2.set_index(df2.date)
df2.known_divisions
True
df = dd.read_hdf(store_file,"/ISE_week_2*")  <---- see the asterisk
df.known_divisions
False
df.set_index(df.date)

KeyError                                  Traceback (most recent call last)
 in ()
----> 1 df.set_index(df.date)
C:\Users\U91437\Documents\xx\Program Files\Anaconda\lib\site-packages\dask\dataframe\core.py in set_index(failed resolving arguments)
2297             from .shuffle import set_index
2298             return set_index(self, other, drop=drop, npartitions=npartitions,
-> 2299                              divisions=divisions, **kwargs)
2300
2301     @derived_from(pd.DataFrame)
C:\Users\U91437\Documents\xx\Program Files\Anaconda\lib\site-packages\dask\dataframe\shuffle.py in set_index(df, index, npartitions, shuffle, compute, drop, upsample, divisions, **kwargs)
61         mins = [ipart.min() for ipart in iparts]
62         maxes = [ipart.max() for ipart in iparts]
---> 63         divisions, sizes, mins, maxes = base.compute(divisions, sizes, mins, maxes)
64         divisions = divisions.tolist()
65
C:\Users\U91437\Documents\xx\Program Files\Anaconda\lib\site-packages\dask\base.py in compute(*args, **kwargs)
201     dsk = collections_to_dsk(variables, optimize_graph, **kwargs)
202     keys = [var._keys() for var in variables]
--> 203     results = get(dsk, keys, **kwargs)
204
205     results_iter = iter(results)
C:\Users\U91437\Documents\xx\Program Files\Anaconda\lib\site-packages\dask\multiprocessing.py in get(dsk, keys, num_workers, func_loads, func_dumps, optimize_graph, **kwargs)
85         result = get_async(pool.apply_async, len(pool._pool), dsk3, keys,
86                            get_id=_process_get_id,
---> 87                            dumps=dumps, loads=loads, **kwargs)
88     finally:
89         if cleanup:
C:\Users\U91437\Documents\xx\Program Files\Anaconda\lib\site-packages\dask\async.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, dumps, loads, **kwargs)
523                     _execute_task(task, data)  # Re-execute locally
524                 else:
--> 525                     raise(remote_exception(res, tb))
526             state['cache'][key] = res
527             finish_task(dsk, key, state, results, keyorder.get)
KeyError: 'date'
Traceback
File "C:\Users\U91437\Documents\xx\Program Files\Anaconda\lib\site-packages\dask\async.py", line 291, in execute_task
result = _execute_task(task, data)
File "C:\Users\U91437\Documents\xx\Program Files\Anaconda\lib\site-packages\dask\async.py", line 272, in _execute_task
return func(*args2)
File "C:\Users\U91437\Documents\xx\Program Files\Anaconda\lib\site-packages\pandas\core\series.py", line 601, in getitem
result = self.index.get_value(self, key)
File "C:\Users\U91437\Documents\xx\Program Files\Anaconda\lib\site-packages\pandas\indexes\base.py", line 2169, in get_value
tz=getattr(series.dtype, 'tz', None))
File "pandas\index.pyx", line 105, in pandas.index.IndexEngine.get_value (pandas\index.c:3567)
File "pandas\index.pyx", line 113, in pandas.index.IndexEngine.get_value (pandas\index.c:3250)
File "pandas\index.pyx", line 163, in pandas.index.IndexEngine.get_loc (pandas\index.c:4373)
`

apologies, ... python 3.5.2, pandas 0.19 dask 0.14
using multiprocessing scheduler

Sorry about missing this. Is it still an issue?
If so, are you able to make a reproducible example? (http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports)
You could create a representative DataFrame in memory with and write to HDF5 with pandas, and then read it in with dask.dataframe.

Thanks Tom for getting back into this.
Unfortunately this is already "overtaken by events".
You can close this if you wish so. I do not have the materials or can reproduce this anymore
JC


Question about server behind load balancer

Hi, first of all thanks for the gem!  It is really useful. I can't find where I can ask question except here.
I'm not sure how to perform SMTP validation when server is behind load balancer with internal IP only.
Truemail.host_audit always fails because server doesn't have public IP and I can't configure one.
Any suggestions?

Hi, Nikola! You're welcome. What version of Truemail you have used? From 0.1.8 and higher I have used third party service for detect IP address instead Resolv.getaddress(Socket.gethostname), and it should work in any case. Could you provide some your error details, what Truemail.host_audit has returned? This feature is for quick host diagnostic only.
You can check it manually, just run this one in bash:
dig -x you_ip_address

Hi Vladislav,
I'm currently using 0.1.10 version in production environment.
Truemail.host_audit returns:
<Truemail::Auditor:0x0000563a72defdc0 @result=#<struct Truemail::Auditor::Result warnings={:ptr=>"ptr record does not reference to current verifier domain"}>>

curl ifconfig.co/json returns:
{"ip":"<IP_ADDRESS>","ip_decimal":600022403,"country":"United States","country_eu":false,"country_iso":"US","hostname":"<IP_ADDRESS>.bc.googleusercontent.com","latitude":47,"longitude":8}

Dig output:
$ dig -x <IP_ADDRESS>

; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> -x <IP_ADDRESS>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49994
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;<IP_ADDRESS>.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
<IP_ADDRESS>.in-addr.arpa. 119 IN	PTR	<IP_ADDRESS>.bc.googleusercontent.com.

;; Query time: 4 msec
;; SERVER: <IP_ADDRESS>#53(<IP_ADDRESS>)
;; WHEN: Wed Jun 05 07:27:02 UTC 2019
;; MSG SIZE  rcvd: 109

My Truemail config:
Truemail.configure do |config|
  config.verifier_email =<EMAIL_ADDRESS>end

I know my verifier domain doens't match PTR record of the instance that I use, Instance officially doesn't have public IP in Google Cloud Console, however when I run curl ifconfig.co/json I get public IP that could be an address of NAT gateway.

Hi Nikola,
Google Cloud instances are blocked from using outgoing SMTP connections (see their documentation) so it is impossible to use SMTP validation directly. You can only use :regex and :mx type validation.

Thank you for you answer. Is it possible to use other ports, I guess most major providers supports 587 and 465? If so I can send pull request to add SMTP port configurable?

No, port 587 is used for client mail submission and usually requires authentication and port 465 is non-standard port for encrypted SMTP.  Most major providers (eg. gmail, outlook, yahoo) don't support those ports on their MX servers.

Ok, thank you very much for the explanation!

Hello, @majksner! Sorry for delay with answer. As @AlexeyDemidov mentioned, for your case you can't use smtp validation. But as alternate way you can use truemail gem on a different host with white public IP and delegate email validation to this service.
Most major providers supports 587 and 465, but what would if you can try to connect? For instance, gmail case:
nc alt3.gmail-smtp-in.l.google.com 587 # nothing
nc alt3.gmail-smtp-in.l.google.com 465 # nothing

@bestwebua Can you elaborate more on "use truemail gem on a different host with white public IP and delegate email validation to this service."

@bestwebua You mean to run truemail as a microservice on  another provider and run email validations though an API?

Got it 👌

@majksner, yes! I plane to realize example app for this case when I will have free time. Cheers!

Hello, @majksner! Now you can use Truemail as independent microservice. Also this app now available as docker image :whale: Cheers!

Thank you so much for letting me know.

Thank you so much for letting me know.

You're welcome, @majksner!

@majksner Also the first version of Truemail web API client library for Ruby was released just now 🚀


fix(nodes): add ngInject to node-select

fix #4182

@ncresswell can you review this one to check if it solves the issue with the empty node selector?


[AMORO2795][helm chart] add variables and descriptions into values.yaml

Why are the changes needed?
Fix #2795
Close #2795
Brief change log

Add variables existed in templates but on in Values.yaml. Those variables are: nameOverride, commonLabels and commonAnnotations.
Change Values.yaml: add more descriptions, refs and modify the file with standard format which can seems pretty.

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[x] Add screenshots for manual tests if appropriate

[x] Run test locally before making a pull request

test with unittest(#2794)
This change is not dependency by #2794 , Just use it as test.

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented?  none

@baiyangtx Another pr, Could you please help to review? Thanks.


$baseUrl problem with Router

I have function like this:
public function testpage()
{
echo 'this is a test page';
}
And added a route like this:
$routes->add('testpage', 'Home::testpage');
When I leave $baseUrl blank, then I access http://ci4.local/testpage, it show the page.
But when I set $baseUrl='http://ci4.local/', then I access http://ci4.local/testpage, it show default CI4 welcome page.
Even I try random uri like http://ci4.local/fdgfdgfdg, it also show default CI4 welcome page instead of 404 error page.
So I think this is a bug that must be solved as soon as possible.

What does your routes file look like, including the various settings in routes.php?

I use default router settings, then I added this line:
$routes->add('testpage', 'Home::testpage');

I can confirm that setting $baseUrl = ''; and leave it to automatically find the domain path works good. Setting to anything else, for example $baseUrl = 'http://ci4.local/ only shows the default homepage on every page.
Also note on routing that absolute paths doesn't work correctly.
Example:
Code in Routes.php:
$routes->get('/profile/change-email', 'Profile::changeEmail', ['as' => 'change-email']);

Code in a view file:
<a href="<?= route_to('change-email') ?>">Change Email

When clicked on link, shows:
Call to undefined method App\Controllers\Profile::change-email()

It uses the as name for controller method, although it should use the Profile::changeEmail

I currently use URLs without leading slashes, and hard-code them in views, because if I open /profile/ then click on link, it goes to /profile/profile/change-email.
Current code that works for me:
Code in Routes.php:
$routes->get('profile/change-email', 'Profile::changeEmail', ['as' => 'change-email']);

Code in a view file:
<a href="/<?= route_to('change-email') ?>">Change Email

Works as indented, but I don't know which one should be fixed, the Routing or the route_to() function.

The OP's problem should now be fixed. Thanks for reporting it.
@DavisMiculis could you open a new issue for the other problem you're seeing? Too hard to keep things straight with multiple issues in the same post.

You're welcome.


can't use dot notation only bracket notation for setting var to lookup value?

I don't remember being told I can't use dot notation but only bracket notation works for the line where i'm setting the value to result:
works
result = lookup[val];
doesn't work
result = lookup.val;
since there aren't any spaces in the potential values passed to 'val', why isn't dotted notation accepted?
Challenge Using Objects for Lookups has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:

// Setup
function phoneticLookup(val) {
  var result = "";

  // Only change code below this line
  var lookup = {
    "alpha": "Adams",
    "bravo": "Boston",
    "charlie": "Chicago",
    "delta": "Denver",
    "echo": "Easy",
    "foxtrot": "Frank"
  };
  result = lookup[val];

  // Only change code above this line
  return result;
}

// Change this value to test
phoneticLookup("charlie");

ok, someone explained that "brackets evaluate variables into keys but dot notations take the variables as actual keys without evaluating them"


Provide a way to select adapter

Currently the adapter is hardcoded to the first available one. In my case this doesn't work, since the integrated GPU shows up before the discrete. It would be helpful to provide some way of selecting a different adapter.

I guess it is possible to add FnOnce to the init function that will receive adapters to choose from.
Would this work for your case?

Not sure if I should open a new issue for this, but it looks like the members for AdapterByName and AdapterByNameRe are private. Is there currently a way to instantiate those structs that I'm missing?

Should be OK now 😄


CDK-989: HttpMetricsMorphlineTest assumes only ConnectionException

The documentation of URLConnection.getInputStream() is that it may throw IOException. Which derived IOException which are thrown though is somewhat platform dependent. For some of the platforms a closed socket will instead throw SocketException.

Thanks for the fix!


Test: virtual video/screen

A basic full-range screen test with dub test -b screen.

Fully functional in Windows and Linux.
Cursor is placed at the end for unittest message.


Status of support for Input Arguments

Hello,
I'm using the Actions SDK to extend the functionality of my Python Assistant SDK based app.
I would like my application to provide some arguments (e.g. a unique_id) to a conversation. Therefore allowing my custom Action webhook to provide information tailored specifically to a particular assistant session.
This input argument functionality is supported by the Actions SDK for Google Assistant. Thus, I would also expect this functionality to be supported by the Google Assistant SDK itself.
Can said argument functionality be used within assistant-sdk-python?
Best Regards,

That exact API wouldn't work for Actions on Google actions, but you can now use Device Actions for on-device fulfillment.

You can use custom device actions now.


Some features for your consideration

[ ] PDF Viewer (Printing/Download option)
[ ] HTML Printing
[ ] Notification
[ ] Embed Fonts
[x] Role Claims (Permission)
[ ] JWT Refresh Token and Revoke Token
[ ] HttpInterceptor

Others nice to have features:

Audit Trai: https://codewithmukesh.com/blog/audit-trail-implementation-in-aspnet-core/?utm_source=csharpdigest&utm_medium=email&utm_campaign=344
Console Project: https://www.devlead.se/posts/2021/2021-01-15-my-preferred-console-stack?utm_source=csharpdigest&utm_medium=email&utm_campaign=346

Additional Authentication options eg like Azure AD.  Authorization still kept in db - just the auth can be outsourced to externals


404 at: /userguide/dockerlinks/

URL: /userguide/dockerlinks/
Brought by google search. Was expected documentation about linking containers.

Thanks for reporting! In the meantime use https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/

Fixed in bd630e9e3bed19c0f1f4c80166fff61536d3dc57
Thanks!


[v1.13] Connection disruption on agent restart due to XfrmInStateProtoError errors when IPSec per-tunnel keys are enabled

Is there an existing issue for this?

[X] I have searched the existing issues

Version
higher than v1.13.18 and lower than v1.14.0
What happened?
Discovered while attempting to backport https://github.com/cilium/cilium/pull/33769 to v1.13 and v1.14. The Cilium IPsec upgrade tests failed due to connection disruption and increase of XfrmInStateProtoError errors, and more specifically:

On the v1.14 branch, when downgrading from v1.14 (tip) to v1.13.18 (e.g., https://github.com/cilium/cilium/actions/runs/9939762286/job/27454998914)
On the v1.13 branch, when upgrading from v1.13.18 to v1.13 (tip) (e.g., https://github.com/cilium/cilium/actions/runs/9943013136/job/27465833835)

Given the latter failure, and the fact that the tip of v1.13 practically matches v1.13.18, this issues is most likely happening on agent restart.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Not very resultful, some interesting facts:

XfrmInStateProtoError isn't transient, I can see it keep going up after upgrade completes.
It's a general breakdown, connectivity is gone even across new pods.
ip x s on all nodes looks good I think? SPIs and aead keys are all right.
Dropped packets are carrying correct SPI.

Suspicious points:

ip x s shows last used time like lastused 2024-07-18 18:33:12, I see some of these were last used long time ago. This may imply the kernel did xfrm lookup and matched a wrong sa for decryption?

Actions I'll do tomorrow:

I can use bpftrace to see what xfrm policy/state is found for encryption and decryption, to see if they match my expectation.
I can compare the ip x s and ip x p with the original xfrm configures, to see if there is anything leftover / missing / changed slightly.
I can tcpdump the dropped esp packets and try to decrypt them in wireshark, to confirm the encryption key is what we expect.

Given the latter failure, and the fact that the tip of v1.13 practically matches v1.13.18, this issues is most likely happening on agent restart.

If this holds true and agent restart is sufficient to repro, it might be worth to roughly bisect the culprit (or see whether this affects even the v1.13 patch release that introduced support for the per-node-pair key system).

Thank you Julian, but restarting agent couldn't reproduce it 100%, so I didn't git bisect. Thank you all the same!
I haven't made good progress, will continue.
The direct reason is mismatching anti-play seq/oseq.
Consider pod1@host1 -> pod2@host2 via ipsec, the src host1 has an xfrm state:
src <IP_ADDRESS> dst <IP_ADDRESS>
	proto esp spi 0x00000003 reqid 1 mode tunnel
	replay-window 0 flag esn
	mark 0x6ec50d00/0xffff0f00 output-mark 0xd00/0xffffff00
	aead rfc4106(gcm(aes)) 0xbe0f550e9e3282395cec9b4efaee868fd056879f 128
	lastused 2024-07-19 22:14:18
	anti-replay esn context:
	seq-hi 0x0, seq 0x2bb8d7, oseq-hi 0x0, oseq 0x0
	replay_window 1024, bitmap-length 32
	ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
	ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
	ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
	ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
	sel src <IP_ADDRESS>/0 dst <IP_ADDRESS>/0

and dst host2 has an xfrm state:
src <IP_ADDRESS> dst <IP_ADDRESS>
	proto esp spi 0x00000003 reqid 1 mode tunnel
	replay-window 0 flag esn
	mark 0xa5a03e00/0xffffff00 output-mark 0xe00/0xffffff00
	aead rfc4106(gcm(aes)) 0xbe0f550e9e3282395cec9b4efaee868fd056879f 128
	lastused 2024-07-19 22:14:18
	anti-replay esn context:
	seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x2bb8d7
	replay_window 1024, bitmap-length 32
	00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
	00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
	00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
	00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
	sel src <IP_ADDRESS>/0 dst <IP_ADDRESS>/0

The key to this issue is the replay-seq at src host and replay-oseq at dst host. The above xfrm states are matching each other, src host has replay-seq 0x2bb8d7 and dst host has reply-oseq 0x2bb8d7, two number must be same.
What happened during upgrade is, node2 agent restarted and decided to replace xfrm states (by removing current ones and adding new ones). Unfortunately our cilium-agent always creates xfrm state with zero replay-seq/oseq, but the src host is still encrypting packets using non-zero replay-seq, that leads to dst host drops packets due to replay-seq mismatch.
Why we only see issue on 1.14 -> 1.13 downgrade? My guess is agent has a logic to check whether host already has the needed xfrm states, and do nothing if there are. For higher version agent doesn't replace xfrm states and we luckily never saw the issue. For 1.13, maybe something slightly different in code makes agent replace xfrm states, flush replay-seq/oseq.
(to be continued)

(I wasn't aware this could also cause XfrmInStateProtoError. We will need to update the IPsec troubleshooting guide to mention that.)

cc @rgo3 might be good for you to read this thread since it looks somewhat related to the BootID investigation you conducted some time ago.

This is the part I still don't understand: cilium config -a | grep -i ipam says it's kubernetes, but 1.13 agent says it's cluster-pool (by adding log). So maybe something in ipam mode settings makes a difference? That's everything I have for now, will continue tomorrow.

Beware of the following subtlety with respect to Cilium CLI and IPAM mode, which already confused me in the past.
TL;DR; Although the default IPAM mode configured in the values file is cluster-pool, the CIlium CLI configures the kubernetes IPAM mode for kind cluster. However, this setting takes only effect for the initial installation, and may get reset (depending on the additional flags) during subsequent upgrade/downgrade operations. Hence, you may end up with a cluster initially running in kubernetes IPAM mode transitioning into cluster-pool mode.
Although in most cases this does not lead to actual issues on Kind (as the PodCIDR field of CiliumNodes is not subsequently modified), it may definitely trigger different code paths. If you observe this issue, you may want to explicitly configure the IPAM mode to prevent these types of inconsistencies, as done for instance here for the clustermesh upgrade/downgrade workflow.

Not contributing to the root-cause investigation but I do recall that we had the following patch (which we never merged) that would ensure that we wouldn't update xfrm states if the triggering node update contained an empty bootid:
diff --git a/pkg/datapath/linux/ipsec.go b/pkg/datapath/linux/ipsec.go
index 27d2491aef..e85d409498 100644
--- a/pkg/datapath/linux/ipsec.go
+++ b/pkg/datapath/linux/ipsec.go
@@ -84,7 +84,7 @@ func (n *linuxNodeHandler) enableIPsec(oldNode, newNode *nodeTypes.Node, nodeID
 		}
 	}

-	if oldNode != nil && oldNode.BootID != newNode.BootID {
+	if oldNode != nil && oldNode.BootID != newNode.BootID && newNode.BootID != "" {
 		n.ipsecUpdateNeeded[newNode.Identity()] = true
 	}
 	_, updateExisting := n.ipsecUpdateNeeded[newNode.Identity()]

We never merged it because it wasn't clear if it was strictly necessary to include it, and the manual testing of it wasn't conclusive. But after reading through grays investigation it seems like it could have prevented the issue?

Fixed by https://github.com/cilium/cilium/pull/34035. Thank you all!

I'm pretty sure we are seeing the same root issue in 1.15.7 - every restart of the cilium daemonset results in XfrmInStateProtoError errors in all cilium agent pods. Could you backport this fix to 1.15?

I'm pretty sure we are seeing the same root issue in 1.15.7 - every restart of the cilium daemonset results in XfrmInStateProtoError errors in all cilium agent pods. Could you backport this fix to 1.15?

If I remember correctly the fix was not applicable to v1.15 - so please open a fresh issue with the relevant info, you're most likely seeing a different problem.


Update to v0.14.0-rc3

Backlinking to purescript/purescript#3942

Currently blocked by exists and likely some kind signature warnings, but I won't know what they are until exists is updated.

purescript-benchotron is not a core repo. By including it here, this repo will never pass CI because benchotron will pull in non-master dependencies.
This should be re-added later. Before merging this PR, we need to add an issue that tracks this.
Also, we will need to decide whether to merge #104 or not before or after this one.

I've opened issues in this repo and the benchotron one.


ModuleNotFoundError: No module named 'torch_directml'

I want to install and run Stable Diffusion on my PC but I always get this error.

I installed it a few times and it should be installed. It also says that its already satisfied:

I already looked up other issues on github but it didnt help.

You are running with sudo so it doesn't check your user specific packages. Running such scripts sudo is a bad idea in the first place.

Hi @lordboettcher, thank you for submitting this issue. While I can't provide a timeline for resolution as the moment, please know that your feedback is valuable to us. We will follow up once we can review this issue.


Create page object model for CreateDate/Timestamp

Create page object model for CreateDate/Timestamp

#567 Merged


Remove subtitle animation

Since this mod is about removing all fade out animations, I thought it would be cool if it also removed the subtitle animation of slowly turning darker when a sound stops.

https://github.com/user-attachments/assets/25d6f7e2-aeba-4b20-990c-08a9f1720ce2
managed to get it working

managed to get it working

You're the greatest. I hope it didn't bloat your minimal size mod jar too much 😅

currently working a bit of a bigger(ish) update for the mod that removes a couple of other fade out animations and adds a config.

its been added in release 2.0.0, you can get it here or on modrinth: https://github.com/DerpDerpling/Fadeless/releases/tag/2.0.0


Susy Debug Output and the Shadow Dom / Web Components

I just noticed that the way the debug toggle works as a pseudo element of the <head>  isn't working with the Shadow DOM.
I see multiple issues:
1.Toggle Trigger Missing
If Susy is used in a component the toggle trigger is created in the CSS inside the component and therefore doesn't get applied to the root DOMs <head> element. As there is no <head> element in the component the toggle trigger doesn't exist at all.
2. CSS from the root DOM doesn't pierce into the Shadow DOM
As intended the Shadow DOM is encapsulated and ignores outside styles. The problem is that even if we had the trigger on the <head> of the root element it wouldn't affect anything inside the components.
Thoughts
I see two approaches:
Either we add another trigger inside each component not attached to the <head> but to some other element. This keeps stuff encapsulated inside the component, but will create a lot of triggers at various places which is kind of messy.
The other approach is to somehow trigger even inside the Shadow DOM. As far as I know there is no way to affect the root DOM via CSS inside the component. This would also contradict the general flow of CSS where you can never select "up". So you need to manually create the trigger in the root document. So the usual CSS needs to be amended using the /deep/ (or >>> in the more recent not yet implemented draft) combinator roughly in this way:
head:hover ~ body /deep/ .in-shadow-dom {
  position: relative;
  &:before {
    @include grid-overlay-base;
    @include background-grid();
  }
}

Here is a link to a working example.
If this was incorporated in the overlay-grid mixin it would make things a bit easier, but I'm not sure if this should be a default as piercing in the Shadow DOM may be unexpected by a lot of people. Maybe just make it optional?
Finally I can also see how you might see all these issues as an edge case Susy shouldn't bother with at all. To show that I am not just theorizing here I just want to quickly talk about my real world use case:
I use the SC5 Styleguide Generator which encapsulates examples in Shadow DOM elements. I'd love to be able to see the grid in there, but unfortunately this doesn't work.
Looking forward to your thoughts. :smile:

I think this is a problem best solved outside of Susy. The real problem isn't that it's an edge case, but that we're getting too dependent on knowledge of the DOM. I'm already skeptical of the DOM antics needed for debug-overlay (your issue reinforces that skepticism), and I'm hesitant to take that even further. I'd rather give you the basics (in this case a grid background generator) and let you apply it to your situation (in this case the shadow dom) however you see fit.
For example, I'm never going to add JS to Susy, but you could probably solve your problem easily with a little bit of JS written specifically for your site. Inject a trigger anywhere you want, and tell it exactly what elements should have grid backgrounds applied. Make a class that shows the grid, and toggle that class wherever you want it. We'll never be able to write a universal sass-only mixin with that kind of flexibility.

First of all thanks for the fast reaction. I think you're spot on - this is probably nothing Susy should be solving all on its own. Especially not using JS. That isn't what I was trying to achieve here at all.
As I've shown with the linked CodePen it is already somehow solvable right now. So yes, the way to go probably is to just offer the tools to solve this on your own.
In this spirit I and if we're looking at the overlay option I was wondering if you may consider making the selectors that are currently hardcoded inside overlay-grid configurable. But while thinking about it we're probably already entering the aforementioned non-Susy-territory  :wink:
Another idea: in container you're automatically calling show-grid. Would it be possible to globally store all containers in a map at that point? One could then use this to create all debug grids in any way needed. Be it a trigger on hover, a special body class that is only added by a CMS for admins or by JS. This would be pretty flexible without baking any assumptions directly into Susy.
I'm just thinking loud here. It would be great to have your thoughts on this.

Good thoughts! I'll have to dig into them a bit more — or look at pull requests if you want to make some suggestions. At the very least, making more things configurable is always a positive in my book. :)

Cool! I just wanted to see if you're generally interested before getting my hands dirty with PRs and stuff. I'll look into it on the weekend. Until then thanks for the feedback and Susy in general.


[just a question] about the priority

hi @alexislg2 (priority author) and @macdonst
according to this
https://github.com/phonegap/phonegap-plugin-push#priority-in-notifications
could I ask what the priority true meaning ?
2 > 1 > 0 > -1 > -2
what the any different to show in my phone ?

@bau720123 read up on it here:
http://developer.android.com/intl/pt-br/reference/android/app/Notification.html#PRIORITY_DEFAULT

hi @macdonst
thanks your document
very clear
the document told us that "priority" feature "Added in API level 16"
well...according to this
http://developer.android.com/intl/pt-br/guide/topics/manifest/uses-sdk-element.html#ApiLevels
you can find a table that list and describe the api level with the Android version
api level 16 is Android 4.1, 4.1.1
is it mean our Android initial system must equal or greater than Android 4.1, 4.1.1 ? right ?
your document also told us "The UI may choose to show these items different size, or at a different position in notification lists"，if have a screenshot can show this，it will more better

You're right on your first assumption, you need to have an android version of 16(api level) or higher for these settings to work (it doesn't mean it will crash on lower versions, it'll just be a no-op function).
On your last question, it's up to the OS to determine what it'll do with each priority, not to the developer. For example, the UI from Samsung devices might do things differently than the UI from LG.

hi @fredgalvao
very clear again
close it


[SPARK-18617][Test]Fix flaky test: StreamingContextSuite.SPARK-18560 Receiver data should be deserialized properly

What changes were proposed in this pull request?
Fixed the potential SparkContext leak in StreamingContextSuite.SPARK-18560 Receiver data should be deserialized properly which was added in #16052. I also removed FakeByteArrayReceiver and used TestReceiver directly.
How was this patch tested?
Jenkins

cc @rxin

Test build #69438 has started for PR 16091 at commit dc07283.

LGTM

Test build #69439 has started for PR 16091 at commit 4785df8.

Test build #69438 has finished for PR 16091 at commit dc07283.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69438/
Test PASSed.

Merging in master/branch-2.1.

Test build #69439 has finished for PR 16091 at commit 4785df8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69439/
Test PASSed.


fix credential bugs

This pr is to fix credential related bugs including:

Fail to use resource manager to retrieve connection string.
Default token credential still takes effects when other token credentials are provided.
The auto-complete properties from binder is not passed to container.

/azp run java - spring - tests

API changes have been detected in com.azure.spring:spring-messaging-azure. You can review API changes here

API changes have been detected in com.azure.spring:spring-messaging-azure-eventhubs. You can review API changes here

API changes have been detected in com.azure.spring:spring-messaging-azure-servicebus. You can review API changes here

API changes have been detected in com.azure.spring:spring-cloud-azure-autoconfigure. You can review API changes here

API changes have been detected in com.azure.spring:spring-cloud-azure-stream-binder-servicebus. You can review API changes here

API changes have been detected in com.azure.spring:spring-cloud-azure-stream-binder-eventhubs. You can review API changes here

/azp run java - spring - tests


Fixed Invalid Syntax Error

Fixed error where, in the subtitle file, the Angle would be a float instead of a double, causing strconv.ParseInt to error

Good stuff thanks man


Building .NET Core from source on RHEL

This item is to track general progress on building from source on RHEL.

This high level tracking issue has served its purpose. Specific issues will continue to be opened individually.


pubspec.yaml dependencies updated

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

updated in another commit
https://github.com/livekit/client-sdk-flutter/pull/313/commits/217498e9f3e96bc2c6b545ce963410d966a948ef


Fix failure to render during interactive readyState

This is a follow-up to PR 1151.  I apologize for doubling up on this, but the last change was insufficient and we've found additional edge cases of undesired behavior with aggressive preloading of deferred scripts.
In addition to solving behavior for non-turbolinks users, this change also restores the original intent of clientStartup(), which explicitly calls out the turbolink check as needing to be done AFTER page load.  Since PR 1099 this check has potentially been done prior to load.

This change is

Coverage remained the same at ?% when pulling 7bde2d17fa68c1892e67750caca2ecefa1c8b427 on rakelley:init-event-handling into 3283f3d5d9bdd35a3560d71d9b0b69fa0a551b9e on shakacode:master.

I just released 11.0.6. 11.0.5 is broken.
@rakelley can you check that this comment is addressed. https://github.com/shakacode/react_on_rails/commit/46350cec56297aef1f94b98f1606ad02780ae107#commitcomment-30794250
If there are more changes to this, I'll push another release.


Update README.md

Description of Change

Checklist

[ ] PR description included and stakeholders cc'd
[ ] npm test passes
[ ] tests are changed or added
[ ] relevant documentation is changed or added
[ ] PR title follows semantic commit guidelines

Release Notes

Notes:

/trop run backport-to 4-0-x

/trop run backport-to 4-0-x


Warning about crossing stream, but then no add a bridge dialog

URL
https://www.openstreetmap.org/edit#map=19/24.17380/120.87923
How to reproduce the issue?
Try to save a change.
You get this warning.
But then clicking it,
doesn't show a dialog about adding a bridge.
Screenshot(s) or anything else?

Which iD Editor versions do you see the issue on?
Released version at openstreetmap.org/edit
Which browsers are you seeing this problem on?
Chrome

In most cases I encounter, adding a bridge would be an error. Adding a tunnel=culvert on the stream is more common. Or even a ford for paths.

Well it would be great if when it did offer fixes,
it would offer both choices,
and be clear about which item will pass over/under which item,
even if it is assumed that bridges pass over streams. (But sometimes they don't !)

When "Save" button is clicked, the displayed list of errors and warnings is not directly actionable. Instead, one has to double-click an entry and it will zoom in the affected map features and then offer the standard resolution choices.
The behavior has been changed at some recent release. I don't remember exactly how it was before, but I'm sure it used to only took a single click to zoom.


[lint] Another round of lint waiver updates and fixes

I wonder if we should add a line length check to CI? Should be quick and save us some bother, maybe it's just me who keeps messing that one up!
Merge state

Verible lint already checks for that, but we have set the limit to 150 as there were too many violations (and I think back in the day there were no waivers for Verible). It's probably time to revisit that.
https://github.com/lowRISC/opentitan/blob/757278a744c20096f5c56bc288deeb8defd9cbfa/hw/lint/tools/veriblelint/rules.vbl#L7-L10

I wonder if we should add a line length check to CI? Should be quick and save us some bother, maybe it's just me who keeps messing that one up!
Merge state

Verible lint already checks for that, but we have set the limit to 150 as there were too many violations (and I think back in the day there were no waivers for Verible). It's probably time to revisit that.
https://github.com/lowRISC/opentitan/blob/757278a744c20096f5c56bc288deeb8defd9cbfa/hw/lint/tools/veriblelint/rules.vbl#L7-L10

Yeah I think we've relaxed that to 150 to get things passing.
There is still a TODO in the configuratio file:
https://github.com/lowRISC/opentitan/blob/d0a0012e13d7befb8e9831227b34921b879fe711/hw/lint/tools/veriblelint/rules.vbl#L7-L10
We should revisit that yes.

Yeah I we've relaxed that to 150 to get things passing.
We should revisit that yes.

I had a look at this this morning. I've got a pending change to regtool to get rid of a whole host of long lines. Once that's in, we've vendored in some icache changes and this PR is merged, everything will be clean.


[9.x] Fix in rule for values with trailing backslash

The In Rule fails, if the value has a trailing backslash.

If this is broken in 8.x as well please send this to 8.x instead then.

If this is broken in 8.x as well please send this to 8.x instead then.

Sorry, I will close this pull request and create another one for 8.x instead.


🧹 lint code

PR Type

[x] Refactoring

Context

[x] Closes #999
[x] fix libs/ui eslint config

Саме так
Please fix deeepscan and let it roll

Ooops.. can't pull it locally


chaos test for streaming and agg in AggregateStreamingNode when compi…

…le with ASAN

be analogous to #1584
add compile option make when run in mode of ASAN, it's easy to test

run starrocks_be_unittest


Overload QuestionAnswerAdvisor to use SearchRequest.defaults()

Although it's helpful to be able to customize the SearchRequest given to QuestionAnswerAdvisor, it's not uncommon to just rely on SearchRequest.defaults(). Therefore, this PR overloads the constructor with a version that only requires a VectorStore and uses SearchRequest.defaults()...in the interest of simplifying the application code a little.

Makes perfect sense. Thanks @habuma


Make buildx optional with a flag

For some reasons when using builx my addon fails to build armhf images with base python:3.10-slim-buster images.
I get this error:
#2 [internal] load .dockerignore
#2 transferring context: 211B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/python:3.10-slim-buster
#3 ...

#4 [auth] library/python:pull token for registry-1.docker.io
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/python:3.10-slim-buster
#3 ERROR: no match for platform in manifest sha256:f17c94905c9cd56dce9ef6ce63229045a75f395f7b5b68eb69ef617079c51848: not found
------
 > [internal] load metadata for docker.io/library/python:3.10-slim-buster:
------
error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: no match for platform in manifest sha256:f17c94905c9cd56dce9ef6ce63229045a75f395f7b5b68eb69ef6[170](https://github.com/asosnovsky/Shortumation/runs/8121900517?check_suite_focus=true#step:5:179)79c51848: not found
Error: Process completed with exit code 1.

So after some googling I came across https://github.com/docker/buildx/issues/476 which coupled with the fact that I have been experiencing this issue ever since the last update believe that buildx is the problem.
Please merge this PR so that I can disable it where needed.
Thank you!

Hi @asosnovsky,
It seems you haven't yet signed a CLA. Please do so here.
Once you do that we will be able to review and accept this pull request.
Thanks!

make that still sense, also since it's now the default on docker itself? Maybe it got fix with new docker version?

Good question indeed.
@asosnovsky is this still a problem for you?


Windows enable credssp script - on exception Write-Error

Other than that looks good.

Thanks @bostko - merging now.


Umbrella issue for go modules

golang/dep is no longer the official tool for dependency management in the community. Go 1.11 adds preliminary support for a new concept called "modules", an alternative to GOPATH with integrated support for versioning and package distribution. Using modules, developers are no longer confined to working inside GOPATH, version dependency information is explicit yet lightweight, and builds are more reliable and reproducible.
Starting in Go 1.13, module mode will be the default for all development.
Below is the TODO list of project in the gardener org. The projects transitioned to go modules are the marked ones:

[ ] gardener/gardener
[x] gardener/gardenctl (ref gardener/gardenctl#78)
[x] gardener/machine-controller-manager (ref gardener/machine-controller-manager#280)
[ ] gardener/etcd-backup-restore
[ ] gardener/gardener-extensions
[ ] gardener/external-dns-management
... and others

P.S I didn't want to duplicate the same issue across the project, so opening a "umbrella" one. Please feel free to comment if you have any concerns or objections.

Go modules wiki in golang/go

From Go modules wiki in golang/go:

Modules must be semantically versioned according to semver, usually in the form v(major).(minor).(patch), such as v0.1.0, v1.2.3, or v1.5.0-rc.1. The leading v is required.

Any update?


perf(contract): Replace Multicall with Solady's Multicallable

Removed internal Multicall implementation and interface. Updated references to use Multicallable from the Solady library for batch function calls. This optimizes and simplifies the contract dependencies.

Should we consider https://github.com/Vectorized/multicaller/blob/main/src/Multicaller.sol since this implementation is not payable aware

Should we consider https://github.com/Vectorized/multicaller/blob/main/src/Multicaller.sol since this implementation is not payable aware

@giuseppecrj This one is a more general multicall for calling external contracts and not a drop-in replacement.

should also add tests for how multicaller would be used in space proxy context.

There's no dedicated test contract for SpaceProxy but I added a fuzz test for multicall.

should also add tests for how multicaller would be used in space proxy context.

There's no dedicated test contract for SpaceProxy but I added a fuzz test for multicall.

the space proxy is an instance of a space, you can create test for it by inheriting BaseSetup and using either the space or everyoneSpace address since they're proxies

should also add tests for how multicaller would be used in space proxy context.

There's no dedicated test contract for SpaceProxy but I added a fuzz test for multicall.

the space proxy is an instance of a space, you can create test for it by inheriting BaseSetup and using either the space or everyoneSpace address since they're proxies

@giuseppecrj Added.


FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

meet some problem in the build step

<EMAIL_ADDRESS>build:linux /home/xuchen/Downloads/electronic-wechat
./scripts/build.sh linux x64

Start packaging for linux x64.
Packaging app for platform linux x64 using electron v1.4.15
<--- Last few GCs --->
655057 ms: Mark-sweep 1396.9 (1458.1) -> 1396.6 (1458.1) MB, 120.3 / 1 ms (+ 1154.9 ms in 20 steps since start of marking, biggest step 792.0 ms) [allocation failure] [GC in old space requested].
656204 ms: Mark-sweep 1396.6 (1458.1) -> 1396.4 (1458.1) MB, 1147.4 / 1 ms [last resort gc].
657363 ms: Mark-sweep 1396.4 (1458.1) -> 1396.4 (1458.1) MB, 1159.4 / 1 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x57ba1eb4629
1: pipe [_stream_readable.js:~452] [pc=0x3a23abda73e4] (this=0xed740c1acf9 <a ReadStream with map 0x2613506c4ed9>,dest=0x27d2b15f2d79 <a WriteStream with map 0x2613506c4a61>,pipeOpts=0x57ba1e041b9 )
2: arguments adaptor frame: 1->2
3: /* anonymous */ [/home/xuchen/Downloads/electronic-wechat/node_modules/.2.0.0@fs-extra/lib/copy/ncp.js:108] [pc=0x3a23a0382099] (this=0x27d2...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
./scripts/build.sh: line 30:  7969 Aborted                 electron-packager . "${APP_NAME}" --platform=$PLATFORM --arch=$ARCH --electronVersion=1.4.15 --app-version=1.4.0 --asar --icon=assets/icon.icns --overwrite --out=./dist --ignore=${ignore_list}

@xuchendev Looks like it's related to your own environment. What's its memory size? Can you reproduce it everytime?


Fixed a null dereference in UsingStatementRule()

Fixes #2992.
Single line fix that returns an ErrorStatementAst after a null check, rather than continuing to next check and dereferencing.

Hi @rjmholt, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. Real humans will now evaluate your PR.

TTYL, MSBOT;

Fixed properly by @iSazonov in #3007


missing article: grid_proxy

background story
we are moving relevant articles from old manual to new manual because we wanna archive the old one after migration is completed.
issue:
https://github.com/threefoldfoundation/info_manual3/issues/176
issue
not sure if its still applicable or not, not sure if we still offr this solution or not, but web team is doing research and found a lot of articles are not in the new manual... for example this one https://library.threefold.me/info/manual/#/manual3_iac/client_grid_proxy/manual__grid_proxy
if code owners accept this FR, id gladly help to make this one just assign to me, thank you

existed https://www2.manual.grid.tf/proxy/proxy.html?highlight=TF Grid DB client#tf-grid-db-client