[Bug] Heatmap does not work with value axis

Version
5.3.3
Link to Minimal Reproduction
https://jsfiddle.net/amiraj/5Lka0wmx/4/
Steps to Reproduce

Create a regular heatmap and set the axis types to 'value'
zoom in and out and you see bad rendering of the pixels

Current Behavior
value type axis won't work with heatmaps. One issue for example is when zooming you see bad pixel locations:

These issues can be fixed by changing axis type to category. But that has it's own issues and this fix is a hacky fix.
Expected Behavior
heatmap chart behavior should not depend on axis type.
Environment
- OS:
- Browser:
- Framework:

Any additional comments?
No response

It's suggested to use the unminimized version in the development environment to get some helpful debug information. The heatmap series must have two category axes currently. The value axis is not supported.


Clarity Changes in Concepts Section

The Concepts section needs to have a bit more clarity and consistency when defining usage types.  First, I think we need to lose the idea of "Core Concepts" and focus on different Concept types.  I think combining the "like" types into one table as is done in the Core Concepts section is fine, but we should also have some text up front that describes that the tables below have the usage patterns dictated by the value of "type".
I also think that the sections should re-iterate the types by avoiding words like "these" and say "Concepts with the type of Activity", for example.
As is an already known global change, requirements should be separated from guiding text.  This section does this in some places but not others.  We also may want to say that unless specified as "Optional", all table names/values are MUSTs for the specified type.

I like core concepts as a way to organize things, but I'm not overly attached to it.

We also may want to say that unless specified as "Optional", all table names/values are MUSTs for the specified type.

This is already fully handled by the document interpretation section: https://github.com/DataInteroperability/xapi-profiles/blob/master/xapi-profiles-spec.md#document-interpretation-and-general-restrictions

I also came to the conclusion of ditching "core concepts". See suggested new wording here: https://github.com/DataInteroperability/xapi-profiles/issues/149


azure - Added link to mailer readme in Azure notify example

@erwelch @aluong

Should we add an example of how to setup mailer in Azure? https://github.com/capitalone/cloud-custodian/blob/master/tools/c7n_mailer/README.md


[6528] - Add logins to the smoke tests

Context
An incident happened Incident report some_users_unable_to_sign_in_to_register 20231207
This adds smoke testing that uses a test user account to log into the service on deploy.
Changes proposed in this pull request

creates a new smoke test user - added to production, productiondata and staging
simple capybara test that visits the live deployed
username/pass are added to the ENV for each relevant site
this skips the tests if OTP sign-in method is enabled (this will need to be a separate spec if wanted)

 Add logins to the smoke tests

Looks good. I was able to login to productiondata by running the test locally.
The only thing I wasn't able to check was the smoke_test.username and password settings. How are these configured?

so these are stored on the server - if testing locally you can put them in your config/settings/development.local.yml as
smoke_test:
  username: do-not-store-here
  password: do-not-store-here


Update reset-the-audio-conferencing-pin-in-teams.md

Minor edit to remove extra word

@JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@tonysmit PR has been copyedited and is ready for final review, could you please check and merge? Thanks!


Use Legacy light

The useLegacy part showing deprecated in future

Thanks. I noticed the same thing a couple of weeks ago when I switched to using the newer three.js module. The latest release of the TalkingHead class no longer refers to useLegacy.


Fixed broken link.

It seems there was a renaming with the file pointed to. I think this is the intended url now.

Thank you for fixing that. I merged it here: https://github.com/kareman/FootlessParser/pull/12


[TS migration] Migrate 'BootSplash' lib to TypeScript #24846

Details

Fixed Issues

$ https://github.com/Expensify/App/issues/24846
PROPOSAL: N/A
Tests

Upon starting the application, the splash screen should work normally as no behaviour updates were made

Offline tests

N/A
QA Steps

N/A
PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] If we are not using the full Onyx data that we loaded, I've added the proper selector in order to ensure the component only re-renders when the data it is using changes
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
https://github.com/Expensify/App/assets/55854142/3589d050-ee8e-4208-854a-5c6a8017d18f

Mobile Web - Chrome
https://github.com/Expensify/App/assets/55854142/53b043d9-e8f4-4f83-b659-bc4b6d92499c

Mobile Web - Safari
https://github.com/Expensify/App/assets/55854142/e6b3b8b4-25da-49e2-be07-5a89ec2806e6

Desktop
https://github.com/Expensify/App/assets/55854142/e9edc853-7ca8-4453-b4aa-22f34ff2c902

iOS
https://github.com/Expensify/App/assets/55854142/9b4057cd-736c-4cc2-a5bc-fb776cd90d4e

Android
https://github.com/Expensify/App/assets/55854142/f3f27e19-baba-4679-980d-85f676dbe08b

Reviewing...

LGTM!

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[x] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

https://github.com/Expensify/App/assets/41169160/2f69873d-7885-48d2-a957-01b7bf908cba

Mobile Web - Chrome

https://github.com/Expensify/App/assets/41169160/14b70285-0243-464a-8743-75ea2b1897ce

Mobile Web - Safari

https://github.com/Expensify/App/assets/41169160/723c5ded-83f1-43f8-93cb-16ed5b1acfb5

Desktop

https://github.com/Expensify/App/assets/41169160/fccca6cf-c8da-4bcb-afb1-75e2955c7c1f

iOS

https://github.com/Expensify/App/assets/41169160/7844d91f-2202-4b59-8db9-103f0c59a17d

Android

https://github.com/Expensify/App/assets/41169160/34d9617b-508b-45cf-a715-b76a9c8fd12f

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by https://github.com/amyevans in version: 1.3.73-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/mountiny in version: 1.3.73-1 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
failure ❌

🕸 web 🕸
success ✅

🚀 Deployed to staging by https://github.com/amyevans in version: 1.3.74-0 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/chiragsalian in version: 1.3.74-3 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
failure ❌

🕸 web 🕸
success ✅


Support Node 8

This is now built in although considered experimental:
https://nodejs.org/api/inspector.html

Briefly discussed with @coreylight for fidelity and it seems with this we'd create a v1 branch of the current code to support Node v6.10. Then, we'd maintain master off the new node stdlib inspector module.

Work in progress branch: https://github.com/ewindisch/iopipe-plugin-profiler/tree/node8
I expect this should be ready for become a PR within a couple days!

#49

Branch with continued work to attempt getting heap profiling working. This has a bunch of wolf-fencing and other things built-in (as of writing this), and is non-working... so I didn't want to commit it to the otherwise working 'node8' branch: https://github.com/ewindisch/iopipe-plugin-profiler/tree/node8heap


Bug: Audio driver has to be reinstalled every day

Disclaimer:
Please go through existing issues to avoid creating duplicates. If you found a similar issue please post a comment there. If you are uncertain your issue is related to others then create the issue. Obvious ignorance of this rule might get you blocked from posting new issues. Please tick the box below to show that you have followed the guideline.

[x] I have checked for a similar issue and sure it hasn't been reported before.

Describe the bug
The pop-up informing that the audio driver needs to be installed (again), randomly appears.
It's most likely a bug, since there hasn't been updates since I'm getting the problem.
Steps to Reproduce

Open Eqmac, and occasionally, you'll get the pop-up.

Expected behaviour
That not appearing every time.
Setup information:

Audio device used for playback: Any
Audio transmission interface: Any
macOS Version: 13.2.1
eqMac Version 1.7.10
eqMac UI Version 4.6.3

Screenshots or Console.app logs

Yes same issue here. And eqMac stop working after there is no sound in my browser, FireFox after a couple of minutes. I see that headphone icon in my systemtray where normally that speaker icon is where you can change the volume.

Happening here too. MacOS 13.3, M1 MacBook Pro, eqMac 1.7.10.

Same here...very annoying!
MacOS 13.2
M1 MacBook Air
eqMac v1.7.10
driver v2.4.3

i also still have this bug

I am having the same bug. It may have something to do with macOS, because I had the same thing when I was using Boom
MacBook Air 2020 M1
MacOS 13.2.1 (22D68)
eqMac v1.7.10

Same issue
Apple M2 Pro 2023
macOS 13.3.1 (22E261)
eqMac v1.7.10

Still happening basically every time I wake from sleep. Version 1.7.10, driver v2.4.3.
MacBook Pro M1 Pro
macOS 13.3.1(a).

I was struggling with this issue too, and for as crazy as it might sound i might have solved it just setting the app in window mode instead of popover mode (the green circular button on top left). I'm not saying it 100% works but i can tell that since i did it a week and a half ago i havent had the app asking me to install the drivers, as soon as i switched again to popover mode it lasted one day and then i got asked to install the drivers. Now I'm back to window mode and haven't had any request in the last 3 days.
Maybe you can try and give a feedback to understand if it's just been shear luck or if it actually does something.
Really hope to help, this bug is so annoying...

Same issue
Version 1.7.10, driver v2.4.3
MacBook Pro M1 Pro
macOS 13.4

Duplicate to #780.


Add readme.html to every exported project

#Problem
Some offline exporters are having issues, (Think BLE and the need for a 3rd party flash algorithm with Keil V4), currently there is no good way to help people out
#Solution
Add a readme.html to every offline exporter that redirects people to a page where we keep a list of the known errata for each platform and compiler. Part of the web address is the platform and compiler so they will automatically jump down the page to where their platform and compiler are.

@sg- look over here

;)

@screamerbg  Can we get this merged into the release branch?

@BlackstoneEngineering release branch? we have just master here

This looks like it will fail on any systems other than Windows - it should probably be os.path.join(temdir, 'README.html') so it's platform independent.

@matthewelse true

@screamerbg srry, not release branch, meant get it pushed live.
@matthewelse good catch, i'll change that and do a new pull request

fixed in pull request https://github.com/mbedmicro/mbed/pull/855


CI - Have test_code_length_estimate() test MACHINE_CODE_PER_INSTRUCTION_METER_CHECKPOINT

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 90.07%. Comparing base (a996494) to head (dd41ce7).
Report is 8 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #588      +/-   ##
==========================================
+ Coverage   89.92%   90.07%   +0.15%
==========================================
  Files          22       22
  Lines        9735     9713      -22
==========================================
- Hits         8754     8749       -5
+ Misses        981      964      -17

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[XGBoost4J-Spark] Random Forest mode

Is it possible to use the Random Forest mode (https://xgboost.readthedocs.io/en/stable/tutorials/rf.html) in XGBoost4J-Spark?

Took a quick look, the num_parallel_tree parameter needs to be supported for the spark package.


Massive memory (100GB) used by dask-scheduler

Trying to use rapids 0.14, conda, Ubuntu 16.04, python 3.6 on xgboost 1.2.
I can't tell if this issue is a dask problem or xgboost one:  https://github.com/dask/dask/issues/6833
So I'm cross-posting here.  It's a serious problem.
Just running sample of airlines data in a fit.  About 5M x 10 columns.  Workers never show too much memory use, but dask-scheduler keeps accumulating memory.  The work is well-distributed on the cluster, using all GPUs on a 3-node cluster each with 2 GPUs.
I can possiblly imagine that xgboost is persisting memory and when work is done that data gets pushed to the scheduler, but I'm not clear on what is going on.  I can imagine my use of data is sticking data into the graph, but unclear why persists once xgboost is done and fork is closed.
Any ideas @trivialfis ? Thanks!

Here is the basic workflow in xgboost:

persist
identify the owning worker of each partition.
construct DMatrix local to that worker by referencing identified partitions.
train on each worker.
return the booster.

I don't understand why the scheduler blows up...

I think histogram-merging happens on the scheduler because that's where the rabit tracker process runs, right?
https://github.com/dmlc/xgboost/blob/fcfeb4959c6e361f2fd1cd18c3b61b598dc205ae/python-package/xgboost/dask.py#L612
Doesn't that mean the scheduler acts as the rabit "master"? So it would be handling the feature histograms sent back from workers and the work of merging them, right?

No. The tracker is only for coordinating the allreduce tree/ring.  Like assigning worker rank.  No actual data is flowed in there.

Once the allreduce tree is constructed, all peers pass messages directly.

Ah ok I see! I misunderstood the role of the tracker in this process @hcho3 explained to me

@trivialfis I'll write a complete repro soon that hopefully is fairly minimum (hopefully only 1 node).
Also, what would you recommend?  If I have an in-CPU-memory frame, what is the best way to (at low CPU and GPU memory cost) get that to xgboost?  i.e. I can't use dask_cudf.read_csv() since the file is only on one node in the cluster, the disk is not shared.  I don't want to use s3/hdfs/etc. this is just an on-prem cluster.
I'm currently doing what I showed in the dask issue:
with client:
    import xgboost as xgb

    import dask.dataframe as dd
    chunksize=500000
    X = dd.from_array(X, columns=columns, chunksize=chunksize)
    y = dd.from_array(y, chunksize=chunksize)
    model = xgb.dask.DaskXGBClassifier()
    model.fit(X, y)

over and over in separate forks, where the client is just connecting to the scheduler.
I see excessive scheduler memory use and accumulation even when I added a persist call (I didn't need because you already do that) and with client.cancel(X) and client.cancel(y) after the fit is done.  But still the scheduler continues to accumulate memory every fit.
Do I have to also run client.scatter(X) and client.scatter(y) to get it off the scheduler?  Seems like alot of extra work.
But also, what if I want to get the data on the GPU ASAP?  But, I don't want to ever have all the data on 1 GPU.  Can I call dask_cudf.DataFrame(dd.from_array(X, columns=columns, chunksize=chunksize)) ?

# One shell run:
# dask-scheduler --port 8790 --host <IP_ADDRESS> --pid-file dask_cuda_scheduler.pid --protocol tcp --dashboard-address 8791

# another shell run:
# dask-cuda-worker tcp://<IP_ADDRESS>:8790 --pid-file ./tmp/dask_cuda_worker_n_jobs.pid --nthreads 1 --rmm-pool-size 1GB

# then run this script:

# suppose alot of things were done, and now have in-memory frame.  We will mimic this by loading a file.
import datatable as dt
# -rw------- 1 jon jon<PHONE_NUMBER> Oct 25 02:16  /home/jon/Downloads/HIGGS.head.csv
X = dt.fread("/home/jon/Downloads/HIGGS5M.csv")
X = X[:, [bool, int, float]]
target = 'C1'
y_np = X[:, target].to_numpy().ravel()
del X[:, target]
X_np = X.to_numpy()
columns = list(X.names)
del X
print("Done getting data into memory")

# from here on we just have some in-memory numpy frame
for i in range(0, 100):
    from dask.distributed import get_client
    with get_client(address="%s://%s:%s" % ("tcp", "<IP_ADDRESS>", "8790")) as client:

        params = {'tree_method': 'gpu_hist', 'n_estimators': 2, 'objective': 'binary:logistic'}
        import xgboost as xgb
        model = xgb.dask.DaskXGBClassifier(**params)

        chunksize = 5000
        import dask.dataframe as dd
        X = dd.from_array(X_np, columns=columns, chunksize=chunksize)
        y = dd.from_array(y_np, chunksize=chunksize)
        print("Done getting data into dask")
        model.fit(X, y)
        print("Done with fit")
        preds = model.predict(X)
        print("Done with predict")
        print(preds[0].compute())

        client.cancel(X)
        client.cancel(y)
        del X, y
        print("Done with del")

Where that is just HIGGS with 5000000 rows and 29 columns.
Problem 1) On 64GB system the first (iteration 0 of the loop) model.fit() step goes through all my system memory and OOM killer kills it.  I don't see why model.fit should use so much memory.  It's not even on GPU yet.

If I lower the rows to 1M and chunksize to 50000
# One shell run:
# dask-scheduler --port 8790 --host <IP_ADDRESS> --pid-file dask_cuda_scheduler.pid --protocol tcp --dashboard-address 8791

# another shell run:
# dask-cuda-worker tcp://<IP_ADDRESS>:8790 --pid-file dask_cuda_worker_n_jobs.pid --nthreads 1 --rmm-pool-size 1GB

# then run this script:

# suppose alot of things were done, and now have in-memory frame.  We will mimic this by loading a file.
import datatable as dt
# -rw------- 1 jon jon<PHONE_NUMBER> Oct 25 02:16  /home/jon/Downloads/HIGGS.head.csv
X = dt.fread("/home/jon/Downloads/HIGGS5M.csv")
X = X[0:1000000, [bool, int, float]]
target = 'C1'
y_np = X[:, target].to_numpy().ravel()
del X[:, target]
X_np = X.to_numpy()
columns = list(X.names)
del X
print("Done getting data into memory")

# from here on we just have some in-memory numpy frame
for i in range(0, 100):
    from dask.distributed import Client
    with Client(address="%s://%s:%s" % ("tcp", "<IP_ADDRESS>", "8790")) as client:

        params = {'tree_method': 'gpu_hist', 'n_estimators': 2, 'objective': 'binary:logistic'}
        import xgboost as xgb
        model = xgb.dask.DaskXGBClassifier(**params)

        chunksize = 50000
        import dask.dataframe as dd
        X = dd.from_array(X_np, columns=columns, chunksize=chunksize)
        y = dd.from_array(y_np, chunksize=chunksize)
        print("Done getting data into dask")
        model.fit(X, y)
        print("Done with fit")
        preds = model.predict(X)
        print("Done with predict")
        print(preds[0].compute())

        client.cancel(X)
        client.cancel(y)
        del X, y
        print("Done with del")

Then I get GPU OOM and xgoost hangs in client code:
worker restarts fine:
distributed.core - INFO - Starting established connection
task [xgboost.dask]:tcp://<IP_ADDRESS>:37071 connected to the tracker
task [xgboost.dask]:tcp://<IP_ADDRESS>:37071 got new rank 0
terminate called after throwing an instance of 'thrust::system::system_error'
  what():  parallel_for failed: out of memory
distributed.nanny - INFO - Worker process 10804 was killed by signal 6
distributed.nanny - WARNING - Restarting worker

but client code using xgboost hangs:
(base) jon@pseudotensor:~/h2oai.fullcondatest$ PYTHONPATH=. python scheduler_problem.py
Done getting data into memory
Done getting data into dask
/home/jon/minicondadai/lib/python3.6/site-packages/distributed/client.py:3479: RuntimeWarning: coroutine 'Client._update_scheduler_info' was never awaited
  self.sync(self._update_scheduler_info)

If I lower to 1M rows but only 11 columns:
# One shell run:
# dask-scheduler --port 8790 --host <IP_ADDRESS> --pid-file dask_cuda_scheduler.pid --protocol tcp --dashboard-address 8791

# another shell run:
# dask-cuda-worker tcp://<IP_ADDRESS>:8790 --pid-file dask_cuda_worker_n_jobs.pid --nthreads 1 --rmm-pool-size 1GB

# then run this script:

# suppose alot of things were done, and now have in-memory frame.  We will mimic this by loading a file.
import datatable as dt
# -rw------- 1 jon jon<PHONE_NUMBER> Oct 25 02:16  /home/jon/Downloads/HIGGS.head.csv
X = dt.fread("/home/jon/Downloads/HIGGS5M.csv")
X = X[0:1000000, [bool, int, float]]
X = X[:, ["C%d" % i for i in range(0, 10)]]
target = 'C0'
y_np = X[:, target].to_numpy().ravel()
del X[:, target]
X_np = X.to_numpy()
columns = list(X.names)
del X
print("Done getting data into memory")

# from here on we just have some in-memory numpy frame
for i in range(0, 100):
    from dask.distributed import Client
    with Client(address="%s://%s:%s" % ("tcp", "<IP_ADDRESS>", "8790")) as client:

        params = {'tree_method': 'gpu_hist', 'n_estimators': 2, 'objective': 'binary:logistic'}
        import xgboost as xgb
        model = xgb.dask.DaskXGBClassifier(**params)

        chunksize = 50000
        import dask.dataframe as dd
        X = dd.from_array(X_np, columns=columns, chunksize=chunksize)
        y = dd.from_array(y_np, chunksize=chunksize)
        print("Done getting data into dask")
        model.fit(X, y)
        print("Done with fit")
        preds = model.predict(X)
        print("Done with predict")
        print(preds[0].compute())

        client.cancel(X)
        client.cancel(y)
        del X, y
        print("Done with del")

then things finally run.
jon@pseudotensor:~/h2oai.fullcondatest$ nvidia-smi
Thu Nov 12 23:59:58 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2080    On   | 00000000:01:00.0  On |                  N/A |
| 41%   52C    P0    50W / 215W |   3157MiB /  7979MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2552      G   /usr/lib/xorg/Xorg                            55MiB |
|    0      2791      G   /usr/bin/gnome-shell                          57MiB |
|    0      3358      G   /usr/lib/xorg/Xorg                           595MiB |
|    0      3492      G   /usr/bin/gnome-shell                         501MiB |
|    0      3953      G   ...AAAAAAAAAAAAAAgAAAAAAAAA --shared-files    22MiB |
|    0      4672      G   ...AAAAAAAAAAAACAAAAAAAAAA= --shared-files   334MiB |
|    0      5146      G   /usr/bin/nvidia-settings                      46MiB |
|    0      6039      G   ...charm-community-2019.1.1/jre64/bin/java    38MiB |
|    0     16029      C   dask-worker [tcp://<IP_ADDRESS>:43127]      1489MiB |
+-----------------------------------------------------------------------------+

But so far I'm not able with this code reproduce the creep-up in dask-scheduler memory use.  Here is just bounces up and down.
So something else is required, even though I'm pretty closely following my use.  One difference is I'm using multiple nodes in the bad case where scheduler accumulates memory.

However, I do see worker using excessive memory, even once the entire loop is done.  And the scheduler is using 10% of 64GB even though it started at 1%.

Another full python script and both using even more memory:

Maybe related: https://github.com/dask/distributed/issues/3898

Moved dask-specific discussion to: dask/distributed#4243 since seems to be dask.distributed problem only perhaps.
However, @teju85 , this issue makes using dask and xgboost with multi-GPU or multi-node with NVIDIA rapids/xgboost impossible.
As for the other problem https://github.com/dmlc/xgboost/issues/6388#issuecomment-726578903 , it would seem that is a highly excessive amount of memory being used by xgboost during model.fit before going to dask or using GPU.  That also is a major issue.

Well, I'm not sure the memory problem is not xgboost.  If I perform the same exact task but without xgboost, there is never any accumulation of memory:
# One shell run:
# dask-scheduler --port 8790 --host <IP_ADDRESS> --pid-file dask_cuda_scheduler.pid --protocol tcp --dashboard-address 8791

# another shell run:
# dask-cuda-worker tcp://<IP_ADDRESS>:8790 --pid-file dask_cuda_worker_n_jobs.pid --nthreads 1 --rmm-pool-size 1GB

# then run this script:

# suppose alot of things were done, and now have in-memory frame.  We will mimic this by loading a file.
import datatable as dt
# -rw------- 1 jon jon<PHONE_NUMBER> Oct 25 02:16  /home/jon/Downloads/HIGGS.head.csv
X = dt.fread("/home/jon/Downloads/HIGGS5M.csv")
X = X[0:1000000, [bool, int, float]]
X = X[:, ["C%d" % i for i in range(0, 10)]]
target = 'C0'
y_np = X[:, target].to_numpy().ravel()
del X[:, target]
X_np = X.to_numpy()
columns = list(X.names)
del X
print("Done getting data into memory")

# from here on we just have some in-memory numpy frame
for i in range(0, 10000):
    from dask.distributed import Client
    with Client(address="%s://%s:%s" % ("tcp", "<IP_ADDRESS>", "8790")) as client:

        params = {'tree_method': 'gpu_hist', 'n_estimators': 2, 'objective': 'binary:logistic'}
        import xgboost as xgb
        model = xgb.dask.DaskXGBClassifier(**params)

        chunksize = 50000
        import dask.dataframe as dd
        X = dd.from_array(X_np, columns=columns, chunksize=chunksize)
        y = dd.from_array(y_np, chunksize=chunksize)
        print("Done getting data into dask")
        if False:
            model.fit(X, y)
            print("Done with fit")
            preds = model.predict(X)
            print("Done with predict")
            print(preds[0].compute())

        client.cancel(X)
        client.cancel(y)
        del X, y
        print("Done with del")

This never accumulates.
@teju85 , so new theory is that xgboost is creating frames and leaking them.  So then all of the problems mentioned are xgboosts fault.

Could you please run this script and see how it goes:
dask-scheduler --scheduler-file='scheduler.json'
# new shell
dask-cuda-worker --scheduler-file='scheduler.json'

import pandas as pd
import dask.dataframe as dd
import xgboost as xgb

def main():
    colnames = ['label'] + ['feature-%02d' % i for i in range(1, 29)]
    df = pd.read_csv('HIGGS.csv',  names=colnames)

    y = df['label']
    X = df[df.columns.difference(['label'])]
    print('X.shape', X.shape)

    for i in range(0, 100):
        from dask.distributed import Client
        with Client(scheduler_file='scheduler.json') as client:
            params = {
                'tree_method': 'gpu_hist',
                'n_estimators': 2,
                'objective': 'binary:logistic'
            }

            model = xgb.dask.DaskXGBClassifier(**params)

            chunksize = 50000
            dX = dd.from_pandas(X, chunksize=chunksize)
            dy = dd.from_pandas(y, chunksize=chunksize)

            print("Done getting data into dask")
            model.fit(dX, dy)
            print("Done with fit")
            preds = model.predict(dX)
            print("Done with predict")
            print(preds[0].compute())

            client.cancel(dX)
            client.cancel(dy)
            print("Done with del")

if __name__ == '__main__':
    main()

The HIGGS.csv is unmodified version of HIGGS with X of shape (11000000, 28).

Could you please run this script and see how it goes:
dask-scheduler --scheduler-file='scheduler.json'
# new shell
dask-cuda-worker --scheduler-file='scheduler.json'

import pandas as pd
import dask.dataframe as dd
import xgboost as xgb

def main():
    colnames = ['label'] + ['feature-%02d' % i for i in range(1, 29)]
    df = pd.read_csv('HIGGS.csv',  names=colnames)

    y = df['label']
    X = df[df.columns.difference(['label'])]
    print('X.shape', X.shape)

    for i in range(0, 100):
        from dask.distributed import Client
        with Client(scheduler_file='scheduler.json') as client:
            params = {
                'tree_method': 'gpu_hist',
                'n_estimators': 2,
                'objective': 'binary:logistic'
            }

            model = xgb.dask.DaskXGBClassifier(**params)

            chunksize = 50000
            dX = dd.from_pandas(X, chunksize=chunksize)
            dy = dd.from_pandas(y, chunksize=chunksize)

            print("Done getting data into dask")
            model.fit(dX, dy)
            print("Done with fit")
            preds = model.predict(dX)
            print("Done with predict")
            print(preds[0].compute())

            client.cancel(dX)
            client.cancel(dy)
            print("Done with del")

if __name__ == '__main__':
    main()

The HIGGS.csv is unmodified version of HIGGS with X of shape (11000000, 28).

What is scheduler.json here?

It's a file created by dask-scheduler command.

Also, I'm no sure what you are asking me to do.  It appears that your script is just the same as mine.  But I'll try the json thing.
But on even on 5M higgs with all 28 columns I hit the https://github.com/dmlc/xgboost/issues/6388#issuecomment-726578903 massive memory use already.  With 10 columns I hit the GPU OOM on my 1 GPU 1080ti.  Only with 1M x 10 did I avoid those issues.

Yup.  I modified your script to be minimal.  I'm curious as I got to 12 rounds and there's no sign of memory leak.

12 rounds?  You mean 12/100 iterations?  That's probably too little to see.  I showed the increment is about 10% of 64GB every 100 iterations for the scheduler and for worker is more.

I ran  your exact case.  There are some differences:

5M x 28 columns runs fine.  Uses not much memory when starting model.fit.  Doesn't run out of GPU memory.

So somehow my version of setup and script burns through all 64GB of memory just doing model.fit and even 5M x 10 hits GPU OOM.
How do you explain?  Did you try my version?

I still see dask-worker eating up tons of memory and accumulating fast.  As before the dask-scheduler accumulates memory just much more slowly.  Again, this is opposite of my original airlines data case, where I only see the scheduler rapidly (crazily) using lots of memory on the scheduler.

Even within 1 overall cycle of 100 iterations, at about 50 iterations the worker dies due to being killed by OOM killer:
distributed.nanny - INFO - Worker process 9018 was killed by signal 9
distributed.nanny - WARNING - Restarting worker
distributed.preloading - INFO - Import preload module: dask_cuda.initialize
distributed.worker - INFO -       Start worker at:   tcp://<IP_ADDRESS>:42173
distributed.worker - INFO -          Listening to:   tcp://<IP_ADDRESS>:42173
distributed.worker - INFO -          dashboard at:         <IP_ADDRESS>:40011
distributed.worker - INFO - Waiting to connect to:    tcp://<IP_ADDRESS>:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                   67.38 GB
distributed.worker - INFO -       Local Directory: /home/jon/h2oai.fullcondatest/worker-0cqyubhi
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.RMMPool object at 0x7fce3caab390>-1fb48df2-9e8a-4a0b-8ea9-11270a198c15
distributed.worker - INFO - Starting Worker plugin <dask_cuda.utils.CPUAffinity object at 0x7fce3caab-41432ce7-62a3-4944-8d4a-1b61ee314d49
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to:    tcp://<IP_ADDRESS>:8786
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection

At that point xgboost/client seems to hang, unlike before.  No progress in the iterations of that loop.

To be clear, as mentioned at first, I'm using rapids 0.14 xgboost 1.2  So there may be improvements.  But maybe you would know if such things were fixed.

Sure.  Could you provide a script that I can exactly follow without making any edits (except for data path)?

To be clear, as mentioned at first, I'm using rapids 0.14 xgboost 1.2 So there may be improvements. But maybe you would know if such things were fixed.

I'm not aware of memory leak in xgboost.  We run benchmarks with larger datasets every now and then.

12 rounds? You mean 12/100 iterations? That's probably too little to see.

In the meanwhile, it's still running.  Hopefully can reproduce it later.

If I start to bisect between our codes to look into the first-iteration-CPU-OOM-in-model-fit issue and the GPU OOM issue, some notes:

Running the scheduler/worker/client like your case has no effect.

dask-scheduler --port 8790 --host <IP_ADDRESS> --pid-file dask_cuda_scheduler.pid --protocol tcp --dashboard-address 8791 --scheduler-file='scheduler.json'
dask-cuda-worker --scheduler-file='scheduler.json'
and `Client(scheduler_file='scheduler.json') as client:` in the script

Adding main has no effect

Consuming pandas instead of numpy fixes the first-iteration-CPU-OOM-in-model-fit issue problem at least.  Why should model.fit() consume all 64GB of memory just because I used from_array instead of from_pandas?

Can you just try your script but use from_array()?

i.e. if I use your exact sequence but only 5M rows of HIGGS, I hit the CPU OOM:
import pandas as pd
import dask.dataframe as dd
import xgboost as xgb

def main():
    colnames = ['label'] + ['feature-%02d' % i for i in range(1, 29)]
    df = pd.read_csv('/home/jon/Downloads/HIGGS5M.csv',  names=colnames)

    y = df['label']
    X = df[df.columns.difference(['label'])]
    print('X.shape', X.shape)

    for i in range(0, 100):
        from dask.distributed import Client
        with Client(scheduler_file='scheduler.json') as client:
            params = {
                'tree_method': 'gpu_hist',
                'n_estimators': 2,
                'objective': 'binary:logistic'
            }

            model = xgb.dask.DaskXGBClassifier(**params)

            chunksize = 50000
            #dX = dd.from_pandas(X, chunksize=chunksize)
            dX = dd.from_array(X.values, columns=X.columns, chunksize=chunksize)   # only diff.
            dy = dd.from_pandas(y, chunksize=chunksize)

            print("Done getting data into dask")
            model.fit(dX, dy)
            print("Done with fit")
            preds = model.predict(dX)
            print("Done with predict")
            print(preds[0].compute())

            client.cancel(dX)
            client.cancel(dy)
            print("Done with del")

if __name__ == '__main__':
    main()

Replying to:
@pseudotensor So here we have a number of issues:

On your setup, client process OOM at first iteration.

After using from_pandas instead of from_array, this issue is mitigated, and more issues follow:

scheduler OOM for some reason?  As scheduler doesn't use GPU, so I believe it CPU OOM.
worker OOM, which is mitigated by not using RMM.

Is this a fair summary?  Some more questions: How did you install xgboost?  pip/conda/conda with rapids build/built from source?

Feel free to correct me if I'm wrong about the summary.  But it's a bit difficult for me to comprehend with many comments mixed together.  It would be great if you can list out some precise details:
With which script, under what condition, how large is your data, which process OOM, and whether OOM is happening on CPU or GPU.

I'll summarize:

Tried 5M x 28 in my original script, hit CPU OOM on first model.fit.  Yes, "mitigated" by using from_pandas, but seems like serious issue that from_array leads to such problems.  I modified your script to just use from_array and it also hits the CPU OOM in model.fit on client process.

Tried 1M x 28 in my original script, but hit GPU OOM on first model fit.  i.e. this exact script: https://github.com/dmlc/xgboost/issues/6388#issuecomment-726584729 .  Every time, first fit GPU OOM is hit.  Hangs client, crashes worker.  I did not yet try to bisect this to see how changing the script will affect thigns.

task [xgboost.dask]:tcp://<IP_ADDRESS>:40307 connected to the tracker
task [xgboost.dask]:tcp://<IP_ADDRESS>:40307 got new rank 0
terminate called after throwing an instance of 'thrust::system::system_error'
  what():  parallel_for failed: out of memory

RMM option there or not doesn't matter.  Still hits GPU OOM message.  No mitigation so far.

I tried 1M x 10, so I could get around issues 1 and 2 above.   This reproduces a worker (especially) but also scheduler acumulation of memory, across iterations within 1 python client script call, but across any number of separate clean runs of the script as well.  This is similar to what I originally saw with airlines data, but in that case the scheduler has massive increases in memory use very quickly.

No mitigation so far.

Thanks for summarizing.  How about:

How did you install xgboost? pip/conda/conda with rapids build/built from source?

Thanks for summarizing. How about:

How did you install xgboost? pip/conda/conda with rapids build/built from source?

Yes, as for last month all my issues have been on exact same setup: https://github.com/dmlc/xgboost/issues/6232#issue-720915573  Yes, as I mentioned, on xgboost 1.2

Great!  I will try reproducing it over this weekend.

I made comment above, but your version of script without xgboost leads to no scheduler/worker memory accumulation.  So it does seem like a pure xgboost issue of (I guess) not cleaning up temporary dask objects, e.g. cancelling futures.
import pandas as pd
import dask.dataframe as dd

def main():
    colnames = ['label'] + ['feature-%02d' % i for i in range(1, 29)]
    df = pd.read_csv('/home/jon/Downloads/HIGGS5M.csv',  names=colnames)

    y = df['label']
    X = df[df.columns.difference(['label'])]
    print('X.shape', X.shape)

    for i in range(0, 100):
        print("i=%d" % i)
        from dask.distributed import Client
        with Client(scheduler_file='scheduler.json') as client:
            chunksize = 50000
            dX = dd.from_pandas(X, chunksize=chunksize)
            #dX = dd.from_array(X.values, columns=X.columns, chunksize=chunksize)
            dy = dd.from_pandas(y, chunksize=chunksize)

            print("Done getting data into dask")
            print(dX.mean().compute())
            print(dy.mean().compute())
            print("Done with means")

            client.cancel(dX)
            client.cancel(dy)
            print("Done with del")

if __name__ == '__main__':
    main()

With this otherwise same script but doing no xgboost stuff, worker stays at about 3.6% of 64GB of memory and scheduler at 2%.

A number of things have changed since RAPIDS 0.14.  Can you try with 0.16?  This was just released in late October?

Yes, although I think these are pure xgboost issues for all 3 problems: https://github.com/dmlc/xgboost/issues/6388#issuecomment-726709410
For problem 1) This happens before any GPU stuff, the numpy frame is poorly handled by xgboost.
For problem 2) This is not rapids specific
For problem 3) This is not rapids specific, likely temporary dask frames that are not cancelled in xgboost.  dask_cudf shouldn't be issue since the frames are on CPU not GPU.

Continued discussion in https://github.com/dask/dask/issues/6833 .  A reproducible example without xgboost is created in https://github.com/dask/dask/issues/6833#issuecomment-729554422


allow some cool down time after allocation measurements

Motivation:
We have code running on multiple threads and certain allocations/frees
happen for example after a promise has been fulfilled. We can't
deterministically wait for that so the only thing we can do is sleep a
little and wait...
Modifications:
Added a 100ms delay after an allocation counted test run.
Result:
Hopefully we don't see this again:
["total_allocations": 322051, "remaining_allocations": 24], ["total_allocations": 322051, "remaining_allocations": -24]

where the first test run had 24 allocations that weren't freed and the
second one on the other hand had 24 frees for allocations that "didn't
happen"...

@tomerd this seems to be quite important, just tested it locally and even after 50 runs I always got exactly 0 remaining allocations 🙌. I’d merge this asap as it will (hopefully) stop other PRs going red. Sorry about that...

Merged thanks


AFT v1.6.5 fails aft_account_provisioning_framework_account_metadata_ssm with >10 custom-fields SSM parameters

AFT Version:
1.6.5
Bug Description
I've upgraded AFT from v1.6.4 to v1.6.5 and we are experiencing issues when updating account-request file of an account with more than 10 custom-fields. The Step Function aft-account-provisioning-framework fails at Lambda function  aft_account_provisioning_framework_account_metadata_ssm.
However, creation of a new AWS account succeeds.
To Reproduce
Steps to reproduce the behavior:

Update account-request file of an account with more 10 custom-fields
Step Function aft-account-provisioning-framework will fail at Lamba function aft_account_provisioning_framework_account_metadata_ssm with the error below.

Expected behavior
Step Function aft-account-provisioning-framework should success with no errors.
Related Logs
Lambda Function aft_account_provisioning_framework_account_metadata_ssm error message:
{
  "resourceType": "lambda",
  "resource": "invoke",
  "error": "ClientError",
  "cause": {
    "errorMessage": "An error occurred (ValidationException) when calling the DeleteParameters operation: 1 validation error detected: Value '[...]' at 'names' failed to satisfy constraint: Member must have length less than or equal to 10.",
    "errorType": "ClientError",
    "stackTrace": [
      "  File \"/var/task/aft_account_provisioning_framework_account_metadata_ssm.py\", line 70, in lambda_handler\n    delete_ssm_parameters(target_account_session, params_to_remove)\n",
      "  File \"/opt/python/lib/python3.8/site-packages/aft_common/account_provisioning_framework.py\", line 315, in delete_ssm_parameters\n    response = client.delete_parameters(Names=parameters)\n",
      "  File \"/opt/python/lib/python3.8/site-packages/botocore/client.py\", line 508, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n",
      "  File \"/opt/python/lib/python3.8/site-packages/botocore/client.py\", line 915, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n"
    ]
  }
}

Additional context
It's weird the changelog of the v1.6.5 says this issue is resolved while we did not experience this issue before and started to face this issue with this version.
Thanks,
Francisco

@fjromerom thank you for reporting this issue.
We are looking into it.

@fjromerom We've just released 1.6.6 to resolve this issue.
Thanks for the report!


Support for VectorDBs

Hii @mufeedvh,
Is there any plan for adding support for the vector database?
I can take the initiative to add support for it

sound goods. fetch the latest changes. create a branch and start working on it.

Sure, I'll create a branch and update


Processing Job not showing failure of notebook fails

Hi,
We discovered that even when a notebook causes an error, the processing job stops in status "complete".
What we would need is in the case of a notebook having an exception to show a failed state.
I had a look at the execute.py and found this section:
https://github.com/aws-samples/sagemaker-run-notebook/blob/master/sagemaker_run_notebook/container/execute.py#L92
    except Exception as e:
        # Write out an error file. This will be returned as the failureReason in the
        # DescribeProcessingJob result.
        trc = traceback.format_exc()
        # with open(os.path.join(output_path, 'failure'), 'w') as s:
        #    s.write('Exception during processing: ' + str(e) + '\n' + trc)
        # Printing this causes the exception to be in the training job logs, as well.
        print("Exception during processing: " + str(e) + "\n" + trc, file=sys.stderr)
        # A non-zero exit code causes the training job to be marked as Failed.
        # sys.exit(255)
        output_notebook = "xyzzy"  # Dummy for print, below

I tried to change it to the following and rebuilding the docker image, but the processing job still shows "complete" instead of "failed" on a failure.
    except Exception as e:
        # Write out an error file. This will be returned as the failureReason in the
        # DescribeTrainingJob result.
        trc = traceback.format_exc()
        with open(os.path.join(os.path.dirname(output_notebook), 'failure'), 'w') as s:
            s.write('Exception during execution: ' + str(e) + '\n' + trc)
        # Printing this causes the exception to be in the training job logs, as well.
        print('Exception during execution: ' + str(e) + '\n' + trc, file=sys.stderr)
        # A non-zero exit code causes the training job to be marked as Failed.
        sys.exit(255)

I'm pretty sure that's got to do with how AWS Sagemaker determines what is a 'failed' processing job. It only causes it to 'fail' if there are specific kernel issues I think.

OK I found a solution for this.
Additionally to the change I mentioned above, I also changed this:
execute.py
from:
        papermill.execute_notebook(
            notebook_file,
            output_notebook,
            params,
            **arg_map,
        )

to:
        papermill.execute_notebook(
            input_path=notebook_file,
            output_path=output_notebook,
            parameters=params,
            log_output=True,
            autosave_cell_every=60,
            stdout_file=sys.stdout,
            stderr_file=sys.stderr,
            **arg_map,
        )

run_notebook
from:
python -u /opt/program/execute.py 2>&1 | stdbuf -o0 tr '\r' '\n'

to:
python -u /opt/program/execute.py


Detect dependencies introduced using the walrus operator

This is a surprisingly big task as it will require traversing into expressions in get_bindings.  Currently, get_bindings only looks into statements.

99% resolved by #12


Use PyO3 0.22 and rust-numpy 0.22

Summary
This commit migrates to using PyO3 0.22 and rust-numpy and updates all the deprecated and changed interfaces in the libraries to their new syntax. No new or alternative interfaces are used as part of this PR except for where deprecation warnings pointed to do so.
One thing to note is that in pyo3 0.22 a new feature py-clone was added to re-enable Py<T>::clone(). This was disabled by default in pyo3 0.22 because of a soundness issue around calling that in situations when the GIL was not held. [1] Right now qiskit relies on using Clone on Py objects because we have container types that own a PyObject that we sometimes clone at the top level so this flag needs to be set. This only occurs in the circuit crate. In the future we can look at adding interfaces to avoid needing this flag in the future.
Another thing to note is that rust-numpy includes support for ndarray 0.16, however this PR doesn't migrate to it because ndarray_einsum_beta used in the qiskit_accelerate::unitary_compose module still depends on ndarray 0.15 and we can't upgrade until that does.
Details and comments
[1] https://pyo3.rs/v0.22.3/migration#pyclone-is-now-gated-behind-the-py-clone-feature

Pull Request Test Coverage Report for Build<PHONE_NUMBER>5
Details

13 of 16 (81.25%)  changed or added relevant lines in 10 files are covered.
10 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.02%) to 88.676%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

crates/accelerate/src/nlayout.rs
1
4
25.0%

Files with Coverage Reduction
New Missed Lines
%

crates/qasm2/src/lex.rs
2
92.73%

crates/accelerate/src/target_transpiler/mod.rs
8
79.11%

Totals

Change from base Build<PHONE_NUMBER>5:
0.02%

Covered Lines:
73194

Relevant Lines:
82541

💛  - Coveralls


Update README.md

Hmm, I guess you're not that familiar with GitHub yet and that's ok. :) But this code snippet should be pushed to your fork and not my code so I'll close this PR. Thank you for using this theme.


Drop-down list items are not XML escaped

Description
If any of the XML special characters <, >, &, " were used in SetDropList() choices, the produced file will be reported as corrupted.

Steps to reproduce the issue:
package main

import (
	"github.com/360EntSecGroup-Skylar/excelize/v2"
)

func main() {
	var jobs = []string{"R&D", "PM", "UX", "HR"}
	newWorkbookWithDropList(jobs)
}

func newWorkbookWithDropList(choices []string) {
	f := excelize.NewFile()
	dv := excelize.NewDataValidation(false)
	dv.Sqref = "A:A"
	if err := dv.SetDropList(choices); err != nil {
		panic(err)
	}
	f.AddDataValidation(f.GetSheetName(0), dv)
	f.SaveAs("TestDropList.xlsx")
}

Describe the results you received:

Describe the results you expected:

Output of go version:
go version go1.17rc1 linux/amd64

Excelize version or commit ID:

master (e9ae9b4)
v2.4.0 (d42834f)
v2.1.0 (e51aff2)

Environment details (OS, Microsoft Excel™ version, etc.):
The following official implementation were tested, and all of them could reproduce this issue:

Microsoft Office 365 for Mac, 16.49 (21050901)
Microsoft Office 365 for Windows, 2102 (Build 13801.20808)
Microsoft Office 2019 Professional Plus for Windows, 2105 (Build 14026.20302)
Microsoft Office 2013 Professional Plus for Windows, 15.0.4919.1002

The following third-party implementations were also tested, but none of them was affected by the issue:

Kingsoft WPS
Apple Numbers
Google Sheets
LibreOffice

Thanks for your feedback, I have fixed it, please try to use the master branch code, and this patch will be released in the next version.


feat: added ansible kcl custom module

1. Does this PR affect any open issues?(Y/N)

[ ] N
[x] Y

fixes #952
I added the kcl run functionality in this kcl ansible custom module, cc @Peefy

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 71.844%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
44274

Relevant Lines:
61625

💛  - Coveralls


Daily...

Is this the best label? I'm guessing (as I've not driven anywhere today), this is actually "distance driven today" etc.
Or it just not updating/showing correctly for me for some reason?
Monthly makes sense that it's an average...

The efficiency entities are in miles/KwH, but I believe I had to choose miles as the unit when I implemented it so the conversion to km works properly.

Ah I confused this with your other issue. Yes, daily efficiency as in since midnight. I guess the wording could be better

Closing for now but agree it could be worded better. Will have a think and see if I can come up with something better


Remove Empty Space Around Sprites

Nearly all of the sprites in the game have a empty boarder around the art. I deliberately did this to prevent sprites from "leaking" into different frames. We should try removing this boarder to improve the workflow of adding new sprites to the game. If we still encounter the issue we should ask in the Bevy discord what we can do to address it.

Is this baked into the code or into the sprites themselves?


No longer works on Fire OS <IP_ADDRESS>

App has suddenly stopped working after an automatic update to Fire OS <IP_ADDRESS> (Insignia Fire TV). Tried to add permissions again to no avail. Home button and startup hijacking no longer work.

Apparently the wrong time for me to try this but my Toshiba fire TV is also having permission issues.

My Toshiba FireTV updated automatically and is not working now. One of my FireSticks is still working.

Any update on this issue?

I am aware of the issue but unfortunately I can't do anything about it (yet) because all of my FireTV devices are still on <IP_ADDRESS> – it looks like Amazon is still rolling out the updates and Germany hasn't received any yet.
So I will have to wait until the update pops up over here... I'm sorry.

Thanks @ErikAbele
You do amazing work!

@ErikAbele please let us know when you get it, were at the edge of our seats 😅

Ok, I also got the update yesterday and see that Amazon did re-work home button handling so I'll have to dig a little deeper.
Booting into your custom launcher should still work though; if not then try setting it up again in an ADB shell:
settings put secure enabled_accessibility_services de.codefaktor.ftvlaunchx/de.codefaktor.ftvlaunchx.HomeService
settings put secure accessibility_enabled 1

Ok, I also got the update yesterday and see that Amazon did re-work home button handling so I'll have to dig a little deeper.
Booting into your custom launcher should still work though; if not then try setting it up again in an ADB shell:
settings put secure enabled_accessibility_services de.codefaktor.ftvlaunchx/de.codefaktor.ftvlaunchx.HomeService
settings put secure accessibility_enabled 1

Okay thanks a lot man, hopefully you can get it sorted out :)

On 2nd. gen Firestick the command adb shell pm grant de.codefaktor.ftvlaunchx android.permission.WRITE_SECURE_SETTINGS came back with an "package not found error". So I decompiled and changed the package name to com.codefactor.ftvlaunchx. Disinstalled and installed the changed apk. This time the command worked without an error. Now to find out how Amazon changed the home button handling.

On 2nd. gen Firestick the command adb shell pm grant de.codefaktor.ftvlaunchx android.permission.WRITE_SECURE_SETTINGS came back with a "package not found error". So I decompiled and changed the package name to com.codefactor.ftvlaunchx. Disinstalled and installed the changed apk. This time the command worked without an error. Now to find out how Amazon changed the home button handling.

Let me know if you can figure out the home button issue. I've tried to debug aswell, can't figure it out

Well the pressing of the Home button is an system event and can't be intercepted, thus I'm sure that Erik knows a lot more about how the Homeservice works. Now if you give the command secure accessibility_enabled 1 it always turns back to 0 after rebooting. Maybe this can be handled like before it did with Hijack and the app-debug.

Well the pressing of the Home button is an system event and can't be intercepted, thus I'm sure that Erik knows a lot more about how the Homeservice works. Now if you give the command secure accessibility_enabled 1 it always turns back to 0 after rebooting. Maybe this can be handled like before it did with Hijack and the app-debug.

Yeah that same issue happened before.
I got my button mapper to say accessibility is enabled with an app I made to trigger it. But accessibility isn't actually enabled.

Well at least for now the stick boots into the custom launcher.

Well at least for now the stick boots into the custom launcher.

Yeah I don't think Amazon could ever patch that

any luck @ErikAbele ?

Ok, I also got the update yesterday and see that Amazon did re-work home button handling so I'll have to dig a little deeper.
Booting into your custom launcher should still work though; if not then try setting it up again in an ADB shell:
settings put secure enabled_accessibility_services de.codefaktor.ftvlaunchx/de.codefaktor.ftvlaunchx.HomeService
settings put secure accessibility_enabled 1

still not working with   FireTV Stick 4K. Fire OS <IP_ADDRESS>

-You cant use Wolf Launcher.
-Reinstall everything. Use Tv Launcher 3 (VERY IMPORTANT USE THIS, DO NOT OPEN IT) its not selectable in the native FireOS launcher chooser.
-adb shell pm grant de.codefaktor.ftvlaunchx android.permission.WRITE_SECURE_SETTINGS (this is what you use for 4k\tank)
-adb reboot
-after reboot you should have FireOS native launcher chooser pop up. Choose TVL 3 and "always."
again, You cannot use Wolf Launcher for this. Its probably not even "all" FTV LaunchX tbh.

-You cant use Wolf Launcher.
-Reinstall everything. Use Tv Launcher 3 (VERY IMPORTANT USE THIS, DO NOT OPEN IT) Wolf is not selectable in the native FireOS launcher chooser.
-adb shell pm grant de.codefaktor.ftvlaunchx android.permission.WRITE_SECURE_SETTINGS (this is what you use for 4k\tank)
-adb reboot
-after reboot you should have FireOS native launcher chooser pop up. Choose TVL 3 and "always."
again, You cannot use Wolf Launcher for this. Its probably not even "all" FTV LaunchX tbh.

You can use the home button to launch tv launcher 3?
You don't need root right?

Yeah you need root. Sorry should mentioned that.

Yeah you need root. Sorry should mentioned that.

Lol. Yeah this stuff is easy with root

Nobody can get it but me and this other user on <IP_ADDRESS> rooted. I think its cause everyone is using Wolf. This app doesn't work straightforward with root anymore.

Nobody can get it but me and this other user on <IP_ADDRESS> rooted. I think its cause everyone is using Wolf. This app doesn't work straightforward with root anymore.
I also think Amazon may have specifically flagged Wolf since it did work. That could possibly help the dev of this.

Atv launcher pro (which wolf is based off) will work with root.
You can always probably change the package name of wolf if that is what amazon blocked.

Actually ill give ATV a shot. I wanna see if it invokes the launcher select menu.

Well I use ATV Pro without root on <IP_ADDRESS> (Firestick 2nd. gen.) which works more or less like on the 4K.
settings put secure enabled_accessibility_services de.codefaktor.ftvlaunchx/de.codefaktor.ftvlaunchx.HomeService
settings put secure accessibility_enabled 1
Does not work without root. The last command always returns 0. I could however chose ATV as the default launcher. Home button of course always takes me back to the amazon home.
I've decompiled the native Amazon launcher and seen that they changed the way the Home button works, but this would not be a problem as you can also intercept the Home_BUTTON_PRESSED, which of course you can't without root as the accessibility system also has been reworked. So far ATV works, just to take care not to press the Home Button to leave an app.

Well I use ATV Pro without root on <IP_ADDRESS> (Firestick 2nd. gen.) which works more or less like on the 4K.
settings put secure enabled_accessibility_services de.codefaktor.ftvlaunchx/de.codefaktor.ftvlaunchx.HomeService
settings put secure accessibility_enabled 1
Does not work without root. The last command always returns 0. I could however chose ATV as the default launcher. Home button of course always takes me back to the amazon home.
I've decompiled the native Amazon launcher and seen that they changed the way the Home button works, but this would not be a problem as you can also intercept the Home_BUTTON_PRESSED, which of course you can't without root as the accessibility system also has been reworked. So far ATV works, just to take care not to press the Home Button to leave an app.

Hopefully @ErikAbele can figure something out, but it's not looking good.

Well I use ATV Pro without root on <IP_ADDRESS> (Firestick 2nd. gen.) which works more or less like on the 4K.

settings put secure enabled_accessibility_services de.codefaktor.ftvlaunchx/de.codefaktor.ftvlaunchx.HomeService
settings put secure accessibility_enabled 1

Does not work without root. The last command always returns 0. I could however choose ATV as the default launcher. Home button of course always takes me back to the amazon home.

I've decompiled the native Amazon launcher and seen that they changed the way the Home button works, but this would not be a problem as you can also intercept the Home_BUTTON_PRESSED, which of course you can't without root as the accessibility system also has been reworked. So far ATV works, just to take care not to press the Home Button to leave an app.

Its a legit solution. You can use "keymap editor" for kodi, to set it to back out with the back button from the home screen of Kodi.
Netflix exits out with back. So do both Silk and Youtube.

its not though, thats not seamless at all lol

@ErikAbele any news? :(

Sween wolf got a fix for us! https://forum.xda-developers.com/showpost.php?p=83468741&postcount=461

Sorry I was confused when the xda post mentioned "4 shortcuts are included in apps (important ones)
To access full settings you have to enable Amazon launcher just like you disabled it."
It appeared that there was no way to access settings unless you use the amazon launcher.
But it does state that only 4 shortcuts to settings are included.

Sorry I was confused when the xda post mentioned "4 shortcuts are included in apps (important ones)
To access full settings you have to enable Amazon launcher just like you disabled it."
It appeared that there was no way to access settings unless you use the amazon launcher.
But it does state that only 4 shortcuts to settings are included.

It's ok I had the same thought. But yeah every setting that's important is in the settings app

can not install settings app on any of my firesticks.  I get "there was a problem parsing the package"
Also I really do not want the firetv logo on the top of the screen.  The reason I wanted a replacement launcher is to get rid of the amazon junk and advertisements
Please fix FTVlaunchX so I can go back to using my fire sticks again

can not install settings app on any of my firesticks.  I get "there was a problem parsing the package"
Also I really do not want the firetv logo on the top of the screen.  The reason I wanted a replacement launcher is to get rid of the amazon junk and advertisements
Please fix FTVlaunchX so I can go back to using my fire sticks again

Read the above post.

Any progress on getting FTVLainchX to work?  I am so tired of the amazon propaganda garbage

FWIW
pm disable-user com.amazon.firehomestarter
pm disable-user com.amazon.tv.launcher

Seems to work.  But it has side effects, such as losing access to the Settings control.

FWIW
pm disable-user com.amazon.firehomestarter
pm disable-user com.amazon.tv.launcher

Seems to work. But it has side effects, such as losing access to the Settings control.

This works with current ftvlaunchx on <IP_ADDRESS>?

It seems to work with:
ro.build.version.name=Fire OS <IP_ADDRESS> (NS6273/2773)
FTVLaunchX-1.0.1.apk
LeanbackOnFire_1.5-release.apk

on my firestick 4K and my Toshiba Fire-TV edition.

How it going ?  If you need some testing please let me know

Fire OS <IP_ADDRESS>... Same problem, launcher opens at startup but home button doesn't work

Fire OS <IP_ADDRESS>... Same problem, launcher opens at startup but home button doesn't work


Feature: JDBC-like driver

Would be nice to have a JDBC-like driver (so one can access an RDF-store in reporting tools etc) similar to Jena's JDBC framework.
No SPARQL-to-SQL translation needed, just the connection and getting results from a SPARQL query.

what about using Calcite?

Thanks for the link. I'm not familiar with Calcite, but it looks like Calcite either needs a JDBC connection (so this enhancement would come in handy) or a yet-to-be-written RDF4J adapter (which would also be useful, and perhaps less work) ?

I have to study it better, but it seems something that should simplify the
development of an adapter
2017-07-06 7:46 GMT+02:00 Bart Hanssens<EMAIL_ADDRESS>
Thanks for the link. I'm not familiar with Calcite, but it looks like
Calcite either needs a JDBC connection (so this enhancement would come in
handy) or a yet-to-be-written RDF4J adapter (which would also be useful,
and perhaps less work) ?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/eclipse/rdf4j/issues/838#issuecomment-313301835, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAFYfBPI5gCF0yxmb-sOVlhfFjMStB9Vks5sLHS8gaJpZM4NrNKy
.

We at Ontotext also want a JDBC driver, as mentioned at https://docs.google.com/document/d/1guwFHi9p4-ujFkrHF6dwMUZndzCmlX_gPyiBi6JlPTs/pub#h.etad8qvxqd6t (2017)

Jena: http://jena.apache.org/documentation/jdbc/
(older) jdbc4sparql: https://code.google.com/archive/p/jdbc4sparql/
(older) SCON: https://code.google.com/archive/p/scon/wikis/Introduction.wiki

For the record, here's the doc of GraphDB's JDBC "connector": https://graphdb.ontotext.com/documentation/10.0/sql-access-over-jdbc.html. It uses Apache Calcite and "stored SPARQL views" exposed as relational tables

@seralf I doubt this will be open sources, but I'll ask


I did remove all the broken links

[x] My submission is formatted according to the guidelines in the contributing guide
[x] My addition is ordered alphabetically
[x] My submission has a useful description
[x] The description does not have more than 100 characters
[x] The description does not end with punctuation
[x] Each table column is padded with one space on either side
[x] I have searched the repository for any relevant issues or pull requests
[x] Any category I am creating has the minimum requirement of 3 items
[x] All changes have been squashed into a single commit

Hi @aliefe04
Thanks for your contribution ;)
We have some problems with Public APIs, so we won't be able to review/add this improvement right now.
Read this issue for more information, please: https://github.com/public-apis/public-apis/issues/3104

@aliefe04 Excuser API link is updated, it is no longer broken.


Feature/1410 show hide fields and dialog tabs based on dropdown and checkbox selection

Coverage remained the same at 40.7% when pulling 2ee63340d51de6ef6cd93a208370365decbff818 on muhammad-naveed:feature/1410_Show-hide-fields-and-dialog-tabs-based-on-dropdown-and-checkbox-selection into a4f1569c5e9aabc427b3993a1fa33b358157fa7a on Adobe-Consulting-Services:master.

Hi @davidjgonzalez , do you think the functionality in the PR is generic enough to be included in ACS Commons? Would really appreciate your comments!

Hi @schoudry, it would be great if you would find some time to review the pull request, Thanks

@muhammad-naveed good extension and sorry for the delay! would it be possible to share the dialog package and possibly a video recording to get a sense of how it works (in addition to the doc comments)

@schoudry thanks for your response. I have created a test component (named: Test) with the dialog box. You can download the package from this link: https://drive.google.com/open?id=12UQim-1aHVM6TmkJWwpkn1zQY8s0d4GH
It can be used on WeRetail pages.
The dialog box contains two Tabs namely Tab 1 and Tab 2. Initially, only Tab 1 displayed. The first Tab contains a drop down. If you select option 1, then a checkbox is displyed. If option 1 is selected and checkbox is checked than two field are shown and Tab 2 is displyed. Out of these fields, one has required field validation. This is to show the case when required field is hidden, then the form can be submitted with the custom validator i.e. requiredvalidation.js.
Other options are self explainatory. Do let me know if you require some more information.

@muhammad-naveed thank you, currently on PTO, will check it upon my return, sep 1
(@davidjgonzalez can someone else take a look? )

Hi @justinedelson,
I have made the changes based on your review comment and also updated test content package the instructions to test it
https://github.com/Adobe-Consulting-Services/acs-aem-commons/pull/1412#issuecomment-410538459

Hi @justinedelson , @schoudry,
would be great if you can have a look at the changes.
Thanks

@muhammad-naveed looks good to me; i guess you could replace $(document) occurrences with $document by passing $(document) to the outer closure

@muhammad-naveed looks good to me; i guess you could replace $(document) occurrences with $document by passing $(document) to the outer closure

@schoudry I have replaced $(document) occurrences with $document

@justinedelson @schoudry If you're happy with the changes, can you please approve the PR. Thanks

Hi @justinedelson , I had done the changes based on your review comments. It would be great if you could find the time to have a look. If any other change is needed, do let me know and if everything is fine, please approve the PR. Thanks!


show date (or number) range for small time steps

When creating a torque map for a small number of steps, it would be good to show the date range in a legend or in the slider bar. For example, if the data is aggregated into three time steps from 1 Jan - 31 Jan, the slider would show 1 Jan - 10 Jan for the first step, 11 Jan - 20 Jan for the second, and 21 Jan - 31 Jan for the third. For a one step map, the slider could turn into a legend showing the date range for the data.
@fdansv what do you think?

@ohasselblad what would you consider a reasonable minimum range to show this instead of just one date?

What about this?

One day (ignoring hours) when the time is not being shown: 1 Jan - 2 Jan
On the level of a day, maybe show a range in hours? 1 Jan, 08:00 - 09:00

I've done it for dates so far:

Whoops, accidental close

Well actually this is a cartodb.js issue, I'll open a PR there.

This was added in #450


How does it compare with ffmpeg.js?

Hey! It would be nice if there was some explanations about the differences with ffmpeg.js
There is a big warning at the beginning of the README, but nothing more than that.

Yes please address this in the README.

Both ffmpeg.wasm and ffmpeg.js are built using emscripten, but ffmpeg.js does not output WASM.
This is because ffmpeg.js is built with the WASM=0 flag, and you can also check this by npm install and tree.
ffmpeg.js is probably outputting asm.js, which can be very slow.

https://github.com/Kagami/ffmpeg.js/blob/v4.2.9003/Makefile#L223
https://github.com/emscripten-core/emscripten/blob/2.0.5/src/settings.js#L1056-L1066

$ npm install<EMAIL_ADDRESS>$ tree node_modules/ffmpeg.js
├── COPYING.GPLv2
├── COPYING.LGPLv2.1
├── LICENSE.MP4
├── LICENSE.WEBM
├── README.md
├── ffmpeg-mp4.js
├── ffmpeg-webm.js
├── ffmpeg-worker-mp4.js
├── ffmpeg-worker-webm.js
└── package.json

On the other hand, ffmpeg.wasm uses WASM threads and is fast, but requires SharedArrayBuffer which is difficult to implement due to Spectre, so it works only with the latest Chrome.

https://developers.google.com/web/updates/2018/10/wasm-threads
https://v8.dev/blog/spectre

Thanks a lot for the explanations.
For what it's worth, I did a quick comparison with both libraries, and I found out that this one is indeed faster (about 2x faster for my use case).
Unfortunately in my use case, it's still too slow to be usable in production. I'm following https://github.com/ffmpegwasm/ffmpeg.wasm/issues/84 to see if this could be another performance improvement.

I took a stab at porting som of the SIMD code for x264 to intrinsics (https://github.com/AlexVestin/x264-simd). There's definitely some room for speedups, though I couldn't get it match the native (relative) performance. Also since it's not released yet (although it seems to be coming soonisch) it might be worth to wait for something like WebCodecs

@AlexVestin Looking forward to x264-simd! I believe it will be a huge speed up to ffmpeg.wasm!


Sun disk position is not adjusted for sky_rotation

Godot version
4.0.1.stable.arch_linux
System information
Arch Linux x86_64
Issue description
Setting the sky_rotation property of a Sky to a nonzero rotation changes the horizon, but also moves the sun disk along with the rotated sky. I would expect the sun disk to match the global rotation of the directional light in the scene.
This is problematic for games that take place on a sphere, because they need a nonzero sky rotation unless the camera is on the north pole.
Steps to reproduce

Add a Camera3D, DirectionalLight and some geometry so you can see the effect of lighting.
Give the Camera3D an Environment, set its background mode to Sky, and create a ProceduralSkyMaterial or PhysicalSkyMaterial for it.
Enable camera preview mode.
Change the rotation on the sky material, for example the y component. Notice how the sun disk moves in the sky, but the lighting on the geometry remains the same.

For example, here you can see that the sphere is lit from behind and a little to the right, but the sun disk appears to the left:

Minimal reproduction project
sky_rotation.zip

This can likely be fixed by subtracting the sky rotation when drawing the sun disks (it needs to be exposed to sky shaders).

I'm not sure I understand the use-case well enough to determine if this is a bug or if this is a misuse of a feature.
It sounds like you are rotating the sky in order to keep the "UP" direction of the sky in line with the player's local "UP" direction as they circumnavigate a spherical planet. I wonder if there are other ways to achieve that without rotating the sky (I imagine you are having to manually control a lot of other values like Gravity, Camera-Y-UP, etc.) as this breaks a lot of assumptions the engine makes (primarily the fact that +Y is up). The reason I bring that up is many other rendering effects will also break (e.g. height-based fog, SDFGI) and at a certain point it may be better to revisit your approach.
The intended behavior for sky_rotation is to rotate the entire sky sphere, it is done at mesh draw time (rather than when running the sky shader). If we were to change the sky itself based on the rotation, then we would have to update the radiance cubemap every time that sky_rotation was changed. To be clear, I am not saying that is a reason not to fix this issue. I am just pointing out that there is a performance tradeoff here. The expected behaviour would result in substantial performance loss in the OP's project.
At any rate, this deserves more discussion.
In the meantime, you can work around this issue by using two DirectionalLight3Ds and setting one to "light only" and the other to "sky only"

Whether the sun should move or not, it should at least be consistent with the light cast from the directional light that puts it there.

Confirmed this and noted that shadows cast become misaligned with the sun disk as well (which is likely assumed but figured I'd state it anyway)
Aside, my solution for a day/night/weather cycle was to offset each frame an alpha noise sky cover texture imitating clouds moving, instead of the sky itself. Adjusting noise color ramps thickens or clears the sky (for westher changes). then I only rotated the sunlight on the x to keep it simple. Not sure if that's handy

I've got a similar use-case. Two other things that would be useful along the same lines:

Ability to offset where the horizon is. When very high above a planet, the horizon is below the horizontal. I imagine there are other use-cases where this would be useful for artistic reasons.
Ability to rotate the night-sky image independently from the rest of the sky.

EDIT: And two suns, of course :)

This is supported with ProceduralSkyMaterial but not PhysicalSkyMaterial for performance reasons.

@Calinou Thanks! That's really useful.
I managed to get what I wanted by two parameters to the shader. The up parameter defines the direction the sky, and horizon_angle defines how far the horizon is from horizontal (in radians):
shader_type sky;
render_mode use_debanding;

uniform float rayleigh : hint_range(0, 64) = 2.0;
uniform vec4 rayleigh_color : source_color = vec4(0.3, 0.405, 0.6, 1.0);
uniform float mie : hint_range(0, 1) = 0.005;
uniform float mie_eccentricity : hint_range(-1, 1) = 0.8;
uniform vec4 mie_color : source_color = vec4(0.69, 0.729, 0.812, 1.0);

uniform float turbidity : hint_range(0, 1000) = 10.0;
uniform float sun_disk_scale : hint_range(0, 360) = 1.0;
uniform vec4 ground_color : source_color = vec4(0.1, 0.07, 0.034, 1.0);
uniform float exposure : hint_range(0, 128) = 1.0;

uniform vec3 up = vec3(0.0, 1.0, 0.0);
uniform float horizon_angle : hint_range(0, 1.5708);

uniform sampler2D night_sky : filter_linear, source_color, hint_default_black;

// Optical length at zenith for molecules.
const float rayleigh_zenith_size = 8.4e3;
const float mie_zenith_size = 1.25e3;

float henyey_greenstein(float cos_theta, float g) {
	const float k = 0.0795774715459;
	return k * (1.0 - g * g) / (pow(1.0 + g * g - 2.0 * g * cos_theta, 1.5));
}

void sky() {
	if (LIGHT0_ENABLED) {
		float angle_fac = (horizon_angle / (PI/2.0));
		float dot_up_dir = (dot(up, EYEDIR) + angle_fac) / (1.0 + angle_fac);

		float zenith_angle = (clamp(dot(up, normalize(LIGHT0_DIRECTION)), -1.0, 1.0) + angle_fac) / (1.0 + angle_fac);
		float sun_energy = max(0.0, 1.0 - exp(-((PI * 0.5) - acos(zenith_angle)))) * LIGHT0_ENERGY;
		float sun_fade = 1.0 - clamp(1.0 - exp(LIGHT0_DIRECTION.y), 0.0, 1.0);

		// Rayleigh coefficients.
		float rayleigh_coefficient = rayleigh - ( 1.0 * ( 1.0 - sun_fade ) );
		vec3 rayleigh_beta = rayleigh_coefficient * rayleigh_color.rgb * 0.0001;
		// mie coefficients from Preetham
		vec3 mie_beta = turbidity * mie * mie_color.rgb * 0.000434;

		// Optical length.
		float zenith = acos(max(0.0, dot_up_dir));
		float optical_mass = 1.0 / (cos(zenith) + 0.15 * pow(93.885 - degrees(zenith), -1.253));
		float rayleigh_scatter = rayleigh_zenith_size * optical_mass;
		float mie_scatter = mie_zenith_size * optical_mass;

		// Light extinction based on thickness of atmosphere.
		vec3 extinction = exp(-(rayleigh_beta * rayleigh_scatter + mie_beta * mie_scatter));

		// In scattering.
		float cos_theta = dot(EYEDIR, normalize(LIGHT0_DIRECTION));

		float rayleigh_phase = (3.0 / (16.0 * PI)) * (1.0 + pow(cos_theta * 0.5 + 0.5, 2.0));
		vec3 betaRTheta = rayleigh_beta * rayleigh_phase;

		float mie_phase = henyey_greenstein(cos_theta, mie_eccentricity);
		vec3 betaMTheta = mie_beta * mie_phase;

		vec3 Lin = pow(sun_energy * ((betaRTheta + betaMTheta) / (rayleigh_beta + mie_beta)) * (1.0 - extinction), vec3(1.5));
		// Hack from https://github.com/mrdoob/three.js/blob/master/examples/jsm/objects/Sky.js
		Lin *= mix(vec3(1.0), pow(sun_energy * ((betaRTheta + betaMTheta) / (rayleigh_beta + mie_beta)) * extinction, vec3(0.5)), clamp(pow(1.0 - zenith_angle, 5.0), 0.0, 1.0));

		// Hack in the ground color.
		Lin  *= mix(ground_color.rgb, vec3(1.0), smoothstep(-0.1, 0.1, dot_up_dir));

		// Solar disk and out-scattering.
		float sunAngularDiameterCos = cos(LIGHT0_SIZE * sun_disk_scale);
		float sunAngularDiameterCos2 = cos(LIGHT0_SIZE * sun_disk_scale*0.5);
		float sundisk = smoothstep(sunAngularDiameterCos, sunAngularDiameterCos2, cos_theta);
		vec3 L0 = (sun_energy * extinction) * sundisk * LIGHT0_COLOR;
		L0 += texture(night_sky, SKY_COORDS).xyz * extinction;

		vec3 color = Lin + L0;
		COLOR = pow(color, vec3(1.0 / (1.2 + (1.2 * sun_fade))));
		COLOR *= exposure;
	} else {
		// There is no sun, so display night_sky and nothing else.
		COLOR = texture(night_sky, SKY_COORDS).xyz;
		COLOR *= exposure;
	}
}

For my use-case an "up" vector was a lot easier to get than Euler angles, so this is actually more convenient.
I still ended up using two lights though, just to artificially dim the directional light to match the sun fading in the sky.


Virtual CSS modules

Description
From discord:

Okay, so it looks like that fixed one problem, but another problem showed up. It looks like the null character from the virtual module isn't being handled properly (running with pnpm run dev). I've created a reproduction of the bug here https://stackblitz.com/edit/node-usvd3r?file=vite-ssr-project/pages/about/index.page.vue. The button on the about page should have red text, and it does sometimes at page load for a very brief moment, but it immediately goes away after.

Error Stack
No response

Requesting to reopen:
I updated to the latest version of vike and have resolve problems in dev mode:
In dev mode I get this html served, where you can see alot of <link rel="stylesheet" type="text/css" href="/@id/plugin-vuetify:components/...?direct"> (screenshot 1).

Those can all not be resolved and end up in a 404 (screenshot 2).

Later on the assets are properly loaded using the correct URL (screenshot 3).

The URLS differe like this:

http://localhost:3000/@id/plugin-vuetify:components/VBtn/VBtn.sass?direct
http://localhost:3000/@id/__x00__plugin-vuetify:components/VBtn/VBtn.sass

According to this https://vitejs.dev/guide/api-plugin.html#virtual-modules-convention this seems to be a virtual plugin transformation.
This transformation is not properly happening on serverside dev rendering.
I am using vite-plugin-vuetify to dynamically load those vuetify components required, reducing bundle size.
This only is happening in dev mode (using vite middleware).
A repro can be found here https://github.com/IT4Change/boilerplate-frontend/ (commit 28d80346a87ff8d6635c8100a26f7358ccc7ad9d in case things get fixed there)
This seems to be the right issue describing the problem: https://github.com/vikejs/vike/issues/479
Any hint how to solve this? :heart:

A fix, which might not be the best way to solve things, but works on my machine:
 else if (mod.url.startsWith('\0')) {
            // Virtual modules
            //  - https://vitejs.dev/guide/api-plugin.html#virtual-modules-convention
            //    - I believe some Vite plugins don't respect the \0 virtual module convention. What should we do then?
            //  - https://github.com/vikejs/vike/issues/1327
            //  - https://github.com/vikejs/vike/commit/3f7b9916dddc84e29e2c20d2b0df7211b6f1acbd
            styleUrls.add(`/@id/__x00__${mod.url.substring(1)}`);
        }

I added the __x00__.
This is in the file node_modules/vike/dist/esm/node/runtime/renderPage/getPageAssets/retrieveAssetsDev.js  line 46.
In the sources it would be: https://github.com/vikejs/vike/blob/main/vike/node/runtime/renderPage/getPageAssets/retrieveAssetsDev.ts#L49

https://github.com/vikejs/vike/commit/3d2561887127a5ba8564ba1a98309c77a5325922 pre-released in 0.4.152-commit-3d25618. Let us know if you still run into issues.

Works like a charm @brillout - no more errors with release 0.4.152-commit-3d25618
Tho I would suggest a code change to match the code @vite - see https://github.com/vitejs/vite/blob/56ae92c33cba6c86ab5819877c19b9ea39f7121b/packages/vite/src/node/utils.ts#L71 for reference
styleUrls.add(`/@id/${mod.url.replace('\0','__x00__'}`)

Consider to explain __x00__:
VITE_NULL_BYTE_PLACEHOLDER = '__x00__ '
...
styleUrls.add(`/@id/${mod.url.replace('\0',VITE_NULL_BYTE_PLACEHOLDER}`)

Why is this important? As far as I read the code @vite it would replace any occurrence of the \0 character not only the first.

Vite does .replace() not .repalceAll(), so it only replaces the first occurence.
The fix adds a link to your comment, I think it's good enough.
Thanks for the bug report and digging into a fix.
In case your compnay is up for it: Sponsor Vike.


Improve Appwrite integration page

Hey there 👋
The page https://docs.astro.build/en/guides/backend/appwriteio/ is currently a stub.
It could do with some more information along with a basic guide running through how to integrate Astro and Appwrite.
Will follow up with a PR :)

Hi @loks0n!
We do intend for those stub pages to be PR'd, yup! And we'd love to get more content there!
Please do read carefully our writing/style guide as well as our guide to submitting recipes, both linked to from https://docs.astro.build/en/contribute/. Please also look at other existing backend/CMS guides to get a sense of what we expect, both in terms of structure and voice/ton, and you could even go back and look over the PRs for those guides to see what feedback we have given to others in the past. We do have pretty particular requirements and standards for the content on these pages, so this often takes a bit of time and effort to massage into our format.
Otherwise, if you've written something already somewhere, we are happy to more quickly accept a PR with a link to it under a "Community Contributions" heading, and then we can also look at it and offer some general feedback before turning it into a PR.
Just wanted to let you know that submitting one of these guides can be a bit of a project, so buckle up!  :rocket:

Thanks @gewenyu99 ! I realized I never closed this issue after the discussion concluded, so just doing that now!  :raised_hands:


Fix error for changing favicon on external login page

Description
Fixed the problem that the external client does not work when setting the change to the desired favicon on the external login page.
And after logging in, updated to set the client name in the modal contents displaying the connection information.
Related Issue : #3192
How Has This Been Tested?
The test method is the same as #3192.

You should check a favicon image matching the "faviconPath" attribute.

You should check a name that matches the "clientName" attribute.

Need additional checks?
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have read the CONTRIBUTING document.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[x] I have added tests to cover my changes.

Additional Context

deploy to 1

deploy to 1


Add parking spaces/prices

Seems like we've agreed this is already solved by another app and is more of a want than a need.

I think that since it's handled by another website we can at least add a link to it on our website, as a consolidation of resources available to students.