Fuchsia Dashboard leaks memory overnight

Steps to Reproduce

Run Fuchsia.
Launch Fuchsia Dashboard (source: https://fuchsia.googlesource.com/topaz/+/master/app/dashboard/)
Wait overnight.

Eventually, the flutter:dashboard process will consume all available memory (in the case of this mechine, 6GB) and be killed by the OOM killer.

FYI @zanderso @collinjackson who both have expressed interest in supporting Flutter on Fuchsia.

@abarth Can the OOM killer dump the memory map before killing a job?

@zanderso We were just talking about how that would be useful!  Unfortunately, it doesn't do that.
We don't need to push it all the way to the wall to get the memory map.  The endurance test is running again.  We can grab a memory map from it in a few hours to see what's going on.  If it's not obvious, then we can schedule memory map dumps overnight.

After three hours of leaking:
before:
P flutter:dashboard | 350.1M | 367059336 | 100.000% | 4.332%
* V jemalloc-heap | 101.7M | 106635264 | 29.051% | 1.259%
* V magma_create_buffer[342] | 88.2M | 92504064 | 25.201% | 1.092%
* V dart-newspace dashboard:main()[2] | 64M | 67108864 | 18.283% | 0.792%
* V dart-oldspace dashboard:main()[118] | 29.4M | 30842880 | 8.403% | 0.364%
* V flutter_jit_runner | 28.0M | 29376512 | 8.003% | 0.347%
* V dart-profiler | 10.5M | 11042816 | 3.008% | 0.130%
* V dart-oldspace vm-service[36] | 9740k | 9973760 | 2.717% | 0.118%

after:
P flutter:dashboard | 363.6M | 381284739 | 29.832% | 4.500%
* V jemalloc-heap | 110.6M | 115937280 | 30.407% | 1.368%
* V magma_create_buffer[344] | 92.4M | 96870400 | 25.406% | 1.143%
* V dart-newspace dashboard:main()[2] | 64M | 67108864 | 17.601% | 0.792%
* V dart-oldspace dashboard:main()[121] | 29.8M | 31182848 | 8.178% | 0.368%
* V flutter_jit_runner | 28.0M | 29376512 | 7.705% | 0.347%
* V dart-profiler | 10.5M | 11042816 | 2.896% | 0.130%
* V dart-oldspace vm-service[36] | 9740k | 9973760 | 2.616% | 0.118%

after 4.5 hours:

entity
memory used
memory used in bytes

P flutter:dashboard
364.2M
381839747

V jemalloc-heap
114.2M
119754752

V magma_create_buffer[343]
89.0M
93358080

V dart-newspace dashboard:main()[2]
64M
67108864

V dart-oldspace dashboard:main()[122]
30.2M
31674368

V flutter_jit_runner
28.0M
29376512

V dart-profiler
10.5M
11042816

V dart-oldspace vm-service[36]
9740k
9973760

Looks like the leak is in jemalloc-heap.

Can we shut off various bits of the dashboard app to try to eliminate some possibilities? For example, shutting off the UI, or using mock data instead of actual http requests.

Yes, we're going to try a build without networking.

We haven't been able to reproduce the leak at this velocity.  Let's keep this issue open for a while to see if we can reproduce it again.

I had some issues creating a test that reproduces the quick leak do to an unrelated bug.  I made the dashboard continually update but without networking and saw the following memory behaviour over a short period of time.
2017-11-27T14:17:31-0800 257679360
2017-11-27T14:18:32-0800 261615616
2017-11-27T14:19:33-0800 274214912
2017-11-27T14:20:33-0800 285057024
2017-11-27T14:21:33-0800 288759808
2017-11-27T14:22:33-0800 290541568
2017-11-27T14:23:34-0800 291065856
2017-11-27T14:24:34-0800 291696640
2017-11-27T14:25:34-0800 292151296
2017-11-27T14:26:35-0800 292675584
2017-11-27T14:27:35-0800 296411136
2017-11-27T14:28:36-0800 296456192
2017-11-27T14:29:37-0800 296476672
2017-11-27T14:30:38-0800 297058304
2017-11-27T14:31:39-0800 297058304
2017-11-27T14:32:40-0800 297078784
2017-11-27T14:33:41-0800 297070592
2017-11-27T14:34:43-0800 297226240
2017-11-27T14:35:44-0800 297320448
2017-11-27T14:36:45-0800 297451520
2017-11-27T14:37:46-0800 297607168

You can see it leaks quickly at the beginning and then slows down.  My theory is the framerate drops over time so the leaking slows.

I switched to a release build (the previous numbers were from a debug build) in the hopes that a higher framerate would leak faster:
2017-11-27T14:45:55-0800 250617856
2017-11-27T14:46:55-0800 257036288
2017-11-27T14:47:56-0800 268750848
2017-11-27T14:48:56-0800 272285696
2017-11-27T14:49:57-0800 278020096
2017-11-27T14:50:58-0800 276627456
2017-11-27T14:51:58-0800 276738048
2017-11-27T14:52:59-0800 276754432
2017-11-27T14:53:59-0800 280948736
2017-11-27T14:55:00-0800 282619904
2017-11-27T14:56:01-0800 282624000
2017-11-27T14:57:02-0800 282624000
2017-11-27T14:58:02-0800 284676096
2017-11-27T14:59:03-0800 283148288
2017-11-27T15:00:03-0800 283148288
2017-11-27T15:01:04-0800 282927104
2017-11-27T15:02:05-0800 287354880
2017-11-27T15:03:05-0800 287498240
2017-11-27T15:04:06-0800 287891456
2017-11-27T15:05:06-0800 287887360
2017-11-27T15:06:07-0800 287735808
2017-11-27T15:07:08-0800 287891456
2017-11-27T15:08:09-0800 287985664
2017-11-27T15:09:09-0800 288161792
2017-11-27T15:10:10-0800 288755712
2017-11-27T15:11:10-0800 288956416
2017-11-27T15:12:11-0800 288653312
2017-11-27T15:13:12-0800 288694272
2017-11-27T15:14:13-0800 288223232
2017-11-27T15:15:13-0800 288628736
2017-11-27T15:16:14-0800 289230848
2017-11-27T15:17:15-0800 292073472
2017-11-27T15:18:16-0800 289673216
2017-11-27T15:19:16-0800 312344576
2017-11-27T15:20:17-0800 307277824
2017-11-27T15:21:18-0800 307441664
2017-11-27T15:22:18-0800 307539968
2017-11-27T15:23:19-0800 306753536
2017-11-27T15:24:20-0800 308072448
2017-11-27T15:25:20-0800 308584448
2017-11-27T15:26:21-0800 312524800
2017-11-27T15:27:22-0800 308846592
2017-11-27T15:28:23-0800 317153280

I ran it overnight, at some point this morning someone rebooted my test device but here's what it looked like before that happened:

Chart from running all day today:

What is the mechanism for turning off networking? What is the time interval between the peaks? Is it related to any hard-coded interval in the program? Do both jemalloc-heap and dart-oldspace oscillate, or is it just one or the other?
If dart-oldspace is large, then the allocation profile in the Observatory can tell us the types of the objects being leaked, and possibly also show retaining paths to the objects.

@zanderso It's the same patch that broke the safe stack thing in dart earlier this month: https://fuchsia-review.googlesource.com/c/topaz/+/89461  I just run the dashboard with that patch

We discovered I was effectively adding 24 new 10 second timers every minute which explains at least some of the leak in the dart heap.  I think the jemalloc leak is unrelated tho.  I've updated my test to not leak the timers and we'll see what comes out of it

cc @devoncarew @mit-mit for user journey around debugging and solving memory leaks

An interesting chart of the last run.  At some point the app gives up and locks up.  When we tried to connect to the observatory everything timed out ontil @zanderso when and touched the app.  That seemed to wake it up.  Also, we manually ran the GC from the observatory and saw it cleaned a lot up around the same time:

For @devoncarew and @mit-mit, we used the allocation profile in the Observatory to discover that lots of callbacks in the timer heap were piling up. @apwilson then manually inspected how timers in the app were used, and discovered the above issue. The feedback here for user journeys is probably that the UI for the Observatory's allocation profile could be more targeted at Dart programmers instead of VM engineers.

The feedback here

awesome, thanks for the description of the process of finding the leak; /cc @stereotype441

/cc @rmacnak-google

We were exploring the timer heap in Observatory, the timer objects we looked at (the ones at the beginning of the list) all had null callbacks. We initially thought this might be cancelled timers not being removed from this list. But after looking at the timer heap implementation and exploring a smaller test program with lots of timers, I think the callbacks were null just because the timers had fired between the time we first visited the timer heap in Observatory and the time we visited the individual timer objects. I.e., if an isolate is not paused, what you see when expanding objects doesn't necessary represent one snapshot in time.

So I'm still seeing a leak, but with a lot less variablility:

@kulakowski thinks this might be an issue in the Fuchsia port of jemalloc.

Since newspace looks maxed out, a 3MB increase in oldspace isn't necessarily a leak. It may just be objects surviving newspace GC, but not so many that it triggers oldspace GC. (The oldspace policy is currently biased towards heap growth over GC, but I believe the plan going forward is to schedule GC during otherwise idle periods.)

To me it definitely looks like something repeatedly allocates a memory pool, uses it all up then doubles its allocation.

I spoke with @abarth and @zanderso.  @zanderso is happy to look further, but would really like the heap profiling support in jemalloc to be turned on in Fuchsia's port first:
https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling @kulakowski likely knows how or who to do that.

I can test things if people want to suggest fixes.  I've reached the end of what I can contribute without that tho.
I guess we could create a reduced test case, but due to the time spans involved it'd be a slow, involved process.
If anyone wants to try, I think the key is to have a flutter element always updating.  In my case I had 20 or so spinners constantly spinning.

I instrumented the Dart VM to spit out stats from jemalloc (with malloc_stats_print) every 1000 crossings between the runtime and generated code:
https://dart-review.googlesource.com/#/c/sdk/+/28461
Output looks like:
Allocated: 25372840, active: 28979200, metadata: 19585632, resident: 109907968, mapped: 148393984, retained: 26439680

where the fields have the following meanings:

Allocated: Total number of bytes allocated by the application.
active: Total number of bytes in active pages allocated by the application.
metadata: Total number of bytes dedicated to metadata
resident: Maximum number of bytes in physically resident data pages mapped by the allocator, comprising all pages dedicated to allocator metadata, pages backing active allocations, and unused dirty pages
mapped: Total number of bytes in active extents mapped by the allocator
retained: Total number of bytes in virtual memory mappings that were retained rather than being returned to the operating system

Given these descriptions, I would expect allocated + metadata + retained to be reasonably close to either resident or mapped, but in the logs I'm seeing, allocated + metadata + retained is significantly less. In the example above it is: 71398152.
Additionally, in running an unpatched dashboard, "allocated" and "active" go up and down and generally stay below 30MB. "metadata" and "resident" also appears stable. However, "mapped" does not ever appear to shrink. Here is an entry from several minutes later:
Allocated: 27296368, active: 30715904, metadata: 20028448, resident: 101961728, mapped: 329338880, retained: 34508800

Given this information, it looks as though there might be two problems:

jemalloc is failing to unmap pages, or maybe even completely losing track of some pages.
jemalloc's policies around retaining unused mappings may need to be revisited. There also seems to be a lot of metadata, but I'm not sure what to say about that.

After running the unpatched dashboard for ~1h, it's at:
Allocated: 32366224, active: 36335616, metadata: 22234864, resident: 136491008, mapped: 898494464, retained: 39157760

Now, I'll run the patched dashboard overnight, and see if there is anything interesting in the logs tomorrow.

After running the patched dashboard overnight, it looks like there is a real leak at the rate of ~10MB every 5 or 6 hours:

However, I'm not seeing the large jumps in total memory usage that @apwilson saw in his run. PSS for dart:dashboard is at about 280MB after ~20 hours.

@zanderso what are the next steps here?

We've got a new build of the dashboard.  I need to characterize the leak more carefully in the new dashboard.

@abarth Any luck with your new build?

Let's close this for now.  The bigger problem is the file descriptor leak.  I'll file a separate bug for that.


Allow DeepSpeed models to be initialized with optimizer=None

The optimizer uses quite a bit of GPU memory. For training, this is necessary, but not for simply running the model. Therefore, allowing optimizer to be set to None can dramatically reduce GPU memory usage when running the model. However, in the current version of the code, doing so will cause an exception to be thrown.

Hi @gcooper-isi , thanks for this contribution! I pushed a small commit that adds a unit test and also avoids an issue when no optimizer is specified in the JSON configuration.
If you haven't yet, can you sign the CLA from the above bot? We will need it before merging.

Thanks so much for adding the unit test! I have asked my employer for permission to sign the CLA, which I don't expect to be an issue, and I will sign it as soon as they say it's OK.

@gcooper-isi just wanted to bump this back up. Okay to sign the CLA?

@gcooper-isi just wanted to bump this back up. Okay to sign the CLA?

I'm OK to sign it personally, but since this requires permission from my company it has had to work its way through the various levels of bureaucracy before they finish the approval. I keep getting periodic updates that it has made it through each next step and I'll continue to follow up.

@gcooper-isi just wanted to bump this back up. Okay to sign the CLA?

I'm OK to sign it personally, but since this requires permission from my company it has had to work its way through the various levels of bureaucracy before they finish the approval. I keep getting periodic updates that it has made it through each next step and I'll continue to follow up.

As far as my contracts team is concerned, though, I don't have a deadline for signing this contract. If you can provide any information that could help convince them that doing this in a timely way would be important, that may help speed things along.

@gcooper-isi just wanted to bump this back up. Okay to sign the CLA?

I'm OK to sign it personally, but since this requires permission from my company it has had to work its way through the various levels of bureaucracy before they finish the approval. I keep getting periodic updates that it has made it through each next step and I'll continue to follow up.

As far as my contracts team is concerned, though, I don't have a deadline for signing this contract. If you can provide any information that could help convince them that doing this in a timely way would be important, that may help speed things along.

@gcooper-isi DeepSpeed is updating at a rapid pace. You might tell them that if they are too slow the risk is that your contributions might be too out of date to be successfully merged. We all see value in your contributions, so I hope they agree soon. The CLA as far as I understand shouldn’t be too controversial for this type of commit, but of course I’m not a lawyer :)

I finally got permission from my employer to sign the CLA and have done so. Please let me know if there is anything else I can do.

I finally got permission from my employer to sign the CLA and have done so. Please let me know if there is anything else I can do.


Keep self out of testDependencies

Now that DT is a pnpm monorepo, getTestDependencies is no more. Closing.


update README.md

Hello,
When using Python 3, it would be better to replace "Fürtinger" by "Fuertinger" in README.md.
If not, Python complains about the UTF-8 which could be fixed by writing
         with open(filename, encoding='UTF-8') as fd:

Regards,
Stefan

Hi Stefan,
my Python 3.4.3 does not complain - what error do you get? Anyway, have a look at #25 if that works for you.

Hello,
This is the error. pypandoc itself is not an issue, since it's just a warning.
warning: pypandoc module not found, could not convert to RST!
Traceback (most recent call last):
  File "setup.py", line 7, in read_as_rst
    from pypandoc import convert
ImportError: No module named 'pypandoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 23, in <module>
    LONG_DESCRIPTION    = read_as_rst('README.md')
  File "setup.py", line 14, in read_as_rst
    out = fd.read()
  File "[...]/sl6/Python/3.5.1-intel-2016a/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1047: ordinal not in range(128)

As indicated, #25 is a solution indeed. Thanks!

OK, merged!


Wait longer for first tip in restore-bench

Issue Number

ADP-846
Overview

[x] Wait longer for first node tip in restore bench

Comments
Attempt to fix this timeout:
https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds/964#fb8228aa-158e-4517-b5bb-883e2be30b24
[bench-restore:Notice:15] [2021-05-31 01:01:17.09 UTC] [cardano-node.66880] withBackend action done. Terminating child process
restore: UnliftIO.Exception.throwString called with:

Gave up in waitForNodeSync, waiting a tip
Called from:
  throwString (bench/Restore.hs:777:18 in main:Main)

It might just be the node re-validating its entire state, taking a lot
of time. In this case, waiting longer should help.
Update
https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds/966#32b592d5-567d-41b9-91bb-d99d89209506
After a while we start seeing Initial node synchronization: still restoring (97.04%), so it seems to work.

bors r+

Bro you are my home by!;I don't know how that wallets on me something to do
with, it MEW cx does that me I can re claim them?.
On Mon, May 31, 2021, 9:25 AM iohk-bors[bot] @.***>
wrote:

Merged #2681 https://github.com/input-output-hk/cardano-wallet/pull/2681
into master.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/input-output-hk/cardano-wallet/pull/2681#event-4821235010,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ATTBKNTNKLAOAXO2VOI6JCLTQOS6NANCNFSM452VCUPA
.

Fixed this error, but the previous heap exhausted error remains.

Two hours is quite a while.
But usually the chain should be cached on the benchmark machine between runs

Yes. We should mostly wait for either the ledger-validation, or full synchronisation, but never both.


fix doc references to parameters and grammar inconsistencies

I noticed a few references in the docs to non-existent parameters. While I was at it, I found some inconsistencies, e.g. a few instances of non-imperative language in the short function summaries (inconsistent with the others in those modules).

Rebased, merged, and closed (GitHub's merge resolution interface wasn't resolving correctly). Thanks!


Make sure only accessible links are present on each page

We should make sure that any link displayed on a page should be accessible by the current user.
At the moment, for example, if an anonymous user visits a user's page, there will be links to edit the user, or edit email addresses, even though the anonymous user will not be able to go to these pages. We should remove these links depending on who the current user is.

Fixed in #655


Videos from Vine are not downloading.

Hello, the videos from vine.co are not working with your code but it downloads with youtube-dl in terminal. heres an example of the videos, i tried different ones already. https://vine.co/v/5zDDlpmiQJW.
Others websites work pretty good . I am using the youtube-dl api server from you, I created my own server on herokuapp.com

You have to upgrade youtube-dl (2016.07.02  or higher is needed).

I have updated youtube-dl to the last version on my server but i have no idea how to update youtube-dl on youtube-dl-api  app using heroku

$ heroku plugins:install https://github.com/heroku/heroku-repo.git
$ heroku repo:reset
$ git push heroku master

Im getting !   repo:resetis not a heroku command.  !    Perhaps you meantpg:reset.  !    See heroku help for a list of available commands.

Yes i did this is my output

Installing https://github.com/heroku/heroku-repo.git... User-MBP:youtube-dl-api-server User$

Whats the output of heroku plugins?

Thats all I gives me after I runheroku plugins:install https://github.com/heroku/heroku-repo.git
This:
Installing https://github.com/heroku/heroku-repo.git... User-MBP:youtube-dl-api-server User$

If i do heroku plugins
I get

You have no installed plugins.

If the plugin fails to install, I can't help you. I guess you should ask heroku.

Sorry, try with heroku plugins:install heroku-repo instead.

Yes I just did and it worked, but now when I run git push heroku master it gives me another error

fatal: Not a git repository (or any of the parent directories): .git

Sorry for bothering so much

fatal: Not a git repository (or any of the parent directories): .git

You have to run the command from the youtube-dl-api-server repo.

Done!  When I checked the

youtube-dl

version  inside the youtube-dl-api-server
it returns version 2016.06.25
But when I try to use the  pip install --user youtube-dl from the youtube-dl-api-server directory I receive

pip: command not found

I don't know your setup, but it's not really relevant because youtube-dl is automatically installed on heroku's servers.

I got it, i simply had to do youtube-dl -U to update youtube-dl to the latest version but, the Vine videos are still not working though

Check you have 2016.07.02 or higher.

I do
this is my code when i run this commanduser-MBP:youtube-dl-api-server User$ youtube-dl --version' I get  2<IP_ADDRESS>`

It works for me with the URL you posted, does it fail locally or on heroku?

user-MBP:youtube-dl-api-server User$ youtube-dl https://vine.co/v/5zDDlpmiQJW
[Vine] 5zDDlpmiQJW: Downloading webpage
[download] Destination: Spider-man VS Captain America freestyle rap battle. 🔥🔥😳😳-5zDDlpmiQJW.mp4
[download] 100% of 2.48MiB in 00:00

Im guessing is on heroku? because on my website it doesn't but when i download it through terminal it downloads

Run heroku repo:purge_cache before heroku repo:resetheroku repo:reset.

Thanks! You a master :) Finally I got it to work


Window no longer responding when trying to delete files

VSCode Version:
Version: 1.36.1 (user setup)
Commit: 2213894ea0415ee8c85c5eea0d0ff81ecc191529
Date: 2019-07-08T22:59:35.033Z
Electron: 4.2.5
Chrome: 69.0.3497.128
Node.js: 10.11.0
V8: 6.9.427.31-electron.0
OS: Windows_NT x64 6.1.7601
OS Version:
OS: Windows_NT x64 6.1.7601

Steps to Reproduce:
Don't know the steps. This is first time this issue happens. Can't reproduce it.
I just right clicked folder (with few subfolders and files) and delete, and it shows this after some time:

Only noticed thing is: when deleting one by one, the file that won't delete is one, and is TypeScript class file.

Does this issue occur when all extensions are disabled?: Yes

Probably not vscode issue, as it looks OS also can't delete the file:


[Bug report]: The balance of shimmer and assembly in firefly is no longer displayed

Platform
Desktop
Version
1.5.1
Description
The balance of shimmer and assembly in firefly is no longer displayed since the update to 1.5.1
Problem on ledger ans stronghold
Expected behaviour
The balance of shimmer and assembly in firefly is no longer displayed since the update to 1.5.1
Problem on ledger ans stronghold
Actual behaviour
The balance of shimmer and assembly in firefly is no longer displayed since the update to 1.5.1
Problem on ledger ans stronghold
Can the issue reliably be reproduced?
Yes
Steps to reproduce the issue
No response
Operating System
Linux
Diagnostics
No response
Developer Profile
No
Profile Type
Hardware (Ledger)
Staking
Actively staking
Crash Status
Non-responsive
Error Messages
No response
Error Logs
No response
Duplicate declaration

[X] I have searched the issues tracker this issue and there is none

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I dont use VPN. I do what I always do. With version 1.5.1, the shimmer and assembly balance is not displayed

I live in Reunion Island. The submarine cable seems to be damaged. Maybe the problem is with the cable.

Have you already tried to use the balance finder in the settings a few times in a row? I´m going to close this issue as it is a duplicate to https://github.com/iotaledger/firefly/issues/4016. Feel free to open the issue again if you have further problems.


Adipietro18 cherry pick

This is a cherry-pick of #21 to resolve the conflicts introduced by #22.
Checklist

[X] Used a personal fork of the feedstock to propose changes
[X] Bumped the build number (if the version is unchanged)
[X] Reset the build number to 0 (if the version changed)
[X] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[X] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

Whenever possible python packages should use pip. See https://conda-forge.org/docs/maintainer/adding_pkgs.html#use-pip

@conda-forge-admin, please rerender

Closes #24


Error when autocompleting appearance

I get the following error whenever the CSS language service attempts to autocomplete the appearance property.
Please let me know if there's any other information I can provide to help!
TypeError: Cannot read property 'indexOf' of undefined
    at /Applications/Visual Studio Code.app/Contents/Resources/app/extensions/css-language-features/server/dist/cssServerMain.js:1:578879
    at Array.filter (<anonymous>)
    at t.getTermProposals (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/css-language-features/server/dist/cssServerMain.js:1:578825)
    at t.e.getCompletionsForDeclarationValue (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/css-language-features/server/dist/cssServerMain.js:1:520870)
    at t.e.getCompletionsForExpression (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/css-language-features/server/dist/cssServerMain.js:1:530352)
    at t.e.doComplete (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/css-language-features/server/dist/cssServerMain.js:1:516088)
    at c.runSafe (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/css-language-features/server/dist/cssServerMain.js:1:71319)
    at Immediate.setImmediate [as _onImmediate] (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/css-language-features/server/dist/cssServerMain.js:1:100717)
    at runCallback (timers.js:696:18)
    at tryOnImmediate (timers.js:667:5)```

https://github.com/Microsoft/vscode/issues/71715
This fix is available VS Code insiders https://code.visualstudio.com/insiders/, or you can wait until our next release in ~2 weeks.

Thanks so much!


Create Basic Task Structure

Task

Create structure for task object with relevant fields (as discussed as a team)
Ensure tasks can be grouped by tags

Deliverables

Create the object that is usable throughout the front-end and ensure easy use for all tasks on the front end

Pushed to Sprint 4, delivery by 12/5
Utilize CRUD structure

Too similar to Issue #48 , closing due to redundancy


fix: update webchat via ui

:tada: This PR is included in version 0.320.1 :tada:
The release is available on:

npm package (@next dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Product page: tab "Data"

TODO
https://www.sketch.com/s/11375b6d-9c02-4920-846d-a2b1376600b9/a/aMqWEMA#Inspect

on it !

I did not find nutritionTags, where can I find it ?

Related fork : https://github.com/AdrienMDevMobile/smooth-app

Here is the first pull request : https://github.com/openfoodfacts/smooth-app/pull/5953


Add reason tag to kubernetes_state.job.failed

What does this PR do?

Add reason:backofflimitexceeded,deadlineexceeded to the kubernetes_state.job.failed.
Motivation

Currently, we cannot monitor why a job failed.
Additional Notes

Job termination and cleanup
BackoffLimitExceeded and DeadlineExceeded are GA. Other reasons are still beta or alpha.

https://github.com/kubernetes/kubernetes/blob/f6fde3ba7fb0755eb0d982b6ed2617258f1f4f8e/staging/src/k8s.io/api/batch/v1/types.go#L638-L642

kube_job_status_failed in KSM(code) is the only metric that contains reason in labels. kube_job_failed doesn't have reason label, see https://github.com/kubernetes/kube-state-metrics/issues/2382
kube_pod_container_status_terminated_reason in KSM (code) does not have reason:backofflimitexceeded,deadlineexceeded even it can have reason:ContainerStatusUnknown, etc.

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Collect KSM metrics from this job.
apiVersion: batch/v1
kind: Job
metadata:
  name: exit-fail
spec:
  backoffLimit: 2
  template:
    spec:
      containers:
      - name: busybox
        image: busybox
        command: ["exit", "1"]
      restartPolicy: OnFailure

/merge


A variety of performance and stability fixes

Removing the earlier PR. Replace with this one.
Have been able to run with 16 clients processing 5000 requests each. No network errors.

Two General comments on the storage service model:

Will 155 work, if the eservices are deployed in physically distinct machines? Say contract runs on eservice1, and eservice1 is configured to use blockstore of sservice2 attached to esservice2 running on different machine? How does the contract enclave access the lmdb unless the file is in the same machine?

the lmdb must be shared.

My guess is that this wont work, and I think in this case, we should not permit this configuration in the toml for now.

i think this is best covered by documentation.

It might be more meaningful to use the storage service  client as a separate object, instead of using it via the eservice client,  when we replicate the change set. I guess this can also be taken up when we implement the replication part.

you can instantiate a storage service separate from the eservice. however, it makes no sense to instantiate an eservice without a storage service. so i suggest leaving it as is.


EQMac2 requires a driver update!!!

I have tried multiple solutions and none have worked, I've allowed anything, I'm running macOS Catalina 10.15.2 (beta). Please help with this, I've tried multiple things, manual kext installation, brew cask install eqmac. I get a popup when I open EQMac2 saying it needs a driver update. I do as follows but have to quit it.

I'm also having the same issue, unfortunately.

Forget the v2 it has more issues than that, dev is working on v3.

Interesting, this must be an issue with installation. I have eqMac installed on Mohave, updated to Catalina, and it is still working mostly fine ( volume issues with bluetooth device, volume level jumping when output device changes, other known bugs etc.)

Hey everyone. Excuse me for the lack of updates. I have been really busy on the new version of eqMac. I am happy to announce an open beta release.
You can download it here: https://eqmac.app
There are a lot of known bugs that I could not fix just yet:

Driver not installing from the first try. Sometimes need to restart eqMac a couple times.
Complex Device switching logic is not perfect (for example when switching off the device that's currently playing, plugging in new device etc.)
UI takes long to load (a lot can be done to improve that)
UI dimensions are not synching properly
There's a noticeable audio delay in the playback, so not suitable for video yet (I know where the problem lies just haven't had the time to fix yet)
Sometimes the playback doesn't start, in that case please try to switch the output to the device you want and eqMac should try and build the audio pipeline again.

So as you can see there's still some work to be done but at least it's a completely modern tech stack that I am fully willing to work on: 100% Swift, User Space driver and a Web UI, instead of C++, Kernel Drivers. As I said, should work on Catalina and should not have any pesky ear drum blasting volume bugs (at least I didn't come across any).
I'm open to feedback and issue reports here: https://github.com/bitgapp/eqmac
That repo will become the main repo for eqMac and this repo will be archived eventually.
And yes it's called just eqMac, not eqMac3 :) And it will stay that way forever.

This little penguin is ready to dance thanks to the new beta version of EQMac, it works on Catalina now which is awesome. Now I can enjoy my bass tunes all day

OMG YESSSS! IVE BEEN LOOKIN FOR A EQ FOR MAC AND I JUST DOWNLOADED THE BETA ONE AND NOW MY AIRPODS HAVE SOO MUCH BASS

Hey everyone 👋
I will be open sourcing the new version of eqMac, therefore, I am closing this issue and repository as I will be transfering nodeful/eqMac2 repo to bitgapp/eqMac
Please bear with me for the next couple of days as I setup that new repo. The eqMac2 code will be living in the old-eqMac2 branch if anyone is interested in referencing it for whatever reason.
Please try out the new version and report bugs to the new repo.
Thanks for using eqMac.

I'm SO glad you got a new version released. It's so much better than no EQ. But, the old version used to produce the most amazing bass in my headphones and no longer does. The previous version had way more frequencies to adjust. Is this coming in the next release (please!)?


update jm compensation figures

Deploy preview looks good, but getting warning that "This branch is out-of-date with the base branch" (presumably to do with https://github.com/oaworks/oa-works/commit/f21e3a00d0326844c45fe1428da167de60db3723?) so not going to merge for now.

Merged on the command line — develop and main are now precisely the same.

Thanks!
Joe McArthur
Director / Co-Founder: OA.Works https://oa.works
The Open Access Button is now OA.Works! Find out more about our rebrand
https://blog.oa.works/open-access-button-is-now-oa-works/.
On Tue, Sept 19 2023 at 8:52 am, Sophy @.***> wrote:

Merged on the command line — develop and main are now precisely the same.
—
Reply to this email directly, view it on GitHub
https://github.com/oaworks/oa-works/pull/186#issuecomment-1725455655,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABFZMZYHWZXQ66U2KSWADWLX3GIQTANCNFSM6AAAAAA4S32Y44
.
You are receiving this because you authored the thread.Message ID:
@.***>


LanguageDetector.js Fix detections

Manage detections if it's necessary

merged and published in v0.1.2


oidc-login documentation looks very outdated

What would you like to be added:
The Authentication documentation is very outdated.
https://gardener.cloud/documentation/guides/client_tools/oidc-login/
Why is this needed:
It references auth0.com
but the information needed to configure this is insufficient in many places and needs to be reviewed from someone experienced.
Trying to match auth0 page configuration with the documentation is frustrating because too much is missing or outdated.
That' s the thing with examples. they should work.

Every identity provider is different in technology, somewhat in terminology and Ux.
The relevant sections in the Gardener documentation is not about configuring the IdP (where auth0 was only used as a generic example). The relevant information is about a) generally needing to configure a kubectl oidc plugin and the attributes of the kubeconfig, and b) the options in the Gardener shoot specification.
Dependig on the used identity provider, you might even get a specialized oidc plugin where configuration of the kubeconfig is different. Also, depending on the provider setup, the configuration of scopes and usernameClaim etc. can be very different. The information needed to configure all possible providers and setups is surely out of scope of the Gardener documentation.
The auth0 documentation works and should be used, lastly, only as a guiding principle.


Fixed MeshInstance2d copying Line2d's default color

MeshInstance2d now has a default color. MeshInstance2d was inheriting the color of the Line2d that was rendered before the MeshInstance2d.
Fixes: #25760

Please ignore commits f34e1db and a8207a2. Still learning the interaction between Git and Github. I believe commit 0331bf2 fixes the indentation issue.

You'll need to rebase to remove the first commits and keep only the last one, see PR workflow.

When I try and run the command git rebase -i HEAD~2 to change commits f34e1db and a8207a2 to fixup status,  I can't seem to access either of them. I am not quite sure what command to run to fix those commits?

That's normal, git rebase will automatically remove those merge commits, which are no longer needed once rebased.

I still do not think I did the rebasing right. My apologies for this. Should I have applied fixup status to EVERY commit before the one I wanted to force push? So that commits 2fd3a19 and d1402fe were not part of the forced push?

You included too many commits in your rebase yes, and modified some made by other authors. You can do this to fix it:
git remote add upstream https://github.com/godotengine/godot
git fetch upstream
git rebase upstream/master

So would I then force push the branch again after running those 3 previous commands?

Also, I noticed the static check in the Travis CI build continues to fail. I believe I fixed the earlier indentation issue to be in-line with the clang format rules, but it still seems to be throwing an error. Any suggestions there?

So would I then force push the branch again after running those 3 previous commands?

If git log looks correct, yes.
Not sure why Travis still fails, the final diff doesn't seem to include this trailing whitespace. I guess we'll see what happens once it's fully rebased.

Before I force push incorrect commits again, I want to make sure I am only pushing the commit that adds the default color to the mesh render (commit ba56b39). My issue with attempting to rebase is that I can't find commits 2fd3a19 and d1402fe (from the commit_issue_screenshot) in either the git log or in the text editor when I am attempting to change them to fixup status. I have attached 3 screenshots of my issues to try and make it clearer. I'm sure this is just me running incorrect commands or not understand the git rebase as I should. Thank you for the help.

What command did you run? git rebase upstream/master should automatically get rid of those two duplicate commits (as they already pre-exist in upstream/master) and apply your own commit on top of the current upstream master branch.

I ran git rebase upstream/master and received the message "Current branch MeshColor is up to date."
This did not seem to affect anything with the commit_issue_screenshot I attached in the previous message.

The problem is that you included your own changes in https://github.com/godotengine/godot/pull/26738/commits/2fd3a19ff2b893adb0e8a9dc6365b1097417149e
So you need to do git rebase -i upstream/master, drop the line for commit 2fd3a19, and then solve the conflict that will arise. Then git rebase --continue and it should be fine.
I've done it myself to confirm, so I'll just force push the fixed commit, you've been wrestling enough with Git for today ;)

Ok thank you for all your help and explanations! I think I understand those Git commands better for the future.
Also, it looks like the fully rebased commit you force pushed passed the Travis static test it was originally failing :) Thanks again for all your help!

Thanks!


Helm sometimes renders unnecessary additional yaml document containing only a comment when templating

Output of helm version: version.BuildInfo{Version:"v3.14.3", GitCommit:"v3.14.3", GitTreeState:"", GoVersion:"go1.21.7"}
Output of kubectl version: N/A
Cloud Provider/Platform (AKS, GKE, Minikube etc.): N/A
Context
Recently when trying to install Envoy Gateway via Jsonnet I hit an obscure problem using std.prune.  The reason I had been trying to use std.prune was to remove empty objects from the Envoy Gateway install.yaml file.  This file is created via helm template here.
When the install.yaml file is rendered, it ends up with yaml documents that contain only the "Source: " comment before being followed by another document containing the actual manifest.  Example:
Actual output:
---
# Source: gateway-helm/crds/generated/gateway.envoyproxy.io_securitypolicies.yaml
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.14.0
  name: securitypolicies.gateway.envoyproxy.io

Expected output:
---
# Source: gateway-helm/crds/generated/gateway.envoyproxy.io_securitypolicies.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.14.0
  name: securitypolicies.gateway.envoyproxy.io

The original issue I raised with Envoy Gateway is here.
This happens because helm template doesn't take in to account how a manifest begins.  If it doesn't start with the --- then it adds it in along with the comment and all is well.  However, if it does start with --- then helm template will insert a document that has a comment only and then the manifest will be added after that and because the manifest starts with ---, it is added as a separate document.
The offending line appears to be https://github.com/helm/helm/blob/main/cmd/helm/template.go#L116
My suggestion is that helm template should check for the yaml start document line ---\n prior to the comment addition and remove it if it exists so that the comment will always appear in the same document as the manifest it is rendering.
I'm happy to work on the issue but just want some feedback on my proposed solution.

@wimnat looking at https://github.com/envoyproxy/gateway/releases/download/v1.0.1/install.yaml, it seems to contain multiple documents inside one YAML file i.e parts split with ---\n. Some of the documents contain comments starting with # Source: ...
Here is an output after running helm template
#
# Gateway API Experimental channel install
#
---
# Source: foo/templates/install.yaml
# Source: gateway-helm/crds/generated/gateway.envoyproxy.io_backendtrafficpolicies.yaml
---
# Source: foo/templates/install.yaml
# Source: gateway-helm/crds/generated/gateway.envoyproxy.io_clienttrafficpolicies.yaml
---
# Source: foo/templates/install.yaml
# Source: gateway-helm/crds/generated/gateway.envoyproxy.io_envoypatchpolicies.yaml
---
# Source: foo/templates/install.yaml
# Source: gateway-helm/crds/generated/gateway.envoyproxy.io_envoyproxies.yaml
---
# Source: foo/templates/install.yaml
# Source: gateway-helm/crds/generated/gateway.envoyproxy.io_securitypolicies.yaml
---
# Source: foo/templates/install.yaml
# Source: gateway-helm/templates/certgen-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eg-gateway-helm-certgen
  namespace: 'envoy-gateway-system'
  labels:
    helm.sh/chart: gateway-helm-v0.0.0-latest

My suggestion is that helm template should check for the yaml start document line ---\n prior to the comment addition and remove it if it exists so that the comment will always appear in the same document as the manifest it is rendering.

I'm not sure I follow your suggestion here. Specifically, what do you mean by will always appear in the same document as the manifest? The manifest in question contains multiple documents which helm renders as expected.

closing as dup: https://github.com/helm/helm/issues/12953
(please comment if I'm mistaken)


Create fmf for each tcms case when converting.

Using tmt test convert will now create fmf file for each test case found
in tcms apart from main.fmf, which contains common attributes found in
all of the individual test cases. New files are named according to their
tcms id and contain only attributes with values that are unique to it.

I wasn't sure where to document the changed behaviour so I would appreciate an advice.

Tests are failing:
<Result UnboundLocalError("local variable 'testcase_data' referenced before assignment")>.exit_code

I wasn't sure where to document the changed behaviour so I would appreciate an advice.

Documentation updates should go here:

https://tmt.readthedocs.io/en/latest/examples.html#convert-tests

You can find the source code under docs/examples.rst. To review updated docs locally:
pip install -e .[docs]
make docs
firefox docs/_build/html/index.html

Thanks.

Looks good now. Thanks. I see a couple of minor issues. We can improve them later.


PickRandomMethod returns exception

Describe the bug
When running a transformer on a dataset with the perClass transformation scope the program tries to apply the transformers to every class. This includes enums. However, when running the pickRandomMethod function the list of all methods is empty. This means that the bound for picking a random method is 0 which is not allowed.
Expected behavior
I expect this to be handled by either logging what is wrong or just ignoring the enums for this scope.

Closed with #120


Duplicate music title play only one path

when you play song that have the same title it play song only one path.
example:
tracks title : A, path: 1.mp3
tracks title : A, path: 2.mp3
tracks title : A, path: 3.mp3
it alway play path: 1.mp3

Ok, let me check it out. I recently added a music cache, which may be caused by this issue.

Can you provide more information?

I think that it issues from music cache .


TODO

TODO

[x] add tcp rpc
[x] check the conf change v2
[ ] Repair wall
[ ] add more logs.

both TCP and conf change 2 are still, under consideration


Use drone for Docker image build/deploy

Fixes #16
Build and deploy quay.io/kontena/pharos-host-upgrades:edge images from the git master branch.
TODO for tagged release builds + arm64 builds... I'd like to see the amd64 :edge build working first, though.

@nevalla PT(Another)L... I added a go test stage which is working, and also a second docker stage for tagged release builds.


git clone HTTPS URL

Changed the git clone URL into an URL that is available to all people that have Internet access.

Changed the git clone URL into an URL that is available to all people that have Internet access.

Could be read as

Changed the git clone URL into an URL that is also available to git clients that don't have an SSH-key at github.com.  Both human and CI-bots.

robot@ci-cd-host:$ git clone git@github.com:sunface/rust-by-practice.git
Cloning into 'rust-by-practice'...
The authenticity of host 'github.com (<IP_ADDRESS>)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
robot@ci-cd-host:$ git clone https://github.com/sunface/rust-by-practice
Cloning into 'rust-by-practice'...
remote: Enumerating objects: 4272, done.
remote: Counting objects: 100% (874/874), done.
remote: Compressing objects: 100% (436/436), done.
remote: Total 4272 (delta 581), reused 662 (delta 436), pack-reused 3398
Receiving objects: 100% (4272/4272), 2.79 MiB | 18.10 MiB/s, done.
Resolving deltas: 100% (2747/2747), done.
robot@ci-cd-host:$


Issue with ResourceAuthorizationContext context Principal

Hi,
If I load more than 45 claims then ResourceAuthorizationContext claim context failed to load all the claims in web api resulting in authorized access but I am able to load all the claims into claims context in MVC.

See if on your controller the .User property has all the claims. We are not really "loading" them - we just take what has been populated by Web API.

Since we are loading claims more than 45 records, in web api under ResourceAuthorizationContext we seen that automatically changing of ClaimsPrincipal to WindowsPrincipal and claims are 0.

Hi,
When we are adding only 32 rows of privileges in the AppPriv table then, ResourceAuthorizationContext context Principal is showing below.

Showing Claims list and able to access protected resources (Web API methods).
If we are adding more than 32 rows of privileges in the AppPriv table then, ResourceAuthorizationContext context Principal is showing below.

Showing unauthorized error, Claims list is empty and not able to access protected resources (Web API methods).

Try it without the resource authZ MW and inspect the .User property on your controller.


[w4.4b][W13-A1]Dominic Kenn Lim

Encapsulate CommandResult class members.
Split Address to more classes.
Edit Test Cases.

@dominickenn
Some comments added. Please ack & close the PR after reading comments.
This PR seems to have unrelated changes.
Avoid accumulating unrelated changes into a single PR.
Tip: If you are currently on branch1 and want to create a new branch2, you need to switch back to master branch before creating branch2. If not, changes in branch1 will appear in the PR created from branch2 later.

Acknowledged comments.


root_scalar fails when passed a function wrapped with functools.lru_cache

When you provide scipy.optimize.root_scalar with a function wrapped with functools.lru_cache it raises a SystemError. This bug is also present in previous versions of scipy (I tested back to 0.19) when using the more specific root-finding methods.
Reproducing code example:
import scipy.optimize
import functools

@functools.lru_cache()
def f(x):
    return x

scipy.optimize.root_scalar(f, bracket=(-1, 1))

Error message:
SystemError: /home/conda/feedstock_root/build_artifacts/python_1562015400360/work/Objects/tupleobject.c:168: bad argument to internal function

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 8, in <module>
    scipy.optimize.root_scalar(f, bracket=(-1, 1))
  File "/home/jbw/.local/miniconda/envs/scipy-test/lib/python3.7/site-packages/scipy-1.4.0.dev0+746aaef-py3.7-linux-x86_64.egg/scipy/optimize/_root_scalar.py", line 249, in root_scalar
    r, sol = methodc(f, a, b, args=args, **kwargs)
  File "/home/jbw/.local/miniconda/envs/scipy-test/lib/python3.7/site-packages/scipy-1.4.0.dev0+746aaef-py3.7-linux-x86_64.egg/scipy/optimize/zeros.py", line 775, in brentq
    r = _zeros._brentq(f, a, b, xtol, rtol, maxiter, args, full_output, disp)
SystemError: <functools._lru_cache_wrapper object at 0x7f48bd1ba930> returned a result with an error set

Scipy/Numpy/Python version information:
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)

1.4.0.dev0+746aaef 1.17.2 sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)

but I also observed this problem in scipy 0.19 when using any of the root-finding functions

Repeated calls with the same arguments to a function should not happen often in the root solving algorithms, so wrapping in a functools.lru_cache is unlikely to produce result much sooner for a single run.  But the code should still work.
I looked into it and have a few observations:

The problem affects scipy.optimize.{bisect,brentq,ridder} and usually root_scalar as its default method is brentq.
The example f(x)=x above (and also f(x)=x**3) didn't trigger the issue for scipy.optimize.{newton,toms748}
The Exceptions disappeared if the call to scipy.optimize.METHOD() was repeated once or twice more.
Changing the endpoints caused the Exceptions to reappear.
scipy.optimize.{bisect,brentq,ridder} are all written in C.  scipy.optimize.{newton,toms748} are written in Python.

I do not know what exactly is the cause, but it does seem that it is a property of the C functions only, not the Python functions.

Ok I've managed to narrow down the problem to this line:
PyArgs(SetItem)(args, 0, Py_BuildValue("d",x));

which expands to
PyTuple_SetItem(args, 0, , Py_BuildValue("d",x));

This returns an error code that we do not catch, which gives rise to the bad argument to internal function.
The reason that it returns an error code is because you cannot set an item in a tuple if that tuple is used elsewhere (i.e. it has a refcount > 1).  This is because tuples are meant to be immutable.
lru-cached functions use the args tuple as a key in their internal dictionary, which increases the refcount on the args tuple.
TL;DR: we should be creating a new args tuple on every call instead of modifying its elements.


usage of vllm for extracting embeddings

Following is a little piece of code to extract embeddings from a certain layer of LLM:
def process_row(prompt: str, model, tokenizer, layers_to_use: list, remove_period: bool):
    """
    Processes a row of data and returns the embeddings.
    """
    if remove_period:
        prompt = prompt.rstrip(". ")
    inputs = tokenizer(prompt, return_tensors="pt")
    with torch.no_grad():
        outputs = model.generate(inputs.input_ids, output_hidden_states=True, return_dict_in_generate=True, max_new_tokens=1, min_new_tokens=1)
    embeddings = {}
    for layer in layers_to_use:
        last_hidden_state = outputs.hidden_states[0][layer][0][-1]
        embeddings[layer] = [last_hidden_state.numpy().tolist()]
    return embeddings

It's pretty standard way, but it's pretty slow. Is there any way to use vllm to make it faster without needing to call generate function everytime? I've tried batching, but it's slow too. Any help is appreciated!

Similarly, for me, I need to extract the last hidden state of the model. But it seems difficult to do, especially i need to modify the worker.py to get better performance. (KVcache ,etc)
class OPTModel(nn.Module):

    def __init__(self, config: OPTConfig):
        super().__init__()
        self.decoder = OPTDecoder(config)

    def forward(
        self,
        input_ids: torch.Tensor,
        positions: torch.Tensor,
        kv_caches: List[KVCache],
        input_metadata: InputMetadata,
        cache_events: Optional[List[torch.cuda.Event]],
    ) -> torch.Tensor:
        return self.decoder(input_ids, positions, kv_caches, input_metadata,
                            cache_events) # i need this

Similarly, for me, I need to extract the last hidden state of the model. But it seems difficult to do, especially i need to modify the worker.py to get better performance. (KVcache ,etc)
class OPTModel(nn.Module):

    def __init__(self, config: OPTConfig):
        super().__init__()
        self.decoder = OPTDecoder(config)

    def forward(
        self,
        input_ids: torch.Tensor,
        positions: torch.Tensor,
        kv_caches: List[KVCache],
        input_metadata: InputMetadata,
        cache_events: Optional[List[torch.cuda.Event]],
    ) -> torch.Tensor:
        return self.decoder(input_ids, positions, kv_caches, input_metadata,
                            cache_events) # i need this

So, does that work? Are you able to make it faster?

Similarly, for me, I need to extract the last hidden state of the model. But it seems difficult to do, especially i need to modify the worker.py to get better performance. (KVcache ,etc)
class OPTModel(nn.Module):

    def __init__(self, config: OPTConfig):
        super().__init__()
        self.decoder = OPTDecoder(config)

    def forward(
        self,
        input_ids: torch.Tensor,
        positions: torch.Tensor,
        kv_caches: List[KVCache],
        input_metadata: InputMetadata,
        cache_events: Optional[List[torch.cuda.Event]],
    ) -> torch.Tensor:
        return self.decoder(input_ids, positions, kv_caches, input_metadata,
                            cache_events) # i need this

So, does that work? Are you able to make it faster?

No! It has error or not init worker! the worker is too complex

Similarly, for me, I need to extract the last hidden state of the model. But it seems difficult to do, especially i need to modify the worker.py to get better performance. (KVcache ,etc)
class OPTModel(nn.Module):

    def __init__(self, config: OPTConfig):
        super().__init__()
        self.decoder = OPTDecoder(config)

    def forward(
        self,
        input_ids: torch.Tensor,
        positions: torch.Tensor,
        kv_caches: List[KVCache],
        input_metadata: InputMetadata,
        cache_events: Optional[List[torch.cuda.Event]],
    ) -> torch.Tensor:
        return self.decoder(input_ids, positions, kv_caches, input_metadata,
                            cache_events) # i need this

So, does that work? Are you able to make it faster?

No! It has error of not init worker! the worker is too complex

well, ok! let's see if someone has solved the problem

I made a solution, without changing the library, i can run. But i cannot certain it can accerate.
from vllm import LLM, SamplingParams
llm = LLM(model="/root/hdd/llm/opt-1.3b")
from vllm.sequence import (SamplerOutput, Sequence, SequenceGroup,
                           SequenceGroupMetadata, SequenceGroupOutputs,
                           SequenceOutputs, SequenceStatus)

prompt = 'I am a'

prompt_token_ids = llm.llm_engine.tokenizer.encode(prompt)
block_size = llm.llm_engine.cache_config.block_size
seq_id = next(llm.llm_engine.seq_counter)
seq = Sequence(seq_id, prompt, prompt_token_ids, block_size)
request_id = str(next(llm.request_counter))
# Create the sequence group.
seq_group = SequenceGroup(request_id, [seq], sampling_params,
                            None)
from vllm.sequence import SamplerOutput, SequenceData, SequenceGroupMetadata
import torch
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()

# Profile memory usage with max_num_sequences sequences and the total
# number of tokens equal to max_num_batched_tokens.

# Enable top-k sampling to reflect the accurate memory usage.
vocab_size = llm.llm_engine.workers[0].model.config.vocab_size
sampling_params = SamplingParams(top_p=0.99, top_k=vocab_size - 1)
max_num_batched_tokens = llm.llm_engine.workers[0].scheduler_config.max_num_batched_tokens
max_num_seqs = llm.llm_engine.workers[0].scheduler_config.max_num_seqs

prompt = 'I am a'

prompt_token_ids = llm.llm_engine.tokenizer.encode(prompt) #[2, 100, 524, 10]

seqs = []
# for group_id in range(max_num_seqs):
#     seq_len = (max_num_batched_tokens // max_num_seqs +
#                 (group_id < max_num_batched_tokens % max_num_seqs))
#     print(seq_len)
group_id = 1
seq_data = SequenceData(prompt_token_ids)
seq = SequenceGroupMetadata(
    request_id=str(group_id),
    is_prompt=True,
    seq_data={group_id: seq_data},
    sampling_params=sampling_params,
    block_tables=None,
)
seqs.append(seq)

input_tokens, input_positions, input_metadata = llm.llm_engine.workers[0]._prepare_inputs(
    seqs)
# Execute the model.
num_layers = llm.llm_engine.workers[0].model_config.get_num_layers(llm.llm_engine.workers[0].parallel_config)
tempOut = llm.llm_engine.workers[0].model.model(
    input_ids=input_tokens,
    positions=input_positions,
    kv_caches=[(None, None)] * num_layers,
    input_metadata=input_metadata,
    cache_events=None,
)
tempOut.size() #torch.Size([1, 4, 2048])

If you improve this code, share with me !

@WuNein using your code, is there any way I can also extract the output tokens along with the embeddings?  With vllm, I'm running model.generate(...) and getting different number of output tokens. For an example prompt, I'm getting 125 output tokens, with vllm's model.generate(...). However, with your approach, the dimensions of my output embedding are 108, 5120, which does not make sense

@WuNein using your code, is there any way I can also extract the output tokens along with the embeddings? With vllm, I'm running model.generate(...) and getting different number of output tokens. For an example prompt, I'm getting 125 output tokens, with vllm's model.generate(...). However, with your approach, the dimensions of my output embedding are 108, 5120, which does not make sense

@WuNein using your code, is there any way I can also extract the output tokens along with the embeddings? With vllm, I'm running model.generate(...) and getting different number of output tokens. For an example prompt, I'm getting 125 output tokens, with vllm's model.generate(...). However, with your approach, the dimensions of my output embedding are 108, 5120, which does not make sense

I made a demo, check it out.
https://github.com/WuNein/vllm4mteb

I made a solution, without changing the library, i can run. But i cannot certain it can accerate.
from vllm import LLM, SamplingParams
llm = LLM(model="/root/hdd/llm/opt-1.3b")
from vllm.sequence import (SamplerOutput, Sequence, SequenceGroup,
                           SequenceGroupMetadata, SequenceGroupOutputs,
                           SequenceOutputs, SequenceStatus)

prompt = 'I am a'

prompt_token_ids = llm.llm_engine.tokenizer.encode(prompt)
block_size = llm.llm_engine.cache_config.block_size
seq_id = next(llm.llm_engine.seq_counter)
seq = Sequence(seq_id, prompt, prompt_token_ids, block_size)
request_id = str(next(llm.request_counter))
# Create the sequence group.
seq_group = SequenceGroup(request_id, [seq], sampling_params,
                            None)
from vllm.sequence import SamplerOutput, SequenceData, SequenceGroupMetadata
import torch
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()

# Profile memory usage with max_num_sequences sequences and the total
# number of tokens equal to max_num_batched_tokens.

# Enable top-k sampling to reflect the accurate memory usage.
vocab_size = llm.llm_engine.workers[0].model.config.vocab_size
sampling_params = SamplingParams(top_p=0.99, top_k=vocab_size - 1)
max_num_batched_tokens = llm.llm_engine.workers[0].scheduler_config.max_num_batched_tokens
max_num_seqs = llm.llm_engine.workers[0].scheduler_config.max_num_seqs

prompt = 'I am a'

prompt_token_ids = llm.llm_engine.tokenizer.encode(prompt) #[2, 100, 524, 10]

seqs = []
# for group_id in range(max_num_seqs):
#     seq_len = (max_num_batched_tokens // max_num_seqs +
#                 (group_id < max_num_batched_tokens % max_num_seqs))
#     print(seq_len)
group_id = 1
seq_data = SequenceData(prompt_token_ids)
seq = SequenceGroupMetadata(
    request_id=str(group_id),
    is_prompt=True,
    seq_data={group_id: seq_data},
    sampling_params=sampling_params,
    block_tables=None,
)
seqs.append(seq)

input_tokens, input_positions, input_metadata = llm.llm_engine.workers[0]._prepare_inputs(
    seqs)
# Execute the model.
num_layers = llm.llm_engine.workers[0].model_config.get_num_layers(llm.llm_engine.workers[0].parallel_config)
tempOut = llm.llm_engine.workers[0].model.model(
    input_ids=input_tokens,
    positions=input_positions,
    kv_caches=[(None, None)] * num_layers,
    input_metadata=input_metadata,
    cache_events=None,
)
tempOut.size() #torch.Size([1, 4, 2048])

I check the answer  the cos sim showed, they are the same.
If you improve this code, share with me !

Hi @WuNein, this code will probably provide the last hidden state, is there any way to get values of intermediate hidden states?

@ra-MANUJ-an  https://github.com/vllm-project/vllm/blob/abfc4f3387c436d46d6701e9ba916de8f9ed9329/vllm/model_executor/models/llama.py#L274
May have, you should try, https://github.com/WuNein/vllm4mteb/blob/main/vllm-new.py
My code for newer version is here.
Maybe star for me.

@WuNein Thanks for taking time to write the code properly, but how are you implementing hidden_states with your following code:
outputs = engine.driver_worker.model_runner.model(input_ids=input_tokens,
positions=input_positions,
kv_caches=[(None, None)] * num_layers,
input_metadata=input_metadata)

@WuNein Hi, if you have time, could you please answer the above question?

@WuNein Hi, if you have time, could you please answer the above question?

There may not any solution if the hidden state do not have intermediate states.
I don't have the requirement to get the intermediate states.


Resource type is enum

These two RPs use an enum to abstract a group of similar resources. I wonder what is the best way to model them in TSP.
example

This could only be represented by low-level APIs.  This will not be supported by TypeScript templates because it is an anti-pattern.


Within schema, fragments annotated with @schema/type any=True are erroneously validated as schema (rather than as a default value)

(Inspired by #515)
What steps did you take:
#@data/values-schema
---
spec:
  #@schema/type any=True
  additionalConfig:
    args:
    - "cmd"
    - "arg1"

What happened:
Got this error:
ytt: Error:
Invalid schema
==============

Invalid schema - wrong number of items in array definition
==========================================================

schema.yml:
    |
  6 |     args:
    |

    = found: 2 array items
    = expected: exactly 1 array item, of the desired type
    = hint: in schema, the one item of the array implies the type of its elements.
    = hint: in schema, the default value for an array is always an empty list.
    = hint: default values can be overridden via a data values overlay.

What did you expect:
That anything within a YAML fragment that's annotated with @schema/type any=True:

can contain any valid YAML
is the default value for the annotated node.

Anything else you would like to add:
This expectation was implied, but made explicit with #516.
From an implementation perspective, this might well be "simply" to:

stop processing this subtree of the schema AST for types ;
consider this subtree as the default for the annotated node.

Environment:

ytt version (use ytt --version): v0.37.0

Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.

Available in v0.38.0


MB-16118: Remove build/deployment of webhook client in dp3 environments

Jira ticket
Summary
This change means we no longer build or deploy the webhook client. Another PR to the infrastructure will disable the service once this PR is deployed. We need to do this step first so that nothing is referencing the service.
I've done this in loadtest with no problems
Verification Steps for the Author
These are to be checked by the author.

[ ] Tested in the Loadtest environment (for changes to containers, app startup, or connection to data stores)
[ ] Have the Jira acceptance criteria been met for this change?

Verification Steps for Reviewers
This change is entirely in CircleCI, so there's nothing to test locally

Messages

:book:
:link: MB-16118

  Generated by :no_entry_sign: dangerJS against b0e8e1ef16ded642c72ebb73a0d9e1908215103e


Data Page Additional Routing Fixes

It seems there are a few more places where the routing needs to be fixed.
For example, if I'm on the data page at https://hivelab.tst.biochemistry.gwu.edu/biomarker-partnership/data:

And I click on "... view details" for the first row (BCO ID of ARGOS_000009), it routes to https://hivelab.tst.biochemistry.gwu.edu/ARGOS_000009, instead of the correct page at https://hivelab.tst.biochemistry.gwu.edu/biomarker-partnership/data/ARGOS_000009:

If I manually visit the correct url at https://hivelab.tst.biochemistry.gwu.edu/biomarker-partnership/data/ARGOS_000009 I get the correct page:

This is similar to the App.js routing that was initially fixed. I'll look for where the additional routing prefix of "/biomarker-partnership/data" needs to get placed.

Edit src/components/columns.jsx  and try to modify the value shown below:

This has been fixed.


Redirect to RP after logout

I have a question about performing a logout from the app and ADFS and returning the user back to the application (RP).
Description
I've setup Windows Server 2016 with ADFS 4.0 and enabled OpenID Connect authentication for users in the local AD.
The authentication (login) using OpenId Connect works (the authorization code flow as well as implicit flow) works. Sign-out (logout) works as well. The problem is that after the signout, the user is left on the ADFS signout page and not redirected back to the RP, even though the RP provides the URL in the post_logout_redirect_uri variable. An example a URL is this:
https://win-910etfa6g5q.mydomain.local/adfs/oauth2/logout?post_logout_redirect_uri=http%3A%2F%2Flocalhost%3A58196%2Fsignout-callback-oidc&state=state-here&x-client-SKU=ID_NET&x-client-ver=<IP_ADDRESS>
I am referring to the documentation in this repo: https://github.com/MicrosoftDocs/windowsserverdocs/blob/master/WindowsServerDocs/identity/ad-fs/development/ad-fs-logout-openid-connect.md
Expected outcome
The browser should be redirected to the post_logout_redirect_uri after logout on the ADFS
Actual outcome
The browser stays on the signout page on the ADFS (https://win-910etfa6g5q.mydomain.local/adfs/oauth2/logout?etc...)
Integration details
I'm using ASP.NET Core MVC application with Microsoft.AspNetCore.Authentication.OpenIdConnect package to integrate with it. I tried both the Implicit flow and Authorization code flow.
I figured that the logout url should be set for this application and couldn't find it in the UI, so I've set it using PowerShell: Get-AdfsServerApplication | Set-AdfsServerApplication -LogoutUri http://localhost:58196/signout-callback-oidc and verified with Get-AdfsServerApplication:
ADUserPrincipalName                  :
ClientSecret                         : ********
JWTSigningCertificateRevocationCheck : None
JWTSigningKeys                       : {}
JWKSUri                              :
Name                                 : TestAppGroup - Server application
Identifier                           : 1e0ae584-52d9-45e3-a3d9-b3216a0a26bc
ApplicationGroupIdentifier           : TestAppGroup
Description                          :
Enabled                              : True
RedirectUri                          : {http://localhost:58196/signin-oidc}
LogoutUri                            : http://localhost:58196/signout-callback-oidc

The ADFS is still not redirecting to the specified logout URI and I'm wondering if I understood the documentation incorrectly. Perhaps the documentation is referring to the fact that ADFS is supposed to "call" all applications that the user signed into on the following URLs? But that is not happening either.
I am performing tests from a dev machine (localhost) and connecting to ADFS win-910etfa6g5q.mydomain.local (different machine). I am using self-signed certificates, FWIW.

Closing due to inactivity.  Please reopen to pursue further.  Thanks!


Confused for the "Seeds" part

https://btcguide.github.io/backup-wallet/
"These are your 24 word BIP39 phrases. You only need to have m (of your total n) seeds to recover your funds."
Here you want to say - you need m of n seed to sign a transaction, right?
"recover your funds" feels confusing to me, it could refer to 1) sign a transaction or 2) recover your wallet.

Thanks. Look good now?


Blazor - Javascript interop in thread from Invoke() breaks.

Hi,
If you try to change the pages html from any function/thread starting from Invoke() it breaks the page with no errors.
e.g.

public void ButtonClicked()
{
    // do stuff...

    Invoke(() => {
        this.StateHasChanged();
        UpdateHTMLDOM();  // <---------- this breaks the entire page.
    });
}

private async void UpdateHTMLDOM()
{
    await JSRuntime.invokemethod<>("CreateImage", "myImageString.png");
}

Can you provide a minimal example that reproduces the problem? Given the default project template, what is the minimal set of steps needed to update it to show the problem you're describing?

Using the default project template "Server-side razor" given in the visual studio 2019 preview version. Using netcore sdk version "3.0.0-preview6-27804-01".
Replace the counter.razor code with the following:
@page "/counter"
@inject IJSRuntime JSRuntime

Counter

Current count: @currentCount

<button @onclick="@ButtonClicked">Increment Counter.</button>

@code {
    private int currentCount = 0;
    private string Title = "";

    bool PreventRenderLoop = false;
    protected override void OnAfterRender()
    {
        if (PreventRenderLoop) return; // stop the page from entering a render loop.

        Task getDataTask = Task.Run(GetData);
        getDataTask.Wait();
    }

    public async void GetData()
    {
        // demo just load some variables.
        Title = "Hello World";
        // e.g. data = MyService.LoadData();

        PreventRenderLoop = true;
        // refresh the page after loading new values
        Invoke(() =>
        {
            this.StateHasChanged();

            // now the page has the new values loaded from the service
            // we want to display the DOM which is relavent
            Task.Run(DisplayDOM).Wait();
        });
    }

    public async Task DisplayDOM()
    {
        currentCount = -1;

        await JSRuntime.InvokeAsync<string>("Notify", "Successfully displayed dom!", "success");
    }

    public void ButtonClicked()
    {
        currentCount++;
    }
}

And the code for the javascript notification (which goes in wwwroot/js and you reference in _Host.cshtml) is:

function Notify(pStrMessage, pStrMode)
{
    alert(pStrMessage + " _ " + pStrMode);
}

If you run the above code, when you click the button to increment the counter, nothing happens. I resolved this by changing it to this:
@page "/counter"
@inject IJSRuntime JSRuntime

Counter

Current count: @currentCount

<button @onclick="@ButtonClicked">Increment Counter.</button>

@code {
    private int currentCount = 0;
    private string Title = "";

    bool PreventRenderLoop = false;
    protected override void OnAfterRender()
    {
        if (PreventRenderLoop) return; // stop the page from entering a render loop.

        Task getDataTask = Task.Run(GetData);
        getDataTask.Wait();
    }

    public async void GetData()
    {
        // demo just load some variables.
        Title = "Hello World";
        // e.g. data = MyService.LoadData();

        PreventRenderLoop = true;
        // refresh the page after loading new values
        await Invoke(() =>
        {
            this.StateHasChanged();
        });

        // now the page has the new values loaded from the service
        // we want to display the DOM which is relavent
        Task domTask = Task.Run(DisplayDOM);
        await domTask;
    }

    public async Task DisplayDOM()
    {
        currentCount = -1;

        await JSRuntime.InvokeAsync<string>("Notify", "Successfully displayed dom!", "success");
    }

    public void ButtonClicked()
    {
        currentCount++;
    }
}

I notice now that i set the PreventRender boolean and never unset it, therefore the OnAfterRender() function is constantly blocked. Could that cause the rest of the program to stop working? if so why does the above code allow the counter to work?
Cheers! Richard

Thanks for contacting us, @Bambofy.
It's not clear what you're trying to do as it most probably be simpler than this. Give us some more context, so that we are able to guide you better.

i'm trying to load data from a service in the OnAfterRender function and then refreshing the page to update the values.

Why use OnAfterRender? Typically the pattern for loading data asynchronously in Blazor looks like this:
@if (myData == null)
{

something to show while data is loading

}
else
{

show your data here

}

@code {

    private MyData myData;

    protected override async Task OnInitialized()
    {
          myData = await LoadMyData();
    }
}

This pattern will render twice, once to show a "loading" UI and then again after the LoadMyData() method completes.
Do you have something special that you're trying to do here?

Why use OnAfterRender? Typically the pattern for loading data asynchronously in Blazor looks like this:
@if (myData == null)
{

something to show while data is loading

}
else
{

show your data here

}

@code {

    private MyData myData;

    protected override async Task OnInitialized()
    {
          myData = await LoadMyData();
    }
}

This pattern will render twice, once to show a "loading" UI and then again after the LoadMyData() method completes.
Do you have something special that you're trying to do here?

Hey i did not find the root cause of the original problem to be honest!
The way i'm making it work at the moment is like this:
    private static Type AccountType = null;
    private static string EmailHash = "";
    private static string PasswordHash = "";

    protected override async Task OnAfterRenderAsync()
    {
        // Ensure user is authenticated.
        if (AccountType == null)
        {
            await Authenticate();

            // Show new data.
            await Invoke(() =>
            {
                this.StateHasChanged();
            });
        }

        await JSRuntime.InvokeAsync<string>("WebpageReady");
    }

    protected async Task Authenticate()
    {
        /*
         * This authorization check can potentially be overriden since this code is clientside
         * Therefore we only use this clientside authorization check to render the page's DOM.
         */
        EmailHash = await JSRuntime.InvokeAsync<string>("GetSessionStorage", "EMAIL");
        PasswordHash = await JSRuntime.InvokeAsync<string>("GetSessionStorage", "PASSWORD");

        // Get the account type.
        AccountType = AccountService.GetAccountType(EmailHash, PasswordHash);
    }


[$500] Emojis - Adding emoji in middle of text moves the text to new line

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.8-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Navigate to staging.new.expensify.com
Open any conversation
Add few lines of text
Place the focus in the middle of the text and add emoji

Expected Result:
Emoji should be added in the middle of the text
Actual Result:
Adding emoji in middle of text moves the text to new line
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/2bdfd168-d926-49ae-875f-df93e92d8b5b
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01c5d5d24c7703f381
Upwork Job ID:<PHONE_NUMBER>482465280
Last Price Increase: 2023-12-05

set the value={state}
onchange={function}
inside of onchange
I'm grabbing the value and updating state
when they click an emoji im going to take the previous value add a space and then append the emoji inside of it.
This will fix this bug


Ratios not visible .. need a Gauge

It is important to have changes in UI / UX  so that user can experience the product seamlessly
What we need to do is to add the gauge on the screen so that users can better understand ratios

preferred way would be to have the user click on filter or slicer open/close