CARDS-1520: Update Visit Numbers for Cardiac Rehab Questionnaires

(cherry picked from commit aced3829e1348eeba9ab5391f3c2ac87a6a7db79)

@sdumitriu
There is a bug in this PR. I was able to create 12 Graded Walk Test (max is 100 and 6 CPET forms (max is 4) for the same patient.

@sdumitriu There is a bug in this PR. I was able to create 12 Graded Walk Test forms (max is 10) and 6 CPET forms (max is 4) for the same patient.

How did you create the forms? Manually through the UI or import?

How did you create the forms? Manually through the UI or import?

Manually.

Tested the dev branch after the merge and the problem stated above is not present, thus removing the bug tag.


Browser equivalent

The reactive stream bus and factory could be extended to support browsers where you have 3 tiers:
service worker
Web worker
Main window.
Then a developer can reuse wasm on the server and browser.
the only thing missing is the factory that would use Message Passing which browsers use to send data across the 3 boundaries list above.
There is a spec / project to try to do this and a few implementations going on. Can dig out the URL later

That's a cool idea!
Currently, Yomo focus on wasi, Presencejs would be proper consider this.
Wanna hear more about this topic.

https://github.com/magodo/go-wasmww Is worth a look.

hey @fanweixiao
https://wanix.sh
https://github.com/tractordev/wanix
can run Yomo in browser probably. it can run Hugo in browser for example.

@gedw99 let me check 🎅🏻


Using a Lambda@Edge function requires 'cdk bootstrap' to be run

I'm trying to use a Lambda@Edge function in a StaticSite (based on the docs), but whenever I add the function while running locally, I get a stack error:
stack: mnorman-myproject-distEdgeFunction: SSM parameter /cdk-bootstrap/abcdefgh/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)

Relevant stack setup:
  const edgeFunction = new experimental.EdgeFunction(
    stack,
    "distEdgeFunction",
    {
      runtime: Runtime.NODEJS_18_X,
      code: Code.fromAsset("packages/functions/src/project"),
      handler: "edge.handler",
      stackId: `${app.logicalPrefixedName("distEdgeFunction")}`,
    },
  );

  const dist = new StaticSite(stack, "dist", {
    cdk: {
      distribution: {
        defaultBehavior: {
          origin: new S3Origin(bucket.cdk.bucket, {}),
          viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
          trustedKeyGroups: [keyGroup],
          edgeLambdas: {
            functionVersion: edgeFunction.currentVersion,
            eventType: LambdaEdgeEventType.VIEWER_REQUEST,
          },
        },
        enableLogging: true,
      },
    },
  });

If I take out the edge lambda, there are no problems with local development, so it has something to do with local development and creating an edge function.
I wanted to use the EdgeFunction contract, but that doesn't appear to be exposed in the sst/constructs API.

If you've already posted your issue on Discord, make sure to leave a link to it here.

No sooner than I had written this, it appears that running cdk bootstrap aws://<accountid>/us-east-1 will solve the problem.
Maybe there's a way to configure SST to do this when not using us-east-1 for other resources, but I couldn't find any options in cdk-bootstrap config.

Ah interesting. Yeah these CDK resources need to specifically go to us-east-1. Running the CDK command works. We should also document that you can run sst bootstrap and that'll do the same.

We are transition this repo to https://github.com/sst/v2 and are closing this issue for now. If it is still relevant please reopen it there - sorry for the trouble


LoveLetters: Improve Share Feature

Requirements and design inspiration from [discord thread](https://discord.com/channels/841390338324824096/964698337230086174/989582461556105276).

Improve share feature for LoveLetters to surface the actual loveletters card as a PNG alongside the polyscan transaction details. (current state below)

@Atmosfearful

Does this task need a design first or is it pure dev work?
On the dev side, is this happening on the Wix end?

This is purely Wix work, I have no control over this at the moment and don't know what Wix supports-- should be possible though. We would need to update the <meta> tags in html to point at an image, examples in our code here: https://github.com/KlimaDAO/klimadao/blob/f272b23253fc3cec39d157cd906458b07a0d44ca/app/components/PageHead.tsx#L46

Not doing any more active feature dev for LL.


updating_the_managedVirtualNetworkSettings_property

MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[x] I commit to follow the Breaking Change Policy of “no breaking changes
[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

[x] Ensure to check this box if one of the following scenarios meet updates in the PR, so that label “WaitForARMFeedback” will be added automatically to involve ARM API Review. Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs, all “removals” and “adding a new property” no more require ARM API review.

Adding new API(s)
Adding a new API version
Adding a new service

[ ] Please ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If there are following updates in the PR, ensure to request an approval from API Review Board as defined in the Breaking Change Policy.

[ ] Removing API(s) in stable version
[ ] Removing properties in stable version
[ ] Removing API version(s) in stable version
[ ] Updating API in stable version with Breaking Change Validation errors
[ ] Updating API(s) in preview over 1 year

Please follow the link to find more details on PR review process.

/azp run automation - sdk

This is not a breaking change for service. This PR is an extension of https://github.com/Azure/azure-rest-api-specs/pull/10964

Hi,  as stated above This is not a breaking change for service. This PR is an extension of #10964
@idear1203  can you please approve in help in merging it  ?

Service is not GA, so approved the breaking change.


How to use markdown-it in nuxt

how to use markdown-it in nuxt ?

No idea what is nuxt. I guess, generic programming questions are out of this repo scope.


Downgrade build script for MSBuild 12.0.

Major caveat: I don't actually know what I am doing.
Instead of creating my comment as an issue I have made it a pull request.
When I try to run the build (build.bat) with MSBuild 12.0 the build tools complain that there are missing xmlns attributes on the .target Project tags, and that they do not understand the new(?) inlined CopyToOutputDirectory attributes for the None tags in the .fsproj files.
Can this be fixed, or will it just ruin other scenarios instead?

Hi @dsyme.
I will give the long answer to what I am doing. Feel free to skip or glance over my answer if that was not how your question was meant.
My main example is plugin programming for a computer game named Kerbal Space Program. Plugins are written for Unity (a product designed for .Net-like scripting via Mono) and need to target the equivalent of .NET 3.5 (a .NET Standard 2.0 compatible version of Unity has since been announced, and may or may not be retrofitted to existing games).
Most players/developers write their game plugins in C#. I favour F# and I think it is quite cool that we are able to write our cross platform game plugins in F#. However, the average player/developer probably does not usually dabble in F# and therefore needs access to an appropriate FSharp.Core.dll to get going in earnest. In an attempt to make it a bit easier to find I have uploaded a Nuget package with an Apache 2.0 licensed FSharp.Core.dll targeting .NET 3.5 (and thus suitable for Unity).
I then thought I would check if I could compile a newer FSharp.Core.dll targeting .NET 3.5 with its new MIT license. I happened to have MSBuild 12.0 (Visual Studio 2013) on my computer already and I found in build.bat that the build seems to explicitly support MSBuild 12.0. I then ran the default (net40 targeted) build and found out that I needed to resolve the two mentioned issues (xmlns attribute and CopyToOutputDirectory attribute) to have MSBuild 12.0 succeed.
So no good reason for using MSBuild 12.0 really, just happened to work out that way.

@Rodhern Thanks for the explanation!


Need a way to Reset settings (Runcher gets stuck when opened).

Describe your problem
Hey, I have downloaded the Runcher today and opened it. Initially it worked but then I tried to manually add a path for a game that it didn't recognize. I have closed the program and reopened it and now it no longer works. I figured out the new setting might have caused the issue so I went to the AppData folder and deleted it in hopes of removing the setting, however it didn't work. I figure out the settings are saved somewhere else (not in the extracted folder and not in Roaming).
Can you help me out in resetting the program to the default settings ?
Extra info?
I have used the latest version.

Runcher settings are not saved to appdata, but to the windows registry. Not sure exactly where QSettings saves them, but they have the "runcher" name on them so they should not be too hard to find.

Thanks, the problem was that I accidently made the default game the one that I added, I removed the setting in the registry.
Now there is the second problem to address, I am not sure why but the Runcher gets stuck when I try to enter the Thrones of Britanna (the game I manually added the path to), it does work fine with the games it did recognize (WH3 for example).
Are there specific instructions in the way that manual adding of games needs to be done ? (I just entered the folder's location)


Report a bug, in djstripe

Please check the line 20 in file init.py under djstripe.
if get_django_version() <= '1.7.x':   <--- Here, the version string can't directly compare
if django.VERSION <= (1,7):     <---  compared naturally
...
OS: Mac OSX 10.11
Python: 3.5.2
Django: 1.10.1,

@woolf-wen This was fixed in PR #348, but probably only exists on the 1.0.0 branch.  I'm not the maintainer for this repository but I'd recommend migrating to 1.0.0 ASAP or at least evaluating it for migration.  Check out the "master issue" for the 1.0.0 release (including details of changes) at: #307
HTH!

@woolf-wen: I cherry-picked the fix @lskillen mentioned onto the master branch a couple weeks ago. If you can't do the migration right away, I suggest pip installing from the master branch.
The next official pip release will be 1.0.0 (That'll happen once we finish up the remaining TODOs)


readHistory: type assertion

My is dead simple:
await tg.readHistory(message.chat);

But in the console more and more messages like:
| MtTypeAssertionError: Type assertion error at channels.readHistory: expected true, but got false
|     at assertTrue (file:///app/node_modules/@mtcute/core/utils/type-assertions.js:27:11)
|     at readHistory (file:///app/node_modules/@mtcute/core/highlevel/methods/messages/read-history.js:27:5)
|     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
|     at async Object.newMessageHandler [as callback] (file:///app/handlers/client/new-messages.js:26:5)
|     at async Dispatcher._dispatchUpdateNowImpl (file:///app/node_modules/@mtcute/dispatcher/dispatcher.js:286:26) {
|   context: 'channels.readHistory',
|   expected: 'true',
|   actual: 'false'
| }

It shows this warning in docker alpine container, macOS, anywhere.
Please give me advice how I can fix it.

does the history actually get read?

Looks like yes, every time actually read. But with warning.


Table of contents rendered incorrectly if no empty line is given before and after the [[toc]]

Example content:
a
[[toc]]
b

# heading1

Expected HTML:
a

// list of headings

b

heading1

Actual HTML is as below: The TOC is getting rendered weirdly.

This is the sample runKit code I tried for this repro.
var MarkdownIt = require("markdown-it")
var md = new MarkdownIt();

md.use(require("markdown-it-anchor"));
md.use(require("markdown-it-table-of-contents"), {
markerPattern: /^\[\[toc\]\]/im
});

var content = "a\n" +
"[[toc]]\n" +
"b\n" +
"# Heading1\n" +
"## Sub heading 1\n" +
"Some even nicer text\n"
md.render(content);

I debugged and I could get the root cause. We are assuming that there could be no '\n' character before the [[toc]] syntax in the given state based on which we set the state.pos. Hence the index we set for the next processing is incorrect and changes depending upon the position of the first '\n' in state.src. It works in if there is a line gap before the [[toc]] and after it.
I shall raise a PR trying to fix this issue.

Thanks! Fixed in https://github.com/Oktavilla/markdown-it-table-of-contents/releases/tag/v0.4.1


Kerl operations are pretty slow

I was trying to figure out why get_account_data sometimes takes seemingly forever. For example, take the following script:
from iota import Iota
import time

seed = 'A' * 81

api = Iota('http://astra2261.startdedicated.net:14265', seed)

start = time.time()
api.get_account_data()
print(f'total time: {time.time() - start}')

This took me 87.39378452301025 seconds to run.
I started profiling and noticed that one of the reasons is that that seed has a lot of addresses that have been used, and generating those addresses is not particularly fast. For example, here's another script that will generate 10 addresses:
from iota import Iota
import time
from iota.commands.extended.utils import iter_used_addresses
from iota.crypto.addresses import AddressGenerator

seed = 'A' * 81

api = Iota('http://foo.bar:14265', seed)
adapter = api.adapter

start = time.time()
addresses = []
for addy in AddressGenerator(api.seed).create_iterator(0):
    addresses.append(addy)
    if len(addresses) == 10:
        break
print(addresses)
print(f'total time: {time.time() - start}')

It takes me 4.875983476638794 seconds to just generate those addresses.
It seems a lot of the time is spent in iota/crypto/kerl/conv.py doing the conversion from bytes to trits and
trits to bytes. I played with re-working these functions to use cython and am getting some promising results. My hacking around so far is here (forgive the ugliness, it definitely needs polishing; I just wanted to see if I could get it working).
If I run the same scripts using the cython version as it is (and it could definitely be improved some), I got a time of 2.0970706939697266 to create the 10 addresses (less than half the time of the python version) and 68.44585180282593 to do get_account_data (about 20 seconds faster).
I might try to write a cython extension that can be installed (similar to how the curl extension is installed) that would provide the speedups for the kerl module. Obviously it would need extensive testing, but fortunately some test vectors already exist so with that and tests to compare it to the python version it should be possible to show that it produces equivalent outputs for a given input.
Would you be open to something like that?

Hey Scott!  Great observations, thanks for investigating this!
Using C extensions to speed up the crypto functionality is awesome, and we already have a precedent for that.
Using Cython is an interesting approach.  I must confess; I've never used it before.  Is this something that will work "out of the box", or will users have to install Cython before they can pip install pyota?
Is it possible to make this an optional install (e.g., pip install pyota[ccurl,ckerl])?

Users do not have to have cython installed if the c files are generated and packaged up on pypi. This is what we do on fastavro and what I would plan to do. They would of course need a compiler if compiled wheels aren't provided (as with any c extension AFAIK).
And yes, I think an optional install as you mentioned would make sense.

@scottbelden I know this might be an older post, but thank you for the C rework of this file! I'm doing a password recovery now and it is indeed much much faster at generating addresses. I didn't do a benchmark yet but from the console output I'm seeing is working blazing fast.

Thanks! I haven't looked at this in years but it's good to hear that at least some people have found it useful. It was a fun experience while implementing the changes.

After some headscratching regarding the huge memory leak, I've added this instruction  free(bytesArray) at line 196. From my limited understanding the memory was not being freed up at the end of the function. Works great now with no issues.


Webpage not found error 404

Problem encountered on https://dotnet.microsoft.com/en-us/learn/aspnet/microservice-tutorial/run
Operating System: ubuntu linux 22.04Lts

I'm sorry to hear you're running into issues @Yaswhanth.
Have you tried opening the route as instructed in the tutorial?
In your case, it should be localhost:5090/weatherforecast.


Scrollbar scroller goes behind navbars and toolbars

Scrollbar scroller goes behind navbars and toolbars. Also it is noticeable if additional paddings applied to pages.

It is by design, look at #690

Thanks for the info. Yes, seems logical. I applied custom CSS to fix this in cases when no backdrop or transparent navbars are needed. Also not needed for Material design.

Is it possible to introduce an option for showing proper scrollbar without having those features?
I'm showing quite large amount of text in Material Design and this beheaded scrollbar doesn't feel good.


Wget button glitched in "Download Your Cart"

Describe the bug

The button of wget download located in the bottom of the Cart page doesn't work.
Sometimes it helps me download the wget script for the files I need when I enter the cart page for the first time of the day, and it just never work again for the rest of the time.
Desktop (please complete the following information):

OS: Windows 10

Browser and Version: Microsoft Edge Version 126.0.2592.61 (Official Build) (x86_64)

To Reproduce

Add 100 files into my cart.
Enter my cart and choose to show 100 files per page.
Select all 100 files in every page
Scroll down to the bottom and choose 'wget' for 'Download Your Cart'
Click on 'Download'
Nothing happens

Expected behavior

After clicking on 'Download' the browser should start downloading a wget script for all selected files.
Screenshots

Additional context

(1) Can you try Chrome or Firefox?  Edge is not supported by the development team.  (2) I have found that logging out , back in (though the cart should persist) seems to fix the issue.  Also refresh the tab?

(1) Can you try Chrome or Firefox? Edge is not supported by the development team. (2) I have found that logging out , back in (though the cart should persist) seems to fix the issue. Also refresh the tab?

Logging out seems to work! ( sometimes however, I still looking forward to the fixing of that button thus it will be much more convenient for downloading data~)
Thanks for your assistance!

I am reopening as a reminder to the development team that we would like the button to work without the workaround in Chrome and Firefox...

Update: we have fixed this issue, being staged on https://esgf-dev1.llnl.gov/ note this is a test server and chance to be offline for frequent updates.

Deployed on aims2.llnl.gov


baseURL: allow value without host

If I set:
baseURL: https://example.com/sunday

I can create a link like this:
Sunday

result:
Sunday

the issue is that if I prefer to use "path absolute" rather than
"protocol absolute", Hugo doesnt really have an elegant way to do that. One of
these:
Sunday
Sunday

I tried changing baseURL to /sunday, but then hugo server returns:
Web Server is available at //localhost:1313///sunday/ (bind address <IP_ADDRESS>)

which gives 404 page not found. I am confused why Hugo requires a host with
baseURL if the typical workflow is to just build a site and host the result
in an arbitrary location. It seems Hugo doesnt need to know more beyond the
"path" part of the URL.

Here is a workaround:
Sunday

https://gohugo.io/variables/site

baseURL: ./sunday


Convert to Mono

Check how many channels it has
Convert to Mono

Google Cloud Speech only accepts 1 channel

Updated to check the number of channels of the audio file and convert to mono if channels exceed one.


Holistic solution hand_detections_from_pose_to_rects_calculator edge cases

Solution
Holistic
Describe the actual behavior
When using the holistic solution, the calculator first estimates the body pose and three points for each hand (wrist, index mcp and pinky mcp) - then, it estimates a rectangle of the hand area of interest, which should cover the full hand, to be sent for hand keypoints.
https://github.com/google/mediapipe/blob/master/mediapipe/modules/holistic_landmark/calculators/hand_detections_from_pose_to_rects_calculator.cc#L110-L121
There are some edge cases that in my view, do not create correct hand rects, and so fail to estimate the hands.
This happens when the hand estimation is off, or when the plane of the hands (the triangle created by the three points of interest) lies directly perpendicular to the camera. When this happens, the "area" of interest is tiny and so the crop will be wrong.
Hands model
If we look at the hands model that uses hand detection, it works well enough:
https://github.com/google/mediapipe/assets/5757359/7183a017-1256-48fa-a20d-083b0807cf47
Pose model
The pose model also works well enough, correctly predicting the general hand position
https://github.com/google/mediapipe/assets/5757359/a06b31a6-8a11-43d5-9bc3-e77ca73e80fc
Holistic model (pose + area + hands)
Holistic model works really well when parallel to the camera, but not when parallel to the floor.
https://github.com/google/mediapipe/assets/5757359/8ac50e97-d50b-4e23-b6d5-41b52fbdf3cf
If I recreate the holistic ROI cropping behavior (without rotation, and with rotation), I get the following. Note how the crop goes crazy.

Describe the expected behavior
I expect that the area of interest will always be correct if the hand estimation is correct, and if not, the hand model will be activated using the hand detection model as a backup.
Possible solution
Before the recroping model

If the hand pose estimation is bad, use the hands detection model
In the area-of-interest calculation, separate two cases based on the hand plane (calculation of the normal are trivial):
a. if the hand plane is parallel to the camera, do as you do now
b. if the hand plane is perpendicular to the camera, use a different calculation.
generic solution, would be to learn a tiny regression model, from the hand points (4 points in total) to the hand crop as predicted by the hand detection model. This is generic since it can just train on data, and is probably the best area of interest because it will predict similar bounding boxes to how the keypoints model was trained.

Alright, here is a solution
Optimizing_Hand_Area_Detection_in_MediaPipe_Holistic.pdf
https://github.com/sign-language-processing/mediapipe-hand-crop-fix


Adding test for model in image end to end use case

Adding test for Mii end to end use case :
create unique namespace for operator
create unique namespace for domain
create service account for operator
install operator
verify operator pod is running
create image using model file
create domain using model in image
verify domain is created
verify admin and managed pods are started and ready
verify admin and managed services are running
delete domain
delete domain image
uninstall operator
delete service account
delete domain namespace
delete operator namespace
Adding Docker Primitive class.

Closing this PR as I will be creating another PR against develop branch.


Remove duplicate ko-deploy-cert-manager make target

This is a fix for an oversight  in #5655  where I renamed one of the targets in response to a code review comment, but forgot to also update the .PHONY line above.
NONE

/kind cleanup

/retest

Thank you for fixing this typo.
/approve
/lgtm

/retest


Read-only properties should not show up in the request examples

Expected Behavior
Properties with readOnly: true should not be included in the client payload examples that show up in the right sidebar.
Actual Behavior
Readonly properties are included in the request payload for the client example requests.
Reproduction steps
1. Use Scalar with an OpenAPI schema that contains readOnly properties
2. Scroll down to an operation that specifies a request with readOnly properties
4. Example payload to the right contains readOnly properties

Link to site / OpenAPI Spec
No response
Version
latest
What browsers are you seeing the problem on?
Firefox

Thanks for the report, fixed in #782! :)

Thank you!

I think this might not be fixed - I just spotted this in my own code?

@adnweedon note that this issue is only for the payload in the example requests that appear to the right.

@federicobond I'm experiencing the same issue that @adnweedon reported. Should we open another issue regarding the request documentation? It can be very confusing seeing an id, read-only, property as part of the request body.

Yes, I agree. I think it merits opening another issue.


Open search dialogs in separate windows from a button next to each base

Already in progress according to @KataBoku 's report.

@KataBoku will use https://angular-ui.github.io/bootstrap/ Collapse element.

Will also resolve the #150 .

Closed by #210 .


pip 10.0 dropped support for Python 2.6

From https://pip.pypa.io/en/stable/news/#b1-2018-03-31

pip 10.0.0b1 (2018-03-31) - dropped support for Python 2.6. (#4343)

How can I force some some machine to use pip <10? (pip 9.0.3 works just fine!)
'Cause this code stopped to work:
python_runtime '2.6'
python_virtualenv '/acme/tellah/env/'

Thanks!

Some code like this https://github.com/poise/poise-python/blob/e7528df332cea433e8676194b33bc2f8c59cc5e1/lib/poise_python/resources/python_runtime_pip.rb#L112-L125

Is it not already working correctly?


blockchain: add utxocache/utxoviewpoint code from btcd

The utxo cache code in the current code is fragile and less performant.
Thus we're replacing it with the better code from btcd.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

605 of 769 (78.67%)  changed or added relevant lines in 6 files are covered.
22 unchanged lines in 5 files lost coverage.
Overall coverage increased (+0.2%) to 52.634%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

blockchain/validate.go
16
18
88.89%

blockchain/sizehelper.go
84
87
96.55%

blockchain/chainio.go
33
42
78.57%

blockchain/chain.go
62
86
72.09%

blockchain/utxocache.go
244
299
81.61%

blockchain/utxoviewpoint.go
166
237
70.04%

Files with Coverage Reduction
New Missed Lines
%

blockchain/chainio.go
1
57.97%

blockchain/utxoviewpoint.go
2
63.57%

database/ffldb/blockio.go
4
91.44%

blockchain/utxocache.go
7
76.14%

peer/peer.go
8
74.82%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
28271

Relevant Lines:
53712

💛  - Coveralls


Remove left-over .diff file, changes already applied in git

@atodorov Thanks for catching my mistake, haha


Performance issues in huge files

I need to process huge files with very custom syntax. So I made a fast parser and took 10MB file to test it. The parser worked as follows: the text is buffered (Buffer = Ctl.GetTextRange(..)), then it's parsed into virtual regions (2 x int struct), then styled - each distinct style is applied to editor control with Ctl.SetStyling. And the results: "Buffer loaded in: 0.1s, Parsed in: 7.1s, Syled in: 19.1s". Of course it's done only once, when you show the last page.
Why is applying styles so slow? I believe visual operations are not performed for invisible part, so most of the time is used for calculations. I'm afraid it's the evil UTF-8 internal buffer. Is there any hope it will be replaced with native UTF-16 support? Maybe a single offset map could be created on loading, and updated before huge reparse operations? Memory is cheap. BTW, what is 100MB in modern PC? Geez, even in ancient ones it's nothing. Basic byte shuffling operations are blazing fast in C#. Maybe not fast enough for real-time 3D graphics, but surely fast enough to handle Ctrl+End event on 10MB file.
First: is styling slow because UTF-16 / UTF-8 conversions? If so, I would try to address it by creating offset buffer for styling, but what's the best way to access native offsets and native methods? Or maybe ScintillaNET already does it?
There is another way to speed up displaying of huge files (I'm going for it right now). I'll create a new event "VisibleContentChanged" and connect my parser to it. This would unfortunately break automatic OnStyleNeeded() action. I tested it with quick and dirty code - it seems to work pretty fast (like real-time), however it will fail to correctly highlight regions larger than single screen content. It's a minor issue which can be solved by implementing new OnStyleNeeded() event based on virtual region maps.
Last but not least - for huge files I use BeginInvoke() and Application.DoEvents() triggered each 1KB - the performance impact is completely insignificant, but the application responsiveness is way better. You can view and edit the text with no problems while it's still finishing parsing and styling.

Why is applying styles so slow? I believe visual operations are not performed for invisible part, so most of the time is used for calculations.

I'm not entire sure that is a true statement. Depending on the style definitions, it's possible that applying a style--even to an unseen portion of the document--could require a redraw. This is because the text dimensions (height and width) could change from the styles used. Those measurements require a redraw to calculate the updated dimensions and scroll positions. This is just a hunch. To be sure you could attached Spy++ and watch the WM_PAINT messages.

I'm afraid it's the evil UTF-8 internal buffer. Is there any hope it will be replaced with native UTF-16 support?

You can be sure that if I maintained Scintilla I would support UTF-16, but I only maintain the wrapper. It is what it is and internally Scintilla will continue to be UTF-8 for the foreseeable future.

Maybe a single offset map could be created on loading, and updated before huge reparse operations?

That's actually how the UTF-8 to UTF-16 conversion layer in ScintillaNET works. However, instead of maintaining a map of each byte/character offset, it keeps one for each line. This usually provides the best balance. A translation therefore usually first identifies the delta for the line, and then for the character within that line. I suppose it's possible to maintain a map for each character but that would likely cause a performance hit when the document changes. Sure the lookups would be fast, but a modification would require updating the offsets to each character.

is styling slow because UTF-16 / UTF-8 conversions? If so, I would try to address it by creating offset buffer for styling, but what's the best way to access native offsets and native methods?

If for any reason you don't like the way ScintillaNET handles something you can use the DirectMessage API and native Scintilla documentation to sidestep it (i.e. DirectMessage(SCI_STARTSTYLING, ...). It's not without risk, but you could directly interface with the underlying Scintilla control if you wanted to. Not knowing more about the issue I couldn't say whether you would get a performance increase or not by doing this. As I mentioned earlier, there is already a byte/character map in ScintillaNET to make lookups fast. If your performance problem is in styling, there is likely to be little gain because there is currently very little overhead in the ScintillaNET mapping lookup time; but I could be wrong....

There is another way to speed up displaying of huge files (I'm going for it right now). I'll create a new event "VisibleContentChanged" and connect my parser to it.

That's a perfectly fine approach and definitely one I would recommend if you can. As you pointed out, it means that you might not know where in the document you are starting your styling and contextually sensitive styling might get messed up. The StyleNeeded event avoids this by playing catch-up, but if you don't need to then don't use that event.
If it is acceptable in your application to show moments of unstyled text, you could be very leisurely about how you approach styling. Once you're not using the StyleNeeded event, you could style the text at any pace/priority you want. Most editors take this approach and will show moments of unstyled text while the parser/styler catch up.

for huge files I use BeginInvoke() and Application.DoEvents()

Without more context I couldn't say whether this is a good idea or not, but generally speaking using Application.DoEvents() is a bad idea. That might be a sign that something else is wrong in your application. You could inadvertently be preventing the Windows message loop from batching multiple paint events into one cycle.
Final thoughts... loading a 10MB file is not what most developers use Scintilla(NET) for and thus isn't something that is specifically optimized for. There could very well be a performance problem in ScintillaNET and if you want to help identify it I would be happy to work with you on a fix. But it might not be a "performance problem" at all. It might just be an unreal expectation that ScintillaNET is going to handle a 10MB file without using some tricks in your code to work around those design decisions/limitations.

Thank you for the great insight. I'm very glad the UTF-8 calculations are already made perfectly in ScintillaNET. You were right about Application.DoEvents() being no good in lexer code. I used UpdateUI event to apply lexer and now it works with 10MB files in real-time, no problems whatsoever.
I agree, it's not a performance issue, but I still can make some little enhancements to implement custom (Container) lexers easier, with various optimizations. I'll be glad to share my the code when it's ready.
(BTW, yesterday I took closer look at the source code, I'm impressed with the quality.)

Cool. :)


RuntimeError: Not compiled with GPU support

Hi,
The following error message happens on my cuda 10.1 machine.
  File "/home/shuchen/miniconda3/envs/cl/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shuchen/miniconda3/envs/cl/lib/python3.9/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/shuchen/miniconda3/envs/cl/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shuchen/data/github/prj-uma-keypoint/DCNv2_latest/dcn_v2.py", line 182, in forward
    return dcn_v2_conv(
  File "/home/shuchen/data/github/prj-uma-keypoint/DCNv2_latest/dcn_v2.py", line 25, in forward
    output = _backend.dcn_v2_forward(
RuntimeError: Not compiled with GPU support

And the same problem happen to CUDA 10.2 as well

Hi, same happens to cuda 11.0

have you solved this problem?please help me.

So frustrated!
Has anyone solved this problem?
Please do me a favor.


Remove server auth filter allocation

****************************************************************

libgrpc.so

     VM SIZE                                                             FILE SIZE
 ++++++++++++++ GROWING                                               ++++++++++++++
  +0.0%     +29 [None]                                                   +109  +0.0%
  +0.6%     +19 src/core/lib/security/transport/client_auth_filter.cc     +19  +0.6%
      +5.1%      +8 init_call_elem                                             +8  +5.1%
      +9.2%      +6 [Unmapped]                                                 +6  +9.2%
      +0.5%      +3 auth_start_transport_stream_op_batch                       +3  +0.5%
      +1.7%      +1 cancel_check_call_host                                     +1  +1.7%
      +2.3%      +1 cancel_get_request_metadata                                +1  +2.3%

 -------------- SHRINKING                                             --------------
  -0.5%     -16 src/core/lib/security/context/security_context.cc         -16  -0.5%
     -10.7%     -19 [Unmapped]                                                -19 -10.7%
      [DEL]     -10 grpc_client_security_context_create                       -10  [DEL]
      -2.9%      -8 grpc_client_security_context_destroy                       -8  -2.9%

  +0.0%     +32 TOTAL                                                    +112  +0.0%

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[trickle] No significant performance differences

[microbenchmarks] No significant performance differences


Load markers dynamically

Hi guys,
I need to load markers in my map, but I want to load just those who are in the shown area of the app.
To do this I'd need the most northeast and the most southwest coordinates shown in the map. So I could find the markers using the $geoWithin and $box operator from MongoDB.
How I could get the map bounds? The coordinate more southwest and northeast of the shwon area of the map?

Code for bounds is in here. Search for bounds or delta

@texas697 the region gives me an object with the following fields:
timestamp
coords: { speed, heading, accuracy, longitude, altitude, latitude}
do you have a sample code?
Thanks

https://github.com/lelandrichardson/react-native-maps/issues/356

Thanks @texas697 :D


DOP-1573: Add directive for banner support

About:
Adds support for a banner directive as a part of https://jira.mongodb.org/browse/DOP-1573.

Seems reasonable enough - I'll look at making a change to support 👍

This is a TOML syntax thing: double brackets indicating a list of dictionaries.
If it were invalid it would fail in CI 😸
On Thu, Jul 8, 2021 at 9:38 AM, Allison Reinheimer Moore @.***> wrote:

@schmalliso commented on this pull request.

In test_data/test_postprocessor/snooty.toml:

@@ -12,3 +12,8 @@ blank = ""
[deprecated_versions]
manual = ["v2.2", "v2.4", "v2.6", "v3.0", "v3.2", "v3.4"]
mms = ["v1.1", "v1.2", "v1.3"]

+[[banners]]
are there too many [ here?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

This is a TOML syntax thing: double brackets indicating a list of dictionaries. If it were invalid it would fail in CI 😸
…
On Thu, Jul 8, 2021 at 9:38 AM, Allison Reinheimer Moore @.***> wrote: @schmalliso commented on this pull request. --------------------------------------------------------------- In [test_data/test_postprocessor/snooty.toml](#308 (comment)): > @@ -12,3 +12,8 @@ blank = "" [deprecated_versions] manual = ["v2.2", "v2.4", "v2.6", "v3.0", "v3.2", "v3.4"] mms = ["v1.1", "v1.2", "v1.3"] + +[[banners]] are there too many [ here? — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#308 (review)), or unsubscribe.

FANCY


Add notification for change in attendance

What?
The app does not seem to keep track of attendance of users.
How?
It'll be really nice to have the app watch the attendance and notify the user that the attendance has been updated.
Here's what I have in mind :
Let the user have enable it in settings (if he requires attendance modified alerts ) but this will require the credentials to be stored somewhere I guess? Feasible?
If enabled, there can also be an option to let the user decide how frequently (in days) the user wants the app to check for changes in the attendance
Relevance : Some teachers update attendance regularly whereas some do so only during the weekends / Tuesdays when the attendance gets locked.

Sounds cool. I'll give it a try.

@GKGokul sure ! Go ahead 😄


Install cri-tools on Amazon Linux 2

What this PR does / why we need it:
Amazon Linux 2 doesn't install crictl by default. We use crictl to restart the API server if it's affected by #1222. Additionally, kubeadm verifies is crictl present if containerd is used.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1281
Does this PR introduce a user-facing change?:
Install cri-tools (crictl) on Amazon Linux 2. This fixes the issue with provisioning Kubernetes and Amazon EKS-D clusters on Amazon Linux 2

/assign @kron4eg

/hold
to manually test the changes

/retest

/retest
/hold cancel

/cherrypick release/v1.2


Add generic Psr17Factory

Q
A

Bug fix?
no

New feature?
yes

BC breaks?
no

Deprecations?
no

Related tickets
-

Documentation
https://github.com/php-http/documentation/pull/303

License
MIT

This PR ports WellKnownPsr17Factory to this repository.
This approach should make it even simpler to write generic PSR17-consuming code.
It would also make the package compatible with factories declared as default values.
Last but not least, the implementation in Psr17Factory provides two additional methods

Psr17Factory::createServerRequestFromGlobals()
Psr17Factory::createUriFromGlobals()

Providing these two methods provides true portability of PSR-7 implementations.
An example that shows this is missing is Sentry's RequestFetcher, which is currently coupled to guzzlehttp/psr7 because of the call to ServerRequest::fromGlobals().
Example Usage
Inline:
$factory = new Psr17Factory();

// Now we can call any PSR-17 method, e.g.
$factory->createRequest();

As default value on PHP8+:
class Foo
{
    public function __construct(
        private RequestFactoryInterface $requestFactory = new Psr17Factory(),
    ) {
        //...
    }
}

To Do

[x] Updated CHANGELOG.md
[x] Documentation pull request created
[x] Fix tests

it does not directly have to do with discovery, not convinced if it fits into this repository.

Arf. To me, discovery is the package where this makes most sense, because that class is using discovery, and because it fills the same need - decoupling to any specific PSR-17 implem. Making a separate package would just make maintenance harder and slow down adoption to me.

thanks for the discussion in slack @nicolas-grekas , now i got what it does. its a decorator around the findX methods that provides you with an all-of-psr17 factory that autodiscovers its things if needed but accepts constructor arguments when autodiscovery is not wanted.
yep, that fits into this package then.

I just pushed tests for the *FromGlobals() methods. PR is ready on my side.


Option to open the function definitions from within the program

Motivation
When someone installs the program using the .msi installer to the default location, it is hard to locate the functions.json and constants.json files, as well as the formulas folder. However, reaching these is necessary, if you want to extend the functionality of the program.
Task description
Add a new menu called e.g. Edit by similar procedure as the one used here
https://github.com/SargonCZ/laser-calculator/blob/9675425834e7c90ecfbad657703596b9bdb31ffc/laser-calculator.py#L65-L68
This menu will have three commands, for opening the two files in the default editor and the mentioned folder in the file explorer. Relative path to the targeted files/folder can be used, so this will be simple. Since the three indiviual functions should be very short and contain only one command, they can be written as arguments of the command keyword with lambda notation.

Please have a look at the PR for this.

Hi @Aadityajoshi151, thanks. It works well. However, when I look at it, maybe calling it (Re)define functions and (Re)define contanst could be better?
Could you also update the readme? You don't have to update all the screenshots, but make a mention in the the appropriate section abou thte possibility to open the files form the program itself.
Afterwards, I will approve the PR.

Hi @SargonCZ
I am sorry for asking again but I am afraid I did not understand it.
Are you talking about replacing "Open functions.json" and "Open constants.json" with "(Re)define functions" and "(Re)define constants" respectively?

Exactly. Maybe it will be more user friendly? But I don't insist.

Yes, I think it can be more user-friendly because, for a developer, it would make sense but if someone is not from a programming background, he/she may not understand the "JSON" terminology.
I will make the required changes.

Hello, I have made the required changes in the names of menu items and also in the readme file to the best of my knowledge.
The "Prettier" extension on my vs-code has refactored the readme file without changing the order or content of the file. I hope that is not an issue.
Thank You


Bug: extract service will enter CrashingOnLoop until Privileged mode is disabled

Describe the bug
We noticed that the extract service will enter into crashing on loop if privileged mode is enabled. This seems to be related to heartbeat, as we can see a heartbeat error when you describe the pod.
To Reproduce
Steps to reproduce the behaviour:

Go to Services tab
Click on Extract, and make sure Privileged mode is enabled
Save settings and wait for new pod to come up, you'll notice this pod will never come up unless you disable privileged mode

Expected behaviour
Should work in both privileged and normal mode.
Screenshots
Can replicate if required
Environment (please complete the following information if pertinent):

Assemblyline Version: <IP_ADDRESS>
Browser: Edge

Additional context
Happy to provide any additional information as needed.

Some questions (assuming Kubernetes deployment):

Is the issue exclusive to Extract running in privileged or any service?
"heartbeat error when you describe the pod": sounds related to the probes, is the cluster particularly busy at the time when this happens?
TLS enabled?
"you'll notice this pod will never come up unless you disable privileged mode": is Scaler not scaling up the deployments or is the pod not reaching a ready state?

Some questions (assuming Kubernetes deployment):

Is the issue exclusive to Extract running in privileged or any service?

Yes just the extract service

"heartbeat error when you describe the pod": sounds related to the probes, is the cluster particularly busy at the time when this happens?

No, this is the dev environment no processing happening at this time.

TLS enabled?

Not on Dev no

"you'll notice this pod will never come up unless you disable privileged mode": is Scaler not scaling up the deployments or is the pod not reaching a ready state?

The pod never reaches a ready state, describing pods shows "healthz" failed

Would you be able to provide:

Service version
Resource allocation for the service (requests/limits for CPU, RAM)

If Extract is the only privileged service that's probe is failing (but others are working), I'd be curious as to why. Perhaps it's a broken service build or the config.yaml isn't being shared properly to that service and needs to be re-created by Scaler.

Any updates regarding this issue?

Issue is present on both dev and prod, even on latest version. Pods have been restarted multiple times, and issue is only on Extract, other services are set to privileged and work fine.

Actually I see the same issue on the new Ancestry service. VirusTotal Service works fine.

Any updates on this?

Going to close the issue for now. If this is still an issue, feel free to re-open with more information😀


yum_package on Red Hat 8 fails

Description
yum_package does not work on Red Hat 8.
Chef Version
15.x.x
Platform Version
Red Hat 8
Replication Case

yum_package 'bingo'

Client Output
cannot find yum libraries, you may need to use dnf_package

@johnypony3 This is the expected output. RHEL 8 no longer ships with yum out of the box so there's no yum utilities for chef to interact with. You can update this code to be dnf_package, which will use the dnf utility in RHEL 8, but you'd probably be better off just using package instead. Package includes the magic to use the appropriate tooling under the hood depending on release.

This should not be chefs stance on this. We now need to convert hundreds of cookbooks to use package. It would be much better if chef switched to dnf_package and added a deprecated call out to the log.

We've always taken the stance that users should not use yum_package directly unless they absolutely needed to access yum on a system. The documentation for yum_package itself suggests using package instead: https://docs.chef.io/resources/yum_package. The package resource is designed for a user that wants a package installed and doesn't particularly care how chef-client goes about doing that. The lower level specific resources are designed for power users that want to override the guidance of chef-client. Since these are designed to interface with particular package managers it wouldn't be appropriate to have yum_package using dnf_package instead.
If you need a quick workaround you can just install yum during bootstrap or use chef to run package 'yum'.

You can do that manually yourself just by throwing something like:
Chef::Resource::DnfPackage.provides :yum_package, platform_family: "rhel", platform_version: ">= 8"
Chef::Provider::Package::Yum.provides :yum_package, platform_family: "rhel", platform_version: ">= 8"

Into any library file

we have hundreds of cookbooks controlled by two chef servers across multiple orgs with policies and environments. this is not a great solution.

That globally fixes it across all cookbooks running on the machine, you just have to add that to any base cookbook or drop a library into any set of base cookbooks that winds up on every server.

thanks! i will test that. i appreciate your time.


Add XML documentation

Add XML docs in the code, generate the XML files during build, and include them in the NuGet packages.

We definitely need to write more xmldocs overall, its been a long standing issue and something we can hopefully block a week or two soon for.
We do not currently include documentation file for the netstandard1.3 artifacts in our nuget package, we'll remedy this ASAP.
net45 and net46 do already include sourcelink enabled XML files in the published nuget packages.


Using ß in Templates causes invalid byte sequence error

I'm getting the Error:  invalid byte sequence in US-ASCII when using the # as part of a template. I found jekyll/jekyll#960 and tried using utf-8 in _config.yml without any success. Guess this shouldn't be the problem as I'm running Jekyll in Version 2.5.3 with ruby 2.1.6p336.
I get the below error when trying to create a new post with octopress in version 3.0.0.rc.33. Any suggestions? Not sure whether this is an octopress issue or an jekyll issue. The used Template for post is also shown below.
> octopress new post "Dateien erstellen" --dir c-programmieren --trace
/usr/local/bundle/gems/octopress-3.0.0.rc.33/lib/octopress/page.rb:176:in `parse_template': invalid byte sequence in US-ASCII (ArgumentError)
	from /usr/local/bundle/gems/octopress-3.0.0.rc.33/lib/octopress/page.rb:135:in `content'
	from /usr/local/bundle/gems/octopress-3.0.0.rc.33/lib/octopress/page.rb:30:in `initialize'
	from /usr/local/bundle/gems/octopress-3.0.0.rc.33/lib/octopress/commands/new.rb:52:in `new'
	from /usr/local/bundle/gems/octopress-3.0.0.rc.33/lib/octopress/commands/new.rb:52:in `block (3 levels) in init_with_program'
	from /usr/local/bundle/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `call'
	from /usr/local/bundle/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `block in execute'
	from /usr/local/bundle/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `each'
	from /usr/local/bundle/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `execute'
	from /usr/local/bundle/gems/mercenary-0.3.5/lib/mercenary/program.rb:42:in `go'
	from /usr/local/bundle/gems/mercenary-0.3.5/lib/mercenary.rb:19:in `program'
	from /usr/local/bundle/gems/octopress-3.0.0.rc.33/bin/octopress:16:in `<top (required)>'
	from /usr/local/bundle/bin/octopress:16:in `load'
	from /usr/local/bundle/bin/octopress:16:in `<main>

Here the content of _drafts/post file:
---
layout: {{ layout }}
title: {{ title }}
excerpt:
categories: {{ dir }}
lernmoment: {{ dir }}
tags: []
image:
  feature: {{ dir }}.jpg
modified:
---

Jetzt viel Spaß mit

Jan

### Merke

-

### LernQuiz

Verwende folgende Fragen um das gelernte von heute zu festigen:

-

### Weitere Informationen

Perhaps this helps
export LANG=en_US.utf-8 :)

@tooh Just tried it, but the problem remains.
I just realised that the problem isn't the #. Instead I'm using an ß in one of my sentences. This seems to cause the trouble. I had a look for "Umlaute", but missed the ß. Sorry for this confusion!

This is usually because of editor or shell encoding not being set to utf-8. I assume your editor is writing ASCII to the template and that's causing the error.

This really is something that should be solved in the shell or the editor, but I'm trying something to see if I can force the template to UTF-8. If you're using a Gemfile for your site, update it to gem 'octopress', '3.0.12.pre.1' or if you don't run gem update octopress --pre. Then give this another try to see if you still  get the error.

@imathis Such a quick commit. Thanks! Unfortunately only the error message changed. I now get octopress 3.0.12.pre.1 | Error:  "\xC3" on US-ASCII. Do you need any other information from my setup?
By every means I don't understand this. When looking into env I see LANG=en_US.utf-8. Sublime, my editor, shows, that I'm saving my file in utf-8.
Not sure whether it is related, but I'm wondering whether the locale: de_DE entry in my _config.yml has any impact on this. I'm not sure what it is for, but it was used in the theme I'm using. So I changed it from en_US to de_DE. Tried it with the old setting without any impact, but you never know.

Finally! I figured out that it is not an issue with Octopress. I'm running it inside a docker container. There the settings for localisation were not properly set.
Running the command locale inside the container showed that everything was set to "Posix". After setting -e "LANG=C.UTF-8" on docker run, everything works great.
Thanks for your support!

By the way. Keep in mind that on many docker images en_US.UTF-8 is not available. When using debian as base distribution only C.UTF-8 will be available.


UnifiedDiffParserException on apparently sane file (3rd variant)

Describe the bug
UnifiedDiffReader.parseUnifiedDiff throws UnifiedDiffParserException on a patch which apparently looks good:
https://git.centos.org/rpms/avahi/raw/c8/f/SOURCES/0001-avahi-python-Use-the-agnostic-DBM-interface.patch
To Reproduce
Run this program: https://gist.github.com/moio/18c8452246bc46a8b6e36c1719c0f831
(contains the literal patch as a string)
Expected behavior
No Exceptions.
Actual behavior
Dec 21, 2020 10:12:32 AM com.github.difflib.unifieddiff.UnifiedDiffReader processLine
WARNING:   >>> no rule matched similarity index 87%
Exception in thread "main" com.github.difflib.unifieddiff.UnifiedDiffParserException: expected file start line not found
	at com.github.difflib.unifieddiff.UnifiedDiffReader.parse(UnifiedDiffReader.java:95)
	at com.github.difflib.unifieddiff.UnifiedDiffReader.parseUnifiedDiff(UnifiedDiffReader.java:151)

System

Java version: openjdk-11
Version: 4.9

Notes
:wave: java-diff-utils maintainers! Here's the third one. I might produce even more going forward, please bear with me :-)

support for

similarity index
rename from
rename to

Excellent!!!

Excellent!!!


feat: allow overriding keymap description

Description
The fields of keymaps will now accept either a mapping as a string or a table that includes a description override for that mapping.
Overview
Users can now provide a table for a keymap which will include the mapping and a desc to override the description. This is useful for which-key, since the default descriptions are verbose and get truncated.
Example:
keymaps = {
   insert = { mapping = "<C-g>s", desc = "surround cursor",
   insert_line = false,
   normal = { mapping = "gs" },
   normal_cur = { mapping = "gss", desc = "Add a surrounding pair around the current line (normal mode)" },
   ...
}

I also went ahead and did a few minor refactors, such as removing duplication of default keymap descriptions.
Please let me know if this isn't wanted or if there should be a different way to implement it. THANKS!

I'm a bit torn about this, since I think this is moreso a symptom of a larger issue that the setup function creates an unnecessary layer of abstraction over configuration. In this particular case, what comes to mind is as a workaround: simply set all keymaps to false via the current setup function, then use vim.keymap.set to customize as much as one wants. However, this seems like just a bad replacement for having the user create all their keymaps themselves, and merely expose <Plug> mappings with suggestions for how to use the plugin. I'll definitely have to think about this more, but at the moment am not particularly keen on merging this PR as it adds complexity for something that isn't exactly part of the core functionality of the plugin.
Further considerations for setup: https://mrcjkb.dev/posts/2023-08-22-setup.html

@kylechui fair enough! Thanks for the quick response. I think setting all to false is fine for now, since there aren't that many keymaps. Closing.


Add remove followers functionality

Hi there, I implemented this feature for myself. I'm not sure if you're looking for it but this is was maintained and a great starting point so I figured I should pay it back :). I don't write much Go and did this fairly quickly so very open to any feedback.
One this I wasn't sure was if I should be sleeping the loop for longer since I'm making 3 requests to the API in a row.

Add a new command: remove-followers / remove / rm
This new command will remove a list of your followers passed in by first blocking then unblocking them.
Renames processNotMutual -> unfollowUsers - this seemed like a function that could be used more generally, so I felt renaming it may be nice for the future.

Ghost followers are becoming increasingly important with Instagram's algorithm; users that follow you but are inactive lower your overall engagement, which means your posts/stories/etc are shown to less people.
Initially, I wanted to unfollow a user directly, but this required additional updates to the downstream instagram API package that I didn't want to deal with currently (and I couldn't get it working, haha).
Blocking and unblocking has the same effect. DMs are left in place and comments + previous likes are only removed for the duration of the block.

https://www.techuntold.com/what-happens-when-you-block-someone-on-instagram/

Thanks for this PR!  I didn't know about this Instagram feature. Especial thanks for the detailed description!


Language feature: barriers

We want to have the same types of safety on the GPU as we do on the CPU. One of those areas that make it easy to introduce race conditions is with memory barriers. HLSL and GLSL require the programmer to explicitly put them in the right locations without any formal verifiction.
This is an attempt at solving that within the Rust type system (attempt still has racy bugs and is largely incomplete).
use core::ops::Deref;
use core::ops::DerefMut;
use core::ops::Index;
use core::ops::IndexMut;

struct MyFoo {
    test: GroupSharedArray<u32>,
}

struct GroupSharedArray<T>
where
    T: Default + Copy,
{
    data: Box<[T]>,
    size: usize,
}

impl<T: Default + Copy> GroupSharedArray<T> {
    fn new() -> Self {
        Self {
            size: 100,
            data: Box::new([T::default(); 100]),
        }
    }
}

struct GroupSharedWriter<T> {
    data: T,
}

impl<T> Deref for GroupSharedWriter<T> {
    type Target = T;

    fn deref(&self) -> &Self::Target {
        &self.data
    }
}

impl<T> DerefMut for GroupSharedWriter<T> {
    fn deref_mut(&mut self) -> &mut Self::Target {
        &mut self.data
    }
}

impl<T> GroupSharedWriter<T> {
    fn new(data: T) -> Self {
        Self { data }
    }
    fn barrier(self) -> GroupSharedReader<T> {
        GroupSharedReader { data: self.data }
    }
}

struct GroupSharedReader<T> {
    data: T,
}

impl<T> Deref for GroupSharedReader<T> {
    type Target = T;

    fn deref(&self) -> &Self::Target {
        &self.data
    }
}

impl<T> GroupSharedReader<T> {
    fn barrier(self) -> GroupSharedWriter<T> {
        GroupSharedWriter { data: self.data }
    }
}

impl<T: Default + Copy> Index<Uniform<u32>> for GroupSharedArray<T> {
    type Output = T;

    fn index(&self, index: Uniform<u32>) -> &Self::Output {
        &self.data[index.data as usize]
    }
}

impl<T: Default + Copy> IndexMut<Uniform<u32>> for GroupSharedArray<T> {
    fn index_mut(&mut self, index: Uniform<u32>) -> &mut Self::Output {
        &mut self.data[index.data as usize]
    }
}

struct Uniform<T> {
    data: T,
}

impl<T> Uniform<T> {
    fn new(data: T) -> Self {
        Self { data }
    }
}

fn thread_idx() -> Uniform<u32> {
    Uniform::new(0)
}

fn main() {
    let mut data = GroupSharedWriter::new(MyFoo {
        test: GroupSharedArray::new(),
    }); // new does barrier

    data.test[thread_idx()] = thread_idx().data;
    data.test[thread_idx()] = 666; // should be illegal

    let value = data.test[thread_idx()]; // should be illegal

    //data.test[0] = thread_idx().data; // what happens in this case? it's unsafe/unsound. race cnd over contents of `test`

    let other_data = data.barrier(); // `barrier` return GroupSharedReader

    // some race-y cases handled correctly:
    // data.test[thread_idx()] = 666; // is correctly marked illegal
    // other_data.test[thread_idx()] = 666; // is correctly marked illegal

    let test = other_data.data.test[thread_idx()];
}

// tl;dr
// - use move semantics to enforce barriers betwen reading and writing
// - broken right now because DerefMut and IndexMut require Deref and Index to be implemented
// - broken right now because we can write to the same location multiple times (data race)

We had the following discussion in the Discord about trying to bring some of Rust's "fearless concurrency" to rust-gpu, by tweaking the way that accesses to shared memory (Buffers/UAVs, Workgroup memory, etc.) is managed when translating to SPIR-V. Just so we don't lose the conversation, I've pasted it below - can try to tease out more details later but for now we just want to be able to stow this so we don't lose it...

Tobias Hector Today at 00:30
So I've been thinking about the buffer/workgroup memory interface and how they need to ideally interact with synchronization. There's two key parts that kind of need to interact. The first is memory ordering - your typical acquire/release semantics as baseline here. Vulkan/SPIR-V also have some interesting additional semantics though once you look at the way the memory model works; notably values are not "coherent" by default like they are on CPU - you have to explicitly tell the compiler to make them visible to other invocations at a given scope. Acquire and release are also scoped - typically you want these scopes to agree with the visibility scope. (The handful of use cases for NOT doing that are really not worth worrying about in the first iteration outside of a single exception which is easy to deal with).
The other part is execution synchronization - telling threads in e.g. a workgroup, subgroup (/warp/wave), or whatever else, to get together so you don't race accesses. This needs to interact with the acquire/release operations, but the crucial part of it initially is simply that you probably need the acquire/release operations to know about sync operations in order to do useful compile time checks. The details of that aren't my concern right now.
So what I'm thinking is that what would be incredibly nice, is if you could write code like this:
struct Buffer {
    data: [u32; 32],
}

fn shader(buffer: &mut Buffer) {
    buffer.data[SubgroupInvocationId()] = 2;
}

And have it fill in the appropriate acquire/release/coherency - without toooo much pain in terms of annotations.
Exploding out the "shader" function above, it should generate spir-v something like this:
fn shader(buffer: &mut Buffer) {
    let blah = &mut buffer.data[SubgroupInvocationId()];    // OpMemoryBarrier, Acquire

    *blah = 2;                                              // OpStore, MakePointerAvailable

    drop(blah);                                             // OpMemoryBarrier, Release
}

The rust-gpu backend can't just universally emit this for all shader function arguments because you need to handle the different storage classes differently and somewhere the synchronization scope needs annotating.
Trying to turn this concept into something rusty is eluding me however - borrow/drop/access all need overriding somewhere, but the only place I can see to do that is by overriding the basic types or adding annoying syntax. I think/hope I'm just being a rust noob though :upside_down:
What I imagine would be quite nice is if you could just throw some attributes on the buffer struct that tell it that all accesses to its data have these semantics - but I don't know how realistic that is. Maybe this is something that would make more sense if I had a better sense of rustc internals. I'm hoping someone here has some good ideas :sweat_smile:
Jasper-BekkersToday at 00:31
Is this something we would need to apply to all buffer types?
I'm trying to understand the plan a bit
Tobias HectorToday at 00:32
Uniform/Constant buffers wouldn't have any of these sync semantics
Jasper-BekkersToday at 00:32
You're thinking about making safety guarantees for concurrent access to buffers
Tobias HectorToday at 00:32
yes
Jasper-BekkersToday at 00:32
Yeah let's assume uavs
Tobias HectorToday at 00:32
trying to make synchronization and lifetime management one and the same as much as possible
because that's... kind of what they are
Synchronization in GLSL/HLSL is enormously painful, and rust has some nice tools to do much better, so ideally want to try to use them
Jasper-BekkersToday at 00:33
If we add . store and .load functions, could that do it?
Tobias HectorToday at 00:33
you'd probably need to have explicit acquire/release too
I mean you could do it that way, but my gut says that seems like unnecessary programmer burden :woman_shrugging:
Jasper-BekkersToday at 00:34
Fair
Tobias HectorToday at 00:34
might work as an initial prototype, but I don't think it's tenable long term
if you can make this work for UAVs it would also work for LDS/workgroup memory too :grimacing:
Jasper-BekkersToday at 00:35
Right so this would, on deref return a RAII object, that can acquire when constructed and release when dropped?
Jasper-BekkersToday at 00:35
And access to the data would go through this RAII guard
Tobias HectorToday at 00:36
yea broadly speaking that's basically it:

Acquire on borrow
Make available on store
Make visible on load
Release on drop

One additional detail that got discussed after this little back and forth was that the aim here is to get borrow checking working across GPU threads - being able to know explicitly that indexing into the buffer is safe. For instance, if you're mutably borrowing from a buffer via a unique index (say indexing by the workgroup ID per workgroup), then a compile time check should be able to tell you your access is safe. For debugging, it should also be possible to automatically insert (somewhat expensive) checks that invocations aren't racing. There's probably also a place in there somewhere for something akin to rust's explicit lifetimes to allow more complex indexing to be compile-time checked.
From the SPIR-V side the aim is to make painless things that typically developers get frustrated over or make mistakes with - acquire/release/barriers are a pain in the prominent shading languages, and experience shows developers very much tend to get them wrong. The proposal discussed above takes that decision making away from the developer and hard codes what the most useful case as a default behavior - unlike GLSL/HLSL. We may want to provide hooks for less common patterns further down the line, but for now this behaviour should cover the needs of any developer that isn't trying to experiment with synchronization patterns...

Putting this on the correct issue this time....
Ok so turning this into something a bit more concrete, I think I'd propose three new attributes that affect struct definitions for shader inputs:
#[SubgroupVisible]
#[WorkgroupVisible]
#[DeviceVisible]

Each of these attributes modifies what happens during borrow and drop, and memory semantics used for OpLoad/OpStore when accessing references to these structs.
Note: If none of these attributes are present it should be a compiler error to borrow a mutable reference from the struct, (unless the code is declared unsafe?).
Scope
The word before "Visible" in the attribute determines the scope at which writes to the structure are made visible to other shader invocations.
So "SubgroupVisible" indicates that other invocations in the same subgroup can see values written by other invocations, assuming correct synchronization.
This scope is used as the %scope operand for the instructions detailed below.
Acquire on Borrow
When a reference is borrowed from the structure, an OpMemoryBarrier should be performed as follows:
OpMemoryBarrier %scope Acquire

Release on Drop
When a reference to the structure is dropped, an OpMemoryBarrier should be performed as follows:
OpMemoryBarrier %scope Release

Loads
When a reference to the structure is read from, an OpLoad should be generated with the following Memory Operands:
%value = OpLoad %pointer MakePointerVisible|NonPrivatePointer %scope

Stores
When a reference to the structure is written to, an OpStore should be generated with the following Memory Operands:
OpStore %pointer %value MakePointerAvailable|NonPrivatePointer %scope

A couple issues:

AFAIK, there is no way to tell in the backend when a borrow starts/stops, and so we cannot emit code at those points.
How do you intend to deal with OpAccessChain to a field on the struct? All attributes for the resulting pointer would be dropped, and we would no longer be able to emit the proper memory operands for load/store.

@khyperia Thanks for looking at this!

AFAIK, there is no way to tell in the backend when a borrow starts/stops, and so we cannot emit code at those points.

My understanding is that it's possible to override borrow/drop in the language itself, rather than in the backend, so I was imagining this be dealt with at least partially above the compiler; perhaps emitting some sort of intrinsic that the backend would turn into the right memory barriers. I was experimenting with this on the rust playground and made some decent progress in getting it to output something, but like a fool I hadn't saved it locally and lost it all when my browser tab crashed 🤦♀

How do you intend to deal with OpAccessChain to a field on the struct? All attributes for the resulting pointer would be dropped, and we would no longer be able to emit the proper memory operands for load/store.

Ok so again I'm not necessarily expecting this to be a wholly backend issue. My hope here is that we can coax the front-end into tagging the variables (and/or the variable accesses), such that when it gets to the backend all you'd need to do is check if a relevant tag is there or not. Unlike the first issue you pointed out, I don't have a handle on how this might be done with rust's attribute system yet - so this is kind of an open issue that I'm hoping we can have more discussion on.

So I've been thinking about the implementation complexity, and given the newness of this project and that it's probably useful to bootstrap this in the short term, maybe there should be two phases to this? The initial phase will have dedicated acquire/release and load/store ops, and then the more complex/ergonomic thing here can be a future investigation? Unless anyone thinks that's a terrible idea I'll write this into the draft RFC I'm working on.


Filter query tree serialization bugfix

Fix the serialization of the filter query tree. During filter query tree
serialization, we convert the tree into a map of ids to individual query
predicates. Since the id is generated automatically, if there is a
collision on the id on the leaf nodes, individual query predicates can
be overwritten, causing incorrect query processing.
This patch fixes this issue by removing the individually generated node
ids and generates them sequentially at serialization time, ensuring that
there is no possible collisions, assuming that there are fewer than
$2^31-1$ query filter predicates.

I tested this with a slightly modified version of the code that would generate the same id numbers for leaf nodes and confirmed that this bugfix does pass the integration tests even if there are id collisions (and fails if there are id collisions without the bugfix).

Codecov Report

Merging #1740 into master will decrease coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1740      +/-   ##
==========================================
- Coverage   68.05%   68.01%   -0.05%
==========================================
  Files         834      834
  Lines       40543    40539       -4
  Branches     5336     5336
==========================================
- Hits        27592    27571      -21
- Misses      11094    11104      +10
- Partials     1857     1864       +7

Impacted Files
Coverage Δ

...in/pinot/common/utils/request/FilterQueryTree.java
100% <100%> (ø)
:arrow_up:

...kedin/pinot/common/utils/request/RequestUtils.java
70.29% <100%> (+1.54%)
:arrow_up:

...e/impl/dictionary/LongOnHeapMutableDictionary.java
82.22% <0%> (-6.67%)
:arrow_down:

.../impl/dictionary/FloatOnHeapMutableDictionary.java
88.88% <0%> (-6.67%)
:arrow_down:

...not/core/query/pruner/DataSchemaSegmentPruner.java
53.65% <0%> (-4.88%)
:arrow_down:

...nkedin/pinot/common/metrics/ValidationMetrics.java
71.01% <0%> (-4.35%)
:arrow_down:

...r/filter/predicate/PredicateEvaluatorProvider.java
68% <0%> (-4%)
:arrow_down:

...filter/SortedInvertedIndexBasedFilterOperator.java
77.02% <0%> (-2.71%)
:arrow_down:

...m/linkedin/pinot/common/metrics/MetricsHelper.java
68.83% <0%> (-2.6%)
:arrow_down:

...impl/dictionary/DoubleOnHeapMutableDictionary.java
75.55% <0%> (-2.23%)
:arrow_down:

... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 75c10f4...0f16142. Read the comment docs.


Update docdown to fix vulnerability in lodash.

This PR updates docdown from 0.7.2 to 0.7.3, so that lodash is upgraded as well in order to fix a vulnerability in lodash <4.17.5:
https://nvd.nist.gov/vuln/detail/CVE-2018-3721
I also reran the doc generation, which apperently just removed some blank lines.

Hi @sepal!
Could you pull out the readme whitespace changes please?

yes of course. I reverted the changes to the readme.

Thank you!

Thank you!


Update repositories.txt

Hi @iwandwip it looks like the Kastara-Digital-Technology/Kadita repo has been deleted. Do you know why?


Feature document api changes

Description :sparkles:
Add service field to document api, allows filtering with service name.
Allow setting document user to anonymous document for API key post and patch requests.
Add Status history to documents and update status format on document GET
Issues :bug:
Content field on GET is returned as a string instead of JSON. Will be fixed next.
Closes :no_good_woman:
[ATV-88: ]
[ATV-103: ]
[ATV-104: ]
[ATV-105: ]
Related :handshake:
Testing :alembic:
Automated tests :gear:
Yes
Manual testing :construction_worker_man:
Screenshots :camera_flash:
Additional notes :spiral_notepad:

Codecov Report

Merging #50 (2baa958) into main (dca73da) will increase coverage by 0.41%.
The diff coverage is 100.00%.

:exclamation: Current head 2baa958 differs from pull request most recent head a9e6788. Consider uploading reports for the commit a9e6788 to get more accurate results

@@            Coverage Diff             @@
##             main      #50      +/-   ##
==========================================
+ Coverage   90.00%   90.42%   +0.41%
==========================================
  Files          51       52       +1
  Lines        1201     1243      +42
  Branches      114      119       +5
==========================================
+ Hits         1081     1124      +43
+ Misses         85       84       -1
  Partials       35       35

Impacted Files
Coverage Δ

documents/api/docs.py
100.00% <ø> (ø)

documents/api/filtersets.py
100.00% <100.00%> (ø)

documents/api/querysets.py
80.00% <100.00%> (ø)

documents/api/viewsets.py
94.93% <100.00%> (+0.37%)
:arrow_up:

documents/models.py
100.00% <100.00%> (ø)

documents/serializers/document.py
98.30% <100.00%> (+0.68%)
:arrow_up:

documents/serializers/status_history.py
100.00% <100.00%> (ø)

audit_log/viewsets.py
88.88% <0.00%> (+1.38%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dca73da...a9e6788. Read the comment docs.


Audio Player: Back button functionality

Should the back button jump to the beginning of the current track, if it's in the middle of playing?

Yes, I think so. We could do something where its like, if youre within 3 seconds of the beginning of the audio file, it goes back to the previous clip's beginning, and if its after 3 seconds it jumps to the beginning of the SAME audio file currently playing.


Larger dimensionality sparse array inputs cause many FLAML use cases to fail

Classification and regression models often leverage sparse arrays. For example when composing a Sklearn pipeline, sparse arrays are formed as the output of categorical OneHot transforms and ngramming like CountVectorizer and TfidfVectorizer.
These sparse arrays are passed into the learners (eg LightGbm, Xgboost, LogisticRegression), and these learners are designed to support these sparse arrays efficiently allowing large dimensionality classifiers to be trained.
The fix I think it to ensure we are keeping arrays in their sparse array formats, eg scipy.sparse.csr_matrix at least when passing into the underlying learner and preferably through the handling of the data by FLAML as well.
Here are a few repro examples that illustrate the issue, note that generally any example using OneHot, CountVectorizer, or TfidfVectorizer will repro similarly and these transforms are common:
Example 1 - LightGbm regression example repro. The FLAML will execution part looks to densify the sparse matrix output from the ColumnTransformer (takes up 5GB RAM), and gets stuck failing to fit the learner. A couple notes:

This is just 700K distinct categorical features, but the same problem is present for smaller numbers like ~70K distinct features too. LightGbm can handle this.
Without FLAML it takes 3 seconds to fit.
With FLAML it hangs.
Python takes up 5GB RAM as soon as it tries to start the FLAML part.

import numpy as np
import pandas as pd

from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.metrics import accuracy_score

from flaml import AutoML

from lightgbm import LGBMRegressor

# Create an example dataframe with random integers and a target column
df = pd.DataFrame(np.random.randint(0, 1000000, size=(500000, 2)), columns=['A', 'B'])
df['target'] = ((df['A'] + df['B']) > 1000000).astype(int)
print(df.head(10))

# Print dataset statistics
print("Dataset statistics:")
print(f"Number of rows: {len(df)}")
print(f"Number of unique categorical features: {df['A'].nunique() + df['B'].nunique()}")

# Build a sklearn pipeline that one-hot encodes the integers

# Create the pipeline with a LightGBM regression model
pipeline = Pipeline([
    ('preprocessor', ColumnTransformer([
        ('onehot', OneHotEncoder(handle_unknown='ignore'), ['A', 'B'])
    ])),
    ('classifier', LGBMRegressor())
])

# Fit the model while timing the process
import time
start = time.time()
pipeline.fit(df[['A', 'B']], df['target'])
end = time.time()

# Print the metrics
print("LightGBM")
print("MAE:", np.mean(np.abs(pipeline.predict(df[['A', 'B']]) - df['target'])))
print("Number of parameters:", pipeline['classifier'].n_features_)
print("Fitting took {} seconds".format(end - start))

# Now fit a FLAML model instead with LightGBM and the same data
pipeline = Pipeline(
    [
        (
            'preprocessor', ColumnTransformer(
                [
                    ('onehot', OneHotEncoder(handle_unknown='ignore'), ['A', 'B'])
                ]
            )
        ),
        (
            'classifier', AutoML(
                task='regression',
                estimator_list=['lgbm'],
            )
        )
    ]
)
start = time.time()
pipeline.fit(df[['A', 'B']], df['target'])
end = time.time()

# Print the metrics
print("FLAML")
print("MAE:", np.mean(np.abs(pipeline.predict(df[['A', 'B']]) - df['target'])))
print("Fitting took {} seconds".format(end - start))

Output:
        A       B  target
0  392214  547064       0
1  733104   70527       0
2  435315  879075       1
3    3221  115858       0
4  507700  113589       0
5  746200  312285       1
6  671553  164952       0
7  486703  540223       1
8  540288  793887       1
9  201758  563042       0
Dataset statistics:
Number of rows: 500000
Number of unique categorical features: 786525
LightGBM
MAE: 0.499999640448
Number of parameters: 786525
Fitting took 3.1742513179779053 seconds
[flaml.automl: 11-04 21:18:01] {2600} INFO - task = regression
[flaml.automl: 11-04 21:18:01] {2602} INFO - Data split method: uniform
[flaml.automl: 11-04 21:18:01] {2605} INFO - Evaluation method: holdout
[flaml.automl: 11-04 21:18:01] {2727} INFO - Minimizing error metric: 1-r2
[flaml.automl: 11-04 21:18:01] {2777} WARNING - No search budget is provided via time_budget or max_iter. Training only one model per estimator. To tune hyperparameters for each estimator, please provide budget either via time_budget or max_iter.
[flaml.automl: 11-04 21:18:01] {2869} INFO - List of ML learners in AutoML Run: ['lgbm']

Example 2 - FLAML binary classification without learner specified (note: binary classification limited to lgbm works)

Without FLAML it takes 3.2 seconds to fit
With FLAML it hangs, but does NOT take up a large amount of RAM right away. Maybe an iteration issue rather than a dense conversion issue at least at first?

import numpy as np
import pandas as pd

from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.metrics import accuracy_score

from flaml import AutoML

from lightgbm import LGBMRegressor, LGBMClassifier

# Create an example dataframe with random integers and a target column
df = pd.DataFrame(np.random.randint(0, 1000000, size=(500000, 2)), columns=['A', 'B'])
df['target'] = ((df['A'] + df['B']) > 1000000).astype(int)
print(df.head(10))

# Print dataset statistics
print("Dataset statistics:")
print(f"Number of rows: {len(df)}")
print(f"Number of unique categorical features: {df['A'].nunique() + df['B'].nunique()}")

# Build a sklearn pipeline that one-hot encodes the integers

# Create the pipeline with a LightGBM regression model
pipeline = Pipeline([
    ('preprocessor', ColumnTransformer([
        ('onehot', OneHotEncoder(handle_unknown='ignore'), ['A', 'B'])
    ])),
    ('classifier', LGBMClassifier())
])

# Fit the model while timing the process
import time
start = time.time()
pipeline.fit(df[['A', 'B']], df['target'])
end = time.time()

# Print the metrics
print("LightGBM")
print("MAE:", np.mean(np.abs(pipeline.predict(df[['A', 'B']]) - df['target'])))
print("Number of parameters:", pipeline['classifier'].n_features_)
print("Fitting took {} seconds".format(end - start))

# Now fit a FLAML model instead with LightGBM and the same data
pipeline = Pipeline(
    [
        (
            'preprocessor', ColumnTransformer(
                [
                    ('onehot', OneHotEncoder(handle_unknown='ignore'), ['A', 'B'])
                ]
            )
        ),
        (
            'classifier', AutoML(
                task='classification',
            )
        )
    ]
)
start = time.time()
pipeline.fit(df[['A', 'B']], df['target'])
end = time.time()

# Print the metrics
print("FLAML")
print("MAE:", np.mean(np.abs(pipeline.predict(df[['A', 'B']]) - df['target'])))
print("Fitting took {} seconds".format(end - start))

Output:
    A       B  target
0  435201   20896       0
1  650113  830984       1
2  296903  335422       0
3  733399  881420       1
4  968169  600231       1
5  243170   94188       0
6  303474  476584       0
7  950125  645745       1
8  607060  432579       1
9  741146  550150       1
Dataset statistics:
Number of rows: 500000
Number of unique categorical features: 787371
LightGBM
MAE: 0.499826
Number of parameters: 787371
Fitting took 3.257169246673584 seconds
[flaml.automl: 11-04 21:20:45] {2600} INFO - task = classification
[flaml.automl: 11-04 21:20:45] {2602} INFO - Data split method: stratified
[flaml.automl: 11-04 21:20:45] {2605} INFO - Evaluation method: holdout
[flaml.automl: 11-04 21:20:46] {2727} INFO - Minimizing error metric: 1-roc_auc
[flaml.automl: 11-04 21:20:46] {2777} WARNING - No search budget is provided via time_budget or max_iter. Training only one model per estimator. To tune hyperparameters for each estimator, please provide budget either via time_budget or max_iter.
[flaml.automl: 11-04 21:20:46] {2869} INFO - List of ML learners in AutoML Run: ['extra_tree', 'lgbm', 'rf', 'xgboost', 'xgb_limitdepth', 'lrl1']
[flaml.automl: 11-04 21:20:46] {3164} INFO - iteration 0, current learner extra_tree

Hi, I tried to look into this and found that FLAML actually preserved the sparsity of the data. I did not finish debugging this but I found that FLAML uses LGBM with different parameters, which may be the cause of this.
For your first example, the LGBMRegressor will create boosters with parameters.
boosting_type=gbdt colsample_bytree=1.0 learning_rate=0.1 max_depth=-1 min_child_samples=20 min_child_weight=0.001 min_split_gain=0.0 n_jobs=-1 num_leaves=31 reg_alpha=0.0 reg_lambda=0.0 subsample=1.0 subsample_for_bin=200000 subsample_freq=0 verbose=-1 objective=regression metric=regression num_iterations=100
While FLAML will create boosters with parameters.
boosting_type=gbdt colsample_bytree=0.7019911744574896 learning_rate=0.022635758411078528 max_depth=-1 min_child_samples=2 min_child_weight=0.001 min_split_gain=0.0 n_jobs=-1 num_leaves=122 reg_alpha=0.004252223402511765 reg_lambda=0.11288241427227624 subsample=1.0 subsample_for_bin=200000 subsample_freq=0 max_bin=511 verbose=-1 objective=regression metric=regression num_iterations=4797
Notice that FLAML allows for many more iterations and leaves, which (I think) is due to the specification of the search space. I am not quite sure why does FLAML takes such a long time, but I don't think this is due to the sparsity of the data.

Hi @glmcdona ,
I think I've found a fix to this problem. So for the first task, what you can do to significantly reduce the training time is to set a max_iter constraint and specify the metric to 'mae'.
Here's my code and output, for which FLAML achieves roughly the same accuracy in a much shorter time.
automl = AutoML( task='regression', estimator_list=['lgbm'], max_iter = 10 ) automl.fit(X_train, y_train, metric="mae")
Outputs: Dataset statistics: Number of rows: 500000 Number of unique categorical features: 786695 Data type after one-hot encoding: <class 'scipy.sparse.csr.csr_matrix'> LightGBM MAE: 0.49999983175033913 Fitting took 1.551173210144043 seconds
FLAML best estimator: LGBMRegressor(learning_rate=0.09999999999999995, max_bin=255, n_estimators=4, num_leaves=4, reg_alpha=0.0009765625, reg_lambda=1.0, verbose=-1) MAE: 0.49999983175033913 Fitting took 78.22783994674683 seconds

If you check the console output or the log, how long does the first trial take in total?


Version 1.17.70 => Freescout 1.8.133

Summary
I am using the docker "latest" tag and need a bugfix in Freescout version 1.8.139, but in the "latest" or "php8.3-1.17.70" tag contains the version 1.8.133.
Steps to reproduce
Nothing special, just pulled the "latest" or "php8.3-1.17.70" tagged image.
What is the expected correct behavior?
Should contain Freescout version 1.8.139
Relevant logs and/or screenshots

Environment

Image version / tag:  "latest" or "php8.3-1.17.70"
Host OS: Ubuntu 23.10

docker-compose.yaml

services:
  freescout-app:
    image: tiredofit/freescout:latest
    container_name: freescout-app
....

Same problem over here!

Sorry about this, looks like I messed up the releases the other night. Latest or 1.17.71 now contains 1.8.139.

Yes got it running at 1.8.139 now! Thanks for this super fast fix 👍

Now everything is as it should be. Thank you for that very quick fix!


Update README.md

Update the README.md file for basic instructions.

Yay!! This is exciting!
Curious question, what is 99-user.ldif?

It is an LDIF file used to extend the LDAP schema in DS. It lets you store custom attributes in your LDAP directory tree.


Update config.pp

typo

Well spotted, thanks!
Please consider signing the Contributor Licence Agreement (see "Checks" above) so we can get this merged in.
Cheers.


Improve Footer

Issue Template

Issue type

Other

Description
Footer is not at all matching full webapp, so making it more appealling according to UI.
Screenshots or videos

Additional information
Participating in swoc24

allow me to work on it

Hello @AaadityaG ,
This issue already exist in issue no. #6 . Kindly go through other issues or create a new one.
Thank You


LionWebModelConverter issues

It's stateful and uses maps so it can't be reliably shared across threads;
However, it doesn't allow to provide its languageConverter from the outside so if you don't share the model converter you have to re-export the used languages.

Either the maps should be made thread-safe, or it should be possible to pass the language converter as a constructor parameter.

It's stateful and uses maps so it can't be reliably shared across threads;

Yes, that is the case because otherwise references would cause the creation of different copies of the nodes (either in the LionWeb->Kolasu or the Kolasu->LionWeb scenario)

However, it doesn't allow to provide its languageConverter from the outside so if you don't share the model converter you have to re-export the used languages.

Right, perhaps we can change that

Either the maps should be made thread-safe, or it should be possible to pass the language converter as a constructor parameter.

The latter seems quite easy. Regarding maps thread-safe I guess it would make sense to do so. I do not have much experience with that but I can study the problem


Citations with bibtex and jekyll-scholar

https://drscotthawley.github.io/devblog4/2020/07/01/Citations-Via-Bibtex.html

done


add GeoreferencedBehavior for behaviors common to georeferenced/geore…

inserts an georeferenced behavior since the BasicGeoMetadata is too low a level to require a CRS

This makes sense. Was trying to sketch these out quickly. What do you think about the RDF URIs for these? As this is somewhat uncharted territory, we're going to have to have make it up as we go for the most part (with outside input, of course).

maybe http://projecthydra-labs/pcdm-geo-models#xxx -- I don't think the URI has to actually be a working URL

I suppose http://projecthydra-labs/pcdm-geo-models#xxx looks cleaner. However, "http://www.georss.org/georss/box" seems pretty clear and is the suggested RDF URI according to georss.org. crsURI is pretty well defined by OGC. Envelope is a bit tricky. It was hard to pin down exactly where that syntax (used by SOLR) is defined. As far as  I can tell, it came from the CSW spec. We may not actually need :solr_geom, so that may not be an issue.

Yeah, anything that has a URI already we should use -- the hydra URI is only for new things we're defining for this model

Looks good to me. I'll merge unless you want to add anything else.

nope, i'm ok with merging both


Model type hierarchy doesn't scale

We could be doing much better if instead of a type hierarchy we had a set of trait functions for isprobabilistic issupervised like I do in GeoStatsBase.jl Currently we have Unsupervised models that are Probabilistic like GMMClusterer and yet we can't place them in the type hierarchy.
Any plans to refactor this types into a set of trait functions in the long-term?

Reference to trait-based approach: https://github.com/JuliaEarth/GeoStatsBase.jl/blob/d5a4bdc0ee676b1077a4fbff929cfc6e272e8088/src/learning/traits.jl#L18-L32

@juliohm Thanks for these observations.

Any plans to refactor this types into a set of trait functions in the long-term?

Not really, as this would be a huge refactor.
However, there is already a trait (shared by models and metrics) called prediction_type which can have value :deterministic, :probabilistic, :interval or :unknown.  (Do info(model) to get a list of all the traits of a model.)
Naturally, for Probabilistic, Deterministic and Interval subtypes of Supervised, there is the obvious fallback.  In principle the existence of this trait means we could do away with the with this subtyping, but this has not been considered because of the disruption.

See also https://github.com/JuliaAI/MLJModelInterface.jl/issues/10 for a previous discussion.


Enable Task Environment Variables Section

Can the task level environment variables section be enabled?

I use Terraform to deploy infrastructure to the Azure China. There's ARM_ENVIRONMENT variable which can help. Despite the fact I can use -environment=china it does't work with current setup and gives me an error.

@vladkasianenko a workaround would be to use secure env files to set ARM_ENVIRONMENT.

Would be great if this task could read TF_VAR_* environment variables from the build agent, without having to use the env input.

It already does.  For example, if you have a variable group with variables of that format (TF_VAR_*) and reference it at the start of the job, all the terraform CLI tasks will see these variables.

@AdamCoulterOz @favoretti @vladkasianenko is this still an issue?  We are passing TF_VAR_* vars to terraform CLI directly from variable groups without issue

@jason-johnson not for me personally. Out of curiosity, do you have an example of passing the vars from a vargroup? I'm pondering moving away from secret files to a KV-backed vargroups, but if I can steal an example somewhere - would be nice :)

I don't have any publicly available examples available but the usage is straight forward.  First, I make a variable group and put variables in it, e.g. TF_VAR_COLOR.  Next, in the top level terraform I will define a variable color.  Now in the job which will call terraformCli  I specify
variables:
  - group: my_var_group

With this in place, when terraform plan runs it will pull the value of TF_VAR_COLOR into the color variable automatically.  This works because referencing the variable group in the job puts all the variables from the variable group into the environment of all tasks that run in that job.
NOTE:  This will only work for non-secret variables.  All variables get injected into the environment but secret variables will not propagate further.  To make use of secret variables, my convention was to simply use the regular name (e.g. COLOR) and have the a task where I specify which secrets to pull in and have the template generate env variables which inject these manually.  A template which does this can be seen in the attached template here.


Feature/test elan

Commit Message Guidelines
Header
Blank Line
Body
Blank Line
Footer

The header should look like:
<type>(<scope>): <subject>

The body should have any necessary detailed info about the commit:
An example, references as to where this idea came from, etc.

The footer should have all the issues tagged:
Fixes #123, Fixes #456

So a commit should look like:
feat(users): Add new Yahoo authentication

Yahoo authentication idea proposed by @codydaig
Example implementation in file.js

Fixes #82

Types:

feat - Features, Enhancements, and overall Improvements
fix - Fixes, Bugs, HotFixs, etc...
doc - Changes to the Documentation and doesn't actually touch any code.

Scope:

The scope should be where the change took place.
Examples: users, core, config, articles

Subject:

The subject line should be clear and concise as to what is being accomplished in the commit.

General Rules:

No Line in the Commit message can be longer than 80 characters.

Reference: Angular Conventions

Hey @erasabi , please make your own repository to learn pull requesting. Now you're just causing extra work for volunteers here. Thanks.


#569: use getWidgetBuilder(context); from CoreRenderer

#569: in PrimeFaces 6.3-SNAPSHOT the RequestScope is replaced by the PrimeRequestScope,
to reduce usage of RequestScope replace
RequestContext.getCurrentInstance().getWidgetBuilder();
by
getWidgetBuilder(context);
in Renderer

Thanks!


test/rubocops: Fix mismatched test descriptions

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[x] Have you successfully run brew tests with your changes locally?
[x] Have you successfully run brew man locally and committed any changes?

After doing #10658, I moved onto other TODOs in that file. Enabling RSpec/RepeatedDescription momentarily revealed that two test descriptions had been copypasted between two similar, but not identical, tests. (Actually fixing the other offenses for that cop will be for another day!)

Review period will end on 2021-02-23 at 00:00:00 UTC.

👏🏻

Review period ended.