Improve default value reporting in --help

Description
As a user, I want --help to show default values for all options where it makes sense, especially the boolean ones, so that I know exactly what is enabled and what isn't.
For example --eth1-enabled defaults to true, whereas --rest-api-docs-enabled seems to default to false. But there is no way to know this.
(Related: although the docs for --rest-api-docs-enabled in the code seem to give true for the fallbackValue, the default appears to be false.)
Acceptance Criteria

running teku --help shows the correct default settings for all relevant options.

Expected behavior: [What you expect to happen]
      --eth1-enabled[=<BOOLEAN>]
                             Whether to connect to the ETH1 chain
                             Default: true

Actual behavior: [What actually happens]
      --eth1-enabled[=<BOOLEAN>]
                             Whether to connect to the ETH1 chain

Versions (Add all that apply)

Software version: [teku --version]

teku/v0.11.0-dev-e3ad800d/linux-x86_64/-privatebuild-openjdk64bitservervm-java-13

Default values are also missing for --p2p-private-key-file and --validators-graffiti, and maybe others.

--p2p-private-key-file

I wonder what should be displayed here as the default value. If not specified then the key is read from the KV-Store (i.e. <beacon-data-dir>/kvstore/generated-node-key.dat). However that file cannot be specified as a value to this argument.

Ah yes, it would be really nice if --p2p-private-key-file could read either the current text based hex format or the binary format used for generated-node-key.dat.
I think the default however is none and it probably just needs a note stating that one will be generated and stored in the beacon node data directory if not specified.  I believe if you specify a value for this we expect it to be present which seems like a useful feature (if you intended to use a specific node key, then you wouldn't want us to silently generate a new one if you happened to get the path wrong).

Hi,
I would like to take this up if no one else is currently working on it.
Initial thoughts:
The reason why the default values for boolean options are currently not being displayed appears to be because Picocli ignores the Command.showDefaultValues flag for boolean options. The rationale for this is mentioned here and is based on the assumption that boolean options would default to false, which isn’t the case in this instance.
There are probably two ways to get around this 1) specifically set Option.showDefaultValue = Visibility.ALWAYS for each non-hidden boolean option specifically 2) Interpolate the default value of an option into its description (${DEFAULT-VALUE} variable ). The first approach is perhaps slightly more consistent with the current use of Command.showDefaultValues, so I might use that unless you have other thoughts.
If I am not mistaken, the second point raised here seems to be that some non-boolean options don’t have specific default values that can be specified, even though there is some mechanism being employed to obtain their value if one isn't explicitly provided. For such options, I assume the consensus is to update the descriptions and note this behavior?

I would like to take this up if no one else is currently working on it.

That would be fantastic.

There are probably two ways to get around this 1) specifically set Option.showDefaultValue = Visibility.ALWAYS for each non-hidden boolean option specifically 2) Interpolate the default value of an option into its description (${DEFAULT-VALUE} variable ). The first approach is perhaps slightly more consistent with the current use of Command.showDefaultValues, so I might use that unless you have other thoughts.

I didn't realise setting Visibility.ALWAYS worked for booleans (I'm not deeply familiar with PicoCLI).  If it does that sounds like a great solution so the formatting of showing the defaults is consistent.  We already habitually add a couple of flags particularly for boolean fields so that would be easy to add as well.  Actually now I think about it we probably should write a custom error-prone check to enforce those options are always set for boolean options, but that's a separate thing. :)

If I am not mistaken, the second point raised here seems to be that some non-boolean options don’t have specific default values that can be specified, even though there is some mechanism being employed to obtain their value if one isn't explicitly provided. For such options, I assume the consensus is to update the descriptions and note this behavior?

Yes, there's a few options where the default is platform dependent or depends on other values.  Just ensuring the description explains how the default is calculated is the right approach for those.

Thanks, @ajsutton. I will work on making the changes for the first part, then come back to confirm the details of the second.

I've submitted PR #4272  with the changes for both parts. The changes are mostly minor and address the following:

fixed the issue where boolean command options were not displaying in the usage help message
fixed an issue where a couple of subcommands weren't displaying default values for any option in their usage help messages
updated the descriptions of options whose defaults can't automatically be inferred by Picocli. This is based on what I could infer from the code and the docs. Let me know if I missed anything.
other minor improvements to the descriptions of commands and options

On a related note, it looks like the CLI reference docs are a bit out of sync with the code. I am not sure if this is already being addressed, but if not, I can raise a separate issue in the docs repo for this. I've listed below some of the minor discrepancies I could see on a quick pass.
Teku command - teku
— --log-file-name-pattern documentation should perhaps show the default pattern used <data-path>/logs/teku_%d{yyyy-MM-dd}.log
— —metrics-categories missing DISCOVERY category (i.e. TekuMetricCategory.DISCOVERY)
— —validators-external-signer-timeout defaults to 5000 in code, but docs state 1000
—  --p2p-private-key-file  documentation should describe what happens when this value is not provided.
Slashing-protection subcommand - teku slashing-protection
— Documentation for ImportCommand and ExportCommand should  mention —data-base-path as an alias to —data-path
— data-validator-path option not present in the documentation
Validator-client subcommand - teku validator-client
— —log-include-validator-duties-enabled default to true in code but the docs report the default as false Validator client subcommand options
—--log-file-name-pattern documentation should show the default pattern used <data-path>/logs/teku_%d{yyyy-MM-dd}.log
— —metrics-categories missing DISCOVERY category
— —metrics-interface defaults to “<IP_ADDRESS>” in code, but docs note “<IP_ADDRESS>”
— --validators-early-attestations-enabled option not present in the documentation
— --validators-external-signer-keystore and --validators-external-signer-keystore-password-file options not present in the documentation
— —validators-external-signer-timeout defaults to 5000 in code, but docs state 1000
—--validators-external-signer-truststore and —--validators-external-signer-truststore-password-file options not present in the documentation
— --validators-graffiti-file option not present in the documentation
Voluntary-exit subcommand -- teku voluntary-exit
— --validators-external-signer-keystore and --validators-external-signer-keystore-password-file options not present in the documentation
—--validators-external-signer-truststore and —--validators-external-signer-truststore-password-file options not present in the documentation
— —validators-external-signer-timeout defaults to 5000 in code, but docs state 1000


Widgets won't render until map is panned

Context
In this map https://team.carto.com/u/noguerol/builder/99349ec4-1f6c-11e7-a204-0e3ff518bd15/embed
widgets are not rendered until the map is panned in the viewport
Browser and version
Chrome
.carto file
Stores and routes in NY (on 2017-04-12 at 11.46.39).carto.zip

dupped in https://github.com/CartoDB/cartodb.js/issues/1610 , so closing


Implement name lookup for more loop headers

This implements name (and type) lookup for more loop headers, i.e. for-in and while let.

Awesome! Thanks @birkenfeld


Résolution automatique des conflits du vault

:tada: This PR is included in version 1.39.0 :tada:
The release is available on:

v1.39.0
GitHub release

Your semantic-release bot :package::rocket:


Server OOM killed while starting up, if too much data > 15 GB

Prometheus version: v2.15.2
Google Stable Chart Version: 10.1.0
I have a Prometheus server running, on it's own node in GCP, with the following specs:
8 CPU, 32 GB MEM and 100 GB SSD.
When Prometheus get's above 15 GB in data, then when the node restarts, Prometheus get's OOM Killed, and i can see that it's using all the memory on the node. The memory usage is increasing, as the WAL segments is loaded. If the amount of data is below 15 GB, then it just has enough memory to start, before it get's OOM Killed. Right now we can only store one to one and a half day of data, so it's really not that much.
This is how the log looks like when it starts up, and it just stops right after the last WAL segment is loaded, or sometimes before of course it get's through them all.

Logs:

[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.308Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)"
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.308Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)"
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.308Z caller=main.go:332 host_details="(Linux 4.15.0-1044-gke #46-Ubuntu SMP Fri Sep 13 15:02:00 UTC 2019 x86_64 prometheus-server-6977c6c64d-btcsp (none))"
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.308Z caller=main.go:333 fd_limits="(soft=1048576, hard=1048576)"
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.308Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)"
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.312Z caller=main.go:648 msg="Starting TSDB ..."
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.312Z caller=web.go:506 component=web msg="Start listening for connections" address=<IP_ADDRESS>:9090
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.312Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1579012558178 maxt=1579017600000 ulid=01DYJWA94QC4E518X3THW8J5ZD
[prometheus-server-6977c6c64d-btcsp prometheus-server-configmap-reload] 2020/01/15 07:56:42 Watching directory: "/etc/config"
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.313Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1579017600000 maxt=1579032000000 ulid=01DYK144VZC9E79P1H2HTZPE5C
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.313Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1579032000000 maxt=1579046400000 ulid=01DYKEVKG4STX0T679AMXWZJC7
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:56:43.313Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1579046400000 maxt=1579060800000 ulid=01DYKWK21CNZDW9XAEXEM298ZR
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:59:18.547Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile"
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:59:18.547Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=118 maxSegment=119
[prometheus-server-6977c6c64d-btcsp prometheus-server] level=info ts=2020-01-15T07:59:18.547Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=119 maxSegment=119

I have faced this problem too, How can we prevent it? is there any flag or something?
because after loading wal files, it frees memory. which means just for starting it needs this huge memory.


Feature Request - Preference for Install Drive Location

Many of us have limited space on the C: drive and would much prefer to install to D: or some other drive location and specific folder (and there are other reasons for choosing a different location).
Since Pinokio is geared toward super easy installation for non-experts this would be a wonderful feature to have in Pinokio.  This way one could choose Preferences prior to starting Pinokio install, and or prior to package installs and choose the desired location.
Thanks

same here.and I can't change the installation folder,don't know why

Despite the claim that you can move the entire Home folder somewhere else, it does not seem to work at all. This is in contrast with Stability matrix (Multi Package home) which does allow you to move the folder in which you store everything.

This.


[Snyk] Security upgrade @babel/plugin-transform-parameters from 0.0.0-use.local to 7.0.0

Snyk has created this PR to fix 1 vulnerabilities in the yarn dependencies of this project.
Snyk changed the following file(s):

packages/babel-plugin-proposal-object-rest-spread/package.json

Note for zero-installs users
If you are using the Yarn feature zero-installs that was introduced in Yarn V2, note that this PR does not update the .yarn/cache/ directory meaning this code cannot be pulled and immediately developed on as one would expect for a zero-install project - you will need to run yarn to update the contents of the ./yarn/cache directory.
If you are not using zero-install you can ignore this as your flow should likely be unchanged.

⚠ Warning
Failed to update the yarn.lock, please update manually before merging.

Vulnerabilities that will be fixed with an upgrade:

Issue
Score

Directory Traversal SNYK-JS-SOURCEMAPSUPPORT-6112477
  170

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Directory Traversal

:tada: Snyk checks have passed. No issues have been found so far.
:white_check_mark: security/snyk check is complete. No issues have been found. (View Details)
:white_check_mark: license/snyk check is complete. No issues have been found. (View Details)


fix(vcs): Smaller array size in pagination hack

This PR fixes problem with allocation of large size Array during pagination.
Fixes: https://github.com/superfaceai/station/issues/266
Description

Motivation and Context

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have read the CONTRIBUTING document.
[x] I haven't repeated the code. (DRY)
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Geocoding tests are failing 🤔 VCS tests pass correctly(there isn't any auth).

Okey, so some are just flaky :/


use-releases does not handle tens properly

I don't know if I have a particular setup on my side but if I do
$ mvn versions:use-releases -U -DallowRangeMatching=true -DprocessParent=true -DfailIfNotReplaced=true

I am getting a far bigger version than what I expect:
[INFO] Scanning for projects...
<...>
[INFO] --- versions-maven-plugin:2.14.2:use-releases (default-cli) @ commons-updater-repository ---
<...>
[INFO] Updated com.dentalwings.commons:versioninfo:jar:1.6-SNAPSHOT to version 1.69-76

I don't have the same result if I go through a property instead:
$ mvn versions:update-properties -U -Dincludes=com.dentalwings*:*:*:*:*-SNAPSHOT -DallowMajorUpdates=false -DallowMinorUpdates=false -DallowIncrementalUpdates=false

Here I get the result I would have expected from the use-release goal:
[INFO] Updated ${commons.version} from 1.6-SNAPSHOT to 1.6-93

You compare use-releases with update-properties using different option sets. The options that you selected have different semantics, hence the results differ.
allowRangeMatching=true will select the highest found version where the version string starts with 1.6. This includes 1.69-76...
On the other hand, for update-properties, you selected -DallowMajorUpdates=false -DallowMinorUpdates=false -DallowIncrementalUpdates=false which will restrict the plugin not to update even the minor nor the incremental segment of the version.
Because the minor segment happens to be 6, and not 69, it will not select the 1.69-76.
So, I don't think this is a bug per se. I'd say that the meaning of allowRangeMatching might be a bit unfortunate, but it simply does what the documentation says -- matches the start of the version string.

Ok, but how can I restrict use-release to behave the same as update-properties then (finding the latest release with the same major.minor) ?
I'm also having difficulties understanding why those 2 goals don't share the same flags. Their mission is essentially the same, no ?

Not really.
The thing is that those targets appeared evolutionally, and they don't necessarily share common options. The puprpose of use-releases is not finding the latest release, but simply stripping the -SNAPSHOT suffix from the version, with an additional option of using prefix search.
If you want to mimic the options you used for update-properties, you should use https://www.mojohaus.org/versions/versions-maven-plugin/use-latest-releases-mojo.html

Thanks ! I tested use-latest-releases and it was exactly what I needed.
If I understand properly your last comment, if I have the version 1.6-SNAPSHOT, use-releases will return:

the highest version that starts with 1.6

whereas on my side I understood that it would return:

the highest version that starts with 1.6.

If I may add, the documentation about allowRangeMatching could be clarified to better represent that behavior (maybe adding 1.11 to the list of versions returned ?)

If I understand properly your last comment, if I have the version 1.6-SNAPSHOT, use-releases will return:
the highest version that starts with 1.6

Only if you use the -DallowRangeMatching=true.
Otherwise it will attempt to simply strip the -SNAPSHOT suffix.

If I may add, the documentation about allowRangeMatching could be clarified to better represent that behavior (maybe adding 1.11 to the list of versions returned ?)

Good point.

Since it was a misunderstanding from my part on the behaviour of allowRangeMatching, I will simply close that issue.
Thanks again for the quick and excellent feedback !


Clarify data management process

Summary
We need to clarify what happens with the "research" data

Will it be able to be downloadable
How will the permission process work?
What is the extent of dynamic consent which is possible - how can we explain this clearly?

Superseeded by #666


Use debug binaries when when debugging.

Also adds /proc/self/cwd as the location for the sourceFileMap when debugging in linux.

Just pinging to see if anyone is monitoring this repo. Looks promising so far. Would love to use it, and I think this PR would make it a lot more useful for some.


Report data fixes

Some fixes to the report data covering all seasons. Current data was a cumulative view but now it shows real numbers for each season

deployed

thanks :+1:


[Vue] Add support for lazy-loading with Async Components

Q
A

Bug fix?
no

New feature?
yes

Tickets
Fix #...

License
MIT

I was a bit surprised when trying the Symfony UX' Vue integration, to see all Vue components/controllers to be loaded to render at least only one controllers.
This PR aims to reduce the bundle size and only load the required code and Vue controllers to render on the page.
To enable lazy-loading, add the 4th parameter "lazy" to require.context():
- registerVueControllerComponents(require.context('./vue/controllers', true, /\.vue$/));
+ registerVueControllerComponents(require.context('./vue/controllers', true, /\.vue$/, "lazy"));

Note: the sync-loading still works
Real life example
Given 6 really simple Vue controllers which all look the same:

If I render only two of them on a single page:

Before

You can see the app.js is about 4.8 kB, it contains all of the 6 Vue controllers.
After

You can see the app.js is now about 3.4 kB, and two more files have been lazy-loaded (our two Vue controllers rendered on the page).
On a real project, with a lot of Vue controllers, the difference would be very significative. I will try to post some results when migrating our main project at work.

Note: ~I still have to add tests and update the documentation.~ done

Thanks for the review @weaverryan!
On this PR, I think nothing is missing, however I thought about a usecase for a 2nd PR: I think some people would like to have both lazy-loaded and not lazy-loaded controllers/components at the same time.
E.g.: common components like a NavBar or Aside that would be always shown on every pages, and specific components like a date picker that would be sometimes shown in some context.
Currently, that's not possible to have both registerVueControllerComponents calls like this:
registerVueControllerComponents(require.context('./vue/controllers', true, /\.vue$/));
registerVueControllerComponents(require.context('./vue/controllers-lazy', true, /\.vue$/, 'lazy'));

because vueControllers and loadComponent, which are used by the global function resolveVueComponent(), will be erased.

Thanks Hugo!


Fix typo in couch_mrview comment

Overview
Fixes a very small typo in a comment in couch_mrview,erl
"uknown languages" to "unknown languages"
Testing recommendations
N/A
Related Issues or Pull Requests
N/A
Checklist

[ ] Code is written and works correctly
[ ] Changes are covered by tests
[ ] Any new configurable parameters are documented in rel/overlay/etc/default.ini
[ ] A PR for documentation changes has been made in https://github.com/apache/couchdb-documentation

+1 thanks for your contribution. I re-triggered test of travis to make it happy and then merge it to master.


How to set up multiple input

SetInputPath only one input can be set,how to set up multiple input?

You can use SetOutputFormat("concat") and give to ffmpeg text file as input
Text file example (tested on windows):

file 'C:\somefolder\somefile1.mp4'
file 'C:\somefolder\somefile2.mp4'
file 'C:\somefolder\somefile3.mp4'

But for concat you need execute ffmpeg with "-safe 0" option and this library does not implement it yet. So you need to add a PR.
Hope it helps you


[Docs] Test Configurations examples have wrong paths

Examples in Test Configurations page show that you should set CAQE_CONFIG='../test_configurations/general_mushra.cfg' as a path, but CAQE builds this path in __init__.py, so the examples should be CAQE_CONFIG='general_mushra.cfg' instead.

....make sure you run create_db.py after changing the config.

Oh, no this isn't what I was talking about. The examples on this page: https://interactiveaudiolab.github.io/CAQE/test_configurations.html are wrong. For example, for general MUSHRA that page says:
$ export CAQE_CONFIG='../test_configurations/general_mushra.cfg'
but line 24 in caqe/init.py does this:
app.config.from_pyfile('../test_configurations/' + os.getenv('CAQE_CONFIG', 'general_mushra.cfg'))


Increment nonce after encrypt / decrypt

Counterpart to https://github.com/stratum-mining/stratum/pull/729

Also added a check for the maximum nonce value of 2^^64 - 1 as chapter 5.1 of the Noise paper requires, though that's not in the spec. @jakubtrnka thoughts?


[RFE] Group or sort Apps category by virtual IP address

Is your feature request related to a problem? Please describe.
The Apps list does not automatically sort by IP address ascending or descending, so it is not always clear how many ports and virtual servers are being configured for a specific virtual IP. If we make use of the AS3 "Application" class the 80/443/whatever port of an application can be grouped together in one AS3 Application class to allow more commonality in profile and pool definitions - this would be made easier if virtual servers of the same IP address were listed together or in order.
Describe the solution you'd like
Either sort the Apps output by ascending or descending IP address, or allow each App category to be port independent to group all ports of an application together.
Describe alternatives you've considered
Currently I end up re-declaring an application if I have to add more virtual servers to it, which is more complex.
Additional context
N/A

apps list is now sorted by IP descending to put all the <IP_ADDRESS> (and no matches) at the bottom - pending v1.1 release


Instance error with contemporary GHC

$ ghci
GHCi, version 8.10.7: https://www.haskell.org/ghc/  :? for help
Prelude> :load Queens.hs
[1 of 3] Compiling Domain           ( Domain.hs, interpreted )

Domain.hs:54:10: error:
    • No instance for (Semigroup Domain)
        arising from the superclasses of an instance declaration
    • In the instance declaration for ‘Monoid Domain’
   |
54 | instance Monoid Domain where
   |          ^^^^^^^^^^^^^
Failed, no modules loaded.
Prelude>

You can to stack ghci to build using the correct GHC and library versions. But it is quite old. I'll see if I can update it to something more recent.

PR #3 updates to Stack LTS 18.12 and GHC 8.10.7.


Integrate Boost.Redis in the Boost build system

The time has come. Changes from the boost review have been implemented, users were gained, tests and new features look stable etc. It is time to integrate with Boost and reach more users with official Boost releases.

Any plans on this please?

Yes, this is top priority. Someone (@anarthal) offered some help so we might be able to get into 1.84. I have been sick for some weeks and could not dedicate much time here.


Inbox - Hmm… page opens briefly when click on Account settings and then on Inbox

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: v9.0.8-1
Reproducible in staging?: Y
Reproducible in production?: N
Email or phone of affected tester (no customers<EMAIL_ADDRESS>Logs: https://stackoverflow.com/c/expensify/questions/4856
Issue reported by: Applause-Internal team
Action Performed:

Go to https://staging.new.expensify.com/
Log in
Click on Account settings
Click on Inbox

Expected Result:
Chat open without Hmm... page
Actual Result:
Hmm… page opens briefly in the chat
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/user-attachments/assets/ded5d1d6-736a-4c5a-b800-f7e06f752f70

View all open jobs on GitHub

We think this issue might be related to the  #vip-vsb

I can repro, but I wouldn't imagine this is a deploy blocker right @madmax330 ?

Agree, doesn't really look like a blocker

Also, looks related to https://github.com/Expensify/App/issues/45565

Waiting on @madmax330

commented on the issue Jack linked, if it's not the same we probably just need to make this external

Going to close this one out, as Fedi could not replicate anymore.


preserve original synchronous compile function as compileSync

I have tens of tests for handlebars helpers and having compileSync function makes it easier to test them. Other than that it just seems like a nice idea to preserve the original synchronous function.

Coverage remained the same at 100.0% when pulling b19dc6dbbcaf8722ad3c0bfee6e04affbe9d1c7b on Gottwik:master into b53ec5848a7cec29158d8682ea2e35eeabc22d1b on nknapp:master.

Coverage remained the same at 100.0% when pulling b19dc6dbbcaf8722ad3c0bfee6e04affbe9d1c7b on Gottwik:master into b53ec5848a7cec29158d8682ea2e35eeabc22d1b on nknapp:master.

Are those helpers synchronous or asynchronous? I am still wondering about the exact use case.
If your helpers are synchronous, why not just use Handlebars directly? If your helpers are asynchronous, a compileSync-function probably won't help you a lot, because it will just return the placeholder for the promise result. It does not feel like the method makes much sense, but maybe I have missed something.
And since I just thought if it's worth to discuss this for a single line of code: There is a little more work to be done. The final PR should include:

A usage example and a little bit of written documentation about the use case (here and here)
A test case that tests the documented usage.
The line-comment in front of the compileSync should be a JSDoc-comment so that it can be used to generate documentation.

@Gottwik I'm closing this PR for now. If you are still interested, please add a comment.


Homeserver event stream

Goals
To decouple Indexers from Homeservers, and for general subscriptions/notifications usecases, we need an event stream interface that notify any subscriper that a resource has been updated.
Subscriptions need to a bit more dynamic than what we had in the web-relay (where you can only subscripe to exacte file), but it needs to be simple enough that the Homeserver doesn't need to do too much work to match data events to their subscriptions.
Users should be able to:

Subscribe to all events from a certain user.
Subscribe to all events in a named folder (or prefix of a path) from a certain user.
Subscribe to all events in a named folder (or prefix of a path) from all users.
Subscribe to all public events from everyone.

Design
Request
The API for subscriptions is suggested to be a concatenation of <user pubky><prefix>:

User pubky; can be either Exact or Wildcard
Prefix; where any entry that shares that prefix is matched.

Wildcard example:
*/pubky.app/

Exact example:
3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/pubky.app/

Servers then store subscriptions in a key-value store, separtated to two logical tables (wildcards and exact ids), for example:
"*/foo/bar" => HashSet<Subscription>,
"3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/foo/bar" => HashSet<Subscription>,

Once a new operation is done (PUT/DELETE), the server iterates over all wildcards and exact tables, until the prefixes no longer match the current event's path.
This way, the server only iterates over subscriptions that match the new event, instead of linearily checking all active subscriptions. More importantly these checks are lexographic so easy to reason about and compute.
Event
An event is encoded in a Server-Sent Events as follows:
id: <event-id>
data: <event type> <user-id>/<path> [<future data>]

event-id: (u64 encoded as base32) is the sequence of the event internally to the server, which allows subscripers to resume after a disconnection from the last Seq they saw if the server didn't discard these old events.
event type: PUT or DEL
user-id: Pubky
path: for example /pubky.app/potsts/XXXX
future data: once we implement a copy-on-write Merkle Treap, we can also share the Seq of the treap as a version of the file, and maybe the hash of the root or the hash of the value of the resource!

Example
Subscribe to all events from a certain user.
Request
GET /stream?q=3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/  HTTP/2
Last-Event-Id: 7

Response
id: 8
data: PUT 3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/foo

id: 10
data: PUT 3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/bar

Subscribe to all events in a named folder (or prefix of a path) from a certain user.
Request
GET /stream?q=3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/fo  HTTP/2
Last-Event-Id: 7

Response
id: 8
data: PUT 3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/foo

Subscribe to all events in a named folder (or prefix of a path) from all users.
Request
GET /stream?q=*/fo  HTTP/2
Last-Event-Id: 7

Response
id: 8
data: PUT 3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/foo

id: 11
data: PUT wfhww7zb54mirr9gi1ricxrfck7po3oyftkf7yc154znnri8u888/foo

Subscribe to all public events from everyone.
Request
GET /stream?q=*/  HTTP/2
Last-Event-Id: 7

Response
id: 8
data: PUT 3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/foo

id: 10
data: PUT 3oyftkf7yc154znnri8u888wfhww7zb54mirr9gi1ricxrfck7po/bar

id: 11
data: PUT wfhww7zb54mirr9gi1ricxrfck7po3oyftkf7yc154znnri8u888/foo

id: 12
data: PUT wfhww7zb54mirr9gi1ricxrfck7po3oyftkf7yc154znnri8u888/bar

Implemented a fire hose subscription in #54 ... no plans to add granular subscriptions yet.


Early publishing segments in IndexTask

As discussed here, the current implementation of IndexTask has a potential out of disk problem, which is caused by keeping all segments in a local disk until publishing them at the very end of IndexTask.
The solution what I'm considering is publishing segments early when the size of generated segments exceeds a predefined threshold in the middle of segment generation. Thus, the behavior of FiniteAppenderatorDriver.add() will be changed to

Add a row to appenderator.
Check the size of segments generated so far.
If it exceeds the predefined threshold,
3.1) Persist all data indexed through FiniteAppenderatorDriver so far.
3.2) Publish all data indexed through FiniteAppenderatorDriver so far, and waits for segment hand-off. This is because data stored in a local disk are removed after hand-off.
Repeat from 1 to 3 until all data are read from the firehose.

Also, this proposal requires to use LinearShardSpec instead of HashBasedNumberedShardSpec when an interval needs to be partitioned. Currently, shardSpecs must be determined before starting ingestion. However, with this proposal, it is very difficult to determine required shardSpecs in advance. Instead, LinearShardSpec can be used for incremental sharding, i.e., creating a new shard whenever a segment is published.
This proposal changes the rollup mode of IndexTask from guaranteed rollup to non-guaranteed rollup when the rollup is enabled because two rows that should roll up might be in different publish cycles. I think it's ok for now because non-guaranteed rollup does not affect to the query result accuracy even though it can increase the space usage. However, we need to devise another solution to support guaranteed rollup in the near future.
I think this will also work well with KafkaIndexTask.

For (3.2) there's no need to wait for handoff (meaning historical nodes loading the data), we just need to wait for publishing.
Regular NumberedShardSpecs should be fine. They support "extending" beyond a core set, by setting the partitionNum to something higher than the corePartitions.
This writeup makes me realize that there's a couple of things we lose by publishing early:

Guaranteed rollup
Atomic publishing (linear shard specs, and numbered ones with partitions outside the core set, are published piecemeal rather than as an atomic unit)

The lack of atomic publishing would mean that replacing existing data won't be done atomically. So I guess while this is a useful option, we would really want an option for loading large datasets "all at once" too so we can get guaranteed rollup and atomic publishing. Probably through parallelization. It could be a later patch though.

@gianm thanks for your comment.

For (3.2) there's no need to wait for handoff (meaning historical nodes loading the data), we just need to wait for publishing.

Actually, IndexTask uses NoopSegmentHandoffNotifierFactory which does nothing, so I think it would be fine to wait for handoff. However, your another comment sounds good and it would be better if we can skip unnecessary call of NoopSegmentHandoffNotifierFactory.

The lack of atomic publishing would mean that replacing existing data will be done non-atomically, which for some users is not acceptable. So I guess while early publishing is a useful option for other users, we would really want an option for loading large datasets "all at once" too so we can get guaranteed rollup and atomic publishing. Probably through parallelization. It could be a later patch though.

Do you mean that it would be ok if I add a way for guaranteed rollup in a follow-up patch?

Actually, IndexTask uses NoopSegmentHandoffNotifierFactory which does nothing, so I think it would be fine to wait for handoff. However, your another comment sounds good and it would be better if we can skip unnecessary call of NoopSegmentHandoffNotifierFactory.

FYI we're having a discussion about that design on a thread in https://github.com/druid-io/druid/pull/4178#issuecomment-299977548 too; I think that PR and #4238 have a lot of similarity in goals. I'm hoping we can find a design that will be simple and effective for everyone…

Do you mean that it would be ok if I add a way for guaranteed rollup in a follow-up patch?

Yeah, that's what I mean. Assuming guaranteeing rollup also means doing atomic publishing (it probably could; they seem related to me).

Right, they would be related each other.
Ok. I'll first read #4178 and finish #4238, and then raise another pr for guaranteed rollup.


[WIP] - Experiment with full removal of cancel token and multiprocessing.

What was wrong?
How was it fixed?
To-Do

[ ] Clean up commit history

[ ] Add entry to the release notes

Cute Animal Picture

Just a brain dump of what I've learned here.
We can replace the cross-process logging mechanism which is actually based on the multiprocessing library with a custom implementation that uses roughly the same pattern as our DBManager.  This is necessary because you cannot pass a multiprocessing.Queue across a process boundary unless you are using the multiprocessing utils.
With the cross process logging tools replaced we can then use asyncio-run-in-process.  This opens the door for an improved Component API.
The Component API can be cleaned up significantly.  This can be seen here.  The component APIs end up being slimmed down to just the following two APIs:
class BaseComponentAPI(ABC):
    def configure_parser(cls, arg_parser: ArgumentParser, subparser: _SubParsersAction) -> None:
        ...

    def validate_cli(cls, boot_info: TrinityBootInfo) -> None:
        ...

class ApplicationComponentAPI(BaseComponentAPI):
    name: str

    def __init__(self, trinity_boot_info: TrinityBootInfo, endpoint: EndpointAPI) -> None:
        ...

    @property
    def is_enabled(self) -> bool:
        ...

    async def run(self) -> None:
        ...

The base API is all that is needed for components like fix-unclean-shutdown.  The ApplicationComponentAPI is what is needed for things like the transaction pool or other components that run concurrently.
The simplification to the component APIs also extends to the ComponentManager class which is easier modeled as just a plain Service.  This allows for easy concurrent spin-up of components, however this might expose a deficiency in our component APIs as some components are dependent on others and we probably need something to allow declaring dependencies between them so that they can be started in the correct order.
The new Service model doesn't use a CancelToken.  Assuming that this works, this allows removal of a lot of machinery like the trinity._utils.shutdown and things like kill_process_gracefully.  The main logic for the whole application can then be coded as just another Service and the startup logic for the whole application becomes as simple as running the service and installing some signal handlers at the outermost layer to catch SIGTERM or KeyboardInterrupt and trigger shutdowns in a clean manner.

Alright, after another day and a decent bit of hacking at the async-run-in-process to get it behaving correctly, I believe I've validated my hypothesis.
I can run the Trinity client and get clean shutdowns without any cancel tokens stuff.
There's still a lot of cleanup to be done but I'm leaning towards this being a viable path forward.
One cool thing that came from this is that we are now completely free of multiprocessing and any related magic.  All of the issues I ran into while debugging this branch were now in parts of code that I was in control of, meaning that I could actually fix them so that they shutdown cleanly rather than having to jump through hoops to try and fix them with indirection.  This included a decent bit of work in the asyncio-run-in-process to get it handling child processes correctly.
I feel pretty good about this.  I think it gives us a level of control over the application that we previously didn't have, specifically in managing the lifecycles of things which is important if we want users to have a nice UX.


Makefile: If go-bindata is available, warn if bindata.go is out of date

[ ]   General Comment
 > Should we add .PHONY: check-bindata to make sure this target is always out of date and needs to be run?
Good point, will update

This pull request is now managed with CodeReviewHub. Refresh.

Should we add .PHONY: check-bindata to make sure this target is always out of date and needs to be run?

Good point, will update

Should we add .PHONY: check-bindata to make sure this target is always out of date and needs to be run?

Does that means gobindata regeneration will run for every build?

Since the target depends on it, we test for potential out of date bindata every time. Doesn't mean you need to rebuild the daemon each time. From my side, the go bindata generation is done rather quick, so I think it makes sense, otherwise why having the check then.


安装最新版本报错呢

pip install -U MiService==2.1.2
Collecting MiService==2.1.2
Using cached miservice-2.1.2.tar.gz (12 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [22 lines of output]
Traceback (most recent call last):
File "/Users/yuanxiao/.pyenv/versions/py3.11_virtualenv_test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/Users/yuanxiao/.pyenv/versions/py3.11_virtualenv_test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/yuanxiao/.pyenv/versions/py3.11_virtualenv_test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/yz/wr4mf5654m306_wpj23bx7vm0000gn/T/pip-build-env-q62g21_v/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/yz/wr4mf5654m306_wpj23bx7vm0000gn/T/pip-build-env-q62g21_v/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
self.run_setup()
File "/private/var/folders/yz/wr4mf5654m306_wpj23bx7vm0000gn/T/pip-build-env-q62g21_v/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 497, in run_setup
super().run_setup(setup_script=setup_script)
File "/private/var/folders/yz/wr4mf5654m306_wpj23bx7vm0000gn/T/pip-build-env-q62g21_v/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 313, in run_setup
exec(code, locals())
File "", line 17, in
File "/private/var/folders/yz/wr4mf5654m306_wpj23bx7vm0000gn/T/pip-install-w73i_04y/miservice_8561a24430bf4f04b3ca223dff320819/micli.py", line 2, in
from aiohttp import ClientSession
ModuleNotFoundError: No module named 'aiohttp'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

是呢，我也是报同样的错。     from aiohttp import ClientSession
ModuleNotFoundError: No module named 'aiohttp'

aiohttp 这个模块已经安装了 ，还是报错 ModuleNotFoundError: No module named 'aiohttp'

我也出这个问题，很奇怪，突然就出现了...

我更新了一下wheel，然后发现yarl报错，又把yarl删掉重新装了一下，然后发现miservice安装没错误了。。。


Launch AVD before getting devices

In Appium 1.5, getDeviceInfoFromCaps() fails when avd capability is configured and no device is connected.
I have moved prepareEmulator call from initDevice to getDeviceInfoFromCaps to ensure that the emulator is up before trying to access it.
This PR and https://github.com/appium/appium-adb/pull/145 fix the issue https://github.com/appium/appium/issues/6233

appium/appium-adb#145 published in 2.5.1.
Thanks for this. Looks good.


setuptools doesn't replace shebang from build directory when installing

Pip version: 8.1.1
Python version: 2.6
Operating System: CentOS 6

Description:
I have a small python module with some scripts that I run in a virtual environment. My Makefile looks like this:
default: build

.DEFAULT:
    ./setup.py $@

build:
    ./setup.py build

/path/to/venv/bin/activate:
    virtualenv --system-site-packages /path/to/venv

install: /path/to/venv/bin/activate
    /path/to/venv/bin/pip install .

Now if I have a fresh install and run make install everything works great, and at the top of all of my scripts in $VENV/bin I have the following shebang:
#!/path/to/venv/bin/python

but if I first run make build and then make install then I get the system python in the shebang:
#!/usr/bin/python

I guess I would have expected the setup process to check and replace the shebang to the virtual environment's python.

make build makes no reference to the virtualenv, so it's probably making a build of the project using the system python environment, probably creating the scripts at this step.
Then by invoking make install, you're invoking pip on the current directory. My guess is that the setup.py build is causing the scripts to be generated and shebangs set then. Still, it's not obvious to me why build artifacts in the build directory would affect a pip install.
I did try to replicate your findings, but when I used Python 3 and venv, the issue did not occur.
I think I'm going to need you to dig in a bit more and put together a description that I can replicate. Can you create a Github repository that experiences this issue?

The setup is very simple. Here is an example package:
~/foo/foo.py
#!/usr/bin/env python

print("Hello world!")

~/foo/setup.py
#!/usr/bin/env python
from setuptools import setup

setup(name='foo', version='1.0', scripts=['foo.py'])

Then, to reproduce the issue, create a virtual environment (but don't activate it):
$ virtualenv ~/myvenv

And then run:
$ cd foo
$ python setup.py build
$ ~/myvenv/bin/pip install .

I think you are right that the build step uses the system python environment. Not sure if this is really a bug or not, but it caught me off guard.

And if you skip the last python setup.py build step, the script gets built with the expected shebang?

Yep.

I think we had the same issue. A developer had run python setup.py build and took us a while to understand what was going on.
Maybe an alternative here would be if we could define somewhere in setup.py that it should always use a fresh build directory.

I ran into this with Python 3.7, venv, and pip 19.1.1., while installing a local package into a virtual environment.  Steps to reproduce:

create a virtual env: python3.7 -m venv cat
activate cat and run python setup.py build

this creates a ./build directory; its script shebangs pointing to the cat environment

deactivate cat, then create another called dog
activate dog and run pip install .

this installs the local package into dog, but its internal shebangs point to cat
reason is, pip copied in the contents of the old build directory


Update Email Service Dependencies

Updates Ruby and Dependencies.
There is currently an issue with this where it builds, but does not run. Please do not merge.

I've converted to draft to avoid being merged


[3/N] Refactor scheduler for chunked prefill scheduling

Refactor the current scheduler to make it easy to understand with chunked prefill later.
This simply moves logic for prefill scheduling and decoding scheudling to a dedicated function. The purpose of doing this is we want the different scheduling policy for chunked prefill (by default, we do prefill -> decoding. But when chunked prefill is enabled, we want decoding -> prefill to reduce ITL impact).
The functionality must be exactly the same except that I made it use lora_enabled instead of directly checking if lora config is None.
Related: https://github.com/vllm-project/vllm/issues/3130
BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

 PR Checklist (Click to Expand)
Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

[Bugfix] for bug fixes.
[CI/Build] for build or continuous integration improvements.
[Doc] for documentation fixes and improvements.
[Model] for adding a new model or improving an existing model. Model name should appear in the title.
[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
[Kernel] for changes affecting CUDA kernels or other compute kernels.
[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
[Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:

We adhere to Google Python style guide and Google C++ style guide.
Pass all linter checks. Please use format.sh to format your code.
The code need to be well-documented to ensure future contributors can easily understand the code.
Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes
Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.
What to Expect for the Reviews
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

 After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
 After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
 After the review, the reviewer will put an  action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
 Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You
 Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

Thanks for the quality review! I will finish addressing comments by eod today :)!

Update: Swapping is currently only for decoding, so I think decoupling should not be a scope of this PR. But I can address this better in the next PR

Here's the e2e PR for scheduler on top of this branch; https://github.com/rkooo567/vllm/pull/15

Addressed comments relevant to this PR https://github.com/rkooo567/vllm/pull/15#issuecomment-2021809284. cc @simon-mo to take another look!

Status: will separate out swapping and add a lot more unit tests.

Throughput: 2.01 requests/s, 972.94 tokens/s
(base) ray@ip-10-0-58-45:~/default/vllm/benchmarks$ Throughput: 1.99 requests/s, 961.40 tokens/s
Benchmark result

@simon-mo Updated (plz take a look one more time);

swap is a separate API now
each API is tested. Also better unit testing for swapping.
general cleanup (e.g., use dataclass, use better naming).

@zhuohan123 @simon-mo
We discussed offline for direction. I updated code based on the proposal I made.

Made all _schedule APIs as stateless as possible.
Each APIs are order agonistic.
Increase test coverage A LOT compared to before including beam search case, block updates, swap & preemption, individual APIs, loras, max_seqs and max_batched_tokens.
Fix 2 bugs.

the first prefill doesn't include running batched tokens when it counts num_batched_tokens.
swapping doesn't break when it reaches to max_num_batched_tokens.

sampler test failure seems unrelated

lora test failure unrelated


Help/Question: How to make the following use-case database agnostic.

I have the following table:
  // names of the attributes are changed.
  // But I need a column to store a list of JSON.
  return knex.schema
  .createTable("table_name", table => {
    table.string("id", 40).notNullable();
    table.string("name", 40).notNullable();
    table.primary(["id", "name"]);
    table.specificType("some_list", "jsonb[]");
  })
};

I have written a query that does the following:

If the primary key is not there in the table then create a row, but if it is already there then append the respective list entry at the end of the some_list array.

The condition to append is that the some_list array length should be equal to a received index,  called lastReceivedIndex in the code below.

The query is the following:
try {
            await OperationMaster.transaction(async (trx) => {
                const updatedRows = await trx.raw(
                    `UPDATE operation_details
                    SET some_list= array_append(list_item, ?)
                    WHERE meeting_id = ? AND part_spec = ? AND array_length(some_list, 1) = ? RETURNING CARDINALITY(some_list)`,
                    [list_item, id, ops.name, lastReceivedIndex]
                );

                if(updatedRows.rowCount === 0){
                    const existingEntry = await OperationMaster.query(trx)
                    .where('id', id)
                    .andWhere('name', ops.name)
                    .first()
                    .forUpdate();

                    if(!existingEntry){
                        await OperationMaster.query(trx)
                        .insert({
                            'id' : id,
                            'name' : ops.m_partSpec,
                            'some_list' : [list_item],
                        })
                        .forUpdate();

                        indexWhereOperationIsAdded = 1; // I want the index where the item was appended.
                        operationSuccess = true;
                    }
                }
                else{
                    indexWhereOperationIsAdded = updatedRows.rows[0]?.cardinality; // It is known that there will be only one row that will be updated here
                    operationSuccess = true;
                }
            });
        }
        catch (error) {
            console.log('Error inserting or appending:', error, newOperationJSON);
        }
        return [indexWhereOperationIsAdded, operationSuccess];

I am using many array functions that are PostgreSQL specific, but I want a Knex-based query for it. Please suggest any better way to design a schema for this use case.
I am new to this.

Duplicate question. https://github.com/knex/documentation/issues/535


remove underscore from _actorAttributes

Fixes issue with token mold config not opening on D&D system version > 3

Closed as duplicate.


Refined Github doesn't work on Chrome/Edge until page is refreshed (reloaded)

Steps
I go to PR, click on Files tab, and Refined is not enabled.
I refresh the page (doesn't matter how, F5, hitting enter again in address bar, using mouse gesture) then Refined is enabled.
By enabled I mean I can see whitespace button and other changes which are being brought by the plugin.

What I've tried?

I've disabled every single plugin on my Chrome
I've reinstalled plugin
Different browsers (Edge, Chrome) the same issue.

It started to happen about a month ago, I expected a fix to come out, but it's still not working.

Duplicate of #4695, please read the pinned issues carefully next time.
@fregante maybe the title for that issue is too technical? Replace "ajax" with "page" would be better.

It's ok, duplicate issues are fine.

Duplicate of #4695, please read the pinned issues carefully next time.
@fregante maybe the title for that issue is too technical? Replace "ajax" with "page" would be better.

I would have never worked out that both of these issues are one and the same :)
I'm glad that this was already registered and obviously with more information what's not working.
Have a good day! :)


askama workspace build problem

When trying to build e.g. askama_escape (https://github.com/djc/askama/blob/main/askama_escape/Cargo.toml) with cargo2nix, I get Cargo.toml: No such file or directory. I believe it happens because Cargo looks for the workspace root Cargo.toml which has been moved/renamed during the buildPhase.
Only when patching out https://github.com/djc/askama/blob/main/askama_escape/Cargo.toml#L10 (workspace = "..") things build correctly.
Forgive my lack of knowledge on workspace builds. Am I doing something wrong or are the askama authors using the workspace directive wrong? Is cargo2nix lacking something here?

  I'll make sure that the TOML used in the build phase doesn't try to look up to a workspace that doesn't exist.

Documenting how to adjust the retained cargo manifest tables / values for future readers:
https://github.com/cargo2nix/cargo2nix/blob/release-0.11.0/overlay/mkcrate.nix#L240-L250
This jq query needs to filter out attributes that interfere with the builder's behavioral intent to build each crate isolation.
The workspace attribute is one such cargo manifest value.
To fix issues like this, add attributes that break the crate, and then configure a query that will successfully remove the offending attribute.   In this case, I pointed the cargo workspace to nowhere and then verified that nix build succeeds by removing the erroneous manifest value.
https://github.com/cargo2nix/cargo2nix/pull/292/commits/e95e9601257602bbec8c05a18edf114855f0efc5
Fix will be pushed to unstable after CI passes.  Closing.


Card payment method should be 'scheme'?

Describe the bug
The credit-card payment method did not show up for me. It seems like that is because the response from /paymentMethods gives the cards with the 'scheme' type payment method.
To Reproduce
Steps to reproduce the behavior:

Add credit-cards in your Adyen customer center
Open hyca checkout
No credit-card payment method

Expected behavior
Have a credit-card payment method
Magento version
2.4.7-p3
Plugin version
adyen/module-hyva-checkout 1.2.1
adyen/module-payment 9.9.1
adyen/php-api-library 19.1.0
Desktop (please complete the following information):

OS: Ubuntu
Browser chrome
Version latest

Additional context
This was with the test-environment of Adyen.
Pull-request
I created a pull-request fixing the issue for me.

Hello @JustinElst,
Thank you for creating this issue and providing a potential solution with your contribution.
However, we are trying to understand the root cause of the issue on your end as we can't reproduce this behaviour and indeed successfully see card payment component on Hyvä checkout. I've already left some comments on the PR. I believe those changes shouldn't affect your implementation unless you have a customisation on Adyen_Payment module.

For further diagnostic of the issue on your end, I would recommend checking the output of this return value. Basically, it returns an array with payment method adapter interfaces. It should have a value for AdyenCc payment method instance.
Best Regards,
Can

Hi thanks for your reply. So I tried to debug this line. And I got this result:

The first block is with my patch of AdyenConfigured the second one is with the master code.
I checked what the response was for the /paymentMethods api:

And that clearly shows the "type": "scheme" for the card methods.
Could you explain why the Adyen_Payment module has this map, but this module does not? Because i am just 'mapping' the 'scheme' type payment method to 'adyen_cc' just as in Adyen_Payment. I don't know what the original 'card' type meant, but i can't reproduce getting that method type from the  /paymentMethods api.


Python extension can't process escape in the string like r"""C:\windows""" or r"C:\windows" correctly

Environment data

VS Code version: 1.48.1
Extension version: 2020.8.103604

Unexpected behaviour

and also it will wrongly process the end char "\", make the end quote char be escaped

We don't control syntax highlighting for Python files. That comes from VS Code which gets it from MagicPython. It's best to report an issue there.


Virt

Type of change

[ ] Refactor
[x] New feature
[ ] Bug fix
[ ] Optimization
[ ] Documentation Update

Description
Adding support for Virtual Machines Instances (VMI).
This integration will create a svc + route to communicate with the Virtual Machine.
The workflow is :

Prepare VMI instance with cloud-init, inject the machine running k8s-netperf sshkey into the guest
Deploy the Client VMI
Deploy the Server VMI (cloud-init will start the workload driver servers, ie, netserver)
Create svc and route for the Client VMI
Attempt to ssh to the Client VMI using ssh-key
Check to see if cloud-init has completed (it installs packages at startup)
Attempt to launch the workload -- In some cases the server might not be done installing packages, so we have a retry.
Run workload (from here on, the workflow is the same as podNet)

Future - We could look to generate our own VM Image with our packages pre-loaded, so the install timing mess can be avoided.

Related Tickets & Documents

Related Issue #
Closes #

Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.

Testing

Please describe the System Under Test.
Please provide detailed steps to perform tests related to this code change.
How were the fix/results from this change verified? Please provide relevant screenshots or results.

We may need to test different VM network "binding" types in the future, i.e. in spec.devices
            interfaces:
            - masquerade: {}
              model: virtio
              name: default

masquerade could eventually be "bridge" and/or "passt"

Why don't you import kubevirt as dependency rather than including it in the repo?

It is not recommended to import it as a dependency. I attempted this at first but ran into significant issues. When I reached out to the CNV / Kubevirt community - they shared with me how they implemented this in their own csi-driver https://github.com/kubevirt/csi-driver/blob/main/pkg/kubevirt/client.go#L26
Others have ran into this problem too - https://groups.google.com/g/kubevirt-dev/c/vuuvW2E6Kow
Recommended way is described here - https://github.com/kubevirt/api/?tab=readme-ov-file#how-to-use-it

I have to try this out personally. But looks like there is a way to ssh into a VM without any ssh keys.
If our cluster has VLAN support and have a VLAN created like this
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: vlan-network
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge",
    "ipam": {
      "type": "host-local",
      "subnet": "<IP_ADDRESS>/24",
      "rangeStart": "<IP_ADDRESS>",
      "rangeEnd": "<IP_ADDRESS>",
      "routes": [{"dst": "<IP_ADDRESS>/0"}],
      "gateway": "<IP_ADDRESS>"
    }
  }'

We can create VM with the below configuration and ssh into it.
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: my-vm
spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
          - name: default
            bridge: {}
          - name: vlan-network
            bridge: {}
      networks:
      - name: default
        pod: {}
      - name: vlan-network
        multus:
          networkName: vlan-network

I have to try this out personally. But looks like there is a way to ssh into a VM without any ssh keys.
If our cluster has VLAN support and have a VLAN created like this
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: vlan-network
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge",
    "ipam": {
      "type": "host-local",
      "subnet": "<IP_ADDRESS>/24",
      "rangeStart": "<IP_ADDRESS>",
      "rangeEnd": "<IP_ADDRESS>",
      "routes": [{"dst": "<IP_ADDRESS>/0"}],
      "gateway": "<IP_ADDRESS>"
    }
  }'

We can create VM with the below configuration and ssh into it.
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: my-vm
spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
          - name: default
            bridge: {}
          - name: vlan-network
            bridge: {}
      networks:
      - name: default
        pod: {}
      - name: vlan-network
        multus:
          networkName: vlan-network

This is significantly more complex and not an option. The assumption of vlan creation isn't always true, where we can always inject a SSH key.

Yeah on the bridge network type, that would allow direct L2 connectivity to the VM from the bastion, however VMs typically use linux bridge not macvlan, which requires an nncp on a secondary interface (i.e. can't be shared w/ the OVN port), and that changes which network type we measure against.

We may need to test different VM network "binding" types in the future, i.e. in spec.devices
            interfaces:
            - masquerade: {}
              model: virtio
              name: default

masquerade could eventually be "bridge" and/or "passt"

Trying to future proof things here -- will this be the name?  -- Model seems to map to bridge 😕
type Interface struct {
    // Logical name of the interface as well as a reference to the associated networks.
    // Must match the Name of a Network.
    Name string `json:"name"`
    // Interface model.
    // One of: e1000, e1000e, ne2k_pci, pcnet, rtl8139, virtio.
    // Defaults to virtio.
    // TODO:(ihar) switch to enums once opengen-api supports them. See: https://github.com/kubernetes/kube-openapi/issues/51
    Model string `json:"model,omitempty"`
    // BindingMethod specifies the method which will be used to connect the interface to the guest.
    // Defaults to Bridge.
    InterfaceBindingMethod `json:",inline"`
    // Binding specifies the binding plugin that will be used to connect the interface to the guest.
    // It provides an alternative to InterfaceBindingMethod.
    // version: 1alphav1
    Binding *PluginBinding `json:"binding,omitempty"`
    // List of ports to be forwarded to the virtual machine.
    Ports []Port `json:"ports,omitempty"`
    // Interface MAC address. For example: de:ad:00:00:be:af or DE-AD-00-00-BE-AF.
    MacAddress string `json:"macAddress,omitempty"`
    // BootOrder is an integer value > 0, used to determine ordering of boot devices.
    // Lower values take precedence.
    // Each interface or disk that has a boot order must have a unique value.
    // Interfaces without a boot order are not tried.
    // +optional
    BootOrder *uint `json:"bootOrder,omitempty"`
    // If specified, the virtual network interface will be placed on the guests pci address with the specified PCI address. For example: 0000:81:01.10
    // +optional
    PciAddress string `json:"pciAddress,omitempty"`
    // If specified the network interface will pass additional DHCP options to the VMI
    // +optional
    DHCPOptions *DHCPOptions `json:"dhcpOptions,omitempty"`
    // If specified, the virtual network interface address and its tag will be provided to the guest via config drive
    // +optional
    Tag string `json:"tag,omitempty"`
    // If specified, the ACPI index is used to provide network interface device naming, that is stable across changes
    // in PCI addresses assigned to the device.
    // This value is required to be unique across all devices and be between 1 and (16*1024-1).
    // +optional
    ACPIIndex int `json:"acpiIndex,omitempty"`
    // State represents the requested operational state of the interface.
    // The (only) value supported is `absent`, expressing a request to remove the interface.
    // +optional
    State InterfaceState `json:"state,omitempty"`
}

func (*v1.Interface).DeepCopy() *v1.Interface
func (*v1.Interface).DeepCopyInto(out *v1.Interface)
func (v1.Interface).SwaggerDoc() map[string]string```

@jtaleric  I think its InterfaceBindingMethod

Need to add virt to the results until we get this captured in go-commons.