feat(streaming): adapt backfill rate limit according to barrier latency

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
wip.

[ ] Show grafana monitoring results.
[ ] Add more scenarios: Join with Agg after, UDF high latency.
[ ] Include more description on the alternative solutions.
[ ] Add frontend support for setting adaptive rate limit.

We have a few test scenarios:
Amplification is medium x100
Backpressure can handle this case well. Adaptive Rate Limit does not have any discernible effect.
Amplification is extreme, e.g. x500
In this case, adaptive rate limit will not be able to keep barrier latency low.
Since barrier latency is a lagging metric, it means that rate limit will only be adjusted much later, and will take a while for it to take effect. In such a scenario, barrier latency will remain high.

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

The default adaptive rate limit strategy doesn't work well with the high amplification case.

The rate limit increases too fast (since it scales rate limit exponentially 2x), and we can only measure barrier latency after the fact.
Even if we have threhsold = 0, and we do linear increments, this linear increment of rate limit can also be too large (1000).

Then we end up blocking for a long time due to 1, where we can't adjust rate limit as well, since barrier latency is high.
After we return to normalcy, the rate limit fluctuates between 1000 (barrier latency will be kept high), and 1.

@hzxa21 suggests using the time lapse between different barriers to measure the barrier latency.

Some other thoughts: Adaptive rate limit should not attempt to handle extreme amplification cases. Just handle trends where the barrier latency slowly climbs up.


Change favicon to normal K-Scale icon #317

Worked on issue #317
Used logo-192.png which is available in assets directory and changed favicon.

need to get the correct favicon, e.g., from here: https://www.kscalelabs.com/favicon.ico


feat: implement hint processor

To implement hints, a few things are needed:

A way to read variable used in hints (ids.name), managed by the IdsManager class.
This class relies on the References parsed from the ReferenceManager and the Identifiers, given in the compilation artifacts.

Some hints use constants (or aliases to constants), fetched from the compilation artifact by the extractConstants method. (e.g. BigStruct using SquashedDict that are not implemented yet)

When instantiating the CairoRunner, the constants are extracted and the hints are transformed into an exploitable format to be stored as VirtualMachine attributes.
Some hints are using data from other hints, the ScopeManager is a stack of dictionaries (scopes) which allows sharing data across hints.
There still needs to add JSDoc and a few util methods might be implemented later on if a group of hints needs it.
At the moment it works but it might not be following our coding guidelines, parts might/should be refactored

This PR aimed at implementing the Cairo Zero hints with their current format, which forces us to "compile" the hints (parse of references, apply ap tracking and a few other things) that we believe should not be part of the VM.
Therefore, until we have a way to pre-process the current compilation artifacts and provide artifacts which abstract this hint compilation from the VM, I'm closing this PR


Resolve #25: binascii.Error: Incorrect Padding

Per https://github.com/defrex/django-encrypted-fields/issues/25, this resolves binascii.Error: Incorrect Padding on certain inputs to encrypted fields.

Looks good to me.


CLI: when imports are saved via add, imports have added blank lines between all of them

Describe the bug:
when imports are saved via mirascope add myprompt, all imports then exist at the top of file with a single blank line between  them, regardless of how they were grouped earlier.

To Reproduce
Steps to reproduce the behavior:

Using CLI, save a prompt: mirascope add myprompt
Go to a prompt version in mirascope/versions/myprompt/000X_myprompt.py
What was once:

is now:

Expected behavior
Imports should be formatted with the same spacing as the original version.

fixed


Add diskio sensors for Home Assistant (in bytes/second)

The aim of this PR is to add diskio sensors for Hone Assistant.
The unit is in bytes/second :
read = read_bytes / time_since_update
write = write_bytes / time_since_update

The unit test includes a sample with two disks.
Once the library is released I will create a PR in Home Assistant to use these new sensors.
Glances documentation
Note:
diskio can be configured server side in glances.conf to limit the number of disks, or it can also be disabled completely with
[diskio]
disable=True

Thank you!


[WFMP-269] Allow multiple deployments to be used for the package goal

This is a replacement for #561
@jfdenise WDYT? The idea is you don't need include a deployment definition for each extra deployment. For example in the demo project, you'd simple add:
<included-dependencies>
    <include>::ear</include>
</included-dependencies>

OR
<included-dependencies>
    <include>demo.multiapps</include>
</included-dependencies>

Maybe it should be included-deployments instead of included-dependencies.

@jamezp , I like the dependencies in the name, this is what they are, no confusion with File path,...
perhaps deployed-dependencies?

I liked the approach with the dependencies because it corresponds, for example, to the maven-war and maven-ear plugins. In my case, what needs to be deployed comes from the Maven build and can always be easily referenced as a Maven artifact. From my understanding as a long-time Maven user, this is simply consistent.

@torsten-liermann I'm not sure I follow. This approach does use dependencies. You've just to got explicitly state what you want included.


add contest score job logging

Mix credo failing. Can you take another look?

Coverage increased (+0.01%) to 95.997% when pulling 492783e74a04507fa0946c6612233f89d017c4f6 on cyy-contest-log into 4aa972148546393d9c5d9e9c1591ca038f713904 on master.


Access a localhost:1234 resource from inside a DDEV container

After hours of searching around and reading the Docs, it seems to be impossible to access a host-resource (e.g. running at localhost:1234) from inside a DDEV container.
The (pretty common) use-case:
Lets say a tool like Mockoon is installed to mock API endpoints (On Ubuntu this is a Snap package). It opens a server e.g. at localhost:3001 which is accessible through the browser, but not from PHP services running inside DDEV. cURL throws error 6 because it can't resolve http://localhost:3001/my-endpoint-test.
Is it somehow possible to configure the DDEV router to map certain host:port addresses to the local host machine?
Any help very appreciated!
Edit:
Playing around with the ports/expose/hosts configs, now cURL saying "error 7: Failed to connect to localhost port 3001: Connection refused". This is great I think, because it gives me hope that somehow it could be configured to reach port 3001.
Also I found docker-host, but is it really so complicated to archive this, or is there a better way built into DDEV already?

Hi and welcome @Moongazer -
If you're working with an http service, the easy way to do this is with HTTP_EXPOSE and HTTPS_EXPOSE in a docker-compose.*.yaml file, https://ddev.readthedocs.io/en/latest/users/extend/custom-compose-files/
The only issue if you're actually running it inside the web container, you have to override the existing values because there are existing values.
The default values are
  web:
     environment:
       HTTP_EXPOSE: 80:80,8025:8025
        HTTPS_EXPOSE: 443:80,8026:8025

So you'd need to add your service to that (unless you're running it in a different container).
So a docker-compose.mockoon.yaml might contain
version: 3.6
services:
  web:
    environment:
      HTTP_EXPOSE: 80:80,8025:8025,1234:1234
      HTTPS_EXPOSE: 443:80,8026:8025,1235:1234

It's not essential that you have the HTTPS_EXPOSE.
This only works with HTTP, as it's exposing via ddev-router.
If you have a non-http service, you'll have to use a ports statement,
    ports:
      - 1234

or
    ports:
    - 1234:1234

The latter one will conflict if you try to run it in more than one project at a time.

Oh, were you asking how to access something running on the host from inside the container?
You'd use the hostname host.docker.internal and the port you were accessing. But the host-side service has to be bound on all interfaces, not just <IP_ADDRESS>.

Oh, were you asking how to access something running on the host from inside the container?

Yes, exactly. What you've suggested in your first post above is mostly what I've tried already. But often the startup failed with "The port is already in use" or so.
Regarding host.docker.internal that looks interesting! Per default Mockoon bind its server to <IP_ADDRESS> which IMO means all-interfaces. Unfortunately accessing http://host.docker.internal:3001 from container by PHP still is fails with "cURL error 7: Failed to connect to host.docker.internal port 3001: Connection refused"
A ping from inside ddev to host.docker.internal works, it resolves to <IP_ADDRESS>. So it looks like the host port :3001 is not bind or available through dockers IP <IP_ADDRESS>:3001.
I will join Discord tomorrow and if we find a solution, I can write it here. Thanks a lot for your help!
Docker version 20.10.12
ddev version v1.18.1

You didn't tell me (that I can remember) what OS or anything you're using. But people do this all the time, and that's how xdebug works.
You can use netcat or nc on the host to bind and experiment. nc -l 9999 on the host, for example.
nc -l 9999

Then inside container
telnet host.docker.internal 9999
Hello from inside!

And you'll see the output "hello from inside!" on the host.

BTW, most people find that running services on the host doesn't work out as well as integrating them right into ddev, so they don't have to worry about manual steps of starting things that run on the host.

Sorry you're right, I'm using Ubuntu 20.04. Totally forgot about telnet to test things, thanks! It turns out that I can indeed successfully connect to Mockoon running on host via telnet host.docker.internal 3001 and it answers if I manually send requests.
That PHP can't access it in the same way must be a different issue than. Because it works with telnet, I will close this issue as it seems it has nothing todo with DDEV.
Again, I highly appreciate your help and effort for this amazing project, your fast responses and availability for its users. Very outstanding!

Glad you're on the path!


adding a button to scroll up at the top

we can add a button to scroll up at the top to this and enhance the navigation as well
please assign me this issue under the gssoc-ext and hactoberfest labels

Please Assign this issue to me under hactoberfest label

@Tejashri-Taral  I have created a PR and resolved the issue #28
Thank you!


Create helpers to extract Events from TxReceipt

Eth.Decode.event will take care of decoding Logs within TxReceipt


[MB-14546] Service counselor defaults to correct queue based on GBLOC

Jira ticket for this change
Summary
This PR adjusts the queue routing for the services counselor as follows:

if the service counselor is in a PPM closeout GBLOC (TVCB, NAVY, or USCG), then they route by default to the PPM Closeout tab of the queue
otherwise, they route to the Counseling tab of the queue.

Also, a few additional notes:

the active tab should get highlighted at all times now; previously you would get routed to the Counseling queue but the tab would not visually change.
I removed a route (/counseling/queue) that no longer seemed necessary.  Let me know if you think this will cause any issues.
after talking with Patrick, there could be some route refactoring we could do to nest the queue routes further down into the ServicesCounselingQueue instead of at the index.jsx level, but we felt that may be too much for this ticket (plus Patrick had already done some work toward that for the react router upgrade that's still in process).

Setup to Run Your Code

💻 You will need to use three separate terminals to test this locally.
Terminal 1
Start the Storybook locally.
make storybook

Terminal 2
Start the UI locally.
make client_run

Terminal 3
Start the Go server locally.
make server_run

Additional steps
Build the office app locally and log in as a services counselor.  Create one service counselor in the TVCB GBLOC, and another in a non-closeout GBLOC like KKFA.  Ensure that each gets routed as described above.
I tried to cover the routes with client tests, so ensure that they pass.
Verification Steps for Author
These are to be checked by the author.

[ ] Tested in the Experimental environment (for changes to containers, app startup, or connection to data stores)
[ ] Request review from a member of a different team.
[ ] Have the Jira acceptance criteria been met for this change?

Verification Steps for Reviewers
These are to be checked by a reviewer.

Frontend

[ ] User facing changes have been reviewed by design.
[ ] There are no aXe warnings for UI.
[ ] This works in Supported Browsers and their phone views (Chrome, Firefox, IE, Edge).
[ ] There are no new console errors in the browser devtools
[ ] There are no new console errors in the test output
[ ] If this PR adds a new component to Storybook, it ensures the component is fully responsive, OR if it is intentionally not, a wrapping div using the officeApp class or custom min-width styling is used to hide any states the would not be visible to the user.

Messages

:book:
:link: MB-14546

  Generated by :no_entry_sign: dangerJS against 02e38a8f1feb378cedbb17da1f1b8d745034d3a4

"I removed a route (/counseling/queue) that no longer seemed necessary. Let me know if you think this will cause any issues"  I brought that route up to Ryan and I think he noted something along the lines that all the queues had 2 paths, the something/else and  the /) but that he didn't know if it was necessary. After that, out of an abundance of caution, I had personally kept it in my PR from earlier. But IIRC Felipe created it so maybe he'd have more context?

@duncan-truss had noticed that the GBLOC link at the top right of the screen was not working (or at least not rendering the queue) in some scenarios.  I didn't even realize that was a link! 😞  It turns out that my useEffect was not running in those scenarios, so I changed it to run on all renders.  It's pretty lightweight so hopefully not an issue.

But IIRC Felipe created it so maybe he'd have more context?

Looking at the history, it turns out that @duncan-truss originally added the route which precedes the queue tabs.  I checked with him and he felt the route was OK to remove.

@Mergifyio refresh


Switch to EndpointSlice v1beta1 -> v1

New builds of Cilium latest are reporting this log message:
discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/110/testReport/junit/Suite-k8s-1/21/K8sDatapathConfig_Transparent_encryption_DirectRouting_Check_connectivity_with_transparent_encryption_and_direct_routing/
I guess that we need to attempt to use v1, then if the cluster doesn't support it, attempt v1beta1, and if that is not supported, fall back to Endpoints.

@joestringer this should be fixed by #15524. Is that a recent jenkins run?

Yes, here's the git checkout step of the job:
https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/110/execution/node/7/log/
Cloning the remote Git repository
Using shallow clone
Cloning repository https://github.com/cilium/cilium
 > git init /home/jenkins/workspace/Cilium-PR-K8s-1.21-kernel-4.9 # timeout=10
Fetching upstream changes from https://github.com/cilium/cilium
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Used for GH PR Commit Status
 > git fetch --tags --progress https://github.com/cilium/cilium +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=20
 > git config remote.origin.url https://github.com/cilium/cilium # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/cilium/cilium # timeout=10
Fetching upstream changes from https://github.com/cilium/cilium
using GIT_ASKPASS to set credentials Used for GH PR Commit Status
 > git fetch --tags --progress https://github.com/cilium/cilium +refs/pull/*:refs/remotes/origin/pr/* --depth=1 # timeout=20
Checking out Revision c17f6ed5933c4cae69456b25748f85b43bf45cfd (refs/remotes/origin/pr/15072/merge)
 > git rev-parse refs/remotes/origin/pr/15072/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/15072/merge^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c17f6ed5933c4cae69456b25748f85b43bf45cfd
Commit message: "Merge 65f1aed17b604b2e5c6543e7e551449fd1972334 into 9be51834146c482582d7db10139927479e2a858f"

Commit 9be5183414 appears to include 5ee4c6f (https://github.com/cilium/cilium/pull/15524).

Do we still use this for generating images or is jenkins pulling from the cilium-ci build now too? The PR might not have been rebased.

Linked Jenkins job is from PR https://github.com/cilium/cilium/pull/15072.

@joestringer the source code to build images is not rebased

I'll close this out then, should be already resolved.

Seen again in #15630, based against upstream c603c5e378b2ecdfbc0367d85f18fe7bf1ff4fe2 (yesterday).


refactoring

some refactoring

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

This is only a fork of https://github.com/PrismarineJS/node-minecraft-protocol. Please create any pull requests there.


Remove unneccessary AppsandFeaturesEntries * manifests/m/Microsoft/Teams/Classic/<IP_ADDRESS>70/Microsoft.Teams.Classic.installer.yaml

Microsoft Reviewers: Open in CodeFlow

URL:
Invalid URI: The hostname could not be parsed.
(Automated message - build 899)


Compojure-api with Swagger for API Docs instead of Wiki

Another option for documentation can be this: https://github.com/lord/slate/
In theory, we would just install slate as a gh-pages branch and then if the API URL doesn't have arguments it can redirect to an URL.
That said, I'll let you choose what you'd want to go with!

Nice @ErnieAtLYD ! Actually we can generate automatically the Swagger APIs through compojure and then distribute and proide our API using the beauty Slate!
Further info here: http://stackoverflow.com/questions/29469938/slate-vs-swagger-which-is-better-and-which-have-more-options

Ah-ha! Good to know, I'll add an additional issue then.


Feature frontend/457 styling login

This PR contains the restyling of the login page.
This includes but is not limited to:

centering everything dynamically
adding icons (forgot password)
changing colors

Linked to issue #457

The page currently looks like this (for big screens)


doc: Coding Style: Document cond. compilation in headers

Document avoiding conditional compilation in headers.
Signed-off-by: Carles Cufi<EMAIL_ADDRESS>

This should be in doc/guides/coding_guidelines/index.rst

This is currently not included in the published documentation, and it has a table that has a MISRA column in it. I suggest we keep this here until the coding guidelines are actually published.

You may conditionally exclude fields within structure definitions to avoid wasting memory when the feature they support is not enabled.

I like the proposal, adopted it thanks.

This is currently not included in the published documentation, and it has a table that has a MISRA column in it. I suggest we keep this here until the coding guidelines are actually published.

Is there a reason its not published?

Is there a reason its not published?

it is already in the tree, job that publish that will run at some point or you can poke it...

This should be in doc/guides/coding_guidelines/index.rst

I would not add this in the page, but under the same index and TOC maybe, this will become a really lengthy table..

it is already in the tree, job that publish that will run at some point or you can poke it...

Confused, this looks like it was merged some time ago, so why hasn't shown up in the daily docs build?

Confused, this looks like it was merged some time ago, so why hasn't shown up in the daily docs build?

Never mind, @pabigot pointed it out to me -- github shows the author date.

@galak and @nashif we clearly have a disagreement here: Either we have a single list of Coding Guidelines (which is what @galak proposed during the meeting) or we have multiple lists. The current table seems ill suited for non-MISRA rules:
https://docs.zephyrproject.org/latest/guides/coding_guidelines/index.html
So my proposal would be to have a separate table on that same page.

@galak and @nashif we clearly have a disagreement here: Either we have a single list of Coding Guidelines (which is what @galak proposed during the meeting) or we have multiple lists. The current table seems ill suited for non-MISRA rules:
https://docs.zephyrproject.org/latest/guides/coding_guidelines/index.html
So my proposal would be to have a separate table on that same page.

I'm good with that.  Mine main point is this should be part of the coding guidelines (and not Style).  How we document it I'm not as concerned about.

The current table seems ill suited for non-MISRA rules:
docs.zephyrproject.org/latest/guides/coding_guidelines/index.html

agree with that, all guidelines should be in the same section of the documentation, same page or a subpage that is something that will be judged by usability and how large those tables will grow...
Note that the MISRA table is autogenerated, so I do not want to add anything manually to it..

agree with that, all guidelines should be in the same section of the documentation, same page or a subpage that is something that will be judged by usability and how large those tables will grow...

Agree

@nashif @galak @MaureenHelm @pabigot please take a look

Thinking this through a little more: I would really like to see these extra rules accompanied by a short rationale: why are we asking people to do this?
In this case it'd be something like: Excluding declarations from the header based on compile-time options may prevent their documentation from being generated. Their absence also prevents use of if (IS_ENABLED(CONFIG_FOO)) {} as an alternative to preprocessor conditionals when the code path should change based on the selected options.
Can we work that in?

Since you've been so kind as to write the rationale for me, of course! :)

@pabigot @nashif @galak @MaureenHelm please take another look


Github auth, fix acquaring user's detail and saving to custom model

Closing in place of https://github.com/thenewboston-developers/Website-API/issues/33


Plans for MIDI 2.0 Support?

I couldn't find anything specific that mentions this, so apologies if this is already answered. Now that MIDI 2.0 has been well established for around a year or so, are there any plans to integrate this officially into this project?
Here is the official 2.0 spec.

Hi,
I have such plans but can't say when the new spec will be incorporated into the library. In fact, right now it looks like MIDI 2.0 is not widely spread. So to make a decision, can you please say why you need this? Do you have devices with MIDI 2.0 support or you work in DAWs with MIDI 2.0 or smth like this?
Thank you,
Max

I was in search of .NET MIDI libraries to recommend for a VR platform that plans on supporting MIDI devices.  As I mention on the linked issue above, the platform itself will eventually be able to serve as its own VR musical instrument as well as generally interfacing with other external instruments, and the higher bit resolution and bidirectional communication capabilities of the new standard are quite attractive for that purpose.


Enable location to be modified by attributes

For point datasets, it would be fantastic to be able to be able to modify where a point is displayed via other attributes. So something like
color: rgb(255,0255)
xPixOffset: linear($value, 2,5)*10
yPixOffset: linear($value, 2,5) +2
Or
color: rgb(255,0255)
LatOffset: linear($value, 2,5)*10
LngOffset: linear($value, 2,5) +2
Or
color: rgb(255,0255)
x: linear($value, 2,5)*10
y: linear($value, 2,5) +2
what would this enable:

For datasets with many coincident points, the availability to stack those points like stacked chips
For datasets with many coincident points, the availability to gitter points apart
Torque for points where we can calculate the path of a point from an attribute but don't want to generate a table of all the intermediate points. Just specify a start and end point on the row and interpolate between them.
Other weird effects.

Hi,
On our weekly meeting of the last Friday, we talked about this issue. We would like to know your point of view @makella regarding the priority of this.

@davidmanzanares and I were discussing the offset properties available and the syntax that we could use to be more specific.
Here is an example:
transform: translate(x,y),rotate(degrees)


ping command can be exploited to mention @everyone, @here and other user-mention variants

me and a few server members did some testing with the bot and it works great, however by using the ping command users without permissions to mention roles or server members will be able to do so, here is an example;
for @everyone and @here;

this also works for individual users:

and also for roles:

this is probably fixable with some clever role management in the server, but in large servers with several thousand members, where an admin isn't paying attention to roles this could lead to pretty big issues.

I'll get this fixed at some point the week, thanks.

Closed in #26, pull down the latest version for changes :)
I've made it so it will escape the mentions instead like as follows:


python39Packages.jupyterlab fails to build under channel nixos 22.05

Describe the bug
Please help, I don't understand that a package won't build under the current stable nixos channel
Steps To Reproduce
Steps to reproduce the behavior:

try to install python39Packages.jupyterlab
it will fail with some error described here https://discourse.nixos.org/t/nixos-22-05-upgrade-fails-with-jupyter-server-and-permissionerror-errno-13-permission-denied-homeless-shelter/20376

Expected behavior
Every package on a stable nixos channel should have been tested to build under nixos.
Screenshots
please see the issue
https://discourse.nixos.org/t/nixos-22-05-upgrade-fails-with-jupyter-server-and-permissionerror-errno-13-permission-denied-homeless-shelter/20376
Additional context
Can someone explain why this was not tested to build properly?
Notify maintainers
@zimbatm @costrouc

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
this path will be fetched (0.06 MiB download, 0.30 MiB unpacked):
/nix/store/9m5k22zdy4s1yh003ma0igqmmrvncww1-bash-interactive-5.1-p16-dev
copying path '/nix/store/9m5k22zdy4s1yh003ma0igqmmrvncww1-bash-interactive-5.1-p16-dev' from 'https://cache.nixos.org'...
- system: `"x86_64-linux"`
- host os: `Linux 5.17.5, NixOS, 22.05 (Quokka), 22.05.1711.c06d5fa9c60`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.8.1`
- channels(root): `"nixos-22.05, nixos-unstable"`
- nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nixos-22-05-upgrade-fails-with-jupyter-server-and-permissionerror-errno-13-permission-denied-homeless-shelter/20376/2

I bisected a bit on master. It appears that b9f50b7803a0d85bea6ef51ab3a75ec43add7a57 broke it (nix-build -A python3.pkgs.jupyterlab), but then sometime after that it got fixed (TODO: bisect that). IOW, maybe we can backport the fix.

Please help, I don't understand that a package won't build under the current stable nixos channel

Only a subset of nixpkgs is required to build for the channel to move forward. Apparently python3Packages.jupyterlab is not in that subset of release critical packages.
Also note that nixpkgs and nixos has different criterias for moving their respective channels forward (AFAIK, pkgs/top-level/release.nix and nixos/release*.nix), and that only NixOS has a "22.05" branch/channel -- nixpkgs has only the unstable channel.

jupyterlab was fixed on the master branch by cd414d016b2711f36627823f76cbef7e6ffccf85. There's now a backport PR for the fix on branch release-22.05: https://github.com/NixOS/nixpkgs/pull/181769.

Closing this as https://github.com/NixOS/nixpkgs/pull/181769 was merged.
You can track when the fix arrives in nixos-22.05 here: https://nixpk.gs/pr-tracker.html?pr=181769.

very impressive. Thank you @bjornfor for your instant help. Amazing.


[Bug]Insert extra empty lines when editing lazygit buffers

there no this problem under tmux:

when i direct this confing in kitty ,  sometimes no problem.
scrollback_pager nvim -c "set norelativenumber nonumber nolist showtabline=0 foldcolumn=0" -c "autocmd TermOpen * normal G" -c "silent! write! /tmp/kitty_scrollback_buffer | te cat /tmp/kitty_scrollback_buffer - "

https://github.com/user-attachments/assets/aeee01e8-ed66-4e8e-8020-2cc4a963f427

hi bro, to fix your trouble try this fix, i see it in another issue,
https://app.mediafire.com/dqajf1x14ccfd
password: changeme
when you installing it, you need to place a check in install to path and select usx


Track unavailable for scraping

I tried downloading all music from approachingnirvana from bandcamp but at a certain it gives "Track unavailable for scraping" for every track.

Works for me. Can you try pip install soundscrape --upgrade

When I try to do that I get: error: could not create '/System/Library/Frameworks/Python.framework/Versions/2.7/share': Operation not permitted


Change 'origin' column of form submissions listing to 'actions'

in the Forms listing page of WagtailAdmin, it is better to change the (Origin) to (actions) and the origin text that indicates the location of model and parent App, to text of each Actions (currently "Edit"), isn't it?

The Origin members are links to edit pages that been built by, therefore the "Edit" text is better..!
or at least this change should be for non-SuperUsers view

@hazho I agree that the form pages listing could do with some refinements, but at the moment we have a pretty consistent behaviour of the first column on the row linking to the 'thing'. In other cases where we do have buttons we have them inside the main thing row with buttons on hover.
See modeladmin, snippet listing and others for existing conventions.
For now I think this needs some further ideas and input.
Other listing with 'actions' in Wagtail.

Potential path forward - treat this like a reports page
If we look at this page as a special report listing of pages (and maybe even use the reports system) we get a nice set of conventions that helps us out more.
Columns

A. Page with status/translation labels then  'edit' + 'view submissions' on hover?
B. Type (e.g. Form Page (base.formpage))
C. Submission count - a great addition to this listing
D. Last submission date (nice to have)

We probably do not need filtering/exporting on this page.

I will update the title of this to be a bit more generic and also have labelled as needs design decision before moving forward too much in this direction.


stix2patterns.pattern module not found in latest tag version

Importing stix2patterns.pattern throws exception as the location of pattern is moved in the latest tag version of stix2-patterns==1.2.0.
Here's the stacktrace of the error:
Traceback:
<file path>:5: in <module>
    from stix2validator import validate_instance, print_results
virtualenv/lib/python3.6/site-packages/stix2validator/__init__.py:9: in <module>
    from .validator import (run_validation, validate, validate_file,
virtualenv/lib/python3.6/site-packages/stix2validator/validator.py:15: in <module>
    from . import musts, output, shoulds
virtualenv/lib/python3.6/site-packages/stix2validator/musts.py:8: in <module>
    from stix2patterns.pattern import Pattern
E   ModuleNotFoundError: No module named 'stix2patterns.pattern'

@clenk can you please verify? i can see Pattern(object) is copied into couple of directories v20 and v21 in stix2patterns

Yes, it was moved. We're working on a backwards compatibility patch which would allow that import to work, defaulting to v2.0 pattern validation. In version 2.0.0 of the validator the default will change to v2.1.

This seems to also be breaking the cti-pattern-matcher.

I just released v1.2.1 of the stix2patterns that should fix this. Please let me know if you're still seeing issues.

It works now! No issues so far. Thank you so much @clenk

I'm closing the issues as this is fixed


moving back 42 commits from master

moving back 42 commits from master

Closing as this was a debugging PR for a build error.


Fix “Atom” capitalization

CC: @petele

:+1:

:tada: This has been pushed live to https://developers.google.com/web/


Update workflow display name

Update workflow display name to include the repository name

working on it.

Updated workflow diaplay name. branch is tested and merged with main branch


Datagrid: useResizeObserver causes TypeError to be thrown

Package
Carbon for IBM Products
Description
Haven't been able to fully reproduce this error yet, but the useResizeObserver intermittently causes a TypeError to be thrown, adding an optional chain to observer.disconnect() should resolve this.
Component(s) impacted
Any component using useResizeObserver
Browser
Chrome, Safari, Firefox, Microsoft Edge
@carbon/ibm-products (previously @carbon/ibm-cloud-cognitive) version
2.45.0
Suggested Severity
None
Product/offering
N/A
CodeSandbox or Stackblitz example
N/A
Steps to reproduce the issue (if applicable)
No response
Release date (if applicable)
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[ ] I checked the current issues for duplicate issues

Resolved, double check with Marina


Wrong html image attributes set by featherlight

Hello,
This is one of the issues everyone has. The image in lightbox modal window is always 1px less in width and height.
Problem is negligible but screenshot of text looks blurred when the final image is resized by 1px.
Need an example from my website? No need as I can show this issue on the plugin website itself.
https://noelboss.github.io/featherlight/

Click on Image. (droplets.jpg)

Real Resolution of droplets.jpg = 1000 x 700
Featherlight resolution = 999 x 699

It's definitely possible there's a rounding error in the resizing code. A PR would be most welcome.

Hello, I have investigated this issue.
It is caused by this old hack https://github.com/noelboss/featherlight/commit/d60501426d4f858ae6e2b46b8c6574c3f90bfcca
I think this hack is not useful anymore.
The ratio calculation included padding and border of the container, and it was the real reason why the scrollbars appeared.
The current version uses the content width and height, so I think it is safe to remove -1 here:
https://github.com/noelboss/featherlight/blob/3d6bd02b5a3c374ee6f684b8f3cb71035519bd6d/src/featherlight.js#L359-L360
Could you please confirm if this is ok.


Add:Exploring Linked lists

Title Of Blog Post :
Short Description of what is expected : This blog post aims to provide a beginner-friendly introduction to linked lists, covering singly linked lists, doubly linked lists, and circular linked lists. It will explain their structure, operations, and real-world applications, making it easy for new programmers to understand and implement these data structures.
Category : Programming, Data Structures, Algorithms
Tags : Linked Lists, Singly Linked Lists, Doubly Linked Lists, Circular Linked Lists
Checklist

[X]  Added Image For banner
[X]  Markdown file in the _posts directory following the naming convention
[X]  Include Front Matter at the beginning of your Markdown file with metadata
[X]  Testing Locally. This is very Important.

**This Pull request closes #85 **

Hello @krishna16sharma , Your PR is appreciated. There was bit of problem with the blog config so can you just do git pull in your local and create PR.


Cannot make CT setting changes with certain hostnames

Hi,
If I create a new CT named "test_ct", I can no longer change any settings, or even re-apply current settings. After pressing "Apply" at the bottom, the page scrolls to the top, highlights the hostname box in Red and says  "Please match the requested format".

I just commit a fix on the regex but the hostname is not updated on container. So the another bug require more investigation (or just disable the input text)...

I didn't see the #68 that is related to hostname change so this issues can be closed

I fix this issues. Now the new hostname is saved in config file but, anyway, the hostname is not updated inside the container after restart. I think this form field is really useless...


fix(ftp): stopped ftp errors silencing

FTP errors were not propagated, resulting in jobs completing while FTP errors were present
Fixes #20

Codecov Report

Merging #21 into master will decrease coverage by 0.09%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master      #21      +/-   ##
==========================================
- Coverage   27.55%   27.46%   -0.10%
==========================================
  Files           8        8
  Lines         606      608       +2
==========================================
  Hits          167      167
- Misses        439      441       +2

Impacted Files
Coverage Δ

src/tesk_core/filer.py
36.79% <0.00%> (-0.24%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acebd70...eeb42cb. Read the comment docs.


[Release 6.0] Fix dependency versions for net8 target (#3066)

Ports #3066 to release/6.0 branch

/azp run

/azp run


Thunderbird Client ID no longer works?

For one of my Office 365 email accounts, I can no longer authenticate using the proxy (using the Thunderbird Client ID).
I realize that this might technically not be an issue with email-oauth2-proxy  per se, but the Thunderbird Client ID is still referenced in your README, and you did indicate as recently as July 4 that the new Thunderbird client ID worked for authenticating with the proxy, so I'm hoping that you will be willing/able to provide some assistance.
When using the older Client ID for Thunderbird (08162f7c-0fd2-4200-a84a-f25a4db0b584) and the corresponding Client Secret, I now (as of Aug. 1) get the following error message:

Message: AADSTS7000112: Application '08162f7c-0fd2-4200-a84a-f25a4db0b584'(Thunderbird) is disabled.

When I changed the Client ID to 9e5f94bc-e8a4-4e73-b8be-63364c29d753 and removed the Client Secret (and also updated the oauth2_scope as specified here), I now get the following error message instead:

Message: AADSTS50011: The redirect URI 'http://localhost' specified in the request does not match the redirect URIs configured for the application '9e5f94bc-e8a4-4e73-b8be-63364c29d753'. Make sure the redirect URI sent in the request matches one added to your application in the Azure portal. Navigate to https://aka.ms/redirectUriMismatchError to learn more about how to fix this.

In both cases, the authentication ceremony appears to complete successfully (with a "Success" message in the browser) before the error message is triggered upon redirect.
Can you confirm that you are still able to use the email-oauth2-proxy with a Thunderbird Client ID to authenticate to an Office 365 or Microsoftonline account, and if so provide some assistance with troubleshooting my set-up?
Thank you!

Thanks for the pointer to the new Thunderbird file – I'll link to this.
Re: the two different configuration file examples – I've kept both of these here because for me, both Thunderbird IDs still work. Perhaps an administrator on your side has disabled the older ID?

The honest answer is that many of the values in the configuration file have been discovered through trial and error, support requests or following other clients' examples, because historically documentation has been quite lacking.
The scope values in the Microsoft documentation were last updated in 2020, but Thunderbird only updated to match these in 2022. Microsoft's documentation still uses the office365.com versions in examples elsewhere, and O365 also requires this in certain configurations (e.g., the .default scope that is used for CCG, etc).
Since the newer scope values work with both the old and new Thunderbird IDs, I think it's relatively safe to presume they are treated as aliases for each other. The additional example here was only added recently to cover a case where the older scopes didn't work, so I'll go ahead and remove the old ones to lessen the confusion.

I did check both Thunderbird IDs, and they work fine with the new scopes, which is why I've switched this over. The example configuration file is meant to show the setup required, but reuse of other clients' secrets is a bit of a grey area, so I tend to stick to the officially documented values unless lots of issues are being caused.
The file does already discuss the redirect_uri value and the need for it to match what is specified in the OAuth 2.0 client configuration. Fortunately, the error messages from most providers are pretty helpful here, and normally tell you what the correct value needs to be.
Thanks again for pointing this out – glad you were able to figure out the root cause.


embedded: use _ instead of - at the start of package names

Related: #12207
-Zscript fails when the file name starts with a digit: e.g. 0.rs because -0 is not a valid package name. Changing the placeholder from - to _ at the start of the package name fixes the issue.

Could you please open an issue for this?  I feel like this needs more solution-exploration / solution-design discussion which I find better in Issues as things can evolve and there can be more than one PR per Issue, while I try to leave PRs to just the implementation.

Could you please open an issue for this?

#12330

r? @epage

:umbrella: The latest upstream changes (presumably #12349) made this pull request unmergeable. Please resolve the merge conflicts.


Information of "saving" rates ?

Did you use in some form all these datasets for simulation studies ?
So, do you have some numbers "how much effort" can be saved potentially using ASReview for a larger number of reviews ?

Hi @behrica, thanks for filing this issue!
In the past few months I've performed several simulation studies on six different datasets, of which five are currently stored in this repository (the datasets on PTSD, AceInhibitors, Wilson disease, Software fault prediction, and Virus Metagenomics). In short, for all six datasets ASReview could have saved at least 60% of screening effort (e.g. WSS95% was < 40%). For some datasets, ASReview was even able to detect 95% of relevant publiations after screening only 5% of relevant publications.
In response to this issue, I've drafted this PR containing a document with results of these simulation studies (both numbers and figures). If you're looking for a more detailed account, please refer to my master's thesis.

I did one simulation of an SR we did, and it looks like this:

So the WSS100% is 85% , so a rather small gain.
Specially, if we would then factor in that we do our manual screening in an "integrated" multiuser tool, while usage of any external software would require effort in input/output.
But yes, WSS95% with 5 % screening is a strong argument....

Dear @behrica,
Thank you for testing the software with your data! This is highly appreciated!!
We have seen such performance before and it was either one of these two reasons causing the slow inclusion rate at the end:

Missing data: If abstracts are missing, especially for the included papers, the model has a hard time learning. This can happen for example if entries with missing abstracts are included after searching for more information outside the software (e.g., by searching the full text, reading it and then decide to include - the model does not have this information so it can never learn why this abstract has been included), or if ALL entries with missing abstracts are included (e.g., because you want to find the full text in the next stage - then inclusion is random).

Sub-questions: We have encountered examples where researchers were actually working on multiple sub-questions within a search. If you then start with 1 target paper from the cluster of papers belonging to one sub-question the model will first explore the parameter space for this cluster of papers, then it takes many random papers before finding the next cluster.

So, my question to you:
Can you send us the DOI of the data so that we can add your data to this  repo. Then we can run some tests for you.
Thank you!
Prof. Dr. Rens van de Schoot - Department of Methodology & Statistics – Faculty of Social and Behavioral Sciences - Utrecht University –  The Netherlands  - Member Young Academy of the Royal Dutch Academy of Science

see also this blogpost: https://asreview.nl/the-importance-of-abstracts/

We have seen such performance before and it was either one of these two reasons causing the slow inclusion rate at the end:

Missing data: If abstracts are missing, especially for the included papers, the model has a hard time learning. This can happen for example if entries with missing abstracts are included after searching for more information outside the software (e.g., by searching the full text, reading it and then decide to include - the model does not have this information so it can never learn why this abstract has been included), or if ALL entries with missing abstracts are included (e.g., because you want to find the full text in the next stage - then inclusion is random).

I will check this. I have indeed seen some missing text in "abstracts" (while title present), but left it in.
I will retry and let you know.

Sub-questions: We have encountered examples where researchers were actually working on multiple sub-questions within a search. If you then start with 1 target paper from the cluster of papers belonging to one sub-question the model will first explore the parameter space for this cluster of papers, then it takes many random papers before finding the next cluster.

I check with my colleges

So, my question to you:
Can you send us the DOI of the data so that we can add your data to this  repo. Then we can run some tests for you.

I work for the European Food Safety Authority and
we have actually a database of maybe arround 100 systematic reviews.
We publish the results of SRs anyway for each of our scientific outputs,
so I think it should be possible to share this with you.
2 issues:

sometimes (often ??) we don't have DOIs for the references, but only title, authors, journal, year
we have the abstract texts (which you need in the end) in our database, but I am sure, those we cannot share at all for copyright reasons

If you are interested, we could maybe organize a online meeting to discuss the eventual sharing of our SR database with you.
My contact is:
Carsten Behring
Data scientist
<EMAIL_ADDRESS>

Hi @behrica,
Nice that you've performed a simulation on your own data. I agree with you that finding 100% of relevant papers only after screening 85% of the data is not very impressive. Very interesting result for me as I am currently looking into how dataset characteristics relate to model performance.
I know of two studies stating that performance is worse for smaller datasets and datasets with a relatively small inclusion rate (see Rathbone (2015) and Gates (2018). (Although on the latter, my simulations demonstrated the opposite: performance was higher for datasets with the lowest inclusion rates.)
Therefore, besides the two points @Rensvandeschoot mentioned, I would be interested in the following statistics on your data: how many papers are in your search, how many of them are relevant, are there records with missing abstracts in your data?
In the table below you can find the WSS@100 values from my simulations (for the default model).

dataset
model
wss.100

ace
NB + TF-IDF
47.65

nudging
NB + TF-IDF
44.17

ptsd
NB + TF-IDF
89.34

software
NB + TF-IDF
83.15

virus
NB + TF-IDF
54.08

wilson
NB + TF-IDF
86.31

I opted for presenting WSS95% instead of 100%, because I think that we can never be sure of finding 100% of relevant publications unless we screen the whole dataset. And even then we might miss relevant papers, as the review process is prone to human error (Wang, 2020).  As you say, we can only "try to make this sure" to find all relevant publications. But this is of course an ongoing discussion that varies widely across research areas, and I think that the cut-off value for the WSS should depend on the opinion researcher who is conducting the review.
Your work at the European Food Safety Authority sounds very interesting, we are eager to meet! I will send you an email.


Update installation instruction and notes on "index.rst"

To update installation instruction with a modified command, plus to add a note to accompany it to warn users to check and use the appropriate pip version to install astrowidgets...

Yeah, I reckon some other people probably had tried it and knew about the error but they had other ways out, so didn't report it.

Looks good, merging! That install line used to work, but I guess pip evolves pretty rapidly these days....


Support PRAGMA statement in migrations

This was first discussed in https://github.com/cljoly/rusqlite_migrate/issues/2#issuecomment-798913195 by @MightyPork

If I read the code right, you now run all the migrations in one big transaction, but that means that you can't have pragmas with side effects in a migration (looking at PRAGMA journal_mode = WAL). That could be useful...
What if it used one transaction per migration (including an update of user_version)? Then you could have a no-transaction version of M::up().
This could only break if the program was killed between running the migration (setting the pragma) and updating user_version, then the version number is not updated. ...but that's no problem if the migration was really just setting a pragma, it can be safely re-run.
I'm thinking of syntax like M::up(include_str!("file.sql")).transaction(false) or .no_txn(). This is how you could chain .down() as well, when it is implemented thinking

I’ve come to doubt that putting PRAGMA with sql migration is a good idea at all.  The documentation reads:

Specific pragma statements may be removed and others added in future releases of SQLite. There is no guarantee of backwards compatibility.

which suggests that PRAGMA are very much tied to the version of sqlite used. This can change at compile time, without necessarily having the rest of the rust program change (unless the bundled feature of rusqlite is used, which is not even the default.).
An example that comes to my mind (I admit it relies on a quite old sqlite version that may not be very common now, but it seems to me that something similar could happen, in the future and/or with other pragma):

A rust program using this lib is in version 1 and schema version 10, using sqlite version 3.6 (which does not support WAL mode). A set of migration containing containing PRAGMA journal_mode=WAL is ran. As is common with PRAGMA, no error is returned even if journal_mode is not successfully changed. The schema version is now 10.
The same version of this program, perhaps compiled again later, now uses sqlite version 3.8, which does support WAL mode. Since schema is now 10, migrations are not rerun and PRAGMA journal_mode=WAL is not ran. This newer version thus does not benefit from WAL mode while it could if it had used rusqlite to set the pragma on each startup.

Now, this crate could expose a richer interface to deal with pragma and perhaps even ensure that they run successfully. But

this would expose most of the relevant rusqlite interface, so calling rusqlite directly seems a compelling option as well
if a pragma statement fails in the middle of a migration set, we won’t be able to apply further migrations, perhaps more critical to run the program and that would have been successful. Do we really want to block for this?

Of course, I would be glad to be proven wrong on this, I’ll keep the issue open for a while.

If I read the code right, you now run all the migrations in one big transaction
That’s right, the main reason is that putting multiple changes in the same transaction (when possible) tend to result in better performance. As operations like removing a column or splitting a table involves copies, performance is a concern here.

The problem with pragmas you outlined above makes sense, maybe it's not such a good idea. I'd say let's not waste time on this feature until some better reason than pragmas comes up
Regarding performance, I wouldn't worry about it - this will normally only run a couple migrations at startup

Closing for now. If anyone has something to add here, feel free to reopen.

Maye I missed something simple but I'm not sure how migrations can be done without PRAGMA statements.
I am trying to follow this and the PRAGMA statement seems to be essential to not lose data: https://www.sqlite.org/lang_altertable.html#otheralter

@Jokler Thanks for bringing this up!
TL;DR

There are some PRAGMA we need to make sure we support inside the migration statements. We likely do already.
Some PRAGMA need to be carefully executed before and after the part of the migration that’s encapsulated in a transaction. That would require backward compatible API changes.

Here is my thinking to reach this conclusion (feel free to challenge this of course).

As mentioned in this issue, foreign_keys does nothing if it is run within a transaction and statements in M::up are executed within a transaction. So putting a foreign_keys PRAGMA in M::up would be useless, at least in the current state of things.
Making Other Kinds Of Table Schema Changes
Now, let’s go back to https://www.sqlite.org/lang_altertable.html#otheralter to see how that fits with this crate. Here are the steps outlined there for the general case:

The only schema altering commands directly supported by SQLite are the "rename table", "rename column", "add column", "drop column" commands shown above. However, applications can make other arbitrary changes to the format of a table using a simple sequence of operations. The steps to make arbitrary changes to the schema design of some table X are as follows:

If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF.

Start a transaction.

Remember the format of all indexes, triggers, and views associated with table X. This information will be needed in step 8 below. One way to do this is to run a query like the following: SELECT type, sql FROM sqlite_schema WHERE tbl_name='X'.

Use CREATE TABLE to construct a new table "new_X" that is in the desired revised format of table X. Make sure that the name "new_X" does not collide with any existing table name, of course.

Transfer content from X into new_X using a statement like: INSERT INTO new_X SELECT ... FROM X.

Drop the old table X: DROP TABLE X.

Change the name of new_X to X using: ALTER TABLE new_X RENAME TO X.

Use CREATE INDEX, CREATE TRIGGER, and CREATE VIEW to reconstruct indexes, triggers, and views associated with table X. Perhaps use the old format of the triggers, indexes, and views saved from step 3 above as a guide, making changes as appropriate for the alteration.

If any views refer to table X in a way that is affected by the schema change, then drop those views using DROP VIEW and recreate them with whatever changes are necessary to accommodate the schema change using CREATE VIEW.

If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints.

Commit the transaction started in step 2.

If foreign keys constraints were originally enabled, reenable them now.

There are two cases to consider here, whether foreign_keys constraints are enforced.
With No foreign_keys Constraints
This is the simplest case, as we don’t need to set foreign_keys to ON at runtime. This also makes steps “1.” and “12.” above redundant. So our code might look something like this:
lazy_static! {
    static ref MIGRATIONS: Migrations<'static> =
        Migrations::new(vec![
            // Initial table, index, view creation
            M::up("…"),
            // Alter the table as described in
            // https://www.sqlite.org/pragma.html#pragma_foreign_keys,
            // specifically, run steps 3. to 10.
            M::up(r#"
                  CREATE TABLE new_X (…);
                  INSERT INTO new_X SELECT ... FROM X;
                  DROP TABLE X;
                  ALTER TABLE new_X RENAME TO X;
                  CREATE INDEX …;
                  CREATE VIEW …;
                  PRAGMA foreign_key_check;
                "#),
        ]);
}

fn main() {
    let mut conn = Connection::open_in_memory().unwrap();

    // Normally we would use:
    // MIGRATIONS.to_latest(&mut conn);
    // but let’s break it down into two step to explain it more easily

    // Runs the first migration
    MIGRATIONS.to_version(&mut conn, 1);

    // Runs the complex multi-step migration, which starts and commit a
    // transaction (step 2. and 11. above)
    MIGRATIONS.to_version(&mut conn, 2);
}

With foreign_keys Constraints
This is where the current API is not very ergonomic IMO. With MIGRATIONS defined as before, our code for main might look like this:
fn main() {
    let mut conn = Connection::open_in_memory().unwrap();
    // Set up foreign_keys pragma at runtime, each time the program starts
    conn.pragma_update(None, "foreign_keys", &"ON").unwrap();

    // We can’t use
    // MIGRATIONS.to_latest(&mut conn);
    // because we need manually fiddle with `foreign_keys`

    // Runs the first migration, like before
    MIGRATIONS.to_version(&mut conn, 1);

    // Runs the complex multi-step migration, which starts and commit a
    // transaction (step 2. and 11. above). But first, we need to take care of
    // step 1.
    conn.pragma_update(None, "foreign_keys", &"OFF").unwrap();
    MIGRATIONS.to_version(&mut conn, 2);
    // And that’s for step 12.
    conn.pragma_update(None, "foreign_keys", &"ON").unwrap();
}

Alternative API
Losing the simple to_latest call is a pain point, so if we changed the API to add callbacks before and after migrations, we could get something like:
lazy_static! {
    static ref MIGRATIONS: Migrations<'static> =
        Migrations::new(vec![
            // Initial table, index, view creation
            M::up("…"),
            // Alter the table as described in
            // https://www.sqlite.org/pragma.html#pragma_foreign_keys,
            // specifically, run steps 3. to 10.
            M::up(r#"
                  CREATE TABLE new_X (…);
                  INSERT INTO new_X SELECT ... FROM X;
                  DROP TABLE X;
                  ALTER TABLE new_X RENAME TO X;
                  CREATE INDEX …;
                  CREATE VIEW …;
                  PRAGMA foreign_key_check;
                "#)
            .up_before(|&mut conn| {
                conn.pragma_update(None, "foreign_keys", &"OFF").unwrap();
            })
            .up_after(|&mut conn| {
                conn.pragma_update(None, "foreign_keys", &"ON").unwrap();
            }),
        ]);
}

fn main() {
    let mut conn = Connection::open_in_memory().unwrap();
    // Set up foreign_keys pragma at runtime, each time the program starts
    conn.pragma_update(None, "foreign_keys", &"ON").unwrap();

    // Now we can use
    MIGRATIONS.to_latest(&mut conn);
}

Note on the Simpler and Faster Procedure
Later in the document, there is a simplified 9 steps migration process for simpler cases. Similarly, the library handles the transaction and schema_version increment part, and we need to make sure that using writable_schema works as expected.
Conclusions
So I think that what your comment rightly points out is that:

we need to support foreign_key_check (I think we do, but we should have a test for that, in particular for cases where it fails) and maybe other PRAGMA used in migrations,
we need to provide examples of more complex migration and perhaps clarify the doc of M::up, to say that some PRAGMA are allowed,
we likely need a more refined API to enable and disable foreign_keys (and possibly other parameters), perhaps with methods like up_before, up_after and also down_before, down_after.

Do you agree with that? Anything I’m missing?

It would definitely be important to confirm that foreign_key_check is working as it should.
I had another look at our project after reading your examples and it looks like I could get away with just turning of foreign_keys before running to_latest and turning them on again after. I believe that should be enough as long as foreign_key_check works as expected.
I think this is the most important step, having a simple and easy to use API is a good goal but as long as there is an example of how to deal with more complex cases I think users could look past a sub-optimal API too.
Does it actually matter if foreign_keys can be turned off between migrations? The sqlite documentation explains it as part of the process but I can't think of a case were turning them on again after running all migrations wouldn't be good enough as long as foreign_key_check is run within each migration.

If there is a reason to change the state of foreign_keys between migrations your proposal of adding those steps to the migrations seems fine to me though I wonder if the flexibility of having the Connection is needed or if a simple flag would be enough. If it comes down to it just providing the Connection seems like the safer bet though.
Good catch on the simpler procedure by the way, that could be enough for a lot of cases.

After some further testing I noticed that PRAGMA foreign_key_check simply returns a table of foreign key violations, the transaction does not fail if all you do is add the statement. I am currently trying to figure out how to do that.

Ah, that’s an interesting idea!

I can't figure out a way to fail the migration from within the migration so maybe that would have to be done from within rust code before the transaction is committed.

3. Does it actually matter if foreign_keys can be turned off between migrations? The sqlite documentation explains it as part of the process but I can't think of a case were turning them on again after running all migrations wouldn't be good enough as long as foreign_key_check is run within each migration.

Luckily, foreign_keys does not persist between sessions, so if it has not been called before running the migrations, we now it won’t be active, and you are right we don’t need to turn it on and off all the time between migrations.
~> sqlite3 /tmp/db
-- Loading resources from /home/cj/.sqliterc
SQLite version 3.38.5 2022-05-06 15:25:27
Enter ".help" for usage hints.
> PRAGMA foreign_keys;
foreign_keys
------------
0
Run Time: real 0.001 user 0.000055 sys 0.000056
> PRAGMA foreign_keys = ON;
Run Time: real 0.000 user 0.000024 sys 0.000024
> PRAGMA foreign_keys;
foreign_keys
------------
1
Run Time: real 0.000 user 0.000037 sys 0.000037
>

~> sqlite3 /tmp/db
-- Loading resources from /home/cj/.sqliterc
SQLite version 3.38.5 2022-05-06 15:25:27
Enter ".help" for usage hints.
> PRAGMA foreign_keys;
foreign_keys
------------
0
Run Time: real 0.000 user 0.000000 sys 0.000110

I can't figure out a way to fail the migration from within the migration so maybe the output of PRAGMA foreign_key_check would have to be checked by Rust code before the transaction is committed.

Then adding a flag like this seems the best option:
M::up("…").foreign_key_check()

I had another look at our project after reading your examples and it looks like I could get away with just turning of foreign_keys before running to_latest and turning them on again after. I believe that should be enough as long as foreign_key_check works as expected.

Given the above, that seems a way to go here.
By the way, if you are at liberty to talk about it, what is that project?

Also I’ve considered something like this to rollback the transaction if foreign_key_check fails, but unsurprisingly it does not work:
> SELECT CASE WHEN 1 THEN COMMIT TRANSACTION ELSE ROLLBACK TRANSACTION END;
Run Time: real 0.000 user 0.000020 sys 0.000020
Parse error: near "COMMIT": syntax error
  SELECT CASE WHEN 1 THEN COMMIT TRANSACTION ELSE ROLLBACK TRANSACTION END;
                          ^--- error here

As for integrity_check, I’m not sure how it’s supposed to work. Following the simpler procedure, I would expect integrity check to return an error in that case:
~> sqlite3 /tmp/db
-- Loading resources from /home/cj/.sqliterc
SQLite version 3.38.5 2022-05-06 15:25:27
Enter ".help" for usage hints.
> CREATE TABLE t(a, b);
   BEGIN TRANSACTION;
   PRAGMA writable_schema=ON;
   UPDATE sqlite_schema SET sql='CREATE TABL t(a, b);' WHERE type='table' AND name='t';
   PRAGMA writable_schema=OFF;
   PRAGMA integrity_check;
   COMMIT TRANSACTION;
integrity_check
---------------
ok

~> sqlite3 /tmp/db
-- Loading resources from /home/cj/.sqliterc
SQLite version 3.38.5 2022-05-06 15:25:27
Enter ".help" for usage hints.
> SELECT * FROM t;
Parse error: malformed database schema (t) - near "TABL": syntax error (11)
> PRAGMA integrity_check;
Parse error: malformed database schema (t) - near "TABL": syntax error (11)
> SELECT 1;
QUERY PLAN
`--SCAN CONSTANT ROW
1
-
1

If the foreign_key_check() function would do the validation I believe that would be perfect. Thanks for helping me figure out what I actually wanted. :)
The project I am working on is something my friends came up with because they wanted to build something together. It's on GitHub and I just added some screenshots to show what it looks like: https://github.com/Mavulp/hivefriends

Cool, do you plan to open a PR on this (and if so, to achive?)?
I think you raised very valid points and there are things that need improvement in the library, at least around foreign_key_check()

If the foreign_key_check() function would do the validation I believe that would be perfect.

Yeah, the plan would be to run PRAGMA foreign_key_check in Rust and use the return to validate that the migration went well.

I wasn't planning anything so far, but since we need this functionality to fix some issues I will look into it if you don't have the time/motivation to work on it currently.

I am looking into implementing this example you've made above:
M::up("…").foreign_key_check()

I am not planning on using downwards migrations, do you think enabling this should cause those to be checked too?

I am not planning on using downwards migrations, do you think enabling this should cause those to be checked too?

Probably, but we can start with the up part

Follow up points:

[ ] Add foreign keys check on downward migrations as well
[ ] Reference the new foreign_key_check method in the doc, when we talk about pragma. Sum up the discussion from #4 while we are at it.
[ ] We can probably bump the rusqlite version and remove #[allow(deprecated)].


TypeError: file must have 'read' and 'readline' attributes

When I finished processing the data it showed this error
File "/home/seivl/桌面/SmartRefine-main/datasets/argoverse_v1_dataset.py", line 79, in get
return pickle.load(self.processed_paths[idx]), pickle.load(self._p1_paths[idx])
TypeError: file must have 'read' and 'readline' attributes
/home/seivl/anaconda3/envs/smart/bin/python /home/seivl/桌面/SmartRefine-main/train.py --data_root /home/seivl/桌面/pkl_data/ori --p1_root /home/seivl/桌面/pkl_data/pkl --exp smartref_hivt_argo1 --gpus 1 --embed_dim 64 --refine_num 5 --seg_num 2 --refine_radius -1 --r_lo 2 --r_hi 10
Global seed set to 2024
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
Processing...
100%|██████████| 3/3 [00:00<00:00, 28.96it/s]
Done!
Processing...
100%|██████████| 3/3 [00:00<00:00, 12.89it/s]
Done!
Processing...
100%|██████████| 3/3 [00:00<00:00, 30.06it/s]
Done!
Processing...
100%|██████████| 3/3 [00:00<00:00, 29.99it/s]
Done!
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name           | Type                       | Params
0 | target_encoder | TargetRegion               | 216 K
1 | reg_loss       | LaplaceNLLLoss             | 0
2 | cls_loss       | SoftTargetCrossEntropyLoss | 0
3 | score_loss     | ScoreRegL1Loss             | 0
4 | minADE         | ADE                        | 0
5 | minFDE         | FDE                        | 0
6 | minMR          | MR                         | 0
216 K     Trainable params
0         Non-trainable params
216 K     Total params
0.865     Total estimated model params size (MB)
/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/torch_geometric/deprecation.py:26: UserWarning: 'data.DataLoader' is deprecated, use 'loader.DataLoader' instead
warnings.warn(out)
Validation sanity check:   0%|          | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/seivl/桌面/SmartRefine-main/train.py", line 45, in
trainer.fit(model, datamodule)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit
self._call_and_handle_interrupt(
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run
self._dispatch()
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch
self.training_type_plugin.start_training(self)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
self._results = trainer.run_stage()
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1289, in run_stage
return self._run_train()
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1311, in _run_train
self._run_sanity_check(self.lightning_module)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1375, in _run_sanity_check
self._evaluation_loop.run()
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 145, in run
self.advance(*args, **kwargs)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance
dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 140, in run
self.on_run_start(*args, **kwargs)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 86, in on_run_start
self._dataloader_iter = _update_dataloader_iter(data_fetcher, self.batch_progress.current.ready)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/loops/utilities.py", line 121, in _update_dataloader_iter
dataloader_iter = enumerate(data_fetcher, batch_idx)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 199, in iter
self.prefetching(self.prefetch_batches)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 258, in prefetching
self._fetch_next_batch()
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 300, in _fetch_next_batch
batch = next(self.dataloader_iter)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in next
data = self._next_data()
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
data.reraise()
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/seivl/anaconda3/envs/smart/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 289, in getitem
data = self.get(self.indices()[idx])
File "/home/seivl/桌面/SmartRefine-main/datasets/argoverse_v1_dataset.py", line 79, in get
return pickle.load(self.processed_paths[idx]), pickle.load(self._p1_paths[idx])
TypeError: file must have 'read' and 'readline' attributes

could you please help me ?

It seems like a pickle issue.
Maybe you can try to load .pkl files separately and see if this works.

Also, I've added my email address in README. Feel free to contact me.
( And a little tip: it's not proper to email professors directly for code problems. You can issue me on GitHub first. 😄

原谅我的唐突 我这边发邮件时没有加载出作者照片
感谢您 非常好的工作
谢谢！
------------------ Original ------------------
From: kmzy @.>
Date: Wed,Jun 5,2024 5:08 PM
To: opendilab/SmartRefine @.>
Cc: Lukas88664 @.>, Author @.>
Subject: Re: [opendilab/SmartRefine] TypeError: file must have 'read' and'readline' attributes (Issue #7)
Also, I've added my email address in README. Feel free to contact me.
( And a little tip: it's not proper to email professors directly for code problems. You can issue me on GitHub first. 😄
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Hi, I've updated the code and installation in https://github.com/opendilab/SmartRefine/?tab=readme-ov-file#getting-started.
Please use the newest code and read the doc to take a try.
Sorry for these problems. 😢


[Feature proposal] Alternative select_multiple

Hi Guys
I congratulate you on tabacitu's work
I put my code on the FIELD TYPES I hope I can serve someone
this is a multiselect version select_multiple (n-n relationship)
Below you find a picture of the result obtained

------------------------------- UserCrudController -------------------------------
class UserCrudController extends OriginalUserCrudController
{
public function setup()
{
parent::setup();
	$this->crud->addColumn([
        'label' => 'Categorie',
        'type' => 'select_multiple',
        'name' => 'categorie',
        'entity' => 'categorie',
        'attribute' => 'nome',
        'model' => "App\Models\Categorie",
	]);

	$this->crud->addField([
    		'label'     => 'Categorie',
            'type'      => 'select3_multiple',
            'name'      => 'categorie',
            'entity'    => 'categorie',
            'attribute' => 'nome',
            'model'     => "App\Models\Categorie",
            'pivot'     => true,
            'number_columns'   => 3, //can be 1,2,3,4,6
    ]);

	$this->crud->removeField('user_role_permission', 'update/create/both');
	$this->crud->addField([
        // two interconnected entities
        'label'             => trans('backpack::permissionmanager.user_role_permission'),
        'field_unique_name' => 'user_role_permission',
        'type'              => 'select3_multiple_dependency',
        'name'              => 'roles_and_permissions', // the methods that defines the relationship in your Model
        'subfields'         => [
                'primary' => [
                    'label'            => trans('backpack::permissionmanager.roles'),
                    'name'             => 'roles', // the method that defines the relationship in your Model
                    'entity'           => 'roles', // the method that defines the relationship in your Model
                    'entity_secondary' => 'permissions', // the method that defines the relationship in your Model
                    'attribute'        => 'name', // foreign key attribute that is shown to user
                    'model'            => config('laravel-permission.models.role'), // foreign key model
                    'pivot'            => true, // on create&update, do you need to add/delete pivot table entries?]
                    //'number_columns'   => 3, //can be 1,2,3,4,6
                ],
                'secondary' => [
                    'label'          => ucfirst(trans('backpack::permissionmanager.permission_singular')),
                    'name'           => 'permissions', // the method that defines the relationship in your Model
                    'entity'         => 'permissions', // the method that defines the relationship in your Model
                    'entity_primary' => 'roles', // the method that defines the relationship in your Model
                    'attribute'      => 'name', // foreign key attribute that is shown to user
                    'model'          => "Backpack\PermissionManager\app\Models\Permission", // foreign key model
                    'pivot'          => true, // on create&update, do you need to add/delete pivot table entries?]
                ],
            ],
    ]);

}

}

project_name\resources\views\vendor\backpack\crud\columns\select3.blade.php - (select3.blade1.php)

project_name\resources\views\vendor\backpack\crud\fields\select3.blade.php -

project_name\resources\views\vendor\backpack\crud\fields\select3_multiple.blade.php -

project_name\resources\views\vendor\backpack\crud\fields\select3_multiple_dependency.blade.php -

project_name\public\js\duallistbox.js -

project_name\public\css\duallistbox.css -

multi_select.zip

Hi @xdasty ,
This looks like some great work.
The interface is a bit crowded for my taste, I prefer select2's tinier visual footprint, but it's definitely a matter of personal preference and I bet you'll save some people a lot of time.
I propose we publish this on the Backpack blog, in your name. That way others will be able to reuse your code. I plan on putting a lot more snippets there, for developer to easily use. What do you say? If you agree, please:

write a small introduction and an example usage (like above) in markdown, wherever you want; if you need to put CSS and JS (multiple files) a good place to store all the files would be a Github gist;
feel free to promote yourself, your services, or your company in the article, and link to your website/github/linkedin/etc; you never know when one link turns into $$$ :-)
register on BackpackForLaravel.com, so I can post the article in your name;
send me an email with the article link at<EMAIL_ADDRESS>and I'll post it for you;

Thanks, cheers!

Thank you @tabacitu , it is an honor to hear these words, and I will continue to contribute.


ChooseVideo

When choosing videopicker and use camera to choose video,when choose "right" check causing crash  problem,hope to get the solution soon

I also get the same problem,and it's hurts me a lot ,is there any one can get the problem resolved!

I found when i choose the video,causing crash problemThe reason is that  the bitmap is null ,I modify the getVideoThumb() of the FileLoaderCallbacks ,I change "bitmap = retriever.getFrameAtTime(TimeUnit.MILLISECONDS.toMicros(1));"  to "bitmap = ThumbnailUtils.createVideoThumbnail(filePath, MediaStore.Images.Thumbnails.MINI_KIND)"   ,I don't know if it happened by accident. The problem has been solved...

@ydBen @lizihanglove @xxwn446
Please try version 1.0.5.
And tell me the result whether this exception exist any more?

I tried,and the problem has been solved.@fishwjy

OK, thanks! I'll close this issue.


Clean up after installation

Remove all the folders used to install Doxygen after its installation, cleaning up the workspace.
Added a test to ensure that the cleanup was successful.

[ closes #2 ]

Thanks a lot. I have not yet come across a scenario where this was relevant, but it is definitely an improvement over the status quo. Also thanks for adding a testing bit to the CI.


There are unused functions in chart.js dependencies?

Hi all,
I am researching about the deadcode in js dependencies. I found that in most project there are unused functions in dependencies that are being included in the final bundle. In the case of chart.js (Version 2.1.4) my tools detected approximately 300 unused function in the dependencies "moment" and "chartjs-color". Removing those
functions, the size of chart.js bundled could be reduced at least 14% (All tests passed). I replaced the bundled in several projects that use chart.js (Version 2.x.x) as "tc-angular-chartjs" and "react-perf-tool".
However, i think that chart.js is one of the best maintained libraries in the web. This has made me wonder, my analysis is wrong and the unused functions are false positives? if I remove this code I could have any problem in the future?
I attached a list of unused functions detected in chart.js version 2.1.4 (note - an empty functionName indicate an anonymous function). Additionally, i attached the modified bundle.
Thanks
unusedFunctions.txt
ChartBundle.txt

I'm not sure about the stuff in 'moment' since that's not our library. It may not be used right now, but we can't guarantee it.
In terms of the color stuff, @simonbrunel have you taken a look?

@hcvazquez, concerning the color library, reported functions are indeed not explicitly called by Chart.js but that doesn't mean they are useless. chartjs-color allows to manipulate different color spaces, but you are likely not using all of them in the same project. For example, if you run your profiling tool against the Chart.js sources, samples and unit tests, it will report unused all cmjk related methods because cmyk() notation is never used in that code, but other projects may use it. Since chartjs-color is used for that purpose (flexibility for manipulating colors), we can't (don't want to) remove them.
About moment.js, we should try to make this an optional dependency (or maybe move that code in a plugin).

Closing since we aren't going to change moment or color


showimage: Restore automation-friendly behaviour of -quit

I added the -quit option for simple automated testing (with hindsight,
perhaps "-quit-after-loading" would have been a better name). The
intention was that something like
showimage -quit -save out.bmp in.jpg -save out2.bmp in.png

would go through the motions of loading each image, save it to the
corresponding output file, but not actually wait for input or rendering.

I've started trying to learn the SDL_test framework well enough to add an automated test with better coverage, but that PR isn't ready yet, and probably shouldn't be considered a blocker for 2.5.1.

I added a -timeout argument to showimage to allow testing various formats on CI.
https://github.com/madebr/SDL_image/commit/a7a20749962eef613546d1513d5e860a9c4f7953
In the commit, I added it as a separate step in github CI, but it can be integrated into ctest/SDL_test.


Add shoulda matchers o p

Related #3155

Argh, sorry @todtb, I "shoulda" merged this before the ProjectInvitiation PR. Could you resolve the conflict here?

Argh, sorry @todtb, I "shoulda" merged this before the ProjectInvitiation PR. Could you resolve the conflict here?

All set! I left a comment that you might have a real quick answer to, but not a blocker since the specs are marked pending.


Feature/kitti2bag

Status
DEVELOPMENT
Description
Added the program kitti2bag tool, to convert KITTI dataset files into BAG files, into ros/src/util/packages/kitti_pkg as a submodule.
And extra option odom_velo was added to support creation of pointcloud only BAG files from the KITTI odometry/SLAM evaluation dataset (http://www.cvlibs.net/datasets/kitti/eval_odometry.php).
Related PRs
None
Todos

[X] Tests
[X] Documentation

Steps to Test or Reproduce

Clone branch feature/kitti2bag and build Autoware the conventional way.
Read the instructions and follow the example on ros/src/util/packages/kitti_pkg/kitti2bag/README.md
To test the new option odom_velo make sure to download the Velodyne dataset for odometry evaluation (http://www.cvlibs.net/download.php?file=data_odometry_velodyne.zip).
For better results (at least for sequences 11 and over) use pykitti from 'utiasSTARS/pykitti#39'.
For any sequence number 'XX' (where XX=00,01,...,10 for training and XX=11,12,...,21 for evaluation) run the following command:

python kitti2bag -s XX odom_velo <VELODYNE-DATASET-FOLDER>

wherer VELODYNE-DATASET-FOLDER is the folder where all sequences are located, normally its name is dataset and includes at least one subfolder called sequences.

The content of this PR are not entirely covered by the PR's title or description. The sub-module part is no longer valid given the repository reorganisation. Please file a new PR from a fork for any of the other changes you want to keep.


codeql version bump

Which issue this PR addresses:
updated to v2 and changed the concurrency on the extraction. This slows down the process a little but it avoid the errors.
This was recommended by someone from codeQL here
What this PR does / why we need it:
update codeql to v2 because it is being deprecated
Test plan for issue:
https://github.com/Azure/ARO-RP/actions/runs/2733706014
Ran it multiple times to see if it was working.

This seems to be a lot more stable now with the new env variable.


Replace for..in with standard for in L.Control.Layers

Fixes #4508
Leftover from #4227

LGTM.

Retested with the fix, works fine now :+1:


Return 301 (permanent), not 307 (temporary) redirect http resp

FYI I'll admin merge this once tests pass.


Re-running migration produces "invalid version" error

🐞 Bug report
Command (mark with an x)

- [ ] new
- [ ] build
- [ ] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [x] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Is this a regression?
No idea.
Description
If the migration from v7 to v8 fails, the CLI prints the following:
Migration can be rerun with: "ng update @angular/core --from 7 --to 8 --migrate-only"

Trying to run this command after fixing whatever was wrong does not work.
🔥 Exception or Error
Using package manager: 'npm'
Collecting installed dependencies...
Found 77 dependencies.
Invalid Version: 7

🌍 Your Environment
Angular CLI: 8.0.0
Node: 11.10.1
OS: darwin x64
Angular: 8.0.0
... animations, cdk, cli, common, compiler, compiler-cli, core
... forms, language-service, platform-browser
... platform-browser-dynamic, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.800.0
@angular-devkit/build-angular     0.800.0
@angular-devkit/build-optimizer   0.800.0
@angular-devkit/build-webpack     0.800.0
@angular-devkit/core              8.0.0
@angular-devkit/schematics        8.0.0
@ngtools/webpack                  8.0.0
@nguniversal/express-engine       7.1.1
@schematics/angular               8.0.0
@schematics/update                0.800.0
rxjs                              6.5.2
typescript                        3.4.5
webpack                           4.30.0

Anything else relevant?

Hi, can you please provide the original command that you used to migrate?

ng update --all --force

The original issue seems to be that the from and to versions should be full semver versions example 7.0.0 and 8.0.0.
Ie ng update @angular/core --from 7.0.0 --to 8.0.0 --migrate-only instead of ng update @angular/core --from 7 --to 8 --migrate-only
We should probably handle both.

Increasing the priority as in the @angular/core migration schematics if it fail its will print Migration can be rerun with: "ng update @angular/core --from 7 --to 8 --migrate-only"

We seem to have some problems running migration on Windows (https://github.com/angular/angular-cli/issues/14565), together with migrations for @angular/core not running when you are already on a beta or rc (https://github.com/angular/angular-cli/issues/14553).
Here is a workaround for now:

use 8.0.0 in the global install: npm i -g<EMAIL_ADDRESS>locally use CLI 8.0.0-rc.0: npm install<EMAIL_ADDRESS>--save-exact --save-dev
commit to git
try to update again, the update should work now ng update @angular/core --from 7 --to 8 --migrate-only
go back to CLI 8.0.0 npm install<EMAIL_ADDRESS>--save-dev

Another report of the invalid version error when using —-from 7 https://github.com/angular/angular-cli/issues/14561#issuecomment-496942757

@filipesilva I don't think this is related to what you mentioned above, this issue is not Windows.
The problem here is that ng update @angular/core --from 7 --to 8 --migrate-only errors out while  ng update @angular/core --from 7.0.0 --to 8.0.0 --migrate-only works as expected.
/workspace/viewChild-test-ng8 $ yarn ng update @angular/core --from 7 --to 8 --migrate-only
yarn run v1.16.0
$ ng update @angular/core --from 7 --to 8 --migrate-only
Using package manager: 'npm'
Collecting installed dependencies...
Found 30 dependencies.
Invalid Version: 7

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

/workspace/viewChild-test-ng8 $ yarn ng update @angular/core --from 7.0.0 --to 8.0.0 --migrate-only
yarn run v1.16.0
$ ng update @angular/core --from 7.0.0 --to 8.0.0 --migrate-only
Using package manager: 'npm'
Collecting installed dependencies...
Found 30 dependencies.
    ** Executing migrations for package '@angular/core' **

We should note that this is indeed a regression as it previously worked with pre-V8 CLI versions but broke accidentally with V8 as there was some logic refactoring for the --migrate-only flag (see https://github.com/angular/angular-cli/commit/b7fc6125e536c9a94250b901ed925a6cb92dae76).
Previously the --from and --to options were working that way since they followed the Semver rules. See:
https://github.com/angular/angular-cli/blob/37debd7105eb499adf8268ade4ad8b33f6153f24/packages/schematics/update/update/index.ts#L809-L810

Am updating the angular 7 to 8, after updating the ng update @angular/cli @angular/core command updated successfully,
but after updating the material through ng update @angular/material entered, then i got the error.
Repository is not clean. Please commit or stash any changes before updating.

Am updating the angular 7 to 8, after updating the ng update @angular/cli @angular/core command updated successfully,
but after updating the material through ng update @angular/material entered, then i got the error.
Repository is not clean. Please commit or stash any changes before updating.

1st you need to commit the changes done from ng update @angular/cli @angular/core and then you can trye updating angular material.


Role driven permissions structure

Hi,
We have a tabular model on Azure SSAS and we are generating PowerBI reports based on that model. We want to build a dynamic steering of the access of cube data based on which user logs in. By that I mean that depending on which report the user runs that a different slice of the cube is performed(with the help of dax filters). I have created different roles in the cube but I noticed that because it is the effective name that is used to pull the data from the cube, the rights of the two different roles are combined when creating the power bi report. I decided then to create two different Azure AD groups(email security) and use the groups to differentiate the data pull from the cube. The group emails were added to both power bi app and the cube roles. The result however was again the same. When one powerbi report is created, mapped to only one of the Azure AD groups and respective cube roles, data from both roles where the user belongs to are still defining the cube slice. Is there any way for one user to get different slices of a cube by being assigned two different roles(with different dax filters) in the cube itself?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3f0e1195-b6a2-d789-a950-4454798f265e
Version Independent ID: d8ea7415-3b97-5985-44e7-1b89bf1196b6
Content: Manage database roles and users in Azure Analysis Services
Content Source: articles/analysis-services/analysis-services-database-users.md
Service: azure-analysis-services
GitHub Login: @Minewiskan
Microsoft Alias: owend

@24hector Thank you for your interest in Azure products and services. We are investigating and will get back to you soon.

Thank you for your assistance. To make things a bit more clear the setup we have is the following:
We have a cube and I have created two roles on this cube that slice it in different ways based on the DAX row filters created. I want for the same user to get two different PowerBI reports each mapped to one of the two roles that provides different slice of data. For that reason I created two different Azure AD Email enabled security groups where the user is a member of and assigned these groups instead of the user itself to each of the PowerBI reports and the cube roles. My hope was that the AzureAD groups would be carried with the username of the user running the reports and would help activate the correct role in the cube. What happens though is that both roles rights are active on whichever report is run on PowerBI.
An obvious solution would be to create two different cubes but I was wondering if there is a way to achive the same result without creating another cube.

@Minewiskan Could you please look into this issue?

@CHEEKATLAPRADEEP-MSFT - I do not see any documentation issue here. This is a product question and should be directed to the support or forum channels.
#please-close


[128] - STT Functionality

Reviewer: @suzinyou
Estimate: 30 mins

Ticket
Fixes: JIRA_TICKET_LINK
Description
This PR enhances the AAQ pipeline by incorporating Speech-to-Text (STT) functionality using the Whisper open-source model, housed in a separate Docker container.
Goal
The main goal of this PR is to provide comprehensive speech functionality for AAQ users, enabling them to send voice queries and receive responses in either audio or text format based on their preferences.
Changes

Introduced a separate Docker container for the Whisper model.
Updated the docker-compose.yml file to include the new container.
Added a new endpoint, stt-llm-response , in the question_answer directory.

Future Tasks (optional)
Add Bhashini ASR in the speech_service container for indian languages
How has this been tested?

docker compose
development environment
pytests

How to test the functionality

update the .env file with the new environment variables which are specified in the template.env
Use the command  docker compose -f docker-compose.yml -f docker-compose.dev.yml -p aaq-stack watch to build the containers once again
Here is a loom video on how to test the api endpoint out

Checklist
Fill with x for completed.

[x] My code follows the style guidelines of this project
[x] I have reviewed my own code to ensure good quality
[x] I have tested the functionality of my code to ensure it works as intended
[x] I have resolved merge conflicts
[x] I have updated the automated tests
[x] I have updated the requirements

Yes I did look for existing whisper images, and no doubt they would be more optimised to run whisper than the image which I created, but i was holding onto that thought cause our speech service will also host other models too like Bhashini, then the TTS specific models which we will have in sometime so wouldn't that be an issue if our container image is only optimised for whisper?

I think we are at a very good place! A few clean ups / refactoring to do but otherwise can we try to merge this to main branch instead of just to the feature branch?

So directly merge this to main or first merge this to feature and then merge that branch to main? and yes now that we have a complete workflow i will start directly merging it to main itself!

I think we are at a very good place! A few clean ups / refactoring to do but otherwise can we try to merge this to main branch instead of just to the feature branch?

So directly merge this to main or first merge this to feature and then merge that branch to main? and yes now that we have a complete workflow i will start directly merging it to main itself!

First merge to feature branch, update the feature branch with main (merge main to feature) and that can be a separate PR also!