Publisher.cycleCamera() crashes

Hi, I have a problem of camera toggle.
I provide a button to toggle camera. When user was publishing and clicked the toggle camera button, app crashes.
below is my code:
toggleButton.setOnClickListener(v -> mPublisher.cycleCamera());

and the crash logs:
Fatal Exception: java.lang.RuntimeException: Fail to connect to camera service
       at android.hardware.Camera.native_setup(Camera.java)
       at android.hardware.Camera.<init>(Camera.java:374)
       at android.hardware.Camera.open(Camera.java:329)
       at com.opentok.android.DefaultVideoCapturer.swapCamera(DefaultVideoCapturer.java:298)
       at com.opentok.android.DefaultVideoCapturer.cycleCamera(DefaultVideoCapturer.java:278)
       at com.opentok.android.Publisher.cycleCamera(Publisher.java:518)
       at com.fancred.android.live.LivePublisherActivity.toggleCamera(LivePublisherActivity.java:299)
       at com.fancred.android.live.LivePublisherActivity.onClick(LivePublisherActivity.java:307)
       at android.view.View.performClick(View.java:4438)
       at android.view.View$PerformClick.run(View.java:18441)
       at android.os.Handler.handleCallback(Handler.java:733)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:136)
       at android.app.ActivityThread.main(ActivityThread.java:5111)
       at java.lang.reflect.Method.invokeNative(Method.java)
       at java.lang.reflect.Method.invoke(Method.java:515)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:786)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:602)
       at dalvik.system.NativeStart.main(NativeStart.java)

sdk version is 2.11.0, my phone has two cameras
hope someone help me, thanks

Hi @lby1992,
What device model are you using?

hi, @marinaserranomontes, this occur on each my device, I think it's caused by the incorrect code.
I have adjusted my code, now it seems working fine:
 BaseVideoCapturer capturer = livePublisher.getCapturer();
        if (capturer.isCaptureStarted()) {
            capturer.stopCapture();
        }
        capturer.destroy();
        livePublisher.cycleCamera();
        capturer.init();
        capturer.startCapture();

If it's the correct way to toggle camera

hi @lby1992 ,
You don't need to manage the capturer if you want to cycle the camera. If the publisher exists, you should only call the publisher.cycleCamera() and it should work.
You can find an example here: https://github.com/opentok/opentok-android-sdk-samples/blob/bdec9e5d14549134c75b28d89f500d921f08d53d/Simple-Multiparty/app/src/main/java/com/tokbox/android/tutorials/simple_multiparty/MainActivity.java#L60

@marinaserranomontes I just called the same as sample code before, but it crashed. I search on Google with the exception message, and then manage the capturer by myself.

@lby1992 just to reproduce and debug it, please, could you detail the Android versions and device models you are using, please? Thanks!!

hi @marinaserranomontes , below models I am using:
1. ZTE N958St, api 19
2.Emulator Nexus_5x_API_24

I am looking for more device to try this

Hi @lby1992 ,
I am not able to reproduce it with the emulator and API 24.
We are working on the possible camera swap exceptions for the next release.
Thanks for your feedback.

@marinaserranomontes Ok, thanks for your help.


Fix: isRequired radio component support

📝 Description
Support for isRequired parameter in Radio.Group
⛳ Current behavior (updates)
Currently isRequired does not work as intended, as it does not create "required" props for inner  radio tags
🚀 New behavior
Adds expected behavior for isRequired prop.
💣 Is this a breaking change (Yes/No):
No

Codecov Report

Merging #638 (989e97b) into main (c64c96a) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #638      +/-   ##
==========================================
+ Coverage   74.82%   74.83%   +0.01%
==========================================
  Files         200      200
  Lines        3098     3100       +2
  Branches      956      957       +1
==========================================
+ Hits         2318     2320       +2
  Misses        765      765
  Partials       15       15

Impacted Files
Coverage Δ

packages/react/src/radio/radio.tsx
100.00% <ø> (ø)

packages/react/src/radio/use-radio.ts
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c48528...989e97b. Read the comment docs.

Hey @richtone I appreciate your contribution to the project 🙏🏻, however, we decided to implement this by ourselves, apologies


FROM fixed centos:7.5.1804 or just centos:7 ?

@rhuss while looking more closely at #165 than I had before (because of #171), I was just wondering why you chose FROM centos:7.5.1804 (fixed version from that month) instead of just using centos:7 ? Shall we change this?

OK sure let's keep it as-is then, I can see and agree with the stability point, no worries.
Do we already have a RELEASE.md doc describing who we release this?
Let's just add a section to bump the centos base image to latest in doc.

no, we don't have one as well as we don't have a CHANGELOG.md. Both would be a good idea to have.

@vorburger suggested I chime in here, since he was asking in the #centos IRC channel.
Pinning your centos version in the FROM section of a Dockerfile, then running 'yum install ...' isn't going to get you packages from anything except the latest release of CentOS, so you might as well use FROM centos:7.  There are not separate yum repositories for 7.5.1804 than 7.6.1810, so all you are doing is forcing your container to have an out-of-date image that most likely will pull in more packages when you use 'yum install'.  It won't make your build any more reproducible.

@rhuss #206 has a longer write up about what I learnt about this kind of stuff today.

I really would like to pin it so that a respin of the release produces the same image (and least for the base image). There might be different spots which hinder such a reproducible build (like unpinned package updates).

My understanding, based on what I learnt on #centos today (see https://gist.github.com/vorburger/ea367a76f69aafb8260c62f6bedf2788 and summary in #206) is that this is futile and won't really work anyway, because, at least when there are events like today's release of the CentOS 7.6 image, and on future updates, there apparently basically no longer really is e.g. a 7.5 and its RPM packages, anyway (unless we want to get really creative and do serious gymnastics involving vault.centos.org).
I'm therefore going to change the FROM now, and also make it do a yum update (which is NOT the cause of the underlying problem; that's just a good idea while we are at cleaning this sort of thing up now).

Do we already have a RELEASE.md doc describing who we release this?
no, we don't have one as well as we don't have a CHANGELOG.md. Both would be a good idea to have.

I'll do that in #208.


tracing: use weak references when caching HttpTracers inside HttpTracerManager

Description: use weak references when caching HttpTracers inside HttpTracerManager
Risk Level: Medium
Testing: unit tests
Docs Changes: N/A
Release Notes: N/A
Context:

part of #9998

Sorry can you merge master? It will fix tidy.
/wait


Slopes

Is your feature request related to a problem? Please describe.
Not really related to a problem, not sure how slopes work in 8-bit games either but I think they would probably come in handy for certain Gameboy Studio games, specifically ones that are built around speed like a sonic the hedgehog clone.
Describe the solution you'd like
adding slopes to the collision paint, like red triangles
Describe alternatives you've considered
Don't think there are any alternatives.
Additional context

I think slopes are a good idea

There should be at least 45deg slopes. Better also 22.5deg slopes.


candy still binding  on connection....

I use candy js with mongooseIM as an XMPP server.
I add a room to my server as following:
{mod_muc, [
{host, "conference.localhost"},
{access, all},
{access_create, muc_create}
]},
I add this room to my index.html as following:
//<EMAIL_ADDRESS>//     -> array of rooms to join after connecting
autojoin: ['conference.localhost']
but when I try to connect I had this log under and the window still binding on "Connecting ..."
This is my log:
[]([Init] Debugging enabled
candy.bundle.js:202 [Connection] Connecting
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 1 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 1.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 1.0 state changed to 1
candy.bundle.js:202 SENT:
candy.bundle.js:234 [Strophe][DEBUG]: request id 1.1 state changed to 2
candy.bundle.js:234 [Strophe][DEBUG]: request id 1.1 state changed to 3
candy.bundle.js:234 [Strophe][DEBUG]: request id 1.1 state changed to 4
candy.bundle.js:234 [Strophe][DEBUG]: removing request
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 1 should now be removed
candy.bundle.js:234 [Strophe][DEBUG]: request id 1.1 got 200
candy.bundle.js:234 [Strophe][INFO]: _connect_cb was called
candy.bundle.js:202 RECV: stream:featuresX-OAUTHDIGEST-MD5SCRAM-SHA-1PLAIN</stream:features>
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 2.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 2.0 state changed to 1
candy.bundle.js:202 SENT: biwsbj1hZG1pbixyPWQ0MWQ4Y2Q5OGYwMGIyMDRlOTgwMDk5OGVjZjg0Mjdl
candy.bundle.js:234 [Strophe][DEBUG]: request id 2.1 state changed to 2
candy.bundle.js:234 [Strophe][DEBUG]: request id 2.1 state changed to 3
candy.bundle.js:234 [Strophe][DEBUG]: request id 2.1 state changed to 4
candy.bundle.js:234 [Strophe][DEBUG]: removing request
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 2 should now be removed
candy.bundle.js:234 [Strophe][DEBUG]: request id 2.1 got 200
candy.bundle.js:234 [Strophe][INFO]: _dataRecv called
candy.bundle.js:202 RECV: cj1kNDFkOGNkOThmMDBiMjA0ZTk4MDA5OThlY2Y4NDI3ZTdwcXZLZ25weFdlSy9SdGp3V281aFE9PSxzPW9IR0xBM3hYY2ZpU2tjRWRwalNwUWc9PSxpPTQwOTY=
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 3.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 3.0 state changed to 1
candy.bundle.js:202 SENT: Yz1iaXdzLHI9ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U3cHF2S2ducHhXZUsvUnRqd1dvNWhRPT0scD1hdFNySGVSeWpVMGdFZUQwN0ozSWt6aWpLSE09
candy.bundle.js:234 [Strophe][DEBUG]: request id 3.1 state changed to 2
candy.bundle.js:234 [Strophe][DEBUG]: request id 3.1 state changed to 3
candy.bundle.js:234 [Strophe][DEBUG]: request id 3.1 state changed to 4
candy.bundle.js:234 [Strophe][DEBUG]: removing request
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 3 should now be removed
candy.bundle.js:234 [Strophe][DEBUG]: request id 3.1 got 200
candy.bundle.js:234 [Strophe][INFO]: _dataRecv called
candy.bundle.js:202 RECV: dj0xc2M2dzh6VHkvTjQ4WisrYVVmL05OazFHT3c9
candy.bundle.js:234 [Strophe][INFO]: SASL authentication succeeded.
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 4.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 4.0 state changed to 1
candy.bundle.js:202 SENT:
candy.bundle.js:234 [Strophe][DEBUG]: request id 4.1 state changed to 2
candy.bundle.js:234 [Strophe][DEBUG]: request id 4.1 state changed to 3
candy.bundle.js:234 [Strophe][DEBUG]: request id 4.1 state changed to 4
candy.bundle.js:234 [Strophe][DEBUG]: removing request
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 4 should now be removed
candy.bundle.js:234 [Strophe][DEBUG]: request id 4.1 got 200
candy.bundle.js:234 [Strophe][INFO]: _dataRecv called
candy.bundle.js:202 RECV: stream:features</stream:features>
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 5.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 5.0 state changed to 1
candy.bundle.js:202 SENT: Candy
candy.bundle.js:234 [Strophe][DEBUG]: request id 5.1 state changed to 2
candy.bundle.js:234 [Strophe][DEBUG]: request id 5.1 state changed to 3
candy.bundle.js:234 [Strophe][DEBUG]: request id 5.1 state changed to 4
candy.bundle.js:234 [Strophe][DEBUG]: removing request
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 5 should now be removed
candy.bundle.js:234 [Strophe][DEBUG]: request id 5.1 got 200
candy.bundle.js:234 [Strophe][INFO]: _dataRecv called
candy.bundle.js:202 RECV: admin@localhost/Candy
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 6.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 6.0 state changed to 1
candy.bundle.js:202 SENT:
candy.bundle.js:234 [Strophe][DEBUG]: request id 6.1 state changed to 2
candy.bundle.js:234 [Strophe][DEBUG]: request id 6.1 state changed to 3
candy.bundle.js:234 [Strophe][DEBUG]: request id 6.1 state changed to 4
candy.bundle.js:234 [Strophe][DEBUG]: removing request
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 6 should now be removed
candy.bundle.js:234 [Strophe][DEBUG]: request id 6.1 got 200
candy.bundle.js:234 [Strophe][INFO]: _dataRecv called
candy.bundle.js:202 RECV:
candy.bundle.js:202 [Connection] Connected
candy.bundle.js:202 [Connection] Attached
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][ERROR]: User connection callback caused an exception: TypeError: Cannot read property 'replace' of nullk.debug.b.log @ candy.bundle.js:234error @ libs.min.js:1_changeConnectStatus @ libs.min.js:1_sasl_session_cb @ libs.min.js:1run @ libs.min.js:1(anonymous function) @ libs.min.js:1forEachChild @ libs.min.js:1_dataRecv @ libs.min.js:1_onRequestStateChange @ libs.min.js:2
candy.bundle.js:234 [Strophe][DEBUG]: request id 7.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 7.0 state changed to 1
candy.bundle.js:202 SENT:
candy.bundle.js:234 [Strophe][DEBUG]: request id 7.1 state changed to 2
candy.bundle.js:234 [Strophe][DEBUG]: request id 7.1 state changed to 3
candy.bundle.js:234 [Strophe][DEBUG]: request id 7.1 state changed to 4
candy.bundle.js:234 [Strophe][DEBUG]: removing request
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 7 should now be removed
candy.bundle.js:234 [Strophe][DEBUG]: request id 7.1 got 200
candy.bundle.js:234 [Strophe][INFO]: _dataRecv called
candy.bundle.js:202 RECV:
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 8.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 8.0 state changed to 1
candy.bundle.js:202 SENT: 1
candy.bundle.js:234 [Strophe][DEBUG]: request id 8.1 state changed to 2
candy.bundle.js:234 [Strophe][DEBUG]: request id 8.1 state changed to 3
candy.bundle.js:234 [Strophe][DEBUG]: request id 8.1 state changed to 4
candy.bundle.js:234 [Strophe][DEBUG]: removing request
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 8 should now be removed
candy.bundle.js:234 [Strophe][DEBUG]: request id 8.1 got 200
candy.bundle.js:234 [Strophe][INFO]: _dataRecv called
candy.bundle.js:202 RECV:
candy.bundle.js:234 [Strophe][INFO]: no requests during idle cycle, sending blank request
candy.bundle.js:234 [Strophe][DEBUG]: request id 9.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 9.0 state changed to 1
candy.bundle.js:202 SENT:
candy.bundle.js:234 [Strophe][DEBUG]: request id 9.1 state changed to 2
candy.bundle.js:234 [Strophe][DEBUG]: request id 9.1 state changed to 3
candy.bundle.js:234 [Strophe][DEBUG]: request id 9.1 state changed to 4
candy.bundle.js:234 [Strophe][DEBUG]: removing request
candy.bundle.js:234 [Strophe][DEBUG]: _throttledRequestHandler called with 0 requests
candy.bundle.js:234 [Strophe][DEBUG]: request id 9 should now be removed
candy.bundle.js:234 [Strophe][DEBUG]: request id 9.1 got 200
candy.bundle.js:234 [Strophe][INFO]: _dataRecv called
candy.bundle.js:202 RECV: 1
candy.bundle.js:202 [Jabber] Presence
candy.bundle.js:234 [Strophe][INFO]: no requests during idle cycle, sending blank request
candy.bundle.js:234 [Strophe][DEBUG]: request id 10.0 posting
candy.bundle.js:234 [Strophe][DEBUG]: request id 10.0 state changed to 1
candy.bundle.js:202 SENT:
Why my app still binding on Conncting ... and does not pass to other view ??

I think that the source of bug is this error:
candy.bundle.js:234 [Strophe][ERROR]: User connection callback caused an exception: TypeError: Cannot read property 'replace' of nullk.debug.b.log @ candy.bundle.js:234error @
but I don't know how to fix it !!!

This is a JID for a room with your config<EMAIL_ADDRESS>What you specified is: conference.localhost
You specified a wrong argument to autojoin. What you configured in the config is the muc host which is
kind of like a domain for all MUC rooms.
Then you set this host as the autojoin argument, when in fact you'd need to specify an actual room. So first create a room to autojoin and then in the config specify it as documented.

@mweibel Thank you for this indication, but, I did not found the command line in mongooseIM that create a muc room !!!!


Kubernetes Running Container Scanning

Following design #527 we should implement scanning running containers that were discovered by the provider. For now the simplest approach that avoids any requirement for persistent storage will be chosen so that the most Kubernetes cluster as possible can be supported.
A Kubernetes job should be started which is nodeSelected to the same node as the target container, it should mount the container runtime socket from the host in order to operate directly on the containers.
Once started the job should commit and then save the container's image to a tar using the correct container runtime's "save" command. For example docker save or nerdctl save.
If the image is not in the OCI format it should be converted, for example:
skopeo copy docker-archive://image.tar oci:image:latest

Once converted it can be unpacked into a rootfs directory using umoci raw unpack https://github.com/opencontainers/umoci https://github.com/opencontainers/umoci/blob/main/doc/man/umoci-raw-unpack.1.md#name
Once unpacked it can be passed into the VMClarity CLI the same as any other rootfs scanned with the families and the results returned to the API server. If a family can support being passed the image.tar then we should pass that in too to ensure that results like the SBOM contain the layer information.

#864


[BUG] Restore with hardlinks fails in IMPORT phase

Describe the bug
I'm trying to restore a created snapshot which fails with error:
Error{source=cassandra-dev01-ird, message=Hardlinking phase finished with errors, the linking of downloaded SSTables to Cassandra directory has failed., throwable=com.instaclustr.esop.impl.restore.RestorationPhase$RestorationPhaseException: Unable to pass IMPORT phase.}
To Reproduce
make backup
curl --header "Content-Type: application/json" --data '{"type":"backup", "globalRequest":"true", "storageLocation" : "ceph://cassandra-icarus2-ird-backup-dev/cassandra_gms_dev/rc3/1", "metadataDirective":"REPLACE", "dataDirs":["/icarus/cassandra/data/data"], "skipRefreshing":"true"}' cassandra-dev00-ird:4567/operations
restore backup
curl --header "Content-Type: application/json" --data '{"type":"restore", "globalRequest":"true", "dataDirs":["/icarus/cassandra/data/data"], "snapshotTag":"autosnap-1645605747", "restorationPhase":"INIT", "restorationStrategyType":"HARDLINKS", "storageLocation":"ceph://cassandra-icarus2-ird-backup-dev/cassandra_gms_dev/rc3/2", "import":{"type":"import", "sourceDir":"/icarus/tmp/"}, "resolveHostIdFromTopology":"true", "cassandraDirectory":"/icarus/cassandra/data/"}' cassandra-dev00-ird:4567/operations
Expected behavior
restore should complete without errors.
Versions (please complete the following information):

OS "Ubuntu 20.04.3 LTS (Focal Fossa)"
cassandra_version: 3.11.6
openjdk version "1.8.0_322"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_322-b06)
OpenJDK 64-Bit Server VM (Temurin)(build 25.322-b06, mixed mode)
icarus version 2.0.2

Additional context
command used:
curl --header "Content-Type: application/json" --data '{"type":"restore", "globalRequest":"true", "dataDirs":["/icarus/cassandra/data/data"], "snapshotTag":"autosnap-1645605747", "restorationPhase":"INIT", "restorationStrategyType":"HARDLINKS", "storageLocation":"ceph://cassandra-icarus2-ird-backup-dev/cassandra_gms_dev/rc3/2", "import":{"type":"import", "sourceDir":"/icarus/tmp/"}, "resolveHostIdFromTopology":"true", "cassandraDirectory":"/icarus/cassandra/data/"}' cassandra-dev00-ird:4567/operations
Part of icarus logging
[] - 08:53:42.480 [pool-4-thread-3] INFO  c.i.e.i.r.RestorationPhase$HardlinkingPhase - Hardlinking phase has started.
[] - 08:53:43.565 [pool-4-thread-3] INFO  c.i.e.i.Manifest - Resolved manifest: cassandra_gms_dev/rc3/982ab74c-cd42-4885-a986-b61c3eb186b4/manifests/autosnap-1645605747-73919840-436c-3d25-a9d8-42659f4e5722-1645605756900.json
[] - 08:54:31.818 [pool-4-thread-3] ERROR c.i.e.i.r.RestorationPhase$HardlinkingPhase - Unable to create a hardlink from /icarus/tmp/gms/labels-bf41945013b011ebb610898044132165/md-146-big-CompressionInfo.db to /icarus/cassandra/data/data/gms/labels-bf41945013b011ebb610898044132165/md-146-big-CompressionInfo.db, skipping the linking of all other resources and deleting already linked ones.
java.nio.file.FileAlreadyExistsException: /icarus/cassandra/data/data/gms/labels-bf41945013b011ebb610898044132165/md-146-big-CompressionInfo.db -> /icarus/tmp/gms/labels-bf41945013b011ebb610898044132165/md-146-big-CompressionInfo.db
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
at java.nio.file.Files.createLink(Files.java:1086)
at com.instaclustr.esop.impl.restore.RestorationPhase$HardlinkingPhase.execute(RestorationPhase.java:539)
at com.instaclustr.esop.impl.restore.strategy.AbstractRestorationStrategy.restore(AbstractRestorationStrategy.java:93)
at com.instaclustr.esop.impl.restore.coordination.BaseRestoreOperationCoordinator.coordinate(BaseRestoreOperationCoordinator.java:57)
at com.instaclustr.icarus.coordination.IcarusRestoreOperationCoordinator.coordinate(IcarusRestoreOperationCoordinator.java:139)
at com.instaclustr.esop.impl.restore.RestoreOperation.run0(RestoreOperation.java:148)
at com.instaclustr.operations.Operation.run(Operation.java:247)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
[] - 08:54:31.840 [pool-4-thread-3] ERROR c.i.e.i.r.RestorationPhase$HardlinkingPhase - Hardlinking phase has failed: Hardlinking phase finished with errors, the linking of downloaded SSTables to Cassandra directory has failed.
Data on filesystem
ls -al /icarus/cassandra/data/data/gms/labels-bf41945013b011ebb610898044132165/md-146-big-CompressionInfo.db
-rw-r--r-- 2 1001 root 43 Feb 22 14:56 /icarus/cassandra/data/data/gms/labels-bf41945013b011ebb610898044132165/md-146-big-CompressionInfo.db

Wow. Are you starting from scratch with the latest stuff? I have junit tests testing this and it just passes fine.

What did the previous stage doing? There should be TRUNCATE phase before hardlinking which should remove all stuff from Cassandra table so linking will link it. The error you got says that the file is still there (FileAlreadyExistsException).

Yes, it could be that those tables are empty, because the restore did fail before (see [BUG] icarus restore fails when dropped tables still have cassandra snapshots or backups #9)
Which tructated all the tables and stopped.
I tried to restore the backup from before this failure and got this error.
After this I tried a backup with the old version, but somehow i get checksum errors on those backups (still not sure why and how that happened). But when i make a backup (of course with some empty tables) and restore it using version 1.1.0, it does works But i don't know how the tooling handle empty tables. It make sense it works if the tool skips those empty tables.
It looks like you say that it is not possible to restore tables that are currently empty, but that sounds strange to me.
I isn't that is one of the reasons, you have to do a restore?
About setting entities:
I need to restore the complete database to see if all the tables are correctly backed-up and could be restored. And when there are problems, which needs a restore, you want to set the database back to a consistent state, which means in most cases the complete database. Or do i miss something?

If you want to restore all database on running cluster, you can realistically restore only user keyspaces / tables, not system ones, you just can not change system tables while your node is running. So if you want to restore all user keyspaces, the best shot is to enumerate them all like "entities": "ks1,ks2,ks3", this will automatically restore all tables a particular keyspace consists of.
How many keyspaces your database consists of? Is it really too problematic to enumerate all of them upon restoration?

You can either stop the node, remove sstables, start the node and try to restore again so linking will be successful as the files will not be there anymore, or you can restart that restoration process again and see if TRUNCATE phase will remove them?

You can either stop the node, remove sstables, start the node and try to restore again so linking will be successful as the files will not be there anymore
I don't like this idea, because it is a test to see if we can restore when there is a problem. Stopping and deleting manually feels like i don't need iCarus at all.

you can restart that restoration process again and see if TRUNCATE phase will remove them?
I did rerun this restore (without entities) multiple times and it didn't remove the files before.
I can't see a reason why it will work when i supply the entities in the restore request.

For me the big question is: In case there is an empty table (for example a delete cleared all entries by mistake) and i want to restore the content. Will the restore fails, due to the empty table? If it does isn't that a BUG?
It doesn't feel right to manually try to restore the state, because our cassandra state is reached, with normal database commands (as far i can see). Therefore (in my opinion) the restore should be able to deal with this situation.

I can't see a reason why it will work when i supply the entities in the restore request.
As I said previously, because you not specifying any entities will likely evaluate it like there is nothing to truncate (1). So it will not truncate anything, hence all your sstables will be still there. So when you are going to restore and it wants to link the file from your import dir (where they were downloaded) to the actual destination in live table dir in Cassandra, it will not be successful because that target file for link is already there - because truncating process havent removed it - because your entities were empty.
(1) https://github.com/instaclustr/esop/blob/master/src/main/java/com/instaclustr/esop/impl/restore/RestorationPhase.java#L361

Oh, Sorry now i understand.
I have the feeling i get in quicksand at the moment.
I tried it, but somehow we got the dropped view problem again, but in the reshresh state???
So i removed the view related directories again and tried the restore again.
And now i get the hardlink problem with an index???
/icarus/cassandra/data/data/gms/incident_users-b4b2551013b011eba5f4b14d6a129718/.incident_users_user_id_idx/md-589-big-Data.db -> /icarus/restore/gms/incident_users-b4b2551013b011eba5f4b14d6a129718/.incident_users_user_id_idx/md-589-big-Data.db
So i'm not sure how to get to a stable situation :-(

I am not completely sure if we support indexes. I need to check that more closely.

What you mean about schema not being "restored" specifically?

if you have a schema of version X, you take a backup, then you alter it to version X+1 and you restore, it will restore SSTables from times of version X to schema of version X+1. Yes. This is true.
read this please: https://github.com/instaclustr/esop#restoring-into-different-schemas

Oh i missed that too.
We added some columns to a table and expected them to be removed too during the restore.
Still need to do a new backup and restore round. Will keep you posted

That is in general too complicated to do. We would have to create keyspace too if you managed to drop that as well with table, we need to know the topology upon keyspace creation so restoring fits the  nodes and so on ... lot of problems.

did a backup and restore: and this worked as expected.
Then i added a record to a table and run the restore again and run into the same problem again
[] - 10:14:52.744 [pool-4-thread-3] ERROR c.i.e.i.r.RestorationPhase$HardlinkingPhase - Unable to create a hardlink from /icarus/restore/gms/incident_users-b4b2551013b011eba5f4b14d6a129718/.incident_users_user_id_idx/md-23-big-CompressionInfo.db to /icarus/cassandra/data/data/gms/incident_users-b4b2551013b011eba5f4b14d6a129718/.incident_users_user_id_idx/md-23-big-CompressionInfo.db, skipping the linking of all other resources and deleting already linked ones.
java.nio.file.FileAlreadyExistsException: /icarus/cassandra/data/data/gms/incident_users-b4b2551013b011eba5f4b14d6a129718/.incident_users_user_id_idx/md-23-big-CompressionInfo.db -> /icarus/restore/gms/incident_users-b4b2551013b011eba5f4b14d6a129718/.incident_users_user_id_idx/md-23-big-CompressionInfo.db
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)

yeah, indexes again?

Yep the index, when i manually remove them and rerun the IMPORT phase it works,
But isn't wanted behavior

Yeah I ll take a loot next week. I am sorry you are hitting this issues. I have to admit it is little bit rough around the edges but we are trying.

I know, we also looked to see if we can make it simpler, without icarus, but i'm afraid we just rebuild icarus and hit all the same problems we do now.

I don't think, we did something special, but in case you need this:
CREATE TABLE IF NOT EXISTS gms.incident_users (
   incident_id text,
   user_id text,
   full_name text,
   created timestamp,
   organisational_unit text,
   PRIMARY KEY (incident_id, user_id)
);

CREATE INDEX IF NOT EXISTS ON gms.incident_users (user_id);

So backup works but restore doesnt? When indexes are involved. Can you check you have indexes backed up too?

I checked in our S3 repository and there are several directories (41) in
....gms/incident_users-b4b2551013b011eba5f4b14d6a129718/.incident_users_user_id_idx
with db files.
So i would say the indexes are backed-up

any progress?

Wow time flies ... i should have some time in couple of days to look at this.

Still no progress?
Can I help with something?

hey, there was actually some movement in Esop, check the lastest master of Esop. It fixes a case when there is a dropped empty snapshot. You can build it and then build Icarus (mvn clean install -DskipTests) for both. Appart from that I have not progressed with indexes sorry ...

looking into indexes issue now

@rjb1971 please try this one, I hit your issue with indexes and I fixed that (at least it does not occur anymore for me :) )
Let me know how it goes.
https://oss.sonatype.org/content/repositories/snapshots/com/instaclustr/icarus/2.0.3-SNAPSHOT/icarus-2.0.3-20220412.202823-5.jar

This is good news, I will try to look at it tomorrow.
Op di 12 apr. 2022 om 22:31 schreef Štefan Miklošovič <
@.***>:

@rjb1971 https://github.com/rjb1971 please try this one, I hit your
issue with indexes and I fixed that (at least it does not occur anymore for
me :) )
Let me know how it goes.
https://oss.sonatype.org/content/repositories/snapshots/com/instaclustr/icarus/2.0.3-SNAPSHOT/icarus-2.0.3-20220412.202823-5.jar
—
Reply to this email directly, view it on GitHub
https://github.com/instaclustr/icarus/issues/10#issuecomment-1097187472,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB55DW2J67URCNQ7YIXWGSTVEXMRHANCNFSM5QYA6F4Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

Tested it and it complete successfully now, but i still see the errors in the log.
Not sure if that happened in previous versions too.

I just ignored it and I do not throw it anymore but it might be still logged yet ... that is fine.

I agree.. ThanX for your effort to fix our problems and i think your product is improved by doing so.
It is really appreciated.
All a need now is a new release. :-)
And again thank you.

Yes, next week I ll release it. I ll ping you.

OK thX

reminder: what is the ETA of the new release?

hey, I am a little bit busy atm, I should do it until the end of the week.

I am on it, we have freeze in Cassandra project from 1st May so all my attention went there.

@rjb1971 it is released as 2.0.3. I am closing this.

It is ok I can start with the new release not sooner than next week
Op ma 2 mei 2022 15:49 schreef Štefan Miklošovič @.***>:

I am on it, we have freeze in Cassandra project from 1st May so all my
attention went there.
—
Reply to this email directly, view it on GitHub
https://github.com/instaclustr/icarus/issues/10#issuecomment-1114905970,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB55DW4PBWRXKOSBUGV4LOLVH7MPVANCNFSM5QYA6F4Q
.
You are receiving this because you were mentioned.Message ID:
@.***>


Create docs and update API docs

Q
A

Bug?
yes

Version Used
-

Feature request?
yes

PHP-FFMPep needs a whole documentation on e.g. readthedocs.org, so the README get's smaller and the usability and content of the docs can be improved.
Besides that the API docs must be fixed to be reachable again. But it should be considered if API docs are really needed, if a good documentation exists.

A doc has already been generated using SAMI.
Could we update it this way?
It would be great to share the SAMI config file, therefore, each person submitting a PR could update the doc accordingly.


drawio: unbreak on darwin

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg build drawio drawio.passthru.tests

Result of nixpkgs-review pr 262081 run on aarch64-darwin 1

6 packages marked as broken and skipped:

pandoc-drawio-filter
pandoc-drawio-filter.dist
python310Packages.mkdocs-drawio-exporter
python310Packages.mkdocs-drawio-exporter.dist
python311Packages.mkdocs-drawio-exporter
python311Packages.mkdocs-drawio-exporter.dist

1 package built:

drawio


Refactor BlinkIDLFileDAO to be backed by local git repository

BlinkIDLFileDAO now uses some shell scripts and utilities to perform the
following initialization:

Look for a local git repository

If the local repository is not found, create a shallow and sparse clone of
the remote repository

Find web platform IDL files in the local repository, load their contents,
then serve IDLFileContents objects out of the DAO

This eliminates the need for PlatformBlinkIDLFileDAO and will later be
abstracted so that Github- or Gitiles-backed git repositories for other
vendors' IDL can be implemented using the same pattern.

Addressed initial comments. @arobins PTAL.

Comments addressed. @arobins PTAL.

Towards #4


Starlark decoupling

break out big bramble package into store/build and runtime/lang/project

lgtm when tests past


Streaming Response

Sure! Here’s a template filled out for a feature request to enable streaming responses in the RAG (Retrieval-Augmented Generation) chatbot on GitHub:

Motivation
The ability to stream responses in the RAG chatbot will improve the user experience by allowing real-time interaction, especially when handling large or complex queries. This feature is particularly useful when responses take time to generate, ensuring the user sees a response in parts rather than waiting for the entire completion. It would enhance usability by providing immediate feedback, preventing the user from feeling disconnected or uncertain about the status of their request.
Alternatives considered, such as waiting for the full response to be generated before returning it, do not provide the same level of interactivity and responsiveness, which could lead to frustration during long-running queries.

How would you feel if this feature request was implemented?

Requirements

Implement a streaming API that allows the RAG chatbot to return parts of the response progressively.
Ensure that the frontend (chat interface) can handle partial responses and display them to the user in real-time.
The solution should support incremental content generation, including handling multiple chunks or tokens of the response being sent back.
The backend should support asynchronous streaming, without blocking other processes or interactions.
Add proper error handling for interruptions in streaming or incomplete data.

Tasks

[ ] Research streaming capabilities for the existing RAG model and framework.
[ ] Implement an API endpoint that supports streaming responses.
[ ] Modify the frontend chat interface to render streamed responses dynamically.
[ ] Test streaming with sample queries, ensuring the experience is smooth and efficient.
[ ] Ensure backward compatibility with non-streaming responses.
[ ] Document the new feature for both developers and users.

This template includes the key aspects for adding streaming functionality. Let me know if you'd like me to refine anything!

Thanks @TapanKumarBarik for logging this enhancement request. Will forward this request to our engineering team.

Hi @Vinay-Microsoft  and  @TapanKumarBarik
I was trying to understand why this is not suported as in this example:
https://github.com/Azure-Samples/azure-search-openai-demo/tree/main
This is possible.
When looking to the code:

There seems to be a reference to the need of implementing a custom callback in the UI.
However there seems to be a reference as well:

To this environment variable called SHOULD_STREAM. However this env parameter is not mentioned in any of the ones here:
https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/blob/main/docs/LOCAL_DEPLOYMENT.md
I am bit confused if the streaming support is due to the mentioned callback in the comments of the code or if there is need of something else.
Even I have no clue of vite Im fairly seasoned with python and I think copilot can do the rest. Could you point me in which direction to go to implement this?
BR
E

@edgBR i have also faced similar issues as frontend also needs to be adjusted accordingly for streaming response

Hi @TapanKumarBarik would you mind to share what exactly did you adjust in the frontend?
BR
E


Support FLUX series models

These models have just been released and appear to be amazing.  Links below:
Blog from fal.ai: https://blog.fal.ai/flux-the-largest-open-sourced-text2img-model-now-available-on-fal/
Huggingface: https://huggingface.co/black-forest-labs
There is a schell version and dev version.

When you prune it in comfy it takes up 11GB, so it should be doable to train lora at 640x640 on 3090?

is it possible to finetune the model on a 3090 or do we have to do a lora due to the size?

I totally agree. Since SD3 may not be able to fit a slightly larger dataset due to model problems (scripts include SimpleTuner, SD-scripts, OneTrainer), it is recommended to stop developing SD3 training scripts. I did a simple test on Flux-dev, and its capabilities are completely superior to SD3. Here are some examples:

It’s worth pointing out that this is the first model I’ve seen that can correctly draw the position of the umbrella handle and the umbrella cover, and the text prompt on the road sign is “iiilllllbddbwW”. Although the AI didn’t draw it correctly, I haven’t seen any model that can draw it correctly either.

I totally agree. Since SD3 may not be able to fit a slightly larger dataset due to model problems (scripts include SimpleTuner, SD-scripts, OneTrainer), it is recommended to stop developing SD3 training scripts. I did a simple test on Flux-dev, and its capabilities are completely superior to SD3. Here are some examples:

I strongly disagree. While the SD3 Medium model has certain drawbacks, it possesses a crucial advantage that FLUX lacks: its weights are publicly available. In contrast, FLUX only provides access to the base model's weights through an API, with no indication or information suggesting they plan to make it open-source. The models that are publicly accessible are derived through distillation of the base model; they are truncated, incomplete, and practically unsuitable for further training. It only makes sense to train the model we weren't given, as fine-tuning the distilled models would require roughly the same effort as training from scratch, if not more. Even the SDXL model was superior in this regard

I totally agree. Since SD3 may not be able to fit a slightly larger dataset due to model problems (scripts include SimpleTuner, SD-scripts, OneTrainer), it is recommended to stop developing SD3 training scripts. I did a simple test on Flux-dev, and its capabilities are completely superior to SD3. Here are some examples:

I strongly disagree. While the SD3 Medium model has certain drawbacks, it possesses a crucial advantage that FLUX lacks: its weights are publicly available. In contrast, FLUX only provides access to the base model's weights through an API, with no indication or information suggesting they plan to make it open-source. The models that are publicly accessible are derived through distillation of the base model; they are truncated, incomplete, and practically unsuitable for further training. It only makes sense to train the model we weren't given, as fine-tuning the distilled models would require roughly the same effort as training from scratch, if not more. Even the SDXL model was superior in this regard.
Calling it open-source is akin to labeling GPT-4o as open-source simply because we were given GPT-3 weights and the ability to fine-tune it. I'm concerned that we'll be wasting time that could be better spent studying SD3, debugging and optimizing its training script. SD3 has more potential, and Stability AI has promised to eventually release all models, including their weights, as open-source. This makes SD3 a more promising avenue for our efforts

You do realise the schnell and dev model weights are freely available, and the schnell one is even under the apache licence....

Hello, the weights for the Flux series models have been released, including the dev version and the schnell version.

Please read this https://blog.fal.ai/flux-the-largest-open-sourced-text2img-model-now-available-on-fal/
Dev and schnell are obtained by distillation of pro scales. It is possible to create LoRAs for them, they will work. But full model training is practically impossible because of this

Hello, the weights for the Flux series models have been released, including the dev version and the schnell version.

Please read this https://blog.fal.ai/flux-the-largest-open-sourced-text2img-model-now-available-on-fal/ Dev and schnell are obtained by distillation of pro scales. It is possible to create LoRAs for them, they will work. But full model training is practically impossible because of this

It should be possible to fine-tune distilled models.

It should be possible to fine-tune distilled models.

Why should it? I just did a quick search for information about training SDXL Turbo, and it turns out it was also obtained through distillation from the base model. There are tons of such models on Civitai, but they're all created by merging SDXL Turbo with something else. I couldn't find a single one obtained through fine-tuning. As I expected. https://www.reddit.com/r/StableDiffusion/comments/18l2qp0/sdxl_turbo_fine_tunemerging/

It should be possible to fine-tune distilled models.

Why should it? I just did a quick search for information about training SDXL Turbo, and it turns out it was also obtained through distillation from the base model. There are tons of such models on Civitai, but they're all created by merging SDXL Turbo with something else. I couldn't find a single one obtained through fine-tuning.The only relevant post I came across was a complaint on Reddit about how training SDXL Turbo produces very poor results. As I expected. https://www.reddit.com/r/StableDiffusion/comments/18l2qp0/sdxl_turbo_fine_tunemerging/

That is because the training code for Turbo was never released and nobody wrote one.  It's not fundamentally impossible.

even training schnell with lora or full tune is fine. they're just big models and require the use of LoRA with quantised base weights, but Kohya should probably wait for the bugs to be worked out in Quanto first before going ahead and trying to integrate it. it makes a mess of the model state dict keys.

@kohya-ss Training scripts released : https://github.com/XLabs-AI/x-flux

those are pretty minimal and eg. it doesn't implement cosmap/logit-norm or any of the SD3 training details, just about the same as cloneofsimo/minRF implementation. in fact it's basically identical - the interesting thing there is probably their ControlNet training implementation details

diffusers scripts arrived
https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux.md

@FurkanGozukara , you are amazing as usual!

@ddpasa thanks
pull request arrived already :d
https://github.com/kohya-ss/sd-scripts/pull/1374/files/da4d0fe0165b3e0143c237de8cf307d53a9de45a..36b2e6fc288c57f496a061e4d638f5641c32c9ea

It should be possible to fine-tune distilled models.

Why should it? I just did a quick search for information about training SDXL Turbo, and it turns out it was also obtained through distillation from the base model. There are tons of such models on Civitai, but they're all created by merging SDXL Turbo with something else. I couldn't find a single one obtained through fine-tuning.The only relevant post I came across was a complaint on Reddit about how training SDXL Turbo produces very poor results. As I expected. https://www.reddit.com/r/StableDiffusion/comments/18l2qp0/sdxl_turbo_fine_tunemerging/

Do you guys also love it when someone is so confidently incorrect?
My flux finetune is coming in very nicely. Huge upgrade compared to SDXL and Pony, also way more trainable than SD3 Medium. It's literally impossible to add NSFW to SD3 medium because of the complete lack of training data.
Flux seem to seen plenty of NSFW images, and it's just filtered via captioning. So the context and knowledge already exists in the latent space, and it only needs to.. well get finetuned.
 Pyro's NSFW model <3 FLUX

It should be possible to fine-tune distilled models.

Why should it? I just did a quick search for information about training SDXL Turbo, and it turns out it was also obtained through distillation from the base model. There are tons of such models on Civitai, but they're all created by merging SDXL Turbo with something else. I couldn't find a single one obtained through fine-tuning.The only relevant post I came across was a complaint on Reddit about how training SDXL Turbo produces very poor results. As I expected. https://www.reddit.com/r/StableDiffusion/comments/18l2qp0/sdxl_turbo_fine_tunemerging/

Do you guys also love it when someone is so confidently incorrect?
My flux finetune is coming in very nicely. Huge upgrade compared to SDXL and Pony, also way more trainable than SD3 Medium. It's literally impossible to add NSFW to SD3 medium because of the complete lack of NSFW content in its training data.  Nobody is ever going to create any kind of content for SD3 when you can create better results for the same money with flux. so yeah, rip.
Flux seems to have seen plenty of NSFW images, and it's just filtered and dropped out via captioning. So the context and knowledge already exists in the latent space, and it only needs to... well get finetuned.
So, yeah f*ck SD3. Pyro's NSFW model goes FLUX.

what are you talking about? i trained 3.0 for 30 minutes and it can generate nsfw just fine. NSFW link https://imgur.com/a/sd-30-test-G7G7G6u

It should be possible to fine-tune distilled models.

Why should it? I just did a quick search for information about training SDXL Turbo, and it turns out it was also obtained through distillation from the base model. There are tons of such models on Civitai, but they're all created by merging SDXL Turbo with something else. I couldn't find a single one obtained through fine-tuning.The only relevant post I came across was a complaint on Reddit about how training SDXL Turbo produces very poor results. As I expected. https://www.reddit.com/r/StableDiffusion/comments/18l2qp0/sdxl_turbo_fine_tunemerging/

Do you guys also love it when someone is so confidently incorrect?
My flux finetune is coming in very nicely. Huge upgrade compared to SDXL and Pony, also way more trainable than SD3 Medium. It's literally impossible to add NSFW to SD3 medium because of the complete lack of NSFW content in its training data.  Nobody is ever going to create any kind of content for SD3 when you can create better results for the same money with flux. so yeah, rip.
Flux seems to have seen plenty of NSFW images, and it's just filtered and dropped out via captioning. So the context and knowledge already exists in the latent space, and it only needs to... well get finetuned.
So, yeah f*ck SD3. Pyro's NSFW model goes FLUX.

what are you talking about? i trained 3.0 for 30 minutes and it can generate nsfw just fine. NSFW link https://imgur.com/a/sd-30-test-G7G7G6u

Someone just reads too much reddit and similar places where everyone is convinced that if a model wasn't trained on nsfw then they will never be able to create such things. How they used to create models for anime, furry and the rest for sdxl no one knows. Lost technology
In all seriousness, there's nothing stopping sd3 from learning to create any nsfw content and even worse. Due to the more efficient architecture, training does not require as much GPU overhead as sdxl.

We (group of SDXL finetuners) spend like 5k bucks making NSFW in SD3 work, but a model that can't even render women lying in grass is so lobotomized that re-introducing NSFW takes immense ressources, as in the ballpark of SAI's training infrastructure. No hobby finetuner is going to pay for that. Nobody is going to pay for that, if they can get way better results for a fraction of the cost with FLUX.
It's not hard to understand. It took 20 bucks to teach FLUX NSFW concepts.... 5k$ vs 20$ pretty clear cut.

How they used to create models for anime, furry and the rest for sdxl no one knows. Lost technology

Well it seems that you don't know the basic of how training such models work and how self-organisation of embeddings in the latent space works.  SD3 data corpus has exactly 0 NSFW images in it. And you honestly have difficulties to understand why one is trainable and the later isn't? You're on the wrong board then.
Please stop talking about things you don't have a clue about.
Also FLUX is runnable locally and the weights are public, so I don't even know what " it has no scales and access only by api" even means.

We (group of SDXL finetuners) spend like 5k bucks making NSFW in SD3 work, but a model that can't even render women lying in grass is so lobotomized

Stabilityai promised to release the 3.1 model soon. They promised to fix this problem in it. You've been too quick to educate yourself

Well it seems that you don't know the basic of how training such models work and how self-organisation of embeddings in the latent space works.  That it didn't use NSFW content in training so nsfw training is impossible, "it's a terrible model, stabilityai killed their reputation by refusing to train on nsfw content, we can't use it, we stay on sd1.5".... Just like they wrote about sd2.... Let's wait a year and find out that there was nsfw in the sd3 dataset but it was removed from the sd4 dataset so we stay on sd3 and boycott the new model....

People are using simple tuner for flux lora creation. Unfortunately it has no windows support. Waiting for kohya ss :) . Flux dev is so much better than sd3 💯

Stabilityai promised to release the 3.1 model soon. They promised to fix this problem in it. You've been too quick to educate yourself

If sd3.1 can have the performance of flux dev and allow training and sharing, and the machine cost of fine-tuning is smaller than flux dev, then I will be very happy to use sd3.1. But as far as sd3 8b and the license of sd3 are concerned, I am pessimistic about this possibility.

Now sd3 branch supports FLUX.1 dev LoRA training experimentally :) https://github.com/kohya-ss/sd-scripts/tree/sd3

Thank you for your excellent work. The fine-tuning effect of sd-scripts with Flux has completely met my expectations, and its performance is on par with Simple Tuner.
Additionally, is there any plan to support Flux in some of the LoRA processing scripts? These scripts could help the community more quickly develop models like "detail enhancer."

People are using simple tuner for flux lora creation. Unfortunately it has no windows support. Waiting for kohya ss :) . Flux dev is so much better than sd3 💯

Now sd3 branch supports FLUX.1 dev LoRA training experimentally :) https://github.com/kohya-ss/sd-scripts/tree/sd3

Will this work for the much more efficient NF4 model that was released yesterday?
Up to 4x speedups, reduced vram, increased quality. Hopefully more accessible to people without a spare A100 lying around.
https://civitai.com/models/638572/nf4-flux1
https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/981

you don't need an A100 for flux. imo kohya should release sooner than keep trying to add the million features. you can train on 16G VRAM without any quantisation at all.

you don't need an A100 for flux. imo kohya should release sooner than keep trying to add the million features. you can train on 16G VRAM without any quantisation at all.

It did in the other training setups I'd seen but yeah apparently not anymore. The NF4 model is far superior though and more accessible for inference.
FP8 used to be virtually unusable on my 4080 because it'd take about 5-10 mins for 1 overquantized generation, and now it's about 20 sec for outputs that look on par with Pro.
Don't really wanna waste a week training an FP8 model that's already obsolete and can't be used be most people.

it's not like that at all though. fp8 is fine, especially in pytorch 2.4. you can read back through the comments in this issue to see.

also, NF4 is definitely not "on par with Pro" 🤪

In addition to the links above citing the same thing I did, prominent youtubers testing it are saying the same thing.
"the regular flux Dev takes about 50 seconds to run on my machine on a RTX 4090, this takes 14 seconds to run"
https://www.youtube.com/watch?v=L5BjuPgF1Ds

just as a quickie and teaser for all people on the fence shelling out for 4090 or an online GPU and doubters of FLUX's capabilities.
my pre-release ATJ lora - will get published on civitai in a couple of hours

It's like on a completely different level than SD3

@bghira finally got your shit running, and the results are amazing! thanks for entertaining me on reddit, hope I didn't piss you off all that much :)

just... try not to call people the 'n word' when they don't meet your expectations of expedience. that genuinely hurt. i am glad it is working for you now.

Perhaps you misread me, or mixed me up with someone else, but I never used the n-word in my life.  The haters have been out in full force lately

The dev and snhnel versions of flux are based on the distillation model of the pro version. It is meaningless to support fine-tuning. It is completely impossible to learn complex tasks. It seems that there are still a large number of amateur users in this discussion.

The dev and snhnel versions of flux are based on the distillation model of the pro version. It is meaningless to support fine-tuning. It is completely impossible to learn complex tasks. It seems that there are still a large number of amateur users in this discussion.

I trained cfg back into flux.dev with a 6 hour LoRA accidentally. If you just set the "fake" CFG value (the one they baked into with distillation) to anything greater than 1, the distillation breaks and CFG is restored. You just have to skip the first couple of steps because for some reason I don't yet understand if you use CFG on them it makes blurry images.

@ostris is training CFG back into the schnell model on huggingface, dunno how that's going.

I trained cfg back into flux.dev with a 6 hour LoRA accidentally.

LoRA ≠ full finetune

^ this is incorrect. see LoRA paper about rank 128.

I trained cfg back into flux.dev with a 6 hour LoRA accidentally.

LoRA ≠ full finetune

are you speedrunning "how often can I be wrong in one thread"?

Latest update to SD3 script made this smooth sailing for 12GB vram flux lora training, I've got a 4 hour timer on 1600 steps at the recommended settings and its at most using 7GB of my 12GB GPU.  I get 200s/it with the kohya flux lora which I get 1.8s/it using other flux lora from civitai

For some reason my flux lora trained on kohya ss when using in comfyui runs extremely slow. I get 200s/it with the kohya flux lora while I get 1.8s/it using other flux lora from civitai

Is the slow inference possibly due to insufficient VRAM or RAM? You could try training LoRA with a lower dim value, or consider using a machine with more VRAM and RAM for inference. The fine-tuning results of Kohya's script are quite good; this is my result of fine-tuning Flux using Kohya's script, and there hasn't been any slowdown in inference speed.

Yes, using my kohya trained lora eats all my vram in f8 , tried with 16gb vram and 20 gb vram machines . Using other lora eats like around 12 gb vram . I might have done something wrong . I did use 512 resolution which gave me around 3sec/it during training on 4060ti 16 gb . I used this - accelerate launch  --mixed_precision bf16 --num_cpu_threads_per_process 1 flux_train_network.py --pretrained_model_name_or_path "E:/ComfyUI/models/checkpoints/flux1-dev.safetensors" --clip_l "E:/ComfyUI/models/clip/clip_l.safetensors" --t5xxl "E:/ComfyUI/models/clip/t5xxl_fp16.safetensors" --ae "E:/ComfyUI/models/vae/ae.safetensors" --cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers --max_data_loader_n_workers 2 --seed 42 --gradient_checkpointing --mixed_precision bf16 --save_precision bf16 --network_module networks.lora_flux --network_dim 4 --optimizer_type adafactor --optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False" --learning_rate 1e-4 --network_train_unet_only --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --fp8_base --highvram --max_train_epochs 100 --save_every_n_epochs 10 --sample_every_n_steps 200 --sample_prompts "D:/github/ee/pt.txt" --sample_sampler "euler" --dataset_config "D:/github/ee/fx.toml" --output_dir "D:/github/kohya_ss/outputs" --output_name flux-lora-name --timestep_sampling sigmoid --model_prediction_type raw --guidance_scale 1.0 --loss_type l2      (might have changed a little if it gave error but this is what i saved on my text file to copy paste again)

Yes, using my kohya trained lora eats all my vram in f8 , tried with 16gb vram and 20 gb vram machines . Using other lora eats like around 12 gb vram .

That's strange because a LoRA with a dim value of 4 shouldn't consume more VRAM. Assuming that most of the LoRAs on Civitai are currently fine-tuned with Simple Tuner, and considering that both Simple Tuner and SD scripts are now only training the Unet, the VRAM requirements should be similar if the file sizes are comparable. Therefore, the slowdown in inference speed is likely due to insufficient RAM. 20GB of RAM might indeed be a bit tight for this kind of task. You could try restarting your computer to fully clear the VRAM and RAM, then test the inference speed of SD scripts' LoRA again.

For me it just goes out of memory when i use my lora trained on kohya . Lora rank 4 alpha 1 .

For me it just goes out of memory when i use my lora trained on kohya . Lora rank 4 alpha 1 .

Would you mind sharing this lora? I would like to see how much video memory this lora would occupy on my machine.

Unfortunately the lora is a person lora(ai person but still not mine)  so i cannot send it. I will train a celeb lora again tomorrow and will send the link. Only thing i can think of is that i did cropped the images to 512x512 as i am using 512 res for training . It is sad because the sampling images during training were very good and close to the actual look. Here 1st one is training sampling image and second one is training image.

@D3voz the high memory usage with some LoRAs has been reported in the ComfyUI repo: https://github.com/comfyanonymous/ComfyUI/issues/4343 . The slow inference is specific to Windows and the way it uses shared GPU memory, on Linux it simply runs out of memory (or since the recent updates, partially loads the LoRA with undefined results).

Long thread, and I am late to the party.  I want no part of SD3, or SAI, but we now have other options we can train on as good as Flux is that is truly open?  I, along with a lot of lawyers, don't like their licence, but we all wait for clarification from them (if it comes).  I have a 4090 and training at BS1 will be the one thing to cause me to toss in the towel on all this.  A lora taking 90 minutes to train without even the clip (that I drastically need for what I do) is far too long.  If this is the future, then I rather rot in antiquity.

Long thread, and I am late to the party. I want no part of SD3, or SAI, but we now have other options we can train on as good as Flux is that is truly open? I, along with a lot of lawyers, don't like their licence, but we all wait for clarification from them (if it comes). I have a 4090 and training at BS1 will be the one thing to cause me to toss in the towel on all this. A lora taking 90 minutes to train without even the clip (that I drastically need for what I do) is far too long. If this is the future, then I rather rot in antiquity.

SD3 is a terrible model with very complicated legal mess of a license.  Given how much better Flux.1 is, I see no reason to waste any of my time on SD3.  Stability AI is a huge mess right now, and it's really good that new models are coming to the scene.

flux and sd3 have the same license, but both are unenforceable anyway. just have fun and do less drama.

what does it have to do with this thread? act like an adult if you are one. some of us are researchers who dont care about how open a model is and can make a living regardless.

It is enforceable and even as a researcher if you break it I'd even come after you lawyers if I were they.  What is with "researchers" willing to be pirates?

I didn't demand anything I only stated that people who obey licenses, and the law, look at the license and don't want to risk it then you come barging in to basically say "screw the licence and just have fun".  No, you don't break laws.  Beyond that whatever.

we are here obeying the law which doesnt support the license. what an odd situation. why dont you just use AuaFlow?

You know, there are adults who make a living and don't wish to spend it all on lawyers and court costs defending ourselves. It isn't about just popping out some waifu/husbando.

I can assure you, researchers in the field of image-synthesis (outside Medical) are not moral busybodies opposed to waifus and husbandos.
Regardless, your opinion is irrelevant to the discussion. Go right a tumblr blog about it, if you must.

  File "/home/mikey/kohya_ss/sd-scripts/finetune/prepare_buckets_latents.py", line 286, in <module>
    main(args)
  File "/home/mikey/kohya_ss/sd-scripts/finetune/prepare_buckets_latents.py", line 89, in main
    vae = model_util.load_vae(args.model_name_or_path, weight_dtype)
  File "/home/mikey/kohya_ss/sd-scripts/library/model_util.py", line 1304, in load_vae
    converted_vae_checkpoint = convert_ldm_vae_checkpoint(vae_sd, vae_config)
  File "/home/mikey/kohya_ss/sd-scripts/library/model_util.py", line 429, in convert_ldm_vae_checkpoint
    new_checkpoint["quant_conv.weight"] = vae_state_dict["quant_conv.weight"]```

What is this quant_conv.weight it is trying to find in the vae? I can't see it in the file. It looks like it's ignoring that I provided a path to the vae file also and trying to load it from the flux model file.

Flux would be awesome

Latest update to SD3 branch script made this smooth sailing for 12GB vram flux lora training, I've got a 4 hour timer on 1600 steps at the recommended settings and its at most using 7GB of my 12GB GPU. BLESS YOU KOHYA!!! <3

@b-7777777 my 12GB 3060 is running out of memory. Can you tell me the base model, clip and vae you're using, please?

4 hours for 1600 steps is really really slow. you can rent a $0.200/hr 4090 and train 1600 steps in one hour. for less than $1. it probably costs you more per-kWh than it would to rent a cloud GPU

I tried to see if I can run with Prodigy and it appears to work… weirdly only using ~17GB of VRAM, so not much more than when I was using AdamW8bit. That seems… wrong. Is there anything I’m missing on why Prodigy would “work” but potentially not be doing what I expect?

Well, I can confirm training with prodigy works great. Just using my same settings as what I’d use in SDXL, setting d_coef to 2, betas 0.9,0.999, weight_decay 0.01. No warmup steps. Was able to get a great Lora of my dog at 1000 steps in about 35 mins on a 4090 (~2.2s/it) All other settings the same as the default recommendation on sd3 readme (except LR is 1 of course). Only used 17GB VRAM.
Got better results than my attempted training using AdamW8bit constant for 3000 steps at 1e-4 and 1e-3. In those it seemed both undertrained (sometimes prompted other unrelated things) and also overfit (realistic style when prompting line art or comic book style)

Just to note for anyone else that’s been trying this or is just starting - at least for me, if you go with full bf16 and the fused backward pass option, I get I get messed up hands and then distorted body parts or bodies very quickly regardless of learning rate. No full bf16 and the fused optimizer option and I don’t get that. Surprisingly the latter also appears to use less VRAM, certainly not any more, and is faster. Of course, keep in mind things are changing, but that’s been my experience the last few days.

Can you add an script to quantify checkpoint (BF16) to NF4?
Many users do not have powerful hardware. I hope my model can be used for more people.

I tried to see if I can train Lora with Prodigy and it appears to work… weirdly only using ~17GB of VRAM, so not much more than when I was using AdamW8bit. That seems… wrong. Is there anything I’m missing on why Prodigy would “work” but potentially not be doing what I expect?

What settings are you using to use such low VRAM? AdamW8bit training with 512px images is using almost 24gb ram for me.

I tried to see if I can train Lora with Prodigy and it appears to work… weirdly only using ~17GB of VRAM, so not much more than when I was using AdamW8bit. That seems… wrong. Is there anything I’m missing on why Prodigy would “work” but potentially not be doing what I expect?

What settings are you using to use such low VRAM? AdamW8bit training with 512px images is using almost 24gb ram for me.
Edit: if you have any caption drop out and it doesn't cache the captions, the text encoders are loaded into the GPU memory during training time which leads to the high GPU usage.

with adafactor and 512px i go as low as 7.5 GB : https://youtu.be/nySGu12Y05k

@wogam "The training can be done with 12GB VRAM GPUs with Adafactor optimizer, --split_mode and train_blocks=single options." from the readme.


(maint) Fix incorrect non-capturing group escapes

The VAR_NAME regexp in Puppet::Pops::Patterns had ':?' instead of
'?:' to denote non-capturing groups. The problem has gone unnoticed
since a) the ':' became optional, and b) no code attempts to use
that expression and extract the groups.

CLA signed by all contributors.


Update glossary.md

Looks like this should be indented differently. @roll can you please confirm. Thanks!

@sapetti9
Technically "Custom Properties" belongs to "Descriptor" so both heading levels are possible. @peterdesmet WDYT?

I think it makes the most sense as its own heading in the dropdown.


aceunit claims to support C89/C90, but requires multiple C99 features

Some  main files require C99:

aceunit.h uses the predefined macro "__func__", which is a C99 feature

aceunit.h is including '<stdbool.h>', which is a C99 header

aceunit.h is using the 'bool' type, which is a C23 feature (or with <stdbool.h> a C99 feature)

AceUnit_SetJmpRunner.c mixes declarations and code, which is not allowed in C90

additional warnings from 'clang' for most files:
"warning: a function declaration without a prototype is deprecated in all versions of C"

Most selftest source files require C99
(probably all of them, but i didn't check that):

the predefined macro "__func__" is used in many locations

'bool' is used in many locations

I used for testing:
CC=gcc  CFLAGS="-Wall -pedantic -std=c89"

or

CC=clang  CFLAGS="-Wall -pedantic -std=c89"

Fixed all of that. Thanks for reporting.


No md5sum on OSX

Had to replace md5sum with "openssl md5" on OSX. Is openssl common enough on Linux, or should I make a pull request that has the Makefile determine if it is building for OSX?
Also, I have a Haskell Criterion benchmark wrapper. Want me to host on my github or want me to do a pull request with a bench_crit Make option? Just Stack config files and a small Haskell script.
Looking into adding streaming top-k, t-digest, ... spiking with a script around Algebird to get a baseline.

Hmm, a victim of my overly broad homebrew installs.
Okay, I think you could also change the script that uses it. Creating a variable in the top that either points to md5sum or openssl md5 or another checksum tool, and use that variable?
Regarding your other questions: if you can explain what (preferable in a separate issue/pull request) the Haskall Criterion wrapper offers this repo (pros/cons)? And if the the haskall dependency is only for the benchmarks, yes this could be a useful addition.
the additions of new tools: sounds cool, looking forward to your pull requests!
Could you explain what you mean with:

spiking with a script around Algebird to get a baseline.
?

Submitted a PR on just the md5sum issue. Need some more work on the Criterion benchmarks. Open this in a browser for an example Criterion report, https://raw.githubusercontent.com/chadbrewbaker/rosalind/master/DNA/count.html

okay, so #12 is merged.
criterion looks pretty, if you can keep the dependency separated, I'm all fine with merging it.


Update .gitmodules

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

Checklist

[ ]  documentation is added or updated
[ ]  tests are added or updated


Revert "Add ".git" as per examples"

This reverts commit e606e6b6773c611b57e423f3323c88b5ea90ed8f.
This fixes #463

Thanks!


Sioyek build fails with mupdf 1.20.x

Steps To Reproduce
Steps to reproduce the behavior:

build sioyek on current nixpkgs-unstable or master (e.g. nix build nixpkgs#sioyek)

Build log
Gist link from nix log /nix/store/w1rjcjfx1knhf5vc94vbmg6pc221wpaz-sioyek-1.4.0.drv, unfortunately I couldn't find a way to strip the colour codes.
The important errors appear to be
pdf_viewer/document.cpp: In member function 'void Document::convert_toc_tree(fz_outline*, std::vector<TocNode*>&)':
pdf_viewer/document.cpp:545:32: error: no match for 'operator==' (operand types are 'fz_location' and 'int')
  545 |                 if (root->page == -1) {
      |                     ~~~~~~~~~~ ^~ ~~
      |                           |       |
      |                           |       int
      |                           fz_location

pdf_viewer/pdf_renderer.cpp:318:125: error: cannot convert 'fz_quad*' to 'int*'
  318 |                                 int num_results = fz_search_page(mupdf_context, page, utf8_encode(req.search_term).c_str(), hitboxes, max_hits_per_page);
      |                                                                                ^~~~~~~~
      |                                                                                |
      |                                                                                fz_quad*

pdf_viewer/document.cpp:552:52: error: cannot convert 'fz_location' to 'int' in assignment
  552 |                         current_node->page = root->page;
      |                                              ~~~~~~^~~~
      |                                                    |
      |                                                    fz_location

Additional context
This was broken by mupdf being updated to version 1.20.3 (#184363), which also broke zathura (#187305, fixed in #187813). sioyek upstream has mupdf as a submodule that's set at 1.19.x but nixpkgs's derivation uses its mupdf package in buildInputs instead.
Upstream is unlikely to upgrade to mupdf (https://github.com/ahrm/sioyek/pull/294).
Both using the unaccepted PR to upstream to make sioyek compatible with 1.20.x (https://github.com/ahrm/sioyek/pull/294) and using 1.19.0 mupdf work, but I'm not sure what would be the best solution moving forward.
Should we

have sioyek build using its mupdf submodule somehow, or
apply a patch to make it compatible with mupdf 1.20.x, or
keep a separate version of mupdf 1.19.0 around in nixpkgs to provide to this derivation, or
override mupdf in this derivation (such as using the following)?let mupdf' = mupdf.overrideAttrs (old: rec {
  inherit (old) pname;
  version = "1.19.0";
  src = fetchurl {
    url = "https://mupdf.com/downloads/archive/${pname}-${version}-source.tar.gz";
    sha256 = "1vfyhlqq1a0k0drcggly4bgsjasmf6lmpfbdi5xcrwdbzkagrbr1";
  };
}); in

Notify maintainers
Notifying @podocarp as mentioned in meta.maintainers.
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.59, NixOS, 22.11 (Raccoon), 22.11.20220820.13711c9`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.10.3`
 - channels(root): `""`
 - nixpkgs: `/nix/store/bxiy4hzvzsml5knnj0labdw6y2qgvmvj-source`

Can we just use the vendored copy of mupdf, that's the only way supported by upstream I guess.

Can we just use the vendored copy of mupdf, that's the only way supported by upstream I guess.

I think that works the best, though I'm unsure of how that's normally done in nixpkgs. Do you perhaps know of any other packages with vendored submodule dependencies I could reference?

If supported by the upstream build system, that should not require extra efforts, just set fetchSubmodules and all is fine. But I just noticed that is not the case for sioyek, it's just a bunch of shell scripts... Just go for the override path is easier in this case.

Ah, so that's why it didn't work when I just tried with fetchSubmodules to see if it would. I'll look closer at Sioyek's build scripts and see if I can get it working, thanks!

Ah, so that's why it didn't work when I just tried with fetchSubmodules to see if it would. I'll look closer at Sioyek's build scripts and see if I can get it working, thanks!


test: update actions

Alaska Airlines Pull Request
Summary:
This pull request tests new features regarding better Github actions.
Type of change:
Please delete options that are not relevant.

[ ] New capability
[x] Revision of an existing capability
[ ] Infrastructure change (automation, etc.)
[ ] Other (please elaborate)

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


can't find direction of the wind

Thanks for the great app. Everything works great. I do not have enough understanding of the grib file itself. How can I get the wind direction from this json?

got it


lane select underlying bug.

https://github.com/CPFL/Autoware/blob/master/ros/src/computing/planning/mission/packages/lane_planner/nodes/lane_select/lane_select_core.cpp#L632
at line 627, here defined current_lane_idx_ = -1; , so it is always execute line 632,  always initialize lane select, never touch processing().
please check.
Bug
Expected Behavior
Actual Behavior
Steps to Reproduce the Problem

Specifications

Ubuntu version:
ROS Version:
Autoware branch:

New Feature
Describe where does new feature request come from. Add high-level picture of how the new feature should work.

should remove line 627-630.

any update? I can submit a PR for this bug.

thanks for report!
It certainly looks like it will not be right.
@TomohitoAndo
could you check it please?

any update?

hi, dear, it seems no response of this issue...

hi dear, any update? I think there is need someone to maintain the debug code.
if ok, i'm also glad to join in your team and maintain.

That would be appreciated if you can contribute to the repo.
@TomohitoAndo @aohsato Can you check the bug?
2017/12/09 午前10:06 "asimay"<EMAIL_ADDRESS>
hi dear, any update? I think there is need someone to maintain the debug
code.
if ok, i'm also glad to join in your team and maintain.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/CPFL/Autoware/issues/933#issuecomment-350410642, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAToC1EAblOOcR0GbCWOKNl9aK8mdlwkks5s-d0DgaJpZM4Qspf3
.

@asimay thx for the report.
@shinpei0208 @TomohitoAndo it seems that with this bug we will never be able to do lane change: https://github.com/CPFL/Autoware/blob/master/ros/src/computing/planning/mission/packages/lane_planner/nodes/lane_select/lane_select_core.cpp#L170, correct?


Update WASM SIMD implementation

I would like to propose a set of changes to update WASM SIMD to the current state of the proposal:

Re-enable and clean up tests
Scrub references to build instructions and m128 WASM type
Update opcodes to the latest revision of the spec

@MikeHolman


Fixes partner link on Fair booth header

Fixes: https://artsyproduct.atlassian.net/browse/LD-247

@ashleyjelks Gonna merge this one as it's #trivial


ecs/service always times out

Given the following:
const cloud = require("@pulumi/cloud-aws");

const REDIS_URL = "redis://foo";
const DATABASE_URL = "psql://bar";

let service = new cloud.Service("django-app", {
  containers: {
    django: {
      build: "../",
      memory: 512,
      cpu: 2,
      // ports: [{ port: 443, protocol: "https", targetPort: 8000 }],
      ports: [{ port: 8000 }],
      environment: {
        DATABASE_URL: DATABASE_URL,
        CELERY_BROKER_URL: REDIS_URL,
        CELERY_RESULT_BACKEND: REDIS_URL
      }
    }
  }
  // replicas: 2
});

let workerService = new cloud.Service("django-app-worker", {
  containers: {
    celery: {
      build: "../",
      command: ["celery", "-A", "app", "worker", "-l", "info"],
      memory: 512,
      cpu: 2,
      environment: {
        DATABASE_URL: DATABASE_URL,
        CELERY_BROKER_URL: REDIS_URL,
        CELERY_RESULT_BACKEND: REDIS_URL
      }
    }
  }
  // replicas: 2
});

// export just the hostname property of the container frontend
exports.url = service.defaultEndpoint.apply(e => `http://${e.hostname}`);

Creating the resources with pulumi up the first time seems to work fine. After that any update to the Dockerfile results in a timeout of the update:
Diagnostics:
  aws:ecs:Service (django-app-worker):
    error: Plan apply failed: 1 error occurred:

    * updating urn:pulumi:backend-dev::django-deploy::cloud:service:Service$aws:ecs/service:Service::django-app-worker: timeout while waiting for state to become 'true' (last state: 'false', timeout: 10m0s)

  aws:ecs:Service (django-app):
    error: Plan apply failed: 1 error occurred:

    * updating urn:pulumi:backend-dev::django-deploy::cloud:service:Service$aws:ecs/service:Service::django-app: timeout while waiting for state to become 'true' (last state: 'false', timeout: 10m0s)

Times out after 10 minutes, when I run pulumi up after this timeout it seems to cleanup and not timeout. Any subsequent update again will timeout, tried many times to destroy and redo the entire setup, always end up here.
Any ideas on how to debug this issue?

Followed up with @eelkeh at https://pulumi-community.slack.com/archives/C84L4E3N1/p1542817597870700.  It appears the most immediate issue was due to ENI limits in ECS:
service django-app-e087161 was unable to place a task because no container instance met all of its requirements. The closest matching container-instance c02faa85-e355-43c2-af55-925ffeba89ce encountered error "RESOURCE:ENI

I'll close this out for now, and continue the debugging conversation in Slack. If it turns out there is an issue in Pulumi that is causing this we'll open a new issue as appropriate.

@lukehoban has any progress been made on this issue? I'm not sure if it's simply related to the number of changes or the application growth but we are no longer able to deploy as it always times out.

@lukehoban @joeduffy I seem to be experiencing this every time I deploy a stack using the Pulumi GitHub action.
updating urn:pulumi:dev::infra::awsx:x:ecs:FargateService$aws:ecs/service:Service::backend: timeout while waiting for state to become 'true' (last state: 'false', timeout: 10m0s)
The service does eventually complete in AWS, and reach the intended state.

Is there  solution to this issue? Here is the explanation of the error from AWS docs:
For RESOURCE:ENI errors, your cluster does not have any available elastic network interface attachment points, which are required for tasks that use the awsvpc network mode. Amazon EC2 instances have a limit to the number of network interfaces that can be attached to them, and the primary network interface counts as one. For more information about how many network interfaces are supported per instance type, see IP Addresses Per Network Interface Per Instance Type in the Amazon EC2 User Guide for Linux Instances.

What I don't understand is how this error only comes up on updating a deployed infrastructure.

I think in my case I was using an instance, t2-nano, that is limited to 2 network interfaces. And when I run an update, 2 revisions of the same task definition have to be running simultaneously on the same instance, apparently. (Probably a graceful way to redirect network traffic to the new revision?)
So, I upgraded my instance to t2-small, which has 3 network interfaces, and it seems to have gotten rid of the timeouts and "RESOURCE:ENI" errors.


pull request for e2e test at 12/22/2016 13:56:18

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Lowercase the issuer URI if not specified explicitly

Great work, guys! I added a post in the Async issue that was probably off topic. Not trying to spam. I know you all are busy. So here it is:
Is using default collation inside a database when searching based on IS4 interfaces acceptable? There is no hint passed in from the implemented interfaces as to how to do string comparisons. Maybe this isn't a big deal, but I have been passing in StringComparison comparison = StringComparison.OrdinalIgnoreCase into other methods I have created. That allows different cultures to change how Entity Framework, for example, should test for equality. I am also not sure if it makes sense to pass in a Context object to these interfaces. AutoMapper, for instance, has re-written some of their methods to pass in a context providing more information.

For questions or comments that are new or unrelated, open new issues please.

I'll review the rfcs some more. I remember reading something about token comparisons.

done on dev


Detectable, generated and vendored attributes unexpected behaviour

Detectable, generated and vendored attributes are behaving differently from what they are supposed to do
Preliminary Steps
Please confirm you have...

[x] reviewed How Linguist Works,
[x] reviewed the Troubleshooting docs,
[x] considered implementing an override,
[x] verified an issue has not already been logged for your issue (linguist issues).

Problem Description
I'm not sure if it's just me unable to use linguist attributes properly, or it's a cache thing or it's a bug.
I have a repo containing mainly kicad and c sources. I have made this .gitattributes file:
*.kicad_pcb linguist-detectable=true
*.sch linguist-detectable=true
fp-lib-table linguist-generated=true
*.xml linguist-generated=true
*.txt linguist-generated=true
*.kicad_mod linguist-vendored=true

To:

Make kicad_pcb and kicad schematic files listed in the language breakdown
Hide fp-lib-table, xml and txt files as generated
Hide .kicad_mod files as vendored

As a result:

kicad_pcb are not shown in the language breakdown ❌
fp-lib-table, xml and txt files are hidden ✔
.kicad_mod files are shown as "kicad layout" in the language breakdown ❌

As of today i've made a small modification to the gitattributes file that may fix the kicad_mod problem, but the other problem should persist. As i said, i'm not totally sure about my usage of the attributes. Please excuse the issue if i'm using them wrong. The latest version states:
. . .
*.kicad_mod linguist-vendored=true
Hardware/customfootprints.pretty/* linguist-vendored

URL of the affected repository:
.gitattributes in 5N44P/Banana-Random-Number-Generator
Older version of .gitattributes in 5N44P/Banana-Random-Number-Generator
Last modified on:
2018-05-02
Expected behaviour:
Kicad_pcb detected and kicad_mod hidden
Current behaviour:
Kicad_mod detected and kicad_pcb hidden

I've checked your repo and everything is working as you expect from a Linguist perspective.  The only files that are contributing to your language stats are:

KiCad Layout:
Hardware/BRNG.kicad_pcb
KiCad Schematic:
Hardware/BRNG.sch

All other files are being ignored or not considered.
You can verify this locally too (this is with your old .gitattributes file in place locally):
$ cat .gitattributes
*.kicad_pcb linguist-detectable=true
*.sch linguist-detectable=true
fp-lib-table linguist-generated=true
*.xml linguist-generated=true
*.txt linguist-generated=true
*.kicad_mod linguist-vendored=true
$
$ linguist --breakdown .
91.51%  KiCad Layout
8.49%   KiCad Schematic

KiCad Layout:
Hardware/BRNG.kicad_pcb

KiCad Schematic:
Hardware/BRNG.sch

$

The issue that comes up is how you're verifying things. Clicking the language within the stats bar actually hands things over to the search side of things (note the URL) and for the most part, this is independent of Linguist. Linguist is only used to determine the language and calculate the repo breakdown statistics. Search will use the language info but will not honour the other Linguist gitattributes as they're not really relevant to search.
You can't hide files from search and wouldn't I think you'd want to either.
When it comes to creating a pull request, you should find your overrides will take effect as you expect then too.

@lildude Thank you! So i'm using the attributes right.
Is Hardware/customfootprints.pretty/* linguist-vendored a good practice?
So it is a search-related issue, breakdown bar percentage are confirmed.
But if that's the case:

Why fp-lib-table, XML and txt files are hidden in the search?
In this other repo, where the same .gitattributes are set, i can see .kicad_pcb and fp-lib-tables, .txt files, and xml files.
Why is kicad_pcb hidden or not recognized in the search?

Does this have any relevance on the linguist side?
Is this a github-linguist integration issue? I understand that we're a bit on the github-linguist dividing edge...

So i'm using the attributes right.

Yup 😄

Is Hardware/customfootprints.pretty/* linguist-vendored a good practice?

I couldn't really say as I know nothing about it 😄 but it's certainly the right thing to do if you don't it to count towards your language stats.

Why fp-lib-table, XML and txt files are hidden in the search?

This one is easy to answer: search won't index what's not there 😉.
Your new repo doesn't appear have any of those files from what I can see:
$ find ./Banana-Random-Number-Generator | egrep '.txt|fp-lib-tables|.xml'
$

It does however identify your LICENSE file as txt as it does for your omega-dock-new repo.

Why is kicad_pcb hidden or not recognized in the search?

Considerations for code search 😉. Your one .kicad_pcb file is 508.21 KB so exceeds the 384 KB indexing limit.
That reminds me, I really need to reword point 1 to make the size thing clearer as this is commonly missed.

Does this have any relevance on the linguist side?
Is this a github-linguist integration issue? I understand that we're a bit on the github-linguist dividing edge (well... linguist is a standalone project, i guess that's not technically correct to speak of a dividing edge)...

Nope cos of the explanation on each point above.

This one is easy to answer: search won't index what's not there wink.

Ok so it's totally my fault. Thank you for your help and for further clarifying the usage of the attributes
I'm closing this, but I've one more question left:
I'm trying to make my .gitattributes file as reusable as possible. Would something like *.pretty/* linguist-vendored work for classifying as vendored every file in a "something.pretty" folder as vendored?

I'm trying to make my .gitattributes file as reusable as possible. Would something like .pretty/ linguist-vendored work for classifying as vendored every file in a "something.pretty" folder?

I'm not sure off the top of my head. I know .gitattributes behaves slightly differently when it comes to wildcards in paths. I recall @Alhadis mentioning something about this difference recently, but can't find where right now.

@5N44P My comment was a bit long-winded, so I'll link instead of copy+paste:
https://github.com/github/linguist/issues/4102#issuecomment-381550820


Bluetooth: Controller: Consolidated fixes applicable for Zephyr v3.1-branch

Consolidated fixes applicable for Zephyr v3.1-branch.
Signed-off-by: Vinayak Kariappa Chettimada<EMAIL_ADDRESS>

Stale


[Security]: code injection in add_llm which can lead to full control of the online RAGflow server

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Branch name
main
Commit ID
main
Other environment information
No response
Actual behavior
Code injection which can lead to RCE in add_llm and attacker can compromise the online RAGflow server.
https://github.com/infiniflow/ragflow/blob/06dfb83529e0ab8aab26ee258208f037f509c6c2/api/apps/llm_app.py#L116
@manager.route('/add_llm', methods=['POST'])
@login_required
@validate_request("llm_factory", "llm_name", "model_type")
def add_llm():
    req = request.json
    factory = req["llm_factory"]

    if factory == "VolcEngine":
        # For VolcEngine, due to its special authentication method
        # Assemble volc_ak, volc_sk, endpoint_id into api_key
        temp = list(eval(req["llm_name"]).items())[0]
        llm_name = temp[0]
        endpoint_id = temp[1]
        api_key = '{' + f'"volc_ak": "{req.get("volc_ak", "")}", ' \
                        f'"volc_sk": "{req.get("volc_sk", "")}", ' \
                        f'"ep_id": "{endpoint_id}", ' + '}'

Use unsafe eval which can cause RCE and get full control of the RAGflow server.
Expected behavior
No code executed.
Steps to reproduce
To reproduce the security issue quickly, you can just use online RAGflow demo in `http://demo.ragflow.io/`. Attacker can get full control of the demo server by reversing a shell.

1. login and go to profile `http://demo.ragflow.io/user-setting/profile`.
2. Add `VolcEngine` model.
3. Inject the code in model name.

Please refer Additional information to see the screenshot.

Additional information
Add VolcEngine  model:

Code injection, my VPS received the request from the server, if the attacker want, it can reverse a shell to its VPS and gain full control of the server:

Thanks a lot for reminding.

The fix can be quite easy, just change the eval to str or ast.literal_eval. I can submit a pr to fix it.
So, how can I get a CVE based on this report?


Openscenario logic

Description
All sequences of actions now occur in parallel
ConditionGroups allow for multiple conditions to be evaluated in parallel for logical ORs
Fixes #365 Improve logical compliance with OpenSCENARIO intent
Where has this been tested?
This has not yet been tested
Possible Drawbacks
This could change the behaviour of existing scenarios including the order of events and triggering of conditions

This change is

I've marked this as draft since there are some additional informational comments I'd like to add but I'd like the logic to be testable and reviewable asap

Hi,
just two small comments. Please also fix the Travis issues (pylint is unhappy it seems). Thank you 👍

Hi,
I reworked the handling of the result_writer, as we had the same piece of code at different locations. Instead of adding the sequence check to the result_writer, you have to add it now to get_criteria() of the Scenario class. Thank you,


Error squigglies don't appear for zero-length diagnostics on empty line

See below:

Duplicate of #1688


Gemfile modification doesn't handle gemfiles in subdirectories

My guess is path is relative to the gemfile location, and if the gemfile is in a subdirectory, you would need ../../.. (or something).  It's probably safer to determine the absolute path to the current directory, and pass that instead of trying to calculate an appropriate relative path.
Found when trying to add Grape as an external test to Rack, by adding the following to Rack's config/external.yaml
grape:
  url: https://github.com/ruby-grape/grape
  command: bundle exec rspec --exclude-pattern=spec/integration/**/*_spec.rb
  gemfile: gemfiles/rack_3_1.gemfile

Unrelated to this, but probably also a bug, is that failures due to bad gemfile or missing command are not reported as errors in CI.  See https://github.com/jeremyevans/rack/actions/runs/9539758130/job/26290479234 , where it doesn't even execute grape's tests due to errors, but still reports CI is passing.

I did get grape's tests passing in Rack's external tests by not specifying a gemfile, but the above still appear to be issues.

Ah yeah this makes sense, I'll fix it.


[multistage] Support SumPrecision Aggregation function

Support sumPrecision in v2
Add integration test for it

Codecov Report

Merging #11162 (c935d7a) into master (35fb1d0) will increase coverage by 0.11%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master   #11162      +/-   ##
==========================================
+ Coverage    0.00%    0.11%   +0.11%
==========================================
  Files        2192     2153      -39
  Lines      118150   116058    -2092
  Branches    17889    17637     -252
==========================================
+ Hits            0      137     +137
+ Misses     118150   115901    -2249
- Partials        0       20      +20

Flag
Coverage Δ

integration2temurin11
?

integration2temurin17
?

integration2temurin20
?

unittests1temurin11
?

unittests2temurin17
0.11% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed
Coverage Δ

...che/pinot/segment/spi/AggregationFunctionType.java
0.00% <0.00%> (ø)

... and 73 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Revert OrcSuite to fix deployment build [databricks]

Reverts #8940, #8979.  Fixes #8982. Reopens #8763.
The new OrcSuite test added in is in integration_tests in order to pick up DBGen, but it is importing internal com.nvidia.spark classes which is not allowed in the integration_tests module (that's one of the main distinguishing points between tests/ and integration_tests/).
Long-term fix is to move OrcSuite to tests/ and make DBGen its own module that tests/ can import.  However to quickly fix 23.08 and reduce risk, reverting the changes in 23.08 and will do the proper fix in 23.10.

build