adam8bit/lion8bit for Sdxl bitsandbytes 0.38.1 cuda118 windows

#447
Old pr

it works well,just slow...

Does this let you train without loss going to nan? Related Issue: #523

i had met Nan，and use --no-half-vae

yes,and i am testing 0.39.1。I am contacting jlllll to fix the bug in 0.39.1, which can optimize the VRAM usage of SDXL.

Thank you for this! It looks good!
Just to confirm, is the DLL from jllllll's repository?

0.38.1 it works well for long time i used.
and 0.39.1 support for bf16 training,i will test it soon.

I like lion over adam
what are your parameters for training?
currently i am doing a training with Using rank adaptation algo: lokr
i used this preset

and how can i use the generated safetensors file? with diffusers can i use?
@sdbds
I would like to test your pr can you give me some more info please

I like lion over adam what are your parameters for training?
currently i am doing a training with Using rank adaptation algo: lokr
i used this preset

and how can i use the generated safetensors file? with diffusers can i use?
@sdbds
I would like to test your pr can you give me some more info please

i recommend #631 for testing
it can use full_bf16 and PagedLion8bit
(I test most save 75% memory on ,it very useful to SDXL with bigger batch size set)
i use these↓
--cache_text_encoder_outputs
--gradient_checkpointing
--full_bf16

I like lion over adam what are your parameters for training?
currently i am doing a training with Using rank adaptation algo: lokr
i used this preset

and how can i use the generated safetensors file? with diffusers can i use?
@sdbds
I would like to test your pr can you give me some more info please

i recommend #631 for testing it can use full_bf16 and PagedLion8bit (I test most save 75% memory on PageAdamW8bit,it very useful to SDXL with bigger batch size set)
i use these↓ --cache_text_encoder_outputs --gradient_checkpointing --full_bf16

Aweosme
So you test checkpoints with comfy ui? Can you share json please?

Also what learning rates and network ranks do you use for lion sdxl training
@sdbds

I like lion over adam what are your parameters for training?
currently i am doing a training with Using rank adaptation algo: lokr
i used this preset

and how can i use the generated safetensors file? with diffusers can i use?
@sdbds
I would like to test your pr can you give me some more info please

i recommend #631 for testing it can use full_bf16 and PagedLion8bit (I test most save 75% memory on PageAdamW8bit,it very useful to SDXL with bigger batch size set)
i use these↓ --cache_text_encoder_outputs --gradient_checkpointing --full_bf16

Aweosme
So you test checkpoints with comfy ui? Can you share json please?

https://civitai.com/models/104515/sdxlor30costumesrevue-starlight-saijoclaudine-lora
you can download any sample image form comfyui to load json workflow

Also what learning rates and network ranks do you use for lion sdxl training
@sdbds

sorry i was busy these days so have not test with lion

@kohya-ss I tried with this PR and sadly, it seems to suffer the same issue as it is mentioned on this issue: https://github.com/kohya-ss/sd-scripts/issues/523 (On Paged AdamW 8bit)
Also, I've tested on pure AdamW 8bit (bitsandbytes==0.35.0) with cu116 on SDXL and it worked, mentioned here https://github.com/kohya-ss/sd-scripts/issues/686
Hope that helps a little!

i recommend #631 for testing it can use full_bf16 and PagedLion8bit (I test most save 75% memory on PageAdamW8bit,it very useful to SDXL with bigger batch size set)
i use these↓ --cache_text_encoder_outputs --gradient_checkpointing --full_bf16

How did you achieve 75% memory reduction?
I am just testing the PagedAdamW8bit right now and it's basically using the same VRAM memory when training dreambooth and lora as regular AdamW/AdamW8bit or AdaFactor.
If you can tell me more about your setup or if you could show your script that would be very helpful as I am not seeing any difference between PagedAdamW8bit and AdamW/AdamW8bit.
Thanks!

what is the used default weight decay for lion optimizer atm?
I am doing heavy testing with it for SDXL dreambooth training atm
i have done over 24 different experimentation so far

what is the used default weight decay for lion optimizer atm?
I am doing heavy testing with it for SDXL dreambooth training atm
i have done over 24 different experimentation so far

for lion recommend wd=0.02

what is the used default weight decay for lion optimizer atm?
I am doing heavy testing with it for SDXL dreambooth training atm
i have done over 24 different experimentation so far

for lion recommend wd=0.02

Thanks a lot
After 30 diff experiments not satisfactory results
Do you have suggestion for sdxl dreambooth?


Issues/1710 404 page navigating to routes manually

This is linked to the https://github.com/provectus/kafka-ui/issues/1710

[ ] Breaking change? (if so, please describe the impact and migration path for existing application instances)

What changes did you make? (Give an overview)
I reverted https://github.com/provectus/kafka-ui/pull/1658 since the Backend does not serve the index.html from any other urls
The issue was not visible to the FE developers since the react node server always serves the html , while that is not the case after build.
Is there anything you'd like reviewers to focus on?
How Has This Been Tested? (put an "x" (case-sensitive!) next to an item)

[ ] No need to
[x] Manually (please, describe, if necessary)
[ ] Unit checks
[ ] Integration checks
[x] Covered by existing automation

Checklist (put an "x" (case-sensitive!) next to all the items, otherwise the build will fail)

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation (e.g. ENVIRONMENT VARIABLES)
[x] My changes generate no new warnings (e.g. Sonar is happy)
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged

Check out Contributing and Code of Conduct
A picture of a cute animal (not mandatory but encouraged)

@sasunprov yep @Mgrdich fixed this, dashboard link works fine now


"see-saw" latch

In various places, including service pool threads and the child context CALL_FUNCTION dispatcher, there is a pattern where a message is sent just as a message or object is fetched.
In the case of service pool, it sends its function return value message to the broker thread immediately prior to accepting the next request message which arrived from the broker thread. These operations currently involve separate locks and queues, and therefore context switches, writes and sleeps, and there may be an opportunity to combine them somehow.
The idea is simply that a lock is taken once, a new work item is popped, and a result item is pushed. The result push somehow triggers (maybe like select.Select) wakeup of notification code.
Before (ideal case): lock.acquire(), list.append(), waker sock.write(), lock.release(), lock.acquire(), list.pop(), lock.release()
After: lock.acquire(), list.append(), list.pop(), waker sock.write(), lock.release()
This is another case where core.Waker's Latch-but-not-quite-Latch design is annoying. If common functionality was combined somehow, one implementation of the above concept could serve in two places.
This also ties into something not ticketed yet -- latch's queuing fairness is actually worse for performance (latency, cache, context switches, ..). The Latch.get() caller that is awake now should always receive an item if one is available, rather than being forced to sleep because another previously-sleeping thread hasn't yet woken far enough to pop the item

Might revisit this later, but would rather do threadless mode #534 first, it's likely to provide a much more general win than the idea here


IIS installation

Is it possible to deploy Raneto to an IIS server? if so, how can i do it?
Thanks.

yes you can!
any OS that runs NodeJS can, as far as I know the navigation filesystem is fullcompatible with Windows and Unix systems 😄
it's just a matter of configuring the IIS server with the right dependencies and tweaks for the language

As long as you configure Raneto to run on a different port to IIS the two can run side-by-side quite nicely. I wrote a little tutorial based on my experience of setting up Raneto to run under a Windows Service and it's quite happy on the same box as IIS: https://collection.code.blog/2018/12/01/running-raneto-as-a-windows-service/

Thanks both for your anwsers..
The question is that i have limited access to the server. I can only access the iis window. I was expecting to have de possibility to somehow build a package and then deploy it.
Is it possible? I've looked into webpack but so far without any luck :(

It is possible to run node.js applications in IIS (see https://github.com/Azure/iisnode, for example), but without access to configure the server beyond just IIS I don't think you're going to have much joy, unfortunately.

Well. At least on my machine i can proceed with it :)
Also, the final version will be published on Azure, so if iisnode already runs on Azure I think there is no problem there.
But looking at iisnode, i'm a little bit stuck on how to build it or use it..

Thank you for the discussion, I'll mark this as "add-to-docs" for inclusion in the Raneto guides on how to host on Windows/IIS.

Closing to clean up and moving to megathread for documentation update (see above).
Would any of you be able to assist in testing? I do not have IIS experience.
If you're available to help, just comment here and re-open. Thanks


Performance significantly worse than building from a terminal

Hi, I'm trying atom from another editor where building from the editor is standard and ran into this package.
I've noticed that build times are significantly longer, at least on a project I build often (the Zephyr RTOS, https://www.zephyrproject.org/, building on Arch Linux with an ARM cross toolchain provided by Zephyr).
Here's my build file (which just times how long it takes to build one of the sample applications):
cmd: time
name: "shell"
args:
  - make
  - -C
  - samples/shell
  - -j8
sh: true,
cwd: /home/mbolivar/src/zephyr
env:
        ZEPHYR_GCC_VARIANT: zephyr
        ZEPHYR_SDK_INSTALL_DIR: /home/mbolivar/bin/zephyr-sdk
        ZEPHYR_BASE: /home/mbolivar/src/zephyr
        PATH: /usr/lib/ccache/bin/:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/mbolivar/bin:/home/mbolivar/bin/arm-none-eabi/bin:/home/mbolivar/bin/arm-eabi/bin:/home/mbolivar/.local/bin:/home/mbolivar/src/zephyr/scripts

The environment variables are needed by the build system; normally you'd do:
. zephyr-env.sh
make -C samples/shell -j8

Results when building from within atom are consistently around 10-12 seconds of wall clock as reported by "time".
Here's an example when building from atom:
make: Entering directory '/home/mbolivar/src/zephyr/samples/shell'
make[1]: Entering directory '/home/mbolivar/src/zephyr'
make[2]: Entering directory '/home/mbolivar/src/zephyr/samples/shell/outdir/qemu_x86'
  Using /home/mbolivar/src/zephyr as source for kernel
  GEN     ./Makefile
  CHK     include/generated/version.h
  CHK     include/generated/offsets.h
  CHK     misc/generated/configs.c
  CHK     misc/generated/sysgen/prj.mdef
make[2]: Leaving directory '/home/mbolivar/src/zephyr/samples/shell/outdir/qemu_x86'
make[1]: Leaving directory '/home/mbolivar/src/zephyr'
make: Leaving directory '/home/mbolivar/src/zephyr/samples/shell'

real    0m12.265s
user    0m7.187s
sys     0m4.037s

Building from the shell is significantly faster, almost exactly 2.7 seconds on the nose each time.
dopey: zephyr (master) mbolivar$ time make -C samples/shell -j8
make: Entering directory '/home/mbolivar/src/zephyr/samples/shell'
make[1]: Entering directory '/home/mbolivar/src/zephyr'
make[2]: Entering directory '/home/mbolivar/src/zephyr/samples/shell/outdir/qemu_x86'
  Using /home/mbolivar/src/zephyr as source for kernel
  GEN     ./Makefile
  CHK     include/generated/version.h
  CHK     misc/generated/configs.c
  CHK     include/generated/offsets.h
  CHK     misc/generated/sysgen/prj.mdef
make[2]: Leaving directory '/home/mbolivar/src/zephyr/samples/shell/outdir/qemu_x86'
make[1]: Leaving directory '/home/mbolivar/src/zephyr'
make: Leaving directory '/home/mbolivar/src/zephyr/samples/shell'

real	0m2.738s
user	0m0.640s
sys	0m0.237s

I took a couple of javascript timelines. Some of the slowness can be attributed to the spawn calls when starting the build from within atom taking about a second or so each. But the extra time is confusing -- the CPU is idle a lot of the time.
Any ideas on how I could diagnose further and maybe speed this up? I'd really like to use this extension (and Atom in general) but this is a significantly slower build at the moment.
Thanks!

That is very odd indeed. I have not experienced this.
What OS are you on?

Arch Linux.

I'm not sure if this is because of atom-build, or because of Node. It's worth trying to make a javascript file only calling this build command and timing that

OK, I've re-run the measurements, including with a Javscript file that runs the commands. I've updated Atom and other system software, so all of these measurements are fresh.
In summary, I find that:

atom-build is 5.5x slower than command line on clean builds

running javascript directly is only 1.1x slower than command line on clean builds

atom-build is 1.9x slower on warm builds

running javascript directly is just as fast as command line builds on warm builds

Here is my Atom etc. version:
atom --version
Atom    : 1.15.0
Electron: 1.4.15
Chrome  : 53.0.2785.143
Node    : 6.5.0

atom-build is at version 0.67.0.
Here's the JS I ran (please forgive my naive Javascript):
#!/usr/bin/env node

exec = require('child_process').exec
exec('time make -C samples/shell -j8',
        { 'cwd': '/home/mbolivar/src/zephyr',
          'env': { 'ZEPHYR_GCC_VARIANT': 'zephyr',
                   'ZEPHYR_SDK_INSTALL_DIR': '/home/mbolivar/bin/zephyr-sdk',
                   'ZEPHYR_BASE': '/home/mbolivar/src/zephyr',
                   'PATH': '/usr/lib/ccache/bin/:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/mbolivar/bin:/home/mbolivar/bin/arm-none-eabi/bin:/home/mbolivar/bin/arm-eabi/bin:/home/mbolivar/.local/bin:/home/mbolivar/src/zephyr/scripts'
                 }
        },
        (error, stdout, stderr) => {
                if (error) {
                        console.error(`error ${error}`)
                }
                console.log(`${stdout}`);
                console.log(`${stderr}`);
        });

For better comparison with atom-build, I ran this from within Atom, with atom-runner 2.7.1:
https://atom.io/packages/atom-runner
I ran each build three times, and took the median wall-clock time. I left the Atom runner window open between runs, as that seemed to cut down on ~1 second overhead needed to create the window and populate it, etc. This doesn't seem to be possible with atom-build.
Here is a summary table (with relative speeds to one decimal place for real / wall-clock time) for clean builds:

Real (s)
Relative

Command line
5.600
1.0

atom-runner
6.168
1.1

atom-build
30.971
5.5

Here is the same table for warm builds (note atom-runner's median time was very slightly faster):

Real (s)
Relative

Command line
2.743
1.0

atom-runner
2.739
1.0

atom-build
5.134
1.9

Full details for median runs:
Build from command line, clean build (delete build directory between runs, but in-memory file cache is still warm):
real	0m5.600s
user	0m1.797s
sys	0m0.653s

Build with atom-runner, clean build:
real	0m6.168s
user	0m5.093s
sys	0m0.917s

Build with atom-build, clean build:
real    0m30.971s
user    0m19.707s
sys     0m9.967s

Build from command line, warm build (no files changed, object files available in build directory):
real	0m2.743s
user	0m0.653s
sys	0m0.253s

Build from Javascript, warm build:
real	0m2.739s
user	0m0.767s
sys	0m0.350s

Build with atom-build, warm build:
real    0m15.134s
user    0m9.313s
sys     0m5.053s

I speculate that all of the live UI updates atom-build is doing as the build is ongoing may account for some of the slowdown, since the running the Javascript file directly and then printing the final output should share much of the same Atom-specific overhead, but is still much faster.

Update: UI updates in atom-build do account for the slowdown.

Variation 1
I tried a similar .atom-build.yml that:

Forces a clean build every time
Silences all build output except the final time values

Here it is:
cmd: rm -r samples/shell/outdir/; time make -C samples/shell -j8 > /dev/null
name: "shell"
sh: true
cwd: /home/mbolivar/src/zephyr
env:
        ZEPHYR_GCC_VARIANT: zephyr
        ZEPHYR_SDK_INSTALL_DIR: /home/mbolivar/bin/zephyr-sdk
        ZEPHYR_BASE: /home/mbolivar/src/zephyr
        PATH: /usr/lib/ccache/bin/:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/mbolivar/bin:/home/mbolivar/bin/arm-none-eabi/bin:/home/mbolivar/bin/arm-eabi/bin:/home/mbolivar/.local/bin:/home/mbolivar/src/zephyr/scripts

The median of three runs by wall clock time is:
real    0m8.182s
user    0m3.753s
sys     0m1.047s

This brings atom-build down from 5.5x slower to 3.1x slower than the command line.

Variation 2
With the same build YML as variation 1, I turned off the running timer from the build view with this patch:
 diff -uN lib/build-view.js.orig lib/build-view.js
--- lib/build-view.js.orig	2017-03-16 23:40:53.940183853 -0400
+++ lib/build-view.js	2017-03-16 23:40:47.743517256 -0400
@@ -281,8 +281,8 @@
   }

   updateTitle() {
-    this.buildTimer.text(((new Date() - this.starttime) / 1000).toFixed(1) + ' s');
-    this.titleTimer = setTimeout(this.updateTitle.bind(this), 100);
+    // this.buildTimer.text(((new Date() - this.starttime) / 1000).toFixed(1) + ' s');
+    // this.titleTimer = setTimeout(this.updateTitle.bind(this), 100);
   }

   close() {

The median was then:
real    0m6.974s
user    0m2.827s
sys     0m0.790s

This brings atom-build down from 3.1x slower to 1.2x slower than command line, which is comparable to the 1.1x slowdown from executing the javascript using atom-runner.

Conclusion the 5.5x slowdown can be explained by UI updates in atom-build taking a long time.
Any suggestions on where to go from here? I wonder if the code in build-view.js can be optimized. A 1.1x or 1.2x slowdown is OK, but 5.5x is unusable for me.

Thanks a lot! This will probably speed up the fixing progress. Could you also run a javascript timeline to show us exactly what takes long?

Thank you for the very detailed and in depth investigation.
We definitely need to to do something about build-view. I'm also pretty sure the terminal emulator I'm using (modified version of term.js) is slow as well as it constantly checks and re-renders control characters common in terminals.
While these may be worthwhile features I think an alternative must be provided which has better performance.
I've been thinking about it for a while and every report I get regarding this indicates that the solution has to be an ability to choose between build views. I'll be looking into extracting the build view component and then have an API integrating with it. Settings in atom-build should allow selecting which View to use.
Once again. Thanks for the report.

Sure thing! This is a 'killer' feature for me and I'm very interested in Atom, so I'm glad it's useful.
I'll try to find some time this weekend to take a Javascript timeline.

I am having similar performance issues due to UI updates and the above mentioned term.js.
CPU load greatly increases when a build command is running, and CPU intensive builds (e.g. cpp) see their build time explode.
Is there any progress on the approach proposed above for a lightweight build-view implementation? Thanks!


Network randomly goes unstable with no commands being received or sent

Is your problem within Home Assistant (Core or Z-Wave JS Integration)?
NO, my problem is NOT within Home Assistant or the ZWave JS integration
Is your problem within Z-Wave JS UI (formerly ZwaveJS2MQTT)?
NO, my problem is NOT within Z-Wave JS UI
Checklist

[X] I have checked the troubleshooting section and my problem is not described there.

[X] I have read the changelog and my problem was not mentioned there.

Describe the bug
What causes the bug?
I'm not able to track down specifically what is causing the issue. But consistently at arbitrary times (sometimes it takes 1 day, sometimes it takes 3, but definitely happens at least 1x a week) in the last month, I've noticed that the entire network stops receiving or sending any commands.
What do you observe?
When this happens, I see the Dropped RX Messages in Z-Wave JS UI climb up constantly (as sensors are trying to provide updates). Looking in the log file every incoming message has the following error:
2023-09-10T18:27:30.500Z SERIAL « 0x010e0004003705310505012fbf000067                                  (16 bytes)
2023-09-10T18:27:30.500Z DRIVER   Dropping message because it could not be deserialized: The command class unkno
                                  wn (0x05) is not implemented (ZW0303)
2023-09-10T18:27:30.501Z SERIAL » [ACK]                                                                   (0x06)

Once this occurs, commands from Z-Wave JS UI (or Home Assistant) to the network do not get received as well.
Going into Z-Wave JS UI settings and clicking save (which reloads Z-Wave JS) fixes the problem immediately.
What did you expect to happen?
Messages should continue to be processed and connectivity should be fine.
Steps to reproduce the behavior:
I do not have a reproducible way of doing so, it just happens on it's own, psuedo randomly.
Device information
N/A
How are you using node-zwave-js?

[X] zwave-js-ui (formerly zwavejs2mqtt) Docker image (latest)
[ ] zwave-js-ui (formerly zwavejs2mqtt) Docker image (dev)
[ ] zwave-js-ui (formerly zwavejs2mqtt) Docker manually built (please specify branches)
[ ] ioBroker.zwave2 adapter (please specify version)
[ ] HomeAssistant zwave_js integration  (please specify version)
[ ] pkg
[ ] node-red-contrib-zwave-js (please specify version, double click node to find out)
[ ] Manually built from GitHub (please specify branch)
[ ] Other (please describe)

Which branches or versions?
version:
zwave-js-ui: <IP_ADDRESS>058e8
zwave-js: 11.13.0
Did you change anything?
no
If yes, what did you change?
No response
Did this work before?
Yes (please describe)
If yes, where did it work?
I've run with the same Z-Wave JS setup for several years now. I did add some new devices from Inovelli (VZW31-SN) a couple of months ago (July). I had previously been running a beta version of this device with no issues since December.
Attach Driver Logfile
zwavejs_2023-09-10.log

Looks like the controller starts hanging in the middle of a transmission, then restarts itself. However, Z-Wave JS still thinks the controller is communicating with 16-bit node IDs, although it reverted to 8-bit IDs after the restart.
2023-09-10T11:27:13.003Z DRIVER « [REQ] [SerialAPIStarted]
                                    wake up reason:        SoftwareReset
                                    watchdog enabled:      false
                                    generic device class:  0x02
                                    specific device class: 0x01
                                    always listening:      false
                                    supports Long Range:   false
2023-09-10T11:27:13.003Z DRIVER   handling request SerialAPIStarted (10)
2023-09-10T11:27:13.004Z DRIVER     no handlers registered!
2023-09-10T11:27:28.306Z SERIAL « 0x01130004004d0a32022174000000000000ad000067                        (21 bytes)
2023-09-10T11:27:28.306Z DRIVER   Dropping message because it could not be deserialized: The command class unkno
                                  wn (0x02) is not implemented (ZW0303)
2023-09-10T11:27:28.306Z SERIAL » [ACK]                                                                   (0x06)
2023-09-10T11:27:49.052Z SERIAL « 0x011b0004006c129f030100367791c58df8df8aff9dfca91636a3000082        (29 bytes)
2023-09-10T11:27:49.053Z DRIVER   Dropping message because it could not be deserialized: The command class unkno
                                  wn (0x03) is not implemented (ZW0303)
2023-09-10T11:27:49.053Z SERIAL » [ACK]                                                                   (0x06)
2023-09-10T11:27:49.157Z SERIAL « 0x01180004006c0f9f030000629ad5a2f5ae6f36c468c5a8000050              (26 bytes)
2023-09-10T11:27:49.158Z DRIVER   Dropping message because it could not be deserialized: The command class unkno
                                  wn (0x03) is not implemented (ZW0303)
2023-09-10T11:27:49.158Z SERIAL » [ACK]                                                                   (0x06)
2023-09-10T11:27:58.306Z SERIAL « 0x01130004004d0a32022174000000000000ad000067                        (21 bytes)
2023-09-10T11:27:58.306Z DRIVER   Dropping message because it could not be deserialized: The command class unkno
                                  wn (0x02) is not implemented (ZW0303)

Nice, Just updated to it. I'll report back if the problem reoccurs
Thanks!


Fix: Abort the upload-RemoteFile if pscp is stuck

Bug #176
This issue is due to PSCP gets stuck in Hyper-V platform, which causes timeout delay (600 seconds) for each upload retry
Reference log for PSCP as stuck:
PS F:\prashant\upload_lisav2\lisav2> .\Tools\pscp -2 -v -pw '####' -P 22 '.\README.md'<EMAIL_ADDRESS>Connecting to <IP_ADDRESS> port 22
We claim version: SSH-2.0-PuTTY_Release_0.70
Code changes fixes this issue with having a check of PSCP console log and aborts the file upload.
The job 554 (pipeline-lis-rpm-hyperv-lisav2-test) shows the this failure scenario
[centos_6.5_x64-ws2016] 08/26/2019 11:08:47 : [INFO ] dos2unix: converting file C:\LISAv2\ZC88\TestResults\2019-26-08-03-52-03-2266\VMBUS_PANIC_NOTIFIER\runtest.sh to Unix format...
[centos_6.5_x64-ws2016] 08/26/2019 11:08:47 : [INFO ] Uploading C:\LISAv2\ZC88\TestResults\2019-26-08-03-52-03-2266\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : <IP_ADDRESS>, port 22 using Password authentication
[centos_6.5_x64-ws2016] 08/26/2019 11:09:47 : [INFO ] .\Tools\pscp is stuck for 60 seconds. Aborting upload.
[centos_6.5_x64-ws2016] 08/26/2019 11:09:47 : [INFO ] dos2unix: converting file .\Testscripts\Linux\CollectLogFile.sh to Unix format...

Please verify big file which takes more than 60 seconds to upload.

Please verify big file which takes more than 60 seconds to upload.

Verified with uploading large file which take 60+ seconds
PS F:\prashant\upload_lisav2\lisav2> date; Upload-RemoteFile -uploadTo <IP_ADDRESS> -port 22 -file F:\prashant\rhel7.1x64.vhdx  -username lisa -password '#####' -maxRetry 5; date
Monday, August 26, 2019 6:43:46 AM
< -- Removed Write-LogInfo warnings -- >
Write-LogInfo "Upload successful after $retry attempt"
Monday, August 26, 2019 6:45:39 AM

Please run negative and positive cases on both hyperv and azure platform, thanks!


Rename raw_t to any_value_t and introduce encoded_value

As described in #16, but without changing the return type of json::encode. Instead I introduced a new function json::encode_value that returns an encoded_value. The reason for this is that I ran into what I believe is a bug in the Clang compiler, getting confused about which implicit conversion operator to pick when one of them returns a const reference and the other one a non-const rvalue reference. The code compiled fine on GCC (and ICC). I tried so so many things, but ended up simply making two functions. Even if Clang would fix this in the next version, we still want to maintain compatibility with (somewhat) older compilers.

For review. I still need to update the documentation.

Plus, I haven't updated any_value_t to use the new encoded_value type.

This can be reviewed now, @pereckerdal @fxb @emil-e

I think we might want to add an implicit constructor that takes a T and creates an encoded_value<T>. Then we could support the use case of decoding into e.g., a std::string, like Felix wanted to do for Cosmos over WebSockets.


Ubuntu WSL2 install issue 0x80370102 got resolved

Like quite many, my attempt to install WSL2 on a brand new Win 11 (Lenovo Tablet) gave the sad message:
WslRegisterDistribution failed  with error: 0x80370102
Please enable the Virtual Machine Platform Windows feature and ensure virtualization is enabled in the BIOS
Quite some searching, cause the error code seems used for several things, I found that this page proved useful to me:
https://woshub.com/powershell-view-change-bios-settings/
I started my PowerShell 7, in Admin mode and entered (since mine was Lenovo):
get-wmiObject -class Lenovo_BiosSetting -namespace root\wmi | select-object InstanceName, currentsetting
and voila, I found that I missed one setting:
ACPI\PNP0C14\1_48  VirtualizationTechnology,Disable
I believed i did change Virtualization stuff in my BIOS, but I did not see that one setting was in the unexpected BIOS category : Security
After enabling, my Ubuntu  22.03 WSL2 began finishing its installation, and two minsn after. it was done.
Thanks sofar, for clear messages around requirements, although there are many.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 4fb08c5b-865d-26b5-0867-de14e87a205c
Version Independent ID: 5eacc1db-a948-722b-d3d5-ae0f281e9e30
Content: System requirements for Hyper-V on Windows Server
Content Source: WindowsServerDocs/virtualization/hyper-v/System-requirements-for-Hyper-V-on-Windows.md
Product: windows-server
Technology: hyper-v
GitHub Login: @BenjaminArmstrong
Microsoft Alias: benarm

We hugely appreciate your contribution, feedback, and support to our technical documentation.
We’ve phased out GitHub Issues on this repository as the feedback mechanism for content and have replaced it with a new feedback system. For more information see, Content User Feedback. As part of this rollout, we are migrating your issue where our team will use your comments to improve the documentation experience. Unfortunately, this does mean you won't be able to access this issue after the migration is complete.
All issues will be migrated by Thursday 29th August 2024.
Once again, thank you for your valuable feedback, we hope you’ll continue to contribute using our new feedback system.


[Feat]: Create Issue forms and make the issue options more visually appealing

Currently, there is an issue template for this repo, but no issue form which will help users in contributing.

I will add Issue Forms to the respective issues to make it easier for the contributor to raise it.

Here is an example of how a Bug Issue Form will look like 👇

and issue templates:

I want to work on this issue. So, kindly assign this issue to me ( GSSOC'24 ).

@TonyStark-47 okk, start working on it

Hey @sethshivam11, I have raised a PR #62  related to this issue. Please check it out.


eval is not in parallel while traning

Hi, thx for this great work!
I found when evaluating on the training stage, the evaluating is not gpu parallel, and is very slow compare to training time, have you have any idea of how to fix it?

the eval code has some redundant operation, i will improve it later.

val set evaluation speed has been fast


[PROFILING] Add ability to profile a single function_profiling

Add a new function tvm.runtime.profiling.profile_function which collects performance metrics for a single function in an IRModule. For example, collecting performance counters using PAPIMetricCollector. This is helpful for optimizing kernels and schedules for a single operator.

@areusch Could you re-review?

@areusch gentle ping


Perf: GetMany doesn't use ContentManagerSession

https://github.com/OrchardCMS/Orchard/blob/610b3c4f53ce9a4bdbe26a5a095bd606bfabc811/src/Orchard/ContentManagement/DefaultContentManager.cs#L284
I was checking the performances of some test pages. In one of them I have a bunch of MediaLibraryPickerFields. Since this is only a test tenant on a dev box, all those fields have the same Media items selected.
I noticed that the query corresponding to the LazyField fetching the information for the Media from the db (https://github.com/OrchardCMS/Orchard/blob/610b3c4f53ce9a4bdbe26a5a095bd606bfabc811/src/Orchard.Web/Modules/Orchard.MediaLibrary/Handlers/MediaLibraryPickerFieldHandler.cs#L40) is fired for every field, always with the same parameters.
This is unlike fetching ContentItems one by one using the Get method, where the ContentManagerSession prevents refiring the same query multiple times.
Looking at the code, I think ContentPickerFields have the same issue.
I think GetMany should only be doing the query if any of the Ids provided to it corresponds to a ContentItem that IS NOT available in ContentManagerSession. It should still, of course, not change the order of returned items.

I think GetMany should only be doing the query if any of the Ids provided to it corresponds to a ContentItem that IS NOT available in ContentManagerSession

Correct. I am pretty sure we do that in OC.

Further testing: TaxonomyField seems to be affected by the same issue.

Has anyone looked more into this yet maybe with a PoC change? I would love to have this fixed!

I haven't had a chance yet. Hopefully over the next few weeks.

I did some test during a couple meetings.
I think I have a working version, but I'm not sure about its performances.
I'll draft a PR early next week.

Sounds great! I'll test it with an application that uses Taxonomies extensively once you've got the PR submitted.


Investigate about:srcdoc errors

Some times we get debugger permission errors which freeze the extension.

Repro: go to https://www.w3schools.com/jsref/event_onscroll.asp and reload a few times, eventually all requests will be stuck in "sent"

Repro: go to https://www.w3schools.com/jsref/event_onscroll.asp and reload a few times, eventually all requests will be stuck in "sent"

error happens on Fetch.continueRequest of "https://www.google.com/images/cleardot.gif"
I don't see anything special in that request.
0: {name: "Accept", value: "image/avif,image/webp,image/apng,image/*,*/*;q=0.8"}
1: {name: "Cookie", value: "CONSENT=YES+ES.en+20150607-14-1; __Secure-3PSID=..."}
2: {name: "Referer", value: "https://www.w3schools.com/"}
3: {name: "User-Agent", value: "Mozilla/5.0 (X11; CrOS x86_64 13505.73.0) AppleWeb…L, like Gecko) Chrome/87.0.4280.109 Safari/537.36"}
4: {name: "sec-ch-ua", value: ""Google Chrome";v="87", " Not;A Brand";v="99", "Chromium";v="87""}
5: {name: "sec-ch-ua-mobile", value: "?0"}

error happens on Fetch.continueRequest of "https://www.google.com/images/cleardot.gif"
I don't see anything special in that request.
0: {name: "Accept", value: "image/avif,image/webp,image/apng,image/*,*/*;q=0.8"}
1: {name: "Cookie", value: "CONSENT=YES+ES.en+20150607-14-1; __Secure-3PSID=..."}
2: {name: "Referer", value: "https://www.w3schools.com/"}
3: {name: "User-Agent", value: "Mozilla/5.0 (X11; CrOS x86_64 13505.73.0) AppleWeb…L, like Gecko) Chrome/87.0.4280.109 Safari/537.36"}
4: {name: "sec-ch-ua", value: ""Google Chrome";v="87", " Not;A Brand";v="99", "Chromium";v="87""}
5: {name: "sec-ch-ua-mobile", value: "?0"}

Full error is Cannot access contents of url "about:srcdoc". Extension manifest must request permission to access this host.

Full error is Cannot access contents of url "about:srcdoc". Extension manifest must request permission to access this host.

a simpler testcase is http://evilwebsite.com/xss.php?plain_xss=%3Ciframe%20srcdoc%3E something breaks with srcdoc

a simpler testcase is http://evilwebsite.com/xss.php?plain_xss=%3Ciframe%20srcdoc%3E something breaks with srcdoc

this is not related to activeTab, requesting with <all_urls> fails as well.

this is not related to activeTab, requesting with <all_urls> fails as well.

requesting about:srcdoc permission explicitly doesnt work either

requesting about:srcdoc permission explicitly doesnt work either

looks like a chrome bug.

looks like a chrome bug.

seems related to https://bugs.chromium.org/p/chromium/issues/detail?id=55084 - but will create a new bug

seems related to https://bugs.chromium.org/p/chromium/issues/detail?id=55084 - but will create a new bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1162935

https://bugs.chromium.org/p/chromium/issues/detail?id=1162935

https://chromium-review.googlesource.com/c/chromium/src/+/2692548 fixes this


Config option to have workers raise SignalException on SIGTERM and SIGINT

This adds the config option Delayed::Worker.raise_signal_exceptions, changing the workers behavior on receiving SIGTERM and SIGINT (but defaulting to the current behavior).
Currently, they set @exit, and terminate after the current job completes. With this option enabled, they'll do the same thing, but also raise the appropriate SignalException.
Relevance
On heroku, when workers are being terminated or restarted for any reason (and unix in general, on shutdown - http://en.wikipedia.org/wiki/SIGTERM), a SIGTERM is sent, followed a few seconds later by a SIGKILL. If the worker is performing a task that takes longer than those few seconds, it will get hard-killed, not running any cleanup exception handling that may be necessary.

I guess there's not much interest in this. I've switched most of my workers from heroku over to ec2 anyway, so I'm closing the request.


New user guide to jar_jar and bazel-deps

Preface: I'm new to bazel, bazel-deps and jar_jar. I've tried reading the available documentation but haven't found satisfactory answers suitable for a new user.
Problem:
We use bazel-deps to download and use a bunch of 3rd party dependencies. Two of these libraries (azure-storage and cassandra-driver) have conflicting dependencies on the version of guava that they require (20 and 19 respectively). I'm trying to use bazel_jar_jar to shade the azure-storage library so that I can have 2 different versions of guava coexist.
Questions:

Is there a way to make bazel_jar_jar play nice with bazel-deps? Bazel deps seems to generate it's own BUILD.bazel files for every dependency specified. There's a warning saying that we shouldn't edit these files manually, which I would have to do to use jar_jar. I see a bunch of options in the documentation but it's not clear if any of them can help setting up bazel_jar_jar rules
Could someone please give point me to a beginner friendly guide/documentation which I can follow to get some more insight

bazel-deps wants to normalize all version jars down to 1 single jar. There isn't a mechanism to nave separate ones.
What I would do if I were trying to solve this would be to have two configuration files:

your main/default versions.
the minimal set you need for the exceptional cases.

then generate two separate third party directories, and then you could use jar_jar to shade one of them.
We use shading in a few cases at Stripe, but we usually manage this by manually adding those jars via http_jar or similar and constructing by hand the target we need. It's not fun, but the goal of the monorepo is a single version of all jars, so we work more to get there than to solve workarounds for mixing versions.
I hope this is helpful.
Good luck.

Thanks for the insight! My current approach was to do it by hand, but will give what you suggested a shot as well.
There doesn't really seem to be a clean way around this, since the issue is inherently messy.


Wait for a file to be stable before computing diff

Signed-off-by: Sebastian Malton<EMAIL_ADDRESS>fixes https://github.com/lensapp/lens/issues/3405

@jakolehm Done, this PR just has the stability check now.


Notes become left-indent

Hi there, I notice that all the notes became left-indent instead of centralized view (default) after I have updated the recent patch of yours. May I know is this on purpose or?

@AirconLah  oh huh, could you show me with a screenshot what you mean exactly?

Currently it looks like this:

Before the upgrade (this is not a solidate theme, but rather to demonstrate the differences):

this before the update:

Ooooh I see what you mean now. The first picture is how it used to be in my theme until they reworked themes, which broke it. I now changed it back, but I see you prefer the original design with large paddings.
Hmm, I think I can provide you with a css snippet you can enable to restore that.

That would be fantastic because I've been used to the original design for a long time and would love to have the original version.
How can I obtain your CSS snippet? =)

I'll write it sometime soon™, upload it to the repo and I'll @ you when it's there :)

Thanks mate! You have been a great help! :)

@AirconLah sorry it took me a while, but this snippet should make the content 700px wide as your screenshot has it https://github.com/tomzorz/Sodalite/blob/main/snippets/content-max-width-700.css - I've also added a 900px and 1100px version


Unable to edit server settings after a migration

I've just migrated my paperless server which meant the machine had a new IP. When launching the app, it's unable to connect, but unless I just wasn't patient enough the connection never seems to time out and allow me to alter server settings.
Workaround was to uninstall and reinstall the app and go through onboarding again. Worked fine, but not ideal :)

Hey @gstevenson,
Thanks for reporting this. I assume where you got stuck is this top level loading scree? You're right that if it gets stuck there you can't actually get to the log out button.
I'll have to come up with a solution for this!


Missing Visible focus indicator after Close Button

Description
Visible focus indicator is lost after moving from the close button. The user needs to press two missing tabs to reach the next focussable link "gdrive"
To Reproduce
Steps to reproduce the behavior:

Go to URL https://main--franklinsidekickrepo--puja-iaccessible.hlx.page/
Use the sidekick extension
Use the tab to reach the close button
After reaching the close button
Press tab and notice that the visible focus indicator is missing.

Expected behavior
Current focus should always be visible on the screen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
https://www.w3.org/TR/WCAG21/#focus-order
Why it matters?
When users navigate through a web page, they expect to encounter controls and other content in an order that makes sense and makes it easy to use the page's functionality. Lost or Poor focus order can be disorienting to people who use screen readers or screen magnifiers and to people with reading disorders. Lost or Poor focus order can also make it difficult or even painful for people who use keyboards because of mobility impairments.

@puja-iaccessible the tab index follows the natural DOM order. After the close button, you leave the scope of the sidekick and continue on to the first focusable element on the underlying web page. By default, this link may be hidden underneath the sidekick (see also #252).

Closing this as duplicate of #252.


Update Getting Started with new example repo

Change the Getting Started example from dps-unit-test to dps_tutorial (https://github.com/MAAP-Project/dps_tutorial)

done


[BUG] no POS user and passs

We don't support cracking communities here, if you want to use OpenBullet for that purpose, figure it out yourself.

Also you could simply do some research before posting the same question 32789 people asked before you https://forum.openbullet.dev/topic/8/environment-ini-a-common-problem


Add complete build option to package.json

The current build process requires executing a number of npm commands in a particular sequence, this can be simplified with an additional script in package.json such as
    "build": "npm run build:contracts && npm run build:sdk && npm run build:cli && npm run build:web",

in the # Building section, add documentation such as
For a first time build, you may build all the modules in one go as follows:
  npm run build

the "npm install" command was added to the root "package.json" file.


Add ability to preload words

Add ability through the admin to preload the tweets through the admin panel without the tweet content or url yet. Then, in the daily kick off process, check if that day/word already exists in the database. If so, go to Twitter to get the url and content and update the record and make it visible. If not, create the record from Twitter.

All queries have been updated to support this and not leak any words, so now "all" that's needed is the admin panel.


Properties: Unnecessary character removed.

This is my mistake in my pr #211, which I found during unicode instruction generator implementation.

Thanks.


how to change the code snippet insert shortcut?

how to change the shortcut from double click tab to just click enter?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ba6dc7b9-f7d2-9cf3-d0ce-70066af7daa9
Version Independent ID: 053a95f0-ab8c-e684-dccc-1e6f4063a3ce
Content: C# code snippets - Visual Studio
Content Source: docs/ide/visual-csharp-code-snippets.md
Product: visual-studio-windows
Technology: vs-ide-general
GitHub Login: @jillre
Microsoft Alias: jillfra

@felixmaomao Thank you for the question.
Are you referring to "...pressing Tab at any time will dismiss the Code Snippet Inserter and insert the currently selected code snippet"?
Unfortunately this is built-in functionality local to the Code Snippet Inserter.  Please also note use of Enter can be restricted as a shortcut key, see Keyboard shortcuts.

@msebolt   Hi， thanks for you response.
I mean ,  for example .  when i print "svm" then i have a code snippet "static void main " for this . But i have to tab twice to insert this code snippet.  This is not quite friendly. I just want to click Enter after i print "svm",and then the snippet can be inserted. In other IDE like Intellj Idea,  this is supported.
So i think this can also be supported in visual studio.

@felixmaomao Thank you for clarifying this further.
I can assign this to the author to see if this option is currently available, though for feature requests you'll need to use Suggest a feature in Visual Studio communty.

@felixmaomao Here are a few options where you might consider asking your question:
https://social.msdn.microsoft.com/Forums/vstudio
https://www.stackoverflow.com
https://www.reddit.com/r/VisualStudio/


Refactoring naming and folder structure according to convention

https://github.com/johnpapa/angular-styleguide

https://github.com/naver/pinpoint/issues/444


Cant move screen/camera

Describe the bug
A clear and concise description of what the bug is.
i can't move the camera
To Reproduce
Steps to reproduce the behavior:
click onto object
try moving screen
Expected behavior
A clear and concise description of what you expected to happen.
the screen is ment to move
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

WinDows EXE

Additional context
Add any other context about the problem here.

Use middle mouse button to move the camera


Building distribution at the SubState level.

If following #189 a SubState gets a construction budget from it's nation in construction packets(CPa). CPa $= \dfrac{\text{Construction Points}}{50}$. It can than spend these CPa on buildings based on their building costs, all within 1-12 CPa. It prioritizes buildings based SubState terrain, national budgetary factors, state modifiers and other concerns, but here is where things get complicated.
Complications

Balance

We don't want each economy to be completely autarkic, but they shouldn't start irrecoverably destroyed either.

Bureaucracy

In vanilla all nations start with approximately balanced bureaucracy budgets.
Some bureaucracy expenses are predictable, the # of incorporated states and population(tech/law modifiers?)
Institutions which we will convert cost bureaucracy scaled to incorporated population.
Military Leaders and trade routes cost bureaucracy scaled by law and trade agreements.

Infrastructure

Lack of infrastructure = market access penalties. Only ~3 Vanilla states start with penalties.
Infrastructure beyond base levels is from ports and railways.

Military

Military strength != more barracks. It's just not a simple more EU4 troops = more barracks
PMs
laws (and autogenerated conscription centers)
the arms industry's ability to supply
or trade's ability to purchase supply
soldier salary

Exceptions
Because of these complications I'm thinking some building will be handled outside of this apportionment process, namely

Government Administration
Ports (Maybe)
Railways (Maybe)
Barracks
Naval Bases

Process
So for RGOs and factories which is most buildings, we spend the construction budget where each building gets a priority based on

NationalBudget

struct or class belonging to country, basically what %extraction vs factory economy

Terrain

build more mines in SubStates that are primarily hills and mountains, lumber mills in forests

Technology

can only start with what you know how to make

State Modifiers

If a state has a bonus to coal mines, the mines in that state should gravitate towards being coal mines.


Portal dashboard page, under books the UI is off for longer wishlist

Describe the task
In the portal dashboard, if there are more items in the books wishlist or you have read section, the UI is breaking. We need scrollable content, like the project card.

The task is  complete


ProtocolProcess.processConnect() processes closed sessions.

When server is too busy, some ConnectMessage cannot be processed untill Constants.DEFAULT_CONNECT_TIMEOUT (10 seconds).
In this case, IdleStateHandler and MoquetteIdleTimoutHandler make channel to close.
But ProtocolProcess.processConnect() processes all messages connected and not connected.
Here is my logs and test code.
20:27:48,052 [main] INFO  Server startServer 95  - Persistent store file: C:\Users\runafter\git\runafter\moquette\broker\moquette_store.mapdb
20:27:48,345 [main] DEBUG SubscriptionsStore init 102  - init invoked
20:27:48,348 [main] DEBUG SubscriptionsStore init 107  - Reloading all stored subscriptions...subscription tree before

20:27:48,348 [main] DEBUG SubscriptionsStore init 115  - Finished loading. Subscription tree after

20:27:48,349 [main] INFO  SimpleMessaging processInit 250  - Starting without ACL definition
20:27:48,350 [main] DEBUG ProtocolProcessor init 137  - subscription tree on init

20:27:48,595 [main] INFO  NettyAcceptor initFactory 152  - Server binded host: <IP_ADDRESS>, port: 1883
20:27:48,598 [main] INFO  NettyAcceptor initFactory 152  - Server binded host: <IP_ADDRESS>, port: 8080
20:27:48,692 [nioEventLoopGroup-3-1] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@2adf16cd
20:27:48,693 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@2adf16cd
20:27:48,694 [nioEventLoopGroup-3-2] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event ProtocolEvent wrapping CONNECT
20:27:48,694 [nioEventLoopGroup-3-4] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event ProtocolEvent wrapping CONNECT
20:27:48,694 [nioEventLoopGroup-3-3] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event ProtocolEvent wrapping CONNECT
20:27:48,694 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer ProtocolEvent wrapping CONNECT
20:27:48,694 [nioEventLoopGroup-3-1] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@21e85538
20:27:48,695 [nioEventLoopGroup-3-5] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event ProtocolEvent wrapping CONNECT
20:27:48,696 [nioEventLoopGroup-3-2] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@76fd5428
20:27:48,696 [nioEventLoopGroup-3-4] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@76fd5428
20:27:48,696 [nioEventLoopGroup-3-3] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@7cd0e73c
20:27:48,697 [nioEventLoopGroup-3-4] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@4ce3c61
20:27:48,697 [nioEventLoopGroup-3-2] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@229198ea
20:27:48,697 [nioEventLoopGroup-3-3] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@305aecd9
20:27:48,698 [nioEventLoopGroup-3-5] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@8664e41
20:27:48,698 [nioEventLoopGroup-3-5] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@63643de1
20:27:48,699 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 160  - CONNECT for client <client1>
20:27:48,699 [nioEventLoopGroup-3-7] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event ProtocolEvent wrapping CONNECT
20:27:48,699 [nioEventLoopGroup-3-8] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event ProtocolEvent wrapping CONNECT
20:27:48,699 [nioEventLoopGroup-3-6] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event ProtocolEvent wrapping CONNECT
20:27:48,699 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 224  - Connect with keepAlive 60 s
20:27:48,700 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 229  - Connect create session <session [clientID: client1]org.eclipse.moquette.server.netty.NettyChannel@4b658b53>
20:27:48,700 [pool-1-thread-1] DEBUG SubscriptionsStore activate 209  - Activating subscriptions for clientID <client1>
20:27:48,700 [nioEventLoopGroup-3-7] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@779e1cd0
20:27:48,701 [nioEventLoopGroup-3-7] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@2d75d90a
20:27:48,702 [nioEventLoopGroup-3-8] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@1f20a0ab
20:27:48,702 [nioEventLoopGroup-3-6] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@17144a84
20:27:48,702 [nioEventLoopGroup-3-8] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@35282bf2
20:27:48,702 [nioEventLoopGroup-3-6] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.LostConnectionEvent@605d3821
20:27:48,748 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client1>
20:27:48,777 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 260  - Create persistent session for clientID <client1>
20:27:48,812 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 262  - Connected client ID <client1> with clean session true
20:27:48,812 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer ProtocolEvent wrapping CONNECT
20:27:48,817 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 160  - CONNECT for client <client4>
20:27:48,817 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 224  - Connect with keepAlive 60 s
20:27:48,817 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 229  - Connect create session <session [clientID: client4]org.eclipse.moquette.server.netty.NettyChannel@5da1fbed>
20:27:48,817 [pool-1-thread-1] DEBUG SubscriptionsStore activate 209  - Activating subscriptions for clientID <client4>
20:27:48,841 [main] ERROR ServerIntegrationConnectionTest waitForCompletion 101  - token.waitForCompletion() client[null] Connection lost (32109) - java.io.EOFException
20:27:48,841 [main] ERROR ServerIntegrationConnectionTest waitForCompletion 102  - Throwable :
Connection lost (32109) - java.io.EOFException
	at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:146)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readByte(DataInputStream.java:267)
	at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:65)
	at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:107)
	... 1 more
20:27:48,843 [main] ERROR ServerIntegrationConnectionTest waitForCompletion 101  - token.waitForCompletion() client[null] Connection lost (32109) - java.io.EOFException
20:27:48,843 [main] ERROR ServerIntegrationConnectionTest waitForCompletion 102  - Throwable :
Connection lost (32109) - java.io.EOFException
	at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:146)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readByte(DataInputStream.java:267)
	at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:65)
	at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:107)
	... 1 more
20:27:48,843 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client4>
20:27:48,843 [main] ERROR ServerIntegrationConnectionTest waitForCompletion 101  - token.waitForCompletion() client[null] Connection lost (32109) - java.io.EOFException
20:27:48,844 [main] ERROR ServerIntegrationConnectionTest waitForCompletion 102  - Throwable :
Connection lost (32109) - java.io.EOFException
	at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:146)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readByte(DataInputStream.java:267)
	at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:65)
	at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:107)
	... 1 more
20:27:48,844 [main] ERROR ServerIntegrationConnectionTest waitForCompletion 101  - token.waitForCompletion() client[null] Connection lost (32109) - java.io.EOFException
20:27:48,844 [main] ERROR ServerIntegrationConnectionTest waitForCompletion 102  - Throwable :
Connection lost (32109) - java.io.EOFException
	at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:146)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readByte(DataInputStream.java:267)
	at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:65)
	at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:107)
	... 1 more
20:27:48,857 [main] INFO  Server stopServer 112  - Server stopping...
20:27:48,870 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 260  - Create persistent session for clientID <client4>
20:27:48,902 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 262  - Connected client ID <client4> with clean session true
20:27:48,902 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer ProtocolEvent wrapping CONNECT
20:27:48,908 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 160  - CONNECT for client <client2>
20:27:48,908 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 224  - Connect with keepAlive 60 s
20:27:48,908 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 229  - Connect create session <session [clientID: client2]org.eclipse.moquette.server.netty.NettyChannel@11f55379>
20:27:48,908 [pool-1-thread-1] DEBUG SubscriptionsStore activate 209  - Activating subscriptions for clientID <client2>
20:27:48,934 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client2>
20:27:48,967 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 260  - Create persistent session for clientID <client2>
20:27:48,994 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 262  - Connected client ID <client2> with clean session true
20:27:48,994 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@21e85538
20:27:48,994 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer ProtocolEvent wrapping CONNECT
20:27:48,999 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 160  - CONNECT for client <client3>
20:27:48,999 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 224  - Connect with keepAlive 60 s
20:27:48,999 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 229  - Connect create session <session [clientID: client3]org.eclipse.moquette.server.netty.NettyChannel@68f2ee49>
20:27:48,999 [pool-1-thread-1] DEBUG SubscriptionsStore activate 209  - Activating subscriptions for clientID <client3>
20:27:49,019 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client3>
20:27:49,050 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 260  - Create persistent session for clientID <client3>
20:27:49,061 [main] INFO  NettyAcceptor close 298  - Msg read: 14, msg wrote: 0
20:27:49,061 [main] INFO  NettyAcceptor close 301  - Bytes read: 465, bytes wrote: 0
20:27:49,061 [main] DEBUG SimpleMessaging disruptorPublish 116  - disruptorPublish publishing event org.eclipse.moquette.spi.impl.events.StopEvent@4ec60971
20:27:49,061 [main] DEBUG SimpleMessaging stop 141  - waiting 10 sec to m_stopLatch
20:27:49,069 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 262  - Connected client ID <client3> with clean session true
20:27:49,069 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@76fd5428
20:27:49,070 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@76fd5428
20:27:49,070 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@7cd0e73c
20:27:49,070 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@4ce3c61
20:27:49,070 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@229198ea
20:27:49,070 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@305aecd9
20:27:49,070 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@8664e41
20:27:49,070 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@63643de1
20:27:49,070 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer ProtocolEvent wrapping CONNECT
20:27:49,075 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 160  - CONNECT for client <client4>
20:27:49,075 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 207  - Found an existing connection with same client ID <client4>, forcing to close
20:27:49,075 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client4>
20:27:49,094 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 217  - Existing connection with same client ID <client4>, forced to close
20:27:49,094 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 224  - Connect with keepAlive 60 s
20:27:49,094 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 229  - Connect create session <session [clientID: client4]org.eclipse.moquette.server.netty.NettyChannel@7dcf7147>
20:27:49,095 [pool-1-thread-1] DEBUG SubscriptionsStore activate 209  - Activating subscriptions for clientID <client4>
20:27:49,114 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client4>
20:27:49,133 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 260  - Create persistent session for clientID <client4>
20:27:49,159 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 262  - Connected client ID <client4> with clean session true
20:27:49,159 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer ProtocolEvent wrapping CONNECT
20:27:49,164 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 160  - CONNECT for client <client2>
20:27:49,164 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 207  - Found an existing connection with same client ID <client2>, forcing to close
20:27:49,164 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client2>
20:27:49,183 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 217  - Existing connection with same client ID <client2>, forced to close
20:27:49,183 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 224  - Connect with keepAlive 60 s
20:27:49,183 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 229  - Connect create session <session [clientID: client2]org.eclipse.moquette.server.netty.NettyChannel@31a48a84>
20:27:49,184 [pool-1-thread-1] DEBUG SubscriptionsStore activate 209  - Activating subscriptions for clientID <client2>
20:27:49,208 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client2>
20:27:49,234 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 260  - Create persistent session for clientID <client2>
20:27:49,253 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 262  - Connected client ID <client2> with clean session true
20:27:49,254 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer ProtocolEvent wrapping CONNECT
20:27:49,259 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 160  - CONNECT for client <client1>
20:27:49,259 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 207  - Found an existing connection with same client ID <client1>, forcing to close
20:27:49,259 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client1>
20:27:49,279 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 217  - Existing connection with same client ID <client1>, forced to close
20:27:49,279 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 224  - Connect with keepAlive 60 s
20:27:49,279 [pool-1-thread-1] DEBUG ProtocolProcessor processConnect 229  - Connect create session <session [clientID: client1]org.eclipse.moquette.server.netty.NettyChannel@4995afcf>
20:27:49,279 [pool-1-thread-1] DEBUG SubscriptionsStore activate 209  - Activating subscriptions for clientID <client1>
20:27:49,304 [pool-1-thread-1] INFO  ProtocolProcessor cleanSession 304  - cleaning old saved subscriptions for client <client1>
20:27:49,336 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 260  - Create persistent session for clientID <client1>
20:27:49,386 [pool-1-thread-1] INFO  ProtocolProcessor processConnect 262  - Connected client ID <client1> with clean session true
20:27:49,387 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@779e1cd0
20:27:49,387 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@2d75d90a
20:27:49,388 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@1f20a0ab
20:27:49,388 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@17144a84
20:27:49,388 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@35282bf2
20:27:49,389 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.LostConnectionEvent@605d3821
20:27:49,389 [pool-1-thread-1] INFO  SimpleMessaging onEvent 158  - onEvent processing messaging event from input ringbuffer org.eclipse.moquette.spi.impl.events.StopEvent@4ec60971
20:27:49,390 [pool-1-thread-1] DEBUG SimpleMessaging processStop 298  - processStop invoked
20:27:49,390 [pool-1-thread-1] DEBUG SimpleMessaging processStop 303  - interceptor firer stopped
20:27:49,396 [pool-1-thread-1] DEBUG SimpleMessaging processStop 306  - subscription tree

20:27:49,396 [main] DEBUG SimpleMessaging stop 143  - after m_stopLatch
20:27:49,396 [main] INFO  Server stopServer 115  - Server stopped

	@Test
	public void shouldNotConnectedConnectionDelayLagerThanConnectionTimeout() throws MqttException, IOException {
		long connectionDelay = 5L;
		long connectionTimeout = 1L;
		startServer(connectionDelay, connectionTimeout);

		IMqttToken token1 = connect("client1");
		IMqttToken token2 = connect("client2");
		IMqttToken token3 = connect("client3");
		IMqttToken token4 = connect("client4");

		sleep(200L);

		waitForCompletion(token1);
		waitForCompletion(token2);
		waitForCompletion(token3);
		waitForCompletion(token4);

		verify(mockInterceptHandler, Mockito.never()).onConnect(any(InterceptConnectMessage.class));
	}

Whole test code is here
https://github.com/runafter/moquette/commit/2f510ed61167b0e4573147345360afea37403d91

This bug can be reproduced with massive CONNECT message from over 10K clients.
This is the process of CONNECT message.

receive CONNECT message form clients
Be fired ProtocolEvent.
Put ProtocolEvent in SimpleMessaging.m_ringBuffer at SimpleMessaging.disruptorPublish()
ProtocolEvent in SimpleMessaging.m_ringBuffer is processed by SimpleMessaging.m_executor sequentially.
ProtocolProcessor.processConnect() process CONNECT message with ServerChannel.

Usually theses process are no problem.
But if massive CONNECT messages are received, it can be issue.
Let's suppose that 1000 CONNECT messages are received.
Received messages will be put in SimpleMessaging.m_ringBuffer.
And then SimpleMessaging.m_executor will process message in SimpleMessaging.m_ringBuffer one by one.
If it takes 100ms to process a message, 1000 CONNECT messages are processed in 100 seconds.
So worst case, last CONNECT message will be processed in 100 seconds.
But if ServerChannel is not processed within 10 seconds, it is closed by MoquetteIdleTimoutHandler.
(https://github.com/runafter/moquette/blob/master/broker/src/main/java/org/eclipse/moquette/server/netty/MoquetteIdleTimoutHandler.java#L31)
Whether or not ServerChannel is closed, CONNECT message is processed by ProtocolProcessor.processConnect().
In my opinion, CONNECT message with closed ServerChannel is not necessary to process.
But it is processed.
This situation makes performance degradation.

Hi @runafter your analysis is correct, but on the fix we should speak a little bit, this issue is also related to #102 and is due to an architectural fail (in my opinion), the ring buffer do a great job to keep the ProtocolProcessor single threaded and simple to hack, without concerns on concurrent access to common data  structures, but this produce these latency related bugs. I think that I've to rework this aspect of the boker.

Hi @runafter I changed the way Moquette works so no more a ring buffer and single thread processor,this means that latency errors shouldn't happened and this bug is fixed.
Andrea

As I know, LMAX Disruptor was built to address the latency of inter-thread communication but now you remove it because it seems to produces latency related bugs. So I'm a bit confused...
Anyway you are the master therefore:

I will redo all our stress tests for evaluate the new implementation
maybe you need to change the main-page description of Moquette ;-)

Hi Diego,
you are right,disruptor is for low latency inter thread comunication,but
the structure (single thread Protocol Processor) was limiting the
threading. In front we have n
Netty that uses 1 thread per core and never block,so it's time to move
Moquette to use full power of Netty.Now the protocol processor is executed
in Netty's threads and use persistent data structures(the subscription tree
is always copied upon modification). In this picture the ring buffer was
just a removeable step (I like that LMAX stuff).
That's the motivation.
Let me know of new bugs:-)
Andrea
Il giorno 11/ott/2015 01:19, "diegovisentin"<EMAIL_ADDRESS>ha
scritto:

As I know, LMAX Disruptor was built to address the latency of inter-thread
communication but now you remove it because it seems to produces latency
related bugs. So I'm a bit confused...
Anyway you are the master therefore:

I will redo all our stress tests for evaluate the new implementation
maybe you need to change the main-page description of Moquette ;-)

—
Reply to this email directly or view it on GitHub
https://github.com/andsel/moquette/issues/101#issuecomment-147130265.

Hi @andsel
Thanks for your work.
This changes are expected to make performance improvement for processing MQTT connection.


Intellisense, auto complete, confirmed and selected using tab!

Issue Type: Bug
Hi,
The productivity loss I have from the space key, automatically selecting an item, is insane.
Please return the functionality to be tab, because everytime I type "fieldA :" I now have my field name change some auto suggestioned delay pop up value,
because I am typing faster than the computer.
I must now type "FieldA" [escape] [space] [colon]
Kind Regards,
Wesley Oliver
VS Code version: Code 1.27.2 (f46c4c469d6e6d8c46f268d1553c5dc4b475840f, 2018-09-12T07:04:11.329Z)
OS version: Darwin x64 17.7.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz (8 x 2200)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: enabledrasterization: enabledvideo_decode: enabledvideo_encode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
2, 2, 2

Memory (System)
16.00GB (0.45GB free)

Process Argv
/Applications/Visual Studio Code.app/Contents/MacOS/Electron

Screen Reader
no

VM
0%

Extensions (2)

Extension
Author (truncated)
Version

markdown-preview-enhanced
shd
0.3.8

json-schema-validator
tbe
0.1.0

The productivity loss I have from the space key, automatically selecting an item, is insane.

This isn't something that we have defined - it's still tab and enter. This can be a custom keybinding configuration of yours (please check) or a commit character of the suggestion in question. To know that I need more information, like the language in which this happens, the extension your are using, and the completion in question. In short, I need reproducible steps

I am using the stock standard out the box, install of VSCode.
Were space now seems to be the bind key for code completion.
If you like any other settings check, please ask away.

To know that I need more information, like the language in which this happens, the extension your are using, and the completion in question. In short, I need reproducible steps


Updating devops extension

How do I update a previously installed devops extension?
Looks like what I need is to publish a devops extension.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 688236b5-3de5-2adf-4e45-d080dfeac2aa
Version Independent ID: 3b62252a-245b-1df1-9024-1f62b5338557
Content: az devops extension
Content Source: latest/docs-ref-autogen/devops/extension.yml
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

route to service team

You can use this command az extension update -n azure-devops to update an already installed extension.


Questions about project status

Hi again! Could you share code status and your plans?

How stable is current code?
What do you think about rewrite to protothreads? Will you accept API change proposals or PR?
Would be nice to support PlatformIO (library distribution).

Code review notes:

Current code is mixture of cpp & c. It would be nice to follow single style. For example, if you like classes, you could pass IO functions to constructor instead of hardcoded calls of fusb_read_buf & fusb_write_buf.
fusb_read_buf & fusb_write_buf are sync (no callbacks). That makes need of RTOS mandatory

Of cause, i could create fork to implement my vision of "ideal opensource usb-pd library". But i'd prefer to collaborate if possible. What do you think?

100% happy to collaborate.
This is actually a longer term plan to be an eject out of the IronOS project.
This is not actually finished ( I have changes to pull over).
Plan is to pull out threads to just function calls (so you can use your choice of library to create) and for cross thread notifications to be pulled out to a macro for user implementation.
In terms of not handling driver callbacks, I expect that to be handled by the user implementation of the read/write buf functions. You can perform any thread/hardware awareness there was my plan?
Open to ideas on restructure, but have found LTO not the best at removing indirection on storing points to functions inside classes.
1000% happy to collaborate into a library on platformio.

In terms of not handling driver callbacks, I expect that to be handled by the user implementation of the read/write buf functions.

read/write buf functions should say somehow "i'm finished data transfer". This cam be done in 3 major ways:

Invoke callback
Send message (very close to callback, but different signature)
Hidden by RTOS (function looks sync, but OS switch context inside)

If you say, user-implemented functions have sync signature, the only choice is (3)

Open to ideas on restructure, but have found LTO not the best at removing indirection on storing points to functions inside classes.

I don't know your hardware limits, but in general, right architecture is more important than saving some bytes. If you really have lack of flash/ram - let me know (i will take into account). Everything is discuss-able, and i never had situation when it was impossible to find satisfying approach. For example, cpp can use templates and so on.

At first glance, it would be enough to have .tick() function, called by timer interrupt every 1ms, instead of threads. The rest will organized by protothreads macroses. That will cause up to 1ms extra lag to responses, but IMO that's acceptable price for flexibility.
About collaboration. I'm software architect, it's my strong side to design optimal data flows and API-s. My weak side is to debug hardware :). So, if you have something working at bare metal, i can transform it to something "desired" (and i will avoid need to debug).
I would take 1-2 weeks to study details and propose refactored API draft.

In terms of not handling driver callbacks, I expect that to be handled by the user implementation of the read/write buf functions.

read/write buf functions should say somehow "i'm finished data transfer". This cam be done in 3 major ways:
1. Invoke callback

2. Send message (very close to callback, but different signature)

3. Hidden by RTOS (function looks sync, but OS switch context inside)

If you say, user-implemented functions have sync signature, the only choice is (3)

I'm not trying to be against this; but I dont see why you would use 1/2 when 3 is cleaner to this code and often trivial when using an rtos based implementation?

Open to ideas on restructure, but have found LTO not the best at removing indirection on storing points to functions inside classes.

I don't know your hardware limits, but in general, right architecture is more important than saving some bytes. If you really have lack of flash/ram - let me know (i will take into account). Everything is discuss-able, and i never had situation when it was impossible to find satisfying approach. For example, cpp can use templates and so on.

True, goal here is minimal indirection to help the flash cache.
Open to discussion, but this code is very much designed as a singleton in a system for only a single FUSB302 at the moment

At first glance, it would be enough to have .tick() function, called by timer interrupt every 1ms, instead of threads. The rest will organized by protothreads macroses. That will cause up to 1ms extra lag to responses, but IMO that's acceptable price for flexibility.

1ms is generally ok so long as its only 1ms. Some chargers only give 20ms for the entire transaction to complete.
(This should generally be fine).
Can definitely split into tick functions for these.

About collaboration. I'm software architect, it's my strong side to design optimal data flows and API-s. My weak side is to debug hardware :). So, if you have something working at bare metal, i can transform it to something "desired" (and i will avoid need to debug).
I would take 1-2 weeks to study details and propose refactored API draft.

If debugging hardware is not your strong point, PD is a mess 🙃  I've seen more not-to-spec than i have to-spec chargers
I would be interested to see what your proposal is, though also probably I would advise waiting if you can for a little bit.
I'm currently focusing on the MHP30 bring up, but then after that coming back to PD is the plan.
Once thats done this should be in a "stable" place; and when I would be far more open to re-arranging the code a lot more.

Sorry I meant to add in addition to this;
If you do think of a more ideally API for this (just do a dot point draft, nothing fancy) and throw it here, I'll try and shape closer to that as I work at the least :)

When this merges expect me to be poking this more in earnest.

I'm not trying to be against this; but I dont see why you would use 1/2 when 3 is cleaner to this code and often trivial when using an rtos based implementation?

Answer is obvious. Library, useable with and without RTOS is more flexible than library, useable with RTOS only :). We can not know user requirements and preferences. It's not good practice to force user use RTOS if he not wish to do so. Also, using protothreads simplifies portage abstraction layer significantly.
Of cause, i don't mean we should pay with messy code for such flexibility. USB-PD code with protothreads will be almost as nice as with RTOS, and easy to maintain. I have some practice with this, and i can do such refactoring to prove.

1ms is generally ok so long as its only 1ms. Some chargers only give 20ms for the entire transaction to complete.
(This should generally be fine).

To be more precise, +1 tick per each data transfer call. Some functions to 2-3 calls per message. That's the worst estimate. But probably can be improved. Or ticks frequency can be increased. Anyway, that should not become fatal problem.

If debugging hardware is not your strong point, PD is a mess upside_down_face I've seen more not-to-spec than i have to-spec chargers

That's my reasons for collaboration. I can do things i like, where i have good experience and avoid things i don't like :). Everybody should win.

I would be interested to see what your proposal is, though also probably I would advise waiting if you can for a little bit.
I'm currently focusing on the MHP30 bring up, but then after that coming back to PD is the plan.

No problem. Result of API design is pure list of function signatures (+ may be templates skeletons) with verified data flows. The second checkpoint is to verify "rewriting to protothreads is possible" - existing code is enough for that.


[Request] How to listen variable change in getxcontroller?

here simple code.
thank you!!!
class Controller extends GetXController{

int count = 0;

@override
void onInit(){

this.addListenner((){

 print(this.count);

});

super.onInit();

}

void increase (){
  this.count += 1;
  update();

}
void decrease(){
  this.count -= 1;
  update();
}

@override
void close(){
 this.removeListenner((){});
 super.close();
}

}

i don't quite understand your request, but here is not the place to ask questions about how to write correctly. The correct place is in the Communication and support channels
But since i am already here, maybe this is what you need: Workers
By the way, where did you find those this.addListener() and removeListener? it is not anywhere to be found in documentation nor any issue posted here. With Getx you don't need that kind of boilerplate, is simple. You should pay a visit do state management documentation, there are several examples to help you start

Please read the Workers documentation.
GetX does not work with changeNotifier to have addListener and removeListener methods.
Please use the support channels for questions. Github is exclusive for issues and requests for resources, and there are exclusive channels on the community platform for questions.
Before opening a resource request, please read the doc, and make sure that the resource does not exist.
As was answered, I am closing this.

how way to listen none observe variable change?
because sometime  i dont use observable variable.

If you want to listen variables, maybe there is no reason to use simple management.
I may even implement something like this in the future (I have no commitment to that statement), but I like to keep things well divided.
Simple management is very light because it is simple. Adding more features to it could add additional overhead for both (the reactive also uses GetxController), so I don't know if I want to do this.


Change version numbers

I think it would be easier to have versions for dev channel in x.x.x-dev and x.x.x for stable channel.
The current versions are unclear.

That's a good idea - I think what we can do is tag 0.7.0 as the next stable channel release and let 0.7.x-dev be reserved for the dev channel

0.7.0 has been released and is compatible with stable an master (for now)


move all hyphen filename to camelCase

If a file exports only a function then we should name it myFunction.js, its a good way to quickly identify what's inside the file

:horse: CLOSING BOT SAYS:

This is very old and not done, reopen if you have the need and want to fix it


Setup a staging repo and image pushing prowjob for this repo

Let's use cloudbuild to build and push docker images to a publicly accessible repo

I hit a fun snag: k8s-staging-secret-sync-rotation is longer than 30 chars, staging repo project names can't be longer than 30 chars.  Chatted offline, and we decided we're going to rename this repo to k8s-gsm-tools ("Kubernetes and Google Secret Manager Tools")

One #8 merges, the next step will be to open a PR against kubernetes/test-infra that mimics the following file: https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/k8s-staging-boskos.yaml

rename things accordingly (k8s-staging-boskos -> k8s-staging-k8s-gsm-tools)
for testgrid settings use sig-testing-images, k8s-gsm-tools, no alert email
assign the PR to me

/assign @b01901143


[Feature Request] Zenstack

Is your feature request related to a problem? Please describe.
Zenstack is an extension of Prisma for implementing authorization, that works with Next.js pages & app routers, and has documented integrations with Clerk and NextAuth, as well as tRPC.
Authorization is a missing piece in Kirimase, and adding Zenstack seems like an obvious choice given that it plays nice with the other pieces that are already supported by Kirimase.
It will also make bootstrapping a Zenstack project from create-t3-app easier, and authorization is also a missing piece in the T3 Stack.
Describe the solution you'd like
Add Zenstack to Kirimase
Additional context
—

Hey @andrictham, have you tried adding Zenstack to a current Kirimase app?
I wonder if it's already compatible and what's missing. Since in the CLI, you can choose to create Model, View or Controller, there might be some options to integrate them more easily. I will explore it myself too.

Hey @andrictham, have you tried adding Zenstack to a current Kirimase app?
I wonder if it's already compatible and what's missing. Since in the CLI, you can choose to create Model, View or Controller, there might be some options to integrate them more easily. I will explore it myself too.

I will try too


Security

Write about possible necessary security measures, while doing this, touch upon the theory and possible implementation of:

[x] Authentication
[x] Authorization

Finished both, added some content for JWT, XSS, CSRF

I have read it and it looks okay to me. My only note is, that I am not sure whether the wording should make it clearer how we would implement it in our project if we had the time.

Looks understandable, quite long for a future improvement stuff tbh, and would be nice to see it better connect to how we would implement in the project

separated the possible attacks to API and website


Fix masking order

This fixes the same issue as #14831 but for the BatchNormalization layer and Bidirectional wrapper.

Nagging Assignee @fchollet: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Reviewer @fchollet: It has been 14 days with no activity and the awaiting review label was assigned. Can you please take a look?

Thanks for the PR, and sorry for the delayed review. Could you please rebase, since the layers have changed recently? I believe the fix is no longer necessary in BN but may be necessary in Bidirectional.

Rebased, and should be ready to go.

Nagging Assignee @fchollet: It has been 17 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I think we can merge this when the tests pass.

The master this was based on previously was causing the test error, rebased to current master to (hopefully) fix the test...

Nagging Assignee @fchollet: It has been 18 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Sorry for the delay in getting this tested. Could you please rebase to fix the merge conflicts?

@droidicus any chance to pull rebase and push again?

It looks like the branch is in conflict due to the bug being fixed upstream already. This PR can be closed.


Email subject is empty when subject contains blue heart symbol

Hi,
Email subject is empty when it contains blue heart symbol. It is empty even when retrieved from API.
Example: 💙 Welcome 💙
Symbol unicode: https://charbase.com/1f499-unicode-blue-heart
Is this a bug or intentional?

Is this on the guerrillamail.com website?
On Wed, Nov 10, 2021 at 2:34 AM Georgi Georgiev @.***>
wrote:

Hi,
Email subject is empty when it contains blue heart symbol. It is empty
even when retrieved from API.
Example: 💙 Welcome 💙
Symbol unicode: https://charbase.com/1f499-unicode-blue-heart
Is this a bug or intentional?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/flashmob/go-guerrilla/issues/232, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE6MPZSNSLYAK7WVDOAVUTULFLUFANCNFSM5HV3XFSA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Hi, yes


Secrets

The extension should leverage Secrets as an option to safely store parameters across sessions.

Ciao @lmangani , I have a related question: now is there no way to save default api_url, api_token and api_key?
Do I have to set them every time at startup?
Thank you for this great tool

CIao @aborruso at this time the settings are indeed required, but if there's interest I'll get started on stored SECRETS 👍

CIao @aborruso at this time the settings are indeed required, but if there's interest I'll get started on stored SECRETS 👍

Grazie @lmangani, my use case is very trivial: I would like to open duckdb cli and use this fantastic work of yours, as available by default (with automatic load in .rc file), and without every time writing these parameters.
And set these parameters only when I don't want to use the default ones. If you think this is helpful, and you can put time and head into it, thank you very much. I would like to help you, but I don't know how to do it.

@aborruso here's an untested PR with SECRETS support and instructions. Would you able to build and validate it locally?
Instructions
Clone and build the PR branch locally:
cd /usr/src
git clone -b secret-manager --recurse-submodules https://github.com/quackscience/duckdb-extension-openprompt
cd duckdb-extension-openprompt
OVERRIDE_GIT_DESCRIBE="v1.1.3" GEN=ninja make

Run the local DuckDB build:
./build/release/duckdb

Add and test SECRETS
D CREATE SECRET IF NOT EXISTS open_prompt (
      TYPE open_prompt,
      PROVIDER config,
      api_token 'your-api-token',
      api_url 'http://localhost:11434/v1/chat/completions',
      model_name 'qwen2.5:0.5b',
      api_timeout '30'
  );

Testing
At this point the parameters should be derived from SECRETS instead of SET parameters.

I will also add ENV parameters to allow provisioning the process from the system.

Grazie @lmangani , I will try tomorrow.


Include licensing information in acme_tiny.py itself

Many people simply copy the acme_tiny.py script, meaning they
lose the licensing information that is included in a separate file.
This adds one extra comment line to the top of the file stating
who the copyright owner is, and what license it is under.

Thanks!


working with mRuby classes from C

Can I instantiate/call methods of mruby classes from a c program?
If yes, can you point me to any sample code?
thanks,
Maryam

#include <stdio.h>
#include <mruby.h>
#include <mruby/variable.h>
#include <mruby/string.h>
#include <mruby/proc.h>
#include <mruby/compile.h>
#include <stdio.h>

int
main(int argc, char* argv[]) {
  mrb_state* mrb = mrb_open();
  struct RClass* tc = mrb_class_get(mrb, "Time");
  mrb_value ti = mrb_obj_new(mrb, tc, 0, NULL);
  // add 3 seconds
  mrb_value ret = mrb_funcall(mrb, ti, "+", 1, mrb_fixnum_value(3));
  mrb_p(mrb, ret);
  return 0;
}

Yes, of course! I use this to communicate data between the c side and the mruby side of my game engine. You can find a C file chock full of useful helpers here:
https://github.com/beoran/eruta/blob/master/src/rh.c
It's free to use under the Sleepycat license.