Cant install gintro on Arch. Using libsoup2 and libsoup3 in the same process is not supported.

First we try generating bindings for GTK4, this may fail when GTK4 is not properly installed
on your computer. But don't worry, you can still use GTK3
gtk4....................: Remaining delayed methods: 0 (Fine!)
gdk4....................: Remaining delayed methods: 0 (Fine!)
gdkx114.................: Remaining delayed methods: 0 (Fine!)
gsk.....................: Remaining delayed methods: 0 (Fine!)
graphene................: Remaining delayed methods: 0 (Fine!)
gobject.................: Remaining delayed methods: 0 (Fine!)
glib....................: Remaining delayed methods: 0 (Fine!)
gio.....................: Remaining delayed methods: 0 (Fine!)
gdkpixbuf...............: Remaining delayed methods: 0 (Fine!)
gmodule.................: Remaining delayed methods: 0 (Fine!)
Failed to load GtkSource
Requiring namespace 'Gtk' version '3.0', but '4.0' is already loaded
Maybe for your OS you have to install additional GTK related packages?
We continue with the remaining packages...
atk.....................: Remaining delayed methods: 0 (Fine!)
pango...................: Remaining delayed methods: 0 (Fine!)
pangocairo..............: Remaining delayed methods: 0 (Fine!)
pangoft2................: Remaining delayed methods: 0 (Fine!)
fontconfig..............: Remaining delayed methods: 0 (Fine!)
freetype2...............: Remaining delayed methods: 0 (Fine!)
harfbuzz................: Remaining delayed methods: 0 (Fine!)
rsvg....................: Remaining delayed methods: 0 (Fine!)
xlib....................: Remaining delayed methods: 0 (Fine!)
notify..................: Remaining delayed methods: 0 (Fine!)
adw.....................: Remaining delayed methods: 0 (Fine!)
soup....................: Remaining delayed methods: 0 (Fine!)

(process:69675): libsoup-ERROR **: 22:27:13.714: libsoup3 symbols detected. Using libsoup2 and libsoup3 in the same process is not supported.
stack trace: (most recent call last)
/tmp/nimblecache-4098377420/nimscriptapi_3780256293.nim(199, 29)
/tmp/nimble_69566/githubcom_stefansalewskigintro/gintro.nimble(77, 7) installBefore
/tmp/nimble_69566/githubcom_stefansalewskigintro/gintro.nimble(64, 7) prep
/usr/lib/nim/system/nimscript.nim(273, 7) exec
/usr/lib/nim/system/nimscript.nim(273, 7) Error: unhandled exception: FAILED: /tmp/gintrosalewski/gen 1 [OSError]
       Tip: 4 messages have been suppressed, use --verbose to show them.
     Error: Exception raised during nimble script execution

And yes, I have installed libsoup 2.74.2-1 and libsoup3 3.0.3-1

related https://github.com/StefanSalewski/gintro/issues/183

Strange. I had the hope we fixed that.
I assume that you did some GTK update?
Well, if I remember correctly, we had a discussion that a really clean fix would need a very large rewrite of the whole generator script?

For me testing would imply installing latest GTK4 from git sources, which unfortunately is some initial work. And indeed I did not found any motivation doing some Nim work at all in the last 4 weeks, with the exception shipping some sections of the Nim book which I mostly had already on my local box or at least in my head.
Are you already able to do fixes on the gen.nim file and then run it? If you can do that, you may move the line with
main("Soup", "3.0") # process Soup before Nice, preventing the load of not matching libsoup version
upwards, that was what we did last time. Maybe adw or another lib is loading libsopup2.0. When that moving up does not help, then someone would have to do a drastically rewrite.
Have to watch AnneWill in German TV now, about latest Covid news.

Sorry, can not reproduce.
First I tested my current GTK4 install --works.
Then I have completely reinstalled GTK4 from git sources with all the corresponding modules like adw, libnice, libhandy and that, all from git sources. I leave out webkitgtk for GTK4, because that takes an hour to build and needs a lot of free space on the SSD. But I think that can not be the problem. So I assume that you have at least one lib that is built before libsoup3 and that loads libsoup2. So it would be good if you can test yourself.
For the rewrite, I think we would have to split the gisub files then into many, one for each module, maybe integrated into the modules. And then we have to process each module fully isolated on its own, in its own process. Of course we have to modify the mconnect macro again, as it has not to search in the current gisup files but in the new location then. I would guess that we ca get the filename for the gisup content from the first parameter of the arguments of connect -- e.g. for connect("clicked", myButton) owner(myButton) would give us gtk4, and so we search for the gisup content in gtk4 module then.
What do you think, may that work?

So, as I understand, you cant reproduce issue, yerstaday itwas fixed for me with commented both libsoup 2 and 3.
Now Im in office with different pc, will try to reinstall gintro, update gtk and reinstall gintro again to check will this reproduce.
For the rewrite, ofc I agree that processing of every GIR file in separate process is a good idea.

If I comment in gen.nim only first soup
    # main("Soup", "2.4") # process Soup before Nice, preventing the load of not matching libsoup version
...
  main("Soup", "3.0")

I still got the error libsoup3 symbols detected. Using libsoup2 and libsoup3 in the same process is not supported
So, may be some of other libs start to use libsoup 2, and didnt do that before.
Now the temp solution for me is comment both soup in gen.nim

From your initial post the issue should be in
  else:
    echo "First we try generating bindings for GTK4, this may fail when GTK4 is not properly installed"
    echo "on your computer. But don't worry, you can still use GTK3"
    main("Gtk", "4.0") # the 3 new 4.0 releases
    main("Gdk", "4.0")
    main("GdkX11", "4.0")
    main("Gsk") # and two new ones for gtk4
    main("Graphene")
    main("GObject") # and the old common onces. GObject before Glib!
    main("GLib")
    main("Gio")
    main("GdkPixbuf")
    main("GModule")
    main("GtkSource")
    main("Atk")
    main("Pango")
    main("PangoCairo")
    main("PangoFT2")
    main("fontconfig")
    main("freetype2")
    main("HarfBuzz")
    main("Rsvg")
    main("xlib")
    # main("Vte") # not yet available for GTK4
    main("Notify")
    # main("Handy") # not yet available for GTK4
    main("Adw") # replaces libhandy for GTK4
    main("Soup", "3.0") # process Soup before Nice, preventing the load of not matching libsoup version

So commenting the last line of that block should fix it, and hopefully moving that line up may fix it also? That is moving up multiple steps.
Maybe you can use ldd to find which lib uses libsoup. For me it is only libnice.so
When you should have adw use of libsoup, then maybe exchanging the last two lines would work. If not, you would have to move main("Soup", "3.0") more upwards.
But yes, all that is ugly.

I think the problem might (also) be related to different versions of WebKit. I have webkit2gtk and webkit2gkt-4.1 installed and they pull in libsoup2 and libsoup3 respectively. When I comment out the main("Soup", "3.0") I can install gintro, but I get a warning:
adw.....................: Remaining delayed methods: 0 (Fine!)
nice....................: Remaining delayed methods: 0 (Fine!)
cairo...................: Remaining delayed methods: 0 (Fine!)
Failed to load WebKit2
Typelib file for namespace 'WebKit2', version '5.0' not found
Maybe for your OS you have to install additional GTK related packages?
We continue with the remaining packages...
Failed to load JavaScriptCore
Typelib file for namespace 'JavaScriptCore', version '5.0' not found
Maybe for your OS you have to install additional GTK related packages?
We continue with the remaining packages...
Failed to load WebKit2WebExtension
Typelib file for namespace 'WebKit2WebExtension', version '5.0' not found
Maybe for your OS you have to install additional GTK related packages?
We continue with the remaining packages...

Still, the install goes through ("Success: gintro installed successfully.") and I can also run the example from here: https://github.com/StefanSalewski/gintro/blob/master/examples/gtk3/webkitgtk.nim
I then tried to install webkit2gtk-5.0 and to reinstall gintro. But this completely breaks it. I get the same libsoup2 and libsoup3 conflict even though I commented out all Soup related lines in gen.nim.

Thanks for reporting. Will think about it this evening or tomorrow, have to do some other work now.

hello idem problem "manjaro"  this trisub

Yes, I will investigate it. Was busy with https://github.com/StefanSalewski/gintro/issues/198 in the last two weeks.

hello idem problem "manjaro" this trisub

Have just shipped the PEGs section of the book, so maybe I can investigate this issue again this weekend. But it may be very hard and may enforce a larger rewrite of the gen.nim generator script.
But I just had an interesting idea: I think Nim loads dynamic libs with dlopen, so may it be possible to use dlclose to close it, and then reopen in a virgin state? What do you think?

Have you made some progress?
And did you saw my comments at https://discourse.gnome.org/t/will-dlclose-reset-internal-state-of-libgirepository/8963
Maybe I have to actually install Arch Linux to verify where exactly the problem arises from.

Thank you, I read correctly, I tried to make a basic XFCE manjaro thinking that the addition of some software could have an action, results the generation is not done and always blocks libsoup and libsoup3 moreover webkit2gtk and webkit2gkt-4.1 are installed automatically.
webkit2gtk use libsoup
webkit2gkt-4.1 and webkit2gtk5 use libsoup3

I just took a brake from book writing, and so did some investigations of this issue again, and I had an idea, which turns out not working. But for my Gentoo box, with additional ONLY libsoup installed from git sources to /opt, all works fine. I can even modify the order of installs, and for example put libsoup to the bottom. All works fine. For details see https://discourse.gnome.org/t/libnice-should-have-libsoup-as-depency/9320/6
So I will hope that for ArchLinux it will work now, or soon also. Unfortunately nimble allows no parameters, so there is no good solution for people with problem to pass a parameter to skip libsoup or libnice, maybe even webkit. Maybe we can use environment variables instead of parameters?
I just tried to fix the latest issue with https://github.com/StefanSalewski/gintro/issues/200 which resulted from a rename of uref to unref of latest gobject. For me it works again, I hope my fix will not break it for people with older gobject. I will push that change tomorrow.

We have just applied the uref/unref fix which was necessary for latest gobject, and we removed libnice from the GTK4 process. Seems to work now. The only disadvantage is, that users can not use libnice with GTK4 currently. But we have not that many libnice users at all. And when libnice may much much later decide to use libsoup3, we get new trouble. Another possible solution is to use g_typelib_free() when processing libsoup and libnice, but that has other issues, and may not work for Windows and Mac, see https://discourse.gnome.org/t/libnice-should-have-libsoup-as-depency/9320/13. Please test with
nimble uninstall gintro
nimble install gintro@#head

it works! thanks for this investigation
...
gstapp..................: Remaining delayed methods: 0 (Fine!)
gstpbutils..............: Remaining delayed methods: 0 (Fine!)
Failed to load GtkLayerShell
Typelib file for namespace 'GtkLayerShell' (any version) not found
Maybe for your OS you have to install additional GTK related packages?
We continue with the remaining packages...
  Verifying dependencies for gintro@#head
 Installing gintro@#head
   Success: gintro installed successfully.

It's not urgent for me but just thought I'd report this issue still persists on EndeavourOS(Arch derivative)

I can confirm that I have the same problem in vanilla Arch

So this issue popped up again? We had the impression that we solved it in spring this year? The libsoup2 and libsoup3 issue is a very ugly one, in principle caused by how GTK and Gobject-introspection works internally. I have the impression that libsoup3 will replace libsoup2 soon totally, so this issue may vanish. I will try a a fresh gintro install on my Gentoo box soon (not this weekend, here in Germany our fiber cable is damaged since a few days, damage will persist at least for two day...) , to test if I can reproduce any issue. If not, and is again a poor Arg Linux issue, then I may not be able to help, sorry.

I have some confidence that I may be able to reproduce your issue.
https://packages.gentoo.org/packages/net-libs/webkit-gtk
webkit-gtk-2.38 seems to use libsoup3 now, while it used libsoup2 for 2.36. I should be able to fix that in the gen.nim install script. Unfortunately that will break it then for people who have still installed webkit-gtk-2.36. A parameter for nimble to influence the install process would be nice to solve such issues, but as you know, I have good reasons not to ask for things like that :-)
Currently my box is busy, Gentoo has to update a lot of packages including  webkit-gtk-2.38. Maybe I can start more detailed investigations in a few hours.

I have been able to install latest webkit versions:
ls -lt /usr/share/gir-1.0/Web*
-rw-r--r-- 1 root root 1728408 Dec 12 21:52 /usr/share/gir-1.0/WebKit2-5.0.gir
-rw-r--r-- 1 root root 2357784 Dec 12 21:52 /usr/share/gir-1.0/WebKit2WebExtension-5.0.gir
-rw-r--r-- 1 root root 1732055 Dec 12 21:18 /usr/share/gir-1.0/WebKit2-4.1.gir
-rw-r--r-- 1 root root 2360529 Dec 12 21:18 /usr/share/gir-1.0/WebKit2WebExtension-4.1.gir
-rw-r--r-- 1 root root 1732056 Dec 11 18:14 /usr/share/gir-1.0/WebKit2-4.0.gir
-rw-r--r-- 1 root root 2360530 Dec 11 18:14 /usr/share/gir-1.0/WebKit2WebExtension-4.0.gir

I think 4.1 and 5.0 are using libsoup3 -- that combinations seems to be shipped for Arch Linux. But most others may have still 4.0 which uses libsopup2.4. It is not easy to install all that combinations. I will try with autodetect, and when that fails, maybe controlled by environment variables? Another problem is, that libnice still needs libsoup2.4, and we had a libnice user some years ago. I can not remember what libnice really is? Without libnice, we could just drop WebKit2-4.0 and  use libsoup3 only.

I just tested an "nimble install gintro@#head" and its works without any problems for me.
I get
ls -lt webk*
-rw-r--r-- 1 salewski salewski 820928 Dec 13 09:57 webkit2webextension.nim
-rw-r--r-- 1 salewski salewski 460709 Dec 13 09:57 webkit2.nim
-rw-r--r-- 1 salewski salewski 821350 Dec 13 09:55 webkit2webextension5.nim
-rw-r--r-- 1 salewski salewski 457630 Dec 13 09:55 webkit25.nim

so the maybe problematic webkitgtk 4.1 is just ignored. This is the same behaviour, which we had two years ago, when webkitgtk5 was not shipped by most distributions, and we had to install it from sources to /opt.
Well, my box has currently no libnice installed, maybe I should install that one and test again. Will do.

After libnice install, still no issues. Are you sure you actually tried
gintro uninstall gintro
nimble install gintro@#head

So what is the actual issue? Maybe Arch Linux ships broken Gobject-Introspection files? But to test that, I would have to install Arch Linux on my box, and learn to use it.
Or is it that you are just missing webkitgtk 4.1? Well, we could try to create that.

jason@jason-desktop ~> ls -lt /usr/share/gir-1.0/Web*
-rw-r--r-- 1 root root 1728289 Nov 30 16:52 /usr/share/gir-1.0/WebKit2-5.0.gir
-rw-r--r-- 1 root root 2357784 Nov 30 16:52 /usr/share/gir-1.0/WebKit2WebExtension-5.0.gir
-rw-r--r-- 1 root root 1731936 Nov 30 16:52 /usr/share/gir-1.0/WebKit2-4.1.gir
-rw-r--r-- 1 root root 2360529 Nov 30 16:52 /usr/share/gir-1.0/WebKit2WebExtension-4.1.gir
-rw-r--r-- 1 root root 1731937 Nov 30 16:29 /usr/share/gir-1.0/WebKit2-4.0.gir
-rw-r--r-- 1 root root 2360530 Nov 30 16:29 /usr/share/gir-1.0/WebKit2WebExtension-4.0.gir

I did not have gintro installed prior, and I am trying to install head. No clue if I can provide any more insight though.

Thanks for reporting. So there seems to be really again an issue for Arch Linux. But I have no idea for the reason.

Hello Jason B.,
as I had the feeling that you are indeed some of the few people who are still using Nim, I finally decided to install Arch Linux myself to find and fix your issue. So yesterday, I took an old laptop of my deceased mom and installed Arch Linux, Gnome, Nim, and latest gintro. The obvious issue is, that latest Arch Linux ships vte 3.91 for GTK4, while most other distributions including my Gentoo still have only v2.91 for GTK3. I was not aware that a VTE for GTK4 is already available, otherwise the issue would have been obvious. I guess fixing this is not that difficult, I will tell you when the fix is available at Github.
But note that I very seriously intend to remove gintro from Github soon. This issue proves again, how lazy (or stupid) users are. Five minutes test for someone with Arch Linux installed, vs. a whole day for me. Well, I know from IRC logs, that recently some "them" complained about this issue too.
For Arch Linux, I still prefer Gentoo. Installing Arch is a bit simpler at first, but still more complicated than Ubuntu. Of course, on an old laptop with only 2 cores and only 4 GB RAM a source based distribution like Gentoo makes not much sense, I would be not able to install Firefox from sources, and GCC or CLang would be problematic already. So Ubuntu or Arch Linux are candidates, I will continue with Arch now, as we have it now. But well, I do not really use that device, it is an Lenovo Thinkpad R500, twelve years old.
Best regards,
Dr. Stefan Salewski

Actually, there seems to be at least one more issue. When generating the GTK3 bindings, at least one of the Arch libs seems to pull in libsoup3 now, without really advertising. Have to comment them out one by one, to find the culprit. Maybe Arch Linux has switched to libsoub3 fully already, so that we could drop libsoup2. That would simplify things a lot. Will continue tomorrow with testing.
Maybe, we could fully drop GTK3? I think GTK4 is available for more than two years now, so why would someone still use GTK3? GTK4 has much more interesting bugs :-)

Actually Arch Linux has a libnice that uses libsoup3 now, that seems to be the main issue. Arch is ahead to most other distributions, we have to find a way to fix it for Arch, without breaking it for old distributions. Problem is, libnice does not really advertise libsoup use, see https://discourse.gnome.org/t/libnice-should-have-libsoup-as-depency/9320

But note that I very seriously intend to remove gintro from Github soon

You could archive the repo and explain why in the readme.

Hello @StefanSalewski and thank you for your efforts.
I am a dedicated Arch user, especially because I can see and validate "things to come"
If you believe that we can drop GTK4 on recent distributions (like Arch), I think it's a good idea anyway. There's no reason to have backwards compatibility forever, especially on Linux world.

If you believe that we can drop GTK4 on

Actually drop GTK3 support. But maybe dropping libsoup2.4 would be good enough. My feeling is, that Arch Linux does not need old libsoup any longer, all seems to be libsoup3 now. Other distributions should follow soon. And libsoup is needed only for webkitgtk, which no one really uses currently. And if someone wants webkitgtk, they can use the gtk4 version with libsoup3. All that would be easier, if "nimble install" would allow parameters to influence the install process. Maybe environment variables will do.

You could archive the repo and explain why in the readme.

Would make not really much sense. An archived version without any support would get bitrotting soon, and no one would be willing to fix it, in the same way as no one beside me has touched it in the last eight years. Maybe I could re-upload it with a new name, maybe as sgb for "Salewski's gobject bindings". Then all the fools would not discover it easily, the issue with too high version number would be solved (v0.99 already), and all the old issues would be gone also. And actually I was never very happy with the old name, was suggested by Mr. Rumpf seven years ago. Will try to fix the issues in the next days.

Actually drop GTK3 support.

Yes sorry, I had in my mind "drop GTK3 and keep only GTK4", but my brain was faster than my fingers :)
Still, the rest of your thoughts look valid to me. And don't get it wrong: I think the community wants and needs what you are doing :smiley:

I think the community wants and needs what you are doing

Not really. We had one user, maybe in 2017 or 2018, who created some game unlocking tool with gintro. But not more. Maybe we have a few unpublished toy tools, but for toy tools the 20 other Nim GUI libs should be good enough. Actually, my feeling is, that GTK has no more users at all, for C and all the languages with bindings, the appearance of GTK4 has not improved the situation. When you go to Github and search for new serious GTK projects, there is nothing. With Gnome, GTK has still a passive, consumer community, asking for new features or complaining from time to time.

Hello Jason B.,
as you may have already noticed, we have just uploaded the latest fix for the libsoup issue, from which currently Arch Linux users and others with latest libnice version may suffer. You may test with
nimble uninstall gintro
nimble install gintro@#head
For me it installs fine now for Gentoo and Arch Linux. For tiny restrictions see https://github.com/StefanSalewski/gintro/issues/209#issuecomment-1363434492
Currently we have two known issues: The webkitgtk2.nim test for gtk4 does not compile, and for Arch Linux gstBasicTutorial1.nim does not compile. But these issues should be unrelated to our current fix. I will investigate these issues in the next days, and will do some more tests. We will see which new issues the upcoming Nim 2.0 may bring.
Best regards,
Stefan Salewski

I'd like to confirm that this works for me now. Thank you @StefanSalewski

OK, I have just uploaded the fix for webkitgtk2 and for gst. The examples/gtk4/webkitgtk.nim should run now and display the Nim home page, and examples/gtk3/gstBasicTutorial1.nim should play the video.

Also can confirm works here!


Chunks

Three changes:

Building out some checks in the chunks schema.
Adding a check that the input is actually a da, since if it's a dataset the chunk routines will fail for the wrong reason.
Remove the return of the da at the end since I think we just want it to continue without returning anything.

Codecov Report

Merging #2 (10416d4) into main (0a60625) will decrease coverage by 1.99%.
The diff coverage is 71.42%.

:exclamation: Current head 10416d4 differs from pull request most recent head 37ecca9. Consider uploading reports for the commit 37ecca9 to get more accurate results

@@            Coverage Diff             @@
##             main       #2      +/-   ##
==========================================
- Coverage   87.17%   85.18%   -2.00%
==========================================
  Files           2        2
  Lines          78       81       +3
==========================================
+ Hits           68       69       +1
- Misses         10       12       +2

Impacted Files
Coverage Δ

xarray_schema/core.py
84.41% <71.42%> (-2.08%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a60625...37ecca9. Read the comment docs.

@jhamman Take a look- added logic to cover chunks being True, False or a dict, and added a few more tests to cover some corner cases.

Thanks @orianac. I'm going to merge now!


Fix pigweed RPC logging on ESP32

Introduced in https://github.com/project-chip/connectedhomeip/pull/26227/ we no longer get matter logs in pigweed RPC. This reverts src/platform/ESP32/Logging.cpp to before the regression. Not only is it not logging matter logs, but it is causing some back to back RPC calls to hang and timeout.
This PR also changes the default of the newly introduced DISPATCH_EVENT_LONG_DISPATCH_TIME_WARNING_THRESHOLD_MS to match the default in the SDK, which is 100ms. Having a lower default will more likely help people catch real issues as the dispatcher should not be taking 100ms to start with. If there is a project that needs a higher threshold it would make sense for that individual project to use a higher value.
Test:

Compiled using ./scripts/build/build_examples.py --target esp32-m5stack-all-clusters-rpc build.
Flashed onto M5Stack using ./out/esp32-m5stack-all-clusters-rpc/chip-all-clusters-app.flash.py
Attached to pigweed rpc console python3 -m chip_rpc.console --device /dev/ttyUSB0
Confirmed

That we are getting chip logs again
Confirmed that doing multiple back to backrpcs.chip.rpc.Device.SetPairingState(pairing_enabled=False) and rpcs.chip.rpc.Device.SetPairingState(pairing_enabled=True) does not result in a timeout

@dhrishi any chance you could look at this and advise on how to use these things?

This PR also changes the default of the newly introduced DISPATCH_EVENT_LONG_DISPATCH_TIME_WARNING_THRESHOLD_MS to match the default in the SDK, which is 100ms. Having a lower default will more likely help people catch real issues as the dispatcher should not be taking 100ms to start with. If there is a project that needs a higher threshold it would make sense for that individual project to use a higher value.

I had taken a look at the origin of DISPATCH_EVENT_LONG_DISPATCH_TIME_WARNING_THRESHOLD_MS and it was there from the legacy code. Apart from printing the error message, it does not do anything like propagating it or taking some action (for any platform).
The change here was induced to provide an option to the application to set the threshold as per their requirement. Also, it resulted into large number of error prints on the console .
Even though the default is set to 700, you can always set it to a lower value in the application.

I had taken a look at the origin of DISPATCH_EVENT_LONG_DISPATCH_TIME_WARNING_THRESHOLD_MS and it was there from the legacy code. Apart from printing the error message, it does not do anything like propagating it or taking some action (for any platform).
The change here was induced to provide an option to the application to set the threshold as per their requirement. Also, it resulted into large number of error prints on the console .
Even though the default is set to 700, you can always set it to a lower value in the application.

Thing taking 500ms to dispatch is likely going to have downstream impacts. Now when someone looks at a log they are going have that context by default and spend more time, IMO potentially looking into the wrong thing. If any application find this log spammy, and they know that it is not an issue for their application, that application should use a higher value. How I see it you can start masking an issue that can progressively get worse over time. But if this is the default that your platform wants to give I will remove that part of the PR


[LIVY-692][THRIFT] Use configured principal for Kerberos authentication

What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/LIVY-692
Currently, if Kerberos authentication is used in the Thrift server (livy.server.thrift.authentication=kerberos), the TTransportFactory used in the ThriftCLIService uses a UserGroupInformation for the service (livy) principal; instead it should use a UserGroupInformation for the principal specified in livy.server.auth.kerberos.principal.
How was this patch tested?
Manual testing with both a Livy Thrift server using Kerberos authentication and one using simple authentication.

Codecov Report

Merging #241 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #241      +/-   ##
============================================
- Coverage     68.45%   68.42%   -0.04%
+ Complexity      927      924       -3
============================================
  Files           100      100
  Lines          5729     5729
  Branches        870      870
============================================
- Hits           3922     3920       -2
+ Misses         1247     1246       -1
- Partials        560      563       +3

Impacted Files
Coverage Δ
Complexity Δ

...ain/java/org/apache/livy/rsc/driver/RSCDriver.java
78.66% <0%> (-2.1%)
43% <0%> (-3%)

...ain/scala/org/apache/livy/utils/SparkYarnApp.scala
67.32% <0%> (+1.3%)
40% <0%> (ø)
:arrow_down:

...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala
55.88% <0%> (+2.94%)
7% <0%> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0804c8e...9647b2f. Read the comment docs.

I don't think this patch is correct. I think the current behavior is correct. Please see {{LivyCLIService}} where the two tokens are created. If we'd go ahead with this patch, impersonation would break on many clusters, as it may/should not be enabled on the http UGI.

@mgaido91 binary mode seems to work with this patch. Let me look into http mode.

@wypoon have you checked also impersonation? Moreover, I think the current behavior is correct and I don't see any benefit introduced by this patch, rather than breaking the existing behavior.

By impersonation, do you mean that the Spark application Livy spawns (to run queries) is run as the user connecting to the Thrift server? I have a user kinit'd, let's say using principal testuser, who connects via beeline to the Thrift server; and the Spark application for the session runs as testuser. If this is what you mean, then impersonation is working.
I tested my patch using both binary and http mode. In binary mode, everything works as I expect. In http mode, everything still works, except that on the server side, there are many exceptions logged, due to
Caused by: org.apache.hive.service.auth.HttpAuthenticationException: Authorization header received from the client is empty.
but they do not cause any failure. I believe these exceptions are an unrelated issue.
Without my patch, I am able to connect when using http mode (the same slew of exceptions about the empty authorization header also occur in the server log), but I am unable to connect when using binary mode, due to the error as described in JIRA.
Thus I fail to see why you say the existing behavior is correct.
This patch fixes the binary mode case, and as far as I can tell, does not break existing behavior.
If you think the existing behavior is correct, and that the failure I experience in binary mode is due to some kind of misconfiguration or user error, I'd very much appreciate knowing what configuration I need to correct or what I am missing in my beeline command. Please comment in JIRA with the information.

I think the point is that in your JIRA you are using HTTP/... as principal when you connect to beeline. You should use the principal Livy server is using instead.

I have a user kinit'd, let's say using principal testuser, who connects via beeline to the Thrift server; and the Spark application for the session runs as testuser. If this is what you mean, then impersonation is working.

yes, that is impersonation. Yes, it may not be a problem because in your case you're forcing to use the HTTP principal only for the authentication, but the livy one is used everywhere else, the problem in this case would be token renewal most likely...you are creating the UGI once and never renewing it, so it would not work after 1 week or even 1 day probably.

The reason I use the HTTP principal in my beeline connection is because that is the principal configured in livy.server.auth.kerberos.principal. If I use the livy principal in my beeline connection, the connection still fails with the "GSS initiate failed" error. (This is with the existing code, before I made any changes.)
I understand that my patch may not be correct (not dealing with token renewal), but there is a real issue here.
Concerning renewal, can you point me to where/how the current code takes care of token renewal?

If I use the livy principal in my beeline connection, the connection still fails with the "GSS initiate failed" error. (This is with the existing code, before I made any changes.)

this is weird. I'd need to investigate this but I don't have a cluster so I am not sure how to help you.

Concerning renewal, can you point me to where/how the current code takes care of token renewal?

Please check the LivyServer class.

GSS initiate failed

can you share full exception stack? perhaps it's some sort of kerberos client misconfiguration on
livy client server, or livy server side.. hard to tell without knowing details.

I updated the JIRA to explain that the problem was due to using an internal version of Livy that had some changes incompatible with the Thrift server; the changes caused the UGI for the livy principal to be obtained from ticket cache instead of keytab; this is what caused the "GSS initiate failed" error when trying to connect from beeline.
Closing this PR.


#Fixes 82 implemented the page not found

This PR addresses the "404 Page Not Found" issue that users encounter when attempting to access non-existent routes in the application. The goal is to enhance user experience by providing a visually appealing and informative error page instead of the default browser error message.

This PR addresses the "404 Page Not Found" issue that users encounter when attempting to access non-existent routes in the application. The goal is to enhance user experience by providing a visually appealing and informative error page instead of the default browser error message.

But it's not working as you've intended it to.


Graceful stop for tasks and OOM conditions handling

Added graceful stop support for user tasks

Now, when the user clicks on "Stop" button in the web UI or stops a task by a REST request, the task receives a signal. After a period, the task receives a KILL signal and gets stopped.
This enables, for example, to save a training checkpoint before the exit.

Added Out-Of-Memory conditions handling

Now, when one of the container processes gets killed due to OOM, the user command receives a signal. The user command decides if it is needed to stop.

Implementation details:

Signals are sent to all running container processes in user session. Thus, if some do not have a registered handler, they will stop.
Signals and timeouts are customizable by setting environment variables for a task. By default, the OOM kill signal is INT, the task stop signal is TERM, task stop timeout is 30s.
No signal specified = no notification.
User timeouts have non-modifiable "hard" timeout - 65 seconds (set in cluster configuration)
Hard timeouts and signal sequence during task stopping:kill -TERM <yarn container> # <timeout 60s + 65s from YARN config> (needs some clarification)
kill -KILL <yarn container>

OOM kill checks are performed periodically during the task run and after by dmesg output checking. This is not very reliable in some cases, but for most situations it should be enough. The reason is restricted dmesg log length. In can probably be resolved by listening cgroup events, but it is not portable way.
If a task has failed and there were some OOM notifications, the task exit code is 55 (unknown error)

Examples
To execute a command, which expects some child can fail, do:
In the task description set:
{
  "PAI_GRACEFUL_EXIT_TIMEOUT": 30, # set some custom exit timeout
  "PAI_GRACEFUL_EXIT_SIGNAL": "USR1", # some custom signal to send on stop
  "PAI_OOM_KILLED_SIGNAL": "USR2" # and on OOM kill
}

And launch the following user command:
set +e # the most important line, script exits on children kill by default (with -e)
trap "echo 'got USR1!, exiting'; exit 0" USR1
trap "echo 'got USR2!'" USR2

# launch some big task in a separate shell periodically
(while :; sleep 7; do /oom_test/alloc_test<PHONE_NUMBER>0; done) &

# and just wait in the main shell
while :; do sleep 5; done

stdout output:
[INFO] USER COMMAND START

+ set +e
+ trap 'echo '\''got USR1!, exiting'\''; exit 0' USR1
+ trap 'echo '\''got USR2!'\''' USR2
+ :
+ sleep 5
+ :
+ sleep 7
+ :
+ sleep 5
+ /oom_test/alloc_test<PHONE_NUMBER>0
Trying to allocate<PHONE_NUMBER>0 bytes...
/pai/taskData/command.sh: line 9:   163 Killed                  /oom_test/alloc_test<PHONE_NUMBER>0
+ :
+ sleep 7
+ :
+ sleep 5
+ /oom_test/alloc_test<PHONE_NUMBER>0
Trying to allocate<PHONE_NUMBER>0 bytes...
/pai/taskData/command.sh: line 9:   166 Killed                  /oom_test/alloc_test<PHONE_NUMBER>0
                               ^--- (comment: from the kernel)
+ :
+ sleep 7
+ :
+ sleep 5
+ :
+ sleep 5
[WARNING] One of the child processes has been killed by the OOM killer. Notifying the others with the 'USR2' signal
User defined signal 2 <-- (comment: from Bash, can not be suppressed)
++ echo 'got USR2!'
got USR2!
          ^---- (comment: user command received an OOM kill notification signal and decided to continue)
+ :
+ sleep 5
+ :
...

Fixed AM hang with infinite spin waiting on an error

AM now supports graceful stop

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: zhiltsov-max sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Depends on #2407

Coverage decreased (-0.6%) to 52.05% when pulling 99ee45dbfdc08ff94751452e54ceb32550a678b4 on zhiltsov-max:zhiltsov-max/feature-graceful-stop into 1e0c14204b4abba584cfbc4c18e7327c48eb3a9f on Microsoft:master.

Coverage decreased (-0.6%) to 52.05% when pulling 99ee45dbfdc08ff94751452e54ceb32550a678b4 on zhiltsov-max:zhiltsov-max/feature-graceful-stop into 1e0c14204b4abba584cfbc4c18e7327c48eb3a9f on Microsoft:master.

@yqwang-ms, please read my comments above and here.

Do we have to change Launcher code to achieve your goal?

Probably yes, since Launcher is responsible for building the command to be executed in Docker container. YARN is also responsible as it adds an extra shell to the one Docker starts with. To resolve the issue, we have either:

to remove bash -c and install a TERM handler in YARN Docker command config
to remove bash -c and install a TERM handler in Launcher command

Currently, both traps are required. If you think it is simpler to modify YARN - I have nothing against.

What is the benefit?

Users can opt in to save their work (long running training) before the container stop by a signal. Caffe framework does this, for instance. Python training scripts also can do this.

I mean "What is the benefit to change Launcher code".
After app is killed (Launcher Framework Stopped), YARN can already graceful kill containers by sending signal first then force kill. So all container can catch this signal and graceful shutdown.
It is reasonable to let task container to graceful shutdown, such as saving checkpoint etc, but we do not need AM container to graceful shutdown, because LauncherAM can tolerate unexpected shutdown, it does not need graceful shutdown.

@yqwang-ms,

YARN can already graceful kill containers

Yes, and this is used here, actually. I've just checked one more time - the only reason for the change in Launcher is graceful stop of AM. If AM does not need to unregister in this case, I think there are no reasons left to modify AM or Launcher, like you are suggesting. The only reason to stop gracefully for AM is just a correct stop then.
What I'm not sure about is the difference between yarn.nodemanager.sleep-delay-before-sigkill.ms and yarn.nodemanager.process-kill-wait.ms.

NM try to kill container pipeline:

getContainerPid (This can fail, so retry for max yarn.nodemanager.process-kill-wait.ms, but usually this can succeed immediately)
TERM the container identified by the pid
Wait yarn.nodemanager.sleep-delay-before-sigkill.ms
KILL the container identified by the pid

So I think your PR only need to take care of yarn.nodemanager.sleep-delay-before-sigkill.ms

Might be a good idea to increase the value of sleep before sigkill

@yqwang-ms, removed AM graceful stop as it is not very important.
@fanyangCS, what do you think the default value should be? This value is a cluster administrator responsibility, it could probably have a more natural way for its modification - through the cluster model, for example.

@zhiltsov-max For Launcher, it would be better if you could remove all callers of zkStore.stop(); instead of just move around publishFrameworkInfo, i.e. remove all

Thanks!

Hi @zhiltsov-max, to unblock the launcher fix, I have moved it to another PR:
https://github.com/Microsoft/pai/pull/2435
Thanks for reporting and the proposal solution!

We add yarn container graceful stop in #2536. Big thanks for your contribution.

Implemented in k8s version

Implemented in k8s version


feat: Generate Tests for new middleware or routes

Description
As a User I would love to get generated Tests for new routes/middlewares each time I run dart_frog new middleware [routeName] or dart_frog new routes [routeName].
This can save me some time so that I don't have to create the tests manually.
Requirements

[ ] create tests for newly generated routes or middlewares

Additional Context
It could be even more beneficial to have this feature once dart_frog_test is released

That is a great idea! We are actively working on dart_frog_test and once we have definition about that package, we can come back to this one.


More gates

In particular ZZMax and PhasedX, perhaps also TK1 and TK2

ZZMax and PhasedX added in #58


Doesn't work with Android project using com.google.gms:google-services:3.2.1

I am using the Firebase services but these all seem to use v11.8.0 and I keep getting a "merge dex" Exception. I know I am a bad person for hardcoding your implementation to use
implementation "com.google.android.gms:play-services-location:11.8.0"
Is there a better way to do handle this situation?

This is a much safer fix for now.
I have removed my hardcoded version from your plugin and added to my android/app/build.gradle, just before dependencies:
configurations.all {
resolutionStrategy {
force 'com.google.android.gms:play-services-location:11.8.0'
}
}
dependencies {
...

Indeed, I would even suggest:
configurations.all {
    resolutionStrategy {
        eachDependency { details ->
            if (details.requested.group == 'com.google.android.gms' ||
                    details.requested.group == 'com.google.firebase') {
                details.useVersion "your version"
            }
        }
    }
}


[BUG] Fix keychain and SSL certificate conflicts with chia

Describe the bug

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior

Screenshots

Desktop

OS:
OS Version/Flavor:
CPU:

Additional context

Sorry, wrong repository! Delete if you can!


Enhance install script to handle commented and uncommented lines (#3632)

Enhance install script to handle commented and uncommented lines in shell file with user prompts for modification.

Track commented and uncommented lines in the file.
Prompt user to append or skip if the line is commented.
Ensure new lines are added only when necessary, based on user input.
To the fish_user_key_bindings.fish, the original logic would append the line to the end if no corresponding statement was found. I’ve adopted the same behavior for commented lines.

This is my first contribution to this open-source project, and I’m really excited to get involved.
I saw the discussion in issue #3632 and wanted to try implementing the proposed feature.
One concern I have is whether I should write test cases for this change. The existing Ruby tests seem not to support testing the fzf installation process directly. Additionally, I worry that the logic in the updated code might be a bit complex (though I’ve tried my best to keep it readable).
Please feel free to provide feedback on my changes. If there’s anything I can improve, I’d greatly appreciate your guidance.

Thanks for your interest in this project. If we're going to display the matching lines, we can just use the output of the grep comment.
Here's my take on the problem. What do you think?
diff --git a/install b/install
index 5b67e30..607c7ef 100755
--- a/install
+++ b/install
@@ -295,35 +295,43 @@ EOF
 fi

 append_line() {
-  set -e
-
-  local update line file pat lno
+  local update line file pat lines
   update="$1"
   line="$2"
   file="$3"
   pat="${4:-}"
-  lno=""
+  lines=""

   echo "Update $file:"
   echo "  - $line"
   if [ -f "$file" ]; then
     if [ $# -lt 4 ]; then
-      lno=$(\grep -nF "$line" "$file" | sed 's/:.*//' | tr '\n' ' ')
+      lines=$(\grep -nF "$line" "$file")
     else
-      lno=$(\grep -nF "$pat" "$file" | sed 's/:.*//' | tr '\n' ' ')
+      lines=$(\grep -nF "$pat" "$file")
     fi
   fi
-  if [ -n "$lno" ]; then
-    echo "    - Already exists: line #$lno"
-  else
-    if [ $update -eq 1 ]; then
-      [ -f "$file" ] && echo >> "$file"
-      echo "$line" >> "$file"
-      echo "    + Added"
-    else
-      echo "    ~ Skipped"
+  if [ -n "$lines" ]; then
+    echo "    - Already exists:"
+    sed 's/^/        Line /' <<< "$lines"
+
+    update=0
+    if [ $(\grep -vn "^[0-9]*:[[:space:]]*#" <<< "$lines" | wc -l) -eq 0 ]; then
+      echo "    - But they all seem to be commented"
+      ask "    - Continue modifying $file?"
+      update=$?
     fi
   fi
+
+  set -e
+  if [ $update -eq 1 ]; then
+    [ -f "$file" ] && echo >> "$file"
+    echo "$line" >> "$file"
+    echo "    + Added"
+  else
+    echo "    ~ Skipped"
+  fi
+
   echo
   set +e
 }

One concern I have is whether I should write test cases for this change. The existing Ruby tests seem not to support testing the fzf installation process directly.

No, we currently don't test install script. So don't worry about it. The importance of the script has declined since most users use a package manager to install fzf, and setting up shell integration is much easier now than it used to be, and it's feasible to manually add a single line to your shell configuration file.
https://github.com/junegunn/fzf/blob/master/README.md#setting-up-shell-integration

I’ve learned a lot about Shell scripting from your explanation. I truly admire the elegance and simplicity of your code.

Merged, thanks for taking the initiative!


Multiple brokers

The current implementation of Secor supports a single broker server only. Do you have any plan to extend to support multiple brokers? I've noticed something related has been discussed in a different ticket (https://github.com/pinterest/secor/issues/134#issuecomment-219106650), but couldn't find any further development on this topic.

I think it should be easy to extend the bootstrap servers to a list of
brokers, https://github.com/pinterest/secor/issues/134 has context on how
to do it.
On Tue, Nov 19, 2019 at 9:08 PM Sumin Byeon<EMAIL_ADDRESS>wrote:

The current implementation of Secor supports a single broker server only.
Do you have any plan to extend to support multiple brokers? I've noticed
something related has been discussed in a different ticket (#134 (comment)
https://github.com/pinterest/secor/issues/134#issuecomment-219106650),
but couldn't find any further development on this topic.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/1076?email_source=notifications&email_token=ABYJP753GSTQ5K7EES2ATE3QUTA5ZA5CNFSM4JPNLXB2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H2SWZJA,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYJP76FPRG3RNHG7HZLC6TQUTA5ZANCNFSM4JPNLXBQ
.

It should be easy to add a config property kafka.bootstrap.servers=host1:9092,host2:9092
Let me know if you still require this.


Streamlines example is failing with ValueError: URL request failed: Forbidden

Streamlines tests are currently disabled (commented out).

Closing as (another) duplicate of #266


Periodic Table Should Log Colors Chosen

For example, see https://boardgamearena.com/table?table=318026229 moves 210-215.

This also happened...empire was achieved twice?
Move 213 :5:34:23 PM
You draw and meld 8 Rocketry.
You achieve Empire.
You return 7 Periodic Table from your display.
Sharing bonus.
You draw 8 Quantum theory.
MadisonSilver draws and melds 8 Rocketry.
MadisonSilver achieves Empire.
MadisonSilver returns 7 Periodic Table from his display.
Sharing bonus.
MadisonSilver draws a 8.

That's a weird quirk of how BGA displays logs; if you go to the "game review" page (as opposed to the "archive/replay" page) it will show notifications from both players.  (You can see also "You draw and meld 8 Rocketry."/"MadisonSilver draws and melds 8 Rocketry." and "You return 7 Periodic Table from your display."/"MadisonSilver returns 7 Periodic Table from his display.")
I'm not sure if there's a way to fix it on our side or if it's just an issue with BGA.

Submitted the duplicated logs as #756.  Not sure if there's a way to fix them (or if we even care), but it may be worth investigation.


ECCV 2020 tutorial: Interactivity and Advanced Features

Motivation and Context
Tutorial Colab staging.
Types of changes

[x] Docs change / refactoring / dependency upgrade
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have completed my CLA (see CONTRIBUTING)
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

@aclegg3 Should pass the CI tests now. vut.display_video is also blocking on circleci and we need to conditionally use/import ipython widgets.

Here is the temporary link for this notebook: https://colab.research.google.com/github/facebookresearch/habitat-sim/blob/colabs-interaction/examples/tutorials/colabs/ECCV_2020_Interactivity.ipynb

@aclegg3 Do we want to use the 'auto' feature for any of these interactivity sliders? Like below:
#@title After running this cell manually, it will auto-run if you change the selected value. { run: "auto" }

option2 = "A" #@param ["A", "B", "C"]
print('You selected', option2)


Touch overlay event consumed early in scroll view

Observed behaviour
In a scrollview, touch overlay stops working if you move slightly up or down and trigger the scroll event of the parent.
version
1.2.0

Unfortunately this is a feature of the Android Touch system. Without ruining the ScrollView functionality from the library side, a working solution is to call the LiveChart view's parent requestDisallowInterceptTouchEvent(disallowIntercept: Boolean) :
liveChart.setDataset(dataset)
                .setOnTouchCallbackListener(object : LiveChart.OnTouchCallback {
                    override fun onTouchCallback(point: DataPoint) {
                        liveChart
                            .parent
                            .requestDisallowInterceptTouchEvent(true)
                    }

                    override fun onTouchFinished() {
                         liveChart
                              .parent
                              .requestDisallowInterceptTouchEvent(false)
                     }
                })
                .drawDataset()


Gsoc2021 double float chris

Hi @sinandredemption I noticed a few issues when handling the big name changes for the convenience classes.
A few points on this PR.

I was not fully happy with the specializations of eval_add(), eval_sub() and eval_mul() so I actually removed them. We can replace these if needed. But I would actually prefer to handle mixed arithmetic proberly at a later time post GSoC.
There is a problem calling sqrt in quad-float when the input parameter is, for instance, exactly equal to 4 and expression_templates is turned on. So I left the quad-float Boost-style tests having expression templates off.

There is a problem calling sqrt in quad-float when the input parameter is, for instance, exactly equal to 4 and expression_templates is turned on.

I am not yet extremely familiar with expression_templates, but I will be willing to look into the problem. Since we are targeting a complete implementation over a fast one, is this something you suggest we should work on before GSoC '21 ends?

we are targeting a complete implementation over a fast one, is this something you suggest we should work on before GSoC '21 ends

We have already over-achieved for this GSoC. And @sinandredemption many thanks for your fantastic work.
I intend to try one more push to this PR with some specializations of eval_add(), eval_sub(), eval_mul() and eval_div(). We discussed these in another thread.
After that, I would like to track down the remaining known bugs. The bugs include the failure of quad-float on square root of a pure integer valued-quad-float. And the bugs also include #35, which I can look into myself.


Make a Model configurable

Please, make a model configurable in the settings.
I'd like to be able to switch to different model (e.g., gpt3.5, gpt-4o) which should be cheaper and for some scenarios more suitable then default gpt4.
Please provide a dropdown with latest available models, or make the setting editable so that I can set the model name manually.

Definitely agree!
I would also love to see an option to use Anthropic Claude Sonnet 3.5 or Ollama local LLM.

also request for an option that allows custom urls, instead of the defualt openai url.


Allow nativepicker for mobile version

@zakibinary i changed it to flex, for better support across browsers.


Update ARMEmulateInstructionStateModifier.java

Reasons for changes:
Method Extraction: Moved register initialization into a separate method to improve readability and maintainability. This makes the constructor cleaner and easier to understand.
Simplified Conditionals: Simplified conditional checks in initialExecuteCallback and postExecuteCallback for better clarity and readability. This reduces cognitive load for developers who need to understand the logic at a glance.
Modularization: Extracted logic for handling thumb mode and ARM mode into separate methods to improve modularity and readability. This makes each method focus on a single responsibility, making the code easier to understand and maintain.
Removed Unnecessary Comments: Removed unnecessary comments for improved code readability. Comments should only be used when they add valuable information or clarification, not for restating what the code does.

I just wanted to confirm that your intent was not to make any behavior changes, correct?

Readability is non-trivial.
On Mon, Apr 15, 2024 at 4:46 PM Ryan Kurtz @.***> wrote:

I just wanted to confirm that your intent was not to make any behavior
changes, correct?
—
Reply to this email directly, view it on GitHub
https://github.com/NationalSecurityAgency/ghidra/pull/6419#issuecomment-2056902572,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BFMUSBJRQTDIJV4V5HUMYHLY5PKZ7AVCNFSM6AAAAABGGCTNHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJWHEYDENJXGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

See Patch Submission Tips.


[NETBEANS-54] Module Review o.apache.commons.codec

Added maven coordinate for external resource commons-codec-1.3.jar

commons-codec has a notice file, that should be included in the distribution. See for an example here:
https://github.com/apache/incubator-netbeans/commit/a204dcca4b7bd99954e6e67a0331ec172542332c

Added the Notice file from http://svn.apache.org/repos/asf/commons/proper/codec/tags/CODEC_1_3/NOTICE.txt


add 2 comment lines

Your Pull Request code is being promoted!  Please follow the link below.
Main Merge Workflow


avoid calling buggy convert/piecewise

Ken says:

I'm thinking we'd keep the call in KB:-assert_deny and remove the call in Loop:-intssums.

I assigned it to 3 people, whoever gets to it first should close this issue.

By the way, here's a less severe problem I just hit in convert/piecewise as called in KB:-assert_deny:
> try convert(piecewise(i<=size[idx[a,i]]-2,true,false),piecewise,i); catch: print(caught); end try
Error, (in unknown) too many levels of recursion

Why do you say 'less severe'?  For one thing, the try/catch didn't, so this bug can't be worked around!
You are using convert/piecewise as a proxy for normalizing (simplifying, whatever you want to call it) boolean expressions, right?  Yes, it does seem to work that way, sometimes, in more powerful ways than other routines.  But maybe we should be more precise about the questions we ask.
Anyways, you should probably file an SCR about this bug.


Fix exception causes in api.py

Thanks for the contribution!
Could you explain a bit what this makes better? That is, do you have an example where this change leads to better error messages?

@mattjj
In all the cases I've touched, before the change, you would get a 2 tracebacks for the 2 exceptions, with this message between them:
During handling of the above exception, another exception occurred:

This text is not appropriate here. It basically means "Your error-handling code has a bug in it" which is not the case here, your error-handling code is working well. After my change, the correct message will be displayed:
The above exception was the direct cause of the following exception:

@hawkinsp
My pleasure. I'd like to raise awareness about this feature, so people will pay attention to the message and understand its meaning.

I wonder if we should be suppressing the inner exception entirely because it describes an implementation detail

I'm gonna recommend against that. When I'm debugging, I'm like a man who got lost in the desert and is about to die of thirst. Any possible insight into what happened is like an oasis, even if there are just a few drops of water there.
Also, some tools like Django and Sentry show you all the local variables for your stacktraces, which is a godsend. These often have important information that sheds light on what went wrong, and if you remove the traceback they'll be gone.

Let me know if you'll be interested in getting PRs for this same issue for other parts of the code.


Precise execution according to waterline

What type of PR is this?
Feature
What this PR does / why we need it:
Add waterline for precise execution, select how many pods will be executed.
The sort logic will be put into executor module use with waterline.
Proposal: #357
Some things to complete

[x] add a proposal to expound sort rule and how it works
[x] there are some repeated codes like pod_info same with #201, and should be rebase on that

Please have a square for the commits.

What relationship between this PR and PR #201 ? Is this PR depends on PR #201 to be merged into the main branch?

How about writing some overview for these pics?
Makes the whole logic flow smoothly.

Ok, these pics are test results, i have added some comments. This pr has a related proposal https://github.com/gocrane/crane/pull/357, thx~

How about writing some overview for these pics? Makes the whole logic flow smoothly.

Ok, these pics are test results, i have added some comments. This pr has a related proposal https://github.com/gocrane/crane/pull/357, thx~

Hi,pull requests that are too big can be difficult to review effectively. here are some suggestions to make code readable.

Thanks for your code review! There are so many helpful comments! I have fixed them. Thx!

Because of the complexity of this requirement, let's merge into the PR first, and we can continue to discuss it on the PR and propose a new PR for modification. Thanks very much to everyone.

/LGTM


Getting an error calling model.score

Hi!
Maybe I did something wrong but I think this is the simplest example to get a score out.
When I call model.score(xTest, yTest) I get the error Labels can't be converted to a 1D Tensor.
const btcusdtData = await ensureCryptoData('BTCUSDT', '1d')
const df = new dnf.DataFrame(btcusdtData)
// Couldn't use a list of column names here unfortunately so I used their index numbers. Would be a lot cooler if I could use 'open', 'high', etc.
const x = df.iloc({ columns: [1, 2, 3, 5, 6, 7, 8, 9, 10, 11] })
const y = df.iloc({ columns: [4] })
const [xTrain, xTest, yTrain, yTest] = sk.trainTestSplit(x.tensor, y.tensor)
const model = new sk.LinearRegression()
await model.fit(xTrain, yTrain)
const score = model.score(xTest, yTest)

Hey @Industrial, thanks for reaching out. I think what's happening is your
const y = df.iloc({ columns: [4] }) line is actually returning a Dataframe to you which has rank 2, instead of rank 1. Meaning it's a 2D matrix with only 1 column.
An easy way to check that is to print y.shape you'd (you'd likely see an array with 2 values instead of 1). Basically everything that we use as a y variable in the whole library needs to be 1D (either a simple array, or a 1-D Tensor, etc)
There are functions that will just grab a single column with danfo, namely df.column("open") or even just df['open'] but those require column headers which it looks like you don't have. Another (admittedly hacky) way it to simply take your 2D tensor and convert to a 1D tensor before passing it into trainTestSplit, by flattening it with .reshape.
Maybe something like
y.tensor.reshape([-1]) instead of just y.tensor
Hopefully this helps get you out of the jam, but if not feel free to post again, with your data, and we can go through it together. All the best 😄


[test] Fix QuantumESPRESSO 6.8.0 with 21.08

Fixing https://jira.cscs.ch/browse/UES-1735

@jenkins-cscs retry daint dom eiger

Codecov Report

Merging #2150 (f59704a) into master (5b17b95) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2150   +/-   ##
=======================================
  Coverage   86.30%   86.30%
=======================================
  Files          53       53
  Lines        9563     9563
=======================================
  Hits         8253     8253
  Misses       1310     1310

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b17b95...f59704a. Read the comment docs.


New event log experience

All policy rules are derived from the fields in the {system event logs, arbitrary .evtx event log files}
Users can select from publisher rules, attribute rules, file path, hash and packaged apps
Within publisher and attribute rules, multiple properties can be combined using the checkboxes

Publisher:

File Attributes:

Original filename, internal name, product name, file description are all selectable/can co-exist

File Path:

The user has the option between allowing by the exact file path and/or using the parent dir + wildcard

File Hash:

All 4 hashes are always added to the policy

"Add Allow Rule" button creates an SiPolicy object behind the scenes, and updates the table to reflect that the rule has been added to the new policy.

WDAC finally supports creating rules from the info in the event logs woohoo, @jsuther1974


feat: add Jina API for Embeddings

PR to add Jina Embeddings API as provider.
https://jina.ai/embeddings/
I have a question about how the documentation must change to show this feature and how tests need to be added?

You also need to add a info.json for the providers.

You also need to add a info.json for the providers.

Done


Add priority field to Record

Some record types like MX and SRV support a priority. This change adds an additional field to the Record definition to support the priority setting in cloudflare.

To make sure custom_certificate_quota is always right I can think of somethink like this: http://play.golang.org/p/B0Tqb3sG0G

@crackcomm Thanks for the input. Added another commit including your suggestions.

Looks good, thanks.


"vboxmanage: command not found" running freebsd-vm action on macos-latest

/bin/chmod 700 /Users/runner/.ssh
/usr/bin/sudo vboxmanage import /Users/runner/work/_actions/vmactions/freebsd-vm/v0.1.5/freebsd-13.0.ova
sudo: vboxmanage: command not found
Error: The process '/usr/bin/sudo' failed with exit code 1

I've noticed this issue in our CI which I tried to overcome by switching to the latest action (v0.1.5). This did not help for the reason that I was running on 'macos-latest' instead of 'macos-10.15'. I eventually overcame it on my side starting to use 'macos-10.15', but you might be interested in looking why some 'macos-latest' start to signify macos 11.6.1 which miss 'vboxmanage'.
See: https://github.com/intel/libva/pull/556
Job failed: https://github.com/intel/libva/runs/4452520572 (that's with macos-latest, before I switched to macos-10.15)
Jon passed: https://github.com/intel/libva/runs/4453434425 (that's with macos-10.15)

@dvrogozh Finally, somebody managed to spot this! I have been trying for days to see what might be going on...
Although it's not a definitive solution, it certainly solves the issue.
Thanks for posting it! 😉


⚠ Bofrost.de has degraded performance

In 57bb351, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1605 ms

Resolved: Bofrost.de performance has improved in 359631f after 9 minutes.


Purpose of using 1D convolutions

Hello,
Could you explain me the purpose of using 1D convolutions in the encoder layer?
self.attention = attention
self.conv1 = nn.Conv1d(in_channels=d_model, out_channels=d_ff, kernel_size=1)
 self.conv2 = nn.Conv1d(in_channels=d_ff, out_channels=d_model, kernel_size=1)
 self.norm1 = nn.LayerNorm(d_model)
self.norm2 = nn.LayerNorm(d_model)

Hello,
Could you explain me the purpose of using 1D convolutions in the encoder layer?
self.attention = attention
self.conv1 = nn.Conv1d(in_channels=d_model, out_channels=d_ff, kernel_size=1)
 self.conv2 = nn.Conv1d(in_channels=d_ff, out_channels=d_model, kernel_size=1)
 self.norm1 = nn.LayerNorm(d_model)
self.norm2 = nn.LayerNorm(d_model)

Seems that now many implementations use Con1d as the "MLP" projection to transform the original input to latent space embeddings, especially Transformers in the recent few years. Maybe just base on claims, you could try a Linear layer and see the effects. It's also worth noting that for Conv layers, there are some weight parameter initialization methods such as Kaiming He's method, which may leads to better performance.


Show incorrect address book type

Describe the bug
Show incorrect address book type
To reproduce the behavior
Steps:

Go to the send fund screen
Input new address on the "Send to" field
Input valid data for remain fields
Click on Transfer button
Go to the Manage address book screen
Observe the address book above

Actual: Show address on the "Saved contact" type

Expect: Show address on the "Recent" type

If use address to create transaction, this address move from recent to save contact type

Version test: https://file.cdmteck.com/s/4oimk9zDJ8R5Aai/download?path=%2F&files=./upgrade-ui-pr-1540-build-1.0.9-2023-06-14T04-45-57UTC.zip
Validation passed.


[GHE Integration] Report Review Results to PR

As a Zally GHE Integration Solution user
I want to receive the results of the API review in the respective PR
So that I get a feedback about my API changes and react to them
Acceptance Criteria

"No API changes detected" comment is sent if API specs haven't changed
"API Review: #x MUST, #y SHOULD, #MAY, #HINT violations found" comment is sent if any violations
"API Review: Wow! No violations! Great Job!! Zalando API Linter Loves You :)"

Nice to have
Some nice/funny phrases for some special use cases (e.g. no MUST violations, only one violation, too many violations,..).

Closed via #488


time zone inconsistency timers

From @tomvdduin on August 12, 2014 7:3
the type ts_array is a table of timestamp. But the procedures time_start and time_stop uses systimestamp; that has a time zone specified. I am at GMT+2, and regularly have something like this in the logger table:
START: test
STOP : test- 01:59:59.969
So the ts_array shold be defined by timestamp with time zone or you should use localtimestamp instead of systimestamp
Copied from original issue: tmuth/Logger---A-PL-SQL-Logging-Utility#74

We're looking at re-writing the timer methods and this is good information to take into consideration when designing them.


[Bug]: Debugging doesn't work

All of the following criteria must be met

[X] All Requirements must be installed.
[X] Full Latest.log file included. If no file exists then leave this unchecked and state so.

All of the following are optional to answer

[ ] Tried reinstalling the Game.
[ ] Tried reinstalling MelonLoader.
[ ] Tried restarting PC.
[X] Was able to see the Start Screen.

Describe the issue.
Latest.log
I'm using dnSpy version 6.1.8 (x64), when I try to debug the game, following the steps in the wiki, it fails:
unable to connect to the debugged process [...] make sure you have patched the mono.dll file of the game
Unity: 5.5.3f1
net35
Did you attach your log file?

[X] Yes, I attached my log file to the text box above.
[ ] No, I could not find a log file at {Game_Directory}\MelonLoader\Latest.log

well, did you get the patched mono? https://github.com/dnSpy/dnSpy/wiki/Debugging-Unity-Games

Allright I had to compile mono.dll from the official repo, but now it works!


Make AR.js work on iOS 11

I got AR.js to work on iOS 11 and here are the changes I needed to apply to aframe-ar.js in our project:
https://github.com/freinbichler/ar-room-availability/commit/bdbdaaf4a53c8f541557a0d9b453f42ea9e383c3
iOS 11 only supports the newer navigator.mediaDevices.getUserMedia in favor of the older navigator.getUserMedia. Also, the video element needs to have the attributes autoplay, muted and playsinline for it to work.
With these changes, this example code works perfectly:
<a-scene embedded artoolkit='sourceType: webcam;'>
	<a-box position='0 0.5 0' material='opacity: 0.5;'></a-box>
	<a-marker-camera preset='hiro'></a-marker-camera>
</a-scene>

An issue I am having is that in portrait mode the camera image is rotated by 180 degrees, but this might be an iOS beta issue, as I am also seeing this behavior in an unrelated test.
I don't know if the migration to navigator.mediaDevices.getUserMedia will break something, but in my tests it still works in the latest Chrome for Android and Mac. Also, I have only tested this in the aframe version, so I don't know where else this change might be needed.
Demo here: https://twitter.com/fr3ino/status/872228441390813185

i ported your modifications.
https://github.com/jeromeetienne/AR.js/blob/master/three.js/threex-artoolkitsource.js#L230
Can you test on ios11 and see how it goes ? thanks :)

Yes, it works 🎉
There is a bug with the resize event though, see this video (ignore the flipped image in portrait mode, this is an early iOS beta bug). As soon as I rotate the device to landscape mode, the cube is not perfectly aligned with the marker anymore. If another resize happens (when the address bar appears in the video at 0:18), the cube is aligned again. So basically the problem is that the resize event fires and it tries to recalculate everything, when the browser is not finished rotating. This could be related to issue #91. Maybe a short delay after the resize event happens and before the recalculations are done would fix this. I am only experiencing this on iOS though.
Another problem is that in portrait mode the cube is stretched. I am seeing this on Android and iOS.
Thank you!

@freinbichler Thanks A LOT for your help during this whole process :)

doesn't work on my iPhone 7 iOs 11.1 What i doing wrong(
"webcam Error Name: Message: WebRTC issue-! navigator.mediaDevices not present in your browser"

You have to open it in standalone Safari, not Safari View Controller embedded in several apps. If you tap on the Safari icon in the bottom right of your screen, Safari opens and it should work.

it even doesn't open in standalone version of safari!

I feel that this is a new issue.
It was working last week.

I didnt have probs using iOS 11 and iphones with ar.js and aframe-ar
Ideami Studios
http://ideami.com
http://torchprinciple.com
On Thu, Nov 16, 2017 at 6:21 AM, Kevin Curry<EMAIL_ADDRESS>wrote:

[image: d1c85a20-c0fb-4b9d-bdf9-b18e2cb1ead2]
https://user-images.githubusercontent.com/69151/32875079-0555a57e-ca64-11e7-910a-395b01d5cdc5.png
[image: e6a4a742-91ef-48f9-9c54-253b23ec7b49]
https://user-images.githubusercontent.com/69151/32875080-056252ce-ca64-11e7-94bd-da2290b51470.png
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jeromeetienne/AR.js/issues/90#issuecomment-344819056,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA9EBCumR8lDUPSN5WMBfE7X4-HB7Hyaks5s28ZWgaJpZM4NyBu5
.

@freinbichler "You have to open it in standalone Safari, not Safari View Controller embedded in several apps"
I used aframe ar within a phonegap app...on android everything it's ok...on IOs it's not working...so you're saying currently Aframe just works in IOS Safari standalone browser?

As far as I know that is correct.

I am running iOS11.2.5 on an iPhone 7, using the latest version of Chrome and Safari. This example is working with Safari but not Chrome.

When using the example code on my own site, it's working just fine - again, only in Safari, not chrome.

Only standalone Safari supports getUserMedia right now. iOS 11.3 should change that.

Hello All,
I am having this issue of the webcam error.  I have even tried going to the codepen version in safari browser on iPad 10.3.2 and iPhone 11.2.6.  Always the same.  Any thoughts as to why this is happening?

yeah..here too. i tried in a iphone 7, with 11.0 on safari and seems not work.
this fix it's already in the library when we download from the github? or i have to fix something in the js?

Hello I'm trying to use with the iphone, but the camera will zoom automatically until it gives an error. Do you know what that mistake might be?

@felipemacch
I had the same problem but I've found a solution!
I allowed access to the camera and microphone, then I enabled ALL experimental WebKit features of Safari in advanced settings. For details, please, check screenshots below of my Safari browser settings. I use iOS 11.4 on Iphone 7plus.

I have found that this example : https://jeromeetienne.github.io/AR.js/three.js/examples/mobile-performance.html works on my iPhone with latest iOS 11 and all the experimental features and permissions set for Safari. But the A-Frame examples do not work.

Apparently the gyro and other features have been blocked per default by Apple, suggesting it could be used by thrid party services for tracking users. See: https://www.macrumors.com/2019/02/04/ios-12-2-safari-motion-orientation-access-toggle/

Leeprobert is right, that example works on my ipad

Hello, i have some problems with the Iphone with IOS 12.1.4
What can i do to work in my phone.
Thanks a lot.

You have to open it in standalone Safari, not Safari View Controller embedded in several apps. If you tap on the Safari icon in the bottom right of your screen, Safari opens and it should work.

Hello,  I am wondering, is it impossible for Safari View Controller? Any clue to make it possible on webView (UIwebview / WKwebview / SFSafariViewController )

On a wkwebview you won't have access to the camera. The MediaDevices.getUserMedia() return undefined This is a feature or bug (depends if you work at apple or not).
You'll need to play along with something like :  https://github.com/amelierosser/arkit-web
I Tested it a bit and was happy. It has been deployed for kikk 2018 by Amelie Rosser.
Based on this you will need to rework part of the AR.js api.
If you want to test something else, you can try : https://github.com/VirtuoWorks/CanvasCameraDemo which stream to a canvas and is fairly stable. You'll need phonegap or getcapacitor.
GD


iProxy

Do you have a real tutorial or details on how to configure this and use this? I'm running Kali can list the apps, but there is little to no documentation on how to configure this. I think it would be helpful to see step by step how to do this.

Found this guide
http://tony.coadou.fr/2017/05/16/how-to-mount-an-iphone-with-ios-10-on-kali-linux/


[improve][ci] skip CI cronjob in forks

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: https://github.com/tisonkun/pulsar/pull/14

@nicoloboschi Thanks for your review! I push one more commit e861b6bda23bd010fd5cd34ee381c6ac8eff9f75 to skip the always running pulsar-ci-checks-completed job. I'm unsure whether it's the proper way, perhaps we can alternatively check the needs.*.result but I found the pushed way is shorter.

Codecov Report

Merging #18411 (3866ab9) into master (b31c5a6) will increase coverage by 0.52%.
The diff coverage is 5.00%.

@@             Coverage Diff              @@
##             master   #18411      +/-   ##
============================================
+ Coverage     46.98%   47.50%   +0.52%
+ Complexity    10343     9274    -1069
============================================
  Files           692      613      -79
  Lines         67766    58393    -9373
  Branches       7259     6086    -1173
============================================
- Hits          31842    27742    -4100
+ Misses        32344    27654    -4690
+ Partials       3580     2997     -583

Flag
Coverage Δ

unittests
47.50% <5.00%> (+0.52%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...va/org/apache/pulsar/broker/service/ServerCnx.java
48.40% <ø> (-0.29%)
:arrow_down:

.../java/org/apache/pulsar/client/impl/ClientCnx.java
30.16% <ø> (ø)

...a/org/apache/pulsar/client/impl/TableViewImpl.java
0.00% <0.00%> (ø)

...ar/client/impl/conf/ProducerConfigurationData.java
84.70% <ø> (-0.18%)
:arrow_down:

...va/org/apache/pulsar/client/impl/ConsumerImpl.java
15.09% <12.50%> (+0.05%)
:arrow_up:

.../pulsar/broker/service/SharedConsumerAssignor.java
3.70% <0.00%> (-64.82%)
:arrow_down:

...apache/pulsar/broker/service/EntryAndMetadata.java
0.00% <0.00%> (-40.75%)
:arrow_down:

...oker/loadbalance/LoadResourceQuotaUpdaterTask.java
44.44% <0.00%> (-33.34%)
:arrow_down:

...lsar/broker/loadbalance/impl/ThresholdShedder.java
3.26% <0.00%> (-30.07%)
:arrow_down:

.../apache/pulsar/broker/loadbalance/LoadManager.java
61.11% <0.00%> (-16.67%)
:arrow_down:

... and 121 more

/pulsarbot run-failure-checks

It's ready to merge.
cc @lhotari @nicoloboschi please take a look.

@nicoloboschi You're right. I recover the always() && ... pattern and it seems the issue you reported vanished.
If we can reach a consensus on this version, let's see how the next cronjob works :)

ping @nicoloboschi @lhotari

/pulsarbot rerun-failure-checks

Thanks for your reviews! Merging...


[Slashtags] Red Updated to Discord.py v2.4 breaking compatibility with upstream dependency AdvancedTagScriptEngine

As per the announcement on the Red discord server the latest version now uses discord.py==2.4 but the dependency of Slashtags AdvancedTagScriptEngine is locked to v2.3.4 as per the requirements.txt.
Example of error from break in dependency chain compatibility:
redbot   | ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
redbot   | advancedtagscriptengine 3.1.4 requires discord.py==2.3.2, but you have discord-py 2.4.0 which is incompatible.

Just as an FYI, I have attempted to manually install AdvancedTagScriptEngine with --no-deps and while slashtags will load and there aren't any "errors" at the very least the user command I have configured won't execute. Just results in a silent fail, the worst kind of fail!


Failed installation on Linux causes syslog spam

After attempting and failing to install the application from the nightly build (geteduroam-cli-linux-amd64) my syslog was spammed with messages which look like this:
Nov  1 09:21:25 jonathan org.freedesktop.FileManager1[2958754]: Please enter your organization (e.g. SURF):

I was never presented with this option when attempting to install (in fact, the installer didn't present me with any options). This was likely an error on my part but it suggests that something isn't caught correctly. The message spam caused my syslog to grow to a problematic size.
Edit: Changed the above after I realised that problem was not persistent.

Is this not just the log that you double clicked on the CLI client in your file manager? You need to open the client in your terminal by navigating to the directory with cd and then run ./geteduroam-cli-linux-amd64 after making it executable with chmod +x geteduroam-cli-linux-amd64

Also if you prefer a GUI you can try the GUI package

Is this not just the log that you double clicked on the CLI client in your file manager?  I thought it was worth raising anyway as it struck me as a sharp edge. In my case, it filled my OS partition by bloating the log. At the very least, there is now a trail of bread crumbs if any other user makes a similar error. Feel free to close the issue if you don't want to take further action and thank you for your quick response!

Ah yes indeed it does spam a lot. I will see if I can handle this, thanks!

Did the client work for you in the end?

It did, thank you!

Could you test the binary in https://github.com/geteduroam/linux-app/releases/tag/nightly I made a commit that hopefully handles this

Tested and I can confirm that it no longer spams the syslog when double clicked in executable mode.

Great! It should also send a notifcation if "notify-send" is installed.


Logs say Detected blocking call to open inside the event loop by custom integration 'smart_irrigation'

What happened?
Logs say Detected blocking call to open inside the event loop by custom integration 'smart_irrigation' - no other side effects though.
diagnostics.json
How to reproduce
Run Smart Irrigation
Relevant log output
2024-06-19 10:51:49.834 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/calcmodules/localize.py, line 23: with open(os.path.join(main_path, LANGUAGE_FILES_DIR+os.sep+"en.json")) as f: (offender: /config/custom_components/smart_irrigation/calcmodules/localize.py, line 23: with open(os.path.join(main_path, LANGUAGE_FILES_DIR+os.sep+"en.json")) as f:), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1988, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 114, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 83, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 85, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 242, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 32, in headers_middleware
    response = await handler(request)
  File "/usr/src/homeassistant/homeassistant/helpers/http.py", line 73, in handle
    result = await handler(request, **request.match_info)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 54, in get
    return await WebSocketHandler(request.app[KEY_HASS], request).async_handle()
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 438, in async_handle
    async_handle_str(command_msg_data)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/connection.py", line 229, in async_handle
    handler(self.hass, self, schema(msg))
  File "/config/custom_components/smart_irrigation/websockets.py", line 226, in websocket_get_all_modules
    modules = coordinator.async_get_all_modules()
  File "/config/custom_components/smart_irrigation/__init__.py", line 1421, in async_get_all_modules
    s = m(self.hass, {})
  File "/config/custom_components/smart_irrigation/calcmodules/pyeto/__init__.py", line 57, in __init__
    super().__init__(name="PyETO", description=localize("calcmodules.pyeto.description",hass.config.language)+".", schema=SCHEMA, config=config)
  File "/config/custom_components/smart_irrigation/calcmodules/localize.py", line 23, in localize
    with open(os.path.join(main_path, LANGUAGE_FILES_DIR+os.sep+"en.json")) as f:

Which version are you running?
v2024.4.3
Diagnostics file

[X] I have attached a diagnostics file

Additional information
No response

I can confirm:
2024-06-27 14:11:52.833 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/localize.py, line 24: with open(os.path.join(main_path, LANGUAGE_FILES_DIR+os.sep+"en.json")) as f: (offender: /etc/homeassistant/custom_components/smart_irrigation/localize.py, line 24: with open(os.path.join(main_path, LANGUAGE_FILES_DIR+os.sep+"en.json")) as f:), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues Traceback (most recent call last):   File "/srv/homeassistant/bin/hass", line 8, in <module>     sys.exit(main())   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/__main__.py", line 209, in main     exit_code = runner.run(runtime_conf)   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/runner.py", line 190, in run     return loop.run_until_complete(setup_and_run_hass(runtime_config))   File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete     self.run_forever()   File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever     self._run_once()   File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1987, in _run_once     handle._run()   File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run     self._context.run(self._callback, *self._args)   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/helpers/event.py", line 1603, in _interval_listener     self.hass.async_run_hass_job(self._run_job, dt_util.utcnow(), background=True)   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/core.py", line 939, in async_run_hass_job     return self._async_add_hass_job(hassjob, *args, background=background)   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/core.py", line 748, in _async_add_hass_job     task = create_eager_task(   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/util/async_.py", line 37, in create_eager_task     return Task(coro, loop=loop, name=name, eager_start=True)   File "/etc/homeassistant/custom_components/smart_irrigation/__init__.py", line 494, in _async_update_all     async_dispatcher_send(   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/helpers/dispatcher.py", line 215, in async_dispatcher_send     async_dispatcher_send_internal(hass, signal, *args)   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/helpers/dispatcher.py", line 247, in async_dispatcher_send_internal     job.target(*args)   File "/etc/homeassistant/custom_components/smart_irrigation/sensor.py", line 126, in async_update_sensor_entity     self.async_schedule_update_ha_state()   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/helpers/entity.py", line 1254, in async_schedule_update_ha_state     self.async_write_ha_state()   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/helpers/entity.py", line 1000, in async_write_ha_state     self._async_write_ha_state()   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/helpers/entity.py", line 1126, in _async_write_ha_state     state, attr, capabilities, shadowed_attr = self.__async_calculate_state()   File "/srv/homeassistant/lib/python3.12/site-packages/homeassistant/helpers/entity.py", line 1065, in __async_calculate_state     if extra_state_attributes := self.extra_state_attributes:   File "/etc/homeassistant/custom_components/smart_irrigation/sensor.py", line 203, in extra_state_attributes     localize(   File "/etc/homeassistant/custom_components/smart_irrigation/localize.py", line 24, in localize     with open(os.path.join(main_path, LANGUAGE_FILES_DIR+os.sep+"en.json")) as f:

@jeroenterheerdt I hope you find the time to fix this soon as this is polluting the logs to an extent where i can not get any meaningful info out of them.

Already fixed in beta versions, so fix will come to production versions eventually.


Fixes to demo containers and juttle examples

@mstemm this is what's needed to release the new juttle-engine and deploy to demo.juttle.io:

made names of elastic containers unique
dc-postgres.yml must be last in the docker-compose command, for unknown reasons
worked around an elastic-adapter bug in the news demo
worked around a change in our batch semantics in the postgres demo
Verified locally that with this, all examples work.

lgtm.

Travis run failed on Node5, fine on Node4, @rlgomes will look into the failure (timing issue) but it's unrelated to these eminently good changes I'm merging.


Remove Has Digital Image facet from Concepts and Places (from 1881)

Problem Description: LUX shows positive results for using Has Digital Image facet on Places and Concepts, even a decision was made not to render these thumbnails, as they mostly stink.
Expected behavior:  Users will not see Has Digital Image facet on Concepts and Places.  Remove facet from UI

Links:

https://www.bugherd.com/projects/284041/tasks/1785
https://www.bugherd.com/projects/284041/tasks/2033
https://www.bugherd.com/projects/284041/tasks/1968
For testing
Removed
https://lux-front-tst.collections.yale.edu/view/results/concepts?cp=1&facetRequest=true&q={"AND"%3A[{"text"%3A"oil"%2C"_lang"%3A"en"}%2C{"text"%3A"paint"%2C"_lang"%3A"en"}]}&sq=oil+paint
Not removed (PRD)
https://lux.collections.yale.edu/view/results/concepts?cp=1&facetRequest=true&q={"AND"%3A[{"text"%3A"oil"%2C"_lang"%3A"en"}%2C{"text"%3A"paint"%2C"_lang"%3A"en"}]}&sq=oil+paint

Approved per UAT

deployed


Make page accessible

I ran an accessibility audit on the page and found accessibility issues like:

Missing form labels
Missing alternative text
Language missing/invalid
Redundant links
Broken ARIA reference etc

I would like to work on these, if this is approved.

@bl-aire Sorry for the late response, I'm not having so much free time to work on this project. If you are still up to it, feel free to work on it, I'd really appreciate it!

I'm on it. Thanks


#1293 Implement Matrix Trace

As discussed in #1293, this is an attempt to implement matrix trace operator for rank 2 tensors. It only takes tensors with rank 2 (similar to the trace design in Theano). Trace for higher rank tensors requires more discussion on how we want to implement it. One approach is to calculate trace for all rank 2 sub-tensors as numpy. Another is to implement tensor contraction (generalization of trace).

Can one of the admins verify this patch?

Jenkins, test this please.

Just some docs and naming nits, otherwise looks good!

@martinwicke Thank you so much for the review! I have updated them accordingly.

@martinwicke Thanks again for the review! Let me know if I need to squash these commits.

One more run of tests, then we're good.
Jenkins, test this please.

Merged.

Thank you!


Allow marking entries for opening without having to navigate to them first

Hello,
it would be nice if it would be possible to select entries without having to navigate to them first.
If there are several entries in the startify screen it is "a bit too much work" having to navigate there.
It would be nice if one could select entries as simple as directly opening them. For example:
b3b8s9 to select entries 3 and 8 for opening in a new buffer and entry 9 in an additional split.

Since I'd like to avoid introducing a new option that changes how b/s/v/t behave, what about using uppercase versions of them to "switch modes"?
So, B38S9 instead of b3b8s9. Then, you could even mix both modes and don't have to decide for setting an option to either this or that. That is, all indexes after S will be marked to be opened in a split, but you can still navigate with j/J and friends and a lowercase version like v would cancel the... "batch mode".
And if you really wanted that behaviour by default, you could overwrite the default mappings like this:
autocmd User StartifyReady nmap <buffer> b B

What do you think about this approach?

I also don't intent to lose the current functionality.
Adding additional keymappings sounds very good.

That is, all indexes after S will be marked to be opened with :split,

Is it guaranteed that indexes always have a single letter? If that is the case that sounds very well. B38S9 would then be possible.
I could also imagine a prefix to be used for it, eg. -b38-s9, but I think you should decide what you think fits best.

Is it guaranteed that indexes always have a single letter?

Nope, ambiguous mappings will always lead to a small delay determined by 'timeoutlen'. E.g. when you have indices 1 and 10, then hitting 1 will open index 1 only after a small delay.
So, to open 1 and 10 in batch mode:
V1<delay>10

I'll put up a PR soon and CC you. It's easier to discuss potential disadvantages when having a PoC. :)

Another idea I can think of is having something like a mode switcher to switch selection modes. I use the above mentioned - (but don't think it would be the best choice for such a mapping).
When starting startify it is in normal selection mode like today. b then marks the entry below the cursor for opening in a new buffer.
After pressing the mode switcher -. It switches to "remote selection mode" (or whatever you'd like to call it). Pressing b then acts as an operator and waits for the index of the entry to select. Pressing 1 selects entry 1. Pressing 12 within timeoutlen will select enty 12.
I am not sure how to react if timeoutlen is exceeded between the 1 and the 2. Directly opening entry 2 may not seem to be the expected outcome. But that applies to the existing multi selection mode as well (haven't tried it).
I think it would be good to visualize the "remote selection mode" somehow.

Implemented by https://github.com/mhinz/vim-startify/pull/359
Thanks for the request! ✨


some then to done (#393,#381) and more

please make sure in your scripts you use done when DONE with a promise instead of then for proper debugging/releasing,
remember: ...->done(function ($response) { ok }, function ($response) { fail });
chunks: push into $discord->guilds after every chunk, not when all chunks have been received
replaced get/push by offset* where possible, david says its faster

will do it over someday


Equivalent for P's null event / transition?

In plain P, we can write a machine like this one:
machine M {
  start state S {
    entry { print "Entry!\n"; }
    on null do { print "Queue is empty!\n"; }
  }
}

Null transitions are called whenever the machine is scheduled and performs a dequeue operation while the queue is empty. So this program would print the following as long as the machine keeps getting scheduled.
Entry!
Queue is empty!
Queue is empty!
...

How do we get this behavior in P#?
I'm working on the P->P# translation with @ankushdesai

It’s called Default:
https://github.com/p-org/PSharp/blob/master/Source/Core/Runtime/Events/Default.cs
Sent from my iPhone

On 12-Sep-2018, at 6:39 AM, Alex Reinking<EMAIL_ADDRESS>wrote:
In plain P, we can write a machine like this one:
machine M {
start state S {
entry { print "Entry!\n"; }
on null do { print "Queue is empty!\n"; }
}
}
Null transitions are called whenever the machine is scheduled and performs a dequeue operation while the queue is empty. So this program would print the following as long as the machine keeps getting scheduled.
Entry!
Queue is empty!
Queue is empty!
...
How do we get this behavior in P#?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Closing; let me know if there are any followups.

Thanks, @akashlal


Remove alternatives during prerm phase

Unexpected behavior
After executing apt remove temurin-8-jdk, the update-alternatives --remove will not be executed.
Cause

The existence of tool_path will be checked before update-alternatives --remove executes.
However, the tool_path does not exist in the postrm phase.

Solution

Execute update-alternatives --remove in the prerm phase

https://github.com/adoptium/installer/blob/master/linux/jdk/debian/src/main/packaging/temurin/8/debian/postrm.in#L14-L18

@karianna This is my first commit and I signed the Eclipse CLA just now.