Support U/Int128 parsing and U/Int128 literals

This PR adds:

String#to_(u/i)128(?) methods
(U)Int128 literal parsing
(U)Int128 modulo/divide methods in compiler-rt (to allow this to even work)

With this you can do crazy things like:
$ bin/crystal eval 'puts 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF_u128 - 1'
Using compiled compiler at .build/crystal
340282366920938463463374607431768211454

To not introduce breaking changes, the method deduce_integer_kind now uses the following priority:
Int32 > Int64 > UInt64 > Int128 > UInt128.
There aren't currently a lot of comments, I will hopefully add those soon.
Requires/Includes #11093
Related to #8373
Supersedes #10975
Closes #9516
Closes #10962

I :heart: this.  I would love to see U/Int128 support in Crystal.

A lot of tests seem to fail because they try to run the specs with the current release of crystal.
Of course all those UInt128 tests fail using the old compiler that doesn't support it.
When I run make crystal and then make std_spec, it always works.

I just noticed there is a little breaking change:
$ crystal eval 'puts 0b_u32'
0

$ bin/crystal eval 'puts 0b_u32'
syntax error in eval:1
Error: numbers cannot end with a prefix

I could easily change that behaviour however.

I think you might need specs.  I took a few swings at specs in some older PRs but I think you would probably want to take a slightly different approach.  Probably convert the test cases into the base10 number and do the assertions.
https://github.com/crystal-lang/crystal/pull/7093
https://github.com/crystal-lang/crystal/pull/8313
Also when I last worked on this I had to update some code in the symantics and syntax.  Have you tried using the Int128 in a program.  Does it work as expected?

I think you might need specs. I took a few swings at specs in some older PRs but I think you would probably want to take a slightly different approach. Probably convert the test cases into the base10 number and do the assertions.
#7093
#8313
Also when I last worked on this I had to update some code in the symantics and syntax. Have you tried using the Int128 in a program. Does it work as expected?

I have added specs (compiler specs and std specs), but I have only tested some basic math operations, nothing complicated. I'll probably need to do that.

Now there are specs for the new compiler-rt methods!

Everything should work now, specs are also there.
Reviews would now be very appreciated!

The only issue right now is the windows CI.
It returns the error "170141183460469231731687303715884105727_i128 doesn't fit in an Int128", which is wrong..

Alright.. after some testing it seems like windows is a major obstacle.
Here's a demonstration to show how bad the situation is (both crystal-master + my branch):
C:\crystal>crystal eval "puts 4_i128"
0

Windows really doesn't like my division/modulo implementation.
C:\crystal>crystal eval "puts (10.to_i128 // 2.to_i128).unsafe_as(Tuple(Int64, Int64))"
{0, 0}

C:\crystal>crystal eval "puts (10.to_i128 % 2.to_i128).unsafe_as(Tuple(Int64, Int64))"
{0, 0}

C:\crystal>crystal eval "puts (10.to_i128 * 2.to_i128).unsafe_as(Tuple(Int64, Int64))"
{20, 0}

C:\crystal>crystal eval "puts (10.to_i128 << 2.to_i128).unsafe_as(Tuple(Int64, Int64))"
{40, 0}

But at least I know where to search for errors now..

Okay, appearantly windows doesn't call my compiler-rt methods.
Using them explicitly works.
C:\crystal>crystal eval "puts __udivti3(UInt128::MAX.to_u128,32.to_u128).unsafe_as(Tuple(UInt64, UInt64))"
{18446744073709551615,<PHONE_NUMBER>03423487}

I think I found a fix in rust's combiler-builtins repo:
https://github.com/rust-lang/compiler-builtins/blob/dab437981078f052a87ff7a435005b276be68770/src/macros.rs#L168

It seems like msvc passes the u/int128's using a pointer to compiler-rt.
Now my compiler-rt methods finally have the numbers they need and can correctly calculate the result.
The only problem now is that I have no clue how to return the calculated numbers.

Thanks for this PR @BlobCodes, it's really awesome!
In order to get this merged, we'll have to split it into two stages. The first stage adding compiler support and all the stdlib requirements for that. Only after that's been merged and released we can then merge the changes that actually make use of those compiler features.
I wouldn't worry too much about windows support for now. It seems more complicated to work our. We can defer that after merging the feature for other platforms. It's not a requirement. You can just use flags to disable failing pieces on win32.
There should be more unit test for overflowing multiplication. There is already mulodi4_spec.cr, we should have respective ones for mulosi4 and muloti4. I actually started working on the latter, you can grab the code from https://github.com/straight-shoota/crystal/commit/2f714ec19cf23ca37b9235ac99a8240d90a43bf8 .

Thanks for this PR @BlobCodes, it's really awesome!
In order to get this merged, we'll have to split it into two stages. The first stage adding compiler support and all the stdlib requirements for that. Only after that's been merged and released we can then merge the changes that actually make use of those compiler features.
I wouldn't worry too much about windows support for now. It seems more complicated to work our. We can defer that after merging the feature for other platforms. It's not a requirement. You can just use flags to disable failing pieces on win32.
There should be more unit test for overflowing multiplication. There is already mulodi4_spec.cr, we should have respective ones for mulosi4 and muloti4. I actually started working on the latter, you can grab the code from straight-shoota@2f714ec .

Thanks, that's helpful!
3 _muloti4 specs actually failed because it uses the 128-bit division algorithm, in which I had mistakenly swapped two variables..
Now everything passes locally, will create the PR soon

Superseded by #11196

@straight-shoota I honestly think the most fair thing to do is for us, core team members, to take care of i128 support, at least for multiplying them. And that can be in a separate PR. Mixing that with parsing Int128 doesn't feel right. Why don't we break them appart?

Not sure if that's particularly fair, but yes of course, we can take care of that. But I have the impression that @BlobCodes has fun working on this, so why take it away?
Obviously, if nobody was working on that (or stops working), we should take it. Especially for the more complex tasks.
But as long as others want to work on something, I think our role should be primarly to support them to get their patches ready and merged.
However, of course nobody should feel responsible for finishing what they started, especially when the tasks grow while being worked on. Don't hesitate to step away if you don't want keep going.

Regarding the specific note about i128 arithmetics, I think that's really already completed in #11196. Except for windows support which I suggested to postpone (and I'm happy to work on afterwards, if nobody else is interested).

Regarding the specific note about i128 arithmetics, I think that's really already completed in #11196. Except for windows support which I suggested to postpone (and I'm happy to work on afterwards, if nobody else is interested).

Supporting the int128 arithmetics is not the problem.
The problem is that windows requires the divti/modti methods to return the integer in the xmm0 register.
To get it there, crystal needs SIMD support.
Basically, we'd need to do this:
{% if flag?(:win32) %}
  # Windows
  @[SIMD]
  record U64x2, a : UInt64, b : UInt64
  fun __divti3(a : Int128*, b : Int128*) : U64x2
{% else %}
# Unix
  fun __divti3(a : Int128, b : Int128) : Int128
{% end %}

It would be really nice if the core members could add this, because I have no clue about crystal codegen and llvm (I think there was already a implementation in a crystal issue)

@BlobCodes It seems SIMD in llvm is represented by the vector type: https://llvm.org/docs/LangRef.html#vector-type . It's almost like a StaticArray, except that it uses a different internal representation. I guess creating a type for it should be pretty easy. The hardest part will be to come up with a name for it 😁
But I was talking about how to dissect this PR. The interpreter doesn't work right now in linux because multiplication between Int128 doesn't work. I think a PR that just tackles that makes sense. It has nothing to do with parsing Int128. That's all I was saying. And for that, there's no need for you, @BlobCodes, to do it. The code is already in some of these PRs. One core team member could extract those changes and add specs on their own.


clone objects to trigger rendering

Fixes https://github.com/open-rmf/rmf-web/issues/496 by doing a shallow copy that forces a render.
Naively I am understanding there are some drawbacks to using shallow copies, but i'm not sure if they are relevant in our case. The alternative deep copy seems to be JSON.parse(JSON.stringify()).
It is taking about 1 seconds to load icons now switching tabs away / back to the buildings tab, when before it took minimal time, i'm wondering if that is something that can be fixed.

Think we can merge this in for now so that at least we have some visual icons first and address the time to load subsequently :)

The delays might have something to do with setInterval only running on the first timeout.


Qt 6.4.2 doesn't build properly on Lassen.

Qt 6.4.2 doesn't build the portions enabled by xcb on Lassen. Lassen is missing some of the xcb-util packages.
I tried installing the xcb packages in a directory in the thirdparty library directory. The following commands were used to install xcb and the required utilities.
tar zxf xcb-proto-1.13.tar.gz
cd xcb-proto-1.13
./configure --prefix=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3
make install

tar zxf libxcb-1.13.tar.gz
cd libxcb-1.13
./configure --prefix=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3 PKG_CONFIG_PATH=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3/lib/pkgconfig

tar zxf xcb-util-keysyms-0.4.0.tar.gz
cd xcb-util-keysyms-0.4.0
./configure --prefix=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3 PKG_CONFIG_PATH=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3/lib/pkgconfig

tar zxf xcb-util-renderutil-0.3.9.tar.gz
cd xcb-util-renderutil-0.3.9
./configure --prefix=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3 PKG_CONFIG_PATH=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3/lib/pkgconfig

tar zxf xcb-util-0.4.0.tar.gz
cd xcb-util-0.4.0
./configure --prefix=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3 PKG_CONFIG_PATH=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3/lib/pkgconfig

tar zxf xcb-util-wm-0.4.0.tar.gz
cd xcb-util-wm-0.4.0
./configure --prefix=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3 PKG_CONFIG_PATH=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3/lib/pkgconfig

tar zxf xcb-util-image-0.4.0.tar.gz
cd xcb-util-image-0.4.0
./configure --prefix=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3 PKG_CONFIG_PATH=/usr/workspace/wsa/visit/visit/thirdparty_shared/3.4.0/blueos/xcb/1.13/linux-ppc64le_gcc-8.3/lib/pkgconfig

I then tried pointing Qt at the xcb install location, but it always found the system one in /usr/lib64 with the partial xcb installation.

The cmake file used to determine if xcb is supported is src/gui/configure.cmake.
Logic along the lines of:
if((X11_SUPPORTED) OR QT_FIND_ALL_PACKAGES_ALWAYS)
    qt_find_package(XCB 1.11 PROVIDED_TARGETS XCB::XCB MODULE_NAME gui QMAKE_LIB xcb)
endif()
if((X11_SUPPORTED) OR QT_FIND_ALL_PACKAGES_ALWAYS)
    qt_find_package(XCB 0.3.9 COMPONENTS ICCCM PROVIDED_TARGETS XCB::ICCCM MODULE_NAME gui QMAKE_LIB xcb_icccm)
endif()
qt_add_qmake_lib_dependency(xcb_icccm xcb)
if((X11_SUPPORTED) OR QT_FIND_ALL_PACKAGES_ALWAYS)
    qt_find_package(XCB 0.3.9 COMPONENTS IMAGE PROVIDED_TARGETS XCB::IMAGE MODULE_NAME gui QMAKE_LIB xcb_image)
endif()

is used to determine if the required libraries are installed.
qt_find_package does eventually call find_package.
The following test determines if it has all the libraries and header files necessary.
# xcb_syslibs
qt_config_compile_test(xcb_syslibs
    LABEL "XCB (extensions)"
    LIBRARIES
        XCB::ICCCM
        XCB::IMAGE
        XCB::KEYSYMS
        XCB::RANDR
        XCB::RENDER
        XCB::RENDERUTIL
        XCB::SHAPE
        XCB::SHM
        XCB::SYNC
        XCB::XFIXES
        XCB::XKB
        XCB::XCB
    CODE
"// xkb.h is using a variable called 'explicit', which is a reserved keyword in C++
#define explicit dont_use_cxx_explicit
#include <xcb/xcb.h>
#include <xcb/xcb_image.h>
#include <xcb/xcb_keysyms.h>
#include <xcb/randr.h>
#include <xcb/render.h>
#include <xcb/shape.h>
#include <xcb/shm.h>
#include <xcb/sync.h>
#include <xcb/xfixes.h>
#include <xcb/xcb_icccm.h>
#include <xcb/xcb_renderutil.h>
#include <xcb/xkb.h>
#undef explicit

int main(void)
{
    /* BEGIN TEST: */
int primaryScreen = 0;
xcb_connection_t *c = xcb_connect(\"\", &primaryScreen);
/* RENDER */
xcb_generic_error_t *error = nullptr;
xcb_render_query_pict_formats_cookie_t formatsCookie =
    xcb_render_query_pict_formats(c);
xcb_render_query_pict_formats_reply_t *formatsReply =
    xcb_render_query_pict_formats_reply(c, formatsCookie, &error);
/* RENDERUTIL: xcb_renderutil.h include won't compile unless version >= 0.3.9 */
xcb_render_util_find_standard_format(nullptr, XCB_PICT_STANDARD_ARGB_32);
/* XKB: This takes more arguments in xcb-xkb < 1.11 */
xcb_xkb_get_kbd_by_name_replies_key_names_value_list_sizeof(nullptr, 0, 0, 0, 0, 0, 0, 0, 0);
    /* END TEST: */
    return 0;
}
")

@brugger1 is the the symptom you would see in VisIt if we try w/o xcb features?
/usr/workspace/cyrush/2024.01.build_visit/visit/src/winutil/WindowMetrics.C: In static member function 'static void WindowMetrics::CalculateBorders(QWidget*, int&, int&, int&, int&)':
/usr/workspace/cyrush/2024.01.build_visit/visit/src/winutil/WindowMetrics.C:480:71: error: 'QX11Application' is not a member of 'QNativeInterface'
     auto x11NativeInterface = qApp->nativeInterface<QNativeInterface::QX11Application>();

qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in ""

Yes, different symptom -- same cause.

More info from cmake when I try to force XCB on via -DFEATURE_xcb=ON
(I suggest we use this option for linux systems so that we stop building before we create a qt that lacks x11 support.)
Note: Disabling X11 Accessibility Bridge: D-Bus or AT-SPI is missing.

ERROR: Feature "xcb": Forcing to "ON" breaks its condition:
    QT_FEATURE_thread AND TARGET XCB::XCB AND TEST_xcb_syslibs AND QT_FEATURE_xkbcommon_x11
Condition values dump:
    QT_FEATURE_thread = "ON"
    TARGET XCB::XCB found
    TEST_xcb_syslibs = "FALSE"
    QT_FEATURE_xkbcommon_x11 not evaluated

I built against Qt 5 on lassen, figuring that there isn't any new significant functionality that we are using from Qt 6. I was mainly documenting the issues I ran into when I filed this ticket.
I also remember running into the issue with WindowMetrics.C mentioned earlier in this thread. I couldn't find a way around it when I disabled xcb.

@brugger1  can we use new VTK w/ Qt5 ?

@cyrush yes VTK 9.6.2 works with 5.14.2. My guess is it would even build with 5.10.

Fantastic, we will try that!

We will release with Qt5 for BlueOS systems

@cyrush I ran into this issue also and resolved it by installing the development packages listed here. Most of these you'll already have installed but I went through each one just in case. I ended up needing to install 4-5 development packages.

@griffin28  thanks - we asked LC to install the packages on lassen and other ibm+nvidia systems, but its non-trivial for them to do so and make take a while. So we will use VTK-9 with Qt5 for our 3.4.1 release.


WebDAV delete doesn't work id-based

Describe the bug
Deleting a file via an id-based webdav request doesn't work.
Steps to reproduce

Create a resource
Delete that resource via a webdav DELETE request: https://localhost:9200/remote.php/dav/spaces/{fileId}

Expected behavior
The file is being deleted. According to https://github.com/owncloud/ocis/issues/6526#issuecomment-1591030387 this should work.
Actual behavior
The server responds with 405 Method Not Allowed.

Fixed via https://github.com/cs3org/reva/pull/4771.


Finetuning on ModelNet40 and ScanObjectNN

According to my understanding, the results of Table 1 and 2 in the paper are finetuned on ScanObjectNN and ModelNet40. The current version provides the codes of pretraining and zero-shot evaluation on downstream datasets.
So could you release the implementations of finetuing on these two downstream datasets, including the definition of ScanObjectNN dataset?
Best Regards.

Hi @auniquesun, By default, we didn't integrate the finetune codes in our repo, since it will make the code over-complicated, and also because we don't change any fine-tune codes from their original repos; just use our pre-train framework to get the pre-trained weights then take the weights to their original repos then initialize and finetune on it, typically you might need to tune the hyper-parameters like lr since you are not training from scratch anymore.
here are the repos we take for the fine-tuning tasks:
https://github.com/guochengqian/PointNeXt
https://github.com/lulutang0608/Point-BERT
https://github.com/ma-xu/pointMLP-pytorch
https://github.com/yanx27/Pointnet_Pointnet2_pytorch

Thanks, I will try  them out.


camunda login still showing after logging in keycloak

Hello, I'm having trouble setting up camunda with keycloak, currently I followed two guides:

https://github.com/camunda/camunda-bpm-identity-keycloak
https://github.com/camunda/camunda-bpm-identity-keycloak/tree/master/examples/sso-kubernetes

I copied the three main classes KeycloakAuthenticationProvider.java , KeycloakIdentityProvider and WebAppSecurityConfig that are mainly involved in configuring this with the exception of this line:
 filterRegistration.setInitParameters(Collections.singletonMap("authentication-provider", "org.camunda.bpm.extension.keycloak.showcase.sso.KeycloakAuthenticationProvider")); I changed the second param to point to my current project directories so it won't error out.
Here is also my application.yml : https://pastebin.com/d6SJs6CD
For the keycloak settings I mainly followed the first guide. Only difference I made was I change the valid redirect uri to this: http://localhost:8081/* instead of http://localhost:8081/camunda/* if I changed it to the second one it results to a invalid redirect uri error.
I'm not sure what I'm doing wrong here, whenever I access the localhost:8081 (camunda) via browser it proceeds to keycloak's login once I entered the correct credentials it directs me to camunda's login.

Hi @JetTorres,

Do you have a user in Keycloak who is a member of the Keycloak "camunda-admin" group? Do you have a group called "camunda-admin" in Keycloak (see administratorGroupName of your configuration)? Upon the very first startup of your application you should see some log entries that the Identity Provider grants authorizations to that group. Creating groups and users is your part / the responsibility of Keycloak - the Identity Provider is readonly and thus only reads this information.
The user you're using to login - have you checked the user's groups?
In order to find out what's potentially wrong with your SSO setup you can set the log level of Spring Security to debug.

Hi @JetTorres,

Do you have a user in Keycloak who is a member of the Keycloak "camunda-admin" group? Do you have a group called "camunda-admin" in Keycloak (see administratorGroupName of your configuration)? Upon the very first startup of your application you should see some log entries that the Identity Provider grants authorizations to that group. Creating groups and users is your part / the responsibility of Keycloak - the Identity Provider is readonly and thus only reads this information.
The user you're using to login - have you checked the user's groups?
In order to find out what's potentially wrong with your SSO setup you can set the log level of Spring Security to debug.

Hi @VonDerBeck

Yes I have a user in keycloak under camunda-admin

I also have this session active in keycloak when I logged in the demo account:

How can I check that? The roles I had was query-group, query-user and view-user on realm-management

I saw these logs when I tried logging in https://pastebin.com/2VuLCvJ3

Hi @VonDerBeck

Yes I have a user in keycloak under camunda-admin

I also have this session active in keycloak when I logged in the demo account:

How can I check that? The roles I had was query-group, query-user and view-user on realm-management

I saw these logs when I tried logging in https://pastebin.com/2VuLCvJ3

Hi @JetTorres,
your application.yml misses any camunda.bpm configuration parameters...
If you look at the SSO example, you'll find especially the following parameters:
camunda.bpm:
  authorization:
    enabled: true
  webapp.application-path:

Wether you want to enable authorization in Camunda is your choice. Ok. But Camunda's web application path must match the path's within the Spring WebSecurity configuration. If you just copied my sample this will not match any more. Because in the example Cockpit is directly running under root. But from your log file I can see that your application is under the path camunda:
2021-01-26 19:55:14.975 DEBUG 25816 --- [io-8081-exec-10] o.s.security.web.FilterChainProxy        : /camunda/lib/ngDefine.js?bust=7.14.0 reached end of additional filter chain; proceeding with original chain
...
2021-01-26 19:55:15.328 DEBUG 25816 --- [nio-8081-exec-8] o.s.security.web.FilterChainProxy        : /camunda/app/welcome/fonts/IBMPlexSans-Italic.otf at position 1 of 17 in additional filter chain; firing Filter: 'WebAsyncManagerIntegrationFilter'

So it is very likely that the KeycloakAuthenticationProvider, which is responsible for wiring Spring Security with Camunda, hasn't been involved at all. Your Camunda application hasn't any knowledge about the authenticated user and its groups. You must either consider the /camunda application path in your Spring WebSecurity configuration (everywhere where there is /app), or you must add the configuration
camunda.bpm:
  webapp.application-path:

to your application.yml and move the Camunda Cockpit to the root path.
Does that help?
Kind regards,
Gunnar

Hi @JetTorres,
your application.yml misses any camunda.bpm configuration parameters...
If you look at the SSO example, you'll find especially the following parameters:
camunda.bpm:
  authorization:
    enabled: true
  webapp.application-path:

Wether you want to enable authorization in Camunda is your choice. Ok. But Camunda's web application path must match the path's within the Spring WebSecurity configuration. If you just copied my sample this will not match any more. Because in the example Cockpit is directly running under root. But from your log file I can see that your application is under the path camunda:
2021-01-26 19:55:14.975 DEBUG 25816 --- [io-8081-exec-10] o.s.security.web.FilterChainProxy        : /camunda/lib/ngDefine.js?bust=7.14.0 reached end of additional filter chain; proceeding with original chain
...
2021-01-26 19:55:15.328 DEBUG 25816 --- [nio-8081-exec-8] o.s.security.web.FilterChainProxy        : /camunda/app/welcome/fonts/IBMPlexSans-Italic.otf at position 1 of 17 in additional filter chain; firing Filter: 'WebAsyncManagerIntegrationFilter'

So it is very likely that the KeycloakAuthenticationProvider, which is responsible for wiring Spring Security with Camunda, hasn't been involved at all. Your Camunda application hasn't any knowledge about the authenticated user and its groups. You must either consider the /camunda application path in your Spring WebSecurity configuration (everywhere where there is /app), or you must add the configuration
camunda.bpm:
  webapp.application-path:

to your application.yml and move the Camunda Cockpit to the root path.
Does that help?
Kind regards,
Gunnar

Hi @VonDerBeck ,
I updated the settings in keycloak and specifically added this redirect URI:

Also, I updated my application.yml to explicitly call the URIs: https://pastebin.com/H92DE6Pj and added the config I missed. I ran camunda and typed it in the URL bar localhost:8081 and localhost:8081/camunda and we got this error log: https://pastebin.com/gx2TMPR6
I also tried debugging by adding breakpoints on the config classes, I noticed that it doesn't go through there.

Hi @VonDerBeck ,
I updated the settings in keycloak and specifically added this redirect URI:

Also, I updated my application.yml to explicitly call the URIs: https://pastebin.com/H92DE6Pj and added the config I missed. I ran camunda and typed it in the URL bar localhost:8081 and localhost:8081/camunda and we got this error log: https://pastebin.com/gx2TMPR6
I also tried debugging by adding breakpoints on the config classes, I noticed that it doesn't go through there.

Setting camunda.bpm.webapp.application-path: camunda means setting it to default. Which has no effect.
Camunda Cockpit consists of 3 paths, /app, /api, /lib - these are all under /camunda/* now. So you should cleanup your code a little bit. And check the package of the KeycloakAuthenticationProvider. If it still does not work you should debug the KeycloakAuthenticationProvider and check, if it is called and the token info can be sucessfully extracted or not.

Setting camunda.bpm.webapp.application-path: camunda means setting it to default. Which has no effect.
Camunda Cockpit consists of 3 paths, /app, /api, /lib - these are all under /camunda/* now. So you should cleanup your code a little bit. And check the package of the KeycloakAuthenticationProvider. If it still does not work you should debug the KeycloakAuthenticationProvider and check, if it is called and the token info can be sucessfully extracted or not.

Hi @VonDerBeck sir,
Even I am facing this issue. Could you please help me out ?
When I run my Camunda Application:
2024-12-05T12:17:49.438+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Application.
2024-12-05T12:17:49.441+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource User.
2024-12-05T12:17:49.444+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Group.
2024-12-05T12:17:49.447+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Group membership.
2024-12-05T12:17:49.450+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Authorization.
2024-12-05T12:17:49.455+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Filter.
2024-12-05T12:17:49.458+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource ProcessDefinition.
2024-12-05T12:17:49.464+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Task.
2024-12-05T12:17:49.466+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource ProcessInstance.
2024-12-05T12:17:49.472+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Deployment.
2024-12-05T12:17:49.475+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource DecisionDefinition.
2024-12-05T12:17:49.476+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Tenant.
2024-12-05T12:17:49.478+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource TenantMembership.
2024-12-05T12:17:49.480+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Batch.
2024-12-05T12:17:49.481+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource DecisionRequirementsDefinition.
2024-12-05T12:17:49.486+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Report.
2024-12-05T12:17:49.490+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Dashboard.
2024-12-05T12:17:49.493+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource OperationLogCatgeory.
2024-12-05T12:17:49.496+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource Optimize.
2024-12-05T12:17:49.499+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource HistoricTask.
2024-12-05T12:17:49.500+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource HistoricProcessInstance.
2024-12-05T12:17:49.501+05:30  INFO 22288 --- [  restartedMain] org.camunda.bpm.extension.keycloak       : KEYCLOAK-01002 GRANT group camunda-admin[camunda-admin] ALL permissions on resource System.

I am getting these in the logs.
And when I am rerouted back from the keycloak signin page, I am getting:
o.s.security.web.FilterChainProxy        : Secured GET /api/engine/engine/

--- User details in between ---

 o.s.security.web.FilterChainProxy        : Secured GET /error

I have followed the steps from the guide as well as the SSO files in your github repo.

@srikar1234,
This is a very old ticket and has been closed long ago...
What does your application.yaml look like? What does your WebAppSecurityConfig look like? What does your Keycloak setup, especially all the URLs (redirect, ...) look like? What happens if you temporarily allow all URLs to be valid in Keycloak?

Hi sir,
Thank you for taking the time to reply. Here we go:
Camunda version: 7.22.0
Spring: 3.2.2
Application.properties:
server.port = 8081
# Server Servlet Context Path
server.servlet.context-path=/camunda

spring.datasource.url=${JDBC_URL:jdbc:h2:./camunda-db-3;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=TRUE}
spring.datasource.username=${JDBC_USER:sa}
spring.datasource.password=${JDBC_PASSWORD:sa}
#spring.h2.console.enabled=true

#Org data
cms.service.endpoint.url=http://xxxxxx

#ass.hikari.maxpoolsize = 200

# Hibernate
#hibernate.dialect= org.hibernate.dialect.PostgreSQLDialect
#hibernate.show_sql = true
#hibernate.formate_sql = true
#hibernate.hbm2ddl.auto = none
#hibernate.auto_commit = false

spring.jersey.application-path=/engine-rest

#camunda.bpm.history-level=audit
camunda.bpm.authorization.enabled=true
camunda.bpm.filter.create=All tasks
camunda.bpm.webapp.application-path=

# Keycloak SSO URLs
keycloak.url.auth=https://xxx.co
keycloak.url.token=https://xxx.co
keycloak.url.plugin=https://xxx.co

# Keycloak Camunda Identity Client
keycloak.client.id=camunda-identity-service
keycloak.client.secret=XXXXXXXXXXXXXXXX

# Spring Boot Security OAuth2 SSO
spring.security.oauth2.client.registration.keycloak.provider=keycloak
spring.security.oauth2.client.registration.keycloak.client-id=${keycloak.client.id}
spring.security.oauth2.client.registration.keycloak.client-secret=${keycloak.client.secret}
spring.security.oauth2.client.registration.keycloak.authorization-grant-type=authorization_code
spring.security.oauth2.client.registration.keycloak.redirect-uri={baseUrl}/{action}/oauth2/code/{registrationId}
spring.security.oauth2.client.registration.keycloak.scope=openid, profile, email

spring.security.oauth2.client.provider.keycloak.issuer-uri=${keycloak.url.auth}/realms/CamundaTest
spring.security.oauth2.client.provider.keycloak.authorization-uri=${keycloak.url.auth}/realms/CamundaTest/protocol/openid-connect/auth
spring.security.oauth2.client.provider.keycloak.user-info-uri=${keycloak.url.auth}/realms/CamundaTest/protocol/openid-connect/userinfo
spring.security.oauth2.client.provider.keycloak.token-uri=${keycloak.url.token}/realms/CamundaTest/protocol/openid-connect/token
spring.security.oauth2.client.provider.keycloak.jwk-set-uri=${keycloak.url.token}/realms/CamundaTest/protocol/openid-connect/certs
spring.security.oauth2.client.provider.keycloak.user-name-attribute=preferred_username

# Logging
logging.level.org.springframework.security=DEBUG

## Camunda Rest API Security
rest.security.enabled=true
rest.security.provider=keycloak
rest.security.required-audience=camunda-rest-api

# Camunda Keycloak Identity Provider Plugin
plugin.identity.keycloak.keycloakIssuerUrl=${keycloak.url.plugin}/realms/CamundaTest
plugin.identity.keycloak.keycloakAdminUrl=${keycloak.url.plugin}/admin/realms/CamundaTest
plugin.identity.keycloak.clientId=${keycloak.client.id}
plugin.identity.keycloak.clientSecret=${keycloak.client.secret}
plugin.identity.keycloak.useEmailAsCamundaUserId=false
plugin.identity.keycloak.useUsernameAsCamundaUserId=true
plugin.identity.keycloak.useGroupPathAsCamundaGroupId=true
plugin.identity.keycloak.enforceSubgroupsInGroupQuery=true
plugin.identity.keycloak.administratorGroupName=camunda-admin

WebAppSecurityConfig.java
package com.vam.kshema;

import java.util.Collections;

import org.camunda.bpm.webapp.impl.security.auth.ContainerBasedAuthenticationFilter;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingClass;
import org.springframework.boot.web.servlet.FilterRegistrationBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.web.SecurityFilterChain;
import org.springframework.web.context.request.RequestContextListener;
import org.springframework.web.filter.ForwardedHeaderFilter;

/**
 * Camunda Web application SSO configuration for usage with KeycloakIdentityProviderPlugin.
 */
@ConditionalOnMissingClass("org.springframework.test.context.junit.jupiter.SpringExtension")
@EnableWebSecurity
@Configuration
public class WebAppSecurityConfig {

    @Bean
    @Order(2)
    public SecurityFilterChain httpSecurity(HttpSecurity http) throws Exception {
        http
                .csrf(csrf -> csrf.ignoringRequestMatchers("/api/**", "/engine-rest/**"))
                .authorizeHttpRequests(authorize -> authorize
                        .requestMatchers("/assets/**", "/app/**", "/api/**", "/lib/**").authenticated()
                        .anyRequest().permitAll());

        // If oauth2Login() is deprecated, verify the new method for OAuth2 configuration
        // This is a placeholder, replace with actual configuration if known
        // http.oauth2Login();

        return http.build();
    }

    @SuppressWarnings({ "rawtypes", "unchecked" })
    @Bean
    public FilterRegistrationBean containerBasedAuthenticationFilter(){
        FilterRegistrationBean filterRegistration = new FilterRegistrationBean();
        filterRegistration.setFilter(new ContainerBasedAuthenticationFilter());
        filterRegistration.setInitParameters(Collections.singletonMap("authentication-provider", "com.vam.kshema.KeycloakAuthenticationProvider"));
        filterRegistration.setOrder(201); // ensure the filter is registered after the Spring Security Filter Chain
        filterRegistration.addUrlPatterns("/app/*");
        return filterRegistration;
    }

    // The ForwardedHeaderFilter is needed to correctly assemble the redirect URL for OAuth2 login.
    // Without the filter, Spring might generate an HTTP URL even though the container route is accessed via HTTPS.
    @Bean
    public FilterRegistrationBean<ForwardedHeaderFilter> forwardedHeaderFilter() {
        FilterRegistrationBean<ForwardedHeaderFilter> filterRegistrationBean = new FilterRegistrationBean<>();
        filterRegistrationBean.setFilter(new ForwardedHeaderFilter());
        filterRegistrationBean.setOrder(Ordered.HIGHEST_PRECEDENCE);
        return filterRegistrationBean;
    }

    @Bean
    @Order(0)
    public RequestContextListener requestContextListener() {
        return new RequestContextListener();
    }
}

All the org and camunda java files are under the same folder: com.vam.kshema
Other dependencies in pom.xml file:
       <dependencies>
           <dependency>
               <groupId>org.springframework.boot</groupId>
               <artifactId>spring-boot-dependencies</artifactId>
               <version>3.2.2</version>
               <type>pom</type>
               <scope>import</scope>
           </dependency>
           <dependency>
               <groupId>org.camunda.bpm</groupId>
               <artifactId>camunda-bom</artifactId>
               <version>7.22.0</version>
               <scope>import</scope>
               <type>pom</type>
           </dependency>
       <dependency>
           <groupId>org.camunda.bpm.extension</groupId>
           <artifactId>camunda-platform-7-keycloak</artifactId>
           <version>7.22.0</version>
       </dependency>
       <dependency>
			<groupId>com.h2database</groupId>
			<artifactId>h2</artifactId>
		</dependency>
       <dependency>
           <groupId>org.camunda.bpm.springboot</groupId>
           <artifactId>camunda-bpm-spring-boot-starter-rest</artifactId>
       </dependency>
       <dependency>
           <groupId>org.camunda.bpm.springboot</groupId>
           <artifactId>camunda-bpm-spring-boot-starter-webapp</artifactId>
       </dependency>
       <dependency>
           <groupId>org.camunda.bpm</groupId>
           <artifactId>camunda-engine-plugin-spin</artifactId>
       </dependency>
       <dependency>
           <groupId>org.camunda.spin</groupId>
           <artifactId>camunda-spin-dataformat-all</artifactId>
       </dependency>
       <dependency>
           <groupId>org.camunda.bpm.springboot</groupId>
           <artifactId>camunda-bpm-spring-boot-starter-test</artifactId>
       </dependency>
       <dependency>
           <groupId>org.springframework.boot</groupId>
           <artifactId>spring-boot-starter-data-jpa</artifactId>
           <exclusions>
               <exclusion>
                   <groupId>org.apache.tomcat</groupId>
                   <artifactId>tomcat-jdbc</artifactId>
               </exclusion>
           </exclusions>
       </dependency>
       <dependency>
           <groupId>org.springframework.boot</groupId>
           <artifactId>spring-boot-devtools</artifactId>
           <scope>runtime</scope>
           <optional>true</optional>
       </dependency>
       <dependency>
           <groupId>org.projectlombok</groupId>
           <artifactId>lombok</artifactId>
           <scope>provided</scope>
       </dependency>
       <dependency>
           <groupId>org.springframework.boot</groupId>
           <artifactId>spring-boot-starter-security</artifactId>
       </dependency>
       <dependency>
           <groupId>org.springframework.boot</groupId>
           <artifactId>spring-boot-starter-oauth2-client</artifactId>
       </dependency>
       <dependency>
           <groupId>org.slf4j</groupId>
           <artifactId>slf4j-api</artifactId>
       </dependency>
       <dependency>
           <groupId>com.fasterxml.jackson.dataformat</groupId>
           <artifactId>jackson-dataformat-xml</artifactId>
       </dependency>
       <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
       <dependency>
           <groupId>org.apache.commons</groupId>
           <artifactId>commons-lang3</artifactId>
       </dependency>
       <dependency>
           <groupId>org.apache.commons</groupId>
           <artifactId>commons-collections4</artifactId>
           <version>4.1</version>
       </dependency>
       <dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
		</dependency>
       <dependency>
           <groupId>org.postgresql</groupId>
           <artifactId>postgresql</artifactId>
           <scope>runtime</scope>
       </dependency>
       <dependency>
           <groupId>org.springframework.boot</groupId>
           <artifactId>spring-boot-starter-jdbc</artifactId>
       </dependency>
       <dependency>
           <groupId>com.sun.xml.bind</groupId>
           <artifactId>jaxb-impl</artifactId>
       </dependency>
       <dependency>
           <groupId>org.camunda.template-engines</groupId>
           <artifactId>camunda-template-engines-freemarker</artifactId>
       </dependency>
       <dependency>
           <groupId>org.camunda.bpm</groupId>
           <artifactId>camunda-engine-plugin-connect</artifactId>
       </dependency>
       <dependency>
           <groupId>org.camunda.connect</groupId>
           <artifactId>camunda-connect-http-client</artifactId>
       </dependency>
       <dependency>
           <groupId>org.springdoc</groupId>
           <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
           <version>2.0.2</version>
       </dependency>
   </dependencies>

Please let me know if we can connect via a call next week incase you need to take a deeper look into the project file. We will try to fix this by then

Have you adapted your Keycloak URLs accordingly?
Example: if you take the SSO example as is and want to move the application root from /camunda to / you would have to do the following on localhost:
application.yaml:

Keycloak:

That's all. The WebAppSecurityConfig remains the same, because URLs are relative to Spring's servlet.context-path.

Btw: your WebAppSecurityConfig misses .oauth2Login(withDefaults()). It's not deprecated in spring-security-config:6.3.3
If you use newer versions you should have a look into Spring Security on how to migrate.


Relocate guava classes with maven shade plugin

to avoid class conflicts with other guava versions.
Fixes #826

Codecov Report

Merging #850 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #850   +/-   ##
=========================================
  Coverage     66.37%   66.37%
+ Complexity      743      742    -1
=========================================
  Files           166      166
  Lines          3138     3138
  Branches        357      357
=========================================
  Hits           2083     2083
  Misses          898      898
  Partials        157      157

@mattnworb What do you think?


Build failure for Android (libunwind remote is out of date)

"make android-all" is not working.
git remote for libunwind is out of date (make can't clone it).

Please fix it (AOSP version is building, but it is 1.9 version).

git remote is fine - it was just firewall.
Truly sorry for that mistake.
It would be great if you would change submodule url (.gitmodules & .git/config), to https (https://git.savannah.gnu.org/git/libunwind.git) as https protocol isn't affected by network setup.


Did you read the on-boarding document
Coming from https://github.com/opensearch-project/logstash-output-opensearch/issues/228#issuecomment-1719180937, please release opensearchproject/logstash-oss-with-opensearch-output-plugin docker image.
What is the name of your component?
opensearchproject/logstash-oss-with-opensearch-output-plugin
What is the link to your GitHub repo?
https://github.com/opensearch-project/logstash-output-opensearch/
Targeted release date
Any date possible.
Where should we publish this component?
Docker Hub
What type of artifact(s) will be generated for this component?
opensearchproject/logstash-oss-with-opensearch-output-plugin:latest should be pulling the latest logstash-output-opensearch 2.0.2
Have you completed the required reviews including security reviews, UX reviews?
this is a minor update
Have you on-boarded automated security scanning for the GitHub repo associated with this component?
yes
Additional context
No response

@dblock , This is correct. We are not releasing updates to the Docker image or the tar.gz file. We are releasing updates to logstash-output-opensearch as Ruby Gems which are available here.

@dblock , This is correct. We are not releasing updates to the Docker image or the tar.gz file. We are releasing updates to logstash-output-opensearch as Ruby Gems which are available here.

So is the docker image EOLed?

Yes, the Docker image and the tarball are EOL.

Yes, the Docker image and the tarball are EOL.

Who can add this information to the description for the public docker-image?
It seems to has 30k downloads a week. https://hub.docker.com/r/opensearchproject/logstash-oss-with-opensearch-output-plugin

@SebastianThorn We should. @peterzhuamazon do you know how?

@dblock We can add the below information to the description. Let me know if it looks ok.
DEPRECATION NOTICE:
This image is no longer supported / maintained (The image was last updated August 10, 2023). Please read this blog for more information.

Looks good to me! Thanks.

Looks good to me! Thanks.

done!


Cannot read property 'dragHandle' of undefined

I've been getting reports from production of an exception 'Cannot read property 'dragHandle' of undefined' on imageCropper.ts:927
I was not able to reproduce the error nor I understand what impacts it has on the application.
Maybe the fix would change the if to just if (dragTouch)

Same issue here.

fixed... will be release today. thanks


No scores of VQA evaluation

I was running python -m train_vqa.py --evaluate to try to reproduce the VQA result, but I only get the message of result file saved to output/VQA/result/vqa_result.json. And it is using default settings datasets = create_dataset('vqa', config)   . Please tell me how to get the final scores. Should I do this by myself?
Thank you!

I found that I should work on VQA challenge

Hi! thank you for your sharing. Could you please tell me how to calculate the score?

Hi! thank you for your sharing. Could you please tell me how to calculate the score?

You could visit VQA-challenge. Use the model generate the answer and evaluate on challenge site

By the way, is the evaluation score consistent with their reported results?


All links in FAQ article content open in new tab

All links that are somewhere in the content of an FAQ article must be opened in new tab.

I tried to do this the proper way - https://github.com/markdown-it/markdown-it/blob/master/docs/architecture.md#renderer but I could not get it to work :(

this issue is close by this pr https://github.com/kiwicom/smart-faq/pull/344


Update documentation links on PyPI

The path.py page on PyPI still links to the now-absent documentation on Pythonhosted.org
It would be nice if:

The PyPI page pointed to https://pathpy.readthedocs.io/en/latest instead
The Pythonhosted page had a link directing us to the new documentation location.

Yeah, this is the hassle I'm forced to do for every project. It's not possible to delete the link. My only option is to hand-assemble some documentation as a zip file and upload it to PyPI. That's what I did to add the now-absent documentation.
So (1) isn't possible, and (2) involves me crafting the docs in a zip file. If you want to do (2), I'll upload it. Otherwise, there's nothing to be done here except encourage others to stop using PyPI and use https://pypi.org/ instead.

On my projects there is a button on pypi that says "If you would like to DESTROY any existing documentation hosted at http://pythonhosted.org/ Use this button, There is no undo." Would that not delete the link?
The README has the RTD link now, so next release that will get updated in the pypi text.

That's a new feature. Pretty sure it wasn't there last I checked. Thanks! Documentation destroyed.

And 9.0.1 pending release.


Adding send_mail feature [WIP].

Implementation for #1272.

@dstufft  This is the basic implementation of email feature, Could you please verify and let me know your thoughts?

@dstufft : Could you please review and let me know your thoughts?

Hey!
Sorry for the delay in reviewing this. I've gone through and reviewed it and made some inline comments. Overall this looks like it's going in a good direction.

@dstufft : I think this is complete now, could you please review and let me know if anything else needs to changed?

✨


Complete queries in kolibri provider middleware

Josh has done research on querying kolibri's database as described in the notion document.
This document outlines most of the relevant tables needed for these queries.
A sqlite database has been provided with some basic test data, which can be used to explore their schema, test and develop the two remaining queries:
func (ks *KolibriService) ImportMilestones(coursePair map[string]interface{}, mapping []map[string]interface{}, db *gorm.DB, lastRun time.Time) error {

a coursePair is a map with two k/v's: course_id and external_course_id
the mapping slice is an array of user_id + external_user_id
These can be used to access the relevant ID's of the courses+users the data is being fetched for.
Milestones are points of achievement for a student, and are represented by the following types:
	Enrollment           MilestoneType = "enrollment"
	QuizSubmission       MilestoneType = "quiz_submission"
	AssignmentSubmission MilestoneType = "assignment_submission"
	GradeReceived        MilestoneType = "grade_received"
	DiscussionPost       MilestoneType = "discussion_post"

And Activity is any user quantifiable interaction with a provider.
	CourseInteraction  ActivityType = "interaction" // watching video, clicking link, etc
	ContentInteraction ActivityType = "completion"  // completing submodule, video, etc

and
func (ks *KolibriService) ImportActivityForCourse(courseIdPair map[string]interface{}, db *gorm.DB) error {

Found an ID mismatch issue.  When a kolibri user is being created via the POST webservice call to kolibri, it returns a hash id w/out hyphens...whereas the source hash id is hyphenated.  So for example a user in provider_user_mappings=10aab22d07b583d5dfd3ab3a6d636686 and the associated one in kolibriauth_facilityuser=10aab22d-07b5-83d5-dfd3-ab3a6d636686.  This is causing a mismatch issue within the middleware.


fix(template): add CONTROL_PLANE_ENDPOINT_IP to ignoredNodeIPs list in CCM

What this PR does / why we need it:
By adding CONTROL_PLANE_ENDPOINT_IP to the ignoredNodeIPs in the CCM config, we ensure
CCM will not assign the CONTROL_PLANE_ENDPOINT_IP as the node IP for any node. This could
previously happen as both CONTROL_PLANE_ENDPOINT_IP and node IP on an external DHCP network
would be learned IPs. In case of a slow DHCP assignment, the CONTROL_PLANE_ENDPOINT_IP
could be assigned as the first learned IP on the VM leading to potential issues when the CCM would then
assign the first IP as the node IP even though it was a floating IP.
How Has This Been Tested?:
manual deployment of modified ressources
Release note:
- Set CCM to ignore Control Plane IP

@tuxtof we're bumping up CCM in https://github.com/nutanix-cloud-native/cluster-api-provider-nutanix/pull/479 already. Can you approve that one, rebase this and keep the scope limited to adding control plane endpoint IP to the ignore IPs list?

We also need to update the template in test/e2e/data/infrastructure-nutanix/ccm-update.yaml


Some questions about dataset

Under the folder of data, there are only two folders of images and classes. Where can I get the labels of labels?

You still need to download the data from the VOC project website or did you mean something else?

Am 15.05.2018 um 09:36 schrieb fantasyzl<EMAIL_ADDRESS>Under the folder of data, there are only two folders of images and classes. Where can I get the labels of labels?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I downloaded voctrainval_11-may-2012, which has five folders: Annotations, ImageSets, JPEGImages, SegmentationClass, and SegmentationClassObject.Can you tell me which label is labels?I'm a rookie, please.

OK，thank you


⚠ Secret Website 3 has degraded performance

In 520cd69, Secret Website 3 ($SECRET_WEBSITE_3) experienced degraded performance:

HTTP code: 200
Response time: 8177 ms

Resolved: Secret Website 3 performance has improved in 9b1d9ae.


Error in OrderItem.php when using Digital Downloads

When i'm using the digital downloads feature, i receive the following error during the checkout process:
Type error: Argument 1 passed to Concrete\Core\Permission\Access\Entity\UserEntity::getOrCreate() must be an instance of Concrete\Core\Support\Facade\UserInfoFactory, instance of Concrete\Core\User\UserInfo given, called in <path_to_my_c5_instance>/packages/community_store/src/CommunityStore/Order/OrderItem.php on line 280
I'm using C5 v<IP_ADDRESS>
Any ideas how to fix this ?

Digital downloads is something I haven't tested much, I've had it listed as something that is likely to have a few issues. I think it may need to be reworked a bit, I'll try to look at this myself soon.

I believe I've fixed this with:
https://github.com/concrete5-community-store/community_store/commits/master
I basically removed code that was manipulating permissions on the files - I found it a bit too risky.
I also fixed a couple of other bugs along the way.
Are you able to confirm this is fixed for you?

Yes, it looks good to me. Thx!


Parameterized the edge bundling code

One of the great things about Datashader in general is that there are not normally any "magic" parameters, i.e., arbitrary values that need to be set just so an algorithm can run, with no clear constraints that determine those values.  For better or worse, the new edge bundling code contains multiple such arbitrary values, and it's now crucial to make sure that the values are declared clearly, documented, interpreted consistently, and easily modifiable by the user.  Moreover, in part because of these arbitrary choices, users are likely to need multiple such edge bundling algorithms, each with tradeoffs, and so it's important to be able to bundle up each algorithm into a clearly maintainable chunk.
Accordingly, I have packaged up the two edge bundling algorithms currently provided into classes that can be called just as the old functions were, except that some positional arguments have been made keyword-only arguments for consistency.
Making this change would not normally affect the actual results, but it does in this case, because two of the parameters were not previously being respected consistently: MIN_SEG and MAX_SEG.  These parameters were defined as module constants and then passed around divided by 2 in some cases, and defaulting to the module-level value in others.  They should now all be respected consistently, with declared values that can safely be overridden in subclasses if needed.

Note that I didn't actually move much of the code into the new classes.  Presumably most of it could be made into class methods, but I'm not sure how all that works with Numba, so I left it alone for now.  Once we have multiple such classes sharing code we'll probably need to sort it out more cleanly.

The clipped plot is presumably because the range is 0,1 and the circular one is placed around zero; I didn't investigate further as I was already planning to update that for other reasons.


[Bug]: When distributing multi-database changes, there are no restrictions on different types of databases.

ODC version
ODC 4.3.2
OB version
No related.
What happened?
I can select obmysql and Oracle data sources to distribute multi-database change tasks simultaneously.

What did you expect to happen?
You can only select one type of database to distribute multi-database change tasks.
How can we reproduce it (as minimally and precisely as possible)?
Follow what happened.
Anything else we need to know?
No response

pass


Need for async version of NestedDiagnosticsContext => NestedDiagnosticsLogicalContext

Need for async variant of NestedDiagnosticsContext for  https://github.com/aspnet/Logging/issues/156
Name: NestedDiagnosticsLogicalContext
Analogues to MappedDiagnosticsContext and MappedDiagnosticsLogicalContext

fixed

MDLC is now working, but still missing https://github.com/NLog/NLog/pull/2110

Resolved with #2110 ?

yes your right! thanks!


Ghost haven't support on NodeJS 5.x yet

I got an installation error
Ghost needs Node version ~0.10.0 || ~0.12.0 || ~4.2.0 you are using version 5.1.0

The error message would have said this:
ERROR: Unsupported version of Node
Ghost needs Node version ~0.10.0 || ~0.12.0 || ~4.2.0 you are using version 5.1.0
Please see http://support.ghost.org/supported-node-versions/ for more information

Please read the linked document http://support.ghost.org/supported-node-versions/

Wow. And we're supposed to "Try Ghost" ?


⚠ Bofrost.be has degraded performance

In 48e3f3b, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1150 ms

Resolved: Bofrost.be performance has improved in e380715 after 20 minutes.


CAMEL-18261 - Add connection parameters to MongoDB component.

Description
Issue - https://issues.apache.org/jira/browse/CAMEL-18261
Added more options than in the story, list of options - https://www.mongodb.com/docs/drivers/java/sync/current/fundamentals/connection/connection-options/
Target

[ ] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[ ] I checked that each commit in the pull request has a meaningful subject line and body.

[ ] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

/component-test camel-mongodb

@VaishnaviR18 component-test has been removed since it is no more needed as the affected components are tested automatically


Merging DLRM integration with MLPerf loadgen

@googlebot I fixed it.

Hi folks,
Any chance you can fix the commit authorship?
Here are the bad addresses:
<EMAIL_ADDRESS>need_author_cla
mnaumovfb<EMAIL_ADDRESS>need_author_cla
It likes the following addresses:
dkorchevgithub<EMAIL_ADDRESS>ok
mnaumovfb<EMAIL_ADDRESS>ok
Best,
Peter

@googlebot I fixed it.

@googlebot I consent

I recommend squashing the PR on merge to avoid the messy history of multiple commits.

@googlebot I consent

@googlebot I consent


Wandb logging cannot be disabled

Describe the bug
It is not possible to disable logging through wandb and tensorflow. While tensorflow is silent, wandb requires interaction, so it is impossible to automate the script due to this problem. The only workaround I found is using WANDB_DISABLED=true when running the script.
Expected behavior
Since llm-compressor uses HfArgumentParser under hood, one would expect that passing report_to="none" to the oneshot(...) call would disable all logging. However, the problem is there, and when I run the script (more in the "To Reproduce" section), I get the following prompt from the console:
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice:

Environment

Ubuntu 22.04.3
Python 3.10
List of all packages:

Package                           Version
--------------------------------- -------------
accelerate                        1.2.0
annotated-types                   0.7.0
clearml                           1.16.5
cloudpickle                       3.1.0
compressed-tensors                0.8.0
datasets                          3.1.0
diskcache                         5.6.3
fastapi                           0.115.6
gguf                              0.10.0
h11                               0.14.0
httpcore                          1.0.7
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.26.5
importlib_metadata                8.5.0
interegular                       0.3.3
jiter                             0.8.2
lark                              1.2.2
llmcompressor                     0.3.0
lm-format-enforcer                0.10.9
mistral_common                    1.5.1
msgspec                           0.18.6
numpy                             1.26.4
nvidia-cublas-cu12                <IP_ADDRESS>
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 <IP_ADDRESS>
nvidia-cufft-cu12                 <IP_ADDRESS>
nvidia-curand-cu12                <IP_ADDRESS>
nvidia-cusolver-cu12              <IP_ADDRESS>
nvidia-cusparse-cu12              <IP_ADDRESS>
nvidia-ml-py                      12.560.30
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
openai                            1.57.3
opencv-python-headless            <IP_ADDRESS>
outlines                          0.0.46
partial-json-parser               <IP_ADDRESS>.post4
pillow                            10.4.0
prometheus-fastapi-instrumentator 7.0.0
py-cpuinfo                        9.0.0
pyairports                        2.1.1
pycountry                         24.6.1
pydantic                          2.10.3
pydantic_core                     2.27.1
pynvml                            11.5.3
python-dotenv                     1.0.1
ray                               2.40.0
requests                          2.32.3
safetensors                       0.4.5
sentencepiece                     0.2.0
starlette                         0.41.3
sympy                             1.13.1
tiktoken                          0.7.0
tokenizers                        0.20.3
torch                             2.5.1
torchvision                       0.20.1
tqdm                              4.67.0
tqdm-multiprocess                 0.0.11
transformers                      4.46.3
triton                            3.1.0
typing_extensions                 4.12.2
uvicorn                           0.32.1
uvloop                            0.21.0
vllm                              0.6.4.post1
watchfiles                        1.0.3
websockets                        14.1
xformers                          0.0.28.post3
zipp                              3.21.0

To Reproduce
To reproduce the issue, use the following script, which is a modified version of the one in the README.md
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
from llmcompressor.transformers import oneshot
from transformers import AutoModelForCausalLM

recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(scheme="W8A8", targets="Linear", ignore=["lm_head"]),
]

oneshot(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    dataset="open_platypus",
    recipe=recipe,
    output_dir="TinyLlama-1.1B-Chat-v1.0-INT8",
    max_seq_length=2048,
    num_calibration_samples=512,
    report_to="none"
)

maybe you can use os.environ['WANDB_MODE'] = 'offline' to avoid interaction for temporary.


PR: Change the current cell color of the Spyder Dark theme

I was never satisfied with the current color because it doesn't have enough contrast with respect to the background one. However, I think this new color has.
Before

After

@goanpeca, what do you think?

Looks good @ccordoba12. Question, there color are taken from some sort of palette generator online or are more tweaked by hand. The palette generator might help keeping them in sync with the overall them

Question, there color are taken from some sort of palette generator online or are more tweaked by hand

Spyder dark is very similar to "One Dark Pro" (the Atom default theme), but adapted to look good with QDarkstyle.
I took this color in particular from
https://github.com/material-theme/vsc-material-theme
which has some variants similar to "One Dark Pro".


How to extract_feature with XLMR using multiple GPU

What is your question?
How to extract_feature with XLMR using multiple GPU? If I use the following code with DataParallel() I can only utilize a single GPU..
Code
xlmr = torch.hub.load('pytorch/fairseq', 'xlmr.large')
xlmr.eval()
xlmr = xlmr.cuda()
xlmr = nn.DataParallel(xlmr)
with open("XLMR_embedding.txt", "w") as out_f:
for data in tqdm(loader):
for key in data:
data[key] = data[key].cuda()
    last_layer_embedding = xlmr.module.extract_features(data["encoding"])

CC @ngoyal2707 , is this supported?

This is not really supported. The easiest way would be to use python multiprocessing and load a separate model instance on each GPU.
The reason nn.DataParallel won’t work is that (1) you’d need to feed a batch of inputs, (2) nn.DataParallel only works if you call forward, but here you’re calling extract_features. You can see how it’s implemented here: https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py#L141


Expiration time for tokens

Perhaps it's sensible to expire app tokens after they haven't been used for a month?

Done with https://github.com/untitaker/mysteryshack/commit/f337a0d7284e344abc0fd3d1831e8ce80fb1f713


Have it all

May you have auspiciousness and causes of success
May you have the confidence to always do your best
May you take no effort in your being generous
Sharing what you can, nothing more nothing less
May you know the meaning of the word happiness
May you always lead from the beating in your chest
May you be treated like an esteemed guest
May you get to rest, may you catch your breath
And may the best of your todays be the worst of your tomorrows
And may the road less paved be the road that you follow
Well here's to the hearts that you're gonna break
Here's to the lives that you're gonna change
Here's to the infinite possible ways to love you
I want you to have it
Here's to the good times we're gonna have
You don't need money, you got a free pass
Here's to the fact that I'll be sad without you
I want you to have it all
Oh!
I feel particularly sad, that I am moving on and moving away from a guy called Ted. He is so worried that I am harassing him too much.
I am secretly embedding some best wishes in Ted’s GitHub repository. Lmao.


WIP: Support external stacks and local function calls using local memory

Add support for invoking the interpreter and JIT'd code with an external stack. This feature is generally useful and will also make it easier to fuzz the runtime and check for correctness.
Add support for local functions that use local memory. Prior to this commit, a local function could be called but could not use any local memory (without overwriting memory from another function).

coverage: 81.454%. first build
when pulling 9a426634f345667e5fae0a2388c31409e65823aa on hawkinsw:local_call_external_stack
into 3fb3da0ffef98be661bebae05043af0f14cfbed4 on iovisor:main.

Can you please format the code using script/format-code? It looks like there are formatting changes that are unexpected.

Can you please format the code using script/format-code? It looks like there are formatting changes that are unexpected.

Of course!! I thought that I did. Sorry!!

Can you please format the code using script/format-code? It looks like there are formatting changes that are unexpected.

Of course!! I thought that I did. Sorry!!

I believe that I ran the formatter on the code (and did it correctly!).

Can you please format the code using script/format-code? It looks like there are formatting changes that are unexpected.

Of course!! I thought that I did. Sorry!!

I believe that I ran the formatter on the code (and did it correctly!).

OK, it might just be possible the code wasn't formatted correctly to begin with. Which version of clang-format did you use?

Can you please format the code using script/format-code? It looks like there are formatting changes that are unexpected.

Of course!! I thought that I did. Sorry!!

I believe that I ran the formatter on the code (and did it correctly!).

OK, it might just be possible the code wasn't formatted correctly to begin with. Which version of clang-format did you use?

I will check!! I am constantly amazed how version-specific the formatter is! This problem occurs with another project I am working on, too.

Can you please format the code using script/format-code? It looks like there are formatting changes that are unexpected.

Of course!! I thought that I did. Sorry!!

I believe that I ran the formatter on the code (and did it correctly!).

OK, it might just be possible the code wasn't formatted correctly to begin with. Which version of clang-format did you use?

I will check!! I am constantly amazed how version-specific the formatter is! This problem occurs with another project I am working on, too.

If you want, we can pick-up a newer version of the formatter if that helps. I think this is using the same one as eBPF-for-Windows, which is using a really old version (due to a dependency on an older version of Clang).
If that is a route we want to pursue, I would suggest making the changes as a separate formatting only PR.


Tutorials - the best and the greatest!

hey @yarikoptic and team! I wanted to see if (after the big neurohack events) this tutorial was still the best and the greatest for getting started with neurosynth:
http://nbviewer.jupyter.org/github/neurosynth/neurosynth/blob/master/examples/neurosynth_demo.ipynb
I'm putting together an interactive lesson with neurosynth + cognitive atlas, and if this one is top of the list, I'll start my version from it. I know there was a lot of cool / new content generated with you and @arokem so I wanted to make sure I started with the creamiest of the corn in this brain crop. Thanks much!

oh, it should be @tyarkoni and @neurosynth Team ;-)  what I remember of a newborn neurosynth is so different from the magnificent beast it is now, that I would be not the one to judge or to guide you among the creamy parts of it.  Looks good to me, I would make it python2/3 compatible though (spotted print statements instead of functions)

@vsoch I don't think there's been any serious work on Neurosynth in a couple of years... the package is basically deprecated at this point (though it still works fine) in favor of NiMARE (though the latter is still a work in progress, and due almost entirely to @tsalo's heroic efforts).

And I'm still tickled by @neurostuffs - I think you came up with that in @chrisfilo office when you were visiting, and it's amazing that it's now a thing :)  #allthestuffs!

And just catching up on things - is this a different API to interact with the (updated?) database --> http://neurosynth.org/api/v2/#!/studies and is there a python (or similar) wrapper for it? Tutorials? Are we rid of the database and features.txt? :P

heyo! Here is a start to what I had in mind --> https://github.com/vsoch/dinosaur-lessons/blob/master/neuro/notebook/cognitive-atlas-intro.ipynb You'll see I go through some human-wordy explanation bits, and then set up a use case that starts in the Cognitive Atlas interface (and I'd like to end up with the equivalent in neurosynth - "here is the web interface way to accomplish this, and the matching functions." What I'm hoping is that I can use these functions (and is there a matching tutorial for them?) to derive the maps, but it may be the case that I still need to install the (old school) NeuroSynth python that is associated with the notebook I linked previously. Please take a look at the tutorial, and special note to the bottom section where I started to go into NeuroSynth (and then realized I didn't know to work with the Python library or just online API).

I am also going to suggest to the user to go to NeuroSynth via NeuroVault as a middle man (decoding) but that's a slightly different use case.  But I made the button for it directly from Cognitive Atlas, so  it must be important!

@vsoch I think neurosynth.org is a good tool for teaching 1st year graduate students. I don't know that the Python package is incrementally useful.
The neurosynth.org API is unrelated to the Python package, it just serves pre-generated data. There's no API to drive the package itself via the web, but I wouldn't recommend doing that (i.e., creating new analyses based on manual study selection) in any case.

Okey doke, video is recorded! And I updated the notebook to generally say "okay, go now use the web interface." If you guys have any other feedback on the notebook it would be appreciated, otherwise I'm good to close the issue. Thanks for the help, and @yarikoptic @tyarkoni if you need any help with said neurostuffs, I hope that you ask me :)

I wanted to quickly weigh in given that I'm the faculty member that pestered @vsoch to create a notebook/lesson that I could use for my undergrad neuro class and my Masters neuro seminar :) The students in both of these courses have a psych background and very little if any programming experience. As such, the web versions of both Cognitive Atlas and Neurosynth are ideal teaching tools for this population of students. It sounds like @tyarkoni is in agreement that the web interface is the right approach for undergrads/new grad students, and I like that @vsoch has also included additional resources for students with Python knowledge who are interested in digging into things further. Thanks, all!

+1!
And for those interested in the future, the technologies I'm using for editing and recording (respectively) are:

kdenlive open source (freeee) video editing software, hasn't screwed anything up majorly yet (when I was younger Adobe Premiere always used to crash)
obs equally free and open source, and really cool for setting up different configurations of "what am I recording on my screen"

The rest of my technology is (unfortunately? lol) paper, markers, tiny toy dinosaurs that I got in bath bombs over the years, and a shoebox with my phone attached to it.  I'm going to adapt it for a Neural Data Science course I'm teaching to undergrads/grads. :)

Awesome! If you want to see the final video, https://youtu.be/0zaFJb0tvtg and I think the links to the repo should be below that.

shot out to OBS! It is really awesome.  I used it a few years back for a few video recordings, had a green screen, and could present "live" while switching between different views of myself (full screen vs in the corner), slides (active, so I could select and point) vs (live) terminal, etc.  You can establish shortcuts to do such quick switches between different scenes. In this abomination: https://www.youtube.com/watch?v=59CE6iOr45k&t=2198s I have switched between me "live" to the previously recorded video (thus audio a bit screwy, and also it was zoom recording), and then back (overlayed on top of desired backgrounds).  OBS is really really powerful and featureful.  Definitely worth learning and using!.


Add information for .net core 2.1 and .net framework 4.7.2

Hi,
Could you please include information for the version of .net standards supported for .net core 2.1 and .net framework 4.7.2?
Thank you!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9eb8b774-c6f0-8235-6964-15219baf2ea7
Version Independent ID: 6819a4a7-7527-d8c7-e132-767f7344e00e
Content: .NET Standard
Content Source: docs/standard/net-standard.md
Product: dotnet
GitHub Login: @mairaw
Microsoft Alias: mairaw

I saw a closed issue and I understand that the table The lists the minimum platform versions that support each .NET Standard version. Thus .net standards 2.0 should be supported by .net core 2.1 and .net framework 4.7.2. From this table, should I understand that .net standards 2.1 has not a minimum platform version to support yet?

Thanks for your feedback, @nicoleta-scrimint. .NET Standard has not yet been finalized and has not been released, so it is not documented; the latest version of .NET Standard is still 2.0.
Since the table lists the minimum supported versions of .NET Core and .NET Framework, .NET Standard 2.0 is implemented by .NET Core 2.0 and all later versions (i.e., .NET Core 2.1, .NET Core 2.2, and the preview version of .NET Core 3.0; similarly, .NET Core 2.0 is implemented by .NET Framework 4.6.1 and later versions (4.6.2, 4.7, 4.7.1, and 4.7.2).
Since I think that this answers your question, I'll close this issue. Feel free to reopen it if you disagree.


Rewrite semantic analysis in Eclair itself

This is an important step towards bootstrapping! We can now generate Souffle code from Eclair, and create a "stage0" compiler that way. Then, we can use the stage0 compiler to compile the Eclair code directly (resulting in "stage1", or the final compiler). The stage1 compiler does not depend on Souffle!
Some constructs were rewritten in an equivalent form for now (because Eclair doesn't support some syntactic sugar yet), but they can be reintroduced later. It shouldn't be any difference to the compiler though, because the desugared form is what Souffle compiles it to anyway.

TODO:

[ ] generate souffle file in CI before running all the tests
[ ] fix semantic analysis check for rules with negation
[ ] add makefile targets for stage0 and stage1
[ ] test both stage0 and stage1 compiler in CI

#144 is an up to date version of this branch, closing..


chore(ci): tune up CI scripts and feature flags

Coverage remained the same at 62.617% when pulling 8c3ae802546c92bf06642c2920a990d9c4f9ef3f on ci-fix into da8edfb750da20cebb8b3380f9bfa5fdc41e45c5 on master.


1.12.2 Support

The issue is exactly what the title says, theres no 1.12.2 support for a version some servers run

There isn't and there won't ever be.


Feature Request: Add VAST (Video Ad Serving Template) support for AVOD (Advertising Video On Demand)

Is there an existing issue for this?

[X] I have searched the existing issues

Which Mux Elements/Packages does this apply to? Select all that apply
mux-video, mux-video-react, mux-player, mux-player-react
Description
VAST is a widely adopted standard for delivering video advertisements within video content. AVOD platforms, which rely on advertising revenue, could significantly benefit from integrating VAST support into their video players. This enhancement would allow for the seamless delivery of video ads, resulting in a better viewing experience for users and increased monetization opportunities for AVOD providers.
Requested Features:

VAST Parsing: Implement the ability to parse VAST XML documents to retrieve ad information, including ad duration, click-through URLs, and media files.
Seamless Integration: Ensure that VAST ads seamlessly integrate with the video playback experience, including proper ad insertion and ad countdown.
Error Handling: Implement robust error handling for cases where VAST ads cannot be displayed due to network issues or invalid ad responses.
Ad Tracking: Include support for tracking events such as ad impressions, clicks, and quartile tracking as per the VAST specification.
Customization: Allow for customization of VAST ad behavior and appearance to suit the platform's branding and user experience.

References:
IAB VAST Standard
VAST 4.0 Specification
Expected Behavior
Upon implementation, the video player should seamlessly parse, integrate, and display VAST-compliant video ads within AVOD content, ensuring smooth transitions, accurate ad tracking, user interactivity, and error handling. Customization options should be available, and thorough documentation and testing should accompany the feature for optimal functionality.

Closing due to inactivity. Can reopen if conversation picks back up.


'Transfer-Encoding: chunked' support

Should be able to have large file uploads stream via a pipe without knowing the length/size.
See mark's email "ANN: Streaming uploads" to manta-private-beta.

I'll be looking into this as part of my work on triton-mysql

@trentm is there anything from "mark's email" that should be moved into this ticket?

the body of that email (which had been sent to the early beta users of Manta):
This morning we pushed an updated manta storage stack that allows for
"streaming" uploads (meaning HTTP transfer-encoding: chunked).  The
semantics of Manta with regards to PUT requests are now such that if
you send a Content-Length header, then nothing has changed.  If you
send a Transfer-Encoding: chunked header, then you do not need to know
the length of the object ahead of time, as Manta will keep track of it
for you, and record the real size at the end.  One caveat: by default,
the maximum length of an object when using streaming uploads is 5Gb;
this can be overridden with a new header 'max-content-length', which
you can use to "reserve" enough space for your object.

The node-manta repo (https://github.com/joyent/node-manta)  is updated
to support this, so please git pull/reinstall it to leverage this. The
CLI works with it as well, as an example (from the root of
node-manta):

$ cat README.md | ./bin/mput -H 'content-type: text/plain'
/mcavage/stor/README.md

Will work just fine.  Note that "content-type" can't be determined in
this case since node-manta just gets a byte stream, so whereas before
you were able to leverage MIME type lookups, you'll almost assuredly
want to explicitly set the content-type now.


cni: "cilium" not working

When I only add cni: "cilium" the node never gets into Ready state, and when I describe it, I get:
Name:               i1
Roles:              control-plane,etcd,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=rke2
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=i1
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/etcd=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=rke2
Annotations:        etcd.rke2.cattle.io/node-address: <IP_ADDRESS>
                    etcd.rke2.cattle.io/node-name: i1-fa7b4bdb
                    node.alpha.kubernetes.io/ttl: 0
                    rke2.io/encryption-config-hash: start-b04df5a426037812bd55e52ea3f030270f995a4cfb1272e08bef7cc1c7d76ede
                    rke2.io/hostname: i1
                    rke2.io/internal-ip: <IP_ADDRESS>
                    rke2.io/node-args: ["server","--cni","cilium","--debug","true","--disable","rke2-ingress-nginx","--disable-kube-proxy","true"]
                    rke2.io/node-config-hash: WXGTHKWCWQS7EC6SSNOJWHVQAEW5IP4HBBVNQ6TWBGQA7F6LPXBA====
                    rke2.io/node-env: {}
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 12 Oct 2023 12:42:26 +0000
Taints:             node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  i1
  AcquireTime:     <unset>
  RenewTime:       Thu, 12 Oct 2023 12:45:19 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 12 Oct 2023 12:43:37 +0000   Thu, 12 Oct 2023 12:42:25 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 12 Oct 2023 12:43:37 +0000   Thu, 12 Oct 2023 12:42:25 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 12 Oct 2023 12:43:37 +0000   Thu, 12 Oct 2023 12:42:25 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Thu, 12 Oct 2023 12:43:37 +0000   Thu, 12 Oct 2023 12:42:25 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Addresses:
  InternalIP:  <IP_ADDRESS>
  Hostname:    i1
Capacity:
  cpu:                256
  ephemeral-storage:  920812304Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             527975872Ki
  pods:               110
Allocatable:
  cpu:                256
  ephemeral-storage:<PHONE_NUMBER>29
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             527975872Ki
  pods:               110
System Info:
  Machine ID:                 cddee597aea040e5b17106df99052f22
  System UUID:                4f87b200-d08a-11ec-8000-3ceceffd301e
  Boot ID:                    4124092d-7d0b-4209-8d2b-6157e9c524c7
  Kernel Version:             5.15.0-86-generic
  OS Image:                   Ubuntu 22.04.2 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.3-k3s1
  Kubelet Version:            v1.26.9+rke2r1
  Kube-Proxy Version:         v1.26.9+rke2r1
PodCIDR:                      <IP_ADDRESS>/24
PodCIDRs:                     <IP_ADDRESS>/24
ProviderID:                   rke2://i1
Non-terminated Pods:          (7 in total)
  Namespace                   Name                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                               ------------  ----------  ---------------  -------------  ---
  kube-system                 cilium-operator-77f9f5495-wpn6k    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2m25s
  kube-system                 cilium-vk5sm                       100m (0%)     0 (0%)      100Mi (0%)       0 (0%)         2m25s
  kube-system                 cloud-controller-manager-i1        100m (0%)     0 (0%)      128Mi (0%)       0 (0%)         2m44s
  kube-system                 etcd-i1                            200m (0%)     0 (0%)      512Mi (0%)       0 (0%)         2m36s
  kube-system                 kube-apiserver-i1                  250m (0%)     0 (0%)      1Gi (0%)         0 (0%)         2m9s
  kube-system                 kube-controller-manager-i1         200m (0%)     0 (0%)      256Mi (0%)       0 (0%)         2m46s
  kube-system                 kube-scheduler-i1                  100m (0%)     0 (0%)      128Mi (0%)       0 (0%)         2m46s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                950m (0%)    0 (0%)
  memory             2148Mi (0%)  0 (0%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-1Gi      0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
Events:
  Type     Reason                          Age                    From                   Message
  ----     ------                          ----                   ----                   -------
  Normal   Starting                        3m21s                  kubelet                Starting kubelet.
  Warning  InvalidDiskCapacity             3m21s                  kubelet                invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced         3m21s                  kubelet                Updated Node Allocatable limit across pods
  Normal   NodeHasNoDiskPressure           3m20s (x7 over 3m21s)  kubelet                Node i1 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID            3m20s (x7 over 3m21s)  kubelet                Node i1 status is now: NodeHasSufficientPID
  Normal   NodeHasSufficientMemory         3m18s (x8 over 3m21s)  kubelet                Node i1 status is now: NodeHasSufficientMemory
  Normal   Synced                          2m53s                  cloud-node-controller  Node synced successfully
  Normal   NodePasswordValidationComplete  2m51s                  rke2-supervisor        Deferred node password secret validation complete
  Normal   RegisteredNode                  2m43s                  node-controller        Node i1 event: Registered Node i1 in Controller

Notice the line:
KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Which to my understanding indicates that the Cilium CNI plugin is not installed, which I also confirm by listing /etc/cni/net.d/ where there is no sign of Cilium.

Hey I know this is old, but if anyone comes across this again cilium can be deployed like so:
rke2_config:
  cni:
    - cilium

This can be added to your rke2_common/vars/main.yml file. So the item should be a list not a string. Also note this will not disable kube-proxy, to do that, the following is needed:
rke2_config:
  cni:
    - cilium
  disable-kube-proxy: true


point envtest in suite to 1.30

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 68.301%

Totals

Change from base Build<PHONE_NUMBER>7:
0.0%

Covered Lines:
989

Relevant Lines:
1448

💛  - Coveralls


Add functions to detect highlight colours and extract highlighted sentences

Done

e2c colors ./path/to/PDF detect all highlight colours in the given PDF file.
e2c extract ./path/to/PDF [0.5, 0.5, 0.5] extract highlighted sentences and print them in terminal. (#31)
Line-split hyphens between two parts of an English word are removed.
Necessary hyphens are detected and kept unchanged.
All kinds of punctuations are removed.
Add dependency on numpy, textbolb, and pymupdf.
Ignore E203 whitespace before ':' because of the conflict with black.

Problems Left

Tests are not complete.
e2c colors and e2c extract cannot be used in the terminal.

Codecov Report

Merging #35 into master will increase coverage by 1.49%.
The diff coverage is 97.36%.

@@            Coverage Diff             @@
##           master      #35      +/-   ##
==========================================
+ Coverage   94.80%   96.29%   +1.49%
==========================================
  Files           5        7       +2
  Lines          77      189     +112
==========================================
+ Hits           73      182     +109
- Misses          4        7       +3

Impacted Files
Coverage Δ

src/e2c/pdf_tools.py
96.84% <96.84%> (ø)

src/e2c/__init__.py
100.00% <100.00%> (ø)

tests/e2c/test_pdf_tools.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f838a64...45e5e8b. Read the comment docs.

Problem 2 is solved by importing click commands in run_script.py and adding to cli.

Please rebase before submitting.

black keeps formatting the following code
while annot:
    if (
        annot.type[0] == 8  # The annotation is a highlight.
        and _compare_color(annot.colors["stroke"], color)
    ):
        sentences[i] = _extract_annot(annot, words_on_page)
    annot = annot.next
    i += 1

to:
while annot:
    if annot.type[
        0
    ] == 8 and _compare_color(  # The annotation is a highlight.
        annot.colors["stroke"], color
    ):
        sentences[i] = _extract_annot(annot, words_on_page)
    annot = annot.next
    i += 1

Unable to reply this comment directly.
It looks like TextBolb have to be initiated by:

In my computer, following messages are printed. Corpora are stored in Users/edxu96/nltk_data.

But the program without this step is able pass CI tests. I think it is because TextBolb is still able to execute some commands without these corpora.

How to install Textblob for Python without admin right?:
NLTK_DATA=/home/user/nltk_data python -m textblob.download_corpora

How to config nltk data directory from code?


BC.bluetooth.isopen is always false on android

Hello
I'm testing your plugin and it's working good on an iPhone 6s.
But on an android phone Nexus 5, this value "BC.bluetooth.isopen" is always false, so I can't scan for new devices.
I have registered your code as a cordova plugin.
It seems that I never Go into this function:
BC.bluetooth.getEnvironment(function(data){
				if(DEBUG){
					alert(JSON.stringify(data));
				}
				window.APPURL = window.location.href;
				window.DEVICEADDRESS = data.deviceAddress;
				window.API = data.api;
				window.VERSION = data.version;
				window.DEVICETYPE = data.deviceType;

				BC.bluetooth.getBluetoothState(function(arg){
					if(arg.state == "false"){
						BC.bluetooth.isopen = false;
					} else {
					    alert("PLUGIN   BC.bluetooth.isopen = true;");
						BC.bluetooth.isopen = true;
					}
					BC.Tools.FireDocumentEvent("bccoreready");
				},testFunc);
			},function(mes){alert(JSON.stringify(mes));});

I found a little bit more about my problem  :
BC.bluetooth.getBluetoothState(function(arg){ ...}

this function never responds on my android device so I never get into .isopen intialization.

Sorry,it's working now.
I had a cache problem when deploying on android.
Is there something planned for a Windows 10 plugin ?
Thank you

Cool!


docs: add explanation on how to use yandex-gpt embedding and model for self-query retriever

Description
This doc provides a detailed guide on integrating and utilizing YandexGPT embeddings and YandexGPT/ChatYandexGPT model for a self-query retriever.

Thanks for this. At first glance the material here is covered in existing docs:

There are some existing guides on Yandex in the integrations section of the docs. The Yandex provider page links out to the chat model/llm; there's also a page on the embeddings.
The self-query retriever is covered in its own how-to guide.

It's unclear to me that having separate guides for different combinations of chat models / retrievers is warranted.
Closing for now but let me know if I've misunderstood or there is material here that is not covered by existing docs and we can re-open.


Set instance attributes into return value.

Please answer these questions before submitting pull request

Why submit this pull request?
[ ] Bug fix
[X] New feature provided
[ ] Improve performance

@wu-sheng H2 storage implementation need to change.

Coverage decreased (-0.1%) to 14.044% when pulling 5f671c6741a8f02c80d6a2eb830ef0a7d3c87d8b on peng-yongsheng:feature/getServiceInstances into cb0b817ffbfd30a1dd5bd8cfd8d34fadba6e6b1f on apache:master.


Add INSERT IGNORE possibility

Hey,
it would be very nice if there is a possibility to have a InsertIgnore Sql object to do some INSERT IGNORE things.
This is how it can look like:
https://stackoverflow.com/a/29219296/2220237

@dpauli I've created PR #336 for it

can be closed in favor of https://github.com/zendframework/zend-db/pull/336

This repository has been closed and moved to laminas/laminas-db; a new issue has been opened at https://github.com/laminas/laminas-db/issues/29.


We should smartly put together default topologies depending on size of the etmodel scenario

Could we have something that scales with your etmodel scenario? E.g. depending on the number of households? So it adds more nodes, and more 'middenspanningsruimtes'? Also see #523: it is related.

:+1:

Assigning to @antw


unable to fetch reference from an URI - invalid peer certificate: UnknownIssuer

hi
got an issue while validating a schema
loading a schema:
$ boon writer/seqway/request/2-0-6.json
schema path/to/file/2-0-6.json: failed
error loading https://internal.gitlab.repo/schema/meta/2-0-0: https://internal.gitlab.repo/schema/meta/2-0-0: Connection Failed: tls connection init failed: invalid peer certificate: UnknownIssuer

reference in schema:
$ cat writer/seqway/request/2-0-6.json
{
  "$schema": "https://internal.gitlab.repo/schema/meta/2-0-0#",

maybe something like
SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

for defining certificates to use for this
thanks in advance

good idea;
I have recently added -k flag in 18df725, which will skip cert validation similar to curl. You can try that flag for the time being.

installed today with cargo install i dont see the -k support there
am i missing something?

ok, build it from sources and now its a bit differenct, seems the -k does not solves it for me
a version command and a -v option for verbose run would be a help there too
$ ./boon -k writer/seqway/request/2-0-6.json
schema /writer/seqway/request/2-0-6.json : failed
error loading https://internal.gitlab.repo/schema/meta/2-0-0: https://internal.gitlab.repo/schema/meta/2-0-0: Connection Failed: tls connection init failed: received fatal alert: HandshakeFailure

@etele I made few changes to fix this in f7b2c6c
could you please retry -k flag  from updated sources

yea, it looks like working now, ty

will see how user can provide certs in command line

I added --cacert PEM_FILE option to cli in d719292
it is similar option in curl. only PEM encoded file is supported. the file can have multiple certs.
could you please very at your end if this flag works fine

seems working, thanks
$ ./boon --cacert /etc/ssl/certs/ca-certificates.crt /path/to/file/2-0-6.json
schema /path/to/file/2-0-6.json: ok

great; thanks


Test/jax components datasets unit test

What?
Test: Unit tests for Jax datasets components
Why?
Components are currently not tested. Needs to be robust for Jax Mava release
How?
Test TransitionDataset and TrajectoryDataset components from mava/components/jax/building/datasets.py
Extra
Close #522

Thanks, Omayma. I made a small commit as per our discussion to fix the pre-commit and change the var name (as per your comment. One more small change requested