Quarkus integration

@casid I'm creating this thread to track the integration (or example) with Quarkus as there may be more interested people in the topic.
When we're finally able to integrate, I would suggest to possibly contribute the example to this repository so that this can be part of your pipeline + showing other people how to do it.
Here's the repository where I'll be doing the tests (contributions are welcome): https://github.com/renannprado/quarkus-jte-example
It would be very nice if I could get a bit of guidance from you to be able to integrate with Quarkus, but I could also seek for help in the Quarkus community if that's necessary.

@renannprado cool, thanks!
Just a quick guess from the error message you receive. Maybe Quarkus is using its own classloader and we need to pass that to the Java Compiler. There was a similiar issue with Ktor: https://github.com/casid/jte/issues/35

@casid I've tried to create the TemplateEngine like this:
val templateEngine: TemplateEngine = TemplateEngine.create(
        codeResolver,
        Path.of("${workingDir}/target/jte-classes/gg/jte/generated/"),
        ContentType.Html,
        Thread.currentThread().contextClassLoader
)

But no luck. I get the same exception.
Do you have any ideas in mind that I could try? otherwise I can seek for help in the Quarkus community too. However I'm still not sure what are the right questions to ask.

Hmm I just cloned your repository and on Windows I get this error message:
2021-03-01 07:36:29,696 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /test failed, error id: f0922e9a-7cbf-435a-b11e-2957c67ae3a9-1: org.jboss.resteasy.spi.UnhandledException: gg.jte.TemplateException: Failed to compile template, error at
An exception has occurred in the compiler (11.0.2). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.nio.file.InvalidPathException: Illegal char <:> at index 4: file:///C:/Users/casid/.m2/repository/io/quarkus/quarkus-core-deployment/1.11.3.Final/quarkus-core-deployment-1.11.3.Final.jar
	at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
	at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
	at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
	at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)
	at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:229)
	at jdk.compiler/com.sun.tools.javac.file.FSInfo.getJarClassPath(FSInfo.java:112)
	at jdk.compiler/com.sun.tools.javac.file.CacheFSInfo.getJarClassPath(CacheFSInfo.java:93)
	at jdk.compiler/com.sun.tools.javac.file.Locations$SearchPath.addJarClassPath(Locations.java:423)
	at jdk.compiler/com.sun.tools.javac.file.Locations$SearchPath.addFile(Locations.java:413)
	at jdk.compiler/com.sun.tools.javac.file.Locations$SearchPath.addFiles(Locations.java:345)
	at jdk.compiler/com.sun.tools.javac.file.Locations$SearchPath.addFiles(Locations.java:333)
	at jdk.compiler/com.sun.tools.javac.file.Locations$SearchPath.addFiles(Locations.java:339)
	at jdk.compiler/com.sun.tools.javac.file.Locations$ClassPathLocationHandler.computePath(Locations.java:779)
	at jdk.compiler/com.sun.tools.javac.file.Locations$SimpleLocationHandler.setPaths(Locations.java:721)
	at jdk.compiler/com.sun.tools.javac.file.Locations$ClassPathLocationHandler.lazy(Locations.java:791)
	at jdk.compiler/com.sun.tools.javac.file.Locations$ClassPathLocationHandler.getPaths(Locations.java:755)
	at jdk.compiler/com.sun.tools.javac.file.Locations.getLocation(Locations.java:2077)
	at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.getLocation(JavacFileManager.java:942)
	at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.getClassLoader(JavacFileManager.java:678)
	at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.initProcessorLoader(JavacProcessingEnvironment.java:269)
	at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.<init>(JavacProcessingEnvironment.java:237)
	at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.instance(JavacProcessingEnvironment.java:193)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.initProcessAnnotations(JavaCompiler.java:1136)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:922)
	at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:311)
	at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:170)
	at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:75)
	at jdk.compiler/com.sun.tools.javac.api.JavacTool.run(JavacTool.java:208)
	at gg.jte.compiler.ClassFilesCompiler.runCompiler(ClassFilesCompiler.java:40)
	at gg.jte.compiler.ClassFilesCompiler.compile(ClassFilesCompiler.java:31)
	at gg.jte.compiler.TemplateCompiler.precompile(TemplateCompiler.java:90)
	at gg.jte.compiler.TemplateCompiler.load(TemplateCompiler.java:53)
	at gg.jte.TemplateEngine.resolveTemplate(TemplateEngine.java:324)
	at gg.jte.TemplateEngine.render(TemplateEngine.java:171)

This is path comes from the MANIFEST.MF Class-Path entry quarkus puts in code-with-quarkus-dev.jar.

I could reproduce the error you got on my Macbook :-)
I have an idea what's wrong, will post as soon as I found out more.

@casid I could be wrong, but I think the error in Windows is related to the System.getProperty call that is relying on a property that only exists in linux/mac.
That's good news! Thanks for taking a look.

No, the Windows error is a bug in the JDK compiler: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8218268
After some digging, I found this issue on Quarkus: https://github.com/quarkusio/quarkus/issues/3592
I will try two things. First, make the classpath fully configurable (currently it's only configurable for precompile.
Second, see if I jte can parse the classpath itself and to a better job than the JDK ;-)
If the second approach fails, we'd probably replicate some of what this class does for the Quarkus jte integration: https://github.com/quarkusio/quarkus/blob/ef55a323d52af0969c7507289c37ddcf2259da7b/core/deployment/src/main/java/io/quarkus/deployment/dev/ClassLoaderCompiler.java

I did some more digging, and this will be harder than expected!
I guess what we'll need is a quarkus extension, to integrate jte: https://quarkus.io/guides/building-my-first-extension
I will give this a try this week, implementing io.quarkus.deployment.dev.CompilationProvider.

@casid looks promising!

@renannprado I got a first very, very, very rough draft of a Quarkus extension.
But! I got a hacky DevMode test running with hot reload :-D
https://github.com/casid/quarkus-jte-example/blob/master/jte-quarkus/deployment/src/test/java/gg/jte/quarkus/test/JteQuarkusDevModeTest.java
My breakthrough was https://github.com/casid/quarkus-jte-example/blob/master/jte-quarkus/deployment/src/main/java/gg/jte/quarkus/deployment/JteCodeGenProvider.java, it allowed me to access the Quarkus runtime classpath (I'm currently storing it as system property and access it later when creating the hot reload template engine, hacky, hacky XD). If this step is executed during production we'd need to generate all template sources instead. I still don't know how to access the configuration of the jte extension from there. That'd be required to setup the template engine the exact same way for production and development.
Let me know what you think.

@renannprado I created a PR for your example repository: https://github.com/renannprado/quarkus-jte-example/pull/1
Let me know if this works on your end :-)

@casid thanks a lot for taking a look at it. I'll check as soon as possible.
You mentioned dev mode, but will it be able to generate a runnable jar and native images already?
Even if not, it will be good enough to start playing with it :)

@renannprado I think the runnable jar should work. mvn package includes all jte classes in the runner jar. I haven't tried native images yet, would be awesome if those worked as well :-D

@casid both pre-compiled and non-precompiled templates worked for me. The runnable jar also worked!
The only thing that didn't work was the native image. There I got the below exception.
I'll soon merge your PR! Thanks again!
2021-03-06 17:22:48,738 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /test failed, error id: 4a2f5d3b-1963-4bba-9154-463019802f11-1: org.jboss.resteasy.spi.UnhandledException: gg.jte.TemplateNotFoundException: Failed to load example1.jte
	at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
	at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
	at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:519)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
	at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:138)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.access$000(VertxRequestHandler.java:41)
	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
	at java.lang.Thread.run(Thread.java:834)
	at org.jboss.threads.JBossThread.run(JBossThread.java:501)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: gg.jte.TemplateNotFoundException: Failed to load example1.jte
	at gg.jte.runtime.TemplateLoader.load(TemplateLoader.java:30)
	at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
	at gg.jte.TemplateEngine.resolveTemplate(TemplateEngine.java:371)
	at gg.jte.TemplateEngine.render(TemplateEngine.java:229)
	at gg.jte.quarkus.runtime.JteTemplateRenderer.render(JteTemplateRenderer.java:21)
	at gg.jte.quarkus.runtime.JteTemplateRendererPrecompiledProvider_ProducerMethod_templateRenderer_834c801a7c0c73f86d26110e439b6abb3ef46215_ClientProxy.render(JteTemplateRendererPrecompiledProvider_ProducerMethod_templateRenderer_834c801a7c0c73f86d26110e439b6abb3ef46215_ClientProxy.zig:117)
	at org.acme.GreetingResource.hello(GreetingResource.kt:25)
	at java.lang.reflect.Method.invoke(Method.java:566)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
	at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:643)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:507)
	at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:457)
	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:459)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:419)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:393)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:68)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
	... 17 more
Caused by: java.lang.ClassNotFoundException: gg.jte.generated.precompiled.Jteexample1Generated
	at com.oracle.svm.core.hub.ClassForNameSupport.forName(ClassForNameSupport.java:60)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:275)
	at gg.jte.runtime.TemplateLoader.load(TemplateLoader.java:27)
	... 35 more

@renannprado, awesome that it works on your end!
Hmm, I suppose we need to somehow expose those classes to native code?
After a quick search I found this, which looks promising:
https://quarkus.io/guides/writing-native-applications-tips#register-reflection
Looks like we could simply add this as a build step to the extension.

I gave it a try:
https://github.com/casid/quarkus-jte-example/commit/b3882a3aa0b936b4defe1d43479977fe8a2a739a
How are you building the native image? I think I need to download and configure GraalVM first, since maven complains when I run quarkus:native-image.

@renannprado just tried it, it seems it does require GraalVM. I installed it and installed the native image support. Unfortunately I'm on Windows and need MSVC. This brought back dark memories from the days before Java when coding in C/C++ :-D After downloading 3GB just for MSVC Visual Studio told me to restart my PC. Seems some things never change ;-)
Will give it a try tomorrow after my PC restarted naturally...

@edward3h wow that looks pretty straight forward. I'm not sure if comparing with Quarkus helps much with native image generation. My impression so far is that Quarkus does this in a very specific, high-level way and there isn't much to reuse if one doesn't use Quarkus.

@renannprado Native image works on my end after this change: https://github.com/casid/quarkus-jte-example/commit/96edea205677c5007a5106e400c0e2a85ecfe782 👍

@casid I took your code and I put master of jte as a dependency ('cause I didn't have 1.8.1-SNAPSHOT. Same error.
I think I missed something.

@renannprado oh right, I just changed the jte dependency to 1.9.0
Hmm, I tested on Windows, but that shouldn't make a difference.
What I did:

Run mvn clean install on jte-quarkus
Run mvnw package -Pnative on the example project

@casid now with 1.9.0 it worked!
Thanks

@casid are you planning to create a real quarkus-extension out of this quest we've just went through?

@renannprado would be nice to have.. I'm a bit unsure where this should be placed. To me it looks like all official quarkus extensions are part of the quarkus repository. But I'm not sure if quarkus would like to have a jte extension on board. Did you already talk to the quarkus guys about this? Maybe you could open an issue there with a link to our demo project?

@casid no I haven't. But I can do this. It's minimum I can do after you put so much effort in getting it to work.
If for some reason it's not possible to put there, then I believe we should just publish the jar so that we can use it as a dependency

@casid I found the docs where it says how to get to the "official" extensions page of quarkus, but it might demand more commitment than you want to do.
Check it out:
https://github.com/quarkusio/quarkus/wiki/Adding-extensions-to-the-Quarkus-ecosystem#process-to-join-the-platform
I believe the quarkus extensions for jte doesn't necessarily need to be there to be useful, you might as well just publish another jar as part of this project repo (if you want). I do believe it would help to create more visibility to jte by doing that though.

@renannprado yeah, looking at the documentation, I'm afraid maintaining a quarkus extension is out of scope for me. I'm not using Quarkus for any of my projects (and do not plan to do so), so I'd be a very bad fit for maintaining it.
I added the help wanted label, in case some other jte user would like to transform our hacky experiment into a solid quarkus extension :-)

@casid It would be enough to publish a jar of it as is just so that one can use as any other dependency. Doesn't that work for you? The extension doesn't need to be part of the "official" ones until maybe it gets more attention from others and from you.
Just adding it as another module in this repo I suppose. If you don't want to do it as part of this repo, I could do it on my own just to scratch my itch, but we both know it's your work. Can we find a compromise "solution"?

@renannprado sure thing, you can definitely publish that jar.
I just think it shouldn't be part of the jte repository. jte is a general purpose library. It's built in a way that let's you easily plug it into your software and configure it as you're needs see fit. This worked fine so far to integrate in all web frameworks, except Quarkus.
Quarkus is kind of the opposite of the spectrum. It's a very huge framework that enforces its ideology onto your entire project. The quarkus build process seems very complex to me and I used APIs that look like they could potentially break in the future and to be honest I would be a bit afraid to use the current extension in production, since I just don't know if it is doing the right thing for all scenarios quarkus can run. I think what would be needed here is someone very proficient in Quarkus to get the extension in production ready shape. Since Quarkus has its own template engine, I'm not sure how high their motivation would be to do that though :-)
Usually just old-school embedded Jetty & Jax-RS. For something more lightweight, Javalin is also very nice to work with.

@casid yeah... I can totally understand.
I'm also not sure if I'm sold to Quarkus to be honest. In the last years so many frameworks appeared, and now it's really hard decide which one to use. I've used Spring in the last ~4 years and it's ok, but its startup time is annoyingly slow.
Alright. If you agree, I would create a new repo quarkus-jte-extension under my github user and see where it goes.
What do you think it's the best way to start the repository? e.g. do I copy/paste your code and commit into the new repo...? Do you want to do the first commit? The last thing I want is to do something that make it look like I did everything on my own without mentioning you.

@casid I didn't forget about it, it's just been a bit hard to find time to work on this.
So I've started to do something, I've already registered my group ID in Sonatype and so on...
But now I'm trying to create a simple test: https://github.com/renannprado/quarkus-jte-extension/pull/1/files#diff-efb5fd392ef0bac05e7100874533722b3058eef3879769d3c53ffc33807a3676
On top of what you have in the DevMode test... but I simply can't.
It would be nice if I get this to work because users of the lib would be writing similar tests I can imagine...
Anyhow if you have 5min and want to take a look at the PR and want to give some feedback (I've upgraded quarkus and renamed several things, but everything is almost the same).
Once I get this test to work and add a test app to the repo, then I'll build a pipeline to finally release it.

Cool, congratulations on registering with Sonatype!
I'm going to clone your repo next week and give it a shot.

@renannprado I cloned the project and looked into the test. Sorry, I have no idea what the problem is. By now, I almost forgot everything I hacked into the Quarkus extension.

@renannprado I cloned the project and looked into the test. Sorry, I have no idea what the problem is. By now, I almost forgot everything I hacked into the Quarkus extension.

@casid thanks for taking a look. I'll release it without the tests then.

@casid it's finally released:
https://github.com/renannprado/quarkus-jte-extension/

Hey, congratulations! I just added a link to your repository in the jte readme 👍
Thank you for your hard work on the plugin!
I believe we can now close this issue, right?


[FEAT] Simplified, Automated and Consistent Forms

Is your feature request related to a problem? Please describe.
Currently, the Show, Edit, and Create pages/forms need to be implemented manually, which can become cumbersome when dealing with a large number of tables - even if using common layout pages.
Describe alternatives you've considered
No response
Additional context
No response
Describe the thing to improve
It would be great to infer column types from data fetched in databases like Supabase and implement a global renderer to maintain a consistent user experience with minimal code.
For example, a foreign key relationship could be represented as a select or autocomplete field, while an int8 type would correspond to a number input, and so forth.

Hey @aress31 thanks for the issue. I guess we already have such feature. It's called inferencer. You can check it out here: https://refine.dev/docs/ui-integrations/material-ui/components/inferencer/

Closing the issue now. Feel free to re-open it if you see fit.


[enhancement] trigger onLoad automatically when scroll is in bottom without extra hold gesture

The package is awesome,
I just realized that in order to trigger onLoad the user has to do an additional scroll gesture (hold + scroll). I think for the user experience it would be nice if the onLoad could be triggered automatically when the scroll position is already at the bottom (with the condition that the Footer is also shown automatically without any additional gestures). hope there is an option for this in the future
or maybe I missed something for this option? because I've been diving in the code examples but couldn't figure out how

Set the infiniteOffset parameters and set the infiniteHitOver to false is the solution


Rbac: belongs to scope by ids

Part of the rbac scope initiative.
This one is working on the rbac belongs to filters.
Currently it is making many queries to lookup records.
This change allows the outer portion to access the ids by scope.
This way if filtered_ids is a scope, we can avoid querying the database multiple times.

LGTM.


Add support for python 3.12

I have a private repository that uses this action and when we tried to upgrade the python version to 3.12, we use it like this in our yaml file:
      - name: Publish Unit Test Results
        uses: EnricoMi/publish-unit-test-result-action/linux@v2
        if: github.actor != 'dependabot[bot]'
        with:
          event_name: Integration Tests
          check_name: Integration Tests After Approval
          report_individual_runs: true

But after trying to upgrade the python version from 3.10.13 (on which it works well) to 3.12.0, this is the log:
> Install Python dependencies
...
  Collecting cffi==1.15.1 (from -r /home/ubuntu/.../_actions/EnricoMi/publish-unit-test-result-action/v2/linux/../python/requirements.txt (line 12))
    Using cached cffi-1.15.1.tar.gz (508 kB)
...
Building wheels for collected packages: cffi
    Building wheel for cffi (pyproject.toml): started
    Building wheel for cffi (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error

    × Building wheel for cffi (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [36 lines of output]
        running bdist_wheel
        running build
        running build_py
        creating build
        creating build/lib.linux-x86_64-cpython-312
        creating build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/__init__.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/pkgconfig.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/recompiler.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/model.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/api.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/commontypes.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/ffiplatform.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/error.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/lock.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/vengine_gen.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/verifier.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/cparser.py -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/_cffi_include.h -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/parse_c_type.h -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/_embedding.h -> build/lib.linux-x86_64-cpython-312/cffi
        copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-cpython-312/cffi
        running build_ext
        building '_cffi_backend' extension
        creating build/temp.linux-x86_64-cpython-312
        creating build/temp.linux-x86_64-cpython-312/c
        gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/home/ubuntu/.../enricomi-publish-action-venv/include -I/opt/hostedtoolcache/Python/3.12.0/x64/include/python3.12 -c c/_cffi_backend.c -o build/temp.linux-x86_64-cpython-312/c/_cffi_backend.o
        c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
           15 | #include <ffi.h>
              |          ^~~~~~~
        compilation terminated.
        error: command '/usr/bin/gcc' failed with exit code 1
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for cffi
  ERROR: Could not build wheels for cffi, which is required to install pyproject.toml-based projects
  Failed to build cffi
  Error: Process completed with exit code 1.

I guess this error makes sense since this action pins cffi to 1.15.1. According to the official changelog (https://cffi.readthedocs.io/en/latest/whatsnew.html#v1-16-0rc1) support for python 3.12 was added in 1.16.0rc1.

Please test my fix:
uses<EMAIL_ADDRESS>

The fix has been released.


Remove copying of meshconfig to app namespace

What this PR does / why we need it:
This PR removes mounting of configmap in kube-injected pods since that configmap is not used by the agent. This eliminates the need to copy the configmap to application namespaces.
Also moves ingress proxies to istio installation. That requires enabling cluster-wide service registry (since it has to listen to both app and istio namespaces). That should not cause problems on shared cluster (that's how we operated during 0.1.x development). The ingresses are contained by marking each resource with namespace, and setting the controller mode to STRICT.
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #1272, fixes #1273
Special notes for your reviewer:
Release note:

Are you also removing the ability to run ingress per namespace ? What is the relation with configmap mount and copy removal, and why is it necessary ?
Is it the smallest safe change ? It seems a lot of renames and removals - can you instead just suppress the generation of the map with a flag or constant, so we can add it back quickly ? Since this change seems pretty big - it may be tricky to revert when we start adding back the config map.

To answer your first point, configmap mount requires copy or the pod does not start.
Linters do a good job finding unused variables. Once you remove the use, you start pulling the string.

Waiting for ack from @andraxylia and @costinm

/lgtm with a few suggestions.


[Bug]: Incognito mode in chromium with version 1.43

Version
1.43.0
Steps to reproduce
Open a new context like always, it opens a non incognito browser.
https://playwright.dev/java/docs/api/class-browsercontext
// Create a new incognito browser context BrowserContext context = browser.newContext(); // Create a new page inside context. Page page = context.newPage(); page.navigate("https://example.com"); // Dispose context once it is no longer needed. context.close();
Expected behavior
The browser opens in incognito mode
Actual behavior
The browser opens in non incognito mode
Additional context
Just check that in 1.42.0 version works like always
Environment
Operating System: Ubuntu 22.04
Browser: Chromium
Java: OpenJdk 11

Due to a Chromium change, it does not look like Incognito anymore but it is actually an incognito tab - do you experience issues with your page behaving different?

Closing as per above. Feel free to re-file if you still experience issues.


Backport config option skip_physical_aggregate_schema_check #13176 to 42

Which issue does this PR close?

Part of https://github.com/apache/datafusion/issues/13166
Rationale for this change
By backporting this option (disabled) I hope we can upblock other upgrades
What changes are included in this PR?
This PR backports https://github.com/apache/datafusion/pull/13176 to the 42 branch to fix the issue with the physical aggregate schema check.
Note unlike the original PR, the default value is true (the PR defaults to disabling the check)
Are these changes tested?
By CI
Are there any user-facing changes?
A newly added check in 42 is now disabled by default

Thank you @comphead  -- I will move on to preparing the release now


Frame rate drops to 30 fps when rendering Unity UI

Has anybody else experienced frame rate drops down to 30 fps using the Unity UI? Especially when using a Scroll View.

Closing this because it's not a toolkit specific question.
Please use the Microsoft Developers Forum or Unity's forums.

I use all of the forums but I truly think this is the most frequently used forum by experienced HoloLens developers right now. Having said that, I understand where you're coming from. It was worth a try, though. Sorry =)

No worries,  Try the Slack Channel too


Upgrade PythonScriptV0 to Node JS 16

Task name: PythonScriptV0
Description: Task upgraded to NodeJS 16
Documentation changes required: N
Added unit tests: N
Attached related issue: https://github.com/microsoft/build-task-team/issues/3597
Checklist:

[x] Task version was bumped - please check instruction how to do it
[x] Checked that applied changes work as expected

Duplicate


Multi line selection disappears after Ctrl+Arrow combinations

Type: Bug
Hi there,
When I select multi-line by pressing Ctrl+Alt+(Up or Down)Arrow keys to start multi-line editing, if I press Ctrl+(Right or Left)Arrow combination to move word by word on my selected lines, the multi-line selection disappears and my multi-cursor turns into single-cursor on the first line that I selected.
This issue fixes itself sometimes when I restart the VSCode but in the last two days I could not fix this issue after restarting. I hope I was able to explain the issue.

If it doesn't happen in that state I suggest you use the Start Extension Bisect command to try and pinpoint which one is causing it.

Hi John,
I disabled all of the extensions and tried again, it was same, issue still
there. I have run the Start Extension Bisect command but it was same. After
I uninstalled all of the extensions and it was same. I uninstalled VSCode
and it was same. So, I am using Sublime Text temporarily until the issue
will be resolved.
Additional info: I am using Windows 11 ver.22H2 and Huawei Matebook D16
Core i5-1135G7
I tried by using an external keyboard and it was same... Crazy!
Arda BALKAN
John Murray @.***>, 26 Oca 2023 Per, 12:51 tarihinde
şunu yazdı:

Are you able to reproduce the issue after running the command to restart
with all added extensions temporarily disabled?
If it doesn't happen in that state I suggest you use the Start Extension
Bisect command to try and pinpoint which one is causing it.
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode/issues/172500#issuecomment-1404768160,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACJVS46SKED5MSDSRAPX2KTWUJCLHANCNFSM6AAAAAAUHHFL6I
.
You are receiving this because you authored the thread.Message ID:
@.***>

Problem solved after clean uninstallation of VSCode. I uninstalled VSCode, I deleted all folders about VSCode, I deleted all Registry Settings about VSCode and restarted Windows. I installed VSCode again and the issue was gone.

Hi everybody.
Problem repeated. I fixed it permanently, the source of the problem was Accessibility Support -> On in Settings->Text Editor->Accessibilty Support=On I changed this option to "Off" and ta-daaa! Thank you.


addon updates

PR introduces addon updates for

pcidevices-controller
vm-import-controller
seeder

Problem:

Solution:

Related Issue:
Test plan:

@mergifyio backport v1.2


Add custom field validators for build

It's possible to input bad values into certain fields despite valid types. For instance, rotation takes number types but should be rangebound -- terms greater than 1 will mess up the scale of an object.
This seems like we can write some custom validators https://github.com/json-editor/json-editor#custom-validation

Using the standard json schema also works here: https://github.com/arenaxr/arena-web-core/commit/e4ab61c70ad5197178533fe0ee47dfd62d6a89ef


Set Up SAP Fiori Tools in Your Development Environment

Tutorials: https://developers.sap.com/tutorials/fiori-tools-vscode-setup.html
In step 3, it is described to install the extension SAP Fiori tools - Extensions Pack.
However, it is called SAP Fiori tools - Extension Pack (without s).

Thank you.  This has been corrected.


Bug report: Upgrade to 1.10.0 gives incomplete import suggestion to OUIFR

Description
Upgrade report suggests updating sass import to
@import '~office-ui-fabric-react/dist/sass/References.scss'
But this does not import the theme variables
Correct import should be
@import<EMAIL_ADDRESS>Steps to reproduce
Run command to generate upgrade report from 1.7.1 (version tested) to 1.10.0
Expected result
@import<EMAIL_ADDRESS>Actual result
@import '~office-ui-fabric-react/dist/sass/References.scss'
Environment
Windows
Bash
O365 CLI version: 2.7.0

I see on the rules that this was done intentionally, but being @microsoft/sp-office-ui-fabric-core the default source on projects, I wonder if it should not be kept as the default
This doesn't cause a build error but the variables are not imported, so in our case we only noticed it after the solution was deployed

Thanks @joelfmrodrigues. We'll have a look at it and see how to best tackle it 👍

Looking at projects generated using SPFx v1.9.1 and v1.10.0, they use @import '~office-ui-fabric-react/dist/sass/References.scss'; instead of @import '~@microsoft/sp-office-ui-fabric-core/dist/sass/SPFabricCore.scss';. As such, I'd say the upgrade guidance is correct and aligned with the projects generated by SPFx. If the project contents are incorrect, then perhaps we should raise it in sp-dev-docs as a separate issue.

@waldekmastykarz  🤦♂ I should have created a new solution before adding the bug
Thanks and sorry for wasting your time testing it

Nah, don't worry about it. Better check twice than have a bug. Appreciate you reaching out ❤


fix: 🐛 Correct validation for removing asset requirements

Description

Correct the condition to check if given requirement id exists in existing requirements for an Asset

JIRA Link
DA-384
Checklist

[ ] Updated the Readme.md (if required) ?

:tada: This PR is included in version 14.1.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 15.0.0-beta.2 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 15.0.0-alpha.16 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Auto-generate regexp from path

Feature Request
Compose_setup currently has manually written regexp in order to convert an actual path with path parameters (/workflows/5) used by an enduser into a generalized path (/workflows/{workflowId})
Desired behaviour
Figure out a better way to get this generalized path.  This may involve:

not using a regular expression at all using some clever way
using auto-generated regular expression which may require:

creating something that fetches all generalized paths from swagger.yaml
writing a converter to convert it to a regexp and then reverse-sort the order of it
Throwing it into a dictionary file
Hook up to Travis

Resources:
https://www.elastic.co/guide/en/logstash/current/_how_to_write_a_logstash_filter_plugin.html
https://github.com/pillarjs/path-to-regexp
┆Issue is synchronized with this Jira Story
┆Epic: DevOps: CloudFormation
┆Fix Versions: Dockstore 2.X
┆Issue Number: DOCK-335
┆Sprint: Backlog
┆Issue Type: Story

there's a script to generate the regex, but still no automated way of running it. this is for ES logstash btw


SampleSet.resolve() method or similar

It would be useful to have a method that can be used to ensure that a sample set is resolved. This would be useful for #474

Thanks!


Don't hardcode GCC in build-logger Makefile

There may not be a gcc program, and CC may be pointing
at the right one, but this would completely override it,
failing the build.

Thank you!


Bump upjet to v1.1.1

Description of your changes

Fixes #
I have:

[ ] Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

/test-examples="examples/ec2/vpc.yaml"

/test-examples="examples/networkfirewall/firewall.yaml"

/test-examples="examples/ec2/vpc.yaml"


NetBSD with Pkgin Provider initial support

Pkgin package provider is initially written. Right now it will handle only installing packages. Will implement bootstrap as I find a best strategy for doing this via code.

I had to merge this manually due to a conlict with Cargo.toml, but authorship was maintained.
Thank you, @martintc

Merged 9c5b2578121177f5c7b9f0f8a80e7226b2987964


Adding the option to trim the string with others characters, and define which trim function to use

Adding the option to trim the string with others characters, not only with the default characters provided by php. Also added the option to define the function which will be used to trim the string, if it's normal trim, left trim or right trim

Thanks for PR! Would you mind adding some tests for this functionality?

:+1:

Added the tests and reforced the method to accept only trim, rtrim and ltrim.

Hey Dude, why didn't you accepted my pull request? I wrote the tests as you asked and wrote some more code to ensure the params sent to the function. Ok so, sorry for anything.

@kbca You added them 3 days ago, and I just got back from a trip this morning. Haven't gotten around to reviewing it since I've been at work all day. I appreciate your contribution, but please cut me some slack.

OK, sorry man. It was my mistake. I was viewing my fork and I saw there were modifications, 2 more commits, actually the commits I myself did. So I thought I was viewing your project, instead of my fork, but I just should have seen you didn't accepted nor rejected my pr. Anyway, sorry again and feel free to accept or reject wherever you are able to.

No worries - I hope I didn't offend you. And I really do appreciate your work on this. Just haven't had as much time for Stringy.
I squashed your commits and merged into master here: https://github.com/danielstjules/Stringy/commit/1a2e771ff8a9cb8cc240abbce7a63dd8613785fa (one of the commits seemed to be from a diff author)
I'm tempted to split this function out into trim, trimLeft and trimRight, as I think it might be nicer than using constants in this case.
Thanks again!


CARP VIP should not require existing IP

It seems CARP no longer requires an existing IP on the interface it is assigned.   This requirement should be removed from OPNsense.
Besides pfsense has done this since their version 2.2.

Why not choose some private unused IP in the main interface?

Because I do not use NAT, and there appears to be no other way to make outgoing connections use the CARP IP instead of an unneeded preconfigured IP.   This also causes split routes that break connections with some systems.

From: Michael<EMAIL_ADDRESS>Sent: Friday, April 13, 2018 12:13 PM
To: opnsense/core
Cc: Schweiss, Chip; Author
Subject: Re: [opnsense/core] CARP VIP should not require existing IP (#2358)
Why not choose some private unused IP in the main interface?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/opnsense/core/issues/2358#issuecomment-381202147, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHa72zCYLpEoojzoESRS_oX_6L5b8hRMks5toNypgaJpZM4TTnIc.

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

Oh you mean like local squid binding to a specific IP for outgoing connections?

As far as I know a rule of carp is to never use the carp ip for creating connections, only for answering, which would mean you need a regular IP as well. Do you have a reference when and where this restriction was lifted in FreeBSD?

On 14. Apr 2018, at 06:25, Michael<EMAIL_ADDRESS>wrote:
Oh you mean like local squid binding to a specific IP for outgoing connections?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I don't think it's a problem to remove the constraint, although I can't find a reference for the intended behaviour in FreeBSD on this topic (it seems to work, the doc's don't seem to suggest otherwise)
I do have the changes needed stashed on my machine, if we want this, I can commit them.

... on the plus side, this dumps two other pointless file_notice() executions as well

It would also make HA setup possible where WAN is only /30
Ad Schellevis<EMAIL_ADDRESS>schrieb am Sa., 14. Apr. 2018,
14:28:

... on the plus side, this dumps two other pointless file_notice()
executions as well
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/opnsense/core/issues/2358#issuecomment-381325744, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ATfeoTb4Mb0bsI15n8TcWgXc426n-es-ks5toetdgaJpZM4TTnIc
.

@fichtner are you ok with ditching the validation too?

fine by me, but will not look at kernel bugs related to this. it sounds like a protocol violation unless someone shares the details and related FreeBSD code changes.


Transaction form has a bug in the date field

This bug was reported to me and still needs to be reproduced.
Screenshot was taken approximately on September 19, 2022 ~ 8pm Pacific on Microsoft Edge (probably ~v105.xxx):

Month names are displayed as Invalid date
The day number field should have been 19 instead of 18

Was not able to reproduce this issue, checked on:

Canutin v1.1.0
Microsoft Edge 105.0.1343.42 (Official build) (64-bit)
Windows 10 Version 21H2 (Build 19044.1889)

This is an issue in Firefox (not Edge)


Adding example notebook with binder link

Here is the rendered notebook example.
I've been saying that reading HDF5  is not significantly different than reading zarr, but it's actually about 20% slower for this example.   With our original test, the times were not significantly different.   @martindurant, Would you expect our new mapping approach to be slower?

@martindurant , any thoughts?
@kwilcox, give this a try!

I'll merge this, so that the binder link appears and it is easier to play with.

Yep, I think it's a nice demonstration.
I ran it with no problems now: 39s for HDF and 30s for zarr. It may be that the second one is faster just because it is second (workers need to import pandas, etc). When I ran it again, HDF came in at 28s.
Is it worth showing the contents of the catalogue or mentioning Intake here? I suppose the audience already knows, but it's nice to show that this works either way transparently.

(note that it comes up as "classic" notebook for me, which is totally fine, but you could have a jlab interface with the dask panels populated for a bit more splash)

Is it worth showing the contents of the catalogue or mentioning Intake here? I suppose the audience already knows, but it's nice to show that this works either way transparently.

I had to spend some time digging into intake_xarray.xzarr.ZarrSource and the parameters being passed as part of the catalog config to figure out what was actually happening, I wouldn't mind the explanation!

Here are my latest results, running the 2 cases twice to make sure the cluster was spun up.  So @martindurant I guess you might be right.  These are my fastest times yet:

These are my fastest times yet!

Except you can get even better by using partitions containing multiple zarr chunks!

And I can do that in the catalog, right?   Can you provide (refresh) my memory?

Yes, you add a chunks key to the zarr args, like {coord: size, ...}, but I'm not sure what the right set of chunks would be.

I think you deleted your branch, and the binder link pointed to it, not to the main repo.
My AWS credentials have disappeared, so I can't access it from here, but it would be something like
  ike-hdf5:
    driver: intake_xarray.xzarr.ZarrSource
    description: 'Hurricane Ike simulation in NetCDF4/HDF5 format'
    args:
      storage_options:
        ref_storage_args:
          requester_pays: true
        references: "s3://pangeo-data-uswest2/esip/adcirc/adcirc_01d_offsets.json"
        target_options:
          requester_pays: true
        target_protocol: s3
      urlpath: "reference://"
      chunks:
        time: 30

where the number 30 is some multiple of the default chunk size that you get.


Documentation how to use routing engines

We should document how to use a routing backend to prevent issues like #88. I’ll actually have to educate myself there a bit too, so I can take that.

The code where i define the profile is the next:
problem_instance.add_vehicle([vroom.Vehicle(vehicle['id'], start=[start_lat, start_lng], end=[end_lat, end_lng], profile="car")])
I am not passing any custom matrix because I am using a JSON file that contains the addresses of all the points, with their respective longitudes and latitudes.


Allow arrow keys to move the slider

On prod each tap of an arrow key moves the slider by 1 percentage point. @SylvainChevalier do you feel like this is worth implementing?
Requested by probahilliby

low priority.


276 docs explain tp page

Description

Added TP explanation page and improved homepage. Fixes #276.
Motivation and Context

Add a page defining transport performance
Introduce home page improvements, especially package description
Add jupyter: python3 to _quarto.yml as a universal python kernel selection for docs previewing/rendering.

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

All added content added modified renders as expected locally.
Test configuration details:

OS: macOS
Python version: 3.9.13
Java version: N/A
Python management system: conda

Advice for reviewer

A separate ticket has been opened to capture the overall workflow to go on the explanation 2. page - will work on this next.
Checklist:

[X] My code follows the intended structure of this project
[X] I have performed a self-review of my own code
[X] I have commented my code, particularly in hard-to-understand areas
[X] I have made corresponding changes to the documentation
[X] My changes generate no new warnings
[X] I have added tests that prove my fix is effective or that my feature works (N/A)
[X] New and existing unit tests pass locally with my changes
[X] Any dependent changes have been merged and published in downstream modules

Additional comments

None

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.13%. Comparing base (15434fc) to head (529ac42).

Additional details and impacted files
@@           Coverage Diff           @@
##              dev     #291   +/-   ##
=======================================
  Coverage   98.13%   98.13%
=======================================
  Files          21       21
  Lines        1927     1927
=======================================
  Hits         1891     1891
  Misses         36       36

Flag
Coverage Δ

unittests
98.13% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


replace hard coded tmaster ports in tmaster_unittest

Prerequisite: #1852
Fix: #1838

Start 3 servers at the beginning of TMaster startup
Set port 0 in tmaster_unittest

are we good on this? @kramasamy

This PR has been refactored. Ping @srkukarni @nlu90 @kramasamy for review


Partial outputs

Hey there!
I have a network where only one of the outputs is known for any training example.  How can I prevent the other outputs from propagating an error?  Is there some sentinel value I can set them to, or some way to customize the loss function to ignore all but one output node?
Thanks!

Actually I think I have the objective function that does this (assuming the undefined outputs are set to 0):
def non_zero_mean_squared_error(y_true, y_pred):
    return theano.tensor.mean(theano.tensor.square(theano.tensor.sgn(y_true)*(y_pred - y_true)), axis=-1)

Does this look reasonable?  The idea is that only one of the true label outputs is non-zero (and always positive) so the sgn function will allow the non-zero outputs to contribute to the error and the rest have no effect.

Why not just assign some arbitrarily unlucky value to confirm the 0? Such as let's say 65535 or something.

I don't understand your comment?  The issue is that the training examples do not have a value for all of the output nodes. If I put a sentinel value (even 0 or 65535) and use a normal loss function, it will train the network to learn these sentinel values. What I need is a loss function that purposely ignores some of the outputs. I believe the one I wrote above will do this.

Actually the function I wrote doesn't work, because the "mean" function will consider the 0's in the list when dividing.

This function should work correctly:
def non_zero_mean_squared_error(y_true, y_pred):
    total = theano.tensor.sum(theano.tensor.square(theano.tensor.sgn(y_true)*(y_pred - y_true)), axis=-1)
    count = theano.tensor.sum(theano.tensor.sgn(y_true))
    return total/count

I was thinking more like:
theano.tensor.mean(theano.tensor.square((y_true==sentinel_value)*(y_pred - y_true)), axis=-1)
That way it should also work for regression.
Does the "mean" actually matter for the loss function's behavior? In the end it's only some scalar.

The size of the gradient affects the learning rate, so yes it is important to get the mean correct.
Your point about using eq instead of sgn makes sense.  Here's a function that uses eq instead:
def non_zero_mean_squared_error(y_true, y_pred):
    total = theano.tensor.sum(theano.tensor.square(theano.tensor.eq(y_true,0)*(y_pred - y_true)), axis=-1)
    count = theano.tensor.sum(theano.tensor.eq(y_true,0))
    return total/count


update readme to install from https

Closes #95

looks like this is blocked on https://github.com/actions/runner-images/issues/10020


Modify: EclipseAdoptium.Temurin.18.JDK version <IP_ADDRESS>

Microsoft Reviewers: Open in CodeFlow

Hi @SpecterShell,
This manifest adds Apps and Features entries that aren't present in previous PR versions. These entries should be added to the previous versions, or removed from this version.
(Automated response - build 886.)


ADBDEV-3999: ORCA generates incorrect plan with exists clause for partitioned table

ORCA generates incorrect plan with exists clause for partitioned table
For subqueries with EXISTS, the first column is always selected by ORCA.
But if this column does not participate anywhere else in the query and is neither a system nor a distribution key, then it is not marked as used.
Therefore, when translating to DXL, such a column is skipped, and in a debug build, such a plan is generally not allowed!
On a release build, when a column is skipped, that column is not added to table_descr.
And therefore, its number is not inserted into the m_colid_to_attno_map hashmap of the correspondence of the column number to the attribute number.
And now the attribute number is not found by the column number.
And so a variable with an unknown attribute number is created.
Such a variable with an unknown attribute number is treated as a whole-row.
In this case, the oid of this variable does not correspond to the composite type and an error occurs.
The solution to this problem is to limit the output columns to only those referenced in the query for partitioned tables, as is done for non-partitioned tables.
The Union-On-HJNs test changed because in its plan the unused column B of the partitioned table disappeared.
Reproduction on release build:
CREATE TABLE offers (
    id int,
    product int,
    date date
) DISTRIBUTED BY (id);

INSERT INTO offers SELECT 0, 0, '2023-01-01';

CREATE TABLE contacts (
    contact text,
    id int,
    date date
) DISTRIBUTED BY (id) PARTITION BY RANGE(date) (START (date '2023-01-01') INCLUSIVE END (date '2023-02-01') EXCLUSIVE EVERY (INTERVAL '1 month'));

INSERT INTO contacts SELECT '0', 0, '2023-01-01';

EXPLAIN (ANALYZE on, COSTS off, VERBOSE on)
SELECT id FROM offers WHERE EXISTS (
    SELECT id FROM contacts WHERE product = 0 OR contacts.id = offers.id
);

ERROR:  type text is not composite  (seg1 slice2 <IP_ADDRESS>:6435 pid=55051)

On a debug build, ORCA can't build a plan.
CTranslatorExprToDXL.cpp:7580: Failed assertion: !reqd_prop_plan->PcrsRequired()->FMember(colref)
LOG:  Pivotal Optimizer (GPORCA) failed to produce plan (unexpected)

Allure report

Allure report https://allure-ee.adsw.io/launch/53148

Allure report https://allure-ee.adsw.io/launch/53651

Allure report https://allure-ee.adsw.io/launch/54790

have you researched why the problem is not reproduced on an unpartitioned table? it looks like this case should also be affected but it's not.

have you researched why the problem is not reproduced on an unpartitioned table? it looks like this case should also be affected but it's not.

The solution is not based on partitioned tables.

looks like DeriveOutputColumns should return a list of used columns, as is done for an unpartitioned table.
https://github.com/arenadata/gpdb/blob/3f979993d4832e4c68a8f0be339903de8de0aa6a/src/backend/gporca/libgpopt/src/operators/CLogicalGet.cpp#L212-L232
whereas in the case of a partitioned table, all columns are returned:
https://github.com/arenadata/gpdb/blob/3f979993d4832e4c68a8f0be339903de8de0aa6a/src/backend/gporca/libgpopt/src/operators/CLogicalDynamicGetBase.cpp#L177-L186
I would suggest bringing the behavior of a partitioned table to the behavior of a non-partitioned one. and thus not marking the column as used in FCreateCorrelatedApplyForExistentialSubquery and FRemoveExistentialSubquery is valid behavior. what do you think?

what do you think?

Applied

Allure report https://allure-ee.adsw.io/launch/57831

Failed job Orca unittests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/757160

Failed job Orca unittests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/757161

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/757164

Failed job Regression tests with ORCA on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/757158

Failed job Regression tests with ORCA on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/757159

bringing the behavior of a partitioned table to the behavior of a non-partitioned one

This approach didn't work because CLogicalDynamicGetBase::DeriveOutputColumns is used not only for exists queries, but also in many other places where unused columns are needed.

not marking the column as used in FCreateCorrelatedApplyForExistentialSubquery and FRemoveExistentialSubquery

I only mark the first column as used if all columns were unused.

what do you think?

Since queries with exists always require a used column, the approach I propose covers not only partitioned and non-partitioned tables, but also generally all cases in which the DeriveOutputColumns function will return the unused column first.

Allure report https://allure-ee.adsw.io/launch/57838

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/758259

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/758260

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/759684

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/759685

Allure report https://allure-ee.adsw.io/launch/57878

Failed job Regression tests with ORCA on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/759682

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/759943

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/759944

Failed job Behave tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/759683

I was able to reproduce the query from the Union-On-HJNs test, but was unable to reproduce the same plan.
create table mpp_r6756 (a int, b int, x int, y int) distributed randomly partition by list (a) (DEFAULT PARTITION o);
create table mpp_s6756 (c int, d int, e int) distributed randomly;
select a, b, count(x), count(y)
from mpp_r6756, mpp_s6756 q, mpp_s6756 w, mpp_s6756 e
where a = q.c and a = w.d and a = e.e
group by 1, 2
union all
select 1, * from (
select a, count(x), count(y)
from mpp_r6756, mpp_s6756 q, mpp_s6756 w, mpp_s6756 e
where a = q.c and a = w.d and a = e.e
group by 1
) a;

I think we should still mark the received attribute as used, or if we expect that only used attributes will returned then assert it.

if we expect that only used attributes will be returned then assert it.

I tried this and 2 tests are follbacks to postgres :(
explain SELECT * FROM csq_r WHERE exists (SELECT * FROM csq_f(csq_r.a));
explain SELECT * FROM csq_r WHERE not exists (SELECT * FROM csq_f(csq_r.a));

then let's mark them as used.

then let's mark them as used.

I reverted to my previous solution.

Ok. but add PR description and maybe to the code why it works without a patch in the case of non-partitioned tables.

Allure report https://allure-ee.adsw.io/launch/57932

Allure report https://allure-ee.adsw.io/launch/57933

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/762860

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/762877

I try more global refactor https://github.com/arenadata/gpdb/pull/642

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/762862

Failed job Behave tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/762857

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/762878

Failed job Regression tests with ORCA on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/762853

Allure report https://allure-ee.adsw.io/launch/61034

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/925834

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/925835

Failed job Behave tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/925832

Allure report https://allure-ee.adsw.io/launch/61225

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/935664

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/935665

Allure report https://allure-ee.adsw.io/launch/61230

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/935763

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/935764

Failed job Behave tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/935761

I rewrote the solution and corrected the description of the pull request. The problem only occurs with partitioned tables, so I only fixed them.

Allure report https://allure-ee.adsw.io/launch/61258

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/937051

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/937052

Failed job Regression tests with ORCA on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/937045

You say that only partitioned tables affected by problem and changed a code of a base class. This class is used by both, ordinary Dynamic Get and Bitmap ones. In my own tests I created a query with Bitmap Get, which has Unused column in exist clause too, but this query is not breaks on unpatched version.
So, I have a question: Are queries with Bitmap Get affected too?

If yes, we should include one example in tests file.
If no, then why? Probably it's possible to copy a solution from Bitmap Get.

Allure report https://allure-ee.adsw.io/launch/61417

Failed job ORCA code linter on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/949783

Allure report https://allure-ee.adsw.io/launch/61419

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/949851

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/949852

Are queries with Bitmap Get affected too?

I added bitmap and index scans to tests

Allure report https://allure-ee.adsw.io/launch/61571

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/957241

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/957242

Failed job Regression tests with ORCA on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/957235

Are queries with Bitmap Get affected too?

Yes, and also with Dynamic Index Get too

Are queries with Bitmap Get affected too?

Yes, and also with Dynamic Index Get too

Sorry, but I can't understand. You say that Bitmap Get is affected too, but code was moved from a base class.
More, when I try to run tests on not patched version the second select is successfully planned by orca. What is this test for? Please add some comments about key moments of what are you testing.

Allure report https://allure-ee.adsw.io/launch/61773

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/965737

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/965738

Failed job Regression tests with ORCA on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/965732

Allure report https://allure-ee.adsw.io/launch/61792

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/966961

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/966962

Failed job Regression tests with ORCA on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/966955

Failed job Regression tests with ORCA on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/966956

Failed job Regression tests with ORCA on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/967721

Modifying the CLogicalDynamicBitmapTableGet class does not solve the problem with the bitmap index, just as modifying the CLogicalDynamicIndexGet class does not solve the problem with the btree index, while modifying the CLogicalDynamicGet class, as well as the base class CLogicalDynamicGetBase, solves problems with both indexes and without indexes at all!
Modifications to index classes contain buggy alternatives, e.g.
diff --git a/src/backend/gporca/libgpopt/include/gpopt/operators/CLogicalDynamicIndexGet.h b/src/backend/gporca/libgpopt/include/gpopt/operators/CLogicalDynamicIndexGet.h
index e9be53c814..d569754c23 100644
--- a/src/backend/gporca/libgpopt/include/gpopt/operators/CLogicalDynamicIndexGet.h
+++ b/src/backend/gporca/libgpopt/include/gpopt/operators/CLogicalDynamicIndexGet.h
@@ -134,6 +134,10 @@ public:
 	virtual COperator *PopCopyWithRemappedColumns(
 		CMemoryPool *mp, UlongToColRefMap *colref_mapping, BOOL must_exist);

+	// derive output columns
+	virtual CColRefSet *DeriveOutputColumns(CMemoryPool *mp,
+											CExpressionHandle &exprhdl);
+
 	//-------------------------------------------------------------------------------------
 	// Required Relational Properties
 	//-------------------------------------------------------------------------------------
diff --git a/src/backend/gporca/libgpopt/src/operators/CLogicalDynamicIndexGet.cpp b/src/backend/gporca/libgpopt/src/operators/CLogicalDynamicIndexGet.cpp
index 2c6b2919f9..2bafee3be6 100644
--- a/src/backend/gporca/libgpopt/src/operators/CLogicalDynamicIndexGet.cpp
+++ b/src/backend/gporca/libgpopt/src/operators/CLogicalDynamicIndexGet.cpp
@@ -180,6 +180,38 @@ CLogicalDynamicIndexGet::PopCopyWithRemappedColumns(
 		ppartcnstrRel);
 }

+
+//---------------------------------------------------------------------------
+//	@function:
+//		CLogicalDynamicGet::DeriveOutputColumns
+//
+//	@doc:
+//		Derive output columns
+//
+//---------------------------------------------------------------------------
+CColRefSet *
+CLogicalDynamicIndexGet::DeriveOutputColumns(CMemoryPool *mp,
+											CExpressionHandle &	 // exprhdl
+)
+{
+	CColRefSet *pcrs = GPOS_NEW(mp) CColRefSet(mp);
+	for (ULONG i = 0; i < m_pdrgpcrOutput->Size(); i++)
+	{
+		// We want to limit the output columns to only those which are referenced in the query
+		// We will know the entire list of columns which are referenced in the query only after
+		// translating the entire DXL to an expression. Hence we should not limit the output columns
+		// before we have processed the entire DXL.
+		if ((*m_pdrgpcrOutput)[i]->GetUsage() == CColRef::EUsed ||
+			(*m_pdrgpcrOutput)[i]->GetUsage() == CColRef::EUnknown)
+		{
+			pcrs->Include((*m_pdrgpcrOutput)[i]);
+		}
+	}
+
+	return pcrs;
+}
+
+
 // Checking if index is partial given the table descriptor and mdid of the index
 BOOL
 CLogicalDynamicIndexGet::IsPartialIndex(CTableDescriptor *ptabdesc,

CREATE TABLE offers (
    id int,
    product int,
    date date
) DISTRIBUTED BY (id);
INSERT INTO offers SELECT i/100, i, '2023-01-01'::date + (i||' min')::interval FROM generate_series(1, 1000) i;

CREATE TABLE contacts (
    contact int,
    id int,
    date date
) DISTRIBUTED BY (id) PARTITION BY RANGE(date) (START (date '2023-01-01') INCLUSIVE END (date '2023-04-01') EXCLUSIVE EVERY (INTERVAL '1 month'));
INSERT INTO contacts SELECT i, i/100, '2023-01-01'::date + (i||' min')::interval FROM generate_series(1, 1000) i;

set optimizer_enforce_subplans = on;
set optimizer_enumerate_plans = on;

CREATE INDEX ON contacts USING btree(id);

set optimizer_plan_id = 13;
EXPLAIN (COSTS off, VERBOSE on)
SELECT id FROM offers WHERE EXISTS (
    SELECT id FROM contacts WHERE id = 1
);
                                               QUERY PLAN
--------------------------------------------------------------------------------------------------------
 Result
   Output: offers.id
   Filter: (SubPlan 1)
   ->  Gather Motion 3:1  (slice1; segments: 3)
         Output: offers.id
         ->  Seq Scan on postgres.offers
               Output: offers.id
   SubPlan 1  (slice0)
     ->  Limit
           Output: contacts.*
           ->  Materialize
                 Output: contacts.*
                 ->  Gather Motion 3:1  (slice2; segments: 3)
                       Output: contacts.*
                       ->  Limit
                             Output: contacts.*
                             ->  Result
                                   Output: contacts.*
                                   ->  Sequence
                                         Output: contacts.*, contacts.id
                                         ->  Partition Selector for contacts (dynamic scan id: 1)
                                               Partitions selected: 3 (out of 3)
                                         ->  Dynamic Seq Scan on postgres.contacts (dynamic scan id: 1)
                                               Output: contacts.*, contacts.id
                                               Filter: (contacts.id = 1)
 Optimizer: Pivotal Optimizer (GPORCA)
 Settings: optimizer=on
(27 rows)

set optimizer_plan_id = 14;
EXPLAIN (COSTS off, VERBOSE on)
SELECT id FROM offers WHERE EXISTS (
    SELECT id FROM contacts WHERE id = 1
);
                                            QUERY PLAN
--------------------------------------------------------------------------------------------------
 Result
   Output: offers.id
   Filter: (SubPlan 1)
   ->  Gather Motion 3:1  (slice1; segments: 3)
         Output: offers.id
         ->  Seq Scan on postgres.offers
               Output: offers.id
   SubPlan 1  (slice0)
     ->  Limit
           Output: contacts.*
           ->  Materialize
                 Output: contacts.*
                 ->  Gather Motion 3:1  (slice2; segments: 3)
                       Output: contacts.*
                       ->  Result
                             Output: contacts.*
                             ->  Sequence
                                   Output: contacts.*, contacts.id
                                   ->  Partition Selector for contacts (dynamic scan id: 1)
                                         Partitions selected: 3 (out of 3)
                                   ->  Dynamic Seq Scan on postgres.contacts (dynamic scan id: 1)
                                         Output: contacts.*, contacts.id
                                         Filter: (contacts.id = 1)
 Optimizer: Pivotal Optimizer (GPORCA)
 Settings: optimizer=on
(25 rows)

set optimizer_plan_id = 27;
EXPLAIN (COSTS off, VERBOSE on)
SELECT id FROM offers WHERE EXISTS (
    SELECT id FROM contacts WHERE id = 1
);
                                                     QUERY PLAN
--------------------------------------------------------------------------------------------------------------------
 Gather Motion 3:1  (slice3; segments: 3)
   Output: offers.id
   ->  Seq Scan on postgres.offers
         Output: offers.id
         Filter: (SubPlan 1)
         SubPlan 1  (slice3; segments: 3)
           ->  Materialize
                 Output: contacts.*
                 ->  Broadcast Motion 1:3  (slice2)
                       Output: contacts.*
                       ->  Limit
                             Output: contacts.*
                             ->  Gather Motion 3:1  (slice1; segments: 3)
                                   Output: contacts.*
                                   ->  Limit
                                         Output: contacts.*
                                         ->  Result
                                               Output: contacts.*
                                               ->  Sequence
                                                     Output: contacts.*, contacts.id
                                                     ->  Partition Selector for contacts (dynamic scan id: 1)
                                                           Partitions selected: 3 (out of 3)
                                                     ->  Dynamic Seq Scan on postgres.contacts (dynamic scan id: 1)
                                                           Output: contacts.*, contacts.id
                                                           Filter: (contacts.id = 1)
 Optimizer: Pivotal Optimizer (GPORCA)
 Settings: optimizer=on
(27 rows)

set optimizer_plan_id = 28;
EXPLAIN (COSTS off, VERBOSE on)
SELECT id FROM offers WHERE EXISTS (
    SELECT id FROM contacts WHERE id = 1
);
                                                  QUERY PLAN
--------------------------------------------------------------------------------------------------------------
 Gather Motion 3:1  (slice3; segments: 3)
   Output: offers.id
   ->  Seq Scan on postgres.offers
         Output: offers.id
         Filter: (SubPlan 1)
         SubPlan 1  (slice3; segments: 3)
           ->  Materialize
                 Output: contacts.*
                 ->  Broadcast Motion 1:3  (slice2)
                       Output: contacts.*
                       ->  Limit
                             Output: contacts.*
                             ->  Gather Motion 3:1  (slice1; segments: 3)
                                   Output: contacts.*
                                   ->  Result
                                         Output: contacts.*
                                         ->  Sequence
                                               Output: contacts.*, contacts.id
                                               ->  Partition Selector for contacts (dynamic scan id: 1)
                                                     Partitions selected: 3 (out of 3)
                                               ->  Dynamic Seq Scan on postgres.contacts (dynamic scan id: 1)
                                                     Output: contacts.*, contacts.id
                                                     Filter: (contacts.id = 1)
 Optimizer: Pivotal Optimizer (GPORCA)
 Settings: optimizer=on
(25 rows)

Allure report https://allure-ee.adsw.io/launch/62824

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1011576

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1011577

Failed job Regression tests with ORCA on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1011571

I'm ready to approve, but we don't have a proper commit message for this. Old message is related to old solution and probably bad formatted.

a proper commit message for this

added empty commit

Allure report https://allure-ee.adsw.io/launch/63286

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1033248

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1033249

Failed job Regression tests with ORCA on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1033242

Allure report https://allure-ee.adsw.io/launch/63349

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1036527

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1036528

Please, update the branch it's out of date

Allure report https://allure-ee.adsw.io/launch/63746

Failed job Resource group isolation tests on x86_64: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1051551

Failed job Resource group isolation tests on ppc64le: https://gitlab.adsw.io/arenadata/github_mirroring/gpdb/-/jobs/1051552


[SDP-1132] serve: Multi-tenant instance breaks when user is logged in using the base domain.

What
Multi-tenant instance breaks when the user is logged in using the base domain.
Why
InjectTenantMiddleware was trying to fetch tenant ID from the token, but AuthenticateMiddleware only added the token to the request context later.
Known limitations
N/A
Checklist
PR Structure

[x] This PR has a reasonably narrow scope (if not, break it down into smaller PRs).
[x] This PR title and description are clear enough for anyone to review it.
[x] This PR does not mix refactoring changes with feature changes (split into two PRs otherwise).

Thoroughness

[x] This PR adds tests for the new functionality or fixes.
[x] This PR contains the link to the Jira ticket it addresses.

Configs and Secrets

[x] No new CONFIG variables are required -OR- the new required ones were added to the helmchart's values.yaml file.
[x] No new CONFIG variables are required -OR- the new required ones were added to the deployments (pr-preview, dev, demo, prd).
[x] No new SECRETS variables are required -OR- the new required ones were mentioned in the helmchart's values.yaml file.
[x] No new SECRETS variables are required -OR- the new required ones were added to the deployments (pr-preview secrets, dev secrets, demo secrets, prd secrets).

Release

[x] This is not a breaking change.
[x] This is ready for production.. If your PR is not ready for production, please consider opening additional complementary PRs using this one as the base. Only merge this into develop or main after it's ready for production!

Deployment

[ ] Does the deployment work after merging?

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr231.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr231.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr231.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr231.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr231.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr231.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr231.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr231.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr231.previews.kube001.services.stellar-ops.com

Something went wrong with PR preview build please check

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr231.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr231.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr231.previews.kube001.services.stellar-ops.com


Add validation to manipulations

Some manipulations can break the equation LaTeX if the wrong thing is selected. This shouldn't happen, and can be fixed with some validation code.

Have added some validation in commit 0f4efece4bc9

Have added more validation from the new_design branch, merged into master in commit f3e1aa35828960116107b24bd878e0c76b1a40e0


RStudio is not supported for current syntax,

Hi! Here a happy SendCode/R-Box user!
After experimenting for a while with SendCode and R scripts, I would like to use Sublime Text to author R Markdown documents. Thanks in advance for any help!
After composing a very document in RStudio and publishing it here...
http://rpubs.com/rrodriguez/ggplot2_colors
... I would also like to move its edition to Sublime Text. Thus, I've created a Gist with the same code, opened it in Sublime Text 3, and submitted it by using R-Box > Render Rmd to HTML. And get this error...

SendCode has been previously configured to sending to RStudio. If I select SendCode: Choose REPL Program when the Rmd file is opened, no R GUI or RStudio Desktop options appear. Those are only available to R files.
Please, what am I using wrong? How could I use Sublime Text 3 to edit and render R Markdown scripts? Is it possible?
Thanks for any help!

@rrodrigueznt sorry for the long delay. Somehow I have missed this thread. It should be fixed with the latest SendCode.

I am experiencing the same issue.  I am trying to send to rgui, but receive an error saying Rgui is not supported for the current system.

it should be fixed in the latest SendCode. Try update SendCode first and let me know if you can still reproduce it.

Randy, please, could you provide me with a working R Markdown example that you can confirm MUST work when sent from SendCode to RStudio? My test code is now travelling nicely from Sublime/RStudio but I'm not able yet to get the output. The same code directly edited with/in RStudio produces a nice output! I would like to debug the problem here: an example working for you will be a great help! Thanks!

What do you mean by not getting the output?

Sorry for not being clear! For instance, this code really simple, the one RStudio includes as an example/template when creating a new RMarkdown script, submitted (Knit to PDF) directly at RStudio...
http://portal.idisantiago.es/xwiki/bin/download/R/WebHome/PDFtest.Rmd
.... gives this PDF file...
http://portal.idisantiago.es/xwiki/bin/download/R/WebHome/PDFtest.pdf
If I open the script with Sublime Text and R-Box > Knit Rnw to PDF it, I get this messages...
http://portal.idisantiago.es/xwiki/bin/download/R/WebHome/PDFtest.Rmd.log.txt
... and no PDF is produced. it seems that the intermediary *.tex file is not correct! What I see is a *.tex file similar to the original *.Rmd one. Here it is...
http://portal.idisantiago.es/xwiki/bin/download/R/WebHome/PDFtest.tex
Same box. Please, any idea? Thanks!

Because it is a Rmd file but not a Rmw file. I guess you have mixed up Rmarkdowm and RSweave files.
Btw, though the menu only says Rmarkdow to HTML, it is still capable of generating PDF by specifying PDF output in the yank header.

Ummm... I've made a mix during the trials, but I think I'm able to differentiate between Rnw and Rmd files. I'll do my best to clarify the situation and report back in the following few days. Going to use these two files as "case study"...
Rnw file...
http://portal.idisantiago.es/xwiki/bin/download/R/WebHome/SweaveTest.Rnw
Rmd file...
http://portal.idisantiago.es/xwiki/bin/download/R/WebHome/PDFtest.Rmd
I'm getting again the "rstudio is not supported for this syntax error message" when R-Box > Knit or Sweave Rnw files to RStudio from Sublime Text, but perhaps my failure is too obvious. Please, allow me to think a bit about this! Thanks!
Thanks for your time!

Yeap... in fact, if I don't set the syntax, Sublime Text R-Box menu doesn't appear!

Ai...there is a bug for Rnw files. Will fix it soon


Add a way to preview an images interim status

It would be great if we could get the data of each intermediate step as an image.
This way we could build a preview in our UIs like this:

This would increase user experience especially on iPhones and iPads, where computing times are usually longer than on the Mac.

As an example, in my sample iOS app, I display the intermediate images step by step.

https://github.com/ynagatomo/ARDiffMuseum

Yes I’d love this

@ynagatomo Thanks so much for your example code! Could you point us to the right direction in which place you have implemented the preview method? I wasn’t able to find it.

Hi. In the ImageGenerator.swift,

set the progress handler when calling generateImages() method;
let cgImages = try sdPipeline.generateImages(prompt: param.prompt,
negativePrompt: param.negativePrompt,
imageCount: param.imageCount,
stepCount: param.stepCount,
seed: UInt32(param.seed),
guidanceScale: param.guidanceScale,
disableSafety: param.disableSafety,
progressHandler: self.progressHandler) // <-- progress handler

The apple/ml-stable-diffusion library provides the immediate images via the progress handler's StableDiffusionPipeline.Progress parameter;
nonisolated func progressHandler(progress: StableDiffusionPipeline.Progress) -> Bool {
debugLog("IG: Progress: step / stepCount = (progress.step) / (progress.stepCount)")
 if ProcessInfo.processInfo.isiOSAppOnMac {
     let generatedImages = GeneratedImages(parameter: GenerationParameter(prompt: progress.prompt,
                                          negativePrompt: "unknow", // progress does not provide this now
                                          guidanceScale: 0.0, // progress does not provide this now
                                          seed: 0,
                                          stepCount: progress.stepCount,
                                          imageCount: progress.currentImages.count,
                                          disableSafety: progress.isSafetyEnabled),
                                          images: progress.currentImages.compactMap {
         if let cgImage = $0 {
             return UIImage(cgImage: cgImage)
         } else {
             return nil
         }
     })

     DispatchQueue.main.async {
         self.setGeneratedImages(generatedImages)
         self.setProgressStep(step: progress.step, stepCount: progress.stepCount)
     }
 } else {
     DispatchQueue.main.async {
         self.setProgressStep(step: progress.step, stepCount: progress.stepCount)
     }
 }

 return true // continue

}

In my sample code, I convert the immediage images (CGImage) to UIImage and display them in SwiftUI views.

Getting the intermediate image takes time, so the above code, as the simplest case, doesn't do it on iPhone or iPad, but does it on macOS.
Ex. using an iPad Pro 2020/A12Z, for generating a image (512x512 px) with 20 steps;

with displaying 20 intermediate images, it takes 7 min 20 sec
without displaying intermediate images, it takes 2 min
I recommend adding UIs to enable/disable intermediate image display and to set the step interval for displaying intermediate images. (I will do this on my project later: https://github.com/ynagatomo/ARDiffMuseum)


fix(index): update gtag container id

https://openforge.teamwork.com/index.cfm#/projects/235303/tasks/17524767

QA Passes


Use trusted publisher deployment

Description
PyPI now allows and encourages uploading files without using any API TOKENS. The trusted publisher deployment can be configured from PyBaMM's PyPI dashboard -
Your account > pybamm > publishing
The tab will look something like this -

Environment name should be set to pypi while filling in the details, and the workflow should work!

[!NOTE]
The owner account will have to enable 2FA (which should not be an extra overhead, given that 2FA will be mandatory on all PyPI accounts by the end of this year).

I don't have access to it, @tinosulzer do you have access? If not, I can reset the password and set this up.

Any update on this? Should I chase for the access details?

Should I chase for the access details?

Yes, that would be great. To be on the safer side, we can enable this for v24.1.

@Saransh-cpp, @brosaplanella Is this still being done for v24.1

This has now been approved, so happy to go with it. @Saransh-cpp let me know if you need the credentials to log in.

@Saransh-cpp let me know if you need the credentials to log in.

That would be great, @brosaplanella. I'll be able to update everything in PyPI.

this should modify the TestPyPI job too with an accompanying addition of a testpypi environment in the repository

maybe sometime after the v24.1 release (or if someone else could take a look at it before the release)

Now should be a good time to get this in. Also, to note: the API credentials should be removed from the TestPyPI step as well.

@Saransh-cpp, @agriyakhetarpal, @brosaplanella What still needs to be done for this one?

The changes in the workflow file are correct, we just need the pypi environment set up in the repository settings and the OIDC credentials set up on PyPI

Ahh, I always forget about this PR. Not this time.

Okay this should be good to go now (thanks for updating the settings!)

the API credentials should be removed from the TestPyPI step as well

Do we need this, @Saransh-cpp? Both of these uploads are in steps that use the same environment/job, so removing the API keys will break the TestPyPI uploads. We never know when we might require them.

the API credentials should be removed from the TestPyPI step as well

Do we need this, @Saransh-cpp? Both of these uploads are in steps that use the same environment/job, so removing the API keys will break the TestPyPI uploads. We never know when we might require them.

I would say we should merge this and then open a new PR to also fix the TestPyPi stuff. That way we have this in place and if we forget the next step the failing test will remind us

@agriyakhetarpal I created a ticket for the TestPyPi job


Rename master branch to main branch in this repo

What is the expected behavior?
Many (if not all) repos in qiskit github org has renamed master branch to main branch, we should do that in this repo as well. I am not sure about the implications and the steps to do that though.

I am not sure if it is that simple. @mtreinish Can you shed some lights here? I did a quick search and found a few places with master https://github.com/Qiskit/qiskit/search?q=master

Yeah, actually swapping the "default" branch on GitHub is very easy, the extra bits are just making sure that all our hyperlinks point to the new default branch.  That's a bit harder, because there's probably quite a few places across a bunch of different Qiskit components, and possibly the textbook, that might need updating.

With this repo being sunset now, we won't be going ahead with a change.  Terra, which will for most purposes replace this repository, already uses main as its default branch.


rkt: version information is old/wrong

coreos-stable-00 rocket-v0.1.1 # ./rkt version
rkt version 0.1.0
coreos-stable-00 rocket-v0.1.1 # ./rkt help
NAME:
rkt - rocket, the application container runner
USAGE:
rkt [global options]  [command options] [arguments...]
VERSION:
0.1.0

Wow, that's embarassing. I think this got lost when the appc/spec versioning change happened. Thanks for catching!


[PAPP-32386] threatstream: Support for custom Observable Types

Please ensure your pull request (PR) adheres to the following guidelines:

Please refer to our contributing documentation for any questions on submitting a pull request, link: Contribution Guide

Pull Request Checklist
Please check if your PR fulfills the following requirements:

[ ] Testing of all the changes has been performed (for bug fixes / features)
[x] The manual_readme_content.md has been reviewed and added / updated if needed (for bug fixes / features)
[x] Use the following format for the PR description: <App Name>: <PR Type> - <PR Description>
[ ] Provide release notes as part of the PR submission which describe high level points about the changes for the upcoming GA release.
[ ] Verify all checks are passing.
[x] Do NOT use the next branch of the forked repo. Create separate feature branch for raising the PR.
[x] Do NOT submit updates to dependencies unless it fixes an issue.

Pull Request Type
Please check the type of change your PR introduces:

[ ] New App
[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation
[ ] Other (please describe):

Security Considerations (REQUIRED)

[x] If you are exposing any endpoints using a REST handler,
please document them in the manual_readme_content.md.
[x] If this is a new connector or you are adding new actions

[x] Please document in the manual_readme_content.md all methods (eg, OAuth) used to authenticate
with the service that the connector is integrating with.
[x] If any actions are unable to run on SOAR Cloud, please document this in the manual_readme_content.md.

[x] Are you introducing any new cryptography modules? If yes, please elaborate their purpose:
[x] Are you are accessing the file system? If yes, please verify that you are only accessing paths returned through
the Vault API.
[x] Are you are marking code to be ignored by Semgrep with nosemgrep?
If yes, please provide justification in an additional comment next to the ignored code.

Release Notes (REQUIRED)

Provide release notes as part of the PR submission which describe high level points about the changes for the upcoming GA release.

What is the current behavior? (OPTIONAL)

Currently we are validating provided observable types and limiting choice in action.

What is the new behavior? (OPTIONAL)

We are no longer validating observable types, and we are suggesting default types in action in description. There are no dropdown options.

Other information (OPTIONAL)

Any other information that is important to this PR such as screenshots of how the component looks before and after the change.

Pay close attention to (OPTIONAL)

Any specific code change or test case points which must be addressed/reviewed at the time of GA release.

Screenshots (if relevant)

Thanks for contributing!

threatstream tests has around 93% of pass rate. Errors are due to resources being unavailable.
Changed action: "import observable" is passing 100% tests.


display certain output formats in detailed simulation view

See #5

Figure out which output formats we want to support

Right now, I think images are the most important

Fix #25 first, that would be more important


Using L20n To Localize XUL Documents

@zbraniecki what do you think?

yeah, looks good. I'll take a stab at some edits later, but you should land it :)

Thanks!


Link Twitter after migrating to Parse server

Hello,
We are trying to test twitter link and facebook link after migrating to Parse server and hitting issues with Twitter link. We see following error on console log,
Twitter auth is invalid for this user. (Code: 101, Version: 1.13.0)
We have all config set on index.js with Twitter secret and client key and using latest SDKs
Facebook link works fine.
Any suggestions on how to debug the issue?

Probably more relevant place for reporting this, so let's move overall discussion on this to ParsePlatform/ParseTwitterUtils-iOS#35


Do we have any document/steps for Migrating Oracle on Premises to Azure Oracle DB

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9837f109-e89c-621e-63fd-a9acd8a090dc
Version Independent ID: ef17dd8c-6dae-bc06-5b19-8eea3bb53e63
Content: Oracle Database Migration to Azure - Azure Solution Ideas
Content Source: docs/solution-ideas/articles/reference-architecture-for-oracle-database-migration-to-azure.md
Service: architecture-center
Sub-service: solution-idea
GitHub Login: @adamboeglin
Microsoft Alias: pnp

Thanks for reporting - this issue is under review.  This is a Microsoft Internal DevOps Bug ID AB#195541

@jameroym the following Oracle DB scenario are available in Azure
https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/oracle/oracle-overview
In order to migrate form on-prem, Azure supports https://docs.microsoft.com/en-us/azure/dms/tutorial-oracle-azure-postgresql-online


editor.lineNumbers description needs to be updated for "on", "off", "relative"

VSCode Version:1.6.0 insiders
OS Version:Windows 10

From Default Settings:
// Controls visibility of line numbers
"editor.lineNumbers": "on",
Should be something like:
"Controls display of line numbers. Possible values are "on", "off", and "relative". The "relative" options shows the line count relative to the current cursor position."

This is also a good oportunity for a pr so assigning also Greg if he wants to fix this. Here is a code pointer
https://github.com/Microsoft/vscode/blob/master/src/vs/editor/common/config/commonEditorConfig.ts#L656


scope.chart.update does not refresh data, updateWithData works instead

In version 1.0.3 the default of refreshDataOnly is true, which leads to trying to call scope.chart.update(). In line 382:
if (!scope._config.disabled && scope._config.autorefresh) {
    scope._config.refreshDataOnly && scope.chart.update ? scope.chart.update() : scope.api.refresh();
}

But scope.chart.update() does is not update my chart after is called. However updateWithData does work:
if (!scope._config.disabled && scope._config.autorefresh) {
    scope._config.refreshDataOnly ? scope.api.updateWithData() : scope.api.refresh();
}

I am using the 'stackedAreaChart'.

Can you give some examples how you resolved the refresh issue? My scatter plot only updates the first time. Every time new data are loaded, still the previous are shown.
I use the following:
[html]

[javascript]
$http.get(url).then(function(response) {
$scope.paretoData   = response.data;
$scope.api.updateWithData($scope.paretoData);
});

Versions 1.0.2 & 1.0.3 brought much pain for people, sorry for that... In the new release v1.0.4 all it's fixed. Now the charts are dynamic by default again. In addition, we can specify the data change detection strategy, to avoid some performance issue. Details in the docs.


about the learning rate for single GPU

我想请问一下，根据get start.txt，config的optimizer中的学习率是根据8GPUS设置的，即Batch size为2的情况下，8GPUs的lr为0.01。为何get start.txt中说0.01对应的是4GPUS？ 如果我用的是单核GPU，那么我的学习率应该设置为0.00125还是0.0025呢？

The learning rate of SGD depends on the batch size. If your batch size is bs, then the lr = 0.01/8 * bs.

The learning rate of SGD depends on the batch size. If your batch size is bs, then the lr = 0.01/8 * bs.

Okay, Thanks for you reply!