[SPARK-18218][ML][MLLib] Reduce shuffled data size of BlockMatrix multiplication and solve potential OOM and low parallelism usage problem By split middle dimension in matrix multiplication

What changes were proposed in this pull request?
The problem in current block matrix mulitiplication
As in JIRA https://issues.apache.org/jira/browse/SPARK-18218 described, block matrix multiplication in spark may cause some problem, suppose we have M*N dimensions matrix A multiply N*P dimensions matrix B, when N is much larger than M and P, then the following problem may occur:

when the middle dimension N is too large, it will cause reducer OOM.
even if OOM do not occur, it will still cause parallism too low.
when N is much large than M and P, and matrix A and B have many partitions, it may cause too many partition on M and P dimension, it will cause much larger shuffled data size. (I will expain this in detail in the following.)

Key point of my improvement
In this PR, I introduce midDimSplitNum parameter, and improve the algorithm, to resolve this problem.
In order to understand the improvement in this PR, first let me give a simple case to explain how the current mulitiplication works and what cause the problems above:
suppose we have block matrix A, contains 200 blocks (2 numRowBlocks * 100 numColBlocks), blocks arranged in 2 rows, 100 cols:
A00 A01 A02 ... A0,99
A10 A11 A12 ... A1,99

and we have block matrix B, also contains 200 blocks (100 numRowBlocks * 2 numColBlocks), blocks arranged in 100 rows, 2 cols:
B00    B01
B10    B11
B20    B21
...
B99,0  B99,1

Suppose all blocks in the two matrices are dense for now.
Now we call A.multiply(B), suppose the generated resultPartitioner contains 2 rowPartitions and 2 colPartitions (can't be more partitions because the result matrix only contains 2 * 2 blocks), the current algorithm will contains two shuffle steps:
step-1
Step-1 will generate 4 reducer, I tag them as reducer-00, reducer-01, reducer-10, reducer-11, and shuffle data as following:
A00 A01 A02 ... A0,99
B00 B10 B20 ... B99,0    shuffled into reducer-00

A00 A01 A02 ... A0,99
B01 B11 B21 ... B99,1    shuffled into reducer-01

A10 A11 A12 ... A1,99
B00 B10 B20 ... B99,0    shuffled into reducer-10

A10 A11 A12 ... A1,99
B01 B11 B21 ... B99,1    shuffled into reducer-11

and the shuffling above is a cogroup transform, note that each reducer contains only one group.
step-2
Step-2 will do an aggregateByKey transform on the result of step-1, will also generate 4 reducers, and generate the final result RDD, contains 4 partitions, each partition contains one block.
The main problems are in step-1. Now we have only 4 reducers, but matrix A and B have 400 blocks in total, obviously the reducer number is too small.
and, we can see that, each reducer contains only one group(the group concept in coGroup transform), each group contains 200 blocks. This is terrible because we know that coGroup transformer will load each group into memory when computing. It is un-extensable in the algorithm level. Suppose matrix A has 10000 cols blocks or more instead of 100? Than each reducer will load 20000 blocks into memory. It will easily cause reducer OOM.
This PR try to resolve the problem described above.
When matrix A with dimension M * N multiply matrix B with dimension N * P, the middle dimension N is the keypoint. If N is large, the current mulitiplication implementation works badly.
In this PR, I introduce a numMidDimSplits parameter, represent how many splits it will cut on the middle dimension N.
Still using the example described above, now we set numMidDimSplits = 10, now we can generate 40 reducers in step-1:
the reducer-ij above now will be splited into 10 reducers: reducer-ij0, reducer-ij1, ... reducer-ij9, each reducer will receive 20 blocks.
now the shuffle works as following:
reducer-000 to reducer-009
A0,0 A0,10 A0,20 ... A0,90
B0,0 B10,0 B20,0 ... B90,0    shuffled into reducer-000

A0,1 A0,11 A0,21 ... A0,91
B1,0 B11,0 B21,0 ... B91,0    shuffled into reducer-001

A0,2 A0,12 A0,22 ... A0,92
B2,0 B12,0 B22,0 ... B92,0    shuffled into reducer-002

...

A0,9 A0,19 A0,29 ... A0,99
B9,0 B19,0 B29,0 ... B99,0    shuffled into reducer-009

reducer-010 to reducer-019
A0,0 A0,10 A0,20 ... A0,90
B0,1 B10,1 B20,1 ... B90,1    shuffled into reducer-010

A0,1 A0,11 A0,21 ... A0,91
B1,1 B11,1 B21,1 ... B91,1    shuffled into reducer-011

A0,2 A0,12 A0,22 ... A0,92
B2,1 B12,1 B22,1 ... B92,1    shuffled into reducer-012

...

A0,9 A0,19 A0,29 ... A0,99
B9,1 B19,1 B29,1 ... B99,1    shuffled into reducer-019

reducer-100 to reducer-109 and reducer-110 to reducer-119 is similar to the above, I omit to write them out.
API for this optimized algorithm
I add a new API as following:
  def multiply(
      other: BlockMatrix,
      numMidDimSplits: Int // middle dimension split number, expained above
): BlockMatrix

Shuffled data size analysis (compared under the same parallelism)
The optimization has some subtle influence on the total shuffled data size. Appropriate numMidDimSplits will significantly reduce the shuffled data size,
but too large numMidDimSplits may increase the shuffled data in reverse. For now I don't want to introduce formula to make thing too complex, I only use a simple case to represent it here:
Suppose we have two same size square matrices X and Y, both have 16 numRowBlocks * 16 numColBlocks. X and Y are both dense matrix. Now let me analysis the shuffling data size in the following case:
case 1: X and Y both partitioned in 16 rowPartitions and 16 colPartitions, numMidDimSplits = 1
ShufflingDataSize = (16 * 16 * (16 + 16) + 16 * 16) blocks = 8448 blocks
parallelism = 16 * 16 * 1 = 256 //use step-1 reducers number as the parallism because it cost most of the computation time in this algorithm.
case 2: X and Y both partitioned in 8 rowPartitions and 8 colPartitions, numMidDimSplits = 4
ShufflingDataSize = (8 * 8 * (32 + 32) + 16 * 16 * 4) blocks = 5120 blocks
parallelism = 8 * 8 * 4 = 256 //use step-1 reducers number as the parallism because it cost most of the computation time in this algorithm.
The two cases above all have parallism = 256, case 1 numMidDimSplits = 1 is equivalent with current implementation in mllib, but case 2 shuffling data is 60.6% of case 1, it shows that under the same parallelism, proper numMidDimSplits will significantly reduce the shuffling data size.
How was this patch tested?
Test suites added.
Running result:

Test build #67969 has started for PR 15730 at commit a6bc12d.

@yanboliang @sethah I will be very pleasure to hear your opinions, thanks!

Test build #67969 has finished for PR 15730 at commit a6bc12d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BlockMatrix @Since(\"2.1.0\") (
trait DistributedMatrix extends Serializable

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67969/
Test PASSed.

Test build #67977 has started for PR 15730 at commit 4324a52.

Test build #67977 has finished for PR 15730 at commit 4324a52.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67977/
Test PASSed.

ping @brkyvz who wrote these libraries.

Hi @WeichenXu123 Thank you for this PR. Sorry for taking so long to get back to you. Your optimization would be very helpful. I have a couple thoughts though. Your examples always take into account fully dense matrices, i.e. that all blocks exist all the time. How would sparsity affect shuffling? Would there ever be a case where sparsity of blocks and unlucky alignment of blocks could actually cause a lot more shuffling with your parameter?
Nevertheless, I can see fully dense matrix multiplications benefitting significantly from your optimization. I guess we will need to work on the APIs a bit and document it a bit more clearly.

@brkyvz
Good question about the shuffling data in the sparse case. Now I give some simple analysis on it (maybe not very strict):
as discussed above, the shuffling data contains step-1 and step-2,
if we keep parallelism the same, we need to increase the midDimSplitNum and decrease ShuffleRowPartitions and ShuffleColPartitions, in such case,  we can make sure that step-1 shuffling data will always reduce, and step-2 shuffling data will always increase.
Now let us consider the sparse case, the step-2  shuffling data increasing because when each pair of left-matrix row-sets multiplying right-matrix col-sets, we split it into multiple parts (midDimSplitNum parts), in each parts we do the multiplying and we need to aggregate all parts together, the more parts need to be aggregate, the more shuffling data it needs. BUT, in sparse case, these parts(after multiplying) will be empty in high probability, so to those empty parts it shuffle nothing. So, we can expect that in sparse case, step-2 shuffling data won't increase much rather than the midDimSplitNum = 1 case, because most split parts will shuffle nothing.
Now I am considering improve the API interface to make it easier to use.
I would like to let user specified the parallism, and the algorithm automatically calculate the optimal midDimSplitNum, what do you think about it ?
And thanks for careful review!

@WeichenXu123 How about if we only add:
def multiply(other: BlockMatrix, numMidDimSplits: Integer): BlockMatrix

as the public API. Users shouldn't have to define GridPartitioners themselves.

This way, all that needs to change in the implementation of multiply is:
val intermediatePartitioner = new Partitioner {
  override def numPartitions: Int = resultPartitioner.numPartitions * numMidDimSplits
  override def getPartition(key: Any): Int = key.asInstanceOf[Int]
}
val newBlocks = flatA.cogroup(flatB, intermediatePartitioner).flatMap { case (pId, (a, b)) =>

@brkyvz All right, I'll update code ASAP. Thanks!

Test build #70574 has started for PR 15730 at commit 13ccfff.

Test build #70574 has finished for PR 15730 at commit 13ccfff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70574/
Test PASSed.

@brkyvz I update code and attach a running result screenshot, waiting for your review, thanks!

@WeichenXu123 Thanks! Will take a look once I get back from vacation (in a week). Happy new year!

Looks pretty good overall. Left one major comment about documentation and one about tests.

@brkyvz Done. Thanks!

Test build #71219 has started for PR 15730 at commit bf2a603.

Test build #71219 has finished for PR 15730 at commit bf2a603.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71219/
Test FAILed.

Jenkins, test this please.

Test build #71242 has started for PR 15730 at commit bf2a603.

Test build #71242 has finished for PR 15730 at commit bf2a603.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71242/
Test PASSed.

cc @yanboliang Thanks!

Test build #71390 has started for PR 15730 at commit 55dabe0.

Test build #71390 has finished for PR 15730 at commit 55dabe0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71390/
Test PASSed.

Merged build finished. Test PASSed.

@WeichenXu123 This LGTM thanks. @mengxr Would you also like to take a look?

The API looks good to me.  I have not reviewed the internals carefully.
One comment: Let's add a check to verify that numMidDimSplits is > 0.

Test build #71691 has started for PR 15730 at commit 3feca58.

Test build #71691 has finished for PR 15730 at commit 3feca58.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71691/
Test PASSed.

Merged build finished. Test PASSed.

@WeichenXu123 This LGTM! Thank you for providing this functionality. I left one final comment then I'll merge this

@brkyvz Also thanks for your careful code review! ^_^

Test build #72063 has started for PR 15730 at commit 4edcd74.

Test build #72063 has finished for PR 15730 at commit 4edcd74.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72063/
Test PASSed.

Merged build finished. Test PASSed.

Merging to master! Thanks!


[Add]

Site URL
https://youtube.com/playlist?list=PLwgFb6VsUj_n15Cg_y2ULKfsOR1XiQqPx&si=UJyXUgEH0_94SpYj
Description of the above provided site
Complete javascript from beginner to mastery.
Category
Languages
Sub Category
JavascriptInHindi
Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue (optional)
[X] I am participating in Hacktoberfest 2023

@Anmol-Baranwal  ,  As i see there is no sub category in youtube for javascript. Would i create one and then update it in the data.ts and index.ts or add it in language category.

@Atul72
It's here.

revise the description of the issue and raise PR accordingly.


Rename generated front test folders?

Should we rename generated frontend application test folders to "unit" and "component" to match the nammings used in the generator app and add some consistence to that? This is a breaking change for the generated applications
cc @murdos  @pascalgrimaud @qmonmert any advice on that?

I think it would be nice to do it.
Sure it might be considered a breaking change, but I'm not sure that we have a lot of users for the frontend modules.

Same opinion than @murdos
Let's do it

Do we all agree to also rename the test/javascript folder to test/webapp?
This is how the generator itself is structured, and that way it's coherent with the src/main/webapp folder


Update to discord.js version 13.6.0

This can wait until other pull requests (like #32, #25, and #28) are closed because merging them first makes it easier to update


SDFGI leaks light into enclosed spaces, even when Use Occlusion is enabled

Godot version
Godot 4 nightly build
System information
Windows 10 v20H2, Intel i7 6700, Nvidia GTX1080, 16GB RAM
Issue description
Depending on distance from the subject mesh, SDFGI will unrealistically leak light into an enclosed or partially-enclosed space. This seems to be due to cascade switching due to the camera's proximity and will change as the camera approaches. These shifts otherwise appear as upscaling of GI detail but are highly noticeable in spaces where light should not be leaking. These spaces will begin bright and progressively darken.
This could simply be a limitation of SDFGI, but I thought it worth bringing to attention. This could not be resolved with quality tweaking (setting the cascades to the maximum of 8 and Y-scale to 50%) and adjusting the bias simply resulted in more extreme distortions.
Steps to reproduce
SDFGI will show traces of light leaking in any project with an enclosed space. In my experience thickness does not seem to affect this since when SDFGI is having a minimal effect due to camera proximity, Godot will default to displaying the space without GI information. In many cases, an enclosed space will be brighter under these conditions than SDFGI would usually introduce after enough resolution has been achieved to accurately portray the lighting.
Since the steps to reproduce are not exact and slightly arbitrary, I have provided screenshots and a reproduction below.
At a distance, the light leak is quite obvious:

As the camera moves closer, the leak changes in nature and reduces but is still visible:

Finally, when maximum resolution is achieved the resulting GI will be considerably different than at a distance:

Minimal reproduction project
I have provided an example file with a couple basic meshes and a capsule to represent the size of a regular player:
SDFGI_Leaking_Bug.zip
Here are some screenshots of the effects that may be observed:
The leftmost structure mimics the above example. From a distance, it appears that light is spilling in from the roof:

Upon closer inspection, the lighting adjusts and the space appears enclosed with some minor leaking in the left corner:

The rightmost structure is entirely enclosed. Although this improves the situation, there is still some noticeable leaking in the right corners. Some strange lighting artifacts will also appear on the walls as the camera travels through the space (not displayed here):

All shots taken with 8 cascades and a Y scale of 50%.

Check Use Occlusion in the environment SDFGI properties to reduce the amount of light leaking, and use thicker walls to make leaks even less noticeable.
Occlusion has a performance cost, which is why it's disabled by default.

Occlusion is ticked on in these examples and the walls of the meshes are appropriately thick for what could be expected in a game space that blends interior and exterior spaces. In the case of the rightmost structure, the walls are considerably thicker than what could be expected in structures in-game.

When using GIProbe, you can add additional geometry before baking GI then hide the geometry after baking, but it's not possible to do this with SDFGI.

That's all good and definitely what I expected. Since these effects aren't well-catalogued, I thought it best to take the time to make some examples. My hope is that if the issue is known and well-reproduced it will make future actions to address it a bit easier to identify.


Feeds don't autoupdate in Heroku

Hi everyone,
I've set up Miniflux on Heroku using the instructions here.
However, the feeds don't seem to auto update periodically in the background. Every day, I have to manually press "Refresh all feeds in background" to get the process started.
I appreciate your help. Thanks!

If you use the free version of Heroku, your instance sleeps after 30 minutes. You have to use the paid version.

@fguillot Oh I see. Thanks for the clarification. Is there anyway I can use something like a scheduler to automatically refresh it every morning?

You can probably write a script that prevent your instance to sleep if you really don't want to pay.
I'm closing this since it's related mostly to Heroku service.


Use convention plugin in generated java-library when split-project is used

Part-of #14599
Context
Currently java-library generated with --split-project --incubating use core java-library plugin instead of the generated convention plugin. java-application generated libraries use convention plugin as expected.
Contributor Checklist

[x] Review Contribution Guidelines
[x] Make sure that all commits are signed off to indicate that you agree to the terms of Developer Certificate of Origin.
[x] Make sure all contributed code can be distributed under the terms of the Apache License 2.0, e.g. the code was written by yourself or the original code is licensed under a license compatible to Apache License 2.0.
[x] Check "Allow edit from maintainers" option in pull request so that additional changes can be pushed by Gradle team
[ ] Provide integration tests (under <subproject>/src/integTest) to verify changes from a user perspective
[ ] Provide unit tests (under <subproject>/src/test) to verify logic
[ ] Update User Guide, DSL Reference, and Javadoc for public-facing changes
[x] Ensure that tests pass sanity check: ./gradlew sanityCheck
[x] Ensure that tests pass locally: ./gradlew <changed-subproject>:quickTest

Gradle Core Team Checklist

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation

https://scans.gradle.com/s/zjmh5cwy5vwow/tests/:build-logic:buildquality:test/gradlebuild.testcleanup.TestFilesCleanupServiceTest/leftover files are archived when test fails()?top-execution=1
It doesn't seem to be anythin related to the PR

Sorry for the late reply!
The DCO check is failing, please look into that as we can't accept the PR otherwise.

Please rebase the changes on the current master tip and add tests to verify the implementation. Know that the area has changed and this may no longer be relevant.


docs: Add new blog post 10 usecases of an API Gateway

Fixes: #[Add issue number here]
Changes:
A new blog post about the 10 most use cases of an API Gateway in API-Led architecture
Screenshots of the change:
A draft version of the content can be found on dev.to
https://dev.to/bobur/10-most-use-cases-of-an-api-gateway-in-api-led-architecture-3md9-temp-slug-4413613?preview=57dce358a48ae0224e2f3a3f6beae69e9bf277a6ad8272e7acc6bf161cdcb48546803e4ac3eee0afb14bd89dd1701de32059379a4638c379650cde5f

@SylviaBABY and @hf400159 I need one more round of approvals, please:)
Merged the master branch to get fixed build failure.


New dashboard: remember selected view setting and defaults

Description:
Related to #854 and #874
Dashboard will be enabled always for all users, the setting to enable or disable Dashboard in app settings will be removed.

Dashboard showing images enabled will be set as the new default view for all Ruuvi Station installations (both old and new users). When user adds a sensor he will be forwarded to full sensor settings view, then when he chooses back button he will land to full sensor card view. Dashboard will be accessed using the left arrow on full sensor card view.
Compact view or show images view should be remembered in Ruuvi Cloud.
User was on dashboard view, then killed app, reopening app will remember to open in dashboard view including compact view or images view that was selected previously. If user was on full card view the app will remember this view on reopening.

During meeting we've decided to not implement: If user was on full card view the app will remember this view on reopening.
We may return to this in future.

QA passed, discussing add new sensor feature in main card


[WIP] Stop hiding kubernetes topologies in the backend.

Fixes #1181
TODO:

[ ] hide in the frontend if empty (using the hide_if_empty flag)

A k8s report: k8s_report.json.zip

@tom we should be able to read topo.stats.edge_count to determine emptiness?
E.g.
const visibleTopologies = topos.filter(t => !t.hide_if_empty || t.stats.node_count > 0);

I would also look at t.stats.filtered_nodes

@davkal r4r.

@paulbellamy for backend review.

LGTM


🐞 [BUG] - items are not properly contained in the boxes

Issue Verification

[X] I have verified that this is a new bug report.
[X] I'm interested in working on this issue
[X] I'm part of GSSOC organization

Describe the bug
items are not properly contained in the boxes. Looks awful.

Expected behavior
No response
Steps to reproduce
No response
Additional Context
No response

You mean the text oeverflow?

Duuplicate issue


Add helm labels to secrets and service account

Closes #93

Should I bump the helm chart version?


Epub javascript interface on readium 2.3.0

Hello, sorry for being late after I first asked this in slack.
So, we have a js file which is responsible for extracting the text from the footnotes in epub and then deliver it to a function in kotlin to show a popup with the given text. After upgrading to version 2.3.0 and following the migration guide, this functionality is not working anymore, because our js files are not injected into the webview.  In our current version we were injecting the files via the server.loadCustomResource, but not anymore since it is deprecated. We are attaching screenshots and videos from our current version and the new one. I tested it on emulator and real device as well.
This is the functionality on our current version. As you can see "inject.js" and "jquery.js" files are added into the webview
https://github.com/readium/kotlin-toolkit/assets/109721963/f6a4411c-d9c6-4d51-a93b-35d337773bf5

This is on version 2.3.0
https://github.com/readium/kotlin-toolkit/assets/109721963/41b35ca6-4ba1-4775-88d9-676249d0f353

This is where we inject our js and register the javascript interface

Would be great if you can point us in the right direction and tell us if something wrong is done on our side.

Hmm you did nothing wrong, you need a feature called Injectable API that is not yet implemented in this new version.
The servedAssets property is only used to make some asset resources available to the web view with HTTP requests. But it doesn't inject any script in the content.
Until we come up with an easy-to-use Injectable API, you can do the injection manually by transforming the fetcher when creating the Streamer, e.g.:
val streamer = Streamer(
    context = context,
    onCreatePublication = {
        if (
            manifest.conformsTo(Publication.Profile.EPUB) &&
            manifest.metadata.presentation.layout == EpubLayout.REFLOWABLE
        ) {
            fetcher = TransformingFetcher(fetcher) { resource ->
                resource.injectScripts()
            }
        }
    }
)

private fun Resource.injectScripts(): Resource =
    TransformingResource(this) { bytes ->
        val link = link()
        if (!link.mediaType.isHtml) {
            return@TransformingResource bytes
        }

        var content = bytes.toString(link.mediaType.charset ?: Charsets.UTF_8).trim()
        val injectables = listOf(
            """<script type="text/javascript" src="https://readium/assets/inject.js"></script>"""
        )

        val headEndIndex = content.indexOf("</head>", 0, true)
        if (headEndIndex != -1) {
            content = StringBuilder(content)
                .insert(headEndIndex, "\n" + injectables.joinToString("\n") + "\n")
                .toString()
        }

        content.toByteArray()
    }

The assets you declare with servedAssets are available under the base URL https://readium/assets/.

I'm glad 👍


Optimizer

First pass at an API for these things, as requested by #263. @andrewtarzia  let me know if this works for you.


add several IAM resources, fix some others

Looks good, tests for me, but breaks microservice-vpc IF you upgrade the version in a few places related to this change, I know you're working on that


Feat: Add support for Swift 4.0

I am part of the Kitura team and we would like to update to the latest version of Stencil, however since we support Swift 4.0 and so are unable to use the latest version unless Stencil also supports Swift 4.0.
This pull requests adds support for Swift 4.0 by

adding the auto-generated functions for equatable
Using #if swift(>=4.1) to alternate between flatMap and compactMap
Using #if swift(>=4.1) to alternate the requirement for where T.IndexDistance == Int with collections

Swift 4.0.3 builds have also been added to travis to check for future Swift 4.0 is supported in future builds.
These changes allow Stencil to compile and run on Swift 4.0 while still compiling with no build warnings for Swift 4.1

I agree that we should define compactMap for older swift versions, and maybe we should do this in a _SwiftSupport.swift file, where we can collect all these backwards compat. fixes.
The == can also be implemented there.

Hey
Thank you for the quick review. As suggested I have moved all the changes into a single file. This extends Collection to so future compact maps and index should work for Swift 4.0 as well.

Note for those wondering: the file should be named _SwiftSupport so that it's the first to compile, otherwise Swift doesn't always find the extensions.


Project 4 vagueness of GO ID

in the process decoding input section  ".... each batch in target_data and concat the GO ID to the ..." perhaps GO should be replaced  with

Though we are a little late to respond, we suggest you add a PR for this if possible. Thanks :)

While I do applaud you for your diligence in clearing old issues, I don't think that I'm in the best position to create a PR for this issue anymore.


Add NMOL/L as acceptable unit for IODATE and UREA

Specific cruise in question that reports these data in this unit, which matches up with their cruise report/figures: 49NZ20191205

The existing UREA param has units of umol/kg. The usual volume variant would be umol/l. Can you confirm that the nmol is correct/expected? I will often start looking at the oceanographic literature for external verification of expected ranges/units for some parameter.

For IODATE: https://doi.org/10.1016/j.dsr2.2011.02.002

this was one of the linked references in the cruise report
this study was in the southern ocean (Atlantic sector) during the summer and found values of 400-450 nmol/L, this aligns with the magnitude of values found in this file. Although this transect had a much lower minimum value in the 100s.

For UREA: https://doi.org/10.1002/lom3.10026

figure 8 shows urea in nmol/L in the same order of magnitude found in this file
however, values in this file vary significantly more than in that paper (minimum = 1.4, maximum = 144.4 for the referenced file versus a range of 50ish nmol/L to 125 nmol/L in the paper), perhaps just a regional difference since the study was in the s. China Sea?

This one gets a partial for now, the IODATE change goes in as proposed. I did some checks on UREA and the data we have already.
All the existing data at CCHDO have UREA in umol/kg, with ranges comparable to what is here proposed here, just smaller:

Existing data is 0~0.5 UMOL/KG which is ≈ 0~500NMOL/L
Checking though all the existing params, we do not have any mixed SI prefix units for any given param, this type of alternate unit appears somewhat unprecedented at CCHDO. Even some of the literature talking about urea will mix/swap between umol and nmol, so there might not be consensus.

I'd like to ask our science advisors for... advice on this one. I have a desire to keep the prefix consistency, i.e. the final param is umol/l and we convert from nmol/l to umol/l so the only difference between our units is mass vs volume. Pointing out that the conversion between nmol/l and umol/l is exact.

Asked Jim and Sarah at the meting on August 23. Since this is an unambiguous conversion, the advise is to convert from NMOL to UMOL by simple division by 1000 (pure SI conversions).

@cberys Where would you like the above decision documented?

@DocOtak it should be mentioned in the monthly report and also make an issue on the operations manual repo to add it (or you can add it somewhere but that needs a major overhaul/revision so I suggest just making an issue)

Closing since IODATE was added and the UREA decision now exists in the ops manual as an issue and won't be added as a name.


Identify owning SIG

Workgroups are temporary time bounded groups, this project should specify the owning sig and be listed as a subproject of one of the SIGs in the metadata in github.com/kubernetes/community, you can identify the workgroup as a place to discuss the project but it cannot be the owner and we should sort that out sooner than later so we don't have confusion when the workgroup winds down.
cc @kubernetes/steering-committee

sig network or sig autoscaling might be a good home
added as. an agenda item for the wednesday meeting: https://docs.google.com/document/d/1aExJFtaLnO-TM6_2uILgI8NI0IjOm7FcwLABBKEMEo0/edit?tab=t.0#heading=h.zd57bzqasuuz

I think sig networking would be the most appropriate

We just need agreement from a SIG to be responsible for hosting the project long term and to setup pointers in sigs.yaml and the README, it sounds like it could be SIG net and that sounds like the closest fit to me personally.

@robscott fyi

FYI https://github.com/kubernetes/community/pull/8056, it was introduced under SIG Apps, actually.
I have a thread with the github management team about preventing this sort of confusion in the future by tweaking the request template to clearly ask for the sponsoring SIG for future repos to be one particular SIG.
In my case I saw the WG serving README PR and this repo's README but not the commit to SIG Apps, and both sigs listed in the request.
If you all are fine with SIG Apps being the sponsoring SIG, then lets just clarify the README here and in the wg-serving README/charter to make it clear that SIG Apps owns it and wg-serving is where it's being discussed.

I have a thread with the github management team about preventing this sort of confusion in the future by tweaking the request template to clearly ask for the sponsoring SIG for future repos to be one particular SIG.

[this part is done: https://github.com/kubernetes/org/pull/5266]

I am open to the suggestion to put this under SIG Network, especially given that it has close ties with Gateway API and deals with application level routing and load-balancing for LLM inference traffic. However given what @BenTheElder is saying, it sounds like we should discuss this at some length. Looking forward to further discussion on this.

ACK, from talking to a few folks, it seems like SIG-Apps is OK but SIG Net may be a better fit, when you all have a conclusion if we need to change things just send a PR to kubernetes/community and this repo and we can update things.
I just ask that even if we don't move it, please cleanup references in this repo and in kubernetes/community wg-serving that indicate that wg serving "owns" this as opposed to a place to discuss this project being sponsored by Apps or Network.
Sometimes workgroups last for many years but they eventually wind down and the distinction is important, we make sure things roll up to SIGs because of this and because SIGs have carefully reviewed charters to avoid conflict, while as we see here it is normal and expected for working groups and conceptual efforts to span many SIGs, as opposed to projects.
Thanks all! Let me know if there's anything I can do to help.

Filed one of the interim cleanup PRs at https://github.com/kubernetes/community/pull/8179

Based on @BenTheElder's comment https://github.com/kubernetes/community/pull/8179#discussion_r1853071389, Serving Catalog is not a project yet as it needs sponsorship from SIGs. cc @ahg-g @jjk-g

Looking at https://github.com/kubernetes/community/pull/7988 it seems, it landed under sig-arch, although looking at https://github.com/kubernetes/org/issues/5072 I only see folks from sig-scheduling acking that repo.

The wg-serving repo is a generic playground for the working group, it is not meant to be only for the serving catalog.

Based on @BenTheElder's comment https://github.com/kubernetes/community/pull/8179#discussion_r1853071389, Serving Catalog is not a project yet as it needs sponsorship from SIGs. cc @ahg-g @jjk-g

Thanks for flagging, it is fine if we remove it.

regarding:

I just ask that even if we don't move it, please cleanup references in this repo and in kubernetes/community wg-serving that indicate that wg serving "owns" this as opposed to a place to discuss this project being sponsored by Apps or Network.

Filed https://github.com/kubernetes-sigs/llm-instance-gateway/pull/52 to close that out:

I think this is reasonable? It allows WG-Serving to claim credit for launching this project, drives interest in the WG, but still clearly identifies the sponsoring SIG and the official subproject listing. I think the split into two sentences is more readable.

With https://github.com/kubernetes/community/pull/8211, it seems like we're all set :+1:
/close
Re-open if there's anything else here we need to figure out.

Oh, actually right after this I noticed the README needs updates about this, as well as the community documentation it links to, so maybe we can hold this one open to cover the work needed to update the relevant documentation regarding SIG Network's role with this.
/reopen

https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/92 Updated the readme.

Roger that, somehow I was looking at something out of date, perhaps my own fork. :+1:
/close

Thanks all.


Task: update copyright year at the beginning of 2024

TRG 7.00 requires that we update the copyright header of files to point to the new year. This should be taken care of shortly after 2024 begins.

Not required for all files, only files which were touched during the new year.


Create enaml-native server so the app can be "reloaded" without rebuilding

Should be able to do this by implementing an AbstractEnamlImporter

No need to do a special importer. Done via this:
https://github.com/frmdstryr/enaml-native/commit/be274922c649a6ef674de48d29bcfc22fbed88d8


Username is not used

Expected behavior
Use the username of the RedisProperties
Actual behavior
Even if the username is set, the RedissonAutoConfiguration ignore the username
Steps to reproduce or test case
I'm using the following dependency
        <dependency>
            <groupId>org.redisson</groupId>
            <artifactId>redisson-spring-boot-starter</artifactId>
            <version>3.17.6</version>
        </dependency>

Create a user 'my-username' on the server. Assume the user's password is my-password
spring:
  redis:
    host: localhost
    port: 6379
    username: my-username
    password: my-password

Redis version
7.0.4
Redisson version
3.17.6
Redisson configuration
No redisson configuration. Just redis

Fixed. Thanks for report


0.5.0-rc upgrade guide

Thank you your contribution. I'm happy using this lib :)
I'm planning to use 0.5.0 rc version.
So, all the relations I can use are these, right?
OneToOne -> HasOneRelation,
OneToMany -> HasManyRelation,
OneToOne -> BelongsToOneRelation,
ManyToManyRelation,
Could you explain some other features about 0.5.0?
Or, when will you release next version(document) ?
I want to know other features about next version 😄

The old OneToOne is now BelongsToOne and there previously was no equivalent for the HasOne.
Other new features in 0.5.0:

joinRelation method. This is just a shortcut for joining a related table and giving the joined table an alias based on on the relation name. For example

Person.query().joinRelation('children').where('children.age', '>', 10)

withSchema method now works as expected and passes the given schema to all executed queries.

filterEager method. This is a better syntax for filtering eager expressions.  Example:

Person
  .query()
  .eager('[children.movies, pets]')
  .filterEager('children.movies', builder => builder.where('releaseYear', '<', '1985')
  .filterEager('pets', builder => builder.where('species', 'dog')

Extra properties for ManyToMany relations. You can select which properties of the model are inserted to the join table instead of the related table. Example:

// In Person.js
...
relationMappings = {
  movies: {
    relation: Model.ManyToManyRelation,
    modelClass: Movie,
    join: {
      from: 'Person.id',
      through: {
        from: 'Movie_Person.personId',
        to: 'Movie_Person.movieId',
        extra: ['type'] // <--- NOTE THIS LINE
      },
      to: 'Movie.id'
    }
  }
}
...

// Now this query inserts a new movie for a person and sets the `type`
// column of the created join row in the join table to value `Actor`
person
  .$relatedQuery('movies')
  .insert({firstName: 'Foo', lastName: 'Bar', type: 'Actor'})

// This fetches the movies with the `extra` properties joined.
let movies = person.$relatedQuery('movies')
console.log(movies[0].type)

It is now possible to limit eager expression recursion like this: person.parent^3. That would only fetch three levels of ancestors for the person.

The 0.5.0 version will be released soon. I'm currently just cleaning up the code and writing more tests. I won't write any new features for it.

By the way, you can always find the up-to-date documentation frion the doc folder. That folder contains the markdown files that are used to generate the documentation website. They are pretty readable.

This info can now be found in the changelog. Closing.


Adds new section (Guide) to the docs

This PR adds a new section (called Guide) to the docs.
This is what it looks like currently from a local build.

The main idea here is to flesh out the docs to take them beyond just the API and example notebooks. So the key thing here is approachability.
The "benefits of PyMC-Marketing" section contains information from the release announcement. I figure that the information here is timeless, so we'd want it in the website rather prominently - it could easily get 'lost' in the Labs blog post in the future.
The Introduction to MMM section basically lifts the material I wrote for one of the PyMC Labs' blog posts. I think this makes sense because it's a pretty good (IMHO) intro to MMM's for the non-expert.
The corresponding CLV material is not yet written. But I think having a placeholder page there is fine - it makes things more visually balanced. The plan would be to follow up with intro CLV explanation content soon.
I've also embedded some relevant Labs members where relevant.
The idea is also that additional guide type pages could be added over time. Such material would serve a distinctly different purpose as the Examples which are much more hands on and code focussed. Material in the Guides section will be more discursive, with explanations on concepts, discussions about best practice, or non-technical explanations of particular models, for example.
TODO's based on feedback

[x] Clarify in all places that it can be a generic outcome metric, doesn't have to be new customers
[x] Get internal links to display properly!
[x] Add RTD preview link to PR via a GitHub action
[x] Add "Book a Call" which closes #217

Thanks for harmonizing the CLV intro with the MMM intro.  Hopefully this brings things up to the level needed for release!  I will take a quick look through the entire thing and then hopefully we can get it merged.

Do we want images in the CLV intro like those in the MMM intro?  I'm not sure how those images were created given their style nor do I know what sorts of images we would want to add to the CLV intro.  But I thought I would ask.

Maybe a couple of customers and their purchase-over-time behavior (sort of like this kind of thing) and then a couple of distributions describing the purchase magnitude and churn probability/hazard/survival?  Maybe @larryshamalama or @ColtAllen have better ideas?

Maybe a couple of customers and their purchase-over-time behavior (sort of like this kind of thing) and then a couple of distributions describing the purchase magnitude and churn probability/hazard/survival? Maybe @larryshamalama or @ColtAllen have better ideas?

@cluhmann can you open an issue with the ideas? Will merge this in the meantime


Add description of SDLC github labels

One of my action items from the June 20th meeting was to write up the github labels.
I realized in doing so that most of them were not used in our recent release of the API.  Given then, I am not sure if many of these are meaningful.

suggestion
gathering evidence: github issue submitted. must include evidence of business value; use the issue to gather additional use cases; could be in a google doc.
proposal under review: design proposal submitted.
validating through implementation: design proposal approved; PMC agrees which platform will be used to validate design through implementation. PMC approves proceeding with implementation to validate design.
reviewing PR: PR submitted
release-candidate: PR merged

@shalako for your suggestions, I'd suggest that validation through implementation should say PMC agrees which platforms.

@mattmcneeney updated

Some more queries:

At what point is an issue moved from gathering evidence to reviewing proposal? Who decides this?
Are we going to use any labels to track features moving into alpha before being merged into the main spec, or will that be done via a follow up PR with no labels?

Suggestion: let's scope this PR to documenting what is there, minus phases we agree to remove.  I will do the alpha features in that PR, unless people feel that we should just combine them.  Suggest we punt about 'gathering evidence' as it is a new to this PR can of worms and we have a big chunk we already agree on.
I'll incorporate the feedback in this PR during my next work session/

A suggestion for a potentially simpler process.
Downsides (IMO):

'alpha PRs' would hang around for a while, could lead to merge conflicts down the line

Upsides (IMO):

It's easy to see alpha features by just filtering PRs using the alpha tag
Only 4 stages

Please let me know your thoughts!
1. A new issue is created containing a link to a proposal (Google Doc) which outlines the motivation, evidence and other contextual information. If the proposal has enough information, the "reviewing proposal" label is set by a member of the PMC.

2. Once a proposal has been discussed and any necessary changes have been made, if the group is happy to proceed then a PR is created from the proposal to the master branch. The description of the PR should contain a link to the proposal and the original issue, and the diff will be to the spec on the master branch (making it easy to understand the changes). The "validation through implementation" label is added to both the issue and the PR. Conversations regarding technicalities/implementation should now be done on the PR rather than the issue.

3. Once the proposal has been implemented and any feedback has been taken into account, the PR should be labeled "alpha". This is equivalent to the current label "validated through implementation". The PR should not be updated from now on without discussion with the working group.

4. Once the group agree that enough feedback has been collected on the alpha feature, the PMC should move to LGTM and merge the PR (resolving any merge conflicts if needed). The original issue should be closed as assigned the label "merged".

The main problem I see with your proposal @mattmcneeney is that you can't get a consolidated view of all the alpha features. If two alpha features are on the same API endpoint, this could become troublesome.
For new features, not minor copy changes, why don't we:

Raise Issue with the problem and supporting evidence.
PMC review and apply label review-proposal
PMC LGTM if they think the issue should be solved in the API
PR is made to master with [alpha-#issue-id] prefix on each change in the spec
OSBAPI review PR and give LGTM
PR is merged, label validating-through-implementation is applied to issue.
Implementation feedback provided in issue and follwup PR's made
Once the feedback issues are resolved, open a PR to remove [alpha-#issue-id] tags, PMC LGTM.
Issue is moved into release-candiate.

This means all agreed changes will be in the spec as [alpha] giving a single consolidated view. It will avoid long running branches and the need to rebase. We also only have three labels to manage, as any more I feel is admin overhead we don't need.
We just need to writeup what alpha means, i.e. not guaranteed to be supported in any platform and may change at any time without warning.

@avade I think this is much clearer then the process today. Couple of queries:

How does an issue get the review-proposal label? Discussion on the weekly call? When can a PMC member apply this label to an issue?
What qualifies as "Once the feedback issues are resolved" in step 8? I think a bit more clarity on this is what would have really helped with the context field change.

@avade I don't like the idea that features are merged into the spec without implementation, even if they are alpha. This leads to astronaut architecture, and there's no platform for broker authors to test with. Why then, would broker authors invest any effort in testing alpha features?

Proposed labels:

GitHub Labels
When new design proposals for the API specification are submitted, the working group will use a set of pre-defined GitHub labels to highlight the current stage of the proposal. Note that these labels will not be used for minor changes.
The labels that will be used are as follows:

1 - reviewing proposal: The API working group is actively reviewing a proposal that has been submitted as a GitHub Issue with the aim of validating both the problem statements and any proposed solutions.
2 - validating through implementation: One of more platforms are actively working on the proposal with the aim of providing feedback on the implementation of the proposed solution.
3 - reviewing PR: Feedback has been received on the implementation of the proposed solution and a Pull Request has been created containing the validated specification changes.

Not all issues will require a validation through implementation stage, and proposals can move back to a previous label at any time. When a PR is merged, the associated issue should be closed and any labels removed from the issue.

Note this doesn't cover how we would tie this in with 'alpha' features etc. I guess if we go down that route then between labels 1 and 2, a PR should be submitted to the alpha doc. And label 3 should remove the feature from the alpha doc and add it to the main spec.

@mattmcneeney Sounds good to me. Let's update the PR.

Need @pmorie for that one

Following yesterday's call I've updated this PR. Thanks @pmorie for push access.
@shalako @avade @pmorie @vaikas-google @fmui @duglin @angarg12
Please review!

LGTM
According to this, the fact that there are currently open PRs in the validation through implementation step, means that we are not following the process properly? :smile:

LGTM
Simplicity FTW.

LGTM

One more pls (I'm not going to do it since I've now been updating the branch!)
@shalako @fmui @pmorie @vaikas-google

LGTM

On Aug 9, 2017, at 6:27 AM, Matt McNeeney<EMAIL_ADDRESS>wrote:
One more pls (I'm not going to do it since I've now been updating the branch!)
@shalako @fmui @pmorie @vaikas-google
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Updating the actual labels now


retry-on-snapshot-warnings - not working as expected on separate snapshot/review workflows

Using retry-on-snapshot-warnings for a submission from a different workflow as described in the docs. If the snapshot upload completes during the phase where the review task is waiting for an upload against the head SHA - none of the retries pick it up.  If you re-run the review workflow it picks up the newly committed snapshot.
On Push:

Submission Workflow - Submits dependency snapshot (takes 2m 3s to complete)

On PR:

Review Workflow - run 1 - Dependency review with retry and long timeout (Retry timeout exceeded. Proceeding... after 4m 37s)

Submission Workflow

after 2m submits its dependency snapshot for the commit 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f
logs_22.zip

Snapshot submission to     "sha": "63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f" +     "ref": "refs/heads/feature/FSharp-Data"
2023-12-04T18:10:54.1830002Z ##[notice]Submitting snapshot...
2023-12-04T18:10:54.1858756Z ##[notice]{
    "detector": {
        "name": "Component Detection",
        "version": "0.0.1",
        "url": "https://github.com/advanced-security/component-detection-dependency-submission-action"
    },
    "version": 0,
    "job": {
        "correlator": "dependency-submission",
        "id": "7090712486"
    },
    "sha": "63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f",
    "ref": "refs/heads/feature/FSharp-Data",

... and completes within a second at 2023-12-04T18:10:55.414Z:
2023-12-04T18:10:55.4988607Z ##[notice]Snapshot successfully created at 2023-12-04T18:10:55.414Z

Review Workflow

logs_23.zip

ref: refs/pull/8/merge
base ref: master
head ref: feature/FSharp-Data

2023-12-04T18:09:23.5277218Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:09:23.5277924Z Retrying in 10 seconds...
2023-12-04T18:09:33.8899944Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:09:33.8900873Z Retrying in 10 seconds...
2023-12-04T18:09:44.4259065Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:09:44.4259945Z Retrying in 10 seconds...
2023-12-04T18:09:54.7741812Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:09:54.7742670Z Retrying in 10 seconds...
2023-12-04T18:10:05.1531584Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:05.1533086Z Retrying in 10 seconds...
2023-12-04T18:10:15.4874900Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:15.4876137Z Retrying in 10 seconds...
2023-12-04T18:10:25.8345344Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:25.8346111Z Retrying in 10 seconds...
2023-12-04T18:10:36.1931593Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:36.1932274Z Retrying in 10 seconds...
2023-12-04T18:10:46.5429500Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:46.5430353Z Retrying in 10 seconds...

at this point the snapshot has been submitted

for the next 2m30s looking for snapshot but not finding it

2023-12-04T18:10:56.9246186Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:10:56.9246809Z Retrying in 10 seconds...
2023-12-04T18:11:07.2595829Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:07.2596661Z Retrying in 10 seconds...
2023-12-04T18:11:17.6323447Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:17.6324235Z Retrying in 10 seconds...
2023-12-04T18:11:27.9848187Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:27.9849254Z Retrying in 10 seconds...
2023-12-04T18:11:38.3612135Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:38.3612905Z Retrying in 10 seconds...
2023-12-04T18:11:48.7582440Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:48.7583606Z Retrying in 10 seconds...
2023-12-04T18:11:59.1062944Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:11:59.1063681Z Retrying in 10 seconds...
2023-12-04T18:12:09.4695845Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:09.4696654Z Retrying in 10 seconds...
2023-12-04T18:12:19.8232333Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:19.8232973Z Retrying in 10 seconds...
2023-12-04T18:12:30.1726324Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:30.1727051Z Retrying in 10 seconds...
2023-12-04T18:12:40.4742149Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:40.4744475Z Retrying in 10 seconds...
2023-12-04T18:12:50.7921268Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:12:50.7921899Z Retrying in 10 seconds...
2023-12-04T18:13:01.1266767Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:13:01.1267423Z Retrying in 10 seconds...
2023-12-04T18:13:11.4858149Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:13:11.4858916Z Retrying in 10 seconds...
2023-12-04T18:13:21.8385364Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:13:21.8386034Z Retrying in 10 seconds...
2023-12-04T18:13:32.1692568Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:13:32.1693442Z Retry timeout exceeded. Proceeding...
2023-12-04T18:13:32.6080649Z Dependency review did not detect any denied packages

Review Workflow - run 2
Most interesting is that re-running dependency review task at any point in the future succeeds after 2 tries (doesnt mention a snapshot found but looking at the detections it has found dependencies that only exist in the snapshot manifest):

Review Workflow - run2

logs_24.zip

ref: refs/pull/8/merge
base ref: master
head ref: feature/FSharp-Data

2023-12-04T18:30:08.1275306Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:30:08.1276356Z Retrying in 10 seconds...
2023-12-04T18:30:18.3518175Z No snapshots were found for the head SHA 63d50c7154fc8bfb6ce9173f0d0edfe5f31d810f.
2023-12-04T18:30:18.3518906Z Retrying in 10 seconds...
2023-12-04T18:30:30.0878871Z ##[debug]Filtered Changes: [{"change_type":"added","manifest":"Fsharp-WebAPI.fsproj","ecosystem":"nuget","name":"FSharp.Data","version":"6.3.0","package_url":"pkg:nuget/FSharp.Data@6.3.0","license":null,"source_repository_url":"https://github.com/fsprojects/FSharp.Data","scope":"runtime","vulnerabilities":[]},{"change_type":"added","manifest":"/Fsharp-WebAPI.fsproj","ecosystem":"nuget","name":"FSharp.Data","version":"6.3.0","package_url":"pkg:nuget/FSharp.Data@6.3.0","license":null,"source_repository_url":"https://github.com/fsprojects/FSharp.Data","scope":"runtime","vulnerabilities":[]},{"change_type":"added","manifest":"/Fsharp-

Hi @felickz, I really appreciate the reproduction and detailed logs! Much of this does sound very strange to me, but one thing sticks out as especially strange:

Most interesting is that re-running dependency review task at any point in the future succeeds after 2 tries

Can you clarify that this applies truly at any point? In other words, if you wait for a very long time (like an hour), does it still retry at least once before succeeding?

Hi @felickz, I really appreciate the reproduction and detailed logs! Much of this does sound very strange to me, but one thing sticks out as especially strange:

Most interesting is that re-running dependency review task at any point in the future succeeds after 2 tries

Can you clarify that this applies truly at any point? In other words, if you wait for a very long time (like an hour), does it still retry at least once before succeeding?

In the sample is showed - it was @ 15 mins:

2023-12-04T18:13:32.1693442Z Retry timeout exceeded. Proceeding...
Re-run

2023-12-04T18:30:30.0878871Z - retry succeed

In a previous repro - i had run another time with 5 minutes between with similar behavior (initial run):

2023-12-04T17:59:17.1762701Z Retry timeout exceeded. Proceeding...
Re-Run

2023-12-04T18:04:56.0137937Z Dependency review did not detect any denied packages

@febuiles I don't have a ton of time to devote to this and nothing's really jumping out at me, but I do have a vague suspicion that this may be related to https://github.com/github/dependency-snapshots-api/pull/615. I thought that would only change the way we handle new canonical snapshots and that shouldn't matter here, but if there's some reason why it's taking a while for new snapshots to be written to the ds_snapshots table then that could explain what's going on here.

@juxtin thanks for the extra feedback 🙇
@jamisonhyatt any ideas if the locks-related change could be having an impact downstream like this? Does any of this look suspicious to you?

I have another oddity where it seems submission actions like anchore/sbom-action are using a merge commit when they upload a snapshot when running as an on pull_request trigger:
  ##[debug]  "eventName": "pull_request",
  ##[debug]  "sha": "6f50a7568be909f93a78e164a90362a365f92a44",
  ##[debug]  "ref": "refs/pull/22/merge",
  ##[debug]  "workflow": "Syft SBOM Action",
  ##[debug]  "action": "__anchore_sbom-action_2",
  ##[debug]  "actor": "felickz",
  ##[debug]  "job": "buildAndUpload",
  ##[debug]  "runNumber": 214,
  ##[debug]  "runId":<PHONE_NUMBER>,

Then the DR action (with on pull_reqeust trigger) is looking for sha of the latest commit in the HEAD branch.
Leading to:  No snapshots were found for the head SHA a9e00023489e612bb5cfbc81ea97202a30af124e.
Is this something  the toolkit is intended to handle?
I am not totally clear what this sha is showing me, this might be a temporary hidden merge branch that is used to check "This branch has no conflicts with the base branch"?   Would it make any sense for DR to be also looking at this SHA for snapshot submissions?

Need to do research if the issue is with the backend service

I also encountered this problem, only I'm using the gradle/actions/dependency-submission action to submit the dependency snapshot.
With an added known vulnerable dependency I could re-run the review workflow at least 3 times all ending in a timeout and no vulnerabilites reported. The timeout was set to 10 minutes and the three times were run consecutively ...
Then, after waiting for maybe 30+ minutes (searching for answers and coming back for debug-logs), the re-runs suddenly started consistently reporting a "vulnerable dependency detected". It reminded me of how replication lag behaves as the snapshot propagates across data stores ..
I couldn't find an option for failing the build on timeouts, and with the green checkmark on the PR I doubt many people will study the logs to see if it all ran as they expected.


Added norwegian translation

Checklist
Common

[x] I am ran the app before creating PR
[x] I am ran all tests before creating PR

UI

[x] I am ran the app for visual checks.

Logic

[x] I am tested basic app functionality.

Changes
Describe all changes here:

Added Norwegian translation

Comments

Added Norwegian translation

Done. Please update version number if this will not be included in 3.6.9.


Link updates

Docs Build status updates of commit 01035c9:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/quickstart/create-and-publish-a-package-using-the-dotnet-cli.md
:white_check_mark:Succeeded
View

docs/quickstart/create-and-publish-a-package-using-visual-studio-net-framework.md
:white_check_mark:Succeeded
View

docs/quickstart/install-and-use-a-package-in-visual-studio.md
:white_check_mark:Succeeded
View

docs/quickstart/install-and-use-a-package-using-the-dotnet-cli.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 0371f36:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/install-nuget-client-tools.md
:white_check_mark:Succeeded
View

docs/quickstart/create-and-publish-a-package-using-the-dotnet-cli.md
:white_check_mark:Succeeded
View

docs/quickstart/create-and-publish-a-package-using-visual-studio-net-framework.md
:white_check_mark:Succeeded
View

docs/quickstart/create-and-publish-a-package-using-visual-studio.md
:white_check_mark:Succeeded
View

docs/quickstart/install-and-use-a-package-in-visual-studio.md
:white_check_mark:Succeeded
View

docs/quickstart/install-and-use-a-package-using-the-dotnet-cli.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[Request] Add search for builds

Searching for builds would be much easier or better bringing the old buildbot back would be the best because navigation on that is horrible.

Can you make one build compatible with lg astrio because when I try to install it on to my phone it said not compatible.


Error occurred when executing DownloadAndLoadFlorence2Model:

System Info
CUDA Setup failed despite GPU being available. Please run the following command to get more information:
python -m bitsandbytes
Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
Reproduction
File "E:\Comfy UI\ComfyUI-aki-v1.3\execution.py", line 316, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "E:\Comfy UI\ComfyUI-aki-v1.3\execution.py", line 191, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "E:\Comfy UI\ComfyUI-aki-v1.3\execution.py", line 168, in _map_node_over_list
process_inputs(input_dict, i)
File "E:\Comfy UI\ComfyUI-aki-v1.3\execution.py", line 157, in process_inputs
results.append(getattr(obj, func)(**inputs))
File "E:\Comfy UI\ComfyUI-aki-v1.3\custom_nodes\ComfyUI-Florence2\nodes.py", line 96, in loadmodel
model = PeftModel.from_pretrained(model, adapter_name, trust_remote_code=True)
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\peft\peft_model.py", line 429, in from_pretrained
model = MODEL_TYPE_TO_PEFT_MODEL_MAPPING[config.task_type](model, config, adapter_name)
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\peft\peft_model.py", line 1395, in init
super().init(model, peft_config, adapter_name)
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\peft\peft_model.py", line 138, in init
self.base_model = cls(model, {adapter_name: peft_config}, adapter_name)
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\peft\tuners\lora\model.py", line 139, in init
super().init(model, config, adapter_name)
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\peft\tuners\tuners_utils.py", line 166, in init
self.inject_adapter(self.model, adapter_name)
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\peft\tuners\tuners_utils.py", line 372, in inject_adapter
self.create_and_replace(peft_config, adapter_name, target, target_name, parent, current_key=key)
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\peft\tuners\lora\model.py", line 223, in create_and_replace
new_module = self.create_new_module(lora_config, adapter_name, target, **kwargs)
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\peft\tuners\lora\model.py", line 291, in create_new_module
from .bnb import dispatch_bnb_8bit
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\peft\tuners\lora\bnb.py", line 19, in
import bitsandbytes as bnb
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\bitsandbytes_init.py", line 6, in
from . import cuda_setup, utils, research
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\bitsandbytes\research_init.py", line 1, in
from . import nn
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\bitsandbytes\research\nn_init.py", line 1, in
from .modules import LinearFP8Mixed, LinearFP8Global
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in
from bitsandbytes.optim import GlobalOptimManager
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\bitsandbytes\optim_init.py", line 6, in
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "E:\Comfy UI\ComfyUI-aki-v1.3\python\lib\site-packages\bitsandbytes\cextension.py", line 20, in
raise RuntimeError('''
Expected behavior
Improvements to address such issues

I solved it by
pip list
bitsandbytes    0.43.3


Minor tweaks to the ScalaPresentationCompiler

Two minor tweaks to the ScalaPresentationCompiler.

retest this please

Looks good, thanks!


Issue 7281 on CodePlex: Unable to receive SNMPv3 traps

*Makes authenticated SNMPv3 TrapV2Pdus pass instead of insisting on Reportable message header flag.
*Modified IsInTime(...) method for timeliness check of received SNMPv3 messages.

Should simply let TRAP v2 messages pass authentication. Closed this PR and use c55d7a46eec74bf78182b8085af7f6e6317374e0 instead.


Create Sequence Diagram(s)

I have created and committed version 1 diagrams

i created and committed version 1 diagrams for start screen to circuit world


top-bar-right offset?

How to reproduce this bug:

Go to http://foundation.zurb.com/templates-previews-sites-f6/product-page.html
Hit F12 and replace the .min.css in head with Version 6.3.0. (https://cdnjs.cloudflare.com/ajax/libs/foundation/6.3.0/css/foundation.min.css)
As you can see the top-bar where the "Search" Button is jumps slightly to the right.

What should happen:
Stay at the same position?
What happened instead:
It positions itself slightly to the right.
Is this a bug or some codebreaking change in Version 6.3.0?
What do I have to change in my project? This is very ugly.
Thanks!

@rafibomb can you take a look at this?

Anything?

@canterberrie can you take a look?

@Mellnik I can reproduce and have attached 2 screenshots for reference.
ORIGINAL:

WITH 6.3.0 CSS SWAPPED-IN:

You can certainly tell there is different CSS providing the rounded-corner search box styling in 6.3.0. But that makes sense because we’re talking about 2, completely different CSS files:
ORIGINAL CSS:
https://dhbhdrzi4tiry.cloudfront.net/cdn/sites/foundation.min.css
6.3.0 CSS:
https://cdnjs.cloudflare.com/ajax/libs/foundation/6.3.0/css/foundation.min.css
I have a hunch that you may also need to swap-in a matching 6.3.0 JS file from the CDN to have an accurate test. I also didn’t have time to do further testing but wanted to get these screenshots posted sooner than later.
Speaking of testing... @Mellnik: What are you trying to accomplish? Upgrade to 6.3.0 from an earlier release?
Cheers!

https://dhbhdrzi4tiry.cloudfront.net/cdn/sites/foundation.min.css

This uses a very old normalize.css version.
We support only the latest releases of Foundation Sites.
I think we can close this as this is probably fixed.
6.5.3 is the latest release.


Smart lockout feasibility with  ADFS authentication method in Azure AD connect.

We have ADFS authentication method in Azure AD connect. In that case can implement Azure smart lock out functionality to configure account threshold and all in Azure for our on prem user.
Reference:
https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-password-smart-lockout

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9c7b90ca-a2b1-9cd8-9769-cd48c27f10ea
Version Independent ID: b8c29389-c435-b2f0-4186-1e9c47ab4f3e
Content: Preventing attacks using smart lockout - Azure Active Directory
Content Source: articles/active-directory/authentication/howto-password-smart-lockout.md
Service: active-directory
Sub-service: authentication
GitHub Login: @iainfoulds
Microsoft Alias: iainfou

@AmitavaHazra , We are looking into this query and will update you as soon as possible.

@AmitavaHazra Please refer to below section in the document. For federated deployments, you need to configure Extranet Smart Lockout (ESL) in ADFS. This document is for managed users or the users who leverage Pass-Through Authentication (PTA) and not for federated users.

Federated deployments using AD FS 2016 and AF FS 2019 can enable similar benefits using AD FS Extranet Lockout and Extranet Smart Lockout.

Thanks @amanmcse
Can you please have a look on the below query if you can answer.
https://github.com/MicrosoftDocs/azure-docs/issues/56431


Fixed sidebar/navbar

Prepare css class to make sidebar/navbar either fixed or static
Make it possible to control the state of navbar/sidebar type via vuex

Quick and dirty styles might be found in #116

awaiting !! :)

Hi smartapant, I followed your method in #116 , but the sidebar is not fixed and still movable. The navbar is fixed.

@smartapant , then this feature is not possible to go live after this weekend? I see the milestone is scheduled for April 30.

hi @harishankards!
We've included the task in the current milestone, will be done by the end of April

Thanks for the update @smartapant!

@asvae fixed sidebar and navbar issue? How do I replicate it in my project?

@harishankards merge from remote : 3

@asvae I have changed the remote in my project. Only there is a dependency of vuestic. How do I handle in this case?

@harishankards you can have several remotes no problem : 3.


Input Norms and Channel Order for EfficientNet

📚 The doc issue
The documentation for all pretrained models lacks clear details regarding the order of color channels for input images, as well as the specific normalization mean and standard deviation values. I am particularly looking for this information in relation to the EfficientNet model.
Suggest a potential alternative/fix
No response

I assume it is RGB order, and
mean = (0.485, 0.456, 0.406)
std = (0.229, 0.224, 0.225)

Hi @ivanstepanovftw , yes the number of channels is RGB, it's a widely adopted convention and applies everywhere in torchvision.
You're correct about the mean and std, this is documented on each model builder's page, e.g. for efficientnet_b0: https://pytorch.org/vision/main/models/generated/torchvision.models.efficientnet_b0.html#torchvision.models.EfficientNet_B0_Weights
Note that you would be able to find that in the .transforms() attribute of the weights as well (EfficientNet_B0_Weights.IMAGENET1K_V1.transforms) : https://pytorch.org/vision/main/models.html#initializing-pre-trained-models


[FLINK-7143] [kafka] Forward ports of new Kafka tests to master

This PR forward ports all new tests added in #4357 to master, so that the behaviors is correctly guarded there also.
Changes

Introduce KafkaTopicPartitionAssigner class to master branch, which strictly defines the partition assignment contract (discussed in #4301).
Port rescaling unit test. Note that some tested behaviors needed to be changed due to the differences between 1.3 and 1.4 for the Kafka consumer.
Make checkpoint methods final.
(new change, not a port) Remove invalid checkRestoredNullCheckpointWhenFetcherNotReady test, which was testing a legacy behavior of the consumer

Merging this now, as the changes were already reviewed when applying them onto release-1.3.


feat: use Immutables for Argument subclasses

allows for equality comparisons of Argument subclasses
BREAKING CHANGE: EnumArgument, TypeArgument and ValueArgument are now abstract

I was trying to compare plans and noticed that the Argument subclasses didn't implement equals. Updated them to use Immutable like everything else in the file uses.
Determined required fields based on:
https://github.com/substrait-io/substrait/blob/f2bfe15585943a137fafa560401e0cf0266c0650/text/simple_extensions_schema.yaml#L91-L123


Better musical score and graph drawing control for registers via wire_symbol

change the wire_symbol method to explicitly take a Register and idx: Tuple[int, ...] rather than one Soquet since the symbol shouldn't depend on the binst.
Eliminate or rename or merge short_name to explicitly compliment the wire symbols. This string should serve (solely) as a "title" to be written above the wire symbols and should be included if and only if the wire symbols don't already imply the gate. E.g. it's not necessary to write "CNOT" over something that's clearly a CNOT. Practically: we can rename the existing method and let it return None or we can use a sentinel value to the wire_symbol method (e.g. reg=None) to imply that we're asking for the (optional) title.
Single source of truth for drawing: modify the GraphvizDrawer for drawing composite bloq graphs to use the wire_symbol method and values to label the soquet rows in the table for each bloq. Use the title method for the title, and again omit if not required. Practically: we'll need a new method on WireSymbol to return a textual equivalent of the symbol

For the third point, since we don't actually draw the symbols in the graphviz drawer: I propose instead to follow #791 where we should have meaningful, concise __str__ methods on all the bloqs and use that

Ok, I just actually read the issue that I linked (and wrote) and it seems in all these places I seemed pretty convinced that the diagrams should use the wire_symbols for composite bloq diagrams (graphviz and musical_score). Let's try that. We can move the graphviz drawer to use str() later if it's too ambiguous otherwise

This has been completed.  See above PRs for details.


Mme dep removal master

EDGEPOD-103 : SGW should not take MME address as config parameter
Removing the Mme address configuration from the SPGW code.
SPGW can get to know about MME address when any UE attaches to the network.

it feature is already done in tmopl repo.


New version: Displaz v1.0.2

Registering package: Displaz
Repository: https://github.com/c42f/Displaz.jl
Created by: @c42f
Version: v1.0.2
Commit: e4dbc6a4a358a476f9531e17fa0cabc05c286dd5
Reviewed by: @c42f
Reference: https://github.com/c42f/Displaz.jl/commit/e4dbc6a4a358a476f9531e17fa0cabc05c286dd5#commitcomment-41806160

Looks like this very old package never got a 1.0.1 release; I will fix the project config and make a 1.0.1 release instead.
I think this PR can be closed.


Introduce StringBuilderPool

Motivation:
A lot of StringBuilders are allocated on the hotpath, in particular in utils used Lagarto.
Modifications:

StringBuilderPool uses a ConcurrentLinkedQueue pool instead of a ThreadLocal so that containers don't complain about leaks on classloader reboot. Also, this strategy supports re-entrant calls.
Offered StringBuilders length is capped to avoid large resident memory usage.
Usage is a borrow pattern where user has to pass a Function<StringBuilder, T>.

Result:
Util to pool StringBuilder instances

@igr I don't have time to run benchmarks atm, sorry. Feel free to run some on your side or wait for me before merging.

I will wait with this a bit, some tests are failing. Will be moved to new jodd-util repo.

@igr Could you please point to the failing tests?

 I commented on your PR.

It works! I am adding a simple JMH test and will release

Mmm, I see that StringUtils#remove can be implemented in a more efficient fashion. Please let me send a PR before benchmarking and releasing.
Also, please note I only used StringBuilderPool on the methods that are used in Lagarto. If experiment is successful, you'll probably want to update lots of code in Jodd (eg StringUtils is full for allocated StringBuilders).

Yes yes, just want to do it a step by step. The first release is for the Lagarto, and then I will slowly apply furthermore.


logout with expired token fails

If i logout in console with an expired token i get an error id_token_hint invalid as response from accounts.

could only find the following in the session management specs:

RP-Initiated Logout (https://openid.net/specs/openid-connect-session-1_0.html#RPLogout)

id_token_hint
RECOMMENDED. Previously issued ID Token passed to the logout endpoint as a hint about the End-User's current authenticated session with the Client. This is used as an indication of the identity of the End-User that the RP is requesting be logged out by the OP. The OP need not be listed as an audience of the ID Token when it is used as an id_token_hint value.

Security Considerations (https://openid.net/specs/openid-connect-session-1_0.html#Security)

The id_token_hint parameter to a logout request can be used to determine which RP initiated the logout request. Logout requests without a valid id_token_hint value are a potential means of denial of service; therefore, OPs may want to require explicit user confirmation before acting upon them.

@fforootd do you know if you should be able to log out with an expired id_token?

I think we should accept the expired id_token.
We still need to validate the users state with the cookie placed in the user-agent.
So it might be a security benefit to check that this id_token (even when expired) was issued to that specific user-agent.
This could IMO be done by storing a hash or id inside the token to bind the cookie to the token.
@livio-a what do you think of this?


add a parameter to control whether to refine the plan

Is your feature request related to a problem? Please describe.
Some tasks are easy and can usually be finished in 1 or 2 steps. In this case, a refining process is unnecessary and calling LLMs may waste a lot of time (especially with large model).
Describe the solution you'd like
Add a parameter like disable_refine_plan which force the TaskWeaver not to refine the plan.
Describe alternatives you've considered
None.
Additional context
None.

Hi, @BarryRun! We suggest you using planner.skip_planning to enable the simpliest version of Planner, only including sending request to CodeInterpreter and reporting reuslts to user. If you need to customized your planner prompt, you can modify the dummy_plan.json. More deatils can be found in planner doc.

Thanks for your reply @zhangxu0307! If I don't want to skip the planning but disable the plan refining, what should I do? Hopping for your replay.

If your task is simple and can be completed in a few steps, the Code Interpreter can take responsibility for creating a basic plan in its thoughts and generating the corresponding code. This means that you can use the planner.skip_planning directly.


Investigate performance as database grows

Is your feature request related to a problem? Please describe.
Load large volumes of data and track how ingestion performance varies over time.
Asses how access and search performance varies.
Understand scaling, such as memory and disk requirements for a large volume of data.
Use the new fhir-bucket project to load resources stored in COS.

A few months back I ran a quick test to measure the throughput of the IBM FHIR server over time. Big disclaimer: these results should be taken with a grain of salt since they were mostly used as a quick benchmark, but here's the throughput in FHIR resources/s for a total of 6 million resources:

The resources were sent to the server as update-as-create Bundles each containing between 10 and 100 resources (each bundle contained one Encounter and some associated Conditions, Procedures, Observations). The left side shows the throughput using a Postgres 12.3 with default options, the right one with the attached, optimized postgres.conf.txt active. DB, server, and client were run on the same VM with 16GB of RAM and 8 vCPUs. If I recall correctly, the CPUs weren't entirely saturated. The containers were reset between the runs.
The sender/client is a Spring Boot Batch app reading data from a DB, mapping it to FHIR Bundles and sending them via REST, using 8 threads in parallel. When sending the resources to S3 (minio) the throughput was around 10.000 resources/s.
I ran the same tests against a HAPI FHIR server (v4, R4) and averaged 800 resources/s, but it remained more consistent over time.
Definitely interested in some sort of "FHIRBench" which can benchmark servers in various scenarios:

single-resource posts
multi-resource bundle posts
conditional creates
conditional updates
... the same for reading
bulk im/export
horizontal scalability (do YugaByte/CockroachDB help)

(each one multi-threaded, at various fill-stages of the DB, etc.)
jvm.options.txt
postgres.conf.txt
docker-compose.yml.txt

@chgl thank you for this - really useful. There's a new project called fhir-bucket (not yet merged to master) which periodically scans S3 buckets looking for new files (NDJSON and JSON) to load and is designed to do so with a lot of parallelism (loading activity is coordinated across several instances). This covers bulk-load type scenarios. The FHIRBench scenarios you described are also being discussed, and contributions are welcome if you are able to.

@chgl also, we recently introduced a change to our logical id generation to force right-hand index inserts. This change is included in our 4.3.0 release. This particularly benefits PostgreSQL data sources because it reduces the number of blocks being modified and therefore the number of full page writes being logged, which otherwise gets worse as the database grows.

Its probably also worth noting that we've made a lot of performance updates since 4.3.0 as we started testing in earnest. I'd be really curious to see how 4.3.3 compares to what you saw on 4.3.0.
@punktilious can go into the details, but here are some highlights from the release notes:

DISTINCT in subquery prevents Db2 from selecting optimal execution plan
Added index for _lastUpdated
Use larger cache for identity sequences in Db2
Improve performance of search requests using multiple filters that have large differences in cardinality
Use encoded timestamp prefix for server-assigned Resource ids to reduce random reads during ingestion
Values Table have duplicate data for _id and _lastUpdated #1388

A couple of those were db2-specific, but most will benefit PostgreSQL as well.

Oops, I should have refreshed before submitting that :-)

Thanks for the information! I've seen the performance improvements in the release notes and am curious as well :) - I will hopefully find some time in the next two weeks.

Some update running on v4.3.3: I've played around with the memory limits a bit more because I believe the drops in throughput are caused by either swapping or GC or both - note that these runs are executed on just 8GB of RAM. It's still not optimized enough yet, but I've found another interesting discovery: the job I use supports sending either transactions with conditional updates or update-as-create bundles.

The left shows the throughput when using transactions, the right when using update-as-create. I assumed update-as-create would generally achieve higher throughput as it requires less processing. Do you have any insights/recommendations here?
CPU and RAM stats:

@chgl I'm actively looking into performance right now so hopefully will get a chance to take a look at what you're showing here. Are you running with openjdk? JDK Mission Control is pretty useful looking for GC pauses. Add something like this to the jvm.options file in your WLPHOME/usr/servers/fhir-server directory:
-XX:StartFlightRecording=disk=true,dumponexit=true,filename=run1.jfr,settings=profile,path-to-gc-roots=true
-XX:FlightRecorderOptions=stackdepth=96

Then you can load the run1.jfr file with JDK Mission Control.
My personal suspicion is that the dips could be related to database checkpoints. If you have the database files and WAL on different mount points then it's easy to tell looking at iostat if you get a surge in writes to the database files with a drop in writes to the WAL. I'll cook up some queries to try and glean more info from the db stats tables.

With over 70 million resources resident in the target database (PostgreSQL 12), a bulk load of resources using the fhir-bucket project was able to ingest over 700 resources/second using a concurrency of 100 threads processing bundles containing up to 100 resources each. We are currently planning a refactor of the part of the schema used to store references and this should improve scaling by reducing the size of our indexes significantly.


Stepper does not work on 3 page deep level

First of all, I could not provide jsfiddle sample as it is big and involdes multiple pages. So I am trying to describe in as much details as possible.
Version: Framework7 3.1.1
Platform: Browser and Android
I have the app structure like this.
Home > Restaurant page > Menu page
In Restarurant page, I show menus and steppers to add items to cart.
I initialize the steppers as below:
for(var i = 0; i < menus.length; i++) {
    $$("#menus").append('

<input class="stepperval stepperval'+ i +'" type="number" readonly value=0>

';
    app.stepper.create({  el: $$(".stepper" + i) });
}

In Restaurant page, when clicked on Menu name, the Menu page opens.
In Menu page, I have link to Restaurant page again.
Now, when I navigate as Home > Restaurant then the steppers works perfect.
But if I navigate as Home > Restaurant > Menu > Restaurant then the stepper does not work.
What I have analyzed.
In the second navigation, assignment only works when I write like this:
$$("body #restaurantaddress").html('My restaurant');
-- adding the body element is important here. Otherwise the html assignment does not work.
The same does not work for steppers.
If I write
app.stepper.create({  el: $$("body .stepper" + i) });
then no help.
Please let me know how to get this work.
Also let me know if you need any other informaton.
Screenshots for reference:

There is something wrong with your selectors when you create Steppers, just make sure you call it on every page withing the page scope, or they all have unique selectors


make sql db wait for connections instead of returning 'too many connections'

we need to probably figure out how to make this stop across all the various backends, we would prefer for them to wait and then possibly get hit with a timeout instead of just failing.
see #125 for error (now masked to user, still happening)
wrt mysql, it is mysql returning this error (not the database/sql package). see https://github.com/golang/go/issues/4805 for context on the pool, this package will wait for a free connection & we are using this feature. we need a better way to sync our local max_conns with the mysql in use's max_conns as these 2 can be different numbers and is the source of this error.

this was mostly related to the mysql bindings, we're locked on an old version until further notice, so closing since this is irrelevant


Adding plugin to project fails to install NPM dependency

Problem
pnpm tauri add process --verbose
> tauri "add" "process" "--verbose"
    Info [tauri_cli::helpers::cargo] Installing Cargo dependency "tauri-plugin-process"...
    Info [tauri_cli::helpers::npm] Installing NPM dependency "<EMAIL_ADDRESS>    Error [tauri_cli_node] Failed to install NPM dependency
 ELIFECYCLE  Command failed with exit code 1.

Reproduce
The issue is reproducable on a dummy project:
pnpm create tauri-app --rc
cd tauri-app
npm install
npm run tauri dev

Hypothesis
The script attempts to install<EMAIL_ADDRESS>Version 2.0.0-rc doesn't exist, but 2.0.0-rc.0 does (npm versions).

https://github.com/tauri-apps/tauri/pull/10795 doesn't work for npm at least.

What's the problematic npm and pnpm versions did you test with?
I ran npm add "<EMAIL_ADDRESS>with npm and pnpm and both worked fine

I actually can confirm that the latest version of cli works fine. I forgot that cli is a cargo package and not npm. So updating cargo (cargo update) in /src-tauri, and then attempting to add plugins works now.

@amrbashir

What's the problematic npm and pnpm versions did you test with?

Latest pnpm (9.10) and whatever comes with lts node for npm (10.8.2). Latest tauri cli rc (rc.16).
pnpm tauri add sql:
...
      Adding wry v0.43.1 (latest: v0.44.0)
    Info Installing NPM dependency "<EMAIL_ADDRESS> ERR_PNPM_NO_MATCHING_VERSION  No matching version found for<EMAIL_ADDRESS>
This error happened while installing a direct dependency of A:\fabianlars\temp\add

The latest release of @tauri-apps/plugin-sql is "2.0.0-rc.1".

If you need the full list of all 17 published versions run "$ pnpm view @tauri-apps/plugin-sql versions".
Progress: resolved 5, reused 5, downloaded 0, added 0
Failed to install NPM dependency
    Error Failed to install NPM dependency
 ELIFECYCLE  Command failed with exit code 1.

^ same for<EMAIL_ADDRESS>and<EMAIL_ADDRESS>btw - same for the process plugin mentioned above.
npm run tauri add sql:
...
      Adding wry v0.43.1 (latest: v0.44.0)
    Info Installing NPM dependency "<EMAIL_ADDRESS>npm error code ETARGET
npm error notarget No matching version found for<EMAIL_ADDRESS>npm error notarget In most cases you or one of your dependencies are requesting
npm error notarget a package version that doesn't exist.
npm error A complete log of this run can be found in: A:\_packages\npm\_logs\2024-09-18T10_17_55_748Z-debug-0.log
Failed to install NPM dependency
    Error Failed to install NPM dependency

I also had similar reports on discord so if i am using the command wrong, i'm not the only one at least.

I'm using tauri-cli 2.0.0 with pnpm 9.12.0 and this still doesn't work. I get this:

Should this be a different issue?

Nvm, my problem was my usage of pnpm workspaces and also installing dependencies in workspace root (which is my tauri app). If I tried manually adding that with:
pnpm add @tauri-apps/plugin-dialog@~2

I got:

ERR_PNPM_ADDING_TO_ROOT  Running this command will add the dependency to the workspace root, which might not be what you want - if you really meant it, make it explicit by running this command again with the -w flag (or --workspace-root). If you don't want to see this warning anymore, you may set the ignore-workspace-root-check setting to true.

I was able to get this working just by adding ignore-workspace-root-check = true into my .npmrc

However it would be nice if Tauri CLI would show the pnpm error message so this would have been faster to figure out.

try with --verbose flag, it should show the error


Atom (linter-pylint?) bogging down CPU with spike of Python processes

Hello,
I've got a large (~500 line) script that I'm writing that just comes to a crawl whenever I'm typing. I've watched the task monitor and determined it's a spike in Python processes that have a high cycle usage. This problem becomes worse for the duration of typing and then resolves itself slowly over the next few minutes. It's quite frustrating, but I can't be sure if this is the way Atom's handling it or an issue with PyLint or what?
Here's the task manager prior to typing in the file:

Here they are shortly after typing a block of text with multiple lines:

I'm not sure what else to make of it. If this isn't an ATOM Issue, does anyone have an idea what it might be?
Thanks!

This sounds like an issue to file on the repo for the linter package that you are using. Atom doesn't fire off python processes itself.

Moreso, it has to be pylint itself. The problem was reproducible on
Sublime-Text until I disabled pylint on both.
So strange. So saddening.
On Wed, Jul 8, 2015, 20:33 Daniel Hengeveld<EMAIL_ADDRESS>wrote:

Closed #7801 https://github.com/atom/atom/issues/7801.
—
Reply to this email directly or view it on GitHub
https://github.com/atom/atom/issues/7801#event-351370126.


Unique on columns which are composite key

Problem
When using 2 or more columns as a composite key (using isPrimary function). All the columns receive the unique property which is wrong and leads to errors.
Purposed solution
Remove the unique property from the columns upon receiving the primary property if there are 2 or more columns unless specified otherwise which is unlikely.

The bug was fixed in This Commit. Thanks for reporting.