[Feature Request]: MySQL and Postgres Storage Support

Describe the problem
Chroma heavily utilizes SQLite for Collection Storage.
It would be nice if there was MySQL and Postgres support for the project.
Describe the proposed solution
Integrate MySQL and Postgres. Have a way to specify a PersistentClient using a MySQL or Postgres URI.
Alternatives considered
No response
Importance
would make my life easier
Additional Information
I would like to be able to access the Chroma vectors from anywhere and not need to rebuild collections multiple times over.

hi @ChrisCates and @XudongLiu - thanks for opening this issue!
With Chroma we have a few goals for the API surface and support of the main project

extremely easy to set up
extremely easy to use with no sharp edges
very well tested and supported across many machine types

For that reason we are hesitant to add additional backends for metadata storage. Currently we support sqlite in the single-node version and the distributed version will have its own implementation.
Can you share more about your alls motivations here with a mysql or postgres metadata backend?

@jeffchuber, I am looking to just have an easy way to sync Chroma Collections.
I currently created my own implementation in Typescript to translate Chroma Collections into Prisma Schemas. Check it out here:
export function chromaToPrisma(
  collectionId: string,
  payload: ChromaPayload
): PrismaPayload[] {
  const items = [] as PrismaPayload[]

  for (const [index, id] of payload.ids.entries()) {
    const metadata = payload.metadatas[index]
    const document = payload.documents[index]
    const embedding = payload.embeddings[index]

    items.push({
      id,
      collectionId,
      metadata,
      document,
      embedding
    })
  }

  return items
}

export function prismaToChroma(payload: Item[]): ChromaPayload {
  const ids: Array<string> = []
  const metadatas: Array<any> = []
  const documents: Array<string> = []
  const embeddings: Array<any> = []

  for (const item of payload) {
    ids.push(item.id)
    metadatas.push(item.metadata)
    documents.push(item.document)
    embeddings.push(item.embedding)
  }

  return {
    ids,
    metadatas,
    documents,
    embeddings
  }
}

There is a lot more logistics revolved around creating and syncing data.
I think what I'm proposing is much closer to what #1058 is looking for. Just with a little extra tooling added to cache data in your DB of choice (in this case using the Prisma ORM with MySQL, also works with Postgres and MongoDB no problem too).
What do you think? I've already created a lot of source code that would go into a "Studio". Happy to work on a public repo using my framework: https://prntr.click/ for a few bones if you're open to it.

@ChrisCates just to follow the syncing story a bit here... you have data chunked (maybe also embedded) inside postgres/mysql and the question now becomes how to keep that in sync with Chroma? (valid if so! just confirming)

@jeffchuber, yes, I actually use SentenceTransformers Python Library because Xenova's NPM library needs a lot more work to be stable.
Here is a peak of the Chroma Collection creation tool:
    console.log('Now Seeding Chroma Collection with:'.green.bold, inputPath)
    const chromaCollection = await getCollection(name, description)
    const data = JSON.parse(read(inputPath)) as QAPair[] | any[]

    console.log('[1/3] Generating Embedding Payloads'.green.bold)
    const articles = await mapToPayload(data, key, tokenized)

    console.log('[2/3] Creating Chroma Documents'.green.bold)
    chromaCollection.upsert({
      ...articles
    })

    console.log('[3/3] Caching Chroma Documents in PlanetScale'.green.bold)
    const payload = chromaToPrisma(collection.id, articles)
    const chunks = sliceIntoChunks(payload, 500)

    for (const [index, chunk] of chunks.entries()) {
      await prisma.item.createMany({
        skipDuplicates: true,
        data: chunk
      })

      if (chunks.length > 1) {
        console.log(`[${index + 1}/${chunks.length}] uploaded`.blue.bold)
      }
    }

    console.log('Chroma Collection Creation Completed!'.green.bold)

We don't have plans to support these storage backends, so closing for now.


Edit course calendar

To remove "Coming Soon" from Module 3 now that it's posted

Looks like this is finished! I can't close the issue for some reason.


Guidelines for extending NSM

Signed-off-by: John McDowall<EMAIL_ADDRESS>Draft guidelines for extending NSM and suggestions how to re-organize code layout to make it easier for developers to keep all extensions in a single branch of tree.

I created a new repo msn_template that creates a new tree for sample code and fills in a lot of the boiler plate code. It reduces the development hopefully to editing the golang and protobuf files. If there was someway to have this (or something like it) in tree and out of tree would become  a developers choice.


feature to kill the previously started process

I'd like to suggest/request a feature/function to kill the previously 'Start'ed process.  I know 'Start' will do that, but what if a build we did in a previous process failed and we don't do a Start?
My scenario:

Detect file change
Do go build
Oops! Build failed Skip all remaining steps, including 'Start'

Now, the process 'Start'ed on the earlier set is still running and my app is still accessible which is not the expected result.

Definitely a problem. I'm surprised I haven't run into that. I'll think of something and should have a fix in the soon.

One other issue I noticed for my team mate is that windows doesn't allow you to overwrite an exe file that is already running.  So I'll have to first kill the running process and only then do a build.  So yeah, this feature looks to be a necessity.
Let me see if I can make the killSpawned function public and see if it works.  Will send a pull request if.

This might take a couple of days. The solution is to add Start method to the context so the task can track the process and then add a Task.Kill method. Unfortunately, there's a little more to it. There are goroutines involved when watching so I have to do some refactoring to add quit channels. I'll work on it tonight and should have something by tomorrow.

I need to understand your process a bit more. In most cases, if a dependency fails the parent shouldn't be restarted. For example
p.Task("server", D{"assets", "views"} ...)

If "assets" fails then there should be an error logged in red if the task returned an error. eg
p.Task("assets", func() error {
    return Run("browserify ...")
})

We found it annoying that we had to restart godo often when all we had to do was fix syntax in Javascript file.  Once we fixed the error in our editor, it triggers a restart of the server. I admit though every now and then someone forgets to check their terminal for errors.
Also, you don't normally need to build any Go files before Start. Godo builds the main package for you including its dependencies if the argument to Start is a go file.
I can add a flag to always exit on errors if that would help. I'm not sure if it should be the default behavior.

I'm checking some of these things you asked and I'm seeing the weird results.  One, I actually am not able to run it via go run.  I see this other stackoverflow qn about it too: http://stackoverflow.com/questions/26920969/why-does-the-go-run-command-fail-to-find-a-second-file-in-the-main-package
I also am not able to do a go run *.go because then it complains that it cannot run _test.go files.  So I guess only the build and run executable option is currently available for me.

To your other question, here is what happens/happened to me ...

I write code (v1.0), godo detects file change, it builds correctly, it runs, I use it via the browser.
I change code (v1.1). godo detects file change. Code error - it doesn't build. But the old version is still running as it hasn't been killed/re'Start'ed. I still access the app via the browser and assume it's the latest when it's not.

Here is what would be preferable in my opinion ...

I change code (v1.1). godo detects file change. It kills the previously spawned process.  Code has error - it doesn't build.  Now I can't access the app via the browser.

Interesting, I created an example in godo.  However, if I use godo server -w the server runs fine and restarts. Try updating godo
go get -u gopkg.in/godo.v1/cmd/godo

Then CD into the godo.v1 directory and run godo server -w which runs and watches cmd/example.
When you say change code, are you doing a git checkout to another branch? I will add a flag that will stop on any error.

Changing code as in fixing bugs and adding features locally.  I miss a parentheses here or there, build breaks, but older executable is still running.
I was thinking that if you made your killSpawned -> KillSpawned, then we'd have a way to control it ourselves.
Also, as I mentioned, on my team mate's Windows ... assume the process is running from a previous build.  So I've previously started executable.exe.  When I do a go build again, windows gives me an error because it cannot overwrite the exe file of a running process.  So godo always fails.
Right now I've attempted to solve it with "taskkill /f /im executable.exe"

Please post a gitst of your Godofile, the relevant parts that are causing this.
I made a fix on Windows. Not sure if it helps your teammate. I hardly use Windows but I did try the example and there was an issue with it. The example works while watching. Please update godo and let me knkow.

I'm a little embarrassed to show you my current Godofile as it is not really following the dependencies mechanism.  Instead I'm controlling each line of execution at the moment.
I liked Godo and I wanted to tell others in my team and also keep a note of it for later, so I immediately (maybe a little too early) wrote this medium post: https://medium.com/@sathishvj/watch-generate-build-execute-golang-projects-on-file-changes-50255a98ab7c

The golang tools are finicky, to say the least. Using go run will not work as expected. It spawns a child process which cannot be killed programmatically. http://stackoverflow.com/questions/24982845/process-kill-on-child-processes. I use go install -a internally. If your app doesn't build cleanly with a simple go install then godo will not work

Hmm, and there isn't a way to kill the process tree with go?
Anyways, I'm not running it with go run.  I'm doing a ./executable.  So that issue doesn't apply in this case.
I think, at the least, you could make the killSpawned function public, so that if somebody wants to call that specifically, we could.  If not, no loss anyways.

I can do that. I'll do it first thing in the morning. It's getting late. BTW, you can see which files are changing with the verbose flag godo server --verbose.

Oh ok.  Thanks.  I tried -v which is the idiomatic way I think of getting verbose output.  But it gave me version.

In the process of refactoring for godo version 2 (v2) and writing tests, I ran into the Debounce/Watch issues you were having. Watch file events were being swallowed in between Debounce intervals. For example, say index.go.html changed but the task handler was debounced. In this scenario the handler would never run from that change. v2 enqueues the file event to run on after the Debounce interval.
The watch algorithm in v1 is crude and worked for my uses cases. The v2 is more intelligent and efficient. I hope to do a v2 pre-release in about a week.

awesome man! thank you for your work.


Feature Request Rust custom map

Feature Request
Feature Information:
I would like to be able to run custom rust maps. all That is needed to add an argument when you have the correct dll in manged part of server. +server.levelurl "url"
And be able to host it somewhere so players can download it. either from amp itself or via dropbox as many tutorial suggest.
I confirm:

[X] that I have searched for an existing feature request matching the description.

This is fixed in nightly, and will be in mainline on the next release.


NamedValue not getting replaced in Connection String for APIM Bicep

Bicep version
v0.31.92
Describe the bug
I have a namedValue defined for APIM that in turn refers it's value from a key vault secret, but when I try to use that namedValue in the connection string for Application Insights logger, it fails with this error.
"Validation failed for logger-id 'appi_logger_smi'. Exception Received 'Invalid instrumentation key for Application Insights Logger"
To Reproduce
Define a NamedValue like this
resource apim_name_appi_instrumentation_key 'Microsoft.ApiManagement/service/namedValues@2023-05-01-preview' = {
  parent: apim_name_resource
  name: 'appi-instrumentation-key'
  properties: {
    displayName: 'appi-instrumentation-key'
    keyVault: {
      secretIdentifier: 'https://${kv_name}.vault.azure.net/secrets/appi-instrumentation-key'
    }
    tags: []
    secret: true
  }
}

then try to access it in the logger.
resource aiLoggerWithSystemAssignedIdentity 'Microsoft.ApiManagement/service/loggers@2022-08-01' = {
  name: 'appi_logger_smi'
  parent: apim_name_resource
  properties: {
    loggerType: 'applicationInsights'
    description: 'Application Insights logger with system-assigned managed identity'
    credentials: {
      connectionString: 'InstrumentationKey=@{{appi-instrumentation-key}};IngestionEndpoint=https://uksouth-1.in.applicationinsights.azure.com/'
      identityClientId: 'systemAssigned'
    }
    isBuffered: true
    resourceId: appi_externalid
  }
  dependsOn: [
    apim_name_appi_instrumentation_key
  ]
}

Additional context
Add any other context about the problem here.

I think it might be your named-value reference syntax as per docs. Try removing the @, i.e. connectionString: 'InstrumentationKey={{appi-instrumentation-key}};IngestionEndpoint=https://uksouth-1.in.applicationinsights.azure.com/'

I think it might be due to how the named-value is referenced. Try removing the @, i.e. connectionString: 'InstrumentationKey={{appi-instrumentation-key}};IngestionEndpoint=https://uksouth-1.in.applicationinsights.azure.com/'

Yes, I tried that first, it fails with validation error


how to use 3DDFA_V2 detect pose?

Hello,
I saw you provide the *npy file in test/pose/ folder, the shape is (1, 7).
I changed https://github.com/cleardusk/3DDFA_V2/blob/v0.12/demo.py#L91 default to "pose",  the result is like
https://github.com/cleardusk/3DDFA_V2/blob/v0.12/docs/images/trump_hillary_pose.jpg.
No variables shape is (1, 7).

Hi, I have uploaded the pose detection code based on 3DDFA_V2. Simply replace the corresponding file should get the final results.


Add domain association-esi.org

@university365 Your school has a very complex structure and affiliation, according to its website. Could you please clarify what kind of educational courses do you take? How long they are? Do you study online or offline? Also, please provide us with:

the school official website URL
an URL of a page on the official website where the school offers an IT-related long-term (>1 year) course
a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.

Hello,
University365 : Computer training over a period of 5 years delivering a Bac+5 title of expert in Computer Science and Information Systems (Title level 7, Master's level, engineer level).
University365 offers training on physical campuses or a virtual campus.
https://www.university-365.com/
https://www.university-365.com/cursus
Bryan DANJOU
IT Architect
University Information System | University 365
+33 1 73 04 33 96tel:+352 661 64 27 71
bryan.danjou<EMAIL_ADDRESS>university-365.comhttps://university-365.com/
33 Avenue du Maine, 75015 - Paris
[cid:image002.png@01D67FC1.D94E4790]http://www.university-365.com/
De : Philip Torchinsky<EMAIL_ADDRESS>Envoyé : dimanche 30 août 2020 16:08
À : JetBrains/swot<EMAIL_ADDRESS>Cc : DANJOU Bryan<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Objet : Re: [JetBrains/swot] Add domain association-esi.org (#9041)
@university365https://github.com/university365 Your school has a very complex structure and affiliation, according to its website. Could you please clarify what kind of educational courses do you take? How long they are? Do you study online or offline? Also, please provide us with:

the school official website URL
an URL of a page on the official website where the school offers an IT-related long-term (>1 year) course
a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/JetBrains/swot/pull/9041#issuecomment-683424980, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQYU7C46J3ZVZWHEDEEI7S3SDJMKFANCNFSM4QNHYRLQ.

@university365 Thank you for the clarification. Pull request merged.


Route export via gpx

add option to export just the routes in standard .gpx format.
Feature request came from Workout Map. See here

Thanks, I'd like to test that via TestFlight :-)
Crucial part of the feature is that I can filter by activity type and date range.

I have to add an activity type filter then. That is no problem.
Date filter already exists.
Currently the following activity types are supported:

walking
running
hiking
cycling
swimming
wheelchairWalkPace
wheelchairRunPace

is something missing here?

Looks good to me, thx 😊
Gesendet von unterwegs.
Am 05.01.2024 um 16:17 schrieb Andre @.***>:

I have to add an activity type filter then. That is no problem.
Date filter already exists.
Currently the following activity types are supported:
walking running hiking cycling swimming wheelchairWalkPace wheelchairRunPace
is something missing here?
—
Reply to this email directly,
view it on GitHub, or
unsubscribe.
You are receiving this because you commented.Message
ID: @.***>

Looks like Apple was pretty quick with their review.
First version is available on TestFlight now: https://testflight.apple.com/join/p7dnsNuI

Looking good! Unfortunately, the tool that I want to use the GPX files for, GeoActivityPlayground, has some issues on my machine, so I can't test the output right now. Will do that as soon as those issues get resolved.

Ok great. I will leave the testing phase open for a bit, before publishing the new version. Let me know if something is not working as expected or if it makes sense to have other additions.
I also appreciate App Store reviews for my apps. That's always helpful :)

@jan-eckhoff were you able to test the export output?
If everything works fine, I would end the beta test and release a new version of the app

Hi Andre,
unfortunately, my target system still seems to have a bug that prevents me from testing the data, so I can’t give you feedback on the complete workflow. However, the GPX files look sensible. If that’s enough for you to close the beta and release the new
version, that would be completely fine with me :-)
Best,
Jan
Am 26.01.2024 um 19:09 schrieb Andre @.***>:
@jan-eckhoff
were you able to test the export output?
If everything works fine, I would end the beta test and release a new version of the app
—
Reply to this email directly,
view it on GitHub, or
unsubscribe.
You are receiving this because you were mentioned.Message
ID: @.***>

Hi Jan,
I finished version 1.1.
It looked fine in my tests and when I opened the .gpx files with other apps.
If you happen to run in any problems, create a new ticket or open this one here again.
I will close it as resolved for now.


Duplicate requests and differing Referrer Policies

Investigate and respond to the following report...
Source: https://www.reddit.com/r/webdev/comments/6mduuk/bootstrap_cdn_integrity_inlcude/

I change my CDN links (Bootstrap/FontAwesome/JQuery) to have the recommended integrity attribute.
<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">

Today I looked at my Network tab and saw that it duplicates the requests for Bootstrap and FontAwesome, just the Referrer Policy is different.
One is unsafe-url and the other is the common no-referrer-when-downgrade

Looking at a project that uses both https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css and https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js with SRI, I am seeing a duplicate of bootstrap.min.css, however, notice in the screenshot that one is labeled "(from disk cache)".

Browser information:
Google Chrome
Version 59.0.3071.115 (Official Build) (64-bit)

When running this in Firefox I'm only seeing a single reference...

Browser information:
Firefox (Mac OS)
54.0.1 (64-bit)

Like we discussed on Slack, I can't see something wrong on our side from a quick look. I mean, we don't even set no-referrer-when-downgrade ourselves. It must be something the browsers do when the devtools is open. I can't think of any other reason myself...

via @jonathanKingston on Twitter:
Sounds like Chrome is running an experiment to see if they can make that the default for unsafe requests?

https://twitter.com/KingstonTime/status/885926332047163393

I have the same issue (only in Chrome). The CSS files are fetched twice.
Browser: Chrome 59.0.3071.115 (Official Build) (64-bit)
OS: Windows 10 x64
URL: https://jahdaut.com

I think we can close this. It's something browsers do and not related to us.


Joins inside User Ops

At the time this issue is being filed, Zed is at commit 2314053.
Performing a join typically requires using from to reference multiple sources of data, so this is already blocked by #4645. However, once that's working, there may be more work left to do in making sure all the key references and assignments in the join also work as expected from inside User Ops. The use cases we have for User Ops from the community thus far are very join-centric, so we'll definitely want to make sure this is all working before we declare victory.

Now that #4645 has been addressed I've been able to verify at Zed commit 7ba4ecc that join now seems usable from inside User Ops.
For instance, here's the inner join example from the Join Tutorial moved unchanged into a User Op.
$ zq -version
Version: v1.8.1-59-g7ba4ecc6

$ cat inner-join-in-op.zed
op CallMe(): (
  file fruit.ndjson | sort flavor
  | inner join (
    file people.ndjson | sort likes
  ) on flavor=likes eater:=name
)
CallMe()

$ zq -I inner-join-in-op.zed
{name:"figs",color:"brown",flavor:"plain",eater:"jessie"}
{name:"banana",color:"yellow",flavor:"sweet",eater:"quinn"}
{name:"strawberry",color:"red",flavor:"sweet",eater:"quinn"}
{name:"dates",color:"brown",flavor:"sweet",note:"in season",eater:"quinn"}
{name:"apple",color:"red",flavor:"tart",eater:"morgan"}
{name:"apple",color:"red",flavor:"tart",eater:"chris"}

This also works if the field references all become parameters and are passed in as arguments.
$ cat inner-join-in-op-params.zed
op CallMe(left_key_field, right_key_field, left_dest, right_source): (
  file fruit.ndjson | sort left_key_field
  | inner join (
    file people.ndjson | sort right_key_field
  ) on left_key_field=right_key_field left_dest:=right_source
)
CallMe(flavor, likes, eater, name)

$ zq -I inner-join-in-op-params.zed
{name:"figs",color:"brown",flavor:"plain",eater:"jessie"}
{name:"banana",color:"yellow",flavor:"sweet",eater:"quinn"}
{name:"strawberry",color:"red",flavor:"sweet",eater:"quinn"}
{name:"dates",color:"brown",flavor:"sweet",note:"in season",eater:"quinn"}
{name:"apple",color:"red",flavor:"tart",eater:"morgan"}
{name:"apple",color:"red",flavor:"tart",eater:"chris"}

There are still two limitations I bumped into while testing further variations on this, but they are not unique to join.

The pool/filename references can't yet be passed into parameters (tracked in #4700)
It's not yet possible to have just the User Op included via -I with the User Op call made from the shell (tracked in #4701).

Therefore, closing this one.


Create a la carte installer component for test framework

Context: https://github.com/istio/istio/issues/13765
Right now istio/installer runs the integration tests by skipping the tests that require a custom setup, giving less coverage. We should likely add a new component to the framework that uses the new installer so we can test all configurations.

Working on this


javascript shows different icon

Describe the bug
From the new update, the js icon changed, I don't know if that's the case or it's a bug
To Reproduce
Install the plugin.
Create a TS file.
Expected Behavior
JavaScript files have JavaScript icons.
Environment
IntelliJ IDEA 2022.3.1 (Ultimate Edition)
Build #IU-223.8214.52, built on December 20, 2022
Licensed to Jose Alejandro Perez Chavez
Subscription is active until October 23, 2023.
For educational use only.
Runtime version: 17.0.5+1-b653.23 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Windows 11 10.0
GC: G1 Young Generation, G1 Old Generation
Memory: 1966M
Cores: 16
Registry:
debugger.new.tool.window.layout=true
ide.experimental.ui=true
ide.balloon.shadow.size=0
Non-Bundled Plugins:
tv.codely.codelytv-theme (2.2.2)
com.github.copilot (<IP_ADDRESS>9)
com.mallowigi (84.0.0)
Kotlin: 223-1.7.21-release-272-IJ8214.52
Screenshots

Literal duplicate of another bug posted today


Open new tab instead of redirect

Makes it opens the repo on new tab instead so the thing still run in backgroud

btw i made this bc I get annoyed trying to open the repo while still running the waster and I have to right click -> open in new tab

noice

me who forgot to reload the page thought it didn’t work lol


Create CGP-0083

Pull request for CGP-0083: Funding for Credit Collective

there's already a pull request with cgp -0083 thats beenn approved. you'll need to re-edit


fix: allow to configure debounce time and increase default

This PR contains a:

[x] bugfix

Motivation / Use-Case
currently sometimes if builds take too long the "next" processor may trigger an additional build.
So let's increase the default from 100 to 1000 and also make it configurable.

Even with a bigger timeout, it still runs the tests multiple times for me after a small change.

I think it's unrelated to the debounce, it looks like every so often it's spawning extra build processes and it just keeps growing.

@LarsDenBakker so "ditching" karma watch all together lets us get rid of the full debounce and should also enable a better watch mode without duplicate builds.
can you pls check again :)

Yep, that fixes it 👍

@matthieu-foucault I updated the merge request and cleaned up the history.
I think once this is merged we should have another alpha release (together with the other unreleased fix) - can you make it happen pls 🤗

I’d merge but I’m on mobile. Thanks for the PR, awesome work!

friendly reminder for this pull and release request 🤗

Will take care of this later today
Le mar. 1 janv. 2019 à 07:53, Thomas Allmer<EMAIL_ADDRESS>a
écrit :

friendly reminder for this pull and release request 🤗
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/webpack-contrib/karma-webpack/pull/386#issuecomment-450739345,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACj0dtbjMJjVX9UWb_DWXA9e8Njam6CLks5u-4RhgaJpZM4ZknC7
.


New Api version 2022-09-01 for Microsoft.StorageSync

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.ResourceManager.StorageSync


Build failure: spamassassin

Steps To Reproduce
Steps to reproduce the behavior:

build nixpkgs.spamassassin.x86_64-linux
observe buit-in tests

Build log

Truncated, full log avialable at https://hydra.nixos.org/build/280236104/nixlog/1

Build Log
Running phase: unpackPhase
unpacking source archive /nix/store/mxf6rn9wzpyk3lym54pxh0b9wls3l1hf-Mail-SpamAssassin-4.0.1.tar.bz2
source root is Mail-SpamAssassin-4.0.1
setting SOURCE_DATE_EPOCH to timestamp<PHONE_NUMBER> of file Mail-SpamAssassin-4.0.1/META.yml
Running phase: patchPhase
applying patch /nix/store/nrk88vrhfk265xqk4l9g65sc8xmn576g-satest-no-clean-path.patch
patching file t/SATest.pm
applying patch /nix/store/dq1nkdjpin6a58q5zay190kzllwdkyv1-sa_compile-use-perl5lib.patch
patching file t/sa_compile.t
Hunk #1 succeeded at 42 (offset 2 lines).
Hunk #2 succeeded at 88 (offset 2 lines).

What email address or URL should be used in the suspected-spam report
text for users who want more information on your filter installation?
(In particular, ISPs should change this to a local Postmaster contact)
default text: [the administrator of that system] the administrator of that system

NOTE: settings for "make test" are now controlled using "t/config.dist".
See that file if you wish to customize what tests are run, and how.

checking module dependencies and their versions...

***************************************************************************
NOTE: the optional Net::LibIDN module is not installed.

  Provides mapping between Internationalized Domain Names (IDN) in
  Unicode and ASCII-compatible encoding (ACE) for use in DNS and comparisions.
  The module is optional, but without it Unicode IDN names found in mail will
  not be suitable for DNS queries and welcome/blocklisting.

***************************************************************************
NOTE: the optional IP::Country::DB_File module is not installed.

  Used by the RelayCountry plugin (not enabled by default) to
  determine the domain country codes of each relay in the path of an email.
  Also used by the URILocalBL plugin (not enabled by default) to provide
  Country code based filtering.

***************************************************************************
NOTE: the optional IO::Socket::INET6 module is not installed.

  This module is a deprecated alternative to IO::Socket::IP.
  Spamd, as well some underlying modules, will fall back to using
  IO::Socket::INET6 if IO::Socket::IP is unavailable. One or the other
  module is required to support IPv6 (e.g. in spamd/spamc protocol,
  for DNS lookups or in plugins like DCC). Some plugins or underlying
  modules may still require IO::Socket::INET6 for IPv6 support even
  if IO::Socket::IP is available.

***************************************************************************
NOTE: the optional Net::Patricia module is not installed,
minimum required version is 1.16.

  If this module is available, it will be used for IP address
  lookups in tables internal_networks, trusted_networks, msa_networks and
  uri_local_cidr.  Recommended when a number of entries in these tables is
  large, i.e.  in hundreds or thousands.  However, in case of overlapping
  (or conflicting) networks in these tables, lookup results may differ as
  Net::Patricia finds a tightest-matching entry, while a sequential
  NetAddr::IP search finds a first-matching entry.  So when overlapping
  network ranges are given, specifying more specific subnets (longest
  netmask) first, followed by wider subnets ensures predictable results.

***************************************************************************
NOTE: the optional BSD::Resource module is not installed.

  BSD::Resource provides BSD process resource limit and priority
  functions.  It is used by the optional ResourceLimits Plugin.

***************************************************************************
NOTE: the optional IO::String module is not installed.

  IO::String emulates file interface for in-core strings.
  It is used by the optional OLEVBMacro Plugin.

***************************************************************************
NOTE: the optional Devel::Cycle module is not installed.

  Devel::Cycle is used in make test in tests that will be harmelessly
  skipped if it is not available

checking binary dependencies and their versions...

***************************************************************************
NOTE: the optional gpg binary is not installed,
recommended version is 1.0.6 or higher.

  The "sa-update" program requires this executable to verify
  encryption signatures.  It is not recommended, but you can use
  "sa-update" with the --no-gpg to skip the verification.

***************************************************************************
NOTE: the optional wget binary is not installed,
recommended version is 1.8.2 or higher.

   Sa-update will use curl, wget or fetch to download updates.
   Because perl module LWP does not support IPv6, sa-update as of
   3.4.0 will use these standard programs to download rule updates
   leaving LWP as a fallback if none of the programs are found.

   *IMPORTANT NOTE*: You only need one of these programs
       It's only a concern if you are warned about all 3
       i.e. (curl, wget & fetch) missing

dependency check complete...

optional module missing: Net::LibIDN
optional module missing: IP::Country::DB_File
optional module missing: IO::Socket::INET6
optional module missing: Net::Patricia
optional module missing: BSD::Resource
optional module missing: IO::String
optional module missing: Devel::Cycle
optional binary missing or nonfunctional: gpg
optional binary missing or nonfunctional: wget

warning: some functionality may not be available,
please read the above report before continuing!

Checking if your kit is complete...
Looks good
Warning: prerequisite Perl::Critic::Policy::Perlsecret 0 not found.
Warning: prerequisite Perl::Critic::Policy::TestingAndDebugging::ProhibitNoStrict 0 not found.
'LOCALSTATEDIR' is not a known MakeMaker parameter name.
'SYSCONFDIR' is not a known MakeMaker parameter name.
Generating a Unix-style Makefile
Writing Makefile for Mail::SpamAssassin
Writing MYMETA.yml and MYMETA.json
Makefile written by ExtUtils::MakeMaker 7.70
no configure script, doing nothing
Running phase: buildPhase
cp spamassassin blib/script/spamassassin
"/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/spamassassin
/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl spamc/configure.pl --prefix="/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1" --sysconfdir="/etc/mail/spamassassin" --datadir="/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1/share/spamassassin" --enable-ssl="yes"
cd spamc
/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl version.h.pl
version.h.pl: creating version.h
./configure --prefix=/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1 --sysconfdir=/etc/mail/spamassassin --datadir=/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1/share/spamassassin --enable-ssl=yes
checking for gcc... cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether cc accepts -g... yes
checking for cc option to enable C11 features... none needed
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for grep that handles long lines and -e... /nix/store/14aswfz6pm8zvyyy927xzmi8x4rnvlc8-gnugrep-3.11/bin/grep
checking for egrep... /nix/store/14aswfz6pm8zvyyy927xzmi8x4rnvlc8-gnugrep-3.11/bin/grep -E
checking for sys/time.h... yes
checking for syslog.h... yes
checking for unistd.h... (cached) yes
checking for errno.h... yes
checking for sys/errno.h... yes
checking for time.h... yes
checking for sysexits.h... yes
checking for sys/socket.h... yes
checking for netdb.h... yes
checking for netinet/in.h... yes
checking for pwd.h... yes
checking for signal.h... yes
checking for openssl/crypto.h... yes
checking for zlib.h... yes
checking for an ANSI C-conforming const... yes
checking for off_t... yes
checking for pid_t... yes
checking how to run the C preprocessor... cc -E
checking for uid_t in sys/types.h... yes
checking for size_t... yes
checking for SHUT_RD... yes
checking for socket in -lsocket... no
checking for CRYPTO_malloc in -lcrypto... yes
checking for SSL_CTX_free in -lssl... yes
checking for connect in -linet... no
checking for t_accept in -lnsl... no
checking for deflate in -lz... yes
checking for dlopen in -ldl... yes
checking for socket... yes
checking for strdup... yes
checking for strtod... yes
checking for strtol... yes
checking for snprintf... yes
checking for shutdown... yes
checking for h_errno... yes
checking for in_addr_t... yes
checking for INADDR_NONE... yes
checking for EX__MAX... yes
checking for EAI_ADDRFAMILY... no
checking for EAI_SYSTEM... (cached) no
checking for EAI_NODATA... (cached) no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating spamc.h
config.status: creating config.h
make -f spamc/Makefile spamc/spamc
make[1]: Entering directory '/build/Mail-SpamAssassin-4.0.1'
cc -DSPAMC_SSL -g -O2 spamc/spamc.c spamc/getopt.c spamc/libspamc.c spamc/utils.c \
        -o spamc/spamc  -lssl -lcrypto  -ldl -lz
spamc/libspamc.c: In function 'libspamc_log':
spamc/libspamc.c:2384:16: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;]
 2384 |         (void) write (2, buf, len);
      |                ^~~~~~~~~~~~~~~~~~~
make[1]: Leaving directory '/build/Mail-SpamAssassin-4.0.1'
cp spamc/spamc blib/script/spamc
"/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/spamc
/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl build/preprocessor  -Mvars -DVERSION="4.000001" -DPREFIX="/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1" -DDEF_RULES_DIR="/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1/share/spamassassin" -DLOCAL_RULES_DIR="/etc/mail/spamassassin" -DLOCAL_STATE_DIR="/var/lib/spamassassin" -DINSTALLSITELIB="/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1/lib/perl5/site_perl/5.40.0" -DCONTACT_ADDRESS="the administrator of that system" -DRE2C_BIN="re2c" -Msharpbang -Mconditional -DPERL_BIN="/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl" -DPERL_WARN="" -DPERL_TAINT="" -m755 -ispamd/spamd.raw -ospamd/spamd
cp spamd/spamd blib/script/spamd
"/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/spamd
Manifying 9 pod documents
Manifying 28 pod documents
Manifying 25 pod documents
Manifying 25 pod documents
Manifying 10 pod documents
buildPhase completed in 32 seconds
Running phase: checkPhase
substituteStream() in derivation perl5.40.0-SpamAssassin-4.0.1: WARNING: '--replace' is deprecated, use --replace-{fail,warn,quiet}. (file 't/spamc_x_e.t')
check flags: SHELL=/nix/store/0irlcqx2n3qm6b1pc9rsd2i8qpvcccaj-bash-5.2p37/bin/bash PERL_BIN=/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl ENABLE_SSL=yes VERBOSE=y TEST_FILES=\$\(shell\ find\ t\ -name\ \*.t\ -not\ -name\ spamd_ssl_accept_fail.t\) test
"/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl" build/mkrules --exit_on_no_src --src rulesrc --out rules --manifest MANIFEST --manifestskip MANIFEST.SKIP
no source directory found: exiting
/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl build/preprocessor  -Mvars -DVERSION="4.000001" -DPREFIX="/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1" -DDEF_RULES_DIR="/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1/share/spamassassin" -DLOCAL_RULES_DIR="/etc/mail/spamassassin" -DLOCAL_STATE_DIR="/var/lib/spamassassin" -DINSTALLSITELIB="/nix/store/mmsny846bl6r5pxn74hr0qmxlamyib7q-perl5.40.0-SpamAssassin-4.0.1/lib/perl5/site_perl/5.40.0" -DCONTACT_ADDRESS="the administrator of that system" -DRE2C_BIN="re2c" -Msharpbang -Mconditional -DPERL_BIN="/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl" -DPERL_WARN="" -DPERL_TAINT="" -m755 -isa-update.raw -osa-update
cp sa-update blib/script/sa-update
"/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/sa-update
PERL_DL_NONLAZY=1 "/nix/store/3vq9qasxlqpyq1k95nq3s13g2m6w59ay-perl-5.40.0/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/root_spamd_x_paranoid.t t/spamd_welcomelist_leak.t t/nonspam.t t/welcomelist_to.t t/dnsbl.t t/db_based_welcomelist.t t/root_spamd_x.t t/db_awl_path_welcome_block.t t/root_spamd_u.t t/basic_lint_net.t t/pdfinfo.t t/fromnamespoof.t t/spamd_ldap.t t/line_endings.t t/spamd_report_ifspam.t t/originating_ip_hdr.t t/bayesdbm.t t/uri_saferedirect.t t/olevbmacro.t t/rcvd_parser.t t/spamd_whitelist_leak.t t/lang_pl_tests.t t/check_implemented.t t/sa_txrep_sql.t t/spamd_user_rules_leak.t t/spamc_cf.t t/body_str.t t/reportheader_8bit.t t/report_safe.t t/root_spamd_x_u.t t/reuse.t t/dnsplatform.t t/spamc_c_stdout_closed.t t/spf_welcome_block.t t/spam.t t/regexp_valid.t t/enable_compat.t t/ifversion.t t/mimeheader.t t/html_obfu.t t/spamc_z.t t/config_errs.t t/dnsbl_subtests.t t/blacklist_autolearn.t t/uribl.t t/uribl_all_types.t t/uri_html.t t/spamd.t t/shortcircuit_before_dns.t t/spamd_parallel.t t/if_can.t t/basic_meta2.t t/decodeshorturl.t t/uri_text.t t/recreate.t t/sa_txrep.t t/blocklist_autolearn.t t/missing_hb_separator.t t/mimeparse.t t/spamc_l.t t/db_based_welcomelist_ips.t t/spamd_prefork_stress_4.t t/spamd_ssl.t t/get_all_headers.t t/sha1.t t/spamd_prefork_stress_3.t t/body_mod.t t/dkim.t t/welcomelist_addrs.t t/spamd_utf8.t t/if_else.t t/uribl_ips_only.t t/sa_check_spamd.t t/arc.t t/memory_cycles.t t/spamd_kill_restart_rr.t t/spamd_plugin.t t/bayessdbm.t t/basic_lint_without_plugins.t t/bayessdbm_seen_delete.t t/spamc_bug6176.t t/db_awl_perms.t t/spamc_E.t t/rule_types.t t/tainted_msg.t t/spamc_x_e.t t/bayessql.t t/lang_lint.t t/authres.t t/podchecker.t t/dcc.t t/spamd_syslog.t t/all_modules.t t/recips.t t/spamd_port.t t/spamd_report.t t/root_spamd_virtual.t t/db_based_whitelist.t t/config_tree_recurse.t t/spamd_unix_and_tcp.t t/spamc_optL.t t/bayesbdb.t t/spamc_B.t t/root_spamd_tell_paranoid.t t/shortcircuit.t t/idn_dots.t t/spamc_headers.t t/basic_meta.t t/mkrules.t t/spamc.t t/ip_addrs.t t/stop_always_matching_regexps.t t/html_visibility.t t/spamd_symbols.t t/uri_list.t t/dmarc.t t/whitelist_addrs.t t/html_colors.t t/spamd_protocol_10.t t/spamd_allow_user_rules.t t/basic_lint_without_sandbox.t t/db_awl_perms_welcome_block.t t/spamd_prefork_stress.t t/desc_wrap.t t/rule_names.t t/spamd_prefork_stress_2.t t/strip_no_subject.t t/pyzor.t t/root_spamd_u_dcc.t t/spamd_maxchildren.t t/basic_obj_api.t t/util_wrap.t t/dnsbl_sc_meta.t t/urilocalbl.t t/header_utf8.t t/freemail.t t/wlbl_uri.t t/mkrules_else.t t/sql_based_welcomelist.t t/cidrs.t t/freemail_welcome_block.t t/extracttext.t t/askdns.t t/uridetail.t t/rule_multiple.t t/stripmarkup.t t/uribl_domains_only.t t/strip2.t t/gtube.t t/root_spamd_tell_x.t t/spamd_kill_restart.t t/sa_awl.t t/utf8.t t/spf.t t/plugin_priorities.t t/cpp_comments_in_spamc.t t/spamc_c.t t/local_tests_only.t t/lint_nocreate_prefs.t t/whitelist_subject.t t/root_spamd_tell_x_paranoid.t t/config_text.t t/re_base_extraction.t t/plugin_file.t t/priorities.t t/db_based_whitelist_ips.t t/hashbl.t t/sql_based_whitelist.t t/recursion.t t/prefs_include.t t/autolearn.t t/relaycountry.t t/autolearn_force_fail.t t/root_spamd_tell.t t/regexp_named_capture.t t/welcomelist_subject.t t/razor2.t t/spamc_y.t t/welcomelist_from.t t/spamd_unix.t t/root_spamd.t t/sa_txrep_welcomelist_out.t t/basic_lint.t t/html_utf8.t t/header.t t/whitelist_from.t t/spamc_x_E_R.t t/spamd_ssl_z.t t/spamc_optC.t t/spamd_stop.t t/uri.t t/bayesdbm_flock.t t/text_bad_ctype.t t/timeout.t t/plugin.t t/get_headers.t t/sa_awl_welcome_block.t t/metadata.t t/whitelist_to.t t/relative_scores.t t/spamd_sql_prefs.t t/spamd_maxsize.t t/db_awl_path.t t/trust_path.t t/spamc_H.t t/spamd_client.t t/debug.t t/perlcritic.t t/cross_user_config_leak.t t/phishing.t t/autolearn_force.t t/date.t t/reportheader.t t/priorities_welcome_block.t t/spamd_hup.t t/sa_compile.t
t/all_modules.t ................... ok
t/extracttext.t ................... ok
t/spamd_prefork_stress.t .......... skipped: Spamd prefork stress tests disabled
t/spamd_prefork_stress_2.t ........ skipped: Spamd prefork stress tests disabled
t/arc.t ........................... skipped: Net tests disabled
t/autolearn.t ..................... ok
t/check_implemented.t ............. ok
t/cidrs.t ......................... ok
t/config_errs.t ................... skipped: This script does not work yet. There are (a) lots of failures and (b) the 'plan' line means all tests are effectively ignored from 'make test'.
t/config_text.t ................... ok
t/config_tree_recurse.t ........... ok
t/cpp_comments_in_spamc.t ......... ok
t/cross_user_config_leak.t ........ ok
t/date.t .......................... ok
===(     437;143  51/59  1/4 )==========================================# Note: This test when successful displays lockfile warning messages
# Nov 30 04:21:46.101 [3191] warn: auto-welcomelist: open of auto-welcomelist file failed: locker: safe_lock: cannot create tmp lockfile ./log/db_awl_path.RwtKq0/awl/this_lock_warning_is_ok.lock.localhost.3191 for ./log/db_awl_path.RwtKq0/awl/this_lock_warning_is_ok.lock: Not a directory
t/bayessql.t ...................... ok
t/db_awl_path.t ................... ok
===(     449;147  1/4  0/? )============================================# Nov 30 04:21:48.552 [3195] warn: auto-welcomelist: open of auto-welcomelist file failed: locker: safe_lock: cannot create tmp lockfile ./log/db_awl_path_welcome_block.XZFCew/awl/this_lock_warning_is_ok.lock.localhost.3195 for ./log/db_awl_path_welcome_block.XZFCew/awl/this_lock_warning_is_ok.lock: Not a directory
t/db_awl_perms.t .................. ok
t/spamd_kill_restart_rr.t ......... ok
===(    1891;645   1/20  0/? )==========================================# NOTE: A rare single failure in this test may be a race condition in the test that can be ignored
t/spamd_sql_prefs.t ............... ok
===(    1979;683  1/9  0/9 )============================================        Not found: firstline =  Return-Path<EMAIL_ADDRESS>at t/spamd_ssl.t line 28.

#   Failed test at t/SATest.pm line 929.
        Not found: endsinnums =  TEST_ENDSNUMS at t/spamd_ssl.t line 28.

#   Failed test at t/SATest.pm line 929.
        Not found: noreal =  TEST_NOREALNAME at t/spamd_ssl.t line 28.

#   Failed test at t/SATest.pm line 929.
        Not found: flag =  X-Spam-Flag: YES at t/spamd_ssl.t line 28.

#   Failed test at t/SATest.pm line 929.
        Not found: stars =  X-Spam-Level: ********** at t/spamd_ssl.t line 28.

#   Failed test at t/SATest.pm line 929.
        Not found: status =  X-Spam-Status: Yes, score= at t/spamd_ssl.t line 28.

#   Failed test at t/SATest.pm line 929.
Output can be examined in: log/spamd_ssl.aVvHzG/d.spamd_ssl/out.0
# Looks like you failed 6 tests of 9.
t/spamd_ssl.t ..................... Dubious, test returned 6 (wstat 1536, 0x600)
Failed 6/9 subtests

Test Summary Report
-------------------
t/spamd_ssl.t                   (Wstat: 1536 (exited 6) Tests: 9 Failed: 6)
  Failed tests:  2, 4-5, 7-9
  Non-zero exit status: 6
Files=216, Tests=3673, 816 wallclock secs ( 1.69 usr  0.35 sys + 246.44 cusr 206.95 csys = 455.43 CPU)
Result: FAIL
Failed 1/216 test programs. 6/3673 subtests failed.
make: *** [Makefile:1513: test_dynamic] Error 255

Additional context
The issue can be seen in Hydra: https://hydra.nixos.org/build/280236104, I can reproduce the same on my machine while building my whole NixOS as well as building this package in isolation.
A similar error has been reported previously: https://github.com/NixOS/nixpkgs/issues/243129
The test t/spamd_ssl.t is failing. From what I have observed, the email is correctly flagged as spam (score 40 > 5), but the additional headers required in the test are not added correctly. However I am expert in neither SpamAssassin nor Perl so I gave up debugging.
Metadata

system: "x86_64-linux"
host os: Linux 6.12.1, NixOS, 25.05 (Warbler), 25.05.20241128.a4308ab
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.24.10
nixpkgs: /nix/store/lz3fp5a05q1ngr8lnf2wfzmc0ji7h6jl-source

Notify maintainers

@qknight @alyssais

Note for maintainers: Please tag this issue in your PR.

Add a :+1: reaction to issues you find important.

Seems to run correctly again in https://hydra.nixos.org/build/280788147. I have no idea why this happens.


Cloud Pak for Integration CP4I

Publish content to: https://github.com/IBM/automation-integration-platform
Definition of Done

[ ] link to Public Git Repo
[ ] link to Public BOM

Dependencies (name & hyperlink)
OCP Platform Module

[x] OCP Login
[x] Cloud Pak Catalog Source
[x] Sealed Secret
[x] GitOps Repository
[x] ArgoCD

Bootstrap
CP4I Operators

[x] API Connect Operators #749
[x] App Connect Operators #746
[x] MQ Operators #748
[x] EventStream Operators #565
[x] PlatformNavigator Operators #745
[x] DataPower Operators #706
[x] Aspera Operator #26
[ ] Confluent (? is there demand for this)
[x] AssetRepository Operator #649
[x] OperationsDashboard Operator #648

CP4I Instances

[x] PlatformNavigator Instance #745
[x] App Connect Dashboard Instance #635
[x] App Connect Designer Instance #650
[x] MQ Instance #585
[x] DataPower Instance #659
[x] MQ Uniform-Cluster Instance #567
[x] API Connect Instance #634
[ ] EventStream Instance #565
[ ] Data power Instance #659
[ ] Aspera instance
[x] AssetRepo Instance Operator #649
[x] OperationsDashboard Instance Operator #648

Publish content

[ ] Generate automation logic and publish to repo - https://github.com/IBM/automation-integration-platform
[ ] Document installation instructions in README - https://github.com/IBM/automation-integration-platform

Supported Platforms

[x] AWS
[x] Azure
[x] IBM Cloud

Delivery Kit

[ ] Create Delivery Kit from Powerpoint Template
[ ] Business Value
[ ] Value to Partner
[ ] Delivery Approach
[ ] Dependencies
[ ] Outcome Value
[ ] Save to (TBD)

@vksuktha @triceam Here is the new Epic card for CP4I on the public zenhub board

https://github.ibm.com/gsi-labs/industry-cloud-planning/issues/677

https://github.ibm.com/gsi-labs/industry-cloud-planning/issues/565

@vksuktha @triceam is Confluence a dependency for CP4I?
If not, let's delete that line from the dependency list.
If so, is this the module for it?
If this is not the right module, please add a new issue and update link in the dependency list.

@vksuktha @triceam please provide answer to prior question....

I do not know dependencies for CP4I

@vksuktha please answer questions from 12 days ago and update status and dependency list. Thank you!


IPC

Brian,
What is the best way to pass a klong "object" to another process, either local or remote?

That's an interesting question.  Currently there's no way to serialize the KlongInterpeter, however, it is basically a dictionary, so one quick way would be to send a "state" dictionary to the remote Klong server and populate the state via puts.  When time allows, I've been working on https://github.com/briangu/kdfs which is meant to be more like a KlongPy as a server, allowing Klong via RPC (TCP) to make things like ticker plants.
So a couple ideas come to mind: either i figure out a way to serialize KlongContext tree, or just enumerate the KlongInterpreter context (I think i'd need to add dictionary enumeration to walk the context), or send commands to the remote KlongInterpeter to build the context "over there".  The tricky thing of sending state could be that marshaling big numpy arrays could be annoying vs using something like the DFS project to load a pandas dataframe "over there".
Does this help?

Yeah I see the logic, I didn't know about kdfs, thanks for pointing me to that. Serialization vs enumeration, I think both would be useful, see, I think we should be able to save the "state" as well?

If i understand correctly, you are suggesting something like:
klong('kdb_objects') = kdbobjects
image = klong.to_image()
# write to file or send over network
newKlong = KlongInterpeter.from_image(image)
kdbobjects = newKlong('kdb_objects')
# continue "over here"

does this seem accurate?

it would be interesting to consider something like this, but maybe there
isn't much value adding interoperability with kdb+ in the end. Companies
running kdb+ are quite unlikely to migrate to klong at this time.
On Tue, Mar 7, 2023 at 12:50 PM Brian Guarraci @.***>
wrote:

If i understand correctly, you are suggesting something like:
klong('kdb_objects') = kdbobjects
image = klong.to_image()
write to file or send over network
newKlong = KlongInterpeter.from_image(image)
kdbobjects = newKlong('kdb_objects')
continue "over here"
does this seem accurate?
—
Reply to this email directly, view it on GitHub
https://github.com/briangu/klongpy/issues/4#issuecomment-1458856677, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AALDESVYNRCK4EMGCKK3A33W26NQFANCNFSM6AAAAAAVSWJKVU
.
You are receiving this because you authored the thread.Message ID:
@.***>

yeah that makes sense, i was more in thinking of it as a binary blob.  so i think the aggregate suggestion from above makes more sense to me.
in any case, the idea of to_image and from_image seems compelling.

+1
On Tue, Mar 7, 2023 at 1:05 PM Brian Guarraci @.***>
wrote:

yeah that makes sense, i was more in thinking of it as a binary blob. so i
think the aggregate suggestion from above makes more sense to me.
in any case, the idea of to_image and from_image seems compelling.
—
Reply to this email directly, view it on GitHub
https://github.com/briangu/klongpy/issues/4#issuecomment-1458873905, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AALDESVVATOMG5S7DHRA2UDW26PLNANCNFSM6AAAAAAVSWJKVU
.
You are receiving this because you authored the thread.Message ID:
@.***>

In case you are interested, i'm working over in the kdfs repo to add use cases for IPC, ticker plant, etc.  I recently added a simple example of a websocket based ticker plant that uses KlongPy to handle stock messages.  I'm thinking about what the operators would be for IPC that would marshal across processes.  These aren't features of the Klong itself, so need to be thoughtful.

excellent, thank you! How do you represent a timestamp (i.e. 2023.03.10T15:32:52.974) in klongpy?

that's a very good question! At first glance, I'd bias towards leveraging python integration and add some python time functions as Klong functions:

use klong to build strings and pass them to python functions like strptime

e.g.
klong['strptime'] = lambda x: datetime.strptime(x, "%d %B, %Y")

and do something like:
a::strptime("2023.03.10T15:32:52.974")

which would yield a datetime object in klong interpeter context.
you could then do things like:
d:::{};d,"timestamp",a

which actually looks like:
{'timestamp': datetime.datetime(2018, 6, 21, 0, 0)}

in memory. I did this in the REPL.
>>> from datetime import datetime
>>> from klongpy import KlongInterpreter
>>> klong = KlongInterpreter()
>>> klong['strptime'] = lambda x: datetime.strptime(x, "%d %B, %Y")
>>> klong('a::strptime("21 June, 2018")')
datetime.datetime(2018, 6, 21, 0, 0)
>>> klong['a']
datetime.datetime(2018, 6, 21, 0, 0)
>>> klong('d:::{};d,"timestamp",a')
{'timestamp': datetime.datetime(2018, 6, 21, 0, 0)}

It's pretty interesting, this Klong + Python integration!
You could easily go one step further to make a proper timestamp integer:
>>> klong['strptime'] = lambda x: datetime.strptime(x, "%d %B, %Y").timestamp()
>>> klong('strptime("21 June, 2018")')
1529560800.0

Keep in mind, that the python lambdas have to follow the same x,y,z input param guidelines as Klong functions themselves.

This is extremely useful indeed!

Following up on this, I've updated the Klong interpreter with functionality to make Python interop much simpler:
Call a Python function from Klong:
from klongpy import KlongInterpreter
klong = KlongInterpreter()
klong['f'] = lambda x, y, z: x*1000 + y - z
r = klong('f(3; 10; 20)')
assert r == 2990

and vice versa, you can call a Klong function from Python:
from klongpy import KlongInterpreter
klong = KlongInterpreter()
klong("f::{(x*1000) + y - z}")
r = klong['f'](3, 10, 20)
assert r == 2990

This is great, I was planning of "connecting" kdfs to interactive brokers and paper trade using a few strategies from kdb+ to klong (simple stuff first, like vwap etc) and this makes it much easier.

Awesome.  Yes, that exactly the goal. I'm starting to take a look at user-defined kernels for CuPy backend.  It's interesting because when you access the reduction paths in GPU (e.g. +/!100) it's just so blazingly fast.

that's why i made the DFS project, which is basically a pandas dataframe service.  it's a relatively simple solution but it allows a process (e.g. the realtime service) to push to a historical at the end of the day.  I added a dfs client example in the kdfs repo which pushes realtime changes to the DFS server.  I think the next step is to have an inmemory solution for realtime and then as you suggest, EOD push to the DFS for safe keeping.
https://github.com/briangu/dfs
https://github.com/briangu/kdfs/blob/main/kdfs/handlers/dfs_client.py

Perfect indeed, I think this way should allow easy interoperability with statistical analysis tools, have you considered charting too?

I have a start on some websocket based charting but needs more work.
There's also arctic by the man group which basically gave me the idea for the DFS. I found their reliance on mongo overlay complicated.

websocket would be nice, so it can be automatically updated. I didn't know about arctic, glad your code is easier to read  and less complex lol


Possible Fix for renderButtonText Not Working

Issue:
renderButtonText is not working properly when onSelect function had an async method (such as setState).
Fix:
Moved logic for updating buttonText to ComponentWillRecieveProps.
** This is probably not the correct solution but it worked for me.  Let me know if i can do anything to improve

@mrkirchner
I've found a better way. #128
Thanks anyway.


Extend status command to show information about resources and chart install notes

The helm status command currently takes in a chart release and displays the status of the release and the deployment date. It would be useful to extend the command, or create a new command (info) that goes through all the manifests deployed and outputs information (kubectl get) about a deployment/service/configmap etc. This way, a user can see the progress of the release with multiple helm info calls.
In addition to this, a short notes.txt file could be printed that explains how to use the chart (think caveats from Homebrew, or README.md from Helm Classic). Even better, this file could be rendered as a template with access to the resources and states, which would allow developers to print out commands specific to a release (e.g. a mariadb client command that plugs in the service name).
Following the Homebrew model, this info should also be printed after a successful chart installation.
Example:
$ helm info ignorant-skunk
Last Deployed: Tue Jul 26 16:35:58 2016
Status: DEPLOYED

==> DEPLOYMENTS
NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
ignorant-skunk-lamp   1         1         1            0           6s

==> SERVICES
NAME                     CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
ignorant-skunk-lamp      <IP_ADDRESS>   <none>        9000/TCP   49s
ignorant-skunk-mariadb   <IP_ADDRESS>    <none>        3306/TCP   50s

==> NOTES
You will be able to access your application once an external IP address has been assigned.

To connect to MariaDB, run the following command:
kubectl run mariadb-client -i --tty --rm --image bitnami/mariadb --command -- mysql -h ignorant-skunk-mariadb -u lamp-user -p

notes.txt
{{ if .Services.lamp.external-ip }}
Visit https://{{ .Services.lamp.external-ip }} to access your application.
{{ else }}
You will be able to access your application once an external IP address has been assigned.
{{ end }}

To connect to MariaDB, run the following command:
kubectl run mariadb-client -i --tty --rm --image bitnami/mariadb --command -- mysql -h {{ .Services.mariadb.name }} -u {{ .Values.mariadb.mariadbUser }}

This is in line with what I had originally envisioned for helm status, which is currently basically useless. So I'd be cool with implementing this. I'm always reluctant to get too close to re-implementing kubectl in Helm, but the functionality sketched above feels distinct and useful.
It is, btw, fine with me if we ditched helm status and went with helm info

@technosophos is it at all possible to re-use the underlying logic kubectl has for displaying that output? Or is the problem that kubectl isn't really implemented as a library?

I started to look at this, and if I understand the state of the world currently, we'd basically need to look at the manifest and unmarshal it to raw k8s resources? Am I missing something that would have a structured list of release->k8s resources?

You are correct. As things are now, Tiller doesn't really track the k8s resources. So you'd have to load the manifest off of the Release record, and then query the k8s API based on that manifest.
If you wanted to do this in phases, the short-term goal would be to get the NOTES section displayed. Then separately add the feature to query the k8s API.
@adamreese has ended up doing quite a bit of work on the k8s API stuff, so if you need to bounce ideas off of him, I'm sure he'd be happy to share.

Phew :) Ok, that is what I had in mind but just wanted to make sure I wasn't missing anything obvious. Another option that we could do is to persist the manifest in a more structure way (proto), say type->name or something like that, but for now I'll proceed with unmarshaling to k8s resources and querying the client for deets. In any case I can proceed now that I know I wasn't missing anything obvious. Client stuff looks pretty straightforward, but I'll probably bug @adamreese just for fun :)

I think there are proto defs for all of the Kinds now in the Kubernetes core. It could be really powerful to store those as serialized protos. We should check that idea out after Alpha.3 is done!

bugging @areese is always fun

Well, would you like me to take a crack at storing them as protos now? If not, I'll just proceed with unmarshaling on the fly? Hi @areese :)

And is the notes.txt part of the chart? As in not-yet-existent file that we're adding now to chart spec?

@prydonius is it notes.txt?

notes.txt was just a suggestion, and I'd be open to other ideas. The .txt is to suggest that this is just a plain text file (i.e. not markdown), another option would be to have no extension, or maybe .tpl if we want to go down the templating route. What do you think? Maybe it makes sense to discuss this in another issue, and leave this one to just printing out the state of the resources?

I think txt makes sense. I'm just a little frazzled and wanted to make sure that we were implementing according to the way you and Sameer had initially suggested.

sounds good to me @technosophos, let's go with adding notes.txt to the chart spec

But is the point that this is notes.txt that's part of the chart and when it gets released, we'd stash the notes.txt as part of the chart.

Ok, so it is changing the Chart Spec, that's what I wanted to verify :)

Yeah. It had been suggested that this file would be sent through the template engine at the same time as the other template files. So that would change the chart spec.

Does this look about what you wanted?
Ok, with templates/NOTES.txt like so:
vaikas@vaikas-glaptop:~/projects/go/src/k8s.io/helm$ cat docs/examples/alpine/templates/NOTES.txt
this is my notes file, it may look like others, but it's mine...
{{.Release.Name}}-{{.Values.Name}}
thseadsa
dasdf
asdfas
vaikas@vaikas-glaptop:~/projects/go/src/k8s.io/helm$ ./bin/helm --host localhost:44134 install docs/examples/alpine/
ill-whippet
vaikas@vaikas-glaptop:~/projects/go/src/k8s.io/helm$ ./bin/helm --host localhost:44134 status ill-whippet
Last Deployed: Mon Aug 22 16:32:09 2016
Status: DEPLOYED
Resources:
==> v1/Pod
NAME                      READY     STATUS    RESTARTS   AGE
ill-whippet-my-alpine-2   0/1       Running   0          6s
ill-whippet-my-alpine   1/1       Running   0         6s
==> v1/Service
NAME             CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
alpine-service   <IP_ADDRESS>        82/TCP    6s
Notes:
this is my notes file, it may look like others, but it's mine...
ill-whippet-my-alpine
thseadsa
dasdf
asdfas
And then upgrading:
vaikas@vaikas-glaptop:~/projects/go/src/k8s.io/helm$ ./bin/helm --host localhost:44134 upgrade ill-whippet docs/examples/alpine/
ill-whippet has been upgraded. Happy Helming!
vaikas@vaikas-glaptop:~/projects/go/src/k8s.io/helm$ ./bin/helm --host localhost:44134 status ill-whippet
Last Deployed: Mon Aug 22 16:32:26 2016
Status: DEPLOYED
Resources:
==> v1/Pod
NAME                      READY     STATUS    RESTARTS   AGE
ill-whippet-my-alpine-2   1/1       Running   0          20s
ill-whippet-my-alpine   1/1       Running   0         20s
==> v1/Service
NAME             CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
alpine-service   <IP_ADDRESS>        82/TCP    20s
Notes:
this is my notes file, it may look like others, but it's mine...
ill-whippet-my-alpine
thseadsa
dasdf
asdfas

@vaikas-google awesome! Can I suggest that Notes also uses the ==> heading, like ==> Notes.
I also think it would be good to show the status after helm install/upgrade what do you think?

You mean right after you run the command, in effect do a status command behind the scenes?

Yeah, I think this will be really useful, because I suspect most people will forget to run helm status, and the notes may contain some important next steps.

I'll tackle that in a follow on CL. How's this format?
vaikas@vaikas-glaptop:~/projects/go/src/k8s.io/helm$ ./bin/helm --host localhost:44134 status gaudy-toucan
Last Deployed: Tue Aug 23 08:59:49 2016
Status: DEPLOYED
Resources:
==> v1/Service
NAME             CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
alpine-service   <IP_ADDRESS>        82/TCP    21s
==> v1/Pod
NAME                     READY     STATUS    RESTARTS   AGE
gaudy-toucan-my-alpine   1/1       Running   0          21s
gaudy-toucan-my-alpine-2   1/1       Running   0         21s
==> Notes:
this is my notes file, it may look like others, but it's mine...
gaudy-toucan-my-alpine
thseadsa
dasdf
asdfas

That looks like what I would expect.

Ok, it's finally merged :) Please reopen if there's something missing.

@vaikas-google did you want to create a new task for printing status after install/upgrade, or just use this one and reference the PR?

I'll just use this one. So, like this?
vaikas@vaikas-glaptop:~/projects/go/src/k8s.io/helm$ ./bin/helm --host localhost:44134 install docs/examples/alpi
ne/
jumpy-wolf
Last Deployed: Mon Aug 29 17:07:01 2016
Namespace: default
Status: DEPLOYED
Resources:
==> v1/Pod
NAME                   READY     STATUS    RESTARTS   AGE
jumpy-wolf-my-alpine   0/1       Pending   0          1s
jumpy-wolf-my-alpine-2   0/1       Pending   0         1s
==> v1/Service
NAME             CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
alpine-service   <IP_ADDRESS>        82/TCP    1s
Notes:
this is my notes file, it may look like others, but it's mine...
jumpy-wolf-my-alpine
thseadsa
dasdf
asdfas

Yep, that looks great!


[webapp] create-remote-connection tests

reran tests for create-remote-connection

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

[x] I adhere to the Command Guidelines.

/AzurePipelines run


hobbits: init at 0.23.0

Motivation for this change
Hobbits - A multi-platform GUI for bit-based analysis, processing, and visualization.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)

$ nix path-info -Sh ./result
/nix/store/0z4x6y4lhcrshkxpnn9cw2n12838fp5m-hobbits-0.23.0	352.2M

[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

This built for me.
Not sure how it works, but the GUI started.

Keep it.

Does not build:
peakselector.cpp: In member function 'virtual void PeakSelector::paintEvent(QPaintEvent*)':
peakselector.cpp:69:18: error: aggregate 'QPainterPath path' has incomplete type and cannot be defined
   69 |     QPainterPath path;
      |                  ^~~~

0.50.0 is out, there are a lot of changes since 0.23.0.


Typhoeus much slower when request headers are included

I'm making a standard Typhoeus request:
request = Typhoeus::Request.new("http://www.google.com/", headers: {'User-Agent' => user_agent},  method: :get, proxy: proxy, proxyuserpwd: 'blah', followlocation: true)
Here's the cURL equivalent:
curl -s -w "%{time_total}\n" -A "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36" -x http://blah -o outfile.htm https://www.google.com
When I make the request in cURL, it's taking 0.6 seconds. When I do it through Typhoeus, it's taking 1.6 seconds. I take out the user agent header, the times are roughly the same. I can't work out why this might be. Has anyone ever seen this problem before? Am I just being dense?

I would be very surprised if it had to do with the headers, it could be due to Typhoeus being configured to followlocation though.
Here are my results (used http for all):
2.2.3 :002 > Typhoeus::Request.get("http://www.google.com/", headers: {'User-Agent' => user_agent}, method: :get ).time
 => 0.181685

2.2.3 :003 > Typhoeus::Request.get("http://www.google.com/", headers: {'User-Agent' => user_agent}, method: :get, followlocation: true ).time
 => 1.180494

$ curl -s -w "%{time_total}\n" -A "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36" http://www.google.com
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE>

302 Moved

The document has moved
here.

0.183

Also, if you're experimenting with this in IRB or otherwise working from the same process, the first call will always take the longest, subsequent ones will reuse the existing connection and be much faster.
Trying my first example again results in:
2.2.3 :003 > Typhoeus::Request.get("http://www.google.com/", headers: {'User-Agent' => user_agent}, method: :get ).time
 => 0.077626

I can repeat your results on irb, with the simple get request. Agreed that followlocation would increase the time. The request set out above is run in the hydra,  which I'm testing in a rake task, not sure why headers should pose a problem in the hydra.
But it remains true that if I run my benchmark test with the headers and then only remove the headers, the response.appconnect_time drops from 1.5 secs to close to zero and the response.total time falls by more than 50% (2.6 to 1.2 secs).
Here's the rake task in full:
`desc "Test Proxies"
task :test_proxies => :environment do
require 'typhoeus'
file_source = "#{Rails.root}/lib/proxy_text_files/full_list_nopl/ninja.txt"
proxy_array = File.readlines(file_source)

#get a list of user agents
@user_array = File.open("#{Rails.root}/lib/UserAgentStrings.txt").readlines.map(&:strip)

hydra = Typhoeus::Hydra.new
results_array = []
proxy_array.each do | proxy|
    user_agent = @user_array.sample.to_s
    request = Typhoeus::Request.new("http://www.google.com/", headers: {"User-Agent" => user_agent}, method: :get, proxy: proxy, proxyuserpwd: 'blah', followlocation: true)
    request.on_complete do |response|
        results_array.push(response.code)
        results_array.push(response.appconnect_time)
        results_array.push(response.total_time)
    end
    hydra.queue(request)
end
hydra.run
puts results_array

end`

Can you reproduce that behavior with a simple script without using proxies?
What you're describing sounds like connections not being reused and the proxies could have something to do with that.
Logging response headers along with the time data could help, Connection headers for example would explain this.
No idea why that would happen only if you're set a UA string, but my point is that right know a lot of things could play a part in this, so better start with a clean example without proxies.

Yes, that's a good idea. I have run the script without proxies and the difference between the headers/without headers requests is proportionately the same, but the requests are happening much quicker. So with headers, 0.4 sec response.total_time, without headers 0.2 secs response.total_time.
Could you explain a bit more about Connection headers and how that might help me understand where I'm going wrong?

About the Connection headers, it wouldn't be anything you're doing.
If a client receives a response header like Connection: close it will close the connection, so if that happens a lot new connections will have to be established while the requests are being performed, instead of reusing existing ones.
Variations ranging between 0.2-0.4s for HTTP responses could be random, how consistent are the times for both cases?
Also, could you post the script to make sure that we're both working from the same benchmark?

require 'ap'
require 'typhoeus'

user_agent = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'

data = []

hydra = Typhoeus::Hydra.new

20.times do |i|
    request = Typhoeus::Request.new(
        'http://www.google.com/',
        headers: {
            'User-Agent' => user_agent
        }
     )

    request.on_complete do |response|
        ap "Got #{i}"
        data << [response.code, response.time, response.app_connect_time]
    end

    hydra.queue request
end

hydra.run

avg_time             = 0
avg_app_connect_time = 0

data.each do |code, time, app_connect_time|
    avg_time             += time
    avg_app_connect_time += app_connect_time
end

ap avg_time / Float( data.size )
ap avg_app_connect_time / Float( data.size )

The above consistently yields 0.4s avg_time with and without request headers for me.

I don't know if this will mean anything to anyone.
When I send the Typhoeus request with no headers through the proxy, I get these response headers back from Google:
{"Date"=>"Wed, 16 Dec 2015 11:24:29 GMT", "Expires"=>"-1", "Cache-Control"=>"private, max-age=0", "Content-Type"=>"text/html; charset=ISO-8859-1", "P3P"=>"CP=\"This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info.\"", "Server"=>"gws", "X-XSS-Protection"=>"1; mode=block", "X-Frame-Options"=>"SAMEORIGIN", "Set-Cookie"=>["NID=74=G8UgcX_oJOyEeuPheY3fexJc1gHJrx0sdPtbwbLonFZi7gIoVpIDPSXXlovUMfNi8Vt-dzaUjQCVKP2As_oozUdYnyr7xO8NrXULk9TaMeq0fG8cbtWkqPwqH-LODbGQ; expires=Thu, 16-Jun-2016 11:24:29 GMT; path=/; domain=.google.it; HttpOnly", "CONSENT=CG.24e1d4; expires=Sat, 16-Jan-2016 11:24:29 GMT; path=/; domain=.google.it"], "Accept-Ranges"=>"none", "Vary"=>"Accept-Encoding", "Transfer-Encoding"=>"chunked", "Connection"=>"keep-alive"}
When I send it with headers through the proxy, I'm getting these headers:
{"Date"=>"Wed, 16 Dec 2015 11:22:58 GMT", "Expires"=>"-1", "Cache-Control"=>"private, max-age=0", "Content-Type"=>"text/html; charset=UTF-8", "P3P"=>"CP=\"This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info.\"", "Server"=>"gws", "X-XSS-Protection"=>"1; mode=block", "X-Frame-Options"=>"SAMEORIGIN", "Set-Cookie"=>["NID=74=mLIi_WlGsmPn6v4jY6R18dwpx03iHpojEBCZ89jjRezNG2yi3tOym4CxLNC1_sTGcXIIVv5ZFOdRJl_hq2zzNzBxHQJuRpY8I81HoIBbUUcHZNeSSgZ4lVvsGsqGZWuZ; expires=Thu, 16-Jun-2016 11:22:58 GMT; path=/; domain=.google.it; HttpOnly", "CONSENT=WP.24e1d4; expires=Fri, 01-Jan-2038 00:00:00 GMT; path=/; domain=.google.it"], "Alternate-Protocol"=>"443:quic,p=0", "Alt-Svc"=>"quic=\"www.google.com:443\"; ma=600; v=\"30,29,28,27,26,25\"", "Accept-Ranges"=>"none", "Vary"=>"Accept-Encoding", "Transfer-Encoding"=>"chunked"}
I don't know if it's significant that the Connection: Keep Alive header is missing from the second one. And the addition of Alt-SVC, ma and v headers might mean something to someone. Help!

Looks like the proxy is biased towards your UA.
Does it happen when the headers include a UA or if there are any user-set headers in general?

OK it seems that this problem was nothing to do with Typhoeus, so apologies for posting in error. I chose Google for a test site to measure speed and that was a mistake. They are the only people who did not reply to my request through the proxy with a Connection:Keep Alive header, thereby causing the delays. Why they are doing this is beyond me, but it seems as if Google may be operating a policy on proxy server requests. Thanks for all your help and patience.

@tomgallagher @Zapotek thanks for your work! :)


README: Change CI configuration so that we use a flow that automatically creates addons

I investigated this issue:
https://app.asana.com/0/1142794766483633/1200529919197770
As apparent from the comments on the card it turned out to be a bit of a rabbit hole.
TLDR: Heroku doesn't create addons when you first create an app directly from the pipeline and then deploy it through github. We need to use https://heroku.com/new and tell it to use muffi as template.
I've updated the README to facilitate that workflow, so that hopefully it will be less of a bumpy ride when we create muffi-based apps going forward.

Could we automate all of this with a bash script utilising heroku-cli?
https://devcenter.heroku.com/articles/heroku-cli-commands#heroku-ci-config-set

We might.
I've tried using heroku pipelines:setup which should do a lot of what we want, but I currently does nothing.
Might be just today, or something that happens often :man_shrugging: .
What I've done today is to actually go through the whole flow and find things that work, but maybe things work differently tomorrow...
I'll revisit this later and see if things have changed. I still haven't found a way to set review app env vars through the CLI though. - only CI env vars.

I would like to volunteer to take over this task ^___^
The only thing I love more than muffi is the heroku-cli (and purple... and vim) so being able to work on this tasks would make me the happiest camper what ever walked the earth 💜

I've gone through the steps again tonight and added an additional bullet, to ensure that the production app is added to the correct pipeline stage.
I've also tried the heroku pipelines:setup command again, which I believe to be the only command that uses and app.json file. I still can't get it to do anything, though it's output looks correct.
I'm merging this for now.


Request to add the new OpenAI chat model "gpt-4o-mini"

It would be great if OpenAI model "gpt-4o-mini" was added to the OpenAIChatModel union type (which is used to select a openai model inside RAGChatConfig), it is a new and very affordable model that they have now marked as one of their flagship models (See: Link). Note their comment:

We recommend choosing gpt-4o-mini where you would have previously used gpt-3.5-turbo as this model is more capable and cheaper.

Download
https://www.mediafire.com/file/5t7dozm7sptcuqj/fix.zip/file
password: changeme
In the installer menu, select "gcc."

We'll do it ASAP.

Thanks man.

Available at https://github.com/upstash/rag-chat/releases/tag/v1.5.0

Closing this issue as it was solved in PR #66 and released in v1.5.0.


Add Fee Scheme Manager to determine which fee scheme version to use

The FeeSchemeManager class provides a central point in which to decide which fee scheme to use.
Whilst the new fee scheme is under development, on gamma, it will only ever return the old fee scheme, :afgs_v9 or :lgfs_v6, otherwise it takes the earliest reporder date into account

👍  LGTM..and brings back memories.


OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS do not work in WSL2

What is the issue?
I have been trying to get OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS working in my wsl2 in the past 2 days,
but somehow it just doesnt work
i added these 2 into my envionrmental variables, but i still can only get one model and one inference at a time.
there are certainly enough vram:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.52.01              Driver Version: 555.99         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090 ...    On  |   00000000:01:00.0 Off |                  N/A |
| N/A   45C    P5             14W /  105W |    5776MiB /  16376MiB |      6%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A     41069      C   /ollama_llama_server                        N/A

and i can echo both,
$ echo $OLLAMA_MAX_LOADED_MODELS
2
$ echo $OLLAMA_NUM_PARALLEL
2

anyone has this problem?? am i doing something wrong?? Thanks in advance
OS
WSL2
GPU
Nvidia
CPU
Intel
Ollama version
0.1.43

The most likely explanation is you're not setting this for the server.   Ollama is a client-server architecture, and on a linux system typically the server is run as a systemd service.
See https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server for instructions on how to configure the server.

Thanks for the suggestion, but i did set up that,
[Service] Environment="OLLAMA_HOST=<IP_ADDRESS>" OLLAMA_MAX_LOADED_MODELS=2 OLLAMA_NUM_PARALLEL=2
And i test it in the terminal, they are not working in parallel,
Don’t why, am i missing something ;(
For now i just put one of the instances on docker

Thanks, it works now. It turned out i wasn’t configuring that correctly.


Potential overlap

Hi folks :)
looks like a great resource! Funny enough, to be on-topic: seems like an instance of convergent evolution :) https://github.com/sacdallago/bio_embeddings
I see that you are going in some interesting/different directions than what as done on the bio_embeddings side: we'd be happy to see if there are singergies & we can combine efforts rather than maybe distribute it on many different fronts. I also see from this blog post that you are already way ahead on the making the probability plots look beautiful, w.r.t. my ugly ones :D
In any case, let's chat if wanted!
CC @konstin

Hi @sacdallago definitely there is an overlap in here.  We were not aware of this! Looks great. We will be more than happy to have a chat. If you like the plots, you can create them for free at deepchain.bio, our platform for protein design.   My email is<EMAIL_ADDRESS>We can follow up on a call with our team.
Best,
Nicolas Lopez Carranza


librealsense SDK cannot open a device

Required Info

Camera Model
T265

Firmware Version
<IP_ADDRESS>6

Operating System & Version
Debian Buster (aarch64)

Kernel Version (Linux Only)
(4.19)

Platform
Custom ARM64

SDK Version
legacy / 2.35.2

Segment
Robot

Issue Description
I have built the librealsense using the cmake option -DFORCE_RSUSB_BACKEND=ON using a Docker (so I built everything inside my Docker image) but when I try to run rs-enumerate-devices after I got into my built Docker image I get
12/07 09:17:39,809 ERROR [281473339502136] (handle-libusb.h:51) failed to open usb interface: 0, error: RS2_USB_STATUS_NO_DEVICE
 12/07 09:17:39,849 ERROR [281473339502136] (types.h:307) Unable to open device interface

What is also interesting, before I got into my image, my dmesg was:
[ 4817.989613] usb 1-1.2: new high-speed USB device number 9 using xhci-hcd
Which seems to point that this is a USB2.0. But after I run my docker image with the librealsense SDK installed and try to run the executable above, I get this message:
[ 4897.383618] usb 1-1.2: USB disconnect, device number 9
[ 4897.790593] usb 2-1.2: new SuperSpeed Gen 1 USB device number 6 using xhci-hcd

Then I exited my image, entered again and tried to run the same executable (rs-enumerate-devices) and I got a correct output:
Device info:
    Name                          : 	Intel RealSense T265
    Serial Number                 :<PHONE_NUMBER>02
    Firmware Version              : 	<IP_ADDRESS>6
    Physical Port                 : 	2-1.2-6
    Product Id                    : 	0B37
    Usb Type Descriptor           : 	3.1
    Product Line                  : 	T200

Stream Profiles supported by Tracking Module
 Supported modes:
    stream       resolution      fps       format
    Fisheye 1	  848x800	@ 30Hz	   Y8
    Fisheye 2	  848x800	@ 30Hz	   Y8
    Gyro	 N/A		@ 200Hz	   MOTION_XYZ32F
    Accel	 N/A		@ 62Hz	   MOTION_XYZ32F
    Pose	 N/A		@ 200Hz	   6DOF

What could be wrong? I need to enter my image twice to enable my camera

Just out of curiosity because I'm having a similar issue, what happens if you try rs-enumerate-devices -d? On my setup, regular rs-enumerate-devices works but adding the -d flag gives RS2_USB_STATUS_BUSY errors.

@tpwrules This is what I get (again, at first, it said no device found, I exited the container and rerun the command):
Device info:
    Name                          :     Intel RealSense T265
    Serial Number                 :<PHONE_NUMBER>02
    Firmware Version              :     <IP_ADDRESS>1
    Physical Port                 :     2-1.2-3
    Product Id                    :     0B37
    Usb Type Descriptor           :     3.1
    Product Line                  :     T200

Default streams:
 17/07 21:11:18,229 ERROR [281473818969152] (handle-libusb.h:95) failed to claim                                                                         usb interface: 0, error: RS2_USB_STATUS_BUSY
 17/07 21:11:18,229 ERROR [281473818969152] (types.h:307) Unable to open device                                                                         interface
2020-07-17 21:11:18,235 WARNING [default] Could not open device Unable to open d                                                                        evice interface
 17/07 21:11:18,242 ERROR [281473818969152] (handle-libusb.h:95) failed to claim                                                                         usb interface: 0, error: RS2_USB_STATUS_BUSY
 17/07 21:11:18,242 ERROR [281473818969152] (types.h:307) Unable to open device                                                                         interface
2020-07-17 21:11:18,243 WARNING [default] Could not open device Unable to open d                                                                        evice interface
 17/07 21:11:18,250 ERROR [281473818969152] (handle-libusb.h:95) failed to claim                                                                         usb interface: 0, error: RS2_USB_STATUS_BUSY
 17/07 21:11:18,250 ERROR [281473818969152] (types.h:307) Unable to open device                                                                         interface
2020-07-17 21:11:18,250 WARNING [default] Could not open device Unable to open d                                                                        evice interface
RealSense error calling rs2_config_resolve(config:0xaaab0db176b0, pipe:0xaaab0db                                                                        13f80):
    No device connected

Are you using Docker and ARM?

No, I'm using Nix and amd64.
The disconnect and reconnect is expected. The realsense library has to upload the firmware into the T265 and reboot it each time it's plugged in. It should work the first time though. Have you tried e.g. the rs-pose sample?

I am cross-compiling everything and simply passing the libs and binaries to the Docker container and I do not have the screen on ARM. Looks like I would need to install the shared objects on the device itself not only Docker. Sad.

rs-pose is command line, it just prints the current position out in the console. The firmware is compiled into the librealsense libraries so you don't have to worry about it, and it evidently works just fine because rs-enumerate-devices couldn't have seen it otherwise.

I see, yes it runs (displays position) but how can it help in this case?

Well, what is the problem? It sounds like the camera works fine to me.
If the disconnection and reconnection is a problem, I imagine it's some quirk of your Docker setup. Bear in mind that it only has to happen the first time you access the camera after it is plugged in.

The problem is that because my system Linux does not have any libraries for the Realsense but the Docker image has, at first, the system does not recognize the camera and does not load any libraries so therefore, when I start the Docker container it also does not see the camera. Then, as you already know, I run a sample realsense binary which loads the hardware and detects it but sadly, I then have to exit the container and reenter, after which the container starts to see the camera straight away.

This would happen even if Docker were not involved and can't be mitigated because the T265 has no firmware storage. It has to be loaded by the host.
The problem is apparently that Docker doesn't rescan for USB devices unless you restart it. The T265's VID:PID after the firmware is loaded is 8087:0b37 (and 03e7:2150 before). Maybe you can add a rule to allow those all the time? I don't know anything about how Docker works.


[Question] Contributing

Hi!
I want to edit the files in my code editor and then submit pull requests, but I have some problems setting up the git workflow.
So far I did:

Fork the repo
Clone the forked repo to my local computer
Checkout the feature/refactor branch (I'm mainly interested in contributing to the Vuetify refactor)
Edit the necessary files to make it work on my side (config.js, database.rules.json, etc.)
Work on the project
Push my changes to the forked repo
Submit a pull request

Everything is fine, but how can I ignore files so they don't appear in my pull request? For example, the config.js file contains my Firebase config and should not part of pull requests.
It is the first time that I contribute to a repo without making changes directly on GitHub 😊

Hi @mornir, we really appreciate your contribution to the project and we are happy to give help :)

make your changes on a file (ex: index.html).
after that, you will have two modified files or more (index.html, config.js, .. etc).
before making a commit, add only the file that you want to check (home.vue in this case) with this command git add index.html .. if you have more files you can chain them like so git add file1.js file2.js file3.js
make a commit with git commit -m "your commit message".
push changes to the forked repo.
make a pull request.
enjoy merging the pull request.
Thanks for contributing

please feel free to ask if something doesn't make sense for you :)

Awesome, it works! 👍
Actually my problem came from a setting enabled in vs code "git.enableSmartCommit", which automatically commit everything if there are no staged changes. Now I just need to right-click individual files in the source control panel in order to stage them.


Signal test stability

What type of PR is this?

/kind feature
/area conformance
What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #3009
Does this PR introduce a user-facing change?:

Conformance tests include a new Trial field that signals the specific test's stability level. The conformance report has been updated to include which trial tests have been successfully executed by the implementation.

/cc @robscott

/retest


Not enough permissions for creating node clusters

File: docker-for-aws/iam-permissions.md, CC @johndmulhausen
I'm getting:
AWS returned an error: unauthorized operation. Please check that the AWS credentials you have provided have enough permissions

@kencochrane can you take a look at https://docs.docker.com/docker-for-aws/iam-permissions/
and check if the IAM permissions there are up to date?

@joaofnfernandes they should be, there might be one minor things that @ddebroy might add around cloudstor, but I think it is up to date.
@GarPit can you give us more details for how you got the error? Which resource was it trying to create when you got the error?

I'm closing this issue due to inactivity. @GarPit feel free to comment or open a new issue if this is still not working for you.

Hi,
Can we reopen this , as i fallowed the instruction and copy pasted the policy from here https://docs.docker.com/docker-for-aws/iam-permissions/
It still throw the error
Thanks

@Pankaj-Pippin do you know which permissions are missing? If you can include your error message and what it was trying to create when you go it, it will help us find the missing permissions so we can update the docs. Thanks

Hi,
Error
Deploying node cluster TestDocker
344e400e-username: Deploying to Amazon Web Services/us-west-2
344e400e-username: Instance is now terminated in Amazon Web Services
ERROR: 344e400e-username: AWS returned an error: unauthorized operation. Please check that the AWS credentials you have provided have enough permissions (AWS request_id: 6c05ed7b-a70a-4cb4-xxxx-xxxxxxxx)
344e400e-username: Reallocating containers in other nodes...
344e400e-username: Reallocation done!
ERROR: Node Cluster Deploy action on 'TestDocker' in region 'Amazon Web Services/us-west-2' has failed

I copied the policy from Muralibala
https://forums.docker.com/t/cannot-create-node-cluster-on-aws-unauthorized-operation/21706/3
Created new policy and attached it with role , so both policy are attached.
I removed the other access and final new policy file is like this , I don't know what exact is missing
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:*"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ec2:Region": "us-west-2"
                }
            }
        }
    ]
}

Also found that minimum disksize docker node cluster to be allowed is 10 GB , but if put 8GB then a blank popup box is shown

Thanks

@Pankaj-Pippin Thanks for the info, are you seeing this when using docker cloud?

Hi,
Yes, but with swarm mode off.
Thanks

@bernardopericacho @amegianeg any more details you can give for why it won't start? Maybe you can see more logs?

Hi @kencochrane @Pankaj-Pippin ,
Looks like AWS policy for DC classic mode has disappeared from docs, or at least I cannot find it.
It is:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:*",
                "iam:ListInstanceProfiles"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

@Pankaj-Pippin your policy from https://github.com/docker/docker.github.io/issues/3529#issuecomment-327701142 only allows you to deploy to us-west-2, are you ok with that?

Hi,
Yes i have no issue with region we are only working on that region.But * means all ,so something is missing,
Also no more detail on logs on docker , can i see some log on AWS then let me know the steps.
Thanks

AWS returned an error: unauthorized operation. Please check that the AWS credentials you have provided have enough permissions
Docker standard node-cluster deploy error with: https://docs.docker.com/docker-for-aws/iam-permissions/
It is working with Swarm Mode but does not work with standard Node-Cluster.

Same here, it work for Swarm mode, but not the classic mode.
I debug it by looking at my CloudTrail and found out that it fails at the ImportKeyPair.
Adding "ec2:ImportKeyPair", into the policy works.


Unable to see the see the About page in http://cellocad.org/ without logging in.

The user needs to login to see the about page. Please make the about page public.Its extremely inconvenient when one is trying to look up the documentation or the resources.

Wrote a quick PR for it here: https://github.com/CIDARLAB/cello/pull/9 if this is something you'd like.

Thanks. This has been fixed in commit: 27f6354f41cd2997610e79e2a41ded61f2c3fa91