GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v5.15.11",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=dfc3fff63793c571147930b13c0f8c689c4281ac"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "dfc3fff63793c571147930b13c0f8c689c4281ac",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "scsi: scsi_debug: Sanity check block descriptor length in resp_mode_select()\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.15.11 by commit dfc3fff63793c571147930b13c0f8c689c4281ac. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2021-1002768


[BAD-587] - PMR Mapper log is full of errors related to org.eclipse.jface.window.ApplicationWindow

pmr job log full of exceptions when some common step classes contains import to swt libraries
localized needed for OpenMappingExtension is from spoon package,
so to fix that situation and have correct localized message PKG is now SpoonLifecycleListener which doesn't have import for swt

Build Completed
:white_check_mark: This pull request has passed all validations.
Build Commands
mvn -B -fn -f plugins/meta-inject-plugin/pom.xml clean install && mvn -B -f plugins/meta-inject-plugin/pom.xml site

Cleanup Commands
mvn -B -f plugins/meta-inject-plugin/pom.xml build-helper:remove-project-artifact

Changed files
  plugins/meta-inject-plugin/src/main/java/org/pentaho/di/trans/steps/metainject/OpenMappingExtension.java
  plugins/meta-inject-plugin/src/test/java/org/pentaho/di/trans/steps/metainject/OpenMappingExtensionTest.java

Unit test coverage change
These statistics help you identify how your changes have affected the coverage of the following files. If a file is not in this list, then its coverage was not affected by your changes. To get some help interpreting these metrics, please refer to Jacoco's documentation.
org.pentaho.di.trans.steps.metainject.OpenMappingExtension

Branch Change:  + 50.00%
Complexity Change:  + 60.00%
Instruction Change:  + 100.00%
Line Change:  + 100.00%
Method Change:  + 100.00%

@mattcasters @duarteteixeira could you please take a view and if looks ok merge?

Build Completed
:white_check_mark: This pull request has passed all validations.
Build Commands
mvn -B -fn -f plugins/meta-inject-plugin/pom.xml clean install && mvn -B -f plugins/meta-inject-plugin/pom.xml site

Cleanup Commands
mvn -B -f plugins/meta-inject-plugin/pom.xml build-helper:remove-project-artifact

Changed files
  plugins/meta-inject-plugin/src/main/java/org/pentaho/di/trans/steps/metainject/OpenMappingExtension.java
  plugins/meta-inject-plugin/src/test/java/org/pentaho/di/trans/steps/metainject/OpenMappingExtensionTest.java

Unit test coverage change
These statistics help you identify how your changes have affected the coverage of the following files. If a file is not in this list, then its coverage was not affected by your changes. To get some help interpreting these metrics, please refer to Jacoco's documentation.
org.pentaho.di.trans.steps.metainject.OpenMappingExtension

Branch Change:  + 50.00%
Complexity Change:  + 60.00%
Instruction Change:  + 100.00%
Line Change:  + 100.00%
Method Change:  + 100.00%

@mbatchelor updated, could you merge?


feat: Enable Net Performance Analyzer

What does the pull request do?

Added Net Performance Analyzer for whole solution, but disabled for default.In the next PRs he will address each rule his own PR.
What is the current behavior?
none
What is the updated/expected behavior with this PR?

How was the solution implemented (if it's not obvious)?

Checklist

[ ] Added unit tests (if possible)?
[ ] Added XML documentation to any related classes?
[ ] Consider submitting a PR to https://github.com/AvaloniaUI/Documentation with user documentation

Breaking changes

Obsoletions / Deprecations

Fixed issues

Part of #8944

You can test this PR using the following package version. 11.0.999-cibuild0024571-beta. (feed url: https://nuget.avaloniaui.net/repository/avalonia-all/index.json) [PRBUILDID]


Add Elastic search E2E tests

Summary

Add E2E cypress tests for Elastic search
Added ids and test ids to various elements
Added additional API commands
On some preexisting API commands, added assertion post-request.
Removed logging in and out when doing some API commands, updated test where they relied on that logic.

Misc:

Renamed some files that did not follow the *_spec.js pattern

Ticket Link
JIRA

@saturninoabril  does not run on cypress.test.mattermost.com at the moment, we'll need to start running elasticsearch there for it to be viable. Will chat with Carlos about it.

Master merged, should be all up to date.
@sudheerDev Can you take a peek at this or want me to assign to someone else?

@thekiiingbob Oh sorry it got lost. I will do this today


Organise and chat with One Login developers (week 1)

What
Have conversation, document actions and next steps for integrating a One Login component into the work on Navigation.
Why
To allow both sets of users to have a consistent experience across services.
Done whens

[x] Meeting created in Establish relationship with One Login coded examples (week 1) #3846
[x] Notes taken in call
[ ] Direction decided
[ ] Guidance decisions passed onto the squad
[ ] Development work to achieve decision completed

We've had a meeting, but One Login have asked for more time to mull over their options before deciding on a direction.

Notes from the call on 21 June 2024
The way forward we got to is that we show a visual example of One Login in the black header to the Help users to navigate a service pattern. We provide a contact to their team (similarly to the Step-by-step navigation pattern) for implementation.
Meanwhile we share with them our npm preview branch and pull request code so they can work out how to use our slots with their custom css, js and form work that underpins the component they have.
We also mentioned we'd like them to update their guidance to include that the Service name should be a link.
From the technical perspective, this sounds like they're happy to use the slot and we won't need to include any code specific to One Login in govuk-frontend.
(This is good because they're going to be changing the HTML structure for the One Login elements soon, which otherwise would have been a blocker for us.)
Rather than having service teams use the slot directly, however, their plan is to create a package that inherits govuk-frontend and uses our Header and Service Header components; and injects their code into the slot at that point, as well as requisite CSS/JS.
Service teams would install the One Login Header as an npm package and use it in place of the DS Header.
Further conversation after the call 21 June 2024
They think they should actually think through the options in some more detail. They may need to revisit the broad decision  made on that call
(waiting to hear back on updates)

Checked in with One Login this morning, I've sent them a copy of our current guidance and how we are speaking about using their component.
Here's what the current guidance looks like, we reference the One Login header under ## Plan your GOV.UK header

The GOV.UK header component should only be used to show the GOV.UK logo and any GOV.UK-wide tools used in your service, such as GOV.UK One Login.
Also see GOV.UK One Login guidance to Let users navigate to their GOV.UK One Login and sign out easily.
Services do not need to duplicate the menu bar used in the GOV.UK homepage and mainstream guidance pages.

A meeting has been scheduled for 16/07/2024. In retro 09/07/2024 we suggested detaching One Login involvement as a dependency to our work in order to remain agile.


Dequeing from Queue error

So I'm writing a python script to dequeue some messages from the queue and then print them. It's usually working fine but it's sometimes giving me this error:
TypeError: can only concatenate str (not "NoneType") to str

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 59cdec64-6b86-6e2b-db74-5952d825bea4
Version Independent ID: e41416cc-85b4-743b-7bcf-ac2a97297a9d
Content: How to use Queue storage from Python
Content Source: articles/storage/queues/storage-python-how-to-use-queue-storage.md
Service: storage
Sub-service: queues
GitHub Login: @tamram
Microsoft Alias: tamram

The error message means that one of your functions/variables is returning a None value while concatenation (somewhere where you are combining two values (+)) , and it could be that you are casting a  None value to a str value which causes that message. I'd suggest to debug your code by printing your values before doing the concatenation to find out which variable is giving the None value.

We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.

Hello Sir, I am printing the values before concatenating and I am getting None sometimes although I am adding elements to the queue. What I'm doing is performing a while(True) loop that dequeues elements and prints them every few seconds if there are elements in the queue, if there aren't any elements in the queue it will print for instance "Empty Queue" and wait for a few seconds and go back again to the start of the while loop.

I fixed the problem. I was checking if the queue is empty by using if(queue_service.get_messages('taskqueue') == []) which was causing it to change the visibility of the message. Instead I just wrote the Dequeue function and in the main code wrote:
x = Dequeue()
if x is not None:
#do stuff
Thank You for the help


Finished remaining compositional charts

Genericized how I'm extracting the data from the server, with minimal duplication of code. Charts now visible are:

Number of Posts by Subreddit
Score by Subreddit
Number of Times Gilded by Subreddit
Number of Posts by Domain
Score by Domain
Number of Times Gilded by Domain
Number of Posts by User
Score by User
Number of Times Gilded by User

       __  _
       .-.'  `; `-._  __  _
      (_,         .-:'  `; `-._
    ,'o"(        (_,           )
   (__,-'      ,'o"(            )>
      (       (__,-'            )
       `-'._.--._(             )
          |||  |||`-'._.--._.-'
                     |||  |||

Sheep it


Bump identity.organization.login version

$version

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/6521364137

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/6521364137
Status: success


[SPARK-4902][CORE] gap-sampling performance optimization

jira: SPARK-4902
cc @mengxr

@witgo is this still live and have you followed up on Xiangrui's comment?

Two months ago. I talked to @mengxr in an email, I will post the performance test results

test("bernoulli sampling benchmark") {
    class BernoulliSamplerBenchmark(val fraction: Double, items: () => Iterator[Int]) extends scala.testing.Benchmark {
      override def run(): Unit = {
        val sampler = new BernoulliSampler[Int](fraction)
        val count = sampler.sample(items()).size
      }
    }

    val context = new org.apache.spark.TaskContextImpl(0, 0, 0, 0)
    var fraction = 0.2
    var len = 1e6.toInt
    var noTimes = 1000
    var array = (1 to len).toArray

    var iter: () => Iterator[Int] = () => {
      new Iterator[Int] {
        var i = 0

        override def hasNext = {
          i < len
        }

        override def next = {
          i += 1
          i
        }
      }
    }
    var sampler = new BernoulliSamplerBenchmark(fraction, iter)
    var time = sampler.runBenchmark(noTimes).sum.toDouble / noTimes
    println(s"general sampling fraction=$fraction len=$len use time: $time Ms")

    iter = () => {
      new org.apache.spark.InterruptibleIterator(context, array.iterator)
    }
    sampler = new BernoulliSamplerBenchmark(fraction, iter)
    time = sampler.runBenchmark(noTimes).sum.toDouble / noTimes
    println(s"gap sampling fraction=$fraction len=$len use time: $time Ms")

    fraction = 0.05
    iter = () => {
      new Iterator[Int] {
        var i = 0

        override def hasNext = {
          i < len
        }

        override def next = {
          i += 1
          i
        }
      }
    }
    sampler = new BernoulliSamplerBenchmark(fraction, iter)
    time = sampler.runBenchmark(noTimes).sum.toDouble / noTimes
    println(s"general sampling fraction=$fraction len=$len use time: $time Ms")

    array = (1 to len).toArray
    iter = () => {
      new org.apache.spark.InterruptibleIterator(context, array.iterator)
    }
    sampler = new BernoulliSamplerBenchmark(fraction, iter)
    time = sampler.runBenchmark(noTimes).sum.toDouble / noTimes
    println(s"gap sampling fraction=$fraction len=$len use time: $time Ms")

    fraction = 0.01
    iter = () => {
      new Iterator[Int] {
        var i = 0

        override def hasNext = {
          i < len
        }

        override def next = {
          i += 1
          i
        }
      }
    }
    sampler = new BernoulliSamplerBenchmark(fraction, iter)
    time = sampler.runBenchmark(noTimes).sum.toDouble / noTimes
    println(s"general sampling fraction=$fraction len=$len use time: $time Ms")

    array = (1 to len).toArray
    iter = () => {
      new org.apache.spark.InterruptibleIterator(context, array.iterator)
    }
    sampler = new BernoulliSamplerBenchmark(fraction, iter)
    time = sampler.runBenchmark(noTimes).sum.toDouble / noTimes
    println(s"gap sampling fraction=$fraction len=$len use time: $time Ms")
  }

=>
general sampling fraction=0.2 len=1000000 use time: 14.562 Ms
gap sampling fraction=0.2 len=1000000 use time: 16.352 Ms
general sampling fraction=0.05 len=1000000 use time: 5.408 Ms
gap sampling fraction=0.05 len=1000000 use time: 4.251 Ms
general sampling fraction=0.01 len=1000000 use time: 7.528 Ms
gap sampling fraction=0.01 len=1000000 use time: 1.009 Ms

Test build #30736 has started for   PR 3744 at commit d18f877.

Test build #30736 has finished for   PR 3744 at commit d18f877.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30736/
Test FAILed.

Test build #30767 has started for   PR 3744 at commit c66115c.

Test build #30767 has finished for   PR 3744 at commit c66115c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30767/
Test FAILed.

@mengxr @srowen any updates on this one?


Transitioning to the Angular 6+ is not possible for Angular CLI

Hello,
I have a project that is working on Angular 5.
The table component is working like a charm.
However, after I update the project to Angular 6, by using this link
I get the following error:

Package "@angular-generic-table/core" has an incompatible peer dependency to "@angular/core" (requires "^4.0.0" (extended), would install "6.1.10").
Package "@angular-generic-table/core" has an incompatible peer dependency to "@angular/common" (requires "^4.0.0" (extended), would install "6.1.10").

Could you please help?

Hi @demid5111 , in angular cli documentation you can find parameters when running ng update for this kind of errors,
please try
ng update --all --force
It basically avoids/ignores all the dependency-checks and updates all the packages wherever applicable.
I hope it helps.
Thanks

Does that mean that tables will be still working with Angular 6? пятница, 02 ноября 2018г., 23:30 +03:00 от Dennis Gonzalez<EMAIL_ADDRESS>:

Hi  @demid5111 , in angular cli documentation you can find parameters when running ng update for this kind of errors,
please try
ng update --all --force
It basically avoids/ignores all the dependency-checks and updates all the packages wherever applicable.
I hope it helps.
Thanks
—
You are receiving this because you were mentioned.
Reply to this email directly,  view it on GitHub , or  mute the thread .

however, I get the error as updating to the 6th version also requires to use rxjs v6

@demid5111 Hi, this is not my library, Im just trying to help,
Did you try to run in your angular folder?
 npm install rxjs-compat tether --save
That library is for compatibility to previous version of rxjs lib( i read somewhere they plan to removed in next versions but no sure is they did it on version 7)
i have it running on version 6 with no problem and i use very complex concepts like lazy loading in some cases, customs header components and custom rows.
Also if you can't make it work check other alternatives like material https://material.angular.io/components/table/examples
Thanks

Sorry for being a little late to the party and thanks @crash1988 for helping out:) It should work just fine with angular 6 following the steps crash explained above. However I haven’t tried using it with angular 7 just yet but it should work too as there are no major breaking changes between angular 6 and 7 that should affect angular-generic-table.
That said I hope I’ll get some time soon to release a new version using the latest version of angular including version 6 of rxjs.
There are a lot of things on the list of things that could be improved since both angular and angular cli has come a long way since this library was written. Not to mention some interesting things to come with the new ivy compiler that hopefully will be release soon:)
Anyways, let me know if the above solotion doesn’t work for you @demid5111 until a new version is released:)

@hjalmers thanks for the response, please create a new branch like you said https://github.com/hjalmers/angular-generic-table/issues/253#issuecomment-393270462  and i will try to help, i know you have to change a lot since now we can move everything to a library but once that step is done the rest should be easier, and also update the rxjs

Yeah first step would just be to use latest angular version and update rxjs (hopefully I'll find some time to do it this week). I just tried using the library with latest version of angular (7.0.3) and I can confirm that it works when doing:
npm install rxjs-compat tether --save

#253


added a warning for cc-faq to not change cluster settings for enterpr…

references https://github.com/cockroachdb/docs/issues/6136

This change is

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/31df01868c9e53b55d7a8480857075b187295856/
Edited pages:

v19.2/cockroachcloud-frequently-asked-questions.md


tls connection client options and gossiped servers

[x] Feature Request or Change Proposal

Defects
Currently (some?) clients require a tls option to be set in a connection order to enforce whether a server connection should be requiring TLS or not. This however doesn't scale, because regardless of cluster correctness, it is possible to have a cluster where some members require a TLS connection while others don't.
The main issue is that gossiped hostports for other cluster members don't provide a protocol, so some of the clients have to sanitize the received hostports into urls, and add a nats:// or tls://.
The requirement for a client to only connect to a tls or bare server should be implicit by the protocol in the url. This means that any tls connection option should only be to provide additional options such as CA certificates, or additional tls options required by the connection.
Proposed Solution

The protocol in the nats server url, should dictate the requires tls or not.
The gossiped hostports should include the protocol exposed by the server
Any client TLS connection option becomes additional TLS configuration options and only used if the server is sporting a secured connection.
Since gossiped servers are notified to the client, the client could remove implicit servers that don't comply with their required connection policy from the submitted list.

While at it, it would be great if the protocol for tls connections was nats-tls

I don't think that any change is required and if client libraries follow the Go library implementation, it works seamlessly. It does not matter what the protocol in front of the URL is, actually most languages would fail if trying to dial to anything but an host[<ip>]:port. The INFO protocol sent from the server to the client after accepting the TCP connection does contain the information regarding the need or not for the client to switch to TLS.
The client library logic should be as follow:

If the user configures TLS for the connect, he/she wants to create a secure connection, so if the server is not configured for TLS, the connection to this server should fail.
If the server is configured to use TLS but the client has not specifically specified TLS, then the library should try to switch automatically to TLS. In some languages, it is possible to do a TLS handshake without the user specifying any certificates (especially if the server is not configured to verify client certificate). So here the user does not require TLS, but there is nothing wrong making the connection secured. If the user has not provided any TLS configuration and the automatic switch to TLS is not possible, then the connection to this server will fail.

If there are servers in the cluster with different client TLS settings (some require, some not - although I doubt that this is common), a client may be able to connect to a server while failing to (re)connect to others with the above logic. Again, I don't think the heterogenous setup will be common, but if it happens, the library will still do the right thing with the current protocol.


Fix timing facility for reference benchmarks

A recent move to C++-driven time measurement changed the timing facility
in a way that is not suitable for reference NumPy and PyTorch
benchmarks: they are not returning a list of times like the generated
code. Put back the Python timing facility for these caess.

forgot to approve and just merged, sorry 😛


keepalive 2.X: new package request

Issue description
keepalive 1.X is already packaged, but we should have both versions available, because version 2 is rather new.

@coretemp I this still relevant?


Allow /email path in CORS

coverage: 94.284%. remained the same when pulling 5ebd226747692259dae1d1c798b5c3fe093ca128 on rahul-enable-email-cors into 87aa1318a766f52dfdd8135a28d1ed8f0375f88a on main.


Clarify interaction docs to indicate they apply to all interactions

Resolves #6200

Maby also change this line "When configuring interaction with the graph via hover or tooltips, a number of different modes are available." in interactions/modes.md to "When configuring the interaction with the graph via interaction, hover or tooltips, a number of different modes are available."


feat: New yaml output type: Folder_per_chart_file_per_resource

New feature:
Allows users to elect to export their yaml resources as follows:

Each chart creates a folder
Each resource is created as a file and placed in the correct chart folder
This provides better organization and flexibility as the number of resources created increases.

There are two files modified:

app.ts: Created the new yaml output type and implemented the logic to create the yaml files
app.test.ts: Added additional test cases to verify new functionality

All tests passed locally before submitting PR

Hi @camba1, thanks for submitting this pull request. :)
Before I review the code I'm wondering if you could briefly describe what your use case for this feature is? The reason I ask is just to see if there are other more general ways we might want to address this.
I wonder if it would be helpful if there was a way to customize the output paths of resources directly based on a user-provided template pattern. For example, a user could specify they want resources output as "{chart}/{name}.{hash}.yaml" or "{name}.{chart}.yml" - and then files would be automatically output in this format.

Hi @Chriscbr,
Thanks for taking a look at the PR.
This PR was created to help manage the K8s manifests that cdk8s generates for an application. Reality is that once an application starts to grow, the number of K8s resources that need to be generated increases quickly. This can easily become a chore to maintain unless the K8s manifests are organized properly.
Currently cdk8s offers 3 ways to generate the manifests:

File per resource: (the one I currently use) creates one file per K8s resource but puts them all in a single folder. This is good for small projects, but even for a the sample application that I built (node+mariadb+redis+prometheus), finding anything within that folder is less than optimal.
File per chart: This would create only 4 files in my case, but I do not use it because it means that I have to scroll through what could be a huge file to find anything (not to mention I could type something by mistake and break something completely unrelated to what I am doing)
File per App: This is similar to the discussion above but with an even bigger file

The new export mode 'Folder per chart. file per resource' allows the files to be setup in a way that is easy to maintain. Each chart folder contains only the relevant resource files, making it easy to find what I am looking for (i.e. the service for redis would be found in the redis chart folder). Furthermore, having a file per resource means that the files remain small and make it easy to find anything within the file
In terms of making it configurable as you mention, I think it would be a nice addition but I think the four export options (3 + new one) would catch the bulk of the uses cases. Maybe it could be added as another option for the super users that really want o configure the output? I suspect most people though would just pick one of the four options though.
What do you think ?


Can't edit collection documents not created via CMS admin console

I have converted my Meteor collections to Orion for use with this CMS, but I have found that I can only update documents that were created via the admin interface, not ones that I have inserted through other means.
Here are two examples of entries:
Non working; entered via application
{
	"_id" : ObjectId("5585630935f8834cf6939a59"),
	"fieldOfStudy" : "Theology",
	"fieldOfStudyId" : "Theology",
	"mappedKeywords" : [ "theology", "ministry" ]
}

Working entered via admin console
{
	"_id" : "PuS9pkZ82K5BLcYJu",
	"fieldOfStudy" : "Awesomeness",
	"fieldOfStudyId" : "Awesomeness",
	"mappedKeywords" : [ "Awesome" ],
}

When I click the first entry via the admin console I get a blank screen as follows:

I have no idea why one works, but the other doesn't. The only difference I can see is the value of the '_id", but I am not setting either of these manually.
For reference, here is my database schema for this collection:
// create Orion collection to keep qualification field of study mapping
QualificationFieldOfStudyMap = new orion.collection(
  // collection name
  'qualification_fos_map',

  // collection options
  {
    singularName: 'Field of Study Entry',
    pluralName: 'Field of Study Entries',
    link: {title: 'Fields of Study'},
    tabular: {
      columns: [
        {data: 'fieldOfStudy', title: 'Field of Study'},
        {data: 'mappedKeywords', title: 'Mapped Keywords'}
      ]
    }
  }
);

// schema for the qualification field of study map
QualificationFieldOfStudySchema = new SimpleSchema({
  fieldOfStudy: {
    type: String,
    label: 'Field of Study'
  },
  fieldOfStudyId: {
    type: String,
    label: 'ID (don\'t set or update)',
    autoValue: function () {
      var fos = this.field('fieldOfStudy');
      if (fos.isSet) {
        return fos.value.replace(/\s/g, ''); // remove spaces
      } else {
        this.unset(); // prevent user from setting their own value
      }
    }
  },
  mappedKeywords: {
    type: [String],
    label: 'Mapped Keywords',
    minCount: 1
  }
})

// attach schema to collection
QualificationFieldOfStudyMap.attachSchema(QualificationFieldOfStudySchema);

// create unique index on the server collection based on the field of study id
if (Meteor.isServer) {
  QualificationFieldOfStudyMap._ensureIndex({
      fieldOfStudyId: 1
    },
    {
      unique: true
    });
}

Please let me know if you need any additional information.

Maybe there is a SimpleSchema error. Try to run in debug mode
https://github.com/aldeed/meteor-simple-schema#debug-mode

I've made some progress by running all the objects in the collection through validation using:

check(document, schema);

The error related to the '_id' field not being set. After adding this to the schema, I didn't get any validation errors, but the symptoms explained above still persist.
I attempted to enable debug mode on simple schema, but didn't get any output.
So far the only difference I can see is the '_id' values generated by SimpleSchema versus those generated by MongoDB.

I have instead logged an issue against simple-schema:
Match error reported on '_id' field #387

I've narrowed this down to AutoForm expecting the '_id' of documents to be of type String, not Object (ObjectId).
Any idea how I can convert the '_id' values between what AutoForm requires and what MongoDB stores in the database?

I thought they were all ObjectId in the DB, and in meteor they are treated like strings, so you don't need to use ObjectId? I'm not 100% clear though, because this is something that I have wondered about as well.

Hmm. Intersting. That would make sense, but Orion (via Simple-Schema and AutoForm) can only display documents that were created vaie the interface it seems.
Docs created using Orion use a String '_id', whereas the other documents in the collection (added with mongoimport) use ObjectId's. The latter of which can't ve displayed by Orion's individual Doc view. Both Documents (Orion and mongoimport) show up in the list view though.
I must be missing something, but can't find anything in the documentation or via Google searches yet.

I found the issue.
I am using mongoimport to import some collection documents. Mongoimport defaults to using ObjectID's instead of String values. This is incompatible with Meteor and Orion CMS it seems.
To fix this I manually generated String _id values in my Nodejs script that builds the JSON that I pass to mongoimport as follows:
var ObjectID = require('mongodb').ObjectID;

// _id
var _id = (new ObjectID()).toString();
output += '"_id": "' + _id + '",';

I created a blog post that adds a little more detail on this topic: https://devillex.com/using-mongoimport-with-meteor-and-orion-cms/

I am stuck with this issue, but in my case, I have to be able to use ObjectID in the admin (Orion) because most data will be inserted via Robomongo. @devillex's solution only solves the issue partially, where Orion won't be using ObjectID to refer back to individual data, but falters when Admin users only need to view the data, without creating it in Orion in the first place.


TwinModel instance keep allocating memory when used in a loop

🔍 Before submitting the issue

[X] I have searched among the existing issues
[X] I am using a Python virtual environment

🐞 Description of the bug
When instantiating a TwinModel in a loop, the allocated memory keep increasing.

📝 Steps to reproduce
import os
import tracemalloc
from pytwin import TwinModel
tracemalloc.start()
snapshot = tracemalloc.take_snapshot()
twin_file = os.path.join("path_to_a_twin_model_with_tbrom.twin")
for i in range(1, 100):
twin_model = TwinModel(model_filepath=twin_file)
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot, 'lineno')
stat_tbrom_read_basis = top_stats[0]
print(stat_tbrom_read_basis)
💻 Which operating system are you using?
Windows
📀 Which ANSYS version are you using?
NA
🐍 Which Python version are you using?
3.9
📦 Installed packages
certifi==2023.7.22
charset-normalizer==3.3.2
colorama==0.4.6
coverage==7.3.2
cycler==0.11.0
exceptiongroup==1.1.3
fonttools==4.38.0
idna==3.4
imageio==2.31.2
iniconfig==2.0.0
kiwisolver==1.4.5
matplotlib==3.5.3
numpy==1.21.1
packaging==23.2
pandas==1.3.5
Pillow==9.5.0
platformdirs==4.0.0
pluggy==1.3.0
pooch==1.8.0
pyparsing==3.1.1
pytest==7.4.3
pytest-cov==4.1.0
python-dateutil==2.8.2
-e git+https://github.com/ansys/pytwin@16cf78a84ed27a2881d0c7781a8afe66750a76e9#egg=pytwin
pytz==2023.3.post1
pyvista==0.38.6
pywin32==306
requests==2.31.0
scooby==0.7.3
six==1.16.0
tomli==2.0.1
tqdm==4.66.1
urllib3==2.0.7
vtk==9.3.0

Adding the following statement twin_model._tbroms = None at the end of the loop solve the issue, see below:

Investigations: We are registering the del method of the TwinModel with the atexit module in order to avoid the settings module tries to delete pytwin temp folder of the process before the log file of the TwinRuntime object owned by the TwinModel has been closed. This has a side effect which is that a TwinModel instance is referenced by the atexit module and thus preventing the garbage to collect the TwinModel at the end of the loop.

@chrpetre see proposed solution in PR #130


Move Add database reference validation error message

Right now, the validation error message is in the Example Usage box, which is what SSDT does

There's support for validation for input boxes, so it would be nicer for the error message to show up with the associated input box like this:

Can I take a crack at this? Is this still an open issue?

@CarlSargunar sure go for it! :)

Ok - I've got some time at the weekend, I'll have a go - thanks :-)

Hello


Feedbacks to your preprint

Hi, I really enjoyed your preprint! An homage to AIMLESS is also nice :)
It is impressive to see that a neural network and variational inference framework can "learn" a scaling model that performs almost as good as existing programs with hand-implemented physical models.
I am looking forward to seeing if it can perform even better in future.
Below are my comments.
Introduction (Section 1)

scale parameters are learned per image

This is not necessarily true. Scale factors might depend not only on time (i.e. image no = goniometer angle) but also on the position on the detector, for example.
AIMLESS uses spherical harmonics and smoothly time varying functions.
XSCALE decomposes the scaling function into a product of three functions defined on 2D grids.
http://xds.mpimf-heidelberg.mpg.de/html_doc/xscale_parameters.html#CORRECTIONS=

DECAY: image number vs resolution
MODULATION: spot position on the detector (X vs Y)
ABSORPTION: image number vs 13 detector positions

Model complexity
How did you decide the structure of the neural network, such as the number of layers? Is it by cross validation? Should it be adjusted depending on the size and complexity of the dataset (e.g. single sweep or a few crystals or tens of thousands of crystals)?
Choice of metrics
CC1/2
Was the network trained independently for each half set or not? In cryo-EM, it is common to split half sets at the beginning and optimize parameters independently, while in crystallography this is not necessarily done. It is worth mentioning your choice.
Anomalous half dataset correlation
This metric is very sensitive to systematic bias. Since you have an atomic model, anomalous correlation between scaled intensities and intensities calculated  from the model might be better.
Comparison with existing approaches
Can you compare your results with those from existing programs? You might not have Precognition at hand but running cxi.merge with and without post-refinement and partiality correction on the XFEL dataset should be trivial.
Model interpretation (Section 4.1)
Can you interrogate the neural network by for example plotting its output when position on the detector is changed while other metadata are kept fixed?

Welcome @biochem-fan, and thanks for some seriously helpful feedback! We're looking to do a revised version of the manuscript shortly, so your comments are timely and useful. It's also need to be getting feedback through GitHub which is definitely a first for me.
Introduction (Section 1)
I will try to beef up this section by including more details such as the spherical harmonic absorption corrections. This is decidely a weak point. Thanks!
Model complexity
This part has been rather ad hoc. The neural net that careless uses is a very simple multi-layer perceptron. I haven't tried to do anything fancy with the architecture yet. Although that is something I am actively working on as I attempt to extend the model to address larger datasets. Regardless, I determined early on that 20 layers was sufficient to scale all of my test data sets which vary dramatically in terms of size and experiment type. I have tried larger nets, and anecdotally they do not seem prone to overfitting. Although overfitting is a legitmate concern, I haven't encountered it in practice yet. I will think about adding a section which explores number of layers against vs crossvalidation performance. I don't think that should be too difficult to include in a future draft.
CC1/2
Our CC1/2 values are computed by first dividing the data in two and training the model on each half independently. In that sense they are likely to be more pessimistic than the classic CC1/2 values you get from conventional scaling and merging programs.
Anomalous half dataset correlation
This is simple enough to add. I'll have a go thanks!
Comparison with existing approaches
I am planning to run cxi.merge and/or prime before the next draft. I wouldn't be shocked if these sort of domain expert programs outperform careless. It will be nice to understand that though.
Regarding Precognition, we do currently have a license and might explore running it. Because it is basically unheard of to run Precognition without an I/Sigma cutoff, I'd worry about how to make a fair comparison with careless. I'd be happy to entertain any thoughts on the matter.
Model interpretation (Section 4.1)
I could imaging playing some games like that. It'd be a decent amount of work to implement right now, so it's not super high priority for me. In the long term, I am less interested in trying to "open the black box" and understand what the multi-layer perceptron is doing and more interested in replacing it with more transparent physical models.
Thanks again! Have a nice weekend.

Welcome @biochem-fan, and thanks for some seriously helpful feedback! We're looking to do a revised version of the manuscript shortly, so your comments are timely and useful. It's also need to be getting feedback through GitHub which is definitely a first for me.
Introduction (Section 1)
I will try to beef up this section by including more details such as the spherical harmonic absorption corrections. This is decidely a weak point. Thanks!
Model complexity
This part has been rather ad hoc. The neural net that careless uses is a very simple multi-layer perceptron. I haven't tried to do anything fancy with the architecture yet. Although that is something I am actively working on as I attempt to extend the model to address larger datasets. Regardless, I determined early on that 20 layers was sufficient to scale all of my test data sets which vary dramatically in terms of size and experiment type. I have tried larger nets, and anecdotally they do not seem prone to overfitting. Although overfitting is a legitmate concern, I haven't encountered it in practice yet. I will think about adding a section which explores number of layers against vs crossvalidation performance. I don't think that should be too difficult to include in a future draft.
CC1/2
Our CC1/2 values are computed by first dividing the data in two and training the model on each half independently. In that sense they are likely to be more pessimistic than the classic CC1/2 values you get from conventional scaling and merging programs.
Anomalous half dataset correlation
This is simple enough to add. I'll have a go thanks!
Comparison with existing approaches
I am planning to run cxi.merge and/or prime before the next draft. I wouldn't be shocked if these sort of domain expert programs outperform careless. It will be nice to understand that though.
Regarding Precognition, we do currently have a license and might explore running it. Because it is basically unheard of to run Precognition without an I/Sigma cutoff, I'd worry about how to make a fair comparison with careless. I'd be happy to entertain any thoughts on the matter.
Model interpretation (Section 4.1)
I could imaging playing some games like that. It'd be a decent amount of work to implement right now, so it's not super high priority for me. In the long term, I am less interested in trying to "open the black box" and understand what the multi-layer perceptron is doing and more interested in replacing it with more transparent physical models.
Thanks again! Have a nice weekend.

I have tried larger nets, and anecdotally they do not seem prone to overfitting. Although overfitting is a legitimate concern, I haven't encountered it in practice yet.

This is an interesting observation.
A related note, not about overfitting, but on the uniqueness of the solution:
Scaling has many equivalent (degenerate) solutions. For example, you can multiply all scaling factors with a constant, or apply a B factor. This only changes the scale of the merged intensities and all solutions are equally valid. To make the solution unique, traditional scaling programs arbitrarily fix the scale and B factor of one input (e.g. the first frame or the first crystal). Does CARELESS have this degeneracy?
In other applications of neural networks, it is often said/observed (but not proven?) that SGD is only a local optimizer but finds one of many similarly good solutions. So probably this does not matter in practice, but worth keeping in mind when comparing maps or anomalous peak heights. Maps look very different at different B factors (consider blurring a 1.0 A map with B factor of 100; it will look like a 3 A map). Anomalous peak height (sigma level) is a real space metric, so it also depends on the B factor. High B factors usually lead to lower sigma levels (because of peak blurring) but sometimes increase it if they suppress noisy high resolution components.

In the long term, I am less interested in trying to "open the black box" and understand what the multi-layer perceptron is doing and more interested in replacing it with more transparent physical models.

Yes, your variational inference framework naturally allows combination of physical models and black-box (neural network) models. Considering what is better implemented as physical models and what should be left to be learned from data is a very interesting question for future.

I have tried larger nets, and anecdotally they do not seem prone to overfitting. Although overfitting is a legitimate concern, I haven't encountered it in practice yet.

This is an interesting observation.
A related note, not about overfitting, but on the uniqueness of the solution:
Scaling has many equivalent (degenerate) solutions. For example, you can multiply all scaling factors with a constant, or apply a B factor. This only changes the scale of the merged intensities and all solutions are equally valid. To make the solution unique, traditional scaling programs arbitrarily fix the scale and B factor of one input (e.g. the first frame or the first crystal). Does CARELESS have this degeneracy?
In other applications of neural networks, it is often said/observed (but not proven?) that SGD is only a local optimizer but finds one of many similarly good solutions. So probably this does not matter in practice, but worth keeping in mind when comparing maps or anomalous peak heights. Maps look very different at different B factors (consider blurring a 1.0 A map with B factor of 100; it will look like a 3 A map). Anomalous peak height (sigma level) is a real space metric, so it also depends on the B factor. High B factors usually lead to lower sigma levels (because of peak blurring) but sometimes increase it if they suppress noisy high resolution components.

In the long term, I am less interested in trying to "open the black box" and understand what the multi-layer perceptron is doing and more interested in replacing it with more transparent physical models.

Yes, your variational inference framework naturally allows combination of physical models and black-box (neural network) models. Considering what is better implemented as physical models and what should be left to be learned from data is a very interesting question for future.

Because we impose Wilson's prior on the structure factors, we avoid the degeneracy inherent in simultaneously optimizing the merged structure factors along with the scale function.  Therefore, careless only has degeneracy to the extent that neural networks have degenerate local minima. Global optimization of non-convex objectives is a hard problem, and in the case of neural nets it is an active area of research. I'm unsure what the current "state of the art" perspective is on the optimization landscape of feed-forward neural nets. Irrespective, it is not de rigueur to fix any parameters during optimization of MLPs. My preference is to wait for the theory to settle down before making any strong claims about optimality.
That being said, I don't believe that the scaling loss function, even in the least squares case, has ever been shown to be convex. So, there have never been any guarantees of global optimality even with the parameters carefully fixed to avoid degeneracy.
When processing serial data for the preprint, I did find it necessary to add per-image scale parameters. These are implemented in the classical way with one of the scales arbitrarily fixed to one. However, it is also possible to break the degeneracy by imposing a prior distribution over scale factors. This is an experimental feature which is currently under development. In the future, I hope to provide a flexible set of priors on image scale distributions which beamline scientists can tailor for each end station by cross-validation.

Because we impose Wilson's prior on the structure factors, we avoid the degeneracy inherent in simultaneously optimizing the merged structure factors along with the scale function.  Therefore, careless only has degeneracy to the extent that neural networks have degenerate local minima. Global optimization of non-convex objectives is a hard problem, and in the case of neural nets it is an active area of research. I'm unsure what the current "state of the art" perspective is on the optimization landscape of feed-forward neural nets. Irrespective, it is not de rigueur to fix any parameters during optimization of MLPs. My preference is to wait for the theory to settle down before making any strong claims about optimality.
That being said, I don't believe that the scaling loss function, even in the least squares case, has ever been shown to be convex. So, there have never been any guarantees of global optimality even with the parameters carefully fixed to avoid degeneracy.
When processing serial data for the preprint, I did find it necessary to add per-image scale parameters. These are implemented in the classical way with one of the scales arbitrarily fixed to one. However, it is also possible to break the degeneracy by imposing a prior distribution over scale factors. This is an experimental feature which is currently under development. In the future, I hope to provide a flexible set of priors on image scale distributions which beamline scientists can tailor for each end station by cross-validation.


Failed to save results.

Issue Type: Bug
Error. Failed to save to file.xlsx: The writer is closed or in error state.
Azure Data Studio version: azuredatastudio 1.24.0 (eb3c0f5d9df86b9490606aab7fffaa253b0aca4d, 2020-11-10T23:35:27.682Z)
OS version: Darwin x64 18.7.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz (12 x 2900)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onprotected_video_decode: unavailable_offrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
1, 4, 3

Memory (System)
32.00GB (17.57GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions: none

@aott88 do you see any error messages in the "Help->Developer Tools" Console?  Any idea if there is anything noteworthy about the data being save to Excel that may be triggering the issue? i.e. is it just "typical" numeric and text data, or are there BLOB, JSON, etc. columns?

Closing based on pending Needs More Info.  Please let us know if you are still impacted by this issue and have additional info that could help diagnosis the issue.  Thanks!

Seeing this also, Azure Data Studio 1.42.0, Date: 2023-03-17
notificationsAlerts.ts:42 Failed to save results. Error: Failed to save Yuriy-Results.xlsx: The Writer is closed or in error state.
onDidChangeNotification @ notificationsAlerts.ts:42
(anonymous) @ notificationsAlerts.ts:28
invoke @ event.ts:575
deliver @ event.ts:779
fire @ event.ts:740
addNotification @ notifications.ts:206
notify @ notificationService.ts:137
doSave @ resultSerializer.ts:343
processTicksAndRejections @ node:internal/process/task_queues:96
log.ts:313   ERR Error loading database names for query editor  TypeError: Cannot read properties of undefined (reading 'databaseNames')
at U.getDatabaseNames (queryActions.ts:859:18)
at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
notificationsAlerts.ts:42 Failed to save results. Error: Failed to save Yuriy-Results.xlsx: The Writer is closed or in error state.
Tried saving .csv or Excel, same results.


Release 1.0.0.RC2

When the following fixes have been approved and merged:
https://github.com/reactive-streams/reactive-streams/pull/212
https://github.com/reactive-streams/reactive-streams/pull/213
https://github.com/reactive-streams/reactive-streams/pull/214
@reactive-streams/contributors 1.0.0.final after this? :-)

:+1:
I'm sure there is something we've missed, but 1.0.0 has to happen at some point.

@benjchristensen Haha, I was thinking the exact same thing :)

@reactive-streams/contributors All PRs merged!

@smaldini Would you perchance do us the honor of releasing 1.0.0.RC2 from master? :)

Yay


Mouse position is incorrect unless at 90 degree angles

Basically the code that converts the mouse position to screen position in 3D space only seems to work at 90 degree angles, or else it starts to slip and go wrong:
http://i.imgur.com/n4JQRlK.gifv
I also edited the code a bit to give a helper + symbol where the mouse thinks it is

Issue appears to be somewhere in here:
https://github.com/HandsomeMatt/3d2d-vgui/blob/master/3d2dvgui.lua#L41-L46
What's strange is that the height is fine, but the width isn't. Also seems independent of aspect ratio, even 1:1 (square).
I made another gif to show the issue more clearly, the number displayed is the yaw angle, the (non moving) crosshair is the mouse's actual position, and the two intersecting lines show where the screen thinks the mouse is.
Note how it approaches the correct position at 0 and -90, right angles. I'm really shit at maths so I don't really know what's causing it - and can't think of a way to fix it, @HandsomeMatt pls

think I got it, will commit asap https://dl.dropboxusercontent.com/u/3659637/2015-05-16_01-13-30.webm

thanks babe


FSB Quilt Support

Hello
I Use Quilt And Can Be Incompatible
The Quilted Fabric API Can Be Too Incompatible
Pls @AMerebagatelle Make A Quilt Version
This Is A Request (Add The Enhancement Label)

I don't see what's the issue you are having, as you can see on the screenshot above it's compatible with Quilt.

But Is the QSL


Python f-strings: highlight expressions in curly braces

Fixes #1228

I'll rebase a send another request


UI - username sometimes not set properly

no username displayed in horizontal main nav bar
might be connected to settings - users - sync to ldap: after executing this, the issue goes away
JWT refresh leads to "Username must not be empty" error

When logging out as some user and logging in as admin next, the before logged in username will be displayed in the nav bar.

duplicate to #1853


Compatibility with<EMAIL_ADDRESS>

{ Cannot find module 'react/lib/CSSPropertyOperations'
    at $root/node_modules/jsxstyle/lib/createCSS.js:3:29 code: 'MODULE_NOT_FOUND' }

@appsforartists this change will break React 0.14 compatibility. Will chat with @petehunt IRL about what we should do.

@appsforartists Superb, thanks for PR. Hope it gets merged soon!

Yes please. Just updated and can't build.

@appsforartists thanks for the PR! We decided to extract CSSPropertyOperations to its own module.
@Andreyco @tehfailsafe this should be fixed with the next release.

@meyer Thank you!


Running Example on Free T4 GPU through Google Colab

Please check that this issue hasn't been reported before.

[X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior
The setup or first 2 cells in the notebook can install the environment. This set of installation code should run all subsequent code to successfully recreate the training and inference steps of LLM's through Axolotl. https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/colab-notebooks/colab-axolotl-example.ipynb
Current behaviour
The first cell is successful,
import torch
# Check so there is a gpu available, a T4(free tier) is enough to run this notebook
assert (torch.cuda.is_available()==True)

The second cell never completes and even if we try to change some of the code, the accelerate command doesn't work. It hangs at flash attention.
!pip install -e git+https://github.com/axolotl-ai-cloud/axolotl#egg=axolotl
!pip install flash-attn=="2.5.0"
!pip install deepspeed=="0.13.1"!pip install mlflow=="2.13.0"

Here's the tail of the installation output from the cell,
:
:
  Attempting uninstall: gcsfs
    Found existing installation: gcsfs 2024.6.1
    Uninstalling gcsfs-2024.6.1:
      Successfully uninstalled gcsfs-2024.6.1
  Running setup.py develop for axolotl
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
albumentations 1.4.14 requires pydantic>=2.7.0, but you have pydantic 2.6.3 which is incompatible.
cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 17.0.0 which is incompatible.
ibis-framework 8.0.0 requires pyarrow<16,>=2, but you have pyarrow 17.0.0 which is incompatible.
Successfully installed accelerate-0.34.2 addict-2.4.0 aiobotocore-2.14.0 aiofiles-23.2.1 aioitertools-0.12.0 art-6.2 autoawq-0.2.5 autoawq-kernels-0.0.6 axolotl-0.4.1 bitsandbytes-0.43.3 botocore-1.35.7 colorama-0.4.6 coloredlogs-15.0.1 datasets-2.20.0 dill-0.3.8 docker-pycreds-0.4.0 evaluate-0.4.1 fastapi-0.114.0 ffmpy-0.4.0 fire-0.6.0 fschat-0.2.36 fsspec-2024.5.0 gcsfs-2024.5.0 gitdb-4.0.11 gitpython-3.1.43 gradio-3.50.2 gradio-client-0.6.1 h11-0.14.0 hf_transfer-0.1.8 httpcore-1.0.5 httpx-0.27.2 humanfriendly-10.0 jmespath-1.0.1 latex2mathml-3.77.0 liger-kernel-0.2.1 markdown2-2.5.0 multiprocess-0.70.16 nh3-0.2.18 optimum-1.16.2 orjson-3.10.7 packaging-23.2 peft-0.12.0 pyarrow-17.0.0 pydantic-2.6.3 pydantic-core-2.16.3 pydub-0.25.1 pynvml-11.5.3 python-dotenv-1.0.1 python-multipart-0.0.9 responses-0.18.0 s3fs-2024.5.0 scikit-learn-1.4.2 semantic-version-2.10.0 sentry-sdk-2.13.0 setproctitle-1.3.3 shortuuid-1.0.13 shtab-1.7.1 smmap-5.0.1 starlette-0.38.5 svgwrite-1.4.3 tiktoken-0.7.0 triton-3.0.0 trl-0.9.6 tyro-0.8.10 uvicorn-0.30.6 wandb-0.17.9 wavedrom-2.0.3.post3 websockets-11.0.3 xformers-0.0.27.post2 xxhash-3.5.0 zstandard-0.22.0
Collecting flash-attn==2.5.0
  Downloading flash_attn-2.5.0.tar.gz (2.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 38.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.0) (2.4.0+cu121)
Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.0) (0.8.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from flash-attn==2.5.0) (23.2)
Collecting ninja (from flash-attn==2.5.0)
  Downloading ninja-<IP_ADDRESS>-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.0) (3.15.4)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.0) (4.12.2)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.0) (1.13.2)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.0) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.0) (3.1.4)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->flash-attn==2.5.0) (2024.5.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->flash-attn==2.5.0) (2.1.5)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->flash-attn==2.5.0) (1.3.0)
Downloading ninja-<IP_ADDRESS>-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.2/307.2 kB 23.2 MB/s eta 0:00:00
Building wheels for collected packages: flash-attn

Steps to reproduce

Open https://colab.research.google.com/ and login if needed
Click open Github, type in axolotl-ai-cloud for organization and select axolotl for repository
Select the example notebook
Run the cells in the notebook for installation

Config yaml
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: mhenrichsen/alpaca_2k_test
    type: alpaca
dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/qlora-out

adapter: qlora
lora_model_dir:

sequence_len: 4096
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:

Possible solution
This might be because the T4 GPU is not supported by this library. Here's the documentation about which GPU's are supported, https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#nvidia-cuda-support
I used the instructions in the README instead to install and train TinyLLAMA using Axolotl instead. I also changed the config to set the flash attention layer to false. This still installs the flash-attn somehow but we get RuntimeError: FlashAttention only supports Ampere GPUs or newer error at runtime instead.
This cell will install it,
!git clone https://github.com/axolotl-ai-cloud/axolotl
%cd axolotl
!pip3 install packaging ninja
!pip3 install -e '.[flash-attn,deepspeed]'

Then in the config.yaml,
flash_attention: false

This does run on the free T4 GPU but still takes hours to finish and may need a different config.
Which Operating Systems are you using?

[X] Linux
[ ] macOS
[ ] Windows

Python Version
3.10
axolotl branch-commit
main
Acknowledgements

[X] My issue title is concise, descriptive, and in title casing.
[X] I have searched the existing issues to make sure this bug has not been reported yet.
[X] I am using the latest version of axolotl.
[X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

can confirm this. @winglian could you kindly take a look and update the colab? thanks

If anyone finds this thread looking to fix the error with installing Axolotl on Colab, I was getting an error where there were conflicting versions of torch required. This is because the setup for Axolotl keeps the already installed version of torch as the required version. However, xformers required a different version.
Therefore, if you first run:
pip install torch==2.3.1

before running the axolotl install commands
pip install -e git+https://github.com/axolotl-ai-cloud/axolotl#egg=axolotl

then it installs correctly on my system.

Thanks for the suggestion @Peter-Devine but if you try running this in Colab, it still doesn't work yet.

I have got it working using these packages:
! pip install torch==2.3.1
! pip install -e git+https://github.com/axolotl-ai-cloud/axolotl#egg=axolotl

! pip install -q datasets evaluate trl
! pip install torchvision==0.18.1 torchaudio==2.3.1
! pip install flash-attn

and running using this command:
! accelerate launch -m axolotl.cli.train /content/training.yaml

My environment is L4 (High memory).
Can you not reproduce my results with these installs/commands?

I was unable to reproduce on Colab with your commands, but I did manage to get it working with these:
!git clone --depth 1 https://github.com/axolotl-ai-cloud/axolotl
!cd axolotl

!pip install -Uqq pip uv

!uv pip uninstall --system torchaudio torchvision

!uv pip install --system -U --no-progress packaging ninja setuptools wheel
!uv pip install --system --no-progress --no-build-isolation torch==2.4.0
!uv pip install --system --no-progress wavedrom==2.0.3 transformers==4.45.1 peft==0.13.0
!uv pip install --system --no-progress --no-build-isolation -e 'axolotl/.[deepspeed]'

Hmmm, I guess that different usages require different installations as I have the code working, but I am not performing QLoRA so I don't require PEFT etc.
For the sake of clarity, here is a fully reproducible example Colab of training that is currently working for me:
https://colab.research.google.com/drive/1yBxLqCVZP4IMqMQmgeCL4Dqpj_fr2pCV?usp=sharing
If you could post a similar notebook, I think that would be helpful to other users of Axolotl.
Thanks!

Here's the working Colab example (as of October 10, 2024) https://colab.research.google.com/drive/1YSCRpWWqGdGXjPlsja1c3rJzePo8VNN-?usp=sharing from my end. Good idea to post the whole thing!

https://github.com/axolotl-ai-cloud/axolotl/pull/2074


Request to add: local NIC custom routing table

https://github.com/sabersalv/freedom-routes
https://www.175.es/web/windowsroute.rar
The following is the client local routing file of the changed Windows version. Since I don't know how to integrate it into the softether client, I sent it to people in need. I also hope that the developers of softether can add local to the local. The control function of the routing table can import a routing table that the user wants. Anyone can pass the pair. This is a generated file. Because ADD.txt is all Chinese IP, you can now implement 2 functions. 1: Proxy All China IP, 2: Skip all Chinese IP, I hope SOFTETHER is getting better and better, because we can access the real Internet in the West through softether

Usage: 1. Use the routing table before the computer links to the VPN site: routes-up.bat and then link to the VPN site, then skip the Chinese site. 2: After connecting the VPN, use routes-up.bat, then only all Chinese IPs are delegated, and routes-down.bat deletes all added custom routing tables.

https://www.175.es/windowsroute.rar

https://github.com/fivesheep/chnroutes


Reorganize into modules

Mainly a breakup of the ibc module into smaller parts.
Overall, now the project is composed of the following modules:

chain/cosmos for the cosmos Chain implementation.
relayer/rly for the cosmos/relayer Relayer implementation.
ibc for the core interfaces and types
ibctest for the test setup, chains, and factories
cmd/ibctest for the test CLI
relayertest for the relayer related test cases
utils for common helpers (currently file and docker helpers)
test for tests to be run by CI in PRs
trophies for archived ITF use cases

I have some changes queued up that depend on this, so I'm merging the PR now. The small naming suggestions are okay to happen separately.


Is there any plan to release the official version of sdxl?

Good job, but sdxl's beta is not perfect，Is there any plan to release the official version of sdxl?

Same here. I don't know why SDXL + AnimateDiff seems to have fallen by the wayside. Really hoping for a better motion model soon. The current model results range from "okay" to "terrible".
It's painful to see such bad quality animations considering SDXL's potential.


Add Leases, Runtime Classes, and Limit Ranges sections

We should add the Leases  + Runtime Classes+ Limit Ranges sections.
The sections will be similar to the HPA and others we have.

fixed by #1008 .


How to select the specific version arrow

Hi, Can we select the specific version for the arrow crate?
e.g.
select 46 for arrow-array = { version = ">=46" }
I try to import the icelake and it will always import the latest arrow-* 47 even there is the arrow-* 46 in project.
@Xuanwo

Can we select the specific version for the arrow crate?

How about using cargo update -p<EMAIL_ADDRESS>

How about using cargo update -p<EMAIL_ADDRESS>
Seems it doesn't work.


Vite serve hanging up.

Describe the bug
vite serve will hang up in the step of pre-bundling dependencies.
But run vite build  can build dist successfully.
vite version v2.6.5
Reproduction
none
System Info
System:
    OS: macOS 11.5.1
    CPU: (8) x64 Apple M1
    Memory: 208.60 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.17.5 - /usr/local/bin/node
    Yarn: 1.22.11 - /usr/local/bin/yarn
    npm: 6.14.14 - /usr/local/bin/npm
  Browsers:
    Chrome: 94.0.4606.71
    Safari: 14.1.2

Used Package Manager
yarn
Logs
Pre-bundling dependencies:
  vue
  vue-router
  ant-design-vue
  (...and 35 more)
(this will be run only when your dependencies or config have changed)

Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to https://github.com/vuejs/vue-next instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

@fanhaoyuan Are you using Vite's JS API? If so, they removed the terminal output when the bundling finishes in 2.6.0 so maybe the server is not hanging up, just not telling you it has finished. Try checking localhost:<port> and see if it works.
In case it helps, here is a commit that adds the lines again to a library using JS API.


Chat Langchain is not working properly if we ask something outside of context instead of saying Hmm, Im not sure, its generating data from internet

Chat Langchain is not working properly if we ask something outside of context instead of saying Hmm, Im not sure, its generating data from internet

In code, i can see we mentioned to not do any assumptions and provide data only on context. But as mentioned in above screenshot its is taking data from  internet providing result as above, I don't want that
what code change is required to fix it

Thanks for flagging! There are two straightforward things we could do to improve:

Better prompt engineering
Classifiers on inputs + outputs
Better evals

Will try to add when I have more time, but since it's not a major security issue right now, it may be a bit


Create CODE_OF_CONDUCT.md

I looked at your project and it looks really interesting but I couldn't find a code of conduct so I added a code of conduct from Contributor Covenant, I took a guess at the contact information<EMAIL_ADDRESS>from your personal profile but feel free to change it.

Thanks, so far people have been behaving quite civilized here, but extra reminders can't hurt.


lxd failed with result 'dependency

Installing lxd on 16.04 after upgrading from 15.10 (lxc1). lxd daemon does not start, syslog reports "dependency failed for lxd", and running any lxc command fails with permission denied. I checked my user is a member of lxd and followed other issues suggestion (exiting terminal, doing newgrp).

Kernel version: 4.4.0-22-generic
LXC version: stable release
LXD version: stable release
Storage backend in use: ZFS

Issue description
lxd doesn't start after installing.
Information to attach

[ ] any relevant kernel output (dmesg)
[ ] container log (lxc info NAME --show-log)
[ ] main daemon log (/var/log/lxd.log)
[ ] output of the client with --debug
[ ] output of the daemon with --debug

May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
May 25 17:40:23 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency
lxc list
error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied
May 25 17:40:14 baZinga systemd[1]: Dependency failed for LXD - main daemon.
May 25 17:40:14 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
May 25 17:40:14 baZinga systemd[1]: lxd-bridge.service: Start request repeated too quickly.
May 25 17:40:14 baZinga systemd[1]: Failed to start LXD - network bridge.
May 25 17:40:14 baZinga systemd[1]: Dependency failed for LXD - main daemon.
May 25 17:40:14 baZinga systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
May 25 17:40:14 baZinga systemd[1]: lxd-bridge.service: Start request repeated too quickly.
May 25 17:40:14 baZinga systemd[1]: Failed to start LXD - network bridge.
I cannot figure out what dependency is missing. My assumption lxd-bridge is not created, running lxd init hangs.
Thanks

systemctl status lxd-bridge

systemctl status lxd-bridge
● lxd-bridge.service
Loaded: not-found (Reason: No such file or directory)
Active: failed (Result: exit-code) since Wed 2016-05-25 17:42:03 CEST; 34min ago
Main PID: 17895 (code=exited, status=1/FAILURE)
May 25 17:42:03 baZinga lxd-bridge.start[17895]: iptables: Bad rule (does a matching rule exist in that chain?).
May 25 17:42:03 baZinga lxd-bridge.start[17895]: iptables: Bad rule (does a matching rule exist in that chain?).
May 25 17:42:03 baZinga lxd-bridge.start[17895]: iptables: Bad rule (does a matching rule exist in that chain?).
May 25 17:42:03 baZinga lxd-bridge.start[17895]: iptables: Bad rule (does a matching rule exist in that chain?).
May 25 17:42:03 baZinga lxd-bridge.start[17895]: iptables: Bad rule (does a matching rule exist in that chain?).
May 25 17:42:03 baZinga lxd-bridge.start[17895]: iptables: Bad rule (does a matching rule exist in that chain?).
May 25 17:42:03 baZinga lxd-bridge.start[17895]: iptables: No chain/target/match by that name.
May 25 17:42:03 baZinga lxd-bridge.start[17895]: iptables v1.6.0: host/network 10.30.630.1' not found May 25 17:42:03 baZinga lxd-bridge.start[17895]: Try iptables -h' or 'iptables --help' for more information.
May 25 17:42:03 baZinga lxd-bridge.start[17895]: ip6tables: No chain/target/match by that name.

Hmm, how did you configure that bridge?
10.30.630.1 sure isn't a valid IP

I ran "sudo dpkg-reconfigure -p medium lxd" hoping that would create the bridge or whatever dependency that is missing. I don't see any lxcbr0. Is it supposed to be created as part of lxd installation?

The reconfigure will create a lxdbr0 bridge when the config is valid.
The IP address of your bridge however is very much invalid which causes it to fail to come up and therefore blocks lxd too.
I'd suggest you re-run "dpkg-reconfigure -p medium lxd" and make sure the values are valid IP addresses.

Reconfigured the bridge to a valid IP and restarted lxd-bridge and lxd. It worked!
Thanks stgraber!

Good to hear!

Hi, I have the same problem. I don't understand what do you means by "valid ip". By default the "sudo dpkg-reconfigure -p medium lxd" tools proposed me to set the bridge with "<IP_ADDRESS>, with DHCP from <IP_ADDRESS> to <IP_ADDRESS>. Why these settings are wrong ? What should I do to choose a valid ip ?
Below my logs:
`➜  tmp sudo service lxd status
● lxd.service - LXD - main daemon
Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
Active: inactive (dead) since Mon 2017-03-13 10:18:14 CET; 18min ago
Docs: man:lxd(1)
Main PID: 968 (code=exited, status=0/SUCCESS)
Mar 13 10:21:36 absgZ systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Mar 13 10:21:36 absgZ systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Mar 13 10:21:36 absgZ systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Mar 13 10:21:36 absgZ systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Mar 13 10:21:36 absgZ systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Mar 13 10:21:36 absgZ systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Mar 13 10:21:36 absgZ systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Mar 13 10:22:23 absgZ systemd[1]: Dependency failed for LXD - main daemon.
Mar 13 10:22:23 absgZ systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Mar 13 10:35:59 absgZ systemd[1]: Stopped LXD - main daemon.
➜  tmp systemctl status lxd-bridge
● lxd-bridge.service - LXD - network bridge
Loaded: loaded (/lib/systemd/system/lxd-bridge.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2017-03-13 10:22:23 CET; 14min ago
Docs: man:lxd(1)
Main PID: 7715 (code=exited, status=2)
Mar 13 10:22:23 absgZ systemd[1]: Starting LXD - network bridge...
Mar 13 10:22:23 absgZ lxd-bridge.start[7715]: dnsmasq: failed to create listening socket for fe80:...:2d99%lxdbr0: Address already in use
Mar 13 10:22:23 absgZ lxd-bridge.start[7715]: Failed to setup lxd-bridge.
Mar 13 10:22:23 absgZ systemd[1]: lxd-bridge.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 13 10:22:23 absgZ systemd[1]: Failed to start LXD - network bridge.
Mar 13 10:22:23 absgZ systemd[1]: lxd-bridge.service: Unit entered failed state.
Mar 13 10:22:23 absgZ systemd[1]: lxd-bridge.service: Failed with result 'exit-code'.
Mar 13 10:36:32 absgZ systemd[1]: Stopped LXD - network bridge.`

Ok, the problem was that my server has an instance of bind9 service running and locking the port 53.
I solved my problem, by stopping bind9 service before restarting lxd. So no issue with the configuration of my bridge in fact.
Thanks for your job,
regards


OCM. user cannot delete accepted user

precondition:

run two ocis instances ocis-server and federation-ocis-server see here https://owncloud.dev/ocis/development/testing/#running-test-suite-with-federated-sharing-ocm

Steps:
And "Alice" has generated invitation
And using server "REMOTE"
And "Brian" has accepted invitation
And using server "LOCAL"
When "Alice" deletes accepted user "Brian"
Then the HTTP status code should be "204"
Actual:
https://github.com/owncloud/ocis/assets/84779829/e2f669cf-c672-4a32-bc6e-88d2b3fe4a0a
Expected:
I guess web uses wrong DELETE sciencemesh/delete-accepted-user request. should be like DELETE sciencemesh/delete-accepted-user/user-uuid but it doesn't work-> I get 404 error with log:
2024-07-08T11:30:09+02:00 WRN http end="08/Jul/2024:11:30:09 +0200" host=<IP_ADDRESS> line=/Users/scharfviktor/Work/ocis/vendor/github.com/cs3org/reva/v2/internal/http/interceptors/log/log.go:112 method=DELETE pkg=rhttp proto=HTTP/1.1 service=ocm size=19 start="08/Jul/2024:11:30:09 +0200" status=404 time_ns=83250 traceid=6a3800c9de492e66fbb41c96775d3761 uri=/sciencemesh/delete-accepted-user/7effd1e3-7446-4d23-9311-1eb8e8f5c96a url=/delete-accepted-user/7effd1e3-7446-4d23-9311-1eb8e8f5c96a
2024-07-08T11:30:09+02:00 INF access-log bytes=19 duration=63.588667 line=/Users/scharfviktor/Work/ocis/services/proxy/pkg/middleware/accesslog.go:34 method=DELETE path=/sciencemesh/delete-accepted-user/7effd1e3-7446-4d23-9311-1eb8e8f5c96a proto=HTTP/1.1 remote-addr=<IP_ADDRESS>:58187 request-id=vscharf/QqQOxX23Rk-002517 service=proxy status=404 traceid=6fe6932c5aa6029b7fbda0b13bdc685b
2024/07/08 11:30:14 http: TLS handshake error from <IP_ADDRESS>:60588: remote error: tls: unknown certificate
2024-07-08T11:30:14+02:00 ERR IdentifierIdentityManager: fetch failed to get user from userID error="ldap identifier backend get user error: user does not exist or too many entries returned" line=/Users/scharfviktor/Work/ocis/ocis-pkg/log/logrus_wrapper.go:50 service=idp

@kulmann is that an issue of the sciencemesh app?

The web ui does not send any payload, so the server cannot identify which user to delete.

no payload

could you provide some example how should look request? then I can add api test for that

@rhafer amd me tried to use it curl -X DELETE "https://federation-ocis-server:10200/sciencemesh/delete-accepted-user?idp=ocis-server:9200&user_id=f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c" -uadmin:admin -vk
but it doen't work. we got 200 but federated connection still exist

curl -X DELETE "https://federation-ocis-server:10200/sciencemesh/delete-accepted-user?idp=ocis-server:9200&user_id=f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c" -uadmin:admin -vk

This one just didn't work because we were using the wrong userid. The federated users userids look slightlyg diffferent since https://github.com/cs3org/reva/commit/3e6ce6954609265e549e5821f7f136df8dc0fc04. Using the correct ids (those returned by sciencemesh/find-accepted-users) the delete requests works just fine.
So the remaining issue is basically a bug in web. It is sending a DELETE request with an empty body.
To delete a federated user you can either use a json BODY like this:
curl -X DELETE "https://ocis-federation.owncloud.test/sciencemesh/delete-accepted-user" \
   -uadmin:admin -vk -H "Content-Type: application/json" \
   --data '{"idp":"ocis.owncloud.test", "user_id":"MWRhZTRkMTctMzU5Yy00NmNhLWE2ZTUtMGIxOTQ4YmY1ZGJkQG9jaXMub3duY2xvdWQudGVzdA=="}'

Or form-urlencoded parameters as in @ScharfViktor  example above.
I am transferring this issue to web, since the ocis side seems to work AFAICS.


Feature request: Improve support for SQS-wrapped S3 event notifications

Use case
Originally discussed in https://github.com/awslabs/aws-lambda-powertools-python/issues/1656#issuecomment-1384670725.
S3 event notifications can be sometimes be ingested into Lambda via an intermediary such as an SQS queue (i.e. Lambda event source), for various architectural reasons - batching, retries, etc.. However, from the Lambda function's perspective, the intermediary might not be too important; what's important is the S3 event notification itself.
With the current Powertools built-ins, it is possible to parse the S3 event notification data structure out of the SQS message data structure, but this requires some awkward boilerplate code to chain the two data structures together (mostly due to the JSON-formatted wrapper in the middle):
from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor
from aws_lambda_powertools.utilities.parser.models import SqsRecordModel, S3Model
from pydantic import Json

class SqsS3EventNotificationModel(SqsRecordModel):
  body: Json[S3Model]

processor = BatchProcessor(event_type=EventType.SQS, model=SqsS3EventNotificationModel)

def record_handler(record: SqsS3EventNotificationModel):
  for s3_record in record.body.Records:
    do_something_useful(s3_record)

@batch_processor(record_handler=record_handler, processor=processor)
def handler(event, context):
  return processor.response()

Solution/User Experience
A simple quick-win to the user-experience would be to make a new data model available as a built-in:
from aws_lambda_powertools.utilities.parser.models import SqsRecordModel, S3Model
from pydantic import Json

class SqsS3EventNotificationModel(SqsRecordModel):
  body: Json[S3Model]

This would be compatible with the existing framework and could be used as just another model, e.g.:
from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor

processor = BatchProcessor(event_type=EventType.SQS, model=SqsS3EventNotificationModel)

def record_handler(record: SqsS3EventNotificationModel):
  for s3_record in record.body.Records:
    do_something_useful(s3_record)

@batch_processor(record_handler=record_handler, processor=processor)
def handler(event, context):
  return processor.response()

Note: see this comment about Mypy compatibility, although I believe it may no longer be a concern due to a recent typing improvement.
Alternative solutions
If there's a way to skip the intermediate handler function, that would be even nicer (but see trade-off below).
In other words, rather than needing to define a def record_handler(record: SqsS3EventNotificationModel) that iterates through the record.body.Records, it would be nice to just directly define a def record_handler(record: S3Model) and let the processor unwrap everything instead.
This would keep the business logic clean, and not having to care whether it came via SQS or EventBridge, etc..
However, the trade-off to this even simpler interface would be that any event metadata (from both the SQS event and the S3 event notification) would be unavailable.
Acknowledgment

[X] This feature request meets Lambda Powertools Tenets
[X] Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

Really love this suggestion. Even passing your own user defined type/pydantic definition/interface to extend the sqs typing would be great and help out the development process. Thank you for this suggestion

Thank you Alan for following up on that comment and creating this feature —
feel free to go ahead with a PR and we can get it released in our next
release ;-)
<3
On Mon, 3 Apr 2023 at 17:28, Thomas Von Heill @.***>
wrote:

Really love this suggestion. Even passing your own user defined
type/pydantic definition/interface to extend the sqs typing would be great
and help out the development process. Thank you for this suggestion
—
Reply to this email directly, view it on GitHub
https://github.com/awslabs/aws-lambda-powertools-python/issues/2078#issuecomment-1494537138,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZPQBB6MLBHDN53H6VHBZTW7LUB7ANCNFSM6AAAAAAWQTEF5E
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***
com>

Hi @theipster! We are planning the next release and would like to know if you have the bandwidth to submit this PR! It would be amazing to have your contribution here. But don't worry if you can't (we know everyone is busy), we can do it and give you credit.

Hi @leandrodamascena - no guarantees, but I can try to squeeze something in! 🙂
The class itself shouldn't be a problem, but I'm not too familiar with the existing test suite, so any high level guidance on what categories of tests is appropriate for this change would be greatly appreciated - thanks!

Hi @leandrodamascena - no guarantees, but I can try to squeeze something in! slightly_smiling_face
The class itself shouldn't be a problem, but I'm not too familiar with the existing test suite, so any high level guidance on what categories of tests is appropriate for this change would be greatly appreciated - thanks!

I saw you submitted the PR and that's awesome! What do you think converting the PR from draft to work in progress? So we can work together to check the best user experience, to write the tests and documentation.
We use pytest to write our tests; you can see examples of how we do this.
Example: S3 Object Event
Test: https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/tests/functional/parser/test_s3 object_event.py
Event: https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/tests/events/s3ObjectEventIAMUser.json
You can copy these files and change the event and fields to test that the parser is parsing the fields as expected.
Please reach out if you need additional guidance and we can work together to get this done.


Refactored event hooks. Closes#1299

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, 4 unresolved discussions, some commit checks failed.

addon/components/sl-select.js, line 42 [r1] (raw file):
Same question about execution order as left on other PRs

addon/components/sl-select.js, line 53 [r1] (raw file):
Same question about execution order as left on other PRs

addon/components/sl-select.js, line 59 [r1] (raw file):
Do we want to mark this method as @private?

addon/components/sl-select.js, line 69 [r1] (raw file):
Do we want to mark this method as @private?

Comments from the review on Reviewable.io

Please merge latest changes from v0.11.0 branch so that tests can run.

Review status: all files reviewed at latest revision, 4 unresolved discussions.

Comments from the review on Reviewable.io

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 4 unresolved discussions.

Comments from the review on Reviewable.io