CMSImageWithTitle

Summary
Compontent  throws error after mouseOver an selected Image. In scrachtOrga as well as in our dev Sandbox. Hope I've done nothing wrong...
Salesforce Org Type
Scratch Org
Steps To Reproduce

Insert component cmsImageWithTitle on LWR Site.
Add picture from CMS
Move mouse over picture
Error happens in edit as well as in preview mode

Current Behavior

Image is displayed
Error occurs on mouseOver
After reload, error still occurs but image isn't displayed

Expected Behavior
No response
Relevant Log Output
{"subject":"window error","message":"Cannot read properties of undefined (reading 'apply')","stack":"TypeError: Cannot read properties of undefined (reading 'apply')\n

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Does this happen with a component in this sample app, or a custom implementation?

happens also in scratch org

@Maserick74 Got it. Thanks for sharing the screenshot, that explains what you see.
The cmsImageWithTitle component for this sample app is designed to be used within the ligthningRecordForm custom component (the one in the top-right of the shared screenshot), and not standalone.
Specifically the mouse over/out effects are used to invert the colors, and to show different images based on the specified CMS type.
This is what a simplified version of the used JavaScript class would have to look like if you want a standalone component that displays a single CMS item.
import { LightningElement, api, wire } from 'lwc';
import getContent from<EMAIL_ADDRESS>import basePath from '@salesforce/community/basePath';
export default class CmsImageWithTitle extends LightningElement {
    @api contentId;
    @api title;

    @wire(getContent, {
        contentId: '$contentId',
        page: 0,
        pageSize: 1,
        language: 'en_US',
        filterby: ''
    })
    wiredContent({ data, error }) {
        if (data) {
            this._cmsData = data;
            this.altText = this._cmsData.altText.value;
            this.url = basePath + '/sfsites/c' + this._cmsData.source.url;
        }
        if (error) {
            console.log('Error: ' + JSON.stringify(error));
        }
    }

    altText;
    url;

    _cmsData;

    handleClick(e) {
        // do something on click
    }

    get wrapperCss() {
        return this.action !== undefined
            ? `slds-grid slds-text-align_center slds-wrap action`
            : 'slds-grid slds-text-align_center slds-wrap';
    }
}

Hope that helps.

@muenzpraeger - wow, how cool is that!! Helps yeah! Amazing!


Information about time zones added to readme

I have added some how-to about the time zones indo the readme.md file. Related to issue #15 and #27

Somehow this PR went completely unnoticed 😧 I’m so sorry!
Looks really good, thank you very much. This will really help people avoid some hours of debugging 👍


feat(optimus): add dryrun backup api path, request response model

Adding:

DryRunBackup api path
DryRunBackupRequest model
DryRunBackupResponse model
to support dry run backup feature in Optimus

Why are files like odpf/optimus/cluster/command.proto and odpf/common/v1/service.proto shown as updated? I think the PR is not properly rebased?

yes 👍  i have rebased it just now

LGTM


ng-quill or quill memory leak in Firefox in AngularJS 1.6 and uiRouter

I'm not sure whether the issue below is related to quill or ng-quill. I have, however, posted it on the quill repository (https://github.com/quilljs/quill/issues/2088) almost two weeks ago - without any reaction. So, maybe it is indeed related to the AngularJS wrapper. Anyway, here goes again.
It seems that a quill editor in a ui-Router state template of an Angular 1.6 app prevents the whole state/scope from being garbage collectible in Firefox.
Steps for Reproduction

Either go with Firefox to JSfiddle https://jsfiddle.net/ur24r1Lk/
or deploy the code below on a local web server and open it in Firefox:

<!DOCTYPE html>

    <meta charset="utf-8" />
    <link rel="stylesheet" href="//cdn.quilljs.com/1.3.1/quill.snow.css">
    <link rel="stylesheet" href="//cdn.quilljs.com/1.3.1/quill.bubble.css">

    Small state
    State with quill

    <ui-view></ui-view>

(I personally prefer the latter option for memory investigation because that way I can be sure it's not the fiddle/plunker itself causing the problem.)
2.  Take a Heap snapshot (F12 -> Memory -> Take snapshot)
4. On the example webpage provided, click "State with quill" and wait till loaded (the template is on purposed bloated with a big array to make the memory difference more significant).
5. Click on "Small state"
6.  Take again a heap snapshot
Expected behavior:
The amount of memory used in "small state" should be more or less the same after having visited and closed the state with the quill editor.
Actual behavior:
After having visited the state with the quill editor, the amount of memory used is much bigger. The existence of quill obviously prevents the memory from being collected. Kicking the quill editor out of the template, solves the problem: The memory gets collected.
Platforms:
Firefox version 58.0.1
Ubuntu 16.10
Version:
Quill version 1.3.6, ng-quill 3.5.1"
Firefox dominators view screenshot

Also this screenshot point into the direction that the problem is quill-related.
Weird additional infos on Chrome
Chrome definitely has also a quill-related memory problem in my (real-world) application. But it seems, Chrome is under some strange circumstances able to cope with it.
For Chrome, I was not able to isolate the problem into a small demo/plunker. In Chrome it was - strange enough - dependent not only on quill, but the HTML name-attribute of four vanilla inputs, the user-defined language of my web application (switching to English solved the problem, whereas in German, it was always there) and - to make the confusion even bigger - in Chrome the problem was just there in Ubuntu, while in Windows 7 everything was fine concerning the memory.

Maybe it is more or less related how You are using quilljs angular and so
on. I Do Not Know the latest ui Router, but. Maybe There are some Caching
mechanis to reuse components.
You should keep in mind that quilljs is Part of the Window Object. I Do Not
Know, If there is an opportunity to access quill instances through that. So
check If Your quill instance is not longer existing and it should be fine.
And some hint. Since angular 1.5 You should Use components and component
Routing. There You have the component life cycle to handle such things very
elegant, like doing cleanup work.
But i used this with ionic and never got memory issues.
cschwaderer<EMAIL_ADDRESS>schrieb am Di., 8. Mai 2018, 13:43:

I'm not sure whether the issue below is related to quill or ng-quill. I
have, however, posted it on the quill repository (quilljs/quill#2088
https://github.com/quilljs/quill/issues/2088) almost two weeks ago -
without any reaction. So, maybe it is indeed related to the AngularJS
wrapper. Anyway, here goes again.
It seems that a quill editor in a ui-Router state template of an Angular
1.6 app prevents the whole state/scope from being garbage collectible in
Firefox.
Steps for Reproduction

Either go with Firefox to JSfiddle https://jsfiddle.net/ur24r1Lk/
or deploy the code below on a local web server and open it in
Firefox:

Small state
State with quill
<ui-view></ui-view>

(I personally prefer the latter option for memory investigation because
that way I can be sure it's not the fiddle/plunker itself causing the
problem.)
2.  Take a Heap snapshot (F12 -> Memory -> Take snapshot)
4. On the example webpage provided, click "State with quill" and wait till
loaded (the template is on purposed bloated with a big array to make the
memory difference more significant).
5. Click on "Small state"
6.  Take again a heap snapshot
Expected behavior:
The amount of memory used in "small state" should be more or less the same
after having visited and closed the state with the quill editor.
Actual behavior:
After having visited the state with the quill editor, the amount of memory
used is much bigger. The existence of quill obviously prevents the memory
from being collected. Kicking the quill editor out of the template, solves
the problem: The memory gets collected.
Platforms:
Firefox version 58.0.1
Ubuntu 16.10
Version:
Quill version 1.3.6, ng-quill 3.5.1"
Firefox dominators view screenshot
[image: image]
https://user-images.githubusercontent.com/18464127/39290886-65f8f8bc-4931-11e8-84a3-1d34665a1b64.png
Also this screenshot point into the direction that the problem is
quill-related.
Weird additional infos on Chrome
Chrome definitely has also a quill-related memory problem in my
(real-world) application. But it seems, Chrome is under some strange
circumstances able to cope with it.
For Chrome, I was not able to isolate the problem into a small
demo/plunker. In Chrome it was - strange enough - dependent not only on
quill, but the HTML name-attribute of four vanilla inputs, the user-defined
language of my web application (switching to English solved the problem,
whereas in German, it was always there) and - to make the confusion even
bigger - in Chrome the problem was just there in Ubuntu, while in Windows 7
everything was fine concerning the memory.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/KillerCodeMonkey/ng-quill/issues/157, or mute the
thread
https://github.com/notifications/unsubscribe-auth/ACKOYLwQSLbcq-4RXIcGIgLIrRuSUTVFks5twYTfgaJpZM4T2fkT
.

Thanks for your reply!

Refactoring my app for AngularJS 1.5 components won't happen. The code base is simply too huge.
I don't think that the problem is directly routed in uiRouter, because without (ng-)quill, the memory is correctly cleared both in Firefox and in Chrome. uiRouter doesn't seem to have a general memory problem.
Of course, the quill instance lives on as you can see from the screenshot. That is exactly the problem. The question is: Why does it live on? And how to solve that? Killing all quill instances in the window object with $rootScope.$on("$stateChangeStart"?

yep i see the only possibility to remove them with a routing event then. So stateChangeStart or success. which quilljs version and ng-quill do you use?
Because since quilljs 1.0 they should not store global instances for garbage collection

I'm using quill version 1.3.6, ng-quill 3.5.1

May it possible that you test this without ng-quill?
simply use quilljs and init an editor by your self https://quilljs.com/docs/quickstart/#quickstart and check if the problem still exists?

It seems indeed that the problem is gone with just plain quill. This is the code I ended up with, putting the quill definition in the controller:
<!DOCTYPE html>

    <meta charset="utf-8" />
    <link rel="stylesheet" href="//cdn.quilljs.com/1.3.1/quill.snow.css">
    <link rel="stylesheet" href="//cdn.quilljs.com/1.3.1/quill.bubble.css">

    Small state
    State with quill

    <ui-view></ui-view>

strange thing, i did not hold any global references in my component 🤔

@cschwaderer hey, seems like it has something todo, that you directly push the array items to the scope. so you are updating more than 200 times the scope. each scope change rerenders the view.
So i think you are rendering 200 quill editors than.
If build up your demo array and only set the final array on the scope, there is no memory leak.

I cannot confirm that observation. Having changed the code to
var bigArrayPrepare = [];
      for(var i = 0; i < 200; i++) {
       var newArr = [];
       for(var j = 0; j < 200; j++) {
          newArr.push("Inner: " + i + " -- " + j);
        }
       bigArrayPrepare.push(newArr);
      }

      $scope.iAmVeryBig = bigArrayPrepare;

I still get the same behaviour.
Anyway: Pure DOM-rendering of (ng-)quill shouldn't be a problem and shouldn't cause any memory leaks. And changing the scope 200 times while being in one state, is something which actually could happen in a real-world application.

yeah but i can not reproduce the error now with your fiddle... i reloaded the page and my memory snapshots are all 13mb-16mb ... even if i am on the quill page or not

That's strange. I can observe that memory leak in Firefox on three machines with both version 58 and 59 and both on Kubuntu 16.04 and on Windows 7.
Don't know what to do about it. If you cannot reproduce it, you can't fix the problem, obviously.
So, could you maybe tell me, what's the exact path of the quill instanc(es) in the window object to be set to null?

thats the curious part... there are no global quill instances --> check the migration guide of quilljs to 1.0.0.
they removed the remove function because no global instances are stored Oo

do not know how to solve this, because i need to bind it to this, to add the angular functionalities.
Maybe you could investigate if there is the possibility to remove the listeners?
I will try something in the meantime

could you test your code with https://github.com/KillerCodeMonkey/ng-quill/blob/develop/src/ng-quill.js ? i set the "editor" var to null $onDestroy.
It is only a shot in the dark

Thanks! Well, I had only tried that, was also my first thought. However, it's too primitive, probably. It doesn't change anything. I guess, the problem is somewhat deeper.

I'm still on this and was able to further narrow down the problem.
It seems that any reference to $scope, the controller or whatever inside a quill-editor event handler prevents the $scope form being garbage-collected in Firefox.
Leaving aside everything which is obviously not related to the problem, my minimized ng-quill looks now like this:
(function ngQuillFunc01 (root, factory) {
  if (typeof define === 'function' && define.amd) {
    define(['quill'], factory)
  } else if (typeof module !== 'undefined' && typeof exports === 'object') {
    module.exports = factory(require('quill'))
  } else {
    root.Requester = factory(root.Quill)
  }
}(this, function ngQuillFunc02(Quill) {
  'use strict'

  var app
  // declare ngQuill module
  app = angular.module('ngQuill', [])

  app.component('ngQuillEditor', {
    controller: ['$scope', '$element', '$timeout', function ngQuillFunc06($scope, $element, $timeout) {

      var editorElem
      var editor

      this.$postLink = function ngQuillFunc10() {
        // create quill instance after dom is rendered
        $timeout(function  ngQuillFunc11() {
          this._initEditor(editorElem)
        }.bind(this), 0)
      }

      this._initEditor = function  ngQuillFunc12(editorElem) {
        var $editorElem = angular.element('

')
        var container = $element.children()

        editorElem = $editorElem[0]

        container.append($editorElem)

        editor = new Quill(editorElem, {})

        editor.on('text-change', function () {
          if (1==2) {
             console.log($scope); // Any arbitrary reference to $scope. It doesn't matter that it never will get executed
          }
        })
      }
    }]
  })

  return app.name
}))

The crucial line is: console.log($scope); - without this line, there is no memory leak. As soon as $scope becomes part of the editor.on('text-change', function () { closure it seems that it is retained forever.
And at this point, I'm now not sure anymore whether ng-quill is actually the problem. The root of the problem could perhaps be the way quill handles event handlers. Unfortunately, the quill guys rarely respond on anything on github.

did you have any updates on this?

Not really big updates. I tried a lot of things - all in vain. However, fortunately, I was at least able to narrow down the issue even more. I sort of rewrote the problematic bit as an AngularJS directive just to rule out that the problem is the component structure.
The code looks now like this:
angular.module('ngQuill',[]);
angular.module('ngQuill').directive('ngQuillEditor',
    [function()
        {
           return {
                restrict: 'E',
                scope: {
                },
               controller: ['$scope','$element',function($scope,$element) {
                   var editor = new Quill($element[0], {});
                   editor.on('text-change', function () {
                   if (1==2) {
                        console.log($scope);
                    }
                   });
               }]
            };
        }]);

and the memory leak is still there.

I think I've knocked it off. It seems to necessary to unregister the event listeners like so:
var selectionChangeEvent = editor.on('selection-change',  ....
var textChangeEvent = editor.on('text-change', ...

$scope.$on('$destroy', function() {
		 textChangeEvent.removeListener('text-change');
		 selectionChangeEvent.removeListener('selection-change');
});

At least according to my tests, that solves the problem in Firefox. The weird Chrome behavior is still to be tested.

added in 2.8.0


Feature: any way with inactive metrics

We have gauge, and send data to it.
It works fine, but after time we can have some changes:

gauge labels changes
gauge on specific labels not availiable

In first case prometheus create new metric with new labels, but old metric work too, and contains constant invalid data (we dont send it, but prometheus use last catched result).
In second case too: we dont send any new values, but metric work, and contains invalid constant data (last catched result).
If we try use sum, or avg on this gauge - we got invalid results (with invalid metrics). And no way to filter only "actual" values.
Yes, we can use delete_series+clean_tombstones, yes they remove data. But metrics not be removed, and after some time prometheus fill it again with invalid value. And no way to remove it permanently, or mark "invalid"/"inactive", etc...
I attach screenshot after delete_series+clean_tombstones with this problem.
Can you add any way to work with non-actual metrics (may be function to remove it, may be function to filter it, may be any another)?

Thanks for your report. It looks as if this is actually a question about usage and not development.
To make your question, and all replies, easier to find, we suggest you move this over to our user mailing list, which you can also search. If you prefer more interactive help, join or our IRC channel, #prometheus on irc.freenode.net. Please be aware that our IRC channel has no logs, is not searchable, and that people might not answer quickly if they are busy or asleep. If in doubt, you should choose the mailing list.
Once your questions have been answered, please add a short line pointing to relevant replies in case anyone stumbles here via a search engine in the future.


Synthesis failed for node-gtoken

Hello! Autosynth couldn't regenerate node-gtoken. :broken_heart:
Here's the output from running synth.py:
2020-05-15 10:27:02,252 autosynth [INFO] > logs will be written to: /usr/local/google/home/rennie/gitrepos/synthtool/logs/googleapis/node-gtoken
2020-05-15 10:27:03,408 autosynth [DEBUG] > Running: git config --global core.excludesfile /usr/local/google/home/rennie/.autosynth-gitignore
2020-05-15 10:27:03,413 autosynth [DEBUG] > Running: git config user.name Jeffrey Rennie
2020-05-15 10:27:03,418 autosynth [DEBUG] > Running: git config user.email<EMAIL_ADDRESS>2020-05-15 10:27:03,424 autosynth [DEBUG] > Running: git config push.default simple
2020-05-15 10:27:03,430 autosynth [DEBUG] > Running: git branch -f autosynth
2020-05-15 10:27:03,437 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2020-05-15 10:27:03,785 autosynth [DEBUG] > Running: git rev-parse --show-toplevel
2020-05-15 10:27:03,795 autosynth [DEBUG] > Running: git log -1 --pretty=%H
2020-05-15 10:27:03,804 autosynth [DEBUG] > Running: git remote get-url origin
2020-05-15 10:27:04,293 autosynth [DEBUG] > Running: git log be74d3e532faa47eb59f1a0eaebde0860d1d8ab4..HEAD --pretty=%H --no-decorate
2020-05-15 10:27:04,303 autosynth [DEBUG] > Running: git log -1 --pretty=%at be74d3e532faa47eb59f1a0eaebde0860d1d8ab4
2020-05-15 10:27:04,309 autosynth [DEBUG] > Running: git log -1 --pretty=%at<PHONE_NUMBER>c0c7ada19e6a8219d7963ff174b392
2020-05-15 10:27:04,315 autosynth [DEBUG] > Running: git log -1 --pretty=%at 5bbfd095faedfe273819d266f21e402192a29041
2020-05-15 10:27:04,322 autosynth [DEBUG] > Running: git log -1 --pretty=%at 4fa923bd3dafb91df8613accbe2230299cc5b98e
2020-05-15 10:27:04,329 autosynth [DEBUG] > Running: git log -1 --pretty=%at 55cdc844877d97139f25004229842624a6a86a02
2020-05-15 10:27:04,337 autosynth [DEBUG] > Running: git log -1 --pretty=%at 98c50772ec23295c64cf0d2ddf199ea52961fd4c
2020-05-15 10:27:04,344 autosynth [DEBUG] > Running: git log -1 --pretty=%at ba909fca409f6b38eae0fa735614e127d1fc0deb
2020-05-15 10:27:04,350 autosynth [DEBUG] > Running: git log -1 --pretty=%at 7482e79a82e353248769d819788adc1213e8c207
2020-05-15 10:27:04,355 autosynth [DEBUG] > Running: git log -1 --pretty=%at a7759f81c25396207d46532ed389ad4d34879857
2020-05-15 10:27:04,362 autosynth [DEBUG] > Running: git log -1 --pretty=%at 5b48b0716a36ca069db3038da7e205c87a22ed19
2020-05-15 10:27:04,369 autosynth [DEBUG] > Running: git log -1 --pretty=%at c585ac3b5eff5cd2097a5315ffd9cf4823cc1ed2
2020-05-15 10:27:04,376 autosynth [DEBUG] > Running: git log -1 --pretty=%at b0461724be19443075b08c10d4a345cb217002b5
2020-05-15 10:27:04,383 autosynth [DEBUG] > Running: git log -1 --pretty=%at 84c4156c49be9dcabacc8fd7b0585b6fd789ae47
2020-05-15 10:27:04,389 autosynth [DEBUG] > Running: git log -1 --pretty=%at f503622985e230a6792730bbc3b7746c11fce09e
2020-05-15 10:27:04,396 autosynth [DEBUG] > Running: git log -1 --pretty=%at 3d2a7d0e21387ed455c966da9f9897b0a4bc5bb8
2020-05-15 10:27:04,402 autosynth [DEBUG] > Running: git log -1 --pretty=%at 7b7f386b393947a542b87707499f4458136f4f61
2020-05-15 10:27:04,409 autosynth [DEBUG] > Running: git log -1 --pretty=%at f395615039665af6599f69305efcd886685e74f9
2020-05-15 10:27:04,416 autosynth [DEBUG] > Running: git log -1 --pretty=%at b6bdd4783f396f9252ce28af43f7215834a55c3c
2020-05-15 10:27:04,422 autosynth [DEBUG] > Running: git log -1 --pretty=%at 3593e3a995510c0570648d9a48fc756ab2bfc2cb
2020-05-15 10:27:04,429 autosynth [DEBUG] > Running: git log -1 --pretty=%at cb3433f7f554ea751584bdd3631d45ec56a32eb5
2020-05-15 10:27:04,437 autosynth [DEBUG] > Running: git checkout 60c1b3517de08fa7e38db47f0dc6261241a46125
Note: switching to '60c1b3517de08fa7e38db47f0dc6261241a46125'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 60c1b35 chore(deps): update dependency js-green-licenses to v2 (#302)
2020-05-15 10:27:04,446 autosynth [DEBUG] > Running: git checkout cb3433f7f554ea751584bdd3631d45ec56a32eb5
Note: switching to 'cb3433f7f554ea751584bdd3631d45ec56a32eb5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at cb3433f fix: uses http for links to internal logs (#556)
2020-05-15 10:27:04,456 autosynth [DEBUG] > Running: git branch -f autosynth-20
2020-05-15 10:27:04,463 autosynth [DEBUG] > Running: git checkout autosynth-20
Switched to branch 'autosynth-20'
2020-05-15 10:27:04,471 autosynth [INFO] > Running synthtool
2020-05-15 10:27:04,472 autosynth [INFO] > ['/usr/local/google/home/rennie/env3.6/bin/python', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']
2020-05-15 10:27:04,474 autosynth [DEBUG] > Running: /usr/local/google/home/rennie/env3.6/bin/python -m synthtool --metadata synth.metadata synth.py --
/usr/local/google/home/rennie/env3.6/bin/python: No module named synthtool
2020-05-15 10:27:04,507 autosynth [ERROR] > Synthesis failed
2020-05-15 10:27:04,507 autosynth [DEBUG] > Running: git reset --hard HEAD
HEAD is now at 60c1b35 chore(deps): update dependency js-green-licenses to v2 (#302)
2020-05-15 10:27:04,515 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2020-05-15 10:27:04,523 autosynth [DEBUG] > Running: git clean -fdx
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/google/home/rennie/gitrepos/synthtool/autosynth/synth.py", line 612, in <module>
    main()
  File "/usr/local/google/home/rennie/gitrepos/synthtool/autosynth/synth.py", line 473, in main
    return _inner_main(temp_dir)
  File "/usr/local/google/home/rennie/gitrepos/synthtool/autosynth/synth.py", line 592, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/usr/local/google/home/rennie/gitrepos/synthtool/autosynth/synth.py", line 368, in synthesize_loop
    synthesize_inner_loop(toolbox, synthesizer)
  File "/usr/local/google/home/rennie/gitrepos/synthtool/autosynth/synth.py", line 378, in synthesize_inner_loop
    synthesizer, len(toolbox.versions) - 1
  File "/usr/local/google/home/rennie/gitrepos/synthtool/autosynth/synth.py", line 266, in synthesize_version_in_new_branch
    synthesizer.synthesize(synth_log_path, self.environ)
  File "/usr/local/google/home/rennie/gitrepos/synthtool/autosynth/synthesizer.py", line 119, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/usr/local/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/usr/local/google/home/rennie/env3.6/bin/python', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

Autosynth passed, closing! :green_heart:


fix LanguagePrimitives.ErrorStrings messages depends on culture

the SR.GetString use CultureInfo.CurrentUICulture, so need to be invoked at runtime not bound when the module ErrorStrings is opened
the public surface area is not changed (checked with surface unit test and decompile ) because i  converted to the LanguagePrimitives.ErrorStrings module -> type to maintain same signature
Now messages are not initialized at module open, so there is a little performance improvemenet, but is ~ zero, also for programs like:
[<EntryPoint>]
let main argv =
    let d = (1,2)
    printfn "%A" d
    0 // return an integer exit code

There are 5 less invocation of SR.GetString ( i can see it with profiler ), but there is .exe startup cost

Yes, this is a good fix.  I'd be happy to take it without testing, since adding the testing may be fairly hard - but will leave it to @latkin  and @KevinRansom  to decide

Is this just a proposed perf improvement, or is there a functionality bug?

Is the idea that the culture of the error string was previously determined too early, and now is determined when the error occurs?

yes, exactly
    try
        Seq.min []
    with e ->
        printfn "%A" e //use default currentUICulture of process

    // change culture now
    System.Threading.Thread.CurrentThread.CurrentUICulture <- new CultureInfo("it-IT")

    try
        Seq.min []
    with e ->
        printfn "%A" e // e message is wrong, because is not of 'it-IT' culture

a regression test is pretty useless, the rest of the code use the good pattern ( format a message only when needed ).
Easy to check anyway, empty main, open all modules, check for useless ResourceManager.GetString ( like screenshot )
I was checking the performance, but the improvement is very very small and in special cases only :disappointed:
Maybe is a good idea to try to open all modules, and check for useless constructor work too, i'll try

Ok, can you please log this as a dedicated issue, and link it here?


Skip user presence check directly after boot

This patch adds a configuration option to skip the additional user
presence check for the first Get Assertion or Authenticate request
within a certain duration after boot.  In this case, the device
insertion is interpreted as a user presence indicator.

This is a breaking change because a field is added to the exhaustive Config struct.  If you prefer, we could put it behind a feature flag that is disabled by default.  Generally, it could make sense to make Config non-exhaustive so that it can be extended without breaking compatibility.

This seems nice from a UX perspective, two thoughts:

Is this behaviour allowed/suggested anywhere in the spec?
Seems this adds one more fault injection opportunity (but we're not very hardened currently anyway)

I'll go ahead and merge this, and include in the upcoming 0.1.1 release (ignoring that it's breaking since it's just us currently).
We can discuss pros+cons of non-exhaustiveness another time :)


Error when navigating

Hi, i keep getting this error, although i copied and pasted the example.
flutter: ══╡ EXCEPTION CAUGHT BY GESTURE
flutter: The following NoSuchMethodError was thrown while handling a gesture:
flutter: The getter 'isInitialRoute' was called on null.
flutter: Receiver: null
Any ideas?

Problem fixed. Please update your version with 1.1.4

Problem fixed. Please update your version with 1.1.4

Oh thanks! now works perfectly.
Thanks for taking the time to answer me.
Salutes!

Your welcome.


TeX tables

Convert CSV tables to TeX
This will be helpful: https://github.com/narimiran/tably/blob/master/tably.py

Needed this again today :smile_cat:

This would be wonderful! Great idea.


fix(driver): remove useless include which causes compilation issues on Centos6

What type of PR is this?
/kind bug
Any specific area of the project related to this PR?
/area driver-kmod
Does this PR require a change in the driver versions?
/version driver-API-version-patch
but we already bumped it in https://github.com/falcosecurity/libs/pull/1147
What this PR does / why we need it:
This PR fixes the build on old Centos6 kernels
2023-06-14 10:54:53,213  fatal error: linux/kconfig.h: No such file or directory
2023-06-14 10:54:53,213  #include <linux/kconfig.h>

Which issue(s) this PR fixes:
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
NONE

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

driver/OWNERS [Andreagit97]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

driver/OWNERS [Andreagit97,LucaGuerra]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 42f206826a6af1fefd887d06d92f51ea1b7d2d4b

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, jasondellaluce, LucaGuerra
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

driver/OWNERS [Andreagit97,LucaGuerra,jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/milestone 5.1.0+driver


Inappropriate Error Message for Long Path Names for Image Component

Describe the bug
When trying to use long path names for the Image component Source property, the error raised isn't appropriate. The error of choice is E_NETWORK_ERROR which isn't descriptive and doesn't highlight the underlying issue. We have spent just under an hour trying to debug something that ultimately was an issue with the limitation in the way Windows 10 handles long paths.
To Reproduce
Steps to reproduce the behavior:

Create an Image component
Create a file with a long file name (300 characters long)
Use the file picker to pick the image
Attempt to display the image using the Path property from the resulting StorageFile class
The image doesn't display, nothing is logged
Adding an error handler on the image doesn't produce a useful error

Expected behavior
The image displays as it should or an appropriate error is thrown
Screenshots
N/A
Desktop (please complete the following information):

OS: Windows 10 Pro
Browser N/A
Version 1903

@drewbatgit Can you please take a look at this issue?


WEB-5782 Fixed tests and allow Symfony 2.x and 3.x and bumped to PHPUnit 6

Apologies, this was sent by mistake, instead of using our private repository.


Bitvehicle dataset

The bitvehicle dataset used in the tensorflow ssd detection tutorial is not available under the link in the readme, is there another way to obtain it?

@davidrij
Yeap, we know about it.
But unfortunately, we didnt find any mirror. And we have no legal approve to publish BITVehicle dataset by ourselves, but we are trying get it.

链接: https://pan.baidu.com/s/1VcZ5ZueySTgLhyNrAB5wGg 提取码: bfx6

Are those new images available by any chance?
It would be much appreciated!

HI
We found that SG plates are not suitable for LPRNet.

They are multi row
They have spaces and characters are kerned
Also LPRNet does not provide a measure of confidence.

We have stopped using LPRnet in favour of our traditional segmentation type LPR.


Feature request: Making e-mail, MTU and keepalive optional

First of all thanks for the great work, you've done! Makes managing wireguard tunnels a breeze!
I'd be glad to see on the client side e-mail to be optional. Currently it is possible to achieve this by directly editing the json files, but it'd be better if one could simply omit the e-mail field for simple and fast addition of clients.
On the service side, if you don't specify KeepAlive it is set to 0, rather than being completely omitted.
The same goes for MTU - the WireGuard Android app even recommends leaving it to auto.

Client email input is optional now. #109
I will check the other two

Thank you!


ifort build all

Finishing touch: I think we just need to find the type specification for ifort, like the ones we have for gfrotran, to avoid the type mismatch errors. I think this will let us build UM and LFRic ATM with ifort.
This is really an issue with the nightly run configs, not Fab itself.
done:

replaced the hard-coded gfortran -J flag
both gfortran and ifort are tested regularly in #160

Got UM building but will probably need assistance from an LFRic expert to configure a build for ATM.

I presume this is obsolete now since there have been IFort builds.


[BUG] Version >0.14.0 leads to RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Describe the bug
[BUG] Version >0.14.0 leads to RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Config: alpha=0.100000, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1
[2024-05-15 02:35:41,860] [INFO] [utils.py:779:see_memory_usage] Stage 3 initialize beginning
[2024-05-15 02:35:41,861] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.75 GB         CA 0.0 GB         Max_CA 1 GB
[2024-05-15 02:35:41,861] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
[2024-05-15 02:35:41,862] [INFO] [stage3.py:130:__init__] Reduce bucket size 1
[2024-05-15 02:35:41,862] [INFO] [stage3.py:131:__init__] Prefetch bucket size 0
[2024-05-15 02:35:41,999] [INFO] [utils.py:779:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2024-05-15 02:35:42,000] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2024-05-15 02:35:42,000] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
Parameter Offload: Total persistent parameters: 2 in 2 params
[2024-05-15 02:35:42,131] [INFO] [utils.py:779:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2024-05-15 02:35:42,131] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2024-05-15 02:35:42,132] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
[2024-05-15 02:35:42,270] [INFO] [utils.py:779:see_memory_usage] Before creating fp16 partitions
[2024-05-15 02:35:42,270] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2024-05-15 02:35:42,271] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
[2024-05-15 02:35:42,403] [INFO] [utils.py:779:see_memory_usage] After creating fp16 partitions: 1
[2024-05-15 02:35:42,404] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2024-05-15 02:35:42,404] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
[2024-05-15 02:35:42,543] [INFO] [utils.py:779:see_memory_usage] Before creating fp32 partitions
[2024-05-15 02:35:42,544] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2024-05-15 02:35:42,544] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
[2024-05-15 02:35:42,675] [INFO] [utils.py:779:see_memory_usage] After creating fp32 partitions
[2024-05-15 02:35:42,676] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2024-05-15 02:35:42,676] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
[2024-05-15 02:35:42,805] [INFO] [utils.py:779:see_memory_usage] Before initializing optimizer states
[2024-05-15 02:35:42,806] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2024-05-15 02:35:42,806] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
[2024-05-15 02:35:42,941] [INFO] [utils.py:779:see_memory_usage] After initializing optimizer states
[2024-05-15 02:35:42,942] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2024-05-15 02:35:42,942] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
[2024-05-15 02:35:42,942] [INFO] [stage3.py:486:_setup_for_real_optimizer] optimizer state initialized
[2024-05-15 02:35:43,073] [INFO] [utils.py:779:see_memory_usage] After initializing ZeRO optimizer
[2024-05-15 02:35:43,074] [INFO] [utils.py:780:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2024-05-15 02:35:43,074] [INFO] [utils.py:787:see_memory_usage] CPU Virtual Memory:  used = 13.56 GB, percent = 21.9%
[2024-05-15 02:35:43,074] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
[2024-05-15 02:35:43,074] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupLR
[2024-05-15 02:35:43,074] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7fed3c460220>
[2024-05-15 02:35:43,074] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.1], mom=[[0.9, 0.999]]
[2024-05-15 02:35:43,074] [INFO] [config.py:996:print] DeepSpeedEngine configuration:
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   activation_checkpointing_config  {
    "partition_activations": false,
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "synchronize_checkpoint_boundary": false,
    "profile": false
}
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   amp_enabled .................. False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   amp_params ................... False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   autotuning_config ............ {
    "enabled": false,
    "start_step": null,
    "end_step": null,
    "metric_path": null,
    "arg_mappings": null,
    "metric": "throughput",
    "model_info": null,
    "results_dir": "autotuning_results",
    "exps_dir": "autotuning_exps",
    "overwrite": true,
    "fast": true,
    "start_profile_step": 3,
    "end_profile_step": 5,
    "tuner_type": "gridsearch",
    "tuner_early_stopping": 5,
    "tuner_num_trials": 50,
    "model_info_path": null,
    "mp_size": 1,
    "max_train_batch_size": null,
    "min_train_batch_size": 1,
    "max_train_micro_batch_size_per_gpu": 1.024000e+03,
    "min_train_micro_batch_size_per_gpu": 1,
    "num_tuning_micro_batch_sizes": 3
}
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   bfloat16_enabled ............. False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   bfloat16_immediate_grad_update  False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   checkpoint_parallel_write_pipeline  False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   checkpoint_tag_validation_enabled  True
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   checkpoint_tag_validation_fail  False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fed3a9bfd60>
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   communication_data_type ......  False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   curriculum_params_legacy ..... False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   data_efficiency_enabled ...... False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   dataloader_drop_last ......... False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   disable_allgather ............ False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   dump_state ................... False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   dynamic_loss_scale_args ...... {'init_scale': 2, 'scale_window': 1000, 'delayed_shift': 2, 'consecutive_hysteresis': False, 'min_scale': 1}
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   eigenvalue_enabled ........... False
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   eigenvalue_gas_boundary_resolution  1
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   eigenvalue_layer_num ......... 0
[2024-05-15 02:35:43,075] [INFO] [config.py:1000:print]   eigenvalue_max_iter .......... 100
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   eigenvalue_stability ......... 1e-06
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   eigenvalue_tol ............... 0.01
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   eigenvalue_verbose ........... False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   elasticity_enabled ........... False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   flops_profiler_config ........ {
    "enabled": false,
    "recompute_fwd_factor": 0.0,
    "profile_step": 1,
    "module_depth": -1,
    "top_modules": 1,
    "detailed": true,
    "output_file": null
}
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   fp16_auto_cast ............... False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   fp16_enabled ................. True
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   fp16_master_weights_and_gradients  False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   global_rank .................. 0
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   grad_accum_dtype ............. None
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   gradient_accumulation_steps .. 1
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   gradient_clipping ............ 1.0
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   gradient_predivide_factor .... 1.0
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   graph_harvesting ............. False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   initial_dynamic_scale ........ 2
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   load_universal_checkpoint .... False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   loss_scale ................... 0
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   memory_breakdown ............. False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   mics_hierarchial_params_gather  False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   mics_shard_size .............. -1
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   nebula_config ................ {
    "enabled": false,
    "persistent_storage_path": null,
    "persistent_time_interval": 100,
    "num_of_version_in_retention": 2,
    "enable_nebula_load": true,
    "load_path": null
}
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   optimizer_legacy_fusion ...... False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   optimizer_name ............... adamw
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   optimizer_params ............. {'lr': 0.1, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   pld_enabled .................. False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   pld_params ................... False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   prescale_gradients ........... False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   scheduler_name ............... WarmupLR
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 0.1, 'warmup_num_steps': 0}
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   seq_parallel_communication_data_type  torch.float32
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   sparse_attention ............. None
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   sparse_gradients_enabled ..... False
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   steps_per_print .............. inf
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   train_batch_size ............. 8
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   train_micro_batch_size_per_gpu  8
[2024-05-15 02:35:43,076] [INFO] [config.py:1000:print]   use_data_before_expert_parallel_  False
[2024-05-15 02:35:43,077] [INFO] [config.py:1000:print]   use_node_local_storage ....... False
[2024-05-15 02:35:43,077] [INFO] [config.py:1000:print]   wall_clock_breakdown ......... False
[2024-05-15 02:35:43,077] [INFO] [config.py:1000:print]   weight_quantization_config ... None
[2024-05-15 02:35:43,077] [INFO] [config.py:1000:print]   world_size ................... 1
[2024-05-15 02:35:43,077] [INFO] [config.py:1000:print]   zero_allow_untested_optimizer  False
[2024-05-15 02:35:43,077] [INFO] [config.py:1000:print]   zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='cpu', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=True) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=0 param_persistence_threshold=10 model_persistence_threshold=sys.maxsize max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2024-05-15 02:35:43,077] [INFO] [config.py:1000:print]   zero_enabled ................. True
[2024-05-15 02:35:43,077] [INFO] [config.py:1000:print]   zero_force_ds_cpu_optimizer .. True
[2024-05-15 02:35:43,077] [INFO] [config.py:1000:print]   zero_optimization_stage ...... 3
[2024-05-15 02:35:43,077] [INFO] [config.py:986:print_user_config]   json = {
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 1,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "bf16": {
        "enabled": false
    },
    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": 0.1,
            "betas": [0.9, 0.999],
            "eps": 1e-08,
            "weight_decay": 0.0
        }
    },
    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": 0,
            "warmup_max_lr": 0.1,
            "warmup_num_steps": 0
        }
    },
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1.000000e+09,
        "reduce_bucket_size": 1,
        "stage3_prefetch_bucket_size": 0.9,
        "stage3_param_persistence_threshold": 10,
        "stage3_max_live_parameters": 1.000000e+09,
        "stage3_max_reuse_distance": 1.000000e+09,
        "stage3_gather_16bit_weights_on_model_save": true
    },
    "gradient_accumulation_steps": 1,
    "gradient_clipping": 1.0,
    "steps_per_print": inf,
    "train_batch_size": 8,
    "train_micro_batch_size_per_gpu": 8,
    "wall_clock_breakdown": false
}
----------------------------- Captured stderr call -----------------------------
PyTorch: setting up devices
torch.distributed process group is initialized, but parallel_mode != ParallelMode.DISTRIBUTED. In order to use Torch DDP, launch your script with `python -m torch.distributed.launch
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
PyTorch: setting up devices
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).

Hi @pacman100 - thanks for making this issue here to better track it.  Does this also happen with the latest changes in the master branch?

Hi @pacman100 - thanks for making this issue here to better track it. Does this also happen with the latest changes in the master branch?

I confirm that test passes when using the master branch.It would be great to have a patch release if possible.

The problem will exist in Zero3-offload. It seems the problem lies in the partition parameter's part in Zero3 if the model has multiple parallel modules or frozen parameters, the offload procedure cannot load correct parameter mapping. Please take a look and fix it. Thanks.

@bug-fixed, are you able to share repro for zero3-offload case? Thanks!

@tjruwase please try this example (https://github.com/haotian-liu/LLaVA/blob/main/scripts/v1_5/finetune.sh) with zero3-offload. Thanks.

@bug-fixed that repro does not work. Please provide a more precise single script reproductin.

@jomayeri , thanks for the response. The file needed in the script can be downloaded in here: https://huggingface.co/liuhaotian/llava-v1.5-mlp2x-336px-pretrain-vicuna-13b-v1.5/tree/main.
Unfortunately, I think it's difficult for me to prepare a more concise script, apologize for this. I checked the model with only Llama-3, the Zero3-offload works fine. But when I tested it using the script above, i.e., equipped with a vision transformer and another simple linear module, the problem occurred. I guess many factors may lead to the problem. Please note that the conclusion in my previous comment might be wrong because of my very limited knowledge in DeepSpeed. I have the following partial error information for your check:
**deepspeed_aio:   fstat for read failed on /lscratch/26730337/offload/param/zero_stage_3/bfloat16params/rank0/291_param.tensor.swp error = 2**

[cn1112:0]:  File "/vf/users/Panaji/anaconda3/envs/th21_ds/mypkgs/DeepSpeed_0142/deepspeed/runtime/engine.py", line 1582, in _configure_zero_optimizer                                                    [25/1960]
[cn1112:0]:    optimizer = DeepSpeedZeroOptimizer_Stage3(
[cn1112:0]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[cn1112:0]:  File "/vf/users/Panaji/anaconda3/envs/th21_ds/mypkgs/DeepSpeed_0142/deepspeed/runtime/zero/stage3.py", line 362, in __init__
[cn1112:0]:    self._setup_for_real_optimizer()
[cn1112:0]:  File "/vf/users/Panaji/anaconda3/envs/th21_ds/mypkgs/DeepSpeed_0142/deepspeed/runtime/zero/stage3.py", line 472, in _setup_for_real_optimizer
[cn1112:0]:    self._create_fp32_partitions()
[cn1112:0]:  File "/vf/users/Panaji/anaconda3/envs/th21_ds/mypkgs/DeepSpeed_0142/deepspeed/runtime/zero/stage3.py", line 845, in _create_fp32_partitions
[cn1112:0]:    self._swap_in_sub_group_to_flat_buffer(unpinned_fp32_buffer, i)
[cn1112:0]:  File "/vf/users/Panaji/anaconda3/envs/th21_ds/mypkgs/DeepSpeed_0142/deepspeed/runtime/zero/stage3.py", line 762, in _swap_in_sub_group_to_flat_buffer
[cn1112:0]:    param.nvme_swapper.swap_in([param], async_op=False)
[cn1112:0]:  File "/vf/users/Panaji/anaconda3/envs/th21_ds/mypkgs/DeepSpeed_0142/deepspeed/runtime/swap_tensor/partitioned_param_swapper.py", line 306, in swap_in
[cn1112:0]:    swap_in_tensors(self.aio_read_handle, swap_in_buffers, swap_in_paths)
[cn1112:0]:  File "/vf/users/Panaji/anaconda3/envs/th21_ds/mypkgs/DeepSpeed_0142/deepspeed/runtime/swap_tensor/utils.py", line 21, in swap_in_tensors
[cn1112:0]:    assert (swap_handle.async_pread(buffer, path) == 0)

It shows that some parameter file was not saved in the storage. I guess one possible reason is that it failed to build the correct parameter mapping.
My Zero3-offload is:
"zero_optimization": {
  "stage": 3,
  "offload_optimizer": {
    "device": "nvme",
    "nvme_path": "/lscratch/26730337/offload/optimizer",
    "pin_memory": true,
    "ratio": 0.2,
    "buffer_count": 4,
    "fast_init": false
  },
  "offload_param": {
    "device": "nvme",
    "nvme_path": "/lscratch/26730337/offload/param",
    "pin_memory": true,
    "buffer_count": 5,
    "buffer_size": 1e9,
    "max_in_cpu": 1e9
  },
  "overlap_comm": true,
  "contiguous_gradients": true,
  "sub_group_size": 1e9,
  "reduce_bucket_size": "auto",
  "stage3_prefetch_bucket_size": 0,
  "stage3_param_persistence_threshold": "auto",
  "stage3_max_live_parameters": 1e9,
  "stage3_max_reuse_distance": 0,
  "gather_16bit_weights_on_model_save": true
},

@tjruwase I have updated my comment, please kindly check it. Thanks.

@bug-fixed Does the same thing happen when you offload to CPU?

@jomayeri Just encountered this problem. I use CPU offloading, and here is my deepspeed config:
    "zero_optimization": {
        "stage": 3,
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_16bit_weights_on_model_save": true
    }

The specific traceback is
    ret_val = func(*args, **kwargs)  File ".../site-packages/deepspeed/runtime/zero/stage3.py", line 2117, in unscale_and_clip_grads

              ^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/deepspeed/runtime/zero/stage3.py", line 2047, in step
    self.fp32_partitioned_groups_flat[sub_group_id].grad.mul_(1. / combined_scale)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cpu!
    self.unscale_and_clip_grads(sub_group_id, scaled_global_grad_norm)
  File ".../site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/deepspeed/runtime/zero/stage3.py", line 2117, in unscale_and_clip_grads
    self.fp32_partitioned_groups_flat[sub_group_id].grad.mul_(1. / combined_scale)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

I found a workaround. Just manually patching your runtime/zero/stage3.py according to PR 5461 will fix everything.

I found a workaround. Just manually patching your runtime/zero/stage3.py according to PR 5461 will fix everything.

@lihe07 - so using the latest deepspeed built from source works?  You don't hit any issues with Zero stage 3?

@loadams I directly modified the source in my deepspeed 0.14.2 installation, and ZeRO stage 3 is working fluently now.

@bug-fixed Does the same thing happen when you offload to CPU?

@jomayeri The machine I'm working on has very limited memory and is shared with others. it is difficult for me to test the "device": "cpu", option.

Both this issue and https://github.com/microsoft/DeepSpeed/issues/5422 are referring to this line in Zero3 which appears to be reverted to the correct state in master. If any user is having similar issues (@bug-fixed) please open a separate thread.


Read me image broken

Hi, i'm a big fan of yours. I found some error in readme file. The Images in readme are broken, because they were written in relative paths.
I think it is needed to modify.
So I have been modified with Github row paths instead of relative path in my fork repository.
if you allow the pull request, I will send it.
Wait for your answer.

Ah I see. Thanks for bringing this to my attention. Feel free to submit a PR, for sure

Actually I'll just fix it quickly

I know, it's very simple. I hope accept my PR.


Is Torii addon abandonned ?

The Torii addon has 72 open issues and 38 pull requests.
It has not been updated for over 3 years.
At this time, Torii is required in order to use 3rd party authentications .. But with Ember 3.28 and more, the Torii addon starts to be deprecated in a lot of things
What will the deal be for the future? For now ember-simple-auth needs Torii. So users need to fork Torii to update it? Will the ember-simple-auth maintainers do something integrated?

The Torii authenticator was recently deprecated in #2352 and will be removed from ESA in a future breaking release. It is possible to built something that replicates what Torii in a custom authenticator inside ESA. There is no roadmap for this as of now.

Wow ! Ok .. ESA is the most used addon by the Ember community for authentication/authorization implementation and its next breaking release is to remove its social connection feature without a workaround in its roadmap ?
I think propose a full built-in solution, an abstract layer or a real example (without Torii) should be do.

IMHO this is the right decision. Torii is (and always was) unrelated to ESA, and the integration was never nice. Not it seems dead, so better don't hold ESA with it. It is actually pretty straightforward to just use ESA and a custom authenticator.

The Torii authenticator was recently deprecated in #2352 and will be removed from ESA in a future breaking release. It is possible to built something that replicates what Torii in a custom authenticator inside ESA. There is no roadmap for this as of now.

@pichfl I've wrote down a custom Authenticator to replace Torii (and a service to manage the popup). Do you think is better between opening a PR on this repo with that code or creating a separate ember addon for handling that type of authentication flow?

The Torii authenticator was recently deprecated in #2352 and will be removed from ESA in a future breaking release. It is possible to built something that replicates what Torii in a custom authenticator inside ESA. There is no roadmap for this as of now.

@pichfl I've wrote down a custom Authenticator to replace Torii (and a service to manage the popup). Do you think is better opening a PR on this repo with that code or creating a separate ember addon for handling that type of authentication flow?

@davideferre I am looking to write the same thing and using yours as a reference would be a great help!

@pichfl I've wrote down a custom Authenticator to replace Torii (and a service to manage the popup). Do you think is better opening a PR on this repo with that code or creating a separate ember addon for handling that type of authentication flow?

@davideferre I did some work in #2367 but wasn't able to allocate the necessary time yet, maybe we can join efforts here. A PR is always welcome.

I can share a PR later today, thanks!

Version 1.0.0-beta.1 of torii has been published.
Source code is located at https://github.com/adopted-ember-addons/torii

ESA is raising a deprecation for using torii, this should be updated


sql/analyzer: resolve_views: Make resolving views an independent analyzer rules that runs before resolve_subqueries.

@zachmu Sorry for the noise. This is actually the right way to do this...we should resolve a view to its definition very early in the analyzer process, and let it go through normal resolve_subqueries resolution.


Doesn't work on last mobile Chrome on Android

Try to open demo link on android chrome on Android, it doesn't work.

Web Notifications aren't supported in Chrome on Android I believe, and must instead be used from within the Service Worker API. Trying new Notification('hello'); in the browser gives the error:
TypeError: Failed to construct 'Notification': Illegal constructor. Use ServiceWorkerRegistration.showNotification() instead.


This program was not compiled for SM 89   : cudaErrorInvalidDevice: invalid device ordinal

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I'm convinced that this is not my fault but a bug.

Description
I just followed the tutorial for building the newest autoware inside docker. I am trying to run autoware universe with AWSIM according to the documentation, but the perception ist not working properly. I dont see bounding boxes. Looking at the Console I can see the following error:

[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'lidar_centerpoint' of type 'centerpoint::LidarCenterPointNode' in container 'pointcloud_container': Component constructor threw an exception: This program was not compiled for SM 89
: cudaErrorInvalidDevice: invalid device ordinal

Expected behavior
Perception should work and I should see bounding boxes.
Actual behavior
Lidarcenterpoint throws and error.
Steps to reproduce

ros2 launch autoware_launch e2e_simulator.launch.xml vehicle_model:=sample_vehicle sensor_model:=awsim_sensor_kit map_path:=

Versions

ROS2 Humble
Newest Autoware Universe
AWSIM 1.2.0
NVIDIA RTX 4090

Possible causes
No response
Additional context
No response

I will test this with the latest docker and see if I can reproduce the error.

@Aysenayilmaz could you test it on an RTX4080 too?

I fixed the Issue by updating my nvidia driver from 525 to 545.

Hello, may I ask if the problem has been resolved? I am currently facing the same problem as you.

Hello, may I ask if the problem has been resolved? I am currently facing the same problem as you.

which nvidia driver are you using?

Hello, may I ask if the problem has been resolved? I am currently facing the same problem as you.

which nvidia driver are you using?

hello, nvidia drinver 535

Hello, may I ask if the problem has been resolved? I am currently facing the same problem as you.

which nvidia driver are you using?

Hello, my computer configuration is as follows:
Ubuntu20.04, 4060Ti, nvidia driver 535
I can't find the NVIDIA driver 545 version on my computer. Do you have any good methods to update the NVIDIA driver from 535 to 545?

Autoware requires Ubuntu 22.04 to function.
If you use Ubuntu 20.04, maybe the galactic branch might work for you but I don't know.

Autoware requires Ubuntu 22.04 to function.
If you use Ubuntu 20.04, maybe the galactic branch might work for you but I don't know.

Hello, thank you very much for your suggestion, but I installed it through Docker. I modified the nvidia driver from 535 to 545 and the error was resolved.

Hello, may I ask if the problem has been resolved? I am currently facing the same problem as you.

which nvidia driver are you using?

Hello, my computer configuration is as follows: Ubuntu20.04, 4060Ti, nvidia driver 535 I can't find the NVIDIA driver 545 version on my computer. Do you have any good methods to update the NVIDIA driver from 535 to 545?

Update your system:

sudo apt update
sudo apt upgrade

Add the NVIDIA PPA

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update

Install the NVIDIA driver

sudo apt install nvidia-driver-545

Reboot you system to apply the changes

sudo reboot


[deck] deleting objects does not close the plank they were open in

This was resolved by #6720, please reopen if this is still the case on main.