API not working

I just checked that, the API has not returned the user data by calling the function
 $api = new \Sovit\TikTok\Api();         $account = $api->getUser($username);
Could you please confirm it?

I think the TikTok changed the code.

The code was updated, but the followers are rounded.

I think the TikTok changed the code.

Yes, I checked with other API they also facing the same issue.
Do you find any other solution to it.?


fix msbuild references, only load one set

Building inside of visual studio was failing with
1>C:\repos\visualfsharp\src\fsharp\MSBuildReferenceResolver.fs(95,29): error FS0001: This expression was expected to have type    'Microsoft.Build.Utilities.TargetDotNetFrameworkVersion (Microsoft.Build.Utilities.v12.0, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)'    but here has type    'Microsoft.Build.Utilities.TargetDotNetFrameworkVersion (Microsoft.Build.Utilities.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)'
1>C:\repos\visualfsharp\src\fsharp\MSBuildReferenceResolver.fs(96,30): error FS0001: This expression was expected to have type    'Microsoft.Build.Utilities.TargetDotNetFrameworkVersion (Microsoft.Build.Utilities.v12.0, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)'    but here has type    'Microsoft.Build.Utilities.TargetDotNetFrameworkVersion (Microsoft.Build.Utilities.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)'
1>C:\repos\visualfsharp\src\fsharp\MSBuildReferenceResolver.fs(137,61): error FS0001: This expression was expected to have type    'Microsoft.Build.Utilities.TargetDotNetFrameworkVersion (Microsoft.Build.Utilities.v12.0, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)'    but here has type    'Microsoft.Build.Utilities.TargetDotNetFrameworkVersion (Microsoft.Build.Utilities.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)'
1>C:\repos\visualfsharp\src\fsharp\MSBuildReferenceResolver.fs(138,63): error FS0001: This expression was expected to have type    'Microsoft.Build.Utilities.TargetDotNetFrameworkVersion (Microsoft.Build.Utilities.v12.0, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)'    but here has type    'Microsoft.Build.Utilities.TargetDotNetFrameworkVersion (Microsoft.Build.Utilities.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)'

because the references were getting resolved to
1>-r:C:\repos\visualfsharp\src\fsharp\FSharp.Compiler\..\..\..\packages\Microsoft.VisualFSharp.Msbuild.<IP_ADDRESS>.1\lib\net45\Microsoft.Build.dll
1>-r:C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.Build\v4.0_<IP_ADDRESS>__b03f5f7f11d50a3a\Microsoft.Build.dll
1>-r:C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.Build.Engine\v4.0_<IP_ADDRESS>__b03f5f7f11d50a3a\Microsoft.Build.Engine.dll
1>-r:C:\repos\visualfsharp\src\fsharp\FSharp.Compiler\..\..\..\packages\Microsoft.VisualFSharp.Msbuild.<IP_ADDRESS>.1\lib\net45\Microsoft.Build.Framework.dll
1>-r:C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.Build.Framework\v4.0_<IP_ADDRESS>__b03f5f7f11d50a3a\Microsoft.Build.Framework.dll
1>-r:C:\repos\visualfsharp\src\fsharp\FSharp.Compiler\..\..\..\packages\Microsoft.VisualFSharp.Msbuild.<IP_ADDRESS>.1\lib\net45\Microsoft.Build.Tasks.Core.dll
1>-r:C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.Build.Tasks.v12.0\v4.0_<IP_ADDRESS>__b03f5f7f11d50a3a\Microsoft.Build.Tasks.v12.0.dll
1>-r:C:\repos\visualfsharp\src\fsharp\FSharp.Compiler\..\..\..\packages\Microsoft.VisualFSharp.Msbuild.<IP_ADDRESS>.1\lib\net45\Microsoft.Build.Utilities.Core.dll
1>-r:C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.Build.Utilities.v12.0\v4.0_<IP_ADDRESS>__b03f5f7f11d50a3a\Microsoft.Build.Utilities.v12.0.dll

this stops it from loading msbuild12 and msbuild15 dlls and building in VS works again

Faced that problem and had to manually fix the fsproj too, is the change to FSharp.Core project needed?
Is it worth to also change the nuget restore command to not use the version number, we will have less churn and chance to break stuff with future updates.

this fixed it for me! good work

@cloudRoutine I have a fix for the coreclr build I will push it, after the tests have completed on my machine.

awesome I didn't get a chance to dig into it too far, I was trying to port VFPT's metadata signature page generation all day, but I can't seem to get the new text buffers to spawn and focus properly 😑

@cloudRoutine think I may have fixed the Ubuntu build issue.

Cool. Can you please merge to master then?


redundant executions in Ensemble Model

Issue Description
What did you expect to see happen?
In each Picard iteration, I expect there will be only one execution for each model.
What did you see instead?
Multiple executions of the same model can happen. For example, if we have two models 'a' and 'b', and there is a loop between them. In other word, the outputs of model 'a' will be the inputs of model 'b', and there will be a feedback from model 'b' to model 'a'. In this case, during each Picard iteration, model 'a' will be executed twice. I think one of the execution of model 'a' is unnecessary.
Do you have a suggested fix for the development team?
Please attach the input file(s) that generate this error.  The simpler the input, the faster we can find the issue.

For Change Control Board: Issue Review
This review should occur before any development is performed as a response to this issue.

[x] 1. Is it tagged with a type: defect or improvement?
[x] 2. Is it tagged with a priority: critical, normal or minor?
[x] 3. If it will impact requirements or requirements tests, is it tagged with requirements?
[x] 4. If it is a defect, can it cause wrong results for users? If so an email needs to be sent to the users.
[x] 5. Is a rationale provided? (Such as explaining why the improvement is needed or why current code is wrong.)

I don't think it can cause wrong results for the users.
Issue Reviewed by @joshua-cogliati-inl on 2017-Sept-12

For Change Control Board: Issue Closure
This review should occur when the issue is imminently going to be closed.

[x] 1. If the issue is a defect, is the defect fixed? yes
[x] 2. If the issue is a defect, is the defect tested for in the regression test system?  (If not explain why not.) Tests updated
[x] 3. If the issue can impact users, has an email to the users group been written (the email should specify if the defect impacts stable or master)? Email written
[x] 4. If the issue is a defect, does it impact the latest stable branch? If yes, is there any issue tagged with stable (create if needed)? no stable
[x] 5. If the issue is being closed without a merge request, has an explanation of why it is being closed been provided? Will be closed with merge

Issue Close Reviewed by @joshua-cogliati-inl on 2017-Sept-14

This issue can be closed with PR #348


fix: Support Yarn Berry when creating sources zip

Fixes yarn zip -b firefox command on a monorepo with the following file structure
and browser extension dependencies such as "@xxx/foo-package-a@workspace:*".
.
├── apps
│   ├── xxx
│   └── browser-extension   <-- command is executed from here
├── packages
│   ├── foo
│   │   ├── package-a
│   │   └── package-b
│   └── bar
│       ├── xxx

TODO: add source code export test (dependencies install & build) for every supported package manager.
For now, I tested the build on my project with the following shell script.
#!/bin/sh

# Make script stop on errors
set -euo pipefail

# Cleanup previous build artifacts
rm -Rf /tmp/browser-extension-sources ?> /dev/null

# Build from mono-repository
yarn
yarn zip -b firefox

# Extract sources
mkdir -p /tmp/browser-extension-sources
unzip ./.output/browser-extension-1.0.0-sources.zip -d /tmp/browser-extension-sources
cd /tmp/browser-extension-sources

# Build from extracted sources to ensure that the source archive from the mono-repo is OK
yarn # or `yarn install --mode="skip-build"``
yarn zip -b firefox

This is a great PR, thanks for working on it!

I'm a bit stuck on this one.
The modifications I'm proposing are suitable for my use case, but I feel they might not be suitable for others.
Firstly, I systematically download workspace-prefixed dependencies, because I only want to publish the source code linked to my extension, and not the one linked to my other applications. So I have to extract the folder containing my extension source code from the monorepo hosting all my projets and update its package.json to make it work outside of a workspace before publishing it.
If someone wanted to publish everything from their monorepo, packaging workspace-prefixed dependencies would make little sense for them.
Secondly, I introduce into the zip.ts file exceptions related to yarn-specific protocols. There may be a cleaner way to do it.
Thirdly, I didn’t add zipping & unzipping tests for the workspace architecture mentioned in the first post of this pull request. It seems that it would need a bit of reflexion. How to test this behavior on NPM, pnpm, Bun and Yarn while keeping the tests small and compatible with the CI process?
Finally, I now feel that the hooks you recently introduced to the zip command may be a better way to handle some of the modifications I made in this pull request.
@aklinker1 I will mark this pull request as ready, but feel free to leave your thoughts on it here if you feel that it shouldn’t be merged as is.

Please note that the first commit I made (changing file://./ to file:./) could probably be cherry-picked and merged with another branch.

I also had problems using wxt to generate a sources zip when using a pnpm monorepo with a similar structure you, @mezannic. I ended up following your idea to use the hooks in the zip command to make a sources zip that worked.
Hoping maybe this will help others:
export default defineWxtModule({
  name: 'wxt-monorepo-hooks',
  hooks: {
    'zip:sources:done': async (wxt, sourcesZipPath) => {
      console.log('\x1b[36mℹ\x1b[0m', 'Re-zipping sources...');

      const projectDirName = path.basename(wxt.config.root);

      /*
       * assumes that the workspace is structured like this:
       *
       * packages/
       *   - package-a/
       * apps/
       *   - app-a/ <- wxt.config.root
       */
      const workspaceRoot = path.resolve(wxt.config.root, '../..');

      const zip = new JSZip();

      // zip the workspace root's pnpm-workspace.yaml
      zip.file(
        'pnpm-workspace.yaml',
        fs.readFileSync(path.resolve(workspaceRoot, 'pnpm-workspace.yaml'))
      );

      // zip the workspace root's package.json
      zip.file(
        'package.json',
        fs.readFileSync(path.resolve(workspaceRoot, 'package.json'))
      );

      // get all files in all packages
      const packageFiles = glob.sync('packages/**/*', {
        cwd: workspaceRoot,
        ignore: ['**/node_modules/**/*'],
      });

      // get all files in the current app
      const appFiles = glob.sync(`apps/${projectDirName}/**/*`, {
        cwd: workspaceRoot,
        ignore: ['**/node_modules/**/*'],
      });

      const files = [...packageFiles, ...appFiles];

      // add SOURCE_CODE_REVIEW.md of the current app to the zip root
      zip.file(
        'SOURCE_CODE_REVIEW.md',
        fs.readFileSync(path.resolve(wxt.config.root, 'SOURCE_CODE_REVIEW.md'))
      );

      for (const file of files) {
        const absolutePath = path.resolve(workspaceRoot, file);
        const content = fs.readFileSync(absolutePath);
        zip.file(file, content);
      }

      await new Promise((resolve, reject) => {
        zip
          .generateNodeStream({
            type: 'nodebuffer',
            ...(wxt.config.zip.compressionLevel === 0
              ? { compression: 'STORE' }
              : {
                  compression: 'DEFLATE',
                  compressionOptions: {
                    level: wxt.config.zip.compressionLevel,
                  },
                }),
          })
          .pipe(fs.createWriteStream(sourcesZipPath))
          .on('error', reject)
          .on('close', resolve);
      });
    },
  },
});

That wasn’t working for me. npm pack doesn’t work with the values for version field that pnpm ls outputs for other packages in a workspace (link:${relativePath}). I can open another issue and/or pull request with a failing test that shows this.


Seems like emberfire is broken with 2.11

Hello,
I just upgraded my app to 2.11, and emberfire does not seems to live update from firebase anymore. Does anyone have the same problem ?

@olivierchatry I'm experiencing the same thing on 2.0.5

@olivierchatry if you're using findAll and not seeing the data update, it might be related to #471

@olivierchatry can confirm that using query instead of findAll will allow data to update when the value is changed, so it appears to be related to #471.

I'm not sure, it works very well using ember + ember-data 2.10, in 2.11 the emberfire callback that is called when firebase receive an update message is never called. I will try to investigate more tomorrow.

Yeah, I just pulled in my updates to ember-try; I've been meaning to carve some time out this week to address but haven't been able to yet. @olivierchatry let us know if you find anything, if we don't cut a fix this week we should aim to set the ember-data dep to "< 2.11" :/

Ok, I found the issue. Ember data internal code replace all call from store.push to the private  store._push, meaning that the hook that is installed at init by ember ( Store reopen ) will not work anymore. I will see if I can change it, but we might be better of finding a better solution than doing the reopen thing if possible.


Documentation of "useTextField" from React-Aria is not up to date

Provide a general summary of the issue here
[Sorry for english mistakes, i'm french]
I am currently doing React components using React-Aria. I was doing a TextField with the useTextField hook but it seems that his documentation is not up to date ! In the code example, "isInvalid" and "validationErrors" are two props that can be used but appears to not exist anymore !
🤔 Expected Behavior?
I should be able to get those props to handle the error cases when a query is sent.
If i'm just doing something wrong, please let me know.

😯 Current Behavior
I am getting two errors, the same, for "isInvalid" and "validationErrors"

Property 'isInvalid' does not exist on type 'TextFieldAria<"input">'.

Property 'validationErrors' does not exist on type 'TextFieldAria<"input">'

💁 Possible Solution
Update the documentation 😅
🔦 Context
No specific context.
🖥 Steps to Reproduce

Install react aria
Create a file.tsx
Paste the example code from the documentation
ERROR 🟥

Version
react-aria: ^3.28.0
What browsers are you seeing the problem on?
Firefox, Chrome, Safari, Microsoft Edge, Other
If other, please specify.
No response
What operating system are you using?
Linux, Xubuntu
🧢 Your Company/Team
No response
🕷 Tracking Issue
No response

Hmm interesting...I’m not sure what’s going on since isInvalid and validationErrors are getting returned by useTextField in the source code
https://github.com/adobe/react-spectrum/blob/3973a772cbc82ae0c25ab41fd75c3dba57794a6e/packages/%40react-aria/textfield/src/useTextField.ts#L212-L213
I believe isInvalid and validationError were added in v3.30 in this PR(https://github.com/adobe/react-spectrum/pull/5288) so I’d make sure the package is up to date but otherwise that’s strange

Hmm interesting...I’m not sure what’s going on since isInvalid and validationErrors are getting returned by useTextField in the source code
https://github.com/adobe/react-spectrum/blob/3973a772cbc82ae0c25ab41fd75c3dba57794a6e/packages/%40react-aria/textfield/src/useTextField.ts#L212-L213
I believe isInvalid and validationError were added in v3.30 in this PR(#5288) so I’d make sure the package is up to date but otherwise that’s strange

God damn it, the easiest thing to do is check the version and update if needed, and i didn't do it !
In fact, i was in 3.28 and just needed to go to the latest version of Aria .....
Thanks @yihuiliao 🙄


add/1 implementation defeats its purpose

In gojq, add/1 is currently defined as:
def add(f): [f] | add;

This requires a memory allocation and thus defeats its purpose, as illustrated by jqlang's def:
def add(f): reduce f as $x (null; . + $x);

Prove it with profiling.

@itchyny wrote:

Prove it with profiling.

$ cat add.jq
def add(s): reduce s as $x (null; . + $x);
add(range(0;$n) | 1)

$ /usr/bin/time -lp gojq -n --argjson n 10000000 -f add.jq
10000000
real 3.62
user 3.33
sys 0.07
            10428416  maximum resident set size

               10060  involuntary context switches
<PHONE_NUMBER>0  instructions retired
<PHONE_NUMBER>9  cycles elapsed
             7876608  peak memory footprint

$ /usr/bin/time -lp gojq -n --argjson n 10000000 '[range(0;$n)|1] | add'
10000000
real 3.16
user 3.93
sys 0.30
           445923328  maximum resident set size

               34947  involuntary context switches
<PHONE_NUMBER>3  instructions retired
<PHONE_NUMBER>3  cycles elapsed
           443813888  peak memory footprint

The surprise here, I think, is that the u+s time for add/0 is significantly higher, no?

Can you exclude the time for constructing the array for add/0? Typical use case is adding the values from an array in given JSON. Also, please profile with various types.

Typical use case is adding the values from an array

Certainly it's a typical use case, but not in the stream-oriented style, e.g. [inputs | add].  One of the strengths of jq (like awk and many others) is its support for the stream-oriented style.  Conversely one of the weaknesses of various tools is that they want to load everything into memory.  gojq has enough memory-related issues that it seems to me one would want to seize every opportunity to alleviate them.
Anyway, I've provided the evidence you asked for and explained the rationale.  If you want to experiment further, please be my guest.


Add adapter for imagecorruptions

Just as we provide adapters for imgaug and albumentations, the imagecorruptions is another library we could add support for:
https://github.com/bethgelab/imagecorruptions
For some reason, they claim this library is not built for augmentations, but this library is being used by imgaug.  Further, this library seems to be maintained while imgaug has not been touched in over a year.

Good news: imgaug uses these in this module, wrapping them in imgaug augmenters. Those can be used with our ForeignAugmentation wrapper already, and imgaug actually already supports all of them (compare the imgaug class list here with the list of imagecorruptions here). The best code is the code we don't have to write!
I updated the README to reflect our support for these.


Wanted to know how to run Subexperiments on Aer Banckends and reconstruct the result to get exaxct probabilities

I am trying to use the newly introduced Automatic Cut finding method from CKT 0.7.2 in which in the tutorial there isnt mentioned that how to run those subcircuits and knitt their result to get exact outcomes. Currently trying Cutting for Deustch Jozsa Algorithm.

Currently the new cutting code supports the reconstruction of expectation values.  There is a draft in #428 for how we might add support for reconstructing (or sampling from) probability distributions.


(Not an issue) Is it possible to suppress the autogenerated section while rendering an array?

Guys, I have more of a question. I want to add a "+" sign to generate a new section for an array as opposed to auto-generate it when the previous item is filled. And in the same way I want to add "x" to remove a section. From what I see, I can add "+" and "-" in the section wrapper, but how would I suppress a new array element rendering? Thanks.

Unfortunately, that's hard-coded, so there's no clean way to suppress it without changing plexus-form itself.


Implement support for classifications on entity proxies

As per https://github.com/odpi/egeria/issues/7153
Also the implementation of the new classifyEntity method the @cmgrote added to the XTDB repository connector is not correct. The purpose of these three new methods (and getHomeClassifications) is to allow a repository to store a classification independently of its entity using the entityProxy as a stub to connect it to. The repository needs to look to see if it has the entity and if so, it can use the old method to add/update/delete the classification. If the entity is not stored then it should add the entity proxy to the store with the classification. When the entity finally turns up in the repository via an event typically, the entity proxy is upgraded to an entity, with care that the home classifications are not lost.

I suspect this is already fixed in snapshot versions of the connector going back as far as September, per https://github.com/odpi/egeria-connector-xtdb/issues/390 — just haven't done a release of the connector with it included yet 🙈

My apologies - the code is there - I was looking at the wrong version ...

No worries — here's the PR for reference: https://github.com/odpi/egeria-connector-xtdb/pull/400
The logic of these transaction functions is a bit confusing to follow, but broadly goes like this:

At connector startup, the connector injects (Clojure) transaction functions into the storage layer (XTDB) — just placeholder code that we can run at any point later.
The XtdbOMRSMetadataCollection methods ultimately call a (Java) transaction function using the <function>.transact() method.
These static .transact() methods prepare all the inputs necessary for the (Clojure) transaction function to run and then invoke the Clojure transaction function (Transaction.builder().invokeFunction(...)).

The logic of the function is defined in Clojure at the top of each Java transaction function (as a static FN string). Immediately after (fn [... is where you'll see the arguments that the Clojure function requires — this is what the static .transact() method is preparing and passing in step (3).

The .invokeFunction() method actually passes over to XTDB to run the (injected at (1)) Clojure transaction function, passing through the arguments prepared.
The Clojure logic then executes: in the case of these classifyMethods, that includes steps like setting a specific transaction point (for consistency), retrieving any existing entity by its guid, calculating any required updates, and then writing any updates back to XTDB. All of these steps happen atomically thanks to the transaction point being used for consistency.
Typically the "calculating any required updates" step is where the Clojure then passes back to the Java transaction function and calls the constructor of the Java class. These constructors are then where most of the logic exists to validate what we've pulled back from the previous steps in Clojure (the result of retrieving any existing entity), apply various validations, revise the object, etc — all in-memory — and then ultimately store the result back into a member named xtdbDoc. (Or throw an Exception if any error is found during the validations.)
This member xtdbDoc is then usually returned back to the Clojure code.
And the Clojure can then pick-up the results of (6) and write them back to XTDB.

I think this is now addressed — but if not, please re-open and let me know what I missed 🙈


not able to change the default like dislike comments section

Hi,
I am unable to see why there are default 123 replies and 23 likes to every new blog added. Can you please tell me about it?

@Ishita-pahwa for blog posts we won't be having likes and replies, it is dummy data,this information will be removed in the latest release.


Add storage implementation based on a different DB

RIght now, all the storage implementations are in-memory or use Redis. It'd be nice to have another implementation based on a different database. The process would help us to verify whether the current storage interface is flexible enough.

Done in #38


RAM usage during fit() on a tf.keras model

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): conda repository
TensorFlow version (use command below): 2.0.1
Python version: 3.7.3
CUDA/cuDNN version: CUDA 10, cuDNN 7.6
GPU model and memory: GeForce GTX 1050 Ti, 4 GB memory

Describe the problem
Situation: running the fit command on a tf.keras model.
Problem: When running on CPU, the fit command takes ~25MB. When running on GPU, it takes ~970MB.
Source code / logs
Use the following code, in a script called test.py:
from memory_profiler import profile
from tensorflow import keras
import tensorflow as tf
import numpy as np

@profile
def test():
    optimizer = keras.optimizers.Adam(lr=1e-3)
    loss = keras.losses.BinaryCrossentropy()

    model = keras.models.Sequential()
    model.add(keras.layers.Flatten())
    model.add(keras.layers.Dense(1))

    model.compile(loss=loss, optimizer=optimizer, metrics=['acc'])

    num_samples = 1280
    batch_size = 64

    inputs = np.random.uniform(size=(num_samples, 100, 100, 3)).astype(np.float32)
    targets = np.random.randint(0, 2, size=(num_samples, 1)).astype(np.int32)

    history = model.fit(inputs, targets, epochs=3, validation_data=(inputs, targets))

test()

and use the following commands, in an environment with tensorflow-gpu and memory_profiler:
export CUDA_VISIBLE_DEVICES=0; python -m memory_profiler test.py
export CUDA_VISIBLE_DEVICES="-1"; python -m memory_profiler test.py
The outputs I get:
Line #    Mem usage    Increment   Line Contents
================================================
     9    249.9 MiB    249.9 MiB   @profile
    10                             def test():
    11    249.9 MiB      0.0 MiB       optimizer = keras.optimizers.Adam(lr=1e-3)
    12    249.9 MiB      0.0 MiB       loss = keras.losses.BinaryCrossentropy()
    13
    14    252.6 MiB      2.7 MiB       model = keras.models.Sequential()
    15    252.6 MiB      0.0 MiB       model.add(keras.layers.Flatten())
    16    252.6 MiB      0.0 MiB       model.add(keras.layers.Dense(1))
    17
    18    252.9 MiB      0.3 MiB       model.compile(loss=loss, optimizer=optimizer, metrics=['acc'])
    19
    20    252.9 MiB      0.0 MiB       num_samples = 1280
    21    252.9 MiB      0.0 MiB       batch_size = 64
    22
    23    399.6 MiB    146.6 MiB       inputs = np.random.uniform(size=(num_samples, 100, 100, 3)).astype(np.float32)
    24    399.6 MiB      0.0 MiB       targets = np.random.randint(0, 2, size=(num_samples, 1)).astype(np.int32)
    25
    26   1383.0 MiB    983.5 MiB       history = model.fit(inputs, targets, epochs=3, validation_data=(inputs, targets))

Line #    Mem usage    Increment   Line Contents
================================================
     9    249.8 MiB    249.8 MiB   @profile
    10                             def test():
    11    249.8 MiB      0.0 MiB       optimizer = keras.optimizers.Adam(lr=1e-3)
    12    249.8 MiB      0.0 MiB       loss = keras.losses.BinaryCrossentropy()
    13
    14    252.6 MiB      2.7 MiB       model = keras.models.Sequential()
    15    252.6 MiB      0.0 MiB       model.add(keras.layers.Flatten())
    16    252.9 MiB      0.3 MiB       model.add(keras.layers.Dense(1))
    17
    18    252.9 MiB      0.0 MiB       model.compile(loss=loss, optimizer=optimizer, metrics=['acc'])
    19
    20    252.9 MiB      0.0 MiB       num_samples = 1280
    21    252.9 MiB      0.0 MiB       batch_size = 64
    22
    23    399.5 MiB    146.6 MiB       inputs = np.random.uniform(size=(num_samples, 100, 100, 3)).astype(np.float32)
    24    399.5 MiB      0.0 MiB       targets = np.random.randint(0, 2, size=(num_samples, 1)).astype(np.int32)
    25
    26    429.3 MiB     29.8 MiB       history = model.fit(inputs, targets, epochs=3, validation_data=(inputs, targets))

Moreover, when I try to use the InceptionV3 model, I get the following results:
RAM used when training on GPU, using the fit method: 13.4 GB
RAM used when training on GPU, using a custom training loop: 943 MB
RAM used when training on CPU, using the fit method: 13 GB
RAM used when training on CPU, using a custom training loop: 582 MB

might be related to #30443

I am experiencing the same problem. Similar to https://github.com/tensorflow/tensorflow/issues/30443, the first epoch is very slow (probably because of data loading) and the memory usage increases constantly after every batch. The other epochs run quickly and memory is stable. Calling the model.fit method seems to cache the data into memory (even though RAM caching was never explicitly turned on).

Same issue here. Using model.fit results in surprisingly high usage of RAM, I couldn't even start training before running out of memory. Then I tried using model.train_on_batch hoping it would not have the problem because only one batch is presented at a time, but sadly the same issue. I'm using tf.data.Dataset with keras model for training.
The same model can be trained using custom training loop with no problem. Just wondering if there is a solution yet?

I have the same issue. No problems running the exact same script with TF 1.14, but with TF 2.0 RAM usage keeps increasing with each epoch until my Jupyter Notebook freezes.

Same issue as well. I am using Keras 2.2.4, CPU and python 3.7.4 under Windows 10.
model = Sequential()
# Encoder
model.add(LSTM(32, activation='relu', input_shape =(timesteps, n_features ), return_sequences=True))
model.add(LSTM(16, activation='relu', return_sequences=False))
model.add(RepeatVector(timesteps))
# Decoder
model.add(LSTM(16, activation='relu', return_sequences=True))
model.add(LSTM(32, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(n_features)))

data.shape
(23764, 20, 2)
model.fit(data, data,
epochs=num_epochs,
batch_size = 32)
I have 16GB of memory, and usage creeps up with each epoch. If I'm lucky I can train 30 epochs, which uses 97% of memory. With more epochs, it generally gets out of memory altogether.
Is there a decent workaround until they fix this?

I am facing the same problem with ResNet50 from keras applications. In the first epoch my 16GB memory is rapidly exceeded. Tried tensorflow 1.14 and tensorflow-gpu 1.14. Also, decreasing the batch size tremendously did not help. So far only facing the issues with models from applications.

I have the same problem, memory usage is increasing after every call to fit. I don't know if this has to do with the callbacks history object,but after around 20-30 calls to fit, all RAM is used (I have 16 GB).

I am facing the same problem with ResNet50 from keras applications. In the first epoch my 16GB memory is rapidly exceeded. Tried tensorflow 1.14 and tensorflow-gpu 1.14. Also, decreasing the batch size tremendously did not help. So far only facing the issues with models from applications.

Update: Tried TF 2.0.0 and it does not show this issue.

Update: I also tried updating to TF 2.0.0. Memory usage of my 16GB tops out at 98%, after 9 out of 30 epochs but seems to stabilise there, dropping to 96-97% and then going up again, but not exceeding 98%.  Seems kind of close to the brink, but it went through.
From: engineerByNature<EMAIL_ADDRESS>Sent: 29 October 2019 16:15
To: tensorflow/tensorflow<EMAIL_ADDRESS>Cc: Garry<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [tensorflow/tensorflow] RAM usage during fit() on a tf.keras model (#31312)
I am facing the same problem with ResNet50 from keras applications. In the first epoch my 16GB memory is rapidly exceeded. Tried tensorflow 1.14 and tensorflow-gpu 1.14. Also, decreasing the batch size tremendously did not help. So far only facing the issues with models from applications.
Update: Tried TF 2.0.0 and it does not show this issue.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/tensorflow/tensorflow/issues/31312?email_source=notifications&email_token=ACCWNRHY5YR3SWLB7TH5GNDQRBOP7A5CNFSM4IJETMI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECRBF5I#issuecomment-547492597 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCWNRGGTQDKICIWOJIEV63QRBOP7ANCNFSM4IJETMIQ .

I'm facing the same issue, but using TF 2.0 alpha works for me. Using the latest release spends all 25 GB of RAM on Google Colab, but with alpha it barely uses 3 GB.

Any progress on this? I'm having the same issue with the current release of TF 2.0 in Colab, memory usage increases dramatically every epoch until the runtime crashes. I'm running an implementation of Wide ResNet.

Don't know if I'm experiencing the same issue, or a different one, but I've also just started encountering an issue with Tensorflow sucking up all available RAM. However for me the trigger was a system upgrade.
I was using Fedora 29 with Tensorflow 2.1.0, and everything worked fine. I then upgraded to Fedora 31, same exact code and Tensorflow version, and now model.fit sucks down all the system memory (~29gb) and OOMs. And it does this without making it through a single epoch. In fact it doesn't appear to make it through a single batch, as the keras progress indicators never show up.
The last thing that shows is Epoch 1/30

Well after spending the entire day trying to get Tensorflow to build from source, my memory issue is resolved.
When I was experiencing the issue I described earlier (model.fit consuming all available memory), I was using the 2.1.0 package from pip. After building Tensorflow from source (same version, 2.1.0) the issue has gone away, and Tensorflow is operating normally again.

The issue is similar to #35030, so you may check the detailed analysis there. Basically, the root cause is that  tf.convert_to_tensor is making a copy when called on NumPy Arrays, leading more memory usage in model.fit().
We are working on the fix now. Please stay tuned. Thanks.

The issue is similar to #35030, so you may check the detailed analysis there. Basically, the root cause is that tf.convert_to_tensor is making a copy when called on NumPy Arrays, leading more memory usage in model.fit().
We are working on the fix now. Please stay tuned. Thanks.

So should I use tensorflow dataset instead of numpy dataset and there will no be memory leak? I'm asking about tensorflow 2.1.0 stable version.

In my case, where I was having memory issues with the PIP version but compiled worked fine, I was using tf.data.Dataset, not numpy. So there may be more than one leak going around.
On February 8, 2020 3:07:56 AM EST, Dmitry Kravchuk<EMAIL_ADDRESS>wrote:

The issue is similar to #35030, so you may check the detailed
analysis there. Basically, the root cause is that
tf.convert_to_tensor is making a copy when called on NumPy Arrays,
leading more memory usage in model.fit().
We are working on the fix now. Please stay tuned. Thanks.

So should I use tensorflow dataset instead of numpy dataset and there
will no be memory leak? I'm asking about tensorflow 2.1.0 stable
version.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/tensorflow/tensorflow/issues/31312#issuecomment-583714439

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

In my case, where I was having memory issues with the PIP version but compiled worked fine, I was using tf.data.Dataset, not numpy. So there may be more than one leak going around.
…
On February 8, 2020 3:07:56 AM EST, Dmitry Kravchuk @.***> wrote: > The issue is similar to #35030, so you may check the detailed analysis there. Basically, the root cause is that tf.convert_to_tensor is making a copy when called on NumPy Arrays, leading more memory usage in model.fit(). > > We are working on the fix now. Please stay tuned. Thanks. So should I use tensorflow dataset instead of numpy dataset and there will no be memory leak? I'm asking about tensorflow 2.1.0 stable version. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #31312 (comment)
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

I'll try tensorflow dataset class instead of numpy. Thx.

Similar leakage case while using tf.data.Dataset on Colab.

@noodlez04 I think this was resolved in recent tf-nightly. I ran your code in GPU and fit took 57.86 MB. Here is a gist with GPU using tf-nightly.
When I ran same code in CPU with tf-nightly, fit took 17.77 MB. Please take a look at the gist here. Thanks!
Please close the issue if this was resolved for you. Thanks!

As it's already fixed, let's close this bug. Feel free to reopen it if needed.

this bug has returned in tf 2.3.0

The issue is similar to #35030, so you may check the detailed analysis there. Basically, the root cause is that tf.convert_to_tensor is making a copy when called on NumPy Arrays, leading more memory usage in model.fit().
We are working on the fix now. Please stay tuned. Thanks.

@yhliang2018 This conversion happens in TF 2.2 when NumPy arrays are passed to model.fit(). https://github.com/tensorflow/tensorflow/blob/v2.2.0/tensorflow/python/keras/engine/data_adapter.py#L265

I had the same problem today, using tensorflow-gpu==2.0.0
My solution was very simple:
pip uninstall tensorflow-gpu==2.0.0
and
pip install --user tensorflow-gpu==1.14.0

@nihir27 Hi. Are there any updates on this issue? Essentially no training process can be completed while this is not resolved.

@lidless-vision @nihir27 @markodjordjic What TF version do you use? Could you try tf-nightly? If this issue still exists in tf-nightly, would you provide a small colab example for repro? So we can investigate the problem.

@yhliang2018 any TF upwards 2.0.0. The gist to what you are referring to is nowhere near a hp tuning process which can produce memory leak.

Yeah, TF2.0.0 does have this issue, and it should be fixed in the following versions (unless there is regression in some later version). So you may try the latest stable version(2.3.0) or tf-nightly which should have no such memory leak.

@yhliang2018 every version upwards from 2.0.0 has exactly the same issue and there is a lot of people talking about it around GH however no progress is being made.

@yhliang2018 I have opened new issue: https://github.com/tensorflow/tensorflow/issues/43627

@yhliang2018 However I think it just creates clutter like many other issues that have the same underlying problem, and have stayed unresolved, or someone has made a statement that version X.X is working, while it is not.

@yhliang2018 thanks for responding, but what is really needed is that someone pushes for this.

@yhliang2018 TF2.3.0 is not working, just to confirm.
algo-1-eprqd_1  | Fitting fold: 0
algo-1-eprqd_1  | memory used: 2072.0
algo-1-eprqd_1  | Fitting fold: 0
algo-1-eprqd_1  | memory uses: 2097.0
algo-1-eprqd_1  | Fitting fold: 0
algo-1-eprqd_1  | memory uses: 2104.0
algo-1-eprqd_1  | Fitting fold: 0
algo-1-eprqd_1  | memory uses: 2114.0
algo-1-eprqd_1  | Fitting fold: 0 for estimator: FFD AEN, with HP: (Dep.: 3, wid.: 26, red.:0.2, epo.:50)
. . .
algo-1-eprqd_1  | memory used: 2188.0

@noodlez04 I think this was resolved in recent tf-nightly. I ran your code in GPU and fit took 57.86 MB. Here is a gist with GPU using tf-nightly.
When I ran same code in CPU with tf-nightly, fit took 17.77 MB. Please take a look at the gist here. Thanks!
Please close the issue if this was resolved for you. Thanks!

According to this comment, we thought this issue was fixed. It might not be true in the following stable versions, sorry about it.
I will look into the issue #43627 and keep you updated on the progress.

@yhliang2018 Thank you.

any update on this on any suggestion on a particular tf version which does not have this issue?

Have the same problem. using model.fit() bit it stops at epoch 18 while loading the dlls. The committed RAM stacks too high.

Have the same problem. using model.fit() bit it stops at epoch 18 while loading the dlls. The committed RAM stacks too high.

@ml118 and @gioxc88 Thank you for keeping this topic alive. It is sad that this issue is not yet resolved. The only workaround I was able to find is to avoid the usage of model.fit methods and run a custom training process as described in:
https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch
There is also the issue #43627 that is not attended at all, and most likely connected to this issue.

@ml118 and @gioxc88 Thank you for keeping this topic alive. It is sad that this issue is not yet resolved. The only workaround I was able to find is to avoid the usage of model.fit methods and run a custom training process as described in:
https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch
There is also the issue #43627 that is not attended at all, and most likely connected to this issue.

I'll also link this other one which I opened a a few weeks ago #45592

I'll also link this other one which I opened a a few weeks ago #45592

@gioxc88 Yup. That's the same issue. Incredible how wide spread the problem is, and it is still persisting; not being attended to.

@gioxc88 Yup. That's the same issue. Incredible how wide spread the problem is, and it is still persisting; not being attended to.

@markodjordjic Thanks for sending the link about implementing the training from scratch. Seems like i have no other solution and has to try this out.

@markodjordjic Thanks for sending the link about implementing the training from scratch. Seems like i have no other solution and has to try this out.

@ml118 You're welcome. I'm glad if I can be of assistance, especially since TF is not doing anything. I need to add a note, that the problem will not be completely removed, but it will be reduced to the point that training is possible. Also, it is recommended to remove all the unnecessary objects like the keras.model at the first point possible, and subsequently use gc.collect() and tf.keras.backend.clear_session(). Let me know how this works out for you.

@ml118 You're welcome. I'm glad if I can be of assistance, especially since TF is not doing anything. I need to add a note, that the problem will not be completely removed, but it will be reduced to the point that training is possible. Also, it is recommended to remove all the unnecessary objects like the keras.model at the first point possible, and subsequently use gc.collect() and tf.keras.backend.clear_session(). Let me know how this works out for you.

@markodjordjic first i will try to expand the paging file (i am on Win10) and see if that helps. It isnt possible to use gc.collect() and tf.keras.backend.clear_session() in model.fit, right? So doing the training from scratch will be my last solution. But i will try it if expanding the paging file wont help. And chare my results here for sure :)

@markodjordjic first i will try to expand the paging file (i am on Win10) and see if that helps. It isnt possible to use gc.collect() and tf.keras.backend.clear_session() in model.fit, right? So doing the training from scratch will be my last solution. But i will try it if expanding the paging file wont help. And chare my results here for sure :)

I've worked with this issue for years now, there's truly no solution. Good luck on your tests, but don't have high hopes. The workaround is really simple: just have your training phase in a dedicated script, call it as a shell command os.process # or whatever, save weights in that script and load them post-train in the calling script.

I've worked with this issue for years now, there's truly no solution. Good luck on your tests, but don't have high hopes. The workaround is really simple: just have your training phase in a dedicated script, call it as a shell command os.process # or whatever, save weights in that script and load them post-train in the calling script.

@lefnire Oh ok. You mean something like model.fit() for 10 epochs. Then safe model, load again and continue for another 10 epochs? Or did you write the training from scratch and then calling it whit a shell command?

@lefnire Oh ok. You mean something like model.fit() for 10 epochs. Then safe model, load again and continue for another 10 epochs? Or did you write the training from scratch and then calling it whit a shell command?

@lefnire Have you tried to use:
https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch
It has solved the problem for me. I was able to complete numerous quite lengthy training jobs on AWS without ever needing to go into saving and restoring model weights in between the (certain number of) epochs.

@lefnire Have you tried to use:
https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch
It has solved the problem for me. I was able to complete numerous quite lengthy training jobs on AWS without ever needing to go into saving and restoring model weights in between the (certain number of) epochs.

@ml118 There is no advantage in running the training from scratch with a shell command. Simply run the script for a training job from scratch, delete objects, gc.collect() them, and clear the session. An increase in memory will be minimal.

@ml118 There is no advantage in running the training from scratch with a shell command. Simply run the script for a training job from scratch, delete objects, gc.collect() them, and clear the session. An increase in memory will be minimal.

Ah, I guess my situation's a bit different. I serve a model for inference, which can online-train new data. Since it's forever-on, mem-leak is a show-stopper eventually. So what I do is:

run.py -> receives jobs, runs inference or training
train.py -> training only. saves model after done
inference.py -> inference only, can be imported & called directly from run.py

run.py: if it needs to train, it calls os.subprocess('./train.py') (I can't remember actual syntax) which will save the model after completion. The os call is blocking, so it knows when it's done; no coordination necessary. Then inference.py loads the model before use each time.
Kinda complex sounding, but it's really simple. TL;DR: put training in a separate script. Point being when the script dies, all RAM is re-claimed. Maybe not what you're looking for in your situation. @markodjordjic man alive, I've tried every possible combo of clear_session() gc.collect() etc, I've tried so many variants of that, don't know how you got it working but I'm cashed out 😓

Ah, I guess my situation's a bit different. I serve a model for inference, which can online-train new data. Since it's forever-on, mem-leak is a show-stopper eventually. So what I do is:

run.py -> receives jobs, runs inference or training
train.py -> training only. saves model after done
inference.py -> inference only, can be imported & called directly from run.py

run.py: if it needs to train, it calls os.subprocess('./train.py') (I can't remember actual syntax) which will save the model after completion. The os call is blocking, so it knows when it's done; no coordination necessary. Then inference.py loads the model before use each time.
Kinda complex sounding, but it's really simple. TL;DR: put training in a separate script. Point being when the script dies, all RAM is re-claimed. Maybe not what you're looking for in your situation. @markodjordjic man alive, I've tried every possible combo of clear_session() gc.collect() etc, I've tried so many variants of that, don't know how you got it working but I'm cashed out 😓

@lefnire Many thanks for sharing the details of your approach. I find it very useful for OL training. The case here was for off-line training (especially dockerized via AWS) and the secret of success was to drop the use of model.fit() an switch completely to training jobs from scratch. This minimized the consumption of RAM after every epoch, and allowed for training to finish.

@lefnire Many thanks for sharing the details of your approach. I find it very useful for OL training. The case here was for off-line training (especially dockerized via AWS) and the secret of success was to drop the use of model.fit() an switch completely to training jobs from scratch. This minimized the consumption of RAM after every epoch, and allowed for training to finish.

I raies the paging file and got to Epoch 26. Its an improvement, not very good, but better. Another thought i have is just train for 20 Epochs. Safe the model, load it again and train for another 20 Epochs with the same training/validation split. This should also work, right?

I raies the paging file and got to Epoch 26. Its an improvement, not very good, but better. Another thought i have is just train for 20 Epochs. Safe the model, load it again and train for another 20 Epochs with the same training/validation split. This should also work, right?

@ml118 I have already given my best advice, and an example of what worked for me. And that is training from scratch without using model.fit() method, described here:
https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch
After ending all the epochs, you should clear session, delete model, and gc.collect().
If you wish to use any other approach (eg. going for 20 epochs, saving, restoring, etc.) that is up to you. I have no information about that, and I cannot give any advice, except that it is unnecessary and cumbersome, especially since the training from scratch works.

@ml118 I have already given my best advice, and an example of what worked for me. And that is training from scratch without using model.fit() method, described here:
https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch
After ending all the epochs, you should clear session, delete model, and gc.collect().
If you wish to use any other approach (eg. going for 20 epochs, saving, restoring, etc.) that is up to you. I have no information about that, and I cannot give any advice, except that it is unnecessary and cumbersome, especially since the training from scratch works.

Any updates on this issue? I implemented a large and complex network inheriting from keras.Model, so of course it uses the builtin fit method. After some testing, I wanted to scale my experiments up and it looks like this is absolutely impossible with this bug. I'll try to go for the training loop from scratch solution, but I'm somewhat suprised that this issue still seems to persist in tf version 2.4.1.

Update: I used a combination of the workarounds recommended by @gabehope and @markodjordjic with success. I can use model.fit() with the following callback:
class MyCustomCallback(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        gc.collect()
        tf.keras.backend.clear_session()

Memory leaks seem to be still present, but very minor. I can train my model now.

Thanks to everyone who was persistently pushing for this issue to not get inactive. As basically everyone who uses TensorFlow 2.x.x and the built-in functions like model.fit() I was experiencing the following issues:

Loads of RAM usage even though I am running NVIDIA GeForce RTX 2080 TI GPUs.
Very slow training.
Some kind of memory leakage (feels like it was somewhat linear).

All of these three issues have been fixed by the solution posted by @rschiewer (https://github.com/tensorflow/tensorflow/issues/31312#issuecomment-813944860) and by adding run_eagerly=True in the model.compile() function.
Hope this might help others, too.

Using run_eagerly=True like @paweller suggests fixes the memory leak for me without any other action needed. However, as far as I'm aware it completely removes the benefit of graph optimizations and should mean a noticeable performance hit. For my application, it did.
Two findings if yout don't want to use run_eagerly=True:

I tried to use tmalloc with no change regarding the memory leak.
I'm currently not able to find the related issue here on github, but I read elswhere that the problem might come from using the builtin relu activation function in one or more layers. My workaround is to use activation=None and add a ReLu activation layer after each of my layers. While this increases clutter and sounds more than a bit esoteric, it eliminated memory leakage in my project without the need for other measures. I'm running my code in graph mode.

@rschiewer you may mean #46475 and/or #46661.

However, as far as I'm aware it completely removes the benefit of graph optimizations and should mean a noticeable performance hit.

@rschiewer: How does it make sense then that my training times per epoch decreased siginificantely to about 1/3 of what they were before? Or may that be caused by the ClearMemory callback? I cannot imagine it does but I might be wrong. Anyways, I will give your finding 2. a try as I am indeed using ReLU as the activation function for all my convolutional layers. I will let you know what it did for me.
Thanks to @bersbersbers for providing the related issue links.

@bersbersbers Yes exactly, it's #46475. Thanks for mentioning it here!
@paweller I'd be a lucky man if I understood how tensorflow graph mode really works. If your training runs faster in eager mode,  there's maybe something going wrong during the generation of the optimized graph. According to my experience, using graph mode is usually faster than not using it. But I've also sunken more than a few hours into debugging cryptic autograph errors, more than once without final success. So you might be better off with the eager execution mode, which just works.

I have now tried putting the activation function as an additional layer and changing it from ReLU to LeakyReLU. It solved the memory leak for me as well. I also tried training on a different machine. Here it does not make a difference in performance whether I use run_eagerly=True or not. Epoch times stay the same regardless (the model is a small one with only four convolutional, one dense and one lambda layer).

I am not a 100% sure about this topic, but it seems the latests gpu docker image fixed the leakage problem for me.

@UntotaufUrlaub which version is it please ?

@wissambenhaddad I used FROM tensorflow/tensorflow:latest-gpu.
So i guess the latest version available back in May. Sadly I got no clue how to tell the exact tag of it today.

I'm using Tensorflow 2.9.1 with Python 3.9 and the memory leak is still present

Hello all,
I've also run into this problem while training a Siamese network with a custom training and testing step (inheriting from tf.keras.Model) and using model.fit(). I faced this exact problem with at least 3 Tensorflow versions (2.6, 2.8 and 2.9).
I tried several of the solutions proposed in previous posts:

Run gc.collect() and  K.clear_session() in a Keras callback (reduced the memory leak but ended up crashing after more epochs).
Move the activation operations to their own layers (e.g.: tf.keras.layers.ReLU instead of using the activation parameter in Conv2D). I appreciated no difference.

I must mention that I did not try passing run_eagerly=True to model.compile() fearing the performance impact it may have.
Finally, what work for me was to instead implement a custom training loop as suggested above. Although, there is more code, you gain customisation and a little bit of performance by individually wrapping the training and validation steps in @tf.function.
I understand it may be a hard to solve issue because it may be caused by a combination of factors, but seeing how many developers are facing the same problems, I think more attention and time are necessary to fix it.

See similar large memory leakage issues in TF 2.6 when calling keras fit() function with tensorflow dataset.

an issue like this is a great example why TF lost the battle to PyTorch (there are other reasons as well) :)

I'm seeming a similar pattern unfortunately, IDK why this issue is closed.

Mac M1 pro
tensorflow-macos  2.9.2
tensorflow-metal   0.5.1

I tried

running gc.collect(), k.clear_session() on my Sequence generator class (helped, but did not solve it)
using a separate Activation layer (didn't work)
model.compile(.. with run_eagerly=True (I'm not sure if it helped, but it did not solve it)

I haven't figured out how to trytcmalloc on my mac.

this is still an issue in October 2023, I am building an image classifier and it keeps crashing

this is still an issue in October 2023, I am building an image classifier and it keeps crashing

what version of keras and tensorflow do you use? probably I have the same problem with keras 2.6.0 and tf 2.6.2

I can confirm the problem is still present.
Running Tensorflow 2.12.0 as the backend of NVidia Merlin (a recommender models framework) on a A10 GPU with 24GB of VRAM, and I see random patterns of crash.
Sometimes the crash happens in the middle of epoch 3, sometimes just epoch 1 etc...
Memory stays almost constant but at some point....💥 it explodes and very long Cuda OOM errors are displayed.
I am trying now all the solutions and hope it will disappear:

compile mode with eager=False
reset state callback at end of each epoch by means of the ClearMemory callback as described above.

2.12 is too old, won't get updated anymore. Can you try on a more recent version, please?


Landing page html and css

Updated landing page html and created css for landing page.

updated landing.css

I updated the html file and connected to the main CSS file.


#258 - ENH: Add Plugin API for 'default providers' and BeanScopeBuilder.provideDefault() methods

Inspiration is really striking, isn't it? I like it.


Rails 5 - Undefined Method 'on' in /capistrano/tasks/rbenv.rake

Steps to reproduce

Setup Rails 5 project as normal
Attempt deploy using Capistrano (bundle exec cap staging deploy --trace)

Expected behaviour
App is deployed to remote server
Actual behaviour
NoMethodError is thrown.  Here is the full stack trace:
| => bundle exec cap staging deploy --trace
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/capistrano-3.1.0/lib/capistrano/i18n.rb:4: warning: key :starting is duplicated and overwritten on line 6
** Invoke staging (first_time)
** Execute staging
** Invoke load:defaults (first_time)
** Execute load:defaults
** Invoke bundler:map_bins (first_time)
** Execute bundler:map_bins
** Invoke deploy:set_rails_env (first_time)
** Execute deploy:set_rails_env
** Invoke deploy:set_linked_dirs (first_time)
** Execute deploy:set_linked_dirs
** Invoke deploy:set_rails_env
** Invoke rbenv:validate (first_time)
** Execute rbenv:validate
cap aborted!
NoMethodError: undefined method `on' for main:Object
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/capistrano-rbenv-2.0.4/lib/capistrano/tasks/rbenv.rake:3:in `block (2 levels) in <top (required)>'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `each'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/capistrano-3.1.0/lib/capistrano/dsl/task_enhancements.rb:12:in `block in after'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `each'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:150:in `invoke_task'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `each'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:106:in `block in top_level'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:115:in `run_with_threads'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:100:in `top_level'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:78:in `block in run'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:176:in `standard_exception_handling'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/rake-10.5.0/lib/rake/application.rb:75:in `run'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/capistrano-3.1.0/lib/capistrano/application.rb:15:in `run'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/capistrano-3.1.0/bin/cap:3:in `<top (required)>'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/bin/cap:23:in `load'
/Users/jeffreyguenther/.rbenv/versions/2.3.0/bin/cap:23:in `<main>'
Tasks: TOP => rbenv:validate

Configuration
OSX El Capitan
Ruby 2.3.0
Rails 5 Beta 3
Capistrano 3.1

Solved:
Updating Capistrano to 3.4 resolved our error.  Whoops ;)


Assert_IsFile.vi hangs when passed large json body requests that are not files

vi.lib\addons_JKI.lib\REST Client\classes\REST Client\Assert_IsFile.vi

The vi above takes in the body of a request to determine if it is a file path. The problem is that "String to Path" takes a very long time to execute for bodies that are large. I have a put request hat has over 1 million characters in it (json string publishing to a web page).  AND for reasons I don't understand, the String to Path returns the massive JSON string as a path!
Even worse, the "File or Folder Exists" hangs when the very large path is passed to it.
So I'm going to have to revert to the previous release of the rest api.
Recommended solutions

get string length. if larger than the OS max path length, this is not a path. (not sure if this will work for both windows and mac).

In any event, an alternate algorithm is needed to determine if this string is a path or not.

@ChrisCilino
How about this? Limiting to 255 char is possible too. I set it to 4x arbitrarily.
It would break as soon as a XML or JSON char is found. We could extend to commas and other punctuations.
Is there an (official) list of disallowed chars for filenames somewhere?

@francois-normandin, You're the best!

I think the 1000 character limit makes sense. Might need to make that a class property in the future but I wouldn't bother for now.
Love breaking on XML or JSON!
I'm not aware of a disallowed character list. And if there were it would be very os specific. And part of the problem would also be network paths. When I try to add an illegal character to a file name on windows I get the error below:

This post has some interesting information.
https://stackoverflow.com/questions/1976007/what-characters-are-forbidden-in-windows-and-linux-directory-names
But as I feared: "Note that the reserved DOS device names and the rule about filenames ending in a dot or spaces are applied by the runtime library when converting a DOS path to a native NT path. If a path starts with the "\?" local-device prefix, this normalization step gets skipped, except to replace "\?" with NT's "??" device prefix. This prefix instructs the object manager to search the logon-session and global DOS device directories for a symbolic link to a native NT device, which is usually a device object in the "\Device" directory."
So I think we'd have to allow question marks.

Good idea to start at the end. We can then definitely limit to 255 chars + 1 for the delimiter.
I thought the native method to check for multiplatform filename would be good, but it does not start on the first fail, so I can only look for inspiration in there.

"I thought the native method to check for multiplatform filename would be good, but it does not start on the first fail, so I can only look for inspiration in there."
i'm not sure I understand what you mean here.

Ha... I'm on my cell phone and auto correct for a bit in the way. I'll make more sense when I'm at the computer. @ChrisCilino 😁

@ChrisCilino I give it more thought and I'm not convinced we should try to optimize for speed, and risk forgetting something. It is a REST call, which is probably not time critical down to the nanosecond.
I propose to cut-off at 4096 (longest Linux path allowed) and simply ensure it is an absolute path to proceed. We can let the "Check If File of Folder Exists" be responsible for the OS independence...
What do you think?

Hmmm..... if memeory serves, an empty path returns "true" from "file or folder exists" so I think what you've coded up will work in that case too since you are checking if the string is empty.
Ship it!

Indeed, it seemed redundant to add the verification in the second case structure since the first should find it, but just to be safe (in case of some weird non-printable or unicode chars in the body, I`ve added more usual checks on empty and not a path.

I'll do a pull request shortly!

happy to verify once the build is out!

@francois-normandin is there anything else I need to do on this or are we just waiting on a build from @jimkring ?

Hi guys, just ran into this issue. Do we know if a fix is foreseen? Or should I stick to previous versions?

@grandangelo This link contains a non-released package with the fix:
https://github.com/vipm-io/HTTP-REST-Client/pull/28#issuecomment-1837557554
The pull request has not yet been accepted and released by JKI, so if you decide to use this unofficial release, please be diligent about managing the distribution on your machines as there is no guarantee that the next package released on vipm.io will be thev from the exact same source code/branch.

I really do wish JKI would release this fix. It's a shame that such a small and quick fix is gated by just rebuilding and releasing, and JKI hasn't yet simply rebuilt. We're now on month 7 of this issue.
@grandangelo I don't know if you have direct connections into JKI, but I've done everything I can to voice my concern. If there's anything you can do to help, I'd be very appreciative.

@ChrisCilino, I've no connection with JKI, apart from having used their library sometimes...
I'm working on a software deployed on more than one hundred machines, with two other colleagues, I found easier to just add a README stating that we should stick to previous versions instead of internally managing a "different" package as suggested by @francois-normandin.

@grandangelo unfortunately that also has been my solution... royal bummer.

Hi @grandangelo @francois-normandin @ChrisCilino,
The PR #28 has been accepted and a new release should be available on vipm.io. Thanks for your extended patience.


feat: Prevent the creation of constant delay schedules that are not a multiple of one second.

As this is not supported anyway by this library it might lead to confusing behaviour
closes #21

Hi @stubents, sorry for getting back to you just now that you have already worked on this.
In my opinion ConstantDelaySchedule, as an alternative implementation of the Schedule interface, should not necessarily suffer from this one second as minimum unit problem, even though I agree with your first assumption about this use case not being very appropriate.
Considering that its logic is contained in the single line return statement of the Next() method, I think similar schedules can be implemented externally case by case. How would you feel about making both the struct and methods private? This way we could handle the error directly in the existing function, without having to deal with the unhandy usage you mentioned.

Hi @janrnc, no worries and thanks for the feedback!
I'm not quite sure if I got you right. You'd like to create another Schedule implementation that is private and only used by the parser?

Not exactly. The idea is to make the ConstantDelaySchedule struct and Every() func private. By doing so, the way we handle the error becomes much less important because it would be just an internal component. The private every() could handle the error itself, without requiring a dedicated function as the MustEvery() you're introducing. What do you think?

Makes sense, actually I think there's no good reason not to use the @every string anyway. I'll change the PR


Fix deadlock during ICETransport.Stop

Calling ConnectionStateChange handler during Stop caused mutex
deadlock. Add another Mutex to protect handler pointer to
avoid the deadlock.
TestPeerConnection_Close_DuringICE is added to reproduces the deadlock in old code.
Reference issue
Fixes #996

Codecov Report

Merging #998 into master will increase coverage by 0.21%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #998      +/-   ##
==========================================
+ Coverage   74.37%   74.59%   +0.21%
==========================================
  Files          66       66
  Lines        5011     5011
==========================================
+ Hits         3727     3738      +11
+ Misses        964      953      -11
  Partials      320      320

Flag
Coverage Δ

#go
76.9% <100%> (+0.24%)
:arrow_up:

#wasm
68.63% <ø> (ø)
:arrow_up:

Impacted Files
Coverage Δ

icetransport.go
76.55% <100%> (ø)
:arrow_up:

dtlstransport.go
74.86% <0%> (+2.09%)
:arrow_up:

sctptransport.go
78.97% <0%> (+3.58%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6165344...00c7234. Read the comment docs.

@at-wat I don't know how you feel about them, but sync.Atomic is really nice in cases like these as well.
I have been slowly moving towards them in simple cases where you just want single ops done (Get/Set/Swap), but keeping multi-operation things as a mutex

@Sean-Der I had thought sync.Atomic.Value has overhead due to handling the value as interface.
https://gist.github.com/at-wat/39bc73f2f87ef16d9b177ed47a529d3c
I'm surprised that atomic.Value.Store/Load is much faster than using sync.Mutex.

One downside of using atomic.Value is that the type of the actual content is not shown at type struct.

@Sean-Der migrated to atomic.Value!

Yea the lack of type safety is shame :( the performance improvement is a real welcome surprise!
The code looks way better now, thanks so much for grabbing this!


sap.m.Table: Hidden elements shift adjacent cells when "minScreenWidth" is applied to all Columns

OpenUI5 version: Since 1.44.5 (affects also current nightly 1.47)
Browser/version (+device/version): All
URL (minimal example if possible): https://embed.plnkr.co/OepBn5/
Steps to reproduce the problem:

Open the Plunker example above
Shrink the preview panel horizontally till columns are shown as pop-in

What is the expected result?
Until 1.44.4

What happens instead?
Since 1.44.5

Any other information? (attach screenshot if possible)
This effect goes away as soon as minScreenWidth of one of the columns is not applied (e.g. change one minScreenWidth value to "Phone" or remove it).

Hi @boghyon if you configure all the columns to be shown as a popin then we cannot have a physical table anymore. There must be always at least one visible real column which is not configured for popin.
Such feature has been asked several times but if we allow this we know people start using table like a form. Thats why this limitation will stay. Please see this to make it work
https://plnkr.co/edit/Dy30RGa3cY00kW7vi6Ii?p=preview

Hey @aborjinik , thank you for the answer. I couldn't find any similar issue about this on GitHub. Sorry about that. Could we at least get a "warning" message placed in the API reference of sap.m.Column that it's not recommended to apply demandPopin to all Columns? Or that at least one column should have demandPopin disabled.

For other readers: since UI5 1.72, at least one column is always kept in the header:

Related commit: https://github.com/SAP/openui5/commit/307b09bc71d3cc2cd6f3fc3f83ffa8c824aa6ee1


db-init.sh is run before db is spun up

Need to add database and collection for API to use.  Here's a solution https://github.com/arangodb/arangodb-docker/issues/83, but for now, work around is to just run it as an entrypoint for API (and that works for now, but not an awesome solution).
Here's the arangodb docker image.
https://hub.docker.com/_/arangodb

Ended up pulling it out completely, and running it along side docker compose.  Will need to readdress this eventually when  deploying with kubernetes.


Update README.md

Made grammatical corrections.

I think you made a pull request to the wrong repo.
Hugo


chore: remove deprecated lime APIs

Related Issues/PRs

#xxx
What changes are proposed in this pull request?
Briefly describe the changes included in this Pull Request.
How is this patch tested?

[ ] I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change any dependencies?

[ ] No. You can skip this section.
[ ] Yes. Make sure the dependencies are resolved correctly, and list changes here.

Does this PR add a new feature? If so, have you added samples on website?

[ ] No. You can skip this section.
[ ] Yes. Make sure you have added samples following below steps.

Find the corresponding markdown file for your new feature in website/docs/documentation folder.
Make sure you choose the correct class estimators/transformers and namespace.
Follow the pattern in markdown file and add another section for your new API, including pyspark, scala (and .NET potentially) samples.
Make sure the DocTable points to correct API link.
Navigate to website folder, and run yarn run start to make sure the website renders correctly.
Don't forget to add <!--pytest-codeblocks:cont--> before each python code blocks to enable auto-tests for python samples.
Make sure the WebsiteSamplesTests job pass in the pipeline.

AB#1936951

/azp run

Codecov Report

Merging #1620 (f5202a2) into master (7e90d19) will decrease coverage by 65.64%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           master    #1620       +/-   ##
===========================================
- Coverage   83.72%   18.07%   -65.65%
===========================================
  Files         292      272       -20
  Lines       15473    14361     -1112
  Branches      752      761        +9
===========================================
- Hits        12955     2596    -10359
- Misses       2518    11765     +9247

Impacted Files
Coverage Δ

...t/azure/synapse/ml/explainers/ImageExplainer.scala
0.00% <ø> (-100.00%)
:arrow_down:

...rosoft/azure/synapse/ml/explainers/ImageLIME.scala
16.12% <ø> (-77.42%)
:arrow_down:

...rosoft/azure/synapse/ml/explainers/ImageSHAP.scala
13.33% <ø> (-80.00%)
:arrow_down:

...crosoft/azure/synapse/ml/explainers/LIMEBase.scala
18.46% <0.00%> (-79.73%)
:arrow_down:

...icrosoft/azure/synapse/ml/explainers/Sampler.scala
0.00% <ø> (-100.00%)
:arrow_down:

.../microsoft/azure/synapse/ml/image/Superpixel.scala
2.36% <ø> (ø)

...azure/synapse/ml/image/SuperpixelTransformer.scala
29.41% <ø> (ø)

...m/src/main/scala/com/microsoft/lightgbm/SWIG.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...cala/org/apache/spark/ml/NamespaceInjections.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...rg/apache/spark/injections/BlockManagerUtils.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 258 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion?


Permit one or more permissions

Supports authorisation of a single permission, any of the given
permissions or all of the given permissions:
authorise_user!(any_of: %(manager editor))

authorise_user!(all_of: %(manager editor))

authorise_user!('manager')

This maintains backwards compatibility with the existing, scalar
permission API - as specified in the final example above.

I'll update the docs if this is considered acceptable, feature-wise.

It's not immediately clear from the method name that this is an "or" check - I think I would've expected this to be behave like "and". There seems potential for confusion about the behaviour.
A little more involved, but perhaps more intention revealing way would be to pass a block. In your case:
authorise_user!(
  User::RESOURCE_MANAGER_PERMISSION,
  User::CONTACT_CENTRE_TEAM_LEADER_PERMISSION
)

becomes
authorise_user! do |user|
  user.has_permission?(User::RESOURCE_MANAGER_PERMISSION) ||
    user.has_permission?(User::CONTACT_CENTRE_TEAM_LEADER_PERMISSION)
end

@tijmenb I do agree and wondered how best to approach this. I'm not really a fan of the block-form. How about would you feel about all: or any: keyword args doing the AND / OR checks respectively? I could maintain backwards compatibility when a string is passed. This feels like a nicer, more obvious API.

Not a block-fan? Blasphemy!
I wondered about kwargs but it still doesn't really read like English. A developer first encountering authorize_user!(any: ["Manager", "Team Lead"] might still not know what's up without diving into this gem. Also the implementation becomes a little fiddly probably.
Perhaps we expose some extra methods to make it easier to keep this logic inside your application?
before_filter do
  deny_access! unless user.has_any_of_permissions?(
    User::RESOURCE_MANAGER_PERMISSION,
    User::CONTACT_CENTRE_TEAM_LEADER_PERMISSION
  )
end

Alternatively, expose a method that spells it out:
before_filter do
  ensure_user_has_at_least_one_of_permissions!(
    User::RESOURCE_MANAGER_PERMISSION,
    User::CONTACT_CENTRE_TEAM_LEADER_PERMISSION
  )
end

My vote is for kwargs of any_of: and all_of:. The intention should be clear enough then I think. Possibly for now raise an ArgumentError if both are supplied, because lol good luck working out what that should mean.

@boffbowsh That makes a lot of sense to me. 👍 I'll be back...

@tijmenb @boffbowsh Should cover every case now but it's not the most straight-forward code I ever wrote. Happy to take a steer on to tidying /clarifying this if you have any!

This kind of confirms my worry about fiddly implementation. I suspect that if you go for separate methods for each behaviour you'll end up with less code and conditionals.
But if you're all hell-bent on kwarging, then something like:
def authorize_user!(permissions)
  case permissions
  when String
    authorize_user_with_permissions!([permissions])
  when Hash
    raise ArgumentError if permissions.keys.size > 1

    if permissions[:any_of]
      authorize_user_with_at_least_one_of_permissions!(permissions)
    elsif permissions[:all_of]
      authorize_user_has_permissions!(permissions)
    else
      raise ArgumentError
    end
  end
end

You can implement authorize_user_with_permissions! and authorize_user_with_at_least_one_of_permissions! as private methods.

@tijmenb I think the kwargs approach definitely results in a nicer public API. I'll play around with some of your suggestions to see if I can tidy up the internals.

So now I wonder if removing the kwargs and having the following (simplified) public API could also make more sense:
authorise_user_with_at_least_one_of_permissions!(permissions)

authorise_user_with_all_permissions!(permissions)

authorise_user!(permission)

Rather than authorise_user!(...) being the entry-point.

Anyway, @tijmenb @boffbowsh - updated again. 👍

@boffbowsh Pushed a small update to provide a message when failing on a single permission.

👍 from me. As this is an external contribution, someone else from GOV.UK will need to merge it.

Thanks for your help with this 😍


Azure SQL DB breaks if disconnected then connected

Azure Data Studio Version: master

Steps to Reproduce:

Connect to any Azure SQL DB from the connection dialog. Check the Remember Password checkbox.
Right click the node from Servers and disconnect.
Connect to that node again.

Expected - Object Explorer just expands.
Actual - Errors out with the following error -

@Charles-Gagnon @kburtram This issue seems to be fixed by my earlier fix https://github.com/Microsoft/azuredatastudio/pull/5256. Closing as I can't repro after the fix.


Using tf.data.dataset for inference with high efficiency

Please make sure that this is a documentation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:doc_template
System information

TensorFlow version: master
Doc Link:  https://www.tensorflow.org/guide/datasets

tf.data.dataset provide a high level api for data importing and preprocessing. It's convenient to be used in training process, however, it is unclear how to use them in the inference scenario.  We want to reuse the data preprocessing code  for inference which using the interface of tf.data.dataset. By google search and self experiment I found the following ways to use tf.data.dataset for inference:
#using a placeholder
x = tf.placeholder(tf.float32, shape=[None,2])
dataset = tf.data.Dataset.from_tensor_slices(x)
data = np.random.sample((100,2))
iter = dataset.make_initializable_iterator() # create the iterator
el = iter.get_next()
with tf.Session() as sess:
    # feed the placeholder with data
    sess.run(iter.initializer, feed_dict={ x: data })
    print(sess.run(el)) # output [ 0.52374458  0.71968478]

But the way above will bring some cost, Specifically ,there are two main operations:

we must run the function sess.run  one more time
dataset object need be to initialize each time

is it efficient to do in this way,  Or we need another way to use tf.data.dataset so that we can keep efficiency?

Update:
using tf.contrib.data.get_single_element  or tf.data.experimental.get_single_element

Regarding your update, does using tf.data.experimental.get_single_element() work well for your case?

Regarding your update, does using tf.data.experimental.get_single_element() work well for your case?

Yes

Great! In that case, I'll close this issue.


Bank of Anthos APP automatically redirect to HTTPS

Expected Behavior
When I use Bank of Anthos following the instructions it does not configure SSL and I should navigate in http
Actual Behavior
When we navigate we are redirected to HTTPS and we get a not found because we don't have HTTPS enable.
Steps to Reproduce the Problem

Follow the instructions up to step 6-anthos-install
Access the application using the public ip http://
click in any of the links

Specifications

Version: 5b02673ce839e2ec0f35b2039f57d365e3fbcab9
Platform: Ubuntu 20.04

We discussed this during our session today. Let's just use "http"


Mrc 5335 Extract leaflet into composable + clean up choropleth

There were a lot of moving parts in the choropleth component and it was getting too complicated so I have tried to make things as neat as I could in this PR by extracting things into composables, any further suggestions are much appreciated! For testing of the useLeaflet we will continue to rely on e2e tests.
Main changes:

fadeColour is now in useColorScales i think this makes sense
backgroundLayer and corresponding global minZoom are extracted into utils since these dont change
added useDataSummary composable since this is completely independent from the functionality, i have not tested this since it is used in the e2e tests, feels weird to test something that is just used for testing idk
Extracted leaflet logic into useLeaflet composable, only functionally different thing ive added is smoothFactor: 0 otherwise should be the same

@EmmaLRussell

I like how useLeaflet doesn't rely on any other composables

I'm not seeing the virtue of having tooltipForFeature being, effectively, imported by Choropleth and then passed to useLeaflet -- this means that I have to read three files to follow the function to its definition, as opposed to if useLeaflet consumed useTooltip directly. It maybe just creates the illusion of useLeaflet not having dependencies, even though it actually does depend on useTooltip?

I'm not seeing the virtue of having tooltipForFeature being, effectively, imported by Choropleth and then passed to useLeaflet -- this means that I have to read three files to follow the function to its definition, as opposed to if useLeaflet consumed useTooltip directly. It maybe just creates the illusion of useLeaflet not having dependencies, even though it actually does depend on useTooltip?

the useLeaflet now relies on useDataSummary anyway haha but the reason why getTooltip exists in choropleth is because it uses the getFeatureId and getFeatureName from it, having said that, i could extract these to functions in a utils file but then i also wanted useLeaflet to be vaguely reminiscent of how you would use leaflet, like an extension of leaflet and how tooltips are displayed is not a leaflet responsibility, it should be the choropleth's responsibility.
so not super sure about what to do

@david-mears-2
I'm not seeing the virtue of having tooltipForFeature being, effectively, imported by Choropleth and then passed to useLeaflet -- this means that I have to read three files to follow the function to its definition, as opposed to if useLeaflet consumed useTooltip directly. It maybe just creates the illusion of useLeaflet not having dependencies, even though it actually does depend on useTooltip?

I think what I like about it is that the composables don't directly depend on eachother - they do have a dependency but they're not assuming what it is, and it's the Choropleth, as the client of all the composables, which coordinates them together, effectively doing dependency injection. Theoretically easier to test.. if we were testing them! :laughing:


Datapack Utilities 2.2.1c creates a grid of barrier blocks at y=255 in the end

In essence, if Datapack Utilities is installed and running, it will constantly generate a grid of barrier blocks at y=255 in the end, starting at 0,0, and going out 32 blocks in each direction. Aside from its invasive nature of generating unbreakable blocks, it will break the ender dragon when she is trying to perch by making her climb up to and past the build height due to the existence of the block there. Furthermore, the blocks will constantly respawn on their own even if they are being destroyed, which, with my testing, could possibly be a result of generating or activating chunks, however I was unable to confirm this.

What datapack(s) do you have installed (including ones made by me- include version if possible)?
Bug occurs when Datapack Utilities 2.2.1c is running on its own without any other datapacks.

Run /reload. Copy/paste or post a screenshot of what appears in chat (should say [Loaded  ]).
[Loaded Datapack Utils v2.2.1c]

What environment are you running the datapack? (Vanilla single player, Vanilla multiplayer, Forge, fabric, spigot, bukkit, paper, optifine, realms, etc). Please specify exact Minecraft version as well.
The bug occurs in 1.16.3 on Optifine Singleplayer as well as Vanilla Multiplayer.

What if happening vs. what you expect to happen?
Datapack Utilities will generate and constantly regenerate barrier blocks at the build limit inside the end at y=255, starting at 0,0, and working out every 32 blocks in each direction, seemingly indefinitely. As a result, the Ender Dragon will attempt to climb to this height whenever she is trying to perch on the end exit gateway, therefor making the fight impossible.

This is part of DU's world generation features. The barriers mark which chunks have received generation. However, I did not know about the issue with the dragon and that will definitely need to be resolved. Unfortunately, right now I think it's a tossup between barriers at y=255 or y=0...

I appreciate the knowledge that this bug is being looked into. I suppose, then, that it probably would not be possible to have the barrier block specifically at the 0,0 chunk be generated at y=0 and then have the surrounding ones be at y=255, then.

Disabled generation in the end in 2.2.1d


Unable to set timeout for idle connections

I am new to go language and want to set an HTTP timeout for requests and connections that have been idle for 10 seconds. I have added a code like so:
func handleServerConn(customer models.User, chans <-chan ssh.NewChannel) {

    for newChan := range chans {

        log.Debugf("New Channel: %v", newChan.ChannelType())

        if newChan.ChannelType() != "session" {
            log.Errorw("channel type rejected", log.Fields{"user": customer.LogFields(), "type": newChan.ChannelType()})
            newChan.Reject(ssh.UnknownChannelType, "unknown channel type")
            continue
        }

        ch, reqs, err := newChan.Accept()
        if err != nil {
            log.Infow("error accepting channel", log.Fields{"user": customer.LogFields(), "error": err})
            continue
        }

        /*
         * Send keepalive pings every now and then so that ELB wont disconnect the client
         */
        go func() {
            var errPing error
            var pingret bool
            for ch != nil && errPing != io.EOF {
                time.Sleep(60 * time.Second)
                pingret, errPing = ch.SendRequest("ping", false, nil)
                log.Debugw("sending ping", log.Fields{"error": err, "return": pingret})
            }
        }()

        serv := &http.Server{
            IdleTimeout: 10 * time.Second,
        }
        serv.ListenAndServe()
        log.Infow("Timeout reached for 10 seconds", log.Fields{"Read timeout response": resp, "Write timeout": resp})
--rest of the code
}

But the server goes on without terminating the connection. Is the placement for the timeout function correct? What else can be done here?

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.
For questions please refer to https://github.com/golang/go/wiki/Questions


Update requirements.txt

Temporary fix for Jinja2

LGTM
We couldve also just updated to the latest Quart but this works just fine to


[FLINK-8117] [runtime] [streaming] Eliminate modulo operation from round robin partitioners

What is the purpose of the change
This PR changes RebalancePartitioner, RescalePartitioner, and RoundRobinChannelSelector to not use % for wrapping around when reaching the number of channels, but use an if-statement instead.
Brief change log

Make the above change in RebalancePartitioner, RescalePartitioner, and RoundRobinChannelSelector
Additionally, I added final to the field that is holding the reused return array in RebalancePartitioner, RoundRobinChannelSelector, ShufflePartitioner, ForwardPartitioner, GlobalPartitioner, CustomPartitionerWrapper.

Verifying this change
This change is already covered by existing tests, such as RebalancePartitionerTest, RescalePartitionerTest.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): yes
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

Very good change, and nice implementation!
+1 to merge this.

Merging this...


add reorg metrics

fixes part of #3817

@Rjected updated! It seems test (partition 3/4) didn't run, looks weird.


Wrong License Displayed

Brief description of your issue
Microsoft's VSCode repository has MIT license. But it is only for the source code. The installer file is under a different license. But when installing Visual Studio Code using Winget, it shows MIT license, even though the installer has different license.
Steps to reproduce
Open Windows Terminal
Type winget install vscode
Press Enter key
Expected behavior
Under license, it shows Microsoft's license for VIsual Studio Code installer.
Actual behavior
It shows MIT license

Environment
Windows Package Manager v1.0.11694
Windows:  Windows.Desktop v10.0.22000.184
Package: Microsoft.DesktopAppInstaller v1.15.11694.0

If you know the actual license you can correct it yourself over on the winget-pkgs repo or you can tell someone and it will get fixed. What is the correct license and I will change it if you don't want to unless someone else gets to it first

@lszomoru has been submitting manifests for Visual Studio Code.


How to create an example of request object?

Do we support example values for request objects?
		public async Task<ActionResult<IEnumerable<DummyDto>>> GetConfigurations(
			[FromQuery] DummyQueryDto query) =>
			Ok(await configurationRepository.GetQueryable()
				.ProjectTo<DummyDto>(m_mapper.ConfigurationProvider)
				.ApplyFilter(query.ToFilter())
				.ApplySorting(query.ToSorter())
				.ApplyPagination(query.Offset, query.Count).ToListAsync());
	}

How do I supply example value for this DummyQueryDto object into Swagger UI?
I use OpenApi, SwaggerUi3, AddOpenApiDocument - a really basic setup.
Is there a support for Swashbuckle.AspNetCore.Filters ?

Is there a support for Swashbuckle.AspNetCore.Filters ?

See https://github.com/RicoSuter/NSwag/wiki/Document-Processors-and-Operation-Processors

Should we use the same approach as stated in this article and implement it by ourselves or we can use the stuff which is already available such as Swashbuckle.AspNetCore.Filters ?
Can we actually use the same semantics as [SwaggerRequestExample(typeof(DeliveryOptionsSearchModel), typeof(DeliveryOptionsSearchModelExample))] without implementing our own IOperationProcessor?
Currently, I do not see any examples of that kind of stuff. Could you please elaborate?

There is nothing outofthebox yet
See also https://blog.rsuter.com/nswag-tutorial-implement-a-custom-operation-processor-to-define-redoc-code-samples/

Will a PR on this be appreciated?

I think it would make sense to add an attribute to specify this out of the box. However what example object/schema would we use? I think we should only support OpenAPI schemas directly (not ReDoc custom formats).

Any update on this?

@RicoSuter is it still a thing?
Would the actual impl be any different than just an attribute which would be an OperationProcessor and work with the current operation via C# code?

I've created a custom IOperationProcessor that can create example responses based on attributes on the controller actions e.g.
[ProducesResponseType(StatusCodes.Status200OK, typeof(Country[]))
[ProducesResponseTypeExample(typeof(CountryExampleGenerator))]
public Task<IActionResult> GetCountries([FromQuery]Country query) {
  return await Execute(query);
}

The first attribute is the ASP.NET Core-provided attribute, which gives the overall shape of the response; the second response is a custom attribute, where the CountryExampleGenerator mentioned exposes an IEnumerable method that returns sample data:
public abstract class ExampleResponseGenerator
{
    private readonly List<object> _examples = new List<object>();

    protected void Register(params object[] exampleData)
    {
        _examples.AddRange(exampleData);
    }

    public IEnumerable Generate()
    {
        foreach (var example in _examples)
        {
            yield return example;
        }
    }
}

public class CountriesExampleProvider : ExampleResponseGenerator
{
    public CountriesExampleProvider()
    {
        Register(new Country
        {
            Name = "Australia",
            CurrencySymbol = 36,
            CurrencyText = "AUD",
            DiallingCode = "61",
            Id = 13,
            IsoCode2 = "AU"
        }, new Country
        {
            Name = "New Zealand",
            CurrencySymbol = 36,
            CurrencyText = "NZD",
            DiallingCode = "64",
            Id = 149,
            IsoCode2 = "NZ"
        }
        );
    }
}

The custom IOperationProcessor then resolves those attributes when generating the OpenApi document (and generates a standard set of responses for e.g. 400 Bad Request, 404 Not Found. However, at the moment, I don't know exactly where to place the final rendered result. On the older Swagger spec, it was on the example property of the response, but with OpenApi, it's in the schema definition (https://swagger.io/docs/specification/2-0/adding-examples/). Any suggestions as to how I can close that particular circle? I'm happy to post the code for my processor if I can get that working.

 I now have a beautiful example of an expected response in my OpenApi spec.

@davidkeaveny can you share the code for your processor?

Actually, these days I’m just using the NSwag.Examples package, which takes a similar approach to mine. Go with that, if you’re still looking for a solution.