Add support for npy/npz files to the command line tools.

Now that we can pass files into the command line tools the next step is to take a file that contains metadata as well. numpy's ndarray serialization format is lingua franca for ML frameworks and supporting it would make interoping between the standard IREE tooling and practically all ML frameworks that themselves interop with numpy easy.
Today:
--function_input=2x2xf32=[[0 1][2 3]]
<EMAIL_ADDRESS>{output on stdout in text form}

Proposed additional flags (sketch):
<EMAIL_ADDRESS>  # no metadata needed, read from npy
<EMAIL_ADDRESS>  # expands to multiple inputs
<EMAIL_ADDRESS> # write one output to a file
<EMAIL_ADDRESS> # write all remaining outputs to a file

Could also sniff the file path and change behavior on the existing flags; the format doesn't require the file extensions but I think it's reasonable if we do (practically everyone I've seen uses the standard ones, cause why not?):
<EMAIL_ADDRESS>  # no metadata needed, read from npy
<EMAIL_ADDRESS>  # expands to multiple inputs
<EMAIL_ADDRESS> # write one output to a file
<EMAIL_ADDRESS> # write all remaining outputs to a file

Could also be an excuse to rename those flags.
Had a prototype patch from ages ago that should be revisited:
// Copyright 2021 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//===----------------------------------------------------------------------===//
// numpy-compatible(ish) IO
//===----------------------------------------------------------------------===//
//
// Provides C variants of the numpy.load/numpy.save routines:
// https://numpy.org/doc/stable/reference/routines.io.html
// which uses the given file format:
// https://numpy.org/doc/stable/reference/generated/numpy.lib.format.html
//
// Pickled objects are not supported (similar to using `allow_pickle=False`) and
// not all dtypes are supported.
//
// .npy and uncompressed .npz files can be mapped into host memory with
// IREE_RUNTIME_IO_NUMPY_LOAD_OPTION_MAP_FILE if the HAL device allocator
// supports using such memory. On devices with discrete memory the contents will
// be loaded into host memory and copied to the device.
//
// This current implementation is very basic; in the future it'd be nice to
// support an iree_io_stream_t to allow for externalizing the file access.
//
// NOTE: this implementation is optimized for code size and is not intended to
// be used in performance-sensitive situations. If you are wanting to run
// through thousands of arrays and GB of data then you're going to want
// something more sophisticated (delay loading with async IO, etc).
//
// TODO(benvanik): conditionally enable compression when zlib is present. For
// now to reduce dependencies we don't support loading compressed npz files or
// compression on save.

#ifndef IREE_RUNTIME_IO_NUMPY_H_
#define IREE_RUNTIME_IO_NUMPY_H_

#include <stdint.h>

#include "iree/base/api.h"
#include "iree/hal/api.h"
#include "iree/vm/api.h"

#ifdef __cplusplus
extern "C" {
#endif  // __cplusplus

//===----------------------------------------------------------------------===//
// .npy (multiple values concatenated)
//===----------------------------------------------------------------------===//

// Options controlling npy load behavior.
enum iree_runtime_io_numpy_npy_load_options_bits_t {
  IREE_RUNTIME_IO_NUMPY_NPY_LOAD_OPTION_DEFAULT = 0u,

  // Tries to map the file into memory and use the contents directly from the
  // file system. Only available if the HAL device supports accessing mapped
  // data.
  // Like providing `mmap_mode` to `numpy.load`.
  // May be ignored if the implementation does not support mapping.
  IREE_RUNTIME_IO_NUMPY_NPY_LOAD_OPTION_MAP_FILE = 1u << 0,
};
typedef uint32_t iree_runtime_io_numpy_npy_load_options_t;

// Options controlling npy save behavior.
enum iree_runtime_io_numpy_npy_save_options_bits_t {
  IREE_RUNTIME_IO_NUMPY_NPY_SAVE_OPTION_DEFAULT = 0u,

  // Appends to the file if it exists and creates it otherwise.
  // When not specified files will be overwritten.
  IREE_RUNTIME_IO_NUMPY_SAVE_OPTION_APPEND_FILE = 1u << 0,
};
typedef uint32_t iree_runtime_io_numpy_npy_save_options_t;

// Loads a single value from an .npy at |file_path| into a buffer view.
// On success |out_buffer_view| will have buffer view matching the parameters
// in the npy file allocated from the given |device_allocator|.
//
// If IREE_RUNTIME_IO_NUMPY_LOAD_OPTION_MAP_FILE is set and the
// |device_allocator| supports mapping then the file will be mapped into the
// host process. Otherwise the file will be loaded into a new allocation.
//
// Fails if the contents of the npy cannot be mapped to IREE HAL types or is
// not an ndarray.
//
// See `numpy.load`:
// https://numpy.org/doc/stable/reference/generated/numpy.load.html
IREE_API_EXPORT iree_status_t iree_runtime_io_numpy_load_ndarray_from_npy_file(
    iree_runtime_io_numpy_npy_load_options_t options, const char* file_path,
    iree_hal_allocator_t* device_allocator,
    iree_hal_buffer_view_t** out_buffer_view);

// Saves |buffer_view| to a .npy file at |file_path|.
//
// See `numpy.save`:
// https://numpy.org/doc/stable/reference/generated/numpy.save.html
IREE_API_EXPORT iree_status_t iree_runtime_io_numpy_save_ndarray_to_npy_file(
    iree_runtime_io_numpy_npy_save_options_t options,
    iree_hal_buffer_view_t* buffer_view, const char* file_path);

//===----------------------------------------------------------------------===//
// .npz (multiple values, possibly compressed)
//===----------------------------------------------------------------------===//

// Options controlling npz load behavior.
enum iree_runtime_io_numpy_npz_load_options_bits_t {
  IREE_RUNTIME_IO_NUMPY_NPZ_LOAD_OPTION_DEFAULT = 0u,

  // Tries to map the file into memory and use the contents directly from the
  // file system. Only available if the HAL device supports accessing mapped
  // data and the file is uncompressed.
  // Like providing `mmap_mode` to `numpy.load`.
  // May be ignored if the implementation does not support mapping.
  IREE_RUNTIME_IO_NUMPY_NPZ_LOAD_OPTION_MAP_FILE = 1u << 0,
};
typedef uint32_t iree_runtime_io_numpy_npz_load_options_t;

// Options controlling npz save behavior.
enum iree_runtime_io_numpy_npz_save_options_bits_t {
  IREE_RUNTIME_IO_NUMPY_NPZ_SAVE_OPTION_DEFAULT = 0u,

  // Compresses the npz contents while saving.
  // Switches from the default uncompressed behavior matching `numpy.savez` to
  // the compressed behavior of `numpy.savez_compressed`.
  IREE_RUNTIME_IO_NUMPY_NPZ_SAVE_OPTION_COMPRESS = 1u << 0,
};
typedef uint32_t iree_runtime_io_numpy_npz_save_options_t;

// Loads the contents of the .npz file at |file_path| into a new |out_list|.
// |device_allocator| will be used to allocate any buffers required.
// Fails if the contents of the file cannot be mapped to IREE HAL or VM types.
//
// See `numpy.load`:
// https://numpy.org/doc/stable/reference/generated/numpy.load.html
IREE_API_EXPORT iree_status_t iree_runtime_io_numpy_load_ndarrays_from_npz_file(
    iree_runtime_io_numpy_npz_load_options_t options, const char* file_path,
    iree_hal_allocator_t* device_allocator, iree_vm_list_t** out_list);

// Saves the conents of the given |list| to a .npz file at |file_path|.
// Fails if any element of the list cannot be mapped to numpy types.
//
// See `numpy.savez` and `numpy.savez_compressed`:
// https://numpy.org/doc/stable/reference/generated/numpy.savez.html
// https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html
IREE_API_EXPORT iree_status_t iree_runtime_io_numpy_save_ndarrays_to_npz_file(
    iree_runtime_io_numpy_npz_save_options_t options, iree_vm_list_t* list,
    const char* file_path);

#ifdef __cplusplus
}  // extern "C"
#endif  // __cplusplus

#endif  // IREE_RUNTIME_IO_NUMPY_H_

#9461 adds npy load/save and flag support for loading. vm_util needs to be rewritten to get output support.
npz is probably not worth doing - npy can do multiple arrays and doesn't have the ordering ambiguity of the npz files (where entries are keyed by name). It also doesn't need zlib.

Marking as done because we can load/store npy files - leaving #9462 for actually writing out the npy files with tensor outputs.


Add Research Papers that contains DVC tool

Hello, I added research papers about DVC.

ah looks like @dberenbaum and I were reviewing at the same time :)

I don't actually think this type of content is a good fit for this repo, but atm can't think of a better existing place for it so fine with merging it here for now.

Because @casperdcl isn't sure this is the best place for this content, @mertbozkir let's instead move to the repo that will have the community content (blog posts), taking into account feedback here on what should be removed.

OK, Jeny. I'm closing this pull request.


Transaction Hashes

There's currently no agreed upon way to hash our SignedTransactions (i.e. no method or trait implemented). This is useful for logging as transactions need to be tracked across services.

[ ] look at how CometBFT implements transaction hashing
[ ] add the functionality to the SignedTransaction type

we should just serialize (ie. proto encode_to_vec) then sha256 hash of that

We should not only have an agreed upon way to report the tx hashes of SignedTransaction, but also of the bundle within (i.e. the transactions hash contained). The reason is that when composer is submitting transactions to sequencer, SignedTransaction is not stable since it will change with the included nonce.

We should not only have an agreed upon way to report the tx hashes of SignedTransaction, but also of the bundle within (i.e. the transactions hash contained). The reason is that when composer is submitting transactions to sequencer, SignedTransaction is not stable since it will change with the included nonce.

The hash of the rollup's transaction (or whatever else they submit, e.g. a block or a bundle) is determined by the rollup developers. Rather than enforcing a specific hash on the rollup tx (in this case an ethers::Transaction, we should just require it to be hashable (is that a word?).

@itamarreif this issue is pertaining to sequencer tx hashes, which are sha256 due to cometbft


Case-sensitivity Issue on OSX

Hello,
Appreciate this ticket #685 has been closed but I'm experiencing similar behaviour on OSX.
Specifically:

rez-env Maya maya results in the highest version Maya package added twice (i.e. in the resolved packages list and by checking the $PATH variable)

Fortran:~ jasonfairley$ rez-env Maya maya

You are now in a rez-configured environment.

resolved by<EMAIL_ADDRESS>on Wed Jul 22 18:22:15 2020, using Rez v2.62.0

requested packages:
Maya
maya
~platform==osx  (implicit)
~arch==x86_64   (implicit)
~os==osx-10.14  (implicit)

resolved packages:
Maya-2020     /Users/jasonfairley/rez/packages/develop/Maya/2020/platform-osx        (local)
maya-2020     /Users/jasonfairley/rez/packages/develop/maya/2020/platform-osx        (local)
platform-osx  /Users/jasonfairley/Google Drive/Software/packages/basic/platform/osx
> bash-3.2$

rez-env maya foo (where foo requires Maya) results in an environment with maya-2020 and Maya-2020 (as above)

Fortran:~ jasonfairley$ rez-env maya helloWorldCmd

You are now in a rez-configured environment.

resolved by<EMAIL_ADDRESS>on Wed Jul 22 21:13:42 2020, using Rez v2.62.0

requested packages:
maya
helloWorldCmd
~platform==osx  (implicit)
~arch==x86_64   (implicit)
~os==osx-10.14  (implicit)

resolved packages:
Maya-2020            /Users/jasonfairley/rez/packages/develop/Maya/2020/platform-osx                                  (local)
arch-x86_64          /Users/jasonfairley/Google Drive/Software/packages/basic/arch/x86_64
helloWorldCmd-1.0.0  /Users/jasonfairley/rez/packages/develop/helloWorldCmd/1.0.0/platform-osx/arch-x86_64/Maya-2020  (local)
maya-2020            /Users/jasonfairley/rez/packages/develop/maya/2020/platform-osx                                  (local)
platform-osx         /Users/jasonfairley/Google Drive/Software/packages/basic/platform/osx
> bash-3.2$

rez-env maya-2019 foo (where foo requires Maya) results in an environment with maya-2019 and Maya-2020 (where 2019 and 2020 are package versions of Maya) ... as well as foo of course.

Fortran:~ jasonfairley$ rez-env maya-2019 helloWorldCmd

You are now in a rez-configured environment.

resolved by<EMAIL_ADDRESS>on Wed Jul 22 21:15:13 2020, using Rez v2.62.0

requested packages:
maya-2019
helloWorldCmd
~platform==osx  (implicit)
~arch==x86_64   (implicit)
~os==osx-10.14  (implicit)

resolved packages:
Maya-2020            /Users/jasonfairley/rez/packages/develop/Maya/2020/platform-osx                                  (local)
arch-x86_64          /Users/jasonfairley/Google Drive/Software/packages/basic/arch/x86_64
helloWorldCmd-1.0.0  /Users/jasonfairley/rez/packages/develop/helloWorldCmd/1.0.0/platform-osx/arch-x86_64/Maya-2020  (local)
maya-2019            /Users/jasonfairley/rez/packages/develop/maya/2019/platform-osx                                  (local)
platform-osx         /Users/jasonfairley/Google Drive/Software/packages/basic/platform/osx
> bash-3.2$

I wanted to check what the expected behaviour is in the case? Should I expect Rez to be case sensitive?
Secondly, since there had been similar issues on Windows, just curious if anyone else is using Rez on OSX and experiencing what I've described above.
Many thanks for any and all pointers :-)
Jason.

PS. I should add that I've used "Maya" in all package.py files (upper case) and on disk.

UPDATE
After shuffling around some paths in the packages_path list in my rez_config.py file (which I point to using the environment variable REZ_CONFIG_FILE) this behaviour only happens when my Maya packages repo lives in a folder location which gets searched before the 'released' area:
packages_path = [
    '${REZ_BASE}/packages/basic',     # core rez packages, deployed
    '${REZ_BASE}/packages/foo',   # adding Maya here leads to the described behaviour
    '${REZ_BASE}/packages/release',   # globally released packages, deployed
]

If I drop that folder path entry foo and add the package repo for Maya to ${REZ_BASE}/packages/release everything behaves as intended and it will only find a package called 'Maya' and not 'maya'.


undefined: "github.com/mikespook/golib/signal".Loop

kentchentekiiMac-23868:gearman kentchen$ go run s1.go
command-line-arguments
./s1.go:73: undefined: "github.com/mikespook/golib/signal".Loop

I redesigned API of the package golib/signal recently that method Loop has been changed into Wait.
It will be OK after you upgrade the codes.


EnumRequest - Table 382 - CV Ledger Entry Buffer - Field - Document Type

EnumRequest - Table 382 - CV Ledger Entry Buffer - Field - Document Type

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Policy enforcement causes terraform to fail if it fixes issues too quickly

If a resource is created with terraform, and it violates policy, sometimes it is fixed too quickly and terraform returns an error message that's difficult to troubleshoot
* google_storage_bucket_iam_member.allauthenticatedusers: 1 error(s) occurred:

* google_storage_bucket_iam_member.allauthenticatedusers: Error applying IAM policy for Storage Bucket "forseti-enforcer-target-m6n9": Error setting IAM policy for Storage Bucket "forseti-enforcer-target-m6n9": googleapi: Error 412: Precondition Failed, conditionNotMet

408e480 sort-of fixes the issue by allowing users to set an optional delay to policy evaluation and enforcement


v-slot navigate function - return optional Promise

What problem does this feature solve?
I am trying to create a button within a router-link that shows a loading indicator when clicked (but still allows opening in a new tab, which should not show a loading indicator).
It would be nice if navigate($event) in the v-slot API could return a promise when it performs a .push/replace() so that I could stop the loading animation when the route fails to load, or succeeds, just like router.push/replace(to) does.
Right now the only way to do this in my app is to implement a programmatic push or replace call, which does not allow opening in a new tab.
Thank you so much!
What does the proposed API look like?
navigate(e) returns a Promise (the result of .push or .replace) if either of those functions are called within navigate(e). If neither is called, return nothing.

It's a bit more complex than just returning the result of push/replace as we also need to support environments with No Promise. https://github.com/vuejs/vue-router/blob/dev/src/components/link.js#L71-L77


python38Packages.databricks-cli: 0.14.3 -> 0.15.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python38Packages.databricks-cli is: "A command line interface for Databricks"
meta.homepage for python38Packages.databricks-cli is: "https://github.com/databricks/databricks-cli"
meta.changelog for python38Packages.databricks-cli is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

/nix/store/j34w6ihfkgprn4fbrlfrjqgphk37lqpj-python3.8-databricks-cli-0.15.0/bin/databricks passed the binary check.

/nix/store/j34w6ihfkgprn4fbrlfrjqgphk37lqpj-python3.8-databricks-cli-0.15.0/bin/dbfs passed the binary check.

/nix/store/j34w6ihfkgprn4fbrlfrjqgphk37lqpj-python3.8-databricks-cli-0.15.0/bin/.databricks-wrapped passed the binary check.

/nix/store/j34w6ihfkgprn4fbrlfrjqgphk37lqpj-python3.8-databricks-cli-0.15.0/bin/.dbfs-wrapped passed the binary check.

4 of 4 passed binary check by having a zero exit code.

4 of 4 passed binary check by having the new version present in output.

found 0.15.0 with grep in /nix/store/j34w6ihfkgprn4fbrlfrjqgphk37lqpj-python3.8-databricks-cli-0.15.0

found 0.15.0 in filename of file in /nix/store/j34w6ihfkgprn4fbrlfrjqgphk37lqpj-python3.8-databricks-cli-0.15.0

directory tree listing: https://gist.github.com/34958814ab8f928bb35e4d6b7eeb29ca

du listing: https://gist.github.com/486a6474ea915e10b20e5f8aea550290

Rebuild report (if merged into master) (click to expand)

20 total rebuild path(s)

5 package rebuild(s)

5 x86_64-linux rebuild(s)
5 i686-linux rebuild(s)
5 x86_64-darwin rebuild(s)
5 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
mlflow-server
python38Packages.databricks-cli
python38Packages.mlflow
python39Packages.databricks-cli
python39Packages.mlflow

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/j34w6ihfkgprn4fbrlfrjqgphk37lqpj-python3.8-databricks-cli-0.15.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python38Packages.databricks-cli https://github.com/r-ryantm/nixpkgs/archive/8d6e9c2730d1128810e5058ebdb41ec635a50e3b.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/j34w6ihfkgprn4fbrlfrjqgphk37lqpj-python3.8-databricks-cli-0.15.0
ls -la /nix/store/j34w6ihfkgprn4fbrlfrjqgphk37lqpj-python3.8-databricks-cli-0.15.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages marked as broken and skipped:

mlflow-server
python38Packages.mlflow
python39Packages.mlflow

2 packages built:

python38Packages.databricks-cli
python39Packages.databricks-cli

Maintainer pings
cc @tbenst for testing.

Result of nixpkgs-review pr 133755 at 8d6e9c27 run on aarch64-linux 1

3 packages failed to build:

mlflow-server
python38Packages.mlflow
python39Packages.mlflow

2 packages built successfully:

python38Packages.databricks-cli
python39Packages.databricks-cli

1 suggestion:

warning: python-include-tests
Consider adding a checkPhase for tests, or if not feasible, pythonImportsCheck.
Near pkgs/development/python-modules/databricks-cli/default.nix:37:0:
   |
37 |     description = "A command line interface for Databricks";
   | ^

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.


reproducing the NTOM in the paper

Hi, thanks for your great work.
In the paper,  table1 there is a performance of NTOM in Long-tailed CIFAR10
However, I have no idea of the implementation details of NTOM using auxiliary data of TINY 300K Image
Could you tell me some implementation details of NTOM using the TINY300K images?
Thanks in advance.

Here is the official implementation of NTOM. Hope useful.
https://github.com/jfc43/informative-outlier-mining/blob/master/train_ntom.py#L144

Hi, thanks for your great work.
In the paper, table1 there is a performance of NTOM in Long-tailed CIFAR10
However, I have no idea of the implementation details of NTOM using auxiliary data of TINY 300K Image
Could you tell me some implementation details of NTOM using the TINY300K images?
Thanks in advance.

Hi,
Have you reproduced the NTOM in the paper? I can't reproduce the results. Thank you very much!


SONARR: delQueue

Currently when requesting to delete from the que I get the response: {'message': 'MethodNotAllowed'}


IgLoginRequestedError when adding post from website

Form
Put an [x] if you meet the condition, else leave [ ].

[ x] I've searched the Issues
[ x] I've read the basic concepts
[ x] I'm using the latest version

Question
Hello, I have problem with logging in to my account when adding post from website. It shows me IgLoginRequiredError. Thought it was process.env problem so now I have parameters as string.Here is my code
Code
const IgApiClient = require('../node_modules/instagram-private-api/dist/core/client');
  const client = new IgApiClient();
            client.state.generateDevice('x');
            client.account.login('x', 'x;');
            client.publish.photo({
                file: createdRecipe.profile,
                caption: req.body.instagram_description
            })
            client.account.editProfile({external_url: `https://${req.headers.host}/recipes/${createdRecipe.link}`})
            done(createdRecipe);

Error and Output
 IgLoginRequiredError: POST /rupload_igphoto/1590605010747_0_7735792673 - 403 Forbidden; login_required

The login method is a promise. You need to wait his response (you can try with async/await)


Fix some more samples and add them to build

There is a warning fixed by ibriano on pr #720
Signed-off-by: Bruno Dilly<EMAIL_ADDRESS>

It's building fine on my machine, on semaphore and tipbot.
We're just adding as dependencies of these samples node types that are built as modules by default. Maybe you're buliding console as module or something like that?
I'll see this keyboard dependency.
Thanks

@ceolin
Ok, it's passing on all these machines because they don't have enough dependencies to build http-server, I've just checked the log.
I'll send a v2 fixing keyboard dependency but if something else goes wrong, please paste a log so I can fix all issues.
Thanks

pr #733


Grow NetSendBuffer_t dynamically when necessary

unless we're on OpenACC, then we throw an error. This uses https://github.com/BlueBrain/CoreNeuron/pull/395#

please retest

please retest

retest this please


How to send emails in a Django application using SMTP server

Be sure to visit our Resources Page for tools, resources, and example articles to go over.
NOTE: We tend to stray away or tend not to publish reviews/comparisons of commercial product offerings.
How to send emails in a python app using SMTP server
Introduction paragraph(s):
Google gmail provides an excellent way where emails can be sent using the SMTP server.
Simple Mail Transfer Protocol (SMTP) is an internet standard communication protocol for electronic mail transmission. There are a lot of free SMTP servers which can be used to send mails, however this article focuses on Google SMTP Server.
Google SMTP server provides for a means where mails can be sent without having to manage your own outgoing mail server.
In an application mails such as a password reset link, invitation or any other information are essential and these can be sent using the google SMTP server.
While building an application, for example a user login in page, the password reset email is usually sent to enable the user to retrieve or set a new password. The major advantage of google SMTP server is that it provides a means for the password of the developer to stay hidden using the app passwords methods. This tutorial would primarily focus on how the google SMTP server works and how to add SMTP to your Django app to send a password reset mail  to users.
Key takeaways:

Understand what SMTP server means
Have knowledge of other free SMPT servers
Learn how google SMTP server and set app passwords.
Send a password reset mail to a user in a Django application

References:
Please list links to any published content/research that you intend to use to support/guide this article.
none
Templates to use as guides

How To Guide Template
Software Review Template
Tutorial Template

Thank you for the topic suggestion @Anitaachu, but it seems there might be quite a bit of overlap with the fundamentals of the article being worked on in #1640. We'd prefer that you select a different topic area to explore.


Invalid imports in the sample code cause project not to build

Describe the bug
In the sample code the value of Static is neither declared or imported.
Also, the default config.ts file has no exported member ENVALID.
Finally, one of the imports is pulling in the entire envalid folder from node-modules which is generating additional build errors.
To Reproduce
Steps to reproduce the behavior:

Start with a working nestjs project
add "nestjs-envalid": "1.2.0", to package.json
npm install
navigate to project/src and touch env-config.ts
paste in the sample code from https://github.com/cobraz/nestjs-envalid#quickstart

Expected behavior
Typescript project builds successfully
Screenshots

Desktop (please complete the following information):

OS: Ubuntu 20
Browser: FireFox
Version: 90

Hello @seantcanavan 👋
Thank for opening this issue! I've seemed to have jumped the fence a bit there, seem to have forgot to export a few things and add a step to the quickstart.
First of all, envalid is a peerDependency. We'll have to add that to the steps. I might be mistaken, but that should take care of your build issues? The ENVALID export should come from nestjs-envalid. I'll look at all this stuff very soon! If you want to open a pull request on this, it would be much appreciated! 👍
Thanks again!

Sorry I'm very new to TS/Angular/NextJS so I don't think I'm qualified to open a bug report


⚠ Jenkins Plugins Artifactory has degraded performance

In 2e7a6fa, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1191 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 3e275db after 20 minutes.


Update the Jackson Databind version to 2.9.10

There is a vulnerability discovered in the Jackson Databind library pre-2.9.10.
This PR updates the version to fix the vulnerability in dependent repositories.

@armiol PTAL.


Export 'Events' from snarkyjs

Would be amazing if we could import Events from snarkyjs directly.
This is the code/export i'd like to access from the main package:
https://github.com/o1-labs/snarkyjs/blob/main/src/provable/transaction-leaves.ts#L17
import { Events } from 'snarkyjs'

Right now i have to do the following, which breaks a couple things down the line when i publish my library packages:
import { Events } from '../../../node_modules/snarkyjs/dist/node/provable/transaction-leaves.js';

^ This triggers error ts(2742) in my libraries when i try to use anything that relies on this specific import.

I'd merge a PR which adds 1 line of code to export Events as AccountUpdate.Events analogous to sequence events here: https://github.com/o1-labs/snarkyjs/blob/d6b7995aa81070c1dcc78eb751ae70578ddeb97a/src/lib/account_update.ts#L650


termguicolors not working

Describe the bug
The termguicolors does not work and use the ctermfg/ctermbg colors instead of the guifg/guibg colors.
To Reproduce
1). Run :set termguicolors
Expected behavior
When termguicolors is set I expect that vim will use the guifg/guibg colors instead of the ctermfg/ctermbg colors.
Environment:

Vim version: 8.2 +termguicolors
OS: Manjaro
Terminal: Alacritty (supports true color)

Duplicate of https://github.com/vim/vim/issues/1740.

I tested it on a newer version and it worked.


SFPF, SFPI: Error by import of big SIFbA forms #2353

The renumbering of the attributes "id" and "href" is reworked

It seems I've placed accidentally both corrections #2353 and #2348 in one pull request.

in your fork, make one branch per change

Yeah, I've got it now, sorry. Is it possible to decline the pull request? Then I could create 2 separate..

On 11 Feb 2019, at 21:10, Lars Hvam<EMAIL_ADDRESS>wrote:
in your fork, make one branch per change
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub https://github.com/larshp/abapGit/pull/2379#issuecomment-462432660, or mute the thread https://github.com/notifications/unsubscribe-auth/AjxxAGoEjjePShhxFokZ0DYx_e3-_z-wks5vMbH5gaJpZM4a0tE8.

this one is closed, feel free to open 2 separate


Add support for all gnuplot's terminal types in Jupyter that has a corresponding MIME type

For all gnuplot's terminal types with corresponding MIME type, the switching in a Jupyter notebook can happen automatically.
This way, even gif and webp animations will be supported.

Implemented by PR #10


Update Type System part of spec wrt "types redux"

Not done function polymorphism, and have not added array (or indeed prelude) to standard library. @ss2165 were you in the process of doing that?

Thanks Alec, many good points there. Feel free to take over, or merge when ready, @ss2165 and @aborgna-q too.


Service Principal ACR Role assignment

Role for the AKS service principal on the ACR should be AcrPull instead of Reader to make sure it's the least priviliged possibility.
az role assignment create --assignee <appId> --scope <acrId> --role acrpull

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d4af71a4-648e-580f-7e86-94ce05196de5
Version Independent ID: 134b8ba0-b085-076a-d2e1-16d0ddff5586
Content: Kubernetes on Azure tutorial - Deploy a cluster
Content Source: articles/aks/tutorial-kubernetes-deploy-cluster.md
Service: container-service
GitHub Login: @iainfoulds
Microsoft Alias: iainfou

@krottiers - Thanks again for pointing this out. The documentation is now updated to assign the AcrPull role to the service principal for AKS.
#please-close


Replace special read_line function with standard fid.readline()

This PR replaces the special read_line() function with the standard fid.readline() so that there's less cognitive load when going through code.
Note this should be merged after #29.

Merging with all tests passing locally.


Make ChannelPool apply scopes where appropriate

This is a PR which would be part of addressing
https://github.com/GoogleCloudPlatform/google-cloud-dotnet/issues/294
if it turns out the fix really should be in our libraries.
This is a breaking change - it requires support in the code generator as well, and we'd need to be careful not to release the GAX package separately, as anyone upgrading to that without upgrading the other packages would be broken. That could be addressed by adding a parameterless constructor here, but that feels a bit of a hack.

LGTM

(LGTM received out-of-band.)


Splunk Logging Driver performance improvements

Add additional options:

splunk-gzip - gzip compress all requests to Splunk HEC
(enabled by default)
splunk-gzip-level - change compression level.

Messages are sent in batches by 1000, with frequency of 5 seconds.
Maximum buffer is 10,000 events. If HEC will not be available, Splunk
Logging Driver will keep retrying while it can hold messages in buffer.
Signed-off-by: Denis Gladkikh<EMAIL_ADDRESS>Manual test I have used:
docker pull alpine
time docker run --log-driver=splunk --log-opt splunk-url=$HEC_URL --log-opt splunk-token=$HEC_TOKEN alpine sh -c "seq 1000000" > /dev/null

Before this change:
real	2m52.159s
user	0m0.080s
sys	0m0.110s

After this change (it uses gzip by default, which slows down it a little):
real	0m16.312s
user	0m0.070s
sys	0m0.070s

Without gzip compression (--log-opt splunk-gzip-compression=false):
real	0m14.122s
user	0m0.060s
sys	0m0.080s

Also added unit tests
go test -v --cover -coverprofile=coverage.out  github.com/docker/docker/daemon/logger/splunk
=== RUN   TestValidateLogOpt
--- PASS: TestValidateLogOpt (0.00s)
=== RUN   TestNewMissedConfig
--- PASS: TestNewMissedConfig (0.00s)
=== RUN   TestNewMissedUrl
--- PASS: TestNewMissedUrl (0.00s)
=== RUN   TestNewMissedToken
--- PASS: TestNewMissedToken (0.00s)
=== RUN   TestDefault
--- PASS: TestDefault (0.00s)
=== RUN   TestInlineFormatWithNonDefaultOptions
--- PASS: TestInlineFormatWithNonDefaultOptions (0.00s)
=== RUN   TestJsonFormat
--- PASS: TestJsonFormat (0.00s)
=== RUN   TestRawFormat
--- PASS: TestRawFormat (0.00s)
=== RUN   TestRawFormatWithLabels
--- PASS: TestRawFormatWithLabels (0.00s)
=== RUN   TestRawFormatWithoutTag
--- PASS: TestRawFormatWithoutTag (0.00s)
=== RUN   TestBatching
--- PASS: TestBatching (0.12s)
=== RUN   TestFrequency
--- PASS: TestFrequency (0.15s)
=== RUN   TestOneMessagePerRequest
--- PASS: TestOneMessagePerRequest (0.01s)
=== RUN   TestVerify
--- PASS: TestVerify (0.00s)
=== RUN   TestSkipVerify
ERRO[0000] splunk: failed to send event - 500 Internal Server Error -
ERRO[0000] splunk: failed to send event - 500 Internal Server Error -
ERRO[0000] splunk: failed to send event - 500 Internal Server Error -
--- PASS: TestSkipVerify (0.13s)
=== RUN   TestBufferMaximum
ERRO[0000] splunk: failed to send event - 500 Internal Server Error -
ERRO[0000] splunk: failed to send event - 500 Internal Server Error -
ERRO[0000] splunk: failed to send event - 500 Internal Server Error -
ERRO[0000] splunk: failed to send event - 500 Internal Server Error -
ERRO[0000] splunk: failed to send event - 500 Internal Server Error -
ERRO[0000] Failed to send a message '{"event":{"line":"0","source":"stdout","tag":"containeriid"},"time":"1472364013.758671","host":"outcoldimac"}'
ERRO[0000] Failed to send a message '{"event":{"line":"1","source":"stdout","tag":"containeriid"},"time":"1472364013.758676","host":"outcoldimac"}'
--- PASS: TestBufferMaximum (0.01s)
PASS
coverage: 83.5% of statements
ok  	github.com/docker/docker/daemon/logger/splunk	0.432s

Is batch size / interval configurable. These options are configurable in our logging libray.

Also is -compression needed in the param name? Seems pretty verbose. Why not splunk-gzip
and splunk-gzip-level?

@glennblock sure, I can remove compression from both of these options.
As for the configuration - yes these parameters are configurable, but not within the driver options, but with the environment variables on docker daemon (see updated splunk.md file in this change). Maybe it is not the best idea, but I just decided that defaults are good enought and not to overwhelming list of logger options for Splunk driver.

Makes sense. As long as there is some way to configure them that is documented, that sounds good to me.
Thanks for updating the paam names.

@glennblock thank you for quick review! Have changed flags as you suggested. Let me know if you will have any other ideas.

LGTM

@clifg changes I made:
diff --git a/daemon/logger/splunk/splunk.go b/daemon/logger/splunk/splunk.go
index f4f0652..43871b0 100644
--- a/daemon/logger/splunk/splunk.go
+++ b/daemon/logger/splunk/splunk.go
@@ -365,9 +365,10 @@ func (l *splunkLoggerRaw) Log(msg *logger.Message) error {
 func (l* splunkLogger) queueMessageAsync(message *splunkMessage) error {
        l.lock.RLock()
        defer l.lock.RUnlock()
-       if l.closedCond == nil {
-               l.stream <- message
+       if l.closedCond != nil {
+               return fmt.Errorf("%s: driver is closed", driverName)
        }
+       l.stream <- message
        return nil
 }

@@ -408,7 +409,11 @@ func (l *splunkLogger) postMessages(messages []*splunkMessage, lastChance bool)
                }
                if err := l.tryPostMessages(messages[i: upperBound]); err != nil {
                        logrus.Error(err)
-                       if messagesLen - i >= l.bufferMaximum {
+                       if messagesLen - i >= l.bufferMaximum || lastChance {
+                               // If this is last chance - print them all to the daemon log
+                               if lastChance {
+                                       upperBound = messagesLen
+                               }
                                // Not all sent, but buffer has got to its maximum, let's log all messages
                                // we could not send and return buffer minus one batch size
                                for j := i; j < upperBound; j++ {
diff --git a/daemon/logger/splunk/splunk_test.go b/daemon/logger/splunk/splunk_test.go
index 5ab2d65..c18e2c8 100644

In general, I'm a bit confused about the calculation you're doing. Usually it's a combination of "total message size in bytes", "total number of messages" and "time since last send" that determine when we send a batch, but I wasn't sure that I saw all three of them in there.

@itay I do not use "total message size in bytes", because I know that messages are limited by the 16K, so I only limiting by total number of messages (maxBufferSize) and time since last send (5 seconds default)

Talked to @outcoldman and based on the comments, LGTM.

Closing, because of https://github.com/docker/docker/pull/26207


Create a

Starting github-check jobs.

Build succeeded.

linters https://sftests.com/logs/cab98f3f5ddf47c7b6e06560be59b14a/ : SUCCESS in 5s


Numerous errors with webpack webpack.config.js with latest

Errors on fsevents, nodefs-handler, and more with latest version of the yo generator and csproj based configuration.

Are these errors or warnings?
Can you post them also?

Appears it may be powershell only related?
PS C:> webpack webpack.config.js
Hash: bb89ba4d65a32e77eeba70a7b2d713a46ab50ed2
Version: webpack 1.14.0
Child
Hash: bb89ba4d65a32e77eeba
Version: webpack 1.14.0
Time: 15332ms
Asset     Size  Chunks             Chunk Names
main.js   1.4 MB       0  [emitted]  main
main-client.js  20.7 kB       1  [emitted]  main-client
main.js.map  1.71 MB       0  [emitted]  main
main-client.js.map  24.3 kB       1  [emitted]  main-client
[0] ./webpack.config.js 2.74 kB {0} [built]
+ 375 hidden modules
WARNING in ./~/chokidar/lib/fsevents-handler.js
Module not found: Error: Cannot resolve module 'fsevents' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_mod

es\chokidar\lib
@ ./~/chokidar/lib/fsevents-handler.js 7:17-36
ERROR in (webpack)/lib/node/NodeMainTemplateAsync.runtime.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\w

pack\lib\node
@ (webpack)/lib/node/NodeMainTemplateAsync.runtime.js 9:2-15 24:2-15
ERROR in ./~/uglify-js/tools/node.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\u

ify-js\tools
@ ./~/uglify-js/tools/node.js 8:9-22
ERROR in (webpack)/lib/node/NodeOutputFileSystem.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\w

pack\lib\node
@ (webpack)/lib/node/NodeOutputFileSystem.js 5:9-22
ERROR in (webpack)/lib/node/OldNodeWatchFileSystem.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\w

pack\lib\node
@ (webpack)/lib/node/OldNodeWatchFileSystem.js 5:9-22
ERROR in ./~/mkdirp/index.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\m

irp
@ ./~/mkdirp/index.js 2:9-22
ERROR in ./~/graceful-fs/graceful-fs.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\g

ceful-fs
@ ./~/graceful-fs/graceful-fs.js 1:9-22
ERROR in ./~/graceful-fs/fs.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\g

ceful-fs
@ ./~/graceful-fs/fs.js 3:9-22
ERROR in ./~/chokidar/index.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\c

kidar
@ ./~/chokidar/index.js 3:9-22
ERROR in ./~/chokidar/lib/nodefs-handler.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\c

kidar\lib
@ ./~/chokidar/lib/nodefs-handler.js 3:9-22
ERROR in ./~/chokidar/lib/fsevents-handler.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\c

kidar\lib
@ ./~/chokidar/lib/fsevents-handler.js 3:9-22
Child
Hash: 70a7b2d713a46ab50ed2
Version: webpack 1.14.0
Time: 15193ms
Asset     Size  Chunks             Chunk Names
main.js  5.29 MB       0  [emitted]  main
main-server.js  50.2 kB       1  [emitted]  main-server
[0] ./webpack.config.js 2.74 kB {0} [built]
+ 521 hidden modules
WARNING in (webpack)/lib/webpack.js
Critical dependencies:
62:11-37 the request of a dependency is an expression
 @ (webpack)/lib/webpack.js 62:11-37

WARNING in ./~/chokidar/lib/fsevents-handler.js
Module not found: Error: Cannot resolve module 'fsevents' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_mod

es\chokidar\lib
@ ./~/chokidar/lib/fsevents-handler.js 7:17-36
PS C:\Repos\CAdlearning\Blogs\CADLearning.Blogs> npm install -g webpack
C:\Users\jhancock.CADLEARNING\AppData\Roaming\npm\webpack -> C:\Users\jhancock.CADLEARNING\AppData\Roaming\npm\node_mo
les\webpack\bin\webpack.js
C:\Users\jhancock.CADLEARNING\AppData\Roaming\npm
--<EMAIL_ADDRESS> +--<EMAIL_ADDRESS> | --<EMAIL_ADDRESS>+--<EMAIL_ADDRESS>| --<EMAIL_ADDRESS> +--<EMAIL_ADDRESS> --<EMAIL_ADDRESS>`--<EMAIL_ADDRESS>npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules\webpack\node_modules\chokidar\node_modul
\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"
y"} (current: {"os":"win32","arch":"x64"})
PS C:\Repos\CAdlearning\Blogs\CADLearning.Blogs> webpack webpack.config.js
Hash: bb89ba4d65a32e77eeba70a7b2d713a46ab50ed2
Version: webpack 1.14.0
Child
Hash: bb89ba4d65a32e77eeba
Version: webpack 1.14.0
Time: 15947ms
Asset     Size  Chunks             Chunk Names
main.js   1.4 MB       0  [emitted]  main
main-client.js  20.7 kB       1  [emitted]  main-client
main.js.map  1.71 MB       0  [emitted]  main
main-client.js.map  24.3 kB       1  [emitted]  main-client
[0] ./webpack.config.js 2.74 kB {0} [built]
+ 375 hidden modules
WARNING in ./~/chokidar/lib/fsevents-handler.js
Module not found: Error: Cannot resolve module 'fsevents' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_mod

es\chokidar\lib
@ ./~/chokidar/lib/fsevents-handler.js 7:17-36
ERROR in (webpack)/lib/node/NodeMainTemplateAsync.runtime.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\w

pack\lib\node
@ (webpack)/lib/node/NodeMainTemplateAsync.runtime.js 9:2-15 24:2-15
ERROR in ./~/uglify-js/tools/node.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\u

ify-js\tools
@ ./~/uglify-js/tools/node.js 8:9-22
ERROR in (webpack)/lib/node/NodeOutputFileSystem.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\w

pack\lib\node
@ (webpack)/lib/node/NodeOutputFileSystem.js 5:9-22
ERROR in (webpack)/lib/node/OldNodeWatchFileSystem.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\w

pack\lib\node
@ (webpack)/lib/node/OldNodeWatchFileSystem.js 5:9-22
ERROR in ./~/mkdirp/index.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\m

irp
@ ./~/mkdirp/index.js 2:9-22
ERROR in ./~/graceful-fs/graceful-fs.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\g

ceful-fs
@ ./~/graceful-fs/graceful-fs.js 1:9-22
ERROR in ./~/graceful-fs/fs.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\g

ceful-fs
@ ./~/graceful-fs/fs.js 3:9-22
ERROR in ./~/chokidar/index.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\c

kidar
@ ./~/chokidar/index.js 3:9-22
ERROR in ./~/chokidar/lib/nodefs-handler.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\c

kidar\lib
@ ./~/chokidar/lib/nodefs-handler.js 3:9-22
ERROR in ./~/chokidar/lib/fsevents-handler.js
Module not found: Error: Cannot resolve module 'fs' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_modules\c

kidar\lib
@ ./~/chokidar/lib/fsevents-handler.js 3:9-22
Child
Hash: 70a7b2d713a46ab50ed2
Version: webpack 1.14.0
Time: 15793ms
Asset     Size  Chunks             Chunk Names
main.js  5.29 MB       0  [emitted]  main
main-server.js  50.2 kB       1  [emitted]  main-server
[0] ./webpack.config.js 2.74 kB {0} [built]
+ 521 hidden modules
WARNING in (webpack)/lib/webpack.js
Critical dependencies:
62:11-37 the request of a dependency is an expression
 @ (webpack)/lib/webpack.js 62:11-37

WARNING in ./~/chokidar/lib/fsevents-handler.js
Module not found: Error: Cannot resolve module 'fsevents' in C:\Repos\CAdlearning\Blogs\CADLearning.Blogs\node_mod

es\chokidar\lib
@ ./~/chokidar/lib/fsevents-handler.js 7:17-36

Actually:
webpack webpack.config.vendor.js
Will fail on cmd as well.

I'm having this issue as well,
In my webpack.config files I added the following to my configs.
Bear in mind the example repos have both a server config and a  clinet config in the same webpack files, so you'll need to add this to both.
node: {
  fs: "empty"
},

I'm still a bit baffled why this even happened though, it was working okay until... yesterday? :/

I tried this and then promptly got endless other errors (it appears to fix that one) even with the default template and no changes at all.

The command isn't webpack webpack.config.js.
It's webpack --config webpack.config.js :)
(Or just run webpack with no args, which defaults to a config file called webpack.config.js).

Head bang Sorry


markdown text entry

Is there a reason why the deployed site has:

but this repo does not?

// CanStyle returns true if this user can style text in comments/stories.
func (u *User) CanStyle() bool {
	return u.Points > 30
}


Activation with powershell scripts opens file instead of executing it

Checks

[X] I have checked that this issue has not already been reported.

[X] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example
On Windnows add an activation section to pixi.toml
[activation]
scripts = ["setup.ps1"]

Create a setup.ps1 with any content.
$env:MY_ENV_VAR="Hello World"

Now activate the shell:
PS> pixi shell

This opens the setup.ps1 file in NotePad, instead of executing it. Shell activation hangs until the file is closed.
Issue description
On Windows activations using powershell scripts doesn't work. They open the script file in Notepad instead of executing it.

Using *.bat files works fine as workaround.
Expected behavior
The powershell script should run.

For now I'll update the documentation to explain this behavior. Would that be enough for you?

Yes . Of course :) Documentation just hinted that it had or was supposed to work.

Oops haha, that was a mistake!

Fixed in : https://github.com/prefix-dev/pixi/pull/563


Add keybind "ctrl+[" to go back to normal mode

Escape key is far from home position. I would like to use "ctrl+[" keybinding like real vim.

What I do is remap Capslock to Escape, so that could be a workaround if you're able to do that. But also you can add the shortcut you request yourself via VSCode's keyboard shortcut system. Just search for "simpleVim". I'm going to close this because I don't think the shortcut should be added by default and it's trivial to add for whomever wants it.


fix incorrect behavior of SplitInfo == operator, speed up comparisons for splits with identical gains

Short Summary
This PR fixes a bug in the == operator for SplitInfo and LightSplitInfo in src/treelearner/split_info.hpp. In the event that two candidate splits have different gains, == currently incorrectly says that those splits are identical.
I'm not certain, but I believe this is a bug that could lead to incorrect behavior in voting parallel learning, at the stage where learners select the top k features to consider for future splits. I'm not sure if this fix would fix #4414, but if I'm right then it might at least help.
Longer Description
The SplitInfo class is used to store information about a candidate tree split, including the gain of the split and the index of the feature it refers to. There is also a lighter-weight alternative to this class, called LightSplitInfo.
LightGBM's training process involves calculating the gain for different splits and comparing those splits to each other to determine the best one. To do this, it defines > and == operators for comparing two SplitInfo or LightSplitInfo instances.
Running cppcheck (#4539), I found what looks like a bug in the definition of the == operator for SplitInfo and LightSplitInfo!
src/treelearner/split_info.hpp:177:25: warning: Opposite inner 'return' condition leads to a dead code block. [oppositeInnerCondition]
      return local_gain == other_gain;
src/treelearner/split_info.hpp:271:25: warning: Opposite inner 'return' condition leads to a dead code block. [oppositeInnerCondition]
      return local_gain == other_gain;

This code (introduced in https://github.com/microsoft/LightGBM/commit/4f77bd2860913a9a210c72a367fc88b4a9c13b3d), has what looks like a mistake:
if (local_gain != other_gain) {
      return local_gain == other_gain;
}

That code is saying "if the gain of this split and the split being compared to it are different, this_split == that_split should be true".
How this fix affects LightGBM
I tried commenting out the definitions of those == operator in src/treelearner/split_info.hpp to see where it's used. It's used much less frequently than the < operator, but I found at least one place...in the stage where voting parallel training evaluates splits to pick the top k features to consider!
https://github.com/microsoft/LightGBM/blob/a926d4fe939118b6377455220e44ca9ded5b964e/src/treelearner/voting_parallel_tree_learner.cpp#L173
https://github.com/microsoft/LightGBM/blob/a926d4fe939118b6377455220e44ca9ded5b964e/include/LightGBM/utils/array_args.h#L160
https://github.com/microsoft/LightGBM/blob/a926d4fe939118b6377455220e44ca9ded5b964e/include/LightGBM/utils/array_args.h#L137
https://github.com/microsoft/LightGBM/blob/a926d4fe939118b6377455220e44ca9ded5b964e/include/LightGBM/utils/array_args.h#L118-L119
A reminder from the original LightGBM paper of how this affects training:

For local voting, the top-k attributes are selected from each machine according to its local data. Then, globally top-2k attributes are determined by a majority voting among these local candidates. Finally, the full-grained histograms of the globally top-2k attributes are collected from local machines in order to identify the best (most informative) attribute and its split point.

I believe now these if (local_gain != other_gain) statements can be moved upper (right before int local_feature = this->feature; line) for the speedup.

oooo yeah, good idea! I'll do that here

updated in https://github.com/microsoft/LightGBM/pull/4542/commits/b40452e75424732381da07adcf7377f28ef719f2
great suggestion @StrikerRUS , this is going to make the majority of split comparisons a little bit faster, very nice 😎

oops, forgot LightSplitInfo too. Updated in https://github.com/microsoft/LightGBM/pull/4542/commits/5a60b7b3b1ddf6e89fc50371109dea8f1f3586f9

But I think the original implementation is also correct.

ahhh ok yes @shiyu1994 you're right! I got confused by this
if (local_gain != other_gain) {
     return local_gain == other_gain;
}

I'll change the title of the PR before we merge, to make it clear that this is a cleanup / small optimization and that it isn't actually fixing anything "incorrect".


Consolidate concurrency limits

Summary
This PR consolidates the concurrency limits used throughout uv and exposes two limits, UV_CONCURRENT_DOWNLOADS and UV_CONCURRENT_BUILDS, as environment variables.
Currently, uv has a number of concurrent streams that it buffers using relatively arbitrary limits for backpressure. However, many of these limits are conflated. We run a relatively small number of tasks overall and should start most things as soon as possible. What we really want to limit are three separate operations:

File I/O. This is managed by tokio's blocking pool and we should not really have to worry about it.
Network I/O.
Python build processes.

Because the current limits span a broad range of tasks, it's possible that a limit meant for network I/O is occupied by tasks performing builds, reading from the file system, or even waiting on a OnceMap. We also don't limit build processes that end up being required to perform a download. While this may not pose a performance problem because our limits are relatively high, it does mean that the limits do not do what we want, making it tricky to expose them to users (https://github.com/astral-sh/uv/issues/1205, https://github.com/astral-sh/uv/issues/3311).
After this change, the limits on network I/O and build processes are centralized and managed by semaphores. All other tasks are unbuffered (note that these tasks are still bounded, so backpressure should not be a problem).

The terminology here is a little weird. From https://github.com/astral-sh/uv/issues/3311, what we want is to expose a way to limit the overall concurrency of downloads. However, for builds, we're really only limiting the number of parallel python processes we spawn. Technically those two are different, but I think sticking to one of concurrency vs. parallelism would be less confusing. Maybe these should be UV_PARALLEL_* instead?
Test Plan

There is one seemingly important limit left in https://github.com/astral-sh/uv/blob/main/crates/uv-resolver/src/resolver/mod.rs#L323, from https://github.com/astral-sh/uv/pull/1163. In practice this bound is quite large compared to our concurrent downloads and should not be a source of contention, it's mainly there to allow us to use a bounded channel and avoid starving the prefetcher.

Regarding parallel vs concurrent: I don't think that's important as a user-facing detail. "CONCURRENT" is fine.

I agree, I was just wondering which one would be more intuitive, regardless of which is technically correct.

👍 I think people have a broader understanding of "concurrency limits" than parallelism.

I've gotten comfortable just using the word "concurrency" because of this: https://doc.rust-lang.org/book/ch16-00-concurrency.html

I have a minor preference for our own setting over a RAYON variable if it's easy.

LGTM! There's maybe one other thing we care about here: the number of concurrent installs (which, today, is controllable by RAYON_NUM_THREADS). Once we've downloaded and built all the wheels, we then have to install them into the virtual environment, and we do this with Rayon. I think it's fine not to add a setting for that, but we could.

We almost might want a way to limit the size of tokio's blocking threadpool for fs operations (https://github.com/astral-sh/uv/issues/3311). Ideally those options could be combined. Installs are fs bound so maybe we should just use tokio's blocking pool instead of rayon for that as well?
Either way I think I'll keep this PR limited to network and build limits.


Updates android dependencies for latest BK

Will be needed to compile TP with latest BK snapshot.
You will need to upgrade, via the sdk manager, the build tools to 23.0.3 and the local maven repo in the extras (still via the sdk manager). It has been renamed but you can figure it out with the latest commit comment and this screenshot :

Coverage remained the same at 78.125% when pulling a3ce11a70c21eec984d83b01355817f7d8fbc931 on sni/update-android-deps into 1db26fbbb286d7c2b2b8a4bfc5b647140a04c316 on master.


[DO NOT MERGE] Make mailing list prompt always show

This PR is just so I can get a custom build. Don't merge it please 😊

@luabud If you mark it as a draft PR we won't be asked for reviews, and you can still get the build.

@luabud what's the status of this PR?

I got what I needed from it at the time and forgot to close it :)


Fix  3 AV1 decode issues with system memory

Following 3 issues are fixed in this PR:

error buffer status, which cause buffer be unmmaped twice.
For all key frame stream,  buffer be overwrited when it is still be used by APP side.
When non display frame be repeated, frame should wait for it be repeated, then release.

@dmitryermilov @onabiull @dvrogozh Could you please review this PR? Thanks.


nmap-formatter: init at 0.2.1

Motivation for this change
Tool that allows you to convert nmap output
https://github.com/vdjagilev/nmap-formatter
Related to #81418
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 136623 run on x86_64-linux 1

1 package built:

nmap-formatter

Result of nixpkgs-review pr 136623 run on x86_64-darwin 1

1 package built:

nmap-formatter

Thank you!


Fixing docs imports

I don't think these changes are necessary, and this confirmed my suspicion that the PPPR token is only available for the CI on some PRs.


Automatically bump version in Homebrew

Overview
Flagd was recently added to Homebrew. The currently deployed version is 3.1. This could quickly become out of sync if the update process isn't automated.
Proposal
Add an additional step in the release process to automatically open a PR in the homebrew-core repo.
Resources

https://github.com/Homebrew/homebrew-core/blob/master/CONTRIBUTING.md#to-submit-a-version-upgrade-for-the-foo-formula
https://github.com/mislav/bump-homebrew-formula-action

@beeme1mr It seems that homebrew already provides the functionality for us: https://github.com/Homebrew/homebrew-core/pull/121354#issuecomment-1401234491

No, that comment indicates that there is a way to get the latest project version. A livecheck does not provide a method to automatically update.

livecheck and bump process are two separate things.

https://github.com/mislav/bump-homebrew-formula-action

That would handle the bump process as part of the release process.

@beeme1mr Since @chenrui333 already has the PR out, we can assign this him


Default implementation of deduplicate is not null-safe

Describe the bug
@belasobral93 discovered that deduplicate doesn't work for Spark when any of the order_by columns are null.
Root cause
The root cause is that Spark defaults to NULLS FIRST for the null_sort_order for the ORDER BY clause.
Why dbt_utils rather than spark_utils?

Explanation

But dbt_utils doesn't officially support Spark, so why is this being reported here (instead of spark_utils)?

dbt_utils is providing five implementations for deduplicate:

default
redshift
postgres
snowflake
bigquery

Since dbt_utils only tests against the four databases listed above and each of those has an override, it means there is no testing for the default implementation which the other dbt adapters inherit (like Spark).

Steps to reproduce

SQL example
The current implementation essentially acts like the following example:
with relation as (

    select 1 as partition_by, 'a' as order_by
    union all
    select 1 as partition_by, 'a' as order_by
    union all
    select 2 as partition_by, 'a' as order_by
    union all
    select 2 as partition_by, 'b' as order_by
    union all
    select 2 as partition_by, NULL as order_by

),

row_numbered as (

    select
        _inner.*,
        row_number() over (
            partition by partition_by
            order by order_by
        ) as rn
    from relation as _inner

)

select
    distinct relation.*
from relation
natural join row_numbered
where row_numbered.rn = 1
;

Expected results

partition_by
order_by

1
a

2
a

Actual results

partition_by
order_by

1
a

Screenshots and log output
Not provided.
System information
Not provided.
Which database are you using dbt with?

[ ] postgres
[ ] redshift
[ ] bigquery
[ ] snowflake
[x] other: spark

The output of dbt --version:
Not provided.
Additional context
To make the fix, it might be as simple as:

adding nulls last after order_by (when an order by clause exists, of course)

To confirm the bug and establish sufficient test cases, we could:

comment out all adapter-specific overrides for deduplicate
re-run the test suite
add test cases as needed
re-run the test suite
make fixes to the implementation(s)
re-run the test suite

Are you interested in contributing the fix?
I will contribute the fix.

To make the fix, it might be as simple as:

adding nulls last after order_by (when an order by clause exists, of course)

Is nulls [last|first] a well-known part of the SQL spec? I've never heard of it, a bit of quick Googling implies that it's specific-ish
A couple of things that are only in my head/true by convention but not documented:

Deviations from the norm (default__ implementations) should be handled in an override. (Strongly held)
"The norm" is vanilla-flavoured things that can reasonably be expected of pretty much everything. 1 (Pretty strongly held)
I tend to assume that Postgres is vanilla-flavoured, as the longest-lived and only-OSS DB in the original core four. (Not at all strongly held, but don't have a better idea)
1 Lookin' at you, BigQuery types

The approach that would be truest to points 1 and 3 would be:

The default implementation changes to Postgres'
Anyone who can't do that builds their own

Which is right if we want a globally coherent and understandable set of principles. It's not super pragmatic though; it appears that 10 data platforms support our current implementation, and as far as I can see most of them would not support Postgres' because they don't have distinct on.
In my opinion, "Vanilla as default" is more important than "every default implementation is what would work best in Postgres". (In fact, this might be a case where the default implementation works on PGSQL, but there is a special optimisation that led to the custom version? I haven't checked).
So that's a lot of words to say that if MySQL, Redshift, Materialize, SingleStore don't understand nulls last then we are inconveniencing some new adapters by throwing it in, and maybe we're better off just specifying nulls last in the Spark version.

Related (in that it's also a deduplicate issue): https://github.com/dbt-labs/dbt-utils/issues/713

anyone any workarounds for this


[ISSUE #4183] When the reportSlaveMaxOffset method fails, end the current loop early

Make sure set the target branch to develop
What is the purpose of the change
#4183
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Looks good to me

Coverage decreased (-0.2%) to 51.851% when pulling ed4595e4e02d3969a8e0840bd11265fdb27bcdc4 on BurningCN:patch-1 into 580b76d3ded98fa906dec56f3bfcf79c3202a263 on apache:develop.

Codecov Report

Merging #4184 (ed4595e) into develop (580b76d) will decrease coverage by 0.17%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             develop    #4184      +/-   ##
=============================================
- Coverage      48.04%   47.86%   -0.18%
+ Complexity      5020     5001      -19
=============================================
  Files            635      635
  Lines          42493    42495       +2
  Branches        5567     5567
=============================================
- Hits           20415    20341      -74
- Misses         19592    19659      +67
- Partials        2486     2495       +9

Impacted Files
Coverage Δ

...n/java/org/apache/rocketmq/store/ha/HAService.java
71.52% <0.00%> (-0.82%)
:arrow_down:

.../broker/subscription/SubscriptionGroupManager.java
64.70% <0.00%> (-16.48%)
:arrow_down:

...in/java/org/apache/rocketmq/test/util/MQAdmin.java
38.88% <0.00%> (-5.56%)
:arrow_down:

...ava/org/apache/rocketmq/filter/util/BitsArray.java
58.11% <0.00%> (-4.28%)
:arrow_down:

...ketmq/common/protocol/body/ConsumerConnection.java
95.83% <0.00%> (-4.17%)
:arrow_down:

...lient/impl/consumer/DefaultMQPushConsumerImpl.java
40.17% <0.00%> (-3.86%)
:arrow_down:

...ava/org/apache/rocketmq/test/util/VerifyUtils.java
46.26% <0.00%> (-2.99%)
:arrow_down:

...ache/rocketmq/store/AllocateMappedFileService.java
39.02% <0.00%> (-2.44%)
:arrow_down:

...a/org/apache/rocketmq/filter/util/BloomFilter.java
60.43% <0.00%> (-2.20%)
:arrow_down:

...e/rocketmq/remoting/netty/NettyRemotingServer.java
57.34% <0.00%> (-1.90%)
:arrow_down:

... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 580b76d...ed4595e. Read the comment docs.

LGTM～


Remove the parm7 and rst7 files from Main directory

It will be better to remove the parm7 and rst7 files so that users won't be confused while running the tutorial. These files were already included in the EXAMPLE folder, and the scripts pointed to the files in the EXAMPLE folder.

The parm7 and rst7 files have been deleted.


Add the newsletter template page, and associated styles.

Associated ticket: #34
The Newsletter WordPress page will use this template to embed the mailchimp signup form.
Page in action:

Also pinging @KillYourFM for visual feedback. It's not live yet, since this code needs to go up before the sign-up form can be displayed.

Good spot, after some thinking, I've adjusted so there's a gap between. Looks more like the search bar now, which also fits in with the thunderbird style guide.

Little fly by comment that I'm sure everyone will love, but I think it would be nice to have a consistent style with the way the subscription input field is styled in the main website.

On some more nitpicky notes:

The input field should have a placeholder
The input field should have the same border radius of  the button
I would give the form a max-width smaller than the post, so to make it centered aligned and not as wide as the whole post since it doesn't need it, and very long input fields "feel" strange.

Sounds good, I was aiming for that originally but wanted to keep it within blog style. I'll cook something more similar up. Thanks for the fly-by feedback!

Slightly more in-line with Thunderbird.net, but still very thunderblog.

I was going to put this on hold until newsletter situation was clarified, but I'll merge this and put the ticket on hold.


Is there a function to add contacts?

I looked all over the app and can't find an add friend button, this is so strange.

Work around - In firebase go to database and click "Try Firestore BETA", click "enable" the application should start working.


Update target-right-audience-for-your-ads-campaign.md

Added Additional Resources + links to other Ads Docs
Description of the change:
Reason for the change:
Link to original source:

Closes #

Hello @negarzipan,Thanks again for the PR! We appreciate your contribution and look forward to                                              continued collaboration. Thanks!                                             Team SendGrid DX


Multi Numeric Column RowTotol Sum

http://orbjs.net/
like orb.js multicolumn sum

@zhaoweijing2003  could you please explain your requirement with screenshots. Thanks.


https://github-roast.pages.dev/

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

Browser & version [e.g. chrome, safari, webworker]
Node version [e.g. 16.0.1]
Starknet.js version
Network [devnet, testnet]

Additional context
Add any other context about the problem here.
Oh, Starknet-io, the self-proclaimed "L2 rollup on Ethereum" – how original! Your bio reads more like a tech jargon generator has gone haywire. With 26 public repos, it’s clear you’re either ambitious or just excessively ambitious about underwhelming contributions.
Your star counts suggest a mild interest in the ecosystem, but let’s be real – for every high count, there’s a wave of open issues that scream, “We started something we can’t finish.” Especially that grim list of stargazers – some with single digits! Is “starknet-docs-antora-ui” a sincere attempt at documentation, or an elaborate way of promising pseudo-clarity?
And what’s with all those “Rust” and “TypeScript” projects? Is anyone still using Rust, or did you mistakenly think you stumbled into a niche that needed rediscovering? Your project names are a blend of vague and uninspired – "Tokei"? Please, it feels like you threw letters together and hoped the universe would accept it.
Also, that lovely zero following? Guess no one wanted to be a part of your “Stark” situation. Bravo on such a unique way to remain in solitary confinement in the coding world!

Hello is this opened for contribution @henry-deriv


framework(engine): remove 'JobMasterID' method from 'JobMaster' interface

What problem does this PR solve?

Issue Number: close #6615
What is changed and how it works?
'JobMasterID' and 'ID' return the same id to user and we don't need the JobMasterID to return JobManagerID actually,
so remove 'JobMasterID'  method to avoid confusion
Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
 `None`.

/area engine
/status ptal

/hold

@GMHDBJD Could you please help fix the unit test error in tiflow/engine/jobmaster/dm?  Just remove the meaningless ID in your cases.

Hold this pr untill https://github.com/pingcap/tiflow/pull/6620 being merged to master.
#6620 will remove the usage of JobMasterID method of dm.

/unhold

/run-dm-compatibility-test
/run-dm-integration-test
/run-kafka-integration-test
/run-integration-test

/merge

Codecov Report

Merging #6619 (4557b1e) into master (2ea67ba) will increase coverage by 0.0749%.
The diff coverage is 85.4054%.

:exclamation: Current head 4557b1e differs from pull request most recent head b79d6de. Consider uploading reports for the commit b79d6de to get more accurate results

Flag
Coverage Δ

cdc
65.9229% <67.9245%> (+0.0548%)
:arrow_up:

dm
52.0573% <ø> (+0.0156%)
:arrow_up:

engine
62.8643% <92.4242%> (+0.3282%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.
@@               Coverage Diff                @@
##             master      #6619        +/-   ##
================================================
+ Coverage   59.2138%   59.2888%   +0.0749%
================================================
  Files           780        780
  Lines         88405      88499        +94
================================================
+ Hits          52348      52470       +122
+ Misses        31415      31390        -25
+ Partials       4642       4639         -3


[chef 12] Use full class name for rbenv_rehash resource

fixes #98

@fnichol Any plans on merging this?

I'm also experiencing the issue that motivates this merge. It would be nice if this can be soon merged into upstream.
Why are the tests failing?

@aminin, wouldn't this require a version bump?

@washingtoneg, version bump? what do you mean? This cookbook version is 0.7.3 for the last two years.

@aminin, I'm just wondering if this would potentially be a breaking change that would be incompatible with previous versions of chef server. If so, that would warrant a version bump, in my opinion. Thoughts?

Installation from supermarket.chef.io is the only rational reason for using
SV in CookBook.
Unfortunately coockbook name rbenv on supermarket is reserved by another
CB (less convinient analog).
On github you may always use sha1 instead  of SV.
2015-02-27 21:42 GMT+03:00 washingtoneg<EMAIL_ADDRESS>
@aminin https://github.com/aminin, I'm just wondering if this would
potentially be a breaking change that would be incompatible with previous
versions of chef server. If so, that would warrant a version bump, in my
opinion. Thoughts?
—
Reply to this email directly or view it on GitHub
https://github.com/fnichol/chef-rbenv/pull/101#issuecomment-76448238.

--
С уважением
Антон Минин.

That makes complete sense and I agree with your standpoint.

Hi @aminin, thanks for this change, it totally fixes the issue for me. I noticed there are a few other changes here as well: removing Gemfile.lock from .gitignore, and it looks like RVM is added in as well. Those seem like bigger changes to me.
As I just want to get running in as simple and forward-compatible manner as possible, I tested with only your change to libraries/chef_provider_package_rbenvrubygems.rb and that worked great. For reference: bplunkert:fix_rbhash_undefined_method
So I opened up #108 hopefully that helps get the fix merged!

@bplunkert, the Gemfile.lock only makes travis CI build pass. It doesn't affect your production chef version (or another gem version).

I've applied a version of this to my fork

Fixed on master


Replace condor log parsing by condor history

Since I'm playing with the job tracking, I decided to get rid of that nasty local log parsing - which tries quite badly - to figure out what happened to jobs that vanished from the condor queue.
So I'm using the condor bindings and the history() method for the Schedd class, you can see the code here:
https://github.com/amaltaro/WMCore/commit/dc573e56be82f8a57289ec095144ca9207ca6db5
which I tested interactively and I also saw it succeeding once in the agent (vocms0231).
However, it failed the second time that that code was executed for one job constraint (and now it fails all the time for ~900 ids). The error reason is Timeout when waiting for remote host, which is raised ~20 seconds after querying the schedd.
Here is the tricky part, after googling for a minute, I found out the generator is indeed supposed to return right away from the history call, however it still didn't produce/yield any results. It does so when we iterate over it in the for loop, which is when the timeout happens.
In case it helps, I locally replaced those 2 lines of for (that was using the grouper function) by only a single line like (it's an iterator already anyways)
for jobAd in jobiter:
and then this line is throwing an exception. Example of traceback can be seen here [1]
@ticoann @ericvaandering do you see any problem with that code?
@bbockelm is 20sec a expected delay in a small node with basically no load? is it configurable? do you see any possible optimization to that code such that we can use condor history in production?
[1]
2016-07-22 23:15:00,880:140496379688704:INFO:PyCondorPlugin:PyCondor is going to track 1005 jobs
2016-07-22 23:15:00,912:140496379688704:INFO:PyCondorPlugin:Retrieved 19 classAds
2016-07-22 23:15:00,913:140496379688704:INFO:PyCondorPlugin:Found 986 jobs missing in condor. Checking the history.
2016-07-22 23:15:00,928:140496379688704:INFO:PyCondorPlugin:<htcondor.HistoryIterator object at 0x7fc7df5028c0>

2016-07-22 23:15:20,949:140496379688704:ERROR:BossAirAPI:Unhandled exception while tracking jobs for plugin PyCondorPlugin!
Timeout when waiting for remote host
2016-07-22 23:15:21,026:140496379688704:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMCore.BossAir.StatusPoller.StatusPoller instance at 0x7fc7df48a7a0> BossAirException
Message: Unhandled exception while tracking jobs for plugin PyCondorPlugin!
Timeout when waiting for remote host
    ModuleName : WMCore.BossAir.BossAirAPI
    MethodName : track
    ClassInstance : None
    FileName : /data/srv/wmagent/v1.0.17.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.patch2/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py
    ClassName : None
    LineNumber : 521
    ErrorNr : 0

Traceback:
  File "/data/srv/wmagent/v1.0.17.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.patch2/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py", line 508, in track
    localRunning, localChanges, localCompletes = pluginInst.track(jobs=jobsToTrack[plugin])

  File "/data/srv/wmagent/v1.0.17.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.patch2/lib/python2.7/site-packages/WMCore/BossAir/Plugins/PyCondorPlugin.py", line 591, in track
    missingJobInfo = self.getMissingClassAds(missingJobIds)

  File "/data/srv/wmagent/v1.0.17.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.patch2/lib/python2.7/site-packages/WMCore/BossAir/Plugins/PyCondorPlugin.py", line 1095, in getMissingClassAds
    for jobAd in jobiter:

  File "/data/srv/wmagent/v1.0.17.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.patch2/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 172, in __call__
    self.algorithm(parameters)
  File "/data/srv/wmagent/v1.0.17.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.patch2/lib/python2.7/site-packages/WMCore/BossAir/StatusPoller.py", line 66, in algorithm
    self.checkStatus()
  File "/data/srv/wmagent/v1.0.17.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.patch2/lib/python2.7/site-packages/WMCore/BossAir/StatusPoller.py", line 92, in checkStatus
    runningJobs = self.bossAir.track()
  File "/data/srv/wmagent/v1.0.17.patch2/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.patch2/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py", line 521, in track
    raise BossAirException(msg)

Hi Alan,
Fundamentally, tracking jobs via the history mechanism isn't a good idea: the history interface isn't meant to scale for these sort of queries.  The log-based mechanism is the recommended mechanism.
That said, the current implementation of the log-based interface looks fairly broken / not-scalable and also isn't the recommended way to do things.
Can we perhaps chat directly on how HTCondor recommends job status be tracked?  Iterating via tickets is likely not going to be efficient.
Brian

@amaltaro have you tried per_job_history_dir? Maybe Brian can comment on efficiency, but for sure CRAB become more stable after using per_job_history_dir.
Even so, CRAB tried to do this with condor_q, and also with parsing log file, but none of them allowed to scale enough and was CPU hungry and was making schedd crashes, huge cpu use, etc.
It should be straight forward to turn this on and make a parser, you can look at the links below. As improvement if you save this job_add together in the final log tarball, you could any time double check all classads and see what happened with specific job if needed, which I would find it very useful.
You can read more here:
https://hypernews.cern.ch/HyperNews/CMS/get/crabDevelopment/2252.html
https://github.com/dmwm/CRABServer/issues/4618

@amaltaro @bbockelm can we re-open this thread and hopefully come to conclusion ?
I found it via google (!) when looking for hints on how to properly parse condor logs.
I have no idea how things are going in WMA land, but while working on porting CRAB job status 'detection' to new HTCondor API I got very worried of the way they wrote CRAB code 3+ years ago trying to interpret condor log heuristically (and not doing a very good job at it). Does really HTCondor not provide the needed fucntionality ? And if not, why not ask them ? In the end all this log parsing is onlly because cannot/wantnot issue too many condor_q ! Performace/lload aside, I am worried about the possible fragility of this code with respect to future HTC evolutions
https://github.com/dmwm/CRABServer/blob/master/scripts/task_process/cache_status.py
Would it make sense to chat about this with Todd & C. ?

Stefano, I sort of got lost going through this issue, and the following two other interesting discussions:
https://github.com/dmwm/WMCore/pull/8469
https://github.com/dmwm/WMCore/issues/8488
but if I recall correctly, the final goal was to retrieve the final state of a job (before it left the condor queue). Such that we could properly map what was Completed, Removed, and maybe removed for being in Held state...
If that's the final goal, then maybe Kenyi @khurtado could discuss it with the HTCondor developers and see what's the recommended and most scalable way of tracking jobs that were in condor in the previous minutes (of course we need to cover a case where problems happen and the job tracking algorithm doesn't run for a day, for instance).
Kenyi is already in close contact with the HTCondor team, so it's likely going to be easier for him to talk to them :-)

thanks Alan. As I wrote I found this issue with a generic google search, it is possible that CRAB and WMA have different needs (and issues). Let me try to talk with @khurtado and see if there's common ground to cover. I am very cautious about changing things in CRAB which are not fully broken (yet), but maybe on job status tracking we do not want to wait for problems before acting.
I will try to come to some understanding.

the CRAB use case for job log parsing is described here https://github.com/dmwm/CRABServer/issues/5942#issuecomment-533189720
While that issue and discussion with HTC developers so far was on "how to read the job log efficiently as it grows very long over time" my question here is "how to make sure that we interprete correctly its content"

@amaltaro If the goal is to retrieve the final state of the job before the job leaving the queue, maybe using the option leave_in_queue would make more sense?
Example 1:

Submit all jobs with: leave_in_queue = True
Query the schedd for the Completed Status, as jobs will remain in the queue
Once completed status is retrieved, edit job attr to: LeaveJobInQueue = False
Wait for HTCondor to remove job from the queue (defined by JOB_IS_FINISHED_INTERVAL ? usually a few seconds)

can also use PER_JOB_HISTORY as Justas already suggested above https://github.com/dmwm/WMCore/issues/7003#issuecomment-240643839
works like this:

when job completes and leaves the queue, HTCondor dumps its classAd to a file
you can read it and parse it whenever you want, then delete it
That may be easier on the schedd than many condor_q

@belforte Thanks, I like that better.

@khurtado is this the only use case you have for parsing jog log ? If so, there's likely no overlap with my problem !

Though I may be interested in learning how WMA does job status tracking, maybe there's something that we are have been overlooking in CRAB.

@belforte:  @amaltaro can expand on it, but I think basically condor_q (well, the iterator based xquery in the python bindings) and the condor job logfile.

Job tracking is performed by the JobTracker component, which has a 1 minute polling cycle. Here is the high level code snippet tracking jobs in condor, which makes a comparison to what we have in the local BossAir table:
https://github.com/dmwm/WMCore/blob/master/src/python/WMComponent/JobTracker/JobTrackerPoller.py#L98
I have to admit I started refactoring the whole job tracking but never completed the task. Besides being far from optimal (and precise), it's a piece of code that has great potential to cause problems and/or hidden bugs along the way.
In addition to JobTracker, we also have JobStatusLite querying condor, every minute as well. There is another couple of components that also interact with condor, but I guess not as a general condor_xquery thing.
Stefano, about the code you pointed out, note that we mostly do not look into the condor.xxxx.err/out/log files, we only do so if a job gets hard-killed from the pool/batch system, which means we don't have a job Report.pkl, basically here:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py#L301

WOW Alan ! Do you meant that WMA goes through all jobs in condor queue in 1 min ? Or that it is so fast (seconds) that it manages to do it every minute ?
I share your concerns about refactoring those things, it is stuff that took O(1year) initially to get robust enough. Any new thing may need a similar time of being exposed to all possible issues before it is as reliable.
Anyhow, clearly atm the use of condor job log in CRAB is unique to it and I have no reason to bug you guys further on that.
Looking in the long term future, looks to me that we all try to do things which in the end should better be done by condor components.

It's the time the thread sleeps before starting a new cycle.
Having a second look at JobTracker, I think I was wrong :-) Times don't make sense, it's just a matter of very few seconds and the cycle is completed. It seems to be using only the BossAir table against the WMBS one.
The component to look at is likely JobStatusLite, which runs basically this code every cycle:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/BossAir/StatusPoller.py#L59
but again, it's extremely overcomplicated, creating wmbs job objects out of runJob objects and back and forth (and eventually fetching live data from condor) ...

Some of that complexity is by design since the system supports multiple submissions systems. Yes, in practice we only use HTCondor now, but that wasn't the case when this was designed/written.


[Rename] Take into account the .fsi file to deduce the scope for renaming

If a top-level symbol is defined without a scope keyword, then renaming it triggers a solution-wide rename. However such a symbol may actually be private or internal due to a declaration (or lack of declaration) in the corresponding .fsi file. In this case it would be nice to avoid the solution-wide rename.

Agreed. That would be a great improvement.


Remove IGNORE_MUTLIPART_ATTACHMENTS option

Remove multipart attachments option from test cases. The issue
making this option needed has been resolved.
See FB 45086 for resolution to issue.

+1

nb typo on "MUTLIPART"


Outline the roadmap for the pipeline

Base paper: https://www.frontiersin.org/articles/10.3389/fbinf.2021.805338/full
The source code is available upon request 🙃

Initiating roadmap:

[ ] @jloubser to share the link to source code
[ ] @Ankita-1211 to initiate outlining the modules
[ ] @Ankita-1211 to sketch-out the main workflow

This task will be taken over by @justicengom and @abhi18av - the goal is to create granular roadmap for the pipeline.
An investment of 20-30 minutes in every 1/2 days should be enough to complete the pipeline.
(Commented by Abhinav from my username)

I have initiated a project for this pipeline here https://github.com/orgs/mtb-bioinformatics/projects/4/views/1?layout=board


New Device Request: Wifi Electric Oil Radiator

Hello, thanks for your great work on this project!
Would it be possible for you to add support for the following heater? (Amazon 1500W 7 Fin Wifi Electric Radiator)
Currently it showing just as a standard switch. Cloud API and Home Assistant Diagnostics below.
https://www.amazon.co.uk/gp/product/B0BGHZJCMJ/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&th=1
Thanks in advance!
Tuya Cloud:
{
"result": {
"category": "qn",
"functions": [
{
"code": "switch",
"desc": "{}",
"name": "开关",
"type": "Boolean",
"values": "{}"
},
{
"code": "temp_set",
"desc": "{"unit":"℃","min":15,"max":50,"scale":0,"step":1}",
"name": "设置温度",
"type": "Integer",
"values": "{"unit":"℃","min":15,"max":50,"scale":0,"step":1}"
},
{
"code": "lock",
"desc": "{}",
"name": "童锁",
"type": "Boolean",
"values": "{}"
},
{
"code": "light",
"desc": "{}",
"name": "灯光",
"type": "Boolean",
"values": "{}"
},
{
"code": "countdown_set",
"desc": "{"range":["cancel","1h","2h","3h","4h","5h","6h"]}",
"name": "倒计时设置",
"type": "Enum",
"values": "{"range":["cancel","1h","2h","3h","4h","5h","6h"]}"
}
],
"status": [
{
"code": "switch",
"name": "开关",
"type": "Boolean",
"values": "{}"
},
{
"code": "temp_set",
"name": "目标温度C",
"type": "Integer",
"values": "{"unit":"℃","min":15,"max":50,"scale":0,"step":1}"
},
{
"code": "lock",
"name": "童锁",
"type": "Boolean",
"values": "{}"
},
{
"code": "light",
"name": "灯光",
"type": "Boolean",
"values": "{}"
},
{
"code": "countdown_set",
"name": "倒计时设置",
"type": "Enum",
"values": "{"range":["cancel","1h","2h","3h","4h","5h","6h"]}"
}
]
},
"success": true,
"t":<PHONE_NUMBER>690,
"tid": "6d885d74750811ed8cfe9edd9c9c26e5"
}
Home Assistant Diagnostics:
},
"data": {
"name": "Loft Heater",
"type": "simple_switch",
"device_id": "REDACTED",
"local_key": "REDACTED",
"host": "REDACTED",
"api_version": 3.3,
"status": {},
"cached_state": {
"1": true,
"2": 29,
"4": "0",
"7": false,
"10": false,
"19": "cancel",
"21": 0,
"updated_at":<PHONE_NUMBER>.4266436
},
"pending_state": {},
"home_assistant": {
"name": "Loft Heater",
"name_by_user": null,
"disabled": false,
"disabled_by": null,
"entities": [
{
"disabled": false,
"disabled_by": null,
"entity_category": null,
"device_class": null,
"original_device_class": "switch",
"icon": null,
"unit_of_measurement": null,
"state": {
"entity_id": "switch.loft_heater",
"state": "on",
"attributes": {
"device_class": "switch",
"friendly_name": "Loft Heater"
},
"last_changed": "2022-12-06T01:39:44.243215+00:00",
"last_updated": "2022-12-06T01:39:44.243215+00:00"
}
}
]
}
}
}

Since the cloud data is missing the dp_id mappings, I have had to guess.  And also some educated guesses at the remaining two unaccounted for dps "4" and "21" based on what is in the screenshots and other similar devices.
"1": true, = "code": "switch",
"2": 29, = "code": "temp_set",
"4": "0", = mode? ["0"=Comfort, "1"=Eco, "2"=Anti-frost]?
"7": false, = "code": "lock" ?,
"10": false, = "code": "light" ?,
"19": "cancel", = "code": "countdown_set",
"21": 0, = countdown_remain?

That’s great. Thanks for your quick response! I do I go about testing this to see if it works?

I reviewed the above mapping you suggested and confirm you have the mappings correct. Although "21":0 doesn't seem to have much use as it still shows 0 even when I have had a countdown set as below. So I don't think that its countdown_remain Maybe 21 is for something else but I am not sure what that could be, as I couldn't get it to show anything but 0. Could it simply be an 'online' status?
"cached_state": {
"1": true,
"2": 30,
"4": "1",
"7": false,
"10": false,
"19": "3h",
"21": 0,

Thanks for pointing me in the right direction! Just taken a look and you are again correct, its 'Fault Alarm'.

I did just try to trip that alarm, by tipping over the heater. It seems the tip over feature completely cut power to the device including wifi access nothing was reported to the cloud other then a 'Device Restart'

Thanks @make-all I have tested the above and confirm all is working perfectly! Really appreciate your work


Cannot get 1 specific channel to reappear

First, thanks for this amazing program. It's working great with my source and all features work. I'm just having an issue with 1 channel coming back after trying to make changes. My system is running Windows Server 2016.
I tried to change the channel numbers to organize them to my liking, but I found my source doesn't work if I do that. I tested this by starting from scratch. I stopped my xTeVe service, deleted the config folder, and restarted. They all came back after re-configuring Plex, as expected. I then went in and just changed the channel number of the channel in question, and reconfigured the DVR in Plex. Said channel was missing. So root cause confirmed, I can't change channel numbers, no big deal. I again, shutdown xTeVe, removed the config folder and turner from Plex and reconfigured.
Now no matter what I do, I cannot get that channel to reappear in the guide. It comes up in the channel list while configuring the tuner, and I confirmed the channel works in VLC. It simply won't match up to the guide again.
I've tried resetting everything I could find, but no luck. I assume there's some kind of configuration cached somewhere that keeps pulling the old channel data.

I should also note that if I map the guide manually in Plex, instead of linking it to my sources EPG, it works. However, I confirmed it's not an issue with my sources EPG or stream because I tested this in my seedboxes Plex server, and it works fine there.

Seems to be working now after waiting a bit and trying again. Weird. Thanks anyway! And again, great app you have here!


Compatibility for SwiftData with CloudKit

I'm considering integrating this package into my project to monitor sync status while using SwiftData with CloudKit. Does it work for SwiftData too? Or is there a similar solution for monitoring the sync status when using SwiftData with CloudKit?
Any insights into compatibility or limitations would be greatly appreciated. Thank you!

This library should be usable in SwiftData projects, as SwiftData's cloud synchronization mechanism is identical to that of Core Data, and it also provides the same notification mechanism.


bluetooth_le_tracker crashes on unnamed iBeacon discovery

Home Assistant release with the issue:

0.81.4
Last working Home Assistant release (if known):
unknown
Operating environment (Hass.io/Docker/Windows/etc.):

docker
Component/platform:

https://www.home-assistant.io/components/device_tracker.bluetooth_le_tracker/
Description of problem:
Discovery of iBeacon without name causes the component to crash and stop working
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
device_tracker:
  - platform: bluetooth_le_tracker
    track_new_devices: true
    interval_seconds: 5

Traceback (if applicable):
2019-02-04 20:22:54 INFO (SyncWorker_8) [pygatt.backends.gatttool.gatttool] Discovered E9:DD:7E:39:44:8B (None)
2019-02-04 20:22:54 INFO (SyncWorker_8) [pygatt.backends.gatttool.gatttool] Discovered E5:74:A7:9E:BB:87 (None)
2019-02-04 20:22:54 INFO (SyncWorker_8) [pygatt.backends.gatttool.gatttool] Found 2 BLE devices
2019-02-04 20:22:54 DEBUG (SyncWorker_8) [homeassistant.components.device_tracker.bluetooth_le_tracker] Bluetooth LE devices discovered = {'E9:DD:7E:39:44:8B': None, 'E5:74:A7:9E:BB:87': None}
2019-02-04 20:22:54 INFO (SyncWorker_8) [homeassistant.components.device_tracker.bluetooth_le_tracker] Discovered Bluetooth LE device E9:DD:7E:39:44:8B
2019-02-04 20:22:54 DEBUG (SyncWorker_8) [homeassistant.components.device_tracker.bluetooth_le_tracker] Seen E9:DD:7E:39:44:8B 4 times
2019-02-04 20:22:54 DEBUG (SyncWorker_8) [homeassistant.components.device_tracker.bluetooth_le_tracker] Adding E9:DD:7E:39:44:8B to tracked devices
2019-02-04 20:22:54 ERROR (MainThread) [homeassistant.core] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/app/homeassistant/components/device_tracker/bluetooth_le_tracker.py", line 107, in update_ble
    see_device(address, devs[address], new_device=True)
  File "/usr/src/app/homeassistant/components/device_tracker/bluetooth_le_tracker.py", line 47, in see_device
    see(mac=BLE_PREFIX + address, host_name=name.strip("\x00"),
AttributeError: 'NoneType' object has no attribute 'strip'

Additional information:
The crash appears top happen when the BLE device is being added to the known_devices.yaml list (which seems to happen after a specific device has been seen 4 times)

Sorry, just found out I am not using the latest hass version, and it seems that the following commit fixed the issue:
https://github.com/home-assistant/home-assistant/commit/7def587c9305646f46ca5f8474bb6d46c7aebfb4