New command: Removing Azure Enterprise app consented user or group permissions

Usage
m365 entra userconsent remove --appId  -u [userid] -g [groupid]
Description
Remove user from enterprise app user consent
Options

Examples
Add channel to the specified Microsoft Teams team with name Team Name
entra userconsent remove --appId 4787c7ff-7cea-43db-8d0d-919f15c6354b --userid aa66f81f-12e7-4aaa-8c0d-68520333136d

Default properties
No response
Additional Info
No response

Thanks for the suggestion @rabwill! @pnp/cli-for-microsoft-365-maintainers let's see what's the best way to do it and which options we should support ideally.


LocalStorage doesn't work

When I set Storage: 'LocalStorage', my React page crashes on startup.
The error is:
TypeError: this.storage.getItem is not a function
CognitoAuth.getLastUser
node_modules/amazon-cognito-auth-js/es/CognitoAuth.js:441
438 | CognitoAuth.prototype.getLastUser = function getLastUser() {
439 |   var keyPrefix = 'CognitoIdentityServiceProvider.' + this.getClientId();
440 |   var lastUserKey = keyPrefix + '.LastAuthUser';

441 |   var lastUserName = this.storage.getItem(lastUserKey);
| ^  442 |
443 |   if (lastUserName) {
444 |     return lastUserName;

My code:
import React, { Component } from 'react';
import { CognitoAuth } from 'amazon-cognito-auth-js';

const authData = {
	ClientId: '<my client id>',
	AppWebDomain: '<my domain>',
	TokenScopesArray: ['profile','openid'],
	RedirectUriSignIn: 'http://localhost:3000',
	RedirectUriSignOut: 'http://localhost:3000',
        UserPoolId: '<my pool id>',
        Storage: 'LocalStorage'
};

class App extends Component {
  constructor() {
    super();

    this.auth = new CognitoAuth(authData);

    this.handleSignIn = this.handleSignIn.bind(this);
    this.handleSignOut = this.handleSignOut.bind(this);
    this.test = this.test.bind(this);
  }

  handleSignIn() {
    this.auth.getSession();
  }

  handleSignOut() {
    this.auth.signOut();
  }

  test() {
    console.log(this.auth.isUserSignedIn());
    console.log(this.auth.getCurrentUser());
    console.log(this.auth.getSignInUserSession());
    console.log(this.auth.getUserContextData());
  }

  render() {
    return (
      <div>
        <button onClick={this.handleSignIn}>SignIn</button>
        <button onClick={this.handleSignOut}>SignOut</button>
        <button onClick={this.test}>Test</button>
      </div>
    );
  }
}

export default App;

Try it without specifying the storage type. by default, it will use localStorage if localStorage not available it will automatically use in memory storage.  (if you need to understand how it works look at the implemntation of StorageHelper.js )
i. e  remove this line  Storage: 'LocalStorage'

Tried that already. When I do that, I see nothing in the local storage,
even after I am redirected back to my page with id token and access token
in the url. That's what made me explicitly specify LocalStorage in the
first place
On Sat, 27 Oct 2018, 22:45 Chamath Silva<EMAIL_ADDRESS>wrote:

Try it without specifying the storage type. by default, it will use
localStorage if localStorage not available it will automatically use in
memory storage. (if you need to understand how it works look at the
implemntation of StorageHelper.js
https://github.com/aws/amazon-cognito-auth-js/blob/master/src/StorageHelper.js
)
i. e remove this line Storage: 'LocalStorage'
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aws/amazon-cognito-auth-js/issues/165#issuecomment-433649908,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHVnyLRrLHphL8U73yUspXCa4sY5ZSikks5upLfugaJpZM4X6rwW
.

@YoniH  how did you solve this problem, i am facing problem. thank you in advance


Clicking package title of uninstalled package in Install search list does nothing

If you go to preferences, open the Install page, enter something into the search form, then a list of packages will show up. Their titles will be styled as links, but clicking on the titles of uninstalled packages does nothing. (I'd expect it to go to a page about that package, or at least open the package's page on atom.io in a browser window.) Clicking the title of an installed package brings you to an atom page for the package that gives you more info about the package.
Atom 0.170.0 on OS X

I believe this is a duplicate of https://github.com/atom/settings-view/issues/328


remove fusd

Closes #???
Description
FUSD has been deprecated in favour of USDC

[ ] Targeted PR against main branch
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work
[ ] Code follows the standards mentioned here
[ ] Updated relevant documentation
[ ] Re-reviewed Files changed in the Github PR explorer
[ ] Added appropriate labels

@nialexsan removal of FUSD still planned?

Yes, we removed almost all references on the frontend, and we still need to do the same for the code

Can we merge this PR?


LetDirective template binding - tests, docs and demo

(This PR is a direct continuation of this one: https://github.com/BioPhoton/rx-angular/pull/178)
Description
This PR adds ng-template bindings to be used with LetDirective. With this, we can create and bind templates for different state of an observable:

next - when values are emitted
complete - on observable completion
error - on observable error
🔥 suspense 🔥 - render something before the first value is emitted from an observable

Checklist
Checklist of things to be done for the feature:

[x] Implement the feature itself
[x] Ensure that all previous test are passing and the previous functionality still works
[x] Add tests
[x] Refactor the TemplateManager class to a factory function
[x] Update docs for LetDirective (jsdocs and Markdown)
[x] Add comments explaining low-level implementation details
[x] Add new page with demo in the template-demo app

Example
Here is some small, example component that uses the newest syntax:
@Component({
  selector: 'let-template-binding',
  template: `
    <button [unpatch] (click)="signals$?.complete()">
      complete
    </button>
    <button [unpatch] (click)="signals$?.next(random())">
      next
    </button>
    <button [unpatch] (click)="signals$?.error(errorStub)">
      error
    </button>

      value: {{ count | json }}

    <ng-template #complete>

COMPLETE

    </ng-template>
    <ng-template #error>

ERROR

    </ng-template>
    <ng-template #suspense>

SUSPENSE

    </ng-template>
  `,
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class LetTemplateBindingComponent {
  errorStub = new Error("Test error");

  signals$ = new Subject<any>();
  // signals$ = fromPromise(fetch('https://swapi.dev/api/people/1').then(a => a.json()));
  // signals$ = from([1,2,3,4,5]).pipe(first());

  // signals$ = interval(1000).pipe(
  //   switchMap(() => fromPromise(fetch(`https://swapi.dev/api/people/${Math.floor(Math.random() * 20)}`).then(a => a.json()))),
  //   takeUntil(interval(5000))
  // );

  visibleStrategy = 'local';

  random() {
    return Math.random();
  }
}

Demo
Inside template-demo app there's a new page - "Template Binding". There you can play with two example scenarios with usage of *rxLet and template binding:

manual Subject triggering,
triggering HTTP request for an external API on certain time intervals.

Any questions?
Comments are welcome! Feature is also ready to play with, so if you want you can switch to my branch and give it a try.

@BioPhoton thanks for the suggestions!
I thought about the name and didn't come up with anything good... Out of your suggestions the LazyTemplateManager might be the best one.
I just don't want to end up with something straight from Java's Spring like CacheableAbstractTemplateProxyManagerFactory :D Nevertheless, I understand, that TemplateManager might look too opaque.
Naming things is hard.

🎉🎉🎉

1.0.0-alpha.2 Should include the pr
😁


HTML/GWT is crashing in compile, how to load an image from url?

Issue details
When compiling WEB with gradlew.bat html:dist getting an exception:
The constructor Pixmap(byte[], int, int) is undefined
How can I load an image from a URL on WEB/GWT/HTML5?
Also, there is a warning there, is it safe to ignore?
Reproduction steps/code
OS: Windows 10
IDE: Intellij
Executing command: gradlew.bat html:dist
Version of LibGDX and/or relevant dependencies
Please provide the version(s) affected.
Stacktrace
Compiling error
> Task :html:compileGwt
Apr 27, 2020 7:54:12 AM java.util.prefs.WindowsPreferences <init>
WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.
Compiling module com.gazman.stories.GdxDefinition
   Tracing compile failure path for type 'my.file.java'
      [ERROR] Errors in 'file:/C:/my.file.java'
         [ERROR] Line 62: The constructor Pixmap(byte[], int, int) is undefined
   [ERROR] Aborting compile due to errors in some input files

Java source code:
    byte[] bytes = ...
    Pixmap pixmap = new Pixmap(bytes, 0, bytes.length);

Please select the affected platforms

[ ] Android
[ ] iOS (robovm)
[ ] iOS (MOE)
[ *] HTML/GWT
[ *] Windows
[ ] Linux
[ ] MacOS

The standard way includes using an InputStream with some REST protocols.  The Google HTTP Client can help some.  You might check StackOverflow for some references.

As you can see here, Pixmap on GWT has constructors which take libGDX's FileHandle (which could work for what you need) or HTML ImageElement. I haven't worked with it personally, but it seems that it should be possible to create an HTML <image> and load it from that.
As this isn't really a libGDX issue, please close it and check Discord or IRC.
Regarding the warning: I have never seen it, but it looks like Java's Preferences are for some reason trying to write into Windows registry. I don't know why would it do that. You will probably have to investigate that yourself, or, again, check Discord/IRC.

GWT / HTML could probably use some extra methods for simplicity's sake that help this.  Other wise, I am not using GWT at the moment.  I would add it my self if so.

@Darkyenus tnx for the tip, I found a working solution using an Image.
I actually had many problems with it

Net.HttpRequest didn't work for me so I used okHttp for handling Desktop and Android
I had cross-origin issues with loading the image, solved it by calling to
 image.getElement().setAttribute("crossOrigin", "anonymous");

I ended up building a library for loading images on Android, Desktop, and Web(don't have an iOS to test it on). Can you please guide me on how I can submit a pool request for it?

@IlyaGazman For info on pull requests, see this (or any other) general info tutorial and this libGDX specific page for contributors. You don't have to get it right the first time - that is what code review is for, so don't worry about it too much.
@shatterblast: By warning, I meant this line:
WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.,
which has nothing to do with any networking and everything to do with windows registry.

This is a long-standing problem that comes up from time to time. In my opinion, we need a Gdx.net.downloadImageToPixmap helper method with two different implementations for GWT and all other backends. It is an easy task using code from GWT backend, but the needed classes and methods are not visible to the core.
Everyone is invited to submit PRs.


TLSH fuzzy hash

TLSH is a fuzzy matching program and library. Given a file (min 50 bytes), TLSH generates a hash value which can be used for similarity comparisons. Similar files will have similar hash values which allows for the detection of similar objects by comparing their hash values TLSH has been adopted by a range of bodies and malware repositories including:

VirusTotal (https://developers.virustotal.com/v3.0/reference#files-tlsh)
Malware Bazaar (https://bazaar.abuse.ch/)
MISP (https://www.misp-project.org/)
STIX: TLSH is a part of the STIX 2.1 standard (https://docs.oasis-open.org/cti/stix/v2.1/stix-v2.1.html).

More information about TLSH, along with diff compare examples and reading in via buffer can be found at: https://tlsh.org/
Below is what we currently have for including TLSH as a Service, but we think TLSH would be a nice addition to the AssemblyLine Core to include with other hashes like SSDEEP.  Installed via py-tlsh
import shutil
import tempfile
from pathlib import Path

import tlsh
from assemblyline_v4_service.common.base import ServiceBase
from assemblyline_v4_service.common.result import Result, ResultJSONSection

class TlshService(ServiceBase):

    def __init__(self, config=None):
        super(TlshService, self).__init__(config)

    def start(self):
        self.log.info(f"start() from {self.service_attributes.name} service called")

    def execute(self, request):
        self.log.info(f"execute() from {self.service_attributes.name} service called for '{request.file_name}'")

        try:
            with open(request.file_path, "rb") as sample_file_object:
                working_dir = Path(tempfile.mkdtemp())

                try:
                    sample_bytes = sample_file_object.read()
                    tlsh_digest = tlsh.hash(sample_bytes).lower()

                    result = Result()

                    json_section = ResultJSONSection("TLSH Response")
                    json_section.set_json({'tlsh': tlsh_digest})
                    result.add_section(json_section)

                    request.result = result
                finally:
                    shutil.rmtree(working_dir, ignore_errors=True)

            result = Result()

        except Exception as ex:
            self.log.error(str(ex))
            raise

Fixed by: https://github.com/CybercentreCanada/assemblyline-base/pull/1205

Fixed in Release <IP_ADDRESS>


does not work with nested tag

Steps to reproduce:

some note:
---
tags:
  - struct/map
---

this is a note.

No effect.   No errors or logs in console as well.

MR with fix: https://github.com/brosell/obsidian-auto-note-organizer/pull/8


crossValidate.toJSON().sets appears to return incorrect data type

What is wrong?
Based on the docstrings, the crossValidate.toJSON().sets result should return an ICrossValidateStats object with the following properties:
export interface ICrossValidateStats {
  truePos: number;
  trueNeg: number;
  falsePos: number;
  falseNeg: number;
  total: number;
}

However, it appears to instead return an object that looks like this:
{ total: 152, testSize: 38, trainSize: 114 }

Which I can't see defined anywhere in the d.ts file, seems like an incorrect type definition/issue with the function return/I'm doing something wrong
Where does it happen?
I am running in a VSCode Devcontainer, here is the container file:
{
	"name": "Node.js 12",
	"dockerFile": "Dockerfile",
	"settings": {
		"terminal.integrated.shell.linux": "/bin/bash"
	},
	"extensions": [
		"dbaeumer.vscode-eslint"
	]
}

And here is the Dockerfile
FROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-12
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
   && apt-get -y install --no-install-recommends build-essential libxi-dev libglu1-mesa-dev libglew-dev pkg-config \
   && apt-get autoremove -y \
   && apt-get clean -y \
   && rm -rf /var/lib/apt/lists/*
ENV DEBIAN_FRONTEND=dialog

How do we replicate the issue?
Here's the code for the network definition (from the docs mostly):
// .... get data
const config = {
    binaryThresh: 0.5,
    hiddenLayers: [1],
    activation: 'sigmoid'
};

const crossValidate = new brain.CrossValidate(
    brain.NeuralNetwork,
    config
);

crossValidate.train(data, { log: true });

const json = crossValidate.toJSON();
console.log(json.stats); // { total: 152, testSize: 38, trainSize: 114 }

const net = crossValidate.toNeuralNetwork();

Expected behavior (i.e. solution)
I would have expected the function to return an ICrossValidateStats object with the props stated above
Version information
Nodejs:
v12.16.3
Browser:
None (running on Node.js)
Brain.js:
2.0.0-beta.1
How important is this (1-5)?
3, not a functional issue but is a bit annoying
Other Comments

Confirmed. We are migrating our code base to typescript, which will fix these type of issues. You can follow the progress/ contribute here: https://github.com/BrainJS/brain.js/issues/551


Start using ITensorMPS.jl

Looks like testing CI isn't set up, but locally I see a lot of test failures when using the latest NDTensors and ITensors which are unrelated to the introduction of ITensorMPS in this PR.

Looks like the function ITensorMPOCompression.get_subtensor_wrapper(::DenseTensor, inds, ::UnitRange{Int}...) defined here:
https://github.com/JanReimers/ITensorMPOCompression.jl/blob/e1aa17a7f7900d092f147f69a0c5631e610025ad/src/subtensor.jl#L171-L175
is hitting this bug in NDTensors.jl:
using ITensors: Index, ITensor, randomITensor
using NDTensors: Tensor
i, j = Index.((4, 4))
it = randomITensor(i, j)
t = Tensor(it)
t_sub = t[1:2, 1:2]
i_sub, j_sub = Index.((2, 2))
ITensor(t_sub, (i_sub, j_sub))

@JanReimers @kmp5VT

I raised an issue here: https://github.com/ITensor/ITensors.jl/issues/1431

I've confirmed locally that with the fix in https://github.com/ITensor/ITensors.jl/pull/1441 and the ITensor constructor in get_subtensor_wrapper changed over to using the TensorStorage instead of the Tensor the error I reported above goes away.
Not all tests pass, I believe because this codebase is relying on this PR: https://github.com/ITensor/ITensors.jl/pull/1099. So this PR is stalled for the time being.


Add tests for jsonrpc id

https://github.com/informalsystems/tendermint-rs/pull/196#discussion_r399790732
part of https://github.com/informalsystems/tendermint-rs/issues/211

[x] Referenced an issue explaining the need for the change
[ ] Updated all relevant documentation in docs
[ ] Updated all code comments where relevant
[x] Wrote tests
[ ] Updated CHANGES.md

Codecov Report

Merging #298 into master will increase coverage by 0.2%.
The diff coverage is 100.0%.

@@           Coverage Diff            @@
##           master    #298     +/-   ##
========================================
+ Coverage    30.8%   31.0%   +0.2%
========================================
  Files         129     129
  Lines        4893    4905     +12
  Branches     1549    1549
========================================
+ Hits         1511    1525     +14
+ Misses       2297    2294      -3
- Partials     1085    1086      +1

Impacted Files
Coverage Δ

tendermint/src/rpc/id.rs
60.8% <100.0%> (+60.8%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 622166b...73e3011. Read the comment docs.


Fix drag and drop of files in multiline textfield when a custom draggedType property is used

Please select one of the following

[ ] I am removing an existing difference between facebook/react-native and microsoft/react-native-macos :thumbsup:
[ ] I am cherry-picking a change from Facebook's react-native into microsoft/react-native-macos :thumbsup:
[x] I am making a fix / change for the macOS implementation of react-native
[ ] I am making a change required for Microsoft usage of react-native

Summary
In https://github.com/microsoft/react-native-macos/pull/1429 I had naively assumed that NSPasteboardTypeFileURL is 1:1 replacement for the deprecated NSFilenamesPboardType property. It is not. It causes drag and drop of files from Finder to stop working when a pastedTypes property is set.
Changelog
[macOS][Fixed] - Fix drag and drop of files in multiline textfield when a custom draggedType property is used
Test Plan
Before
https://user-images.githubusercontent.com/484044/195694913-2bdfa768-4e84-4e2e-9788-7eb5948f009c.mov
After
https://user-images.githubusercontent.com/484044/195694931-66821d49-81ff-4ceb-aeaf-055322648f6e.mov

So if I understand correctly, this is a return to the deprecated API?  Is there a "proper" way to use NSPasteboardTypeFileURL is pastedTypes is set?

It is a deprecated property. Xcode emits this warning
react-native-macos/Libraries/Text/TextInput/RCTBaseTextInputView.m:684:68: warning build: 'NSFilenamesPboardType' is deprecated: first deprecated in macOS 10.14 - Create multiple pasteboard items with NSPasteboardTypeFileURL or kUTTypeFileURL instead
Well the suggestion of simply using NSPasteboardTypeFileURL does not lead to the same behavior.
It seems some Swift users even try to bring back the property > https://stackoverflow.com/questions/44537356/swift-4-nsfilenamespboardtype-not-available-what-to-use-instead-for-registerfo


Add support for css and data attributes that correctly declare and generate inputs and outputs

This is based on https://github.com/ProdriveTechnologies/bazel-pandoc/pull/13 but fixes the issues pending there, it also adds a "data" attribute that can be used to declare stuff like images and other media referenced in the doc, and have them copied over as runfiles,

Hey thanks for looking at this, I had kind of given up on getting traction here and forgot to update this PR with my later fixes. It is updated now.

It is an almost complete rewrite of the CSS support with added support for the pandoc standalone and --self-contained flags.

Ping @ProdriveTechnologies


Update README.md - Fix spelling of 'plagues'.

Previous text said 'plages' — which means beaches by the sea.

Awesome, thanks for the contribution @seichert ❤


Add Attribution destination

Description of the change

Add https://www.attributionapp.com destination.

Type of change

[ ] Bug fix (non-breaking change that fixes an issue)
[x] New feature (non-breaking change that adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklists
Development

[ ] Lint rules pass locally
[ ] The code changed/added as part of this pull request has been covered with tests
[ ] All tests related to the changed code pass in development

Code review

[ ]  This pull request has a descriptive title and information useful to a reviewer. There may be a screenshot or screencast attached
[ ] "Ready for review" label attached to the PR and reviewers mentioned in a comment
[ ] Changes have been reviewed by at least one other engineer
[ ] Issue from task tracker has a link to this pull request

Thanks for the PR @mgz !
If it's not too much trouble, would you mind rebasing your changes on top of the current main branch? We had a busy couple of days after you created this PR, including:

Moving our Gatsby code into the docs repo
changing default branch to main
adding the Gatsby Link component for internal links
adding Netlify preview links for PRs

If you can do that, we should be able to avoid these conflicts which look to be due to the /docs prefix change/Gatsby update. You should also get a link to a Netlify preview build to verify changes on the web when you push the change.
You can also use Gatsby Link now which is much nicer, e.g.:
<Link to ="/rudderstack-cloud/rudderstack-connection-modes">RudderStack Connection Modes</Link>
Thanks again!
-pfd

@mgz thanks for making all the requested changes! While we review the changes, request you to please sign the CLA so that we can add you to the list of approved contributors.

@ameypv-rudder yep, just signed that.

Thanks for incorporating all the suggestions, @mgz :) One last query - are there any specific event/property mappings that we should include in the doc for the readers' information?
For reference, you can check this guide.

@ameypv-rudder no, nothing specific. We are Segment compatible and use any events clients send.

While the docs are good to, I was informed that the transformer code is not live yet. We will wait till the integration is published before pushing these changes.


Custom Head support

A request from the fancy side 😄

As you can see from the pictures, you see just the Steve head as item.
It exist mods/datapacks with multiple Custom Heads.

Display the Custom Heads as texture, instead of the Steve model:


Add poweron architecture ppc64le to travis build

This is part of the Ubuntu distribution for ppc64le. This helps us simplify testing later when distributions are re-building and re-releasing,For more info tag @gerrith3.
We typically build applications for customers and ISVs, and while we don't use this package directly, we do count on all of the packages in debian/ubuntu to build other packages. So we more likely have this as a second or third level dependency and couldn't tell you explicitly which features we use or our usage model.

@asellappen , thanks for the contribution!


docs: Create basic story for < Link />, < StoryDescription />

Monday ticket
Add basic story by structure similar to existing ones
https://github.com/mondaycom/vibe-storybook-components/blob/master/COMPONENTS_DOCUMENTATION_GUIDELINES.md

Hey, @Dhoni77!
Thanks, it's yours!

Closed thanks to @Dhoni77 !


Add support for Qodana

Testing IntelliJ Qodana https://www.jetbrains.com/help/qodana/github.html

I expected Qodana supports setting the threshold by severity (= ignore minor issues), but it seems there's no such configuration as far as I confirmed. failThreshold property is a bar regardless of the severity.


Revolutionary Tricolors

The converter happens to convert revolutionary tricolors to a full-white flag.

The dharmatized build doesn't translate the revolution at all (though it does boost liberals and guillotine reactionaries from upper house). Dharma status for revolutionary France is in fact a simple feudal monarchy, which is something PDX should probably solve internally.
We should override this manually of course.


Enable VSCode tests & fix BBE parsing

Fixed all tests except debugger ones.
Also fixed BBE diagram rendering.

Check List

[ ] Read the Contributing Guide
[ ] Required Balo version update
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #17155 into master will decrease coverage by 59.09%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #17155      +/-   ##
==========================================
- Coverage   68.85%    9.75%   -59.1%
==========================================
  Files           5       46      +41
  Lines         122     1455    +1333
  Branches       21      234     +213
==========================================
+ Hits           84      142      +58
- Misses         38     1300    +1262
- Partials        0       13      +13

Impacted Files
Coverage Δ

tool-plugins/vscode/src/core/extension.ts
26.94% <100%> (ø)

composer/packages/tracing/src/ToolBar.tsx

composer/packages/tracing/src/TraceList.tsx

composer/packages/tracing/src/DetailView.tsx

composer/packages/tracing/src/TraceLogs.tsx

composer/packages/tracing/src/index.ts

tool-plugins/vscode/src/utils/logger.ts
52.63% <0%> (ø)

tool-plugins/vscode/src/core/index.ts
100% <0%> (ø)

tool-plugins/vscode/src/utils/rpc/handler.ts
5.88% <0%> (ø)

tool-plugins/vscode/src/debugger/index.ts
0% <0%> (ø)

... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update beabf35...f51fcb3. Read the comment docs.


ppc64le: add files for supporting 14.04 'make debs'

Followup to #23438, this PR adds the necessary files needed in order to make ubuntu 14.04
ppc64le docker debs. There are two differences between trusty and xenial on power.
1.) trusty doesn't support seccomp out of the box
2.) trusty uses a different package for installing golang
Signed-off-by: Christopher Jones<EMAIL_ADDRESS>

LGTM

LGTM


CSS : Lists

Description
Introduction to Lists
Domain
Frontend Dev HTML/CSS/JS
Type of Contribution
Documentation
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

/assign

/assign

/assign

/assign

/assign


Keypair generation and sign/verify using ECDSA + sha1

Im trying to create a function for this jobs for my application.
Please, take a look at the question i did in lazarus forums:
https://forum.lazarus.freepascal.org/index.php/topic,53228.0.html
Thanks in advance.

Hi @DevTeamNoso, Attached is a sample that does what you want. ECDSASampleLaz.zip

Thanks for your quick reply. I will check it and let you know.


[WFCORE-4711] Upgrade XNIO to 3.7.7.Final

Jira: https://issues.jboss.org/browse/WFCORE-4711
EAP Jira: https://issues.jboss.org/browse/JBEAP-17789
Release notes:
    Release Notes - XNIO - Version 3.7.7.Final

        Enhancement

[XNIO-353] -         WARN message for rejected connections over Undertow max-connections limit

@fl4via thanks


Merge develop branch to v2 branch

Codecov Report

Merging #2474 (f9988df) into v2 (80f9009) will decrease coverage by 0.10%.
The diff coverage is 62.50%.

@@            Coverage Diff             @@
##               v2    #2474      +/-   ##
==========================================
- Coverage   97.64%   97.54%   -0.11%
==========================================
  Files          57       57
  Lines       10823    10693     -130
==========================================
- Hits        10568    10430     -138
- Misses        255      263       +8

Impacted Files
Coverage Δ

botocore/httpsession.py
90.49% <62.50%> (-2.53%)
:arrow_down:

botocore/compat.py
69.54% <0.00%> (-0.35%)
:arrow_down:

botocore/model.py
98.02% <0.00%> (-0.23%)
:arrow_down:

botocore/utils.py
96.82% <0.00%> (-0.18%)
:arrow_down:

botocore/docs/bcdoc/restdoc.py
96.26% <0.00%> (-0.17%)
:arrow_down:

botocore/retries/bucket.py
98.48% <0.00%> (-0.09%)
:arrow_down:

botocore/loaders.py
99.24% <0.00%> (-0.04%)
:arrow_down:

botocore/paginate.py
97.80% <0.00%> (-0.03%)
:arrow_down:

botocore/client.py
98.74% <0.00%> (-0.03%)
:arrow_down:

... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80f9009...f9988df. Read the comment docs.


Build error: 'for' loop initial declarations are only allowed in C99 or C11 mode

Report

[x] I have searched existing issues and this is not a duplicate

Issues and Steps to Reproduce
Building yoga-layout fails when using a Node Docker container, with an error message use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code from error: 'for' loop initial declarations are only allowed in C99 or C11 mode.
Steps to Reproduce:

Install Docker
Download node image: docker pull node
Run interactive container: docker run -it node sh
Try installing yoga-layout in an interactive session: yarn add yoga-layout

This issue should also be present with a standard Debian install, as this is what the Node Docker container is using.
Environment details:
# uname -a
Linux<PHONE_NUMBER>e5 4.9.31-moby #1  x86_64 GNU/Linux

# yarn -v
yarn install v0.24.6

# node -v
v8.1.2

Expected Behavior
Describe what you expected would happen.
Should install yoga-layout without errors.
Actual Behavior
Describe what actually happened.
Fails to install.
Full error log is here:
# yarn add yoga-layout
yarn add v0.24.6
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[1/1] ⠈ yoga-layout: /root/.npm/_logs/2017-07-05T05_25_45_061Z-debug.log
[-/1] ⠈ waiting...
[-/1] ⠈ waiting...
[-/1] ⠈ waiting...
error /node_modules/yoga-layout: Command failed.
Exit code: 1
Command: sh
Arguments: -c npm -- run autogypi && npm -- run build:node
Directory: /node_modules/yoga-layout
Output:
npm info it worked if it ends with ok
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>npm info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>
><EMAIL_ADDRESS>autogypi /node_modules/yoga-layout
> autogypi

npm info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>npm info ok
npm info it worked if it ends with ok
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm info lifecycle yoga-layout@1.5.0~prebuild:node<EMAIL_ADDRESS>npm info lifecycle yoga-layout@1.5.0~build:node<EMAIL_ADDRESS>
><EMAIL_ADDRESS>build:node /node_modules/yoga-layout
> npm -- run copy-sources && npm -- run node-gyp configure build

npm info it worked if it ends with ok
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>npm info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>
><EMAIL_ADDRESS>copy-sources /node_modules/yoga-layout
> ! npm -s -- run is-monolithic || (rsync -r --checksum --delete ../yoga/ sources/yoga/)

npm info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>npm info ok
npm info it worked if it ends with ok
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>npm info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>
><EMAIL_ADDRESS>node-gyp /node_modules/yoga-layout
> node-gyp "configure" "build"

gyp info it worked if it ends with ok
gyp info using<EMAIL_ADDRESS>gyp info using<EMAIL_ADDRESS>| linux | x64
gyp http GET https://nodejs.org/download/release/v8.1.2/node-v8.1.2-headers.tar.gz
gyp http 200 https://nodejs.org/download/release/v8.1.2/node-v8.1.2-headers.tar.gz
gyp http GET https://nodejs.org/download/release/v8.1.2/SHASUMS256.txt
gyp http 200 https://nodejs.org/download/release/v8.1.2/SHASUMS256.txt
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/node_modules/yoga-layout/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/local/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/root/.node-gyp/8.1.2/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/root/.node-gyp/8.1.2',
gyp info spawn args   '-Dnode_gyp_dir=/usr/local/lib/node_modules/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/root/.node-gyp/8.1.2/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/node_modules/yoga-layout',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/node_modules/yoga-layout/build'
  COPY Release/obj.target/nbind/geni/symbols.txt
  CC(target) Release/obj.target/nbind/sources/yoga/YGNodeList.o
../sources/yoga/YGNodeList.c: In function 'YGNodeListInsert':
../sources/yoga/YGNodeList.c:67:3: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
   for (uint32_t i = list->count; i > index; i--) {
   ^
../sources/yoga/YGNodeList.c:67:3: note: use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code
../sources/yoga/YGNodeList.c: In function 'YGNodeListRemove':
../sources/yoga/YGNodeList.c:79:3: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
   for (uint32_t i = index; i < list->count - 1; i++) {
   ^
../sources/yoga/YGNodeList.c: In function 'YGNodeListDelete':
../sources/yoga/YGNodeList.c:89:3: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
   for (uint32_t i = 0; i < list->count; i++) {
   ^
nbind.target.mk:134: recipe for target 'Release/obj.target/nbind/sources/yoga/YGNodeList.o' failed
make: Leaving directory '/node_modules/yoga-layout/build'
make: *** [Release/obj.target/nbind/sources/yoga/YGNodeList.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:258:23)
gyp ERR! stack     at emitTwo (events.js:125:13)
gyp ERR! stack     at ChildProcess.emit (events.js:213:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:197:12)
gyp ERR! System Linux 4.9.31-moby
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "build"
gyp ERR! cwd /node_modules/yoga-layout
gyp ERR! node -v v8.1.2
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
npm info lifecycle<EMAIL_ADDRESS>Failed to exec node-gyp script
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>node-gyp: `node-gyp "configure" "build"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>node-gyp script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!  Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>build:node script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2017-07-05T05_25_45_100Z-debug.log

I had the same problem, you can either:

upgrade g++ version (greater >4.8)
modify this repo: here adding a line '-std=c99' and here switch to c++1y

Tthe first solution is what I used (I was on heroku cedar14).

@theobat thanks so much for the hint.
I was running into the same issues with the docker node:8 container which already provides g++ version 4.9. So instead I've updated g++ and gcc to version 5 using a custom docker container. Here's the content of my Dockerfile which made installing yoga-layout smooth as butter.
# Use official node runtime as parent image
FROM node:8

# Install gcc-5 in order to support yoga-layout
RUN echo "deb http://ftp.us.debian.org/debian unstable main contrib non-free" > /etc/apt/sources.list.d/unstable.list &&\
    apt-get update &&\
    apt-get install --assume-yes -t unstable gcc-5 g++-5 &&\
    update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 60 --slave /usr/bin/g++ g++ /usr/bin/g++-5 &&\
    cc --version

tried @weaintplastic 's suggestion on Ubuntu 16.04 but no dice

It looks like this issue is stale. Please reopen if you see the same behavior on the latest version of Yoga.


Camera feedback

@LorenzMeier Is there any chance in getting camera feedback added to the version 1.5 milestone.  This would be great to have as we help test development of QGC mapping functionality.

Yes, let's do that.

@LorenzMeier One thing I forgot to mention here is that the feedback system would ideally include shutter count to compare how many pictures were taken against how many QGC calculated we should have taken in real time.

@DonLakeFlyer We're sending CAMERA_TRIGGER messages. Are you processing these?

Duplicate of #7164 now.


fixing the imports in the example

This PR just fixes a regression in a comment. The comment is just some code that outlines how to use serverutils to parse incoming messages in an EXTERNAL-Probe in order to implement SERVER-mode there.
The current comment lists imports that are not longer valid and I simply replaced the import statements with the correct ones.
No change on any running code performed through this PR. It should be safe to merge.
But as always please review carefully and use on your own risk.

@googlebot I signed it!


New model request

Hello,
Could you kindly add v5.0 and v5.0 Lightning to your RealVisXL list of supported weights? They were released 1-2 months ago and they are a useful improvement.
https://civitai.com/models/139562?modelVersionId=798204
Kind regards,
Ed

These have been added.


High performance to/from dimble

dimble is useful because it can alleviate IO bottlenecks in training pipelines.
Currently, there has been no work done to optimise conversion to/from the dimble format.
TODO:

Benchmark current dimble conversion times
Address bottlenecks
Repeat until satisfied

Closed as I've added a script https://github.com/StrongResearch/dimble/blob/main/scripts/to_dimble.py to do conversions in parallel. Can be revisited and further optimised as needed.


Full Standardization

See #32, #31.

Best not. Yants will complain (I did not use it correctly). Beside that I've been tinkering with the API, and the CLI doesn't support this yet either.


Is this still alive?

I really like the premises of the book and its style but... 10% completion and not content for a whole year... Please give me a light of hope 😄

Yes and no (she says 3 years 3 months later). The past four years have been a frustrating experience as just prior to the COVID-19 pandemic I cofounded a startup which has been the main focus of my efforts ever since.
That startup is currently looking like it's going to finally crash this summer, which isn't the idea outcome :/
There will definitely be more content for A Go Developer's Notebook as I have more than enough lurking from various talks I've given in recent years to easily double the current volume. And that's before even delving deeper into the topics I've already covered. Quite what that looks like I can't say just now because I don't have the time or energy to devote.
However I was serious when I said in the introduction that this is the only book I ever intend to write on Go - and quite possibly the only technical book. Whilst I'm alive there will be content begging to be added, and when I'm no longer alive the book will be finished. Well, unless I leave notes that suffer the Tolkien's Silmarillion treatment...


[Signal] Handle uuid sender identification

Signal / Signal-cli now allows identifying senders only by UUIDs, which makes us unable to identify contributors' first messages. (100EYES-3M)
As Till found out this can be resolved by creating contacts for each Signal contributor during onboarding, from which signal-cli can infer the UUIDs.

This was closed by #1428, right?


Push to API endpoint on new post

I'm trying to integrate scrapbook into hack hour so I can watch posts/updates on projects to help with review & etc. - could this be implemented?

implemented since #622


500 server error using facebook

Bot Info

Bot handle: informatics-lab-sol
App ID: c0dd83ca-0370-4310-96a6-c418c924870f
SDK Platform: Node.js
SDK Version: 3.9.1
Active Channels: Facebook
Deployment Environment: aws ec2

Issue Description
My service is returning 500s constantly
Error: Request to 'https://state.botframework.com/v3/botstate/facebook/users/1420943404688109' failed: [500] Internal Server Error

I am experiencing the same thing. Is there a status page somewhere?

I found this page, but it shows the service as being up.
https://azure.microsoft.com/en-gb/status/

Same error 500 here. Like 3 hours ago, we started to have this kind of errors in both our development (own openstack servers) and production environments (azure North EU).
Azure services status page does not show any alarm.
Any help is appreciated...

Thanks for reporting this.  We will investigate, and report back our findings asap.
(Note, the azure status page does not show Bot Framework connector status.  We are working on a status page.)

the service is up, but it keeps having the same issue.. Thanks Eric,hope we find it soon,because it was all of the sudden without any change..

can you give us any more information?

Would appreciate an update soon, as we have a release slated for later this afternoon that is being impacted...

I have the exact same issue and this is a serious, high priority bug for our users.
I went into chatconnector.js and printed the body response back from Microsoft, and I get this
{ error:
{ code: 'ServiceError',
message: 'Null properties cannot be encrypted. Please assign a default value to the property if you wish to encrypt it.' } }
Error: Request to 'https://state.botframework.com/v3/botstate/facebook/users/2014981635209074' failed: [500] Internal Server Error
at Request._callback (/app/node_modules/botbuilder/lib/bots/ChatConnector.js:514:46)
at Request.self.callback (/app/node_modules/request/request.js:188:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
at Request. (/app/node_modules/request/request.js:1171:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at IncomingMessage. (/app/node_modules/request/request.js:1091:12)
at IncomingMessage.g (events.js:292:16)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickCallback (internal/process/next_tick.js:104:9)

Likewise here. We have a user study and demo this afternoon. This is a very serious problem.

We are actively investigating this.

We're also experiencing the issue and its having a fairly large impact to us.

ddos? Big brother, speak to us. ETA?
Seems like SMS chat works, slack should not.
Also guess it is on FB and Slack side.

Looks like it has started working.

Yes, confirmed. My dev and prod bots are working again! Thanks!

We were experiencing similar issues earlier; looks like things have settled now. Is there any change of getting a status page anytime soon?

Oh wow, thank goodness, up and running 1.5 hours before our launch. Hopefully it's stable

Yes, thank you! Back up and running now. Any information on what caused this?

There was an issue with null values in bot properties.  We have done a rollback to fix the issue and are working on a hot fix to correct it permanently.

Thank you everyone, for reporting this to us.  Please open a new issue if you run into any new problems.

Everyone please note:  "By default, the Bot Framework SDK for .NET stores state data using the Bot Framework State service, which is intended for prototyping only and is not designed for use by bots in a production environment."  https://docs.microsoft.com/en-us/bot-framework/dotnet/bot-builder-dotnet-state
Please consider using the Azure Extensions, or implementing a custom state client: https://blog.botframework.com/2017/07/18/saving-state-azure-extensions/
Note: Bots using a custom state client did not experience these issues on September 15th.

You mention the .NET SDK, but this issue affected my Node.js client as well.
I'm a little surprised to hear that the State Service isn't meant for production.  This is one of the best features of the Bot Framework.  It significantly speeds up bot development and maintenance.  Without it everyone has to implement their own solutions, many of which are likely to incur an operational cost on top of the added dev costs.  Have you considered hardening the service so it's production-ready and perhaps even charging usage fees?

"It significantly speeds up bot development and maintenance."  <-- yes, there are definitely benefits to the default connector state service.  However, it is also relatively simple to include the Azure Extensions and use CosmosDb, Sql Server, or Azure Table Storage in either Node or .NET.  https://github.com/Microsoft/BotBuilder-Azure
There are Nuget and NPM packages:
https://www.nuget.org/packages/Microsoft.Bot.Builder.Azure/
https://www.npmjs.com/package/botbuilder-azure
I do not know the plans for the current state service.

?Thank you for the crucial information. I tried using bot service and it worked, i totally recommend it.

Από: Eric Dahlvang<EMAIL_ADDRESS>Στάλθηκε: Τρίτη, 19 Σεπτεμβρίου 2017 9:31 πμ
Προς: Microsoft/BotBuilder
Κοιν.: Bolotis Konstantinos; Comment
Θέμα: Re: [Microsoft/BotBuilder] 500 server error using facebook (#3466)
"It significantly speeds up bot development and maintenance." <-- yes, there are definitely benefits to the default connector state service. However, it is also relatively simple to include the Azure Extensions and use CosmosDb, Sql Server, or Azure Table Storage in either Node or .NET. https://github.com/Microsoft/BotBuilder-Azure
There are Nuget and NPM packages:
https://www.nuget.org/packages/Microsoft.Bot.Builder.Azure/
https://www.npmjs.com/package/botbuilder-azure
I do not know the plans for the current state service.

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/BotBuilder/issues/3466#issuecomment-330442753, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ANrwG0bwjBrVJe13U5c4CWOhKkYs0o9vks5sj1-kgaJpZM4PX9hK.

So, i guess that every bot framework solution already stored in the old way has to change to bot service? I am not capable of understanding this part of the problem.

Please confirm:

if node.js based bots must also implement a custom state client for storing conversation and user data.
if we need to remove every reference to session.userData, session.privateConversationData and son on in order to future proof our bots

Thanks!

@advl03 yes, Node.js based Bots should be using external state storage, as advised in the bot framework documentation. You can then continue to use session based storage (e.g. session.privateConversationData) as normal.

@advl03

node.js bots should use one of the Azure Extensions (documentdb, azure table storage, or sql: coming soon), or implement your own state client (not that difficult).  Also: this is not just for conversation and user data, but dialog data (like in a waterfall in node, or formflow in .net) are also stored in the state client.

no need to remove session.userData, etc: if the azure extensions implementation are used.  Azure Extensions will substitute the default.


Revisit TypeScript definitions for dual cjs/esm

For background see: https://github.com/tj/commander.js/issues/1880
Basically, we need a separate type definition file for each of cjs and esm.
Because this repo is all about the TypeScript, I suggest we add the esm.d.mts file beside the entry point and actually delete the "types" entry. Some TypeScript gurus recommend this and suggest the "types" entry is only needed for more complex scenarios when need to point to different folders.
https://arethetypeswrong.github.io/?p=%40commander-js%2Fextra-typings%4010.0.3

I tried to produce an actual warning or error and failed! I suspect because we (deliberately) no longer have a default export, TypeScript silently sorts the fallback and lets us away with a cjs type definition for an esm entry point.

TypeScript silently sorts the fallback and lets us away with a cjs type definition for an esm entry point.

If I got right what you are testing here - this is likely because of the "masquerading as CJS" issue. It looks like it works OK (and for most of the code out there it might be OK-ish) but this allows consuming this module in a way that wasn't intended - and if one would use a runtime export using this way then the output code could end with a runtime error. See the playground here

Released in v11


add handle_methods & authorize to Auth object

Added functionality to handle different methods of authentication. Also, added authorization per method, once a client has been authenticated. Like so:
var Auth = function () {
    this.handle_methods = function(realmName, secureDetails, callback) {
        secureDetails.authenticated = false;
        if (secureDetails.hasOwnProperty('authmethods')) {
            if (secureDetails.authmethods.includes('token')) callback(null, 'token', {});
            else if (secureDetails.authmethods.includes('secret')) {
                secureDetails.challenge = crypto.randomBytes(20).toString('hex');
                callback(null, 'secret', {challenge: secureDetails.challenge});
            }
            else callback(true);
        }
        else callback(true);
    };

    this.authorize = function(session, method, uri, callback) {
        if (session && session.secureDetails && session.secureDetails.authmethod && session.secureDetails.authenticated === true) {
            if (session.secureDetails.authmethod === "secret") callback(null, true);
            else if (session.secureDetails.authmethod === "token" && session.secureDetails.token) {
                jwt.verify(session.secureDetails.token, publicKey, {algorithm: 'RS256'}, (err, decoded) => {
                    if (err) callback(err);
                    else {
                        if (method === "unsubscribe") method = "subscribe";
                        if (method === "unregister") method = "register";
                        if (decoded.permissions.includes("wamp_" + method + "_" + uri)) callback(null, true);
                        else callback(null, false);
                    }
                });
            }
        }
        else callback(null, false);
    };

    this.authenticate = function (realmName, secureDetails, secret, callback) {
        secureDetails.authenticated = false;
        if (secureDetails.authmethod === "secret") {
            let verifier = crypto.createVerify('sha256');
            verifier.update(secureDetails.challenge);
            if(verifier.verify(publicKey, secret,'base64')) {
                secureDetails.authenticated = true;
                callback();
            }
            else callback(true);
        }
        else if (secureDetails.authmethod === "token") {
            jwt.verify(secret, publicKey, {algorithm: 'RS256'}, (err, decoded) => {
                if (err) callback(err);
                else {
                    if (decoded.permissions.includes("ws_connection")) {
                        secureDetails.token = secret;
                        secureDetails.authenticated = true;
                        callback();
                    }
                    else callback(true);
                }
            });
        }
        else callback(true);
    };
};

Hello!
I am sorry for the late answer.
Thank you for your contribution but this project requires unit tests for main functions. File wamp/gate.js is covered by unit tests. Could you please add some of it? This will help to understand how to use functionality later. Also unit tests helps to refactor code. Refactor change that do not break unit tests will be treated as acceptable.
Also lately I am changing formatting of the project according to https://github.com/standard/standard and unfortunately your code has conflicts.
If you add this improvements I will happy to merge you changes. Please do not hesitate to ask me any questions about this project.
Thank you,
Anatoly

Hi @kalmyk
thanks for the response. Right now I'm in the middle of something else, so I don't think I can make those changes straight away. Please leave this PR open until then.
For info, what I'm adding in this PR is mainly this feature on crossbar:
https://crossbar.io/docs/Authorization/
Thanks


Callback for $api->updatePlaylistImage

Hi,
is there a way to know exactly when the Image is uploaded and ready for $api->updatePlaylistImage?
I create a playlist, upload a cover and add tracks to it and then I redirect the user to this fresh playlist. But when he enters the playlist, the cover is not uploaded/saved yet.
Thanks in advance and Cheers, Christian

Hey!
Not that I'm aware of, I'm afraid. You could ask the Spotify folks to add one over at https://github.com/spotify/web-api.

Closing this due to a lack of activity.


Make thread_pool.h a public header

Rename ThreadPool -> FixedSizeThreadPool to allow for later introduction
of DynamicSizeThreadPool

Hmm, I thought that this PR was purely cosmetic, but Jenkins is reporting a lot of leaks on thread_stress_test (reproduced twice) . Does anyone think that leaks were triggered by this change?
https://grpc-testing.appspot.com/job/gRPC_pull_requests/949/config=asan,language=c++,platform=linux/testReport/junit/(root)/tests/bins_asan_thread_stress_test/


bugs in the new version!!!

tcl not packaged well enough.
bugs in parsing command line text
ref before defined for some other variables due to bad logic control

also needed a better copy of the pandas site-package

and dateutil

and secrets.py and hmac.py and six.py and pytz and adi (adi-reader) and fractions.py and multiprocessing and queue.py

multiprocessing won't multiprocess...
seems unable to locate python executable ... unclear if working directory related or if other libraries are needed to make it work


Provision plugin.send with correct janus message

{
  janus: 'message',
  data: {
    //....
  }
}

I didn't make it too narrow cause and include the body field cause we have plugin messages like this:
{
        "janus" : "message",
        "transaction" : "sBJNyUhH6Vc6",
        "body" : {
                "audio" : false
        },
        "jsep" : {
                "type" : "offer",
                "sdp" : "v=0\r\no=[..more sdp stuff..]"
        }
}

@tomaszdurka please review

As discussed we cannot force janus: message as we use other events like detach.


Update Instagram.php

Fixing getPaginateMediasByTag, it seems instagram has changed it's response. i don't know if it's temporary or not, but it did fix my issue.

I think fix for this should be similar to https://github.com/postaddictme/instagram-php-scraper/pull/1020
And also can I please ask you to re-test the code changes on real data.
Thank you


Sun RGB-D Dataset

Hello, thanks for the great work!
Could you please share the source from which you downloaded the SunRGBD dataset for evaluation?

I download the 5,050 SUN-RGBD test images from here: https://github.com/ankurhanda/sunrgbd-meta-data?tab=readme-ov-file#training-and-test-data-for-depth


Correct grammatical error in the JavaDoc of the Application interface

Corrects "this facilities" to "these facilities" in Application.java

I approve of minor fixes.
This paragraph I'm not too fond of anyway. When it says you can query the operating system, I expect to be able to check if we're on Windows, macOS, Solaris. But getType() isn't that granular and UiUtils is required instead.


Set Noble scan speed

Please tag this as an enhancement request
Please can a feature be added to change the interval that Noble scans for peripherals at?
For example, I want Noble to only scan for peripherals every 1 second

Hi @davidmessem,
This is a duplicate request of https://github.com/sandeepmistry/noble/issues/298 - so I will close as a duplicate.
Please note, this feature would be possible on Linux, however it is not possible on OS X because CoreBluetooth does not provide this feature.


Rating settings support for pnp core

https://github.com/OfficeDev/PnP/tree/dev/Samples/Core.ListRatingSettings shows how to enable ratings for a library in a publishing web. This is excellent IP to be included in the PnP core library.

Hi @jansenbe
I've added support to Core. created a pull request from the same.
Added required Test
Thanks
Akhilesh Nirapure

Thanks for the work @akhileshnirapure. Sorry for late follow up.


--stats gives weird info message

I get the message
Diff-Total: 3 changed rows out of None
Looking at the code, it looks like None should be the count of records in table 1. My test case is a really small table (only 3 rows) so wondering if that's what's throwing things off.

This happens when all the rows were downloaded, not checksummed. Will fix

I believe @sirupsen fixed this.


Add documentation for the units module

Coverage remained the same at 93.082% when pulling 464f682aed7823b88905c5bea2856fbc14c7cb40 on razvanvasile:module_docstrings into 7a2a29e3322d7e098838e7ed0d71cd292e026750 on willrogers:master.

Coverage remained the same at 93.082% when pulling 464f682aed7823b88905c5bea2856fbc14c7cb40 on razvanvasile:module_docstrings into 7a2a29e3322d7e098838e7ed0d71cd292e026750 on willrogers:master.

Coverage remained the same at 93.082% when pulling fc09eb24f045cf34ced204ff3cba70dcd62e5ebd on razvanvasile:module_docstrings into 7a2a29e3322d7e098838e7ed0d71cd292e026750 on willrogers:master.

Coverage remained the same at 93.082% when pulling a61573f9275160a50c16aa4c540c8e179498e731 on razvanvasile:module_docstrings into 7a2a29e3322d7e098838e7ed0d71cd292e026750 on willrogers:master.


Correction for raw affiliation CTIFL

Correction needed for raw affiliation CTIFL
raw_affiliation_name: CTIFL
new_rors: 02a63xw04
previous_rors: 03hye4039
works_examples: W4236769799;W4233896992;W4232369798;W4232856958;W3095963911
contact: d058789bdefd6b2fd85d81734ba57688:f306039abaa2c3033631c9 @ ctifl.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


Added support to retrieve metadata fields from milvus

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)

If we connect to a milvus vector DB with pre-existing data, we are able to get only the field that is concerned with the embedding (aka "text" field) but couldn't get other fields that can have metadata. Therefore added a metadata_keys class param that specifies all the other keys to be retrieved from milvus query response.
Fixed another bug in QueryFusionRetriever where the same embed_model was used even if two separate ones are specified in individual retrievers.

New Package?
Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

[ ] Yes
[x] No

Version Bump?
Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

[ ] Yes
[x] No

Type of Change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Added new unit/integration tests
[ ] Added new notebook (that tests end-to-end)
[x] I stared at the code and made sure it makes sense

Suggested Checklist:

[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added Google Colab support for the newly added notebooks.
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I ran make format; make lint to appease the lint gods

For us this is a breaking change. See the example:
my_vector_store = MilvusVectorStore(
    dim=768, overwrite=False, collection_name="my_collection", uri="standalone:19530"
)

Which will result in the following error:
_ ERROR collecting tests/unit/xxx/app/core/rag/test_rag_functions.py _
tests/unit/xxx/app/core/rag/test_xxx.py:86: in <module>
    vector_store = MilvusVectorStore(dim=768, overwrite=False, collection_name="my_collection")
.venv/lib/python3.11/site-packages/llama_index/vector_stores/milvus/base.py:143: in __init__
    super().__init__(
.venv/lib/python3.11/site-packages/pydantic/v1/main.py:341: in __init__
    raise validation_error
E   pydantic.v1.error_wrappers.ValidationError: 1 validation error for MilvusVectorStore
E   output_fields
E     none is not an allowed value (type=type_error.none.not_allowed)

probably caused due to the following change?

Hey, I deleted my previous comment, because I wasn't sure, but I am coming back here, since it still looks like the change from Milvus to Pydantic Model was a breaking change it MilvusVectorStore constructor is broken now.
Whenever we use the MilvusVectorStore constructor (no matter the arguments actually used), we get the following error related to pydantic:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/A78751003/projects/genai/document-service/.venv/lib/python3.11/site-packages/llama_index/vector_stores/milvus/base.py", line 143, in __init__
    super().__init__(
  File "/Users/A78751003/projects/genai/document-service/.venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for MilvusVectorStore
output_fields
  none is not an allowed value (type=type_error.none.not_allowed)

We are using the following libraries (I just re-created the .venv minutes ago):
❯ pip freeze | grep llama
llama-index==0.10.28
llama-index-agent-openai==0.2.2
llama-index-cli==0.1.11
llama-index-core==0.10.28
llama-index-embeddings-openai==0.1.7
llama-index-indices-managed-llama-cloud==0.1.5
llama-index-legacy==0.9.48
llama-index-llms-openai==0.1.15
llama-index-llms-openai-like==0.1.3
llama-index-multi-modal-llms-openai==0.1.5
llama-index-packs-fusion-retriever==0.1.3
llama-index-program-openai==0.1.5
llama-index-question-gen-openai==0.1.3
llama-index-readers-file==0.1.16
llama-index-readers-llama-parse==0.1.4
llama-index-retrievers-bm25==0.1.3
llama-index-storage-docstore-mongodb==0.1.3
llama-index-storage-docstore-redis==0.1.2
llama-index-storage-index-store-mongodb==0.1.2
llama-index-storage-kvstore-mongodb==0.1.2
llama-index-storage-kvstore-redis==0.1.3
llama-index-vector-stores-milvus==0.1.8
llama-index-vector-stores-postgres==0.1.5
llama-parse==0.4.0
llamaindex-py-client==0.1.16

Yes. I have mentioned the same in my comments above to @logan-markewich who made this change.

would adding this work?
        if output_fields is None:
            output_fields = []

Setting output_fields default value in the MilvusVectoreStore arguments to the empty list should work.

@niderhoff this should be fixed and published in another PR I made today
pip install -U llama-index-vector-stores-milvus
https://github.com/run-llama/llama_index/pull/12704

(Cc @andrew-augment @chandrashekarvt )


[SPARK-27064][SS] create StreamingWrite at the beginning of streaming execution

What changes were proposed in this pull request?
According to the design, the life cycle of StreamingWrite should be the same as the read side MicroBatch/ContinuousStream, i.e. each run of the stream query, instead of each epoch.
This PR fixes it.
How was this patch tested?
existing tests

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8521/
Test PASSed.

cc @jose-torres

Test build #103059 has started for PR 23981 at commit 3261ed5.

Test build #103059 has finished for PR 23981 at commit 3261ed5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class WriteToMicroBatchDataSource(write: StreamingWrite, query: LogicalPlan)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103059/
Test FAILed.

Test build #103078 has started for PR 23981 at commit f835a5c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8542/
Test PASSed.

Test build #103078 has finished for PR 23981 at commit f835a5c.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103078/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8554/
Test PASSed.

Test build #103092 has started for PR 23981 at commit f835a5c.

Test build #103092 has finished for PR 23981 at commit f835a5c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103092/
Test PASSed.

LGTM.

retest this please

Test build #103412 has started for PR 23981 at commit f835a5c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8834/
Test PASSed.

Test build #103412 has finished for PR 23981 at commit f835a5c.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103412/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8846/
Test PASSed.

Test build #103426 has started for PR 23981 at commit f835a5c.

Test build #103426 has finished for PR 23981 at commit f835a5c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103426/
Test PASSed.

thanks, merging to master!


Add consumption rate limiter for LLConsumer

Description
Having a consumption rate limiter for a table is beneficial in two ways:

Currently offline tables can be configured to have storage quota. The storage quota for tables safegaurds the pinot cluster hosting those tables in case of excessive uploads of offline segments so that the cluster won't be under provisioned. Consumption rate limiter can play similar role for hybrid tables and realtime-only tables as it limits the amount of data being ingested into the realtime tables.
Bursty pattern of incoming Kafka events introduces long time GC pauses on Pinot server. This behavior has been observed in some tables in LinkedIn. It's also been reported in this issue: #6138.

This PR introduces rate limit on Kafka consumption which can be configured in StreamConfig. The consumption rate limit parameter is defined for the whole topic and the rate limit for each partition will be simply topic rate limit divided by the partition count of that topic.
Testing Done

Locally added consumption rate limit to LLCRealtimeClusterIntegrationTest and verified the consumption rate was limited the the amount set in the table config.
Deployed to a perf cluster with two servers, one with rate limit enabled and one without. Here are some chart comparing a few metrics on the two hosts (red: w/o consumption rate limit, blue: w/ consumption rate limit):

Ingestion rate for a period of two hours when incoming message rate was more than the consumption rate limit in the table config:

Young generation GC collection time for the same time period.

Young generation GC count for the same time period.

other metrics like old gen gc metrics, cpu utilization, ... were pretty much identical.

Codecov Report

Merging #6291 (6c8bb8b) into master (1beaab5) will decrease coverage by 20.58%.
The diff coverage is 52.01%.

@@             Coverage Diff             @@
##           master    #6291       +/-   ##
===========================================
- Coverage   66.44%   45.86%   -20.59%
===========================================
  Files        1075     1253      +178
  Lines       54773    61253     +6480
  Branches     8168     8868      +700
===========================================
- Hits        36396    28093     -8303
- Misses      15700    30870    +15170
+ Partials     2677     2290      -387

Flag
Coverage Δ

integration
45.86% <52.01%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ot/broker/broker/AllowAllAccessControlFactory.java
100.00% <ø> (ø)

.../helix/BrokerUserDefinedMessageHandlerFactory.java
52.83% <0.00%> (-13.84%)
:arrow_down:

...org/apache/pinot/broker/queryquota/HitCounter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...che/pinot/broker/queryquota/MaxHitRateTracker.java
0.00% <0.00%> (ø)

...ache/pinot/broker/queryquota/QueryQuotaEntity.java
0.00% <0.00%> (-50.00%)
:arrow_down:

...ker/routing/instanceselector/InstanceSelector.java
100.00% <ø> (ø)

...ceselector/StrictReplicaGroupInstanceSelector.java
0.00% <0.00%> (ø)

.../main/java/org/apache/pinot/client/Connection.java
22.22% <0.00%> (-26.62%)
:arrow_down:

...not/common/assignment/InstancePartitionsUtils.java
64.28% <ø> (-8.89%)
:arrow_down:

.../apache/pinot/common/exception/QueryException.java
90.27% <ø> (+5.55%)
:arrow_up:

... and 1247 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe9d3c7...6c8bb8b. Read the comment docs.

Can you note in your checkin comments that what we are throttling is not really the consumption, but the processing of messages. We will still consume as much as we can from the stream.
If we were to limit consumption, then the behavior will be somewhat like:
while (true) {
  consumeMsgsAsPerSomeAllowedRate()
  processAllMsgsConsumed()
  sleepAsIndicatedByRateLimiter()
}

Whereas by rate limiting the processing, we are doing the following:
while (true) {
  consumeAllMsgsThatWeCan()
  foreach(msg) {
    processMsg()
    sleepAsDictatedByRateLimiter()
  }
}

So, we may be taking up more heap in the second case ?

Can you note in your checkin comments that what we are throttling is not really the consumption, but the processing of messages. We will still consume as much as we can from the stream.
If we were to limit consumption, then the behavior will be somewhat like:
while (true) {
  consumeMsgsAsPerSomeAllowedRate()
  processAllMsgsConsumed()
  sleepAsIndicatedByRateLimiter()
}

Whereas by rate limiting the processing, we are doing the following:
while (true) {
  consumeAllMsgsThatWeCan()
  foreach(msg) {
    processMsg()
    sleepAsDictatedByRateLimiter()
  }
}

So, we may be taking up more heap in the second case ?

I actually looked into that and Kafka doesn't provide an API to retrieve limited number of messages. IMO having a rate limit on processing will have similar effect as if we put the rate limit on the consumption because we synchronously process the messages after the messages are polled from Kafka. For example, let's assume for a bursty period, the incoming rate of Kafka messages is 100 msgs/sec and we have set the rate limit to 20 msgs/sec. That means for a period of 10 seconds, we only process 200 messages and while we're processing messages, we don't consume new messages. This effectively puts the consumption rate to 20msgs/sec while if there was no rate limit we would've consumed 1000 messages at rate 100 msgs/sec.
Side note:
Kafka consumer has this configuration max.partition.fetch.byte that limits the count of consumed bytes. That is a bit hard to utilize as the intention here is to consume less number of messages than consumed bytes.

Codecov Report

Merging #6291 (bd5ce12) into master (c85ff1d) will decrease coverage by 1.10%.
The diff coverage is 50.00%.

@@             Coverage Diff              @@
##             master    #6291      +/-   ##
============================================
- Coverage     71.30%   70.20%   -1.11%
- Complexity     4214     4260      +46
============================================
  Files          1596     1607      +11
  Lines         82756    83454     +698
  Branches      12348    12460     +112
============================================
- Hits          59006    58585     -421
- Misses        19761    20818    +1057
- Partials       3989     4051      +62

Flag
Coverage Δ

integration1
28.95% <40.38%> (-0.01%)
:arrow_down:

integration2
?

unittests1
67.85% <37.25%> (-0.28%)
:arrow_down:

unittests2
14.23% <0.00%> (-0.06%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...pache/pinot/spi/stream/StreamConfigProperties.java
100.00% <ø> (ø)

...nager/realtime/RealtimeConsumptionRateManager.java
39.47% <39.47%> (ø)

...java/org/apache/pinot/spi/stream/StreamConfig.java
90.76% <71.42%> (-2.43%)
:arrow_down:

...manager/realtime/LLRealtimeSegmentDataManager.java
68.88% <83.33%> (-2.38%)
:arrow_down:

.../pinot/server/starter/helix/BaseServerStarter.java
51.82% <100.00%> (-8.02%)
:arrow_down:

...t/core/plan/StreamingInstanceResponsePlanNode.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ore/operator/streaming/StreamingResponseUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ager/realtime/PeerSchemeSplitSegmentCommitter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...pache/pinot/common/utils/grpc/GrpcQueryClient.java
0.00% <0.00%> (-94.74%)
:arrow_down:

...he/pinot/core/plan/StreamingSelectionPlanNode.java
0.00% <0.00%> (-88.89%)
:arrow_down:

... and 147 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c85ff1d...bd5ce12. Read the comment docs.

This is a good feature to have. Few thoughts

this seems very specific to Kafka and low-level consumers.
It might be better to make this real-time table level config and move this into indexRow within MutableSegment.
is the throttling configuration per table or per server or per partition?

This throttling is a per-table configuration and it's applied to low level consumer. It's not necessarily tied to Kafka. Low level consumer can have any implementation. The throttling happens after events are fetched from the stream and before processing of those fetched events starts.

The multi-partition rate limiter seems to be created only once. If the number of partitions changes, where is it adjusted?

It's created only once and that's in the constructor of LLRealtimeSegmentDataManager. When the number of partitions changes, nothing happens to the existing consuming segments, i.e. existing LLRealtimeSegmentDataManager objects, but there will be new consuming segments and consequently new LLRealtimeSegmentDataManager objects will be created. Each of these new objects will have their own rate limiters with the new partition count.

Then why do we have a 10-minute cache refresh for the number of partitions? Just fetch it once and set the rate limiter right?

Then why do we have a 10-minute cache refresh for the number of partitions? Just fetch it once and set the rate limiter right?
Fetching-it-once works for current consuming segments, but it'll be problematic for the future consuming segments in case the number of partitions changes.
The cache is used to prevent fetching the num partitions for multiple consuming segments of the same topic on the same host.

+1 to this feature.
Does the current implementation allow dynamic rate limit addition or change? For ops, it would be nice to adjust the quota and expect it to be effective immediately. Currently implementation seems to create the rate limiter once upon the LLRealtimeSegmentDataManager  creation

+1 to this feature.
Does the current implementation allow dynamic rate limit addition or change? For ops, it would be nice to adjust the quota and expect it to be effective immediately. Currently implementation seems to create the rate limiter once upon the LLRealtimeSegmentDataManager  creation

You're right, in this implementation changes in the ingestion quota config doesn't get reflected in the current consuming segments and we need to wait till they get completed and then the new consuming segments will pick up the new rate limit.
For immediate reflection of rate limit change, we can add ZK watches on table config or read the table configs in short periods of time like in one minute. Then adjust the rate limit if it's changed. But I think using these approaches adds complexity and maybe extra pressure on ZK.
One simpler and less expensive approach is that after consumption rate limit is changed, the operator calls a force commit endpoint on controllers for existing consuming segments. Then the next consuming segments will automatically pick up the new consumption quota value. I'm finalizing a design doc for pause / resume consumption which includes the force commit logic and will share it soon.
@yupeng9 @mcvsubbu what do you guys think?

+1 to this feature.
Does the current implementation allow dynamic rate limit addition or change? For ops, it would be nice to adjust the quota and expect it to be effective immediately. Currently implementation seems to create the rate limiter once upon the LLRealtimeSegmentDataManager  creation

You're right, in this implementation changes in the ingestion quota config doesn't get reflected in the current consuming segments and we need to wait till they get completed and then the new consuming segments will pick up the new rate limit.
For immediate reflection of rate limit change, we can add ZK watches on table config or read the table configs in short periods of time like in one minute. Then adjust the rate limit if it's changed. But I think using these approaches adds complexity and maybe extra pressure on ZK. One simpler and less expensive approach is that after consumption rate limit is changed, the operator calls a force commit endpoint on controllers for existing consuming segments. Then the next consuming segments will automatically pick up the new consumption quota value. I'm finalizing a design doc for pause / resume consumption which includes the force commit logic and will share it soon.
@yupeng9 @mcvsubbu what do you guys think?

Yes, I think this is a good solution. In fact, some other config changes such as schema evolution may follow a similar approach of force closing the consuming segments, as those changes also encounter similar challenges of waiting for the consuming segment to close.

I suggest we first start off with the simple PR that is here, and then try to improve it with dynamic changes, etc. For the specific use case of operator changing the table config, a force commit is a great option. We are going to put the design out on that soon

I suggest we first start off with the simple PR that is here, and then try to improve it with dynamic changes, etc. For the specific use case of operator changing the table config, a force commit is a great option. We are going to put the design out on that soon

Yes, makes total sense.

@sajjad-moradi Since the feature is merged (and hopefully well tested in prod), could you please add a release-note section to the PR description on how to enable it and also update the pinot documentation?


Fix geocoding batched flag

@Kartones can you please review? (check the commit messages and ask me if any doubt)
Thanks!

Frontend tests were OK :+1: (details)

(I broke the tests with a silly thing related to config, not code, gonna fix it in a sec)

:+1:

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)


Feat: save model weights as hd5 to increase transparency of contents

Problem description
Pickles may contain malicious code that is difficult to check (e.g. see solutions in https://huggingface.co/docs/hub/security-pickle). Wary users may not want to download pickled model weights.
Current situation
torch.save is used to save model weights, pickling weights
Desired situation
A more "open" way of saving model weights is used, or a ready-to-use safe-pickle script is provided.
Suggested solution
Use the suggestion in https://github.com/pytorch/pytorch/issues/52181 to save the model weights as an hd5. This requires a relatively heavy import and install, but is much more transparent and extremely easy to implement. I believe hd5 is already used for saving the features.
As taken from the given solution

We started using hdf5 for this after reading this thread. Made sense to us and wasn't too much work to set up. Something like
import h5py
path = 'item/'
model = torch.load(path_to_model.pt)
with h5py.File(output_path.h5, 'w') as h5file:
    for key, item in model.items():
        h5file[path + key] = item

to secure the model
  from collections import OrderedDict
  file = h5py.File(output_path.h5)
  item = file["item"]
  keys = item.keys()
  dict_ = OrderedDict()
  for key in keys:
      dict_[key] = torch.tensor(item[key][()])
  torch.save(dict_, reconstructed_model.pt)

to get back the original

So likely implement some marugoto.utils.models with save_h5_weights(model: torch.nn.Sequential, path: pathlib.Path) -> None and load_h5_weights(path: pathlib.Path) -> torch.nn.Sequential, or something that's something similar to the torch load and save API.

The way the pipeline currently rely on a lot of the metadata (concerning transforms, etc.) in the model, so it'd probably be more trouble than it's worth at the moment. I was thinking about transitioning the model to lightning and deprecating the fast.ai version at some point (because there's some other problems with fast.ai), so we could definitely incorporate that in the "new" version.


Als Ausgleich zu Vorschlag 2 (da Ritter weniger op sind als berittene Bogenschützen):

Ställe beherbergen statt 4 Pferden 8 Pferde

Da auf beiden Seiten je 4 Pferdeköpfe rausschauen sollte das auch nicht allzu unlogisch sein.

I agree with the proposed changed to the shields 100%.
I am not sure about the sable, however, making them have 8 horses seems rather hacky you can't see the horses on the "other side" after all. maybe instead stables would refill constantly at a much slower rate so you could have 1 stable and if you play long enough infinite knights? or maybe with a higher cap per stable which would kinda go your direction as well but it would never have 8 horses in one stable at the same time but start putting out a new batch of horses after the first one is dispatched.

ByBurton I updated my comment with most of your suggestions with a few exeptions:
Shields are extremely strong and since 2.6 you can build really easy really very much of them. I already played a lot of matches on a map without ressources, shields in combination with archers and monks are the best way to win that kind of match. But not only in that kind of game, also in matches on maps with really much ressources shields are used, so even when a player has an infinite number of ressources. So in my opinion that would be the best way to make them less strong. They would be still extremely usefull, because most battles in Stronghold are decided by arrows and shields have a really high influence on such battles, but they wouldn't be that overpowered anymore.
The thing with knights is: I think it would be logical to make all kind of riders, so I thought increasing the amount of stables would let stables and riders still be a really useful option.
 More resistance would make them too usefull against all kinds of troops, because a good player will protect his siege engines too well and already shoots with his stones on enemies armies.
Monsterfish I'm not sure about your suggestion, because it would have a really high influence on the game as we know it. Stronghold 2 had a system like that, but knights there were also extremly expensive.

There would be a cap for the horses regardless just higher as you suggested with 8 it just means you can't recruit 8 knights in a swoop but first 4 and than more afterward just it's essentially your idea just slower :P
that being said it was mostly food for thought as i don't think knights are particularly underpowered as a disturbing/interception unit.

In my opinion knights are actually too strong.
Yes, they are veery expensive, and the stables take up a lot of space.
But attacking with ~8 knights you can destroy most economy with easy.
Talking about single player of course.
Online players just spam hundreds of crossbowmen, and put them all on a single tower and stuff like that.

Don't agree with the point about the siege towers; it is ridiculously easy to move a siege tower up to the walls of an AI even if they have loads of archers, barely any damage is caused. The AI generally don't do anything about a siege tower on the walls for ages either, this needs to be fixed so they can send their counterattack units eg Pig and Sheriff with macemen, Wazir, Emir and Saladin with their Arabian swordsmen to destroy it sooner.

Actually, you are right. I kinda miss-remembered that. I was thinking about a different Stronghold.
But horse archers are a bit op. They need a slight nerf. With nerfing the range I meant really like just a little bit. They should not be able to shoot as far as an archer that can take his time to aim. Or make their accuracy a bit lower.

blows the dust of the thread
Considering horse archers have no armor and are a bigger target due to the horse why not nerf them by reducing their health to be similar to that of a monk if not like slaves making them more of a glass cannon granted shields would probably still help them a lot but this way once the shields are mostly taken care of they will fall like flies. this way it would also somewhat balance out with the knights where if the horse archers are cornered  by melee units they have little chance to fight back with their short shorts and no armor (atm you can pretty much overrun a group of 20 knights with your 100 horse archers easily in close combat  while "only" loosing around 40 units so around 2 archers for one knight seems like a flat out terrible ratio considering knights will always be very limited in numbers.

I think there are several ways HA can be nerfed, however shields are the real issue. If they can be nerfed a little, then HA arent as much of a threat. still a threat, but not as much.
Ways to adjust HA (sorry if these have been mentioned already):

reduce range (they are using smaller bows which historically had less range than the european longbow)
reduce health (read monsterfisch's excelent point about melee with knights)
reduce fire rate (currently they match arabian archers while moving, and are faster when standing still, that makes little sense to me)
reduce strength (again historically more accurate, the smaller bows cant pack as much of a punch)
increase cost (arabian archer + 5 gold gets a horse? thats one cheap horse!)
Any one of these, or a combination or them, would be a good thing i think.
If I had the option to adjust the balance, I would first modify the shield (make them slower, they should be catchable by spears and macemen, and slightly weaker to fire ballista and mangonels) - this will also fix other OP units that often get protected by shields (e.g. catapults), and then see how the Horse Archers fare without that extra protection, If they still appear OP, then id probably start by increasing the cost to 130-150 range, and maybe reducing the fire rate to be about 3/4s of what it is.

Making shields slower and giving horse archers less hitpoints should i believe be enough nerfing to make it interesting without making the horse archer shield combo unplayable considering the damage and range and accuracy of a real horse archer is not really much less than a stationary archer as with enough training you can sit very stable in the saddle. fire rate could be a different story but I am not speaking of experience there.
Either way, I think making shields slower (keep the health) and horse archers less healthy should do the trick.