Factory HttpClientOptions not used by Client in createAppCheck

Describe the bug
When we set options to the Factory for timeout and proxy, the call to the method createAppCheck() will return a new AppCheck which is using a CachedKeySet that can't use the same options.
        use Kreait\Firebase\Factory;
        $factory = (new Factory)->withServiceAccount('/path/to/firebase_credentials.json');
        $options = HttpClientOptions::default();
        $options = $options->withTimeOut(3.5);
        // Use a proxy that all API requests should be passed through.
        $options = $options->withProxy('tcp://<host>:<port>');
        $factory = $factory->withHttpClientOptions($options);

        $appCheck = $factory->createAppCheck();

Here, $appCheck will not have the proxy and timeout options for the CachedKeySet :
        $keySet = new CachedKeySet(
            'https://firebaseappcheck.googleapis.com/v1/jwks',
            new Client(), // <---- new Guzzle Http Client which will not receive the Guzzle Config
            $this->httpFactory,
            $this->keySetCache,
            21600,
            true,
        );
        return new AppCheck(
            new AppCheck\ApiClient($http),
            new AppCheckTokenGenerator(
                $this->serviceAccount['client_email'],
                $this->serviceAccount['private_key'],
                $this->clock,
            ),
            new AppCheckTokenVerifier($projectId, $keySet),
        );

It works for us only if we modify like this :
        $keySet = new CachedKeySet(
            'https://firebaseappcheck.googleapis.com/v1/jwks',
            new Client($this->httpClientOptions->guzzleConfig()), // <--- we pass the Factory's GuzzleConfig
            $this->httpFactory,
            $this->keySetCache,
            21600,
            true,
        );

Installed packages
firebase/php-jwt : v6.8.0

PHP version and extensions
PHP 8.1.19
Composer 2.2.7

Steps to reproduce the issue.
With this example :
        use Kreait\Firebase\Factory;
        $factory = (new Factory)->withServiceAccount('/path/to/firebase_credentials.json');
        $options = HttpClientOptions::default();
        $options = $options->withTimeOut(3.5);
        // Use a proxy that all API requests should be passed through.
        $options = $options->withProxy('****** put real proxy here ******');
        $factory = $factory->withHttpClientOptions($options);

        $appCheck = $factory->createAppCheck();
        $appCheckToken = '****** put a token here ******';
        $appCheck->verifyToken($appCheckToken);

When we call verifyToken :

instead of having a timeout after 3.5 seconds, it takes 60 seconds (default Guzzle timeout).
the response is a 504
debug of HttpClient used by JWT::keyIdExists shows no options (no timeout, no proxy)

If we use the same code but change how the Factory create a new Client()
When we call verifyToken :

we have a timeout after 3.5 seconds
if we add the proxy, no more timeout and no more 504 Response. We receive a real response from https://firebaseappcheck.googleapis.com/

Error message/Stack trace
-

Additional information
No response

Thank you for the detailed issue description and for providing a solution, much appreciated! I'll get to it as soon as I can 💪🏻
Out of curiosity: since you are on PHP 8.1, have you considered upgrading the SDK to the current 7.x release?

Thanks if it can help.
By the way, we're currently on kreait/firebase-php 7.5 (sorry for only mentionning firebase/php-jwt v6.8)

🚀 https://github.com/kreait/firebase-php/releases/tag/7.5.1
Let me know if it now works/still doesn't work 🙏🏻

Thanks Jérôme, it works fine. Timeout and proxy are now perfectly used. 👍


SSR redirect to same route without trailing slash returns 302 but no Location header

Environment

Operating System: Darwin
Node Version:     v19.0.1
Nuxt Version:     3.1.1
Nitro Version:    2.0.0
Package Manager<EMAIL_ADDRESS>Builder:          vite
User Config:      modules, vite, runtimeConfig, css, app, hooks, content, i18n
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules:    -

Reproduction
Minimal reproduction middleware that fails to produce a valid HTTP 301 or 302 response during server-side rendering:
// Redirect trailing slashes, preserving query string: /foo/ -> /foo
export default defineNuxtRouteMiddleware((to, from) => {
	if (to.path.length > 1 && to.path.slice(-1) === '/') {
		const path = to.path.replace(/\/+$/, '') || '/'
		return navigateTo({ path, query: to.query, hash: to.hash }, { redirectCode: 301 })
	}
}

Describe the bug
The current Nuxt version (3.1.1) will return an invalid HTTP response during SSR if a redirect to the same route happens within defineNuxtRouteMiddleware. The HTTP response contains a status code of 301 or 302, but no Location header.
Additional context
It seems like the issue is here, where the router plugin uses ufo.isEqual in non-strict mode, which considers /about/ and /about to be the same:
https://github.com/nuxt/nuxt/blob/64b835c70701c8f27debe8a534ec99034bda812b/packages/nuxt/src/app/plugins/router.ts#L251-L256
Logs
❯ curl -vvv http://localhost:3000/about/
*   Trying <IP_ADDRESS>:3000...
* Connected to localhost (<IP_ADDRESS>) port 3000 (#0)
> GET /about/ HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.86.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Access-Control-Allow-Origin: *
< content-type: text/html;charset=utf-8
< x-powered-by: Nuxt
< date: Thu, 26 Jan 2023 16:27:36 GMT
< connection: close
< content-length: 22456
<
<!DOCTYPE html>
<html >
...

Let's track in https://github.com/nuxt/nuxt/issues/18561 because it has a pending PR 🙏


Fix The CustomResourceDefinition Installation Error

Description
Customers recently got the following errors after March 24th.
execution failed for kubernetes-open-source-integration: exit status 1: The CustomResourceDefinition "clusterserviceversions.operators.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Code changes
We think the issue is same as this issue and follow the fix to make this PR.
Tests
Test 2>&1 >/dev/null only keeping stderr in ERROR
xqi@TFXCKKXMP9 Chapter09 % ERROR=$(k create -f gs-ping-deployment.yaml 2>&1 >/dev/null)
xqi@TFXCKKXMP9 Chapter09 % echo $ERROR

xqi@TFXCKKXMP9 Chapter09 % ERROR=$(k create -f gs-ping-deployment.yaml 2>&1 >/dev/null)
xqi@TFXCKKXMP9 Chapter09 % echo $ERROR
Error from server (AlreadyExists): error when creating "gs-ping-deployment.yaml": deployments.apps "gs-ping" already exists
xqi@TFXCKKXMP9 Chapter09 %

We need to make sure that the versions are updated here when the charts are too. Can you please add an item to the release checklist to make sure someone updates this script when updating the version in the nri-bundle?

We need to make sure that the versions are updated here when the charts are too. Can you please add an item to the release checklist to make sure someone updates this script when updating the version in the nri-bundle?

Created this PR, Is there a place specific for the release checklist?
https://github.com/newrelic/helm-charts/pull/1061/files


Page becomes unscrollable after chaging theme in the mobilemenu

What version of starlight are you using?
0.5.2
What version of astro are you using?
2.7.3
What package manager are you using?
npm
What operating system are you using?
win11
What browser are you using?
chrome edge
Describe the Bug
On the starlight document site, if the user chages to another theme in the mobilemenu and immediatly adjusts the window width to more than 50rem, the page will become unscrollable.

Link to Minimal Reproducible Example
No response
Participation

[ ] I am willing to submit a pull request for this issue.

Thanks for the report @justinliu1985 — I can confirm it.
It’s actually not even dependent on changing the theme. Just opening the menu, then resizing the window is enough.


Add-ons compatibility

On the Android version of Firefox, i can not change anything (proxy etc) other than the default configuration of "Bandwidth Hero", because i can not access the add-ons settings page. Other extensions like uBlock Origin works well and their add-ons settings are opened on a new page

Thank you for reporting it!
I'll try to take a look on it.


File t does not exist in model openai/whisper-large-v2

Hi there,
When trying to do the model conversion via Ctranslate2 API (as same setup following as CLI), the conversion failed and popped the error 'File t does not exist in model openai/whisper-large-v2'. And also, when trying to transcribe the audio in GPU mode, it always failed with the 'Cannot execute code, session has been disposed. Please try restarting the Kernel.
The Kernel crashed while executing code in the the current cell or a previous cell. ' Is there any suggestion please?

Hi,
Can you post the code you are using to convert the model?

from ctranslate2.converters import TransformersConverter
converter = TransformersConverter(model_name_or_path='openai/whisper-large-v2',copy_files='tokenizer.json')
converter.convert(output_dir='openai/whisper-large-v2-ct2', quantization='float16')

Hi,
Can you post the code you are using to convert the model?

from ctranslate2.converters import TransformersConverter
converter = TransformersConverter(model_name_or_path='openai/whisper-large-v2',copy_files='tokenizer.json')
converter.convert(output_dir='openai/whisper-large-v2-ct2', quantization='float16')

See the documentation https://opennmt.net/CTranslate2/python/ctranslate2.converters.TransformersConverter.html.
copy_files is a List[str] so you should pass copy_files=['tokenizer.json'].

See the documentation https://opennmt.net/CTranslate2/python/ctranslate2.converters.TransformersConverter.html.
copy_files is a List[str] so you should pass copy_files=['tokenizer.json'].

Great! Thanks a lot. By the way, do you have any idea of my second question that
"when trying to transcribe the audio in GPU mode, it always failed with the 'Cannot execute code, session has been disposed. Please try restarting the Kernel. The Kernel crashed while executing code in the the current cell or a previous cell. '
The CUDA installation is correct as I have double checked. And I can call the APIs on Colab. Just only fail on my local Jupyter..

The error message is generic so I don't really know. Try running the code without Jupyter.


Camera flip option missing

after installing and using the wtf os found that the camera options menu is grayed out and so can no longer access the menu item to flip the camera for a build that has it installed upside down. Whats the fix here besides having to move the actual camera?

WTFOS should not be changing this behavior. The only possible cause of this issue that would relate to WTFOS would be boot ordering caused by MSP-OSD, and even that seems unlikely.
First - this build has a capable camera and was able to have the camera flipped prior to WTFOS install? Please double check.
Next, do you have MSP-OSD installed on the Vista/AU? If so, try uninstalling it? Does this fix it?
Finally, if you boot your air unit / Vista using the WTFOS bypass (bind button held down during boot), does the option come back?


cmake: print a warning if SDL version is between 2.16 and 2.20 included, ref #600

See:

#600

The "Fixed event pump starvation" that improves it is part of SDL2.20, and the "Fixed mouse warping" commit is part of SDL2.16. Meaning that every version of SDL2.x where 16≤x≤20 is known bad.
-- @necessarily-equal

@necessarily-equal I tried that instead, but it doesn't work, I have no idea why…
    if (SDL2_VERSION_STRING VERSION_GREATER_EQUAL "2.16"
        AND SDL_VERSION_STRING VERSION_LESS_EQUAL "2.20")
        message(WARNING "SDL is known to be buggy between version 2.16 and 2.20, see https://github.com/DaemonEngine/Daemon/issues/600")
    endif()

I mean, there is no message VERSION_GREATER_EQUAL isn't supported (it's supported since 3.7 and we require 3.12), the test just doesn't catch the version when I do ≥ and ≤ instead of ≮ or ≯.

In fact SDL2_VERSION_STRING is empty…

The CONFIG keyword had to be added to the find_package call in order to get SDL2_VERSION filled:
-     find_package(SDL2 REQUIRED)
+     find_package(SDL2 REQUIRED CONFIG)

Meh, the CI fails:
CMake Error at CMakeLists.txt:626 (find_package):
  Could not find a package configuration file provided by "SDL2" with any of
  the following names:

    SDL2Config.cmake
    sdl2-config.cmake

  Add the installation prefix of "SDL2" to CMAKE_PREFIX_PATH or set
  "SDL2_DIR" to a directory containing one of the above files.  If "SDL2"
  provides a separate development package or SDK, be sure it has been
  installed.

This is how I do it, the first find_package will not be fatal (QUIET) if CONFIG isn't supported, and thenfind_package is called again with REQUIRED but without doing any unsupported version comparison. This happens if SDL2 can be found but CMake doesn't provide sdl2-config.cmake (or SDL2Config.cmake).
    find_package(SDL2 QUIET CONFIG)
    if(SDL2_FOUND)
        if (SDL2_VERSION VERSION_GREATER_EQUAL "2.16"
            AND SDL2_VERSION VERSION_LESS_EQUAL "2.20")
            message(WARNING "SDL ${SDL2_VERSION} between version 2.16 and 2.20 is known to be buggy, see https://github.com/DaemonEngine/Daemon/issues/600")
        endif()
    else()
        # CMake may be able to find SDL2 without supporting CONFIG
        # If sdl2-config.cmake or SDL2Config.cmake isn't provided.
        find_package(SDL2 REQUIRED)
        message(WARNING "SDL version is unknown, version can't be checked for known bugs")
    endif()

I can't test, but based on the comments I'd think we could actually work around the bug. Something like
diff --git a/src/engine/sys/sdl_input.cpp b/src/engine/sys/sdl_input.cpp
index e8a29a26..48270d49 100644
--- a/src/engine/sys/sdl_input.cpp
+++ b/src/engine/sys/sdl_input.cpp
@@ -1079,7 +1079,7 @@ static void IN_ProcessEvents( bool dropInput )
                                        {
                                                Com_QueueEvent( Util::make_unique<Sys::MousePosEvent>(e.motion.x, e.motion.y) );
                                        }
-                                       else
+                                       else if ( e.motion.xrel != 0 || e.motion.yrel != 0 )
                                        {
                                                Com_QueueEvent( Util::make_unique<Sys::MouseEvent>(e.motion.xrel, e.motion.yrel) );
 #if defined( __linux__ ) || defined( __BSD__ )

This patch filters out mouse deltas with 0 motion. It might be necessary to filter out absolute mouse position events with zero deltas too; someone would need to test that.

That workaround would alleviate the issue quite a bit for SDL2.20 I
think, but doesn't fix it for SDL2.16 and 2.18.
On Sun, Apr 3 2022 at 14:20:04 -0700, slipher
@.***> wrote:

I can't test, but based on the comments I'd think we could actually
work around the bug. Something like
diff --git a/src/engine/sys/sdl_input.cpp
b/src/engine/sys/sdl_input.cpp
index e8a29a26..48270d49 100644
--- a/src/engine/sys/sdl_input.cpp
+++ b/src/engine/sys/sdl_input.cpp
@@ -1079,7 +1079,7 @@ static void IN_ProcessEvents( bool dropInput )
{
Com_QueueEvent(
Util::make_uniqueSys::MousePosEvent(e.motion.x, e.motion.y) );
}

                                  else

                                  else if ( e.motion.xrel != 0

|| e.motion.yrel != 0 )
{
Com_QueueEvent(
Util::make_uniqueSys::MouseEvent(e.motion.xrel, e.motion.yrel) );
#if defined( linux ) || defined( BSD )
This patch filters out mouse deltas with 0 motion. It might be
necessary to filter out absolute mouse position events with zero
deltas too; someone would need to test that.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID:
@.***>

Why not? Going on this:

When in-game, when SDL_GetRelativeMouseMode()==SDL_TRUE, SDL's SDL_PollEvent(&e) gives, SDL_MOUSEMOTION events with both e.motion.xrel==0 and e.motion.yrel==0.

That's after the second commit, the one that kinda fixes it. Before
(between the two related commit) there is basically no input for me.
On Sun, Apr 3 2022 at 14:25:40 -0700, slipher
@.***> wrote:

Why not? Going on this:

When in-game, when SDL_GetRelativeMouseMode()==SDL_TRUE, SDL's
SDL_PollEvent(&e) gives, SDL_MOUSEMOTION events with both
e.motion.xrel==0 and e.motion.yrel==0.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID:
@.***>

Our version of daemon/cmake/FindSDL2.cmake doesn't attempt to find a version number.

Right, if there is no FindSDL2.cmake in our module path or the modules shipped with CMake, the next thing is to check for the config.cmake thing. The sdl-config.cmake from the SDL repository also lacks a version number, so it doesn't help us.

So this is cool and the CI pass, but it seems to print this on Windows, Mac and Linux:
CMake Warning at CMakeLists.txt:636 (message):
  SDL version is unknown, version can't be checked for known bugs

Well, maybe it only checks for version on computers of lucky peoples like me…

I don't think it should work for anyone. The cmake/FindSDL2.cmake script we have does not set a version number. If it appears to work for you, maybe you just set SDL2_VERSION in the variable cache while testing...

@slipher you seems to miss something important: FindSDL2.cmake is NOT expected to set a version number.
The file that sets a version number is sdl2-config.cmake, which is only used when CONFIG option is set to find_package(), so to get a version number, you need that your cmake provides sdl2-config.cmake.
On my end:
$ find /usr/ -name sdl2-config.cmake
/usr/lib/x86_64-linux-gnu/cmake/SDL2/sdl2-config.cmake

That's why I am talking about “luck”, it depends on user's own environment.
My code first tries the CONFIG option and then, if there is an error (for example, there is no sdl2-config.cmake), it tries without CONFIG to rely onFindSDL2.cmake or I don't know why.

The sdl-config.cmake from the SDL repository also lacks a version number, so it doesn't help us.

This, as I said above, is true. But there is a second file that does help us! /usr/lib/x86_64-linux-gnu/cmake/SDL2/sdl2-version-config.cmake (note the version in the filename).
So this will only give us the version number on the subset of Linux distros which ship the version config file. (The binary releases from the SDL website don't include the cmake-config stuff.) But hopefully the mouse bug only occurs on Linux so it will be OK.

We can also raise a warning in source code when trying to compile against a broken SDL version, but given our build isn't warning free that can be missed.

If you change the warning to only print for Linux builds then LGTM

Do you mean the SDL issue only affects Linux systems?

Do you mean the SDL issue only affects Linux systems?

I would guess so but it doesn't matter. On Windows and Mac we provide the precompiled SDL, so no one is using an unapproved version. I don't want there to be useless warnings for every Windows or Mac build.

Ah, I see!

OK, now the SDL version is only checked on Linux and the message telling the version can't be checked is a status and not a warning.


[feature request] Array field support

Let's say you have a field that is named: Requests:
type Requests = Array<{url: string, name: string}>

Is there a way to do something like this natively with the lib?
Of course, there are workarounds using  2 levels of formState:

Create one formState per row and add a global onChange listener to change the formState above
Use raw so you can trigger onChange manually for the above formState - this way you lose local validation

I feel the lib could support these cases natively
Thanks in advance

Hi @melloc01!
I'm not exactly sure what your use case is. If possible, please provide code snippets or a live example to help me better understand this limitations (e.g. codesandbox).
It sounds like  .raw() might be what you need here. It was introduced to handle this type of inputs that can have any shape.
For example, if you provide a custom component that controls Requests, you can treat all data related to Requests as a single unit/input in the form state (even though it can be represented as multiple inputs).
const Requests = ({ onChange, validate, value }) => {
  // maintaining the requests state independently
  const [requests, setRequests] = useState(value);
  const handleChange = useCallback((data) => {
    // update the requests data how like
    onChange(requests); // calling the input's onChange to update formState
  });
  return (
    // logic to render and update the requests fields
  );
}

const Form = () => {
  const [formState, { text, raw }] = useFormState({ requests: [] });
  return (
    <>
      <input {...text('name')} />
      <Requests {...raw('requests')} formState={formState} />
    </>
  );
};

Keep in mind that validation errors can be anything (an object or an array) which will be useful if you want to have multiple errors for a single input.
I hope that helps.

That's cool, solved it.  thanks @wsmd and sorry for the late reply.


feature: diagnostic plots

Addresses #43
I've additionally fleshed out the tests to use fixtures instead. Now, all the model draws, model fits etc., are in tests/fixtures.py, so they are more easily accessible for other test files, in this case test_plots.py.

This is what the plot looks like on the test data

@Nathaniel-Haines Thanks! Yes, I thought the same. I will open a different PR for the docs, as I will add a weights plot too.


Routing wrong default value

Hi.
In https://bcit-ci.github.io/CodeIgniter4/general/routing.html
I use in router
$routes->add('blog/joe', 'Blogs::users/34'); but this show 404 page.
Can you recheck it.
Thanks

Duplicate #352

We've got a number of tests on it. In order to see if it's a bug, I need to know how your controller file is setup. Can you provide code please?

In file controller

<?php namespace Modules\Blog\Controllers\Admin;

use App\Controllers\AdminBaseController;

class BlogController extends AdminBaseController
{
    public function detail($slug = '')
    {
           return 'Ok';
    }
}

In router file.

$routes->group('admin', ['namespace' => 'Modules\Blog\Controllers\Admin'], function($routes)
{
    $routes->get('blog/detail/abc', 'BlogController::detail/dsdsd');
});

I access to link: /admin/blog/detail/abc this show 404
But if change router
$routes->group('admin', ['namespace' => 'Modules\Blog\Controllers\Admin'], function($routes)
{
    $routes->get('blog/detail/(:segment)', 'BlogController::detail/$1');
});

then success link
Thanks

The way you changed it is correct and the way that it's intended to be used.


Return results in JSON format with serde

The purpose of this PR is twofold:

To return results in JSON format instead of printing in mock-json style to console
Return with a result type for all functions instead of just printing the error to console

Currently a lot of the responses can only be viewed in the terminal and not all functions fail on an error, instead just printing out the error and returning. It would also be difficult to use the sample node non-interactively, for example via http responses. Serde_json offers consistent formatting for responses and is already a dependency in the sample node.
The PR has the following advantages:

Responses are in a format that can be parsed more easily. For example if implementing an API, definitive responses on channel info, connected peers etc. can be sent remotely
Better error handling since most errors are now returned as a result rather than just been printed to console
Improved readability, printing JSON brackets no longer necessary
Easier to embed objects in responses, for example sent and received payments are now separate return fields

Mh, I'm not sure we want to go this way. ldk-sample is really meant as an educational example to showcase how the different modules of LDK fit together, i.e., it's primary goal is to be read, even more so than to be run. So the more boilerplate and complexity we add, the more the reader has to parse and understand.
Moreover, I really don't think we want to optimize our logging outputs w.r.t. machine readability. If you need to interact with the code non-interactively, you should specify and build-out some kind of API, not just parse in log messages.

It's actually not returning JSON, but simply the default Debug output of the Rust objects.

I didn't know that, guess I'll have to look more into that. Thanks


stats: add checks for drop reason counters - v3

Previous PR: #1376
Changes from last PR:

rebase
Leaving checks for stream reassembly aside as those were already added by another commit.

Is there a redmine ticket for this ?
Is it just to enhance the existing tests with more checks ?
(Should it be labelled tests pass ? )

Is there a redmine ticket for this ?

Added one: https://redmine.openinfosecfoundation.org/issues/6571

Is it just to enhance the existing tests with more checks ?

Was to add checks for the recently-ish introduced drop_reason counters (https://redmine.openinfosecfoundation.org/issues/6230)

(Should it be labelled tests pass ? )

Likely, though I think I'll rebase now, and also add a ticket number to the commit message, I guess...


reordered meta and added failOnWarning flag

#20

@yegor256  I forgot about this PR for 2 month. Please, have a look.

@rultor merge

@rultor merge

@yegor256 OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@yegor256 Done! FYI, the full log is here (took me 3min)


[Problem]:  job called without bash

Description / Beschreibung
ion is called without bash this will fail
https://github.com/buanet/ioBroker.docker/blame/75f6496ffe9fe24710f8666342014f5e6dc47222/debian12/scripts/iobroker_startup.sh#L326
Image version
v8
Docker logs / Docker Protokoll
/opt/scripts/iobroker_startup.sh: /usr/bin/iob: /usr/bin/bash: bad interpreter: No such file or directory

Full log?
Didn't see that failing. So it would be nice to get some more information about that.

2023-10-10 20:54:53.457874+00:00[DEBUG] Checking custom states db settings...
2023-10-10 20:54:53.457879+00:00[DEBUG] No custom states db settings detected.
2023-10-10 20:54:53.457887+00:00[DEBUG] Done.
2023-10-10 20:54:53.457891+00:002023-10-10T20:54:53.457891491Z
2023-10-10 20:54:53.459625+00:00Checking Database connection... Failed.
2023-10-10 20:54:53.459642+00:00[DEBUG] Error message:
2023-10-10 20:54:53.459652+00:00[DEBUG] /opt/scripts/iobroker_startup.sh: /usr/bin/iob: /usr/bin/bash: bad interpreter: No such file or directory
2023-10-10 20:54:53.459667+00:00Please check your configuration and try again.
2023-10-10 20:54:53.459673+00:00For more information see ioBroker Docker Image Docs (https://docs.buanet.de/iobroker-docker-image/docs).
2023-10-10 20:54:53.459678+00:002023-10-10T20:54:53.459678355Z


[ENH] Handle metadata deletes + fix bugs related to Updates/deletes in the metadata writer

Description of changes
Summarize the changes made by this PR.

Improvements & Bug fixes

Handles metadata deletes
Full text writer adds (token, freq) pair even if freq is 0. Fixes this.
Full text writer does not remove postings list of documents that have been deleted. Fixes this.
Fix for test_query_without_add

Test plan

[x] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes
None

Discussed offline that nothing obvious comes to mind wrt simplifying the metadata merge. At this point, ok to move on but revisit this later and assess if there are possibilities to do this


Protobuf fields should not be checked for nullness

Java codegen for list methods should not generate null checks like this one for proto fields https://github.com/googleapis/java-pubsublite/blob/da98b0f8212d33ace2e3f9d034f7a923ef05c43a/google-cloud-pubsublite/src/main/java/com/google/cloud/pubsublite/v1/stub/AdminServiceStubSettings.java#L187
This is a hard blocker for pulling these libraries into the google codebase which has a compiler error for null checks on protobuf fields. Per https://developers.google.com/protocol-buffers/docs/reference/java-generated, java protobuf fields will never return null.
Environment details

Programming language: Java
OS: All
Language runtime version: All
Package version: All

Steps to reproduce

Write a java client library with a list request
Codegen adds redundant check

@Neenu1995 @chingor13 FYI

This null check originates from the monolith generator. On investigating, this was introduced for a very specific reason, context below.

Original bug
Relevant discussion in the bugfix, where the necessity of this check was initially debated.

I'll add a comment to the code to indicate this check is WAI. Removing this check would introduce a bug in client libraries who do return null instead of an empty list, so closing this issue.

Going to reopen. The bug and discussion refer to com.google.api.services.compute.model.Image, note the model in the name. This refers to the old HTTP generated libraries, not protobuf libraries, indeed you can see it overrides GenericJson not AbstractMessage. Are there still generated clients which use the old HTTP bindings, not the protobuf ones? If so, this check may indeed be necessary for those clients. But if no such libraries are generated (or generatable) with the version at head, this check may safely be removed.

The microgenerator supports a superset of the monolith's features, and it's unclear if there are clients who do the same thing as the use case mentioned in the original bug. I would prefer not to introduce a potential breakage to one of the ~200 client libraries we support, but if you can confirm that none of them return null instead of an empty list (and paste that here), we can revisit making this change.


Improved test_rule

Made the config loading path much closer to that of elstalert
If no rule is specified it will load and run all rules (though not very neatly)
Added a --show-query option to print out the ES query
Raises exceptions, so that error codes can be checked

Can you resolve conflicts please :)

Can you resolve conflicts please :)


RKRequestCachePolicyFromCacheAndNetwork

@jeffarena and @blakewatters :
I'm noodling around RestKit on behalf of a secret project and part of it deals with some modifications to the request cache policy.  I'm wondering if these mods are worth a pull request or not... do you think this has much general appeal for other projects beyond mine?  If so, I can put together a pull request with some unit tests to match.
I'm proposing adding an additional policy to the request cache.  For each cacheable request to a single endpoint, it will return a response from the cache if available and then if the network is reachable it will also fire off a request to the network.  So, if a response is available from the cache and from the network, you'll get two didFinishLoad: calls for the price of one get:.
-------------------------- Code/Network/RKRequest.h ---------------------------
index 2c434ee..34d5977 100644
@@ -62,10 +62,12 @@ typedef enum {
     // Load from the cache if we have data stored
     RKRequestCachePolicyEnabled = 1 << 3,

     // Load from the cache if we are within the timeout window
     RKRequestCachePolicyTimeout = 1 << 4,
+
+    RKRequestCachePolicyFromCacheAndNetwork = 1 << 5 | RKRequestCachePolicyEnabled,

     RKRequestCachePolicyDefault = RKRequestCachePolicyEtag | RKRequestCachePolicyTimeout
 } RKRequestCachePolicy;

 /**

--------------------------- Code/Network/RKRequest.m ---------------------------
index 54f1ed2..43b6a24 100644
@@ -468,10 +468,12 @@ RKRequestMethod RKRequestMethodTypeFromName(NSString *methodName) {
     _sentSynchronously = NO;
     if ([self shouldLoadFromCache]) {
         RKResponse* response = [self loadResponseFromCache];
         _isLoading = YES;
         [self performSelector:@selector(didFinishLoad:) withObject:response afterDelay:0];
+        if ( ([self shouldDispatchRequest]) && (self.cachePolicy & RKRequestCachePolicyFromCacheAndNetwork) )
+            [self fireAsynchronousRequest];
     } else if ([self shouldDispatchRequest]) {
 #if TARGET_OS_IPHONE
         // Background Request Policy support
         UIApplication* app = [UIApplication sharedApplication];
         if (self.backgroundPolicy == RKRequestBackgroundPolicyNone ||

I'm not necessarily opposed to such an addition… what's the use case? Load data quickly and then update the UI again when request completes?

Exactly. Populating the UI quickly and then check for updates on the net.
Although now that I'm thinking about it, my implementation should consider some indication of currentness from the server like etags or timestamps too.
For the etags that you're using, are those free with Rails?  I'm wondering what options we have on the backend that would allow us to take advantage of the work you've already put into that part of the caching.
Sent from my iPhone.
On Feb 9, 2012, at 8:02 AM, Blake<EMAIL_ADDRESS>wrote:

I'm not necessarily opposed to such an addition… what's the use case? Load data quickly and then update the UI again when request completes?

Reply to this email directly or view it on GitHub:
https://github.com/RestKit/RestKit/issues/552#issuecomment-3887740

In the current case, for this first release, the project won't use the RK-UI components (hopefully thereafter, though). But more specifically, the timestamp I mentioned might be some data attribute passed back from a preliminary network request, or in an etag.  As I understand it, it's not really the timestamp of the last network request but rather is the time that this slice of data was actually updated on the server's backend database.  I could be wrong, I'm still figuring this one out...


Question: C++ interop

Issue description
Sorry if this random and unwanted but I noticed you have llvm.C++Interop turned on in a couple of places - are there examples of C++ interop somewhere in-tree?

Hello, could you elaborate on what you are looking for regarding C++ interop? We are currently working on a native API that allows using Elide to dispatch calls from native applications, if that's the intent behind your question.
If you are referring to C++ interop from guest code being evaluated by Elide (e.g. a Python script using a C++ library), there are currently no samples or documentation about it, but we would be happy to hear about your use cases.

I was interested in arbitrary C++ interop between whatever and however. I have since done the deep dive on graal and have figured out how to do it (with just graal). Thanks.

@makslevental Sorry about the delay, GitHub notifications are legendarily bad. Yes, I think plain GraalVM would be better at this. If you haven't seen some of the C interface stuff for GVM (it's buried), that could help too. Feel free to file here or contact us on Discord and we'd be happy to help even if it's not related to Elide directly.
We are shipping a native interop layer soon (specifically, with WorkerD, which is in C++) so we have a lot of lessons which may or may not help.

If you haven't seen some of the C interface stuff for GVM (it's buried)

Yes I've seen it and cribbed to implement https://github.com/makslevental/graalvm_cpp_interop
The principle takeaway (after the deep-dive and some follow-up discussions with graal people) is there's a hard limit on how much bitcode graal can parse because of the 32b limit on largest array size (the C++ interop lazily parses bitcode). This is unworkable for me because I want to bind a very large code base (all of LLVM in fact).

We are shipping a native interop layer soon (specifically, with WorkerD, which is in C++) so we have a lot of lessons which may or may not help.

Not sure what "WorkerD" is but my next attempt (no clue when - day job) will involve just binding/using libclang itself.


Router binds props from nested route to parent component

Version
4.0.12
Reproduction link
jsfiddle.net/k8z1vp4b/
Steps to reproduce
In the repro, there are two nested components: Bar -> Foo.
They both have props: true and both components have an id prop.
Because the :id route parameter is at the 2nd level, I would not expect it to "bleed" on the higher level.
This is opposite of regular dataflow and can cause unexpected issues if your top component has these props for other uses in another context.
This creates a brittle base problem. Adding a new prop in the child component can break its parent if the same props existed there.
What is expected?
props defined in nested routes shouldn't be set on parent component.
What is actually happening?
they are.

I am not really set on whether the opposite should happen or not (a parent route prop being set on a child component).
I suppose that could be understandable as it's the regular dataflow direction? Or not? Or maybe make it an explicit option?
Maybe all options are ok.

yeah, this is intended. props: true is the convenient pass-all option. You can use a function instead to selectively pass anything you want or even use the v-slot API to pass props.

This creates a brittle base problem. Adding a new prop in the child component can break its parent if the same props existed there.

This is true with regular components but not wih page components because they are used in very specific scenarios and most of the time they are only passed to routes.

most of the time they are only passed to routes.

Although the docs explicitly say that props exists to enable their re-use as views and regular component in app.
I agree it's pretty rare. Threw me off for a while though.
(I was also thrown off for a while by the fact that router-view doesn't provide you with your route but a kind of global route object with path being the full matched url. You have to fish into matched[] array but if you consider a structure with multiple nesting levels, it's pretty hard to know which level you're at. Let me know if you want me to open an issue for improvements there.)


Entity framework does not interpret .Equals in the query right

Using .Equals instead of == in a query result in an empty list. Tested in 3.0. Breaking change from 2.1.

Seems like EqualsTranslator is improperly identifying string.Equals(string, StringComparison) and eliding it.

I am going to fix issue just for the awesome post by the OP! 😄


Readme tutorial has diverged from code

Description
The tutorial in README.md assumes different code from what is currently on master.  For example readme assumes:
let viewPage model dispatch =
    match model.Page with
    //...
    | TomatoModel ->
        [ words 60 "Tomatoes taste good!"]

...but actual code is currently:
let view model dispatch =
    div [ Key "Application" ] [
        Menu.view (Logout >> dispatch) model.User
        hr []

        div [ centerStyle "column" ] [
            match model.PageModel with
            | HomePageModel ->
                yield Home.view ()
            | LoginModel m ->
                yield Login.view m (LoginMsg >> dispatch)
            | WishListModel m ->
                yield WishList.view m (WishListMsg >> dispatch)
        ]
    ]

Repro steps

Clone master.
Follow tutorial up to step 2.

Expected behavior
Tutorial matches code.
Actual behavior
Tutorial does not match code.
Known workarounds
N/A
Related information
I'd offer to fix this myself, but as I'm following the tutorial in order to understand the code it's a bit of a Catch 22.

I'm having a go at fixing this myself today. Will submit a PR if it goes well.

PR: https://github.com/SAFE-Stack/SAFE-BookStore/pull/382


Oracle Dataset

Hi, thank you for the great work!
Can you share the oracle extractive and abstractive compressions you used for the experiment? Would appreciate it!
Thanks!

Hi there! I have uploaded the prompts for oracle compressions for FLAN-UL2 for NQ, TQA, HotpotQA in the respective directory here.

Thanks so much!


Added basic color control

Added basic support to color control, default to original solution
plot(
    [[1, 2, 3, 5, 7, 2, 3, 4], [4, 2, 2, 5, 1, 2, 2, 4]],
    color=["red", "blue"],
    legend_labels=["test", "test2"],
)

Looks great, thank you!

Is now live as v0.13.0


Base SHA and Head SHA empty

Hello,
I'm having a weird situation when using the action
When I run it on ubuntu-latest runner it works perfectly
When I run it on a self hosted runner or any docker image, it runs, but never returns the Base SHA and Head SHA
get-affected-projects:
    runs-on: ["ubuntu-latest"]
    container:
      image: ${{ vars.CONTAINER_PUBLIC_REGISTRY }}/node:16.14.2-alpine
      credentials:
        username: ${{secrets.CONTAINER_PUBLIC_REGISTRY_USERNAME}}
        password: ${{secrets.CONTAINER_PUBLIC_REGISTRY_TOKEN}}
 steps:
      - name: checkout project
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: install dependencies
        run: |
          apk --update add bash make git jq
      - name: Set Directory as Safe
        run: |
          git config --system --add safe.directory "$GITHUB_WORKSPACE"
      - name: Get SHAs
        uses<EMAIL_ADDRESS>
And this is the return I get

has any one faced this before ? do you have any clues, advices ? I've been struggling for a week now
Thank you

I'm closing this as I fixed my issue finally

@fourass : A big thank to you for sharing your solution.


Avoid ever building rustc_codegen_spirv more than once in release mode.

This PR allows these 3 commands to share the same rustc_codegen_spirv build (this also means spirv-tools-sys also only compiles all the C++ code only once, which can itself take a significant amount of time):
# Examples (via build dependency).
cargo build --release -p example-runner-wgpu

# Old tests (via direct dependency).
cargo test --release -p spirv-builder

# New tests (also direct dependency, but also implicitly release mode).
cargo compiletest

Sadly, I'm not aware of relevant automation (though one could already exist), but here's what I found:

between regular dependencies and build dependencies, features aren't always unified, and these differed:

syn (used by serde_derive -> serde -> rustc_codegen_spirv)
libc (used by jobserver -> cc -> spirv-tools-sys -> rustc_codegen_spirv)
hashbrown (used by indexmap -> rustc_codegen_spirv)
num_traits (used by rspirv -> rustc_codegen_spirv)

tests/Cargo.toml (cargo compiletest) didn't have the system of use-{installed,compiled}-tools features that spirv-builder has, resulting in an extra --cfg 'feature="default"' passed when building rustc_codegen_spirv
our [profile.release.build-override] contains codegen-units = 16, and that is the default for release mode except the actual default is baked into rustc, so Cargo doesn't actually normally pass it in release mode

I'm not really happy with just fixing it right now, we should try to come up with a test for "did we waste time building everything some things twice" (e.g. checking the target dir for multiple copies of some crates?).
But I don't even think we build anything (other than cargo compiletest) in release mode on CI.
We did try release mode CI in #494, but it took longer and I now think it might be the duplication.
Since we're already running cargo compiletest on CI, I suspect that if we land this PR we should be able to move to release mode, and see a benefit. In fact I might just test that myself once we have data for this PR itself.

A silly way to enforce this doesn't regress would be to hash librustc_codegen_spirv.so when first built, then require any further commands to preserve the hash. Not sure if it makes sense without --release in CI (a la #547), though.


Revise API error codes

Issue Number
ADP-291
Overview

f533091070c2ccc7945ffff2143f3b6c4f06599c
Revise error codes:  - 404 when wallet not found  - 403 when transaction is too big

430a31f88b5fa4d3632f4e5a53e4afa9a69bae32
Update swagger

e90b26430eaf90532966a92504af89f338fa89fc
Update integration tests

Comments
Whie reviewing https://github.com/input-output-hk/cardano-wallet/pull/2258 I noticed that some error codes are a bit inconsistent, e.g.:

410 or 404 is returned when the wallet is not found.
400 or 403 - when transaction is too big.

This pr is about to revise it and:

return 404 on wallet not found (removing 410 altogether)
return 403 on transaction too big

Did you consider dropping 415 altogether?

I haven't actually. As far as I see there are some tests against 415 in the test suite, so maybe let's have it here.

bors r+

bors r-

bors r+


fourth try

Checklist

[x] I ran npm test locally and it passed without errors.

[x] I only edited the _data/pixels.json file.

[x] I entered the username in the pixels.json that I'm also using to create this pull request.
[x] I acknowledge that all my contributions will be made under the project's license.

I'm closing this since your pixel has already been merged :) thank you!


TypeError: admin.firestore.FieldPath is not a constructor in v11+ in Emulator mode only

[READ] Step 1: Are you in the right place?
I hope so. This might be a problem with the Emulator, not sure...
[REQUIRED] Step 2: Describe your environment

Operating System version: macOS Monterey 12.6 (also happened on Ubuntu machines)
Firebase SDK version: 11.12.1
Firebase Product: functions
Node.js version: 16.18.1
NPM version: 8.19.2

[REQUIRED] Step 3: Describe the problem
Starting with version 11 of the admin SDK creation of an admin.firestore.FieldPath object does throw an error (in Emulator only).
The call still works when deployed to Firebase.
Error happening:
⚠  functions: TypeError: admin.firestore.FieldPath is not a constructor

Steps to reproduce:
Just call new admin.firestore.FieldPath("test") in a firebase function in emulator environment.
A detailed reproduction with a minimal example is displayed here:
https://github.com/nerdlibfront/firebase-admin-fieldpath-problem
This leads to:
⚠  functions: TypeError: admin.firestore.FieldPath is not a constructor
    at /###/functions/lib/index.js:11:18
    at cloudFunction (/###/functions/node_modules/firebase-functions/lib/cloud-functions.js:78:23)
    at /usr/local/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:516:16
    at runFunction (/usr/local/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:504:15)
    at runBackground (/usr/local/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:515:11)
    at processBackground (/usr/local/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:499:11)
    at /usr/local/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:690:27
    at Layer.handle [as handle_request] (/usr/local/lib/node_modules/firebase-tools/node_modules/express/lib/router/layer.js:95:5)
    at next (/usr/local/lib/node_modules/firebase-tools/node_modules/express/lib/router/route.js:137:13)
    at next (/usr/local/lib/node_modules/firebase-tools/node_modules/express/lib/router/route.js:131:14)

Relevant Code:
import * as functions from "firebase-functions";
import * as admin from "firebase-admin";

admin.initializeApp();

export const onTesting = functions
    .firestore
    .document("/test/{testId}")
    .onWrite(async (change) => {
      const path = new admin.firestore.FieldPath("test");
      const item = await admin.firestore().collection("test")
          .where(path, "==", "test")
          .get();
      console.log(item.docs.length);
    });

Error only happens in emulator, not on deployed functions.
Downgrading to firebase-admin 10 solves the problem.

Please see my comment on https://github.com/firebase/firebase-admin-node/issues/1973#issuecomment-1314323312

We recommend developers to use the new modular entry points to access the various APIs of the SDK, as opposed to using the global admin namespace.

Please try the modular entry points instead of using the global admin namespace.


Fix detection of seeding peers

wire.peerPieces.length is always undefined, which meant the updateSeedStatus function was always returning early, and peers were never being detected as seeds.

Released as 0.98.11


Documentation: expand description Api Reference text for imod.formats.ipf.save

The IPF file is written based on information in the pandas.DataFrame.  Therefore some dataframe columns are required.

Text suggestion:

df (pandas.DataFrame) – DataFrame containing the data to write. Columns required for X ("x") and Y ("y") coordinates.

in case of itype = 1

1 or "timeseries".  Columns requires with temporal information ("time")

in case of itype = 2.

2 or "borehole1d".  Columns required for vertical reference ("top")

in case of itype = 3.

3 or "cpt".  Columns required for vertical reference ("top")

in case of itype = 4.

4 or "borehole3d".  Columns required "x_offset", "y_offset", "top"

And what about similar column requirements for imod.formats.ipf.write?

in addition:
"top" is defined as the individual top of each Interface described in a single borelog.  E.g.
3
2,2
top,1e+20
lithology,1e+20
-1.9,"C"
-2.3,"P"
-4.0,"C"


Debug recent travis consistent failure

In the last few days, the master travis runs have been consistently failing with simulator crashes running the an unchanged RTDB test - FIRMutableDataTests.

Sent #2923 for reverting #2904

Fixes rolled into #2924 and #2926


Support for classes(), etc

This came up in #118. We've found it easier to work with tests that use the assertion library's comparison functions, rather than the comparison functions like hasClass, hasProp, etc.
I'd like to propose adding helpers that are more like html() and text() that simply return a consumable form of the element feature requested. IE, classes() would return an array of class names on the element. I've got an implementation of classes and can submit a PR for it and others that make sense, but wanted to see if it made sense before going further.

I agree. Value assertions are more useful than boolean assertions.
We could replace hasClass, hasAttribute, and hasProp with classes, attributes, and props.
We've got a good opportunity to remove these before the official release.
The one method I think is useful is hasStyle, because it allows you to check hasStyle('color', 'white') instead of returning {color: 'rgb(255, 255, 255)}
What do people think?

I haven't had much occasion to use attributes or props, but I'd be for including them - props seems like it might be handy. classes is solved in #118 , and should probably be renamed classes in light of the updated functionality.
I agree that hasStyle should stay, I have used  it quite a lot (changing button colors, opacities, and the such). I don't see a styles helper being very useful.

I prefer personnaly prefer the notation getClasses() but other than that i like it too.

I also prefer the notation getClasses.
What do other people think?

The getX syntax seems different from the conventions used in this library already (html, text).
Alternatively, it could go fully the other way and implement the functions as getters, so they would expose a property-like API expect(wrapper.classes).to.eql(['foo', 'bar'])

I've changed my mind, as you said we already have html and text methods.

This has been released in 1.0.0-beta.6
Happy to continue discussing API, but thinking about it I think:
wrapper.props().someProp

Is more elegant that
wrapper.getProps().someProp

Plus we can extend the props method by adding an optional param:
wrapper.props('someProp')


Can't write to subindex zero of register 0x1003 Pre-defined error field

It is my understanding that this should be possible, and doing it should reset the error counter.
I also found this piece of code that suggests this has been thought of at some point.
If I try to do this write, the function above never gets executed, instead CO_SDO_AB_READONLY error is thrown here
This happens even though OD entry for this register in initialised like this with attribute of 0x8E and never changes. (Note that CO_ODA_READABLE == 0x04)
What happens is that when the object for this registered is fetched from the OD the attribute member is populated with this function that makes sure that subindex zero is never written to.

I'm currently short with my time. I have to update and test index 0x1003 more precisely. I plan to do this in ten days.

There was a problem with suggested fix, as described in comment to #15.
However, it is fixed now.
Thanks


update ci.yml

Fixes #3

Hey @gunjanmimo, thx for this PR.
I think you borrowed the ci.yml from Meshery project. Since this is an adapter, please use the ci.yml from meshery-istio.

@gunjanmimo quick check - were you able to update with a new commit here?


QuaRot: add weight RTN [WIP]

Tested PPL for A16W4 with and without rotation:
Llama-2 7B

with rotation: 7.17 vs 6.76 in paper
without rotation: 8.65 vs 6.99 in paper

Llama-2 13B

with rotation: 5.63 vs 5.48 in paper
without rotation: 21.26 vs 6.32 in paper [very anomalous, should at least be better than 7B result]

Key code differences to fake_quant: gpu implementation of grid search for weight clipping -> halves time for applying weight RTN in 7 (30s) and 13B (1min) models compared to spcl fake_quant, however untested on 70B and may cause GPU OOM. The previous commit has the equivalent fake_quant more memory efficient method. However, what is very strange is that this takes 10mins on 7B whereas this takes only 1min in spcl repo.
@sashkboos could you have a look at e.g. this PR/branch at the "add weight clipping" commit and see if you can spot anything in the rtn logic that could be causing these accuracy differences? The other possibility is that there is a bug in the "no rotation" logic, and that we're actually doing things to the network that we're not meant to be doing, causing big diffs here.

Thanks @nailimixaM
Before diving into the performance issue, I would like to understand the difference between these numbers and the results in the paper. Do you follow the same arguments (including clipping) as the paper experiments?

Thanks @nailimixaM
Before diving into the performance issue, I would like to understand the difference between these numbers and the results in the paper. Do you follow the same arguments (including clipping) as the paper experiments?

Yes -> symmetric, perchannel and weight clipping all set to true.

With bugfix in clamping:
Llama-2 7B

with rotation: 6.83 vs 6.76 in paper
without rotation: 8.07 vs 6.99 in paper

Llama-2 13B

with rotation: 5.59 vs 5.48 in paper
without rotation: 13.00 vs 6.32 in paper [still very anomalous]

With bugfix in clamping:
Llama-2 7B

with rotation: 6.83 vs 6.76 in paper
without rotation: 8.07 vs 6.99 in paper

Llama-2 13B

with rotation: 5.59 vs 5.48 in paper
without rotation: 13.00 vs 6.32 in paper [still very anomalous]

I suspect the discrepancy in the no-rotation ppls is the fact that I was evaluating quantization on a fused-but-not-rotated model here, whereas in fake_quant this is done on the raw Llama model. The QuaRot MLP and Attn modules I have written assume that the model has fused layernorms. To do no rotation quantization emulation would require some work to either add new odules or modify the QuaRot ones with a switch specifying whether layernorms have been fused or not.

I suspect the discrepancy in the no-rotation ppls is the fact that I was evaluating quantization on a fused-but-not-rotated model here, whereas in fake_quant this is done on the raw Llama model. The QuaRot MLP and Attn modules I have written assume that the model has fused layernorms. To do no rotation quantization emulation would require some work to either add new modules or to modify the QuaRot ones with a switch specifying whether layernorms have been fused or not.

Makes sense. So that's why you got higher ppl values as your fused model has worse distribution which makes sense to me  :-)

Fixed without rotation. Results for A16W4 are now:
Llama-2 7B

with rotation: 6.84 vs 6.76 in paper
without rotation: 7.00 vs 6.99 in paper

Llama-2 13B

with rotation: 5.59 vs 5.48 in paper
without rotation: 6.33 vs 6.32 in paper

The ~0.1ppl discrepancies with rotation have me a bit stumped but at this point I think we can crack on with KV-cache and activation quantization to get end-to-end RTN working. If large discrepancies appear in these it could be worth searching further.


bst build bails if it cannot push to the artifact cache

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Sep 6, 2017, 19:36
It seems that if an artifact push-url is configured and buildstream cannot reach it or push to it; bst build will just error out.
This seems wrong considering that we cache the offline state in places and such.
If we cannot push we should probably warn about it at startup and construct the scheduler without a push queue.

In GitLab by [Gitlab user @jjardon] on Sep 6, 2017, 19:37
marked this issue as related to baserock/definitions#13

In GitLab by [Gitlab user @cs-shadow] on Sep 12, 2017, 14:57
mentioned in merge request !89

In GitLab by [Gitlab user @tristanvb] on Sep 12, 2017, 15:41
closed via merge request !89


Module build failed: TypeError: Cannot read property 'newLine' of undefined

Bug Report or Feature Request (mark with an x)

[x] bug report -> please search issues before submitting
[ ] feature request

Versions.

@angular/cli: 1.0.0-rc.2
node: 6.10.0
os: linux x64
npm version: 3.10.10
Repro steps.

npm uninstall -g @angular/cli
npm cache clean
npm install -g @angular/cli@latest
ng serve

The log given by the failure.

ERROR in ./src/main.ts
Module build failed: TypeError: Cannot read property 'newLine' of undefined
    at Object.getNewLineCharacter (/home/bunyamin/WebstormProjects/c3dp/node_modules/typescript/lib/typescript.js:8062:20)
    at Object.createCompilerHost (/home/bunyamin/WebstormProjects/c3dp/node_modules/typescript/lib/typescript.js:44978:26)
    at Object.ngcLoader (/home/bunyamin/WebstormProjects/c3dp/node_modules/@ngtools/webpack/src/loader.js:202:31)
 @ multi webpack-dev-server/client?http://localhost:4200 ./src/main.ts

Desired functionality.

It was an existing project created by older version of angular-cli. It was working well before updating angular-cli , node and npm. After updates, it produced the error above by just typing ng serve.
Mention any other details that might be useful.

There is a similar issue which is closed (#3781) but I couldn't find a solution from it.

even I am getting the same error.

Dupe of #3781, you can find an answer in https://github.com/angular/angular-cli/issues/3781#issuecomment-285020354.
I also suggest the RC update guide: https://github.com/angular/angular-cli/wiki/stories-rc-update


Unable to create v25 containers "Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions..."

PLEASE DO NOT INCLUDE ANY PASSWORDS OR TOKENS IN YOUR ISSUE!!!
Describe the issue
Unable to create new containers.
Tried creating onprem and sandbox containers.
Tried both process and hyper-v isolation.
Tried versions 25.0, 25.1 and next minor version.
Created a version 24.5 container just now and working fine.
Errors from Event Log:
1.
Application: Microsoft.Dynamics.Nav.Server.exe
CoreCLR Version: 8.0.324.11423
.NET Version: 8.0.3
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
File name: 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60'
at Microsoft.Dynamics.Nav.WindowsServices.DynamicsNavServer.Main(String[] args)

Faulting application name: Microsoft.Dynamics.Nav.Server.exe, version: 25.0.26346.0, time stamp: 0x66e80000
Faulting module name: KERNELBASE.dll, version: 10.0.20348.2340, time stamp: 0xf746f0b2
Exception code: 0xe0434352
Fault offset: 0x000000000003f19c
Faulting process id: 0xa5c
Faulting application start time: 0x01db305349a0c686
Faulting application path: C:\Program Files\Microsoft Dynamics NAV\250\Service\Microsoft.Dynamics.Nav.Server.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: 7c5c6853-e2bc-492f-a6e0-dfe421c3e51a
Faulting package full name:
Faulting package-relative application ID:
Scripts used to create container and cause the issue
$containerName = 'bc250'
$password = 'P@ssw0rd'
$securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force
$credential = New-Object pscredential 'admin', $securePassword
$auth = 'UserPassword'
$artifactUrl = Get-BcArtifactUrl -storageAccount 'bcinsider' -type 'Sandbox' -country 'w1' -select 'SecondToLastMajor' -accept_insiderEula
$licenseFile = 'C:\temp\partner.bclicense'
New-BcContainer `
    -accept_eula `
    -containerName $containerName `
    -credential $credential `
    -auth $auth `
    -accept_insiderEula `
    -artifactUrl $artifactUrl `
    -includeTestToolkit `
    -includePerformanceToolkit `
    -assignPremiumPlan `
    -licenseFile $licenseFile `
    -dns '<IP_ADDRESS>' `
    -isolation 'hyperv' `
    -memoryLimit 12G `
    -updateHosts

Full output of scripts
BcContainerHelper is version 6.0.15
BcContainerHelper is running as administrator
HyperV is Enabled
UsePsSession is True
UsePwshForBc24 is True
Host is Microsoft Windows 11 Enterprise - 10.0.26100.2033
Docker Client Version is 27.3.1
Docker Server Version is 27.3.1
Removing Desktop shortcuts
Fetching all docker images
Fetching all docker volumes
Using image mcr.microsoft.com/businesscentral:ltsc2022
Creating Container bc250
Style: sandbox
Multitenant: Yes
Version: 25.2.26400.0
Platform: 25.0.26346.0
Generic Tag: <IP_ADDRESS>
Container OS Version: 10.0.20348.2340 (ltsc2022)
Host OS Version: 10.0.26100.2033 (Unknown/Insider build)
Using hyperv isolation
Using locale en-US
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file C:\temp\partner.bclicense
Files in C:\ProgramData\BcContainerHelper\Extensions\bc250\my:
- AdditionalOutput.ps1
- HelperFunctions.ps1
- license.bclicense
- MainLoop.ps1
- SetupNavUsers.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container bc250 from image mcr.microsoft.com/businesscentral:ltsc2022
1b4634372754e4b93baeb5ff2ee74f127d824b93ec98ea8fbbd64b73fb4e8d42
Waiting for container bc250 to be ready
Using artifactUrl https://bcinsider-fvh2ekdjecfjd6gk.b02.azurefd.net/sandbox/25.2.26400.0/w1
Using installer from C:\Run\240
Installing Business Central: multitenant=True, installOnly=False, filesOnly=False, includeTestToolkit=False, includeTestLibrariesOnly=False, includeTestFrameworkOnly=False, includePerformanceToolkit=False, appArtifactPath=c:\dl\sandbox\25.2.26400.0\w1, platformArtifactPath=c:\dl\sandbox\25.2.26400.0\platform, databasePath=c:\dl\sandbox\25.2.26400.0\w1\BusinessCentral-W1.bak, licenseFilePath=c:\dl\sandbox\25.2.26400.0\w1\Cronus.bclicense, rebootContainer=True
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
c:\dl\sandbox\25.2.26400.0\platform\ServiceTier\Program Files
c:\dl\sandbox\25.2.26400.0\platform\ServiceTier\System64Folder
Copying Web Client Files
c:\dl\sandbox\25.2.26400.0\platform\WebClient\Microsoft Dynamics NAV
Copying ModernDev Files
c:\dl\sandbox\25.2.26400.0\platform
c:\dl\sandbox\25.2.26400.0\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\sandbox\25.2.26400.0\platform\ConfigurationPackages
Copying Test Assemblies
C:\dl\sandbox\25.2.26400.0\platform\Test Assemblies
Copying Extensions
C:\dl\sandbox\25.2.26400.0\w1\Extensions
Copying Applications
C:\dl\sandbox\25.2.26400.0\platform\Applications
Copying dependencies
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS
Exporting Application to CRONUS
Removing Application from tenant
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Service 'Dynamics 365 Business Central Server [BC] (MicrosoftDynamicsNavServer$BC)' cannot be started due to the following error: Cannot start service MicrosoftDynamicsNavServer$BC on computer '.'.
at <ScriptBlock>, C:\Run\navinstall.ps1: line 311
at <ScriptBlock>, C:\Run\start.ps1: line 246
at <ScriptBlock>, <No file>: line 1Error
Using artifactUrl https://bcinsider-fvh2ekdjecfjd6gk.b02.azurefd.net/sandbox/25.2.26400.0/w1
Using installer from C:\Run\240
Installing Business Central: multitenant=True, installOnly=False, filesOnly=False, includeTestToolkit=False, includeTestLibrariesOnly=False, includeTestFrameworkOnly=False, includePerformanceToolkit=False, appArtifactPath=c:\dl\sandbox\25.2.26400.0\w1, platformArtifactPath=c:\dl\sandbox\25.2.26400.0\platform, databasePath=c:\dl\sandbox\25.2.26400.0\w1\BusinessCentral-W1.bak, licenseFilePath=c:\dl\sandbox\25.2.26400.0\w1\Cronus.bclicense, rebootContainer=True
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
c:\dl\sandbox\25.2.26400.0\platform\ServiceTier\Program Files
c:\dl\sandbox\25.2.26400.0\platform\ServiceTier\System64Folder
Copying Web Client Files
c:\dl\sandbox\25.2.26400.0\platform\WebClient\Microsoft Dynamics NAV
Copying ModernDev Files
c:\dl\sandbox\25.2.26400.0\platform
c:\dl\sandbox\25.2.26400.0\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\sandbox\25.2.26400.0\platform\ConfigurationPackages
Copying Test Assemblies
C:\dl\sandbox\25.2.26400.0\platform\Test Assemblies
Copying Extensions
C:\dl\sandbox\25.2.26400.0\w1\Extensions
Copying Applications
C:\dl\sandbox\25.2.26400.0\platform\Applications
Copying dependencies
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS
Exporting Application to CRONUS
Removing Application from tenant
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Service 'Dynamics 365 Business Central Server [BC] (MicrosoftDynamicsNavServer$BC)' cannot be started due to the following error: Cannot start service MicrosoftDynamicsNavServer$BC on computer '.'.
at <ScriptBlock>, C:\Run\navinstall.ps1: line 311
at <ScriptBlock>, C:\Run\start.ps1: line 246
at <ScriptBlock>, <No file>: line 1
New-BcContainer Telemetry Correlation Id: 047ee489-0dc5-4ddb-bb53-d804e0b2ba5b
Initialization of container bc250 failed
At C:\Users\EwaldVenter\OneDrive - Braintree by Vox Telecom (Pty) Ltd\Documents\WindowsPowerShell\Modules\BcContainerHelper\6.0.15\ContainerHandling\Wait-NavContainerReady.ps1:42 char:17
+ ...             throw "Initialization of container $containerName failed" ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Initialization of container bc250 failed:String) [], RuntimeException
    + FullyQualifiedErrorId : Initialization of container bc250 failed

Screenshots
If applicable, add screenshots to help explain your problem.
Additional context

does it happen all the time? Yes, seemingly from today, I think
did it use to work? Had a version 25.0 that stopped working for obscure reason, complained about "Database transaction detected". Then when I deleted it and recreated it, this started happening.

Please update to the latest version of ContainerHelper and use the latest version of the generic image (using -alwayspull when creating the container)

Got a running container now, thank you. Is it possible to add the alwayspull option as part of the (http://aka.ms/bcdockerscript) script?
Still failed on a missing Business Foundation Test app, but got something to work with now, thank you.

The latest BcContainerHelper should actually check whether you are using the latest generic.


How to add HTML tag type="text/javascript" for scripts in final bundle?

Version
4.5.10
Environment info
vue-cli-service build

Steps to reproduce
Now vue-cli creates bundle without HTML tag type="text/javascript". How can this be configured?
What is expected?
Flexible configuration of the final bundle
What is actually happening?
Cant add tag type

I'll leave a sample configuration here. It might help someone.
module.exports = {
  chainWebpack: config => {
    config
      .plugin('html')
      .tap(args => {
        args[0].minify.removeScriptTypeAttributes = false
        return args
      })
  }
}

But why?
Per MDN:

Omitted or a JavaScript MIME type: This indicates the script is JavaScript. The HTML5 specification urges authors to omit the attribute rather than provide a redundant MIME type.

@sodatea , yes its true. But I faced a problem with Microsoft Edge and sometimes with Firefox as well.

That's likely due to a misconfiguration in the server (wrong Content-Type in the response header).
It's better to be fixed on the server side… otherwise, you might run into other problems again in the future.

my solution was adding the --no-module while building for production.
"build": "vue-cli-service build --no-module"


Fix several mixed parameter native APIs

In cases where the command language leveraged mixed types, the auto-generated API is "unknown".  The fix is to treat these as *argv, **argv. Convert :ref: to the misspelled calculator_functions target into a :doc: reference.

Updated version to 0.3.0.
Imported new .xml export from EnSight builds (including API fixes for several native mixed mode functions)
Change calculator docs functions to :doc: reference.


sync local SHA1 after new online

#1509

Hi,
I guess this will work, however it adds on top of the real problem, https://github.com/larshp/abapGit/issues/1485
We need to investigate what the SHA1 is really used for. I guess it can be removed from persistence, and if so #1509 will probably be solved automatically

Ok, any tipps how to start the investigation?

no, just hard work 😄


Publish on npm?

Is it possible to get this plugin published on npm? That would make it a lot easier to keep track of updates to the plugin. Thanks!

@alexbuijs I took the liberty to publish it on npm myself:
https://www.npmjs.com/package/FilePicker-Phonegap-iOS-Plugin

@MadMaxedy can you send a PR with the package.json?

It's probably a good idea to change the cordova id in package.json and config.xml to 'FilePicker-Phonegap-iOS-Plugin' as well (or better yet, change them and the npm name to something like 'cordova-plugin-filepicker-ios' ;-). Now cordova tries to add the plugin to the project on every prepare or build:
Discovered plugin "FilePicker-Phonegap-iOS-Plugin" in config.xml. Adding it to the project
Fetching plugin<EMAIL_ADDRESS>via npm
Plugin "com.jcesarmobile.filepicker" already installed on ios.

@MadMaxedy I've published it as cordova-plugin-filepicker, can you delete yours?
https://www.npmjs.com/package/cordova-plugin-filepicker
I didn't put ios on it because I plan to also add Android implementation.
And also, I plan to update the plugin to use the new iOS 11 pickers as current ones become deprecated.


Make fn optional

I have a top-level <SWRConfig/> which specifies my fetcher function, so I only need to specify the key for useSWRNative to work. However, because fn is required here, Typescript still asks for the fetcher function. This request makes fn optional in light of SWRConfig.

Could you line this up with master, which is updated for v1?

Done. Released in latest version
yarn add swr-react-native@beta


bug: inner table cannot have 'defs'

when inner table have 'defs' structure in the table object, the 'defs' is not passed properly.
it should fix the type structure internally to use TableProps without partial structure to avoid the situation.

the behavior does not show. it was mistake.


 However, we added it to the partner app but it was still just as slow: https://github.com/lensterxyz/lenster/pull/1358#issuecomment-1343363557

No longer building
Context: The motivation to add that to XMTP.chat was to speed up a partner app. However, we added it to the partner app but it was still just as slow: https://github.com/lensterxyz/lenster/pull/1358#issuecomment-1343363557


Fix #9806 small issue in scrollbar component

This fix the issue #9806 [Small issue with scroll position]. Since the scrollTop is always 0 when the element is display: none, we need to update the visibility of the element before we update the scrollTop.

Please add specs to this.

No response to request to add specs, closing.


SSL Certificate_Verify_Failed: On Mac Machine

[ERROR:certstream] 2020-06-04 01:17:14,112 - Error connecting to CertStream - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108) - Sleeping for a few seconds and trying again...
Source code:
import logging
import sys
import datetime
import certstream

def print_callback(message, context):
    logging.debug("Message -> {}".format(message))

    if message['message_type'] == "heartbeat":
        return

    if message['message_type'] == "certificate_update":
        all_domains = message['data']['leaf_cert']['all_domains']

        if len(all_domains) == 0:
            domain = "NULL"
        else:
            domain = all_domains[0]

        sys.stdout.write(u"[{}] {} (SAN: {})\n".format(datetime.datetime.now().strftime('%m/%d/%y %H:%M:%S'), domain, ", ".join(message['data']['leaf_cert']['all_domains'][1:])))
        sys.stdout.flush()

if __name__ == '__main__':
    logging.basicConfig(format='[%(levelname)s:%(name)s] %(asctime)s - %(message)s', level=logging.INFO)
    certstream.listen_for_events(print_callback, url='wss://certstream.calidog.io/')

Hmm interesting, this should work fine - I wonder if your local trust store is having issues? Please reopen if you still think this is an issue.


Unable to run unity-builder from an organization

Problem Description:
When running unity builder from a PR in an organization repository, unity-builder action cannot find the origin remote.
Error Message:
##[warning]fatal: ambiguous argument 'origin/continuous-integration': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

##[error]Failed to run "git rev-list --count origin/continuous-integration".
 In-command error caught: Error: The process '/usr/bin/git' failed with exit code 128

Entire job log:
 ##[warning]
      Library folder does not exist.
      Consider setting up caching to speed up your workflow,
      if this is not your first build.

##[warning]From https://github.com/rubin-studios/card-game
 * [new branch]      develop    -> origin/develop
 * [new branch]      master     -> origin/master

/bin/sh
0
0

##[warning]fatal: ambiguous argument 'origin/continuous-integration': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

##[error]Failed to run "git rev-list --count origin/continuous-integration".
 In-command error caught: Error: The process '/usr/bin/git' failed with exit code 128

Desired Behaviour:
The unity-builder action should be able to run from an organization repository, within a pull request from a fork.

Here is the YAML
# This is a basic workflow to help you get started with Actions

name: CI

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
  push:
    branches: [ master, develop ]
  pull_request:
    branches: [ master, develop ]

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2

    # Runs a single command using the runners shell
    - name: Super-Linter
      uses<EMAIL_ADDRESS>
    - name: Unity - Builder
      uses<EMAIL_ADDRESS>      env:
        UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
      with:
        unityVersion: 2019.4.0f1
        targetPlatform: StandaloneWindows64 # Build a Windows 64-bit standalone.

@JMLegere thank you for reporting this issue.
This may have been fixed in https://github.com/webbertakken/unity-builder/pull/105
Could you try if unity-builder@master gives you the same behaviour?

This may be a different problem than the one this issue is discussing.
After switching the unity-builder to @master, I am getting a different error. I have both the UNITY_LICENSE secret on the forked repo, and the organization repo.
/usr/bin/docker run --workdir /github/workspace --rm --env UNITY_LICENSE --env UNITY_LICENSE_FILE --env UNITY_EMAIL --env UNITY_PASSWORD --env UNITY_SERIAL --env UNITY_VERSION=2019.4.0f1 --env PROJECT_PATH=. --env BUILD_TARGET=StandaloneWindows64 --env BUILD_NAME=StandaloneWindows64 --env BUILD_PATH=build/StandaloneWindows64 --env BUILD_FILE=StandaloneWindows64.exe --env BUILD_METHOD= --env VERSION=<IP_ADDRESS> --env CUSTOM_PARAMETERS= --env HOME=/github/home --env GITHUB_REF --env GITHUB_SHA --env GITHUB_REPOSITORY --env GITHUB_ACTOR --env GITHUB_WORKFLOW --env GITHUB_HEAD_REF --env GITHUB_BASE_REF --env GITHUB_EVENT_NAME --env GITHUB_WORKSPACE=/github/workspace --env GITHUB_ACTION --env GITHUB_EVENT_PATH --env RUNNER_OS --env RUNNER_TOOL_CACHE --env RUNNER_TEMP --env RUNNER_WORKSPACE --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/runner/work/_temp/_github_home:/github/home --volume /home/runner/work/_temp/_github_workflow:/github/workflow --volume /home/runner/work/card-game/card-game:/github/workspace unity-builder:2019.4.0f1-windows
License activation strategy could not be determined.

Visit https://github.com/webbertakken/unity-builder#usage for more
details on how to set up one of the possible activation strategies.
##[error]The process '/usr/bin/docker' failed with exit code 1

This is indeed a different error, that is indicating that UNITY_LICENSE is in fact not set.
You can find the relevant code in activate.sh of unity-builder. Likely the secrets.UNITY_LICENSE is misspelled or for some reason giving undefined. To solve this problem you may want to read deeper in to how secrets work in unity actions. They may have everything to do with non-protected branches not receiving the contents of secrets or conceptually similar nuances.

Thank you for your help. I have found the issue.
In the YAML file above, my UNITY_LICENSE env variable was declared in the wrong place.


applications supply packet buffers (rather than through the alloc_packet callback)

Up until now, we have had quicly_packet_allocator that is used for allocating memory for storing a packet image, associated with the source and destination addresses. While conceptually clean, this design has had issues:

Unnecessary use of alloc - free. Typically, applications call quicy_send then immediately submits the UDP packets being built to the kernel. In such case, freedom for releasing things in arbitrary order (what heap allocation provides) is unnecessary and just costly.
Unfriendliness with GSO. The prerequisite of GSO is that all packets contain the same 4-tuple. As the 4-tuple tuple contained in each packet generated by quicly can be different, applications are required to consult the 4-tuple of each packet and route them to the correct socket (or underlying network interface). Also, it was hard to build all the UDP packet images in a contiguous range of memory. That resulted in the kernel doing gather operation when using GSO.

This PR changes the approach.
It eliminates the alloc_packet callback, and instead requires application to supply the buffer in which UDP packet images should be built, when calling functions like quicly_send.
In master, quicly_send function looks like:
/**
 * Builds UDP datagrams to be sent for given connection.
 * @param [out] packets         vector of `quicly_datagram_t *`, each of the elements containing the 4-tuple and the datagram
 *                              payload. Each `quicly_datagram_t *` is allocated and freed using the alloc_packet callback provided
 *                              through `quicly_context_t`.
 * @param [in,out] num_packets  Upon entry, the application provides the number of entries that the `packets` vector can contain.
 *                              Upon return, contains the number of packet vectors emitted by `quicly_send`.
 * @return 0 if successful, otherwise an error. When an error is returned, the caller must call `quicly_close` to discard the
 *         connection context.
 */
int quicly_send(quicly_conn_t *conn, quicly_datagram_t **packets, size_t *num_packets);

With the proposed change, it becomes like:
/**
 * Builds UDP datagrams to be sent for given connection.
 * @param [out] dest            destination address
 * @param [out] src             source address
 * @param [out] packets         vector of iovecs pointing to the payloads of UDP datagrams. Each iovec represens a single UDP
 *                              datagram.
 * @param [in,out] num_packets  Upon entry, the application provides the number of entries that the `packets` vector can contain.
 *                              Upon return, contains the number of packet vectors emitted by `quicly_send`.
 * @param buf                   buffer used for building UDP datagrams. It is guaranteed that the first datagram would be built from
 *                              the address provided by `buf`, and that succeeding packets (if any) will be contiguously laid out.
 *                              This constraint reduces the number of vectors that need to be passed to the kernel when using GSO.
 * @return 0 if successful, otherwise an error. When an error is returned, the caller must call `quicly_close` to discard the
 *         connection context.
 */
int quicly_send(quicly_conn_t *conn, quicly_address_t *dest, quicly_address_t *src, struct iovec *packets, size_t *num_packets,
                void *buf, size_t bufsize);

Other functions that emit packets (e.g., quicly_send_version_negotiation, quicly_send_retry) are also modified so that they would receive a buffer to which the UDP payloads are written.

FWIW, with this PR, I see 7% reduction in "CPU time spent per Mbps" metric.

@AloysAugustin I think we want to merge this PR, but wanted to ping you as it is my understanding that you use a custom alloc_packet callback.
As stated at the top, I think our current approach is suboptimal and inadequate as it requires per-packet routing. You might not have noticed this, because at the moment, quicly does not talk to multiple addresses for each connection. But we'd soon start doing that, as we add support for migration.
With this PR, we'd have a contractual guarantee that series of packets returned by a single call to quicly_send can be routed as one big block contained in a contiguous buffer. I think that this is a good simplification and will provide us better performance, reduce maintenance cost.
That said, I understand that there might be something missing in the current form of this PR. For example, this PR builds a series of UDP packet bodies (i.e., excluding IP and UDP header) in a contiguous chunk of memory supplied by the user. Some might want a callback for building packets with appropriate headers. I am eager to discuss what is missing and adding them as necessary.
Thank you in advance.

Hi @kazuho
Jumping in on this discussion, I would +1 for merging this PR. Even if some things are missing I think it will improve the current behavior. In our use-case that'll help removing one layer of memcpy for most packets.
Few thought for potential future improvements :
It might be worth re-introducing some kind of packet allocator that would be called when exhausting the current buffer. This would allow to build chained buffers without having to make a full quicly_send call.
On that specific use-case another open question we have is the handling of 'border packets'. If we want to finish filling the first buffer before switching to the next one, is there a way to write a datagram to two distinct buffers ?
Concerning callbacks for writing UDP packet headers, I don't have a clear opinion. I think the GSO approach is to write them as late as possible, but there might be use-cases I'm not seeing here.
Thanks for your help
Cheers

@sknat Thank you for sharing your thoughts. It's good to know that you could remove memcpy once this PR gets adopted.

Few thought for potential future improvements :
It might be worth re-introducing some kind of packet allocator that would be called when exhausting the current buffer. This would allow to build chained buffers without having to make a full quicly_send call.

We could do that, though I would like to understand why it would be beneficial.
Having such an optimization would make sense if it is going to be used often. But if it is going to be used often, why not supply a larger buffer instead?
In our case, each of our event loop would be doing quicly_send -> sendmsg using at most 64KB buffer. 64KB per CPU core is a tiny number compared to the sum of the sizes of the send windows of connections governed by that event loop.
I would also point out that applications should be reading ingress packets frequently rather than trying to build long list of packets, as building long list of packets without reading worsens the latency, therefore having negative impact on performance once loss occurs. My assumption has been that supplying at most ~64KB buffer to quicly_send would work well even on 10gbe networks, and if the size of the buffer is as large as that, I'd assume that making one more call to quicly_send would not be a cost. Though numbers might prove me wrong.

Concerning callbacks for writing UDP packet headers, I don't have a clear opinion. I think the GSO approach is to write them as late as possible, but there might be use-cases I'm not seeing here.

Right, I think the question here is more about kernel-bypass. My thinking was that when doing kernel-bypass, it might be the case that you want to build IP /UDP headers yourself, enforce alignment on each packet being sent, etc.

Hi Kazuho,
Thanks for reaching out to us, I would also +1 this PR.
My only question would be about the size of the generated datagrams, can we assume that all (or all but the last) datagram have the same length?

Hi Kazuho,
Thanks for reaching out to us, I would also +1 this PR.
My only question would be about the size of the generated datagrams, can we assume that all (or all but the last) datagram have the same length?

@AloysAugustin Thank you for checking. Glad to know that you'd be happy to see this happen.

My only question would be about the size of the generated datagrams, can we assume that all (or all but the last) datagram have the same length?

That's a good point. At the moment we have not had that contract. Having that would simplify things; I've created #333 that accomplishes that.


fix $ref bug

Problem
Currently, any properties named $ref get treated as references, and cause errors because their value is usually { type: 'string' }. This could potentially be a problem with any dynamic-key field (such as headers or scopes, but technically not components.schemas since $ is not an allowed character)
Solution
This PR adds a workaround which keeps track of whether the recursive function is inside a properties field

if you can bump the version of the packages as well that'd be great! i can help publish if needbe but i also think u might be able to run npm publish


Update setup docs

Type of PR
Docs 📖
Description
I tried my best to update the "Manual Configuration" docs with what the installer produces. There's many pitfalls here, e.g. providing all the sorts of jsbundling configs the installer provides, but I'd rather not go down that road now 😁
Closes #642
Checklist

[x] My code follows the style guidelines of this project
[x] Checks (StandardRB & Prettier-Standard) are passing

Touched up some things to account for the new installer 🙌🏼


Scala Core: Drop Scala 2.10

Hey @szareiangm! Is there any necessity for this? I don't have hard feelings for 2.10, but still IMO it has zero-cost here.


Adding a new article in Capacity Planning

"How Back Market SREs prepared for Black Friday" is an article relating an SRE approach to Capacity Planning for a Black Friday in an electronic marketplace.
BTW, thanks for this repository, I've found some jewels ! 🙂

That's great! Thanks for your contribution @thylong! :)


sqlite: 3.28.0 -> 3.30.0, sqlite-analyzer: 3.28.0 -> 3.30.0, resolves #70127 and #68583

replaces #65402
Motivation for this change

sqlite 3.28.0 has a security vulnerability regarding a a division by zero  which is fixed in 3.30.0. See #70127, #68583 and https://nvd.nist.gov/vuln/detail/CVE-2019-16168 for details
Changelog: https://www.sqlite.org/changes.html

Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Notify maintainers
cc @eelco @np

I have no idea what the failing checks mean.

@GrahamcOfBorg build sqlite sqlite-analyzer

I ran the compilation for both on NixOS, and both succeeded. Also tested that this indeed fixes the issue using the snippet from https://www.sqlite.org/src/info/e4598ecbdd18bd82945f6029013296690e719a62 .
Pinging @edolstra as @eelco is non-existent.

@cko: given that nobody has come around yet, I'd suggest to rebase and push force again, that should retrigger ofBorg.

@d-goldin that's a good idea, thank you! Done.

Poke security team: @fpletz @domenkozar @grahamc

@GrahamcOfBorg build pythonPackages.apsw python3Packages.sqlalchemy
Testing these two because I saw build failures when I applied this to release-19.09, not sure if master has the same problem.

@ivan: Thanks for pointing that out.  I briefly looked around and it also does not seem like there is an issue filed with sqlalchemy yet, so it might take a little while. Also because the failure actually looks like some more significant change has happened.
I don't think this issue is too high severity, but since we're at it, maybe we can go with a backported fix like in: https://github.com/d-goldin/nixpkgs/commit/3958e0220e4352bee0507bd24f6e2b4275a423a8
I ran sqlalchemy and apsw against it, and it seemed to pass the tests fine.

One commit per package please.

This seems somewhat stalled a bit. Should I open a PR just for SQLite with the backported fix instead?

Actually, 61017517a34cfd7c542ba8b3a3d19ed8c2c120da went in so closing.

@FRidh: Alright, missed that one. Keep in mind that it might have breakages with sqlalchemy, as outlined above.

Yep, I pushed a fix for sqlalchemy.

@FRidh: Ah, great that there is a fix a vailable upstream by now. Is the plan to backport the bumped version + sqlalchemy fix to release-19.09 too, or might a backport be more suitable for that?

I don't know. I kind of hoped we would not have to backport a sqlite version bump

@FRidh: I agree, that would be not overly stable releasy. I propose the backport PR listed above my comment.

FWIW sqlite 3.30.1 has been released! \o/
In my own testing/usage there were no new failures and the existing problems re:sqlalchemy and APSW are still present.

FWIW sqlite 3.30.1 has been released! \o/
In my own testing/usage there were no new failures and the existing problems re:sqlalchemy and APSW are still present.

Could you open a PR? I think we can keep that one for the next iteration. Most things have already been built now with 3.30.0.


Restart the WS connection if no pongs are received

With this change, the server will actually respond to ping frames with a corresponding pong frame.
We keep track of the outstanding pings on the client side and we restart the connection if we have more than 3 outstanding pings.
The pongs are sent back automatically by the fastwebsockets library, but to do so, the WebsocketTunnelRead struct needs to have access to the WriteHalf of the WS connection, in order to send back the pongs. So we now keep the WriteHalf in an Arc<Mutex<_>>.
The PingState that tracks the sent pings and received pongs is also shared between the WriteHalf and ReadHalf.

Hello,
Thank you for the PR, but sadly there is no chance for this to get merged.
Having to take a lock everytime we need to send websocket message, is a no-go for me.
Not telling you that this is going to be merged, if it changes, but here some comments:

You can remove the Mutex around PingSate if you use AtomicU8 internally, which will be better thant a lock
You can avoid the mutex around the writer half if you use a channel to transmit when it needs to send pong to the writer tasks
Also you are using async mutex, where it is correct for the Writer half, you would have been better with a sync mutex for the PingState https://docs.rs/tokio/latest/tokio/sync/struct.Mutex.html#which-kind-of-mutex-should-you-use

Ok, fair point. I will see if I can implement it without the locking.
Currently wstunnel always hangs for me when my laptop comes out of suspend, and I need to manually restart the service for it to work again, so I wanted to have a good way to be able to detect connection loss and restart the connection.

I had a shot at implementing this without locking, I pushed a new commit.

finish the DNS PR first, I will review this one after :wink:

Going to check it next week, don't have a big chunk of spare of time those coming days

Hi back,
I add some time to look more into it, and indeed the change is not trivial.
First, we need to disable the auto_pong ws.set_auto_pong(false) because it forces to know that we need to send a pong in the frame_reader which cause issue with the borrow.
It will allow handling the ping here, where self is already a &mut instead of doing shenanigan to handle it in frame_reader

We need to add a way for the WebsocketTunnelRead and WebsocketTunnelWriter to exchange information together.
Using a Arc with some atomic for the ping drift, and a channel to the ping payload.
When the read part receives a pong, it decrements the ping_drift, and when it receives a ping, it sends in the channel the ping payload.

I only have added an atomic in the shared state, but it would need to get a channel too.

Add in tunnel writer a method that return a future that you can await in the big select!

Thanks for your feedback. I'm leaving on holidays for 2 weeks and didn't have time to look at this again yet. I will get back to it once I'm back from holidays.

I need to refacto a bit the stuff, but this commit should do the stuff https://github.com/erebe/wstunnel/commit/f55643550b16ccbb6e8a1211a53654256e83c082


Branch stats

@jeeyi heads up I'm going to merge that soon.
I've added JSON support and put the filter as one of the default supported filters. Since it is a common format, this may lead to some JSON files to be processed unexpectedly. I think it make sense to have it enabled by default similarly to what we do for other types but it could be a slight annoyance to reconfigure the filter types in the CLI calls. Thoughts?