Error: BotFrameworkAdapter.processActivity(): 401 ERROR

Versions
botbuilder: 4.9.2 (Latest)
node: 12.18.2 (Latest)
OS: Ubuntu (Docker-Container)
Describe the bug
One of a sudden adapter.processActivity fails with these two error messages when sending a chat-message to the bot:
1. Error Message:
JwtTokenExtractor.getIdentity:err! ReferenceError: XMLHttpRequest is not defined dev_server        |     at /var/huforce/node_modules/whatwg-fetch/dist/fetch.umd.js:484:17 ... dev_server        |     at /var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:14:71 dev_server        |     at new Promise (<anonymous>) dev_server        |     at __awaiter (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:10:12) dev_server        |     at OpenIdMetadata.getKey (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:30:16) dev_server        |     at JwtTokenExtractor.<anonymous> (/var/huforce/node_modules/botframework-connector/lib/auth/jwtTokenExtractor.js:86:56)
2. Error Message:
(node:18) UnhandledPromiseRejectionWarning: Error: BotFrameworkAdapter.processActivity(): 401 ERROR dev_server        |  ReferenceError: XMLHttpRequest is not defined dev_server        |     at /var/huforce/node_modules/whatwg-fetch/dist/fetch.umd.js:484:17 dev_server        |     at new Promise (<anonymous>) dev_server        |     at fetch (/var/huforce/node_modules/whatwg-fetch/dist/fetch.umd.js:477:12) dev_server        |     at OpenIdMetadata.<anonymous> (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:54:31) dev_server        |     at Generator.next (<anonymous>) dev_server        |     at /var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:14:71 dev_server        |     at new Promise (<anonymous>) dev_server        |     at __awaiter (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:10:12) dev_server        |     at OpenIdMetadata.refreshCache (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:53:16) dev_server        |     at OpenIdMetadata.<anonymous> (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:34:32) dev_server        |     at Generator.next (<anonymous>) dev_server        |     at /var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:14:71 dev_server        |     at new Promise (<anonymous>) dev_server        |     at __awaiter (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:10:12) dev_server        |     at OpenIdMetadata.getKey (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:30:16) dev_server        |     at JwtTokenExtractor.<anonymous> (/var/huforce/node_modules/botframework-connector/lib/auth/jwtTokenExtractor.js:86:56) dev_server        |     at BotFrameworkAdapter.<anonymous> (/var/huforce/node_modules/botbuilder/lib/botFrameworkAdapter.js:674:27) dev_server        |     at Generator.throw (<anonymous>) dev_server        |     at rejected (/var/huforce/node_modules/botbuilder/lib/botFrameworkAdapter.js:12:65) dev_server        |     at processTicksAndRejections (internal/process/task_queues.js:97:5)
Details
I created my bot via the App-Studio. I don't have an azure subscription. I neither use the emulator. The bot runs locally. I am using ngrok to tunnel the requests to my PC.
From one second to another this error occurred. At the same time I experienced another issue with our Graph-API. Microsoft returns two public keys instead of 3: https://login.microsoftonline.com/ff14e898-7782-407d-a7ea-8fc44c902168/discovery/keys. I don't know if these issues correlate.
What I have tried

I tried various versions of your (awesome) package
I checked the client-id & the client-secret
I created a new client-secret
I verified my client-id & my client-secret
I checked my messaging endpoint
I verified the system time
I created a new bot via the App Studio and the https://dev.botframework.com

Code:
`
...
const bot = new Bot();
export const adapter = new BotFrameworkAdapter({
appId: msteams.CLIENT_ID,
appPassword: msteams.CLIENT_SECRET
});
MicrosoftAppCredentials.trustServiceUrl(
"https://smba.trafficmanager.net/emea/"
);
...
expressApp.post("/bot", secure((req, res) => {
adapter.processActivity(req, res, async context => {  //  <--  error
await bot.run(context);
});
}));
`
I am desperate, because we need to deliver a new version of our bot. But, I just can't fix this issue.
I really appreciate any help.
[bug]

@anusharavindrar could you take a look at this?

I have the same issue. Any ideas?

@janandreschweiger, I will attempt a repro on my end and update the thread with my findings. Also, do you find any additional errors on the channels tab in dev.botframework.com?
@geckgecko, are you also getting the same error messages when trying to communicate with the bot created via App Studio?

@anusharavindrar No there are neither error messages nor warnings.
Thanks for your support!

@janandreschweiger, thanks for the above info. Also, any specific reason behind using such a setup and not use something like offline-directline if you don't want to have an Azure subscription? Can you elaborate more on what you are trying to accomplish and are you following any specific sample or are building a new bot from scratch as per your requirements?

@anusharavindrar the bot is deployed to the servers of our clients. The bot is running inside a docker container and needs a public ip. In order to simulate a production environment we use ngrok to tunnel the requests to our local PC (private ip). Our clients have a public ip and want to host the bot themselves.

Today something strange happened. I have an old (cached) docker container from my bot. This version works well. But when running the same old code in a new docker the error occurs. I tried different variants of docker-images. I can't fix this issue. I tried different combinations of package versions and docker images (node versions). Installing the xmlhttprequest manually didn't solve the issue either.

@anusharavindrar yes I get the exact same error message. I have created the my bot via the botframework webpage. I experience this error after adding this package: https://www.npmjs.com/package/cld3-asm. However, removing the package doesn't resolve the issue.
I used these samples https://github.com/microsoft/BotBuilder-Samples/tree/master/samples/javascript_nodejs for creating my bot.
If you need any more information or any help just let me know. Thank you!!!

Ok, I now know how to fix the error. But it is extremely strange.
Error:
`
...
const expressApp = express();
import botbuilder from "botbuilder";
const { BotFrameworkAdapter } = botbuilder;
...
`
No Error:
`
...
import botbuilder from "botbuilder";
const { BotFrameworkAdapter } = botbuilder;
const expressApp = express();
...
`
Why on earth do I have to import the botbuilder before initializing express? Can someone please explain this to me.

@geckgecko Thank you! This solved my issue.
@anusharavindrar Do you have any idea why botbuilder throws an error?
This should be fixed or at least mentioned in the docs.

@janandreschweiger I totally agree on this.

@geckgecko & @janandreschweiger, glad to hear that the above method helped to fix the issue.
@geckgecko, can you confirm that you didn't get any error before adding the cld3-asm package?
It does look like that botbuilder and express are having some kind of overlap or conflict which is throwing up these errors. It is recommended that you make use of just 'restify' like in our samples and give it a try. Also, can you try downgrading the botbuilder version and see if you still get the error?

@anusharavindrar Yes I can confirm. However, I removed this package (also from the cache) and I still got the error.
I downgraded the botbuilder to 4.8.0, makes no difference. Unfortunately, we can't move to restify, as we have graphql (/graphql), file-serving (/serve) and our bot (/bot) on the same server.

@geckgecko same. I use express, because I am hosting different services on the same server.
@anusharavindrar I'd appreciate it, if you could take a closer look on this. It is very counterintuitive that botbuilder needs to be imported before initializing express. Now I have to combine the express & my bot into a single file.

@geckgecko @janandreschweiger, thanks for the above info. I will update the thread soon with my findings.

@janandreschweiger
@geckgecko
I can try to help determine the cause of this issue. I can't reproduce it yet so I will need your help with experimentation to narrow down the possible factors.

Does the error occur when you use Emulator instead of Teams?
If so, does the error occur when the bot has no app ID or password?
Does the error occur when you don't use a Docker container?
Does the error occur when you take secure out of your request handler?

Here is my index.js if you'd like to start there and see if you can get the error using the same code as me:
const dotenv = require('dotenv');
const path = require('path');
const express = require('express');
const app = express();

// Import required bot services.
// See https://aka.ms/bot-services to learn more about the different parts of a bot.
const { BotFrameworkAdapter } = require('botbuilder');

// This bot's main dialog.
const { EchoBot } = require('./bot');

// Import required bot configuration.
const ENV_FILE = path.join(__dirname, '.env');
const PORT = 3978;

dotenv.config({ path: ENV_FILE });

app.listen(PORT);

console.log(`Listening on ${PORT}...`);

// Create adapter.
// See https://aka.ms/about-bot-adapter to learn more about how bots work.
const adapter = new BotFrameworkAdapter({
	appId: process.env.MicrosoftAppId,
	appPassword: process.env.MicrosoftAppPassword
});

// Catch-all for errors.
adapter.onTurnError = async (context, error) => {
	// This check writes out errors to console log .vs. app insights.
	console.error(`\n [onTurnError]: ${error}`);
	// Send a message to the user
	await context.sendActivity(`Oops. Something went wrong!`);
};

// Create the main dialog.
const bot = new EchoBot();

// Listen for incoming requests.
app.post('/api/messages', (req, res) => {
	adapter.processActivity(req, res, async (context) => {
		// Route to main dialog.
		await bot.run(context);
	});
});

@janandreschweiger - Are you still working on this?

I am closing this issue due to inactivity. We are still happy to provide help if you need any, though it may be a good idea to make a new post if that's the case. Please have a look at this to find out the best places to ask for help, and consider asking questions on Stack Overflow.


AttributeError: 'NoneType' object has no attribute 'strip'

Hello, I have been using this beautiful application for some time, but after about 2 months I get this error "AttributeError: 'NoneType' object has no attribute 'strip'" I reinstalled it and I still have this error, any suggestions on how to fix it?

I have the same problem here


Encrypt wazuh-secrets.yml once templated

@MaxBed4d you don't need to make PRs for your own branches :slightly_smiling_face:

@MaxBed4d you don't need to make PRs for your own branches 🙂

The changes here were to achieve a different goal from the other branch, however this branch relies and builds on the changes from the other branch.
The two branches felt like two separate PRs, but this could not be merged into main before the other, so to prevent that, I thought that merging this into the other branch was a better idea.

@MaxBed4d you don't need to make PRs for your own branches 🙂

The changes here were to achieve a different goal from the other branch, however this branch relies and builds on the changes from the other branch.
The two branches felt like two separate PRs, but this could not be merged into main before the other, so to prevent that, I thought that merging this into the other branch was a better idea.

In the future, one PR for "Wazuh deployment script improvements" with two commits would be fine. I also don't really see how this one relies on the other, they wouldn't conflict.


[e2e]: generic schedule main multi-uses slsa3

Repo: https://github.com/slsa-framework/example-package/tree/main
Run: https://github.com/slsa-framework/example-package/actions/runs/3020136646
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.generic.schedule.main.multi-uses.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.generic.schedule.main.multi-uses.slsa3.yml
Trigger: schedule
Branch: main
Date: Fri Sep  9 05:22:41 UTC 2022

Repo: https://github.com/slsa-framework/example-package/tree/main
Run: https://github.com/slsa-framework/example-package/actions/runs/3020136646
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.generic.schedule.main.multi-uses.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.generic.schedule.main.multi-uses.slsa3.yml
Trigger: schedule
Branch: main
Date: Fri Sep  9 05:23:47 UTC 2022

Repo: https://github.com/slsa-framework/example-package/tree/main
Run: https://github.com/slsa-framework/example-package/actions/runs/3026799441
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.generic.schedule.main.multi-uses.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.generic.schedule.main.multi-uses.slsa3.yml
Trigger: schedule
Branch: main
Date: Sat Sep 10 05:22:31 UTC 2022
Tests are passing now. Closing this issue.


Use readonly input form to show the validated OpenID URI

there is any screenshot?

No screenshot. Imagine another input field, like the others.
Just you cannot change it.

As I'm now on a computer with easier-to-take-screenshotter, here you go:

@strk Speaking of screenshotters, use Kazam on linux. It is the best one.

LGTM

LGTM


Delta 3 genesis files.

Updating files for Delta-3 including 34553 -> 35000 port change and new known IPs.

bors r+


Prototype Forth VM for filling Awkward Arrays from Uproot.

@ianna This is a conceptual start on the array-filling interface from Uproot to Awkward Array that we talked about last Tuesday.
As a reminder, the motivation is to try to keep Uproot and Awkward Array decoupled. I've made mistakes in the past where Uproot has used Awkward implementation details, which then broke when the wrong versions of Uproot and Awkward Array were used together. Although the new Uproot 4 uses only Awkward's public API, next year's project, in which Uproot offloads ROOT TBaskets to Awkward Array for filling, gets dangerously close to too much coupling. The prototype (scikit-hep/uproot4#96) puts explicit knowledge of how TTrees are serialized in Awkward Array. (In the following, 0x40000000 and 2-byte headers are ROOT details, which should stay in Uproot, not Awkward Array.)
https://github.com/scikit-hep/awkward-1.0/blob/257b24602c343e8420c30499ade40bb7bce1c429/src/libawkward/io/uproot.cpp#L64-L91
That sort of thing should be exclusively contained within Uproot.
So the idea I've been thinking about for decoupling them is to have Uproot create a sequence of ROOT-deserialization instructions that Awkward Array can follow in compiled code. "Instructions," in a general sense, is "code"—Uproot has to send Awkward Array a program. Since the motivation for this is performance (currently, Uproot is generating these instructions as Python code, which it executes using Python's exec), it wouldn't help much for these "instructions" to be something that has to be interpreted (after all, Python is interpreted by a C program, the Python interpreter). Ideally, we might like to just-in-time compile these instructions, but Awkward Array doesn't come with a built-in compiler.
That's why I've been thinking about this idea of using Forth. Forth is an extremely simple language—it has less syntax than LISP!—and it's a half-step above assembly language, a half-step below a conventional language like C or Fortran. The following is a great introduction:
https://skilldrick.github.io/easyforth/
Instead of having proper functions with arguments, it has a single stack and "words" that manipulate that stack. It's an untyped language in the sense that the stack has only one type: integers (typically 16-bit, but we'll be using 64-bit integers).
Here's how I'm thinking it should work: Uproot determines what kind of Form it needs to fill and how to traverse the ROOT TBaskets to fill them, in the form of a Forth program. Uproot sends the raw TBaskets and the Form to Awkward Array. Awkward Array makes GrowableBuffers for each node in the Form, compiles the Forth program, and runs it. Our version of the Forth language includes builtin words for reading from the TBasket data, possibly putting these values on the stack (as 64-bit integers), or putting them into the GrowableBuffers (as whatever types they need to be). Awkward Array then assembles the array from the GrowableBuffers and the Form and sends it back to Uproot.
Here's why Forth seems like a good option:

Whereas declarative or functional languages endeavor to be side-effect free, Forth operates entirely through side-effects. Each word pulls a value off of the TBasket (updating its seek position in-place), pushes or pops values on the stack (for scratch-work), or pushes a value onto one of the GrowableBuffers. Everything we want to do is a side-effect, because our main interest here is I/O. Forth seems like a better fit to this task than a declarative/functional language.
Forth is simple enough to minimize overheads when evaluating it. Documentation about Forth calls it an "interpreted" language, but these words can be mapped 1:1 onto integer instructions for a simple (stack-based) virtual machine. I would call that "compiled," in the same sense as Java, but for an extremely simple machine like NumExpr. The hot loop is a double while loop over a dictionary, which can be implemented as a std::vector<std::vector<int>>. Here's what it looks like in Python:

https://github.com/scikit-hep/awkward-1.0/blob/447d4d6a5622796e0a9fc4ffc6cc921aa2f94898/studies/awkward-forth/virtual-machine.py#L158-L189
The if ... else if can become a switch statement, which the compiler knows how to optimize. The which and where are stacks of integers. With this minimal runtime environment, we can compile and run programs that only use builtin words:
>>> vm.do("3 2 +", verbose=True)
instruction          | stack before instruction
---------------------+-------------------------
3                    | <- top
2                    | 3 <- top
+                    | 3 2 <- top
                     | 5 <- top

Programs that define and use a function:
>>> vm.do(": foo 3 2 + ; foo", verbose=True)
instruction          | stack before instruction
---------------------+-------------------------
foo                  |
  3                  | <- top
  2                  | 3 <- top
  +                  | 3 2 <- top
                     | 5 <- top

Nested function definitions:
>>> vm.do(": foo : bar 1 + ; 3 2 + ; foo bar", verbose=True)
instruction          | stack before instruction
---------------------+-------------------------
foo                  |
  3                  | <- top
  2                  | 3 <- top
  +                  | 3 2 <- top
bar                  |
  1                  | 5 <- top
  +                  | 5 1 <- top
                     | 6 <- top

And recursion (I haven't defined an if builtin word yet, so there's no way to stop the recursion gracefully):
>>> vm.do(": foo + foo ; 1 2 3 4 5 foo", verbose=True)
instruction          | stack before instruction
---------------------+-------------------------
1                    | <- top
2                    | 1 <- top
3                    | 1 2 <- top
4                    | 1 2 3 <- top
5                    | 1 2 3 4 <- top
foo                  |
  +                  | 1 2 3 4 5 <- top
  foo                |
    +                | 1 2 3 9 <- top
    foo              |
      +              | 1 2 12 <- top
      foo            |
        +            | 1 14 <- top
        foo          |
          +          | 15 <- top
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 160, in do
  File "<string>", line 112, in run
  File "<string>", line 20, in pop
ValueError: stack underflow

The rest is adding builtin words to make this language usable for ROOT I/O. I'm going to develop this prototype by testing it with complex data from Uproot, to make sure that it has enough builtin words to parse data like that. When it's clear that it will be capable of all the use-cases Uproot requires, we can think about ways to optimize it in C++.

Interesting ROOT files to try parsing with Forth:
>>> import uproot
>>> import skhep_testdata

>>> # A good place to start.
>>> vector_of_vectors = uproot.open(skhep_testdata.data_path("uproot-vectorVectorDouble.root"))["t/x"]
>>> vector_of_vectors.array().tolist()
[[], [[], []], [[10.0], [], [10.0, 20.0]], [[20.0, -21.0, -22.0]], [[200.0], [-201.0], [202.0]]]

>>> # Everything in this tree is an interesting data structure.
>>> stl_containers = uproot.open(skhep_testdata.data_path("uproot-stl_containers.root"))["tree"]

>>> # A nested record with non-trivial stuff in it that isn't overwhelmingly large.
>>> f = uproot.open(skhep_testdata.data_path("uproot-issue46.root"))
>>> f["tree/evt"].array().type
100 * Event["ArrayBool": 10 * bool, "Bool": bool, "N": int64, "SliceBool": var * bool, "StlVecBool": var * bool]

# Like the stl_containers, but a bit larger.
>>> f = uproot.open(skhep_testdata.data_path("uproot-issue243-new.root"))
>>> f["triggerList/triggerMap.first"].array().type
10252 * var * string
>>> f["triggerList/triggerMap.second"].array().type
10252 * var * float64

>>> # None of the following have been interpreted correctly by the "interpret" function.
>>> # Same with uproot-issue434.root, though it looks like Uproot 3 managed it?
>>> f = uproot.open(skhep_testdata.data_path("uproot-issue124.root"))
>>> f["KM3NET_TIMESLICE"].keys()
['KM3NET_TIMESLICE',
 'KM3NET_TIMESLICE/KM3NETDAQ::JDAQPreamble',
 'KM3NET_TIMESLICE/KM3NETDAQ::JDAQTimesliceHeader',
 'KM3NET_TIMESLICE/vector<KM3NETDAQ::JDAQSuperFrame>']

For some whopping huge examples (performance scaling), there's the std::vector^N<float> from the CHEP 2019 studies. I think I have a copy. Otherwise, they need to be regenerated from the code in this directory.

Note to self: this is how to get the raw data for the Forth interpreter to work on.
>>> vector_of_vectors.basket(0)
<TBasket 0 of 'x' at 0x7843507860d0>
>>> vector_of_vectors.basket(0).data
array([ 64,   0,   0,   6,   0,   9,   0,   0,   0,   0,  64,   0,   0,
        14,   0,   9,   0,   0,   0,   2,   0,   0,   0,   0,   0,   0,
         0,   0,  64,   0,   0,  42,   0,   9,   0,   0,   0,   3,   0,
         0,   0,   1,  64,  36,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   0,   2,  64,  36,   0,   0,   0,   0,   0,
         0,  64,  52,   0,   0,   0,   0,   0,   0,  64,   0,   0,  34,
         0,   9,   0,   0,   0,   1,   0,   0,   0,   3,  64,  52,   0,
         0,   0,   0,   0,   0, 192,  53,   0,   0,   0,   0,   0,   0,
       192,  54,   0,   0,   0,   0,   0,   0,  64,   0,   0,  42,   0,
         9,   0,   0,   0,   3,   0,   0,   0,   1,  64, 105,   0,   0,
         0,   0,   0,   0,   0,   0,   0,   1, 192, 105,  32,   0,   0,
         0,   0,   0,   0,   0,   0,   1,  64, 105,  64,   0,   0,   0,
         0,   0], dtype=uint8)
>>> vector_of_vectors.basket(0).byte_offsets
array([  0,  10,  28,  74, 112, 158], dtype=int32)

The Python prototype is done. The C++ prototype will be a new PR. (I'm merging this one to be sure that the ak.from_buffers correction gets into the main codebase.)

The Python prototype is done. The C++ prototype will be a new PR. (I'm merging this one to be sure that the ak.from_buffers correction gets into the main codebase.)

Interesting.. I assume on the uproot side of this the form keys don't depend on the partition, right? If so, and if we find they are invariant for files with identical branches, then this satisfies the needs for establishing a base form for NanoEvents and then I don't have to worry about maintaining https://github.com/CoffeaTeam/coffea/blob/master/coffea/nanoevents/mapping/uproot.py#L34 for all types of root files (and can drop the !skip and !content DSL commands as well)

Interesting.. I assume on the uproot side of this the form keys don't depend on the partition, right? If so, and if we find they are invariant for files with identical branches, then this satisfies the needs for establishing a base form for NanoEvents and then I don't have to worry about maintaining https://github.com/CoffeaTeam/coffea/blob/master/coffea/nanoevents/mapping/uproot.py#L34 for all types of root files (and can drop the !skip and !content DSL commands as well)


Adding custom node to the dashboard navigation error

Hello,
I added a new custom node to dashboard navigation in settings.py:
OSCAR_DASHBOARD_NAVIGATION += [     {         'label': _('New Admin'),         'icon': 'icon-bar-chart',         'url_name': 'admin:main_school_changelist',     }, ]
but I get this error
ImproperlyConfigured at /shop/dashboard/pages/ Please follow Oscar's default dashboard app layout or set a custom access_fn
It seems like someone has the same problem: https://groups.google.com/forum/#!msg/django-oscar/D96E15xcjPk/LGdHDhIgUXgJ
I'm using Django Oscar 1.1.1 and Django 1.8.13.

You can add the following item in the dictionary:
'access_fn': lambda user, url_name, url_args, url_kwargs: user.is_staff
this needs better documentation..

I also got this problem but I can't figure out how to fix it. The lambda user for access_fn iś not working

I too was struggling with this, so I'm adding what I learned to get it working.  Here is a dashboard menu item that is basically a redirect.
from my settings.py file:
INSTALLED_APPS += ['myredirectapp']
#####
from django.utils.translation import ugettext_lazy as _
OSCAR_DASHBOARD_NAVIGATION += [
    {
        'label': _('My Redirect App Label'),
        'icon': 'icon-globe',
        'children': [
            {
                'label': _('My Redirect App Child Label'),
                'url_name': 'my-redirect-index',
                'access_fn': lambda user, url_name, url_args, url_kwargs: user.is_staff,
            },
         ]
    },
]

from my project's urls.py
urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'myredirect', include("myredirectapp.urls")),
    url(r'', include(application.urls)),
]

from myredirectapp/urls.py
from django.conf.urls import url
from . import views

urlpatterns = [
    url(r'^$', views.index, name='my-redirect-index'),
]

from myredirectapp/views.py
from django.shortcuts import redirect

def index(request):
    return redirect("/some/other/url/")

Fixed by 6f685be795a4888f9b34b91c44882dffecc67260.


pikaday option defaultDate & setDefaultDate not work

See pikaday defaultDate example: http://jsfiddle.net/ppdJx/16/
But with this addon these two options won`t work.
I commented one line of code as below (in mixins/pikaday.js)and it works again:
  setPikadayDate: function() {
    const format = 'YYYY-MM-DD';
    const value = this.get('value');

    if (!value) {
      // this.get('pikaday').setDate(value, true);
    } else {
      const date = this.get('useUTC') ? moment(moment.utc(value).format(format), format).toDate() : value;

      this.get('pikaday').setDate(date, true);
    }
  },

Can you share a minimal Ember Twiddle (https://ember-twiddle.com/) displaying this?
Did you set defaultDate in the option hash?

I tried ember-twiddle, but some error was encountered when installing this addon in ember-twiddle.
Yes I set defaultDate in the option hash, this is my code:
  {{pikaday-input
  format=dateFormat
  theme="report-period-select-theme"
  readonly="readonly"
  firstDay=1
  maxDate=maxDate
  minDate=minDate
  onSelection=(action 'selectedEndDate')
  options=(hash defaultDate=endDate setDefaultDate=true)
  }}

I think maybe the origin code in pikaday.js prevented the underlying pikaday to set defaultDate as the initial value?

BTW, this is the error I encountered in twiddle:

ember-twiddle-7e98278….js:2 Addon ember-pikaday encountered a build error:(anonymous function) @ ember-twiddle-7e98278….js:2n @ vendor-87dd926….js:4(anonymous function) @ vendor-87dd926….js:4e.(anonymous function) @ vendor-87dd926….js:4_takeSafeStep @ vendor-87dd926….js:77_takeStep @ vendor-87dd926….js:77invoke @ vendor-87dd926….js:7flush @ vendor-87dd926….js:7flush @ vendor-87dd926….js:7end @ vendor-87dd926….js:7(anonymous function) @ vendor-87dd926….js:7
ember-twiddle-7e98278….js:2 Could not start watchman; falling back to NodeWatcher for file system events.
Visit http://ember-cli.com/user-guide/#watchman for more info.
Installed packages for tooling via npm.
installing ember-pikaday
install bower package pikaday
not-cached https://github.com/dbushell/Pikaday.git#*
resolved https://github.com/dbushell/Pikaday.git#1.4.0
Installed browser packages via Bower.
install addon ember-cli-moment-shim
Installed packages for tooling via npm.
installing ember-cli-moment-shim
Installed addon package.
Installed addon package.
Could not start watchman; falling back to NodeWatcher for file system events.
Visit http://ember-cli.com/user-guide/#watchman for more info.
DEPRECATION: EmberApp.concatFiles() is deprecated. Please use the broccoli-concat module directly.
DEPRECATION: EmberApp.concatFiles() is deprecated. Please use the broccoli-concat module directly.
headerFiles cannot contain a glob,  [object Object]
TypeError: headerFiles cannot contain a glob,  [object Object]
at /addon-builder/node_modules/broccoli-concat/concat.js:79:13
at Array.forEach (native)
at ensureNoMagic (/addon-builder/node_modules/broccoli-concat/concat.js:77:16)
at new Concat (/addon-builder/node_modules/broccoli-concat/concat.js:56:3)
at module.exports (/addon-builder/node_modules/broccoli-concat/index.js:26:10)
at StubApp.EmberApp.concatFiles (/addon-builder/node_modules/ember-cli/lib/broccoli/ember-app.js:368:10)
at module.exports (/addon-builder/ember-cli-build.js:103:9)
at CoreObject.module.exports.Task.extend.setupBroccoliBuilder (/addon-builder/node_modules/ember-cli/lib/models/builder.js:74:19)
at CoreObject.module.exports.Task.extend.init (/addon-builder/node_modules/ember-cli/lib/models/builder.js:54:10)
at CoreObject.superWrapper [as init] (/addon-builder/node_modules/core-object/lib/assign-properties.js:32:18)

No addon.js created(anonymous function) @ ember-twiddle-7e98278….js:2n @ vendor-87dd926….js:4(anonymous function) @ vendor-87dd926….js:4e.(anonymous function) @ vendor-87dd926….js:4_takeSafeStep @ vendor-87dd926….js:77_takeStep @ vendor-87dd926….js:77invoke @ vendor-87dd926….js:7flush @ vendor-87dd926….js:7flush @ vendor-87dd926….js:7end @ vendor-87dd926….js:7(anonymous function) @ vendor-87dd926….js:7
ember-twiddle-7e98278….js:3 Object {status_date: "2016-10-26T03:25:33.351Z", status: "build_error", addon_js: null, addon_css: null, error_log: "Could not start watchman; falling back to NodeWatc…/assign-properties.js:32:18)↵↵No addon.js created"}(anonymous function) @ ember-twiddle-7e98278….js:3n @ vendor-87dd926….js:4(anonymous function) @ vendor-87dd926….js:4e.(anonymous function) @ vendor-87dd926….js:4_takeSafeStep @ vendor-87dd926….js:77_takeStep @ vendor-87dd926….js:77invoke @ vendor-87dd926….js:7flush @ vendor-87dd926….js:7flush @ vendor-87dd926….js:7end @ vendor-87dd926….js:7(anonymous function) @ vendor-87dd926….js:7

Yea generally speaking the support for the options hash is less thorough as properties we add natively.
The road forward here is:

[ ] add defaultDate as a native argument in pikaday-input, e.g. {{pikaday-input defaultDate=myDate}}
[ ] use setDefaultDate to true if defaultDate is supplied and pass the defaultDate on to Pikaday.

Welcoming a PR, I might not be able to get to this myself in the coming time.

Hello. Is there any progress on this? thanks!


fix Refactor LRU cache policy memory tracking

Proposed changes
Fix #35590, CacheManager get CachePolicy should be used lock.

Add LRUCachePolicyTrackingAllocator and LRUCachePolicyTrackingManual.
tracking memory in LRUCachePolicy::insert, LRUCacheValueBase not need init mem_tracker, so not need to get CachePolicy under lock.

run buildall

run buildall

run buildall

run buildall


[dask] make random port search more resilient to random collisions (fixes #4057)

Changes in this PR

if using training with "search for random ports" strategy (documented in https://lightgbm.readthedocs.io/en/latest/Parallel-Learning-Guide.html#using-specific-ports), lightgbm.dask. will now check for the situation where it found duplicate IP-port pairs. If such duplicates are found, lightgbm.dask will re-run the random search up to 10 times.

Background
https://github.com/microsoft/LightGBM/issues/4057 documents a class of error that can fail with distributed training using lightgbm.dask. If you do not provide machines or local_listen_port at training time, LightGBM will randomly search for ports to use on each worker.
To limit the overhead introduced by this random search, a function _find_random_open_port() is run once at the same time on every worker, using distributed.Client.run() (added in #3823). If you have multiple Dask worker processes on the same physical host (i.e. if you're using distirbuted.LocalCluster or nrprocs > 1), there is a small but non-zero probability that this setup will choose the same random port for multiple workers on that host. When that happens, LightGBM training will fail with an error like

Exception: LightGBMError('Socket send error, code: 104',)

Based on https://github.com/microsoft/LightGBM/pull/4112#issuecomment-808166310, I think my original assumptions about how likely such conflicts were (https://github.com/microsoft/LightGBM/issues/4057#issuecomment-794889388) was not correct.
How this improves lightgbm
Removes a source of instability that could cause distributed training with Dask to fail.
Improves the stability of LightGBM's tests, which should reduce maintainer effort needed in re-running failed builds.

Linking https://github.com/microsoft/LightGBM/pull/4132#issuecomment-809066586, the finding that inspired this pull request.

@jmoralez Are you OK to merge this?


feat(web,vue-lib) alternate minimal tab style

adds new minimal tab style which can be toggled on. For now we'll use wherever we have nested tabs, but may want to consider using in other places as well.

also fixes the diff tab when viewing head and other small tab related issues.

bors merge


Automatically retry failed transactions due to write conflicts

When running multiple concurrent merge/update operations on the same output Delta table, some of the operations might fail due to a concurrent write conflict. This is a result of Delta Lake using optimistic concurrency control. To prevent the entire application from failing, you can catch the exception thrown by the conflict, inspect it, and then retry the operation (if it makes sense to do so). I believe this is not uncommon in streaming scenarios where a Delta table sink is being updated by multiple streaming sources.
My suggestion is to provide the retry mechanism in Delta Lake, which could be controlled by a configuration value
spark.databricks.delta.retryWriteConflict.enabled = true  # would be false by default
spark.databricks.delta.retryWriteConflict.limit = 3  # optionally limit the maximum amout of retries
# can probably add more options like retry delay, exponential backoff, etc...

Retrying any operation on conflict can very expensive as it will require reprocessing the entire data once again as the underlying table snapshot has changed.  So fundamentally, instead of retrying and suffering from the resultant unpredictability, it's much better to design your workloads to avoid conflicts altogether.

@tdas What is the recommended way to "design the workload" to avoid conflicts altogether? How is it possible with multiple streaming queries updating the same table to completely not have this problem? I think this improvement makes sense.

@ZoharMeir What has been your mechanism for effectively retrying the query? We are specifically targeting partitions during upserts and the latency seems to be unbelievably slow, so not sure if we should pivot back to the vanilla merge queries that cause the append conflict exceptions.

@AlexDHoffer Our current solution is not very efficient - we basically wrap the query execution in a try/except and retry the query if the exception is the expected Concurrent*Exception. Something like this:
def run_query_with_retry(query, attempts=3):
    try:
        return query()
    except Exception as e:
        if attempts > 1 and ('ConcurrentAppendException' in str(e) or 'ConcurrentDeleteReadException' in str(e)):
            return run_query_with_retry(query, attempts-1)
        else:
            raise

run_query_with_retry(
    lambda: target_dt.alias('t').merge(source_df.alias('s'), 's.key=t.key').whenNotMatchedInsertAll().execute()
)

Regarding the append conflicts with merge queries - perhaps this would be helpful: https://github.com/delta-io/delta/issues/326

I've been lead to believe by Databricks support that there is more of a built-in way to trigger these retries, specifically with OPTIMIZE. We frequently run into issues with Delta tables that are Streaming Sinks and a separate job that OPTIMIZES said tables. The OPTIMIZE job fails with ConcurrentDeleteReadException. The confusing part is that the only jobs writing to the table are the Stream and the OPTIMIZE. The Stream is only appending and it does not have auto-optimize or auto-compaction turned on, so what else would be deleting data?
Anyway... I got this from support

Enable retry handling logic in optimize job: You can enable retry logic on the explicit optimize command. Concurrent exceptions are always re-triable and there is no harm. So if you retry the optimize operation 3 times, it will succeed and the job will not fail.

I believe the failure you are describing stems from a conflict between the stream and optimize jobs. I submitted https://github.com/delta-io/delta/pull/626 some months ago which should resolve this conflict, but sadly it was never merged :(

We’re currently reviewing the issue and will get back to you shortly.

When running multiple concurrent merge/update operations on the same output Delta table, some of the operations might fail due to a concurrent write conflict. This is a result of Delta Lake using optimistic concurrency control. To prevent the entire application from failing, you can catch the exception thrown by the conflict, inspect it, and then retry the operation (if it makes sense to do so). I believe this is not uncommon in streaming scenarios where a Delta table sink is being updated by multiple streaming sources.
My suggestion is to provide the retry mechanism in Delta Lake, which could be controlled by a configuration value
spark.databricks.delta.retryWriteConflict.enabled = true  # would be false by default
spark.databricks.delta.retryWriteConflict.limit = 3  # optionally limit the maximum amout of retries
# can probably add more options like retry delay, exponential backoff, etc...

hi, i've tried setting these as config of spark session and then tried updating/inserting an entry while simultaneously compacting a particular partition (which isn't the partition that I was updating/inserting). It still leads to "ConcurrentAppendException", even though i've set the limits way above than 3. Is there any alternate solution/ is it something wrong that i am doing here?

Hi @sreeaishwar, these configurations were only a suggestion for a new feature - they do not currently exist in any Delta release.
You can try this approach as a workaround if it makes sense for you: https://github.com/delta-io/delta/issues/445#issuecomment-775085613
(keep in mind this code snipped was written for Delta 0.6.x, so you may need to adjust it to work with the latest version)

Hi @sreeaishwar, these configurations were only a suggestion for a new feature - they do not currently exist in any Delta release.
You can try this approach as a workaround if it makes sense for you: #445 (comment) (keep in mind this code snipped was written for Delta 0.6.x, so you may need to adjust it to work with the latest version)

hi, @zmeir tried the code snippet u mentioned here,it still thows a "ConcurrentAppendException". i hope this makes sense to u. also i tried increasing the limit which also didn't help. delta is same.

Hi @sreeaishwar, these configurations were only a suggestion for a new feature - they do not currently exist in any Delta release.
You can try this approach as a workaround if it makes sense for you: #445 (comment) (keep in mind this code snipped was written for Delta 0.6.x, so you may need to adjust it to work with the latest version)

hi, @zmeir tried the code snippet u mentioned here,it still thows a "ConcurrentAppendException". i hope this makes sense to u. also i tried increasing the limit which also didn't help. delta is same.

the issue is solved.

So what was the fix? Use retry as mentioned here? https://github.com/delta-io/delta/issues/445#issuecomment-775085613

@aish-war Can you please share the latest code that you used that fixed the logic?

@kumarigal Have you found the solution for the problem?


add stress test with an very long list

with the same repeated atom to not use a lot of memory

I added this stress test to ensure it's OK to lower the cost ascribed to c without making a DOS attempt viable. As demonstrated here, it's fine now and would fail if me next batch breaks that.


How to wire interceptors in spring boot AppConfig.

As title, please give an example. thanks.

I apologize. Work for my employer has been keeping me busy for quite a while. I will try to get you and example ASAP.

Sorry for the delay.
Looking at the code I will have to enhance ChannelProcessor to accept a list of Interceptors. In the meantime you can do something like:
  @Bean
  public SourceRunner seqSource(Channel memoryChannel, Map<String, String> source1Properties) {
    ChannelSelector selector = new ReplicatingChannelSelector();
    selector.setChannels(listOf(memoryChannel));
    SourceRunner runner = configureSource("source1", SequenceGeneratorSource.class, selector,
        source1Properties);
    Context context = source1Properties != null ? new Context(source1Properties) : new Context();
    runner.getSource().getChannelProcessor().configure(context);
    return runner;
  }

The properties will need to include the configuration for the interceptor as shown in the manual - you won't be able to configure it as a Spring Bean until I fix this.


When would device id get changed on iOS?

Question
What are the known cases were device id got from getUniqueId  would be changed? ex: when app got removed and installed again? what other cases would there be?
I found some questions on SO but I could not find if the id would get changed while the app is already installed?

Very API dependent I think? As in the old android versions behave differently than the new ones. And it may change again in the future. You'll have to refer to primary documentation (android SDK docs) and/or stackoverflow on this one though, I am not an expert as I haven't needed stability of these ids in my work project so haven't paid too much attention personally.
I'll leave this open for a while though in case an actual expert turns up.

Also - I would test test and re-test if it's a key part of the app that the id behavior is well-understood. I marked it android but I suppose it could be iOS as well? You might be specific on that, as well with what specific version range you care about

I found this for iOS https://developer.apple.com/documentation/devicecheck#overview but I am not sure if this would get persistent id??

@mikehardy is the new getDeviceToken() likely to be more persistent than getUniqueId() on iOS?

I would not even hazard a guess, uniquely identifying specific devices is a dark art and generally discouraged so :man_shrugging: - but I'll expose the APIs if possible :-)

@grit96 The new getDeviceToken() uses the DeviceCheck API (https://developer.apple.com/documentation/devicecheck#overview).
The documentation says:

Your app uses the DeviceCheck APIs to generate an ephemeral token that identifies a device.

So no, it won't be persistent :)

As its not persistent, and I don't know when it would be invalid, what use cases would it be useful in?!

It may not be any more. Remember these APIs change every OS release for privacy reasons. I imagine it was useful before and simply still exists.

So is there a way to get an unchanging devide id ?

@BenjaminSAILcloud I would expect device IDs to only be stable for a while. They should be stable during a single installation of the app, but are subject to change if the app is removed and reinstalled. This is a good thing for your privacy and is the point of the API changes the vendors have put in


Documentation needed for Calling generated client for Typescript

Hello,
I am able to generate the typescript code for WebAPI calls and able to call them as well. I am not sure i am calling it right or wrong way from my angular2 component .ts file
After getting data i am getting many internal errors.
Is there proper documentation available for this which can show how to utilize client.ts proper way.
My client.ts have all generated output for Typescript using nswag windows tool.
I have app.component.ts file from where i am calling client.ts function call getMembers();
I am able to get all data from the web api but it is giving some internal errors later. one of them is every property in returned json in _body is not defined.

The names are converted to js style, try firstName in your html (lower camel case)

Could you solve your problem?

Yes I was able to solve the problem. Thanks...


new_measure__amox_500_15_caps.json

Preview measure here: https://openprescribing.net/measure/preview_amox_500_15_caps/

Looks good - are there any obvious conditions we can think of that have >5 days course recommended. It would be good to hint that it might not be realistic to get this to 0% - I think it reads like that now.

@chrisjwood16 I don't think the "title" field actually does anything...

@chrisjwood16 I don't think the "title" field actually does anything...

Ah, couldn't work out what was missing, but felt weird to leave it as "title"!

Looks good - are there any obvious conditions we can think of that have >5 days course recommended. It would be good to hint that it might not be realistic to get this to 0% - I think it reads like that now.

I've added in H. pylori treatment as an example of where higher doses/longer courses might be used.


Next Task Optimization for the Mulithreaded Executor

Objective
Whenever the multithreaded executor spawns system tasks, it always starts a new task when a system is supposed to run. This presents a potential context switch if the task is stolen by another thread. Even with #11906, its still pretty common to see systems and the mulithreaded executor ping-pong between multiple threads, particularly when systems becomes serialized due to conflicting accesses or explicit ordering, with each bounce incurring a context switch overhead. This PR aims to address that overhead.
Solution
This PR ensures that at least one thread is running a scheduled system or the multithreaded executor at any given time. This is done by withholding one of the systems that are slated to run, and instead of spawning a task, directly run the system inside the same task the mulitthreaded executor is running in after releasing the lock on the executor. So long as there is at least one system the executor can run, and the task can grab the lock on the multithreaded executor, it will continue to run in a loop without yielding to the async task executor.
In strictly serialized schedules, there should now be close to zero OS-level overhead between running a single threaded executor and the multithreaded executor.
Performance
The performance difference here is more noticeable with schedules with a large number of serialized or short-lived systems. Tested on many_foxes (yellow is main, red is this PR), there is about a consistent 5% performance improvement on the main schedule.

On the Render schedule, this yields a more meager 1% improvement on the entire schedule:

result with benchmark  #12873

I experimented with something similar to this recently, and found a few potential issues when looking at the traces more closely.
On my 4-core machine, most of the benefit came from the fact that there were extra threads running.  The main and render threads currently sleep while running a schedule, but with this change they will run systems alongside the compute task pool threads.
My understanding is that some architectures need to send rendering tasks to the main thread, so running more of the main schedule there could delay rendering (until #9122 is merged).
Also, if you have tasks saturating the Io and AsyncCompute task pools, then keeping the main thread running would give you more threads than cores and cause thrashing.  That won't show up in many_foxes since it doesn't do much on those pools, although it might become a problem if #1907 is resolved.
The other potential issue is that system tasks sometimes get picked up by the scope running a par_iter.  With this change, system tasks can run much longer and could block the scope for longer.  It would be possible for a par_iter in the Render schedule to block while it runs the entire PostUpdate schedule, or vice versa!


PowerToys Run doesn't get focus when I invoke it with a keystroke

Description of the new feature / enhancement
When I open PowerToys Run via a keystroke (ALT+SPACE), the search bar appears but the focus is not always on it, so I have to click onto it BEFORE beginning to type my search or command.
Scenario when this would be used?
Everytime I use PowerToys Run.
Supporting information
This weird behaviour just happen to me.

I was using FireFox Nightly in Windows 10.
Opened PowerToys Run.
Begin my typing.
And then the error sound raised. No typing into the PowerToys Run search box.
I then click into the search box and resume my typing.
Everything OK.

Hi @lpiot, can you drag and drop a copy of your logs generated by the system tray?

Seems like #8876 to me.

Seems like #8876 to me.

You're definitely right.

Hi @lpiot, can you drag and drop a copy of your logs generated by the system tray?

PowerToysReport_2021-09-17-15-06-51.zip

@lpiot As https://github.com/microsoft/PowerToys/issues/8876#issuecomment-915440418 says, the fix of this should be delivered with the release of v0.47. We should close this issue first and wait for the v0.47 release which will be delivered at the end of September 2021.
If this issue still persists in v0.47, please create another issue for us to track and investigate this. Thanks and stay safe!

@lpiot As #8876 (comment) says, the fix of this should be delivered with the release of v0.47. We should close this issue first and wait for the v0.47 release which will be delivered at the end of September 2021.
If this issue still persists in v0.47, please create another issue for us to track and investigate this. Thank you and stay safe!

Of course, I tried and look for a similar issue but didn't find it. Sorry for this duplicate one.
And BTW, great job. I just migrated from 15 years of Macintosh to Windows 10 and couldn't live without this kind of tools.
Thank you so much!

/duplicate #8876


Incorrect Gunning Fog Calculation

In the FOG calculation, complex words are defined as those words having 3 or more syllables. Not that Wikipedia is the best source of information, but see list item 3 here. But, as you can see in the snippet below from textstat.py the threshold for a difficult word is: not in the easy_word_set and more than one syllable. (Lines 254-255)
    @repoze.lru.lru_cache(maxsize=128)
    def difficult_words(self, text):
        text_list = re.findall(r"[\w\='‘’]+", text.lower())
        diff_words_set = set()
        for value in text_list:
            if value not in easy_word_set:
                if self.syllable_count(value) > 1:
                    diff_words_set.add(value)
        return len(diff_words_set)

    @repoze.lru.lru_cache(maxsize=128)
    def dale_chall_readability_score(self, text):
        word_count = self.lexicon_count(text)
        count = word_count - self.difficult_words(text)

        try:
            per = float(count) / float(word_count) * 100
        except ZeroDivisionError:
            return 0.0

        difficult_words = 100 - per

        score = (
            (0.1579 * difficult_words)
            + (0.0496 * self.avg_sentence_length(text)))

        if difficult_words > 5:
            score += 3.6365
        return legacy_round(score, 2)

    @repoze.lru.lru_cache(maxsize=128)
    def gunning_fog(self, text):
        try:
            per_diff_words = (
                (self.difficult_words(text) / self.lexicon_count(text) * 100))

            grade = 0.4 * (self.avg_sentence_length(text) + per_diff_words)
            return legacy_round(grade, 2)
        except ZeroDivisionError:
            return 0.0`

I created a #71  for a fix to this.


Clean out IE 9-specific CSS hacks

Specifically, search for and clean up cases where CSS written to work with IE9 may lead to issues with newer browsers.

I will take this

Fixed by @fortMaximus.


Bump @solana/web3.js from 1.91.3 to 1.91.7

LGTM for js-deps-reviewers. We're not bringing in any additional transitive dependencies with this.


Feat/improve registration

Merge spotlight events data with schedule events data (they use the same events and data)
Refactor forms to use Formik library

Easier to write forms fields. and error messages
Easier to create custom validation for each field
Code is cleaner and easier to read

Change profile dialog call to action to specify register button if user has not registered
Profile dialog name no longer displays null if user name is null(before registering)

📝 Changed routes:

/auth
/auth/signup

(view full size)
(view full size)

…mponents/SpotlightCard
/dashboard

(view full size)
(view full size)

/register

(view full size)

And 1 other route:

/api/schedule

Commit ff5c4f08829d016dd047f6a2eee91846bdfca64e (https://hackportal-jfq59dw4x-acmutd.vercel.app).

Also, styling for dropdowns needs to be consistent:

The styling is different cuz they use different types of dropdown. The larger ones like university and majors use datalist instead of the select dropdown because the select dropdown has horrible autocomplete/typeahead which makes it really hard to find your university. I don't think we're able to change the style of the dropdowns since its mostly determined by browsers.

Also forgot to remove asterisks after copying majors from ACT website lol


Added wake-on-activity support

Hey, could you provide a bit of context how to test it so I can verify the code and merge your PR?

Hi Adam
Here is an example that uses it:
https://github.com/gregyedlik/irq_lowpow_canbed
To really test it, you need to monitor the current consumption of the chip (or at least the board) so that you see it moving from sleep mode to listen only mode  mode, and then eventually normal mode if you explicitly make that happen from the software, as I did in the example above.


empty test file

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@26f2835). Click here to learn what that means.
The diff coverage is n/a.

@@          Coverage Diff           @@
##             main      #7   +/-   ##
======================================
  Coverage        ?   3.50%
======================================
  Files           ?       4
  Lines           ?      57
  Branches        ?       0
======================================
  Hits            ?       2
  Misses          ?      55
  Partials        ?       0

Flag
Coverage Δ

unittests
3.50% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


create DBO Arctic project document

Create a project document showing the DBO Arctic project info serialized into the emerging project description format.

Additional details:

We need to support things like vessel searching, and adding international flags to the logos section.
For the filters, we would like to emulate the functionality found here. Their cruise data page might also provide details.

Hi @robyngit - When you create a DBO project definition document, please upload it to both https://dev.nceas.ucsb.edu and https://test.arcticdata.io.
Please include an AccessPolicy that has read/write/changePermission permissions for the CN=knb-data-admins,DC=dataone,DC=org and the CN=arctic-data-admins,DC=dataone,DC=org groups respectively.  Also, assign the seriesId in the system metadata which is found here:
https://github.com/NCEAS/metacatui/blob/b713766a028a8b6d6c1d77b96acc17a58127577f/src/js/themes/arctic/models/AppModel.js#L129
That's the seriesId that allows us to navigate to /projects/DBO.

Initial DBO project xml is now available on dev.nceas with pid urn:uuid:d4f20af6-0134-4bfa-8204-eb3ee5152ad7 and tests.arcticdata with pid urn:uuid:62d1add2-d3dd-470e-b31e-0e82ccb23c72. The seriesID (sid) for both is urn:node:93834148-30ce-420d-8d6d-b6690d93b9bc


Update @nx/workspace to 19.5.0

Update Nx from 19.4.4 to 19.5.0

Auto-generated by create-pull-request

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 97.63%. Comparing base (8292aca) to head (ede5cdc).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #147   +/-   ##
=======================================
  Coverage   97.63%   97.63%
=======================================
  Files          17       17
  Lines         211      211
  Branches       30       30
=======================================
  Hits          206      206
  Misses          5        5

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Blank Priority field on Engagement Plan Task should use default Task Priority on created Tasks

thread: https://powerofus.force.com/0D580000030CD8v
The Visualforce page does not allow option to set Priority to None, but this can be done easily in standard edit. We should enforce that the Priority field on Engagement Plan Task is always populated, even if not editing from VF page.

**lurch: add

Tracking W-011498

Included in beta release 3.83 (Beta 7)

Included in production release rel/3.83


Issue with npm install

OS: Ubuntu 20.04 LTS (via WSL)
Launch command used: npm i
When I try to enter the commands mentioned in the README.md file, it give out the following error when I type the npm install command
/usr/bin/env: ‘bash\r’: No such file or directory

I am not sure what exactly is the issue after I properly cloned the repository as requested.

Verbose  log
PASTE VERBOSE LOG HERE

If you are using WSL, please clone repo from WSL git, NOT windows git.
Windows git will convert line ending to CRLF by default, breaking all the sh1ts


NOISSUE: build master as :latest

Codecov Report

Merging #69 into master will increase coverage by 0.22%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #69      +/-   ##
==========================================
+ Coverage   74.63%   74.86%   +0.22%
==========================================
  Files          43       43
  Lines        2764     2757       -7
==========================================
+ Hits         2063     2064       +1
+ Misses        561      555       -6
+ Partials      140      138       -2

Impacted Files
Coverage Δ

api/handlers.go
79.56% <100.00%> (+0.13%)
:arrow_up:

etl/storage/storage.go
81.96% <100.00%> (+2.43%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e9bff4...de9a550. Read the comment docs.


Use proper API for controlPlaneCluster namespace annotation

Please provide a description of this PR:

@frankbu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
067171f2eefcacf119f5feef5cf42a096cf3f23c
link
true
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test release-notes_istio

/retest


TextRange and LineRange information is incorrect in diagnostic locations

See https://github.com/ballerina-platform/ballerina-lang/blob/ee83d258a4a7e95d1e397dcc334793e9fe4c9603/compiler/ballerina-lang/src/main/java/org/wso2/ballerinalang/compiler/diagnostic/BLangDiagnosticLocation.java#L38
The values set for the columns in line range and the offsets set in text range are incorrect. The text range is supposed to be the absolute position as an offset from (0, 0).

LineRange and TextRange distinguishes as follows:
LineRange --> Verbose line number information
|_ filename
|_ startLine
|    |_line (line number)
|    |_offset (column of the line)
|_ endLine
     |_line (line number)
     |_offset (column of the line)

TestRange --> Offsets relative to the (0,0) location of the file
  |_startOffset
  |_endOffset
  |_length


Updating to stripe/stripe-php=~2.0

Reopening on the master branch. Refer to the prior pull request for details: https://github.com/laravel/cashier/pull/152

This version may be breaking now that I look at the fact you allow passing $apiKey to methods when the parent class expects an array of options. However, this part is untested. I'm doing a lot of work with subscriptions and one off charges this week so I will test this code in depth and report back if this needs more work.
@taylorotwell also notified me on Twitter that he has a test Stripe application that he's going to test this with as well. So if he finds anything himself I'm sure we'll hear about it.

@taylorotwell after I ran composer update Im getting this for new users to my website: Could not determine which URL to request: Laravel\Cashier\Customer instance has invalid ID: Was there an upgrade?

Which version of cashier are you on. This hasn't been merged yet.
On Friday, April 10, 2015, Daveyon Mayne<EMAIL_ADDRESS>wrote:

@taylorotwell https://github.com/taylorotwell after I ran composer
update Im getting this for new users to my website: Could not determine
which URL to request: Laravel\Cashier\Customer instance has invalid ID: Was
there an upgrade?
—
Reply to this email directly or view it on GitHub
https://github.com/laravel/cashier/pull/153#issuecomment-91605473.

 The when I place this in my controller: $invoices = $user->invoices(); I get the error when a new user registers.

This sounds like a bug in your application. That error is you trying to call Stripe with a cus_ ID that does not exist in Stripe.
Have you verified that ID exists?
Sent from my iPhone

On Apr 10, 2015, at 10:15 AM, Daveyon Mayne<EMAIL_ADDRESS>wrote:
@taylorotwell after I ran composer update Im getting this for new users to my website: Could not determine which URL to request: Laravel\Cashier\Customer instance has invalid ID: Was there an upgrade?
—
Reply to this email directly or view it on GitHub.

Ahhhh but how to call that method? It works ok but if a new user registers in my website it throws an error. I cant even wrap it around an if statement.


Animated posters are not animated

I actually had similar issues to this; https://github.com/rmrector/script.artwork.beef/issues/48
However using a Shield.  Do not have the issues when using Windows.
So that brings me onto the issue.
I could not save gifs from the shield due to the samba issue.
I have since created the entire library on Windows instead, ran Artwork beef, saved all posters, gifs etc.
So I wiped the android side, and added everything again in order to get Artwork beef to pick up the newly saved images already in the folders, and it has! However, even though its picking up the animated posters, none are animted.. they are all just static imagery, yet it shows as a Gif and as a 'animated poster'.
Seems in the process of loading the images onto the shield, its just removed all the animation from the gifs.  Leaving lower quality posters.  Other posters however are fine.
Using latest Shield 8.0.2, update released today (27th Jan)
However issue existed on previous.

Thanks, I recently rebuilt my server and am now using a Windows share.
It worked from the Shield using network mount and saving without a issue now its a true windows share.
However, it just failed at saving a file and I am unsure as to why.


Distributed optimizations

This PR consists changes to optimize distributed mode of training.

depthwise changes: Possibility to build 'smaller' nodes was introduced for distributed mode (before this only left nodes were build)  for more balanced work. Than 'subtraction trick' is applied for 'larger' node.
lossguide changes: similar to above.
number of rabit::Allreduce calls is the same as before due to structured histograms adding.
Timers were added to track most time consuming parts for distributed also.

mortgage 3.5Gb

6 workers x 8 threads
loss
depth

Master
76.6
68.5

This PR
48.7
44

Hello @ShvetsKS could you provide some details as to what the plan is here?

@thvasilo,
The changes are close to this PR https://github.com/dmlc/xgboost/pull/4824. Currently we have different behavior for single-node and distributed modes in terms of what histograms we compute explicitly by BuildHist function and what nodes are build by SubtractionTrick.
Now for single-node it's more optimal, because we select smaller set of rows for BuildHist function and SubtractionTrick for larger, it brings performance gain. For distributed mode we just use BuildHist for left siblings always, in general it shows worse performance.
The PR introduce similar logic to distributed mode.

A genuine question, do you have difficulty to read the hist code after many optimizations (I see that you are really good at spotting optimization opportunity so I may be not as skilled)?  I see that specializing for each case is good for performance, but could you try to arrange the source code into finer modules and create a easier to follow structure?  Maybe some of these can help others to understand the code?

policy based http://loki-lib.sourceforge.net/index.php?n=Main.Policy-basedDesign
strategy https://en.wikipedia.org/wiki/Strategy_pattern

Also, if you specialize over something, then there needs to be a new test case for your specialization.

Please don't merge until #5557 (comment) is resolved. Also can we bring some refactoring like #5557 (comment) before merging any further optimization ?

@trivialfis Thanks for your suggestion.
Some refactoring was done, I hope it's more clear now. (Not canonical strategy pattern was implemented for synchronization and adding hist rows. I think the role of 'Context' class is covered by 'QuantileHistMaker::Builder' already. )
Unit tests for new functionality is in progress.

@trivialfis , @SmirnovEgorRu seems all major comments were applied, could you take a look ?

Hello @ShvetsKS since these change aim at improving distributed code, have you tested against the master in a distributed (not local cluster) environment.
My concern with previous changes being that they introduced many synchronization points. That's not happening any more I think, but I'd like to see the differences when run say over dask or Spark (or MPI) with network communication happening.
To stress test I'd suggest using the URL dataset which should introduce large amounts of communication.

@thvasilo We have specialized optimization for super sparse dataset, the opt is not enabled on distributed environment.  If @ShvetsKS is interested I can point out the details.

@thvasilo We have specialized optimization for super sparse dataset, the opt is not enabled on distributed environment. If @ShvetsKS is interested I can point out the details.

As far as I understand changes in this PR also should introduce some impact for sparse datasets (we have BuildHist specialized for sparse data, and Partition kernel). @trivialfis  Could you share the details please?
@thvasilo thanks for suggestions, measurements are in progress.

@ShvetsKS This line: https://github.com/dmlc/xgboost/blob/535479e69fd298c2cbabcb853adebb97f2fdf515/src/common/hist_util.cc#L129
If you run URL without the SparseCuts, the memory usage will blow up (by blow up I mean we don't know how much it needs, no one has the machine with enough memory to try).  But it's not enabled for distributed environment as we need to sync the number of required entries, I wasn't familiar with distributed system back then ...

@thvasilo
please see measurements (over Spark)
mortgage data set was used.

606M Rows on 4 SKL8180 nodes
depthwise, s
lossguide, s

Master
473.64
749.62

This PR
303.51
399.67

@ShvetsKS This line:
https://github.com/dmlc/xgboost/blob/535479e69fd298c2cbabcb853adebb97f2fdf515/src/common/hist_util.cc#L129
If you run URL without the SparseCuts, the memory usage will blow up (by blow up I mean we don't know how much it needs, no one has the machine with enough memory to try). But it's not enabled for distributed environment as we need to sync the number of required entries, I wasn't familiar with distributed system back then ...

That will be interesting to investigate in a future PR.
Thanks @ShvetsKS for the measurements, great improvements!


Hashing techniques in C++

Greetings,
I would like to add hashing Techniques (linear probing, quadratic probing and double hashing) in C++ along with proper function descriptions, space and time complexity.
Kindly assign this to me under Gssoc'21.

Hi,I'm a GSSoC participant.
I wish to add the implementation of Hashing Technique in Java with time and space complexity.


Unable to see 'dist' folders

Using bower if I install jquery I cannot see the dist folder in the file tree and its not picked up in my HTML either when adding the script tags.
This is the case for all folders called dist. I have to open it in finder and drag and drop the file to be able to view it!

I'm running windows 8
Brackets:
Release 1.3 build 1.3.0-16022 (release cd0a6aae5)
build timestamp: Wed Apr 29 2015 11:07:50 GMT+0100

Works in my machine :tm: .
Could you goto 'debug menu>reload without extensions'  and see if the problem persists?

@apricot13 do you the extension File Tree Exclude?

I had the same "problem", and it was an extension hiding it

Closing as most likely an extension behaviour.


Sleight of hand combat redo alex l

I highly suggest these 2 changes in order to keep all weapons from becoming, effectively, semi-automatic capable, for reasons of player choice, and weapon variety.
"Proposed balance change to "Sleight of hand" so that weapons without semi-automatic capability don't become identical to the same.
Allows the user to negate cocking of a weapon by performing another action first (aiming, moving, reloading, hacking, healing, etc.)
Alternative idea: Negate 1 bolt cycle per turn"

ayup. This was a good fix. @trowl223 , can we get a shipit?

@bleehu I made a requirement changes to sleight of hand, and to straight pull bolt, and I nerfed straight-pull bolt a little more.
Please review and let me know what you think?

@trowl223, can we get a shipit?


Read benchmarks

This PR adds some benchmarks to compare the following queries:

selecting a range of rows
performing a series of aggregations across a range of rows
performing a series of rolling window aggregations across a range of rows

and the following execution strategies:
Measuring 'Time to First Row'

runAndReadUntil(q, 1)
start runTask pending.readUntil(1)
start runTask single fetchChunk
streamAndReadUntil(q, 1)
startStream runTask pending.readUntil(1)
startStream runTask single fetchChunk
run single fetchChunk
stream single fetchChunk

Measuring 'Time to Full Result'

runAndReadAll()
start runTask pending.readAll()
start runTask fetchChunks loop
streamAndReadAll()
startStream runTask pending.readAll()
startStream runTask fetchChunks loop
run fetchChunks loop
stream fetchChunks loop

The statistically significant takeaways by my reading are:

Streaming gives you faster 'time to first result', as long as the query can be materialised lazily.

Overall aggregates don't benefit from streaming.
Rolling aggregates benefit greatly from streaming.
Simple row fetching benefits greatly from streaming.

Streaming can give you faster 'time to full result' in certain circumstances.

Overall aggregates again don't benefit from streaming.
Rolling aggregates don't benefit from streaming.
Simple row fetching benefits greatly from streaming.

Here are the full results on my M1 Macbook Pro:
 ✓ test/bench/read.bench.ts (48) 230150ms
   ✓ Row Fetching - Time to First Row (8) 5518ms
     name                                                           hz     min      max    mean     p75     p99    p995     p999      rme  samples
   · Row Fetching - runAndReadUntil(q, 1)                     1,376.76  0.4753  16.3925  0.7263  0.7714  0.9663  1.0138  16.3925   ±6.24%      689
   · Row Fetching - start runTask pending.readUntil(1)        1,079.19  0.4728  11.6620  0.9266  0.7817  7.6780  9.6702  11.6620  ±11.28%      542   slowest
   · Row Fetching - start runTask single fetchChunk           1,105.70  0.5103   9.1201  0.9044  0.7776  7.3376  8.0964   9.1201   ±9.74%      553
   · Row Fetching - streamAndReadUntil(q, 1)                  2,107.37  0.2790  43.5780  0.4745  0.4726  0.6541  0.7103   1.3092  ±16.94%     1054
   · Row Fetching - startStream runTask pending.readUntil(1)  2,323.58  0.2688   9.7647  0.4304  0.4301  0.5947  5.6202   7.1385   ±6.67%     1164
   · Row Fetching - startStream runTask single fetchChunk     2,363.71  0.2729   9.1197  0.4231  0.4259  0.5968  4.8921   8.6742   ±6.72%     1182   fastest
   · Row Fetching - run single fetchChunk                     1,331.20  0.4783  59.6678  0.7512  0.7247  0.8967  0.9563  59.6678  ±23.13%      666
   · Row Fetching - stream single fetchChunk                  2,176.48  0.2682  94.8838  0.4595  0.4277  0.5614  0.5826   0.6777  ±31.38%     1285
   ✓ Overall Aggregates - Time to First Row (8) 5440ms
     name                                                               hz     min      max    mean     p75     p99    p995     p999     rme  samples
   · Overall Aggregates - runAndReadUntil(q, 1)                     877.93  0.7878   1.7058  1.1390  1.2682  1.4572  1.5715   1.7058  ±1.38%      440   fastest
   · Overall Aggregates - start runTask pending.readUntil(1)        828.47  0.8028   3.3645  1.2070  1.3137  2.7739  2.9337   3.3645  ±2.58%      415   slowest
   · Overall Aggregates - start runTask single fetchChunk           831.70  0.7918   3.6258  1.2024  1.3221  2.3136  2.6003   3.6258  ±2.22%      416
   · Overall Aggregates - streamAndReadUntil(q, 1)                  860.32  0.8093  10.5932  1.1624  1.2869  1.4233  1.5662  10.5932  ±3.93%      431
   · Overall Aggregates - startStream runTask pending.readUntil(1)  851.64  0.7908   1.6785  1.1742  1.3078  1.5665  1.6066   1.6785  ±1.40%      426
   · Overall Aggregates - startStream runTask single fetchChunk     863.23  0.7758   1.4962  1.1584  1.2987  1.4348  1.4740   1.4962  ±1.37%      432
   · Overall Aggregates - run single fetchChunk                     863.06  0.8056   8.9131  1.1587  1.2681  1.4543  1.5107   8.9131  ±3.32%      432
   · Overall Aggregates - stream single fetchChunk                  859.57  0.7852   5.0499  1.1634  1.2870  1.5952  2.1741   5.0499  ±2.14%      430
   ✓ Rolling Aggregates - Time to First Row (8) 85961ms
     name                                                                hz      min      max     mean      p75      p99     p995     p999     rme  samples
   · Rolling Aggregates - runAndReadUntil(q, 1)                     13.7236  44.5004   123.83  72.8670  78.2095  94.0872   100.97   123.83  ±2.02%      200
   · Rolling Aggregates - start runTask pending.readUntil(1)        13.8090  44.1295   121.97  72.4166  78.4516  97.5966   100.35   121.97  ±2.20%      200
   · Rolling Aggregates - start runTask single fetchChunk           13.4964  45.5832   156.89  74.0938  79.1863  96.2242  96.3086   156.89  ±2.18%      200   slowest
   · Rolling Aggregates - streamAndReadUntil(q, 1)                  32.4901  23.7333  42.0643  30.7786  32.1036  38.5903  39.3412  42.0643  ±1.70%      200   fastest
   · Rolling Aggregates - startStream runTask pending.readUntil(1)  31.6999  23.7718  44.6693  31.5458  33.0128  40.5613  41.1911  44.6693  ±1.70%      200
   · Rolling Aggregates - startStream runTask single fetchChunk     31.6347  23.8681  52.4038  31.6108  33.0380  47.6970  47.8277  52.4038  ±1.88%      200
   · Rolling Aggregates - run single fetchChunk                     13.9924  44.0817   121.27  71.4675  77.8605  79.8555   107.91   121.27  ±2.19%      200
   · Rolling Aggregates - stream single fetchChunk                  32.2393  23.5144  70.5517  31.0181  32.3448  39.6358  39.7199  70.5517  ±2.13%      200
   ✓ Row Fetching - Full Result (8) 5416ms
     name                                                        hz     min      max    mean     p75     p99     p995     p999     rme  samples
   · Row Fetching - runAndReadAll()                          718.63  0.8328   6.7487  1.3915  1.2545  6.3228   6.7328   6.7487  ±7.17%      360
   · Row Fetching - start runTask pending.readAll()          577.95  0.9156   7.1313  1.7302  1.6273  5.3958   5.8823   7.1313  ±6.66%      289   slowest
   · Row Fetching - start runTask fetchChunks loop           760.48  0.8213   5.4165  1.3150  1.3235  3.4509   3.9240   5.4165  ±4.16%      381
   · Row Fetching - streamAndReadAll()                       998.97  0.5869  17.6965  1.0010  0.9653  2.0408   2.4146  17.6965  ±7.01%      500
   · Row Fetching - startStream runTask pending.readAll()    991.83  0.5981   3.4852  1.0082  1.0139  2.6441   2.6847   3.4852  ±2.92%      498
   · Row Fetching - startStream runTask fetchChunks loop   1,000.72  0.6345   3.4698  0.9993  0.9927  2.7818   2.9266   3.4698  ±3.13%      501   fastest
   · Row Fetching - run fetchChunks loop                     701.55  0.8667  10.6289  1.4254  1.2878  9.3998  10.2593  10.6289  ±9.51%      351
   · Row Fetching - stream fetchChunks loop                  933.68  0.6153  17.5829  1.0710  1.0151  4.1196   4.2596  17.5829  ±7.85%      467
   ✓ Overall Aggregates - Full Result (8) 5432ms
     name                                                            hz     min     max    mean     p75     p99    p995    p999     rme  samples
   · Overall Aggregates - runAndReadAll()                        844.43  0.7865  5.8209  1.1842  1.3093  1.5038  1.5552  5.8209  ±2.27%      423
   · Overall Aggregates - start runTask pending.readAll()        845.97  0.7873  1.6718  1.1821  1.3326  1.5857  1.5935  1.6718  ±1.48%      423
   · Overall Aggregates - start runTask fetchChunks loop         830.36  0.8041  1.6605  1.2043  1.3396  1.5843  1.6173  1.6605  ±1.36%      416
   · Overall Aggregates - streamAndReadAll()                     856.70  0.7828  5.5033  1.1673  1.2991  1.4692  1.7650  5.5033  ±2.20%      429   fastest
   · Overall Aggregates - startStream runTask pending.readAll()  842.75  0.8340  1.7361  1.1866  1.3355  1.5678  1.5780  1.7361  ±1.46%      422
   · Overall Aggregates - startStream runTask fetchChunks loop   818.82  0.8265  1.7138  1.2213  1.3707  1.6475  1.6993  1.7138  ±1.58%      410   slowest
   · Overall Aggregates - run fetchChunks loop                   829.73  0.8407  7.8850  1.2052  1.3224  1.5317  1.6899  7.8850  ±2.95%      415
   · Overall Aggregates - stream fetchChunks loop                851.84  0.7923  7.5480  1.1739  1.2875  1.5922  1.8078  7.5480  ±3.36%      426
   ✓ Rolling Aggregates - Full Result (8) 122381ms
     name                                                             hz      min      max     mean      p75      p99     p995     p999     rme  samples
   · Rolling Aggregates - runAndReadAll()                        13.7514  45.1392   120.32  72.7201  78.8937  87.8110   111.02   120.32  ±2.12%      200
   · Rolling Aggregates - start runTask pending.readAll()        13.6204  46.5226   152.29  73.4192  78.9196  82.9205   105.00   152.29  ±2.14%      200
   · Rolling Aggregates - start runTask fetchChunks loop         13.7576  45.2252  95.2338  72.6872  78.6150  81.7152  82.0683  95.2338  ±1.85%      200   fastest
   · Rolling Aggregates - streamAndReadAll()                     13.2885  63.2723   105.59  75.2528  78.8987  94.2671  99.6013   105.59  ±1.20%      200
   · Rolling Aggregates - startStream runTask pending.readAll()  13.3006  63.4977   102.19  75.1848  79.3148  94.0547   100.32   102.19  ±1.21%      200
   · Rolling Aggregates - startStream runTask fetchChunks loop   13.3335  63.4264   121.22  74.9989  78.6301  82.0112   106.03   121.22  ±1.25%      200
   · Rolling Aggregates - run fetchChunks loop                   13.4990  44.8317   157.66  74.0796  78.4570  99.0749   103.98   157.66  ±2.04%      200
   · Rolling Aggregates - stream fetchChunks loop                13.2082  63.0522   167.50  75.7107  78.7524  91.6548   117.71   167.50  ±1.69%      200   slowest

 BENCH  Summary

  Row Fetching - startStream runTask single fetchChunk - test/bench/read.bench.ts > Row Fetching - Time to First Row
    1.02x faster than Row Fetching - startStream runTask pending.readUntil(1)
    1.09x faster than Row Fetching - stream single fetchChunk
    1.12x faster than Row Fetching - streamAndReadUntil(q, 1)
    1.72x faster than Row Fetching - runAndReadUntil(q, 1)
    1.78x faster than Row Fetching - run single fetchChunk
    2.14x faster than Row Fetching - start runTask single fetchChunk
    2.19x faster than Row Fetching - start runTask pending.readUntil(1)

  Overall Aggregates - runAndReadUntil(q, 1) - test/bench/read.bench.ts > Overall Aggregates - Time to First Row
    1.02x faster than Overall Aggregates - startStream runTask single fetchChunk
    1.02x faster than Overall Aggregates - run single fetchChunk
    1.02x faster than Overall Aggregates - streamAndReadUntil(q, 1)
    1.02x faster than Overall Aggregates - stream single fetchChunk
    1.03x faster than Overall Aggregates - startStream runTask pending.readUntil(1)
    1.06x faster than Overall Aggregates - start runTask single fetchChunk
    1.06x faster than Overall Aggregates - start runTask pending.readUntil(1)

  Rolling Aggregates - streamAndReadUntil(q, 1) - test/bench/read.bench.ts > Rolling Aggregates - Time to First Row
    1.01x faster than Rolling Aggregates - stream single fetchChunk
    1.02x faster than Rolling Aggregates - startStream runTask pending.readUntil(1)
    1.03x faster than Rolling Aggregates - startStream runTask single fetchChunk
    2.32x faster than Rolling Aggregates - run single fetchChunk
    2.35x faster than Rolling Aggregates - start runTask pending.readUntil(1)
    2.37x faster than Rolling Aggregates - runAndReadUntil(q, 1)
    2.41x faster than Rolling Aggregates - start runTask single fetchChunk

  Row Fetching - startStream runTask fetchChunks loop - test/bench/read.bench.ts > Row Fetching - Full Result
    1.00x faster than Row Fetching - streamAndReadAll()
    1.01x faster than Row Fetching - startStream runTask pending.readAll()
    1.07x faster than Row Fetching - stream fetchChunks loop
    1.32x faster than Row Fetching - start runTask fetchChunks loop
    1.39x faster than Row Fetching - runAndReadAll()
    1.43x faster than Row Fetching - run fetchChunks loop
    1.73x faster than Row Fetching - start runTask pending.readAll()

  Overall Aggregates - streamAndReadAll() - test/bench/read.bench.ts > Overall Aggregates - Full Result
    1.01x faster than Overall Aggregates - stream fetchChunks loop
    1.01x faster than Overall Aggregates - start runTask pending.readAll()
    1.01x faster than Overall Aggregates - runAndReadAll()
    1.02x faster than Overall Aggregates - startStream runTask pending.readAll()
    1.03x faster than Overall Aggregates - start runTask fetchChunks loop
    1.03x faster than Overall Aggregates - run fetchChunks loop
    1.05x faster than Overall Aggregates - startStream runTask fetchChunks loop

  Rolling Aggregates - start runTask fetchChunks loop - test/bench/read.bench.ts > Rolling Aggregates - Full Result
    1.00x faster than Rolling Aggregates - runAndReadAll()
    1.01x faster than Rolling Aggregates - start runTask pending.readAll()
    1.02x faster than Rolling Aggregates - run fetchChunks loop
    1.03x faster than Rolling Aggregates - startStream runTask fetchChunks loop
    1.03x faster than Rolling Aggregates - startStream runTask pending.readAll()
    1.04x faster than Rolling Aggregates - streamAndReadAll()
    1.04x faster than Rolling Aggregates - stream fetchChunks loop

This is how many tasks were run through before being 'ready' for each of the queries where applicable, as some additional 'numbers'. I don't think they tell much of a story, but interesting to know none-the-less.
// Full Result
Row Fetching - start runTask pending.readAll() - tasks 304
Row Fetching - start runTask fetchChunks loop - tasks 436
Row Fetching - startStream runTask pending.readAll() - tasks 25
Row Fetching - startStream runTask fetchChunks loop - tasks 114
Overall Aggregates - start runTask pending.readAll() - tasks 2018
Overall Aggregates - start runTask fetchChunks loop - tasks 1516
Overall Aggregates - startStream runTask pending.readAll() - tasks 6
Overall Aggregates - startStream runTask fetchChunks loop - tasks 3645
Rolling Aggregates - start runTask pending.readAll() - tasks 331597
Rolling Aggregates - start runTask fetchChunks loop - tasks 206513
Rolling Aggregates - startStream runTask pending.readAll() - tasks 109053
Rolling Aggregates - startStream runTask fetchChunks loop - tasks 121899

// Time to First Row
Row Fetching - start runTask pending.readUntil(1) - tasks 936
Row Fetching - start runTask single fetchChunk - tasks 211
Row Fetching - startStream runTask pending.readUntil(1) - tasks 363
Row Fetching - startStream runTask single fetchChunk - tasks 43
Overall Aggregates - start runTask pending.readUntil(1) - tasks 1490
Overall Aggregates - start runTask single fetchChunk - tasks 632
Overall Aggregates - startStream runTask pending.readUntil(1) - tasks 2639
Overall Aggregates - startStream runTask single fetchChunk - tasks 580
Rolling Aggregates - start runTask pending.readUntil(1) - tasks 337248
Rolling Aggregates - start runTask single fetchChunk - tasks 315596
Rolling Aggregates - startStream runTask pending.readUntil(1) - tasks 106010
Rolling Aggregates - startStream runTask single fetchChunk - tasks 8674

Thanks! Interesting data. Nice confirmation that streaming can be very useful.


[Simple proxies] See which researchers you manage as a proxy

Feature
As a proxy, I can see which researchers I can manage, and switch to their profile perspective to manage their bibliography
https://booktower-ugent-ui.vercel.app/biblio/researcher/proxies/proxies.html

Navigation

[x] When I have proxies assigned, a menu item appears that guides me to the overview of researcher I am the proxy for

Researcher proxy overview
https://booktower-ugent-ui.vercel.app/biblio/researcher/proxies/proxies.html

[x] Per researcher I proxy for, I can see their name and IDs.
Per researcher I can see...

Suggestions

[x] see how many suggestions there are, and go to the suggestions view

Publications

[x] unlocked should be in URL see how many unlocked draft publications there are, and go to the publications view with the draft and unlocked filter on
[x] unlocked should be in URL see how many unlocked withdrawn publications there are, and go to the publications view with the withdrawn and unlocked filter on
[x] go to all publications

Datasets

[x] unlocked should be in URL see how many unlocked draft datasets there are, and go to the datasets view with the draft and unlocked filter on
[x] unlocked should be in URL see how many unlocked withdrawn datasets there are, and go to the datasets view with the withdrawn and unlocked filter on
[x] go to all datasets

... which I can manage from a proxy perspective

Proxy perspective
https://booktower-ugent-ui.vercel.app/biblio/researcher/publications/overview--proxy.html

[x] When I click one of the options on the Proxy dashboard, I go to the page from a proxy perspective (Publications, datasets, suggestions)
I can manage the researcher's

[x] Publications
[x] Datasets
[x] I can do anything the researcher would be able to do, however, every action will be logged in my own name.

[x] I stay in the proxy perspective until I click "Leave proxy"

When I click a record, I remain in the proxy context
When I create a new record, I go to the regular context

Navigation

[x] Dashboard and Proxy dashboard navigation is hidden
[x] The sidebar colour switches
[x] A button appears to leave the proxy view, and if I click that button, the navigation turns back to 'normal'
[x] I can see who I am proxying for in the title

Publications and datasets
https://booktower-ugent-ui.vercel.app/biblio/researcher/publications/overview--proxy.html

[x] The tabs have a different colour to make clear you are managing someone else's bibliography

Out of scope
Suggestions, see https://github.com/ugent-library/biblio-backoffice/issues/1348
https://booktower-ugent-ui.vercel.app/biblio/researcher/researcher-suggestions/researcher-suggestions--proxy.html

Feedback

dropdown menu is confusing, make clear

all done


Move to bootstrap 4 and bootstrap_form 4

Thanks for creating an issue! Please fill out this form so we can be sure to have all the information we need, and to minimize back and forth.

What are we trying to do?

We're a major version behind on bootstrap, which wouldn't be a terribly huge deal except our form builder is also moving on, and I want to make sure we're taking advantage of what it offers us in terms of security. So we should move to

What feature or behavior is this required for?

Long term health of the project, rails security, the ability to use form_with in code.

How could we solve this issue? (Not knowing is okay!)

At a minimum, gem 'bootstrap' and bundle update bootstrap_form, then work to make all the tests pass again and make sure the views don't get wild.

Anything else?

This is going to be a lot of work, as our CSS is probably the least touched part of this project, but it is absolutely worth doing so we can keep our form building up to snuff.

This is underway. We're doing it in such a way that at the end we'll still be in sprockets, but will be firmly on bootstrap 4 (with much cleaner HTML as a fringe benefit).
After that, we should ticket moving our BS4 install from sprockets to webpacker, and then write a gameplan for a JS framework implementation if that still feels like a good idea.


Add writing support

Test using this:
http://www.sheepwalkelectronics.co.uk/product_info.php?cPath=23&products_id=59

I would like to promote this feature request.  In my 1-wire network there are not only sensors but also actors like DS2405. To be able to switch them would be great.

Request noted. Will try and find some time to get my DS2413 tested/working.

Tried adding write support, but there is a bug in version 0.2.3 of the owfs node module that means writing doesn't work.
A fix has been contributed:
https://github.com/benediktarnold/owfs/pull/14
But it has not been released.
Additionally, because it is written in coffee script, installing directly from git doesn't work - nothing is re-compiling the JavaScript.

I am now the maintainer of the owfs node package. I have released version 0.2.4 that fixes support for owfs.
Writing support added in f522abafe59e045781c1ea4a96055b2daa13a4b4.
Going to do some more testing, then do a release to npm.

I have been running this for quite a while without any problems.
Released version 1.2.0 today.


[Feature] Setting to disable copilot

Motivation
We're running MQTTX inside our organization for debugging our internal MQTT network. We don't want to potentially leak any confidential information to OpenAI through the copilot integration.
Detailed design
Basically just a switch in the settings menu to disable any communication with OpenAI/copilot/any online AI assistant.
Alternatives
For now I've set an invalid API key but I might as well set a line in /etc/hosts to point any OpenAI domain to localhost. I'd rather MQTTX never tried contacting any AI assistant at all though. We could recommend our developers to use another MQTT client but MQTTX is the best one we've tried for debugging so far. Another temporary solution would be to downgrade to MQTTX 1.9.6 but then we'd be stuck not getting any more bugfixes. In theory we could maintain an internal fork with the copilot code ripped out but I don't think maintaining that would be viable long-term as the MQTT team is very small.
More detail (optional)
Personally I want a setting to disable any AI assistant whether it be online or offline, but that's probably just me.

Hello @Djhg2000,
Thanks for using MQTTX! We understand the importance of maintaining the confidentiality of your internal MQTT network and want to make sure you know that we are our top priorities.
To address your concern, MQTTX is designed to communicate with OpenAI or any online AI assistant only if a valid API key is provided. If you do not set an API key in the settings, the application will not attempt to establish any connection with OpenAI's services. This ensures that your data remains within control and is not shared externally.

However, we recognize the importance of providing explicit controls for our users. Because of your feedback, we plan to introduce a switch in the settings menu to turn off any interaction with Copilot or any online AI assistant altogether. This will provide an additional layer of assurance and allow organizations like yours to use MQTTX without any concerns about data leakage.
We appreciate your suggestion and understand that while the Copilot feature can benefit many users, especially for testing and debugging with public data, it is equally important to respect the needs of users handling sensitive information. The upcoming switch will ensure that you can comfortably continue using MQTTX, which you've found to be the best tool for debugging, without the need to resort to workarounds or maintain an internal fork.
Thank you for your understanding and your continued support. We're committed to making MQTTX the best tool for all our users.

@Djhg2000 Thanks for your feedback. Now the https://github.com/emqx/MQTTX/releases/tag/v1.9.8 supported


Missing Station Length Imputation

We used length of the station to calculate delay, and vht. This calculated vht again used as the input of q value calculation and q value for tti. The impacts of null station length is like chain reaction impacts. We need to come up with ideas that can impute the missing station length for better quality performance metric calculation. We have around 6281 distinct missing station length out of total 17056 unique station length in five mins performance file. Overall 1/3 of the length is missing which greatly impacts the performance metrics

I'd like to better understand the cause of the missing lengths. Are they for stations that don't have a neighbor within a reasonable distance? Or some data quality issue?
In theory, I think we could also compute some of the lengths using the postmiles of neighboring stations, unless I misunderstand the meaning of the field.

Are the missing station lengths for station types ML and HV? I agree with @ian-r-rose that with the absolute postmile data we can compute lengths but I would like to know a little bit more about which stations are missing the length and why.
For the definition of station length you can review the PeMS User Guide pages 31 and 32:

These are the statistics of missing length from five minutes performance table

So, this is good, then? If we only expect ML or HV lanes to have lengths, then it seems like they are all populated

yes, this is expected and we are good in quality of data for performance metrics