Expose IPFS API as window.ipfs

There is an area of interest that can be summarized as..

..exposing a subset of IPFS APIs as window.ipfs on every webpage ✨

tl;dr If we have it, websites could detect that window.ipfs already exists and use it instead of spawning own js-ipfs node (saves resources, battery etc).
We had discussions in https://github.com/ipfs/js-ipfs-api/issues/387 and https://github.com/lidel/ipfs-firefox-addon/issues/37, https://github.com/ipfs/in-web-browsers/issues/9, https://github.com/ipfs/in-web-browsers/issues/35  but the consensus at the time was that it is not safe without some kind of access-control (eg. via proposed API Tokens: https://github.com/ipfs/go-ipfs/issues/1532).
@VictorBjelkholm  suggested a simple solution to the lack of access controls in the API itself:
On the first use of window.ipfs user could be prompted with something like "Do you want to allow scripts at <website> to access API of your IPFS node?". The choice would be remembered eg. for the page til the end of browsing session.  There could also be an additional "remember for this site" checkbox to make UX better for users who wants to make permission persist between browser restarts.
Tasks

[ ] Create a content script that creates a proxy object under window.ipfs of the page it is injected into

[ ] This object should expose  a subset of IPFS APIs

<victorbjelkholm>  there is of course a lot more once there is a full ipfs node in the browser, but if we only have the simplest and most useful functions (add, cat, pubsub subscribe/publish, dag get/put), it suddenly becomes a lot simpler and useful quickly

[ ] Operations performed on window.ipfs should be proxied to the real IPFS API object present in webextension's Background page (requires robust (de)serialization, we don't want to mangle upload data etc)
[ ] First use of window.ipfs should trigger a dialog where user confirms that access to IPFS API is granted to requesting site (and decides if permission is for this session only or permanent)

[ ] Add a checkbox under "Preferences/Experiments" to control if content script  is injected to every visited page, make it disabled  by default initially

Potential problems

[ ] Something to investigate and solve before we make window.ipfs enabled by defaul: make sure it is not possible for a website to approve itself 🙃  This may mean we can't inject dialog to the tab itself and the dialog needs to be displayed  in a new Tab or via other means.

I've made two PoCs on this concept. One was separate and you can see it here: https://github.com/victorbjelkholm/js-ipfs-in-the-browser
The other one was a recent try to integrate that work into ipfs-companion and fix the stream issue in .cat, but didn't get really far yet: https://github.com/ipfs/ipfs-companion/commit/6b0ac8b6c0dbb45928a6e2ecaffed5d5e26f0ab7

suggested a simple solution to the lack of access controls in the API itself:
On the first use of window.ipfs user could be prompted with something like

I don't think that's good enough. I think (judging only five different methods) that the functions themselves should trigger the permission dialog (all with a "remember for this site" checkbox)

ipfs.id would trigger a "website would like to know your ID" prompt
ipfs.cat would trigger a "website would like to get content from the IPFS network via you or content in your local node" prompt
ipfs.pubsub.subscribe would trigger a "website would like to read messages from a pubsub channel via you"
ipfs.pubsub.publish would trigger a "website would like to send messages as you"

make sure it is not possible for a website to approve itself
Should not be possible if we store it in the extensions storage.

This may mean we can't inject dialog to the tab itself and the dialog needs to be displayed in a new Tab or via other means.

We absolutely should not inject the dialog into the page itself, or via a new tab but rather use the popup that ipfs-companion provides. Take a look at how MetaMask deals with confirmation when you do a transaction.

@VictorBjelkholm thanks, great points!
For future reference, metamask webextension:
https://github.com/MetaMask/metamask-extension/tree/master/app

@lidel can a quick experiment to let users access window.ipfs after enabling a flag on the settings pane? @joaosantos15 would really appreciate this.
@joaosantos15, perhaps you can grab @VictorBjelkholm PoC and submit a PR to station?

@diasdavid there is some very early wip for that here https://github.com/ipfs/ipfs-companion/pull/333
It's building on the ideas in https://github.com/victorbjelkholm/js-ipfs-in-the-browser
adding window.ipfs as a proxy to a ipfs instance running in ipfs-companion. Only ipfs.id() implemented so far... there are a lot of hurdles to jump through to make it work.

Just like @diasdavid suggested, we can work on PR #333 as an experiment that is disabled by default (another checkbox in  Preferences 🙃 ). I feel that as long  it is an "opt-in experiment" we can ship it as a crude PoC, even without authorization GUI (although it may not be the hardest part, as noted in https://github.com/ipfs/ipfs-companion/pull/333#pullrequestreview-83146426).

Thanks @lidel @olizilla. Yes, shipping it as an opt-in would enable @joaosantos15 use case for now, that would help a ton :)

@diasdavid  @joaosantos15 out of interest, what is the use case?

glad you asked :) it is for @joaosantos15 thesis -> https://github.com/ipfs/notes/issues/268 & https://github.com/inesc-id/hypercerts

@diasdavid given all the limitations of proxy-ing across to an ipfs in the webext background page, would it be of any use if simply provide js-ipfs as a constructor atwindow.IPFS in the meantime?
We can do both, it's just that a useable window.ipfs proxy is a little way off yet. So let pages construct their own jsipfs instance until we can proxy through to a shared one?

Hello, sorry it has taken me this long to reply, and thank you for the help! :)
@olizilla, essentially what I need it to connect my web extension to the go-ipfs daemon (both running on the same machine). What you suggested,

provide js-ipfs as a constructor atwindow.IPFS in the meantime?

would make that work, right?
Thanks!

@joaosantos15 ah, if you are developing a webextension, be sure to see our other proposal:
"Provide API for other browser extensions". Perhaps it is a better match?
Let us know which approach is better for you (runtime.onMessageExternal vs window.ipfs) and we can prioritize accordingly.

Right now my extension is pretty simple, it only injects some js to change the layout of the webpage, I'd prefer window.ipfs.

@joaosantos15 to add and get files with an ifps daemon you'll want window.ipfs set up as a proxy to an ipfs-api client hosted in the background page in ipfs-companion. The work in #333, when it's ready, will make that possible.

@joaosantos15 you could already today use js-ipfs-api easily to build what you need. The Ports or window.ipfs would just be a optimization to minimize dependencies, but what you need is already possible today.
If you check for window.ipfs or a ping message (interface to be defined) over a Port, before actually using it, once we have this feature it should work automagically.
Example:
if (window.ipfs) {
  // do whatever
} {
  // logic for loading js-ipfs-api, require/import async
  node = new ipfs()
  node.once('ready', () => { /* do whatever */ })
}

With this example, your application will work today and once window.ipfs or the Port is available, you'll automatically start using it.

@VictorBjelkholm that's great. I'll go ahead and do it that way and when #333 is ready I'll switch to that. Thanks for the help, everyone 😄

Hey, some progress here:
https://youtu.be/t1ldUp_mjDk

window.ipfs property!

(also adds window.Buffer if not there)

proxies to running js-ipfs in the extension

video demonstrates shared repo across tabs

Buffers sent across postMessage

WIP PR for companion here: https://github.com/ipfs/ipfs-companion/pull/333
IPFS postMessage proxy here: https://github.com/tableflip/ipfs-postmsg-proxy

@alanshaw  to make sure we won't miss anything,  I've extracted remaining work into two issues:

Isolate ACL for window.ipfs on IPFS paths (assume HTTP gateway) (#375)
Custom dialog for window.ipfs ACLs (make it work under Firefox) (#377)

Do you feel there is anything else we should add as a dedicated issue?

This is only for web pages and not service workers, correct? What should I do in order to get access to the local IPFS instance from a service worker?

Rough POC - https://github.com/ipfs/ipfs-service-worker/pull/7

Thinking through this a bit, on the permissions model, It would be really nice to be able to limit which directories in the files api that pages have access to. Maybe give them /apps/DOMAIN/ or something?

@alanshaw more like, restricting calls like ipfs files cp, ipfs files read, ipfs files ls etc to operating on paths under: /apps/example.com/ or so.

We need to scope things based on not just the origin but the path as well. So an IPFS application would be scoped to /apps/ipfs.io/ipfs/Qmasd/. It's actually quite a nice model, and could also open up ways of applications sharing data via common directories, kind of how KBFS (from Keybase) works. /keybase/private/victor is mine, /keybase/private/victor+why is mine + why's.

I’m not sure I’m understanding you both properly...the permissions are scoped to origin + pathname.
If you allow to /apps/example.com then anything under it will get access, but /apps/another.com will not.

@alanshaw I'm talking about paths within the ipfs files api.

Ah, got ya. I didn't pick up that you meant the actual paths of files stored in IPFS.
@VictorBjelkholm has suggested disabling all access to files.* until we can implement this. Which would allow us to ship this feature.
Thanks for taking the time to evaluate this @whyrusleeping - do you have any other concerns? I'm also wondering if there's more people that we should get to review this also...?

So just to flesh this out a little - we're talking about the MFS specific functions of ipfs.files.
When accessing MFS functions via window.ipfs your changes are limited to directories with the same scope as your application and only if the user grants permission.
e.g. if you application is running at https://example.com/mycoolapp then you can only access files and directories in or under /https/example.com/mycoolapp
To distinguish this scope from general access permissions scope we'll call it mfs-scope.

files.cp - both source and destination path must be below mfs-scope
files.mkdir - path must be below mfs-scope
files.stat - path must be mfs-scope or below
files.rm - path must be below mfs-scope
files.read - path must be below mfs-scope
files.write - path must be below mfs-scope
files.mv - both source and destination path must be below mfs-scope
files.flush - path must be below mfs-scope
files.ls - path must be mfs-scope or below

For the beta release I'll temporarily disable access to the above functions until I can implement the scope rules.

Is it worth namespacing access to companion? e.g. /companion/https/example.com/mycoolapp

I think it is a good idea, having https and https in root dir looks like a bad UX.
Perhaps /dapps/https/example.com/mycoolapp would be a better name? or just apps-data ?

I've been mulling this over and had a thought - should mfs-scope be transparent? So the app would read and write to /*, but that would be internally mapped onto /app-data/https/example.com/*?
I think that would be cool, but I don't know if there's any repercussions to doing this that I've perhaps not considered.

MFS is local and it should be fine for different apps to see different roots.
Transparent sandboxing of ipfs.files would be just a syntactic sugar, but makes developer's life more pleasant.

BREAKING NEWS ✨ PR is here ->> https://github.com/ipfs-shipyard/ipfs-companion/pull/431
Demo is here => https://youtu.be/KvgxAafphPg

@whyrusleeping @VictorBjelkholm something that came up today was sharing data between different versions of an application.
The MFS scoping does a good job of stopping other applications writing all over your stuff and each other's stuff but it means that if I have an app at ipfs.io/ipfs/Qm0 and I update it to ipfs.io/ipfs/Qm1 then ipfs.io/ipfs/Qm1 no longer has access to the files saved by ipfs.io/ipfs/Qm0, even though they are the same app.
It's been suggested that IPNS is the answer to this, but I wondered what your thoughts were?
We also discussed an alternative to MFS scoping - instead ask the user for permission to read/write to a directory (and sub directories).

Maybe this will help:
Let's take a look at Origin-based security context that isolates local storage, cookies etc in the regular web. Note that there is no GUI for opting out of Origin-based isolation per website. We have low-level CORS dance just so that one page can access API of the other.  But no GUI for escaping Origin sandbox. That is for a good reason.
Here comes my main fear: as soon as we introduce UI for apps to ask user for access outside of its default security perimeter, we will end up with Windows Vista UAC-like hell of nagging dialogs.  Of course every app will want access to the root of your ipfs.files. Should it? IPFS introduces interesting space for innovation, as we can have zero-cost copies of the same data across multiple sandboxes.
I feel we should at the other direction: remove friction (dialogs) for every sandboxed method. Look closely at various methods and decide which ones are safe to be executed without ACL prompt.
Otherwise users will just learn to press "alllow" without reading, which defeats the purpose of ACLs.

MFS Chroot/Sandboxing Just Landed
PR #431 just got merged. This means your app's MFS root directory is based on the origin and path where your application is running.
Just for the record, I was playing with running this on various sites:
ipfs.files.write('/test-from-' + window.location.host, Buffer.from('ehlo'), { create: true }).then(() => console.log('done'))

and sandboxing of ipfs.files works as expected :ok_hand:
Open Question: Merge IPFS Scopes?
It gets interesting when accessing ipfs.files via window.ipfs at IPNS site.
I've run the snippet above at /ipns/ipfs.io and:

when executed at Public Gateway https://ipfs.io/ipns/ipfs.io/
creates /dapps/https/ipfs.io/ipns/ipfs.io/test-from-ipfs.io
when executed at Local Gateway http://<IP_ADDRESS>:8080/ipns/ipfs.io/
creates /dapps/http/<IP_ADDRESS>%3A8080/ipns/ipfs.io/test-from-<IP_ADDRESS>:8080

Do we want to merge IPFS scopes (/ipns/ and /ipfs/) by removing HTTP component so that both above URLs will have access to the same ipfs.files root at /dapps/ipns/ipfs.io/  ?

Closing this issue: IPFS Companion v2.2.0 shipped with window.ipfs enabled by default.
Remaining work related to refining security and UX constraints for window.ipfs  will be tracked in https://github.com/ipfs-shipyard/ipfs-companion/issues/454.
The window.ipfs label can be used for tracking all issues related to this endeavor.


⚠ Slack has degraded performance

In 4c982de, Slack (https://status.slack.com/api/v2.0.0/current) experienced degraded performance:

HTTP code: 200
Response time: 201 ms

Resolved: Slack performance has improved in b2534fd.


Update and fix eslint rules

This PR:

Turns off some security rules that were only generating false positives
Turn on rule to disallow explicit any
Remove angularjs from linting

This should remove the "Unchanged files with check annotations" section at the bottom of our PRs.

Coverage increased (+0.002%) to 83.702% when pulling 011b388edc4727f4d41c41b567c29322b525a8b5 on update-eslint-rules into e4052aaf3db6192b3ad2399c9291c08a7dc8095f on dev.


feat(lib): Setting main target to export-lib.js

:tada: This PR is included in version 1.3.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


chore(ci): Modernize the plugin

This PR aims to modernize the plugin by allowing it to compile with JDK21 on ci.jenkins.io.
It has been tested to compile, and to pass tests, on my local machine.
Testing done

Submitter checklist

[X] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
[X] Ensure that the pull request title represents the desired changelog entry
[X] Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
[X] Ensure you have provided tests - that demonstrates feature works or fixes the issue

@coderabbitai review


Enviornment Setting Issue

Hello, love ur work.
I have a setting issue when I go through your setting step.
my OS is Ubuntu 22.04, CUDA version is 11.7
I followed the suggestion :
conda env create -f environment.yml
conda activate dift
it's good so far.
but when I key in:
python extract_dift.py
--input_path ./assets/cat.png
--output_path dift_cat.pt
--img_size 768 768
--t 261
--up_ft_index 1
--prompt 'a photo of a cat'
--ensemble_size 8
an error occurs:
Traceback (most recent call last):
File "/home/xxx/Documents/Diffusion/DiFT/dift-main/extract_dift.py", line 4, in
from torchvision.transforms import PILToTensor
File "/home/xxx/.local/lib/python3.10/site-packages/torchvision/init.py", line 10, in
from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils  # usort:skip
File "/home/xxx/.local/lib/python3.10/site-packages/torchvision/_meta_registrations.py", line 4, in
import torch._custom_ops
ModuleNotFoundError: No module named 'torch._custom_ops'
do u encounter this before or could u give any advice. Thx first. @Tsingularity

I changed the setting version, it works now.


Create dataset_privacy table to combine compliance and security

Coverage remained the same at 0.0% when pulling c19e96617bd1c593cb9605f5bca7c4a38d41e076 on alyiwang:linkedin into 27e6c5435f0e36f5227309629bc85d7a443b52d8 on linkedin:linkedin.

Coverage remained the same at 0.0% when pulling 335bfd8da307b17edeb227062aadda846b8669b2 on alyiwang:linkedin into 27e6c5435f0e36f5227309629bc85d7a443b52d8 on linkedin:linkedin.


req.user from Sequelize causes server to hang

When an error is picked up by the connect errorHandler middleware, if that request contains a user object from Sequelize, the server hangs for a long time.
I assume this happens because Raven is trying to serialize the User object, and that gets into all kinds of weird things, including the Postgres driver itself.
Side note: This also causes #210 and #212, because the Postgres driver has a bug where if you enumerate all properties, it will try to require pg-native, even if it's not in your dependencies.
Proposed Solution
There are a few ways to resolve this.  I prefer the first.
Request sanitizing function
Allow a function to be passed into the middleware that accepts a request and returns a sanitized request.  My implementation would look like this:
(req) => {
  const { user, ...sanitizedRequest } = req; // Remove user from req
  sanitizedRequest.user = { id: user.id }; // Add userId for context
  return sanitizedRequest;
}

Recognize Sequelize models in req.user
Since req.user is popularly used with Sequelize, in parseRequest we could recognize that it's a sequelize model and call req.user.get({ plain: true }) to make it a plain JS object.

Awesome report/breakdown @idris - thanks! This brings a lot of clarity around that pg-native issue. I'm definitely on board with fixing this in one way or another; Sequelize is way too popular for us to not work with. For now, the closest thing to a workaround is rolling your own slightly-modified-but-mostly-just-duplicate of the error handler middleware to call parsers.parseRequest(req, { user: req.user.get({ plain: true })); instead of parsers.parseRequest(req);, and that's pretty rough.
In general I think it's better to avoid ecosystem-member-interop-specific code paths like what you describe in option 2, so I would be inclined to agree with you on the request sanitizing function as the flexible solution (even though option 2 does make a simpler UX in this case). I think that function would fit better as a configurable option on the Raven instance rather than an option to the middleware, though, and I still need to figure out how user context data should fit in vs user data off the req obj, which one should take precedence, should they merge or just either-or, etc, but that isn't really the core issue and I think those details will fall in to place.
The other point of consideration is that with upcoming breadcrumb support, we're going to have to have a fair amount of ecosystem-player-specific code for automatically capturing queries, logs, etc from various popular libraries. I'm still getting my thoughts in line on that, but it makes me wonder if maybe library-interop-specific code isn't such a bad thing.
If you need a fix for something right away, I think the self-rolled-error-handler I described should work as a stopgap, but I'll figure this out and get the nicer fix into a raven-node beta release soon.

Re: workaround, I just wrote a super simple wrapper around your middleware to avoid copy/pasting code:
  app.use((err, req, res, next) => {
    const { user, ...sanitizedRequest } = req;
    if (user) {
      sanitizedRequest.user = { id: user.id };
    }

    return raven.middleware.express.errorHandler(SENTRY_DSN_SERVER)(err, sanitizedRequest, res, next);
  });

I think probably the best solution would be to add some sort of callback that gets called by the errorHandler middleware that we could use to include / exclude user context. That would probably be the most straight forward I'd say! Something like this:
app.use(Raven.errorHandler({
    userContext: (req, res) => {
        return {
            user_id: req.user.id
        };
    }
}));

Looking at the code it should be easy enough to add in 👍

Maybe instead of just taking user, we (attempt to) grab a bunch of predefined fields. I'd prefer this worked as "magic" as possible without configuration, but obviously stalling the server is bad.

@MaxBittker – I can tell from the +1s that this is probably an important issue. My suggested easy fix is that, instead of just copying the user object wholesale by default, we only copy a handful of suggested properties: username, email, id, and perhaps the fields documented in passport's API.

Addressed in #274; will publish a new version to npm tomorrow.

The server doesn't hang anymore, but it seems it's not working as expected,
It's not parsing any fields from req.user if req.user is a sequelize instance.
I'll try to look at this again to confirm and/or propose a solution.

@seromenho as of #274 we provide a parseUser option which you can use like this:
Raven.config({
  parseUser: function (req) {
    return req.user.toJSON();
  }
});

@LewisJEllis Oh yeah that!. Thank you.


Internal user research analysis

Goal of Ticket
We've got a lot of great insights and information from CAS staff - now what did we learn?
Describe the task
Analysis of internal user research.
Acceptance Criteria

[ ] Compile findings from internal user interviews
[ ] Pull out challenges, opportunities & insights
[ ] Create summary of what we learned

Hey team! Please add your planning poker estimate with Zenhub @NicoleGovvy @jaimiebutton


Global environments seem not to be fully separated from each other

Checks

[X] I have checked that this issue has not already been reported.

[X] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example
FROM ghcr.io/prefix-dev/pixi:latest
RUN cd /root
RUN pixi global install ipython cowpy
RUN pixi global install pytest

# note: cowpy can be imported from ipython, but pytest cannot.
# Observation so far: packages that have been installed in the same `pixi global install ` command can import each other
# and are thus not truly in different environments:

RUN echo "This is not meant to work:"
RUN /root/.pixi/bin/ipython -c "import cowpy"
RUN echo "This is okay to fail:"
RUN /root/.pixi/bin/ipython -c "import pytest"

Build the docker image by putting the above into a Dockerfile and run in the same directory docker build -t issuepixi ..
Issue description
As shown in the Dockerfile (as a clean environment we trust), it seems that the different .pixi/venvs can somehow interfere which each other.
My hypothesis is that this connection between the different environments exists when they are installed together in a single pixi global install command.
Expected behavior
In the container above, ipython should not be able to import cowpy if they are truly separated.

While investigating it seems that cowpy is placed in the ipython site-packages.

I've quickly investigated and found the source of the issue. Whats actually (incorrectly) happening is that when you run pixi global install a b you get two environments, one for a, and one for b. But both environments will contain the same packages!
However, it also looks like some people currently rely on this behavior...
But we are working on a complete revamp of pixi global that will not exhibit this behavior so to not potentially break peoples workflow twice we'll wait for the revamp instead of providing a fix now.
The "solution" for this issue is for now to call pixi global install for each individual spec instead of combining:
pixi global install ipython
pixi global install cowpy

instead of
pixi global install ipython cowpy

Hi @baszalmstra - thanks for investigating.
Ironically, I was also hoping that the current behaviour is the intended behaviour. (I was looking for a way to provide a global anaconda-like installation which I can recommend to students, and where we can ignore the complexity of environments etc for the first few weeks of learning Python.) Only when looking at the documentation and double checking with @ruben-arts and @wolfv at EuroScipy, I thought this might be bug and the team should be aware of it.
As long as this is on the radar of the pixi team, and we have the issue to document it, that's good. Feel free to close or keep open the ticket - whatever works best for your work flow. From my side, the reporting of the behaviour is completed.

The new pixi global in the v0.33.0 release fixes this.


Improve MVVM support for some Toolkit controls

Search and FloorFilter both depend on setting a DataContext in the constructor. This doesn't work if there is an externally-set DataContext, which is quite common in MVVM scenarios.
The quick fix is to add a wrapping FrameworkElement in the control template, and explicitly set the DataContext to {x:Null}, then set the DataContext on the child.

Custom controls should not set the datacontext anywhere, but instead use TemplateBinding.


The context menu item "Open PowerShell window here" to remove the set-executionpolicy

Steps to reproduce

## Steps to Reproduce: (Add details for how we can reproduce the issue)

"if((Get-ExecutionPolicy ) -ne 'AllSigned') { Set-ExecutionPolicy -Scope Process Bypass }” to be removed when invoking a script as runas.

Expected behavior

Actual behavior

Environment data

Which version of PowerShell are you using? I can't reproduce this on v7.1.3.  What is the output of $PSVersionTable?

@rkeithhill Applies to all open versions.

@krishnayalavarthi The issue doesn't describe the problem, and it doesn't have repro steps and environment data. Hence, I will close this issue. Please following the issue template to provide all needed information when you open a new issue, thanks!


set content-type application/json for generated requests with a json body

Multiple Azure services reject requests without an appropriately set
content-type.  Application Insights is one such.
Long term, this should check the consumes part of the endpoint spec, which every spec I spot checked, if it has it defined, is ["application/json"].
This PR sets the content-type based on the assumption that "any time the BODY is json encoded, we should probably set the content-type".
I split up the PR into two commits:

The generator changes: https://github.com/Azure/azure-sdk-for-rust/commit/d9e890b95d798195182b6e8bfbc73bd8d66f4c09
The generated code: https://github.com/Azure/azure-sdk-for-rust/commit/7f9e63d02976243206199fa03e371cb480dbeb83

Created #398 as a followup.


Changes made to NITK Life Static Page

Changes have been made as discussed on yesterday's chat. Each article has separately been committed so if I'm making a mistake do let me know about it.

@salman-bhai  Test your changes before making the PR! I'm closing this now. Open a new one when you're done.


NIFI-6197 HBase: Fix deprecations, default for client retries & validations

This is a continuation of #3418 which I managed to somehow break.

This fixes deprecations in HBase 2.0 which will be removed in 3.0. Amongst other things the typeByte, sequenceId and tags are now considered internal and are not exposed anymore.

I also made it so that only either an hbase-site.xml or a ZooKeeper quorum need to be provided, previously ZooKeeper Client Port, Client retry and ZK Parent Node were mandatory as well but they are not really needed to connect to HBase

Removed the default for HBase client retries. This means it can still be overridden if needed but if not the defaults from either hbase-site.xml or HBase itself will be used

Some minor warnings (e.g. too broad scope, unchecked cast etc.)

@bbende  Thank you for your time in discussing & reviewing the last one. I hope you have the time to look at this one as well (I'll update the Jira ticket as well)

Overall the code looks good. I'm still on the fence about removing the default value for retries...
I believe the most common scenario where people encounter the retries is during initial setup of the service when they have misconfigured something, or have a network issue, and in that case having the service retrying 31 or 15 times in the background with no info in the NiFi UI is not very helpful because its not going to ever work, and with increased delays it may take many minutes to wait for it finish and many users believing it is stuck.
I'd be onboard increasing to something like 5, and then for users like yourself that believe you need it higher then you can always change it to whatever you like. Also, the 1_1_2 client service currently uses the 1.1.13 hbase-client which still has 31 retries which is way too much.

I disagree but I do understand that we don't want a bad user experience :)
I believe the most common scenario is during normal operation. Because having a region moved, split, merged or writes blocked (due to all kinds of reasons) or a server crash is totally normal in HBase. And if you have a long running flow this'll happen multiple times a day on a busy cluster.
Enabling the service will ideally only ever happen once (or very few times).
I don't have time to look now and don't know the API very well but can we use a low retry number for enabling to see if it works and if it does then reconnect with the "proper" number? Shouldn't add much latency.
5 is too low for normal operations. HBase uses an incremental backoff for some of those retries. I believe the current (HBase 2.x) default of 15 to be fine.

I could be wrong, but my reasoning for thinking this is that the 1_1_2 service has been in existence and heavy use for almost 4 years, and during that time we haven't heard any issue with the retries until your situation just now. We have heard of many supposedly "stuck" issues during initial setup where someone gets the config files with hostnames not accessible from where NiFi is running. I guess it is very possible that everyone has just increased the retry value themselves and we never heard about it.
I would like the idea of different values for first connection vs normal ops, but I'm not sure how to implement that since this retry concept is all part of hbase-client code.
The exponential back-off is part of what I don't like about the 15 or 30 retries during first connect because it may take over 5 minutes before enabling the service truly fails and can be seen in NiFi.

I can only provide anecdotes as well but I've now had two customers faces this issue in a short time which lead me to look into it. And we had good reasons to set this number to the current value in HBase :) In HBase 1.x the MTTR wasn't as good as it is now in 2.x so the retry count can be lowered in newer versions. Another good reason to not hardcode the default in NiFi, the defaults change the more we learn and work on the stability and MTTR.
I don't fully understand the Javadoc on AbstractConfigurableComponent#customValidate but if it's only called after all properties have been set and the user clicked on Apply/Save then we could establish a connection (and tear it down again) there to validate whether the details are correct.
Another option is to make a connection with retry = 1 in onEnabled and if that works tear it down and reconnect with the proper number. This will mean that enabling will take longer than before even if everything's good.
If any of those is acceptable to you I'll work it into the patch.

The customValidate method is called as part of validation after all property level validators have passed. Validation is called frequently which is how we can show the status of the component on the canvas as valid or invalid, so we definitely don't want to be creating connections during validation.
The only way to have separate values for first connection vs other ops, would be to set the hbase-client retries to 1, and then build the retry concept into our own code for the other operations. Not sure I love doing that since it would require quite a bit of work in both hbase service implementations.

Maybe I explained badly. What I meant is to change the onEnabled method to look something like this (semi-pseudo code):
    @OnEnabled
    public void onEnabled(final ConfigurationContext context) throws InitializationException, IOException, InterruptedException {
        // Try connecting with client retries set to 1 here to fail fast if something's wrong.
        connection = createConnectionTestConnection(context);
        connection.close();

        // Now create the real connection
        this.connection = createConnection(context);
....

Just for reference: With all default settings these will be the retry values/backoff times) for each retry:
100ms, 200ms, 300ms, 500ms, 1s, 2s, 4s, 10s, 10s, 10s, 10s, 20s, 20s, ...
So with the default retry of 15 in HBase 2 that'll be 129,1s
For 35 retries (HBase 1) that'll be 528,1s.
I totally get that this is not great to wait for during a connection timeout but 1-1,5 minutes is how long a crashed server recovery can take and I'd rather take the one-time hit than problems during runtime. We are working on getting that down but I believe it really is not a good idea to go lower than the default values (35/15).
If you insist I could set an explicit default of 15 for HBase 1 and rely on the hardcoded default for HBase 2 (which should hopefully just go down in newer versions).

@bbende ping, if you have another few minutes that'd be great.

@bbende FWIW, I think I've had an experience similar to @lfrancke. When I was ingesting genomic data, the basis for creating PutHBaseRecord, we often got hit hard with HBase choking on the volume of data (wasn't a terribly big cluster, just a few nodes).

Sorry, but my stance has not really changed. I'm willing to make the default retries 7 which is a compromise in the middle between 1 and 15, and with the back-off times it would be about 8 seconds of retries where the user doesn't know whats going on, which I can live with. If you need more than that then you can configure it to 30 or whatever you need.

Understood, thank you. I don't think that's a good compromise so I won't implement it but I had an earlier suggestion that I'd be willing to give a try if you'd accept it: https://github.com/apache/nifi/pull/3425#issuecomment-482711295

While most of the time that probably would work (making two connections), the second connection could still fail for some reason. We don't really create two connections like this for any other processor that talks to an external system, and I also don't think many other client libraries have built in retries. You will still need to build the error handling logic in the flow for when a failure happens after 15 retries, which would be the same if it was after 7 retries.

Yes, that's true. The second one could fail but having already established one before we at least know that security works, network connection etc. so all the time consuming stuff should be checked at least.
There is a really really good reason for those HBase defaults and NiFi is the only tool I know that goes lower than that, it not only breaks expected functionality but also expectations (at least mine) so I'd really like to get this fixed properly.
8 seconds is not enough time :(

Hi @bbende I just wanted to check again and see if you'd accept a Pull Request with the proposed two-connection-setup "hack"? If so I'd provide one, if not I can close this PR.

Sorry lost track of this... I'm not really in favor of the double connection.

Alright, thank you.


Post Form with data and Header Properties

I am doing some web scraping using puppeteer.I came across a href link(in "form" format) which i need to download.I need to use the http post method to post a form and also specify the request header !How do I do that?
URL: https://www.xyz.com
Method: POST
Headers:
Accept: text/html, application/xhtml+xml, /
Referer: https://www.xyz.com
Accept-Language: en-US
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Host:
Content-Length: 5035
Connection: Keep-Alive
Cache-Control: no-cache
Cookie:

@priyankapuranik, Why did you close it? How did you manage to do that?

You just use a normal fetch or XMLHttpRequest to do the data submission. Nothing special in puppeteer to make it work.


Setup clang-format in build_devtools and build_tpl scripts

PR #291 changed our code formatting from uncrustify to clang-format and manually updated the host-configs.
We now need to update our build_devtools script to use clang-format, and our axom spack package to add a line for clang-format instead of uncrustify.

@white238 added this for clang10.

@white238 added this for clang10.


Plugin react-tap-event-plugin not found, compile error

Looks like we need to add react-tap-event-plugin to the dependencies for the client-side code?
><EMAIL_ADDRESS>start /Users/kwhinnery/dev/twilio/hangouts-clone-react
> concurrently "npm run server" "npm run client"

[0]
[0] ><EMAIL_ADDRESS>server /Users/kwhinnery/dev/twilio/hangouts-clone-react
[0] > babel-node server.js
[0]
[1]
[1] ><EMAIL_ADDRESS>client /Users/kwhinnery/dev/twilio/hangouts-clone-react
[1] > babel-node start-client.js
[1]
[1]
[1] ><EMAIL_ADDRESS>start /Users/kwhinnery/dev/twilio/hangouts-clone-react/client
[1] > react-scripts start
[1]
[0] App listening on port 3001
[1] Starting the development server...
[1]
[1] Failed to compile.
[1]
[1] Error in ./src/components/app.js
[1] Module not found: 'react-tap-event-plugin' in /Users/kwhinnery/dev/twilio/hangouts-clone-react/client/src/components
[1]
[1]  @ ./src/components/app.js 16:27-60
[1]
[1] Compiling...
[1] Failed to compile.
[1]
[1] Error in ./src/components/app.js
[1] Module not found: 'react-tap-event-plugin' in /Users/kwhinnery/dev/twilio/hangouts-clone-react/client/src/components
[1]
[1]  @ ./src/components/app.js 16:27-60

Oops. Forgot --save on this one. Works now!


[Feature Request]

Add a option that
On app start up tachiyomi looks for reading progress in tracking site (if site is attached to manga) (tracking site like myanimelist and etc)
And if the the reading ch no. on tracking site is greater than in tachiyomi than mark the ch as read in app
Just like how tachiyomi sync ch from app to site
You just need to reverse it on startup of the app
It will be a great feature to have and it's possible to do
Why/User Benefit/User Problem
Can read manga on Multiple devices without remembering last read
What/Requirements
Hard Works😅😂

I probably wont do this, I think something similar has been pr'd to the main app before


FlannBasedMatcher throws exception on large inputs

I'm getting the following exception from the call to knnMatch (see code given below):
opencv/modules/core/src/ocl.cpp:4585: error: (-215) u->origdata == data in function deallocate
The exception is only thrown when the descriptor sets are very large (i.e. dense features for two images).
The exception is thrown when the scope of FlannBasedMatcher::add is exited (features2d/src/matchers.cpp), I'm guessing during deallocation of std::vector<UMat> descriptors, although not sure.
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/xfeatures2d.hpp>
#include <opencv2/flann.hpp>
#include <opencv2/imgcodecs.hpp>

int main(int argc, char* argv[]){

	cv::Mat im_l = cv::imread("im_left.png");
	cv::Mat im_r = cv::imread("im_right.png");

	cv::Ptr<cv::xfeatures2d::DAISY> daisy = cv::xfeatures2d::DAISY::create(30,6);

	std::vector<cv::KeyPoint> keypoints_l;
	std::vector<cv::KeyPoint> keypoints_r;
	cv::Mat descriptors_l;
	cv::Mat descriptors_r;

	daisy->compute(im_l,descriptors_l);
	daisy->compute(im_r,descriptors_r);

	std::vector<cv::DMatch> good_matches;
	cv::Ptr<cv::flann::KDTreeIndexParams> kdIndParams(new cv::flann::KDTreeIndexParams(5));
	cv::Ptr<cv::flann::SearchParams> searchParams(new cv::flann::SearchParams(50));
	cv::FlannBasedMatcher matcher( kdIndParams, searchParams);

	std::vector<std::vector<cv::DMatch>> matches;
	matcher.knnMatch( //OpenCV exception here
				descriptors_l,
				descriptors_r, matches, 2);
	filter_good_matches(matches,good_matches,0.75);

	return 0;
}

P.S.
The brute-force matcher also fails, because the number of descriptors in the train set appears to be limited by 1 << 18 (see IMGIDX_ONE constant at top of BFMatcher::knnMatchImpl). Why?
I'm getting
features2d/src/matchers.cpp:825: error: (-215) trainDescCollection[iIdx].rows < IMGIDX_ONE in function knnMatchImpl
At this point, I'm out of options in OpenCV and will have to implement my own dense feature matcher.
//To test, just substitute this instead of the FlannBasedMatcher into the code above
 cv::BFMatcher matcher(cv::NormTypes::NORM_L2, true);

P.P.S. for the FlannBasedMatcher, repeated calls to knnMatch on different rowRanges of the same input matrices also result in the same error,
opencv/modules/core/src/ocl.cpp:4585: error: (-215) u->origdata == data in function deallocate.
This can be avoided by copying the rowRange / window into a separate matrix before each call.

Every time when i use "FlannBasedMatcher" with "surf", "sift", "orb" in Python 3
return to me "Segmentation fault (core dumped)" and crash my code (and python executable too)
ps: Opencv 3.0, Python 3.4 - Linux/Ubuntu 64 bit.

@jhoonb this seems like more related to this bug here, as it deals with the bindings: https://github.com/Itseez/opencv/issues/5667

I'm seeing this as asymmetrical as well. Calling the brute-force matcher with 5K descriptors as the query and 500K as the target works, but calling BFMatcher reversing the arguments crashes with the error above.

I'm guessing that any of those matchers, BFMatcher, FlannBasedMatcher, or others, simply weren't intended for matching dense features in such quantities.
Aside: that actually makes sense, since dense features like DAISY have been shown to work well only when applied in conjunction with complex stereo-matching algorithms, such as graph-cut-based stereo and the like. I tried replacing the pixel cost function in the standard SGBM with DAISY, but that didn't work as well as the original*  for some reason. The code seems okay, but maybe I'm forgetting something critical.
*SGBM uses Birchfield-Tomasi pixel cost + difference in gradient of pixel's immediate surroundings.

Just for reference, using the stand-alone FLANN matcher works for large number of "target" descriptors. I've tested with up to 2 million target descriptors.


Added new filters to web-bluetooth device-info

R=@jeffposnick, @g-ortuno

:+1: from a code perspective. Unless @g-ortuno has any feedback from the API perspective, feel free to merge.

One small comment. Otherwise lgtm

@g-ortuno Can you just simply look at me and slap me please in the face? Thank you!

:) does name filtering work now?

It does ;)


[ottf, dif/uart] Writing to the UART in ISR may race with UART sending checks

Description
Problem
In dif_uart_send_byte(_polled), the logic is:

wait for TX fifo to not be full,
write byte to fifo,
wait for TX fifo to drain (polled version).

This function is used in e.g. ottf_putbuf for ujson (by the way not sure why we use the polled version for ujson?). But at the same time, all of this runs with interrupts enabled and one possible interrupt is ottf_console_flow_control_isr that may call dif_uart_byte_send that does NOT drain the TX fifo. Therefore, it seems possible that in some cases, an interrupt happens right after the TX fifo check in either dif_uart_send_byte_polled or dif_uart_send_byte, and the flow control decides to send a character. On return from the ISR, the TX fifo may still be full and the code will try to write to full fifo. If that happens, then we might drop a UART character by mistake.
The first step would be to confirm that this really can happen and that I haven't missed something.
Solutions
One possible solution is to disable interrupts when writing to the UART so that the TX fifo check + TX fifo write is atomic. However, as pointed out by @nbdd0121 , this doesn't help if an NMI writes to the UART.
Another solution would be that when writing a UART character to the FIFO in an interrupt context, to always wait for at least one character to drain from the FIFO. This guarantees that any TX fifo check done before the ISR still applies. The downside is that the ISR will have to wait at least one character length hence introduces a delay in an ISR which may not be ideal.

@nbdd0121 @jwnrt @timothytrippel @a-will

I think fundamentally, the OTTF console channel would need to become a managed resource. We can't have ottf_putbuf and ottf_console_flow_control_isr directly throwing characters into the UART FIFO in different contexts. This is just a general multi-context software problem, hehe.

I agree with that, generally using the UART DIF directly seems like a bad idea anyway (unless one is specifically testing the UART).
Maybe a good starting point would be to change the prototype of base_uart_stdout to take a structure with send/send_polled/recv methods (ie a mini driver). In parallel of this, a standard uart "driver" with this interface could be implemented that handles this race correctly.


No documentation available for chartjs-plugin-annotation

Why can I not find any documentation  on this library?
Or where is it hidden?
Searches take me to chart,js, which has a link to this github, but still no documentation.
Are we supposed to guess at the syntax?
Are we supposed to guess at the possible parameters and their possible values?
No documentation IS an issue.

tldr;
Docs for "this library" (annotation plugin for chart.js) https://github.com/chartjs/chartjs-plugin-annotation and scroll down a bit.
Docs for Chart.JS (required for working with this library) https://www.chartjs.org/docs/latest/
It's not clear to me which library's documentation it is that you're looking for so I'll include both of the possibilities that I think you might be looking for.
It's common practice on GitHub to keep documentation in a file called readme.md. That file can always be found in the root of the source. Hold on though! No one expects you to know that.
You'll find the contents of that readme.md are included (rendered) on the main page of the repository. That's also the default page you likely found earlier. That's also the page that has the source code file navigator on it. You can get there from here by clicking on the "<> Code" tab above. Hold on! Don't do that yet.
You'll need to scroll down below the code navigator.
Welcome to GitHub.

tldr;
Docs for "this library" (annotation plugin for chart.js) https://github.com/chartjs/chartjs-plugin-annotation and scroll down a bit.
Docs for Chart.JS (required for working with this library) https://www.chartjs.org/docs/latest/
It's not clear to me which library's documentation it is that you're looking for so I'll include both of the possibilities that I think you might be looking for.
It's common practice on GitHub to keep documentation in a file called readme.md. That file can always be found in the root of the source. Hold on though! No one expects you to know that.
You'll find the contents of that readme.md are included (rendered) on the main page of the repository. That's also the default page you likely found earlier. That's also the page that has the source code file navigator on it. You can get there from here by clicking on the "<> Code" tab above. Hold on! Don't do that yet.
You'll need to scroll down below the code navigator.
Welcome to GitHub.

Fair enough, but I can't get it working with React for the life of me, can you provide React examples for the chartjs v3.0.0-beta.7

tldr;
Docs for "this library" (annotation plugin for chart.js) https://github.com/chartjs/chartjs-plugin-annotation and scroll down a bit.
Docs for Chart.JS (required for working with this library) https://www.chartjs.org/docs/latest/
It's not clear to me which library's documentation it is that you're looking for so I'll include both of the possibilities that I think you might be looking for.
It's common practice on GitHub to keep documentation in a file called readme.md. That file can always be found in the root of the source. Hold on though! No one expects you to know that.
You'll find the contents of that readme.md are included (rendered) on the main page of the repository. That's also the default page you likely found earlier. That's also the page that has the source code file navigator on it. You can get there from here by clicking on the "<> Code" tab above. Hold on! Don't do that yet.
You'll need to scroll down below the code navigator.
Welcome to GitHub.

Fair enough, but I can't get it working with React for the life of me, can you provide React examples for the chartjs v3.0.0-beta.7

tldr1; You most likely have a race condition. Try using this.render() after you resolve your promise or async/await.
tldr2; The Issues tab on GitHub is where we developers, the consumers of the package, give support to the maintainers of a specific package. StackOverflow is where we give (and receive) support for (and from) the larger community where the integration and use of those packages is concerned.
I can see you have an issue, but it appears to me to be more related to your individual use of the code as opposed to a bug in the code itself. By all means, correct me if I'm wrong on that assumption. If I am wrong can you please include more details on any specific error messages you may be seeing? I would need more details to be able to give a more detailed answer. (This is the most common way we can help support the maintainers of any package).
However, based on what you've said so far the only thing I can think of that might assist you is to tell you that Chart JS is not reactive (it is event driven) nor is it's configuration object observable.  If you call render() before your data is returned from the server and before it's set in the config object (which is a deep object structure in and of itself and not directly observable anyway) you will have a race condition. In other words, you may be rendering the chart before the data is available to even be rendered and Chart JS does not react to those changes in your data model. Try using this.Chart.render() after you have resolved your promise, in .then() or after async/await. I don't know your specific use case as you haven't shared any examples of your code.
I've long mused that "Issues" is an inappropriate label for this section, but that's perhaps an example of an issue in and of itself. This is more so the place to report bugs (aka issues) relevant to the Chart JS code base. When someone has questions about the use of the code (eg "how do I..."), it's best to search StackOverflow and, failing any acceptable results, it's best to ask your question there. The Chart JS community from here also helps out there, but more importantly, React community members are there as well. Some subset of those two communities will also work with Chart JS and are probably more familiar with your specific situation than I am. I haven't touched React since switching to Vue 4+ years ago and even then I didn't integrate Chart JS.
The answer to "why" is that the "issues" here are tracked and associated to code changes. The "Issues" tab is meant to assist the developers track their workflow. The purpose of this section of GitHub is to give the community a way to support the developers of the package in question. We support them here primarily by sharing the steps to reproduce a bug in the software so they can fix it faster. I know what you might be thinking, why not offer support for the software they're creating and hosting on GitHub right here on GitHub? To which I will point to your own question above as an example. Your original concern involved the confluence of two unique packages; React and Chart JS. You can't really ask for help with Chart JS on the React JS GitHub page anymore than you might receive help here for your React question. I'll bet your first post is related to that conundrum. So then, where does one go? There needs to be a way for you to get help from a broader community with experience in both? That's where StackOverflow comes in. There are other sites out there besides Stack and many open source projects have dedicated forums. In this case, I would recommend checking and potentially posting on Stack.
Consider answering (at least) 2 questions on Stack for every 1 question you ask. If you aren't sure yet how to do all that or don't feel comfortable answering any questions yet, don't worry about it. Come back around to it later when you do. Karma is patient. Cheers.

tldr1; You most likely have a race condition. Try using this.render() after you resolve your promise or async/await.
tldr2; The Issues tab on GitHub is where we developers, the consumers of the package, give support to the maintainers of a specific package. StackOverflow is where we give (and receive) support for (and from) the larger community where the integration and use of those packages is concerned.
I can see you have an issue, but it appears to me to be more related to your individual use of the code as opposed to a bug in the code itself. By all means, correct me if I'm wrong on that assumption. If I am wrong can you please include more details on any specific error messages you may be seeing? I would need more details to be able to give a more detailed answer. (This is the most common way we can help support the maintainers of any package).
However, based on what you've said so far the only thing I can think of that might assist you is to tell you that Chart JS is not reactive (it is event driven) nor is it's configuration object observable.  If you call render() before your data is returned from the server and before it's set in the config object (which is a deep object structure in and of itself and not directly observable anyway) you will have a race condition. In other words, you may be rendering the chart before the data is available to even be rendered and Chart JS does not react to those changes in your data model. Try using this.Chart.render() after you have resolved your promise, in .then() or after async/await. I don't know your specific use case as you haven't shared any examples of your code.
I've long mused that "Issues" is an inappropriate label for this section, but that's perhaps an example of an issue in and of itself. This is more so the place to report bugs (aka issues) relevant to the Chart JS code base. When someone has questions about the use of the code (eg "how do I..."), it's best to search StackOverflow and, failing any acceptable results, it's best to ask your question there. The Chart JS community from here also helps out there, but more importantly, React community members are there as well. Some subset of those two communities will also work with Chart JS and are probably more familiar with your specific situation than I am. I haven't touched React since switching to Vue 4+ years ago and even then I didn't integrate Chart JS.
The answer to "why" is that the "issues" here are tracked and associated to code changes. The "Issues" tab is meant to assist the developers track their workflow. The purpose of this section of GitHub is to give the community a way to support the developers of the package in question. We support them here primarily by sharing the steps to reproduce a bug in the software so they can fix it faster. I know what you might be thinking, why not offer support for the software they're creating and hosting on GitHub right here on GitHub? To which I will point to your own question above as an example. Your original concern involved the confluence of two unique packages; React and Chart JS. You can't really ask for help with Chart JS on the React JS GitHub page anymore than you might receive help here for your React question. I'll bet your first post is related to that conundrum. So then, where does one go? There needs to be a way for you to get help from a broader community with experience in both? That's where StackOverflow comes in. There are other sites out there besides Stack and many open source projects have dedicated forums. In this case, I would recommend checking and potentially posting on Stack.
Consider answering (at least) 2 questions on Stack for every 1 question you ask. If you aren't sure yet how to do all that or don't feel comfortable answering any questions yet, don't worry about it. Come back around to it later when you do. Karma is patient. Cheers.

There is a react example in https://github.com/chartjs/chartjs-plugin-annotation/issues/216#issuecomment-754511081


src: copy the license files to license directory

Connects to #172

Coverage decreased (-2.4%) to 83.025% when pulling 86fc780e71bbe1912ca5cfc1eb37d5eaf5b4b578 on helio-frota:pr into a9de82bafcb6f3ddba2f6cea7b71111f6e0cad62 on bucharest-gold:master.


export { } from is no longer marked as covered using TypeScript 3.9

🐛 Bug Report
In a codebase with full coverage, lines in the form export { /* something */ } from /* some file */; are marked as not covered after upgrading to TypeScript 3.9.
To Reproduce
Steps to reproduce the behavior:

git clone git@github.com:Vinnl/ts-jest-repro.git
npm install
npm test

For convenience I've also checked in the generated coverage reports.
Expected behavior
The above repo to have 100% coverage.
Link to repl or repo (highly encouraged)
https://github.com/Vinnl/ts-jest-repro
envinfo

  System:
    OS: Linux 5.4 Ubuntu 20.04 LTS (Focal Fossa)
    CPU: (4) x64 Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
  Binaries:
    Node: 12.16.1 - ~/.nvm/versions/node/v12.16.1/bin/node
    Yarn: 1.22.4 - /usr/bin/yarn
    npm: 6.13.4 - ~/.nvm/versions/node/v12.16.1/bin/npm
  npmPackages:
    jest: ^25.5.4 => 25.5.4

See also: https://github.com/kulshekhar/ts-jest/issues/1635

I have similar problem on typescript version 4.0.5. The difference is that only some export as marked as not covered by tests.
My versions:
"@nrwl/jest": "11.1.5",
"@types/jest": "26.0.20",
"jest": "26.6.3",
"jest-preset-angular": "8.3.2",
"ts-jest": "26.4.4",
"typescript": "~4.0.5"

I have similar problem on typescript version 4.0.5. The difference is that only some export as marked as not covered by tests.
My versions:
"@nrwl/jest": "11.1.5",
"@types/jest": "26.0.20",
"jest": "26.6.3",
"jest-preset-angular": "8.3.2",
"ts-jest": "26.4.4",
"typescript": "~4.0.5"


HIVE-25202 Support decimal64 operations for PTF operators

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Initial patch, need more improvements in the successive commits. Doing a test run for qtest.


puppeteer crashes and takes down whole nodejs application.

I have an app that launches puppeteer and performs actions on a page, after a while (about few hours) application crashes with following message:
<--- Last few GCs --->

[4656:000001FC428D4970] 29297362 ms: Mark-sweep 2014.6 (2059.7) -> 2012.7 (2059.7) MB, 1272.0 / 0.1 ms  (average mu = 0.330, current mu = 0.335) allocation failure scavenge might not succeed
[4656:000001FC428D4970] 29299835 ms: Mark-sweep 2014.7 (2059.7) -> 2012.7 (2059.7) MB, 1747.8 / 0.0 ms  (average mu = 0.311, current mu = 0.293) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 00007FF78A792C6D]
Security context: 0x01a3bfffda89 <JSObject>
    1: send [00000153352D0B69] [C:\my_app\cypress-relay-bot\node_modules\puppeteer-core\lib\Connection.js:~178] [pc=0000039B9B4F8639](this=0x031b9fc44e69 <EventEmitter map = 000001EBA6730861>,0x01a53c2e9cf1 <String[#22]: Runtime.callFunctionOn>,0x0240aa38c481 <Object map = 000001EBA6705CA1>)
    2: _evaluateInternal [00000153352D0379] [C:\my_app\cypres...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200312.021420.4656.0.001.json
Node.js report completed
 1: 00007FF789BC094F napi_wrap+124431
 2: 00007FF789B62696 public: bool __cdecl v8::base::CPU::has_sse(void)const __ptr64+34502
 3: 00007FF789B63356 public: bool __cdecl v8::base::CPU::has_sse(void)const __ptr64+37766
 4: 00007FF78A366F4E private: void __cdecl v8::Isolate::ReportExternalAllocationLimitReached(void) __ptr64+94
 5: 00007FF78A34EF91 public: class v8::SharedArrayBuffer::Contents __cdecl v8::SharedArrayBuffer::Externalize(void) __ptr64+833
 6: 00007FF78A21C85C public: static void __cdecl v8::internal::Heap::EphemeronKeyWriteBarrierFromCode(unsigned __int64,unsigned __int64,class v8::internal::Isolate * __ptr64)+1436
 7: 00007FF78A227C00 public: void __cdecl v8::internal::Heap::ProtectUnprotectedMemoryChunks(void) __ptr64+1312
 8: 00007FF78A224734 public: static bool __cdecl v8::internal::Heap::PageFlagsAreConsistent(class v8::internal::HeapObject)+3204
 9: 00007FF78A219FC3 public: bool __cdecl v8::internal::Heap::CollectGarbage(enum v8::internal::AllocationSpace,enum v8::internal::GarbageCollectionReason,enum v8::GCCallbackFlags) __ptr64+1283
10: 00007FF78A218794 public: void __cdecl v8::internal::Heap::AddRetainedMap(class v8::internal::Handle<class v8::internal::Map>) __ptr64+2356
11: 00007FF78A239A65 public: class v8::internal::Handle<class v8::internal::HeapObject> __cdecl v8::internal::Factory::NewFillerObject(int,bool,enum v8::internal::AllocationType) __ptr64+53
12: 00007FF789FA6239 ??4iterator@JumpTableTargetOffsets@interpreter@internal@v8@@QEAAAEAV01234@$$QEAV01234@@Z+4057
13: 00007FF78A792C6D public: virtual bool __cdecl v8::internal::SetupIsolateDelegate::SetupHeap(class v8::internal::Heap * __ptr64) __ptr64+567949

what does this mean and how to fix it?

I have the same issue...


Debugger: Edit components in debugger view

Should be able to add and remove components, and edit component data visually in the editor.

This is something that I think is really important. Even a half-baked implementation would be acceptable compared to nothing.


This Plug-In For Rider

Cant Use In Rider

@AnZuiLC only A, B, and C are supported.

@AnZuiLC Only support pycharm, goloang and  intellij idea.


[rv_core_ibex] New Ibex tests to cover security hardening

This issue is to track the implementation of new tests within the Ibex testbench for testing security hardening features, from gaps highlighted in the M4 V2(S) sign-off review https://github.com/lowRISC/opentitan/issues/23542

[ ] https://github.com/lowRISC/ibex/issues/2175
[ ] https://github.com/lowRISC/ibex/issues/2174
[ ] https://github.com/lowRISC/ibex/issues/2173

I'd suggest adding these tests to the chip_sw_rv_core_ibex_lockstep_glitch FI test. It may not look like a perfect match to do it in there but I fear we otherwise risk to spend too much effort on setup work.

These are actually very easy to implement in the Ibex DV environment and I'd prefer them as block level tests rather than chip level.

A PR (https://github.com/lowRISC/ibex/pull/2182) is available for https://github.com/lowRISC/ibex/issues/2174.
@nasahlpa is working on https://github.com/lowRISC/ibex/issues/2173
Work for https://github.com/lowRISC/ibex/issues/2175 is mostly complete, but I need to create a PR.

PR for lowRISC/ibex#2173 is available here: lowRISC/ibex#2184.

PR for https://github.com/lowRISC/ibex/issues/2175 is available here: https://github.com/lowRISC/ibex/pull/2185

I am closing this issue as all linked PRs have been merged and all boxes ticked off. Thanks @nasahlpa and @GregAC !


Provide an option for puppeteer.launch() to run a docker image

I came here to ask about making it easier to run Puppeteer in Docker and I spotted #3072, which is very much related. Docker is arguably the ideal way to run puppeteer because it guarantees a consistent environment for your puppeteer scripts regardless of the operating system you're running them on. Some examples of differences across operating systems are:

Font rendering (makes visual snapshot tests difficult)
Permissions (for example, it's not possible to grant clipboard access on macOS)

For this reason, we skip some of our E2E tests when running locally and pray that they pass in the build pipeline (we can always use docker locally for debugging if they do fail). What would be ideal is if running in docker was integrated directly with Puppeteer so that very few changes are required to the puppeteer scripts in order to run in the docker environment.
I've been looking at Browserless and it provides a simple way to run a Chrome environment for Puppeteer to connect to:
$ docker run \
  --rm \
  -p 3000:3000 \
  -e "MAX_CONCURRENT_SESSIONS=10" \
  browserless/chrome:latest

However, this has its obvious downside: you have to change your process to ensure the container is running and switch your scripts from launch() to connect(). This is not particularly ideal for our pipeline and neither is maintaining two separate workflows for each environment. What I would like to see is support for puppeteer.launch() to run a docker container and connect to it.
const browser = await puppeteer.launch({
  docker: !process.env.CI && {
    // Let's pretend an official Puppeteer image exists,
    image: "puppeteer/chrome:v13.5.0",

    // Optional, could just defer to the Dockerfile's EXEC statement
    executablePath: "/usr/bin/chromium",
  },

  // ...
});

Other options could also be provided, for instance volume mapping for downloads, etc. In addition, the puppeteer node module could use an environment variable to predownload the docker image instead of a Chrome binary.
According to the Troubleshooting guide, "getting headless Chrome up and running in Docker can be tricky". The advice for Alpine also becomes out of date every time a new version of Chrome is published in the package repo (at the time of writing, copying and pasting the script would install Chrome 100 alongside Puppeteer v10.0.0, which is intended for Chrome 92). Having Puppeteer be able to just launch Chromium in Docker seems like it would be a win here.

I was able to hack this in quite quickly by making changes to BrowserRunner.ts and a couple of other files. Here's the launch() configuration I tested with:
this._browser = await puppeteer.launch({
  docker: {
    image: "browserless/chrome",
    executablePath: "google-chrome"
  },
  args: this._flags,
  userDataDir: "/usr/src/app/workspace"
});

And here's the changes I made:
diff --git a/src/node/BrowserRunner.ts b/src/node/BrowserRunner.ts
index f2dc6075..174a4005 100644
--- a/src/node/BrowserRunner.ts
+++ b/src/node/BrowserRunner.ts
@@ -25,7 +25,7 @@ import { promisify } from 'util';

 import { assert } from '../common/assert.js';
 import { helper, debugError } from '../common/helper.js';
-import { LaunchOptions } from './LaunchOptions.js';
+import { DockerOptions, LaunchOptions } from './LaunchOptions.js';
 import { Connection } from '../common/Connection.js';
 import { NodeWebSocketTransport as WebSocketTransport } from '../node/NodeWebSocketTransport.js';
 import { PipeTransport } from './PipeTransport.js';
@@ -49,6 +49,7 @@ export class BrowserRunner {
   private _processArguments: string[];
   private _userDataDir: string;
   private _isTempUserDataDir?: boolean;
+  private _dockerOptions?: DockerOptions;

   proc = null;
   connection = null;
@@ -62,13 +63,15 @@ export class BrowserRunner {
     executablePath: string,
     processArguments: string[],
     userDataDir: string,
-    isTempUserDataDir?: boolean
+    isTempUserDataDir?: boolean,
+    dockerOptions?: DockerOptions
   ) {
     this._product = product;
     this._executablePath = executablePath;
     this._processArguments = processArguments;
     this._userDataDir = userDataDir;
     this._isTempUserDataDir = isTempUserDataDir;
+    this._dockerOptions = dockerOptions;
   }

   start(options: LaunchOptions): void {
@@ -86,19 +89,36 @@ export class BrowserRunner {
     debugLauncher(
       `Calling ${this._executablePath} ${this._processArguments.join(' ')}`
     );
-    this.proc = childProcess.spawn(
-      this._executablePath,
-      this._processArguments,
-      {
-        // On non-windows platforms, `detached: true` makes child process a
-        // leader of a new process group, making it possible to kill child
-        // process tree with `.kill(-pid)` command. @see
-        // https://nodejs.org/api/child_process.html#child_process_options_detached
-        detached: process.platform !== 'win32',
-        env,
-        stdio,
+    if (this._dockerOptions) {
+      this.proc = childProcess.spawn(
+        '/usr/local/bin/docker',
+        ['run', '-p=9333:9333', '-p=3000:3000', this._dockerOptions.image, this._dockerOptions.executablePath, ...this._processArguments, "--remote-debugging-address=<IP_ADDRESS>"],
+        {
+          // On non-windows platforms, `detached: true` makes child process a
+          // leader of a new process group, making it possible to kill child
+          // process tree with `.kill(-pid)` command. @see
+          // https://nodejs.org/api/child_process.html#child_process_options_detached
+          detached: process.platform !== 'win32',
+          env,
+          stdio,
+        }
+      );
+    }
+    else {
+      this.proc = childProcess.spawn(
+        this._executablePath,
+        this._processArguments,
+        {
+          // On non-windows platforms, `detached: true` makes child process a
+          // leader of a new process group, making it possible to kill child
+          // process tree with `.kill(-pid)` command. @see
+          // https://nodejs.org/api/child_process.html#child_process_options_detached
+          detached: process.platform !== 'win32',
+          env,
+          stdio,
+        }
+        );
       }
-    );
     if (dumpio) {
       this.proc.stderr.pipe(process.stderr);
       this.proc.stdout.pipe(process.stdout);

It works a charm if the docker image is already downloaded. I used it to run ~800 tests with Karma, including several visual snapshot tests. I tested it with one of our internal docker images too (alpine, with a different executable path), that worked just fine also.
I'd be happy to work on contributing this feature properly if the Puppeteer team are willing to accept it.

I've published an npm package that can help with this in the meantime, https://github.com/andyearnshaw/docketeer:
npx docketeer browserless/chrome:latest npm run visual-snapshot

It works by setting the PUPPETEER_EXECUTABLE_PATH environment variable and launching the browser via a docker container.  There are a few kinks to work out, such as the container not properly exiting when Puppeteer calls kill() on the process group. Puppeteer being aware of the docker process would make this simpler, so I'm still hoping there's interest in adding direct docker support and therefore keeping this issue open.

lol docker is the most complicated way to run pupeteer as even chromium does not run well in it because of many isolation issues and i know that as i am Docker/moby contributor since 2014 so over 8 yeary of expirence with that the GPU part for example is relativ problematic and many other issues that normal none OS Programmers are not aware of.
Do not get me wrong it is possible but internal it is a nightmare. Because docker and Chrome are using same system api's and you need to relax dockers own isolation or turn of the chrome isolation.  Chromium OS Shares a lot of code with Chromium it self. so You see where the Problem comes from Chromium is more then a simple browser (render engine) it is a full blown Platform with a gigantic standard set of Libs and Modules that build even into a full OS as soon as you add More window and user management code.

It makes zero sense to implement this inside pupeteer it is out of scope this should get closed:
and all who want to do this they simply create a pupeteer docker image and then use child_process.execSync to do the magic or spawnSync what ever you favor or what ever system management is in between that.
It would not even be clear what that launch() would do!

Should it detect a local docker socket
should it accept docker hosts as target
should it use a local docker cli
should it use the http api
should it support full blown kubernetes?
.-....... the list is endless

Short Conclusion
Pupeteer can be seen as application it is out of scope for it to supply commands or functions that allow to fork the program it self inside a diffrent environment.

Thanks for your comment @frank-dspeed.

docker is the most complicated way to run pupeteer

Yes, this is one of the main motivations for this suggestion: to improve the ergonomics of it.

chromium does not run well in it
[...]
Do not get me wrong it is possible but internal it is a nightmare

Chromium runs fine in Docker, and I'm speaking from plenty of experience too. Our pipelines run various docker images with Chromium many times a day for e2e tests, integration tests (including visual snapshot tests). I haven't run into any such issues in the several years since building and maintaining these images and the extensive tests that run on them.
It's imperitive to use Puppeteer in a Docker environment for visual snapshot tests to get a consistent rendering across different host operating systems.

and all who want to do this they simply create a pupeteer docker image and then use child_process.execSync to do the magic or spawnSync what ever you favor or what ever system management is in between that.

It's not as simple as this for many reasons, some of which I've discovered since implementing such a solution (see the library above). I've given other reasons in my main post too.

It would not even be clear what that launch() would do!

I don't understand your point here. This is merely an implementation detail that would be the choice of whoever should implement this feature if it were to be accepted by the puppeteer team. The fact that there are many different approaches that could be taken is not a good reason to dismiss the idea.
In any case, I would argue that using the docker cli would be the simplest approach and wouldn't require too many additional hoops to how Chromium is currently launched. This is how I've achieved it, but the aforementioned issues make it an imperfect solution. Having the launched instance be directly under the control of the puppeteer library removes the need for workarounds I've had to employ, improves resilience and robustness and, to circle back to the first point I made, improves the ergonomics of having that consistent browser experience on all operating systems.

but all your changes can be easy addressed by a extra package like pupeteer-docker it is not hard to integrate but ok if this issue should get any upvotes why not but i guess it will slow down overall puppeteer development it has already a lot of open issues and i guess keeping up with the chromium api is work enough for this project unit but maybe i am wrong.

The changes cannot be easily addressed by an external package for the reasons I've already pointed out. That's what https://github.com/andyearnshaw/docketeer is about but there are several problems that are difficult to overcome, such as Puppeteer killing the process and leaving the container orphaned, or connecting the fd3/fd4 pipes for --remote-debugging-pipe.
Please feel free to take a look, I'm sure your experience as a Docker/Moby contributor for 8 years would be really useful.

docketeer 2.0
docketeer.cjs
const os = require('os');
const path = require('path');
const fs = require('fs');

const dockerImage = 'browserless/chrome:latest'
const dockerExecutablePath = '/usr/bin/google-chrome'
const executablePath = path.join(__dirname, 'Chrome-Launcher');

const docker = `#!/bin/bash
docker run -p=9333:9333 -p=3000:3000 ${dockerImage} ${dockerExecutablePath} \${@}`

const debugBinary = `#!/bin/bash
echo "\${@}" > ${executablePath}.log.txt`

fs.writeFileSync(executablePath, docker);
fs.chmodSync(executablePath,'777');

const puppeteer = require('puppeteer');
const browser = puppeteer.launch({
    executablePath,
    userDataDir: './',
    args: [
        "--remote-debugging-address=<IP_ADDRESS>",
        "--no-sandbox",
        "--headless",
        "--disable-gpu",
        "--disable-software-rasterizer"
    ]
});

@frank-dspeed thanks for posting that suggestion. I have a few concerns, but I haven't gotten around to evaluating it yet. I've set up https://github.com/andyearnshaw/docketeer/issues/7 to handle that evaluation and any further discussion as I'd prefer to keep this thread noise-free and for discussion about the feature request itself.
@jrandolf it appears you tagged as bug instead of feature-request, was that intentional?

i guess the core issue here is that node/launcher.js does not export the FirefoxLauncher and ChromeLauncher classes
if it would do that it would allow you to import them and modify them without modify the main code in here
import { FirefoxLauncher, ChromeLauncher } from 'puppeteer/node/Launcher.js'

class dockerLauncher extends ChromeLauncher {
  defaultArgs(options) {
    return [
        // docker arguments
        'run',
        '-p=9333:9333',
        '-p=3000:3000',
        `${dockerImage}`,
        `/usr/bin/google-chrome`,
        // end docker arguments
        ...super.defaultArgs(options)
    ]
  }
}
new dockerLauncher({
    executablePath: '/usr/bin/docker',
    pipe: true
})

@jrandolf is this class based application design fixed? or would i be allowed to refactor to a more functional codebase?
i guess it would improve iteration speed a lot as also types and code reuse overall i needed to read the whole code base because of the class inharitance it would be more easy to use functional composition to configure Puppeteer as also make
it more test able and so on i would invest into that it would take less then 3 days includes testing as i would keep the original class based structure as a facade till it is complet.

This is closed in favour of #3072. In particular, puppeteer is already disjoint enough from chromium through the 'connect' interface, so containerising puppeteer fills the gap of "inconsistent" environments.

@jrandolf I feel like #3072 only strengthens the case for this feature request, the two requests compliment each other significantly.
Let me clarify. If official Puppeteer images are published, the typical approach in using them would be to map in your scripts using -v in some docker run command. This is problematic in non-Linux environments when you have a lot of disk I/O because of how volumes are virtualized in Mac and Windows. This pretty much means any TypeScript or other transpiled codebases are much slower inside a Docker container in those environments. For some of the tests run at the company I work for, the difference is >15 minutes in running visual snapshot tests. The biggest benefit of this feature request is making the process of avoiding that trap as ergonomic as possible for Puppeteer users.
If Puppeteer is both included in the container but can also just launch the containerised browser from outside the container, you're covering all bases for users; the container can be used exclusively in the CI pipeline, just the browser can be used outside.

@jrandolf yes, I've tried other mount methods to no avail, but the slowness is acknowledged by the Docker engineers themselves. Docker for Mac 4.6 has an experimental file sharing implementation called virtiofs which improves performance drastically. If you read the release post, you'll see that their own statistics are similar to mine. virtiofs gains, to pick one quote that supports my statement:

An 80% improvement in the time taken to boot a monolithic Typescript app (1m 30s to 18s)

However, my test suites running with virtiofs still take twice as long as when they are running on the host instead of the container (and there are other issues with the "new Virtualization framework" that is required). As for repeated runs, caching, etc, those all depend on configuration of the build tool and such options may or may not be available. Some build tools like WebPack offer watching, which will obviously increase speed for re-runs.

Also, if puppeteer runs in a container, then with this feature, puppeteer will run the containerized browser in the container, so I don't see how https://github.com/puppeteer/puppeteer/issues/3072 compliments this feature.

Because although the idea on that feature (currently) is that Puppeteer will run in a container, it will also enable easier running the browser in the container, with Puppeteer running on the host. Currently we maintain our own Puppeteer builds with Node, so every time we need to upgrade Puppeteer in our projects, we also need to produce a new image. If #3072 gets done, then upgrading would mostly be a simple case of updating the image tag in the same commit as updating package.json. Going even further, Puppeteer could provide default values for the image and tag, reducing the configuration (as suggested in my first post) to docker: true in the simplest of cases. Running Puppeteer in a container just can't get much easier than that.

i guess that it is a design failure to even use defaultArgs for anything the documentation and why they are needed needs to be improved we need something like the devtools-protocol repo for chromium and firefox args and so on
you can not see docker as indipendent target as docker is just the app it self by design docker is not a vm it is a compiler target like you would compile to single binary. so your resulting docker image with chromium included should contain only the entrypoint because of edge cases like CMD usage breaks the signaling and so on
so a docker image should be seen as single file binary while docker handels IPC
the docker arguments do depend highly on the docker hosts configuration they can be seen as individual Os so with every args change your binary supports a diffrent OS

you got me wrong i say it works and it works perfekt and has nothing todo with puppeteer i recoded it now complet including playwright and did many iterations on that i know everything it does and does not
to get it working with remote pipe you need to adjust the docker args and then the puppeteer launcher or you even do it more clever and spawn the docker container your self and then use puppeteer connect and there you can also none documented insert a custom transport that would be the stdio in and out
all this is do able all this works all that is out of scope for the puppeteer project thats my whole point
the core vital part of puppeteer is its Connection.js thats it the rest is sugar and wrappers lets say models around the existing api they are opinionated


Improve remove-aliases.py

[x] 1. Some aliases do not become trivial.
[f| emp & flted_175_250=1+(4*n) & y=2+x |- q_284::arr_seg<b_282>@M & flted_176_269=b_282+flted_175_250 & flted_175_250+x=q_284 |f] is translated by the class VarReplacer by traversing the string from left to right, and for each variable, if the variable is known to be aliased (as given by AliasCollector), replacing the variable with the aliased value, until a fixpoint. Here is one step of the fixpoint computation:
emp & flted_175_250=1+(4*n) & y=2+x |- q_284::arr_seg<b_282>@M & flted_176_269=b_282+flted_175_250 & flted_175_250+x=q_284
emp & 1+(4*n)=1+(4*n) & y=2+x |- q_284::arr_seg<b_282>@M & flted_176_269=b_282+flted_175_250 & flted_175_250+x=q_284
emp & 1+(4*n)=1+(4*n) & 2+x=2+x |- q_284::arr_seg<b_282>@M & flted_176_269=b_282+flted_175_250 & flted_175_250+x=q_284
emp & 1+(4*n)=1+(4*n) & 2+x=2+x |- q_284::arr_seg<b_282>@M & b_282+flted_175_250=b_282+flted_175_250 & flted_175_250+x=q_284
emp & 1+(4*n)=1+(4*n) & 2+x=2+x |- q_284::arr_seg<b_282>@M & b_282+flted_175_250=b_282+1+(4*n) & flted_175_250+x=q_284
emp & 1+(4*n)=1+(4*n) & 2+x=2+x |- q_284::arr_seg<b_282>@M & b_282+flted_175_250=b_282+1+(4*n) & 1+(4*n)+x=q_284

and then the class AliasRemover gives the result: emp  |- q_284::arr_seg<b_282>@M & b_282+flted_175_250=b_282+1+(4*n) & 1+(4*n)+x=q_284. However, it should be translated into  emp  |- q_284::arr_seg<b_282>@M & 1+(4*n)+x=q_284.
flted_175_250 in b_282+flted_175_250 is not replaced, since the NodeVisitor class does not modify parse trees in-place. So the aforementioned flted_175_250 does not exist at that location in the parse tree. However, on the next step of the fixpoint computation, the aforementioned flted_175_250 will exist there.
An initial solution is to not just replace variables in the parse tree, but also replace variables in the collected aliases.
After more analysis, the root cause is that the list of aliased variables (as given by AliasCollector) is emptied on every step of the fixpoint computation.
The solution is to collect aliases given by every step of the fixpoint computation, and to not empty the list of aliased variables. Note that this solution is sound, since for any variable n that appears multiple times in the list of aliased variables, n must be aliased to the same value. Otherwise, the formula contains aliases of the same variable n to multiple different values, and this is already a contradiction, therefore the solution is trivially sound.

[ ] 2. With the current implementation of AliasRemover (with the two existing functions visit_head and visit_restHead), trivial aliases that occur in boolExps cannot be removed, as mentioned in https://github.com/hipsleek/hipsleek/pull/37/commits/54a1a9860d412bf97f2f62002bfe9ec616b6f4ce, since the boolExp may contain subformulas other than trivial aliases, and so removing the entire boolExp is wrong. However, trivial aliases that occur in boolExps may be removed if:

The boolExp rule is broken into boolExpHead and boolExpRestHead, similarly to the existing rules head and restHead, and with similar justification. The justification for this is, a trivial alias in head (e.g. n=n in n=n & ..., or e.g. n=n in n=n | ...) should be removed, and both the operator and the trivial alias in restHead (e.g. & n=n in ... & n=n & ...) should be removed. Note that removing a trivial alias in head is not entirely correct, as the operator following the trivial alias should also be removed (e.g. & in n=n & ..., or e.g. | in n=n | ... should be removed), and
AliasRemover implements the visit_boolExpHead and visit_boolExpRestHead functions in a similar way to  visit_head and visit_restHead. However, these functions need to account for boolExp's arbitrary nestedness, and also account for possible parentheses around a boolExp.

With this, the semantic distinguishment of alias syntaxes in https://github.com/hipsleek/hipsleek/pull/37/commits/54a1a9860d412bf97f2f62002bfe9ec616b6f4ce is no longer needed.

[x] 3. As mentioned above, removing a trivial alias in head is not entirely correct, as the operator following the trivial alias should also be removed (e.g. & in n=n & ..., or e.g. | in n=n | ... should be removed). As the grammar is left-recursive, this should be solved by setting some boolean state in the AliasRemover class on visiting a trivial alias in visit_head, and then whenever this boolean state is true in visit_headRest, the operator should be removed.

Wontfix item 2: aliases of any same variable n two different values separated by disjunction in boolExp, (e.g. n=1|n=2), when collected and used to replace variables, must be handled as such: the resulting formula is a disjunction, where each disjunct is the replacement of n by the corresponding value in the disjunction in boolExp (e.g. (n=1|n=2)&x::int_star<n> becomes x::int_star<1>|x::int_star<2>.
I consider this too troublesome to implement, and without proper formatting, the resulting formula could be too long to read. I leave resolving aliases in boolExps, as mentioned in item 2, to humans. So, the semantic distinguishment in https://github.com/hipsleek/hipsleek/pull/37/commits/54a1a9860d412bf97f2f62002bfe9ec616b6f4ce stays.

Closing until substantial issues come up in remove-aliases.py.

https://github.com/zhengqunkoo/hipsleek/commit/e4ee8543aa79249a26ef76997fae2a8c856384e7 causes nontermination with the invocation echo "[f| a::aseg<flted_123_104>@M & c=1+b & b=1+a & flted_123_104=3+a |- b::int_star<v>@M * c::aseg<flted_123_125>@M & flted_123_125=3+a & b_124=b & a=b_124|f]" | ./remove-aliases.py.

Note: remove-aliases.py at commit https://github.com/zhengqunkoo/hipsleek/commit/0270b052f avoids nontermination experienced by https://github.com/zhengqunkoo/hipsleek/commit/e4ee8543aa79249a26ef76997fae2a8c856384e7 because in the second step of the fixpoint computation, it considers as a variable only the entire left-hand side of the alias 1+b, and does not consider b as a variable. So b does not get substituted for 1+b forever.
See the invocation echo "[f| b=1+b |f]" | ./remove-aliases.py at commit https://github.com/zhengqunkoo/hipsleek/commit/0270b052f, with debug output indicated by the filename, line, and source code:
[f| b=1+b |f]
remove-aliases.py:163: print(formula): b=1+b
remove-aliases.py:164: print(list(map(lambda x: list(map(lambda y: y.text, x)), aliases))): [['b', '1+b']]
remove-aliases.py:70: print(alias.text, node.text): b b
remove-aliases.py:70: print(alias.text, node.text): b 1
remove-aliases.py:70: print(alias.text, node.text): b b
remove-aliases.py:163: print(formula): 1+b=1+1+b
remove-aliases.py:164: print(list(map(lambda x: list(map(lambda y: y.text, x)), aliases))): [['1+b', '1+1+b']]
remove-aliases.py:70: print(alias.text, node.text): 1+b 1
remove-aliases.py:70: print(alias.text, node.text): 1+b b
remove-aliases.py:70: print(alias.text, node.text): 1+b 1
remove-aliases.py:70: print(alias.text, node.text): 1+b 1
remove-aliases.py:70: print(alias.text, node.text): 1+b b
1+b=1+1+b


How to debug or log in javascript test?

Or is it possible to provide a standalone html based test driver, so as to eliminate the steep learning curve?

If you edit the gruntfile you can set singleRun to false at the following line:
https://github.com/facebook/css-layout/blob/master/Gruntfile.js#L96
This ensures that the Chrome browser and Karma do not shut down after the test run.
You can run the tests as follows:
grunt test-javascript

This will launch chrome allowing you to set breakpoints etc ..


[WIP] trying pet ui

Also don't forget to merge with dev to get the updated tests. And add your tests to the runner otherwise you won't get coverage or testing results in CI.

Codecov Report

Merging #51 (15bfcd2) into dev (eb0728e) will decrease coverage by 8.81%.
The diff coverage is 69.30%.

:exclamation: Current head 15bfcd2 differs from pull request most recent head 3db5656. Consider uploading reports for the commit 3db5656 to get more accurate results

@@            Coverage Diff             @@
##              dev      #51      +/-   ##
==========================================
- Coverage   76.40%   67.58%   -8.82%
==========================================
  Files          69       71       +2
  Lines        2606     2767     +161
  Branches      150      164      +14
==========================================
- Hits         1991     1870     -121
- Misses        577      861     +284
+ Partials       38       36       -2

Impacted Files
Coverage Δ

src/DucksNet.API/Controllers/UsersController.cs
0.00% <0.00%> (ø)

...c/DucksNet.Infrastructure/Sqlite/PetsRepository.cs
26.92% <ø> (-23.08%)
:arrow_down:

.../DucksNet.IntegrationTests/CagesControllerTests.cs
70.64% <0.00%> (-29.36%)
:arrow_down:

tests/DucksNet.IntegrationTests/TestDbContext.cs
100.00% <ø> (ø)

src/DucksNet.API/Controllers/PetsController.cs
16.98% <9.09%> (-1.89%)
:arrow_down:

src/DucksNet.Domain/Model/Pet.cs
42.85% <46.80%> (-13.40%)
:arrow_down:

...et.IntegrationTests/AppointmentsControllerTests.cs
36.58% <50.00%> (-55.29%)
:arrow_down:

src/DucksNet.Domain/Model/User.cs
47.82% <54.90%> (+47.82%)
:arrow_up:

src/DucksNet.API/DTO/PetDTO.cs
100.00% <100.00%> (ø)

src/DucksNet.Domain/Model/Enums/Size.cs
100.00% <100.00%> (ø)

... and 16 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

I also don't understand why there's a drop in coverage.

Found out.  Please run all tests locally before pushing. For now the CI won't notify you if you fail tests.


is there any res.locals equivalent?

It only has special meaning in the context of view/template rendering

Is it so? Express's type RequestHandler has a generic param Locals. And it is convenient to use it for creating typed middlewares that store values of a specific type in .locals, which can be used in the next middlewares. But type Middleware in @types/polka types just ignore this parameter

It is not provided by default. The snippet in my last comment adds it manually.


Build and evaluation corpus from Spain medical test repository

From the website:
https://fse.mscbs.gob.es/fseweb/view/public/datosanteriores/cuadernosExamen/busquedaConvocatoria.xhtml
extract information associated to question and request and create a framework for evaluation following the Meditron guidelines in model evaluation.

Check if exist similar repositories in LATAM cuntries as Chili, Argentina, Mexico, Uruguay.

Check copywrite licences and create document with dataset features.

Create a csv, readme and python script files as output.

We have to meet with the team to define wich task we are going to evaluate


Fix std::function missing

The header file <functional> is not implicitly included in some (uncertain) standard library implementations.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


DateTime Humanizations: "never" threshold

I think this is similar to this, a threshold for "never" would be useful for logging purposes, where the DateTime struct sets 01/01/1900 initially. Could this be contributing? I'd like to try to address this.

Please, jump in, pull requests are very welcome. The project's contributing guidelines are here, just let me know if you have any questions.

Thanks, what would be an appropriate property name for the "never"-threshold? Or will naming issues etc. be sorted out by the code review of the PR?

As suggested I forked the repo and switched to the dev branch. However upon running tests I hit the following without quite being sure how to resolve it:
Test Name:	approve_public_api
Test FullName:	Humanizer.Tests.ApiApprover.PublicApiApprovalTest.approve_public_api
Test Source:	D:\Entwicklung git\Humanizer\src\Humanizer.Tests\ApiApprover\PublicApiApprovalTest.cs : line 16
Test Outcome:	Failed
Test Duration:	0:00:01,506

Result StackTrace:
bei ApprovalTests.Approvers.FileApprover.Fail()
   bei ApprovalTests.Core.Approver.Verify(IApprovalApprover approver, IApprovalFailureReporter reporter)
   bei ApprovalTests.Approvals.Verify(IApprovalWriter writer, IApprovalNamer namer, IApprovalFailureReporter reporter)
   bei ApprovalTests.Approvals.Verify(IApprovalWriter writer)
   bei ApprovalTests.Approvals.Verify(String text)
   bei Humanizer.Tests.ApiApprover.PublicApiApprovalTest.approve_public_api() in D:\Entwicklung git\Humanizer\src\Humanizer.Tests\ApiApprover\PublicApiApprovalTest.cs:Zeile 30.
Result Message:	ApprovalTests.Core.Exceptions.ApprovalMismatchException : Failed Approval: Received file D:\Entwicklung git\Humanizer\src\Humanizer.Tests\ApiApprover\PublicApiApprovalTest.approve_public_api.received.txt does not match approved file D:\Entwicklung git\Humanizer\src\Humanizer.Tests\ApiApprover\PublicApiApprovalTest.approve_public_api.approved.txt.

And I am presented the following diff:
--- D:/Entwicklung git/Humanizer/src/Humanizer.Tests/ApiApprover/PublicApiApprovalTest.approve_public_api.received.txt	Di Nov  3 15:05:10 2015
+++ D:/Entwicklung git/Humanizer/src/Humanizer.Tests/ApiApprover/PublicApiApprovalTest.approve_public_api.approved.txt	Di Nov  3 12:33:42 2015
@@ -1028,12 +1028,12 @@ namespace Humanizer.DateTimeHumanizeStrategy
     }
     public class PrecisionDateTimeHumanizeStrategy : Humanizer.DateTimeHumanizeStrategy.IDateTimeHumanizeStrategy
     {
-        public PrecisionDateTimeHumanizeStrategy(double precision = 0,75) { }
+        public PrecisionDateTimeHumanizeStrategy(double precision = 0.75) { }
         public string Humanize(System.DateTime input, System.DateTime comparisonBase, System.Globalization.CultureInfo culture) { }
     }
     public class PrecisionDateTimeOffsetHumanizeStrategy : Humanizer.DateTimeHumanizeStrategy.IDateTimeOffsetHumanizeStrategy
     {
-        public PrecisionDateTimeOffsetHumanizeStrategy(double precision = 0,75) { }
+        public PrecisionDateTimeOffsetHumanizeStrategy(double precision = 0.75) { }
         public string Humanize(System.DateTimeOffset input, System.DateTimeOffset comparisonBase, System.Globalization.CultureInfo culture) { }
     }
 }

Now I can tell that this has to do with the different decimal delimiter, which I assume is interpreted wrong because of the system defaults, as it is a German system.
Do you have any advice?


Feature Request: Search inside Customizer

Would be nice to have some Search function that looks inside Kirki's control sections.  Something along the lines of this - https://wordpress.org/plugins/customizer-search/
thanks for kirki.

add label feature request #2026

@badcat If there's already a plugin for that they why add it in Kirki?

Maybe because Kirki affords other "Kirki-specific" items that could be searchable and the Customizer Search doesn't "See" the Kirki-specific items.

The 'Customizer Search' plugin looks useful, but this doesn't seem like something that would be suitable for the core Kirki Framework.