gn fetcher fails for noexec download dir

We are doing a VM build with a shared download directory on a noexec mount. We are on Styhead using your scarthgap branch.
The gn fetcher does try to execute dart from the download directory, which results in the following error:
PermissionError: [Errno 13] Permission denied: '/efs/shared/yocto-dl-dir/gn/https__github.com_flutter_engine.git/src/flutter/third_party/dart/tools/sdks/dart-sdk/bin/dart'

This is a big problem for us, as our shared directory can only be noexec.
I believe this usage is at best non-standard (it's the first time we've run across such an error in our builds), and perhaps a security hole. Is it possible to rework this so dart is executed from the standard recipe-sysroot-native instead?

@thochstein This happens in "run-hooks" phase, which happens after the engine source tree is fetched.  It executes from the source repo.  See:
https://github.com/flutter/engine/blob/main/DEPS#L939-L1089
Any of the items without a condition always run, and the ones with a condition run based on the variable value.  This is what I set that affects the hooks:
GN_CUSTOM_VARS ?= '\
{\
    "download_android_deps": False, \
    "download_windows_deps": False, \
    "download_linux_deps": False,   \
}'

So any of the hooks with these conditions, will not run.  It narrows down what actually runs.
You can try adding this to your local.conf:
EXTRA_GN_SYNC:pn-flutter-engine = "--no-verify --shallow --no-history -R -D"

It's supposed to disable hooks.
This ran before?  If so with what meta-flutter branch/commit.  Perhaps the delta is --no-verify.  If so, then when it does run (invoked by configure), it would run in the recipes $WORKDIR.  It would then likely require network access which breaks LTS.
I would expect your setup to have issues with all of the gn based recipes in the layer.
Regarding security concerns I build on RedHat (selinux enabled) using rootless Podman.  When something does execute it only has user access of which the container runs in, not root.  I lock down the runner user.  Running non-rootless Docker it has root access.  Running non-rootless Docker/Podman is a serious security concern.  The CI jobs reference these containers:  https://github.com/meta-flutter/docker-containers.  Getting the rootless setup on a selinux system is not that simple.

This function would require modification for the gn fetcher:
https://github.com/meta-flutter/meta-flutter/blob/scarthgap/lib/gn.py#L34-L88
These for the cip fetcher:
https://github.com/meta-flutter/meta-flutter/blob/scarthgap/lib/cipd.py#L58-L89
https://github.com/meta-flutter/meta-flutter/blob/scarthgap/lib/cipd.py#L58-L89

You can try adding this to your local.conf:
EXTRA_GN_SYNC:pn-flutter-engine = "--no-verify --shallow --no-history -R -D"

]';             gclient sync --force --no-verify --shallow --no-history -R -D --revision db49896cf25ceabc44096d5f088d86414e05a7aa -j 1 -v failed with exit code 2, output:
mkdir: missing operand
Try 'mkdir --help' for more information.
WARNING: Your metrics.cfg file was invalid or nonexistent. A new one will be created.
Usage: gclient.py sync [options]

gclient.py: error: no such option: --no-verify

Note the -j 1 was a debugging attempt. Please ignore.

gclient.py: error: no such option: --no-hooks

What does gclient —help output?

Oh, realized I can use -c devshell.
root@yb20:~/styhead/build/tmp/work/armv8a-poky-linux/flutter-engine/3.24.4/src# ../recipe-sysroot-native/usr/share/depot_tools/gclient --help
Usage: gclient.py <command> [options]

Meta checkout dependency manager for Git.

Commands are:
  config       creates a .gclient file in the current directory
  diff         displays local diff for every dependencies
  fetch        fetches upstream commits for all modules
  flatten      flattens the solutions into a single DEPS file
  getdep       gets revision information and variable values from a DEPS file
  gitmodules   adds or updates Git Submodules based on the contents of the DEPS file
  grep         greps through git repos managed by gclient
  help         prints list of commands or help for a specific command
  installhooks installs gclient git hooks
  metrics      reports, and optionally modifies, the status of metric collection
  pack         generates a patch which can be applied at the root of the tree
  recurse      operates [command args ...] on all the dependencies
  revert       reverts all modifications in every dependencies
  revinfo      outputs revision info mapping for the client and its dependencies
  root         outputs the solution root (or current dir if there isn't one)
  runhooks     runs hooks for files that have been modified in the local working copy
  setdep       modifies dependency revisions and variable values in a DEPS file
  status       shows modification status for every dependencies
  sync         checkout/update all modules
  validate     validates the .gclient and DEPS syntax
  verify       verifies the DEPS file deps are only from allowed_hosts

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -j JOBS, --jobs=JOBS  Specify how many SCM commands can run in parallel;
                        defaults to 96 on this machine
  -v, --verbose         Produces additional output for diagnostics. Can be
                        used up to three times for more logging info.
  --gclientfile=CONFIG_FILENAME
                        Specify an alternate .gclient file
  --spec=SPEC           create a gclient file containing the provided string.
                        Due to Cygwin/Python brokenness, it can't contain any
                        newlines.
  --no-nag-max          Ignored for backwards compatibility.

I tried --version as well.
root@yb20:~/styhead/build/tmp/work/armv8a-poky-linux/flutter-engine/3.24.4/src# ../recipe-sysroot-native/usr/share/depot_tools/gclient --version
Your copy of depot_tools is configured to fetch from an obsolete URL:

  https://chromium.googlesource.com/chromium/tools/depot_tools

OK to update it to https://chromium.googlesource.com/chromium/tools/depot_tools.git ? [Y/n] n
./cipd: line 143: ../recipe-sysroot-native/usr/share/depot_tools/cipd_client_version.digests: No such file or directory
Platform linux-amd64 is not supported by the CIPD client bootstrap: there's no pinned SHA256 hash for it in the *.digests file.
../recipe-sysroot-native/usr/share/depot_tools/bootstrap_python3: line 32<EMAIL_ADDRESS>No such file or directory

Looking at a recent version these are the current options
$ gclient sync --help
Usage: gclient.py sync [options]

Checkout/update all modules.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -j JOBS, --jobs=JOBS  Specify how many SCM commands can run in parallel;
                        defaults to 32 on this machine
  -v, --verbose         Produces additional output for diagnostics. Can be
                        used up to three times for more logging info.
  --gclientfile=CONFIG_FILENAME
                        Specify an alternate .gclient file
  --spec=SPEC           create a gclient file containing the provided string.
                        Due to Cygwin/Python brokenness, it can't contain any
                        newlines.
  --no-nag-max          Ignored for backwards compatibility.
  -f, --force           force update even for unchanged modules
  -n, --nohooks         don't run hooks after the update is complete
  -p, --noprehooks      don't run pre-DEPS hooks
  -r REV, --revision=REV
                        Enforces git ref/hash for the solutions with the
                        format src@rev. The src@ part is optional and can be
                        skipped. You can also specify URLs instead of paths
                        and gclient will find the solution corresponding to
                        the given URL. If a path is also specified, the URL
                        takes precedence. -r can be used multiple times when
                        .gclient has multiple solutions configured, and will
                        work even if the src@ part is skipped. Revision
                        numbers (e.g. 31000 or r31000) are not supported.
  --patch-ref=GERRIT_REF
                        Patches the given reference with the format
                        dep@target-ref:patch-ref. For |dep|, you can specify
                        URLs as well as paths, with URLs taking preference.
                        |patch-ref| will be applied to |dep|, rebased on top
                        of what |dep| was synced to, and a soft reset will be
                        done. Use --no-rebase-patch-ref and --no-reset-patch-
                        ref to disable this behavior. |target-ref| is the
                        target branch against which a patch was created, it is
                        used to determine which commits from the |patch-ref|
                        actually constitute a patch.
  -t, --download-topics
                        Downloads and patches locally changes from all open
                        Gerrit CLs that have the same topic as the changes in
                        the specified patch_refs. Only works if atleast one
                        --patch-ref is specified.
  --with_branch_heads   Clone git "branch_heads" refspecs in addition to the
                        default refspecs. This adds about 1/2GB to a full
                        checkout. (git only)
  --with_tags           Clone git tags in addition to the default refspecs.
  -H, --head            DEPRECATED: only made sense with safesync urls.
  -D, --delete_unversioned_trees
                        Deletes from the working copy any dependencies that
                        have been removed since the last sync, as long as
                        there are no local modifications. When used with
                        --force, such dependencies are removed even if they
                        have local modifications. When used with --reset, all
                        untracked directories are removed from the working
                        copy, excluding those which are explicitly ignored in
                        the repository.
  -R, --reset           resets any local changes before updating (git only)
  -M, --merge           merge upstream changes instead of trying to fast-
                        forward or rebase
  -A, --auto_rebase     Automatically rebase repositories against local
                        checkout during update (git only).
  --deps=OS_LIST        override deps for the specified (comma-separated)
                        platform(s); 'all' will process all deps_os references
  --process-all-deps    Check out all deps, even for different OS-es, or with
                        conditions evaluating to false
  --upstream            Make repo state match upstream branch.
  --output-json=OUTPUT_JSON
                        Output a json document to this path containing summary
                        information about the sync.
  --no-history          GIT ONLY - Reduces the size/time of the checkout at
                        the cost of no history. Requires Git 1.9+
  --shallow             GIT ONLY - Do a shallow clone into the cache dir.
                        Requires Git 1.9+
  --no_bootstrap, --no-bootstrap
                        Don't bootstrap from Google Storage.
  --ignore_locks        No longer used.
  --break_repo_locks    No longer used.
  --lock_timeout=LOCK_TIMEOUT
                        GIT ONLY - Deadline (in seconds) to wait for git cache
                        lock to become available. Default is 5000.
  --no-rebase-patch-ref
                        Bypass rebase of the patch ref after checkout.
  --no-reset-patch-ref  Bypass calling reset after patching the ref.
  --experiment=EXPERIMENTS
                        Which experiments should be enabled.

Examples:
  gclient sync
      update files from SCM according to current configuration,
      *for modules which have changed since last update or sync*
  gclient sync --force
      update files from SCM according to current configuration, for
      all modules (useful for recovering files deleted from local copy)
  gclient sync --revision src@GIT_COMMIT_OR_REF
      update src directory to GIT_COMMIT_OR_REF

JSON output format:
If the --output-json option is specified, the following document structure will
be emitted to the provided file. 'null' entries may occur for subprojects which
are present in the gclient solution, but were not processed (due to custom_deps,
os_deps, etc.)

{
  "solutions" : {
    "<name>": {  # <name> is the posix-normalized path to the solution.
      "revision": [<git id hex string>|null],
      "scm": ["git"|null],
    }
  }
}

This is the one to use
-n, --nohooks         don't run hooks after the update is complete

That did get me past the fetcher error. Next problem is in do_compile:
ERROR: Logfile of failure stored in: /home/bamboo/automation/styhead/fsl-imx-internal-xwayland/yocto/build/tmp/work/armv8a-poky-linux/flutter-engine/3.24.4/temp/log.do_compile.36686
Log data follows:
| DEBUG: Executing shell function do_compile
| NOTE: FLUTTER_RUNTIME_MODES=debug profile release
| ninja: Entering directory `out/linux_debug_arm64'
| [0/1] Regenerating ninja files
| WARNING at build arg file (use "gn args <out_dir>" to edit):57:19: Build argument has no effect.
| angle_build_all = false
|                   ^----
| The variable "angle_build_all" was set as a build argument
| but never appeared in a declare_args() block in any buildfile.
|
| To view all possible args, run "gn args --list <out_dir>"
|
| The build continued as if that argument was unspecified.
|
| ninja: error: '../../flutter/tools/const_finder/.dart_tool/package_config.json', needed by 'gen/const_finder.dart.snapshot', missing and no known rule to make it
| WARNING: exit code 1 from a shell command.

Could this be due to the Styhead WORKDIR/UNPACKDIR change?

You need the hooks to run.  To support DL_DIR as a noexec mount you need changes to the code I pointed to prior.
If you plan on shipping product, I would only use an LTS release.  styhead is not an LTS release.


Added Debug.logTime function for easier performance debugging

This lets you "tag" any function for easy performance debugging. More info in this discourse thread.
Wrap any function in your code:

This will log the execution time in the console:

It is also visible in the performance timeline:

I was thinking about renaming this function to Debug.benchmark instead, is that better?

The way I use this locally is

Replace the Debug.elm module somewhere in ~/.elm/packages or wherever they are located on your system (I use Linux)
Remove the *.dat files in the core module (found in the parent dir to where Debug.elm is located)
Remove elm-stuff in my project.

Then you should be able to use Debug.logTime in your project. Probably there is a better way to do this but it works on my computer ;-)

It compiles! That's excellent. Lifesaver!
So someone should at some point be making an "illegal" package manager for Elm that applies and unapplies core patches such as your excellent one.


Another batch of user-agents

(plus a correction to a mistake I made in the previous PR)
This should be my last one for a while hopefully - this is everything that was remaining in my logs.
Thanks

many thanks!


error: failed to select a version for the requirement pnet = "^0.31"

Hello, I met an issue about pnet :
arp-toolkit-rust ±|master|→ cargo build
error: failed to select a version for the requirement `pnet = "^0.31"`
candidate versions found which didn't match: 0.29.0, 0.28.0, 0.27.2, ...
location searched: crates.io index
required by package `arp-toolkit v0.3.0 (/works/arp-toolkit-rust/src)`

arp-toolkit-rust ±|master|→ cargo --version &&  rustc --version
cargo 1.58.0 (f01b232bc 2022-01-19)
rustc 1.58.1 (db9d1b20b 2022-01-20)

Hi,
as pnet 0.31 is available, there must be some issue on your side.
Have you tried e.g. this?

Hello Oli,
I'd try to build it in a clean Ubuntu 20.04 with cargo 1.65.0 , no error occurred. Maybe this is a cargo's compatibly issue.

BTW, maybe these  command can resolve this issue on old-environment :


deploy-qtrpi.sh "unable to locate libinput5"

Good afternoon,
I'm trying to run the deploy-qtrpi.sh but I get the error "unable to locate libinput5". I looked into the script and it is basically the following command line that causes the error:
ssh $TARGET_HOST 'sudo apt-get install -y libts-0.0-0 libinput5 fontconfig'
I was looking to download manually the libinput5 library to the raspian Jessie version I have but it seems that since it depends on other packages the manual installation is halted due to these dependencies missing. Would you please tell me if there is anything I may have overlooked while running this script. Any help would be greatly appreciated. Thank you.

Hello,
got same problem on brand new install stretch raspbian

Raspbian Stretch with desktop
with desktop based on Debian Stretch
Version: August 2017
Release date: 2017-08-16
Kernel version: 4.9

E: Unable to locate package libinput5
rm: cannot remove '/usr/lib/arm-linux-gnueabihf/libEGL.so.1.0.0': No such file or directory
rm: cannot remove '/usr/lib/arm-linux-gnueabihf/libGLESv2.so.2.0.0': No such file or directory

I'm trying again with ./init-qtrpi-full.sh
Best regards,

Don't know if it could solve traviezo problem also, but I manage to fix the missing library by doing following steps (should no be accurate as I tried so many things...):
apt-get install libx11-dev apt-get remove libgles2-mesa --purge (without removing dependencies) sudo apt-get install libgles2-mesa libgles2-mesa-dev

QtRpi expect a "Raspbian Jessie" version. Since august 2017, Raspbian switched from Debian Jessie to Debian Stretch.
Two solutions:

You can use a Raspbian Jessie version (https://downloads.raspberrypi.org/raspbian/images/raspbian-2017-07-05/) with QtRpi
You can help us to enhance and update QtRpi to support the new Raspbian Stretch version

thanks GuillaumeLazar for your cmt, I'm using Raspbian Jessie version but I got same problem on brand new install stretch raspbian.E: Unable to locate package libinput5 rm: cannot remove '/usr/lib/arm-linux-gnueabihf/libGLESv2.so.2.0.0': No such file or directory


authorization for https connections?

Hi. Thank you for your work.
Is there any way to enable authorization for https connections?

hi @legale
could you give more details of how to reproduce this issue?

Hi @chobits
Auth_basic is invalid for https
server {
    listen                         3128;

    # dns resolver used by forward proxying
    resolver                       <IP_ADDRESS>;

    # forward proxy for CONNECT request
    proxy_connect;
    proxy_connect_allow            all;
    #proxy_connect_allow            80 443 563;
    proxy_connect_connect_timeout  100s;
    proxy_connect_read_timeout     100s;
    proxy_connect_send_timeout     100s;

    # forward proxy for non-CONNECT request
    location / {
        # auth
        auth_basic "test auth";        #This is invalid for https
        auth_basic_user_file htpasswd;

        autoindex on;
        proxy_pass http://$host;
        proxy_set_header Host $host;
    }
}

It seems there is issue with HTTPS basic authentication.
I get 401 Unauthorized.
Here is nginx config
server {
    listen                         3129 ssl;
    ssl_certificate                /etc/letsencrypt/live/proxy.address/fullchain.pem;
    ssl_certificate_key            /etc/letsencrypt/live/proxy.address/privkey.pem;
    resolver                       <IP_ADDRESS>;
    auth_pam                       "nginxProxyAuth";
    auth_pam_service_name          "nginx";
    proxy_connect;
    proxy_connect_allow            all;
    location / {
        proxy_pass                 http://$host;
        proxy_set_header           Host $host;
    }
}

Here is curl trace

curl --proxy https://proxy.address:3129 http://example.com/ --proxy-user user:password --include --verbose --proxy-cacert "C:\curl\bin\curl-ca-bundle.crt"

* Connected to proxy.address (<IP_ADDRESS>) port 3129 (#0)
* successfully set certificate verify locations:
*   CAfile: C:\curl\bin\curl-ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Proxy certificate:
*  subject: CN=proxy.address
*  start date: Sep 27 17:32:01 2018 GMT
*  expire date: Dec 26 17:32:01 2018 GMT
*  subjectAltName: host "proxy.address" matched cert's "proxy.address"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Proxy auth using Basic with user 'user'
> GET http://example.com/ HTTP/1.1
> Host: example.com
> Proxy-Authorization: Basic cm9vdDpTTHNza29yZA==
> User-Agent: curl/7.61.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 401 Unauthorized

I happen to have squid configured on the same machine. HTTPS basic auth works on it.
* Connected to proxy.address (<IP_ADDRESS>) port 3129 (#0)
* successfully set certificate verify locations:
*   CAfile: C:\curl\bin\curl-ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
* Proxy certificate:
*  subject: CN=proxy.address
*  start date: Sep 27 17:32:01 2018 GMT
*  expire date: Dec 26 17:32:01 2018 GMT
*  subjectAltName: host "proxy.address" matched cert's "proxy.address"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Proxy auth using Basic with user 'root'
> GET http://example.com/ HTTP/1.1
> Host: example.com
> Proxy-Authorization: Basic cm9vdDpTTHNza29yZA==
> User-Agent: curl/7.61.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK

Actually it seems nginx PAM authentication breaks when used with this module.
HTTP basic authentication also fails.

curl --proxy http://proxy.address:3128 http://example.com/ --proxy-user user:password  --include --verbose

* Connected to proxy.address (<IP_ADDRESS>) port 3128 (#0)
* Proxy auth using Basic with user 'user'
> GET http://example.com/ HTTP/1.1
> Host: example.com
> Proxy-Authorization: Basic cm9vdDpTTHNza29yZA==
> User-Agent: curl/7.61.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 401 Unauthorized

nginx configuration
server {
    listen                         3128;
    resolver                       <IP_ADDRESS>;
    auth_pam                       "nginxProxyAuth";
    auth_pam_service_name          "nginx";
    proxy_connect;
    proxy_connect_allow            all;
    location / {
        proxy_pass                 http://$host;
        proxy_set_header           Host $host;
    }
}

I used nginx PAM auth module. I checked that basic auth on simple page works. Please @chobits advise on authentication options for this module.

hi，
let me check.
(I only check nginx basic auth module before)

More than half year passed, any progress about this critical bug? It provides little value if insists to break the existing authentication mechanism.

@chobits you mentioned nginx basic auth, but according to my result, it fails either. If you have different output, would you please share you sample configuration here? Thanks.

any progress about this critical bug? From my point of view, it provides little value if this module insists to break/deny the existing authentication mechanism.

This module can work with lua-nginx-module. During REWRITE phase (directive rewrite_by_lua_*), you can check request and detect whether to break/deny the request.
But you should know that the request which you can check is only the CONNECT request, not any data proxied under CONNECT tunnel.

@chobits Thanks for your quick reply.  But we all want is nothing else but basic auth to prevent illegal traffic pass through forward proxy. Do you think it is not important?

hi @moralrebuild
@gamelton 's case has two problem, which are not assiciated with proxy connect module.

His test case does not generate CONNECT request, please note that proxy connect module can only handle CONNECT request, other requests (GET/POST) is handled by nginx itself.
The auth module of his test case is not nginx auth basic module, it's nginx PAM auth module. Why this module cannot work, should be debugged with https://github.com/sto/ngx_http_auth_pam_module.

Note that I have checked proxy_connect module can work under auth basic module (Note that my test case is worked under CONNECT request with basic header).

Note that request under CONNECT tunnel, such as some GET/POST requests cannot be checked by auth basic module for your proxy_connect nginx.
For why, see https://github.com/chobits/ngx_http_proxy_connect_module/issues/108#issuecomment-498964963

I think this module should supports
proxy authentication
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/407

@chobits
Let me explain what I observed.

I build this module from its master branch with Nginx 1.16.0.

the nginx configuration
`	server {
listen 443 ssl;
listen [::]:443 ssl;
server_name my.domain;
auth_basic "server auth";
auth_basic_user_file /etc/nginx/htpasswd;
ssl_certificate /etc/letsencrypt/live/my.domain/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/my.domain/privkey.pem;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
ssl_session_timeout 5m;
ssl_protocols TLSv1.3 TLSv1.2 TLSv1.1;
ssl_prefer_server_ciphers on;
ssl_session_cache builtin:1000 shared:SSL:10m;
access_log on;
root         /var/www/html;
index index.html index.htm index.nginx-debian.html;
 # dns resolver used by forward proxying
 resolver                       <IP_ADDRESS>;

 # forward proxy for CONNECT request
 proxy_connect;
 proxy_connect_allow            443 22;
 proxy_connect_connect_timeout  10s;
 proxy_connect_read_timeout     10s;
 proxy_connect_send_timeout     10s;

 # forward proxy for non-CONNECT request
 location / {
     proxy_pass http://$host;
     proxy_set_header Host $host;
 }

}
`

the test result
3.1  curl
`curl https://github.com/ -v -x my.domain:443 --proxy-user user01:password01

Trying <IP_ADDRESS>...
TCP_NODELAY set
Connected to (nil) (<IP_ADDRESS>) port 443 (#0)
Establish HTTP proxy tunnel to github.com:443
Proxy auth using Basic with user 'user01'

CONNECT github.com:443 HTTP/1.1
Host: github.com:443
Proxy-Authorization: Basic cHVuZ2lzemI6TEdtMzZCaXc5SlkwdTGFqNA==
User-Agent: curl/7.52.1
Proxy-Connection: Keep-Alive

< HTTP/1.1 400 Bad Request
< Server: nginx/1.16.0
< Date: Mon, 17 Jun 2019 07:44:24 GMT
< Content-Type: text/html
< Content-Length: 255
< Connection: close
<

Received HTTP code 400 from proxy after CONNECT
Curl_http_done: called premature == 0
Closing connection 0
`

3.2 proxytunnel
`proxytunnel -v -E -p my.domain:443 -P pungiszb:LGm36Biw9JY1Laj4 -d my.domain:22     # for testing only purpose, enable one proxy rather than two
SSL client to proxy enabled
Local proxy my.domain resolves to <IP_ADDRESS>
Connected to my.domain:443 (local proxy)
Set SNI hostname to my.domain
Tunneling to my.domain:22 (destination)
Communication with local proxy:
-> CONNECT my.domain:22 HTTP/1.1
-> Host: my.domain:9443
-> Proxy-Authorization: Basic cHVuZ2lzemI6TEdtMzZCaXc5SlkwdTGFqNA==
-> Proxy-Connection: Keep-Alive
<- HTTP/1.1 401 Unauthorized
HTTP return code: 401 Unauthorized
<- Server: nginx/1.16.0
<- Date: Mon, 17 Jun 2019 07:53:17 GMT
<- Content-Type: text/html
<- Content-Length: 179
<- Connection: keep-alive
<- WWW-Authenticate: Basic realm="server auth"
`
nginx debug log :
2019/06/17 15:58:29 [debug] 30468#30468: *50954 accept: <IP_ADDRESS>:60038 fd:13 2019/06/17 15:58:29 [debug] 30468#30468: *50954 event timer add: 13: 60000:354427063 2019/06/17 15:58:29 [debug] 30468#30468: *50954 reusable connection: 1 2019/06/17 15:58:29 [debug] 30468#30468: *50954 epoll add event: fd:13 op:1 ev:80002001 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http check ssl handshake 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http recv(): 1 2019/06/17 15:58:29 [debug] 30468#30468: *50954 https ssl handshake: 0x16 2019/06/17 15:58:29 [debug] 30468#30468: *50954 tcp_nodelay 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL server name: "my.domain" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL_do_handshake: -1 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL_get_error: 2 2019/06/17 15:58:29 [debug] 30468#30468: *50954 reusable connection: 0 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL handshake handler: 0 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL_do_handshake: 1 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 reusable connection: 1 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http wait request handler 2019/06/17 15:58:29 [debug] 30468#30468: *50954 malloc: 000055DA697591D0:1024 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL_read: -1 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL_get_error: 2 2019/06/17 15:58:29 [debug] 30468#30468: *50954 free: 000055DA697591D0 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http wait request handler 2019/06/17 15:58:29 [debug] 30468#30468: *50954 malloc: 000055DA697591D0:1024 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL_read: 162 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL_read: -1 2019/06/17 15:58:29 [debug] 30468#30468: *50954 SSL_get_error: 2 2019/06/17 15:58:29 [debug] 30468#30468: *50954 reusable connection: 0 2019/06/17 15:58:29 [debug] 30468#30468: *50954 posix_memalign: 000055DA6973B4B0:4096 @16 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http process request line 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http request line: "CONNECT my.domain:22 HTTP/1.1" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http uri: "" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http args: "" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http exten: "" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 posix_memalign: 000055DA6970C250:4096 @16 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http process request header line 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http header: "Host: my.domain:22" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http header: "Proxy-Authorization: Basic  cHVuZ2lzemI6TEdtMzZCaXc5SlkwdTGFqNA==" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http header: "Proxy-Connection: Keep-Alive" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http header done 2019/06/17 15:58:29 [debug] 30468#30468: *50954 event timer del: 13: 354427063 2019/06/17 15:58:29 [debug] 30468#30468: *50954 generic phase: 0 2019/06/17 15:58:29 [debug] 30468#30468: *50954 generic phase: 1 2019/06/17 15:58:29 [debug] 30468#30468: *50954 rewrite phase: 2 2019/06/17 15:58:29 [debug] 30468#30468: *50954 rewrite phase: 4 2019/06/17 15:58:29 [debug] 30468#30468: *50954 post rewrite phase: 5 2019/06/17 15:58:29 [debug] 30468#30468: *50954 generic phase: 6 2019/06/17 15:58:29 [debug] 30468#30468: *50954 generic phase: 7 2019/06/17 15:58:29 [debug] 30468#30468: *50954 generic phase: 8 2019/06/17 15:58:29 [debug] 30468#30468: *50954 generic phase: 9 2019/06/17 15:58:29 [debug] 30468#30468: *50954 access phase: 10 2019/06/17 15:58:29 [debug] 30468#30468: *50954 access phase: 11 2019/06/17 15:58:29 [info] 30468#30468: *50954 no user/password was provided for basic authentication, client: <IP_ADDRESS>, server: my.domain, request: "CONNECT my.domain:22 HTTP/1.1", host: "my.domain:22" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http finalize request: 401, "?" a:1, c:1 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http special response: 401, "?" 2019/06/17 15:58:29 [debug] 30468#30468: *50954 http set discard body 2019/06/17 15:58:29 [debug] 30468#30468: *50954 HTTP/1.1 401 Unauthorized Server: nginx/1.16.0 Date: Mon, 17 Jun 2019 07:58:29 GMT Content-Type: text/html Content-Length: 179 Connection: keep-alive WWW-Authenticate: Basic realm="server auth"

hi @moralrebuild

3.1 curl

For "3.1 curl" case, your nginx server config listens 443 with ssl protocol, while curl command (curl https://... -x ...) creates plain CONNECT request. So nginx server refuses this CONNECT request with 400. Note that curl command cannot create tunnel under SSL protocol. You should disable ssl protocol for listening config.

3.2 proxytunnel

For this case, proxytunnel create tunnel under SSL protocol,  in which case CONNECT request and later proxied stream is under ssl protocol.
But nginx auth_basic module can only handle Authorization header, proxytunnel request header Proxy-Authorization: Basic cHVuZ2lzemI6TEdtMzZCaXc5SlkwdTGFqNA== cannot be checked. Try the following directive to bypass header limit:
        proxy_connect ...;
        ...;

        # set Authorization header via Proxy-Authorization header
        rewrite_by_lua '
            ngx.req.set_header("Authorization", ngx.var.http_proxy_Authorization)
        ';

My test case for nginx auth basic module with proxy_connect module.

nginx config

    server {
        listen 8080;
        server_name my.domain;

        auth_basic "server auth";
        auth_basic_user_file /Users/xiaochen/work/github/tengine/output/conf/pwd;

        resolver                       <IP_ADDRESS>;

        # forward proxy for CONNECT request
        proxy_connect;
        proxy_connect_allow            all;
        proxy_connect_connect_timeout  10s;
        proxy_connect_read_timeout     10s;
        proxy_connect_send_timeout     10s;

        # auth: hello:world
        rewrite_by_lua '
            ngx.req.set_header("Authorization", "Basic aGVsbG86d29ybGQ=")
        ';

        # forward proxy for non-CONNECT request
        location / {
            proxy_pass http://$host;
            proxy_set_header Host $host;
        }
    }

auth_basic_user_file /Users/xiaochen/work/github/tengine/output/conf/pwd:

$ htpasswd -c ./conf/pwd hello
New password:           <<<<<< input "world"
Re-type new password:     <<<<<<< input "world"
Adding password for user hello

$ cat conf/pwd
hello:$apr1$X2Xvv/bM$K17b2lTxih1JlTvyzFyWh1

my curl command:

NOTE I dont add header Authorization: Basic aGVsbG86d29ybGQ=  for this curl command, because curl command can only add header for proxied request under CONNECT tunnel, it cannot add header to CONNECT request. I have added this header in rewrite_by_lua directive, see nginx config of step 1.
$ curl https://github.com -x localhost:8080  -I
HTTP/1.1 200 Connection Established
Proxy-agent: nginx

HTTP/1.1 200 OK
Date: Tue, 18 Jun 2019 07:34:05 GMT
Content-Type: text/html; charset=utf-8
Server: GitHub.com
Status: 200 OK
Vary: X-PJAX
...

It seems there is issue with HTTPS basic authentication.
I get 401 Unauthorized.
Here is nginx config
server {
    listen                         3129 ssl;
    ssl_certificate                /etc/letsencrypt/live/proxy.address/fullchain.pem;
    ssl_certificate_key            /etc/letsencrypt/live/proxy.address/privkey.pem;
    resolver                       <IP_ADDRESS>;
    auth_pam                       "nginxProxyAuth";
    auth_pam_service_name          "nginx";
    proxy_connect;
    proxy_connect_allow            all;
    location / {
        proxy_pass                 http://$host;
        proxy_set_header           Host $host;
    }
}

Here is curl trace

curl --proxy https://proxy.address:3129 http://example.com/ --proxy-user user:password --include --verbose --proxy-cacert "C:\curl\bin\curl-ca-bundle.crt"

* Connected to proxy.address (<IP_ADDRESS>) port 3129 (#0)
* successfully set certificate verify locations:
*   CAfile: C:\curl\bin\curl-ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Proxy certificate:
*  subject: CN=proxy.address
*  start date: Sep 27 17:32:01 2018 GMT
*  expire date: Dec 26 17:32:01 2018 GMT
*  subjectAltName: host "proxy.address" matched cert's "proxy.address"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Proxy auth using Basic with user 'user'
> GET http://example.com/ HTTP/1.1
> Host: example.com
> Proxy-Authorization: Basic cm9vdDpTTHNza29yZA==
> User-Agent: curl/7.61.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 401 Unauthorized

I happen to have squid configured on the same machine. HTTPS basic auth works on it.
* Connected to proxy.address (<IP_ADDRESS>) port 3129 (#0)
* successfully set certificate verify locations:
*   CAfile: C:\curl\bin\curl-ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
* Proxy certificate:
*  subject: CN=proxy.address
*  start date: Sep 27 17:32:01 2018 GMT
*  expire date: Dec 26 17:32:01 2018 GMT
*  subjectAltName: host "proxy.address" matched cert's "proxy.address"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Proxy auth using Basic with user 'root'
> GET http://example.com/ HTTP/1.1
> Host: example.com
> Proxy-Authorization: Basic cm9vdDpTTHNza29yZA==
> User-Agent: curl/7.61.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK

See my comment: https://github.com/chobits/ngx_http_proxy_connect_module/issues/42#issuecomment-502985437.
We should transfer Proxy-Authorization header to Authorization header for auth basic module!!

I think there are two cases interleaved here.

Client have to authenticate it self to nginx-proxy
nginx-proxy have to authenticate it self with upstream proxy

For the first case its fairly simple, just return 407 Proxy Authentication Required (if client does not provide credentials)
Next request client should provide its credentials which can be checked against standard password file.
For second case, we can use proxy_pass_header with client credentials or use proxy_set_header with hard coded credentials

hi @deba12
I have given a tricky way to support Proxy-Authorization via lua-nginx-module(rewrite_by_lua directive) and auth basic module.
See my comment of the test case: https://github.com/chobits/ngx_http_proxy_connect_module/issues/42#issuecomment-502985437.

hi @deba12
It's unnecessary for CONNECT tunnel, because CONNECT request is only handled by proxy_connect module, there is no header passed to backend on CONNECT request. And later request under CONNECT tunnel is proxied to backend without any modified.
I mean proxy_hide_header ( any directive of proxy module) wont work for CONNECT request and tunnel.

However, proxy_hide_header can work for plain http proxying, for example curl http://xxx -x ip:port via proxy_pass ( its http://xxx not https://).

@chobits yes, but for plain http connections we have to hide it anyway so why not to include it.
You know most people will just brainlessly copy paste this example. Their credentials will leak to upsteram

@chobits yes, but for plain http connections we have to hide it anyway so why not to include it.
You know most people will just brainlessly copy paste this example. Their credentials will leak to upsteram

Yes, I agree.  Have updated https://github.com/chobits/ngx_http_proxy_connect_module/issues/42#issuecomment-502985437.

     # If backend wont check Auth header, we should not pass the user/password.
       proxy_hide_header Authorization;
       proxy_hide_header Proxy-Authorization;

Hi, @chobits.
Thank you for the reply. I'm looking at your config.
    server {
        listen 8080;

Does it mean that credentials are sent in clear text?
Is it possible to encrypt connection before sending authentication data?
I'm bit confused. What I'm trying to achieve is fully encrypted connection to proxy.

Hi, @chobits.
Thank you for the reply. I'm looking at your config.
    server {
        listen 8080;

Does it mean that credentials are sent in clear text?
Is it possible to encrypt connection before sending authentication data?
I'm bit confused. What I'm trying to achieve is fully encrypted connection to proxy.

Yes you can config nginx with SSL protocol. For example, you can refer config of https://github.com/chobits/ngx_http_proxy_connect_module/issues/42#issuecomment-502577577.
And you can test it with proxytunnel command from 3.2 proxytunnel test case of https://github.com/chobits/ngx_http_proxy_connect_module/issues/42#issuecomment-502577577.

How to do access control using nginx auth basic module
Note that this auth-basic checking only works for CONNECT request, not for proxied requests on the CONNECT tunnel.

nginx config

enables proxy_connect module
enables auth basic module
exclamation transfer Proxy-Authorization header to Authorization header for CONNECT request via rewrite_by_lua directive (because auth basic module can only handle Authorization header)

$ cat conf/nginx.conf

    server {
        listen 8080;

        auth_basic "server auth";
        auth_basic_user_file /Users/xiaochen/work/github/tengine/output/conf/pwd;

        resolver                       <IP_ADDRESS>;

        # forward proxy for CONNECT request
        proxy_connect;
        proxy_connect_allow            all;

        # transfer Proxy-Authorization header to Authorization header
        rewrite_by_lua_file /Users/xiaochen/work/github/tengine/output/conf/proxy_auth.lua;

        # forward proxy for non-CONNECT request
        location / {
            proxy_pass http://$host;
            proxy_set_header Host $host;

            # If backend wont check Auth header, we should not pass the user/password.
            proxy_hide_header Authorization;
            proxy_hide_header Proxy-Authorization;
        }
    }

lua for header transfering and checking of https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/407

$ cat /Users/xiaochen/work/github/tengine/output/conf/proxy_auth.lua

-- check Proxy-Authorization for https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/407
if not ngx.var.http_proxy_authorization then
    ngx.header["Proxy-Authenticate"] = "Basic realm=\"Access to internal site\""
    ngx.exit(407)
end

-- transfer Proxy-Authorization header to Authorization for auth basic module
ngx.req.set_header("Authorization", ngx.var.http_proxy_authorization)

create user/password hello:world for auth_basic_user_file /Users/xiaochen/work/github/tengine/output/conf/pwd

$ htpasswd -c ./conf/pwd hello
New password:           <<<<<< input "world"
Re-type new password:     <<<<<<< input "world"
Adding password for user hello

$ cat conf/pwd
hello:$apr1$X2Xvv/bM$K17b2lTxih1JlTvyzFyWh1

my curl command:

auth passed

$ curl https://github.com -x localhost:8080  --proxy-user hello:world -sv -o /dev/null
HTTP/1.1 200 Connection Established
Proxy-agent: nginx

HTTP/1.1 200 OK
Date: Tue, 18 Jun 2019 07:34:05 GMT
Content-Type: text/html; charset=utf-8
Server: GitHub.com
Status: 200 OK
Vary: X-PJAX
...

auth failed without Proxy-Authorization header

$ curl https://github.com -x localhost:8080 -sv -o /dev/null
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8080 failed: Connection refused
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to github.com:443
> CONNECT github.com:443 HTTP/1.1
> Host: github.com:443
> User-Agent: curl/7.62.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 407
< Server: Tengine/2.3.0
< Date: Tue, 18 Jun 2019 08:03:59 GMT
< Content-Length: 0
< Connection: keep-alive
< Proxy-Authenticate: Basic realm="Access to internal site"
<
* Received HTTP code 407 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0

See Also

nginx documentation: ngx_http_auth_basic_module
wiki: HTTP Basic Authenticatio Protocol

hi, i donnot know why, but proxy_hide_header is not work here.. your code work?

support for this should really be added within the module not like this

@chobits do you have a suggestio to use #42 comment without using lua module/script?

ok,nice work;
I have made a change,when directly access the site(not use as a proxy)，use 401 WWW-Authenticate
when use as a proxy for CONNECT METHOD，use 407 Proxy-Authenticate
-- check Proxy-Authorization for https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/407
local http_method = ngx.var.request_method
if not ngx.var.http_proxy_authorization and http_method == "CONNECT" then
        ngx.header["Proxy-Authenticate"] = "Basic realm=\"Access to internal site\""
        ngx.exit(407)
end

-- transfer Proxy-Authorization header to Authorization for auth basic module
if http_method == "CONNECT" then
        ngx.req.set_header("Authorization", ngx.var.http_proxy_authorization)
end

Do you recommend using OpenResty to get lua-nginx-module and compiling that with ngx_http_proxy_connect_module?
Or the other way round?
As I have Nginx built with this module, but am having issues following install instructions https://github.com/openresty/lua-nginx-module/#installation

Do you recommend using OpenResty to get lua-nginx-module and compiling that with ngx_http_proxy_connect_module?
Or the other way round?
As I have Nginx built with this module, but am having issues following install instructions https://github.com/openresty/lua-nginx-module/#installation

If you just use a few functions of nginx OR personal use, there is no difference between using nginx and openresty.
But if you need to use a large number of nginx three-party modules to complete online business, and do not want to maintain the update of each module independently, it is highly recommended that you use nginx bundle (maybe openresty, NGINX Plus, Tengine ...)

@chobits do you have a suggestio to use #42 comment without using lua module/script?
Is it possible to rewrite and simplify the line below?
ngx.req.set_header("Authorization", ngx.var.http_proxy_authorization)
Reply late, this thread is too long.

There is no module that can directly modify and handle the request before the nginx proxy module. Maybe nginx-native module: Perl module can do this also, I rarely use it, only use it to write nginx test case. Lua  module is a convenient way to do programming for HTTP request.

This issue was resolved in https://github.com/chobits/ngx_http_proxy_connect_module/issues/42#issuecomment-502985437.
I Closed and locked this issue. If there is a new discussion, please open a new issue.


scram user creation (Upgrade terraform-provider-kafka to v0.5.3)

We are running our own fork of pulumi-kafka to be able to create Scram user and it's causing some headache.
Ideally, we can use the official provider to achieve that.
I understand that's blocked by https://github.com/pulumi/pulumi-kafka/issues/145
But i wonder if this is a real issue at all? We are not able to reproduce ourselves. we have also been using 0.5.3 in prod for managing Kafka creation in other prod workloads for quite some time now.
It would be a huge unblock if pulumi-kafka can support scram user creation.
Thanks!

Hi @seed-doordash - thank you for filing this enhancement request.
As soon as https://github.com/Mongey/terraform-provider-kafka/issues/329 is resolved to our satisfaction (our tests pass), we will resume upgrading. Unfortunately, we do not have the spare engineering cycles to resolve the upstream issue at this time.
If you are not seeing this issue on your end, perhaps you can share which program runs successfully. On our end, creation of a Kafka Topic fails with the same error as described in the upstream issue.
This should be resolved with the next upgrade of the upstream provider.

@guineveresaenger thanks for the response.
We are actually using this SHA
github.com/doordash/terraform-provider-kafka v0.0.0-20230117222634-7bd996447da2

We are on a fork for terraform provider. We will provide more details soon.


Front Camera

I've looked over everything, and I can’t see a way to use the front camera. I was wondering if I have missed something, and if not, how difficult it would be to implement. I don't have a very in-depth knowledge to Swift, but I have tried to look over some possibilities of how to do it, but achieved nothing

Not sure how needed is this as of now, but what I did was download the package, add it as a local package in Xcode, and edit line 170 of CodeScanner to the following:
let videoCaptureDevice = AVCaptureDevice.default(.builtInWideAngleCamera,for: .video, position: .front)

It would be nice to take this in as an argument when calling the scanner!
See more on Apple's Developer Website

@ichbineinnerd0 @AxelSariel I added an argument to CodeScannerView in my Pull Request #52 regarding videoCaptureDevice


NullPointerException in javax.xml.ws.Service.

Call to Provider.provider() from  javax.xml.ws.Service constructors returns null in Java 11 if osgi-resource-locator is on the classpath. This jar can be on the classpath as a result of a transitive dependency. The cause of the NullPointerException is because the call to isOSGi() from javax.xml.ws.spi.FactoryFinder#find can return true in Java 11 even if the application is not OSGi; whereas in Java 1.8, the call would return false under the same circumstance.
The NullPointerException actually comes as a result of org.glassfish.hk2.osgiresourcelocator.ServiceLoader#lookupProviderInstance returning null to javax.xml.ws.spi.FactoryFinder#lookupUsingOSGiServiceLoader. The reason ServiceLoader#lookupProviderInstance returns null is because the ServiceLoader#initialize method has never been called. The reason the initialize method has never been called is because this is not an OSGi application which would normally load and initialize the bundle.
The lookupUsingOSGiServiceLoader method is invoked because the call to javax.xml.ws.spi.FactoryFinder#isOSGi() returns true. On JDK1.8, isOSGi() returns false. So why does isOSGi() return true? All the method does is check to see if Class.forName("org.glassfish.hk2.osgiresourcelocator.ServiceLoader") throws a ClassNotFoundException. On JDK 11, the exception is not thrown, but it is thrown on JDK 1.8. To understand why, we need to look at what happens in Class.forName.
The first thing Class.forName does is find the class for the caller using Reflection#getCallerClass(). Both 1.8 and 11 return the FactoryFinder class, as expected. Next, it finds the ClassLoader for the caller class using (package-private) ClassLoader#getClassLoader. And here lies the difference. On 1.8, this returns null, which is the bootstrap class loader. On 11, this returns the application class loader.
The reason we get the bootstrap class loader on 1.8 is that the javax.xml.ws package is part of the JDK in 1.8. In Java 9, it was removed (along with other Java EE classes - see JEP 320: Remove the Java EE and CORBA Modules). So, on 11, these classes have to be added to the classpath. This is why when Class.forName gets the class loader for javax.xml.ws.spi.FactoryFinder.class on Java 11, it gets the application class loader. And the lookup of the o.g.h.o.ServiceLoader class succeeds on 11 because osgi-resource-locator-1.0.1.jar is also on the classpath (in this case, due to a transitive dependency).
It is worth noting that the FactoryFinder#find method is coded differently in jaxws-api than it was in JDK 1.8. In Java 11, the method goes through a sequence of checks to find the service provider. First it looks to see if there is a jaxws.properties file in the JRE and uses the properties there. If jaxws.properties is not found, it looks for the system property javax.xml.ws.spi.Provider. Failing that, it checks isOSGi(). If the OSGi check fails it assumes the default of com.sun.xml.internal.ws.spi.ProviderImpl.
The workaround is to set the system property javax.xml.ws.spi.Provider=com.sun.xml.internal.ws.spi.ProviderImpl and add a dependency to jaxws-ri to the project.
Recommended diff is:
`
diff --git a/api/src/main/java/jakarta/xml/ws/spi/FactoryFinder.java b/api/src/main/java/jakarta/xml/ws/spi/FactoryFinder.java
index 4e259d5..7375203 100644
--- a/api/src/main/java/jakarta/xml/ws/spi/FactoryFinder.java
+++ b/api/src/main/java/jakarta/xml/ws/spi/FactoryFinder.java
@@ -71,7 +71,8 @@ class FactoryFinder {
     // handling Glassfish (platform specific default)
     if (isOsgi()) {

       return (T) lookupUsingOSGiServiceLoader(factoryId);

       provider = (T) lookupUsingOSGiServiceLoader(factoryId);

       if (provider != null) return provider;
   }

   if (fallbackClassName == null) {

`

Thank you for detailed analysis! It came in just in time :-)

It is worth noting that the FactoryFinder#find method is coded differently in jaxws-api than it was in JDK 1.8.

see https://bugs.openjdk.java.net/browse/JDK-8131667 on when and why...
note that the lookup sequence is defined by the JAX-WS spec or now a days by Jakarta XML Web Services spec


Test coverage being missed with assert_cmd package

Hey,
Awesome package, I really appreciate your work maintaining this!
I'm trying to debug why tarpaulin isn't picking up my binary tests for my rust project rip2: https://github.com/MilesCranmer/rip2. I have started using assert_cmd to test the CLI functionality but it doesn't seem to get picked up by tarpaulin at all.
I have been running tarpaulin with the following options:
cargo tarpaulin --engine llvm --follow-exec --out=Lcov -- test_cli

I have an integration test here that uses assert_cmd to call the rip binary with different options.
However, these don't seem to be picked up. I've read through the solutions on https://github.com/xd009642/tarpaulin/issues/903 but they didn't seem to help me. Here is the current output:
Finished test [unoptimized + debuginfo] target(s) in 13.21s
2024-04-11T03:44:32.262681Z  INFO cargo_tarpaulin::process_handling: running /Users/mcranmer/PermaDocuments/rip/target/debug/deps/rip-9723570d6298e012
2024-04-11T03:44:32.262727Z  INFO cargo_tarpaulin::process_handling: Setting LLVM_PROFILE_FILE

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

2024-04-11T03:44:32.438859Z  INFO cargo_tarpaulin::statemachine::instrumented: For binary: target/debug/deps/rip-9723570d6298e012
2024-04-11T03:44:32.438870Z  INFO cargo_tarpaulin::statemachine::instrumented: Generated: target/tarpaulin/profraws/rip-9723570d6298e012_14156139374758574137_0-62036.profraw
2024-04-11T03:44:32.438873Z  INFO cargo_tarpaulin::statemachine::instrumented: Merging coverage reports
2024-04-11T03:44:32.447034Z  INFO cargo_tarpaulin::statemachine::instrumented: Mapping coverage data to source
2024-04-11T03:44:32.565754Z  INFO cargo_tarpaulin::process_handling: running /Users/mcranmer/PermaDocuments/rip/target/debug/deps/unit_tests-58106ca4ac690a6c
2024-04-11T03:44:32.565804Z  INFO cargo_tarpaulin::process_handling: Setting LLVM_PROFILE_FILE

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.00s

2024-04-11T03:44:32.866785Z  INFO cargo_tarpaulin::statemachine::instrumented: For binary: target/debug/deps/unit_tests-58106ca4ac690a6c
2024-04-11T03:44:32.866797Z  INFO cargo_tarpaulin::statemachine::instrumented: Generated: target/tarpaulin/profraws/unit_tests-58106ca4ac690a6c_11063364119811431340_0-62037.profraw
2024-04-11T03:44:32.866799Z  INFO cargo_tarpaulin::statemachine::instrumented: Merging coverage reports
2024-04-11T03:44:32.882143Z  INFO cargo_tarpaulin::statemachine::instrumented: Mapping coverage data to source
2024-04-11T03:44:33.134842Z  INFO cargo_tarpaulin::process_handling: running /Users/mcranmer/PermaDocuments/rip/target/debug/deps/rip2-5a19ef321941f7f6
2024-04-11T03:44:33.134918Z  INFO cargo_tarpaulin::process_handling: Setting LLVM_PROFILE_FILE

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

2024-04-11T03:44:33.392793Z  INFO cargo_tarpaulin::statemachine::instrumented: For binary: target/debug/deps/rip2-5a19ef321941f7f6
2024-04-11T03:44:33.392804Z  INFO cargo_tarpaulin::statemachine::instrumented: Generated: target/tarpaulin/profraws/rip2-5a19ef321941f7f6_12520403362171216094_0-62038.profraw
2024-04-11T03:44:33.392806Z  INFO cargo_tarpaulin::statemachine::instrumented: Merging coverage reports
2024-04-11T03:44:33.403659Z  INFO cargo_tarpaulin::statemachine::instrumented: Mapping coverage data to source
2024-04-11T03:44:33.517893Z  INFO cargo_tarpaulin::process_handling: running /Users/mcranmer/PermaDocuments/rip/target/debug/deps/integration_tests-7b491feadfa88650
2024-04-11T03:44:33.517943Z  INFO cargo_tarpaulin::process_handling: Setting LLVM_PROFILE_FILE

running 7 tests
test test_cli::scenario_1___help__ ... ok
test test_cli::scenario_2___help2__ ... ok
test test_cli::scenario_3___bury_unbury__ ... ok
test test_cli::scenario_4___bury_seance__ ... ok
test test_cli::scenario_5___bury_unbury_seance__ ... ok
test test_cli::scenario_6___inspect__ ... ok
test test_cli::scenario_7___inspect_no__ ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 11 filtered out; finished in 0.43s

2024-04-11T03:44:34.193509Z  INFO cargo_tarpaulin::statemachine::instrumented: For binary: target/debug/deps/integration_tests-7b491feadfa88650
2024-04-11T03:44:34.193520Z  INFO cargo_tarpaulin::statemachine::instrumented: Generated: target/tarpaulin/profraws/integration_tests-7b491feadfa88650_17302080270570845666_0-62039.profraw
2024-04-11T03:44:34.193524Z  INFO cargo_tarpaulin::statemachine::instrumented: Merging coverage reports
2024-04-11T03:44:34.211205Z  INFO cargo_tarpaulin::statemachine::instrumented: Mapping coverage data to source
2024-04-11T03:44:34.460016Z  INFO cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| src/args.rs: 0/24 +0.00%
|| src/lib.rs: 0/263 +0.00%
|| src/main.rs: 0/16 +0.00%
|| src/util.rs: 0/38 +0.00%
||
0.00% coverage, 0/341 lines covered, +0.00% change in coverage

^ Oddly you can see that despite all the test_cli tests being run, where this line is used to get the binary for the package, the coverage doesn't get picked up. The main tests that should trigger this behavior are test_cli::* in the integration tests.
(If I turn on the other tests which directly call the library functions, the coverage does work. It's just the binary/CLI components I'm struggling with).
The solutions on https://github.com/xd009642/tarpaulin/issues/903 didn't seem to help me sadly. If there are any other CLI flags I can try, please let me know.
Can reproduce this with:
git clone https://github.com/MilesCranmer/rip2
cd rip2
cargo tarpaulin --engine llvm --follow-exec --out=Lcov -- test_cli

^ Which will show the 0%.
Cheers,
Miles

Hmm I'll have to look into it once the nightly issues are resolved, one thought though is that follow-exec doesn't work necessarily properly on llvm coverage depending on your tests the profraw files may be written after your test asserts the command works properly (depending on the method of the assert). So the arg --post-test-delay was designed to make it work by adding a fudge factor delay after the main test process exits to collect all the profraws

Thanks, I tried --post-test-delay=10 but still at 0% unfortunately.


Not sure if this is construct related or not, ut take a look, sending it here because your mod is mentioned in the error.

Exception: java.util.ConcurrentModificationException
16:52:58.148 ERROR core.CardCrawlGame> Exception caught
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(Unknown Source) ~[?:1.8.0_171]
at java.util.ArrayList$Itr.next(Unknown Source) ~[?:1.8.0_171]
at basemod.BaseMod.publishPostDungeonInitialize(BaseMod.java:1887) ~[?:?]
at basemod.patches.com.megacrit.cardcrawl.dungeons.AbstractDungeon.PostDungeonInitializeHook.Postfix(PostDungeonInitializeHook.java:9) ~[BaseMod.jar:?]
at com.megacrit.cardcrawl.dungeons.AbstractDungeon.initializeRelicList(AbstractDungeon.java:1766) ~[?:?]
at com.megacrit.cardcrawl.dungeons.Exordium.(Exordium.java:52) ~[?:?]
at com.megacrit.cardcrawl.core.CardCrawlGame.getDungeon(CardCrawlGame.java:1291) ~[?:?]
at com.megacrit.cardcrawl.core.CardCrawlGame.update(CardCrawlGame.java:810) ~[?:?]
at com.megacrit.cardcrawl.core.CardCrawlGame.render(CardCrawlGame.java:415) [?:?]
at com.badlogic.gdx.backends.lwjgl.LwjglApplication.mainLoop(LwjglApplication.java:225) [?:?]
at com.badlogic.gdx.backends.lwjgl.LwjglApplication$1.run(LwjglApplication.java:126) [desktop-1.0.jar:?]
Controllers: removed manager for application, 0 managers active
16:52:59.157 INFO core.CardCrawlGame> PAUSE()
16:52:59.157 INFO core.CardCrawlGame> Game shutting down...
16:53:01.573 INFO helpers.File> Failed to copy file=D:\SteamLibrary\steamapps\common\SlayTheSpire\preferences\THE_CONSTRUCT_MOD
java.nio.channels.ClosedByInterruptException: null
at java.nio.channels.spi.AbstractInterruptibleChannel.end(Unknown Source) ~[?:1.8.0_171]
at sun.nio.ch.FileChannelImpl.size(Unknown Source) ~[?:1.8.0_171]
at java.nio.file.Files.readAllBytes(Unknown Source) ~[?:1.8.0_171]
at com.megacrit.cardcrawl.helpers.File.validateWrite(File.java:140) ~[desktop-1.0.jar:?]
at com.megacrit.cardcrawl.helpers.File.copyAndValidate(File.java:97) [desktop-1.0.jar:?]
at com.megacrit.cardcrawl.helpers.File.save(File.java:49) [desktop-1.0.jar:?]
at com.megacrit.cardcrawl.helpers.FileSaver.consume(FileSaver.java:42) [desktop-1.0.jar:?]
at com.megacrit.cardcrawl.helpers.FileSaver.run(FileSaver.java:25) [desktop-1.0.jar:?]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_171]
16:53:01.574 INFO helpers.File> java.lang.InterruptedException: sleep interrupted

Hmm, could you provide a bit more context on when the error occurred / what you were doing?
Some specific questions:

What other mods did you have active?
Does this keep happening, or did it work next time you tried to load the game?
If you go to that file path (D:\SteamLibrary\steamapps\common\SlayTheSpire\preferences), is there a THE_CONSTRUCT_MOD file there at all?

The mods that were active at the time were: Basemod, BetterRewards, Black Mage, Colored Map, Construct Mod (obviously), FechMod, FruityMod, HexCores, MadScienceMod, MoreRelics, NecroMod, ReplayTheSpire, STS_Alwayswhale, SynergyRelic and WitchMod.
As for what I were doing. well the latest release put Endless into the main branch so I were on an Endless Run, had just beaten Act 3 and were gong back to see Neow when it just CTD'd me with that error.
There was a THE_CONSTRUCT_MOD file in the preferences menu so not sure why it didn't find it.
Upon starting up the game and retrying it just now (haven't played since this bug occurred) I could enter the new cycle just fine and going for another 3 cycles it still have not crashed, so I am honestly not sure whats going on.

Yeah, that's pretty weird - I just tested an endless run with that setup and it didn't crash or anything. I'll leave the issue up in case you or anyone else experiences it again. It's probably an interaction between two or more mods, but with your list that's a lot of options :)

I will put up a copy of this on Basemod's issue tracker part as that is also mentioned, might be that they can make heads or tails of this.
mentioned part is: basemod.patches.com.megacrit.cardcrawl.dungeons.AbstractDungeon.PostDungeonInitializeHook.Postfix(PostDungeonInitializeHook.java:9) ~[BaseMod.jar:?]
so this might actually be a Basemod issue


fixed function name

I only fixed a typo

Codecov Report

Merging #153 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #153   +/-   ##
=======================================
  Coverage   97.24%   97.24%
=======================================
  Files           1        1
  Lines         145      145
=======================================
  Hits          141      141
  Misses          4        4

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 264e15a...548cd9b. Read the comment docs.

Thanks for the PR. Unfortunately that's not a typo. We could've named that variable batman and it would've been valid.

Thanks for the PR. Unfortunately that's not a typo. We could've named that variable batman and it would've been valid.

But you use a wrong function name three lines later....

And that's why we have a Pull Request template. You chose to remove that template. If you hadn't, you could have explained that and the change would have made sense.


[Fix] Fix clickhouse local generate file has <0x00> (#6089)

Purpose of this pull request
fix clickhouse local generate file has <0x00>,it will cause
Does this PR introduce any user-facing change?
no
How was this patch tested?
after fix，i success transfer 300KW records from hive to ck
Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

Hi @zhanggougou , thanks for contribute this. Could you explain the reason of change?

you can see this issue https://github.com/apache/seatunnel/issues/6089
when 【buffer.position() + byteData.length > buffer.capacity()】，a new MappedByteBuffer will generate,but last MappedByteBuffer also has some space ,because the bufferSize is fixed
so i compute fileChannel.position，when buffer is not enough,the next will continue previous position,so the local_data will not contains unnecessary <0x00>
Is my problem？but when i fixed it，i succesed transfer data 【OR higher version ck local can ignore <0x00>?】
Looking forward to your reply，thanks

Can you add e2e to test this pr?

Can you add e2e to test this pr?

sorry，i am a newcomer in this project，e2e means？
now，i suspect if this is a bug，give me some time，i will transfer data again to test


Reproduction question for SDXL Base 1.0

when I reproduce the v2.1 benchmark，I tested and found SDXL Base 0.9 to be 28.71 and SDXL Base 1.0 to be 28.83, but the scores in the README exceed 30. I used the default settings of diffusers.

Even when I ran the given code, I only got an average score of 28.71.
import hpsv2

print(hpsv2.get_available_models()) # Get models that have access to data
hpsv2.evaluate_benchmark('')

result：
SDXL-base-0.9 concept-art     29.42      0.2775
SDXL-base-0.9 anime           31.03      0.2606
SDXL-base-0.9 paintings       28.83      0.5296
SDXL-base-0.9 photo           25.56      0.5209
SDXL-base-0.9 Average         28.71

hpsv2.evaluate(output_dir, hps_version="v2.1")

I got the same question, the following are results from my reproduction:
SDXL-base-0.9 concept-art     29.42
SDXL-base-0.9 anime           31.03
SDXL-base-0.9 paintings       28.83
SDXL-base-0.9 photo           25.56
SDXL-base-0.9 Average         28.71
shifting from the reported results.

Even when I ran the given code, I only got an average score of 28.71.
import hpsv2

# print(hpsv2.get_available_models()) # Get models that have access to data
hpsv2.evaluate_benchmark('SDXL Base 0.9')

result：
SDXL-base-0.9 concept-art     29.42      0.2775
SDXL-base-0.9 anime           31.03      0.2606
SDXL-base-0.9 paintings       28.83      0.5296
SDXL-base-0.9 photo           25.56      0.5209
SDXL-base-0.9 Average         28.71

same question


Crash in iOS

When xcode is compiled and passed, the app will have a certain probability of crash.
As shown in the figure below

Does anybody have a look?
React Native version:
0.59.4

@xutm Hi, seems you have two bridges? Can you still reproduce? I think it may related to some race condition, can you show how other JavaScript thread is doing like com.facebook.react.JavaScript(41) below.

@zhongwuzw

two bridges when App init.(One cache, one not)
have a certain probability of crash.(Low frequency)

Hello there 👋 this issue doesn't have a repro (which means, a react-native init-ed project with the minimal changes that leads to creating the same issue you are reporting). Unfortunately, I have no way of helping you in a meaningful way – there is no easy way for me to recreate the situation and check that the issue reported is still there when changing the code.
Also, it's really hard to help you when you don't follow the provided template.
Because of this, we are going to close this issue - but if a repro is shared, we are happy to reopen it 🤗

@xutm Emm, seems your infos are not sufficient to investigate by us, as @kelset said, please fill in the issue template and more importantly for this crash is provide the entire crash log, contains all related infos. After that, we can fix this issue and reopen it. :)


Projects imported from github should run npm start automatically after build just like next.js

Describe the bug
npm start is a standard command, so non-next.js projects should be able to utilize this as well
Link to the blitz that caused the error
To Reproduce
Steps to reproduce the behavior:

https://stackblitz.com/github/Rest-Hooks/todo-example

Expected behavior
After npm install finishes, npm start should run

We use a framework detection to figure out the command we want to run. If it's a nextjs project, we will automatically run next dev. If we don't detect which framework it is (we currently only do this for next but extending this very soon), we don't do anything.
You can change the behaviour of how this works by adding a .stackblitzrc file to the blitz and setting startCommand to whatever you want. (https://stackblitz.com/edit/github-tvonzc?file=.stackblitzrc)
It might make sense to run npm start by default for every project that has it. We'll take this under considiration.

Are there docs on stackblitzrc anywhere?


Broadcast input terminal please????

Broadcast Input menu item and click on the Add to Broadcast menu item???

Thanks for the suggestion! This is actually already being tracked by another issue on our repo - please refer to #2634 for more discussion.
/dup #2634


Shutter not linked to button on Squarestation

This shutter on R&D on Squarestation is not linked to it's respective button.
Steps:
1.) Go to Squarestation.
2.) Go to R&D.
3.) Hit the right-most button in robotics.
4.) See results.
Examples:

No there. Open issue on unitystation/unitystation


Speedup stationary qubit

This PR speeds up StationaryQubit by eliminating the usage of par() as normal variables.

This change is

@res6idue hey, could you rebase this?


Cannot add a resource in registry via carbon console

Description
When adding a resource to the registry via carbon console, it keeps on processing without returning a successful response
Steps to Reproduce
Follow the steps under Configure per tenant custom domain mappings [1]
[1] https://apim.docs.wso2.com/en/4.3.0/reference/customize-product/customizations/customize-the-api-store-and-gateway-urls-for-tenants/

Affected Component
APIM
Version
4.3.0
Environment Details (with versions)
No response
Relevant Log Output
No response
Related Issues
No response
Suggested Labels
No response

Tested this on the latest version and couldn't reproduce the behaviour

This issue still persists in the Firefox browser. Works fine with Chrome.

Yes, observed the behaviour in Firefox 124.0.1 in the latest(Post alpha2) pack

This issue is there in API Manager 4.2.0 version as well,


Complete without snippet expansion

I have a question.
Is it possible to confirm completion without snippet expansion by a custom shortcut (using <C-CR>, for example)?
I.e. when completing a function:

Insert it like this:

Instead of this:

Because I trying to pass a function into another function.

I thought this feature was feasible with the current feature set, but the perfect solution seems difficult.
Therefore, it is necessary to add a function.
Note that this feature is a bit special.
nvim-cmp has carefully created the "text inserted when selecting an item", but in some cases it is not perfect.
google translated

What I want to say is that nvim-cmp implements the text state after "expand" as the server intended, but the text state at the time of selection is not always perfect. ..

As a workaround i press Ctrl+n - it inserts function name without substituting arguments. But it gets problematic when there is a single completion item or the item I need is preselected (via rust magic completion, for example). In this case I press Ctrl+n and then Ctrl+p. Not very convenient, but better then nothing.

Maybe I can create a hacky workaround by sending Ctrl+n with Ctrl+p to insert in my configuration?

Yes. <C-n><C-p> combo can be usable as workaround. But you have to write some codes.

It's may work for you.
      ['<C-j>'] = function()
        local active = cmp.get_active_entry()
        if active then
          return
        end
        local next = '<Cmd>lua require("cmp").select_next_item()<CR>'
        local prev = '<Cmd>lua require("cmp").select_prev_item()<CR>'
        local close = '<Cmd>lua require("cmp").close()<CR>'
        vim.api.nvim_feedkeys(vim.api.nvim_replace_termcodes(next .. prev .. close, true, true, true), 'n', true)
      end,

Could you help me with this a little bit?

Thank you!
Can I make this work when an item was selected? It inserts messed text If I comment out the check for active.

Example:

Reluctantly, I needed vim.schedule to make it work.
The nvim-cmp mapping process needs to be refactored.
Anyway, I think the following implementation works.
      ['<C-j>'] = function()
        local active = cmp.get_active_entry()
        if active then
          return cmp.close()
        end
        local selected = cmp.get_selected_entry()
        if not selected then
          cmp.select_next_item()
        else
          cmp.select_next_item()
          cmp.select_prev_item()
        end
        vim.schedule(function()
          cmp.close()
        end)
      end,

Hm... It inserts wrong item for me:

@hrsh7th sorry for the ping, but I think I know why the solution doesn't work for me. cmp.get_selected_entry() for some reason returns nil even in this situation:

Do you have any idea why it happens?

hrsh7th's solution also didn't work for me. Eventually I foud workaround, by remapping Up/Down keys by cmp.select_next_item() / cmp.select_prev_item() ( similar to C-n and C-p ) and pressing Escape keeps the signature without snippet.
 opts.mapping['<Down>'] = cmp.mapping(function(fallback)
                if cmp.visible() then
                    cmp.select_next_item()
                else
                    fallback()
                end
            end
            , {"i", "c"})
        opts.mapping['<Up>'] = cmp.mapping(function(fallback)
                if cmp.visible() then
                    cmp.select_prev_item()
                else
                    fallback()
                end
            end
            , {"i", "c"})

hrsh7th's solution also didn't work for me. Eventually I foud workaround, by remapping Up/Down keys by cmp.select_next_item() / cmp.select_prev_item() ( similar to C-n and C-p ) and pressing Escape keeps the signature without snippet.
 opts.mapping['<Down>'] = cmp.mapping(function(fallback)
                if cmp.visible() then
                    cmp.select_next_item()
                else
                    fallback()
                end
            end
            , {"i", "c"})
        opts.mapping['<Up>'] = cmp.mapping(function(fallback)
                if cmp.visible() then
                    cmp.select_prev_item()
                else
                    fallback()
                end
            end
            , {"i", "c"})

what's the difference of type C-n C-p and esc as mentioned above?

I think a workaround in snippet engine can help:
``lua
snippet = {
expand = function(args)
if not expand then
local function remove_bracket_contents(input)
local pattern = "^(.)%b().$"
local result = string.gsub(input, pattern, "%1")
return result
end
args.body = remove_bracket_contents(args.body)
expand = true
end
require("luasnip").lsp_expand(args.body)
end,
},
[""] = cmp.mapping(function()
if cmp.visible then
expand = false
cmp.confirm()
end
end),

mapping = cmp.mapping.preset.insert {
  ['<CR>'] = function()
    cmp.select_next_item { behavior = cmp.SelectBehavior.Insert, count = 0 }
    cmp.close()
  end,
}

mapping = cmp.mapping.preset.insert {
  ['<CR>'] = function(fallback)
    if cmp.visible() then
      cmp.select_next_item { behavior = cmp.SelectBehavior.Insert, count = 0 }
      cmp.abort()
    else
      fallback()
    end
  end,
}

Thanks!


combust module not defines

I tried to run the code, but I obtained the following error.
ModuleNotFoundError: No module named 'combust'
coming from this line:
https://github.com/THGLab/NewtonNet/blob/fcf2af848a1c998bd08096dcefb58a5610eda03c/newtonnet/data/neighbors.py#L15
Did I miss something during the installation? I tried to find this combust module, but I've been unsuccessful so far.

You can import newtonnet.utils instead.


More than one qBittorrent instance

Hello,
I run two instances of qBittorrent in my pc: one for private trackers, and another for the same files on public trackers. I cannot use one for both as qBittorrent would merge them.
Would this work in this case? If not, is there a workaround? I don't know, maybe creating a docker image with just qBittorrent and this script?
All the best!

Should not be a problem. Just make sure to use different ports for different qBittorent instances, or just don't run them simultaneously.


GPT-2 fine-tuned with LoRA and trainable embedding and predicion head FAILED to be loaded and inference

System Info
I am trying to fine-tune a pre-trained GPT-2 chatbot with LoRA and with some additional special tokens such as '<end of turn>' and '<end of dialog>'. Because the Embedding layer is expanded for new tokens and the prediction head is randomly initialized, they should be trained together with LoRA adapters.
But after I've done the fine-tuning and trying to load and inference in another script, the model only gives out unstructured outputs or just repeating newly added token '<end of dialog>', while another full parameter fine-tuned model with exactly hyperparameters behaves much better (at least can output some chinese characters).
I was convinced that something is wrone during the parameter loading process in the inference script, because the output of the model during training looks quite good.
Here's my training script and inference script:
train.py:
from transformers import GPT2LMHeadModel, GPT2Tokenizer, TrainingArguments, Trainer, TrainerCallback, pipeline
from peft import LoraConfig, get_peft_model
import torch
from sys import argv

from utils import CustomTextDataset

script, LoRA = argv
LoRA = LoRA == 'True'

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Current Device: ', device)

print("LoRA: ", LoRA)

model = GPT2LMHeadModel.from_pretrained('gpt2')
model = model.to(device)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

special_tokens_dict = {'additional_special_tokens': ['<end of turn>', '<end of dialog>']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
assert num_added_toks > 0

model.resize_token_embeddings(len(tokenizer))
model = torch.compile(model)

if LoRA:
    config = LoraConfig(
        r=32,
        lora_alpha=32,
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM",
        # tune the embedding layer and prediction head
        modules_to_save=["wte", "lm_head"]
    )
    model = get_peft_model(model, config)

train_dataset = CustomTextDataset(
    name='train_dataset_medium',
    tokenizer=tokenizer,
    file_path="/home/baily/eureka/datasets/train_medium.txt",
    block_size=128)

val_dataset = CustomTextDataset(
    name='valid_dataset_medium',
    tokenizer=tokenizer,
    file_path="/home/baily/eureka/datasets/val_medium.txt",
    block_size=128)

training_args = TrainingArguments(
    output_dir="/home/baily/eureka/gpt-2-lora",
    overwrite_output_dir=True,
    num_train_epochs=5,
    per_device_train_batch_size=64,
    per_device_eval_batch_size=128,
    logging_steps=200,
    evaluation_strategy='epoch',
    save_steps=2000,
    save_total_limit=10,
    warmup_steps=5000,
    gradient_accumulation_steps=2,
    learning_rate=3e-5,
    fp16=True,
    )

def collate_fn(data):
    out = torch.stack(data)
    return {'input_ids': out,
            'labels': out}

class Callbacks(TrainerCallback):
    def on_save(self, args, state, control, **kwargs):
        if LoRA:
            model.save_pretrained(f'./checkpoint-{state.global_step}/lora/')
        tokenizer.save_pretrained(f'./checkpoint-{state.global_step}/tokenizer/')
        model.config.to_json_file(f'./checkpoint-{state.global_step}/config.json')
    def on_log(self, args, state, control, **kwargs):
        prompt = '你好。<end of turn>'
        chatbot = pipeline('text-generation', model=model, tokenizer=tokenizer, device=0)
        model_input = tokenizer.encode(prompt, return_tensors='pt').to('cuda')
        generated_ids = model.generate(input_ids=model_input, max_length=10, pad_token_id=tokenizer.eos_token_id)
        generated_text = tokenizer.decode(generated_ids[:, model_input.shape[-1]:][0], skip_special_tokens=True)
        print(f"Response to '{prompt}':", generated_text)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=collate_fn,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    callbacks=[Callbacks(),]
)
trainer.train()

chat.py:
from transformers import GPT2LMHeadModel, GPT2Tokenizer, pipeline
import torch
import os
from sys import argv
_, chechpoint_name = argv

model_name = "gpt2"
chechpoint_path = f'./checkpoint-{chechpoint_name}/'

tokenizer = GPT2Tokenizer.from_pretrained(chechpoint_path+'tokenizer/')

def load_base_model(tokenizer, lora):
    model = GPT2LMHeadModel.from_pretrained(model_name)
    model.resize_token_embeddings(len(tokenizer))
    if lora:
        print("Loading Model Trained WITH LoRA...")
        lora_state_dict = torch.load(chechpoint_path+'lora/adapter_model.bin')
        print("Number of fine-tuned parameters:", sum([p.flatten().shape[0] for p in lora_state_dict.values()]))
        processed_state_dict = model.state_dict()
        params_to_load = 0
        for k, v in lora_state_dict.items():
            if 'lm_head' in k:
                processed_state_dict['lm_head.weight'] = v
                params_to_load += 1
            elif 'wte' in k:
                processed_state_dict['transformer.wte.weight'] = v
                params_to_load += 1

        # Assuring that only embedding layer and prediction head
        # were full parameter fine-tuned
        state_dict = torch.load(chechpoint_path+'pytorch_model.bin')
        saved_state_dict = {}
        for k, v in state_dict.items():
            # to get rid of 'base_model.model._orig_mod' prefix of param name
            if 'transformer' in k:
                k = k[k.index('transformer'):]
            if 'original_module' in k or 'lora_' in k:
                continue
            elif 'modules_to_save' in k:
                if 'lm_head' in k:
                    k = 'lm_head.weight'
                elif 'wte' in k:
                    k = 'transformer.wte.weight'
            saved_state_dict[k] = v
        print(len(list(saved_state_dict.keys())) == len(list(processed_state_dict.keys())))
        # saved_state_dict: transformer layers & embedding layer & prediction head from checkpoint
        # processed_state_dict: transformer layers from huggingface pre-trained GPT-2,
        #                       embedding layer & prediction head tuned together with LoRA A & Bs
        value_match_dict = {}
        for key in saved_state_dict.keys():
            assert key in list(processed_state_dict.keys())
            value_match_dict[key] = torch.all(saved_state_dict[key].to('cpu') == processed_state_dict[key].to('cpu')).item()
        assert all(list(value_match_dict.values())) # Check whether two state_dict are exactly the same

    else:
        print("Loading Model Trained WITHOUT LoRA...")
        processed_state_dict = {}
        state_dict = torch.load(chechpoint_path+'pytorch_model.bin')
        for k, v in state_dict.items():
            if '_orig_mod.' in k: # prefix added after model = torch.compile(model)
                k = k.replace('_orig_mod.', '')
            processed_state_dict[k] = v
    print(model.load_state_dict(processed_state_dict))
    if lora:
        from peft import PeftModel
        # load parameters of adapters
        model = PeftModel.from_pretrained(model, chechpoint_path+'lora/')
    return model

model = load_base_model(tokenizer=tokenizer, lora=os.path.exists(chechpoint_path+'lora/'))

chatbot = pipeline('text-generation', model=model, tokenizer=tokenizer, device=0)

turn = 1
dialog_history = ""

while True:
    user_input = input("User {}: ".format(turn))
    dialog_history += user_input + '<end of turn>'

    model_input = tokenizer.encode(dialog_history, return_tensors='pt').to('cuda')  # move input tensor to cuda
    generated_ids = model.generate(input_ids=model_input, max_length=100, pad_token_id=tokenizer.eos_token_id)
    generated_text = tokenizer.decode(generated_ids[:, model_input.shape[-1]:][0], skip_special_tokens=True)
    generated_text = ''.join([token for token in generated_text.split(' ')]) # get rid of space
    dialog_history = (dialog_history + generated_text + ' <end of turn> <end of dialog>')[:-500]

    print("GPT {}: {}".format(turn, generated_text))
    turn += 1

So I'm really asking what is the correct way to load LoRA tuned model parameters when some parts of the base model are not frozen?
Who can help?
No response
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder
[X] My own task or dataset (give details below)

Reproduction

Expected behavior

I'm experiencing a very similar issue. With the special tokens it also stopped interpreting unicode characters


SerNet Samba 4.2: no such option: --workgroup

Using the sernet packages in version 4.2 i got the following error msg:
Notice: /Stage[main]/Samba::Dc/Exec[provisionAD]/returns: samba-tool domain provision: error: no such option: --workgroup
I found the source of the issue here:
https://lists.samba.org/archive/samba/2015-April/190857.html
Looks like it was removed in 4.2 because it was always useless. I would suggest removing that option when samba version is => 4.2

Thanks


Fixed connect(), which now returns the error

Connect was providing a wrong parameter to WlanConnect; as I was at it, I also started to return the error code (0 means success), so that it would be easier to debug issues.

thanks!


Integrating with sentry.io. (or a locally hosted version)

Optional dependency.

Coverage decreased (-0.008%) to 63.285% when pulling 1b7c8bfff3330da7353b4badb1b399cc9f4daf0d on sentry_integration into 82d159a9081abd2e23055d7ebedacc8208381013 on develop.


Support for @FindBys and @FindAll annotations

ExtendedFieldDecorator handles only those fields which has @FindBy and @FindAI annotations now, please add the support for @FindAll and @FindBys annotations so that any variations in locators can be handled.
This featured is needed as the elements inside a component are not same across all the pages in the applications.

Added support of all Selenium and Appium annotations

will be verified in auto test in carina-demo


Find my mouse method is enabled even set to off

Microsoft PowerToys version
0.72.0
Installation method
Microsoft Store
Running as admin
No
Area(s) with issue?
Mouse Utilities
Steps to reproduce
In "Mouse Utilities" tab, set "Enable Find My Mouse" to Off;
In this case even pressed left Ctrl twice Find My Mouse is showed.
✔ Expected Behavior
Find My Mouse shouldn't be shown.
❌ Actual Behavior
Find My Mouse is shown.
Other Software
No response

Is this still a problem for you in the latest version of PowerToys? Alternatively, this issue may shed some light on a potential cause: https://github.com/microsoft/PowerToys/issues/29410
/needinfo

No! There are at least 3 updates that I remember that actually fixed the
problem. Thanks!
Em seg., 22 de jul. de 2024 às 15:46, Ethan Fang @.***>
escreveu:

Is this still a problem for you in the latest version of PowerToys?
Alternatively, this issue may shed some light on a potential cause: #29410
https://github.com/microsoft/PowerToys/issues/29410
/needinfo
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/PowerToys/issues/28106#issuecomment-2243596929,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AO3CVM6W3ZQ7QBKZQ34ZMPDZNVHQHAVCNFSM6AAAAABLI4BUUWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBTGU4TMOJSHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
[image: img] http://ajinsoft.com.br/
Eduardo Andrade
Desenvolvedor
(11) 4382-7188
ajinsoft.com.br
Av. Cauaxi, 350 - Conj. 104 - Alphaville Industrial
Barueri - SP https://maps.app.goo.gl/PUJvn81n58Kq9YgG7


TypeError: Cannot read property 'addReference' of undefined

I'm using Gridsome ^0.3.0, and @gridsome/source-wordpress, ^0.1.0, and I'm running into the following error upon running gridsome develop:
TypeError: Cannot read property 'addReference' of undefined
    at WordPressSource.fetchWordPressContent (/Users/andrew/code/SpiralPraxis/marketing/node_modules/@gridsome/source-wordpress/index.js:99:20)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

I've got a vanilla installation. I haven't modified any files except gridsome.config.js, and that only to provide a URL to my wordpress site, which is enabled to serve wp-json endpoints.
Here's my config file:
module.exports = {
  plugins: [{
    use: '@gridsome/source-wordpress',
    options: {
      baseUrl: 'https://mywordpresssite.com', // required
      typeName: 'WordPress', // GraphQL schema name (Optional)
      perPage: 100, // How many posts to load from server per request (Optional)
      concurrent: 10, // How many requests to run simultaneously (Optional)
      routes: {
        post: '/:year/:month/:day/:slug', //adds route for "post" post type (Optional)
        post_tag: '/tag/:slug' // adds route for "post_tag" post type (Optional)
      }
    }
  }]
}

Here's the relevant code from @gridsome/source-wordpress/index.js:

    for (const type in taxonomies) {
      const options = taxonomies[type]
      const typeName = makeTypeName(type)
      const route = routes[type] || `/${options.rest_base}/:slug`

      restBases.taxonomies[type] = options.rest_base

      for (const type of options.types) {
        const postTypeName = makeTypeName(type)
        const collection = getContentType(postTypeName)

        collection.addReference(options.rest_base, { typeName, key: '_id' })
      }

      addContentType({ typeName, route })
    }

Not sure what to do. I can't see anything in the documentation that clarifies this issue. Help would be appreciated.

Your PR is merged and published :) Can you verify that it is working now?


feat Port/Combinatorics.SimpleGraph.Coloring

@arienmalec Thanks! I finished the last three points:
Coloring.card_colorClasses_le has to do with two things: the first is that Lean 4 can't see that it already has a Fintype (Setoid.classes (Setoid.ker C)) (it used to get it using unification with the goal). I moved that into a haveI. The second is that it gets the wrong instance and you need to do a convert/apply Subsingleton.elim dance.
coloringOfIsEmpty was to do with how implicit arguments for lambdas work. I guess it's subtly different -- I didn't bother to investigate beyond finding a fix.
colorable_chromaticNumber was fixable with classical. It's the same sort of error as the first, where it's not picking up an instance in the goal. (The goal already has a classical instance, so it's not harming anything by being explicit with classical.)


fix: correct onEventDrop parameter order

The order returned from fullcalendar eventDrop is $newEvent, $oldEvent, $relatedEvents.

missed that. thanks!


Anonymous access allowed to private projects

Hello, we've deployed Harbor 1.4 recently, we've made several projects and are pushing images to them with Jenkins, however in our db access logs the username is listed as anonymous instead of the authentication username provided by Jenkins. It appears that we are able to do a manual push through docker to our project without setting any config authentication or docker login (docker client 18.01.0-ce). I see read_only access is available in Harbor 1.5 but I assume that private projects should not be accessible by non-users?
empty docker config for registry:

docker push without logging in

"anonymous" user logged from push

Figured out the issue, we were missing the auth configuration in the registry config for some reason.


Enterprise-specific Values

Hey @benbalter!
So for things like hostname, how would you like this program to fetch those? ENV vars? Hypothetically we can write in enterprise? ? 'ghe.io' : 'github.com' but that doesn't give very much flexibility.
The variables that are custom based on the env are:

hostname
pages_hostname
api_url
environment

How do you want this public code to handle that stuff?

Falling back to Octokit configuration for this kind of thing.