Slower than browscap in many real-world user agents

Hi,
We've been having some memory issues with browscap-php and found your library as an alternative.  The memory use is certainly a lot lower, especially since we use the Full browscap file, so many thanks for that. Most likely we'll move to crossjoin instead of browscap-php for this reason alone, however, it appears that crossjoin can be significantly slower on some real-world user agents, and I just wanted to mention these, along with the benchmarks we've run to see if there is more optimization that can be done.
We've been testing using this project: https://github.com/kenjis/user-agent-parser-benchmarks
I took the top 200 user-agents from our system over the past 24 hours and ran them through that benchmark (removing get_browser (too slow), ua_parser and woothee (not browscap compatible)).  Since we use the Full browscap file, I modified the benchmark scripts to load this file instead, but I was seeing similar results with the default file as well.
browscap-php completes the run in ~24 seconds, using 35 megs of memory.
crossjoin completes the run in ~40 seconds, using 2 megs of memory.
I modified the benchmark utility a bit to time each user-agent individually and then stored the data in a CSV so that I could see which user-agents crossjoin is slower on.  The worst performer (based on the difference between crossjoin's time and browscap-php's time) was this one:

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36

browscap-php was able to do this user-agent in 186ms, while crossjoin takes 855ms.
I also ran the crossjoin run in blackfire, which does show where most of the time is spent, but I wouldn't say it's all that helpful otherwise:
https://blackfire.io/profiles/bc08c0c5-b935-4f2d-bcfd-b47af339d424/graph
Here's a CSV of all of the 200 (196 actually) user-agents with the browscap time, crossjoin time and difference between them.
https://s3.amazonaws.com/uploads.hipchat.com/109285/807583/VHZKpTok5QwIttT/crossjoin_browscap_comparison.csv
Again, thanks for this implementation, we do a lot of caching prior to hitting browscap/crossjoin, so I don't think the extra time here will be that big of a deal, but if these can be optimized, I'm sure it would benefit the library overall.
If there's anything else I can provide, just ask.

Thanks for the info. I'll check it.

So I've spent a fair amount of time looking into this, and have done a lot of benchmarking and inspection using a user-agent that took crossjoin quite a bit of time to process:

Mozilla/5.0 (Linux; Android 4.0.4; Huawei-H867G Build/HuaweiH867G) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19

In my benchmark, this took ~1.2 seconds to lookup.
As mentioned in the commit above, I found a small optimization that saved a couple hundred milliseconds for this particular user-agent (and ~10 seconds or so on the run of our top 200).
Most of the time is still spent in preg_match.  I was curious as to why that is, so put some timings around the two preg_match calls inside the getBrowser method.

A few expressions hit 1 or 2 ms, but a majority of them are at that .03 time (rounded).  The entire run takes ~600 ms.
I was curious if removing that grouped preg_match in crossjoin, and just letting it run against every pattern it found would help at all (assuming the grouped ones are too slow), but that ended up taking longer.  The reason for this is that it has to look at ~65,000 patterns.
I may try looking at the grouping logic some, but I suspect user-agents that start with "Mozilla5.0" as this one does have a lot of potential patterns to match against.  I'm not clear yet why browscap-php decides to only look at 2800 patterns, but crossjoin 65,000 (ungrouped).

Please take a look at the commit above and let me know if this is something you'd be interested in merging in. If so, I can make a PR.
I may try to put together some tests, mainly something that processes useragents in both browscap-php and in crossjoin and compares the output.
For now though, this library is incredibly fast, and we'll be testing it against some live traffic.
Here's a run from that benchmark utility with all 4 parsers enabled:

Running Benchmark using agents in ua-list-100-sample01.txt
Benchmarking browscap-php ...	17.020838secs 	35.25MB
Benchmarking crossjoin-browscap ...	1.5584890secs 	1.5MB
Benchmarking ua-parser ...	1.0967519secs 	2MB
Benchmarking woothee ...	0.2663431secs 	1MB

Giving ua-parser and woothee a run for their money is not an easy task on a Full browscap file :)
I should note that this change will require rebuilding the cache files, but I think if the library's version is bumped that automatically happens no?

Thanks for your work and the helpful information.
I did similar optimizations and could optimize the performance by about 37%. I could eliminate the preg_match calls on the second level in getBrowser and also moved the quoting from parsing to generating the patterns.
The replacement of digits would help a lot, because most performance problems are created by the rising number of patterns, caused by Chrome and Firefox and their countless version numbers.
I will try to merge your changes with my current work.
Independent from this I have another idea to solve the aforementioned problem, but this is something for version 2 and needs some deeper changes.

FYI, I forked and modified the browscap/browscap project to make it run the userAgent tests using my optimized crossjoin branch.
Good news, tests pass 100%.
Here are run times.
browscap-php:

Time: 11.83 minutes, Memory: 706.25Mb
OK, but incomplete, skipped, or risky tests!
Tests: 8499, Assertions: 177184, Skipped: 2413.

crossjoin 1.0:

Time: 14.52 minutes, Memory: 156.75Mb
OK, but incomplete, skipped, or risky tests!
Tests: 8499, Assertions: 177184, Skipped: 2413.

optimized crossjoin:

Time: 2.3 minutes, Memory: 156.75Mb
OK, but incomplete, skipped, or risky tests!
Tests: 8499, Assertions: 177184, Skipped: 2413.

I'm working on getting the recent updates to dev-master to pass all tests and will report back when/if that happens.

So I was able to get the tests to 100% pass, but only with some modifications to the parser.
First, the test run results:

Time: 2.39 minutes, Memory: 163.00Mb
OK, but incomplete, skipped, or risky tests!
Tests: 8499, Assertions: 177184, Skipped: 2413.

Now, I ran into two issues with the changes in the master branch.

The digit replacement you're doing is a bit too optimized :)  Meaning, because you're reducing from something like this (\d)(\d)(\d)\.(\d) to this [\d]+\.[\d]+ it actually causes any .* before or after those patterns to cause issues because of their greediness.  For example, one user agent that I specifically looked at:

UCWEB/2.0(BlackBerry; U; <IP_ADDRESS>; en-us; 9700/<IP_ADDRESS>) U2/1.0.0 UCBrowser/<IP_ADDRESS> U2/1.0.0 Mobile

The pattern it tried to match against:

UCWEB/.(BlackBerry.(\d)+/.) U(\d)+/. UCBrowser/(\d)+.(\d)+.*

And the $matches array that contains the captured digits from that:
Array
(
    [0] => UCWEB/2.0(BlackBerry; U; <IP_ADDRESS>; en-us; 9700/<IP_ADDRESS>) U2/1.0.0 UCBrowser/<IP_ADDRESS> U2/1.0.0 Mobile
    [1] => 0
    [2] => 2
    [3] => 8
    [4] => 1
)

Which when stuck back into the pattern, results in this pattern:

UCWEB/.(BlackBerry.0/.) U2/. UCBrowser/8.1.*

Which ends up not matching, so it moves on to the next possible pattern (which is the incorrect one).
Using the digit replacement I used in my optimize branch:
The pattern it matches against:

UCWEB/.(BlackBerry.(\d)(\d)(\d)(\d)/.) U(\d)/. UCBrowser/(\d).(\d).*

The $matches array:
Array
(
    [0] => UCWEB/2.0(BlackBerry; U; <IP_ADDRESS>; en-us; 9700/<IP_ADDRESS>) U2/1.0.0 UCBrowser/<IP_ADDRESS> U2/1.0.0 Mobile
    [1] => 9
    [2] => 7
    [3] => 0
    [4] => 0
    [5] => 2
    [6] => 8
    [7] => 1
)

The pattern with digits replaced:

SUBBED PATTERN: UCWEB/.(BlackBerry.9700/.) U2/. UCBrowser/8.1.*

Which matches, and is the correct one.

By extending the $sub_key length to 3 characters for the iniparts, it actually broke the Small file test suite, which I haven't fully investigated.  Switching it back to 2 character length fixes it.  The error encountered:

fopen(browscap/tests/BrowscapTest/../../build/browscap-ua-test-1440790661/smallbrowscap_v1440790661_1.0.2/smallbrowscap.iniparts.767): failed to open stream: No such file or directory

This doesn't happen for all user-agents in the suite, just 15 but it is odd.  I haven't looked much further than that.
I'll submit a PR that has the changes necessary to make tests pass. Feel free to mess with that digit replacement to make that work, as I do believe it'll help reduce the number of patterns further if we can make it work properly.
I'll also commit my fork/branch of the test lib (or perhaps @mimmi20 's works still).

And here's my fork for the tests: https://github.com/jaydiablo/browscap/tree/crossjoin

So results of latest dev-master:
browscap userAgent tests:

Time: 2.66 minutes, Memory: 156.75Mb
OK, but incomplete, skipped, or risky tests!
Tests: 8499, Assertions: 177184, Skipped: 2413.

Test suites from the benchmark project:

Running Benchmark using agents in top-200-last-24.txt
Benchmarking browscap-php ...	24.512679secs 	35.25MB
Benchmarking crossjoin-browscap ...	2.2018690secs 	1.75MB

Running Benchmark using agents in top-1000-last-7days.txt
Benchmarking browscap-php ...	124.50185secs 	35.75MB
Benchmarking crossjoin-browscap ...	11.111421secs 	4.5MB

Running Benchmark using agents in ua-list-all.txt
Benchmarking browscap-php ...	71.135257secs 	35.25MB
Benchmarking crossjoin-browscap ...	3.5164999secs 	2.75MB

Running Benchmark using agents in ua-list-100-sample01.txt
Benchmarking browscap-php ...	14.723027secs 	35.25MB
Benchmarking crossjoin-browscap ...	0.6433799secs 	1.25MB

Just like before, the only "diffs" that show up are in the "ua-list-all.txt" file:

Sleipnir/3.3 CFNetwork/711.1.12 Darwin/14.0.0
browscap-php: ["Darwin","CFNetwork",""]
crossjoin-browscap: ["","Sleipnir","2.0"]
Sleipnir/4.5.1 CFNetwork/673.3 Darwin/13.4.0 (x86_64) (iMac14%2C1)
browscap-php: ["Darwin","CFNetwork",""]
crossjoin-browscap: ["","Sleipnir","2.0"]

The "problem" agent from the original post (Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36):

Benchmarking browscap-php ...	0.1350529secs 	35MB
Benchmarking crossjoin-browscap ...	0.0133697secs 	0.5MB

Profile of that parse:
https://blackfire.io/profiles/acfd2421-b1c0-4a25-8537-22e21ac6abf6/graph
I also re-built the CSV that I attached to the original post that had per-agent timings for browscap and crossjoin.  Here's how that looks:
https://s3.amazonaws.com/uploads.hipchat.com/109285/807583/rf3tgwVVDz9b5xl/crossjoin_browscap_comparison_2.csv
I think it's safe to say that this issue is resolved. Thanks for taking the time to implement these optimizations.

I checked the patterns for the user agents "Sleipnir/3.3 CFNetwork/711.1.12 Darwin/14.0.0" and "Sleipnir/4.5.1 CFNetwork/673.3 Darwin/13.4.0 (x86_64) (iMac14%2C1)" manually in the browscap data file and I'd say, that the browscap-php results are wrong, so I think it's okay to ignore these two cases.
I also checked the results with PHP 7RC1 - it's about 2-3 times faster.
I'll update the readme and create a new release. Thanks a lot for your help with that.

No joke about php7:
https://travis-ci.org/jaydiablo/browscap


suggestions

Information

Apktool Version (apktool -version) -2.4.2-dirty
Operating System (Mac, Linux, Window) -Window

I have 2 suggestions:

When the number of dex file methods exceeds 65535, apktool will fail to compile.
So, when apktool recompiles, can it automatically detect the number of methods, exceeding 65535, split the smali folder into 2 subfolders, and then compile again?

apktool will automatically decompile all dex files in the apk, including dex files in other folders such as assets.
Can apktool add a command line parameter without decompiling the dex files in other folders such as assets?

There's a parameter for only decompiling the dex files in the root of the APK in 2.4.1. Use "--only-main-classes".

There's a parameter for only decompiling the dex files in the root of the APK in 2.4.1. Use "--only-main-classes".

Thank you, you are right.

This would require some work rewriting files to new packages which is pretty out of scope of Apktool main purpose. It might be able to be done directly via dexlib with some work.

As mentioned above, this is a feature in the new --only-main-classes parameter


[WIP] Update module imports, use class instead of functions

Import class by default, so, to use this package you need to create an object,

import UploadcareUpload from '@uploadcare/upload-api'

const upload = new UploadcareUpload(settings)

All function will be methods of class
Low-level functions will be in UploadcareUpload.api scope,

upload.api.fromUrl(...)

High-level functions will be in UploadcareUpload scope

upload.fileFrom(...)

To update settings, you can use setSettings method that fired event

upload.addUpdateSettingsListener((prevSettings) => {
  console.log(upload.settings)
  console.log(prevSettings)
})

upload.setSettings({publicKey: 'newkey'})

What do you think?
P.S. Also, move specs to test folder, its more clear

@bautrukevich bump


Add pre-commit hook with checkstyle goal

https://github.com/oslomarketsolutions/pre-commit-maven-plugin


Create seed file in DB

What do we want to seed? Products first and then users and then orders or all three in one file?


Feat/add entry point postonboarding

📝 Description
This PR add entry point for post onboarding
❓ Context

Impacted projects: ledger-live-desktop
Linked resource(s): FAT-308

✅ Checklist

[ ] Test coverage
[X] Atomic delivery
[X] No breaking changes

📸 Demo

🚀 Expectations to reach
Please make sure you follow these Important Steps.
Pull Requests must pass the CI and be internally validated in order to be merged.

Good to merge in FAT-99


DLL file origin?

Where can I find the original source of these files?
Mono.Data.Sqlite.dll
Mono.Data.SqliteClient.dll

Nevermind I figured it out. Mono.Data.SqliteClient is a deprecated assembly

Ah thanks for letting me know!


Allow opt-out of alarms

see #48

So you can turn off an alarm temporarily with Enabled: false.
👍


spelling error

It should be "Air Balloon". I want to fix this issue.
Please assign this issue to me under gssoc-extd and hacktoberfest.

Hi! I am eager to contribute anyhow. Please assign this issue to me.

@iamrahulmahato Hey, I have done this task. Please merge it.


M-RET r should call normal-mode on all buffers

This is because the mode hooks could have been updated, or the things they changed might be overridden by other code in the init-file running again.
It probably wouldn't be too hard to make it so that normal-mode was only run in buffers once the user switched to them, which would improve performance of M-RET r.

On second thought this is probably too much magic. The user should be aware of the pitfalls of reloading the init-file in place rather than restarting, after major changes.


.NET 7 not supported

You must install or update .NET to run this application.
App: /tools/resharper-to-codeclimate
Architecture: x64
Framework: 'Microsoft.NETCore.App', version '6.0.0' (x64)
.NET location: /usr/share/dotnet
The following frameworks were found:
  7.0.0 at [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Hi, I've added .net7 as a target to the latest version

Perfect! Thanks!


new sonatype scanner

Release 174
Powered by Sonatype IQ Server

TNC_PrimeRead-build-f2c3b683f67343329c446f2c3847e447.spdx.json

@johnfelipe FWIW, DefectDojo supports CycloneDX which I see as a supported output.
https://documentation.defectdojo.com/integrations/parsers/file/cyclonedx/
So, while we could add support for SPDX, it seems somewhat unnecessary.

Can we close this issue then, @mtesauro ?


Bear Hunter Ninja | Game

PR Description 📜
In this,the user chooses one from the three options available and the computer does the same simultaneously. And thus, making the combinations such that either the user wins, computer wins or its a draw or tie.
In this game, players use the gestures of a bear, a hunter, and a ninja. The rules are as follows:
Bear beats hunter (bear mauls hunter).
Hunter beats ninja (hunter shoots ninja).
Ninja beats bear (ninja kills bear).
I would like to contribute under gssoc'24 :)
Kindly assign labels
Fixes #3097

Mark the task you have completed ✅

[x] I follow CONTRIBUTING GUIDELINE & CODE OF CONDUCT of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I have followed proper naming convention showed in CONTRIBUTING GUIDELINE
[x] I have added screenshot for website preview in assets/images
[x] I have added entries for my game in main README.md
[x] I have added README.md in my folder
[ ] I have added working video of the game in README.md (optional)
[x] I have specified the respective issue number for which I have requested the new game.

Add your screenshots(Optional) 📸

Thank you soo much for contributing to our repository 💗

Thank you @Niyatizzz ,for creating the PR and contributing to our GameZone 💗
Review team will review the PR and will reach out to you soon! 😇
Make sure that you have marked all the tasks that you are done with ✅.
Thank you for your patience! 😀

Thank you  @Niyatizzz , for your valuable time and contribution in our GameZone 💗.
It’s our GameZone, so Let’s build this GameZone altogether !!🤝
Hoping to see you soon with another PR again 😇
Wishing you all the best for your journey into Open Source🚀


Update README

Hi. For some reason the link was inactive, maybe it will be better this way

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

The link works for me when rendered by Github. How did you view it?


more: move from getopts to clap

Code is in:
https://github.com/uutils/coreutils/blob/master/src/uu/more/src/more.rs
Tests:
https://github.com/uutils/coreutils/blob/master/tests/by-util/test_more.rs
As example:
https://github.com/uutils/coreutils/commit/9081e120d6b866c94679d9e29c7bdb17ccccb4d0

I'm working on this.


Fix pub warnings for tool package before publishing

Draft PR while I fix issues:
Just discovered pub warnings while working on the docs and realizing we hadn't published to pub.dev yet:
Package validation found the following errors:
* You must have a LICENSE file in the root directory.
  An open-source license helps ensure people can legally use your code.
* line 15, column 1 of lib/src/builder/builder.dart: source_gen is in the `dev_dependencies` section of `pubspec.yaml`. Packages used in lib/ must be declared in the `dependencies` section.
     ╷
  15 │ import 'package:source_gen/source_gen.dart';
     │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ╵
* line 16, column 1 of lib/src/builder/builder.dart: build is in the `dev_dependencies` section of `pubspec.yaml`. Packages used in lib/ must be declared in the `dependencies` section.
     ╷
  16 │ import 'package:build/build.dart';
     │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ╵
* line 19, column 1 of lib/src/builder/src/code_generator.dart: build is in the `dev_dependencies` section of `pubspec.yaml`. Packages used in lib/ must be declared in the `dependencies` section.
     ╷
  19 │ import 'package:build/build.dart';
     │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ╵
* line 21, column 1 of lib/src/builder/src/code_generator.dart: glob is in the `dev_dependencies` section of `pubspec.yaml`. Packages used in lib/ must be declared in the `dependencies` section.
     ╷
  21 │ import 'package:glob/glob.dart';
     │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ╵
* line 22, column 1 of lib/src/builder/src/code_generator.dart: source_gen is in the `dev_dependencies` section of `pubspec.yaml`. Packages used in lib/ must be declared in the `dependencies` section.
     ╷
  22 │ import 'package:source_gen/source_gen.dart';
     │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ╵
Package validation found the following potential issue:
* ./CHANGELOG.md doesn't mention current version (0.3.0-dev).
  Consider updating it with notes on this version prior to publication.
Sorry, your package is missing some requirements and can't be published yet.
For more information, see: https://dart.dev/tools/pub/cmd/pub-lish.

@kevmoo Note: I can fix those errors by moving the builder code out of lib (e.g., put under tool dir like stagehand does), but I don't want to for now. I'd like to be able to export the ability to build templates as part of the tool API.

Under the tool dir makes sense to me. These are not things that are needed at runtime on the users machine.

The package isn't meant to just be a CLI -- it's meant to support other tools that can leverage it. The builder can't be exported if not under lib.

Right...
Hrm...
Well, since it's globally installed, having these extra dependencies won't
block others either...unless they are building similar tools.
On Wed, Dec 23, 2020 at 1:26 PM Tony Pujals<EMAIL_ADDRESS>wrote:

The package isn't meant to just be a CLI -- it's meant to support other
tools that can leverage it. The builder can't be exported if not under lib.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/functions-framework-dart/pull/146#issuecomment-750480207,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAEFCV4JELKG5T7SIG76KDSWJN7PANCNFSM4VHN4GBA
.

It could be refactored out into it's own package, but that would take time. I figured this was the least amount of work to get this out today.

It's fine, I can move it -- I just have to go through the cycle of rebuilding and getting presubmit checks to pass, etc.

At some point, various tool package consumers (a CLI, Cloud Code, etc.) will be in their separate packages. I'll go ahead and refactor for now if that's your preference. Just going to take a bit longer than I wanted while working on docs today. Running against the clock now!  :D

@kevmoo PTAL

@kevmoo PTAL


[Bug/Question] pip-compile does not include package

Environment Versions

OS Type: macOS
Python version: 3.11.1
pip version: 22.3.1
pip-tools version: 6.14.0

Steps to replicate
With project format:
.
├── dev-requirements.txt
├── pyproject.toml
├── requirements.txt
└── src
    └── my_package
        ├── __init__.py
        └── foo.py

and a pyproject.toml of:
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "my_package"
version = "0.1"
requires-python = ">=3.11"
dependencies = ["holidays"]

[tool.hatch.build.targets.wheel]
packages = ["src/my_package"]

[project.optional-dependencies]
dev = ["black", "mypy", "ruff", "ipykernel", "pyupgrade"]

Expected result
Similar to what happends if we use hatch natively.
$ hatch env create
$ hatch env run python
>>> import my_package
>>>

I expected my local package to be installed, ideally in editable (-e) mode.
Actual result
Running pip-compile does not seem to install the local package, my_package.
$ pip-compile -o requirements.txt pyproject.toml
$ pip-sync requirements.txt dev-requirement.txt
$ python
>>> import my_package
ModuleNotFoundError: No module named 'my_package'

I'm very new to pip-tools, so I honestly assume I'm missing a key step here, is there a suggested way or option to install the local package here?

Thanks for this! There's definitely been surprise about this kind of thing in other issues, so discussion is warranted.
pip-compile pins dependencies (only), and installs nothing. As you mentioned, the top-level project is more practically installed in editable mode, rather than being pinned.
So to me the thing to do, if you want pip-sync to install the top package, is to run it against something higher-level than the pinned output from pip-compile. For example:
local-requirements.txt:
-e .[dev]
-c dev-requirements.txt

$ pip-compile --extra dev -o dev-requirements.txt pyproject.toml
$ pip-sync local-requirements.txt

Thanks a lot for the help here! So my takeaways are:

pip-compile's scope is to pin dependecies only
A way to have pip-sync install the local package is to somehow to manually include the local package in a requirements-type file that pip-compile doesn't touch—like your example

Again, thanks a lot. Really clear answer and good solution. Appreciate the work on the tool!


Analytics Page, we have duplications on one of the charts.

Here is the chart that has duplications.

surely, I have been able to see that, too. I think what is causing this is that

little spelling errors,
Having the same name but with little extra wording,
Capitalization of the words

For someone to be able to fix this, we need to categorize the data based off the similarity and also be able to create unique values


Update to 4.3.9 or 4.3.10 will fail

Hello,
The update to 4.3.9+ will fail with:
E0413 16:21:53.310311       1 sync_worker.go:329] unable to synchronize image (waiting 2m52.525702462s): Precondition "ClusterVersionUpgradeable" failed because of "DefaultSecurityContextConstraints_Mutated": Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [hostmount-anyuid]
as also discussed here: https://bugzilla.redhat.com/show_bug.cgi?id=1818893
The reason is the custom user for hostmount-anyuid: system:serviceaccount:openshift-nfs-provisioner:nfs-client-provisioner
The solution seems to be to clone the default SCC and put the changes there.

I have been running 4.4 from rc4 and updates have been working ok. Have update to rc-5 and rc-6 and rc-7. If you feel lucky try updating directly to 4.4-rc7 quay.io/openshift-release-dev/ocp-release@sha256:df3b7a74c8590a932c00fd9b1ef6c1fb2a0bfd1c3643b78ae378cadee3258c03

hmm, according to the bugzilla 4.4 will stomp any changes in the default SCCs. So maybe the nfs-provisioner is not added in your config anymore, or you have already a custom/cloned
SCC

Running 4.4rc8 now it is working file.
[root@host01 ~]# oc describe scc hostmount-anyuid
Name:						hostmount-anyuid
Priority:					<none>
Access:
  Users:					system:serviceaccount:openshift-infra:pv-recycler-controller,system:serviceaccount:openshift-nfs-provisioner:nfs-client-provisioner
  Groups:					<none>
Settings:
...

I have this:
`users:

system:serviceaccount:openshift-nfs-provisioner:nfs-client-provisioner`

The user was added during the ansible deployment and above error complains about changes in the default SCC.
When I cloned it (as hostmount-anyuid-custom) and removed it from the default one, it worked.
btw: I installed the cluster with storage_nfs: true
So this task will one add the user to the SCC when storage_nfs is set to true.

😳 - let's wait of BZ 1818893 to check out what is the right solution.
I don't understand the work-a-round in Comment 8:

A workaround to "adding an account to any SCC" is to use RBAC to give a user access to the default SCC. This way the customer can avoid changing the default SCC.

It's good to check the SCC bevor upgrade, but not the members. I don't like to create a copy of hostmount-anyuid  scc.

Solution:

Remove scc member:

oc adm policy remove-scc-from-user hostmount-anyuid system:serviceaccount:openshift-nfs-provisioner:nfs-client-provisioner

Add RBAC to allow nfs-provision to run with hostmount-anyuid:

oc create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: scc-hostmount-anyuid
  namespace: "openshift-nfs-provisioner"
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - hostmount-anyuid
  resources:
  - securitycontextconstraints
  verbs:
  - use
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sa-to-scc-hostmount-anyuid
  namespace: "openshift-nfs-provisioner"
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
roleRef:
  kind: Role
  name: scc-hostmount-anyuid
  apiGroup: rbac.authorization.k8s.io
EOF

PR for new installation will follow or highly appreciated :-)

@rbo Should this fix work as is?

@tfriman  Yes, tested at my cluster

Ok, it doesn't work on mine (4.3.8 and .9/.10/.12/.13 all fail) but maybe I'll fix mine with reinstall anyways :-)

Please check the log of CVO pod. The fix solve only the nfs provisioner. If you install openshift pipeline (for example) then opeshift pipeline is the problem for failing cluster update.

Ah, I have installed plenty of operators (also pipelines).
Cluster version operator log says:
````E0420` 15:29:46.764643       1 precondition.go:49] Precondition "ClusterVersionUpgradeable" failed: Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]`````

What I do not get, but maybe thats more for the bug to track, when you install for example the pipeline operator, the SCC will be modifed. With 4.4 they say any change will be stomped

Ah, I have installed plenty of operators (also pipelines).
Cluster version operator log says:
E0420 15:29:46.764643 1 precondition.go:49] Precondition "ClusterVersionUpgradeable" failed: Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [privileged]

Please compare your privileged scc with the default of 4.3:
allowHostDirVolumePlugin: true
allowHostIPC: true
allowHostNetwork: true
allowHostPID: true
allowHostPorts: true
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowedCapabilities:
- "*"
allowedUnsafeSysctls:
- "*"
apiVersion: security.openshift.io/v1
defaultAddCapabilities:
fsGroup:
  type: RunAsAny
groups:
- system:cluster-admins
- system:nodes
- system:masters
kind: SecurityContextConstraints
metadata:
  annotations:
    kubernetes.io/description: 'privileged allows access to all privileged and host
      features and the ability to run as any user, any group, any fsGroup, and with
      any SELinux context.  WARNING: this is the most relaxed SCC and should be used
      only for cluster administration. Grant with caution.'
  name: privileged
priority:
readOnlyRootFilesystem: false
requiredDropCapabilities:
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
seccompProfiles:
- "*"
supplementalGroups:
  type: RunAsAny
users:
- system:admin
- system:serviceaccount:openshift-infra:build-controller
volumes:
- "*"

Source: https://bugzilla.redhat.com/show_bug.cgi?id=1821905

What I do not get, but maybe thats more for the bug to track, when you install for example the pipeline operator, the SCC will be modifed. With 4.4 they say any change will be stomped.
I fear the rbac is now the correct solution, but I am also with @rbo ... why checking the members of an object.

Let's see whats going next days, there some internal discussing around that. For hetzner-ocp4 and nfs-prov. the problem is solved with merging the PR #98 .
I would prefer to close the issue...

works for me , thank you!

Official Solution Article from Red Hat: Upgrading OpenShift 4.3.8 - 4.3.12 fails with error "it may not be safe to apply this update"


Build types and add them to the bundle

Isn't this already there? I don't know if I get it.

Ahh, I somehow saw a type error when I ran lerna run validate yesterday.


Example: dynamic_values and dynamic_output_new

This takes the example from https://github.com/facebook/buck2/pull/619 rebased on the newly merged dynamic_values support for dynamic_output_new.

Example using dynamic values
fix starlark linter warnings
Test dynamic_value example on CI
Remove the prelude symlink
Update the dynamic value example to the new API

Is it possible to modify this example to emulate some real-word scenarios with recognisable patterns and some stubs? So for people not aware what dynamic values are usable for, could understand what the problem from the solution.
In the internal version of the repo, we have an example, in a form of e2e test. We plan to publish e2e tests after we clean them up from references to Meta internals). If this example is meant just as a test, we already have it.
@JakobDegen do you know, what is the status of publishing test suite?

In the internal version of the repo, we have an example, in a form of e2e test. We plan to publish e2e tests after we clean them up from references to Meta internals). If this example is meant just as a test, we already have it.

I see, yes, this was mainly meant as a simple e2e test. It was also a way for me to validate the new API. So, if you have a test already that you're planning to release than this PR can be closed.

Is it possible to modify this example to emulate some real-word scenarios with recognisable patterns and some stubs? So for people not aware what dynamic values are usable for, could understand what the problem from the use case.

Potentially, yes, though I'm not sure I'll find time for that in the very near future. Our next focus will be to update our Haskell rules to the new API. The example here is inspired by that use-case - it's just stripping away a lot of Haskell specifics and taking a short cut in that the module dependency graph can be read directly from the sources rather than having to invoke a tool first to extract it. So, maybe once we've upgraded the Haskell rules it will be easy to extract an instructive example. We also still want to propose these Haskell rules for merge into the upstream Buck2 prelude. So, if and when they're merged, they could serve as a practical example directly, albeit a much more involved one.

I have exported buck2 e2e tests. For dynamic values, this is it:
https://github.com/facebook/buck2/blob/0877eeac4b423c28dbad0c91be2f7ce104137d8d/tests/e2e/build/actions/test_dynamic_value_data/prelude.bzl
Note tests for now are not executable as tests (test repos should work, but accompanying python tests launching these tests are yet to be enabled on GitHub CI).


Azure Blob - "BlobItem.getMetadata()" returns null but "BlobClient.getProperties().getMetadata()" returns correct value

We are using BlobContainerClient.listBlobsByHierarchy(..)  to get list of blobs while iterating we realised that BlobItem.getMetadata() returns null but BlobClient.getProperties().getMetadata() returns correct value
Prerequisite
Create a blob which should have metadata
Code Snippet To Reproduce
 public void getBlobMetadata() throws Exception {
        String containerName = "**-test";
        String directoryName = "**-test";

        BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().endpoint(BLOB_SAS).buildClient();
        BlobContainerClient blobContainerClient = blobServiceClient.getBlobContainerClient(containerName);

        ListBlobsOptions listBlobOption = new ListBlobsOptions();
        listBlobOption.setMaxResultsPerPage(10);
        listBlobOption.setPrefix(directoryName);

        PagedIterable<BlobItem> pagedIterable = blobContainerClient.listBlobsByHierarchy(null, listBlobOption, null);
        Iterable<PagedResponse<BlobItem>> iterableByPage = pagedIterable.iterableByPage(10);

        Iterator<PagedResponse<BlobItem>> pagedResponseiterator = iterableByPage.iterator();
        while (pagedResponseiterator.hasNext()) {
            PagedResponse<BlobItem> pagedResponse = pagedResponseiterator.next();
            Iterator<BlobItem> pagedBlobItems = pagedResponse.getElements().iterator();
            while (pagedBlobItems.hasNext()) {
                BlobItem blobItem = pagedBlobItems.next();
                LOGGER.info("BlobItem - BlobName '{}', Metadata '{}'", blobItem.getName(), blobItem.getMetadata());

                BlobClient blobClient = blobContainerClient.getBlobClient(blobItem.getName());
                LOGGER.info("BlobClient - BlobName '{}', Metadata '{}'", blobItem.getName(), blobClient.getProperties().getMetadata());
            }
        }
    }

Expected behavior
BlobItem.getMetadata()  should return available metadatas
Screenshots

Setup (please complete the following information if applicable):
OS: window
IDE : eclipse
Version of the Library used: azure-storage-blob-12.10.2

I have one more query on top of above questions. Please find below details -
BlobClient.getProperties() returns com.azure.storage.blob.models.BlobProperties, but
blobItem.getProperties() returns   'com.azure.storage.blob.models.BlobItemProperties'
Both classes (BlobProperties, BlobItemProperties) have approximate same parameters but there are some inconsistent parameters, such as - BlobProperties have expiresOn property but BlobItemProperties  does not have expiresOn
Could you please suggest which one is correct Properties and we should be using which class going forward?
Please let me know for any queries.

Hi @abhikt48, in the code sample you provided it appears that setDetails isn't being configured to retrieve metadata. Without this configuration the listed BlobItem won't return metadata: https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs#response-body

Snapshots, blob metadata, and uncommitted blobs are included in the response only if they are specified with the include parameter on the request URI.

Hi @alzimmermsft, Thanks for quick update. I tested with your suggestion and able to retrieve metadata, but failed to retrieve tags. It is throwing This request is not authorized to perform this operation using this permission.
I checked permission of SAS key as well and I have provided all permissions. Please refer below screenshot.
I am not able to get tags with BlobClient as well.
Could you please suggest your opinion on below two issues -

How can I get tags with BlobItem and BlobClient ? Please refer below sample code and errrorStacktrace.
What is the recommendation from your side to access Blob properties while listing blobs? Should we use BlobItem orBlobClient` ? Please suggest.

Source Code
public void getBlobMetadata() throws Exception {
        String containerName = "***-test";
        String directoryName = "**test";

        BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().endpoint(BLOB_SAS_SERVICE).buildClient();
        BlobContainerClient blobContainerClient = blobServiceClient.getBlobContainerClient(containerName);

        ListBlobsOptions listBlobOption = new ListBlobsOptions();
        listBlobOption.setMaxResultsPerPage(10);
        listBlobOption.setPrefix(directoryName);

        BlobListDetails blobListDetails = new BlobListDetails();
        blobListDetails.setRetrieveMetadata(true);
        blobListDetails.setRetrieveTags(true);
        blobListDetails.setRetrieveVersions(true);

        listBlobOption.setDetails(blobListDetails);

        PagedIterable<BlobItem> pagedIterable = blobContainerClient.listBlobsByHierarchy(null, listBlobOption, null);
        Iterable<PagedResponse<BlobItem>> iterableByPage = pagedIterable.iterableByPage(10);

        Iterator<PagedResponse<BlobItem>> pagedResponseiterator = iterableByPage.iterator();
        while (pagedResponseiterator.hasNext()) {
            PagedResponse<BlobItem> pagedResponse = pagedResponseiterator.next();
            Iterator<BlobItem> pagedBlobItems = pagedResponse.getElements().iterator();
            while (pagedBlobItems.hasNext()) {
                BlobItem blobItem = pagedBlobItems.next();
                LOGGER.info("BlobItem - BlobName '{}', Metadata '{}', tags '{}', versionID '{}'", blobItem.getName(), blobItem.getMetadata(), blobItem.getTags(), blobItem.getVersionId());

                BlobClient blobClient = blobContainerClient.getBlobClient(blobItem.getName());
                BlobProperties blobProperties = blobClient.getProperties();
                LOGGER.info("BlobClient - BlobName '{}', Metadata '{}', tags '{}', versionID '{}'", blobItem.getName(), blobProperties.getMetadata(), blobItem.getTags(), blobProperties.getVersionId());
            }
        }
    }

Permission

ERROR
Exception in thread "main" com.azure.storage.blob.models.BlobStorageException: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.
If you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.
Please remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.
Status code 403, "<?xml version="1.0" encoding="utf-8"?><Error>AuthorizationPermissionMismatch<Message>This request is not authorized to perform this operation using this permission.
RequestId:***
Time:2021-04-23T15:15:03.7787690Z</Message></Error>"
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.azure.core.http.rest.RestProxy.instantiateUnexpectedException(RestProxy.java:337)
	at com.azure.core.http.rest.RestProxy.lambda$ensureExpectedStatus$5(RestProxy.java:378)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:320)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onNext(MonoCacheTime.java:337)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onSubscribe(MonoCacheTime.java:276)
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:191)
	at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:132)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:103)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
	at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144)
	at reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionMainSubscriber.onNext(FluxDelaySubscription.java:180)
	at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
	at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:173)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
	at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:171)
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:252)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1783)
	at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:152)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:138)
	at reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:206)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:262)
	at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:383)
	at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:373)
	at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:429)
	at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:655)
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1533)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1294)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1331)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
		at reactor.core.publisher.Flux.blockLast(Flux.java:2497)
		at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:94)
		at com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator.<init>(ContinuablePagedByPageIterable.java:49)
		at com.azure.core.util.paging.ContinuablePagedByPageIterable.iterator(ContinuablePagedByPageIterable.java:37)
		at com.blob.test.BlobListByPage.getBlobMetadata(BlobListByPage.java:106)
		at com.blob.test.BlobListByPage.main(BlobListByPage.java:125)

@alzimmermsft  - One more input on top of above comment. There are two problems for selection between BlobProperties and BlobItemProperties. Please find below details

BlobProperties and BlobItemProperties are not consistent. Such as - BlobItemProperties.contentLength and BlobProperties.blobSize
While listing I will get BlobItem in return, so I need to create specifically BlobClient  again which could be performance issue "blobContainerClient.getBlobClient(blobItem.getName());"

Please suggest your recommendation, so i can implement same.

Hi @abhikt48
For your permissions issue, could you share the sas token you are using (you can redact any blob related information and the signature part of it)?
I just ran your repro code with a SAS token from portal just as you showed and I was able to get the tags information.

HI @gapra-msft
Please find masked SAS - https://***.blob.core.windows.net/?sv=2019-12-12&ss=b&srt=sco&sp=rw**cx&se=2021-10-31T21:53:13Z&st=2020-11-19T13:53:13Z&spr=https&sig=8xyC***9Ix**
Please let me know for any other inputs.

@abhikt48  Can you un-redact the sp query parameter?

For reference, here is the sas token I got back from portal
https://XXXX.blob.core.windows.net/?sv=2020-02-10&ss=bfqt&srt=sco&**sp=rwdlacuptfx**&se=2021-04-28T01:22:02Z&st=2021-04-23T17:22:02Z&spr=https&sig=XXXXXXX

@gapra-msft  Please find updated one -
https://***.blob.core.windows.net/?sv=2019-12-12&ss=b&srt=sco&sp=rwdlacx&se=2021-10-31T21:53:13Z&st=2020-11-19T13:53:13Z&spr=https&sig=8xyC***9Ix**
Another one - https://***.blob.core.windows.net/?sv=2019-12-12&ss=b&srt=sco&sp=rwdlacupx&se=2021-10-31T21:53:13Z&st=2020-11-19T13:53:13Z&spr=https&sig=8xyC***9Ix**

Hi @abhikt48
I think the problem is Azure Portal has not yet been updated to support sas versions that allow you to specify tag permissions. Can you generate the sas using the SDK
blobServiceClient.generateSas()
The above API should allow you to create a SAS that supports tags permissions.

HI @gapra-msft  - Thanks for providing details. Is it possible for you to raise internal request to generate latest SAS version in Azure Portal ?
Based on your suggestion - AccountName and AccountKey is manadatory for generating SAS.  We are using SAS only in most applications, so we can't enforce user to use AccountName and AccountKey instead of SAS. This will be a huge change for us.
StorageSharedKeyCredential credential = new StorageSharedKeyCredential("accountName", "accountKey");
BlobServiceClientBuilder blobServiceClient = new BlobServiceClientBuilder().credential(credential);
// Client must be authenticated via StorageSharedKeyCredential
String sas = blobServiceClient.generateAccountSas(sasValues);

Could you please suggest your recommendation on below query? We have to take decision based on your suggestion.

Hi @abhikt48
For 1. blobSize and contentLength are the same thing, I believe we chose blobSize for getBlobProperties since it is more idiomatic for users.
2. Could you please clarify what you mean by this question?

Hi @gapra-msft , Thanks for quick reply. Please find below details -
Our developed component using various download and list operations. While

downloading - we are getting BlobClient and BlobClient.getProperties() returns com.azure.storage.blob.models.BlobProperties
Listing - We are getting BlobItem, and blobItem.getProperties() returns   com.azure.storage.blob.models.BlobItemProperties

Eventually both classes (BlobProperties, BlobItemProperties) returns blob properties, but both classes does not have exact property. We want to maintain consistent over all operations. So we would like to have same properties over both operations.
While listing I get BlobItem, I can new BlobClient with blob name which could be a performance issue.
Could you please suggest your recommendation - Which class we should to use for accessing Blob properties?
Please find below property differences -

BlobPropertieshave expiresOn property but BlobItemProperties  does not have expiresOn

BlobPropertieshave copyDestinationSnapshot property but BlobItemProperties  does not have copyDestinationSnapshot

BlobPropertieshave encryptionKeySha256 property but BlobItemProperties  does not have encryptionKeySha256

BlobPropertieshave committedBlockCount property but BlobItemProperties  does not have committedBlockCount

BlobPropertieshave isCurrentVersion property but BlobItemProperties  does not have isCurrentVersion

BlobPropertieshave objectReplicationDestinationPolicyId property but BlobItemProperties  does not have objectReplicationDestinationPolicyId

BlobItemPropertieshave destinationSnapshot property but BlobProperties  does not have destinationSnapshot

BlobItemPropertieshave deletedTime property but BlobProperties  does not have deletedTime

BlobItemPropertieshave remainingRetentionDays property but BlobProperties  does not have remainingRetentionDays

BlobItemPropertieshave customerProvidedKeySha256 property but BlobProperties  does not have customerProvidedKeySha256

 BlobItemhave snapshot property but  BlobClient does not have snapshot

Please let me know for any queries.

Hi @abhikt48
Thanks for clarifying. I am going to check with other platforms to see how they expose these variables and will get back to you.

HI @gapra-msft  - Could you please update on this isuue ?

Hi @abhikt48
I spoke with my team. We will take a second look at the properties returned by listBlobs/getProperties and ensure they align with all the service supported properties.

Hi @abhikt48
It looks like there are a few naming convention mismatches here. I'll cover all your concerns one by one below

BlobPropertieshave expiresOn property but BlobItemProperties does not have expiresOn

The listBlobs API does not support expiresOn according to the REST docs.

BlobPropertieshave copyDestinationSnapshot property but BlobItemProperties does not have copyDestinationSnapshot

The listBlobs API does not support copyDestinationSnapshot according to the REST docs.

BlobPropertieshave encryptionKeySha256 property but BlobItemProperties does not have encryptionKeySha256

BlobProperties.encryptionKeySha256=BlobItemProperties.customerProvidedKeySha256. There is just a naming mismatch here.

BlobPropertieshave committedBlockCount property but BlobItemProperties does not have committedBlockCount

The listBlobs API does not support commitedBlockCount according to the REST docs.

BlobPropertieshave isCurrentVersion property but BlobItemProperties does not have isCurrentVersion

BlobProperties.isCurrentVersion = BlobItem.isCurrentVersion. This property is one level up in the hierarchy

BlobPropertieshave objectReplicationDestinationPolicyId property but BlobItemProperties does not have objectReplicationDestinationPolicyId

The listBlobs API does not support objectReplicationDestinationPolicyId according to the REST docs.

BlobItemPropertieshave destinationSnapshot property but BlobProperties does not have destinationSnapshot

This is BlobProperties.copyDestinationSnapshot

BlobItemPropertieshave deletedTime property but BlobProperties does not have deletedTime

The getProperties API does not support deletedTime according to the REST docs.

BlobItemPropertieshave remainingRetentionDays property but BlobProperties does not have remainingRetentionDays

The getProperties API does not support remainingRetentionDays  according to the REST docs.

BlobItemPropertieshave customerProvidedKeySha256 property but BlobProperties does not have customerProvidedKeySha256

BlobProperties.encryptionKeySha256=BlobItemProperties.customerProvidedKeySha256. There is just a naming mismatch here.

BlobItemhave snapshot property but BlobClient does not have snapshot

You can call BlobClient.getSnapshotId to get the snapshot of a BlobClient.
Hope this helps!

@gapra-msft  Thanks for feedback.
Can you please consider feature request to match same name in both objects?
As list api doesn’t return some properties, so we can ignore other properties.
please suggest.

Hi @abhikt48
As a design principle, the SDK errs on the side of following the naming convention of the service, so we would not be able to change the name unless the service decides to resolve the naming issue.
I will go ahead and raise this issue to the service's attention, so they can move forward appropriately.
Generally we don't track service issues on this repo so if I've answered all your questions, please feel free to close the issue.
Thanks for the feedback!

Hi @abhikt48 I've created an issue tracking this (it isn't publicly visible but I noted this issue number so whoever picks this item up can update you)


Question About Missing Variable

Hello. First off, thank you all very much for releasing the source code to your great paper.
I had a small question about where this node_attr variable, referenced below, comes from, as it does not appear to be defined anywhere else in the SEGNN model's code.
https://github.com/RobDHess/Steerable-E3-GNN/blob/eda3680c94e89ff1da546197d48ac12cbc722430/models/segnn/segnn.py#L95
Is this node_attr variable in fact coming from graph.node_attr?

Hi,
That seems to be an old variable name that was not properly updated. Since node attributes in our experiments are defined as the average of the edge attributes, an isolated node would not be assigned any node attributes. We therefore set it to zero. I believe this should be changed to  node_attr_irreps.shape[-1], but I'll double check later today.


Abbreviate huge numbers

Abbreviate huge numbers so that they don't clobber the UI.

balances abbreviated

[ ] ensure tooltip for abbreviated numbers show entire number


hapOut sporadically gets into a fail state

While trying to figure out why hitting weblogs was not always sending the full HTML page, and why sometimes the Home app returns "No response", I ended up adding a bit of debug in HAPClient::getStatusURL() as such:
  hapOut << "\n";
  hapOut.flush();
  // Let's see if there was an error flushing the stream
  if(!hapOut.good()) {
    LOG0("Error flushing stream. (eofbit: %d, failbit: %d, badbit: %d)\n", hapOut.eof(), hapOut.fail(), hapOut.bad());
  }

And the outcome I get, from time to time, is
Error flushing stream. (eofbit: 0, failbit: 1, badbit: 1)

No idea why at the moment, and no TLS code, just standard weblogs.
And there does not seem to be a way to recover from this other than reboot the device.
Hitting Refresh in Chrome gives
<IP_ADDRESS> didn’t send any data.
ERR_EMPTY_RESPONSE

and turning L2 logs, I get this:
=======================================
** Client #2 Connected (773 sec): <IP_ADDRESS>

<<<<<<<<< <IP_ADDRESS> <<<<<<<<<
GET /status HTTP/1.1
Host: <IP_ADDRESS>
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9,fr;q=0.8
------------ END BODY! ------------

>>>>>>>>>> <IP_ADDRESS> >>>>>>>>>>
Error flushing stream. (eofbit: 0, failbit: 1, badbit: 1)
------------ SENT! --------------
=======================================
** Client #2 Connected (773 sec): <IP_ADDRESS>

** Client #2 DISCONNECTED (773 sec)

hapOut is global right? Wouldn't it need to be protected against concurrent access, like when a controller requests an update from HomeSpan, and when a user hits the weblog?
Wouldn't you have two distinct requests flushing and/or calling setHapClient() ?

Hmmm.  I've not encountered this.
Note there is no concurrent access (this is why things like the AsyncWebServer does not play well with HomeSpan).   Calls to the web log are treated the same as calls from a Controller.  Each is processed to completion before the next request is handled.

I can't find a way to reproduce this.. sporadic and random. Recently, sometimes almost immediately after reflash or reboot.

And not related to flush() either.
Before flushing stream. (goodbit: 0, eofbit: 0, failbit: 1, badbit: 1)
Error flushing stream. (eofbit: 0, failbit: 1, badbit: 1)

A bit more logging, I now got to this:
iostate 10: (goodbit: 1, eofbit: 0, failbit: 0, badbit: 0)
Before flushing stream. (goodbit: 0, eofbit: 0, failbit: 1, badbit: 1)
Error flushing stream. (eofbit: 0, failbit: 1, badbit: 1)

with this code
  LOG0("iostate 10: (goodbit: %d, eofbit: %d, failbit: %d, badbit: %d)\n", hapOut.good(), hapOut.eof(), hapOut.fail(), hapOut.bad());
  if(homeSpan.webLog.maxEntries>0){
    // I'm not reaching 500 entries yet.
    // ...
  }

  hapOut << "\n";

  LOG0("Before flushing stream. (goodbit: %d, eofbit: %d, failbit: %d, badbit: %d)\n", hapOut.good(), hapOut.eof(), hapOut.fail(), hapOut.bad());

Now I wonder something. HTTP requires the use of \r\n, not just \n.. but still, if the buffer hasn't been flushed, can't expect the socket to be closed. I will try with \r\n for a while.

I didn't read correctly and spoke to quickly.
iostate 10: (goodbit: 1, eofbit: 0, failbit: 0, badbit: 0)
iostate 11: (goodbit: 0, eofbit: 0, failbit: 1, badbit: 1)
Before flushing stream. (goodbit: 0, eofbit: 0, failbit: 1, badbit: 1)
Error flushing stream. (eofbit: 0, failbit: 1, badbit: 1)

I'm narrowing it down to something happening inside the IF if(homeSpan.webLog.maxEntries>0){ as I added "iostate 11" right there:
  LOG0("iostate 11: (goodbit: %d, eofbit: %d, failbit: %d, badbit: %d)\n", hapOut.good(), hapOut.eof(), hapOut.fail(), hapOut.bad());
  hapOut << "\n\r\n";

The HTML source when this happens seems to always end like this:
... top is ok ...

Entry	Up Time	Log Time	Client	Message

9	0:00:00:06	Sun Aug  4 17:05:04 2024	<IP_ADDRESS>	Trouble light ON.

8	0:00:00:06	Sun Aug  4 17:05:04 2024	<IP_ADDRESS>	Bureau: Armed for Away

7	0:00:00:06	Sun Aug  4 17:05:04 2024	<IP_ADDRESS>	Keybus reconnected. Trouble light OFF.

6	0:00:00:06	Sun Aug  4 17:05:04 2024	<IP_ADDRESS>	Country code: US, country maxTxPower: 30, currentTxPower: 19.50 (max:19.50), policy: AUTO. Setting country code.

5	0:00:00:00	Unknown

As seen above, the last  is where we'd normally see the IP address, like if homeSpan.webLog.log[index].clientIP.c_str() is somewhat corrupted...
More to come.

That last comment above led to nowhere.
However, one thing for sure, I can easily constantly (but randomly) reproduce it when using the original HS code hapOut << "\n";, and the issue cannot be reproduce when sending CRLF on a new line as per the RFC: hapOut << "\n\r\n";
Steps to reproduce

I have 5-6 lines that gets logged with WEBLOG level prior to wifi being established (if that makes any difference).
Have Chrome open to http:///status
Reboot the device
Wait for HAP server to be ready by seeing this in the serial console: Starting HAP Server on port 80...
Hit refresh constantly. If the issue has not reproduced itself within ~20 refresh, reboot and try again.

I added this:
      if (!homeSpan.webLog.log[index].clientIP.c_str()) {
        LOG0("NO IP !!!\n");
      }

And this is what causes the failbit to be set. I got the logs. Looking at the implementation of operator<< of ostream, it does set the failbit when the string is null.
So the problematic line is this:
      hapOut << "
" << i+1 << "	" << uptime << "	" << clocktime << "	" << homeSpan.webLog.log[index].clientIP.c_str() << "	" << homeSpan.webLog.log[index].message << "
\n";

.. when homeSpan.webLog.log[index].clientIP.c_str() is null.
The questions now is why is clientIP null on some occasions?

I don't understand. In HomeSpan.cpp, in vLog, I added serial logging like such:
    log[index].message=(char *)HS_REALLOC(log[index].message, strlen(buf) + 1);
    strcpy(log[index].message, buf);

    Serial.print("Last IP: ");
    Serial.println(homeSpan.lastClientIP);

    log[index].clientIP=homeSpan.lastClientIP;
    nEntries++;

and in HAP.cpp, in getStatusURL(), I added this logging:
      if(homeSpan.webLog.log[index].clockTime.tm_year>0)
        strftime(clocktime,sizeof(clocktime),"%c",&homeSpan.webLog.log[index].clockTime);
      else
        sprintf(clocktime,"Unknown");

      Serial.print("--- item - IP: ");
      Serial.println(homeSpan.webLog.log[index].clientIP);
      hapOut << "
" << i+1 << "	" << uptime << "	" << clocktime << "	" << homeSpan.webLog.log[index].clientIP.c_str() << "	" << homeSpan.webLog.log[index].message << "
\n";

and interestingly, this is what gets logged:
Last IP: <IP_ADDRESS>
Last IP: <IP_ADDRESS>
Last IP: <IP_ADDRESS>
Last IP: <IP_ADDRESS>
Last IP: <IP_ADDRESS>
Last IP: <IP_ADDRESS>
Last IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP:
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>

The IP that gets clipped comes from a WEBLOG I do in a setWifiCallback callback method.

Still chasing down. In getStatusURL() :
      Serial.print("--- item - IP: ");
      Serial.println(homeSpan.webLog.log[index].clientIP);
      if (!homeSpan.webLog.log[index].clientIP) {
        Serial.println("***  IP is null!");
      }
      hapOut << "
" << i+1 << "	" << uptime << "	" << clocktime << "	" << homeSpan.webLog.log[index].clientIP.c_str() << "	" << homeSpan.webLog.log[index].message << "
\n";

I get this:
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP:
***  IP is null!
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>
--- item - IP: <IP_ADDRESS>

I also added checks for all allocations / assignations of clientIP to check if clientIP = something completes successfully, and there is no error. clientIP is never null after assignation, but it gets to null in the function.
Any advice?

.. there is a threading issue going on. See where it logs "About to copy 3" twice.
Starting HAP Server on port 80...

About to copy 3
   Copy done, 3 entries

ClientIP: <IP_ADDRESS>
About to copy 3
   Copy done, 4 entries
About to copy 5
   Copy done, 5 entries
About to copy 6
   Copy done, 6 entries
About to copy 7
   Copy done, 7 entries
About to copy 8
   Copy done, 8 entries
About to copy 9
   Copy done, 9 entries

This is the modified code:
    Serial.printf("About to copy %d\n", nEntries + 1);
    log[index].clientIP=homeSpan.lastClientIP;
    Serial.printf("   Copy done, %d entries\n", nEntries + 1);
    nEntries++;

That homeSpan.setWifiCallback() callback, does it get executed by a different thread that the main loop?

I nailed it. WEBLOG / vLog is not thread safe.
Using this code as a proof:
    Serial.printf("About to copy %d - task name: %s\n", nEntries + 1, pcTaskGetName(NULL));
    log[index].clientIP=homeSpan.lastClientIP;
    Serial.printf("   Copy done, %d - task name: %s\n", nEntries + 1, pcTaskGetName(NULL));

produces the output:
About to copy 1 - task name: loopTask
   Copy done, 1 - task name: loopTask
WiFi Connected!  IP Address = <IP_ADDRESS>
About to copy 2 - task name: loopTask
   Copy done, 2 - task name: loopTask

Starting MDNS...

<removed>

Starting HAP Server on port 80...

About to copy 3 - task name: timeSeverTaskHa
   Copy done, 3 - task name: timeSeverTaskHa
About to copy 3 - task name: loopTask
   Copy done, 4 - task name: loopTask
About to copy 5 - task name: loopTask
About to copy 5 - task name: timeSeverTaskHa
   Copy done, 5 - task name: timeSeverTaskHa
   Copy done, 6 - task name: loopTask
About to copy 7 - task name: loopTask
   Copy done, 7 - task name: loopTask
About to copy 8 - task name: loopTask
   Copy done, 8 - task name: loopTask
About to copy 9 - task name: loopTask
   Copy done, 9 - task name: loopTask
About to copy 10 - task name: loopTask
   Copy done, 10 - task name: loopTask

Again, you can see there is a copy twice to index 3, which mean index 4 will remain null.
Now the task name shows timeSeverTaskHa ... About a week-or-so ago, I setup my local NTP server. So instead of reaching out to the web for an NTP server, all this local, so NTP response happens really fast, causing HomeSpan to log its own WEBLOG in its own thread called timeSeverTaskHandle.
I'll see if I can submit a PR to get WEBLOG / vLog thread safe, unless @HomeSpan you prefer tackling it?

I finally had a chance to explore this issue and you are indeed correct that there is a problem with the thread safety.  If you were running HomeSpan with autoPolling you would find that any calls to WEBLOG you made from outside of the polling thread, such as using WEBLOG in the main Arduino loop() method, could cause a thread issue if you were simultaneously trying to read the Web Log in a browser.
But even if did not use autoPolling, there is still a separate thread that calls WEBLOG behind the scenes - namely the background task that sets the time via an NTP server.  This background tasks terminates after the time is read, but as you've found above, it still can cause a thread collision.
Both issues can be addressed by making the Web Log code thread-safe.  After doing some research I found there is a standardized way in C++ of using a mutex to solve the problem of inter-locking separate functions that can read and write to the same shared variables.  The key is using a shared mutex and instead of manually locking and unlocking, you simply create a shared_lock or unique_lock object whenever you want to protect a piece of code.  As soon as the code goes out of scope, the object is destroyed and upon destruction it releases the lock.  The other advantage to using this method is that it allows multiple threads to read at the same time, but only one thread can write at any given time.  Since the mutex is shared, the system knows that it should block all threads from reading if anything is being written, and it should wait to start writing until any threads that are reading are done.
The changes to the code were very minimal (just a few extra lines) and I've uploaded to the dev branch.  Please give this a try and let me know if it solved the issue in this thread (no pun intended).  I tested myself using autoPolling with a log of extra weblog writes coming from the main Arduino loop() every 100ms and things seems to work as expected.
NOTE that the issue you have outlined in your other thread (#903) may or may not be related.  Once we tackle this we can explore the next.
Also, if this works I will add similar shared mutex locks to all other entry points, such as getVal and setVal to ensure you can read and write to Characteristics even from an external thread.  I knew there was a good reason I labeled autoPolling as experimental when initially released.  Maybe after all this is solved, I'll take off the "experimental" label.

Feature is now available in latest release v2.1.0.


Uncaught TypeError: mb_strpos(): Argument #1 ($haystack) must be of type string, null given

Laravel Version: 8.22.1
PHP Version: 8.0.1
Database Driver & Version: Postgres 13

Description:
Updated php to last version 8.0.1 on Ubuntu 18
Steps To Reproduce:
Just run php artisan serve
Starting Laravel development server: http://<IP_ADDRESS>:8000
[Tue Jan 19 23:40:37 2021] PHP 8.0.1 Development Server (http://<IP_ADDRESS>:8000) started
[Tue Jan 19 23:40:43 2021] <IP_ADDRESS>:60006 Accepted
[Tue Jan 19 23:40:43 2021] <IP_ADDRESS>:60008 Accepted
[Tue Jan 19 23:40:43 2021] PHP Fatal error:  Uncaught TypeError: mb_strpos(): Argument #1 ($haystack) must be of type string, null given, called in /home/akaderb/Projects/Menu/server/vendor/laravel/framework/src/Illuminate/Support/Str.php on line 183 and defined in /home/akaderb/Projects/Menu/server/vendor/symfony/polyfill-mbstring/bootstrap80.php:60
Stack trace:
#0 /home/akaderb/Projects/Menu/server/vendor/laravel/framework/src/Illuminate/Support/Str.php(183): mb_strpos()
#1 /home/akaderb/Projects/Menu/server/vendor/laravel/framework/src/Illuminate/Http/Concerns/InteractsWithContentTypes.php(16): Illuminate\Support\Str::contains()
#2 /home/akaderb/Projects/Menu/server/vendor/laravel/framework/src/Illuminate/Http/Request.php(371): Illuminate\Http\Request->isJson()
#3 /home/akaderb/Projects/Menu/server/vendor/laravel/framework/src/Illuminate/Http/Request.php(435): Illuminate\Http\Request->getInputSource()
#4 /home/akaderb/Projects/Menu/server/vendor/laravel/framework/src/Illuminate/Http/Request.php(64): Illuminate\Http\Request::createFromBase()
#5 /home/akaderb/Projects/Menu/server/public/index.php(52): Illuminate\Http\Request::capture()
#6 /home/akaderb/Projects/Menu/server/server.php(21): require_once('...')
#7 {main}
  thrown in /home/akaderb/Projects/Menu/server/vendor/symfony/polyfill-mbstring/bootstrap80.php on line 60
[Tue Jan 19 23:40:43 2021] <IP_ADDRESS>:60008 Closing

Argument #1 ($haystack) must be of type string, null given

Here's your error. You're passing null but should be passing a string.

Argument #1 ($haystack) must be of type string, null given

Here's your error. You're passing null but should be passing a string.

@driesvints
But I didn't change any thing, just upgrade php to latest version !
It was working fine before !!!

@driesvints
But I didn't change any thing, just upgrade php to latest version !
It was working fine before !!!

@AEK-BKF PHP 8 is more strict here.

@AEK-BKF PHP 8 is more strict here.

Thank you, I found the issue.

Thank you, I found the issue.

Thank you, I found the issue.

for those wondering, what was the issue and your solution?

Thank you, I found the issue.

for those wondering, what was the issue and your solution?


Dashboard Liquidity section TBTC + ETH pool APY

The rewards for the TBTC + ETH pool were voted to stop on October 7th 2021. The last pool top-up with KEEP was made on October 1st 2021.
https://etherscan.io/tx/0xe5b6f069e6cbc1da6c6a0a8210e03b0b64e410f2fd9428682f51e83df42114cc
These rewards were distributed over a week time-frame, so should have been depleted October 8th.  However, the current pool APY is still displayed as 49%.

@r-czajkowski in my dashboard it still shows APY >999% and no text as for the others
"Incentives removed
The incentives for this pool has been removed and you can no longer deposit the lp tokens. You can still withdraw rewards that you already earned."

@r-czajkowski in my dashboard it still shows APY >999% and no text as for the others
"Incentives removed The incentives for this pool has been removed and you can no longer deposit the lp tokens. You can still withdraw rewards that you already earned."

We closed the ticket because the changes from https://github.com/keep-network/keep-core/pull/2701 have been merged to main branch. The new version of dashboard will be released soon.


TAJO-2130: Remove inet4 data type

It seems to need to rebase.

@hyunsik , master branch is merged.
@jihoonson , I created the issued you mentioned, TAJO-2155 and TAJO-2156.

+1 LGTM.


RFP update and overhaul cordova-docs

My understanding from Apache Cordova PMC discussions is that an anonymous donation was received for the purpose of improving our documentation. I think it should be enough to cover multiple months of full-time work if we can source the work in a strategically intelligent manner.
Issues have been piling up over the years. I would really like to see some things happen in the near future such as:

review of all existing documentation to check for resolve items such as incorrect information, missing information, and any possible lack of clarity
get automatic doc build working again (#1035)
add tutorials and guides for some common tasks such as:

using Cordova with tools such as Babel, Rollup, and Webpack
integration with some larger frameworks such as Angular, Vue, Preact, React, and Ionic
authoring plugins for Android, iOS, and Electron

upgrade to use a super-modern documentation system such as Gatsby, JSDoc, etc.
go through the open issues and get them down to a handful at the most (let's get them fixed please, avoid some other outcomes such as "invalid" or "won't fix")

It would be ideal if we could find a maintainer or similar kind of Cordova expert as well as a well-qualified technical editor who would have the time to do this kind of update and overhaul in the near future.
I would also very strongly favor an agile approach where we start with tasks of limited scope and limited cost. I really hope we can start to see some positive impact before burning too much of the budget.
/cc @DRuggeri

Hey @brodybits!
Here is the original proposal that shaped the site and docs as they are today: https://github.com/apache/cordova-discuss/blob/doc-improvements/proposals/ImprovedDocumentationGenerator.md.
JSDoc is a great option, and was originally proposed as the way forward. I don't remember anymore why we didn't follow through with using it, but I suspect it was just because most of the Cordova docs content is hand-written prose, not automatically generated specs.

I have some more thoughts on this:

I think we need to update all documentation, including plugins & documentation for contributors and maintainers.
There may be quite a bit of investigation, testing, and interactive discussion needed to get it right.
I think it would be ideal if we could include efforts to verify that all existing and updated documentation reflects the actual behavior.
This kind of project should be done in an agile manner, with limited sprints starting with limited pilot phases, to avoid the risk of waste on incorrect priorities.
Domain experts, ideally multiple Cordova maintainers, would be needed to help ensure we get this right.

JSDoc does seem to be the most well-known, stable, and JavaScript documentation system. I have seen some other nice frameworks such as gitbook and Docusaurus, recall seeing at least one case of maintainers just dropping support.
I am also thinking this would be the perfect time for us to get the ball rolling, before an economic recovery leads to higher costs and more limited expert availability.


Durexforth compatibility.

I've studied your code- very cool! I think it is simple to make this work well with Durexforth.
If we not copy BASIC to RAM, and restrain logging to between HERE and LATEST, and keep BASIC banked out, I think it will be both compatible and useful in this context also.
I am capable of this but I thought I would bring it to your attention while the code is fresh in your mind.
Thank you for your efforts!

Unfortunately it appeared to me that it was required to bank BASIC out when banking the KERNAL out.    I don't see a banking scheme where BASIC is ROM, and KERNAL is RAM.   The alternative would be to change to hooking only the vectors in RAM instead -- a different algorithm for those entries which do have vectors.   Not all the entry points use vectors though.
Compare A000 (BASIC) to E000 (KERNAL) in this chart I composed from testing.   This monitor runs in bank 5, while normal operation of a C64 at power up is in bank 7.
; bank  A000  D000  D800  E000
; 0     RAM   RAM   RAM   RAM
; 1     RAM   CHARG CHARG RAM
; 2     RAM   CHARG CHARG ROM
; 3     ROM   CHARG CHARG ROM
; 4     RAM   RAM   RAM   RAM
; 5     RAM   I/O   NYBLE RAM
; 6     RAM   I/O   NYBLE ROM
; 7     ROM   I/O   NYBLE ROM

I hadn't seen your pull request until after I commented.   If that works for you, great!   Seems the $125f, $1261 are specific to Durexforth, so probably not generally useful to the rest of folks.   Now, adding some variables, and a configuration byte could make the code work better for both communities.  Another test could be to check if BASIC is already swapped out, and skip copying it.  But unfortunately probably not one size fits all, unless able to better detect the situation.
Hope this works out for you.

Actually, I just got it working with Durexforth. It's default memory bank is 6.
I was having a problem assembling it on windows 10 and I just patched it.
Skipping the move basic- Hard coding the lda 55's and 56's to lda #a0 and #00,
and moving 'log_end' from $a0 to $bf to suit memory use.


Update Google Drive Label

Package ID is longer correct for the current version of Google Drive.
Update Package ID and add app name because it differs from the label

Thanks


[macOS] Update icon for macOS Big Sur?

Summary
Suggestion to update the icon for the macOS version of the app?
Motivation
to match the new macOS Big Sur icon style
Describe alternatives you've considered
Manually changing the icon, it would be great if Atom could come with it though
Additional context
Proposed new icon: https://github.com/elrumo/macOS-Big-Sur-icons-replacements/raw/master/icons/Atom.icns

EDIT: I just found out that the icon linked above has a different license, so it probably wouldn't work to directly use that icon. However, if Atom would consider making their own, using the linked icon as an idea, that would probably work better...

EDIT 2: I just thought I'd clarify that in this issue, I'm mainly referring to the macOS version of the app as I would probably not expect cross-platform apps to use the Big Sur style on Windows and Linux for instance. Now, if there is no way to change the icon for just the macOS app without affecting the other platforms, that is understandable and I can close the issue.

Here's a good reference
https://github.com/elrumo/macOS_Big_Sur_icons_replacements

There is some existing OS-specific logic around the icon, that may help.
For others reference: icon is under Gpl V3

It's probably worth noting that, while not at all important to change quickly, Atom actually has 2 other icons, for Beta and Nightly

All you have to do is follow these instructions. I changed my Atom's icon by simply doing what they said - copy/pasting the .icns file linked by Jeff above and it worked like a charm. May have to remove Atom from your dock and then re-add it after changing the icon for the new icon to appear in the dock but it instantly changed it for me.

Yes please! :)

Yes please! :)

@ghost @jeff-hykin i dropped the icns icon and the result is this! Beautiful!
but can it be released when atom switches to electron version 9+? it is also fair to consider the versions that @adamsoutar indicates.

@ghost @jeff-hykin i dropped the icns icon and the result is this! Beautiful!
but can it be released when atom switches to electron version 9+? it is also fair to consider the versions that @adamsoutar indicates.

@adamsoutar here is the icon for beta.

@ghost thanks for icon, looks great on my dock. Just spent the last 20 mins appropriating WhatsApp, Photoshop and Notion too. The dock has never looked so good.

I really love this icon!
Thank you 🙏🏻


RG peer review

At least three areas of strength:
• I think this tutorial for learning to simulate data or download twitter data is a very useful tool for people who don't have access to data to work with making graphs!
• I like how much direction you provide to walk the user through all the tutorial steps
• The code is clear and easy to follow for beginners.
At least one thing you learned from reviewing their script:
• Its possible to write a simulation function (likert_sim) and make it look like real data!
At least one and no more than three areas for improvement:
• The data_viz_function.html is quite text heavy and I think it would benefit from some visual aids or possibly an r simulator where users could practice writing code?
• I was not able to write_csv in the guha_twitter.rmd so you might want to apply the package here() to make it more user friendly
Overall, great job with the tutorials! Looking forward to see how it all comes together!
@datalorax

Nice review, Rebecca!


Obsidian icon doesn't get changed

Created links successfully opens right vaults, but icon in task bar doesn't get changed.
I have edited .lnk file with my own icons.
I used this commands to create links, user folder is [redacted]:
.\mklnkwaumid.exe 'C:\Users\<user>\Downloads\aumid-stopgap-tools-0.1.0-x64\ObsidianMain.lnk' Tksh164.Obsidian.VaultMain 'C:\Users\sarae\Downloads\aumid-stopgap-tools-0.1.0-x64\runwaumid.exe' '-tp Tksh164.Obsidian.VaultMain \"main - Obsidian\" obsidian://open?vault=a3baa2db0cb952ea'

.\mklnkwaumid.exe 'C:\Users\<user>\Downloads\aumid-stopgap-tools-0.1.0-x64\ObsidianWork.lnk' Tksh164.Obsidian.VaultWork 'C:\Users\sarae\Downloads\aumid-stopgap-tools-0.1.0-x64\runwaumid.exe' '-tp Tksh164.Obsidian.VaultWork \"work - Obsidian\" obsidian://open?vault=6ac369b45b259074'

Upd: Got it, you need to pin new shortcut to taskbar to get changed icon. Previously I tried to run .lnk by double clicking link in a folder.

Upd: Got it, you need to pin new shortcut to taskbar to get changed icon. Previously I tried to run .lnk by double clicking link in a folder.


Worked on issue #18

Fixed the broken slash command in confessban by restricting the command to only one server, which reduces the latency in which the command is updated on discord. Fixed timeout issue
#18

!bounty 35

!bounty 25

!bounty 20

!bounty 25


Suggest changing the hostname in hosts (<IP_ADDRESS> entry) and hostname

Some folks have another raspberrypi on the network and might get confused when they try to reconnect to raspberrypi.local and wonder where all their changes went. (Like this somebody 😉) Suggest "teslapi" or similar. Simple sed exercise likely.

From: https://github.com/cimryan/teslausb/issues/5

Addressed: 1d19ac5c2f6ced75f34c6fd7f700d453bc2a5578
Network name is set to "teslausb".


Add C420S2 as not supported camera

See https://github.com/JurajNyiri/HomeAssistant-Tapo-Control/issues/243

Thank you!!


Move tests after #620

In #620 we have change coreblocks directory structure. This PR is a step towards aligning tests directory structure with coreblocks. Here I only move tests between directories to reduce number of merge conflicts with other PRs. In future some files with tests will be split to be more fain-grained.

Not merging due to conflicts.


Error when trying to configure PolygonMode

Hi, first of all thanks a lot for your work on this wrapper.
My problem is, I'm wrapping the creation of MaplibreTerradrawControl in a Function Component for use in a react-map-gl Map. It all works out perfectly, until I try to configure modeOptions for PolygonMode.
If I copy and paste the code from your demo website for SelectMode, everything works fine, but when I swap out the modeOptions object with this one, for example:
const modeOptions = {
    polygon: new TerraDrawPolygonMode({
      cursors: {
        start: 'crosshair',
        close: 'crosshair',
      }
    }),
  };

I get this error:

Which is the same error I get if I only use polygon as a mode, without render. But I think that's actually fine, as render should be required if you're not using the open prop.
I don't understand if I'm doing something wrong or if it's a problem of the package? I feel like I'm doing everything by the book here.

@eduardomecchia I think I fixed the bug in this PR (https://github.com/watergis/maplibre-gl-terradraw/pull/74) when no render mode is set. I just tried your modeOptions, it looks working in my side. could you try it again with the latest version?

I installed 0.4.3 and unfortunately the problem is still there with the same error.
I think it's a separate issue from the one you linked because it occurs whenever I configure polygon mode, doesn't matter if I also have render or not.
const TERRA_DRAW_MODES = ['polygon'];
const TERRA_DRAW_MODES = ['render', 'polygon'];

These are the two mode arrays I'm feeding to my TerraDrawControl.

@eduardomecchia I realized m.state sometimes keep remained as registered state. Thus, in some timing, when the plugin is added to maplibre map object repeatedly, terradraw throw the error. I fixed in the below PR to force reset mode state to unregistered. could you try version 0.4.4?
https://github.com/watergis/maplibre-gl-terradraw/pull/86

@eduardomecchia cool! let me close this issue now. Please feel free to open issues if anything


Issue in installation on OSX 10.14.1

I have java version 11.0.1
I have set $JAVA_HOME = /Library/Java/Home
then I cloned the git repo
next I ran the command : sudo python3 setup.py install
the installation was successful but when I opened a python interactive shell and tried to import jpype, i got the following error

import jpype
Traceback (most recent call last):
File "", line 1, in
File "jpype/init.py", line 17, in
from ._jpackage import *
File "jpype/_jpackage.py", line 18, in
import _jpype
ImportError: No module named _jpype

That is very weird. I can not reproduce this. Do you have the same environment for ipython3?

Is this still an issue?


Delete connection

Allow LC to receive delete connection message and delete connection.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 77.465%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
971

Relevant Lines:
1199

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 77.465%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
971

Relevant Lines:
1199

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 77.465%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
971

Relevant Lines:
1199

💛  - Coveralls


Rethink repo structure

Right now everything is chaotically mixed together. There are several separate components, they interoperate and depend on one another in particular ways, and some of them could be distributed while others that wouldn't make sense.
Rearrange all the files into a structure that makes sense. Possibly (probably?) also separate the llmtaskgraph prototyping library stuff into a new repository that this one would depend on.

This is progressing - llmtaskgraph things were organized more better (and tested to work correctly post reorg - that was a battle!) in 5e62de07e4c1e60a7ed34bfa38463cad4c958a84 and removed from this repo in 94261a562c2efc471d073125e4f4701803616052. They're now in https://github.com/knexer/llmtaskgraph.
Last step: link the repos together.


Replaced all prototype element references from $('xx') to $j('#xx')

After converting calls to use $j('#xxx') format, added a common JS method that checks for length of element to check existence of element in DOM.
Updated any other prototype specific calls that didn't work after changing element references to use $j

#1418
@dclarizio @martinpovolny please review/test.
@AparnaKarve if you get a chance can you test this.

@h-kataria @dclarizio Went over the changes, and it looks like all the changes are centered around -
1. .val()
2. miqDomElementExists (function that checks for $j('#' + element).length)
3. $(‘abc’) -> $j(‘abc’) conversions
4. A few visual effects

Tried a few screens that cover these changes, and they all seem to work well.

Did some testing of various areas in the UI as well.


See Grails issue #9467

Tell me (after this) if there is something to improve or to fix.

It looks like these changes may have already been applied independent of this PR.  Is that correct?

Hi Jeff and sorry for the delay ... Nothing (or very little content) from this PR has been applied.
From what I see I think it would be better to drop my pull request because now code base is really different. I already realigned my fork with latest version but I have to rewrite many parts; but to avoid waste anyone time next time we have to check what changes are needed/useful (let's update in the existing issue if still open).
Thanks for now, Sandro


refreshDevices Error

First off, everything seems to be working ok for me, but not sure if this will cause untended issues. That said, every 10 minutes (polling interval set to 300), I get the following error:
[Security System] refreshDevices Error: GET https://www.alarm.com/web/api/systems/systems/6660189 failed: [object Object]
[12/7/2021, 6:55:03 AM] [Security System] refreshDevices Error: GET https://www.alarm.com/web/api/systems/systems/6660189 failed: [object Object]
[12/7/2021, 6:55:03 AM] [Security System] Refreshing session authentication.
[12/7/2021, 7:00:09 AM] [Security System] Logging into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:00:10 AM] [Security System] Logged into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:00:10 AM] [Security System] Updating sensor FAMILY ROOM MOTION (Occupancy Sensor) (96622429-31), state=1, prev=0
[12/7/2021, 7:05:21 AM] [Security System] refreshDevices Error: GET https://www.alarm.com/web/api/systems/systems/6660189 failed: [object Object]
[12/7/2021, 7:05:21 AM] [Security System] Refreshing session authentication.
[12/7/2021, 7:10:49 AM] [Security System] Logging into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:10:50 AM] [Security System] Logged into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:16:17 AM] [Security System] refreshDevices Error: GET https://www.alarm.com/web/api/systems/systems/6660189 failed: [object Object]
[12/7/2021, 7:16:17 AM] [Security System] Refreshing session authentication.
[12/7/2021, 7:21:45 AM] [Security System] Logging into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:21:46 AM] [Security System] Logged into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:26:51 AM] [Security System] refreshDevices Error: GET https://www.alarm.com/web/api/systems/systems/6660189 failed: [object Object]
[12/7/2021, 7:26:51 AM] [Security System] Refreshing session authentication.
[12/7/2021, 7:32:18 AM] [Security System] Logging into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:32:18 AM] [Security System] Logged into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:37:46 AM] [Security System] refreshDevices Error: GET https://www.alarm.com/web/api/systems/systems/6660189 failed: [object Object]
[12/7/2021, 7:37:46 AM] [Security System] Refreshing session authentication.
[12/7/2021, 7:42:56 AM] [Security System] Logging into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:42:57 AM] [Security System] Logged into Alarm.com as<EMAIL_ADDRESS>[12/7/2021, 7:48:07 AM] [Security System] refreshDevices Error: GET https://www.alarm.com/web/api/systems/systems/6660189 failed: [object Object]
[12/7/2021, 7:48:07 AM] [Security System] Refreshing session authentication.
This has been happening for several weeks (at least). Again, don't seem to be having any issues, but maybe is indicative of an underlying problem.
Homebridge System (please complete the following information):

Node.js Version: 16.13.1
NPM Version: 8.1.2
Homebridge Version: 1.3.8
homebridge-node-alarm-dot-com v1.8.0
Operating System: Ubuntu 20.04.3 LTS

Same here. Effect is that my motion detector (seen as occupancy sensor) does not refresh and homekit automations are PITA.
Node.js Version: 16.13.1
NPM Version: 8.3.0
Homebridge Version: 1.3.8
homebridge-node-alarm-dot-com v1.8.0
Operating System: Raspbian GNU/Linux Buster (10)

How many devices do you have? I'm wondering if this is the same as #62

I have 7 configured devices or a total of 30 if I should also be counting ignoredDevices.

@kaganae There was an issue where MFA may not have been properly turned off then the user requested. I believe it was fixed in 1.8.1-beta.1. Can you confirm if you're still having issues with the latest beta?

I haven’t been getting the issue for a while, so I believe it must be fixed in the beta. Thanks!


kubectl not looking in local directory for kubelogin

What happened:
With  kubectl 1.25.0 when running commands kubelogin is not found when adjacent to kubectl in the local working directory. kubelogin is only found when in the PATH
What you expected to happen:
I would hope that kubectl would look in the local working directory as well as all the directories in the PATH.
How to reproduce it (as minimally and precisely as possible):
`
PS C:\ProgramData\Apps1> .\kubectl version --output=json
{
"clientVersion": {
"major": "1",
"minor": "25",
"gitVersion": "v1.25.0",
"gitCommit": "a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2",
"gitTreeState": "clean",
"buildDate": "2022-08-23T17:44:59Z",
"goVersion": "go1.19",
"compiler": "gc",
"platform": "windows/amd64"
},
"kustomizeVersion": "v4.5.7"
}
Unable to connect to the server: getting credentials: exec: executable kubelogin not found
It looks like you are trying to use a client-go credential plugin that is not installed.
To learn more about this feature, consult the documentation available at:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
PS C:\ProgramData\Apps1> dir .\kubelogin.exe
Directory: C:\ProgramData\Apps1

Mode                 LastWriteTime         Length Name

-a---          09/08/2022    19:35       45491712 kubelogin.exe
`
Anything else we need to know?:
Behaviour not seen with 1.24
Environment:
kubectl 1.25.0
PSVersion 7.2.4 Core
Windows 10 20H2
AKS 1.24.6

I would hope that kubectl would look in the local working directory

This is not the case. It will not load it from being in the same directory. You need to make sure it's in your PATH.
Check out the docs here https://github.com/Azure/kubelogin#setup-windows.
/triage support
/close


Updates targeting ElasticSearch v5.x.x

Hi,
This fix seems to not be already integrated in the last available version (3.0.2).
When a new delivery of elasticdump (which works with Elasticsearch 5) is planned ?
Thanks.
Hélène Joanin.


The message is improper when subscribe a service.

This change is

Reviewed 3 of 3 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable