Pdf size issue

Hello. Can you please help, how how I set width for pdf? I see there are only to formats 'A4' and 'A7'.
But if I need have width like 150mm?
Thanks.

With SwissQRBill v4, it is now possible to create the PDF document independently from the QR bill.
This allows you to create pages with any size.
To conform the specifications, the page the QR bill gets attached will still use the sizes required by the specifications.
Example:
import { createWriteStream } from "node:fs";

import PDFDocument from "pdfkit";
import { SwissQRBill } from "swissqrbill/pdf";
import { mm2pt } from "swissqrbill/utils";

const data = {
  amount: 1994.75,
  creditor: {
    account: "CH44 3199 9123 0008 8901 2",
    address: "Musterstrasse",
    buildingNumber: 7,
    city: "Musterstadt",
    country: "CH",
    name: "SwissQRBill",
    zip: 1234
  },
  currency: "CHF",
  debtor: {
    address: "Musterstrasse",
    buildingNumber: 1,
    city: "Musterstadt",
    country: "CH",
    name: "Peter Muster",
    zip: 1234
  },
  reference: "21 00000 00003 13947 14300 09017"
};

const stream = createWriteStream("different-pdf-size.pdf");

const pdf = new PDFDocument({ size: [mm2pt(150), mm2pt(300)] });
const qrBill = new SwissQRBill(data);

qrBill.attachTo(pdf);

pdf.pipe(stream);
pdf.end();


Allow specification of extra kwargs to Runner.

This means that tests can do things like run modules in check mode or request a diff. It's useful in some tests I've written.

It seems like travis and coverage are both broken on master too. I could make a separate PR for fixing those, if you'd like.

@weykent I have code not yet pushed for handling the new ansible-2.0.0 library.  We can track that portion elsewhere.

@weykent thanks for taking the time to construct this PR.  I like the concept, but have some suggestions.  Rather than supporting _runner_kwargs as a module parameter, I'd prefer using the existing pattern for passing data from py.test user to the Runner object.  For that, the code offers command-line options, and a marker for altering ansible parameters on a per-test basis.  My suggestion would be to ...

Add a command-line option --ansible--check to enable it session wide
Also, use the existing pytest.mark.ansible() for allowing function-scope override for check=True

I do like the idea of overriding ansible parameters, but not through the module directly.  We could expose a method/attr on the ansible_module object itself (e.g. ansible_module.__options.check=True .  But that could be a subsequent PR.

I'm not entirely sure I agree about how to specify check=True. Making it a command-line option doesn't make a lot of sense, because as far as I understand it, tests written with pytest-ansible will expect that the module successfully ran if you call ansible_module.whatever and it indicates success. That is, tests will then verify that whatever the module was supposed to do actually happened. If --ansible-check was globally specified, then none of those calls would do anything anymore, and all the tests would fail.
You can see the tests I've been writing here: https://github.com/weykent/ansible-runit-sv/blob/205add8261bb7cabc5bd8f6942727e2adca2a9b5/tests/test_runit_sv.py
In my own tests, most of the tests ensure that if the module succeeded, then its effects are visible. There's another test to ensure that if check mode is specified, then the effects are not visible. I hope this makes it more clear the level of granularity I'm reaching for here, and why.
What I have so far could use the pytest.mark.ansible to indicate check=True, as I only call the module once per test. I was trying to imagine in what circumstance one might want to call a module in both check and not-check modes, and couldn't come up with anything, but I don't think that is to say nobody would need it.
So, I'm definitely not strongly attached the the interface I proposed in this PR (and I'd be willing to change it), but I'm not sure I agree with the interface proposed aove.

@weykent Thanks for your comments.  The intent with the ansible_module fixture is that it allows a tester to call a specific module directly.  The arguments to the called method should match exclusively what the ansible module requires.  I feel a bit nervous passing along unused ansible module attributes that have special meaning to the py.test fixture.  We can head that direction eventually if there is a great need, just don't want to do it in a one-off manner.

If --ansible-check was globally specified, then none of those calls would do anything anymore, and all the tests would fail.

That is correct.  And I'd likely need to add tests for that.  I attempt to mimic supported ansible parameters with the pytest-ansible plugin.  While it may make more sense for uses to enable check=True mode on a specific test class/function, the pattern I've followed so far allows for the tester to optionally pass the parameter to the py.test command-line (much like with ansible).

What I have so far could use the pytest.mark.ansible to indicate check=True, as I only call the module once per test. I was trying to imagine in what circumstance one might want to call a module in both check and not-check modes, and couldn't come up with anything, but I don't think that is to say nobody would need it.

I agree.  While we can create a mechanism for altering ansible command-line parameters within a test, the current structure allows one to specify ansible parameters globally (via py.test command-line options), or per-test/class (via fixtures).
Thanks again for your feedback!


Replace XIVDB tooltips

XIVDB is no more and so are its tooltips. We need a replacement, preferably one that's not dependent on an external service so that it doesn't break unexpectedly.
Garland Tools appears to have all the necessary data about crafting actions, which is available as JSON data files that can be processed into a form we can use.

I do not remember what the tooltips looked like but in my forked version of the project I set up a tooltips that displays at least the name of the action using the translation files

Partially fixed in 76766a7a14c686f07afa0e8039d9d49b127e83cd.


MINA_IMAGE, MINA_AUTOMATION_LOCATION

Use environment variables MINA_IMAGE and MINA_AUTOMATION_LOCATION for the test executive. Change OCaml and Terraform field names, variables to match these changes.
Also change some other coda...image names to mina...image.
Last PR for #9041.

Looks great because LOCATIOPN is only in the PR description

Not any longer! :-)


⚠ Bofrost.at has degraded performance

In a3db370, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2186 ms

Resolved: Bofrost.at performance has improved in 8865151 after 6 minutes.


How to run this code?

Hi,
I am using the EuRoC dataset. I build the code on ROS kinetic on Ubuntu 16.04. I ran the exact code line which appears in the README file. However, after only two IMU callbacks, the data of my IMU is nan and 0.. I already checked my dataset and it is not empty. Please, does someone see what is going on here?
Thank you

Find a timestamp from the groundtruth and copy that to stand_still_end in the launch file.

From which file do I need to copy the timestamp ? From the imu0 one ?

The images themselves are named by the timestamp in the ASL dataset format. They'll be in a folder like this
euroc/MH_03/mav0/cam0/data
Just find the image that is at the end of the second stand still and use the filename without png (make sure to keep the '.' because it needs to be a double because of the size of the number).
Alternatively ROS node support is underway and I'm hoping to have it done by the end of the week if you just want to wait for that.

So I updated the workspace with the ROS support for EuRoC. However, I do not understand how you computed the calibration matrix (since they aren't the matrix from the sensor.yaml files from the EuRoC dataset). Could you please tell me what are they? Thanks.

@ziyangli  How to find the correct timestamp from the groundtruth data file. when I load a dataset from the given link, There is a folder named state_groundtruth_estimateo in which a data.csv file is found. So, which timestamp is correct in the data.csv? Thanks.

In addition to changing the "stand_still_end" parameter in the launch file, the "stand_till_start" parameter also needs to be changed, it will just be the first image in the sequence. Alternatively, you can add the "stand_still_start" parameter in the roslaunch commnad in the same format. Dont forget the dot! Good luck!


indented better

Summary
Fixes #
Links to related discussion
Proposed changes in this pull request
General checklist

[ ] I have updated the CHANGELOG.txt
[ ] I have created backports if necessary
[ ] I have updated the documentation and/or amended the upgrade section
if necessary

@khiladimahan1210 thank you for the contribution 🏆
I agree with @jrief that the indentation is a bit off. I'm closing this PR for now but would accept another request with changes like these:
{% for ance in ancestors %}

	{% if not forloop.last %}
        {{ ance.get_menu_title }}
        »
	{% else %}
	    {{ ance.get_menu_title }}
	{% endif %}

{% endfor %}

Please also add a a changelog entry to reflect the change 👍


Added support for docker's WORKDIR and updated singularity's url

Change folder to docker container's working directory before executing the entrypoint
Updated singularity's url in the Dockerfile (previous one not reachable anymore)

heyo @pierlauro ! This is a very important feature, but I think it belongs with singularity proper - e.g., if there is a final WORKDIR in the container it should be honored by the container natively. Would you be able to open an issue on the singularity repository to suggest this?

@pierlauro see this issue --> https://github.com/sylabs/singularity/issues/380 please comment here if you think Singularity should honor WORKDIR.

sylabs/singularity#380

I think Singularity should provide a WORKDIR feature similar to docker, I added my considerations in the cited issue.
I just adapted and tested the PR to be merged after the new commits in the master.

Since you didn't bring the issue back here and merged in another repo, closing this issue.


PLT-6763 FE: Auditors sign certificates

https://input-output.atlassian.net/browse/PLT-6763
Across BE task - PLT-6367 / PR #74 (merged into master)
@coderabbitai:ignore

Summary by CodeRabbit

New Feature: Introduced CertificationForm component for dynamic certification form handling.
New Feature: Added CertificationMetadata component to submit and manage certification metadata.
New Feature: Integrated CertificationMetadata into CreateCertificate component, allowing users to review and submit certification metadata.
Refactor: Updated ReportUpload component to use the new CertificationForm component, improving code modularity and maintainability.
Refactor: Enhanced CreateCertificate component with improved state management and error handling.
Style: Adjusted CSS styles across various components for better UI consistency.

@coderabbitai: ignore

After I bought the subscription, I had to refresh  several times to see the Auditing menu

Validation error in the case of valid value

I wasn't able to submit it even if the form didn't show any validation error or unfilled mandatory field
https://github.com/input-output-hk/dapps-certification-web/assets/3176255/f238c1a8-1a50-4086-9383-f071739a8855

Subject inpute accepts in the field invalid format

Please consult the Swagger api data model specification

@bogdan-manole

After I bought the subscription, I had to refresh several times to see the Auditing menu

Resolved with a temporary hack (since in revamp the handling of subscriptions will be moved to before entering the pages) in PR#10 PLT-7439 - Fix for Blank page in the tool at times

Validation error in the case of valid value

:) See it isn't a valid value. A valid http(s):// URL must have .json or .pdf.

I wasn't able to submit it even if the form didn't show any validation error or unfilled mandatory field

Thought is was resolved, checking it, will resolve. Issue - Script Hash not showing validation errors

Subject inpute accepts in the field invalid format

Per PLT-4386 - Subject [1-64 chars], and also discussion with @RSoulatIOHK (missed to pass the info to you :pray: )

:) See it isn't a valid value. A valid http(s):// URL must have .json or .pdf.

an URL could be a PDF without being specified as so. Please follow the Regex pattern specified by the Swagger for that field.

Level 0 certification
I don't get L0 audit when submitting and auditor report for L0 audit
{"1304":{"metadata":[["ipfs://bafkreiaerryd4ou6edxrygimjxzgtv4wyjmlhcqqtg47cosz5oygzdnf","le"]],"rootHash":"64346f7e71cdc0b790c3992114f3ab3b15dddb7e4b5ffc10269a06fbb5484828","schemaVersion":1,"subject":"test","type":{"action":"CERTIFY","certificateIssuer":"test","certificationLevel":"l2"}}}
Instead I should get:
{"1304":{"metadata":[["ipfs://bafkreiaerryd4ou6edxrygimjxzgtv4wyjmlhcqqtg47cosz5oygzdnf","le"]],"rootHash":"64346f7e71cdc0b790c3992114f3ab3b15dddb7e4b5ffc10269a06fbb5484828","schemaVersion":1,"subject":"test","type":{"action":"**AUDIT**","certificateIssuer":"test","certificationLevel":**0**}}}
Note: certificationLevel should be an int with just the certification level and not a str.
Level 1 certification
I get "Undefined" in on/off chain metadata

Level 2 certification
Everything looks OK. Just the need for BE to change certificationLevel which should be an int with just the certification level and not a str.

@bogdan-manole

https://github.com/input-output-hk/dapps-certification-web/pull/8#issuecomment-1713760452

Can you fix the validation in BE. The regex used in FE is /^(https?:\/\/)?(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,255}\.[a-z]{2,6}(\b([-a-zA-Z0-9@:%_\+.~#()?&\/\/=]*))?\.(?:jpg|jpeg|png|gif|bmp|svg|webp|tiff|tif)$/

Website URL regex pattern

/^(https?://)?(www.)?[-a-zA-Z0-9@:%.+~#=]{1,255}.[a-z]{2,6}(\b([-a-zA-Z0-9@:%+.~#()?&//=]*))?$/
@RSoulatIOHK ,

Check that Contract address is a valid address

Contact Address under DApp Script will be validated across pattern ^(addr_test1|addr1)[a-zA-Z0-9]{53,}$; and will be non-mandatory

Display when a regex fails for DApp Scripts (Repository URL and Script hash look missing)

Able to see errors now with a minor fix, unable to reproduce the scenario when the errors weren't visible. Can you recheck
https://github.com/input-output-hk/dapps-certification-web/assets/11058223/ee106ddd-2b8f-46f3-989a-c399163e516b
Note -

Received undefined in off-chain and on-chain downloaded JSONs
BE is pointed towards https://dapps-certification-web.scdev.aws.iohkdev.io/ now, and not excuse.ro ( it has latest profile/subscription API updates for the moment)

I'll pull it this morning for test. Which BE version should I be using?

On a failed testing, we should not be able to review Certification metadata because we should not issue any.

I don't know if excuse.ro is the right BE but I can't get "L1" metadata. I get "Something went wrong message"

@RSoulatIOHK ,

"Something went wrong message"

Due to 204 no content in POST api. Must be because the BE API is not the right one. Neither of the deployments excuse.ro nor aws has the right branch for this. Bogdan needs to confirm the branch that this needs to be verified across.

On a failed testing, we should not be able to review Certification metadata because we should not issue any.

Is it OK to ensure this in revamp?

After something went wrong message,  you can never retry as it says " Forbidden - Certification already started". This should fix.

Need some clarifications over the workflow, will address in revamp once that is clear

However we should be able to view the full html report, we currently only view the failed unit tests:

Will handle in revamp, as this will be a tabular view

subject matches ^[A-Za-z0-9_]{1,64}$ ( I know it fits the current one, but let's keep the same regex on both sides.

Updated the FE validation to be .required(...).max(64, ...).matches(/^[A-Za-z0-9_]+$/, .... just to ensure the form errors are thrown properly
Other fixes done -

remove mandatory json/pdf extension checks at the reportUrl
update social.github pattern to ^(?=.{1,39}$)[a-zA-Z0-9]+(-[a-zA-Z0-9]+)*$ rather than url
social has properties - twitter, github, contact*, website*, discord ; fixed value mapping in these fields
ensure optional fields not sent as empty-strings in payload; they are either null or not existing
Github - field was missing in Auditor Report Upload, added it there.

Cc: @bogdan-manole @RSoulatIOHK


CLOUDSTACK-10319: Allow TLSv1, v1.1 for XenServer, Vmware

Description

This reverts changes from #2480, instead moves TLS settings to
java ciphers settings config file. It should be sufficient to enforce
TLS v1.2 on public facing CloudStack services:

CloudStack webserver (Jetty based)
Apache2 for secondary storage VM
CPVM HTTPs server

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Checklist:

[x] I have read the CONTRIBUTING document.
[x] My code follows the code style of this project.
[x] All new and existing tests passed.

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian.  I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1827

Checked test results, test LGTM (the failures are all env issues). I'll merge this to unblock other PRs who are failing testing on older xenserver/vmware based environments.

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian.


exiv2: Patch cmake files to correctly locate libs

Motivation for this change
Exiv2's .so and .a files are placed in the default output, but the cmake file looks for them in the dev output. This causes build failures in at least once case (#77581).
This patch takes the same approach as #33953, which solved a similar issue in the clang derivation.
The two outputs in question (.out and .dev) looks like this:
/nix/store/...-exiv2-0.27.2
|-- bin
|   `-- exiv2
`-- lib
    |-- libexiv2-xmp.a
    |-- libexiv2.so -> libexiv2.so.27
    |-- libexiv2.so.0.27.2
    `-- libexiv2.so.27 -> libexiv2.so.0.27.2

/nix/store/...exiv2-0.27.2-dev
|-- include
|   `-- exiv2
|       |-- *.h
|       `-- *.hpp
|-- lib
|   |-- cmake
|   |   `-- exiv2
|   |       |-- exiv2Config-release.cmake
|   |       |-- exiv2Config.cmake
|   |       `-- exiv2ConfigVersion.cmake
|   `-- pkgconfig
|       `-- exiv2.pc
`-- nix-support
    `-- propagated-build-inputs

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

I'm running nixpkgs-review but it might take a while on my laptop. Will comment with results in the morning.
As mentioned in  #77581, I have been using this patch locally for a few days to successfully build and run digikam 7. Things seem to work as expected.

Also note that I have zero experience with cmake; if there's a better way to resolve this, I'm all ears. 🙂

Ideally someone would make a pull request against the upstream repository to support absolute GNUInstallDirs variables. See https://github.com/jtojnar/cmake-snips

I believe nixpkgs-review is fine:
Result of nixpkgs-review 1

8 packages failed to build:
- calligra
- gimpPlugins.focusblur
- gimpPlugins.texturize
- photivo
- photoflow
- python27Packages.pyexiv2
- python37Packages.py3exiv2
- python38Packages.py3exiv2

110 packages built:
- cataract
- cataract-unstable
- darktable
- deja-dup
- digikam (digikam5)
- dolphin
- doodle
- dragon
- dropbox-cli
- elisa
- empathy
- exiv2
- geeqie
- gegl_0_4
- gexiv2
- gimp
- gimp-with-plugins
- gimpPlugins.exposureBlend
- gimpPlugins.fourier
- gimpPlugins.gap
- gimpPlugins.gimplensfun
- gimpPlugins.gmic
- gimpPlugins.lightning
- gimpPlugins.lqrPlugin
- gimpPlugins.resynthesizer
- gimpPlugins.ufraw
- gimpPlugins.waveletSharpen
- gmic-qt-krita (gmic_krita_qt)
- gnome-photos
- gnome3.cheese
- gnome3.file-roller
- gnome3.gnome-books
- gnome3.gnome-boxes
- gnome3.gnome-color-manager
- gnome3.gnome-contacts
- gnome3.gnome-control-center
- gnome3.gnome-documents
- gnome3.gnome-terminal
- gnome3.gnome-tweak-tool
- gnome3.gnome-user-share
- gnome3.nautilus
- gnome3.nautilus-python
- gnome3.shotwell
- gnome3.totem
- gnome3.tracker-miners
- gnomeExtensions.gsconnect
- gnunet
- gnunet-gtk
- gnunet_git
- gpscorrelate
- gramps
- gthumb
- gwenview
- hugin
- k3b
- kde-cli-tools
- kdeApplications.baloo-widgets
- kdeApplications.dolphin-plugins
- kdeApplications.kdegraphics-thumbnailers
- kdeApplications.kdenlive
- kdeApplications.kfind
- kdeApplications.kio-extras
- kdeApplications.kolourpaint
- libsForQt5.libkexiv2 (kdeApplications.libkexiv2)
- kdeApplications.okular
- libsForQt5.baloo (kdeFrameworks.baloo)
- libsForQt5.kfilemetadata (kdeFrameworks.kfilemetadata)
- kdeplasma-addons
- kdev-php
- kdev-python
- kdevelop
- kdevelop-unwrapped
- kile
- kmenuedit
- krename (krename-qt5)
- krita
- krohnkite
- kwin-tiling
- libextractor
- plasma5.khotkeys (libsForQt5.khotkeys)
- luminanceHDR
- mythtv
- nomacs
- pantheon.elementary-greeter
- pantheon.elementary-photos
- pantheon.extra-elementary-contracts
- pantheon.switchboard-plug-pantheon-shell
- pantheon.switchboard-with-plugs
- pantheon.wingpanel-with-indicators
- pdf2djvu
- peruse
- photoqt
- phototonic
- plasma-browser-integration
- plasma-desktop
- plasma-workspace
- plasma5.powerdevil
- plasma5.systemsettings
- qgis
- qgis-unwrapped
- qimgv
- rapid-photo-downloader
- strigi
- tellico
- ufraw
- variety
- vdrPlugins.xineliboutput
- viewnior
- viking
- wacomtablet

The packages that failed to build also failed to build locally without this pull request, so I don't think the failures are related.

Hmm, this does not seem necessary on master: https://github.com/NixOS/nixpkgs/issues/95232#issuecomment-672528716
But I do not see why.

Oh, it was walked around by f1a58aba151e59c4a5c438c501663944d608011f.

Oh, great! Thank you for investigating.


[macOS] Minimize to Tray/Command-Q not working

Version 5.3.0-20200310172631+4f2689009b
macOS 10.14.6 (18G3020)
"Minimize to Tray" (Command-Q) is not working for me. Command-Q makes Keybase flash in the menu bar but the window remains in view. Clicking the menu item (Keybase > Minimize to Tray Command-Q) also has no effect.
Clicking the red stoplight window menu works, as does Command-W.
"Quit Keybase" in the menubar hamburger menu and "Quit" from control-clicking on the dock icon both work and quit the app completely, including the menubar app.
Related but not identical issues
https://github.com/keybase/client/issues/4721
[…]
https://github.com/keybase/client/issues/20580
Notably, there are tray issues on other platforms as well, possibly shifting the balance to make Command-Q and its equivalents on other platforms all behave like "Quit Keybase" in the menubar hamburger menu and "Quit" from control-clicking on the dock icon. For example:
https://github.com/keybase/client/issues/4715
https://github.com/keybase/client/issues/5824
https://github.com/keybase/client/issues/6447
https://github.com/keybase/client/issues/22930

Came here to file the same bug, just adding on to this one instead of creating a new issue.
On macOS Catalina, 'Minimize to Tray' no longer works in the new Keybase (5.3.0). I confirmed on a second machine that still had 5.2.x on it, that it
worked there, then upgraded it to 5.3, and it stopped working. So something broke between those two versions.
my log id: 8bcbc38506d7a693b3ee9b1c

Same issue on MacOS 10.15.3 and KeyBase yersion 5.3.0-20200310172631+4f2689009b

Same, MacOS 10.14.6, Keybase version 5.3.0-20200310172631+4f2689009b

I was halfway filing
"I love Keybase, really, but it's annoying that the client on macOS doesn't act properly like other apps.  It doesn't hide on Cmd-H, doesn't minimize on Cmd-M, and doesn't close on Cmd-W.  There may be other similar issues."
when I noticed this had partially been filed.

Try Icon vanished after upgrade on Linux Mint 19 Tara Cinnamon 4.0.10 too
keybase version 5.3.1-20200320154633+3e235215b3
I notice that keybase seems to be run with the service option, I'm not sure what this means or whether it's related. keybase may have been running like this for months.
$ run_keybase %u
Shutting down Keybase GUI...
Unmounting /run/user/1000/keybase/kbfs...
Unmounting and shutting down kbfsfuse...
Shutting down keybase service...
Starting via systemd...
run_keybase: Success!

$ pgrep -a keybase
7682 /usr/bin/keybase-redirector /keybase
8730 /usr/bin/keybase --use-default-log-file --debug service

Fixed in 5.4.0. "Minimize to Tray" (Command-Q) and clicking the menu item (Keybase > Minimize to Tray Command-Q) both close the window now.
Also a new command "Quit Keybase Completely" added in Keybase menu and bound to Alt/Option-Command-Q (⌥⌘Q). It behaves like "Quit Keybase" in the menubar hamburger menu and "Quit" from control-clicking on the dock icon.
Probably as close to the the ideal solution of making Command-Q behave normally (instead of like Command-W) as we are likely to get, so I am closing this. Command-H does hide the app, and Command-M does minimize the app.


Fixburst

Local test bench improvement:  120k/sec -> 170k/sec
Added ability to set the ring buffer size/count
Some stats were not updated when compiled with O3, made volatile

I tested that this code doesn't cause any major blowups this afternoon as part of this testing: https://gist.github.com/csibbitt/04353ae6db22f5961c76c2631ba28a2f  It didn't fix my problem, but it did improve behavior, tunables, and output, so I want to merge it.
I'll play with the new knobs in the next round and make sure they do what they look like they do, but the changes are straightforward enough.


New controls, adjustments, modules (oh my!)

Hey Ari.
After several months of adding, modding and testing, I feel its time to actually do a pull request.
There may still be some testing required, BUT I don't want to keep you in the dark too long about what my partner and I have been working on/implementing in our own WP theme.
This framework has been an awesome edition to our theme and we wanted to share the modifications with you.
Some of the new control consist of:
slider-advanced:

Which has the ability to select a unit without hard coding one suffix. This also has media queries (similar to elementor).
border:

Pretty self explanatory.
spacing-advanced:

Pretty much the same as spacing, but with the ability to sync all fields with media queries (similar to Elementor)
toggle tabs:

For our project, we needed an ability to tab controls together. Now, I wasn't sure how to pull that off, BUT I did a little high-jacky method that pulls other controls list items and sticks them into its own while maintaining the original events. Its a little glitchy with requireds, but I haven't figured out a proper fix yet...
Kirki::add_field('theme-options', array(
	'label' => esc_html__('Button background', 'text-domain'),
	'section' => 'buttons_section',
	'settings' => 'btn_bg_tabs',
	'type' => 'toggle-tabs',
	'required' => array(
		array(
			'setting' => 'btn_style',
			'operator' => '==',
			'value' => 'standard'
		)
	),
	'choices' => array(
		esc_html__('Normal') => array(
			'btn_bg_style',
			'btn_bg_classic',
			'btn_bg_gradient'
		),
		esc_html__('Hover') => array(
			'btn_bg_hover_style',
			'btn_bg_hover_classic',
			'btn_bg_hover_gradient'
		)
	),
	'transport' => 'auto'
));

An example of how its use.
typography-advanced:

Pretty much the same as typography, but with media queries (Can be disabled).
box-shadow:

Pretty self explanatory. I think you did add a box shadow control already, so this one is late to the party.
color-gradient:

A color gradient control.
sortable (mod):

I added a 'mode' to sortable which allows both show/hide along with custom text.
Kirki::add_field('theme-options', array(
	'label' => esc_html__('Social media icons', 'konio'),
	'section' => 'social_media_icons_section',
	'settings' => 'social_media_icons',
	'type' => 'sortable',
	'mode' => 'text',
	'choices' => array(
		'facebook' => esc_html__('Facebook', 'konio'),
		'twitter' => esc_html__('Twitter', 'konio'),
		'instagram' => esc_html__('Instagram', 'konio'),
		'snapchat' => esc_html__('Snapchat', 'konio'),
	),
	'default' => array(
		'facebook' => '#',
		'twitter' => '#',
		'email' => '#'
	)
));

I also added a new module for resetting a section inside of customizer. I saw in a post issue post that you removed it cause of stability issues, but I discovered a stable way to handle resetting.

	/**
	 * Handles resetting called by reset button
	 */
	public function ajax_reset_section()
	{
		$section = isset ( $_REQUEST['section'] ) ? $_REQUEST['section'] : '';
		$nonce = isset ( $_REQUEST['nonce'] ) ? $_REQUEST['nonce'] : '';
		if ( !wp_verify_nonce( $nonce, 'kirki-section-reset'  ) )
			wp_send_json_error( __( 'Invalid nonce.', 'kirki' ) );
		if ( empty( $section ) )
			wp_send_json_error( __( 'Section not set', 'kirki' ) );

		if ( isset( Kirki::$sections[$section] ) )
		{
			foreach ( Kirki::$fields as $field )
			{
				if ( $field['section'] !== $section )
					continue;
				if ( $field['option_type'] === 'theme_mod' )
					remove_theme_mod( $field['settings'] );
				else
					delete_option( $field['settings'] );
			}
			Kirki::$skip_output_context = true;
			new Kirki_CSS_To_File();
			do_action( 'customize_save_after' );
			Kirki::$skip_output_context = false;

			update_option( '_kirki_customize_last_section', $section );

			wp_send_json_success();
		}
		else
		{
			wp_send_json_error( __( 'Could not find section', 'kirki' ) );
		}
	}

There are the controls and also internal mods like the dependency postMessage output issue, section reset, etc.
Like I mentioned before, I've been working on this changes for a while and wanted to bring them to your attention before things got out of hand or you decided to do a complete 180 with the project.
That way, these additions/changes can be included in the changes.
I'm a little cramped on time right now (And I may of forgotten some things), but let me know if you have any questions! And I'll be posting more examples later today.
I also merged this with your current commit.
This is also one of my first contributions to open-source projects, so let me know if some things are off.
P.S.
I made different controls (slider-advanced, typography-advanced, etc) to keep the integrity of the current controls.
Nathan

I'll be honest... this PR is way too large to properly review, it's going to take me a while.
So far I've only reviewed the slider and I have to admit I don't quite get it.
First of all, I don't see the media queries anywhere, in any of your controls.
But even if I did, I've been struggling for days with the idea and I don't understand why a developer would want to add media-queries in controls. Don't get me wrong, I get why it's there, Elementor has it too in its elements and when building a page it's useful because typography is not responsive/adaptive.
But why not just make the typography adaptive? Why do we work on workarounds instead of solving the underlying problem?
I guess that's just my naive and somewhat philosophical point of view, but I've always believed that these responsive controls in Elementor are a bad example and are only there to work around the limitations of the theme used in conjunction with Elementor. The plugin can be used with any theme so it has to be able to work well with all of them. So, the best solution in that case was to add media queries in font-sizes etc. It made sense for them. But why follow the same bad example when building a theme and one has the opportunity to build something really good without all these limitations?
Secondly, why save the value as an array? why [value:20, unit:'px'] instead of plain 20px? Would it not be simpler to implement and potentially cleaner?

I figured it would take a while.
The media queries can be enabled by adding:
'media_queries' => true
In say slider-advanced for example.
	'choices' => array(
		'use_media_queries' => true,
		'units' => array(
			'px' => array(
				'min' => 1,
				'max' => 500,
				'step' => 1
			),
			'em' => array(
				'min' => 1,
				'max' => 30,
				'step' => .1
			)
		),
	),

Secondly, why save the value as an array? why [value:20, unit:'px'] instead of plain 20px? Would it not be simpler to implement and potentially cleaner?

Why I split them up is for reloading purposes.
Since we're allowing the user to also select multiple units as well (See example above), I needed to know which unit they selected to reload inside of customizer. Having it as one string would be cleaner output wise, yes, but parsing it back was iffy to me.
I originally had RegEx to parse, but decided to go with this. I'm still on the fence about it.
And about where the media queries are, some defaults are defined in class-kirki-config.php
		$default_queries = [
			'desktop' => 'global',
			'tablet' => '@media screen and (min-width: 768px) and (max-width: 991px)',
			'mobile' => '@media screen and (max-width: 575px)'
		];
		if ( !isset( $this->config_final['media_queries'] ) )
			$this->config_final['media_queries'] = array();
		foreach ( $default_queries as $k => $v )
		{
			if ( !isset( $this->config_final['media_queries'][$k] ) )
				$this->config_final['media_queries'][$k] = $v;
		}

The user/developer can change them when create a config however.
	Kirki::add_config('theme-options', array(
		'capability'    => 'edit_theme_options',
		'option_type'   => 'theme_mod',
		'media_queries' => array(
			'tablet'  => 'custom media query'
			'mobile'  => 'custom media query'
		)
	));

Above was done so you do not have to copy and paste media queries everywhere. So all the advanced controls that utilize desktop/tablet/mobile buttons inherit the same values. I now realize I should make them override-able  per control, but I can add that later on.

@aristath Hey, ignore my pull request for now. I need to tweak some things in the controls that are broken and I'm also going to simplify some things and fix the code standard issues.


jsonapi to support edit

Summary
Today gopass-jsonapi does not support the API to edit a password. Having that feature would definitely be useful.
Steps To Reproduce
https://github.com/gopasspw/gopass/blob/master/docs/jsonapi.md - the documentation clearly mentions which APIs are available, and edit is not a part of it.
Expected behavior
An API corresponding to edit

request

{
   "type": "edit",
   "login": "myusername",
   "password": "mynewpassword"
}

response

{
   "username": "myusername",
   "password": "mynewpassword"
}

Environment

OS: macOS Catalina
OS version: Darwin
gopass Version: 1.10.1
Installation method: source/brew

Additional context
Having this API as a part of the jsonapi would open up multiple other opportunities like feature of editing directly from the gopassbridge extension. Nice to have.

Hi All,
If no one is working on this enhancement, can I start working on this?


Importing external remote modules

Consider the following Javascript code:
import Toastify from<EMAIL_ADDRESS>
If I want to convert it to the equivalent TeaVM code, it should be something like:
@JSBody(
        script = "Toastify({text: \"This is a test toast\",duration: 3000,}).showToast()",
        imports = @JSBodyImport(
                alias = "Toastify",
                fromModule = "https://esm.sh/toastify-js@1.12.0"))
    private static native void callModule();

The code however doesn't work, saying that the object Toastify does not exist (it seems):
Uncaught TypeError: $rt_import_0 is not a function
    at ia_Client_callModule$js_body$_1 (classes.js:1053:10)
    at ia_Client_main (Client.java:53:1)
    at classes.js:607:19
    at $rt_startThread (classes.js:1752:22)
    at classes.js:606:13
    at onload (3000/:8:25)

Also Network tab of Devtools shows no connection has been made to unpkg.sh (again I may be wrong).
In general, does TeaVM support such "remote" imports? If yes, is there an example?
And in general, how does calling third-party Javascript libraries from TeaVM work? Should we dynamically add <script> tags to the HTML head for loading the library and then calling it in native snippets?
Thanks

Update, to solve this, one solution is something like the following code,
 import("https://esm.sh/toastify-js@1.12.0").then( ({default :Toastify}) => {
            Toastify({text: "This is a test toast",duration: 3000,}).showToast()
        })

, which I can run from a <script> tag in index.html (without type="module"), but not in the script argument of a JSBody, because "Uncaught TypeError: $rt_globals.import is not a function". Is there another step I need to do to make this work?

Why don't you just include toastify.js with <script> element and use it from global name? Where did you get the library? Is it this one? So documentation states that you need following:
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/toastify-js"></script>

And then you don't need any imports

TeaVM's import functionality is a little-bit old-fashioned. Currently TeaVM only supports ES5 output. It's a subject of change in the future, but right not it can't use ES6 import. Instead, it generates common.js compatible imports, which can be used with node.js and webpack.

Why don't you just include toastify.js with <script> element and use it from global name? Where did you get the library? Is it this one? So documentation states that you need following:
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/toastify-js"></script>

And then you don't need any imports

Injecting a script tag is fine and works for this library, but not easily for all others (I'm looking at Transformer.js for example). Looking forward to having the ES6 modules.
Now another thing: A very "hacky" solution to have import parsed verbatim would be to wrap it in an eval. I tried and it worked, but it seems TeaVM parser only recognises single-line,  double-quotation strings, and everything else (including multi-line or template strings) is rendered as eval(), i.e. evaluating nothing (as I saw in classes.js).
Hence, this works:
eval("console.log(99) ")

eval(`console.log(99)`)

Is it also part of the consequences of supporting older Javascript standards?
Thanks

Yes, I as said, TeaVM only works with ES5. But I don't understand this thing with eval. How are you going to use it to work-around ES6 module issue?

If I put the code snippet with import() inside an eval, then TeaVM parser
will not prepend it with $rt_globals.
Alexey Andreev @.***> schrieb am So., 29. Okt. 2023,
06:47:

Yes, I as said, TeaVM only works with ES5. But I don't understand this
thing with eval. How are you going to use it to work-around ES6 module
issue?
—
Reply to this email directly, view it on GitHub
https://github.com/konsoletyper/teavm/issues/842#issuecomment-1784006185,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYCZ357HOMKZG2ONXHULFLYBXUXVAVCNFSM6AAAAAA6UCDAZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUGAYDMMJYGU
.
You are receiving this because you authored the thread.Message ID:
@.***>

I can suggest another workaround: you can write JavaScript code that imports everything you need and puts it into global objects, than access these objects from TeaVM.

Yes. I'm thinking of that exactly :)
For the context, assume I want to write an independent Java library on top
of TeaVM, not a one-time thing for my current project. Hence everything
should be dynamic and modifying index.html is not an option.
Alexey Andreev @.***> schrieb am So., 29. Okt. 2023,
06:58:

I can suggest another workaround: you can write JavaScript code that
imports everything you need and puts it into global objects, than access
these objects from TeaVM.
—
Reply to this email directly, view it on GitHub
https://github.com/konsoletyper/teavm/issues/842#issuecomment-1784007489,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYCZ32OOJCWHEBUFVRPZRDYBXV7TAVCNFSM6AAAAAA6UCDAZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUGAYDONBYHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

One working example is this:
private static final String tstfy =  """
        const doc = window.document;

        var head = doc.head;
        var script = doc.createElement('script');
        script.type = 'module';
        script.innerHTML = 'import Toastify from "https://esm.sh/toastify-js@1.12.0";Toastify({text: "This is a test toast",duration: 3000,}).showToast();console.log("Loaded ToastifyJS module");';
        script.async = false

        // Fire the loading
        head.appendChild(script);
        //window.Toastify({text: "This is a test toast",duration: 3000,}).showToast()
    """;

    @JSBody(script = tstfy)
    private static native void callToastify();


Added information/examples about input checking

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

hey @jhoard08, this looks pretty good but it could use a little modification - if you could remove the more information area and the this is a stub area, we could get this merged - Also, it would be great if you could add an example for the other two methods you mentioned. Thanks

This looks good to me.  Thank you for your contribution!


[TA] fhir sample draft

API change check
API changes are not detected in this pull request.

Closing for now, might bring this back later.


Style admin pages

Admin pages are a mess 😅

How do you navigate to this page? I haven't found any links except the one that is up on the navbar.

These are links 😅 It's still a huge WIP

@brendt I cannot find these links either, there is also nothing that points to some sort of page in the routes file.

It's because it's WIP (Work in progress). It's on a development branch

Exactly, that's the branch :) I'll probably merge it today though


Add python packages for fake-useragent, scrapy-fake-useragent, scrapy-deltafetch, scrapy-splash

Motivation for this change
Add several scrapy-related packages
Things done

Add fake-useragent
Add scrapy packages

Add scrapy-fake-useragent
Add scrapy-deltafetch
Add scrapy-splash

All of the above scrapy packages currently fail to compile on master with python38Packages, due to testfixtures failing to compile with such.
Everything seems to compile fine using python37Packages / python3Packages.

All set.  Squashed and cleaned up, etc.
Let me know if there's anything else.


XIAO ESP32-C6 gives: "Unexpected error: Error while writing flash: Packet content transfer stopped (received 2 bytes)"

I'm new to the world of ESP32 and bought a Seeed Studio XIAO ESP32C6 with 4Mb flash. Tried to flash on Windows 11 (winver details below).

tried to flash "released Tasmota for 4MB Flash (english)" using WebInstaller  but that was hanging at 15% complete
tried to flash this Tasmota C6 image using ESP_Flasher but this gave the error msg in title

I found a possibly related case here but couldn't make sense out of it as several topics got mixed there :(
Any suggestion how to get over this hurdle?
Details from Tasmota ESP_Flasher 3.0.3:
Console:
Using 'COM3' as serial port.
Unexpected error: could not open port 'COM3': PermissionError(13, 'Access is denied.', None, 5)

Progress window:
Using 'COM3' as serial port.
Connecting...
Detecting chip type... ESP32-C6
Connecting...

Chip Info:
 - Chip Family: ESP32
 - Chip Model: ESP32-C6FH4 (QFN32) (revision v0.1)
 - Number of Cores: 1
 - Max CPU Frequency: 80MHz
 - Has Bluetooth: YES
 - Has Embedded Flash: NO
 - Has Factory-Calibrated ADC: NO
 - MAC Address: F0:F5:BD:2C:F8:CC
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 1500000
Changed.
 - Flash Size: 4MB
Creating esp32c6 image...
Successfully created esp32c6 image.
Erasing flash (this may take a while)...
Chip erase completed successfully in 15.0s
Flash will be erased from 0x00000000 to 0x00003fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x0000e000 to 0x0000ffff...
Flash will be erased from 0x00010000 to 0x000d2fff...
Flash will be erased from 0x000e0000 to 0x002d9fff...
Compressed 13280 bytes to 9048...
Writing at 0x00000000... (100 %)
Wrote 13280 bytes (9048 compressed) at 0x00000000 in 0.2 seconds (effective 497.6 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 134...
Writing at 0x00008000... (100 %)
Unexpected error: Error while writing flash: Packet content transfer stopped (received 2 bytes)

Windows 11 details:

Update: My first unsuccessful attempt was with Chrome-based browser Vivaldi 6.9.3447.44 (Stable channel) (64-bit)
Success was achieved using Chrome 128.0.6613.138 (Official Build) (64-bit)

Maybe this little detail should be mentioned in ReadMe?

Suspected root cause of error:
I had left the Vivaldi Browser Window open before I started ESP_Flasher!
Likely WebInstaller window interferes with ESP_Flasher

Maybe this little detail should be mentioned in ReadMe?
Maybe the code could check for other processes accessing the same COM port as ESP_Flasher?

Clearly says could not open port 'COM3'

right, but from the message the necessary action to fix issue is not clear to a new user.
This is the part I tried to express via "Maybe this little detail should be mentioned in ReadMe?"


Bulk deletion of files when underlying storage disappears

Describe the bug
This might be better as a feature request, but when syncing a large amount of deletes it may be prudent to check that the storage can be accessed. Though the storage disappearing from underneath the client is probably rather edge case and not sure this could even be handled gracefully!
2021-10-17 06:03:27 sync INFO: Deleting 49773/49779...
2021-10-17 06:03:29 manager INFO: Up to date
2021-10-17 06:03:33 manager INFO: Syncing...
2021-10-17 06:03:33 sync INFO: Fetching remote changes
2021-10-17 06:03:35 sync INFO: Indexing 993...
2021-10-17 06:03:35 sync INFO: Applying deletions...
2021-10-17 06:03:35 sync INFO: Deleting 1/1...
2021-10-17 06:03:37 sync INFO: Deleting 8/337...
2021-10-17 06:03:39 sync INFO: Deleting 235/337...
2021-10-17 06:03:41 sync INFO: Deleting 110/345...
2021-10-17 06:03:43 sync INFO: Deleting 337/345...
2021-10-17 06:03:45 manager ERROR: Cannot create cache directory: Transport endpoint is not connected
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/maestral/sync.py", line 1144, in _new_tmp_file
    with NamedTemporaryFile(dir=self.file_cache_path, delete=False) as f:
  File "/usr/lib/python3.6/tempfile.py", line 690, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/lib/python3.6/tempfile.py", line 401, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
OSError: [Errno 107] Transport endpoint is not connected: '/data/Dropbox/Dropbox/.maestral.cache/tmp324zkcdb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/maestral/manager.py", line 791, in _handle_sync_thread_errors
    yield
  File "/usr/local/lib/python3.6/dist-packages/maestral/manager.py", line 636, in download_worker
    self.sync.download_sync_cycle(client)
  File "/usr/local/lib/python3.6/dist-packages/maestral/sync.py", line 2836, in download_sync_cycle
    downloaded = self.apply_remote_changes(changes)
  File "/usr/local/lib/python3.6/dist-packages/maestral/sync.py", line 3004, in apply_remote_changes
    for n, r in enumerate(res):
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/maestral/sync.py", line 3328, in _create_local_entry
    res = self._on_remote_file(event, client)
  File "/usr/local/lib/python3.6/dist-packages/maestral/sync.py", line 3384, in _on_remote_file
    tmp_fname = self._new_tmp_file()
  File "/usr/local/lib/python3.6/dist-packages/maestral/sync.py", line 1155, in _new_tmp_file
    "Please check if you have write permissions for "
maestral.errors.CacheDirError: Cannot create cache directory: Transport endpoint is not connected. Please check if you have write permissions for /data/Dropbox/Dropbox/.maestral.cache.
2021-10-17 06:03:45 manager INFO: Shutting down threads...
2021-10-17 06:03:45 sync INFO: Sync aborted
2021-10-17 06:03:45 manager INFO: Paused

To Reproduce

Maestral pointed Remote storage
Disconnect storage from underneath the OS

Expected behaviour
Maestral to fail gracefully if it can't read/write to the Dropbox location that is configured
System:

Maestral version: 1.5.0
Python version: 3.6.9
OS: Ubuntu 18.04.6 LTS
Desktop environment: N/A - headless
PyQt version (for Linux GUI): N/A

Additional context
These are glusterfs bricks mounted via glusterfs 3.13.2. It's all pretty low performance and sketchy (couple of BananaPi boards, built when not many SBCs had Sata) and a x86 box based on an AMD Jaguar. It has been working really well, albeit a little slow due to the storage bottleneck since the early releases of maestral after dropbox decided to end support for non ext4 drives.
The only trouble is the rewind feature is either currently broken or can't deal with a changeset with 60,000+ files, so I'm waiting for support to roll it back (though I keep multiple offline external backups of the files as well).
Additional Additional context
Dropbox were pretty responsive and rolled back things to just before it happened, so no harm done. But it has run like this for a very long time, including with the official dropbox client before it got kneecapped. Now it's possible I've gotten lucky, but I know for sure at least a few times the storage layer has gone wonky and the client just bailed out. I just re-mounted and restarted the services. This is the first time I've encountered anything beyond a sync conflict and it was pretty soon after upgrading to 1.5.0.

After the rollback I got around to turning maestral back on. Before that I dug into the Filesystem and the gluster host it was mapped against was quite wonky. So the FS was kinda mapped, but kinda not.
2021-10-21 10:11:10 sync INFO: Up to date
2021-10-21 10:11:10 sync INFO: Indexing local changes...
2021-10-21 10:29:11 sync INFO: Syncing ↑ 1/1
2021-10-21 10:29:11 manager INFO: Up to date
2021-10-21 10:29:12 manager INFO: Syncing...
2021-10-21 10:29:14 sync INFO: Syncing ↑ 1/2
2021-10-21 10:29:15 manager INFO: Up to date
2021-10-21 10:29:16 manager INFO: Syncing...
2021-10-21 10:29:16 sync INFO: Fetching remote changes
2021-10-21 10:29:16 sync INFO: Indexing 1...
2021-10-21 10:29:16 sync INFO: Syncing ↓ 1/1
2021-10-21 10:29:17 sync INFO: Up to date
2021-10-21 10:29:17 manager INFO: Up to date

If this is well in "Won't Fix" territory, I don't mind if it's closed. If it happens again I might dive into the deletion code and see if there is a way to guard against it :slightly_smiling_face:

Closing since this is indeed a very niche issue and in the Won't Fix territory for me.


"No more data to read" for some input level 4 (PACK+ROLZ) w/ 32m+ buffers

kanzi-cpp commit 6b57226e8e2168e5d2c0dadda505fe5f747bc99a (2024-10-19) gives error
No more data to read in the bitstream. Error code: 13
when attempting to decompress a stream created by compressing specific test data with level 4 (or just -t PACK+ROLZ -e NONE) using a buffer size of at least 32m.  This combination of options works fine for test data from e.g. /dev/zero or /dev/urandom but not for my specific log file.  I discovered the bug using a 5+ GiB log file today but can still reproduce it using just the first 32 MiB of the file.  The attached file contains these 32 MiB compressed down to 744448 bytes with kanzi -c -b 64m -t PACK -e TPAQX (the most efficient packing of this kind of highly repetitive log, a factor 1.25 smaller than using level 9 defaults).  The attached test file decompresses fine:
$ kanzi -d < kanzi-l4_b32m_error13_no_more_data.log.knz|wc -c
33554432

$ kanzi -d < kanzi-l4_b32m_error13_no_more_data.log.knz|sha256sum
162ca6bcb213184289b6a5076fcc37b34fdeda60b7a87b4b6bca442e5ce59628  -

$ kanzi -d < kanzi-l4_b32m_error13_no_more_data.log.knz|(head -3;echo ...;tail -3);echo
8 7 11 4 3 6 10 2 9 12 5: Qf= 99.42, sd=  1.15, d4=  2.00, Qc= 99.29, s=0.085, Qo= 90.24, P=12, S=900, N=66, Qf1av= 97.22, Qf1max= 98.17
8 3 11 4 7 6 10 2 9 12 5: Qf= 99.42, sd=  1.15, d4=  2.00, Qc= 99.29, s=0.085, Qo= 90.24, P=12, S=900, N=66, Qf1av= 97.19, Qf1max= 97.94
8 3 7 4 11 9 10 2 6 12 5: Qf= 99.42, sd=  1.15, d4=  2.00, Qc= 99.29, s=0.085, Qo= 90.24, P=12, S=900, N=66, Qf1av= 97.17, Qf1max= 97.82
...
8 3 7 6 12 9 11 4 2 5 10: Qf= 98.65, sd=  1.67, d4=  2.31, Qc= 98.53, s=0.122, Qo= 90.24, P=12, S=900, N=66, Qf1av= 96.20, Qf1max= 97.47
8 3 7 6 12 4 2 11 10 5 9: Qf= 98.65, sd=  1.67, d4=  2.15, Qc= 98.53, s=0.122, Qo= 90.24, P=12, S=900, N=66, Qf1av= 96.20, Qf1max= 97.47
8 3 7 6 12 10 2 4 5 11 9: Qf= 98.65, sd=  1.67, d4=  2.09, Qc= 98.53, s=0.122, Qo= 90.24, P=12, S=900, N=66, Qf1av= 96

But if you then recompress and decompress it like this, it fails:
$ kanzi -d < kanzi-l4_b32m_error13_no_more_data.log.knz|kanzi -c -t PACK+ROLZ -e NONE -b 32m -j 1|kanzi -d -v 5 -j 1 -o none

Kanzi 2.3.0 (c) Frederic Langlet

Fast lossless data compressor.
Compiled with clang version 16.0 - AVX2
Verbosity: 5
Overwrite: false
Using 1 job
Input file name: 'STDIN'
Output file name: 'none'

Decompressing STDIN ...

{ "type":"DECOMPRESSION_START", "size":0, "time":2150 }
Bitstream version: 6
Block checksum: NONE
Block size: 33554432 bytes
Using no entropy codec (stage 1)
Using PACK+ROLZ transform (stage 2)

{ "type":"BLOCK_INFO", "id":1, "offset":-1, "skipFlags":00111111 }
{ "type":"BEFORE_ENTROPY", "id":1, "size":1698481, "time":2484 }
{ "type":"AFTER_ENTROPY", "id":1, "size":1698477, "time":3616 } [2 ms]
{ "type":"BEFORE_TRANSFORM", "id":1, "size":1698477, "time":3623 }
No more data to read in the bitstream. Error code: 13

This was with a binary compiled with clang 16.0.4 but also happens with a gcc 12.2.1 compiled one.  Both produce exactly the same output (same sha256sum) for level 4 (32m) compression of the test data.
The error does not occur with buffer size 8m or 16m.  Also doesn't occur with PACK alone or ROLZ alone.  Or as mentioned with random or pure-zeros input test data.
Tested on: Fedora 36 Linux on AMD Ryzen 9
Built with (for the clang 16.0.4 build, the fastest on this platform among 7 different clang/gcc builds):
make CXX='clang++ -flto -Rpass=vector' CXXFLAGS='-c -std=c++14 -Wall -Wextra -O3 -fomit-frame-pointer -DNDEBUG -pedantic -march=haswell -fno-rtti' LDFLAGS='-fuse-ld=lld -lpthread' kanzi
Thanks for your great work, this is a very promising program!
PS: Attachment wrapped in 'gzip -1' because GitHub refuses to accept .knz files.  With only 'gzip -9' instead of kanzi+gzip, the compressed log excerpt would be 2.7 times larger, 1986 KiB instead of 727 KiB!
kanzi-l4_b32m_error13_no_more_data.log.knz.gz

Thank you for the report.
I am able to reproduce the issue. Here are a few quick notes:
cat /tmp/1 | ./kanzi -c -t PACK+ROLZ -e NONE -b 32m -j 1 -o /tmp/2 -f
Compressed STDIN: 33554432 => 1698506 (5.06%) in 216 ms
md5sum /tmp/2
9e0a1aadcfec3b994088b11680d4202d  /tmp/2
cat /tmp/2 |  ./kanzi -d -j 1 -o none
No more data to read in the bitstream. Error code: 13

cat /tmp/1 | ./Kanzi -c -t PACK+ROLZ -e NONE -b 32m -j 1 -o /tmp/2 -f
Compressed STDIN: 33554432 => 1698506 (5.06%) in 316 ms
md5sum /tmp/2
9e0a1aadcfec3b994088b11680d4202d  /tmp/2
cat /tmp/2 | ./Kanzi -d -j 1 -o none
Decompressed STDIN: 1698506 => 33554432 in 95 ms

The C++ and Go compressors generate the same file. The Go decompressor works, so the bug is in the C++ decompressor. The issue is already present in the 2.3.0 release. I will have to dig.

It should be fixed as of commit 15c7b1d.

Thanks for the fix.  I can confirm that it solves the problem.  kanzi-cpp can now decompress the entire 5.1 GiB log file after compression with kanzi -c -l 4 -b 32m as well as after 37 other more or less interesting combinations of compression levels/algorithms and buffer sizes.
(Due to the log being pre-sorted, it is highly sensitive to block size, with just 8m being optimal for level 9 -- compressed size 153 MiB -- and 128m level 9 blowing up to 402 MiB, much larger than even the 234 MiB for zstd -19; best kanzi is 136 MiB for -t PACK -e TPAQX -b 64m while zpaq -m57 (128 MiB buffers better than smaller here) shrinks it to just 118 MiB at the cost of more than 10x the CPU usage of the 136 MiB kanzi).
I should of course have created this issue in the kanzi-cpp project instead of here.  But I realized my mistake too late.

Thanks for confirming the fix and providing more details.
It is clear to me that the default levels work well in general but for specific data, nothing can beat dedicated transforms and special combinations of transforms and entropy which is why I build kanzi this way. The best option (depending of the chosen target ratio/speed) in these scenarios is really to try different combinations as you did.
As for zpaq, it is currently too slow for my taste. Maybe in the future, CPUs & memory will be fast enough but I preferred to restrict my PAQ implementation (levels 8 & 9) to something "usable" at the cost of compression ratio.

Right, I also think zpaq is way too slow.  I'm grateful that you created kanzi as a much more practical alternative for good compression at quite reasonable speed.


Match Keyboard Animation Curve

The animation when the keyboard appears and hides can be improved by grabbing the exact animation curve from the NSNotification and using it during the frame animation.

Thanks very much for this enhancement. As far as I can tell, it was previously animating with the correct curve, I guess because it inherits the correct curve from the in-flight animation. But it's good to make this explicit. Thanks again.


can update barrierDismissible from page

I make app that open this modal. page are consist pageUpdateProfile, pageSetting (barrierDismissible = true). Sometime when page expired token, i push new page called pageExpiredToken in this modal, i want user cannot hide by tap barrier. the only way to dismissible is by tap button close in the page in this modal, so i need to update barrierDismissible = false. but if token is not expired while other page is opened, this barrierDismissible is true. is this feature exist?

Hi, @developer-medan 👋
Currently, when the route is possible, it is not straightforward to update the barrierDismissible  feature. However, I think this might work and let me know if it does:
WoltModalSheet.show(
  context: context,
  modalDecorator: (child) {
    // Wrap the modal with `ChangeNotifierProvider` to manage the state of
    // the entire modal. You can use any state management solution that you like.
    return ChangeNotifierProvider<MyViewModel>(
      builder: (_, __) => MyViewModel(),
      child: child,
    );
  },
  modalTypeBuilder: (context) {
   // here you watch for the view model changes. When the state updates because the
   // isTokenExpired value changes, it will rebuild the modal.
   bool isTokenExpired = context.watch<MyViewModel>().state.isTokenExpired;
   WoltModalType.bottomSheet().copyWith(barrierDismissible: isTokenExpired),
  }
);


Email listing does not return all fields

Hi all,
Following the instructions to retrieve a message list does not include essential details like from email From: and To: fields. It seems like the "friendly names" are included, but these of course are not email addresses.
The following code lists my Inbox messages, but the getToRecipients() is always 0 elements; the getFrom().getAddress() is always null; and getInternetMessageHeaders() is also null.
How can I specify the fields I want in the result?
private static void checkInbox( ) throws Exception{
	ExchangeService service = new ExchangeService( ExchangeVersion.Exchange2010_SP2 ); //"<IP_ADDRESS>"
	service.setUrl( new URI( "https://webmail.example.com/ews/exchange.asmx" ) );
	service.setCredentials( new WebCredentials( USERNAME, PASSWORD ) );
	Folder inbox = Folder.bind( service, WellKnownFolderName.Inbox );
	ItemView top1000 = new ItemView( 1000 );
	FindItemsResults<Item> result = inbox.findItems( top1000 );
	for ( Iterator<Item> m = result.iterator(); m.hasNext();  ){
		Item peek = m.next();
		if ( peek instanceof EmailMessage ) {
			EmailMessage msg = (EmailMessage) peek;

			System.out.println( "\nMessage   " + msg.getInternetMessageId() );
			System.out.println( "Date:       " + msg.getDateTimeReceived().toGMTString() );
			System.out.println( "From:       " + msg.getFrom().getName() );
			System.out.println( "To:         " + msg.getDisplayTo() );
			System.out.println( "Recipients: " + msg.getToRecipients().getCount() );
			System.out.println( "Subject:    " + msg.getSubject() );

		}
		else
			System.out.println( "\nEncountered a non-email message." );
	}
}

You'll probably need to configure the PropertySet you want to retrieve on the ItemView object.

I would appreciate if you could show how that is done. I did not have very good luck with taking out the PropertySet, adding more ItemSchema.* properties to it, and then putting it back in the ItemView.

There's an example in the Getting Started Guide.

Thank you. That is the original example I was following. I did not have any luck with the following customization of PropertySet to try to return the elements I want (e.g. From: and To: email headers)
I would appreciate if you could show how that is done. I did not have very good luck with this:
...
	PropertySet fields = new PropertySet();
	fields.add( ItemSchema.InternetMessageHeaders );
	top1000.setPropertySet( fields );
	FindItemsResults<Item> result = inbox.findItems( top1000 );

...

you can use PropertySet fields = new PropertySet(ItemSchema.Id,EmailMessageSchema.From, EmailMessageSchema.ToRecipients) (extend this list with the additional properties you want to load) also a taken hint from msdn

To get the message headers, the EWS schema exposes the InternetMessageHeaders first-class property. Unfortunately, this property does not appear to return a complete set of message headers. For example, in Exchange 2010 Service Pack 1 (SP1), the InternetMessageHeaders property doesn't return address headers such as From and To


Cannot run both rails server and guard together

Running both guard and rails server together, forces atleast 1 of them to quit because of notiffany
Stack trace for guard
/Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/notiffany-0.1.0/lib/notiffany/notifier.rb:194:in `_activate': Already connected (RuntimeError)
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/notiffany-0.1.0/lib/notiffany/notifier.rb:87:in `initialize'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/notiffany-0.1.0/lib/notiffany/notifier.rb:42:in `new'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/notiffany-0.1.0/lib/notiffany/notifier.rb:42:in `connect'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/guard-2.14.0/lib/guard/notifier.rb:11:in `connect'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/guard-2.14.0/lib/guard.rb:61:in `setup'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/guard-2.14.0/lib/guard/commander.rb:32:in `start'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/guard-2.14.0/lib/guard/cli/environments/valid.rb:16:in `start_guard'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/guard-2.14.0/lib/guard/cli.rb:122:in `start'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/guard-2.14.0/lib/guard/aruba_adapter.rb:32:in `execute'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/guard-2.14.0/lib/guard/aruba_adapter.rb:19:in `execute!'
    from /Users/ronyvarghese/.rvm/gems/ruby-2.3.1@stack_track/gems/guard-2.14.0/bin/_guard-core:11:in `<main>'

Note: Works for webrick rails server. Fails for Puma

I'd be happy to fix this.
Can you create a minimal example where this fails? (It would help me fix this quickly and release as soon as I do).

This same issue just started for me.  It was caused by a duplicate entry in the Guardfile for guard :teaspoon....  I figured this may help others with a similar mistake.

Can someone create a small repo to reproduce this?
(That way I can work out where both connections are happening and which one should be used, etc).

Hi, I've stumbled across this, also with puma and teaspoon. I've put together a minimal repo as suggested at https://github.com/ianfleeton/notiffany-issue-24
Steps to reproduce (each a commit in the above repo):

rails new notiffany-issue-24 --skip-test-unit
Add guard-rspec to Gemfile
bundle exec guard init rspec
Add guard-teaspoon to Gemfile, in :development, :test
Add gem teaspoon-jasmine to :development, :test
rails generate teaspoon:install
bundle exec guard init teaspoon
Add gem phantomjs to Gemfile, in :test

Run either rails s or bundle exec guard one after another; either will quit the other running before it.

@e2 not sure if you'd seen my comment with the repro repo? Cheers!


[Bug] [UI] Right-clicking the workflow does not respond

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
1.Select a workflow with many tasks.
2.Modify && save && online.
3.Enter the workflow, select a task, and right-click without responding.
This problem was occasional, and the video screen was shown to @lenboo at that time.
4.Format the DAG, select the "grid" layout ("dagre" layout is not OK), save, and right-click to use again.
What you expected to happen
At any time, the right button can be used normally.
How to reproduce
See "What happened"
Anything else
No response
Version
2.0.3
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I didn't reproduce this problem, please confirm


Random access violations problems with VS

The executables produced by VS in Release configuration using either of the rasterizers and SIMD configurations randomly crash on startup because of a memory access violation.
I suspect an uninitialized variable somewhere.

It could also be an alignment problem (less sure). I will investigate this.

Ok I found it, it was a pointer arithmetic error in mipmap calculations.


Initial bootstrap git branch should carry over to installed maverick

If bootstrapping from master branch, maverick-branch.conf should be set to master

This is a pain, should be fixed

Probably have to create a new fact based on the git branch and use that to set the revision during maverick bootstrap


Automatically handle 3rd party svelte imports

Which issue does this fix?

Closes #49
Describe the solution
🚨 Breaking Change 🚨
Svelvet now automatically handles importing 3rd party svelte components without making the user set up a snowpack.config.js.
If you want to override the default svelvet snowpack config, you can do so by adding --snowpack-config which will load your own snowpack.config.js OR custom config via --snowpack-config ./mysnowpackconfig.js.
Why is it a breaking change?
For all people who've added a custom snowpack.config.js file, you need to update your svelvet npm cmds to include the new option --snowpack-config so your config will be loaded instead.
TODO

[ ] add docs
[ ] release as breaking change
[ ] update release info with migration notes (just add --snowpack-config)
[ ] support dev mode, hydration mode, etc... in rollup config https://github.com/jakedeichert/svelvet/issues/69#issuecomment-602098919

👋


Allow unnamed queries with variables

Codecov Report

Merging #21 (ff78197) into main (951bf64) will increase coverage by 0.67%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #21      +/-   ##
==========================================
+ Coverage   97.01%   97.69%   +0.67%
==========================================
  Files           7        7
  Lines         134      130       -4
==========================================
- Hits          130      127       -3
+ Misses          4        3       -1

Impacted Files
Coverage Δ

pygraphic/_gql_query.py
100.00% <100.00%> (+3.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 951bf64...ff78197. Read the comment docs.


crd-generator: option to set where CRDs should be generated

By default the crd-generator generates the yaml files in target/classes/META-INF/fabric8 but it would be nice to have an option to specify where they should be generated so as example, one can check in the CRDs to the repository

There's a PR opened for this in the Quarkus extension: https://github.com/quarkiverse/quarkus-operator-sdk/pull/8. This requires the soon-to-be-released 5.2 version of the fabric8 client.

@lburgazzoli the CRD generation process doesn't really allow much flexibility in the way of where the files can be generated, unfortunately. The quarkus extension offers some flexibility but still is limited to outputting generated code in target. Closing this as won't fix, unfortunately, but more like can't fix… :(


Enable exploration of Thanos Store Gateway by (external) label pairs

Is your proposal related to a problem?

I help manage a deployment of Thanos with a most of the normal components (Store GW + Compactor + Querier + Query FE). We are handling blocks coming from several clusters (hundreds of them) into a single "observability" cluster. When trying to debug memory usage for a particular dashboard or query it is quite hard to figure out which Store Gateway holds blocks for  a specific label set (e.g. `cluster_name="app-cluster-foo") because you have to go into each store gw and check the blocks for it in the UI. To provide more context into this, we are "hashmodding" the blocks from different clusters, so we can evenly distribute load, so this is one of the reasons we can't predict where a block will land.
Describe the solution you'd like

I would like to have an option to deploy a single binary which interacts with the Thanos Stores and is able to create an overview of how blocks are distributed and some summary information like which timespan is also represented in each Store-gw.
Describe alternatives you've considered

After trying to find a tool online and in the Thanos project repos, I ended up writing a tool of my own (just a draft, but already in a working state), because there was nothing available that would satisfy these requirements.
Additional context

The tool I mentioned is a simple Golang appconnects to each store gw via HTTP API and hit /api/v1/blocks?loaded=true and generates summaries in memory of the blocks, then we have a SPA written in react to show this data.

I love this idea in general. The question now is the interface to expose this. Whether it is a UI, API, or a cli tool.
From my perspective, a cli tool to fetch /api/v1/blocks?loaded=true from multiple store gateways are good enough.

I could write an CLI from that data I already have on my "server" that communicates with all stores, I will try to come up with something. The only problem here is that you always have to kubectl exec into some thanos pod to use the cli tool, whereas with a web UI it could be deployed as part of whole deployment.

I think extending the existing web tooling would be great. For example, adding a new Stores web command here: https://thanos.io/tip/components/tools.md/

Still relevant

/remind me 28/11/2022


Deploy with Zero Trust Networking Principles Guide

Overview/Summary
As part of adding Zero Trust guidance to the Landing Zone reference implementation, this PR adds an article describing what variables to update in order to deploy with Zero Trust principles for networking.
This PR fixes/adds/changes/removes

Adds an article for "Deploying with Zero Trust Networking Principles" to the wiki
Updates the wiki to include this article as a 300 level item.

Breaking Changes
1.None.
Testing Evidence
No code change.
As part of this Pull Request I have

[X] Checked for duplicate Pull Requests
[X] Associated it with relevant issues, for tracking and closure.
[X] Ensured my code/branch is up-to-date with the latest changes in the main branch
[X] Performed testing and provided evidence.
[X] Updated relevant and associated documentation.

@jtracey93 For #27294

@jtracey93 for review/discussion.

/azp run unit


Should the wizard show "(Step 1/5)"?

I removed that functionality when I refactored the wizard in this PR: https://github.com/Microsoft/vscode-azuretools/pull/119 because it didn't seem worth the work at the time, but I'm open to re-implementing it if people liked it.
My problems with the previous implementation:

The numbers weren't necessarily consistent or accurate. For example, the 'createWebApp' wizard can show up in the middle of other wizards (like when you deploy), so step 1/5 in that wizard might actually be step 3/7 total
We have problems with longer placeholders getting cut-off and until VS Code fixes that issue I'm not sure the step count was the most important information to display

Agreed if it's re-implemented the steps must be done properly. Not sure it's a high priority, though.

Closing due to lack of feedback and the cost of "fixing" this isn't worth the benefit


⚠ Bofrost.be has degraded performance

In 931a456, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1263 ms

Resolved: Bofrost.be performance has improved in 91b1572 after 9 minutes.


⚠ Bofrost.be has degraded performance

In 3efab60, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1624 ms

Resolved: Bofrost.be performance has improved in 3efe93c.


For contextualbandits the params are not clear.

estimator = VowpalWabbitContextualBandit()
.setArgs("--cb_explore_adf --epsilon 0.2 --quiet")
.setParallelism(6)
.setSharedCol('hour_feature')
.setAdditionalSharedFeatures(['placementId_feature', 'countryId_feature','resolvedAdType_feature', 'isWifi_feature', 'deviceOsId_feature', 'deviceTypeId_feature' ])
.setFeaturesCol('features')
.setUseBarrierExecutionMode(False)
.setChosenActionCol('chosenAction')
.setLabelCol('cost')
.setProbabilityCol('probability')
My questions are:
(a) My context features are ['hour_feature', 'placementId_feature', 'countryId_feature','resolvedAdType_feature', 'isWifi_feature', 'deviceOsId_feature', 'deviceTypeId_feature']. What setSharedCol and setAdditionalSharedFeatures means? Can you provide the link to the docs which explains it?
(b) How do you combine context features and action features? For example, in Vowpal wabbit we can have two spaces U for context and A for action. You can combine it using the command -qUA
(c) How do you generate the model in a human-readable form which can be done using -inv in vowpal wabbit.?
Thanks,
Sai

@jackgerrits For viz

Standard SparkML puts all features into a single column (e.g. features) and expects either the ML algo or upstream components to perform feature crossing. Since VW has the capability to perform at-execution-time feature cross of high-dimensional features, the API was extend to allow for multiple Spark vectors (e.g. sparse or dense) to be passed into the ML algo. (FYI performing feature cross ahead of time as SparkML is designed to, can be computational and space infeasible).
wrt to setSharedCol: specifically for contextual bandit, VW space optimized on how data was passed in for the typical scenario rank multiple items for 1 user. From an ML perspective the algo needs the features for each item and the feature of the user. Internally VW predicts for each item taking both the user and item features into account. To avoid duplicating the user features for each item the concept of "shared" features was invented. The user features are appended for each item.
wrt to setAdditionalSharedFeatures: each VW namespace is represented by a single Spark column backed by a SparkML vector (dense or sparse). the method allows to map additional spark columns to VW shared namespaces.
wrt to combining ontextual and action features: following VW convention, the first letter of the namespace is used when specifying quadratic features (e.g. -qUA). The Spark column name is used as VW namespace (it seems you have h, p, c, r, i, d,d)
wrt to human-readable model: since the VW estimator operates on SparkML vector (dense/spark) the knowledge of the input data is lost. This is a trade-off to enable standard SparkML featurizers. You might be able to recover the hashing by using the VowpalWabbitFeaturizer and mapping it to the input - mapping these values to feature offsets and weights might be an additional adventure. @jackgerrits any ideas on your end?

Yeah I think it would be an adventure for sure, but as you said the VowpalWabbitFeaturizer could potentially maintain a mapping which could be used to reconstruct the input from hashes. This sort of thing is always tricky because there are a number of components which are used to get the final feature/weight index.

the namespace names are "User" and "Action" so -q UA seems right (it's case-sensitive). user=Tom is a feature, not a namespace.
@jackgerrits  can you double check me?

The default features (actions) col is features (https://mmlspark.blob.core.windows.net/docs/0.9.5/pyspark/synapse.ml.vw.html#synapse.ml.vw.VowpalWabbitRegressor.VowpalWabbitRegressor), which is empty right now.
If you update the featurizer to output to features for action features and if the column names correspond to namespaces wouldnt' the required interactions be: -q ut -q ua -q ta?

Thanks, @jackgerrits @eisber.
"If you update the featurizer to output to features for action features and if the column names correspond to namespaces wouldnt' the required interactions be: -q ut -q ua -q ta?"
Yes, this is correct.
This is my understanding. Pls, correct me if I am wrong.
(a) There is no concept of Space as in VW. We have to pass the combinations eg -q ut etc.
(b) As mentioned in https://microsoft.github.io/SynapseML/docs/features/vw/Vowpal Wabbit - Overview/, the support is there for --bag, --cover, and --softmax.
(c) It seems like the current support is for only ips and not for other cb_types like dr, dm, and mtr.
(4) The weightCol  param in https://mmlspark.blob.core.windows.net/docs/0.9.5/pyspark/synapse.ml.vw.html#synapse.ml.vw.VowpalWabbitRegressor.VowpalWabbitRegressor.      corresponds to the weight of the data point. As this feature was not there for VW contextual bandit, the workaround was to divide the prob by weight as discussed in https://github.com/VowpalWabbit/vowpal_wabbit/issues/3472. I think this is resolved in the SynapseML support of contextual bandit.
(5) I will explore VowpalWabbitFeaturizer as mentioned by @jackgerrits and let you know the results.

Hai @jackgerrits
I have one more additional question. I am trying to build at least one workable model which I was able to build using VW.
My code is:
pipeline = Pipeline(stages=[
VowpalWabbitFeaturizer(inputCols=['hour'], outputCol='hour_feature'),
VowpalWabbitFeaturizer(inputCols=['placementId'], outputCol='placementId_feature'),
VowpalWabbitFeaturizer(inputCols=['isAudienceBiddingPlacement'], outputCol='isAudienceBiddingPlacement_feature'),
VowpalWabbitFeaturizer(inputCols=['chosenAction'], outputCol='chosenAction_feature'),
VectorZipper(inputCols=['chosenAction_feature' ], outputCol='features')
])
estimator = VowpalWabbitContextualBandit()
.setArgs("--cb_explore_adf --epsilon 0.2 --quiet --l2 10 --learning_rate 0.5" )
.setParallelism(6)
.setSharedCol('hour_feature')
.setAdditionalSharedFeatures(['placementId_feature', 'isAudienceBiddingPlacement_feature'])
.setFeaturesCol('features')
.setUseBarrierExecutionMode(False)
.setChosenActionCol('chosenAction')
.setLabelCol('cost')
.setProbabilityCol('probability')
I have one action column.
I am getting this error:
22/01/22 16:47:16 INFO CoarseGrainedExecutorBackend: Got assigned task 909
22/01/22 16:47:16 INFO Executor: Running task 1.1 in stage 41.0 (TID 909)
22/01/22 16:47:16 WARN VowpalWabbitContextualBandit: VowpalWabbit args: --cb_explore_adf --epsilon 0.2 --quiet --l2 10 --learning_rate 0.5 --holdout_off --span_server <IP_ADDRESS> --span_server_port 44611 --unique_id<PHONE_NUMBER> --total 4  --node 1 --no_stdin
22/01/22 16:47:16 WARN VowpalWabbitContextualBandit: Casting column 'cost' to float. Loss of precision.
22/01/22 16:47:16 WARN VowpalWabbitContextualBandit: Casting column 'probability' to float. Loss of precision.
22/01/22 16:47:16 INFO FilterCompat: Filtering using predicate: and(noteq(ecpmFloor, null), noteq(revenue, null))
22/01/22 16:47:17 WARN VowpalWabbitContextualBandit: Casting column 'cost' to float. Loss of precision.
22/01/22 16:47:17 WARN VowpalWabbitContextualBandit: Casting column 'probability' to float. Loss of precision.
22/01/22 16:47:17 WARN VowpalWabbitContextualBandit: Casting column 'cost' to float. Loss of precision.
22/01/22 16:47:17 WARN VowpalWabbitContextualBandit: Casting column 'probability' to float. Loss of precision.
22/01/22 16:47:17 WARN VowpalWabbitContextualBandit: Casting column 'cost' to float. Loss of precision.
22/01/22 16:47:17 WARN VowpalWabbitContextualBandit: Casting column 'probability' to float. Loss of precision.
22/01/22 16:47:18 WARN VowpalWabbitContextualBandit: Casting column 'cost' to float. Loss of precision.
22/01/22 16:47:18 WARN VowpalWabbitContextualBandit: Casting column 'probability' to float. Loss of precision.
22/01/22 16:47:18 INFO FileScanRDD: Reading File path: dbfs:/mnt/blobglobal/batchstore/secure/projects/data/dynamiy/2022-01-15-01/part-00000-581d5b70-14b4-4618-b892-7758d894d280-c000.snappy.parquet, range: 134217728-268435456, partition values: [empty row], modificationTime:<PHONE_NUMBER>000.
22/01/22 16:47:18 ERROR Executor: Exception in task 1.1 in stage 41.0 (TID 909)
java.lang.Exception: VW failed with args: --cb_explore_adf --epsilon 0.2 --quiet --l2 10 --learning_rate 0.5 --holdout_off --span_server <IP_ADDRESS> --span_server_port 44611 --unique_id<PHONE_NUMBER> --total 4  --node 1 --no_stdin
at com.microsoft.azure.synapse.ml.vw.VowpalWabbitBase.$anonfun$trainInternal$1(VowpalWabbitBase.scala:406)
at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)
at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)
at com.microsoft.azure.synapse.ml.vw.VowpalWabbitBase.trainIteration$1(VowpalWabbitBase.scala:346)
at com.microsoft.azure.synapse.ml.vw.VowpalWabbitBase.$anonfun$trainInternal$7(VowpalWabbitBase.scala:421)
at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:228)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:903)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:903)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:380)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:344)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:60)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:380)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:344)
at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
at org.apache.spark.scheduler.Task.doRunTask(Task.scala:153)
at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:122)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.scheduler.Task.run(Task.scala:93)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:824)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1621)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:827)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:683)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
at com.microsoft.azure.synapse.ml.vw.VowpalWabbitContextualBandit.$anonfun$trainRow$2(VowpalWabbitContextualBandit.scala:243)
at com.microsoft.azure.synapse.ml.vw.VowpalWabbitContextualBandit.$anonfun$trainRow$2$adapted(VowpalWabbitContextualBandit.scala:232)
at com.microsoft.azure.synapse.ml.vw.VowpalWabbitUtil$.prepareMultilineExample(VowpalWabbitUtil.scala:87)
at com.microsoft.azure.synapse.ml.vw.VowpalWabbitContextualBandit.$anonfun$trainRow$1(VowpalWabbitContextualBandit.scala:232)
at com.microsoft.azure.synapse.ml.vw.VowpalWabbitContextualBandit.$anonfun$trainRow$1$adapted(VowpalWabbitContextualBandit.scala:230)
at s

@jackgerrits  I took an example from the  test case from your repository and modified it as :
def get_data_two_shared():
# create sample data
schema = StructType([
StructField("shared_text", StringType()),
StructField("shared_text2", StringType()),
StructField("action1", IntegerType()),
StructField("chosenAction", IntegerType()),
StructField("label", DoubleType()),
StructField("probability", DoubleType())
])
    data = spark.createDataFrame([
        ("shared_f", "shared_f2", 1,  1, 1.0, 0.8),
        ("shared_f", "shared_f2", 2,  2, 1.0, 0.8),
        ("shared_f", "shared_f2", 2,  2, 4.0, 0.8),
        ("shared_f", "shared_f2", 1,  1, 1.0, 0.8),
        ("shared_f", "shared_f2", 2,  2, 1.0, 0.8),
        ("shared_f", "shared_f2", 1,  1, 1.0, 0.8),
        ("shared_f", "shared_f2", 2,  2, 1.0, 0.8),
        ("shared_f", "shared_f2", 2,  2, 4.0, 0.8),
        ("shared_f", "shared_f2", 1,  1, 1.0, 0.8),
        ("shared_f", "shared_f2", 2,  2, 1.0, 0.8),
        ("shared_f", "shared_f2", 11,  1, 0.0, 0.8)], schema) \
        .coalesce(1)

    # featurize data
    shared_featurizer = VowpalWabbitFeaturizer(inputCols=['shared_text'],
                                               outputCol='shared')
    shared_featurizer2 = VowpalWabbitFeaturizer(inputCols=['shared_text2'],
                                                outputCol='shared2')
    action_one_featurizer = VowpalWabbitFeaturizer(
        inputCols=['action1'], outputCol='action1_features')
    action_merger = VectorZipper(
        inputCols=['action1_features'],
        outputCol='features')
    pipeline = Pipeline(stages=[
        shared_featurizer, shared_featurizer2, action_one_featurizer,
         action_merger
    ])
    tranformation_pipeline = pipeline.fit(data)
    return tranformation_pipeline.transform(data)

featurized_data = get_data_two_shared()
estimator1 = VowpalWabbitContextualBandit()
.setArgs("--cb_explore_adf --epsilon 0.2 --quiet")
.setSharedCol('shared')
.setAdditionalSharedFeatures(["shared2"])
.setUseBarrierExecutionMode(False)
model1 = estimator1.fit(featurized_data)
I am not getting errors if the number of rows in data is less than 8. But more than 8 gives me the error is same as I shared in the previous chat.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 79.0 failed 4 times, most recent failure: Lost task 0.3 in stage 79.0 (TID 2316) (<IP_ADDRESS> executor 2): java.lang.Exception: VW failed with args: --cb_explore_adf --epsilon 0.2 --quiet  --no_stdin
Py4JJavaError                             Traceback (most recent call last)
 in
5     .setAdditionalSharedFeatures(["shared2"])
6     .setUseBarrierExecutionMode(False)
----> 7 model1 = estimator1.fit(featurized_data)
/databricks/python_shell/dbruntime/MLWorkloadsInstrumentation/_pyspark.py in patched_method(self, *args, **kwargs)
28             call_succeeded = False
29             try:
---> 30                 result = original_method(self, *args, **kwargs)
31                 call_succeeded = True
32                 return result
/databricks/spark/python/pyspark/ml/base.py in fit(self, dataset, params)
159                 return self.copy(params)._fit(dataset)
160             else:
--> 161                 return self._fit(dataset)
162         else:
163             raise TypeError("Params must be either a param map or a list/tuple of param maps, "

Hi, @jackgerrits and @eisber. Can you suggest any fix the above problems?

@musram  Jack and I are setting up a debugging env to repro the problem.

@eisber  and @jackgerrits, we were able to look at your source code and were able to pass the data as required by your framework.  We are not getting the above error.
But there is one issue we are facing when we pass chosenAction = 0 because of
val selectedActionIdx = row.getInt(chosenActionColIdx)
if (selectedActionIdx == 0) {
throw new IllegalArgumentException("Chosen action index is 1 based - cannot be 0")
}
https://github.com/microsoft/SynapseML/blob/865d189496f4958e07ca3178155063ddd3b908d0/vw/src/main/scala/com/microsoft/azure/synapse/ml/vw/VowpalWabbitContextualBandit.scala#L235
Is there any constraint that you restrict the chosenAction to take value >= 1?
Sai

The index is simply 1 to n vs 0 to n-1. you are still able to select all actions.

@eisber, yes I understand. We have to make a simple change in our data.
Thanks. I will close this issue.


implement SolutionMappingsHashTable

This data structure may serve as a basis for implementations of both the hash join algorithm and the symmetric hash join algorithm.

While this issue has, in principle, been addressed by PRs #35 #42 and#43, I would still like to clean up the corresponding classes a bit and add some documentation.

PR #52 has brought in the additional changes that I still wanted to have before closing this issue here. Now that PR #52 has been merged, we can close the issue.


Coherence and invalidation ordering

Hi Pawel -
We noticed we were getting occasional issues with the cache being incoherent. We are using EFCache.Redis, but I believe the issue would apply to any implementation of ICache. In ExecuteDbDataReader (and ...Async), for queries that are not cacheable, you execute the query and then invalidate the affected sets. If the invalidation fails, then the cache becomes incoherent.
I'd like to propose that you invalidate the sets first. This gives the cache implementor the option to throw if the invalidation fails, preventing the update/insert/delete from happening at all, ensuring the cache does not become incoherent.
Does this seem like a reasonable change? Am I overlooking anything? If it seems ok, I'll create a PR.
Thank you,
Jason Dufair

Indeed it is more complicated! Thanks for the detailed response. We are basically dealing with the challenge of a two phase commit if we do want to go the "cache everything" route.
For backends that support transactions (i.e. Redis), we could leverage this for cache coherence. The way we could do this is in ExecuteDbDataReader would be to:

Begin a cache backend transaction
Invalidate cache entity set members
Execute the DB query
Only commit the cache backend transaction if the query succeeds

The only remaining opportunity for inconsistency would be if the cache backend commit command fails. Which should be sufficiently rare (especially if it included a retry/backoff mechanism) that it could be monitored and handled out-of-process in one way or another. I believe other backends besides Redis (even in-memory) could support this scheme as well.
How would you feel about extending ICache to include the following members?
bool SupportsTransactions
void BeginTransaction
void EndTransaction
It wouldn't handle, i.e. nested transactions and it still wouldn't support DB transactions, but this would allow EFCache to offer exceptional consistency if one wanted to treat the cache as essentially a (much faster) distributed database.
See any issues with this approach?

Hi Pawel - just an update. I have implemented an approach that more or less takes this approach, but uses optional locking to ensure consistency and leaves it up to the implementor to use or not. I'll go ahead and close this issue. I'll have a PR sometime soon. Hopefully it's something you'll find useful.


Task: Add browser caching for resources

Overview
Closes #2369
Demo

When the first fetch is unsuccessful, and the cache instance is empty, this is displayed to the user:

Otherwise, the resources are displayed (served from DevX API or the cache when DevX API is down).

Notes
Optional. Ancillary topics, caveats, alternative strategies that didn't work out, anything else.
Testing Instructions

Checkout to this branch.
Run the local instance of GE.
Navigate to the Resources tab.
Observe changes.

We should remember to remove the local json files


stroom:source() is failing to return a value after processing part of a source stream

Not sure exactly when this is happening, it may coincide with a subs-stream boundary in an aggregated non XML source.

processing a large aggregated stream, stroom:source() stopped working after 988 records, roughly 1200000 chars into the stream, still on the first aggregate

The way stroom:source() works is that if you have a split count of say 100, the system remembers the source location for 100 records. When 100 records have been parsed the split filter fires an endDocument event that effectively tells the downstream XSLT that there is no more data coming so go ahead and do your transform on the DOM you have built in memory. For the stroom:source() function to work it then expects that function to be called exactly 100 times, once for each input record. After the transform is done it clears the memorised source locations and starts again with another 100....
If a pipeline does the source decoration downstream where some records are filtered out, 100 records in don't equal 100 records out so the function isn't called for each input record. Firstly this means the locations will be totally wrong as soon as you skip a record. Also as soon as it goes wrong for a batch it just stops bothering to try capturing source locations as they will all be wrong anyway.
The only way to fix this is to call stroom:source() for each record in your first XSLT in the pipeline or to process one record at a time, i.e. set a split count of 1. The latter might affect performance but it has been a while since we tried this so I'm not sure by how much.

May also relate to #1736


THORN-2343 correct implementation of fix for THORN-2276

[x] Have you followed the guidelines in our Contributing document?
[x] [v2] Have you created a JIRA and used it in the commit message?
[ ] [v4] Have you created a GitHub Issue and used it in the commit message?
[x] Have you checked to ensure there aren't other open Pull Requests for the same issue?
[x] Have you built the project locally prior to submission with mvn clean install?

CI server was down, had to restart it.
retest this please

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Thanks!