Work better with Haskell.nix?

Hi!
I tried to integrate two things new to me recently, Cabal-doctest and Haskell.nix.  As you might expect, Doctests are already pretty hacky when it comes to Cabal integration.  It only gets trickier when you throw in Nix.  Actually, the default way of building Haskell projects with Nix (Nixpkgs) didn't give me a problem with Doctests, even with Cabal-doctest.  But Haskell.nix, specifically, causes me issues.
I have a couple ideas on how we might improve the state of things.
Background on Nix and Haskell.nix
I don't know much you already know about Nix and Haskell.nix, but I figured it would be good to spell out some relevant context, even if for others finding this ticket.
In Nix, everything is cached at the level of a Nix derivation.  A derivation sets up a very hygenic/controlled environment, with all of the files/folders/permissions/envvars explicitly specified.  Then the crank is turned, and specific outputs are cached as the output of the derivation.  These can then be used to set up the environment for subsequent derivations.  This is important, because it means that transient build artifacts from a previous derivation are not just lying around for the next derivation.  They have to be explicitly copied over.
What Haskell.nix does that breaks the normal Cabal-doctest workflow is have a separate derivation for each component in a Cabal project.  So the libraries, executables, and test suites each get their own derivation.  Furthermore, there is even a separate derivation to run a test that is different from the one that builds the test code.  This is kind of nice because it gives developers more control.  A developer can make sure all these tests build without actually running the tests.  But then, because in Nix everything is cached at the derivation-level, we can run the tests later and use the same artifacts we cached previously.  Stack/Cabal users take this for granted, but in Nix we have to make Nix expressions to regain this flexibility, which is one of the reasons the Haskell.nix project exists as a alternative to the default way of building things with Nix (Nixpkgs).
What's going wrong with Haskell.nix + Cabal-doctest
Components are seen only one at a time
The code for Cabal-doctest assumes that most of the flags we need are derived from the library and executable components.  But in a Haskell.nix derivation for a test component, there are no library or executable components.  They have already been built and made available in the package database provided to the derivation's Cabal run.  However, the source code of these components can be made available (obviously needed for the authored doctests themselves).
Right now, Cabal-doctest when running in a Haskell.nix derivation won't output any flags, packages, or module sources.  Because these flags are output only for libraries and executables, which are not available when compiling the test component.  What would be useful is to get the at least the packages flags (possibly the rest too) from the test component.  The code I need to load up in my doctests have already been shuttled into a package database.  And I can access this package database for normal dependencies from my doctest.  This does mean to list dependencies I need for doctests on my test component, not just the dependencies needed for the doctest fixture.  But that seems like an okay compromise to me.
Some references to missing artifacts
All the references to build/ and dist/, but also Build_doctests aren't useful for a Haskell.nix derivation.  All of this stuff is in the package database.
Possible solutions (not mutually exclusive)
1. At least output flags for the test component
I made a patch of cabal-doctest that does this (not cleaned up for a PR yet, and not sure if you want to go this direction anyway), and it makes this following Build_doctests.hs file:
module Build_doctests where

import Prelude

data Name = NameLib (Maybe String) | NameExe String deriving (Eq, Show)
data Component = Component Name [String] [String] [String] deriving (Eq, Show)

pkgs_test_doctests :: [String]
pkgs_test_doctests = ["-package-id=base-<IP_ADDRESS>","-package-id=base-compat-0.11.0-BTru338SRTmL3dK24Mui3j","-package-id=doctest-0.16.2-6z3pUedJOHdCkF4jrPAXAU","-package=exceptions-checked-0.0.1"]

flags_test_doctests :: [String]
flags_test_doctests = ["-i","-i/build/exceptions-checked/dist/build/doctests/autogen","-i/build/exceptions-checked/dist/build","-i/build/exceptions-checked/test","-i/build/exceptions-checked/test","-package-env=-","-hide-all-packages","-clear-package-db","-package-db=/nix/store/ryxw8lj748jbmvriz3w8k4vw7brnann4-exceptions-checked-0.0.1-test-doctests-config/package.conf.d","-package-db=dist/package.conf.inplace","-optP-include","-optPdist/build/doctests/autogen/cabal_macros.h"]

module_sources_test_doctests :: [String]
module_sources_test_doctests = ["Build_doctests", "Doctest.Checked.Catch","Doctest.Checked.Throw"]

-- [NameLib Nothing]
components :: [Component]
components = []

2. Eliding problematic missing references
Although there's a few references to artifacts that are missing:

-i/build/exceptions-checked/dist/build/doctests/autogen
-i/build/exceptions-checked/dist/build
-i/build/exceptions-checked/test
-i/build/exceptions-checked/test
-package-db=dist/package.conf.inplace
-optPdist/build/doctests/autogen/cabal_macros.h
Build_doctests

The only ones that really cause my Haskell.nix build trouble are the two bolded above.  The Build_doctests file should be elided from module_sources_test_* always, I think.  If we did nothing with the missing dist/package.conf.inplace I can always do a ghc-pkg init dist/package.conf.inplace in my Haskell.nix configuration.  It's a small one-liner.
We could possibly make an option to elide the "inplace" package flag.  But we'd need this flag for non-Haskell.nix builds.  I'd much more prefer to have a static Cabal configuration and source code that works with all builds systems.  Small tweaks seem better on the Nix side than that Cabal side.
3. Nothing
I list this for completeness.  I kind of understand the argument that Cabal-doctest is complicated enough as it stands.  Is Haskell.nix breaking an invariant expected of Cabal?
I'm probably going to point some Haskell.nix developers at this ticket.  Maybe there's something about the Haskell.nix implementation that can be changed to allows all the components to show up in the LocalBuildInfo when Setup.hs runs.

I have to read this properly, but I have a quick comment:
cabal-install doesn’t use per-component build for build-type: Custom packages for related reasons: the custom Setup.hs can do virtually anything (particularly the Cabal hooks interface is not refined to support per component builds)
cabal-doctest is a hack, and I really really don’t want to make it more hacky. So option 3: Nothing is very tempting.

Okay, let me put that PR up anyway, and you can decide if it makes the code noticeably worse.  I'll go out of my way to not break any prior functionality.  It will just generate new things in Build_doctests.hs that it didn't before.


redo(ticdc): fix a bug that flush log executed before writing logs

What problem does this PR solve?

Issue Number: ref #5486
What is changed and how it works?
When calling FlushLog for a table, we must ensure that all data of this table has been written to underlying writer. Since the EmitRowChangedEvents and FlushLog of the same table can't be executed concurrently, we can insert a simple barrier data into data stream to achieve this goal.
Check List
Tests

Unit test
Integration test

Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
Fix a bug in redo log manager that flush log executed before writing logs

/run-all-tests

Codecov Report

Merging #5621 (baa485f) into master (fcea4d5) will increase coverage by 0.1451%.
The diff coverage is 59.4913%.

Flag
Coverage Δ

cdc
61.6194% <62.5199%> (+0.4713%)
:arrow_up:

dm
52.0101% <48.8826%> (-0.0327%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
@@               Coverage Diff                @@
##             master      #5621        +/-   ##
================================================
+ Coverage   56.0768%   56.2219%   +0.1450%
================================================
  Files           535        527         -8
  Lines         70143      69793       -350
================================================
- Hits          39334      39239        -95
+ Misses        27078      26821       -257
- Partials       3731       3733         +2

/run-all-tests

/run-all-tests

/merge


home assistant not starting after update to 0.92

Home Assistant release with the issue:

0.92
Last working Home Assistant release (if known):
0.91.3
Operating environment (Hass.io/Docker/Windows/etc.):

hass venv ubuntu 18.04 intel skull canyon NUC
Component/platform:

Description of problem:
after update from 0.91.3 -> 0.92 HA is running but doesn't generate a log file and i can't connect to ip:8123 ... i've restarted HA and rebooted but with the same result ... no idea what is wrong ?
dan@Home-Assistant:~$ sudo systemctl status<EMAIL_ADDRESS>●<EMAIL_ADDRESS>- Home Assistant
Loaded: loaded<EMAIL_ADDRESS>enabled; vendor preset: enabled)
Active: active (running) since Thu 2019-04-25 04:20:05 PDT; 6min ago
Main PID: 6226 (hass)
Tasks: 2 (limit: 4915)
CGroup<EMAIL_ADDRESS>└─6226 /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
Apr 25 04:20:05 Home-Assistant systemd[1]: Started Home Assistant.
Apr 25 04:20:05 Home-Assistant hass[6226]: 2019-04-25 04:20:05 INFO (SyncWorker_0) [homeassistant.config] Upgrading configuration directory from 0.91.3 to 0.92.0
Apr 25 04:20:05 Home-Assistant hass[6226]: 2019-04-25 04:20:05 INFO (SyncWorker_0) [homeassistant.config] Migrating google tts to google_translate tts
Apr 25 04:20:05 Home-Assistant hass[6226]: Config directory: /home/homeassistant/.homeassistant
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

# configuration.yaml
# SCCS: @(#)	1.12 04/25/19

homeassistant:
# Name of the location where Home Assistant is running
  name: Monte Verde
# Location required to calculate the time the sun rises and sets
  latitude: !secret lat
  longitude: !secret long
  elevation: 30
  temperature_unit: F
  unit_system: imperial
# pick your from here: http://en.wikipedia.org/wiki/List_of_tz_database_time_zones
  time_zone: America/Los_Angeles
  whitelist_external_dirs:
    - '/tmp'
# authentication
  auth_providers:
    - type: legacy_api_password
      api_password: !secret http_password
    - type: homeassistant
    - type: trusted_networks
      trusted_networks:
        - <IP_ADDRESS>/24
        - <IP_ADDRESS>/24
        - <IP_ADDRESS>/24
        - <IP_ADDRESS>
        - <IP_ADDRESS>

# load packages
  packages: !include_dir_named packages

# Show links to resources in log and frontend
#introduction:

# use yaml for lovelace
lovelace:
  mode: yaml

# custom updater and tracker
custom_updater:

# Enables configuration UI
config:

# enable system health
system_health:

http:
# Secrets are defined in the file secrets.yaml
#  server_port: 8123
#  ssl_certificate: /home/homeassistant/.homeassistant/certificate.pem
#  ssl_key: /home/homeassistant/.homeassistant/privkey.pem
#  ssl_certificate: /etc/letsencrypt/live/marotta.ddns.net/fullchain.pem
#  ssl_key: /etc/letsencrypt/live/marotta.ddns.net/privkey.pem
#  base_url: https://marotta.ddns.net:8123
#  ip_ban_enabled: true
#  login_attempts_threshold: 5

updater:
  include_used_components: true

# Discover some devices automatically
#discovery:

# limit home-assistant_v2.db size
recorder:
  purge_keep_days: 7

# iOS configuration
ios:

#Zones
zone:
  - name: Home
    latitude: !secret lat
    longitude: !secret long
    radius: 450
    icon: mdi:home

# Allows you to issue voice commands from the frontend in enabled browsers
conversation:

# Enables support for tracking state changes over time
history:

# View all events in a logbook
logbook:

# Enables a map showing the location of tracked devices
map:

# Track the sun
sun:

# Text to speech
tts:
  - platform: google

# Cloud
cloud:

#group: !include groups.yaml

Traceback (if applicable):

Additional information:

rolled back to 0.91.3 and all is well

How long did you wait after restarting HA?

5 minutes (boots quickly on intel skull canyon NUC) ... the important info is no log file is generated ... immediately, after restart i should be able to  see log file activity right away

Hi @bmf7777 !
I've just did it and it is working perfectly (on Docker).
I personally just did this in the configuration.yaml :
Delete :
# Show links to resources in log and frontend
introduction:

Rename  :
# Text to speech
tts:
  - platform: google

to
# Text to speech
tts:
  - platform: google_translate

Maybe try to change your tts component.

Did you check if any dependencies were being built? Sometimes it takes a while, so I would advise to try again and leave it alone for a while.

I have the same problem. I'll try to see if it's something in configuration.yaml, but it's not showing in loggging (or debug mode).

You have this log
Apr 25 04:20:05 Home-Assistant hass[6226]: 2019-04-25 04:20:05 INFO (SyncWorker_0) [homeassistant.config] Migrating google tts to google_translate tts

But the configuration you posted still use google tts. Any permission setting blocked HA write your configuration file? If so, you need manually change google to google_translate

i commented out below and all is well ! thanks, dan
# Text to speech
tts:
  - platform: google_translate


Adds NavigationActionPolicy to allow without try open app link

Connection with issue(s)
We need keep the website navigation in our webview when exist Universal Links.
searching a little on the internet i find some discursions about solve it like this. And there is a const to do it, its 3. You can see that in the original webkit code source:
static const WKNavigationActionPolicy WK_API_AVAILABLE(macos(10.11), ios(9.0)) _WKNavigationActionPolicyAllowWithoutTryingAppLink = (WKNavigationActionPolicy)(_WKNavigationActionPolicyDownload + 1);

Testing and Review Notes

Screenshots or Videos

To Do

[x] double check the original issue to confirm it is fully satisfied
[x] add testing notes and screenshots in PR description to help guide reviewers
[x] request the "UX" team perform a design review (if/when applicable)

Unfortunately, it's an iOS Private API, so it could lead to App Store rejection.
However, I think you can already manage to open these kind of links inside the webview using the shouldOverrideUrlLoading event.
For example, I have Twitter (X) App installed on my iPhone.
If I go to https://docs.flutter.dev/get-started/install and click on the Twitter (X) logo, it will open the Twitter (X) App.
To avoid that, I can do this in Flutter code:
  initialUrlRequest: URLRequest(url: WebUri('https://docs.flutter.dev/get-started/install')),
  shouldOverrideUrlLoading: (controller, navigationAction) async {
    final uri = navigationAction.request.url;
    if (uri.toString().startsWith('https://twitter.com')) {
      if (navigationAction.navigationType == NavigationType.OTHER) {
        // check for navigationType to be NavigationType.OTHER
        // in order to block infinite load request loop towards https://twitter.com
        return NavigationActionPolicy.ALLOW;
      }
      // load request again inside this webview
      controller.loadUrl(urlRequest: navigationAction.request);
      // and cancel the current request to avoid opening the installed Twitter (X) App
      return NavigationActionPolicy.CANCEL;
    }
    // allow all the other requests
    return NavigationActionPolicy.ALLOW;
  },

HI @pichillilorenzo !
Using shouldOverrideUrlLoading not work. The Universal Link is the same of link navigation in website.

Hi @pichillilorenzo !
This approach using shouldOverrideUrlLoading not work in IOS.

You can use my example and change it with your website url/domain.
Otherwise, post your code of inappwerbview and I will try to help you

So, I don’t know if you can do download this app. MercadoLivre.
If you have this app installed and navigate with this webview using https://www.mercadolivre.com.br It's load OK. But if you try do some search the webview it redirect to to show this search in the app. Because the link of search is https://lista.mercadolivre.com.br/ and it is a UniversalLink

Have you tried using something like:
  shouldOverrideUrlLoading: (controller, navigationAction) async {
    final uri = navigationAction.request.url;
    if (uri.toString().startsWith('https://lista.mercadolivre.com.br')) {
      if (navigationAction.navigationType == NavigationType.OTHER) {
        // check for navigationType to be NavigationType.OTHER
        // in order to block infinite load request loop towards https://lista.mercadolivre.com.br
        return NavigationActionPolicy.ALLOW;
      }
      // load request again inside this webview
      controller.loadUrl(urlRequest: navigationAction.request);
      // and cancel the current request to avoid opening the installed MercadoLivre App
      return NavigationActionPolicy.CANCEL;
    }
    // allow all the other requests
    return NavigationActionPolicy.ALLOW;
  },

otherwise, could you post your current shouldOverrideUrlLoading implementation?

Hi @pichillilorenzo !
https://github.com/pichillilorenzo/flutter_inappwebview/assets/10039275/525ea795-109a-4e79-84a8-eb407b1d2dd6
My shouldOverrideUrlLoading is equal you give me and not works. The flow happen in native code, if returns ALLOW  it will  open in installed app.

if possible, post all the logs that are coming from the InAppWebView.
You should see logs for each event called by InAppWebView.
Maybe https://lista.mercadolivre.com.br is not the only URL to Cancel and you need to check for other links.

I already check the other links and not there is. no deeplinks or urlScheme, just https://lista.mercadolivre.com.br.....

Same issue here, any progress @RafaelBarbosatec?
I check if the link starts with "http" or "https" as well before navigating, but, when using IOS, "https://www.mercadolivre.com.br/" always redirects to the app if i try to create an account.

Yeah. I notified the package developer but he think that it's not happening.
I did a fork and fix it. In the moment I'm using my fork.

if possible, post all the logs that are coming from the InAppWebView.
Without it, I can’t help you.


Convert localization.js to use UMD format

Sorry for all the code style changes, forgot to turn off auto-formatting.  I can fix that, but all the important  bits are at the top/bottom.

Merged it into develop. I think the formatting is fine. Just had no time yet to spare the Lang.addMessages and just export a JSON object yet ;)


In no animation mode, click on overlay pass through

Hi,
I have driver.js 0.7.1
When I trigger a multistep tutorial  with no animation mode, I can click on buttons, links ... etc behind the overlay

That is a known issue. To tackle the z-index stacking context issues, we don't have the overlay in the non-animated version and it is achieved by simply putting an outline around the stage, which does not block the clicks and thus clicks are passed through.
We could use focus-trap, for example, to achieve that behavior but it is intentionally left-out to avoid the bloat. You can however use it in one of the available hooks to achieve that.


Check all messages beginning with $...

... and then if it is a valid command. If it is, continue, if not, say it's not.

On hold until the code is fully optimized and running at maximum speed (for my internet) and efficiency. This would require much faster running code to still be efficient enough to use.

See #53


how to get difference between two volumetric meshes to calculate the deformation distance for each node

I have created a volumetric mesh using gmsh and then exported it as a .vtk file format (i.e. as an unstructured grid). Then fed this into a vtk program and add different deformations (i.e. scaling, rotate, translation) to this unstructured grid and obtain 100 deformation instances.
Now I need to obtain the displacement vector for each node of the reference geometry (i.e. initial volumetric mesh) compared to each of these deformed instances.
For example, I wrote a code as below in meshio package in order to calculate this displacement for a one particular deformation instance.
My question is that, are the nodes appearing in the same order after deforming the initial geometry (i.e. reference unstructured grid) so that we can obtain by simply subtracting as the below step ?
reference_ellipsoid_file = './Generated_data/reference_geometry/ellipsoid.vtk'
reference_ellipsoid = meshio.read(msh_ellipsoid_file)

deformed_ellipsoid_file = './Generated_data/deformations/dataset/meshes/box_with_ellipsoid0.vtk'
deformed_ellipsoid = meshio.read(deformed_ellipsoid_file)

displacement = deformed_ellipsoid.points - reference_ellipsoid.points

Hi,
It mostly depends on the VTK program you are using to produce your deformed meshes, not meshio as it only reads/writes mesh files as is.
One way to find out would be to check that your deformed and reference meshes have the same cells attributes:
for c1, c2 in zip(deformed_ellipsoid.cells, reference_ellipsoid.cells):
    assert c1.type == c2.type
    assert numpy.allclose(c1.data, c2.data)


EnderIO-JEI intergration crush

What happens:
Server crashed after clicking item in JEI item when Inventory Panel Remote (or any other Inventory panel) GUI opened in hand.
Crashes both with jei sync on and off.
What you expected to happen:
Steps to reproduce:
1.Open inventory panel (any)
2.Try to get recipe for an item in JEI  by clicking on any item.
3.
...

Affected Versions (Do not use "latest"):

EnderIO:EnderIO-1.10.2-<IP_ADDRESS>_beta
crash-2016-08-21_13.57.49-client.txt

EnderCore:EnderCore-1.10.2-<IP_ADDRESS>-beta

Minecraft: 1.10.2

Forge:(forge-1.10.2-<IP_ADDRESS>3
-jei_1.10.2-<IP_ADDRESS>(tested also with JEI 244)

Your most recent log file where the issue was present:
[pastebin/gist/etc link here]

update

lol didn't noticed that EnderIO wasn't updated thx


Create Model for PurchaseItem

A PurchaseItem is used to record an instance of when an Item is purchased the schema for this is also used for a WantsItem which is the draft used to create a PurchaseItem. A PurchaseItem Contains the following fields

name: name of the item
id: system generated id of the item
type: type this is the PaymentMethod planned for this Purchase
qpm: this is the number of times this item purchased per a budget this if 1 it would be allocated to the first section if more than one a user needs to select one or more sections for the item
category: This is the LogCategory this item belongs to
price: this is the price as planned for this item
location: this is the planned location of purchase for this item

seeing as this will also be used for the WantsItem model two sets of tests will be described
Wants Tests

[ ] Create a WantsItem if name is provided
[ ] Update a WantsItems name, qpm, price,location, type
[ ] Delete a WantsItem

Purchase Tests

[ ] Convert WantsItem to PurchaseItem if 1 qpm then select a Logsection for the item the default being the first  if more than 1 select 1 or more sections for the item(s)
[ ] Update PurchasItems name, qpm, price, location, type  provided the MoneyLog isn't refined

WantsItems and PurchaseItems are  the same things really it's just that a purchaseitem has been l assigned to a budget section


Unhandled Promise Rejection

@whiteyhat The issue is that the swift module is not being properly imported. This seems to be a result of changes in XCode 10.2, as discussed here: https://github.com/facebook/react-native/issues/24139
Still working on a fix, will keep you updated.

Since the Blockstack can't be evil hackathon was giving voucher if the hackathon app was uploaded before the 25th, and as the minimum requirement, it should as support blockstack auth. But as described above, the SDK itself has compiling issues.
What can I do to still being eligible if this does not depend on me?

@whiteyhat This will hopefully be fixed within the day, so it should not be blocker for your entry. In the mean time, you can use the iOS 12.1 simulator instead of 12.2. This should make things work immediately.
A more permanent fix will come as part of react-native v0.60.0 (relevant commit: https://github.com/facebook/react-native/commit/ff66600224e78fec5d0e902f8a035b78ed31a961). I'm now attempting to configure npm to use the 0.60-stable branch directly from github for the react-native dependency, once I get this working the issue should be resolved.

Nice, ping me back when that is done so I can implement blockstack auth seamlessly

I hit a snag yesterday—once I got it using branch ‘0.60-stable’ I found that they have changed how the subspecs are defined and imported, causing errors when doing pod install. Still figuring out how to import those subspecs (i.e. RCTWebSocket, etc.). I’ll ping you as soon as the fix is ready!
Again if you wish to be unblocked immediately add the iOS 12.1 simulator in XCode. Thanks and hang on 😄.

Okay, but this will affect when creating the IPA? (to upload the final version in the app store)

No, I don't believe it will. But by the time you submit the final version I'd expect react-native to be properly updated.

Yeah, but it is quite complicated.. since I am using expo-cli I am not using an external simulator. So I need to be constantly building the iOS app to compile the blockstack auth... and that is a bit of a pitty

It does not work on the iOS 12.1 simulator either...

Hey--can you expand and screenshot the error you are getting using iOS 12.1?

Hey, I'm trying out Expo-cli.  Would be faster to message you on there. Thanks!

Sure! I am not, but I can log in

@whiteyhat Whew, FINALLY managed to track down and fix the issue. Two ways to get a working version:

Easiest way to fix:
Reclone the repository (not sure if pulling the latest changes will fix it for you, but you can try first).

Alternative way to fix if you don't want to re-download the repo:
a. Pull latest code in develop branch.
b. In XCode, delete the blue RNBlockstackSdk folder and select "Remove References".
c. Re-add that same folder from "example/node_modules/react-native-blockstack/ios/", but make sure "Create Groups" is selected in the file picker window that pops up.
d. The RNBlockstackSdk folder should be back now, but this time in yellow 😯.

That's it! Should be working. At some point it seems react native broke for what appears as "blue folders" in XCode. The blue folder represents folders in the deployed app bundle, and apparently RN related code within them stops working. By doing the fix outlined above, we are turning it into a yellow folder which means that it will not be in its own folder in the app bundle.
The other issue I mentioned above resulted in the same error, but for a different reason, and so fixing that did not actually fix this issue. Sorry for the wait :).

If I go for the 1st one (the easiest way to fix). When I clone it, I had to change to the develop branch?

Just merged everything into master branch, so either develop or master will work :).

Nice. Does it have to be 12.1 or it alos works on 12.2?

Works on 12.2 as well

👍

Okay, It works. Even tho, it has some compiling issues. I am not sure how this will affect

the upload of the IPA to the App Store

That is probably a local issue. Try running your commands with sudo. Other who have had this error seem to have fixed it by reinstalling things (https://github.com/facebook/create-react-app/issues/4316).


ValueError("Invalid padding bytes.") when trying to decrypt Adobe.PubSec encrypted pdf file

Describe the bug
Crash when trying to decrypt Adobe.PubSec encrypted file
To Reproduce
I'm afraid this will be very hard to reproduce, since I can't share the files used to reproduce this. This is the output:
(venv) $ pyhanko --verbose decrypt pkcs12 --force encrypted.pdf decrypted.pdf key.p12
2024-03-25 10:25:57,582 - root - DEBUG - Running with --verbose
2024-03-25 10:25:57,582 - root - DEBUG - There was no configuration to parse.
Key passphrase:
2024-03-25 10:26:03,349 - cli - ERROR - Generic processing error.
Traceback (most recent call last):
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 772, in _import_object
    return reference_map[obj.reference]
KeyError: Reference(idnum=26, generation=0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 772, in _import_object
    return reference_map[obj.reference]
KeyError: Reference(idnum=27, generation=0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 772, in _import_object
    return reference_map[obj.reference]
KeyError: Reference(idnum=34, generation=0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 772, in _import_object
    return reference_map[obj.reference]
KeyError: Reference(idnum=168, generation=0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 772, in _import_object
    return reference_map[obj.reference]
KeyError: Reference(idnum=296, generation=0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 772, in _import_object
    return reference_map[obj.reference]
KeyError: Reference(idnum=300, generation=0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/cli/runtime.py", line 50, in pyhanko_exception_manager
    yield
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/cli/commands/crypt.py", line 187, in _decrypt_pubkey
    w = copy_into_new_writer(r)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 1262, in copy_into_new_writer
    new_root_dict = w._import_object(
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 835, in _import_object
    return generic.ArrayObject(
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 836, in <genexpr>
    self._import_object(v, reference_map, obj_stream) for v in obj
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 835, in _import_object
    return generic.ArrayObject(
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 836, in <genexpr>
    self._import_object(v, reference_map, obj_stream) for v in obj
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 835, in _import_object
    return generic.ArrayObject(
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 836, in <genexpr>
    self._import_object(v, reference_map, obj_stream) for v in obj
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 769, in _import_object
    obj = obj.decrypted
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/generic.py", line 2067, in decrypted
    decrypted = pdf_string(cf.decrypt(local_key, obj.original_bytes))
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/crypt/filter_mixins.py", line 134, in decrypt
    return aes_cbc_decrypt(key, data, iv)
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/pyhanko/pdf_utils/crypt/_util.py", line 20, in aes_cbc_decrypt
    return unpadder.update(plaintext) + unpadder.finalize()
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/cryptography/hazmat/primitives/padding.py", line 160, in finalize
    result = _byte_unpadding_check(
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/cryptography/hazmat/primitives/padding.py", line 97, in _byte_unpadding_check
    raise ValueError("Invalid padding bytes.")
ValueError: Invalid padding bytes.
Error: Generic processing error.

Expected behavior
The document decrypts.
Environment (please complete the following information):

OS: Ubuntu 22.04
Version;

$ pip3 freeze
asn1crypto==1.5.1
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cryptography==42.0.5
idna==3.6
oscrypto==1.3.0
pycparser==2.21
pyHanko==0.23.2
pyhanko-certvalidator==0.26.3
pypng==0.20220715.0
PyYAML==6.0.1
qrcode==7.4.2
requests==2.31.0
typing_extensions==4.10.0
tzlocal==5.2
uritools==4.0.2
urllib3==2.2.1

Python 3.10.12
Additional context
$ openssl pkcs12 -legacy -info -in key.p12 -noout
Enter Import Password:
MAC: sha1, Iteration 100000
MAC length: 20, salt length: 20
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 50000
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 50000
Certificate bag
Certificate bag

I've had problems with the 40 bit RC2 in the past, so I upgraded the key by following https://www.docuseal.co/docs/convert-legacy-p12-pfx-files-to-support-openssl-3, but I still have the same problem.
$ openssl pkcs12 -legacy -info -in key_new.p12 -noout
Enter Import Password:
MAC: sha256, Iteration 2048
MAC length: 32, salt length: 8
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CBC, Iteration 2048, PRF hmacWithSHA256
Certificate bag
Certificate bag
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 2048, PRF hmacWithSHA256

I can confirm that I can decrypt the document with the code in https://github.com/py-pdf/pypdf/pull/2537

Interesting... can you try with the previous release? 0.23.1
I can't test myself right now, unfortunately, but I'll take a look later.

Yes, with pip install 'pyHanko[pkcs11,image-support,opentype,xmp]==0.23.1', I get the same error.
Additionally, while trying to decrypt another document (that I also succesfully decrypted before with my patch to pypdf), I got:
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 780, in _import_object
    imported = self._import_object(refd, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 793, in _import_object
    raw_dict = {
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 794, in <dictcomp>
    k: self._import_object(v, reference_map, obj_stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/writer.py", line 774, in _import_object
    refd = obj.get_object()
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/generic.py", line 529, in get_object
    obj = self.reference.get_object()
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/generic.py", line 204, in get_object
    return self.pdf.get_object(self).get_object()
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/reader.py", line 425, in get_object
    obj = self._read_object(
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/reader.py", line 478, in _read_object
    retval = self._get_object_from_stream(
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/reader.py", line 291, in _get_object_from_stream
    obj = generic.read_object(
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/generic.py", line 246, in read_object
    result = DictionaryObject.read_from_stream(
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/generic.py", line 1273, in read_from_stream
    key = read_object(stream, container_ref)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/generic.py", line 240, in read_object
    result = NameObject.read_from_stream(stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/generic.py", line 1124, in read_from_stream
    name_bytes = read_until_delimiter(stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/misc.py", line 108, in read_until_delimiter
    result = _read_until_class(PDF_WHITESPACE + PDF_DELIMITERS, stream)
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/misc.py", line 125, in _read_until_class
    return b''.join(_build())
  File "/home/user/Documents/secure/venv-oldpyhanko/lib/python3.10/site-packages/pyhanko/pdf_utils/misc.py", line 120, in _build
    tok = stream.read(1)
RecursionError: maximum recursion depth exceeded while calling a Python object
Error: Generic processing error.

The full stacktrace is attached: stacktrace.txt

Hi @redfast00,
Those are all pretty interesting... There seems to be some kind of regression in 0.23.1 -> 0.23.2 regarding permission flags in files using the PubSec handler, but so far that appears to be unrelated to what you're seeing.
I see _get_object_from_stream appear in both stack traces, so I suspect this to be a funny interaction between object streams (AKA "compressed objects") and file encryption. I'll investigate further along those lines...

Thank you for looking at it; I understand that this can be frustrating if you don't have the files, but I unfortunately really can't share them. If you need me to try some things out, feel free to ask :smile:

Hmmm, I tried quite a few permutations but could not reproduce what you're seeing. Can you tell me which software (purportedly) created your PDFs? And can you post the content of the /Encrypt dictionary?
For the first document, can you try breakpointing the failure and check if (a) pyHanko is able to decrypt any strings / streams at all in that document, and (b) if it fails only on a specific one, whether that string "looks" encrypted? I'm starting to think that perhaps there's a string somewhere that has not been encrypted by mistake (I can think of some scenarios that could result in that kind of thing).
In the second issue, it appears that the public-key encryption part is working as it should: the bottom of the call stack seems to indicate that we're successfully looking inside an object stream somewhere, which requires said object stream to have been decrypted without issues. Rather, from the very high object ids in your stack trace, I think there's just a very deep path in the tree somewhere that is blowing up the stack (a particularly convoluted structure tree could do that, I suppose...). I didn't spot any loops in the stack trace (which _import_object defends against, FWIW). The only way to fix that would be to rewrite _import_object to not use recursion, which is of course possible but it requires some rearranging.

PDF metadata
In the PDF metadata, there is:
For the small document (4 pages) that crashes with the value-error:
'/Creator': 'Acrobat PDFMaker 9.1 for Word',
'/Producer': 'Adobe PDF Library 9.0',

For the big document, that crashes with the recursion error:
{'/Creator': 'FrameMaker 16.0.4',
 '/Producer': 'Acrobat Distiller 23.0 (Windows)'}

(identifying information removed)
Can it decrypt any strings?
By breakpointing, I was able to verify that the shared key calculates to the same that the working patch to pypdf does. It is indeed able to decrypt strings: most of the plaintexts look like hex garbage, but there is a pattern in them (they start with the same character); and some plaintexts contain readable metadata.
The place where it fails in aes_cbc_decrypt, data is b''; resulting in plaintext also being b'', and there likely not being any more padding. I fixed this by adding a check for zero-length encrypted data (it will then just return b'', just like pypdf does), and then indeed saw that it was trying to decrypt something that looks like a certificate. I added a check to return this as-is instead of decrypting; the new error I get is:
2024-03-26 10:48:33,584 - root - DEBUG - Running with --verbose
2024-03-26 10:48:33,584 - root - DEBUG - There was no configuration to parse.
Key passphrase:
2024-03-26 10:48:39,451 - tzlocal - DEBUG - /etc/timezone found, contents:
 Europe/Brussels

2024-03-26 10:48:39,451 - tzlocal - DEBUG - /etc/localtime found
2024-03-26 10:48:39,452 - tzlocal - DEBUG - 2 found:
 {'/etc/timezone': 'Europe/Brussels', '/etc/localtime is a symlink to': 'Europe/Brussels'}
2024-03-26 10:48:39,454 - cli - ERROR - Generic processing error.
Traceback (most recent call last):
  File "/home/user/Documents/secure/pyHanko/pyhanko/cli/runtime.py", line 50, in pyhanko_exception_manager
    yield
  File "/home/user/Documents/secure/pyHanko/pyhanko/cli/commands/crypt.py", line 189, in _decrypt_pubkey
    w.write(outf)
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/writer.py", line 608, in write
    self._write(stream)
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/writer.py", line 624, in _write
    self._write_objects(stream, object_positions)
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/writer.py", line 512, in _write_objects
    obj.write_to_stream(stream, handler, container_ref)
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/generic.py", line 1249, in write_to_stream
    value.write_to_stream(stream, handler, container_ref)
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/generic.py", line 487, in write_to_stream
    data.write_to_stream(
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/generic.py", line 1249, in write_to_stream
    value.write_to_stream(stream, handler, container_ref)
AttributeError: 'Reference' object has no attribute 'write_to_stream'
Error: Generic processing error.

Second document
This document is indeed pretty large (334 pages, 16M), so it's possible that the object graph is indeed too big.

Thanks! That helps a lot, I'm now pretty confident that the issues you're seeing are not related to public-key encryption as such (that seems to work), it's got more to do with decrypting the actual objects in the file(s).

The place where it fails in aes_cbc_decrypt, data is b''; resulting in plaintext also being b'', and there likely not being any more padding.

Hmm, this sounds like a bug in the software that produced the document, to be honest. For disambiguation purposes, PKCS#7 padding requires adding a full block of padding bytes even when the message length is an integer multiple of the block size (and AFAIK an empty message is not an exception to this rule). But OK, that's something we can try to tolerate.

AttributeError: 'Reference' object has no attribute 'write_to_stream'

This error is quite curious. Out of curiosity: where did you put that workaround to check for empty strings?

This document is indeed pretty large (334 pages, 16M), so it's possible that the object graph is indeed too big.

Mmkay, I'll look into rewriting _import_object to not work recursively, then.

This is identifying information, I'm afraid.

Fair enough; I think we've ruled out the Encrypt dictionary as a source of problems anyway :).
By the way: are these the only two public-key encrypted documents you have tried, or are there others?

Hi @redfast00,
I think I managed to hunt down all the issues you saw: the object graph thing, the padding issue, and also the AttributeError: 'Reference' object has no attribute 'write_to_stream' error you encountered (coincidentally, I stumbled upon that one while fixing the object importing logic).
Can you try again with the code in #414?

@MatthiasValvekens good news and bad news!
With the code from your PR, I still get:
Traceback (most recent call last):
  File "/home/user/Documents/secure/pyHanko/pyhanko/cli/runtime.py", line 50, in pyhanko_exception_manager
    yield
  File "/home/user/Documents/secure/pyHanko/pyhanko/cli/commands/crypt.py", line 187, in _decrypt_pubkey
    w = copy_into_new_writer(r)
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/writer.py", line 1293, in copy_into_new_writer
    new_root_dict = importer.import_object(input_handler.root)
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/writer.py", line 1173, in import_object
    imported = self._ingest(source_obj)
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/writer.py", line 1195, in _ingest
    raw_dict = {
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/writer.py", line 1196, in <dictcomp>
    k: self._ingest(v) for k, v in obj.items() if k != '/Metadata'
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/writer.py", line 1191, in _ingest
    obj = obj.decrypted
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/generic.py", line 2067, in decrypted
    decrypted = pdf_string(cf.decrypt(local_key, obj.original_bytes))
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/crypt/filter_mixins.py", line 134, in decrypt
    return aes_cbc_decrypt(key, data, iv)
  File "/home/user/Documents/secure/pyHanko/pyhanko/pdf_utils/crypt/_util.py", line 16, in aes_cbc_decrypt
    plaintext = decryptor.update(data) + decryptor.finalize()
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/cryptography/hazmat/primitives/ciphers/base.py", line 184, in finalize
    data = self._ctx.finalize()
  File "/home/user/Documents/secure/venv/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/ciphers.py", line 223, in finalize
    raise ValueError(
ValueError: The length of the provided data is not a multiple of the block length.

but I already got this error before; if I change the AES decrypt function to:
def aes_cbc_decrypt(key, data, iv, use_padding=True):
    if len(data) == 8636:
        print("WARNING: special case for somehow not encrypted data")
        print(data)
        return data
    cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
    decryptor = cipher.decryptor()
    plaintext = decryptor.update(data) + decryptor.finalize()

    # we tolerate empty messages that don't have padding
    if use_padding and len(plaintext) > 0:
        unpadder = padding.PKCS7(128).unpadder()
        return unpadder.update(plaintext) + unpadder.finalize()
    else:
        return plaintext

then the small PDF decrypts fine, and seems to look okay :tada:
I also tested the large PDF with the same code, this also decrypts now, and also hits the same 'somehow not encrypted data' path as the small PDF (with the exact same length).
 It is clearly plaintext, since I can see the string This certificate has been issued in accordance with the GlobalSign CPS located at https://www.globalsign.com in the data. pypdf with my patch does not error on this, so I think it should be possible to somehow see the difference between data that needs to be decrypted and data that doesn't (with a better heuristic than just looking at the ciphertext length and seeing if it is a multiple of the block length).

Hi @redfast00,
Aha, I suspect that your input files contain digital signatures... :D. Does the object reside in a key named /Contents?
That would also immediately explain a lot of things:

The logic to import/export objects from encrypted files was never really tested on signed files because the process (necessarily) destroys any signatures in the input document. Hence it never even occurred to me that anyone would want to do this, but here we are :).
The Contents key in a digital signature dictionary is one of very few "special cases" in the PDF encryption model in that it is exempt from encryption (that provision in the spec has some interesting history behind it, but I won't bore you with that). FWIW, the list of such special objects is very short and explicitly enumerated in the standard.
The AttributeError you ran into before only manifested when a part of the PDF object graph points back to the document catalog. This almost never happens in "regular" PDF documents, unless....the document contains a digital signature with DocMDP.

Anyway, if that's what's going on, then the mystery as to why you're hitting all those wonky corner cases is also solved ;).
I can probably make the importer aware of DigSig data in the input file and make it deal with those things gracefully (but the signatures will be not be valid for the decrypted output either way, so be mindful of that).

@MatthiasValvekens the input file indeed contains a digital signature: it is signed + encrypted.
I don't know what key this object resides in; I couldn't immediately find a way to get the key from the AES decryption function.

but the signatures will be not be valid for the decrypted output either way, so be mindful of that

I really don't mind, I'd already be very happy to be able to read my document in an open-source PDF reader (instead of having to boot Windows just to use Adobe Reader).
As feedback from a user, for me it was not immediately clear from the README that pyHanko could be used to decrypt PDF documents; maybe this is useful to add to the README/documentation?

I can probably make the importer aware of DigSig data in the input file and make it deal with those things gracefully (but the signatures will be not be valid for the decrypted output either way, so be mindful of that).

I've implemented, tested and merged this (see #414). Can you give it a try on your files? If that works, then we can close this issue as well.

As feedback from a user, for me it was not immediately clear from the README that pyHanko could be used to decrypt PDF documents; maybe this is useful to add to the README/documentation?

Right, the README mentions encryption support but does not explicitly call out encryption/decryption of files as a feature. That's indeed worth some clarification. Thanks!

Werkt perfect, bedankt om dit op te lossen!


WIP: THORN-2389: Move CDI Producers code into smallrye-jwt

Motivation
There are duplicate copies of MP JWT CDI producers in Thorntail and Quarkus, so they need to be consolidated in smallrye-jwt
Modifications
All the local CDI producers have been removed and the ones from smallrye-jwt are now being used.
Additionally, an unused ClaimLiteral class has also been removed
Result
CDI producers code can not be controlled in a single place, smallrye-jwt, and easily reused in Thorntail, Quarkus

At the moment it depends on smallrye-jwt 1.1.4-SNAPSHOT

Sure, we will need to release smallrye-jwt 1.1.4 first


Add variable query support

Derived from #100
Fetch query variable options directly from Cassandra.
https://grafana.com/developers/plugin-tools/create-a-plugin/extend-a-plugin/add-support-for-variables#add-support-for-query-variables-to-your-data-source
Mostly probably blocked by #167

@unflag FYI

Hell yeah 🚀

Even better, now with labels :)


Compared with dirsearch ，There are quite a few false positives

1、When all the results are 200, it returns all the results back.These result pages and URL pages are the same.
eg: http://www.example.com/                        length:80  word:30
http://www.example.com/FUZZ            all result  length:80  word:30
Such false positives should not be returned
2、When -r and -fc exist together。 The fc parameter should filter 301/302, not the redirected page status.
eg: http://www.example.com/FUZZ     all result return 302 status code ,Redirect the 200 false positives page

Hi @icezhaoL, how are you using FFUF? with -ac and -acc?

ffuf -w dicc.txt -u https://api*.com/FUZZ  -o /1.json -c  -r
no use -ac and -acc
i don't know how to use this options ac and acc?

Ok, i see.
-ac is mainly which will fix your problem, it will first run looking the pattern of the server and then ignore it.
If you use -ac it will ignore length:80 word:30
ffuf arguments_here -ac
-acc is to ignore all pages similar to one you pass.
-acc my_custom_page.html
Then he will take code/length/words and ignore it.

Thank you for your patience.

Don't worry, anything you need just open a new issue =]

Thanks for clearing this up @bsysop . I consider the issue solved, and will close it.


Feature : Automated Greeting Workflow

Description:
This pull request implements a new GitHub Actions workflow named "Greetings," which automatically greets users who open new issues or pull requests in this repository.
Key Features:

Automatic Greeting: Users will receive an immediate acknowledgment when they create an issue or pull request. The messages are:

For Issues: "👋 Thank you for raising an issue! We appreciate your effort in helping us improve. Our team will review it shortly. Stay tuned!"
For Pull Requests: "🎉 Thank you for your contribution! Your pull request has been submitted successfully. A maintainer will review it as soon as possible. We appreciate your support in making this project better."

Hey @daccotta Please assign this Task to me

Sound's amazing. Go ahead @samyak-aditya , Would love to see this workflow setup.

Hey @sid0000007 I have raised the PR
thanks


fix(github): Include "new notifications" in direct total

Pre-flight Checklist
Please ensure you've completed all of the following.

[X] I have read the Contributing Guidelines for this project.
[X] I agree to follow the Code of Conduct that this project adheres to.

Description of Change

GitHub's notification page no longer updates the notification count when notifications arrive. Instead, it displays a small message saying "COUNT new notifications".
This PR includes that COUNT in the number of direct messages reported to Ferdium.
For example, with this update, my GitHub icon in Ferdium shows 7 direct messages, 4 of which come from the inbox, and 3 of which come from the "new notifications" banner. Prior to this commit, the GitHub icon would only show the 4 direct messages from the inbox.

@all-contributors please add @thehale for code


Write Module#deprecate_constant specs

Thank you for your contribution, clean and fast!

@eregon Thanks to merge! And sorry for my mistaken to check private constants. It is followed in #108.


Add inviscid Burgers equation in 1D

My rationale for this PR is to provide a common basis for further experimentation and code discussions in the stability channel. I am not sure if this is sufficient, but I thought maybe it is enough to have 1D Burgers, since it is mostly useful for numerical experimentation anyways.

What about adding the standard setup used for our local linear stability investigations?

Yes, please do it! I only didn't add it since I was not sure if y'all agree to Burgers 1D and since I thought you probably know much better which exact setup to implement.

What do you make of this error, @ranocha ? Seems to be CI related - should we try again or just merge? https://github.com/trixi-framework/Trixi.jl/pull/467/checks?check_run_id=2106391580#step:6:237

Yeah, I think so. I'll merge and we can see what happens (and fix it before tagging a new release of Trixi if necessary).


sap.m.PlanningCalendar: stickyHeader gets transparent when scrolled down

OpenUI5 version: 1.108.16
Browser/version (+device/version): Chrome 114.0.5735.134
URL (minimal example if possible): https://sdk.openui5.org/1.108.16/#/entity/sap.m.PlanningCalendar/sample/sap.m.sample.PlanningCalendarWithStickyHeader
Steps to reproduce the problem:

scroll down

What is the expected result?
Sticky/Fixed header needs to remain opaque.
What happens instead?
The sticky header gets transparent -> gets visually mixed with content.
Any other information?
In version 1.108.14 the behavior is still fine: https://sdk.openui5.org/1.108.14/#/entity/sap.m.PlanningCalendar/sample/sap.m.sample.PlanningCalendarWithStickyHeader

Hello @mkueng1,
Thank you for reporting this issue.
I've created an internal incident<PHONE_NUMBER>.
The status of the ticket will be updated here in GitHub.
Best regards,
Konstantin
Github Dispatcher

@kgogov
Hello, our team also found the same bug with planning calendar in the app.
It seems the trouble is connected to the version of sapui5 library, that we are using. Before we switched to the newest version - 1.108.19 (https://sapui5.hana.ondemand.com/1.108.19/resources/sap-ui-core.js), we were using 1.91.1, in which there isn't such issue
What will be your proposals?

Are there any updates as it prohibits of using newer 1.108 bugfix versions which include very important fixes.

Hello @mkueng1,
Thank you for reporting this issue.
this issue has been resolved since version 1.108.21
Here it can be tested - https://sapui5.hana.ondemand.com/1.108.21/#/entity/sap.m.PlanningCalendar/sample/sap.m.sample.PlanningCalendarWithStickyHeader
Best regards,
Todor


Flight Bypass

Description : The bug is a Flight bypass.
Steps to reproduce the behavior:

Install FDPClient v5.4.0
And use AAC3.0.5 Flight

Video : https://streamable.com/40akq1

OS :: Windows 11
Plugin Version :: 4.0
Server Version :: 1.8.8 Spigot
Client Version and Type :: Forge 1.8.9

Flight code : https://github.com/SkidderMC/FDPClient/blob/main/src/main/java/net/ccbluex/liquidbounce/features/module/modules/movement/flys/aac/AAC305Fly.kt

Patched, will be released in the next release candidate. Thanks for the report.


build.sh error on OS X 10.10-x64

Executing task: OnlyPublish
Publishing OmniSharp for .NETFramework,Version=v4.5.1/osx.10.10-x64
Project OmniSharp.Abstractions (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.Roslyn (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.Roslyn.CSharp (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.MSBuild (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.ScriptCs (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.Nuget (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.Stdio (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.Plugins (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.Host (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.DotNet (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp.DotNetTest (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
Project OmniSharp (.NETFramework,Version=v4.5.1) was previously compiled. Skipping compilation.
publish: Published to /Users/nathan/development/omnisharp-roslyn/artifacts/publish/OmniSharp/default/net451
Published 1/1 projects successfully
Can not find runtime target for framework '.NETCoreApp,Version=v1.0' compatible with one of the target runtimes: 'osx.10.10-x64'. Possible causes:
1. The project has not been restored or restore failed - run `dotnet restore`
2. The project does not list one of 'osx.10.10-x64' in the 'runtimes' section.
An error occured when executing task 'OnlyPublish'.
Error: Failed to publish OmniSharp / netcoreapp1.0

Your issue seems similar to #532.
If it's the same issue, then you'll need to add "osx.10.10-x64": { }, to https://github.com/OmniSharp/omnisharp-roslyn/blob/dev/src/OmniSharp/project.json#L37

Adding that made the process go further, but now I'm getting System.IO exceptions:
Unhandled Exception:
System.IO.IOException: Invalid handle to path "/Users/nathan/development/omnisharp-roslyn/[Unknown]"
  at System.IO.FileStream.ReadData (System.Runtime.InteropServices.SafeHandle safeHandle, System.Byte[] buf, Int32 offset, Int32 count) <0x108dd0060 + 0x00096> in <filename unknown>:0
  at System.IO.FileStream.ReadInternal (System.Byte[] dest, Int32 offset, Int32 count) <0x108dce1c0 + 0x00077> in <filename unknown>:0
  at System.IO.FileStream.Read (System.Byte[] array, Int32 offset, Int32 count) <0x108dcdfb0 + 0x000e8> in <filename unknown>:0
  at System.IO.StreamReader.ReadBuffer () <0x108c3d890 + 0x002c3> in <filename unknown>:0
  at System.IO.StreamReader.Read () <0x108c3c8a0 + 0x00068> in <filename unknown>:0
  at System.TermInfoDriver.GetCursorPosition () <0x108ee8920 + 0x00099> in <filename unknown>:0
  at System.TermInfoDriver.Init () <0x108ee7d40 + 0x005a0> in <filename unknown>:0
  at System.ConsoleDriver.Init () <0x108ed5250 + 0x00033> in <filename unknown>:0
  at System.Console.add_CancelKeyPress (System.ConsoleCancelEventHandler value) <0x108ed4100 + 0x00018> in <filename unknown>:0
  at OmniSharp.Program.Main (System.String[] args) <0x10715d5d0 + 0x00c3f> in <filename unknown>:0
  at Program.Main (System.String[] args) <0x10715bd90 + 0x0014e> in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.IOException: Invalid handle to path "/Users/nathan/development/omnisharp-roslyn/[Unknown]"
  at System.IO.FileStream.ReadData (System.Runtime.InteropServices.SafeHandle safeHandle, System.Byte[] buf, Int32 offset, Int32 count) <0x108dd0060 + 0x00096> in <filename unknown>:0
  at System.IO.FileStream.ReadInternal (System.Byte[] dest, Int32 offset, Int32 count) <0x108dce1c0 + 0x00077> in <filename unknown>:0
  at System.IO.FileStream.Read (System.Byte[] array, Int32 offset, Int32 count) <0x108dcdfb0 + 0x000e8> in <filename unknown>:0
  at System.IO.StreamReader.ReadBuffer () <0x108c3d890 + 0x002c3> in <filename unknown>:0
  at System.IO.StreamReader.Read () <0x108c3c8a0 + 0x00068> in <filename unknown>:0
  at System.TermInfoDriver.GetCursorPosition () <0x108ee8920 + 0x00099> in <filename unknown>:0
  at System.TermInfoDriver.Init () <0x108ee7d40 + 0x005a0> in <filename unknown>:0
  at System.ConsoleDriver.Init () <0x108ed5250 + 0x00033> in <filename unknown>:0
  at System.Console.add_CancelKeyPress (System.ConsoleCancelEventHandler value) <0x108ed4100 + 0x00018> in <filename unknown>:0
  at OmniSharp.Program.Main (System.String[] args) <0x10715d5d0 + 0x00c3f> in <filename unknown>:0
  at Program.Main (System.String[] args) <0x10715bd90 + 0x0014e> in <filename unknown>:0

Versions and things:
$ mono --version
Mono JIT compiler version 4.2.3 (Stable <IP_ADDRESS>/832de4b Wed Mar 30 13:57:21 PDT 2016)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       altstack
	Notification:  kqueue
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug
	LLVM:          supported, not enabled.
	GC:            sgen

 $ dotnet --info
.NET Command Line Tools (1.0.0-preview1-002702)

Product Information:
 Version:     1.0.0-preview1-002702
 Commit Sha:  6cde21225e

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.10
 OS Platform: Darwin
 RID:         osx.10.10-x64

@nathansizemore Don't forget to restore after adding it.

Boom! That was it. Thanks!


Automate releases using actions

This PR automates releases for different architectures using github actions and goreleaser. This should fix #50 if merged.
An example run can be found here:
https://github.com/lehmann-4178656ch/gantry/commit/b192c47b2f9ef52fb40b07b5c3bb9a4167bf56bf/checks?check_suite_id=320054199
The resulting binaries here:
https://github.com/lehmann-4178656ch/gantry/releases/tag/v0.5.1

By the way, this is really cool. Didn't know it was this easy to automate already

├── gantry_linux_aarch64.tar.gz
├── gantry_linux_armv6l.tar.gz
├── gantry_linux_armv7l.tar.gz
├── gantry_linux_ppc64le.tar.gz
├── gantry_linux_s390x.tar.gz
└── gantry_linux_x86_64.tar.gz


Test notebook editor input refactoring

Refs: https://github.com/microsoft/vscode/pull/100266

[ ] anyOS @bpasero
[x] anyOS @isidorn

Complexity: 2
Authors: @jrieken @rebornix
Create Issue

We have reworked how notebook editors and its inputs are managed. Test everything related to opening notebooks like open, split, pin, unpin, move, close, reopen, and (window) reload. Make sure to have rendered output when splitting or moving and ensure that you never loose the output and that you never see two outputs (webviews)

Output is not preserved on reload, but I believe that is expected.
Running a query autoamticaly pins the Notebook editor but I also believe that is by design?

Have same notebook side by side, running a query in one also runs it in another. If we do this it sort of makes me feel like all state should be synced.

It depends on model state or view state, e.g when you split a text editor, typing is sync'd but find widget or selection isn't

https://github.com/microsoft/vscode/issues/101378 is pretty bad, otherwise seems to work fine 👍


Added plugin_movie video-to-texture plugin

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Do you want me to host it on Solar2D account so you would be able to automate releases?

Yes please

I created the repo and invited you with write access. Every time you push changes to plugins directory release would be created and published to directory automatically. It seems to be live already: https://plugins.solar2d.com/?search=Movie
I added automation file to my fork, so make sure to pull changes from it and push them to your repo. Or just delete your binary repo, I don’t think you need it anymore, since it’s separate from sources repo and you can just use hosted one.


Feature/homepage

set up database connection to flask
passed user data to homepage templates
created custom filters for templates

close Finalize homepage views with data and filters issue


Impro 983 cube manipulation

Partially addresses #742
Migrated remaining cube manipulation utility unit tests to use the new cube setup functions.  Tidied up and reduced duplication.
Testing:

[x] Ran tests and they passed OK

Thanks @cgsandford. I'm happy with this, and the compromise set up of the test data cut it down nicely.


Migrate to bootstrap

This pr resolves #431
All that remains when it comes to CSS are classes or IDs accessed via javascript (should be resolved with #432) or custom overrides for bootstrap it's self.
Updated look for desktop:

Updated look for mobile:

I will setup a meeting to discuss any changes to be made

One note from Kim, can we move the refresh icon to the left here

like it was originally

(but keeping the new opaqueness, that's a good improvement for readability)
"To the left please. That makes it seem more like a button and less like a status"

Also get a big blue outline in the pending state

Just a couple tweaks I noticed above that are worth focusing on but in general looking good.
I think there's some more grid changes and alignment tweaks for the "header-desc" info boxes we should do, however not worth looking at right now as they are working close enough as intended and we should go through a UX review to see if those will dramatically change anyways.
So (after tweaks above) worth merging and deploying to dev (to check on other devices) and then having UX review with Kim before going too much further

Just a couple tweaks I noticed above that are worth focusing on but in general looking good.
I think there's some more grid changes and alignment tweaks for the "header-desc" info boxes we should do, however not worth looking at right now as they are working close enough as intended and we should go through a UX review to see if those will dramatically change anyways.
So (after tweaks above) worth merging and deploying to dev (to check on other devices) and then having UX review with Kim before going too much further

Just pushed these changes. Thanks for the feedback and testing. Should be good to go now.


Upgrade dependencies and modernize code for 2018 edition

We're using this in sentry-cli and this pulls in a really old version of rand. This upgrades the dependency and I also modernized the crate in the process.

@mitsuhiko  thank you for your work!

I'll make a new release tonight.

@mitsuhiko I've cut a 0.1.3 release. Thanks again for the updates!

Thanks for the fast turnaround, upgraded immediately: https://github.com/getsentry/sentry-cli/commit/3fb080e9f10be0d8d171fc38f58f4b27f800143a :)


rails5 compatability

Sorry about the white spaces in readme.  I can re-open the PR without if you want.  I also didn't update the version.  I left that for you guys if the PR gets approved.

@austinrfnd thanks, released 0.3.0

@le0pard 🌮 👍


Autofill state and district after entering pin code

Yes I can work on this issue.

Sure, go-ahead
I'll assign you this issue when the contribution period starts from 10th June, then you can start working on this issue.

Could u please assign me this issue sir even I wanted to work on it.

@SLPoojitha I'll assign you this issue when the contribution period starts from 10th June, then you can start working on this issue.
Since many contributors are willing to resolve this issue, I will accept only the first PR which will completely resolve this issue.

Thank u sir!!

Fine @Zhong-Shan-Hui.
Wait till I review your PR.


Failed backups cleanup

Clean up failed backups when there is a backup in state Error or Unknown.
longhorn/longhorn#3898
longhorn/longhorn#4334
Signed-off-by: James Lu<EMAIL_ADDRESS>

In general, LGTM. Have some feedback, please resolve others' comments as well before the merge.
P.S. this needs longhorn/longhorn change as well because a new setting will be introduced.

Not sure why you prefer to delete the latter part... I think we should retain it but change the log level to warning instead.

@shuo-wu did you mean to keep this part "and it will skip the auto-deletion for the failed backups?"

@shuo-wu did you mean to keep this part "and it will skip the auto-deletion for the failed backups?"

Right. This part is what I suggest in the previous comment. But remember to modify the log level.
cc @derekbit What's the reason for removing this part?

@shuo-wu did you mean to keep this part "and it will skip the auto-deletion for the failed backups?"

Right. This part is what I suggest in the previous comment. But remember to modify the log level.
cc @derekbit What's the reason for removing this part?

Ah, sorry. I was not aware of that it is your recommendation.
Just thought the error message is enough for debugging. But yes, we can keep it here.

Ah, sorry. I was not aware of that is your suggestion. Just thought the simple error message is enough for debugging. But yes, we can keep it ("and it will skip the auto-deletion for the failed backups") here.

Never mind. I think it can help avoid a potential misunderstanding that this error will fail the reconciliation

@mantissahz Please backport this to 1.3.x and 1.2.x.


Cleanup PersistenceTestCluster interface

This change removes methods that are not generic enough and are only used in the internal implementation details.

Coverage decreased (-0.07%) to 67.033% when pulling 99d3ad63fc54a8edbbb9cd9b287544b50a565a16 on vitarb:cleanup-persistence-test-interface into 10fc818c7331b9ecbed9f974310613fd8cee11c6 on uber:master.


merge --no-record

For some workflows, you want to backport a bunch of changes (but not all) from another branch.  In this case, you must not record a merge, because that will prevent any changes you didn't backport from being merged later.
git update-ref -d MERGE_HEAD will do this and is trivial to add after merge.

Presumably we could also re-formulate the merge command from lower-level stuff, but that seems like a hassle.


first pass at full implementation

incomplete test coverage, but seems to be working

Since we only have one action, i'm not sure if we really need to separate out creators from types. I so to like them being together since they are so tightly coupled

And if we do decide to separate them out, I don't think we should name the action type the same as the action, because we should be able to change the action type value without changing the name of the file.


Winston has unexpected output when using  default meta

Example code
import winston from 'winston'
import ecsFormat  from '@elastic/ecs-winston-format'

const logger = winston.createLogger({
    level: 'info',
    format: ecsFormat(),
    defaultMeta: { service: "Service Name" },
    transports: [
        new winston.transports.Console()
    ],
  });

 logger.info('Unable to connect to ElasticSearch')

Output
{"@timestamp":"2021-02-11T01:45:57.337Z","log.level":"error","message":"Unable to connect to ElasticSearch","ecs":{"version":"1.6.0"},"service":{"0":"S","1":"e","2":"r","3":"v","4":"i","5":"c","6":"e","7":" ","8":"N","9":"a","10":"m","11":"e"}}

Expected
{"@timestamp":"2021-02-11T01:45:57.337Z","log.level":"error","message":"Unable to connect to ElasticSearch","ecs":{"version":"1.6.0"},"service":{"Service Name"}}

As you can see above when using defaultmeta the object value is serialized even if it is a string.  Any quick fix I can put in place to solve this?

@Tim-Machine The issue here is that "service" is a name reserved in ECS to be an object at the top-level: https://www.elastic.co/guide/en/ecs/current/ecs-service.html
That combined with the schema-based usage of "fast-json-stringify" in these ecs-logging packages results in that broken rendering of your "service" field.
I don't have a plan for a general fix yet. I'm not sure what an ECS logger should best do when a user attempts to use a conflicting type for an ECS-specified field. Something I need to think about and discuss with others.
As a quick fix, the canonical ECS way to state the service name is via "service.name", so either of:
    defaultMeta: {
        service: {name: "Service Name"},

or
    defaultMeta: {
        "service.name": "Service Name",

would work.  Does that suffice for you in the short term?

Let me test this out and I'll get back to you, good catch.

Just verified, this is working as suggested. Sorry for the false flag 😂


fix(configs/prow-dev): prowjob crd

the full edition will failed when kubectl apply or replace on open api schema validation.

/merge


🔥 Refactor to remove TDF helper class

Functional refactor of tdf.ts
This is a step toward a more functional set of internals that will be simpler to refactor
notably this is a step toward refactoring KAS, policy, metadata, and attribute related functionality to be shared between nanotdf and json container formats

https://virtru.slack.com/archives/GNFF2JBQ9/p1698755761287039?thread_ts=1698687454.262009&cid=GNFF2JBQ9


Compress large sized screenshot image

Compressed the screenshot file so that it can load faster for users.

Same Resolution, but compressed :)

Oh nice, thanks for this!!
I should probably also update this screenshot too 🤔

Yeah. That'd be nice.


IDE and docker integration

Expected Behavior

Add support to debug and run QCAlgorithms, developing locally in an IDE (VS Code, pycharm, visual studio, etc) and executing in Leans docker container. Add required documentation, scripts and configuration files for each IDE

Actual Behavior

Currently, we have run_docker.sh/bat, which don't allow providing an external algorithm to run .py/.dll (nor debugging)

Potential Solution

Implement 1 IDE at the time, first running a custom algorithm, second debugging.
Reproducing the Problem

N/A
System Information

N/A
Checklist

[x] I have completely filled out this template
[x] I have confirmed that this issue exists on the current master branch
[x] I have confirmed that this is not a duplicate issue by searching issues

[x] I have provided detailed steps to reproduce the issue

@Martin-Molinero mind adding the hint from the google-hangout about the switch statement to select the right IDE when setting up debugging


Different prediction from predict() while scoring

Description

Hello. I am getting a different prediction score (for the classification problem, I am looking at the scores, not the actual output class). Is it possible?
I ran the scoring last month and then again ran the scoring on the same dataset this month and for the same input I am getting different scores. Is there a way to explain this?
Reproducible example

Environment info
LightGBM version or commit hash:
Command(s) you used to install LightGBM

Additional Comments

Thanks for using LightGBM!
Without a reproducible example it will be hard for us to help, but I can at least guarantee to you that there should be nothing random about scoring and give you a list of things to double check.

Are you using the same version of lightgbm and all its dependencies?
Are you certain the model file has not been altered?
Are you using the same version of Python?
Is model scoring running in exactly the same operating system, and was lightgbm built exactly the same way (e.g., with/without CUDA support, with/without OpenMP)?
Are you certain the datasets are identical?

Thank you for sending these checkpoints. I will sure check this out.
I am unable to provide any script because the code is in production and the model cannot be publisized. What we are noticing is that this issue occurred only between the runs which were months apart but as we are testing them now repeatedly on consecutive days, the scores are consistent. We will be monitoring it in case anything happens after a certain period of time.

Are you using the same version of lightgbm and all its dependencies?
Are you using the same version of Python?
Is model scoring running in exactly the same operating system?
Will it make difference in the scores if these do not match ? Why should it, just wondering!

@ayush488
can you provide a minimal reproducible example?

Will it make difference in the scores if these do not match ? Why should it, just wondering!

Any changes in the lightgbm version used to a load a model could mean that the predictions are different if that new version contains relevant bugfixes or newly-introduced bugs. We try to limit the risk of new bugs with extensive testing and try to publicize fixes to existing bugs in release notes, but it's not possible to be 100% confident.

same version of Python?... same operating system?

Subtle differences in numeric precision can lead to slightly different predictions in some cases, and changing Python versions or operating systems might introduce such differences.
We try to control for that but again, cannot be 100% confident that all such cases are accounted for in this project's tests.


Added driver support for m24256 eeprom

Added the driver support for m24256 eeprom. It allows to read and write the data

@henrikbrixandersen Please take a review

@henrikbrixandersen Please take a review

I will get to it, you only submitted this 4 days ago. While you wait, I suggest you read up on our documented Contributor Expectations, especially the section on review escalation: https://docs.zephyrproject.org/latest/contribute/contributor_expectations.html#pr-review-escalation

@henrikbrixandersen Waiting for your approval, please take a look.

Lets close this PR as per our discussion
https://discordapp.com/channels/720317445772017664/1218200905044721765/1219957042874155008


use protocol

I think here should be a unified statement with agreement

thanks for your pull requests, could you help to update usage on ViewController.m and RLArithmeticTests.m
Thanks again for your attention to this code repository!

#3


[CR][ENG-481] Update WB base image for Oathpit python36

Ticket
https://openscience.atlassian.net/browse/ENG-481
Purpose
Since we're doing comprehensive regression testing for OATHPIT, take advantage to apply some widespread changes to the base image and codebase.
Changes

Upgrade base Dockerfile image to use python3.6 and be based on buster
Bump tornado dep to v6 and add newrelic support.
Start pulling gosu from apt-get
Remove some code constructs that will be invalid on python3.7
Bump test suite dependencies to latest and fix incompatibilities
Replace raven-python with sentry-sdk
Mark some docstrings as raw to support sphinx formatting
Other minor misc changes.

Side effects
There shouldn't be, but it's possible.
QA Notes
This will be tested as part of the other OATHPIT regression testing.
Deployment Notes
Since this affects the Dockerfile, we may need to pay special attention to deploy step.  Also needs a patch to osf.io's docker-compose.yml to work with local devel.

Coverage decreased (-0.01%) to 54.319% when pulling 0df018d4a6f84d03ea1bcaa17b24e497ba37342f on felliott:feature/oathpit-python36 into 12f540acf2a5bb9983b5bc7e213b1513797beb07 on CenterForOpenScience:feature/oathpit.


Fixed Colors in Panda syntax

I had fixed color issues in CSS file. Please merge it :)

Please open a pull request that is based on the current code. Your patch restores a bug that was fixed since you first submitted this code.


Show demo on using bootc + podman desktop + microshift

Is your enhancement related to a problem? Please describe
Create a demo showcasing microshift with bootc extension
Describe the solution you'd like
Demo showing microshift onto a usb, demo it showing in podman desktop
Describe alternatives you've considered
No response
Additional context
No response

Current progress is getting microshift example working and using RHEL instead of centos stream (since microshift's requirement is RHEL anyways), see issue: https://gitlab.com/fedora/bootc/examples/-/issues/17
Going it to boot with centos stream 9 / active, now going to set it up with RHEL.

Some updates:

Coordinating with microshift team for them to update their documentation, started with PR: https://github.com/openshift/microshift/pull/4004 to add a separate Containerfile
Issue opened regarding using bootc-image-builder in their official documentation:


Fix for several issues

Add some fixes around 3 node cluster and lpar support, dns setting, ip_forward, etc. Include
-initrd-inject "/{{ kvm_host_home.stdout }}/{{ env.file_server.cfgs_dir }}/{{ env.bastion.networking.hostname }}/bastion-ks.cfg" \ in case pf create bastion which works not consistent and env.file_server.cfgs_dir should contain the absolute path.
Furthermore section 1 is being flatten.

:tada: This PR is included in version 2.2.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


CosmosDB - Add support for geo-redundancy

Be able to choose the geo-redundancy option at the creation

Thanks for the issue @julienstroheker. Looking at the ARM template we are using and the portal, that isn't a direct option like the portal, which defaults to East US and West US (based on the tool tip it gave me). In the ARM template, we can provide:
locations | -  An array that contains the georeplication locations enabled for the Cosmos DB account.
-- | -- | -- | --

Do you think exposing that as a provisioning parameter is sufficient?

@jeremyrickard Sound good !

Solved by https://github.com/Azure/open-service-broker-azure/pull/544.


when i am installing the plugin naming npm istall ionic2-calendar --save

it is displaying error naming typings install. I dont know how to rectify the error

Not related to this version.


C++ bindings to the Batch API

An example use of the Batch API (requires CUDA v12.1+):

    // Here we use the batch API to read "/tmp/test-file" into `b_dev` by
    // submitting 4 batch operations.
    constexpr int num_of_batches = 4;
    constexpr int batchsize      = SIZE / num_of_batches;
    kvikio::DriverProperties props;
    check(num_of_batches < props.get_max_batch_io_size());

    // We open the file as usual.
    kvikio::FileHandle f("/tmp/test-file", "r");

    // Then we create a batch
    auto batch = kvikio::BatchHandle(num_of_batches);

    // And submit 4 operations each with its own offset
    std::vector<kvikio::BatchOp> ops;
    for (int i = 0; i < num_of_batches; ++i) {
      ops.push_back(kvikio::BatchOp{.file_handle   = f,
                                    .devPtr_base   = b_dev,
                                    .file_offset   = i * batchsize,
                                    .devPtr_offset = i * batchsize,
                                    .size          = batchsize,
                                    .opcode        = CUFILE_READ});
    }
    batch.submit(ops);

    // Finally, we wait on all 4 operations to be finished and check the result
    auto statuses = batch.status(num_of_batches, num_of_batches);
    check(statuses.size() == num_of_batches);
    size_t total_read = 0;
    for (auto status : statuses) {
      check(status.status == CUFILE_COMPLETE);
      check(status.ret == batchsize);
      total_read += status.ret;
    }
    check(cudaMemcpy(b, b_dev, SIZE, cudaMemcpyDeviceToHost) == cudaSuccess);
    for (int i = 0; i < NELEM; ++i) {
      check(a[i] == b[i]);
    }
    cout << "Batch read using 4 operations: " << total_read << endl;

An example use of the Batch API (requires CUDA v12.1+):

    // Here we use the batch API to read "/tmp/test-file" into `b_dev` by
    // submitting 4 batch operations.
    constexpr int num_of_batches = 4;

may be it will be clear to say this is "num_ops_in_batch".

An example use of the Batch API (requires CUDA v12.1+):

    // Here we use the batch API to read "/tmp/test-file" into `b_dev` by
    // submitting 4 batch operations.
    constexpr int num_of_batches = 4;

may be it will be clear to say this is "num_ops_in_batch".

Good point, fixed

Thanks @wence-, I think I have addressed all of your issues?

Awesome, thanks for the review @wence-. Just in time for the code freeze :)

/merge


What is the equivalent of require('aerial').setup({}) in you have a init.vim (no Lua) config

Neovim version (nvim -v)
NVIM v0.8.3
Operating system/version
MacOS
Output of :AerialInfo
not an editor command
Describe the bug
What is the equivalent of require('aerial').setup({}) in you have a init.vim (no Lua) config
Steps To Reproduce
What is the equivalent of require('aerial').setup({}) in you have a init.vim (no Lua) config
Expected Behavior
What is the equivalent of require('aerial').setup({}) in you have a init.vim (no Lua) config
Minimal example file
No response
Minimal init.lua
No response
Additional context
No response

You can use a :help lua-heredoc
example:
lua << EOF
require("aerial").setup({})
EOF