Cleaning house and rename actual module to OceanParameterizations

This PR kinda serves as a review of PR #17 and reorganizes the code a bit, mostly trying to separate package/library source code utils and scripts.
I also tried to make the code follow Julia standards a bit more:

Julia modules should be in CamelCase so I renamed the submodules to follow this convention.
I noticed some submodules were including files from other submodules. In general I think each submodule should be self-contained and only include files in its own submodule (but could import stuff from other submodules via using and import).
Structs were generally not strongly typed. This violates two of the Julia performance tips: Avoid fields with abstract types and Avoid fields with abstract containers. I strongly typed those structs in case they're used in performance critical code. This doesn't change how they're used so no breaking changes!

The src/ directory was housing a mix of library function and scripts. I think in general src/ just hosts package of library code, mostly self-contained functions to be used by scripts (which live outside of src/).
Won't merge until wind_mixing/uw_train_NDE.jl works as it before (just need some JLD2 LESbrary.jl data to test) on.

Sorry a lot of the docstrings were out of date.

Ah haha still infinitely better than no docstrings :rocket:


[SP] Make the interfaces on SP object avaliable after initialized

We need to make the API avaliable even the SP was paused, so that we can get the volume preview when stoped in the PR: https://github.com/crosswalk-project/realsense-extensions-crosswalk/pull/36

LGTM. Thanks!


GitHub Enterprise support + improved error handling. Fixes #84

Yeah it's tricky to test for sure, but given the way the library and GitHub works where it's only the URL that changes (it's still GitHub on the other side) there isn't much that differs. I've verified it.
I can update the README+CHANGELOG.

A couple of options from --help was added by this change, like --version and --branch. I suppose you just forgot to update it the last time.
I'll let you do the actual tagging and release ;)

Awesome perfect. Thanks for picking these up 👍


API link query

Hello
In case if I am using few computers, should I install your script in each, or it would be possible to install this script on one computer, and to get kind of API link for the others?
Thanks

No, you'll have to install the extension on every browser you use.
Please ask for help and support on the discord: https://discord.gg/RSAf7b5njt.


Fix broken link in getting-started

I think this fixes the 404 link in the getting started page not pointing correctly to first-steps.md

Thanks! :smile:


Service provider order and package auto discovery

I have an issue where I would like my package to get loaded after the rest of the Laravel application service providers however using auto-discovery it gets loaded first. I looked at the code that handles this and there isn't any way to change this besides digging deep into things and overriding core files.
I think there should be support for ordering service providers loaded through auto-discovery.
The "laravel-modules" package does this by adding an additional config value "order". See: https://github.com/AsgardCms/Platform/blob/3.0/Modules/Core/module.json

But where in your package would you define this "order"?
How could it work with other packages / competing orders?

It could be in composer.json with a weight value and lower weights would get loaded first or something like that

This isn't a bug, please move it over to https://github.com/laravel/ideas/issues

Please use this issue tracker only for reporting Laravel bugs.


Fix values in Limitations section

The current limitations values seem outdated.
See

1.3.x https://github.com/modcluster/mod_cluster/blob/1.3.x/native/include/mod_clustersize.h
2.x https://github.com/modcluster/mod_proxy_cluster/blob/main/native/include/mod_clustersize.h

Also, the PROXY_WORKER_MAX_ROUTE_SIZE macro that is referenced in JVMROUTESZ related comment is 64 in 2.4. but 96 in trunk. Can we set it to the macro directly?

Thanks @jajik and @jfclere

Also, the PROXY_WORKER_MAX_ROUTE_SIZE macro that is referenced in JVMROUTESZ related comment is 64 in 2.4. but 96 in trunk. Can we set it to the macro directly?

@jfclere ?

#define PROXY_WORKER_MAX_ROUTE_SIZE      96
In trunk (modules/proxy/mod_proxy.h)
#define PROXY_WORKER_MAX_ROUTE_SIZE     64
IN 2.4.x
So yes please ;-)


Can't locate shellwords.pl in @INC

For certain input data (when default coregistration fails) FreeSurfer calls a tool that requires an older version of Ubuntu.
https://mail.nmr.mgh.harvard.edu/pipermail//freesurfer/2014-May/038342.html
<EMAIL_ADDRESS>

So that would be FROM ubuntu:precise?
Can you foresee any downside?
I guess this is not a problem for FS6.0?


Use store for all primitive properties

This moves primitive properties into the store, and uses the store to seed the primitives object in the model that is used for lookup.
npm install may be needed to test this.
Dispatching an action, e.g. PRIMITIVE_ADD_MARK, will be used to

create the primitive in the primitive store, with appropriate signal references
instantiate any signals in the signal store to match those references
set up streams in the signal store to handle mark interactions

Key known issues:

Channels are completely broken, because rules don't know how to update the store yet. I will need to pick @arvind's brain on the inner workings of rules to address this.
Dragging a mark on the stage has gotten slower; one of the store updates is causing react to re-render more than it needs to or should.

vectors for investigating the selection issue:

Manual inspection of model.view.model().scene().items between lyra2 and this branch shows the structure appears to be similar; but the logic in updateSelectedMarkInVega (hierarchy.findInItemTree) fails to find the mark.
Diffing the JSON created from model.manipulators() between lyra2 and this branch shows that the only difference appears to be in ID numbers, not content, so it's unclear what could be causing the rendered vega graph to diverge.

Heyas, @arvind found this https://github.com/vega/lyra/blob/lyra2/src/js/util/hierarchy.js#L102
When I modify that to
for (i = path.length - 1; i >= 0; --i) {

The selection works

Here's a formal TODO list for when this branch will be ready for review:

[x] Can select a mark by clicking on it in the sidebar
[x] Can delete an individual mark
[ ] Can "clear all"
[ ] Can bind data to mark channels (this may take a while, not sure how best to proceed)

@deathbearbrown mark instantiation, modification (excluding data binding) and removal (both individually and via "clear all") are now ready to test. deleteMark and clearScene actions are worth special note since they use redux-thunk and that's new for this project.

I don't know how you want me to file these. I'm just going to put multiples in a comment for now so it's not too overwhelming.
FILL:   Initial color state is set to black.  Should be default blue.  It's important to have those match for the user to know what they are editing without being explicitly told.

STROKE - the width isn't set, so if I change the color there is no update until I change the width. The property manager shows it as the middle of the graph instead of unset

Can't change the shape of a symbol

Maybe bug, requires more discussion about expectations:
steps to see what I see:

create a group with a mark in it, create a mark outside that group
add data to axis on each mark
delete group
the scales still exist but the legends disappear

Can't change a line's width

Pulse already has a non-zero stamp. this is the error I have  in console.
I'm going to do a fresh checkout to make sure that I don't have any crummy stuff in this branch from the last time I checked it out.

(OK on a clean branch.) I made a line graph. I changed the width of the stroke and it worked fine. I went to change the color and that didn't work and also the stroke property stopped working. Something wonky is happening so I'm gonna hold off until your next commit is in and then start again.

I'm getting a lot of pulse errors when using the signal inspectors, in general. I'll investigate but any further specific examples you can find are helpful!

@arvind what do these alignment properties for rect marks do?

Re: Rectangle spatial properties, they are currently broken in lyra2 as well. So likely not related to this PR, but we should investigate. Here's a GIF from an earlier commit demonstrating the intended behaviour. Basically, the horizontal and vertical position/sizing for rectangles is jointly determined by two properties: typically a start/end, a start/span, or a center/span. The drop downs determine which two properties are being used (and set _disabled on the others).

OK I have functional tested this like crazy.  All the marks are looking amazing.  I tested all properties. The only thing that isn't working is those drop downs on rect, but that's a known issue.

This is beautiful.  +1 functional test.  +1 code review.  I'm going to make sure we have an issue for the following known things:

RECT - spacial properties
ContentEditable react component


add: Shivanshu814

Description
[Include a brief description of the changes you've made]
Related Issues
[Include any related issues or pull requests that this PR addresses or is related to. Use GitHub's shorthand syntax to link to them, like #1234.]
Changes Proposed
[Describe the changes you've made in detail. Be specific and include any relevant code snippets.]
Checklist

[ ] I have read and followed the Contribution Guidelines.
[ ] All new and existing tests passed.
[ ] I have updated the documentation to reflect the changes I've made.
[ ] My code follows the code style of this project.
[ ] The title of my pull request is a short description of the requested changes.

Screenshots
Note to reviewers

@shivanshu814 Thank you so much for your contribution! Your efforts are greatly appreciated and will go a long way in improving our project. Please feel free to share it with others and Star the repo to help us grow even further!


fixes blackburns

About the PR

Why / Balance

Technical details

Media

[ ] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Changelog

sprites were fixed by Aidenkrz


Saved Model support: implements Model.saved_model that enables the user to serialize pred_step as a SavedModel

saved_model gets a sample input and a path, and serializes a pred_step as a SavedModel via jax2tf:
import numpy as np
import elegy
from flax import linen

x = np.random.uniform(size=(32, 6))

model = elegy.Model(module=linen.Dense(4))
model.init(x)

model.saved_model(x, "test_saved_model")

The SavedModel in this example would contain the following signature:

input: (None, 6)
output: (None, 4)

At the moment the previous code doesn't actually work due to google/jax#5915. However, fixing batch_sizes currently works:
import numpy as np
import elegy
from flax import linen

x = np.random.uniform(size=(32, 6))

model = elegy.Model(module=linen.Dense(4))
model.init(x)

model.saved_model(x, "test_saved_model", batch_sizes=[1, 8])

This creates 2 signatures:

input: (1, 6)
output: (1, 4)

and

input: (8, 6)
output: (8, 4)

Codecov Report

Merging #171 (fb484cc) into master (7f4a781) will decrease coverage by 0.08%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##           master     #171      +/-   ##
==========================================
- Coverage   87.21%   87.13%   -0.09%
==========================================
  Files         135      136       +1
  Lines        7360     7374      +14
==========================================
+ Hits         6419     6425       +6
- Misses        941      949       +8

Impacted Files
Coverage Δ

elegy/model/utils.py
82.14% <82.14%> (ø)

elegy/model/model_core.py
85.46% <86.95%> (+0.23%)
:arrow_up:

elegy/model/model_test.py
100.00% <100.00%> (ø)

elegy/nn/linear.py
97.29% <100.00%> (+0.07%)
:arrow_up:

elegy/data/data_adapter.py
57.14% <0.00%> (-6.28%)
:arrow_down:

elegy/data/generator_adapter.py
40.00% <0.00%> (-4.90%)
:arrow_down:

elegy/data/data_handler.py
71.08% <0.00%> (-1.33%)
:arrow_down:

elegy/data/torch_dataloader_adapter.py
84.00% <0.00%> (-0.91%)
:arrow_down:

elegy/data/tf_dataset_adapter.py
85.10% <0.00%> (-0.90%)
:arrow_down:

elegy/types.py
93.18% <0.00%> (-0.81%)
:arrow_down:

... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f4a781...fb484cc. Read the comment docs.


Fix resume

Fixes #878.

Hey, @stefanminke, I used your proposed changes from #878, could you check, if the changes are correct?

Hi, yes it looks good.
Thank you!


Remove throw from ~IlmThread

It's unnecessary and it causes a warning.
Signed-off-by: Cary Phillips<EMAIL_ADDRESS>

The throw in the constructor should suffice to ensure the class is never used on the platform in question.


[Bug] [core] Fix the resource leak for core module

Purpose of this pull request

Fixes #1531
Check list

[x] Code changed are covered with tests, or it does not need tests for reason:
[x] If any new Jar binary package adding in you PR, please add License Notice according
New License Guide
[x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

is it better to use finally block to close stream?

@dailidong SGTM


[x264] build failure when trying to build only release version of the library

Host Environment

OS: Windows
Compiler: Microsoft (R) C/C++ Optimizing Compiler Version 19.23.28106.4 for x8

To Reproduce
Steps to reproduce the behavior:
vcpkg.exe install --overlay-triplets=..\.vscode\custom-triplets x264:x86-windows-static
Custom triplet contents x86-windows-static.cmake:
set(VCPKG_TARGET_ARCHITECTURE x86)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)
set(VCPKG_BUILD_TYPE` release)

Failure logs
See logs for more information:
vcpkg\buildtrees\x264\install-x86-windows-static-dbg-err.log
Call Stack (most recent call first):
scripts/cmake/vcpkg_build_make.cmake:164 (vcpkg_execute_build_process)
scripts/cmake/vcpkg_install_make.cmake:24 (vcpkg_build_make)
ports/x264/portfile.cmake:33 (vcpkg_install_make)
scripts/ports.cmake:94 (include)
Error: Building package x264:x86-windows-static failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: x264:x86-windows-static
Vcpkg version: 2019.09.12-nohash
Contents of install-x86-windows-static-dbg-err.log
make: *** No rule to make target 'install'.  Stop.
Additional context
The issue seems to be in fact that scripts/cmake/vcpkg_build_make.cmake ignores the fact that we are building only release during install step (while does the proper thing during build step). Here is a patch I applied locally:
install_step.patch.txt

Hi @bogdanlytvynovskyi thanks for posting this issue and your suggestion to fix this issue.
The PR has been merged. Please update vcpkg and try to use it again.


test if utility is functionally equivalent to traditional asciidoc syntax

investigate asciidoc syntax support

Thanks for the PR, good work. This will give us a good place to start for functional equivalence.

@manaswinidas I think we're good here, what do you think?

@LightGuard yes


Should not use labelled functions

As that is not idiomatic. It should instead by t-first (eg the List goes in the first position in functions in the List module)

So this involves changing all the functions and tests so that the t goes in first place, removing labels
Eg:
let myFunc = (~count: int, ~sep:string, v: t)

would become
let myFunc = (v: t, count: int, sep:string)

The one exception is where there are multiple values of the same type:
let add = (a : int, b : int)

should become:
let add (a : int, ~addend:int)

to ensure they can't be accidentally misused


luaPackages.magick: fix libMagickWand-7.Q16HDRI path on darwin

Darwin uses .dylib as the extension for dynamic libraries. Replace the hardcoded extension with stdenv.hostPlatform.extensions.sharedLibrary to fix magick support on darwin.
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[x] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Priorities
Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review run on aarch64-darwin 1

4 packages marked as broken and skipped:

lua51Packages.magick
lua52Packages.magick
lua53Packages.magick
lua54Packages.magick

2 packages built:

luajitPackages.magick
vimPlugins.image-nvim


Feature Detail on the BIOM summary doesn't scroll

... but if opened in a new window it works.

This works in the new release, at least using Google Chrome Version 70.0.3538.102 (Official Build) (64-bit), see:

BTW I still prefer to open in another window ...


chore: Update gradle version to 7.2

Hey, I just made a Pull Request!
References to other issues or PRs
Fixes: #20
Brief description of what is fixed or changed
Updated the grade version from 5.0 to 6.8.3 to support the latest OpenJDK version.

@vaish3496 This issue was created some time back. We can actually upgrade Gradle to the latest version.

@vaish3496 This issue was created some time back. We can actually upgrade Gradle to the latest version.

Are you suggesting v7.2? I'm not sure if that is stable. Or do you want me to upgrade it to v7.1.1?

@vaish3496 Yes, we can go to 7.2.

Getting this error while trying to upgrade Gradle to 7.2. Not sure how to deal with this.

@vaish3496 You would need look at all breaking APIs for gradle , migration guide etc. to make build.gradle compatible to 7.2

change protobuf-gradle-plugin version to 0.8.17
        classpath 'com.google.protobuf:protobuf-gradle-plugin:0.8.8'
on line 6 in build.gradle

@vaish3496 Build is failing with deprecated feature of gradle still being used in build file. We would need to upgrade them all. Check build logs for the same.

can we go up to Gradle 8.0 ? @vaish3496

@vaish3496 Build is failing with deprecated feature of gradle still being used in build file. We would need to upgrade them all. Check build logs for the same.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. But works well with 7.2.

can we go up to Gradle 8.0 ? @vaish3496

8.0 is yet to be released. 7.2 is the latest.

@vaish3496  https://github.com/odpf/firehose/pull/101/checks?check_run_id=3469295039#step:5:29

@sumitaich1998 @vaish3496 Some tests are failing. Can you folks check and fix those.

@ravisuhag @sumitaich1998 @lavkesh all checks have been passed. Thanks!

👍🏻 lgtm

@vaish3496 Thanks for the contirbutions. Looking forward to more :)


Collect SSN for Members

/Users/jackyalcine/.asdf/installs/ruby/3.1.2/bin/ruby -I/Users/jackyalcine/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/rspec-core-3.12.0/lib:/Users/jackyalcine/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/rspec-support-3.12.0/lib /Users/jackyalcine/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/rspec-core-3.12.0/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...................................................

Finished in 2.15 seconds (files took 1.47 seconds to load)
51 examples, 0 failures


useIdle should reset on delay change

In useIdle hook, currently the events prop is used as dependency array of useEffect hook.
The ms prop is not included in dependencies which doesn't allow changing delay once component is mounted.
So, instead of
...
useEffect(() => {
  ...
}, events)
...

it should be
...
useEffect(() => {
  ...
}, [ms, events])
...

:tada: This issue has been resolved in version 8.1.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add 'SYSTEM_THREAD(ENABLED)'

https://docs.particle.io/reference/firmware/photon/#system-thread

Seems to resolve issue with Max / Min reset not working when wifi disconnected
[https://github.com/jgc/falco_sw_photon/issues/20]


V2 : binary STL files cannot be loaded properly via URI parameter

Expected Behavior
This URL works:
https://www.jscad.xyz?uri=https://github.com/jscad/sample-files/raw/master/stl/testcube_10mm.stl
Actual Behavior
The cube does not render and the viewer is very strange. The conversion by the stl-deserializer failed.
Steps to Reproduce the Problem

Click on the URL above

Specifications

Version: V2
Platform: web
Environment: all browsers

Fixed via #764


Implement also for Facebook

Send the rest of the posts to Facebook (according to scheduling, if available)

Had lots of confusion with the authorization tokens, but now I've figured out that the ones in the Excel file are for HarperCollinsPublishersUK official facebook page, while the one for the test page I need to get myself.


complete the health check streams and update the handlers

When the streams processor processes the health check tasks from the controller-health-check topic populated by the light-scheduler, we need to complete it and project the health check result with another Kafka streams.
We also need to update some of the handlers to get the health check info from the streams state store instead of in memory hash map for the cluster mode.

 complete the health check streams and update the handlers light-controller master


File Formats in different schemas, but with same name are tainted by each other

Provider Version
0.25.10
Terraform Version
0.13.x
Describe the bug
File Formats in different schemas (in the same database), but with the same name are tainted by each other.
In our use case, we have multiple file formats named "DATALAKE_PARQUET_FORMAT" in 3 different schemas under the same database: SCHEMA1, SCHEMA2 & SCHEMA3.
When a run a terraform plan, it keeps showing the files formats in the schemas SCHEMA1 & SCHEMA2 as tainted, since they are registered with schema = SCHEMA3 instead of the correct schema.
(likely because they come back from the results of SHOW FILE FORMATS in an alphabetical order?).
Expected behavior
File Formats should respect schema when checking state.
Code samples and commands
See explanation above.
Additional context
This issue is very similar to the issue: https://github.com/chanzuckerberg/terraform-provider-snowflake/issues/290, which was fixed in the PR: https://github.com/chanzuckerberg/terraform-provider-snowflake/pull/291, just for file formats instead of stages...

Can you share the terraform you're writing to create the file formats? I'm seeing https://github.com/chanzuckerberg/terraform-provider-snowflake/blob/main/pkg/snowflake/file_format.go#L564 which looks right to me...

Hi @alldoami Yes, this is the row I fixed in my PR...  It was already approved and merged by @edulop91 :-)
Thanks for taking the time to look into it!


Fetch new list of commands

:+1:

:+1:

@qix would you mind updating the list again and adding a 'use strict'; in the file to generate the commands? It would be great to have some tests for the new commands but I'd say it's fine to merge them first and add tests for each command later.
What do you think @bcoe @erinspice?

Superseded by #814


DiscoClient cannot be included in NetBeans

As far as I can tell, NetBeans still expects to build on Java 8. This means that discoclient, which uses that Java 11 HttpClient cannot be included.
If the foojay backend has some documentation we could re-implement discoclient using some Java 8 APIs/libraries like the Apache Commons HttpClient 3.1 which NetBeans already includes.

@geertjanw while there is value in having discoclient as a separate JAR for other clients, I think we should just include the code from discoclient in the plugin. It makes no sense to have another "3rd party jar" wrapped in the module.
It might make a bit of sense if discoclient were to be published on Maven Central which, right now, it is not.
I'm also mildly wondering if it's worth having these 2 codebases under different licenses. In particular, using discoclient introduces the need for a NOTICE file entry.

Sure. Makes sense.

I think this can be closed. discoclient has a java8 branch and is included as a JAR and will be on Maven Central in the future. There isn't much actionable in this issue anymore.


[Proposal] add option to update pull request by rebase

As we all know, do git rebase main(base_branch) and then force push to the head branch also can set the base of head branch update to the base branch, So I think maybe we can also add it as
a optional way on ui. Thanks

Didn't we have implemented that? You can find a button named Update when the pull request base head branch moved.

That button does a git merge base_branch to update the PR, not a rebase.

@lunny I know it, It use git merge base_branch, but as we all know, git rebase base_branch can do same thing. So like we provide more than one way to merge a pull request. I think we can also provide more than one way to update pull request also, Thanks.


Dump stream to file

Hello,
I`m trying to dump  network stream video to a file. I add the options in the player
        Player.AddOption("--demux=dump");
        Player.AddOption("--demuxdump-file backup.ts");

but not make the file.
I try to add the option in the AssemblyInfo.cs like:
[assembly: VlcSettings(@"libvlc", "-I", "--dummy-quiet", "--ignore-config", "--no-video-title", "--no-sub-autodetect-file", "--rtsp-tcp", "--file-logging", "--logfile=log.txt", "--demux=dump","--demux-dump-file c:/backup.ts")]
but then a Exception is thrown " Exception thrown: 'System.Windows.Markup.XamlParseException' in PresentationFramework.dll"
Is possible to dump a stream ? How?
Thanks,

Maybe you can try ':demux=dump' in load media method

The LoadMedia method, only accept one parameter. How I can add :demux=dump in load media?
For example:
vlcPlayer.LoadMedia(new Uri("http://<IP_ADDRESS>"));
Thanks

You can try VlcPlayer.LoadMediaWithOptions(string, params string[]).


Fix the fully typed examples

What
Update the examples in the fully typed docs learn page to use contract IDs and the latest CLI command structure.
Why
It appears it's way out of date and these commands won't work. One of the example commands no longer exist. And the use of hashes for contract IDs is no longer supported.

Preview is available here:http://developers-pr398.previews.kube001.services.stellar-ops.com

Preview is available here:http://developers-pr398.previews.kube001.services.stellar-ops.com


Installation error

OCTIS version: 1.13.1
Python version: 3.12.3
Operating System: Windows

Description
Trying regular to install octis using a regular pip install commands and I keep getting an error that I think is related to a deprecated module.
What I Did
pip install octis

Collecting gensim==4.2.0 (from octis==1.13.1)
  Using cached gensim-4.2.0.tar.gz (23.2 MB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: nltk in c:\users\bns36\appdata\local\programs\python\python312\lib\site-packages (from octis==1.13.1) (3.8.1)
Requirement already satisfied: pandas in c:\users\bns36\appdata\local\programs\python\python312\lib\site-packages (from octis==1.13.1) (2.1.4)
Requirement already satisfied: spacy in c:\users\bns36\appdata\local\programs\python\python312\lib\site-packages (from octis==1.13.1) (3.7.4)
Collecting scikit-learn==1.1.0 (from octis==1.13.1)
  Using cached scikit-learn-1.1.0.tar.gz (6.8 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "C:\Users\bns36\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "C:\Users\bns36\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\bns36\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 112, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "C:\Users\bns36\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 77, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\bns36\AppData\Local\Programs\Python\Python312\Lib\importlib\__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 995, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "C:\Users\bns36\AppData\Local\Temp\pip-build-env-38b2a8n9\overlay\Lib\site-packages\setuptools\__init__.py", line 16, in <module>
          import setuptools.version
        File "C:\Users\bns36\AppData\Local\Temp\pip-build-env-38b2a8n9\overlay\Lib\site-packages\setuptools\version.py", line 1, in <module>
          import pkg_resources
        File "C:\Users\bns36\AppData\Local\Temp\pip-build-env-38b2a8n9\overlay\Lib\site-packages\pkg_resources\__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Hello there,
Downgrade to python version <12 and it will be resolved. You may take a look at link for a github issue.

Hello there,
Downgrade to python version <12 and it will be resolved. I am using it within a conda virtual environment with Python3.10 in it and it works fine. You may take a look at a link for the same issue in a different .
You may find a few more trouble after this which I have posted at #125 .
Hope it helps.

I did a manual install offline using the setup.py file after modifying the requirements and it worked, though some functions that I haven't tested may be broken. It needs to be updated though - serious roadblock for people working with topic modeling as OCTIS is mentioned in the seminal paper on BERTopic.

Hello there,
Downgrade to python version <12 and it will be resolved. I am using it within a conda virtual environment with Python3.10 in it and it works fine. You may take a look at a link for the same issue in a different .
You may find a few more trouble after this which I have posted at #125 .
Hope it helps.

I did a manual install offline using the setup.py file after modifying the requirements and it worked, though some functions that I haven't tested may be broken. It needs to be updated though - serious roadblock for people working with topic modeling as OCTIS is mentioned in the seminal paper on BERTopic.

Downgrading to relevant versions and updating the requirements.txt file here would suffice that.


"Evaluate In Debug Console" without moving focus to Debug Console

I tend to use "Evaluate In Debug Console" to develop like the function "Evaluate Selection" in Matlab.
But moving focus to Debug Console seriously cumulatively takes my time a lot.
Please add these functions...

disable moving focus.

evaluate the current line of keyboard cursor focus if there is no selection.

Or is it possible for me to add these functions as extension ?
thank you for consideration.

I edited by myself in local vscode and found that it is vscode function not vscode-python


Legacy provider should be able to load implementations from other providers

PR #14326 is currently blocked because there is currently no way for the legacy provider to use implementations from other providers. This is a particular issue for PBKDF1.
A proposed solution is to have a config value for legacy which contains a list of the providers to load into the legacy provider internal libctx.

A proposed solution is to have a config value for legacy which contains a list of the providers to load into the legacy provider internal libctx.

Err that sounds a bit horrible and very confusing..
This does not sound like it would allow you to use provider_load to do the equivalent operation?
We should somehow be able to create a libctx and have a legacy + default provider inside it. And that should be enough.
The problem here is that we really need to use the 'parent' libctx when doing any internal fetches. (This is of course different to the fips provider) and the default provider doesnt have a parent..

Providers in general should have a way to use implementations from other providers.
That was a core concept from the original discussions in that providers should be able to provide an implementation using other providers as building blocks.
Of course our default provider doesn't need to do that and the fips provider shouldn't (in our configurations) although it is conceptually permitted if the other providers are themselves fips validated but that is a separate item and not our focus.
The legacy provider should be able to access algorithm implementations.
I don't see this as a provider providing a list of other providers is wants to use - it really should just be the ability to walk through fetching from all the available providers from within a provider so there isn't something separate to go and configure for this context - i.e. this shouldn't be seen as something "special" for legacy - it should just be how providers work except for those that are explicitly set to not use other providers (i.e. are standalone) like the current FIPS provider and the current default.
Note that the default provider also should all for this as there are places where it makes sense to do that with the addition of other digests and ciphers etc.

A list of providers exist inside a libctx, not inside another provider. The issue here is that the legacy provider has its own internal libctx, which has no providers inside it.

Even if the legacy provider (which is assumed to be a loadable module in this comment) wouldn't have its own library context, the default library context that it can reach "belongs" with the libcrypto it's linked with, not with the libcrypto that loaded it, and those libcrypto versions may differ.

Providers in general should have a way to use implementations from other providers.
That was a core concept from the original discussions in that providers should be able to provide an implementation using other providers as building blocks.

My recollection is that we also talked about the view of each provider as a self contained box, where the only interface with it is the set of dispatch tables.
We did briefly discuss having the fetch functions offered by libcrypto as upcalls for providers to use.  It didn't take us many seconds to realize that we would have to make all associated functions into upcalls...  we backed away from that idea, if I remember correctly because we couldn't see all the ramifications.
There was also a time where the idea to pass around a dispatch table instead of the name of an implementation was floated for a brief time (just like we pass around an EVP_MD in pre-3.0), I think around the time that the redesign of the RAND functionality started.  That idea never caught.
So what we currently have in code is support for provider as relatively self-contained blackboxes, where only provider that are built into libcrypto can make exception because we know for a fact that they have access to the exact same libcrypto as they were loaded from.

I don't see this as a provider providing a list of other providers is wants to use

Agree, but that's not the idea here.  The idea is that libcrypto, when loading a provider, could pass a set of "interesting other providers" to the legacy providers.  Through configuration files, we already have all the mechanisms to do this, all that would be needed would be for the legacy provider to actually have them loaded in its (separate) library context.

it should just be how providers work except for those that are explicitly set to not use other providers (i.e. are standalone) like the current FIPS provider and the current default.

The current default provider can use implementations from other providers, because its library context is the same and always will be the same as the loading libcrypto's.  With the self-contained blackbox view of providers, it's the default provider that's "special".
To actually make it possible for one dynamically loaded provider to use any implementation from any other provider, we would need to make quite a lot of internal changes, or assume that a dynamically loaded provider is always linked with the same libcrypto as that one that loaded it...  at least as far as I can see.

We should somehow be able to create a user libctx and have a legacy + default provider inside it. And that should be enough.

It was argued that someone might want to use PBKDF1 from legacy together with the FIPS module, for FIPS compliance...

The problem here is that we really need to use the 'parent' libctx when doing any internal fetches.

One thing that we have said is that these kinds of internal structures (in fact, any opaque structure) must never be shared across the libcrypto <-> provider boundary.  As soon as you call EVP_MD_fetch() with that "parent" libctx from inside a provider, you get into the problem with different libcryptos that I mentioned above.

I went back to the documents we wrote when we did the grand design, and found this snippet in the OpenSSL Strategic Architecture:

A Provider may be entirely self-contained or it may use services that are provided by different providers or the Core Services. For example, an application may use the cryptographic primitives for an algorithm implemented by a hardware accelerated provider, but use the serialisation services of a different provider in order to export keys into PKCS#12 format.

This supports both @t-j-h's view and mine.
The question here is rather that we obviously want the legacy provider to "use services that are provided by different providers", the question is how to technically go about it.  The base idea that's been presented here is to pass down a list of provider names to the legacy provider (through the OSSL_PARAMS passed in the provider init call), and let the legacy provider do what it wants with that.

One thing that we have said is that these kinds of internal structures (in fact, any opaque structure) must never be shared across the libcrypto <-> provider boundary. As soon as you call EVP_MD_fetch() with that "parent" libctx from inside a provider, you get into the problem with different libcryptos that I mentioned above.

Passing a list of providers just seems very wrong and inflexible.

I can only think of 3 possible solutions to this problem (maybe there are more):

Disallow having the provider and the application linked against different versions of libcrypto. This means (for example) you couldn't have the provider dynamically linked, and the application statically linked.

or

Provide upcalls for all possible crypto operations that the provider wants to do.

or

Have the provider load whatever providers it needs into its own internal libctx

IMO, (2) is just a non-starter. There would be a huge proliferation of upcalls. We would effectively have to mirror large parts of the public API as upcalls. Basically anything that ends up doing a fetch which we think a provider might want to do would have to be an upcall. This would be difficult to maintain - and would be a nightmare as far as a provider author is concerned.
(1) is possible technically but would be highly restrictive and significantly devalue the flexibility in what we have produced. I don't think we should go down this route.
That leaves (3). It's simple. It works. It meets the requirement (from @t-j-h's comment above "providers should be able to provide an implementation using other providers as building blocks").

We are just talking about for the legacy provider here...
For (2) wouldnt just having the digest and cipher fetch be enough?

(3) may be simple from a implementation view, but it does seems like a confusing concept for a end user..
If we could get to the internal libctx then couldnt the core also add additional providers manually?

We are just talking about for the legacy provider here...

The legacy provider has sparked the discussion, but the problem is universal for all providers that are (1) not built-in and (2) linked against libcrypto. Whatever we do for the legacy provider should be the approach we recommend for others.

For (2) wouldnt just having the digest and cipher fetch be enough?

That's still horrible, and only works for the legacy provider.

Another variant of (3):
Pass the name of a config file as a config parameter to the provider. Legacy can then initialise its internal ctx with OSSL_LIB_CTX_load_config. In the event that no config file name is supplied then use the default file name.

Pass the name of a config file as a config parameter to the provider.

Errrrr, we already make the config parameters for the provider at hand through the core get_params() call.  The mechanism you talk about would be quite redundant, and it means the provider will have to work out exactly which section it should be concerned with to get the data.  More work for the provider author to achieve something that we already have available.

Errrrr, we already make the config parameters for the provider at hand through the core get_params() call. The mechanism you talk about would be quite redundant, and it means the provider will have to work out exactly which section it should be concerned with to get the data. More work for the provider author to achieve something that we already have available.

I think you misunderstand my proposal. I am seeking to re-use the existing mechanism for automatically loading providers and setting a default property query string that exists for setting up the default libctx in libcrypto, for setting up the legacy provider internal libctx. In order for that to happen the legacy provider would have to call OSSL_LIB_CTX_load_config and provider a filename. In its simplest incarnation I am envisioning that the filename can be exactly the same as the default config file. Hence the internal legacy libctx would be a duplicate of how the default lib ctx is initially set  up. That might be good enough, but I could also imagine that you may want to use a different config file for the legacy provider. So a config parameter could be the name of another config file to use.
So something like:
    OSSL_PARAM core_params[2], *p = core_params;
    char configfile[80];
    OSSL_LIB_CTX *libctx = OSSL_LIB_CTX_new();

    strncpy(configfile, CONF_get1_default_config_file(), sizeof(configfile));

     *p++ = OSSL_PARAM_construct_utf8_ptr(
                    OSSL_PROV_PARAM_PROVIDER_CONFIG_FILENAME,
                    (char **)configfilename,
                    sizeof(configfilename));
    *p = OSSL_PARAM_construct_end();

    if (!c_get_params(handle, core_params)) {
        ERR_raise(ERR_LIB_PROV, PROV_R_FAILED_TO_GET_PARAMETER);
        return 0;
    }

    OSSL_LIB_CTX_load_config(libctx, configfilename);

Ah ok, yeah that makes sense.

Could we (also?) provide an upcall to libcrypto core from a provider to list the provider names currently loaded in the core libctx? And perhaps also the default property query in the core libctx? And use these to load the providers inside the provider's libctx and do the fetch?

This of course assumes that there are similar providers available in the core libcrypto and the provider's libcrypto but...

Could we (also?) provide an upcall to libcrypto core from a provider to list the provider names currently loaded in the core libctx?

We could, yes.

This of course assumes that there are similar providers available in the core libcrypto and the provider's libcrypto but...

Well, if that isn't true, then there's not a lot we can do about that.

We could, yes.

Although, the advantage of the config file approach is that that also deals with any provider specific configuration parameters. If providers need some kind of mandatory configuration, just loading the provider by name might not be enough.

One thing that might help to keep in mind: the project has absolutely no experience with what people want to do here. I think Matt's #3 is simple and understandable. If in a year or two common patterns emerge, then you can add new functions to implement them.

I think we could provide both. For our legacy provider the upcalls would be enough, for third party they would get the config file option as well if the upcalls would not be enough for them.

My reason is that for the simple use-cases there should be no need to do some extra configuration to obtain reasonable behavior.

My reason is that for the simple use-cases there should be no need to do some extra configuration to obtain reasonable behavior.

Actually, I think with my proposal there also would be no need to do extra configuration in the simple case. If you don't do anything then you get exactly the same initialisation as the global default libctx gets. Which is probably enough for most cases.

Yep, it only would not work properly if the application does some explicit provider loading/unloading and default property query manipulation.

What happen if no configuration is used, would the legacy provider then get the fall backs?  I expect this case will be fairly common.

Back when we designed this, it was my understanding that most providers would have access to libcrypto and that the FIPS provider was special in that it wouldn't.  We seem to have reversed this.
I see this as a major problem.  PBKDF1 isn't doing anything special, it should be able to fetch whatever digest/cipher it wants and just use it.  No extra configuration, no mucking around, no special cases.  It should just work.
I view Matt #1 option as the best resolution:

Disallow providers from statically linking libcrypto.  They are shared libraries, there is little need to include libcrypto.
If you want to statically link libcrypto into you application, statically link the (non-FIPS) providers and register them as built in.

If we can be absolutely sure that there is only one per-process writable data section for every individual shared library, across all platforms, then sure.  I've never felt safe assuming that's always the case.

Disallow providers from statically linking libcrypto.

That's nothing we can control, unfortunately (expect mysterious crashes if someone does this).  This also disallows providers linked with, say, libcrypto.so.3 to be loadable by libcrypto.so.4...  something that we, again, can't really control.

If you statically link libcrypto into your application and a provider calls a shared version for an algorithm, things should still hang together

Errr, how does that work?  Say that we have an application that's statically linked with libcrypto, and we have the legacy module legacy.so -> libcrypto.so.3, what is it that would make legacy.so bind symbol references to the libcrypto bits that were linked into the application rather than those in the loaded libcrypto.so.3?

We should not be double loading providers as a solution here. A provider should be able to work with the set of providers that an application has put in place however it is that the application has put them in. That information should be made available for provider implementation to be able to use.
There are contexts where you cannot have multiple loading because initialisation triggers real-world events (like having to go to a computer room and enter pins on secure pinpads or insert smartcards and perform quorum based authentication) which double loading will trigger - as effectively we end up breaking the concept of logical one-time initialisation.
This shouldn't be a major issue to sort out - and if it turns out it is a major issue to sort out then we have something wrong in our design that needs fixing. This was always part of the objective in the restructuring of things to allow for the concept of multiple providers being able to cooperate. The suggested extra loading basically isn't delivering on that - as the providers are still effectively conceptually isolated from each other - and then you'll end up with a combination of providers loading providers loading providers etc which is not the right solution.
We also had a core objective to be able to have providers from different versions of OpenSSL being able to be operational in the one application (forward and backwards compatibility for the provider ABI within a major version of OpenSSL).

@t-j-h, correct me if I'm wrong, but that assumes that all the parts loaded in a process that are linked libcrypto are linked to the same libcrypto.  Correct?

Our design approach made no such assumption - nor would I have expected it to do so - because we wanted to be able to have a mix of versions of openssl - that was a substantial part of the reorganisation so we don't have to have the same versions of openssl across all of the providers. We discussed precisely this issue in our original design meetings - and how much "fun" that was going to be in order to make sure we could get that result.

Solution 4:
The core provides a callback function which has the same signature as OSSL_init_provider. The legacy provider loads a new built-in provider into its libctx using that callback as the entry point. The callback has the responsibility of registering a query function which merges the results of all query functions for all loaded providers. i.e. the callback is a "proxy" for the providers loaded into the libctx used by the application's version of libcrypto.
Solution 4a:
Same as solution 4, except the core provides a callback which provides a way of registering a different built-in provider proxy for each of the providers loaded into the application's libctx. This way you don't have to have a query function that merges the query results.

It sounds like the query function would need to become an iterator, making it possible to return more than one OSSL_ALGORITHM array, unless you enjoy allocating memory and merging tables...

If a provider module does EVP_MD_fetch(libctx, "MD5", NULL), what would libctx be?  If we consider a provider module linked with a different libcrypto than the application that loaded it, I don't understand how it connects back to the applications libcrypto, and thusly "magically" has access to all other providers that the application has loaded.  If the provider module doesn't make its own libctx, i.e. libctx = NULL, it would use the default library context of the libcrypto that the provider module is linked with, not the one that the application is linked with, right?  I'm not able to wrap my brain around the magic that would somehow make one translate into the other...

A provider, if linked with openssl >= 3.0, must always use its own libctx, otherwise nothing but a mess can happen. And I really cannot see how, with the current design, the provider could "simply" fetch implementations from the caller's libctx. That might be a completely different version of libcrypto.
Do we want to set the provider API in stone? Or how do we expect the compatibility between libcrypto versions and providers to work?

Should a 3.0 provider work with 3.1 libcrypto? (Probably yes, with some functionality limitations.)
Should a 3.0 provider work with 4.0 libcrypto? (No idea, but I would not expect it to, otherwise we could not make any ABI breaking changes in the provider API in 4.0.)
Should a 3.1 provider work with 3.0 libcrypto? (Now that's an interesting question and I do not have answer.)
Should a 4.0 provider work with 3.0 libcrypto? (I would not expect that.)

If, and only if, both 1. and 3. combinations were supposed to work and 2. and 4. it might be possible to achieve this but I suppose it would be harder and more limiting than we can think of now.

How about this for a possible way forward.  Create a new opaque structure:
struct ossl_core_lib_ctx {
    int nonce;
    OSSL_LIB_CTX *libctx;
};

typedef struct ossl_core_lib_ctx OSSL_CORE_LIB_CTX;

Pass a pointer to one of these to the provider init.  If the provider wants to use the calling libcrypto, it calls whichever libcrypto asking for the library context:
OSSL_LIB_CTX *ossl_lib_ctx_from_core(const OSSL_CORE_LIB_CTX *core)
{
    if (core->nonce matches)
        return core->libctx;
    return NULL;
}

A NULL return would mean that there is another libcrypto in the way and that direct calls aren't possible.  Any other return means calling libcrypto directly is fine.
In effect, we're explicitly allowing direct callback to the libcrypto that loaded a provider so long as the provider and application play nicely.
Extra thoughts:

The nonce field could be a version field to allow compatible libcrypto's to operate -- there are inherent dangers here with state which would be avoided by a nonce.  The nonce doesn't have to be random, the address of any function or variable is sufficient to distinguish copies of libcrypto.
The function might better be:

int ossl_lib_ctx_from_core(const OSSL_CORE_LIB_CTX *core, OSSL_LIB_CTX **lib_ctx)
{
    if (core->nonce matches) {
        *lib_ctx = core->libctx;
        return 1;
    }
    *lib_ctx = (OSSL_LIB_CTX *)"";
    return 0;
}

which makes using the failure return more fatal.

I had a different thought last night.
It seems that the biggest problem we're facing is that OSSL_LIB_CTX is at least potentially libcrypto version sensitive, as we may make subtle changes, even in update releases.  That's normally perfectly fine...  however, the issue that's come up is that we may want to pass it around between different libcrypto versions and be possible to access by them all.  How that would work seemed like magic to me yesterday.
It seems, though, that the gist of the problem is to have some control over what functions (from which libcrypto) can safely access the library context data.  A library context created by one particular libcrypto version is only safely accessible by functions in that same libcrypto version, right?
One way we can solve that is to couple together the internal data and the functions to handle it in a super-structure, which must be set in stone once and for all, something like this:
typedef struct ossl_lib_ctx2_st OSSL_LIB_CTX;
struct ossl_lib_ctx_st {
    struct ossl_int_lib_ctx_st *data;          /* This structure is what is currently called struct ossl_lib_ctx_st */
    OSSL_LIB_CTX_METHOD *functions;
};

OSSL_LIB_CTX_METHOD is troublesome, though, as it would also have to be set in stone just as much as this super-structure as different libcrypto versions will have to be able to use the functions in that method.
However, we already have an extensible structure for this, so:
struct ossl_lib_ctx_st {
    struct ossl_int_lib_ctx_st *data;          /* This structure is what is currently called struct ossl_lib_ctx_st */
    OSSL_DISPATCH *functions;
};

Exactly what functions might appear in functions, I currently do not know...
We could also, for the benefit of providers that want to have their own library context and still have access to the other providers loaded by the "parent" libcrypto, make it possible to chain library contexts together, making them a (short) searchable list:
struct ossl_lib_ctx_st {
    OSSL_LIB_CTX *parent;
    struct ossl_int_lib_ctx_st *data;          /* This structure is what is currently called struct ossl_lib_ctx_st */
    OSSL_DISPATCH *functions;
};

A few notes:

We may choose to make the super-structure public.  It's not absolutely necessary, but we have a tendency to be more disciplined regarding not changing structures if they are public.
We don't have to make OSSL_LIB_CTX more public than this.
The function numbers and declarations for functions going into that particular dispatch table can be kept private, since they are of no concern to anyone other than libcrypto.  We may choose to make the public, but that's a matter of choice, not of necessity.
Every time some libcrypto functions wants to use the dispatched functions, they will have to search for them on the spot.  It's not possible to build one method structure from them, as that method structure may differ from one libcrypto version to another.  This is extra churn, but would be worth the while to meet the requirements that @t-j-h mentions.

@levitte, a neat idea but I think the problem goes deeper:
libctx = ...magic as above...
mdfetch = find(libctx->function, MD_FETCH_FUNCTION)
EVP_MD *md = mdfetch(libctx, "sha256", "...")

When we use md, won't we need to know how the evp_md_st structure is laid out?

My thought is unfinished.  I was thinking, though, that the functions in libctx->functions would be at a deeper level, such as having a pointer to ossl_method_construct() in there...  it does mean that another internal structure (OSSL_METHOD_CONSTRUCT_METHOD) must be set in stone or become an OSSL_DISPATCH as well, though, so this needs a bit more careful thought.

Could we add an up call like evp_generic_fetch that takes the provider handle libcrypto passes to the provider?  The owning libcrypto can get the libctx out of this.
A provider that wants to fetch from it's containing libctx would call the up call passing pointers to functions in the libcrypto it is linked against.  This avoids all the concerns I mentioned about using fetched things above.  It also seems pretty clean -- we'd need to write EVP_x_fetch wrappers for providers and link against them instead.  The only problem is identifying the provider via a call chain that doesn't pass enough information, but it does have a libctx inside the provider so we might be able to include what we need there.
Have I missed something here?


NIFI-12538 - Add map Record Standalone Function

Summary
NIFI-12538 - Add map Record Standalone Function
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[ ] Apache NiFi Jira issue created

Pull Request Tracking

[ ] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[ ] Pull Request based on current revision of the main branch
[ ] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Thanks for the review @exceptionfactory, your comments make a lot of sense and I pushed a commit with the changes you recommended.


Create more level chunks

Create more interesting level chunks, of #2

I am going to redesign the ones I have now and I would like to streamline the process


Task/package reference

Update references to old package name and repos (GitHub & Travis) and document v3.0.0 in changelog

@ndelangen Some refs were not up-to-date, I think all should be good with this one.


[ooyala] picking formats with m3u8 manifests (was: picking formats)

In my youtube-dl.conf I have -f "18/43/34/h264-sd/flv/mp4-sd/mp4". When trying to download from some sites (particularly ooyala), I still get the highest resolution.

How do I pick the res where 480x270, 640x360, and 1280x720 are available? (See below)
My config also fails when getting soundcloud files. -fwav and -fmp3 also fail. It would be nice if there was some way to disable the -f or all the youtube-dl.conf options.

[Vice] life-as-a-truck-stop-stripper-full-length: Downloading webpage
[Ooyala] 5yeW9mazqGRQO-Kig7MadzZ-kh4nbmDM: Downloading webpage
[Ooyala] 5yeW9mazqGRQO-Kig7MadzZ-kh4nbmDM: Downloading webpage
[download] Destination: Vice_-_Every_Woman_-Full_Length-5yeW9mazqGRQO-Kig7MadzZ-kh4nbmDM.mp4
ffmpeg version 1.2.4 Copyright (c) 2000-2013 the FFmpeg developers
  built on Oct  3 2013 05:43:15 with gcc 4.8 (Debian 4.8.1-10)
  configuration: --prefix=/usr --extra-cflags='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security ' --extra-ldflags='-Wl,-z,relro' --cc='ccache cc' --enable-shared --enable-libmp3lame --enable-gpl --enable-nonfree --enable-libvorbis --enable-pthreads --enable-libfaac --enable-libxvid --enable-postproc --enable-x11grab --enable-libgsm --enable-libtheora --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libx264 --enable-libspeex --enable-nonfree --disable-stripping --enable-libvpx --enable-libschroedinger --disable-encoder=libschroedinger --enable-version3 --enable-libopenjpeg --enable-librtmp --enable-avfilter --enable-libfreetype --enable-libvo-aacenc --disable-decoder=amrnb --enable-libvo-amrwbenc --enable-libaacplus --libdir=/usr/lib/x86_64-linux-gnu --disable-vda --enable-libbluray --enable-libcdio --enable-gnutls --enable-frei0r --enable-openssl --enable-libass --enable-libopus --enable-fontconfig --enable-libpulse --disable-mips32r2 --disable-mipsdspr1 --dis  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[mpegts @ 0x237b8c0] Estimating duration from bitrate, this may be inaccurate
[aac @ 0x24cae40] max_analyze_duration 5000000 reached at 5015510 microseconds
[aac @ 0x24cae40] Estimating duration from bitrate, this may be inaccurate
[mpegts @ 0x2811e80] Estimating duration from bitrate, this may be inaccurate
[mpegts @ 0x2856960] Estimating duration from bitrate, this may be inaccurate
Input #0, hls,applehttp, from 'http://player.ooyala.com/player/ipad/5yeW9mazqGRQO-Kig7MadzZ-kh4nbmDM.m3u8?js=1':
  Duration: 00:27:24.00, start: 0.000000, bitrate: 0 kb/s
  Program 0
    Metadata:
      variant_bitrate : 868000
    Stream #0:0: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p, 640x360, 11.99 fps, 23.98 tbr, 90k tbn, 47.95 tbc
    Metadata:
      variant_bitrate : 868000
    Stream #0:1: Audio: aac ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 130 kb/s
    Metadata:
      variant_bitrate : 868000
  Program 1
    Metadata:
      variant_bitrate : 128000
    Stream #0:2: Audio: aac, 44100 Hz, stereo, fltp, 137 kb/s
    Metadata:
      variant_bitrate : 128000
  Program 2
    Metadata:
      variant_bitrate : 442000
    Stream #0:3: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p, 480x270, 11.99 fps, 23.98 tbr, 90k tbn, 47.95 tbc
    Metadata:
      variant_bitrate : 442000
    Stream #0:4: Audio: aac ([15][0][0][0] / 0x000F), 32000 Hz, stereo, fltp, 68 kb/s
    Metadata:
      variant_bitrate : 442000
  Program 3
    Metadata:
      variant_bitrate : 1530000
    Stream #0:5: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p, 1280x720, 11.99 fps, 23.98 tbr, 90k tbn, 47.95 tbc
    Metadata:
      variant_bitrate : 1530000
    Stream #0:6: Audio: aac ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 130 kb/s
    Metadata:
      variant_bitrate : 1530000
Output #0, mp4, to 'Vice_-_Every_Woman_-Full_Length-5yeW9mazqGRQO-Kig7MadzZ-kh4nbmDM.mp4.part':
  Metadata:
    encoder         : Lavf54.63.104
    Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1280x720, q=2-31, 11.99 fps, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 1530000
    Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo, 130 kb/s
    Metadata:
      variant_bitrate : 868000
Stream mapping:
  Stream #0:5 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
No longer receiving variant 1ze=    1219kB time=00:00:08.01 bitrate=1247.0kbits/s
No longer receiving variant 2ze=    1508kB time=00:00:09.96 bitrate=1239.8kbits/s
frame=  331 fps= 48 q=-1.0 size=    2257kB time=00:00:13.86 bitrate=1333.7kbits/s
ERROR: Interrupted by user
frame=  340 fps= 48 q=-1.0 Lsize=    2349kB time=00:00:14.21 bitrate=1354.1kbits/s
video:2116kB audio:227kB subtitle:0 global headers:0kB muxing overhead 0.273598%
Received signal 2: terminating.

Yes, that is exactly the same request as in Debian bug 753309, only here it is framed as a bug (it does't work, I expect it would) while in the Debian bug it is framed more as a "new feature".
A workable method would be to parse the output of avprobe, or parse the m3u8 file yourself (it doesn't seem to be a complicated format).

Thanks for the suggestion. For Ooyala videos, m3u8 information are extracted now.
@kickerofelves The second URL http://www.mmafighting.com/2014/2/2/5370376/ufc-169-post-fight-show contains a Youtube video and an Ooyala video. However, youtube-dl detects the Youtube video first, so the Ooyala video is not downloaded. We'll have to wait until the feature requested in #5587 implemented to correctly extract both videos.


Update README.md

Added a step by step instructions to run the pipeline

@melsiddieg Great idea!
I added the contributors names but not their description, feel free to do so. I didn't know how to appropriately define each person contribution.

@melsiddieg I noticed some layout issues that I fixed.
Please, can you review again and if it is ok for you, approve the review so that I can merge?
Thanks!

I think the contributions are clear. I developed the SV pipeline, contributed to hail setup and script, and the migration to Nextflow DSL2. Phil contributed to testing and configuration while you and Britanny worked as a pair and should be familiar with each other contributions.

I added contributions, I will run them through Phil and Brittany for approval before merging this request.


NIST PubID scheme: Support document type "Insert"

The document type "insert" is not supported by the 2021-08 scheme.

e.g. "NBS CIRC 25insert"
https://nvlpubs.nist.gov/nistpubs/Legacy/circ/nbscircular25insert.pdf

@mico we also need to add this to the implementation.


FEAT add extend by mini-batch

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (d4306c4) to head (57d5c6b).
Report is 10 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##              main       #22    +/-   ##
==========================================
  Coverage   100.00%   100.00%
==========================================
  Files            3         5     +2
  Lines          110       218   +108
==========================================
+ Hits           110       218   +108

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests


fix: upgrade deps

:tada: This PR is included in version 1.18.4 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


feat: Promote import/no-nodejs-modules to an error

Rule documentation: https://github.com/import-js/eslint-plugin-import/blob/main/docs/rules/no-nodejs-modules.md
This PR promotes the above rule from a warning to an error in our base eslint configuration, but exempts imports from "events" due to the sheer number of uses in the codebase.
This PR also "pre-fixes" violations across the codebase by disabling the rule in packages / inline as needed without making any code changes. Code changes may be made in the future as appropriate to remove rule overrides. But for now, the primary goal is to prevent future violations.
Breaking Change
This PR promotes an existing warn-level rule to an error.

I think all the packages we opted out of this here are server code run under node, which makes sense.
This makes me wonder if we want different base configs for server and browser code (shared code might get an even more strict combination?).

I think all the packages we opted out of this here are server code run under node, which makes sense.
This makes me wonder if we want different base configs for server and browser code (shared code might get an even more strict combination?).

I think you're right. Once we've moved to a world where we have more definite mono-repo (i.e. release group) workspaces, I want to explore hierarchical configs a bit more - let mono-repos define their own baselines derived from a shared config.


Remove Duplicate Element  From a Un-Sorted Liked list(Temporary Buffer Not Allowed)

🚀 Feature
(A clear and concise description of what the feature is.)
Have you read the Contributing Guidelines on Pull Requests?
(Write your answer here.)
YES
Motivation
(Please outline the motivation for the proposal.)
Its a Classical Problem Which is often Asked In Interviews
Pitch
(Please explain why this feature should be implemented and how it would be used.)
We need to Remove the duplicates from the liked List with using any extra space apart from only storing values
i.e Extra Buffer Not Allowed

I want to work on this issue  Using C and Java as a part of GsSoc21


Routing validation (unique ingress host and random default route)

Ref:

#1585

This PR fixes #1585 adding a check for already existing routes on the deployed ingresses, and it adds a short random suffix to the default route to prevent clashing with app with the same name, but in different namespaces.

@andreas-kupries I added the suggested changes, but to fix the tests I needed to refactor a bit the validateRoutes func, so you would probably need to have a look again at it.
The problem was that the update of an app were also failing, because I was not checking the "ownership" of the ingress.
So now I'm checking if the route/ingress is owned by the app that is going to request the routes, and it will fail only if the route is owned by another app.


[extractor/facebook] Improve extraction

Please follow the guide below

Before submitting a pull request make sure you have:

[x] Searched the bugtracker for similar pull requests
[x] Read adding new extractor tutorial
[x] Read youtube-dl coding conventions and adjusted the code to meet them
[x] Covered the code with tests (note that PRs without tests will be REJECTED)
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x]  Except for the commit by @kikuyan who separately agreed to either this or the below, I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Bug fix
[x] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
This PR replaces #29796 which was orphaned when @kikuyan's account was deleted.
The PR makes the following fixes and improvements to the Facebook extractor:

improves title, add description, preferring metadata in ld+json if present
adds another getter for attachments (from PR #30496, issue)
as ld+json VideoObject type seems to be gone since PR was first made, adds SocialMediaPosting type as a fallback
when web pages are analyzed as playlists but consist of single video, treats the pages as regular video pages to extract metadata well (examples: tests 7, 9, 10)
makes test skip for now: TestDownload.test_Facebook_8: the page cannot be parsed (get ERROR: This video is only available for registered users.) while it can be opened by a browser.

The PR makes the following improvement to the extractor/common.py:

adds SocialMediaPosting for author extraction in InfoExtractor._json_ld() (replaces PR #30513).

Resolves #29421, resolves #23627, resolves #23180, resolves #14156.
Resolves #30472, resolves #30474, resolves #30650, resolves #30681.
Closes #29796 (superseded)
Closes #30496 (superseded)
Closes #30513 (superseded).

If you're interested, I found this:
https://facebook.com/video/video_data?video_id=309868367063220
but it only contains the media, not metadata, so its only marginally useful

Hm, it seems those links are useful after all. If you do this:
GET /video.php?v=309868367063220 HTTP/1.1
Host: www.facebook.com

You only get the SD media. You can get HD media like this:
GET /video.php?v=309868367063220 HTTP/1.1
Host: www.facebook.com
Accept: text/html

but its hidden in JSON hell.

Hey @dirkf - do these extractors get images? I tried --write-info-json, but I didn't see any image links. It seems that the link I found expires, and I was wanting to embed a Facebook thumbnail on another page. I noticed Twitter just rehosts the Facebook image on their own server, is that the only way?

The extractor does this
            thumbnail = html_search_meta_non_empty(
                ['og:image', 'twitter:image'], webpage, 'thumbnail', default=None)

Yeah, those a both timestamped, the oe parameter 626DE546 =<PHONE_NUMBER> = April 30 2022
I guess I need to junk my Facebook extractor. That is annoying, but no permalink images is a dealbreaker. If you find anything, let me know. Thanks.

Facebook video downloading has issue now which was already filed before in several related/duplicated issues:

ERROR: Cannot parse data

But it's working fine few days ago. Hopefully, this PR could be merged soon, since I badly needed the fix also. Thanks :)

Are you saying that this happens with the PR code? Or (as I hope) that the PR code is still valid and fixes "Cannot parse data"?
That message basically means that the extractor tried all the tactics it knows to extract from the page and none worked.


The Application does not work with Kubernetes Nodes when deployed with NodePort Mode service

Hi Abhishek,
I have been trying to reach out to you regarding my failed attempts on Kubernetes. I have tried implementing the process using your video. But I am not able to run the application through NodePort Mode and Kubernetes Node EXTERNAL-IP. Please let me know if there is any setting I am missing.
I have got no other medium to talk to you so I am using this issue section to let you know the problem I am facing while trying to deploy your application.
Please let me know what is your thought process. I am not able to ping the node EXTERNAL-IP through browser or curl command.
Please your guidance is required.
I am looking for forward to hearing from you soon.

@ubaidsworld same here too facing same issue

@ubaidsworld Yeah tried it will work
but how abhishek is doing without modifying the SG

@ManojKumarN001 I wonder the same thing. There were possibility that author wants us to discover that thing. Or may be wants us to subscribe his channel so he can earn from what he is teaching. I had tried contacting him but there is no response from his side. So raising issue was the last attempt I made, thinking that the author might respond. But he turn off deaf ears on the issues people raises. So my doubt became a little strong about my second thought over subscription.
But I am glad that you got it resolved from my suggestion.


Introspection Context

Issue to implement an Introspection Context, as described in https://github.com/strimzi/strimzi-ui/blob/master/docs/Architecture.md#introspection and https://github.com/strimzi/strimzi-ui/tree/master/client/Contexts/Introspect

Requires GQL library ( #38 ), testing issues to be completed

I'll take this one - I don't have permissions to assign etc.

I'll take this one - I don't have permissions to assign etc.

@scholzj would you be able to add @pmuir to the Strimzi-ui devs group?

@pmuir You should have an invite in your mail.


IDbAsyncProvider vs IAsyncProvider

The code on this page discusses and uses IDbAsyncProvider which I believe is this one
https://docs.microsoft.com/en-us/dotnet/api/system.data.entity.infrastructure.idbasyncqueryprovider?view=entity-framework-6.2.0
Through MOQ exceptions requesting IAsyncProvider I found out the page should be updated to use that class, which I believe EF Core 6 uses, this one: https://docs.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.query.iasyncqueryprovider?view=efcore-6.0

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1a04a5e4-3060-2331-f78d-8b45064b8983
Version Independent ID: 673d9406-3fa1-a9eb-f5cc-a431155e0260
Content: Testing with a mocking framework - EF6
Content Source: entity-framework/ef6/fundamentals/testing/mocking.md
Product: entity-framework
Technology: entity-framework-6
GitHub Login: @ajcvickers
Microsoft Alias: avickers

This content is for EF6, not EF Core. It would therefore not be correct to update it like you suggest.

Right, sorry, this a result of the naming mess with .NET 6.

EF6 Onwards Only - The features, APIs, etc. discussed in this page were introduced in Entity Framework 6.

Was thinking this is EF6 from .NET 6..

That product is called EF Core 6


Feature Request: Client Access Code

Hi @selvinortiz,
Invaluable plugin!
We found the client access link feature of the Craft 2 plugin IP Whitelist plugin really really useful.
It allowed us to send a unique link to a client which would automatically and their IP address to the allowed list. So rather than requiring (for each new location) to ask the client to google for their IP, add it to config, prepare a release, release let client know, you just send something like this:
https://staging.site.com?access=75hnf75hi38yryu588uh3jyy5huy678h3
Be great to see something like this in the already awesome Patrol!

@samhibberd I'm working on an update to the plugin and this feature request sounds like a great idea. I'll explore how I can bring this into the plugin.

@samhibberd This was added in the today's release https://github.com/selvinortiz/craft-plugin-patrol/releases/tag/3.1.0

Awesome. Nice one @selvinortiz

@selvinortiz this is working great! Just a quick follow up, it would be really useful if you could include the token link in the cp somewhere?

@samhibberd Where do you think it would be the best place for it? In the CP sidebar, or settings screen maybe? Patrol supports multiple tokens, so we would have to support multiple links.

@selvinortiz in the settings screen would be fine, but a separate section for access tokens might be cool as you could introduce a permission such that uses could access / share tokens but not the plugin settings. Just thinking out loud :)


How to serialize a list to a Json object

Hi,
I have a question regarding a list serialization. Since I could not find any pointers in the tests or docs. I would be great if someone could point me in the right direction.
Given I have a type like this:
type Post = {
    Id: Guid
    Title: string
    Body: string
} with static member ToJson(x:Post) = json{
        do! Json.write "id" x.Id
        do! Json.write "title" x.Title
        do! Json.write "body" x.Body
}

and a list of posts like this
 let post1 = {Id= Guid.NewGuid();Title="Title hello world post 1";Body="My body"}
 let post2 = {Id= Guid.NewGuid();Title="Title hello world post 2";Body="My body"}
 let listOfPosts = [post1;post2]

    listOfPosts
        |> Json.serialize
        |> Json.formatWith JsonFormattingOptions.Pretty

this would results obviously in this
"[
  {
    "body": "My body",
    "id": "0356c901-e1fa-4bae-8fe3-70061d2e05f3",
    "title": "Title hello world post 1"
  },
  {
    "body": "My body",
    "id": "292a275b-960e-46b4-b04b-c3e214c78b0d",
    "title": "Title hello world post 2"
  }
]"

But how can I write a custom list serialization function that would produce something like this to an object graph:
"{
  "0356c901-e1fa-4bae-8fe3-70061d2e05f3":{
    "body": "My body",
    "title": "Title hello world post 1"
  },
  "292a275b-960e-46b4-b04b-c3e214c78b0d":{
    "body": "My body",
    "title": "Title hello world post 2"
  }
}"

Thanks for any help.

Oh that's interesting! Sorry i've not got back to you sooner, I was away for the weekend, but that's an interesting question. I'll have to play around with it and have a think about that - but I'll try and get back to you soon 😄

Hi kolektiv & felixbecker,
May I ask you here? Because I think I am doing something like this issue (not very sure), and my (useless / full of bug) method is
e.g.
At first I have a Monkey type, and I hope I could serialize IDictionary<int64, Monkey>
#r @"..\..\..\chiron\src\Chiron\bin\Chiron.dll"
#r @"..\..\..\aether\src\Aether\bin\Aether.dll"
#r @"..\..\..\..\gdrive_anighost\fparsec\Build\VS14-PCL\bin\Debug\FParsec.dll"
#r @"..\..\..\..\gdrive_anighost\fparsec\Build\VS14-PCL\bin\Debug\FParsecCS.dll"

open Aether
open Aether.Operators
open Chiron
open Chiron.Operators
open System.Collections.Concurrent
open System.Collections.Generic
open System.Runtime.CompilerServices

[<CLIMutable>]
type Monkey =
  { sound : string
    height : decimal }

  static member ToJson (x : Monkey) =
    Json.write "sound" x.sound
    *> Json.write "height" x.height

So I define a extension method like this:
[<Extension>]
type ExtensionMethds2 () =
    [<Extension>]
    static member inline ToJson (dic: IDictionary<int64, Monkey>): Json<unit> =
        Seq.fold (fun (sn : Json<unit>) key ->
            let x = dic.[key]
            sn
            *> Json.write (key.ToString()) x
            ) (Json.init ()) dic.Keys

But when I tried to serialize it with:
let dictest = dict [123L, { sound = "ooh"; height = 1.53m }; 456L, { sound = "ooh"; height = 1.53m }]
Json.format (Json.serialize dictest)

Fsi.exe complains:
StatExtract.fsx(744,29): error FS0001: No overloads match for method 'ToJson'. The available overloads are shown below (or in the Error List window).
Possible overload: 'static member ToJsonDefaults.ToJson : x:unit -> Json<unit>'. Type constraint mismatch. The type
    IDictionary<int64,Monkey>
is not compatible with type
    unit
The type 'IDictionary<int64,Monkey>' is not compatible with the type 'unit'.
Possible overload: 'static member ToJsonDefaults.ToJson : x:bool -> Json<unit>'. Type constraint mismatch. The type
    IDictionary<int64,Monkey>
is not compatible with type
    bool

Even I add this code, it still not work...
type ToJsonDefaults with
    static member inline ToJson (x: IDictionary<int64, Monkey>) =
        x.ToJson()

And I take a look at the source code:
    let inline internal toJsonDefaults (a: ^a, _: ^b) =
        ((^a or ^b) : (static member ToJson: ^a -> unit Json) a)

But I couldn't understand this syntax... would you please give me some clue / url / doc for understand this syntax? (esp. ((^a or ^b) : (static member ToJson: ^a -> unit Json) a))
Thank you so much!!

Hi,
I got some workaround:
type Dicttc = Dicttc of IDictionary<int64, Monkey>

[<Extension>]
type ExtensionMethds2 () =
    [<Extension>]
    static member inline ToJson2 (dic: Dicttc): Json<unit> =
        let ddic = (fun (Dicttc d) -> d) dic
        Seq.fold (fun (sn : Json<unit>) key ->
            let x = ddic.[key]
            sn
            *> Json.write (key.ToString()) x
            ) (Json.init ()) ddic.Keys

type Dicttc with
    static member ToJson (x : Dicttc) =
        x.ToJson2()

let dictest = Dicttc (dict [123L, { sound = "ooh"; height = 1.53m }; 456L, { sound = "ooh"; height = 1.53m }])
Json.format (Json.serialize dictest)

The result is:
val it : string =
  "{"123":{"height":1.53,"sound":"ooh"},"456":{"height":1.53,"sound":"ooh"}}"

Hi again,
A refinement... no extension method required! (in the fact, extension method way caused another issues.)
type Ooo = Ooo of IDictionary<int64, Monkey[]> with
    static member ToJson (x: Ooo) =
        let xx = (fun (Ooo d) -> d) x
        Seq.fold (fun (sn : Json<unit>) key ->
            let xxx = xx.[key]
            sn
            *> Json.write (key.ToString()) xxx
            ) (Json.init ()) xx.Keys

let dictest2 = Ooo   (dict [123L, [|{ sound = "ooh"; height = 0.53m }|]; 456L, [|{ sound = "ooh2"; height = 1.53m }|]])

Json.format (Json.serialize dictest2)


basic.target reached before home directory is mounted with dm-crypt

Issue description
Similar to #12281 but not quite.
Assume you have something that runs before basic.target but requires files from $HOME which is mounted during login from a dm-crypt volume (example: systemd-tmpfiles-setup from #62813).
The problem is that basic.target is reached before $HOME is mounted and thus before the configuration files are available.
Steps to reproduce

Apply #62813
Have a machine with $HOME mounted using pam-mount during login from a dm-crypt volume
Stick the following into $HOME/.config/user-tmpfiles.d/test.conf: d %t/test 0700 - - - -
Log in
Notice how systemd-tmpfiles-setup.service was run but /run/user/$(id -u)/test is missing
Manually try running systemd-tmpfiles --user --create --remove --boot
Notice how /run/user/$(id -u)/test is now present

Technical details

system: "x86_64-linux"
host os: Linux 5.1.6, NixOS, 19.09.git.aa42aca8104M (Loris)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.2.2

I believe this is a duplicate of #31113 with a corresponding fix in #89457


Massive performance degredation from <IP_ADDRESS> -> <IP_ADDRESS>

I'm using kafka-net to send around 160,000 messages per second to Kafka, and after an upgrade from <IP_ADDRESS> to <IP_ADDRESS> I'm seeing a massive increase in CPU usage, to the point that kafka-net is unusable.
Before and After:

The only change in the two runs above was in the version of kafka-net.
To get to the level of throughput needed, messages are being batched into 4000 messages chunks before being written to SendMessageAsync(), which worked well under 0.8.
This was running on a VM with 8 cores @ 2.5 Ghz and 40GB RAM.
I have tried using the batching in 0.9 (_client = new Producer(router) { BatchSize = 4000 };) and writing messages one at a time, which improves things initially, but it fails to keep up with the data volume, and memory usage increases as it falls further behind:

I'm not sure where to go from here.
Any advice on how kafka-net should be used to reach throughput levels in the 100,000s of messages per second?
I'm currently looking at implementing a synchronous producer that forgoes the async stuff, as I believe that's where the performance issues are being introduced.
Thanks.

hi,
Thanks for the update, I also running a load and performance testing.
I came across with number of problems that was critical to me so i am trying to fix them in a new branch.
i have plans to make a few big changes during the next week and see their implications.
I will be happy if you could peek and tell me your opinion.
https://github.com/EranOfer/kafka-net/tree/ImprovePerformance
look on ConsumerShouldConsumeInSameOrderAsAsyncProduced_dataLoad It may help you.

I've been experiencing something similar, although with a much lower throughput level than described here. I've been using kafka-net inside an ASP.NET Web API application. I was experiencing frequent producer timeouts when publishing messages individually so I implemented a batching layer after experimenting in a console application.
Initial results were good, but after fully implementing it I could see timings for the production application's endpoints getting progressively worse:

Expected worst-case timings are in the 150-200 msec range. Restarting/recycling the application returns things to normal but timings start climbing again right away.
After experimenting with tweaks and profiling for hours, including some things that are mentioned in other issues here, I've given up and deployed version <IP_ADDRESS> which has been working perfectly so far.

Can you let me know how was the performance measurement done ? Was it via some tool like Sematext SPM?

Which tool did you use it to to run these Performance tests?

This was observed in a production system and timings are collected through Graphite. Our message sizes are fairly large (can be multiple MB, but in most cases will probably be 50-300 KB). I don't know if this would influence performance.


[BUG] 对某一个列设置了style: { border}，位于这个列后的所有列都生效了。

🐛 Bug Report
对某一个列设置了style: { border}，位于这个列后的所有列都生效了。
Steps To Reproduce

const wb = new ExcelJS.Workbook();
const ws = wb.addWorksheet('XYZ');
ws.columns = [
     { header: '', key: 'myBorder', width: 20, style: { border: { right: {style:'double', color: {argb:'FFFF0000'}}, }, },},
     { header: '', key: '', width: 20,},
]

The expected behaviour:
只对一个列的border完成style设置

pr#1573 seems to be talking about this problem, but has not been merged, after the merger should be able to solve the problem

I also encountered the same issue where I believe that two cells were using a shared style. My situation is as follows: I uploaded an xlsx file, parsed it using exceljs, set the background color for a specific cell, and then exported it. However, I noticed that many other cells also had their colors changed.
To address this issue, I suggest clearing the original styles before adding styles to the cells. The following code snippet may help you achieve that:
worksheet.eachRow({  includeEmpty: true }, (row) => {
    row.eachCell({ includeEmpty: true  }, (cell) => {
        cell.style = {};
     });
 });

But I still hope to provide an attribute to indicate that I want to set the styles of certain cells independently, instead of applying the styles to a group of cells. Additionally, it would be helpful to have a utility function, such as worksheet.clearStyle("A1:B5"), that allows me to remove the style association between specific cells.

I also encountered the same issue where I believe that two cells were using a shared style. My situation is as follows: I uploaded an xlsx file, parsed it using exceljs, set the background color for a specific cell, and then exported it. However, I noticed that many other cells also had their colors changed.
To address this issue, I suggest clearing the original styles before adding styles to the cells. The following code snippet may help you achieve that:
worksheet.eachRow({  includeEmpty: true }, (row) => {
    row.eachCell({ includeEmpty: true  }, (cell) => {
        cell.style = {};
     });
 });

But I still hope to provide an attribute to indicate that I want to set the styles of certain cells independently, instead of applying the styles to a group of cells. Additionally, it would be helpful to have a utility function, such as worksheet.clearStyle("A1:B5"), that allows me to remove the style association between specific cells.


Files cannot be downloaded from new teams’ custom app

                                let blob = new Blob([data], { type: 'application/octet-stream' });
                                var url = window.URL.createObjectURL(blob);
                                var a = document.createElement('a');
                                a.href = url;
                                a.download = decodeURIComponent(filename);
                                document.body.appendChild(a);
                                a.click();
                                window.URL.revokeObjectURL(url);

It works very well in Classic Teams. However, in new teams, it only works once for the first time, and you have to log out and log in again for it to work. However, it also works only once for the first time. What's the problem? Is it a bug?

안녕하세요 - 여기에 언급 된 것과 동일한 문제인지 알려주시겠습니까?

Multiple File Download

Download Files

Try running this source
In new teams, only one file is downloaded and there is no response after that.
You must completely close the app and log in again to download only one file.

Try running this source
In new teams, only one file is downloaded and there is no response after that.
You must completely close the app and log in again to download only one file.

Try running this source
In new teams, only one file is downloaded and there is no response after that.
You must completely close the app and log in again to download only one file.

Multiple File Download

Download Files

Hi @takeungrae - Could you please share a repro video along with repro steps, so that we can raise a bug for this?

Hi @Prasad-MSFT  - Any news? This is critical to our teams app...

Hi @Prasad-MSFT - Any news? This is critical to our teams app...

me too

@takeungrae It worked for me, when I do not provide the download attribute:
const link = document.createElement('a');
link.href = url;
try {
  document.body.appendChild(link);
  link.click();
} finally {
  document.body.removeChild(link);
}

@takeungrae It worked for me, when I do not provide the download attribute:
const link = document.createElement('a');
link.href = url;
try {
  document.body.appendChild(link);
  link.click();
} finally {
  document.body.removeChild(link);
}

Just as the browser asks for the option to allow multi-downloading, the same applies to web views. Since the web view of the Teams app does not allow multi-downloading, it is downloaded only once for the first time and does not download after that.

@samuel-weber , @takeungrae - Apologies. At the moment, we have not received any updates from the engineering team. However, we are actively communicating with them to stay updated on the progress. We have also made them aware of the seriousness of the issue. As soon as we receive any updates from them, we will notify you in this thread.
Thank you!

Hello, do you have any news about this issue? In classic teams everything works fine, but old version will be disabled soon and this is critical to our custom add in.

@bciernia - Apologies. Currently there is no ETA to share. We are following up with engineering team about this issue. We will update this thread once we hear from them. Thanks!

Hi @takeungrae , @samuel-weber , @bciernia - according to latest update from engineering team the fix has been released publicly. Please do verify and let us know.


Stop unnecessary TestFlight builds

We've been pushing to TestFlight on every commit for the release branches, this generates a lot of noise and doesn't help us version well. This PR should stop that from happening.

The file .circleci/config.yml is in the blocklist for external contributors. Hence, these changes are not tested by the CI pipeline active until the build is re-triggered by a core committer or the PR is merged. Please be careful when reviewing it.
/cc @mattermost/core-security @mattermost/core-build-engineers

Thank you 😊

/cherry-pick release-5.2

/cherry-pick release-5.2

Cherry pick is scheduled.