[GOBBLIN-228] Add config property to ignore fields in JsonRecordAvroSchemaToAvroConverter

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-228

Description

[x] Here are some details about my PR, including screenshots (if applicable):

Allow fields in the Avro schema to be skipped (set to null in the resulting record) if specified in config property
Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Update unit test
Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@abti please review


Handle undefined token gracefully

If there is no token, then this function errors.

Ping :+1:

Thanks for the PR @knownasilya! I don't think returning is a good idea. I would probably throw an error in this case, as it is an invalid argument. Would you mind making the update to error with a clearly defined message and also document that?
Thanks!

@dschenkelman is that good what you had in mind?

It looks good to me, I released it as v1.2.0.

Thanks.


[Updating ReadMe.md file]: Adding Voyage AI in the list

Voyage AI is also provide API's and is free.

Please assign me this task so that I can work on it as soon as possible under GSSOC'24.


Fresh install, can't upgrade db

I'm doing a reinstall on one of my boxes to the latest version. I've copied the steps for an install directly (cut/paste) so that they match, but am encountering the following errors:
(venv) root@ns1:/opt/web/powerdns-admin# flask db upgrade
Error: While importing 'powerdnsadmin', an ImportError was raised:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/flask/cli.py", line 256, in locate_app
    __import__(module_name)
  File "/opt/web/powerdns-admin/powerdnsadmin/__init__.py", line 4, in <module>
    from flask_mail import Mail
ModuleNotFoundError: No module named 'flask_mail'

Usage: flask [OPTIONS] COMMAND [ARGS]...
Try 'flask --help' for help.

Error: No such command 'db'.

I've declared my ENV variables as such:
(venv) root@ns1:/opt/web/powerdns-admin# export | grep -i flask
declare -x FLASK_APP="powerdnsadmin/__init__.py"
declare -x FLASK_CONF="../configs/production.py"
(venv) root@ns1:/opt/web/powerdns-admin#

nevermind...typo in config file - fat fingers


Asgiref 2.0.1 version conflict

Hi,
After upgrading stuff via pip, asgiref-2.0.1 was installed
asgi_ipc is the latest version as well : 1.4.2
Loading my project I got the error below
Obviously setup.py contains as dependency : asgiref~=1.1.2
ERROR [celery.utils.dispatch.signal:276] Signal handler <bound method DjangoFixup.on_import_modules of <celery.fixups.django.DjangoFixup object at 0x757c89b0>> raised: ContextualVersionConflict(asgiref 2.0.1 (/usr/lib/python3.6/site-packages), Requirement.parse('asgiref~=1.1.2'), {'asgi-ipc'}) Traceback (most recent call last):   File "/usr/lib/python3.6/site-packages/celery/utils/dispatch/signal.py", line 271, in send     response = receiver(signal=self, sender=sender, **named)   File "/usr/lib/python3.6/site-packages/celery/fixups/django.py", line 84, in on_import_modules     self.worker_fixup.validate_models()   File "/usr/lib/python3.6/site-packages/celery/fixups/django.py", line 122, in validate_models     self.django_setup()   File "/usr/lib/python3.6/site-packages/celery/fixups/django.py", line 118, in django_setup     django.setup()   File "/usr/lib/python3.6/site-packages/django/__init__.py", line 27, in setup     apps.populate(settings.INSTALLED_APPS)   File "/usr/lib/python3.6/site-packages/django/apps/registry.py", line 116, in populate     app_config.ready()   File "/usr/lib/python3.6/site-packages/channels/apps.py", line 14, in ready     check_all()   File "/usr/lib/python3.6/site-packages/channels/package_checks.py", line 22, in check_all     module = importlib.import_module(package)   File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module     return _bootstrap._gcd_import(name[level:], package, level)   File "<frozen importlib._bootstrap>", line 994, in _gcd_import   File "<frozen importlib._bootstrap>", line 971, in _find_and_load   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked   File "<frozen importlib._bootstrap_external>", line 678, in exec_module   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed   File "/usr/lib/python3.6/site-packages/asgi_ipc/__init__.py", line 2, in <module>     __version__ = pkg_resources.require('asgi_ipc')[0].version   File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 984, in require     needed = self.resolve(parse_requirements(requirements))   File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 875, in resolve     raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (asgiref 2.0.1 (/usr/lib/python3.6/site-packages), Requirement.parse('asgiref~=1.1.2'), {'asgi-ipc'})

How did you upgrade stuff via pip? asgiref 2.0 is a future release that is not compatible with any of the current Channels packages, so you shouldn't just upgrade it independently - this is why the version is pinned like it is.

I use python packages update website to check versions updates, then I create archlinux pkgbuilds with new versions. With AUR repository, this way all my machines are updated with the base distro upgrade tool yaourt.
I understand that my update method is not mandatory, but nothing in the asgiref description page (https://pypi.python.org/pypi/asgiref/2.0.1) inform that the release is a beta one. Maybe you should add some disclaimer about that fact in the description ?
Anyway thanks for your support
Cheers

It is not a beta release - it is just incompatible with the current version of asgi_ipc. If you are making packages for a distro, you should put the version dependencies into the package metadata so that the distribution's package manager knows this and will solve it correctly.

Ok, understood. I just don't figure out the purpose of this stuff out of the channel usage, so excuse my ignorance


fix: cortex config for demo

Why
I just cloned a fresh repo to my WSL ubuntu distro and was not able to run the cortex container
See:

What
Cortex config according to official docs: https://cortexmetrics.io/docs/configuration/configuration-file/

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Db2 Modules don't show up in catalog

module id/name updated here too: https://github.com/cloud-native-toolkit/terraform-gitops-db2u-operator/commit/4ccee32f3f6a758b32dca879e6a76009e228f433

DB2 metadata has been fixed.  All DB2 modules publishing metadata correctly and now show up in the catalog as expected.
Released:

DB2U Operator
DB2 OLTP Service
DB2 Database Instance

Still in development:

DB2 Warehouse (hasn’t been touched in a while)
DB2 Database Schema


[release-1.16] Fix istioctl pc secret cert validity not accurate

This is an automated cherry-pick of #44929

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

unit-tests-arm64_istio_release-1.16
b2b3a52e7700938191a5bd909aaac02449c1e755
link
true
/test unit-tests-arm64

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

unit-tests-arm64_istio_release-1.16
b2b3a52e7700938191a5bd909aaac02449c1e755
link
true
/test unit-tests-arm64

unit-tests_istio_release-1.16
b2b3a52e7700938191a5bd909aaac02449c1e755
link
true
/test unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Testing Suite

Off the back of #1, it would be good to have a testing suite in place. Seperate issue created for discussion.

How would we like tests to work? There is a test folder currently; do we want to have tests for all components?

@49Indium Ideally we should test things that are within our control. For example how the events are parsed from the ical, rather than testing if a message is sent.

I've added tests (that pass) for Haiku. Will add some for the other cogs soon.

Looking at this further, it seems that we might need to relocate some of the logic of some commands to their own functions (not within the object) to be able to test them effectively without simulating discord (see here).

linking this library for if/when we ever get the time. This looks to be the "best" (most up-to-date and still live) discord mocking library: https://github.com/CraftSpider/dpytest


Check if tokens are returned before attempting to create token entity

Fixes a bug where an error is thrown if an access token is not returned by the server, such as a scenario where login is called with an empty request object (we only expect id_token in the response)

Coverage decreased (-0.02%) to 80.974% when pulling 22a5358328a2acd90752937dad132cbc6093e44b on Fix-token-caching into c6db556ceef9012bc1852dcf27e9d46fae4c2046 on dev.

Agree about tests, otherwise looks good!


TypeError: querystring_1.default.encode is not a function when searching for movie

Describe the bug
When I call the following function
const movies = await tmdb.search.movies({ query: 'Babylon' });
I get this error
TypeError: querystring_1.default.encode is not a function in search.ts:49
However, it seems the code is working in my console as I can log stuff related to the search results. See Additional information why that might be the case
To Reproduce
Steps to reproduce the behavior:

Call tmdb.search.movies with some query parameter
See error

Expected behavior
A search result should be returned
Desktop (please complete the following information):

OS: [Windows 11]
Browser [firefox]
Version [110]

Additional information
I also get the following warning
Module "querystring" has been externalized for browser compatibility. Cannot access "querystring.encode" in client code. See http://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.

Ahh I see. Initially my thought process when creating this wrapper was to only use it server side. It seems like in your case you're using on the client ?

That is the case indeed


feat: id option for addScriptTag

Resolves #5166

Look like something got messed up when rebasing. Considering starting from scratch and opening a new PR from a fresh branch.

Fixed the conflicts. @jschfflr let me know if I can do anything else. Thanks.


Directionality of interactions

Hi there
As you mentioned in the tutorial, the interactions are not symmetric. So, I am wondering how we can infer the directionality of the interactions.
Best

Hi @Pedramto89,
The directionality of the interactions can be found from the "interaction_input" of CellPhoneDB database. You can access it from https://www.cellphonedb.org/, and search by the partners of interactions
Please let us know if this doesn't help.
Best,
Batu


Doc fixes for upstream

These are only the documentation changes, but I also took the liberty to rename some variables in some stack effect declarations, purely for better readability and consistency.

I'm not sure about the vector stack effect change since u and v are frequently used for vectors.
See, for example:
https://en.wikipedia.org/wiki/Vector_notation#Operations

OK, but the word is v+n, not u+n. See what I mean?

Well, I think the desire was to have the same first arg whether it had 1-arity or 2-arity, so if you follow that:
: vadd ( v w -- x )
: v+n ( v n -- x )

But generally, most math seems to use u + v or u * v for operations... ?

Cherrypicked the first three commits.  @erg can weigh in on the math.vectors change?

Yes, and I only changed things for consistency so that in vneg, vnot, vsqrt etc. you actually see the "v" parameter. Same for v^n, v/n, and so on. Purely a cosmetic fix.

u+v and such are left intact.

ahh, gotcha.  @erg?

P.S. thanks for the contributions!!!

Happy to help!

It seems that @erg has left the building : ))

Ha. I'm here, I forgot to reply!
Last patch seems fine!

Okay, pulled the last commit, thanks @AlexIljin!


TLS Runtime Issues for GitHub Artifacts built on Windows

Hello 😊 We have discovered a weird runtime TLS bug happening for artifacts built on the GitHub Windows runners. We have an example repository showcasing a simplified example here:
https://github.com/Thomas-Avery/test-rustls-platform-verifier
You can find the built artifact here:
https://github.com/Thomas-Avery/test-rustls-platform-verifier/actions/runs/10378079748
The error when running this artifact:
PS C:\Users\Colton\Desktop> .\test-rustls-platform-verifier.exe
[2024-08-22T17:19:28Z ERROR rustls_platform_verifier::verification::windows] failed to verify TLS certificate: invalid peer certificate: InvalidPurpose
Error: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("httpbin.org")), port: None, path: "/ip", query: None, fragment: None }, source: hyper_util::client::legacy::Error(Connect, Custom { kind: Other, error: Custom { kind: InvalidData, error: InvalidCertificate(InvalidPurpose) } }) }

So apparently this is CERT_E_WRONG_USAGE. Can you get a packet capture of this going wrong (in case there are any middleboxes between you and httpbin.org)?

@ctz @complexspaces I've captured two requests with Wireshark. One is a successful request with cURL (the cURL on Windows PowerShell), and the other is the failed request sent with the simplified example artifact here.
Within the Wireshark file:
The packets for the successful request are row #'s: ~22 - 44
The packets for the failed request are row #'s: ~66 - 81
Both requests are GET requests to: https://httpbin.org/
In case there is sensitive info within the file, I've uploaded the Wireshark file here: https://send.bitwarden.com/#MyCGYTnaekmYl7HUAQ4OhQ/Ah16xdCywkQdMkVuxW1fQg
This was run on Windows 11 Pro on ARM.
Microsoft Windows
Version 23H2 (OS Build 22631.4037)

Thanks, that looks pretty normal to me (real certificate from a real CA, in CT).
Looking more closely at the involved code I think we might be missing zero termination on the rgpszUsageIdentifier?

I'm not immediately sure if that's it. The CTL_USAGE structure takes a pointer to an array and the length of the array. It should not be looking for a NUL element to mark the end of the valid EKU array. As for the string itself, the docs don't specify but I would have expected this to crash/fail on any input if this was a case of Windows reading a garbage EKU value due to a missing terminator:

The LPSTR type and its alias PSTR specify a pointer to an array of 8-bit characters, which MAY be terminated by a null character.

I can run your test on an ARM64 Windows 11 VM later today and see if that makes a difference. If it doesn't I'll extract the certificates from the capture you provided and see if I can narrow it down by providing the exact chain 👍

My parsing of rgpsz is rg array of p pointer to sz nul terminated string, though I'm not sure if the hungarian notation is more or less trustworthy than anything else. If I'm right, and the matching gets done by strcmp, the behaviour will vary from compilation-to-compilation, platform-to-platform, in a highly unpredictable way.

I spent a while looking at this tonight and I manged to reproduce it. This was caused UB from a missing NULL terminator like @ctz suspected, but in a fun non-obvious way. With a fresh look it makes sense: how would Windows know where the string ends?
I was interested in the actual cause though, so I dug further.
The CERT_USAGE_MATCH we pass into the system is determined at compile time, as are all the pointer offsets (we only make pointers into const structures here):
 let usage = CERT_USAGE_MATCH {
            dwType: USAGE_MATCH_TYPE_AND,
            Usage: CTL_USAGE {
                cUsageIdentifier: ALLOWED_EKUS.len() as u32,
                rgpszUsageIdentifier: ALLOWED_EKUS.as_ptr() as *mut LPSTR,
            },
        };

The first problem, but that actually wasn't breaking things, was the way the string pointer was getting passed in. &str is a fat pointer so we were actually passing the equivalent of *const u128 into windows, slice metadata included. This didn't break anything though since the array only had one element and Windows wouldn't read past the actual string's data.
The second problem was more interesting. To finally reproduce the error, I actually had to set $CARGO_HOME="C:/Users/runneradmin/.cargo" because the broken binary would only be compiled that way when the path string of windows.rs was a specific length. My default username is shorter then runneradmin, hence failing to reproduce it at first.
I opened up both the binary from CI and my local build in Binary Ninja to look at the callsite for differences. Right away something jumped out. Here's what a correct initialization of our CTL_USAGE structure looks like:

Now the binary compiled by GitHub actions:

You can see that its getting a .reloc offset pointer that is just completely wrong in the GHA test binary. Its reading past the end of the &str const and into other random, merged strings. The reason this previously worked fine is because, AFAICT, the string would always* merge differently and result in those garbage bytes not being there, and instead a NUL terminator from the string's section layout being right after the EKU string data. I even checked our production 1Password builds and it accidentally lines up there too 😅. That is the "fun" of UB though: entirely absurd chances and results.
Running the bad binary with syscall tracing confirms the hypothesis, where you can see Windows is receiving an invalid eku:

As a result of that invalid EKU, the valid one in the certificate isn't in the allow-list anymore and it then fails. Small mystery solved 🎉
As an aside: This stopped working unconditionally at codegen time in Rust 1.78, for unknown reasons. Prior to that it worked with older rustc versions despite having the runneradmin path. The incorrect code also seems to work again in the latest nighty but this is just a random datapoint because it doesn't matter, and needs fixed regardless.

Please take a look at https://github.com/rustls/rustls-platform-verifier/pull/127 which contains all of the fixes.

Thanks so much for the quick turnaround with a fix for this!
Just chiming in to confirm that the fix works for artifacts built in GitHub runners:

This is excellent, thank you @complexspaces and @ctz for the help, the fix, and the detailed information!
🎉

This is now released on crates.io as versions 0.3.4 and 0.1.2.
Thank you again for the really helpful bug report! This issue would have probably continued to lurk for much longer otherwise.


What was wrong with this prompt?

Type: Bug
I want a web page that has a button. When I click the button, I want to use an asp.net api to return a list of videos which then get rendered as a list with a thumbnail to each video on a home page. Using react
Extension version: 0.22.4
VS Code version: Code 1.95.3 (f1a4fb101478ce6ec82fe9627c43efbf9e98c813, 2024-11-13T14:50:04.152Z)
OS version: Windows_NT x64 10.0.22631
Modes:

System Info

Item
Value

CPUs
AMD Ryzen 7 5800X3D 8-Core Processor            (16 x 3400)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: enabled_ondirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_graphite: disabled_offvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: enabledwebnn: disabled_off

Load (avg)
undefined

Memory (System)
31.92GB (16.99GB free)

Process Argv
./ --crash-reporter-id 34c85632-0a1c-4c74-b19a-d0f6e82462a3

Screen Reader
no

VM
0%

A/B Experiments
vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805cf:30301675
binariesv615:30325510
vsaa593:30376534
py29gd2263:31024239
vscaat:30438848
c4g48928:30535728
azure-dev_surveyone:30548225
2i9eh265:30646982
962ge761:30959799
f8hc8238:30694864
pythonnoceb:30805159
pythonmypyd1:30879173
2e7ec940:31000449
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
dvdeprecation:31068756
dwnewjupytercf:31046870
nativerepl2:31139839
pythonrstrctxt:31112756
nativeloc2:31192216
cf971741:31144450
iacca1:31171482
notype1cf:31157160
5fd0e150:31155592
dwcopilot:31170013
stablechunks:31184530

Can you please describe what type of request you're making so we can reproduce the issue? Example files / queries will help a lot. You can also try clearing your chat history.

It was a request for some code to do multipart file uploads. I understand
it was likely flagged for copy protection reasons.
All good, I changed the prompt slightly and managed to get what I needed.
Thanks!
Greg
On Thu, Dec 19, 2024 at 6:27 AM Bhavya U @.***> wrote:

Can you please describe what type of request you're making so we can
reproduce the issue? Example files / queries will help a lot. You can also
try clearing your chat history.
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode-copilot-release/issues/3228#issuecomment-2552107434,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BJKUYPC2YDLQCUQ4NH6RPCT2GHEDJAVCNFSM6AAAAABTR4KHFGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJSGEYDONBTGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>


Optionally disable creation of readme.txt on the shared folder

Summary
This PR allows the users to avoid the file 'readme.txt' being created in the shared folder.
Details
Actually, after booting the cointainer and if it has a folder shared from the host, a small readme.txt file is created.  This PR makes this behaviour completely configurable by means of a new variable (README) that can be defined in compose.yml:
environment:
  README: "N"

The new variable README defaults to "Y" so that the default behaviour does not change  (but, from my point of view, it should be set as "N" -- that is, disable the automatic creation of an new, possibly unwanted file).
Testing
1.- Add both the variable README="N" and a shared folder to compose.yml
2.- Start the container
3.- Check, either from the host or from within the container, the shared folder.  The file readme.txt should not have been created.

Definitively, that is a much better idea :-)
I'm currently testing it, will push ASAP

(Not really sure if I have followed the best practices :-/ ... I'm not very used to collaborate and make PR's using git)


AE, PS, TVpaint, Blender: Add option to define path to workfile template

This is a suplemental PR to #1571, which deals with more advanced version currently only for nuke.
This simply adds option to define a template file and have it used as a first working file on a project and host basis.
logic is handled in a global pre-hook and might clash with the more complex behaviour of Nuke, but should get us started.

When setting up a template and trying to launch the application from Launcher, it shows "Permission denied" message, and application will not start.
>>> [  Current context does not have any workfile yet.  ]
>>> [  Creating workfile from template: "C:\Users\qam\template"  ]
WARNING:openpype.tools.launcher.window.LauncherWindow:Action launch failed.
Traceback (most recent call last):
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\tools\launcher\window.py", line 441, in run_action
    action().process(filtered_session)
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\tools\launcher\actions.py", line 105, in process
    task_name=task_name
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\lib\applications.py", line 261, in launch
    return self.manager.launch(self.full_name, *args, **kwargs)
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\lib\applications.py", line 364, in launch
    return context.launch()
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\lib\applications.py", line 868, in launch
    prelaunch_hook.execute()
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\hooks\pre_copy_template_workfile.py", line 31, in execute
    from_template = self.workfile_path(last_workfile)
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\hooks\pre_copy_template_workfile.py", line 78, in workfile_path
    os.path.normpath(last_workfile)
  File "C:\Program Files\Python37\lib\shutil.py", line 266, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\Program Files\Python37\lib\shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\qam\\template'

My bad, I've set only a folder without a file. It would be nice if it checks the correct path with the file.

I am not able to run a workfile template in AE. It always starts in a "factory" template. Other apps are ok.

I am not able to run a workfile template in AE. It always starts in a "factory" template. Other apps are ok.

thanks I'll check it out

this was merged to #1571 and will be finalised in there


Suggestion to have "What's new" page for new releases

A "What's new" page, which exists for most projects, is incredibly helpful to keep up with the new exciting features of a package.
It would be very useful to have one for Holoviews, so that one can get to know new features of new releases.
Searching for "what's new" did not give me a similar page.

We already have notes for each release:
https://github.com/ioam/holoviews/releases
But yes, we probably should haev a more graphical page showing off new capabilities as they appear.  Right now, we are adding features way quicker than anyone can learn about them!

I really like how bokeh announces releases in a blog post style. We should do that too but we would need to set up a static page generator, link to it properly etc as well as write some nice content for each release!

We should do that too but we would need to set up a static page generator, link to it properly etc as well as write some nice content for each release!

I like bokeh's blog style page, but it seems simpler to just have a section like that on our website.

Sure, but then I would still want a page per release.

I don't think we need any special blogging support to do this, but one thing to consider is that the rest of our web site and notebooks are all things that we promise to keep up to date and are dynamically generated each time we have a release.  A blog post isn't like that; it's explicitly dated, and there's no promise that what's on there will be modified later if APIs or dependencies change.
So would we store a static copy of such pages, or would we commit to fixing them forever into the future until we delete them?

As a user I do the following:

Study documentation first
Keep up with changes through "What's new" (instead of scanning the docs for changes)

So, I'm aware that "What's new" may be overridden by a newer entry. I wouldn't know how else to get to know about new features.
Btw, ideally a "What's need" can be added to a news reader (i.e. support RSS)

We now have a releases page: http://holoviews.org/releases.html
That can certainly still be improved (and in fact the headings are broken in the current deployment).


Add DYNAMIC_LIST option for user-defined list of dynamic targets

Hackish proof-of-concept for #1595. Just for discussion, not for immediate merging (seems the local fork I developed this with is out-of-sync as well)

Thanks for taking a whack at this @martin-frbg.


Clifford optimize has no doc string and does not explain what it does

@dabacon dupe of #2961

Closing since this is a dupe.


Hidden input value="undefined" with JSON response

Everything uploads correctly in the background and I get a response from my server in JSON,
{"success":true,"filename":"image1.png","assetId":946}
But it doesn't populate the hidden field with the assetId.
Also is there anyway to change the hidden input name after successful upload as well? As Craft the CMS needs the field which it should be saved/linked to? Annoyingly I need one name for the Craft controller to do the upload to the system and another to save it to the correct field in the entry.
  FilePond.parse(document.body);

    FilePond.setOptions({
        allowDrop: true,
        allowReplace: true,
        instantUpload: true,
        server: {
            url: 'https://domain.com/',
            process: {
                url: './actions/assets/upload',
                ondata: (formData) => {
                    formData.append('folderId', '8');
                    return formData;
                },
                onload: (response) => response.key,
            }
        }
    });

  </script>

As requested in the issue template, please ask questions on Stack Overflow.


Adding Logon Process Trusted

Adding Logon Process Trusted
The event id 4611 for example can be helpful in detecting rogue logon process like Rubeus' User32LogonProcesss with triple 'S'
relationship_id: REL-2022-0187
name: Logon Process Trusted
contributors:
- Hamza OUADIA @Cyb3rSn0rlax
attack:
  data_source: null
  data_component: null
behavior:
  source: logon process
  relationship: trusted
  target: null
security_events:
- event_id: 4611
  name: A trusted logon process has been registered with the Local Security Authority.
  platform: Windows
  audit_category: System
  audit_sub_category: Security System Extension
  log_channel: Security
  log_provider: Microsoft-Windows-Security-Auditing
references:
- https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/event-4611
- https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=4611
notes:
- This event indicates that a logon process has registered with the Local Security Authority (LSA).
- At the technical level, the event does not come from the registration of a trusted logon process, but from a confirmation that the process is a trusted logon process. If it is a trusted logon process, the event generates.

Adding script to generate a UUID for future contributions
The script generate_uuid.py will generate a new UUID based on the already existing ones
import glob
import os
import re

current_directory = os.path.dirname(__file__)
relationships_directory = os.path.join(current_directory, '../relationships')
max_id=0
num_id = []
relationships_files = glob.glob(os.path.join(relationships_directory, "[!_]*.yml"))
for relationship_file in relationships_files:
    file = open(relationship_file,'r+')
    first_line = file.readlines()[0].rstrip() # read first line
    if re.search("^relationship_id\:\sREL\-202[\d]{1}\-\d{4}", first_line): # If file already has an ID
        search = re.search("^relationship_id\:\sREL\-202[\d]{1}\-(.*?)$", first_line) # Grab it
        num_id.append(search.group(1))
# Convert strings to integers
for i in range(0, len(num_id)):
    num_id[i] = int(num_id[i])
# Get max ID in list
for n in num_id:
    if n > max_id: max_id = n
# Generate relationship_id
count = max_id+1
print('relationship_id: REL-2022-' + '0'*(4 - len(str(count))) + str(count))

[ ] Add support for generic year value, currently supports 2022

Hey @H1L021 Thank you for the PR.

What are your thoughts on adding event 4611 to the data source / component: Logon Session / Logon Session Metadata? I understand this event is not related to the creation of a logon session, but it describes the registration of a trusted logon process that will handle the logon process. Also, since the the event provides context of the user that registered the trusted logon process, what are your thoughts on adding another yaml file with the relation: user --> registered --> logon process? This new relationship would be also mapped to the Logon Session / Logon Session Metadata

Regarding the python script. It is really cool :D I was thinking on doing something similar LOL. I just tested it and it works. Just a quick question: Would the script work when creating relationships in a year different than 2022?

I am merging the pull request and open issues for the comments/questions above since the current content is okay and we can update it later.
Best,
Jose


Use standard library for weighted random sampling

The original script is very slow and takes over 15 seconds to process the lottery.
I switched the homegrown method to a standard Python weighted random.choices method.
Note that the results it gives are not the same as the implementation differs, but they are still deterministic.
This method works at least 100 times faster.

@banteg thanks for this PR!
we are aware of random.choices and we do believe it's battle-tested. but on the other hand we also think hand-writing inverse transform sampling is a fun coding practice :) more importantly, the idea of using a random number generator from a uniform distribution to sample arbitrary probabilistic distributions is so cool, that we think it's worthwhile writing it done (bonus question, instead of a discrete distribution, how to use the same technique to sample a continuous distribution?).
as for performance, we're fine for the extra 15 seconds which may only be executed once in our whole life (within 9 hours!).
still, appreciate your suggestions. python does have a super strong toolset for all kinds of applications. hope we will have a chance to collaborate more for QuarkChain project!


H5PY dependencies

I got: ModuleNotFoundError: No module named 'h5py'
at line: from zernike import RZern
So H5PY dependency was not installed along wiht zernike when using pip install. I see it in the setup.py setup_requires, so not sure exactly why this happens. In Ubuntu, I started from a fresh conda environment with:
$ conda create -n <name> python=3.9
$ pip3 install torch torchvision torchaudio
$ pip install zernike

I changed setup_requires to install_requires and that seems to work for me. Is that working for you too? Using the latest version (0.0.33)?

Seems to be working!


Symlink config

This PR aims to clean up the root directory of the repo and make it more appealing to users.
Keep your JavaScript repository clean
See how the repo looks after changes https://github.com/strdr4605/lexical/tree/symlink-config

Before
After

By default the ./support/root/ directory is created.
Maybe a good idea is to move ./scripts/ to ./support/scripts/.

I wonder if this break for Windows users?

I wonder if this break for Windows users?

Asked a friend to clone, npm install, and npm run test-unit.
Looks like on Windows you need to run the shell as admin to allow the creation of symlinks.
After this npm install and npm run test-unit runs as expected.
This might be a small blocker for lexical contributors on windows. But I don't see a big issue here.

Can you re-commit please? I wonder if the CI passes with these changes.

Also, you can add a README.md in the sourceDir (./support/root/README.md) and it will be excluded from symlink. Where you can explain if needed, why config files stay in ./support/root/.

Please can you trigger another update on this so the CI can actually run (maybe with a rebase)? I think this PR was put up before we fixed the detection mechanism somehow. Sorry!

@zurfyx, @trueadm, can you approve the workflow to run to see if the issue disappeared after Symlink-config after npm ci?!

Hey @strdr4605 - thanks for submitting this. We discussed this as a team and we'd prefer to keep the current configuration for now. We may revisit this again in the future.


Remove trailing semicolons

Just some clean up, Had some trailing semicolons and mixed indents/tabs.

Thanks for the cleanup!


When expression "borrow of moved value"

I tried this code:
let can_move = var(true);

Container! {
        when *#{can_move.clone()} {
            mouse::cursor = mouse::CursorIcon::Move;
        }
        mouse::on_mouse_down = hn!( |_| {
            let _use = can_move;
        });

}

I saw this happen:
error[E0382]: borrow of moved value: can_move
I expected this to happen:
It should compile because the when expression is declared before the handler, so#{can_move.clone()} should eval before the hn! capture move.
Error log or stack trace:

stderr
Running: do "run" ["window"]
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on build directory
   Compiling examples v0.0.0 (C:\code\zng\examples)
error[E0382]: borrow of moved value: `can_move`
   --> examples\window.rs:435:17
    |
427 |       let can_move = vars.state().map(|s| matches!(s, WindowState::Normal | WindowState:...
    |           -------- move occurs because `can_move` has type `ReadOnlyVar<bool, ArcVar<bool>>`, which does not implement the `Copy` trait
...
435 |           when *#{can_move.clone()} {
    |                   ^^^^^^^^ value borrowed here after move
...
438 |           mouse::on_mouse_down = hn!(|args: &mouse::MouseInputArgs| {
    |  ________________________________-
439 | |             if args.is_primary() && can_move.get() {
    | |                                     -------- variable moved due to use in closure
440 | |                 window::cmd::DRAG_MOVE_RESIZE_CMD.scoped(WINDOW.id()).notify();
441 | |             }
442 | |         });
    | |__________- value moved into closure here
    |
help: consider cloning the value if the performance cost is acceptable
   --> C:\code\zng\crates\zng-clone-move\src\lib.rs:140:17
    |
140 |             move.clone() | $($rest)+
    |                 ++++++++

For more information about this error, try `rustc --explain E0382`.
error: could not compile `examples` (example "window") due to 1 previous error
error: task "run" failed with exit code: 101

When expressions are expanded after all assigns to support later assigns back when the WidgetBuilder did not exist:
Container! {
    when #widget::background_color.0.red == 0.0 {
        mouse::cursor = mouse::CursorIcon::Move;
    }
    widget::background_color = colors::BLUE;
}

Current WidgetBuilder can handle delayed assigns, even from a later dynamic source like a Style. Time to go mess with the proc-macros again...


Update nodes.go with SafetyNet node

We are online for several months with three validators. Adding to deso nodes list

@lazynina @diamondhands0 if you honor https://github.com/deso-protocol/core/pull/1438 this one can be skipped as @StarGeezerPhil included SafetyNet in his improved proposal. Thank you

@lazynina @diamondhands0 if you honor #1438 this one can be skipped as @StarGeezerPhil included SafetyNet in his improved proposal. Thank you

Thanks @carry2web, I've also made comment on-chain about the prior proposals that were skipped:

PR #1207 - @sungkhum for MyDesoSpace, and
PR #424 - @mohitbadwal for Setu_Deso (now defunct/inactive)
CooperativaLocal node using ID 30 (also now defunct/inactive)

If the team are happy to accept #1438 I'm happy to update it in chronological order so that Nathan's much earlier request can be honoured for node ID 30, which he has been using for some considerable time already.
This would push Focus to 31, SafetyNet to 32 and so on.
On-chain post requesting this is considered:
https://diamondapp.com/posts/5f8d804ca58a6263d93d58fb4cf5ab3cc58d84bd9f2e091f04638832af1074cd


can u add me on discord i have some questions about your dll blocker write up but ur not accepting friend reqs

i dont know how else to contact you your site is out of date but i have some questions regarding the dll blocker ive tried everything possible i can think of and cant overwrite the dll entry point with the handle i grab off the debugger event listener it just does not work and executes and loads like normal

oh yea my discord is: 44xo

https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-updateprocthreadattribute

found a much better way to do this

ok nevermind more research done and an entire blogpost written later after testing the new method and i would still like to go back to this approach but in a modified and improved way to leave less of a trail
https://44.is-a.dev/posts/Building-A-DLL-Blocker/


Please add N64 bios pack

boot.rom  with MD5 hash = 5C124E7948ADA85DA603A522782940D0
boot1.rom with MD5 hash = D4232DC935CAD0650AC2664D52281F3A

Oh wait, you already had them.


Dedupe fragment suggestions in client-preset

When using the client-preset we are seeing a lot of duplicate fragment-suggestions currently 😅

Apparently this is the case because I was working in a mono-repo that had 3 instances of GraphQLSP with the client-preset targeting the same API 😅 Not sure if this is actionable atm


[Draft] Sample step-by-step descriptions

I've opening this PR just to discuss it with @jeffhandley who suggested that we could add similar sections.

Internal previews

📄 File
🔗 Preview link

docs/standard/serialization/binaryformatter-migration-guide/migration-to-data-contract-serializer.md
Migration to DataContractSerializer

docs/standard/serialization/binaryformatter-migration-guide/migration-to-protobuf-net.md
"BinaryFormatter Migration Guide: Migration to protobuf-net"

I think this is a great direction to head and worth continuing down for the other pages.


On localhost landing page, links should go to correct port

Description
The ES example's localhost landing page links to subpages with the wrong port when the port set in the docker-compose.yml is different from the port in the pygeoapi docker.config.yml.
Steps to Reproduce
Steps to reproduce the behavior:

Set the port in https://github.com/geopython/pygeoapi-examples/blob/main/docker/elastic/docker-compose.yml#L44 to something other than 5000 (I was using 5001).
Run docker-compose up from the elastic directory.
Click on any link when opening localhost:5001 in the browser.

Expected behavior
I think the port should change across all files when set in one top level variable file.
Screenshots/Tracebacks
If applicable, add screenshots to help explain your problem.
Environment

OS: Mac
Python version: Python 3.12.1
pygeoapi version: geopython/pygeoapi:latest
pygeoapi-examples version: latest

Additional context
Add any other context about the problem here.

Should be fixed by https://github.com/geopython/pygeoapi-examples/pull/10/files.


Added support for GQ GMC-SE

Hi @Wikilicious, thank you for building this nice project and framework.
I updated it with support for the GMC-SE.  It uses the RFC1201 so was fairly straightforward.
I've tested it with my GMC-SE and everything seems to work ok.
The only exception being that the Gyro gives a tuple of 3 numbers something like (15375, -5885, 769) however I think it is nonsensical and the numbers barely change when the device is moved.
I've updated the documentations as well.  Will create more PR in the future if I see any significant changes needed.
Huy

Hi @huyndao
This is awesome! Thanks!
I really need to add a contribution guide.

What do we do about the ruff lint failure?  I don't see what is wrong with it.

Ok, now all linting and tests are passing...
It's just complaining that we added code but didn't add unit-tests for it.
The tests section isn't organized in the best way...
We can do 2 paths:

We merge it as-is and I can add test cases later.
You can take a look at tests/test_devices/test_gmc320.py and make one for GMCSE (you can simply reuse the gmc320 data since it's also RFC1201)

Ok, now all linting and tests are passing... It's just complaining that we added code but didn't add unit-tests for it. The tests section isn't organized in the best way...
We can do 2 paths:
1. We merge it as-is and I can add test cases later.

2. You can take a look at tests/test_devices/test_gmc320.py and make one for GMCSE (you can simply reuse the gmc320 data since it's also RFC1201)

Thanks, I will do option 2 within an hour.

You can do pip install -r dev_requirements.txt and it'll add invoke to your command line. You can then add a unit test and run invoke test and it'll show you a summary.

You can do pip install -r dev_requirements.txt and it'll add invoke to your command line. You can then add a unit test and run invoke test and it'll show you a summary.

please bear with me, I'd need to get my device hooked up in order to have the correct data for the response mapping correct?
If so, it may take a bit until I finish work.

It's all good you're the first contributor and this is my first pypi project.
You could get the data from your device... BUT I don't think that's needed here since GMCSE inherits DeviceRFC1201 and doesn't override methods. Basically, feeding GMC320 data into GMCSE class is a valid test.
Give me a few minutes to take a look... it may just be a one liner

It's all good you're the first contributor and this is my first pypi project. You could get the data from your device... BUT I don't think that's needed here since GMCSE inherits DeviceRFC1201 and doesn't override methods. Basically, feeding GMC320 data into GMCSE class is a valid test. Give me a few minutes to take a look... it may just be a one liner

okay, I pretty much reused GMC320 data except for the lines about the version, which I used GMCSE.
The tests passed locally, hopefully everything works.
Please check commit 7793fa5.
Thanks!

It's all good you're the first contributor and this is my first pypi project. You could get the data from your device... BUT I don't think that's needed here since GMCSE inherits DeviceRFC1201 and doesn't override methods. Basically, feeding GMC320 data into GMCSE class is a valid test. Give me a few minutes to take a look... it may just be a one liner

okay, I pretty much reused GMC320 data except for the lines about the version, which I used GMCSE.
The tests passed locally, hopefully everything works.
Please check commit 7793fa5.
Thanks!

Thank you for your time!
If you want...
In README.md under Devices section, can you add a checkmark for GMC-SE (✔=physically confirmed works)
Add a Contributors section after Devices section with your github link.
And... if all this wasn't enough... lol... in tests/test_connection_mock_serial.py line 135... add pygmc.GMC (if in linux or mac pip install mock_serial else... can only test in GitHub)
If you're interested in the test coverage rabbit-hole... https://app.codecov.io/gh/Wikilicious/pygmc
It's a measure of lines of logic that a test touched.
More if you're curious... can see the code for github CI tests here: https://github.com/Wikilicious/pygmc/tree/master/.github/workflows

It's all good you're the first contributor and this is my first pypi project. You could get the data from your device... BUT I don't think that's needed here since GMCSE inherits DeviceRFC1201 and doesn't override methods. Basically, feeding GMC320 data into GMCSE class is a valid test. Give me a few minutes to take a look... it may just be a one liner

okay, I pretty much reused GMC320 data except for the lines about the version, which I used GMCSE.
The tests passed locally, hopefully everything works.
Please check commit 7793fa5.
Thanks!

Thank you for your time!
If you want... In README.md under Devices section, can you add a checkmark for GMC-SE (✔=physically confirmed works) Add a Contributors section after Devices section with your github link. And... if all this wasn't enough... lol... in tests/test_connection_mock_serial.py line 135... add pygmc.GMC (if in linux or mac pip install mock_serial else... can only test in GitHub)
If you're interested in the test coverage rabbit-hole... https://app.codecov.io/gh/Wikilicious/pygmc It's a measure of lines of logic that a test touched.
More if you're curious... can see the code for github CI tests here: https://github.com/Wikilicious/pygmc/tree/master/.github/workflows

I think I fixed the mock connection by mimicking what you had for GMC-320 (at least it worked here locally).
I added the checkmark for GMC-SE as you suggested
I also added the contributors section and the github links, I put you on top since you're the owner / main dev.
Coincidentally, I worked at Intel in my previous life lol.

let's try again shall we: mock_serial passed locally.

Well... everything is now perfect!
It's just a matter of a simple merge...
Or do we release a new version?

Well... everything is now perfect! It's just a matter of a simple merge... Or do we release a new version?

You're the boss.  Whichever way gets it to pypi the quickest so I can use it with pip install lol.

Well... everything is now perfect! It's just a matter of a simple merge... Or do we release a new version?

You're the boss. Whichever way gets it to pypi the quickest so I can use it with pip install lol.

I'm a perpetual student...
Let's release
There's just one-more-thing™
The version.
pygmc follows semver
major.minor.patch - Look up the semver spec and see if you agree...
I don't see this as a patch/fix for the 0.9.1 release... it's a new/added feature...
So 0.10.0
There is a tool to do this... bumpversion
bumpversion minor

Well... everything is now perfect! It's just a matter of a simple merge... Or do we release a new version?

You're the boss. Whichever way gets it to pypi the quickest so I can use it with pip install lol.

I'm a perpetual student...
Let's release There's just one-more-thing™
The version. pygmc follows semver major.minor.patch - Look up the semver spec and see if you agree... I don't see this as a patch/fix for the 0.9.1 release... it's a new/added feature... So 0.10.0
There is a tool to do this... bumpversion bumpversion minor

Thanks, I'm learning as well...  Never worked with tests / CI before.

I did, it's updated to 0.10.0, unless I missed something.

I'm seeing v0.10.0 on pypi... https://pypi.org/project/pygmc/

Hi @Wikilicious, thank you for building this nice project and framework.
I updated it with support for the GMC-SE. It uses the RFC1201 so was fairly straightforward. I've tested it with my GMC-SE and everything seems to work ok. The only exception being that the Gyro gives a tuple of 3 numbers something like (15375, -5885, 769) however I think it is nonsensical and the numbers barely change when the device is moved.
I've updated the documentations as well. Will create more PR in the future if I see any significant changes needed.
Huy

For the gyro... The docs don't specify any units so treat it as relative...
Put the GC upside-down and get_gyro

Hi @Wikilicious, thank you for building this nice project and framework.
I updated it with support for the GMC-SE. It uses the RFC1201 so was fairly straightforward. I've tested it with my GMC-SE and everything seems to work ok. The only exception being that the Gyro gives a tuple of 3 numbers something like (15375, -5885, 769) however I think it is nonsensical and the numbers barely change when the device is moved.
I've updated the documentations as well. Will create more PR in the future if I see any significant changes needed.
Huy

For the gyro... The docs don't specify any units so treat it as relative... Put the GC upside-down and get_gyro

The change was quite small.  It would be nice if they document what the units are.

Screen up: (15370, -49, 768)
Screen down: (15364, -49, 768)
Most other orientations give the same last 2 numbers

When I have more time, I'll play with it more but it looks like the numbers may change after some event like a reset or history erasure.

Firstly... Thanks for your contribution!
Second...
On my GMC-500+
up-right: (5, -246, 5)
face-down: (8, -10, 230)
face-up: (11, -15, -230)
your screen up/down numbers don't seem to indicate a useful orientation...
Specs I used: https://www.gqelectronicsllc.com/download/GQ-RFC1201.txt
It may be it's just not supported on GMC-SE... it may be bad pygmc coding... it may be bad usb/device...

No problems, thank you for creating a convenient package/framework for it.  It will be very helpful for some future projects I have in mind later.
Regarding the GMC-SE: according to this page  there is gyro support.  Hopefully I'll have some time this weekend to just connect to it with raw serial and see what is going on.


Inappropriate parameter types of LoadablePlugin's invoke function

Description
The parameter types of the invoke function is inappropriate.
https://github.com/apache/incubator-teaclave-trustzone-sdk/blob/bc14fb63f65bfff77456eea1b9097c1619d30b04/optee-utee/src/extension.rs#L33-L56
First, the parameter data should be declared as mut instead, as when invoking plugin the content of the buffer will be changed:
Secondly, I think the function should just like the read function of std::io::Read, return the length rather than return a new constructed vector, returning a length notifies developers that the output is relative to the limited buffer, while returning a vector is not.
Last, I think the function should add an parameter_size parameter as sometimes the return content is bigger than the request content, for example, fetch file content from plugin.
in conclusion, I think the function should be:
// previous: pub fn invoke(&mut self, command_id: u32, subcommand_id: u32, data: &[u8]) -> Result<Vec<u8>> {
// now: set data as mut and rename it to share_buffer, add parameter_size and return length instead of new vector
pub fn invoke(&mut self, command_id: u32, subcommand_id: u32, share_buffer: &mut [u8], parameter_size: usize) -> Result<usize> {
    // some codes
}

PS: we should use data.as_mut_ptr() (returns a *mut T) instead of data.as_ptr() (return a *const T) too.
https://github.com/apache/incubator-teaclave-trustzone-sdk/blob/bc14fb63f65bfff77456eea1b9097c1619d30b04/optee-utee/src/extension.rs#L41
https://github.com/apache/incubator-teaclave-trustzone-sdk/blob/bc14fb63f65bfff77456eea1b9097c1619d30b04/optee-utee/optee-utee-sys/src/tee_internal_api_extensions.rs#L21-L39
Suggestion
1) change the parameters and return value of the function, but it is a breaking change
impl LoadablePlugin {
    pub fn new(uuid: &Uuid) -> Self {
        Self { uuid: uuid.to_owned() }
    }
    pub fn invoke(&mut self, command_id: u32, subcommand_id: u32, share_buffer: &mut [u8], parameter_size: usize) -> Result<usize> {
        let raw_uuid: Uuid = self.uuid;
        let mut outlen: usize = 0;
        match unsafe {
            raw::tee_invoke_supp_plugin(
                raw_uuid.as_raw_ptr(),
                command_id as u32,
                subcommand_id as u32,
                share_buffer.as_mut_ptr() as _,
                parameter_size,
                &mut outlen as *mut usize,
            )
        } {
            raw::TEE_SUCCESS => {
                assert!(outlen <= (share_buffer.len()));

                Ok(outlen)
            },
            code => Err(Error::from_raw_error(code)),
        }

    }
}

and there is another way, by using generic:
struct ShareBuffer<'a> {
    buffer: &'a mut [u8],
    parameter_size: usize,
}

impl<'a> From<&'a mut [u8]> for ShareBuffer<'a> {
    fn from(value: &'a mut [u8]) -> Self {
        let len = value.len();
        Self::new(value, len)
    }
}

impl<'a> ShareBuffer<'a> {
    pub fn new(buffer: &'a mut [u8], parameter_size: usize) -> Self {
        Self {
            buffer,
            parameter_size,
        }
    }
}

impl LoadablePlugin {
    /// call it like:
    /// let mut buffer = [0_u8; 32];
    /// let parameter_size = 12;
    /// let _ = plugin.invoke(1, 1, buffer.as_mut_slice());  // when parameter size is bigger than or equals to response size
    /// let _ = plugin.invoke(1, 1, ShareBuffer::new(&mut buffer, parameter_size));  // when parameter is smaller than response size
    pub fn invoke<'a, T: Into<ShareBuffer<'a>>>(&mut self, command_id: u32, subcommand_id: u32, share_buffer: T) -> Result<usize> {
        let raw_uuid: Uuid = self.uuid;
        let buffer: ShareBuffer<'a> = share_buffer.into();
        let mut outlen: usize = 0;
        match unsafe {
            raw::tee_invoke_supp_plugin(
                raw_uuid.as_raw_ptr(),
                command_id as u32,
                subcommand_id as u32,
                buffer.buffer.as_mut_ptr() as _,
                buffer.parameter_size,
                &mut outlen as *mut usize,
            )
        } {
            raw::TEE_SUCCESS => {
                assert!(outlen <= (buffer.buffer.len()));

                Ok(outlen)
            },
            code => Err(Error::from_raw_error(code)),
        }
    }

2) mark the invoke method as deprecated, and introduce a new one
impl LoadablePlugin {
    pub fn new(uuid: &Uuid) -> Self {
        Self { uuid: uuid.to_owned() }
    }
    #[deprecated(note = "Please use the invoke2 function instead")]
    pub fn invoke(&mut self, command_id: u32, subcommand_id: u32, data: &[u8]) -> Result<Vec<u8>> {
        // old codes
    }
    pub fn invoke2(&mut self, command_id: u32, subcommand_id: u32, share_buffer: &mut [u8], parameter_size: usize) -> Result<usize> {
        let raw_uuid: Uuid = self.uuid;
        let mut outlen: usize = 0;
        match unsafe {
            raw::tee_invoke_supp_plugin(
                raw_uuid.as_raw_ptr(),
                command_id as u32,
                subcommand_id as u32,
                share_buffer.as_mut_ptr() as _,
                parameter_size,
                &mut outlen as *mut usize,
            )
        } {
            raw::TEE_SUCCESS => {
                assert!(outlen <= (share_buffer.len()));

                Ok(outlen)
            },
            code => Err(Error::from_raw_error(code)),
        }

    }
}

I would prefer the first one, as the codes are wrong here, we should stop people from compiling so that they will get noticed.

sometimes the return content is bigger than the request content, for example, fetch file content from plugin.

Thanks for pointing out this issue and providing valuable advice. Seems the issue exists due to treat input_buffer as shared_buffer.
For the Rust API pub fn invoke(&mut self, command_id: u32, subcommand_id: u32, data: &[u8]) -> Result<Vec<u8>> the intention is to expose the input_buffer and output_buffer for developers, with the shared_buffer is transparent. It's a user-friendly way while developers don't need to truncate the output buffers.
However the current implementation treat input_buffer as shared_buffer, which cause problems when the output_buffer is larger than the input_buffer.
https://github.com/apache/incubator-teaclave-trustzone-sdk/blob/bc14fb63f65bfff77456eea1b9097c1619d30b04/optee-utee/src/extension.rs#L33-L56
To address this issue while maintaining compatibility and simplicity, how about modifying it as follows:
const SHARED_BUFFER_SIZE: usize = 1024; // or other size

 pub fn invoke(&mut self, command_id: u32, subcommand_id: u32, data: &[u8]) -> Result<Vec<u8>> {
     let raw_uuid: Uuid = self.uuid;
     let mut outlen: usize = 0;
     let mut shared_buffer = vec![SHARED_BUFFER_SIZE; u8];
     shared_buffer[..data.len()].copy_from_slice(data);

     match unsafe {
         raw::tee_invoke_supp_plugin(
             raw_uuid.as_raw_ptr(),
             command_id as u32,
             subcommand_id as u32,
             shared_buffer.as_mut_ptr() as _,
             data.len(),
             &mut outlen as *mut usize,
         )
     } {
         raw::TEE_SUCCESS => {
             assert!(outlen <= (shared_buffer.len()));
             let mut outbuf = vec![0; outlen];
             outbuf.copy_from_slice(&shared_buffer[..outlen]);

             Ok(outbuf)
         },
         code => Err(Error::from_raw_error(code)),
     }

 }

This approach ensures shared_buffer can handle outputs correctly while keeping the interface user-friendly and compatible. However, it comes with a disadvantage: the static SHARED_BUFFER_SIZE is not dynamic, which could lead to potential memory waste.

@DemesneGH
What's wrong
Sorry, my previous solution has some mistakes and is misleading, there are limitations that you can not get data from ree more than len bytes:
https://github.com/apache/incubator-teaclave-trustzone-sdk/blob/bc14fb63f65bfff77456eea1b9097c1619d30b04/optee-utee/optee-utee-sys/src/tee_internal_api_extensions.rs#L21-L39
Because OP-TEE will map your memory by the parameter len, see the codes:

So even you have buffer more than len bytes, and you did copy more than this in plugin, and set out_len greater than len, you will only get len bytes copy to you buffer.
Therefore, when your response size greater than you request size, you must not use the request_size as len parameter, it should be max(request_size, response_size).
Test
You can try it by the following test.
Prepare buffer of 32 bytes in both plugin and TA side, and TA use sub_cmd to tell plugin to copy how many bytes into the buffer, and set the out_len.
In Plugin:
static TEEC_Result plugin_invoke(unsigned int cmd, unsigned int sub_cmd,
                                        void *data, size_t data_len,
                                        size_t *out_len)
{
    const char bytes[] = "1111111111111111111111111111111111111111111111111111111111111111";
    memcpy(data, bytes, sub_cmd);
    *out_len = sub_cmd;
    return TEEC_SUCCESS;
}

In TA:
static TEE_Result plugin_copy(uint32_t input_size, uint32_t request_size)
{
    TEE_Result res = TEE_SUCCESS;
    char buffer[32] = {0};
    TEE_UUID uuid = PLUGIN_UUID;
    size_t out_len = 0;
    res = tee_invoke_supp_plugin(&uuid, 0, request_size, buffer, input_size, &out_len);
    if (res)
        EMSG("invoke plugin failed with code 0x%x", res);

    assert(out_len == request_size);
    printf("data: ");
    for (size_t i = 0; i < out_len; i++) {
        printf("%d,", buffer[i]);
    }
    printf("\n");
    return res;
}

TEE_Result TA_InvokeCommandEntryPoint(void __unused *sess_ctx,
                                      uint32_t __unused cmd_id, uint32_t __unused param_types,
                                      TEE_Param __unused params[4])
{
    // request size > response size, out_len is 20,
    // print 49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49
    // work as expected
    plugin_copy(32, 20);
    // request size = response size, out_len is  32,
    // print 49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49,49
    // work as expected
    plugin_copy(32, 32);
    // request size < response size, out_len is 32,
    // print 49,49,49,49,49,49,49,49,49,49,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
    // just copy len of request size (which is 10), not working
    plugin_copy(10, 32);
    return TEE_SUCCESS;
}

you can just download this  plugins.zip, and run the test.
My Scenario
I am trying deep learning in TEE, which I will need to copy large amount of data from REE(getting training data), request size just several bytes(for example, the filename and offset), but the response size is huge(for example, up to 100M), so setting a const ShareBuffer is not working for me, and making a copy is not acceptable too(consumes too many memory).
How do we know the request size
During my development, in 2 ways:

a zerocopy struct: like what we used to do in C, both TEE and REE know the size of it, but it is hard to upgrade.
use sub_cmd: as the request size will not get too big, an int32 is enough for telling REE how many bytes is the request.

What's More
I am trying to find a better solution, currently, I am trying to use generic to make the invoke method flexible(result type could be Vec or usize, just as what developers define, like serde_json), I will update you if I'm done will it.

After trying some different ways, I can now introduce a new way, which keeps compatibility and simplicity, and can prevent developers from misuse (while rest of my other ways don't).
Solution
First, we introduce an enum, call Buffer, it could has its own buffer or borrow from others.
pub enum Buffer<'a> {
    Own(Vec<u8>),
    Borrow(&'a mut [u8]),
}

impl<'a> Buffer<'a> {
    // from immutable slice, create a new buffer instead
    pub fn new_from_parameter(data: &[u8], buffer_size: usize) -> Self {
        assert!(buffer_size >= data.len());
        let mut buffer = vec![0_u8; buffer_size];
        buffer[..data.len()].copy_from_slice(&data);
        Self::Own(buffer)
    }
    // from mutable slice, use that directly
    pub fn new_from_share_buffer(data: &'a mut [u8]) -> Self {
        Self::Borrow(data)
    }
    // get buffer for the parameters of invoke function
    pub fn buffer(&mut self) -> &mut [u8] {
        match self {
            Self::Own(buf) => buf.as_mut_slice(),
            Self::Borrow(buf) => buf,
        }
    }
    // into the result of invoke function
    pub fn into_sized_vector(self, size: usize) -> Vec<u8> {
        match self {
            Self::Own(mut buf) => {
                buf.resize(size, 0);
                buf
            }
            Self::Borrow(buf) => {
                let mut buffer = vec![0_u8; size];
                buffer[..core::cmp::min(size, buf.len())].copy_from_slice(&buf);
                buffer
            }
        }
    }
}

Then, we introduce a wire Trait, we use this to make our invoke function flexible
pub trait Wire<'a, OutputT> {
    // convert the function input to buffer
    fn into_buffer(self) -> Buffer<'a>;
    // construct the result
    fn make_result(buffer: Buffer<'a>, outlen: usize) -> OutputT;
}

With both of them, the invoke method can rewrite to:
impl Plugin {
    pub fn invoke<'a, OutputT, InputT: Wire<'a, OutputT>>(
        &mut self,
        command_id: u32,
        subcommand_id: u32,
        data: InputT,
    ) -> Result<OutputT> {
        let raw_uuid: Uuid = self.uuid;
        let mut outlen: usize = 0;
        let mut data = data.into_buffer();  // transfer data into buffer
        let buffer = data.buffer();
        match unsafe {
            raw::tee_invoke_supp_plugin(
                raw_uuid.as_raw_ptr(),
                command_id as u32,
                subcommand_id as u32,
                buffer.as_mut_ptr(),
                buffer.len(),
                &mut outlen as *mut usize,
            )
        } {
            raw::TEE_SUCCESS => {
                assert!(outlen <= buffer.len());
                // construct result by the type of data
                Ok(InputT::make_result(data, outlen))
            }
            code => Err(Error::from_raw_error(code)),
        }
    }
}

Finally, we can imply Wire trait for the (InputT, OutputT) pair we want:
// use generic type pair (AsRef<[u8]>, Vec<u8>) to keeps compatibility
impl<'a, T: AsRef<[u8]>> Wire<'a, Vec<u8>> for T {
    fn into_buffer(self) -> Buffer<'a> {
        let data = self.as_ref().into();
        Buffer::new_from_parameter(data, data.len())
    }
    fn make_result(buffer: Buffer<'a>, outlen: usize) -> Vec<u8> {
        buffer.into_sized_vector(outlen)
    }
}

// use exact type pair to avoid confliction and keep developers from misuse.
impl<'a> Wire<'a, usize> for &'a mut [u8] {
    fn into_buffer(self) -> Buffer<'a> {
        Buffer::Borrow(self)
    }
    fn make_result(_: Buffer<'a>, outlen: usize) -> usize {
        outlen
    }
}

And now, we can call the invoke function as we want, here is some examples that may or may not compile:
        let mut plugin = Plugin {
            uuid: Uuid::parse_str("7dd54ee6-a705-4e4d-8b6b-aa5024dfcd10").unwrap(),
        };
        let immutable_buffer = [0_u8; 32];
        let mut mutable_buffer = [0_u8; 32];
        // =================== old feature, pass immutable slice ==================
        // response1 infer to Vec<u8> automatically, keeps compatibility.
        let response1 = plugin.invoke(1, 1, &immutable_buffer).unwrap();
        // this won't compile, as we didn't imply the Wire trait for the type pair
        // and I think codes like this must be mistake from developers, developers
        // must get the buffer back
        let response2: usize = plugin.invoke(1, 1, &immutable_buffer).unwrap();

        // =================== new feature, pass mutable slice ====================
        // developers must specific what they want, keeping developers from misuse
        // currently the valid return types are: Vec<u8> or usize
        let response3: Vec<u8> = plugin.invoke(1, 1, &mut mutable_buffer).unwrap();
        let response4: Vec<u8> = plugin.invoke(1, 1, mutable_buffer.as_mut_slice()).unwrap();
        // this won't compile, as we need developers to specific the input type to &mut [u8]
        let response5: usize = plugin.invoke(1, 1, &mut mutable_buffer).unwrap();
        // this is working like std::io::Read
        let response5: usize = plugin.invoke(1, 1, mutable_buffer.as_mut_slice()).unwrap();

Full Codes Demo
Here's the full codes demo.

cargo.toml

[package]
name = "plugin"
version = "0.1.0"
edition = "2024"

[dependencies]
optee-utee = { git = "https://github.com/apache/incubator-teaclave-trustzone-sdk.git", branch = "main", version = "0.2.0", features = ["no_panic_handler"] }
optee-utee-sys = { git = "https://github.com/apache/incubator-teaclave-trustzone-sdk.git", branch = "main", version = "0.2.0" }

[dev-dependencies]
rand = "0.8.5"
once_cell = "1.20.2"

lib.rs

extern crate alloc;
use alloc::vec;
use alloc::vec::Vec;
use optee_utee::{Error, Result, Uuid};
use optee_utee_sys as raw;

pub enum Buffer<'a> {
    Own(Vec<u8>),
    Borrow(&'a mut [u8]),
}

pub trait Wire<'a, OutputT> {
    // convert the function input to buffer
    fn into_buffer(self) -> Buffer<'a>;
    // construct the result
    fn make_result(buffer: Buffer<'a>, outlen: usize) -> OutputT;
}

pub struct Plugin {
    uuid: Uuid,
}

impl Plugin {
    pub fn invoke<'a, OutputT, InputT: Wire<'a, OutputT>>(
        &mut self,
        command_id: u32,
        subcommand_id: u32,
        data: InputT,
    ) -> Result<OutputT> {
        let raw_uuid: Uuid = self.uuid;
        let mut outlen: usize = 0;
        let mut data = data.into_buffer();
        let buffer = data.buffer();
        match unsafe {
            raw::tee_invoke_supp_plugin(
                raw_uuid.as_raw_ptr(),
                command_id as u32,
                subcommand_id as u32,
                buffer.as_mut_ptr(),
                buffer.len(),
                &mut outlen as *mut usize,
            )
        } {
            raw::TEE_SUCCESS => {
                assert!(outlen <= buffer.len());
                // construct result by the type of data
                Ok(InputT::make_result(data, outlen))
            }
            code => Err(Error::from_raw_error(code)),
        }
    }
}

impl<'a> Buffer<'a> {
    // from immutable slice, create a new buffer instead
    pub fn new_from_parameter(data: &[u8], buffer_size: usize) -> Self {
        assert!(buffer_size >= data.len());
        let mut buffer = vec![0_u8; buffer_size];
        buffer[..data.len()].copy_from_slice(&data);
        Self::Own(buffer)
    }
    // from mutable slice, use that directly
    pub fn new_from_buffer(data: &'a mut [u8]) -> Self {
        Self::Borrow(data)
    }
    // get buffer for the parameters of invoke function
    pub fn buffer(&mut self) -> &mut [u8] {
        match self {
            Self::Own(buf) => buf.as_mut_slice(),
            Self::Borrow(buf) => buf,
        }
    }
    // into the result of invoke function
    pub fn into_sized_vector(self, size: usize) -> Vec<u8> {
        match self {
            Self::Own(mut buf) => {
                buf.resize(size, 0);
                buf
            }
            Self::Borrow(buf) => {
                let mut buffer = vec![0_u8; size];
                buffer[..core::cmp::min(size, buf.len())].copy_from_slice(&buf);
                buffer
            }
        }
    }
}

// use generic type pair (AsRef<[u8]>, Vec<u8>) to keeps compatibility
impl<'a, T: AsRef<[u8]>> Wire<'a, Vec<u8>> for T {
    fn into_buffer(self) -> Buffer<'a> {
        let data = self.as_ref().into();
        Buffer::new_from_parameter(data, data.len())
    }
    fn make_result(buffer: Buffer<'a>, outlen: usize) -> Vec<u8> {
        buffer.into_sized_vector(outlen)
    }
}

// use exact type pair to avoid confliction and keep us from misuse.
impl<'a> Wire<'a, usize> for &'a mut [u8] {
    fn into_buffer(self) -> Buffer<'a> {
        Buffer::Borrow(self)
    }
    fn make_result(_: Buffer<'a>, outlen: usize) -> usize {
        outlen
    }
}

#[cfg(test)]
pub mod tests {
    extern crate std;
    use super::*;
    use core::ffi::c_char;
    use once_cell::sync::Lazy;
    use optee_utee_sys::{TEE_Result, TEE_UUID};
    use rand::Rng;
    use rand::distributions::Alphanumeric;
    use std::collections::HashMap;
    use std::string::String;
    use std::sync::RwLock;

    static REE_RETURN_VALUES: RwLock<Lazy<HashMap<u32, String>>> =
        RwLock::new(Lazy::new(|| HashMap::new()));

    fn set_value_for_test(sub_cmd: u32, value: String) {
        let mut values = REE_RETURN_VALUES.write().unwrap();
        assert!(!values.contains_key(&sub_cmd));
        values.insert(sub_cmd, value);
    }

    fn get_test_value(sub_cmd: u32) -> String {
        let values = REE_RETURN_VALUES.read().unwrap();
        values.get(&sub_cmd).unwrap().to_owned()
    }

    fn generate_test_pair(length: usize) -> (u32, String) {
        let sub_cmd: u32 = rand::thread_rng().r#gen();
        let random_string: String = rand::thread_rng()
            .sample_iter(&Alphanumeric)
            .take(length)
            .map(char::from)
            .collect();
        set_value_for_test(sub_cmd, random_string.clone());
        (sub_cmd, random_string)
    }

    #[unsafe(no_mangle)]
    pub extern "C" fn tee_invoke_supp_plugin(
        _uuid: *const TEE_UUID,
        _cmd: u32,
        sub_cmd: u32,
        buf: *mut c_char,
        len: usize,
        outlen: *mut usize,
    ) -> TEE_Result {
        let inbuf = unsafe { core::slice::from_raw_parts_mut(buf, len) };
        std::println!(
            "*plugin*: receive value: {:?} length {:?}",
            inbuf,
            inbuf.len()
        );
        let test_value = get_test_value(sub_cmd);
        assert!(test_value.len() <= len);
        std::println!("*plugin*: write value '{}' to buffer", test_value);

        inbuf[0..test_value.len()].copy_from_slice(test_value.as_bytes());
        unsafe {
            *outlen = test_value.len();
        }
        return raw::TEE_SUCCESS;
    }

    #[test]
    fn test_invoke_with_immutable_array_and_vec_result() {
        let mut plugin = Plugin {
            uuid: Uuid::parse_str("7dd54ee6-a705-4e4d-8b6b-aa5024dfcd10").unwrap(),
        };
        const BUFFER_LEN: usize = 32;

        // test calling with &[u8; N], and output less than that
        let parameter = [0_u8; BUFFER_LEN];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN / 2);
        let response = plugin.invoke(1, sub_cmd, &parameter).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response.as_slice(), random.as_bytes());
        // test calling with &[u8; N], and output equals to that
        let parameter = [0_u8; BUFFER_LEN];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN);
        let response = plugin.invoke(1, sub_cmd, &parameter).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response.as_slice(), random.as_bytes());
        // test calling with &[u8], and output less than that
        let parameter = [0_u8; BUFFER_LEN];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN / 2);
        let response = plugin.invoke(1, sub_cmd, parameter.as_slice()).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response.as_slice(), random.as_bytes());
        // test calling with &[u8], and output equals to that
        let parameter = [0_u8; BUFFER_LEN];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN);
        let response = plugin.invoke(1, sub_cmd, parameter.as_slice()).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response.as_slice(), random.as_bytes());
        // test calling with &Vec, and output less than that
        let parameter = vec![0_u8; BUFFER_LEN];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN / 2);
        let response = plugin.invoke(1, sub_cmd, &parameter).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response.as_slice(), random.as_bytes());
        // test calling with &Vec, and output equals to that
        let parameter = vec![0_u8; BUFFER_LEN];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN);
        let response = plugin.invoke(1, sub_cmd, &parameter).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response.as_slice(), random.as_bytes());
    }
    #[test]
    fn test_invoke_with_mutable_array_and_length_result() {
        let mut plugin = Plugin {
            uuid: Uuid::parse_str("7dd54ee6-a705-4e4d-8b6b-aa5024dfcd10").unwrap(),
        };
        const BUFFER_LEN: usize = 32;

        // test calling with &mut [u8; N], and output less than that
        let mut buffer = [0_u8; 32];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN / 2);
        let response: usize = plugin.invoke(1, sub_cmd, buffer.as_mut_slice()).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response, random.len());
        assert_eq!(&buffer[..random.len()], random.as_bytes());
        assert!(buffer[random.len()..].into_iter().all(|&byte| byte == 0));
        // test calling with &mut [u8; N], and output equals to that
        let mut buffer = [0_u8; 32];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN);
        let response: usize = plugin.invoke(1, sub_cmd, buffer.as_mut_slice()).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response, random.len());
        assert_eq!(&buffer, random.as_bytes());
        // test calling with &mut [u8], and output less than that
        let mut buffer = [0_u8; 32];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN / 2);
        let response: usize = plugin.invoke(1, sub_cmd, buffer.as_mut_slice()).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response, random.len());
        assert_eq!(&buffer[..random.len()], random.as_bytes());
        assert!(buffer[random.len()..].into_iter().all(|&byte| byte == 0));
        // test calling with &mut [u8], and output equals to that
        let mut buffer = [0_u8; 32];
        let (sub_cmd, random) = generate_test_pair(BUFFER_LEN);
        let response: usize = plugin.invoke(1, sub_cmd, buffer.as_mut_slice()).unwrap();
        std::println!("*TA*: response is {:?}", response);
        assert_eq!(response, random.len());
        assert_eq!(&buffer, random.as_bytes());
    }
}

You can test it by SYS_BUILD_TYPE=unit_test cargo test -- --show-output

@DemesneGH Got some time to have a look?

Thanks for the solution. Let’s start by clarifying a few points:
What we need to do:

The current implementation has issues that we need to fix.
Additionally, there is a requirement for zero-copy handling of large outputs, which we should incorporate.

We are open to introducing breaking changes if necessary. Seems there're multiple ways to call the plugin.invoke() in the above solution. Developers may be confused about which way should they choose. I prefer re-design the API even if  sacrificing backward compatibility.

        let immutable_buffer = [0_u8; 32];
        let mut mutable_buffer = [0_u8; 32];
        // =================== old feature, pass immutable slice ==================
        // response1 infer to Vec<u8> automatically, keeps compatibility.
        let response1 = plugin.invoke(1, 1, &immutable_buffer).unwrap();
        // this won't compile, as we didn't imply the Wire trait for the type pair
        // and I think codes like this must be mistake from developers, developers
        // must get the buffer back
        let response2: usize = plugin.invoke(1, 1, &immutable_buffer).unwrap();

        // =================== new feature, pass mutable slice ====================
        // developers must specific what they want, keeping developers from misuse
        // currently the valid return types are: Vec<u8> or usize
        let response3: Vec<u8> = plugin.invoke(1, 1, &mut mutable_buffer).unwrap();
        let response4: Vec<u8> = plugin.invoke(1, 1, mutable_buffer.as_mut_slice()).unwrap();
        // this won't compile, as we need developers to specific the input type to &mut [u8]
        let response5: usize = plugin.invoke(1, 1, &mut mutable_buffer).unwrap();
        // this is working like std::io::Read
        let response5: usize = plugin.invoke(1, 1, mutable_buffer.as_mut_slice()).unwrap();

So let’s first sync the requirements and then discuss the new design. Please let me know if I’ve misunderstood anything:
The requirement is:
TA send a small buffer for request and received a large buffer from NW. The buffer cannot be copied due to memory limitation.
For the design let's discuss the questions:

Should developers care about the shared buffer?

Yes? Because developers need dynamic shared buffer. Developers know the approximate output size and they should alloc a shared buffer when preparing the input.

The plugin_invoke() takes these two parameters data_ptr and data_len. they are description of the shared_buffer, or the data_len is size of input_buffer (as the comment for tee_invoke_supp_plugin)?

Seems the len is size of shared_buffer, when invoke it contains the input data, so TA also need to send the input_data size, like request_size you mentioned above but using sub_cmd to send the input size seems slightly confusing, but we can implement it in any suitable way.
Should developers manually set the input_data size, or should our SDK handle it automatically?

Would zero-copy of Vec in Rust solve the problem?

Compared with returning usize, I prefer Vec which is more Rust way and avoid unsafe memory operations by developers themselves. We may seek ways for zero-copy (maybe unsafe, carefully handled by SDK) and return the Vec to developers.

@DemesneGH
For the questions:

Should developers care about the shared buffer?

Yes, not just my cases, in many of the use cases the response size is bigger than request size, and actually they are not really care about the shared buffer, but just want to make sure they can read all the data from REE.

The plugin_invoke() takes these two parameters data_ptr and data_len. they are description of the shared_buffer, or the data_len is size of input_buffer (as the comment for tee_invoke_supp_plugin)?

Yes, the len is the size of share buffer, But we don't need to consider the size of the input data in teaclave, things is complicated here, I will explain later.
Developers should manually set the input_data size this their own

Would zero-copy of Vec in Rust solve the problem?

Yes, but as what I said, my real problem is not we can't copy, is that I need to set the size of the buffer my own but rather than use a immutable input slice with same size as the output Vector(in my case it wastes 100M of memory by using invoke).

How people use plugin with the share buffer
1. Read data from REE
Some people use it to read file content, or make a request to remote and return its result, in scenarios like these, people prepare a large buffer and use different ways to know the input data size from TA:

by const size: every command is combine with a const struct, which is the same layout in both TEE and REE (just like repr(C)), and they read the buffer directly, when upgrading, they use a new struct with new command pair(new command_id or same command_id with different sub_cmd_id).
by sub_cmd: some people use protobuf or json or flatbuffer for request parameter, and use sub cmd for lens.
by first 8 bytes: they write the length (which is u64) to the prefix of buffer.

2. Post something to REE
Some people use it to write data into REE, for example, some secret keys, and the share buffer is used to pass the error description(if result is not TEE_SUCCESS), in scenarios like these, current implementation is enough.
3. Trigger something in REE
Some people use it to trigger something in REE, for example, TEE listening to a trusty remote, and when the maintainers find something wrong with it (a hacker hack in and take control of the system), they trigger a security alarm, or shutdown the system immediately, etc. In scenarios like these, they don't care about the share buffer, sometimes they don't use the sub_command_id either.
Other considerations

Actually plugin system have existed for a long time, there are some plugins developed by C already, so should we keeps compatibility with them?

Yes? => So people don't have to rewrite them.
No? => So we can keep a clean architecture.

My New Design
I have a new design, which is Command struct with chainable api, just like reqwest.Request and ureq.Request.
First, we propose a Command struct, reference to the Plugin and keeps arguments of tee_invoke_supp_plugin inside:
pub struct Plugin {
    uuid: Uuid,
}

pub struct Command<'a> {
    plugin: &'a Plugin,
    cmd_id: u32,
    sub_cmd_id: u32,
    buffer: Vec<u8>,
}

Then, we provide chainable api for developers, just like the following:
impl<'a> Command<'a> {
    // use this to set sub_cmd if needed
    pub fn sub_cmd(mut self, sub_cmd_id: u32) -> Self {
        self.sub_cmd_id = sub_cmd_id;
        self
    }
    // use this to write request body if needed
    pub fn write_body(&mut self, data: &[u8]) {
        self.buffer.extend_from_slice(data);
    }
    // same with write_body, but chainable
    pub fn chain_write_body(mut self, data: &[u8]) -> Self {
        self.write_body(data);
        self
    }
    // invoke the command, and get result from it
    pub fn call(self) -> Result<Vec<u8>> {
        let mut outlen: usize = 0;
        let mut buffer = self.buffer;
        buffer.resize(buffer.capacity(), 0); // resize to capacity first
        match unsafe {
            raw::tee_invoke_supp_plugin(
                self.plugin.uuid.as_raw_ptr(),
                self.cmd_id,
                self.sub_cmd_id,
                buffer.as_mut_slice().as_mut_ptr(),
                buffer.len(),
                &mut outlen as *mut usize,
            )
        } {
            raw::TEE_SUCCESS => {
                assert!(outlen <= buffer.len());
                buffer.resize(outlen, 0);
                Ok(buffer)
            }
            code => Err(Error::from_raw_error(code)),
        }
    }
}

Finally, we add extra api for Plugin, and rewrite the invoke method with it:
impl<'a> Command<'a> {
    fn new(plugin: &'a Plugin, cmd_id: u32) -> Self {
        const DEFAULT_CAPACITY: usize = 1024;
        Self::new_with_capacity(&plugin, cmd_id, DEFAULT_CAPACITY)
    }
    fn new_with_capacity(plugin: &'a Plugin, cmd_id: u32, capacity: usize) -> Self {
        Self {
            plugin,
            cmd_id,
            sub_cmd_id: 0,
            buffer: Vec::with_capacity(capacity),
        }
    }
}

impl Plugin {
    pub fn invoke(&self, command_id: u32, subcommand_id: u32, data: &[u8]) -> Result<Vec<u8>> {
        self.new_command_with_capacity(command_id, data.len())
            .sub_cmd(subcommand_id)
            .chain_write_body(data)
            .call()
    }
    // for people don't care about the response
    pub fn new_command<'a>(&'a self, command_id: u32) -> Command<'a> {
        Command::new(self, command_id)
    }
    // for people who wants to set the buffer size for response manually
    pub fn new_command_with_capacity<'a>(
        &'a self,
        command_id: u32,
        capacity: usize,
    ) -> Command<'a> {
        Command::new_with_capacity(self, command_id, capacity)
    }
}

By this design, developers of different scenarios could:

developers upgrade from old teaclave SDK: nothing changes.
developers who wants to fetch something from REE(which have bigger response size):

const RESPONSE_LEN: usize = 100;
let result_vec = plugin.new_command_with_capacity(command_id, RESPONSE_LEN)
    .sub_cmd(subcommand_id)
    .write_body(request)
    .call()?;

developers who wants to put something to REE: just use the invoke method.
developers who just want to trigger something:

let _  = plugin.new_command(command_id)
    .sub_cmd(subcommand_id)
    .write_body(request)
    .call()?;

they can even imply a writer of it:

// impl a writer for Command
struct Wrapper<'a, 'b>(&'b mut Command<'a>);
impl<'a, 'b> std::io::Write for Wrapper<'a, 'b> {
    fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
        self.0.write_body(buf);
        Ok(buf.len())
    }
    fn flush(&mut self) -> std::io::Result<()> {
        Ok(())
    }
}
// serialize data into command directly
let test_data = serde_json::json!({
    "code": 100,
    "message": "error"
});
const OUTPUT_LEN: usize = 100;
let (sub_cmd, random) = generate_test_pair(OUTPUT_LEN);
let mut cmd = plugin
    .new_command_with_capacity(1, OUTPUT_LEN)
    .sub_cmd(sub_cmd);
serde_json::to_writer(Wrapper(&mut cmd), &test_data).unwrap();
let result_vec = cmd.call().unwrap();

How about this?

Thanks for the new proposal and I agree with the design. I have a few minor recommendations:

We can simplify the scenario by removing this:

    // for people don't care about the response
    pub fn new_command<'a>(&'a self, command_id: u32) -> Command<'a> {
        Command::new(self, command_id)
    }

just keep the legacy invoke() and introduce the new invoke_with_capacity() with similar parameters (cmd, sub_cmd), I would suggest something like:
pub fn invoke_with_capacity(&self, command_id: u32, subcommand_id: u32, capacity: usize)

for developers:
const RESPONSE_LEN: usize = 100;
let result_vec = plugin.invoke_with_capacity(command_id, subcommand_id, RESPONSE_LEN)
    .write_body(request)
    .call()?;

For the PR: In terms of naming e.g Command  use a more specific name such as LoadablePluginCommand for distinguish.

If there's nothing else to discuss regarding the issue, please feel free to open the PR. Thanks again!


DeprecationWarning: builtin type X has no module attribute

On Python 3.7 and up I see warnings of the following form when running the pyo3 test suite:
DeprecationWarning: builtin type X has no __module__ attribute

I think this is probably related to using a #[pyclass] when not adding it to a module. I haven't investigated yet.
Example test which demonstrates this issue:
$ cargo test --test test_various return_custom_class
    Finished test [unoptimized + debuginfo] target(s) in 0.03s
     Running target/debug/deps/test_various-a73ba5b7affaefde

running 1 test
<string>:1: DeprecationWarning: builtin type PyUsize has no __module__ attribute
test return_custom_class ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 7 filtered out

I'm not sure what changes we need to make to resolve.

I think this is probably related to using a #[pyclass] when not adding it to a module. I haven't investigated yet.
This does seem to be the trend with the issue when re-producing, when exposed to the module this error goes away. Only when you dont expose it directly does this warning get raised from my findings

This appears to cause an error when running doctests:
$ python -m doctest -v myfile.py
Traceback (most recent call last):
  File "/home/ben/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ben/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/ben/miniconda3/lib/python3.8/doctest.py", line 2793, in <module>
    sys.exit(_test())
  File "/home/ben/miniconda3/lib/python3.8/doctest.py", line 2783, in _test
    failures, _ = testmod(m, verbose=verbose, optionflags=options)
  File "/home/ben/miniconda3/lib/python3.8/doctest.py", line 1955, in testmod
    for test in finder.find(m, name, globs=globs, extraglobs=extraglobs):
  File "/home/ben/miniconda3/lib/python3.8/doctest.py", line 939, in find
    self._find(tests, obj, name, module, source_lines, globs, {})
  File "/home/ben/miniconda3/lib/python3.8/doctest.py", line 1000, in _find
    self._from_module(module, val)):
  File "/home/ben/miniconda3/lib/python3.8/doctest.py", line 967, in _from_module
    return module.__name__ == object.__module__
AttributeError: __module__

"__module__ is the name of the module the function was defined in or None if unavailable" https://docs.python.org/3/reference/datamodel.html
So perhaps the __module__ attribute should be set to None if the pyclass is not added to a module?

That sounds like a valid solution, though it's not super clear how to do that correctly with PyType_FromSpec. I'll keep playing with it and see if I can get it right.

@bderrett am I right in thinking this is a new regression for PyO3 0.13? It's quite possibly related to the abi3 work / switch to heap types.

@davidhewitt Yes. A bisection shows that it was introduced when the "PyO3/abi3" branch was merged.


Adds API for compute clusters

Changes proposed in this PR

adding API endpoint for creating and query compute clusters

Why are we making these changes?
This is part of a larger effort to support dynamic (k8s) compute clusters in Cook.

This has been absorbed into #1639


when build the project in a script, the index.html is wrong

Versions
<!--
Output from: `ng --version`.
If nothing, output from: `node --version` and `npm --version`.
  Windows (7/8/10). Linux (incl. distribution). macOS (El Capitan? Sierra?)
-->

<!doctype html>Ui
Repro steps

Step 1
Step 2
Step 3

Observed behavior
<!-- Normally this includes a stack trace and some more information. -->

Desired behavior

"C:/msys64" is wrong. I just use a script , build outside the angular project root. Actually , for combine django and angular.
Mention any other details that might be useful (optional)

There is not enough information here, please open another following the new issue template.
Please make sure to include steps to reproduce and the versions you are using.


Cant generate data in GCAM6 and 5.4. Shiny also not working

Hello everyone, I am trying to use the gcamreport package in R and so far I can generate the report in GCAM7.
For GCAM5.4 and 6.0, it only creates the .dat file without the CSV and xlsx files.
I have tried to get the information from the .dat file using the instructions given on the GitHub page but I get the same error as shown in the attached image.
Also, in all three versions of GCAM, the interactive page doesnt open. It says "No Shiny application exists at the path "inst/gcamreport_ui"". Again, I tried the suggested solution and it doesnt work.
Below is the exact code I use for each of the three versions of GCAM.
devtools::load_all()
db_path <- "D:/gcam-v6.0-Windows-Release-Package/output"
db_name <- "SECT_III"
prj_name <- "my-project.dat"
scen_name <- c("CONV")
generate_report(db_path = db_path, db_name = db_name, prj_name = prj_name,
scenarios = scen_name, final_year = 2050, save_output = TRUE, launch_ui = TRUE, ghg_GWP = gcamreport::ghg_GWP_AR4)
I will be very grateful to hear back from you on how to resolve the data generation and shiny app problems at your most convenient time. @klau506 @russellhz @jonsampedro
Thank you.

closing this now, found my mistake and rectified it. I used version6 of the gcamreport and also made sure to use the gcamreport R project. So the above issues have all been resolved now.


Tidy up readme

Add what NWP variables are used. Would be good to give the units of these too

Hey! Could I be assign to this if that’s ok with you? I’m getting familiar with different NWP model and I think that would be a good exercise. Thanks :)

Yes please @chloepilonv , thank you for getting in touch

Would be great to fix the example.
The ts=datetime... needs to be removed


How to extract preact code from the bundle

Hello,
I'm using preact in CMS and sometimes I need 2 different preact bundles on the same page. In this case I have preact code twice. How is it possible to remove preact and preact-compat code from the bundle and handle it separately?
Thank you very much!

Try server-side rendering for one bundle and client-side rendering for the other. This repo might help: https://github.com/developit/preact-render-to-string

Thanks, I'll try this.