buildah-config: created and created-by confusion

The OCI Spec v1.0.1 suggests that neither --created nor --created-by ought to be user configurable.

Created:

combined date and time at which the image was created, formatted as defined by RFC 3339, section 5.6.

Created by:

The command which created the layer.

@rhatdan, I'm 99% sure what you're asking for.  I think you want me to remove the created and created-by options from buildah config and then determine those values when the config happens and save it to the container.  I'll make that so, holler if I need to adjust.

Believe this would also close #537

@TomSweeneyRedHat Yes on the first statement, but created and createdby should be populated on buildah commit, not no buildah config.

@taqtiqa-mark #537 isn't just the format the date is displayed in, the primary issue is that local timezone isn't being applied when the time is displayed to the user.

@pixdrift the created date shown in buildah inspect is in UTC.  However when we display the creation date of the image via buildah images we convert the time displayed by that command to the time of the machine that runs the command as determined by its locale.  We don't change the time inside the image itself though.

@TomSweeneyRedHat this is how I would expect it to work, but my ticket #537 shows that it is reporting the raw time off the container and not adjusting for my local time (AEST).

@pixdrift @TomSweeneyRedHat I think if the spec format was followed then the timezone decoration would explain the 10 hour difference.
My view is to keep things a simple as possible ... so just stick with UTC throughout and format the time as the spec states.  Chopping and changing is problematic.
Also I think a buildah use case is that it will be used in scripted scenarios where you can't guarantee that data being passed around is being passed between two machines in the same TZ or even with the same locale settings.

@taqtiqa-mark I am not suggesting that the date/time be anything other than UTC in the container. I was raising that the spec doesn't stipulate that UTC should be used for this date/time and the spec probably should define this categorically because using RFC3339 (ISO8601 date/time), there is an 'option' to include the TZ date time in the date.
My issue in #537 is that the displayed  CREATED AT time in buildah images is showing the raw UTC time from the container, and not displaying in local time in the user interface. There is no suggestion that anything should be changed in the container configuration using UTC.

@pixdrift ack.  I'm arguing displayed data should be UTC too.  Humans are only one type of consumer of buildah output. Scripts are another important group.  It's much easier for a Human to recognize a UTC time than to write scripts and pipelines that coordinate/adjust time differences.

@taqtiqa-mark ack. I'd argue that buildah images primary consumer is humans. If you want to interface with the container attributes programatically you could use output from buildah inspect. I accept others may argue the counter.
In comparison docker images produces relative time which is also difficult to interpret eg.

3 weeks ago

I'll let @rhatdan chime in on what we should be displaying time wise.  But just to be totally clear, the times that we store for the containers and images are UTC times.  What we display is UTC time with the exception of the time displayed in the buildah images command which converts the time to locale time based on #537.
@pixdrift I'm a little confused (it's late, so likely me) by  your reply above:  "but my ticket #537 shows that it is reporting the raw UTC time off the container and not adjusting for my local time (AEST which is UTC+10)."  #537 was for the image, not the container.  Is the latest Buildah (from GitHub, don't think we've spun a rpm with the fix yet) not showing that for you?  Is there a spot that the time for a container is being displayed that should be showing UTC in your opinion.
I believe #544 addresses #537, OK to close #53 now?

@TomSweeneyRedHat, my issue #537 is about display only, and is in buildah images, you're absolutely correct. I am commenting on it in this thread because it was suggested above that this issue would also close #537, but in my view they are unrelated.
I will follow up #537 in that issue and provide feedback.

@TomSweeneyRedHat, @taqtiqa-mark To summarise above, this proposal/suggestion looks good, my comments were in response to how it related to #537, which is now closed. Crisis over :smile:

How about buildah images does display in LocaleTime and buildah images --json returns in UTC.

Appreciate the proposal.
The std Linux approach is to pipe command output via cut etc. - keep simple things simple.
I've tried parsing json through bash etc.  I don't think it is a solution here.
Ideally you shouldn't need more than what is in busy-box to consume buildah output.
May be a --pipe or --pipe-output option that produces delimited (un-percent-encoded output).  Trouble here is selecting a delimiter that is not part of the content defined in the spec.... may be | as a default?  Is  part of the OCI spec content?
Anyway, this could be made robust by providing an option --pipe-delimiter <char>?
That way if someone encounters output with '|', etc. they can workaround it.
Agree all dates and times sent in response to --pipe should be UTC.

How about if we check if the output is a tty, if yes, then it is converted, if not then it get UTC.  Although this might confuse certain users.  Perhaps we should add a flag to tell it to not convert.
buildah --utc
buildah --noformat
?

@rhatdan, I tend t prefer explicit --utc over implicit --noformat which presumes the user knows what format the raw is. Even more explicit and self descriptive/documenting is --utc-3339.  If (when?) the spec changes, at least there won't be confusion about one parameter in legacy code ;)
@pixdift, json issue yes, overly complex. I understand by using json the developers get some freedom, the tradeoff is that now buildah is one more tool's output for which busy-box isn't sufficient - note not all cases will buildah --pipe be piped. Stored in a file and passed along to be parsed (cat pipe-file.txt|...) is another another regular use case.
Thanks for your time, thought and effort.

@taqtiqa-mark, can you provide some examples of the buildah commands with proposed switches with proposed/expected output?
eg.
buildah images --pipe
buildah images --pipe --utc

Was interested in how this situation was handled in Docker, and it looks like the solution in Docker is to use the more generic --format parameter that is built in to present json output in a format that is easily parseable.
https://docs.docker.com/config/formatting/
When looking for --format in the other projectatomic utilities, I found this in the skopeo README.md.

NOT TODO
provide a format flag - just use the awesome jq

Should there be some discussion about how all the tools under projectatomic umbrella (ie. buildah, podman, skopeo) provide parseable output so that it is consistent?

Refering to the configuration list in issue #591 table
Ideal workflow would look like this, unless I missed something buildah push... doesn't return the container ID in a way that helps....
OCI_ID=$(buildah push ...)
OCI_CONFIG="$(buildah images --pipe --pipe-delimiter='|' |grep ${OCI_ID})"
IFS='|' read arch authors cmd description documentation remain1 <<< ${OCI_CONFIG}
IFS='|' read entrypoint env licenses os port ref_name remain2 <<< ${remain1}
IFS='|' read revision source title url user vendor remain3 <<< ${remain2}
IFS='|' read version volume workingdir <<< ${remain3}

The assumptions are the output is in alphabetical order of the buildah-config flags.  All flags have a default value if they are not set - even if it is empty strings or some placeholder.
Hope that helps?

OCI_CONFIG='amd_64|"Daniel Walsh, pixdrift"|"/bin/sh consul-entrypoint.sh"|"me oh <my@mailinator.com>"|http://www.google.com|1|@|3|&|'
IFS='|' read arch authors cmd description documentation remain1 <<< ${OCI_CONFIG}
echo $arch
echo $authors
echo $cmd
echo $description
echo $documentation
echo $remain1

Is that what you wanted?

@taqtiqa-mark I am interesting in what the output of this command would be. No parsing or interpretation, just what you would like to see printed to the console?
buildah images --pipe --pipe-delimiter='|'
My view is that the Unix Philosophy is absolutely supported in buildah's current form. You are referring to elements that you want to extract from the spec. The method provided to interrogate the spec of the container is buildah inspect. The output of buildah inspect is json, a format that is easily parsed by jq, which is a single program that does a single thing very well (parses json). jq is available on all Linux platforms i've worked on.
I don't fully understand the constraint of limiting to commands in busybox, can you provide some details on the kind of environment you're building containers in that only has busybox?

If I can make another point for jq. The jq solution is more robust than using positional parameters. Even if the output items are sorted alphabetically, if anything is added to the spec (even elements you don't utilise) the script will break. In buildah inspect + jq, you are referencing specific items by name that won't be impacted by additional elements/fields being added to the spec or command output.

I would think the format command would be best to satisfy your needs then you can format the output of buildah images any way you want.
Take a look at podman images --format  would similar functionality from buildah satisfy your needs?

@rhatdan and @pixdrift I've given the concrete use case.
In 6 lines I show an image build and how buildah-config settings can be easily parsed into env variables using prexisting tools.
(For later readers - the buildah-push and buildah-inspect behavior above is vapoware at the time of writing)
I'm not familiar with jq or podman.  Perhaps you can show how you envision the same results can be achieved using those approaches and then the choice will be clearer?

@taqtiqa-mark, I have provided the --format documentation for Docker in my above post, which describes its operation. What @rhatdan is proposing is a re-implementation of this capability in buildah as this capability has already been replicated in podman. To summarise it basically includes a json parsing/templating mechanism in the tool itself to remove the need for jq.

@taqtiqa-mark, As requested, example using jq in current config (random fields selected)
buildah inspect centos:7 | jq '.OCIv1.architecture+"|"+.OCIv1.os+"|"+.OCIv1.config.Cmd[]'
"amd64|linux|/bin/bash"

Thanks @pixdrift for taking the time to do that.  I appreciate we are all doing this in what would otherwise be family or recreation time.
A couple of observations.

It'll be useful to show the full PoC you have in mind that would set the same suite of environment variables I gave in my use case. Ack buildah doesn't (yet?) support setting all that configuration data through buildah config.
You say the idea is to make the functionality/behavior possible available without jq.  Can you share how you think that will work/look - the example uses jq
It'll be useful if all a user needs to know to retrieve, say os, is that it was set via buildah config --os=....  In the proposal to use json (even without jq) it seems I've now got to know in detail buildah's implementation of the spec.  Ideally I should not even have to know any spec detail.  When either the spec or the implementation of it changes, as they surely will, my scripts should not break so long as os is somehow still available in the same N series of buildah.

@pixdrift : Thanks.  Striving to keep things simple and avoid the need to consult docs etc.  Not easy to write intuitive software, I know. Esp when user base is so diverse.

@pixdrift @taqtiqa-mark Lets add the format and see if we can satisfy your needs with that.  Should be simple to back port the changes from podman to buildah images.  @umohnani8 Could you look at that?

@rhatdan agreed, format to align with other tools would be a good solution.

I am not sure what we are missing now?
buildah inspect --format '{{.OCIv1.Created}}' fedora-working-container
2018-03-07 20:51:34.488688562 +0000 UTC

If you want it in namevalue pairs
buildah inspect --format 'created="{{.OCIv1.Created}}"' fedora-working-container
created="2018-03-07 20:51:34.488688562 +0000 UTC"

@rhatdan, this is for @taqtiqa-mark to answer I think.

@rhatdan add --format to buildah inspect only or to images as well?

@umohnani8 We have it now I believe.


Support for find patient medical record command

Resolves #34

Hey could you try and fix code style issues perhaps?


[SPARK-20842] [SQL] Upgrade to 1.2.2 for Hive Metastore Client 1.2

What changes were proposed in this pull request?
Hive 1.2.2 release is available. Below is the list of bugs fixed in 1.2.2
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12332952&styleName=Text&projectId=12310843
How was this patch tested?
N/A

Test build #77191 has started for PR 18063 at commit 5ff0039.

Test build #77191 has finished for PR 18063 at commit 5ff0039.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77191/
Test PASSed.

+1, LGTM.

cc @cloud-fan

retest this please

Test build #79434 has started for PR 18063 at commit 5ff0039.

Test build #79434 has finished for PR 18063 at commit 5ff0039.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79434/
Test PASSed.

Merged build finished. Test PASSed.

LGTM, merging to master!


Document that most constants in FpParameters and elsewhere should be in Montgomery form.

Should this issue now be closed?

No, not quite; FpParameters still requires writing things down in Montgomery form =/

#379 fixes this.


Fix grpc_saga typo

Codecov Report

Merging #126 (6c77f66) into main (ead7455) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #126   +/-   ##
=======================================
  Coverage   96.26%   96.26%
=======================================
  Files          44       44
  Lines        2037     2037
=======================================
  Hits         1961     1961
  Misses         46       46
  Partials       30       30

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ead7455...6c77f66. Read the comment docs.


Upload attachments shouldn't trigger on secondary button click

When I click the secondary button on the 'upload attachments' icon, shown while writing a comment, the file upload dialog is shown. This shouldn't happen as generally a secondary button shouldn't trigger the primary action (uploading files). At least that's what I expect :relaxed:.
FWIW, I observed this issues in Firefox Nightly 62.0a1 running on Debian GNU/Linux.

What's the "secondary button"?
What does that normally do?
What should happen when clicking the secondary button on the upload icon?

Sorry for the delayed response. I was out-of-town for the past week.

What's the "secondary button"?

I should have been more explicit. It's the secondary button of the mouse (the right mouse button, for me)

What does that normally do?

It usually brings up the context menu for the webpage being viewed. It's generally taken care by the browser and looks something like this (Sorry about the orientation, I'm not sure why it changed):

What should happen when clicking the secondary button on the upload icon?

I would expect nothing to happen. But any secondary action would be meaningful. I filed this as the secondary button shouldn't trigger a primary action such as uploading files.

Thanks! I don't see this as a particularly problematic behavior (solution: "don't right click that button") but if you manage to find the solution and send a PR I can take a look at it :)


Cell output height is not shrinked until cell execution completes

Environment data

VS Code version: Version: 1.85.1
Jupyter Extension version (available under the Extensions sidebar): v2023.11.1003402403
Python Extension version (available under the Extensions sidebar): v2023.22.1
OS (Windows | Mac | Linux distro) and version: Local: MacOS Ventura 13.4.1
Python and/or Anaconda version: Python 3.11.7, miniconda3-3.11-23.11.0-2
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pyenv, miniconda
Jupyter server running: Local

Expected behaviour
I would expect it to behave like Jupyter lab, so that when you run a cell, all previous output is cleared and output height is truncated:
https://github.com/microsoft/vscode-jupyter/assets/4602302/f5013645-52bd-4075-85c7-fd987755e5ff
Actual behaviour
If there is a long running cell that has a lot of output, or if you interrupt it and there is large stack trace, then on subsequent cell reruns there remains quite large empty space until execution completes.
https://github.com/microsoft/vscode-jupyter/assets/4602302/cb50fd13-65aa-40e7-82fd-fdfc8d67b5b9

This was changed by design to reduce the amount of scrolling as cells are re-executed, especially for situations such as this: https://github.com/microsoft/vscode/issues/193835

Hmm, ok, I see that there is long history behind this issue, and there are a few related but different scrolling/anchoring artifacts. I admit, I haven't studied them all, and I don't know implementation details, probably for some issues this fix works well, but in this particular case it looks quite questionable to me.
First, if you run a cell for the first time, its output will still grow, so you will still have that scrolling, which you should handle somehow.
Second, even on second re-run, output might be longer, so again you still can't avoid scrolling.
In one of the issues (https://github.com/microsoft/vscode/issues/196465) you said:

The whitespace is just holding the position for the output to re-appear and reduces the amount of jumpiness while re-executing cells.

You are probably assuming that the new output will be of the same size as previous one, but it's surely not guaranteed. For example you might run a cell and by mistake it produced huge output, so you quickly corrected your mistake and re-run the cell, but it could take long time to execute and you will be left with annoying whitespace all that time, until execution finishes.


Tag the Image with a Version. Currently only "latest".

As there is now a stable version of the dashboard it should be tagged with a version number.

The latest Image is tagged with v0.1 . The versioning of the ankaios dashboard will follow the versioning of the ankaios api.


Settings lock

This PR creates a /lock endpoint that can be used to lock and unlock the device settings.When the lock is active any attempt to update the /settings endpoint will fail with a 409 status.
NOTE: the settings lock DOES NOT currently prevent restart or restore operations on the /settings endpoint.
A GET on /lock will return the state of the device
{
"state": "locked"
}
or
{
"state": "unlocked"
}
A PUT on /lock will attempt to lock the settings. Requests must include a JSON body with a pwd member. The value of pwd should be a password which is used to validate, or create/store a signature in the device_config file. Ex request body...
{
"pwd": "SuPeR SeCrEt"
}
If the lock is active - the value pwd must be the same as the value used when the lock was created. A match will unlock the device.
If the lock is not active, a lock will be created using the value of pwd.
The response body of PUT upon success is the new state of the device (same format as returned by GET).
If an error occurs the /lock endpoint returns an appropriate http status.
400 - pwd member missing
403 - incorrect value for pwd

@heythisisnate I suggest reviewing and merging this PR into #140 before doing the full validation of nodemcu 3.0


KAFKA-4923: Add Streams EOS integration tests

Currently, only 4 out of 7 tests work. We should try to fix shouldBeAbleToPerformMultipleTransactions first.

https://github.com/apache/kafka/pull/2994 fixed shouldBeAbleToPerformMultipleTransactions
shouldNotViolateEosIfOneTaskFails is still flaky -- it sometimes fails with
[2017-05-17 17:31:59,893] ERROR [Transaction Coordinator 0]: transactionalId: appId-1-0_0 -- Current state is Empty, but received transaction marker result: ABORT (kafka.coordinator.transaction.TransactionCoordinator:99)
[2017-05-17 17:31:59,895] ERROR aborting producer batches because the transaction manager is in an error state. (org.apache.kafka.clients.producer.internals.Sender:208)
org.apache.kafka.common.KafkaException: Unhandled error in EndTxnResponse: The producer attempted a transactional operation in an invalid state
    at org.apache.kafka.clients.producer.internals.TransactionManager$EndTxnHandler.handleResponse(TransactionManager.java:759)
    at org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:529)
    at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:100)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:378)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:196)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:152)
    at java.lang.Thread.run(Thread.java:745)

Other times it fails with:
Expected :<[KeyValue(0, 0), KeyValue(0, 1), KeyValue(0, 2), KeyValue(0, 3), KeyValue(0, 4), KeyValue(0, 5), KeyValue(0, 6), KeyValue(0, 7), KeyValue(0, 8), KeyValue(0, 9), KeyValue(0, 10), KeyValue(0, 11), KeyValue(0, 12), KeyValue(0, 13), KeyValue(0, 14), KeyValue(0, 15), KeyValue(0, 16), KeyValue(0, 17), KeyValue(0, 18), KeyValue(0, 19)]>

Actual   :<[KeyValue(0, 0), KeyValue(0, 1), KeyValue(0, 2), KeyValue(0, 3), KeyValue(0, 4), KeyValue(0, 5), KeyValue(0, 6), KeyValue(0, 7), KeyValue(0, 8), KeyValue(0, 9), KeyValue(0, 10), KeyValue(0, 11), KeyValue(0, 12), KeyValue(0, 13), KeyValue(0, 14), KeyValue(0, 10), KeyValue(0, 11), KeyValue(0, 12), KeyValue(0, 13), KeyValue(0, 14), KeyValue(0, 15), KeyValue(0, 16), KeyValue(0, 17), KeyValue(0, 18), KeyValue(0, 19)]>

	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
	at org.apache.kafka.streams.integration.EosIntegrationTest.checkResultPerKey(EosIntegrationTest.java:234)
	at org.apache.kafka.streams.integration.EosIntegrationTest.shouldNotViolateEosIfOneTaskFails(EosIntegrationTest.java:527)

For this case, it read ABORTED data (ie, records with value 10 to 15).

Rebased and fixed checkstyle.
https://github.com/apache/kafka/pull/3082 fixes shouldNotViolateEosIfOneTaskFails

Call for review @enothereska @dguy @bbejeck (One test is still WIP and disabled on purpose -- hope to add it today.)

LGTM - just left one minor comment

We probably also need a test that ensures that data in the state stores adheres to exactly once, i.e., when there is a hard crash and recovers. Probably a system test.

see disabled test shouldNotViolateEosIfOneTaskFailsWithState :)
And yes, this is just a first PR adding test -- we will need more, including system tests.

LGTM. I'm wondering if EoS will/should lead to duplication of several existing integration tests, in which case it would be good to parametrise them. Good start though.

@enothereska We could do that -- but we need to make sure, we can get a 100% deterministic result (think KTable joins which is somewhat non-deterministic)

Updates this.

@mjsax related test fails?
org.apache.kafka.streams.integration.EosIntegrationTest.shouldNotViolateEosIfOneTaskFails

@guozhangwang Seems like -- but the build was deleted already. Did you copy the stack trace?

Retest this please.

Test failure unrelated.

Retest this please.

Triggering some more retests to see if it's stable (as we got one test failure above but no Stacktrace -- unclear what the problem might be.)

Retest this please.

Retest this please.

Retest this please.

Retest this please.

Failed with
java.lang.AssertionError: Condition not met within timeout 60000. Should receive uncaught exception from one StreamThread.
	at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:274)
	at org.apache.kafka.streams.integration.EosIntegrationTest.shouldNotViolateEosIfOneTaskFails(EosIntegrationTest.java:406)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

@guozhangwang Is this the same exception from the failure above you reported?

retest this please

Closing this PR in favor of #3193


[1327] Added level to the course serializer

Context
Course level can only be really one of 3, primary, secondary and further education
Changes proposed in this pull request
Added course level derived from course subject and/or course title

 Course - Basic detail tab Level field (backend)


feat: tx bundle refactor

Start of a very large initiative to refactor how transactions are constructed on the Aave Interface, beginning with supply.

[x] Create separate supplyBundle method to support entire lifecycle and options for supply
[x] Modify EthereumTransactionTypeExtended to pre-compute tx data and gas estimation
[ ] Re-write gas estimation utils to work with this new transaction type ^
[ ] Split pool helpers into smaller methods and generalize for use with other actions
[ ] Unit test coverage

Old Workflow:
Anytime there is a state update in the modal call supply utility which checks approval and generates bundle of action + optional approval txs of EthereumTransactionTypeExtended.
Frontend detects if asset and action support permit, if so they use the result from above to determine if the asset needs to be signed, if so, separately fetch data to sign with signERC20Approval, deconstruct and pass to supplyWithPermit to generate signed tx
New Workflow:
On first load call supplyBundle utility to fetch ActionBundle.
If the approvals array is empty, then action.tx should be ready to submit.
If not, then depending on users preference and asset availability:

Permit: sign messages in signatureRequests, then pass signatures into generateSignedAction. Submit this data or call refreshSignedTxData with any updated parameters.
Approval: submit txns in approvals, then submit action.tx or call refreshTxData with any updated parameters

Link T-5843


Unable install Sample flutter app in TV emulator

E:\Android_Flutter_Project\demo> flutter-tizen doctor -v
[!] Flutter (Channel [user-branch], 3.22.2, on Microsoft Windows [Version 10.0.19045.4651], locale en-IN)
! Flutter version 3.22.2 on channel [user-branch] at E:\Android_Flutter_Project\flutter-tizen\flutter
Currently on an unknown channel. Run flutter channel to switch to an official channel.
If that doesn't fix the issue, reinstall Flutter by following instructions at https://flutter.dev/docs/get-started/install.
! Upstream repository unknown source is not a standard remote.
Set environment variable "FLUTTER_GIT_URL" to unknown source to dismiss this error.
• Framework revision 761747bfc5 (10 weeks ago), 2024-06-05 22:15:13 +0200
• Engine revision edd8546116
• Dart version 3.4.3
• DevTools version 2.34.3
• If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update
checks and upgrades.
[√] Tizen toolchain - develop for Tizen devices
• Framework revision ecb91197f9 (4 weeks ago), 2024-07-16 15:03:41 +0900
• Engine revision 84ee23c69b
• Embedder revision 36f818a8ab
• Tizen Studio 5.6 at E:\tv\tizen-studio
• .NET SDK 8.0.303 at C:\Program Files\dotnet\dotnet.exe
[√] Windows Version (Installed version of Windows is version 10 or higher)
[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
• Android SDK at C:\Users\jayden\AppData\Local\Android\Sdk
• Platform android-34, build-tools 34.0.0
• ANDROID_HOME = C:\Users\jayden\AppData\Local\Android\Sdk
• Java binary at: C:\Program Files\Android\Android Studio1\jbr\bin\java
• Java version OpenJDK Runtime Environment (build 17.0.10+0--11572160)
! Some Android licenses not accepted. To resolve this, run: flutter doctor --android-licenses
[√] Chrome - develop for the web
• Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe
[X] Visual Studio - develop Windows apps
X Visual Studio not installed; this is necessary to develop Windows apps.
Download at https://visualstudio.microsoft.com/downloads/.
Please install the "Desktop development with C++" workload, including all of its default components
[√] Android Studio (version 2023.3)
• Android Studio at C:\Program Files\Android\Android Studio1
• Flutter plugin can be installed from:
https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 17.0.10+0--11572160)
[√] VS Code, 64-bit edition (version 1.82.3)
• VS Code at C:\Program Files\Microsoft VS Code
• Flutter extension version 3.94.0
[√] Connected device (4 available)
• Windows (desktop)       • windows        • windows-x64    • Microsoft Windows [Version 10.0.19045.4651]
• Chrome (web)            • chrome         • web-javascript • Google Chrome 127.0.6533.100
• Edge (web)              • edge           • web-javascript • Microsoft Edge 127.0.2651.98
• Tizen t-0813-1 (mobile) • emulator-26101 • flutter-tester • Tizen 8.0 (emulator)
[√] Network resources
• All expected network resources are available.
! Doctor found issues in 3 categories.
E:\Android_Flutter_Project\demo> flutter-tizen devices
Found 4 connected devices:
Windows (desktop)       • windows        • windows-x64    • Microsoft Windows [Version 10.0.19045.4651]
Chrome (web)            • chrome         • web-javascript • Google Chrome 127.0.6533.100
Edge (web)              • edge           • web-javascript • Microsoft Edge 127.0.2651.98
Tizen t-0813-1 (mobile) • emulator-26101 • flutter-tester • Tizen 8.0 (emulator)
I have also create samsung ceritificate using GUI Certificate manager but still i am getting below error i dont know whats the issue is there anyone who successfully build flutter tv app for samsung  and i cant find  any demo app in flutter and proper doucmentation for flutter setup
flutter-tizen run
Launching tizen\flutter\generated_main.dart on Tizen t-0813-1 in debug mode...
The DemoApp profile is used for signing.
Building a Tizen application in debug mode...                      87.5s
√ Built build\tizen\tpk\com.example.demo-1.0.0.tpk (33.4MB)
Installing build\tizen\tpk\com.example.demo-1.0.0.tpk...           15.6s
Installing TPK failed:
WARNING: Your data are to be sent over an unencrypted connection and could be read by others.
pushed     com.example.demo-1.0.0.tpk     0%          0MB           0KB/spushed     com.example.demo-1.0.0.tpk   12%          4MB
0KB/spushed
com.example.demo-1.0.0.tpk       22%          7MB        3082KB/spushed     com.example.demo-1.0.0.tpk   39%         13MB        5977KB/spushed
com.example.demo-1.0.0.tpk       53%         17MB        2996KB/spushed     com.example.demo-1.0.0.tpk   60%         20MB        3452KB/spushed
com.example.demo-1.0.0.tpk       68%         22MB        3202KB/spushed     com.example.demo-1.0.0.tpk   75%         25MB        2867KB/spushed
com.example.demo-1.0.0.tpk       83%         27MB        2699KB/spushed     com.example.demo-1.0.0.tpk   91%         30MB        2836KB/spushed
com.example.demo-1.0.0.tpk      100%         33MB        3145KB/s
1 file(s) pushed. 0 file(s) skipped.
E:\Android_Flutter_Project\demo\build\tizen\tpk\com.example.demo-1.0.0.tpk   2977KB/s (35061071 bytes in 11.497s)
install com.example.demo
package_path /home/owner/share/tmp/sdk_tools/com.example.demo-1.0.0.tpk
app_id[com.example.demo] install start
app_id[com.example.demo] installing[10]
app_id[com.example.demo] installing[21]
app_id[com.example.demo] install failed[118, -12], reason: Check certificate error :
spend time for wascmd is [3946]ms
Error launching application on Tizen t-0813-1.

Would you take a look at https://github.com/flutter-tizen/flutter-tizen/wiki/TPK-installation-troubleshooting? Was the emulator running when you were creating the distributor certificate (so that the device DUID is registered properly)?


[examples/with-sentry] Better docs + correct dependencies

Based on feedback from the original PR #6362

Thank you!

It shouldn't be the case... why next isn't injecting SENTRY_DSN properly?

Does index.js in pages still necessary ?


Release 3.0.28 on Docker HUB

Version 3.0.28 was released a week ago, together with Java SDK, but there is no docker tag.
https://hub.docker.com/_/orientdb?tab=tags&page=1
Thanks

Hi @leszekgruchala
It's just taking a bit more than expected (there was an error in the dockerfiles and I had to re-submit the release), hopefully it will be available in next few hours
Thanks
Luigi

Hi @leszekgruchala
Now you can find OrientDB v 3.0.28 on DockerHub https://hub.docker.com/_/orientdb
Thanks
Luigi


Added ability to set position with Vector2

Added ability to set position of scene2d.Actor by using a Vector2. This just means it is more convenient when changing an actors position.

I love you.

I have mixed feelings about such convenience methods. What are others' thoughts? This particular method is probably called pretty commonly. If this does get merged, it is not an invitation to add a Vector2 method everywhere x,y is passed -- see the mess from doing this in Intersector.

What about set position from Vector2 with alignment as well?
In my experience I've never wanted to use a Vector2 to store a position, it's always been easier to use 2 primitives. But I could see if you're wanting to use the vector methods all the time it might be easier to keep it in Vector2 form. However, if you're storing it in Vector2 form is it really that much worse to write setPosition(v.x, v.y)? And you have access to setting with alignment as well already - for example setPosition(v.x, v.y, Align.center). I don't know if it's saving much effort but I don't know if I see the problem with it either.

This change, beyond the fact that we will require such methods for most of the stuff, will encourage the use of vector2. Now this isn't bad, but most ppl won't catch the fact that they will need to manage the thousands of vector2 objects with pools and we will get a lot of forum/github noise around GC issues.
Now, for the wise coder that needs this and who takes care of his/her garbage he can always add 1 liner static methods in utility classes like
Util.setActorPosition(Actor,Vector2);
and in any shape, form and parameters he requires to match his use case and speed his development.
And although that's not that pretty, it does the job. Note that no project will need Vector2 convenience methods for everything and everywhere.
Finally in some places the grouping of the data isn't clear and it depends on what the coder wants: What if I'm storing my x,y position along with say, scale or some other datum in a Vector3?
I say no.

I'm not a fan of adding these kinds of methods, usually it just results in cluttered classes and confusing API.  I'm all for convenience methods if the trade off is worth it, which I personally don't is the case here.


Error When Writing out GT_donors.vcf

Hi there,
I am working with vireo v0.1.8 in python. I am running the command:
vireo -c $CELL_DATA -d $DONOR_GT_FILE -o $OUT -N 5

and I have received the following error when it is trying to write out the final vcf:
Traceback (most recent call last):
  File "/path/to/bin/vireo", line 11, in <module>
    load_entry_point('vireoSNP==0.1.8', 'console_scripts', 'vireo')()
  File "/path/to/lib/python3.7/site-packages/vireoSNP-0.1.8-py3.7.egg/vireoSNP/vireo.py", line 232, in main
  File "/path/to/lib/python3.7/site-packages/vireoSNP-0.1.8-py3.7.egg/vireoSNP/utils/vcf_utils.py", line 223, in write_VCF
  File "path/to/lib/python3.7/site-packages/vireoSNP-0.1.8-py3.7.egg/vireoSNP/utils/vcf_utils.py", line 223, in <listcomp>
IndexError: list index out of range

It writes the header but nothing else for the vcf. For reference this is the header:
##source=cellSNP_v0.1.3
##FILTER=<ID=PASS,Description="All filters passed">
##FILTER=<ID=.,Description="Filter info not available">
##INFO=<ID=DP,Number=1,Type=Integer,Description="total counts for ALT and REF">
##INFO=<ID=AD,Number=1,Type=Integer,Description="total counts for ALT">
##INFO=<ID=OTH,Number=1,Type=Integer,Description="total counts for other bases from REF and ALT">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=GL,Number=3,Type=String,Description="Genotype likelihood">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="total counts for ALT and REF">
##FORMAT=<ID=AD,Number=1,Type=Integer,Description="total counts for ALT">
##FORMAT=<ID=OTH,Number=1,Type=Integer,Description="total counts for other bases from REF and ALT">
##FORMAT=<ID=ALL,Number=5,Type=Integer,Description="total counts for all bases in order of A,C,G,T,N">
##contig=<ID=1>
##contig=<ID=2>
##contig=<ID=3>
##contig=<ID=4>
##contig=<ID=5>
##contig=<ID=6>
##contig=<ID=7>
##contig=<ID=8>
##contig=<ID=9>
##contig=<ID=10>
##contig=<ID=11>
##contig=<ID=12>
##contig=<ID=13>
##contig=<ID=14>
##contig=<ID=15>
##contig=<ID=16>
##contig=<ID=17>
##contig=<ID=18>
##contig=<ID=19>
##contig=<ID=20>
##contig=<ID=21>
##contig=<ID=22>
##contig=<ID=X>
##contig=<ID=Y>
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  193.CEL      146.CEL      133.CEL donor1  donor2  donor3  donor4

The number of individuals in this file should be 5 (only 3 genotyped) so I'm not sure why there are so many extra donors.
Any suggestions?
Thanks!

Hi,
This indeed looks odd, but I can't tell the issue immediately. Could you provide more details, e.g., the log print out? Also, I assume that when you don't specify the donor_vcf file, vireo works properly, right?
Thanks,
Yuanhua

Yes, you are right, I just ran
 vireo -c $CELL_DATA -o $OUT -N 5
and had no errors.
Here is the log print out for
vireo -c $CELL_DATA -d $DONOR_GT_FILE -o $OUT -N 5
[vireo] Loading cell VCF file ...
[vireo] Loading cell VCF file, done.
[vireo] Loading donor VCF file ...
[vireo] 285 out 1379 variants matched to donor VCF
[vireo] Loading donor VCF file, done.
[vireo] Demultiplex 1442 cells to 5 donors with 285 variants.
[vireo] warm-up: 50 random initializations for 7 clusters...
donor1  donor6  donor0  donor5  donor4  donor3  donor2
306     271     297     140     116     91      221
[vireo] warm-up: lower bound ranges [-13770.6, -12717.9, -11990.2]
[vireo] pre-RUN: finding 4 from 5 donors without GT ...
donor4  donor0  donor1  donor2  donor3
133     281     502     305     132
[vireo] main RUN with warm initials and tuned GT ...
[vireo] main RUN: 27 iterations; lower bound -11725.8
[vireo] beta parameters for binomial rate:
[[229.2 15526.1]
 [8082.9 9962.8]
 [8594.6 164.4]]
133.CEL      146.CEL      193.CEL      donor1  doublet unassigned
274     483     244     109     65      267
Traceback (most recent call last):
  File "/share/ClusterShare/software/contrib/sccg/anaconda3/bin/vireo", line 11, in <module>
    load_entry_point('vireoSNP==0.1.8', 'console_scripts', 'vireo')()
  File "/share/ClusterShare/software/contrib/sccg/anaconda3/lib/python3.7/site-packages/vireoSNP-0.1.8-py3.7.egg/vireoSNP/vireo.py", line 232, in main
  File "/share/ClusterShare/software/contrib/sccg/anaconda3/lib/python3.7/site-packages/vireoSNP-0.1.8-py3.7.egg/vireoSNP/utils/vcf_utils.py", line 223, in write_VCF
  File "/share/ClusterShare/software/contrib/sccg/anaconda3/lib/python3.7/site-packages/vireoSNP-0.1.8-py3.7.egg/vireoSNP/utils/vcf_utils.py", line 223, in <listcomp>
IndexError: list index out of range

Thanks for your hep!

I also just realized that when I use genotype data, vireo reports for one less individual than the number of individuals that I tell it are in the multiplexed sample. In other words, it only indicates 4 individuals when there are 5 individuals in the multiplexed sample. I think this may contribute to the vcf writing issue. Here is the summary.tsv when N = 5:
ar1    Freq
133.CEL      274
146.CEL      483
193.CEL      244
donor1  109
doublet 65
unassigned      267

The command for that output was:
vireo -c $CELL_DATA -d $DONOR_GT_FILE -o $OUT -N 5
-Drew

Yuanhua,
I'm not sure that I fully understand what you mean by a sample that "dropped out". If you mean that there aren't any cells from that individuals in the multiplexed batch, I don't think that is the case because I find all samples in the batch when I run without genotyping information. Further, this is consistently happening across 10 multiplexed batches (I have just provided the smallest one as an example).
I just ran what you suggested and here is the output:
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from vireoSNP import load_VCF
>>> from vireoSNP.utils.vcf_utils import parse_donor_GPb
>>> donor_vcf = load_VCF("SUB2019_10_09_MoST_SNP_genotypesRprocessedNA_rmSORTED.vcf.gz", biallelic_only=True, sparse=False, format_list=["PL"])
>>> donor_GPb = parse_donor_GPb(donor_vcf['GenoINFO']["PL"], "PL")
>>> print(donor_GPb.shape)
(285, 3, 1)
>>> print(donor_vcf['samples'])
['193.CEL', '146.CEL', '133.CEL']```

I'll send you the first few lines of my vcf by email as well.

Thanks,
Drew

Hi Drew,
Thanks for checking this and sending through the test file. It looks that the PL tag for gentoype (Phred-scaled likelihood) is not available and introduce false values. This bug has been probably introduced in recent upgrade for supporting arbitrary FORMAT tag in donor vcf file.
In short, you can add -t GT in your command line, and it should work. I will fix this bug to automatically detect the gentoype tag in new release.
Thanks a lot!
Yuanhua

Great, pretty easy fix. Thanks for working to figure out the solution!
Cheers,
Drew

Hi Drew,
BTW, the number of SNPs (i.e., 1379) in your cell CVF seems only marginally sufficient. If you got too many unassigned cells, you could try some efforts to increase the usable SNPs, e.g., pileup ~7 million common human SNPs (AF5e2): https://sourceforge.net/projects/cellsnp/files/SNPlist/. Also, you could consider impute your known genotype if it is human data and you haven't done so.
Best
Yuanhua


1_16_R3 incompatible

Describe the bug
A clear and concise description of what the bug is.
NPCLib usage option
NPCLib can be used in multiple ways, please provide the number corresponding to your usage option.
Option 3
NPCLib JAR name
npclib-plugin-2.12.1-SNAPSHOT.jar
Server version
E.g. CraftBukkit version git-Spigot-a1f2566-97b0057 (MC: 1.13.2) (Implementing API version 1.13.2-R0.1-SNAPSHOT) (use command /version)
Purpur version git-Purpur-1131 (MC: 1.16.5) (Implementing API version 1.16.5-R0.1-SNAPSHOT)
To Reproduce
Steps to reproduce the behavior:
Just start your server With purpur 1.16.5

same problem

up to v1_16_R3 == 1.16.5
I think you need an update

I checked my server with Bukkit.getBukkitVersion() and getServer().getClass().getPackage().getName()
I got 1.16.5-R0.1-SNAPSHOT and org.bukkit.craftbukkit.v1_16_R3 back. But there is still this Error
NPCLib Failed to initiate. Your server's version (v1_16_R3) is not supported
The plugin try to get it with  this.getServer().getClass().getPackage().getName().split("\.")[3]; I got v1_16_R3
But why it cannot get with Class.forName("net.jitse.npclib.nms." + versionName + ".NPC_" + versionName); the right class

I figured it out. You need to shade the -plugin and not -api version because the api version does not include the nms packages thus not finding the classes.

Yes you need
 <dependency>
       <groupId>net.jitse</groupId>
        <artifactId>npclib-plugin</artifactId>
         <version>2.12.1-SNAPSHOT</version>
          <scope>compile</scope>
  </dependency>

In Intellij you must exract the project to the root of your plugin.
(Sorry for my english)


Unable to start test validator.

The error log in .anchor/test-validator.log simply had the word "Initializing..." in it. I tried running it solana-test-validator with the --no-bpf-jit flag and anchor test with  --skip-local-validator but had no luck. I got the following error
Error: failed to send transaction: Transaction simulation failed: Attempt to load a program that does not exist
But I noticed that sometimes the tests worked! I finally figured out that if I launch solana-test-validator with --reset flag, close that terminal or ctrl+c out of it and then run anchor test, the tests run smoothly! If I try running anchor test again, it goes back to the original error Unable to start validator.
P.S. I use arch btw
Jokes aside, im on Kernel 5.16.11

Heres a screenshot of it happening.

When tests work only some of the time for seemingly random reasons, it's usually about commitment level/not giving enough time for the test validator to start(e.g. on a slower machine).


IDL generation through compilation

I've implemented an IDL generation method which instead of parsing the program source files generates the idl generation code by hooking into various anchor derive and attribute macros. This IDL generation code is then executed by running cargo test on the program crate. The benefit of this approach is that we're now able to evaluate expressions within the program source which allows for embedding additional information into the IDL.
The scope of this PR is to cover the same features that the current (parse) IDL generation has (I haven't implemented any of the eval stuff from #1927), with the addition of supporting external types (discussed #1972).
The change is fairly big so I've seperated it into multiple stand-alone commits to make the review easier. I strongly suggest to merge this via a rebase vs squash for the purposes of git history.
In summary, here is the diff between the IDLs generated by the parse and gen methods: https://www.diffchecker.com/UzVPnS0o
(for the following test program here)
Also, generics on accounts and AnchorSerialize structs are now supported. Here's the IDL it generates https://gist.github.com/kklas/1617d2b636e8d24e6789f3cf99519b1f generated for this program.
Notes

I have put the new IDL generation behind the anchor idl build command. The parsing method is still default when running anchor build so this PR doesn't actually introduce any breaking changes.
The changes to anchor macros have been put behind a idl-gen feature flag so there's no idl generation code generated when it's not enabled.
The new IDL references defined types via their "full_path" field vs "name". This means the new IDL cannot be used with old clients as they will search the types in the "types" field by their name.  We can fix the clients to support both IDLs by e.g. searching for the name and then full_path (or vice versa).
SOLVED ~The new IDL generation method currently doesn't support generics (i.e. if you have account, event, etc. structs which are generics, the generation will fail with a compile error). There's a way to go around this which I believe would work #2013~
Since it's not possible to know which events are being referenced in the program instructions, the idl generation for events is done slightly differently than accounts (there's a separate test method which prints the IDL for each event). This means that external events (event structs imported from different crates) will not be included. I really like the idl import macro solution for these edge cases #1972). With the compilation approach it should be fairly straightforward to implement.
DEPRECATED I didn't implement "state" stuff because it's going to get deprecated
PDA parsing is not covered here - we can add it later with eval
relations derivations is supported
in order to support generics without creating any breaking changes to the  current IDL generation method, I had to introduce a few new IDL types (lang/syn/src/idl/types.rs)
idl-gen feature needs to be added to the [features] section of Cargo.toml of programs, including all external crates that we import types to be included in the IDL, e.g.:

[features]
idl-gen = [
    "anchor-lang/idl-gen",
    "some-external-program/idl-gen",
]

Implementation Details
The basis of the IDL generation is the hook in the AnchorSerialize derive macro (gated behind 'idl-gen' feature). It generates the following methods for each type that derives AnchorSerialize:

__anchor_private_full_path() - calling this will return the type's full path (implemented using the  std::module_path! macro)
__anchor_private_gen_idl_type() - this will return the IdlTypeDefinition for the type when called
__anchor_private_insert_idl_defined(defined_types: &mut HashMap<String, IdlTypeDefinition>) - a helper method  which will insert any defined types in the types fields into the provided defined_types map.

#[zero_copy] does the same as above and #[account] and #[account(zero_copy)] are covered due to the fact that these attribute macros already add #[zero_copy] and #[derive(AnchorSerialize)] respectively.
#[derive(Accounts)] adds a __anchor_private_gen_idl_accounts method which will return Vec<IdlAccountItem> when called. It also adds any types referenced within Account<...> or AccountLoader<...> into the provided defined_types hash map.
#[program]  generates a __anchor_private_print_idl_program function which is marked as a #[test]. When executed it will collect all the accounts from ctx by calling __anchor_private_gen_idl_accounts on the ctx type, and __anchor_private_gen_idl_type on each ix arg. This is how all the defined types are collected recursively. It will print out the Idl json.
#[event], #[error_code], and #[constant] work a bit different in that they also generate an __anchor_private_print... function marked as a test to print them out. This is because it's not possible to know which events, etc. are being referenced and we can't collect them recursively in #[program] as mentioned in the note 5).
Then the anchor idl build command runs cargo test __anchor_private_print_idl --features idl-gen -- --show-output --quiet on the crate and collects the Idl jsons from the stdout printed by the generated test methods. The events, errors, and constants are printed separately and are combined by the command into the final IDL json which is then outputted to stdout.
When running anchor idl build on the workspace with multiple programs, it will collect all the IDLs and print them in form of an array: [<idl1>, <idl2>]

What implications if any does adding prefixes to type definition names like idl::Bar and extern_program::some_module::Foo have on type inference during IDL parsing in the TS package?

What implications (if any) does adding prefixes to type definition names like idl::Bar and extern_program::some_module::Foo have on type inference during IDL parsing in the TS package?

I touched upon this in note 3). It means that current clients wont be able to parse the new IDL successfully but we can make the new clients work with both IDLs easily.
Alternatively we could have the "name" be default and use "full_path" only when referring to an external type.
The reason I did it this way is because it was simpler to implement and we can have this discussion later when we actually make the compile method the default.

ahh must have missed that part. so the introduction of the path segmented naming via full_path was added for name conflicted resolution (correct me if I'm wrong). would it not make more sense for backwards compatibility to do ExternProgramBaz as the name (converting the crate name to title case as the prefix) instead of a new full_path and doing extern_program::...::Baz that does the resolution?

Seems sad to let this much work go to waste but i don't really know if it would be possible to merge. The IDL stuff isn't something I'm greatly familiar with yet

I will take a look into this again when I finish what I'm currently working on and try to push it through. I think there might be a cleaner solution using schemars which seemingly already implements the functionality of this PR and also supports generics.

Wooh! Everyone I've talked to really likes this approach over the current IDL approach. So I really hope to be able to see this PR reviewed and merged into anchor

Yes, I also think this is the right approach. My only concern with it previously was the generics support, but now with this implemented it should be very solid. It needs to be tested on more programs but I wouldn't be surprised it's already more reliable than the current method, esp. w.r.t. generics.
There's still a bit of work to be done before it can replace the current IDL approach though, and it will require breaking changes to the IDL. I think that should be done in a separate PR, probably along with other breaking changes to the IDL (I feel like some things need to be consolidated in the IDL format).
Also the PDA support needs to be added here. It will probably work differently than the current PDA parsing method (and thus be a breaking change), but will be much more reliable (currently the PDA generation is hidden behind a feature and doesn't work well in many cases from my experience). I already made a POC for this in #1927.
Even so, this PR will already be useful for some people as it introduces:

support for including external types into the IDL (including from other crates)
proper support for generics

What I think is the right way forward:

merge this as a separate cli command anchor idl build so that people can test it out on their programs and edge cases are discovered if any
in the meantime, the PDA support is worked out, IDL format is consolidated (introducing breaking changes if needed), and support for parsing the new IDL is added to the clients
after the above is done and stable, this method is made the default one and the old one is marked deprecated

W.r.t. the review, yea it will be a though review, but I'm happy to hop on a call with whoever will be reviewing this to go through the changes and provide context. But if there's only one commit to review it's this one https://github.com/coral-xyz/anchor/pull/2011/commits/5da2b51d95bf77bbacffd35469e3cfa7c0295a7e which gates the changes to the derive and attribute macros behind the idl-gen flag. So if this is OK, there are no actual changes to the generated BPF programs. There are also multiple pure refactoring commits which should be a straightforward review.

@acheroncrypto

How should the external wrapper types like TokenAccount work? Currently it doesn't seem to build the IDL with an SPL account. Should we skip the known types?

Do you mean that the IDL build fails with the compilation error or they're just not included? I think for this PR they should be skipped to keep it minimal and this functionality added later on.
The general problem with external wrappers is that they each have their own specific encoding / decoding method and this info needs to be included in the IDL also so the clients know how to enc/dec those accounts. I think that wrapper types can implement their own __anchor_private_gen_idl_type method manually to pass this info in.

How should a generic accounts struct work?

This is an entirely different problem far out of scope of this work. The question you should be asking is whether you want to implement this to make derive(Accounts) structs reusable (avoid copy paste on similar structs) or you want to enable generic CPI calls. If it's the former then in my opinion this shouldn't be allowed because it results in very complex code and is a foot-gun. If it's the latter then this is a very different problem to solve (and a tough one). It's not fully clear to me how one would do that and what security implications it has. IIRC the Solana team is working on adding type definitions to the SBF perhaps to cover this kind of functionality, so might be something to look into.

- Add InterfaceAccount to the account types(we are checking for Account and AccountLoader).
- Remove full_path and use name instead because clients can derive the name from full_path anyway.
- Use idl-build instead of idl-gen to make it more clear and also parse command is anchor idl parse and the build command is anchor idl build.
- Convert type_args to camelCase in IDL.

LGTM, just be careful with old IDL / new IDL parsing compatibility for the clients and how you want to roll this out. People parsing the new IDL with the old client could run into issues.

The former, IDL build fails because those types don't have __anchor_private_gen_idl_type method. As you know, they implement AnchorSerialize trait but the trait itself doesn't have the IDL print functionality, derive macro version does.

Oh I'm a bit fuzzy on the details here since this was a while ago, but I thought these would be skipped here https://github.com/coral-xyz/anchor/pull/2011/files#diff-b2688db258a9b7f16103c6bd084af5bc13cfd095a651f723a66d29d27acd7f24R656-R658 for derive(Accounts) (code for inserting IDL types generated only for Account and AccountLoader). Now that I think of it, these can in theory also be fields in #[account] structs which is something that I haven't considered. Nice catch!

Sorry I wasn't clear enough, I've asked this because of the tests here

Yea, not only is it not supported by the current IDL, but I've had even anchor build failing in some cases.

wow, great work @kklas ! You put a serious amount of work on this PR. Do you think this could be a potential fix for the issue mentioned here?

@AnderUstarroz yes, it will fix it

@kklas I've fixed the conflicts and included some changes/fixes. Some parts are still rough around the edges such as external types, duplication of accounts/types, larger IDL size and client compatibility. I'll work on these issues in subsequent PRs.
Is there anything you'd like to change before we merge this?

LGTM

Thanks for continuing the effort!


Getting a type error on trying to export armature with animations

Describe the bug
I am creating an FPS and using some arms I found with a model of a crossbow I made. The arms themselves will export just fine, and my crossbow does as well. However, once they are posed and I've put frames into an animation/action, I can't export it.
To Reproduce
Steps to reproduce the behavior:

Go to File -> Export -> gltf 2.0
Click on Export gltf 2.0
See error

Expected behavior
The file exports successfully as a .glb.
Screenshots
Python: Traceback (most recent call last):
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/__init__.py", line 626, in execute
    return gltf2_blender_export.save(context, export_settings)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_export.py", line 46, in save
    json, buffer = __export(export_settings)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_export.py", line 63, in __export
    __gather_gltf(exporter, export_settings)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_export.py", line 75, in __gather_gltf
    active_scene_idx, scenes, animations = gltf2_blender_gather.gather_gltf2(export_settings)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather.py", line 39, in gather_gltf2
    animations += __gather_animations(blender_scene, export_settings)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather.py", line 83, in __gather_animations
    animations_, merged_tracks = gltf2_blender_gather_animations.gather_animations(_blender_object, merged_tracks, len(animations), export_settings)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animations.py", line 76, in gather_animations
    animation = __gather_animation(blender_action, blender_object, export_settings)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animations.py", line 114, in __gather_animation
    channels=__gather_channels(blender_action, blender_object, export_settings),
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animations.py", line 149, in __gather_channels
    return gltf2_blender_gather_animation_channels.gather_animation_channels(
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_cache.py", line 65, in wrapper_cached
    result = func(*args)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animation_channels.py", line 74, in gather_animation_channels
    channel = __gather_animation_channel(
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animation_channels.py", line 202, in __gather_animation_channel
    sampler=__gather_sampler(channels, blender_object, export_settings, bake_bone, bake_channel, bake_range_start, bake_range_end, action_name, driver_obj),
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animation_channels.py", line 254, in __gather_sampler
    return gltf2_blender_gather_animation_samplers.gather_animation_sampler(
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_cache.py", line 65, in wrapper_cached
    result = func(*args)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animation_samplers.py", line 67, in gather_animation_sampler
    input=__gather_input(channels, blender_object_if_armature, non_keyed_values,
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_cache.py", line 65, in wrapper_cached
    result = func(*args)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animation_samplers.py", line 235, in __gather_input
    keyframes = gltf2_blender_gather_animation_sampler_keyframes.gather_keyframes(blender_object_if_armature,
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_cache.py", line 65, in wrapper_cached
    result = func(*args)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animation_sampler_keyframes.py", line 231, in gather_keyframes
    mat = get_bone_matrix(
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_cache.py", line 90, in wrapper_bonecache
    result = func(*args)
  File "/usr/share/blender/2.93/scripts/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_animation_sampler_keyframes.py", line 159, in get_bone_matrix
    bpy.context.scene.frame_set(frame)
TypeError: Scene.frame_set(): error with argument 1, "frame" -  Function.frame expected an int type, not float

location: <unknown location>:-1

.blend file/ .gltf
https://1.filedit.ch/1/VlLYnIJHjXzUJNoXXIkN.blend
Version

OS: Void Linux
Blender Version: 2.93.5

Additional context
Is it possible I have an outdated version of this addon? Version shows as 1.6.16. This is more than likely me not realizing something with an animation lol, but I can't seem to find a solution :/

Yes, version of this addon for 2.93.5 is 1.6.16
I can not reproduce with your test file, export seems to be ok :

Can you please test on another computer and / or with a 3.0 beta version? (https://builder.blender.org/download/daily/)
Thanks!

@julienduroure Odd, it exports just fine on my laptop (Pop_OS). I downloaded 3.0 beta to my desktop (Void) and tried, possibly a different error message but it just crashes the program. When I turn off animation in the export settings, it exports successfully and no crash. I've got a Ryzen 5 1600 CPU and RX 5700 if that means anything for this. Likely something odd on my system but appreciate any ideas or help. Thank you.
# Python backtrace
  60   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_gather_animation_channels.py", line 75 in <listcomp>
  61   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_gather_animation_channels.py", line 75 in gather_animation_channels
  62   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_gather_cache.py", line 65 in wrapper_cached
  63   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_gather_animations.py", line 149 in __gather_channels
  64   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_gather_animations.py", line 114 in __gather_animation
  65   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_gather_animations.py", line 76 in gather_animations
  66   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_gather.py", line 83 in __gather_animations
  67   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_gather.py", line 39 in gather_gltf2
  68   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_export.py", line 77 in __gather_gltf
  69   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_export.py", line 63 in __export
  70   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/blender/exp/
       │ gltf2_blender_export.py", line 46 in save
  71   │   File "/mnt/main/downloads/blender-3.0.0-beta+v30.3e6907eb8a3d-linux.x86_64-release/3.0/scripts/addons/io_scene_gltf2/__init__.py"
       │ , line 637 in execute

@julienduroure alright, I've spent a little playing with this. In the script mentioned in the initial error, I've added the following below this line:
if type(frame) is float:
    frame = int(frame)

and it successfully exports. I'm unsure of why this is necessary for me though. Also, I have tested this with some basic models and it is indeed an issue with armature animations - if there aren't any armatures with animations, the model exports just fine.

@kedodrill What is your Blender's Python version? It should say at the top of the Python Console.

@kedodrill What is your Blender's Python version? It should say at the top of the Python Console.

3.10.0

Ok, that explain it.
The frame variable is indeed a float (it ultimately comes from fcurve.range), while scene.frame_set is documented to take an integer.
If I run C.scene.frame_set(1.0) in my Blender (Python 3.9) I get
<blender_console>:1: DeprecationWarning: an integer is required (got type float).
Implicit conversion to integers using __int__ is deprecated,
and may be removed in a future version of Python.

So I'm guessing this has actually been removed in 3.10.
Since some distros are ahead of the curve on their Python version, this should hopefully be fixed soon (before 3.0) to avoid a repeat of #1295. The quickest fix is just to replace it with frame_set(int(frame)) since the deprecation warning indicates that is what the current code is actually doing.

@scurest @julienduroure thank you both for the help :) I'll leave that fix in my code until 3.0 then

I had the same problem on Fedora 35 with default version of blender 2.93.5 with python 3.10.  i changed to flatpak version 2.93.6 and everything works now


Change "Communities" label

Is your feature request related to a problem? Please describe.
A number of sites would like to change the "Communities" label to something else ('Collections'. 'Projects', ...). Since the label is spread across the package, it's not easy to change without forking the package.
Describe the solution you'd like
It would be great to have a configuration variable that set the "Communities" label. This would have to be added https://github.com/caltechlibrary/invenio-communities/commit/dbfc9abc37b81fceebac2064e7406520b56a9031 as well as a number of other places in the package.
Describe alternatives you've considered
Translations could be used to change the label, but it's not clear how this could be achieved.

At the University of Münster we want to change "communities" to "projects" and would be very interested in such a solution on the package level!
The alternatives are currently (as far as I can see it):

to use template overrides (i. e. copy over a large amount of templates to change very little content), which does not work for strings in React components
using translations to overwrite certain strings. This misuses the translation feature and might have unforeseen consequences. It also requires quite some maintenance to ensure the "translations" are not accidentally overwritten


[Snyk] Upgrade @mui/icons-material from 5.14.0 to 5.14.1

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to upgrade @mui/icons-material from 5.14.0 to 5.14.1.
:information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project.

The recommended version is 1 version ahead of your current version.
The recommended version was released 23 days ago, on 2023-07-19.

Release notes

Package name: @mui/icons-material

5.14.1 - 2023-07-19 Read more

5.14.0 - 2023-07-11 Read more

    from @mui/icons-material GitHub release notes

Commit messages

Package name: @mui/icons-material

f2e5ded v5.14.1 (#38016)
a5779da [blog] Add blog post about support for Next.js App Router (#37929)
e7b5573 [core] Increase space available for sidenav
3c99d8b [website] Add Base UI marketing page (#36622)
1b9c8ab [core] Adds `component` prop to `OverrideProps` type (#35924)
db2968e [Modal] Remove deprecated `BackdropComponent` and `BackdropProps ` from tests (#38018)
5bedd50 [core] Fix rsc build step in CI (#38019)
e4dc06a [docs] Link Material UI from the landing page (#37971)
6b01133 [docs-infra] Change the Diamond Sponsor block positioning on the side nav (#37933)
56cf2e3 [docs-infra] Support backticks in the codeblocks (#37950)
8ef6ff4 [core] Remove nx dependency (#37964)
64d354b [website] Update MUI X landing page (#37966)
1e92db4 [blog] Blog MUI X pro statement removed  (#38015)
093d487 [example] update EmotionCacheProvider to work with GlobalStyles (#37962)
5131b59 [ToggleButton][Joy] Add `ToggleButton` component (#37716)
a748b51 [Skeleton] Add Joy UI `Skeleton` component  (#37893)
0591739 [docs-infra] Improve performance hideToolbar: true (#37969)
eee34a0 [website] Fix a11y issues (#37999)
1f41932 [docs]: Fix the empty /components page (#38010)
4c885c6 [docs-infra] Fix button label on mobile (#37997)
ac9846f [docs-infra] Square drawer corners (#37970)
0fcf1e5 [website] Make the Core page refer to group of products (#37608)
8653517 [Autocomplete] Make touch and click behavior on an option consistent (#37972)
2e67798 Bump @ argos-ci/core to ^0.9.0 (#37988)

Compare

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.
For more information:
🧐 View latest project report
🛠 Adjust upgrade PR settings
🔕 Ignore this dependency or unsubscribe from future upgrade PRs

Approved


Ignore operators that are known to be unsupported

operatorList.addOp adds the arguments to the list which is then
passed as-is by postMessage to the main thread. But since we don't
parse these operations, they are raw PDF objects and may therefore
cause a serialization error.
This is a conservative patch, and only affects operators which are
known to be unsupported. We should ignore all unknown operators,
but I haven't really looked into the consequences of doing that.
Fixes #6549

looks good.
/botio test


Added Duale Hochschule Baden-Würtemberg Karlsruhe, Germany

Added this domain: @student.dhbw-karlsruhe.de (I hope I did it correctly..)
Website: https://www.dhbw-karlsruhe.de/english/general/general/
Long-term it-related courses: https://www.dhbw-karlsruhe.de/inf/ (or in english: https://www.dhbw-karlsruhe.de/english/general/degree-programs/)
(The other DHBW locations were already in this repo)
~ Mike

Duplicated by #1223


[#335] Avoid infinite loop when listening socket is already bound

This change causes the server to terminate and print a message to the terminal when the socket is already bound by another instance. Admins can expect to see the following when this occurs.
Validating configuration file ...
No JSON schema file provided. Using default.

Error: bind: Address already in use [system:98 at /opt/irods-externals/boost1.81.0-1/include/boost/asio/detail/reactive_socket_service.hpp:161:33 in function 'bind']

#'d


load_untyped vs Custom AssetLoaders

The usage of load_untyped seems to prevent the usage of otherwise valid custom assets (AssetLoaders).
The error looks like this, where bevy_asset::server ends up not being able to find the asset loader.
Could not find an asset loader matching: Loader Name: None; Asset Type: None; Extension: None; Path: Some("uipackSpace_sheet.xml");

I am not currently sure if load_untyped is supposed to work with custom assets (bevy example). I have asked in Discord to see if anyone knows.
load_untyped is used in AssetCollection::load as seen here. I've seen the issue with both loading_state and non-loading_state bevy_asset_loader examples.
example expanded macro
// Recursive expansion of AssetCollection macro
// =============================================

#[automatically_derived]
#[allow(unused_variables)]
impl AssetCollection for ImageAssets {
    fn create(
        world: &mut ::bevy::ecs::world::World,
    ) -> Self {
        let from_world_fields = ();
        world.resource_scope(|world,asset_keys: ::bevy::prelude::Mut< ::bevy_asset_loader::dynamic_asset::DynamicAssets> |{
            ImageAssets {
                space_sheet:{
                    let asset_server = world.get_resource:: < ::bevy::asset::AssetServer>().expect("Cannot get AssetServer");
                    asset_server.load("spaceShooter2_spritesheet_2X.xml")
                },
            }
        },)
    }
    fn load(
        world: &mut ::bevy::ecs::world::World,
    ) -> Vec<::bevy::prelude::UntypedHandle> {
        let cell = world.cell();
        let asset_server = cell
            .get_resource::<::bevy::prelude::AssetServer>()
            .expect("Cannot get AssetServer");
        let asset_keys = cell.get_resource:: <bevy_asset_loader::prelude::DynamicAssets>().expect("Cannot get bevy_asset_loader::prelude::DynamicAssets");
        let mut handles = vec![];
        handles.push(
            asset_server
                .load_untyped(
                    "spaceShooter2_spritesheet_2X.xml",
                )
                .untyped(),
        );
        handles
    }
}

Thanks for opening an issue. I am aware of the problem and don't think this is supposed to work. Bevy needs to get the correct AssetLoader, but in this case, we don't give any possible "identification" to use for loader selection (AssetLoader in .meta file, registered extension or asset type).
You are not the first user to stumble into this, but I don't think I can easily fix this in bevy_asset_loader at the moment. The macro cannot know what initial asset type needs to be loaded since, in some cases, it will be different from the asset collection field type (e.g. a StandardMaterial needs to be loaded as an Image asset first).

Feeling a bit silly on this one 😅
AssetLoader::extensions must be a list of extensions without preceding dots. So [".xml"] fails with the above error while ["xml"] succeeds.
It works for me now.

I opened #208 to see if the plugin can support asset loaders without extensions.


how to use this script ?

how to use this script ?

Which script? HopeBilling?

Which script? HopeBilling?

Yes. I understand you need apache2, php7.3?


Add support for IPyaladin widget

Fixes #9292

Codecov Report

Merging #9502 (fbcca12) into main (feb0d28) will increase coverage by 0%.
The diff coverage is n/a.

@@          Coverage Diff          @@
##            main   #9502   +/-   ##
=====================================
  Coverage     73%     73%
=====================================
  Files        228     228
  Lines      10484   10484
  Branches    1536    1536
=====================================
+ Hits        7678    7727   +49
+ Misses      2187    2132   -55
- Partials     619     625    +6

Impacted Files
Coverage Δ

src/platform/common/cancellation.ts
72% <0%> (-6%)
:arrow_down:

...tform/datascience/variablesView/notebookWatcher.ts
94% <0%> (+1%)
:arrow_up:

src/platform/common/platform/fileSystemUtils.ts
94% <0%> (+1%)
:arrow_up:

src/platform/common/process/pythonDaemonPool.ts
73% <0%> (+1%)
:arrow_up:

...orm/datascience/progress/kernelProgressReporter.ts
81% <0%> (+2%)
:arrow_up:

src/platform/common/process/proc.ts
78% <0%> (+2%)
:arrow_up:

...latform/datascience/plotting/plotViewerProvider.ts
41% <0%> (+2%)
:arrow_up:

src/platform/common/featureDeprecationManager.ts
37% <0%> (+3%)
:arrow_up:

...c/platform/datascience/multiplexingDebugService.ts
55% <0%> (+3%)
:arrow_up:

src/platform/common/process/baseDaemon.ts
36% <0%> (+3%)
:arrow_up:

... and 8 more


Using only encoder part for word accentation

Should it be possible to use only transformers encoder part to train word accentation for Lithuanian language. In Lithuanian language stressing is somewhat tricky as it can vary dependyng on context along with word meaning (e.g. grammar case).
You've mentioned in your post using only encoding part for one to one mapping. In case of Lithuanian language accentation, there are three types of accent and the position of the accent within the word (varies alot). And there can also be no accent at all.
Any suggestions?

So is the accent on particular characters? You could define tags at character levels and basically work with a character level Transformer Encoder.

I think you can map the input directly to the unicode character values. The infrastructure around the Tagger classes currently works at a word (+char) level. We'll have to make it more generic to handle character only input (An incentive for me to work on this!).
But the Transformer module is independent of the input (check this file).
13K sentences should be more than enough if you're working at a character level. Do you have tags for each character (including none)?

I do have tags for each char.


Add approval forms to funds just like the other forms are added

Also rename "Approval form" to "Project approval form".

I still don't see the option to add an approval form to a round https://test-apply.hypha.app/admin/pages/939/edit/?next=/admin/funds/round/%3Fform-state%3Dopen%26o%3D-3

Thank you for clarifying! I see the add approval form option in Funds. 👍


[TODO] Update assimp to new version

TODO
Description
Update assimp to the new version, seems to be stable again (?)
https://github.com/assimp/assimp/releases/tag/v5.4.1
Alternative ... drop assimp and add GLTF / OBJ support for a smaller lib

https://github.com/edunad/rawrbox/pull/197


Add a docs page for casdoor-springcloud-gateway-example

Add a page after: https://casdoor.org/docs/integration/spring-security for: https://github.com/casdoor/casdoor-springcloud-gateway-example

@conghuhu


drain cached buffer to immutable buffer

the bl.duplicate() method appears to be costly (about 60% RPS dropoff compared to res.end(string)) - it'd be nice to find a way to evade copying buffers and just dump bytes from memory to the network. Not quite sure how to do this tho
cc/ @mcollina any suggestions on how to efficiently pipe from a buffer?

Don't pipe. Store it as a buffer and call .end(buf). BufferList.duplicate can probably be made a bit faster, too.

Ah yeah, that makes a lot of sense. For this model we expect a stream to always be returned tho; so perhaps it's time to create a from2-buffer package with a constructor method that just reads the buffer and pumps it out to a stream as fast as possible :rocket: - heaps thanks for replying @mcollina :D :D


Ability to treat empty arrays as relations

there is now a flag on decompose, set to true to get empty set behaviour


Restart all services when restarting the agent.

Ensure we restart everything when the agent is scheduled for a restart.
Signed-off-by: Joe Grund<EMAIL_ADDRESS>

Moved into https://github.com/intel-hpdd/iml-agent/pull/12


Attributes keys and values must be string

As explained on this PR : https://github.com/kainxspirits/laravel-pubsub-queue/pull/18
PubSub message documentation : https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage

Hmm didn't see the issue here after your commit, it's fixed as you requested in PR #20


fix: construction of person href with mailto

Fixed construction of  anchor elements for a person to have an href attribute with a value starting with mailto:.
Previously a mailto attribute was created instead of href which is not a standard attribute of an anchor element.

Why close this?

Nevermind, I see there is another pr. 👍


Make it easier for google to ban fingerprints

Won't this repository make it trivial for google to ban every fingerprint in it. They can just download it and ban everything.
The only way we have any chance of using spoofed fingerprints longterm is if people learn how to find them on their own.
The main thing this repository will accomplish is burning a whole bunch of fingerprints all at once.

Hi there, I believe you have a few misunderstandings here. Google knows the relevant build properties of every Android build instance that's ever used Play Services and, if they wanted to, they could block software-backed integrity for any and all spoofed device profiles easily with or without access to this repository. Even if they didn't already have such information, the public repositories of build properties that were the sources of these JSONs are no less easy a target.
At present, they appear to be "banning" only fingerprints that are used excessively enough to stand out like those included as default values in PIF modules. By having far fewer people using the same fingerprints we substantially reduce the risk of them being blocked. Additionally, this collection is a mere 5% of the device profiles acquired from public dumps by my automation scripts over a 48-hour period and is not curated.
The real issue here is that the Play Integrity servers can still differentiate real and spoofed devices. Android instances on real devices with "blocked" fingerprints appear to achieve DEVICE integrity without hardware-backed key attestation and we still don't entirely understand how.
You are correct in that ideally, everyone should be able to pick a device profile from the web, create a PIF-compatible JSON file, and use it for their device. For a number of people this is just not possible; whether that's due to knowledge, time constraints, web filters, or any number of other reasons. If you're able to do this - awesome! I would ask kindly, however, that you don't make rash complaints based on incorrect assumptions when there are those in less privileged positions for whom this collection is a valuable and necessary resource.
You're welcome to read my numerous similar replies where this has been brought up before, such as here, here, and here.

I have randomly tested about 10 fingerprints and only one worked worked so I assume that most are already blocked.

This ratio seems about right. Because the collection is a random subset of collected public device profiles, you can expect that many will not work, which is also true if trying random ones found on the public web. Some of the profiles are too old or too new, are development or unsigned builds, and some may work if you tweak one or two of the properties.
This collection is by design not curated/filtered, though, and doing so might actually tempt Google engineers to crawl and mass-block. There are a handful of fingerprints that worked when this collection was first published and are now blocked, but looking at my monitoring set of tested-working ones from the published collection, 48/50 are still passing as of Tuesday.


Issue of "type resource != float" when trying to get frozen graph_def from a model of saved_model format

1. System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux release 7.6.1810 (Core)
TensorFlow installation (pip package or built from source): pip install tensorflow
TensorFlow library (version, if pip package or github SHA, if built from source): 2.12.0

2. Code
I am trying to use the code of this function to get frozen_graph_def from a TF model in saved_model format: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/python/convert_saved_model.py#L134
No error occurs during the process of conversion. But a bug is reported after I pass the frozen_graph_def to tf.import_graph_def()
import tensorflow_text
import tensorflow as tf
from convert_saved_model import freeze_saved_model
from tensorflow.python.saved_model import signature_constants

saved_model_dir = "./universal-sentence-encoder-multilingual_3"
tag_set = ['serve']
signature_key = signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY
frozen_graph_def, in_tensors, out_tensors, graph = freeze_saved_model(saved_model_dir, input_arrays=None, input_shapes=None,
                       output_arrays=None, tag_set=tag_set, signature_key=signature_key)

tf.import_graph_def(frozen_graph_def, name='')

The model is downloaded from this tfhub link: https://tfhub.dev/google/universal-sentence-encoder-multilingual/3
3. Failure after conversion
If the conversion is successful, but the generated model is wrong, then state what is wrong:

Can't be loaded by using tf.import_graph_def() API.

5. (optional) Any other info / logs
The code is run on Intel CLX Xeon CPU.
The log of this bug:
2023-04-26 22:31:36.666526: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-04-26 22:31:36.668374: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-26 22:31:36.708706: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-26 22:31:36.709089: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-26 22:31:38.113729: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:tensorflow:From /home/zehaohua/tensorflow/tensorflow/lite/python/convert_saved_model.py:42: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.saved_model.load` instead.
2023-04-26 22:31:40.329195: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:353] MLIR V1 optimization pass is not enabled
2023-04-26 22:31:42.536067: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2023-04-26 22:31:42.536209: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
WARNING:tensorflow:From /home/zehaohua/miniconda3/envs/muse/lib/python3.8/site-packages/tensorflow/lite/python/util.py:306: convert_variables_to_constants (from tensorflow.python.framework.convert_to_constants) is deprecated and will be removed in a future version.
Instructions for updating:
This API was designed for TensorFlow v1. See https://www.tensorflow.org/guide/migrate for instructions on how to migrate your code to TensorFlow v2.
WARNING:tensorflow:From /home/zehaohua/miniconda3/envs/muse/lib/python3.8/site-packages/tensorflow/python/framework/convert_to_constants.py:952: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This API was designed for TensorFlow v1. See https://www.tensorflow.org/guide/migrate for instructions on how to migrate your code to TensorFlow v2.
Traceback (most recent call last):
  File "/home/zehaohua/miniconda3/envs/muse/lib/python3.8/site-packages/tensorflow/python/framework/importer.py", line 510, in _import_graph_def_internal
    results = c_api.TF_GraphImportGraphDefWithResults(
tensorflow.python.framework.errors_impl.InvalidArgumentError: input resource[0] expected type resource != float, the type of embeddings_sharded_0[0]
        In {{node EncoderDNN/EmbeddingLookup/EmbeddingLookupUnique/embedding_lookup/Gather}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_muse.py", line 13, in <module>
    tf.import_graph_def(frozen_graph_def, name='')
  File "/home/zehaohua/miniconda3/envs/muse/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py", line 576, in new_func
    return func(*args, **kwargs)
  File "/home/zehaohua/miniconda3/envs/muse/lib/python3.8/site-packages/tensorflow/python/framework/importer.py", line 406, in import_graph_def
    return _import_graph_def_internal(
  File "/home/zehaohua/miniconda3/envs/muse/lib/python3.8/site-packages/tensorflow/python/framework/importer.py", line 515, in _import_graph_def_internal
    raise ValueError(str(e))
ValueError: input resource[0] expected type resource != float, the type of embeddings_sharded_0[0]
        In {{node EncoderDNN/EmbeddingLookup/EmbeddingLookupUnique/embedding_lookup/Gather}}

Hi @zehao-intel
The Value Error occurs  if input_map, or return_elements contains names that do not appear in graph_def, or graph_def is not well-formed (e.g. it refers to an unknown tensor).
Also, the Saved Model serialisation format is only supported and the TFLite conversion from frozen graph is deprecated.
Could you please try from_saved_model if you are trying convert a saved model from TFLite.
Thanks.

Hi pjpratik，
Thanks for your explanation and suggestions. They are very helpful!
It seems that 'from_saved_model' only convert a saved_model to TFLite model.
I would like to get a correct graph_def from the saved_model. Is that possible to achieve this by using TFLite API?

Hi @zehao-intel
As per documentation, the TFLite API supports converting a TensorFlow model to a TensorFlow Lite mode using saved_model format which is recommended by TFLite.
Converting from frozen_graph  is no longer supported in TF 2.x but can be accessed using tf.compat.v1.lite.TFLiteConverter.from_frozen_graph .
Thanks.

Hi @pjpratik ,
I have tried 'from_saved_model' with the following code:
'''
import tensorflow as tf
import tensorflow_text
saved_model_dir = './universal-sentence-encoder-multilingual_3'
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
'''
But an error occurs:

It seems that some binary contents is printed and the execution is stopped at 'converter.convert()'.
Could you please look into it?
Thanks

Hi @zehao-intel
The TensorFlow Lite builtin operator library only supports a limited number of TensorFlow operators, not every model is convertible. To allow conversion, we can enable the usage of certain TensorFlow ops in their TensorFlow Lite model by using following syntax
converter.target_spec.supported_ops = [
  tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
  tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]

Please find the working gist here and let us know if it helps.
Thanks.

Hi @pjpratik ,
Many thanks to your comments.
I have tried your code and inspected the generated model.tflite with netron. I believe the model is correctly converted.
By the way, I made some investigation. It seems that there is no official way to get graph_def from a TFLite model.
In fact, I still need to find out how to get graph_def from the original saved_model. I have tried many ways, including the tf lite code I mentioned before, but none of them works.
Could you give me some suggestions about this?
Thank!

Hi @zehao-intel
Getting a graph def from tflite is not supported. For getting a frozen graph from checkpoints you can try https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py
An example of command-line usage is:
bazel build tensorflow/python/tools:freeze_graph && \
bazel-bin/tensorflow/python/tools/freeze_graph \
--input_graph=some_graph_def.pb \
--input_checkpoint=model.ckpt-8361242 \
--output_graph=/tmp/frozen_graph.pb --output_node_names=softmax

Thanks.

Hi @pjpratik ,
It's happy to hear that there is a stable method to get  a frozen graph from checkpoints.
But for this model, it is in saved_model format.
I saw some arguments related with saved_model from the script you provided.
Is there a similar usage of getting  a frozen graph from saved_model with this script?
Thanks

Hi @zehao-intel
Importing from tensorflow.python or any other modules not supported, and can break unannounced as anything under tf.python.* is private, intended for development only, rather than for public use.
As mentioned earlier, frozen graphs are deprecated and using Saved Model format is recommended.
Is there any challenge to use Saved Model in your use case?
Thanks.

Hi @pjpratik,
Thanks for your explanation. So TensorFlow don't recommend to use frozen graphs in version 2.X.
The task I am handled is to apply 8bit quantization to the TF model mentioned above. And I need to firstly get the frozen graph from this model so that I can modify the model by change the Op type and fuse Op patterns. I am not able to apply this process to a saved_model.
Thanks.

Hi @zehao-intel
Sorry for delayed response.
We can apply 8 bit quantization while converting the SavedModel to TFLite model by adding these flags:
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

Please refer to this documentation on converting using integer only quantization and let us know if it helps.
Thanks.

Hi @pjpratik ，
Sorry for the late reply.
I think this Issue could be closed.
Really appreciate your patient comments!

Thanks for the confirmation.
Closing this issue. Please feel free to reopen if you'd like to work on this further.


A sample for CRUD operations of a virtual CDROM device

It will use physical device if it exists.
It will also attach to an iso that is provided on a datastore.

Thanks Nathan!


match the new signature of rmw_create_node

since https://github.com/ros2/rmw/pull/95/files#diff-137eb23aab0499b25f47cc6e60812dbf the rmw_create_node function has a new namespace_ parameter. This PR adds it to the signature and doesn't use it in the function.

Got it, and I'll make sure it gets handled in the body of the function.


WikipediaParagraph Collection

Add a new Collection class to be used for indexing Wikipedia articles by paragraphs.

I agree with @Peilin-Yang - paragraph-based indexing should be a "primitive" indexing option. I'll start a separate issue to capture discussion.

Let's have the discuss in #312

There hasn't been any action on this issue for a while... closing for now.


PHOENIX-5755 Ensure that Hadoop and HBase dependencies are not overri…

…dden with older versions

other than one nit suggestions, LGTM. +1


Latest release broke the pacman and keys

Describe the issue
Latest release (<IP_ADDRESS>) broke the pacman and keys
To Reproduce

Download Arch_Online.zip
Extract archive
Run "Arch.exe"
Wait for download rootfs.tar.gz
Press "Enter"
Run "Arch.exe" again
Run "sudo pacman-key --init"

Expected behavior
Should be executed without errors
Screenshots

Enviroment:

Windows build number: 10.0.22621.607
Security Software: -
WSL version 1/2: WSL2
ArchWSL version: <IP_ADDRESS>
ArchWSL Installer type: zip, online
Launcher version: 22020900

I confirmed this issue. I'll fix it.
Thank you.

I fixed this issue in new release!


md-raised-button on anchor always have disabled styles

it appears to have disabled="false", which matches these rules
[md-raised-button].md-primary[disabled], [md-raised-button].md-accent[disabled], [md-raised-button].md-warn[disabled], [md-raised-button][disabled][disabled], [md-fab].md-primary[disabled], [md-fab].md-accent[disabled], [md-fab].md-warn[disabled], [md-fab][disabled][disabled], [md-mini-fab].md-primary[disabled], [md-mini-fab].md-accent[disabled], [md-mini-fab].md-warn[disabled], [md-mini-fab][disabled][disabled]

it shouldn't have disabled attribute at all(should be undefined instead of false).
would make plunker, but it appears to be outdated( @DevVersion )

apparently was fixed in https://github.com/angular/material2/commit/716372bc856a71f18c3d27185a0a8bfae82e41fd#diff-a9be5af2302f13e68145b436ee01073aR145
?

Should be fixed already in master, will be released in alpha.11

https://github.com/angular/material2/milestone/12 is this accurate?

@fxck nope, I've been terrible about keeping the milestones up to date.

You don't say. 😄 OK, are there any big features blocking it, or just some polishing to do?

@fxck #1915, #1907, #2026, #2014, and a fix for #1973


Investigate if full scoping is possible

scoping as in recursively looking up variable names in parent fieldsets.

https://github.com/getkirby/kirby/pull/5011

Closing because Kirby PR was rejected.