add custom control button

Hello from newbie
is there anywhere a sample for a custom control button.
i dont know how to handle controls.js
i just want to add a button for SavetoURL and LoadFromUrl and a div for dropImage
thanks

Hello. I'm want to do exact same thing. Have you solved it?

First of all, thank you very much. I'm new to web development, so I'll need to study this code. Would it be possible for me to see your project along with its source code for educational purposes? I've spent three days trying to add this feature, but I couldn't figure it out. I would really appreciate it if you could share your project with me.
My email<EMAIL_ADDRESS>thanks a lot~~


Listener FilterChainMatch using connection metadata

Commit Message: FilterChainMatch using metadata on the connection.
Co-authored-by: Piumi Abeynayaka
Additional Description:
config.listener.v3.FilterChainMatch¶
specifies the match criteria for selecting a specific filter chain for a listener.
The following order applies:

Destination port.
Destination IP address.
Server name (e.g. SNI for TLS protocol),
Transport protocol.
Application protocols (e.g. ALPN for TLS protocol).
Directly connected source IP address (this will only be different from the source IP address when using a listener filter that overrides the source address, such as the Proxy Protocol listener filter).
Source type (e.g. any, local or external network).
Source IP address.
Source port.

This pull request adds connection metadata to this list before the Destination port is matched.
This is somewhat related to https://github.com/envoyproxy/envoy/issues/18856
Risk Level: Low
Testing: Unit Tests have been added to test different scenarios
Docs Changes: Documentation here (https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/listener/v3/listener_components.proto.html?highlight=filterchainmatch) should be enhanced to hilight the new order.
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

/wait
on CI


chore: remove unnecessary test helpers

per https://github.com/apache/datafusion/pull/13260#discussion_r1830873028

thank you @alamb !

I restarted the vendored code CI check as it seems it was due to a network error

🚀


Form Data/File Upload doesn't work

The following code works perfectly fine. It works with the Form[...] wrap as well.
@api.post("/upload", auth=None)
def upload(request, details: PasswordResetRequestIn, file: File[UploadedFile]):
    print(details.email)
    data = file.read()
    return {"name": file.name, "len": len(data)}

However, when the same code is written inside a controller, it doesn't work. I can't find any values in the parameters. The following code gives 422 error with profile_in missing, however, if it is wrapped with Form[...] then it doesn't give any error but it can't find any values passed to the parameter.
@http_patch(response=ProfileOut)
    def update_profile(
        self, request, profile_in: ProfileIn, photo: File[UploadedFile] = None
    ):
...

I have tried using swagger, postman, curl, etc. none of them works if I write the code inside a controller.
Let me know if more details are required.

@maahad767 Can you please share the error log?

Code(controller):
class PasswordResetRequestIn(Schema):
 email: str

@http_patch("upload", auth=None, response=GenericResponse)
 def upload(
     self, request, details: PasswordResetRequestIn, file: File[UploadedFile]
 ):
     print(details.email)
     data = file.read()
     return {"name": file.name, "len": len(data)}

Response:
"PATCH - ProfileAPI[upload] /api/profile/upload" ([{'type': 'missing', 'loc': ('file', 'file'), 'msg': 'Field required'}, {'type': 'missing', 'loc': ('body', 'details'), 'msg': 'Field required'}],)
Unprocessable Entity: /api/profile/upload

Request:

Response:

@maahad767 I have looked into this here is my findings

using a patch http method for file uploads does not work. I tried this example here using PATCH http method and it did work
If want to combine file with other schema, its good to annotate the schema with Form. That way django-ninja would spread the schema fields as an individual form field. For Example

@http_post("/upload", auth=None)
def upload(request, details:Form[PasswordResetRequestIn], file: File[UploadedFile]):
    print(details.email)
    data = file.read()
    return {"name": file.name, "len": len(data)}

got it. thanks @eadwinCode

@maahad767 You are welcome


[fix][client] Make DeadLetterPolicy & KeySharedPolicy serializable

Fixes #23704
Motivation
We use the pulsar client to consume messages from a Pulsar broker in a Storm topology. Recently, we encountered an issue where the deadLetterPolicy that we set on the PulsarSpout seems to be lost when the topology is started. Upon investigation, we found that this is due to the deadLetterPolicy attribute in ConsumerConfigurationData being marked as transient. This prevents us from utilising the dead letter queue feature in our topology.
Modifications
Made DeadLetterPolicy, KeySharedPolicy and Range classes implement the java.io.Serializable interface and removed the transient keyword from the corresponding fields in the ConsumerConfigurationData class. In addition, two tests were also modified / updated.
Verifying this change

[X] Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

Extended existing unit test for the ConsumerConfigurationData class to validate the retention of deadLetterPolicy through serialization & deserialization.

Does this pull request potentially affect one of the following parts:

[ ] Dependencies (add or upgrade a dependency)
[ ] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The threading model
[ ] The binary protocol
[ ] The REST endpoints
[ ] The admin CLI options
[ ] The metrics
[ ] Anything that affects deployment

Documentation

[ ] doc
[ ] doc-required
[X] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: https://github.com/AnuragReddy2000/pulsar/pull/1

oh, that is a unconventional use case I've been never seen. @lhotari Do you think is this change makes sense?

/pulsarbot rerun-failure-checks

@lhotari A test is failing on this PR which did not fail on the PR that raised in my fork. I checked the test, I'm not sure how my changes may cause it to fail. What do you suggest I do here?

@lhotari A test is failing on this PR which did not fail on the PR that raised in my fork. I checked the test, I'm not sure how my changes may cause it to fail. What do you suggest I do here?

@AnuragReddy2000 there are quite a few flaky tests. Reporting ones that haven't already been reported and retrying is the way to handle it.


CB-7215 Add CB service changes to perform database backup/restore

Adds new cloudbreak API methods backupDatabaseByName and restoreDatabaseByName.
These methods trigger new flows that will invoke salt on datalake master node
to run the backup_db.sh and restore_db.sh scripts.
Tested with unit tests and by running cloudbreak locally and making new API
calls.

please don't forget to squash your commits. we usually do 1 commit/pr

please don't forget to squash your commits. we usually do 1 commit/pr

I thought that merging in a PR would automatically squash the commits in that branch into a single commit. That's what I was told when I first starting using Github PRs, in the cloud-infra-services project.

@lacikaaa I've refactored the code to try and reduce code duplication and move shared logic into the parent class, per your review. Please let me know if I've missed anything.

so 2 things:
* you might want to add your flows to `CloudbreakFlowInformation#RESTARTABLE_FLOWS` so if your flows fails they can be retried

Done

* I don't know this for sure but I see that you don't stop hive and ranger during backup/restore operation, and you dump/restore the databases sequentially. This makes it possible that they drift apart, like you're backing up hive, it will be consistent, but in the meanwhile something new is added which is not included in the backup. You create a new policy/whatever in ranger for that. Then you backup ranger. You will have a ranger backup which will point to something in hive which doesn't exist. I'm not so familiar with these services, so I'm not sure this scenario is valid or not.

That's an overarching decision decision for phase one of the the disaster recovery service, and it's not just limited to the ranger and hive database here. This PR isn't meant to handle or resolve that particular case, so we accept it's possible things can get out of sync. That's a longer term issue for the next phase.

Put a DO NOT MERGE flag on while discussing the end-to-end testing with @kkalvagadda1, to make sure it still looked good after the most recent changes. It does, so branch is now okay to merge.
@lacikaaa would you mind merging this in for me?


kgx-generic: init

Motivation for this change
Adds a variant of the kgx terminal emulator with generic branding (called simply "Terminal") to all-packages, so that Hydra picks it up and it gets cached. This is useful for Mobile NixOS demo images and people who want to use kgx as their default terminal on their phone.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

From their README:

Are you a ‘mobile distro’ wanting to ship KGX as your terminal? Great!
We would suggest in such environments you make use of the -Dgeneric=true option. This uses an alternative Desktop Entry and icon similar to that of GNOME Terminal (i.e., a generic terminal) which will be more familar for your users

So it's a suggestion from upstream.

Result of nixpkgs-review pr 121459 at f6e46593 run on x86_64-linux 1

1 package failed to build:

kgx-generic

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement.

https://gist.github.com/b5c1e4bd963d8a9015c49a3cdc108c33

Very curious build failure. Not entirely sure what the non-determinism came from here (note that kgx-enums.h is a Meson-generated file).

Closing since the app has been renamed to GNOME Console and this is no longer needed.


Rekey the SSH key for primary user

Got a little frustrated and re-installed ProxMox, so the keys are no good. Will have to re-key them using Tom Lawrence's guide to YubiKey SSH keys.

Finished adding a new SSH key and verified it's working correctly.


Exception when running/debugging tests

I am actually seeing 2 issues when running test, if I use the test explorer just to run a single test I see this, in the output window
========== Run test finished: 0 run (0:00:01.019) ==========```

And when I try to debug said test, I get the following exception, which my explain the previous problem.

```C#

System.AggregateException was unhandled by user code
  HResult=-2146233088
  Message=One or more errors occurred.
  Source=mscorlib
  StackTrace:
       at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
       at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
       at System.Threading.Tasks.Task`1.get_Result()
       at JsTestAdapter.TestAdapter.TestRunner.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 138
       at JsTestAdapter.TestAdapter.TestRunner.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle) in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 155
       at WEX.TestExecution.ExecutionPlugin.ExecuteTests(IntPtr context, IDictionary`2 rockSteadySettings)
       at WEX.TestExecution.InteropMethods.ExecuteTests(IntPtr context, String[] rockSteadySettings, Int32 rockSteadySettingsSize)
  InnerException:
       ErrorCode=-2147023728
       HResult=-2147023728
       Message=The specified test case was not found in the current result context (mapping test id to test node).
       Source=""
       StackTrace:
            at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
            at WEX.TestExecution.FrameworkHandle.RecordStart(TestCase testCase)
            at JsTestAdapter.TestAdapter.TestRunner.RunTest(TestSourceSettings settings, ITestLogger logger, IRunContext runContext, IFrameworkHandle frameworkHandle, Spec spec) in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 185
            at JsTestAdapter.TestAdapter.TestRunner.<>c__DisplayClassc.<RunTests>b__a(Spec spec) in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 168
            at JsTestAdapter.TestServerClient.DiscoverCommand.<>c__DisplayClass1.<Run>b__0(JObject message) in c:\Git\JsTestAdapter\JsTestAdapter\TestServerClient\DiscoverCommand.cs:line 21
            at JsTestAdapter.JsonServerClient.JsonServerCommand.<>c__DisplayClassb.<RunInternal>b__4(JObject message) in c:\Git\JsTestAdapter\JsTestAdapter\JsonServerClient\JsonServerCommand.cs:line 81
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
            at JsTestAdapter.JsonServerClient.JsonServerCommand.<RunInternal>d__d.MoveNext() in c:\Git\JsTestAdapter\JsTestAdapter\JsonServerClient\JsonServerCommand.cs:line 99
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
            at JsTestAdapter.JsonServerClient.JsonServerCommand.<RunInternal>d__0.MoveNext() in c:\Git\JsTestAdapter\JsTestAdapter\JsonServerClient\JsonServerCommand.cs:line 53
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
            at JsTestAdapter.TestServerClient.DiscoverCommand.<Run>d__3.MoveNext() in c:\Git\JsTestAdapter\JsTestAdapter\TestServerClient\DiscoverCommand.cs:line 21
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
            at JsTestAdapter.TestAdapter.TestRunner.<RunTests>d__e.MoveNext() in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 166
       InnerException:

seems to throw on this line
frameworkHandle.RecordStart(testCase);
 private void RunTest(TestSourceSettings settings, ITestLogger logger, IRunContext runContext, IFrameworkHandle frameworkHandle, Spec spec)
        {
            var testCase = CreateTestCase(settings, spec);
            var outcome = TestOutcome.None;

         frameworkHandle.RecordStart(testCase);
            foreach (var result in spec.Results)
            {
                if (result.Skipped && outcome != TestOutcome.Failed)
                {
                    outcome = TestOutcome.Skipped;
                }

                if (result.Success && outcome == TestOutcome.None)
                {
                    outcome = TestOutcome.Passed;
                }

                if (!result.Success && !result.Skipped)
                {
                    outcome = TestOutcome.Failed;
                }

                frameworkHandle.RecordResult(GetResult(testCase, result, frameworkHandle));
            }
            frameworkHandle.RecordEnd(testCase, outcome);
        }```

I found a similar reported error here, https://github.com/xunit/xunit/issues/492 looks like it could be a VS update 1 related bug?

Has anyone else seen this? Could I be doing something wrong? Is there some things to double check?

I am getting the same exception whenever I attempt to debug any test.
For example if I set a breakpoint on the second instruction here:
it("TEST0", function () {
   expect(true).toBe(true);
});

I get the same exception with this stack trace:
System.AggregateException was unhandled by user code
HResult=-2146233088
Message=One or more errors occurred.
Source=mscorlib
StackTrace:
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification)        at System.Threading.Tasks.Task1.get_Result()
at JsTestAdapter.TestAdapter.TestRunner.RunTests(IEnumerable1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 138        at JsTestAdapter.TestAdapter.TestRunner.RunTests(IEnumerable1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle) in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 155
at WEX.TestExecution.ExecutionPlugin.ExecuteTests(IntPtr context, IDictionary2 rockSteadySettings)        at WEX.TestExecution.InteropMethods.ExecuteTests(IntPtr context, String[] rockSteadySettings, Int32 rockSteadySettingsSize)   InnerException:         ErrorCode=-2147023728        HResult=-2147023728        Message=The specified test case was not found in the current result context (mapping test id to test node).        Source=""        StackTrace:             at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)             at WEX.TestExecution.FrameworkHandle.RecordStart(TestCase testCase)             at JsTestAdapter.TestAdapter.TestRunner.RunTest(TestSourceSettings settings, ITestLogger logger, IRunContext runContext, IFrameworkHandle frameworkHandle, Spec spec) in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 185             at JsTestAdapter.TestAdapter.TestRunner.<>c__DisplayClassc.<RunTests>b__a(Spec spec) in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 168             at JsTestAdapter.TestServerClient.DiscoverCommand.<>c__DisplayClass1.<Run>b__0(JObject message) in c:\Git\JsTestAdapter\JsTestAdapter\TestServerClient\DiscoverCommand.cs:line 21             at JsTestAdapter.JsonServerClient.JsonServerCommand.<>c__DisplayClassb.<RunInternal>b__4(JObject message) in c:\Git\JsTestAdapter\JsTestAdapter\JsonServerClient\JsonServerCommand.cs:line 81          --- End of stack trace from previous location where exception was thrown ---             at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)             at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)             at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at JsTestAdapter.JsonServerClient.JsonServerCommand.d__d.MoveNext() in c:\Git\JsTestAdapter\JsTestAdapter\JsonServerClient\JsonServerCommand.cs:line 99
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at JsTestAdapter.JsonServerClient.JsonServerCommand.d__0.MoveNext() in c:\Git\JsTestAdapter\JsTestAdapter\JsonServerClient\JsonServerCommand.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at JsTestAdapter.TestServerClient.DiscoverCommand.d__3.MoveNext() in c:\Git\JsTestAdapter\JsTestAdapter\TestServerClient\DiscoverCommand.cs:line 21
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at JsTestAdapter.TestAdapter.TestRunner.d__e.MoveNext() in c:\Git\JsTestAdapter\JsTestAdapter\TestAdapter\TestRunner.cs:line 166
InnerException:


HttpBackendMock expectation fails for Uint8Array response

This was broken starting in 1.2.27. The offending commit was 1426b02
-    if (data && !angular.isString(data)) return angular.equals(data, angular.fromJson(d));
+    if (data && !angular.isString(data)) {
+      return angular.equals(angular.fromJson(angular.toJson(data)), angular.fromJson(d));
+    }

My tests and my service use the same function to convert a payload into a U8IntArray:
function strToBinaryArray(str) {
  // This is the evil incantation to convert our string into a binary data array suitable for XHR.send()
  return new Uint8Array(
    Array.prototype.map.call(str, function byteValue(x) {
      return x.charCodeAt(0) & 0xff;
    }));
}

Now when I run my expectation with the expected payload washed through this function, I get this:
ERROR [karma]: Error: Expected POST adb-manager/adbs with different data
EXPECTED:     {"0":13,"1":10,"2":45, <CUT FOR BREVITY>,"length":434,"byteOffset":0,"buffer":{"byteLength":434},"byteLength":434}
GOT:      [object Uint8Array]
at http://localhost:9018/base/vendor/angular-mocks/angular-mocks.js?eb7eef4597d027fb2bbdc81cfbea2a5906c2e0e7:1150

If I revert the change from 1426b02 my spec passes.
This seems like a bug.

Yes, this sounds like a valid bug.

The workaround for this is to have the expectation be a function that returns the binary data. The fact that this workaround works suggests that the intended fix (for Date objects) will not suffice in the case of a function expectation.

Is this still a problem in 1.4.x?
If so then it seems to me that the problem is with the angular.toJson() method, which is not encoding the Uint8Array correctly. Is that right?


release v0.2.2

Can you also push a 0.2.2 tag? @lgarithm

tag must be created from commits from the master branch


See stack trace for ranks > 0 when tests fail

🚀 Feature Request
Right now, if a rank other than rank 0 crashes while running tests, you end up with timeout errors or Connection closed by peer errors without any stack trace from the rank that crashed. It would be very helpful to see logs / stack traces for ranks other than rank 0 in the case of a test failure.

Ah yes. I think we will need to monkeypatch the distributed barriers to use timeout barriers in tests, so they exit without blocking forever.

Closed via #805, which logs the non-rank-zero stdout and stderr as Jenkins artifacts.


add 辽宁传媒学院

add 辽宁传媒学院

@kpsanmao The first verification stage is complete: the official domain for  辽宁传媒学院 is www.lncu.edu.cn, to the best of my knowledge. This is just an informational note. The pull request is still under review. The review may take some more time. I greatly appreciate your patience. Here is the proof of domain ownership: https://www.lncu.edu.cn/gjjyxy/22270.html

Hello, "lncu.edu.cn" and "lncu.cn" both belong to Liaoning Communication University, because of the historical reason, the mailbox system of Liaoning Communication University uses lncu.cn domain name.
Visiting www.lncu.cn is now a 301 jump to www.lncu.edu.cnhttp://www.lncu.edu.cn.
The following documents are proof of domain ownership：
[cid:8203b548-4ce3-4b1c-b656-15abacd2e620]
[cid:81131ade-b182-4f7e-a175-c884c3dae05a]

发件人: Philip Torchinsky @.>
发送时间: 2024年8月6日 20:23
收件人: JetBrains/swot @.>
抄送: kpsanmao @.>; Mention @.>
主题: Re: [JetBrains/swot] add 辽宁传媒学院 (PR #22299)
@kpsanmaohttps://github.com/kpsanmao The first verification stage is complete: the official domain for 辽宁传媒学院 is www.lncu.edu.cnhttps://www.lncu.edu.cn, to the best of my knowledge. This is just an informational note. The pull request is still under review. The review may take some more time. I greatly appreciate your patience. Here is the proof of domain ownership: https://www.lncu.edu.cn/gjjyxy/22270.html
―
Reply to this email directly, view it on GitHubhttps://github.com/JetBrains/swot/pull/22299#issuecomment-2271161739, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEG7ZCHNDALER3IK5OBNLHLZQC52LAVCNFSM6AAAAABL4DKWIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZRGE3DCNZTHE.
You are receiving this because you were mentioned.Message ID: @.***>

Hello, "lncu.edu.cn" and "lncu.cn" both belong to Liaoning Communication University, because of the historical reason, the mailbox system of Liaoning Communication University uses lncu.cn domain name.
Visiting www.lncu.cn is now a 301 jump to www.lncu.edu.cn.
The following documents are proof of domain ownership：

@kpsanmao The first verification stage is complete: the official domain for  辽宁传媒学院 is www.lncu.edu.cn, to the best of my knowledge. This is just an informational note. The pull request is still under review. The review may take some more time. I greatly appreciate your patience. Here is the proof of domain ownership: https://www.lncu.edu.cn/gjjyxy/22270.html

@kpsanmao Thank you for your patience while waiting for a review and processing your request. Pull request merged.
Please start requesting the licenses in about an hour, to let the changes to propagate through our system.

Thank you


fix: prevent from displaying checkout transactions twice in results

:tada: This PR is included in version 3.3.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Help needed: convert image_captioning example into tensorflow lite model

I need some guidance on how to convert this image_captioning model into a re-usable tensorflow lite model so I can use this model in an Android app for image captioning images taken from the camera.
For reference, this is the full Jupyter notebook: https://gist.github.com/shaunaa126/0ad309208ad1eeb96843e231c121e4b1
The script was taken from here: https://github.com/keras-team/keras-io/blob/master/examples/vision/image_captioning.py (https://keras.io/examples/vision/image_captioning/)
I tried to add tf.function Signature to the ImageCaptioningModel class, I want the model's default_serving signature to accept an image as an input and spit out the caption as a string:
 @tf.function(input_signature=[tf.TensorSpec(shape=[299, 299, 3], dtype=tf.float32)])
    def call(self, image):
        vocab = vectorization.get_vocabulary()
        index_lookup = dict(zip(range(len(vocab)), vocab))
        max_decoded_sentence_length = SEQ_LENGTH - 1

        # Read the image from the disk
        sample_img = decode_and_resize(image)
        img = sample_img.numpy().clip(0, 255).astype(np.uint8)

        # Pass the image to the CNN
        img = tf.expand_dims(sample_img, 0)
        img = self.cnn_model(img)

        # Pass the image features to the Transformer encoder
        encoded_img = self.encoder(img, training=False)

        # Generate the caption using the Transformer decoder
        decoded_caption = "<start> "
        for i in range(max_decoded_sentence_length):
            tokenized_caption = vectorization([decoded_caption])[:, :-1]
            mask = tf.math.not_equal(tokenized_caption, tf.constant(0))
            predictions = self.decoder(
                tokenized_caption, encoded_img, training=False, mask=mask
            )
            sampled_token_index = np.argmax(predictions[0, i, :])
            sampled_token = index_lookup[sampled_token_index]
            # if sampled_token == "<end>":
            #     break
            # tf.cond(sampled_token == "<end>", lambda: "test", lambda: "continue")
            decoded_caption += " " + sampled_token

        decoded_caption = decoded_caption.replace("<start> ", "")
        decoded_caption = decoded_caption.replace(" <end>", "").strip()
        return {
            "result": decoded_caption
        }

However when I try to convert the Model from the Keras Model, i get the following error: OperatorNotAllowedInGraphError: Using a symbolic tf.Tensoras a Pythonbool is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.  I am not familiar with tf.function and its limitations, any help is appreciated. Below code is what I used to execute converting the model:
# Keras model converter API uses the default signature automatically.
converter = tf.lite.TFLiteConverter.from_keras_model(caption_model)

converter.target_spec.supported_ops = [
  tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
  tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]

tflite_model = converter.convert()

# Print the signatures from the converted model
interpreter = tf.lite.Interpreter(model_content=tflite_model)

signatures = interpreter.get_signature_list()
print('Signatures:', signatures)

If I can get past the error above, then I can execute the following script to create the Tensorflow lite model:
# Save the model.
with open('model2.tflite', 'wb') as f:
  f.write(tflite_model)

# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="model2.tflite")
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

print('Signature:', interpreter.get_signature_list())
print('Input:', input_details)
print('Output:',output_details)

@AakashKumarNain, Could you please help us with the above issue related to your published tutorial here https://keras.io/examples/vision/image_captioning/

@AakashKumarNain, let me know if you need any more clarification on this issue. Looking forward to your response.

Can you please post the full traceback?

Does this help? https://gist.github.com/shaunaa126/0ad309208ad1eeb96843e231c121e4b1

I will look into it today. Thanks

Hi @AakashKumarNain, were you able to look into this and hopefully find a solution?

Couple of things:

You need to move these two out of the call function if you want to use tf.function

vocab = vectorization.get_vocabulary()
index_lookup = dict(zip(range(len(vocab)), vocab))

You also have to move the loop outside of the scope of tf.function.

Hope that helps!

@AakashKumarNain thanks for the tip. I am not sure how to move these two out of the function as it still results in the same error. If you are saying to completely remove them from the function, how can I return the decoded caption of the image from the function?
Here is my updated code:
    def vocabfunc(self):
        return vectorization.get_vocabulary()

    def index_lookup(self, vocab):
        return dict(zip(range(len(vocab)), vocab))

    @tf.function(input_signature=[tf.TensorSpec(shape=[299, 299, 3], dtype=tf.float32)])
    def call(self, image):
        vocab = self.vocabfunc()
        index_lookup = self.index_lookup(vocab)
        max_decoded_sentence_length = SEQ_LENGTH - 1

        # Read the image from the disk
        sample_img = decode_and_resize(image)
        img = sample_img.numpy().clip(0, 255).astype(np.uint8)

        # Pass the image to the CNN
        img = tf.expand_dims(sample_img, 0)
        img = self.cnn_model(img)

        # Pass the image features to the Transformer encoder
        encoded_img = self.encoder(img, training=False)

        # Generate the caption using the Transformer decoder
        decoded_caption = "<start> "
        for i in range(max_decoded_sentence_length):
            tokenized_caption = vectorization([decoded_caption])[:, :-1]
            mask = tf.math.not_equal(tokenized_caption, tf.constant(0))
            predictions = self.decoder(
                tokenized_caption, encoded_img, training=False, mask=mask
            )
            sampled_token_index = np.argmax(predictions[0, i, :])
            sampled_token = index_lookup[sampled_token_index]
            if sampled_token == "<end>":
                break
            # tf.cond(sampled_token == "<end>", lambda: "test", lambda: "continue")
            decoded_caption += " " + sampled_token

        decoded_caption = decoded_caption.replace("<start> ", "")
        decoded_caption = decoded_caption.replace(" <end>", "").strip()
        return {
            "result": decoded_caption
        }

Error message
OperatorNotAllowedInGraphError            Traceback (most recent call last)
File [~/miniconda/envs/my-proj/lib/python3.11/site-packages/keras/src/engine/training.py:521](https://file+.vscode-resource.vscode-cdn.net/Users/aa849190/Downloads/github/autopilotai/autopilotai-ml/~/miniconda/envs/my-proj/lib/python3.11/site-packages/keras/src/engine/training.py:521), in Model.build(self, input_shape)
    520 try:
--> 521     self.call(x, **kwargs)
    522 except (tf.errors.InvalidArgumentError, TypeError) as e:

File [~/miniconda/envs/my-proj/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py:153](https://file+.vscode-resource.vscode-cdn.net/Users/aa849190/Downloads/github/autopilotai/autopilotai-ml/~/miniconda/envs/my-proj/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py:153), in filter_traceback..error_handler(*args, **kwargs)
    152   filtered_tb = _process_traceback_frames(e.__traceback__)
--> 153   raise e.with_traceback(filtered_tb) from None
    154 finally:

File [~/miniconda/envs/my-proj/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/autograph_util.py:52](https://file+.vscode-resource.vscode-cdn.net/Users/aa849190/Downloads/github/autopilotai/autopilotai-ml/~/miniconda/envs/my-proj/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/autograph_util.py:52), in py_func_from_autograph..autograph_handler(*args, **kwargs)
     51 if hasattr(e, "ag_error_metadata"):
---> 52   raise e.ag_error_metadata.to_exception(e)
     53 else:

OperatorNotAllowedInGraphError: in user code:

    File "[/var/folders/2b/p4gxgkbj4qlg9wgcqphy88sw0000gq/T/ipykernel_9013/4198633898.py](https://file+.vscode-resource.vscode-cdn.net/var/folders/2b/p4gxgkbj4qlg9wgcqphy88sw0000gq/T/ipykernel_9013/4198633898.py)", line 283, in call  *
        vocab = self.vocabfunc()
    File "[/var/folders/2b/p4gxgkbj4qlg9wgcqphy88sw0000gq/T/ipykernel_9013/4198633898.py](https://file+.vscode-resource.vscode-cdn.net/var/folders/2b/p4gxgkbj4qlg9wgcqphy88sw0000gq/T/ipykernel_9013/4198633898.py)", line 276, in vocabfunc  *
        return vectorization.get_vocabulary()
    File "[/Users/aa849190/miniconda/envs/my-proj/lib/python3.11/site-packages/keras/src/layers/preprocessing/text_vectorization.py](https://file+.vscode-resource.vscode-cdn.net/Users/aa849190/miniconda/envs/my-proj/lib/python3.11/site-packages/keras/src/layers/preprocessing/text_vectorization.py)", line 493, in get_vocabulary  **
        return self._lookup_layer.get_vocabulary(include_special_tokens)
...
    File "[/Users/aa849190/miniconda/envs/my-proj/lib/python3.11/site-packages/keras/src/layers/preprocessing/index_lookup.py](https://file+.vscode-resource.vscode-cdn.net/Users/aa849190/miniconda/envs/my-proj/lib/python3.11/site-packages/keras/src/layers/preprocessing/index_lookup.py)", line 382, in get_vocabulary
        if self.lookup_table.size() == 0:

    OperatorNotAllowedInGraphError: Using a symbolic `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.
.

The vocabfunc, and the index_lookup are still part of the tf.function. You need to separate them completely from the call method


metrics: add type and namespace label to k8s event metric

Description:
This change changes the osm_k8s_api_event_count metric to be
a counter vector so that the counters can be grouped as a counter
collection sharing the same description with variable labels.
Part of #902
Signed-off-by: Shashank Ram<EMAIL_ADDRESS>
Affected area:

New Functionality      [ ]
Documentation          [ ]
Install                [ ]
Control Plane          [ ]
CLI Tool               [ ]
Certificate Management [ ]
Networking             [ ]
Metrics                [X]
SMI Policy             [ ]
Security               [ ]
Tests                  [ ]
CI System              [ ]
Performance            [ ]
Other                  [ ]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution?

Codecov Report

Merging #2285 (c1f02d6) into main (047154e) will increase coverage by 0.03%.
The diff coverage is 86.66%.

@@            Coverage Diff             @@
##             main    #2285      +/-   ##
==========================================
+ Coverage   56.85%   56.89%   +0.03%
==========================================
  Files         147      147
  Lines        6596     6602       +6
==========================================
+ Hits         3750     3756       +6
  Misses       2840     2840
  Partials        6        6

Impacted Files
Coverage Δ

pkg/kubernetes/event_handlers.go
64.10% <66.66%> (+0.21%)
:arrow_up:

pkg/metricsstore/metricsstore.go
91.11% <100.00%> (+0.63%)
:arrow_up:

pkg/envoy/route/config.go
96.03% <0.00%> (+0.79%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 047154e...c1f02d6. Read the comment docs.

Codecov Report

Merging #2285 (c1f02d6) into main (047154e) will increase coverage by 0.03%.
The diff coverage is 86.66%.

@@            Coverage Diff             @@
##             main    #2285      +/-   ##
==========================================
+ Coverage   56.85%   56.89%   +0.03%
==========================================
  Files         147      147
  Lines        6596     6602       +6
==========================================
+ Hits         3750     3756       +6
  Misses       2840     2840
  Partials        6        6

Impacted Files
Coverage Δ

pkg/kubernetes/event_handlers.go
64.10% <66.66%> (+0.21%)
:arrow_up:

pkg/metricsstore/metricsstore.go
91.11% <100.00%> (+0.63%)
:arrow_up:

pkg/envoy/route/config.go
96.03% <0.00%> (+0.79%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 047154e...c1f02d6. Read the comment docs.


python310Packages.venstarcolortouch: 0.18 -> 0.19

Description of changes
Update to latest upstream release 0.19

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 194178 run on x86_64-linux 1

3 packages built:

home-assistant-component-tests.venstar
python310Packages.venstarcolortouch
python39Packages.venstarcolortouch


sample of WebSocketEcho does not work

WebSocketEcho does not work，time out reported，but Dark WebSocket Terminal works with the ws url

It works in my home network both the WebSocketEcho and the Dark WebSocket Terminal, i will check the fire wall in my company tomorrow


Make Array to be workable as a queue, ie respect push+shift pattern.

To achieve it, pointer to allocation is preserved, and @buffer is moved
instead of moving elements on #shift.
This is draft for #5126

Note, I don't propose accepting of this patch.
It has obvious negative effect of increasing Array's header.

@funny-falcon what is the status of this PR?

@funny-falcon seems like from that other thread that this is something we should have.
@RX14 I hope you dont mind me bringing you into this PR.  You seemed to be on the issue thread what are your thoughts on this PR?

There is no agreement on whether we want this, so stirring up the conversation on the PR makes little sense.

In case it wasn't clear: perhaps you want to go to the issue that this references, and add your arguments for or against the idea of adding this.

As long as the code works and it's not too complex I don't really mind merging this optimization. Complexity is really the only problem


Automated e2e test Mirror Node API is functioning correctly

Research notes as possible way to test:

https://docs.hedera.com/hedera/tutorials/consensus/query-messages-with-mirror-node
https://github.com/swirlds/infrastructure/blob/4ed91a9cfc4b807135bc2561fda6748bd8b858ef/.github/ISSUE_TEMPLATE/custom-issue-template-upgrade-testnet.md?plain=1#L77

:tada: This issue has been resolved in version 0.25.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


head pose estimation, aflw dataset

Thanks for your work. I have a question about the CNN_Head pose estimation. Before I use the AFLW dataset to train the network, do I need to resize the face picture to 64*64? How to complete it?

Hi @NieHw
Yes in general it is a good practice to crop, resize, and scale the input images.
The AFLW dataset has associated bounding boxes that define the position of each face. Using these coordinates you can crop the face. You have to be careful because some faces are close to the border of the picture and in this case to get a correct crop you have to translate the box in a valid area.
To resize the cropped subframe to 64x64 pixels you can use the OpenCV method called resize.

Thanks for your reply. I will google it. And I want to implement the experiment of your paper as practice. I have some questions about the cross-validation when you used the Prima Head Pose Dataset. Can you please give me your training code? Thank you again.

Hi @mpatacchiola
I got some information in .sqlite file about the bounding boxes, "x, y, h, w" these four coordinates. But I noticed that there were some negative coordinates numbers. And when my crop programer these pictures(negative coordinates), it will creat a damaged picture. Do u know how to avoid it? Thank you very much!

Hi @mpatacchiola
I got some information in .sqlite file about the bounding boxes, "x, y, h, w" these four coordinates. But I noticed that there were some negative coordinates numbers. And when my crop program process these pictures(with negative coordinates), the program will create a damaged picture. Do u know how to avoid it? Thank you very much!

Hi @mpatacchiola
I got some information in .sqlite file about the bounding boxes, "x, y, h, w" these four coordinates. But I noticed that there were some negative coordinates numbers. And when my crop program processed these pictures(with negative coordinates), the program would create a damaged picture. Do u know how to avoid it? Thank you very much!

Hi @mpatacchiola
I used the ‘ex_prima_parser.py' to handle the Prima Head Pose Dataset, and I noticed that you separated the tilt_label and pan_label. When I used tilt_label to train my cnn, after tf.nn.softmax(), it will putout [1,1,1,1....] or [0,0,0,0,....] as predicition. I don't understand it. How do I to train the network?
Help me please, thank you very much. Sorry to bother you.

Hi @NieHw you have to think about head pose estimation not like a classification problem but like a regression problem. You have to use a Sigmoid or Tanh activation funciton in the output layer and use the Squared error loss function to compare your output and the target.
However, the Prima dataset has discrete poses and you can think of dealing with it as a classification problem. In this case you should have an output unit for each class and use a Softmax activation function in the output, and the crossentropy as loss.

Thanks for your patient reply. I just used regression (not softmax) as a output and I get a result close to yours in paper.

Dear NieHw,
Thanks for your help!
Can you share which script you used?
I tried ex_dnn_head_pose_estimation_training.py, but after 5 million steps,
I use the result, dnn_1600i_4h_3o-5000000, to estimate different faces of different poses.
The inputs' poses varies, but the resulting ckpt's estimation always the same.
mpatacchiola said I might stuck at a local minimum.
But currently I am not sure what to do with it.
So hope you can share how you achieve similar result to the paper.
Another point, the paper uses cnn to do regression, but ex_dnn_head_pose_estimation_training
uses fully connected layers.
My doubt about this is whether fully connected layers can learn angles?
I think to learning angles, the input to the network better to 2D?
BR,
Jimmy


Add support for linting Julia files

Possibly using https://github.com/tonyhffong/Lint.jl

Duplicates #627.


window is not defined  - what is correct way to import component inside vite-ssg?

What is correct way to use client side component with vite-ssg?
code: https://stackblitz.com/edit/github-ijnjm5?file=src%2Fpages%2F[lang]%2Fdocs%2F[doc].vue
I used  but still have issue when building with vite-ssg. What's wrong with it?
[vite-ssg] An internal error occurred.
[vite-ssg] Please report an issue, if none already exists: https://github.com/antfu/vite-ssg/issues
(node:7) UnhandledPromiseRejectionWarning: ReferenceError: window is not defined
    at eval (/home/projects/github-ijnjm5/node_modules/pdfjs-dist/legacy/web/pdf_viewer.js:2141:3)
    at new Promise (<anonymous>)
    at Array.Object.defineProperty.value (/home/projects/github-ijnjm5/node_modules/pdfjs-dist/legacy/web/pdf_viewer.js:2140:24)
    at __w_pdfjs_require__ (/home/projects/github-ijnjm5/node_modules/pdfjs-dist/legacy/web/pdf_viewer.js:10060:41)
    at Array.Object.defineProperty.value (/home/projects/github-ijnjm5/node_modules/pdfjs-dist/legacy/web/pdf_viewer.js:1040:17)
    at __w_pdfjs_require__ (/home/projects/github-ijnjm5/node_modules/pdfjs-dist/legacy/web/pdf_viewer.js:10060:41)
    at Array.pdfjsLib (/home/projects/github-ijnjm5/node_modules/pdfjs-dist/legacy/web/pdf_viewer.js:51:25)
    at __w_pdfjs_require__ (/home/projects/github-ijnjm5/node_modules/pdfjs-dist/legacy/web/pdf_viewer.js:10060:41)
    at eval (/home/projects/github-ijnjm5/node_modules/pdfjs-dist/legacy/web/pdf_viewer.js:10192:33)
    at eval (/home/projects/github-ijnjm5/node_modules/pdfjs-dist/legacy/web/pdf_viewer.js:10220:3)
(Use `node --trace-warnings ...` to show where the warning was created)

https://v3.vuejs.org/guide/ssr/universal.html#access-to-platform-specific-apis

https://github.com/antfu/vite-ssg/issues/21#issuecomment-862333958
https://github.com/antfu/vite-ssg/issues/90


Fix thread leak due to race condition

https://github.com/lavalink-devs/lavaplayer/commit/f602c4794f80d01ee1b57326aabd50097f992a86 fixes a thread leak in lavaplayer that appears to be caused by a race condition. I was able to reproduce the issue with some test code, with the leak no longer occurring after that commit. I can provide the test code used if relevant.
Specifically, the leak concerns LocalAudioTrackExecutor, which each control a thread. The audio buffer is leaked as well. This first commit makes the executor no longer reusable, meaning that once stopped a new one must be created, which happens automatically now.
However, I am still observing a similar leak of LocalAudioTrackExecutor when using Lavalink in a real working environment. https://github.com/lavalink-devs/lavaplayer/commit/895b7cda7f65a7a6a186e81b4369f07257ca8a88 was an attempt at eliminating places where the executor could leak. Heap analysis shows that the stop() function of LocalAudioTrackExecutor is not being called in rare cases, leading to a leak. Any help in resolving this second leak would be greatly appreciated. I suspect that these two leaks are separate issues, so merging this PR would likely still reduce the leak to an extent.

LGTM


Does it work for the model produced by nerfstudio?

Currently this code for SA3D only supports the provided nerf model. But in principle SA3D can support different kinds of NeRF.

@Jumpat thank you.
Would you mind showing me where the changes could be applied to the other NeRFs? I just want to ensure that the code is implemented effectively.

@Zanue Thank you for the reply. I would like to know if it is possible to use pre-trained models like nerfacto with SA3D. I have created a model using nerfacto and I want to fine-tune it using SA3D. I am unsure of where to start or if it is currently not possible. Do you have any suggestions? Thank you very much in advance.

Thank you for your interest in our work! Currently SA3D cannot be directly applied to nerfacto, and we are planning to make nerfstudio support SA3D in a few weeks.

Hi
Just wanted to know... if it is integrated in nerfstudio by now. Thanks!

We have been trying, but it appears to be a hard work🤣. Thanks for your patience.

Thanks.

But is it possible to let the SA3D support nerfacto in first?
Just my opinion.

This is exactly what we are working on now!👍

OK, I am really looking forwards to the support of nerfacto.

Hi,
Just wanted to know if it is integrated with nerfstudio already :)
Thanks!

We have released the nerfstudio-version SA3D in the branch nerfstudio-version. Currently it only supports text prompts as input. Feel free to ask questions!


Use OAM to model Knative Serving and Scaling

Xref: https://github.com/oam-dev/rudr/issues/390#issuecomment-547609263
In serverless world, OAM and Knative are perfectly complementary to each other:

Knative works on vertical dimension:

defines abstractions for serverless workload (serving, scaling etc) and map them to K8s resources

OAM/Rudr works on the horizontal dimension:

organizes and manages abstractions for better interoperability and manageability so platform builder could assemble them into a user facing serverless platform.

We will use this issue to tracking the effort of modeling Knative with OAM as soon as OAM spec 0.3.0 is released. It should be straightforward.
/cc @glyn

More precisely, Rudr implements the core workload defined by OAM spec on Kubernetes platform, and also provide necessary utilities for implementing additional traits/components.
The OAM spec defines a set of concepts/rules. The goal is to makes it easier for different components from different parties to compose as an application.

Just drafted a quick sample about Knative serving:
---
apiVersion: core.oam.dev/v1alpha1
kind: WorkloadDefinition
metadata:
  name: service.serving.knative.dev
spec:
  definitionRef:
    # reference to the name of real-world crd
    name: service.serving.knative.dev
---
apiVersion: core.oam.dev/v1alpha3
kind: Component
metadata:
  name: example-service
spec:
  workload:
    # this is exactly a knative serving CR which will be enforced & validated by WorkloadDefinition above
    apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      name: helloworld-go
      namespace: default
    spec:
      template:
        spec:
          containers:
            - image: docker.io/test/helloworld-go
              env:
                - name: TARGET
                  value: "Go Sample v1"
  # (optional) overridable parameters for this workload
  parameters:
  - name: image
    required: true
    fieldPaths:
    - ".spec.containers[0].image[0]"
---
apiVersion: core.oam.dev/v1alpha3
kind: ApplicationConfiguration
metadata:
  name: cool-example
spec:
  components:
  - componentName: example-service
    # (optional) override image parameters to a production image
    parameterValues:
    - name: image
      value: docker.io/prod/helloworld-go
    traits:
    - kind: Logging
      apiVersion: logging.alibaba-apis.com/v1
      metadata:
        name: alibaba-logger
      spec:
        level: debug

The knative scaling will be modeled as trait, this will need some glue code as right now scaling is represented as annotation.

All done! Knative works like a charming in OAM: https://github.com/oam-dev/samples/tree/master/6.Knative_with_Cloud_Resource_App


make pubsub TLS example work properly out-of-the-box

This lets the pubsub + TLS example work. The certificate + key included are self-signed ones for "localhost".

Codecov Report

Merging #804 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #804   +/-   ##
=======================================
  Coverage   60.35%   60.35%
=======================================
  Files          71       71
  Lines       11893    11893
  Branches     1864     1864
=======================================
  Hits         7178     7178
  Misses       4092     4092
  Partials      623      623

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f620b27...62a3bc1. Read the comment docs.

perfect .. no hassles, tests both TLS and non-TLS


Add gauge charts

This commit adds gauge charts to the gchart renderer. Please let me know if you need documentation, the demo project seemed overly complex to set up.

@nkuttler Can you add it in demo project, i.e at http://localhost:8000/gchart/, otherwise it's hard to verify that it works as expected and that the chart renders properly.
With recent README, demo project shouldn't be hard to setup. If you still think it's complex, a PR for that would be welcome too :)

I have added the chart to the demo page and it works for me.
Re complex, I was surprised about the need for mongo, numpy etc. But I think you run the same code on the live site, so it makes sense I guess.

@nkuttler Merged. Thanks!


ValueError when re-compiling function with a multi-dimensional array as a static field

Description
A ValueError occurs when re-compiling a function with a re-instantiated  flax.struct.dataclass with a static and multi-dimensional field . The issue seems related to how static equality is checked for cached compiled functions, as it happens only when setting the pytree_node to False in the dataclass field. The error is triggered on the second iteration of the loop when the jax.jit compiled function is called a second time with a non-scalar jax.Array.
The error message:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This error does not occur when:

The field is set as a scalar (jax.numpy.array(1)).
The pytree_node is set to True.
The parameters are not re-instantiated within the loop.

Minimal Working Example:
import jax
from flax import struct

@struct.dataclass
class Foo:
    # Error has to do with this field when set as static.
    bar: jax.Array = struct.field(pytree_node=False)  # If True, the error will not occur

def step(_params):
    return _params

params = Foo(bar=jax.numpy.arange(4))
for i in range(2):
    print(f"i: {i}")  # Fails on second iteration

    # When we re-instantiate, but with a scalar, the error will not occur
    # params = Foo(bar=jax.numpy.array(1))  # Works, because foo.bar.all() is valid I guess

    # Jit the step function
    jit_step = jax.jit(step).lower(params).compile()
    # ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

    # Only when re-instantiating the dataclass, the error will occur
    params = Foo(bar=jax.numpy.arange(4))

Expected Behavior:
I expect jax to either throw an error on the first iteration, similar to when non-hashable arrays are provided as static args, or not throw an error at all and function correctly.
System info (python version, jaxlib version, accelerator, etc.)
flax:   0.8.5
jax:    0.4.30
jaxlib: 0.4.30
numpy:  1.26.4
python: 3.9.19 (main, Apr  6 2024, 17:57:55)  [GCC 9.4.0]
jax.devices (1 total, 1 local): [cuda(id=0)]
process_count: 1
platform: uname_result(system='Linux', node='r2ci-Alienware-m15-R4', release='5.15.0-122-generic', version='#132~20.04.1-Ubuntu SMP Fri Aug 30 15:50:07 UTC 2024', machine='x86_64')
$ nvidia-smi
Wed Oct  9 12:10:13 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3070 ...    Off | 00000000:01:00.0  On |                  N/A |
| N/A   57C    P0              45W / 125W |    816MiB /  8192MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1600      G   /usr/lib/xorg/Xorg                          139MiB |
|    0   N/A  N/A      1939      G   /usr/lib/xorg/Xorg                          199MiB |
|    0   N/A  N/A      2070      G   /usr/bin/gnome-shell                         99MiB |
|    0   N/A  N/A      6512      G   /usr/lib/firefox/firefox                    140MiB |
|    0   N/A  N/A      6763      G   ...seed-version=20241008-180117.502000       35MiB |
|    0   N/A  N/A      7480      G   ...ures=SpareRendererForSitePerProcess       25MiB |
|    0   N/A  N/A     11398      C   /home/r2ci/rex/.venv/bin/python             144MiB |
+---------------------------------------------------------------------------------------+

Yes, I understand your explanation. My point is more about the user-facing consistency of the behavior. For users who are not familiar with the internal details, it feels inconsistent that the error doesn't occur on the first iteration if static arrays are not allowed. Furthermore, while arrays are apparently not permitted as static fields, scalar arrays (e.g., jax.numpy.array(1)) don't raise any errors, which adds to the confusion.
Finally, the error message itself suggests that a check for .all() in the case of multidimensional arrays might resolve the issue, potentially enabling static array definitions. It seems like there could be room for improvement here to make this behavior more intuitive or consistent for users who might not be aware of these internal mechanisms.

Thanks, that makes sense.
I'm not sure how flax or JAX could do better on this to be honest: attributes marked static are expected to have certain properties, but pre-emptively checking those properties is somewhat costly. For example, one way to do it would be for every time a pytree object is constructed, we go through the elements and check that __eq__ and __hash__ behave as expected. That would lead to the consistent errors you're asking for.
Unfortunately, pytrees are constructed very frequently, so adding such checks would add a lot of overhead to every JAX program. The alternative is to only check the required properties when they're used: this leads to better performance, but as you note is less consistent from the user perspective.
I think the performance piece is critical enough that the current behavior is a better choice – what do you think?

I agree that adding checks to every pytree creation would introduce a significant performance overhead that likely wouldn't justify the benefit.
However, would it be possible to validate static arguments during the cache-hit check instead? Since the traversal is already happening at that point, adding a check there would only impact cases where a previously jitted function is called, which should be less frequent compared to pytree creation. This way, a more descriptive error could be raised—rather than failing with a .all() check—while still allowing scalar arrays to pass through without issue.
Lastly, I was also surprised that fields marked as "static" are still traversed like pytrees. I had expected them to be treated as static leaves and not further traversed. In such cases, could we allow users to override hash and eq for those fields, making them valid for caching?

I don't think the static leaves are being further traversed – what's your evidence of that?

Well, it seemed that way when I placed a breakpoint here and modified the Foo dataclass to contain a nested Bar dataclass. I wanted to check how often the code would hit the breakpoint.
In this updated example, I encountered the breakpoint twice—once for each field (b and c) in Bar—even though only the multi-dimensional array (c) caused the error.
import jax
from flax import struct

@struct.dataclass
class Bar:
    b: jax.Array  # Scalar array
    c: jax.Array  # Multi-dimensional array

@struct.dataclass
class Foo:
    a: Bar = struct.field(pytree_node=False)  # Static field

def step(_params):
    return _params

params = Foo(a=Bar(b=jax.numpy.array(1), c=jax.numpy.arange(4)))
for i in range(2):
    print(f"i: {i}")  # Fails on the second iteration

    # Jit the step function
    jit_step = jax.jit(step).lower(params).compile()
    # ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

    # Re-instantiating the dataclass triggers the error on the second iteration
    params = Foo(a=Bar(b=jax.numpy.array(1), c=jax.numpy.arange(4)))

This suggests that even though pytree_node=False is set for a, the fields within Bar (b and c) are still being traversed, at least during the cache-hit process. This traversal only causes an issue when encountering the multi-dimensional array (c), but both fields are processed.

Thanks for the example – agreed that is strange: I would also expect Bar to be treated as a static variable, and not traversed by the pytree machinery.

I believe the traversing happens because of how dataclasses implement the __eq__ function, where an element-wise comparison occurs, similar to how tuples are checked. This likely gives the impression that the pytree is being traversed, even though it's a result of the equality check.
Regarding the immutability and hashability concerns, I was under the impression that Flax dataclasses are immutable by default and generate a hash function, since frozen=True is set here.
According to the Python documentation on dataclasses, when both eq=True and frozen=True are set (which is the default behavior in Flax), a hash function is automatically generated, even if unsafe_hash=False.
To summarize from the docs:

When eq=True and frozen=True (as in the default Flax dataclass setup), a __hash__() method is generated.
This makes the class instances hashable, ensuring immutability.
If frozen=False, no __hash__() method is generated, making the class unhashable.

However, I'm not entirely sure what is meant by a "meaningful" hash in this context. From what I understand, the hash is based on the __eq__ function, which compares all elements of the dataclass fields. To me, this seems like a reasonable and consistent approach for determining equality and hashability.

Regarding the immutability and hashability concerns, I was under the impression that Flax dataclasses are immutable by default and generate a hash function, since frozen=True is set here.

Ah, I see, thanks. That said, this does not mean it's OK to mark a non-hashable attribute as static. That's the source of your problem: we should probably improve the error here to make that more clear, but the only fix here is for you to not mark arrays as static.

Here's a more stripped-down repro:
import jax
from dataclasses import dataclass

@dataclass
class Foo:
  a: jax.Array
jax.tree_util.register_dataclass(Foo, data_fields=[], meta_fields=['a'])

func = jax.jit(lambda x: x)
foo1 = Foo(jax.numpy.arange(4))
func(foo1)  # no error

foo2 = Foo(jax.numpy.arange(4))
func(foo2)  # error during check of foo1 == foo2

In this case, func(foo1) should error because of non-hashable static attributes, and I'm not sure why that is not happening.

Maybe even more to the point, the issue is that this hash computation should fail, because the tree structure contains an unhashable array.
import jax
from dataclasses import dataclass

@dataclass
class Foo:
  a: jax.Array
jax.tree_util.register_dataclass(Foo, data_fields=[], meta_fields=['a'])

foo = Foo(jax.numpy.arange(4))
foo_struct = jax.tree.structure(foo)
hash(foo_struct)  # should error but doesn't

If it's an expected error, I would have expected it to happen on the first iteration. However, it only happend in the second iteration, which is unexpected.

Chatting with @hawkinsp, it sounds like this is the expected behavior. Hash functions do not necessarily reference all objects in a structure, and so it's not guaranteed that hashing a pytree structure will always identify a problematic value in that structure. But on the second call, __eq__ is called, and this will always fail for arrays that are marked as static.
I'm wondering how to best document this – it's not clear to me. What do you think?

Problem Description
To clarify, it appears that the unhashability of jax arrays is a deliberate design choice. Despite the immutability of jax arrays, which theoretically could allow hashing, hashing large arrays could be computationally expensive and undesirable, especially in scenarios like repeated jit calls. Is this reasoning correct?
The primary issue seems to stem from how the __eq__ method operates for both numpy and jax arrays. When comparing these arrays, __eq__ does not return a single boolean value, but rather an array of booleans. This becomes problematic in the context of flax.struct.dataclass, which evaluates equality by comparing the fields of the class, effectively treating it as a tuple of its fields. Since Python cannot interpret an array of booleans as a single truth value, the equality check raises an error when it encounters an array.
Potential Solution
Could an initial call to the __eq__ method address this issue? For example, calling __eq__ with itself on the first iteration could trigger a consistent failure if any field returns a non-boolean result. This might help surface the error more predictably and ensure it fails early, filtering out unsupported types (like arrays) from being used as fields of a statically marked argument.
Documentation Feedback
In terms of documentation, this behavior is not immediately clear. When using a flax.struct.dataclass, I expected that the dataclass would handle hashing and equality comparisons out of the box. However, it's not obvious that each field within the dataclass must also have a valid equality operator that returns a single boolean value. Not sure if this is something that must be documented in flax, or in jax...

To clarify, it appears that the unhashability of jax arrays is a deliberate design choice. Despite the immutability of jax arrays, which theoretically could allow hashing, hashing large arrays could be computationally expensive and undesirable, especially in scenarios like repeated jit calls. Is this reasoning correct?

Yes – but also hashability in Python requires particular __eq__ semantics that JAX arrays do not follow: rather than arr1 == arr2 returning a boolean, JAX follows NumPy in returning an array of element-wise comparisons. So we must set __hash__ = None because of how we define __eq__ (see the Python __hash__ docs for more).

Could an initial call to the eq method address this issue? For example, calling eq with itself on the first iteration could trigger a consistent failure if any field returns a non-boolean result.

Maybe, but note that just calling __eq__ between a pytree struct and itself would not be enough. For example, this does not error:
import jax
from dataclasses import dataclass

@dataclass
class Foo:
  a: jax.Array
jax.tree_util.register_dataclass(Foo, data_fields=[], meta_fields=['a'])

foo = Foo(jax.numpy.arange(4))
foo_struct = jax.tree.structure(foo)
foo_struct == foo_struct  # True

This is due to the implementation of __eq__ for Python tuples, which won't call __eq__ on their contents if the object IDs match. So it's not a solution here, and even if it were, it's not clear that adding that overhead to every JIT dispatch would be worth the cost.

In terms of documentation, this behavior is not immediately clear. When using a flax.struct.dataclass, I expected that the dataclass would handle hashing and equality comparisons out of the box. However, it's not obvious that each field within the dataclass must also have a valid equality operator that returns a single boolean value. Not sure if this is something that must be documented in flax, or in jax...

The JAX docs are pretty clear that arrays cannot be marked as static. For example, from the docs for jax.tree_util.register_dataclass, which flax.struct.dataclass uses internally:

meta_fields (Sequence[str]) – auxiliary data field names. These fields must contain static, hashable, immutable objects, as these objects are used to generate JIT cache keys. In particular, meta_fields cannot contain jax.Array or numpy.ndarray objects.

Maybe the flax docs should do more to warn users about this?


DRILL-5601: Rollup of external sort fixes and improvements

DRILL-5513: Managed External Sort : OOM error during the merge phase
DRILL-5519: Sort fails to spill and results in an OOM
DRILL-5522: OOM during the merge and spill process of the managed external sort
DRILL-5594: Excessive buffer reallocations during merge phase of external sort
DRILL-5597: Incorrect "bits" vector allocation in nullable vectors allocateNew()
DRILL-5602: Repeated List Vector fails to initialize the offset vector

All of the bugs have to do with handling low-memory conditions, and with correctly estimating the sizes of vectors, even when those vectors come from the spill file or from an exchange. Hence, the changes for all of the above issues are interrelated.
Vector Sizing
Testing revealed that Drill has three different memory layouts for vectors:

Buffer-per-vector as created by the scan operator.
Buffer-per-batch as created by the exchange receiver.
Buffer-per-top-level-vector as created when reading a spill file

As it turns out, each layout produces radically different memory allocations for the same data. In general, the buffer-per-vector is the most compact, followed by the buffer-per-top-vector, with the buffer-per-batch the most variable. The reason is simple: the larger the allocation, the larger the internal fragmentation due to unused average 25% of memory in the power-of-two rounded buffers.
The original record batch sizer computed batch size by summing vectors. This worked for the first case above, but was far off for the other two cases. The estimation error caused the sort to incorrectly predict memory needs and thus triggered OOM errors.
To solve this, the implementation no longer looks at vectors to infer size. Instead, it reaches down through layers of abstraction to the "ledgers" that Drill uses to track memory assignment. In the buffer-per-batch case, computing size by vector ignores the very large 25% of memory that is unused. On the other hand, when sizing by ledger, the sizer learns that all vectors are backed by the same ledger, and asks the ledger for its size.
Since vector sizes are not reliable, the portion of the sizer based on that size was deprecated. Instead, the sizer computes actual data size per column, and computes total size per batch. The previous "gross" vector size (which was the buffer size) was dropped in favor of the "net" size.
Repeated Map Vector Offset Vector
Another memory estimation error occurred with repeated map vectors. Let us first consider non-repeated maps. A "map" in Drill is really a "struct": just a collection of columns that happen to have nested names. That is, Map "m" may have members "a" and "b", referenced as "m.a" and "m.b". But, the vectors for a and b are implemented the same a any top-level vector. As a result, map vectors have no size; they are just containers. The previous "sizer" ignored maps for this reason.
Repeated map vectors are different. Not only are they containers, they are also arrays, and thus have an offset vector. That offset vector must be considered when computing overall batch size. The sizer needs a place to hold the size information. So, the sizer was extended to store a “ColumnSize” instance even for maps. This instance holds the offset vector size for repeated maps.
Vector Allocation
Drill provides (at least) two tools for allocating buffers for vectors. One is the plain “allocateNew()” method which allocates memory for 4096 entries. The “allocateNew(valueCount)” variant takes additional arguments (here, just value count) needed to determine buffer size. For fixed-width vectors, we need only row count. For Varchar vectors, we need row count and average column width. And so on.
If we use the “basic” method, we end up doubling the buffers multiple times. Assume we start with 4096 entries. If the output needs 64K records, we double buffers from 4K to 8K, 16K, 32K and eventually 64K, copying the data every time.
Worse, some vectors allocate very little memory on initial allocation, causing an extreme number of doublings.
Turning on vector logging during sort tests revealed a very large number of doublings. (As it turns out, Project has the same problem, but that is an exercise for later.)
The solution is to use the “allocateNew()” variant that takes the desired data size. We know the record count, but where do we get average Varchar size or average number of entires per repeated column?
The answer is: from the record batch sizer. So, this PR contains code to pull information from the sizer, organize it as needed to allocate vectors, then use the existing methods to do the allocation. The new “SmartAllocationHelper” class performs this task. It is “Smart” in contrast to the existing “AllocationHelper” which assumes column widths and array sizes.
The “SmartAllocationHelper” is organized around hints that can be associated with each column. For efficiency, these hints should be part of the column metadata. But, for now, to prove the concept, the hints are stored in a hash table indexed by column name. If this approach to vector sizing works, the class will evolve to combine schema and size information into a single column description object, perhaps shared with the record batch sizer.
Code changes require passing the “smart allocation helper” into the sort components that do vector allocation. These components are designed to be independently unit tested. To ease testing, components also allow the helper to be null, in which case they continue to allocate vectors the “old fashioned” way: perfectly fine for testing.
Because the allocation helper depends on observed column sizes, the helper is discarded and recreated each time a significant change occurs in incoming batch size.
Data, Expected and Worst-Case Batch Sizes
The sort code contains a “memory manager” that plans sort operator memory use. As discussed above, the sort must handle three distinct batch layouts. This revealed the complexity of dealing with the internal fragmentation that comes from Drill’s power-of-two vector allocations.
As it turns out, we must consider three distinct sizes for the input batch, spill batch and the merge batch:

The data size (which determines size on disk)
The “expected” size (which assumes, on average, 25% internal fragmentation, used when estimating the total size of a collection of batches)
The “worst” case size (which assumes 50% internal fragmentation, used when computing memory needs for the single, large, output batches)

Several tests happened to hit the worst-case size, pointing out the need for the three estimates.
Some rather complex code is needed to convert between the sizes depending on where we start.

If we start with data size (from the record batch sizer, say), we have to expand the size to get the expected and worst-case size.
If we start from a memory budget (space available for a merge batch, say), we have to shrink that memory estimate to get the data estimate.

Since the sort operates based on row counts, to go from a memory budget to a row count:

Determine the memory budget
Choose whether this represents an expected or worst case batch size
Work backward to the data size, assuming internal fragmentation
Divide the batch size by average row width to get the row count

Sometimes we have to take a round trip if we are in tight memory.
Sort Memory Management
Efforts continue to refine the algorithms used to manage memory within the sort.
A problem seen in several tests is that the incoming batch is so large that we can’t keep two in memory as (thought to be) needed for spilling. The “isSpillNeeded” method was changed to long a message if this case occurs as this problem indicates an issue with the scan operator and/or too little memory provided to the sort. That is, it indicates a configuration issue, not a sort bug.
(The two-batch restriction is a bug: see DRILL-5635.)
Low Memory Revisions
Several revisions were made to the code that handles low-memory situations to better reflect the various buffer layouts and worst-case buffer sizes. Basically, if memory is plentiful, spill and merge batch sizes are of fixed sizes to tune performance.
But, if memory is short, we have to reduce spill and output batch sizes to fit the memory available. (Alas, we cannot do anything about input batch sizes.) In this case, we have to work up from available memory to sizes, subject to a variety of constraints such as the need to put at least one record per batch, and so on.
This code is quite messy; it is hard to explain in the abstract other than as “come up with the best memory layout possible given the constraints provided.” See the code comments for details.
Additional logging was added for the various ways we can be short on memory:

Records and/or input batches too large for available sort memory during loading
Records and spill batches too large for available memory during merge

Since these conditions may lead to query failure, detailed logging was added to help diagnose memory-related issues so we can differentiate between those due to bugs and those due to an unhappy collision of low sort memory and large input batches (or rows.)
Revised Merge Phase Planning
The merge phase occurs when spill files exist on disk, buffered batches occur in memory, and we must merge all of these to produce output batches. Output batches should be large due to Drill’s large per-batch overhead. (More rows per batch amortizes the per-batch cost over more rows.)
The existing merge phase calculations omitted certain obscure cases, such as having a single merge file, and anticipating the extra memory needed to conduct a spill. Again, the code is complex, so see the detailed comments.
Sort Spill Batch Size
It finally dawned on the code author that there is no advantage at all to having large spill batches, though there is advantage to having large spill files. Spill batch size was greatly reduced from 8 MB to 1 MB, which helped with tests that torture the sort with very low memory allocations.
Unit Tests
The sort code allows detailed unit tests. These tests were adjusted for changes made above. New cases were added as needed to reproduce additional issues found in system tests.
Vector Size Methods
The “getPayloadByteCount” method was revised to take a value count. It turned out that using the value count from the mutator did not always work for deserialized vectors.
The “getAllocatedByteCount” method was removed; replaced with the “getLedgers” method to estimate actual memory allocations, as described earlier.
Minor Changes
Some minor fixes include:

Consider power-of-two rounding when estimating the size needed for the in-memory sort SV4.
Log more detail for spilled batches.
Pass the “SmartAllocationHelper” into sort components that do vector allocation.
Moved the “EmptyResults” class within its SortImpl parent file to clean up the code a bit.
The sort “cheats” a bit by giving itself a 10% memory allocation margin for rounding errors. Logged this adjustment to aid debugging.
Random code and comment cleanup.
Minor bug in the profile parser, used for tests, was fixed.

Additional Revisions
The code contains an optimization for short queries. If the query has just one batch then the sort need not copy that batch to produce the output. Instead, the sort simply passes along the input batch along with an SV2. This commit fixes some issues with transferring that single, buffered, input batch to the sort’s output container.
While reproducing problems found by QA, it turned out to be useful to be able to parse SQL statements from a file in the context of the test framework. Added test methods for this purpose.
The “AllocationHelper” class turns out to throw an exception if asked to allocate a vector with zero elements. Since a zero-size can occasionally come from the record batch sizer (via the “smart allocation helper”, special code was added to handle this case.
Backed out some debugging code that accidentally appeared in the original PR.
With this set of improvements, the revised, managed sort has become more stable and reliable than the original version. As a result, this commit enables the managed sort by default.

Fixed; slightly different than suggested, but similar. Didn't want to change this code too terribly much, so did the minimum change that would work.

Squashed commits.

Rebased onto master.

+1
LGTM.