Bug with ExtendedHandshakeMessage.MaxRequests

There is problem with ExtendedHandshakeMessage.MaxRequests because when creating ExtendedHandshakeMessage property MaxRequests not filed by value.
And when this message parsed in Mode.HandleExtendedHandshakeMessage property PeerId.MaxSupportedPendingRequests become 1.
Then in Mode.PreLogicTick value PeerId.MaxPendingRequests always will be 2 and this make bad effect on download speed.

That doesn't seem good at all! Do you think you could submit a pull request updating the code and adding a new test?

@carlisib this might be affecting you?

Hello Alan!
Yes. I think this can be the big issue. When I try to download much files.... All be started but only some are download complete.
Obter o Outlook para Androidhttps://aka.ms/ghei36

From: Alan McGovern<EMAIL_ADDRESS>Sent: Sunday, April 12, 2020 1:15:22 PM
To: alanmcgovern/monotorrent<EMAIL_ADDRESS>Cc: carlosib<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [alanmcgovern/monotorrent] Bug with ExtendedHandshakeMessage.MaxRequests (#287)
@carlosibhttps://github.com/carlosib this might be affecting you?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/alanmcgovern/monotorrent/issues/287#issuecomment-612639892, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADQ3SX5CNXCMKGA7DWT3MLLRMHSJVANCNFSM4MGL7UZA.

@Jashik Got that patched and merged! Do let me know if it helps performance for you!
@carlosib This may help performance for you too. This change will make MonoTorrent -> MonoTorrent connections better, which is one area you showed an issue.

@alanmcgovern
Yes. It's helps.
Thx

Hello Alan!
About download speed I still have issue 🙁
Using monotornt app I have about 5mb download speed, but, using utorrent i have 50...60..80 mb download speed.

De: Alan McGovern<EMAIL_ADDRESS>Enviado: segunda-feira, 13 de abril de 2020 08:31
Para: alanmcgovern/monotorrent<EMAIL_ADDRESS>Cc: carlosib<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Assunto: Re: [alanmcgovern/monotorrent] Bug with ExtendedHandshakeMessage.MaxRequests (#287)
@Jashikhttps://github.com/Jashik Got that patched and merged! Do let me know if it helps performance for you!
@carlosibhttps://github.com/carlosib This may help performance for you too. This change will make MonoTorrent -> MonoTorrent connections better, which is one area you showed an issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/alanmcgovern/monotorrent/issues/287#issuecomment-612862806, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADQ3SX54UKK54EZ5Y4Z34M3RMLZX5ANCNFSM4MGL7UZA.


BUG:Get started button in the info section page is invisible on hovering

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
When the "get started" button of the info section (below the home page) is hovered the "get started" text is invisible because the text color is already white and a white background is applied on hovering. This happens in both the modes(light n dark).
Add ScreenShots

What browser are you seeing the problem on?
Firefox, Chrome
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'23 contributor
[X] I want to work on this issue

I would like to work on it. Please assign this to me


Add ReferenceType field to Events

Events are extremely important to FireFly applications, but purposefully don't contain a large amount of information. After speaking with @eberger727 and @awrichar, it was decided adding a new field, ReferenceType, would be useful to help developers determine which tables the events ref refers.

The above events break down into the following categories, which will be the possible values of ReferenceType

TokenTransfer
TransferOperation
Messages
BlockchainEvent
TransactionSubmitted
TokenApproval [0]
TokenApprovalOperations [0]

[0]: Coming in https://github.com/hyperledger/firefly/pull/529

So I realized after discussing this, that the Reference values of the Definition types actually do not refer to a Message. They refer to the created object (a Namespace, Datatype, Token Pool, etc). So there actually are nearly as many reference types as there are operations. The only common ones are that message_confirmed and message_rejected both refer to a Message, and token_transfer_op_failed and token_approval_op_failed both refer to an Operation. Probably not enough commonality to warrant the ReferenceType now that I've thought on it further.
I think it may be more beneficial to introduce a Category in order to group the events by the 4 colors in the graphic above.

I know we also talked about enhancing the enrichEvents method here:
https://github.com/hyperledger/firefly/blob/d03b868663d460f5eba503ee4652f2fced0e6d01/internal/events/event_dispatcher.go#L161
But as I mentioned above, there are actually a lot of different tables to which the Reference field might point. Need to decide if we want a 12-way switch on Type to always fetch the referenced object, or if we pick and choose only certain events to "enrich".

Another angle to dump in here - when creating a websocket subscription, you're able to filter events by a lot of message-specific things, like author, tag, topics. These filters won't be relevant for non-message event types. At the same time, we may want other filters for non-message things, such as filtering blockchain_event events to only certain event names.
We don't have to solve all of this at once, but there's a lot of subtlety to the picture since events can reference so many different types of things.

Also wondering whether we need to even store this in the database, or if it should be derived and just annotated on the event by a switch case after loading...

I'm good with only annotation

Yea after even further thought/discussion, seems like category is not really necessary all the way down in the database - and maybe not even in the API layer at all. The primary driver behind wanting it was to do color-coding on the UI, but if we have a way to aggregate events by type, then the categorization can be done relatively easily all the way up at the UI level. Feels like we've come full circle on this, but probably no changes to the database table needed here.
I think we do definitely need to extend enrichEvents and filterEvents to support processing events that reference something other than Messages. At the very least we should pull in details from Transactions (for transaction_submitted) and Blockchain Events (for blockchain_event).

Going to jot down the work items here:

Define the current web socket events behavior (thanks @nguyer 😄) and ensure blockchain events are included by default
Outline the applicable filters for each event type
Extend filterEvents to support additional filtering based on type.

i.e. filter on type=blockchain_event and blockchain_event.name=<value>, where the latter is currently missing

Enhance enrichEvents to provide additional information for blockchain_events and transactions
Add the ability for the enriched information to be included in API responses via an optional flag

Might want to change the title/summary of this issue since we just made it into a totally different thing 😄

if we have a way to aggregate events by type

btw is this bit being handled elsewhere by @eberger727? I think it's not in scope above.

Here's my first attempt at mapping the event types to the FF construct you can filter on:

For the time being, we're only going to add filters for Messages, Transactions, and BlockchainEvents.


TechDocs: Documentation Header

The header for documentation currently use the standard "home" color.

We would like to change this to a specific color to use for documentation.

Add new gradient to the page theme provider from Figma sketch.
Use new gradient for documentation.
Use the page component with the new page theme type in TechDocsPageWrapper.
Add new documentation header example to Storybook. in Header.stories.tsx

Hi @emmaindal, I would like to take this up.

Great @twishasaraiya 😄 I'll assign it to you, just let us know if you have any questions! 🎉

Hi @emmaindal, can you confirm if I have picked correct shades
{
    colors: ['#C8077A', '#C8077A'],
    waveColor: '#262425',
    opacity: ['0', '0.8'],
  }

Hi @emmaindal, can you please confirm if I have picked correct shades
{
    colors: ['#C8077A', '#C8077A'],
    waveColor: '#262425',
    opacity: ['0', '0.8'],
  }

@twishasaraiya I adjusted them a little bit, will update the sketch. Lets use the following 😄
{
colors: ['#C8077A', '#FF0F9E'],
waveColor: '#ea93c3',
opacity: ['0.8', '0.0'],
},

Thanks for that @emmaindal.  I have made the changes.


Multiple dates do not work property

Describe the bug
Hello,
Even in the documentation, it is not possible to selected multi dates setting a specific time.
To Reproduce
Select multiple dates > choose to select time > save
Only the last "day" will get the time which was selected, the other selected days will get the current time
Expected behavior
Example: I selected 25.08 26.98 and 29.08 and select time for 4 am. It should save all days with 4am, and not only the last day I selected
Screenshots
Documentation:

My Program:

Thank you!

This is the intended behavior, select date > select time for that date, select another date > select time for that date, and so on. Time will update on last selected date.

This is the intended behavior, select date > select time for that date, select another date > select time for that date, and so on. Time will update on last selected date.

Is it somehow possible ?
Thank you!

@makerovski This is the current behaviour


feat(go/adbc/driver): Adbc Driver for Snowflake

Initial work to start creating a snowflake ADBC driver which we can eventually package up like we do for the Flight SQL driver. Currently only GetInfo and GetObjects are implemented, but it's a start!
Will have to add secrets to the repo eventually to allow tests to work.
This relies on https://github.com/snowflakedb/gosnowflake/pull/781 getting merged first

requires https://github.com/snowflakedb/gosnowflake/pull/769 in order to work properly

@lidavidm This is ready for review now, though the CI/tests aren't going to pass until I get the SNOWFLAKE_URI secret added as per the JIRA ticket I filed. In the meantime, the rest of the code can be reviewed and looked at. Though this still relies on some PRs for the gosnowflake lib that haven't gotten merged yet.

@lidavidm My change on the snowflake repo has been merged and I've updated the go.mod here to point to that commit hash. They release on a monthly schedule so the next actual release which will include this change won't be until the end of May, at which point we can update the go.mod to point to that specific version tag.
So this is good to go!

Actually, I should update the install and package files for this shouldn't I? or should that be a separate change?

Actually, I should update the install and package files for this shouldn't I? or should that be a separate change?

Up to you - whatever is easier


[bug] pragma: allowlist nextline secret isn't always respected

After a few experiments, I haven't been able to figure out the logic governing when it does and doesn't work.
Here's an example where the pragma does work:
# pragma: allowlist nextline secret
{"foo": "a16e770f5fe56019"}

Here's an example where it doesn't:
# pragma: allowlist nextline secret
foo: a16e770f5fe56019

detect-secrets will return no results for the first file but report a high entropy hex string in the second, disregarding the pragma.
The normal allowlist secret pragma seems to work in the second case, but not the nextline version.

I have a similar problem where none of the two variants work. With the nextline:
;; pragma: allowlist nextline secret
{:user {:signing {:gpg-key "5AA361214163F2CD"} ;; my keybase key

This bug makes this project almost unusable.  There are many text file formats where one cannot add an inline pragma -- the pragma must be on a separate line.

@ExplodingCabbage @holyjak @jjneely @choman Can you please outline which file types you are seeing this issue in with an example snippet of the failing case. I am actively working on this issue and would like more test cases for verification. Thank you.

I've just been experiencing this issue this morning. The file extension is .cfg (specifically, it is airflow.cfg). The offending line:
result_backend = db+mysql://airflow:airflow@localhost:3306/airflow```

The response on committing, with detect-secrets in the precommit hooks:

```Secret Type: Basic Auth Credentials
Location:    docker/config/airflow.cfg:618```

@petercj1 What version of detect-secrets are you using?

Ah, good question. It is 1.2.0

Thats, interesting. I am running the same version and it seems to be filtering out fine. Can you tell me the output of:
detect-secrets -v scan docker/config/airflow.cfg and
detect-secrets-hook -v scan docker/config/airflow.cfg
Also in your .pre-commit-config.yaml can you post a snippet of your detect-secrets configuration?

Ok couple things I'm noticing.
A) There are no [INFO] debug lines in your first snipped did you forget to paste those? In addition I am seeing the following debug lines in both command outputs:
[scan]	INFO	Checking file: airflow.cfg
[scan]	INFO	Skipping secret due to `detect_secrets.filters.allowlist.is_line_allowlisted`.

B) The first command output verifies that the secret is filtered out? Which is what we want?
C) The second command output verifies that the secret is filtered out?
D) In your pre-commit config are you using v1.2.0? I don't see a specified version. Mine looks like the following:
-   repo: https://github.com/Yelp/detect-secrets
    rev: v1.2.0
    hooks:
    -   id: detect-secrets
        args: [--baseline, .secrets.baseline]
        additional_dependencies: ['gibberish-detector']


Project file reader should check for no-version

Project file reader should check for a package imported with no-version
This is a thing that happens.
e.g. <PackageReference Include="Microsoft.AspNetCore.App" />
This is a very basic fix for #311

Good point. I have specifically put in a test with a version range, and done the TryParse.
I still want a different wording for log message for nothing vs something not parsed.


#263 Adds support for pi_heating_demand on Thermostats

This adds support for detecting heat demand on thermostats exposing pi_heating_demand.
If the feature is available, the OFF state will be added for current CurrentHeatingCoolingState and TargetHeatingCoolingState. Also a monitor will be setup to detect if the thermostat is heating or not.
A threshold (currently 10) is added so thermostat is not shown as heating if demand is below the threshold. This is aimed at making sure that radiators does not appear as heating when they are in fact still cold. (valve may open just a bit even when the room is 1-2 degrees warmer than target temperature)

Not sure if I have checked in the code formatting style, but I did notice that your code is formatted differently.
Also, you have added the target state "Off" as well, but what does it do if the user selects "Off" in the Home.app?

I tried to run the lint target and it didnt complain about the formatting. I have tried to run a format on it now.
Let me know if there is still some formating you would like different.
I am a bit unsure of the target OFF as well. If I do not add it, then it will always appear with a glowing icon in Homekit as if it is enabled. If I add it and it goes OFF, then icons stops glowing - but user will be able to toggle it on and off in the UI with no actual difference on the device.
So to sum it up for states relevant to a room where no heating occurs:

CurrentState.OFF & TargetState.ON = Top left temperature indicator shows idle, device icon shows enabled.
CurrentState.OFF & TargetState.OFF = Top left temperature indicator shows idle, device icon shows disabled.

Thinking a bit more about it.
The correct way must be to always have the TargetState=HEAT. That is the purpose of the device: Keep the heat at a specific degree. I'll revert that part.

There.  Now it only touches CurrentState.

This is no longer needed as running_state has been added to Danfoss Thermostats.


Indentation breakdown by auto formatting

When using functions requires function definition or code block in their option (e.g. setRefClass, test_that in testthat package) inputting { cause breakdown of indentation by auto formatting.

What is broken exactly - the caret appears properly indented into the function scope.

I expected auto formatting only affects current line like
cls <- setRefClass(
	"test",
	methods = list(
		test = function() {}
	)
)

cls <- setRefClass(
	"test",
	methods = list(
		test = function() {

		}
	)
)

For me, affecting indentation other than current line is not intuitive.

It is not quite clear still what is incorrect. { <ENTER>} appear to correctly indented braces and the caret inside the function scope. And yes, automatic formatting does can affect multiple lines - up to the enclosing scope. Example in C#:
            int x;
            if(x<1){
                |

type } at the caret - the line with the if gets formatted.

My intention was that auto formatting affects multiple lines (including lines I manually formatted) was not intuitive for me.
But if auto formatting provides more options to control its behavior, this may not be a problem (e.g. #1736).

OK, we'll consider #1736 separately (maybe also add more options). Formatting multiple lines is otherwise normal.


MATE "Language Support" (Switch Language of Desktop)

It would be nice to be able to switch the language of the desktop environment through MATE "Language Support". This option has not been enabled in GhostBSD's version of MATE.

issue moved from ghostbsd/ghostbsd-src to ghostbsd/issues


[rfc] Remove 'armeabi' and 'arm64-v8a' architectures from Android builds

Significantly reduces Android library build time and output size at a negligible functional cost. Full release AAR size is reduced from 19.9MB to 5.7MB, a size reduction of 71%.
Dropped architectures:

armeabi, the ABI for ARMv6 CPUs, is no longer supported by Android as of 4.4 (according to the compatibility requirements, section 3.3.1: https://source.android.com/compatibility/4.4/android-4.4-cdd.pdf). Our minimum supported Android version is 4.0.3, so it is theoretically possible that an ARMv6 device could install our AAR and fail to load a compatible native library at runtime, but I expect this case to be vanishingly rare. We could consider increasing our minimum Android version to 4.4, but that would exclude ~7% of currently active devices (according to the Android Dashboards: https://developer.android.com/about/dashboards/index.html).
arm64-v8a, the ABI for 64-bit ARM CPUs, has backward compatibility with binaries using the 32-bit armeabi-v7a ABI (according to: https://en.wikipedia.org/wiki/ARM_architecture#ARMv8-A). There are possible performance benefits to running ARM64 binaries on a 64-bit ARM device and I'd like to do some benchmarks to make sure we're not making a dramatic sacrifice on 64-bit devices.

Kept architectures:

armeabi-v7a, the ABI for ARMv7 CPUs, is the target for the vast majority of active Android devices.
x86, the ABI for 32-bit x86 CPUs, is used rarely for mobile devices (mostly niche products like the ZenFone) but is immensely helpful for running in the Android emulator. The x86 emulator is at least 10x faster than the ARM emulator and is strongly recommended by Android Studio.

👍

Please do not increase the min version to 4.4 though, there are still too many people using these versions.
The device catalogue in the Google Play Console shows me that 10699 different devices are currently compatible with my app. Without armeabi, it's only two devices less (and no device of the currently installed 5k).
That is less than 0.02% of devices tangram-es becomes incompatible with and thus dimensions away from the ~7% that would be lost by a hard increase of minSdkVersion to 19.

Also, FYI, about roughly 55% the devices seem to be currently based on arm64-v8a and about 1% on x86.

Thanks @westnordost. Its good to know the stats from streetcomplete. As you mentioned there are fair bit of devices running arm64-v8a, so it might be beneficial for us to keep that architecture.
However as @matteblair mentioned it does make sense to do a performance benchmark to gauge how much of performance improvement we get with 64bit arch. If its significant enough it does make sense to only get rid of armeabi and stick with arm64-v8a.
Will read more on this and update.

Confirmed from multiple documentations (android, arm and intel), that devices having 64bit archs are backward compatible with 32bit ABIs. So with respect to compatibility and device support we should be good here.
With respect to performance and power utilization, documented evidence points to about 5 to 10% performance and power improvement. I will run some tests with tangram on a device running arm64-v8a to get some concrete evidence with arm64-v8a and armeabi-v7a explicitly.
Reference: https://androidbycode.wordpress.com/tag/armeabi-v7a/, https://software.intel.com/en-us/android/articles/how-to-develop-and-evaluate-64-bit-android-apps-on-intel-x86-platforms

@tallytalwar As I have a dependency on SQLCipher which does not have 64 bit support. All the work I've done with Tangram-es has been excluding the arm64-v8a lib and I have not seen any problems with this across a variety of newer devices.
Be very interested in seeing the results of your arm64-v8a vs armeabi-v7a benchmark.

Ok did some basic benchmarking with just scene loading with different house scene files and compared the approx numbers for armeabi-v7a and arm64-v8a.
Scene | CPU (max) % usage |   | Memory (peak) - Mb |   | GPU (max) - ms |
-- | -- | -- | -- | -- | -- | --
  | armeabi-v7a | arm64-v8a |   | armeabi-v7a | arm64-v8a |   | armeabi-v7a | arm64-v8a |
default | 50 | 44 | 12 | 10 | 8.5 | 15 | 67 | 58 | 13.43283582
bubble | 50 | 40 | 20 | 10 | 10 | 0 | 67 | 67 | 0
refill | 45 | 42 | 6.666666667 | 11 | 9.5 | 13.63636364 | 67 | 60 | 10.44776119
walkabout | 40 | 38 | 5 | 10.2 | 9 | 11.76470588 | 67 | 67 | 0
tron | 55 | 50 | 9.090909091 | 11.2 | 10 | 10.71428571 | 67 | 67 | 0
cinnabar | 40 | 38 | 5 | 11.2 | 11 | 1.785714286 | 67 | 67 | 0
zinc | 40 | 38 | 5 | 11.2 | 10.5 | 6.25 | 67 | 67 | 0
  |   |   | 8.965367965 |   |   | 8.450152788 |   |   | 3.411513859
To Summarize on an average wrt just scene loading for tangram using the demo app:
CPU utilization is optimized by about 8%
Memory utilization is optimized by about 8%
GPU render times are optimized negligibly by a factor of 3%
Will do some other benchmarking with scene updates, etc. But this gives a ballpark of optimization we get with using arm64-v8a vs using armeabi-v7a.

Thanks for checking out some benchmarks @tallytalwar! Can you explain more about how you got these numbers? I'm curious because the CPU and memory usage overall look lower than I would expect, and memory use should almost certainly be higher for the 64-bit architecture (due to pointers and struct alignment being larger). Maybe I'm overlooking something?

I wonder, do you consider a speed improvement of ~8% as significant?
My gut opinion would be (still) "yes" as I usually tend to value lower CPU usage always over lower disk drive / memory usage. The latter is just usually available in plenty, the former a very limited resource.
Plus, in any case, if you'd revisit this topic in a few years, the 55% of 64bit users will have grown into a vast majority.

I tested this change on a Galaxy S7, in Release configuration, using the following diff to print a running average of theoretical FPS:
--- a/platforms/android/tangram/src/main/java/com/mapzen/tangram/MapController.java
+++ b/platforms/android/tangram/src/main/java/com/mapzen/tangram/MapController.java
@@ -1254,6 +1254,9 @@ public class MapController implements Renderer {
     private Map<Long, Marker> markers = new HashMap<>();
     private Handler uiThreadHandler;

+    private long frameCounter = 0;
+    private long frameTimeSum = 0;
+
     // GLSurfaceView.Renderer methods
     // ==============================

@@ -1269,11 +1272,20 @@ public class MapController implements Renderer {
             return;
         }

+        long frameStartTime = System.nanoTime();
         boolean viewComplete;
         synchronized(this) {
             viewComplete = nativeUpdate(mapPointer, delta);
             nativeRender(mapPointer);
         }
+        frameTimeSum += System.nanoTime() - frameStartTime;
+
+        frameCounter++;
+        long frameAverageWindow = 60;
+        if (frameCounter % frameAverageWindow == 0) {
+            long fps = frameCounter *<PHONE_NUMBER> / frameTimeSum;
+            android.util.Log.d("Tangram", String.valueOf(fps) + " FPS");
+        }

         if (viewComplete) {

Using a library compiled for armeabi-v7a I recorded averages of roughly 75-85 FPS.
Using a library compiled for arm64-v8a I recorded averages of roughly 75-85 FPS.
Any difference between the two was not significant compared to normal variation in application performance.
Based on these results, I'm in favor of moving forward with this PR as-is.

Sure in my results also apart from cpu and memory improvement the fps rate was pretty consistent.
I also did some more reading on this and probably make sense to re-evaluate this when the state of the industry and adoption of arm64-v8a increases.
http://www.informit.com/articles/article.aspx?p=1843892


On Total Scroll Offset

Added an offset for onTotalScroll, so that it registers the autoscroll even if you are some pixels away from the bottom.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Adding a second contig to Dataflow tests.

Now, we many Dataflow tests have reads, variants, and reference bases on contig "2" as well as "1". I also added a useful utility DoFn: PrintCollection.

The class comment on ReadsPreprocessingPipelineTestData needs updating.

Other than the class comments this looks good.  (Feel free to merge if those are the only changes you decide to make.)
I think we do need to fix the brittleness in the tests with respect to the reference shard and variant shard sizes.  (I saw this was done in some classes, but not all of them yet.)   I think it's fine not to deal with really impractical cases like like shard size -10 or 1, but I think it should be robust to plausible changes.   Could you either do that in this  pull request or create a new ticket.

@davidadamsphd back to you


EIO-702: Create new chart for git-mirror

Closes: https://github.com/rancherlabs/eio/issues/702

[ ] Remove analyze
[ ] Remove logrotate
[ ] Remove metrics

docker run -e GIT_SYNC_REPO="https://github.com/rancher/rancher-charts.git" -it rancher/mirrored-git-sync:v3.5.0 -v 9


Add README.md for C library

Description
Resolves #75
Type of change

[x] documentation

let's use ctest for running tests, not in this PR but we should consider using it at some point

How about Check?


Add minute directive in second interval

This PR fixes #29
It adds the missing minute directive in the second interval's cache_key
It also adds proper testing around second-day interval specs to make sure this doesn't happen again.

Would love any feedback on this!

Been a couple of days, going to merge this as to fix the bug :bug: :)


dev container is unable to build docs in bind mounted directory

Environment

Nautobot version (Docker tag too if applicable): 2.0.0b2
Python version: 3.10.12 (3.8.17 in container)
Database platform, version: N/A
Middleware(s):

Steps to Reproduce

Build docker container with latest version of next (after #4130)
Try to run inv build-and-check-docs

Expected Behavior
Docs are built

Observed Behavior
Permission denied. mkdocs tries to build the docs in $PWD/project-static/docs which is owned by the user on the host system (uid 1000 for me) but is running as uid 999 in the container
My quick workaround was to run build_nautobot_docs and build_example_plugin_docs as the current user:
diff --git a/tasks.py b/tasks.py
index 9dc59ee2d..e60fb0eaa 100644
--- a/tasks.py
+++ b/tasks.py
@@ -536,7 +536,11 @@ def build_and_check_docs(context):
 def build_nautobot_docs(context):
     "Build Nautobot docs."
     command = "mkdocs build --no-directory-urls --strict"
-    run_command(context, command)
+    if is_truthy(context.nautobot.local):
+        run_command(context, command)
+    else:
+        docker_command = f"run --user {os.getuid()}:{os.getgid()} --entrypoint '{command}' nautobot"
+        docker_compose(context, docker_command, pty=True)

 def build_example_plugin_docs(context):
@@ -547,7 +551,7 @@ def build_example_plugin_docs(context):
         print_command(local_command)
         context.run(local_command, pty=True)
     else:
-        docker_command = f"run --workdir='/source/examples/example_plugin' --entrypoint '{command}' nautobot"
+        docker_command = f"run --user {os.getuid()}:{os.getgid()} --workdir='/source/examples/example_plugin' --entrypoint '{command}' nautobot"
         docker_compose(context, docker_command, pty=True)

I'm still getting permission denied when unittest tries to create .coverage so we may need to run unittest as the local user also.

My preferred alternate option is to continue to run as root in the dev stage. Thoughts @glennmatthews?

I'm not necessarily opposed to that option. I was kinda worried that we might hit issues like this with #4130 given the difference in how Docker Desktop works on Mac versus other platforms. Whatever the cleanest/quickest way to fix this turns out to be is fine with me.

I'd like to propose that we stick with root in the dev stage for now. Even with the workaround proposed above, after switching to the develop branch, the docs get created by root so when switching back to next I have to rm -rf examples/example_plugin/example_plugin/static/example_plugin/docs and rm -rf nautobot/project-static/docs to get back to a working state.

Resolved by #4213


How to get CPLEX in Julia 0.6.4

Hello everyone. I need to get CPLEX in my Julia 0.6.4. I've already done everything that was mentioned in the previous issues. I have CPLEX optimizer installed on my computer but when I use "using CPLEX" I get this error:
INFO: Precompiling module CPLEX.
ERROR: LoadError: CPLEX not properly installed. Please run Pkg.build("CPLEX")
Stacktrace:
 [1] error at .\error.jl:21
 [2] include_from_node1 at .\loading.jl:576
 [3] include at .\sysimg.jl:14
while loading C:\Users\ramin\.julia\v0.6\CPLEX\src\CPLEX.jl, in expression starting on line 16
**********************
and then when I run Pkg.build("CPLEX") I get this error:
INFO: Building CPLEX
================================[ ERROR: CPLEX ]================================

LoadError: Unable to locate CPLEX installation. Note this must be downloaded separately. See the CPLEX.jl README for further instructions.
while loading C:\Users\ramin\.julia\v0.6\CPLEX\deps\build.jl, in expression starting on line 69

================================================================================

================================[ BUILD ERRORS ]================================

WARNING: CPLEX had build errors.

 - packages with build errors remain installed in C:\Users\ramin\.julia\v0.6
 - build the package(s) and all dependencies with `Pkg.build("CPLEX")`
 - build a single package by running its `deps/build.jl` script

================================================================================

Does anyone know how I can resolve this issue? I'm using windows 10, Julia 0.6.4, CPLEX 12.8.0 and Jupyter Notebook.
Thank you.

LoadError: Unable to locate CPLEX installation. Note this must be downloaded separately. See the CPLEX.jl README for further instructions.

Please read: https://github.com/JuliaOpt/CPLEX.jl#note-for-windows
Make sure you have an environment variable called CPLEX_STUDIO_BINARIES that points to the correct location. For example:
julia> ENV["CPLEX_STUDIO_BINARIES"] = "path/to/cplex/installation"
julia> Pkg.build("CPLEX")

Thank you Odow, but I'm still getting the same error.

@general6r, please see #168 and #215.
Your code needs to be CPLEX_STUDIO_BINARIES not CPLEX_STUDIO_BINARY. If that doesn't work (which is probably), the environment variable needs to be ENV["CPLEX_STUDIO_BINARIES128"]

@odow Thank you. I read all the previous issues and those two that you suggested, yet for some reason I'm still getting the same error. Help please :(

What is in the directory of that folder?

I guess you mean the path to cplex installation. This is the directory:

Close Julia, then open a new julia REPL and show the output of
Pkg.status("CPLEX")
using Libdl
@show ENV["CPLEX_STUDIO_BINARIES128"]
@show Libdl.dlopen_e("C:/Program Files/IBM/ILOG/CPLEX_Studio_Community128/cplex/bin/x64_win64/cplex1280")

@odow I got another error for Libdl! :(

Oops, don't worry about the Libdl error. Julia behaves differently on 0.6.
I'm not sure what the problem is. Everything looks correct, but you can't open CPLEX correctly (the last line should not end in 0x00000000.
The only other thing I can think of is that you are using the community edition. Do you have a valid license? Can you actually run CPLEX?

@odow Yes I downloaded the free version of CPLEX from the link in CPLEX.jl file. Thank you so much for taking time for me. Is there anything else that I can do to fix it?

Some things you could try

when did you download it? Has the license expired?
Try installing Julia 1.1 and see if CPLEX will install on that.
If you are an academic, you can get an academic license for the full version.

Thank you so much for taking time for me.

No worries.

@odow I downloaded it on Feb 6 and it is valid for 30 days. I downloaded Julia 1.1 and the funny thing is I can't even add packages. For example Pkg.add("JuMP") results in an error saying "Pkg not defined"!!!

Many things changed between Julia 0.6 and Julia 1.0. You now need to import Pkg.
julia> import Pkg

julia> Pkg.add("JuMP")

You will probably run into many other issues. However, if you search https://discourse.julialang.org/, you should be able to find lots of questions that others had when upgrading.
In addition, JuMP recently updated. See https://discourse.julialang.org/t/jump-v0-19-has-been-released/20878

@odow Thank you so much. I finally managed to fix the issue by reinstalling Julia and defining the environment variable.

Great!


Avoid creating intermediate references to packed fields

Such an intermediate reference is UB, even if it is never dereferenced. The previous code that created addresses / raw pointers via intermediate references will become a hard error in a future version of Rust. Use core::ptr::addr_of instead.
This PR increases the MSRV to 1.51.0

bors r+

bors retry


StartPage: fix colors (tooltip, icon-button)

Fixes #165 and #153

Download:
Environment: MINGW=C:\Qt\Tools\mingw530_32, QTDIR=C:\Qt\5.11\mingw53_32, QT=5.11, ARCH=x86
installer/trik-studio-installer-master-x86.exe
trik-studio-master_qt5.11_x86.zip
Environment: MINGW=C:\Qt\Tools\mingw492_32, QTDIR=C:\Qt\5.6\mingw49_32, QT=5.6, ARCH=x86
installer/trik-studio-installer-master-x86.exe
trik-studio-master_qt5.6_x86.zip

Download:
Environment: MINGW=C:\Qt\Tools\mingw492_32, QTDIR=C:\Qt\5.6\mingw49_32, QT=5.6, ARCH=x86
installer/trik-studio-installer-master-x86.exe
trik-studio-master_qt5.6_x86.zip
Environment: MINGW=C:\Qt\Tools\mingw530_32, QTDIR=C:\Qt\5.11\mingw53_32, QT=5.11, ARCH=x86
installer/trik-studio-installer-master-x86.exe
trik-studio-master_qt5.11_x86.zip

Download:
Environment: MINGW=C:\Qt\Tools\mingw492_32, QTDIR=C:\Qt\5.6\mingw49_32, QT=5.6, ARCH=x86
installer/trik-studio-installer-master-x86.exe
trik-studio-master_qt5.6_x86.zip
Environment: MINGW=C:\Qt\Tools\mingw530_32, QTDIR=C:\Qt\5.11\mingw53_32, QT=5.11, ARCH=x86
installer/trik-studio-installer-master-x86.exe
trik-studio-master_qt5.11_x86.zip


The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #745   +/-   ##
=======================================
  Coverage   79.40%   79.40%
=======================================
  Files         104      104
  Lines        9054     9054
=======================================
  Hits         7189     7189
  Misses       1865     1865

Files Changed
Coverage Δ

crates/wasmi/src/module/instantiate/mod.rs
79.74% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Api service

Adds preliminary OrgBook API services, interfaces, and store setup for application.

Looks good! Only comment I have so far is whether we should be able to define the baseVersion parameter in the app configuration, rather than having a hard-coded value. Thoughts on that?

We could do that, how would we resolve different API versions? Right now there's mixed use between v2 and v3.

I am not 100% sure. The API itself has a middleware that can be configured to resolve one version or another depending on which parameters get passed in the request path/headers. I was thinking we could use the version configured in the settings, and override it on an as-needed basis for each service, if we need to. That would go back to having it hardcoded though, so maybe it is just best to leave it in each service for now.


[Google Translate] ... Proxy settings needed

Extension
https://www.raycast.com/gebeto/translate
Description
In some countries, people can have access to google only via VPN. Now I can get access to google translate in browser, but cannot get response via google translate extension in raycast.
So please support proxy settings in such extensions.
Thanks.
Who will benefit from this feature?
Users in countries in where google is banned.
Anything else?
No response

I think if you will go to the Settings -> Advanced, scroll down and enable Use System Network Preferences, it will use system network proxy

I think if you will go to the Settings -> Advanced, scroll down and enable Use System Network Preferences, it will use system network proxy

I enabled it, but it did not work. I can open google translate in browser, but cannot translate in the extension.

I think if you will go to the Settings -> Advanced, scroll down and enable Use System Network Preferences, it will use system network proxy

I enabled it, but it did not work. I can open google translate in browser, but cannot translate in the extension.

i encountered same issue，Google translate is doesn't work

i encountered same issue，Google translate is doesn't work


fix: using npm 9 with lockfileVersion 3

Fix for issue https://github.com/kreuzerk/import-conductor/issues/99

Hi @mtrefzer thx a lot for this contribution. PR looks good.

@all-contributors please add @mtrefzer for code

:tada: This PR is included in version 2.6.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


UX problems

I swear to Yahve, the CIA is bad at UX just like they are bad at deep fakes

How to not put the functions in freaking folders?
I just want a list

Window -> Functions


UnicodeEncodeError: 'charmap' codec can't encode character

I am facing an error while trying to convert a .docx file to xhtml output.
Similar issue was faced in several other (doc/docx/pdf files)
UnicodeEncodeError Description:

Program Code that I used:

`from tika import parser
file_name = input("Enter the file name :")
def extract_html_text(file_name):
parsed_html = parser.from_file(file_name, xmlContent=True)
parsed_html_text = parsed_html['content']
return parsed_html_text
html_text = extract_html_text(file_name)
print(html_text)
`
Also, Can I get help with this:

looks like your Python code that calls Tika is taking in an input file name that includes unicode. You may want to change your code to to:
file_name = codecs.decode(input('Enter the file name:'), "utf-8")

It raises another error:

See this article. You have to read the filename string in UTF-8. I didn't literally mean take the code I gave and paste it, sorry it was pseudo code and I didn't try to run it. Good luck!

Ok, I found another similar way to do that without using codecs module.
Also, Can I get help with this:

This occurs first time(on each new day) I run the code. Due to this there is a delay in getting the output

Thanks @Tushar-Mehndiratta I don't have a fix for the failed to see the startup log message, however you can safely ignore it, as Tika still works fine even if it doesn't see the log output on first run.

But it causes a huge delay (15 seconds- 50 seconds) in executing the whole code

I suppose we could consider putting in an option to not check but overall you need to check for the log to  be written in order  to run the client (aka the server needs to be running after starting). It's never 15 seconds. It's a few seconds. If you have  a better idea and/or sample PR  that illustrates it I'm all ears.

Maybe we could put a time bound, that if the log file is not found within (say) 5 seconds, if not then it could make a new log file... What do you think about this ?

well the danger is not to get the log file to be created via Python - the thing that creates the log file is the starting of the Tika server ... so if that log file isn't created you want it to error out.


imports changed for React Native .4 and above syntax

Imports syntax changed per this document for upgrade to RN v.4 and above: https://github.com/facebook/react-native/releases/tag/v0.40.0

Thanks @andrewnaeve


Zeppelin-272: Remove option part from REST API

This PR remove the option part from the Interpreter setting REST API
[x] Remove option from list settings
[x] Remove option from Create settings and update settings
[X] Update Create setting to return new created setting (jira 261) replacing pr # 258
Remove option from list setting is done by implementing a Exclusion strategy for the gson serialization.
Remove option from the Create\Update REST API is done by passing new InterpreterOption(remote=true) the the Factory and by omitting the InterpreterOption from the NewInterpreterSetting and UpdateInterpreterSetting classes.
REST API tests are managed in another
Update to the documentation will be done in a separate PR

Fixed RAT issue.
Ready for merge

I think these lines also need to be taken care
https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-web/src/app/interpreter/interpreter.controller.js#L67
https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-web/src/app/interpreter/interpreter.controller.js#L172

I will have a look at this as well although the PR is about the REST API
and as much as I tested this controller is not part of it.
בתאריך יום ג׳, 1 בספט׳ 2015, 20:43 מאת Lee moon soo <
<EMAIL_ADDRESS>
I think these lines also need to be taken care
https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-web/src/app/interpreter/interpreter.controller.js#L67
https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-web/src/app/interpreter/interpreter.controller.js#L172
—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-zeppelin/pull/266#issuecomment-136809023
.

merging if there're no more discussions


is it possible to switch AWS_PROFILE/start a new Session completely for UPath pointing to S3?

this is definitely more of a discussion than an issue, and related to s3fs i guess.
i sometimes need to use different profiles to access different buckets or prefixes. is it simple to "reset" UPath so it creates a new Session and picks up different profile? Am i just missing something due to a lack of understanding of fsspec/s3fs?
this is what i had in mind:

# %%
from upath import UPath
import os

# %%
os.environ["AWS_PROFILE"] = "shared1"
test_path = UPath("s3://data1/test.txt")
txt = test_path.read_text()

# %%
# DO SOMETHING?!

# %%
os.environ["AWS_PROFILE"] = "shared2"
test_path1 = UPath("s3://data2/test.txt")
test_path1.write_text(txt)

i can pass the session directly, but would rather "reset"/not have to, e.g. if i need to create multiple Upaths
# %%
from aiobotocore import session as sess
test_path1 = UPath("s3://data2/test.txt", session=sess.AioSession(profile='shared2'))

sorry for the silly question - i'll keep on digging in the mean time.

Hi @DayalStrub
You should be able to achieve what you want, in a few different ways:

aws profiles can be provided via storage options directly, see: https://s3fs.readthedocs.io/en/latest/#credentialsp1 = UPath("s3://bucket/file", profile="shared1")
p2 = UPath("s3://bucket/file", profile="shared2")

filesystem instances are cached depending on your provided storage_options (the keyword arguments in UPath) see: https://github.com/fsspec/filesystem_spec/blob/d969b9601e4aaf5922cb00d82b869bf4b4affd95/fsspec/spec.py#L69-L78
This means that the filesystem instances are identical if (simplifying a bit) their storage_options are identical UPath("s3://bucket/something").fs is UPath("s3://bucket/bla").fs. You can prevent that caching from occuring by using a special keyword argument:UPath("s3://bucket/file", skip_instance_cache=True)

You can clear clear the instance cache of a specific filesystem class manually by running:import fsspec

os.environ["AWS_PROFILE"] = "shared1"
p1 = UPath("s3://bucket/foo")

# clear the instance cache for the fsspec class associated to the "s3" protocol
fsspec.get_filesystem_class(p1.protocol).clear_instance_cache()

os.environ["AWS_PROFILE"] = "shared2"
p2 = UPath("s3://bucket/bar")

Let me know if that helps,
Andreas

super helpful! thanks!
for what i had in mind, where i might use one profile for a while/some paths then switch, i think option 3 works great, and i would maybe make a helper function and pass the protocol directly, eg something like
def set_profile(profile):
    fsspec.get_filesystem_class("s3").clear_instance_cache()
    os.environ["AWS_PROFILE"] = profile

set_profile("shared1")

p1 = UPath("s3://bucket/foo")
p2 = UPath("s3://bucket/bar")

set_profile("shared2")

p3 = UPath("s3://bucket/baz")

i'll close the issue - but do think it would make a great Discussion, given the great answer. guess people can search old tickets too.


:zap: SNBT Parsing Refactor: Boost Performance by 2x-100x!

🎉 Happy Announcement! 🎉
Good news, fellow developers! 🎊 I've just made some magic happen by refactoring the SNBT parser code, and the results are nothing short of spectacular. 🚀
⚡ Performance Boost Alert! ⚡
I've supercharged SNBT performance, with improvements at least 2x to 100x faster! 🔥 Now there's no reason not to choose SNBT for performance reasons!
💡 What’s Changed? 💡

Major refactoring of the old parsing logic.
Optimized loops and better memory handling for smoother performance.
Enhanced speed across the board without compromising the readability and flexibility that you know and love. 🚀

🎨 Pretty Print for SNBT? Absolutely! 🎨
I've also added a pretty-printing feature that supports indentation and line breaks for better readability! 🖌
Now you can format your SNBT data with ease, making it more readable and developer-friendly than ever. Whether you're debugging or simply working with data, this feature will save you time and headaches! 😌

This announcement was brought to you with a little help from GPT, making coding news fun and exciting! 😄

:zap: Performance Test Results
:wrench: Testing Conditions
Types of Serializers Compared

PMMP(BINARY)

The result of using PMMP's BigEndianNbtSerializer to serialize NBT data into binary format.
This method provides highly efficient and compact data representation.

PMMP(BASE64)

The binary result of "PMMP(BINARY)" encoded into a storable string format using base64_encode().
For testing compatibility in environments where binary data cannot be used directly.

SNBT-LEGACY

The output of the original SNBT serializer before optimization efforts.

SNBT-IMPROVE

The output of the newly optimized SNBT serializer after performance improvements.

Types of Data Compared

serialize-time

The time taken to serialize data into the respective format.
Measured in nanoseconds (ns) to evaluate processing speed.

deserialize-time

The time taken to deserialize data back into its original form.
Also measured in nanoseconds (ns), assessing the performance of data restoration.

serialized-size

The size of the serialized data in bytes.
Used to analyze data efficiency and storage requirements.

escaped-size

The size of the serialize data encoded into a storable string format using json_encode().
For testing compatibility in environments where binary data cannot be used directly (JSON or URL, etc...).

📑 Result table

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

0b

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

64b

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

0s

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

16384s

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

0

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

1073741824

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

0l

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

4611686018427387904l

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

0f

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

1234567890.1235f

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

0d

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

1234567890.1235d

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

""

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

"----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------"

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

"\r\n\t\u0001\u0002\u0003\u0004\u0005\u0006\u0007"

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

[B;]

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

[B;45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b,45b]

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

[I;]

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

[I;0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

[]

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

[[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []]

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

[
    [  1b ],
    [ 1s ],
    [ 1 ],
    [ 1l ],
    [ 1f ],
    [ 1d ],
    [ "1" ],
    [ [B;49b] ],
    [ [I;1] ],
    [ [] ],
    [ {} ]
]

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

{}

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

{0: {}, 1: {}, 2: {}, 3: {}, 4: {}, 5: {}, 6: {}, 7: {}, 8: {}, 9: {}, 10: {}, 11: {}, 12: {}, 13: {}, 14: {}, 15: {}, 16: {}, 17: {}, 18: {}, 19: {}, 20: {}, 21: {}, 22: {}, 23: {}, 24: {}, 25: {}, 26: {}, 27: {}, 28: {}, 29: {}, 30: {}, 31: {}, 32: {}, 33: {}, 34: {}, 35: {}, 36: {}, 37: {}, 38: {}, 39: {}, 40: {}, 41: {}, 42: {}, 43: {}, 44: {}, 45: {}, 46: {}, 47: {}, 48: {}, 49: {}, 50: {}, 51: {}, 52: {}, 53: {}, 54: {}, 55: {}, 56: {}, 57: {}, 58: {}, 59: {}, 60: {}, 61: {}, 62: {}, 63: {}, 64: {}, 65: {}, 66: {}, 67: {}, 68: {}, 69: {}, 70: {}, 71: {}, 72: {}, 73: {}, 74: {}, 75: {}, 76: {}, 77: {}, 78: {}, 79: {}, 80: {}, 81: {}, 82: {}, 83: {}, 84: {}, 85: {}, 86: {}, 87: {}, 88: {}, 89: {}, 90: {}, 91: {}, 92: {}, 93: {}, 94: {}, 95: {}, 96: {}, 97: {}, 98: {}, 99: {}, 100: {}, 101: {}, 102: {}, 103: {}, 104: {}, 105: {}, 106: {}, 107: {}, 108: {}, 109: {}, 110: {}, 111: {}, 112: {}, 113: {}, 114: {}, 115: {}, 116: {}, 117: {}, 118: {}, 119: {}, 120: {}, 121: {}, 122: {}, 123: {}, 124: {}, 125: {}, 126: {}, 127: {}}

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

{1: {2: {3: {4: {5: {6: {7: {8: {9: {10: {11: {12: {13: {14: {15: {16: {17: {18: {19: {20: {21: {22: {23: {24: {25: {26: {27: {28: {29: {30: {31: {32: {33: {34: {35: {36: {37: {38: {39: {40: {41: {42: {43: {44: {45: {46: {47: {48: {49: {50: {51: {52: {53: {54: {55: {56: {57: {58: {59: {60: {61: {62: {63: {}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

{
    byte: 1b,
    int: 1,
    short: 1s,
    long: 1l,
    float: 1f,
    double: 1d,
    string: "1",
    "array:byte": [B;49b],
    "array:int": [I;1],
    ":list:": [],
    "\"compound\"": {},
    : {}
}

serialize-time
deserialize-time
serialized-size
escaped-size

PMMP(BINARY)

PMMP(BASE64)

SNBT-LEGACY

SNBT-IMPROVE

{
    Count: 1b,
    Slot: 12b,
    Name: "minecraft:diamond_sword",
    Damage: 0s,
    tag: {
        rarity: "legendary",
        itemType: "weapon",
        stats: {
            attackDamage: 20,
            criticalChance: 15f,
            skill: "Dragon\u2019s Breath"
        },
        generatedBy: "GPT"
    },
    PMMPDataVersion: 1l
}


Clean up a few things

Publishing this package is more involved than it should be. Publishing version 1.2.0 included files from my local system that should not be included in a release.
This PR cleans up a few things:

Add lodash as an explicit dependency, because it is used in lib/mixins/util.js
Restrict published files to index.js and lib/
Lints tests
Add package-lock.json to .gitignore
Bump version to 1.2.1 for immediate publishing to NPM

The full list of files published to NPM in the tarball is now:
npm notice 11.4kB LICENSE
npm notice 915B   lib/mixins/attributable.js
npm notice 1.3kB  lib/mixins/authenticatable.js
npm notice 745B   lib/mixins/deletable.js
npm notice 1.1kB  lib/mixins/editable.js
npm notice 1.1kB  lib/endpoint.js
npm notice 673B   lib/mixins/enumerable.js
npm notice 361B   lib/error.js
npm notice 706B   lib/util/findIndex.js
npm notice 69B    lib/util/http.js
npm notice 51B    lib/util/https.js
npm notice 5.9kB  index.js
npm notice 2.3kB  lib/util/mock.js
npm notice 1.2kB  lib/mixins/paginatable.js
npm notice 738B   lib/mixins/readable.js
npm notice 921B   lib/mixins/repeatable.js
npm notice 1.9kB  lib/request.js
npm notice 637B   lib/mixins/routable.js
npm notice 616B   lib/mixins/schedulable.js
npm notice 2.7kB  lib/mixins/streamable.js
npm notice 572B   lib/mixins/util.js
npm notice 2.0kB  lib/mixins/watchable.js
npm notice 822B   lib/mixins/writable.js
npm notice 1.4kB  package.json
npm notice 3.2kB  README.md

Old package size: 767.3 kB (packed),  4.4 MB (unpacked).
New package size:  12.4 kB (packed), 43.4 kB (unpacked).


Remove Lockfiles Feature Flag and use feature by default

Thanks for raising this, @edgarrmondragon! I've added to the project board.
Any other feature flags we should toggle? Does it make sense to widen this to cover any others as well?

@aaronsteers

Any other feature flags we should toggle? Does it make sense to widen this to cover any others as well?

I think for the moment the other two (Uvicorn and Scheduled jobs in the API) are good to keep, unless @pandemicsyn thinks different 😄

Uvicorn and Scheduled jobs in the API

Thanks, @edgarrmondragon. For my part, I think (?) those should both stay off by default for now.

Uvicorn and Scheduled jobs in the API

Thanks, @edgarrmondragon. For my part, I think (?) those should both stay off by default for now.

+1

@edgarrmondragon @aaronsteers quick clarification: I'm reading this to mean that we want to remove the feature flag entirely along with the checks around the relevant code paths. Is that correct or do we just want to leave the feature flag in place but have it turned on by default so users can still toggle it off?

@aaronsteers @cjohnhanson I'm okay with the full removal approach. I'd initially thought it was just flipping the default, but I think it makes sense. For users that don't want to use lock files we could engage them and see a) why and b) if adding a toggle would make sense.

@aaronsteers  (CC: @edgarrmondragon) Checking in real quick--removing lockfiles continues to cause some issues in testing. There's a few approaches I could take here:

Go through regressions 1 by 1 and figure out what the desired behavior is that they reflect and whether or not that desired behavior has legitimately changed with lockfiles always on. Then update the tests to reflect the new desired behavior. This is what I'm currently doing.
Assume that new behavior with lockfiles is very likely to be the correct behavior and update tests accordingly without doing a full deep dive to grok the behavior the test is expecting. Obviously this introduces some risk but "in theory" if the lockfile tests are a thorough and correct representation of desired lockfile behavior and they are passing, then this should be relatively safe to assume.
Call an audible and instead of doing the "full removal" approach, just change to the "on by default" approach and then toggle lockfiles off for some of these failings tests. This obviously seems bad because what we're testing is then not the default behavior that new users will actually see, but I thought I'd raise it as a potential option anyway.


Edge returned when asked Vertex

Hi,
When asked a vertex, edge functions of the Java API (e.g., getEdge) return a result. This means that the Blueprint implementation of OrientDB is not compliant. For example, the Blueprint documentation states that getEdge(id) should return null if no edge is referenced by that id.
With OrientDB 2.1-rc4, the following very simple test case:
OrientBaseGraph graph = ...;
Vertex v = graph.addVertex("class:TestVertex");
System.out.println(graph.getEdge(v.getId()));
System.out.println(graph.getEdgesOfClass("TestVertex").iterator().next());

displays:
e[#11:0][#-1:-1-TestVertex->#-1:-1]
v(TestVertex)[#11:0]

In addition, the same problem occurs with getVertex.

Could you execute this?
System.out.println(graph.getEdge(v.getId()).getClass());
System.out.println(graph.getEdgesOfClass("TestVertex").iterator().next().getClass());

Hi,
System.out.println(graph.getEdge(v.getId()).getClass()); displays class com.tinkerpop.blueprints.impls.orient.OrientEdge. However, the second line throws a ClassCastException: com.tinkerpop.blueprints.impls.orient.OrientVertex cannot be cast to com.tinkerpop.blueprints.Edge
I runned the second line step by step. In the next() method of OrientElementIterator, the isEdgeType() test returns false:
if (immutableClass.isEdgeType())
    currentElement = new OrientEdge(graph, currentDocument);
else
    currentElement = new OrientVertex(graph, currentDocument);
...
return (T) currentElement; // fails because currentElement is an OrientVertex and T is OrientEdge

Hope this helps.

Hi,
Any news regarding this issue?

Found the reason: This code:
System.out.println(graph.getEdgesOfClass("TestVertex").iterator().next());

Should return the edges of class TestVertex, but such class has been used for Vertices, not Edges. So this usage is not correct, but the error handling is misleading. Fixing the error...

Ideally, getEdgesOfClass(class) should return an empty iterator or null if class is a vertex type. Nevertheless, this is a matter of personal preference/implementation effort and a clear error message is fine, if documented.
However, according to the Blueprint reference implementation (TinkerGraph), the getEdge usage is correct and must return null:
Vertex v1 = graph1.addVertex("class:TestVertex");
System.out.println("ODB: " + graph1.getEdge(v1.getId()));
Vertex v2 = graph2.addVertex(1);
System.out.println("BP: " + graph2.getEdge(v2.getId()));

ODB: e[#11:0][#-1:-1-TestVertex->#-1:-1]
BP: null

The problem is the same if I use getVertex with an edge id.

Fixed and added some new test cases in GraphPolymorphicTest.java


Global/national usage stats don't update correctly when filtering initial search results

Steps to reproduce:

go to caniuse.com and type in "optional"
wait for the list to filter (the top result should be :optional CSS pseudo-class)
note the global/national usage stats in the upper right corner
type in " ch" (so the complete search is "optional ch" and the top result should be JavaScript operator: Optional chaining operator (?.))
check the global/national usage stats again... they should be the same as the stats from step 3, even though the feature is different
refresh the page... the global/national usage stats should now be correct

Yikes, that's pretty bad. Just pushed a fix, should be working as expected now. Thanks so much for reporting!

No problem, glad to help!


feat: Implement bloom_filter_agg

Which issue does this PR close?

Closes #846.
Rationale for this change

What changes are included in this PR?

Native implementation (bloom_filter_agg.rs) that uses DataFusion's Accumulator trait. We do not have a GroupsAccumulator implementation and leave it as a possible future optimization.
Serde logic (planner.rs, QueryPlanSerde.scala)
Serialization and merging logic for underlying data structures (spark_bloom_filter.rs, spark_bit_array.rs)

How are these changes tested?

New test in CometExecSuite
Spark tests in CI exercise this aggregation
Scala benchmark to compare against Spark code path
Native benchmark for partial and final aggregation modes
Native tests for new bit array logic spark_bit_array.rs

Codecov Report
Attention: Patch coverage is 76.47059% with 4 lines in your changes missing coverage. Please review.

Project coverage is 33.98%. Comparing base (c3023c5) to head (bf22902).
Report is 19 commits behind head on main.

Files with missing lines
Patch %
Lines

.../scala/org/apache/comet/serde/QueryPlanSerde.scala
76.47%
2 Missing and 2 partials :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##               main     #987      +/-   ##
============================================
- Coverage     34.03%   33.98%   -0.06%
+ Complexity      875      857      -18
============================================
  Files           112      112
  Lines         43289    43369      +80
  Branches       9572     9623      +51
============================================
+ Hits          14734    14738       +4
- Misses        25521    25567      +46
- Partials       3034     3064      +30

Flag
Coverage Δ

?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Results from the benchmark I just added:
BloomFilterAggregate Exec (cardinality 100):       Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
---------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (BloomFilterAgg)                         117            136          18         89.4          11.2       1.0X
SQL Parquet - Comet (Scan) (BloomFilterAgg)                  117            134          18         89.4          11.2       1.0X
SQL Parquet - Comet (Scan, Exec) (BloomFilterAgg)             71             78           9        148.3           6.7       1.7X

BloomFilterAggregate Exec (cardinality 1024):      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
---------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (BloomFilterAgg)                         111            128          11         94.7          10.6       1.0X
SQL Parquet - Comet (Scan) (BloomFilterAgg)                  110            135          15         95.7          10.4       1.0X
SQL Parquet - Comet (Scan, Exec) (BloomFilterAgg)             69             78          12        152.9           6.5       1.6X

BloomFilterAggregate Exec (cardinality 1048576):   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
---------------------------------------------------------------------------------------------------------------------------------
SQL Parquet - Spark (BloomFilterAgg)                         165            183          14         63.6          15.7       1.0X
SQL Parquet - Comet (Scan) (BloomFilterAgg)                  169            184          11         62.0          16.1       1.0X
SQL Parquet - Comet (Scan, Exec) (BloomFilterAgg)            117            126           9         89.2          11.2       1.4X

Results from the benchmark I just added:

Looks like a 40% improvement.

Just putting notes for the test failure. It's failing one Spark test in InjectRuntimeFilterSuite. The test is "Merge runtime bloom filters". It's failing in a check in CometArrayImporter when it's bringing Arrow data from Native back over to JVM.
The plan is a bit of a monster, but I'll provide it below:
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [c1#45920, b1#45919], [c2#45926, b2#45925], Inner
   :- Sort [c1#45920 ASC NULLS FIRST, b1#45919 ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(c1#45920, b1#45919, 5), ENSURE_REQUIREMENTS, [plan_id=739]
   :     +- Filter (((isnotnull(c1#45920) AND isnotnull(b1#45919)) AND might_contain(Subquery subquery#45946, [id=#610].bloomFilter, xxhash64(c1#45920, 42))) AND might_contain(Subquery subquery#45949, [id=#678].bloomFilter, xxhash64(b1#45919, 42)))
   :        :  :- Subquery subquery#45946, [id=#610]
   :        :  :  +- AdaptiveSparkPlan isFinalPlan=false
   :        :  :     +- CometProject [mergedValue#45952], [named_struct(bloomFilter, bloomFilter#45945, bloomFilter, bloomFilter#45948) AS mergedValue#45952]
   :        :  :        +- !CometHashAggregate [buf#45954, buf#45955], Final, [bloom_filter_agg(xxhash64(c2#45926, 42), 1000000, 8388608, 0, 0), bloom_filter_agg(xxhash64(b2#45925, 42), 1000000, 8388608, 0, 0)]
   :        :  :           +- CometExchange SinglePartition, ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=605]
   :        :  :              +- !CometHashAggregate [c2#45926, b2#45925], Partial, [partial_bloom_filter_agg(xxhash64(c2#45926, 42), 1000000, 8388608, 0, 0), partial_bloom_filter_agg(xxhash64(b2#45925, 42), 1000000, 8388608, 0, 0)]
   :        :  :                 +- CometProject [c2#45926, b2#45925], [c2#45926, b2#45925]
   :        :  :                    +- CometFilter [a2#45924, b2#45925, c2#45926], (((isnotnull(a2#45924) AND (a2#45924 = 62)) AND isnotnull(c2#45926)) AND isnotnull(b2#45925))
   :        :  :                       +- CometScan parquet spark_catalog.default.bf2[a2#45924,b2#45925,c2#45926] Batched: true, DataFilters: [isnotnull(a2#45924), (a2#45924 = 62), isnotnull(c2#45926), isnotnull(b2#45925)], Format: CometParquet, Location: ... PartitionFilters: [], PushedFilters: [IsNotNull(a2), EqualTo(a2,62), IsNotNull(c2), IsNotNull(b2)], ReadSchema: struct<a2:int,b2:int,c2:int>
   :        :  +- Subquery subquery#45949, [id=#678]
   :        :     +- AdaptiveSparkPlan isFinalPlan=false
   :        :        +- CometProject [mergedValue#45952], [named_struct(bloomFilter, bloomFilter#45945, bloomFilter, bloomFilter#45948) AS mergedValue#45952]
   :        :           +- !CometHashAggregate [buf#45954, buf#45955], Final, [bloom_filter_agg(xxhash64(c2#45926, 42), 1000000, 8388608, 0, 0), bloom_filter_agg(xxhash64(b2#45925, 42), 1000000, 8388608, 0, 0)]
   :        :              +- CometExchange SinglePartition, ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=673]
   :        :                 +- !CometHashAggregate [c2#45926, b2#45925], Partial, [partial_bloom_filter_agg(xxhash64(c2#45926, 42), 1000000, 8388608, 0, 0), partial_bloom_filter_agg(xxhash64(b2#45925, 42), 1000000, 8388608, 0, 0)]
   :        :                    +- CometProject [c2#45926, b2#45925], [c2#45926, b2#45925]
   :        :                       +- CometFilter [a2#45924, b2#45925, c2#45926], (((isnotnull(a2#45924) AND (a2#45924 = 62)) AND isnotnull(c2#45926)) AND isnotnull(b2#45925))
   :        :                          +- CometScan parquet spark_catalog.default.bf2[a2#45924,b2#45925,c2#45926] Batched: true, DataFilters: [isnotnull(a2#45924), (a2#45924 = 62), isnotnull(c2#45926), isnotnull(b2#45925)], Format: CometParquet, Location: ... PartitionFilters: [], PushedFilters: [IsNotNull(a2), EqualTo(a2,62), IsNotNull(c2), IsNotNull(b2)], ReadSchema: struct<a2:int,b2:int,c2:int>
   :        +- CometScan parquet spark_catalog.default.bf1[a1#45918,b1#45919,c1#45920,d1#45921,e1#45922,f1#45923] Batched: true, DataFilters: [isnotnull(c1#45920), isnotnull(b1#45919)], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/Users/matt/git/spark/spark-warehouse/org.apache.spark.sql.Inject..., PartitionFilters: [], PushedFilters: [IsNotNull(c1), IsNotNull(b1)], ReadSchema: struct<a1:int,b1:int,c1:int,d1:int,e1:int,f1:int>
   +- CometSort [a2#45924, b2#45925, c2#45926, d2#45927, e2#45928, f2#45929], [c2#45926 ASC NULLS FIRST, b2#45925 ASC NULLS FIRST]
      +- CometExchange hashpartitioning(c2#45926, b2#45925, 5), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=742]
         +- CometFilter [a2#45924, b2#45925, c2#45926, d2#45927, e2#45928, f2#45929], (((isnotnull(a2#45924) AND (a2#45924 = 62)) AND isnotnull(c2#45926)) AND isnotnull(b2#45925))
            +- CometScan parquet spark_catalog.default.bf2[a2#45924,b2#45925,c2#45926,d2#45927,e2#45928,f2#45929] Batched: true, DataFilters: [isnotnull(a2#45924), (a2#45924 = 62), isnotnull(c2#45926), isnotnull(b2#45925)], Format: CometParquet, Location: ... PartitionFilters: [], PushedFilters: [IsNotNull(a2), EqualTo(a2,62), IsNotNull(c2), IsNotNull(b2)], ReadSchema: struct<a2:int,b2:int,c2:int,d2:int,e2:int,f2:int>

This is what it looks like on the main branch:
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [c1#45920, b1#45919], [c2#45926, b2#45925], Inner
   :- Sort [c1#45920 ASC NULLS FIRST, b1#45919 ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(c1#45920, b1#45919, 5), ENSURE_REQUIREMENTS, [plan_id=729]
   :     +- Filter (((isnotnull(c1#45920) AND isnotnull(b1#45919)) AND might_contain(Subquery subquery#45946, [id=#605].bloomFilter, xxhash64(c1#45920, 42))) AND might_contain(Subquery subquery#45949, [id=#668].bloomFilter, xxhash64(b1#45919, 42)))
   :        :  :- Subquery subquery#45946, [id=#605]
   :        :  :  +- AdaptiveSparkPlan isFinalPlan=false
   :        :  :     +- Project [named_struct(bloomFilter, bloomFilter#45945, bloomFilter, bloomFilter#45948) AS mergedValue#45952]
   :        :  :        +- ObjectHashAggregate(keys=[], functions=[bloom_filter_agg(xxhash64(c2#45926, 42), 1000000, 8388608, 0, 0), bloom_filter_agg(xxhash64(b2#45925, 42), 1000000, 8388608, 0, 0)])
   :        :  :           +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=601]
   :        :  :              +- ObjectHashAggregate(keys=[], functions=[partial_bloom_filter_agg(xxhash64(c2#45926, 42), 1000000, 8388608, 0, 0), partial_bloom_filter_agg(xxhash64(b2#45925, 42), 1000000, 8388608, 0, 0)])
   :        :  :                 +- CometProject [c2#45926, b2#45925], [c2#45926, b2#45925]
   :        :  :                    +- CometFilter [a2#45924, b2#45925, c2#45926], (((isnotnull(a2#45924) AND (a2#45924 = 62)) AND isnotnull(c2#45926)) AND isnotnull(b2#45925))
   :        :  :                       +- CometScan parquet spark_catalog.default.bf2[a2#45924,b2#45925,c2#45926] Batched: true, DataFilters: [isnotnull(a2#45924), (a2#45924 = 62), isnotnull(c2#45926), isnotnull(b2#45925)], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/Users/matt/git/spark/spark-warehouse/org.apache.spark.sql.Inject..., PartitionFilters: [], PushedFilters: [IsNotNull(a2), EqualTo(a2,62), IsNotNull(c2), IsNotNull(b2)], ReadSchema: struct<a2:int,b2:int,c2:int>
   :        :  +- Subquery subquery#45949, [id=#668]
   :        :     +- AdaptiveSparkPlan isFinalPlan=false
   :        :        +- Project [named_struct(bloomFilter, bloomFilter#45945, bloomFilter, bloomFilter#45948) AS mergedValue#45952]
   :        :           +- ObjectHashAggregate(keys=[], functions=[bloom_filter_agg(xxhash64(c2#45926, 42), 1000000, 8388608, 0, 0), bloom_filter_agg(xxhash64(b2#45925, 42), 1000000, 8388608, 0, 0)])
   :        :              +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=664]
   :        :                 +- ObjectHashAggregate(keys=[], functions=[partial_bloom_filter_agg(xxhash64(c2#45926, 42), 1000000, 8388608, 0, 0), partial_bloom_filter_agg(xxhash64(b2#45925, 42), 1000000, 8388608, 0, 0)])
   :        :                    +- CometProject [c2#45926, b2#45925], [c2#45926, b2#45925]
   :        :                       +- CometFilter [a2#45924, b2#45925, c2#45926], (((isnotnull(a2#45924) AND (a2#45924 = 62)) AND isnotnull(c2#45926)) AND isnotnull(b2#45925))
   :        :                          +- CometScan parquet spark_catalog.default.bf2[a2#45924,b2#45925,c2#45926] Batched: true, DataFilters: [isnotnull(a2#45924), (a2#45924 = 62), isnotnull(c2#45926), isnotnull(b2#45925)], Format: CometParquet, Location: ... PartitionFilters: [], PushedFilters: [IsNotNull(a2), EqualTo(a2,62), IsNotNull(c2), IsNotNull(b2)], ReadSchema: struct<a2:int,b2:int,c2:int>
   :        +- CometScan parquet spark_catalog.default.bf1[a1#45918,b1#45919,c1#45920,d1#45921,e1#45922,f1#45923] Batched: true, DataFilters: [isnotnull(c1#45920), isnotnull(b1#45919)], Format: CometParquet, Location: ... PartitionFilters: [], PushedFilters: [IsNotNull(c1), IsNotNull(b1)], ReadSchema: struct<a1:int,b1:int,c1:int,d1:int,e1:int,f1:int>
   +- CometSort [a2#45924, b2#45925, c2#45926, d2#45927, e2#45928, f2#45929], [c2#45926 ASC NULLS FIRST, b2#45925 ASC NULLS FIRST]
      +- CometExchange hashpartitioning(c2#45926, b2#45925, 5), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=732]
         +- CometFilter [a2#45924, b2#45925, c2#45926, d2#45927, e2#45928, f2#45929], (((isnotnull(a2#45924) AND (a2#45924 = 62)) AND isnotnull(c2#45926)) AND isnotnull(b2#45925))
            +- CometScan parquet spark_catalog.default.bf2[a2#45924,b2#45925,c2#45926,d2#45927,e2#45928,f2#45929] Batched: true, DataFilters: [isnotnull(a2#45924), (a2#45924 = 62), isnotnull(c2#45926), isnotnull(b2#45925)], Format: CometParquet, Location: ... PartitionFilters: [], PushedFilters: [IsNotNull(a2), EqualTo(a2,62), IsNotNull(c2), IsNotNull(b2)], ReadSchema: struct<a2:int,b2:int,c2:int,d2:int,e2:int,f2:int>

Debugger output from the failing state in CometArrayImporter.
this = {CometArrayImporter@18451}
snapshot = {ArrowArray$Snapshot@18448}
 length = 1
 null_count = 0
 offset = 0
 n_buffers = 1
 n_children = 2
 buffers =<PHONE_NUMBER>81104
 children =<PHONE_NUMBER>81088
 dictionary = 0
 release =<PHONE_NUMBER>
 private_data =<PHONE_NUMBER>35808
children = {long[2]@18449} [105553174820048,<PHONE_NUMBER>20208]
 0 =<PHONE_NUMBER>20048
 1 =<PHONE_NUMBER>20208
childVectors = {ArrayList@18450}  size = 1
 0 = {VarBinaryVector@22881} "[]"
vector = {StructVector@18452} "[]"
 reader = {NullableStructReaderImpl@22883}
 writer = {NullableStructWriter@22884} "org.apache.comet.shaded.arrow.vector.complex.impl.NullableStructWriter@8a493b[index = 0]"
 validityBuffer = {ArrowBuf@22885} "ArrowBuf[1], address:0, capacity:0"
 validityAllocationSizeInBytes = 497
 NonNullableStructVector.reader = {SingleStructReaderImpl@22886}
 field = {Field@22887} "Struct not null"
 valueCount = 0
 ephPair = null
 vectors = {PromotableMultiMapWithOrdinal@22888}
 allowConflictPolicyChanges = true
 conflictPolicy = {AbstractStructVector$ConflictPolicy@22889} "CONFLICT_REPLACE"
 name = null
 allocator = {RootAllocator@18453} "Allocator(ROOT) 0/2176/4196904/9223372036854775807 (res/actual/peak/limit)\n"
 callBack = null
children.length = 2

The entire subquery runs in native code now, so my guess is that the output from that projection, which looks like it should be a struct with two binary values in it is wrong. I'm not sure if it's a bug in the projection, or something further downstream.

I do not have time to look at this error yet. I may take a look after the conference.

Can't say I see a huge different in TPC-H or TPC-DS locally, but the plans I looked at were typically building filters over very small relations.

@mbutrovich Is it possible to trace back where children and childVectors are populated?

The Spark SQL test failure can be fixed by #1016.

I merged the fix. You can rebase and re-trigger CI now.

Merged in updated main, thanks for the quick fix!

I tested with TPC-H q5 and see that we are now running the bloom filter agg natively


update azureconst

also make work w/ francesouth, australiacentral, and australiacentral2

What this PR does / why we need it: update const and generator code
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
If applicable:

[ ] documentation
[ ] unit tests
[ ] tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

Release note:

update azureconst

/approve


Add ShouldNotHappenException for EasyCI

If you install the Easy CI package separately, it complains that it cannot find the exception.

Thank you
CI needs pass first. It's the replace composer thing

I don't know how to fix that. It's in master.

Like this: https://github.com/symplify/symplify/pull/2617
It's common error for monorepo dependency on external package that depends on local package.

Ok thanks for fixing. I still don't get it 😅

I see, it's a tricky circular dependency :)

symplify requires rector
rector require symplify of 9.0.1
but symplify here is dev-master (self.version)

That's why composer is confused, because it need symplify 9.0.1 and both dev-master
The solution is to fake here 9.0.1 version. It could be done better with on-release update of self-version. Space for monorepo-builder improvement.

I assume this was accidentally committed?

No, the symplify version changed in rector, so it was commited on purpose. This is just something composer does not handle very well. Same goes for the composer ENV for dev-master. I assume it's constraint for performance improvement, as circular matrixes can go performance heavy very easily.
If we use some external package that depends on symplify of version X and here we had self-version, this would happen again.

Here is another "problem" you'll meet while building monorepos:
https://tomasvotruba.com/blog/2019/01/31/monorepo-composer-magic


video doesn't show any first pic of the video before start

Hey!
When I add the video it shows on the blog the controls and gives space to the video but doesn't show a "preview" (the first pic of the video) ....
to see on  http://www.zentraders.net/devsite/blog/  > Test Video
What can I do about that?
THanks

@karok Closing old issues. If you'd like to reopen this, let us know. Thanks!


node_modules/history included in tarball published on npm

Just tried to run npm shrinkwrap on a project that includes react-lazyload as a dependency and got this error:
npm ERR! Problems were encountered
npm ERR! Please correct and try again.
npm ERR! extraneous<EMAIL_ADDRESS>/path/to/node_modules/react-lazyload/node_modules/history

Here's a link to the module on npmcdn.
It seems like the issue with erroneously publishing the history module should be fixed in recent versions of npm -- see https://github.com/npm/npm/issues/9642.

That's weird, I'm building and publishing this package using<EMAIL_ADDRESS>the referenced issue should have been fixed. Anything you suggest should I investigate in?

Looks like the issue for npm 3 was fixed in<EMAIL_ADDRESS>(https://github.com/yahoo/react-intl/pull/395#issuecomment-207217451). Just tried doing the following locally with node 6.4.0 and<EMAIL_ADDRESS>git clone git@github.com:jasonslyvia/react-lazyload.git
cd react-lazyload
npm install
npm pack
tar xzf react-lazyload-2.1.4.tgz
ls package/node_modules

and get the following message: ls: package/node_modules: No such file or directory

Cool, I'll update ASAP.

@jasonslyvia any update on this? Just upgraded to<EMAIL_ADDRESS>and the node_modules folder is still included<EMAIL_ADDRESS>

I sorry it has been delayed, could you please try 2.1.6?

awesome, thanks for pushing that up!

Cheers, thanks for your patience.


Implement is-cross-origin internally so it doesn't throw

is-cross-origin throws if received object isn't a window
this apparently is possible (dom clobbering with indexes 😱)
no reason for it to throw, just continue to next window in frames array
in fact, no reason for this to be an external package at all, implement internally instead

making Snow a deps-free project 🎉

This allowed Snow bypass:
(function(){
    const ifr = document.createElement('iframe');
    const ifr2 = document.createElement('iframe');
    document.body.appendChild(ifr);
    const div = document.createElement('div');
    div.id='0';
    setTimeout(() => { ifr2.contentWindow.alert.call(top,1); }, 500);
    try { ifr.contentWindow.document.body.appendChild(div); } catch {}
    document.body.appendChild(ifr2);
}());

Update, #111 was not strong enough, the isWindow function could have been easily bypassed, e2cf42e introduces a hardening fix to that


Infyom Scaffold: How to return to index page all Articles belongs to Auth User

i created Article Scaffold, and i want to return all articles belongs to specific auth user, note that ArticleDataTable.php is the file response of return all data to the index of articles, and i hope that my problem  is clear to understand.

Open ArticleDataTable.php

Go to query method
Find code that looks like :-

$articles = Article::query();

Replace with below code :-

$articles = Article::query();
$articles->where('user', \Auth::user()->id);


Add stdin provider as an alternative to fs provider

Contributes #187
Example:
cat ./tests/manual/manifests-all/root1.yaml | kahoy apply --dry-run --no-log --kube-provider-id test -n-

▶ Apply (1 resources)
└── ▶ root (1 resources)
    └── apps/v1/Deployment/test-kahoy/app-root1 (stdin)

LGTM this will be great for kubectl users failiar with commands like cat manifest.yaml | kubectl apply -f -

LGTM this will be great for kubectl users failiar with commands like cat manifest.yaml | kubectl apply -f -

Would be awesome, sadly we are already using it with --include-changes flag. It would be a breaking change.


Runloop 'bug' ?

Hi!
My server sometimes a little bit sloooow...
        while(isRunLoop) {
            NSRunLoop.currentRunLoop().runMode(NSDefaultRunLoopMode, beforeDate: NSDate.distantFuture() as! NSDate)
        }

sometimes it takes 7-8 sec while the isRunLoop value will be false (and it freeze the UI.)
So I mean the NSRunLoop.current.... freeze the UI.
I saw that it is on a background thread so it is so weird..
You can reproduce this if you do something like this, and you start a connection in a UITableViewController.
let runAlways = true
        while(runAlways) {
            NSRunLoop.currentRunLoop().runMode(NSDefaultRunLoopMode, beforeDate: NSDate.distantFuture() as! NSDate)
        }

You cannot scroll until the while() is running.
Do you have any idea  what am I doing wrong? Or is it a real bug?

Sorry the "bug" is real but it is not the problem..


Tabs: don't apply :active style when a tab is active

When the tab is active and clicked on, the bottom line turns grey, this shouldn't happen.

Closed with https://github.com/AbsaOSS/cps-shared-ui/pull/352


[Clinical Site] Fix the flow on how the TS status becomes  'Enrolled'

When the Site adds a TS number for the TS, the status of the Patient should automatically change from 'Screened' to 'Enrolled'. In the future, the button should be changed to 'Enroll the Patient'. Clicking on the button, will generate automatically the TS number and it will also change the Status of the Patient from 'Screened' to 'Enrolled'.

Well, this is how it happens now. The TS number could not be automatically generated while the TS number is the sk key with the external hospital db.


Some questions about SoftmaxCrossEntropyLoss

Hello, I have some questions about this function https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/losses/cross_entropy.py#L18C1-L18C7. And I consider that this function could do softmax operation for logits according to the func name. I don't know my understanding is correct or not. However, I found that the sum of logits after computing loss is not one and I am doubted that there is no softmax operation conducted in logits. So, is there any misunderstood?

It's supposed to do the same as pytorch's CrossEntropyLoss. It doesn't compute softmax (e.g. it returns a loss, not probabilities).

Okay, thank you for your reply. I am very curious that why this function is called SoftmaxCrossEntropyLossFn. And I have misunderstood that this function could do softmax for logits.  And I have noticed the there is a parameter called inplace_backward, so what does this parameter can do?

For historical reason, the apex version that we built on calls it SoftmaxCrossEntropyLoss.
inplace_backward will save some memory.

So, if I set the inplace_backward is True. we notice that the logits is connected to the loss. If loss have some changes in backward, then logits will change too. Thank you very much!


Automatic registering of commands outside of /app folder

Laravel Version: ^8|^9|^10
PHP Version: ^7.4|^8

Description:
Documenation at https://laravel.com/docs/10.x/artisan#registering-commands says:
"You are even free to make additional calls to the load method to scan other directories for Artisan commands"
Though automatic registering commands at Kernel.php with ->load() does not work for commands located outside of /app folder.
Steps To Reproduce:
Let's put additional commands location at commands() method:
protected function commands()
{
    $this->load(__DIR__ . '/Commands');

    // Neither of these locations would be scanned
    $this->load(__DIR__.'/../../modules/Integrations/Console/Commands');
    $this->load(base_path().'/modules/Integrations/Console/Commands');
}

Unfortunately commands at this additional location won't be recognized and upon running a command we'll get:
There are no commands defined in the "integration:sync" namespace. Did you mean one of these?
I can see that load() method has a "hardcoded" realpath(app_path()). It would be great if the restriction of placing commands only in the /app folder was lifted and automatic command recognition would work from other locations.

Heya that's correct. This is limited to the app directory. If you want additional directories to be scanned you can maybe attempt a PR?

@boryn were you able workaround this? I did mine by registering the command directly like
use CustomNamespace\Outside\AppFolder\ExportFiles;

protected function commands() {
	$this->load(__DIR__ . '/Commands');
        $this->getArtisan()->resolve(ExportFiles::class);
}

@boryn were you able workaround this? I did mine by registering the command directly like
use CustomNamespace\Outside\AppFolder\ExportFiles;

protected function commands() {
	$this->load(__DIR__ . '/Commands');
        $this->getArtisan()->resolve(ExportFiles::class);
}

The problem with this is you are still manually reregistering the command, instead of auto registering it. And you can also just put this in the the command in $commands in Kernel.php
    protected $commands = [
        ExportFiles::class
    ];

What I did was this and it seems to work. I put this in my Kernel.php
/**
 * Register the commands for the application.
 *
 * @return void
 */
protected function commands()
{
    // we need to load our command from outside the default structure
    $this->loadCommands(
        __DIR__ . "/Commands",
    );

    require base_path('routes/console.php');
}

private function loadCommands(string $path): void {
    $class = new \ReflectionClass($this);
    $namespace = $class->getNamespaceName();

    // go through each file so we can load the command
    foreach ((new Finder)->in($path)->files() as $file) {
        // remove .php, remove the path so we only keep the namespace
        $command = $namespace.str_replace(
            ['/', '.php'],
            ['\\', ''],
            Str::after($file->getRealPath(), realpath(__DIR__))
        );

        // and load the commands
        Application::starting(function ($artisan) use ($command) {
            $artisan->resolve($command);
        });
    }
}

Hi @JeffreyHosler!
I agree, solution with overwriting the load() method would be the best (and finally publishing it as merge request at Laravel framework). It would be good to keep possibility of passing paths both as string or as an array.
But, unfortunately your solution did not want to work out of the box in my case.
For the regular folder (__DIR__ . '/Commands') I get:
Target [App\Console\Commands\ScheduleList] is not instantiable.
For a non standard folder like $this->loadCommands(__DIR__.'/../../modules/Integrations/Console/Commands'); it gets completely mixed up with the namespaces, while testing on Windows I get:
Target class [App\ConsoleE:\www\project\modules\Integrations\Console\Commands\MyCustomCommand] does not exist.


shell_benchmarks failed to run in the release mode

See https://cirrus-ci.com/task/6348658408947712 for an example.

@chinmaygarde @gaaclarke : do I need  --full-dart-sdk in ./flutter/tools/gn in order to run shell_benchmarks? That's the only difference I can tell from our debug and release build, and the debug build can run shell_benchmarks. I'm having trouble ssh into my desktop so I'm unable to test it now. I can try it tomorrow if there's no quick answer on your mind.

No. You never need --full-dart-sdk. Are you able the run the benchmark runner locally. It may not be setup for running in release in that it doesn't setup AOT snapshots.

Fixed in https://github.com/flutter/engine/pull/17051.


Add pedestrian motorbike model

Added models and glb for motorcycle and pedestrian.

Also rescaled other vehicles

closes #756 #755

Were you also going to add .urdf models for physics?

Were you also going to add .urdf models for physics?

nope, in the meeting after standup, it was decided urdf files aren't needed. We currently don't use different urdf for different vehicle types. Also having more urdf increases complexity of modeling physics in my understanding

We will need to fix test observations.

We will need to fix test observations.

right, sorry I missed that, thought only test_notebook is failing. Also were we going to disable test_notebook test?

We will need to fix test observations.

right sorry I missed that. Also were we going to disable test_notebook test?

I will try fixing it by explicitly installing ipykernel.


Sort_by function not work very well with hungarian letters

What system are you running Yazi on?
Linux X11
What terminal are you running Yazi in?
alacritty 0.11.0
Yazi version
yazi 0.2.2 (e607943 2024-01-31)
Did you try the latest main branch to see if the problem has already been fixed?
Not tried, and I'll explain why below
Describe the bug
Yazi does not take into account accented characters when sorting files and folders. He always puts it at the bottom of the list, like this:

Expected Behavior
The correct order would be the following for accented characters (Hungarian):
Alpha
Átrium
Beta
.
Elemér
Érvényes
.
Hedvig
Ilona
Írott
.
Olga
Órmotlan
Öveges
Őzike
.
Ubul
Útvonal
Üveg
Űr
.
To Reproduce
Create files:
Alpha
Átrium
Beta
Elemér
Érvényes
Hedvig
Ilona
Írott
Olga
Órmotlan
Öveges
Őzike
Ubul
Útvonal
Üveg
Űr
and the problem will be visible
Configuration
[manager]
ratio          = [ 0, 2, 2 ]
# ratio          = [ 0, 2, 2 ]
sort_by        = "natural"
sort_sensitive = false
sort_reverse   = false
sort_dir_first = true

Anything else?
No response

This is the expected behavior, Ö and O are different characters. Isn't it more intuitive to separate them into distinct groups, just like Yazi is currently doing?

No, because for us Hungarians, the accents in the abc come in the line described above, we still look for these characters.

Added to the Feature Requests, closing in favor of that list.

Hi, this feature has now been implemented by @clispios in https://github.com/sxyazi/yazi/pull/1053! :tada: