fuse read performance couldn't scale and much lower than Client IO bench results

Alluxio Version:
2.8.0
Describe the bug
We are evaluating alluxio cache acceleration effect for our AI training scenario. The typical user case is loading many small images(~100KB) with POSIX interface. In order to make things easier to analyse, we get start from the single node configuration: just one alluxio worker, and want to know its potential.
After our test, we found that the maximum read performance provided by this single worker could reach about 3.3GB/s under Alluxio Client IO bench, however when read through fuse interface, this single worker could just provide about 1.5GB/s read bandwidth, even though we run the fuse IO bench workload from multiple different physical hosts, it still couldn't unlock the worker's potential.
To Reproduce
To avoid client metadata operations with master affecting test results,  we enable the client metadata cache and disable the file access time update with below configuration:
#User properties
alluxio.user.metadata.cache.enabled=true
alluxio.user.metadata.cache.expiration.time=2h
alluxio.user.metadata.cache.max.size=2000000
alluxio.user.update.file.accesstime.disabled=true

So after the first read test, the metadata could be cache in the client side and client won't need to interact with master anymore in the following read test.
Test with StressClientIO to evaluate maximum read performance single worker could provide:

First create test files:
./alluxio-2.8.0/bin/alluxio runClass alluxio.stress.cli.client.StressClientIOBench --operation Write --base alluxio://<master address>:19998/stress-client-io-base --file-size 100k --threads 256  --warmup 5s --duration 30s

Then read test files with single client host:
./alluxio-2.8.0/bin/alluxio runClass alluxio.stress.cli.client.StressClientIOBench --operation ReadFully --base alluxio://<master address>:19998/stress-client-io-base --file-size 100k --threads 256  --warmup 5s --duration 30s
Test output:
2022-08-11 23:35:22,021 INFO  StressClientIOBench - thread count: 256, errors: 0, IO throughput (MB/s): 2010.835449
Typical network bandwidth and cpu usage of worker are as below:

The read test files from two different client hosts:
mpirun -np 2 --hosts <client_host_A,client_host_B> ./alluxio-2.8.0/bin/alluxio runClass alluxio.stress.cli.client.StressClientIOBench --operation Write --base alluxio://<master address>:19998/stress-client-io-base --file-size 100k --threads 256  --warmup 5s --duration 30s
Test output:
2022-08-12 00:54:15,647 INFO  StressClientIOBench - thread count: 256, errors: 0, IO throughput (MB/s): 1628.829468
2022-08-12 00:54:15,656 INFO  StressClientIOBench - thread count: 256, errors: 0, IO throughput (MB/s): 1638.937622
Typical network bandwidth and cpu usage of worker are as below:

Test with Fuse IO stress bench to evaluate maximum read performance single worker could provide:

First create test files:
./bin/alluxio fs mkdir /fuseIOBench
integration/fuse/bin/alluxio-fuse mount /mnt/test /fuseIOBench
./bin/alluxio runClass alluxio.stress.cli.fuse.FuseIOBench --operation Write --local-path /mnt/ --threads 12 --num-dirs 64 --num-files-per-dir 4000 --file-size 100k --buffer-size 100k
mount it into another directory & List it to preload the metadata
integration/fuse/bin/alluxio-fuse mount /mnt/read_alluxio/ /fuseIOBench
./bin/alluxio runClass alluxio.stress.cli.fuse.FuseIOBench --operation ListFile --local-path /mnt/read_alluxio/
Read test file:
./bin/alluxio runClass alluxio.stress.cli.fuse.FuseIOBench --operation LocalRead --local-path /mnt/read_alluxio/ --threads 32 --num-dirs 64 --num-files-per-dir 4000 --file-size 100k --buffer-size 100k --warmup 5s --duration 10s
Test output:
2022-08-12 01:59:36,529 INFO  FuseIOBench - job worker id: local-task-0, errors: 0, IO throughput (MB/s): 1417.479858
Run the read test from two different client hosts to check if the aggregate read bandwidth could increase, and also before run the read test, we listed the files in the new client host to load metadata in advance:
#on new client host  ./bin/alluxio runClass alluxio.stress.cli.fuse.FuseIOBench --operation ListFile --local-path /mnt/read_alluxio/
mpirun -np 2 --hosts <client_host_A,client_host_B> ./bin/alluxio runClass alluxio.stress.cli.fuse.FuseIOBench --operation LocalRead --local-path /mnt/read_alluxio/ --threads 32 --num-dirs 64 --num-files-per-dir 4000 --file-size 100k --buffer-size 100k --warmup 5s --duration 10s
Test output:
2022-08-12 02:01:48,245 INFO  FuseIOBench - job worker id: local-task-0, errors: 0, IO throughput (MB/s): 778.780273
2022-08-12 02:01:48,885 INFO  FuseIOBench - job worker id: local-task-0, errors: 0, IO throughput (MB/s): 809.390259

Expected behavior
From above test, we see that worker read throughput was able to increase from 2GB/s to 3.3GB/s ( ~65% gain) when benching Alluxio Client number increases from 1 to 2, however when benching Fuse Client doubled, the worker read throughput just got a minor increase, from 1.41GB/s to 1.57GB/s (~10%). And the absolute performance is less than half of the Alluxio Client case: 1.57GB/s Vs. 3.3GB/s, it seems that worker read request process ability is restrained in Fuse mode. When fuse clients doubled, the read bandwidth it could get from worker is nearly halved, although we think it should remain the same since worker has the potential to support it.
Urgency
urgent for our solution selection
Additional context
Further details could be provided if needed

@JianyuLi Hello, may I ask how to get the output like "2022-08-11 23:35:22,021 INFO StressClientIOBench - thread count: 256, errors: 0, IO throughput (MB/s): 2010.835449".
In my env, the output looks like this:
{
"baseParameters" : {
"mCluster" : false,
"mClusterLimit" : 0,
"mClusterStartDelay" : "10s",
"mJavaOpts" : [ ],
"mProfileAgent" : "",
"mBenchTimeout" : "20m",
"mId" : "local-task-0",
"mDistributed" : false,
"mStartMs" : -1,
"mInProcess" : true,
"mHelp" : false
},
"parameters" : {
"className" : "alluxio.stress.client.ClientIOParameters",
"mClientType" : "ALLUXIO_HDFS",
"mReadType" : "CACHE",
"mWriteType" : "ASYNC_THROUGH",
"mOperation" : "READ_BYTE_BUFFER",
"mClients" : 1,
"mThreads" : [ 1 ],
"mBasePath" : "alluxio://<IP_ADDRESS>:19998/stress-client-io-base4",
"mBaseAlias" : "",
"mTag" : "",
"mFileSize" : "1g",
"mBufferSize" : "512k",
"mBlockSize" : "64m",
"mDuration" : "30s",
"mWarmup" : "5s",
"mReadSameFile" : false,
"mReadRandom" : false,
"mWriteNumWorkers" : 1,
"mConf" : { }
},
"recordStartMs" : 0,
"endMs" : 0,
"errors" : [ ],
"timeToFirstBytePerThread" : { },
"threadCountResults" : {
"1" : {
"recordStartMs" :<PHONE_NUMBER>374,
"endMs" :<PHONE_NUMBER>377,
"errors" : [ ],
"iobytes" :<PHONE_NUMBER>
}
},
"className" : "alluxio.stress.client.ClientIOTaskResult"
}

@JianyuLi Hello, may I ask how to get the output like "2022-08-11 23:35:22,021 INFO StressClientIOBench - thread count: 256, errors: 0, IO throughput (MB/s): 2010.835449".
In my env, the output looks like this:
{ "baseParameters" : { "mCluster" : false, "mClusterLimit" : 0, "mClusterStartDelay" : "10s", "mJavaOpts" : [ ], "mProfileAgent" : "", "mBenchTimeout" : "20m", "mId" : "local-task-0", "mDistributed" : false, "mStartMs" : -1, "mInProcess" : true, "mHelp" : false }, "parameters" : { "className" : "alluxio.stress.client.ClientIOParameters", "mClientType" : "ALLUXIO_HDFS", "mReadType" : "CACHE", "mWriteType" : "ASYNC_THROUGH", "mOperation" : "READ_BYTE_BUFFER", "mClients" : 1, "mThreads" : [ 1 ], "mBasePath" : "alluxio://<IP_ADDRESS>:19998/stress-client-io-base4", "mBaseAlias" : "", "mTag" : "", "mFileSize" : "1g", "mBufferSize" : "512k", "mBlockSize" : "64m", "mDuration" : "30s", "mWarmup" : "5s", "mReadSameFile" : false, "mReadRandom" : false, "mWriteNumWorkers" : 1, "mConf" : { } }, "recordStartMs" : 0, "endMs" : 0, "errors" : [ ], "timeToFirstBytePerThread" : { }, "threadCountResults" : { "1" : { "recordStartMs" :<PHONE_NUMBER>374, "endMs" :<PHONE_NUMBER>377, "errors" : [ ], "iobytes" :<PHONE_NUMBER> } }, "className" : "alluxio.stress.client.ClientIOTaskResult" }

In the end of ${ALLUXIO_BASE}/logs/user/user_xxx.log, we could get the test result summary of StressBench.

Thanks very much @JianyuLi    Alluxio is difficult to use after some attempts. In our env, alluxio was deployed in the kubernetes cluster, the speed about 200MB/s with StreeBech. But 1MB/s with fuse mount.

@LuQQiu Could you take a look at this issue?


Implement AggregatorAccount.withdrawAction

escrow_withdraw
aggregator_withdraw

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Use Arcade to install .NET Core runtimes

Removes copies of our dotnet-install scripts in favor of a new Arcade feature that installs .NET Core runtimes for us.

Blocked on https://github.com/dotnet/arcade/issues/2744


[firebase_crashlytics] Can force it to upload the reports?

firebase_crashlytics: 0.0.4+4
Currently, i integrated with firebase_crashlytics. I found that this plugin will upload the crashlytics report when the application restart. But, for dart, the uncaught handling is handled the exception. The application will not be forced to restart. So, the report will not be uploaded.
If i try to add exit(0) into FlutterError.onError to force kill the application, it is not recommended for iOS.
So, any solutions to force firebase_crashlytics to report? Or, any ideas to solve this problem?

I am also interested in forcing reports to crashlytics. In Sentry you can upload an report.

@KingWu
This issue has been moved to https://github.com/FirebaseExtended/flutterfire/issues/1142. Any further collaboration will be done there.

I think it's related to https://github.com/FirebaseExtended/flutterfire/issues/1138


Tooltip not closing

Hi since last update (1.4.2) All my tooltips are not closing.
This is reproducible when you have 2 or more tooltips one near each other and you hover over them.
Some text

Can we have a plunker please?

Here is a plunk that reproduces the issue.

In looking at the test associated with the tooltip for this scenario and I'm trying to figure out why this test is not failing.  I've tried adding timeout flush and using a count of elements, but still haven't gotten the test to fail?

Thanks guys for your quick reaction.
Any idea when this fix would be released?

@bogdanalexe90, we don't have a time frame for 0.14.3 yet, but please feel free to clone the repo, run an npm install and then grunt to build the SNAPSHOT release (which will appear in the dist folder.


Miss spelt gist

Gits => Snippets
should be
Gist => Snippets
source: https://gist.github.com/
https://gitlab.com/explore/snippets

Thank you so much. I fixed that ;)


Newlines are not rendered

Hi,
I found one minor issue, newlines are not being rendered.
For example my note on nostrudel:

This is how it looks on npub.pro:

Fixed, thanks.

Thanks, it's working now.


[release/1.6] Downgrade MinGW to version 10.2.0

There is currently an issue in the race detector in Go on Windows when
used with a newer version of GCC. The issue was first reported here:
https://github.com/golang/go/issues/46099
Fixes #7104
Signed-off-by: Gabriel Adrian Samfira<EMAIL_ADDRESS>(cherry picked from commit 1ef4bda43301ef1de059346a9c37a0876eb635f3)
Signed-off-by: Kazuyoshi Kato<EMAIL_ADDRESS>

#7131 was having the following issue.
https://github.com/containerd/containerd/runs/7199569926?check_suite_focus=true#step:9:220
=== Failed
=== FAIL: pkg/cri/server  (0.00s)
==6156==ERROR: ThreadSanitizer failed to allocate 0x000004299000 (69832704) bytes at 0x200ded1054000 (error code: 87)
FAIL	github.com/containerd/containerd/pkg/cri/server	0.263s
DONE 516 tests, 9 skipped, 1 failure in 242.336s
mingw32-make: *** [makefile:206: test] Error 1
Error: Process completed with exit code 2.

@gabriel-samfira does it make sense to backport the change to 1.5 branch as well?

@kzys this will happen in any environment with a mingw version above 10.2.0 so it will need to be backported to all supported branches. Sadly, the default GitHub runners have an updated version of MinGW.

@gabriel-samfira Thanks! Opening #7134 for 1.5.x


RTree Weights

While we do not utilize the RTree weights builder, we still have the code in the code base.  I believe that the logic will fail using the Virginia county test data for Rook contiguity.  It looks like https://github.com/pysal/pysal/blob/master/pysal/weights/_contW_rtree.py#L89 utilizes a set intersection to see if the vertex count is > 1, but a true poly edge to poly edge check is required.
Am I miss reading this code?  If not, do we need to fix since this is essentially deprecated.

Line 38 is a check of a necessary condition for rook. if it passes then the check for the shared segment is then carried out. if the necessary condition is not satisfied, no check for shared segment is carried otu.

Line  91 then makes the check, with 89 checking the precondition...Got it.


Use Typia's JSON transformation in Express application

Question
How do I replace JSON.stringify() and parse() methods from express to typia's methods
currently the api design looks like this
res.json({
    listingDetail: result.metaData,
    constantData: result.constant_data,
});

Express internally used JSON.stringify() when using res.json, How do I override this with typia?

Make a function getting functional parameter.
export const doSomething = <T>(stringify: (input: T) => string) => {};

If you're inconvenient repeating the typia.json.createAssertStringify<T>() function call for every API endpoints, you can use another option.

Use Nestia with NestJS
Make your custom transformer library reading the nestia code

I get the generic function type, it'll make it easier to use everywhere, I don't understand how Nestia and NestJS come into play here.
I am using ExpressJS here, is there any way to use typia.json.stringify<T>() in this response like
res.send(typia.json.stringify<T>({
    listingDetail: result.metaData,
    constantData: result.constant_data,
}));

@samchon

BTW, I came here after reading this article, there is an express example code, but I didn't quite understand how to use it...
https://dev.to/samchon/i-made-express-faster-than-fastify-4h8g

The benchmark was done by manual mapping through typia.json.createAssertStringify<T>() functions.
Also, about the nestia case, you have to read its implementation code to follow the same way.
In personally, I just recommend to use the nestia.

Will look into it, thanks


install.sh referenced repo updates

Description
The following repo references in install.sh should be replaced with official keylime repo locations
KEYLIME_GIT=https://github.com/mit-ll/keylime.git
TPM4720_GIT=https://github.com/mit-ll/tpm4720-keylime.git

Taking this one as its fixed in the coming Python 3 port:
https://github.com/keylime/keylime/pull/126

Done, thanks for pointing this out @atothRedHat. I also see that the copyright mention in installer.sh is still:

Copyright 2017 Massachusetts Institute of Technology.

Might want to update that.


Tab is equal to N spaces not n. Why does vscode behavior for tab spaces not working?

Steps to Reproduce:

Modified my settings.json and change between 2 if I use javascript and 4 for C++
But everytime I modify my settings.json why does tab spaces set to 4 even I modify it to 2 spaces?

  "editor.insertSpaces": true,
  // When opening both `editor.tabSize` and `editor.insertSpaces`
  // will be detected and overridden based on the file contents
  // set to false to keep the values I explicitly set, above.
  "editor.detectIndentation": false,
  // The number of spaces a tab is equal to 2 spaces not 4
  "editor.tabSize": 2,

Does this issue occur when all extensions are disabled?: Yes/No

@kirbygit VS Code settings can be defined per folder, so it is possible that there is a .vscode/settings.json file in the folder you open that will override the user configured editor.tabSize.
Does the issue reproduce for you when opening a new empty folder ?

@alexdima so everytime I switch to a another defined per project folder. I check if my settings.json and the editor.tabSize is the correct number of spaces. Then tried to Reload Window just to make sure settings.json changes is noticed. Then tried pressing tab:
ex.
Expected behavior - 2 spaces
Unintended - 4 spaces

@kirbygit Did you disable all your extensions? There is one extension in particular named .editorconfig which takes ownership of the editor configuration once installed.

@alexdima I don't think I have .editorconfig. Here is my settings.json
// Current settings on my makefile on github
{
  "editor.fontFamily": "'Fira Code', 'Droid Sans Mono', 'monospace', monospace, 'Droid Sans Fallback'",
  "editor.fontLigatures": true,
  "editor.formatOnSave": false,
  "editor.suggestSelection": "first",
  "editor.wordWrap": "on",
  "files.autoSave": "afterDelay",
  "files.exclude": {
    "**/.classpath": true,
    "**/.project": true,
    "**/.settings": true,
    "**/.factorypath": true
  },
  "git.autofetch": true,
  "terminal.integrated.shellArgs.windows": [
    "--login",
    "-i"
  ],
  "java.home": "/home/sean/.jenv/versions/11.0",
  "python.dataScience.sendSelectionToInteractiveWindow": false,
  "python.jediEnabled": false,
  "vsintellicode.modify.editor.suggestSelection": "automaticallyOverrodeDefaultValue",
  "window.zoomLevel": 0,
  "peacock.favoriteColors": [
    {
      "name": "Angular Red",
      "value": "#b52e31"
    },
    {
      "name": "Auth0 Orange",
      "value": "#eb5424"
    },
    {
      "name": "Azure Blue",
      "value": "#007fff"
    },
    {
      "name": "C# Purple",
      "value": "#68217A"
    },
    {
      "name": "Gatsby Purple",
      "value": "#639"
    },
    {
      "name": "Go Cyan",
      "value": "#5dc9e2"
    },
    {
      "name": "Java Blue-Gray",
      "value": "#557c9b"
    },
    {
      "name": "JavaScript Yellow",
      "value": "#f9e64f"
    },
    {
      "name": "Mandalorian Blue",
      "value": "#1857a4"
    },
    {
      "name": "Node Green",
      "value": "#215732"
    },
    {
      "name": "React Blue",
      "value": "#00b3e6"
    },
    {
      "name": "Something Different",
      "value": "#832561"
    },
    {
      "name": "Vue Green",
      "value": "#42b883"
    }
  ],
  "workbench.iconTheme": "material-icon-theme"
}

{
  // Disable welcome screen on startup
  "workbench.welcome.enabled": false,
  // Controls if the diff editor shows the diff side by side or inline
  "diffEditor.renderSideBySide": false,
  // Insert spaces when pressing Tab
  "editor.insertSpaces": true,
  // When opening both `editor.tabSize` and `editor.insertSpaces`
  // will be detected and overridden based on the file contents
  // set to false to keep the values I explicitly set, above.
  "editor.detectIndentation": false,
  // The number of spaces a tab is equal to 2 spaces not 4
  "editor.tabSize": 2,
  // Minimap
  "editor.minimap.enabled": true,
  // A multiplier to be used on the `deltaX` and `deltaY` of mouse wheel scroll events
  "editor.mouseWheelScrollSensitivity": 5,
  // Controls whether the editor should render whitespace characters
  "editor.renderWhitespace": "all",
  // Extensions to be installed by the Extension Manager
  "extension-manager.extensions": [
    "eg2.tslint", // TypeScript linter
    "ms-vscode.cpptools", // Cpp tools
    "msjsdiag.debugger-for-chrome", // Debugger for Chrome
    "robertohuertasm.vscode-icons" // Cool icons for files
  ],
  // When enabled, will trim trailing whitespace when you save a file
  "files.trimTrailingWhitespace": true,
  // Default end of line character
  "files.eol": "\n",
  // Glob patterns for excluding folders and files
  "files.exclude": {
    "**/*.build": true,
    "**/*.git": true,
    "**/*.err": true,
    "**/*.rec": true,
    "**/*.log": true,
    "**/*.evt": true,
    "**/buildfre.*": true,
    "**/buildchk.*": true,
    "**/.DS_Store": true
  },
  // Glob patterns for excluding files and folders in searches
  "search.exclude": {
    "**/node_modules": true,
    "**/bower_components": true,
    ".build/**": true,
    "out*/**": true,
    "i18n/**": true,
    "extensions/**/out/**": true,
    "*.min.js": true
  },
  // Enable linter for TypeScript files
  "tslint.enable": true,
  "files.associations": {
    "deque": "cpp",
    "list": "cpp",
    "vector": "cpp",
    "xstring": "cpp"
  }
}

@kirbygit You have two JSON objects in there i.e.
{
"some settings"
}

{
"some more settings"
}

this does not work, you need to remove the extraneous }\n\n{ from the middle of your settings.

@alexdima hmm hold on, not rendered with squiggles at all

@alexdima ok I see what you mean about .editorconfig. I did not create a symlink for that yet. Thanks for pointing out that the two json objects

@kirbygit  OK. So everything works fine now?

@alexdima Thank you, all good now


Mini RFC: Switch all linting warnings to errors

Problem

Currently we have several ESLint plugins that only set checks to warnings instead of errors.
Warnings can be easily ignored while coding, but they show up on every PR if the warning gets merged.
What's the use case for an ESLint warning? If something is intentionally going against the ESLint rules, shouldn't we have to force the use of eslint-disable for the rule for the CI to pass? Otherwise the warnings are just pointless noise.
Particularly worrisome ones are for example the warning not to use @ts-ignore because it modifies compilation. @ts-ignore is a powerful tool that is useful for specific situations, but should be avoided whenever possible. If it's used, then an explanation for why would be appropriate.
Description

Should we switch all ESLint rules that are currently warnings to errors, therefore requiring the use of eslint-disable?
cc @WordPress/openverse-developers
Implementation

[ ] 🙋 I would be interested in implementing this feature.

I like the idea of not introducing any additional warnings in the PRs. The warnings might seem tiny in one PR, but when we let them build up, they really degrade the experience of looking through the logs.
One issue with the approach you suggest, @sarayourfriend , is that there were a couple of ESLint warnings that I couldn't remove:

two components in one file in a test file (two wrapper components)
unknown component in a test file: I don't know how to ensure that Vue sees a component inside a test wrapper component as registered.
I remember a couple of times when I didn't know how to disable warnings with eslint-disable, it didn't work for some reason, and the warnings still appeared.

Considering that both my examples are from test files, maybe we should switch warnings to errors in the src/ directory, and leave them in tests?

It looks like you were able to disable the two-components in one file warning: https://github.com/WordPress/openverse-frontend/blob/e8cab9fa310002f175f50705551e6c020a3a91d0/test/unit/specs/components/v-link.spec.js#L12
Is there somewhere else it was an issue?
I'm not sure about the unknown component in a test file one, I'm not able to reproduce that one. If we can reproduce it we can debug why it's not able to be disabled.
But you bring up a good point, some errors don't make sense in test scenarios. For the multiple components in a single file (common when writing test wrappers) we could just disable that rule for the test directory using the test/unit/eslintrc.js.

@WordPress/openverse-frontend Are we fine implementing this? We've got yet another batch of ESLint warnings polluting every PR now (raw text in test components, which is not a valid warning). If they'd been errors (as that one in particular should be for the actual application code) then we would have caught the regression in how ESLint rules are applied to different parts of the codebase before committing it to main and adding a long list of noise (that you still have to pick through to make sure there isn't anything actually valid in them) to every single PR.

@WordPress/openverse-frontend please 👍 / 👎 your support for this in @sarayourfriend's comment and offer additional feedback if you don't want to move forward.

I very much support this, really don't like to have warnings polluting the terminal or the GitHub UI, and I think both problems @obulat mentioned have been fixed in PR #1221, tests were passing when Olga approved it but broke when merging with main. I'll check what happened there. It just needs another approval.
I also sent another to get rid of warning annotations on test files, please check it out #1252.


[Request] Storage dialog box needs to be resizable

Enough said.  Needed for long key values

Since its a complex form with lots of stuff, do you know of a good way to make it resizable without having to handle resizing all controls in resize event?

The storage viewer has just the listview.
From: Sérgio Flores<EMAIL_ADDRESS>Sent: February 22, 2018 9:44 AM
To: CityOfZion/neo-debugger-tools<EMAIL_ADDRESS>Cc: Michael Herman (Parallelspace<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [CityOfZion/neo-debugger-tools] [Request] Storage dialog box needs to be resizable (#44)
Since its a complex form with lots of stuff, do you know of a good way to make it resizable without having to handle resizing all controls in resize event?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/CityOfZion/neo-debugger-tools/issues/44#issuecomment-367702344, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AF0a6GjJqz9tLIIcFJR8oYoyrAEP8xuOks5tXX1HgaJpZM4SPa9q.

Yes true, I was thinking of the Invoke Dialog. Then it is simple to do :)

Closed, as already resizable


error fetching geometry type for layer (land_0-9): ERROR: relation "simplified_land_polygons" does not exist (SQLSTATE 42P01)

v0.4.0-rc2.  Just following steps in the README.

@erictheise did you bring in the Natural Earth dataset?

I suspected that was the problem but I'm only interested in OSM and the README says

Import the Natural Earth dataset (requires gdal. can be skipped if you're only interested in OSM)

Perhaps the meaning of "interested in OSM" should be spelled out?

Appears that in addition to running natural_earth.sh you also need to run the undocumented osm_land.sh script for this to have any chance of working.

@erictheise good point. maybe we need a few different tegola.toml files for the various combinations. i.e. tegola-natural-earth.toml, tegola-osm.toml, tegola-osm-natural-earth.toml

I updated the readme file to mention running the osm_land.sh file. I'm going to close this issue for now. I think we can spin up a different issue for the various config files. @PetersonGIS is already working on updating them for Natural Earth v4.


ubsan: jerryscript/jerry-core/vm/vm.c:3731:77: runtime error: left shift of 33554431 by 22 places cannot be represented in type 'int'

JerryScript revision

ccc1525
Build platform

Ubuntu 20.04.1 LTS (Linux 5.4.0-74-generic x86_64)
Build steps

readonly FLAGS=(
  "-g -O1 -fsanitize=address,undefined -fno-sanitize-recover=undefined,address"
)
cmake -D CMAKE_C_FLAGS="${FLAG}" -D CMAKE_BUILD_TYPE=Debug -D CMAKE_INSTALL_PREFIX:PATH="${INSTALL_DIR}" "${SRC_FOLDER}"
make
make install

Test case

33554431 << 22

Output

jerryscript/jerry-core/vm/vm.c:3731:77: runtime error: left shift of 33554431 by 22 places cannot be represented in type 'int'

Expected behavior

According to C99 standard (6.5.7/4):

The result of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are filled with
zeros. If E1 has an unsigned type, the value of the result is E1 × 2 E2 , reduced modulo
one more than the maximum value representable in the result type. If E1 has a signed
type and nonnegative value, and E1 × 2 E2 is representable in the result type, then that is
the resulting value; otherwise, the behavior is undefined.

A slightly different ub will be triggered in the same line with the following test case
- 1 << 1

the undefined behavior:
runtime error: left shift of negative value -1


fix: loading state added for generate crud CTA on datasource preview

Description
This PR adds a loading state to generate CRUD CTA on datasource preview page, so that a user cannot click on generate crud multiple times and run into race condition. Clicking on this CTA multiple times was causing 400 bad request error to be thrown by server. This PR fixes that issue by adding a loading state.
PR fixes following issue(s)
Fixes #30373
Media

A video or a GIF is preferred. when using Loom, don’t embed because it looks like it’s a GIF. instead, just link to the video

Type of change

Bug fix (non-breaking change which fixes an issue)

Testing

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Also list any relevant details for your test configuration.
Delete anything that is not relevant

[x] Manual
[ ] JUnit
[ ] Jest
[ ] Cypress

Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Speedbreak features have been covered
[ ] Test plan covers all impacted features and areas of interest
[ ] Test plan has been peer reviewed by project stakeholders and other QA members
[ ] Manually tested functionality on DP
[ ] We had an implementation alignment call with stakeholders post QA Round 2
[ ] Cypress test cases have been added and approved by SDET/manual QA
[ ] Added Test Plan Approved label after Cypress tests were reviewed
[ ] Added Test Plan Approved label after JUnit tests were reviewed

/ok-to-test<EMAIL_ADDRESS>

/build-deploy-preview skip-tests=true


Can't Access Interceptor(s)

I need to get the value of one of my outInterceptors properties, an audit id.
I tried to access the interceptor through the cxf client by casting the service to JaxWsClientProxy but this failed even though this is the class the debugger in IntelliJ shows. I can navigate the IntelliJ debugger to see the info I need but I'm unable to access it programatically.
Is this possible?
Suggestions?

More specifically, what are you trying to do with the value?

I figured out how to access the information I needed.


Scala API for restoring delta table

Add possibility to restore delta table using version or timestamp.
Examples:
io.delta.tables.DeltaTable.forPath("/some_delta_path").restoreToVersion(1)
io.delta.tables.DeltaTable.forPath("/some_delta_path").restoreToTimestamp("2021-01-01 00:00:00.000")
io.delta.tables.DeltaTable.forPath("/some_delta_path").restoreToTimestamp("2021-01-01")

Fixes https://github.com/delta-io/delta/issues/632
Signed-off-by: Maksym Dovhal<EMAIL_ADDRESS>Tested locally using spark-shell
sbt package
spark-shell --jars ./core/target/scala-2.12/delta-core_2.12-1.1.0-SNAPSHOT.jar --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

spark.range(2).write.format("delta").mode("overwrite").save("/tmp/delta_restore_test")
spark.range(2,3).withColumnRenamed("id", "id_new").write.option("mergeSchema", "true").format("delta").mode("overwrite").save("/tmp/delta_restore_test")
io.delta.tables.DeltaTable.forPath("/tmp/delta_restore_test").restoreToVersion(0)
io.delta.tables.DeltaTable.forPath("/tmp/delta_restore_test").restoreToTimestamp("2021-12-18 16:40:14.54")
// At next day
io.delta.tables.DeltaTable.forPath("/tmp/delta_restore_test").restoreToVersion(0)
io.delta.tables.DeltaTable.forPath("/tmp/delta_restore_test").restoreToTimestamp("2021-12-19")
io.delta.tables.DeltaTable.forPath("/tmp/delta_restore_test").history().show(false)

Output:
+-------+-----------------------+------+--------+---------+------------------------------------------------------+----+--------+---------+-----------+--------------+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------------------------------------+
|version|timestamp              |userId|userName|operation|operationParameters                                   |job |notebook|clusterId|readVersion|isolationLevel|isBlindAppend|operationMetrics                                                                                |userMetadata|engineInfo                                  |
+-------+-----------------------+------+--------+---------+------------------------------------------------------+----+--------+---------+-----------+--------------+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------------------------------------+
|5      |2021-12-19 09:43:41.604|null  |null    |RESTORE  |{version -> null, timestamp -> 2021-12-19}            |null|null    |null     |4          |Serializable  |false        |{numRestoredFiles -> 2, removedFilesSize -> 1252, numRemovedFiles -> 3, restoredFilesSize -> 794, numOfFilesAfterRestore -> 2, tableSizeAfterRestore -> 794} |null        |Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT|
|4      |2021-12-19 09:43:17.415|null  |null    |RESTORE  |{version -> 0, timestamp -> null}                     |null|null    |null     |3          |Serializable  |false        |{numRestoredFiles -> 3, removedFilesSize -> 794, numRemovedFiles -> 2, restoredFilesSize -> 1252, numOfFilesAfterRestore -> 3, tableSizeAfterRestore -> 1252}|null        |Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT|
|3      |2021-12-18 16:42:14.083|null  |null    |RESTORE  |{version -> null, timestamp -> 2021-12-18 16:40:14.54}|null|null    |null     |2          |Serializable  |false        |{numRestoredFiles -> 2, removedFilesSize -> 1252, numRemovedFiles -> 3, restoredFilesSize -> 794, numOfFilesAfterRestore -> 2, tableSizeAfterRestore -> 794} |null        |Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT|
|2      |2021-12-18 16:40:53.861|null  |null    |RESTORE  |{version -> 0, timestamp -> null}                     |null|null    |null     |1          |Serializable  |false        |{numRestoredFiles -> 3, removedFilesSize -> 794, numRemovedFiles -> 2, restoredFilesSize -> 1252, numOfFilesAfterRestore -> 3, tableSizeAfterRestore -> 1252}|null        |Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT|
|1      |2021-12-18 16:40:14.54 |null  |null    |WRITE    |{mode -> Overwrite, partitionBy -> []}                |null|null    |null     |0          |Serializable  |false        |{numFiles -> 2, numOutputRows -> 1, numOutputBytes -> 794}                                                                                |null        |Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT|
|0      |2021-12-18 16:40:08.045|null  |null    |WRITE    |{mode -> Overwrite, partitionBy -> []}                |null|null    |null     |null       |Serializable  |false        |{numFiles -> 3, numOutputRows -> 2, numOutputBytes -> 1252}                                                                                |null        |Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT|
+-------+-----------------------+------+--------+---------+------------------------------------------------------+----+--------+---------+-----------+--------------+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------------------------------------+

Examples of transactions:
/tmp/delta_restore_test/_delta_log/00000000000000000000.json
{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}
{"metaData":{"id":"b090f082-f927-4372-9537-9623ae280ad8","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1639838404635}}
{"add":{"path":"part-00000-cb74dd35-ae80-4b3a-b97c-ea492e11ddc3-c000.snappy.parquet","partitionValues":{},"size":296,"modificationTime":1639838406641,"dataChange":true}}
{"add":{"path":"part-00005-beac50f7-dbe7-40b7-9ce2-5e0e6a1607ad-c000.snappy.parquet","partitionValues":{},"size":478,"modificationTime":1639838406642,"dataChange":true}}
{"add":{"path":"part-00011-7e35258f-a724-43f3-8622-c7efa51f01a6-c000.snappy.parquet","partitionValues":{},"size":478,"modificationTime":1639838406642,"dataChange":true}}
{"commitInfo":{"timestamp":1639838407868,"operation":"WRITE","operationParameters":{"mode":"Overwrite","partitionBy":"[]"},"isolationLevel":"Serializable","isBlindAppend":false,"operationMetrics":{"numFiles":"3","numOutputRows":"2","numOutputBytes":"1252"},"engineInfo":"Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT"}}

/tmp/delta_restore_test/_delta_log/00000000000000000001.json
{"metaData":{"id":"b090f082-f927-4372-9537-9623ae280ad8","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"id_new\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1639838404635}}
{"add":{"path":"part-00000-b71c3566-429b-4038-8127-6e480656038c-c000.snappy.parquet","partitionValues":{},"size":304,"modificationTime":1639838414041,"dataChange":true}}
{"add":{"path":"part-00011-7a5341e6-4876-467a-b33d-56d8dc0bf243-c000.snappy.parquet","partitionValues":{},"size":490,"modificationTime":1639838414041,"dataChange":true}}
{"remove":{"path":"part-00000-cb74dd35-ae80-4b3a-b97c-ea492e11ddc3-c000.snappy.parquet","deletionTimestamp":1639838414511,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":296}}
{"remove":{"path":"part-00011-7e35258f-a724-43f3-8622-c7efa51f01a6-c000.snappy.parquet","deletionTimestamp":1639838414511,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":478}}
{"remove":{"path":"part-00005-beac50f7-dbe7-40b7-9ce2-5e0e6a1607ad-c000.snappy.parquet","deletionTimestamp":1639838414511,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":478}}
{"commitInfo":{"timestamp":1639838414511,"operation":"WRITE","operationParameters":{"mode":"Overwrite","partitionBy":"[]"},"readVersion":0,"isolationLevel":"Serializable","isBlindAppend":false,"operationMetrics":{"numFiles":"2","numOutputRows":"1","numOutputBytes":"794"},"engineInfo":"Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT"}}

/tmp/delta_restore_test/_delta_log/00000000000000000002.json
{"commitInfo":{"timestamp":1639838436332,"operation":"RESTORE","operationParameters":{"version":0,"timestamp":null},"readVersion":1,"isolationLevel":"Serializable","isBlindAppend":false,"operationMetrics":{"numRestoredFiles":"3","removedFilesSize":"794","numRemovedFiles":"2","restoredFilesSize":"1252","numOfFilesAfterRestore":"3","tableSizeAfterRestore":"1252"},"engineInfo":"Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT"}}
{"metaData":{"id":"b090f082-f927-4372-9537-9623ae280ad8","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1639838404635}}
{"add":{"path":"part-00005-beac50f7-dbe7-40b7-9ce2-5e0e6a1607ad-c000.snappy.parquet","partitionValues":{},"size":478,"modificationTime":1639838406642,"dataChange":true}}
{"add":{"path":"part-00011-7e35258f-a724-43f3-8622-c7efa51f01a6-c000.snappy.parquet","partitionValues":{},"size":478,"modificationTime":1639838406642,"dataChange":true}}
{"add":{"path":"part-00000-cb74dd35-ae80-4b3a-b97c-ea492e11ddc3-c000.snappy.parquet","partitionValues":{},"size":296,"modificationTime":1639838406641,"dataChange":true}}
{"remove":{"path":"part-00011-7a5341e6-4876-467a-b33d-56d8dc0bf243-c000.snappy.parquet","deletionTimestamp":1639838435578,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":490}}
{"remove":{"path":"part-00000-b71c3566-429b-4038-8127-6e480656038c-c000.snappy.parquet","deletionTimestamp":1639838435586,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":304}}

/tmp/delta_restore_test/_delta_log/00000000000000000003.json
{"commitInfo":{"timestamp":1639838517073,"operation":"RESTORE","operationParameters":{"version":null,"timestamp":"2021-12-18 16:40:14.54"},"readVersion":2,"isolationLevel":"Serializable","isBlindAppend":false,"operationMetrics":{"numRestoredFiles":"2","removedFilesSize":"1252","numRemovedFiles":"3","restoredFilesSize":"794","numOfFilesAfterRestore":"2","tableSizeAfterRestore":"794"},"engineInfo":"Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT"}}
{"metaData":{"id":"b090f082-f927-4372-9537-9623ae280ad8","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"id_new\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1639838404635}}
{"add":{"path":"part-00011-7a5341e6-4876-467a-b33d-56d8dc0bf243-c000.snappy.parquet","partitionValues":{},"size":490,"modificationTime":1639838414041,"dataChange":true}}
{"add":{"path":"part-00000-b71c3566-429b-4038-8127-6e480656038c-c000.snappy.parquet","partitionValues":{},"size":304,"modificationTime":1639838414041,"dataChange":true}}
{"remove":{"path":"part-00005-beac50f7-dbe7-40b7-9ce2-5e0e6a1607ad-c000.snappy.parquet","deletionTimestamp":1639838516199,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":478}}
{"remove":{"path":"part-00011-7e35258f-a724-43f3-8622-c7efa51f01a6-c000.snappy.parquet","deletionTimestamp":1639838516199,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":478}}
{"remove":{"path":"part-00000-cb74dd35-ae80-4b3a-b97c-ea492e11ddc3-c000.snappy.parquet","deletionTimestamp":1639838516202,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":296}}

/tmp/delta_restore_test/_delta_log/00000000000000000004.json
{"commitInfo":{"timestamp":1639899780668,"operation":"RESTORE","operationParameters":{"version":0,"timestamp":null},"readVersion":3,"isolationLevel":"Serializable","isBlindAppend":false,"operationMetrics":{"numRestoredFiles":"3","removedFilesSize":"794","numRemovedFiles":"2","restoredFilesSize":"1252","numOfFilesAfterRestore":"3","tableSizeAfterRestore":"1252"},"engineInfo":"Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT"}}
{"metaData":{"id":"b090f082-f927-4372-9537-9623ae280ad8","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1639838404635}}
{"add":{"path":"part-00005-beac50f7-dbe7-40b7-9ce2-5e0e6a1607ad-c000.snappy.parquet","partitionValues":{},"size":478,"modificationTime":1639838406642,"dataChange":true}}
{"add":{"path":"part-00011-7e35258f-a724-43f3-8622-c7efa51f01a6-c000.snappy.parquet","partitionValues":{},"size":478,"modificationTime":1639838406642,"dataChange":true}}
{"add":{"path":"part-00000-cb74dd35-ae80-4b3a-b97c-ea492e11ddc3-c000.snappy.parquet","partitionValues":{},"size":296,"modificationTime":1639838406641,"dataChange":true}}
{"remove":{"path":"part-00011-7a5341e6-4876-467a-b33d-56d8dc0bf243-c000.snappy.parquet","deletionTimestamp":1639899779981,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":490}}
{"remove":{"path":"part-00000-b71c3566-429b-4038-8127-6e480656038c-c000.snappy.parquet","deletionTimestamp":1639899779981,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":304}}

/tmp/delta_restore_test/_delta_log/00000000000000000005.json
{"commitInfo":{"timestamp":1639899805962,"operation":"RESTORE","operationParameters":{"version":null,"timestamp":"2021-12-19"},"readVersion":4,"isolationLevel":"Serializable","isBlindAppend":false,"operationMetrics":{"numRestoredFiles":"2","removedFilesSize":"1252","numRemovedFiles":"3","restoredFilesSize":"794","numOfFilesAfterRestore":"2","tableSizeAfterRestore":"794"},"engineInfo":"Apache-Spark/3.2.0 Delta-Lake/1.1.0-SNAPSHOT"}}
{"metaData":{"id":"b090f082-f927-4372-9537-9623ae280ad8","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"id_new\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1639838404635}}
{"add":{"path":"part-00011-7a5341e6-4876-467a-b33d-56d8dc0bf243-c000.snappy.parquet","partitionValues":{},"size":490,"modificationTime":1639838414041,"dataChange":true}}
{"add":{"path":"part-00000-b71c3566-429b-4038-8127-6e480656038c-c000.snappy.parquet","partitionValues":{},"size":304,"modificationTime":1639838414041,"dataChange":true}}
{"remove":{"path":"part-00005-beac50f7-dbe7-40b7-9ce2-5e0e6a1607ad-c000.snappy.parquet","deletionTimestamp":1639899805448,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":478}}
{"remove":{"path":"part-00011-7e35258f-a724-43f3-8622-c7efa51f01a6-c000.snappy.parquet","deletionTimestamp":1639899805445,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":478}}
{"remove":{"path":"part-00000-cb74dd35-ae80-4b3a-b97c-ea492e11ddc3-c000.snappy.parquet","deletionTimestamp":1639899805444,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":296}}

Hi @Maks-D, thanks for this PR! We will take a look and get back to you.

This is fantastic! Thank you for building this! I am going to take a look at this soon.

@tdas  Thank you for review! I am going to solve requested changes soon.

@tdas I've fixed requested changes.
Except this I decided to refactor RestoreTableCommand.run and now method uses txn.snapshot as a latest available snapshot.

other than the cache part.. this looks good.

@Maks-D I still dont understand the need for caching for the filesToRemove. When partitions are computed one-by-one when toLocalIterator is consumed, data produced by each partition will get cached but never reused, as each partition is going to be read-only once.

@tdas
I can't say that this cache has sense in practice, because I don't have access to the delta tables with millions of files.
More over, for computation of fileToRemove and filesToAdd we use already cached snapshots and maybe we can skip caching at all.
I want to describe this decision (maybe wrong and without sense):
Initially I found Note in toLocalIterator:
Note: this results in multiple Spark jobs, and if the input Dataset is the result of a wide transformation (e.g. join with different partitioners), to avoid recomputing the input Dataset should be cached first.

I decided to look into the code deeper and found that toLocalIterator calls org.apache.spark.rdd.RDD#toLocalIterator and this RDD method triggers sc.runJob (my understanding this is action) for each partition. Due to possible join with shuffle (in the worst case) we need to re-join data to get the data for a specific partition, because for each partition we have a separate action on lazy RDD.
For broadcast join (I think it will be the case for the most of restore operations) this has no sense, because data in partitions depends on incoming data and we can easily get this data without any additional computation.
Maybe my understanding is wrong and I don't take into account some spark optimisation :(.
If in the real life spark doesn't need to recompute RDD for each partition, we definitely don't need cache there and I will be happy remove it. Also, in this case I can execute filesToAdd.cache only if ignoreMissing is false and that will be one more optimisation.

@tdas Could you please review my comment related to caching https://github.com/delta-io/delta/pull/863#issuecomment-999406804 . If my understanding of toLocalIterator is wrong and cache is unnecessary, I can remove it.
Just want to finish this PR and merge new functionality to oss delta.
Thank you!

@Maks-D my apologies for the massive delay. Here are the updates from my side.

I have started the process of merging this PR. It should get merged by tomorrow. I dont want to block the PR over the cache() discussion, we acn always update it.
Regarding the cache, here is what I think will happen. Each instance of runJob started by the localIterator should reuse the shuffle output between them. That is, first call of the toLocalIterator is going to trigger the map side of the shuffle which will produce all the map outputs in the local disks of the executors. The reduce side of the first runJob will however read only the map output parts needed to full the partition(s) to be computed (e.g., map output only for reducer 1). Subsequent runJobs will reuse those map output from the first runJob, but read a different  parts of that map output (e.g., map output for reducer 2). So the map side of the shuffle should not be re-triggered for every run job, and every run job should reuse pre-computed map output. So I dont think cache makes a huge performance improvement here. Rather it carries a much larger risk of perf degradation due to memory spilling, memory cache invalidation, etc.

If you are good with this suggestion, i suggest you make a follow up PR (after this one is merged) to:

remove the cache
add python API (would complete the programmatic APIs)
what do you think?

@tdas Thank you for detailed explanation!
I agree with your proposals 👍
I've already created https://github.com/delta-io/delta/issues/890 for python API and I will remove the cache.


Multiple Menus: Different Breakpoints

By using multiple menus the use of different breakpoints caused problems.
With these changes each menu can have a different breakpoint.
JS:
// New init Option
$(".flexnav").flexNav({
'breakpoint': 'mybreakpoint', // data name for breakpoint
});
HTML:

found a better way


I can't run newest version - 1.6.2 Linux 64

Hello,
I faced following issue while trying to run Bob 1.6.2 for Linux 64:

Error executing boot module $:/plugins/OokTech/Bob/FileSystemMonitor.js: {}
$:/plugins/OokTech/Bob/FileSystemMonitor.js:24
if($tw.node && $tw.settings.disableFileWatchers !== 'yes') {
^
TypeError: Cannot read property 'disableFileWatchers' of undefined
at $:/plugins/OokTech/Bob/FileSystemMonitor.js:24:29
at $:/plugins/OokTech/Bob/FileSystemMonitor.js:229:3
at Script.runInContext (vm.js:127:20)
at Script.runInNewContext (vm.js:133:17)
at Object.runInNewContext (vm.js:299:38)
at Object.$tw.utils.evalSandboxed (/home/slawek/mnt/data/sandbox/tiddlywiki-testy/upgrade-test/TiddlyWiki5

/boot/boot.js:530:5)
at Object.$tw.modules.execute (/home/slawek/mnt/data/sandbox/tiddlywiki-testy/upgrade-test/TiddlyWiki5/boot/boot.js:806:15)
at /home/slawek/mnt/data/sandbox/tiddlywiki-testy/upgrade-test/TiddlyWiki5/boot/boot.js:846:30
at Object.$tw.utils.each (/home/slawek/mnt/data/sandbox/tiddlywiki-testy/upgrade-test/TiddlyWiki5/boot/boot.js:135:12)
at Object.$tw.modules.forEachModuleOfType (/home/slawek/mnt/data/sandbox/tiddlywiki-testy/upgrade-test/TiddlyWiki5/boot/boot.js:845:12)

I haven't been able to get this error. I fixed what is causing the problem for you, but if you could give me more information about your setup so I can try and prevent similar things in the future.
I will be releasing the next version sometime this next week.

Cool, thanks, I will try new release in next week. What kind of data do you need?

I have the same issue both on Debian Buster 10.6 with kernel 4.19.0-12-amd64 and Fedora 32 with kernel 5.8.17-200.fc32.x86_64.
I downloaded the 1.6.2r2 executable and ran it in my existing TW5-Bob wiki, but it exited with the same error. I also tried running it in an empty directory and it created the default wiki files, but then crashed again.
I would be happy to provide any additional information if that would help.

Same here on Windows 10 with a fresh manual installation and on the default index wiki but with the error :
$:/plugins/OokTech/Bob/ServerSide.js:813
  if(data.decoded) {
          ^
TypeError: Cannot read property 'decoded' of undefined

This is a plain manual installation with the two git clones and just accessing the index for the first time. Node.js version is v14.15.0

I think that these problems should be fixed in the new release that is about to come out. I haven't been able to replicate them so I can not be certain. Please let me know one way or the other once I do the release.

I tried the new version 1.7.0b1 release and it did not crash like 1.6.2r2, so thank you for the update.


StacRasterSources improvements

Overview
Checklist

[x] Description of PR is in an appropriate section of the CHANGELOG and grouped with similar changes if possible

Closes #340
Closes #326

@jisantuc ): sry I bumped the scalafmt version up and that led to unpredictable results...

@jisantuc eh yea, well it kind of makes sense to move it out of this repo, but from the other hand it is only used by projects within this repo o: 8 minutes is not that bad :D


Send back the location of the identifier

Addresses: https://github.com/sourcegraph/go-langserver/issues/25
Allows: https://github.com/sourcegraph/sourcegraph/issues/3322

cc @slimsag can this change cause any issues in the global refs pipeline?


Example Inaccuracies

I have noticed two issues while writing some unit tests using the examples provided in this repository:

nodeapi-flows-get-200.json - The IS-07 flows (as opposed to the other flows in the array) have "caps" fields, although they are not defined for flows. That seems to originate from the IS-07 documentation samples and has to be corrected there too. The fix is to remove the fields.
registrationapi-resource-*.json (3 files affected) - The examples show the update for a node, but the node in the example doesn't have "interfaces" property which is mandatory for a node. The fix is to (at least) add an empty "interfaces" array.
I have the fixes for the above locally, but don't have the rights to push them and create a pull request.

👍 Sounds about right. I also feel the example files haven't had as much scrutiny as the schemas and other parts of the specs.
You should be able to fork this repo, and make a branch based on v1.3.x, and push your changes on that. Then you should be able to submit a PR.
Spec authors can usually then rebase on to v1.2.x, etc. if appropriate.

Closed by #156.


[GSoC2024] resolved issue #7443 (navigate over unresolved issues)

Resolved issue #7443
It skips the frames where all issues have been resolved!
Checklist

[x] I submit my changes into the develop branch
[x] I have created a changelog fragment
[x] I have updated the documentation accordingly
[x] I have added tests to cover my changes
[x] I have linked related issues (see GitHub docs)
[x] I have increased versions of npm packages if it is necessary
(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)

License

[x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.

@klakhov please review this pr,  I've made the required changes

@nmanovic @klakhov a reminder, i request you to review my pr

@thekavikumar , could you please take a look at the failed test?

@nmanovic let me check

@klakhov please check it now

@klakhov by any chance can we run tests locally?

@nmanovic @klakhov If there is no failing tests, I guess its good to merge now. What do you think sir?

@thekavikumar

by any chance can we run tests locally?

You can run tests locally. Refer to the tests documentation. You will need first section with E2E tests.

If there are no failing tests, I guess its good to merge now. What do you think sir?

Please, check my comment above where I've asked you to write a test, its good practice to add one with bugfix. I've shared some ideas about how it should be implemented.

@thekavikumar

by any chance can we run tests locally?

You can run tests locally. Refer to the tests documentation. You will need first section with E2E tests.

If there are no failing tests, I guess its good to merge now. What do you think sir?

Please, check my comment above where I've asked you to write a test, its good practice to add one with bugfix. I've shared some ideas about how it should be implemented.

Yeah will commit with a test written for soon!

I'm getting this error, may I know why @klakhov

@klakhov also I'm not getting what to test, because the change I did is already being tested here  right?

@klakhov can please make me get a clear idea of what to test with the bugfix?

@thekavikumar
Probably you get 400 error because the users are already created in the database from previous runs. Try to run remove_users_tasks_projects_organizations.js to delete them
The test that Ive linked test only basic navigation for frames with issues. It doesnt test filter setting. You need to resolve some issues, enable the filter and check that frames with resolved issues are not shown using the filter

@klakhov okay so in the tests I have to write code to create multiple frames with issues, then resolve the issues in one frame, then try to navigate and check the frame number weather the frame has been skipped or not. Am I right?

@klakhov should I create test inside this itself or should I create a separate describe?

@thekavikumar
I think you can extend the current review test after the part where we test usual issue filter. You can do such steps:

Resolve the issue on the frame in the middle(frame 1) (we have issues on 0, 1, 2 frames)
Enable the filter
Check that the frame 1 is skipped using the filter navigation
Check that full test works because we have resolving issue logic after the filter testing and fix that if needed

@klakhov  how can i get all issues like this

for filtering
let issuesResolvedHidden = true;
// Filter frames based on resolved issues and store them in the 'frames' array
let frames = issues
.filter((issue) => !issuesResolvedHidden || !issue.resolved)
.map((issue) => issue.frame)
.sort((a, b) => a - b);
// Resolve the issue on frame 1
cy.goCheckFrameNumber(1);
cy.collectIssueLabel().then((issueLabelList) => {
for (const label of issueLabelList) {
cy.resolveIssue(label, 'Resolved issue on frame 1');
}
});
issuesResolvedHidden = false;
cy.goCheckFrameNumber(0);
cy.get('.cvat-issues-sidebar-previous-frame')
.should('have.attr', 'style')
.and('contain', 'opacity: 0.5;'); // the element is not active
cy.get('.cvat-issues-sidebar-next-frame').should('be.visible').click();
cy.checkFrameNum(2);

@klakhov does this code look good?

@klakhov also when i run remove_user that test file i get

@klakhov can you please review it

@klakhov i mean review the tests, i have committed it :)

@klakhov sir one more thing, I'm not able to run full tests as because due to some reason when I try to remove users by running the remove_users_tasks_projects_organizations.js
could you please help me with that?

As for invalid token error, please provide mode details. On what step the error happens? Do you run test from admin account?

@klakhov yes I'm using the admin username and password only. i got this error while running

@thekavikumar
Check the id of your admin user that you are using to run tests. Maybe its not 1 as expected in the delete test, so the user gets deleted and token is wrong. Change it inside the test if needed.
Also it seems the review test is failing on the CI.

@klakhov okay let me check

@klakhov please give approval for the workflow

@nmanovic @klakhov is there any possibility to run the entire tests in our local system as it runs in GitHub workflow because each time I have to push the changes then the admin have to approve for the workflow to run.

@nmanovic @klakhov Sorry for asking you again for workflow approval. I've no other options. Please help me with this testing issues

@thekavikumar
I've approved your runs, but I see one problem in the test. After you resolved an issue on frame 1, the code on line 329 is trying to do the same and fails on frame 1 as its already resolved, please fix this.
Also, please try to run the test locally before submitting. Do you still have a problem with remove_users_tasks_projects_organizations.js?

@klakhov 'After you resolved an issue on frame 1, the code on line 329 is trying to do the same and fails on frame 1 as its already resolved, please fix this.' for this I am asking, shall we reopen or shall I create new frames for this?

@klakhov

You dont need to reopen or create new frames. The code on 329 tests issue resolving process, so if the issue is already resolved, we dont need to do anything, we need to skip resolving it.

@klakhov okay let me try it

@klakhov review the recent commit, will this work?

You need to update server dependencies after merging develop

@klakhov how do i update?

@klakhov I'm not sure why this occured, any idea? Also can you approve the workflow once again?

@klakhov the filter test was running good, but the above issue in image was in actions_projects_models, not sure why.

You need to update server dependencies after merging develop

@klakhov @nmanovic may i know how to update the server deps?

@klakhov how do i update? using yarn?

To update server dependencies(link to docs):
pip install -U pip wheel setuptools
pip install -r cvat/requirements/development.txt

@klakhov how do i update? using yarn?

To update server dependencies(link to docs):
pip install -U pip wheel setuptools
pip install -r cvat/requirements/development.txt

Alright that works, thanks


Error occurs when switching game view in editor mode

Color resource is null
UnityEngine.Debug:LogError (object)
FidelityFX.Fsr2Context:DebugCheckDispatch (FidelityFX.Fsr2/DispatchDescription) (at ./Packages/com.unity.postprocessing@3.3.1/PostProcessing/Runtime/FSR2/Fsr2Context.cs:441)
FidelityFX.Fsr2Context:Dispatch (FidelityFX.Fsr2/DispatchDescription,UnityEngine.Rendering.CommandBuffer) (at ./Packages/com.unity.postprocessing@3.3.1/PostProcessing/Runtime/FSR2/Fsr2Context.cs:144)
UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)
Depth resource is null
UnityEngine.Debug:LogError (object)
FidelityFX.Fsr2Context:DebugCheckDispatch (FidelityFX.Fsr2/DispatchDescription) (at ./Packages/com.unity.postprocessing@3.3.1/PostProcessing/Runtime/FSR2/Fsr2Context.cs:446)
FidelityFX.Fsr2Context:Dispatch (FidelityFX.Fsr2/DispatchDescription,UnityEngine.Rendering.CommandBuffer) (at ./Packages/com.unity.postprocessing@3.3.1/PostProcessing/Runtime/FSR2/Fsr2Context.cs:144)
UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)
MotionVectors resource is null
UnityEngine.Debug:LogError (object)
FidelityFX.Fsr2Context:DebugCheckDispatch (FidelityFX.Fsr2/DispatchDescription) (at ./Packages/com.unity.postprocessing@3.3.1/PostProcessing/Runtime/FSR2/Fsr2Context.cs:451)
FidelityFX.Fsr2Context:Dispatch (FidelityFX.Fsr2/DispatchDescription,UnityEngine.Rendering.CommandBuffer) (at ./Packages/com.unity.postprocessing@3.3.1/PostProcessing/Runtime/FSR2/Fsr2Context.cs:144)
UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)

When running in the editor

These errors are fairly harmless by themselves, they are ported over from the original FSR2 codebase and only appear inside the Unity Editor and in development builds, to warn developers if/when something might be wrong with their FSR2 setup. They can sometimes appear in Unity when the view gets reset or a camera is activated or deactivated.
Are you using the latest version of the Post-processing Stack V2 + FSR2 package? That's version 3.3.1d. In it, I did some refactoring on how render target resource references are managed, which also affects these error messages. They should appear less often now.


Update sonar build to only run on aurora editor repo

Update sonar build to only run on aurora editor repo

Thanks for submitting a pull request!
We will review it as soon as possible.

Hey @0xWDG,
Thanks for your contribution.
Your PR has been merged!
Your name will appear on https://auroraeditor.com/#contributors in a few moments, if you have contributed to the editor.
Thanks again for your contribution!
Kind Regards,
Aurora Editor Team.


Travis CI integration

As title, I think we can integrate CI service.
And this work can let every upcoming PRs and commits do tests automatically.

@peter279k I too think this is a great idea and will be looking into it shortly. I am not too familiar with CI yet and want to make sure I am well versed in it.

Hi @jason-napolitano, thanks for your concern.
This PR I can help you and you can assign me to be a collaborator if possible :).

Invitation sent. Thank you for your interest. I have some pretty neat plans of actually converting this into a full PHP exception framework, and you're more than welcome to join =)

I've started the CI process (f6c8181) =D

I've started the Travis CI process but I fully admit that I do not know what I am doing lol. I believe I get the gist of it but still have a lot to learn.

@jason-napolitano, that's great.
If you've already initialized the Travis CI build, you can just send the PR.
And I can review this and suggest some recommendations :).

Ok. I see that it has the Travis CI build now.
I will enhance them and send the PR.
Thanks for your initialized effort, @jason-napolitano 💪 !

Thanks for anything you may like to offer =)
Since my knowledge in CI is still so minimal, it took me a while to get things going through trial-and-error, but I believe I've got the hang of it now. The nightly PHP build does not get satisfied due to PHPUnit ^8, so I left it at PHP 7.3 in the interim in the .travis.yml file.


Can't load Sustain 2020 event page

Clicking on the first event on this page takes you to https://opencollective.com/sustainoss/events/sustain-2020-5874aeeb which errors out. The other events seems to be working.

@alanna I can't reproduce on Chrome/Firefox latest versions. Could you see if you get more details in the browser console?
Also, what kind of error do you get on https://opencollective.com/sustainoss/events/sustain-2020-5874aeeb ?

I fixed it today.


Hint for Rails 7 with capistrano deploy user after #131

To save 30 minutes for the community.

favour #140


Make raw field on literal AST types optional, or remove them

The problem
#7211 brought up a problem. Most of our literal types (NumericLiteral etc) have raw field.
This makes sense when the AST has come from the parser, but little sense when the node is generated (e.g. in transformer), because the node has no "raw" representation.
Having to provide "0" here feels like a hack to me, for example:
https://github.com/oxc-project/oxc/blob/44375a5662ee3e6451f1a5335c75d6379d1878a6/crates/oxc_ast/src/ast_builder_impl.rs#L209-L211
The exception is StringLiteral which doesn't have a raw field. This feels like an omission - if the other literal types have a raw field, it should have one too. But adding one would be a pain, as there are a lot of places we generate strings in transformer etc.
Prior art
Babel's types have the raw field under extra which is optional:
export interface StringLiteral extends BaseNode {
  type: "StringLiteral";
  value: string;
}

interface BaseNode {
  type: Node["type"];
  // ...
  extra?: Record<string, unknown>;
}

StringLiteral BaseNode
Acorn's types also have the raw field as optional:
export interface Literal extends Node {
  type: "Literal"
  value?: string | boolean | null | number | RegExp | bigint
  raw?: string
  regex?: {
    pattern: string
    flags: string
  }
  bigint?: string
}

source
Possible solutions
I can see 2 options:
Make raw fields optional

Change raw field on NumericLiteral etc to Option<&'a str>.
Add a raw: Option<&'a str> field to StringLiteral.

Remove all raw fields
Where an AST node has a non-empty span, the raw value can be obtained by slicing source text. So remove all the raw fields, and use methods to get raw value where required:
impl NumericLiteral {
    pub fn raw<'a>(&self, source_text: &'a str) -> Option<&'a str> {
        if self.span.start == 0 && self.span.end == 0 {
            None
        } else {
            Some(&source_text[self.span.start..self.span.end])
        }
    }
}

(as suggested in #5522)
Which?
Personally I prefer the 2nd. I don't think the raw fields are used much, so they're bloating the AST for little value. The methods to get raw value from Span are pretty trivial and cheap.

I did a try to remove raw fields, but there are some issues...
I think the core problem is the mix purpose of AST, different requirement leads to different design.
For example, for compiling, we do not need any origin data, so the AST could be abstract.
But for linter or formatter, some abilities should not be enabled by default. Like, for this, they might not want 1_0_2_2 be converted to 1_022, at least there should be an option to control it.
Roslyn(Biome and RA too, they refers Roslyn) keeps a lossless syntax tree for this purpose. This design makes their AST great for interreact, although it is not pretty memory friendly(because they need to maintain about 2 trees).
So, I would suggest to choose option 1. Any thoughts? @overlookmotel @Boshen

It is true that transformer/minifier and linter/formatter have different needs. However in this case, I don't think it much affects which option we go for.
For example, in linter the numeric_separators_style rule does need the raw code string. But it can equally easily get that from a raw field, or a raw() method. So either option works for linter. Ditto the formatter, I think.
The raw() method is simple and cheap, so personally I still prefer it.

@Boshen Do you have any opinion on which option? If we can agree, perhaps @ShuiRuTian is willing to work on it?

Boshen prefers raw: Option<&'a str>.

#7393 adds raw field to StringLiteral.
Still to do: Change raw fields on other literal types from Atom<'a> to Option<Atom<'a>>.

@overlookmotel I would love to, but @Boshen already have a PR.
Not sure what I need to do now.

Thanks for your willingness!
Boshen's PR will add raw field to StringLiteral. The remaining task is to make the raw fields on all the other literal types optional.
e.g. NumericLiteral's raw field is currently &'a str but should be Option<Atom<'a>>.
FYI, Atom<'a> and &'a str are basically the same - Atom is just a wrapper around &str:
https://github.com/oxc-project/oxc/blob/199076bd14faefeff42ae263217e165042b2fad1/crates/oxc_span/src/atom.rs#L20
Boshen said he ran into a problem using Option<&'a str> on StringLiteral because ast_tools didn't understand that syntax, so let's go with Option<Atom<'a>> to avoid that problem.
Does that make more sense?

Yeah, it makes things much clearer, thanks


reg analysis server

can this same property be used to retrieve sql analysis servers as well?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cef2d29c-1c1e-f9c9-8224-2e61cef002ae
Version Independent ID: 09c6f52b-7aaa-f311-fb33-7e2fdac7ece3
Content: Azure.SqlServers Property (Microsoft.Azure.Management.Fluent)
Content Source: xml/Microsoft.Azure.Management.Fluent/Azure.xml
Service: service-bus
GitHub Login: @erickson-doug
Microsoft Alias: douge

@NambiVenkatesh No, it's actually for Azure SQL Database, which is a serverless platform-as-a-service offering.


"Skip" doesn't stop audio stream when queue is empty.

If the queue is empty, then "skip" will not end the audio process and instead restart the current song that is playing.

This will be worked on either tomorrow or Sunday.

The bug is resolved in this commit.


Adding plugin to Meteor Cordova project downloads demo folder of >63MB

Hello there,
Thank you so much for this awesome plugin. Works flawlessly.
I just have a little issue.
I'm using Meteor framework to build a mobile app. All cordova plugins download and install in seconds (most of them are below 1MB).
This particular plugin takes a long time to download. Upon closer inspection, I realized there's a demo folder which is downloaded along with the tarball which is main reason for the long download time. This is around 63MB. Deleting this immediately speeded up things, and ofcourse it didn't effect the install. The actual source of this plugin is only around 120Kb, which is really sweet.
Unfortunately, Meteor checks and downloads all Cordova plugins on a clean compile. So I'm left with no option but to wait until the entire 63MB demo folder is downloaded along with the source and then delete it.
I'm not sure how this can be fixed, but if you could take a look, that would be of great help.
Thanks!

How are oyu downloading this? Using bower?

I'm downloading it via the
meteor add cordova:plugin-id command
Its documented here https://github.com/meteor/meteor/wiki/Meteor-Cordova-Phonegap-integration under Using Cordova plugins directly in your application
So the actual command i'm using is
meteor add cordova:org.pbernasconi.progressindicator@https://github.com/pbernasconi/cordova-progressIndicator/tarball/564981cbf5ec0c90b5cb0151c09566bf18b3b6a5


Add option to ignore/swallow errors

It would be handy to mark this task to indicate we dont want s3 failures to tank our build:
  restore_from_cache:
    image: docker.target.com/drone/drone-s3-cache
    pull: true
    restore: true
    required: false // look at me!

note the required: false
For many, this plugin might be used as a convenience to improve build performance. Momentary hiccups in s3 connectivity should not break the build, but simply slow it down.

So there are different opinions... Imho it's totally valid to fail the build if an error happens.

Drone has an option for that https://github.com/drone/drone-yaml/blob/master/yaml/testdata/pipeline.yml#L61
Add failure: ignore.

yes, adding failure: ignore to the step should handle this use case assuming you are using Drone 1.0. @dan-kirberger I think we have this use case covered at the yaml-level so I am going to close this. If there are other use cases we have not considered let me know and we can re-open.

We're not on 1.0 yet but that's the perfect solution. Happy to wait for that. Thanks!


fix(common): LW-8396 fix yarn lint skipping workspaces, enable eslint no-console rule

Checklist

[x] JIRA - LW-8396
[ ] Proper tests implemented
[ ] Screenshots added.

Proposed solution
Currently it's easy to forget a console.log() statement in the PR as no eslint check warns the dev about that, leaving up to the dev/reviewer to spot that. Even though webpack drops console statements from the prod build, it still seems worthwhile to prevent the dev build console from being spammed by such "forgotten" console.log statements.
In this PR I'm re-enabling the no-console eslint rule, keeping it disabled just for console.debug/warn/error/info where it seems reasonable to assume that it was the intent of the developer to log something beyond ad-hoc debugging. I took this PR also as a chance to change the existing console.log instances to appropriate console.error or console.info depending on the apparent context around
Additionally, while enabling the no-console rule I noticed that yarn lint doesn't execute any linting for the underlying workspaces which is also fixed in this PR. The issue seems to have been caused by the R flag in the underlying command:
yarn workspaces foreach -ptRv run lint
As per the docs the R flag runs the recursion for the underlying "dependencies", not "workspaces", resulting in an empty execution as the relevant workspaces are not listed as "dependencies":

If -R,--recursive is set, Yarn will find workspaces to run the command on by
recursively evaluating dependencies and devDependencies fields, instead of
looking at the workspaces fields.

Testing

check that CI passes
try adding a console.log in the codebase and check that it's caught by yarn lint
check that yarn lint executes properly (i.e. for all workspaces)

Allure report
allure-report-publisher generated test report!

smokeTests: ❌ test report for 9a3903bc

passed
failed
skipped
flaky
total
result

Total
29
3
0
0
32
❌

I noticed yarn lint started failing on CI with the following error

[@lace/browser-extension-wallet]: Error: packages/ui/.eslintrc.js » plugin:unicorn/recommended:
[@lace/browser-extension-wallet]: 	Environment key "es2022" is unknown

I'm on it
https://github.com/input-output-hk/lace/actions/runs/6146361786/job/16675580818?pr=521

fixed


Weblog: Date in meta data has no influence on post order. [Bug]

Bug Description
Post meta data date does not reorder post order in archive.
Steps to Reproduce
Post 1 Date: Jan 1
Post 2 date: Jan 3
Adding a third post with date Jan 2 does not order it in the middle, but after Jan 3 instead.

This has been resolved!


feat: use helmBin() for kustomize build for helmChart in kustomization.yaml

This adds support --helm-binary for helmChart in kustomization.yaml by:

Changing use helmBin() for kustomize build

Resolves https://github.com/helmfile/helmfile/issues/1121

@moririnson  could you add some tests?

@yxxhero I added tests. Please review 🙇
In the first commit, I added cases for --enable-alpha-plugins=true (current helmfile) and --enable-alpha-plugins=false . The --enable-alpha-plugins=false case is expected to fail.
In the next commit, as the title of the pull request suggests, I made it so that helmBin() is also carried over to kustomize build.  Both test cases for --enable-alpha-plugins=true/false should succeed.
Interestingly, it seems that --enable-helm is enabled when --enable-alpha-plugins=true. I couldn't find any description to that effect, so I might be misunderstanding something.

@yxxhero Should I submit a p-r to use chartify 0.17.0 on the helmfile side? Or should I wait for dependabot?
I hope you can use this correction in your work.Thank you.

@moririnson you can post a PR. thanks so much.

@moririnson you can post a PR. thanks so much.

I will merge it as soon as I can.

@yxxhero I posted PR. Please merge when you can~ 🙏
https://github.com/helmfile/helmfile/pull/1140/files


Suggestions/comment regarding sensitivity talk

Still to do:

[x] font sizes #565
[ ] Summary slide
[ ] copy edit again
[ ] practice what to say (mostly know I think, but should actually go through it)

Font sizes fixed for heatmap axis tick labels - see ecb3d9af73f7ec5c27434bfd99b4bd33b4fddddd

That was a lot of work. Good job Andy. Here are some comments.

I fixed some typos w/ eb265fe9179c74f53219031ce01ae72a8f1c4331
I changed the \itemize structure on slide 27 with 91f38c9c2e73697afbc9d2c7ab1a0d7fcfb2d203
slide 30: It is unclear when "Time-varying fecundity?" is not "Y", how is the model configured? Does it use the mean 1975-2018 for 1975-2018?
slide 30 ...: I am unsure of what you are going to say when certain rows / columns are red, but you don't always highlight the column header. I thought maybe you only did this when an entire column was red, but you don't. So, your logic isn't quite clear.
slide 31-34: I think that in the future it might be good to make these plots with larger labels but not label every year or age. I do not think it is worth worrying about for this presentation. Just something to think about because if people didn't have the document to refer back to then there is no way they can read the years. Sorry this is Andre's voice projecting through because it has not been long enough since defending, so my voice is still dictated by not wanting to aggravate him. I will put it on my todo list for future to make this an option when printing these.
slide 35: can you add a red vertical line or something at year 1975 because my first thought was why in the heck is the yellow line so far below the others in the early years. Duh ..., but it took a while for me to dawn on the fact that this is when fecundity assumptions may differ.
slide 36: I would move this to before slide 35 and do the same for the following slides. So, have the table, then spawning biomass followed by relative spawning biomass, and finally the table before the next set.
slide 42 - 45: I am guessing that these always come after sensitivities? In my brain they would be better placed after the MCMC MLE comparison, but feel free to completely ignore this to remain status quo.
slide 46: Great plot. @aaronmberger do we do this for the PFMC stocks, if not, I think we should bring it up at a team meeting.

The historical  assessment plot hasn't been done for many (if any) PFMC stocks that I'm aware of, partly because there may be 3 three lines for many of them (or 2), others like sablefish it would work nicely.  I do like this retrospective assessments plot.

Thanks guys. Kelli - I'll add notes to your bullet points.....


Prevent normalization of polymorphic association

Polymorphic associations have no klass names, so those should be not
normalized. Fixes #588.

[x] Add tests
[x] Changelog entry

I might need help with the tests.

@coorasse any thought about this pull-request?

Hi! Thanks for your PR.
I like your solution, since this case is currently not well managed IMHO.
In an optimal case, it should be normalised to
expect(rule.conditions).to eq(attachments: { record_id: 11, record_type: 'Article' })

I'd merge it by adding a TODO that explains the best scenario, not currently possible.
What are your thought about it?

Let me check if I can get it working that way.

@coorasse The problem with that normalization is that it depends on querying for id, it will not work on other fields.
    # query for id
    rule = CanCan::Rule.new(true, :read, Blob, articles: { id: 11 })
    CanCan::ModelAdapters::ConditionsNormalizer.normalize(Blob, [rule])
    expect(rule.conditions).to eq(attachments: { record_id: 11, record_type: 'Article' })

    # but does not work for other queries
    rule = CanCan::Rule.new(true, :read, Blob, articles: { status: 'draft' })
    # expect(rule.conditions).to eq(attachments: { ???? })

I've fixed some missing options on the sample associations and updated for rails 6.0 rc2.
Regards.

It could be
expect(rule.conditions).to eq(attachments: { record: {id: 11, type: 'Article', status: 'draft' })?
We need to rebase, also. thanks

@coorasse I've done the rebase.

It could be
expect(rule.conditions).to eq(attachments: { record: {id: 11, type: 'Article', status: 'draft' })?

I'm not sure if that would work in Rails, but the main issue is that I'm not sure how to modify the actual code to do it.


Add a versioning mechanism for cache

Currently, there is no way to know which packages have been created with which version of the tool.
This poses some problems when a new version fixes a bug in the package generation, or changes how they should be.
For instance:

If we go the #128 way, we might have to rename packages tools from <tool-name>+bin+platform to <tool-name>. But that would be a pity to lose all cached packages.
The way we create and expect to be the packages for sandbox compiler might change. Not updating them would create issues such as #130.

We should add a versioning mechanism for those packages.

The two issues should be fixed now.


[SSoC 2.0] Fleury's Algorithm

Is your feature request related to a problem? Please describe.

This Algorithm is the alternative of Hierholzer's Algorithm of directed graph
This algorithm is used to find the euclerian circuit but this algo. can also find the path in the disconnected graph also

Describe the solution you'd like

It's a very useful algorithm to find the over the many disconnected bridges over the hierholzer algo.

Describe alternatives you've considered

We can also go for brute force method but it will time and space consuming which is not a efficient solution

I can solve this algorithm in C, C++, JAVA and PYTHON language

@Kumar-laxmi please assign this algorithm to solve under the program SSoC'23
I can solve this algorithm in C++, C, PYTHON & JAVA programming languages.
/assign


Unable to generate model from database in Visual Studio 2017 RC

I followed the tutorial in this link
When I try to execute Scaffold-DbContext "Server=Data Source=DESKTOP-COJD62N;Initial Catalog=TrainGameDB;Integrated Security=True" Microsoft.EntityFrameworkCore.SqlServer -OutputDir Models -context TrainGameContext -Verbose
I am getting the following output
Build started...
Build succeeded.
dotnet exec needs a managed .dll or .exe extension. The application specified was 'F:_IIT Level 6\UX\CW2 Backend\App2\TrainGameAPI\src\TrainGameAPI\bin\Debug\netcoreapp1.0\TrainGameAPI.runtimeconfig.json'
Process finished with non-zero exit code
I am using:
Visual Studio 2017 RC
SQL Server 2016 Developer
.NET Core 1.1.0 (I uninstalled version 1.0.1 and installed this)
Windows 10 Education
Here is the list of packages from my .csproj file (VS 2017 doesn't have a package.json file)
<PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="1.1.0" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer.Design" Version="1.1.0" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="1.1.0-preview4-final" />
    <PackageReference Include="Microsoft.NETCore.App" Version="1.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc" Version="1.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Routing" Version="1.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Server.IISIntegration" Version="1.0.0" />
    <PackageReference Include="Microsoft.AspNetCore.Server.Kestrel" Version="1.0.1" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="1.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="1.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="1.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging" Version="1.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="1.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="1.0.0" />
    <PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="1.0.0" />

Dupe of #7241

@alexidsa - Changing the framework version to dotNetCore 1.1.0 didn't help. I still have the issue. After updating the framework to 1.1.0, I am not able to open the project anymore (should be a visual studio issue).

@firedeepan can you provide a repro project?

@firedeepan The workaround for #7241 is to add the following to your csproj:
<PropertyGroup>
  <NuGetPackageRoot>$(USERPROFILE)\.nuget\packages\</NuGetPackageRoot>
</PropertyGroup

You can also try it on the latest bits. See #7358

Closing, but please re-open with additional information if you can't get it working on version 1.0.0-msbuild3 of the tools.


Add environment configuration option to set terminal background color

I use dark terminal. It is a pain to write every time --bg "dark".
It should be simple to read some environment variable (or any other type of global configuration file, e.g. ~/.ccat file) and define it globally. Then command line argument can just overwrite the global configuration, if present.
P.S. I want to add that light scheme looks ok on dark terminal, but i use dark terminal with transparency and only then it is unreadable. As it is unreadable for small portion of users, i do not think it is critical.

:+1: I plan to support this!

@igorshubovych The color scheme is not optimized. Feel free to help improve this. I also plan to make the color scheme customizable so that you could define yours. I pre-populate a bunch of color codes here.

Actually just another thought. How about alias ccat="ccat --bg=dark"?

Good idea

@igorshubovych I made a PR to provide options to set color values: https://github.com/jingweno/ccat/pull/27


Replace data base navigation with folder tree navigation

I think that will make it way simpler to understand/maintain and create the navigation tree

https://twitter.com/daKmoR/status/1411678879169880068?s=20
inspiration for implementation
https://github.com/adamduncan/eleventy-auto-navigation

Maybe we can score #151 along with this issue in the same pr

file tree navigation is the default since we use @rocket/engine


JSON encrypted/compressed over HTTPS

After deploying this app to a DigitalOcean droplet I was able to connect my local composer instance and install dependencies.
After enabling HTTP on the server using LetsEncrypt, composer is no longer able to fetch package.json as it appears encrypted/compressed.
https://packagist.cwserve.com/packages.json
I have tried enabling/disabling gzip to no avail.
Update: the file is encrypted/encoded on the server when opened with sudo nano package.json.

This is the output I get:
Composer version 1.6.5 2018-05-04 11:44:59

  [Seld\JsonLint\ParsingException]
  "" does not contain valid JSON
  Parse error on line 1:
  composer
  ^
  Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['

config [-g|--global] [-e|--editor] [-a|--auth] [--unset] [-l|--list] [-f|--file FILE] [--absolute] [--] [<setting-key>] [<setting-value>]...```

Hi @cameronwilby, there a feature, i lack to explain on a tutorial to create your own mirror, next few days i fixed this problem.
The mirror creation save the date as .gz, you need to enable reverse gz decode when a client ask for the version.
To do this, change you nginx configuration of gzip_static and gunzip as is:
server {
    index.html;

    server_name packagist.com.br www.packagist.com.br;

    location / {
        try_files $uri $uri/ =404;
        gzip_static on;
        gunzip on;
    }
}

Source:
http://nginx.org/en/docs/http/ngx_http_gunzip_module.html
http://nginx.org/en/docs/http/ngx_http_gzip_static_module.html

@Webysther that did the trick, thank you!  Here is the nginx config I ended up with:
server {
	listen 80;
	listen [::]:80;

	root /var/www/html;
	index index.php index.html index.htm;

	server_name packagist.com.br;

	location / {
		try_files $uri $uri/ =404;
		gzip_static on;
		gunzip on;
	}

	error_page 404 /404.html;
	error_page 500 502 503 504 /50x.html;
	location = /50x.html {
		root /usr/share/nginx/html;
	}

	location ~ \.php$ {
	        include snippets/fastcgi-php.conf;
        	fastcgi_pass unix:/run/php/php7.2-fpm.sock;
	}

	listen [::]:443 ssl ipv6only=on
	listen 443 ssl;
	ssl_certificate /etc/letsencrypt/live/packagist.com.br/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/packagist.com.br/privkey.pem;
	include /etc/letsencrypt/options-ssl-nginx.conf;
	ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}

Great @cameronwilby! Another tip is using a machine with 2GB at least of memory. With that all metadata keep to the memory helping the nginx and disk to not be consumed.

Good to know, I went with the 2GB droplet on DigitalOcean :+1: Thanks again @Webysther, awesome work.


Ten 121

app-config-base https://github.com/hmrc/app-config-base/pull/7046
journey tests https://github.com/hmrc/tax-enrolment-assignment-journey-tests/pull/28

@ChrisHillCap I've addressed the comments


dont use inverse protocol in SG generation

both src NIF and dst NIF should have the same protocols in their SG rules (inbound and outbound).

#200


Fix issues in Scenario-23

Description:
There are 3 test case failures in scenario 21. Please check the attached JTL file.
scenario23-results.txt
Suggested Labels:
Type/Task, Severity/Blocker, Priority/High
Affected Product Version:
IS 5.5.0

Tested on local an cluster setup multiple times was not able to reproduce this issue. On local setup I even tested by changing the  property

Did another test run today
scenario 1:
step1:remove config3 from WSO2Scenariofile.yaml
step2:run 23-post-scenario-steps.sh
step3:execute run-scenario.sh
Observation: the error can be reproduced
scenario 2:
step1:Keep config3 from WSO2Scenariofile.yaml
step2:run 23-post-scenario-steps.sh
step3:execute run-scenario.sh
Observation: the reported issue  can not be reproduced

JTL Files attached
23052018.zip

note: used jmeter 3.2 when working with the script

Shall we append all results to 1 jtl in available .jmx files ?


Enable pam_slurm.so

pam_slurm.so (from the slurm-pam_slurm rpm package) should be enabled in PAM configuration. It prevents local users from accessing nodes where they don't have jobs running.

Bonus feature related to this: in commit https://github.com/CSC-IT-Center-for-Science/ansible-role-slurm/commit/93af9538fa194182f095476449d64b7a65f97cf1 we allow users from the slurm_sshd_access_groups list to SSH in by pam_access.so