RuntimeError: expand(CUDABoolType) in train mapillary semantic segmentation

how to choice some specify coco categories for train in instance segmentation? I changed some codes,but it doesn't help.

Traceback (most recent call last):
File "/workspace/Mask2Former/train_net.py", line 322, in
launch(
File "/workspace/detectron2/detectron2/engine/launch.py", line 82, in launch
main_func(*args)
File "/workspace/Mask2Former/train_net.py", line 316, in main
return trainer.train()
File "/workspace/detectron2/detectron2/engine/defaults.py", line 484, in train
super().train(self.start_iter, self.max_iter)
File "/workspace/detectron2/detectron2/engine/train_loop.py", line 149, in train
self.run_step()
File "/workspace/detectron2/detectron2/engine/defaults.py", line 494, in run_step
self._trainer.run_step()
File "/workspace/detectron2/detectron2/engine/train_loop.py", line 413, in run_step
loss_dict = self.model(data)
File "/root/anaconda3/envs/segment/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/workspace/Mask2Former/mask2former/maskformer_model.py", line 204, in forward
targets = self.prepare_targets(gt_instances, images)
File "/workspace/Mask2Former/mask2former/maskformer_model.py", line 271, in prepare_targets
padded_masks[:, : gt_masks.shape[1], : gt_masks.shape[2]] = gt_masks
RuntimeError: expand(CUDABoolType{[21, 1024, 1024, 3]}, size=[21, 1024, 1024]): the number of sizes provided (3) must be greater or equal to the number of dimensions in the tensor (4)


Check sk__push function result

In all cases listed below, the result of the sk_push function is not checked. However, when you try to add records with call stack: sk_push -> OPENSSL_sk_push -> sk_reserve -> OPENSSL_realloc, memory manipulation occurs, which can give an error regardless of the input parameters. Therefore, I think that it is necessary to check the result, but I decided to ask for an opinion before creating PR.

https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/apps/engine.c#L355
https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/apps/engine.c#L358
https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/apps/engine.c#L373
https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/apps/engine.c#L378
https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/apps/lib/apps.c#L2493
https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/apps/lib/apps.c#L2499
https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/apps/pkcs12.c#L331

Yes, this is a minor bug. A PR would be welcome.

What about

https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/crypto/conf/conf_lib.c#L231
https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/crypto/conf/conf_mod.c#L547
https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/crypto/property/property.c#L470

Should we change function return type to int and start checking for it in the calling code?

What about
1. https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/crypto/conf/conf_lib.c#L231

2. https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/crypto/conf/conf_mod.c#L547

3. https://github.com/openssl/openssl/blob/8af4c02ea952ca387691c4a077c260ba045fe285/crypto/property/property.c#L470

Should we change function return type to int and start checking for it in the calling code?

Nope. But they should be marked with (void) to indicate the result is ignored.

Thanks.
Related PR: https://github.com/openssl/openssl/pull/25412

What about
https://github.com/openssl/openssl/blob/ad33d62396b7e9db04fdf060481ced394d391688/apps/smime.c#L315
https://github.com/openssl/openssl/blob/ad33d62396b7e9db04fdf060481ced394d391688/apps/smime.c#L321
https://github.com/openssl/openssl/blob/ad33d62396b7e9db04fdf060481ced394d391688/apps/smime.c#L346
https://github.com/openssl/openssl/blob/ad33d62396b7e9db04fdf060481ced394d391688/apps/smime.c#L351


Orchestrator Error

Hello @jvalls-axa ,
By mistake i have closed the request so i raised a new request for the same you can find the attachment below for this error -
It is completely processing some documents but with some documents it is giving error when processing with pdfminer -
There was an error running the orchestrator: TypeError: Cannot read property 'map' of undefined
at thereAreFakeSpaces (D:\Jain University\Final Year\Parsr\Parsr-0.11.1\Parsr-0.11.1\dist\src\input\pdfminer\pdfminer.js:390:10)
at breakLineIntoWords (D:\Jain University\Final Year\Parsr\Parsr-0.11.1\Parsr-0.11.1\dist\src\input\pdfminer\pdfminer.js:307:22)
at D:\Jain University\Final Year\Parsr\Parsr-0.11.1\Parsr-0.11.1\dist\src\input\pdfminer\pdfminer.js:211:47
at Array.forEach ()
at getPage (D:\Jain University\Final Year\Parsr\Parsr-0.11.1\Parsr-0.11.1\dist\src\input\pdfminer\pdfminer.js:206:24)
at D:\Jain University\Final Year\Parsr\Parsr-0.11.1\Parsr-0.11.1\dist\src\input\pdfminer\pdfminer.js:185:78
at Array.forEach ()
at getPages (D:\Jain University\Final Year\Parsr\Parsr-0.11.1\Parsr-0.11.1\dist\src\input\pdfminer\pdfminer.js:185:28)
at jsParser (D:\Jain University\Final Year\Parsr\Parsr-0.11.1\Parsr-0.11.1\dist\src\input\pdfminer\pdfminer.js:177:53)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
DemoInvoice.pdf

Hi @sjaintdl
Fixed in 0.11.2

Thanks @jvalls-axa ,
Now it is working well after installing your latest version.


[BugFix] Routine load job running time

What type of PR is this：

[x] bug
[ ] feature
[ ] enhancement
[ ] refactor
[ ] others

Which issues of this PR fixes ：

Fixes #7908
Problem Summary(Required) ：

This PR fixes the scheduled job running time.

@mergifyio backport branch-2.3 branch-2.2


Bug 1150641 - Add Shumway telemetry probe for cross domain loading issues

r? @tschneidereit

The probes "split" into two categories: loading swf and loading URLStream. For loading SWF we are reporting if SWF loading was rejected based on allowDomain whitelist. For loading via URLStream, we reporting if access was denied based on crossdomain.xml restrictions.

Looks good with comments addressed. One thing is a bit weird, IMO, though: for loads the probe really just reports whether they where allowed to happen or not, but for streams it also reports on whether loading was successful or not. Perhaps that would be better in another probe? We should also have reporting for it in the Loader, after all.

Review status: all files reviewed, 11 unresolved discussions, all commit checks successful.
Reviewed files:

extension/firefox/chrome/FileLoader.jsm @ r1
extension/firefox/chrome/ShumwayCom.jsm @ r1
extension/firefox/content/ShumwayTelemetry.jsm @ r1
extension/mozcentral/browser/extensions/shumway/content/ShumwayTelemetry.jsm @ r1
src/base/utilities.ts @ r1
src/flash/display/Loader.ts @ r1
src/flash/net/URLStream.ts @ r1
src/flash/system/Security.ts @ r1
src/player/external.ts @ r1
src/player/settings.ts @ r1

src/base/utilities.ts, line 3492 [r1] (raw file):
Nit: please use CamelCasing instead of UPPERCASING for enum entries.

src/base/utilities.ts, line 3494 [r1] (raw file):
I'd prefer Denied and Allowed for both the load and the stream cases.

src/flash/display/Loader.ts, line 379 [r1] (raw file):
I would change this to whitelistResult !== CrossDomainSWFLoadingWhitelistResult.Failed.

src/flash/net/URLStream.ts, line 107 [r1] (raw file):
Nit: please wrap this at 100 cols. Same below.

src/flash/system/Security.ts, line 111 [r1] (raw file):
Nit: please use CamelCasing instead of UPPERCASING for enum entries.

src/flash/system/Security.ts, line 112 [r1] (raw file):
Nit: please use line comments here and below. These won't be picked up by doc comment tools anyway, I think. If I'm wrong and they are picked up, ignore this.

src/flash/system/Security.ts, line 113 [r1] (raw file):
I'd prefer a different name for this. I had to go here to understand the code using it because I didn't get that it meant the using SWF's domain. OwnDomain would work for me.

src/player/settings.ts, line 768 [r1] (raw file):
Please add a type definition or even a class for this. Also, please rename source to something like ownDomain for the same reasons given for the enum entry.

src/player/settings.ts, line 770 [r1] (raw file):
Why is the new parameter optional? The two existing call sites should, IMO, clearly communicate the type of entry added here.

src/player/settings.ts, line 797 [r1] (raw file):
Can't the two else blocks be simplified to else { success = entry.insecure || url.protocol === entry.protocol; }?

Comments from the review on Reviewable.io

Review status: 5 of 10 files reviewed, 3 unresolved discussions, some commit checks pending.

src/flash/system/Security.ts, line 112 [r1] (raw file):
no preference here, fixed

src/player/settings.ts, line 797 [r1] (raw file):
out of scope of this PR

Comments from the review on Reviewable.io

Review status: all files reviewed, 2 unresolved discussions, all commit checks successful.

Comments from the review on Reviewable.io


Linked List using Python

💥 Proposal
(A clear and concise description of what the proposal is.)
Have you read the Contributing Guidelines on Pull Requests?
(Write your answer here.)
Yes

/assign


Build failure after upgrading to 2.7.1 version

I'm submitting a ...  (check one with "x")

[ ] question
[x] any problem or bug report

OS:  (check one with "x")

[x] Android
[ ] iOS
[ ] Browser

cordova information:  (run $> cordova plugin list)
cordova-plugin-actionsheet 2.3.3 "ActionSheet"
cordova-plugin-app-version 0.1.9 "AppVersion"
cordova-plugin-browsertab 0.2.0 "cordova-plugin-browsertab"
cordova-plugin-buildinfo 4.0.0 "BuildInfo"
cordova-plugin-camera 4.1.0 "Camera"
cordova-plugin-compat 1.2.0 "Compat"
cordova-plugin-device 2.0.3 "Device"
cordova-plugin-dialogs 2.0.2 "Notification"
cordova-plugin-file 6.0.2 "File"
cordova-plugin-googlemaps 2.7.1 "cordova-plugin-googlemaps"
cordova-plugin-inappbrowser 3.2.0 "InAppBrowser"
cordova-plugin-ionic-keyboard 2.2.0 "cordova-plugin-ionic-keyboard"
cordova-plugin-ionic-webview 4.2.0 "cordova-plugin-ionic-webview"
cordova-plugin-market 1.2.0 "Market"
cordova-plugin-network-information 2.0.2 "Network Information"
cordova-plugin-splashscreen 5.0.3 "Splashscreen"
cordova-plugin-statusbar 2.4.3 "StatusBar"
cordova-plugin-whitelist 1.3.4 "Whitelist"
cordova-plugin-x-toast 2.7.2 "Toast"
cordova-sqlite-storage 5.0.0 "Cordova sqlite storage plugin - cordova-sqlite-storage plugin version"
cordova-support-google-services 1.3.2 "cordova-support-google-services"
cordova-universal-links-plugin-fixed 1.2.1 "Universal Links Plugin"
es6-promise-plugin 4.2.2 "Promise"
phonegap-plugin-multidex 1.0.0 "Multidex"
phonegap-plugin-push 2.3.0 "PushPlugin"
uk.co.workingedge.phonegap.plugin.launchnavigator 5.0.4 "Launch Navigator"

If you use @ionic-native/google-maps, please tell the package.json (only @ionic-native/core and @ionic-native/google-maps are fine mostly)
@ionic-native/core : 5.23.0
@ionic-native/google-maps : 5.5.0

Current behavior:

After upgrading plugin from version 2.6.2 to version 2.7.1 Android build fails.
Expected behavior:

I expect the build doesn't fail.
Screen capture or video record:

Related code, data or error log (please format your code or data):
FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture fingerprint of input files for task ':app:preDebugBuild' property 'compileManifests' during up-to-date check.
> In project 'app' a resolved Google Play services library dependency depends on another at an exact version (e.g. "[15.0.> Task :app:preDebugBuild FAILED
Dependency resolved to an incompatible version: Dependency(fromArtifactVersion=ArtifactVersion(groupId=com.google.android.gms, artifactId=play-services-stats, version=15.0.1), toArtifact=Artifact(groupId=com.google.android.gms, artifactId=play-services-basement), toArtifactVersionString=[15.0.1])

  1]", but isn't being resolved to that version. Behavior exhibited by the library will be unknown.

  Dependency failing: com.google.android.gms:play-services-stats:15.0.1 -> com.google.android.gms:play-services-basement@[
  15.0.1], but play-services-basement version was 16.0.1.

  The following dependencies are project dependencies that are direct or have transitive dependencies that lead to the art
  ifact with the issue.
  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>
  For extended debugging info execute Gradle from the command line with ./gradlew --info :app:assembleDebug to see the dep
  endency paths to the artifact. This error message came from the google-services Gradle plugin, report issues at https://
  github.com/google/play-services-plugins and disable by adding "googleServices { disableVersionCheck = false }" to your b
  uild.gradle file.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 3s
3 actionable tasks: 1 executed, 2 up-to-date
C:\Users\Alessandro\Kaleidos\kaleidos-frontend\platforms\android\gradlew: Command failed with exit code 1 Error output:
FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture fingerprint of input files for task ':app:preDebugBuild' property 'compileManifests' during up-to-date check.
> In project 'app' a resolved Google Play services library dependency depends on another at an exact version (e.g. "[15.0.
  1]", but isn't being resolved to that version. Behavior exhibited by the library will be unknown.

  Dependency failing: com.google.android.gms:play-services-stats:15.0.1 -> com.google.android.gms:play-services-basement@[
  15.0.1], but play-services-basement version was 16.0.1.

  The following dependencies are project dependencies that are direct or have transitive dependencies that lead to the art
  ifact with the issue.
  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>  -- Project 'app' depends onto<EMAIL_ADDRESS>
  For extended debugging info execute Gradle from the command line with ./gradlew --info :app:assembleDebug to see the dep
  endency paths to the artifact. This error message came from the google-services Gradle plugin, report issues at https://
  github.com/google/play-services-plugins and disable by adding "googleServices { disableVersionCheck = false }" to your b
  uild.gradle file.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 3s

There is a good explain in the error log.
 Dependency failing: com.google.android.gms:play-services-stats:15.0.1 -> com.google.android.gms:play-services-basement@[
  15.0.1], but play-services-basement version was 16.0.1.

I'm sorry but for me that is not a good explanation...
What should I do to change play-services-basement version?

You need to specify the same versions for all plugins

Ok.
Yesterday I tried to install googlemaps plugin with this command:
cordova plugin add cordova-plugin-googlemaps --variable PLAY_SERVICES_VERSION="16.0.0"
but still no success.
The strange thing is that with googlemaps plugin version 2.6.2 everything works, so I imagine to be able to make everything work changing something in googlemaps plugin.
Can you give me a hint to understand where I should intervene please?

All variables into config.xml now.
https://github.com/mapsplugin/cordova-plugin-googlemaps/blob/master/README.md#install-optional-variables-configxml

I could find that the problem is due to phonegap-plugin-push plugin: their latest version conflicts with your latest version, and until now I couldn't find any way to let them coexist, also changing their variable "FCM_VERSION".

Please ask to phonegap-plugin-push plugin developer.

I did it!
The key was installing phonegap-plugin-push plugin this way:
ionic cordova plugin add phonegap-plugin-push --variable FCM_VERSION="17.3.4"

Sorry, I'm only in charged of this plugin. Other plugins are out of my cover. I think you should ask at the ionic forum, not blaming to me.

I'm sorry for the misunderstanding, I was just happy because I found the solution... 😉
Thank you for your great plugin!

I'm sorry for the misunderstanding, I was just happy because I found the solution... 😉
Thank you for your great plugin!

those also solved same problem here.thanks!


[Windows] Temporary remove hash check from DotnetSDK

Description
During the investigation it was found that the dotnet SDK archive at URL https://dotnetcli.azureedge.net/dotnet/Sdk has a different hash than the archive from the releases.json file at URL https://download.visualstudio.microsoft.com/download/pr.
It was decided to temporary remove hash checking function and raise a ticket to dotnetSDK repo.
Along with that the signature for SessionManagerPluginSetup.exe was also updated.
Related issue:
#9958
Check list

[x] Related issue / work item is attached
[ ] Tests are written (if applicable)
[ ] Documentation is updated (if applicable)
[ ] Changes are tested and related VM images are successfully generated

/azp run windows2019, windows2022

@Alexey-Ayupov : thanks for working on a workaround.
Did you (or someone) raise a ticket to dotnetSDK repo? If so, where can we track?
(Same package version with on different hosts with different hash needs investigation as this probably indicates security issue.)

Workaround to remove the hash checking for dotnetSDK is no longer needed:
The package downloaded from https://dotnetcli.azureedge.net results in the hash code that matches https://dotnetcli.blob.core.windows.net/dotnet/release-metadata/6.0/releases.json.


Support initialisation with AppOptions

firebase::App::Create() has an overload that takes an object of type AppOptions to permit Firebase to be initialized with non-default values.
We'd like to use it to permit selection of Firebase projects for different environments (dev, staging etc) from a boot screen.
I'm happy to take this on myself, but could do with some guidance on the Lua API, as AppOptions has 6 string members.

I'm happy to take this on myself, but could do with some guidance on the Lua API, as AppOptions has 6 string members.

We should change firebase.init() to take an optional Lua table with options, ie:
local options = {
  api_key = "a",
  app_id = "b",
  project_id = "c",
}
firebase.init(options)

The extension code will look something like this:
	AppOptions options;
	if (!lua_isnil(L, 1)) {
		luaL_checktype(L, 1, LUA_TTABLE);
		lua_pushvalue(L, 1);
		lua_pushnil(L);
		while (lua_next(L, -2)) {
			const char* attr = lua_tostring(L, -2);
			if (strcmp(attr, "api_key") == 0)
			{
				options.set_api_key(luaL_checkstring(L, -1));
			}
			else if (strcmp(attr, "app_id") == 0)
			{
				options.set_app_id(luaL_checkstring(L, -1));
			}
			else if (strcmp(attr, "project_id") == 0)
			{
				options.set_project_id(luaL_checkstring(L, -1));
			}
			lua_pop(L, 1);
		}
		lua_pop(L, 1);
	}

Closed via #5


Exception caught by rendering library ===================================================== The following assertion was thrown during paint():

Either the assertion indicates an error in the framework itself, or we should provide substantially more information in this error message to help you determine and fix the underlying cause.

@Manishkumar76
Although the error says to file an issue, but in order to properly address the root-cause of the error, we need minimal reproducible code sample along with steps to replicate so that we can make the issue actionable.
Closing for now, since no information is provided. If you have concrete details, please use new issue template and fill out requested details so that we can properly analyze the issue.


README.md Contains Link to Archived Repository

Describe the bug
main: latest
In the README.md section quick start

one can read the following sentence after the word Note  For a more detailed or advanced Docker or Kubernetes deployment, as well as a full setup with the Connector and its GUI, see [here]

This link https://github.com/International-Data-Spaces-Association/IDS-Deployment-Examples/tree/main/dataspace-connector is referring to an archived repository.

one can read the following sentence: The official Docker images of the Dataspace Connector can be found [here]

This link https://github.com/International-Data-Spaces-Association/DataspaceConnector/pkgs/container/dataspace-connector is referring to an archived repository.
To Reproduce
Steps to reproduce the behavior:

Press on the links as described above

Expected behavior
Documentation references to actively supported and contributed repositories.

Hello, thanks for reaching out.
Regarding the first link: The repository is archived but still contains a setup to start the connector with GUI (one would just have to upate the versions). For another version you could check out the IDS-Testbed.
Regarding the second: The link to the docker images is correct. The repository is archived but is used for publishing the new images (see images being published 1 day ago) as it was the original repository.


Enhanced telegram functionality

Enhanced telegram functionality.
this should also close issue #5236
/softbans - shows info about possible softbans
/caught - last 25
/evolved -  last 25
/hatched - last 25
/pokestops - last 25
/transfers - last 25
also displays info on events:
bot_random_pause
bot_random_alive_pause
log_stats
show_inventory

Can we have a meaningful PR description before we start reviewing?

My apologies for all the trouble with the pull request. I figured this one would fail also.

The web version was touched, but I’ll get it right later

👍

Have you tested this with "master" absent in the config file?

No. if telegram class is instantiated there should always be a master.

Nevermind i see what you're saying

No, with the new "password" configuration, you do not need a "master" at all.

So whats the solution here. This allows telegram to work for me. The other way always keeps master at None. (for me) at least

If config ['master']
Master = config ["master"]   ??

The master is not needed, and actually subject o be removed in future. Use password.

But if you MUST use the master, always use config.get('master', None), otherwise you will get errors if master is not specified in the config.

Thanks for the heads up....ill abandon this and start using password.

How exactly do you find your telegram password. I have tried everything from my api key to my master to any other password I might have set and I just get invalid password everytime.

seriously is the password thing a joke.

👍

👍


NetworkController: Use the same middleware stack as the extension

Changes

BREAKING:

[@metamask/network-controller] Update initializeProvider to throw if the provider config in state has a type of "localhost" but not a chainId
(NetworkController) Update initializeProvider to throw if the provider config in state has a type of "rpc" but neither a chainId nor an rpcTarget
[@metamask/network-controller] Update provider property returned by getProviderAndBlockTracker so that it is now typed as a SwappableProxy<SafeEventEmitterProvider> instead of any
[@metamask/network-controller] Update initializeProvider to throw if the provider config in state has a type of "mainnet", "goerli", or "sepolia" and infuraProjectId and the NetworkController has been initialized without an infuraProjectId

FIXED:

no longer using completely different provider than metamask-extension

CHANGED:

eth_chainId calls to the provider are now cached, giving the configured chainId (0x prefixed and converted to hex if necessary).
[@metamask/network-controller] Bump dependency eth-json-rpc-infura (now @metamask/eth-json-rpc-infura) from ^7.0.0 to ^8.0.0

REMOVED:

static methods:

web3_clientVersion
net_listening
eth_hashrate
eth_mining

Sanitization middleware
Filter & subscribe middleware (when connection type is not websocket)
Nonce tracker middleware
all the 'hooked wallet subprovider' handlers:

getAccounts
processTransaction
approveTransaction
signTransaction
publishTransaction
processMessage
approveMessage
signMessage
processPersonalMessage
processTypedMessage
approvePersonalMessage
approveTypedMessage
signPersonalMessage
signTypedMessage
personalRecoverSigner

websocket support

ADDED:

BlockRefRewrite middleware
[@metamask/controller-utils] Add InfuraNetworkType enum
[@metamask/network-controller] Add dependency eth-json-rpc-middleware ^11.0.0
[@metamask/network-controller] Add dependency eth-json-rpc-provider ^1.0.0
[@metamask/network-controller] Add dependency eth-block-tracker ^7.0.0
[@metamask/network-controller] Add dependency json-rpc-engine ^6.1.0

Checklist

[x] Tests are included if applicable
[x] Any added code is fully documented

Issue
Fixes #970

@mcmire just rebased

@Gudahtt I've updated this branch with main again. Note that this PR is longer than the original version because I've removed handling the error event, we no longer "stop" the provider, and I've also fixed the types for Provider and BlockTracker, but I've tried to leave comments to explain what each kind of change is doing.
I plan on polishing the PR description a bit but the changes themselves should be reviewable at least.

@Gudahtt I've merged recent changes from main.


Removed stats, added daily telemetry

Removed the old (unused?? cc @aaronrothschild) stats API
Added daily telemetry for #instances, users, subscriptions

Codecov Report

Merging #836 (db1714a) into flow (985cc26) will decrease coverage by 1.07%.
The diff coverage is 3.23%.

:exclamation: Current head db1714a differs from pull request most recent head b876060. Consider uploading reports for the commit b876060 to get more accurate results

@@            Coverage Diff             @@
##             flow     #836      +/-   ##
==========================================
- Coverage   33.11%   32.03%   -1.08%
==========================================
  Files          53       49       -4
  Lines        6276     5875     -401
==========================================
- Hits         2078     1882     -196
+ Misses       3989     3804     -185
+ Partials      209      189      -20

Impacted Files
Coverage Δ

server/atlassian_connect.go
0.00% <0.00%> (ø)

server/http.go
44.15% <ø> (+1.38%)
:arrow_up:

server/instance_server.go
18.05% <ø> (+0.48%)
:arrow_up:

server/kv.go
20.00% <ø> (ø)

server/plugin.go
11.37% <0.00%> (-1.52%)
:arrow_down:

server/setup_flow.go
0.00% <0.00%> (ø)

server/subscribe.go
67.87% <ø> (+1.01%)
:arrow_up:

server/user.go
25.59% <0.00%> (-0.16%)
:arrow_down:

server/user_cloud.go
0.00% <ø> (ø)

server/utils/utils.go
76.36% <ø> (-0.83%)
:arrow_down:

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6917d4a...b876060. Read the comment docs.


A series of Questions

below is a list of questions of which require two answers:

one from a conceptual point of view,
how should it work?
the other from the point of view of a technical solution, the mechanism by which the difficulty is solved,
how does it work?

the technical answer should underpin and build upon the answer from the conceptual point of view.

the aim of the list is to explore and clarify what rotonde is, pinpoint the difficulties that need to be surmounted, and to guide everyone to a similar place in order to solve them

can people run rotonde without an instance?

can people run rotonde without a dedicated server?
~will every person wanting to run rotonde be able to run an instance?~

will there be rotonde visitors without instances?
what is the lifetime of a rotonde instance?
how will people connect to instances?
how are new instances found?
how will bots be handled?
how do you script your instance?
how will chatlogs of instances be handled?

will a visiting user be able to view logs from when they weren't there, as is possible with slack?

how does an instance tell all of its spectators it is now alive?
how does an instance notify all of its spectators of notifications?
how are messages sent between instances?
how are files exchanged within an instance?
how are an instance's modules accessed?
what is an example of an instance's module?
can you send notifications to an instance without a client?
how easy is it to deploy an instance?

if you haven't programmed before?
if you don't know any javascript?
if you can't configure your webserver?

is rotonde simple?
is rotonde accessible?
what is rotonde?

~is each instance simply a hosted rotonde core (and plugins)?~
~when an instance is saved by a visitor, what is actually saved?~

can people run rotonde without an instance?
No
will there be rotonde visitors without instances?
No
what is the lifetime of a rotonde instance?
As long as the connection is active
how will people connect to instances?
/visit neauoire, Join a pair
/visit The-Lobby,
how are new instances found?
word of mouth, direct links
how will bots be handled?
plugins
how do you script your instance?
plugins, connecting to different APIs of your instances and client
how will chatlogs of instances be handled?
plugins
will a visiting user be able to view logs from when they weren't there, as is possible with slack?
possible through asking the logging bot for a backlog, auto sending a notification with the history.
how does an instance tell all of its spectators it is now alive?
Your instance constantly pings your spectated instances for activity.
how does an instance notify all of its spectators of notifications?
what?
how are messages sent between instances?
sockets
how are files exchanged within an instance?
No idea
how are an instance's modules accessed?
APIs, possibly metaprograming like Faun's Open Sky
what is an example of an instance's module?
A bot
can you send notifications to an instance without a client?
yes, instances open a website that could have APIs themselves
how easy is it to deploy an instance?
rotonde connect
if you haven't programmed before?
rotonde connect
if you don't know any javascript?
This is not for you.
if you can't configure your webserver?
It should be handled by Rotonde. Similar to Python's SimpleHTTPServer -8000
is rotonde simple?
No, it's an engine.
is rotonde accessible?
No, it's an engine for Merveilles-like people.
what is rotonde?
An engine to communicate, create utilities, share, build identity and a personal space online.

can people run rotonde without an instance?

can people run rotonde without a dedicated server?

I know @neauoire says that we do not need a server, but currently, having a server of some sorts is a requirement. If we can figure out a way to do it in the future, then the system will support adding a deployment method that supports local systems.

will there be rotonde visitors without instances?

Currently, it is possible for clients to connect to Rotonde without having an instance of their own.

what is the lifetime of a rotonde instance?
how will people connect to instances?
People will connect to instances using one of the Rotonde clients, either first-party or third-party.

how are new instances found?

how will bots be handled?

Bots will be handled via plugins.

how do you script your instance?

You script your instance by creating plugins.

how will chatlogs of instances be handled?

will a visiting user be able to view logs from when they weren't there, as is possible with slack?

Any sort of logging/persistence would be handled by a plugin. But if you had a plugin saving all messages, then I would think it would show all messages to clients when they reconnect.

how does an instance tell all of its spectators it is now alive?

Since an instance is a websocket server, if it goes offline all of its clients will have their sockets disconnected. Now, if the client maintains a list of instances that it has connected to, we could potentially poll them and attempt to reconnect once they come online.

how does an instance notify all of its spectators of notifications?

The instance notifies spectators by broadcasting a message over websockets.

how are messages sent between instances?

Messages are sent between instances over websockets.

how are files exchanged within an instance?

This would be delegated to a plugin, and could be done via websockets/HTTP.

how are an instance's modules accessed?

Are you referring to accessing the plugins from within one of the Rotonde clients? There would be some sort of API for accessing the plugins.

what is an example of an instance's module?
can you send notifications to an instance without a client?

No, you need a Rotonde client to communicate with a Rotonde instance.

how easy is it to deploy an instance?

if you haven't programmed before?
if you don't know any javascript?
if you can't configure your webserver?

Each deployment method will have varying levels of complexity, which is tied entirely to the complexity of the environment you are deploying too.
But the way it works from the CLI client is you type create, and it will prompt you with one of the supported deployment targets. Once you select an option, it would then walk you through additional configuration and then deploy to the environment.

is rotonde simple?

Depends on what you mean by simple. It's core will definitely not be simple, by nature. But plugin development will be simple. Creating and consuming plugins should be dead-simple.

is rotonde accessible?
what is rotonde?


Hold being lost in SetReplace functions

Describe the bug
The following should not remove Hold:
In[] := SetReplace[{1, 2}, {a_, b_} :> {Hold[a + b]}]

Out[] = {3}

It should do the same thing as Replace:
In[] := Replace[{1, 2}, {a_, b_} :> {Hold[a + b]}]

Out[] = {Hold[1 + 2]}

Version (please complete the following information):

OS: 10.15.2 (19C57)
Wolfram Language Version: "12.1.0 for Mac OS X x86 (64-bit) (December 21, 2019)"
SetReplace git sha: ba9234cd62794ef55cd9473320ec7eeb9473e21e

Might not worth fixing before #67 or #401.


networkmanager: 1.18.0 -> 1.18.1

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/network-manager/versions.
meta.description for networkmanager is: '"Network configuration and management tool"'.
meta.homepage for networkmanager is: '"https://wiki.gnome.org/Projects/NetworkManager"

Checks done (click to expand)

built on NixOS
/bin/sh passed the binary check.
1 of 1 passed binary check by having a zero exit code.
0 of 1 passed binary check by having the new version present in output.
found 1.18.1 with grep in /nix/store/sr08zswidchm2i9z4v1635qv8870dycf-network-manager-1.18.1
directory tree listing: https://gist.github.com/fa1e69c0be0bc5125ed049fa48c35094
du listing: https://gist.github.com/b59a575a802e9221ce021748bf8ac3a8

Rebuild report (if merged into master) (click to expand)

1595 total rebuild path(s)
552 package rebuild(s)
552 x86_64-linux rebuild(s)
523 i686-linux rebuild(s)
6 x86_64-darwin rebuild(s)
514 aarch64-linux rebuild(s)
First fifty rebuilds by attrpath
adapta-gtk-theme
akregator
almanah
arc-theme
areca
astroid
azureus
balsa
birdfont
bolt
bomi
bookworm
chrome-gnome-shell
claws-mail
corebird
deepin.dde-daemon
deepin.dde-dock
deja-dup
digikam
dino
discover
dolphin
dragon
dropbox-cli
eclipses.eclipse-cpp
eclipses.eclipse-java
eclipses.eclipse-modeling
eclipses.eclipse-platform
eclipses.eclipse-scala-sdk
eclipses.eclipse-sdk
eid-mw
ekiga
empathy
eolie
ephemeral
epiphany
evolution-data-server
feedreader
flatpak
flatpak-builder
folks
fondo
fwupd
geoclue2
gfbgraph
gimp
gimp-with-plugins
gimpPlugins.exposureBlend
gimpPlugins.focusblur
gimpPlugins.fourier

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/sr08zswidchm2i9z4v1635qv8870dycf-network-manager-1.18.1 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
Or, build yourself:
nix-build -A networkmanager https://github.com/r-ryantm/nixpkgs/archive/367d6b556ef4ac82ffc8b30613fe326a0876d244.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/sr08zswidchm2i9z4v1635qv8870dycf-network-manager-1.18.1
ls -la /nix/store/sr08zswidchm2i9z4v1635qv8870dycf-network-manager-1.18.1/bin

cc @phreedom @rickynils @domenkozar @obadz for testing.

Is this stable https://gitlab.freedesktop.org/NetworkManager/NetworkManager/tags/1.18.1-dev ?

Apparently not: https://download.gnome.org/sources/network-manager-applet/1.8/
Good catch @worldofpeace.


Strange behavior of display different error message for signup if username/email already exists on Mongo 3.0/3.2

@trendzetter @ilanbiala
I'm reading Mongoose doc. I did not find any info about configuring err msg for unique.
I'm running Ubuntu 14.04 with MongoDB 3.2.3. Nodejs is v4.3.2
I attached a screenshot. This is not good to display such a detailed error.

I tested with clean enviroment in vagrant. The only difference is: MongoDB 3.0.9 and 3.2.3.
I wonder where does the sentence "Email already exists" or '11000 duplicate key error collection: mean-test.users index: username already exists' to be 'Username already exists' come from?
I enabled the debug mode of mongoose. I tested with both Mongo 3.0 and 3.2.
The query generated by mongoose is the same (execept salt).
Then I execute the insert query manually in Mongo 3.0 and 3.2.
The output is also the same.
Mongo 3.0
Mongoose: users.insert({ salt: 'RphFQ5BmRcgAxoa9vAaCTQ==', displayName: 'Test User', provider: 'local', username: 'testuser', _id: ObjectId("56d767da3ccc59b408859f90"), created: new Date("Wed, 02 Mar 2016 22:23:22 GMT"), roles: [ 'user' ], profileImageURL: 'modules/users/client/img/profile/default.png', password: '9WsKha9q20AQdR6jYfBIv9LSYg5Qsfa+MTLFka0KmazFcGKh9VKVxFm1ft3S2AjqIVJorN4ihEz1nOoH6U22UA==', email<EMAIL_ADDRESS>lastName: 'User', firstName: 'Test', __v: 0 })
WriteResult({
"nInserted" : 0,
"writeError" : {
"code" : 11000,
"errmsg" : "E11000 duplicate key error index: mean-dev.users.$username_1 dup key: { : "testuser" }"
}
})
Mongo 3.2
Mongoose: users.insert({ salt: 'd4SiQVR3xDk0rqHQ3dajNw==', displayName: 'Test User', provider: 'local', username: 'testuser', _id: ObjectId("56d7684543b5be880b8aedd7"), created: new Date("Wed, 02 Mar 2016 22:25:09 GMT"), roles: [ 'user' ], profileImageURL: 'modules/users/client/img/profile/default.png', password: 'hsKel5Gew/IVqoo8oynaOFayV35sKP07xbcApvVosAhILPg99l0ccv74+mu7k14+ph2LJpxQ5ntPsDBTMtYhag==', email<EMAIL_ADDRESS>lastName: 'User', firstName: 'Test', __v: 0 })
WriteResult({
"nInserted" : 0,
"writeError" : {
"code" : 11000,
"errmsg" : "E11000 duplicate key error index: mean-dev.users.$username_1 dup key: { : "testuser" }"
}
})
I think mongodb does not change error reply. I wonder whether mongoose or other plugin changes the err msg behavior.

@trendzetter @ilanbiala
I figured out the problem.
If you upgrade mongodb from 3.0 to 3.2, there is no issue because you still use MMAPv1 engine.
But if you install new mongodb 3.2, there is a problem because you use new WiredTiger engine.
The output of these  2 engine is slightly different.
MMAPv1
WriteResult({
"nInserted" : 0,
"writeError" : {
"code" : 11000,
"errmsg" : "E11000 duplicate key error index: mean-dev.users.$email_1 dup key: {<EMAIL_ADDRESS>}"
}
})
WiredTiger
WriteResult({
"nInserted" : 0,
"writeError" : {
"code" : 11000,
"errmsg" : "E11000 duplicate key error collection: mean-dev.users index: email_1 dup key: {<EMAIL_ADDRESS>}"
}
})
There is a minor diff between .$ and index:
The method in the file modules/core/server/controllers/errors.server.controller.js
is used to parse the error message and output it to front webpage:
/**

Get unique error field name
*/
var getUniqueErrorMessage = function (err) {
var output;

try {
var fieldName = err.errmsg.substring(err.errmsg.lastIndexOf('.$') + 2, err.errmsg.lastIndexOf('_1'));
output = fieldName.charAt(0).toUpperCase() + fieldName.slice(1) + ' already exists';
} catch (ex) {
output = 'Unique field already exists';
}
return output;
};
It only deals with .$
If we also parse index:, it will be done without modifying any test script.

@QiyuLi I don't quite follow your suggestion, but if you'd like to submit a PR we'd gladly review it. Thanks for getting to the bottom of this.


Mypy compliance; then enable it in github workflow

We are very close to being mypy compliant with only 10 remaining errors. Once we have fixed them, we should enable mypy errors in the github workflow so that we are always mypy complient.

This is now completed https://github.com/zxcalc/zxlive/commit/c0801323a441959c4f114bbff50369a2da425da1


Handle overloaded mapper methods

MyBatis version
3.4.1-SNAPSHOT/master
Test case or example project
o.a.i.submitted.overload.OverloadTest  - See pull request #704 for the testcase and more details.
Steps to reproduce
Given

a mapper interface with two methods with the same name but different parameter sets (e.g. one and two parameters)
a mapper xml with a single statement with the same name as the interface method, using all parameters from the interface, but properly checking for null values
Expected result
One can use both methods, both being properly mapped to the sql statement from the mapper xml
Actual result
Trying to execute the method not having all parameters used in the mapper xml statement will result in a BindingException

This has been asked several times before, but it is labeled as "won't fix".
https://github.com/mybatis/old-google-code-issues/issues/231
Basically, we couldn't find a simple way to differentiate those statements.
When you want to use one of the overloaded methods as a nested select, for example, what would the statement ID look like?

This still is "won't fix" until someone came up with an elegant solution.

so there are two ways to handle  variable parameter method overload:

use Map parameter
use different method name

Met this issue and google lead me here. My2Cents. one could leverage one optional Id attribute as part of @Select annotation, to explicitly supply one Id to the method and the framework can notify the user to use this annotation when there is duplication observed, during startup. This de-couples the method name and id. Cannot this be more graceful than restrict the language feature by a given framework.
@Select("SELECT * FROM student", id="selectAllFromStudent")
List loadStudents();
@Select("SELECT * FROM student WHERE name = #{name}" id="selectStudentsByName")
List loadStudents(String name);

Thank you for the input @grantsunny ,
There is an open PR #1519 which basically is the same idea as yours and some people seem to like it.
However, I am still not convinced if it's worth supporting overloaded methods.
I think the result is simpler/better if you simply rename the method(s).
In your case, for example, if you name the second method loadStudentsByName(String), you don't lose much and it's easier for future readers to understand the code, especially if the statement is referenced from result maps as a nested select.
Anyway, please upvote #1519 if you like it.
We might merge it if it gets many upvotes.

Thank you @harawata - I will be there and upvote it.
I agree renaming a method does not lose much though, as getXXXById, getXXXByType. It works in principle but I think the upvoters just feel hesitated to loose the flexibility. Have a nice day!


What do gamma correction values mean for the RGB value?

Required Info

Camera Model
D400

Issue Description
For some real sense cameras, developers can set the gamma correction value of the realsense cameras, e.g., for the D435 between 100 and 500 according to the data sheet.
I need to extract linear RGB data without gamma correction though, ideally by deactivating gamma correction or removing gamma correction from the RGB values of the camera.
My question is, what does the gamma value range mean? Does the range 100 to 500 represent floating point values? E.g., a gamma correction of 100 means 1.00, i.e., no gamma correction? Then I could just use this setting for uncorrected RGB values.
Thanks in advance.

Hi @RealitiesMediaCube   Do you require further assistance with this case, please?   Thanks!

Hi @MartyG-RealSense,
thanks for the info about the raw bayered RGB images. I will look into that.
I would appreciate any input your team can give me on the configuration of the gamma value and its impact on the image. The realsense documentation is not clear about how gamma is applied and what the range of values means.

Gamma can be configured with the RS2_OPTION_GAMMA instruction.  A script in the link below provides an example of the use of the instruction to configure a gamma value between 300 and 500:
https://github.com/IntelRealSense/librealsense/issues/6560#issuecomment-641869430

I could not find documentation that provided further descriptive explanation.  In general though, reducing RGB gamma darkens the RGB image and increasing gamma brightens the image.  Below is shown a scene with 100 gamma (minimum) in the upper image and 500 gamma (maximum) in the lower image.
RGB gamma is independent of the RGB auto-exposure.  So whilst changing exposure and gain whikst auto-exposure is active disables the auto-exposure, changing the gamma does not affect the status of auto-exposure.

Thanks for the detailed information and your support @MartyG-RealSense.
I need a linear color response from the camera for further processing. Hence, I either turn gamma of, or know how it is applied, so that I can reverse the process.
I determined from the data sheet that the camera uses the OV9282 image sensor. Theoretically, the gamma correction can even be turned off for some image sensors, maybe also for this one. Is there a possibility in the realsense SDK to send instructions directly to the image sensor, some kind of raw input signals for controlling the sensor?
I could also contact the chip designers directly for details on gamma correction.

Thanks for the detailed information and your support @MartyG-RealSense.
I need a linear color response from the camera for further processing. Hence, I either turn gamma of, or know how it is applied, so that I can reverse the process.
I determined from the data sheet that the camera uses the OV9282 image sensor. Theoretically, the gamma correction can even be turned off for some image sensors, maybe also for this one. Is there a possibility in the realsense SDK to send instructions directly to the image sensor, some kind of raw input signals for controlling the sensor?
I could also contact the chip designers directly for details on gamma correction.

The camera sensors can be accessed directly by using callbacks with the Low Level API.
https://github.com/IntelRealSense/librealsense/blob/master/doc/api_arch.md#low-level-device-api
Low-level “sensor” callbacks API calls:

Minimum SDK-imposed overhead == minimum latency.
Each frame is handled with a separate callback - “first come – first serve”.
All the synchronization stuff is left for the user side to implement.
The rs-data-collect tool of the SDK uses low-level sensor API to minimize software-imposed latencies.   It is useful for performance profiling

https://github.com/IntelRealSense/librealsense/tree/master/tools/data-collect

Hi @RealitiesMediaCube  Do you require further assistance with this case, please?  Thanks!

@MartyG-RealSense I couldn't solve the problem yet. I will have to check for alternative camera solutions. I think you can close the issue, if there is no more information. Thank you for you support.

I just did some further research of your problem.  I found a comment where the Chief Technical Officer of the RealSense Group at Intel (agrunnet) states that if a preset file is loaded that does not provide settings for particular options then the camera uses the previous settings.
This would suggest that you may be able to achieve your goal of no gamma if you define a preset file that sets '0' gamma and then load the file in to apply the defined settings when your program launches.
https://github.com/IntelRealSense/librealsense/issues/1815#issuecomment-393936631
It looks as though gamma is set in the json with controls_color_gamma
https://github.com/IntelRealSense/librealsense/issues/3277#issuecomment-465066625
In the above link, the control_color settings for Hue and Brightness are set to zero, suggesting that gamma should be able to be zero too (I do not have confirmation of this though).

Hi @RealitiesMediaCube  Do you require further assistance with this case, please?  Thanks!

Hi, that could be an option, thanks for checking it out. I will have to try that out.
However, without knowing how gamma is actually influenced by the value, it likely does not solve my use case. I really need to know for sure that it is on or off, or which relationship exists so that I can remove the gamma correction from the image.

I carried out further research into your case.   Ultimately, if you want the certainly of obtaining raw images before they undergo processing then I believe that the earlier mentioned method of accessing and converting raw color frames may be the best option.
https://github.com/IntelRealSense/librealsense/issues/7856#issuecomment-733776542

Thanks I will try that.


fix: typo in nvim_set_hl

Self-explanatory. Enables setting color for a focused window once again.

Seems like there is an style issue


adds post install message

In the Sprint 1 demo, I noticed you used the phrase "tcramer is in the house" but we don't notify users that the gem is installed. This PR addresses that issue, and also adds a relevant sheriff emoji.

@mejackreed (jk :rofl: )

Could you please add a CONTRIBUTING.md to the repo so I can understand the tcramer community norms?


scripts: only suggest running clang-format on files that don't match …

…coding standard

I'm not quite sure what this does so I haven't approved it. However I was wondering about commit message formatting. Do you think we should adhere to gitlint rules (and commit signing) on this repo?
➜ gitlint
1: T1 Title exceeds max length (84>72): "scripts: only suggest running clang-format on files that don't match coding standard"
3: B6 Body message is missing


add reference node (wip)

Adds the reference node.  Very straightforward, adds another node type and then sets up the conditions for how to handle the new node type everywhere that the branch/action nodes are already handled.  I've also added a "type" property to all the nodes, so that they can be easily differentiated without needing complex logic.
The only thing that needs work is the cpg output.  I'm not sure exactly what's supposed to be output for the cpg, so for now this is a "wip".

still a bit shaky on what exactly we wanna see out of the CPG export.  The referenced pathways show up, but I'm not sure if they show up the right way.

The referenced pathways show up, but I'm not sure if they show up the right way.

So it seems like all the ActivityDefinitions are being added in the correct place (top level resources in Bundle.entry). But I only see two PlanDefinitions. This is how you have have it organized:
Bundle.entry
|- StrategyDefinition
     |-action
          |-PlanDefinition

The StrategyDefinition is also named PathwayA while the RecommendationDefinition is named PathwayB. The correct structure is
Bundle.entry
|- StrategyDefinition
     |- action
          |- PlanDefinition (PathwayA)
          |- PlanDefinition (PathwayB)

Let me work on an example for you. I think that might help.

ok so now the cpg export should be putting things in the right place.
Though there are still some outlying questions.  Like what happens when Pathway A conditionally references Pathway B an C, but pathway C also references pathway B conditionally?  The conditions aren't tied to a specific pathway reference.

What happens when Pathway A conditionally references Pathway B and C, but pathway C also references pathway B conditionally?

So ideally I think the result should be this:

Check if Pathway B has already been exported to a RecommendationDefinition
If it has don't convert it again (the resource is already in the bundle), but still add the reference in StrategyDefinition.action

This will probably require another data structure (probably just a map from PathwayId to RecommendationId)

I can't take a picture of it but the arrow in the sidebar next to pathway reference (which brings you to that pathway) should make the cursor a pointer like the other arrow buttons

Everything looks good to me in the exported CPG file except for two small details:

The StrategyDefinition.action don't reference the RecommendationDefinitions (via definitionCanonical)
The Library for PathwayB only has CQL and not ELM (or preconditions). The Library for PathwayA has both and when I export only PathwayB it too has both.

Exported Files using the new CPG exporter: CPG Pathways.zip
For everyone else's reference here are the documents I gave Keeyan:


Exception on multi-touch for useDrag and v10

Describe the bug
When using multi-touch and dragging an element, I receive an exception _event$currentTarget.contains is not a function.
TypeError: _event$currentTarget.contains is not a function. (In '_event$currentTarget.contains(e.target)', '_event$currentTarget.contains' is undefined)

Sandbox or Video
https://9bk8y.csb.app/
https://codesandbox.io/s/react-spring-carousel-9bk8y?file=/src/Slider.tsx
Can be reproduced with iOS (tested on 14) Safari and using one, then 2 fingers to drag the carousel.
Information:

React Use Gesture version: v.10.0.0-beta.14
Device: Safari IOS
OS: iOS
Browser Safari

Checklist:

[x ] I've read the documentation.

Should be fixed in 10.0.0-beta.15, let me know!

Fixed :) Thank you!!


Feature/code file manager

The modification of job manager allow loading file to submit script code to hardware platform. The persistant bug of form validation is now fixed.

This pull request seems to include a bunch of commits from an entirely different repository - perhaps that repository was cloned into a subdirectory of the hbp_neuromorphic_platform repository.
I have cherry-picked the relevant commits into master, so this PR can now be closed.


Update the mkdocs theme to material

The old theme kind of sucked. I was looking for a theme that was black and white to match with Arcadia's brand assets, which was why I picked lux, but it had weird rules like making headers all caps, not showing bold font, and making it hard to see linked text. The font itself was also sort of hard to read.
This PR changes to the material theme, which is much simpler and doesn't have any of the problems listed above. I also added more Arcadia assets so that the icons on the page and tabs are the arcadia triangle logos, and switched out the computational training image to be compatible with the brand assets as well (the old one wasnt 😬 ). Some of these images don't follow a principled naming scheme bc some needed specific path names to render correctly in the material theme, while others I could do whatever I wanted and it worked 🤷♀
I removed the css formatting file because it no longer did anything with this theme.

Looks good, in the future can you add screenshots to the PRs that involve UI changes?

yes will do!


Change installation disk space limit to 180Gi when using different disk for data.

Problem:

When selecting different disk for data, 250Gi limit for installation disk is too large.
Solution:

Change the limit to 180Gi when using different disk for data.
Related Issue:
https://github.com/harvester/harvester/issues/4133
Test plan:

ISO Installation
Single Disk

Build an ISO from this branch;
Boot into the installer, the installer should limit the installation disk at 250Gi minimum.

Multiple Disk

Build an ISO from this branch;
Boot into the installer, and use the same disk for installation disk and data disk;
The installer should limit the installation disk at 250Gi minimum;
When choosing different disk for installation and data, the installer should limit the installation disk at 180Gi minimum.

PXE Installation
Single disk

Build artifacts from this branch;
Use PXE to install Harvester artifacts you just built, when disk is smaller than 250Gi it should not continue.

Multiple disk

Build artifacts from this branch;
Prepare a Harvester config yaml, use another disk for data;
Use PXE to install Harvester artifacts you just built, when installation disk is smaller than 180Gi or data disk is smaller than 50Gi the installer should not continue.

FYI: It's ready for review.

Excellent job; the merge makes the UX better. Can we wait for one more enter before proceeding to the next page when using a data disk? It's too rushed to jump to the next page immediately:
https://github.com/harvester/harvester-installer/assets/1691518/3e0fc8d1-f7c2-425c-bb69-c71f875be1af

@mergify backport v1.2


Update xargo.

Ping me for an approval when ci has passed

bors try

@Dylan-DPC, ping. 😉

bors: r+


[Question] Is there a timeline on making include directives a stable feature?

Any information on the timeline or outstanding blockers for include directives being added as a stable feature to just would be great. This seems highly useful but I do not want to bake a feature into my workflow that will change or be removed.

Fair question!
Not really, unfortunately. The #!include feature as it is a bit sub-optimal. It does literal text inclusion, i.e., it just copies the contents of a file into the justfile that's including it, so error messages have the wrong line number and don't tell you which file the error is coming from. I'm not sure I want to stabilize it before we can make sure that line numbers are decent, and point to the right file.


Auto update gdax/coinbase currencies

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
Auto-generated assets/currencies/pairs for GDAX/Coinbase

Other information:
As per convo here - https://github.com/askmike/gekko/pull/2597.

I've tested a couple of imports and market watchers - it seems to work fine. I'd give it a good test though!! :)
Two issues -
1/ The importer shows an error as follows -
Some markets were unscannable, click here for details.
Unable to find datasets in the following markets:

gdax:GBP:ZRX

I know this to be correct and when you import date/start a bot then it doesn't offer up GBP/ZRX as an option so not sure why this is, but it doesn't seem to affect functionality.
2/ The last issue is the gdax/coinbase identy crisis! I've left the file as gdax.js but the new .json is coinbase-marketdata.js. I guess this is the next thing to look at?

I need to sort some bits out here where I seem to have working on an older version. Back shortly!


multi GPU training problem

When I use multiple GPUs for training, the inconsistent size of tensor types in instance_masks leads to data parallelism errors. How can I solve this error?

So instance_masks is just a list of tensors. The length is equal to the number of centers that are detected in a given sample (the local maxima of centerness), hence the variability. So one possible workaround would be to pad this list to a larger constant length.
One way to do this is to replace l.215 of src/panoptic/paps.py with:
instance_masks = [torch.zeros(1, device=stack.device) for _ in range(10000)]
and l.241 with :
instance_masks[i]=pred_mask

I think it may be caused by different sizes of tensors in instance_mask:

Ah ok I see. Can you make sure that this is the problem ? If you replace the tensors in instance_mask with a list of dummy tensors with constant shape, does code run on multi-gpu? Also, did you make sure that the code runs properly on a single gpu?
If this is really the problem, I don't think there is a simple workaround. One option would be to store the candidate instance masks in a tensor of shape NxHxW with as many channels as there are candidate instances. And maybe use sparse tensors to avoid useless memory usage. But that would involve a bit more changes to the code.

I make sure that the code runs properly on a single gpu. And I replace the tensors in instance_mask with a list of dummy tensors with constant shape, the code runs properly on multi gpu.

Do you have other solutions to complete multi GPU training?

Hi, I have solved this problem by replacing DataParallel model with DistributedDataParallel.


[bug] libarchive.13.dylib' (no such file),

Describe the bug
dyld[82730]: Library not loaded: /opt/homebrew/opt/libarchive/lib/libarchive.13.dylib
Referenced from: <3A92023B-E694-3FA1-B4FC-5056A4F353A7> /Users/suzhiwei/Documents/gitsource/fom/src-tauri/target/release/omg-ff
Reason: tried: '/opt/homebrew/opt/libarchive/lib/libarchive.13.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/libarchive/lib/libarchive.13.dylib' (no such file), '/opt/homebrew/opt/libarchive/lib/libarchive.13.dylib' (no such file)
Reproduction
No response
Expected behavior
No response
Full tauri info output
><EMAIL_ADDRESS>tauri
> tauri info

[✔] Environment
    - OS: Mac OS 15.0.1 x86_64 (X64)
    ✔ Xcode Command Line Tools: installed
    ✔ rustc: 1.81.0 (eeb90cda1 2024-09-04)
    ✔ cargo: 1.81.0 (2dbb1af80 2024-08-20)
    ✔ rustup: 1.27.1 (54dd3d00f 2024-04-24)
    ✔ Rust toolchain: 1.81-x86_64-apple-darwin (default)
    - node: 20.11.1
    - pnpm: 8.6.11
    - yarn: 1.22.22
    - npm: 10.2.4

[-] Packages
    - tauri 🦀: 2.1.1
    - tauri-build 🦀: 2.0.3
    - wry 🦀: 0.47.2
    - tao 🦀: 0.30.8
    - @tauri-apps/api : 2.1.1
    - @tauri-apps/cli : 2.1.0

[-] Plugins
    - tauri-plugin-fs 🦀: 2.1.0
    - @tauri-apps/plugin-fs : not installed!
    - tauri-plugin-dialog 🦀: 2.0.4
    - @tauri-apps/plugin-dialog : 2.0.1
    - tauri-plugin-shell 🦀: 2.0.2
    - @tauri-apps/plugin-shell : 2.0.1

[-] App
    - build-type: bundle
    - CSP: unset
    - frontendDist: ../dist
    - devUrl: http://localhost:1420/
    - framework: React
    - bundler: Vite

Stack trace
No response
Additional context
No response

did you add aditional rust dependencies? are you compiling for macos or windows? are dev and build affected?

What I need to make is a cross-platform application，My dependency information is as follows
@FabianLars
[package]
name = "omg-ff"
version = "0.1.0"
description = "A Tauri App"
authors = ["you"]
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[lib]
# The `_lib` suffix may seem redundant but it is necessary
# to make the lib name unique and wouldn't conflict with the bin name.
# This seems to be only an issue on Windows, see https://github.com/rust-lang/cargo/issues/8519
name = "omg_ff_lib"
crate-type = ["staticlib", "cdylib", "rlib"]

[build-dependencies]
tauri-build = { version = "2", features = [] }

[dependencies]
tauri = { version = "2", features = [] }
tauri-plugin-shell = "2"
tauri-plugin-dialog = "2"
blake3 = "1.5.0"
rusqlite = "0.26"
serde = { version = "1.0", features = ["derive"] }
log = "0.4"
serde_json = "1.0"
chrono = "0.4"
chardet = "0.2"
jieba-rs = "0.6"
encoding_rs = "0.8"
log4rs = { version = "1.2", features = ["file_appender", "pattern_encoder", "threshold_filter"] }
rayon = "1.5"
rust_search = "2.0.0"
memmap2 = "0.7"
docx-rs = "0.4"
calamine = "0.22"
tantivy = "0.22.0"
msoffice_pptx = "0.2.1"
leptess = "=0.14.0"
dirs = "5.0.1"
glob = "0.3.1"
tantivy-jieba = "0.11.0"
directories = "4.0"


search weights PATO terms higher than GO terms

I'm not sure why when I search amigo I am seeing PATO terms, this might be confusing for users ...there a quite a lot of PATO terms, which are not going to give any results.
The top hit for "phosphorylation" in AmiGO is PATO PATO:0002262

This will fall under the general search overhaul we need to do, such as https://github.com/geneontology/amigo/issues/102 and company.


xercesc: 3.2.4 -> 3.2.5

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/xerces-c/versions.
meta.description for xercesc is: Validating XML parser written in a portable subset of C++
meta.homepage for xercesc is: https://xerces.apache.org/xerces-c/
Updates performed

Version update

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 3.2.5 with grep in /nix/store/pkx8i2v3xf8lx624kqn6ic226q3nv91b-xerces-c-3.2.5
found 3.2.5 in filename of file in /nix/store/pkx8i2v3xf8lx624kqn6ic226q3nv91b-xerces-c-3.2.5

Rebuild report (if merged into master) (click to expand)

106 total rebuild path(s)

106 package rebuild(s)

First fifty rebuilds by attrpath
apacheHttpdPackages.mod_tile
apacheHttpdPackages_2_4.mod_tile
blahtexml
brewtarget
cloudcompare
dbxml
enigma
entwine
far2l
freecad
gdal
gdalMinimal
geant4
gmt
gplates
grass
haskellPackages.hgdal
libcutl
libdigidocpp
libe57format
liblinphone
linphone
mapcache
mapnik
mapproxy
mapserver
megaglest
merkaartor
meshlab
mysql-workbench
octavePackages.mapping
open-vm-tools
open-vm-tools-headless
openorienteering-mapper
opensaml-cpp
ovftool
paraview
pdal
perl536Packages.Tirex
perl538Packages.Tirex
postgresql12JitPackages.postgis
postgresql12Packages.postgis
postgresql13JitPackages.postgis
postgresql13Packages.postgis
postgresql14JitPackages.postgis
postgresql14Packages.postgis
postgresql15JitPackages.postgis
postgresql15Packages.postgis
postgresql16JitPackages.postgis
postgresql16Packages.postgis

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/pkx8i2v3xf8lx624kqn6ic226q3nv91b-xerces-c-3.2.5 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A xercesc https://github.com/r-ryantm/nixpkgs/archive/5e727c91bda06e5ba421fa279ae3c687054ddb63.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/5e727c91bda06e5ba421fa279ae3c687054ddb63#xercesc

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/pkx8i2v3xf8lx624kqn6ic226q3nv91b-xerces-c-3.2.5
ls -la /nix/store/pkx8i2v3xf8lx624kqn6ic226q3nv91b-xerces-c-3.2.5/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
nixpkgs-review took longer than 45m and timed out

Maintainer pings

CVE-2024-23807


the 'file list' window disappears

hello, I modify your code to add a small function. But now I find the file list window disappears that let me crazy. my code is here. can you give some advises?
thank you very much~
looking forward to your reply.

@tzutalin can you give some advises?

Hi @IvyGongoogle ,
Sorry about that. There is a workaround for that, you can try to click File/ResetAll.
I also submitted a change that 'file list' won't be hidden anymore. https://github.com/tzutalin/labelImg/commit/3391ab632ec20c9038b10dfb03c536ac58128465

@tzutalin it works, thanks a million


python bindings

Thanks for your great works.
I read the paper "GSLAM: A General SLAM Framework and Benchmark"
It indicates the python binding is provided for the modern deep learning platforms and developers prefer python for coding.
But I cannot any python bindings and python files.
is the development of this feature in progress?
Thank you

oops, I found the https://github.com/zdzhaoyong/GSLAM-Python
close.


202306-docs-notebook topn

A notebook on the topN engine in Apache Druid.
Description
This notebook walks through running a series of queries in approximate and accurate modes, with a comparison between the results of each.
This PR has:

[x] been self-reviewed.
[x] been tested in a test Druid cluster.
[ ] been tested in single node.
[ ] been tested in the notebook docker environment

@techdocsmith @hellmarbecker @sergioferragut @ektravel I warmly welcome you testing out and commenting on this new notebook.
I was unable to test it completely in the docker container for notebooks due to local resource constraints, but I did get a consistent experience on a cluster with both multiple and single historicals.

@sergioferragut I got the same feedback from @writer-jill so I've amended the flow in the notebook to call out needing to do the ingestion. The other notebook (events from the flight data) isn't suitable for use in this notebook unfortunately - it's built around the data in an all-defaults ingestion of the flight data, so I can't make that direct link (yet!)


Update uri and product attribute names

Updates docs to reflect changes made in Crux-UI.  Changes product url to uri and uri_type syntax.  And Changes product attributes.
fixes #76
fixes #164

Not merging yet as these things go live immediately. We'll release all the changes at the same time.


modify all settings with only ONE command

solved in https://github.com/valerio-vaccaro/SFYL-Wallet/pull/14


Enable PGD attack on PyTorch Faster-RCNN using np object arrays as input

Is your feature request related to a problem? Please describe.
I'm looking to run PGD on ART's PyTorch Faster-RCNN using the xView dataset. This dataset contains images of varying shapes, so in order to use batch_size > 1, the inputs are stored as 1D numpy object arrays. For example, if using batch_size=16, the input x has shape (16,) and dtype np.object. x[i] would be of dtype float (or int) and of shape (image_i_height, image_i_width, 3).
Describe the solution you'd like
Based on the testing I've done locally, it appears there are two places where modifications would need to be made:

The loss_gradient() of the PyTorch Faster-RCNN.  The attack will initially crash here. np.stack() requires that the elements in grad_list are of the same shape (which they are not, in my case). A solution could be to check if x.dtype == np.object, and if so, make grads a 1D object array and then loop through the i elements of the batch, assigning grads[i] to the gradient of image i.

The next function where the attack crashes is the _apply_perturbation() of fast_gradient.py, here. The same kind of logic in (1) above could be used to avoid np.clip() breaking.

Additional context
It appears this issue might've surfaced recently for another scenario (perhaps ASR?) because I see this kind of logic already implemented here and [here].(https://github.com/Trusted-AI/adversarial-robustness-toolbox/blob/dev_1.4.2/art/attacks/evasion/fast_gradient.py#L374)

Hi @lcadalzo Thank you very much for raising this feature request.
Yes, it's correct that PyTorchFasterRCNN is expecting images of the same size in a single array of shape NHWC.
I think this feature would be very useful for datasets with images of various sizes. We will have to investigate the compatibility of object arrays with varying element shape with the torchvision.models.detection.fasterrcnn_resnet50_fpn model, there could be an uncertainty in the models performance if we have to use batch_size=1.


Path no use in continue_pretrain.sh

https://github.com/princeton-nlp/LLM-Shearing/blob/8bb2f7c6b494edba50e52ee70ac334ec315cc43a/llmshearing/scripts/continue_pretrain.sh#L14C1-L14C5
Path is defined but not used. Where to pass the parameters of the model?

Thanks for catching this! Scripts are updated.

Hi mengzhou, i change the code and it prints load weight from my path, its ok
but it raise a problem that the loss keep all the same
It seems that the gradient is not calculated and it is not training normally.
And the loss (10) of the hot start after loading is much higher than the loss (2) I trimmed before, on the same data set.
Is this a normal phenomenon?
[batch=366/48000]:
         Train time/batch: 365
         Train time/sample: 93440
         Train time/batch_in_epoch: 365
         Train time/sample_in_epoch: 93440
         Train time/token: 382730240
         Train time/token_in_epoch: 382730240
         Train metrics/train/cc_weight: 0.2192
         Train metrics/train/github_weight: 0.0002
         Train metrics/train/book_weight: 0.0791
         Train metrics/train/stackexchange_weight: 0.0064
         Train metrics/train/wiki_weight: 0.0096
         Train metrics/train/arxiv_weight: 0.0010
         Train metrics/train/c4-rp_weight: 0.6845
         Train memory/current_allocated_mem: 9.7173
         Train memory/current_active_mem: 9.7173
         Train memory/current_inactive_mem: 0.6447
         Train memory/current_reserved_mem: 51.3280
         Train memory/peak_allocated_mem: 44.6420
         Train memory/peak_active_mem: 44.8020
         Train memory/peak_inactive_mem: 17.7940
         Train memory/peak_reserved_mem: 51.3280
         Train memory/alloc_retries: 0
         Train trainer/device_train_microbatch_size: 16
         Train loss/train/total: 10.3750
         Train loss/train/ce_loss: 10.3750
         Train metrics/train/LanguageCrossEntropy: 10.3750
         Train metrics/train/Perplexity: 32048.3164
         Train metrics/train/cc_LanguageCrossEntropy: 10.3750
         Train metrics/train/cc_count: 746
         Train metrics/train/github_LanguageCrossEntropy: nan
         Train metrics/train/github_count: 0
         Train metrics/train/book_LanguageCrossEntropy: 10.3750
         Train metrics/train/book_count: 250
         Train metrics/train/stackexchange_LanguageCrossEntropy: 10.3750
         Train metrics/train/stackexchange_count: 18
         Train metrics/train/wiki_LanguageCrossEntropy: 10.3750
         Train metrics/train/wiki_count: 33
         Train metrics/train/arxiv_LanguageCrossEntropy: nan
         Train metrics/train/arxiv_count: 2
         Train metrics/train/c4-rp_LanguageCrossEntropy: 10.3750
         Train metrics/train/c4-rp_count: 2279
         Train throughput/batches_per_sec: 0.0414
         Train throughput/samples_per_sec: 10.5889
         Train throughput/device/batches_per_sec: 0.0052
         Train throughput/device/samples_per_sec: 1.3236
         Train throughput/tokens_per_sec: 43372.1727
         Train throughput/device/tokens_per_sec: 5421.5216
         Train throughput/flops_per_sec:<PHONE_NUMBER>99506.0000
         Train throughput/device/flops_per_sec:<PHONE_NUMBER>99938.2500
         Train throughput/device/mfu: 0.3516
         Train time/train: 2.4628
         Train time/val: 0.0000
         Train time/total: 2.4628
[batch=367/48000]:
         Train time/batch: 366
         Train time/sample: 93696
         Train time/batch_in_epoch: 366
         Train time/sample_in_epoch: 93696
         Train time/token: 383778816
         Train time/token_in_epoch: 383778816
         Train metrics/train/cc_weight: 0.2192
         Train metrics/train/github_weight: 0.0002
         Train metrics/train/book_weight: 0.0791
         Train metrics/train/stackexchange_weight: 0.0064
         Train metrics/train/wiki_weight: 0.0096
         Train metrics/train/arxiv_weight: 0.0010
         Train metrics/train/c4-rp_weight: 0.6845
         Train memory/current_allocated_mem: 9.7173
         Train memory/current_active_mem: 9.7173
         Train memory/current_inactive_mem: 0.6447
         Train memory/current_reserved_mem: 51.3280
         Train memory/peak_allocated_mem: 44.6420
         Train memory/peak_active_mem: 44.8020
         Train memory/peak_inactive_mem: 17.7940
         Train memory/peak_reserved_mem: 51.3280
         Train memory/alloc_retries: 0
         Train trainer/device_train_microbatch_size: 16
         Train loss/train/total: 10.3750
         Train loss/train/ce_loss: 10.3750
         Train metrics/train/LanguageCrossEntropy: 10.3750
         Train metrics/train/Perplexity: 32048.3164
         Train metrics/train/cc_LanguageCrossEntropy: 10.3750
         Train metrics/train/cc_count: 803
         Train metrics/train/github_LanguageCrossEntropy: nan
         Train metrics/train/github_count: 0
         Train metrics/train/book_LanguageCrossEntropy: 10.3750
         Train metrics/train/book_count: 273
         Train metrics/train/stackexchange_LanguageCrossEntropy: 10.3750
         Train metrics/train/stackexchange_count: 20
         Train metrics/train/wiki_LanguageCrossEntropy: 10.3750
         Train metrics/train/wiki_count: 35
         Train metrics/train/arxiv_LanguageCrossEntropy: 10.3750
         Train metrics/train/arxiv_count: 3
         Train metrics/train/c4-rp_LanguageCrossEntropy: 10.3750
         Train metrics/train/c4-rp_count: 2450
         Train throughput/batches_per_sec: 0.0413
         Train throughput/samples_per_sec: 10.5839
         Train throughput/device/batches_per_sec: 0.0052
         Train throughput/device/samples_per_sec: 1.3230
         Train throughput/tokens_per_sec: 43351.5384
         Train throughput/device/tokens_per_sec: 5418.9423
         Train throughput/flops_per_sec:<PHONE_NUMBER>81446.8750
         Train throughput/device/flops_per_sec:<PHONE_NUMBER>72680.8594
         Train throughput/device/mfu: 0.3515
         Train time/train: 2.4695
         Train time/val: 0.0000
         Train time/total: 2.4695
[batch=368/48000]:
         Train time/batch: 367
         Train time/sample: 93952
         Train time/batch_in_epoch: 367
         Train time/sample_in_epoch: 93952
         Train time/token: 384827392
         Train time/token_in_epoch: 384827392
         Train metrics/train/cc_weight: 0.2192
         Train metrics/train/github_weight: 0.0002
         Train metrics/train/book_weight: 0.0791
         Train metrics/train/stackexchange_weight: 0.0064
         Train metrics/train/wiki_weight: 0.0096
         Train metrics/train/arxiv_weight: 0.0010
         Train metrics/train/c4-rp_weight: 0.6845
         Train memory/current_allocated_mem: 9.7173
         Train memory/current_active_mem: 9.7173
         Train memory/current_inactive_mem: 0.6447
         Train memory/current_reserved_mem: 51.3280
         Train memory/peak_allocated_mem: 44.6430
         Train memory/peak_active_mem: 44.8020
         Train memory/peak_inactive_mem: 17.7940
         Train memory/peak_reserved_mem: 51.3280
         Train memory/alloc_retries: 0
         Train trainer/device_train_microbatch_size: 16
         Train loss/train/total: 10.3750
         Train loss/train/ce_loss: 10.3750
         Train metrics/train/LanguageCrossEntropy: 10.3750
         Train metrics/train/Perplexity: 32048.3164
         Train metrics/train/cc_LanguageCrossEntropy: 10.3750
         Train metrics/train/cc_count: 861
         Train metrics/train/github_LanguageCrossEntropy: nan
         Train metrics/train/github_count: 0
         Train metrics/train/book_LanguageCrossEntropy: 10.3750
         Train metrics/train/book_count: 293
         Train metrics/train/stackexchange_LanguageCrossEntropy: 10.3750
         Train metrics/train/stackexchange_count: 22
         Train metrics/train/wiki_LanguageCrossEntropy: 10.3750
         Train metrics/train/wiki_count: 40
         Train metrics/train/arxiv_LanguageCrossEntropy: 10.3750
         Train metrics/train/arxiv_count: 4
         Train metrics/train/c4-rp_LanguageCrossEntropy: 10.3750
         Train metrics/train/c4-rp_count: 2620
         Train throughput/batches_per_sec: 0.0412
         Train throughput/samples_per_sec: 10.5394
         Train throughput/device/batches_per_sec: 0.0051
         Train throughput/device/samples_per_sec: 1.3174
         Train throughput/tokens_per_sec: 43169.4561
         Train throughput/device/tokens_per_sec: 5396.1820
         Train throughput/flops_per_sec:<PHONE_NUMBER>49509.1250
         Train throughput/device/flops_per_sec:<PHONE_NUMBER>43688.6406
         Train throughput/device/mfu: 0.3500
         Train time/train: 2.4766
         Train time/val: 0.0000
         Train time/total: 2.4766

Hi mengzhou, i change the code and it prints load weight from my path, its ok
but it raise a problem that the loss keep all the same
It seems that the gradient is not calculated and it is not training normally. And the loss (10) of the hot start after loading is much higher than the loss (2) I pruned before, on the same data set. Is this a normal phenomenon?
[batch=366/48000]:
         Train time/batch: 365
         Train time/sample: 93440
         Train time/batch_in_epoch: 365
         Train time/sample_in_epoch: 93440
         Train time/token: 382730240
         Train time/token_in_epoch: 382730240
         Train metrics/train/cc_weight: 0.2192
         Train metrics/train/github_weight: 0.0002
         Train metrics/train/book_weight: 0.0791
         Train metrics/train/stackexchange_weight: 0.0064
         Train metrics/train/wiki_weight: 0.0096
         Train metrics/train/arxiv_weight: 0.0010
         Train metrics/train/c4-rp_weight: 0.6845
         Train memory/current_allocated_mem: 9.7173
         Train memory/current_active_mem: 9.7173
         Train memory/current_inactive_mem: 0.6447
         Train memory/current_reserved_mem: 51.3280
         Train memory/peak_allocated_mem: 44.6420
         Train memory/peak_active_mem: 44.8020
         Train memory/peak_inactive_mem: 17.7940
         Train memory/peak_reserved_mem: 51.3280
         Train memory/alloc_retries: 0
         Train trainer/device_train_microbatch_size: 16
         Train loss/train/total: 10.3750
         Train loss/train/ce_loss: 10.3750
         Train metrics/train/LanguageCrossEntropy: 10.3750
         Train metrics/train/Perplexity: 32048.3164
         Train metrics/train/cc_LanguageCrossEntropy: 10.3750
         Train metrics/train/cc_count: 746
         Train metrics/train/github_LanguageCrossEntropy: nan
         Train metrics/train/github_count: 0
         Train metrics/train/book_LanguageCrossEntropy: 10.3750
         Train metrics/train/book_count: 250
         Train metrics/train/stackexchange_LanguageCrossEntropy: 10.3750
         Train metrics/train/stackexchange_count: 18
         Train metrics/train/wiki_LanguageCrossEntropy: 10.3750
         Train metrics/train/wiki_count: 33
         Train metrics/train/arxiv_LanguageCrossEntropy: nan
         Train metrics/train/arxiv_count: 2
         Train metrics/train/c4-rp_LanguageCrossEntropy: 10.3750
         Train metrics/train/c4-rp_count: 2279
         Train throughput/batches_per_sec: 0.0414
         Train throughput/samples_per_sec: 10.5889
         Train throughput/device/batches_per_sec: 0.0052
         Train throughput/device/samples_per_sec: 1.3236
         Train throughput/tokens_per_sec: 43372.1727
         Train throughput/device/tokens_per_sec: 5421.5216
         Train throughput/flops_per_sec:<PHONE_NUMBER>99506.0000
         Train throughput/device/flops_per_sec:<PHONE_NUMBER>99938.2500
         Train throughput/device/mfu: 0.3516
         Train time/train: 2.4628
         Train time/val: 0.0000
         Train time/total: 2.4628
[batch=367/48000]:
         Train time/batch: 366
         Train time/sample: 93696
         Train time/batch_in_epoch: 366
         Train time/sample_in_epoch: 93696
         Train time/token: 383778816
         Train time/token_in_epoch: 383778816
         Train metrics/train/cc_weight: 0.2192
         Train metrics/train/github_weight: 0.0002
         Train metrics/train/book_weight: 0.0791
         Train metrics/train/stackexchange_weight: 0.0064
         Train metrics/train/wiki_weight: 0.0096
         Train metrics/train/arxiv_weight: 0.0010
         Train metrics/train/c4-rp_weight: 0.6845
         Train memory/current_allocated_mem: 9.7173
         Train memory/current_active_mem: 9.7173
         Train memory/current_inactive_mem: 0.6447
         Train memory/current_reserved_mem: 51.3280
         Train memory/peak_allocated_mem: 44.6420
         Train memory/peak_active_mem: 44.8020
         Train memory/peak_inactive_mem: 17.7940
         Train memory/peak_reserved_mem: 51.3280
         Train memory/alloc_retries: 0
         Train trainer/device_train_microbatch_size: 16
         Train loss/train/total: 10.3750
         Train loss/train/ce_loss: 10.3750
         Train metrics/train/LanguageCrossEntropy: 10.3750
         Train metrics/train/Perplexity: 32048.3164
         Train metrics/train/cc_LanguageCrossEntropy: 10.3750
         Train metrics/train/cc_count: 803
         Train metrics/train/github_LanguageCrossEntropy: nan
         Train metrics/train/github_count: 0
         Train metrics/train/book_LanguageCrossEntropy: 10.3750
         Train metrics/train/book_count: 273
         Train metrics/train/stackexchange_LanguageCrossEntropy: 10.3750
         Train metrics/train/stackexchange_count: 20
         Train metrics/train/wiki_LanguageCrossEntropy: 10.3750
         Train metrics/train/wiki_count: 35
         Train metrics/train/arxiv_LanguageCrossEntropy: 10.3750
         Train metrics/train/arxiv_count: 3
         Train metrics/train/c4-rp_LanguageCrossEntropy: 10.3750
         Train metrics/train/c4-rp_count: 2450
         Train throughput/batches_per_sec: 0.0413
         Train throughput/samples_per_sec: 10.5839
         Train throughput/device/batches_per_sec: 0.0052
         Train throughput/device/samples_per_sec: 1.3230
         Train throughput/tokens_per_sec: 43351.5384
         Train throughput/device/tokens_per_sec: 5418.9423
         Train throughput/flops_per_sec:<PHONE_NUMBER>81446.8750
         Train throughput/device/flops_per_sec:<PHONE_NUMBER>72680.8594
         Train throughput/device/mfu: 0.3515
         Train time/train: 2.4695
         Train time/val: 0.0000
         Train time/total: 2.4695
[batch=368/48000]:
         Train time/batch: 367
         Train time/sample: 93952
         Train time/batch_in_epoch: 367
         Train time/sample_in_epoch: 93952
         Train time/token: 384827392
         Train time/token_in_epoch: 384827392
         Train metrics/train/cc_weight: 0.2192
         Train metrics/train/github_weight: 0.0002
         Train metrics/train/book_weight: 0.0791
         Train metrics/train/stackexchange_weight: 0.0064
         Train metrics/train/wiki_weight: 0.0096
         Train metrics/train/arxiv_weight: 0.0010
         Train metrics/train/c4-rp_weight: 0.6845
         Train memory/current_allocated_mem: 9.7173
         Train memory/current_active_mem: 9.7173
         Train memory/current_inactive_mem: 0.6447
         Train memory/current_reserved_mem: 51.3280
         Train memory/peak_allocated_mem: 44.6430
         Train memory/peak_active_mem: 44.8020
         Train memory/peak_inactive_mem: 17.7940
         Train memory/peak_reserved_mem: 51.3280
         Train memory/alloc_retries: 0
         Train trainer/device_train_microbatch_size: 16
         Train loss/train/total: 10.3750
         Train loss/train/ce_loss: 10.3750
         Train metrics/train/LanguageCrossEntropy: 10.3750
         Train metrics/train/Perplexity: 32048.3164
         Train metrics/train/cc_LanguageCrossEntropy: 10.3750
         Train metrics/train/cc_count: 861
         Train metrics/train/github_LanguageCrossEntropy: nan
         Train metrics/train/github_count: 0
         Train metrics/train/book_LanguageCrossEntropy: 10.3750
         Train metrics/train/book_count: 293
         Train metrics/train/stackexchange_LanguageCrossEntropy: 10.3750
         Train metrics/train/stackexchange_count: 22
         Train metrics/train/wiki_LanguageCrossEntropy: 10.3750
         Train metrics/train/wiki_count: 40
         Train metrics/train/arxiv_LanguageCrossEntropy: 10.3750
         Train metrics/train/arxiv_count: 4
         Train metrics/train/c4-rp_LanguageCrossEntropy: 10.3750
         Train metrics/train/c4-rp_count: 2620
         Train throughput/batches_per_sec: 0.0412
         Train throughput/samples_per_sec: 10.5394
         Train throughput/device/batches_per_sec: 0.0051
         Train throughput/device/samples_per_sec: 1.3174
         Train throughput/tokens_per_sec: 43169.4561
         Train throughput/device/tokens_per_sec: 5396.1820
         Train throughput/flops_per_sec:<PHONE_NUMBER>49509.1250
         Train throughput/device/flops_per_sec:<PHONE_NUMBER>43688.6406
         Train throughput/device/mfu: 0.3500
         Train time/train: 2.4766
         Train time/val: 0.0000
         Train time/total: 2.4766

I trained on a single card and it can run normally. It seems that there is still a compatibility issue with the composer code on multiple cards. There may be issues with model distributed loading and data sharding. Is it possible to run this code using the deepspeed framework?

Hey I think there is a bug here -- not sure what it is. I am working on it now.

Hi the issue the solved! It stemmed from the init_device setup from the yaml files. Originally it was set to meta, yet it causes unexpected issues in loading the model. Now it is switched to cpu. Ideally, we would love to support meta loading as it is faster, but I am not sure how to integrate it with the current codebase yet. Thanks for spotting this!
PS. The codebase has been changed a lot after my runs for the paper (mostly to make it adaptable for the update to date composer package.) So there could be issues here and there, as it's not fully tested. Thanks for your work on this!

Hi mengzhou,
Thank you very much for your quick fix, I will try again tomorrow to test your new code.
It is normal to have problems in the code, but fortunately, I have basically run through the framework process now. Although the process is a bit difficult, your work is very interesting and meaningful, so I am happy to participate in the realization process.
Another happy thing is that I have found a solution to the problem of "train.fit" not reporting errors and multi-card process blocking that bothered me before. I raised an issue in the composer library and found the relevant solution with their help. When the code in this warehouse is interrupted, the shared memory will be blocked by some bugs in Steam. The zombie memory needs to be cleaned up in time to avoid lagging. For specific circumstances, please refer to the following two issues.
https://github.com/mosaicml/llm-foundry/issues/436#issuecomment-1627712085
https://github.com/mosaicml/composer/issues/2733
I wrote a script for cleaning. If you need it, I will create a new branch and merge it in.

Hi! Awesome :) Feel free to start a PR on it!

Hey @Longyichen , I am also facing issues with pruning getting stuck after 'Starting Training'.
Could you guide me with the changes you made to solve this issue?

Hey @Longyichen , I am also facing issues with pruning getting stuck after 'Starting Training'.
Could you guide me with the changes you made to solve this issue?

@argitrage see https://github.com/princeton-nlp/LLM-Shearing/pull/30


literal data truncated in header exception

Since upgrading to 1.79 I get a literal data truncated in header exception when reading LiteralDataPacket.
I'm not sure but I think this might be down to the changes in reading the modification time.
Before 1.79 the code was...
(((long)in.read() << 24) | (in.read() << 16) | (in.read() << 8) | in.read()) * 1000L
which would cope with a maximum date of new Date(Long.MAX_VALUE) // Sun Aug 17 07:12:55 GMT 292278994
Now it is...
(long) ((in.read() << 24) | (in.read() << 16) | (in.read() << 8) | in.read()) * 1000L
which would cope with a maximum date of new Date(Integer.MAX_VALUE * 1000L) // Tue Jan 19 03:14:07 GMT 2038
I was using random data 1 Jan 2074 for testing and hit this as a negative date throws the exception.

Fixed in https://github.com/bcgit/bc-java/commit/8902307a09adc76195d7e3985a4d22660bd102c5


It won't cross compile for Linux/amd64 on MacOs.

Error message on MacOS:
../../../gopkg.in/kothar/brotli-go.v0/dec/decode.go:36:2: no buildable Go source files in /Users/mehmetgerceker/workspace/goWorkspace/src/gopkg.in/kothar/brotli-go.v0/shared
Error message on Linux:
main.go:14:2: C source files not allowed when not using cgo or SWIG: bit_reader.c decode.c huffman.c state.c streams.c
../../../gopkg.in/kothar/brotli-go.v0/dec/decode.go:36:2: no buildable Go source files in /home/mehmetg/gohome/src/gopkg.in/kothar/brotli-go.v0/shared
Project: github.com/mehmetg/simpleJSONService

Hi, this package wraps the C/C++ implementation of Brotli, which means you can't use the usual GOOS=xxxx cross-compilation method. I think it's possible to cross compile with cgo, but I've not tried it myself.
https://gist.github.com/steeve/6905542
I imagine it would be easier to install Docker for Mac and use a suitable image containing go to do the actual compilation, rather than trying to cross-compile directly. You could still get that to map your working directory and run everything from the command line.
Alternatively, @dsnet has a pure-go implementation of the decompression algorithm, which may be easier for your use case as you don't need to compress anything: https://github.com/dsnet/compress/tree/master/brotli

Thank you for the pointers.
For now I'll stick with the VM/Docker solution as it is the fastest.
Do you expect performance gain/loss between the wrapper and pure go solution?

Currently the pure go version is about 2-3x slower than the pure C version, but I'm not sure how much overhead this wrapper adds to that (it should be minimal).
dsnet has done some extensive benchmarking of his implementation here:
https://github.com/dsnet/compress/wiki/Performance-Metrics#brotli

I've had success cross-compiling cgo code using gox - see https://github.com/itchio/butler/blob/master/scripts/ci-build.sh for how we use it in production.

FYI, you can compile brotli as a shared object, which is significantly easier to dynamically link in than the static approach you taking here.
See https://github.com/dsnet/compress/blob/619c68bc20d0be0da0563304ea093066c4190cad/internal/cgo/brotli/brotli.go


Linking plot axis across multiple plots

I am trying to create an audio time signal plot combined with a time-frequency spectrogram that are linked in time x-axis, but not in the y-axis. This is so that when zooming in on one of the plots, the same range of time is displayed in the other plot. I have searched through the examples, and all examples seem to be linked in both x, and y axis. In my example below, the two plots worked correctly when shown separately, but when combined, the y-axis range for the time signal plot is forced to follow the spectrogram.
I think this is an important feature that should be possible, and documented and/or have a clean example for reference.
My example notebook code:
import numpy as np
import holoviews as hv
from scipy.fftpack import fft
from holoviews import opts, Curve, Image

hv.extension('bokeh')
plot_width = 900
plot_height = plot_width // 2
opts.defaults(
    opts.Curve(
        height=plot_height,
        width=plot_width,
        line_width=1.0,
        line_alpha=.5,
        show_grid=True,
    ),
    opts.Image(
        height=plot_height,
        width=plot_width,
    ),
)

Time signal generation and plot:
FS = 48000
T = 2
N = int(FS * T)
FZ = 5000  # Hz
t = np.linspace(0, T, N)
x = np.sin(2*np.pi*FZ*np.linspace(0,N-1,N) / FS) * .5
x += np.random.rand(N) * .5
x_plt = Curve((t, x)).opts(title='Time Signal')
x_plt

Spectrogram plot:
FFT_LEN = 1024
K = int(FFT_LEN // 2 + 1)
FFT_SHIFT = 512
FFT_WIN = np.hanning(FFT_LEN)
M = N // FFT_SHIFT - 1
X = np.zeros((M, K))
for m in range(M):
    x_idx = m * FFT_SHIFT
    xx = x[x_idx:x_idx+FFT_LEN] * FFT_WIN
    X[m, :] = np.log10(np.abs(fft(xx)[:K])+1e-21)
X_plt = Image(np.flipud(X.T), bounds=(0, 0, T, FS/2.)).opts(title='Spectrogram')
X_plt

Combining time and spectrogram plots:
(x_plt + X_plt).cols(1)

By default, HoloViews assumes that multiple plots with the same dimensions should be scaled together and linked to more easily show the relationship between the data in each one. But it won't do that if the dimensions are different, and here the problem is that you haven't told HoloViews that the dimensions are different.  Just label each dimension properly (not just x and y, but "time", "frequency", amplitude", etc., and then it should behave the way you expect:
x_plt = Curve((t, x),"time","amplitude").opts(title='Time Signal')
...
X_plt = Image(np.flipud(X.T),['time','frequency'], bounds=(0, 0, T, FS/2.)).opts(title='Spectrogram')
...

 I think there are already examples like that, but please open a PR clarifying this issue in the docs if you can see a good place for it!


mongo

Expected behavior
Actual behavior
Information about the Issue
Steps to reproduce the behavior

...
...

I have solved this problem. Thanks for your help.


블로그에 css글 추가

Summary

Context

@pkb9239 you must not submit a PR here, you should fork the repository and keep the modifications in your fork.
Additionally, read carefully the documentation and pay attention to what you share online, like personal email, Google tracking IDs, and so forth... I'm sure @mmistakes will close this.

sorry. I'm still a beginner and made a mistake.


Added constructors to provide CellTreeMessages and defaultNodeSize.

See: https://github.com/ArcBees/gwtquery-droppable-plugin/issues/1

Thanks for the contribution! We will review shortly :D

LGTM

LGTM


Feri the Ferret Stickers

1000 stickers received from Sticker Robot. :D
Social media posts made on Facebook, Twitter, and LinkedIn. Should post again in the future.

To-Do

Ask original artist of Feri for their address in the UK.
Mail out sticker packages to various friends, contacts, and family.
Consider posting a free Feri stickers offer (for US residents) on the project page.

Free sticker info now listed on the main project page.
Social media posts done and will keep distributing physical stickers at various meetups when the opportunity arises.