Make a Note on Licensing, Sources, and Copyrights

Feature Branch
Current feature branch for this issue: not created yet.
Progress

[ ] Make note of external sources of code and copyrights/licences.
[ ] Update/add comments when additional sources are used.

Background
Because of the nature of the application, it is going to be based upon various sources, including algorithms, Bitcoin Improvement Proposals (BIPs), Satoshi Labs Improvement Proposals (SLIPs), Requests For Comments (RFCs), articles, code samples, my original code, etc.
Ordinarily, I would just add the copyright lines to the LICENSE file if published under a compatible licence and/or requires the copyright notice line(s) be included in any subsequent work.
The purpose of this issue is to make a note of the sources of code the code in this repository is built upon, to expand upon the copyright lines in the LICENSE file.

Z85
The implementation of the Z85 specification in this repository is based on the Z85 reference implementation written in C.
The Z85 specification is licensed GPL version 3 or later and has the following copyright notice:
Copyright (c) 2013 iMatix Corporation.

The reference implementation is licensed MIT and has the following copyright notice:
Copyright (c) 2010-2013 iMatix Corporation and Contributors

My C# implementation (and any subsequent changes) is based on the reference implementation and has the following copyright notice:
Copyright (c) 2010-2013 iMatix Corporation and Contributors
Copyright (c) 2021-2022 John Cook

RelayCommand, EventRaiser, EventArgs<T>
The classes RelayCommand, EventRaiser, EventArgs<T> (collectively, the RelayCommand classes), are based on the code in the Using RelayCommand / ICommand to handle events in WPF and MVVM article, with the Web site author stating the following on the site's about page:

My postings have been donated to the public domain, including any new posts I write, so they’re not copyrighted. You’re welcome to republish, use, or share them in any way you desire. There’s no need to ask permission in advance.
Andy

The idea of RelayCommand is likely based upon Josh Smith's  MSDN article Patterns - WPF Apps With The Model-View-ViewModel Design Pattern. GitHub user djangojazz has copied the code used in that article to a GitHub repository.
Whilst looking for the original source, I noticed some links in Google with titles/descriptions suggesting a potential issue with the original source, that suggest a possible memory leak. I may need to look more at the issue, see if the issue exists in the code I'm using, and work out how to mitigate the issue.
BIP-0039
BIP-0039, or BIP39, is Bitcoin Improvement Proposal number 39, with title Mnemonic code for generating deterministic keys.
BIP39's authors are Marek Palatinus, Pavol Rusnak, Aaron Voisine, Sean Bowe.
The American English word list from BIP39 is used in some form in this application (currently, by functions TryGetIntFromWord() and TryGetWordFromInt()), and the test vectors used in the Tests package are those from Trezor that are referenced in the BIP39 proposal.
All of the code in this application implementing BIP39 is my own. Given the amount of maths involved in splitting bytes into 11 bit chunks, creating/confirming the checksum, et cetera, there is a reason my implementation is heavily commented (it took me a while to get the maths working, particularly the bit shifts).
Having said that, my code implements the maths in BIP39 in a way that looks obvious, so I consider the code for the maths to be an obvious implementation and probably not subject to copyright for reason of obviousness. Likewise, there are probably some obvious optimisations that could be made to my implementation that I would eventually notice if I looked at the code long enough.
SLIP-0021
SLIP-0021, or SLIP21, is Satoshi Labs Improvement Proposal 21, with title Hierarchical derivation of symmetric keys.
SLIP21's authors are Andrew R. Kozlik, Ondrej Vejpustek, Pavol Rusnak.
All of the code in this application implementing SLIP21 is my own, implementing in code the maths described in SLIP-0021 with references to BIP-0032.
The test vectors used in the Tests package are those from SLIP-0021.
ViewModelBase
The class ViewModelBase is based upon various sources, that seem to be based themselves upon various sources such as the BindableBase class in MIT-licensed Prism framework.
Prism has the following copyright notice:
Copyright (c) .NET Foundation

SecureString to char[]
At present, my method of converting a SecureString to a byte[] and then a UTF-8 char[] is based on a StackOverflow answer to a question with title "C# convert SecureString to UTF-8 byte[] securely".
The author of the answer follows up with a comment, linking to the class in their GitHub repository that the answer is based upon, with the repository using The Unlicense.
MVVM Pattern
The Model-View-ViewModel pattern (MVVM) is one way of achieving a separation of concerns, and one of the typical patterns used in C# WPF development. The Model-View-Container (MVC) pattern is the other typical pattern.
I don't particularly like using frameworks when programming because I find they tend to obscure what the code is really doing, so the namespace/folder layout used, the naming of files and classes, and which files are used for which functionality, is based upon what various sources say is "the right way".
Some of my other C# WPF projects that have reached the point of having a release build package created have gone through the process of being refactored into using the MVVM pattern, but I haven't yet settled on what my "right way" is. For example, the start of my refactoring had me turn to my 16 page note document MVVM that is some handwritten source code based on a YouTube tutorial series about MVVM, but I have deviated from it in my MVVM refactoring branch.
Switching to the MVVM pattern, or other refactoring of code to split things into smaller files, is not really a question about licensing or copyright. It is something that happens when a project reaches the point where the file(s) are becoming so long they are becoming unwieldly.


Permissions always returned denied even though user approved

Describe the bug
I'm having issues with the permissions. The users are prompted with the approve permissions to share read access to their StepCount, Workout, DistanceWalkingRunning and FlightsClimbed data. Even though they approve all I get status denied. All of them are set to approved when checking the Apple Health app and sharing tab.
To Reproduce

Initialize AppleHealthKit.initHealthKit ... when navigating to that component
Approve all read permissions when that is prompted
Checking auth status is status 1 (SharingDenied)

Expected behavior
Once user approved permissions auth status should be set to 2 (SharingAuthorized) and data should be fetched from Apple Health.
Screenshots

Smartphone (please complete the following information):

Device: iPhone 11 Pro Max
OS: iOS16.1
Version: 2

Additional context

Latest react-native-health version is being used
Same issue occurs when testing on multiple devices (e.g iPhone 13 Pro, iOS 16.1.1)

Hello @amandadahlin did you manage to work around this? Is this still an issue?

+1 happening here

I see the same issue as well..

Hey ppl 😄
After taking a better look at this issue, I need to clarify some things:

In HealthKit, the term share permissions is synonymous to write permissions, so when the word share appears in the docs, it can be read as write. It's a little bit confusing 😅
It's impossible to check the read permissions for each type. HealthKit doesn't allow it for privacy reasons. Per HealthKit docs:

To help prevent possible leaks of sensitive health information, your app cannot determine whether or not a user has granted permission to read data. If you are not given permission, it simply appears as if there is no data of the requested type in the HealthKit store. If your app is given share permission but not read permission, you see only the data that your app has written to the store. Data from other sources remains hidden.

Our docs mentions that getAuthStatus should return all types as SharingAuthorized, although this isn't what is currently happening. Both the read and the write fields of the getAuthStatus return are being populated by the write authentication status.
tldr: Lib is bugged and returning the write auth status as the read auth status.
Honestly, I believe you can just ignore the read auth status, as HealthKit doesn't even allow us to get them.

To contributors:
I believe we should deprecate this function in favor of one without anything related to read permissions, as it's pointless and just misleading. I can do that in the next few days, but contributions until that are welcome 😄.

I have noticed the following behavior in addition to the above described behavior. I currently am sharing read permissions for HeartRate, RestingHearRate, SleepAnalysis, and ActivitySummary. Here is my request:
const permissions = {
        permissions: {
          read: [
            'HeartRate',
            'RestingHeartRate',
            'SleepAnalysis',
            'ActivitySummary',
          ],
          write: ['HeartRate', 'HeartRateVariability'],
        },
      }

      AppleHealthKit.getAuthStatus(permissions, (err, results) => {
        if (err) {
          console.log('error getting Healthkit auth status: ', err)
        }

        if (results) {
          console.log('results: ', results)
        }
      })

And this is the response:
results:  Object {
  "permissions": Object {
    "read": Array [
      1,
      1,
      1,
      1,
    ],
    "write": Array [],
  },
}

So it looks like in addition to receiving the confusing 1 value for reads, it looks like I'm not getting the same length response for other types (see where I have HeartRate which is authorized as read but not write, while HeartRateVariability - of which I have neither read nor write - doesn't return any value for write)

The read permissions return 1 regardless of the state, so we're checking the steps from the last 48 hours and marking them as denied if the step count is zero. It's working very well. As for getAuthStatus, it is useful because it returns 0 if Apple Health hasn't been initiated before, or if it was closed without approving or denying the permissions.

getAuthStatus

The read permissions return 1 regardless of the state, so we're checking the steps from the last 48 hours and marking them as denied if the step count is zero. It's working very well. As for getAuthStatus, it is useful because it returns 0 if Apple Health hasn't been initiated before, or if it was closed without approving or denying the permissions.

checking the steps is smart I might try that

yup we actually did a similar workaround, but checking for sleep data (as it was the data we needed)

Having the same issue, still, after unchecking permissions in the Health settings for the app - it still shows "1" for permissions even though the user has since unchecked the specific permissions.
Will this be fixed in the future? Or do I code around it since its been an issue for years? I'm confused on what I should do here because its been broken for so long it seems.


routing: add cltv limit

This PR adds a maximum cltv limit for payments, similar to the currently existing fee limit.

Somewhat unrelated -- I noticed that we check for the fee limit when constructing the route from the path. Is there a reason that check is still needed since we already do it within path finding?

I don't think there is. It is a sanity check, but I think it can be removed. I didn't bother adding another one for cltvLimit.

@wpaulino PTAL

I don't think there is. It is a sanity check, but I think it can be removed. I didn't bother adding another one for cltvLimit.

Gotcha, would be nice to remove it in a follow-up PR then.

I don't think there is. It is a sanity check, but I think it can be removed. I didn't bother adding another one for cltvLimit.

Gotcha, would be nice to remove it in a follow-up PR then.

Yes, I can do that.

@wpaulino ptal

@halseth  comments replied to, ptal

Rebased

I don't think there is. It is a sanity check, but I think it can be removed. I didn't bother adding another one for cltvLimit.

Gotcha, would be nice to remove it in a follow-up PR then.

It is happening in #2497

Needs a rebase.

ptal


Scope cloning method was not setting the user

:scroll: Description

:bulb: Motivation and Context
Fixes https://github.com/getsentry/sentry-dart/issues/1010
:green_heart: How did you test it?
:pencil: Checklist

[X] I reviewed submitted code
[X] I added tests to verify changes
[ ] I updated the docs if needed
[X] All tests passing
[X] No breaking changes

:crystal_ball: Next steps

Codecov Report
Base: 91.14% // Head: 95.55% // Increases project coverage by +4.40% :tada:

Coverage data is based on head (4c02b58) compared to base (ef7527b).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1013      +/-   ##
==========================================
+ Coverage   91.14%   95.55%   +4.40%
==========================================
  Files           9        2       -7
  Lines         192       45     -147
==========================================
- Hits          175       43     -132
+ Misses         17        2      -15

Impacted Files
Coverage Δ

dio/lib/src/dio_event_processor.dart

dio/lib/src/sentry_dio_client_adapter.dart

dio/lib/src/failed_request_interceptor.dart

dio/lib/src/breadcrumb_client_adapter.dart

dio/lib/src/tracing_client_adapter.dart

dio/lib/src/sentry_dio_extension.dart

dio/lib/src/sentry_transformer.dart

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


How to use front camera to come out when first executed?

I'm using RtpService(backgroundexample).
I wanna set for front camera to come out when first executed.
I did "RtpService.Companion.startPreview (CameraHelper.Facing.FRONT);" at surfaceChanged, but it didn't work.
RtpService.Companion.init(this);

Intent intent2 = new Intent(getApplicationContext(), RtpService.class);
 intent2.putExtra("endpoint", rtmpUrlForSending);
startService(intent2);

 openGlView.getHolder().addCallback(this);

@Override
    public void surfaceChanged(SurfaceHolder surfaceHolder, int i, int i1, int i2) {
        RtpService.Companion.setView(openGlView);
        RtpService.Companion.startPreview(CameraHelper.Facing.FRONT);
    }

//RtpService.kt edited
fun startPreview(cameraFacing: CameraHelper.Facing ) {
       camera2Base?.startPreview(cameraFacing);
}

This should be fixed in last commit: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/commit/d00fe498812d455287e49e54bc7ee68e1fa62ca4
Use this gradle meanwhile I upgrade version:
	        implementation 'com.github.pedroSG94:rtmp-rtsp-stream-client-java:d00fe49881'


MarkerLayer: tell type system that all features have IDs

This fixes some Typescript warnings, and improves type safety in general.

Thanks! I’ll put out a new release later tonight.


Istio-operator not working on Mac m1

Bug Description
I try to install istio on my mac m1 by istio-operator,but it doesn't seems to work.

And I can't open the log of istio-operator.

Version
% istioctl version
no running Istio pods in "istio-system"
1.12.4
% kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/arm64"}

Additional Information
No response

arm 上支持 istioctl，但 arm 集群上不支持 istio。支持有望很快到来。有关一些讨论，请参阅#36762。

Thanked your replay.
Hope to use istio on M1 soon

you can try https://github.com/querycap/istio, but I'm not 100% it will work.

you can try https://github.com/querycap/istio, but I'm not 100% it will work.

Thx , i will try

Confirm querycap istio builds works for me. An option in conjunction with capistio, kind and docker-for-mac is to add this patch so ingress/egress gateways get scheduled.
`kubeadmConfigPatches:

|
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "kubernetes.io/arch=amd64"`


1.19 updgrader ignores seeds in Dimension json files

The seed field in dimension json files remains even though it should be removed, and importing into the Dimension Generator removes it also.
e.g.
{
	"type": "orc:void",
	"generator": {
		"type": "flat",
		"seed": -1148089079,
		"settings": {
			"biome": "the_void",
			"features": false,
			"layers": [
				{
					"block": "air",
					"height": 1
				}
			],
			"structures": {
				"structures": {}
			}
		}
	}
}

should become
{
	"type": "orc:void",
	"generator": {
		"type": "flat",
		"settings": {
			"biome": "the_void",
			"features": false,
			"layers": [
				{
					"block": "air",
					"height": 1
				}
			]
		}
	}
}

Hm, I wonder where I ever got that from then haha


Write Tests

Need to test for:
[ ] installation and presence of awsagent binary
[ ] removal and absence of awsagent binary
[ ] PGP failures

Tests covered by https://github.com/eternaltyro/chef-aws-inspector/pull/2


Add Endb

Endb

🗃 Simple key-value storage with support for multiple backends.

Features

Easy-to-use: Endb has a simplistic and easy-to-use promise-based API.
Adapters: By default, data is stored in memory. You can optionally install and utilize a "storage adapter".
Third-Party Adapters: You can optionally utilize third-party adapters or build your own. Endb will integrate the third-party adapter and handle complex data types internally.
Namespaces: Namespaces isolate elements within a database.
Custom Serializers: Utilizes data serialization to ensure consistency across different backends.
Embeddable: Endb is designed to be easily embeddable inside modules.
Data Types: Handles all the JSON types including Buffer.
Error-Handling: Connection errors are sent through, from the adapter to the main instance; connection errors won't exit or kill the process.
Custom Serializers: Utilizes data serialization methods that encode Buffer data as a base64-encoded string, and decode JSON objects which contain buffer-like data (either as arrays of numbers or strings) into Buffer instances to ensure consistency across different backends.
Optionally, pass your own data serialization methods to support extra data types.
Data Types: Handles all the JSON types including Buffer using its data serialization methods.
Error-Handling: Connection errors are sent through, from the adapter to the main instance; connection errors won't exit or kill the process.

Useful Links

Documentation
GitHub
NPM

By submitting this pull request, I promise I have read the contribution guidelines twice and ensured my submission follows it. I realize not doing so wastes the maintainers' time that they could have spent making the world better. 🖖
⬆⬆⬆⬆⬆⬆⬆⬆⬆⬆

/thoughts

Duplicate of #965

Hey @sindresorhus,
I have deleted the pull-request branch for the previous pull-request, not only that, but Endb has changed a lot since then. Therefore, I created a new pull-request addressing all the issues with the previous submission.

Last time, I asked you a question and you just ignored it. You have still not answered it.

Last time, I asked you a question and you just ignored it. You have still not answered it.

You see Endb has changed its implementation, that is, from ORM type database to key-value type database. Thus, the question is no longer valid. Further, I am sorry for that.

Last time, I asked you a question and you just ignored it. You have still not answered it.

I am sorry for not answering that question. You see Endb has improved and developed significantly and as it did, the question became unreasonable and unrelated to Endb. Again, I am sorry for not reporting this earlier. I have replied to that comment recently.


Unable to connect YH002-A blinds controller to vtrust-flash SSID / SmartConfig not working as expected

I am not certain the brand but it is basically the Zemismart Blinds Controller and the model is YH002-A. I have opened the device and confirmed it contains a ESP8266 (TYWE1S), firmware is unknown. Device is brand new out of the box and never been connected to any network or app.
Running Raspberry Pi OS Lite (Legacy) (Buster) (Released 2022-04-04) on a Raspberry Pi 3B v1.2. OS is fully updated, prereqs for tuya-convert are installed. I am connected to the vtrust-flash SSID (which by default has no password) from my Windows PC (I also have tried with my Android phone as well). I can see my PC/phone join the WiFi from the logs but none of my YH002-A devices will connect to the vtrust-flash SSID for any reason. Device is in EZ pairing mode (held middle button for ~5 seconds then middle light flashes red fast). Tested with 2 of these new devices so far and getting the same results. It's as if they never receive the SmartConfig packets and/or it never connects to the vtrust-flash SSID. Any help is greatly appreciated
`IMPORTANT

Connect any other device (a smartphone or something) to the WIFI vtrust-flash
This step is IMPORTANT otherwise the smartconfig may not work!
Put your IoT device in autoconfig/smartconfig/pairing mode (LED will blink fast). This is usually done by pressing and holding the primary button of the device
Make sure nothing else is plugged into your IoT device while attempting to flash.
Press ENTER to continue

======================================================
Starting smart config pairing procedure
Waiting for the device to install the intermediate firmware
Put device in EZ config mode (blinking fast)
Sending SSID                  vtrust-flash
Sending wifiPassword
Sending token                 00000000
Sending secret                0101
................
SmartConfig complete.
Resending SmartConfig Packets
.................
SmartConfig complete.
Resending SmartConfig Packets
.................
SmartConfig complete.
Resending SmartConfig Packets
................
SmartConfig complete.
Resending SmartConfig Packets
.................
SmartConfig complete.
Resending SmartConfig Packets
.................
SmartConfig complete.
Resending SmartConfig Packets
................
SmartConfig complete.
Resending SmartConfig Packets
.................
SmartConfig complete.
Resending SmartConfig Packets
.................
SmartConfig complete.
Resending SmartConfig Packets
.................
SmartConfig complete.
Resending SmartConfig Packets
..............
Timed out while waiting for the device to (re)connect
Attempting to diagnose the issue...
No ESP82xx based devices connected according to your wifi log.
Here is a list of all the MAC addresses that connected:
0a:91:bf:3f:7b:bf <--- This is my Android Phone
80:32:53:d1:a7:09 <--- This is my Windows PC
If you see your IoT device in this list, it is not an ESP82xx based device.
Otherwise, another issue may be preventing it from connecting.
For additional information, check the *.log files inside the scripts folder.
Please include these logs when opening a new issue on our GitHub issue tracker.
Do you want to try flashing another device? [y/N]`
smarthack-mqtt.log
smarthack-psk.log
smarthack-udp.log
smarthack-web.log
smarthack-wifi.log

Figured out the issue through more troubleshooting. The instructions included with this unit are really terrible. EZ pairing mode is actually achieved on this device by holding the left button (Up) and the center button (Stop). until the blue and red lights start blinking. Once I did this and re-ran tuya-convert it got the packets, connected, and I was able to determine the firmware is too new for tuya-convert. Closing this issue.


Is there an Azure Policy to export Microsoft Defender Alerts with Subscription Scope?

Hi,
i was wondering if Azure also offers a built-in policy like:

Deploy export to Event Hubs for Microsoft Defender for Cloud alerts and recommendations
cdfcce10-4578-4ecd-9703-530938e4abcb

But with subscription scope, so that all security alerts from all resources within all resourcegroups of this subcription will be gathered and pushed to an event-hub. the policy above needs a resourcegroup as a required attribute.
Thanks in advance for answering !

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: efaf8e6d-7f77-33b4-04ea-e692fa377bfc
Version Independent ID: 26885d0b-3696-ef26-8959-87b09db7c58a
Content: Continuous export can send Microsoft Defender for Cloud's alerts and recommendations to Log Analytics or Azure Event Hubs
Content Source: articles/defender-for-cloud/continuous-export.md
Service: defender-for-cloud
GitHub Login: @bmansheim
Microsoft Alias: benmansheim

@syndimann
Thanks for your feedback! We will investigate and update as appropriate.

@ManoharLakkoju-MSFT thanks for reaching out and your support ! Looking forward to an answer

@bmansheim
Can you please check and add your comments on this doc update request as applicable.

@berlihie Can you comment on this issue, please?

@syndimann Unfortunately I don't have the answer to that question. Please contact support for assistance.
#please-close


Adding volume mount to dockerfile command not working

Describe the bug
Hello!
When I run the Dockerfile provided in the github repo with some additional commands in windows:
ENTRYPOINT ["./deploy/docker-entrypoint.sh"]
CMD ["label-studio","start", "test", "-l", "/mnt/settings/label_studio_ui.xml", \
    "--init", "--username"<EMAIL_ADDRESS>"--password", "123456"]

And then in docker console in windows add the optional setting to add a volume mount, everything works perfectly. it sees the settings automatically and starts the test project.
However, when i want to deploy it with the volume mount set in the Dockerfile like so:
ENTRYPOINT ["./deploy/docker-entrypoint.sh"]
CMD ["label-studio","-v", "C:/mnt:/mnt", "start", "test", "-l", "/mnt/settings/label_studio_ui.xml", \
    "--init", "--username"<EMAIL_ADDRESS>"--password", "123456"]

The -v argument is not recognized. My question is: Am i doing something wrong, or is label-studio expecting something else? Where does docker on windows add the volume mount as an argument?
I've been racking my brain over this for 2 hours now, any help would be greatly appreciated!

Hi @SjoerdBraaksma
-v argument is a Docker argument not label studio argument
so your docker run command should look similar to:
docker run -it -p 8080:8080 -v "C:/mnt:/mnt" heartexlabs/label-studio:latest label-studio start test -l "/mnt/settings/label_studio_ui.xml" --init --username<EMAIL_ADDRESS>--password "123456"


Implement RBF via nonce arg

Implement support for passing a nonce arg which can be used for performing a replace-by-fee.

This arg is already supported:
 --nonce=nonce                          Optionally specify a nonce for this transaction
The task here could be improving the command doc / readme to explain how this can be used for performing a replace-by-fee.
@MarvinJanssen do you have any insight into what might be helpful here for the entities who are using this util?

Most entities that use this tool regularly have it integrated into their application flow. I should investigate which of them have a resolution mechanism for nonce issues / RBF built into their platform.


68-points trained face model should be distributed along with dlib

Since a few months, dlib comes with a state-of-art face tracker, and provides an use example that is impressive.
This example requires a trained model, and suggests to download it from here: http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2. The model is however not part of dlib source, and no information regarding its license is provided.
Since this 68-features face tracker with this model is by itself extremely valuable, I would suggest to have it directly bundled with dlib (like OpenCV which provides trained haar classifiers for face detection).
If the size of the file is a concern, it could possibly belong to a different repo (dlib-models or dlib-data for instance).
In any case, it would be nice to know about the license of the model (at first sight, CC-BY would be a possible match to the apache 2 license used for dlib source).

I don't want to put the file into the repo or dlib archive file since it's so big.  But clarifying the license is a good idea.  I'll put the file into a github repo with an appropriate license statement later today.

There is now a repo with the model in it and an appropriate license: https://github.com/davisking/dlib-models

Great, thank you!

No problem.

Is there a smaller model? Suitable for use on mobile devices?

No.  If you want a smaller model you will have to train your own.

What would be really useful in this repo is to add code and scripts to reproduce that model file. Hopefully over time people would contribute more complex models, enhance it etc...

Reg smaller model for mobile use case:
Is it possible to extract data from "shape_predictor_68_face_landmarks.dat" file (of size 95MB) and make smaller version of dat file (say 50MB) ?, little loss in landmark accuracy is acceptable.
This will help to avoid training by our own.

No.  Just train a new model.  The data is available and it doesn't take
that long.

We would like to reduce the dat file size by retaining the the No of landmarks to 68,
we are ready to compromise in landmark detect accuracy.
I experimented by converting dat file's deserialized data (forests, anchor_idx & deltas) from 32 bits into 16 bits and stored it as smaller dat (serialized code also modified), but unable to succeed.
So retraining new model with 68 points, will it result the into 95MB of dat file again?
Or any change needed/recommended on shape_predictor_trainer parameters to reduce this dat size (68 points case)?
Also please provide me the links to get 68 points training data like

training_with_face_landmarks.xml,
testing_with_face_landmarks.xml,
images,
(in the past, i created 194 points xmls using many scripts)

Thanks

Data is available here: http://dlib.net/files/data/.   Just remove
landmarks you don't want and the resulting file will be smaller.
Experiment with training parameters and see what you get and you like it
for your application.

I think what Gopi is asking would be quite useful -- if the landmarks dat used 16-bit floats instead of 32-bit floats, it would fit on mobile better, i.e. ~50MB vs ~100MB (Apple recommends apps be less than 100MB). And it would preserve all of the existing 68 landmarks. Is this doable by modifying the dlib serialize / deserialize code? Or does dlib require full 32-bits of precision?

Yes I get it.  You can modify it however you like.  The only way to see if
the accuracy is still good enough for what you care about is to try it and
see.

I trained with your ibug_300W_large_face_landmark_dataset files and got the dat file of size 63.3MB (45.5MB after zip) This size will fit into mobile app store, thanks a lot :)
Note:
I commented out the below lines at train_shape_predictor_ex.cpp.
trainer.set_oversampling_amount(300);
trainer.set_nu(0.05);
trainer.set_tree_depth(2);

@gopi77 and what loss of precision do you get?

I didn't measure the loss,
visually the new 68 landmarks appears to be proper and acceptable.

@gopi77 i also need the minified .dat file for android app, but i have problem training it by my self (hardware issue). Can you please share your .dat file to us? Thanks

Hi
Please find the link for dat file.
https://www.dropbox.com/s/ycbpznbjp165kae/sp.dat?dl=0
Regards
Gopi. J

Thanks for the link @gopi77 , you're the best!

Does somebody have a model he could share with less than 68 landmarks? Or a code to train such model?
I'm trying to generate a model that will be less than 50mb

There is an example program that shows how to train it: http://dlib.net/train_shape_predictor_ex.cpp.html

@davisking I'm also interested in training say 20 points model. How do I achieve that?
Do I have to manually (with some script) adjust iBUG dataset? what are the general advices to make the model smaller? We are ready to sacrifice some precision but we would like to minimize the losses.
I would appreciate any hints.

All you need to do is run the training code on training data with just 20 points in it.  Either read the paper describing the algorithm to understand how it works or just run it and see what happens with different settings.  I'm not going to explain how it works here when there is a very well written paper that explains it in detail.

@davisking yeah, I understand. Is it possible to tell which exactly 20 points we want?
Is there any info about quantizing .dat file to 8/16 bit floats?
Thanks for you cooperation

You want the points you want.  How am I to know what those points are?  I'm
not a mind reader.
Anyway, read the paper.

I understand the people wanting the trained file. Cause the same happened to me. I haven't got cpp compiling experience and don't know how to train it with less points. I have also tried the quantization but there's very poor info about it :c
Hope you understand

@davisking may I know any hardware minimum requirement to train 68 landmarks with "https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/" this data?
I am trying to run on my laptop but it seems stop working in the after 2 hours..
the code I'm using is exactly the same with "http://dlib.net/train_shape_predictor_ex.cpp.html", I only change the xml name file to match with their data.

No fany hardware is required.  Any normal laptop or desktop should suffice.  http://dlib.net/faq.html#Whyisdlibslow

可以将检测到的68个特征点数据运用到一个做好的人脸模型让他一起动吗

Can we detect the 68 feature points data to a good facial model and let him move together?

Hi friends,
I trained dlib by myself with the data got:  http://dlib.net/files/data/ and I set parameter:
trainer.set_oversampling_amount(300);
trainer.set_nu(0.05);
trainer.set_tree_depth(2);
But I got accuracy ~14% on training and model size: ~17MB.
Please let me know why?
Note: I try to modify some parameter like: set_nu(0.01), tree_depth(5), cascade,.. but I still got the result with low accuracy.
Thanks!


how to know current callstack finished

hi
I want to know when current callstack finished, and then I will do some other things after that;
I debug the function  exit in the finally of namespace.bind or run, But It couldn't work
Is there some good way to know when current callstack finished(include callback)?
thanks
zeng

Hi! I decided to put expressjs response object in the context and added this to the lib.
https://github.com/AleksMeshkov/node-continuation-local-storage/commit/7eee8cf67a4c3f79ca69bd7b5246c183dbdaaedd


Rephrase link diagnostic error messages to provide more specific information first

Bug/issue #, if applicable: rdar://99990978
Summary
This PR does 4 things related to link diagnostics:

It adds a new diagnostic when a known name but an unknown disambiguation
It leverages the authored disambiguation to narrow down the suggestions (for example to only make suggestions of the matching symbol type)
It rephrases the link diagnostic error messages to be more to the point and surface the most relevant and specific information first.
It shrinks the diagnostic range to the path component with the issue

For example, instead of highlighting the full link range and using this error message
- ``MyKit/MyClas/myFunction()``
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Topic reference 'MyKit/MyClas/myFunction()' couldn't be resolved. Reference at '/MyKit' can't resolve 'MyClas/myFunction()'. Did you mean 'MyClass'?"
Replace 'MyClas' with 'MyClass'.  [Fix]

the same diagnostic now highlights the "MyClas" path component with this error message
- ``MyKit/MyClas/myFunction()``
          ~~~~~~

'MyClas' doesn't exist at '/MyKit'
Replace 'MyClas' with 'MyClass'  [Fix]

Dependencies
None
Testing
Describe how a reviewer can test the functionality of your PR. Provide test content to test with if
applicable.
Steps:

Provide setup instructions.
Explain in detail how the functionality can be tested.

Checklist
Make sure you check off the following items. If they cannot be completed, provide a reason.

[x] Added tests
[x] Ran the ./bin/test script and it succeeded
[x] Updated documentation if necessary

@swift-ci please test

@swift-ci please test


Final attribute adjustment

I misjudged this one - the attribute in GATK-SV VCFs is RD_CN, and this is annotated later as CN. 🙄

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 78.50%. Comparing base (8ac64ca) to head (cea2766).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #886   +/-   ##
=======================================
  Coverage   78.50%   78.50%
=======================================
  Files          10       10
  Lines        1791     1791
=======================================
  Hits         1406     1406
  Misses        385      385

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Interaction between Rust wasm/foreign modules and encapsulation

I hope this is the right place for this discussion; I tried finding a forum that would reach the right target audience and this seemed like the best place.
I've been working with Rust/wasm for quite a while.  For that I've been using wasm-bindgen quite heavily (for a not-yet-released project) using a Webpack loader I've authored: https://github.com/dflemstr/rust-native-wasm-loader
One thing I have noticed is that there doesn't seem to be a clear direction regarding how code modules are to be treated between Rust and foreign modules (such as ES6 modules or other wasm modules) on the source level.
Incidentally, it seems to be working similar to C but not consistently.  This behavior is quite confusing when interfacing between two languages that both support modules/namespacing (Rust+JS)
Remember, this is the perhaps naive impression of somebody using Rust together with wasm.  I'm aware of how the current wasm FFI is closely modeled after the C FFI.
Imagine the following tree (with the module hierarchy like you would expect):
.
└── src
    ├── lib.rs
    ├── reducer
    │   └── editor.js
    └── util
        ├── leftpad.js
        └── leftpad.rs

What should the semantics be around visibility and scope for these files?  This is what my intuition would tell me:

editor.js should import leftpad.rs as ../util/leftpad.rs, or import some myrustlib.wasm module where the symbol is available as util::leftpad::xyz in some fashion.
leftpad.rs should import editor.js as ../reducer/editor.js.
(Stretch:) There should be a separate notion of "exported in my Rust crate" and "available to Javascript"
(Stretch:) It should be possible to mark a symbol as being available only in the util module, so that editor.js would not see such a symbol defined in leftpad.rs, but leftpad.js would be able to.

With that said, here is what current tools are doing:

Raw rustc symbols that are export "C"'d are exported in a global shared namespace.
wasm-bindgen will export everything marked with #[wasm_bindgen] also in a global shared namespace (including more advanced things such as structs/classes etc).
wasm-bindgen will resolve all imports (#[wasm_bindgen(module = "...")]) relative to the location of the final generated module artifact, wherever that may be.
wasm-bindgen only supports symbols that are both crate-public and exported.
All imports/exports have no notion of encapsulation.
(Feel free to suggest other items if we want to make a complete list; I just highlighted some examples)

I find this behavior surprising and feel like one could do either of these things:

Prohibit imports/exports except on the crate top level (lib.rs/main.rs) and resolve them relative to that location.
Properly namespace imports/exports such that they behave as I described above, or in some other intuitive fashion.

Does this sound reasonable? What are everybody's thoughts on this?

Hey @dflemstr I would say this is the right place since this is where the working group for wasm and Rust coordinate! I would like to here what @alexcrichton thinks on this considering he's done most of the compiler work regarding the wasm32 target. I'm sure many others here have thoughts on the matter as well!

@dflemstr You're definitely in the right place. Let me give my thoughts on your post:

editor.js should import leftpad.rs as ../util/leftpad.rs, or import some myrustlib.wasm module where the symbol is available as util::leftpad::xyz in some fashion.

I'm undecided about this. On the one hand importing the .rs file makes sense, but on the other hand Rust compilation units are at the crate level, not the module level.
So I guess it depends on whether we want .js files to be seen as a "part of the crate", or as an external thing which imports the Rust crate. I suppose both use-cases will need to be supported.

leftpad.rs should import editor.js as ../reducer/editor.js.

If we treat .js files as being a part of the crate, then yes absolutely I would expect that.

(Stretch:) There should be a separate notion of "exported in my Rust crate" and "available to Javascript"

I believe Rust already has a notion like that: https://doc.rust-lang.org/book/first-edition/ffi.html#callbacks-from-c-code-to-rust-functions
I imagine it would look like this:
extern "wasm" fn foo() -> i32 {
  5 + 10
}

(Stretch:) It should be possible to mark a symbol as being available only in the util module, so that editor.js would not see such a symbol defined in leftpad.rs, but leftpad.js would be able to.

I'm not so sure about this one. That sounds like it's trying to apply Rust's module system and scoping rules to JS, but JS has its own module system.

As for the rest of your post, I generally agree. I think this is less a matter of "the Rust team decided things should be this way" and more a matter of "we haven't decided how things should be yet, so the current status-quo is a bit wonky".

FWIW right now wasm and rustc have no knowledge of import paths really, they're sort of just opaque strings. AFAIK it's bundlers that attach meaning to these import paths to allow files to import one another. In that sense I'm not entirely sure what we'd do differently in rustc/wasm-bindgen/etc?
Specifically on the topic of FFI and "what symbols are exported" or "what is their ABI", that's quite similar to https://github.com/rust-lang-nursery/rust-wasm/issues/29 I think

@alexcrichton It might be necessary for rustc to carefully generate the relative imports in such a way that they work correctly after running wasm-pack. That will require us to figure out exactly how the custom sections and relative paths work.

Actually, the situation is much more complicated than I thought. rustc bundles up all the Rust code into a single .wasm file, and then wasm-pack grabs the .js files out of the .wasm file.
But that means if the .js files try to import a .rs file, it doesn't work because the .rs files don't exist!
So we either have to give up on the idea of .js files importing .rs files, or we have to parse the .js and rewrite the imports to point to the .wasm instead.
And then there's issues about name collisions, if multiple Rust crates export the same variable name...

@Pauan oh but my point is that rustc isn't generating imports at all, authors are writing them down in source code and rustc is moving those strings along to the output file.

@alexcrichton Yes, but there's all sorts of problems if the imports are just passed through unchanged:

The author of a crate called mycrate has a src/subdir/foo.rs file and a src/bar.js file.

Inside of foo.rs it has #[wasm_import_module = "../bar.js"]

When that crate is compiled and run through wasm-pack it gets converted into a mycrate.wasm and bar.js file, but inside of mycrate.wasm it has an import for ../bar.js, which is wrong.

Doing this right will be hard, since authors will want to import file paths relative to the .rs file, but the compiled .wasm file won't necessarily have the same directory structure as the original .rs file.

Oh sure, but this is also serious hypothetical territory right now. Nothing in the toolchain supports specification of the import module, much less actually resolving it. I think before we debate what should happen we should get something working, see how it works, and go from there.

Does it makes sense to use a path specific to JS there at all anyway? That means the rust crates would be written with specific JS files in mind, which honestly I don't see working out if we consider normal rust crates like chrono that you have as a dependency. So for something like chrono it makes a lot more sense to just define something like this:
#[wasm_import_module = "chrono"]
extern {
    fn time() -> f64; // Made up signature
}

That way you have a clean namespacing on the rust side that isn't prone to potential conflicts.
However I think it definitely does make sense to be able to specify JS dependencies there too, so it probably makes sense to have a custom section in the wasm that specifies how those map to JS.
WAT-like pseudo-code:
(JSImportMappings ("chrono" "chrono-helper.js"))

And on Rust's side it could looks like:
#[js_import_mapping = "chrono-helper.js"]
#[wasm_import_module = "chrono"]
extern {
    fn time() -> f64; // Made up signature
}

@alexcrichton Absolutely, I'm not trying to stop the prototyping work. But since ES6 modules and npm are stabilized, we can discuss theoretical use-cases, to make sure that we will be able to (eventually) support them.

@CryZe That would require that any Rust crate that wants to use JS files would have to publish those JS files as an npm module. That's true even if the Rust crate only wants to include a tiny snippet of JS code.
Also how would the naming go? Would the Rust crate be published to npm as chrono, and the JS snippets published to npm as chrono-js? That seems really weird (and unusual for npm).
And then there's things like wasm-bindgen and stdweb which need to be able to generate JS files. So we will need some way of including JS code in a Rust crate. Maybe it would be inline like the stdweb js! macro, or maybe it would be an external .js file (which is much cleaner), but either way the use case is definitely necessary.

And on Rust's side it could look like [...]

How is that better than specifying a relative path to the .js file? No matter what API we decide on, it will be compiled into wasm custom headers, but your API seems a lot more cumbersome than the relative path API.

My thinking is this: when you compile a crate to wasm32-unknown-unknown, rustc will add in a custom header to the .wasm file. This header would contain the various metadata we need. This header contains the metadata for all the crates which were compiled (including direct and transitive dependencies).
As such, the header needs to be able to uniquely identify each crate. I assume rustc already has some way of uniquely disambiguating crates, so that can be used. Or perhaps it could use UUIDs. But I don't really care what method is used, as long as each crate can be uniquely identified.
When a crate uses #[wasm_import_module = "./path/to/foo.js"], rustc would resolve the relative path based upon the .rs file which contains the wasm_import_module. And then it would insert three things into the wasm custom header:

The unique identifier for the crate.

The absolute path to the crate.

The relative path from the crate to the .js file which is referenced within wasm_import_module

In addition, when rustc is generating the wasm import for wasm_import_module, it should contain the unique identifier for the crate followed by the relative path to the .js file.
As an example, let's consider a Rust crate located at /path/to/crate:
// File `src/foo/bar.rs`
#[wasm_import_module = "../qux.js"]
extern {
    fn time() -> f64;
}

#[wasm_import_module = "./corge.js"]
extern {
    fn random() -> f64;
}

// File `src/qux.js`
export function time() {
    return Date.now();
}

// File `src/foo/corge.js`
export function random() {
    return Math.random();
}

And lets suppose that rustc gives the above crate the unique identifier some_unique_identifier
The .wasm custom header would contain this information:
{
    "crates": [{
        "id": "some_unique_identifier",
        "path": "/path/to/crate",
        "imports": ["src/qux.js", "src/foo/corge.js"]
    }]
}

For the sake of clarity I described the data format using JSON, but I don't really care about the format: it could be binary or anything else.
And within the import section of the .wasm it would contain the following imports:
(import "some_unique_identifier/src/qux.js" "time" (func time (result f64)))
(import "some_unique_identifier/src/foo/corge.js" "random" (func random (result f64)))

Now that the .wasm file contains the above information, it is now possible for wasm-pack to correctly handle everything. It can parse the .wasm file, extract the information out of the custom header, copy the .js files into the current folder, and then fix the wasm imports to point to the correct paths.
Of course wasm-pack will need to handle filename conflicts, and relative imports in the .js files, but that's an issue for wasm-pack to solve, not rustc. The important thing is that rustc gives wasm-pack enough information so that it's possible for wasm-pack to solve those problems.
Everything I described above doesn't require rustc to do anything complicated: it doesn't need to parse .js files or anything like that.
It already knows the absolute path to the crate, and it already knows the relative path to the .rs file, so all it needs to do is resolve the wasm_import_module path relative to the .rs file.
In addition, the above format is generic, so it can work with other systems like C++. In other words, it's not tied specifically to Rust, it simply contains some filepath information, nothing else. And because it simply contains filepaths, it doesn't need to be limited to .js: the same header format could be used to import .wasm files, .css files, etc.
The above idea doesn't allow for .js files to import .rs files, but at least it allows .rs files to import .js files. I think importing .rs files inside .js will require a lot more design work (and probably a new header format, separate from the above header format).

Oh, also, in my above proposal I assumed that wasm_import_module would be importing relative paths (since that's consistent with ES6 modules).
I still think that's the best idea, however there is an acceptable alternative:
#[wasm_import_module = "/src/qux.js"]

In this case the import is relative to the crate, not relative to the current .rs file.
The custom header format would be exactly the same, the .wasm import would be exactly the same, and wasm-pack would be exactly the same, so the only difference is the syntax of wasm_import_module.
However, I don't like this because it's inconsistent with filepaths in ES6 modules. I think it's really nice if wasm_import_module mirrors ES6 imports as much as possible (especially because we're already committed to importing npm modules).
But if other people are convinced that this approach is better, then I'll accept it. The exact syntax for wasm_import_module isn't that important, the important thing is the support for the custom header format (which is the same regardless of how wasm_import_module looks).

Also, I mentioned the need to uniquely identify each crate. Rather than generating a unique identifier, it could instead simply use the index to uniquely identify each crate.
Looking at the /path/to/crate example again, it could generate this custom header:
[
    {
        "path": "/path/to/crate",
        "imports": ["src/qux.js", "src/foo/corge.js"]
    }
]

And generate these imports within the .wasm:
(import "0/src/qux.js" "time" (func time (result f64)))
(import "0/src/foo/corge.js" "random" (func random (result f64)))

Rather than using some_unique_identifier it is instead using the index within the JSON array (which in this case is 0)
Either approach works fine, but this index-based approach might be easier to implement.

I'm new to this discussion but...  in general this is a well-exercised but somewhat arcane problem called "linking".  So I'm going to ramble a little bit and try to describe the problem from that perspective, mainly so I can try to think about it.
WASM modules describe all the information needed for linking: what names a module exports, what module names and symbol names a module imports.  However, it says nothing at all about how that relates to Javascript or Rust anything else outside of wasm itself.
The first problem is one of uniqueness of names, and Rust traditionally solves this by mangling names.  If the names need to be something that other programming languages can understand, there's extern (usually extern "C" which uses C's name and function call conventions).  Mangled names exported from crates are guaranteed to be unique because they include crate names and you can't have two crates in the same project with the same name.  If they're not unique, it's a compile-time problem and it doesn't compile, or if dynamic linking is involved the program doesn't run (in wasm most linking is dynamic, more or less).  This is kindasorta-not-a-problem for wasm because names live in different modules, and so it's the programmer's job to just make sure modules have unique names.  wasm modules don't actually have a built-in name, the host decides what to name them, and in general name conflicts are the programmer's problem, not the wasm host's.  So, we have an easy way of dealing with this.
It seems like if you want to treat JS files like Rust you have to fit them into Rust's assumptions (modules, crates, etc) in a reasonable way, and conflating symbol names with file names is not the way to do this.  You could easily import the JS file into the Rust code as a string, but that may not accomplish what you want (I frankly know very little about the JS <-> Rust bridging process, just how it looks from Rust and wasm.)  From the Rust side, it seems semi-obvious that if you want to be able to call JS code from Rust, it needs to be treated like a Rust module.  This probably means just making it a Rust module and filling it with FFI code generated by wasm_bindgen or some other FFI-generator.  There may be crates consisting purely of these FFI bindings, and that's fine; there are plenty of foo-sys crates that do exactly that but with C code.  They usually don't include C code themselves but they do usually have code to link against the C libraries on various platforms... but not to download and build the C library themselves.  Usually.  If we're talking about including raw JS in Rust code, then that's basically what we're doing: building a library in a different language, and providing an FFI layer as a Rust module.  So I dunno about incorporating JS files into Rust crates, but making Rust crates that link to JS files seems like a problem that just gets solved the same way it gets solved for any other language.
From the perspective of the wasm host, it's the wasm host that's in charge of (dynamically) linking things at runtime.  wasm itself has no idea how any of this language/crate/module/function nonsense; it just knows about names in wasm modules.  The host provides modules full of names, and how it gets them is none of wasm's business.  A crate is not really a unit of dynamic linking, and Rust has no language-level story for dynamic linking, so a .wasm file built from Rust source is going to either include all the code necessary for running the whole program, or just a subset of it, a list of symbols it imports and exports, and a big sign saying "no longer rustc's problem".  If we take the cue from the C ffi, then the foo-js-sys FFI crate will simply fail to link at runtime if the wasm host can't provide the foo module somehow or another (presumably by loading foo.js).  If we want to be able to guarantee at compile-time that foo.js exists and will work the way foo-js-sys expects it to, then we need to be building some loader infrastructure off in Web land and I have no idea how any of that works.
For calling Rust from JS, I don't know much about how JS modules or the wasm<->JS host interface operates, but a Rust program will export some (non-mangled, probably) names of functions or data and JS will ask the wasm host to provide them.
Binary distribution of a library is a different ball-game; rustc helps you not at all there, as far as I know, it's generally up to the programmer and the operating system to make sure things slot together the way they're supposed to.  In this case, wasm-pack is our linker it seems, and it is trying to put everything together so the wasm host doesn't need to figure out where to find multiple Rust libraries and Javascript files.  I have no idea how it incorporates JS files into this, but if it's dealing with Rust crates and they're interacting with JS via FFI the same way they interact with everything else then I don't think there's any need for a header in your wasm file.  Maybe it will output dependency info for JS functions as part of the linking process but that's a purely temporary thing.
I don't have any real answers, I just want to make sure everyone is 100% aware of what is going on: we're talking about building a format and tools for dynamic linking.  WebAssembly is our .dll format, and it provides fairly complete but also quite minimal information.  We need to figure out how to map the assumptions of Rust crates and Javascript modules onto the information it provides us.  Rust provides assumptions that make linking easy: mangled names, no dynamic linking, no unlinked libraries hanging around, dedicated FFI crates.  wasm works fine with these assumptions.  So it feels like the real question is "how do we easily make nice FFI crates out of Javascript and bundle it all up to deliver to a web browser?"

@icefoxen I appreciate you sharing your thoughts. Allow me to respond to some of the things you said:

If you want to specify the location of a JS file relative to a Rust file you could easily import the JS file into the Rust code as a string, but I don't know what that accomplishes (I frankly know very little about the JS <-> Rust bridging process, just how it looks from Rust and wasm.)

I think it's important for this discussion to understand exactly how Rust, JS, and wasm interact with each other (which includes the linking process).
This is our current plan (though it might change):
Let's suppose that you want to compile a Rust crate to wasm:

This Rust crate contains dependencies on other Rust crates (by using Cargo.toml, like usual).

Each Rust crate might have some dependencies on npm packages. These npm dependencies are specified in a package.json file which is within each crate.
You can think of package.json as being similar to Cargo.toml, except it describes npm dependencies rather than Rust dependencies.
It's important to note that each Rust crate has their own separate package.json, just like how each Rust crate has their own separate Cargo.toml

Each Rust crate might have some .js files that they use.

Within the .rs files, the Rust crates can use the following to import the JS files:
#[wasm_import_module = "./foo.js"]
extern {
    fn foo() -> f64;
}

Or they can use the following to import npm packages (which are specified in the package.json file within the crate):
#[wasm_import_module = "foo"]
extern {
    fn foo() -> f64;
}

The above import syntax is standard for JavaScript:

It imports individual .js files.

The import filepath is either an npm package, or a relative path from the current file to the .js file.

After compiling the Rust crate (using rustc or cargo build) the final output is a single .wasm file (which is put into the target folder, as usual).
This .wasm file contains the compiled Rust code for the current crate, and also the compiled Rust code for all the dependencies of the current crate (recursively).
In other words, all of the Rust static linking has already occurred, there is no dynamic linking between Rust crates.
For each usage of wasm_import_module the .wasm file contains a wasm import which will load the .js file at runtime (i.e. dynamic linking, as you said).
However, the .wasm file does not contain the .js files or npm dependencies, those are kept separate (they are not statically linked).
At this point rustc has successfully compiled the Rust crates into .wasm, and therefore its job is complete.
But we're not quite done yet. You now need to use the wasm-pack tool, which does two things:

For each Rust crate it reads the package.json file and merges all of the npm dependencies together. It then generates a package.json file in the current folder which contains the merged dependencies.

It copies the .js files from all of the Rust crates into the current folder.

The end result is that the current folder contains three things:

A package.json file which describes all of the npm dependencies for all of the Rust crates.

A single .wasm file which contains all of the compiled Rust code for all of the Rust crates.

Multiple .js files which were copied from all of the Rust crates.

At this point the .js files still haven't been linked into the .wasm file, they are still separate.
Now you must link the .wasm and .js files together. There are two ways to do that:

You can simply run the .wasm file in the browser, since the browser will automatically link the imports. This only works in browsers that support both WebAssembly and ES6 modules.

You can use Webpack, Rollup, or Parcel to do the linking between .wasm and .js

And now you can finally run the program, because everything has been successfully linked together.
There will be a tool which streamlines the above process and makes it easy (by calling cargo build + wasm-pack + webpack all in one step), but for the sake of understanding my proposal I needed to explain how it all works under the hood.
Most of what I just said has already been implemented. The part that hasn't been implemented (and what my proposal is trying to fix) is allowing Rust crates to contain .js files.
By using #[wasm_import_module = "./foo.js"] it's currently possible for Rust crates to import .js files.
However, the import paths in the compiled .wasm file are wrong, which causes the final linking step to fail.
My proposal fixes that by adding a tiny bit of filepath information to the .wasm file so that wasm-pack can then read that information and fix the import paths.
Note: my proposal has nothing at all to do with variable or function names. As you said that is a solved problem with ES6 modules.
My proposal also has nothing at all to do with FFI interop between JS <-> Rust, that's already handled by extern (and wasm-bindgen / stdweb)
My proposal is only about making sure that the import filepaths within .wasm are correct, so that the files can be linked together in the final step.

From the Rust side, it seems semi-obvious that if you want to be able to call JS code from Rust, it needs to be treated like a Rust module. This probably means just making it a Rust module and filling it with FFI code generated by wasm_bindgen or some other FFI-generator.

Yes of course Rust will need to use extern blocks (and probably wasm-bindgen) to describe which JS files to import (and which variables to import from those JS files).
But that's completely separate to my proposal. The problem my proposal is trying to fix is the following:

A Rust crate (let's call it foo) includes a foo.js file, which is intended to be imported from within Rust.

The foo crate uses #[wasm_import_module = "./foo.js"] or whatever to import the .js file. The exact mechanism for importing the file isn't important.

Somebody else creates a Rust crate (let's call it bar) which uses the foo crate as a dependency.

After compiling the bar crate the end result is that it has a single bar.wasm file in the target folder which contains the Rust code for both the foo and bar crates.
However, the foo.js file doesn't exist in the target folder, and so things break.
Or maybe the foo.js file does exist, but the ./foo.js import path inside of the .wasm file is wrong.
My proposal fixes both of those problems.

They usually don't include C code themselves but they do usually have code to link against the C libraries on various platforms...

The situation with JS is very different from the situation with C. Being able to include .js files inside of a Rust crate is a critical use-case (since it will often be needed to create glue code which converts the JS API into an API that can be used within Rust).

but making Rust crates that link to JS files seems like a problem that just gets solved the same way it gets solved for any other language. [...] If we take the cue from the C ffi, then the foo-js-sys FFI crate will simply fail to link at runtime if the wasm host can't provide the foo module somehow or another (presumably by loading foo.js).

Unfortunately it's not that simple, because of the way that wasm + JS + npm are, things cannot just be handled in the same way it's handled with C.

If we want to be able to guarantee at compile-time that foo.js exists and will work the way foo-js-sys expects it to, then we need to be building some loader infrastructure off in Web land and I have no idea how any of that works.

That infrastructure already exists (Webpack, Rollup, Parcel, etc.). And the plan is for us to reuse that infrastructure as much as possible.
The purpose of my proposal is just to ensure that the filepaths are correct so that Webpack/Rollup/Parcel/etc. can do the final linking.

I have no idea how it incorporates JS files into this, but if it's dealing with Rust crates and they're interacting with JS via FFI the same way they interact with everything else then I don't think there's any need for a header in your wasm file.

If we can avoid the header information then that would be great! I'm definitely open to alternative proposals which fix the filepath problem.
However, the status quo is not sufficient to fix the filepath problem. So we need some solution (not necessarily my solution).

Maybe it will output dependency info for JS functions as part of the linking process but that's a purely temporary thing.

Yes, after wasm-pack has done its work it deletes the header from the .wasm file. Though in my proposal the header has nothing to do with functions, it only has to do with filepaths.

Do we have any issues opened for this on the wasm-pack ?

The JS ecosystem is very complex and it's constantly changing

I could not agree more on this.

Multiple .js files which were copied from all of the Rust crates.

How the npm dependencies will be handled here? I agree this will work great if you have JS dependencies (as separate files) But how to relatively link the entire lib inside.

@sendilkumarn Thanks for taking a look!

Do we have any issues opened for this on the wasm-pack ?

I don't think so. There's nothing wasm-pack can do until after rustc (or wasm-bindgen) supports this.

How the npm dependencies will be handled here? I agree this will work great if you have JS dependencies (as separate files) But how to relatively link the entire lib inside.

They'll be handled the same way wasm-pack handles them right now: it creates a package.json file and then you run npm install (or yarn install).
Bundlers (like Webpack, Rollup, and Parcel) understand the node_modules structure, so the final linking step will work just fine.

@Pauan Thanks for all those detailed explanations. I really enjoy reading them 👍

They'll be handled the same way wasm-pack handles them right now: it creates a package.json file and then you run npm install (or yarn install).

Even inside the .rs file ? Like I want to use a npm library inside the .rs file. How this will be handled? How will be the linking happens here?
P.S: There are a lot of use cases. Mapping & Framing them in a single mental model is really difficult. (atleast for me)

Even inside the .rs file ? Like I want to use a npm library inside the .rs file. How this will be handled? How will be the linking happens here?

You import it with normal npm import syntax, like this:
#[wasm_import_module = "foo"]
extern {
    fn foo() -> f64;
}

That will import the foo npm package (which must be specified in that crate's package.json file).
Then that will translate into this wasm import:
(import "foo" "foo" (func foo (result f64)))

I haven't tested it, but I believe the above wasm import will Just Work(tm) with the JS bundlers (Webpack, etc.).
So in the case of npm modules, the only thing wasm-pack needs to do is generate the package.json, the bundlers handle everything else.
I suppose there might be some conflicts between my proposal and npm packages (e.g. if there is an npm package called 0, then is 0 an npm package, or is it a unique identifier for a Rust crate?).
That can be solved easily enough by using prefixes/namespaces:
(import "crate:0/src/qux.js" "qux" (func qux (result f64)))
(import "npm:foo" "foo" (func foo (result f64)))

So the crate: namespace means to import the file from the Rust crate (which has the unique id of 0), and npm: means to import the npm module called foo.
Then wasm-pack would:

Copy the .js files from the crate: imports into the pkg folder (rewriting the wasm import path as appropriate).

Strip out the npm: namespace.

So the end result would be something like this:
(import "./0/src/qux.js" "qux" (func qux (result f64)))
(import "foo" "foo" (func foo (result f64)))

Notice that it copied the .js file into the pkg/0/src folder, and it changed the import path to be relative. And it stripped out npm: as well.
Now everything is set up correctly so that Webpack (etc.) can correctly do the final linking.
This handles the conflict correctly, because crate:0 and npm:0 would be treated as different.

I think this is handled by RFCs like https://github.com/rustwasm/rfcs/pull/8 and https://github.com/rustwasm/rfcs/pull/6, so closing.


Issue with k8s.io/docs/tutorials/configuration/configure-redis-using-configmap/

Warning  FailedMount  67s (x10 over 21m)  kubelet, docker-desktop  Unable to mount volumes for pod "redis_default(fb68a597-838e-4dbb-a252-36cd8eeba1a4)": timeout expired waiting for volumes to attach or mount for pod "default"/"redis". list of unmounted volumes=[config]. list of unattached volumes=[data config default-token-b9nw7]
Warning  FailedMount  59s (x19 over 23m)  kubelet, docker-desktop  MountVolume.SetUp failed for volume "config" : configmap references non-existent config key: redis-config

I am trying the sample with Docker desktop on windows

This list of issues tracks bug reports and feature requests for the Kubernetes documentation.
If you're looking for help, check Stack Overflow and the troubleshooting guide.
You can also post your question on the Kubernetes Slack or the Discuss Kubernetes forum.
/close


Remove deprecated code

Description

closes #13262
Additionally I removed deprecated code from two more classes: VH_BE_DOUBLE (static member in BitUtil) and variant of BlockJoinSelector.wrap method. Both classes are tagged with @lucene.internal and I searched in Solr and elasticSearch. Let me know If any potential issues arise from its removal, I'll reinstate it.

Thanks for looking into this. I think I'd rather keep VH_BE_DOUBLE for now for consistency, since we're also keeping the INT, LONG and FLOAT BE variants.

In addition to IntField, can you also remove deprecated constructors from other *Field classes such as DoubleField?
Can you also remove deprecated code in BooleanQuery and ByteBuffersDataInput?

I looked into different Search engines whether they use it or not  -- Many of them still using the deprecated code.

LongPoint.newDistanceFeatureQuery (ElasticSearch uses it https://github.com/search?q=repo%3Aelastic%2Felasticsearch+LongPoint.newDistanceFeatureQuery&type=code and OpenSearch as well https://github.com/search?q=repo%3Aopensearch-project%2FOpenSearch newDistanceFeatureQuery&type=code)
BooleanQuery.TooManyClauses (Elastic Search https://github.com/search?q=repo%3Aelastic%2Felasticsearch TooManyClauses&type=code)
BooleanQuery.getMaxClauseCount and BooleanQuery.setMaxClauseCount are still in use.

Should I go ahead and remove it?

@iamsanjay Yes please go ahead. All these problems should be easy to address by these various projects. Thanks for checking!

I added MIGRATE docs. Let me know if that's not required.
There is one more class pending SortField. I am still looking into some of the test cases that are failing , we may have to remove those test cases along with deprecated code. We can also deal with it in separate pull request.


az cli "hybrid connections not supported on linux app."

Describe the bug
Since hybrid-connections are now supported on Linux webapps via the portal, it should also be supported via CLI,
The following warning message appears: "hybrid connections not supported on linux app." and it does not add the hybrid-connection.
To Reproduce
execute command on linux webapp:
az webapp hybrid-connection add --hybrid-connection
--name
--namespace
--resource-group
[--slot]
[--subscription]
Expected behavior
A hybrid connection should be added to the linux webapp
Environment summary
Linux-4.15.0-1106-azure-x86_64-with-debian-10.2 (Cloud Shell)
Python 3.6.10
Installer: DEB

azure-cli 2.19.0 *

Additional context
Documentation does not mention that this does not work on linux webapps

route to appropriate team

This was marked as resolved by #17146, which was released in version 2.23.0 (according to the release notes).  After upgrading to that version, this error persists.

az --version
azure-cli                         2.23.0

core                              2.23.0
telemetry                          1.0.6


Catch & wrap EntryPointNotFoundException with a friendly message

In .NET Core 3.1 and .NET 5 apps using Ookii.Dialogs need to add an app.manifest file to the application using Ookii Dialogs for it to to work, otherwise the user gets and EntryPointNotFoundException.
One idea would be to catch this exception and wrap it into an OokiiDialogsException with a friendly message explaining the need for the app.manifest, with a hyperlink to docs on how to do it.

Superseded by #31.


Fix incorrect message after cleaning up local setup

Resolves https://github.com/wso2-cellery/sdk/issues/288

Codecov Report

Merging #553 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #553   +/-   ##
=======================================
  Coverage   64.48%   64.48%
=======================================
  Files          34       34
  Lines        1177     1177
  Branches       98       98
=======================================
  Hits          759      759
  Misses        365      365
  Partials       53       53

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4d2059...21f20ef. Read the comment docs.


EXT-1438: Compact Design

What?
In various ways, changed the Sandbox UI to be more compact and align better with Storyblok's design.
From

To

Note: we're still waiting for designs, so it's going to keep changing.

Removed the Story view. This view was originally created to develop and debug the getContext feature in @storyblok/field-plugin, but it's not necessary to include this in the container. The getContext feature has not been implemented in V2, and since nobody has reported an issue on this, it is pretty clear that this feature is not used by clients.

Removed the JSON view of the Schema. The users can already see the string key-value pairs in the options section.

Removed the accordion and merged the State and Schema sections into a single Data section. The idea behind this view is to give the developer insight into FieldPluginResponse.data.

Using the design team's preferred style for code blocks:

Removed dependency on highlight.js. It's overkill at this point, as the only thing to highlight would be a json string.

Renamed the Value section to Content.

How to test? (optional)
See the preview deployment.

I like how it has changed overall, but can we bring back the Story? I think it's essential to see which value they just have passed to setValue(). Or even if they failed (or forgot) to send one, it will be empty, which is more noticeable.

I like how it has changed overall, but can we bring back the Story? I think it's essential to see which value they just have passed to setValue(). Or even if they failed (or forgot) to send one, it will be empty, which is more noticeable.

Do you mean this View?

This is actually not the same counter as in setValue. data.value can be seen in the Content view

Btw, I realize that we should probably add some indication of what the different sections refer to. For example, in the the Content section, we should have data.value, and under the Height sections, we should have data.height. What do you think?

@johannes-lindgren Oh I see. What if we rename Content back to Value? It might be more consistent. And I like the indications
Value (data.value)
And it could be good to have a button like "View Raw Data", and it shows that whole data object. What do you think?

What do you think of the idea of changing data.value to data.content? That would make it pretty clear that that is the only data that will be included the JSON. It would also be consistent with the Storybook Delivery API which stores the content/value in the content property on the story object.
I haven't made up my mind on this, just throwing this idea out there... I will change the title back to Value for this pull request, then we can consider this for a separate ticket.
I will see if I can add an accordion that will show the full value.

What do you think of the idea of changing data.value to data.content? That would make it pretty clear that that is the only data that will be included the JSON. It would also be consistent with the Storybook Delivery API which stores the content/value in the content property on the story object.
I haven't made up my mind on this, just throwing this idea out there... I will change the title back to Value for this pull request, then we can consider this for a separate ticket.
I will see if I can add an accordion that will show the full value.

Hmmm, yeah then could you create a jira ticket where three of us can have a discussion next week regarding the term? Let me know if this PR is ready for review again. Thanks!

@eunjae-lee It's ready to be reviewed again :)
As you suggested, I added a new section where you can preview the data object (it is not complete, as there are some properties that we haven't implemented yet, such as data.spaced)
I also created a ticket for the value/content topic: https://storyblok.atlassian.net/browse/EXT-1497

Thanks for the work. Looks good to me!

Thanks for the valuable feedback!


Edit UG for consistency and style

Codecov Report

Merging #173 (6d07a39) into master (d277aa0) will decrease coverage by 0.22%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #173      +/-   ##
============================================
- Coverage     65.46%   65.24%   -0.23%
- Complexity      734      735       +1
============================================
  Files           132      132
  Lines          2754     2768      +14
  Branches        333      334       +1
============================================
+ Hits           1803     1806       +3
- Misses          872      883      +11
  Partials         79       79

Impacted Files
Coverage Δ

src/main/java/seedu/address/ui/MainWindow.java
0.00% <0.00%> (ø)

src/main/java/seedu/address/ui/PersonCard.java
0.00% <0.00%> (ø)

...rc/main/java/seedu/address/ui/PersonListPanel.java
0.00% <0.00%> (ø)

...a/seedu/address/logic/commands/NextSemCommand.java
100.00% <0.00%> (ø)

...rc/main/java/seedu/address/model/ModelManager.java
80.00% <0.00%> (+0.77%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[3.x] Editor Debugger: auto scrolling is activated at every log start.

Fix #37476
Note: for the master branch I can't test if it works (godot crashes at the start) and the changes are not entirely compatible, given the different structure of the current debugger.

Note: for the master branch I can't test if it works (godot crashes at the start) and the changes are not entirely compatible, given the different structure of the current debugger.

This will need a master PR merged before it can be merged for 3.2 though.

Done #37751

Sorry, my mistake :(

Is this still relevant for the 3.x branch? The master PR was closed as it was no longer relevant but I think it might be different in 3.x.


Fix visibility form collapse/expand for batches

Fixes #930
This fixes a bug in the visibility pane on the new batch form.  Clicking on the Embargo radio button will now expand and reveal the "Restricted to..." or "then open it up to.." options like it does on the single work form.
@projecthydra-labs/hyrax-code-reviewers

Thanks, @hortongn!


Add alternate "View more" URL field to Related Posts block

We have a page owner that would like to be able to use a different destination for the "View more" link on their Related Posts block, to go to the Blog, rather than the default of the Activity Log. This PR adds a new field to override that destination URL.
Background: https://GHE/CFGOV/platform/issues/2714
Additions

alternate_view_more_url field added to RelatedPost
Related Post _view_more() macro updated to check for a value in the new field

Testing

Pull branch
Run the migration
Edit/add a Related Posts block, adding something like /about-us/blog/?categories=info-for-consumers to the new field
Preview the page and see that the "View more" link is going to that URL, not the Activity Log

Screenshots

Checklist

[x] PR has an informative and human-readable title
[x] Changes are limited to a single goal (no scope creep)
[x] Code can be automatically merged (no conflicts)
[x] Code follows the standards laid out in the development playbook
[x] Passes all existing automated tests
[x] Any change in functionality is tested
[x] New functions are documented (with a description, list of inputs, and expected output)
[x] Placeholder code is flagged / future todos are captured in comments
[x] Visually tested in supported browsers and devices (see checklist below :point_down:)
[x] Project documentation has been updated
[x] Reviewers requested with the Reviewers tool :arrow_right:

Generally looks good to me from a UX standpoint.


core-components: Move inline styles to classes for Avatar component

chore: move the styles into the avatar theme so it can be overriden
chore: remove styles
chore: add changeset Signed-off-by: blam<EMAIL_ADDRESS>
Hey, I just made a Pull Request!

:heavy_check_mark: Checklist

[ ] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[ ] All your commits have a Signed-off-by line in the message. (more info)

@marleypowell do you see any impact by this change? I'm currently travelling and don't have the time to set this up in Azure DevOps right now to confirm myself.

@marleypowell do you see any impact by this change? I'm currently travelling and don't have the time to set this up in Azure DevOps right now to confirm myself.

The changes look good and shouldn't have an impact 😄


Update imports for files refactored in core PR #8157

Issue #, if available:
N/A
Description of changes:
Update imports for files refactored in core PR https://github.com/opensearch-project/OpenSearch/pull/8157
CheckList:

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Currently need https://github.com/opensearch-project/notifications to fix this.
@Hailong-am

Currently need https://github.com/opensearch-project/notifications to fix this.
@Hailong-am

will take fix in notification and publish new jar/zip file

Currently need https://github.com/opensearch-project/notifications to fix this.
@Hailong-am

@lezzago you should unblocked.


Creating your own annotated dataset?

Any software you guys are using to create your own dataset?
Thanks.

I am using VOCdevkit to create the data set in the format of VOC2007. And the LabelImage is used as labeling tool. So far so good.

Try using Labelimg


[UPGRADE] Upgrade PostgreSQL to v13

Is your feature request related to a problem? Please describe.

PostgreSQL v10 has 20 vulnerabilities : CVE-2019-10211, CVE-2018-16850, CVE-2018-1115, CVE-2019-10208, CVE-2019-10164, CVE-2018-1058, CVE-2018-10925, CVE-2018-10915, CVE-2020-10733, CVE-2020-14350,CVE-2019-9193,
CVE-2020-14349, CVE-2019-10210, CVE-2018-1053, CVE-2018-1052, CVE-2017-15099, CVE-2020-1720, CVE-2019-10130
PostgreSQL v12 have also additional functional, performance and security improvements comparing to PostgreSQL v10

Describe the solution you'd like

Upgrade PostgreSQL to version 13
Backport to supported Epiphany versions (please consider creating new issues for this purpose)

Describe alternatives you've considered
None
Additional context
External components related to PostgreSQL should be also upgraded: Rep Mgr, PGAudit, pgpool, PgBouncer. There are separate tasks created for those upgrades. We also need to check possibility of this upgrade in context of our components used for replication and if those components are still needed(can be replaced by PostgreSQL embedded or other components functionality).

DoD checklist

[ ] Changelog updated
[ ] COMPONENTS.md updated / doesn't need to be updated
[ ] Feature has automated tests
[ ] Automated tests passed (QA pipelines)

[ ] apply
[ ] upgrade

[ ] Idempotency tested
[ ] Documentation added / updated / doesn't need to be updated
[ ] All conversations in PR resolved
[ ] Solution meets requirements and is done according to design doc
[ ] Usage compliant with license

After Postgres upgrading related components have to be checked: repmgr, pgpool, pgbouncer, pgaudit. So this task may be large, especially with testing part.
Let's discuss backporting tasks after implementation, during DoD.

we should split in 3 tasks:

installation
spike about upgrade
upgrade

Also ensure that we have all for ARM architecture before we start here.

Also start with checking that we can upgrade PostgreSQL and it's surrounding would be compatible.

Additional links:

https://repmgr.org/docs/current/upgrading-repmgr-extension.html

@erzetpe @mkyc could you clarify if we are going to support upgrade of postgresql or only apply.

In my opinion on every environment where the data has some business value the upgrade should be performed manually with someone that is experienced Postgres DBA.

Decision:

do upgrades automatically
ask user a question about backup before starting upgrade
check if there is a "dry run" option for upgrade. If yes, add it to preflight

if there is no "dry run" perform queries @to-bar mentioned, to check that unsupported types of data types do not exist in DB.

Merged to feature/upgrade-postgresql-13 branch, but moved to 'on hold' as waiting for other postgres parts to be merged.

testing in progress...
:heavy_check_mark: single machine: apply, re-apply
:heavy_check_mark: single node: apply, re-apply
:heavy_check_mark: master + replica: apply, re-apply


Revert "notebook: more clean results code and output" and fix CI

See individual commits.

For posterity, old approach:
[leseb@tarox~/cli][main] find notebooks/ -name "*.ipynb" -exec python scripts/validate_notebook.py {} \;
An error occurred while validating notebooks/Training_a_LoRA_With_Instruct_Lab.ipynb: Notebook does not appear to be JSON: '{\n "cells": [\n  {\n   "cell_type": "m...
[leseb@tarox~/cli][main] echo $?
0

With the new approach:
[leseb@tarox~/cli][main] find notebooks/ -name "*.ipynb" | xargs python scripts/validate_notebook.py
An error occurred while validating notebooks/Training_a_LoRA_With_Instruct_Lab.ipynb: Notebook does not appear to be JSON: '{\n "cells": [\n  {\n   "cell_type": "m...
[leseb@tarox~/cli][main] echo $?
123


Remove redundant friend class statements

Okay, obviously these are redundant because CI passed. Hmm. It seems that I have had a bug in my brain about this. It seems that a nested class has access to its parents' private interface. Somehow I did not think that was the case. I'm going to mark "dirty" all my brain storage about outer and inner classes' access to each other's private stuff across different languages.


Update dependencies

I just went through all the Cargo.toml files and updated all dependencies to their latest versions.

Workspaces don't support global dependency configuration. Each sub SDK has to specifiy the dependencies it depends on. I believe there is tooling that we can run in CI though to ensure we're not using dependencies anywhere that we don't want to use due to CVEs.

Perhaps I'm misunderstanding this, then: https://doc.rust-lang.org/cargo/reference/workspaces.html#:~:text=a workspace is a collection of one or more packages that share common dependency resolution (with a shared cargo.lock)%2C output directory%2C and various settings such as profiles.
If that's the case, yes, we'll need some sort of check to make sure we're all using the same ones - or at least newer than some baseline. I doubt Azure Pipelines and GitHub Actions support crates in their component governance checks - neither (which share the same managed host environments) even ship the rust toolset for hosted agents yet.


Unique indicies

The most incredibly unnecessary PR
feel free to reject if you don't like the aesthetics of the change
(Mainly made because it took me a quarter second to follow the lines)

Thanks for sending this. I prefer the current aesthetics.


Mocking mraa (python) for development?

Hi does someone us mocking for development with python and mraa?
many thanks

Sorry for me it was not clear that there exists some mocking already. https://github.com/intel-iot-devkit/mraa/blob/master/docs/mock.md


Fix: Instances and instance rootfs were stored indifferently

But the two are used very differently.

This would cause backward compatibility issues with existing nodes.


feat: remove signatures in container deployments by default

Make skopeo copy to remove the signatures of signed containers
by default to avoid build failures.
Depends: osbuild/osbuild#1906
Resolves: osbuild/bootc-image-builder#681
Signed-off-by: Miguel Martín<EMAIL_ADDRESS>

Please also canonically reference https://github.com/containers/image/issues/2599 (in the code as well)


TypeError: __init__() got an unexpected keyword argument 'save_safetensors'

Comon! I can't use safetensors if I want to quantize my merge with AWQ.
https://github.com/huggingface/transformers/blob/976189a6df796a2ff442dd81b022626c840d8c27/src/transformers/training_args.py#L835
https://github.com/huggingface/trl/blob/7a95cc86966ec5e59a9347384ba572f93e931827/trl/trainer/sft_trainer.py#L120-L146
Need to add **kwargs at the end of parameters to pass keyword arguments to transformers' Trainer.

Nevermind. If you face this issue (or a similar one), set save_safetensors (or the argument you want) as parameter when you initialize TrainingAguments...
from transformers import AutoModelForCausalLM, TrainingArguments
from trl import SFTTrainer

training_args = TrainingArguments(
    save_safetensors=False,
)

trainer = SFTTrainer(
    args=training_args,
)

Maybe we should fix this in general, what do you think @younesbelkada?

Maybe we should fix this in general
@lvwerra What do you mean?

From my POV, its my lack of understanding of the code's behavior that lead to this error. To be honest I was hyper focused on exporting my checkpoints in the correct format to later quantize my FP merge with AWQ when looking at the trl docs that I didn't even thought about the dedicated way to pass arguments to the trainer. Plus the fact I thought SFTTainer was a subclass of Trainer didn't helped me either...
Something that can be improved is the docs of trl. Like I said in my PR's review, it would be nice to have a small section dedicated to saving your checkpoint in supported formats with trl trainers. That way it's easy to find when you are looking for it. Because atm, without prior knowledge, you would have to piece it from a few different projects docs, the code and a lot of trial and error.


Possible regression: simultaneous multis will not call back

With 2.1.0, you could run two multis simultaneously and still have multi.exec call back.
In 2.2.0, multi.exec will not call back if two multis are in progress on the same connection at the same time.
Test case: https://gist.github.com/barn3y/b838cbed93042da24ca0
Sample results under 2.1.0:
$ node simultaneous_multi.js
exec multi 1
exec multi 2
multi 1 complete
multi 2 complete
<process ends>

Sample results under 2.2.0:
$ node simultaneous_multi.js
exec multi 1
exec multi 2
<process does not end>

Hm, I see. I thought about something like that just yesterday after releasing 2.2 and I have to check a couple of things. I'm uncertain if this was fully supported earlier, as the results might get mixed up under some conditions.

@barn3y I just tried your gist and a couple of different similar actions and all worked for me. What node version do you use?

Interesting! Admittedly I'm still on v0.10.38

That's it - I tried under 0.12.2 and it worked

I'll have a closer look at it.

I tried my test case again under node 0.10.38 on Mac OS 10.10.5, still no callbacks. Here's the output with NODE_DEBUG=redis on:
$ NODE_DEBUG=redis node simultaneous_multi.js
Hiredis parser not installed.
exec multi 1
Queueing multi for next server connection.
Queueing set for next server connection.
Queueing exec for next server connection.
exec multi 2
Queueing multi for next server connection.
Queueing set for next server connection.
Queueing exec for next server connection.
Stream connected <IP_ADDRESS>:6379 id 1
Using default parser module: javascript
Checking server ready state...
Send <IP_ADDRESS>:6379 id 1: *1
$4
info

True, that's even another issue than the one that I just fixed. The issue is only occuring because you're not yet ready :) Fix is incoming in a few minutes.

Amazing! Thank you very much!

@barn3y The issue should be fixed now. Would you be so kind and test the master right quick?

$ node simultaneous_multi.js
exec multi 1
exec multi 2
multi 1 complete
multi 2 complete

Yep working!

:) Thx


[Fix] Fix compiling error on windows

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
Please describe the motivation of this PR and the goal you want to achieve through this PR.
Modification
Please briefly describe what modification is made in this PR.
BC-breaking (Optional)
Does the modification introduce changes that break the back-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects.
The documentation has been modified accordingly, like docstring or example tutorials.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@wangruohui Seems that we can successfully build these ops, but will cause errors when importing them. Can you successfully train models using these modified ops in your Linux server?

Hi @wangruohui, can you try re-training one model which uses these ops to benchmark the result? For example, PartA2Net that uses iou3d. If you have trouble downloading the data, I can train it for you.

Hi @wangruohui, can you try re-training one model which uses these ops to benchmark the result? For example, PartA2Net that uses iou3d. If you have trouble downloading the data, I can train it for you.

Please train one with this PR. Actually, if the related model can test well, we can merge that. The modification should be fine.

Thank you very much!


Add support for status code 422

We're using an API that provides 422 responses for well-formed and understood requests that cannot be completed due to other errors (e.g. duplicate entry).  Is it possible to add this to the $statusTexts array.
i've added it locally as 422 => 'Unprocessable Entity',

Absolutely.  Would you be able to send a PR to implement this?


Stack exists but output does not.

Anyone know what I am doing wrong here....I keep getting the following error when I run the following command "stack_master apply --trace" I have two stacks...test and test2
/home/centos/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/stack_master-0.5.0/lib/stack_master/parameter_resolvers/stack_output.rb:24:in `resolve': Stack exists (test), but output does not: testVar (StackMaster::ParameterResolvers::StackOutput::StackOutputNotFound)
Why do I see testVar when I do: "stack_master outputs us-east-1 test --trace"?

*The below is in the template (test.yaml) I want to get it from....
"Outputs" : {
"testVar" : {
"Description" : "Test Variable",
"Value" : "TESTDATA"
}
}
In my parameters file for my second template I am referencing it as follows
testVar2:
stack_output: test/testVar
On my second template I am I referencing testVar2 as follows
Parameters section
"testVar2" : {
"Type" : "String",
"Description" : "Test variable that is read from the output of the test template."
}
Under tags section of EC2Instance
"Tags": [
{ "Key": "Name", "Value": {"Ref": "testVar2" } }
]

Have you applied the test stack with the changes?
If you update a stack with just some output, it won't actually update.

My fault. I wasn't referencing the parameter correctly.


Emit newGuid back to clients after sending message through private api

https://github.com/BlueBubblesApp/BlueBubbles-Server/blob/b8c483095beaad99399740ee6453abc98f0ff147/src/main/server/services/helperProcess/index.ts#L199
The chatGuid, tempGuid, and newGuid for the recently sent message are received by the server here ^.

not necessary with rest api